Monitoring and Evaluating the Food Security and Nutrition Effects of Agricultural Projects

1. Monitoring and Evaluating the Food Security and Nutrition Effects of Agricultural Projects Anna Herforth - June 13, 2013 - LCIRAH

2. Nutrition Indicators in Agriculture Survey: Preliminary results Anna Herforth and Terri Ballard FAO consultants Funded by the EU-FAO Improved Global Governance for Hunger Reduction Programme (2012-2015)

3. Nutrition indicators in Agriculture projects - Survey • Aim: to understand how agriculture projects will measure impact on nutrition: which indicators are being used, how, and why • Why? • Researchers may be able to connect if desired; informal technical support possible • Discussions about the evidence base can be informed about the current generation of new evidence • How: • Follow up on DFID-funded LCIRAH mapping study • 50% of studies identified were applicable • Excluded secondary data analysis, unspecified research, and research with no agricultural intervention

4. Survey questions based on theory • Key principles: • Incorporate explicit nutrition objectives and indicators into their design • Maintain or improve the natural resource base • Empower women • Facilitate production diversification, and increase production of nutritious  foods • Expand markets and market access for vulnerable groups • Incorporate nutrition promotion and education to enhance the impact of production and income • Collaborate and coordinate with other sectors See: http://unscn.org/files/Agriculture-Nutrition-CoP/Agriculture-Nutrition_Key_recommendations.pdf

5. Preliminary Results: Response • 77 project PIs contacted • 68 responded (88%) • 3 of these excluded • 2 did not complete • 82% with complete data

6. Preliminary Results: Indicators

7. Preliminary Results: Design • Majority are measuring in a comparison population • Majority are collecting qualitative data • Sample sizes range from 120 to 4000 (one 9000)

8. Early conclusions • Focus is on nutrition impact among producers • Total shift from previous generation of research regarding measurement of diet quality • Newly developed indicators get used • High number of studies measure nutritional status, but available sample sizes suggest low power • Seems to be attention to program impact pathways • Indicators chosen mostly because important to project goals, or evaluates a specific aspect of project • Interest in support for: “Adapting indicators to fit your particular study activities and aims”

9. What these results represent • This sample describes projects that set out to affect nutrition • It does not describe larger agriculture programs or investments where nutrition is not necessarily the primary goal

10. Monitoring and Evaluating the Food Security and Nutrition Effects of Agricultural Projects F. James Levinson and Anna Herforth Ag2Nut Community of Practice http://knowledge-gateway.org/ag2nut

11. Purpose Monitoring and evaluation of ag projects seeking to address food security and nutrition (in addition to their generally primary production objectives) would be highly desirable: • there is, to date, so little empirical data documenting successes and failures; and • possible adverse effects need to be identified and addressed rapidly within programs

12. Constraints • Impact on food security on nutrition not necessarily first priority of agricultural managers • Not familiar with how to measure • M&E in general is often not very strong • FAO/World Bank study (2010) of M&E in agriculture projects in Asia, Africa and Latin America

13. Common shortcomings of Ag M&E • M&E is often perceived as an externally imposed obligation with findings seldom integrated into management and action systems • Ag managers complain of unmanageable data collection and reporting demands • What M&E is carried out gives primary attention to physical achievements to the neglect of project outcomes • Inadequate institutional capacity consistently limit M&E (All of these, of course, are problems common to M&E in development projects more generally.)

14. A role for external Ag-Nut M&E teams? • Roles • identify sensible indicators to measure nutrition-relevant impact based on the type of activities in the program • carry out the key M&E necessary for tracking progress • feed back to the program management (monitoring) • Understand reasons for impact or lack thereof (eval) • could support nutrition-sensitive program design or adjustment

15. Sentinel Sites Geographically representative sentinel sites within the overall project area are one approach to M&E in ag projects with Ag2Nut interests Baseline data would be followed by the collection of quantitative and qualitative data at 6 month intervals from these sites, and from comparable sites in non-project areas.

16. Examples of data • Extent to which households have been reached/affected by the project • Relevant outputs specific to project (e.g. certain crops/foods consumed) • Household food insecurity levels • Dietary quality • Women’s empowerment • If it makes sense, nutritional status among young children and women All complemented by qualitative data to better understand the dynamics of project effects

17. Identifying harmful food security or nutrition effects • Employment levels have remained static or deteriorated; • Small producers have been excluded; • Household food insecurity has deteriorated (overall or seasonally); • Intra-household equity of income has declined; • The labor burden of women has increased; • Debt burden has increased; • In irrigation/water use projects, changes in water-borne diseases; • In livestock projects, changes in zoonotic disease • Harm to natural resources (particularly water, soil, biodiversity)

18. Data of particular interest to project managers Among possible indicators: • Access, use and satisfaction with services provided under the project • Changes in farmer assets • Percentage of households considering themselves better off now than 12 months ago  • Percentage of the labor force underemployed or unemployed

19. Conditions Such sentinel site data collection is likely to be useful if: • Good quality data can be sensibly aggregated and presented to project management in timely fashion • Challenge to collect data that are meaningful enough to be useful, and brief enough to be usable • Information indicating harmful effects or shortcomings in project implementation – will be addressed by project management

20. How would it be supported? Adequate staff and funding: • There is a need to identify Ag-Nut M&E teams capable of participating actively in an initial stream of nutrition-sensitive agriculture projects. • External funds probably necessary

21. In Sum Through the creative use of separately managed sentinel site-based M&E, it should be possible to: • Generate cooperative efforts • Generate much needed data • Document successes in nutrition-sensitive agriculture • Develop prototypes, training modules and TA mechanisms for subsequent use

22. Building the evidence base • Need more examples of successful programs • Not just nutrition outcomes, but win-wins with other goals • Ultimately, what is it that we want to scale up? • Probably not individual programs • Rather, principles that explain how individual programs have positive impact • Evaluations need to offer generalizable lessons learned

23. Discussion Ag2Nut Community of Practice http://knowledge-gateway.org/ag2nut