Training Workshop on Evaluation of Agricultural and Rural Development Programmes

1. Training Workshop on Evaluation of Agricultural and Rural Development Programmes Independent Office of Evaluation (IOE), IFAD Beijing, China 1 November 2013

2. Training Workshop Agenda Module 1: IntroductiontoDevelopmentEvaluation Module 2. Evaluation of Agricultural and Rural PovertyAlleviationInterventions (ARD) Module 3: HowtoManage and Conduct ARD Evaluations Module 4: A PracticalEvaluationExercise Module 5: Wrap-up and PracticalSuggestions

3. Module1. Introduction to Development Evaluation • Development Evaluation: Development + Evaluation • Evaluation of development interventions • Different types of development interventions • Different types of evaluation

4. Why are EvaluationsImportant? • Evaluations for accountability: to show for which purposes resources were used. Were the programme’s or project’s resources used to achieve its objectives? • Evaluations for learning: to draw lessons from experience and apply them to improve operations. What lessons can be drawn from the programme’s experience?

5. RelationsbetweenMonitoringand Evaluation • MONITORING: observing and collecting data on the implementation of an intervention, including inputs, outputs and outcomes, & on contextual factors • EVALUATION: on the basis of monitoring data and a set of criteria, assessing the effectiveness of interventions • Evaluation uses monitoring data and evaluation requirements determines which data should be monitored ME & EM

6. Evaluation Criteria • Relevance: were the objectives consistent with beneficiaries’ requirements, country needs, institutional priorities and partner and donor policies? • Effectiveness: to which extent the development intervention’s objectives were achieved, or are expected to be achieved?

7. Evaluation Criteria • Efficiency: how economically resources/inputs (funds, expertise, time, etc.) were converted into results? how reasonable were the costs? • Sustainability: will the results achieved be durable? Is it likely that those results will be sustainable? • Innovativeness: were innovative pro-poor approaches introduced by the intervention? • Replicability and Scaling-up: to what extent the interventions introduced have been (or are likely to be) replicated and scaled up?

8. Evaluation Criteria • The first three criteria (relevance, effectiveness and efficiency) are called “core performance criteria” • IFAD also uses other evaluation criteria such as “rural poverty impact”, “performance of partners” • Equity: how equitable is the distribution of benefits (by gender, social group or other attributes) is an additional criterion

9. ImpactDomains • Household income and assets • Human and social capital and empowerment • Food security and agricultural productivity • Natural resources and the environment • Institutions and Policies See key questions for each of the impact domains in Table 7, ps. 37 & 38 of the IFAD Evaluation Manual.

10. DifferentTypesofDevelopmentInterventions • Projects: achieving development objectives (e.g. reducing rural poverty) through a set of instruments or components (e.g. microfinance) • Programmes: set of projects • Policies: sets of objectives that are implemented through a set of programmes or projects • POLICIES  PROGRAMMES  PROJECTS

11. Development Evaluation andDifferentTypes of Evaluation Cont.’ • Ex-ante evaluation (appraisal) • Process evaluation • Mid-term evaluation • Completion evaluation • Ex-post evaluation – Impact evaluations • ----------------------------------------------- > time • Formative (to improve) Summative (to judge)

12. Reference e-materials • Conducting Quality Impact Evaluations Under Budget Time and Data Constraints (2006) http://siteresources.worldbank.org/EXTEVACAPDEV/Resources/4585672-1251461875432/ conduct_qual_impact.pdf • Book Impact Evaluation in Practice (2011) Eng/Fr/Sp. http://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1295455628620/ Impact_Evaluation_in_Practice.pdf • IFAD Evaluation Manual www.ifad.org/evaluation/process_methodology • Glossary of Key Terms in Evaluation and ResultsBased Management www.oecd.org/dataoecd/29/21/2754804.pdf http://www.oecd.org/dac/evaluation/evaluationandaideffectivenessseries.htm (Chineseversionunder doc. No. 6)

13. Module 2: Evaluationof Agricultural and Rural PovertyAlleviationInterventions • Interventions whose objective is to alleviate rural poverty, through agriculture and rural development • Programmes and projects with different types of components: microfinance, agricultural extension, support for rural non agricultural activities • Area based interventions and country wide programmes.

14. Examples of Evaluationsof ARD • Evaluation of a rural development project in the Northeast of Brazil • Country Portfolio Evaluation (CPE) of IFAD in Brazil and CPE for China • Evaluation of a rural microfinance project in Colombia • Thematic evaluation of agricultural extension services • IFAD and AfDB joint evaluation of ARD in Africa • IFAD Completion evaluation of the West Guangxi Poverty-Alleviation Project http://www.ifad.org/evaluation/public_html/eksyst/doc/prj/region/pi/china/guangxi

15. IFAD’sMethodologyforthe Evaluation of ARD • IFAD’sEvaluation Manual, Methodology and Processes www.ifad.org/evaluation/process_methodology • The manual buildsoninternationallyrecognizedevaluationpractices and criteria • Guidelinesforprojectcompletionreportvalidation (PCRVs) and project performance assessment (PPA), complementthe Evaluation Manual http://www.ifad.org/evaluation/policy/new_policy.htm whereyou can alsofindIFAD’s “Guide to Project M&E”

16. IFAD’sMethodologyforthe Evaluation of ARD: Evaluation Criteria • Project performance • Relevance • Effectiveness • Efficiency • Other Performance Criteria • Sustainability • Promotion of pro-poororientation, replication and scaling-up Drawingupon 1,2 and 3, overallprojectachievement of theproject • Rural Povertyimpact • Householdincome and assets • Human and social capital and empowerment • Foodsecurity and agriculturalproductivity • Natural resources and theenvironment • Institutions and policies • Performance of Partners • IFAD • Government • CooperatingInstitution • NGO/CBO

17. IFAD’sMethodolgyforthe Evaluation of ARD: Rating System • A six-point rating systemfortheevaluationcriteria as a waytoquantifyqualitativejudgements of evaluators • 6: highlysatisfactory 5: Satisfactory • 4: moderatelysatisfactory ------------------------------------------------ • 3: moderatelyunsatisfactory • 2: unsatisfatory 1: highlyunsatisfactory

18. Other Approachestothe Evaluation of ARD • Goal-free evaluation Itsstrength in avoiding a limitedfocusonobjectives can becompensatedbyanexplicitfocuson “unintendedconsequences” Weak in accountability. • Impactevaluationwith control groups Explicitcounterfactualtomodelthewithoutprogrammesituation

19. M&E of innovation, replication and scaling up

20. Evaluation Challenges • The contribution of the IFAD portfolio to innovation and scaling up is important to justify IFAD’s lending to China, - the financial volume of which is insignificant from a national financial perspective. However, extremely challenging to evaluate this contribution • Project designs seldom have specific and concrete innovation agenda – what is an innovation in the local context? • Often innovations emerge during implementation in response to problems/opportunities – by farmers, NGOs, researchers, implementers, IFAD supervision missions etc. • Often, implementation and project M&E systems do not clearly identify innovations and lack systems to confirm value added/viability of innovations and their replication and scaling up - M&E methodology for innovations is the same as for standard interventions • A large part of the scaling up may take place after project completion and in areas not covered by the project

21. Identifying innovation and scaling up and the processes involved Which interventions are innovative and what is the innovation process? Where and how are innovations scaled up – processes and mechanisms?

22. Innovation and scaling updifferent processes and evaluation approaches Innovation Replication and Scaling up • Define the hypothesis of the innovation – ”a scientific experiment” • In addition to before-after, with/without analysis may be necessary • Engage independent third party to assess viability, relevance and applicability • If viable and relevant, formulate scaling up strategy • Spontaneous market-driven replication with some facilitation or managed scaling up, requiring substantial support, - policy dialogue, KM, partnerships etc. • Who are the partners and the financial sources required in the scaling up process? • How can the scaling up process be monitored – proxy indicators, data sources etc.

23. M&E of support for enterprise development and employment generation&The Standard of the OECD Donor Committee on Enterprise Development (DCED)

24. Rural poor will increasingly depend on enterprise development and employment • Current strong trends of land consolidation, agricultural commercialisation, outmigration and ageing of farm population are changing rural socio-economic landscape • Small commercial farms and rural enterprises, including agro-industries, are becoming important players • For their livelihoods, rural poor will increasingly depend on the welfare system, leasing out their user rights, remittances and employment on- and off-farm (often seasonal) • For development partners, obvious niche to support development of MSMEs and cooperatives, creating employment for the poor – but how to monitor and evaluate • DCED Standard provides a common tool to design support and measure results and impact of support for enterprise development

25. Universal Impact Indicators • Scale: Number of target enterprises who realize a financial benefit as a result of the programme’s activities per year and cumulatively. The programme must define its “target enterprises.” (may be farms, MSMEs, cooperatives etc but also poor workers) • Net income: Net additional income (additional sales minus additional costs) accrued to target enterprises as a result of the programme per year and cumulatively. In addition, the program must explain why this income is likely to be sustainable. • Net additional jobs created: Net additional, full time equivalent jobs created in target enterprises as a result of the programme, per year and cumulatively. “Additional” means jobs created minus jobs lost (displacement and crowding in). “Per year” comprises 240 working days. The program must explain why these jobs are likely to be sustainable. Jobs saved or sustained may be reported separately. These enterprise-level impact indicators can be translated into household-level impact but it is a demanding process, particularly with respect to attribution, because households may have multiple income streams.

26. Eight Control Points • Articulating the theory of change/results chain • Defining the indicators of change • Good measurement practices • Handling attribution • Capturing wider changes in the systems or markets • Relating to programme costs • Reporting results (internal/external) • Managing the system for results measurement

27. Articulating the Results Chain Obligatory • An appropriate, sufficiently detailed and logical results chain(s) is articulated explicitly for each of the interventions. • Each results chain is supported by adequate research and analysis. • Mid and senior level programme staff are familiar with the results chain(s) and use them to guide their activities; key partners can explain the logic of interventions. • The results chain(s) are regularly reviewed to reflect changes in the programme strategy, external players and the programme circumstances. Recommended • The results chain(s) include the results of broader systemic change at key levels. • The research and analysis underlying the results chain(s) take into account the risk of displacement.

28. Example of Results Chain www.Enterprise-Development.org

29. Module 3: Howtomanage and conductARD evaluations • The first strategic decision is choosing WHAT to evaluate. Decide on the basis of expected value added for accountability and/or learning • The terms of reference should clarify expectations concerning the evaluation that will be carried out. Answers to WHY, HOW, and WHEN TO EVALUATE? See “Writing Terms of Reference for an Evaluation: A How-To Guide” https://ieg.worldbankgroup.org/evaluations/evaluation-capacity-development-ecd

30. Howtomanage and conductARD evaluations Cont.’ • Selectingthetype(s) of evaluationto be used Methodsadequatetothecircumstances (purpose, availability of time and resources). See IFAD Manual and “Monitoring and Evaluation: Some Tools, Methods and Approaches” https://ieg.worldbankgroup.org/evaluations/evaluation-capacity-development-ecd

31. Howtomanage and conductARD evaluationsCont.’ • Consulting with stakeholders Why? To get their insights, and to nurture their interest in the evaluation (which will facilitate the use of the evaluation). • Identifying key stakeholders: Partners, intended beneficiaries, government officers, civil society representatives

32. Howtomanage and conductARD evaluationsCont.’ • Reviewing relevant documentation In addition to PROJECT DOCUMENTS, reports on the country, internet searches for thesis. other evaluations and academic papers on the project/programme area and on the instruments used (e.g., rural microfinance or agricultural extension in the area) During the review prepare notes and a bibliography, as inputs for the report.

33. Howtomanage and conductARD evaluationsCont.’ • Applying evaluation criteria • Relevance, Effectiveness, Efficiency, Sustainability, Innovativeness, Replicability and Scaling-up, Rural Poverty Impact , Performance of partners • Ratings for the criteria • Rating overall project achievement • Providing a justification for the ratings

34. Howtomanage and conductARD evaluationsCont.’ • Drafting and reviewing evaluation reports • See section 3B of IFAD Evaluation Manual,p.30 • Using evaluation reports • Evaluation reports are important INPUTS for evaluations, as indicated before under “reviewing documentation”. • Evaluation reports are also key OUTPUTS of the evaluation process. Show in a clear way the links between findings and recommendations to facilitate the use of the evaluation report.

35. Module 4: Exercise: Evaluation of theNorthernSomeland ARD Project • With support from the XFund, the 10,000,000 euros Northern Someland ARD project aimed to provide low prized fertilizers to 5,000 farmers in the northern region of Someland, a land-locked country, where small farmers are fruit producers. • The objective of the project was to improve the welfare of the farmers and their families. A project implementation unit was set up to organize the delivery of fertilizers and to establish links between the farmers and local institutions involved in the provision of complementary services. • How would you evaluate this project? What would you do to evaluate the project? Please discuss with your colleagues and prepare a 5 minutes presentation

36. Module 5: Wrap-up and Practical Suggestions • Review key points discussed • Your suggestions for ARD evaluations

37. Feedbacksurveyonthe training • What have you learned? • Do you have any suggestion for future training on evaluation in China? • To what extent to you think that the objectives of the workshop were achieved?

38. Thanks for your participation!