CROSS-COUNTRY WORKSHOP FOR IMPACT EVALUATIONS IN AGRICULTURE AND COMMUNITY DRIVEN DEVELOPMENT Addis Ababa, April 13-16,

1. CROSS-COUNTRY WORKSHOP FOR IMPACT EVALUATIONS IN AGRICULTURE AND COMMUNITY DRIVEN DEVELOPMENT Addis Ababa, April 13-16, 2009 A CONCEPT NOTE FOR IMPACT EVALUATION OF THE ETHIOPIA NEWLY RE-DESIGNED FOOD SECURITY PROGRAM

2. Team Members • Ato Kebede Tafesse, Food Security Coordination Directorate • Ato Ahmed Adem, Pastoralist Community Development • Ato Behailu Shewangzaw, Food Security Coordination Directorate • Ato Hailu Ankisso, Food Security Coordination Directorate • Ato Welelaw Sendeku, WB-CIDA Food Security Project • Ato Zeleke Aged, Food Security Coordination Directorate • Ato Zena Habewold, Food Security Coordination Directorate • Dr. Hussien Hamda, Addis Ababa, University • Ato Isayas Abate, The World Bank • Ato Getahun Tafesse, CIDA • Ms. Laketch Mickael, The WB • Ms. Maddalena Honorati, The WB • Ms. Marie Gaarder, 3ie • Mr. Ludovic Subran, The WB

3. Background • Long period of Emergency Assistance • Decision to differentiate Chronic vs Transitory Food Insecure people • Food Security Program that has three pillars was implemented during the past five years • Only one of the pillar program is under regular evaluation • The Current Redesign Exercise is aimed at Integrating the three pillar programs into one.

4. Framework for Welfare Status Categorization

5. FOOD SECURITY PROGRAM COMPONENTS Title • Productive Safety Net • 1 Direct Support - Cash or Food transfer • 1.2 Public Works - Cash or Food transfer • Household Investment & Financial Service • Complementary Community Infrastructure Investment (newly incorporated) • Land Access

6. Figure 1: Graphic Representation of the Food Security Programme Logical Hierarchy of Objectives 8.3 million chronically cally food insecure people graduate to food secure status. 6.7 million more people have improved food security. Food availability improved Food access impr oved Vulnerability to Shocks Food utilization improved through Decreased through asset through increased on - farm increased income from cash through health and nutrition production and productivity (food crop and livestock sales and protection/ promotion and timely interventions (mostly outside of and cash crops and livestock) non - farm income generation. safety net interventions. the scope of the FSP) Interventions, of l non - FSP institutions Resettlement Programme Productive Safety Net Programm e Other Food Security Programmes – Up to 440,000 chronically food insecure – HH assets protected and – Assets of HHs in 290 woredas households resettled and provided with community assets built in 263 food promoted through d iverse insecure woredas through timely and basic materials and services. development interventions. consistent safety net activities. Selected Outputs: Selected Outputs: Selected Outputs: § Sites selected, HHs identified and § HHs with available labour § Support given for cropping and resettl ed. participate in public works based livestock production. § Land provided and support given on local priorities and economic § Water resources devel oped and for cropping and gardening. opportuniti es. irrigation systems installed. § Livestock production supported § Food or cash provided in timely § Opportunities introduced for and oxen provided for ploughing. manner. credit and savings. § Social services established and § HHs without labour supported § Off - farm income generation made functional. with direct transfers of food/cash. promoted/supported. § Environmental protection assured § Shocks mitigated with timely § Market links created and/or and monitored. delivery of food/cash. reinforced.

7. FOOD SECURITY PROGRAM COMPONENTS Title • The Safety Net Program is planned to address: • 8.3 million chronically food insecure people in 290 woredas • The Credit Support, Ag. Extension, etc (household investment package) will be primarily targeted to Safety Net beneficiaries

8. PRIORITIES FOR LEARNING: INTERVENTIONS Title • Does provision of Credit makes a difference? • Do different types of households benefit differently from Credit? • Is Credit more effective in certain type of intervention setting than other?

9. EVALUATION QUESTIONS Title • Does Provision of Credit on top of Safety Net Makes a Difference in making progress towards Graduation? • Magnitude • Speed • Sustainability

10. EVALUATION DESIGN Title Sample Population: Safety Net Kebelles (households receiving the transfer cash, food or mix) within the Kebelle Community/ kebelle is unit of Intervention for the credit component and therefore will be unit for randomization The Plan is to Use a Randomized design that will be based on: • The Roll-out / Phasing of the Intervention by the GoE over the next year • Excluding current RUSACA kebelles that have a pseudo-household investment component HH level is Unit of observation/ analysis for Outcome Measurement namely: Asset value index and livelihoods measurement, both currently used as benchmark indicators for measuring ‘graduation’

11. EVALUATION DESIGN Title INDICATORS • Asset value index currently used as benchmark indicators for measuring ‘graduation’ • Livelihood measurement Additional indicators that can potentially be considered • Human capital • Anthropometric • Link with the national poverty line (per adult expenditure that takes into account minimum calorie intake and non-food need)

12. EVALUATION DESIGN Title Sampling Frame • A total of 290 woreda, composed of 15 Kebelles on average, ie 700 Households on average per Kebelle • Approx 3000 Kebelle to be considered (excluding RUSACA ones), out of which (tbd) 300 will be included in the evaluation design, with a hundred household in the each Kebelle (tbd) • Control group: Kebelle with safety net support, without access to rural finance

13. EVALUATION DESIGN Title Effects that have to be Captured/ Controlled: • Households in the Control group should be households that are eligible, offered the credit and have accepted to take credit but did not get the credit yet • Purposeful manipulation of the selection criteria by potential beneficiaries • In the survey design, households will be asked on whether they would have received Credit if they had been offered

14. EVALUATION DESIGN Title • Stratification/ Results disaggregation • Different Levels of Support provided under household investment (three credit categories: subsidized I, subsidized II and commercial rate) will be offered according to household food insecurity levels – see ‘further options’ slide) • Different timeline for support (3-, 6- and 9-months) • Communities/ kebelles that have acquired complementary Infrastructure investment –vs- those without (see ‘further options’ slide) • Cash beneficiaries, Food and mix Beneficiaries • Last outcomes for male-headed and female-headed households will be particularly compared

15. EVALUATION DESIGN Title • Assumptions: • Households who are offered Credit will take it (so far acceptance rate close to 100%) • The financial institutions are new ones so that a treated kebelle is not at the expenses of a control kebelle (available ‘supply’)

16. SAMPLING AND DATA Title • Sampling will be done among Safety Net kebelle/beneficiaries • The Main Data Source will be the on-going IFPRI Survey for both baseline and follow-up (annual panel data) • Need for Over-sampling and adding of additional Module (set of questions)

17. SAMPLING AND DATA Title • Explore potential linkage with other surveys such as • CSA HHICE Survey • CSA Welfare Monitoring Survey • Agricultural Census • Planned LSMS

18. TIMELINE FOR IMPACT EVALUATION Title • Evaluation Concept Note to be discussed at the next Program Design Workshop in May, 2009 • Baseline will be conducted end 2009 and/ or early 2010 • Surveys will be repeated annually

19. IMPACT EVALUATION TEAM: STAFFING Title • The GoE and Donor Joint Food Security Monitoring and Evaluation Technical Task Force will steer the Process – METT • METT will also get involved in technical issues of survey design, methodology and sampling with close support from local Academic/ research institutes • The GoE and Donor Joint Food Security Coordination Committee (JCC) will deal with policy and resource issues

20. IMPACT EVALUATION TEAM: STAFFING Title • IFPRI will be responsible for Survey Design, follow-up on quality of data collection and undertaking in-depth analyses • The CSA will be responsible for Data Collection and production of Basic Report • The GoE and Donor Joint Food Security Coordination Committee (JCC) will deal with policy and resource issues • Technical Assistance is expected from 3ie, AADAPT and DIME

21. BUDGET Title • The Study is expected to be financed through the Multi- Donor Trust Fund (MDTF) • To Pickyback on IFPRI, about 300,000 USD is needed • 3ie, AADAP or DIME to cover TA on • Impact Evaluation Design • Subject matter (Rural Finance) expert

22. Amasagenallehu

23. FURTHER OPTIONS Title • Compare impact of different programs in addition to the safety net: household investment, community infrastructure and household investment+community infrastructure • Check for opportunity for proper impact evaluation (randomized phasing or non experimental toolbox). • If take-up much less than expected among households, a randomized encouragement design (two-step approach) could be explored to capture both the impact of ‘offering the package’ and the impact of ‘actually borrowing’ • Unfortunately, rural finance packages are targeted to different strata among the food insecure and randomization would need to be possible in practice (offering the credit to households irrespective of food insecurity characteristics)