Impact Evaluation in Ethiopia

1. Impact Evaluation in Ethiopia Arianna Legovini Africa Impact Evaluation Initiative www.worldbank.org/africa/impact Addis Ababa June 16, 2006

2. The Africa Impact Evaluation Initiative • OBJECTIVES • Build government capacity for evidence-based policy-making, and • Mainstream rigorous impact evaluation in Bank-supported operations • APPROACH • THEMATIC : to learn how similar innovations perform in different contexts and apply these lessons to scale up successful experiences • COUNTRY : to build in-country capacity to measure the cost-effectiveness of different interventions and use this information to improve programs and policies • SECTOR/PROJECT: to test innovations in the short run and provide early feedback into program design and roll-out

3. Education in Kenya: addressing basic health problems Evaluations of cost-effectiveness require knowledge of a program’s costs as well as its impact. Because the Kenyan programs were conducted in similar environments, cost-effectiveness estimates from these randomized evaluations can be readily compared. • Deworming was found to be extraordinarily cost-effective at only $3.50 per additional year of schooling. • Provision of free uniforms would cost $99 per additional year of school participation induced. • The school meals program, which targeted preschoolers, cost $36 per additional year of schooling induced. • In other words, $10000 of public resources put in: • Deworming will keep 2800 children in school for an additional year • Uniforms or school feeding will keep only 100-277 children in school for an additional year • Kenya adopted de-worming treatment as part of its Universal Primary Education strategy

4. Africa Impact Evaluation Initiative THEMES & COUNTRIES • Infrastructure for Africa • Ethiopia, Malawi, Mozambique, South Africa, Tanzania • Strengthening Safety-nets • Angola, Ethiopia, Gambia, Kenya, Madagascar, Malawi, Sierra Leone, Tanzania, Zambia • Promoting Inclusion and Access to Justice • Ethiopia, Mozambique, Niger, Uganda • Education: what works and how • Kenya, Madagascar, Rwanda, Sierra Leone, Uganda • Health: what works and how • Ethiopia, Mali, Mozambique, Rwanda, Uganda • Roll-Back Malaria: improving take-up • Kenya, Congo, Eritrea, Ethiopia, Nigeria, Senegal, Sudan • Agriculture & environment: strengthen sustainable dev’t • Kenya

5. Ethiopia Impact Evaluation Program OBJECTIVE • Support government to evaluate progress of the PASDEP BY • Generating sound evidence on what works to achieve the targets in the PASDEP, • Building capacity in the country and among implementing institutions for evidence-based programs and policies, and thus, • Securing longer-term effectiveness of programs in achieving PASDEP outcomes.

6. How does Impact Evaluation fit within the M&E strategy of the government? • Monitoring is the best tool to track input to output changes (implementation efficiency) • Monitoring is not adequate to establish output to outcome linkages • Impact evaluation help identify the causal link between outputs and outcomes, the size of the effect and its cost (policy effectiveness) • Monitoring and evaluation are complementary tools

7. Alignment with PASDEP 1. Agriculture and rural development 2. Food Security 3. Infrastructure 4. Health 5. Education 6. Governance 7. Environment 8. Private sector and exportations development 9. Tourism and culture 10. Mining

8. Ethiopia Impact Evaluation Program 2 2 1 2 2 1 1 1 3 1 • Food security • Evaluation of the national food security program • Infrastructure • Evaluation of Electricity Access Expansion • Evaluation of ICT Expansion • Poverty Impact of Transport Operations on Road Corridors • Health • Evaluation of the Community-Based Child Growth Promotion • Evaluation of the health extension package and health extension workers (Basic Services Project component) • Evaluation of Malaria Booster Program • Governance • Basic Services Project • Women Development Initiative project • Gender & Law program

9. Health Evaluation of the Community-Based Child Growth Promotion • In 2005, the Government of Ethiopia initiated the Child Growth Promotion Component (CGPC) of the broader Food Security Project (FSP). • The CGPC seeks to improve the nutritional status of children below 2 years of age through growth monitoring and advice to the caregivers. • A baseline survey was conducted in July and August 2005 of 5706 children in 7 woredas/70 kebeles of SNNPR. • The sample is divided into two groups, both of which receive the core FSP services. One of the groups will also receive CGPC. • The baseline provides background information on the nutritional status and practices prior to the CGPC as well as some information on general FSP service delivery.

10. Health Protection of Basic Services (PBS) The PBS supports Ethiopia to make continued progress toward its MDG targets for health, education, and water, by: • Scaling up financial resource available at the local government level for the delivery of basic services, and • Increasing the distribution of commodities (such as immunization, family planning and malaria control) to health providers.

11. Health The evaluation will seek to establish: • Whether the PBS improves the allocation of funds across sectors and government levels, and the availability of resources to front-line services (before-after comparison using PETS/QSDS at municipal and facility level). • The extent to which approaches to strengthen voice of citizens and civil society groups bring about positive change in service delivery (before and after and relative to control using household surveys of perceptions/satisfaction). • The extent to which procurement of commodities improve front line availability and use of preventive health (before and after).

12. Malaria • Malaria Booster Programs scale-up malaria control efforts through the delivery of • Insecticide treated nets (ITNs) • Effective anti-malarial medication (ACTs), and • Intermittent preventive treatment (IPT)

13. Malaria • While the causal link between use of ITNs and effective anti-malarial medication and improved malaria outcomes IS well established, • The relative effectiveness of alternative delivery mechanisms and intervention packages to induce a change in people’s health behavior for the adoption and use of ITNs and effective anti-malarial medication IS NOT

14. Malaria • The evaluation will test alternatives for their impact on adoption behavior across several country contexts. • Alternative delivery modes include: • House to house visits • Mass ITN campaigns • ITN distribution through existing prime care services • Health vs school system-based delivery • And different emphasis on: • Improving human capital • Improving physical capital • Strengthening social capital

15. Malaria • Impact would be measured on: • Ownership of ITNs by household • Use of ITNs by children under five, pregnant women, and households • Children under five with fever receiving treatment with effective anti-malarial medication within 24 hours of the onset of symptoms • Percentage of pregnant women receiving intermittent preventive treatment (IPT) • All cause under-five mortality • Household income and consumption

16. Access to JusticeGender & Law • This regional program builds capacity for governments and civil society partnerships to support: • gender-responsive legal reform and institutional and policy environment; • gender-responsive legal education and literacy activities at the grassroots level; and • improved access to justice and legal services for the poor and socially vulnerable

17. Access to JusticeGender & Law The program has been implemented in 13 SSA countries and is planned for expansion. During this new phase the program plans to develop a rigorous framework of evaluation to measure to what extent the program is an effective mechanism for expanding women’s access to justice and to learn what features of the program are most successful. The evaluations are planned to take place in a number of countries (Niger, Uganda and proposed Ethiopia) from which a series of lessons will be drawn

18. Access to JusticeGender & Law Different communities will be randomly assigned to alternative : • Training and capacity building packages for legal service providers and practitioners, and • Direct provision of legal assistance (new) Impact will be measured on indicators of access to justice, outcomes of legal disputes, and economic outcomes.

19. GovernanceWomen Development Initiatives • The project, in operation since 2001, facilitates the formation and strengthening of self-help women’s groups, to enhance women’s capacity to act collectively and increase the social and economic welfare of their households.

20. Women Development Initiatives • When the evaluation started, the project was already under implementation, and therefore we could not take a baseline • There were doubts that a rigorous counterfactual could be established • But, during the first year of implementation, the project had rationed participation (one group of max 30 women per kebele) • The officials stated that to keep the number of participants under 30, they had carried out a lottery. • This offered an opportunity • We tracked down the women who had been rationed out (from official records and participant recollection) & • Administered a 1000 household survey to a random sample of participants and non-participants

21. Some findings from WDIP mid-term evaluation: Participatory stage • Economic: • Exclusion from many sectors of economic activity • Restricted freedom of movement • Wage gap and low bargaining power • Poor access to education, transport, financing • No access to water, electricity • Time diverted to household chores--in rural areas women report spending 3-4 hours a day collecting water and grinding grains • Judicial: • Traditional court system upholding tradition rather than national law • No access to formal courts • Limited recourse

22. Some findings from WDIP mid-term evaluation: Participatory stage • Political: • Political representation and government male dominated • Women involved in social activities (iqub, idir), seldom comfortable to participate in male dominated political gatherings • Household & social: • Traditional roles and mores • Prevalence of harmful practices (FGM, early marriage,..) • Domestic violence common and accepted

23. Some findings from WDIP mid-term evaluation: Quantitative analysis • Economic √ • WDIP participants earn 100 birr more on average per month than non participants • Income, in turn, significantly increases women’s say on household purchases • Project cost-effectiveness • Assuming the gains can be sustained for 5 years, average group size of 12, a transfer of $4,000 per group, and 40% project overhead, the Internal Rate of Return for this project would be about 15%.

24. Some findings from WDIP mid-term evaluation: Quantitative analysis • Mental health √ • On average women report being neither happy nor sad (3 on a scale of 1-5). 40% of women show signs of depression. • WDIP participants are happier. WDIP adds 0.7 to the mean, when controlling for several variables including employment (+1.1), income (no effect) and social capital (+.3 per additional person they can rely on)) • WDIP effect on happiness is not economic

25. Some findings from WDIP mid-term evaluation: Quantitative analysis • Social customs • Fewer participants (about 6 percentage points) uphold traditional rules that women should not • Farm (plough) • Ride animals or carts • Walk alone at night • Yet more participants uphold their husband right to beat them

26. Some findings from WDIP mid-term evaluation: Quantitative analysis • Traditional harmful practices: spill over? • Even though many more WDIP participants received training on THP, more of them are likely to practice them • Domestic violence • WDIP participants are as likely to suffer from domestic violence. The women who chose their husband suffer less domestic violence. • Political awareness • WDIP participants are as politically aware as non-participants (on political representation, civil rights, etc). Rural women are more aware.

27. Some findings from WDIP mid-term evaluation: Quantitative analysis • Participant recommendations to improve project: • Speed up delivery of funds • Relax minimum group size condition • Strengthen technology transfer and transfer of ideas

28. Follow-up WDIP evaluation • Collect follow up survey data and evaluate impact at project end • Come up with detailed project costs to estimate returns to the project accurately

29. THANK YOU