IMPACT EVALUATION FOR POLICY MAKING A look at LAC countries with weaker research capacities

1. IMPACT EVALUATION FOR POLICY MAKINGA look at LAC countries with weaker research capacities María Laura Alzúa Habiba Djebbari Martín Valdivia Session: Demand and Supply of Impact Evaluations Mind the Gap: From Evidence to Policy Impact Cuernavaca, June 15-17, 2011

2. Outline • Why do we need to evaluate the impact of programs? • The Evidence to Policy Process • Key Hypotheses of this study • What we do? • Some preliminary results

3. The need for impact evaluations • Money for foreign aid/taxpayers have alternative uses for a set of goals • Social behavior and economic policies • Modeling incentives, behavioral constraints and the objectives of policies/programs (Karlan & Appel, 2011; Banerjee & Dufló, 2011) • Hidden/unexpected effects • The challenge to identify a sound counterfactual • What would have happened to the beneficiaries in the absence of the program? • Experimental methods • Non-experimental methods (PSM, IV, RDD, etc.)

4. Research questions • How many studies have been done and on what topics or policy issues and with what methodologies (RCT, q-experimental, IV, etc.)? What are the emerging fields or sectors of interested being evaluated? • Who is doing it? The role of local universities and research centers vis-à-vis the work by northern-based researchers and institutions and initiatives such as J-PAL, IPA, and donors and the multilaterals, etc. • Who is funding it (3IE, Gates, MCC, the multilateral banks)? How have their project approval processes being modified to give stellar importance to impact evaluation designs? • How are they being used to shape policy? What are the institutional arrangements that can better insert impact evaluations into policy design?

5. Key hypotheses • The number of programs with rigorous IEs have increased significantly in the region, especially in countries with more capacities for research and in policy making • Institutionalization of the use of IEs for policy making has been key

6. What we do? • Collect all IE studies done in the region since 1995, and especially in the following set of countries • South America: Bolivia, Ecuador, Paraguay, Perú • Central America: El Salvador, Guatemala, Honduras, Nicaragua • Caribbean: Dominican Republic, Jamaica, Trinidad and Tobago, Guyana • Sources: • Bouillón & Tejerina (2007) • Most common academic papers databases: IDEAS/RePEC, EconLit and JSTOR, and SSRN Randomized Social Experiments • Other databases: World Bank DIME, IADB OVE/Development Effectiveness, IPA, J-PAL, 3ie, MCA, the Bill and Melinda Gates Foundation, IFPRI

7. What we do? • Final selection criteria restricts the sample to those with reasonable identification strategies • We exclude studies with identification strategies based on community characteristics as IVs, PSM without baseline, etc. • Three case studies: Dominican Republic, El Salvador, Peru • What are the restrictions to the implementation and use of rigorous IEs? • Are there conditions for the institutionalization of IEs? (Briceño and Gaarder, 2009) • Supply of training courses on IE in LAC • Sources: IPA, J-PAL, World Bank, IADB, IEN-LACEA • Who are they serving? Local policy makers , researchers