ex ante program evaluation

1. Ex Ante Program Evaluation Petra E. Todd University of Pennsylvania (Based on joint work with Ken Wolpin)

3. Ex Post Evaluation Methods Evaluate program impacts after implementation Alternative Approaches: Randomization Difference-in-Difference Matching (Cross-sectional and Difference-in-difference) Control function methods Regression-Discontinuity IV Methods, MTE, Local IV (LIV), LATE All methods require data on a treatment group and on a comparison group

4. Advances in Ex-post Evaluation Matching Does not require functional form assumption on the outcome equation (Rosenbaum and Rubin, 1983) Propensity scores can be estimated semi-parametrically, (Heckman, Ichimura and Todd, 1997, Buchinsky, 1998) Regression-Discontinuity (RD) method Requires discontinuity in the probability of receiving treatment (Hahn, Todd and Van der Klaauw, 2001) Does not require specifying the functional form of the outcome equation Control function methods Implementable without distributional assumptions on the error terms in the participation and outcome equations (e.g. Heckman, 1980, Newey (1988), Andrews (1991)) Usually requires an exclusion restriction

5. IV estimators Has LATE interpretation under weak assumptions (e.g. Imbens and Angrist, 1994) MTE, LIV estimators (Heckman and Vytlacil (2005)) Require a continuous instrument Permit investigation of program impact heterogeneity Relax assumptions about additive separability of error terms

6. Predict program impacts prior to implementation Needed for optimal program design and placement Requires simulating program effects and costs (take-up rates) Experimental approach often not feasible (high cost, time delay) Identify range of potential impacts, helpful in choosing sample sizes for future evaluation Evaluate effects of counterfactual programs Study how impacts change if parameters of an existing program are altered For example, changing school subsidy levels Evaluate effects of longer terms of exposure than are observed in the data

7. Using Static Models Forecast demand for a new good prior to its being introduced into the choice set e.g. McFadden (1977) � BART subway Impose structure on utility function and on the distribution of the error terms (e.g. multivariate probit or logit) Forecast effect of changing the characteristics of a good Berry, Levensohn, Pakes (1985) � changing car characteristics (e.g. price, fuel efficiency)

8. Using Dynamic Models Impose functional form assumptions on utility function and on the joint distribution of error terms Evaluate model performance by comparing forecast based on structural predictions to experimental results Wise (1985) : effect of housing subsidy on housing demand Lumsdaine, Stock and Wise (1992): retirement bonus Lise, Seitz, and Smith (2003) � welfare bonus program Todd and Wolpin (2006) � effects of Mexican school subsidy program

9. Early Efforts to Relax Functional Forms for Ex Ante Evaluation Marschak (1953) and Hurwicz (1962) Observe that it is not necessary to know the entire structure of the problem to answer certain policy questions (studied tax changes) Recognize that an economic model is required to extrapolate from historical experience

10. More recent efforts Ichimura and Taber (1998,2002) Present general set of conditions under which nonparametric policy evaluation is possible Estimate the effects of a college tuition subsidy using tuition variation in the data Heckman (2000, 2001) Discusses �Marschak�s Maxim� Provides some new examples where nonparametric assessment of new policies is feasible Blomquist and Newey (2002) Nonparametric estimation of labor supply responses with nonlinear budget sets. Bourguignon, Ferreira, and Leite (2002) Use reduced form random utility model for forecast impact of school subsidy program in Brasil

11. Goals of this paper Consider nonparametric and semiparametric methods for evaluating the impacts of social programs prior to their implementation Illustrate use of behavioral models in evaluating effects of hypothetical programs Show that fully nonparametric strategy sometimes feasible Suggest estimation strategy based on a modified version of the method of matching Study the performance of the methods using data from the PROGRESA school subsidy experiment in Mexico Compare ex ante predictions to experimentally estimated impacts Evaluate the effects of counterfactual programs Changes to the subsidy schedule Unconditional income transfer

17. Combination wage subsidy and income transfer

18. Estimation

19. School attendance subsidies when child wages are observed

21. Required Assumption

22. Intent-to-treat estimator

23. Coverage Rate and Treatment-on-the-Treated Estimator

24. Extension to multiple children, fertility assumed to be exogenous

25. Multiple children, endogenous fertility

29. Example: Only accepted child wages observed, selection on unobservables

34. Extension to Two Period Model

36. Description of PROGRESA, Oportunidades(Programa de Educacion, Salud, y Alimentacion) Large scale anti-poverty program begun in 1997 originally provided aid to about 10 million poor families (40% of all rural households) operates in 31 states with a budget ? 1 billion U.S. dollars Recent expansion into urban areas Provides educational grants to parents (mothers) to encourage children�s school attendance. Must attend 85% of days Benefit levels increase with grade level, higher for girls Subsidies amounted to about 25 percent of average annual income over all children that actually attended in the first year of the program.

37. Experimental design and Data Program implemented as a randomized social experiment 506 villages randomly selected from 7 states in Mexico (of 31 states) 320 randomly assigned to the treatment group and 186 to the control group Controls incorporated after third year of the program, but not told about the program until incorporated Use Oct. 1997 Baseline and Oct. 1998 Follow-up Surveys Data elements: school attendance and grade attainment, information on employment and wages (to construct total family income net of child income) Village level data on the minimum wage paid to daily laborers Subsample children from program eligible families, age 12 to 15 in 1998, who are the son or daughter of the household head, and for whom information is available in the 1997 and 1998 surveys.

38. Overview of Empirical Results Compare the predicted ex-ante impacts to the actual impacts (These are ITT impacts) Multiple child model Single child model Implement exact matching on age and gender Evaluate effects of counterfactual programs Doubling subsidy, cutting subsidy by 25% Unconditional income transfer of 5000 pesos per year (about half of family income)

47. Conclusions and future research Considered nonparametric methods for evaluating the impacts of social programs prior to their implementation. Behavioral models required to justify particular estimation strategies. Estimators are modified versions of matching estimators. Require stronger assumptions on unobservables (future research) In some cases, can accommodate other endogenous choices Studied performance of the ex-ante prediction method using data from the Mexican PROGRESA experiment. The predictions are generally of the correct sign and usually come within 30% of the experimental impact. Predictions more accurate for girls than for boys Counterfactual programs Changes in subsidy schedule � enrollment of older children more elastic with respect to level of subsidy Unconditional income transfers unlikely to be effective