1 / 45

Impact Evaluation Methods: Randomization, IV, Regression Discontinuity

Africa Program for Education Impact Evaluation. Impact Evaluation Methods: Randomization, IV, Regression Discontinuity. Muna Meky Impact Evaluation Cluster, AFTRL. Slides by Paul J. Gertler & Sebastian Martinez. AFRICA IMPACT EVALUATION INITIATIVE, AFTRL. Measuring Impact.

laken
Télécharger la présentation

Impact Evaluation Methods: Randomization, IV, Regression Discontinuity

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Africa Program for Education Impact Evaluation Impact Evaluation Methods: Randomization, IV, Regression Discontinuity Muna Meky Impact Evaluation Cluster, AFTRL Slides by Paul J. Gertler & Sebastian Martinez AFRICA IMPACT EVALUATION INITIATIVE, AFTRL

  2. Measuring Impact • Randomized Experiments • Quasi-experiments • Randomized Promotion-Instrumental Variables • Regression Discontinuity • Difference in difference – panel data • Matching

  3. Randomization • The “gold standard” in evaluating the effects of interventions • It allows us to form a “treatment” and “control” groups • identical characteristics • differ only by intervention • Closest approximation to counterfactual

  4. Random Assignment • Each eligible unit has the same chance of receiving the intervention • Mimics chocolate experiment • Allows us to compare the “treatment” and “control group”

  5. Random Assignment vs. Random Sample • Random Assignment • Are the observed results due to the intervention rather than other confounding factors? (internal validity) • Random Sample • Do the results found in the sample apply to the general population/are they generalizable? (external validity)

  6. Randomization Random Sample (external validity) Randomization Randomization Random Assignment (internal validity)

  7. Example of Randomization • What is the impact of providing free books to students on test scores? • Randomly assign a group of school children to either: - TreatmentGroup – receives free books - Control Group – does not receive free books

  8. Randomization Random Assignment

  9. How Do You Randomize? • At what level? • Individual • Group • School • Community • District

  10. When would you use randomization? • Universe of eligible individuals typically larger than available resources at a single point in time • Fair and transparent way to assign benefits • Gives an equal chance to everyone in the sample • Good times to randomize: • Pilot programs • Programs with budget/capacity constraints • Phase in programs

  11. Oportunidades Example • Randomized treatment/controls • Community level randomization • 320 treatment communities • 186 control communities • Pre-intervention characteristics well balanced

  12. Oportunidades Example

  13. Difference in Means

  14. Impact Evaluation Example: Randomization vs. Other Methods

  15. Other Analyses Often Lack Internal Validity • Enrolled vs non-enrolled • The baseline characteristics will be different because people have chosen which group they want to be in • Before and after • There are often other interventions going on at the same time

  16. Measuring Impact • Randomized Experiments • Quasi-experiments • Randomized Promotion-Instrumental Variables • Regression Discontinuity • Difference in difference – panel data • Matching

  17. When Do We Use Random Promotion? • Common scenario: • National Program with universal eligibility • Voluntary enrollment in program • Can not control who enrolls and who does not

  18. Randomized Promotion • Possible solution: random promotion or incentives into the program • Information • Money • Other help/Incentives • Also called • Encouragement designs • Incentive schemes

  19. Study Components • Intervention • Chocolate • Randomized Promotion • Encouragement to take chocolate

  20. Example of Promotion Design for SATs • What is the impact of supplementary learning material on student test scores? • Outcome • Student test scores • Intervention • Supplementary learning materials • (all teachers can access these) • Randomized promotion • Letters encouraging students to use materials • (sent to randomly assigned teachers)

  21. What Information Does Randomized Promotion Give Us? • How effective is the treatment? • We can analyze the effect the treatment had on the outcome in the sub-group of subjects who would not have used the intervention unless encouragement was present

  22. How Effective is the Treatment? • Local Average Treatment Effect • Effect of the intervention on those who would not have enrolled unless encouraged

  23. Encouragement Design

  24. Example: Community Based School Management • What is the effect of decentralization of school management on learning outcomes? • All communities are eligible • Community management of hiring, budgeting, oversight • Need to write proposal

  25. Community Based School Management • 1500 schools in the evaluation • Each community chooses whether to participate in program

  26. Promotion Design • Community based school management • Provision of technical assistance and training by NGOs for submission of grant application • Random selection of communities with NGO support

  27. Community Based School Management • Outcome – learning outcome • Intervention • decentralization of management to community • 1500 schools • Promotion • NGO support • schools randomized to receive this support

  28. Examples – Randomized Promotion • Maternal Child Health Insurance in Argentina • Intensive information campaigns • Employment Program in Argentina • Transport voucher • Community Based School Management in Nepal • Assistance from NGO • Health Risk Funds in India • Assistance from Community Resource Teams

  29. Randomized Promotion • Just an example of an Instrumental Variable • A variable correlated with treatment but nothing else (i.e. random promotion) • Again, we really just need to understand how the benefits are assigned • Don’t have to exclude anyone

  30. Measuring Impact • Randomized Experiments • Quasi-experiments • Randomized Promotion-Instrumental Variables • Regression Discontinuity • Difference in difference – panel data • Matching

  31. Introduction • What is the impact of monetary incentives on test scores to schools below some ranking? • Compare the schools right above the ranking point to schools below the cutoff point

  32. Indexes are Common in Targeting of Social Programs • Anti-poverty programs • targeted to households below a given poverty index • Pension programs • targeted to population above a certain age • Scholarships • targeted to students with high scores on standardized test

  33. Example: What is the Effect of Cash Transfer on Consumption? • Intervention: • Cash transfer to poor households • Evaluation: • Measure outcomes (i.e. consumption, school attendance rates) before and after transfer, comparing households just above and below the cut-off point.

  34. Example: What is the Effect of Cash Transfer on Consumption? • Target poorest households for cash transfer • Method: • construct poverty index from 1 to 100 with pre-intervention characteristics • households with a score <=50 are poor • households with a score >50 are non-poor

  35. Non-poor Poor

  36. Treatment effect

  37. Regression Discontinuity • When to use this method? • the beneficiaries/non-beneficiaries can be ordered • there is a cut-off point for eligibility. • cut-off determines the assignment of a potential beneficiary to the treatment or no-treatment

  38. Regression Discontinuity Baseline – No treatment 2

  39. Regression Discontinuity Treatment Period

  40. Sharp and Fuzzy Discontinuity • Sharp discontinuity • the discontinuity precisely determines treatment • equivalent to random assignment in a neighborhood • e.g. social security payment depend directly and immediately on a person’s age • Fuzzy discontinuity • discontinuity is highly correlated with treatment . • use the assignment as an IV for program participation. • e.g. rules determine eligibility but there is a margin of administrative error.

  41. Examples • Effect of class size on scholastic achievement (Angrist and Lavy, 1999) • Effect of transfers on labor supply (Lemieux and Milligan, 2005) • Effect of old age pensions on consumption -BONOSOL in Bolivia (Martinez, 2005)

  42. PotentialDisadvantages of RD • We estimate the effect of the program around the cut-off point. • the effect is estimated at the discontinuity • fewer observations than in a randomized experiment with the same sample size • limits generalizability • Make sure the relationship between the assignment variable and the outcome variable is correctly modeled, including: • nonlinear relationships • interactions

  43. Advantages of RD for Evaluation • RD allows one to estimate the effect of an intervention at the discontinuity • Can take advantage of a known rule for assigning the benefit that are common in the designs of social policy • No need to “exclude” a group of eligible households/individuals from treatment

  44. Summary

More Related