Making the Most out of Discontinuities

1. Making the Most out of Discontinuities Bilal Zia World Bank

2. Introduction • Many times random assignment is not possible • Universal take-up • Non-excludable intervention • Treatment already assigned • When randomization is not feasible, how can we exploit implementation features of the program to “measure” its impact? • Answer: Quasi-experiments • Example: Regression Discontinuity Design.

3. Regression Discontinuity Design • RDD closer to randomized experiments than other quasi-experimental methods • Relies on knowledge of the selection process • Need to know quantifiable selection criteria – a “score” • Assignment to “treatment” depends discontinuously on this “score” • Example: A matching grants policy that applies to firms with annual sales less than or equal to $5,000. • A firm with sales = $5,001 would not be treated but would be very similar to a firm with sales = $5,000. • RDD would compare firms just above and just below the $5k threshold.

4. Another RDD Example • Policy: US minimum legal drinking age is 21  alcohol consumption illegal for people younger than 21 • Observation: • People aged 20 years, 11 months and 29 days • 21 year olds • Treated differently under policy because of arbitrary age cut off • But not inherently different (likelihood to go to parties, obedience, propensity to engage in risky behavior, etc)

5. Effect of Alcohol on Mortality • In effect: This policy rule assigns people to “treatment” and “comparison” groups • Treatment group: People between ages 20 years and 11 months and 20 years 11 months and 29 days • Comparison group: individuals who just turned 21 and can now legally drink alcohol. • They should be similar in terms of observable and unobservable characteristics that affect outcomes (mortality rates). • This way, it is possible to isolate the causal impact of alcohol consumption on mortality rates among young adults.

6. Graphical Depiction

7. Increased alcohol consumption causes higher mortality rates around the age of 21 All deaths All deaths associated with injuries, alcohol or drug use All other deaths

8. Sharp and Fuzzy Discontinuities • Sharp discontinuity • Discontinuity precisely determines treatment status • All people 21 and older drink alcohol and no one else does • All firms with less than $5,000 in sales receive vouchers and larger firms do not. • Fuzzy discontinuity • Percentage of participants changes discontinuously at cut-off, but not from 0% to 100% (or from 100% to 0%) • Some people younger than 21 end up consuming alcohol and/or some older than 21 don’t consume at all • Policy rule determines eligibility but among firms with less than $5,000 in sales, there is only partial compliance / take-up

9. Probability of Participation: Sharp ruleand “fuzzy” rule 100% 75% 0% 0%

10. Internal Validity • General idea • If cut-off is arbitrary, individuals to the immediate left and right of the cut-off should be very similar. • Differences in outcomes can be attributed to the policy. • Major assumption • Nothing else is happening: in absence of policy, we would not observe a discontinuity in the outcomes around this particular cut off. • Might not be the case if • Bike-helmet policy stops also stops applying at age 21. • Another policy gives equipment to firms with sales less than $5,000

11. Outcome Profile Before and After the Intervention

12. External Validity • Would the results generalize past the two groups you are comparing? • Counterfactual group in RDD • Individuals marginally excluded from benefits • Examples: people less than 21 but older than 20 years and 10 months, firms with sales less than $5,000 but more than $4,500. • Causal conclusions are limited to individuals, households, villages, near the cut-off • The effect estimated is for individuals marginally or just eligible for benefits • Extrapolation beyond this point needs additional, often unwarranted, assumptions (or multiple cut-offs) • Fuzzy designs exacerbate the problem

13. RDD Implementation: Nuts and Bolts • Major advantage of the RDD • Transparency • Illustrated possible using graphical methods • Major disadvantage • Requires many observations around cut-off • All observations away from the cut-off should have less weight • Why? • Only near the cut-off can we assume that people find themselves by chance to the left and to the right of the cut-off. • Think about firm with annual sales = $5,000 vs firm with sales = $500. • Or a 16 year old vs a 25 year old.

14. Graphical Analysis

15. Wrap Up • RDD as a powerful tool to identify causal effects • Pros • RDDs share the same properties as an experiment locally at the cut-off • Can be used to evaluate interventions ex post by treating cut-offs as “natural experiments” (approximate properties of an experiment but not designed as experiments) • Cons • Estimated program effects are representative only of individuals/firms near the cut off • RDDs require a lot of data • Individuals/firms may adjust their behavior in subsequent years in response to the cut-off • Example: Stop reporting sales above $5,000.

16. Wrap Up • Can be used to design a prospective evaluation when randomization is not feasible • The design applies to all means tested programs • Multiple cut-offs to enhance external validity • Can be used to evaluate ex-post interventions using discontinuities as “natural experiments”.

17. Thank You