Natural Experiments as Multi-Method Research

1. Natural Experiments as Multi-Method Research Thad Dunning Yale University Prepared for a short course on Multi-Method Research, American Political Science Association meetings, Seattle, Washington, August 30, 2011

2. The Growth of Natural Experiments (1960-2009) Articles published in major political science and economics journals with “natural experiment” in the title or abstract (as tracked in the online archive JSTOR).

3. A “quantitative” technique? • Natural experiments are often regarded as quantitative: • Standard Natural Experiments • Regression-Discontinuity Designs • Instrumental-Variables Designs • Gathering systematic data on outcomes, exposure to treatment, and pre-treatment covariates for each case—a.k.a. Data-Set Observations (DSOs)—is indeed critical • One advantage of strong natural experiments: the quantitative analysis may be simple and transparent • Often, a simple difference of means suffices to estimate causal effects • Another advantage: models of the data-generating process are often credible (e.g., the Neyman model)

4. The crucial role of qualitative methods • However, qualitative methods are invaluable and often prerequisite for natural experiments. • As Angrist and Krueger (2001) put it, “Our view is that progress in the application of instrumental variables methods depends mostly on the gritty work of finding or creating plausible experiments that can be used to measure important economic relationships—what the statistician David Freedman (1991) has called “shoe-leather” research.Here the challenges are not primarily technical in the sense of requiring new theorems or estimators. Rather, progress comes from detailed institutional knowledge and the careful investigation and quantification of the forces at work in a particular setting. Of course, such endeavors are not really new. They have always been at the heart of good empirical research” [emphasis added].

5. What Role Do Qualitative Methods Play? • A “Causal-Process Observation” (CPO) is “an insight or piece of data that provides information about context, process, or mechanism” (Collier et al. 2010) • Dunning (book manuscript) describes the contribution of five types of CPOs to natural experiments (see Mahoney 2010): • Treatment-Assignment CPOs • Independent-Variable CPOs • Mechanism CPOs • Auxiliary-Outcome CPOs • Model-Validation CPOs • The idea is to put the role of “detailed institutional knowledge and careful investigation…of the forces at work in a particular setting” on a more systematic foundation • Chapter 6 of Dunning, Natural Experiments in the Social Sciences

6. 1. Assessing as-if random • Relative to conventional observational studies, the claim that treatment assignment is as good as random is the defining feature of a natural experiment • Definition: A natural experiment is an observational study (that is, a study without an experimental manipulation) in which assignment to treatment and control groups is done at random—or “as if” at random • Since there is (usually) no true randomization, this assertion is, to some extent, not testable • Yet evidence and a priori reasoning, informed by substantive knowledge, can help validate a natural experiment

7. Quantitative Tools for Assessing As-If Random • Standard Natural Experiments • Balance tests: Treatment and control groups are statistically balanced on pre-treatment covariates—just as they should be in true (randomized controlled) experiments • Regression-Discontinuity Designs • Balance tests, plus histograms or “conditional density” tests: units do not bunch on one side of the RD threshold (e.g., effects of anti-discrimination laws) • Placebo tests: no apparent effect away from the RD threshold • Instrumental-Variables Designs • Balance tests; specification tests (note many applications include covariates in specifications—not so good) • These are all based on Data-Set Observations (DSOs)

8. Treatment-Assignment CPOs • These are nuggets of information about the process by which cases ended up in treatment or control groups • Analyst must explain why thisprocessplausibly leads to an as-if random allocation of units to treatment and control. • This is quite different from saying that one cannot think of any potential confounders, and thus assignment must be as good as random (or worse, that confounders have been “controlled”) • Knowledge of this process does not typically come by knowing the values of independent and dependent variables for each case (i.e., DSOs) • Treatment assignment CPOs provide crucial leverage for causal inference

9. Example: Impact of Land Titling • Squatters with land titles seem to do better than those without them (De Soto 2000). Do titles have a causal effect? • Maybe—but squatters who obtain titles are different from those who don’t. • They may have, say, different family backgrounds • Differences in outcomes could be due to the effect of titles, due to the confounders, or both. • We might try to control for confounders we can measure—e.g., compare titled and non-titled squatters with similar family backgrounds. • But within these strata, more determined or motivated squatters may obtain titles—and determination is hard to measure

10. A Natural Experiment on Property Rights • In 1981, squatters organized by the Catholic church occupied an urban wasteland in the province of Buenos Aires, dividing the land into similar parcels • After the return to democracy, a 1984 law expropriated the land, with the intention of transferring title to the squatters • Some of the original owners challenged the expropriation in court, leading to long delays, while other titles were ceded and transferred to squatters • The legal action created a “treatment” group – squatters to whom titles were ceded – and a control group – squatters whose titles were not ceded

11. A Natural Experiment? Quantitative Evidence • Galiani and Schargrodsky (2006, 2007) use DSOs to argue that whether title was ceded is “as if” random • For example, they show “pre-treatment equivalence” of treated and untreated units: • Titled and untitled parcels are side-by-side (reminiscent of Snow) • Pre-treatment characteristics of the parcels (such as distance from polluted creeks) are similar in both groups • The compensation offered by the government (in square meter terms) was very similar across parcels • Pre-treatment characteristics of squatters (age, sex, etc.) do not predict whether they received title • Treatment and control groups are statistically indistinguishable—just as they usually would be after randomization

12. Qualitative evidence • Yet, just as important is qualitative evidence – for example, on the process by which the squatting took place • Squatters and Catholic Church organizers did not appear to know the land was privately owned at the time they occupied it • They did not anticipate the later expropriation of land by the state • They had no basis for predicting which particular plots of land would be expropriated, and thus could not assign plots to particular squatters with this in mind • These are Treatment-Assignment CPOs: • They do not come in the form of systematic data on the values of independent and dependent variables (DSOs) • They help the analysts understand the process by which units were eventually allocated to treatment and control groups—and to validate the claim that this assignment was as good as random

13. The effects of property rights • Galiani and Schargrodsky (2006, 2007) find significant differences in housing investment, household structure, and educational attainment of children (but not in access to credit markets, contradicting De Soto) • Squatters who received titles also believed more in individual efficacy (!)—e.g. “people get ahead in life through hard work, rather than luck” (Di Tella et al.) • Without the natural experiment, such a finding could be due to confounding: people who acquired land titles might have had stronger ex-ante faith in their efficacy than those who didn’t

14. Snow on Cholera • Nineteenth-century London: Against the predominant theory—miasma—John Snow hypothesized that cholera was a water- or waste-born disease The predominant theory • Snow’s natural experiment • In 1852, the Lambeth water company moved its intake pipe upstream on the Thames, to a purer water source • The Southwark & Vauxhall company left its intake pipe in place.

15. Death rate from cholera in London, by source of water supply (adapted from Snow 1855, Table IX)

16. Snow on Cholera: DSOs “The pipes of each Company go down all the streets…A few houses are supplied by one Company and a few by the other… In many cases a single house has a supply different from that on either side. Each company supplies both rich and poor, both large houses and small; there is no difference either in the condition or occupation of the persons receiving the water of either company… “It is obvious no experiment could have been designed which would more thoroughly test the effect of water supply on the progress of cholera than this.” -- John Snow (1885: 74-75) • The quote speaks to external validity, but also to the “pre-treatment equivalence” of treatment and control groups

17. Snow on cholera: CPOs • Households typically did not self-select into sources of water supply; absentee landlords often took the decision regarding water supply source, thereby dividing “more than three hundred thousand people of all ages and social strata…into two groups without their choice, and, in most cases, without their knowledge; one group being supplied with water containing the sewage of London, and, amongst it, whatever might have come from the cholera patients, the other group having water quite free from such impurity” (Snow 1855: 75; italics added). • Moreover, water pipes were laid down and water-supply source was determined “according to the decision of the owner or occupier at that time when the Water Companies were in active competition” (ibid), long before the cholera outbreak • Such Treatment-Assignment CPOs play a critical role in making the claim of as-if random persuasive

18. Exercise: Treatment-Assignment CPOs Snow on cholera. The impurity of water in the Thames was a source of concern to public authorities before the cholera outbreak, and the Metropolis Water Act of 1852 made it unlawful for any water company to supply houses with water from the tidal reaches of the Thames after August 31, 1855. Yet, while the Lambeth’s move of its intake pipe upstream was planned in the late 1840s and completed in 1852—before the cholera outbreak of 1853-54—the Southwark and Vauxhall company did not move its pipe until 1855. In other words, the Lambeth Waterworks Company chose to move its pipe upstream before it was legally required to do so, while Southwark & Vauxhall left its intake pipe in place. Questions: Could this fact pose a threat to as-if random? How might Snow’s discussion of the process by which water was purchased and supplied counter some potential threats to the validity of the natural experiment? Are these Treatment-Assignment CPOs?

19. Validating As-If Random, In Sum • Some key questions to ask of any study: • Are treatment and control groups unbalanced with respect to variables other than the treatment? • Do units “self-select” into groups? • Are policy interventions applied in a way that anticipates the behavioral response of units? • “No” to all questions may be necessary, if not sufficient, for a plausible natural experiment • Thus, both CPOs and DSOs play a critical role • Similar points may be made about regression-discontinuity and instrumental-variables designs

20. 2. Independent-Variable CPOs • These nuggets of information provide information about the presence or values of an independent variable (a treatment); they can contribute both to natural experiments and in exploratory or confirmatory research undertaken in conjunction with a natural experiment. • They can also sometimes be useful for investigating what aspect or component of a treatment is plausibly responsible for an estimated causal effect. • Snow on cholera: • Early in Snow’s research, he found infected water or waste was present during cases of cholera transmission • The Broad Street pump: brewers get water from private pumps • Verifying source of water supply in his natural experiment—a case of “shoe leather” epidemiology

21. 3. Mechanism CPOs • In many natural-experimental contexts, using selected units in treatment and control groups to “extract… ideas at close range” (Collier 1999) can be useful for explaining effects • There may be rich opportunities for “experimental ethnography” (Sherman and Strang 2004), in which qualitative data shapes the interpretation of causal effects • Mechanisms may be understood as invariant processes, not necessarily mediating variables: • E.g., “deterrence,” “empowerment,” or “solidarity” • Close range fieldwork may be quite important for identifying such processes that might account for an effect (e.g. the Argentina land titling study)

22. 4. Auxiliary-Outcome CPOs • These provide “information about particular occurrences that should occur alongside(or perhaps as a result of) the main outcome of interest if in fact thatoutcome were caused in the way stipulated by the theory under investigation” (Mahoney 2010: 129). • They may thus be closely linked to theory testing; the metaphor of a criminal detective searching for clues is useful here (Collier, Brady and Seawright 2010). • Using a natural experiment, Dunning and Nilekani (2011) find that caste-based quotas in India have very weak distributive effects • They attribute this to patterns of party competition engendered by the rotation of quotas across electoral jurisdictions (village councils)

23. 5. Model-Validation CPOs • These are nuggets of information about causal process that can support or invalidate assumptions of causal models • Causal inference depends on maintained assumptions—but these can sometimes be validated, at least partially • In principle, experiments and natural experiments are “design-based” rather than “model-based” methods • Inferential problems (such as confounding) are confronted through research design, not ex-post statistical adjustment • Causal leverage comes (or should come) from the design and not from modeling (e.g., multivariate regression analysis, matching • Thus, a simple difference-of-means test can suffice • Yet, the assumptions of models—e.g., the Neyman model—matter here as well

24. The Effects of Land Titles on Children’s Health Source: Galiani and Schargrodsky (2004). Notice that this is intention-to-treat analysis. In the first two rows, data for children ages 0-11 are shown; in the third row, data for teenage girls aged 14-17 are shown. The number of observations is in brackets.

25. Is the Model Credible? • The Neyman causal model—which defines the estimands that difference-of-means tests estimate—is very general • However, there are assumptions that may or may not be credible, depending on the setting • One unit’s potential response is assumed not to depend on the treatment assignment of other units • Rubin referred to this “no interference between units” as the Stable Unit Treatment Value (SUTVA) assumption • Thus the model may not be an adequate description of happens in certain natural experiments • E.g., in Argentina, the fertility decisions of squatters in the control group may be influenced by the fertility decisions of their neighbors in the treatment group • Model-validation CPOs can be helpful here • E.g., how do squatters interact with each other?

26. Regression-Based Inference • With conventional regression analysis, the need to validate the model may be even greater • Quantitative studies using conventional observational data often rely on regression modeling to approximate an experimental ideal • “The power of multiple regression analysis is that it allows us to do in non-experimental environments what natural scientists are able to do in a controlled laboratory setting: keep other factors fixed” (Wooldridge 2009: 77). • Yet, the assumptions behind the models play a key role—and if the assumptions aren’t valid, the results shouldn’t be trusted • I won’t thoroughly discuss the limitations of conventional regression analysis here, but this is important background

27. Model-Validation CPOs: Regression • Question: How does affluence affect attitudes towards economic policy? • Doherty, Green, and Gerber (2005, 2006) take a survey of lottery winners and ask questions about political attitudes, e.g., towards redistributive taxation or social spending • An innovative natural experiment: lottery winners are randomized to the level of winnings

28. Doherty et al. as a natural experiment • Findings: lottery winnings influence attitudes about narrow redistributive issues (the estate tax) but not broader issues (role of government) • A key merit: there really is a random mechanism determining assignment to treatment status • However, our interest may lie in the effects of overall income, not lottery winnings • Instrumental-variables regression may help—if the assumptions of IV model are correct

29. Lottery Winnings as an Instrument • The impact of income on political attitudes? Regress the latter on the former: • The problem: Income and are not independent, so OLS estimates of will be biased. • The solution: Use lottery winnings as an instrument for income. • Notice that INCOME = LOTTERY_WINNINGS + EARNED_INCOME ATTITUDES = INCOME

30. Thus, the model says: ATTITUDES = (LOTTERY_WINNINGS + EARNED_INCOME ) • According to the model, a marginal increment in either lottery winnings or earned income is associated with the same expected marginal increment in attitudes • This could be called the assumption of “homogenous partial effects” (Dunning 2008) • Yet, windfall earnings associated with the lottery may have different effects than money earned through work

31. Suppose: ATTITUDES = LOTTERY_WINNINGS + EARNED_INCOME • To estimate consistently, we would need another instrument for earned income—and getting rid of the confounding of earned income is supposed to be the point • The point is that as-if random is not enough; the model of the data-generating process must be credible as well • In principle, model-validation CPOs could help here • Do lottery winners say they treat windfall income differently?

32. Exercise: Model-Validation CPOs • Horiuchi and Saito (2009), in their study of the effect of turnout on fiscal disbursements to municipalities in Japan, use election-day rainfall as an instrumental variable for municipal turnout. They argue that in this context, the assumption of “homogenous partial effects” (Dunning 2008)—that is, the assumption that variation in turnout related to rainfall has the same effect as variation unrelated to rainfall—is likely to be valid. • Why do these authors use instrumental variables here? • What does the assumption of “homogenous partial effects” imply in this substantive context? • What sorts of model-validation CPOs might validate the assumption? • What model-validation CPOs would undermine it?

33. Other Model-Validation CPOs • In instrumental-variables analysis, model-validation CPOs can help assess whether the instrument affects the outcome through channels other than the treatment variable • Does rainfall affect conflict in Africa through channels other than economic growth (Miguel et al. 2004)? • Does draft lottery number affect later earnings through channels other than service in Vietnam (Angrist 1990)? • Validation of the model is crucial in any study, and qualitative methods can play a central role • However, there are fewer examples of successful Model-Validation CPOs than the other types of CPOs • This may be because model validation is too-infrequently discussed…

34. Conclusions: Natural Experiments as Multi-Method Research • Quantitative analysis is critical and has the potential advantage of simplicity and transparency • Reliance mainly on differences-of-means tests gives analysts more space to discuss the substantive context, research design and results • Qualitative methods can bolster the credibility of quantitative analysis • Qualitative research is critical in other ways • Substantive knowledge and close attention to context are crucial for recognizing the potential existence of a natural experiment • Thus, natural experiments are pre-eminently a form of multi-method research • Yet, the quantitative analysis of natural experiments has received much more methodological attention

35. Conclusions: Next Directions • What distinguishes a good (convincing, compelling) CPO from a bad (misleading) one? • Knowledge of context is critical, and only a few experts may possess this • A possible tendency towards confirmation bias—ignore disconfirming CPOs while highlighting helpful ones? • Does the adversarial tradition (e.g. debates among case experts) help? • How can qualitative methods best bolster the strength of natural experiments along many dimensions?

36. Strong Research Design Typology of Natural Experiments Most Credibility of the Models Most Relevance of the Intervention Least Least Least Most Weak Research Design Plausibility of As-If Random