Non-Experimental Evaluations

Non-Experimental Evaluations Methods of Economic Investigation Lecture 5

Today’s Lecture • Review the Evaluation Problem • Review the Experimental Approach • Issues of Compliance • Natural Experiments

Review: The Evaluation Problem • In words: • We want to evaluated the effect of a treatment on an outcome • Worried that some omitted/unobservable factor affects both who is “treated” and the outcome. So let’s do this very simply: there’s some treatment T it occurs at some time k

Homogeneous Treatment Effects • Before the treatment (so at time t<k) • After the treatment (so at time t≥ k) Relationship between characteristics X and the Outcome Y Treatment Effect

The Fundamental Identification Problem • Worry that assignment to treatment is not random • Assignment process leads to non-zero correlation between treatment (T) and the error (U) • Probably because participation decision is based on characteristics that affect outcome (Y) too • If you can’t observe EVERYTHING that affects both treatment probability and outcome then your OLS will not be a valid approach

Experimental Approach • Randomization implies that T is independent of X • Don’t have to worry about what factors generated selection into treatment because it is by design • May still include some Xs to increase efficiency • Can use these X’s to show that X and d are uncorrelated

Heterogeneous Treatment Effects • For simplicity for the moment ignore the other regressors (X) • Define the population mean of the treatment effect • Last time, we showed that in this situation when the heterogeneity does not affect probability of treatment then OLS estimates ATE consistently ( ) Treatment Effect for individual i

Defining TOT • Let’s parameterize this a bit more so that an individual’s treatment effect is the mean treatment effect and some individual specific component • Then, we can define the mean impact of the treatment on the treated (TOT) as Mean deviation of the treatment effect among the treatment group population

OLS with Heterogeneous Treatment Effects • The OLS estimate will be: • Our treatment effect is now: • Errors are heteroskedastic (that’s ok)

Heterogeneous TE and ATE • If E(εidi)≠0 then the OLS estimator identifies: • Worry that individuals factors that are unobservable might be correlated with treatment =0 because of Random Assignment TOT

In general there are 4 kinds of Non-Experimental Evaluations 1. Controls, controls, controls… 2. “Natural Experiments” 3. Matching 4. Selection Models 5. Structural Models “Reduced Form” Replicate experiment with an ‘engineered’ control group “Parametric/ Structural” Model the form of the selection effect or bias and apply a statistical correction

Controls method • Basic goal—conditional independence so that it’s like our conditional/stratified randomization • Focus of next week’s class: • If we don’t omit anything, then we’re set • Trade-off between omitted variables and “over-controlling” • Even if we can’t measure everything, maybe be able to control with “fixed effects” and “interactions” (both are error component models)

Pros and Cons • Pros • Transparent to others • Can be convincing with robust controls and well-defined outcomes • Cons • VERY data intensive • Can be hard to measure everything • Not very convincing in complicated settings

“Natural Experiment” • Use an explicit source of variation • Don’t model (or at least don’t use a model) of underlying behavior based on behavior/preferences • ‘Natural’ or ‘Quasi’ Experiments • Used to refer to situation that is not experimental but is ‘as if’ it was • Not a precise definition – saying your data is a ‘natural experiment’ makes it sound better • Refers to case where variation in X is ‘good variation’ (directly or indirectly via instrument)

A Famous Example: London, 1854

The Case of the Broad Street Pump • Regular cholera epidemics in 19th century London • Two Theories on Why: • Widely believed to be caused by ‘bad air’ • John Snow thought ‘bad water’ was cause • An ideal experimental design: • give some people good water and some bad water • Ethical Problems with this

Soho Outbreak: August/September 1854 • Observation 1: People closest to Broad Street Pump most likely to die • Does this distinguish between the two theories? • NO—breathe same air so does not resolve air vs. water hypothesis

Distinguishing between theories • Observation 2: Some places near the well had few deaths • Nearby workhouse few deaths (own well • Nearby brewery had own no deaths (workers all drank beer & own well) • Observation 3: Some people far away died • Woman died in Hampstead • Had the habit of having water from pump deliver by her niece in Islington

Why is this a Natural experiment? • Good variation: • water supply ‘as if’ it had been randomly assigned (existed before outbreak) • other factors (‘air’) held constant • Can estimate treatment effect • difference in means • run regression of death on water source distance to pump, other factors

What’s that got to do with it? • Strongly suggests water the cause • Certainly relative to the air counter-theory • Investigation of well found contamination by sewer • This is non-experimental data but analysed in a way that makes a very powerful case

Elements of Non-experimental Evaluation • Good source of variation (we’re going to call this “EXOGENOUS VARIATION”) • Good measure of treatment and control group outcomes • Good measure of treatment and control group control variables

Methods for Analysing Data from Natural Experiments • If data is ‘as if’ it were experimental then can use all techniques described for experimental data and then some: • OLS • Simple conditional means difference as in experimental data • Difference-in-differences (really just like OLS but with a pre-treatment regression too) • Instrumental Variables

‘Natural Experiments’ Pros and Cons • Advantages: • source of variation is clear • “model free” • Disadvantages: • Identifying assumption may be questionable or hard to believe (internal validity) • may tell you about this experiment but “local” in that it does not tell you about preference parameters (External validity) • using it to make simulations as to other policy changes may be bad

Matching Methods • Non-parametric (no functional form assumption) approach • Take controls and use combined variation to reconstruct experimental conditions • The goal: on everything one can observe for a treatment group individual find a ‘nearest neighbor’

Matching Pros and Cons • Pros • Very flexible in determining “propensity” for treatment • Can use all the variation across all variables (relative to simple control methods) • Cons • Need a lot of common support for you control and treatment groups (i.e. need good matches) • Heavy data requirements • Heterogeneous treatments hard to measure

Selection Correction • Parametrically model the way selection occurs • Famous Example: Female Labor Force Participation • A special case of “structural estimation” but can be used in combination with any methods when you’re worried about a specific form of selection bias

A quick word on Structural Estimation • In some cases, we can use theory to model the endogeneity. • Often this is based on utility or objective function (e.g. profit) maximization theory • In the original structural literature, there was little attention to identification and the results may be identified by nonlinearities and parametrization. • This is no longer the norm, with more attention being placed on identification. • Can also use this more loosely to constrain your estimation to realm of the likely or feasible

Structural Estimation Pros and Cons • Advantages: • once you recover the parameters of the utility function (or other pref), you can use those parameters to simulate what will happen if policy changes. • Disadvantages: • have to implement possibly untestable assumptions about economic and statistical model • often generate wide range of estimates • Can be very sensitive to specification and assumptions—not great external validity

Conclusion • Natural experiments useful source of knowledge • Often requires use of IV • Instrument exogeneity and relevance need justification • Weak instruments potentially serious • Good practice to present first-stage regression • Finding more robust alternative to IV an active research area

Non-Experimental Evaluations