sources of bias in experiments and quasi-experiments

1. sources of bias in experiments and quasi-experiments sean f. reardon stanford university 11 december, 2006

2. three populations • population of interest (POI): the population for whom we would like to estimate the average treatment effect • population of study (POS): the population of whom the study sample is representative • population of causal inference (POC): the population for whom we can make a causal inference

3. tradeoffs between bias and generalizability • we want to estimate the average treatment effect () in some population of interest (POI). •  tells us how much would should we expect the outcome Y of a person randomly chosen from P to differ depending on whether we assign them to T or C. • but we only estimate  in POC • bias arises when POC is not the same as POI: • bias 1: POC is not the same as POS • bias 2: POS is not the same as POI

4. strategies for minimizing bias • randomized experiments: • we get unbiased estimate of POC • and POC=POS, so RCT eliminates bias 1 • but bias 2 may be large or small (external validity) • regression discontinuity: • we get unbiased estimate of POC (under weak assumptions) • but at the cost of making POC≠ POS • POC is generally small (POC is the region of the population near the discontinuity) relative to both POS and POI • but sometimes the region around discontinuity is the population of interest

5. strategies for minimizing bias (cont.) • matching (including fixed effects): • attempts to get unbiased estimate of POC through matching • but at the cost of making POC smaller relative to POS (and POI), because matching allows estimation of treatment effect only in region of common support • but observational matching studies are easier to do with sample of the population of interest than are experiments, so bias 2 may be smaller in matching if region of common support (POC) approximates POI. • fixed effects a form of matching (matching on invariant observed or unobserved factors)

6. How well do quasi-experimental methods do at eliminating bias? • Shadish & Clark paper • Lalonde (1986)-type studies estimate bias remaining after matching • but generally can’t disentangle residual bias in POC from bias 2 • Shadish & Clark paper solves this problem • Theoretically-informed matching can eliminate most/all of bias in POC • What about bias 2? • Extensions? • can use this to assess average effects in population of those who would select the treatment if available

7. How well do quasi-experimental methods do at eliminating bias? • lessons of the Bloom paper • consider ways of reducing variance of estimated treatment effect • need to worry about functional form • tie-breaking experiments enable us to evaluate bias in regression discontinuity (Black, Galdo, Smith 2005) • RD estimates sensitive to functional form unless cases are near threshold • treatment effect varies across thresholds

8. How well do quasi-experimental methods do at eliminating bias? • lessons of the Raudenbush paper • adaptive centering provides same estimates as two-way fixed effects models • better estimates of uncertainty, computationally easier, than fixed effects • under what conditions do such designs reduce bias? • fixed-effects (or centering) eliminates bias in POC under the assumption the within-cell assignment to treatment is ignorable (under what conditions is this reasonable?) • fixed-effects may increase bias 2 by reducing region of common support (domain of POC)

9. remaining questions • important to consider population of interest in research design, concerns about external validity • how can we asses the extent to which we should worry about bias 2? • meta-analysis of multiple studies with different populations of interest? • multi-site randomized trials • draw study samples from known population, assess participation selection