Missing Data in Randomized Control Trials

1. Missing Data in Randomized Control Trials John W. Graham The Prevention Research Center and Department of Biobehavioral Health Penn State University IES/NCER Summer Research Training Institute, July 2008 jgraham@psu.edu

2. Sessions in Four Parts • (1) Introduction: Missing Data Theory • (2) Attrition: Bias and Lost Power • (3) A brief analysis demonstration • Multiple Imputation with • NORM and • Proc MI • (4) Hands-on Intro to Multiple Imputation

3. Recent Papers • Graham, J. W., Cumsille, P. E.,& Elek-Fisk,E. (2003).Methods for handling missing data. In J. A. Schinka & W. F. Velicer (Eds.). Research Methods in Psychology (pp. 87_114). Volume 2 of Handbook of Psychology (I. B. Weiner, Editor-in-Chief). New York: John Wiley & Sons. • Graham, J. W., (2009, in press).Missing data analysis: making it work in the real world. Annual Review of Psychology, 60. • Collins, L. M., Schafer, J. L.,& Kam, C. M.(2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6, 330_351. • Schafer, J. L.,& Graham,J. W.(2002).Missing data: our view of the state of the art. Psychological Methods, 7, 147-177.

4. Part 1:A Brief Introduction toAnalysis with Missing Data

5. Problem with Missing Data • Analysis procedures were designed for complete data. . .

6. Solution 1 • Design new model-based procedures • Missing Data + Parameter Estimation in One Step • Full Information Maximum Likelihood (FIML)SEM and Other Latent Variable Programs(Amos, LISREL, Mplus, Mx, LTA)

7. Solution 2 • Data based procedures • e.g., Multiple Imputation (MI) • Two Steps • Step 1: Deal with the missing data • (e.g., replace missing values with plausible values • Produce a product • Step 2: Analyze the product as if there were no missing data

8. FAQ • Aren't you somehow helping yourself with imputation?. . .

9. NO. Missing data imputation . . . • does NOT give you something for nothing • DOES let you make use of all data you have . . .

10. FAQ • Is the imputed value what the person would have given?

11. NO. When we impute a value . . • We do not impute for the sake of the value itself • We impute to preserve important characteristics of the whole data set . . .

12. We want . . . • unbiased parameter estimation • e.g., b-weights • Good estimate of variability • e.g., standard errors • best statistical power

13. Causes of Missingness • Ignorable • MCAR: Missing Completely At Random • MAR: Missing At Random • Non-Ignorable • MNAR: Missing Not At Random

14. MCAR(Missing Completely At Random) • MCAR 1: Cause of missingness completely random process (like coin flip) • MCAR 2: • Cause uncorrelated with variables of interest • Example: parents move • No bias if cause omitted

15. MAR (Missing At Random) • Missingness may be related to measured variables • But no residual relationship with unmeasured variables • Example: reading speed • No bias if you control for measured variables

16. MNAR (Missing Not At Random) • Even after controlling for measured variables ... • Residual relationship with unmeasured variables • Example: drug use reason for absence

17. MNAR Causes • The recommended methods assume missingness is MAR • But what if the cause of missingness is not MAR? • Should these methods be used when MAR assumptions not met? . . .

18. YES! These Methods Work! • Suggested methods work better than “old” methods • Multiple causes of missingness • Only small part of missingness may be MNAR • Suggested methods usually work very well

19. Methods:"Old" vs MAR vs MNAR • MAR methods (MI and ML) • are ALWAYS at least as good as, • usually better than "old" methods (e.g., listwise deletion) • Methods designed to handle MNAR missingness are NOT always better than MAR methods

20. Analysis: Old and New

21. Old Procedures: Analyze Complete Cases(listwise deletion) • may produce bias • you always lose some power • (because you are throwing away data) • reasonable if you lose only 5% of cases • often lose substantial power

22. Analyze Complete Cases(listwise deletion) • 1 1 1 1 • 0 1 1 1 • 1 0 1 1 • 1 1 0 1 • 1 1 1 0 • very common situation • only 20% (4 of 20) data points missing • but discard 80% of the cases

23. Other "Old" Procedures • Pairwise deletion • May be of occasional use for preliminary analyses • Mean substitution • Never use it • Regression-based single imputation • generally not recommended ... except ...

24. Recommended Model-Based Procedures • Multiple Group SEM (Structural Equation Modeling) • LatentTransitionAnalysis (Collins et al.) • A latent class procedure

25. Recommended Model-Based Procedures • Raw Data Maximum Likelihood SEMaka Full Information Maximum Likelihood (FIML) • Amos (James Arbuckle) • LISREL 8.5+ (Jöreskog & Sörbom) • Mplus (Bengt Muthén) • Mx (Michael Neale)

26. Amos 7, Mx, Mplus, LISREL 8.8 • Structural Equation Modeling (SEM) Programs • In Single Analysis ... • Good Estimation • Reasonable standard errors • Windows Graphical Interface

27. Limitation with Model-Based Procedures • That particular model must be what you want

28. Recommended Data-Based Procedures EM Algorithm (ML parameter estimation) • Norm-Cat-Mix, EMcov, SAS, SPSS Multiple Imputation • NORM, Cat, Mix, Pan (Joe Schafer) • SAS Proc MI • LISREL 8.5+ • Amos 7

29. EM Algorithm • Expectation - Maximization Alternate between E-step: predict missing data M-step: estimate parameters • Excellent (ML) parameter estimates • But no standard errors • must use bootstrap • or multiple imputation

30. Multiple Imputation • Problem with Single Imputation:Too Little Variability • Because of Error Variance • Because covariance matrix is only one estimate

31. Too Little Error Variance • Imputed value lies on regression line

32. Imputed Values on Regression Line

33. Restore Error . . . • Add random normal residual

34. Regression Line only One Estimate

35. Covariance Matrix (Regression Line) only One Estimate • Obtain multiple plausible estimates of the covariance matrix • ideally draw multiple covariance matrices from population • Approximate this with • Bootstrap • Data Augmentation (Norm) • MCMC (SAS 8.2, 9)

36. Data Augmentation • stochastic version of EM • EM • E (expectation) step: predict missing data • M (maximization) step: estimate parameters • Data Augmentation • I (imputation) step: simulate missing data • P (posterior) step: simulate parameters

37. Data Augmentation • Parameters from consecutive steps ... • too related • i.e., not enough variability • after 50 or 100 steps of DA ... covariance matrices are like random draws from the population

38. Multiple Imputation Allows: • Unbiased Estimation • Good standard errors • provided number of imputations (m) is large enough • too few imputations  reduced power with small effect sizes

39. ρ From Graham, J.W., Olchowski, A.E., & Gilreath, T.D. (2007). How many imputations are really needed? Some practical clarifications of multiple imputation theory. Prevention Science, 8, 206-213.

40. Part 2Attrition: Bias and Loss of Power

41. Relevant Papers • Graham, J.W., (in press).Missing data analysis: making it work in the real world. Annual Review of Psychology, 60. • Collins, L. M., Schafer, J. L.,& Kam, C. M.(2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6, 330_351. • Hedeker, D.,& Gibbons,R.D.(1997).Application of random-effects pattern-mixture models for missing data in longitudinal studies, Psychological Methods, 2, 64-78. • Graham, J.W.,& Collins, L.M. (2008). Using Modern Missing Data Methods with Auxiliary Variables to Mitigate the Effects of Attrition on Statistical Power. Annual Meetings of the Society for Prevention Research, San Francisco, CA. (available upon request) • Graham, J.W.,Palen, L.A., et al. (2008). Attrition: MAR & MNAR missingness, and estimation bias. Annual Meetings of the Society for Prevention Research, San Francisco, CA. (available upon request)

42. What if the cause of missingness is MNAR? Problems with this statement • MAR & MNAR are widely misunderstood concepts • I argue that the cause of missingness is never purely MNAR • The cause of missingness is virtually never purely MAR either.

43. MAR vs MNAR • "Pure" MCAR, MAR, MNAR never occur in field research • Each requires untenable assumptions • e.g., that all possible correlations and partial correlations are r = 0

44. MAR vs MNAR • Better to think of MAR and MNAR asforming a continuum • MAR vs MNAR NOT even the dimension of interest

45. MAR vs MNAR: What IS the Dimension of Interest? • How much estimation bias? • when cause of missingness cannot be included in the model

46. Bottom Line ... • All missing data situations are partly MAR and partly MNAR • Sometimes it matters ... • bias affects statistical conclusions • Often it does not matter • bias has tolerably little effect on statistical conclusions (Collins, Schafer, & Kam, Psych Methods, 2001)

47. Methods:"Old" vs MAR vs MNAR • MAR methods (MI and ML) • are ALWAYS at least as good as, • usually better than "old" methods (e.g., listwise deletion) • Methods designed to handle MNAR missingness are NOT always better than MAR methods

48. Yardstick for Measuring Bias Standardized Bias = (average parameter est) – (population value) -------------------------------------------------------- X 100 Standard Error (SE) • |bias| < 40 considered small enough to be tolerable

49. A little background for Collins, Schafer, & Kam (2001; CSK) • Example model of interest: X  Y X = Program (prog vs control) Y = Cigarette Smoking Z = Cause of missingness: say, Rebelliousness (or smoking itself) • Factors to be considered: • % Missing (e.g., % attrition) • rYZ • rZR

50. rYZ • Correlation between • cause of missingness (Z) • e.g., rebelliousness (or smoking itself) • and the variable of interest (Y) • e.g., Cigarette Smoking