Julian R. Betts and Y. Emily Tang, University of California, San Diego

1. Julian R. Betts and Y. Emily Tang, University of California, San Diego (jbetts@ucsd.edu, yetang@ucsd.edu) We are grateful to the Center on Reinventing Public Education, University of Washington, Bothell, for funding this research The effect of charter schools on student achievement: A meta-analysis of the literatureCampbell Colloquium Education panel, May 2012

2. Introduction and Motivation Selecting Studies to Include Assessment of Alternative Methods of Evaluating the Impact of Charter Schools Challenges in Study Collection/Review Process Description of Methods Used in Review Results Future Research and Policy Implications outline

3. Persistent concern over the performance of US public schools at the elementary and secondary levels • Elementary • Grades K-5 (ages 5-11) • Secondary • Middle: Grades 6-8 (ages 11-14) • High: Grades 9-12 (ages 14-18) Some background on US education

4. The US spends a lot (per primary school pupil) on education, obtains average educational outcomes Source: Gruber (2010)

5. The US spends about average (% of Gdp) on education, obtains average educational outcomes Source: OECD (2011)

6. In the us The school that a student attends is primarily determined by where he/she lives San Diego Unified School District Elementary School Boundaries 2011-12

7. Charter schools are a relatively new alternative to traditional neighborhood public schools • ~ 20 years, substantial growth in the 2000s • A succession of U.S. presidents has named charter schools as important agents of school reform What is a charter school?

8. Approximately 5% of public schools are Charter schools, this number is growing Source: Lake and Gross (2011)

9. Charter schools are publicly funded, governed by organization under contract with the state Charter schools are exempted from parts of the state education code, freeing them to innovate with respect to curriculum, pedagogy and hiring of teachers What is a charter school?

10. Albert Einstein Academy: • “independent charter school that would have a dual instructional focus of German-English immersion within the context of a rigorous academic instructional model” • Charter School of San Diego: • initially developed from a state bill “designed to reduce the dropout rate by recovering students who had been out of school for more than 45 days” charter schools are different from each other, Examples from san diego

11. Scope: Include studies of US elementary and secondary charter school performance • US public K-12 education is decentralized • Most data on student performance are collected at the level of a US state, or the level of a school district (smaller than a US state) • Outcomes: Include studies that use student performance on math and reading standardized tests as an outcome measure • Methods: Include studies that use credible approaches to address selection bias Selecting Studies for this Literature Review

12. Snapshots of average student achievement at one point in time can be misleading as they do not account for self-selection into schools • US school attendance based largely on geographic residence. • Students choosing to attend charter schools are likely different in observable and unobservable ways Selection bias: Main Concerns with alternative approaches leading to exclusion

13. Negative selection (downward bias) Example: An underprivileged, disadvantaged student without family support is at high risk of dropping out of school. She is advised by her high school counseling staff to transfer to a charter school, and she chooses to transfer. Problem: Underprivileged, disadvantaged students without family support are not likely to obtain high test scores in any school, traditional or charter. The estimate of charter school effectiveness based on comparison of charter school student performance and traditional school student performance would be biased downwards. Unobserved characteristics correlated with charter school attendance

14. Positive selection (upward bias) Example: An active, concerned, involved parent is dissatisfied with the traditional public school in his/her neighborhood. The parent decides to opt-out of the traditional school and enroll his/her child in a charter school. Problem: Students with active, concerned, involved parents are likely to obtain high test scores in any school, traditional or charter. Implication: The estimate of charter school effectiveness based on comparison of charter school student performance and traditional school student performance would be upwardly biased. Unobserved characteristics correlated with charter school attendance

15. National Charter School Research Project issued a White Paper (drafters: Betts and Hill, 2006) arguing that lottery-based studies and student-level longitudinal “value-added” studies were the two most credible approaches These methods more convincing than other methods. Selecting Studies for this Literature Review

16. Methods matter Source: Hill (2006)

17. In the set of studies we include, there are four approaches used 1) Lottery-based studies 2) Fixed-effect studies, that compare a student’s gains in achievement in years attended a charter to his or her gains in years attended a traditional public school 3) Propensity score matching 4) Other types of matching (e.g. CREDO) 4 Commonly used Methods of Analysis in the included studies

18. Lottery-Based Analysis Source: Waiting for Superman movie (2010)

19. Obvious benefit: expected outcomes identical for lottery winners and losers if lottery conducted fairly • But several weaknesses to this “gold standard” • External validity • Most charter schools not oversubscribed • Mathematica study of charter middle schools: only 130/492 oversubscribed • Could be bias from attrition Lottery-Based Analysis

20. Assumes “selection on observables” If students in charter schools have unobserved variations in ability or motivation, will be biased Two major studies of KIPP (Knowledge is Power Program) schools have used this approach CREDO at Stanford has produced string of influential state-level studies. Uses a unique matching process. Not propensity score but has similar issue with “selection on observables” Propensity score matching

21. Benefit: Avoids need to compare one student with another, instead comparing individual students’ trajectories in charter schools and traditional public schools • But many elementary students never switch between the two types of schools – external validity issue • Zimmer et al (2009) compare test-score gains of charter “stayers” and switchers and do not get clear-cut result. But in some cases “stayers” have higher test-score gains • Suggests downward bias from using this method • Zimmer et al (2009) also raise concerns about reversibility – are the effects of attending a charter dependent on the order in which a student attends the charter and the traditional public school? Find some evidence that this is the case. • Unobserved heterogeneity may change over time. Fixed effects cannot solve Student fixed-effects

22. 40 reports now available, with just under 100 estimates of effects for each of math and English Language Arts (reading) • Lottery-based studies still quite rare: still only 8 papers that use lotteries, covering 90 charter schools • We exclude studies using less rigorous methods, specifically, those that do not use student-level test score gains as outcomes. Included studies

23. Handling large weight (large number of students and large number of schools) studies • Solution: Analyze with and without large weight studies • Handling the different methods used in different studies • Solution: Investigate whether method of analysis matters • Some reports omit important information, e.g. number of schools in the sample • Solution: Email exchange with authors Challenges in study collection/review process

24. Introduction and Motivation Assessment of Alternative Methods of Evaluating the Impact of Charter Schools Selecting Studies to Include Challenges in Study Collection/Review Process Description of Methods Used in Review Results Future Research and Policy Implications

25. Fisher test – Is there evidence that no study finds negative effects; conversely, evidence of no positive effects? • Formal meta-analysis provides overall estimated effect, its statistical significance and measures of how much true underlying variation there is across studies • Histograms • Show variability and the influence of weighting of studies • Vote-counting as a way of assessing variation in results Our Methods of Analysis

26. Look for variations in effect by: • Subject area tested (math vs. reading) • Grade span (E, M, H) • Geographic location • KIPP vs. non-KIPP • Is there a systematic difference in results based on the method researchers use? Heterogeneity is an Underlying Theme

27. Testing Whether Charter Schools in Any Study Increase or Decrease Achievement Relative to Traditional Public Schools Meta-Analysis of Effect Size Histograms and Vote Counting as Measures of Variation Methods used in Review

28. Fisher’s combined test S is distributed with df=2k Null hypothesis: No positive effects Null hypothesis: No negative effects Method #1: Evidence of No Positive Effects, or No Negative Effects?

29. We conduct this analysis 12 times: 6 ways of combining grades, and two subjects (math and ELA) First sign of heterogeneous effects of charter schools: in 9/12 cases there is clear evidence of BOTH negative and positive effects Three exceptions with evidence of positive effects but no evidence of negative effects: elementary and middle school ELA scores, and middle school math scores Method #1: Evidence of No Positive Effects, or No Negative Effects?

30. Probability of No Positive Effects in Any of the Studies: Almost Zero 30

31. Probability of No Negative Effects in Any of the Studies: Almost zero in most cases, and quite high in 3 cases 31

32. Testing Whether Charter Schools in Any Study Increase or Decrease Achievement Relative to Traditional Public Schools Meta-Analysis of Effect Size Histograms and Vote Counting as Measures of Variation Methods used in Review

33. Assume charter school estimates are randomly distributed • Therefore it is important to estimate both the mean and the variation • Underlying “true” variation across studies is the extent to which variation cannot be explaining by sampling error (“uncertainty”) in individual estimates • Omitted many studies of individual KIPP schools as they would have disproportionate influence • Include KIPP schools in subsidiary analysis Method #2: Formal Meta-Analysis

34. In a random effects meta-analysis, we take a weighted average of the effect sizes across studies. If Yiis the effect size for the ithof k studies, and Wi is the weight for each study, our overall estimated effect size M is: (1) The Mean Effect is a Weighted Average

35. The weight for each study is the inverse of the sum of the within-study variance (based on the standard error) and an estimate of the true between-study variance, T2: (2) T2 based on a method of moments  estimate  of the variance of true effect sizes. Note that as T2becomes large relative to the average within-study variance estimate, then we will tend toward equal weighting across studies; whereas as T2becomes relatively small, the weights can become highly unequal with heavier weight given to studies with the lowest sampling variance. Weights depend on within-study variance and estimated across-study (true) variance

36. Use the I2 statistic (Higgins et al., 2003) • Provides estimate of the percentage of variation across studies that reflects true underlying variation An estimate of what % of the variance across studies is true

37. * Indicates statistically significant (5% level) Sample of Our Results on Effect Sizes

38. * Indicates statistically significant (5% level) Sample of Our Results “On average, attending a charter school is associated with an increase in test scores in reading equal to 0.022 of a standard deviation per year.”

39. * Indicates statistically significant (5% level) Sample of Our Results Nine studies covering 7 geographic areas 77.7% of the variation across studies represents true variation in charter school effects, rather than “noise”

40. Overall Effect Size Estimates

41. Enough to move a student at the 50th percentile to the 52nd percentile after attending a charter for one year Elementary school reading impact is smaller: enough to boost a student from 50th to about percentile 50.8 Elementary/Middle School Math Effects: Meaningful but Not Huge

42. Elementary School Reading Effect Sizes

43. Elementary school math effect sizes

44. Middle School Reading Effect Sizes

45. Middle School Math Effect Sizes

46. High School Reading Effect Sizes

47. High school math effect sizes

48. Reading Effect Sizes for Studies that combine elementary and middle schools

49. MATH Effect Sizes for Studies that combine elementary and middle schools

50. Testing Whether Charter Schools in Any Study Increase or Decrease Achievement Relative to Traditional Public Schools Meta-Analysis of Effect Size Histograms and Vote Counting as Measures of Variation Methods used in Review