The Inter-temporal Stability of Teacher Effect Estimates

1. The Inter-temporal Stabilityof Teacher Effect Estimates J. R. Lockwood Daniel F. McCaffrey Tim R. Sass The RAND Corporation The RAND Corporation Florida State University National Conference on Value-Added Modeling, April 2008 This presentation has not been formally reviewed and should not be cited or distributed without the authors’ permission.

2. Introduction • Several school districts and states have begun using measures of teachers’ contributions to student achievement to assess and reward teachers • Denver, Houston, Florida • For “value added” measures to provide correct incentives and be acceptable to stakeholders they must: • be relatively accurate measures of productivity (ie.unbiased) • be relatively stable over time • Most observers (implicitly) assume that a given teacher’s productivity doesn’t vary much from year to year

3. Research Questions How stable are estimated teacher effects? What factors affect the stability of estimated teacher effects? Are there methods to enhance the stability of estimated teacher effects?

4. Previous Literature Ballou (2005) Elementary and middle school teachers in a “moderately large” Tennessee school district in two consecutive years Nearly 50 percent of math teachers in top quartile in one year stay in the top quartile the next year Precision increases with number of student observations per teacher Aaronson, et al. (2007) High school teachers in Chicago over two years 57 percent of teachers in the top quartile in one year remain there in the next year

5. Previous Literature Koedel and Betts (2007) Teachers in San Diego Within-school estimates of teacher quality Models with student and school fixed effects 35 percent of teachers ranked in the top quintile remain there in the next year Omission of student and school fixed effects increases stability of estimated teacher effects

6. Models of Teacher Effects General Value-added Model • Student i, classroom j, teacher k, school m • X = time-varying student characteristics • P = time-varying classroom peer characteristics • T = time-varying teacher characteristics • S = time-varying school characteristics • Teacher Classroom Average Effect

7. Data Seven Large Countywide School Districts in Florida Two among 10 largest in the U.S. (Dade, Broward) Remainder among 25 largest in the U.S. (Hillsborough, Palm Beach, Orange, Duval and Pinellas) Testing in Grades 3-10 FCAT-NRT (Stanford Achievement Test) 1999/2000-2004/05 (SAT-9 1999/2000, SAT-10 2004/05) FCAT-SSS (Criterion reference exam) 2000/2001-2004/05 Focus on Middle School Math Teachers Teacher effects greater in math More students per teacher in middle school

8. How Stable are Estimated Teacher Effects? Year-to-Year Correlations Proportion of Top-Quintile Teachers Remaining in the Top Quintile the Next Year

9. Inter-temporal Correlation in Estimated Teacher Classroom Average Effects

10. Quintile Ranking of Estimated Teacher Classroom Average Effect in 2001/02 by Quintile Ranking in 2000/01 (in Percent)Broward County [Cross-Year Correlation = 0.48]

11. Quintile Ranking of Estimated Teacher Classroom Average Effect in 2001/02 by Quintile Ranking in 2000/01 (in Percent)Orange County [Cross-Year Correlation = 0.23]

12. Percentage of Teachers Who Remain in Top Quintile from One Year to the Next

13. What Factors Affect the Stability of Estimated Teacher Effects? Changes in the Measurement of Achievement Test Scale If scaling changes over time, could decrease stability Norming by grade and year should reduce fluctuations due to scaling changes Could increase stability if distribution changes over time Test Content If teacher ability varies across content, changes in test could contribute to instability in measured teacher effectiveness Compare FCAT-SSS and FCAT-NRT

14. Inter-Temporal Correlation of Estimated Teacher Classroom Average Effects Under Alternative Test Score Measures

15. Inter-Temporal Correlation of Estimated Teacher Classroom Average Effects Under Alternative Achievement Tests

16. What Factors Affect the Stability of Estimated Teacher Effects? Changes in Reference Point (Stratification) Individual Teacher Effectiveness Must be Measured Relative to Some Reference Point “Holdout” teacher or Average teacher If reference teacher changes, measured effectiveness changes Comparisons Can Only be Made to Other Teachers Who Are Interconnected by Common Students Different strata will have different reference points Within-school vs. between-school measures

17. Number of Teacher-Years in Interconnected Groups

18. What Factors Affect the Stability of Estimated Teacher Effects? Omitted Variable Bias If Measured Teacher Effects Reflect Omitted Variables, Stability of Measured Teacher Effects Will Depend on Stability of Omitted Variables and Extent of Selection Past educational inputs (persistence) Achivement levels vs. achievement gains Student heterogeneity No controls vs. student covariates vs. student fixed effects Peer heterogeneity No controls vs. controls for peer characteristics

19. Inter-Temporal Correlation of Estimated Teacher Classroom Average Effects Under Alternative Persistence Assumptions

20. Inter-Temporal Correlation of Estimated Teacher Classroom Average Effects Under Alternative Controls for Time-Invariant Student Heterogeneity

21. Inter-Temporal Correlation of Estimated Teacher Classroom Average Effects Under Alternative Controls for Time-Varying Factors (Baseline Model – Gain on Normed FCAT-NRT, Student Fixed Effects, Minimum 10 Students per Class)

22. What Factors Affect the Stability of Estimated Teacher Effects? Measurement Error in Student Achievement If measurement error is uncorrelated across students within a classroom, then precision should be higher for teachers with larger classes Minimum class size Minimum number of “movers” per teacher If measurement error is correlated across students within a classroom, but not across classrooms, precision should increase with the number of classes per teacher Using middle school teachers who generally teach multiple class per term

23. Inter-Temporal Correlation of Estimated Teacher Classroom Average Effects Under Alternative Class Size Restrictions

24. Inter-Temporal Correlation of Estimated Teacher Classroom Average Effects Under Alternative “Student Mover” Restrictions (Minimum 10 students per class restriction)

25. What Factors Affect the Stability of Estimated Teacher Effects? True Variation in Teacher Quality Over Time Instability in Estimated Effects Could Reflect Changes in True Teacher Quality Over Time Add time-varying teacher covariates to model Regress estimated teacher-by-year effects on teacher fixed effect and time-varying teacher covariates

26. Inter-Temporal Correlation of Estimated Teacher Classroom Average Effects Under Alternative Controls for Time-Varying Factors (Baseline Model – Gain on Normed FCAT-NRT, Student Fixed Effects, Minimum 10 Students per Class)

27. Are There Methods to Enhance the Stability of Estimated Teacher Effects? Are there methods to enhance the stability of teacher effect estimates? 3-Year Running Averages Reduces noise by averaging sampling errors Could add bias if true performance is changing across years Empirical Bayes or “Shrinkage” Estimators Place greater weight on more reliable estimates and push less reliable estimates toward population mean If already have a significant minimum class size restriction, “simple” EB adjustments which account for differences in the number of classes per teacher or students per teacher not likely to yield large improvements to stability Accounting for variability at the individual teacher level requires computation of standard errors on individual teacher effects, which can be problematic

28. Problems in Computing Shrinkage Estimators Default estimates from most software packages are estimating contrasts between every teacher and a holdout teacher Such estimates support within-year comparisons and cross-year correlations We cannot shrink these estimates directly and we cannot use the resulting standard errors for shrinkage We cannot average these estimates without removing yearly means because changes to the holdout create year-to-year fluctuations

29. Summary • Moderate Stability in Teacher Effects • Cross-year correlations in range on 0.2-0.5 • About 40-50 percent of teachers in top quintile remain in the top quintile the following year • Stability increases with number of students per teacher and when persistence is assumed to equal 0 • Nothing else has a consistent appreciable effect on stability • Variation across districts appears substantial • Shrinkage estimators or running averages could improve stability of teacher-by-year effects, but need to get appropriate estimates and standard errors • Findings suggest caution in using value-added measures for high-stakes personnel decisions

30. Next Steps • Determine sources of year-to-year variability • Improve estimation techniques to obtain comparable estimates across years with accurate measures of within-year standard errors • Current software does not support such estimation • Separate noise from “true” short-term variation • Model sources of short-term variation • If true year-to-year variations exists more efficient estimation than three year averages might be possible via smoothing or filtering