Final Reports from the Measures of Effective Teaching Project

1. By Tom Kane, Harvard University Final Reports from the Measures of Effective TeachingProject

3. The MET project is unique … • in the variety of indicators tested, 5 instruments for classroom observations Student surveys (Tripod Survey) Value-added on state tests • in itsscale, 3,000 teachers 22,500 observation scores (7,500 lesson videos x 3 scores) 900 + trained observers 44,500 students completing surveys and supplemental assessments in year 1 3,120 additional observations by principals/peer observers in Hillsborough County, FL • and in the variety of student outcomes studied. Gains onstate math and ELA tests Gains on supplemental tests (BAM & SAT9 OE) Student-reported outcomes (effort and enjoyment in class)

4. The MET project would like to thank… Denver Public Schools MaeganDaigler Susan McHugh Danielle Ongart TheressPidick Hillsborough County Public Schools Faychone Durant Geri Neizman Danni Resnick David Steele Memphis Public Schools TequillaBanks AnasaFranklin Rorie Harris Monica Jordan New York City Schools Sarah Foster Jennifer Milk David Osborne Katie Ruddy Ha My Vu Jeremy Walter Pittsburgh Public Schools Paulette Poncelet Mary Wolfson Charlotte-Mecklenburg Vanessa Benton RhyshellaGaddist Bettina Spivey Dallas Independent Schools Jacqueline Edwards Judi Guilbeaux Lori Garcia Wendy Hawthorne Johannes Starkes

5. Two Past Reports: • Learning about Teaching (Student Surveys) • Gathering Feedback for Teaching (Classroom Observations)

6. Have we identified effective teachers or … teachers with exceptional students? To find out, we randomly assigned classrooms to 1,591 teachers.

8. Have We Identified Effective Teachers?Key Findings • Following random assignment in Year 2, the teachers with greater measured effectiveness in Year 1 did produce higher student achievement. • The magnitude of the impacts were consistent with predictions. • They also produced higher achievement on the supplemental assessments 70 percent as large as impacts on state tests.

11. Framework for Teaching (Danielson) Actual scores for 7500 lessons.

12. Organizing Observations by School PersonnelKey Findings • Adding an observation by a second observer increases reliability twice as much as having the same observer score an additional lesson. • Short observations may provide a time-efficient way to incorporate more than one observer per teacher. • Principals rate their own teachers higher than do outside observers. However, (1) their ratings are highly correlated with others and (2) administrators discern bigger differences between teachers than peers do (which increases reliability). • Although average scores are higher, letting teachers choose which lessons are observed does not make it harder to identify the teachers who are struggling. In fact, it is easier.

13. There are many roads to reliability. One more observer +.16 One more lesson +.07

14. Combining Measures Using WeightsKey Findings • When the goal is simply to identify teachers who will produce large gains on state tests, put 65 to 90 percent of the weight on that measure. • Balanced weights offer two benefits: • Better ability to predict other outcomes and • Improved reliability (less volatility). • But student achievement should not be weighted too lightly. Going too far results in (1) worse predictive power with respect to other outcomes and (2) less reliability.

17. Can the measures be used for “high stakes”? • High-stakes decisions are being made now, with little or no data. • No information is perfect, but better information should lead to better decisions and fewer mistakes.