What the research tells us about evaluating teacher effectiveness

1. What the research tells us about evaluating teacher effectiveness Laura Goe, Ph.D. Principal Investigator for Research & Dissemination, National Comprehensive Center for Teacher Quality Research Scientist, ETS Presentation to OSEP Washington, DC July 20, 2009

2. How Do We Measure Teacher Effectiveness? • "It's a hard nut to crack. The things that are easy to measure don't matter that much, and the things that matter aren't easy to measure." Adam Gamoran, interim dean at the University of Wisconsin School of Education, talking to Teacher Magazine in December 2008

3. Teacher Qualifications and Effectiveness • We know from research that qualifications may be useful for screening purposes but are poor predictors of teacher effectiveness* • Thus, it is important to measure teacher effectiveness not by initial qualifications but by performance in classrooms and schools • However, which aspects of performance and which student outcomes constitute teacher effectiveness are the source of much debate *New study from Feng & Sass may prove differently!

4. Measuring Teacher Effectiveness • How we measure teacher effectiveness is impacted by the following: • What is valued • Our technological advances and limitations • The data, evidence, and information we have or can acquire • The resources (staff, money, time, policy levers) available to us • The cooperation of the teachers themselves 4

5. Emphasis on Student Achievement • Teacher effectiveness is often discussed by researchers and politicians in terms of teachers’ contributions to students’ learning as measured by test scores • This unfortunately means that other ways teachers contribute to student learning and well-being or to the culture and stability of the school are often not measured at all and may be given little consideration

6. Five-Point Definition of Highly Effective Teachers • Have high expectations for all students and help students learn, as measured by value-added or alternative measures. • Contribute to positive academic, attitudinal, and social outcomes for students, such as regular attendance, on-time promotion to the next grade, on-time graduation, self-efficacy, and cooperative behavior.

7. Five-Point Definition of Highly Effective Teachers • Use diverse resources to plan and structure engaging learning opportunities; monitor student progress formatively, adapting instruction as needed; and evaluate learning using multiple sources of evidence. • Contribute to the development of classrooms and schools that value diversity and civic-mindedness.

8. Five-Point Definition of Highly Effective Teachers • Collaborate with other teachers, administrators, parents, and education professionals to ensure student success, particularly the success of students with special needs and those at high risk for failure. Goe, L., Bell, C., & Little, O. (2008). Approaches to evaluating teacher effectiveness: A research synthesis. Washington, DC: National Comprehensive Center for Teacher Quality. http://www.tqsource.org/publications/EvaluatingTeachEffectiveness.pdf

9. Measures of teacher effectiveness • Value-Added Models • Becoming increasingly common • Considered “more objective” • Of little value in helping teachers improve their practice because value-added scores tell us nothing about what goes on in teachers’ classrooms • Teacher Observations • Great for formative evaluation but expensive to conduct (personnel time, training, calibrating) • Only as good as the instruments and the observers • Considered “less objective”

10. What are value-added models? Value-added models There are many versions of value-added models (VAMs), but results from the different models are quite similar Most states and districts that use VAMs use the Sanders’ model, also called TVAAS Prior test scores (3+ years in the Sanders’ model) are used to predict the next test score for a student They can identify teachers whose students are performing better or worse than expected (but say nothing about why) 10

11. Concerns with value-added models Ceiling effect (depending on state test used) Non-random matching of students/teachers Context effects (classroom and school) Missing data patterns Horizontal and vertical test equating Student growth trajectories Influences on test scores other than knowledge (students’ motivation, attitudes) 11

12. What Value-Added Models Cannot Tell You • Value-added models are really measuring classroom effects, not teacher effects • Value-added models can’t tell you why a particular teacher’s students are scoring higher than expected • Maybe the teacher is focusing instruction narrowly on test content • Or maybe the teacher is offering a rich, engaging curriculum that fosters deep student learning • How the teacher is achieving results matters

13. VAMs don’t measure most teachers 60-70% of teachers can’t be accurately assessed with VAMs Teachers in subject areas that are not tested Teachers in grade levels (lower elementary) where no prior test scores are available Special education & ELL 13

14. Validity in Measurement • Validity: The extent to which evidence and theory support an interpretation of scores for the use of the measure (i.e., a higher standard for evidence should be used when high-stakes decisions are associated with scores)

15. Threats to Validity in Observations • When the instrument is not sensitive enough to detect differences in teacher effectiveness • When the instrument doesn’t measure what is important and valued to all stakeholders, given the purpose of the observation • When raters are poorly trained in the use of the observation instrument • When raters have “drifted” over time, meaning they are no longer using the instrument in the way it was intended to be used

16. Other Measures • Many types of evidence*—including portfolios, administrator recommendations, analysis of teachers’ assignments, analysis of students’ work, documentation of teachers’ positive contributions to the school, student and parent reports, and documentation of teacher leadership and mentoring—can be used in addition to student test scores * For descriptions and discussions of instruments for measuring various aspects of teacher performance, see Goe, Bell, and Little (2008).

17. Evaluating Special Educators • Observers must take into account • Individualized, intensive nature of instruction to meet the IEP goals for each student • The greater focus on behavioral and social goals • Using achievement scores for teacher evaluation is problematic • Many students with special needs receive modifications such as extra time • Some are not tested at all, depending on IEP • Learning trajectories may be erratic across time • Which teacher gets “credit” for achievement?

18. Collaboration is Seldom Measured • Although collaboration between general and special educators is critical to instructional success for students with special needs, we are not very sophisticated about measuring it • How do you evaluate the very different role of a special educator in a co-teaching situation? • How do you evaluate the collaborative planning that goes on outside the classroom to ensure that the team will provide effective instruction for both general students and students with special needs?

19. Measuring What Is Important • Measure what is most important (to states, districts, administrators, teachers, and other education stakeholders) • The system will drive improvement • Teachers will strive to improve in areas that they know will be measured • Ensure that what they are striving for is truly important in a consensus definition of successful teaching

20. The Goal of Teacher Evaluation • The goal of all teacher evaluation should be to improve teaching and learning • Give all teachers opportunities to improve in areas in which they score poorly • Provide assistance in determining problem areas and planning strategies to address them • Provide resources to help them improve • Mentors or expert teachers (observe and be observed) • Targeted professional development • College courses (with tuition paid or reimbursed)

21. Laura Goe, Ph.D. P: 609-734-1076 E-Mail: lgoe@ets.org National Comprehensive Center for Teacher Quality 1100 17th Street NW, Suite 500Washington, DC 20036-4632877-322-8700 > www.tqsource.org