Striving to Link Teacher and Student Outcomes: Results from an Analysis of Whole-school Interventions

1. Striving to Link Teacher and Student Outcomes: Results from an Analysis of Whole-school Interventions Kelly Feighan, Elena Kirtcheva, and Eric Kucharik Research for Better Schools, Philadelphia, PA American Evaluation Association Annual Meeting, November 12, 2009 in Orlando, Florida

2. Study Purpose • Investigate which variables best explain student reading outcomes following teacher professional development • Explore the contextual reasons that help explain why no intervention “impact” was detected • Inform educational policy and improve rigor of educational research

3. Project Background • Federal Striving Readers program aimed at improving pedagogy and student achievement • Schools were matched in pairs and then randomly assigned to the treatment or control condition • Professional Development: four-semester course, onsite literacy coaching, leadership seminar, and curricular material • Developer’s hypothesis: integrating literacy strategies in content areas will yield student gains

4. Factors Affecting Student Learning • Student-level: SES, socio-demographic variables, family background, early development (Barton & Coley, 2009) • Teacher/classroom-level: expectations, preparation, experience, class size (Cohen, McCabe, Mitchelli, and Pickeral, 2009) • School-level: school climate - safety, student- adult and peer relationships, curriculum rigor (Cohen, McCabe, Mitchelli, and Pickeral, 2009)

5. Study Participants • 30 ELA teachers taught at eight schools • 16 taught at intervention schools • 14 taught at comparison schools • 2,114 students linked to these teachers • state assessment reading scores (N = 2,064) • ITBS scale reading scores (N = 1741)

6. Methodology • Quantitative data sources: • RBS teacher survey • School district school climate survey • Department of Education teacher HQT statistics and student discipline data • Students’ scores on state assessment and ITBS

7. Methodology • Qualitative data sources: • Observations • 56 classrooms (Year 1) • 48 classrooms (fall of Year 2) • 10 paired observations (spring of Year 2) • Interviews • 8 principals and 19 school improvement team members in Years 1 and 2 • Focus groups: seven groups with 62 teachers

8. Research Hypotheses • Exposure to professional development participants will yield gains in reading achievement • Including contextual variables in impact analysis will increase explanatory power of results

9. Quantitative Analysis • Used Hierarchical Linear Modeling (HLM) to predict student performance based on student-, teacher-, and school-level characteristics • Fully unconditional model represents how variation in an outcome measure is allocated across the three different levels

10. Variables Included in the HLM

11. Outcome Variables: Reading Scores

12. Student-Level Variation • Across multiple model specifications, the only predictors with statistical significance were the student’s • Pre-test score • Gender • ELL status • Modeling teacher-level factors produced no significant results

13. Classroom Observation Results • No baseline differences in levels of engagement & cognitive demand, or in instructional strategies • Cognitive demand level of lessons was low in Year 2, irrespective of research condition • Intervention teachers tended to use more literacy strategies than comparison teachers in Year 2 • 38.5% of intervention teachers used multiple literacy strategies vs 18.2% of comparison teachers

14. Why We May Not Find Impact • Low cognitive demand of lessons • Counterfactual situations may “water down” the treatment’s effect • Low implementation fidelity • Limitations in outcomes measures (just say measurement error)

15. Implications for Further Research • Better understanding of • the relationship between a school-level intervention and its potential to affect student achievement • Correlates of student achievement • Why an intervention that did not show impact may nevertheless be of value