210 likes | 326 Vues
Impact of Interruptions on Test Scores in Indiana. Richard Hill June 25, 2014. Two Parts. Initial study prior to presentation to Legislative committee Quick Outline Details in paper: http://www.nciea.org/publication_PDFs/ISTEP%20RH072713.pdf Follow-up study
E N D
Impact of Interruptions on Test Scores in Indiana Richard Hill June 25, 2014
Two Parts • Initial study prior to presentation to Legislative committee • Quick Outline • Details in paper: http://www.nciea.org/publication_PDFs/ISTEP%20RH072713.pdf • Follow-up study • http://www.nciea.org/publication_PDFs/GainsMadebyInterruptedStudents_RH092513.pdf
Three Confounding Factors • New policy for retention of students in Grade 3 • Transition from paper and pencil to computer administration • Interruptions
Change from Paper and Pencil to Computer Administration • 2009 and 2010 – < 10% • 2011 – 36% • 2012 – 71% • 2013 – 95%
Change Not Even Across Grades • 2012 • Grade 6 – 66% • Grade 7 – 86% • Grade 8 – 92%
Additional Center Analyses • School-by-school improvement at same grade between 2012 and 2013 • School-by-school gain, following same cohort of students across grades • Student-level gain by students matched from 2012 to 2013
CTB Analyses • Group Analyses • Overall statewide averages • Interrupted vs. non-interrupted within 2013 • Scores before interruption vs. scores after interruption • Individual Analyses • Before vs. after interruption • Performance predicted from previous tests
Presentation to Legislative Committee • Was overall finding of no change a function of two factors? • Some students adversely affected by interruption • Other students taking advantage of interruption to learn answers from outside sources, then changing answers when testing restarted
Available Data • CTB could provide A+B and C+D • That is, they knew at which item student was interrupted, and they knew, by item, whether student had made a change • But they couldn’t tell (easily) exactly the time the change was made
Analysis of Table • Cell B is the event of interest—changes made after interruption to items presented before interruption • But C = 0, and A should equal D • So B = (A + B) – (C + D)
Conclusion • Concern raised by legislators was confirmed—data show that students changed answers from wrong to right more often after interruption than before • Impact on overall results was negligible—less than 0.2 scaled score points (on tests with standard deviations of 50-75)
% Making No Changes from Wrong to Right (Grades 3-5, Math Only)
Average % of Change from Wrong to Right (Grades 3-5, Math Only)
Conclusions from Second Analysis • Students reported by CTB as interrupted had higher rates of change from wrong to right • Again, estimated impact is less than 0.2 scaled score points • Students reported as interrupted by locals had lower rates of change from wrong to right
Consistent with first Impact on overall results negligible-- Conclusion from Second Analysis