Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd

1. Enhanced Assessment Grant Integrating Simulation-Based Science Assessments into Balanced State Science Assessment Systems:Findings and Implications Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd Presented to the Council of State Science Supervisors, San Francisco, CA, March 9, 2011

2. Enhanced Assessment Grant State Partners • Nevada (Lead State) • Utah • North Carolina • Massachusetts • Connecticut • Vermont

3. Advisors • Rebecca Kopriva, Wisconsin Center for Education Research • Assessments for English Learners • Martha Thurlow, National Center on Education Outcomes • Assessments for Students with Disabilities

4. Subcontractors • WestEd STEM (SimScientists) • Project Management • Content Development • Redhill Studios • Interactive (Flash) Programming • UCLA/CRESST • External Evaluation

5. EAG Goals • Establish the technical quality of simulation-based assessments • for two topics, ecosystems and force and motion • Study the feasibility and utility of simulation-based assessments in classrooms from 3 states

6. EAG Goals • Study the measurement of science knowledge and inquiry by simulation-based curriculum-embedded assessments and unit benchmark assessments in comparison to traditional items for: • All students • ELLs • Students with disabilities • Propose alternative models for integrating simulation-based assessments into state assessment systems

7. Relevance to Current Assessment Programs • Need innovative, technology-enhanced assessments that align with new frameworks that focus on • fewer, deeper, more integrated core knowledge targets • E.g., the new Framework for Science Education and next generation national science standards • Models as structures for understanding and studying science systems (model-based learning) • Science practices for using knowledge and inquiry in significant, recurring authentic tasks

8. Relevance to Current Assessment Programs Need innovative, technology-enhanced assessments that • Target 21st century skills within STEM • Use technology to engage students in use of 21st century “tools of the trade” • Provide evidence supporting one type of technology-enhanced performance assessment • For summative and formative purposes • Called for in the RTT Consortia

9. EAG Design • Align to national science standards and to science standards of the pilot states • Ecosystems and force & motion • At middle school level • Nevada, North Carolina, Utah • Accommodations • Extended time • Audio • Zoom

10. Ecosystem Target Model

11. Multiple Modes of RepresentationActive Inquiry Organism box shows icons that appear and disappear Graph shows size of population over time. Data inspector shows population for one point on graph. Table shows population at start, end, and point selected by data inspector.

12. Assessment Module Components for Each Topic/System Online assessment with feedback and coaching Follow up Classroom Reflection Activity Progress report Embedded in Classroom Instruction EmbeddedAssessments for Formative Purposes (2 to 4) Benchmark Summative Unit Assessments Online assessment without feedback Teacher scores constructed responses Bayes Net Proficiency report

13. Ecosystem Embedded Assessment: Feedback, Coaching, and Self-Assessment Demo

14. Progress Reports to Students

15. Progress Reports to Teachers NH = needs help P = making progress OT = on track

16. Grouping Recommendations for Classroom Reflection Activity

17. Classroom Reflection Activity • Formative use of assessment results • Students assigned to teams based on embedded results • Transfer to different, more complex system • Jigsaw structure • Allows differentiated instruction via tasks of varying difficulty • Promotes small and large group discourse and collaboration • Guidance for teacher • Teacher review of key points in simulation • What to look for during group work and questions to pose in response • Presentations • Evaluation of presentations

18. Benchmark Assessment:Assess Transfer to New Context Demo

19. Benchmark Report: Class Summary

20. Benchmark Report: Student Details

21. Samples • 28 think aloud sessions • Pilot testing in 4 states • 60 participating teachers • ~5,800 students • 40 schools • 28 districts • Response data on the assessments, post tests and classroom observations

22. Research Evidence • Quality of the assessments (external review) • Alignment with standards and curriculum • Coherence • Science content and inquiry • Item design • Implementation • Feasibility in schools and classrooms • Utility for students and teachers

23. Research Evidence • Technical Quality (Validity & Reliability) • Alignment with science standards • Cognitive labs • Analyses of responses • Comparison of student responses on simulation-based and traditional items • • Cognitive labs • • IRT model fits

24. Reliability of Benchmark and Post Tests • Ecosystems • Benchmark (n=3844) • Mean score was 28.14 (max 39) • Standard Deviation 6.64 • Coefficient Alpha: 0.85 • Post Test (n=3422) • Mean score: 16.52 (max 30) • Standard deviation: 6.68 • Coefficient Alpha (KR 20): 0.88

25. Reliability of Benchmark and Post Tests Force and Motion • Benchmark (n=1504) • Mean score: 28.15 (max 40) • Standard Deviation: 5.51 • Coefficient Alpha: 0.79 • Post Test (n=2166) • Mean score: 14.87 (max 28) • Standard Deviation: 5.73 • Coefficient Alpha (KR 20): 0.82

26. Correlation of Benchmark to Posttest ** Correlation is significant at the 0.01 level (2-tailed)

27. CRESST Evaluation Questions • How effective was SimScientists implementation? • Was SimScientists feasible and useful in middle school classrooms? • How effective were SimScientists’ accommodations?

28. CRESST Evaluation Methods • Convenience sample of teachers from all three pilot states, both topics • Eight teachers with 3-4 classes • Five schools • Observation and interview protocols • Assessments • Reflection Activities • Teacher survey

29. Teacher’s Role (Assessment)

30. Class Organization (RA)

31. CRESST Interviews “I think that [students] are way more engaged … using this kind of interactive assessment … it was more engaging them simply, “Here is a pencil, here is a paper, answer questions” .”

32. CRESST Interviews • “…I like that kind of feedback when it doesn’t just go to the next page and they don’t know whether they did it right or not.” • “…Yes, the science content is really being tested. Students are asked to conduct experiments, investigate, and draw conclusions and to use scientific skills. Students are not able to guess on the multiple choice questions because it probes them until they choose the right answer. Students are also taught about food webs in one biome and they are tested on another biome.”

33. CRESST Evaluation Conclusions • Observations showed that students were actively engaged most of the time during assessments. • Both teachers and students generally believed that the SimScientists program was beneficial to learning. • Teachers found the automatically scored, immediate feedback—especially the reports generated by the questions—helpful to students. The instant reports allowed teachers to easily see which questions students had the most difficulty with so that they could tailor their lessons accordingly.

34. CRESST Evaluation Conclusions • Teachers collectively agreed that the simulation assessments had greater benefits than traditional paper-and-pencil tests because of the simulation’s instant feedback, interaction, and visuals. • Teachers agreed that the assessments would be useful in measuring their individual state standards.

35. Multilevel Balanced Assessment System Models • Reporting benchmark results alongside district and state data • Matrix sampling of short “signature” tasks from different topics

37. Matrix Sampling Short “Signature Tasks” Across Topics

38. Policy Brief • Summary of alternative models for integrating simulation-based assessments into balanced state assessment reporting systems • Summary of study findings on feasibility, utility, and technical quality • Implications for further study

39. Related Activities • Increased use of technology in classroom instruction and assessment • Continued (and growing) frustration around measuring student’s knowledge and skills above conceptual level • Growth in costs associated with hand scoring of CR items on science assessments at grades 5 and 8 • Development of electronic and on-line resources for instruction, assessment, and professional development for math and science • MINES, Science Builder, Pathways to Nevada’s Future

40. Other Projects • Race to the Top • Assessment consortium related to CCSS in math and reading • Multilevel Assessments of Science Standards • Science Assessments with UDL • EAG proposal for 2011? • Study series of assessment suites within a grade

41. Contact information Email: rvineyard@doe.nv.gov equellm@wested.org msilber@wested.org Website: www.simscientists.org