More Z to A: Setting the STAAR Cut Points from End to Beginning

1. More Z to A: Setting the STAAR Cut Points from End to Beginning Gloria Zyskowski, TEA Laurie Davis, Pearson

2. Session Overview • Standard Setting 101 • STAAR Standard Setting Process • Studies • Performance labels and descriptors • Policy and standard setting committees • Implementation and Standards Review • Phase-In • Timelines • Questions?

3. Standard Setting 101 • Different types of standards • Content Standards • Performance Standards • Accountability Standards • Process of determining the level of knowledge and skill students need to demonstrate fall into the various performance levels. • Involves obtaining recommendations from those who are knowledgeable of • the content/skills to be assessed • the test population • the ways in which test scores are used

4. Why is 70% not passing? • Common passing score for classroom tests • However…standardized tests work differently • Use scale scores to account for differences in difficulty of test questions across the many years of an assessment program • i.e. TAKS 2003-2011 • Therefore, passing score is a scale score rather than a percent correct score • This lets us hold students to the same standard across years regardless of the specific set of test questions they took

5. Standard Setting 101 • Combines content, data, and policy expertise

6. Alignment of Performance Standards • Start with college readiness • Align to high school standards • Vertically align down to elementary school

7. Performance standards based on empirical evidence from student performance • across courses and grades • on external assessments

8. STAAR Standard-Setting Process Nine steps for setting STAAR performance standards: • Conduct validity and linking studies • Develop performance labels and policy definitions • Develop grade/course specific performance level descriptors • Policy committee • Standard-setting committee • Reasonableness review • Approval of performance standards • Implementation of performance standards • Review of performance standards

9. Step 1:Conduct Validity and Linking Studies

10. Why Studies for Setting Performance Standards? • Texas’ goal is to be in the Top 10 in terms of college readiness by 2019–2020. • Comparisons of Texas standards with national and international standards are important in meeting this goal. • Studies will be conducted at least every three years to update standards and monitor progress.

11. Conduct Validity and Linking Studies Studies to inform the college readiness standard: • Follow students from high school to college • Compare EOC performance and other test performance • SAT • ACT • ACCUPLACER • THEA • College students take STAAR EOC assessments

12. Conduct Validity and Linking Studies Studies to inform standards for STAAR EOC assessments: • Relationship between test performance in the same content area • Relationship between test and course performance • Comparison to TAKS • Comparison with NAEP

13. Conduct Validity and Linking Studies Studies to inform standards for STAAR 3–8 assessments: • Relationship between test performance in different grades in the same content area • Relationship between grade 8 and high school • Comparison with TAKS • Comparison with NAEP • Vertical Scale

14. SAT Mathematics (not SAT Subject Test) Arithmetic operations Algebra Geometry Statistics Probability Etc… STAAR Algebra II Quadratic Functions and Relations Square Root Functions Rational Functions Exponential and Logarithmic Functions Etc… Example: Algebra II and SAT • Content Overlap • SAT (general) is broader, but STAAR is deeper

15. Example: Algebra II and SAT STAAR EOC Algebra II SAT Mathematics Validity Study Sample

16. Additional Considerations • Student motivation now and then • 2010 was a stand-alone field test for Algebra II • Does not count for students prior to 2012 • Algebra II is a newly-tested content area • Required course as part of 4x4 • Required assessment for graduation (recommended and distinguished achievement programs) • Instruction will improve • Curriculum may be changed/updated • SAT is typically taken by college-bound students

17. Step 2:Performance Labels and Policy Definitions

18. Performance Labels and Policy Definitions Developed by committee convened in September 2010 Describe the general level of knowledge and skills evident at each performance level Describe students in the middle of each performance level Apply to all assessments November 2011 Texas Education Agency Student Assessment Division 18

19. There will be two cut scores, identifying three performance categories Level III: Advanced Academic Performance Level II: Satisfactory Academic Performance Level I: Unsatisfactory Academic Performance Performance Labels and Policy Definitions November 2011 Texas Education Agency Student Assessment Division 19

20. Policy Definition of Level III: Advanced Academic Performance Performance in this category indicates that students Are well prepared for the next grade or course Demonstrate the ability to think critically Demonstrate the ability to apply the assessed knowledge and skills in varied contexts, both familiar and unfamiliar Have a high likelihood of success in next grade or course with little or no academic intervention November 2011 Texas Education Agency Student Assessment Division 20

21. Policy Definition of Level II: Satisfactory Academic Performance Performance in this category indicates that students Are sufficiently prepared for the next grade or course Generally demonstrate the ability to think critically Generally demonstrate the ability to apply the assessed knowledge and skills in familiar contexts Have a reasonable likelihood of success in the next grade or course May need short-term, targeted academic intervention November 2011 Texas Education Agency Student Assessment Division 21

22. Policy Definition of Level I:Unsatisfactory Academic Performance Performance in this category indicates that students Are inadequately prepared for the next grade or course Do not demonstrate a sufficient understanding of the assessed knowledge and skills Are unlikely to succeed in next grade or course without significant, ongoing academic intervention November 2011 Texas Education Agency Student Assessment Division 22

23. Performance Labels Texas Education Agency Student Assessment Division 23 November 2011

24. Step 3:Specific Performance Level Descriptors (PLDs)

25. Specific Performance Level Descriptors (PLDs) Descriptions of what students know and can do in relation to the content standards at each performance level Translate the policy definitions into course- specific descriptions of student achievement at each performance level November 2011 Texas Education Agency Student Assessment Division 25

26. Specific Performance Level Descriptors (PLDs) November 23 EOC Science EOC Social studies November 89 EOC Mathematics EOC English November 2011 Texas Education Agency Student Assessment Division 26

27. Specific Performance Level Descriptors (PLDs) Are required by USDE for peer review Are a vital part of setting performance standards May be used as a resource for classroom instruction May be used as a resource for item development November 2011 Texas Education Agency Student Assessment Division 27

28. Characteristics of Specific PLDs Connect directly to the knowledge and skills in the content standards for each course Reflect the level of cognitive demand represented in the standards Describe students in the middle of each performance category November 2011 Texas Education Agency Student Assessment Division 28

29. Tasks in Developing Specific PLDs Culminating Skills Represent the “big ideas” in the content standards May subsume prerequisite and/or enabling skills Are associated with satisfactory level achievement Serve as a starting point for specific PLD development Were developed by TEA content specialists in consultation with educators from across the state November 2011 Texas Education Agency Student Assessment Division 29

30. Approach for Developing Specific PLDs November 2011 Texas Education Agency Student Assessment Division 30

31. Step 4:Policy Committee

32. Policy Committee Scheduled for February 2012 Considers policy implications of the performance required at different achievement levels Provides guidance on the results of the research studies Provides advice about the reasonable ranges or “neighborhoods” for cut scores November 2011 Texas Education Agency Student Assessment Division 32

34. Validity Studies Support Neighborhoods

35. Step 5:Standard-Setting Committee

36. Standard-Setting Committee • Committee will follow a research-based standard-setting process • Cut scores will be informed by • Test content (item difficulty, required skills) • Alignment within content area • Neighborhoods set by policy committee • External study results (e.g. SAT, ACT) • Linking studies (e.g. Algebra I to Algebra II) • Student performance (estimated % passing) • Expert judgment • Cut score will be set starting with highest grade/course, with lower grade/course vertically aligned to higher grade/course

37. Cut Score Placement

38. Step 6:Reasonableness Review

39. Reasonableness Review • March 2012 • Look across subject areas to determine reasonableness of system of EOC standards • Look at recommended cut scores relative to policy considerations • Conducted by TEA staff with advice from representatives from policy committee

40. Step 7:Approve Performance Standards

41. Approve Performance Standards • STAAR EOC: March 2012 • STAAR 3–8: December 2012 • College readiness performance standards on English III and Algebra II approved by commissioner of education and commissioner of higher education • All other performance standards approved by commissioner of education

42. Step 8:Implement Performance Standards

43. Implement Performance Standards • STAAR EOC: May 2012 (first high stakes administration) • STAAR 3–8: Late fall 2012 or early 2013 • New standards are expected to be used in state and federal accountability systems starting in 2013

44. Step 9:Review of Performance Standards

45. Review Performance Standards • Legislative requirement to review performance standards at least once every three years • First review in 2014 • Consider additional data from research studies • Longitudinal data (follow cohorts of TX students from EOC to college and careers) • Substitute tests (AP, IB, SAT subject) • Military service • Workforce • College readiness for science and social studies (depending on findings of feasibility studies)

46. Phase-In of Performance Standards • Like TAKS, STAAR will have a phase-in period to provide school districts with an appropriate amount of time to • improve instruction • provide new professional developmentincrease teacher effectiveness • close knowledge gaps • Phase-in will apply to Level II: Satisfactory Academic Performance

47. Timelines for Standard Setting • Standard Setting Committees will convene in: • EOC: February 2012 • STAAR Alternate: August 2012 • STAAR EOC Modified: September 2012 • STAAR 3-8: October 2012 • STAAR 3-8 Modified: November 2012

48. Questions?

49. Session Code <Insert Session Code here>