Scoring and Standard Setting

1. Scoring and Standard Setting Jean Clayton, ILSSA June Quitugua, Guam CEDDERS

2. Introductions • “Agenda”/Housekeeping • History of the project • Purpose related to GSEG project • Cadre development ILSSA/CEDDERS/UoG 2009

3. Two Presentations in One Real World Jean’s Pretend World

4. Agenda • Day 1: • Mock scoring training • Score mock portfolios • Entity work on Discussions and Action Steps • Day 2: • Quick refresher on scoring • Complete scoring • Enter scores into database • Entity work on Discussions and Action Steps • Day 3: • Mock standard setting training • Mock standard setting with previously scored portfolios • Apply cut sores • Entity work on Discussions and Action steps

5. Purpose of Scoring Session • By participating in a mock scoring session participants will: • expand knowledge of the process, • determine minimum evidence requirements, • identify areas to design or refine entity scoring process, • and make revisions to entity scoring rubric as needed.

6. Purpose of Standard Setting • By participating in a mock standard setting participants will: • gain knowledge of the process, • identify purposes of the performance level descriptors, • practice applying cut scores, • identify areas to design or refine entity standard setting process, • and make revisions to entity performance level descriptors as needed.

7. Discussion and Action Steps • As scoring proceeds questions and clarifications will arise. • Use these issues to reflect upon the process already in place

8. Mock Scoring Training • Purpose is to practice the process: • Training is based on a mock rubric and mock portfolios. • Rubric may be similar to entity but is not intended to be exact. • Exact training, scoring rules, and examples need to be modified if used for entity training.

9. Scoring Materials • Scoring Rubric with Definitions • Score Sheet • Sample Portfolios • Training PowerPoint • Depth of Knowledge and Bloom’s Taxonomy References

10. Before Scoring • Score each standard separately. • Ensure that the work sample matches the grade level standard/benchmark/indicator. • Ensure that the evidence shows student work. • Ensure that the student performance is indicated by the teacher (i.e., % correct). • Ensure that the student performance indicated is independent.

11. Scoring Steps - Performance • Review the student work to determine how accurately the student performed. • Carefully read each item to ensure it was scored correctly. • Carefully read each item to ensure that it addresses the standard/benchmark/indicator. • Use the rubric to determine at which level the student is performing

12. Example of Performance – Level 4 The evidence shows that the student knew all of the information asked.

13. Example of Performance – Level 3 The evidence shows that the student had a couple of mistakes but didn’t appear to interfere with understanding.

14. Example of Performance – Level 2 The evidence shows that the student missed enough of the asked information to show that the student’s understanding was emerging.

15. Example of Performance – Level 1 The evidence shows many mistakes that does not indicate any student understanding.

16. Scoring Steps – Performance Evidence • Review the evidence provided to determine if it has provided accommodations, modified, or reflective or lower grade level work. • Has it been put in another format but kept the same rigor? • Has it been adapted but kept the same context? • Is it lower grade level work but age appropriate? • Is it age appropriate? • Use the rubric to determine at which level the work is best described.

17. Example of Performance Evidence – Level 4 The work on which the student was able to evidence his performance is age appropriate and on grade level. Although the accommodation of adding pictures to assist with reading, the text is exactly the same (not reduced or simplified) and the skill of identifying the correct meaning of a word using context maintained the same rigor.

18. Example of Performance Evidence – Level 3 The work on which the student was able to evidence performance is age appropriate and similar to the grade level work. However, the work was simplified from the work that of same grade level peers.

19. Example of Performance Evidence – Level 2 The work on which the student was able to evidence performance is age appropriate but is work typically completed by peers at lower grade levels.

20. Example of Performance Evidence – Level 1 The work on which the student was able to evidence performance is not age appropriate and is work typically performed by lower level peers.

21. Scoring Steps – Cognitive Level • Review each required task for the student to determine student level of understanding. • Use the Depth of Knowledge (DOK) and Bloom’s Tools as references. • Don’t rely only on the verbs – think about the level of understanding. • Use the rubric to determine the level that best describes the student’s highest level of understanding.

22. Example of Cognitive Level – Level 4 Student performance evidenced an understanding at the comprehension level and moves to a strategic thinking level when comparing the information in the two graphs.

23. Example of Cognitive Level – Level 3 Student performance evidenced an understanding at the comprehension level. The work required the student to understand the concept, inform, persuade, and entertain and then apply each to a unique passage.

24. Example of Cognitive Level – Level 2 The student performance evidences understanding at a basic recall as the student matches the word to the corresponding shape.

25. Example of Cognitive Level – Level 1 The student performance evidences understanding at a basic recognition level as the student points to the picture when asked to point to a show a particular item.

26. Scoring Steps – Performance Context • Carefully review the student work to determine how it relates to authentic tasks and what students would typically do at school. • Does the work relate to “real life” activities? • Is it the type of work that all students would do at school? • Use the rubric to determine the level that best describes the type of task being required of the student.

27. Example of Performance Context – Level 4 Authentic: Read from short passages from a biography and determined author’s purpose. Novel: Able to apply skill to a different context – reading newspaper and determined author’s purpose. Tasks completed at this grade level: These are activities that would be completed by same grade peers.

28. Example of Performance Context – Level 3 Authentic: Using a graph to identify the number of animals in a section of forest would be something that a researcher might do. Tasks completed at this grade level: A simulated authentic activity such as this would be done by general education students.

29. Example of Performance Context – Level 2 Not authentic: Isolated sentences without a reason to read them or identify the word meaning. Task completed at this grade level: this is a task that would be routinely given to students at this grade level but it is still age appropriate.

30. Example of Performance Context – Level 1 Not an authentic task: This is not an authentic task that would be completed by this grade level. The task is not age appropriate for 8th graders.

31. Common Errors - How to Avoid • Scoring by how it feels. • Use tools (e.g., rubric, definitions, examples, DOK chart, etc.). • Score only what is evidenced on the student work. • Develop detailed scoring rules. • Adjusting score to student ability. • Stick with scoring rules regardless of the disability. • Do not score lower due to accommodations and supports that do not change the context or rigor.

32. Scoring Process Round 1: Score each portfolio. Label score sheet and place in folder. Place portfolio in second score tray. Round 2: Score each portfolio from second score tray (do not look at first scores). When finished scoring, remove first score sheet from folder and paperclip two score sheets to the portfolio. Place the portfolio with score sheets into the final score tray. Round 3: Compare the first and second scores. Have the two people who scored and a third person (if available) to discuss any differences and agree upon a final score. Record score information into the database.

33. Discussion and Action Steps • Discuss issues that have risen during scoring. • Discuss possible solutions. • Determine next steps: • What • Who • when

34. Purpose of Standard Setting • By participating in a mock standard setting participants will: • gain knowledge of the process, • identify purposes of the performance level descriptors, • practice applying cut scores, • identify areas to design or refine entity standard setting process, • and make revisions to entity performance level descriptors as needed.

35. Performance Level Descriptors ILSSA 2009

36. Achievement Standard • Three parts: • Name of each level (e.g., below basic, basic, proficient, advanced) • Written description of each level (Performance Level Descriptor or PLD) • Minimum cut score for each level

37. Purpose of PLD • Setting cut scores • Reporting tool: • Helps teachers and parents see what their students know and can do in academic areas • Development tool • Helps teachers plan instruction and set teacher expectations

38. Development of PLD • Should be developed prior to applying cut scores. • Developed at a policy level prior to standard setting. • Additional information can be added during the standard setting process (e.g., to add more content detail as viewed in sample portfolios).

39. Policy and Stakeholder Members • Policy makers may determine wording to describe intended rigor meant for each performance level • Stakeholder members, including content experts, general education and special education teachers will expand on generic terms using content specific explanations across grade levels

40. Considerations • Ensure that PLDs appropriately reflect the content assessed. • A progression should occur across performance levels and across grade levels.

41. Steps for Drafting PLDs for AA-AAS • Convene the stakeholder group • Provide background information on the population of students • Focus first on writing Proficient PLD • Have Participants brainstorm what a student should know regarding each content strand (benchmark, performance indicator, etc.) and how well the student should be able to do it in relationship to the rubric dimensions

42. Steps for Drafting PLDs - continued • Keep a list of the ideas in bullet format • Move to Basic and write statements for that level • Move to Advanced and follow the same process • Continue with any other levels • Consider adjacent grades • Use as a bulleted list or write into a descriptive paragraph • Review all PLDs for a clear progression and ability to translate to instruction (Perie, Hess, and Gong, 2008)

43. Marianne Perie, K. H. (2008, March 27). Writing Performance Level Descriptors: Applying lessons learned from the general assessment to alternate assessments based on alternate and modified achievement standards. Retrieved January 5, 2009, from nciea: http://www.nciea.org/publications/ALDs_BGKHMAP08.pdf •  Perie, M. (2007, April 27). A Guide to Understanding and Developing Performance Level Descriptors. Retrieved January 5, 2009, from NCIEA: http://www.nciea.org/cgi-bin/pubspage.cgi?pgRequested=11&curPage=12&sortby=auth_lastnm&itemsToDisplay=10

44. What is Standard Setting? • Applying professional judgment to the question “How much is just enough?” • It results in a set of performance level descriptors and recommended cut scores that separate performance levels.

45. What Is a Cut Score? Minimum test score a student must earn to be considered at a certain performance level Below Basic Basic Proficient Advanced Cut Score A Cut Score B Cut Score C

46. Overview of Standard-Setting Process • Analyze the rubric and determine the minimum score required to reach each performance level by • Content strand • Performance dimension • Discuss cut scores as a group and come to consensus • Examine student papers • Review impact/consequences data • Make a final recommendation of the cut score for each performance level • On overall total score

47. Performance Level Descriptors • Review the reading PLDs • PLDs describe what students should know and be able to do at each performance level • With respect to the grade level standards/benchmarks/indicators • What should Basic performance look like? • What should Proficient performance look like? • What should Advanced performance look like?

48. First Task: Rubric Analysis • Individually determine the minimum score you would expect a student in each performance level to demonstrate • What is the lowest score you would feel comfortable calling Basic? Proficient? Advanced? • How would they likely perform on each performance dimension? • Would it vary by content standard? • Ultimately we will sum the scores across content standards to calculate a cut score for each dimension • Record it, submit it, and then we will discuss it

49. Reminder: Scoring • Students receive 0–4 points on each dimension within each standard • There are 4 dimensions and 2 standards • Total possible score = 0–32 points • Many ways to get different scores

50. Second Task: Discussion of cuts • Examine the results as a group • Achieve consensus on the minimum score for each performance level • What would be the minimum score on each performance dimension? • What would the minimal acceptable total score at each level?