Holistic Scoring in Social Studies

1. Holistic Scoring in Social Studies Curriculum, Instruction, and Instructional Technology Team New York State Education Department Albany, NY 2005

2. Curriculum & Instruction Jo Ann Larson jlarson@mail.nysed.gov Lawrence Paska lpaska@mail.nysed.gov Phone: (518) 474-5922 State Assessment Donna Merlau dmerlau@mail.nysed.gov Gary Warren gwarren@mail.nysed.gov Phone: (518) 474-3860 Contact Information NYS Education Department 2005

3. Purposes of the Regents • Accountability: To measure student attainment of the Learning Standards for Social Studies at the commencement level • Diagnostic: To help school districts identify those students who are in need of Academic Intervention Services (AIS) NYS Education Department 2005

4. Goal • To achieve consistent and accurate scores Note: not simply to have everyone assign the same scores, but to assign them for the right reasons. NYS Education Department 2005

5. Purpose of This Workshop • Make the rating task as simple and painless as possible NYS Education Department 2005

6. Cast of Characters PRINCIPAL / SOCIAL STUDIES COORDINATOR • Appoints a Scoring Coordinator who will manage the training and logistics of the scoring process • Appoints a Trainer NYS Education Department 2005

7. Cast of Characters • SCORING COORDINATOR • Manages the training and logistics of scoring • Organizes the movement of papers throughout the entire scoring process • Records scores on Part I answer sheets NYS Education Department 2005

8. Cast of Characters • TRAINER • Responsible for the overall training session, including time management (starting on time, moving from step to step of training procedures, keeping to schedule) • Responsible for the substance of scoring • Leads review of: specific rubric, procedures for assigning holistic scores, each anchor paper and commentary, practice papers NYS Education Department 2005

9. Cast of Characters • TRAINER • Follows the procedures and adapts them when necessary to ensure that everyone can assign valid and consistent scores • Answers questions that arise over how to evaluate papers or assign scores • Monitors scoring session progress; helps keeps Raters on schedule • Assists the Table Leaders with back-reading NYS Education Department 2005

10. Cast of Characters • TABLE LEADERS • Distributes the bundles of papers to be scored (with Support Staff help) • Leads review of the generic rubric and specific rubric with reference to the task by matching evidence from the student responses to the rubric • Leads review of each anchor paper and commentary NYS Education Department 2005

11. Cast of Characters • TABLE LEADERS • Leads discussion until Raters feel confident to move on to actual rating; discusses any discrepancies • Helps the Trainer with the substantive issues that arise during discussion (e.g., why a paper received a certain score, how to explain a distinction between score levels) NYS Education Department 2005

12. Cast of Characters • TABLE LEADERS • Back-reads several papers for each scorer, to confirm that Raters assign scores consistently • Serves as a third Rater to resolve scores that differ by more than one point • Calculates a final score for each student’s essay • Ensures that Raters fill out score sheets correctly • Assembles all score sheets and training materials at end of scoring session NYS Education Department 2005

13. Cast of Characters • RATERS • Scores the five practice papers without looking at the scores and commentaries provided • Records own individual rating for each student’s essay on the rating sheet provided (not directly on the student’s paper/answer sheet), when the actual rating begins; the scores of each scaffold question may be recorded in the student’s examination booklet NYS Education Department 2005

14. Cast of Characters • RATERS • Should not correct the student’s work by making insertions or changes of any kind • Note: Each essay must be scored by two different Raters; scaffold questions only need scoring by one Rater NYS Education Department 2005

15. Cast of Characters • RATERS: QUALIFICATIONS • Teachers of Grades 7-12 social studies • Special Education Teachers who are knowledgeable about the GHG or USHG social studies curriculum • Should have previous school, district, and/or regional training on scoring social studies essays or scaffold questions as part of the turnkey training process NYS Education Department 2005

16. Cast of Characters • SUPPORT STAFF • Distributes the bundles of papers to the tables • Moves the papers so that all parts and all papers are completely scored • Collects the score sheets and training materials, moving them to shipping centers or a storage facility, at the end of the session NYS Education Department 2005

17. The Rating Process • 1. BEGIN • Select a Table Leader. • Review each scaffold question and/or essay task. • Discuss possible student responses. • Review and discuss the specific rubric. • Discuss the nature of responses that would be expected at each score point of the rubric. • Remember: This is a holistic scoring. A student does not need to achieve a “5” for each bullet in the rubric in order to be awarded a “5” on the essay. A “5” is not necessarily a perfect paper. NYS Education Department 2005

18. The Rating Process • 2. REVIEW THE SET OF ANCHOR PAPERS • Read the student anchor papers individually. • Starting with the first anchor paper, the group should discuss the scores that were awarded. Using the specific rubric, explain why each anchor paper earned the score that it did. • Read the commentaries at the back of the packet for each of the anchor papers, to find the rationale for each paper’s score. NYS Education Department 2005

19. The Rating Process • 3. REVIEW THE PRACTICE PAPERS • Read the student practice papers individually. • Using the specific rubric, each Rater at the table awards a score to each practice paper without looking at the scores and commentaries provided after the papers. • After each Rater has separately scored the practice papers, the Table Leader charts the scores and leads discussion on the scores that were awarded. NYS Education Department 2005

20. The Rating Process • 3. REVIEW THE PRACTICE PAPERS • Where there are discrepancies, the Table Leader should ask individuals with the most discrepant scores to discuss their rationale for awarding credit. The group should continue to discuss the scores until consensus is reached. (See the commentaries at the back of the packet for the scores awarded for each practice paper.) Discuss any discrepancies between the SED commentaries and the scores awarded above. NYS Education Department 2005

21. The Rating Process • 3. REVIEW THE PRACTICE PAPERS • If additional practice is required to reach scoring consensus, a sample of actual student answer papers from the test may be used. NYS Education Department 2005

22. Setting the Scene … • Provide frequent breaks. • Provide incentives (e.g., water, snacks, meals). • Conduct the scoring session in a pleasant environment. • Check for understanding of rating procedures frequently. • Review the rubric after any long-term break (e.g., a lunch period, overnight). • Work toward exact agreement in awarding credit. NYS Education Department 2005

23. Materials Needed • Photocopiers • Fax machines • Computers and printers • Paper • Chart paper • Paper clips and rubber bands • Pens, pencils, markers, and highlighters NYS Education Department 2005

24. Checking for Accuracy • Do the Raters share a common interpretation of the rubric, or do some Raters have their own personal construct of the rubric? • Are Raters applying the rubric appropriately, or are they influenced by irrelevant factors when scoring student performance? • Are Raters using all of the criteria, or are they ignoring some aspects? • Are Raters assigning scores reflective of the rubric? NYS Education Department 2005

25. Other Reminders • “Scoring Key and Rating Guide” may not be used until after the Uniform Statewide Admission Deadline (45 mins. after exam start time). • Set aside one room as a central rating room for collecting, storing answer sheets/essay booklets, and for preparing/maintaining records, for the exam. • Avoid having a teacher rate his/her own students’ responses. • Provide adequate time for training and for rating. NYS Education Department 2005