Distributed Scoring Regents June 2012

1. Distributed ScoringRegents June 2012 CFN 603, May 14 2012 Michael Tancredi <mtancre@schools.nyc.gov> Gary Carlin <gcarlin@schoos.nyc.gov>

2. Agenda • Overview: Background • Scoring Models • CFN 603: Non-Electronic Distributed Scoring • Pre-scoring Procedures • Scorer Training • Scanning • Resources and Q&A

3. Distributed Scoring • Rules and processes by which exams are assigned to teachers for scoring. • NYSED requirement: 2012-13 school year, teachers can longer score the state exams of students that attend their school (MS 6/13) • Scoring sites: score exams collectively using a non-electronic and electronic distribution system

4. NYC January 2012 Pilot • Only schools with grades 9-12 (7 sites/27 sch) • BxHSVA, Astor, PPA • Only English language versions • 3 Exams: • Comprehensive English, • Integrated Algebra, • Living Environment

5. Scoring Model: Non-Electronic • Regional Scoring: Staff members from multiple schools (feeder schools) gather at a host school site to collectively score exams. • Campus Scoring: Staff members located in one building with multiple high schools remain in their building to score exams.

6. Electronic Model • Scanned images of student responses are electronically distributed to scorers either at a central (Regional) scoring site or at individual schools (School-based Scoring). • Exams picked-up by courier and shipped to vendor who will scan student responses for display in the online system.

7. CFN 603 Schools: Non-Electronic • Regional: (1) Choir Academy, (2) Felisa Rincon, (3) Wings, (4) ASE, (5) Astor, (6) BxHSVA, (7) Bronxwood, (8)PPA, (9) Law Enforcement • Campus: (1) Writing & Communication, (2) Global Citizenship, (3) Fordham HS for the Arts

8. Exam Distribution • Regional (10/102): Exams will be picked-up by courier from participating schools for scoring at host site (Afternoon 3-5pm and Next day morning) • Campus (10/44): Exams will be brought to the appropriate room within the campus for scoring.

9. Regional Host Sites • Choir Academy -> A. Philip Randolph • Felisa Rincon -> Bronx HS of Science • Wings -> Bronx HS of Science • ASE -> Clinton • Astor -> Clinton • BxHSVA -> Clinton • Bronxwood -> Clinton • PPA -> Clinton • Law Enforcement -> Forest Hills

10. Campus Host Schools • Writing & Commun -> Evander • Global Citizenship -> Prospect Heights • Fordham HSA -> Roosevelt

11. NYC June 2012 • Selection: Number of exams expected to administered and location/proximity to other schools • 4 exams: (Non-Electronic) Comprehensive English, Integrated Algebra, Living Environment, US History • 3 exams: (Electronic) Comp Eng, Int Alg, Global • Only English versions

12. Scoring Site Staff Structure

13. Site Supervisor (1-2) • Appointed Supervisor: (Regional)selected by host school principal, (Campus) selected by committee of principals • Manage scoring site and supervise all activities related to the scoring facility, RGTU; be on-site for duration of scoring

14. Scoring Content Leaders(4+) • Content expert (AP or teacher): 4 exams • Train scorers and oversee scoring; be on-site for duration of scoring for content exam • Determined at principal’s planning session

15. Scorers (varied) • Teachers: certified in subject area of exam • Score accurately; complete a sufficient number of test booklets to finish scoring within allotted time frame. • Selected by principals (mid May and entered online with selection tool)

16. Organizational Team Lead (1) • Teacher or Supervisor with ATS access • Scan answer documents for scoring site; manage error correction with oversight of site supervisor; additional site logistics support • Selected by principals at planning meeting

17. Organizational Team Members (5% of scorers) • Teacher or Non-Instructional Staff with strong organizational skills and attention to detail (LIFT EXAM BOXES!!!) • Assist scoring site supervisors and content leaders as needed to check-in, distribute, and check-out test materials • Determined by principals at planning meeting

18. Wednesday, May 2 • The Scoring Site Staff (Site Supervisor, Scoring Content Leaders, Organizational Team lead and Members) must be identified and communicated to Networks – with all CONTACT INFROMATION: Email, School Phone and Extension, [and Cell Phone] • No later than Wednesday, May 2.

19. Scorer Selection • Principal receives an email that indicated the # of scorers per subject that the school is required to assign to scoring location (end of April) x • Principals enters scores via web-based scorer selection tool (early-mid May) • Scorers are notified of selection and site logistics (after principal submission)

20. Staff Time Commitments • 8:30 am – 2:50 pm (Campus can be flexible) Bring PHOTO IDENTIFICATION • Schools should try to complete final examinations, LOTE Speaking section, Regents Performance Tests and portfolio presentations as early as possible. • Teachers selected for scoring will not be able to proctor exams on the days they have been selected to proctor. • Three (3) Day commitment for scoring

21. Scoring Administration Procedures • Scorers and teachers are required to report to scoring sites with copies of rating guide released on the day of the exam. (note: IA is a PM exam!) • Test booklets will be scored and answer documents will be scanned at the scoring site (except for exams in a language other than English). • Morning exams will be delivered in the afternoon (+3 pm) and afternoon exams will be delivered the following school day (+8 am) by a courier service.

22. Scoring Selection Timeline

23. Scorer Eligibility • Eligibility is ultimately determined by the principal, but the initial eligibility screening uses the following information from Galaxy: • Scorers must be currently appointed pedagogues to be eligible to score • Staff who are not pedagogues (e.g. guidance counselors) cannot score, even if they have eligible state certifications, SpEd must have certification in exam area (dual certification) • State Certification - Scorers must have a subject-specific license to be eligible to score: • Living Environment scorers must have a Science license, such as: • Biology 7-12, Chemistry 7-12, Physics 7-12, Earth Science • English scorers must have an English license, such as: • English 7-12, English To Speakers Of Other Languages • Integrated Algebra scorers must have a Math license, such as: • Mathematics 7-12, SWD Mathematics, • Global/U.S. History scorers must have a Social Studies license, such as: • Social Studies 7-12, SWD Social Studies 7-12 • Only staff that are appointed pedagogues and have certifications will appear in the Scorer Selection Tool.

24. Principal Scorer Selection Tool Log-In • Go to http://intranet.nycboe.net/applications/oa/pssat/default.aspx via Internet Explorer • Enter your DOE username and password (PRINCIPAL)

25. Start Scorer Selection • User’s name and role (e.g. Principal) will be automatically populated in the upper-right. • Choose “Assign Teachers For Scoring” from the menu at the top of the page

26. Review Scorer Requirement • The summary table at the top of the page shows the scorer requirement for each exam • If a school has identified Scoring Content Leader(s), this contribution will be noted in the summary table, and count towards the scorer requirement. • Schools cannot select staff that have been identified to be Site Supervisors, Organizational Team Leads, or Organizational Team Members to score.

27. View the Scoring Calendar • Place the mouse cursor over the exam scoring dates to bring up a calendar

28. To Select Teachers For Scoring • For each subject, click on the “Assign Teachers” button to see a list of eligible scorers

29. Select Scorers • Choose scorers by clicking in the checkboxes to the left of their names • The number required is shown at the top of the page • Click on the “Assign Selected Teachers” button when finished selecting • Teachers not eligible to score an exam will be listed at the bottom of the screen; principals will not be able to assign these individuals as scorers

30. Scorer Required/Assigned Numbers Adjust As Scorers Are Chosen • Once selected, teachers appear in the Teachers Assigned window, where principals have the option of deleting them from the scoring assignment • The summary table at the top of the screen updates the number of scorers required/assigned as principals make their choices • Scoring Content Leaders will appear here as well and will be prepopulated

31. Select Scorers For Each Exam • Principals should make sure to select scorers for each exam where the required number is greater then zero • Principals can confirm they have met their school’s requirement by reviewing the summary table

32. Click Save And Submit When All Selections Are Made • Principals can click on the “Save And Return” button if they want to logout and return later to finalize/make changes • Principals can click on the “Save And Submit Final” button when they are ready to finalize their selections

33. Confirm Submission Is Final • Once principals click on the “Save And Submit Final” button a confirmation message will appear • Principals should click “OK” to confirm selections • Clicking “OK” will automatically send notification emails to the teachers that have been selected, alerting them to their scoring assignment

34. Scorer Selection Complete • Scorer selection is complete • An email confirmation that includes a summary of all scorer selections will be sent to the principal

35. Principal Email Confirmation • Here is an example of the assignment confirmation email

36. Adding/Replacing Scorers • Contact Luke Burrows at lburrows3@schools.nyc.gov for exceptions processing, including: • Adding new teachers • Replacing scorers after scorer selection deadline

37. School Preparation

38. Exam Pick-up & ReturnRegional Site • Participating Schools: Deluxe Courier will pick-up test materials for delivery to scoring site (see memo for pick-up times) • Host School: Consult with site supervisor for date/time the materials should be delivered to (and picked-up from) the scoring room(s). --------------------------------------------------------------------- • Campus: All school, in consultation with site supervisor will arrange for materials to be delivered and picked-up from scoring room(s).

39. Exam & Scoring TimelineNon-Electronic

40. Exams & Scoring Timeline • Scoring period for each exam will be 3 days or until scoring of all included exams is complete. • Scoring staff are expected to be at the scoring site by 8:30 a.m. each day and will be expected to stay for the full length of the teacher work day (i.e., 6 hours and 20 minutes plus extended time). • Campuses may alter their scoring schedule as long as all schools agree to the change.

41. Scorer Training • All scorers will be trained at the scoring site by the subject area Scoring Content Leaders. • Scorers are required to attend the ENTIRE training session and remain at the scoring site for the duration of the scoring. *** (clarify absence) • Each scorer must bring a copy of the scoring key and rating guide for the exam to be scored

42. May Training: Scoring Content Leaders • Training will follow NYSED policy outlined in the exam-specific Information Booklet. • Consistent with current requirements for scorer training regardless of exam or scoring model. • Anchor Papers will be used to focus on developing scorers understanding of the rubric and its consistent and accurate application.

43. Scoring Committees • Scoring committees will be established by the Scoring Content Leader, in consultation with the Site Supervisor, and structured according to SED policy. (single or multi-school swaps) • No teacher will grade an exam from a student from their own school • Scorers are expected to maintain security of test materials (sign a declaration)

44. Regents Scanning • All exams will be scanned in their entirety*** at the scoring site (absent students) • Any omissions or errors (student/teacher section) will be corrected or updated at the scoring site in accordance with NYCDOE policy. • Organization Team Leaders will assist Site Supervisors in utilizing RGTU function to correct inaccurate information from answer documents

45. ATS Reports • ATS:REDS - Status Report and Item Report, and the Item Distribution Report • ATS Data is sent each night to the transcript system and STARS. • If a students took the exam in their own school the record will be available in STARS (for report cards, exam reports) the next day. • Student who tool the exam in another school, the records will be available on the transcript only.

46. Finality of Examination Scores • When the scoring committee completes the scoring process, test scores must be considered final and will be entered onto the students’ permanent records via the scanning process. • Scoring committees, once they have rate an exam, are NOT permitted to re-score them, regardless of the final score. Specifically, exams receiving a final score of 60-64 or 50-54 may not be rescored. • Principals and other administrative staff in a school or district do NOT have the authority to set aside the scores arrived at by the teacher scoring committee and rescore student examination papers or to change any scores assigned through the procedures outlined in the scoring materials

47. Absent Students • Still waiting on clarification for whether we will pull ABSENT answer documents and scan in the Home or Regional/Campus Site. • FYI: Scan page 1 only and REDS Report has been updated so that page 2 will NOT indicated “Not Scanned”.

48. Absent Teachers (Scoring) • Call Principal to report absence • Principal sends replacement teacher (if teacher is available) • Replacement Teacher either trained as scorer or used for Organization

49. Implementation Timeline

50. Non-Electronic Distributed Scoring Process Overview Supervisors and school staff receive training. Deliver exams to scoring room/site Planning Administer Check-In Exams/Train Scorers Schools identify site management staff and work with Supervisor to plan for scoring. Schools plan for proctoring to occur without teachers selected for scoring. 2. Schools administer exams and package materials for pick-up/delivery according to instructions and schedule in administration memo. 3. Supervisors and organizational team members check-in test materials at scoring site while Scoring Content Leaders train scorers. Distribute booklets for scoring QA Process Score exams. • Review answer documents for omissions on teacher section after scoring of tests in a poly-bag is complete. Prepare answer documents for scanning. 4. Once test materials are received and contents are verified in the scoring room; the Scoring Content Leader (with support of Organizational Team Member) distributes booklets for scoring. 7a. Answer document with teacher section omissions should be returned to appropriate scoring committee for completion. Scanned answer documents returned to scoring room ATS Reports Package and Return Test Materials to Schools Scanning • Org. Team Lead runs REDS reports to identify data capture issues and teacher response omissions or multiples. Supervisor makes RGTU updates as needed. 8. Materials packaged for return to schools. Courier pick-up scheduled by Supervisor as needed. • Organizational Team Lead scans answers documents at Campus or Regional Scoring Site.