Developing Monitoring and Pre-Scoring Plans for Alternate/Alternative Assessments

1. Developing Monitoring and Pre-Scoring Plans for Alternate/Alternative Assessments Virginia Department of Education Division of Student Assessment and School Improvement August 2011

2. Purpose of this Presentation • To assist school divisions in developing plans to monitor and pre-score Collections of Evidence (COE) for the Virginia Alternate Assessment Program (VAAP) and the Virginia Grade Level Alternative (VGLA) and Course Work Compilations (CWC) for the Virginia Substitute Evaluation Program (VSEP).

3. Developing Monitoring and Pre-Scoring Plans for Alternate/Alternative Assessments • What are monitoring and pre-scoring? • Why should divisions develop monitoring and pre-scoring plans? • Who should be involved in developing the monitoring and pre-scoring plans? • What components should the monitoring plan include? • What components should the pre-scoring plan include?

4. What are monitoring and pre-scoring?

5. Monitoring A periodic and systematic review of COEs and CWCs while they are in the development process with opportunities for feedback and intervention.

6. What Monitoring is Not • Checking in with the teacher • Flipping through COEs or CWCs • Reviewing COEs and CWCs without providing feedback

7. Pre-Scoring A detailed review of COEs and CWCs to address technical errors (i.e., grading, SEI tags) prior to scoring with opportunities for feedback and corrections.

8. What Pre-Scoring is Not • Assigning a rubric score to the evidence • Altering student evidence or student responses • Judging the instructional quality of the evidence

9. Why should divisions develop monitoring and pre-scoring plans?

10. Monitoring • Keeps division staff abreast of the status of each COE and/or CWC • Ensures that teachers are providing instruction based on the Standards of Learning (SOL) • Ensures that instruction is on target with the division’s pacing guide

11. Monitoring • Ensures that technical issues (i.e., grading, SEI tags) are appropriately addressed • Provides the opportunity to address issues with ample time for correction and/or intervention

12. Pre-Scoring • Provides a final opportunity to correct technical errors and omissions that may result in the loss of credit to students prior scoring • Results in the creation of better COEs and CWCs

13. Who should be involved in developing the monitoring and pre-scoring plans?

14. Who should be involved in developing the plans? Depends upon: • Number of COEs and CWCs • Available central office resources • Available building level resources

15. Central Office Staff Division Director of Testing (DDOT) or designee Special Education Director or designee Director of Instruction or designee Title III Coordinator or designee Building Level Staff Principal or designee School Test Coordinator Special Education Chairperson Who should be involved in developing the plans?

16. What components should the monitoring plan include?

17. Critical Planning Information Develop a Master Participation List for each alternate/alternative assessment program that includes... • Name of student • Grade level of student • School • Content area being assessed by alternate/alternative assessment • Teacher(s) responsible for instruction in the content area assessed

18. Monitoring Plan Decisions • Who will monitor? • How will monitors be trained? • When will monitoring occur? • Who will receive feedback? • How will feedback be given? • What options are available if corrections are needed?

19. Remember! Your monitors will provide training and/or technical guidance to teachers submitting COEs and are not allowed to score these COEs.

20. Who will monitor? Staff with content knowledge and alternate/alternative assessment knowledge • Hired consultants • Special education or testing specialists • Instructional specialists • LEP specialists • School test coordinators • Title I mathematics and reading specialists • Experienced teachers • Building administrators

21. How will monitors be trained? • Provide access to implementation manuals and other resources (i.e. testing memos, VGLA worksheets, SEI tags, Curriculum Frameworks) • Conduct local training for monitors to include review and discussion of scoring rules, SOL or ASOL information, issues generated from the previous year’s scoring, etc.

22. When will monitoring occur? • Monthly • End of each grading period • At the same time as benchmark testing • Every other month • Other

23. Who will receive feedback? • Teachers • Principals • Alternate/Alternative Assessment Leaders • Central Office Staff • Trainers/Technical Assistance Providers

24. How will feedback be given? • Post It notes on COEs and CWCs • Locally created feedback forms given to teachers, principals, and/or others • Meetings with teachers, principals, and/or others about monitoring results • Other

25. What options are available if corrections are needed? • Individualized consultation and support • Peer or small group training sessions • Division-level professional development • Access to state and local SOL and/or ASOL assessment resources • Other

26. A Sample Monitoring Planfor 500 COEs • School-based Review Teams will be created in each school. • DDOT and Special Education Director will train School-based Review Teams using implementation manuals, VDOE power points, division pacing charts, and other resources. • Teachers submit COEs or CWCs to the School-based Review Team at the end of each nine-weeks.

27. A Sample Monitoring Plan • School-based Review Teams will report their finding on each COE or CWC using a locally created form. A copy of the form will be distributed to the teachers, principals and central office staff. • Central office will dispatch instructional specialist and assessment specialist to schools in need of additional support and training.

28. A Sample Monitoring Plan 2 School-based Review Teams check COEs every 9 weeks according to Pacing Chart and report to principals and central office staff 1 Select And Train School-based Review Teams 3 Instructional and assessment specialists dispatched to schools based on 9 Week Reports

29. A Sample School-Based Review Team 9-Week Monitoring Form – Grade 4 Reading

30. A Sample Teacher Review Form VGLA 9 Week Review Sheet School-Based Teams Teacher: ____________________ Student: ___________________________ Grade Level:______ Content Area:_________________________________________ REVIEW for: ____ 1st Nine Weeks ____ 3rd Nine Weeks ____ 2nd Nine Weeks ____ 4th Nine Weeks Reviewed By:____________________________ Date:________ Collection Status: Address the following questions: (1) Is there evidence for all of the standards for the nine weeks according to the Pacing Chart? (2) Does the work submitted align with the standards satisfactorily? (2) Does the evidence demonstrate student mastery? (3) Has the student work been graded accurately? (4) Other? Recommendation(s): Be as specific as possible Follow-up review needed for implementation of recommendations: ___Yes ____No Follow –up Review Date: ________________

31. Pre-Scoring Decisions • Who will pre-score? • How will pre-scorers be trained? • When will pre-scoring occur in relation to submission date? • Who will receive feedback? • How will feedback be given? • What options are available if corrections are needed?

32. Who will pre-score? • Persons selected for pre-scoring VAAP COEs can not serve as scorers. • Pre-scorers should be knowledgeable about all scoring rules and the SOL and/or ASOL content. • Pre-scorers should be detail oriented.

33. Remember! Your pre-scorers will provide training and/or technical guidance to teachers submitting VAAP COEs and are not allowed to score these COEs.

34. How will pre-scorers be trained? Training should include a list of specific issues to look for in each COE or CWC • grading • correctly completed SEI tags • detailed anecdotal records (VAAP only) • captioned photographs (VAAP only)

35. Other helpful questions to ask in Pre-scoring • Is the evidence organized according to the scoring worksheet? • Are all required state and local forms included and completed?

36. When will pre-scoring occur in relation to local due date? Select a time to pre-score: • that will give ample time for COEs or CWCs to be complete or near completion. • that will give opportunity to return COEs or CWCs to teachers for corrections and returned by local due date.

37. Who will receive feedback? • Teacher(s) submitting COE or CWC. • Building level principals • Others

38. How will feedback be given? • Post-it Notes on COEs and CWCs • Local created forms given to teachers, principals, and/or others • Meetings with teachers, principals, and/or others • Other

39. What options are available if corrections are needed? • Return COEs to teachers for corrections. • Assign staff to assist teacher in corrections such as grading evidence, organizing evidence, completing SEI tags. • Options may be limited or non-existent due to time constraints • Other

40. Sample Pre-scoring Plan • Pre-scoring will be conducted by teachers, building level staff and central office staff in three phases • Teacher Level • Building Level • Central Office Level • Scoring rules will be reviewed with all persons involved in pre-scoring. Checklists will be provided.

41. PHASE 1 – Teacher Review Conducted April 12-16 Submitting Teacher reviews COEs or CWCs using Teacher Checklists in Implementation Manuals PHASE 2 – School-based Review Conducted April 19 – 26 COEs or CWCs using notes and local forms Building Administrator reviews COEs or CWCs using Administrator Checklist in Implementation Manuals Sample Pre-Scoring Plan

42. Phase 3 – Central Office Pre-scoring Team Review Conducted April 27 – May 7 Central Office team reviews COEs and CWC against scoring rules. Findings noted on Post-It notes placed on evidence. Central Office teams return COEs or CWCs to schools to correct errors and omissions. COEs or CWCs are corrected by teachers. Teachers sign affidavits COEs or CWCs submitted to principals or designee. Principal or designee signs affidavits. COEs or CWC submitted for scoring. Sample Pre-Scoring Plan

43. Monitoring & Pre-Scoring Results Reduces or Eliminates COEs and CWCs with: • Missing Evidence or SEI tags • Unacceptable Evidence (textbooks, homework) • Inaccurate or Ungraded Evidence

44. Monitoring & Pre-Scoring Results Eliminates Surprises: • COEs or CWCs not done for students educated outside of the division • Incomplete COEs or CWCs in the division • COEs or CWCs not driven by IEP, 504 or LEP Plans

45. Questions Virginia Department of Education Office of Test Administration, Scoring, and Reporting 804 225-2102 Office of Special Education Instructional Services 804-225-2932