New England Common Assessment Program (NECAP)

1. New EnglandCommon Assessment Program(NECAP) Setting Performance Standards

2. Purpose • Provide data to establish the following cut scores for Math, Grades 3-8: • Proficient/Proficient with Distinction • Partially Proficient/Proficient • Substantially Below Proficient/Partially Proficient

3. What is Standard Setting? • Set of activities that result in the determination of threshold or cut scores on an assessment • We are trying to answer the question: • How much is enough?

4. What is Standard Setting • Data collection phase • Policy/Decision making phase

5. Many Standard Setting Methods • Angoff • Body of Work • Bookmark

6. Choice of Method is Based on Many Factors • Prior usage/history • Recommendation/requirement by some policy making authority • Type of assessment

7. Choice of Method is Based on Many Factors • Weighing all these factors, it was determined that the Bookmark Method would be used for NECAP Math

8. The Bookmark Procedure • Well established procedure that has been successfully used on many assessments • Has produced defensible results • Appropriate for assessments that consist primarily of multiple-choice items but also include some constructed-response items

9. Standard Setting vs. Standards Validation • Standard setting • Generally three rounds: • Round 1: individual ratings • Round 2: table group discussion of Round 1 results, followed by second round ratings • Round 3: whole room discussion of Round 2 results, followed by final ratings

10. Standard Setting vs. Standards Validation • Standards validation • Starting cut points are provided to panelists at the beginning of the process • Two rounds: • Round 1: table group discussion of starting cut points, followed by first round ratings • Round 2: whole group discussion of Round 1 results, followed by second round ratings • The process you will be following will be more of a standards validation process

11. Details for Standards Validation using the Bookmark Procedure

12. What is the bookmark procedure? • A standard setting procedure that uses a book of items (ordered from easiest to hardest) • Panelists place bookmarks in that book of items

13. What is the bookmark procedure?

14. What is the bookmark procedure? • For purposes of NECAP standard setting, you will be provided with starting cut points and will either validate those cuts, or recommend modifications to them

15. Starting Cut Points • Starting cuts were determined using teacher judgment data: • At the time of testing, teachers were asked to categorize each student into an achievement level category, based on classroom performance (not test performance) • Using these categorizations as well as the students’ test scores, Measured Progress staff calculated starting cut points

16. Starting Cut Points • Analyses of teacher judgment data indicate: • Good participation rates • Teachers appear to have been conscientious in completing the task • In other words, available evidence supports the validity of the starting cuts; therefore…

17. Starting Cut Points • You are free to recommend changing the starting cuts, but we expect that changes will be relatively minor • You should have justification for any changes you recommend • rationale for changes should be based on the Achievement Level Descriptions and the test items

18. Starting Cut Points • Note that, in some cases, the lowest starting cut appears very early in the Ordered Item Booklet • What this means is that the test was difficult for these students: • for several of the math tests, borderline Partially Proficient students (as identified by the teacher judgment data) have less than a two-thirds chance of answering even the easiest items correctly

19. Starting Cut Points • What you need to know is that the ordered item cut point for a given cut does not equal the raw score a student must obtain to be categorized into the higher achievement level • For example, the Substantially BelowProficient/ Partially Proficient starting cut for grade 7 math is between ordered items 3 and 4; however, a student must obtain at least 20 points on the test in order to be classified into the Partially Proficient level

20. How to Place a Bookmark • A few concepts you will need to know: • The starting cut points • The achievement level descriptions • ‘Borderline’ students • What knowledge, skills, and abilities (KSAs) are needed to answer each question

21. How to Place a Bookmark • Start at the beginning of the ordered item book • Evaluate whether at least 2 out of 3 students demonstrating skills at the ‘borderline’ of Partially Proficient would correctly answer item 1 • Moving through the book, make this evaluation of each item • The bookmark should go where you no longer think 2 out of 3 Partially Proficient ‘borderline’ students would correctly answer the question. • You may decide that the starting cuts are in the right place, or you might recommend moving them.

22. How to Place a Bookmark

23. How to Place a Bookmark • In the example, the bookmark would go between items 8 and 9 • However, it won’t be that easy; there will be gray areas • You will have the opportunity to discuss your bookmark placements and change them if desired • Place one bookmark for each cut score

24. How to Place a Bookmark • To place your bookmarks you will need to be familiar with the achievement level descriptions and the assessment items

25. How to Place a Bookmark • Don’t worry, we have procedures, materials and staff to assist you in this process.

26. Any questions about the Bookmark Procedure?

27. What Next? • After this session, you will break into grade-level groups, where you will: • take the assessment to familiarize yourself with the test items • discuss the Achievement Level Descriptions and develop definitions of “borderline” Partially Proficient, Proficient, and Proficient with Distinction students

28. What Next? • You will: • complete the Item Map, which is a document that will help you with the bookmark placement process • discuss the starting cut points in table groups and do the first round of bookmark placement • discuss the first round bookmark placements as a whole room then do the second round of bookmark placement

29. What Next? • After the second round of bookmark placement, you will have an opportunity to provide feedback about the Achievement Level Descriptions • As the final step, we will ask you to complete an evaluation of the standard setting process

30. Good Luck!