Issues Related to Judging the Alignment of Curriculum Standards and Assessments

1. Norman L. Webb Wisconsin Center for Education Research University of Wisconsin-Madison Annual Meeting of the American Educational Research Association Montreal, April 11, 2005 Issues Related to Judging the Alignment of Curriculum Standards and Assessments This work was supported by a subgrant from the U. S. Department of Education (S368A030011) to the State of Oklahoma and a grant from the National Science Foundation, (EHR 0233445) to the University of Wisconsin—Madison. Any opinions, findings, or conclusions are those of the author and do not necessarily reflect the view of the supporting agencies.

2. Webb Alignment Process • Identify Standards and Assessments • Select 6-8 Reviewers (Content Experts) • Train Reviewers on DOK Levels • Part I: Code DOK Levels of the Standards/Objectives • Part II: Code DOK Levels and Corresponding Objectives of Assessment Items

3. Reports Standards DOK analysis Degree of alignment by standard Source of Challenge Reviewers’ Notes General Comments Made by Reviewers Reliability Among Reviewers Overall Finding of the Degree of Alignment

4. Data Tables • Acceptable levels on four alignment criteria • Source of challenge • DOK by item and intraclass correlation • Notes • DOK level and objective code by item • Objectives by item • Items for each objective • Number of reviewers coding an item by objective

5. Alignment Criteria • Categorical Concurrence Number of items per standards • Depth-of-Knowledge Consistency Percent below, at, and above • Range-of-Knowledge Correspondence Percent of objectives with at least one item • Balance of Representation Index 0 to 100

6. What are appropriate acceptable levels for each criterion? • Number of items for one standard • Distribution of items by complexity • Number of standard-objectives with at least one item • Distribution of items among objectives under a standard

7. Alignment Levels Using the Four Criteria

8. State ACategorical Concurrence for Grade 3 Science (N=55 items)

9. State BDepth-of-Knowledge Consistency High School Mathematics (N=51 items)

11. State BRange of Knowledge CorrespondenceHigh School Mathematics (N=51 items)

13. State BBalance of RepresentationHigh School Language Arts (3 of 12 standards)(N=116 items)

15. What considerations should be given to different item types? • How to consider a multiple-point assessment item? • What is the trade off between multiple-choice items and open-ended items?

16. What are issues related to vertical alignment? • Appropriate progression of complexity across grades • Appropriate progression of content across grades

17. State A Mathematics DOK Levels for Objectives by Grade

18. State A Reading Language Arts DOK Levels for Objectives by Grade

19. Vertical Alignment Questions • What level of concurrence is there between objectives for the two grades? • To what extent do comparable objectives increase in depth from one grade to the next? • To what extent does the range of content increase from one grade to the next? • How does the balance of representation change from one grade to the next?

20. Type of Vertical Relationships • Broader • Deeper • Prerequisite • New • Identical