Cultivating Success in Fertile Soil

1. Cultivating Success in Fertile Soil Raising Expectations and Outcomes for Students with Disabilities through Assessment and Accountability Systems Martha Thurlow, Rachel Quenemoen, Sandy Thompson, John Bielinski, and Jane Minnema National Center on Educational Outcomes (NCEO) University of Minnesota

2. Clinic Agenda Part I: Separate the Wheat from the Chaff Part II: Plow New Ground Part III: Apply the Best Fertilizers Part IV: Harvest a Rich Return

3. NCEO Resources Visit: education.umn.edu/nceo or Search for NCEO

4. Part I.Separate the Wheat from the Chaff

5. In 1989, the Education Summit set an agenda for education reform that called for – • Higher Expectations • Rigorous Educational Standards • Assessments of Progress toward Standards NCEO was funded in 1990 to look at the educational outcomes of students with disabilities

6. IDEA 97 New Assessment Provisions • Participation of students with disabilities in state and district assessments • Alternate assessments for those students unable to participate in general state or district assessments • Inclusion of disaggregated participation and performance data of students with disabilities in public reports whenever data are provided for all students

7. Title I Includes ALL Students • All “eligible” students can receive Title I services, regardless of other services provided • Title I evaluation is based on statewide assessment, which is to include all students • States must report statewide data, with disaggregations for students with disabilities, LEP students, and other groups • States must define adequate yearly progress (AYP) and evaluate schools against AYP

8. Participation of ALL students in state and district assessments Reported information about the performance of special populations, relative to other students Measurement against consistent goals and standards for ALL students (to the maximum extent appropriate) Common Themes Include:

9. Standards-Based Reform Context High Standards All Students --- Everything else is negotiable --- schedules, place, time, structure, curriculum, methods of assessment, instructional methods . . . AcCOUNTability

10. Accountability System Components Goals (Content Standards) Indicators of Success (Performance Standards) Measures of Performance (Assessment System) Reporting Consequences

11. Clarification of Assessments Eligibility Assessments Classroom Tests Large-ScaleAssessments Districtwide Statewide National

12. VARYINGContext of State Assessments • Some measure basics, others high standards • Some are high stakes for students, some high stakes for systems, some are both • Grades administered vary, as do content areas (all have Reading and Math) • Some are norm-referenced, some are criterion referenced, and some are both • Varying approaches to accommodations and alternate assessments

13. Principle 1. All students with disabilities are included in the assessment system. Principle 2. Decisions about how students with disabilities participate in the assessment system are the result of clearly articulated decision-making processes. Principles of Inclusive Assessment and Accountability Systems

14. Principle 3. All students with disabilities are included when student scores are publicly reported, in the same frequency and format as all other students. Principle 4. The assessment performance of students with disabilities has the same impact of the final accountability index as the performance of other students. Principles of Inclusive Assessment and Accountability Systems

15. Principle 5. There is improvement of both the assessment system and the accountability system over time… Principle 6. Every policy and practice reflects the belief that all students must be included in state and district assessment and accountability systems. Principles of Inclusive Assessment and Accountability Systems

16. Part II.Plow New Ground

17. Student Accountability – students are held responsible and consequences are assigned to them (e.g., must pass test to graduate or move to next grade) 20 States System Accountability – educators, schools, or districts are held responsible and consequences are assigned to them (e.g., schools rated according to test scores, teachers receive rewards for student performance) 38 States High Stakes Testing

18. Norm-Referenced Test (NRT):A test that allows its users to make score interpretations of a test taker’s performance in relationship to the performance of other people in a specified reference population. Criterion-Referenced Test (CRT):A test that allows its users to make score interpretations in relation to a functional performance level, typically through “cut score” definitions. *From OCR Resource Guide, December 2000 Definitions*

19. Norm-Referenced Tests Criterion-Referenced Tests Proficient Levels 1 2 3 4

20. True Score True Score True Score Reliability Reliability is an index of the precision with which an examinee’s score is estimated with a particular set of items. Not reliable Or valid Reliable, But not valid Reliable & Valid

21. High Performance Math Ability True Score Math Ability True Score Poor Performance Validity The degree to which test scores accurately reflect the types of inferences made. Non- Accommodated Accommodated

22. Test score utility for school improvement resulting in improved outcomes for students with disabilities, depends on alignment with standards Curriculum and Instruction Improvement CONTENT STANDARDS Standardized Test

23. IDEA and IASA require states to report the number of students with disabilities participating in the regular assessment, and the number participating in the alternate assessment Two important numbers Numerator DENOMINATOR When the Numbers Are Not Enough

24. How does the performance of students receiving special ed. services compare to the performance of all students? Number receiving SpEd Number of students Math Proficiency 300 70 Partially Proficient 250 20 Proficient 50 10 Advanced • Now, look how easy it is to compare groups when • percentages are reported in each category

25. Which School did better, A or B? Now decide Who is counted affects interpretation of results

26. Louisiana Data (NRT) IEP IEP 504 504 Grade In Out In Out 3 5.7 94.3 12.8 87.2 5 5.8 94.2 15.4 84.6 6 6.1 93.9 19.2 81.8 9 11.2 88.8 22.6 77.4

27. Assessment Participation Rates of Students with Disabilities Rates Vary Tremendously Across States The lowest rate is 15% The highest rate is 100%

28. 14 states disaggregated data on the participation of students with disabilities 17 states disaggregated data on the performance of students with disabilities

29. Current Research and Technical Challenges • Accommodations and Modifications • Alternate Assessment • Out-of-Level Testing • Other “GAP” Assessments

30. Accommodations and ModificationsFertilizer Guaranteedto BoostYield

31. Is on the rise About 50% of the LD students accommodated Most common accommodations are: small group administration read-aloud extended time Accumulating evidence from experimental studies indicates that some accommodations boost performance Accommodation Use

32. An accommodation is a change in testing materials or procedures that: The Metaphor • - increases access to the test for students with disabilities. • - results in measurement of student abilities not disabilities • - levels-the-playing field

33. An accommodation represents an alteration to standard test conditions that neutralizes extraneous sources of difficulty that result from an interaction between standard administration and the student’s disability while preserving the measurement goals of the test. Psychometric Definition

34. AS = OS - D - S - D*S = 100 AS = Actual Score OS = Optimal score D = Disability S = standard conditions D*S = Interaction Example 85 - 0 - 5 - 10

35. An accommodation should boost performance for students with disabilities but not for students without disabilities necessary but NOT sufficient Since 1995, there were 38 empirical studies of test score boost & 6 studies examining construct validity Single subject design Test individuals under many conditions Use very short (usually single item) performance tests look for accommodations that result in large boost do not account for measurement error in the comparison of performance Test Score Boost

36. The construct the test was designed to measure should remain unchanged by the presence of an accommodation Requires construct validation studies Test score boost associations with other measures invariance of the item characteristics Difficult to do with small samples Extant data well suited for construct validity study large samples real-world less expensive/time-consuming Preserving Measurement Goals

37. Compare item characteristics across groups Differential Item Functioning Analysis Structural Equation Modeling Four possibilities if an effect is found Accommodation not appropriately administered Accommodation not administered to appropriate population Accommodation doesn’t work Some combination of these Construct-validation

38. DIF analysis across four groups Non-disabled, non-accommodated Low performing, non-disabled, non-accommodated Reading disabled, no read-aloud accommodation Reading disabled, with read-aloud accommodation Results # DIF Items MATH READING Performance-matched: 0 1 No read-aloud: 1 10 6 Read-aloud: 19 Research Findings

39. Alternate Assessment – for those students unable to participate in general state assessments New part of state and district assessment systems - Did not exist in most places before IDEA - Lots of activity in the past year!

40. Alternate Assessments are intended to provide the missing piece that makes it possible to include ALL students with disabilities Many states have found the need for more than one missing piece

41. Focus of Alternate Assessments is Evolving Number of States ’99 ’00 ‘01 State Standards/Expanded 19 28 19 Skills Linked to Standards -- 3 14 Standards + Additional Skills 1 7 9 Skills Only 16 9 4 Other or Uncertain 24 3 3

42. As Focus Evolves, So Does Assessment Decision-Making Process

43. A student with a disability… Who requires substantial modifications to instructional level and learning standards in a content area, and Who requires intensive, individualized instruction in order to acquire and generalize knowledge, and Who is unable to demonstrate achievement of learning standards on a paper and pencil test, even with accommodations Example from MA training -Who should take MCAS-Alt?

44. Variations in Approach • Body of Evidence/ Portfolio 24 states • Checklist 9 states • IEP team determines strategy 4 states • IEP analysis 3 states • Combination of strategies 4 states • Specific performance assessment 4 states • No decision 2 states

45. Alternate assessment developers in nearly all states included: State special education and assessment personnel Local administrators, special and general educators, assessment coordinators, and related service providers Parents and advocates A few states included students and adults with disabilities Stakeholders Bring Different Values and Beliefs to the Table

46. Variations in StudentPerformance Measures • Skill/competence 40 states • Independence 32 states • Progress 24 states • Ability to generalize 18 states • Other 7 states

47. Variations in System Performance Measures • Variety of Settings 21 states • Staff Support 20 states • Appropriateness (e.g, age, challenge) 20 states • Gen. Ed. Participation 12 states • Parent Satisfaction 9 states • No system measures 8 states

48. Performance - demonstration of skill while attempting a given task. Each entry is scored Support - assistance provided to a student during performance of tasks. Each entry is scored Appropriateness - The degree to which the tasks 1) reflect the chronological age of a student, 2) provide a challenge for the student, and 3) are representative of real-world activities that promote increased independence. Each entry is scored Settings - settings or environments in which tasks are administered/performed for math entries; and for ELA entries. Scored once for each content area across entries Example: Arkansas Scoring DomainDefinitions

49. Alternate Assessment Performance Descriptors About one-third of states are using the same performancedescriptors for their alternate and general assessments Slightly more states are using different performance descriptors

50. Absolute vs. Relative Performance Standards • Some states emphasize measurement against absolutestandards over the relative emphasis on individualized needs and abilities. In these states, most students participating in the alternate assessment are performing at the “0” or “1” levels. • Other states have a separate definition of performance levels for the alternate assessment that emphasizes student-by-student growth of skill toward therelative standard based on the high expectation bridge, not in comparison to absolute standards. • With this approach, student results can be at any of the proficiency levels.