190 likes | 390 Vues
CCSSO NSA Conference / Elliott 2009. 2. Projects
E N D
1. Designing More Accessible Achievement Tests for All Students Stephen N. Elliott
Learning Sciences Institute
and
Department of Special Education
Vanderbilt University
CCSSO 2009
National Conference on Student Assessment
2. CCSSO NSA Conference / Elliott 2009 2 Projects & Partners CAAVES: Consortium for Alternate Assessment
Validity and Experimental Studies
USDE funded; 2006-2009
Partners: AZ, HI, ID, & IN + Vanderbilt Measurement Group + Discovery Education Assessment
CMAADI: Consortium for Modified Alternate Assessment Development and Implementation
USDE funded; 2007-2010
Arizona Dept. of Education
Indiana Dept. of Education
Visit Websites for Resources Discussed Today
http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home.xml
http://peabody.vanderbilt.edu/LSI_Projects/C-MAADI_Project_Home.xml
http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home/TAMI_Project.xml
3. CCSSO NSA Conference / Elliott 2009 3 Inclusive Testing & Better Results NCLB Act 2007
Amendments on AA-MAS
Students with disabilities who exhibit persistent academic difficulties.
Inattention
Organizational difficulties
Poor reading fluency
History of below proficient test performances
Low self-efficacy with testing
4. CCSSO NSA Conference / Elliott 2009 4 Key Terms Access: the opportunity for test-takers to demonstrate proficiency on the target construct of a test or a test item. In essence, complete access is manifest when a test-taker is able show the degree to which he/she knows the tested content. Access, therefore, must be understood as an interaction between individual test-taker characteristics and features of the test itself.
Accommodation: widely recognized in state testing guidelines as individualized changes to the setting, scheduling, presentation format, or response format of an assessment.
Modification: alterations or adjustments of test items to facilitate access for virtually all test takers. Appropriate modifications .
Remove extraneous material,
Maintain the same depth of knowledge (DOK),
Do NOT change the grade-level construct being measured, and
Increase the validity of the inference from the test score.
5. CCSSO NSA Conference / Elliott 2009 5 Anatomy of an Item
6. CCSSO NSA Conference / Elliott 2009 6 CAAVES Procedures We completed the following..
Modified a common set of existing reading and math items to create items designed to be more accessible and still measure the same grade-level content as the original items.
Conducted a cognitive lab study with a small sample of students with and without disabilities to gain their insights into which item modifications are preferred and most likely to improve test access for students whose disability involves reading difficulties.
Conducted a cross-state experimental study to compare the effects of tests with and without modified items on students test performances and test score comparability.
Conducted post-assessment survey of all students concerning their perceptions of item types and cognitive ease.
7. CCSSO NSA Conference / Elliott 2009 7 Guiding Theories & Research Evidenced-based model of test score validity,
Universal design principles,
Cognitive Load Theory for designing instructional materials, and
Item writing research and practices.
8. CCSSO NSA Conference / Elliott 2009 8 Examples of Theory-Guided & Data-Based Item Modifications
9. CCSSO NSA Conference / Elliott 2009 9 Example: Original to Modified Item
10. CCSSO NSA Conference / Elliott 2009 10 Overview of Results
11. Modifications Benefited all Groups CCSSO NSA Conference / Elliott 2009 11
12. CCSSO NSA Conference / Elliott 2009 12 Item Summary Reports: An Example
13. Estimating Impact Will AA-MASs result in more students with disabilities being considered proficient for AYP?
We have explored the impact of some hypothetical cut scores for the CAAVES Reading and Math Scores.
An actual Standard Setting is needed. CCSSO NSA Conference / Elliott 2009 13
14. CCSSO NSA Conference / Elliott 2009 14 Method for Documenting OTL
15. CCSSO NSA Conference / Elliott 2009 15 Cognitive Labs
16. CCSSO NSA Conference / Elliott 2009 Post-Assessment Focus Groups 16
17. CCSSO NSA Conference / Elliott 2009 17 Evolving Modification Paradigm Step 1. Evaluate original item accessibility.
Step 2. Reduce sources of construct-irrelevant variance in items.
Step 3. Document changes to items.
Step 4. Pilot test with student cognitive labs & post-assessment focus groups.
Step 5. Field test with large sample of students.
Step 6. Conduct psychometric & related analyses.
18. Characteristics of Appropriate Modifications Design Elements
Simply words and text structure
Delete extraneous words
Improve visuals and locate within item
Use bold text for important words
Eliminate least plausible distractor so there are 3 answer choices
Desired Outcomes
Increase accessibility
Decrease item difficulty
Increase item discrimination
Increase reliability estimates
Reduce readability level w/i grade range
Maintain alignment w/ content stds.
Maintain DOK for all items
Increase validity of test scores
Reduce need for accommodations
Increase reading fluency
Improve students perceptions of tests & motivation to engage in testing
CCSSO NSA Conference / Elliott 2009 18
19. CCSSO NSA Conference / Elliott 2009 19 Colleagues Presentations Quantifying and Improving Item & Test Accessibility Peter Beddow, Vanderbilt
Using Students Insights to Influence Item & Test Design Andrew Roach, Georgia State
Plausible Attractors & Item Psychometrics- Michael Rodriguez, University of Minnesota
20. CCSSO NSA Conference / Elliott 2009 20 Thanks! Thank you very much for your time and joining us for this session.
Please provide follow-up questions and suggestions in writing to:
Steve.elliott@vanderbilt.edu
http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home.xml
http://peabody.vanderbilt.edu/LSI_Projects/C-MAADI_Project_Home.xml