Designing High Quality Assessments that are Affordable: Conclusions and Recommendations

1. Designing High Quality Assessments that are Affordable: Conclusions and Recommendations John F. Olson CCSSO National Conference on Student Assessment Detroit, MI June 21, 2010

2. Reference • Conclusions and recommendations from the report: Topol, B., Olson, J., & Roeber, E. (2010). The Cost of New Higher Quality Assessments: A Comprehensive Analysis of the Potential Costs for Future State Assessments. Stanford, CA: Stanford University, Stanford Center for Opportunity Policy in Education. Assessment Solutions Group www.assessmentgroup.org

3. Cost Study Conclusions • The development cost of a new HQA is relatively inexpensive relative to the total cost of the assessment • A key factor in determining whether states can adopt and sustain new improved assessments is ongoing administration costs • In order to reduce costs, states should participate in an assessment consortium to share the overhead associated with development, administration, and management of assessments

4. Cost Study Conclusions • Larger consortia are somewhat more cost-effective • The majority of cost savings relative to the single state case are seen at a 10-state consortium size • States should strongly consider being part of a large consortium where certain costs can be shared across many states, such as for item development and project management

5. Cost Study Conclusions • Implementing a HQA system with performance items is affordable, with teacher scoring of performance items at a price comparable to today’s assessments, when procured by a consortium of states • In order to implement and afford an HQA system that includes a variety of performance items, it is essential to have teachers involved in the scoring process • Note that the cost impact of increases in the time to score performance items is very significant

6. Cost Study Conclusions • The use of online technology (i.e., CBTs, CATs, or online assessments) should be encouraged • It has the potential to reduce assessment cost and improve quality • The procurement of PCs to improve the student- to-PC ratio should be encouraged at all levels of the educational system

7. Recommendations • Developing and implementing an HQA will likely cost more than most current state assessments. It can be affordable for states if they look carefully at • the design of the summative assessment component • finding a balance in the number of CR items, PEs, and PTs used • consider various cost-reduction strategies

8. Recommendations States should consider using a scoring model that has teachers scoring the performance items as part of their PD via a distributed scoring system. Having all scoring done by the testing vendor is likely to be cost-prohibitive for most states. Ultimately, automated scoring of essay responses should lower scoring costs for these items significantly and further enable the implementation of HQAs at reasonable prices. AI scoring should be encouraged and its progress monitored.

9. Recommendations • Sustainability • A key factor in the sustainability of new improved assessments and whether or not states can adopt and use them will be the ongoing administration costs that need to be managed. • State consortia interested in implementing a HQA should make sure they can afford the ongoing administration costs of the assessment before they embark on developing it

10. Recommendations It is recommended that state consortia go about the process of designing a new assessment in a thoughtful manner, then use a comprehensive costing model to analyze and determine the price in advance of any new assessment system they would like to implement.

11. Questions • John Olson – jolson@assessmentgroup.org • Ed Roeber – eroeber@assessmentgroup.org • Barry Topol – btopol@assessmentgroup.org Assessment Solutions Group www.assessmentgroup.org