Presented to: CCSSO 2012 National Conference on Student Assessment June 27, 2012

1. Quality Assurance in a Chaotic World An External Perspective Presented to: CCSSO 2012 National Conference on Student Assessment June 27, 2012

2. The External Perspective • HumRRO serves as • National Assessment of Educational Progress (NAEP) Quality Assurance contractor • Independent evaluator of California High School Exit Examination (CAHSEE) • Psychometric second-checker for several states • In these roles, we stand outside operational assessments and provide feedback, warnings, guidance to federal and state clients • This presentation focuses on NAEP.

3. NAEP Quality Assurance • Several years ago, the National Center for Education Statistics (NCES) contracted with a single vendor for NAEP. This vendor, in turn, managed subcontracts. • NCES now contracts directly with several agencies for: • Item Development • Web/Technology Development, Operations and Maintenance • Materials Distribution, Processing and Scoring • Sampling and Data Collection • Design, Analysis, and Reporting • NAEP State Service Center • Alliance • Quality Assurance

4. NAEP QA: Definitions • Quality Control (QC): integrated steps embedded within a process that check, verify, and document that data, analysis, and results are accurate. • Quality Assurance (QA): an external set of steps that verify the QC process is effective, complete, and efficient.

5. Categories of QA Activities • Participation in planning and monitoring meetings • Observation of operational meetings (NAGB, DAC, NVS) • Review of contractor QC plans • Observation of operational processes • Special studies

6. Participation in planning and monitoring meetings • Annual Design Summit • Annual Quality Summit • Periodic contractor meetings • Inter-contractor quality teams

7. Observation of operational meetings • National Assessment Governing Board (NAGB) • Design and Analysis Committee (DAC) • NAEP Validity Studies Panel (NVS)

8. Review of contractor QC plans • Annual advance review of QC plans • Critical components include: • Completeness • Ongoing self-monitoring • Recording relevant data • Monitoring of trends • Continuous process improvement • Feedback loops • Importance of closing the loop with operational contractors

9. Observation of operational processes • “Site visits” in four phases: • Item development • Test administration • Processing and scoring • Analysis and reporting • Brief site visit reports after each event • Includes recommendations & commendations that are logged centrally • Annual summary report for each phase • Reflects upon overall patterns

10. Special studies • Quality Assurance Technical Panel (QATP) • Small number of studies with a quality focus, such as: • Use of trend papers in scoring • Investigation of “interesting” analytic results in advance of public release • Investigation of alternative approaches to full population estimates • Evaluation of alternative approaches to NAEP-TIMSS linkage • Evaluating improvements to processes fir development and review of math items

11. Examples of Specific Contributions • Improved contractor QC plans (e.g., specific decision criteria and follow-up actions) • Improved scoring process • Improved training of field staff • Increased self-monitoring by operational contractors • Independent confirmation of a variety of findings before public release • Specific suggestions for following up on NAGB questions and issues

12. Benefits of an External QA Contractor • Another set of eyes • Unbiased perspective • Free from some of the time and workload stresses of operational contractors