1. Quest for Perfection

2. Florida Comprehensive Assessment Tests and End-of-Course Assessments • FCAT 2.0 Reading Grades 3 – 10 • FCAT 2.0 Mathematics Grades 3 – 8 • FCAT 2.0 Science Grades 5 & 8 • FCAT Writing Grades 4, 8 &10 • Algebra 1 & Geometry • Biology • Under development: U.S. History, Civics

3. Pervasive Quality Assurance • Built into every process from planning and designing, to executing • Process and outcome documentation • Process and outcome audits • Transparency • Continuous refinement and redesign • Discipline and vigilance; trust but verify

4. Pervasive Quality Assurance • Design and Development • Test Construction • Document Quality • Test Security and Administration • Scoring • Calibration/Equating • Reporting Validity

5. Design and Development • Test Development Center • Contracted center, staffed by educators • Ensures educator involvement & content quality • Standing and Rotating Committees (e.g., Content Advisory, STEM Experts, College-ready Standards, …) • Annual Report to District Superintendents • District educators who served in past year • District educators who are in database (statewide ~4200) • Request for nominations

6. Test and Item Content Quality HS Meet statistical criteria? Content Advisory Committees Item Review Committees Items are field tested Review and select field-test items HS Test and Item Specs Yes Appropriate? Standards Assessed Clarifications Item Types Examples Yes Key Educator Involvement HS Handscoring

7. Test Construction • Annual training conducted for Department and contractor staff • Test Construction Specifications • Revised and enhanced annually • Incorporates checklists • Checklists follow documents during test construction • Relentless focus on item quality, overall coherence of test content, statistical quality • Example: location of reading passages compared to prior location is strictly controlled

8. Document Quality • Test Development Center • Editors • Accommodations specialist • Editorial and content checklists • Variety of staff take each test • Intense and synchronized process

9. Test Administration • Annual “Test Administration Debrief” meeting • Review of all products against specifications, including shipment samples • Webinars for preparation and training on CBT • Tests are administered and monitored by certified educators. • Multiple equated core forms of CBT are randomly assigned within classrooms.

10. Test Security Extensive and enhanced test security requirements • Test Security Statute, s. 1008.24, F.S. • Prohibited Activities Agreement • Seating charts, test group codes • Test materials chain of custody form • Student pledge • Data forensics on all grades and tests • Some student results automatically invalidated but have an appeals process. • Investigations, including Department of Law Enforcement and FDOE Inspector General

11. Scoring and Data Quality • Specifications and checklists • Extensive planning and dry runs with mock data • Contractor site monitoring by FDOE • Document processing and equating audits • Triangulation with FDOE and multiple contractors • Additional assistance from FSU for data checking

12. Handscoring Quality Assurance Extensive quality assurance processes are utilized for FCAT Writing handscoring • FDOE and educators review, revise, and approve all training and qualifying materials. • FDOE staff provide on-site monitoring for training, qualifying, and first full week of scoring • One week of “pseudoscoring” • External audit and public reporting

13. Reporting Results • Statute (s.1008.22(c)14, F.S.) dictates testing and reporting timeframes. • Cross-agency planning team meets weekly for several months prior to reporting. • Hire seasonal data checkers to verify files/paper reports. • Choreographed file transfers (contractor, FDOE, web team, data warehouse), version control, analyses, and briefings

14. Pervasive Quality Assurance • Built into every process from planning and designing, to executing • Process and outcome documentation • Process and outcome audits • Transparency • Continuous refinement and redesign • Discipline and vigilance; trust but verify

15. More information Websites: fcat.fldoe.org/default.asp www.floridapathtosuccess.org/ Technical: fcat.fldoe.org/fcatpub5.asp : @EducationFL Florida Department of Education FL Department of Education