Conducted by

1. Evaluation of the DC Opportunity Scholarship Program:Achievement Impacts After Three Years of Treatment Exposure Conducted by Westat, University of Arkansas, Chesapeake Research Associates, Georgetown University Presented by Patrick Wolf, Principal Investigator “Reversion to the Mean or Does Dosage Matter?” IES Annual Research Conference June 8, 2009 Washington, DC

2. Presentation Overview • The Program • The Study • Achievement Impacts in Year 3 • Achievement Impacts Over Time • Dosage in the Context of School Choice

3. The Opportunity Scholarship Program • DC School Choice Incentive Act signed in early 2004 • $14 million/yr for 1,700 student vouchers worth up to $7,500 • Eligible students must be: • DC residents • entering grades K-12 • income <= 185 percent of poverty • Priority to applicants from “needs improvement” (SINI) schools • 68 participating private schools in D.C.

4. The Impact Evaluation • Randomized Control Trial (RCT) Design: baselineeligible applicants assigned by lottery to be offered (“treatment” group) or not offered (“control” group) a scholarship • Impact Sample: • First two cohorts in 2004 (Cohort 1) and 2005 (Cohort 2) • 1,387 students assigned to treatment, 921 to control • Cohort 1 outcomes lagged one year and combined with Cohort 2 • Outcomes tracked annually: SAT-9 in reading and math, parent surveys, student surveys, principal surveys (public and private) • Analysis: Regression-adjusted intent-to-treat (ITT) impact, Bloom-adjusted impact-on-treated (IOT), Instrumental Variable Analysis of effect of private schooling

5. DC Choice: Year 3 Achievement Impacts • Overall: • Reading impact of 4.5 scale score points (.13 SD or 3.1 months of learning) • No impact in math • Subgroups by student baseline conditions: • Reading impacts for (1) non-SINI, (2) higher performing, (3) female, (4) K-8, (5) Cohort 1 • No impacts for (1) SINI, (2) lower performing, (3) male, (4) 9-12, (5) Cohort 2 • No impacts for any subgroups in math

6. Regression Adjusted Impact and Confidence Interval: Year 3 Reading Scale score points

7. Regression Adjusted Impact and Confidence Interval: Year 3 Math

8. DC Choice: Achievement Impacts Over Time • Overall effects across 3 outcome years: • Apparent trend of cumulating impacts in reading • No apparent trend in math • Subgroup reading effects across 3 outcome years: • Of 5 with significant impacts in year 3, all but Cohort 1 suggest cumulating impacts • Of 5 without impacts in year 3, X suggest cumulating impacts • Data not pooled across years so significance of trends unknown

9. Impacts Over Three Year for the Full Sample Scale score points

10. Impacts Over Three Years: SINI at Baseline Subgroups Scale score points

11. Impacts Over Three Years: Baseline Performance Subgroups Scale score points

12. Impacts Over Three Years: Gender Subgroups Scale score points

13. Impacts Over Three Years: Grade Subgroups Scale score points

14. Impacts Over Three Years: Cohort Subgroups Scale score points

15. The DC Choice Evaluation and Treatment Dosage • Impact patterns differ by domain: • Math may have suffered reversion to the mean • Reading impacts appear to cumulate • No obvious reason for the difference • School choice interventions may require long treatment exposure to demonstrate results: • Starts with adjustment to a school switch • May then place student on steeper growth curve • Bring on Year 4!

16. Contact Info Patrick Wolf, Ph.D. Professor and 21st Century Endowed Chair in School Choice Department of Education Reform College of Education and Health Professions 201 Graduate Education Building University of Arkansas Fayetteville, AR  72701 Phone: 479-575-2084 FAX: 479-575-3196 pwolf@uark.edu