School Accountability and the Distribution of Student Achievement

1. School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University

2. No Child Left Behind • States must adopt accountability systems that assign ratings to schools based on student pass rates on exams in elementary, middle school, and high school grades • School is not making ‘Adequate Yearly Progress’ if a pass rate is not sufficiently high, where the required pass rate increases each year • Consequences • Stigma, financial rewards/penalties, loss of local control, changes in property values • Intra-district public school choice provision • Tutoring for economically disadvantaged children

3. Texas Accountability Program • Precursor to No Child Left Behind • Assigns schools one of four ratings based on • Dropout Rates • Attendance Rates • Fraction of students who pass exams (overall and within subgroups by race and family income) • Testing Incentives are based on Pass Rates, not value-added measures of student-achievement

4. Key Provisions of the Texas Accountability System

5. Previous Research Related to School Accountability/Minimum Proficiency • Relative performance of students at different points in distribution (Holmes, 2004; Deere & Strayer, 2001) • Achievement trends • Grissmer & Flanagan (98)- Math NAEP in TX • Hanushek & Raymond- Math NAEP • Carnoy, Loeb, Smith (2002)- TX improvements didn’t correspond with improved 10-12th grade transitions, SAT participation, SAT performance • Low-performing versus high-performing schools (Jacob, forthcoming • States with or without HS graduation exams (Jacobsen, 1993) • Gaming • Exemptions: Figlio & Getzler, Cullen & Reback • School meals: Figlio & Winicki • Disciplinary practices: Figlio

6. Theoretical Framework • Subject Specific but not Student Specific Inputs (as) • Not Subject Specific but Student Specific Inputs (bi) • Subject Specific and Student Specific Inputs (cs) • Assume only campus-wide Math (m) and Reading (r) pass rates count. Call all other subjects (z). Schools want to maximize:

7. The Data • Texas Assessment of Academic Skills • Math Tested Grades 3-8 and 10 • Writing Tested Grades 4, 8, and 10 • Test Documents Submitted for Every Student • Includes Student Descriptors • Campus Level Data on Attendance/Dropouts • Texas Learning Index • Measures How Student Performs Compared to Grade Level • I Do Not Measure Test Score Gains for Observations with Prior Year’s Scores Below 30 or Above 84

8. Pass Rate Probabilities Based on Prior Year Test Score Range Passing Score=70

9. Estimating the Marginal Benefit to the School from a Moderate Increase in a Student’s Expected Performance • estimate the probability that each student passes by grouping students based on their performance during other years • use these student-level pass probabilities to compute the probability that the school will obtain each rating • find the marginal effect of a moderate improvement in the expected achievement of a particular student on the probability that the school obtains the various ratings….

10. How a ‘moderate improvement’ in a student’s achievement is determined • hypothetical pass probability by re-estimating the student’s pass probability after dropping the bottom X% of the current year score distribution among students with identical prior year scores • For example: distribution of this year’s Math scores for students scoring 53 last year in Math 0%: 36 Actual pass probability= .20 20%: 49 40%: 55 60%: 59 Pass probability with X% set at 20%: 80%: 70 .2 / .8=.25 100%: 86

11. Dependent Variable • Year-to-year variation in test scores might be greater at certain points of achievement distribution • Value-added models examining distributional effects SHOULD NOT simply look at changes in levels or in relative place in distribution • Instead, use conditional Z-score… Z-score among students with similar prior year scores • This way, results are compared to typical progress at that place in the test score distribution

12. Empirical Model #1: Campus-Year Fixed Effects Student i in grade g during year y at school s • Si,t includes control variables for student characteristics: • Cubic terms for prior year scores in other subject • Racial dummy variables, Low-income family dummy, • and Race-income interactions

13. Achievement Gains and Marginal Accountability Incentives within Schools and within the Same Year

14. Model #2: Response to Infra-marginal Incentives (Cross-sectional comparisons) • Schools might consider the impact of improving the expected performance of 5% of the students • Define as the marginal change in the schools’ probability of a higher rating if all students in the ‘group’ are expected to do better

15. Achievement Gains and Infra-marginal Accountability Incentives

16. Model #3: Incentive to Improve Performance within a Grade-level • Schools might use inputs that simultaneously affect multiple students • Define as change in school’s probability of receiving a higher rating if all students in student i‘s grade at the school improve

17. Approximate Effect Sizes (SD change in Statewide Achievement Distribution from 1 SD increase in Accountability Incentive

18. Effects of Sample Selection Due to Student Exemptions & Grade Repetition • Exemptions • negative relationship between student-level accountability incentive and likelihood that student is exempted from accountability pool • suggests that estimated effect of student-level accountability incentive may understate the true effect • Also suggests that estimated effect of grade-level incentives overstate the effects for lowest achievers and understate for others • Grade Retention • small, positive relationship between the student-level accountability incentives and the probability of grade retention • effect on main results is unclear, but likely small

19. Conclusions • Schools respond to specific incentives of a rating system • Appear to respond with broad changes in teaching or resource allocation rather than narrowly-targeted changes • Current findings may understate distributional effects • High achievers (top 50% Reading, top 33% math) are not included • May be permanent changes rather than response to short-run incentives • NCLB-style ratings. Are they good? bad?... depends on one’s preferences.