Centre for Market and Public Organisation

1. Centre for Market and Public Organisation A natural experiment in school accountability: the impact of school performance information on pupil progress and sorting Simon Burgess, Deborah Wilson and Jack Worth

2. Introduction • How to improve educational attainment? • And reduce educational inequalities? • Role of school accountability ... • System in England and Wales since 1988: • Market-based accountability • Administrative accountability • It is “consequential accountability” (Hanushek and Raymond) www.bris.ac.uk/cmpo

3. Evaluating school accountability • Hard to get at a causal effect (Figlio and Ladd, 2008): • Lack of adequate control group • Introduction of multi-faceted system all at once. • Hanushek and Raymond (2005) and Dee and Jacob (2009) – evaluate the effects of NCLB • We exploit an event that gets round these problems: www.bris.ac.uk/cmpo

4. From 1992 to 2001, school performance tables (“league tables”) published annually in England and Wales • Devolution of power to Welsh Assembly Government (WAG) after a referendum in 1999 • WAG abolished the publication of these league tables from 2001; they continued in England. • Otherwise, the educational systems continued to be very similar; some later changes in KS testing. • We set up a difference-in-difference approach. www.bris.ac.uk/cmpo

5. What we do • Test whether the removal of this accountability mechanism: • Reduces school effectiveness • Reduces sorting across schools. • Why might these happen (or not)? • Principal agent model – performance tables provide public scrutiny of the output of the school • Teachers are professionals ... • Information as basis for sorting (for some?) www.bris.ac.uk/cmpo

6. Results • Significant and sizeable negative effect on pupil progress: 2 GCSE grades = 0.23 sd (school) = 0.09 sd (pupil) • Equivalent to raising class-sizes from 30 to 38 • Heterogeneity: • Greatest effect on schools with most poor children • No effect in the top quartile of schools or schools with fewest poor children • No effect on sorting; if anything a hint of polarisation in Wales. www.bris.ac.uk/cmpo

7. What is the mechanism? • Less information for choice? • No effect for top schools • But, little variation by degree of competition • Diminished government scrutiny? • Different culture and different emphasis of WAG • Diminished local public accountability? www.bris.ac.uk/cmpo

8. Objections? • Schools in Wales have lower funding • We match schools and control for funding • Poorer neighbourhoods and families • We match schools and use school fixed effects • Other sources of information • We can’t find any, nor can The Times • “Gaming” and manipulation of qualifications • Our evidence does not support this; PISA findings • Confounded with other policies • No obvious ones fit with the timing and distribution www.bris.ac.uk/cmpo

9. PISA test scores for England and Wales www.bris.ac.uk/cmpo

10. Broader issues? • This is a specific but important question (HE, jobs, …) • Other issues: • Learning in a broader sense than GCSE performance • PISA evidence • Pupil well-being • … and GCSE performance • For its own sake • Teacher well-being • … and GCSE performance • For its own sake www.bris.ac.uk/cmpo

11. Change of policy: • Leighton Andrews’ Speech, 2nd February, 2011, Cardiff. (Minister for Children, Education and Lifelong Learning): “We will introduce a national system for the grading of schools which will be operated by all local authorities/consortia. … All schools will produce an annual public profile containing performance information to a common format” www.bris.ac.uk/cmpo

12. Plan • Introduction • Policy Environment and the Policy Change • Methodology • Data • Results • School effectiveness • Sorting • Conclusion www.bris.ac.uk/cmpo

13. School system • State-funded schools = 94% students • National Curriculum, four Keystages • Primary Education, to age 11, compulsory secondary education to age 16. • Keystage 3 exams at age 14, Keystage 4 exams at age 16, also called GCSEs • GCSEs are high stakes exams for students – access to higher education and to jobs www.bris.ac.uk/cmpo

14. School accountability • GCSE exams taken May/June, results reported privately to pupils in August, league tables published November. • Very widespread media attention for these: national and local tv, radio, newspapers ... • Key indicator is % students getting 5 or more “good passes” (grade C or higher). • Other main aspect of accountability: school inspections by OFSTED (E) and ESTYN (W) www.bris.ac.uk/cmpo

15. Policy change in Wales • Referendum in Wales voted in favour of devolved government • Welsh Assembly Government set up in Cardiff, and took over education policy. • Announced in July 2001 the abolition of league tables, and they were not published in November 2001 in Wales, nor since. • Rest of system continued: National Curriculum, GCSEs, ... (some other aspects of testing changed later on). www.bris.ac.uk/cmpo

16. Why? • Arguments against league tables ... • In Wales, perhaps largely ideological, and an explicit avowed preference for “producer” interests over a “consumerist” approach. • Reynolds (2008): reform was “motivated by the left wing political history of Wales ... use of government to ensure enhanced social justice .... greater trust in producer-determined solutions” www.bris.ac.uk/cmpo

17. Potential off-setting changes 1? • Increased school inspections? • No, no change of policy by Estyn (WAG) • Schools publish own results? • Yes, some do; • We surveyed schools in Cardiff and (similar) Plymouth and Newcastle. • Low-performing schools tend not to, and are not obliged to (our survey); • Nothing overtly comparative in Wales. www.bris.ac.uk/cmpo

18. Potential off-setting changes 2? • Other ad hoc private websites? • Not that we (or The Times) could find • Broader accountability system: • Nature and characteristics of choice systems have not changed differentially in the two countries • Top-down accountability: • school performance data continue to be analysed by local government in Wales “as a basis for challenge and discussion” www.bris.ac.uk/cmpo

19. Methodology • Difference-in-difference, with: • school fixed effects • time varying controls • sample 1: all schools (balanced) • sample 2: matched schools • Two outcomes: • School %5+ good passes (published in league tables) • Mean GCSE points (not consistently published) www.bris.ac.uk/cmpo

20. School is the appropriate unit • Usual assumptions: • Non-controlled factors have common time patterns • No change in population characteristics • These are discussed below www.bris.ac.uk/cmpo

21. Matching schools • Take all secondary schools in Wales from balanced sample • Matching variables: performance (level and trend), composition (FSM, ethnicity), school resources, local competition, all averaged over the “before” period. • Match English schools by propensity score to one nearest neighbour in Wales, without replacement • Differences in the means of matching variables are individually and jointly insignificant. www.bris.ac.uk/cmpo

22. Propensity Score Matching (School - Performance) www.bris.ac.uk/cmpo

23. Propensity Score Matching (LEA - Sorting) www.bris.ac.uk/cmpo

24. Data • NPD/PLASC • School performance • Data • Sample • Sorting • Data • Sample www.bris.ac.uk/cmpo

25. Schools data • PLASC/NPD – Pupil Level Annual Schools Census, part of the National Pupil Database • School-year (cohort) level • GCSE scores (two different metrics) • Prior attainment, Keystage 3 scores (KS3) • Poverty measure – eligibility for free school meals (FSM) • School resources (annual expenditure outturn), deflated by CPI and by Area Cost Adjustment • %White British, % Female www.bris.ac.uk/cmpo

26. Sample • Exclude all schools in areas > 10% selective • Wholly-before and wholly-after • Effectiveness (key ages 14 – 16): • Wholly before cohorts are A and B, taking GCSEs in 2000 and 2001; wholly after cohorts are E onwards, taking GCSEs from 2004 • Sorting (key ages 10 – 11): • Cohorts E – H are before, cohort I – ... after www.bris.ac.uk/cmpo

27. Performance Timeline www.bris.ac.uk/cmpo

28. Sorting Timeline www.bris.ac.uk/cmpo

29. Results • Impact of league tables on performance • Simple diff-in-diff • Full diff-in-diff • Heterogeneity • Robustness • Impact of league tables on sorting • Dissimilarity indices • School poverty rates • School – Neighbourhood assignment www.bris.ac.uk/cmpo

30. School Mean GCSE Points score in England and Wales over time www.bris.ac.uk/cmpo

31. School Percent 5 A*-C in England and Wales over time www.bris.ac.uk/cmpo

32. PISA scores www.bris.ac.uk/cmpo

33. Simple Difference-in-differences www.bris.ac.uk/cmpo

34. Full Difference-in-Difference Model Also included: school fixed effects, year effects, prior attainment. SE’s clustered at LA level www.bris.ac.uk/cmpo

35. Results • Heterogeneity • Year • Quartiles of school prior attainment • Quartiles of school poverty • Quartiles of league table position • Quartiles of school local competition www.bris.ac.uk/cmpo

36. Full D-in-D allowing for Heterogeneity through time www.bris.ac.uk/cmpo

37. Full D-in-D allowing for Heterogeneity in Prior Attainment www.bris.ac.uk/cmpo

38. Full D-in-D allowing for Heterogeneity in Poverty Status www.bris.ac.uk/cmpo

39. Full D-in-D allowing for Heterogeneity in Local Competition www.bris.ac.uk/cmpo

40. Results • Robustness • Serial correlation (Bertrand, Duflo and Mullainathan, 2004) – collapse time dimension • Changes in group composition: • Private schools • Cross-border movement • Other aspects of devolution ... • Coincident policy changes: • Literacy hour • Staggered introduction of new equivalent qualifications www.bris.ac.uk/cmpo

41. Collapse to Pre- and Post-reform periods www.bris.ac.uk/cmpo

42. Difference-in-difference of Composition Variables www.bris.ac.uk/cmpo

43. Primary - Secondary Triple Difference www.bris.ac.uk/cmpo

44. Coincident policy changes 1 • Literacy hour introduced in primary schools in England in 1998, so our first “after” cohort exposed in England and not Wales • Control for prior attainment; hard to see an age 5-11 policy affecting age 16 score, conditioning on age 14 ability • Effect only in urban areas, and most of matched sample is rural www.bris.ac.uk/cmpo

45. Coincident policy changes 2 • Staggered introduction of replacement GCSE-equivalent qualifications, in England in 2005 and in Wales in 2007 • Minority activity: affects 4.3% of GCSE points at the median in England in 2006, only 9.3% in lowest decile • Would expect to see differential time trends off the effect over KS3 distribution, but not so. www.bris.ac.uk/cmpo

46. D-in-D allowing for Heterogeneity in Prior Attainment and Through Time www.bris.ac.uk/cmpo

47. Results • Impact of league tables on sorting • Dissimilarity indices, national averages over LAs • Figures • Diff-in-diff regressions • Differential evolution of school poverty rates • Figures • Diff-in-diff regressions • School – Neighbourhood assignment • Figures • Diff-in-diff regressions www.bris.ac.uk/cmpo

48. LEA-level FSM Dissimilarity Index in England and Wales over time www.bris.ac.uk/cmpo

49. Difference-in-Difference model: Dissimilarity Index: FSM www.bris.ac.uk/cmpo

50. School poverty evolution • Match all Welsh schools to English schools on FSM % in the ‘before’ period • Split into quartiles • Trace out evolution of school FSM% quartile-by-quartile separately in England and Wales • Normalised by national trends • Poor schools less poor in Wales, affluent schools poorer?? www.bris.ac.uk/cmpo