Daphne Kounali and Tony Robinson Department of Mathematical Sciences University of Bath

1. Merging Local Research Data with PLASC:An application on the Appropriateness of Free School Meals as a Measure of Deprivation in Educational Research in Hampshire. Daphne Kounali and Tony Robinson Department of Mathematical Sciences University of Bath

2. The HARPS study The HARPS project is an acronym for ‘Hampshire Research with Primary Schools’ Objective: Assess the impact of school composition upon student academic progress. The main aim of the study is to estimate and better understand compositional effects at the primary school level. Definitions: Compositional effects are a subset of contextual effects. Compositional effects attempt to operationalise the influences of fellow pupils characteristics on individual pupil’s performance. Compositional variables included in this study will be; social economic status (SES), ethnicity, gender, prior achievement, special educational needs (SEN) and age. A further remit of the study is to evaluate the probity of FSM as a proxy for the economic element of SES.

3. Study Design The research design is both quantitative and qualitative. Like a set of Russian dolls the project design is of 3 nested parts: • A large scale analysis of over 300 primary schools • A study of a subsample of 46 schools in the Basingstoke and Dean area. • More detailed case studies of 12 schools.

4. The Basingstoke sub-sample One purpose of the Basingstoke subsample is to test the sensitivity of the quantitative measures and methodologies, including FSM, used to identify compositional effects. To this end we have collected and are analysing detailed family background information from the year 3 children in the 46 Basingstoke schools. These schools have been selected so as to compare the findings from these schools with those of the overall Hampshire cohort. This will not only test the probity of measures such as FSM for the Hampshire dataset but will also throw light on their use for school effectiveness studies in general.

5. The Basingstoke subsample dataset contains family background data on 1653 pupils from a total of 2012 students attending 46 out of all 50 schools in the Basingstoke and Dean area. Relevant to economic status these data include: • Occupational group, Working status, Home ownership, Whether in receipt of Working Tax Credit, Whether in receipt of FSM. • Other information was collected on ethnic background, family structure, out-of-school childcare arrangements and cultural behaviour. These data were collected at the beginning of 2004 for year 3 pupils (7 year old) from these schools.

6. One other measure that is becoming popular in research is the Index of Multiple Deprivation (IMD)(Noble et al., 2004) However this relates not directly to individuals but to the small geographical area in which they live, known as a low level Super Output Area (SOA) containing on average about 1500 people. IMD is a composite index based on indices grouped within seven domains: Income; Employment; Health deprivation and disability; Education, skills and training; Barriers to housing and services; Living environment; Crime

7. The Deprivation Geography of Hampshire based on the Multiple Deprivation Index Our Basingstoke Schools on the Deprivation Map The schools are represented by their DFES codes

8. As a competitor to FSM, the IMD or its income component could be used as a distant proxy for economic deprivation but technical problems arise when it is desired to assign a deprivation index to an individual who may or may not share the fortunes of an average neighbour. In general it is not too difficult to assign a deprivation index to a school using IMD information from the SOAs from which the school draws its pupils. This is then an alternative to using the percentage of pupils eligible for FSM as an indicator of economic deprivation at the school level. Note that the education component of the IMD contains achievement data on performance at Key Stages 2, 3 and 4 which may well preclude it as independent information used as input to an analysis of such performance.

9. FSM and how it is used • FSM is frequently used as a factor in representing economic disadvantage in educational research (studies of attainment or truancy) • LEAs incorporate FSM figures in calculating SEN and AEN provision • Only statutory available information on economic disadvantage • Used as a proxy indicator of economic disadvantage and more generally low SES

10. Current FSM eligibility The current eligibility criteria are that parents do not have to pay for school lunches if they receive any of the following: • Income Support • Income-based Jobseeker's Allowance • Support under Part VI of the Immigration and Asylum Act 1999 • Child Tax Credit, provided they are not entitled to Working Tax Credit and have an annual income (as assessed by HM Revenue & Customs) that does not exceed £14,155 • the Guarantee element of State Pension Credit. • Children who receive Income Support or income-based Job Seeker's Allowance in their own right qualify as well.

11. What is FSM • FSM eligibility is a measure of extreme socio-economic disadvantage • At pupil level indicates the pupil’s family economic disadvantage • At school level its aggregate indicates proportion of families suffering extreme economic disadvantage • It is not a direct indicator of non-economic SES disadvantage

12. Criticisms The popularity of such uses of FSM eligibility in studies in educational research is based on its availability. There is no other measure of socio-economic disadvantage that is universally or even widely available at individual pupil level. • Registration is voluntary – affected by shame, social class differentials? • Schools encourage registration because it affects school income • Do families consistently qualify over time? • Often used as a broad indicator of SES but for example does not directly indicate cultural capital

13. The use of Large-Scale Data-Bases on Educational Data • Opportunities for advantages of using large pre-existing data sets • Conceptual and logistic problems in working with these data sets • Statistical Issues germane to the analysis of large, complex data bases

14. The pros • A means for identifying or enumerating cases: they can provide the sampling frame – can be especially helpful in identifying rare events i.e. our subsample statistics indicate that 7.5% incidence of FSM eligibility in the Basingstoke and Dean area compared with a yearly estimate of 9% between 2002-2005. • Provide a broad, if imprecise overview: Frequent existing large data bases can be utilized to measure and define the magnitude and distribution of a problem to more definitive study requiring primary data collection. This is particularly important in studies in social sciences and education where the phenomena under study are not themselves directly measurable and must be studied indirectly through the measurements of other observable phenomena. This is true for both the outcomes we are measuring i.e. we want student progress and we measure test performance as well as the predictors used i.e. we want socioeconomic disadvantage and we measure a number of indicators for this including FSM.

15. Could provide directions of the size of measurement error in our predictors as well as the direction of bias induced by non-response. Our Basingstoke subsample was found to have higher intakes (baseline tests) in literacy, lower incidence of SEN and FSM compared with the rest of Hampshire. Provide opportunities to Validate the quality of research data in terms of what these indicators actually measure as well as address missingness. In our sample for example non-responders were found to be twice more likely to be FSM eligible (using the PLASC information)

16. Conceptual and structural pitfalls They can be frequently flawed. There are 2 reasons for this and both related to the fact that those who design and compile data bases cannot consider the particular needs of every person who eventually might use them. First, PLASC is primarily used for administrative functions related to LEA funding and related policies not always explicitly stated and could be changing. It is simply not designed or maintained to maximize data quality or consistency. Codes and records layouts (i.e. SEN) may be changed periodically. Data elements (i.e. UPN’s from the earlier years might be unreliable pupil identifiers and the same applies for school identifiers after school mergings or closings) are often incomplete and unreliably coded. Information might be inconsistently recorded across LEAs (i.e. literacy tests at reception)

17. Pitfalls continue … • Second, even in well-maintained data bases (Hampshire LEA with a long history of research collaborations) the original goals that dictated what data were collected will not coincide with those of subsequent users and coding practices (i.e. SEN assessed externally) might make secondary analysis difficult if not impossible. Inclusion of both persons and services might be linked to program policies that might change over time. Another example is associated with the LEA’s communication with the individual school systems and heterogeneity among schools in the type of the actual tests administered (e.g. QCA). In most cases we do not know: How were the data collected and processed The time-frame of data collection Completeness and coherence among databases of different LEAs

18. Another example of complex problems in usage • Our original population cohort consists of 11793 (51% boys) of all Hampshire pupils who took the baseline test during 2001-2002 and their KS1 tests during 2003-2004. We have test results for approximately 84% of this cohort. The Hampshire-wide size of the cohort of Reception pupils from PLASC 2001/02 is 14329 and the size of the Yr2 pupils from PLASC 2003/04 is 14308. • The pattern of longitudinal losses in terms of test-results seems quite typical for Hampshire for this phase, judging for other historical cohort data that we also had access to. A wave of around 2000 pupils are lost as we move in time because they are leaving Hampshire schools and another wave of 2000 new pupils are coming-into to these schools. • Because of data-inconsistencies related to correctly identifying and recording school changes (school merging or closing) we end-up looking at 11702 pupils for the phase Baseline to KS1 data. These pupils come from 318 (Infant or Junior and Primary Schools) at baseline and 302 schools at KS1.

19. Back to FSM in HARPS and the pros Some subsample statistics • Out of the 1653 Basingstoke families 124 were receiving FSM (7.5%) • Of these 13 claimed to receive working tax-credit • Single parents and non-working couples account for the majority of FSM recipients So FSM is a very coarse indicator, can we do better? • What about the economically deprived that do not fall below the FSM economic threshold? • No indication of how much the “better off” actually are?

20. A scale of economic status We estimate a latent score for the unobserved economic status using relevant data and Use this Instead of FSM as input to modelling X1 Y1 X2 Latent Index  Y2 Y3 X3 X1: Working status X2: SES (assessed by occupation records) X3: education level of the parent Y1: Tax-Credit Y2: FSM eligibility Y3: Home Renting A graphical representation of the MIMIC model (Multiple-Indicator-Multiple-Cause)

21. FSM stability There is a widely held view that FSM eligibility status is stable over time as the following quote shows:

22. FSM stability in Hampshire 2002-05

23. Who of these ever received FSM?

25. Observations on FSM stability

26. Assessing the effect of deprivation on school performance So, FSM does NOT work

27. More Consequences of the choice of deprivation indexRanking school Under-Performance • School Ranks for the Lowest threshold of the KS1 testing scale: Comparison of school rankings based on FSM and our index suggest that schools can be judged performing worse when FSM is used instead of the deprivation index. • So, when we use FSM we tend to under-estimate school performance. This is because FSM eligibility under-estimates the prevalence of economic deprivation. • Moreover, we also under-estimate the uncertainty associated with these rankings. The differences in rankings affect mostly Average (typical) performing schools The plots depict the school-level residuals (probit scale) against their rank-order. Large negative values in the probit scale characterize the best performing schools.

28. The area where FSM under-estimates school performance is below the diagonal line.Higher Residual Ranks indicate Worse Performance • 61% - 71% (depending on the residuals considered for each threshold of the KS1 scale) of the schools are judged as performing worse when using FSM rather than using our index. • Moreover, the bias also seem to be differential More deprived schools (in terms of intake and/or FSM eligibility) being affected more.

29. Derivation of the index of disadvantage A MIMIC model (Multiple-Indicator-Multiple-Cause) model was used for the derivation of a scale representing economic disadvantage. Binary data on three indicators of economic disadvantage Y1: working tax-credit, Y2: FSM eligibility, Y3: Home renting) are used in a two-parameter logistic model (Rasch-model) as follows: logit[Pr(yijj)]= diji+ dijij, for indicator i=1,2,3 and subject j where di’ is a three-dimensional vector for the i-th element equal to 1 and all other elements equal to zero and  is the latent unobserved measure of economic disadvantage. The parameter estimates for i (location parameters, or intercepts) represent the potency of each indicator. The parameters i (factor loadings) represent the discriminating ability of each indicator.

30. We also specified a structural model for the latent economic disadvantage, allowing the mean indicator potencies to differ between groups as follows: j=0+1X1+2X2+3X3+di1X1, where X1: working status, X2: SES assessed from occupation records, X3: level of education The term di1X1 allows us to introduce a direct effect of working tax credit on the receipt of working tax credit, apart from its indirect effect through the latent variable. This is needed since only employed people would be in receipt of tax-credit.

31. The intercepts’ estimates suggest that the estimated indicator potencies increase from FSM to renting with FSM being the most difficult benefit to get. FSM eligibility does have the highest discriminating ability (slope) as expected followed by home renting. • As expected the estimated parameter  indicates that the odds of receipt of of working tax-credit are significantly reduced for non-working couples compared with the typical couple where one is working full-time and the other part-time.

32. Higher SES reduces economic disadvantage compared to middle classes • Low or unknown SES increases economic disadvantage. • Higher education level is associated with reduced economic disadvantage. • Non-working couple are significantly more likely to be economically disadvantaged, as expected, compared with the typical working status of a couple in this sample (one working full-time and one part time) with the single part-time parents following closely. • Couples consisting of partners where only one full-time working time are also significantly more disadvantaged compared to the typical working couple which appears similar to couples where both partners work full-time. • It is interesting to note that couples consisting of partners where one is working full-time and the other is self-employed, are significantly less disadvantaged compared with the typical couple

33. Working tax-credit is a much More useful indicator especially For middle class occupations • FSM refers to very disadvantaged children even among low SES families

34. Receipt of Working tax credit as an indicator of economic disadvantage seem to INDEX occupation based SES, MOSTLY but not always. This might be due to bias associated with • reporting occupation and/or • assumptions on occupation rankings. • Receipt of Working tax credit as an indicator of economic disadvantage seem to INDEX level of parental education less well. • It seems that level of education is consistent with economic disadvantage for the very poor. • The economic value of vocational qualifications seem less clear for more typical incomes. • This could also be due to the fact that in our sample 90% of the responders are women and only 60% of these women contribute to the determination of SES.

35. Further developments • Improvements in the construction of the index: • Allow for differential measurement error through joint modelling instead of plugging-in index estimates. • In the structural part of the model include interaction terms for education level and the identity of the major bread-winner. • Address problems related to endogeneity i.e. economic disadvantage might affect both school intakes as well as KS1 test results, and there might be both direct and indirect effects of economic disadvantage on school performance.

36. Conclusions

37. As far as the use of PLASC data is concerned the conclusion is • Documentation and Documentation and … • The importance for the research community to get involved with the development and progress of maintaining and improving such invaluable data resources as the PLASC data.