The EPPE 3-11 Research: combining methods to study the impact of pre-school

1. The EPPE 3-11 Research: combining methods to study the impact of pre-school The Effective Provision of Pre-school Education Project TEAM: Kathy Sylva - University of Oxford Edward Melhuish - Birkbeck, University of London Pam Sammons - School of Education, University of Nottingham Iram Siraj-Blatchford - Institute of Education, University of London Brenda Taggart - Institute of Education, University of London Karen Elliot - formerly Institute of Education, University of London EPPE is an Associate ESRC TLRP project Presentation for the LSRI Seminar Series University of Nottingham 9 June 2009 Pam Sammons School of Education University of Nottingham pam.sammons@nottingham.ac.uk

2. Mixed Method (multi-method or combined) Research: 1. Multi-method approach involves using two methods from the same tradition (e.g. ethnography and case study). Often justified in terms of triangulation. 2. Mixed or Combined research usually used to refer to studies that involve both ‘qualitative’ and ‘quantitative’ methods.

3. “Gray and Densten (1998) suggest considering the two approaches of qualitative and quantitative as being on a continuum rather than in the form of a dichotomy that researchers have to select between. ‘Qualitative and quantitative evidence’, therefore, refers to a false dualism (Frazer 1995), and one that we are probably better off without.” (Gorard and Taylor, 2004)

4. The ‘compleat researcher’ (Gorard) – means accomplished; expert' Rather than worrying about rather petty distinctions between constructivism and social constructs, given that no one is suggesting that we have direct experience of an objective reality, we should be more concerned with finding better ways of describing what we do experience (Rorty 1999). For the present we could do worse than adopt a position of being ontologically largely realist (there must be something for us to research), epistemologically somewhat relativist (trying to make sense of and unify different perspectives), and methodologically fairly pragmatic (using whatever methods it takes to get the job done). Gorard, S. (2004) Sceptical or clerical? Theory as a barrier to the combination of research methods, Journal of Educational Enquiry, Vol. 5, No. 1,

5. Parallels Between Research on Child Care and Research on School Effects The study of child care effects & school effects have developed through 3 parallel phases of research questions. For practical & ethical reasons few studies have included experimental manipulation of variables • Early Phase - Does Educational Environment Matter? • Second Phase - What Matters? • Third Phase - What Matters for Which Types of Children ‘The parallels illustrate the value of each field monitoring the progress of the other. In this way , each may benefit from the conceptual and methodological advances made by the other. The small but growing numbers of longitudinal studies completed in both areas point to the need for studies of long and short term processes and effects.’ (McCartney & Jordan, 1990)

6. Focus of SER • The central focus a belief in the potency of social institutions ‘the idea that schools matter, that schools do have major effects upon children’s development and that, to put it simply, schools do make a difference’ (Reynolds & Creemers, 1990) Interest in • the impact of social institutions • characteristics that promote students’ educational outcomes • the influence of contexts on student outcomes • a specific methodology focusing on student progress SER has shown that effectiveness is a relative and retrospective concept dependent on choice of outcomes studied, adequacy of methodology & timescale. These same limitations apply to studies of pre-school influences.

7. Defining Quality in Education • Judgements about quality can refer to different, but closely related aspects, for example: the goals or functions, the contents, the processes, the effects and the conditions or means (Lagerweij & Voogt 1990, p100) • The assessment of quality is complex & value laden.... In the same way as the definition of what constitutes high quality in education is multi-dimensional, so there is no simple prescription of the ingredients necessary to achieve high quality education; many factors interact - students and their backgrounds; staff and their skills; schools and their structure and ethos; curricular; and societal expectations (OECD 1989, p27) • School effectiveness research can help to disentangle and clarify such interactions, and because of this has an important role to play in analysing the constituents of quality in education (Sammons 1999, p155)

8. EPPE Research Design • Adopts an educational effectiveness design • Employs a mixed methodology involving: - large scale quantitative study, longitudinal tracking of sample approx 3000 children from 141 different pre-school settings from age 3+ to 7 years, including multilevel analyses of pre-school centre effects - focuses on a broad range of child outcomes (cognitive progress and social behavioural development) and investigates impact of processes eg measures of centre quality - detailed qualitative case studies of selected centres identified from multilevel analyses of value added as having positive effects on different child outcomes

9. Why Mixed Methods & An Effectiveness based Design? Possible Alternatives: • Randomised Control Trials eg High Scope • Longitudinal Birth Cohort studies eg NCDS - ethical and practical issues and problems of potential bias in RCTs. Not suited to study of impacts of range of ‘typical’ pre-school provision experienced by most young children, best suited to evaluate specific ‘interventions’. Also questions as to how far results of interventions can be generalised to different groups and contexts. - birth cohort research valuable for exploring family and background effects, but too few children in any one pre-school setting or in some types of provision to explore institutional effects in any depth. - mixed methods provides a range of data to explore processes especially pedagogy and facilitates triangulation. In particular, evidence from carefully chosen case studies relevant to practitioners.

10. The EPPE Mixed Method Model ‘evolved a dialectical’ approach: Green and Caracelli (1997) attribute the term to Geertz (1979): “…who argued for a continuous ‘dialogical tacking’ between experience-near (particular, context specific, ideographic) and experience-distant (general, universal, nomothetic) concepts, because both types of concepts are needed for comprehensiveness and meaningful understanding”. (op cit, p10)

11. Aims of the EPPE Research • To establish the impact of pre-school on young children’s intellectual and social/behavioural development. • To identify pre-schools that are more effective than others in promoting children’s development. • To describe the characteristics of effective pre-school settings. • To establish the impact of the home and childcare history (before age 3) on children’s intellectual and behavioural development. • To explore whether pre-school experience can help reduce social inequalities.

12. Features of Mixed Methods • Move away from ‘paradigm wars’ • Recognition of commonalities between quantitative & qualitative approaches - both use empirical observations to address research questions - both incorporate safeguards to minimise sources of invalidity/untrustworthiness - both have goal of increasing understanding seeking to provide warranted assertions about human beings/groups and their environments • Research becoming increasingly interdisciplinary, complex and dynamic, researchers need a solid appreciation of different methods to facilitate communication, promote collaboration and provide superior research • The link between research method and paradigm is not absolute • A pragmatic/balanced/pluralist position encourages researchers to mix research approaches in ways that offer the best opportunities for answering important research questions

13. Key Features of EPPE Mixed Research Design • Quantitative analyses enable comparisons across settings and by type of provision taking account intake differences in terms of significant child, family and home factors including prior attainments or social behaviour (estimates of pre-school centre effectiveness based on ‘value-added’ for cognitive progress and social behavioural development). • Detailed qualitative cases studies of more effective pre-school settings explore organisation and processes, including pedagogy, associated with positive child outcomes and increase understanding of best practice.

14. Operating as a Mixed Method Team • Research team varied backgrounds and experiences of different research traditions • A systematic, non-linear process of iterative triangulation • Worked back and forwards between inductive and deductive models of thinking • Regular weekly all day team meetings for research planning, management and discussion of data and emerging findings facilitated this dialogue

15. Advantages of the mixed method approach 1: (p15 Teddlie & Tashakkori, 2003)

16. Advantages of the mixed method approach 2 Triangulation is defined very broadly as: “the combinations and comparisons of multiple data sources, data collection and analysis procedures, research methods, and or inferences that occur at the end of a study”. (p674 Tashakkori, & Teddlie 2003)

17. EPPE’s Integrated, Mixed Method Design Meta Inferences

18. A Stratified Random Sample • Six local authorities in England, covering urban, inner city, suburban and rural locations and a range of ethnic diversity and social disadvantage • Pre-school centres randomly selected within authorities to include: • playgroups • nursery classes • private day nurseries • day care centres run by local authorities • nursery schools • fully integrated centres • A ‘home’ sample approx 300 who have no significant pre-school experience • Approx 3000 children and 141 centres in total

19. Types of Data • Child assessments over 4 years e.g. cognitive tasks and social-emotional profile • Interviews e.g. with parents and heads of centres, and local authority officers • Systematic rating of ‘quality’ in centres • Documents e.g. curriculum statements, policy documents etc. • Qualitative case studies of centres` including documents, observations and interviews

20. Rationale for Using Multilevel Models in EPPE • Many kinds of data have a hierarchical (clustered) structure. eg children are grouped within pre-school centres, pupils are grouped within primary school, children live in neighbourhoods etc s. Such groupings may reflect geographical factors (housing patterns, accessibility) & social factors, including ability to pay fees • Ignoring the hierarchical structure of data can invalidate findings from traditional analyses eg the ‘Teaching Styles’ research by Bennett (1976) concluded that ‘formal’ teaching approaches were more effective than ‘informal’ ones Reanalysis by Aitkin et al (1976) using multilevel models which took into account the clustering of pupils within classes in comparing teaching approaches overturned the original findings • Cross-classified models can be used to identify and separate the joint impact of different institutional membershipseg both pre-school and primary school may affect young children’s progress over Key Stage 1 (age rising 5 to 7 years)

21. Advantages of Multilevel Analysis for a Pre-school Study • Enables calculation of statistically efficient estimates of regression coefficients Necessary to establish relationships between young children’s personal, family and home learning environment characteristics and on attainment at a given time point (eg age 3 years at entry to study and later at entry to reception )and on progress/development over the pre-school period before testing impact of process measures of pre-school quality, type etc • Enables the interpretation of results in relation to: - the impact of type of pre-school provision - effects of individual pre-school centres - the impact of measures of quality by partitioning variance at child and centre level, providing better estimates of size of effects and standard errors and enabling adequate control of differences between centres in the intake characteristics of children served • Identifies more effective centres for in depth case studies to illuminate different aspects of practice

22. Measuring Value Added for EPPE Sample • using multilevel models to measure children’s cognitive progress / social behavioural development gains • 5 cognitive outcomes: pre-reading, early number concepts, language, non verbal reasoning, spatial awareness/reasoning • 4 social behavioural outcomes: 4 factors from the Child Social Behavioural Questionnaire (CSBQ) Independence & Concentration, Co-operation & Conformity, Peer Sociability, Anti-social / Worried • controlling for prior attainment / social behavioural development at entry to the study at age 3+ using: • baseline cognitive measures: verbal & non verbal composites • baseline social behavioural measures: Adaptive Social Behavioural Inventory factors • controlling for child, family, home learning environment and other significant measures

23. Using Multilevel Models to Explore the Influence of child, family and home environment characteristics Contextualised (cross-sectional) multilevel models were employed to explore children’s attainments or aspects of social behavioural development at entry to pre-school and again at start of primary school, controlling for child, family and home learning environment characteristics (but not prior attainment). Value added models (controlling for these factors and prior attainment or prior social behaviour were used to establish pre-school centre effects, and to explore the influence of type and quality of provision.

24. Child Measures gender age at outcome test ethnicity number of siblings first language birth weight Family Measures eligibility to FSM mother’s / father’s highest level of qualification highest social class Other Measures parent reported developmental problems number of non parental carers length of time spent in reception Home Learning Environment Measures frequency reading to child frequency of library visits frequency child paints/draws at home frequency parent teaches letters/numbers frequency parent teaches the alphabet frequency parent teaches songs, nursery rhymes, etc Other Home Environment Measures frequency child plays with friends at home/elsewhere frequency child visits relatives, etc Measures found to be significant predictors of attainment

25. Simple Example of Measuring Progress Score at entry to EPPE project (3 years+) Score at entry to primary school 0

26. Caterpillar plot showing pre-school centre effects (with 95% confidence intervals)

27. Examples of contrasting Pre-school Centre Effect Profiles for Cognitive Progress Equivalent profiles constructed for social behavioural development Factors: Independence & concentration, Peer sociability, Cooperation & conformity, Anti-social/Worried/Upset X denotes a broadly positive centre profile O denotes a broadly negative centre profile

28. Measures of Quality & Processes • A wide range of information from Centre Managers’ Interviews e.g. aims, organisation, policies, provision, staff qualification etc • Observational measures of quality by trained researchers ECERS_R (Harms, Clifford and Cryer 1998) • total average score • 7 subscales: space & furnishings, personal care practices, language & reasoning, pre-school activities, social interaction, organisation & routines, adults working together ECERS_E (Sylva, Siraj-Baltchford and Taggart 1999) • total average score • 4 subscales: literacy, maths, science/environment, diversity Caregiver Interaction Scale (Arnett 1989) • 4 subscales: Positive relationships, Punitiveness, Permissiveness, Detachment

29. Research Questions addressed via Quantitative analyses • What is the impact of children’s background on cognitive attainment and progress at different ages? • What is the impact of amount and duration of pre-school experience? • Are ‘home’ children (without pre-school experience) at a disadvantage when they start primary school? • Are some pre-school centres more effective than others? • Does type of pre-school experience matter? • Do children attending higher quality settings have better outcomes at entry to primary school?

30. Impact of Pre-school (1) • Pre-school provision helps all children in terms of social and intellectual development. • If children come from disadvantaged backgrounds and are ‘at risk’ of developing special educational needs, then high quality pre-school can make a significant contribution to improving their social and cognitive development. • Children with no pre-school experience (the ‘home’ group) had poorer intellectual attainment, sociability and concentration when they started school, even after taking account of home background.

31. Impact of Pre-school (2) • Duration is important. More months in pre-school (after the age of 2 years) is related to better cognitive and social progress. • Children who attend pre-school settings part-time develop as well as those children attending full-time • Higher quality provision is associated with better child outcomes

32. Type of Provision • Outlier centres (those with significant positive or by contrast negative effects on child outcomes ) were found in all types of provision • Overall children from integrated settings and nursery schools tended to do better on cognitive outcomes even after taking account of children’s backgrounds. • Overall children from integrated settings (which have fully integrated education with care) and nursery classes tended to do better on social behavioural development.

33. Quality and Type of Provision • Good quality pre-school education can be found in all types of settings, more variation within than between types. • In the EPPE sample, nursery schools and centres that integrated education and care tended to be rated highest on quality, including effective pedagogy and responsive social interactions. • Good quality and better cognitive outcomes for children are associated with higher qualifications in staff- especially trained teachers

34. Impact of quality and duration

35. Young children ‘at risk’ of SEN • At the start of EPPE, about one third of children were ‘at risk’ of SEN in terms of low cognitive development. By primary school entry, only 20% were – suggesting pre-school can help the most vulnerable groups of young children make a better start at primary school. • Home children were more likely to be at risk than those who had experienced pre-school, even when the impact of child, family and home environment was controlled • Lack of pre-school may act as an additional disadvantage for the most vulnerable groups of young children.

36. The contribution of social class and pre-school to literacy attainment (age 7) READING at key stage 1, social class and pre-school experience WRITING at key stage 1, social class and pre-school experience

37. The Home Learning Environment What parents and carers do is important and makes a real difference to development. Activities for parents/carers that help children’s development include: • reading to children; • teaching children songs and nursery rhymes; • playing with letters and numbers; • painting and drawing; • taking children to libraries; • (for social outcomes) creating regular opportunities for play with friends.

38. Qualitative case studies of typical to more effective outliers • Qualitative researchers trained in conducting • case studies • Not informed which were more effective • centres during data collection or initial analysis • Both qualitative and quantitative data collected • and categorised

39. Pedagogy: condensed model from a review of the literature on effective pedagogy Involvement Adult and child Co-construction of knowledge Instruction Modes of teaching demonstration explanation questioning modelling

40. Effective Pedagogy in the Early Years

41. Phase 1 analysis – description of case study centres based on interviews, observations and documentary analysis • 1.CENTRE PROFILE • a) accommodation, people, location • b) child outcomes and ECERS scores • 2. STAFFING (subheadings) • 3. MANAGEMENT AND LEADERSHIP (subheadings) • 4. CLASSROOM ORGANISATION • pupil organisation (age mix, grouping arrangements) b) layout • resources • 5. PARENT INVOLVEMENT • a) communication to parents to help them understand the centre • b) parents and the day-to-day life of the nursery • c) how parents are made to feel part of their child's education • d) what's expected of parents • e) parent education

42. Phase 1 analysis – description of case study centres 6. ETHOS a) the atmosphere of the place (i) the welcome (ii) the working climate for children (iii) the emotional climate for children (catering for gender, ethnicity and SENs / expectations of behaviour) (iv) staff co-operation / collegiality (v) support for staff b) the philosophy c) ethos as portrayed through displays and booklets

43. Phase 1 analysis – description of case study centres 7. CURRICULUM a) curriculum policies b) balance and breadth (across subjects and including ICT) c) practitioner emphasis d) assessment e) curriculum planning, continuity and progression f) the role of visits and visitors 8. PEDAGOGY a) practice b) the quality of interaction c) the role of play and direct instruction d) the role of the teacher e) identifying children with SENs f) ensuring continuity and progression g) developing dispositions 9. COMMUNITY OUTREACH

44. Data entered into QSR NUD*IST – vivo 107 parent interviews and 14 centre plans 14 files of documentary and case study analysis 42 staff and manager interviews 204 transcribed naturalistic observations of 28 staff (2 staff/setting, 400+hrs taken over 56 whole days) 254 systematic target child observations 20-40 mins each

45. Pedagogy- the key findings are in the following areas: • Management and staff • Ethos and climate of the settings • Adult-child verbal Interactions • Differentiation and formative assessment • Discipline and adult support in talking through conflicts • Parental partnership with settings and the home education environment • Pedagogy • Knowledge of the curriculum and child development

46. Sustained shared thinking Good outcomes are linked: Adult-child interactions that involve ‘sustained shared thinking’ and open-ended questioning to extend children’s thinking Sustained shared thinking: An episode in which two or more individuals “work together” in an intellectual way to solve a problem, clarify a concept, evaluate activities, extend a narrative etc. Both parties must contribute to the thinking and it must develop and extend.

47. Percentage of pedagogical interactions (cognitive and monitoring) in settings varying in effectiveness

48. Percentage of high cognitive challenge activities within each initiation category in each setting type

49. Effective Pedagogy in the Early Years (1) • Effective pedagogues observe and assess children to ensure the provision of challenging yet achievable experiences • Effective pedagogues model appropriate language, values and practices, encourage socio-dramatic play, praise, encourage, ask questions, interact verbally with children • The Foundation Stage has clearly improved the relationship between nurseries and schools but transition remains a significant issue

50. Effective Pedagogy in the Early Years (2) • Effective pedagogy is both ‘teaching’, and the provision of instructive learning and play environments and routines • The most effective settings provide both teacher-initiated group work and freely chosen yet potentially instructive play activities • Excellent settings tend to achieve an equal balance between adult-led and child-initiated interactions, play and activities