Building Evidence in Education: Conference for EEF Evaluators 11 th July: Theory 12 th July: Practice www.educationend

1. Building Evidence in Education:Conference for EEF Evaluators11th July: Theory12th July: Practicewww.educationendowmentfoundation.org.uk

2. Panel session 1: Grand designs

3. Experimental design for Achieve Together Ellen Greaves and Luke Sibieta

4. Achieve Together • Bring together three programmes in a school: • Teach First • Teaching Leaders • Future Leaders • Intensive human capital investment • Original motivation was also to encourage schools to work together and to engage the local community and organisation in school-improvement • Cluster-design (two clusters) • Difficult to evaluate quantitatively

5. Achieve Together • Two pilots: • Area-based design • School-level human capital investment

6. Achieve Together • Two pilots: • Area-based design • One-cluster in Bournemouth • 4 primary schools and 6 secondary schools • Involvement of local community/organisations • Process evaluation • School-level human capital investment

7. Achieve Together • Two pilots: • Area-based design • School-level human capital investment • School-level intervention • No co-ordination within clusters or involvement of external organisations • 48 schools recruited for RCT (24 treatment, 24 control) • Quantitative evaluation and process evaluation

8. Power calculations

9. Challenges • School-level RCT began to look clustered... • Cluster based recruitment • Co-ordination between schools • Complicates and creates risks for evaluation: • What can we learn from the evaluation? • How will the power calculations be affected?

10. Challenges • School-level RCT began to look clustered... • Cluster based recruitment • Co-ordination between schools • Complicates and creates risks for evaluation: • What can we learn from the evaluation? • Is positive impact due to the human capital approach? • Or better co-ordination between schools? • Our findings would be inconclusive • How will the power calculations be affected?

11. Challenges • School-level RCT began to look clustered... • Cluster based recruitment • Co-ordination between schools • Complicates and creates risks for evaluation: • What can we learn from the evaluation? • How will the power calculations be affected? • Required treatment effect from power calculations with clustering at the school level looked ambitious... • Clustering may increase the intra-cluster correlation and increase the challenge

12. Power calculations and the risk of clustering • The ICC may be higher when schools are close together, all else equal (pupils in an area are more similar than pupils in different areas) • Minor

13. Power calculations and the risk of clustering • The ICC may be higher when schools are close together, all else equal (pupils in an area are more similar than pupils in different areas) • Minor • The ICC may be higher when schools work together • Less minor • Co-ordination across schools may make the pilot more homogeneous, so pupils’ outcomes will become more correlated • In the extreme case, we can think of the unit of treatment as the cluster • Uncertain risk for detectable effect size • Distinct from cluster-based delivery where treatment may be more heterogeneous across schools

14. Solutions • Discussion • Negotiate the focus with the project team • Highlight the importance of: • Clear conclusions from this important evaluation • Maximising the probability that the effect size is detectable

15. Conclusions • Cluster-based interventions increase risk for detectable effect size • Fine distinction between cluster-based interventions and cluster-based delivery • Most crucial to define a clear treatment, and for the project team to understand the importance of this • Negotiation with project team is essential, and can be done!

16. Panel session 1: Grand designs Piloting regression discontinuity design to evaluate the effectiveness of a Saturday school programme Carole Torgerson, Andy Wiggins, Victoria Menzies and Kirsty Younger Durham University

17. Using randomised designs to evaluate writing interventions for children during the transition between primary and secondary school Conference for EEF Evaluators: Building evidence in education 11th and 12th July Hannah Ainsworth, Research Fellow, York Trials Unit, University of York Buckley, H., Heaps, C., Jefferson, L., Mitchell, N., Torgerson, C, and Torgerson, D.

18. Trial Design • Individually Randomised (Discover) • Cluster Randomised (Calderdale) • Split Plot Design (Exeter)

19. Background • EEF Transitions Round: • Evaluate 3 different writing interventions with aim to improve children’s writing skills during transition for primary school to secondary school • In all 3 trials: the primary outcome measure will be the combined score on the 2 writing tasks within the Progress in English 11 (Long Form), GL Assessment (PiE11LF). • Secondary outcomes will be scores on the reading, spelling and grammar components of the PiE11LF

20. Discover Summer School Research question: “What is the effectiveness of the Discover summer writing workshop intervention compared with a “business as usual” control group on the writing abilities of participating children?”

21. Discover: Intervention Intervention group • Discover Children’s Story Centre summer writing workshop • 4 week programme • Specifically aimed at children who are less able writers. • The summer school programme will include: • Poetry and literacy sessions, including opportunities to engage with professional poets and writers • Other ‘enrichment activities,’ including drama and sports activities • Trips to West End performances and visits to the Olympic Park. Control group • Business as Usual • Offered opportunity to attend a Saturday writing workshop in the autumn term.

22. Discover: Trial Design • Individually randomised controlled trial • Aim to recruit 250 pupils, randomised on a 1:1 ratio to the intervention and control groups Year 6 pupils will be eligible to be included if they: 1) Are predicted to achieve between level 3c and level 4c in English by the end of Key Stage 2 (based on teacher assessment). 2) Attend a participating primary school 3) Attend a participating secondary school

23. Discover: Trial Design Diagram

24. Discover: Current Progress Progress to date • Recruitment: 124 pupils • Randomisation ratio changed to 3:2 to allow greater numbers in the intervention group in order to fill more spaces in the summer school • Pupils have now been allocated to Intervention/Control • Summer school taking place 29th July to 23rd August • Post-test being delivered in first 2 weeks of autumn term. Challenges encountered to date • Short time frame involved – parents already had plans for summer holidays (reduced consent rates, but also expected to reduce summer school attendance rates) • Potential contamination as other summer schools available to control group (and intervention group) pupils • Identification of pupils challenging in transition from Year 6 to Year 7 as must be able to complete post-test to be eligible

25. Calderdale Improving Writing Quality Research question: • What is the effectiveness of the Improving Writing Quality programme compared with “business as usual” on the writing skills of participating children?

26. Calderdale: Intervention Intervention Group • The intervention will be offered to all pupils in Year 6 • Pupils who are predicted to achieve Level 3 or an insecure Level 4 in English (based on teacher assessment) will continue to receive intervention in Year 7 (Autumn term). • Memorable experiences for pupils in Year 6. • Professional development for primary and secondary school teachers in key elements of the writing intervention Self-Regulated Strategy Development (SRSD), including discussion, modelling and planning. Control Group • Business as Usual • Primary schools will be trained following year

27. Calderdale: Trial Design • Pragmatic cluster randomised controlled trial. • Approximately 24 schools (feeding into 3 secondary schools) randomised on a 1:1 ratio to intervention and control groups

28. Calderdale: Trial Design Diagram

29. Calderdale: Current Progress Progress to date • Recruitment: 23 Primary Schools (possible 24); 3 secondary schools (possible 4) • Successful Information Meeting, to explain intervention and evaluation, hosted by delivery partner and evaluators • Opt Out consent for data sharing: 22 parents opted out • Randomised Primary Schools in March 2013 • Intervention delivery on-going • Post test December 2013 Challenges encountered to date • Secure provision of pupil level data at appropriate time

30. Exeter: Grammar for Writing Research questions: 1. “What is the effectiveness of the whole class Grammar for writing intervention compared with a “business as usual” control group on writing skills of participating children?” 2. “What is the effectiveness of the whole class Grammar for writing intervention plus additional small group intervention compared with a “business as usual” control group on writing skills of participating children?” 3. “What is the effectiveness of the whole class Grammar for writing intervention plus additional small group intervention compared with the whole class Grammar for writing intervention only on writing skills of participating children?”

31. Exeter: Intervention Intervention: Cluster Level • 15 sequential guided writing sessions delivered during literacy classes for the whole glass • Writing sessions delivered using a number of techniques (teacher led, DVD, power point, group exercises) Individual level • Whole class intervention plus additional guided writing sessions delivered in a small group setting Control Cluster Level • Business as usual Individual Level • Whole class intervention only

32. Exeter: Trial Design • Pragmatic cluster randomised trial with a split plot design • Two levels of randomisation – class and individual Class • Within each school, one class randomly allocated to whole class intervention and one to business as usual Individual • Within intervention class, pupils expected to achieve Level 3 or borderline Level 4 in English by the end of Key Stage 2 (based on teacher assessment), individually randomised to whole class plus small group intervention or whole class only intervention

33. Exeter: Trial Design Diagram

34. Exeter: Current Progress Progress to date: • 55 schools recruited & randomised • 2 schools have since dropped out • Intervention delivered between 27th May – 5th July • Post intervention testing between 1st July – 18th July Challenges encountered: • Short time frames challenging • Numerous protocol deviations

35. Conclusions • Large variety of randomised designs available • Consider specific nature and circumstances • Consider carefully the exact research question/s you will be able to answer with chosen design • Any Questions?