1 / 21

Experimental Evidence on the Early Effectiveness of Intervention in Childhood Orla Doyle

Experimental Evidence on the Early Effectiveness of Intervention in Childhood Orla Doyle ( UCD School of Economics & UCD Geary Institute) Conference on Irish Economic Policy 1 st February 2013. Importance of Early Childhood Investment.

ken
Télécharger la présentation

Experimental Evidence on the Early Effectiveness of Intervention in Childhood Orla Doyle

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Experimental Evidence on the Early Effectiveness of Intervention in Childhood Orla Doyle (UCD School of Economics & UCD Geary Institute) Conference on Irish Economic Policy 1st February 2013

  2. Importance of Early Childhood Investment Targeted early intervention programmes effective way of reducing socio-economic inequalities in children’s skills Well-designed interventions can generate a return to society ranging from $1.80 to $17.07 for each dollar spent YET primarily US-based evidence only on the effectiveness & cost-effectiveness of such interventions Argument for early intervention in more generous welfare systems is not rooted in strong empirical evidence

  3. Irish Context • Prevention & Early Intervention Programme (The Atlantic Philanthropies) & Government , DCYA) initiated in 2004 • Aim: to support the development, implementation and evaluation of strategies for disadvantaged children • All programmes evaluated using experimental or quasi-experimentaldesign • Little tradition of social experimental evaluation in Ireland • Comparing the effectiveness of multiple programmes will enhanced resource allocation and policy design

  4. Preparing for Life Programme • Community-led initiative: operated by Northside Partnership in highly disadvantaged area of Dublin, Ireland • Bottom-up approach: community initiative involving 28 community groups, service providers, & local representatives • Evidence of need: Children scored below the norm on cognitive & non-cognitive skills on school entry (Doyle & McNamara, 2011) • Aim:Improve levels of school readiness by assisting parents in developing skills to prepare their children for school • Evaluation Design: RCT & embedded quasi-experimental design

  5. Design of Preparing for Life

  6. PFL Evaluation Impact Evaluation • Aim: To determine if the programme/service is effective • PFL data collection: Pre-intervention (baseline), 6mths, 12mths, 18mths, 24mths, 3yrs, 4yrs, school entry Implementation Evaluation • Aim: Delve into the blackbox of programme effectiveness & evaluate fidelity to the model • PFL data collection: • Implementation data • Focus groups with participants • Semi-structured interviews with programme staff

  7. Recruitment • Eligibility Criteria: • Cohort of pregnant women residing in PFL catchment area between Jan 2008-August 2010 (32 months) • Includes nulliparous and multiparous women • Recruitment: • Maternity hospital at first booking visit (b/w 12-26 weeks) • Within the local community • Population-based recruitment rate, based on all live births during the recruitment phase, was 52%

  8. Randomisation Procedure • Individual-level randomisation • Unconditional probability randomisation strategy • Design procedure that preserves the integrity of randomization ie. no opportunity for recruiter to intentionally influence assignment • Randomisation process • After informed consent obtained, participant pressed key on tablet laptop to allocate her assignment condition • Evaluation & Implementation team automatically received an email with the assigned ID number and treatment condition • Thus no opportunity to compromise the randomisation process

  9. Baseline Analysis • Determine the effectiveness of the randomisation procedure • Baseline assessment conducted post randomisation, pre intervention • Randomisation worked!

  10. Baseline Differences in Selected Socio-Demographics

  11. Methodology • Estimate treatment effects at birth, 6 months, 12 months • Permutation based hypothesis testing (Heckman et al. 2010) • Classical hypothesis tests unreliable when the sample size is small and the data are not normally distributed • Permutation test are distribution free, thus suitable in small samples • Based on the assumption of exchangeability between treatment conditions under the null hypothesis • Stepdown procedure (Romano & Wolf, 2005) • Ignoring the multiplicity of tests may lead to the rejection of “too many” null hypotheses • Test multiple hypotheses simultaneously by controlling overall error rates for vectors of hypotheses using the family-wise error rate (FWER) as a criterion • Less conservative & more powerful than other methods as takes account of statistical dependencies between tests

  12. Analysis of Maternity Records • Data Collection • Summer 2012 accessed maternity records for PFL participants who attended Rotunda Hospital (85%) & NMH (15%) • 283 parents gave consent to access records - available records for 272

  13. PFL Maternity Records – Summary • No programme impact on birth weight or gestational age, but programme altered the birth experiences of the participating families • Significantly fewer mothers in the high treatment group had a c-section compared to those in the low treatment group • Result remains when the incidence of a previous c-section and pariouness are controlled for • Programme particularly effective in preventing high treatment mothers from having a repeat c-section • Result possibly linked to Tip Sheets which specifically focus on preparing the mother for the birth and being aware of the birthing process and possible labour outcomes

  14. Analysis of 6 Month Results • Sample: PFLHigh/Low treatment groups: 84/90 • Test for treatment effects across 160 measures & 25 Step-down categories

  15. Example of Results: Child development @ 6 Months Notes:1 one-tailed (right-sided) p value from an individual permutation test with 1000 replications. 2 one-tailed (right-sided) p value from a Step Down permutation test with 1000 replications. * indicates the variable was reverse coded for the testing procedure. ‘ns’ indicates the variable is not statistically significant.

  16. Positive Treatment Effects @ 6 Months • Child Development:none • Child Health:Appropriateness & frequency of eating; Immunizations • Parenting:Parent-child dysfunctional interactions; Baby comparison score; Parental hostile-reactive behaviour; Interaction with baby • Home Environment: HOME sub-scales – variety, childcare, toys & books, physical environment, learning materials; Electrical socket covers, Framingham safety survey • Mother’s Health:Mother less hospitalisation for complications after birth • Social support: Frequently sees grandparent; Discuss programme with partner & others; Knows children same age as baby; Meet friends frequently • Childcare & Service use: Voted in last election • Household & SES:none

  17. Analysis of 12 Month Results • Sample: 12 Months: PFLHigh/Low treatment groups: 83/82 • Test for treatment effects across 140 measures & 23 Step-down categories

  18. Positive Treatment Effects @ 12 Months Child Development:ASQ Fine Motor skills; BITSEA Social Competence cut-off Child Health:Chest Infections; Immunizations; Appropriateness eating Parenting: none Home Environment: none Mother’s Health: Drank alcohol Social support: Meet friends frequently; Voted in last elections Childcare & Service use: ~ Household & SES:none Results consistent with the home visiting literature

  19. Attrition & Dosage • Attrition/disengagement • Attrition may bias evaluation results if non-random • 29% of high treatment & 29% of low treatment dropped out/disengaged between randomisation & 12 months • Few individual characteristics predict attrition – IQ more important • Dosage/engagement • Variation in dosage, may lead of variation in effect • Collected data on number/duration of home visits • On average, high treatment group received 21 home visits of ~1 hr in duration between pregnancy & 12 months • 58% of prescribed visits delivered based on bi-weekly visits • Higher maternal IQ associated with higher engagement

  20. Conclusions to date • Many of the results are in hypothesized direction • Few significant effects, yet consistent with literature • Significant findings on birthing experiences, parenting, home environment, child health, social support • Corresponded directly to information on the PFL Tip Sheets • No significant impact on many key domains such as child development, birth weight, breastfeeding, maternal well-being etc. • Attrition & disengagement between 6 & 12 months very low • Engagement below prescribed dosage • Participate satisfaction is relatively high (focus groups)

  21. Update on PFL • Oldest PFL child is 4 years and old & youngest is 18 months • 24, 36 & 48 month surveys are currently in the field • 18 month results available very soon • Over 1,600 interviews conducted as part of impact evaluation • Conducted five school readiness surveys with junior infant cohorts in 2008, 2009, 2010, 2011, 2012 • Evaluation will continue until all children start school Programme website:www.preparingforlife.com Evaluation website: http://geary.ucd.ie/preparingforlife/

More Related