Success and CBAS: An Economic Evaluation of the Incredible Years

1. Success and CBAS: An Economic Evaluation of the Incredible Years Biglan, Taylor, Meltzer, Rusby, and others Oregon Research Institute Foster Central PA Office & Penn State

2. www.personal.psu.edu/emf10 Recent In-Press or Working Papers on • economic issues in prevention forthcoming in Applied Developmental Science • public costs of improved mental health services • “How Economists Think about Family Resources and Child Development” forthcoming in Child Development • dosage that uses propensity score methods • other topics • Paper using FT data on screening • Paper using FT data on non-ignorable non-response and multiple imputation Stay tuned: social costs of conduct disorder Econ Eval of Incredible Years

3. There are many economic questions one might consider • What is the most cost-effective research design? • What does an intervention cost (efficacy)? • What does an intervention cost II (effectiveness)? • Inputs (personnel, space)? • Scale? • How do those costs vary? Econ Eval of Incredible Years

4. There are many economic… (cont) • Is an intervention cost-effective? • Does an intervention reduce the costs of illness? Econ Eval of Incredible Years

5. Let’s talk about two grants at ORISpecific Aims • Evaluate the indicated and universal components of a multi-component intervention • Estimate the full costs of the intervention • Estimate the • Impact on net social costs (a circumscribed benefit-cost analysis) • Relative cost-effectiveness of different levels of intervention Econ Eval of Incredible Years

6. Outline • Design—Intervention • Design—Evaluation • Overview of the economic evaluation • Estimating the full costs of the intervention • Impact on net social costs • Cost-effectiveness analysis • Hypothetical Illustration • Problems and Challenges Econ Eval of Incredible Years

7. 1) Design--Intervention Three components • Videotape-mediated training of teachers in behavior management • Videotape-mediated social problem-solving skills training for elementary school children • Empirically based parent training Econ Eval of Incredible Years

8. 2) Design—EvaluationSuccess • Evaluation of targeted components • Longitudinal study of 162 children (3 children per class X 2 classrooms X 27 schools) • Participants identified as at risk in K using teacher reports Econ Eval of Incredible Years

9. 2) Design—EvaluationSuccess • Random assignment of high-risk children within schools • Data collection through • Annual parent interview + two phone interviews • Through grades 2-3-4 (depending on the cohort) Econ Eval of Incredible Years

10. 2) Design—EvaluationCBAS • Track 648 additional children (12 children per class X 2 classrooms X 27 schools) • Extends data collection two years • Adds economic analysis of entire project Econ Eval of Incredible Years

11. 3) Overview of economic evaluation • Estimating the full costs of the intervention • Impact on net social costs (circumscribed benefit-cost analysis) • Cost-effectiveness analysis Econ Eval of Incredible Years

12. NOTE: we’re only capturing a portion of the economic impact.

13. 3a) Estimating the full costs of the intervention • Allocating explicit costs • Divide labor costs into intervention and evaluation • Identify and measure other variable costs (e.g., materials) Econ Eval of Incredible Years

14. 3) Estimating the …. (cont.) • Allocating explicit costs (cont.) • Allocated other facility and fixed costs • Allocate total costs to intervention components Aside: how to handle training costs Econ Eval of Incredible Years

15. 3) Estimating the …. (cont.) • Allocating implicit costs • Space costs • Teacher time • Parent time • Lost instruction Aside: implications for parent interview Econ Eval of Incredible Years

16. 3b) Impact on net social costs • Identify the relevant outcomes • Use of alternative services, such as school, mental health and social services • Care giver employment (including impact on transfers) • Measure treatment impact Econ Eval of Incredible Years

17. 3b) Impact on net social costs • Express those behaviors in dollar terms • School budgets • Billing records • Estimating net costs • Discounting • Sensitivity Analysis • Precision Econ Eval of Incredible Years

18. Aside: What should a services instrument look like? • What’s the right time period? • How many modules (or service types) do we include? • Do we ask about expenditures? • What level of other detail do we specify (e.g., # visits)? Econ Eval of Incredible Years

19. Aside: Why medical and administrative records? • Advantages: • Parents are inaccurate • Parents are not knowledgeable • Reduces respondent burden • Creates new and more mysterious forms of missing data Econ Eval of Incredible Years

20. Aside: Why medical (cont). • Disadvantages: • Privacy concerns • IRB issues • Agency and provider burden • Health providers & HIPPA • Public agencies • Legal issues • formatting of authorization • Whose authority Econ Eval of Incredible Years

21. 3c) Cost-effectiveness analysis • Reports of child’s behavior • Academic performance • Care giver strain • Health-related quality of life Econ Eval of Incredible Years

22. 4) A Hypothetical Example We’ll replace these figures with real ones from our current study. Econ Eval of Incredible Years

23. What Does IY Cost? Costs as disseminated under reasonable circumstances • Agency would run program in two classes with 14 children • Agency would need parent group leader (who would have to be trained) • Group and classroom sizes consistent with experience Econ Eval of Incredible Years

24. What does IY Cost? (cont) • Included appropriate materials • Other costs included (such as meals and sitters) • Included parent and teacher time (including preparation time) • Includes training costs Econ Eval of Incredible Years

25. What does IY Cost? (cont) • Reasonable assumptions • number of parent groups a trainer leads during tenure • Salaries • Value of parental time (?) • Excludes • costs of development (e.g., tapes) • space costs • opportunity costs for classroom time • additional administrative costs (such as receptionists) Econ Eval of Incredible Years

26. Bottom Line Econ Eval of Incredible Years

27. Breakdown of Costs Econ Eval of Incredible Years

29. $535 $467 $400 $368 $332 $313 $291 Costs 600 500 400 To pick the most cost-effective parent group size, we need the outcome side. 300 200 100 0 10 12 15 17 20 22 25 Size of P Groups Econ Eval of Incredible Years

30. 5) Problems and Challenges • Design • Power • Methodology • Combining record review and self-report data • Fully capturing the future (without waiting for it to arrive) • Dealing with highly influential cases Econ Eval of Incredible Years

31. 5) Problems … (cont) • Administrative data • All the usual problems • Combining data from multiple sources • Measurement • Overall well-being • Broader conceptual issues • Efficacy v. effectiveness Econ Eval of Incredible Years