Evidence-Informed Program Improvement: Using principles of effectiveness to enhance the quality and impact of youth and

1. Evidence-Informed Program Improvement:Using principles of effectiveness to enhance the quality and impact of youth and family programs STEPHEN SMALL Professor of Human Development & Family Studies, University of Wisconsin-Madison and Extension Human Development & Family Relations Specialist University of Wisconsin-Extension Iowa State Webinar November 17, 2009

2. Agenda • What are evidence-based programs, practices, and organizations? • What are some common principles of effective evidence-based youth and family programs? • How can Evidence-Based Program Improvement (EIPI) be used to improve existing programs?

3. Evidence-based Programs and Practices Years of research have demonstrated that specific approaches, practices and strategies can reduce problem behaviors and enhance positive developmental outcomes The most effective programs and practices are termed “evidence-based” – these are the “gold standard”

4. Why the growing interest in evidence-based programs and practices? • Accountability: Increasingly required by funders within political climate • Efficiency: Reduces development costs associated with creating a new program • Availability of good science: Body of scientific evidence has reached a critical mass • Cost-effectiveness: Increased likelihood that limited resources will be used wisely • Marketability: Documented evidence can help sell and justify program • Ethics: Responsibility to use what is known to be effective

5. The Evidence-Based Programs and Practices Continuum Evidence-based Programs Evidence-based Kernels and core elements Evidence-based Practices Evidence-based Principles Evidence-based Organizational Systems

6. Evidence-Based Programs A new class of programs that: • Are based on a sound scientific evidence and theory • Have been carefully implemented, evaluated and replicated using rigorous scientific methods • Have evaluation findings that have been subjected to critical review • Often “endorsed” as evidence-based by a federal agencies or respected research organizations (e.g., Campbell Collaboration, SAMHSA, CDC)

7. Terminology Evidence-based Research-based Evidence-based = Research-based + Rigorously tested

8. Examples of Evidence-based Programs Family/Parent Education Programs • Nurse Family Partnership: 15-year follow-up showed ↓arrests (54%) among adolescents; $3.59 return on the dollar • Strengthening Families Program for Parents and Youth 10-14:↓ aggression, alcohol, tobacco, marijuana, meth; $7.82 return

9. Examples of Evidence-based Programs School-based programs • Life Skills Training: 6-year follow-up showed ↓alcohol, tobacco, marijuana, narcotics, hallucinogens; $25.61 return on the dollar • Olweus Bullying Prevention Program:↓in bullying and victimization, vandalism, fighting, truancy; ↑ in school attachment

10. Examples of Evidence-based Programs Mentoring Big Brothers Big Sisters:↓ in violence, alcohol and drug use; ↑ in school performance; $1.01 return

11. The effect of the Strengthening Families 10-14 Program on teen aggressive and hostile behaviors Aggression and Hostility Index SOURCE: Spoth, R., Redmond, C., & Shin, C. (2000) Reducing adolescents' aggressive and hostile behaviors: Randomized trial effects of a brief family intervention four years past baseline. Archives of Pediatrics and Adolescent Medicine 154, 1248-1257

12. What are some limitations of evidence-based programs? • Not always easy to find • Can be costly to implement in both time & $ • May not address particular issues or audiences • Tend to be problem focused • May downplay local knowledge and community ownership • Overlooks value of promising grassroots programs • Often difficult to transport to new settings • Other?

13. The Evidence-Based Programs and Practices Continuum Evidence-based Programs Evidence-based Kernels and core elements Evidence-based Practices Evidence-based Principles Evidence-based Organizational Systems

14. Evidence-Based Kernels • Kernels = units of behavioral influence and practice that have been shown through experimental evaluation to produce reliable effects on behavior • Kernels =Active ingredients of effective programs

15. The Evidence-Based Programs and Practices Continuum Evidence-based Programs Evidence-based Kernels Evidence-based Practices Evidence-based Principles Evidence-based Organizational Systems

16. Definitions of Evidence-based practices • “The integration of the best available research with clinical expertise in the context of patient characteristics, culture and preferences.” (American Psychological Association) • “The integration of best-researched evidence and clinical expertise with patient values.” (Institute of Medicine of the National Academies)

17. Evidence-Based Practice Practitioner Expertise & Experience Evidence-Based Practices Research Evidence

18. The Evidence-Based Programs and Practices Continuum Evidence-based Programs Evidence-based Kernels Evidence-based Practices Evidence-based Principles Evidence-based Organizational Systems

20. EVIDENCE-INFORMED PROGRAM IMPROVEMENT: Using principles of effectiveness to enhance the quality and impact of family-based prevention programs February 2009, Family Relations, 58, 1-13. http://whatworks.uwex.edu

21. Principles of evidence-based youth and family programs • Design & Content • Relevance • Delivery/Implementation • Assessment & Quality Assurance

22. Program design and content Effective programs… • Target relevant assets and/or risk and protective factors • Are theory-driven • Are of sufficient dosage and intensity • Are comprehensive • Use active learning approaches & appropriate delivery formats

23. Theory-driven • Based on strong, scientific theory • Target relevant risk, protective factors and assets to accomplish objectives • Logical, well-developed program theory A well thought out recipeis essential

24. Theory driven - Example Nurse-Family Partnership (early childhood/parent ed)- grounded in theories of human ecology (Bronfenbrenner, 1979, 1992), self-efficacy (Bandura, 1977, 1982), and human attachment (Bowlby, 1969)

25. Theory-driven • Based on strong, scientific theory • Target relevant risk, protective factors and assets to accomplish objectives • Logical, well-developed program theory A well thought out recipeis essential

26. Have sufficient dosage and intensity Enduring change takes effort

27. Thinking about sufficient dosage and intensity… For a mentoring program to be effective, the relationship on average has to exist for at least… 2 weeks 1 month 6 months 9 months 1 year

28. Comprehensive Simple solutions rarely work

29. Use active learning techniques and appropriate delivery format

30. Program relevance Effective programs… • Are developmentally appropriate • Are socio-culturally relevant • Are appropriately timed • Are sensitive to local context • Are responsive to individuals One size does not fit all

31. Program delivery Effective programs… Foster good relationships Are delivered by well-trained and committed staff Address environmental barriers to program delivery and behavior change -Provide a safe setting for the intervention Are embedded in a supportive organization infrastructure

32. Well trained and committed staff - Examples Big Brothers, Big Sisters (mentoring program) – Orientation, volunteer screening, youth assessments, matches, supervision Promoting Alternative Thinking Strategies (PATHS, classroom program)- Teachers receive training in a two- to three-day workshop and in bi-weekly meetings with the curriculum consultant.

33. Program assessment and quality assurance Effective programs… • Are well-documented • Focus on evaluation and refinement • Provide opportunities for ongoing feedback

34. Make sure the program is well-documented It’s important to understand the program in order to implement it well Maintaining program fidelity requires that people know what the program is about Program replication and evaluation is difficult if the program isn’t well documented

35. Focus onevaluation & refinement See evaluation as your friend Evaluation takes many forms Evaluation isn’t always easy Consider EIPI as a first step

36. Evidence Informed Program Improvement http://whatworks.uwex.edu/attachment/whatworks_manual.pdf

37. SUMMARYPrinciples of Effective Youth and Family Prevention Programs Program design and content Program relevance Program delivery Program assessment and quality assurance

38. The Evidence-Based Programs and Practices Continuum Evidence-based Programs Evidence-based Kernels Evidence-based Practices Evidence-based Principles Evidence-based Organizational Systems

39. Evidence-based organization • Promotes an organizational culture that values science and evidence-based approaches • Creates an infrastructure that supports effective practices and programs: • Uses existing evidence-based programs when available and appropriate • Employs a strong theoretical/science approach to new program design • Provides regular opportunities to reflect on programs and effective practices • Uses scientific tools to assess needs/assets, understand local context and adapt programs • Provides ongoing technical support for selecting, implementing and evaluating effective programs and practices • Builds organizational and community capacity to identify relevant goals and implement appropriate and effective approaches to address these goals

41. Questions/Discussion