Innovations in Prevention & Implementation Science: Implications for Improvement in Schools

1. Innovations in Prevention & Implementation Science: Implications for Improvement in Schools C Hendricks Brown, Northwestern University Feinberg School of Medicine Sheppard Kellam, Johns Hopkins University Jeanne Poduska, American Institutes for Research Juan Villamar, Northwestern University Feinberg School of Medicine Gracelyn Cruden, Northwestern University Feinberg School of Medicine

2. Acknowledgments: for Ce-PIM • National Institute on Drug Abuse: CENTER FOR PREVENTION IMPLEMENTATION METHODOLOGY FOR DRUG ABUSE AND SEXUAL RISK BEHAVIOR (PI Brown P30 DA027828) • Substance Abuse and Mental Health Services Administration SAMHSA Ce-PIM Partnership

3. Acknowledgements: Work to Date Funding Agencies: • IES • NIDA • NIMH • NICHD

4. Process in this Workshop: Active Engagement

5. Workshop Outline • A Challenging Tour of Implementation Science and its Home in Service Delivery Systems • Public Education and Public Health: Toward a System of Systems • Systems Thinking and Partnerships in Daily Life • Illustrative Innovations of Ce-PIM • Summary: Building an Integrated Structure to Support Children and Youth Reaching their Full Potential

6. 1. A Challenging Tour of Implementation Science and its Home in Service Delivery Systems The Perspective at Ce-PIM A. Implementation as an action strategy working on systems change B. Why Study Implementation? C. Approaches to Moving Effective Interventions into Practice D. Three Interacting and Evolving Components of Implementation E. A Systems Approach involving 3 interacting components Small group exercise

7. Ce-PIM Mission: Building scientific and methodologic rigor into implementation research and practice. AIMS: • Develop Systems Methods (Systems Science, Engineering, Computational) Methods for Implementation Science • Infuse systems methods into the field of Implementation Research • integrating methods within the practice of prevention implementation at the federal, state, county, and local levels.

8. A. What is Implementation Research? According to NIH (2008): The use of strategiesto adopt and integrate evidence-based health interventions and change practice patterns within and across specific systems Action Oriented Within Settings or Systems AND collects data Chambers DA. Advancing the science of implementation: A workshop summary. Administration and Policy in Mental Health and Mental Health Services Research. 2008;35(1-2):3-10.

9. B. Why Study Implementation? 1. We know a lot about what works 10K reviewed studies in What Works Clearinghouse 2. We are short on implementation action strategies to put what works into practice: WWC Practice Guides: Reducing Behavior Problems in the Elementary School Classroom 2008 3. It takes too long for research to affect practice

10. Closing the Gap between What We “Know” Works and What We Do. “17 Year Gap” in Health Care

11. 17 Years in Implementing an Statistical MethodologyGeneralized Estimating Equations (GEE: Brown et al., APMH 2012)

12. Is the gap between research and practice similar in education to that existing in health? Types of Gaps? As long as? As important to shorten? Which way? As resistant to change?

13. C. Approaches to Moving Effective Programs into Practice

14. Sustainment Implementation Implementation Adoption / Preparation Exploration Effectiveness Studies Efficacy Studies Preintervention Traditional Research Pipeline for Implementation Making a Program Work Generalized knowledge Does a Program Work? Local knowledge IOM 2009 Landsverk, Brown et al. 2012 Aarons et al., 2011 Real World Relevance Intervention Could a Program Work? Traditional Translation Pipeline

15. A Focus on the Intervention itself is Not Sufficient • The use of effective interventions without implementation strategies is like serum without a syringe; the cure is available, but the delivery system is not Fixsen, Blase, Duda, Naoom, Van Dyke,2010.

16. D. Three Interacting and Evolving Components of Implementation (Chambers et al., 2013 Imp Sci) • Intervention: Program, Practice, Policy, Principles • Practice Setting: Delivery Support System • Ecological System: Population and Community/Cultural Context

17. Key Parameters on Implementation Science A. Transitions of Research Questions: Effectiveness – Can a Program Work in real life systems? Implementation – Making a Program Work B. 4 Stages: Exploratory, Adoption, Implement w/ Fidelity, Sustainability C. Focus on Interactions of Program, Delivery System, and Ecology D. Science: Produce Generalizable Knowledge rather than Local Knowledge Measurement, Modeling, and Testing Using Rigorous Methods E. Partnerships

18. Definition of System A set of connected parts forming a complex whole

19. Group Exercise: Produce a System Map of the Educational System Three Steps: • Who needs to be at the table? What are the components needed represented? • How do they Interact with one another? • Where would you intervene to affect intervention usage? Education Policy, i.e. Common Core Standards Identify one person from your small group to bring this back to the group

20. Follow-up Question on Systems: Would your list of players and location intervention differ based on implementing New policy Replacement of an existing curriculum Removal of a expulsion policy

21. Summary of Part 1 • Multi-goal, Interacting, Dynamic and Multilevel System • System View Looks at Interactions between components. • Behavior of a system is not linear: complex. • A useful way to understand a system is to intervene and study its behavior.

22. 2. Public Education and Public Health: Toward a System of Systems Sheppard G. Kellam, MD Center for Prevention Implementation Methodology Johns Hopkins Bloomberg School of Public Health Society for Research on Educational Effectiveness Washington, DC, September 4-5, 2014

23. The Baltimore Education and Prevention Partnership • The Baltimore City Public School System (BCPSS) has collaborated in 3 generations of education and prevention field trials. • Trials were directed at helping children master obeying rules of behaving, attending, academic learning, socializing appropriately in 1st grade classroom. • Interventions were tested separately in 1st generation (our focus today), then together in later trials.

24. Two Dimensional View of Mental Health • Social Adaptational Status—level of success or failure in main social fields at each stage of life • Psychological/physical Well-Being—includes affect, cognitive development, i.e., internal condition of the individual

25. Prevention Research and Service Strategies DEVELOPMENTAL EPIDEMIOLOGY: directed at early proximal targets COMMUNITY PREVENTION: directed at Community & School proximal targets MORE IMMEDIATE RISK: directed at more recent proximal targets COMMUNITY / SOCIETAL: directed at Policies & Laws as proximal targets Kellam/Langevin, 2003

26. Levels of Prevention and Treatment Universal Entire Population Selective Indicated Rx Med, MH, Soc Welfare

27. Current Problems in System Coordination • Prevention and treatment not integrated • Service Agencies are Silos—minimal collaboration • No shared information system monitoring developmental progress • Mystery as to who gets “referred” for services • Little follow-up of “no-shows”

28. Early Risk in Prevention Research • Over the last four decades much has been learned about early risk factors and paths leading to drug abuse, and other behavioral, mental health, and school problems. • Most if not all are strongly related to school failure, also a major risk factor for later drug abuse, alcohol, tobacco, depression, anti-social and other problem outcomes. • Aggressive, disruptive behavior as early as 1st grade has been repeatedly found a risk factor for later drug and alcohol abuse and disorders, delinquency, violence, tobacco use, high risk sex, school failure and other high risk behaviors.

29. When a child misbehaves…

30. The Baltimore Education and Prevention Partnership • The Baltimore City Public School System (BCPSS) has collaborated in 3 generations of education and prevention field trials. • They were directed at helping children master key social task demands in 1st grade classroom. • Interventions were tested separately, then together. • The 1st generation will be our main focus here, where the Good Behavior Game (GBG) was tested by itself and the children were followed up at ages 19-21 and beyond.

31. High Risk Children in Well vs. Poorly-Managed Classrooms (control classrooms only) • If the top 25% of children on aggressive behavior were in disrupted classrooms, their risk of severe aggressive behavior by middle school was up to 59 times the average child’s. • If similar children were in well-managed classrooms, the risk was up to 2.7 times the average child’s.

32. Kellam et al., Addiction Science & Clinical Practice; 6:73-84, 2011

33. The Role of School Information Systems in our New System of Systems • Following individual children over time, schools and communities is required for assessing their progress, needs and for integrating services across agencies • Information systems tracking each child exist in most school districts and can be extended and used to integrate services and monitor progress • Confidentiality is by law guaranteed by school authorities and can be expanded to include data as needed for services

34. Partnerships for Building and Overseeing the New System of Systems • Analyze political constituencies • Engage and work through trust with each • Learn their mission and vision • Identify mutual self-interests and • Encourage their coming together as oversight committee

35. Whole Group Activity: Partnerships • What do we mean by “who” needs to be included? • The nature of the process of engaging and developing the base of support/collaboration.

36. 3. Systems Thinking and Partnerships in Daily Life Jeanne Poduska, Sc.D. American Institutes for Research Society for Research on Educational Effectiveness Washington, DC, September 4-5, 2014

37. Systems Concepts • Introduction of programs and action strategies requires adaptation to systems, e.g. staffing, fiscal allocations • We operate in complex systems: multi-level, multi-actor • Change occurs locally (even if simultaneously) • Systems thinking focuses on interactions

38. Working Definition of Scaling-Out “Scaling out is the deliberate use of strategies to implement and sustain evidence-based interventions through or across settings to promote the greatest public health impact”. Ce-PIM Workgroup on Scaling Out, unpublished, 2014

39. Scaling Out Perspective on Implementation Research Scale Up Sustain Implement Generalized knowledge Adopt/Prepare Explore NE rural Baltimore NE urban Houston Local knowledge Scale Out Across Diverse Contexts

40. AIR’S mission is to conduct and apply the best behavioral social science research and evaluation towards improving peoples’ lives, with a special emphasis on the disadvantage.

41. Chambers et al, 2013 Implementation Science

42. Case Study: Good Behavior Game at AIR Houston ‘08- present Baltimore ’83-’07 Nebraska rural ’13- present Nebraska urban ’11- present American Institutes for Research JHSPH Implementation Science Development grant: Develop and pilot web-based tools to deliver training to teachers Under review: Develop and pilot web-based tools for coaching and on-going support to teachers Prevention Science into…. Johns Hopkins: 1983-present RCT 1: GBG | ML RCT 2: GBG + curricula | family AIR: 2003-2007 RCT 3: GBG + reading + family Develop model of training and support for GBG Implementation Science RCT 1: -GBG: 2 models of PD -Implement/Sustain -”Adaptation” Service contract to develop local GBG capacity: Face-to-face training and support Implementation Science Service contract: Face-to-face training and support Under review: GBG for Middle School Domitrovich et al., 2008; Poduska et al., 2009; Poduska et al., 2012; Poduska & Kurki, 2014

43. Scaling Out Perspective on Implementation Research Scale Up Sustain Implement Generalized knowledge Adopt/Prepare Explore NE rural Baltimore NE urban Houston Local knowledge Scale Out Across Diverse Contexts

44. Who needs to be at table at AIR? Early Childhood Quantitative Analysts Legal Data Use Contracts Mental Health Services Professional Development Colleagues: Researchers Practitioners Middle School Financial Adolescence Leadership: CEO/President Board of Directors Dir, Ed Program Qualitative Analysts Ed Tech Social Emotional Learning Publication Services Human Resources Communications ? ? Web Services Institutional Review Board (IRB) Information Technology (IT) ??

45. Additional Systems Concepts • Systems have developmental trajectories • Systems are adaptive, fluid • Adaptation occurs locally • A note about silos: people may know one another, talk with one another and not be working towards a common goal

46. 4. Innovations: Activities of Ce-PIM Juan Villamar, MS.Ed Gracelyn Cruden, M.A. C Hendricks Brown, Ph.D. Center for Prevention Implementation Methodology Department of Psychiatry and Behavioral Sciences Northwestern University

47. Broker, Intermediary Implementation Agency or Agencies Target 7. Monitoring Large-Scale Implementation Developer/ Purveyor 2. Delivery medium Funder / Oversight 1. Person, Place, and Time, Reach, Engagement 4. Training, fidelity monitoring and feedback 6. Intervention R&D 3. Simplify and support intervention delivery 5. Improve efficiency and effectiveness of implementation delivery 8. Modeling and instituting implementation and oversight policies General Model to Support Implementation of Evidence Based Intervention Brown et al, 2013 JAIDS

48. Using a mobile application to improve fidelity of the GBG – Rapid Development of a Prototype

49. Visualization Tool – Classroom Disruptiveness Comparison Model 2 classrooms, 24 students Chance that students will act disruptively and their disruptiveness will spread.

50. Major Gap – Sustainable Partnerships • Partnerships between content experts and technology developers “What do you recommend we use?” VS “Tell me what you want to see in a report” • Bring to the table those players that will interact with the report FIRST • Partnerships between Research and Practice Iterative testing with feedback loops • Technology System, Educational System, Public Health System Integrated System?