Emily Ozer, Ph.D. University of California, Berkeley School of Public Health eozer@berkeley

1. Adaptation and Fidelity In the Implementation of Evidence-Based Programs Emily Ozer, Ph.D. University of California, Berkeley School of Public Health eozer@berkeley.edu

2. For Today • Give overview of research program and motivation for it • Describe recent study relevant to violence prevention • Answer any questions re career development, methods, etc • PLEASE “Interrupt”

3. Roots of Research Interests • Human rights and community-based work • India - Community-based health promotion, research • Central America - Human rights, impact of violence • California – Democratic business with Latina immigrant women and families • Informs research on: • Complex interventions in schools and communities -- linking process, context, and outcomes • Individual and collective “resilience” to range of stressors – violence, extreme poverty

4. Research on Factors Promoting Mental and Physical Health • Protective factors for psychological problems and aggression in adolescents who have experienced community violence (e.g. Ozer & Weinstein, 2004, Journal of Clinical Child and Adol Psych; Ozer, 2005 J. of Adolescent Research) • Perpetration of violence in dating and peer contexts among Mexican and European American adolescents (Ozer, et al., 2005, J of Adolescent Health) • Risk and protective factors for PTSD and other stress-related problems across the lifespan • Meta-analysis of risk factors for PTSD (Ozer et al. Psych Bull 2004) • Child mental health in context of domestic violence (Lieberman, VanHorn, & Ozer, Development and Psychopathology 2006) • Oportunidades: Impact on child behavior problems (Ozer et al. Pediatrics, 2009) • Resilience-oriented; factors amenable to intervention*

5. Population Health: School-Based Interventions • Peer-led HIV prevention: How peer educator-student relationships and classroom context predict program outcomes (Ozer, Maslach, Weinstein, et al. 1996, American Journal of Community Psych) • Contextual factors that influence effectiveness of school-based violence prevention (e.g. Ozer, 2006, J of Primary Prevention) • School gardens as a model for strengthening school setting and improving child/adolescent health (Ozer, 2006, Health Educ and Behavior) • Sources of school connection – qual follow up to exposure to violence study (Ozer, 2008, Journal of Adolescent Research).

6. Community-Based Interventions: Focus on Urban Public Schools • Excellent way to reach diverse youth and families from under-served communities • Critical setting for youth development • Strengthen resources in setting through research • Ultimate goal of school-level change

7. Community-Based Interventions: Opening up the “Black Box” • Beyond overall effects, fine-grained intervention research: • For whom and under what conditions is the intervention more effective? • Specify theory of change • Operationalize intervention processes and contexts expected to promote change • Link intervention processes and outcomes • Implications for theory and policy

8. Overview of Current Research • Two related studies that develop and test models with adolescents as: • “Agents” and resources in secondary school settings • Collaborators in efforts to improve adolescent health and wellbeing • Bridging participatory research and traditional intervention research – longstanding interest

9. Overview of WT Grant Scholars’ Award Research • Developing and testing models of teens as collaborators in urban public schools • Youth-led participatory action research • Adaption/fidelity of evidence-based programs • What happens when diffused? Are teacher and student adaptations consistent with program theory? • Ozer et al. in press Prev. Science • “Stretch” Areas • Mixed methods, qual, quant

10. Adaptation/Fidelity Study Rationale • Competing pressures in diffusion of evidence-based prevention programs • High-fidelity implementation of programs • Involvement of local stakeholders, community boards • Little study of integration of local expertise and evidence-based programs

11. Central Questions and GoalsInitial Phase of Research • How programs “fit” local settings when diffused? • Eliciting views of local stakeholders, especially youth, on fit with norms, practices in schools, communities • Characterize if suggestions support core components or not • Longer term goal: Test added value of collaborative implementation

12. Specific Questions Addressed Today • What changes do teachers make when implementing evidence-based prevention programs, and why? • What kinds of adaptations do teachers vs. youth suggest regarding evidence-based prevention programs? • Are these suggested changes consistent with program theory?

13. Site and Teacher Selection • Two urban school districts – Oakland, SF • Diverse student populations (ethnicity, immigration status, and socio-economic status) • “Higher end” of implementation spectrum • Roll-out of evidence-based programs in high schools • Understudied; few programs; likely more adaptation • Collaborative process developmentally-appropriate • District nominated health teachers likely to do high-quality job – make curriculum focus

14. Prevention Programs • Project Towards No Drug Abuse (TND) • 12-session curriculum • Theoretical model focusing on • “Motivation-skills-decision-making” • Building social and self-control skills • Strengthening rational decision-making processes • Too Good for Drugs and Violence (TGDV) • 14-session curriculum • Theory of change drawing from range of theories, e.g. • Social norms • Social cognitive theory • Social development model

15. Multiple Sources of Data • Classroom Observations (163) • Majority of sessions observed in each class • Student Group Interviews (25 interviews) • Teacher Interviews (22 formal) • Report of any adaptations made to curriculum • Rationale for making any adaptations • Suggested adaptations to the curriculum

16. Data Sources: Program Developer Consultations • Asked program developers to characterize each suggested change: • How consistent with core program theory and teaching approaches? • Integrated these responses into qualitative dataset of suggested adaptations

17. Qualitative Analysis • Coded by at least two members of the research team • Discrepancies resolved by discussion in weekly meetings and revision of the codebook • Codes entered in Nvivo qualitative software program • Observed Adaptations: Field notes and teacher interviews coded for observed or reported adaptations • Suggested Adaptations: Interviews with teachers and students coded for adaptation suggestions

18. Sensitizing Concepts Based on Prior Conceptual Frameworks • Frameworks for cultural tailoring(e.g. Resnicow; Kreuter) • Surface/peripheral • Deep/socio-cultural • Evidential • Others emerged from collected data • Content, real life, resources, teaching approach

19. Coding of Feedback From Program Developers • “Stoplight” categories (CDC, ETR) • Theoretical, logistical “stoplight” code for each suggested adaptation • Green, yellow, red

20. Overview of Results • Analysis of changes made by teachers • Analysis of teachers’ and students’ suggested adaptations to curriculum • What types of changes suggested? • Consistency between teachers and students? • Acceptability to program developers?

21. Results: Teachers’ Observed Adaptations • What changes did teachers make? • Changes in instructional format (45%) • Integration of real-life experiences of the teacher or others into the curriculum (21%) • Supplementation with additional resources such those used before (16%) • And why? • Primarily pedagogical reasons • Experiential learning; addition of visual, kinesthetic materials • Classroom dynamics and management • E.g. “unsafe” to do stereotype activity in racially charged school • Time and sequencing • E.g. Already taught communication skills in health class

22. Results: Teachers’ ObservedAdaptations • Do teachers differ in adaptations in 1st vs. 2nd time teaching the curriculum? • Overall, same number of adaptations observed… • But 2nd time more adaptations in teaching and resources (χ2= 3.97, df = 1,89, p < .05; χ2= 7.25, df = 1,89, p < .01) • Parallels teachers’ comments about “following the script” the 1st time, then “making it their own.” • Tension between fidelity and effective teaching.

23. Results: Students’ Suggested Adaptations • Content: More information (29%) • e.g. “positive aspects of drug use” • Surface: Wording, appearance (26%) • e.g. “no one says ‘dweeb.’” • Teaching approach: (20%) • e.g. making the curriculum more interactive by including more games or group activities

24. Results: Teachers’ Suggested Adaptations: • Instructional practices such as enacting experiential activities and adding homework to deepen students’ reflection on topics (34%) • More information, particularly about different types of drugs and their effects (16%) • Surface aspects of the curriculum such as language and including more visuals (13%)

25. Acceptability of Teachers’ and Students’ Suggestions • Theoretical consistency: Equally likely to propose “green-light” changes to each program. • Logistical concerns: Suggestions equally likely to be judged acceptable or problematic by program developers.

26. Type of Adaptation by Acceptability: TND • Some students’ suggestions “red lighted” due to concerns about deviancy training or scare tactics • Outside speakers talk about addiction • More differentiated characters and trajectories • Some teachers’ suggestions were “red lighted” • Replaced empirically-tested activities with those of questionable effectiveness • e.g. Demo of drug effects • Detracted from main goals of session

27. Type of Adaptation by Acceptability: TGDV • Small number of suggestions “red-lighted” • Main concerns • Covering content re intervening in fight • Dropping of an activity • Any discussion that might deviate from “zero tolerance” of fighting and drug use • Students say you have to fight back to save face

28. Discussion and Next Steps: Actual Changes • All teachers made changes • Tended to make more interactive to address different learning styles • Contrast to prior observational study by Hansen • Concerns about knowledge transmission • State standards are not skills-based • Next steps: • Deeper qualitative inquiry into tensions across practice communities, meaning of fidelity • Engaging stakeholders in dialogue about meaning of fidelity (“scalpel vs. chainsaw”)

29. Discussion and Next Steps: Suggested Changes • Some differences in type of suggestions made by teachers versus students • But youth equally likely to suggest “green light” • Many surface suggestions could increase perceived relevance without sacrificing fidelity to program core • Next steps • Comparative evaluation of regular versus adapted • Program developers’ integration of changes into new versions of curricula

30. Youth-Led Participatory Research Youth participatory/action research in urban secondary schools What are core features and processes? Impact on youth participants? Impact on school settings? Extensive development Collaboration, capacity (Ozer et al. 2008, AJCP) Measures of key processes and outcomes (Ozer et al. in press, AJCP, Ozer & Schotland, in press, HEB)

31. What is Youth-Led Action Research? • Iterative process of inquiry and action led by youth and guided by adult allies: • Identify problems and questions • Conduct research (with training) to guide recommendations and solutions • Engage and advise decision-makers • Recent topics: Diversity, drop-out, stress, school lunch, teachers’ best practices for culturally responsive teaching, sexual health, dating relationships

32. YPAR Study: Within-School Multi-Method Design 4 diverse sites x 5 years: Randomized at Classroom Level Other Electives N= (240) Latino (25%) Asian Am (52%) African American (6%) “Peers” Youth-Led Action Research Elective Class N= (135) Latino (34%) Asian American (37%) African American (28%) “Peers” Mentoring Direct Service Elective Class N= (171) Latino (48%) Asian Am (42%) African American (8%) Pre/Post/FU Surveys, Classroom Observations, Interviews Pre/Post/FU Surveys (measure development) Pre/Post/FU Surveys, Classroom Observations, Interviews Figure 3. Youth Research Intervention: Study Design and Sample

33. Intervention Key Processes Youth-level Outcomes Teacher-student power-sharing Skills, efficacy in research, communication, advocacy Networking opportunities Positive ethnic identity, Sense of purpose PAR Classroom Group work Perceived school connection Positive class climate (e.g. engagement, student perspectives) Psychological and political empowerment Opportunities for skill development (e.g. research, advocacy) Expanded social networks and support Figure 1a. School-Based PAR: Youth-Level Effects Model

34. Settings Targeted School-Level Outcomes Alliances between students and adult staff Classroom Expansion of meaningful student roles -school policies and practices PAR School Student-adult inquiry and learning Figure 1b. School-Based PAR: School-Level Effects Model

35. Sample Thus Far • Baseline: 24 experimental classes; 23 experimental control • Challenges of small class sizes, attrition, and semester vs. year long structure • BL: experimental=135, exp control=171 • 3 mos: exp=130, exp control=154 • 22 quasi-experimental control classes

36. Individual-Level Analyses • Cluster-randomized structure • Challenges • Likely different effects across sites • Small samples • Different expert advice, “cultures” • Current analyses in STATA with robust SE adjusted for clustering

37. Individual Preliminary Findings • In the midst of analyses…. • Higher dosage associated with: • Participatory behavior • Motivation to influence • Self-esteem

38. School Setting Analysis • Developmental mismatch in public secondary schools (e.g. Eccles, Midgley, et al.) • Recent attention to characteristics of settings that promote positive development • Ways of intervening to improve settings • YPAR as potential means to expand meaningful participation, efficacy, “mattering”

39. Design + Sample • Does YPAR process expand opportunities for influence, participation, “mattering?” • Definitely within the class. On level of school? • How experienced by participants? • Organize around critical incidents/interactions? • What conditions set the stage for new or expanded interactions and roles? • Which if any are sustained over time?

40. Mentors and Collaborators for YPAR Research Project Mentors: Tom Cook (Northwestern), Meredith Minkler (UCB), Larry Green (UCSF) School Collaborators: Elizabeth Hubbard, Gary Cruz, Adee Horn, Morgan Wallace Research team: Marieka Schotland, Laura Douglas, Sami Newlan, Dana Wright, Maggie Gaddis, Miranda Ritterman, Yolanda Anyon, Eddy Jara, Kathryn Clark, Huong Dao, Eric Koo, Becky Lee, Christopher Wu Funding: WT Grant Foundation, CDC