UNPACKING THE BLACK BOX ENGINEERING MORE POTENT BEHAVIORAL INTERVENTIONS TO IMPROVE PUBLIC HEALTH

1. UNPACKINGTHEBLACKBOXENGINEERING MORE POTENT BEHAVIORALINTERVENTIONS TO IMPROVE PUBLIC HEALTH Linda M. Collins, Ph.D.

2. Outline • What is a behavioral intervention? • What is the black box, and why unpack it? • What is optimization? • A few examples • Concluding remarks

3. What is a behavioral intervention? • Definition: A program aimed at modifying behavior for the purpose of preventing/treating disease, promoting health, and/or enhancing well-being.

4. What is a behavioral intervention? A few examples from PSU (out of many): • PROSPER (Greenberg, Bierman, Feinberg, Welsh, Perkins, Mincemoyer, Corbin) • Keepin’ it REAL (Hecht, Miller-Day, Graham) • Project ACT (Turrisi) • Active MOMS (Downs) • Interventions for at-risk caregivers (Zarit) • Head Start REDI (Bierman, Domitrovich) • HealthWise (Caldwell, Smith) • RESERVE (Kolanowski, Fick)

5. What is a behavioral intervention? • Most behavioral interventions are made up of multiple components. • Some components may be pharmaceutical or medical.

6. Outline • What is a behavioral intervention? • What is the black box, and why unpack it? • What is optimization? • A few examples • Concluding remarks

7. Treatment package approach component component component component component RCT

8. What is the black box, and why unpack it? If RCT finds significant effect, it is UNKNOWN • Which components are making positive contribution to overall effect • Whether all the components are really needed • Whether a component’s contribution offsets its cost • How to make the intervention more effective, efficient, cost-effective

9. What is the black box, and why unpack it? If RCT finds non-significant effect, it is UNKNOWN • Whether any components in the box are worth retaining • Whether one component in the box had a negative effect that offset the positive effect of others • Specifically what went wrong and how to do it better the next time

10. What is the black box, and why unpack it? The treatment package approach • Encourages stuffing the black box with as many components as possible to get a significant effect • Downplays considerations such as efficiency, cost-effectiveness, time-effectiveness • Places focus on attaining statistical significance rather than meeting a criterion

11. What is the black box, and why unpack it? • This is NOT how engineers build products. They take an approach that is • Systematic • Efficient • Focused on the clear objective of optimizing the product • Can we borrow ideas from engineering… • … and build optimized behavioral interventions?

12. Resource management principle • How engineers think: • This is what I need to find out: ______ • These are the resources I have: ______ • How can I manage my resources strategically to find out what I need to know?

13. Resource management principle • Logic: Objective is to identify ONE OF THE TWO OR THREE BEST approaches

15. Resource management principle • Logic: Objective is to identify ONE OF THE TWO OR THREE BEST approaches • Manage research resources strategically • Decide what information most important, and target resources there • Choose efficient experimental designs • Take calculated risks

16. Resource management principle • Note that the starting point is the resources you have • By definition, MOST does not require an increase in research resources • But may require a realignment of research resources

17. The Multiphase Optimization Strategy (MOST) component component Note: MOST is a framework, not an off-the-shelf procedure. component component component Optimized intervention Screening (component selection) and refinement experiment(s) component RCT component component

18. Outline • What is a behavioral intervention? • What is the black box, and why unpack it? • What is optimization? • A few examples • Concluding remarks

19. Definition of optimized • “The best possible solution… subject to given constraints” [emphasis added] (The Concise Oxford Dictionary of Mathematics) • Optimized does not mean best in an absolute or ideal sense • Instead, realistic because it includes constraints • Optimization always involves a clearly stated optimization criterion • A working definition of what YOU mean by “better”

20. Selecting an optimization criterion: what you mean by “best” • Your definition of “best possible, given constraints” • This is the goal you want to achieve • Suppose you are developing a behavioral intervention to encourage HAART adherence in HIV+ people called “Living with HIV”

21. One possible optimization criterion: • Intervention with no “dead wood” • Example: Health care settings are finding it difficult to fit Living with HIV into their busy day, and are watching costs carefully. The investigators want to be confident that every component is necessary so that no time or money is wasted. • Achieve this by selecting only active intervention components.

22. Another possible optimization criterion: • Most effective intervention that can be delivered for ≤ some $$ • Example: To have a realistic chance of being adopted by HMOs, Living with HIV must cost no more than $200/participant to deliver, including materials and staff time. • Achieve this by selecting set of components that represents the most effective intervention that can be delivered for ≤ $200.

23. Another possible optimization criterion: • Most effective intervention that can be delivered in ≤ some amount of time • Example: Interviews with health care clinic staff suggest that Living with HIV has the best chance of being implemented well if it takes no more than 15 minutes to deliver. • Achieve this by selecting set of components that represents the most effective intervention that can be delivered in ≤ 15 minutes.

25. Outline • What is a behavioral intervention? • What is the black box, and why unpack it? • What is optimization? • A few examples • Concluding remarks

26. Three examples of MOST • All currently in the field • All three have the same optimization criterion: No inactive components

27. Example 1: Clinic-based smoking cessation study funded by NCI Tim Baker Mike Fiore University of Wisconsin Team also includes B.A. Christiansen, L.M. Collins, J.W. Cook, D.E. Jorenby, R.J. Mermelstein, M.E. Piper, T.R. Schlam, S.S. Smith Project funded by NCI grant P50CA143188

28. Example 1: Clinic-based smoking cessation study funded by NCI Tim Baker Mike Fiore University of Wisconsin Objective: Develop an effective “lean” clinic-based smoking cessation intervention (no inactive components)

29. Some interesting features of Example 1 • Study being implemented in health care settings • Involves both behavioral and pharmaceutical components

30. Baker & Fiore’s model of the smoking cessation process MOTIVATION PRECESSATION (3 weeks prior up to quit day) CESSATION (quit day to 2 weeks after quit day) MAINTENANCE (2 weeks to 6 months after quit day)

31. Six components being considered for the smoking cessation intervention • Precessation nicotine patch (No, Yes) • Precessation ad lib nicotine gum (No, Yes) • Precessation in-person counseling (No, Yes) • Cessation in-person counseling (Minimal, Intensive) • Cessation phone counseling (Minimal, Intensive) • Maintenance medication duration (Short, Long)

32. Experiment to examine individual component effects • We decided to conduct a factorial experiment • Special type called a fractional factorial • N=512 subjects TOTAL provides power ≥ .8

33. Engineering the intervention • Experiment will give us • Main effect of each individual intervention component on outcomes • e.g. number days abstinent in 2-wk post-quit period • Selected interactions between intervention components • This information will be used to select components/component levels • Result: optimized clinic-based smoking cessation intervention • Plan to conduct an RCT to establish statistical significance of effect

34. Example 2: School-based drug abuse/HIV prevention study funded by NIDA Linda L. Caldwell Edward A. Smith Penn State Team also includes D. Coffman, L. Collins, J. Cox, I. Evans, J. Graham, M. Greenberg, J. Jacobs, D. Jones, M. Lai, C. Matthews, R. Spoth, L. Wegner, T. Vergnani, E. Weybright Project funded by NIDA grant R01DA029084

35. Example 2: School-based drug abuse/HIV prevention study funded by NIDA Linda L. Caldwell Edward A. Smith Penn State Objective: To develop a strategy for maintaining implementation fidelity in which all components contribute

36. Some interesting features of Example 2 • Components being examined relate to how the intervention is delivered • “sealed intervention” • Cluster randomization

37. Background • HealthWise school-based ATOD/HIV prevention intervention • Has previously been evaluated in South Africa • Metropolitan South Education District in South Africa wants to implement HealthWise in all its schools • Question: how to maintain fidelity? • Metro South allowed us to conduct an experiment

38. Caldwell & Smith’s model of implementation fidelity

39. Components • Enhanced teacher training • Standard training (one and one-half days) vs. enhanced (three days + two additional days four months later) • Structure, support, and supervision • No additional vs. additional (e.g., weekly text messages; monthly visits from support staff; option to call support staff with questions as needed) • Enhanced school climate • No climate enhancement vs. climate enhancement (e.g., form committee of parents and teachers to promote HealthWise; develop visuals; issue newsletter)

40. How to conduct an experiment to examine individual component effects • We decided to conduct a factorial experiment. Why? • Enables examination of individual component effects AND • Statistical power achieved with smallersample sizes than alternative designs • Yes, I mean smaller • BUT they also usually require more experimental conditions than we may be accustomed to • Experiment uses all 56 schools in district

41. Factorial experiments 101 • Example: 2 X 2, or 22, factorial design • Factorial experiments can have • ≥ 2 factors • ≥ 2 levels per factor

42. HealthWise experiment in South African school district. 56 schools in all; 7 schools assigned to each experimental condition

43. Are 7 schools per experimental condition enough? • We estimated power ≥ .8 for main effects • Remember that each main effect estimate is based on ALL schools • In a factorial experiment you DO NOT compare individual conditions

44. Main effect of Training is mean of (5,6,7,8) vs. mean of (1,2,3,4). Note that all 56 schools are used in estimating the main effect.

45. Main effect of Structure, support, & supervision is mean of (3,4,7,8) vs. mean of (1,2,5,6). Note that all 56 schools are used in estimating the main effect.

46. Main effect of Enhanced school climate is mean of (2.4.6.8) vs. mean of (1,3,5,7). Note that all 56 schools are used in estimating the main effect.

47. Engineering the intervention • Experiment will give us • Main effect of each individual intervention component on outcome variables • Also interactions between intervention components • This information will be used to select the best set of the three components • Result: Intervention engineered to optimize fidelity according to our criterion • Note: optimized ≠ the best possible

48. Example 3: Internet-delivered drug abuse prevention program aimed at NCAA athletes David Wyrick UNC Greensboro Prevention Strategies, LLC MelodieFearnow-Kenney Prevention Strategies, LLC • Team includes L. M. Collins, J. Milroy, K. Rulison • Project funded by NIDA grant R44DA023735

49. Example 3: Internet-delivered drug abuse prevention program aimed at NCAA athletes David Wyrick UNC Greensboro Prevention Strategies, LLC MelodieFearnow-Kenney Prevention Strategies, LLC • OBJECTIVE: No inactive components. Secondary aim: Maximize overall program effect within available research resources.

50. Some interesting features of Example 3 Intervention is internet-delivered Experimental design is a cluster-randomized fractional factorial Takes an iterative approach