Designing & Implementing Randomized Controlled Trials For Community-Based Psychosocial Interventions

1. Designing & Implementing Randomized Controlled Trials For Community-Based Psychosocial Interventions Phyllis Solomon, Ph.D. Professor School of Social Policy & Practice University of Pennsylvania March 17, 2010

2. Overview of Workshop • Introduction • So you think you want to do an RCT? • RCT Ethical Considerations • Planning an RCT • NIH Exploratory Research Grants • Developing Conceptual Foundation • Designing an RCT • Implementing an RCT • Generalizing RCT Outcomes

3. Introduction

4. What is an RCT? • True experimental design. Participants assigned by chance, following consent, to one of at least two conditions • Key features of classic experimental design: • Random assignment • determines who assigned to which group • Pre & post tests • outcome measured before & after intervention • Control group • same experiences as experimental group except no exposure to experimental stimulus

5. What is an RCT? • Can have more than two groups • Sometimes no pre-test measures • Chance does not necessarily mean equal, but known probability

6. Community-Based Psychosocial Interventions • Psychosocial Intervention – any service, program, educational curriculum, or workshop whose goal is to produce positive outcomes for individuals confronted with social &/or behavioral issues & challenges • Community-Based -Conducted in agency & social work settings

7. Community-Based Psychosocial Interventions • Community-based psychosocial intervention – reflects impact of environmental context in which interventions are imbedded, on clients and providers & interactions between both/all systems • Less control, more complex environmental context with participants with multiple problems

8. RCT vs. Evaluation • Research uses scientific methodology to generate generalizable knowledge • Evaluation uses same methodology but primary goal is not for generalizable knowledge • For NIH grants do not use term evaluation

9. RCT vs. Evaluation • In evaluation RCT known as experimental study or a randomized field experiment • Both examine a program or policy • Both addresses effectiveness & cost effectiveness • Evaluation experimental studies closely resemble community-based psychosocial RCTs • literature in this area may be helpful

10. RCT vs. Evaluation • Purpose of RCTs & field experiments may differ • RCT – research –generalized knowledge • Field experiments – evaluation – answer local questions – but also policy questions of broader application • Semantic difference

11. Psychosocial Community-Based Interventions = Effectiveness Studies • Efficacy studies occur under ideal or optimum conditions • Effectiveness studies occur in “real world” • Efficacy studies greater internal validity • Effectiveness studies greater external validity

12. So You Think You Want To Do An RCT?

13. Appraising Whether to Move Forward with an RCT Preliminary questions to be addressed before moving forward: • Is the question well justified? • Is the question an important one to answer? • Is the question addressing a gap in the literature? • Is the question an ethical one? • Is the question posing the correct question? • Would you fund this RCT?

14. Appraising Whether to Move Forward with Your RCT Case Example 1 • Is a 90 day Advanced Practice Nurse-Transitional Care Model more effective than usual discharge in improving adherence to treatment & quality of life for persons with SMI being released from a psychiatric hospital? • “hand-off” from hospital to home of SMI linked to gaps in delivery of MH services • Consequently high rates of rehosp & poor outcomes • EBP – Advanced Practice Nurse-Transitional Care Model improves outcomes following acute medical care discharge for elderly adults with complex medical problems

15. Appraising Whether to Move Forward with RCT Case Example 1(continued) - Intervention hybrid of case management, disease management, & home health care - Nurse works with hospital team to develop discharge plan & then implement in the community - Believe adapting this intervention potential to be equally successful with adults with SMI being discharge from acute hospital

16. Appraising Whether to Move Forward with RCT Case Example 2 • Is Multidimensional Treatment Foster Care (MTFC) Program more effective in reduction of disruptive behaviors than traditional Therapeutic Foster Care (TFC) among children in foster care? - Instability in foster care placement ranges from 22%-56% - Instability in placement due to child’s disruptive behaviors - TFC typically used for children with more demanding emotional & behavior needs & has more intensive structure & MH services

17. Appraising Whether to Move Forward with RCT Case Example 2 (continued) - Data on disruptions for TFC sparse but estimated 38%-70% - Limited evidence on TFC effectiveness – most studies descriptive, methodologically flawed - Lack of clear standards & specification of actual implementation of TFC -MTFC – manualized intervention with goals to improve well-being & reduce disruptions - MTFC placement augmented with coordinating an array of clinical interventions in family, school, & peer group

18. Appraising Whether to Move Forward with RCT Case Example 3 Is CBT for adolescents with sickle cell disease (SCD) more effective than medical management of the disease in increasing coping strategies? - adolescents with SCD have a number of adjustment difficulties that have received little attention - some psychosocial difficulties include stress-processing e.g. decreased coping strategies, lack of knowledge of SCD - need to promote biological & psychosocial adjustment

19. Appraising Whether to Move Forward with RCT Case Example 4 • Is Forensic Assertive Community Treatment (FACT) more effective than forensic intensive case management (FICM) in a variety of psychosocial and clinical outcomes for homeless adults with SMI leaving jail? - Pop. has multiplicity of needs due to mental illness, homelessness, & criminal justice involvement

20. Appraising Whether to Move Forward with RCT Case Example 4(continued) - cognitive deficits & poor social skills complicate ability to coordinate efforts to meet needs - FICM single point of planning, monitoring & accountability considered beneficial for this pop. - FICM specialized ICM - FACT –team approach (shared caseload), self contained intervention to meet all needs of client – includes psychiatrist, case managers, etc. - Based on ACT for criminally involved

21. RCT Ethical Considerations

22. RCT Ethical Considerations • Appropriate question to ask • Who ethically eligible to randomize • What ethical comparison • How & when to randomize • When are providers human subjects • What is ethical responsibility at termination

23. Justifying the RCT to Doubters • Want to provide most effective services to clients • Expectation when treated by a doctor • RCTs best means to making causal inference with high degree of confidence • Unethical to offer untested intervention • Not denying better treatment to controls • if answer known, there would be no need for study • Frequently those who receive services determined on a haphazard or a biased basis

24. Ethical Justification For Randomization • Lack of adequate evidence of effectiveness of exp. intervention understudy • Experimental intervention theoretically justified to potentially benefit target pop. • Uncertainty of effectiveness (equipoise) – otherwise no scientific basis for RCT

25. Principle of Equipoise • Substantial degree of uncertainty / ambiguity necessary • Specific population • Setting

26. Integration of Practice & Research Ethics • Practice – interventions designed solely to enhance well-being of client & has reasonable chance of success (Belmont Report, 1979) • Research – activities designed to test hypothesis, permit conclusions to be drawn, thereby contribute to generalized knowledge (Belmont Report, 1979) • RCTs = Practice & Research

27. Integration of Practice & Research Ethics • Practice ethics = human subject protections – may conflict w/ scientific rigor • Participant deterioration in experimental condition results in biased attrition • Exclusion criteria for clinical reasons – reduce external validity

28. Ethics of Scientific Untested Interventions • Experimental intervention at least as effective as TAU • Do no harm - even if voluntarily consents • Risks assessment for participant • Extends to others & community-at-large

29. Ethics of Selecting Control Group • Justify no service comparison • Gas to no gas • Waitlist may be justified if agency normally has waiting list, or no service offered • Inert intervention may be justified • TAU may be most justifiable comparison

30. Consent Forms • Must inform potential participant that will receive experimental intervention by chance • i.e., like flipping a coin • Indicate chance of receiving experimental intervention • equal chance or 1 out of 3 chance • People grasp natural frequencies rather than probabilities

31. Consent Forms • Describe all interventions • Merely saying ‘standard care’ not helpful • Remember need to provide reasonable information to make a decision • Dishonest to promise benefit – uncertainty justification for study • Need to ensure non-participation will not jeopardize usual services to which entitled

32. When to Gain Consent • Gain consent prior to random assignment • Unethical to indicate allocate by chance when already assigned • If assigned prior to consent, require two separate consent forms • Allocation prior to consent - result in biased attrition

33. Multiple Consent Forms • Screening for eligibility may require consent form • Children require assent & possible multiple consents • Process assessments may require consents from family members, providers etc

34. RCT Providers • Consents for providers – When are consent forms needed? • Need for Federal-Wide Assurance

35. Incentive Payments to Participants • Negotiate payments with agencies • Clients • Providers • Types of payments

36. Responsibilities at Termination of RCT • Provision for ongoing care of participants • Experimental service to control condition • Feedback & dissemination to agency

37. Data Safety & Monitoring • NIH require Board for RCT oversight • Often 3-4 members – meet quarterly in person or via phone • Report adverse events – also to IRB • Review of adverse events

38. Considerations for Internet RCT • Consents handled either by mail or via Internet • Monitored or unmonitored interventions • Are internet communities public or private spaces? • Consent forms – need to specify potential risks due to internet

39. Planning An RCT

40. Determining Whether to Undertake an RCT • Selecting a site • Pipeline of available & willing eligible participants • Setting prepared & willing to commit & support RCT • Financially, space, & supervision • Others willing to financially support • Sustainability of effective intervention

41. Negotiating with the Setting • Top down & bottom up approach • Honesty in negotiating • “You’ll hardly know we are here” • Collaborative partnership

42. REAL SCORE • Respect for providers & clients • Establish credibility • Acknowledge strengths • Low burden • Shared ownership – reciprocity • Collaborative relationship • Offer incentives – be responsive & appreciative • Recognize environmental strengths • Ensure trust – be sure providers feel heard

43. Feasibility & Pilot Studies • Worthiness, practicality, feasibility & acceptability of intervention • Modification of intervention for new population • Pilot testing recruitment, retention, & data collection • Estimate required sample size

44. Defining Treatment / Program Manuals • Specifies: • Intervention • Standards for evaluating adherence • Guidance for training • Quality assurance & monitoring standards • Facilitation or replication • Stimulates dissemination & replication (Carroll & Rounsaville, 2008)

45. Treatment / Program Manuals • Brief literature review • Guidelines for establishing therapeutic relationship • Defining & specifying intervention • Contrast to other approaches • Specific techniques & content • Suggestions for sequencing activities (Carroll & Rounsville, 2008)

46. Treatment / Program Manuals • Suggestions for dealing with specific problems • Implementation issues • Termination issues • Qualifications of providers • Training providers • Supervising of providers (Carroll & Rounsville, 2008)

47. Treatment / Program Manuals Deal with structural aspects - Caseload - Staff qualifications - Location/setting e.g., space - Integration into service setting (Carroll & Rounsville, 2008)

48. Criticisms of Treatment Manuals • Limited application to diversified population with complex problems • Overemphasis on specific techniques – rather than competencies • Focus on technique rather than theory • Reduction of provider competence • Lack of applicability to diverse providers • Designed for highly motivated & single problem clients

49. Adapting Existing Manuals • Use of qualitative methods • Focus groups • In-depth interviews • Group processes – nominal group process, Delphi method, & concept mapping • Ethnographic methods

50. Fidelity Assessment • Determining whether the intervention was conducted as planned and is consistent with service or program elements delineated in manual, including structures & goals • Fidelity measure – scale or tool assessing adequacy of implementation of service or program - means to quantify degree to which program service elements or services are implemented