Consumer-Operated Service Programs Results

1. Consumer-Operated Service Programs Results Jean Campbell, Ph.D. Greg Teague, Ph.D. E. Sally Rogers, Sc.D. Asya Lyass, M.S. Ph.D. (Cand.) FROM INNOVATIONS TO PRACTICE: THE PROMISE AND CHALLENGE OF ACHIEVINGRECOVERY FOR ALL Hyatt Regency Hotel in Cambridge, MA., on April 14, 2008

2. Background • Over the past three decades peer-run support services have • matured, • diversified, • increased in numbers across the United States

3. What is a COSP? • A Consumer–Operated Services Program is peer-run service program that is administratively controlled and operated by mental health consumers and emphasizes self-help as its operational approach. • Today, most COSPs operate as an adjunct to traditional mental health services within the continuum of community care.

4. COSP Service Models • Mutual Support Groups • Drop-in Centers • Education and Advocacy Programs • Multi-Service Agencies • Specialized Supportive Programs • Warm Lines

5. Number of COSP Services • SAMHSA estimates there are 7,467 groups, organizations, and services run by and for mental health consumers and/or families in the United States (2002) • 44.4% are mental health mutual support groups • 40.4% are mental health self-help organizations • 15.2% are consumer-operated service programs

6. COSP Effectiveness • Studies using non-randomized control groups or pretest scores as comparisons found that participation in peer support • Reduces psychiatric symptoms • Decreases hospitalization • Enlarges social networks • Enhances self-esteem and social functioning

7. COSP Effectiveness • Recent studies using randomized control groups and pretest scores as comparisons have found that participation in peer support promotes wellness (a sense of well-being). • Hope • Empowerment • Social connectedness • Meaning in life • Self-esteem/personhood

8. COSP Study Overview • Eight program sites • CA, CT, FL, IL, ME, MO, PA, TN • Three general program models • Drop-In (4 sites) • Mutual Support (2 sites) • Education/Advocacy (2 sites)

9. Participating Study Sites

10. COSP Study Overview • One-year longitudinal follow-up • 4 measurement points: 0, 4, 8, 12 months • Participants • Persons 18+ with diagnosable mental / behavioral / emotional disorder and functional impairment • N = 1827 enrolled in study; 1600+ in analysis • Common interview protocol • Logic Model • Conventional RCT approach • Intent-to-treat analysis • Optimized, common a priori hypothesis

11. Primary Hypothesis • Participants offered both traditional and consumer-operated services would show greater improvement in well-being over time than participants offered only traditional mental health services.

12. Well-Being Outcome Rationale • To develop a measure that was supported by theory and peer literature • Hypothesized to be most sensitive to primary peer support program effect • Short term outcome • Realization of participants that “We are not alone.”

13. Composite Well-being Measure: Constituent Scales • Scales incorporated in Well-being measure (WB2) • Total HerthHope Index • Meaning of Life Framework Subscale • Empowerment / Making Decisions (EMD) Self-esteem/ Self-efficacy Subscale • Recovery Assessment Scale (RAS) Goal Subscale • Recovery Assessment Scale (RAS) Hope Subscale • Internal consistency • Cronbach’salpha= .92 at all four measurement points

14. Results: Well-being Over Time by Random Assignment Group

15. Intent-To-Treat Results for Well-being – All Sites • Increase in well-being over time for participants overall • Possible differences in change in well-being over time across sites • Significantly greater increase in well-being for persons offered use of COSP (p < .05) overall • Possibly important differences in this effect across site

16. Well-being Effect Sizes By Site

17. Well-being Effect Sizes By Site: Between Conditions & Within COSP+

18. Well-being Effect Sizes By Site: Between and Within Conditions

19. Intent-To-Treat Results for Well-being – Seven Sites • Increase in Well-being over time for participants overall • No significant site difference in change in Well-being over time – very consistent positive ES for COSP across all 7 sites, along with positive trend for both groups • Significantly greater increase in Well-being for persons offered use of COSP (p < .01) • Negligible differences in this effect across site • Site variations in experimental effect for this outcome are driven by variations in TMHS programs

20. As-Treated Analysis • Engagement rates and adherence to assigned condition low • 57% of participants assigned to COS+ used COS • 15% of participants assigned to TMHS only used COS • Consequently, ITT results may be too conservative • Scores calculated for propensity to use COS • Analysis limited to middle third of propensity • Removed those who especially seek/avoid COS • Effects can be attributed to intervention, not selection • Smaller N means positive findings are conservative

21. As-Treated Analysis: Two Measures of Engagement in COSP • Binary measure of simple engagement – visited at least once vs. didn’t use at all • Three-point measure of intensity of use • No use • Low use = less than study median (8.5 visits to COSP over 12-month period) • High use = more than study median

22. Well-being Over Time By Engagement (Middle 1/3 Propensity)

23. Well-being By Intensity of COSP Use (Middle 1/3 Propensity)

24. COSP-Findings for Well-being • Significant gains in well-being were found for the group that was offered use of COSP (the formal experiment – intent-to-treat analysis) • Greater gains in well-being were found for the group of participants who actually used COSP services (as-treated analysis) • Greatest gains in well-being were found for the group of participants who used COSP the most, those in the upper half of frequency of use • Variations in effects across sites were unrelated to formal COSP type

25. Sub-Study: How is empowerment effected by involvement in consumer run programs • Why study empowerment? • Many programs espouse empowerment as both an aspect of the program structure (“empowering practices”) and as an outcome to be achieved • However, few rigorous studies of the effects of consumer run programs on empowerment exist • Most studies not randomized, small, descriptive • Complicating matters: there is no consensus of the definition of empowerment

26. Sub-Study: Empowerment • Making Decisions Empowerment (MDE), 28 item scale to measure subjective feelings of empowerment: self-efficacy, perceived power, optimism about and control over the future, and community activism • Personal Empowerment (PE), 20 item tool with 2 subscales: Choice and Reduction in Chance • Organizational Empowerment (OME), 17 item scale about involvement in an community, organization or club

27. Purpose • Determine whether COSP sites varied with respect to differences between E & C or change over time • Describe pattern in change over time • Examine whether “engagement intensity” (binary or no, low, high use) resulted in greater changes in empowerment

28. Intent to Treat Results • Results using strict (ITT) approach yielded marginal results--perhaps due to modest engagement in the COSP • Results of ITT analyses with both measures of personal empowerment had overall results below threshold for significance and very small effect sizes • As with well-being analyses, one site obscured the more positive results from remaining sites. • Without this site, both measures showed a significant, small and positive effects • There was still significant cross-site variation remaining for PE Choice

29. Analysis of differences in slopes over time

31. Differences between groups over time-personal empowerment • Sample items: “I can pretty much determine what will happen in my life” “People working together can have an effect on their community” • Half of the sites showed a positive difference between E and C groups in personal empowerment (MD) over time with an effect size of .2 or more • One site had a significant negative difference in change over time between E and C • One site had slight negative difference; remaining had no difference • Conclusion: positive effect on personal empowerment from participation in COSP

33. Differences between groups over time- PE-Choice • Sample item: “How much choice do you have about how you spend your free time” • Three sites had positive difference in change over time between E and C groups with effect sizes above .2 • Five sites had a negative difference in change over time, or no change • Conclusion: Some COSP affect perception of choice

35. Differences between groups over time-PE-Reduction in Chance • Sample item: “How likely is it that you will get enough to eat in the next month” • Two sites had positive differences in change over time between E and C • Six sites had no positive differences in change over time between E and C, or negative differences • Conclusion: Perception of life being left to chance or control over life not affected by COSP

36. Organizational Empowerment • There was a general downward trend in organizational empowerment but significant variation across sites • There was less decline among COSP programs than traditional programs • Data raises questions about the fit of construct “organizational empowerment” with COSP programs • Conclusion: Relatively few participants (in either condition) experienced an increase in the number of organizational roles or activities tapped in this measure.

37. As treated analyses • Needed to go beyond ITT analyses because of modest engagement in the COSP for experimental folks • Using a procedure described earlier and propensity scores, created two groups of balanced individuals • Individuals in both groups showed equal tendency to attend COSP • Yielded a better comparison of E and C participants

38. Results Results of the as-treated analyses confirmed the inferences that emerged from the ITT analyses: Use of consumer-operated services was positively associated with increases in personal empowerment as measured by both the MD and the PE Choice scales, results that held without significant variation across the eight sites. Gains from using COSPs become apparent only with higher levels of use.

39. Conclusions • Results support the conclusion that COSPs in general have a positive impact on aspects of empowerment (MD and PE Choice scales) • Some COSPs have the effect of improving empowerment, while others are less effective • Effects significant but small in magnitude and consistent with other studies

40. Conclusions • Individuals with greater engagement in and attendance at COSPs fared better in their personal empowerment outcomes • Additional analyses suggested positive changes in self-efficacy items: “I believe I am a person of worth”; “I am usually confident about the decisions I make”. • COSP may have more of an effect on self-efficacy than other aspects of personal empowerment

41. What is Fidelity? • Fidelity measures provide an objective rating system to assess the extent that components of a program are faithfully implemented according to intended program model, theory, or philosophy.

42. Why Measure Fidelity? • The use of fidelity measures has become a widely accepted methodological tool in mental health services research and serves a number of important purposes.

43. Why Measure Fidelity? • In addition to establishing that a set of well-defined services leads to predicted outcomes, it is critical for researchers to establish the integrity of the service delivery. • Providers need to adhere to critical elements of an evidence-based practice in order to achieve the positive outcomes identified in the original research.

44. COSP Fidelity Measurement • Common measurement, diverse programs • Analytic challenges • Some theoretically important program aspects were not being measured • Comparison conditions were highly variable • Initial program measurement goal • Develop cross-site program-level implementation measure • Assess interventions within traditional pooled data framework

45. Fidelity Assessment Common Ingredients Tool: FACIT • Process of Developing FACIT • Identification of common ingredients • Definition of common ingredients • Feasible performance indicators (48 items) • Performance anchors (typically 4-5) • Involvement of COSP directors and staff as well as researchers/evaluators at all stages of FACIT development process • Involvement of the CAP in the definition of common ingredients

46. FACIT Operationalization • Data collection • During site visits, questions devised to elicit information about common ingredients from program directors and program staff. • COSP recipient focus group • Program observation • While FACIT was developed to measure characteristics thought to be common to COSPs, it was also used to measure the extent to which TMHS were “consumer-friendly.”

47. FACIT Operationalization • Independent rating • After site visits, site visitors independently rated each program on each dimension. • Conciliation • Site visitors came to agreement on any dimensions on which there was disagreement.

48. FACIT Operationalization • Pilot testing of FACIT (Round 1 site visits) • Average interrater reliability of items .8; use of FACIT feasible with both COSP and Traditional Mental Health Services • FACIT Psychometrics • CIs present within COSPs • Differences between COSPs and TMHS detected • Differences among COSP models

49. FACIT Operationalization • COSP Fidelity Measurement (Round 2 site visits) • Analysis & further psychometrics • Factor analysis and internal consistency within major scales • Identification of subscales for use in fidelity-outcome analyses

50. STRUCTURE Consumer Ownership Responsiveness ENVIRONMENT Inclusion Accessibility BELIEF SYSTEMS Peer Ideology Choice & Respect Spirituality & Accountability PEER SUPPORT Encouragement Self-Expression EDUCATION ADVOCACY FACIT Scales and SubscalesUsed in COSP Study