1 / 18

Measuring Consumer Perception of Care: Challenges and Opportunities in Substance Use Disorder Programs

This presentation by Dr. John Bartlett explores the nuances of measuring consumer perception of care within California's substance use disorder programs. It reviews key concepts, success criteria, and the development and testing of the CSAT Modular Survey. The discussion emphasizes the importance of meaningful, scientifically-sound, and actionable measures for comprehensibility and benchmarking. Insights include the differences between consumer perception and satisfaction, pilot testing phases, and the consensus-driven approach that supports effective outcomes in healthcare.

dyani
Télécharger la présentation

Measuring Consumer Perception of Care: Challenges and Opportunities in Substance Use Disorder Programs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Measuring Consumer Perception of Care Challenges & Opportunities John Bartlett, M.D.,M.P.H. March 20-21 Meeting with California’s Division of Alcohol and Drug Programs

  2. Goals for Presentation • Review consumer perception of care as a concept • Review success criteria for its measurement • Review the development & testing of CSAT’s Modular Survey

  3. The Concept • Measuring consumer response = core business function • In healthcare tied to growth of consumerism & CQI initiatives • A NOMS domain • Consumer perception of care ≠ satisfaction

  4. Measuring Consumer Perception of Care • Approaches to measurement differ depending on scope & purpose • For purposes of comparability, improvement over time, & benchmarking measures must be : • Meaningful • Scientifically-sound • Actionable

  5. The Problems with Satisfaction • No evidence linking the measurement of satisfaction to client outcomes • Few satisfaction surveys scientifically validated • Data is not actionable (ceiling effect)

  6. The Modular Survey • SAMHSA –supported initiative • Conducted under the auspices of the Forum on Performance Measurement & the Washington Circle • Conducted in 2 phases • Phase 1 in conjunction with mental health Rx • Phase 2 substance abuse – specific

  7. Design Requirements • Phase 1 focus on commonality, not comprehensiveness • Short • Scientifically sound • Actionable • Use of existing, widely-used, non-proprietary surveys • Consensus-driven

  8. Modular Survey Flow of Common Questions for Individual Respondent Common Design Template Adult Mental Health Core Measures Adult Common Measures Adult Substance Abuse Core Measures “Field-level” Common Measures C/Adol Mental Health Core Measures Child/ Adolescent Common Measures Adol Substance Abuse Core Measures

  9. Approach to Phase 1 • 4 workgroups to develop consensus • Selection of instruments • Identification of concerns • Identification of potential items • Ranking of items • Final item selection (modified Delphi) • Pilot testing

  10. Phase 1 Pilot Testing • Conducted during summer/fall 2004 • Primary data collection in Cincinnati United Way agencies (N = 1157) • Secondary analysis using MHSIP data (16 state & LA County data sets) • Final N > 22,000 respondents • Pool of items reduced from 28 to 11 • All items common to both fields, both populations

  11. Approach to Phase 2 • Stand-alone SUD Rx initiative • Under Washington Circle with Forum as “subcontractor” • New item development (no existing SUD survey) • Content work group co-chaired by Tom McLellan (TRI) & Doreen Cavanaugh (Georgetown) • Support from Forum Methods Work Group & Ann Doucette (George Washington • Public Provider & Consumer Advisory Groups

  12. Phase 2 SUD Initiative • Closely coordinated with NOMS • Identification of concerns • Relationship to treatment program • Self awareness of problem/commitment to change • Perceived outcomes • Social connectedness • Generation of items (35 in testing pool)

  13. Phase 2 Pilot Testing • OMB & IRB approval spring 2006 • Conducted in 3 rounds • Round 1 – Adult & Adolescent (summer 2006) • 14 programs, N = 1207 • Round 2 – Adult & Adolescent (winter 2006 – 07) • 6 programs, N = 585 • Round 3 – Adolescent (spring-summer 2007) • 8 programs, N = 268 • Final adult N = 1549 (2 samples) • Final adolescent N = 492 (1 sample) • All demographic groups covered except Native American

  14. Phase 2 Completion • Analysis & recommendations by Ann Doucette PH.D. • Use of IRT • For a copy of the technical report, e-mail jbartlett@avisagroup.com • Review by Forum Methods Work Group (November 2007) • Review & Approval by SUD Content Committee (November 2007)

  15. Final SUD Modular Survey • 21 items (11 from Phase 1, 10 from Phase 2) • Quality – 6 items • Perceived Outcomes – 6 items • Social Connectedness – 7 items • Commitment to Change – 2 items • 10 demographic & background items • Spanish translation available

  16. Modular Survey Flow of Common Questions for Individual Respondent Final Version Phase 2 Mental Health Items Phase 1 Common Items * Phase 2 SUD Items Common Design Template * All populations, all fields

  17. Convergence with NOMS • In Spring 2006 NOMS Technical Consulting Group convened • Recommended 17 items from 8 different instruments • 9 of the 17 from the Modular Survey • 5 are in the Final Modular Survey

  18. In summary…… • Consumer perception of care key measurement domain • Its measurement must meet certain criteria in order to be worth the effort • The Modular Survey is the only current instrument measuring consumer perception of care that is: • SUD Rx – specific • Product of both consensus and empirical analysis • Short and actionable

More Related