Measuring QI Intervention Implementation: Helping the Blind Men See?

1. Measuring QI Intervention Implementation: Helping the Blind Men See? Becky Yano, PhD, MSPH VA Greater Los Angeles HSR&D Center of Excellence UCLA School of Public Health

2. Overview of this Talk • Brief orientation to example QI interventions • How context matters  sets us up for variable QI intervention deployment • EBQI process for intentional adaptation of evidence into context of local practice • Types of data sources brought to bear on measuring implementation • Including development of a fidelity score • Triangulation of data sources to tell story

3. QI Intervention (QII) Examples • TIDES (Translating Interventions for Depression into Evidence-based Solutions) • Depression collaborative care model • Lisa Rubenstein, MD & Ed Chaney, PhD (Co-PIs) • QUITS (Quality Improvement Trial for Smoking cessation) • Evidence-based quality improvement approach to implementing smoking cessation guidelines • Scott Sherman, MD & Becky Yano, PhD (Co-PIs)

4. TIDES Depression Collaborative Care • Evidence base: • >20 RCTs • Depression • toolkit Provider/patient education Depression care manager EBQI QI Informatics support Performance feedback “adaptation” “priority-setting” Leadership support

5. QUITS Smoking Cessation Trial • Evidence base: • SC clinic referrals • Tobacco quitlines • PC-based intn’s Education “toolkit” Local QI plan development EBQI Expert review/feedback Performance feedback “local buy-in” “priority-setting” Leadership support

6. Context Matters: Design for It • TIDES • 2:1 intervention-to-control sites x 3 networks (6 intervention + 3 control sites total) • VA network leaders chose sites, we randomized within network (block on network characteristics) • QUITS • Regional concentration in southwest (3 networks) • Matched on size/academic affiliation within network • We chose sites and randomized within network

7. VISN MAP of TIDES and QUITS QUITS Sites in VISNs 18, 21, 22

8. Context Matters: Input from Sites • Attitudes/beliefs/experiences • Perceived need for the intervention • Competing demands • Staff open to innovation • PC-MH relationship (relevant to both TIDES and QUITS) • Resources • Perceived time to use program and participate in implementation • Organizational structure, staffing, prior QI experience, tools (access to informatics support) Source: Kirchner JE, Parker LE, Yano EM, COVES evaluation (2007).

9. Multiple Data Sources: Measuring Implementation

10. Measuring TIDES Implementation:Development of a Fidelity Score • Used semi-structured interviews • Network leaders, medical center leaders, providers, care managers, consumers (patients) • Mental health and primary care at all levels • 106 interviews in week-long site visits at each intervention site (2 interviewers/visit) • Audiotaped and professionally transcribed • Qualitatively analyzed in 5 phases (Atlas.ti) • 22 top level codes  top level coding + 20% re-review  subcodes  subcoding + 100% partner review  interpret Source: Kirchner JE, Parker LE, Yano EM, Ritchie MJ, COVES evaluation (2007).

11. Measuring TIDES Implementation:Development of a Fidelity Score • TIDES activity top level code • Subcodes for TIDES components • Provider education • Depression care management (DCM) • Patient education • Provider reminders/other informatics support • Performance feedback • Leadership support Source: Kirchner JE, Parker LE, Yano EM, Ritchie MJ, COVES evaluation (2007).

12. Measuring TIDES Implementation:Development of a Fidelity Score • Sub-subcodes in each TIDES component(eg, depression care management or DCM) • DCM/patient interaction content • DCM/patient interaction non-content • DCM/provider interaction content • DCM/provider interaction non-content • DCM supervisory issues • other roles and relationships of DCM • Two coders rated each site based on quotes Source: Kirchner JE, Parker LE, Yano EM, Ritchie MJ, COVES evaluation (2007).

13. TIDES Fidelity Score Example* *Examined awareness, mentions, how disseminated, penetration, consistency.

14. TIDES Fidelity Scoring • Level of implementation (high/medium/low) • Two coders re-reviewed all quotes • Considered awareness (y/n), mentions (y/n), how disseminated, penetration, consistency • 1 person saying something 10x ≠ 10 people saying 1x Yes, recall an email Yes, went to 5 sessions Yes, saw those LA folks Yes, rec’d guide Yes, amazing info!!! Yes, saw weblink and all No, never got training TIDES? What’s that? vs.

15. QUITS Organizational Site Surveys Source: Yano, Rubenstein, Farmer, et al., HSR, 2008, in press.

16. QUITS Administrative Data Source: Yano, Rubenstein, Farmer, et al., HSR, 2008, in press.

17. QUITS Patient Surveys * Screened >36,000 primary care patients to identify, enroll >2,000 current smokers. Source: Yano, Rubenstein, Farmer, et al., HSR, 2008, in press.

18. QUITS Practice Checklist • Smoking cessation expert review of: • Local QI plans (with feedback to practices) • Implementation activities • Completed practice checklist of intervention components • Evidence-based vs. non-evidenced based • Changes from QI plan to implementation

19. QUITS Post-Implementation Survey * Brief counseling program, computerized referral in PC or counselor/nurse hired in SC clinic Source: Yano, Rubenstein, Farmer, et al., HSR, 2008, in press.

20. QUITS Post-Implementation Survey Source: Yano, Rubenstein, Farmer, et al., HSR, 2008, in press.

21. Multiple Data Sources:Strengths and Challenges

22. Triangulation • Critical to collect information about implementation from multiple sources • Be prepared for disagreement • Perspectives and opportunities for observation differ for managers, providers vs. patients • Recognize differences between “exposed” sample and practice population • Does the “enrolled” group represent the practice? • Did the intervention penetrate among all providers?