Achieving Synergy Between Designing and Reporting Improvement Projects

1. Achieving Synergy Between Designing and Reporting Improvement Projects Kaveh G. Shojania, MD Editor-in-Chief, BMJ Quality & Safety Director, Centre for Patient Safety University of Toronto

2. Rube Goldberg Pencil Sharpener

4. Messiness of Practice Settings • Hard to understand target problems • Difficult to attribute amidst noise of randomness and other deliberate changes • Unintended consequences • Lots of implementation challenges • Makes having a theory to guide you very helpful

5. Hospital infections affect thousands each year Hospital staff do not wash their hands consistently We implemented a multifaceted strategy Staff education Telling patients to ask staff if they have washed their hands Briefly stated design, analytic strategy and main outcomes Typical Improvement Report Introduction Methods Results • It didn't work Discussion • We can only guess why

6. Commonly identified barriers to hand hygiene compliance include A, B, and C  Staff Education and engaging patients to speak up about hand hygiene address A, B, and C by doing X, Y, and Z. Describes the education content and format Describes the way in which patients interacted with staff Methods Better Report Introduction Results • No improvement • Many patients felt uncomfortable questioning their doctors or nurses Discussion • This intervention did not succeed and we know why

7. That report is an improvement, but.. • Ideally, we would have recognized (through doing more background work or pilot testing) that the intervention had problems • The intervention theory was wrong or incomplete • Implementation required attention to additional issues • There has to be a better way to plan interventions (and their evaluations), so that we increase success

8. A Good Evaluation Plan Increases Chance of Success • Explicit theory for why intervention will work • Specifies the “active ingredients” of the intervention • Explains how those ingredients address the causes of the target safety or quality problem • Outcomes that measure success but also capture the degree to which intervention worked as expected • Framework for refining the intervention and addressing implementation problems

9. Improvements often not matched to underlying problem “We’ll have a lecture series on the subject” “Let’s develop a clinical practice guideline” “Do an audit and send performance reports” “Create a new a new order form or clinical pathway” • Each presupposes a certain type of problem: • Education implies knowledge problem • Performance report implies the problem is largely under control of the recipients of the reports

10. Framework for Understanding Quality Problems • Organisational –infrastructure, staffing, culture • Equipment/human factors • Professional • knowledge/skills • Attitudes, peer opinion • Memory, time factors • Patient --e.g. knowledge, expectations

11. Overuse of Antibiotics for Upper Respiratory Tract Infections • Organisational –infrastructure, staffing, culture • Professional • Knowledge/skills • Attitudes, peer opinion • Memory, time factors • Patient --e.g. knowledge, expectations Maybe – some doctors think they can identify patients who need ATBx Probably not a major factor, but still possible Definitely!!!

12. Patient Expectations and Antibiotic Prescriptions for URIs

13. Dissatisfaction When Not Given ATBx Phillips. J Am Board Fam Pract, 2005

14. Importance of Understanding Behavior

15. Target: decreased use of Foley catheters Successful Application of Matching Solution to Problem : reduction of nosocomial UTIs • Approx 20% of hospitalized patients have Foley catheters • Main cause of nosocomial UTI • Insertion not indicated 25% of the time • often placed in Emerg before MD sees patient • Continued use not indicated 50% of time • MDs unaware of catheter 30-50% of time Education directed at MD likely to be ineffective

16. Trial of Automatic Stop Orders for Urinary Catheters Cornia et al. Am J Med. 2003

17. Ten Specific Steps to Consider • Specify active ingredients of intervention • Think about dose and intensity: how frequently should performance feedback occur and in what format? • How often should nurses screen for fall risks or ask about x, y, or z – just at admission? Every few days? • Explicit theory for why intervention will work • Match ingredients of your intervention to the causes of the target problem • E.g., Why will a, b, and c fix the target problem?

18. Theory aids design of intervention not just the eventual report • Why will screening for fall risk work? • Because if we identify the subset of “high risk” patients, then we can focus on them. But, can you effectively identify such a subset? Well…actually, most patients end up being `high risk´ 2. How will bed alarms reduce falls? • Because nurses love nothing better than more alarms going off

19. Articulation of Alternate Theories Even Better: E.g., METs • Main theory: deteriorating patients not recognized early enough AND ward physicians often difficult to reach AND ward physicians may not escalate care in timely fashion • Transfers to ICU often delayed even when ward staff want them; RRTs facilitate transfers to ICU ( when staffed by ICU personnel) • METs make more patients DNR → fewer unexpected deaths Specification of main and alternate theories directly suggests different outcomes worth measuring

20. House Calls for Homebound Elders • Intuitively sensible but consider 2 types of patients: • Homebound due to multiple chronic problems and frailty, plus frequent acute problems (UTIs, delirium from meds)  rapid access to care via house calls may be beneficial • Homebound with multiple chronic problems, but relatively stable – does well being seen every few weeks  regular visits with transportation support probably sufficient and more cost-effective than house calls Specifying theory helps identify important issues with patient selection : hard to achieve impact cost-effectively, if 1st group can’t easily be identified

21. Thinking through theory enhances • Patient selection – sharpens thinking about what types of patients will most benefit from intervention • Recognition of key ingredients – you may realize that you’re missing some E.g., for house calls, maybe availability of an on-call physician for advice, reassurance will be crucial for some patients (or their caregivers) • Evaluation – articulating mechanism will facilitate choice of outcomes, design, markers of implementation fidelity • Recognition of key contextual factors

22. Contextual Factors • ‘Table 1’ in traditional research reports relevant characteristics of patients and/or institutions • Some items there routinely (e.g., age, gender) • Others chosen based on theoretic understanding (E.g., stage of disease, specific comorbid conditions, other risk factors for outcomes of interest, socioeconomic status) • Similarly, long list of potential contextual factors • Some may appear routinely (e.g., reason for interest in target problem or intervention, support by management, infrastructure commonly related to QI, like clinical IT) • Others require a theory for the intervention (e.g., interprofessional culture or teamwork highly relevant to some interventions but not to others)

23. Ten Specific Steps • Specify the active ingredients of your intervention • Think about dose and intensity, too • Explicit theory for why intervention will work • Match ingredients to the causes of the target problem • Identify the providers or patients most likely to benefit from the intervention • Evidence that supports or refutes aspects of the intervention; recognize areas of uncertainty

24. CHF Disease Management Program • Evidence-based protocols for choosing medications, investigations, interventions • Protocols for adjusting medications based on patient-reported symptoms and functional status • Daily telephone based contact using automated response system with concerning responses relayed to clinicians Ingredients – fairly clear Theory - not hard to guess from description Evidence....

26. Patients made daily, toll-free calls to automated system During each call patients heard a series of questions about their symptoms entered responses using keypad Responses reviewed daily and issues relayed to clinicians If patients did not use system for 2 consecutive days, they received a reminder call after that, they were contacted by site staff to encourage participation What Was The Intervention? Primary Outcome: readmission for any reason or death from any cause within 6 months

27. Results • 85% of patients made at least one call • Adherence highest (90%) during Week 1 • Decreased to 55% by Week 26. • Median of 21 clinical variances per patient • No significant differences in the primary endpoint or any of the secondary endpoints • Subgroup analyses failed to identify a group for which the intervention was effective, despite efforts to include sites and patients who demonstrated enthusiasm for participation and screening of the patients for their ability to follow the protocol.

28. Domestic violence emergency department quality improvement

29. Screening for intimate partner violence in health care settings: a randomized trial. JAMA. 2009

30. OBJECTIVE: To determine the effectiveness of IPV screening and communication of positive results to clinicians. Randomized controlled trial conducted in 11 emergency departments, 12 family practices, and 3 obstetrics/gynecology clinics in… Randomized controlled trial conducted in 11 emergency departments, 12 family practices, and 3 obstetrics/gynecology clinics in Ontario, Canada, among 6743 English-speaking female patients aged 18 to 64 years who presented between July 2005 and December 2006 CONCLUSIONS: Although sample attrition urges cautious interpretation, the results of this trial do not provide sufficient evidence to support IPV screening in health care settings. Evaluation of services for women after identification of IPV remains a priority.

31. CONCLUSIONS: In screening for IPV, women preferred self-completed approaches over face-to-face questioning; computer-based screening did not increase prevalence; and written screens had fewest missing data. These are important considerations for both clinical and research efforts in IPV screening.

32. Looking for Evidence • Most interventions have been tried before – always look for reports of similar interventions • You may find important information on likely effectiveness (or lack thereof) • Previous work may suggest specific refinements • Components of your intervention may have been studied as part of other interventions • E.g., telephone contact in the heart failure disease management example • May convey useful information about likely effectiveness and implementation for your intervention

33. 4. Anticipated Timeline of Impact Quality Improvement Outcome 1 Outcome 2 Outcome 3 0 1 6 2 3 5 4 Years Some outcomes may even worsen initially…

34. INTERVENTION: Two home visits by a pharmacist within two weeks and eight weeks of discharge to educate patients and carers about their drugs, remove out of date drugs, inform general practitioners of drug reactions or interactions, and inform the local pharmacist if a compliance aid is needed. CONCLUSIONS: The intervention was associated with a significantly higher rate of hospital admissions and did not significantly improve quality of life or reduce deaths. Further research is needed to explain this counterintuitive finding and to identify more effective methods of medication review.

35. We studied the effect of an intervention designed to increase access to primary care after discharge from the hospital, with the goals of reducing readmissions and emergency department visits and increasing patients' quality of life and satisfaction with care. ...at nine VA Medical Centers, we randomly assigned 1396 veterans hospitalized with diabetes, COPD, or CHF to receive either usual care or an intensive primary care intervention. The intervention involved close follow-up by a nurse and a primary care physician, beginning before discharge and continuing for the next six months. CONCLUSIONS...the primary care intervention we studied increased rather than decreased the rate of rehospitalization, although patients in the intervention group were more satisfied with their care. .

36. Ten Specific Steps (cont’d) • Process for refining intervention (learning ) • Include outcomes that assess “fidelity of implementation” in order to identify and address implementation issues • E.g., how often did patients use the phone system? Did they make selections consistent with appropriate use? • Modify theory of the intervention: maybe frequently eliciting symptoms isn’t the key issue ? • Unintended consequences

38. “Safety of Patients Isolated for Infection Control” • Isolated patients also more likely to • have no vital signs recorded as ordered (51% vs 31%; P<.001) • have days with no physician progress note (26% vs 13%; P<.001) Isolated patients twice as likely to experience adverse events (31 vs 15 adverse events per 1000 days; P<.001).

39. Lots of other examples of unintended consequences • CPOE – entering orders for wrong patients, problematic default doses or decision support • EMR – template driven records that contain little usable information, copy and pasting from previous admissions leading to perpetuation of mistakes and/or harder to understand assessments • Fall programs that reduce mobility (and don’t even reduce fall-related injuries) • Various policies and procedures that lead to hazardous work arounds

40. Ten Specific Steps (cont’d) • Traditional methodologic questions: What is you project's Achilles Heel and do your design choices deal with it? • Are secular trends an issue? • Might the intervention take awhile to “turn on”? • Is your primary outcome valid? • Does your primary outcome tell the whole story? • Will you be able to say anything useful if the intervention doesn’t work?

41. Common problems with Outcomes • Unknown connection to outcomes of interest • Measure that has evidence but depends on other processes • e.g., smoking cessation counseling: too easy to claim counseling delivered with no actual impact • Outcomes that don’t tell whole story • E.g., Reduced length of stay but no idea of readmissions increased or more problems in ambulatory setting • “cardiac arrests on ward” – arrests may just be happening in intensive care unit with no change in survival • Subjective outcomes – need to have blinded outcome assessment or more than one assessor with reliability

42. Ten Specific Steps (cont’d) • Traditional methodologic questions: make sure your choices for design and outcomes deal with the major potential threats to validity • Spread: identify parts your want to disseminate • not all components may be necessary or appropriate; identify the ones you think can and should be spread • Sustainability – think about costs, but also potential erosion of engagement or attention

43. Summary • Considering these steps will increase chance of success for QI projects • Explicit theory for why intervention will work • Outcomes that capture success but also measure the intervention’s impact on mediators of success • Anticipation of implementation barriers and process for refining intervention and/or implementation • Attention to these elements will also substantially enhance your writing and success in publication

44. Key Writing Points • What is your story? Be able to tell it in a couple of sentences • What is the main weakness? • Abstract is single most important element • What are the Tables and Figures going to be? • Show your outline and then your draft to a senior colleague