Strengths & Weaknesses of a Pre-Post Controlled Randomized Trial

1. Strengths & Weaknesses of a Pre-Post Controlled Randomized Trial Michael E. Matheny, MD MS Decision Systems Group Brigham & Women’s Hospital, Boston, MA

2. Trial Design

3. Trial Design

4. Pre-Post Randomized DesignStrengths • Allows the study design to account for changes in measured outcomes by unmeasured factors over time • Provides a single p value for the effect of the intervention that incorporates all of the data • Used primarily for: • Studies that occur over a longer time period • Studies in an environment where other changes taking place could affect the measured outcomes

5. Pre-Post Randomized DesignWeaknesses • Requires allocation of significant time and resources prior to introduction of the intervention • Doubles the cost of data collection • Very little power gained from “doubling” the sample size • calculations are still based on two arm design • Unable to provide a single odds ratio result for the intervention

6. Data AnalysisSAS Code

7. Data Analysis Pre-Post Controlled Design • Use of interaction term in this study design • There are 4 study groups • Multiplicative interaction term of (Pre/Post) * (Control/Intervention) can be interpreted as the relative change in outcome between comparison groups from the baseline to the follow-up evaluation • Pre/Post • Control/Intervention • Post*Intervention • Reported as a p value only • Odds Ratios are reported for both the Control arm and for the Intervention arm

8. Example Results150 Per Arm

9. Example Results150 Per Arm

10. Example Results150 Per Arm

11. Impact of an Automated Test Results Management System on Patients’ Satisfaction of Test Result Communication Michael E. Matheny, Tejal K. Gandhi, E. John Orav, Zahra Ladak-Merchant, David W. Bates, Gilad J. Kuperman, Eric G. Poon Decision Systems Group / Division of General Medicine Brigham & Women’s Hospital, Boston, MA

12. BackgroundTest Result Communication • Test result communication between patients and physicians is a critical part of the diagnostic and therapeutic process • However, follow-up of test results in the primary care setting is often challenging: • High volume of test results • Test results arrive when physician not focused on the patient • Lack of systems to ensure reliability and efficiency

13. BackgroundTest Result Communication Problems • 31% of women with abnormal mammograms did not receive care consistent with established guidelines • 39% of abnormal TSH at BWH were not followed up within 60 days • 36% of abnormal pap smear were lost to follow-up

14. BackgroundPhysician Workflow • 33% of physicians reported they did not always notify patients of abnormal test results • ~30% of physicians reported they did not have a reliable method of test result communication • 59% of physicians were dissatisfied with how well they managed test results despite spending over an hour a day in this activity

15. BackgroundPatient Expectations • Patients do not normally discuss their preferences for test result notification with their providers • Patients preferred telephone notification to regular mail, and found electronic notification to be uncomfortable due to security issues • Patients wanted to be notified of all test results, regardless of whether the results were abnormal

16. BackgroundPatient Satisfaction • These problems reduce patient satisfaction with their medical care, and impair future patient-physician interactions • Improving patient satisfaction has been identified as one of the most important issues currently facing healthcare

17. Objective • To evaluate the impact of an automated test results notification system imbedded into an electronic health record on patient satisfaction regarding test results communication

18. MethodsStudy Setting • Partners HealthCare System • Brigham & Women’s Hospital • Massachusetts General Hospital • Faulkner Hospital • McLean Hospital • Newton-Wellesley Hospital • Free Standing Outpatient Clinics • Longitudinal Medical Record (LMR) • Released July 2000 • Scheduling • Medication lists • Problem lists • Health maintenance record • Clinic notes (free form & templates)

19. MethodsLMR Summary Screen

20. MethodsStudy Setting • Usual care regarding test results management • Test results were embedded directly into the patients’ electronic health record • No automated test results tracking • All test results were mailed to the physician’s clinic office • Physicians were paged directly for critical results

21. MethodsPatient Test Results Screen

22. MethodsIntervention • Results Manager - an electronic test results management system embedded into the LMR • Tracks and displays all test results associated with an ordering physician • Prioritizes by degree of test result abnormality • Facilitates review of test results in context of patient’s history • Generates test result letters • Allows clinicians to set reminders for future testing

23. MethodsResults Manager Summary Screen

24. MethodsResults Manager Letter Generation Screen

25. MethodsStudy Design

26. MethodsRandomization • Stratified randomization of 26 primary care clinics based on: • Primary Hospital Affiliation (BWH / MGH) • Academic or Community Based • Average Patient Socioeconomic Status • Rolling implementation of Results Manager for intervention clinics was conducted from July, 2003 to March, 2004.

27. MethodsStudy Criteria • Patients were randomly sampled in both control and intervention clinics during: • Pre-Intervention: 12/2002 – 06/2003 • Post-Intervention: 09/2003 – 04/2005 • Inclusion Criteria: All patients who had any laboratory, pathology, microbiology, or radiology tests from participating clinics. • Exclusion Criteria: Primary care physician determined that patient should not be contacted or patient did not speak English.

28. MethodsSurvey Instrument • Internally developed • Outcomes were measured on dichotomized Likert scale • Administered by trained research assistants 5 to 7 weeks after test results were posted • Up to three attempts were made to contact each patient

29. MethodsPrimary Outcome Measure • Overall satisfaction with test result communication • “I am satisfied with the way test results are communicated to me”

30. MethodsSecondary Outcome Measures • Satisfaction with PCP listening skills • “My primary care doctor always listens to my concerns” • Satisfaction with information given about treatment and condition • “My primary care doctor gives me as much information about my condition and treatment as I wanted” • Satisfaction with general PCP communication • “My primary care doctor and I communicate very well”

31. MethodsOutcomes • Whether a patient’s expectations were met by the method of test result communication was determined by: • Test result type: normal / abnormal • Defined as requiring follow-up or a management plan change • Method of test result receipt • Patient’s expected delivery method for test • Hierarchy of test result communication • Same Visit > Telephone > Letter > Email > Next Visit > Never • If receipt was by a more desired method, it was counted

32. MethodsData Analysis • Intention-to-treat analysis • Multivariate logistic regression (GEE) • Clustered by primary care physician • Adjusted for Patient Age, Gender, Race, Insurance Status, and Self-Reported Health Status • SAS v9.1

33. Results Response Rates

34. Results Demographics

35. Results Demographics

36. Results Primary Outcome

37. Results Secondary Outcome

38. Results Secondary Outcome

39. Results Secondary Outcome

40. Results Secondary Outcome

41. Discussion • These findings could be related to a number of potential workflow improvements in RM • Provided a concise summary page for the management of test results ordered by a provider • Template-based results letter generator • Can imbed actual test results into letter • Improve patient-friendly interpretations of results • One-click patient reference information

42. DiscussionLimitations • Generalizability • Imbedded in non-commercial EHR • English speaking only • Telephone Survey Bias • Responders vs. Non-Responders • Patient Recall

43. Conclusions • An automated management system that provides centralized test result tracking and facilitates contact with patients: • improved overall patient satisfaction with communication of test results • Increased patient satisfaction with the discussion of treatments/conditions • Improved receipt of results by an expected method of communication

44. Acknowledgements • Co-Authors • Tejal K. Gandhi, MD MPH • John Orav, PhD • Zahra Ladak-Merchant, BDS MPH • David W. Bates, MD MS • Gilad J. Kuperman, MD PhD • Eric G. Poon, MD MPH • Funding • AHRQ U18-HS-11046 • NLM T15-LM-07092

45. The End Michael Matheny, MD MS mmatheny@partners.orgBrigham & Women’s HospitalThorn 30975 Francis StreetBoston, MA 02115

46. MethodsSurvey Response Data