Jeffrey G. Chipman MD, Constance C. Schmitz PhD, Travis P. Webb MD, Mohsen Shabahang MD PhD,

1. Evaluating the Reliability and Validity of the Family Conference OSCE Across Multiple Training Sites Jeffrey G. Chipman MD, Constance C. Schmitz PhD, Travis P. Webb MD, Mohsen Shabahang MD PhD, Stephanie F. Donnelly MD, Joan M. VanCamp MD, and Amy L. Waer MD University of Minnesota, Department of Surgery Funded by the Association for Surgical Education Center for Excellence in Surgical Education, Research & Training (CESERT) Surgical Education Research Fellowship (SERF)

2. Introduction • ACGME Outcome Project • Professionalism • Interpersonal & Communication skills • Need test with validated measures

3. Professionalism & Communication • More important than clinical skills in the ICU Crit Care Clin 20:363-80, 2004 • Communication • Accessibility • Continuity • 1 out of 5 deaths in the US occurs in an ICU Crit Care Med 32(3):638, 2004 • < 5% of ICU patients can communicate when end-of-life decisions are made Am. J. Resp. Crit. Care. Med. 155:15-20, 1997

4. Family Conference OSCE • Two 20-minute encounters (cases) • End-of-life • Disclosure of a complication • Literature-based rating tools • Trained family actors and raters • Ratings by family, clinicians, self • Debriefing, video Chipman et al. J Surg Ed, 64(2):79-87, 2007.

5. Family Conference OSCEMinnesota Experience • High internal consistency reliability • Strong inter-rater agreement • Raw differences favored PGY3s over PGY1s • Small numbers Chipman et al. J Surg Ed, 64(2):79-87, 2007 Schmitz et al. Crit Care Med 35(12):A122, 2007 Schmitz et al. Simulation in Health Care 3(4):224-238, 2008

6. Replication StudyPurpose • Test the feasibility of replicating the OSCE • Examine generalizability of scores • Institutions • Types of raters (clinical, family, resident) • Examine construct validity • PGY1s vs. PGY3s

7. Replication StudyMethods • 5 institutions • IRB approved at each site • Training Conference (Minnesota) • Site Training • Detailed case scripts • Role plays • Videos of prior “good” and “bad” performances

8. Replication StudyMethods – Learner Assessment • Assessment by: • Clinical raters (MD & RN) • Family actors • Self • Only family raters were blinded • Rating forms sent to Minnesota • Data analyzed separately for DOC, EOL

9. Classical test theory considers only one type of measurement error at a time Test-retest Alternate forms Internal consistency Inter-rater agreement Generalizability theory allows for errors that occur from multiple sources Institutions Rater type Family actors Provides overall summary as well as breakdown by error sources and their combinations Generalizabilty Theory Mushquash C & O’Connor. SPSS and SAS programs for generalizability theory analyses Behavior Research Methods 38(3):542-47, 2006

10. Generalizabilty Theory • Summary statistics (0 to 1) • 1.0 = perfectly reliable (generalizable) assessment • Relative generalizability • Stablility in relative position (rank order) • Absolute generalizability • Agreement in actual score

11. ResultsFeasibility • N = 61 residents • Implementation fidelity was achieved at each site • Key factors: • Local surgeon champions • Experienced standardized patient program • On-site training (4 hrs) by PIs • Standardized materials & processes

12. ResultsInternal Consistency Reliability

13. ResultsGeneralizability The relative G-coefficients we obtained suggest the exam results can be used for formative or summative classroom assessment. The absolute G-coefficients suggest we wouldn’t want to set a passing score for the exam. Downing. Reliability: On the reproducibility of assessment data Med Educ 38:19009-1010, 2004

14. ResultsConstruct Validity MANOVA Disclosure End-of-Life p = 0.44 p = 0.41 Between subjects effect (PGY 1 vs. PGY 3) was not significant (p = 0.66 DOC, p = .0.26 EOL).

15. Study Qualifications • Only family members were blinded • Clinician and family ratings were significantly correlated on EOL & DOC • Nested vs. fully crossed design

16. ConclusionsFamily Conference OSCE • Feasible at multiple sites • Generalizeable Scores • Useful for formative, summative feedback • Raters were greatest source of error variance • Did not demonstrate construct validity • Questions the assumption that PGY-3 residents are inherently better than PGY-1 residents, particularly in communication

17. Study PartnersLurcat Group • Amy Waer, MD • University of Arizona • Travis Webb, MD • Medical College of Wisconsin • Joan Van Camp, MD • Hennepin County Medical Center • Mohsen Shabahang, MD, PhD • Scott & White Clinic, Texas A&M • Stephanie Donnelly MD • Mayo Clinic Rochester • Connie Schmitz, PhD • University of Minnesota Acknowledgments Jane Miller, PhD, and Ann Wohl, University of Minnesota IERC (Inter-professional Education Resource Center) Michael G. Luxenberg, PhD, and Matt Christenson, Professional Data Analysts, Inc., Minneapolis, Minnesota