Interactive Case-based Assessment Tool ( iCAT ) Bringing Together the “What” and “How of Learning:

1. Interactive Case-based Assessment Tool(iCAT)Bringing Together the “What” and “Howof Learning: Wendy Ham, Andrew Rosenfeld, Lukasz Babiarz, Sandy Mong, Chris Dede, Susan Pasquale, Robert Leffert,  and James Quattrochi Harvard University, ICON Program for Multidisciplinary Learning, Cambridge, MA University of Massachusetts Medical School, Worcester, MA Supported by the Harvard Provost Innovations in Instructional Technology Award

2. Background • We have developed a case simulation method, called ICON (Interactive Case-based Online Network) • In ICON, students interact with case characters online • Case characters (patient, physicians, case consultants, family members) are assumed by faculty members • Events in ICON unfold in real time • Students learn to: • Integrate the theory and practice of medicine • Solve problems • Navigate resources • Make decisions • Develop teamwork skills • Be accountable to case characters and teammates • How do we measure the competencies listed above?

3. Survey of Student Perception of ICON Hypothesis 1 ICON facilitates a learning process that is more engaging Hypothesis 2 ICON increases peer contribution in learning Hypothesis 3 ICON increases active faculty participation in student learning Hypothesis 4 ICON decreases learning dependence on faculty lecturer due to increased contribution by peer and faculty mentor

4. Yr 1 26 (81%) 12 (38%) Year of survey Entrance Exit Yr 2 14 (58%) 17 (71%) Survey Method Comparison of entrance and exit surveys from two years of ICON implementation in the second-year Human Nervous Systems and Behavior course at Harvard Medical School: # of Responses (% Response) Total 40 (71%) 29 (52%)

5. Student Perception of ICON: Survey Results Hypothesis 1: ICON facilitates a learning process that is more engaging # hours spent discussing tutorial cases outside of tutorial session P=0.004 Without ICON With ICON

6. Student Perception of ICON: Survey Results Hypothesis 2: ICON increases peer contribution in learning Mean ranking of peer relative to faculty mentor and faculty lecturer in contribution to learning P=0.02 Mean ranking Min value= 1 Max value=3 Without ICON With ICON

7. Student Perception of ICON: Survey Results Hypothesis 3: ICON increases active faculty participation in student learning Mean agreement rating to the statement that “faculty (as case characters) actively participate in student learning” P=0.00003 1=completely disagree 2=somewhat disagree 3=somewhat agree 4=completely agree Agreement rating Min value= 1 Max value=4 Without ICON With ICON

8. Min value= 1 Max value=3 Without ICON With ICON Student Perception of ICON: Survey Results Hypothesis 4: ICON decreases learning dependence on faculty lecturer due to increased contribution by peer and faculty mentor Mean ranking of faculty lecturer relative to peer and faculty mentor in contribution to learning P=0.009 Mean peer ranking Mean ranking

9. OK… so students think ICON: • is engaging • increases peer contribution • promotes active faculty participation • decreases relative role of lectures But how does ICON really add value in their learning?

10. iCATInteractive Case-based Assessment Tool • Characterization of process, not only outcome • Case-based • Emphasis on problem solving in the context of: • Hypothesis formulation • Diagnosis and treatment plan • Risk management

11. Case: Morning Trouble (adapted from Risk Management Foundation case study, 2006) A 42-year-old uninsured woman, who had not seen a doctor in 20 years, presented to the emergency department (ED) at 7:50 a.m., complaining of chest pain and trouble breathing. She was accompanied by her husband and her son, who helped interpret because English was her second language. The husband reported that his wife had been experiencing chest pain since 11:00 the prior evening, which she treated with aspirin. In the morning, she had sudden onset of severe chest pain and fainted on her bed. The patient was first seen by the ED attending and then by a resident. Her initial vital signs were: HR: 107, BP: 146/99, RR:29. Her chest pain was documented as “sudden onset, right-sided, sharp, under the right breast, started while the patient was lying in bed and worse with inspiration, movement, and palpation.” The patient’s medical history was documented by the resident as: “fainting spells, no family history of coronary artery disease or clots, father suffered a stroke.” Toolbox Background H&P Test Consult Notebook Incurred cost: $0.00 Select a tool from the Toolbox Time: 00:00:17

12. English as a second language Click on a keyword for more information: Is there someone with a medical background who can reliably interpret for the patient? Is English the son’s/husband’s first language? Was the history documented with the presence of an interpreter other than family members? Was there any ambiguity in taking the patient’s history? A 42-year-olduninsured woman, who had not seen a doctor in 20 years, presented to the emergency department (ED) at 7:50 a.m., complaining of chest pain and trouble breathing. She was accompanied by her husband and her son, who helped interpret because English was her second language. The husband reported that his wife had been experiencing chest pain since 11:00 the prior evening, which she treated with aspirin. In the morning, she had sudden onset of severe chest pain and fainted on her bed. The patient was first seen by the ED attending and then by a resident. Her initial vital signs were: HR: 107, BP: 146/99, RR:29. Her chest pain was documented as “sudden onset, right-sided, sharp, under the right breast, started while the patient was lying in bed and worse with inspiration, movement, and palpation.” The patient’s medical history was documented by the resident as: “fainting spells, no family history of coronary artery disease or clots, father suffered a stroke.” Toolbox Background H&P Test Consult Notebook Incurred cost: $0.00 Time: 00:01:35

13. English as a second language Click on a question for more information A reliable interpreter with a medical background is not available. Is there someone with a medical background who can reliably interpret for the patient? Is English the son’s/husband’s first language? Was the history documented with the presence of an interpreter other than family members? Was there any ambiguity in taking the patient’s history? Save in Notebook A 42-year-olduninsured woman, who had not seen a doctor in 20 years, presented to the emergency department (ED) at 7:50 a.m., complaining of chest pain and trouble breathing. She was accompanied by her husband and her son, who helped interpret because English was her second language. The husband reported that his wife had been experiencing chest pain since 11:00 the prior evening, which she treated with aspirin. In the morning, she had sudden onset of severe chest pain and fainted on her bed. The patient was first seen by the ED attending and then by a resident. Her initial vital signs were: HR: 107, BP: 146/99, RR:29. Her chest pain was documented as “sudden onset, right-sided, sharp, under the right breast, started while the patient was lying in bed and worse with inspiration, movement, and palpation.” The patient’s medical history was documented by the resident as: “fainting spells, no family history of coronary artery disease or clots, father suffered a stroke.” Toolbox Background H&P Test Consult Notebook Incurred cost: $0.00 Time: 00:02:09

14. Test ECG Chest X-Ray ECG Lab Tests: $650 Blood gassesAcetoneArteritis (ANA)CO2EKGGlucosePotassiumSodiumLiver (SGOT)Liver (SGPT)StoolType & cross blood Studies/Tests: $1,000 Angiogram Audiometry Biopsy Bone Scan Cardiac Eval Cisternogram CSF Studies CT EEG ECG EMG/NCS ENG Evoked Potentials Genetic Tests Ischemic Exercise Test MRA MRI Muscle Biochem Myelogram Neuropsych Testing Pap Smear Pulmonary Function Schilling Test SPECT Scan Tensilon Test Visual Fields X-rays Click to enlarge Save in Notebook Toolbox Background H&P Test Consult Notebook Incurred cost: $1,650.00 Time: 00:07:42

15. Test ECG Chest X-Ray ECG Lab Tests: $650 Blood gassesAcetoneArteritis (ANA)CO2EKGGlucosePotassiumSodiumLiver (SGOT)Liver (SGPT)StoolType & cross blood Studies/Tests: $1,000 Angiogram Audiometry Biopsy Bone Scan Cardiac Eval Cisternogram CSF Studies CT EEG ECG EMG/NCS ENG Evoked Potentials Genetic Tests Ischemic Exercise Test MRA MRI Muscle Biochem Myelogram Neuropsych Testing Pap Smear Pulmonary Function Schilling Test SPECT Scan Tensilon Test Visual Fields X-rays Click to enlarge Save in Notebook Toolbox Background H&P Also notice the timer and cost counter Test Consult Notebook Incurred cost: $1,650.00 Time: 00:07:42

16. Notebook Chest pain Shortness of breath Stroke in family history Absence of trained interpreter Age Aspirin Gender Background Final vital signs Initial vital signs H&P ECG No cardiac enzymes results Elevated glucose X-Rays Blood gases Test Absence of cardiology consult Consult

17. Notebook Key Student Chest pain Shortness of breath Stroke in family history Absence of trained interpreter Age Aspirin Gender Background Why does the student not consider these high risk? Final vital signs Initial vital signs H&P ECG No cardiac enzymes results Elevated glucose X-Rays Blood gases Test Absence of cardiology consult Consult

18. Model of Student’s Rating Profile during Case Progression Profile of Student’s Rating during Case Progression Cumulative value of information accessed (Scale 1-5) (High) 5 4 Student X (Moderate) 3 2 What causes the profile to shift? Baseline (Low) 1 0 (Start) Time (seconds) (End)

19. iCAT beta

20. Instructional technology impacts the process of learning, which impacts the outcome of learning Design, Development, and Implementation of Instructional Technology Outcome of Learning Process of Learning Impact Inform

21. Impact is tightly coupled with information, allowing progress in educational development to be driven by hypothesis Design, Development, and Implementation of Instructional Technology Outcome of Learning Process of Learning Impact Inform

22. Contact Information • http://icon.fas.harvard.edu • wendyham@gmail.com