Qualitative Methods in Educational Research

1. Qualitative Methods in Educational Research Aaron W. Bernard, MD Jonathan Fisher, MD

2. Objectives • Define qualitative research • Contrast with quantitative research • Describe how to conduct qualitative research by using examples in the literature

3. Quantitative Research • Test well-specified hypothesis concerning some predetermined variables • Answer questions like; • Whether…….. • How much….

4. Quantitative Educational Research • Attempts to study links between teaching factors and learning outcomes • Improved Medical Student Satisfaction and Test Performance with a Simulation-based Emergency Medicine Curriculum: A Randomized Controlled Trial. Ten Eyck. Ann Emerg Med. 2009;54:684-691

5. Qualitative Research • Qualitative research questions tend not to ask whether or how much but rather; • What • Why • How • Qualitative research offers insight into social, emotional, and experiential phenomena. • Qualitative reports do not typically generate answers but rather generate narrative accounts, explanations, conceptual frameworks.

6. Qualitative Research • Qualitative research is inquiry-guided, inductive, and descriptive. • Similar to an anthropologic or sociologic descriptive study.

7. Data Collection Methods • Field observations • Interviews • Acad Med. 2009. • Document analysis • Teach Learn Med. 2004.

8. Data Analysis • Grounded theory • Coding • Iteration • Theoretical saturation

9. Qualitative Educational Research • Acad Emerg Med 2005;12:856-861.

10. What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg Med 2005. • Background • Effective teaching practices in the ambulatory setting have been documented • The ED is a different type of learning environment • A qualitative study to determine what EM teachers consider important has been done1 • What do EM learners consider important to effective EM teaching? • Quantitative methods vs.. qualitative methods Bandiera . Ann Emerg Med 2005; 45:253-261

11. What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg Med 2005. • Methods • Focus groups at Five EM programs in Ontario • Multiple sites adds validity • Medical student, EM Resident, Off-service resident, Junior FRCP, Senior FRCP • Purposeful sampling • 28 participants total

12. What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg Med 2005. • Methods • Semi-structure interviews by a trained moderator • Tape-recorded and transcribed by a third party • Participants used terms like “junior” to identify themselves to allow the data to be matched to type of participant

13. What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg Med 2005. • Methods/Data Analysis • Transcripts were reviewed by two independent assessors and a coding framework was derived using grounded theory methods • Novel ideas were assigned a new teaching code or a new qualifier as they emerged • When a new code was added the entire transcript was reviewed • The two codes that emerged were compared and consensus was reached • Agreement on 41 coded items and disagreement on 9 • 5/9 were subthemes

14. What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg Med 2005.

15. What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg Med 2005. • Data analysis • The transcripts were then reviewed against the final code by one investigator to check for frequency of codes • A second investigator reviewed 20 percent of the transcript to allow for an estimation of interrater reliability • Computer software exists to help with this!

16. What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg Med 2005. • Summary • Qualitative methods were most appropriate for this investigation • Focus group interviews are a common method to obtain data • Purposeful sampling (low numbers, key players) • Grounded theory to create code • Frequency of themes to assess importance of themes to subject

17. Qualitative Educational Research Academic Medicine 2010; 85:124-133

18. Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010 • Background • The informal and hidden curriculum drives professionalism development • There is a lack of evidence regarding what students experience. • Objective • To understand students experiences as they relate to an understanding of professionalism

19. Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010 Methods Students on IM required to post 2 narratives over 2 months Retrospective review of narratives June – November 2007 272 narratives from 135 students

20. Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010 Methods/Data Analysis Thematic content analysis Immersion/crystallization method Reading and rereading narratives until themes emerge Code was created with the first 50 narratives A narrative could be coded more then once

21. Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010 Reliability One author created the code based on the first 50 narratives then two others independently reviewed the work. Created focus groups of students to review themes

22. Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010 Two major domains 14 themes 63.4% positive, 29.1% negative, 7.5 % hybrid

23. Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010

24. Direct representative quotes Smith AK, Fisher J. Ann Emerg Med. 2009.

25. Direct representative quotes

26. Direct representative quotes

27. Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010 Summary Qualitative methods were most appropriate for this investigation Document analysis is a common method to obtain data Methodological steps to improve validity is important The use of direct representative quotes

28. Qualitative Educational Research Teach Learn Med. 2010;22(1):28-32

29. Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn Med 2010 Background Disaster training is relevant to medical students who as physicians may be asked for guidance during training or real events. Medical school offer little formal curriculum Objective Provide 4th year medical students with a didactic and clinically experience in disaster medicine Evaluate the effectiveness through lecture evaluations and a qualitative analysis of focus groups

30. Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn Med 2010 Methods 43 students 4 hour didactic session the evening before the drill Training in the use of a standardized assessment tool for disaster medicine drills Evaluation and Focus Group 5 point likert scale surveys regarding the didactics 2 Focus groups after the drill Semi-structured interviews

31. Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn Med 2010 Methods/Data Analysis Median, means, SD for the survey data Notes were taken in real time One researcher performed the qualitative analysis and created the thematic domains of the code

32. Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn Med 2010 28 of the 42 students participated in both the didactic and the drill 17 agreed to participate in the focus group Themes Changes in self-perceived attitude toward DM Changes in student’s ability to apply this knowledge in a simulated setting.

33. Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn Med 2010 Effective use of qualitative research to create a descriptive study of an educational curriculum that is publishable. Combination of qualitative and quantitative methods

34. Final Conclusions Qualitative research is a tool that can be used for educational investigations. Qualitative research is inquiry-guided, inductive, and descriptive. Data usually comes from field observations, interviews, or document analysis. Grounded theory is used to review the data to create themes and subthemes. This is known as coding. Frequency of themes is often used as a marker of importance of themes to subjects studied Various techniques can be used to increase reliability of the analysis

35. References Tehaerani A, O’Brien B, Masters DE, et al. Burden, responsibility, and reward: preceptor experiences with the continuity of teaching in a longitudinal integrated clerkship. Acad Med. 2009;84(10 Supp):s50-s53. Alford CL, Currie DM. Introducing first-year medical students to clinical practice by having them “shadow” third-year clerks. Teach Learn Med. 2004;16(3):260-263. Thurgur L, Bandiera G, Lee S, et al. What do emergency medicine learners want from their teachers? A multicenter focus group analysis. Acad Emerg Med 2005; 12:856-861. Karnieli-Miller O, Vu R, Holtman M, et al. Medical students’ professionalism narratives: a window on the informal and hidden curriculum. Acad Med. 2010;85:124-133. Giacomini MK, Cook DJ, et al. Users’ guides to the medical literature XXIII. Qualitative research in health care A. Are the results of the study valid? JAMA. 2000;284:357-362. Smith AK, Fisher J, Schonberg MA, et al. Am I doing the right thing? Provider perspectives on improving palliative care in the Emergency Department. An Emerg Med. 2009;54:86-93. Kaji AH, Coates WC, Fung C-C. Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn Med. 2010;22(1):28-32