Formulating Research Questions and Designing Studies

1. Formulating Research Questions and Designing Studies 2 March 2011 Facilitator: Larry Gruppen, Ph.D. University of Michigan Development team: Carol Hodgson, LuAnn Wilkerson, David Irby, Judy Shea, Clair Kuykendall, Larry Gruppen

2. Preview • Moving from general issues to specific research questions • Using the literature to refine the question • Measuring the variables • Selecting the right research design

3. The Research Process Define Research Question No hole in the literature Hole in literature Conduct Literature Review Refine Question Operationalize variables Design Study Obtain IRB Approval Collect & Analyze Data Write and Report Results

4. Recognizing & Choosing Among Research Opportunities • What situations or problems tend to puzzle, fascinate, challenge, or interest you? • List as many research questions as you can as quickly as possible • Identify which are: • - Most interesting (I) • - Feasible (F) • - Fundable ($) • - Best overall (*) • Rewrite your best research question at least twice

5. The Research Question • Based on literature/theory • Includes sample description (e.g., 4th-year medical students) • Includes study design (e.g., relationship, difference between groups, etc.) • Includes the independent & dependent variables • Is measurable • Stated as a question or hypothesis

6. Definitions • Variable • Independent and dependent variables • Covariates • Operationalization and measurement • Sample and population

7. Example Research Question • Does training in airway management on a simulator improve resident performance (correct selection of equipment, # of errors, time to intubation), compared with standard didactic/demonstration instruction.

8. Example Research Question • What is the independent variable? • What is the dependent variable? • What are possible covariates? • What is the population and sample? • What is your design?

9. Example Research Question Do first-year medical students who complete a student-run anatomy review course score higher on the anatomy final exam than students who do not complete the review course? • What is the independent variable? • What is the dependent variable? • What are possible control variables? • What is the sample? • What is your design?

10. The Research Process Define Research Question No hole in the literature Hole in literature Conduct Literature Review Refine Question Operationalize variables Operationalize variables Design Study Obtain IRB Approval Collect & Analyze Data Write and Report Results

11. Searching for Related Work • Identify possible sources of information • Colleagues and librarians • Databases, PUBMED, PSYCLIT, Science Citation Index • Journals, chapters, books, publications • Read critically and summarize • Citations referenced • Sample size • Study design and limitations • Overall conclusions

12. What to Look For • Conceptualization and theory • How variables are defined & measured • Sample size • Study design and limitations • Overall conclusions • Citations referenced

13. The Research Process Define Research Question No hole in the literature Hole in literature Conduct Literature Review Refine Question Operationalize variables Operationalize variables Design Study Obtain IRB Approval Collect & Analyze Data Write and Report Results

14. Refining the Research Question • Share your best question • Critique each other’s research questions • Determine which is the best/most interesting question • Refine this group question based on critique • Write final version of question

15. Critique Your Question • Clearly stated? • Stated as a question? • Testable? • Defines variables to be studied? • Defines sample to be studied? • Describes the setting for the study?

16. The Research Process Define Research Question No hole in the literature Hole in literature Conduct Literature Review Refine Question Operationalize variables Design Study Obtain IRB Approval Collect & Analyze Data Write and Report Results

17. Operationalization and Measurement • Three basic questions • What do you measure? • How do you measure? • How well do you measure?

18. What do you measure? • Defined by research question • Dependent, independent variables • Covariates

19. How do You Measure? • Operationalization is essential to the conduct of the study • Counting events • Measuring time and physical quantities • Externalizing internal (psychological) states, events, and processes

20. Measurement Instruments • Knowledge • Tests (MCQ, essay, oral) • Attitudes • Questionnaires, surveys • Behavior or performance (skills) • Clinical exams (OSCE), standardized patients, direct observation • Checklists, ratings, error rates

21. How Well do You Measure? • Reliability — score accuracy or stability • Would the score be reproduced if tested again? • Would the score be reproduced by different raters? • Validity — score meaning • Does the score measure what you intend to measure?

22. The Research Process Define Research Question No hole in the literature Hole in literature Conduct Literature Review Refine Question Operationalize variables Design Study Obtain IRB Approval Collect & Analyze Data Write and Report Results

23. Use When Focus on meaning and context In-depth recording and triangulation Inductively derived interpretation Methods Interview Observation Think aloud, stimulated recall Chart review Surveys Qualitative ResearchWhat is the phenomenon?

24. Use When Predictors can’t be randomized Subjects &/or treatments not controllable Control groups not available Methods Surveys Chart review Archived data Correlational Research All data is confidential

25. Use When Specific intervention Temporal relationship Feasible explanatory mechanism Subjects and treatments controlled Methods Control over treatment and measurement Randomization Control Groups Experimental and Quasi-Experimental Research

26. Defining Your Sample • Target population • Accessible population • Intended sample • Inclusion and exclusion criteria • Availability • Time frame • Willingness to participate • Intended variables • Actual sample

27. Research DesignsQuasi Experimental • One shot case study X -----O • One group Pretest-Posttest O-----X-----O (O=Observation X=Treatment)

28. Research Designs Quasi-Experimental • Post-test only control group X---------O -----------O • Pre-test/Post-test Control Group O--------X-------O O-----------------O

29. Research Designs Experimental • Post-test only control group (Randomized) X---------O -----------O • Pre-test/Post-test Control Group (Randomized) O--------X-------O O-----------------O • Solomon Four Group Design (Randomized) (1) O------X------O (2) O--------------O (3) ------X------O (4) --------------O R = randomization R R R

30. Threats to Internal Validity • History • Maturation • Repeated measurement • Statistical regression • Selection • Loss of Subjects/mortality • Investigator bias

31. External Validity • Is the sample representative of the population? Can the study be generalized to the population? • Are the conditions the same? For example, laboratory setting versus natural setting. • Did the subjects act differently because they were subjects in the study (Hawthorne Effect)?

32. The Research Process Define Research Question No hole in the literature Hole in literature Conduct Literature Review Refine Question Operationalize variables Design Study Obtain IRB Approval Collect & Analyze Data Write and Report Results

33. Is Your Study Research? • Research means a systematic investigation including research, development, testing, and evaluation to develop or contribute to generalizable knowledge • If you might publish the results, it’s research AND NEEDS IRB REVIEW • What about quality or evaluation studies?

34. Issues for the IRB • Data normally collected in your course? • Data collected specifically for your study? • Will the data be anonymous? • Will you use an any identifier? Is it linked? • Do you need an informed consent? • What kind of review? Exempt, Expedited, Full

35. The Research Process Define Research Question No hole in the literature Hole in literature Conduct Literature Review Refine Question Operationalize variables Design Study Obtain IRB Approval Collect & Analyze Data Write and Report Results

36. Analyzing Data • Get a consultation on the analysis • Even if you know biostatistics - ed research uses some different methods • Do it before you start the study!

37. Analyzing Data • Where to look for consultations • Department - colleagues who have done this before • Medical school - Dean’s office, ed research unit • University - school of ed, public health, psychology, educational support centers

38. The Research Process Define Research Question No hole in the literature Hole in literature Conduct Literature Review Refine Question Operationalize variables Design Study Obtain IRB Approval Collect & Analyze Data Write and Report Results

39. Publishing Med Ed Research • Emergency Medicine journals & meetings • General Medical Education journals & meetings • Academic Medicine, Medical Education, Teaching and Learning in Medicine, Advances in Health Sciences Education, Medical Teacher, … • Assoc. of American Medical Colleges, Ottawa Conference, Assoc. for Medical Education in Europe, ACGME, and many more

40. Research That Makes a Difference • Investigates important questions • Connects study to prior research and to theory • Uses appropriate research design & data analysis procedures • Disseminates results

41. Planning Next Steps • Create a project plan: tasks & deadlines • Find collaborators, mentors, consultants • Search for funding • Protect time for research • Keep a research journal • Have fun

42. References • Bass, Dunn, Norton, Stewart, & Tudiver. (1993). Conducting Research in the Practice Setting. Newbury Park, CA: Sage. • Campbell & Stanley (1963). Experimental and Quasi-experimental Designs for Research. Dallas: Houghton Mifflin. • Glesne & Peshkin (1992). Becoming Qualitative Researcher: An Introduction. Longman. • Hulley & Cummings (1988). Designing Clinical Research. Baltimore: Williams & Wilkins.