A Statistical Perspective on Mixed Methods

1. A Statistical Perspective on Mixed Methods MIXED METHODS RESEARCH WORKSHOP Institute For Social Research – Room 6080 May 16th, 2012 Thomas N. Templin, PhD Office of Health Research College of Nursing Wayne State University Please do not duplicate or use these slides without the express permission of the author.

2. Research designs in context • The literature on mixed methods (MM) is large and complex so it is worth taking a look first at what we stand to gain. • Put these developments in the context of research design today. • The growth of MM designs over the past few decades is part of growing trend to make research designs more intelligent, more capable, and more adaptive. Tom Templin, MCUAAAR workshop 5/2012

3. Research designs in context • Intelligent • Uses what we know to learn what we don’t know • Example: pretest information • Capable • Arrives at the correct population inference • Example: all statistical assumptions are met • Adaptive • Does not waste resources • Example: stopping early Tom Templin, MCUAAAR workshop 5/2012

4. Research designs in context • Experimental designs and clinical trials • Interim analyses allow multiple looks at the data without inflating Type 1 error. • Trial can be stopped early for either success or failure (futility). • Sequential multiple assignment randomized trial (SMART) for the development of dynamic treatment regimes. Susan Murphy • Brown, C. H., T. R. Ten Have, et al. (2009). Adaptive Designs for Randomized Trials in Public Health. Annual Review of Public Health. 30: 1-25. Tom Templin, MCUAAAR workshop 5/2012

5. Research designs in context • Quasi experimental designs extended the range the interventions we could examine • Cook, T. D. and D. T. Campbell (1979). Quasi-Experimentation: Design and Analysis Issues for Field Settings. Chicago, Rand McNally. • Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and Quasi-Experimental Designs: for Generalized Causal Inference. Tom Templin, MCUAAAR workshop 5/2012

6. Research designs in context • Causal Modeling • Development of causal modeling theory increased the sophistication of research questions we could address including threats to validity in randomized designs and quasi-experiments • Rubin, D. B. (1974). Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies. Journal of Educational Psychology, 66(5). • Pearl, J. (2000). Causality: Models, reasoning, and inference. New York, NY, Cambridge University Press. • Morgan, S. L., & Winship, C. (2007). Counterfactuals and Causal Inference: Methods and Principles for Social Research. New York: Cambridge Univ Press. • Hernan, M., & Robins, J. (2012). Causal Inference: Chapman & Hall/CRC. • Equations plus qualitative information Tom Templin, MCUAAAR workshop 5/2012

7. Research designs in context • The Hope for Mixed Methods • Increased credibility of qualitative findings • Increased role of qualitative exploration to shape the direction of the research • Increased value of individual differences and individual variation • More intelligent, capable, adaptive • Jennifer Green’s mixed method story Tom Templin, MCUAAAR workshop 5/2012

8. What’s New in Mixed Methods? We have routinely used open coded items on surveys in convergent (concurrent) designs. We have always used multi-phased sequential designs in instrument development. It is not uncommon to include qualitative interviews to determine how participants experience an intervention in embedded designs We just didn’t use these particular terms Tom Templin, MCUAAAR workshop 5/2012

9. What’s New in Mixed Methods? We have routinely used open coded items in survey designs. We have always used multi-phased designs in instrument development. It is not uncommon to include qualitative interviews to determine how participants experience an intervention Tom Templin, MCUAAAR workshop 5/2012

10. What’s New in Mixed Methods? • Mixed methods research design terminology • convergent (concurrent or parallel) • Sequential (exploratory sequential, explanatory sequential) • Embedded • Terminology to describe how qualitative and quantitative data can be integrated • Merged • Connected • Embedded Tom Templin, MCUAAAR workshop 5/2012

11. What’s New in Mixed Methods? • NIH OBSSR “Best Practices for Mixed Methods Research in the Health Sciences • Directions for mixed methods research applications to NIH • Explicit reference to qualitative and quantitative data in the Specific Aims, Significance and Innovation sections • Description of rigorous qualitative and quantitative methodology in the Approach Section Tom Templin, MCUAAAR workshop 5/2012

12. What’s New in Mixed Methods? In a concurrent design, investigators often discuss the collection of both types of data (quantitative and qualitative) before discussing the analysis of both types of data (quantitative and qualitative). Data Collection Quantitative Qualitative Data Analysis and Interpretation Quantitative Qualitative Integration/Merging Procedures Tom Templin, MCUAAAR workshop 5/2012

13. What’s New in Mixed Methods? In a sequential design discuss the collection and analysis of the first type of data (quantitative or qualitative) and then discuss the collection and analysis of the subsequent type of data (qualitative or quantitative). First Phase (quantitative or qualitative) Data Collection Data Analysis and Interpretation Connecting Procedures (e.g., development of sampling procedures or materials based on the results from the first phase) Second Phase (qualitative or quantitative) * Data Collection * Data Analysis and Interpretation Tom Templin, MCUAAAR workshop 5/2012

14. Three Ways of Integrating Qualitative and Quantitative Data Merging Connecting Embedding Tom Templin, MCUAAAR workshop 5/2012

15. Integrating qualitative and quantitative data by merging Merging—1. Qualitative and Quantitative data can be integrated by using words and numbers together. Describing the statistical result and giving examples from the qualitative analysis. In the opening Chapter of Creswell and Plano Clark’s (2011) handbook on mixed methods they give this description of mixed methods as used in everyday life. Creswell, J. W. & Plano Clark (2011) Designing and Conducting Mixed Methods Research. Thousand Oaks, CA: Sage Publications Inc. Tom Templin, MCUAAAR workshop 5/2012

16. Integrating qualitative and quantitative data by merging Merging qual and quant data Example 1. “Consider for a moment An Inconvenient Truth, the award-winning documentary on global warming featuring the former U.S. vice president and Nobel Prize winner Al Gore .. In the documentary, Gore narrated both the statistical trends and the stories of his personal journey related to the changing climate and global warming. This documentary brings together both quantitative and qualitative data to tell the story. Also, listen closely to CNN's broadcast reports about hurricanes or about the votes cast in elections. The trends are again supported by the individual stories” (Creswell & Plano Clark, 2011). Tom Templin, MCUAAAR workshop 5/2012

17. Integrating qualitative and quantitative data by merging Merging ql and qn data-Example 2. Numerically coded qualitative data can be merged with quantitative data in cross tabulated tables. For example if a qualitative analysis classifies participants in three mutually exclusive classes, say Type A, Type B, and Type C. Then it might be in interest to examine the Type by Gender crosstab. This crosstab is an example of integration by merging Tom Templin, MCUAAAR workshop 5/2012

18. Integrating qualitative and quantitative data by merging Merging ql and qn data-Example 3. If the survey contained both Likert type response items and qualitative items with open codes items, it might be of interest to examine the correlation between the Likert items and various recodings of the open ended item. A specific example is actual report card grade and self reported grade in response to the question, What kinds of grades do you generally get in school? Tom Templin, MCUAAAR workshop 5/2012

19. Integrating qualitative and quantitative data by merging Merging ql and qn data-Example 4. Short answer questions added to standardized test are coded and factor analyzed with other test items Tom Templin, MCUAAAR workshop 5/2012

20. Integrating qualitative and quantitative data by merging Mixed method research designs that collect qual and quant data concurrently combine qual and quant by merging as described in the examples above. Tom Templin, MCUAAAR workshop 5/2012

21. Integrating qual and quant data by connecting. Data are connected when the results of the qual (or quant) in one phase of the study depend in some way on the quant (or qual) data collected in another phase of the study. Connecting ql and qn--Example 1 Focus groups and experts are used during Phase 1 to develop questionnaire items for a test that will be administered to the target population in Phase 2. Tom Templin, MCUAAAR workshop 5/2012

22. Integrating qual and quant data by connecting. Connecting ql and qn--Example 2 In Phase 1, the quantitative analysis ranks students in terms of their reading ability. In Phase 2, the top 5 and the lowest 5 students are selected of in depth interviews about reading attitudes, interests, etc. Mixed method sequential research designs use connecting to integrate qual and quant data. Tom Templin, MCUAAAR workshop 5/2012

23. Integrating qual and quant by embedding Embedding ql and qn—Example 1. “In this form of integration, a dataset of secondary priority is embedded within a larger, primary design. An example is the collection of supplemental qualitative data about how participants are experiencing an intervention during an experimental trial” (p. 6). Tom Templin, MCUAAAR workshop 5/2012

24. Integrating qual and quant by embedding Embedding ql and qn—Example 2. “The [researchers] implemented an RCT study to compare the two treatments in terms of various repeated measure patient outcomes, including pain levels. Embedded within the RCT study, they also gathered qualitative data in the form of audiotapes of the intervention sessions, along with nurse and patient notes, to describe the issues, strategies, and interactions experienced during the intervention. The results provide evaluation of both the outcomes and process of the intervention.” (BPMM, p. 6). Tom Templin, MCUAAAR workshop 5/2012

25. Designs for Mixed Methods Research Convergent Sequential Embedded Tom Templin, MCUAAAR workshop 5/2012

26. The OBSSR Best Practices document gives examples of three kinds of mixed methods research designs and notes that many more are available in the literature. • That is an understatement. Interest in these methods has resulted in a plethora of MM research designs and frameworks for research designs since the Jennifer Greene’s (1989) pioneering review. This can make literature difficult for the novice. • Creswell and Plano Clark (2011) listed 15 different classification schemes. Most used different terminology. • Onwueegbuzie and Collins (2007) noted that a popular handbook of mixed methods contained approximately 35 MM designs. • Jennifer Greene (2007) stated from Maxwell and Loomis, that “the actual diversity in mixed methods studies is far greater than any typology can adequately encompass” (2003), pl. 244). Tom Templin, MCUAAAR workshop 5/2012

27. Some discontent surrounding mixed method design typologies In Greene’s mixed method text (2007) she writes, “Muted by the emphasis on design typologies are possible contributions to better understanding that could come from mixes in philosophy, substantive theory, and disciplinary thinking, alongside mixes of differences in personal experience, education, values, and beliefs” (p 15). KatrinNiglas (2009) noted that in their, “attempts to build more and more exhaustive typologies” … “the terminology in the field: it gets very specific and complicated but remains ambiguous at the same time. She attempted to deconstruct MM design categories using conventional research design concepts. An interesting approach. Niglas, Katrin (2009). How the novice researcher can make sense of mixed methods designs. International Journal of Multiple Research Approaches, 1, 13 - 33. Tom Templin, MCUAAAR workshop 5/2012

28. Convergent (or parallel or concurrent) MM design. Example from Tony Omwuegbuzie (2007). A study interested in student attitudes toward reading and reading strategies administered a survey containing predetermined questions with Likert type response options and open ended questions to elicit qualitative information about reading strategies. Since all the information is collected at one time, standard research design terminology would call this a cross sectional design. Tom Templin, MCUAAAR workshop 5/2012

29. Sequential MM design (or explanatory sequential or exploratory sequential) One data set builds on another. This design involves two distinct interactive phases. Data is collected and analyzed in each phase. The results of Phase 1 inform data collection in Phase 2. Example 1. Interviews are conducted following the administration of a quality of life instrument to better understand the mechanisms underlying the responses. What else do we need to specify here? Tom Templin, MCUAAAR workshop 5/2012

30. Sequential MM design (or explanatory sequential or exploratory sequential) Example 2. Qualitative interviews are conducted to identify risks involved in the treatment of diabetes. A questionnaire is developed and administered to a population of patients. Typical instrument development design (exploratory sequential); type of data integration is connecting. See example on connecting. Tom Templin, MCUAAAR workshop 5/2012

31. Embedded MM design The embedded design can be a variation of a convergent or sequential design but either qual or quant is dominant throughout. “A prototype would be to conduct an intervention study and to embed qualitative data within the intervention procedures to understand how experimental participants experience the treatment.” Tom Templin, MCUAAAR workshop 5/2012

32. Questions Seems like the convergent design is the same as a cross sectional design in monomethod research and the sequential design is a phased design approach. Tom Templin, MCUAAAR workshop 5/2012

33. Conclusions Mixed Methods in context New terminology New best practice guidelines Tom Templin, MCUAAAR workshop 5/2012

34. Mixed methods in context • Think about using mixed methods with advances used in mono method designs • Adaptive experimental designs • Causal modeling Tom Templin, MCUAAAR workshop 5/2012

35. New terminology • Terms advanced in the Best Practice Guidelines are probably worth knowing even if not intuitive • Data integration terms • Merging • Connecting • Embedding • Research design • Convergent (concurrent or parallel) • Sequential (sequential exploratory and sequential explanatory) • Embeded Tom Templin, MCUAAAR workshop 5/2012

36. New best practice guidelines • Guidelines advocate discussion of qual and quant in each part of the application • While qual methods are often used in designs that are not mixed methods designs, mixed methods greatly extends the role of qualitative data by including qual research in study aims, significance, and innovation Tom Templin, MCUAAAR workshop 5/2012 Please do not duplicate or use these slides without the express permission of the author.