Mixed-methods data analysis September 2011

1. Mixed-methods data analysisSeptember 2011 Richard Watson Todd http://arts.kmutt.ac.th/crs/research/mmda2.ppt

2. Types of data • Quantitative data • Rating scales, experiments • Qualitative data • Discourse, multimodal

3. “A mixed methods study involves the collection or analysis of both quantitative and qualitative data in a single study with some attempts to integrate the two approaches at one or more stages of the research process” (Dörnyei, 2007)

4. Types of Mixed-Methods Research (MMR)

5. Mixed methods research designs • Embedded concurrent MMR • Example 1 • Research into attitudes: survey a large number and interview a predetermined small number of subjects • Purpose: unclear • Concurrent MMR • Example 2 • Research into beliefs: interview 4 teachers but survey 80 students • Purpose: accounting for practicality in using instruments

6. Mixed methods research designs • Exploratory sequential MMR • Example 3 • Interview a small number to gain insights to design a questionnaire, then survey a large number • Purpose: informing instrument design • Explanatory sequential MMR • Example 4 • Research into attitudes: survey a large number of subjects, then, selecting based on questionnaire responses, interview a small number • Purpose: follow-up on interesting results or providing a basis for subject selection

7. MMR v. MMDA • “The most common perception of mixed methods research is that it is a modular process in which qualitative and quantitative components are carried out either concurrently or sequentially. Although this perception is by and large true, it also suggests that the analysis of the data should proceed independently for the QUANT and QUAL phases and mixing should occur only at the final interpretation stage. This conclusion is only partially true … we can also start integrating the data at the analysis stage, resulting in what can be called mixed methods data analysis [MMDA]” • Dörnyei (2007)

8. MMDA • QUANT data can rarely be analysed qualitatively • MMDA usually focuses on QUAL data • Quantitising QUAL data allows broad patterns to be identified • Quantitising QUAL data is only worth doing when the data is substantial • Information is always lost when converting QUAL to QUANT • Most quantitisation of QUAL data views discourse as a corpus, not a narrative

9. Explanatory sequential MMDA • Explanatory sequential MMR: QUAL first (collect and analyse), then QUANT to allow generalisation • You want to identify categories/patterns from QUAL data inductively • E.g. whether respondents in an online discussion agree or disagree with the original posting and how they do this; what identities students manifest in a groupwork task; how supervisors mitigate criticisms in a thesis supervision

10. Explanatory sequential MMDA • Start with QUAL analysis of a subset of the data • Generate categories/patterns • Problems: Why that subset? Impractical to show QUAL analysis of whole data set • Provide summary of analysis of whole data set through QUANT analysis (e.g. frequency of categories) • Can treat data as narrative with dispersion plot summary

11. Explanatory sequential MMDAAn example • Goal: To investigate the functions of within-sentence repetitions by teachers in classroom discourse (e.g. “In an aeroplane, yes, in an aeroplane.”) • No previous research in the area so functions need to be induced from the data qualitatively • After the QUAL analysis, functions are counted to identify frequencies of use to provide a summary of how teachers use the various functions

12. Exploratory sequential MMDA • Exploratory sequential MMR: QUANT first (collect and analyse), then QUAL to explain in depth • You want to deductively apply existing categories/patterns to data • E.g. the use of linguistic features associated with stance; metacognitive strategies from think-aloud protocols; uses of 3 key purposes of L1 use in groupwork tasks

13. Exploratory sequential MMDA • Start by applying existing categories to the data and counting them • Provide QUANT overview of categories in the data • Problem: QUANT overview hides actual usage and complexity • Conduct in-depth QUAL analysis of a subset to show details • Does a corpus analysis starting with keyness and moving on to examination of concordance lines illustrate this approach?

14. Exploratory sequential MMDAExample 1 • To investigate how and why a curriculum has changed over 4 years, in-depth interviews with teachers were conducted • Themes needed to be identified from the interview data • Danger of bias in identifying themes • Use QUANT frequency analysis to identify keywords indicative of potential themes • Apply these keyword-generated themes in the QUAL analysis

15. Exploratory sequential MMDAExample 2 • Goal: To identify patterns of functions in teacher instructions through an exchange structure analysis • Functions identified through interviews with teachers and counted • Chi-square applied to identify salient sequences of functions • Typical patterns of functions in teacher instructions identified through QUANT analysis • Sample extract presented to make findings tangible and concrete

16. Embedded concurrent MMDA • Embedded concurrent MMR: QUAL + QUANT (from same source collected at same time), but analysed separately • Embedded concurrent MMDA: (separate) QUAL and QUANT analyses of same data • You want to examine 2 different aspects of the data or you want to examine the data at 2 different levels of specificity • E.g. in an online discussion, what category of contributor responds to a posting and how do they respond; what personal pronouns are used in near/distant Thai interaction and how does their use relate to other linguistic features; length of pauses in student talk and placement of pauses

17. Embedded concurrent MMDA • Apply 2 analyses (1 QUAL, 1 QUANT) with different focuses to the same data • Integrate the findings by integrating the discussion of the 2 focuses

18. Embedded concurrent MMDAAn example • Goal: To investigate whether Red and Yellow communicate in online forums promotes antagonism or agonism • Need to identify: 1. levels of antagonism of Reds and Yellows; 2. content concerns of Reds and Yellows; 3. how Reds and Yellows respond to each other in discussion threads • For 1., identify colour of contributor, code for level of antagonism, calculate mean values and correlations between colour and antagonism (QUANT) • For 2., conduct keyness analysis comparing content of Red and Yellow contributions (QUANT) • For 3., in-depth examination of threads (QUAL)

19. MMDA task • Explanatory sequential MMDA: QUAL → QUANT for generalisation • Exploratory sequential MMDA: QUANT → QUAL for depth • Embedded concurrent MMDA: 1 QUAL, 1 QUANT for different focuses • Task 1: embedded concurrent • Task 2: exploratory sequential • Task 3 explanatory sequential