Synthesising & Integrating Qualitative Research Evidence Methods @ Plymouth 2011 Friday 20 May

1. Synthesising & Integrating Qualitative Research EvidenceMethods @ Plymouth 2011Friday 20 May E. van Teijlingen, A. Avenell, F. Douglas & P. Hoddinott

2. Road map • Introduction • Rise of qualitative synthesis; • Methodological conundrums (tensions & questions raised); • Tools use to address some of those; • Our thinking around qualitative synthesis for a HTA project on men’s health and obesity prevention & treatment.

3. Introduction • Research team • Substantive topic • HTA funding Synthesising & Integrating Qual. Res. • A growing methodological field • Different approaches have grown organically over past two decades • Our ideas to attempt synthesis.

4. HTA Project I Research team: • Alison Avenell • Flora Douglas • Pat Hoddinott • Edwin van Teijlingen Topic areas: • Men’s Health • Obesity Prevention & Treatment Funding body: • HTA Project 09/127/01

5. HTA Project I Multidisciplinary Research Team

6. Men’s Health I Men die younger than women Men generally have less healthy lifestyles than women, more likely to take risk, etc. Men tend to seek help later in disease stage (i.e. wait longer before going to get medical care)

7. Men’s Health II Perhaps a gender-sensitive approach is needed to better understand men's health seeking behaviour. If existing services were more “male friendly”, i.e. convenient and anonymous they might be better used by men. (Schofield et al., 2000; Banks, 2001; Lloyd & Forrest, 2001).

8. Weight management Obesity & associated diseases are huge challenge to NHS and UK. Lack of engagement of men in services for obesity is a serious hindrance to mitigating its effects. Obesity interventions, especially weight management interventions, often seen as ‘for women’.

9. The funded study Overarching objective is to integrate quantitative & qualitative evidence base for engagement and management of men with obesity in weight loss services: (1) Systematically reviewing the effectiveness and cost-effectiveness of interventions for obesity in men, and men in contrast to women (2) Systematically reviewing the effectiveness and cost-effectiveness of interventions to engage men in their weight reduction (3) Systematically reviewing qualitative research with men about obesity management, and providers of such services for men

10. Road map • Introduction • Rise of qualitative synthesis; • Methodological conundrums (tensions & questions raised); • Tools use to address some of those; • Our thinking around qualitative synthesis for a HTA project on men’s health and obesity prevention & treatment.

11. Background I • Field of synthesising evidence from qualitative primary studies has witnessed the emergence of several different approaches, each is based on different epistemological viewpoint.  • These approaches not only develop research questions, but also conduct a critique of the literature included in the review. 

12. Background II BUT, end product of synthesis can be: • too conceptual/ complex for policy-makers, managers & practitioners;  • Too costly, some methods are time and resource intensive. Policy makers often / usually want a rapid answer, which limits the choice of analysis procedures. 

13. Background III • We reflect on approaching synthesis from ‘realist’ perspective.  We are trying to find out not only “what works” for men in terms of weight management, but also “for which men, and under what circumstances”.  • We are developing an eclectic study design, drawing from several methodological approaches, employing deductive and inductive analytical approaches throughout review process.

14. Why do we do it? When to use Qualitative Methods: • Focusing on the ‘why’ question. • Preparing for quantitative studies • In parallel with quantitative studies: • explain / examine in-depth meaning findings of quantitative studies. • as part of triangulation.

15. Range of Methods • In-depth interviews • Focus groups • Observation • Content analysis • Critical incident technique • Case studies

16. Road map • Introduction • Rise of qualitative synthesis; • Methodological conundrums (tensions & questions raised); • Tools use to address some of those; • Our thinking around qualitative synthesis for a HTA project on men’s health and obesity prevention & treatment.

17. Can we compare? I In order to be able to synthesise one has to agree that we actually can compare studies based on very different qualitative research methods in different populations based on different methodological approaches within qualitative research.

18. Can we compare? II Which leads to next step that one has to agree that we can assess the quality of qualitative research across studies using different methods and/or approaches .

19. Range of views • All research perspectives are unique, each valid in its own right, i.e. no general quality criteria; • Qualitative and quantitative methods can be assessed by the same (or very similar) criteria; • Qualitative and quantitative methods can be assessed, but not by the same criteria (checklists for qualitative research).

20. NO to comparing I • Some people argue that we can’t apply quantitative / Positivist ideas of systematic reviewing and meta-analysis to methods as diverse as qualitative ones.

21. NO to comparing II • SAMPLING • Purposive / theoretical • Criterion / quota • Random • Snowball • etc • Purposive sampling: • Extreme Case • Maximum Variation • Homogeneous • Typical Case • Critical Case • Etc.

22. No to comparing III Can we really mix qualitative methodologies? Ethnographers study shared experiences and their construction within a specific environment and culture, whilst Grounded Theorists uses the data more to explain (i.e. build theory around) a phenomenon.

23. No common criteria • All research perspectives are unique and each is valid in its own terms. This view rejects the notion of establishing general quality criteria; • View of some purists. Mays N, Pope C. Assessing quality in qualitative research. BMJ 2000; 320:50-52.

24. Same or similar criteria • Qualitative and quantitative methods can be assessed by the same (or very similar) criteria, such as: • acceptability, cost, validity, reliability, generalisability (or relevance) and objectivity. (Ryan et al. 2001)

25. Checklists Predefined criteria in assessing quality of qualitative studies: ‘checklists’ Blaxter, M. Criteria for qualitative research. Med. Sociol. News 2000; 26: 34-7. Mays & Pope. Rigour and qualitative research. BMJ 1995; 311:109-112. Mays & Pope. Assessing quality in qualitative research. BMJ 2000; 320:50-2. Critical Appraisal Skills Programme (CASP) 10 questions to help you make sense of qualitative research, www.sph.nhs.uk/sph-files/casp-appraisal-tools/Qualitative%20Appraisal%20Tool.pdf Malterud K. Qualitative research: standards, challenges & guidelines, The Lancet 2001 358: 483-88

26. Typical checklist I • Worth or relevance of research? • Clarity of research question: able to set aside preconceptions? • Appropriateness of research design to question? • Is the context or setting adequately described? • Sampling: full range possible cases/settings so that conceptual rather than statistical generalisations could be made? Cont.

27. Typical checklist II • Analysis: analysis incorporating all observations? Could it explain key processes or respondents’ accounts or observations? Search for disconfirming cases? • Reflexivity of account: Self-conscious assessment impact of methods used on data obtained? Are sufficient data included in report to assess whether analytical criteria had been met? (Mays& Pope, BMJ 2000: 50-52)

28. Case for checklists • Checklist give outsiders a chance to evaluate the quality of our work. • At the same time it is important to stress that there a many different ways to conduct qualitative research.

29. Against checklists I • Are qualitative methods really different in general outline? • Are there no common criteria for a good piece of research using either qualitative or quantitative methods? • There are different ways to conduct quantitative research: • (e.g. parametric versus non-parametric statistics).

30. Against checklists II • Checklists focus on how research is conducted rather than on the quality of the method itself. • There is a risk that criteria become ‘bureaucratic’ or ‘prescriptive’. Whilst flexibility is exactly the strength of qualitative methods! Barbour (2001)

31. Not same criteria • Qualitative and quantitative methods can be assessed, but not by the same criteria. • E.g. Lincoln & Guba (1985) suggested four criteria specifically for the evaluation of qualitative methods: • credibility, transferability, consistency (or dependability) and confirmability.

32. Case for common criteria I • Main quality criteria: (1) acceptability; (2) validity; (3) reliability; or (4) generalisability (or relevance); (5) objectivity. • Criteria apply to the valuation of quantitative and qualitative techniques, perhaps some criteria require minor modification for qualitative research.

33. Common criteria: example ‘validity’ (2) Validity • Triangulation • Respondent validation • Reflexivity (relevant to validity & reliability) • Data analysis: • Computer packages • Grounded theory • Analytical induction (e.g. identify ‘deviant’ cases)

34. Common criteria: example ‘objectivity’ Methods employed to maximise objectivity: • Reflexivity: a process where researchers continually reflect upon how their own interests and potential biases could alter the interpretation of the results; • intersubjectivity (interrater reliability).

35. Grading studies A: No, or few flaws. The study credibility, transferability, dependability and confirmability is high. B: Some flaws, unlikely to affect the credibility, transferability, dependability and/or confirmability of the study. C: Some flaws that may affect the credibility, transferability, dependability and/or confirmability of the study. D: Significant flaws that are very likely to affect the credibility, transferability, dependability and/or confirmability of the study. (Downe et al. 2007)

36. Synthesizing Table 1 Process of meta-synthesis Steps in process of doing a qualitative meta-synthesis: • aim of study: establish appropriate research question • set inclusion and exclusion criteria methodologies • identify and locate relevant papers/publications • appraise studies located through process above. • synthesize findings reported in papers above.

37. 1. Research question • Some of earlier meta-synthesis have been criticised for being too broad, e.g. synthesis by Sherwood (1997) on ‘clients perspective of caring’ • Walsh and Downe (2005) concluded that the debate of a broader more inclusive stance versus a narrower more precise stance is still ongoing, although they seem to favour a narrower focus.

38. 2 Inclusion/ exclusion criteria • Include all qualitative research, all methods and all approaches. • Include only selected methods • Include only selected approaches • Etc.

39. 3 Identifying and locating studies Qualitative researchers are less likely to publish in journals. Most electronic databases comprise mainly journal articles, not reports and book chapters which often accepts longer manuscripts needed for reporting qualitative research in detail. Hence need to search outwith the standard electronic databases, and check references of articles found in the initial trawl. Methods sections in papers are often too short for authors to give detailed information on how the study was exactly conducted, this creates the need to contact the original authors for further details.

40. 4 Appraising qualitative literature • Arguably the most crucial step in meta-analysis, but most uncertain. • Some meta-synthesists refer to existing appraisal methods (‘tools’), such as CASP (as mentioned before) or the UK Cabinet Office quality framework (Spencer et al. 2003).

41. 5 Approaches to meta analysis • Meta narrative • Meta synthesis • Thematic synthesis • Grounded Theory • Critical interpretive synthesis • Meta ethnography • Ecological Triangulation • Qualitative Metasummary • Content Analysis • Framework synthesis • Etc. (review of range of methods by Barnett-Page & Thomas 2009)

42. 5a Meta-narrative Narrative inductive method of analysis Greenhalgh et al. (2005) on diffusion of innovation in health services. Bring together research which contains many different theories from many different disciplines & study designs. “In the best narrative work, descriptive/ interpretative analysis is a story about stories.” (Jones 2004: 107)

43. 5b Thematic synthesis Adopts approaches from meta-ethnography and grounded theory. Method developed to assist reviews on interventions. Thomas & Harden (2008) advocated thematic analysis in three stages: (a) coding ‘line by line’ of included studies; (b) develop ‘descriptive’ themes and (c) generate analytical themes.

44. 5c Critical interpretive synthesis Adopts approaches from meta-ethnography and grounded theory. Critical interpretive synthesis is approach to doing the whole reviewing process not just the synthesis part of it (Dixon Woods et al. 2007). E.g. iterative approach redefining research question and searching and selecting from literature.

45. 5d Meta ethnography A comparative textural analysis using three ways to order them: • in terms of one another; • set against one another; • tied to one another. Meta-ethnography aims to recover the social and theoretical context in which substantive findings emerge (Noblit & Hare, 1988, p. 5-6). It is driven by interpretation, not analysis.

46. But complex & expensive Barnett-Page and Thomas (2009) argued that products of some qualitative synthesis reviews can be complex and conceptual and (in their opinion), tend to require further interpretation by policy makers and practitioners. • Some synthesis methods can be very time and resource intensive

47. Road map • Introduction • Rise of qualitative synthesis; • Methodological conundrums (tensions & questions raised); • Tools use to address some of those; • Our thinking around qualitative synthesis for a HTA project on men’s health and obesity prevention & treatment.

48. Realist Approach I Realist evaluation has as its underlying principles of what works best, for whom, and under what circumstances (Kazi 2003:1). Realistic or Realist evaluation has particular strengths in assessing interventions delivered in context; (Pawson & Tilley 1997).

49. Realist Approach II Realist evaluation is particularly useful “in the evaluation of practice in an open system that consists of a constellation of interconnected structures, mechanisms and contexts” (Kazi 2003:158).

50. Realist Approach III • We aim to uncover how effective interventions work (if they exist), and describe a recipe for the intervention ingredients, processes, environmental and contextual factors that contribute to effectiveness (Pawson 1997).