Question, theory and method in systematic reviews Research Methods Festival 2008

1. Question, theory and method in systematic reviewsResearch Methods Festival 2008 Sandy Oliver

2. Karen Bird Jeff Brunton Adam Fletcher David Gough Angela Harden Ann Oakley Sandy Oliver Rebecca Rees Philip Rose James Thomas Lisa Underwood AcknowledgementsThe Methods for Research Synthesis Programme of the ESRC National Centre for Research Methods

3. Outline • Types of research questions • Synthesis methods for • Generating theory • Testing theory • Exploring theory • Comparison of research questions, theory and methods • Choice of approaches for synthesis

4. Types of research questions • What types of questions do primary studies ask? • Working typologies for appraising research (web sites) • Typologies resulting from analyses of literature (academic journals) • Analysis of social science research questions across 16 disciplines of the social sciences

5. Research questions What types of questions do primary research studies ask?

6. Types of research questions What types of questions do primary research studies ask? Typical distinction: qualitative or quantitative research

7. Types of research questions What types of questions do research syntheses ask? Typical cluster of questions: How are concepts related? How salient, important, valuable are they?

8. Types of research questions What types of questions do research syntheses ask? Typical cluster of questions: Are concepts related? How much are concepts related?

9. Types of research questions What types of questions do research syntheses ask? Typical cluster of questions: How are concepts related? How salient, important, valuable are they? How much are they related?

10. Types of research questions What types of questions do research syntheses ask? Typical cluster of questions: How are concepts related? How salient, important, valuable are they?

11. Example: generating theory • Research question: • How do the perceived meanings of medicines affect patients’ medicine taking behaviour and communication with health professionals? • Britten N, Campbell R, Pope C, Donovan J, Morgan M, Pill R (2002) Using meta ethnography to synthesise qualitative research: a worked example. J Health Serv Res Policy 7(4):209-215

12. Steps taken in the synthesis • The researchers looked across the papers for common and recurring concepts. • Used Schutz’s notion of ‘first- and second-order constructs. • First order: everyday understandings of ordinary people • Second order: constructs of the social sciences • ‘Third order interpretations’ were derived from a ‘line of argument’ based on 1st and 2nd order concepts.

13. Translating studies into one another

14. Meta-ethnographic synthesisAdapted from Britten et al., 2002

15. Conclusions about theory and method • Methods developed for synthesising ethnographic research are applicable to the synthesis of other kinds of (non-ethnographic) study. • By developing ‘third order interpretations’, this method goes beyond the primary studies and develops new concepts. • The hypotheses it produces could be tested by other researchers.

16. Other qualitative syntheses • Beck 1996 • Six qualitative studies revealed a shared set of five themes that help increase our understanding of mothering multiples Smith et al 2005 • 32 qualitative studies provide insight into patients' experiences of recognising symptoms of cancer and seeking help. Paterson et al 1998 • 43 qualitative interpretive research synthesised to advance understanding of the lived experience of diabetes Harden et al 2004 • Thematic analyses of views studies to identify factors influencing behaviour change

17. Synthesis methods for generating theory • Few concepts pre-specified, most emerge during the analysis • Tend to use qualitative data in a qualitative synthesis • Theory is the product – it appears as the synthesis is completed • Qualitative analysis • Meta-ethnography • Content analysis • Grounded theory • Thematic synthesis • Interpretive synthesis

18. Types of research questions What types of questions do research syntheses ask? Typical cluster of questions: Are concepts related? How much are concepts related?

19. Example: testing theory • Research question: • Are secondary school-based mental health promotion interventions based on cognitive behavioural treatments effective in preventing or alleviating depression, anxiety and suicidality amongst children and young people? • KavanaghJ, Oliver S, Lorenc T, Caird J, Tucker H, Greaves A, Harden A, Oliver K, Thomas J, Oakley A. Reducing inequalities in young people's mental health: a systematic review of the effectiveness of school-based cognitive-behavioural interventions (in preparation)

20. Steps taken in the synthesis • The researchers searched for studies with the relevant: • Population • Intervention • Outcomes • Study design They prepared a statistical meta-analysis based on the theory that the intervention has an effect on pre-specified outcomes They also prepared population subgroup analyses and meta-regression to test whether the intervention reduced health inequalities

23. Meta-regression • As the groups increased in SES, the effect size went down by 0.25. • This was not quite statistically significant, co-efficient = 0.25 (p=0.072), but is close. • This trend may therefore be a real effect, or the result of co-incidence. However, the findings should be treated with a high level of caution, as the three analyses which provided data for the meta-regression contain only two studies each. They are suitable only for generating hypotheses for future evaluation.

24. Other meta-analyses, sub-group analyses and meta-regressions Thomas et al 2004 • Investigated the content of interventions with respect to children’s views of healthy eating • Oliver et al 2008 • Investigated study designs, intervention content and preparation to determine whether RCTs provide the same effect size and variance as non-randomised studies of similar policy interventions; and whether these findings can be explained by other associated factors.

25. Conclusions about theory and method Statistical methods are well developed for testing pre-specified theories about • Effectiveness • Effectiveness with specific populations • Comparative effectiveness with different populations • Comparative effectiveness of different intervention designs/ components

26. Synthesis methods for testing theory The theory is explicit, precise and pre-specified in its detail • Theory informs the search for studies and • The analysis is quantitative and the theory is open to refutation • Does it make a difference? How much difference? • Does it make a difference for different groups? How much difference? • Statistical meta-analysis • Bayesian analysis • What are the active elements of an effective intervention? • Meta-regression

27. Types of research questions What types of questions do research syntheses ask? Typical cluster of questions: How are concepts related? How salient, important, valuable are they? How much are they related?

28. Example: exploring theory • Research question: • What is known about the factors which promote or hinder young people’s health behaviour change across a number of health topics/settings? • Oliver S, Harden A, Rees R, Shepherd J, Brunton G, Garcia J, Oakley A. An emerging framework for including different types of evidence in systematic reviews for public policy. Evaluation, the International Journal of Theory, Research and Practice 2005; 11 (4): 428-446.

29. Steps taken in the synthesis • The researchers searched for and synthesised two types of studies: • ‘Views’ studies and controlled trials about Young people and mental health promotion Juxtaposed in a matrix studies according to health promotion theory • Individual • Family and friends • School (community) • Physical and material resources (wider environment) and study design • Views studies • Controlled trials To facilitate a qualitative analysis

30. Other mixed methods syntheses Gaber 2000 • Narrative procedure and vote counting for a social service meta-needs assessment in Nebrasksa • Oliver et al 2001 • Framework analysis of qualitative and quantitative reports of public involvement in research agenda setting Evans and Fitzgerald 2002 • why people are physically restrained in the acute and residential care settings.

31. Conclusions about theory and method Juxtaposing findings from the literature within a theoretical and methodological matrix: • Allowed easy comparison of studies with similar findings • Allowed easy comparison of studies of similar interventions • Divided the literature into subsets to facilitate qualitative analysis Where there is a clear framework and structured methods, it is easier: • To describe in advance • To involve non-researchers in guiding the review • To involve novice reviewers in conducting the review • To prepare maps or “rapid reviews”, and meet funders information needs/timetables Mixing methods increased the policy relevance of reviews by combining evidence of the effectiveness of interventions with qualitative understanding from people’s lives

32. Synthesis methods for exploring theory • Some concepts pre-specified, some emerge during the synthesis • Can use qualitative or quantitative data in a qualitative synthesis • Theory is introduced after the studies are identified, to guide the analysis, and is open to refinement • Framework synthesis • Realist synthesis • Content analysis

33. Comparison of approaches

34. Conclusions • Synthesis methods are complementary rather than competing • Choice of method depends upon: • Whether concepts and theories are clear in advance • The purpose of the work • To generate, explore or test theory • To advance understanding, to inform the choice of interventions, or to inform the development and implementation of interventions • Whether data is qualitative, quantitative or mixed • The skills and traditions of the review team • The relationship with policy decisions