Synthesis of multiple forms of evidence

1. Synthesis of multiple forms of evidence Professor Elizabeth Waters Dr Belinda Hall

2. Synthesis… ...refers to a combination of two or more entities that together form something new

3. Complex systems • Systems Thinking • An overall approach to problem solving: component parts of system best understood in context of relationships with each other and with other systems • View “problems” as parts of an overall system, rather than reacting to specific parts/outcomes • Examine the linkages and interactions between elements • Cyclical thinking rather than linear cause and effect • Need multiple forms of evidence to understand the system

4. MRC framework for complex interventions Key elements of the development and evaluation proce Figure 1 Key elements of the development and evaluation process

5. Reviews of complex interventions • Evaluating complex interventions poses a challenge and requires a substantial investment of time • Unless the intervention studies illuminate processes and mechanisms they often fail to provide useful information • If the result is negative, is it because: • the intervention is inherently ineffective (either because the intervention was inadequately developed or because all similar interventions are ineffective)? • it was inadequately applied or applied in an appropriate context? • the trial used an inappropriate design, comparison group or outcomes?

6. Heterogeneity: the antithesis of synthesis? • Multiple forms of evidence will produce heterogeneity • How much are we comfortable with? • Different forms of difference: • Evidence that answers different questions: feature in different parts of the review (eg. theory vs effectiveness data) • Answering a similar question in a different way (eg. different study designs, populations, interventions, outcomes)

7. Narrative synthesis Framework1: • Developing a theoretical model of how the intervention works, why and for whom • Developing a preliminary synthesis of findings of included studies • Exploring relationships in the data • Assessing the robustness of the synthesis 1Popay et al. 2006. Guidance on the conduct of narrative synthesis in systematic reviews. ESRC Methods Programme.

8. Example: Health and social effects of housing improvements (Thomson et al, in press) • Synthesis structure: • Type of housing intervention • Context and population • Outcomes (respiratory health, general health, mental health) • Study quality • Study design • Data represented in several ways • Forest plot for similar quantitative outcomes without pooling (due to heterogeneity) • Table visually representing the direction of health impact from all included studies • Logic model using review data to show nature and direction of health and socio-economic impacts following housing improvements

9. Logic model developed using review data (Thomson et al, in press) Logic model mapping impact types and direction, and links to health impacts reported in qualitative and quantitative studies of modern day housing improvements in developed world (warmth & energy efficiency improvements, and rehousing/retrofitting)

10. Using qualitative synthesis to explore heterogeneity • 2 reviews: • Qualitative review on HIV/AIDS patients perspectives on adherence • Cochrane Review of intervention trials to improve adherence to therapy in HIV/AIDS patients • Tabulated whether the interventions in the CR corresponded with the patients’ perspectives on how to help them Candy et al. BMC Medical Research Methodology 2011, 11:124

11. Take home messages • Synthesis must generate something “new” • A range of evidence forms should be considered in developing and evaluating complex interventions • Frameworks are useful for organising the different forms and aligning these with the questions they answer • Commit to including process and contextual information as standard ‘evidence synthesis’ practice • Heterogeneity is a way of life • Contextual information helps us make sense of it

12. Acknowledgements • Hilary Thomson and the housing review team • CPHG Review authors • CPHG Editors • Sasha Shepperd