Aims of this session….

1. Aims of this session…. What is a systematic review? How to formulate an appropriate review question? Why and how to write a protocol? How to find relevant studies? How to extract appropriate data? How to critically appraise studies? Introduction to synthesis BRIEF introduction and signposting!

2. What is a systematic review? SYSTEMATIC: Done or acting according to a fixed plan or system: methodical REVIEW: A critical appraisal of a book, play or other work

3. What is a systematic review? “A systematic review is a review in which there is a comprehensive search for relevant studies on a specific topic, and those identified are then appraised and synthesized according to a predetermined and explicit method.” (Klassen 1998) A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing more reliable findings from which conclusions can be drawn and decisions made (Antman 1992, Oxman 1993)

4. Why use systematic reviews? • Minimise the impact of bias/errors • Can help to end confusion • Highlight where there is not sufficient evidence • Combining findings from different studies can highlight new findings • Can mitigate the need for further trials

5. Why use systematic reviews? • Facilitate rational decision making • Health care providers, researchers and policy makers are inundated with unmanageable amounts of information • Over 20 million citations in PubMed • Approx. 75 to 100 RCTs published daily • Usually impossible to consider all relevant individual primary research studies in a decision making context • Enable practitioners to keep up to date and practice evidence-based medicine

6. Why not traditional reviews?

7. Hierarchy of evidence

8. Who undertakes systematic reviews? • Cochrane/Campbell Collaboration • NICE/Regulatory bodies • Health Technology Assessment • Academics/researchers/clinicians • MSc/PhD students

9. Who undertakes systematic reviews? • Multidisciplinary teams • Clinicians • Health services researchers • Information scientists • Statisticians • Health economists • Patient and public involvement

10. Systematic Reviews - the process (1) • Define review question – very precisely; in partnership with commissioners, clinicians, patients (as appropriate) • Develop protocol – provides transparency; defines exact inclusion criteria and methods • Identify relevant studies – usually comprehensive search across multiple bibliographic databases plus reference checking • Assess eligibility – careful matching of studies against inclusion criteria

11. Systematic Reviews - the process (2) • Extract relevant data – only what is required to answer the question • Critically appraise studies – use a published tool to compare methodological features across studies • Synthesise appropriately – depending on type of question and study designs quantitative or qualitative synthesis • Disseminate to appropriate audience – full transparent write-up plus as appropriate short report to funder, journal article, patient leaflet…

12. Formulating the review question

13. Formulating review question • Questions may be broad or narrow • Well-formulated questions will guide many aspects of the review process • Searching strategy • Inclusion/exclusion criteria • Data extraction • Choice of synthesis method • Presentation/dissemination of findings

14. Quick Activity • Discuss a very broad question and how you might narrow it? (10 mins) • Discuss the potential limitations of your review questions • If time and resource were not a limitation – consider how useful would the answer to your review question be?

15. Current guidance • a clear and concise statement of a review's objectives (or questions) is critical and should begin with a precise statement of the primary objective, including the interventions reviewed and the targeted problem; ideally, this would be presented in a single sentence Cochrane & PRISMA Statment

16. Current guidance - Cochrane • “To assess the effects of [intervention or comparison] for [health problem] in [types of people, disease or problem, and setting if specified].”

17. Formulating review question • Determining the scope is a decision dependent upon multiple factors: • Perspectives regarding a question’s relevance and potential impact; • Supporting theoretical, biologic and epidemiological information; • The potential generalizability and validity of answers to the questions; • Available resources; • The wider literature base – has a recent high-quality SR been conducted?

18. Formulating review question • Often dealing with complex interventions • Might be a need to develop working definitions of the intervention of interest • Several options on how to do this (pragmatic real world v theoretical, logic models, etc.) • Use content experts outside the review team to ensure that the resulting definitions are likely to be robust and meaningful • Need to carefully consider appropriate study design which will answer your particular question

19. Formulating review question

20. Formulating review question: tools (1) PICO is for effectiveness questions which usually use controlled trials Glasziou et al, 2003

21. Formulating review question: tools (2) SPICE is designed for questions around public health interventions 1. Setting what is the context of the question? e.g. European countries 2. Perspectivewho are the users/potential users of the outcomes? e.g. chronically ill 3. Interventionwhat is being done to them? e.g. increased cost-sharing 4. Comparisonwhat are the alternatives? e.g. no increase 5. Evaluationhow will you measure if the intervention is successful? e.g. access to health services Booth 2004

22. Formulating review question: tools (3) Sample a description of what is likely to be a smaller group of participants Phenomenon (of) may not be an intervention or exposure Interest Designdetails of the design will help to make decisions about the robustness of the research Evaluationwhat are you measuring? Research typequalitative, mixed methods? Cooke et al 2006

23. Any questions??

24. Developing the protocol

25. Developing protocol • A protocol is an essential component of the systematic review process • Helps to ensure careful a priori planning of whole process • Consistency • Transparency • Integrity • Journals often require a protocol registration number before publishing systematic reviews

26. Developing protocol • One of the features that distinguish a systematic review from a traditional review is the pre-specification of criteria and methods • What studies will be included and excluded? (use previous question formulation tools) • How (exactly) will each stage be conducted? (search strategy – consult librarian, study selection, critical appraisal, synthesis method – consult statistician) • What are the primary and secondary outcomes of interest?

27. PROSPERO – CRD initiative • Search for ongoing reviews • Register planned review online (& update as work progresses) • Avoids duplication of reviews

28. Finding eligible studies

29. Searching for eligible studies • Can use the same tools to help define search as for question formulation (PICO, SPICE, SPIDER) • Use only the items needed to define search (usually at minimum population and intervention/phenomenon of interest), sometimes outcomes • Consult information specialist or librarian for advice (the search is the first potential major source of bias in your review)

30. Searching for eligible studies • Thesaurus headings (e.g. MeSH in MEDLINE) • Controlled vocabulary thesaurus used for indexing articles • Standardised search terms regardless of how author has described the study • Each database has a different thesaurus – must translate

31. Searching for eligible studies • Keywords • Also search title and abstract fields in case of imprecise or non existent indexing • Need synonyms and truncation to search word stems (e.g. child* retrieves child, children, childless, childbirth, etc.)

32. Searching for eligible studies • Where to search • Electronic databases according to the topic area: MEDLINE, EMBASE, Cochrane databases, PsycInfo, etc. • Grey literature, dissertations, theses, conference proceedings, national bodies (NICE, HTA), clinical trial database (www.clincialtrails.gov/) • University library web pages give advice about databases available in different topic areas • Check existing systematic reviews in a similar area for ideas • Look at the databases own guidance for searching they vary! • Download results to reference management software (e.g. University supports Endnote, alternatively Mendeley)

33. Boolean operators

34. Selecting included studies • Select potentially eligible abstracts (in Endnote) • Assess titles/abstracts against your predetermined inclusion/exclusion criteria, use groups or custom fields to record • Retrieve full text articles of initial selections • Assess full text for inclusion • Requires judgement (>1 reviewer) • Check reviewer agreement (3rd review to resolve) • Use a selection form to ensure consistency and record decisions

35. Any questions??

36. Data Extraction

37. Data Extraction (1) • Pilot a data extraction form • See options later • Collect the following information: • Citation details to identify the study • Characteristics of population/intervention • Outcomes at baseline and follow-up • Methodological details according to chosen critical appraisal tool

38. Data Extraction (2) • What effect measures do you want to calculate? • What data do you need to do this? • How are you planning to group studies for the analysis? • By intervention, study design?? • What information do you need to extract to enable you to organise and analyse the way you want?

39. Data Extraction (3) • Extract sufficient information: • to describe studies • to allow you to undertake the planned analysis • so you do not need to return to the full text papers • But not too much • Don’t waste time extracting ‘nice to have’ but unnecessary detail REMEMBER YOUR PROTOCOL IT IS YOUR ROADMAP, FOLLOW IT!

40. Data Extraction (4) • There is a wide selection of software to choose from • Consider: • What are you are familiar with? • What package best suits your data? • How many included studies do you have?

41. Software Options • Word • Excel • Access • EPPI reviewer • COVIDENCE • REVMAN • ????

42. Consistency/Standardisation • Data extractors need to collect and interpret data in a standard way • Essential >one reviewer is extracting data • Independent piloting of data extraction forms – always one standardised form • Regular discussion of progress/disagreements • Regular comparison of data extraction – don’t wait till the end

43. Efficient data extraction • Once data extraction is complete you may need to: • Sort/search your data • Filter data • Calculate frequencies • Transform data (e.g. SE to SD) • Categorising/coding data will make these tasks easier: • Needs to be implemented with consistency by the whole team

44. Things to consider • Are you including more than one study design? • You may need separate forms for each study design • However, you are still answering the same question, so make sure the core information extracted is the same • You may need to update the form, or have more than one form • Any changes need to be agreed and made consistently

45. Things to consider • Have one or a few studies reported data differently from the others? • Will the data still be useful? • Should you include it? • Make sure the core information extracted is the same • You may need to update the form, or have more than one form • Any changes need to be agreed and made consistently

46. Any questions??

47. Critical Appraisal

48. Critical Appraisal • Why - what are we trying to achieve? • Not all published and unpublished literature is rigorous! • being in a journal doesn’t mean it is good • Quality may be used as an explanation for differences in study results or to guide interpretation of findings, strength of inferences

49. Critical Appraisal • Quantitative studies • Internal validity • Bias: selection; performance; detection; attrition; reporting • External validity • Better to use domain based assessment not numerical scores • Cochrane Risk of Bias 2.0 – RCTs (Higgins 2016) • QUADAS 2 – diagnostic accuracy (Whiting 2006) • ROBIS for systematic reviews (Whiting 2016)

50. Critical Appraisal • Qualitative studies • Three broad categories • Rigour: has a thorough and appropriate approach been applied to key research methods in the study? • Credibility: are the findings well presented and meaningful? • Relevance: how useful are the findings to you and your organisation?