Systematic Review: Workshop 1 26 July 2007 warwick.ac.uk/go/chrisbridle/sr

1. Systematic Review:Workshop 126 July 2007www.warwick.ac.uk/go/chrisbridle/sr Chris Bridle, PhD, CPsychol Institute of Clinical Education Warwick Medical School University of Warwick Tel: +44(24) 761 50222 Email: C.Bridle@warwick.ac.ukwww.warwick.ac.uk/go/hpsych

2. Workshop Overview • Systematic review 70 mins • Break 15 mins • Programme structure 10 mins • Protocol development 45 mins • Questions -

3. Why is research synthesis needed? • 2 million articles published in 20,000 health journals pa • Equivalent to a pile of paper 900 metres high • If you tried to stay current by reading two articles each day, within 1 year you would fall 55 centuries behind! • Put another way, if you try to read everything of possible relevance, you would have to read 5,500 articles per day. Seeing is believing ... 381m ‘What burns me up is that the answer is right here somewhere, staring us in the face … … if only Chris B was here!

4. Expert Advice?

5. OR (fixed) OR Study 95% CI (fixed) [95% CI] 101 3.47 [0.35, 14.51] 102 5.10 [0.24, 18.96] 103 2.43 [0.21, 12.71] 104 3.40 [0.35, 13.43] 105 2.43 [0.43, 11.69] Total (95%CI) 3.06 [1.13, 8.31] Heterogeneity: Chi²=0.23, df=4 (P=0.92) Effect: Z=2.20 (P=0.03) .1 0.2 0.5 1 2 5 10 Favours treatment Favours control What and When? What? Thalidomide When? Before 1960 The evidence was there, but the methods for making it accessible and meaningful were not ◊ Treatment <Favours> Control(Fewer <Adverse Events> More)

6. Don’t forget the slides handout!

7. Why we need research synthesis • Vast quantity of research  • Single trials rarely definitive  • Hidden effects  • Research and practice built upon  biased &/or unfounded assumption • Need for research synthesis  How can research evidence be synthesized?

8. Chalk and Cheese

9. Traditional Review: Take a simmering topic, extract the juice of one argument, add the essence of one filling cabinet, sprinkle liberally with your own publications and sift out the work of noted detractors A non-scientific method prone to bias in evidence retrieval, ‘evaluation’, synthesis and interpretation Systematic Review: A review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, evaluate and synthesise relevant primary research A scientific methodology that is reproducible and amenable to rigorous evaluation You Decide

10. Formulate Review Question Key stages in conducting a systematic review Develop Protocol Search, Identify & Select Evidence Data extraction Quality Assessment Data Synthesis & Interpretation Conclusions & Recommendations Write Review Report

11. Formulating a Review Topic / Question • Example question: Effects of interventions for overweight children • This is a bad question! • It is much too broad, would take years to answer, and results unlikely to be meaningful • Needs to be narrower, i.e. more focussed • What effects? • Effects of what exposure? • Among who? • Compared to what / who? Etc., etc., etc.

12. Focussing the Review Overweight children Community walking Orilstat Weight loss Answerable questions include: • A description of the population P • An identified interventionI • An explicit comparison C • Relevant outcomes O Community walking versus orilstat for weight loss among overweight children

13. PICO Components • Identify the PICO components: • Effectiveness of homeopathy • Stage-based interventions for smoking cessation • Psychosocial interventions for prevention and treatment of childhood obesity • Effectiveness of atypical antipsychotics for treatment of mania in bipolar disorder

14. The Review Protocol • The review protocol is the first major milestone of any systematic review • Provides a rigid, well-specified plan for how each stage of the review will be conducted – a roadmap • Helps to avoid or minimise bias at each stage of the review – but only if followed rigidly • Should be sent for external peer review, e.g. advisory group comprising researchers, practitioners, users, etc. • Should be published, e.g. project webpage, research register, or peer-reviewed journal

15. Comprehensive Literature Search • Necessary to reduce bias: • Publication, language, and geographical • Search media: • Electronic, manual, and personal • Publication type: • Journal, conference, dissertation, grey, and ongoing

16. For each PICO component identify and record: Abbreviations, e.g. CBT, TTM, MS, CFS Synonyms, e.g. CBT - Theory or Therapy Related terms: cognitive theory/therapy, behavior theory/therapy Transatlantic differences, e.g. pressure sores / decubitus ulcers Spelling, e.g. behavior or behaviour Set limits? Publication, e.g. published; language; date, etc. Study, e.g. sample size, follow-up time, type of outcome assessment, etc. Evidence reliability, e.g. study design; specific criteria … Hierarchy of evidence From question to search terms

17. Systematic Review RCT Experimental Study + + Potential For Bias - - Cohort Studies Case-Control Studies Observational Studies Case Reports Cross Sectional Surveys You, … and me Expert Opinion Drunk Bob Anecdotal

18. Question – Evidence Relationship Question Type Relevant Evidence Intervention - SR, RCT, Cluster RCT Prognosis - Cohort study Etiology - Cohort, case-control Harm - Cohort, case-control Diagnosis - Cross-sectional, case-control Experience - Qualitative research

19. Building a Search Strategy • Record everything!* • Search history, e.g. database, host and date • Search strategy, e.g. terms and combinations • Search results • Design a search-term table • Enter search terms in relevant column • Combine search terms for each column with ‘or’ = more • Combine column results with ‘and’ = less * See example

20. Preparing to be flooded How will you manage the many (thousands) of references and abstracts your search has identified? • Bibliographic software • Endnote • RefWorks • ProCite • RevMan

21. Study Selection • What references to retrieve? • Study information limited to Abstract or Title • Assess potential relevance of studies against inclusion / exclusion criteria (i.e. P.I.C.O) – retrieve relevant full text papers • What studies to include? • Use full text papers to assess the relevance of studies against the study inclusion / exclusion criteria - PICO • The remaining papers are those relevant to your review • Keep precise records from start to finish • Account for all references (studies)

22. Example: Flow chart of study papers through a reviewStudy papers are to a review, what participants are to a trial

23. Data Extraction • Recording of information presented in primary studies • Ranges from superficial reporting to systematic recording • Systematic extraction / recording is important for data synthesis and interpretation • Important to strike the right balance between reporting / recording too much or too little information • Determined by the ‘nature’ of the review and its intended use / audience

24. Extracting Study Data • Important data to extract: • Participants: demographic / disease characteristics • Intervention: description of intervention and control • Analysis: statistical method, number of Ps, ITT analysis • Results: be selective and sensible, e.g. primary outcome • Authors’ conclusions: word-for-word reporting of conclusions • Reviewer’s comments: validity of results, calculation of summary (effect) statistic, etc.

25. Data Extraction: Example

26. Tabulated Data: Example • Tabulated data are usually presented in the text of the report / paper

27. Critical Appraisal • Estimates potential for bias • Extent to which results attributable to factors other than named intervention • Independent application of a priori criteria • Criteria depend on study design • Common criteria for RCTs: • Adequate randomisation; Concealed allocation; a priori sample size calculation; Blinding; Baseline comparability; Adequate follow-up; Intention-to-treat analysis

28. Importance of Criteria

29. Uses of Quality Assessment • Determine the strength of conclusions • A threshold for inclusion • Potential source of heterogeneity • As weights in statistical analyses – unwise! • Recommendations for improving future research

30. Data Synthesis • Quantitative: • statistically combine results of several studies into a single numerical estimate of effect, e.g. meta-analysis • Only used for (some) quantitative data • Qualitative: • a narrative summary and synthesis of primary studies • Used for both quantitative and qualitative data

31. Selecting a Method of Synthesis • Decision based on degree of heterogeneity • Variation within the group of included studies • Types of heterogeneity: • Clinical: participants, interventions and outcomes, e.g. age, intervention intensity, outcome definition, etc. • Methodological: how studies were conducted, e.g. study design, study quality • Statistical: measured intervention effect – you should calculate a summary statistics for each study yourself

32. Workplace exercise intervention for mild depression

33. Assessing Statistical Heterogeneity • Visual inspection • Overlap of CIs • Chi-square statistic • Significant p<.01 • Power, too much, too little • Statistic Vs degrees of freedom • Meaningless question • Is heterogeneity present? Of course it is! • We want to know how much heterogeneity is present, where, and is it important?

34. Dealing With Heterogeneity • Subgroup analyses • Comparison of separate meta-analyses based on subgroups of the studies • Sensitivity analyses • Repeating meta-analyses with amended criteria to determine robustness of review findings

35. Subgroup analyses: Mortality results of zidovudine trials in HIV, stratified by infection stage (early or late).

36. Sensitivity analyses:Case-control studies relating residential exposure to electromagnetic fields to childhood leukaemia, stratified by methodological quality.

37. Caution • Subgroup and sensitivity analyses need to interpreted with caution • Strong conclusions are often drawn from subgroup / sensitivity analyses –misleading and dangerous • Results based on indirect comparisons • Any differences are observational in nature • Prone to bias and confounding

38. Narrative Synthesis • Used for both quantitative and qualitative data but no formal method of synthesis • Studies meaningfully grouped and explored for potential sources of effect heterogeneity • Participants: age, gender, ethnicity, SES • Interventions: setting, provider, intensity, duration • Outcomes: definition, measured, collection, follow-up • Methods: quality, design, sample size, protocol adherence

39. Conclusions • Must be based solely on the evidence reviewed • Should highlight implications for research, practice, policy, decision makers, etc. • May be graded according to the strength of the evidence • May indicate a time period within which the findings of the review are relevant

40. Core Systematic Review Stages Formulate Review Question Advisory Group Scoping Search Develop Protocol Publish Protocol Independently Assessed Search, Identify & Select Evidence Reasons for Exclusion Check for accuracy Data extraction Review Processes - Inputs Review Outputs Independently Assessed Quality Assessment Investigate Heterogeneity Data Synthesis & Interpretation Advisory Group Review Drafts Conclusions & Recommendations Develop Research Bid Publish Review Write Review Report

41. Critical Appraisal of SRs • Question Is the review question clear, specific and answerable? • Search Have attempts to identify relevant evidence been sufficiently comprehensive? • Evaluation Have included studies been critically appraised against appropriate criteria? • Synthesis Is the method of synthesis appropriate, and have potential sources of heterogeneity been investigated? • Conclusions Do conclusions reflect both the quality and quantity of evidence? • Process Is the review process independent and transparent so as to limit potential bias?

42. Summary • Systematic reviews are important: • Meet the need for research synthesis • Use scientifically rigorous methods • Provide evidence-based answers to meaningful questions • Identify gaps and weaknesses in evidence base • Improve research quality • Enhance effectiveness of clinical practice • Strengthen bids for research funding

43. Good Guidance Centre for Reviews & Dissemination: Report 4 www.york.ac.uk/inst/crd/report4.htm Cochrane Collaboration: Reviewer’s Handbook www.cochrane.org/resources/handbook/index.htm Cochrane Behavioral Medicine: Recommended paper http://www.cochranebehavmed.org/ WMS Systematic Review Training Materials www.warwick.ac.uk/go/chrisbridle/sr

44. Any questions? Now, or later Chris Bridle PhD, CPsychol Institute of Clinical Education Warwick Medical School University of Warwick Tel: (024) 761 50222 Email: C.Bridle@warwick.ac.uk Web: www.warwick.ac.uk/go/chrisbridle

45. Programme Overview