An introduction to systematic reviews and meta-analyses

1. An introduction to systematic reviews and meta-analyses Colin Josephson Assistant Professor of Neurology University of Calgary

2. Faculty/Presenter Disclosure • Faculty: Colin Josephson • Relationships with commercial interests: • Grants/Research Support: Nil • Speakers Bureau/Honoraria: Nil • Consulting fees: Nil • Other: Nil

3. Disclosure of Commercial Support • This program has received financial support from: Nil • This program has received in-kind support from: Nil • Potential conflict(s) of interest: • Nil

4. “That’s so meta…” • Greek: prefix for ‘after’ or ‘beyond’ • English: abstraction form a concept that is used to complete or add to said concept • Unfortunately now a ‘hipsterism’

5. Pyramid of evidence

6. A lot of work…but for what? • Traditional ‘narrative’ reviews are at high risk of bias • Systematic reviews address a specific question in a standardised, reproducible manner • If low heterogeneity (statistical inconsistency) then aggregating data is possible

7. What we want to avoid… Publication bias (selective reporting) Cochrane Handbook

8. The ideal result Cochrane Handbook

9. Normal distribution

10. Sampling error

11. The ‘systematic’ approach PICOST JAMA Users Guide to the Medical Literature

12. Judge a man by his questions… • What was the question of the review? • Was it sensible, clinically relevant, and focused? • PICOST • A narrow, precise question diminishes the risk of heterogeneity, thus providing accurate conclusions

13. Examples • What is the effect of surgery on epilepsy outcomes (e.g. seizure-freedom and quality of life)? • What is the effect of resective surgery on seizure freedom? • In patients with TLE secondary to MTS (P), what is the effect of an ATL (I) compared to medical management (C) on seizure-freedom (O) in randomised controlled trials (S) measured at one-year (T)

14. The ‘systematic’ approach Statistical plan (PROSPERO) JAMA Users Guide to the Medical Literature

15. Heterogeneity • Clinical: • Population, intervention, outcome (systematic bias) • Methodology: • Study design (systematic bias) • Statistical • Variability in intervention effects (random error)

16. Means of addressing heterogeneity • Refrain from performing a meta-analysis • Explore heterogeneity through subgroup analyses or meta-regression • Pre-specify heterogeneity level (fixed-effects) • Perform a random-effects meta-analysis • Change the effect measure • Exclude studies after careful consideration

17. Fixed effects meta-analyses • Assumes one true treatment effect • Thus, differences across trials can only be due to one source of error (i.e. random error) e.g. TLE secondary to MTS Borenstein et al., 2007

18. Random effects meta-analyses • Assumes many treatment effects of which there is a ‘mean’ true effect e.g. epilepsy type

19. Random effects meta-analyses • Assumes many effects of which there is a ‘mean’ true effect • Two sources of error (random error around the true effect and random error around the mean of true effects) e.g. epilepsy type Borenstein et al., 2007

20. The ‘systematic’ approach Conducting the review JAMA Users Guide to the Medical Literature

21. Conducting the review • Comprehensive search strategy using multiple databases • Some argue that searching MEDLINE, EMBASE, and Cochrane Central is the bare minimum • Additional sources and grey literature: • Trial registries • Reference lists • Personal communications

22. ‘Searching ain’t easy…’ Josephson et al., Cochrane Database of Systematic Reviews 2014

23. Databases…just a sampler • MEDLINE • PUBMED • EMBASE • Cochrane Central • TRIP • DARE • WHO ICTRP • Google Scholar

24. Publication bias Cochrane Handbook

25. Duplication,duplication • Involvement of two or more reviewers and abstractors ensures reproducibility of the process • Why? Because it is impossible to avoid some degree of subjectivity and, thus, error • The kappa statistic is highly informative (greater IRR suggests more confidence in the process)

26. Kappa Byrt, Epidemiology, 1996

27. The ‘systematic’ approach Performing the analyses JAMA Users Guide to the Medical Literature

28. See the forest for the trees Josephson et al., Neurology, 2013

29. Pooling the effect

30. Mantel-Haenszelfixed effect Contingency table M-H weighted average

31. Mantel-Haenszelfixed effect ssRR= 0.88 ssRR= 1.16 M-H weighted average: ((400*400) + (7*5))/(1000+20) 160035/1020 156.8 RR= 1 1 2 2 1 2 = = = 0.89 ((300*600) + (2*15))/(1000+20) 180030/1020 176.5

32. DerSimonian and Laird Contingency table DSL Estimate: v = within study variance t = between study variance

33. Was it correct to pool? Josephson et al., Neurology, 2013

34. Measuring heterogeneity – Cochran Q Nomenclature: wi = individual study’s weight (1/v) Ti = individual study’s effect size T-bar = mean effect size • Form of chi-square test • p-value of 0.1 is typically used for significance • Caveat: • Low power = insensitive • High power = too sensitive Cochrane Handbook

35. Measuring heterogeneity – I2 statistic • % variability in the effect estimate that is more than chance alone (i.e. due to heterogeneity rather than random error) • 0-40% = might not be important • 30-60% = may represent moderate heterogeneity • 50-90% = may represent substantial heterogeneity • 75-100% = considerable heterogeneity Cochrane Handbook

36. Confidence = ‘GRADE’ the results GRADE Type of evidence (RCT vs. observational) Quality (blinding, allocation, f/u, sparse data, methodology) Consistency of results (within or between studies) Directness (generalisability) Effect size (e.g. >5 or <0.2 for all studies)

37. Additional quality scales Jadad: RCTs Newcastle-Ottawa Scale: non-RCTs

38. Quality summary table(e.g. QUADAS scale) Josephson et al., Cochrane Database of Systematic Reviews 2014

39. Summary of findings table Cochrane Handbook

40. Required for all studies! • Checklist outlining all essential steps of the review to facilitate quality control • www.prisma-statement.org

41. Conclusions PICOST Statistical plan (PROSPERO) Performing the analyses JAMA Users Guide to the Medical Literature

42. Thank you