Meta Analyses and Systematic Reviews

1. Meta Analyses and Systematic Reviews HINF 371 - Medical Methodologies Session 12

2. Objective • Understand what is meant by “evidence-based medicine” • Understand the types of research informing evidence • Understand meta-analyses and compare two meta analyses examples

3. Reading • Ioannidis JPA and Lau J (2000) Chapter 4: Evidence Based Medicine: A Quantitative Approach to Decision Making, in Decision Making In Health Care: Theory, Psychology and Applications, Cambridge University Press, USA • Campbell CL, Smyth S, Montalescot G, Steinhubl SR. (2007) Aspirin dose for the prevention of cardiovascular disease: a systematic review, JAMA, 2007 May 9;297(18):2018-24. • Clark RA, Inglis SC, McAlister FA, Cleland JG, Stewart S. (2007) Telemonitoring or structured telephone support programmes for patients with chronic heart failure: systematic review and meta-analysis, BMJ. 2007 May 5;334(7600):942. • Schechtman, E (2002) Odds Ratio, Relative Risk, Absolute Risk Reduction, and the Number Needed to Treat – Which of These Should We Use? Value In Health, Vol:5, No:5, pp.431 - 36

4. Evidence-based Medicine • To introduce more objective, quantifiable estimates of clinical variables to the practice of medicine • Real data is better than speculation or opinion – application of formal synthesis of evidence complements traditional “experience based” medicine • Evidence-based Medicine = literature based medicine • High quality, controlled experimental studies have long been preferable to poor quality, uncontrolled studies

5. Clinical Studies • Double blind (masked) both subjects and evaluators, randomized, placebo controlled studies • Single blind (masked) Subjects, randomized, placebo controlled studies • Open-label (unmasked) randomized, placebo controlled studies • Non-randomized, controlled, observational studies with concurrent groups • Observational studies with historical control groups – Framingham study • Uncontrolled observational studies, especially individual case reports, case series, and descriptive reports based on large databases and clinical registries.

6. Framingham Study • The original study cohort consisted of respondents of a random sample of 2/3 of adults, 30 to 62 years of age, residing in Framingham, Massachusetts in 1948. Of the original 5209, there are approximately 1095 known alive as of February 1998. • The Offspring Study was initiated in 1971 when the need for establishing a prospective epidemiologic study of young adults was recognized. A sample of 5135 men and women, consisting of the offspring of the original cohort and their spouses was established. As of February 24, 1998, there were approximately 4524 offspring surviving with only 20 lost to follow-up and 4 in whom survival status was unknown.

7. Problems with Studies • Lack of concealment (open label, historical controls) – amplification of effects • No controlled environment in observational studies – hard to link cause and effect • Studies without control groups – bias, spurious claims of large treatment effects • Length of time – Average randomized trial takes 5 to 7 years from conception to publishing

8. Characteristics of Studies • Internal Validity • Robust comparison throughout the study • Lack of adherence • Cross over to the other arm • Reduced number of follow-up patients • Missing measurements • External Validity • Study results must be generalizable to the population • More controlled the study is harder to generalize • Baseline risk of the population must be considered • Information about patient opt out of treatment is necessary

9. Measurement of End Points • Example: • Dyskinesia after Ropinirole (ROP) or Levodopa (LD) for early Parkinson’s disease • 17 out of 179 patients developed dyskinesia in the ROP arm • 23 out of 89 patients developed dyskinesia in the LD arm

10. Measurement of End PointsAbsolute Risk Reduction • Absolute risk: The observed or calculated probability of an event in the population under study • Absolute Risk Reduction (risk difference): Is the risk of an event reduced by a clinically meaningful amount • The difference between the risk of an event in the control group and the risk of an event in the treatment group • Absolute risk: ROP arm 17/179=0.095 • Absolute risk: LD arm 23/89=0.258 • Absolute risk reduction: 0.258-0.095=0.163

11. Measurement of End PointsNumber needed to treat (NNT) • Based on Absolute risk reduction. • The number of patients that need to be treated, to get the desired outcome of in one patient who would not have benefited otherwise • NNT: 1/0.163=6.13 • When negative than Number needed to Harm (NNH)

12. Measurement of End PointsRelative Risk and Relative Risk Reduction • Relative risk: the ratio of risks of the treated group and the control group, also called risk ratio • Relative Risk: 0.095/0.258=0.368 • Relative Risk Reduction: relative risk subtracted from 1. • Relative Risk Reduction: 1-0.368=0.632 • Relative Risk Reduction=ARR/Control Group risk = 0.163/0.258 = 0.632

13. Measurement of End PointsOdds and Odds Ratio • Odds: a proportion in which the numerator contains the number of times an event occurs and the denominator includes the number of times the event does not occur • Odds of ROP arm:17/(179-17)=17/162=0.105 • Odds of LD arm: 23/(89-23)=23/89=0.348 • Odds ratio: is a common measure of the size of and effect and the goal is to look at associations rather than differences. • The ratio between the odds of the treated group and the odds of the control group • The Odds ratio (OR) less than 1 means that the odds have decreased, and similarly OR greater than 1 means that the odds have increased. • Odds ratio: 0.105/0.348=0.302 – the odds of dyskinesia is reduced for the ROP arm.

14. Measurement of End PointsIntention to Treat - Cautions • Intention to treat: "Intention to treat" is a strategy for the analysis of randomised controlled trials that compares patients in the groups to which they were originally randomly assigned. This is generally interpreted as including all patients, regardless of whether they actually satisfied the entry criteria, the treatment actually received, and subsequent withdrawal or deviation from the protocol. • Better measure of Effectiveness • Intention to treat gives a pragmatic estimate of the benefit of a change in treatment policy rather than of potential benefit in patients who receive treatment exactly as planned • Full application of intention to treat is possible only when complete outcome data are available for all randomised subjects • About half of all published reports of randomised controlled trials stated that intention to treat was used, but handling of deviations from randomised allocation varied widely • Many trials had some missing data on the primary outcome variable, and methods used to deal with this were generally inadequate, potentially leading to bias • Intention to treat analyses are often inadequately described and inadequately applied

15. Other Characteristics • Adjustment – may be required in non randomized studies • Sub-group analyses – may distort the results, especially not intended in the original study • Stratification – must happen before the randomization • Multiple and Secondary end points – must be defined before the study • Significance – not a good indicator – confidence intervals are better because they allow for sensitivity analyses

16. Systematic Review of EvidenceMeta-Analyses • Meta analysis: a set of qualitative methods for statistically combining the results of different studies on the same topic to explore the degree of and reasons for heterogeneity and bias in the combined results and to provide a quantitative synthesis of these results • Example: Cochrane collaboration • Meta analyses are • Important for clinical decision making • Conducted if there is better than average quality research • Conducted if adequate information available

17. Meta Analysis Methodology • Develop a protocol • Search literature • Extract data (from literature and may be directly from authors of studies • Assess the evidence qualitatively – may be for assigning weights • Assess the evidence quantitatively – • Correlation co-efficients, mean difference in the event rate, standardized mean difference (mean difference/control group SD) • Combine treatment effects • Truth is fixed – fixed effects model – variation within studies (all studies approximated the same truth, no variation between them, so variation is zero) • Truth is not fixed – random effects model – variation within and across studies • Presenting results – confidence intervals and overall results

18. Presenting Results

19. Methods for Conducting Analyses • Assess heterogeneity • Meta regression analyses – relationship between the magnitude of the treatment effects and different predictors • Differences in sample size and variance – publication bias – funnel plot • Differences in the duration of follow-up • Effect of event rate in the control group

20. Measurement of Heterogeneity Funnel Plot Precision of the estimate of treatment effect Treatment effect Log Relative Risk or Sample size

21. Meta AnalysisSpecific types • Meta-analyses of diagnostic tests – receiver operating characteristics = trade off between sensitivity and specificity • Meta-analyses of individual patients • More detailed time to event analyses • Ability to generate individual based predictive models • But retrieval bias and cumbersome

22. Other concerns • Non-statistically significant results - our concern is magnitude of treatment effects or the strength of the association • Use in public policy – averages might be good but identifying specific populations are also very useful • Sensitivity analysis - might be necessary to identify the most beneficial patient group