1 / 34

EPI-820 Evidence-Based Medicine

EPI-820 Evidence-Based Medicine. LECTURE 10: Meta-Analysis II Mat Reeves BVSc, PhD. 4. Analysis. Several considerations Primary focus/motivation of study? Summary effect or explore heterogeneity? What kind of data are you combining? Dichotomous (categorical) (OR/RR, RD )

carolena
Télécharger la présentation

EPI-820 Evidence-Based Medicine

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EPI-820 Evidence-Based Medicine LECTURE 10: Meta-Analysis II Mat Reeves BVSc, PhD

  2. 4. Analysis • Several considerations • Primary focus/motivation of study? • Summary effect or explore heterogeneity? • What kind of data are you combining? • Dichotomous (categorical) (OR/RR, RD ) • Continuous (effect size [diff/SD]) • Diagnostic data (sensitivity and specificity) • Fixed vs random effects model?

  3. Analysis – Primary Goal • What should be the primary goal? • To provide a summary estimate or explore of presence and sources of heterogeneity? It depends. • If studies are homogeneous then generate a summary estimate with 95% CI • Much more likely to happen in RCT’s where randomization has helped control bias and confounding • If studies are heterogeneous then focus of study should be to investigate the sources of this variability • More likely to happen in observational studies where differences in populations, methods and uncontrolled bias and confounding are rampant. But also occurs in RCT’s.

  4. Exploring heterogeneity • Heterogeneity is the norm rather than the exception • Heterogeneity can result from • Methodological differences • Biological differences • Heterogeneity Statistics (Q) • Power generally low (because study N’s are typically small) • Power also affected by size of deviations between studies • Statistical vs clinical heterogeneity • What is the size of the statistical heterogeneity? Does it make sense? Could it have arisen due to random error? (chance)

  5. Sources of clinical heterogeneity • Clinical heterogeneity can be due to differences in: • study design or characteristics: • hospital vs population-based observational designs • DBPC vs open trials • Study population (sources), study quality • Selection criteria for subjects, treatments or follow-up • Sub-group responses (biological interaction) – esp. in RCT’s • Bias or confounding – esp. in observation studies • Explored using stratification/sub-group analyses • See Bernal, 1998

  6. RR of Vasectomy on Prostate CA Risk Effect of study characteristics/quality (Bernal, 1998)

  7. Fixed and random effects models • Homogeneity and heterogeneity • Heterogeneity depends on the degree of between-study variability in a group of studies. • Fixed effects models: • consider only within-study variability. • assumption is that studies use identical methods, patients, and measurements; that they should produce identical results - any differences are only due to within-study variation only. • Answer the question: • “Did the treatment produce benefit on average in the studies at hand?”

  8. Random Effects models • consider both between-studyandwithin-study variability. • assumption is that studies are a random sample from the universe of all possible studies. • Answer the question: • “Will the treatment produce a benefit ‘on average’?” • Note that random effects models do not “adjust for”, “account for”, or “explain” heterogeneity • A random effects model does not therefore solve the problem of heterogeneity!

  9. Fixed and random effects models • Can give very different answers, and you can create examples where either model gives counterintuitive results (see Petitti, page 96) • Random effects gives non-significant summary statistic for two studies that are each significant • Fixed effects model gives the same confidence interval when you would expect a broader and narrower CI • Usually, though, answers are similar. • Example: Comparison of 22 meta-analyses, fixed and random effects models gave the same answer in 19/22. In 3 cases, fixed effects models were significant while random effects models were not (Berlin, 1989).

  10. Fixed and random effects models • Differences only arise when studies are not homogenous. • When there is significant heterogeneity, the between-study variance becomes much larger than the within, and studies of different sample size receive relatively similar weight. • When there is homogeneity, sample size dominates, and both models give similar results. • Random effects models are more “conservative” and generate a wider confidence interval (because they add in the between-study variance). • Random effects models also tend to give greater weight to small studies (which maybe more biased?)

  11. What to do?, what to do? • If homogenous, use fixed effects model • random will give same results • fixed is computationally simpler • If heterogeneous…then first ask why?! • In the face of heterogeneity, focus of analysis should be to describe possible sources of variability - attempt to identify sources of important subgroup differences • Example: studies using one dose showed significant effect, while lower dose did not. Then do fixed effects analysis of each sub-group and report all results.

  12. Use of the Random Effects Model?…. • Many observers dispute the rationale for random-effect based analyses. For example: • Petitti (2000) “…. in the very situations where application of the method matters (= heterogeneity), a single summary estimate of effect is inappropriate” • Greenland (1994) – “the random effects model is the model or summary of last resort”

  13. Statistical Tests of Homogeneity (heterogeneity) • Homogeneity calculations • Ho = studies are homogeneous • Based on testing the sum of weighted differences between the summary effect and individual effects • Calculate Mantel Haenszel Q, where: • Q = [weighti x (lnORmh - lnORi)2] • To interpret, use the chi-square distribution where the degrees of freedom = S - 1 (where S is the number of studies). If p < 0.05, then there is significant heterogeneity. • Power of such statistical tests is low (a non-significant test does not rule out clinically important heterogeneity)

  14. Specific methods for dichotomous data • Mantel-Haenszel method (fixed effects) • originally developed to handle analysis of data in multiple strata. If you think of each study as a stratum, you can do a meta-analysis! • data must be in form of 2 x 2 table for Mantel-Haenszel • odds ratio, rate ratio, risk ratio • Most commonly used method for meta-analysis (has optimal statistical properties) • Only accounts for confounding if it is incorporated into the study design (matching or randomization) • therefore, can’t use multivariable adjusted data.

  15. Mantel-Haenszel Method

  16. Mantel-Haenszel Method ORmh =  (weighti x ORi) /  weighti • ORi = (ai x di) / (bi x ci) • weighti = 1 / variancei • variancei = ni / (bi x ci) • 95% CI = e ln(ORmh) +/- 1.96 x sqrt(var ORmh) • var ORmh = • (F / 2 x R2) + [G / (2 x R x S)] + (H/(2 x S2) • where: • F = [ai x di x (ai + di)]/ni2 • G = [ai x di x (bi+ci)] + (bi x ci x (ai + di))] / ni2 • H = (bi x ci x (bi+ci)) / ni2 • R = (ai x di) / ni • S = (bi x ci) / ni

  17. (Sir Richard) Peto Method • Fixed effects • very similar to Mantel-Haenszel method (same 2x2 requirement) • see Pettiti pages 104-107 or Hasselblad article for formulae • computationally somewhat simpler, especially to calculate the confidence interval • may provide biased results under some circumstances in which Mantel-Haenszel would not • Best applied to RCT’s and not observational studies

  18. General Variance Methods • Used to summarize rate/risk differences (RD) • Fixed effects method • RDs = (wi x RDi) / wi • wi = 1 / variancei • 95%CI = RDs +/- 1.96 (variances)0.5 • Variances = 1/wi • see text page 107 for more details • formulas differ if analyzing rate ratio data (incidence-density) or risk ratio data (cumulative risk) • General variance-based methods also used for observational studies when study results are presented as RR with 95% CI

  19. Random effects models • DerSimonian and Laird statistic • Uses odds ratios only! • lnORdl = (wi* x lnORi) / wi* • wi* = 1 / [D + (1/wi)] • wi = 1 / variancei • D = ([Q - (S - 1)] x wi ) / [(wi)2 - wi2] • Q = [wi x (lnORi - lnORmh)2] • CI = exp(lnORdl + 1.96 x (variances*)^0.5 • variances* = weighti*

  20. Continuous outcomes • Two approaches: • 1. Each study used the same scale or variable (i.e. all measured SBP, serum creatinine or Mini-Mental State score). Based on ANOVA model where studies are “groups”. • meansummary = (weighti x meani) / weighti • meani = meantx - meancontrol • weighti = 1 / variancei = 1 / SDi2 • (use pooled variance) • 95% CI = means +/- (1.96 x (variances)^0.5) • variances = 1 / weighti • Test of homogeneity: Q = [weighti x (means - meani)2]

  21. Continuous outcomes • 2. Each study used a similar but different scale (e.g., CAGE and MAST for diagnosis of alcoholism, pulmonary function tests [PEFR, FEV1]) • dsummary = (weighti x di) / weighti • dsummary = summary estimate of the difference in effect sizes • di = effect size = (meantx - meancontrol) / SDpooled • weighti = 1 / variancei = (2 x Ni) / (8 + di2) • (use pooled variance) • 95% CI = ds +/- (1.96 x (variances)^0.5) • variances = 1 / weighti • Test of homogeneity: Q = [weighti x (ds - di)2]

  22. Other Issues in Meta-Analysis • Cumulative M-A • See article by Antman for example • Pooling Studies • See article by Blettner (Type III study) • M-A of observational studies • M-A of diagnostic tests • Meta-regression

  23. M-A of Observation Studies • Very controversial application with some authors rejecting the approach outright (Shapiro, 1994) • Often applied to controversial topics where previous studies are inconclusive (due to small risks and/or small studies) • Exam: Chlorination and CA risk, EMF and CA risk. • But can never exclude bias……. • Important to regard process as a “study of studies” and not a means of providing a summary estimate • Very valuable process at identifying deficiencies in published literature • See Stroup et al (JAMA 2000) – proposal for reporting

  24. Meta-analysis of diagnostic tests • See Irwig article (bibliography) for an excellent overview. • Simply averaging sensitivity and specificity is not useful: SeSp • Study 1 0 100 • Study 2 99 99 • Study 3 100 0 • Mean 67 67

  25. What to do? • Can calculate a summary ROC curve, by plotting the sensitivity and specificity for each study of a diagnostic test. • Especially useful for comparing tests • e.g. stress thallium vs stress echocardiogram for heart disease. • See Irwig article for details of calculations.

  26. Plotting an ROC curve O O O Se O Each circle represents an individual study O 1 - Sp

  27. Figure 3. Summary receiver-operating characteristic (SROC) curve analysis of ELISA D-dimer in the diagnosis of PE. Plotted in each of the SROC graphs are individual studies depicted as ellipses. The x- and y-dimensions of the ellipses are proportional to the square root of the number of patients available to study the sensitivity and specificity, respectively, within the analysis. Also shown is the unweighted SROC curve limited to the range where data are available. The cross (x) represents the independent random-effects pooling of sensitivity and specificity values of the studies.

  28. Meta-regression • Multivariate approach: • Use the study characteristics as independent variables • Design, age, population source, quality score etc etc • Use effect size or other outcome as the dependent variable • Identify significant study characteristics • Unit of observation = study • Can be useful to identify sources of heterogeneity, clarify importance of quality scores • Exploratory only

  29. Meta-regression – Example(Phillips 1991: 26 HIV studies, Dependent var = Specificity)

  30. Final comments • Remember the “art” of meta-analysis: knowing when to use which technique, rather than “mindlessly” applying formulae to studies. • Understanding the underlying clinical rationale for treatment, differences in populations, and differences in outcomes is critical. • An important contribution of MA is to highlight the variability in the design, conduct, analysis and findings of a particular body of literature.

  31. Bibliography • Highly recommended reading • Hasselblad V, McCrory DC. Meta-analytic tools for medical decision-making: a practical guide. Med Decis Mak 1997; 15: 81-96. • Irwig L, Tosteson AN, Gatsonis C, et al. Guidelines for meta-analyses evaluating diagnostic tests. Ann Intern Med 1994; 120: 667-76.

  32. Other recommended reading • Cook DJ, Guyatt GH, Ryan G, et al. Should unpublished data be included in meta-analyses? JAMA 1993;p 269: 2749-53. • Greenland S. A critical look at some popular meta-analytic methods. Am J Epid 1994; 140: 290-6. • L’Abbe K, Detsky AS< O’Rourke K. Meta-analysis in clinical research. Ann Intern Med 1987; 107: 224-33. • LeLorier J, Gregoire G, Benhaddad A, et al. Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med 1997; 337: 536-42. • Eddy DM, Hasselblad V, and Schachter. An introduction to a Bayesian method for meta-analysis. Med Decis Mak 1990; 10: 15-23. (REQUIRES SPECIAL SOFTWARE) • Sacks HS, Berrier J, Reitman D, et al. Meta-analyses of randomized controlled trials. N Engl J Med 1987; 316: 450-5.

  33. Other recommended reading • Cook DJ, Sackett DL, Spitzer WO. Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on Meta-Analysis. J Clin Epidemiol 1995; 48: 167-71. • Chalmers TC, Smith H, Blackburn B, et al. A method for assessing the quality of a randomized control trial. Control Clin Trials 1981; 2: 31-49. • Sackett DL. Applying overviews and meta-analyses at the bedside. J Clin Epidemiol 1995; 48: 61-6. • Olkin I. Statistical and theoretical considerations in meta-analysis. J Clin Epidemiol 1995; 48: 133-46.

  34. Sample meta-analyses • Clark P, Tugwell P, Bennett K, Bombardier C. Meta-analysis of injectable gold in rheumatoid arthritis. J Rheumatol 1989; 16: 442-7. • Rowe BH, Keller JL, Oxman AD. Effectiveness of steroid therapy in acute exacerbations of asthma: a meta-analysis. Am J Emerg Med 1992; 10: 301-10. • Cummings P. Antibiotics to prevent infection in patients with dog bite wounds: a meta-analysis of randomized trials. Ann Emerg Med 1994; 23: 535-40. • Callahan CM, Drake BG, Heck DA, Dittus RS. Patient outcomes following tricompartmental total knee replacement: a meta-analysis. JAMA 1994; 271: 1349-57. • Phillips KA. The use of meta-analysis in technology assessment: a meta-analysis of the enzyme immunosorbent assay HIV antibody tests. J Clin Epidemiol 1991; 44: 925-31.

More Related