Approaches to Synthesis of Heterogeneous Evidence

1. The following slides were presented at a meeting of potential editors and methods advisors for the proposed Cochrane review group in February 2008. The slides were designed to promote discussion rather than represent the views and directions of this group.

2. Approaches to Synthesis of Heterogeneous Evidence Randy Elder, PhD, MEd Scientific Director for Systematic Reviews Guide to Community Preventive Services National Center for Health Marketing Centers for Disease Control and Prevention (CDC)

3. It is the mark of an educated man to look for precision in each class of things just so far as the nature of the subject admits.Aristotle, c. 350 BC

4. Goals • Review of tools for addressing heterogeneity • Discuss utility of descriptive statistical approaches when MA is not feasible • Raise conceptual and practical issues related to heterogeneity • Substantive sources of variance • Methodological sources of variance

5. Methods for SynthesizingHeterogeneous Data

6. Inferential Statistical Approaches: Meta-analysis • Requires sufficient homogeneity for estimate of central tendency to be useful • Likely to be relatively uncommon in HPPH reviews • Less complex interventions are most likely candidates • E.g., safety belt laws • Subgroup analysis can be used to account for some heterogeneity

7. Inferential Statistical Approaches: Meta-regression • Able to account for sources of heterogeneity in more complex interventions • Partially addresses colinearity issues that bedevil univariate subgroup analyses • Potentially useful for selected interventions with large evidence base • E.g., some school-based interventions • Pitfalls include • Poor reporting/measurement of effect modifiers • Underpowered analyses of effect modification • Potential for false positives with multiple comparisons • Susceptibility to ecological fallacy

8. Descriptive Approaches:Narrative Synthesis • Likely to be the most common approach for complex HPPH reviews • ESRS guidance on narrative synthesis is a valuable tool for editors and authors • Pro • Can be applied to any data • Often only option given heterogeneous interventions, populations, and outcomes • Allows thoughtful synthesis of small bodies of evidence • Con • Challenging for larger bodies of evidence • Tabular and graphical techniques can be helpful additions • More prone to biased interpretation • E.g., temptation to engage in vote-counting • More difficult to evaluate effect modification

9. Use of Descriptive Statisticswith Narrative Synthesis • Descriptive summary statistics can provide a useful supplement to tabular and graphical methods • Facilitate simple, concise text summaries of distribution of study results • What is the central tendency? (e.g., median) • How much variation in results can be expected? (e.g., range, interquartile interval)

10. Conceptual and Practical Issuesin Addressing Heterogeneity

11. Accounting for Heterogeneity: Effect Modification and Subgroup Analysis • HPPH and ECRS guidance on subgroup analysis* • Do it (within reason and with theoretical justification) • Report results • Interpret them cautiously • This has the practical benefits of providing end-users with information they need • Decisions re: when, where, how, and with whom to implement interventions need to be made • Any information is preferable to none • An a priori assumption of homogeneity is a far less conservative approach • Analyses done from a hypothesis-testing perspective face issues of confounding and tend to be underpowered (substantial risk of Type II error) • Not doing analyses effectively guarantees Type II error (of uncertain magnitude) *As I understand it

12. Incorporating Non-randomized Studies: Cochrane NRS Guidance • Cochrane NRS Group guidance • Don’t use NRSs to supplement RCT data on effectiveness • Few RCTs provide imprecise, unbiased estimate • Including NRSs increases precision, but at the unacceptable cost of accuracy • This position has some merit, but ignores some important characteristics of HPPH interventions and reviews

13. Sources of Variance in HPPH Reviews • Meta-synthesis of psychological, behavioral and educational interventions (Wilson & Lipsey , 2001) • Reasonable generalizability to HP interventions • Substantive variance (25% of total) • Methodological Variance (21% of total) • Study design (4%) • Operationalization of outcome (8%) • EPPI meta-synthesis on policy studies will be useful

14. Rationale for Including NRSs in Complex Population-level Interventions (1) • Bias needs to be considered at two levels • The study (internal validity) • The systematic review (generalizability) • Distinction between systematic and non-systematic biases is also important

15. Rationale for Including NRSs in Complex Population-level Interventions (2) • Non-systematic sources of bias appear to contribute variance within an acceptable range of “noise” • Selection bias is the major systematic threat in NRSs • Self-selection • Researcher-selection • Threats of self-selection bias are not identical across interventions • Most likely with individual-level interventions • Less likely with population-level interventions • Complicated causal pathway to implementation reduces risk of confounding • Availability of data on comparability of groups pre-intervention • Registry of PH interventions would help address researcher-driven selection biases

16. Rationale for Including NRSs in Complex Population-level Interventions (3) • Limiting reviews to RCTs may introduce more bias than it prevents • Bias=systematic error in the population effect estimate • RCTs may provide biased effect estimates for complex interventions due to • ITT analysis (difference between the effectiveness of the intervention and of randomization to the intervention condition) • Resources • Population selection • Adherence to protocol • Benefits of including NRSs • Power

17. Rationale for Including NRSs in Complex Population-level Interventions (4) • Generalizability • Power • Increases potential to provide useful guidance on “lumped” effects • Dramatically increases potential to provide useful guidance on effect modification issues (but only when there is no “firewall” between RCTs and other studies)

18. A Judgment Call • Is study design such a unique and important source of variance that it should be singled out from among all other potential sources of bias and effect modification? • Or do the harms of treating study design as qualitatively different from all other potential modifiers of effect estimates outweigh the benefits?

19. If the Latter.. • Guidance re: addressing “quantitative” differences in study quality should apply: • Consider limiting review to studies above a threshold “design quality” • Considering plausible systematic sources of variance for given subject matter • Use sensitivity analysis to evaluate robustness of findings (giving up on the quest for precision) • Avoid or cautiously apply “quality weighting” by design alone

20. Beware of the “Outlier” Randomized Trial • Shatterproof glassware • Students Against Drunk Driving • Any multi-million dollar trial that can’t feasibly be brought to scale

21. Thank You! Randy Elder rfe3@cdc.gov www.thecommunityguide.org