Evaluating Methodological Quality and Recommendations in Clinical Evidence

1. Two questions in grading recommendations • Are you sure? • Yes: Grade 1 • No: Grade 2 • What is the methodological quality of the underlying evidence • High quality: Grade A • Intermediate quality: Grade B • Poor quality: Grade C

2. What is the methodological quality of the underlying evidence? • High quality evidence: Grade A - Randomized trials, few limitations • Intermediate quality evidence: Grade B • Randomized trials with important limitations • varying results (heterogeneity) • major methodological flaws • total sample size in all studies combined under 100 • Poor quality: Grade C • Observational studies B: Randomized trials, inconsistent results C: Observational studies • Benefits vs Risks/costs Tradeoff I: Clear that benefits do/don’t outweight risks/cost II: Benefit vs risk/cost tradeoff not clear

3. Generalizing result • ASA in unstable angina 50% RRR • trials of patients up to 80 years old • no trials in those over 80 -- still Grade A, or C? • Warfarin in atrial fibrillation • lots of trials in non-valvular atrial fibrillation • no trials in valvular a fib -- still Grade A, or C? • IV heparin for pregnant women with DVT • lots of trials in non-pregnant • no trials in pregnant -- still Grade A, or C

4. What do we mean by “are you sure? • 1st: Is there uniformity in assessment of risk/benefit in your consensus group and in the community • If yes, probably Grade 1 • If no, probably Grade 2 • 1st: Is the risk/benefit clear • Grade 1: Benefit clearly greater than risk or risk clearly greater than benefit • Grade 2: Risk/benefit uncertain

5. What do we mean by “are you sure” • Consider patient values • Example: Different values of stroke/bleeding • 3rd: Would (almost) all your patients make the same choice? • Yes: Grade 1 • No: Grade 2 • 4th: Would a decision aid be useful and worthwhile? • No, no need: Grade 1 • Yes, needed: Grade 2

6. What do we mean by “are you sure? • 5th: Directive to clinicians • Grade 1: just do it • Grade 2: think about it • your own judgment of strength of evidence • your own judgment of risk/benefit • talk to your patients, their values may impact

7. Risk/Benefit clear • Aspirin with acute myocardial infarction • 25% reduction in relative risk, narrow confidence interval • side effects trivial, cost negligible • benefit obviously much greater than risk/cost, 1(A) • Thrombolysis in MI symptoms with only ST changes • no difference from placebo, narrow confidence interval • small risk of intracranial hemorrhage • risk obviously greater than possible benefit, 1(A)

8. Judgment: Benefits vs Risks/Costs • Seriousness of outcome • Magnitude of effect • Precision of treatment effect • Risk of target event • Risk of serious adverse events • Cost of therapy • Values

9. Cost and Magnitude of Effect • Clopidigrel vs ASA for atheroembolism • 8.7% RRR relative to ASA • 5.83% to 5.32% in MI, ischemic stoke and death • NNT 200, cost $1,052 vs. $21 • some will feel benefits not worth extra costs and therefore 2(A)

10. Cost and toxicity • TPA versus streptokinase • RCT shows 15% RRR with TPA • TPA larger cost • TPA increased risk of intracranial hemorrhage • Varying practice, unclear risk/benefit • Grade 2 (B)

11. Imprecision of treatment effect • Should dipyridamole be added to aspirin after MI? • 1998 single RCT • 85 deaths in 810 ASA alone, 87 in 810 ASA and dipyridamole • RR with ASA 0.98 (95% CI 0.70 to 1.26) • Recommendation: don’t use dipyridamole • Clearly Grade A; ? 1 or 2 • Consensus criterion: Grade 1(A)

12. Precision of estimate • RR with ASA 0.98 (95% CI 0.70 to 1.26) • ASA may reduce risk relative to combination by 30% • combination may reduce risk relative to ASA by 26% • Are we sure dipyridamole doesn’t add - No • Patient: I’ll take any low cost low toxicity medication that MIGHT help • Risk/benefit or patient value criteria: 2 (A) • How to use confidence interval • look at boundaries, is decision same at either end?

13. Judgement: benefits versus risks/costs Seriousness of outcome Death vs post-phlebitic syndrome Magnitude of effect 68% RRR warfarin in a fib, vs 9% RRR with clopidigrel in CAD Precision of treatment effect warfarin in a fib vs. ASA in a fib Risk of target event warfarin in high vs low risk a fib Risk of serious adverse event coumadin versus aspirin Costs ASA vs. clopidigrel Values (every decision) high value on avoiding stroke: TPA; clopidigrel; warfarin

14. 1 A recommendation • Patients with atrial fibrillation and additional risk factors for arterial embolism without excessive bleeding risk should receive warfarin • strong recommendation, can apply to most patients in most circumstances with no reservations

15. 1 B recommendation • Clinicians should not administer magnesium sulfate to patients with acute myocardial infarction • meta-analysis of smaller RCTs +ve, large RCT -ve • Strong recommendation, likely to apply to most patients

16. 1 C recommendation • Patients with acute peripheral arterial thrombi or emboli should be systematically heparinized • No RCTs, strong biological rationale • Intermediate strength recommendation, may change when stronger evidence available

17. 2 A recommendation • Men over 50 without established CAD, but with one or more additional risk factors for CAD should take daily ASA • RCT shows lower risk of MI but may be higher risk of cerebral bleed • both risks very low, individual values may determine decision • Intermediate strength recommendation, best action may differ depending on circumstances or patients’ or societal values

18. 2 B recommendation • Intra-arterial thrombolytic therapy may be used as an alternative to surgery in patients with acute peripheral arterial thrombi or emboli • 2 RCTs, show comparable results, 1 surgery clearly better • Weak recommendation, alternative approaches likely to be better for some patients or circumstances

19. 2 C recommendation • Pregnant women with previous venous thrombosis associated with a transient risk factor should receive surveillance only during pregnancy and heparin and warfarin post-partum • incidence of thrombosis and magnitude of benefit with therapy unestablished • Very weak recommendation, alternatives equally reasonable

20. Are we producing guidelines? • Yes! • Recommendations from authoritative body intended to influence clinical practice • Shanneyfelt, JAMA;1999;281:1900 • 279 guidelines published 1985 to 1997 • adherence to standards for evidence summary 34% • adherence to standards for making recommendations 6% • Conclusions • we should do better or • everyone else doing equally badly, we don’t need to worry

21. Systematic review of evidence • Formal statement of eligibility criteria for each question • patients, interventions, outcomes, methodology • Systematic search for evidence • Explicit process of data abstraction • Pooling of results • wherever appropriate • systematic approach

22. From evidence to recommendations • Value judgments implicit in every recommendation • Whose values • Ours? • Society? • Patients? • Possibilities • explicit elicitation of values • include people with different values/perspectives • patient; primary care doctor