Why Grade Recommendations?

2. Why Grade Recommendations? • strong recommendations • strong methods • large precise effect • few down sides of therapy • weak recommendations • weak methods • imprecise estimate • small effect • substantial down sides

3. Which grading system to use? • many available • Australian National and MRC • Oxford Center for Evidence-based Medicine • Scottish Intercollegiate Guidelines (SIGN) • US Preventative Services Task Force • American professional organizations • AHA/ACC, ACCP, AAP, Endocrine society, etc.... • cause of confusion, dismay

5. A common international grading system? • GRADE (Grades of recommendation, assessment, development and evaluation) • international methodologists, guideline developers • Australian NMRC, SIGN, USPSTF, WHO, NICE, Oxford CEBM, CDC, CC • ~ 20 meetings over last eight years • (~10 – 50 attendants) • BMJ 2004, six part series 2008

6. GRADE Uptake • UpToDate World Health Organization • British Medical Journal American Thoracic Society • American College of Physicians Cochrane Collaboration • BMJ Clinical Evidence KDIGO • Polish Institute for EBM EBM Guidelines Finland • Society of Vascular Surgery Society of Pediatric Endocrinology • European Respiratory Society American Endocrine Society • Society of Critical Care Medicine Surviving sepsis campaign • American College of Chest Physicians European Soc of Thoracic Surgeons • Allergic Rhinitis in Asthma Guidelines Society of Vascular Surgery • Infectious Disease Society of America • National Institute for Clinical Excellence (NICE) • Agency for Health Care Research and Quality (AHRQ) • Swedish National Board of Health and Welfare • Canadian Agency for Drugs and Technology in Health • Ontario MOH Medical Advisory Secretariat • Agencia sanitaria regionale, Bologna, Italia • The German Agency for Quality in Medicine • Evidence-based Nursing Sudtirol, Alta Adiga, Italy • Norwegian Knowledge Centre for the Health Services • University of Pennsylvania Health System Center for EB Practice • Journal of Infection in Developing Countries – International • Japanese Society of Oral and Maxilofacial Radiology • Emergency Medical Services for Children National Resource Center

7. What are we grading? • two components • quality of body of evidence • confidence in estimate of effect • high, moderate, low, very low • strength of recommendation • strong and weak

8. Studies S1 S2 S3 S4 S5 Health Care Question (PICO) Systematic reviews Outcomes OC1 OC2 OC3 OC4 Important outcomes Critical outcomes OC1 OC2 OC3 OC4 Generate an estimate of effect for each outcome Rate the quality of evidence for each outcome, across studies RCTs start high, observational studies start low (-) Study limitations Imprecision Inconsistency of results Indirectness of evidence Publication bias likely Final rating of quality for each outcome: high, moderate, low, or very low (+) Large magnitude of effect Dose response Plausible confounders would ↓ effect when an effect is present or ↑ effect if effect is absent Rate overall quality of evidence (lowest quality among critical outcomes) Decide on the direction (for/against) and grade strength (strong/weak*) of the recommendation considering: Quality of the evidence Balance of desirable/undesirable outcomes Values and preferences Decide if any revision of direction or strength is necessary considering: Resource use *also labeled “conditional” or “discretionary”

9. Structured question • patients: lymphoma patients at risk of developing chemotherapy-induced febrile neutropenia • granulocyte colony-stimulating (G-CSF) • alternative not using G-CSF

10. Need to define all patient-important outcomes and evaluate their importance

11. Design and Execution • well established • concealment • intention to treat principle observed • blinding • completeness of follow-up • more recent • early stopping for benefit • selective outcome reporting bias

12. Consistency of results • consistency of results • if inconsistency, look for explanation • patients, intervention, outcome, methods • judgment of consistency • variation in size of effect • overlap in confidence intervals • statistical significance of heterogeneity • I2

13. Fluoroquinolone prophylaxis in neutropenia: infection-related mortality

14. Directness of Evidence • differences in patients • age, sex, ethnicity, condition – avian versus regular influenza • interventions • dose, class • outcomes • health-related quality of life, functional capacity, laboratory exercise

15. Critical for decision making Important, butnot critical for decision making Of low patient- importance Figure 6: Hierarchy of outcomes according to their patient-importance to assess the effect of phosphate lowering drugs in patients with renal failure and hyperphophatemia Importance of endpoints Surrogates of declining importance Mortality 9 Coronary calcification Ca2+/P- Product Myocardial infarction 8 Bone density Ca2+/P- Product Fractures 7 Pain due to soft tissue Calcification / function 6 Soft tissue calcification Ca2+/P- Product 5 4 Lower by one level for indirectness 3 2 Flatulence Lower by two levels for indirectness 1

16. Directness interested in A versus B available data A vs C, B vs C Alendronate Risedronate Placebo

17. Imprecision • small sample size • small number of events • wide confidence intervals • uncertainty about magnitude of effect • how to decide if CI too wide? • grade down one level? • grade down two levels? • extent of confidence in estimate of effect

18. Offer all effective treatments? • atrial fib at risk of stroke • warfarin increases serious gi bleeding • 3% per year • 1,000 patients 1 less stroke • 30 more bleeds for each stroke prevented • 1,000 patients 100 less strokes • 3 strokes prevented for each bleed • where is your threshold? • how many strokes in 100 with 3% bleeding?

19. 1.0% 0

20. 1.0% 0

21. 1.0% 0

22. 1.0% 0

23. Publication bias • high likelihood could lower quality • reporting of studies • publication bias • number of small studies • industry sponsored

26. What can lower quality? • detailed design and execution • inconsistency • indirectness • imprecision • publication bias

27. What can raise quality? • large magnitude can upgrade one level • very large two levels • common criteria • everyone used to do badly • almost everyone does well • quick action • hip replacement for severe osteoarthritis • dialysis vs no dialysis for prolonging life

28. Dose-response gradient • childhood lymphoblastic leukemia • risk for CNS malignancies 15 years after cranial irradiation • no radiation: 1% (95% CI 0% to 2.1%) • 12 Gy: 1.6% (95% CI 0% to 3.4%) • 18 Gy: 3.3% (95% CI 0.9% to 5.6%).

29. Quality assessment criteria

30. Whipples procedure pancreatic cancerwith or without duodenectomy

32. Strength of Recommendation • strong recommendation • benefits clearly outweigh risks/hassle/cost • risk/hassle/cost clearly outweighs benefit • what can downgrade strength? • low quality evidence • close balance between up and downsides

33. Risk/Benefit tradeoff • aspirin after myocardial infarction • 25% reduction in relative risk • side effects minimal, cost minimal • benefit obviously much greater than risk/cost • warfarin in low risk atrial fibrillation • warfarin reduces stroke vs ASA by 50% • but if risk only 1% per year, ARR 0.5% • increased bleeds by 1% per year

34. Strength of Recommendations Aspirin after MI – do it Warfarin rather than ASA in Afib -- probably do it -- probably don’t do it

35. Significance of strong vs weak • variability in patient preference • strong, almost all same choice (> 90%) • weak, choice varies appreciably • interaction with patient • strong, just inform patient • weak, ensure choice reflects values • use of decision aid • strong, don’t bother • weak, use the aid • quality of care criterion • strong, consider • weak, don’t consider

36. When evidence is low quality • choice more preference dependent • risk aversion • steroids for pulmonary fibrosis • low quality evidence in support of benefit • high quality evidence of toxicity

37. When evidence is low quality • recommendation to the hopeful patient • I’m likely to deteriorate • if something might work, let’s try it • damn the torpedoes • recommendation to the fearful patient • doctor, you mean you know it’s toxic • diabetes, skin changes, body habitus, infection, osteoporosis • you don’t know for sure it works? • are you crazy? • discretionary recommendation mandated

38. Strong recommendation when evidence is low quality? • recommendations against • uncertainty of benefit • confidence in down sides • whole body CT or MRI screening • maybe benefit, maybe not • true positives some harm • false positive some harm

39. Strong recommendation when evidence is low quality? • known benefit, strong recommendation for one of two alternatives • antipyretics in children with chickenpox • but which one: ASA or acetaminophen • benefit: high quality evidence of equivalence • harm: low quality evidence that harm differs appreciably • Reye syndrome from ASA • strong recommendation for acetaminophen?

40. Strong recommendation when evidence is low quality? • Blastomycosis • low quality evidence amphotericin more effective than itraconazole • high quality evidence more toxic • patients with life threatening blasto • life and death situation • strong recommendation for ampho

41. Value and preference statements • underlying values and preferences always present • sometimes crucial • important to make explicit

42. Values and preferences Stroke guideline: patients with TIA clopidogrel over aspirin (Grade 2B). Underlying values and preferences: This recommendation to use clopidogrel over aspirin places a relatively high value on a small absolute risk reduction in stroke rates, and a relatively low value on minimizing drug expenditures.

43. Values and preferences peripheral vascular disease: aspirin be used instead of clopidogrel (Grade 2A). Underlying values and preferences: This recommendation places a relatively high value on avoiding large expenditures to achieve small reductions in vascular events.

44. Flavanoids for Hemorrhoids • venotonic agents • mechanism unclear, increase venous return • popularity • 90 venotonics commercialized in France • none in Sweden and Norway • France 70% of world market • possibilities • French misguided • rest of world missing out

45. Systematic Review • 14 trials, 1432 patients • key outcome • risk not improving/persistent symptoms • 11 studies, 1002 patients, 375 events • RR 0.4, 95% CI 0.29 to 0.57 • minimal side effects • is France right? • what is the quality of evidence?

46. What can lower quality? • detailed design and execution • lack of detail re concealment • questionnaires not validated • rate down quality for study limitations? • indirectness – no problem • inconsistency, need to look at the results

48. Publication bias? • size of studies • 40 to 234 patients, most around 100 • all industry sponsored

50. What can lower quality? • detailed design and execution • lack of detail re concealment • questionnaires not validated • inconsistency • almost all show positive effect, trend • heterogeneity p < 0.001; I2 65.1% • indirectness • imprecision • RR 0.4, 95% CI 0.29 to 0.57 • reporting bias • 40 to 234 patients, most around 100