Lessons about risk-benefit assessment from OMERACT

1. Lessons about risk-benefit assessment from OMERACT Jasvinder Singh, MBBS, MPH Associate Professor of Medicine, UAB School of Medicine & VA Medical Center, Birmingham, Alabama Mayo Clinic School of Medicine, Rochester, MN IMMPACT-XIV, Arlington, VA June 16-17, 2011

2. Acknowledgement • Entire OMERACT Executive: Developed OMERACT 3 X 3 • Slides: Dr. Maarten Boers • Comments: Drs. Maarten Boers & Lee Simon

3. Overview • Definitions • Summary and Discussions at 2008 OMERACT Drug Safety Summit • OMERACT approach • Benefit Risk Action Team (BRAT) approach • Pharmaceutical Research and Manufacturers of America • Questions and potential solutions • The Future

4. Definitions: Risk and Benefit • Risk: • Oxford: “a situation involving exposure to danger” • Merriam-Webster: “possibility of loss or injury” • Medical context: an effect that is harmful to the patient’s or public’s health and which can relate to safety, efficacy or quality of a product. • Benefit: • Oxford: “an advantage or profit gained from something” • Merriam-Webster: “something that promotes well-being “

5. Overview • Definitions • Summary and Discussions at 2008 OMERACT Drug Safety Summit • OMERACT approach • Benefit Risk Action Team (BRAT) approach • Pharmaceutical Research and Manufacturers of America • Questions and potential solutions • The Future

6. OMERACT Drug Safety Summit (2008) • Assigning causality: • Bradford-Hill criteria + biological plausibility + Bayesian methods • No agreement on magnitude of frequency, i.e., when does something become a “signal” • Utility of large trials to define risk • Large, simple trials such as prospective randomized open blinded endpoint (PROBE) • Randomization to two effective treatments • Alternative: observational studies with confounding by indication and channeling bias

7. OMERACT Drug Safety Summit (2008) • Utility of metanalyses of RCTs • Multiple small RCTs ?statistically equivalent to a large single trials with same denominator • 200-300 events needed for credible estimates • Role for indirect comparisons- unclear • Observational studies: may be included; • Selection and confounding bias issues • Outcome definition and measurement and follow-up

8. OMERACT Drug Safety Summit (2008) • Postmarketing Surveillance • Essential for drug safety evaluation • All drugs in a class should be analyzed identically and concurrently • Desired components: • More than one defined population • Full protocol with outcomes assessed at regular intervals • Concurrent control subjects • Outcomes- patient-recoded + documented in electronic records • Pre-defined hypothesis • Oversight by FDA and data management council

9. OMERACT Drug Safety Summit (2008) • Utilities of Drug registries • Advantages: real world experience, long-term FU • Disadvantages: lack of comparator group, data quality, patients not obligated to follow protocol, loss to follow-up • Pharmacoepidemiologic studies • Cohort studies generally better than case-control studies in providing risk estimates • Issues need to addressed • Misclassification bias • Validation of outcomes • Guidelines for confounding bias and methods for adjustment

10. OMERACT Drug Safety Summit (2008) • Simple versus complex metric • Simple: OMERACT 3 x 3 • Benefit and risk categorized into 3 levels each- none, substantial, (near) remission or death • Complex: multicriteria decision analysis • Complex Frameworks: Quantitative methods • Decision analysis method • Conjoint analysis • Incremental net-benefit • BRAT approach

11. OMERACT Drug Safety Summit (2008) • GRADE approach • Classifies evidence into 4 levels: high, moderate, low and very low based on • Study design, weaknesses, special strengths (large effect, dose response) • 2 recommendations: strong and weak • RCTs always starting at high and non-RCTs starting at low? • Other models of Risk: Nontreatment • Risk and value of available treatment versus nontreatment options • Type, intensity and severity of adverse event • Acute, subacute or chronic; manageable, treatable or not

12. Overview • Definitions • Summary and Discussions at 2008 OMERACT Drug Safety Summit • OMERACT approach • Benefit Risk Action Team (BRAT) approach • Pharmaceutical Research and Manufacturers of America • Questions and potential solutions • The Future

13. OMERACT 3 x3 • Create three categories of harm and benefit • Benefit: none/minimal, major, (near) remission • Harm: none/minimal, major, (near) death • Key components of this approach: • Harms versus benefits

14. Problems in benefit-risk assessment • measurement of benefit is specific and evolved, but measurement of risk (or harm) is generic and more primitive • Benefit and harm not placed on a single scale single scale benefit-harm

15. Conceptual problems • placing value judgments on scientific facts • values will vary depending on the perspective of the assessor • comparing benefit and risk involves: • comparing apples and pears • trading off (and discounting) long-term against short-term effects • assessing multiple benefits and risks • assessment mostly driven by the need to make decisions, whereas most research is ‘truth-driven’ single scale benefit-harm

16. Practical problems • ...placing value judgments on scientific facts • Requires qualitative research • ...multiple comparisons and trade-offs • Difficult science • Requires decision analysis • ...mostly driven by decision-makers • Low speed of developments • Literature difficult to access single scale benefit-harm

17. OMERACT 3 x 3 Table single scale benefit-harm

18. Examples • COBRA trial • VIGOR trial single scale benefit-harm

19. COBRA trial • RCT comparing combination therapy (COBRA) including step-down high-dose prednisolone with single drug therapy (sulfasalazine, SSZ) in early rheumatoid arthritis • Main result showed COBRA to be more effective and result in less side effects than SSZ single scale benefit-harm

20. 3 x 3 COBRA Major benefit = Disease activity score <=3.7 (near) Remission = Disease activity score <=2.4 Severe Harm = treatment discontinuation due to serious harm, loss of efficacy or both (near) death = death or severe inacapacitation

21. 2 x 2 COBRA ARR: Unqualified success: 80-54% = 26%; NNT = 4 ARR: Unmitigated failure: 5-14% = –9%; NNH = ? single scale benefit-harm

22. Success/failure Unqualified success Unmitigated failure single scale benefit-harm

23. VIGOR trial • large industry-sponsored trial to compare high-dose rofecoxib with naproxen in RA • object was GI safety, not efficacy • rofecoxib proved safer for GI, but less safe for cardiovascular events • Merck withdrew the drug voluntarily amidst much controversy • no access to individual patient data • safety and efficacy assumed independent single scale benefit-harm

24. 2 x 2 VIGOR Data from original report and from Strand V. Lancet 2007;370:2138-51. Major harm is defined as cardiovascular event; gastrointestinal event; death. single scale benefit-harm

25. Success/failure Unqualified success Unmitigated failure single scale benefit-harm

26. Overview • Definitions • Summary and Discussions at 2008 OMERACT Drug Safety Summit • OMERACT approach • Benefit Risk Action Team (BRAT) approach • Pharmaceutical Research and Manufacturers of America • Questions and potential solutions • The Future

27. STEP 1: Defining the decision context • Specifying the therapeutic context • Explicit definition of product, indication, target population, formulation, dosage and contraindications • Specifying the comparator • Standard of care, best in class, watchful waiting, placebo and nonpharmacological intervention • Defining the time horizon • Duration of exposure and time period over which benefit-risk events are measured • Specifying the stakeholder perspective • Sponsor, regulators, patients, physicians 1Coplan PM Clin Pharma Therap 2011; 89:2: 312-15

28. Step 2: Identifying benefit and risk outcomes: Building the value tree • Identifying pool of candidate outcomes for the assessment • Deciding which outcomes to include or exclude from the framework • Documenting all critical assumptions for these inclusion and exclusion decisions • Value tree is a visual hierarchical presentation of key ideas, values or concepts 1Coplan PM Clin Pharma Therap 2011; 89:2: 312-15

29. Step 3: Identifying data sources for the framework • Information may come from multiple data sources • A repository of all data, called data source table is kept 1Coplan PM Clin Pharma Therap 2011; 89:2: 312-15

30. Step 4: Customizing the framework • Customization done based on data quality and limitations • Study end-points are organized in value tree at 2 levels • Body system category of the benefit or risk • The end-point measured in studies • Approaches to capturing level of severity of outcomes identified and the value tree is enhanced 1Coplan PM Clin Pharma Therap 2011; 89:2: 312-15

31. Step 5: Assess relative importance of different outcomes • Assess relative weights • Stated choice methods • QALYs • Utilities • Value functions • Application of weighted approach in analysis • Net clinical benefit • Number needed to harm (NNH) • Multicriteria decision analysis 1Coplan PM Clin Pharma Therap 2011; 89:2: 312-15

32. Step 6: Display and interpret key benefit-risk metrics • Key risk benefit table • Flexible table summarizing the key information needed to quantify outcomes in the value trees • Additional items to enhance key table • Heat map color coding • Embedded graphs 1Coplan PM Clin Pharma Therap 2011; 89:2: 312-15

33. BRAT framework Example: Acute treatment of headache phase of migraine 1Levitan Clin Pharma Therap 2011; 89:2: 217-234

34. Example 2: BRAT framework 1Levitan Clin Pharma Therap 2011; 89:2: 217-234

35. Example 2: Key Benefit-risk table 1Levitan Clin Pharma Therap 2011; 89:2: 217-234

36. Overview • Definitions • Summary and Discussions at 2008 OMERACT Drug Safety Summit • OMERACT approach • Benefit Risk Action Team (BRAT) approach • Pharmaceutical Research and Manufacturers of America • Questions and potential solutions • The Future

37. Questions and potential solutions • How to come up with method (s) for risk-benefit assessment that is/are • Easily understood by regulators, patients and physicians • Take into account non-treatment or alternative treatments • Allow sensitivity analyses? • How to mandate and conduct efficient postmarketing surveillance studies? • Requirement by FDA and EMEA etc. • Funding • Preventing bias • Design and follow-up methodology

38. Questions and potential solutions • Can efficacy trials be better designed to allow better short-term risk-benefit analysis? • What steps must be taken to use large databases to conduct postmarketing studies? • Standardization of methods for reduction of bias • Data integrity and privacy issues • How to achieve an agreement on methods by major regulators (FDA, EMEA)?

39. Overview • Definitions • Summary and Discussions at 2008 OMERACT Drug Safety Summit • OMERACT approach • Benefit Risk Action Team (BRAT) approach • Pharmaceutical Research and Manufacturers of America • Questions and potential solutions • The Future

40. Future • Collaborative networks of academia, regulatory agencies, patients and pharma • Use of preferences, values, cost and delay in drug approval (risks of alternatives) in risk-benefit analysis • Universally applicable methods • Ability to vary values, weights and perform sensitivity analyses • Utilize non-randomized studies • Design and conduct useful large simple comparative effectiveness trials

41. Thank You