Statistical Issues in Specification of D

1. Statistical Issues in Specification of D Daphne Lin, Ph.D. Erica Brittain, Ph.D. Division of Biometrics III

2. Outline • Intro: Non-inferiority trials • What is D? • History of D Selection • Principles for determining D • Difficulties in practice • Alternate designs • Summary

3. The Problem • How can we show the new drug has identical efficacy to the standard drug (active control drug)? • Answer: WE CAN’T! • SOLUTION: We must allow some potential difference in efficacy: D

4. What is D? • The largest clinically acceptable difference • Should be smaller than differences observed in superiority trials of active comparator

5. Non-inferiority Trials • Non-inferiority trial: to show the new drug is no worse than the standard drug by some margin D.

6. Estimation of treatment effect & Confidence Interval • Define treatment effect as the absolute difference of percent cure rates (e.g.; 85%-75% = 10%) • A confidence interval around this estimate of treatment effect is used as the primary analysis for non-inferiority trials • Interpretation of the 95% confidence interval: 95% confident that the true difference in efficacy between these two drugs is contained in the confidence interval. )

7. -10% Example 1: Non-inferiority demonstrated 200 pts/arm, test drug: 88%, control drug: 90%, CI: (-8.6% 4.6%) D = 10% Observed success ratebetween test and control drug e.g.; - 2% 95% confidence interval - 8.6% 4.6%

8. -10% Example 2: Non-inferiority not demonstrated 200 pts/arm, test drug: 84%, control drug: 90%, CI: (-13%, 1.1%) D = 10% Observed success ratebetween test and control drug e.g.; - 6% 95% confidence interval - 13% 1.1%

9. Goals of non-inferiority trials: • Indirectly determine if the test drug is better than placebo • Directly determine if the test drug is similar to the active control drug

10. History of D Selection in FDA’s Division of Anti-infective Drug Products

11. D 25% 20% 15% 10% 5% 0% 70% 80% 90% 100% Success Rate 1992 PTC Step Function Approach

12. Concerns about 1992 PTC approach • Seriousness of disease and consequence of treatment failure were not taken into account • whether D was small enough that a drug with no efficacy could meet the standard was not considered • Has undesirable statistical properties ( e.g.; D jumps from .15 to .2 when success rate changes from .8 to .79)

13. Placebo Drug 3 48% Concern of Potential “Bio-Creep” • Trials over time used progressively less effective control arms • Example: if D of 20% is used Drug 1 Drug 2 100% 60% 70% 0%

14. July 1998 AC Meeting • The FDA stated D should reflect: • historical cure rate (with and without therapy) • risk associated with treatment failure • advantages and disadvantages of study drug

15. July 1998 AC Meeting: Proposal • Clinically relevant delta • Indication specific • Special situations • Discuss with Medical Division during protocol development • Sponsor should provide rationale for selection of control arm

16. Committee for Proprietary Medicinal Products (CPMP) • Published “Guidance on evaluation of new anti-bacterial medicinal products” in 1997 • D=10% for “common non-serious infections” • Smaller for very high cure rates • Based on “minimum clinically relevant difference” • Justified in protocol

17. Transition • For past 1-2 years, have worked with sponsors on case-by-case basis to specify D • In February 2001, a disclaimer added to PTC document stating that PTC approach phased out • There is a need to establish standards (ICH-E10 principles)

18. Now what?

19. Road Map • Principles for determining D • Based on ICH-E10 • Difficulties in practice • This is the hard part: we need advice • Alternate designs • Summary • GOAL: Choice of D is not a technical matter

20. To demonstrate efficacy: • Experimental drug should be better than placebo • In some settings, experimental drug should also have similar efficacy to existing therapy • Choose D to assure that both of these goals are met

21. ICH-E10 says • Non-inferiority design “is appropriate and reliable only when the historical estimate of drug effect size can be well supported by reference to results of previous studies of the control drug”

22. Translation • We must KNOW with good precision the magnitude of advantage of the active control drug over placebo in setting of clinical trial • In practice: if advantage is large, precision of estimate probably won’t matter • If potentially modest, precision is critical • If only a single trial of AC w/ borderline efficacy -- poor info about magnitude (Statistical significance not enough)

23. Principles from E10 • D “based on both statistical reasoning and clinical judgment” • D cannot be larger than the advantage the “active drug would be reliably expected to have compared with placebo in the setting of the planned trial” • Usually choose D to be even smaller to ensure some clinically acceptable treatment benefit maintained

24. Possible Choices for D TrueSuccess Rate of Placebo True Success rate of Active Control D = 3% D = 7% D = 15% 70 72 74 7678808284 85 True% Success Rate

25. General approach to D • Determine two values: • Conservative estimate of advantage of AC over placebo (D1) • DATA-BASED • Largest clinically acceptable difference between AC and experimental drug (D2). • JUDGMENT-BASED • D = smaller of these two values: Value used in upcoming NI trial

26. Estimation of benefit of AC over placebo (D1) • The true success rate of active control minus the true success rate of placebo in setting of clinical trial • By how much is AC better than placebo in the NI trial, if placebo were present? • Based on historical data -- NOT A CHOICE • Not important when benefit is large

27. Estimating benefit of AC (D1): Be Conservative • E10: D “should reflect uncertainties in evidence on which choice is based and should be suitably conservative” • If D1 is over estimated, chance of concluding efficacy when new drug is no better than placebo is too high • Err on side of underestimating benefit. IF poor historical info: don’t use “best” guess, use smallest of reasonable values

28. Estimating benefit of AC(D1 ):What is best information? Best information for determining D1 Multiple PCTs w/ same design as NI Single PCT with same design as NI Multiple PCTs with different design Single PCTs with different design Observational studies Anecdotal / Case Series No information for determining D1

29. Estimating benefit of AC(D1):Placebo Control Trial Data • What if placebo controlled data exist, but: • Trials are old: antibiotic resistance, changes in clinical care management • Proposed active control not studied • Few in number / consistency? • Differences in entry criteria, assessment criteria, timing, populations • Take data with big grain of salt --Then what?

30. Estimating D1: Bottom Line • Use historic data -- preferably from placebo controlled trials with same designs as upcoming NI trial • Bad news: historic data is often poor because of ethical constraints • conservative estimate not straightforward • Good news: precision is probably irrelevant if benefit is KNOWN to be very large

31. D: Step 2 • Recall two step process • Step 1: Determine estimate of advantage of AC over placebo (D1) • Step 2: Select acceptable loss from AC (D2) • D = smaller of D1 and D2 • Selection of D2 is primary concern for most Anti-infective indications

32. Selection of clinically acceptable loss (D2) • NOT STATISTICAL DECISION • CLINICAL judgement of largest acceptable difference between AC and the new drug • a difference >D2 is so important clinically that it must be ruled out -- no tolerance for any difference >D2 • D2 is borderline value between just barely acceptable and NOT acceptable

33. Selection of clinically acceptable loss (D2) • Consequence of treatment failure is important factor in determining D2 • What proportion of study failures are deaths or other very serious morbidity? • Can treatment failure be easily reversed/addressed?

34. Clinically Acceptable Loss (D2):Consider consequence to patients

35. Clinically Acceptable Loss (D2):Consider Clinical Trial Realities • Some patients do not have disease • Trt diff :Pts with bacterial infection: 12% • Trt diff: Pts without bacterial infection: 0% • Measured trt diff: If 50/50 mix: 6% • If D=10%, may conclude New is efficacious • BUT key population difference>10% • BE CAREFUL: Other factors dilute observed treatment difference too

36. Selection of clinically acceptable loss (D2) • SUMMARY: • Consequence of treatment failure • Potentially large impact on patient care • Be careful about clinical trial realities • CLINICAL judgement

37. General Approach:D for each indication • Provides regulatory consistency • But, requires vigilance • New AC may be less efficacious than original AC (is D smaller than advantage of AC over placebo?) • Emerging resistance and other temporal changes may have diminished the efficacy of any AC

38. Consequence to Sample Size • D cut in half, sample size quadruples • But if new drug is slightly better than AC, sample size can be sharply reduced • 80% Power, D=10% • Assume cure rates both 80%, SS/grp=252 • Assume new drug rate is 82%, SS/grp=168 • Sample Size cut by 1/3

39. BIGGEST CHALLENGES • Biggest challenges in selection of D • Indications where treatment effect is potentially modest but not precisely known • Serious indications with low incidence • Superiority designs may offer an important alternative to NI design: stronger evidence and potentially smaller sample size • Can they be done ethically?

40. Early Escape: Ethical? • Applicable to few indications (less serious) • Two arms: Experimental vs. placebo • KEY: Pts seen several days after baseline • If blinded assessment no improvement • Failure in analysis, therapy switched • Ethically consistent with “wait and see” • Variant: Early Escape with 3 arms

41. Other Superiority Designs • Placebo Add-on: Existing + New vs. Existing + Placebo • Does New have benefit in presence of Existing therapy; labeling implications • Dose-response (High vs Low dose) • Superiority to some comparator • Combination

42. Selection of D: BIG PICURE Take Home Message

43. Choice of D impacts patients • If D is incorrectly chosen so that greater than advantage of AC over placebo • Patients may get drugs with no benefit, while exposed to toxicity and potential for development of resistance • Potential benefits of using smaller D even where no concern about placebo rate: • More pts cured overall, higher survival • Subtle but important differences detected • Smaller D: larger and longer studies

44. Take Home Message • Absolute: D MUST be smaller than conservative estimate of the advantage of Comparator over placebo • Challenge: insufficient historic data • Clinical judgement: may decrease D further • To rule out important loss in efficacy • Superiority designs can play important role in some settings