Prospective Subset Analysis in Therapeutic Vaccine Studies

1. Prospective Subset Analysis in Therapeutic Vaccine Studies Richard Simon, D.Sc. Chief, Biometric Research Branch National Cancer Institute http://linus.nci.nih.gov/brb

2. Not • Statistical Credibility of Subset Analyses in Phase 3 Trials: Is There Room for a Different Perspective for Cancer Vaccines?

3. Clinical trial designs for early clinical development of therapeutic cancer vaccines, JCO 19:1848-54, 2001 • R Simon, S Steinberg, M Hamilton, A Hildesheim, S Khleif, L Kwak, C Mackall, J Schlom, S Topalian and J Berzofsky

4. Why focus on early clinical development? • Principles for phase III trials apply equally to vaccines • Randomized control group • Endpoint reflecting clinical benefit • Defined target population • Differences between vaccines and chemotherapeutic agents have important implications for early clinical trials

5. Phase III Clinical Trials of Therapeutic Cancer Vaccines should conform to good statistical principles • Vaccine studies should be performed professionally • There are no perfect studies • Adaptive prospective analysis plans are consistent with good statistical principles. • Some aspects of therapeutic vaccine studies make adaptive analysis plans more important than for other therapeutic areas

6. Objectives of Phase II Trials • Determine whether regimen is sufficiently promising to warrant phase III trial • Optimize regimen • Generally using non-clinical endpoint • Identify the right population of patients to include in phase III trial

7. Differences Between Therapeutic Vaccines and Chemotherapeutic Agents • Many vaccines are incapable of causing immediate serious or life threatening toxicity at doses feasible to manufacture • Appropriate target population may not have measurable tumor • Vaccination strategies often combine multiple agents and components (adjuvants, cytokines, costimulatory molecules)

8. Trying to determine whether there is a dose-response relationship is a phase III objective. • Using more than two dose levels to determine an OBD is even more ambitious.

9. Optimization of vaccine regimen by comparing results of single arm studies using immunological response is problematic • Much time can be wasted on uninterpretable or unreliable phase I or phase II trials • Randomized screening studies can be used to optimize immunogenicity. • Phase II studies of time to progression should have randomized controls

10. Phase 2.5 Trials • Randomized control group •  = 0.10 type 1 error rate • Endpoint without established clinical benefit • Detect relatively large treatment effect • E.g. Power 0.8 for detecting 40% reduction in 12 month median time to recurrence with =0.10 requires 44 patients per arm

11. Prototypic Phase III Clinical Trial • One hypothesis tested • Randomized treatment assignment • One investigational treatment arm • One control arm • Single pre-defined target population • Single pre-defined primary endpoint • Measure of clinical benefit • Evaluate statistical significance with type I error 5% • Interim analyses consume the type I error

12. Test Multiple Hypotheses • Multiple treatment regimens • Doses, schedules, adjuvants, etc • Multiple target populations • All eligible patients • Pre-defined subsets • Non-pre-defined subsets • Multiple endpoints

14. Testing Multiple Hypotheses • Specify hypotheses to be tested in advance • Share the type I error among the tests so that if all of the null hypotheses are true the probability of rejecting any of them is no more than .05

15. Principle • Hypotheses to be tested must not be determined by the same data used to do the testing • Unless the way that the data determines the hypothesis to be tested is specified in advance, we cannot control the type I error

16. Develop Predictor of Response to New Rx Predicted Responsive To New Rx Predicted Non-responsive to New Rx New RX Control New RX Control Developmental Strategy (II)

17. Developmental Strategy (II) • Do not use the diagnostic to restrict eligibility, but to structure a prospective analysis plan. • Compare the new drug to the control overall for all patients ignoring the classifier. • If poverall 0.04 claim effectiveness for the eligible population as a whole • Otherwise perform a single subset analysis evaluating the new drug in the classifier + patients • If psubset 0.01 claim effectiveness for the classifier + patients.

18. The purpose of the RCT is to evaluate the new treatment overall and for the pre-defined subset • The purpose is not to re-evaluate the components of the classifier, or to modify or refine the classifier • The purpose is not to demonstrate that repeating the classifier development process on independent data results in the same classifier

19. Developmental Strategy III • Do not use the diagnostic to restrict eligibility, but to structure a prospective analysis plan. • Compare the new drug to the control for classifier positive patients • If p+>0.05 make no claim of effectiveness • If p+ 0.05 claim effectiveness for the classifier positive patients and • Continue accrual of classifier negative patients and eventually test treatment effect at 0.05 level

20. The Roadmap • Develop a completely specified genomic classifier of the patients likely to benefit from a new drug • Establish reproducibility of measurement of the classifier • Use the completely specified classifier to design and analyze a new clinical trial to evaluate effectiveness of the new treatment with a pre-defined analysis plan.

21. Guiding Principle • The data used to develop the classifier must be distinct from the data used to test hypotheses about treatment effect in subsets determined by the classifier • Developmental studies are exploratory • And not closely regulated by FDA • FDA should not regulate classifier development • Studies on which treatment effectiveness claims are to be based should be definitive studies that test a treatment hypothesis in a patient population completely pre-specified by the classifier

22. Adaptive Clinical Trials • Modify some aspect of the trial based on data developed in the same trial

23. Seamless Phase II/III TrialFutility Analysis • Randomized comparison of vaccine based regimen to non-vaccine based control • Size trial as phase III study with clinical endpoint • Perform interim analysis using immunogenicity • If results are not “promising”, terminate trial • Otherwise, continue randomizing additional patients and do analysis of clinical endpoint at end of trial • Interim futility analysis does not consume any of the type I error

24. Seamless Phase II/III TrialOptimize Treatment Arm • Randomized comparison of 2 vaccine based regimens to non-vaccine based control • Size trial as phase III study with clinical endpoint • Perform interim analysis using immunologic response • select vaccine arm with most promising immunologic response data • Continue accrual as 2-arm phase III trial of the selected vaccine arm and the control arm • Do analysis of clinical endpoint at end of trial using .05/2 level of significance

25. Adaptive Methods • PF Thall, R Simon, SS Ellenberg, (1989) “A two-stage design for choosing among several experimental treatments and a control in clinical trials”, Biometrika 75:303-310. • 1st phase of trial randomize patients among k investigational arms and a control arm • Select the most promising investigational arm to complete the trial • The final analysis comparing the selected investigational arm to control takes into account that the first phase of the trial was used to select the arm

26. Adaptive Signature Design An adaptive design for generating and prospectively testing a gene expression signature for sensitive patients Boris Freidlin and Richard Simon Clinical Cancer Research 11:7872-8, 2005

27. Adaptive Signature DesignEnd of Trial Analysis • Compare E to C for all patients at significance level 0.04 • If overall H0 is rejected, then claim effectiveness of E for eligible patients • Otherwise

28. Otherwise: • Using only the first half of patients accrued during the trial, develop a binary classifier that predicts the subset of patients most likely to benefit from the new treatment E compared to control C • Compare E to C for patients accrued in second stage who are predicted responsive to E based on classifier • Perform test at significance level 0.01 • If H0 is rejected, claim effectiveness of E for subset defined by classifier

29. Treatment effect restricted to subset.10% of patients sensitive, 10 sensitivity genes, 10,000 genes, 400 patients.

30. Overall treatment effect, no subset effect.10,000 genes, 400 patients.

31. Biomarker Adaptive Threshold Design Wenyu Jiang, Boris Freidlin & Richard Simon JNCI 99:1036-43, 2007 http://linus.nci.nih.gov/brb

32. Biomarker Adaptive Threshold Design • Randomized pivotal trial comparing new treatment E to control C • Survival or DFS endpoint • Have identified a univariate biomarker index B thought to be predictive of patients likely to benefit from E relative to C • Eligibility not restricted by biomarker • No threshold for biomarker determined • Biomarker value scaled to range (0,1)

33. Global Test for Treatment EffectOptimized Threshold • Compare E vs C for patients with B > b • S(b)=log likelihood ratio statistic for treatment versus control comparison in subset of patients with Bb • Compute S(b) for all possible threshold values • Determine b* value for which S(b) is maximum • T=S(b*) • Compute null distribution of T by permuting treatment labels • Re-analyze the data • Compute T for the permuted data • Repeat for 10,000 permutations

34. If the data value of T is significant at 0.05 level using the permutation null distribution of T, then reject null hypothesis that E is ineffective • Compute point and interval estimates of the threshold b

36. Adaptive Methods • The algorithm for adapting should be specified in advance in the protocol • In assessing statistical significance, the analysis should take into account the adaptiveness algorithm used • The rejection region should be calibrated to limit the experiment-wise type I error (probability of making any false positive claim from a study) to 5%, taking into account the adaptiveness algorithm used

37. But I’m a Bayesian • Bayesian inference usually does not control type I error • Bayesian methods can lead to a high frequency of erroneous conclusions if prior distributions are not selected carefully • Dumb things are dumb to careful Bayesians • Bayesian methods are sharp scalpels, and need to be used carefully by experts with knowledge of anatomy

38. Bayesian analysis is based on specifying a model for all aspects of the data and a “prior distribution” for all parameters of the model • eg True hazard ratio of treatment effect on survival is a parameter • The true hazard ratio is regarded as a random sample from the specified prior distribution • True values of parameters are not really random samples from a distribution • Prior distributions represent one’s subjective beliefs about the unknown parameters before doing the study • Different individuals have different prior distributions • Bayesian methods provide a way of updating the subjective prior based on the data obtained from the trial • The Bayesian “posterior distribution” mixes the prior with the data • Bayesian methods are more easily used in phase I/II trials where you don’t need to convince others of your prior distributions and your conclusions

39. Conclusions • Phase III trials of therapeutic vaccines should meet the standards of phase III trials in other therapeutic areas • Prospectively planned subset analyses are consistent with good statistical practice so long as the subsets are pre-defined and the study-wise type I error is preserved at 5% • Therapeutic vaccine trials can benefit from adaptive analysis plans • The adaptivity algorithms should be completely pre-defined • Regulators should be receptive to innovative clinical trial designs that are consistent with sound statistical principles • The standard paradigm of broad eligibility clinical trials with no subset analysis unless the overall treatment effect is significant is often inappropriate in cancer therapeutics