FDA/INDUSTRY STATISTICS WORKSHOP: Washington, D.C. Sept. 29, 2006 Statistical Issues in Medical Device Trials

1. FDA/INDUSTRY STATISTICS WORKSHOP: Washington, D.C. Sept. 29, 2006Statistical Issues in Medical Device Trials George Koustenis, FDA-CDRH David Breiter, Boston Scientific Roseann White, Abbott Vascular George Woodworth, Univ. of Iowa

2. The Use of Objective Performance Criteria (OPC) in Medical Device Evaluation FDA/Industry Statistics Workshop September 29, 2006 Washington, D.C. George Koustenis Division of Biostatistics Office of Surveillance and Biometrics Center for Devices and Radiological Health U.S. Food and Drug Administration

3. OBJECTIVE The purpose of this presentation is to provide a brief introduction to the possible use of Objective Performance Criteria (OPC) in the evaluation of medical devices during the regulatory approval process.

4. DEFINITION OF OPC • Fixed Target(s) • Objective and Meaningful Standard • Provides Comparison in Evaluating Safety and Effectiveness • Usually a Rate • Surrogate for Control Group • Benchmark for Minimally Acceptable Values • Not a Control Group

5. CONTROLS • Standard Comparison for an Experimental Treatment • Group of Patients with Same Condition, Demographics, & Prognostic Values • By Controlling for as Many Variables as Possible – Any Differences are Presumed Due to New Intervention

6. TYPES OF CONTROLS • Concurrent Randomized Controlled Trial (RCT) • The Gold Standard • Effectively Minimizes Bias • Effectively Balances Demographics & Unknowns • Supports Basic Assumptions of Standard Statistical Methodology

7. Historical Controls • Compares Current Therapy & Patients Against Others Studied in Previous Investigations • Not Randomized to Current Conditions • May Be Too Far Removed in Time • Very Difficult to Validate the Data from Historical Trials • Tend to Produce More Positive Results

8. ESTIMATES OF OPC’S • Necessarily Driven by Historical Data • Requires Appropriate Pooling of Different Investigations • OPC’s Inherit All of Problems Seen with Historical Controls • Even Sophisticated Pooling/Analytical Techniques Cannot Eliminate Basic Problems

9. USE OF OPC’s In CDRH • Some Current Use of OPC’s • Cardiac Ablation Catheters • Replacement Heart Valves • Ophthalmics • Hip Replacement Systems

10. USE OF OPC’s In DEVICE TRIALS • Given that the use of OPC represents a significant departure from the standard scientific approach to the design and analysis of medical device clinical trials, the question now becomes: when might it be appropriate to use this type of non-control comparison in the medical device approval process?

11. CITED ADVANTAGES Of OPC’s • Smaller Sample Size • Standard Value for All Sponsors • Save Time and Money • Easier to Execute • I.E. – “Least Burdensome”

12. DISADVANTAGES Of OPC’s • All the Problems Associated with Historical Controls • Problems with Validity of Data & Analysis • Problems with Advances in Practice of Medicine • May be Disagreement on Final OPC Value • Problems with Single Arm Trials • Selection Bias • May Not See Pre-Post Benefits

13. DISADVANTAGES Of OPC’s • May Not Be Least Burdensome • Smaller Sample Size? N=100 or 150? • Time and Resource Intensive to Develop • Sets a Minimum Standard • Superiority? • Older and Older Data • Protect and Promote the Public Health?

14. WHEN MIGHT OPC’s Be USED? • History of OPC’s in Medical Devices • Non-Experimental or Quasi-Experimental Designs May be Valid in Certain Situations

15. MINIMUM REQUIREMENTS WHEN OPC’s MIGHT Be USED • Great Deal is Known About the Natural History of the Disease or Condition • Underlying Patient Population is Well Described & Relatively Stable • Extensive Clinical History & Experience with This Device

16. MINIMUM REQUIREMENTS WHEN OPC’s MIGHT Be USED (cont.) • Stable and Well Known Standard of Care • Appropriate Ancillary Technology is Relatively Stable • No Significant New Questions of Safety or Effectiveness • Consensus Among FDA, Industry, Clinical, Academic and Patient Communities • Expectation of Significantly Positive Treatment Effect

17. HOW SHOULD OPC’s BE DERIVED? • Data Driven • Rigorous And Scientifically Valid Methodologies • Valid Databases • Appropriate Statistical Modeling • Appropriately Designed and Powered Pivotal Trial • Periodical Re-Evaluation and Updating the OPC’s

18. OTHER IMPORTANT OPC ISSUES • All Parties Must Have a Clear Understanding of the OPC in Medical Device Trials – Success and Failure • OPC’s Already in Limited Use in Device Trials • No Current General Policy of OPC Use in CDRH • Need for Comprehensive, Coherent, and Consistent OPC Policy for Device Trials

19. WHAT SHOULD THIS POLICY CONTAIN? • Clear Definitions of All Appropriate Terms • Established Guidance on Minimum Standards for a Device or Device Class • Specific Roles Played by FDA, Industry, Clinical Community, Academia and Other Interested Parties • Provisions for Periodic Updating of OPC • Specific Guidance on Methodology to Derive an OPC • Unambiguous Policy Regarding Failure to Meet OPC

20. SUMMARY • An introduction to the history and use of Objective Performance Criteria (OPC) in medical device evaluation • Advantages and disadvantages of OPC’s • When and how OPC might be used & developed • Discussed the need for developing a CDRH policy regarding use of OPC in medical device trials • General outline of this policy is presented in an effort to begin dialogue between CDRH, industry, clinical community & other interested parties

21. REFERENCES • Please see accompanying brief commentary • Request electronic copy from george.koustenis@fda.hhs.gov

22. Q&A