Data Mining in VAERS to Enhance Vaccine Safety Monitoring at the FDA

1. Data Mining in VAERS to Enhance Vaccine Safety Monitoring at the FDA Robert Ball, MD, MPH, ScM Dale Burwen MD, MPH M. Miles Braun, MD, MPH Division of Epidemiology Office of Biostatistics and Epidemiology DIMACS, October 18, 2002

2. What is the Vaccine Adverse Event Reporting System (VAERS)? • National system for surveillance of adverse events after vaccination initiated by National Childhood Vaccine Injury Act 1986 and established 1990 • Jointly managed by FDA and CDC • Reports received from health professionals, vaccine manufacturers, and the public

3. Post-licensure Safety Monitoring • How we do it • VAERS • Potentially rapid detection of signal of new safety concern • Rarely allows determination of causality • Enhanced surveillance • Obtain standardized information on reports • Controlled studies of hypothesized causal relationships raised in surveillance • Communicate results

4. Uses of VAERS • Detecting unrecognized adverse events • Monitoring known reactions • Identifying possible risk factors • Vaccine lot surveillance

5. Limitations of VAERS • Reported diagnoses are not verified • Lack of consistent diagnostic criteria • Wide range in data quality • Underreporting • Inadequate denominator data • No unvaccinated control group • Usually not possible to assess whether a vaccine caused the reported adverse event

6. Analysis of VAERS Data • Describe characteristics and look for patterns to detect “signals” of adverse events plausibly linked to a vaccine • Signals detected through analysis of VAERS data almost always require confirmation through a controlled study

7. Fundamental Problem in Assessing Spontaneous Reports • VAERS ~10-15K reports / year • AERS ~20K reports / year (CBER) • How can a sensitive system to detect potential product problems not be overloaded and overwhelmed by information to which we have to respond?

8. “Data Mining” • Identify events reported more commonly for one product than others • Proportional Reporting Ratios (PRR) • Empirical Bayesian Geometric Mean (EBGM) • Don’t account for medical knowledge or biases in reporting • EBGM algorithm implemented by Lincoln Technologies and PPD Informatics • VAERS Data Mining Environment (VDME) • PRR algorithm implemented in standard packages (e.g. SAS, STATA) on an ad hoc basis

9. Proportional Reporting Ratio • Compares the adverse event profile of one vaccine to other vaccines Number of reports with Adverse Other Adverse Total Event Y Events Vaccine X a b (a+b) Other vaccines c d (c+d) PRR = [a/(a+b)] / [c/(c+d)]

10. Proportional Reporting Ratio • Compares the adverse event profile of one vaccine to other vaccines • Evans has proposed using PRR  2, n  3, and chi square  4 as criteria for selecting pairs for further evaluation

11. Background: Empirical Bayesian Data Mining • Similar to PRR in comparing one vaccine to others • Calculates observed and expected frequencies • Observed: # of reported events/vaccine • Expected: Based on overall frequency of the event for all vaccines, and the total # of reports of the vaccine of interest • Identifies cells with very small expected counts – accounts for the instability of the small number

12. Empirical Bayesian Data Mining • Ranks vaccine-event combinations by Empirical Bayesian Geometric Mean (EBGM) • Dumouchel has proposed EBGM  2 as criterion to select pairs for further evaluation • Multi-item Gamma Poisson Shrinkage (MGPS) algorithm detects multi-way combinations • V=vaccine; S=symptom • V S • V S S • V S S S

13. Rotavirus Vaccine-Intussusception • Clinical Trials Signal • Wild type RV & intussusception study • FDA - licensure • CDC - recommendations for use • Post-marketing Surveillance (VAERS) • Background rates • Population-based incidence rates • Withdrawal

14. Rotavirus Vaccine and Intussusception:Signal Emergence

15. Vaccine Profiles

16. Anthrax Vaccine: 3-Dimensional Assessment (V-S-S)

17. Effect of Stratification on EBGM:Anthrax Vaccine and Selected COSTARTS EBGM

18. Selection of “Item Sets” for Empirical Bayesian Data Mining • The choice of “Item Sets” influences the Multi-item Gamma Poisson Shrinkage (MGPS) algorithm • Currently all combinations (e.g. 2D v-v, s-s, v-s where v=vaccine; s=symptom) • If input is restricted to only v-s combinations the magnitude of the EBGM and rank for pairs with small numbers are affected • Appropriate selection of Item Sets needs systematic evaluation

19. Effect of Item Set Selection on EBGM

20. Challenges • What is the best method? • Bayesian vs. PRR vs. other? • What are criteria for making this decision? • How should each method be applied and interpreted? • What level of PRR/EBGM? • How should statistics be interpreted?

21. Challenges • Should data mining methods be used for automated screening or as analytic tools? • Importance of stratification suggests need for intermediate level epi/stat sophistication in users • Users need training to properly interpret results • Computing resources • Substantial effort required for data preparation • Software needs user-friendly features to enhance end-user control over: • Defining data subsets of interest • Stratification • Combining adverse event terms • Selecting item sets prior to data mining

22. Challenges • Usual method of monitoring for signals: • Physician review of individual reports as they arrive • Physician review of serious reports • Committee review of serious reports at weekly meeting • Physician review of monthly numerical summaries of selected vaccines • Periodic vaccine or disease-specific surveillance summaries • Where does data mining best fit in this process? • How can data mining results be best communicated to decision makers, health care providers, and the public?

23. Next Steps and Future Challenges • Continue using PRR and Empirical Bayesian methods in routine practice • Systematic comparison of methods • Simulation study in collaboration with CDC • Large size of AERS database, especially with 2 way and 3 way interactions • Is simpler better? e.g. PRR with chi-square • Drug dictionary in AERS

24. Summary • Automated summary of a large amount of data • Potential for improving usual methods of monitoring for signals • Other methods should also be considered • Further understanding and experience is needed

25. Acknowledgments • FDA • Manette Niu, Phil Perucci, other CBER staff, Ana Szarfman and other CDER staff • CDC • Henry Rolka, Vitali Poole, Penina Haber, John Iskander, and other CDC staff • Others • Lincoln Technologies, Inc. • PPD Informatics • William DuMouchel

26. Selected References • Dumouchel W. Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. American Statistician 1999;53:177-190. • Evans SW, Waller PC, Davis S. Use of proportional reporting ratios for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf 2001;10:483-486. • Niu MT, Erwin DE, Braun MM. Data mining in the US Vaccine Adverse Event Reporting System (VAERS): early detection of intussusception and other events after rotavirus vaccination. Vaccine 2001;19:4627-4634.