The Science of Outbreak Detection: A tutorial

1. The Science of Outbreak Detection: A tutorial RODS: http://www.health.pitt.edu/rods Auton Lab: http://www.autonlab.org

2. Two Basic Goals of R&D • Improve the detection of cases • Improve the detection of outbreaks

3. What are the scientific questions? • What types of data should be collected (and how)? • Which data analytic methods (algorithms) are best for each problem? • How much earlier can an outbreak be detected? • How small of an outbreak can be detected? • Ultimately, how much reduction in morbidity and mortality can be achieved? For each organism and for every type of outbreak of that organism!

4. What are the Methods? • Experimental design for Case detection • Experimental designs for outbreak detection • Real data (“Experiments of nature”) • Case studies • N>1 studies (studies with statistical power) • Simulated data (“Injections”) • Semi synthetic (inject synthetic outbreaks into real surveillance data) • Hi fidelity injects • Completely synthetic • Experimental designs for assessing a data source • Correlation analysis • Detection algorithm method

5. Experimental Design for Case Detection • What is measured: sensitivity and specificity for detecting cases of say “Respiratory Illness” • Standard medical informatics experiment • Assemble 100 cases of TB and 100 controls • Run some case classification algorithm on surveillance data • Count up the true positives, false positives, true negatives, false negatives • Compute the sensitivity and specificity of the algorithm • Draw ROC curves and measure the Area Under the Curve (AUC) • Preferably also have a benchmark (human expert, alternative algorithm) against which to compare the results

6. Example: Evaluation of Natural Language Processing Naive Bayes Text Classifier P(Respiratory|NVD)= .05 P(Botulinic|NVD)= .001 P(Constitutional|NVD)= .01 P(GI|NVD) = .9 P(Hemorrhagic|NVD)= .001 P(Neurologic|NVD)= .001 P(Rash|NVD)= .001 P(None|NVD)= .036 1. Ivanov, O., M.M. Wagner, W.W. Chapman, and R.T. Olszewski, Accuracy of three classifiers of acute gastrointestinal syndrome for syndromic surveillance. Proc AMIA Symp, 2002: p. 345-9. 2. Chapman, W.W., J.U. Espino, J.N. Dowling, and M.M. Wagner, Detection of Acute Lower Respiratory Syndrome from Chief Complaints and ICD-9 Codes. Technical Report, CBMI Report Series, 2003. 3. Wagner, M., J. Espino, F.-C. Tsui, P. Gesteland, B.E. Chapman, O. Ivanov, A. Moore, W. Wong, J.N. Dowling, and J. Hutman, Syndrome and Outbreak Detection from Chief Complaints: The Experience of the Real-Time Outbreak and Disease Surveillance Project. MMWR (under review), 2004. 4. Chapman, W.W., J.U. Espino, J.N. Dowling, and M.M. Wagner, Detection of multiple symptoms from chief complaints. Technical Report, CBMI Report Series, 2003. 5. Chapman, W.W., J.N. Dowling, and M.M. Wagner, Fever detection from free-text clinical records for biosurveillance. J Biomed Inform, 2004. 37(2): p. 120-7. 6. Chapman, W., J.N. Dowling, and M. Wagner, Syndromic Case Classification from Chief Complaints: a Retrospective Analysis of 527,228 Patients. 2004, Technical Report, CBMI Report Series. • Chapman, W.W., L.M. Christensen, M.M. Wagner, P.J. Haug, O. Ivanov, J.N. Dowling, and R.T. Olszewski, Classifying free-text triage chief complaints into syndromic categories with natural language processing. Artificial Intelligence in Medicine, 2004: p. in press. • Gesteland, P.H., Unpublished results (see appendix). 2003. GI “N/V/D” Chief complaint Results of Many Studies

7. Experimental Design for Outbreak Detection Similar in principle to evaluation of Case detection What is measured: sensitivity, specificity, and earliness of outbreak detection • Get surveillance data from 100 outbreaks of disease X and 100 non outbreak periods • Run the detection algorithm on the surveillance data • Count the number of true positives, false positives, true negatives, false negatives • Also measure the time of detection relative to the start of the outbreak • Compute ROC curves Problem: There usually aren’t 100 outbreaks!

8. Nevertheless, it is occasionally possible

9. SCRANTON, PA HARRISBURG, PA PITTSBURGH, PA PHILADELPHIA, PA SALT LAKE CITY, UTAH INDIANAPOLIS, IN Study of 18 Outbreaks: Electrolyte Sales Precede Hospitalization by >2 Weeks Data courtesy IRI, Utah DOH, Indianapolis Network for Patient Care, and PA HC4 Council

10. CASE STUDY: PA CARBON MONOXIDE 2003 Chronology July 18, 2003: RODS pages the on-call epidemiologist at 8:06 PM about a very anomalous number of cases of respiratory syndrome from a single county outside of Pittsburgh The on-call epidemiologist reviewed the verbatim chief complaints of these patients from home Significance • Detection of the event occurred by automatic statistical analysis of chief complaint data within five hours of the first case presenting to hospital. • The system worked end-to-end, including the human element. • If this outbreak had been Anthrax, the alert would have triggered an immediate, focused investigation. More case studies: https://www.rods.pitt.edu/rods2

11. Case Studies • Case Study #1: PA Carbon Monoxide 2003 • Case Study #2: Utah Flu Outbreak 2003 • Case Study #3: Southern California Wildfires 2003 • Case Study #4: Philadelphia Gastroenteritis at Loews Hotel 2003 • Case Study #5: Philadelphia Shigella Outbreak 2003 • Case Study #6: Fort Wayne, Indiana, Cough&Cold 2003 • San Diego Heat Wave & Electrolyte sales 2004 (Jeff Johnson) • Influenza in Kentucky 2003/04 (Carl Hall) • Norovirus in Washoe County, Nevada 2003 (Lei Chen) • Influenza in LA 2003 (Ray Aller) • Influenza in Allegheny County 2004 • Hepatitis A in Beaver County • GI in Putin-Bay, Ohio • Norovirus in Coraopolis

12. Semisynthetic Injection • Basic idea: Inject a geometric spike into a surveillance data time series taken from non outbreak period • Advantage: • Can vary the size of the spike to find the smallest detectable spike • Can inject the spike on each day of the time series to get a large sample to compute confidence intervals • Disadvantages: • Geometric shape • Leaves unanswered the question of how much epidemic does it take to create that big of a spike

13. Hi Fidelity Injection North Battleford Crypto Outbreak Method Public Water Drinking Advisory • Bayesian adaptive regression splines are used to estimate the shape and size of a spike in surveillance data during a real outbreak in region A. • The spike is re-scaled and injected into real surveillance data for region B. • Detection algorithms are used to determine how small of an outbreak could be detected and to measure the timeliness of detection. Antidiarrheal Sales ROC-like curve • Advantages • Can vary the size of the spike to find the smallest detectable spike • Can inject the spike on each day of the time series to get a large sample to compute confidence intervals • Answers the question of how small of an outbreak is detectable 10% 1% Sensitivity 0.1% Affected Days into outbreak More info: paper on CD

14. Experimental Designs for Assessing a Data Source • The questions are: • Is there signal of an outbreak? • How early is it? • The methods are: • Correlation analysis • Detection Algorithm Method

15. Pneumonia and influenza Respiratory chief complaints Daily counts Date Daily counts of respiratory chief complaints predict Influenza. 1996-2003 Utah. Correlation Analysis • Obtain two time series: surveillance data and gold standard • Normalize by plotting number of standard deviations from mean. Result is the strength of correlation And the time shift at which correlation is maximum

16. Detection Algorithm Method • A simple idea • Simply measure whether outbreaks can be detected using the data source in question and some detection algorithm, and when • Output is sensitivity, false alarm rate, and earliness • Requires a gold standard determination of existence and date of onset of outbreak

17. Do we have to do evaluations of every disease (and route of transmission)? -Or-How to deal with a very big problem space Remember: For each organism and for every type of outbreak of that organism!

18. Which Organisms? The Nation's Current Capacity for the Early Detection of Public Health Threats including Bioterrorism. June 8, 2001.

19. Method to Aggregate Threats into 9 Detection Problems • Assemble exhaustive list of pathogens • For each pathogen (loop) • Expand the list by identifying the spectrum of outbreak size, routes of transmission or differences in release (indoor outdoor) • Refer to each as a ‘threat’ • For each threat (loop) • Experts asked what kind of automated detection system would be needed to detect each threat. • IF the detection system involved DIFFERENT data and algorithm create a new category ELSE add the threat to an existing category • end

20. List of Detection Problems • Large scale bioaerosol (e.g., Anthrax) • Communicable (e.g., SARS) • Waterborne • Building contamination • Foodborne • Vector borne • Continuous release • Single case • Sexual/blood borne Some of these categories happen to correspond to divisions in health departments, others are new

21. Example of Two Patterns and Implications for Detection Systems 4 5 3 6 Symptoms Presentation to physician Deaths 3 4 5 6 Anthrax or other bioaerosol, food contamination, water contamination Contagious disease like Smallpox Early warning system that collects unorthodox, nonspecific data and looks for anomalous patterns Embedded diagnostic expert systems at the point of care, probably requires POE Wagner, Dato et al. Data Required for an Effective Bioterrorism Detection System. Report to AHRQ, 11/28/01 180 pp.