Reinforcement Learning for Optimizing Disease Outbreak Detection Methods

1. Optimizing Disease Outbreak Detection Methods Using Reinforcement Learning Masoumeh IzadiClinical & Health Informatics Research Group Faculty of Medicine, McGill

2. Overview • Motivation • Problem formulation • Basic definitions • The suggested method • Experimental results • Concluding remarks

3. The Surveillance Cycle 1. Identifying individual cases 2. Detecting population patterns 3. Conveying information for action Individual Event Definitions Population Pattern Definitions Intervention Guidelines Public Health Action Event Reports Pattern Report Event Detection Algorithm Pattern Detection Algorithm Intervention Decision Data Describing Population Population Under Surveillance (Buckeridge DL & Cadieux G, 2007)

4. Surveillance Research • Achieving the National Electronic Disease Surveillance System (NEDSS) architecture • Data fusion (linkage) • New data sources • Case definitions (automation/validation) • Geographic Information System (GIS) indices • Forecasting • Evaluation and quality control

5. The Surveillance Cycle 1. Identifying individual cases 2. Detecting population patterns 3. Conveying information for action 2. Using RL to identify optimal policies for responding to statistical alarms. 1. Accounting for population mobility in detecting spatial disease clusters. Individual Event Definitions Population Pattern Definitions Intervention Guidelines Public Health Action Event Reports Pattern Report Event Detection Algorithm Pattern Detection Algorithm Intervention Decision 3. Simulation modeling to evaluate outbreak detection. Data Describing Population Population Under Surveillance Decision Algorithm Knowledge (Buckeridge DL & Cadieux G, 2007)

6. Outbreak Detection Environment Data Detection Method Warning??? Knowledge

7. Data Sources

8. Outbreak Problems • Large scale bioaerosol (e.g., Anthrax) • Communicable (e.g., SARS) • Waterborne • Building contamination • Foodborne • Continuous release • Sexual/blood borne

9. Detection Methods • Define a threshold . • Signal an alarm when the # of ED visits per day exceeds the threshold.

10. Anthrax Attacks Flu Flu Flu Data courtesy of Medstar & Georgetown University Anthrax Cases in DC

11. Existing Detection Methods • Temporal methods e.g. Moving average • Spatio-temporal methods e.g. Space-time scan

12. Features Shared by Most Detection Methods • Design a baseline. • Define an important event when the p-value of a statistic is less than an expected value by the baseline.

13. Obtaining Baseline Data All Historical Data Bayesian Biosurveillance of Disease Outbreaks [UAI04 Cooper et al] • Learn Bayesian Network using Optimal Reinsertion [Moore and Wong 2003] Today’s Environment Baseline 2. Generate baseline given today’s environment

14. Important Events • determine which of these p-values are significant for a specific problem. Idea: use association rules to define cases

15. Key Observations • There is a great amount of uncertainty about suspicious events. • An action has to be taken in response to any suspicious change in the environmental patterns. • Surveillance systems faced by high-risk decision problems under uncertainty.

16. Surveillance algorithms are inaccurate in practice • How precisely can we detect if an outbreak is happening? (sensitivity) • How early can we detect it? (timeliness) Research to address this problem • Novel or ‘improved’ data streams • Better forecasts or detection methods • Improve decision making after alarms

17. Our Approach Instead of trying to improve the detection method, we ‘post-process’ the signals: • Use a standard surveillance method to provide alarm signals • Feed this signal to the model of outbreak detection as a partially observed Markov decision process (POMDP)

18. Partially Observable MDP • POMDPs are characterized by: • States: sS • Actions: aA • Observations: oO • Transition probabilities: T(s,a,s’)=Pr(s’|s,a) • Observation probabilities: T(o,a,s’)=Pr(o|s,a) • Rewards: R(s,a)

19. Solving POMDPs • To solve a POMDP is to find, for any action/observation history, the action that maximizes the expected discounted reward. V(b)= max a [Σs R(s,a)b(s)+ Σs’ [T(s,a,s’)O(s’,a,z)α(s’)]] OUTCOME: an optimal policy over belief space

20. Suitability • The ‘true’ state of the outbreak cannot be observed • Statistical algorithms provide imperfect measurements of the true state • That the probability of success of (i.e., effectiveness) of actions can be determined • The that costs of actions and of outcomes can be determined

21. Limitations for inhalational anthrax • Limited data from actual anthrax attacks available: • Postal attacks 2001 (Only 11 people affected, not representative of a large scale attack) • Sverdlovsk 1979 • But literature contains studies on the characteristics of inhalational anthrax

22. Background knowledge for inhalational anthrax Can coherently incorporate different types of simulation data : • Progression of symptoms • Incubation period • Spatial dispersion pattern

23. The POMDP Model S - True epidemic state {No Outbreak, D1, ….} O - Output from detection algorithm {0,1} A - Possible public health actions T(s,a,s’) - Impact of actions given the state R(s,a) - Costs of actions and of epidemic states Action Transition Do nothing Review records Investigate cases Declare outbreak (Izadi M & Buckeridge DL, 2007)

24. Clear Day 1 s Day 2 Day 3 Day 4 Detected Clear D1 D2 D3 D4 Det Transition Functions • The transition functions reflect the probability of moving to another state if an action is performed in each state of the model. T: Review records 0.99 0.01 0.0 0.0 0.0 0.0 0.0 0.0 0.9 0.0 0.0 0.1 0.0 0.0 0.0 0.7 0.0 0.3 0.0 0.0 0.0 0.0 0.5 0.5 0.0 0.0 0.0 0.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 T: Investigate 0.99 0.01 0.0 0.0 0.0 0.0 0.0 0.0 0.7 0.0 0.0 0.3 0.0 0.0 0.0 0.4 0.0 0.6 0.0 0.0 0.0 0.0 0.1 0.9 0.0 0.0 0.0 0.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 s’

25. Observation Functions Observations are noisy output of the detection algorithm • Alarm -sensitivity at outbreak states and 1 - specificity in the no outbreak state. • No Alarm -specificity at normal states and 1 - sensitivity in each outbreak state.

26. Sensitivity versus Specificity

27. Sensitivity in Days of Outbreak Reis et al. (2003) Proc. Natl. Acad. Sci. USA 100, 1961-1965

28. Costs and Reward • Costs • Investigation (false and true positive) • Intervention (false and true positive) • Outbreak by day (false negative) calculated as (# deaths* future earnings) + (# hospitalized * cost of hospitalization) + (# outpatient visits * cost of visit) • Rewards • Preventable costs each day - investigation / intervention costs Sources • Investigation costs are estimated from wages • Intervention and outbreak costs from (Kaufman, 1997)

29. Experimental Setup • There is a constant probability of an outbreak. • Epidemic curve taken from historical outbreak. • After 4 days, the outbreak is detected clinically. • Population size is 50,000 exposed and the outbreak results in a mean increase in surveillance data of 8% or 15% • POMDP solution • Point-based approximation • Ran simulation for ten years.

30. Things to Notice • Any alerts before actual anthrax release are considered a false positive • Detection time calculated as first selection of C/P action after anthrax release. • Maximum detection time is 4 days.

31. Preliminary Results

32. Initial Evaluation Results 8% Increase in ED visits 15% Increase in ED visits Sensitivity Sensitivity Day of Outbreak Day of Outbreak • Compared POMDP operating on detection method, to detection method alone • Method was SARIMA + MA on residuals • Specificity of 0.97 for the detection method used

33. Final Words • Conclusion: • POMDP improves the timeliness and the sensitivity of detection processes • Future work: • Sensitivity analysis over parameter values. • Apply to other diseases and in other settings!

34. Thank You