Adaptive Stimulation Design for the Treatment of Epilepsy

1. Adaptive Stimulation Design for the Treatment of Epilepsy Joelle Pineau School of Computer Science,McGill University Montreal, QC CANADA Jointly with Robert Vincent, Aaron Courville, Massimo Avoli SAMSI Program on Challenges in Dynamic Treatment Regimes and Multistage Decision-Making June 21, 2007

2. Background Vagus nerve and deep brain stimulation are used to treat various neurological disorders, including epilepsy. Images from www.epilepsyfoundation.org and www.cyberonics.com SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

3. Project goal • Problem: Existing devices offer limited control and do not adapt to the patient’s condition. • Idea: Create an improved class of devices with closed-loop control. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

4. A reinforcement formulation • Objective: Minimize occurrence of seizures and overall amount of stimulation. • The MDP model: • States, st: recordings of electrical activity • Actions, at: stimulation (frequency, voltage, location) • Transitions, P(st|st-1, at) : unknown • Rewards, rt: large cost for seizures, small cost for stimulation at at+1 at-1 st st-1 st+1 … rt rt-1 rt+1 SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

5. Technical challenges • Investigate supervised learning methods for automatic seizure detection to inform choice of good state representation. • Design a computational (generative) model of epilepsy. • Run reinforcement learning using online data from the computational model. • Run reinforcement learning using batch data from an in-vitro model. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

6. Recordings of electrical activity • Recorded from single sensing electrode in in-vitro model of epilepsy. • Raw data:4096-sample frames  windowed, normalized, FFT • xt = 83 real-valued features: mean, range, energy, 80 FFT magnitudes • yt ={normal, spike, seizure} hand-labeled for each frame. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

7. Seizure detection • Long literature on seizure detection using supervised learning (e.g. RBFs, wavelets, neural networks, energy methods). • Usually uninterpretable results. • Related literature in time-series prediction (e.g. HMMs, CRFs). • Requires feature selection. • Instead, focus on ensemble boosting methods: • Extend standard multi-class Adaboost [Schapire&Singer, 1999] to a recurrent formulation, in whichyt = f (xt , yt-1 , yt-2 , …, yt-k). • Relatively interpretable results (assuming simple learners). • Requires no prior model of the distribution of features. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

8. Classification accuracy yt = f (xt , yt-1 ) SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

9. Feature extraction = insight into state design • In all recurrent examples, first weak hypothesis recruited was frequency band 62 or 63 (= 76-77 Hz). • High value normal • Low value spike • Frequency bands 6-8 (=7-10 Hz) also consistently recruited early. • High value spike • Low value normal • Often recruited in the first 20 rounds is Energy. • High value spike • In recurrent Adaboost, prior label often recruited early and acts as memory. • High spike prior spike etc. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

10. Technical challenges • Investigate supervised learning methods for automatic seizure detection to inform choice of good state representation. • Design a computational (generative) model of epilepsy. • Run reinforcement learning using online data from the computational model. • Run reinforcement learning using batch data from an in-vitro model. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

11. A computational model of epilepsy • Aims of computational modeling: • To understand basic mechanisms of epilepsy sufficiently to design a good state representation. • To understand the appropriate class of policies to consider. • To have an inexpensive testing environment for RL algorithms. • A word of caution: • Epilepsy is a complex disease. Many animal models are found in the literature. We focus on model by [Avoli et al., 2002], which we will use for online exploration in-vitro. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

12. Model overview (a) Assume a simple stochastic neuron model (leaky integrate-and-fire). (b) Connect many neurons in small-world network configuration. (a) Leaky integrate-and-fire neuron model (b) Network structure SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

13. Model overview (cont’d) (c) Define a sensor model: • Voltage measurement over a patch of adjacent neurons (neuron contribution falls off in inverse-square relationship with distance from the patch center). (d) Define the stimulation model: • Input current applied uniformely to a “patch” of neural units. vsensor istim SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

14. Traces from the computational model # neurons firing sensor voltage SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

15. Results from fixed stimulation strategies 0 Hz 0.5 Hz 1.0 Hz 2.0 Hz 4.0 Hz 5.0 Hz SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

16. Technical challenges • Investigate supervised learning methods for automatic seizure detection to inform choice of good state representation. • Design a computational (generative) model of epilepsy. • Run reinforcement learning using online data from the computational model. • Run reinforcement learning using batch data from an in-vitro model. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

17. Reinforcement learning agent • States: st = {sensor reading} x {# time steps since last stimulation} discretized into small number of (independent) features • Actions: a1 = Stimulator on for 1 time-step. a2 = Stimulator off for 1 time-step. • Rewards: R(s,a) = -100 for a seizure (i.e. firing count > 50) R(s,a) = -10 for a stimulation R(s,a) = 0 otherwise • Training details: • Online data, finite horizon (60 x 100-sec traces), -greedy exploration, Sarsa() with eligibility SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

18. Results: Learning (a) After 1 learning episode (b) After 10 learning episodes SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

19. Results: Expected return SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

20. Results: Policy SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

21. Technical challenges • Investigate supervised learning methods for automatic seizure detection to inform choice of good state representation. • Design a computational (generative) model of epilepsy. • Run reinforcement learning using online data from the computational model. • Run reinforcement learning using batch data from an in-vitro model. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

22. Learning from batch in-vitro data We have data of the type used in the classification task, but which includes fixed-policy stimulation. • States: 83 real-valued features (electrical signal sampled + FFT) • Actions: {0 Hz, 0.5 Hz, 1.0 Hz, 2.0 Hz, 5.0 Hz} • Training: batch (~30 x 60-sec traces), tree-based regression. No results SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

23. Wrap-up • Preliminary evidence for: • ability to detect seizures using in-vitro data. • generation of synthetic data with epileptiform behavior. • controllability of the stimulator in the computational model. • Results apply to a specific model of epilepsy - generalization to other models is unknown. • Issues I did not discuss today: • Designing the reward function. • Using the features found in Adaboost within the RL agent. • Transferring what we have learned in the computational model to the biological model. • Learning from very few data points. SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

24. Available data • Recordings used for classification (no stimulation) • http://www.cs.mcgill.ca/~jpineau/datasets/epilepsy.tar.gz • Recordings used for batch reinforcement learning (fixed policy stimulation) • Subject to approval by neuroscientists (1-2 weeks delay). • Generative model of epilepsy • Still under development, but publicly available soon (1-2 months). SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy