Adaptive Stream Resource Management Using Kalman Filters

1. Adaptive Stream Resource Management Using Kalman Filters Ankur Jain٭, Edward Y. Chang and Yuan-Fang Wang Univ. of California, Santa Barbara SIGMOD 2004

2. Outline • Data Streams • Introduction to data streams and common applications • Resource management in data streams • Application of Kalman Filters • Introduction to Kalman Filters • Adaptive Stream Resource Management using Kalman Filters SIGMOD 2004

3. Data Streams • A Data stream is a continuous sequence of tuples • Unbounded in size • Tuples arrive online • Unpredictable/variable data arrival characteristics • Real-time requirements • Imprecise/noisy data (from sensors) SIGMOD 2004

4. Applications • Sensor networks • Monitor temperature in a nuclear reactor • Network monitoring & traffic engineering • Monitor HTTP traffic on a network link • Financial tickers • Find stocks gaining more than 5% in last 30 minutes SIGMOD 2004

5. Query Precision Sampling Rate Sliding Window Size Data Filtering Data Sampling Data Forwarding Query Processing Query Evaluation Resource Management Storage Stream Synopsis A Data Stream Management System Streaming Data Sources DSMS User Query Streaming Query Result SIGMOD 2004

6. Resource Management • Communication • Limited bandwidth and high variance in availability • Power • Processing and transmitting data at remote source • CPU • Processing data at the server and the remote source • Memory • Limited memory for processing unbounded streams SIGMOD 2004

7. Communication Resource Management • Adaptive data filtering • STREAM [OJW03] • Adaptive load shedding • Aurora [TCZ+03], STREAM[BDM03] • Adaptive data sampling • TinyDB[MFH+03] SIGMOD 2004

8. Adaptive Filtering of Data Streams • Some applications do not require exact precision for the queries • Tradeoff between query precision and resource usage • Data filtering according to the query precision SIGMOD 2004

9. 8 6 Value 4 2 1 2 3 4 5 Time Caching vs. Prediction Kalman Filter ? Prediction Model Caching Data Model SIGMOD 2004

10. Outline of the Remaining Talk • Data Streams • Introduction to data streams and common applications • Resource Management in data streams • Data Streams and the Kalman Filter • Introduction to Kalman Filters • Adaptive Stream Resource Management using Kalman Filters SIGMOD 2004

11. Introduction to Kalman Filter (KF) • A prediction/correction algorithm used for state estimation (developed in 1960 by R.E. Kalman) • KF is used for • Prediction – based on previous measurements and given state model • Estimation – when measurements are made in noisy environment SIGMOD 2004

12. Common Applications of the KF • Tracking missiles • Tracking moving objects • Computer vision • Extracting lip motion from video • Data fusion/integration • Integration of spatio-temporal video segments • Robotics • Robust estimation and sensor data noise reduction SIGMOD 2004

13. The Discrete Kalman Filter State Model Measurement Model SIGMOD 2004

14. The Discrete Kalman Filter State Estimate Kalman Gain SIGMOD 2004

15. Projects the current state estimate Adjusts the current state estimate The KF cycle Time Update (Predict) Measurement Update (Correct) SIGMOD 2004

16. A Simple Example - Tracking Y X SIGMOD 2004

17. Actual Estimate Tracking Example SIGMOD 2004

18. KF and Data Streaming • Capability to model wide range of state transition functions • Robustness during unavailability of measurements • Low computational complexity for simple problems SIGMOD 2004

19. Central Server (Running KFs) Remote Source (Running KFc) Forward update received from the remote source Update the central server with new value YES YES Update available from remote source ? Prediction at central server outside the query precision ? Streaming Source NO NO Drop the data tuple Forward prediction from KFs Dual Kalman Filter (DKF) SIGMOD 2004

20. Design goals of DKF • Develop DKF as a general and adaptive stream filtering solution • Static precision thresholds • Make tradeoff between query precision and resource usage • Test performance on real and synthetic data sets • Compare against data caching model SIGMOD 2004

21. Source value when estimation error is large Central Server Continuous query results Vn V1 V2 Continuous Query KF state transition/error covariance matrices (rare) Streaming Sources DKF Architecture Continuous Query Evaluator SIGMOD 2004

22. Tracking - Dataset SIGMOD 2004

23. Results - Tracking SIGMOD 2004

24. Results - Monitoring Electric Load in a Power Zone SIGMOD 2004

25. Issues and Challenges • Setting sampling rates and thresholds • Avoiding too much computation at sensors • Sensitivity vs. Precision vs. Adaptability! SIGMOD 2004

26. Future Work • Adaptive update of state transition matrices can further improve performance • Evaluation of more complicated filters (e.g. particle filters) that can improve effectiveness • Models for non-linear systems can increase generality SIGMOD 2004

27. Central Server (Running KFs) Remote Source (Running KFc) Forward update received from the remote source Update the central server with new value YES YES Is update available from remote source ? Is prediction at central server outside the query precision ? Streaming Source NO NO Drop the data tuple Forward prediction from KFs Thank You ! SIGMOD 2004