160 likes | 353 Vues
Multi-Scale Analysis for Network Traffic Prediction and Anomaly Detection. Ling Huang Joint work with Anthony Joseph and Nina Taft January, 2005. Problem Statement. Accurate network traffic modeling and prediction are important for: Network provisioning Problem diagnosis
E N D
Multi-Scale Analysis for Network Traffic Prediction and Anomaly Detection Ling Huang Joint work with Anthony Joseph and Nina Taft January, 2005
Problem Statement • Accurate network traffic modeling and prediction are important for: • Network provisioning • Problem diagnosis • But, network traffic is highly dynamic • Exhibits multi-timescale properties (temporal domain) • Anomalies (failures, attacks) interfere with analysis • Need to isolate anomalies from normal traffic variation to achieve better modeling and prediction
Our Work • We decompose traffic signals into two parts • Normal variations: follow certain law, can be modeled and are predictable • Anomalies: consist of sudden changes and are not predictable • Method: Multi-scale signal analysis and modeling for • Network traffic prediction [almost done] • Volume anomaly detection and identification [in progress]
Properties of Real Life Data Streams Three real-world data traces and a random trace
Long-term trend Observation: Multi-Scale Property Sudden Changes (anomalies) Transient Variation Multi-scale property of traffic data in a weekly measurement Data source: bandwidth measurements for the CUDI network interface on an Abilene router with 5-minute average.
Multi-Scale Prediction of Future Data • Traditional Approaches • Last seen data as approximation for current estimation • Linear Prediction • Exploit and leverage statistical properties of data stream on temporal domain in a window of size T • Exploit temporal correlation in different time scale • Long term trend: B-spline estimation • High frequency residual: ARMA modeling • ARMA stands for AutoRegressive and Moving Average model, which is a standard time series technique to model chaotic data stream
Two-Level Modeling and Prediction • B-spline modeling for long term trend • Piecewise continuous, low-degree B-spline can represent complex shapes • Least-square B-spline regression for two-level decomposition • B-Spline extension for future forecasting • ARMA forecasting for transient oscillation • System Identification to determine the order of the model • Parameter estimation by optimization algorithm • Low complexity recursive equation for future forecasting • Statistical properties for the calibration of prediction results
Complexity of Prediction Algorithms • Legend • T: the window size of history data • m: the order of the linear predictor • K: the order of the ARMA model • d: the degree of B-spline curve • c: the increase in storage due to multi-level data representation
Performance of Prediction Algorithms Mean Relative Increment Performance of Prediction Algorithms On Network Traffic
Unpredictability: Anomalies in Data Signal Long-term trend (modeled) Residual with anomalies Monday Data The data stream can be decomposed into two layers: the long-term trend, which is the modeled pattern; the residual, high frequency with anomalies
Anomalies Detection and Identification • Volume anomaly: Sudden change in link/flow’s traffic count • Network failures, attacks, flash crowds • Measurement anomalies • Anomalies are not normal variations of network traffic and are not predictable • Worse yet, anomalies skew the prediction models! • For better modeling and prediction, need to detect and isolate anomalies from data • The rest of the talk focuses on anomaly detection algorithms • Existing algorithms: single-link vs. network-wide analysis • New directions
Single-link Anomalies Detection • Multi-scale analysis to capture temporal correlation • Use wavelets for multi-scale data decomposition • Isolate characteristics of traffic signal on different timescales • Expose the details of both ambient (modeled) and anomalous traffic • Detection of sharp increase in the local variance in a moving time window on different time-scale • Disadvantages • Diagnose on single link or at single router, and is impractical to do analysis for large network • Many anomalies are across multiple links, and is not obvious on single link
Network-Wide Anomalies Detection • Diagnose traffic anomalies spanning multiple links • Capture the spatial correlation cross links • Analyze origin-destination flows with known traffic matrix • Principle Component Analysis (PCA) for dimension reduction and signal decomposition • Separate traffic signal into modeled and residual space • Scoring/prediction method to detect abnormal changes in residual space • Disadvantages • Need ISP support • Single time scale analysis • Centralized algorithm
New Direction (1): Multi-Scale PCA • PCA analysis on wavelets representation of traffic data • Wavelets capture correlation within a single link • PCA captures the correlation across links and transforms the multivariate space into a subspace which preserves maximum variance of the original space • PCA analyzes data on single time-scale and can not utilize the information pertaining to the frequency • Multi-scale PCA combines two extremes of wavelets and PCA based analysis of multivariate data • Benefits: detection of multi-timescale anomalies
New Direction (2): Distributed Algorithm • Distributed anomalies detection based on partial flow information • No ISP support, no information about OD flows and traffic matrix • Diagnose volume anomalies based on network monitoring data – flow and link information from a subset of places in the network • Network-wide traffic modeling, inference and prediction improve measured data • Distributed algorithms for network-wide data reduction, decomposition and anomaly detection • Collective PCA • Benefit: local analysis and low cost
Conclusion and Future Work • Summary • Apply statistical algorithms to network traffic analysis • Multi-scale analysis is effective in traffic modeling and prediction • Contribution: using multi-scale network-wide analysis to capture both temporal correlation within single link and spatial correlation cross links • Future Work • Develop distributed algorithms based on multi-scale PCA • Exploit tradeoff between detection accuracy, false positive and computation cost • Build real system for applications in traffic analysis and network health monitoring