Enhancing Network Monitoring and Anomaly Detection at the Wisconsin Advanced Internet Lab

1. Distributed Network Monitoring in the Wisconsin Advanced Internet Lab Paul Barford Computer Science Department University of Wisconsin – Madison Spring, 2002

2. Motivation • Many applications that run over the Internet have minimum performance requirements • The network is one of the two possible sources of poor performance • Wide area network behavior is unpredictable • IP networks are best effort • Constant change is normal • Quality of service capability is not widely deployed • Will it ever be available? pb@cs.wisc.edu

3. Monitoring is a First Step • Accurate monitoring of network state can enable application adaptivity and improved network management • Data provides basis for improved models and protocols • There are many challenges in network monitoring • All features of the Internet make monitoring difficult • When, where, what, how… • Today’s focus • Network monitoring efforts at Wisconsin • Combining monitoring and analysis to understand network traffic anomalies pb@cs.wisc.edu

4. The Wisconsin Advanced Internet Lab • Next generation environment for network research • Our focus: performance, management, security • Platform for testbeds: storage, grid computing , … • Internal environment • Instances of end-to-end-through-core Internet paths • External environment • Measurement nodes deployed across the Internet pb@cs.wisc.edu

5. WAIL’s External Environment • Existing infrastructure • WAWM systems (10) • Surveyor systems (60) • Partnership with Advanced Systems • NIMI systems (45) • Partnership with PCS and ICIR • Condor/Grid Infrastructures • Prototype system is under development • Passive flow measurements • FlowScan data from UW, Internet2, others(?) pb@cs.wisc.edu

6. WAIL’s Internal Environment • Complement to external facilities • Hands-on test bed which creates paths identical to those in the Internet from end-to-end-through-core • Variety of highly configurable equipment • Why do we need an internal lab? • Enables instrumentation and measurement of entire end-to-end system • Enables new systems and protocols to be implemented in places where access is not possible in wide area • Vision of internal lab: New means for doing network research • Status: Significant commitment from industry partners (Cisco, EMC, Fujitsu) and the university – rev. 1.0 by 5/1/02 pb@cs.wisc.edu

7. Distributed Anomaly Detection • Motivation: Anomaly detection and identification is an important task for network operators • Operators typically monitor by eye using SNMP or IP flows • Simple thresholding is ineffective • Some anomalies are obvious, other are not • Focus: Characterize and develop distributed means for detecting classes of anomalies • Network outages, Flash crowds, Attacks, Measurement failures • Approach: Use statistical and wavelet techniques to analyze anomalies from IP flow and SNMP data from UW and other sites • Implications: Tools and infrastructure which quickly and accurately identify and adapt to traffic anomalies pb@cs.wisc.edu

8. Characteristics of “Normal” traffic pb@cs.wisc.edu

9. Our Approach to Analysis • Analyze examples of each type of anomaly via statistics, time series and wavelets (our initial focus) • Wavelets provide a means for describing time series data that considers both frequency and scale • Particularly useful for characterizing data with sharp spikes and discontinuities • More robust than Fourier analysis which only shows what frequencies exist in a signal • Tricky to determine which wavelets provide best resolution of signals in data • We use tools developed at UW Wavelet IDR center • First step: Identify which filters isolate anomalies pb@cs.wisc.edu

10. Analysis of “Normal” Traffic • Wavelets easily localize familiar daily/weekly signals pb@cs.wisc.edu

11. Example Anomaly: Attacks • DoS: sharp increase in flows and/or packets in one direction • Linear splines seem to be a good filter to distinguish DoS attacks pb@cs.wisc.edu

12. Characteristics of Flash Crowds • Sharp increase in packets/bytes/flows followed by slow return to normal behavior eg. Linux releases • Leading edge not significantly different from DoS signal so next step is to look within the spikes pb@cs.wisc.edu

13. Characteristics of Network Anomalies • Typically a steep drop off in packets/bytes/flows followed a short time later by restoration pb@cs.wisc.edu

14. Summary and Conclusion • Accurate network monitoring is essential for improving application performance and network management • The Wisconsin Advanced Internet Lab provides a unique environment for network monitoring • Wavelets are an effective means for identifying anomalous behavior in data gathered from IP flow and SNMP interface monitors • Details on distributed and coordinated monitoring and analysis available this spring pb@cs.wisc.edu