Traffic Instrumentation and Management

1. Traffic Instrumentation and Management CSG, January 2002

2. Traffic Instrumentation • What are you looking for? • How’s the bandwidth being spent? • Locate anomalies • Intrusions • Outgoing Denial of Service (DoS) attacks • Where should you look? • Gateway routers get us most of what we want

3. Solution: Network Logs • Network logs let you analyze past events • Log specific information: source, dest, time, amount of traffic, etc. • Packet contents are overkill • Privacy issues • Disk space • Do you need to log all connections? • Doing so allows forensics • Our disk space usage: 100GB gives 3 months

4. Network Forensics • What happened on the network? • Three Examples • Who’s launching a DoS? • Where’s the network bandwidth gone? • A compromised machine • How was it compromised? When? Where?

5. Outgoing DoS • DoSes are generally spoofed • Network logs aren’t too useful • Egress filtering helps, but the DoS tools figure out how much spoofing they can safely do (spoof from the same class C) • Blocked spoof attacks can flood network logs

6. Graphs for identifying DoS • Graphs are useful as DoS attacks stand out • How much can you graph? • Graphing each network port may be impractical • Other traffic may interfere

7. DoS Identification • If you catch the DoS while it’s occurring, you can check the current bandwidth usage on the switches • “show top pkts” • If it’s spoofed, and you don’t catch it while it’s happening… now what?

8. Where’s the Bandwidth Gone? • The “Napster” question • Use statistical analysis • Which udp/tcp ports and/or ip addresses are using a lot of bandwidth at times of high bandwidth?

9. Is it an abuser? • Is one machine using more than their fair share of bandwidth? • Look at the top ten bandwidth users • Maybe… most of the IPs are of known high bandwidth services (usenet, ftp, backup) % flow-stat -f11 < ft-v06.2002-01-06.140000 | sort -nr +2 -3 | head -10 # IPaddr flows octets packets 128.135.137.92 9543 2477490152 2897200 224.2.177.155 160 2337752653 4446397 128.135.136.147 1258 1947979335 2123565 128.135.108.92 4775 1676599523 2105520 128.135.12.170 1391 1510492347 2570530 128.135.147.43 6765 1079172157 1396155 198.49.215.223 16 868834755 979761 128.135.221.135 1610 848575034 866508 128.135.112.72 3855 829361150 940891 66.27.181.42 43 807246316 876126

10. Is it a specific program • File sharing is high • KaZaA (port 1214) and eDonkey (4662) • http is high (no surprise) • Port 55524 only has a few flows. • Probably a few large file transfers • Flow-extract shows us that it is multicast traffic % flow-stat -f7 < ft-v06.2002-01-06.140000 | sort -nr +2 -3 | head -10 # port flows octets packets 1214 232837 16971643884 20705354 80 1696461 10397156269 21971307 4662 14292 2652388190 3526641 55524 83 2313245819 4410164 119 1503 1571612208 2510833 6346 86042 1067187821 3034293 1737 787 809319373 882134 6348 1695 799340259 1403576 1156 1592 715081006 754911 47087 10 678618965 691006

11. The Compromise • willard.uchicago.edu compromsed • We know the approximate time of the compromise: the morning of December 18th. • We want to know what else they got into and how they got in.

12. Logs, Part 1 • Look for connections to machine at right time • Compromise was via ssh • ftp’d to a home.com address • Weird connections to port 40911 % flow-extract -d willard.uchicago.edu.ft-v06 -e ' since 2001-12-18 00:00 { print }' 12/18/2001 04:59:42 -> 12/18/2001 04:59:42 6 02 willard.uchicago.edu 22 <-> 101 host230.avlogic.com 4658 2 100 00 -SR-A- 12/18/2001 04:59:42 -> 12/18/2001 04:59:42 6 11 willard.uchicago.edu 22 <-> 01 host230.avlogic.com 4658 2 100 00 -SR-A- 12/18/2001 04:59:42 -> 12/18/2001 04:59:43 6 11 willard.uchicago.edu 22 <-> 01 host230.avlogic.com 4658 6 543 10 F-RPA- 12/18/2001 04:59:42 -> 12/18/2001 04:59:43 6 02 willard.uchicago.edu 22 <-> 101 host230.avlogic.com 4658 6 543 10 F-RPA- 12/18/2001 04:59:42 -> 12/18/2001 04:59:42 6 103 host230.avlogic.com 4658 <-> 02 willard.uchicago.edu 22 5 296 00 FS-PA- 12/18/2001 04:59:42 -> 12/18/2001 04:59:43 6 103 host230.avlogic.com 4658 <-> 02 willard.uchicago.edu 22 3 120 10 --R--- 12/18/2001 04:59:42 -> 12/18/2001 04:59:42 6 11 willard.uchicago.edu 22 <-> 01 host230.avlogic.com 4157 1 60 00 -S--A- [clip] 12/18/2001 05:03:57 -> 12/18/2001 05:03:58 6 11 willard.uchicago.edu 2170 <-> 01 cc17926-a.wlgrv1.pa.home.com ftp 2 112 00 -S--A- 12/18/2001 05:03:58 -> 12/18/2001 05:03:58 6 103 cc17926-a.wlgrv1.pa.home.com ftp <-> 02 willard.uchicago.edu 2170 1 60 00 -S--A- 12/18/2001 05:03:57 -> 12/18/2001 05:03:58 6 02 willard.uchicago.edu 2170 <-> 101 cc17926-a.wlgrv1.pa.home.com ftp 2 112 00 -S--A- 12/18/2001 05:05:46 -> 12/18/2001 05:05:46 6 02 willard.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1705 1 48 00 -S--A- 12/18/2001 05:05:49 -> 12/18/2001 05:06:11 6 02 willard.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1705 48 3107 10 ---PA- 12/18/2001 05:05:46 -> 12/18/2001 05:06:12 6 103 55.icafe.euroweb.ro 1705 <-> 02 willard.uchicago.edu 40911 43 2602 00 -S-PA-

13. More forensics • What’s on port 40911 • Looks like a back door % telnet willard.uchicago.edu 40911 Trying 128.135.149.73... Connected to willard.uchicago.edu (128.135.149.73). Escape character is '^]'. SSH-1.5-1.2.27

14. Investigating 40911 • Did they connect to other machines on port 40911? (Yes, ultraviolet) • Could also scan the whole network for port 40911 % flow-cat * | flow-extract -e 'port = 40911 { print }' 12/18/2001 04:52:01 -> 12/18/2001 04:52:01 6 02 ultraviolet.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1662 1 48 00 -S--A- 12/18/2001 04:52:01 -> 12/18/2001 04:52:29 6 103 55.icafe.euroweb.ro 1662 <-> 02 ultraviolet.uchicago.edu 40911 36 2294 00 -S-PA- 12/18/2001 04:52:01 -> 12/18/2001 04:52:29 6 02 ultraviolet.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1662 37 3131 10 ---PA- 12/18/2001 05:01:50 -> 12/18/2001 05:02:10 6 02 ultraviolet.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1662 39 2512 10 ---PA- 12/18/2001 05:01:50 -> 12/18/2001 05:02:11 6 103 55.icafe.euroweb.ro 1662 <-> 02 ultraviolet.uchicago.edu 40911 48 2800 00 ---PA- 12/18/2001 05:03:03 -> 12/18/2001 05:03:05 6 103 55.icafe.euroweb.ro 1662 <-> 02 ultraviolet.uchicago.edu 40911 6 340 00 ---PA- 12/18/2001 05:03:03 -> 12/18/2001 05:03:04 6 02 ultraviolet.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1662 4 296 10 ---PA- 12/18/2001 05:05:46 -> 12/18/2001 05:05:46 6 02 willard.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1705 1 48 00 -S--A- 12/18/2001 05:05:49 -> 12/18/2001 05:06:11 6 02 willard.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1705 48 3107 10 ---PA- 12/18/2001 05:05:46 -> 12/18/2001 05:06:12 6 103 55.icafe.euroweb.ro 1705 <-> 02 willard.uchicago.edu 40911 43 2602 00 -S-PA-

15. Reading the logs through • What happened later on in the logs? • This can give us more information on what else was compromised • Connections in from avanti0.hab.de 12/17/2001 21:50:40 -> 12/17/2001 21:50:41 6 90 avanti0.hab.de 4222 <-> 02 willard.uchicago.edu 22 5 296 00 FS-PA- 12/17/2001 21:50:40 -> 12/17/2001 21:50:41 6 02 willard.uchicago.edu 22 <-> 90 avanti0.hab.de 4222 2 100 00 -SR-A- 12/17/2001 21:50:41 -> 12/17/2001 21:50:42 6 02 willard.uchicago.edu 22 <-> 90 avanti0.hab.de 4222 6 543 10 F-RPA- 12/17/2001 21:50:41 -> 12/17/2001 21:50:42 6 90 avanti0.hab.de 4222 <-> 02 willard.uchicago.edu 22 3 120 10 --R--- 12/18/2001 06:12:05 -> 12/18/2001 06:12:05 6 02 willard.uchicago.edu 22 <-> 90 avanti0.hab.de 1023 1 60 00 -S--A- 12/18/2001 06:12:05 -> 12/18/2001 06:12:05 6 90 avanti0.hab.de 1023 <-> 02 willard.uchicago.edu 22 2 112 00 -S--A- 12/18/2001 06:12:05 -> 12/18/2001 06:12:10 6 90 avanti0.hab.de 1023 <-> 02 willard.uchicago.edu 22 20 1647 10 ---PA-

16. Repeat the process • Looking at who avanti0.hab.de connected to can reveal more compromised machines • We find one more… aupc1.uchicago.edu % flow-cat * |flow-extract -e 'host = avanti0.hab.de && host != willard { print }' 12/18/2001 00:40:14 -> 12/18/2001 00:40:14 6 90 avanti0.hab.de 4284 <-> 02 aupc1.uchicago.edu 22 5 248 00 FS-PA- 12/18/2001 00:40:14 -> 12/18/2001 00:40:14 6 90 avanti0.hab.de 4284 <-> 02 aupc1.uchicago.edu 22 2 80 10 --R--- 12/18/2001 00:40:14 -> 12/18/2001 00:40:14 6 02 aupc1.uchicago.edu 22 <-> 90 avanti0.hab.de 4284 4 283 10 F--PA- 12/18/2001 00:40:14 -> 12/18/2001 00:40:14 6 02 aupc1.uchicago.edu 22 <-> 90 avanti0.hab.de 4284 1 44 00 -S--A- 12/18/2001 08:00:52 -> 12/18/2001 08:00:52 6 02 aupc1.uchicago.edu 22 <-> 90 avanti0.hab.de 1023 1 44 00 -S--A-

17. Logging Methods • We use flow logs from routers (and some switched). • Mark Fulmer’s flow-tools • <http://www.splintered.net/sw/flow-tools> • Flow-extract • <http://security.uchicago.edu/tools/net-forensics

18. Advantages Straight from router No sense of state Authoritative Disadvantages Need to have a router that supports flows where you want to log Missing useful information (e.g. sequence number) No sense of state Flow Logs

19. Logging Methods • Argus – QoSient, LLC – Carter Bullard • <http://www.qosient.com/argus> • OpenSource effort and proprietary version • Same flow model, performance and scaling • Origin/History: • Early 1990’s Work at CERT • Guerilla work until startup in 1999 • Continued analysis/experimentation at CMU • Validation, IDS, web logging (FlowScan-style)

20. Argus • Applications – audit • Edge Traffic Characterization • Security • Anonymized research data (use analysis) • Traffic accounting • Service/Policy Discovery • who/how/how much • Unexpected service delivery? • QoS validation • Internet Call records • Who talks to whom – not what’s said • Contrast to Carnivore

21. Advantages Authoritative Transaction flow aggregation Strong flow model/semantic TCP window delta/retrans ICMP aggregation Accurate timestamps TCPdump selection syntax Scalable – multiple probes Flexible – put probe anywhere Subnet/switch/host Limited access to user data Higher level tools for analysis/indexing Disadvantages Technology, no sexy apps Limited documentation Probe Architecture Vs switches, IPSEC, etc Scaling factors DoS vulnerability Argus Flow Logs

22. Argus • Quick Demo

23. Interesting Questions • Aggregate transaction analysis • Web trans frames smtp spam • Probes followed by specific connections • Application fingerprinting • Regardless of port • Network service Provision • End2End or Edge2Ether • Ask for a service, not a connection

24. Problems in identifying traffic • What if the port number jumps around • Many file sharing programs are beginning to do this to evade firewalls • If it’s used by a lot of people it will look like random traffic from a statistical view point and will just appear as noise • Application layer analysis can help • What if the traffic is encrypted? • Need lots of storage and a fast machine to keep up

25. Network Graphs • Allows quick visualization of network use • MRTG • <http://people.ee.ethz.ch/~oetiker/webtools/mrtg.html> • Cricket • <http://cricket.sourceforge.net> • FlowScan • <http://net.dois.wisc.edu/~plonka/FlowScan>

26. Traffic Management • Traditional Rate Limiting • Who to rate limit? • Just the dorms? • Everyone? • Known abusers? • How much to Rate Limit? • Can’t do application layer limiting, so it may be ineffective to programs that jump ports

27. Traffic Management • Packeteer, etc. • Can do application level • What if the traffic is encrypted? • Can’t do high bandwidth • ~100Mb/sec okay, ~1Gb/sec not • Other options…?