Download
tranalyzer feel the packets be the packets n.
Skip this Video
Loading SlideShow in 5 Seconds..
Stefan Burschka PowerPoint Presentation
Download Presentation
Stefan Burschka

Stefan Burschka

0 Vues Download Presentation
Télécharger la présentation

Stefan Burschka

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Tranalyzer Feel the packets, be the packets Stefan Burschka

  2. What we do: • Network Troubleshooting, Security: • TRANALYZER(T2/3): High Speed and Volume Traffic Analyzer • TRAVIZ: Graphical Toolset for Tranalyzer • Complete Tool Sets for Traffic Mining (TM), Forensics • Artificial Intelligence Research: TM & Visualisation Brain support 4 multi-dim datasets Encrypted Traffic Mining Operational Picture Malware and covert channel detection Nifty stuff 2

  3. “It's the network – go fix it!” • 3

  4. The Network is slow, The Network is insecure; NO, it's not Microsoft, shut up, It wasn't me ... Production (poor Techie) Knows, Always warned, Always his fault: FUBAR License to get fired Finance (MBA) Knows basic calculus License to Excel Manager (MBA) Always right, DoR License to Powerpoint We didn't find the problem in 4 months, can you do the job in 2 weeks? (We supply 20TB data)

  5. Troubleshooting, Security Traffic Mining: Change your perspective 5

  6. What is wrong here? 6

  7. See the disaster now? Now you have context! 7

  8. Traffic Mining(TM): Hidden Knowledge: Listen | See, Understand, Invariants, Model • Application in • Troubleshooting, Security (Classification, Encrypted TM ) • Netzwerk usage(VoiP, P2P traffic shaping, application/user profiling) • Profiling & Marketing (usage performance- & market- index) • Law enforcement and Legal Interception (Indication/Evidence) 8

  9. Basic Need: Versatile Flow Compression A B Definition: (6-Tuple) Vlan(s), srcIP, srcpPort, dstIP, dstPort, L4Protocol Or why not a bit more context and meaning ? srcWho, dstWho srcNetwork, dstNetwork Bad, Good Internal / External 9

  10. Closed source loud Tools • Netflow (Sometimes not so loud, comes with routers) • Pro: Good hands-on tool, flow statistics, header parameters, standard • Cons: Not all statistics we need, no developer support GigaStor (Horrible loud and exceptional expensive HW) Pro: heuristic expert system, Graphics, reports, whatever is in the DB Cons: What we needed is not in the DB, no developer support DPI (Elacoya, Sandvine,..) (Terrible loud and expensive HW) Pro: good protocol resolution, nice reports Cons: Its a DPI not a verstile flow engine with developer support 10

  11. Open source silent SW • Wireshark, T-Shark (packet, flow statistics) • Pro: Hands-on tool, protocol db, GUI, command line, filtering • Cons: Limited flow statistics and file size, post processing difficult Silk (flow based) Cons:Not even close to Netflow, 5 tuple, esoteric config Netmate Pro: Flow, packet based, nice features, Cons: Config , handling, 5 tuple, that is, ... University NTOP(ng) Pro: Monitoring, flow statistics, config, GUI, Graphics Cons: not really flow based as we need it, protocol encapsulation? IDS (SNORT, BRO) Pro: Alarming, regex, flexible Cons: Alarming, no Flows, BRO: memory leaks, university stuff 11

  12. Need an Allrounder, script friendly between Wireshark, Netflow and 2006: Somebody has to develop me !!

  13. Tranalyzer2(T2), C99, (Geek/Dev/Prof) • High Volume Traffic Preprocessing and Troubleshooting Open Source • Speed and Memory optimized by *.h“, config and ./autogen.sh -n • Command line based, full pcap, eth and dag cards • Post processing : HEX, ‘text \t’; Bash, AWK, Perl, … friendly • C Plugin based, Linux, Mac, (Windoof) • Subnet labeling (Who, Where, What) • BPF • Hands-on: Anomaly and security related flags • Researchers: Full Statistical and Packet Signal Analysis support • Interfaces: Matlab, GnuPlot, SPSS, Excel, oocacl, soon Netflow tools • The “-s” option: The command line AWK, Perl friendly packet mode • GUI: Traviz (http://sourceforge.net/projects/traviz) • Easy to use but, You have to know your shit • 13

  14. T3, C99, (Geek/Normalo NonDev/Prof) High Speed and Volume Troubleshooting, Security, Monitoring • Complete new Concept and Design • Full IPv4/6, more protocols as T2 • Basic Features from T2 + new nifty Plugins • Full Subnet labeling and flexible flow aggregation • Multi Threading and Interface: High performance • GUI Support via professional Tool Set: Unlimited flows and files • ipSOM: AI Tool Set to answer ANY question • Core functions into DSP and FPGA in future for the 40Gig+ • More non geek/dev user friendly but, • You still have to know your shit • 14

  15. Report T2 • /tranalyzer -r ~/wurst/data/weichwurst.dmp -w ~/wurst/results/hartwurst • ================================================================================ • Tranalyzer 0.5.8 (Anteater), beta. PID: 6123 • ================================================================================ • Active plugins: • 00: protocolStatistics, version 0.5.8 --> _protocols.txt, ports.txt • 01: basicFlowOutput, version 0.5.8 --> _flow.txt / bin subnet.txt • 02: macRecorder, version 0.5.0 --> _flow.txt / bin • 03: portBasedClassifier, version 0.5.8 --> _flow.txt / bin, portmap.txt • 04: basicLayer4CalcStatistics, version 0.5.6 --> _flow.txt / bin • 05: tcpFlags, version 0.5.8 --> _flow.txt / bin • 06: tcpStates, version 0.5.6 --> _flow.txt / bin • 07: icmpDecode, version 0.5.8 --> _flow.txt / bin, _icmpStats.txt • 08: connectionCounter, version 0.5.5 --> _flow.txt / bin • 09: descriptiveStatistics, version 0.5.6 --> _flow.txt / bin • 10: nFirstPacketsStats, version 0.5.8 --> _flow.txt / bin • 11: packetSizeInterArrivalTimeHisto, version 0.5.8 --> _flow.txt / bin • 12: standardFileSink, version 0.5.0 --> creates text output _flow.txt • 13: textFileSink, version 0.5.8 --> creates binary output _flow.bin • Start processing file: /home/wurst//data/weichwurst.dmp • BPF: (null) • Dump start: 1351794649.186547 sec : Wed 01 Nov 2012 18:30:49.186547 • Shutting down Tranalyzer 0.5.8... • Dump stop: 1351837376.118852 sec : Thu 02 Nov 2012 06:22:42.118852 • Total dump duration: 42712.932305 sec • Number of processed packets: 6497970 • Number of processed traffic bytes: 1749617780 • Number of ARP packets: 1603 • Number of RARP packets: 5 • Number of IPv4 fragmented packets: 299 • Number of IPv6 packets: 0 • Number of IPv4 flows: 3395325 • Average snapped Bandwidth: 327.634 KBit/s • Average full IP Bandwidth: 326.386 Kbit/s • Warning: IPv4 Fragmentation header packet missing • 15

  16. T2 Protocol File • Total packets captured: 42278 • L4 Protocol # Packets Relative Frequency[%] Protocol description • 1 21 0.049671 Internet Control Message Protocol • 2 6 0.014192 Internet Group Management Protocol • 6 41698 98.628128 Transmission Control Protocol • 17 250 0.591324 User Datagram Protocol • 103 28 0.066228 Protocol Independent Multicast • Total TCP packets: 41698 • Port # Packets Relative Frequency[%] • 80 41519 99.570723 World Wide Web HTTP • 445 8 0.019186 Win2k+ Server Message Block • 5557 147 0.352535 • Total UDP packets: 250 • Port # Packets Relative Frequency[%] • 53 2 0.800000 Domain Name Server • 137 50 20.000000 NETBIOS, [trojan] Msinit • 138 21 8.400000 NETBIOS Datagram Service • 1900 18 7.200000 SSDP • 1908 2 0.800000 Dawn • 1985 156 62.400000 Hot Standby Router Protocol • 16

  17. T2 ICMP Stats File • Total # of ICMP messages: 22258 • ICMP / Total traffic percentage[%]: 0.343 • Echo reply / request ratio: 0.892 • Type Code # of Messages Relative Frequency [%] • ICMP_ECHOREQUEST - 111 0.499 • ICMP_ECHOREPLY - 99 0.445 • ICMP_SOURCE_QUENCH - 15 0.067 • ICMP_TRACEROUTE - 0 0.000 • ICMP_DEST_UNREACH ICMP_NET_UNREACH 60 0.270 • ICMP_DEST_UNREACH ICMP_HOST_UNREACH 15674 70.420 • ICMP_DEST_UNREACH ICMP_PROT_UNREACH 0 0.000 • ICMP_DEST_UNREACH ICMP_PORT_UNREACH 3100 13.928 • ICMP_DEST_UNREACH ICMP_FRAG_NEEDED 0 0.000 • ICMP_DEST_UNREACH ICMP_SR_FAILED 0 0.000 • ICMP_DEST_UNREACH ICMP_NET_UNKNOWN 0 0.000 • ICMP_DEST_UNREACH ICMP_HOST_UNKNOWN 0 0.000 • ICMP_DEST_UNREACH ICMP_HOST_ISOLATED 0 0.000 • ICMP_DEST_UNREACH ICMP_NET_ANO 8 0.036 • ICMP_DEST_UNREACH ICMP_HOST_ANO 600 2.696 • ICMP_DEST_UNREACH ICMP_NET_UNR_TOS 0 0.000 • ICMP_DEST_UNREACH ICMP_HOST_UNR_TOS 0 0.000 • ICMP_DEST_UNREACH ICMP_PKT_FILTERED 776 3.486 • ICMP_DEST_UNREACH ICMP_PREC_VIOLATION 0 0.000 • ICMP_DEST_UNREACH ICMP_PREC_CUTOFF 0 0.000 • ICMP_REDIRECT ICMP_REDIR_NET 1125 5.054 • ICMP_REDIRECT ICMP_REDIR_HOST 589 2.646 • ICMP_REDIRECT ICMP_REDIR_NETTOS 0 0.000 • ICMP_REDIRECT ICMP_REDIR_HOSTTOS 0 0.000 • ICMP_TIME_EXCEEDED ICMP_EXC_TTL 95 0.427 • ICMP_TIME_EXCEEDED ICMP_EXC_FRAGTIME 0 0.000 • ICMP_TRACEROUTE - 0 0.000 • 17

  18. T2 Flow Header File: Hands-On 20 ..... 21 8:NR Minimum layer3 packet size 22 8:NR Maximum layer3 packet size 23 19:NR Average packet load ratio 24 19:NR Send packets per second 25 19:NR Send bytes per second 26 19:NR Packet stream asymmetry 27 19:NR Byte stream asymmetry 28 8:NR IP Minimum delta IP ID 29 8:NR IP Maximum delta IP ID 30 7:NR IP Minimum TTL 31 7:NR IP Maximum TTL 32 7:NR IP TTL Change count 33 13:NR IP Type of Service 34 14:NR IP aggregated flags 35 8:NR IP options count 36 13,15:NR IP aggregated options • 18

  19. T2 Flow Header View: Hands-On 37 8:NR TCP packet seq count 38 10:NR TCP sent seq diff bytes 39 8:NR TCP sequence number fault count 40 8:NR TCP packet ack count 41 10:NR TCP flawless ack received bytes 42 8:NR TCP ack number fault count 43 8:NR TCP initial window size 44 19:NR TCP average window size 45 8:NR TCP minimum window size 46 8:NR TCP maximum window size 47 8:NR TCP window size change down count 48 8:NR TCP window size change up count 49 8:NR TCP window size direction change count 50 13:NR TCP aggregated protocol flags (cwr, ecn, urgent, ack, push, reset, syn, fin) 51 14:NR TCP aggregated header anomaly flags 52 8:NR TCP options Packet count 53 8:NR TCP options count 54 15:NR TCP aggregated options 55 8:NR TCP Maximum Segment Length 56 7:NR TCP Window Scale 57 19:NR TCP Trip Time Syn, Syn-Ack | Syn-Ack, Ack 58 19:NR TCP Round Trip Time Syn, Syn-Ack, Ack | TCP Ack-Ack RTT 59 19:NR TCP Ack Trip Min 60 19:NR TCP Ack Trip Max 61 19:NR TCP Ack Trip Average 62 13:NR TCP aggregated protocol state flags 63 15,14:NR ICMP Aggregated type & code bit field 64 19:NR ICMP Echo reply/request success ratio 65 9:NR Number of connections from source IP to different hosts 66 9:NR Number of connections from destination IP to different hosts 67 9:NR Number of connections between source IP and destination IP Yes I know, I should do something special for the TimeStamp option • 19

  20. T2 Flow Header View: TM geeks 68 19:NR Minimum packet length 69 19:NR Maximum packet length 70 19:NR Mean packet length 71 19:NR Lower quartile of packet lengths 72 19:NR Median of packet lengths 73 19:NR Upper quartile of packet lengths 74 19:NR Inter quartile distance of packet lengths 75 19:NR Mode of packet lengths 76 19:NR Range of packet lengths 77 19:NR Standard deviation of packet lengths 78 19:NR Robust standard deviation of packet lengths 79 19:NR Skewness of packet lengths 80 19:NR Excess of packet lengths 81 19:NR Minimum inter arrival time 82 19:NR Maximum inter arrival time 83 19:NR Mean inter arrival time 84 19:NR Lower quartile of inter arrival times 85 19:NR Median inter arrival times 86 19:NR Upper quartile of inter arrival times 87 19:NR Inter quartile distance of inter arrival times 88 19:NR Mode of inter arrival times 89 19:NR Range of inter arrival times 90 19:NR Standard deviation of inter arrival times 91 19:NR Robust standard deviation of inter arrival times 92 19:NR Skewness of inter arrival times 93 19:NR Excess of inter arrival times 94 8,25:R L2L3/L4/Payload( s. PACKETLENGTH in packetCapture.h) length and inter-arrival times for the N first packets 95 8,9,9,9,9:R Packetsize Inter Arrival Time histogram bins All you never wanted to know about statistics in a flow L2/3/4/7 configurable Packet Statistics • 20

  21. HOW TO find the needle in the flow stack? Have a break have a HEX & ¦ scripting!

  22. T2 Text Flow File: Basic plugins • A 1196278772.439355 1196279184.642073 412.202718 0x9B42 22 192.168.1.10 0x00000001 2119 68.3.4.5 0x800806034 80 6 00:0f:1f:cf:7c:45_00:00:0c:07:ac:0a_6387 http 6387 8272 464 5437587 0 4 15.494803 1.125660 -0.128590 -0.999829 1 87 128 128 0x00 0x42 0x0000 116 464 6231 4116 5437724 2253 63754 64831.988281 62501 65535 3342 2904 5713 0x18 0xF900 0x0000 0x03 0x00000000 0x0000 -1.0 1 1 1 ... B 1196278772.409312 1196279184.642073 412.232761 0x9B43 22 192.168.1.10 0x00000001 80 68.3.4.5 0x80080634 2119 6 00:d0:00:64:d0:00_00:0f:1f:cf:7c:45_8272 http 8272 6387 5437587 464 0 1380 20.066333 13190.574633 0.128590 0.999829 1 3 63 63 0x00 0x42 0x0000 8146 5440245 109 116 464 8104 5840 5840.000000 65535 0 0 0 0 0x18 0x1B00 0x0000 0x03 0x00000000 0x0000 -1.0 1 1 1 ... • 22

  23. T2 Binary Coding Status: • 2^0 0x0001 Flow Warning Flag: If A flow: Invert Flow, NOT client flow • 2^1 0x0002 Dump/flow: L3 Snaplength too short • 2^2 0x0004 Dump/flow: L2 header length too short • 2^3 0x0008 Dump/flow: L3 header length too short • 2^4 0x0010 Dump: Warning: IP Fragmentation Detected • 2^5 0x0020 Flow: ERROR: Severe Fragmentation Error • 2^6 0x0040 Flow: ERROR: Fragmentation Header Sequence Error • 2^7 0x0080 Flow ERROR: Fragmentation Pending at end of flow • 2^8 0x0100 Flow/Dump: Warning: VLAN(s) detected • 2^9 0x0200 Flow/Dump: Warning: MPLS unicast detected • 2^10 0x0400 Flow/Dump: Warning: MPLS multicast detected • 2^11 0x0800 Flow/Dump: Warning: L2TP detected • 2^12 0x1000 Flow/Dump: Warning: PPP detected • 2^13 0x2000 Flow/Dump: 0/1: IPv4/IPv6 detected • 2^14 0x4000 Flow/Dump: Warning: Land Attack detected • 2^15 0x8000 Flow/Dump: Warning: Time Jump So what is: 0x9B43 • 23

  24. T2 Flow Binary Coding: ipFlags • 2^0 0x0001 IP Options present, s. IP Options Type Bit field • 2^1 0x0002 IPID out of order • 2^2 0x0004 IPID rollover • 2^3 0x0008 Fragmentation: Below expected RFC minimum fragment size: 576 • 2^4 0x0010 Fragmentation: Fragments out of range (Possible tear drop attack) • 2^5 0x0020 Fragmentation: MF Flag • 2^6 0x0040 Fragmentation: DF Flag • 2^7 0x0080 Fragmentation: x Reserved flag bit from IP Header • 2^8 0x0100 Fragmentation: Unexpected position of fragment (distance) • 2^9 0x0200 Fragmentation: Unexpected sequence of fragment • 2^10 0x0400 L3 Checksum Error • 2^11 0x0800 L4 Checksum Error • 2^12 0x1000 SnapLength Warning: IP Packet truncated, L4 Checksums invalid • 2^13 0x2000 Packet Interdistance == 0 • 2^14 0x4000 Packet Interdistance < 0 • 2^15 0x8000 Internal State Bit for Interdistance assessment So what is: 0x1C21 • 24

  25. T2 Flow Binary Coding: tcpFlags 2^0 0x01 FIN No more data, finish connection 2^1 0x02 SYN Synchronize sequence numbers 2^2 0x04 RST Reset connection 2^3 0x08 PSH Push data 2^4 0x10 ACK Acknowledgement field value valid 2^5 0x20 URG Urgent pointer valid 2^6 0x40 ECE ECN-Echo 2^7 0x80 CWR Congestion Window Reduced flag is set 2^0 0x0001 Fin-Ack Flag 2^1 0x0002 Syn-Ack Flag 2^2 0x0004 Rst-Ack Flag 2^3 0x0008 Syn-Fin Flag, Scan or malicious packet 2^4 0x0010 Syn-Fin-Rst Flag, potential malicious scan packet or malicious channel 2^4 0x0020 Fin-Rst Flag, abnormal flow termination 2^5 0x0040 Null Flag, potential NULL scan packet, or malicious channel 2^6 0x0080 XMas Flag, potential Xmas scan packet, or malicious channel 2^8 0x0100 Due to packet loss, Sequence Number Retry, retransmit 2^9 0x0200 Sequence Number out of order 2^10 0x0400 Sequence mess in flow order due to pcappkt loss 2^11 0x0800 Warning: L4 Option field corrupt or not acquired 2^12 0x1000 Syn retransmission 2^13 0x2000 Ack number out of order 2^14 0x4000 Ack Packet loss, probably on the sniffing interface 2^15 0x8000 Internal state: TCP Window Size Machine So what is: 0x1B 0xC403 • 25

  26. T2 Flow Binary Coding: icmpFlags • Aggregated ICMP Type & Code bit Field • So what is: 0x00000100_0x0001 • 26

  27. T2 Packet Signal: Encrypted VoIP Mining PacketLength_Packet-Interdistance; … 1023_0.000000;758_0.030043;1380_0.110201;80_0.00000;369_0.000010;230_0.020029;1380_0.070101;80_0.000000;50_0.060086;1380_0.070101;80_0.090130; … Packet Length Post processing scripts: /tranalyzer/trunk/scripts time 27

  28. T2 Statistical Application / User profiling Packet length-Interdistance Statistics: Fingerprint PktLen_Packet-IAT_cnt_cntPktLen_cntIAT; … • 0_0_2322_6271_2396;0_2_82_6271_90;0_4_114_6271_114;0_6_138_6271_140;0_8_162_6271_164;0_10_157_6271_160;0_12_220_6271_224;0_14_217_6271_222;0_16_325_6271_325;0_18_373_6271_376;0_20_493_6271_498;0_22_340_6271_343;0_24_238_6271_238;0_26_283_6271_284;0_28_143_6271_143;0_30_114_6271_114;0_32_139_6271_140;0_34_175_6271_176;0_36_72_6271_73;0_38_25_6271_25;0_40_20_6271_20;0_41_12_6271_13;0_42_8_6271_8;0_43_6_6271_6;0_44_6_6271_6;0_45_4_6271_4;0_46_5_6271_5;0_47_9_6271_10;0_48_9_6271_9;0_49_6_6271_6;0_50_4_6271_4;0_51_4_6271_4;0_52_5_6271_5;0_53_3_6271_3;0_54_9_6271_9;0_55_7_6271_8;0_56_1_6271_1;0_57_4_6271_4;0_58_1_6271_1;0_59_3_6271_3;0_60_4_6271_4;0_61_4_6271_4;0_62_2_6271_2;0_63_1_6271_1;0_64_1_6271_1;0_65_1_6271_1;4_0_74_116_2396;4_2_8_116_90;4_6_2_116_140;4_8_2_116_164;4_10_3_116_160;4_12_4_116_224;4_14_5_116_222;4_18_3_116_376;4_20_5_116_498;4_22_3_116_343;4_26_1_116_284;4_32_1_116_140;4_34_1_116_176;4_36_1_116_73;4_41_1_116_13;4_47_1_116_10;4_55_1_116_8 ….. Post processing scripts: /tranalyzer/trunk/scripts Skype: Vulnerable against TM Attack • 28

  29. Some T3 Plugins • L7 Protocols: Mail, HTTP, etc • Routing: OSPF • DNS / DHCP • Full PCRE Regex • Signal Processing • Artificial Intelligence (RNN, Bayes, ESOM), nifty entropy shit • Connection Matrix, Centrality • IP Statistics: Host • Database • 29

  30. So what? Some Examples

  31. The one way TCP Flow problem • Symptom: on and off access problems • TCP flows established, unidirectional • T2 proofed: Reverse connection exists, not through firewall • Not communicated online mis-configuration of firewall Trampel OSPF

  32. FFT of some Packet Signals • Packet Length • time • 32

  33. Traffic Mining: Encrypted Content Guessing • SSH Command Guessing • IP Tunnel Content Profiling • Pitch based Classification • Encrypted Voip Guessing: CCC 2011 33

  34. TM Your OWN: Packet Length Signal See the features? Codec training SN Ping min l =3 Burschka (Fischkopp) Linux Dominic (Student) Windows

  35. Connection plugin: Social Behaviour • 35

  36. What is the Unknown? • 36

  37. HOW TO find Bad Guys? Day: 0.7% of all users 42% bandwidth, WTF? P2P Traffic P2P Traffic Average Users ??? Percentil User Percentil User Normal Traffic Normal Traffic 37

  38. HOW TO find Bad Guys? Night: Same guys @ night 3am, ... P2P Traffic Average Users Machines of WAREZ guys Percentil User Normal Traffic 38

  39. Layer3/4/whatever Visualization Graphviz --> Operational Picture in Bootcamp _flow.txt Your AWK script Graphviz: dotty • 39

  40. Layer3/4 Visualization Graphviz --> simple forensic Picture

  41. Network Classification Centrality Connection Matrix PCA Largest Eigenvector Plot / t • 41

  42. Network / Host Classification Centrality

  43. ipSOM Operational Picture: 13 Dim statistical T2 Flow parameters Now conceivable by human brain Bot Scanner DNS Zone Transfer 43

  44. Questions / Comments RFM and try me Join the development force Who wants Bootcamp? http://sourceforge.net/projects/tranalyzer/ http://tranalyzer.com http://sourceforge.net/projects/traviz Google: Dataming for Hackers stefan.burschka@ruag.com 44