1 / 72

Towards High Performance Network Defense

Towards High Performance Network Defense. Zhichun Li EECS Department Northwestern University. Motivation. Attackers. Botnets. Professional attackers exploit networks for profit $$$. Worms. Network Level Defense.

ariasb
Télécharger la présentation

Towards High Performance Network Defense

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Towards High Performance Network Defense Zhichun Li EECS Department Northwestern University

  2. Motivation Attackers Botnets Professional attackers exploit networks for profit $$$ Worms

  3. Network Level Defense • Network gateways/routers are the vantage points for detecting large scale attacks • Only host based detection/prevention is not enough • Some users do not apply the host-based schemes due to the reliability, overhead, and conflicts • Many users do not update or patch their system on time • E.g., Conficker worm in the end of 2008 infected 9~15 millions of hosts • Cannot only reply on end users for security protection

  4. Challenges • Scalable to high speed networks with a large number of users • Highly accurate • Adapt fast to the emerging threats • Have good attack coverage

  5. Network-based Intrusion Detection, Prevention, and Forensics System Scalability • Framework (I) Sketch based monitoring & detection Accuracy & Scalability & Coverage Accuracy & adapt fast (III) Signature matching engines (II) Polymorphic worm signature generation Packet streams (IV) Network situational awareness Accuracy & adapt fast

  6. h1(k) … 0 1 K-1 1 … hj(k) j hH(k) … H High-speed Network Monitoringand Anomaly Detection • Online traffic monitoring and recording [SIGCOMM IMC 2004, INFOCOM 2006, ToN 2007] [INFOCOM 2008] • Reversible sketch for data streaming computation • Record millions of flows (GB traffic) in a few hundred KB • Small # of memory access per packet • Scalable to large key space size (232 or 264) • Online sketch-based flow-level anomaly detection [IEEE ICDCS 2006] [Journal of Computer Networks 2010] [IEEE CG&A, Security Visualization 2006] • Online stealthy botnet scan detection [IEEE IWQoS 2007]

  7. Network and Distributed System Diagnosis • Overlay network monitoring and diagnosis [SIGCOMM IMC 2003, SIGCOMM 2004, ToN 2007] [SIGCOMM 2006] • End-user network diagnosis [INFOCOM 2007 (2)] • Internet-scale Virtual Private Network (VPN) and backbone monitoring and diagnosis [INFOCOM 2009] • Internet-scale Data Center and dist system profiling and diagnosis [NSDI 2010]

  8. Polymorphic Worm Signature Generation Exploit invariant signature generation [IEEE Symposium on Security and Privacy 2006] (cited by ~100, code and test cases release to Columbia U., UT Austin, Purdue, Georgia Tech, UC Davis, etc) Vulnerability signature generation [IEEE ICNP 2007, ToN 2010] [NSF CyberTrust 06 Award] 1010101 10111101 11111100 00010111 Network gateway Internet Our network 8

  9. NetShield vulnerability signature based NIDS/NIPS [NSF CyberTrust 08 Award] [under submission] [patent filed] Interested by Cisco (IPS ruleset & site visit) Code release has been used by researchers in University of Toronto Using failure information to detect enterprise zombies [SecureCom09] Spamming botnet detection [NSDI09] Online Protocol Parsing and Signature Matching 9

  10. Network Situational Awareness Large-scale botnet and P2P misconfiguration event situational-aware forensics Botnet attack target/strategy inference [ASIACCS09] Root cause analysis of the P2P misconfiguration/poisoning traffic [INFOCOM10] Analysis of 2TB data across 4 years over 5 /8 IPs 10

  11. Current Work • Data center management and configuration • Internet emergency response • AS topology study [CoNEXT09] • Recovery via IXP [Infocom10] • Network based web dynamic vulnerability defense • Social network security

  12. NetShield: Matching a Large Vulnerability Signature Ruleset for High Performance Network Defense

  13. Outline Motivation High Speed Matching for Large Rulesets High Speed Parsing Evaluation Research Contributions 13 13

  14. Signature DB NetShield Overview NIDS/NIPS (Network Intrusion Detection/Prevention System) operation NIDS/NIPS Packets • Accuracy • Speed • Attack Coverage Security alerts

  15. State Of The Art Regular expression (regex) based approaches Used by: Cisco IPS, Juniper IPS, open source Bro Pros • Can efficiently match multiple sigs simultaneously, through DFA • Can describe the syntactic context Example: .*Abc.*\x90+de[^\r\n]{30}

  16. Regex Context Sensitive Context Free Cons of Regex Limited expressive power, cannot describe semantic context, thus inaccurate Theoretical prospective Protocol grammar Practical prospective • HTTP chunk encoding • DNS label pointers

  17. Vulnerability: design flaws enable the bad inputs lead the program to a bad state Good state Bad input Bad state Vulnerability Signature State Of The Art Vulnerability Signature [Wang et al. 04] Blaster Worm (WINRPC) Example: BIND: rpc_vers==5 && rpc_vers_minor==1 && packed_drep==\x10\x00\x00\x00 && context[0].abstract_syntax.uuid=UUID_RemoteActivation BIND-ACK: rpc_vers==5 && rpc_vers_minor==1 CALL: rpc_vers==5 && rpc_vers_minors==1 && packed_drep==\x10\x00\x00\x00 && opnum==0x00 && stub.RemoteActivationBody.actual_length>=40 && matchRE(stub.buffer, /^\x5c\x00\x5c\x00/) Pros • Directly describe semantic context • Very expressive, can express the vulnerability condition exactly • Accurate Cons • Slow! • Existing approaches all use sequential matching • Require protocol parsing

  18. Motivation of NetShield 18

  19. Motivation Desired Features for Signature-based NIDS/NIPS Accuracy (especially for IPS) Speed Coverage: Large ruleset Focus of this work Cannot capture vulnerability condition well! Shield [sigcomm’04] 19

  20. Research Challenges and Solutions Challenges Matching thousands of vulnerability signatures simultaneously Sequential matching match multiple sigs. simultaneously High speed protocol parsing Solutions An efficient algorithm which matches multiple sigs simultaneously A tailored parsing design for high-speed signature matching 20

  21. Background Vulnerability signature basic Use protocol semantics to express vulnerabilities Defined on a sequence of PDUs & one predicate for each PDU Example: ver==1 && method==“put” && len(buf)>300 Data representations For all the vulnerability signatures we studied, we only need numbers and strings number operators: ==, >, <, >=, <= String operators: ==, match_re(.,.), len(.). Blaster Worm (WINRPC) Example: BIND: rpc_vers==5 && rpc_vers_minor==1 && packed_drep==\x10\x00\x00\x00 && context[0].abstract_syntax.uuid=UUID_RemoteActivation BIND-ACK: rpc_vers==5 && rpc_vers_minor==1 CALL: rpc_vers==5 && rpc_vers_minors==1 && packed_drep==\x10\x00\x00\x00 && opnum==0x00 && stub.RemoteActivationBody.actual_length>=40 && matchRE(stub.buffer, /^\x5c\x00\x5c\x00/) 21

  22. Outline Motivation High Speed Matching for Large Rulesets High Speed Parsing Evaluation Research Contributions 22

  23. Matching Problem Formulation Suppose we have n signatures, defined on k matching dimensions (matchers) A matcher is a two-tuple (field, operation) or a four-tuple for the associative array elements Translate the n signatures to a n by k table This translation unlocks the potential of matching multiple signatures simultaneously Rule 4: URI.Filename=“fp40reg.dll” && len(Headers[“host”])>300

  24. Matching Problem Formulation Challenges for Single PDU matching problem (SPM) Large number of signatures n Large number of matchers k Large number of “don’t cares” Cannot reorder matchers arbitrarily -- buffering constraint Field dependency Arrays, associative arrays Mutually exclusive fields. 24

  25. Difficulty of the SPM • Bad News • A well-known computational geometric problem can be reduced to this problem. • And that problem has bad worst case bound O((log N)K-1) time or O(NK) space (worst case ruleset) • Good News • Measurement study on Snort and Cisco ruleset • The real-world rulesets are good: the matchers are selective. • With our design O(K)

  26. Matching Algorithms Candidate Selection Algorithm Pre-computation decides the rule order and matcher order Decomposition. Match each matcher separately and iteratively combine the results efficiently • Integer range checking  balanced binary search tree • String exact matching  Trie • Regex  DFA (XFA) 26

  27. Step 1: Pre-Computation Optimize the matcher order based on buffering constraint & field arrival order Rule reorder: 1 Require Matcher 1 Require Matcher 1 Require Matcher 2 Don’t care Matcher 1 Don’t care Matcher 1 & 2 n

  28. Step 2: Iterative Matching S1 A2 Don’t care matcher i+1 S3=S2 A3+B3 ={4} {4}+{}={4}+{}={4} Si require matcher i+1 In Ai+1 PDU={Method=POST, Filename=fp40reg.dll, Header: name=“host”, len(value)=450} S1={2} Candidates after match Column 1 (method==) S2= +B2 ={2} {}+{4}={}+{4}={4} R1 R2 R3 28

  29. Complexity Analysis Three HTTP traces: avg(|Si|)<0.04 Two WINRPC traces: avg(|Si|)<1.5 • Merging complexity • Need k-1 merging iterations • For each iteration • Merge complexity O(n) the worst case, since Si can have O(n) candidates in the worst case rulesets • For real-world rulesets, # of candidates is a small constant. Therefore, O(1) • For real-world rulesets: O(k) which is the optimal we can get

  30. Refinement and Extension SPM improvement Allow negative conditions Handle array cases Handle associative array cases Handle mutual exclusive cases Extend to Multiple PDU Matching (MPM) Allow checkpoints. 30

  31. Outline Motivation High Speed Matching for Large Rulesets. High Speed Parsing Evaluation Research Contribution 31

  32. High Speed Parsing General V.S. Special Purpose • Design a parsing state machine • Build an automated parsing state machine generator Keep the whole parse tree in memory Parsing and matching on the fly V.S. Parse all the nodes in the tree Only signature related fields (leaf nodes) V.S.

  33. Outline Motivation High Speed Matching for Large Rulesets. High Speed Parsing Evaluation Research Contributions 33

  34. Evaluation Methodology Fully implemented prototype • 12,000 lines of C++ and 3,000 lines of Python Release at: www.nshield.org Deployed at a university DC with up to 106Mbps 26GB+ Traces from Tsinghua Univ. (TH), Northwestern (NU) and DARPA Run on a P4 3.8Ghz single core PC w/ 4GB memory After TCP reassembly and preload the PDUs in memory For HTTP we have 794 vulnerability signatures which cover 973 Snort rules. For WINRPC we have 45 vulnerability signatures which cover 3,519 Snort rules 34

  35. Parsing Results 35

  36. Matching Results 11.0 8-core 36

  37. Scalability and Accuracy Results Rule scaling results Accuracy • Create two polymorphic WINRPC exploits which bypass the original Snort rules but detect accurately by our scheme. • For 10-minute “clean” HTTP trace, Snort reported 42 alerts, NetShield reported 0 alerts. Manually verify the 42 alerts are false positives Performance decrease gracefully

  38. Research Contribution Make vulnerability signature a practical solution for NIDS/NIPS • Multiple sig. matching candidate selection algorithm • Parsing parsing state machine Build a better Snort alternative! 38

  39. Future work Client Server Network Security Social network security Data Center Security Web/WebSecurity • WebPropeht[NSDI10] • WebShield

  40. Q & A Thanks!

  41. Observations • PDU  parse tree • Leaf nodes are numbers or strings PDU array General V.S. Special Purpose Keep the whole parse tree in memory Parsing and matching on the fly V.S. Parse all the nodes in the tree Only signature related fields (leaf nodes) V.S. 41

  42. Efficient Parsing with State Machines Studied eight protocols: HTTP, FTP, SMTP, eMule, BitTorrent, WINRPC, SNMP and DNS as well as their vulnerability signatures Common relationships among leaf nodes Pre-construct parsing state machines based on parse trees and vulnerability signatures Automated parsing state machine generator: UltraPAC 42

  43. Example for WINRPC Rectangles are states Parsing variables: R0 .. R4 0.61 instruction/byte for BIND PDU 43

  44. Experiences • Working in process • In collaboration with MSR, apply the semantic rich analysis for cloud Web service profiling. To understand why slow and how to improve. • Interdisciplinary research • Student mentoring (three undergraduates, six junior graduates)

  45. Future Work • Near term • Web security (browser security, web server security) • Data center security • High speed network intrusion prevention system with hardware support • Long term research interests • Combating professional profit-driven attackers will be a continuous arm race • Online applications (including Web 2.0 applications) become more complex and vulnerable. • Network speed keeps increasing, which demands highly scalable approaches.

  46. Research Contributions Demonstrate vulnerability signatures can be applied to NIDS/NIPS, which can significantly improve the accuracy of current NIDS/NIPS Propose the candidate selection algorithm for matching a large number of vulnerability signatures efficiently Propose parsing state machine for fast protocol parsing Implement the NetShield 46

  47. Comparing With Regex • Memory for 973 Snort rules: DFA 5.29GB (XFA 863 rules1.08MB), NetShield 2.3MB • Per flow memory: XFA 36 bytes, NetShield 20 bytes. • Throughput: XFA 756Mbps, NetShield 1.9+Gbps (*XFA [SIGCOMM08][Oakland08])

  48. Measure Snort Rules Semi-manually classify the rules. Group by CVE-ID Manually look at each vulnerability Results 86.7% of rules can be improved by protocol semantic vulnerability signatures. Most of remaining rules (9.9%) are web DHTML and scripts related which are not suitable for signature based approach. On average 4.5 Snort rules are reduced to one vulnerability signature. For binary protocol the reduction ratio is much higher than that of text based ones. For netbios.rules the ratio is 67.6. 48

  49. Matcher order Reduce Si+1 Enlarge Si+1 Merging Overhead |Si| (use hash table to calculate in Ai+1, O(1)) fixed, put the matcher later, reduce Bi+1

  50. Matcher order optimization • Worth buffering only if estmaxB(Mj)<=MaxB • For Mi in AllMatchers • Try to clear all the Mj in the buffer which estmaxB(Mj)<=MaxB • Buffer Mi if (estmaxB(Mi)>MaxB) • When len(Buf)>Buflen, remove the Mj with minimum estmaxB(Mj)

More Related