1 / 31

Polymorphic Malware Detection

Polymorphic Malware Detection . Connor Schnaith, Taiyo Sogawa 9 April 2012. Motivation. “ 5 000 new malware samples per day” --David Perry of Trend Micro Large variance between attacks Polymorphic attacks Perform the same function Altered immediate values or addressing

asis
Télécharger la présentation

Polymorphic Malware Detection

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Polymorphic Malware Detection Connor Schnaith, Taiyo Sogawa 9 April 2012

  2. Motivation • “5000 new malware samples per day” • --David Perry of Trend Micro • Large variance between attacks • Polymorphic attacks • Perform the same function • Altered immediate values or addressing • Added extraneous instructions • Current detection methods insufficient • Signature-based matching not accurate • Behavioral-based detection requires human analysis and engineering

  3. Malware Families •Classified into related clusters (families) •Tracking of development •Correlating information •Identifying new variants •Based on similarity of code •Koobface •Bredolab •PoisonIvy •Conficker (7 mil. Infected) Source: Carrera, Ero, and Peter Silberman. "State of Malware: Family Ties." Media.blackhat.com. 2010. Web. 7 Apr. 2012. <https://media.blackhat.com/bh-eu-10/presentations/Carrera_Silberman/BlackHat-EU-2010-Carrera-Silberman-State-of-Malware-slides.pdf>.

  4. ~300 samples of malware with 60% similarity threshold

  5. Current Research • Techniques for identifying malicious behavior • Mining and clustering • Building behavior trees • Industry • ThreatFire and Sana Security developing behavioral-based malware detection

  6. Design challenges • Discerning malicious portions of code • Dynamic program slicing • accounting for control flow dependencies • Reliable automation • Must be able to be reliable w/o human intervention • Minimal false positives

  7. Holmes: Main Ideas • Two major tasks • Mining significant behaviors from a set of samples • Synthesizing an optimally discriminative specification from multiple sets of samples • Key distinction in approach • "positive" set - malicious • "negative" set - benign • Malware: fully described in the positive set, while not fully described in the negative set

  8. Main Ideas: behavior mining • Extracts portions of the dependence graphs of programs from the positive set that correspond to behaviors that are significant to the programs’ intent. • The algorithm determines what behaviors are significant (next slide) • Can be thought of as contrasting the graphs of positive programs against the graphs of negative programs, and extracting the subgraphs that provide the best contrast.

  9. Main ideas: behavior mining • A "behavior" is a data dependence graph • G = (V, E, a, B) • V is the set of vertices that correspond to operations (system calls) • E is the edges of the graph and correspond to dependencies between operations • a is the labeling function that associates nodes with the operations they represent • B is the labeling function that associates the edges with the logic that represents the dependencies

  10. Main ideas: behavior mining • A program P exhibits a behavior G if it can produce an execution trace T with the following properties • Every operation in the behavior corresponds to an operation invocation and its arguments satisfy certain logical constraints • the logic formula on edges connecting behavior operations is satisfied by a corresponding pair of operation invocations in the trace • Must capture information flow in dependence graphs • two key characteristics • the path taken by the data in the program • security labels assigned to the data source and the data sink

  11. Main ideas: behavior mining • Information gain is used to determine if a behavior is significant. A behavior that is not significant is ignored when constructing the dependency graph • Information gain is defined in terms of Shannon entropy and it means gaining additional information to increase the accuracy of determining if a G is in G+ or G- • Shannon entropy • H(G+ U G-) corresponds to the uncertainty that a graph G belongs to G+ or G- • partition G+ and G- into smaller subsets to decrease that uncertainty • process called subgraph isomorphism

  12. Main ideas: behavior mining • A significant behavior g is a subgraph of a dependence graph in in G+ such that: Gain(G+ U G- , g) is maximized • Information gain is used as the quality measure to guide the behavior mining process • Some non-significant actions can get passed as significant • these actions may or may not throw off the algorithm that determines if the program is malicious

  13. Main ideas: behavior mining • Significant behaviors mined from malware Ldpinch • Leaking bugfix information over the network • Adding a new entry to the system autostart list • Bypassing firewall to allow for malicious traffic • Could say any program that exhibits all three of these behaviors should be flagged malicious • This is too specific of a statement • Doesn't account for variations within a family • It is known that smaller subsets of behaviors that only include one of these actions could still be malicious • Need discriminative specifications

  14. Main ideas: discriminative specifications • Creates clusters of behaviors that can be classified into as characteristic subset • Program matches specification if it matches all of the behaviors in a subset • "Discriminative" in that it matches the malicious but not the benign programs

  15. Main ideas: discriminative specifications • Each set of subset of behaviors induces a cluster of samples • Malicious and benign samples are mined are organized into these clusters • Goal: find an optimal clustering technique to organize the malicious into the positive subset and the benign into negative subset

  16. Main ideas: discriminative specifications • Three part algorithm • Formal concept analysis • Simulated annealing • Constructing optimal specifications • Formal concept analysis • O is a cluster of samples • A is the set of mined behaviors in O • A concept is the pair (A, O) Set of concepts: {c1, c2, c3 , ... , cN) Behavior specification: S(c1, c2, c3, ... , cN)

  17. Main ideas: discriminative specifications Formal Concept Analysis (continued) • Begins by constructing all concepts and computes pairwise intersection of the intent sets of these concepts • Repeated until a fixpoint is reached and no new concepts can be constructed • When algorithm terminates, left with an explicit listing of all of the sample clusters that can be specified in terms of one or more mined behaviors • Goal is to find {c1, c2, c3, ... , cN} such that S(c1, c2, c3, ... , cN) is optimal (based on threshold)

  18. Main ideas: discriminative specifications Simulated annealing • Probabilistic technique for finding approximate solution to global optimization problem • At each step, a candidate solution i is examined and one of its neighbors j is selected for comparison • The algorithm moves to j with some probability • A cooling parameter T is reduced throughout process and when it gets to a minimum the process stops

  19. Main ideas: discriminative specifications Constructing Optimal Specifications • Threshold t, a set containing positive and negative samples, and a set of behaviors mined with the previous process • Called SpecSynth • Constructs full set of concepts • Removes redundant concepts • Run simulated annealing until convergence, then return the best solution

  20. Holmes: Mining an Clustering

  21. Evaluation and Results: Holmes • Used six malware families to develop specifications • Tested final product against 19 malware families • Collected 912 malware samples and 49 benign

  22. Holmes Continued • Experiments carried over varying threshold values (t) • Demonstrates high sensitivity to system accuracy • Perhaps only efficient for a specific subset of malware

  23. Holmes Scalability • Worst-case complexity is exponential • Behaviors of repeated executions (Stration and Delf) took 12-48 hours to analyze • Scalability for Holmes is a nightmare! “scary and scaled”

  24. USENIX • The Advanced Computing Systems Association • (Unix Users Group) • 2009 article: automatic behavior matching • Behavior graphs (slices) • Tracking data and control dependencies • Matching functions • Performance evaluations Source: Kolbitsch, Clemens. "Effective and Efficient Malware Detection at the End Host." Usenix Security Symposium (2009). Web. 8 Apr. 2012. <http://www.iseclab.org/papers/usenix_sec09_slicing.pdf>.

  25. USENIX: Producing Behavior Graphs • Instruction log • Trace instruction dependencies • Slicing doesn't reflect stack manipulation • Memory log • Access memory locations Partial behavior graph of Netsky (Kolbitsch et al)

  26. USENIX: Behavior Slices to Functions • Use instruction and memory log to determine input arguments • Identify repeated instructions as loops • Include memory read functions • We can now compare to known malware

  27. Evaluation Six families used for development (mostly mass-mailing worm) Expanded test set

  28. Performance Evaluation • Installed Internet Explorer, Firefox, Thunderbird, Putty, and Notepad on Windows XP test machine • Single-core, 1.8 GHz, 1GB RAM, Pentium 4 processor

  29. USENIX Limitations • Evading system emulator • USENIX detector uses Qemu emulator • delays • time-triggered behavior • command and control mechanisms • Modifying algorithms behavior • A more fundamental change, but cannot be detected using same signatures • End-host based system • Cannot track network activity

  30. Questions/Discussion

More Related