Best Known Methods in Security Events Correlation

Best Known Methods in Security Events Correlation Mohammed Fadzil Haron GSEC GCIA April 12, 2005

Agenda • Correlation overview • Knowledge requirements • Methodology • Data representation • Reaction IT@Intel

Correlation defined • A relation existing between phenomena or things or between mathematical or statistical variables which tend to vary, be associated, or occur together in a way not expected on the basis of chance alone…[1] [1] http://www.webster.com IT@Intel

Overview • Correlation is the next security big thing in importance • An important tool in the security analyst’s toolbox for monitoring security events • To be most effective, most – if not all – events should be examined • Defense in depth means more data from different technologies, vendors, and products • Huge amount of data to analyze; terabytes in size and growing • Reduce false-positive and false-negative findings compared to use of a single product/technology • Expensive manned 24x7 monitoring capabilities IT@Intel

Ultimate goal Et = Dt + Rt • Exposure time (Et): The time the resource, information, or organization is susceptible to attack or compromise. • Detection time (Dt): The time it takes for the vulnerability or the threat to be detected. • Reaction time (Rt): The time it takes for the individual, group, or organization to respond and eliminate or mediate the vulnerability or risk. “Time Based Security” by Winn Schwartau IT@Intel

Security events flow IT@Intel

Axiom on correlation • You only see the tip of the iceberg • Know the environment and perimeter of defense well • Don’t trust the tool; trust your judgment • “Automate whenever possible” [1] • Use the simplest data representation possible • Balance between over-correlated and under-correlated • Get the big picture • “The truth is in the packet” [1] [1] Toby Kohlenberg, Intel Corp. IT@Intel

Knowledge requirements • Know your environment • Know your perimeter of defense • Automate tasks • Simplify data representation IT@Intel

Know your environment Knowing the ins and outs of your network is a necessity • External network, DMZ and internal network architecture • Other networks, such as VPN and dial-up • Logistical and geographical locations of servers and users • Different operation systems, applications and functionality of servers and client machines • Network switches and routers in use • Logistical and geographical locations of critical servers (DNS, WINS, DHCP) as well as high-valued servers (web servers, servers containing intellectual properties) • You cannot know everything yourself, so know the individual experts on each piece of the network puzzle IT@Intel

Example of environment knowledge usage • Can isolate IP addresses of Internet, DMZ and internal network for different categorization • Potential detection of external attack versus inside job • VPN and dial-up services introduce other threats and need to be given separate consideration • Allows assignment of customized severity levels for different services, such as DNS and servers housing intellectual property, for upgraded security needs IT@Intel

Source of events • Host level – Syslog, HIDS/HIPS, eventlog, log files, apps logs, anti-virus signature level • Network level – NIDS/NIPS, NBAD, firewall, network routers and switch logs, active directory logs, VPN logs, third-party authentication logs • Audit – Vulnerability scanning, OS and patch level • Knowledgebase – Software vulnerabilities and exploits IT@Intel

Know your perimeter of defense • Firewall • IDS • IPS • Audit capabilities • Host level defenses • PENS • Vulnerability scanning data • And so on. IT@Intel

Know your firewalls • Location – Outer-facing, inner-facing, DMZ, internal, internal isolated network • Type – Packet filter, stateful, application firewall/proxy • What’s allowed versus denied • Capabilities versus shortcomings IT@Intel

Know your IDS/IPS • Which product deployed? NIDS, HIDS/HIPS, NIPS • Where were they deployed? What kind of traffic is being monitored? • What product/vendor deployed? • Capabilities versus shortcomings IT@Intel

Know your audit capabilities • Where are logs being kept? Syslog server or logs on host? • How long have logs being kept? Rotated? • Know your syslog servers IT@Intel

Host level defenses • Anti-virus logs • Minimum security specification compliance enforcement software logs • OS, service packs, patches-level information IT@Intel

Automate tasks as much as possible • Daunting tasks to detect intrusion due to: • Amount of data involved reaching terabyte range • Complexity of network environment architecture with Internet presence, DMZ, WAN, MAN, PAN, LAN, VOIP, VPN, Dial-up • Complexity of perimeter of defense • Large IP address ranges used internally, that is, using Class A 10.x.x.x • Multiple internally isolated networks with different type of policies, and access controls IT@Intel

What and where to automate • Data aggregation – at data source and event manager • Manual, repetitive tasks – at event manager and reaction • Data correlation – event manager • Simplify data representation – event manager console • Incident notification – event manager IT@Intel

Group your assets • Break down IP addresses into groups, such as internal, DMZ and others for Internet • Determine and group all critical servers, such as DNS, WINS, and DHCP • Determine and group all high valued servers, such as file shares, web servers, and FTP servers, and encrypted content servers for intellectual properties IT@Intel

Types of correlation • Sets • String a group of events together to generate a trigger • Sequences • String a group of events together in sequence or particular order to generate a trigger • Statistical • Deviation of normal behavior, such as mean or normal curve IT@Intel

Methods of correlation • Rule • Manually constructed, easy to create/update. Usually explicit in nature and can be applied to set, sequence and threshold types. Contains three elements: condition, time interval, and response. • Heuristic • Similar to anti-virus signature. One signature can detect multiple variations. More implicit than explicit in nature, thus potential for higher false positives/negatives. • Fuzzy Logic / Artificial Intelligence • Model approach to correlation that can dynamically adapt to changing environment. Difficult to produce and still immature; very cutting-edge. • Hybrid • No one doing them all yet. Commonly used are heuristic and rule. IT@Intel

Correlation constraint • Time • Time should be considered when creating time box correlation • Correct time is critical in correlation • Time synchronization is crucial • Context • Order of events sequence is important • Context can be necessary in correlation rules IT@Intel

Sample of correlation flow IT@Intel

Graphical representation • Seeing is believing • Pros • Can represent huge data in simple and easy to understand graphs • Cons • Not many tools (commercial/open source) with this capability • If exist, limited capabilities IT@Intel

Effective graphics should… • Show the data • Avoid distorting data • Present a large volume of data in small space • Make large data sets coherent • Show several levels of detail • Provide clear purpose of data presentation • Represent the data and not the underlying technology, methodology, and design IT@Intel

Forms of data representation • Graphs • Link graph • Charts • Data maps • Time series • Narrative graphics (space and time) • Animation • Visualization • Virtual reality IT@Intel

Scanning graph (One source to many target relationship) Mar 14 08:33:20 66.34.244.12:2827 -> xxx.yyy.1.1:18905 SYN ******S* Mar 14 08:33:20 66.34.244.12:2830 -> xxx.yyy.1.2:18905 SYN ******S* Mar 14 08:33:20 66.34.244.12:2833 -> xxx.yyy.1.3:18905 SYN ******S* Mar 14 08:33:22 66.34.244.12:2836 -> xxx.yyy.1.4:18905 SYN ******S* Mar 14 08:33:22 66.34.244.12:2839 -> xxx.yyy.1.5:18905 SYN ******S* Mar 14 08:33:22 66.34.244.12:2842 -> xxx.yyy.1.6:18905 SYN ******S* Mar 14 08:33:22 66.34.244.12:2845 -> xxx.yyy.1.7:18905 SYN ******S* Mar 14 08:33:20 66.34.244.12:2848 -> xxx.yyy.1.8:18905 SYN ******S* Harder to internalize Scan activity easily recognized IT@Intel

Link graph Stage 1 of worm propagation IT@Intel

Moving average (Simple network anomaly detection) Example: Monitoring port 445 Increase in moving average, showing an increase in activities IT@Intel

Animation movie • Inbound connection attempts to San Diego State University (SDSU) from external source (unauthorized) • Representing 332 GB of raw data, 3.4 billion raw syslog records, and 1 million events • Period of 1996-2002 (6 years) • Available at http://security.sdsc.edu/probes-animations/index.shtml IT@Intel

Animation movie IT@Intel

Reaction to correlated data • Enforcement for malware cleaning • Blocking to minimize malware propagation and attack • Investigation for malicious non-worm activities • Learning mode for improving data (reducing false-positives and false-negatives) IT@Intel

Conclusion • Correlation is a must tool for information security professionals • Time saved in detection will allow faster response time • Faster response time will minimize damages to your assets IT@Intel

Questions? IT@Intel

References • Event correlation; http://www.computerworld.com/networkingtopics/networking/management/story/0,10801,83396,00.html • “Protecting the Enterprise with Scalable Security Event Management, Part II - Intelligent Event Correlation”; Michael Mychalczuk; https://www.sans.org/webcasts/show.php?webcastid=90468 • “Thinking about Security Monitoring and Event Correlation“; http://www.securityfocus.com/infocus/1231 IT@Intel

Best Known Methods in Security Events Correlation