Responding to Data Breaches

1. Responding to Data Breaches Security Professionals Conference San Antonio, TX 5 April 2011

2. Approaches to Responding to a Breach • So, you have had a data breach, now what? • What do you do to investigate if a breach occurred? • Is a virus infection a data breach? • How do you effectively engage the community? • Training • Delegated Authority • Notification • Internal direction • Public notice

3. How we got here • Due to the number of information security reports originating from computers located on campus it became necessary to: • Actively deploy sensitive data locating tools (IdentityFinder) to limit notification obligations • Document incident response procedures to work in conjunction with home grown software system • Expedite identification, investigation, remediation • and if necessary, Develop notification procedures to insure compliance with University policy as well state laws that pertain to sensitive data. 

4. History.. • Prior to deployment of many data breach statutes, we develop informal workflow to notify academic staff of virus infected computers • Identified via ePO, REN-ISAC, etc • Procedure involved first shutting off switch port • System owner lookup procedures using Remedy jack tracking and ticketing systems • Generally a telephone call from me…

5. After passage of MGL 93h • Focus turned to identify a combination of: • compromised system + sensitive data • A compromised system alone, while needing care and attention, did not run afoul of statute… • However, far too many of our systems contained sensitive data.

6. Incident Response Procedures • Contain/Quarantine suspected compromised system • Preserve an image of the local disks • Identify any sensitive data that may have been exposed • If sensitive data may be at risk: • Identify and assess malware • Complete technical report outlining the findings above • Work with legal counsel, internal audit, etc to assess risk and take appropriate actions

7. Incident Response Procedures • One important condition to make the process both efficient and effective is determining who has responsibility and/or authority over the suspected compromised system • We currently use a multi-class system that drives our incident response procedures • While this may not fit all campuses, it works for our purposes

8. Incident Response Procedures • In the course of formally documenting incident response procedures, we identified several distinct classes of networks: • Academic Managed • Academic Default • Residential Default • These are internal terms that define whether the computer is owned by the University or if it may contain University owned data • While compromises or undergraduate student/residents computers are a serious issue, they are generally out scope for our breach procedures

9. Incident Response Procedures • We developed a DNS and/or Subnet based cross-walk table identifying who to contact in the case of an incident • Due to the highly fractionalized IT support infrastructures on campus, understanding who is responsible for what is one of the most challenging tasks • Especially on a large, research campus • This required us to focus on consistent handling procedures, regardless of which group supports the departments IT needs

10. Documenting procedures • All of our current incident response procedures are posted on the OIT website • These are the public-facing parts of the process that we use for consistency, as well as for awareness and training • This material represents the work of many individuals across campus http://www.oit.umass.edu/security/incident/index.html

11. Challenges • We are delegating substantial authority to these departments to appropriately handle these procedures • However, they are already in a position of trust as they are often either the owners, stewards, or custodians of the data in question • We provide oversight to the procedures • Building trust with the campus IT community is the most critical element to keeping these procedures effective

12. Data Breach Notification Procedures • Having a consistent protocol of communication between the impacted entities is important • As a matter of University policy, we engage: • President’s office CIO • Campus CIO(s) • Legal Counsel • Law Enforcement (if necessary) • As lessons learned, we engage Internal Audit earlier in the process than prescribed by policy

13. Data Breach Notification Procedures • Engagement with University Relations/News Office has helped • Given the media awareness of data breaches, this is a key relationship to have built before the incident occurs • Data Breach Notices are legal documents, having departments craft their own may lead to more problems than benefits • There are deep political and ownership issues that arise when discussing data breaches, you need to learn the context of an organization

14. Locating Sensitive Data • This could be a whole presentation by itself…However, since we are about 40 slides in, I’ll simplify… • It is complicated…. • We site-license and use IdentityFinder in a number of functions • Proactive detection, isolation, deletion and/or remediation of all sensitive data • Reactive detection of sensitive data for breach notification timeline determination • Opportunistic stored email scanning to identify business processes that need changing

15. Lessons Learned • Our network registration procedures in academic networks are currently not effective enough • Developing an Information Security community of practice for IT administrators on campus has been invaluable • And inexpensive… • Our procedures continue to evolve.

16. Additional Lessons Learned • It has been a long time since my staff has been wanting for work. • The concern of staff overload is currently a major issue, incident response takes a lot of staff time and can be a stressful environment • Engagement with legal and audit has been key to successful procedures • Notification procedures are complicated, political, and nuanced, tread with caution…

17. Questions and Discussion…

18. Appendixes • Incident Response Definitions • Documented Response Procedures • Malware Incident Response Checklist • Network Registration • Community of Practice • Training, Awareness, Prevention • Tools we have used to positive effect…

19. Incident Response Definitions • Academic Managed domains are those academic or staff departments that are managed via their own support infrastructure.  For identification of an academic managed network see internal contacts list. • Academic Default domains are all academic networks that are not listed as Academic Managed.  If the source IP address is in an academic domain, but it is not listed in Incident Response Contacts as an Academic Managed then the network is classified as Academic Default. 

20. Academic Default • For departments without previously known IT support staff, we process these incidents through our help desk • These procedures remain a work in progress, but we have an existing workflow • Built from our historical workflows for handling DMCA complaints and compromised student computers • These procedures were born out of the bad old days of Code Red, Nimda, etc.

21. Academic Default Containment • In general, we shut the switch port off first for academic default cases • As a risk reduction step • Notification is provided to the help desk via Remedy functionality (homegrown workflow) • Help desk uses additional meta-data to identify some responsible party • Often located by seeing who is paying the bills for that jack • Results is in a call from the Help Desk

22. Academic Default • When the suspected compromised system arrives at the help desk: • Ticket is sent to our Software Support group (tier2 help desk) • Suspected system is handled by full time staff, not students • Normalized data gathering procedures to identify sensitive data • Reports sent back to campus Information Security Office

23. Academic Managed • In departments with known IT support staff, we generally notify the department before shutting off any switch ports • This has proven to be a more effective means of both trust building and risk reduction • The same generalized procedures above are followed, but some responsibilities are delegated to the departmental IT staff • While a work in progress, we have had excellent engagement from the campus and positive feedback

24. Example Contact Information • We use a combination of IP address, DNS suffix, ADS domain, and physical building location to identify hosts • This is a bit more art than science, but thankfully is no longer just in my head • Which was part of our previous processes

25. Documented Incident Response Procedures • Respond to Data Security Incidents • IT Administrators need to notify OIT if any of the following occur in their department. Note: Faculty and staff should contact their departmental IT Administrator. If your department does not have an IT Administrator, contact the OIT Help Desk immediately.

26. Documented Incident Response Procedures • Computing Devices Compromised by Malware (Most Common) • These include departmental or personal devices that may contain sensitive data. If a server is compromised, contact security@oit.umass.edu for instructions. • Conduct a preliminary analysis using the Malware Incident Response Checklist - or - contact OIT as soon as possible. • Submit an Incident Report to security@oit.umass.edu.

27. Documented Incident Response Procedures • Computing Devices Accessed without Authorization (Non-Malware) • These include University devices accessed without permission - stolen or compromised credentials, credentials lost to phishing scams, and other attempts to access a device without authorization (e.g., former employees, etc.). • Submit an Incident Report to security@oit.umass.edu.

28. Documented Incident Response Procedures • Lost or Stolen Computing Devices • These include departmental laptops, USB drives, cell phones, or other devices that may contain sensitive data, or personal computing devices with sensitive University data. • Contact the UMass Amherst Police. • Submit an Incident Report to security@oit.umass.edu.

29. Incident Response Procedures • General Incident Response Procedures • IT Administrators who suspect a data security incident in their department or who were notified of a potential incident need to complete the following steps: Note: This is a general overview of the incident response process. Depending on the complexity of the incident, additional steps may be required. • (Optional) Preliminary Analysis: If this is a malware infection, perform a preliminary analysis using the Malware Incident Response Checklist.

30. Incident Response Procedures • Incident History: Gather the incident details, including symptoms and how you first responded. • Incident Report: Contact security@oit.umass.edu if OIT first notified you of the incident, sensitive data was stored on the compromised device, or you cannot rule out the presence of sensitive data on this device. A report is required even when encryption is available on the affected device.

31. Incident Response Procedures • If the incident is confirmed: • Forensic Analysis: OIT will perform an in-depth forensic analysis of the compromised device (if the device is available). • Legal Counsel Review: The University Legal Counsel will review the incident to determine the University's legal obligations. • User Notification: The University is required to notify the individuals whose personal information may have been compromised as a result of this incident. Not all incidents will result in a notice obligation.

32. Motivation… • Consequences of Mishandling Data Security Incidents • Responding to data security incidents promptly and efficiently helps protect the University's assets (e.g., data, computers, networks) and ensures compliance with state and federal law, and University policy. • Under Chapter 93H of the Massachusetts General Law, the University is required to notify those individuals whose personal information may have been compromised as a result of a security breach.

33. Motivation… • Consequences of Mishandling Data Security Incidents • Failure to respond to a data security incident appropriately can lead to: • Regulatory fines • Legal action • Loss of funding • Loss of reputation • Most people on campus want to do the right thing

34. Malware Incident Response Checklist • Use this checklist for your preliminary analysis. Through the entire process, it is critical to: • Keep detailed notes. Depending on the severity of the incident, you may have to provide details about the incident, including how you first responded, to other staff, management, University Legal Counsel, or Internal Audit. • Minimize system changes. Keep the system intact as changes can destroy valuable data related to the incident. Do not power off, run anti-virus software, or attempt to back up data. • Contact security@oit.umass.edu if you need assistance with any of the steps below.

35. Malware Incident Response Checklist • 1. Gather volatile information while the system is running (if possible) • Document any open network connections, running processes, logged-in users, and connected drives. Capture an image of the computer’s memory. • 2. Shut the system down & preserve hard drive data • You need to shut the system down before completing the next steps.

36. Malware Incident Response Checklist • Option A: Get a forensically-sound copy of the hard drive • Get a forensically-sound 'bit-by-bit' copy of the affected hard drive(s) and keep this information until the incident is resolved. You should also preserve an MD5 hash of the original drive(s) and image(s). Note: You will need a hard drive write blocker to complete this step (see details below).

37. Malware Incident Response Checklist • Option B: Connect the hard drive to a write blocker • Alternatively, you can connect the hard drive to a hard drive write blocker before performing the next steps. Write blockers enable you to acquire information from a drive without damaging its contents. We recommend Tableau products, available from multiple online retailers.

38. Malware Incident Response Checklist • 3. Run IdentityFinder & a malware detection scan • With the write blocker in place or after you obtained a forensically-sound copy of the affected hard drive(s): • Run Identity Finder to determine whether personally identifiable information is stored on this device and where it is located. • Complete a virus/malware detection scan using your preferred anti-virus/malware application. • Gather any other information relevant to this incident.

39. Malware Incident Response Checklist • 4. Provide OIT with an Incident Report • You must contact OIT if Identity Finder finds anypersonally identifiable information, if OIT first contacted you about this incident, or if you cannot rule out the presence of sensitive data on this device. Email security@oit.umass.edu the following information: • Incident history (date, time, symptoms, how you first responded)

40. Malware Incident Response Checklist • 4. Provide OIT with an Incident Report • Identity Finder and malware scan results (if available) • Host name • IP Address • MAC Address • Building and room number • Your email address and campus telephone number

41. Malware Incident Response Checklist • Preliminary Analysis: Findings & Next Steps • If you have completed a preliminary analysis, these are some general recommendations based on the most common findings. For additional information, contact security@oit.umass.edu. • Malware and personally identifiable information foundSubmit an Incident Report (see Step 6 above). OIT will need the compromised device (or the forensically-sound copy) for an in-depth analysis.

42. Malware Incident Response Checklist • Personally identifiable information found, but no malwareContact OIT for a secondary analysis (additional detection tools may be required). Remove the data if no longer necessary or save it in a safe location (e.g., server). Review the business processes that require sensitive data to be placed in this location.

43. Malware Incident Response Checklist • Malware found, but no personally identifiable informationReview the scope of the incident to ensure other devices are not affected. Change all passwords and complete the appropriate recovery steps for this device. Submit an Incident Report if OIT originally notified you of this incident. Alternatively, email your malware scan results to security@oit.umass.edu (we'll share them with other IT Administrators).

44. Malware Incident Response Checklist • No malware, no personally identifiable informationYou may need to re-diagnose the problem: check the incident symptoms and contact OIT for assistance.

45. Malware Incident Response Checklist • OIT can help! We can provide assistance with any of the steps below. We can also provide additional information related to an incident, such as network logs or centralized system/application logs (epo, ISS, AD, DNS). • This is a critical piece of our approach, we engage with and support the departments in complying with our obligations • Issues of responsibility, discipline, etc are not in scope of this analysis

46. Network Registration • Due to inefficiencies in some of the above processes we are pushing network registration in to all academic buildings • We have had Netreg deployed for many years in residence halls and classrooms to address these same issues • Using the same network registration procedures, except that we facilitate bulk or group registrations for academic departments • Hopefully lowering the staff time commitment to find relevant contacts person(s).

47. Community of Practice • Problem: How to engage already overburdened departmental IT staff to help secure campus data • Solution: Buy them lunch… • Quick lessons learned • Buying lunch the first time helped drive attendance, money well-spent…Can probably get away with cookies. • Most IT administrators want to do the right thing, they just need some assistance in what that is… • Engaging IT staff in the development of the security program increased support and buy in… • Having many eyes and ears across campus provides far better detection of business process failure than lots of expensive technology…

48. Community of Practice • Compliance obligations frequently change, thus are responses also change. • e.g. 60-day notice obligation to HIPAA under ARRA • Regular communications with the campus community helps align practice with obligation • Given recent funding issues, most of these departmental administrators have little to no travel budget, local training events are often readily welcomed

49. Training, Awareness, and Prevention • Using our CoP as a communications vehicle, we have extended much of our training and awareness materials • Prevent Data Security Incidents • To reduce the likelihood of data security incidents and prepare to respond if an incident occurs, OIT recommends that IT Administrators: • Keep an updated inventory of sensitive data in your department. Periodically review your department's business requirements and purge the information that is no longer needed.

50. Training, Awareness, and Prevention • Use the protection tools available from OIT. OIT offers departments several enterprise solutions to help manage sensitive data and mitigate potential incidents. • Develop an incident response plan for the most common incidents. Designate staff members who can act as 'first responders' and keep their contact information readily available.