1. Stuff Steve Romig romig@net.ohio-state.edu

2. Introduction • Summary: things we’ve learned about incident response, computer crime. • Things we’ve done right • Things we’ve done wrong • Vehicle: an investigation that started 4 years ago.

3. Pre-incident • OSU didn't have much of an incident response team • Incident response was ad-hoc • Response depended on who responded • Had recently hired me part time • I started some minimal initiatives: • Tracking incidents • Logging (authentication, network traffic) • Education/awareness meetings

4. 19:00 August 27, 1996 • California ISP calls me at home: they'd been compromised • Attack came via our modem pool. • They named a suspect: someone using the nickname XXX on IRC. • I confirmed the activity • Intruder had been logged in through modem pool since 2:00 that morning. • We had several previous incidents for this intruder

5. Lessons • Publish your contact info • Log lots, log often, retain your logs • Early action can prevent later nastiness

6. 00:30 August 28, 1996 • Intruder is *still* logged in • Phone traces through Ameritech: • A promising start, sort of • Phone traces work “just like in the movies”

7. 10:00 August 28, 1996 • Intruder is *still* logged in • Phone traces through Ameritech: • I've definitely seen too many movies • It doesn't work the way I thought! • Lessons: • Publish contact info • Everyone you talk to • Carry at all times

8. August 28, 1996 • Phone traces through Ameritech: • They keep records • We can request traces after the fact • Lessons: • Work out procedures, info required with your local police, phone company.

9. August 29, 1996 • Set up tcpdump logging of intruder sessions. • We had to identify sessions through our authentication logs, start/stop tcpdump by hand. Ick. • Also raised legal issues – ECPA? • Talked to our lawyer – “no”. • This indemnifies me (to some degree) - now its the University's problem

10. Lessons: • Talk to your lawyers. • Create an incident response “team” • Not necessarily full time • Key players: legal, IT, communications, student affairs, help desk, etc. • Make a plan – who decides how/whether incidents will be handled.

11. August 30, 1996 • We enter the next level of phone trace hell: • Confusion over what sorts of court order/subpoena/search warrant we needed to request the trace. • I don’t recall how this was resolved.

12. September 3, 1996 • Got tired of starting tcpdump by hand • tacacs-action • Config file lists accounts and actions to take on login/logout. • Actions include "log" and "page" • "page" does what you'd expect • "log" invokes tcpdump on a sniffer on the correct subnet to capture their traffic on login (filtering for just their IP address), or stops tcpdump for that session on logout.

13. Lessons • Automation is a wonderful thing. • We discovered that there were several people using several accounts.

14. September 5, 1996 • I got insanely sick of getting paged all the time. Turned off the paging in the control file for tacacs-action. • We discovered that one of the local groups hangs out in #614 on IRC. • Started lurking in #614…

15. Meanwhile… • Tcpdump logs are piling up • We read through the logs with tcpdump and strings and a program called cleanup that Mark Fullmer wrote. • This is tedious, icky, and prone to errors. Its hard to read terminal escape sequences and other obfuscated traffic.

16. Review • GUI to browse list of logs, view contents of logs (by "sessions") and contents of sessions.

18. Log Listing Window • List of logs, sizes • Double click log to see summary

19. Session Summary Window • List of sessions from one log • Double click to see contents

20. Session Contents

21. Session Replay • Escape sequences are hard to read • Replay takes the server to client traffic and writes it at a controlled rate to a terminal emulator

26. September 13, 1996 • Morning ritual - check mail, download tcpdump logs, run the pre-processing stuff, get a cup of coffee, and settle down to read. • They were doing lots of IRC, email, some probing, some exploits. • They used SSH and PGP • Through telnet sessions • Sent passphrases for private keys via telnet • Sent private keys via FTP and IRC

27. Lessons: • Weakest link • When you send encrypted email, encrypt it to your public key also :-(

28. September 21, 1996 • We have been trying to ID the suspects • Maintained “players” list • Original theory: ID them and jump directly to search warrants. • Nope, it doesn’t work that way: phone trace, pen register, search warrant. Builds body of proof. • Phone traces are still up in the air...

29. Ah, breaking news… • YYY notes that XXX gets accounts by sniffing passwords in an OSU public lab and shares them with friends. • Yes, the labs were sniffable • Despite recommendations to fix this the year before

30. Lesson • Fix known security problems • Learn from past mistakes • Our labs are mostly fixed now • Now we’re deploying wireless networking…

31. October 1, 1996 • We tried to find the local 2600 meeting • 2600 magazine claimed they met at a local mall • Not as far as we could tell • XXX says that the local 2600 meeting isn't where its advertised. Aha! • Took some time before we learned true location

32. October 15, 1996 • The first of the military/government intrusions. • The issue of notification arises again. • We call the FBI and the various military CERTs.

33. The Issue of Notification • These guys ran a domain/host • They’d run probes, exploits from there • Guess who answers postmaster email? • They’d receive complaints about their activity • Rarely • They’d respond with a polite note “so sorry, we’ve been hacked…”

34. Notification • At one point they broke into host Q.com • We were all set to send q.com a warning about it • Saw email between “our” crackers and them joking about the breakin – they were friends!

35. Lesson • To notify or not, that is the question… • Don’t know who you are talking to • Don’t know whether they will follow your instructions (if you have any) • Sticky question

36. Phf Exploits • They were using the canonical "execute xterm on the remote box as root with DISPLAY set to my X server" version of the phf exploit. • Tom’s nasty xterm…

37. Review Revisited • X traffic is obscure – requests, results, events are sent in binary form. • I mangled an X debugger called xmond to replay X sessions from the tcpdump logs • Later, Justin Dolske rewrote this in Perl.

38. Browsing an X session • Server side (next 2 slides) • Key press and other events, replies, errors • What the user typed

41. Browsing an X session • Client side traffic now (next 2 slides) • Requests sent to the server • ``What the user sees'' • What is the user seeing now?

44. Replay of an X session • More obvious now: the user was running vi • Works for simple cases

50. October 23, 1996 • Many of “our” intruders made various confessions to other crimes: drugs, credit card fraud, cell phone fraud… • XXX passes out OSU accounts • Practice sessions, training, playing with new exploits. particularly XXX, WWW.