COEN 252 Computer Forensics

1. COEN 252 Computer Forensics Investigating Hacker Tools

2. Program Analysis • Given an executable, how do we find out what it does? • Try to find the program online. • Analyze source code to find clues. • Search for the name of the program. • Perform source code review. • Execute the program in a sandbox. • Some programs can break out of a sandbox / jail.

3. Program Compilation • Compiler • Translates HLL code to Assembly / ILL • Assembler • Translates Assembly code to machine language • Linker • Creates object code out of several modules. • A program usually makes library calls (stdio)

4. Program Compilation • Statically Linked: All library code is part of the object code • Dynamically Linked: Program calls library functions. (DLL) • Stripping: Removes all human-readable symbols from object code. • Combats reverse engineering. • Packing with UPX, etc. • upx.sourceforge.net • Compresses source code (achieves ratios of 20% - 40%)

5. Program Compilation • Static compilation needs more memory

6. Program Analysis • Static Analysis: • Determine the type of executable. • ELF file in Unix • Exe-type in Windows • Symbol Extraction: • Use a program like strings to find symbols left in object code. • Names give hints on program. • Will not work for stripped files.

7. Static Program Analysis • Example for strings output:

8. Program Analysis • Find the program online: • Use the name of the file to find online versions. • Use strings to check whether this is a similar file. • Use same compiler to compile the online version and check for similarity.

9. Static Program Analysis • Investigate source code • Use Reversing Tools: • Disassembler: • Decodes binary machine code into a readable assembly language text • IDA-Pro • ILDasm (Microsoft .Net IL disassembler)

10. Static Program Analysis • Investigate source code • Use Reversing Tools: • Debuggers • Kernel-mode: • Component that sits alongside the system’s kernel • Allows for stopping and observing the entire system. • User-mode: • Attach to a process. • Take full control of process. • Tools: • OllyDbg • WinDbg (MS tool) • IDA-Pro • Numega-SoftIce (no longer available in isolation)

11. Static Program Analysis • Investigate source code • Use Reversing Tools: • Decompilers • Attempt to produce a high-level language source-code-like representation from a binary. • Never completely possible because • The compiler removes some information, • The compiler optimizes the code. • System Monitoring Tools • Filemon • TCPView • RegMon • PortMon • WinObj • Process Explorer

12. Static Program Analysis • Investigate source code • Executable-Dumping • Dumpbin (MS) • PEView • PEBrowse Professional

13. Program Analysis • Using disassembly:

14. Program Analysis

15. Static Program Analysis • Artifacts to look for: • Names of functions • Especially API functions. • Data strings • Names of constant strings • Names of directories • Identification of compiler

16. Program Analysis

17. Static Program Analysis • Compilers generate different types of code for the same HLL feature • Function Calls: • Order in which parameters are pushed on stack. • Use of certain registers to pass variables. • Use of stack / registers to return a value. • Division of labor between callee and caller. • This allows us to recognize the compiler with which an executable was created. • Programmers using assembly will not follow the same standards throughout the code. • Hence, we can recognize assembly writers as well.

18. Dynamic Program Analysis • Run the program and see what it is doing. • Requires security mechanisms: • Dedicated machine. • Not connected to the internet. • Or: Virtual machine. • However: Code can recognize whether it is running in VMWare. • E.g. by the internal MAC addresses, … • Transport malware on a non-writable CD / DVD

19. Dynamic Program Analysis • Strace, systrace: • Run the programming, but keep track of the system calls that it makes with parameters. • More relevant calls (Unix): • open • read • write • Unlink • lstat • socket • close • Strace has an option that intercepts all network related calls.

20. Dynamic Program Analysis • Use fport, netstat, … to determine ports opened by the program. • On Windows systems. • Use regmon • Use ListDlls • Use psList • to find out processes created by program.

21. Dynamic Program Analysis • Intercept communication of program. • Need to generate a fake network. • E.g.: Static analysis reveals that the program tries to contact www.evil.org on the IRC port. • Hence, name an additional machine on separated net www.evil.org.

22. Dynamic Program Analysis • Run program on a debugger. • IDA-Pro • OllyDbg • SoftIce

23. Dynamic Program Analysis • Do a web-search for unique names.

24. Program Analysis • Malware writers can use antireversing techniques. • Eliminate symbolic information. • Encrypt code. • Code obfuscation. • Make HLL constructs difficult to understand. • Antidebugger Methods: • Use the IsDebuggerPresent API to protect against user-level debuggers. • Use the NTQuerySystemInformation API to determine if a kernel debugger is attached to the system. • Set a trap flag and check whether it is still there. • A debugger would “swallow” it. • Put in bogus bytes over which the code jumps. • Does not work for all disassemblers.