Automated Exploit Detection in Binaries: Finding Vulnerabilities in Software

1. Automated Exploit Detection in Binaries Finding exploitable vulnerabilities in binaries Matt Hargett http://www.clock.org/~matt matt {hizzat} use {dizznot} net Luis Miras http://dwerd.blogspot.com lmiras {hizzat} gmail {dizznot} com

2. Agenda • Definition • Architecture • Challenges

3. bugreport • Set of tests for analysis tools • Proof Of Concept tool • Not a product or real-world tool • Released under GPLv3 draft 2 • http://sf.net/projects/bugreport • Enhancements as issues come up

4. Why C#? • Very similar to Java and C++ • Open ECMA standard • 3 open source implementations • It has specific features we like • high-speed generics • nullable value types • strong typing • high quality and simple open source tools

5. Target of Detection • Many vendors have their own definitions of exploitable bugs. "depends on what you mean by exploit and by bug" • Our definition is Out-Of-Bound (OOB) memory write using tainted data.

6. Out-Of-Bound Write Tests • C code • x86 code • Test • C# code

7. bugreport architecture • Set of tests • x86 emulator • Other processors will be added later. • Analysis engine

8. Challenges • Branches • Inter-function Analysis • Non-Contiguous functions • Self Modifying code • Loops

9. Dealing with Branches • Known values • Results in one machine state • Unknown values • Results in two machine states • Constraints are used • x <= value <= y

10. Dealing with Branches • cmp, test, math instructions set flags based on input • jxx, sbb, etc. instructions act on flags

11. Dealing with Branches 1: cmp eax, 0 2: jne 4 3: ret 4: cmp eax, 255 5: jle 7 6: ret 7: ...

12. Choosing Branches • Cheat, take branches (follow jxx, sbb). • Randomly pick branches • Take all branches (drop through and follow jxx, sbb) • Take some branches (drop through and follow jxx, sbb)

13. Choosing Branches • Many functions have guards at the entry. • Guards generally drop through on failure. • Taking all branches increases code coverage

14. Dealing with Branches void main(int argc, char** argv) { if(argc < 2){ exit(-1); } printf("23c3\n"); }

15. Dealing with Branches cmp [ebp+argc], 1 jg short postGuard push 0FFFFFFFFh; status call _exit postGuard: push offset a23c3 ; ”23c3\n" call _printf

16. Choosing Branches • Prefix is a tool that randomly took branches. • Found many bugs for customers. • Produced different results each run. • Bought by Microsoft and shelved. • Many customers keep old versions around. • Prefast comes with DDK. • Does not do interfunction value tracking

17. Choosing Branches • Taking all branches results in multiple machine states. • Taking a branch sets constraints on input. • These constraints must not be broken.

18. Dealing with Branches int getSize(char *ch){ int size = 1; char x = *ch; if(x != 0){ if(x != '\n'){ size++; } else{ size += 2; } } else{ size--; } return size; }

19. Dealing with Branches What are the potential states? • (x <=-1 || x >= 1) && (x != ‘\n’) && (size == 2) • (x == ‘\n’) && (size == 3) • (x == 0) && (size == 0) Real world code will have many potential states.

20. Inter-function: Top-down • Start at an export or entry point. • Traverse code through functions

21. Inter-function: Top-down main(){foo(); x(); bar();} foo(){x(); } bar(){y(); } x(){y(); z(); } y(){z(); } z(){return 0; } // Code omitted for brevity

22. Inter-function: Top-down Function Count main() 0 foo() 0 bar() 0 x() 0 y() 0 z() 0

23. Inter-function: Top-down Function Count main() 1 foo() 0 bar() 0 x() 0 y() 0 z() 0

24. Inter-function: Top-down Function Count main() 1 foo() 1 bar() 0 x() 0 y() 0 z() 0

25. Inter-function: Top-down Function Count main() 1 foo() 1 bar() 0 x() 1 y() 0 z() 0

26. Inter-function: Top-down Function Count main() 1 foo() 1 bar() 0 x() 1 y() 1 z() 0

27. Inter-function: Top-down Function Count main() 1 foo() 1 bar() 0 x() 1 y() 1 z() 1

28. Inter-function: Top-down Function Count main() 1 foo() 1 bar() 0 x() 1 y() 1 z() 2

29. Inter-function: Top-down Function Count main() 1 foo() 1 bar() 0 x() 2 y() 1 z() 2

30. Inter-function: Top-down Function Count main() 1 foo() 1 bar() 0 x() 2 y() 2 z() 2

31. Inter-function: Top-down Function Count main() 1 foo() 1 bar() 0 x() 2 y() 2 z() 3

32. Inter-function: Top-down Function Count main() 1 foo() 1 bar() 0 x() 2 y() 2 z() 4

33. Inter-function: Top-down Function Count main() 1 foo() 1 bar() 1 x() 2 y() 2 z() 4

34. Inter-function: Top-down Function Count main() 1 foo() 1 bar() 1 x() 2 y() 3 z() 4

35. Inter-function: Top-down Function Count main() 1 foo() 1 bar() 1 x() 2 y() 3 z() 5

36. Inter-function: Top-down • Complexity can explode. • Very time consuming. • Hitting the same functions multiple times. • z() visited 5 times. • Larger programs can have very large call chains. • “like playing with a yo-yo in the grand canyon”

37. Inter-function: Bottom-up • Describe each function in isolation • Taint return value • Store return values for a function based on constraints • Use it when function call is evaluated • Creating a machine state diff.

38. Inter-function: Bottom-up • With deeply nested calls • Taint return value • Requires multiple sweeps

39. Inter-function: Bottom-up main(){foo(); x(); bar();} foo(){ x(); } bar(){ y(); } x(){y(); z(); } y(){ z(); } z(){return 0; } // Code omitted for brevity

40. Inter-function: Bottom-up Pass #1 main() { foo(); x(); bar(); } Done: <None>

41. Inter-function: Bottom-up Pass #1 foo() { x(); } Done: <None>

42. Inter-function: Bottom-up Pass #1 bar() { y(); } Done: <None>

43. Inter-function: Bottom-up Pass #1 y() { z(); } Done: <None>

44. Inter-function: Bottom-up Pass #1 x() { y(); z(); } Done: <None>

45. Inter-function: Bottom-up Pass #1 z() { return 0; } Done: z()

46. Inter-function: Bottom-up • One pass through call graph seems similar to top-down. • What is the difference? • The difference is z() is evaluated as a machine state diff. • z()’s analysis is cached

47. Inter-function: Bottom-up Pass #2 main() { foo(); x(); bar(); } Done: z()

48. Inter-function: Bottom-up Pass #2 foo() { x(); } Done: z()

49. Inter-function: Bottom-up Pass #2 bar() { y(); } Done: z()

50. Inter-function: Bottom-up Pass #2 x() { y(); z(); } Done: z()