An Attack Simulator for Systematically Testing Program-Based Security Mechanisms

1. An Attack Simulator for Systematically Testing Program-Based Security Mechanisms Ben Breech Lori Pollock Mike Tegtmeyer University of Delaware Army Research Lab

2. Insecure Programming Practices • Lead to malicious and costly exploits • Zotob worm, 2005 ($10,000 - $100,000 / company) • Sasser worm, 2004 (kept ~40 Delta planes on ground, shutdown part of UK Coastguard) • Examples: • gets() • strcpy() • strncpy() (with wrong n)

3. Example Attack: Stack Smashing Runtime Stack Caller int f (parameters) { local_vars char buf2 [5]; gets (buf2) // vulnerability …. } 0x1200 return address f local vars buf2

4. Possible Solutions • Fix vulnerability • Can be expensive to locate • Rewrite entire code with more secure language/libraries • Really expensive • Use security mechanism to prevent exploit

5. Security Mechanisms • Any procedure in program’s environment to prevent successful exploitation of a vulnerability • Program based security mechanisms compiled into program • Problem: Testing of program based security mechanisms tends to be poor

6. Example Security Mechanism: ProPolice Runtime Stack Caller int f (parameters) { local_vars char buf2 [5]; gets (buf2) // vulnerability …. } 0x1200 return address f canary canary local vars buf2 (Etoh and Yoda, 2000)

7. Current testing of security mechanisms Implement Mechanism Exploit no Attack Prevented? Execute Vulnerable Prog w/ Sec. Mechanism Compile yes Vuln. Prog.c

8. What’s Wrong with Security Mechanism Testing? • Limited amount of “exploitable” programs available • Not a lot of testing performed • Some mistakes may go unnoticed • movb instead of movl Our research: create attack simulator to test program based security mechanisms

9. Goals of Security Mechanism Testing • Systematic -- insert attacks at any appropriate point • Reliable -- accurately report effectiveness of security mechanism tested • Low Overhead -- should be efficient and have reasonable overhead

10. Testing Framework Test Case Existing Input Program w/ Sec. Mechanism Attack Spec Attack Simulator Coverage Report Monitor Oracle Test Report Framework (Breech and Pollock, SESS 2005)

11. Attack Simulator Approach • Utilize a dynamic compiler to analyze code during execution • Test security mechanisms by dynamically adding code to simulate attack • No modification of program code • Allows testing of mechanisms without vulnerable programs

12. Architecture of Attack Simulator Dynamic Compiler Program w/ Sec. Mech. Create Basic Block Attack Simulator Should Attack? Program Input yes no Insert Attack Execute Block on CPU

13. Recovering During Testing • Most attacks terminate test run • Only do one attack / test run • Can “recover” from some simulated attacks by undoing damage done • Enables more efficient testing of security mechanisms by avoiding startup costs • Requires some extra code to be compiled into program

14. Testing Stack Smashing Mechanisms • Goal: modify return address • each function call is possible attack point • 3 modes of testing • Direct -- only modify return address • Tailored -- modify enough to trigger security mechanism • Random -- overflow a buffer by writing random values into it

15. Attack Mode: Direct Runtime Stack Modify return address directly Caller 0x1200 0x5555 return address • Final outcome identical to stack smashing • Can recover • Does not exactly simulate stack smashing attack Current 0xff12 canary local vars

16. Attack Mode: Tailored Runtime Stack Modify both return address and canary (tailor attack to sec. mechanism) Caller 0x1200 0x5555 return address • Mechanism should identify all such attacks • Can recover • Does not exactly simulate stack smashing attack Current 0xff12 0xf0ba canary local vars

17. Attack Mode: Random Runtime Stack Write random values into (and beyond) local buffer Caller yz 0x1200 uvwx return address • Exactly simulates buffer overflow • Can’t recover qrst Current canary 0xff12 mnop abcd efgh ijkl local vars

18. Evaluation Questions • How well does the attack simulator identify attack points and provide systematic testing? • How well do some known security mechanisms identify attacks? • How much overhead is incurred?

19. Evaluation Methodology • Implemented attack modes as DynamoRIO1 modules • Security Mechanisms • ProPolice2 from GCC 4.1 • RAD3 (testing details in paper) • Executed 4 C programs • None had known vulnerabilities • ~1k - 10k LOC 1 (Bruening et al. CGO 2003) 2(Etoh and Yoda, 2000) 3(Chiueh and Hsu, ICDCS 2001)

20. Results: Identifying Attack Points Found almost all attack points in direct and tailored modes • Missed main

21. Results: ProPolice Detection of Simulated Attacks ProPolice detected all attacks it should have • 4 nondetected strcpy wrote to global variables (can’t be used for stack smashing attack)

22. Results: Tailored Mode Overhead (seconds) Reasonable overhead

23. Results: Summary • Attack Simulator identified almost all possible attack points (missed main) • ProPolice detected all attacks designed to detect • Reasonable overhead for attack simulator

24. Conclusions and Future Work • Presented attack simulator to provide more systematic testing of security mechanisms • Demonstrated on known security mechs. • Does not require vulnerable programs • Future work: • Simulate other attacks (function pointers, etc) • Seed faults in security mechanisms to further evaluate attack simulator

25. Example Vulnerability Caller AR 42 int blah (int a, int b, int c) { char buf1 [3]; int z; char buf2 [5]; gets (buf2) …. } c -> 6 b -> a -> 7 return address -> 0x1200 QRST saved fp -> 0x150 MNOP blah AR JKL buf1 [] -> z -> FGHI Input: ABCDEFGHIJKLMNOPQRSTUVWX ABCDE buf2 [] ->

26. Example Mechanism: ProPolice Caller AR 42 c -> int blah (int a, int b, int c) { char buf1 [3]; int z; char buf2 [5]; gets (buf2) …. } 6 b -> a -> 7 return address -> UVWX 0x1200 saved fp -> 0x150 QRST MNOP canary blah AR JKL buf1 [] -> z -> FGHI Input: ABCDEFGHIJKLMNOPQRSTUVWX ABCDE buf2 [] ->