2006. 2. 20. Researcher : Lee, Seung Min (Presenter : Lee, Seung Ick)

1. An Enhanced Buffer Separation Scheme to Protect Security Sensitive Data against Buffer Overflow Attacks 2006. 2. 20. Researcher : Lee, Seung Min (Presenter : Lee, Seung Ick) High Performance Computing Laboratory at POSTECH

2. Contents • Introduction • Related Works • Motivation • Problem Definition • Proposed Idea • Performance Analysis • Conclusion and Future Works

3. Introduction • Buffer overflow • Occur when a program or process stores more data in buffer than the buffer size • Security sensitive data • Data to be a target of buffer overflow attack for a successful change of control flow • Basic steps of buffer overflow attack • The first step • To find, discover and identify vulnerability of buffer overflow • The second step • To overflow and overwrite security sensitive data near buffer • The third step • To change control flow of process • The fourth step • To execute intended instruction or process

4. Related Works ( 1/2 ) Protection Methods • Protection Methods are classified depending on the step the prevention method is applied. • The first step • To eliminate buffer overflow vulnerability • The second step • To prevent overflow or overwrite data • The third step • To prevent no intended control flow of a program or process • The fourth step • To apply that data of buffer can’t be executed through hardware support

5. Related Works ( 2/2 ) Problems of Previous Works • Problem of the prevention method on the first step • Generate too many false warnings and miss errors in the code • Problem of the prevention method on the second step • Performance degradation through array bounds checking • Not to provide complete protection against vulnerabilities in user defined or non standard library code • Problem of prevention method on the third and fourth steps • Occur an exception or termination because of attack failure

6. Motivation ( 1/2 ) • The prevention methods applied in these steps but the second have the problem of process availability, since the process stops its execution because of false warnings, exceptions or termination. • Our focus is to provide a reliable process availability and more secure protection method. • Buffer separation approach is the method applied on the second step for prevention. • Remove buffers from stack • Allow the occurrence of buffer overflow but prevent security sensitive data from being overwritten • Gemini and DYBOC

7. Motivation ( 2/2 ) • Shortcoming of previous works • Gemini • Can generate heap overflow because of buffer using heap area instead of stack • DYBOC • Has a memory overhead problem because of using write- protected page

8. Problem Definition • To prevent stack and heap overflow for using enhanced buffer separation approach • To have minimal performance penalty

9. Proposed Idea ( 1/3 ) Basic Idea • Assumption • We can know the size of arrays at compile time. • Enhanced buffer separation schemes • Buffer stack • Separate buffer from the security sensitive data on the stack area. • Separated meta data • Separate meta data from buffer on the heap area.

10. Proposed Idea ( 2/3 ) Buffer Stack Architecture • Buffer stack is to prevent against stack overflow. • Buffer stack will be determined as using heap or stack at the compile time.

11. Proposed Idea ( 3/3 ) Separated Meta Data Architecture • Separated meta data is to prevent against heap overflow.

12. Performance Analysis • Performance comparison with Gemini, DYBOC and our solutions • Limitation • If the security sensitive data exists inside buffers, it is very hard to split them from the buffers. • Our solutions can’t prevent data from pointer operations that use the primitive type variables. • Compiler has to know the size of stack.

13. We chose two vulnerabilities which is similar to the source of finger demon and PCT SSL vulnerability. We applied single thread program but it can be applied in multi-threaded environment easily. In both cases, the security sensitive data of the original program is overwritten and make an exception. But, the program applied our approaches correctly executes because it protects the security sensitive data even though the buffer data is tainted. Experiments

14. Conclusion and Future Works • Conclusion • Buffer stack prevents the stack overflow attack. • Separated meta data prevents the heap overflow attack. • Future works • We must find trade-off between performance and memory overhead on case 1 and 2 of the Buffer stack architecture, and then will choose one of cases. • Throughout the implementation, we can provide more exact performance analysis. • We can provide more powerful solutions to change the main assumption: security sensitive data is located near the buffer and also in the buffer.

15. Thank you!Questions?

16. Backup slide

17. References • [1] Hiroaki Etoh and Kunikazu Yoda. Protecting from Stack-Smashing Attacks. Published on World-Wide Web at URL http://www.trl.ibm.com/projets/security/ssp/main.html, June 2000 • [2] Donaldson, Mark E. Inside The Buffer Overflow Attack: Mechanism, Method, & Prevention. April 3, 2002. URL:http://www.sans.org/rr/paper.php?id=386 • [3] Bharath Madhusudan, John Lockwood. Design of a System for Real-Time Worm • [4] H. Wang, C. Guo, D. Simon, and A. Zugenmaier. Shield: Vulnerability-driven network filters for preventing known vulnerability exploits. In Proceedings of ACM SIGCOMM, Portland, OR, Aug. 2004 Detection, 12th Annual Proceedings of IEEE Hot Interconnects 2004 • [5] J. Newsome and D. Song. Dynamic taint analysis for automatic detection, analysis, and signature generation of exploits on commodity software. In Proceedings of the 12th Annual Network and Distributed System Security Symposium(NDSS05), Feb. 2005 • [6] Rinard. M., Cada. C., Dumitran. D., Roy. D., Leu.T. A Dynamic Technique for Eliminating Buffer Overflow Vulnerabilities (and Other Memory Errors). In: Proceedings 20th Annual Computer Security Applications Conference (ACSAC), 2004 • [7] StackShield. http://www.angelfire.com/sk/stackshield • [8] A. Baratloo, T. Tsai, and N. Singh. Transparent Run-Time Defense Against Stack Smashing Attacks. In Proceedings. of the USENIX Annual Technical Conference, June 2000

18. Reference (Cont.) • [9] Crispin Cowan, Steve Beattie, John Johansen, and Perry Wagle. Pointguard: Protecting pointers from buffer overflow vulnerabilities. In Proceedings of the 12th USENIX Security Symposium, Washington, D.C., August 2003 • [10] BB. Madan, S. Phoha, G. NIST, KS Trivedi, StackOfence: A Technique for Defending Against Buffer Overflow Attacks, In Proceedings of the International COnference on Information Technology: Coding and Computing(ITCC05), 2005 • [11] J. Xu, Z. Kalbarczyk, S. Patel, and R. K. Iyer. Architecture support for defending against buffer overflow attacks. In 2nd Workshop on Evaluating and Architecting Systems for Dependability, 2002 • [12] S Bhatkar, DC DuVarney, R Sekar, Address obfuscation: An efficient approach to combat a broad range of memory error exploits, In Proceedings of the 12th USENIX Security Symposium, 2003 • [13] Wilander, J. and M. Kamkar, Comparison of Publicly Available Tools for Dynamic Buffer Overflow Prevention, 10th Network and Distributed System Security Symposium, 2003 • [14] Stelios Sidiroglou, Giannis Giovanidis, and Angelos D. Keromytis, A Dynamic Mechanism for Recovering from Buffer Overflow Attacks • [15] R Hieb, RK Dybvig, C Bruggeman, L Hall, Representing Control in the Presence of First-Class Continuations, In Proceedings of the ACM SIGPLAN 1990 conference on 1990 [16] EG Barrantes, DH Ackley, S Forrest, D Stefanovic, Randomized Instruction Set Emulation, ACM Transactions on Information and System Security, 2005 • [16] Christopher Dahn, Spiros Mancoridis, Using Program Transformation to Secure C Programs Against Buffer Overflows, in Proceedings of the 10th Working Conference on Reverse Engineering (WCRE03)

19. Scenario Modified Pseudo Assembly Code Original Pseudo Assembly Code a Original Code push ebpmov ebp, espsub exp, 10push [ebp+4]push expcall strcpyadd exp,10leaveretpush ebpmov ebp, espsub esp, 4sub exp, 20mov [exp+5],4push expcall f1add exp,20add esp,4leaveret Return Address of main push ebpmov ebp, espsub esp, 10push [ebp+8]push [ebp-10]call strcpyadd esp,10leaveretpush ebpmov ebp, espsub esp, 24mov [ebp-15],4push ebp-20call f1add esp,24leaveret Old EBP void f(char *a) { char c[10]; strcpy(c,a);}void main() { char a[20]; int i; a[5] = 4; f1(a);} Prologue EBP a EXP c i address of a Return Address of f ESP Epilogue Old EBP c Return Address of main Old EBP EBP i address of a ESP Return Address of f Old EBP

20. Scenario (Cont.) Modified Pseudo Assembly Code Original Code Return Address of main push ebpmov ebp, espsub esp, 10push [ebp+4+S]push ebp-10+Scall strcpyadd esp,10leaveretpush ebpmov ebp, espsub esp, 24mov [exp-15+S],4push expcall f1add esp,24leaveret Old EBP void f(char *a) { char c[10]; strcpy(c,a);}void main() { char a[20]; int i; a[5] = 4; f1(a);} Prologue EBP a i address of a Return Address of f ESP Epilogue Old EBP c S : Size of Stack

21. Vulnerable code of PCT SSL vulnerability function(char *packet, unsigned int N){         char buf[32];         unsigned int register i;         if(N < 32)         {              memcpy(buf,packet,N);             for(i = 0; i < N; i++)             buf[i+N] = ~buf[i];         }}