Instructor: Erol Sahin

Machine Programming – IA32 memory layout and buffer overflowCENG331: Introduction to Computer Systems7th Lecture Instructor: ErolSahin • Acknowledgement: Most of the slides are adapted from the ones prepared • by R.E. Bryant, D.R. O’Hallaron of Carnegie-Mellon Univ.

FF C0 BF Stack 80 Heap 7F 40 DLLs 3F Heap Data 08 Text 00 Linux Memory Layout • Stack • Runtime stack (8MB limit) • Heap • Dynamically allocated storage • When call malloc, calloc, new • DLLs • Dynamically Linked Libraries • Library routines (e.g., printf, malloc) • Linked into object code when first executed • Data • Statically allocated data • E.g., arrays & strings declared in code • Text • Executable machine instructions • Read-only Upper 2 hex digits of address Red Hat v. 6.2 ~1920MB memory limit

Some Heap More Heap Initially Linked BF BF BF BF Stack Stack Stack Stack 80 80 80 80 Heap 7F 7F 7F 7F Heap 40 40 DLLs 40 DLLs 40 DLLs 3F 3F 3F 3F Heap Data Data Data Data Text Text Text 08 08 08 08 Text 00 00 00 00 Linux Memory Allocation

Initially BF Stack 80 7F 40 3F Data Text 08 00 Text & Stack Example • Main • Address 0x804856f should be read 0x0804856f • Stack • Address 0xbffffc78 (gdb) break main (gdb) run Breakpoint 1, 0x804856f in main () (gdb) print $esp $3 = (void *) 0xbffffc78

Linked BF Stack 80 7F 40 DLLs 3F Data Text 08 00 Dynamic Linking Example (gdb) print malloc $1 = {<text variable, no debug info>} 0x8048454 <malloc> (gdb) run Program exited normally. (gdb) print malloc $2 = {void *(unsigned int)} 0x40006240 <malloc> • Initially • Code in text segment that invokes dynamic linker • Address 0x8048454 should be read 0x08048454 • Final • Code in DLL region

Memory Allocation Example char big_array[1<<24]; /* 16 MB */ char huge_array[1<<28]; /* 256 MB */ int beyond; char *p1, *p2, *p3, *p4; int useless() { return 0; } int main() { p1 = malloc(1 <<28); /* 256 MB */ p2 = malloc(1 << 8); /* 256 B */ p3 = malloc(1 <<28); /* 256 MB */ p4 = malloc(1 << 8); /* 256 B */ /* Some print statements ... */ }

BF Stack 80 Heap 7F 40 DLLs 3F Heap Data 08 Text 00 Example Addresses $esp 0xbffffc78 p3 0x500b5008 p1 0x400b4008 Final malloc 0x40006240 p4 0x1904a640 p2 0x1904a538 beyond 0x1904a524 big_array 0x1804a520 huge_array 0x0804a510 main() 0x0804856f useless() 0x08048560 Initial malloc 0x08048454

Internet Worm and IM War • November, 1988 • Internet Worm attacks thousands of Internet hosts. • How did it happen? • July, 1999 • Microsoft launches MSN Messenger (instant messaging system). • Messenger clients can access popular AOL Instant Messaging Service (AIM) servers AIM client MSN server MSN client AIM server AIM client

Internet Worm and IM War (cont.) • August 1999 • Mysteriously, Messenger clients can no longer access AIM servers. • Microsoft and AOL begin the IM war: • AOL changes server to disallow Messenger clients • Microsoft makes changes to clients to defeat AOL changes. • At least 13 such skirmishes. • How did it happen? • The Internet Worm and AOL/Microsoft War were both based on stack buffer overflow exploits! • many Unix functions do not check argument sizes. • allows target buffers to overflow.

String Library Code • Implementation of Unix function gets • No way to specify limit on number of characters to read • Similar problems with other Unix functions • strcpy: Copies string of arbitrary length • scanf, fscanf, sscanf, when given %s conversion specification /* Get string from stdin */ char *gets(char *dest){int c = getc(); char *p = dest; while (c != EOF && c != '\n') { *p++ = c; c = getc(); } *p = '\0'; return dest; }

Vulnerable Buffer Code /* Echo Line */void echo(){ char buf[4]; /* Way too small! */ gets(buf); puts(buf);} int main(){printf("Type a string:"); echo(); return 0;}

Buffer Overflow Executions unix>./bufdemo Type a string:123 123 unix>./bufdemo Type a string:12345 Segmentation Fault unix>./bufdemo Type a string:12345678 Segmentation Fault

Stack Frame for main Return Address Saved %ebp %ebp [3] [2] [1] [0] buf Stack Frame for echo Buffer Overflow Stack /* Echo Line */void echo(){ char buf[4]; /* Way too small! */ gets(buf); puts(buf);} echo:pushl %ebp # Save %ebp on stackmovl %esp,%ebpsubl $20,%esp # Allocate space on stackpushl %ebx # Save %ebxaddl $-12,%esp # Allocate space on stackleal -4(%ebp),%ebx # Compute buf as %ebp-4pushl %ebx # Push buf on stack call gets # Call gets . . .

Stack Frame for main Stack Frame for main Return Address Return Address Saved %ebp %ebp Saved %ebp 0xbffff8d8 [3] [2] [1] [0] buf [3] [2] [1] [0] buf Stack Frame for echo bf Stack Frame for echo ff f8 f8 08 04 86 4d xx xx xx xx unix> gdb bufdemo (gdb) break echo Breakpoint 1 at 0x8048583 (gdb) run Breakpoint 1, 0x8048583 in echo () (gdb) print /x *(unsigned *)$ebp $1 = 0xbffff8f8 (gdb) print /x *((unsigned *)$ebp + 1) $3 = 0x804864d Buffer Overflow Stack Example Before call to gets 8048648: call 804857c <echo> 804864d: mov 0xffffffe8(%ebp),%ebx # Return Point

Stack Frame for main Stack Frame for main Return Address Return Address Saved %ebp 0xbffff8d8 Saved %ebp %ebp [3] [2] [1] [0] buf [3] [2] [1] [0] buf Stack Frame for echo Stack Frame for echo bf ff f8 f8 08 04 86 4d 00 33 32 31 Buffer Overflow Example #1 Before Call to gets Input = “123” No Problem

Stack Frame for main Return Address Stack Frame for main Saved %ebp 0xbffff8d8 [3] [2] [1] [0] buf Stack Frame for echo Return Address Saved %ebp %ebp [3] [2] [1] [0] buf Stack Frame for echo bf ff 00 35 08 04 86 4d 34 33 32 31 Buffer Overflow Stack Example #2 Input = “12345” Saved value of %ebp set to 0xbfff0035 Bad news when later attempt to restore %ebp echo code: 8048592: push %ebx 8048593: call 80483e4 <_init+0x50> # gets 8048598: mov 0xffffffe8(%ebp),%ebx 804859b: mov %ebp,%esp 804859d: pop %ebp # %ebp gets set to invalid value 804859e: ret

Stack Frame for main Return Address Stack Frame for main Saved %ebp 0xbffff8d8 [3] [2] [1] [0] buf Stack Frame for echo Return Address Saved %ebp %ebp [3] [2] [1] [0] buf Invalid address Stack Frame for echo 38 37 36 35 No longer pointing to desired return point 08 04 86 00 34 33 32 31 Buffer Overflow Stack Example #3 Input = “12345678” %ebp and return address corrupted 8048648: call 804857c <echo> 804864d: mov 0xffffffe8(%ebp),%ebx # Return Point

Malicious Use of Buffer Overflow Stack after call to gets() • Input string contains byte representation of executable code • Overwrite return address with address of buffer • When bar() executes ret, will jump to exploit code void foo(){ bar(); ... } foo stack frame return address A B data written by gets() pad void bar() { char buf[64]; gets(buf); ... } exploit code bar stack frame B

Exploits Based on Buffer Overflows • Buffer overflow bugs allow remote machines to execute arbitrary code on victim machines. • Internet worm • Early versions of the finger server (fingerd) used gets() to read the argument sent by the client: • finger droh@cs.cmu.edu • Worm attacked fingerd server by sending phony argument: • finger “exploit-code padding new-return-address” • exploit code: executed a root shell on the victim machine with a direct TCP connection to the attacker.

Exploits Based on Buffer Overflows • Buffer overflow bugs allow remote machines to execute arbitrary code on victim machines. • IM War • AOL exploited existing buffer overflow bug in AIM clients • exploit code: returned 4-byte signature (the bytes at some location in the AIM client) to server. • When Microsoft changed code to match signature, AOL changed signature location.

Date: Wed, 11 Aug 1999 11:30:57 -0700 (PDT) • From: Phil Bucking <philbucking@yahoo.com> • Subject: AOL exploiting buffer overrun bug in their own software! • To: rms@pharlap.com • Mr. Smith, • I am writing you because I have discovered something that I think you • might find interesting because you are an Internet security expert with • experience in this area. I have also tried to contact AOL but received • no response. • I am a developer who has been working on a revolutionary new instant • messaging client that should be released later this year. • ... • It appears that the AIM client has a buffer overrun bug. By itself • this might not be the end of the world, as MS surely has had its share. • But AOL is now *exploiting their own buffer overrun bug* to help in • its efforts to block MS Instant Messenger. • .... • Since you have significant credibility with the press I hope that you • can use this information to help inform people that behind AOL's • friendly exterior they are nefariously compromising peoples' security. • Sincerely, • Phil Bucking • Founder, Bucking Consulting • philbucking@yahoo.com It was later determined that this email originated from within Microsoft!

Code Red Worm • History • June 18, 2001. Microsoft announces buffer overflow vulnerability in IIS Internet server • July 19, 2001. over 250,000 machines infected by new virus in 9 hours • White house must change its IP address. Pentagon shut down public WWW servers for day • When We Set Up CS:APP Web Site • Received strings of form GET /default.ida?NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN....NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u9090%u8190%u00c3%u0003%u8b00%u531b%u53ff%u0078%u0000%u00=a HTTP/1.0" 400 325 "-" "-"

Code Red Exploit Code • Starts 100 threads running • Spread self • Generate random IP addresses & send attack string • Between 1st & 19th of month • Attack www.whitehouse.gov • Send 98,304 packets; sleep for 4-1/2 hours; repeat • Denial of service attack • Between 21st & 27th of month • Deface server’s home page • After waiting 2 hours

Code Red Effects • Later Version Even More Malicious • Code Red II • As of April, 2002, over 18,000 machines infected • Still spreading • Paved Way for NIMDA • Variety of propagation methods • One was to exploit vulnerabilities left behind by Code Red II

Avoiding Overflow Vulnerability • Use Library Routines that Limit String Lengths • fgets instead of gets • strncpy instead of strcpy • Don’t use scanf with %s conversion specification • Use fgets to read the string /* Echo Line */void echo(){ char buf[4]; /* Way too small! */fgets(buf, 4, stdin); puts(buf);}

Final Observations • Memory Layout • OS/machine dependent (including kernel version) • Basic partitioning: stack/data/text/heap/DLL found in most machines • Type Declarations in C • Notation obscure, but very systematic • Working with Strange Code • Important to analyze nonstandard cases • E.g., what happens when stack corrupted due to buffer overflow • Helps to step through with GDB

Machine Programming – x86-64CENG331: Introduction to Computer Systems7th Lecture Instructor: ErolSahin • Acknowledgement: Most of the slides are adapted from the ones prepared • by R.E. Bryant, D.R. O’Hallaron of Carnegie-Mellon Univ.

x86-64 Integer Registers %rax %r8 %eax %r8d • Twice the number of registers • Accessible as 8, 16, 32, 64 bits %rbx %r9 %ebx %r9d %rcx %r10 %ecx %r10d %rdx %r11 %edx %r11d %rsi %r12 %esi %r12d %rdi %r13 %edi %r13d %rsp %r14 %esp %r14d %rbp %r15 %ebp %r15d

x86-64 Integer Registers %rax %r8 Return value Argument #5 %rbx %r9 Callee saved Argument #6 %rcx %r10 Callee saved Argument #4 %rdx %r11 Used for linking Argument #3 %rsi %r12 C: Callee saved Argument #2 %rdi %r13 Argument #1 Callee saved %rsp %r14 Stack pointer Callee saved %rbp %r15 Callee saved Callee saved

x86-64 Registers • Arguments passed to functions via registers • If more than 6 integral parameters, then pass rest on stack • These registers can be used as caller-saved as well • All references to stack frame via stack pointer • Eliminates need to update %ebp/%rbp • Other Registers • 6+1 callee saved • 2 or 3 have special uses

x86-64 Long Swap swap: movq (%rdi), %rdx movq (%rsi), %rax movq %rax, (%rdi) movq %rdx, (%rsi) ret void swap(long *xp, long *yp) { long t0 = *xp; long t1 = *yp; *xp = t1; *yp = t0; } • Operands passed in registers • First (xp) in %rdi, second (yp) in %rsi • 64-bit pointers • No stack operations required (except ret) • Avoiding stack • Can hold all local information in registers

x86-64 Locals in the Red Zone swap_a: movq (%rdi), %rax movq %rax, -24(%rsp) movq (%rsi), %rax movq %rax, -16(%rsp) movq -16(%rsp), %rax movq %rax, (%rdi) movq -24(%rsp), %rax movq %rax, (%rsi) ret • Avoiding Stack Pointer Change • Can hold all information within small window beyond stack pointer /* Swap, using local array */ void swap_a(long *xp, long *yp) { volatile long loc[2]; loc[0] = *xp; loc[1] = *yp; *xp = loc[1]; *yp = loc[0]; } rtnPtr %rsp unused −8 loc[1] −16 Volatile tells the compiler that the value of the variable may change at any time--without any action being taken by the code the compiler finds nearby. Useful when working with I/O devices and interrupts. loc[0] −24

x86-64 NonLeaf without Stack Frame • No values held while swap being invoked • No callee save registers needed long scount = 0; /* Swap a[i] & a[i+1] */ void swap_ele_se (long a[], inti) { swap(&a[i], &a[i+1]); scount++; } swap_ele_se: movslq %esi,%rsi # Sign extend i leaq (%rdi,%rsi,8), %rdi # &a[i] leaq 8(%rdi), %rsi # &a[i+1] call swap # swap() incq scount(%rip) # scount++; ret

x86-64 Call using Jump long scount = 0; /* Swap a[i] & a[i+1] */ void swap_ele(long a[], inti) { swap(&a[i], &a[i+1]); } swap_ele: movslq %esi,%rsi # Sign extend i leaq (%rdi,%rsi,8), %rdi # &a[i] leaq 8(%rdi), %rsi # &a[i+1] jmp swap # swap() Will disappear Blackboard?

x86-64 Call using Jump • When swap executes ret, it will return from swap_ele • Possible since swap is a “tail call”(no instructions afterwards) long scount = 0; /* Swap a[i] & a[i+1] */ void swap_ele(long a[], inti) { swap(&a[i], &a[i+1]); } swap_ele: movslq %esi,%rsi # Sign extend i leaq (%rdi,%rsi,8), %rdi # &a[i] leaq 8(%rdi), %rsi # &a[i+1] jmp swap # swap()

x86-64 Stack Frame Example swap_ele_su: movq %rbx, -16(%rsp) movslq %esi,%rbx movq %r12, -8(%rsp) movq %rdi, %r12 leaq (%rdi,%rbx,8), %rdi subq $16, %rsp leaq 8(%rdi), %rsi call swap movq (%r12,%rbx,8), %rax addq %rax, sum(%rip) movq (%rsp), %rbx movq 8(%rsp), %r12 addq $16, %rsp ret • Keeps values of a and i in callee save registers • Must set up stack frame to save these registers long sum = 0; /* Swap a[i] & a[i+1] */ void swap_ele_su (long a[], inti) { swap(&a[i], &a[i+1]); sum += a[i]; }

Understanding x86-64 Stack Frame swap_ele_su: movq %rbx, -16(%rsp) # Save %rbx movslq %esi,%rbx # Extend & save i movq %r12, -8(%rsp) # Save %r12 movq %rdi, %r12 # Save a leaq (%rdi,%rbx,8), %rdi # &a[i] subq $16, %rsp # Allocate stack frame leaq 8(%rdi), %rsi # &a[i+1] call swap # swap() movq (%r12,%rbx,8), %rax # a[i] addq %rax, sum(%rip) # sum += a[i] movq (%rsp), %rbx # Restore %rbx movq 8(%rsp), %r12 # Restore %r12 addq $16, %rsp# Deallocate stack frame ret

%rsp rtnaddr rtnaddr +8 %r12 %r12 −8 %rsp %rbx %rbx −16 Understanding x86-64 Stack Frame swap_ele_su: movq %rbx, -16(%rsp) # Save %rbx movslq %esi,%rbx # Extend & save i movq %r12, -8(%rsp) # Save %r12 movq %rdi, %r12 # Save a leaq (%rdi,%rbx,8), %rdi # &a[i] subq $16, %rsp # Allocate stack frame leaq 8(%rdi), %rsi # &a[i+1] call swap # swap() movq (%r12,%rbx,8), %rax # a[i] addq %rax, sum(%rip) # sum += a[i] movq (%rsp), %rbx # Restore %rbx movq 8(%rsp), %r12 # Restore %r12 addq $16, %rsp# Deallocate stack frame ret

Interesting Features of Stack Frame • Allocate entire frame at once • All stack accesses can be relative to %rsp • Do by decrementing stack pointer • Can delay allocation, since safe to temporarily use red zone • Simple deallocation • Increment stack pointer • No base/frame pointer needed

x86-64 Procedure Summary • Heavy use of registers • Parameter passing • More temporaries since more registers • Minimal use of stack • Sometimes none • Allocate/deallocate entire block • Many tricky optimizations • What kind of stack frame to use • Calling with jump • Various allocation techniques

IA32 Floating Point (x87) • History • 8086: first computer to implement IEEE FP • separate 8087 FPU (floating point unit) • 486: merged FPU and Integer Unit onto one chip • Becoming obsolete with x86-64 • Summary • Hardware to add, multiply, and divide • Floating point data registers • Various control & status registers • Floating Point Formats • single precision (C float): 32 bits • double precision (C double): 64 bits • extended precision (C long double): 80 bits Instruction decoder and sequencer Integer Unit FPU Memory

FPU Data Register Stack (x87) • FPU register format (80 bit extended precision) • FPU registers • 8 registers %st(0) - %st(7) • Logically form stack • Top: %st(0) • Bottom disappears (drops out) after too many pushs 0 79 78 64 63 s exp frac %st(3) %st(2) %st(1) “Top” %st(0)

FPU instructions (x87) • Large number of floating point instructions and formats • ~50 basic instruction types • load, store, add, multiply • sin, cos, tan, arctan, and log • Often slower than math lib • Sample instructions: Instruction EffectDescription fldz push 0.0Load zero fldsAddr pushMem[Addr] Load single precision real fmulsAddr %st(0)  %st(0)*M[Addr] Multiply faddp %st(1)  %st(0)+%st(1);popAdd and pop

FP Code Example (x87) pushl %ebp # setup movl %esp,%ebp pushl %ebx movl 8(%ebp),%ebx # %ebx=&x movl 12(%ebp),%ecx # %ecx=&y movl 16(%ebp),%edx # %edx=n fldz # push +0.0 xorl %eax,%eax # i=0 cmpl %edx,%eax # if i>=n done jge .L3 .L5: flds (%ebx,%eax,4) # push x[i] fmuls (%ecx,%eax,4) # st(0)*=y[i] faddp # st(1)+=st(0); pop incl %eax # i++ cmpl %edx,%eax # if i<n repeat jl .L5 .L3: movl -4(%ebp),%ebx # finish movl %ebp, %esp popl %ebp ret # st(0) = result • Compute inner product of two vectors • Single precision arithmetic • Common computation float ipf (float x[], float y[], int n) { inti; float result = 0.0; for (i = 0; i < n; i++) result += x[i]*y[i]; return result; }

Inner Product Stack Trace eax = iebx = *x ecx = *y Initialization 1.fldz 0.0 %st(0) Iteration 0 Iteration 1 2.flds (%ebx,%eax,4) 5.flds (%ebx,%eax,4) 0.0 %st(1) x[0]*y[0] %st(1) x[0] %st(0) x[1] %st(0) 3.fmuls (%ecx,%eax,4) 6.fmuls (%ecx,%eax,4) 0.0 %st(1) x[0]*y[0] %st(1) x[0]*y[0] %st(0) x[1]*y[1] %st(0) 4.faddp 7.faddp x[0]*y[0]+x[1]*y[1] 0.0+x[0]*y[0] %st(0) %st(0)

Vector Instructions: SSE Family • SSE : Streaming SIMD Extensions • SIMD (single-instruction, multiple data) vector instructions • New data types, registers, operations • Parallel operation on small (length 2-8) vectors of integers or floats • Example: • Floating point vector instructions • Available with Intel’s SSE (streaming SIMD extensions) family • SSE starting with Pentium III: 4-way single precision • SSE2 starting with Pentium 4: 2-way double precision • All x86-64 have SSE3 (superset of SSE2, SSE) “4-way” x +

Intel Architectures (Focus Floating Point) Processors Architectures Features x86-16 8086 286 x86-32 386 486 Pentium Pentium MMX Pentium III Pentium 4 Pentium 4E MMX SSE SSE2 SSE3 4-way single precision fp 2-way double precision fp time x86-64 / em64t Pentium 4F Core 2 Duo Our focus: SSE3 used for scalar (non-vector)floating point SSE4

SSE3 Registers • All caller saved • %xmm0 for floating point return value 128 bit = 2 doubles = 4 singles %xmm0 %xmm8 Argument #1 %xmm1 %xmm9 Argument #2 %xmm2 %xmm10 Argument #3 %xmm3 %xmm11 Argument #4 %xmm4 %xmm12 Argument #5 %xmm5 %xmm13 Argument #6 %xmm6 %xmm14 Argument #7 %xmm7 %xmm15 Argument #8

SSE3 Registers • Different data types and associated instructions • Integer vectors: • 16-way byte • 8-way 2 bytes • 4-way 4 bytes • Floating point vectors: • 4-way single • 2-way double • Floating point scalars: • single • double 128 bit LSB

SSE3 Instructions: Examples • Single precision 4-way vector add: addps %xmm0 %xmm1 • Single precision scalar add: addss %xmm0 %xmm1 %xmm0 + %xmm1 %xmm0 + %xmm1

Instructor: Erol Sahin