A Binary Agent Technology for COTS Software Integrity

1. A Binary Agent Technology for COTS Software Integrity Richard Schooler Anant Agarwal InCert Software

2. Operating System Input COTS Binary COTS Binary SAP Output The Mission Critical Environment The development environment The deployment environment

3. COTS Binary Objective Operating System Input To improve the integrity of the deployment environment with COTS software in the presence of attacks, bugs SAP Output

4. COTS Binary Our Focus Operating System Outer security defences will be breached by attackers Use a practical, systems level approach – execution-time monitoring On COTS program or data corruption, rapidly d-detect problems a- trigger an alarm p- try to protect r- recover Input SAP Output

5. COTS COTS COTS New New New Legacy Legacy Legacy Missing source Missing source Missing source Our Approach: Execution-Time Monitoring of COTS through Binary Instrumentation d- Policy specs for detection d- Heartbeat insertion d- Argument range checks d- Rare code execution/sigs. a- Alarm messages to console p- Defaults for fault tolerance p- Access constraints, redund. r- Logging COTS Binary The development environment The deployment environment

6. Our Approach to Integrity • A pragmatic system level approach • Works with COTS binaries, even legacy codes • Allows a user to establish desired security levels and to some extent modify policy on the fly • Allows monitoring of all user code execution, not selected OS calls • Can work completely at the user’s deployment site

7. Drawbacks of Binary Insertion • Specific to a single platform, needs new technology development for different platform • Challenging to relate low-level observable events back to high-level user actions • hard to detect some types of intrusions that only affect data corruption • hard to protect or correct problems at higher semantic levels

8. Three Major Components in the Prototype,Three Major Tasks • Core technology for customizable agent insertion into PC/NT, PC/Linux • Anomaly detection and reporting • Rapid recovery and problem pinpointing

9. Selected Challenges and Ideas • Core technology for agent insertion into binary • How to deal with unknown relocations, e.g., for dusty decks – incremental control and dataflow analysis; an integrated static and dynamic method • Dealing with multithreading, in particular, time syncing and monitoring events in a distributed environment • Anomaly detection • Runtime comparison against execution path signatures? • State machines for control flow checks (e.g., Abraham) • How to minimize runtime overhead – borrow compiler optimization techniques (e.g., steal registers, inline code, sampling, multilevel checks) • Rapid recovery and problem pinpointing technology • Instrumentation records trace in circular buffer to pinpoint problem and facilitate recovery from crash/alert following an actual or suspected attack • Can we get data values? Use dataflow analysis and offline simulation to obtain intermediate data values

10. Measures of Success • Core technology for agent insertion into binary: • Can we handle all binaries, even dusty decks? • Target: Performance degradation to be under 1 percent • Anomaly detection • What fraction of injected problems can we detect • automatically • with user spec • Rapid recovery technology • Performance degradation to be under 1 percent • Can we cut recovery time significantly? We will measure recovery time with and without • As a bonus, can we catch problems before system goes down? • We are building a prototype system, work with real users, and measure

11. Current Progress • Work on NT binary insertion prototype ongoing • Demo of early capability showing • instrumentation • simple recovery log • detecting application has crashed • taking control and • writing out log • Handling multithreading, DLLs imminent • Ongoing thinking on detection capability

12. Instrumentation for Recovery: Basic Idea • At each program block, record the progress of program execution. AGENT test al,0x3 jnz 0x1143 1 ... 1 2 4 5 AGENT test al,0x3 jnz 0x1143 2 3 AGENT shr edx,0x1 add ebx,edx 4 AGENT add ebx,ecx jc 0x1101 while ((c = ++ci)) { INSTRUCTION_ITERATOR ii = c->Instructions(); while ((inst = ++ii)) inst->Lift(null_state); while ((inst = ++ii)) inst->Lift(null_state); 5 AGENT inc eax add ecx,edi add edx,esi cmp eax,0xa

13. DAG Header • setup new trace record • DAG Block 1 • set bit in current record Instrumentation Runtime • Naïve implementation: • Write block id • Increment index (in memory) • Check for wrap-around • Too big/slow: many instructions per block; large buffer 2 3 4 DAG id DAG id DAG id 5

14. Instrumentation Example 1 test al,0x3 jnz 0x1143 add ebx,ecx jmp short 0x1156 shr edx,0x1 add ebx,edx inc eax add ecx,edi add edx,esi cmp eax,byte 0xa jc 0x1101 Original

15. Instrumentation Example 2 mov eax,[0x300c] mov ecx,[byte eax+0xc] mov [byte eax+0xc],dword 0x1 mov edx,[byte eax+0x8] mov [dword eax+edx+0xffe],ecx add edx,0x4 and edx,[byte eax+0x4] mov [byte eax+0x8],edx test al,0x3 jnz 0x1143 or [byte ebp+0xe],byte 0x2 add ebx,ecx jmp short 0x1156 or [byte ebp+0xe],byte 0x1 shr edx,0x1 add ebx,edx or [byte ebp+0xe],byte 0x4 inc eax add ecx,edi add edx,esi cmp eax,byte 0xa jc 0x1101 Original Instrumentation

16. Instrumentation Example 3 mov eax,[0x300c] mov ecx,[byte eax+0xc] mov [byte eax+0xc],dword 0x1 mov edx,[byte eax+0x8] mov [dword eax+edx+0xffe],ecx add edx,0x4 and edx,[byte eax+0x4] mov [byte eax+0x8],edx test al,0x3 jnz 0x1147 or [byte ebp+0xe],byte 0x2 add ebx,ecx jmp short 0x1164 or [byte ebp+0xe],byte 0x1 shr edx,0x1 add ebx,edx or [byte ecp+0xe],byte 0x4 inc eax add ecx,edi add edx,esi cmp eax,byte 0xa jc 0x1103 Original Instrumentation Displacement Fixup

17. Instrumentation Example 4 mov eax,[0x300c] mov ecx,[byte eax+0xc] mov [byte eax+0xc],dword 0x1 mov edx,[byte eax+0x8] mov [dword eax+edx+0xffe],ecx add edx,0x4 and edx,[byte eax+0x4] mov [byte eax+0x8],edx test al,0x3 jnz 0x1147 or [byte ebp+0xe],byte 0x2 add ebx,ecx jmp short 0x1164 or [byte ebp+0xe],byte 0x1 shr edx,0x1 add ebx,edx mov ecp,ebp or [byte ecp+0xe],byte 0x4 inc eax add ecx,edi add edx,esi cmp eax,byte 0xa jc 0x1103 Original Instrumentation Displacement Fixup Register Transfer

18. Related Approaches • Applied at source level during development • e.g., type based safety; work of Lee et al. • Applied at link time with special object formats • e.g., software fault isolation; work of Pandey et al. • Applied through interpretor • e.g., safe Java interpreters • Applied during program execution – works with existing COTS • e.g., sandboxing through middleware software • e.g., modify OS with extensions to catch OS calls, Ghosh • e.g., wrap application to steer OS calls to intermediary, Ghosh

19. COTS Binary The Current Commercial Solution: Middleware Operating System Input • Slow • Cannot deal with viruses • Cannot improve fault tolerance of COTS package itself Middleware wrappers SAP Output

20. COTS Binary Sandboxing through OS Call Interception(OS extensions or wrapping executable) Operating System Input • This is the right first step, and less intrusive than binary instrumentation • Cannot improve fault tolerance of COTS package itself • Cannot handle communication through shared memory • Cannot observe internal app behavior • OS extensions can slow OS for others SAP Output

21. Summary • A systems approach to COTS Integrity • Approach based on execution-time monitoring using binary insertion • We have an early prototype version of NT binary insertion implemented