Chapter 4 Operating System Security

1. Chapter 4 Operating System Security Sue Fitzgerald Metropolitan State University CS 328 Computer Security Fall 2008

2. Overview • Hardware • Operating systems and processes • Authentication • Principle of Least Privilege • Attacks - vulnerabilities and exploits • Key loggers and rootkits • Prevention and recovery

3. Hardware • Computers are binary machines • Voltage level output by an electronic device (high/low) • Magnetic device polarity (positive/negative) • One binary digit (true/false) is called a bit • A collection of 8 bits is a byte • A collection of bits can be used to represent a binary number

4. Memory • The memory of a computer is composed of many bytes (megabytes or gigabytes) • Each byte or collection of bytes holds a binary number • Data is encoded and stored as binary numbers • Computer instructions are encoded and stored as binary numbers

5. Memory (continued) • Each byte of memory has a unique address • The address is also represented as a binary number • Components of the computer are linked by wires • Data, instructions and addresses are sent across the wires as high/low voltage levels (bits)

6. Architecture • See figure 4.1, p. 62 • The CPU processes or executes instructions such as add, compare, jump • The CPU requests the next instruction or a piece of data from the memory • The memory returns that piece of information • The CPU sends a piece of data to the memory • The memory stores that piece of data

7. CPU • The central processing unit controls the order in which instructions are executed • The CPU executes instructions • The CPU has registers (very fast memory) for keeping track of a small amount of very important information • The Program Counter (PC) is a register that holds the memory address of the next instruction to be fetched and executed

8. Fetch-Execute Cycle • The CPU takes the address of the next instruction (held in the PC) and places it on the address bus • The CPU sends a signal via the control bus, asking to read that information • The memory returns that piece of information • The CPU decodes and executes that instruction • The CPU updates the PC

9. Modern Architecture • Instructions are pipelined • One or more instructions are pre-fetched before the previous instruction is finished • Instructions are broken into stages • Several instructions can be executing at the same time, but in different stages • Instructions and data are cached

10. Operating System • An operating system manages the resources of the computer (CPU, memory, I/O devices, disk, network access, etc.) • Hides details of device management from users • Permits multi-processing – many programs (processes) executing ‘at once’ • Protects each process from interference from another process

11. CPU Modes • CPU runs in one of two modes • Kernel or privileged mode (Ring 0) • User or unprivileged mode (Ring 3) • Some instructions can only be executed when the CPU is in privileged mode • Operating systems run in privileged mode • Everything else runs in user mode

12. Privileged vs. Unprivileged Mode • For the protection of the OS, unprivileged code cannot freely change modes • The only way to go from unprivileged mode to privileged (kernel) mode is via an interrupt • An interrupt from external hardware (I/O, clock) • A runtime exception (divide by zero, illegal memory access • A trap instruction (software interrupt)

13. Interrupt Handling • When an interrupt occurs, the CPU • Suspends the execution of the running (user) process • Changes to privileged (kernel) mode • Looks in an operating system data structure, called an interrupt vector, to see where to go next • Starts running the operating system code at the address given in the interrupt vector

14. Memory Management • Must protect user processes from interfering with one another (reading/writing memory) • Must protect operating system kernel from user processes (reading/writing memory) • Solution: virtual addressing • To each process (running program), the memory of the computer appears to belong only to that process; it thinks it has the whole memory in one long contiguous piece

15. Virtual Memory • This is called the virtual address space • Instead, physical memory is broken up into chunks (pages or frames) • The physical memory is shared among all the processes • The user process’s memory may not be physically contiguous • Parts of the user process’s memory may be temporarily stored on disk

16. Virtual Memory (continued) • The memory management unit (MMU) is hardware that translates the virtual address to a physical address • The operating system sets up and controls this mapping • See Figure 4.2, p. 65 • The operating system moves missing pages of memory from disk to memory as needed

17. MMU • The MMU may mark certain parts of memory as ‘read-only’ • Attempts by user processes to write to that memory will cause a runtime exception • The MMU may prevent user processes from even reading some parts of memory which are used by the kernel for data and buffers

18. MMU (continued) • MMU tables are initialized for each user process such that the kernel’s memory is located in the same place every time for every process • When the user process traps to the kernel, the kernel can see the user data and does not need to change the MMU tables for itself

19. Process • A process is a running program • It is one of the fundamental units controlled by the operating system • Processes should be self-contained for security

20. System Calls • A user process requests an operating system service via a system call or syscall • System calls are available via a library of functions • The system call identifies what service is needed. • Additional information is provided via parameters/arguments • The system call issues a trap instruction (software interrupt)

21. System Calls (continued) • The system saves the user process’s state (program counter, variables, etc.) • Hardware switches to kernel (privileged) mode • OS runs the system call trap handler function • The trap handler looks up the requested service and retrieves an address for that service • Control passes to the service • See Figure 4.4, p. 67

22. Context Switching • Changing from running one process to another is called a ‘context switch’ • It is a relatively slow thing to do • When the service is done, the CPU drops back into user mode and control passes back to the calling (user) process • The operating systems keeps track of all processes (running, waiting, blocked for I/O)

23. User Interfaces • Two types of user interface (UI) • Command line (shell) • Graphical user interface (GUI) • Technically, the UI is not part of the OS

24. OS Security • Process isolation • Use memory management to make sure one process cannot read or write to the memory of another • Processes communicate via standard mechanisms • Interprocess communication (IPC) • Pipes, message passing, shared memory, shared files

25. IPC • IPC opens the door to leaks • Examples • Shared memory does not occupy an entire page but the operating system maps an entire page to both processes • A sends a message to B but B terminates before the message arrives. C starts and is assigned the same process number as B. The message is delivered to C.

26. IPC (continued • More examples of leaks • Object reuse • Memory pages/frames • Kernel heap regions • Temporal variations • What shared code is cached or in memory

27. Authentication • Authentication – verifying someone is who they say they are • Authorization – given the user has authenticated, to what resources are they permitted access and what sort of access are they permitted? • Users – people or processes

28. Users • Users have accounts • Users are authenticated via login/password • Special accounts are set up for groups of users (e.g., guest, games) • Some users need special privileges • Special accounts (root, superuser, admin) • Specific users are granted admin privileges

29. Principle of Least Privilege • Assign fewest privileges necessary to get the job done • Operate at lowest level of privilege • Always operating at highest level of privilege can lead to lots of damage if errors are made • What mode do you run in?

30. Filesystem Access Control • Most objects (not all) are managed by the file system • Data and programs are stored in files • Some programs are OS programs • Some files contain configuration information • Typical permissions for files are: • read, write, execute

31. Filesystems (continued) • Impersonation – changes the owner of a program so that a user can run an system program with privileges • Allows users to do necessary tasks that require root privilege • Necessary, but dangerous

32. Access Control List • Access Control List (ACL) – a column from the access control matrix • Each object has a list of access control entries • (user/group, operation, permission – allow/deny) • The operating system checks the permissions on each file access • This is the Windows approach

33. Access Control - Capabilities • A row from the access control matrix is associated with each subject • Each subject has a list of objects and what operations can be performed on them • Unix (Linux, OSX, FreeBSD) takes a hybrid approach

34. Unix Approach • Each file has an owner and a group • A group is a set of users • Users can belong to more than one group • Each file has three sets of permissions – one for the owner, one for the group and one for everyone else • The permissions are read, write and execute

35. Unix Approach (continued) • The operating system checks the permissions on each access • Objects carry their permission list (ACL-like) • Group membership grants access rights (capability-like) • Root/superuser overrides all permission checking • DAC

36. Reference Monitors • Separate policy from mechanism • Policy is documentation • Mechanism is the implementation of policy • System must practice complete mediation (check every access) in order to be secure • The ‘reference monitor’ is responsible for checking all accesses

37. Trusted Computing Base (TCB) • The set of things we are forced to trust • Should include the CPU and MMU • Should NOT include user processes • TCB should be as small as possible

38. Operating System Components • The essentials – kernel, drivers, command interpreter, filesystem • Add-ons • Networking • Remote procedure calls • Cryptography

39. Access to the OS • System calls – interface to the OS • POSIX • Portable Operating System Interface • 1980’s • IEEE standard • Standard API for operating systems • Portability • Moderately successful

40. Access to OS (continued) • Win32 API • Standard interface to Windows • Similar to POSIX • Permits underlying changes without changes to user programs making system calls

41. Remote Procedure Calls • Remote Procedure Calls (RPC) allow one computer to call procedures located on a different computer, more or less transparently • Library of functions to support this • Marshalling – serializing arguments so they can be sent over a network • Local call to stub -> marshall -> pass over network -> unmarshall -> actual call • Possible security hole

42. RPC Technologies • SunRPC for Unix • MSRPC/DCOM for Microsoft • DCOM has been replaced by .NET • RPC, in general, is being replaced by web services

43. Crypto • Operating systems support encryption and key storage • Applications use operating systems services • Another avenue for attacks

44. Extending the Kernel • Drivers for peripheral hardware – execute with kernel privileges

45. Attacks • Learn confidential information • Set up a computer so it can serve as the basis for attacking another computer • Hide the real attacker

46. Attack Strategies • 0wn or r00t – attacker has complete control over a system • Can install and run programs • Can access all files • Can modify user accounts • Access a low-privilege account and escalate to system ownership

47. Attack Strategies (continued) • Services available via the Internet are especially vulnerable to attack (ftp, http servers, DCOM) • Remote code execution • Attacker exploits service • Transfers program to victim • Runs program on victim

48. Attacks • Keystroke loggers • Allows attacker to capture keystrokes • Used to discover passwords and elevating privilege • rootkits – larger set of attack tools • bots – remotely controlled programs that perform denial of service attacks (robots) • botnets – a set of machines infected with bots

49. Attacks (continued) • Denial of service – bombarding a site with requests for service or malformed requests which prevent real users from accessing the service • Distributed denial of service attacks – a denial of service attack in which the requests for service come from many computers • Threats/blackmail

50. Attacks (continued) • guest accounts may be exploited to escalate privilege • Accounts with weak or no passwords may be exploited similarly • Worms – self-replicating programs which take advantage of network vulnerabilities to propagate themselves