Interfacing with Hardware

1. Interfacing with Hardware

2. I/O Devices • Two primary aspects of computer system • Processing (CPU + Memory) • Input/Output • Role of Operating System • Provide a consistent interface • Simplify access to hardware devices • Implement mechanisms for interacting with devices • Allocate and manage resources • Protection • Fairness • Obtain Efficient performance • Understand performance characteristics of device • Develop policies

3. I/O Subsystem User Process Kernel Kernel I/O Subsystem Harddisk Mouse GPU SCSI Bus Keyboard PCI Bus Device Drivers Software Hardware Harddisk Mouse GPU Device Controllers SCSI Bus Keyboard PCI Bus Harddisk Mouse GPU SCSI Bus Keyboard PCI Bus Devices

4. User View of I/O • User Processes cannot have direct access to devices • Manage resources fairly • Protects data from access-control violations • Protect system from crashing • OS exports higher level functions • User process performs system calls (e.g. read() and write()) • Blocking vs. Nonblocking I/O • Blocking: Suspends execution of process until I/O completes • Simple and easy to understand • Inefficient • Nonblocking: Returns from system calls immediately • Process is notified when I/O completes • Complex but better performance

5. User View: Types of devices • Character-stream • Transfer one byte (character) at a time • Interface: get() or put() • Implemented as restricted forms of read()/write() • Example: keyboard, mouse, modem, console • Block • Transfer blocks of bytes as a unit (defined by hardware) • Interface: read() and write() • Random access: seek() specifies which bytes to transfer next • Example: Disks and tapes

6. Kernel I/O Subsystem • I/O scheduled from pool of requests • Requests rearranged to optimize efficiency • Example: Disk requests are reordered to reduce head seeks • Buffering • Deal with different transfer rates • Adjustable transfer sizes • Fragmentation and reassembly • Copy Semantics • Can calling process reuse buffer immediately? • Caching: Avoid device accesses as much as possible • I/O is SLOW • Block devices can read ahead

7. Device Drivers • Encapsulate details of device • Wide variety of I/O devices (different manufacturers and features) • Kernel I/O subsystem not aware of hardware details • Load at boot time or on demand • IOCTLs: Special UNIX system call (I/O control) • Alternative to adding a new system call • Interface between user processes and device drivers • Device specific operation • Looks like a system call, but also takes a file descriptor argument • Why?

8. Device Driver: Device Configuration • Interactions directly with DeviceController • Special Instructions • Valid only in kernel mode • X86: In/Out instructions • No longer popular • Memory-mapped • Read and write operations in special memory regions • How are memory operations delivered to controller? • OS protects interfaces by not mapping memory into user processes • Some devices can map subsets of I/O space to processes • Buffer queues (i.e. network cards)

9. Interacting with Device Controllers • How to know when I/O is complete? • Polling • Disadvantage: Busy Waiting • CPU cycles wasted when I/O is slow • Often need to be careful with timing • Interrupts • Goal: Enable asynchronous events • Device signals CPU by asserting interrupt request line • CPU automatically jumps to Interrupt Service Routine • Interrupt vector: Table of ISR addresses • Indexed by interrupt number • Lower priority interrupts postponed until higher priority finished • Interrupts can nest • Disadvantage: Interrupts “interrupt” processing • Interrupt storms

10. Device Driver: Data transfer • Programmed I/O (PIO) • Initiate operation and read in every byte/word of data • Direct Memory Access (DMA) • Offload data xfer work to special-purpose processor • CPU configures DMA transfer • Writes DMA command block into main memory • Target addresses and xfer sizes • Give command block address to DMA engine • DMA engine xfers data from device to memory specified in command block • DMA engine raises interrupt when entire xfer is complete • Virtual or Physical address?

11. Interrupt Descriptor Table (IDT) • The IDT is a segment of memory containing pointers to irq handlers • For now think of them as function pointers • Location of the IDT configured using a special HW register(IDTR) • HW IRQ specifies which entry to use, called automatically by CPU

12. IDT Descriptors

13. Calling the IRQ handler

14. Interrupt Context (32 bit)

15. Linux Interrupt handling • All IRQ vectors mapped to a single low level handler • HW vectors interrupt to CPU and IDT entry • IDT entry points to low level (ASM) code • ASM code calls do_irq() • do_irq()calls handle_irq_event() • handle_irq_event()calls device driver

16. handle_IRQ_event() fastcallinthandle_IRQ_event(unsigned intirq, structpt_regs *regs, structirqaction *action) { int ret, retval = 0, status = 0; if (!(action->flags & SA_INTERRUPT)) local_irq_enable(); do { ret = action->handler(irq, action->dev_id, regs); if (ret == IRQ_HANDLED) status |= action->flags; retval |= ret; action = action->next; } while (action); if (status & SA_SAMPLE_RANDOM) add_interrupt_randomness(irq); local_irq_disable(); return retval; }

17. Exceptions • Events generated by the CPU • Page fault, Divide by zero, invalid instruction, etc • Full list in the CPU architecture manuals • First 32 IRQ vectors in IDT • Generally its an “error” or “exception” encountered during CPU instruction execution • IDT is referenced directly by the CPU • Completely internalized by HW

18. External Interrupts • Events triggered by devices connected to the system • Network packet arrivals, disk operation completion, timer updates, etc • Can be mapped to any IRQ vector above the exceptions • Vectors: 32-255 • External because they happen outside the CPU • External logic signals CPU and notifies it which handler to execute • Managed by Interrupt Controller

19. Interrupt Controllers • Translate device IRQ signals to CPU IRQ vectors • Interrupt controller maps devices to vectors • Two x86 controller classes • Legacy: 8259 PIC • Connected to a set of default I/O ports on CPU • Modern: APIC + IOAPIC • Memory mapped into each CPUs physical memory

20. 8259 PIC • Allows the mapping of 8 IRQ pins (from devices) to 8 separate vectors (to CPU) • Assumes continuous numbering • Assign the PIC a vector offset, • Each pin index is added to that offset to calculate the vector

21. 8259 Cont’d • 1 PIC only supports 8 device lines • Often more than 8 devices in a system • Solution: Add more PICs • But x86 CPUs only have 1 INTR input pin • X86 Solution: • Chain the PICs together (master and slave) • Slave PIC is attached to the 2nd IRQ pin of the master • CPU interfaces directly with master

22. PC architecture defines common default devices to each PIC input pin • Initially PIC vector offsets set to 32, just above exceptions

23. APIC • How to support multiple CPUs • You only want one CPU to receive an interrupt • SMP required a special solution • APIC + IOAPIC • Separates the responsibility of the Interrupt Controller into two components • APIC = Interfaces with CPU • IOAPIC = Interfaces with devices

24. APIC • Each CPU has its own local APIC • Tracks interrupts bound for its assigned CPU • Modern CPUs include APIC on the CPU die • APIC interfaces with CPUs interrupt pins to invoke correct IDT vector • But it does other things as well • Timer – APIC has its own embedded timer device • Allows each CPU to have its own timer • Inter-Processor Interrupt – Allows cross CPU communication • 1 CPU can send an interrupt to another one

25. Managing the APIC • Best to think of the APIC as a special external device • Exposed to OS as a set of (32 bit) device registers • See Intel Manual Vol 3, Chapter 10 • Common device mechanism • Each CPU core has its own device state • Mostly Read-Only representation of current/pending IRQs • Accessed via memory mapped I/O • 4KB region located at address 0xFEE0000000 • Each register located at fixed position within page • Registers accessed via 4 byte memory loads/stores • More recent versions allow MSR based accesses

26. ICC bus • APICs and IOAPICs share a common communication bus • ICC bus: Interrupt Controller Communication Bus • Handles routing of interrupts to the correct APIC • Interrupts orginate from either IOAPICs or other APICs

27. Interprocessor Interrupts (IPIs) • Mechanism for CPU cores to send signals to each other • Source APIC sends interrupt vector to destination APIC through ICC bus • Any core can send any vector to any other core • Accomplished by writing commands to special APIC register • ICR: Interrupt Command Register • Common Linux IPIs • CALL_FUNCTION_VECTOR (vector 251) • Calls an abitrary function at destination • RESCHEDULE_VECTOR (vector 252) • Forces destination core to invoke the scheduler to reschedule tasks • INVALIDATE_TLB_VECTOR (vector 253) • Broadcast to all cores to invalidate TLB entries • More details when we talk about paging

28. Interrupt Command Register Destination APIC(s) Interrupt Vector

29. IOAPIC • Connects devices to ICC bus • Translates device IRQ pins to CPU vectors • But now must also select destination APIC • Typically has 24 I/O Redirection Table Registers • Specifies vector # to send to APIC • Specifies which APIC (or group of APICS) can accept the IRQ • Several methods of specifying APIC addresses • Allows masking of IRQs from a device

30. Device raises INTR signal on Pin 3 Linux Kernel routes IRQ to handler function in device drvier I/O Device Device Driver APIC IO-APIC APIC IO-APIC Translates pin signal to IRQ Vector # and destination APIC ID APIC CPU stops execution and vectors through IDT to software IRQ handler APIC accepts IRQ to deliver to CPU

31. Interrupt Vectors

32. IRQ Example

33. Interrupt Handlers • A function the kernel runs in response to interrupt • More than one handler can exist per IRQ • Must run quickly. • Resume execution of interrupted code. • How to deal with high work interrupts?

34. Interrupt Context Not associated with a process. Cannot sleep: no task to reschedule. current macro points to interrupted process. Shares kernel stack of interrupted process. Be very frugal in stack usage.

35. Registering a Handler request_irq() Register an interrupt handler on a given line. free_irq() Unregister a given interrupt handler. Disable interrupt line if all handlers unregistered.

36. Registering a Handler int request_irq(unsigned int irq, irqreturn_t (*handler)(int, void *, struct pt_regs *), unsigned long irqflags, const char * devname, void *dev_id) irqflaqs = SA_INTERRUPT | SA_SAMPLE_RANDOM | SA_SHIRQ

37. Top and Bottom Halves Top Half • The “interrupt handler” • Disables lower priority interrupts • Runs in interrupt context, not process context. • Can’t sleep or block • Acknowledges receipt of interrupt. • Schedules bottom half to run later. • Re-enables Interrupts Bottom Half • Runs in process context with interrupts enabled • Performs most work required • Can sleep or block • Eg. copies network data to memory buffers

38. Bottom Halves • Three ways to defer work • SoftIRQs – Foundation for most bottom halves • Tasklets • Work Queues • ksoftirqd – Kernel thread to handle SoftIRQs • SoftIRQs may occur at high frequencies • Queues SoftIRQs and handles them in order • One thread per processor. • Runs at lowest priority (nice +19).

39. Timer Interrupt • Executed HZ times a second. • #define HZ 1000 /* <asm/param.h> */ • Called the tick rate. • Time between two interrupts is a tick. • Driven by Timer device • Several possible timer devices on modern HW • Interrupt handler responsibilities • Updating uptime, system time, kernel stats. • Rescheduling if current has exhausted time slice. • Balancing scheduler runqueues. • Running dynamic timers.