Rapid Development of High Performance Servers

1. Rapid Development of High Performance Servers Khaled ElMeleegy Alan Cox Willy Zwaenepoel

2. Goals • Maximize server utilization and decrease its latency. • Server process doesn’t block! • Robust under heavy loads. • Relatively easy to develop.

3. Lazy threaded servers Key Idea • Get the best of the two worlds. Multi-threaded servers Event driven servers Programming level: Multi-Threaded Event Driven Implementation level: Multi-Threaded Event Driven

4. 1-KQueue Background • Kernel event notification mechanism. • Supported by FreeBSD Kernel. • Events queued in a kernel queue, and delivered to user by kernel calls. • Events usually correspond to AIO completions, or socket reads/writes.

5. 2-AIO • Asynchronous I/O mainly used for disk reads/writes. • Issue disk request and return immediately without blocking for results. • On completion an event is sent to KQueue.

6. 3-Servers Architectures:a- Multi-threaded Servers • Kernel thread per request executing the request handler. • A thread blocking on I/O doesn’t block the server. • Ex: Apache, MySql. Request Handler A B Function Calls C

7. Multi-threaded Servers (Cont.) • Advantages: + Relatively easy to develop. • Disadvantages: - Not good performance. • Thread scheduling. • Context switching. - Doesn’t scale. • Under heavy load , many threads would be created exhausting kernel’s memory, crashing the server.

8. b- Event Driven Servers • Single thread (event loop). • AIO operations. • I/O completions / Requests (events) put in Event Queue. • Continuations. • Ex:Flash web server (slightly different architecture). + High performance. + Scalable/Robust. - Hard to write. Event Loop Events From Kernel A1 A2 B2 B1 Event Queue (KQueue) C2 C1 “Blocking” operation

9. Similar design to event driven but no continuations. Threads yield control on fixed points (I/O). If request handler doesn’t “block” Run handler to completion then use same thread to handle next event. Else Suspend current thread, create another user thread to handle next event. Lazy Threading

10. IO_Wrapper(..) { uc = save_user_context(..); /* save the current user thread context to be able to resume it later */ aio_func(..); /* non-blocking I/O call */ stack=allocate_stack(); switch_thread(event_loop,stack); /* suspend current thread and create a new thread running the event loop using the newly allocated stack */ . . } Wrappers around “blocking” operations to create the blocking illusion. Uses AIO. Saves current stack instead of unwinding it. Creates a user thread by jumping to the event loop and using a new stack. Wrappers / Thread Suspension

11. Event_loop() { . . . e = get_event(..); /* read new event from the kernel’s kqueue */ if e is completion of aio_func(..) run_thread(uc); /* dispose current thread and resume the IO_Wrapper from where it has stopped */ . . } Restores registers and jumps back to wrapper as if blocking has finished. Current thread is disposed as it’s useless. Wrapper’s thread eventually returns to the event loop after handling its request. Thread Resumption

12. ServLib • Library for fast server development. • Provides an infrastructure for building lazy-threaded servers. • Hides all the lazy threading details from the server developer. • Used to upgrade existing servers, or build new servers.

13. How to use ServLib • Extract the event handler code from the old server/some library or write your own. • Initialize ServLib with : • request handler function. • listening port number. • Change all blocking calls in the request handler to use the wrappers exported by ServLib.

14. Experimentation • Implementing then comparing the performance of 3 versions of a database server (from sqlite library). • And of a similar 3 versions of a web server (from mini_httpd). • A multi-threaded version, • An event driven version and • A lazy threaded version. • Use FreeBSD 4.6

15. Summary Multi-Threaded Event Driven Lazy-Threaded Scalable No Yes Yes High Performance No Yes Yes “Easy” to build Yes No Yes >1 thread Yes(kernel) No Yes(user) Thread / request? Yes No No Arbitrary thread yield Yes N/A No

16. Questions?