Read-Copy Update P. E. McKenney, J. Appavoo, A. Kleen, O. Krieger, R. Russell, D. Saram, M. Soni

1. Read-Copy Update P. E. McKenney, J. Appavoo, A. Kleen, O. Krieger, R. Russell, D. Saram, M. Soni Ottawa Linux Symposium 2001 Presented by Bogdan Simion

2. Motivation • Locking can be expensive • Overhead of locking code • Cache bouncing • Linux uses locking to protect against infrequent destructive modifications • e.g., racy accesses to unloaded modules • Want to avoid locking expense for reads of data that are infrequently modified

3. Key Idea • Example: module unloading • Give ongoing operations a grace period to finish

4. Grace Periods • Starts when new operations see new state • e.g., remove pointer to module from a list • First phase of the update • No new references made once period starts • Extends until after all operations that started before the grace period finish • Operations with outstanding references finish safely • When period ends, system may cleanup • Second phase of the update, e.g., free module data

5. Grace Period Duration • Safe to end the grace period when all CPUs have finished prior operations • A non-preemptive operating system finishes all operations when it context switches • Thus, grace period ends after all CPUs have context switched at least once • Zero reference count deduced without using any shared data!

6. RCU So Far • RCU performs updates in two phases: • Update enough so new operations see new state but old operations can proceed using old state • Complete the update after the grace period • RCU works well when • Updates can be done in two phases • Operations still work with stale state • Destructive updates are infrequent • Let's look at an example of how it's used

7. Example: Reference Counted Search • Simple circular doubly linked-list • Compare a reference-counting locking algorithm taken from Linux with its read-copy-update equivalent

8. Reference Counted Search • For each algorithm: • search() • delete() • search(): returns a pointer to an element in the list given its addr, and ensures that element is not being freed up • delete(): arranges for the specified element to eventually be freed up • delete() may not be able to free the element immediately due to concurrent searches

9. Reference Counted Search

10. Reference-Counted Usage • Read-only and update (including delete) operation

11. RCU Search / Delete

12. Search / Delete Discussion • Searching scales perfectly • No locks – scales well • No cache line bouncing • Clear advantage over reference counting • Search can return stale data • There is a race between search and delete • Reference counting + locks does not have this problem • Delete is similar – global lock • Good speedups only if many more searches than del • kfree_rcu is neither trivial nor inexpensive

13. Read-Copy Deletion Scenario • To delete element B, the updater task acquires list lock to exclude other list manipulation, unlinks element B from the list and releases list lock

14. Read-Copy Deletion Scenario • The updater task passes a pointer to B to the kfree_rcu() primitive, which adds the memory to a list waiting to be freed. • Safe to return B to the freelist at the end of the grace period (when all pre-existing ops complete) List After Element B Returned to Freelist List After Grace Period

15. Implementing kfree_rcu • Basic idea:

16. Implementing kfree_rcu • Execute updater on each CPU:

17. Implementing kfree_rcu • Delay deletion until the end of the grace period: • wait_for_rcu() { • ... current->cpus_allowed = (1 << num_cpus) - 1; • while(true) { • current->cpus_allowed &= ~(1 << cpu_index()); • if (current->cpus_allowed == 0) break; schedule(); • } /* Grace period now over. Now it's safe to delete. */ ... • }

18. Implementing kfree_rcu • Delay deletion until the end of the grace period: • wait_for_rcu() { • ... current->cpus_allowed = (1 << num_cpus) - 1; • while(true) { • current->cpus_allowed &= ~(1 << cpu_index()); • if (current->cpus_allowed == 0) break; schedule(); • } /* Grace period now over. Now it's safe to delete. */ ... • } • Doesn't work with preemptible kernels. Why? • Can't be called from an interrupt handler or while a spin lock is held. Why? • Can be relatively slow. Why?

19. Deferring wait_for_rcu struct rcu_head { tq_struct task; }; • void* kmalloc_rcu(size_t size, int flags) { • rcu_head* ret = kmalloc(size + sizeof(*ret), flags); • return ret + 1; • } • void sync_and_destroy(void* head) { • wait_for_rcu(); • kfree(head); • } • kfree_rcu(void* obj) { • rcu_head* head = ((rcu_head*) obj) – 1; • head->task.routine = &sync_and_destroy; • head->task.data = head; • schedule_task(&head->task); • }

20. Deferring wait_for_rcu struct rcu_head { tq_struct task; }; • void* kmalloc_rcu(size_t size, int flags) { • rcu_head* ret = kmalloc(size + sizeof(*ret), flags); • return ret + 1; • } • void sync_and_destroy(void* head) { • wait_for_rcu(); • kfree(head); • } • kfree_rcu(void* obj) { • rcu_head* head = ((rcu_head*) obj) – 1; • head->task.routine = &sync_and_destroy; • head->task.data = head; • schedule_task(&head->task); • }

21. Deferring wait_for_rcu struct rcu_head { tq_struct task; }; • void* kmalloc_rcu(size_t size, int flags) { • rcu_head* ret = kmalloc(size + sizeof(*ret), flags); • return ret + 1; • } • void sync_and_destroy(void* head) { • wait_for_rcu(); • kfree(head); • } • kfree_rcu(void* obj) { • rcu_head* head = ((rcu_head*) obj) – 1; • head->task.routine = &sync_and_destroy; • head->task.data = head; • schedule_task(&head->task); • } Why is kmalloc_rcu necessary?

22. RCU Application: File Descriptors • Kernel maintains mapping of file descriptors to instances of struct file with an array • Expansion of the array is a destructive update: • Copies the old elements into a new array • Updates pointers and deletes the old array • RCU employed: • Phase 1: Create new arrays and update pointers • Phase 2: Delete the old arrays

23. RCU Performance: File Descriptors • Chat benchmark, 2.4.2 SMP Kernel • Why does R/W lock incur so much overhead?

24. RCU Performance Improvements • A number of improvements to the basic mechanism • Batch grace period measurements • wait_for_rcu is expensive • A single measurement satisfies multiple deferred free requests • Maintain per-CPU request lists • Faster grace period algorithm • See the paper for details

25. Comparing RCU to other Locking Algorithms • Data locking • Does not avoid reader locks • Also prone to deadlocks • Although list elements can be manipulated in parallel, searches cannot be done in parallel • Can be used to prevent stale reads in RCU • brlock • Effectively lock-free reads • Not clear how its performance differs from RCU • i.e., Can't brlock be used for the file descriptor arrays?

26. Conclusions • RCU is an effective approach for avoiding locking for read-mostly data structures • An elegant method for implicit reference counting • Main advantage: readers need not acquire locks, perform any atomic ops, write to shared memory or use barriers. • The destructive update is delayed until the grace period finishes – until all CPUs context switch (if non-preemptible) • Since 2001, it has been used in hundreds of places in the Linux kernel