Speculative Execution in a Distributed File System

1. Speculative Execution in a Distributed File System - Sumanth Gelle

2. Outline • Introduction • Motivation: Speculation in NFS • Conditions For Success • Interface for Speculation • Implementing Speculation • Multi process Speculation • Using Speculation • Evaluation

3. Motivation • Why are distributed file systems slow(er)? • Sync n/w messages provide consistency • Sync disk writes provide safety • Sacrifice guarantees for speed • Can DFS can be safe, consistent and fast? • Yes! With OS support for speculative execution

4. Big Idea: Slow Way Big Idea: Speculator Client Server 1) Checkpoint RPC Req RPC Req 2) Speculate! Block! RPC Resp RPC Resp 3) Correct? No: restore process & re-execute Yes: discard ckpt. RPC Req RPC Resp • Guarantees without blocking I/O!

5. Motivation: Speculation in NFS • To close a file NFSv3 client needs to • Write RPC • Commit RPC • To open a file NFSv3 clinet needs to • Getattr RPC

6. Example Scenario

7. Conditions for Success • Operations are highly predictable • Conflicts are rare • Checkpoints are cheaper than network I/O • 52 µs for small process • Computers have resources to spare • Need memory and CPU cycles for speculation

8. An Interface for Speculation • To initiate speculative process calls create_ speculation. • This function returns the spec_id that uniquely identifies the particular speculation. • Based on the Speculation result the process executes • commit_speculation or fail_speculation.

9. Implementing Speculation • Process Checkpoint and Rollback • Copy-on-write fork of the currently running process. • On failure mark the process as failed in its task structure . • Check point process assumes the identity of failed process. • Speculation • Adds two data structures to the kernel • Speculation structure – created during create_speculation. Contains set of kernel objects that depend on the new speculation. • Undo log - associated with each kernel object that has speculative state • Sharing of a checkpoint for those speculations which do not change state. • Ensuring Correct Speculation Execution.

10. Spec Undo log Implementing Speculation 1) System call 2) Create speculation Time Process Checkpoint

11. Spec Undo log Speculation Success 1) System call 2) Create speculation 3) Commit speculation Time Process Checkpoint

12. Spec Undo log Speculation Failure 2)Create speculation 1) System call 3)Fail speculation Time Process Process Checkpoint

13. Ensuring Correctness • Spec processes often affect external state • Three ways to ensure correct execution • Block • Buffer • Propagate speculations (dependencies) • New system call Jump table. • sys_spec_deny – a function to block the calling process until the speculations are resolved.

14. Multi Process Speculation • Speculator’s strategy for IPC allows selected data structures within the kernel to have speculative state. • Speculator propagates dependencies from a process P to an object X whenever P modifies X and P depends on speculations that X does not. • Speculator propagates dependencies from an object X to a process P whenever P observes X’s state and X depends on speculations that P does not.

15. Spec 1 Spec 1 Spec 2 Multi-Process Speculation Checkpoint Checkpoint Checkpoint Checkpoint Checkpoint pid 8000 pid 8001 Chown-1 Chown-1 Write-1 Write-1 inode 3456

17. Using Speculation • Modifying a distributed file system to use Speculator typically involves changes to the client, server, and network protocol. • The client calls commit speculation if it correctly predicted the result of the RPC and fail speculation if it did not.

18. Example: NFSv3 Linux Client 1 Server Client 2 Modify B Write Commit Open B Getattr

19. Example: SpecNFS Client 1 Server Client 2 Write+Commit Modify B speculate Getattr Open B speculate Getattr Open B speculate

20. Problem: Mutating Operations Client 1 1. cat foo > bar Client 2 2. cat bar • bar depends on cat foo • What does client 2 view in bar?

21. Solution: Mutating Operations • Server determines speculation success/failure • State at server never speculative • Send server hypothesis speculation based on • List of speculations an operation depends on • Requires server to track failed speculations • Requires in-order processing of messages

22. Group Commit • Previously sequential ops now concurrent • Sync ops usually committed to disk • Speculator makes group commit possible Client Client Server Server write commit write commit

23. Putting it all Together: SpecNFS • Apply Speculator to an existing file system • Modified NFSv3 in Linux 2.4 kernel • Same RPCs issued (but many now asynchronous) • SpecNFS has same consistency, safety as NFS • Getattr, lookup, access speculate if data in cache • Create, mkdir, commit, etc. always speculate

24. Putting it all Together: BlueFS • Design a new file system for Speculator • Single copy semantics • Synchronous I/O • Each file, directory, etc. has version number • Incremented on each mutating op (e.g. on write) • Checked prior to all operations. • Many ops speculate and check version async

25. Apache Benchmark • SpecNFS up to 14 times faster

26. The Cost of Rollback • All files out of date SpecNFS up to 11x faster

27. Conclusion • Speculator greatly improves performance of existing distributed file systems • Speculator enables new file systems to be safe, consistent and fast

28. References • http://www.eecs.umich.edu/~pmchen/papers/nightingale05.slides.ppt • Speculative Execution in a Distributed File System EDMUND B. NIGHTINGALE, PETER M. CHEN, and JASON FLINN University of Michigan