1. Fault Tolerance Chapter 7

2. Basic Concepts • Dependability Includes • Availability • Reliability • Safety • Maintainability

3. Failure Models • Different types of failures.

4. Failure Masking by Redundancy • Triple modular redundancy.

5. Flat Groups versus Hierarchical Groups • Communication in a flat group. • Communication in a simple hierarchical group

6. Agreement in Faulty Systems (1) • The Byzantine generals problem for 3 loyal generals and1 traitor. • The generals announce their troop strengths (in units of 1 kilosoldiers). • The vectors that each general assembles based on (a) • The vectors that each general receives in step 3.

7. Agreement in Faulty Systems (2) • The same as in previous slide, except now with 2 loyal generals and one traitor.

8. Lost Request MessagesServer Crashes (1) • A server in client-server communication • Normal case • Crash after execution • Crash before execution

9. Server Crashes (2) • Different combinations of client and server strategies in the presence of server crashes.

10. Basic Reliable-Multicasting Schemes • A simple solution to reliable multicasting when all receivers are known and are assumed not to fail • Message transmission • Reporting feedback

11. Nonhierarchical Feedback Control • Several receivers have scheduled a request for retransmission, but the first retransmission request leads to the suppression of others.

12. Hierarchical Feedback Control • The essence of hierarchical reliable multicasting. • Each local coordinator forwards the message to its children. • A local coordinator handles retransmission requests.

13. Virtual Synchrony (1) • The logical organization of a distributed system to distinguish between message receipt and message delivery

14. Virtual Synchrony (2) • The principle of virtual synchronous multicast.

15. Message Ordering (1) • Three communicating processes in the same group. The ordering of events per process is shown along the vertical axis.

16. Message Ordering (2) • Four processes in the same group with two different senders, and a possible delivery order of messages under FIFO-ordered multicasting

17. Implementing Virtual Synchrony (1) • Six different versions of virtually synchronous reliable multicasting.

18. Implementing Virtual Synchrony (2) • Process 4 notices that process 7 has crashed, sends a view change • Process 6 sends out all its unstable messages, followed by a flush message • Process 6 installs the new view when it has received a flush message from everyone else

19. Two-Phase Commit (1) • The finite state machine for the coordinator in 2PC. • The finite state machine for a participant.

20. Two-Phase Commit (2) • Actions taken by a participant P when residing in state READY and having contacted another participant Q.

21. Two-Phase Commit (3) actions by coordinator: while START _2PC to local log;multicast VOTE_REQUEST to all participants;while not all votes have been collected { wait for any incoming vote; if timeout { while GLOBAL_ABORT to local log; multicast GLOBAL_ABORT to all participants; exit; } record vote;}if all participants sent VOTE_COMMIT and coordinator votes COMMIT{ write GLOBAL_COMMIT to local log; multicast GLOBAL_COMMIT to all participants;} else { write GLOBAL_ABORT to local log; multicast GLOBAL_ABORT to all participants;} • Outline of the steps taken by the coordinator in a two phase commit protocol

22. Two-Phase Commit (4) actions by participant: write INIT to local log;wait for VOTE_REQUEST from coordinator;if timeout { write VOTE_ABORT to local log; exit;}if participant votes COMMIT { write VOTE_COMMIT to local log; send VOTE_COMMIT to coordinator; wait for DECISION from coordinator; if timeout { multicast DECISION_REQUEST to other participants; wait until DECISION is received; /* remain blocked */ write DECISION to local log; } if DECISION == GLOBAL_COMMIT write GLOBAL_COMMIT to local log; else if DECISION == GLOBAL_ABORT write GLOBAL_ABORT to local log;} else { write VOTE_ABORT to local log; send VOTE ABORT to coordinator;} • Steps taken by participant process in 2PC.

23. Two-Phase Commit (5) actions for handling decision requests: /* executed by separate thread */ while true { wait until any incoming DECISION_REQUEST is received; /* remain blocked */ read most recently recorded STATE from the local log; if STATE == GLOBAL_COMMIT send GLOBAL_COMMIT to requesting participant; else if STATE == INIT or STATE == GLOBAL_ABORT send GLOBAL_ABORT to requesting participant; else skip; /* participant remains blocked */ • Steps taken for handling incoming decision requests.

24. Three-Phase Commit • Finite state machine for the coordinator in 3PC • Finite state machine for a participant

25. Recovery Stable Storage • Stable Storage • Crash after drive 1 is updated • Bad spot

26. Checkpointing • A recovery line.

27. Independent Checkpointing • The domino effect.

28. Message Logging • Incorrect replay of messages after recovery, leading to an orphan process.