Isis 2 guarantees

1. Isis2 guarantees Did you understand how Isis2 ordering and durability work?

2. Group multicast • In the Isis2 programming manual, you’ll see that multicast primatives like g.Send allow you to specify the destinations for a message: you can call g.Send(rcode, List<Address> dests, args) • True or false: g.Send(rcode, args) behaves exactly the same as g.Send(rcode, g.GetView().members, args)? Time  S P Q g.GetView().members = {S,P,Q}. Here S is the sender.

3. Group Multicast • False.When you use g.Send(rcode, args) to send a message to the entire group, Isis2 uses a lock to synchronize sending with the group membership • This ensures that your message goes to all members in some view of the group (the sender gets a copy too) • In the second approach , a race could arise if some member was joining the group just as you do the send. g.GetView().members = {S,P,Q}. Here S is the sender. S P Q S P Q R The problem is that the g.Send() really is issued after the GetView() was evaluated. During that split second, R might have joined!

4. Fault-Tolerance for Multicasts • Suppose that you use g.Send() to send a message and some member of the group, process p, receives it and prints the message to the local console (without doing any other Isis2 system calls first). • True or false: If g was in view {p,q,r} when p received the message, q and r will receive it too.

5. Fault-Tolerance for Multicasts • False. • g.Send() is an optimistic delivery multicast • It doesn’t wait to be certain that every member has received a copy. Thus there are patterns of failures in which the sender could crash, and in which process p also crashes simultaneously, where q and r might never receive the message S sends a message, but then crashes S P Q R P receives it. Then P crashes too Q R … Q and R do not receive it … new view reports the failures

6. Fault-Tolerance for Multicasts • True or False: In Isis2 the only way to prevent the scenario seen here is to use the more costly g.SafeSend() S P Q R Q R

7. Fault-Tolerance for Multicasts • As worded, False: The SafeSend protocol has a pessimistic mode of delivery: before any copy is delivered, every member logs the message. But you can also avoid the risk of an outcome like this one by using an optimistic protocol but calling the g.Flush() method before “talking” to the outside world. S P Q R Q R

8. Fault-Tolerance for Multicasts • True or False: In Isis2 there is a way to “mask” the optimistic delivery behavior so that no external observer could ever know it arose. S P Q R Q R

9. Fault-Tolerance for Multicasts • True. If the recipient of a g.Send() calls g.Flush() before printing to the console, an outside observer will never see a non-atomic delivery. P calls g.Flush() before printing anything. It has received the message, but until g.Flush finishes, doesn’t act upon it. S P Q R Q R g.Flush() delays until delivery is finished

10. Ordering for Multicasts • Suppose S uses g.Send to send M0, then M1 True or false: Every receiver gets M0 before M1 M0 M1 S P Q R

11. Ordering for Multicasts • True.g.Send() preserves the “FIFO” or sender ordering. FIFO is short for “first in, first out” but in Isis2 means “first sent, first delivered” M0 M1 S P Q R

12. Ordering for Multicasts • True or false. Suppose that only the leader (0’th ranked member) sends multicasts in a group. Then with g.Send(), every member sees every multicast in the identical order. M0 M1 S P Q R M2 M3 P Q R

13. Ordering for Multicasts • True. After the old leader fails and the new view is reported, we won’t see further messages from the old leader. Thus the ordering guarantee of Send (FIFO) is all we need, on a “view by view” basis. The group will never receive additional multicasts from S after S leaves the view M0 M1 S P Q R M2 M3 P Q R

14. Ordering for Multicasts • Suppose S uses g.Send to send M0, and then much later in real-time, R sends M1 True or false: Every receiver gets M0 before M1 M0 S P Q R M1

15. Ordering for Multicasts • False. Message loss or scheduling delays could affect delivery ordering. g.Send() only promises to maintain the ordering for messages that had the identical sender. These messages had different senders. P gets M1, then M0 M0 S P Q R M1 All others get M0 before M1

16. g.OrderedSend • Suppose S uses g.OrderedSend to send M0, and then much later in real-time, R sends M1 True or false: Every receiver gets M0 before M1 M0 S P Q R M1

17. g.OrderedSend • False. With OrderedSend, every process gets messages in the identical order. But it might not be the real-time order in which messages were sent. So every process gets M0 M1, or every process gets M1 M0. Every process gets M0 M1 M0 M2 S P Q R M1 M3 Every process gets M3 M2 even though M2 was sent first

18. g.OrderedSend • With OrderedSend, every process gets messages in the order they were sent, assuming they were sent by some single source. M0 M2 S P Q R M1 M3

19. g.OrderedSend • True. In this example, M0 will always be delivered before M2. And M1 will always be delivered before M3. OrderedSend respects the FIFO ordering, but might not follow a real-time (clock) ordering if the senders are different (like for M2 and M3) M0 M2 S P Q R M1 M3

20. Fault-Tolerance for g.OrderedSend • Suppose that you use g.OrderedSend() to send a message and some member of the group, process p, receives it and prints the message to the local console (without doing any other Isis2 system calls first). • True or false: If g was in view {p,q,r} when p received the message, q and r will receive it too.

21. Fault-Tolerance for Multicasts • False. • g.OrderedSend() is an optimistic delivery multicast • It doesn’t wait to be certain that every member has received a copy. Thus there are patterns of failures in which the sender could crash, and in which process p also crashes simultaneously, where q and r might never receive the message S sends a message, but then crashes S P Q R P receives it. Then P crashes too Q R … Q and R do not receive it … new view reports the failures

22. Fault-Tolerance for g.OrderedSend • Suppose that you use g.OrderedSend() to send a message and some member of the group, process p, receives it, but calls g.Flush() before any external action is taken. • Then no external observer will ever observe a non-atomic delivery.

23. Fault-Tolerance for g.OrderedSend • True. If the recipient of a g.OrderedSend() calls g.Flush() before printing to the console, an outside observer will never see a non-atomic delivery. P calls g.Flush() before printing anything. It has received the message, but until g.Flush finishes, doesn’t act upon it. S P Q R Q R g.Flush() delays until delivery is finished

24. Ordering for g.OrderedSend • Like g.Send(), g.OrderedSend() will always select an ordering that preserves the sender (FIFO) order. g.OrderedSend() goes beyond what g.Send() offers by also ensuring that even with multiple concurrent senders, ever process sees messages in the same order.

25. Ordering for g.OrderedSend • True. All Isis2 multicast primitives maintain the sender order, even if a message is lost and must be retransmitted. Due to a message loss, delivering M1 immediately would violate FIFO order. Isis2 delays M1 until after M0 is retransmitted. M0 M1 S P Q R

26. Ordering for g.SafeSend • The ordering provided by g.SafeSend() is actually identical to that of g.OrderedSend().

27. Ordering for g.SafeSend • True. The ordering provided by g.SafeSend() is actually identical to that of g.OrderedSend(). • Both provide FIFO ordering for the sender. And both place concurrent multicasts from different senders into some fixed order, which cannot easily be predicted and might not be the real-time ordering, and then deliver in that fixed order.

28. g.OrderedSend vs. g.Send • g.OrderedSend is always slower than g.Send, but faster than g.SafeSend

29. g.OrderedSend vs. g.Send • False. g.OrderedSend adapts to the situation. It will be as fast as g.Send if there is just one sender in the group. But it will run slower if there are multiple concurrent senders. • Even so, g.OrderedSend is always faster than g.SafeSend, which logs messages to a disk file before delivering them.

30. g.OrderedSend+g.Flush • If g.OrderedSend() is used, and g.Flush() is called prior to interacting with external entities, the result is identical to g.SafeSend()

31. g.OrderedSend+g.Flush • This issue is very subtle and goes beyond what was covered in the video. • [Case A: true]In a soft-state (first tier) service this statement would be true. In such a situation, you cannot distinguish between g.OrderedSend+g.Flush and g.SafeSend, provided that g.Flush is called prior to interacting with the outside world.

32. g.OrderedSend+g.Flush • [Case B: false]With a durable service things get more complex. In the event of a total failure where all group members crash concurrently, g.SafeSend maintains a log and will replay all messages after failure. g.OrderedSend doesn’t log its messages, hence no replay occurs. • This matters when a group is used as the “front end” to a durable replicated database. In such cases we need g.SafeSend.

33. g.Flush(k) • True or false: Calling g.Flush(k) causes Isis2: • To form a list of known active multicasts, namely messages that have been sent or received but that are still active in the system. • Check how many recipients each of the multicasts has. • And then wait until there are at least k recipients that have acknowledge receipt of each. • Thus, after g.Flush(k) returns, a multicast can’t vanish from the system unless k concurrent failures occur, and they would need to include every one of the processes that acknowledged receipt.

34. g.Flush(k) • True. Without a value specified for k, g.Flush() waits until every group member has received any pending multicasts. With k>1 specified, g.Flush() waits until k group members have acknowledged each pending multicast.

35. g.Flush(k) • True or False: it makes no sense at all to call g.Flush(1).

36. g.Flush(k) • True. The value of k tells Isis2 how many “acknowledged copies” must exist of every pending multicast known to the process P at which the call to g.Flush(k) was done. • But any multicast known to P clearly has k=1, since P itself has a copy. Thus k=1 is meaningless. We normally use k=2 or k=3.

37. Let’s try a harder case • Mx is an update and YT means “your turn”. We want them processed in the identical order. The idea is that S “owns” some variable X and sends updates M0 and M1. Then R takes over and sends updates. M0 M1 YT S P Q R M2 M3 YT

38. Let’s try a harder case • Assume the service is running in the soft-state tier. Will this algorithm work properly if OrderedSend() is used for the Mx and YT multicasts? M0 M1 YT S P Q R M2 M3 YT

39. Let’s try a harder case • … Yes.OrderedSend() respects FIFO order, and as long as R waits to receive YT before sending M2 M3, every process will receive every multicast in the identical order. M0 M1 YT S P Q R M2 M3 YT

40. Let’s try a harder case • If FIFO order is what we need, will this code be correct if g.Send() is used for the Mx and YT multicasts? M0 M1 YT S P Q R M2 M3 YT

41. Let’s try a harder case • … No. Consider this example. g.Send() could reach R before it reaches P and Q. This creates a race condition, and M2 and M3 might be received out of order. M0 M1 YT S P Q R M2 M3 YT

42. Let’s try a harder case • …What if we use g.Send(), but insert a call to g.Flush() before sending the YT multicasts? M0 M1 YT flush S P Q R flush M2 M3 YT

43. Let’s try a harder case • … yes. This version works because M1 M2 are already stable before the YT multicast is sent. But by waiting before sending YT it could be slow… the flush waits for acknowledgements M0 M1 YT flush S P Q R flush M2 M3 YT

44. Let’s try a harder case • …What if we use g.Send(), but insert a call to g.Flushafterreceivingthe YT multicasts? M0 M1 YT flush S P Q R flush M2 M3 YT

45. Let’s try a harder case • … yes! The g.Flush() waits for unstable multicasts to finish, and we know that R has seen M0 M1 YT. So those finish, and every process receives M2 M3 after M0 M1 M0 M1 YT flush S P Q R flush M2 M3 YT

46. Another hard case • Suppose a group manages red objects and blue objects, and they have nothing to do with one-another. Moreover, suppose there is a single member that sends red updates, and a different single member that sends blue updates. • We can use g.Send() for the updates • We should use g.OrderedSend() for updates

47. Another hard case • Suppose a group manages red objects and blue objects, and they have nothing to do with one-another. Moreover, suppose there is a single member that sends red updates, and a different single member that sends blue updates. • We can use g.Send() for the updates • We should use g.OrderedSend() for updates g.Send() will have one order for red updates (based on the FIFO send order) and another one for blue. Thus updates may interleave in different orders at different members, but this won’t matter because the group treats red and blue objects as being independent.

48. Another hard case • Same setup, but now suppose we also need to do a green operation that looks at both red and blue objects and computes some sort of aggregate statistic over the full set. • We can still use g.Send() for the updates, but should use g.OrderedSend() for the green operations. • We should use g.OrderedSend() for everything

49. Another hard case • Same setup, but now suppose we also need to do a green operation that looks at both red and blue objects and computes some sort of aggregate statistic over the full set. • We can still use g.Send() for the updates, but should use g.OrderedSend() for the green operations. • We should use g.OrderedSend() for everything This won’t work because OrderedSend isn’t ordered relative to Send. Thus the green operations could “see” an inconsistent state in which the red and blue updates are only partially completed at some members If every operation uses OrderedSend, the green operations see some well defined single state consistent across the entire group.

50. Which is the most efficient? • OrderedSend() is slower than Send() in this setting • Flush() by the sender is faster than by a receiver. • The sender is first to learn that multicasts are stable. • Receiver needs to wait until the sender reports stability. • Best solution: call Flush before sending YT • Mystery: Will the new sender delay waiting to learn about stability of the prior YT? • Yes if the code that sends the multicasts is slow enough, but perhaps not if the code is very fast • May need to experiment to find out!