Correctness of Gossip-Based Membership under Message Loss

1. Correctness of Gossip-Based Membership under Message Loss Maxim Gurevich Idit Keidar Technion

2. The Setting • Many nodes – n • 10,000s, 100,000s, 1,000,000s, … • Come and go • Churn • Fully connected network • Like the Internet • Every joining node knows some others • (Initial) Connectivity

3. Membership: Each node needs to know some live nodes • Each node has a view • Set of node ids • Supplied to the application • Constantly refreshed • Typical size – log n

4. Applications • Applications • Gossip-based algorithm • Unstructured overlay networks • Gathering statistics • Work best with random node sample • Gossip algorithms converge fast • Overlay networks are robust, good expanders • Statistics are accurate

5. Modeling Membership Views • Modeled as a directed graph w y u v

6. Modeling Protocols: Graph Transformations • View is used for maintenance • Example: push protocol w z u v

7. Desirable Properties? • Randomness • View should include random samples • Holy grail for samples: IID • Each sample uniformly distributed • Each sample independent of other samples • Avoid spatial dependencies among view entries • Avoid correlations between nodes • Good load balance among nodes

8. What About Churn? • Views should constantly evolve • Remove failed nodes, add joining ones • Views should evolve to IID from anystate • Minimize temporal dependencies • Dependence on the past should decay quickly • Useful for application requiring fresh samples

9. Global Markov Chain • A global state – all n views in the system • A protocol action – transition between global states • Global Markov Chain G u v u v

10. Defining Properties Formally • Small views • Bounded dout(u) • Load balance • Low variance of din(u) • From any starting state, eventually(In the stationary distribution of MC on G) • Uniformity • Pr(v  u.view) = Pr(w  u.view) • Spatial independence • Pr(v  u. view| y w. view) = Pr(v  u. view) • Perfect uniformity + spatial independence  load balance

11. Temporal Independence • Time to obtain views independent of the past • From an expected state • Refresh rate in the steady state • Would have been much longer had we considered starting from arbitrary state • O(n14) [Cooper09]

12. Existing Work: Practical Protocols Push protocol • Tolerates asynchrony, message loss • Studied only empirically  • Good load balance [Lpbcast, Jelasity et al 07] • Fast decay of temporal dependencies [Jelasity et al 07] • Induce spatial dependence  w w z z v v u u

13. Existing Work: Analysis w z Shuffle protocol • Analyzed theoretically [Allavena et al 05, Mahlmann et al 06] • Uniformity, load balance, spatial independence  • Weak bounds (worst case) on temporal independence  • Unrealistic assumptions – hard to implement  • Atomic actions with bi-directional communication • No message loss * u v

14. Our Contribution : Bridge This Gap • A practical protocol • Tolerates message loss, churn, failures • No complex bookkeeping for atomic actions • Formally prove the desirable properties • Including under message loss

15. Send & Forget Membership • The best of push and shuffle • Some view entries may be empty w u v

16. S&F: Message Loss • Message loss • Or no empty entries in v’s view w w u v u v

17. S&F: Compensating for Loss • Edges (view entries) disappear due to loss • Need to prevent views from emptying out • Keep the sent ids when too little ids in view • Push-like when views are too small w w u v u v

18. S&F: Advantages over Other Protocols • No bi-directional communication • No complex bookkeeping • Tolerates message loss • Simple • Without unrealistic assumptions • Amenable to formal analysis Easy to implement

19. Key Contribution: Analysis • Degree distribution • Closed-form approximation without loss • Degree Markov Chain with loss • Stationary distribution of MC on the global graph G • Uniformity • Spatial Independence • Temporal Independence • Hold even under (reasonable) message loss!

20. Degree Distribution without loss • In all reachable graphs: • dout(u) + 2din(u) = const • Better than in a random graph – indegree bounded • Uniform stationary distribution on reachable states in G • Combinatorial approximation of degree distribution • The fraction of reachable graphs with specified node degree • Ignoring dependencies among nodes

21. Degree Distribution without Loss: Results • Similar (better) to that of a random graph • Validated by a more accurate Markov model

22. Setting Degree Thresholds to Compensate for Loss • Note: dout(u) + 2din(u) = const invariant no longer holds – indegree not bounded

23. Key Contribution: Analysis • Degree distribution • Closed-form approximation without loss • Degree Markov Chain with loss • Stationary distribution of MC on the global graph G • Uniformity • Spatial Independence • Temporal Independence

24. Degree Markov Chain • Given loss rate, degree thresholds, and degree distributions • Iteratively compute the stationary distribution … outdegree 0 2 4 6 … … State corresponding to isolated node 0 Transitions without loss … 1 … indegree Transitions due to loss 2 … … 3 …

25. Results • Outdegree is bounded by the protocol • Decreases with increasing loss • Indegree is not bounded by the protocol • Still, its variance is low, even under loss • Typical overload at most 2x

26. Key Contribution: Analysis • Degree distribution • Closed-form approximation without loss • Degree Markov Chain with loss • Stationary distribution of MC on the global graph G • Uniformity • Spatial Independence • Temporal Independence

27. Uniformity • Simple! • Nodes are identical • Graphs where uv isomorphic to graphs where uw • Same probability in stationary distribution

28. Key Contribution: Analysis • Degree distribution • Closed-form approximation without loss • Degree Markov Chain with loss • Stationary distribution of MC on the global graph G • Uniformity • Spatial Independence • Temporal Independence

29. Decay of Spatial Dependencies w w • Assume initially > 2/3 independent  good expander • For uniform loss < 15%, dependencies decay faster than they are created … … u u v v u does not delete the sent ids

30. Decay of Spatial Dependencies: Results • 1 – 2loss ratefraction of view entries are independent • E.g., for loss rate of 3%  more than 90% of entries are independent

31. Key Contribution: Analysis • Degree distribution • Closed-form approximation without loss • Degree Markov Chain with loss • Stationary distribution of MC on the global graph G • Uniformity • Spatial Independence • Temporal Independence

32. Temporal Independence • Start from expected state • Uniform and spatially independent views • High “expected conductance” of G • Short mixing time • While staying in the “good” component

33. Temporal Independence: Results • Ids travel fast enough • Reach random nodes in O(log n)hops • Due to “sufficiently many” independent ids in views • Dependence on past views decays within O(log n  view size)time

34. Conclusions • Formalized the desired properties of a membership protocol • Send & Forget protocol • Simple for both implementation and analysis • Analysis under message loss • Load balance • Uniformity • Spatial Independence • Temporal Independence

35. Thank You