Gossip Protocols from Epidemic Algorithms for Replicated Database Maintenance , Alan Demers et al

1. Gossip Protocolsfrom Epidemic Algorithms for Replicated Database Maintenance, Alan Demers et al Russell Greenspan CS523 Spring, 2006

2. Gossip Protocols • Useful for routing information through an unreliable network • Great for replicating database updates from one site to many • Introduce randomness leading to “probabilistic” reliability

3. Epidemic Algorithms for Replicated Database Maintenance, Alan Demers et al • Gossip protocols in replicated domain name service on Clearinghouse Servers for Xerox Corporate Internet • Direct Mail • Anti-Entropy • Rumor Mongering • Demers, A., Greene, D., Hauser, C., Irish, W., Larson, J., Shenker, S., Sturgis, H., Swinehart, D., and Terry, D. 1987. Epidemic algorithms for replicated database maintenance. In Proceedings of the Sixth Annual ACM Symposium on Principles of Distributed Computing (Vancouver, British Columbia, Canada, August 10 - 12, 1987). F. B. Schneider, Ed. PODC '87. ACM Press, New York, NY, 1-12. DOI= http://doi.acm.org/10.1145/41840.41841

4. Direct Mail(What not to do!) • Timestamp each operation • Notify all sites of every operation when it occurs • Sending pseudocodeforeach (site s in Sites) send(to:s, item:i); • Receiving pseudocodeif (localItem.timeStamp < i.timeStamp) localItem.value = i.value; • Problems • No guarantee of message delivery • Unreachable destinations • Incorrect destination list • Generates n messages (1 per site s in Sites) • Each message traverses the entire network between source and destination

5. Anti-Entropy • Based on epidemic theory • Single infected site eventually infects entire population of susceptible sites • In database replication, infected site is the one with the latest update, susceptible sites are those needing the update • Total time to update population is proportional to log of population size • Periodically sync entire copy of DB with one or more remote copies • Pseudocode • forsome (site s in Sites) resolveDifferences(localDB, s);

6. Anti-EntropyResolving Differences • Push • If local item is more recent, tell remote copy to update • Site susceptible at time t remains susceptible at time t + 1 if no infected site contacted it at time t + 1 • Good when most sites need the update • Pull • Ask remote copy if its copy is more recent; if so, update local copy • Site susceptible at time t remains susceptible at time t + 1 if it did not contact an infected site at time t + 1 • Good when most sites are already update-to-date • Push-Pull • Do both:if (localItem.timeStamp < i.timeStamp) localItem.value = i.value; //pullelse if (localItem.timeStamp > i.timeStamp) i.value = localItem.value; //push

7. Anti-EntropyPerformance Enhancements • Compare DB checksums (recomputed after each update) before exchanging full copy of the DB • Only exchange DB if checksums differ • Need to use time window that allows updates to propagate before checksums are compared • Keep inverted index by timestamp and only exchange recent updates

8. Rumor Mongering • Like a game of network-telephone, infected site s1 shares update with susceptible site s2 • s2 then becomes infected and spreads the update to n other susceptible sites (exponential growth) • At a certain point, infected site realizes that rumor has spread sufficiently and stops sharing it via: • Feedback-based probability (stop with probability 1/k if site was already infected) • Blind probability (always stop with probability 1/k) • Fixed count (stop after k sites report they are already infected)

9. Rumor MongeringConnection Limit • Consider a limit on the number of rumor requests a server can process at once • Effects of Push and Pull methods • On mostly-stale database, Push excels since if two sites try to push to the same recipient, one is rejected; as epidemic spreads, this has the effect of canceling unnecessary update network traffic • With Pull method, connection limit introduces slight chance that a site might miss an update

10. Replication Strategy Recap • Direct mail creates high volume of network traffic • Anti-Entropy is only practical as a backup to some other mechanism since it can be so expensive • Rumor Mongering introduces nonzero probability of failure • So... use Rumor Mongering with infrequent Anti-Entropy backup

11. Deleted Data • Need to use “Death Certificate” to notify sites of deleted data • Similar to imagining a “Deleted” flag on every piece of data, with delete operations updating this flag instead of actually deleting data • Compare timestamp of Death Certificate in same way update timestamps are compared • Store Death Certificates: • For a fixed period of time • In “dormant” state at small number of sites and activate it when site s1 realizes site s2 has not seen it; keep “activation timestamp” to determine dormancy

12. Network Node Distribution • Consider extremes • Only nearest neighbors • Traffic per link per cycle = O(1) • Cycles to spread update = O(n) • Random • Traffic per link per cycle = O(n) • Cycles to spread update = O(log n) • Distance proportional to d-2 • Traffic per link per cycle = O(log n) • Cycles to spread update = O(log n) • Great... but achieving d-2 in real-world network not immediately obvious; suggests rectilinear meshes of sites

13. Network Node DistributionAchieving Rectilinear Grids • Xerox’s topology contains hundreds of nodes in US and tens of nodes in Europe connected by a pair of transatlantic links • Each site keeps list of other sites sorted by distance and chooses nearest recipients with greatest probabilities

14. Network Node DistributionPerformance by Distribution Factor

15. Network Node DistributionPush and Pull • Push alone and Pull alone extremely sensitive to network topology • Consider a network with two isolated sites s1 and s2 • With Push, if the update is introduced at s1 or s2 and these two sites continuously select each other as partners, the update will not propagate • With Pull, if the update is introduced in the main part of the network, high likelihood that the update is no longer hot when s1 or s2 finally go to pull

16. References • Demers, A., Greene, D., Hauser, C., Irish, W., Larson, J., Shenker, S., Sturgis, H., Swinehart, D., and Terry, D. 1987. Epidemic algorithms for replicated database maintenance. In Proceedings of the Sixth Annual ACM Symposium on Principles of Distributed Computing (Vancouver, British Columbia, Canada, August 10 - 12, 1987). F. B. Schneider, Ed. PODC '87. ACM Press, New York, NY, 1-12. DOI= http://doi.acm.org/10.1145/41840.41841 • M. -J. Lin and K. Marzullo. Directional gossip: gossip in a wide area network. To be published in Proceedings of the Third European Dependable Computing Conference (SpringerVerlag LNCS). • D. Agrawal, A. El Abbadi, and R.C. Steinke, "Epidemic Algorithms in Replicated Databases," Proc. 16th Symp. Principles of Database Systems, pp. 161-172, May 1997. • Chandra, R., Ramasubramanian, V., Birman, K.P. Anonymous Gossip: Improving multicast reliability in mobile ad-hoc networks. International Conference on Distributed Computing Systems (2001) 275-283.