Evaluating the Running Time of a Communication Round over the Internet

1. Evaluating the Running Time of a Communication Round over the Internet Omar Bakr Idit Keidar MIT MIT/Technion PODC 2002

2. Communication Round • Exchange of information from all hosts to all hosts • Part of many distributed algorithms, systems • consensus, atomic commit, replication, ...

3. Common Metric for Evaluating Algorithms • Number of rounds (or steps) they require

4. Questions • What is the best way to implement a communication round over the Internet • decentralized vs. centralized • How long is a communication round over the Internet?

5. Prediction is Hard • Internet is unpredictable, diverse, … • Different answers for different topologies, different times • Different performance metrics • local runningtimeone host is engaged in algorithm • overall runningtimefrom when first host starts to when last host finishes

6. “Communication Round” Primitive • Initiated by some host • Propagates data from every host to every other host connected to it

7. All-to-all Leader Secondary Leader Example Implementations

8. Experiment I • 10 hosts: Taiwan, Korea, US academia, ISPs • TCP/IP (connections always up) • Algorithms: • All-to-all • Leader (initiator) • Secondary leader (not initiator) periodically initiated at each host - 650 times over 3.5 days

9. Computing Overall Running Time • Elapsed time from initiation (at initiator) until all hosts terminate • Requires estimating clock differences • Clocks not synchronized, drift • We compute difference over short intervals • Compute 3 different ways • Achieve accuracy within 20 ms. on 90% of runs

10. All-to-all: 2 Leader: 3 Secondary Leader: 4 Teaser: Comparing Performance Based on Number of Steps

11. 150+240 = 390 150+150+150 = 450 Predicting Overall Runnig Times From MIT • Ping-measured latencies (IP): • Longest link latency 240 milliseconds • Longest link to MIT 150 milliseconds

12. Measured Running Times Runs Initiated at MIT Running times in milliseconds

13. What’s Going On? • Loss rates on two links are very high • 42% and 37% • Taiwan to two ISPs in the US • Loss rates on other links up to 8% • Upon loss, TCP’s timeout is big • More than round-trip-time • All-to-all sends messages on lossy links • Often delayed by loss

14. Distribution of Running Times Up to 1.3 sec. at MIT

15. All-to-all overall local Leader overall local Sec. Leader overall local Average (runs under 2 sec) 866 645 1120 844 679 607 % runs over 2 seconds 54% 24% 64% 43% 13% 7% Running Times Runs Initiated at Taiwan Running times in milliseconds

16. Distribution of Running Timesin Taiwan

17. Taiwan MIT Hosts with bad links to Taiwan Leader Secondary Leader What’s Going On? Good link Lossy link Other Hosts All-to-all

18. Experiment II: Removing Taiwan • Overall running times much better • For every initiator and algorithm, less than 10% over 2 seconds (as opposed to 55% previously) • All-to-all overall still worse than others! • either Leader or Secondary Leader best, depending on initiator • loss rates of 2% - 8% are not negligible • all-to-all sends O(n2) messages; suffers • But, all-to-all has best local running times

19. Probability of Delay due to Loss • If all links would have same latency • assume 1% loss on all links; 10 hosts (n=10) • Leader sends 3(n-1) = 27 messages • probability of at least one loss: 1 -.9927 »24% • All-2-all sends n(n-1) = 90 messages • probability of at least one loss: 1 -.9990 »60% • In reality, links don’t have same latency • only loss on long links matters

20. Conclusions • Message loss causes high variation in TCP link latencies • latency distribution has high variance, heavy tail • Latency distribution determines expected time for receiving O(n) concurrent messages • Secondary leader helps • No triangle inequality, especially for loss • Different for overall vs. local running times • Number of rounds/steps not sufficient metric • One-to-all and all-to-all have different costs