UCL Data Center Networking with Multipath TCP Costin Raiciu University College London & Universitatea Politehnica Bucuresti Christopher Pluntke, UCL Adam Greenhalgh, UCL Sebastien Barre, Universite Catholique Louvain Damon Wischik. UCL Mark Handley, UCL
Data Center Networking Today Resource Allocation TCP Path Selection Random load balancing Routing OSPF, VLANs, TRILL Topology FatTree, VL2, BCube, multi-rooted tree
Data Center Networking Tomorrow Resource Allocation TCP Multipath TCP Path Selection Random load balancing Routing OSPF, VLANs, TRILL Topology FatTree, VL2, BCube, multi-rooted tree
Data Centers are Important Cloud computing • Economies of scale: networks of tens of thousands of hosts Cool apps • Web search, GFS, BigTable, DryadLINQ, MapReduce • Dense traffic patterns
Flexibility is Important in Data Centers • Apps distributed across thousands of machines. • Flexibility: want any machine to be able to play any role. But: • Traditional data center topologies are tree based. • Don’t cope well with non-local traffic patterns. Many recent proposals for better topologies.
Traditional Data Center Topology Core Switch 10Gbps Aggregation Switches 10Gbps Top of Rack Switches 1Gbps Racks of servers …
Fat Tree Topology [Fares et al., 2008; Clos, 1953] K=4 Aggregation Switches 1Gbps K Pods with K Switches each 1Gbps Racks of servers
VL2 Topology [Greenberg et al, 2009, Clos topology] 10Gbps … 10Gbps 20 hosts
BCube Topology [Guo et al, 2009] BCube (4,1)
How Do We Use this Capacity? • Need to distribute flows across paths. • Basic solution: Random Load Balancing. • Use Equal-Cost Multipath (ECMP) routing. • Hash to a path at random. • Use many differently rooted VLANs. • End-host hashes to a VLAN; determines path.
Collisions 1Gbps 1Gbps Racks of servers
Can MPTCP self-optimize data-center traffic? With Multipath TCP we can explore many paths: • Instead of using one random path, use many random paths • Don’t worry about collisions. • Just don’t send (much) traffic on colliding paths
Simulation Setup • ~8000 hosts • Long-lived flows • Permutation traffic matrix • Each hosts sends and receives from a single other randomly chosen host • Smallest amount of traffic that can fill the network
Multipath TCP in the Fat Tree Topology Throughput Allocation
Performance depends on topology VL2 BCube
Centralized Scheduling • With RLB, it’s really hard to utilize FatTree. • Hedera [Fares et al.,2010] uses a centralized scheduler and flow switching. • Start by using RLB • Measure all flow throughput periodically. • Any flow using more than 10% of its interface rate is explicitly scheduled onto an unloaded link. How does centralized scheduling compare with MPTCP?
MPTCP vs Centralized Dynamic Scheduling Centralized Scheduling MPTCP Infinite Scheduling Interval
Can’t we just use many TCP connections? Loss rate of MP-TCP (“linked”) vs multiple uncoupled TCP flows Retransmit timeouts with MPTCP (“linked”) vs uncoupled TCP flows
MPTCP Linked Increases in DCs • Better fairness and less aggressive than uncoupled TCP • Improves throughput in dense traffic in BCube (25%)
The bigger picture Resource Allocation Path Selection Multipath TCP Routing OSPF, VLANs, etc. Topology ? FatTree, VL2, Bcube, multi-rooted tree
Multipath TCP can utilize topologies TCP can’t • Requirement: a subset of hosts should be able to communicate at 10Gb/s 1Gb/s 1Gb/s 10Gb/s
Multipath TCP can utilize topologies TCP can’t  • Problem • ToR switch failures wipe out tens of hosts • Repair time is on the order of days • Solution: use two ToRs/rack, multi-home servers • Single path TCP • Single flows still get same max throughput • Which interface do I use? • With Multipath TCP • Flows double their maximum throughput • Path selection automatic
Summary • Data center networking offers many paths between end-hosts. Yet: • Random Load Balancing does a poor job of utilizing them • Centralized scheduling is laggy and has inherently limited knowledge • Multipath TCP naturally optimizes data center networks: • Improves throughput • Improves fairness • More robust than centralized scheduling • Question: what topologies does multipath TCP enable?
Centralized Scheduling: Setting the Threshold Throughput Hope 1Gbps 17% worse than multipath TCP 100Mbps App Limited
Centralized Scheduling: Setting the Threshold Throughput 1Gbps 21% worse than multipath TCP 100Mbps App Limited Hope
Centralized Scheduling: Setting the Threshold Throughput 1Gbps 51% 100Mbps 500Mbps 17% 45% 21%