Routers with Small Buffers

1. Routers with Small Buffers NANOG, October 25, 2005 Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford.edu http://yuba.stanford.edu/~yganjali/ Joint work with: Guido Appenzeller, Mihaela Enachescu, Ashish Goel, Tim Roughgarden, Nick McKeown Special thanks to: Level 3 Communications

2. # packets at 10Gb/s 1,000,000 10,000 20 Sawtooth Peak-to-trough Non-bursty arrivals Smoothing of many sawtooths Intuition & Proofs Simulations, Experiments Simulated ManyTCP Flows Simulated Single TCP Flow Evidence The Story • Assume: Large number of desynchronized flows; 100% utilization • Assume: Large number of flows; <100% utilization Routers with Small Buffers

3. Backbone Router Buffers Source Destination Router C • Universally applied rule-of-thumb: • A router needs a buffer size: • 2T is the two-way propagation delay (or just 250ms) • C is capacity of bottleneck link • Context • Mandated in backbone and edge routers. • Appears in RFPs and IETF architectural guidelines. • Usually referenced to Villamizar and Song: “High Performance TCP in ANSNET”, CCR, 1994. • Already known by inventors of TCP [Van Jacobson, 1988] • Has major consequences for router design 2T Routers with Small Buffers

4. Single TCP Flow Rule for adjusting W • If an ACK is received: W ← W+1/W • If a packet is lost: W ← W/2 Only Wpackets may be outstanding Routers with Small Buffers

5. Window size t Single TCP Flow Rule for adjusting W • If an ACK is received: W ← W+1/W • If a packet is lost: W ← W/2 Only Wpackets may be outstanding Source Dest Routers with Small Buffers

6. Time evolution of a single TCP flow through a router. Buffer is 2T*C Time evolution of a single TCP flow through a router. Buffer is < 2T*C Time Evolution of a Single TCP Flow Routers with Small Buffers

7. Synchronized Flows • Aggregate window has same dynamics • Therefore buffer occupancy has same dynamics • Rule-of-thumb still holds. t Routers with Small Buffers

8. Probability Distribution Buffer Size Many TCP Flows B 0 Routers with Small Buffers

9. Required Buffer Size Simulation Routers with Small Buffers

10. Real Network Experiments • Stanford University dorm traffic • Network Lab (Cisco routers) at University of Wisconsin • Internet2 • Operational Internet backbone Routers with Small Buffers

11. Buffer sizes 190ms, 10ms, 5ms, 2.5 and 1ms Load balancing High link utilization Long duration (about two weeks) Drops, utilization data collected every 30 seconds Test flows Internet Backbone Experiment Routers with Small Buffers

12. MAX Packet Drops vs. Link Load Buffer size = 190ms, 10ms, 5ms Routers with Small Buffers

13. Packet Drops vs. Link Load Buffer size = 1ms Routers with Small Buffers

14. Relative Link Utilization Utilization of the link with 1ms buffer / Utilization of the link with 190ms buffer Routers with Small Buffers

15. Relative Utilization (Cont’d) Routers with Small Buffers

16. Practice Typical OC192 router linecard buffers over 1,000,000 packets 5 orders of magnitude difference! Theory vs. Practice Theory (benign conditions) M/D/1 Poisson D B Loss independent of link rate, RTT, number of flows, etc. Can we make traffic look “Poisson-enough” when it arrives to the routers…? Routers with Small Buffers

17. Paced Injections Assume: • Buffer size > • Distance between consecutive packets of a single flow S > • Limited injection rate • Flows are not synchronized; and • Start times picked randomly and independently We can prove that the packet drop probability is very low. Similar results from Cambridge/UCL, UMass and Stanford See papers in: ACM Computer Communications Review, July 2005 Routers with Small Buffers

18. O(log W) Buffers Assumptions: • Internet core is over-provisioned • Example: Load < 80% • There is spacing between packets of the same flow: • Natural: Slow access links • Artificial: Paced TCP Result: Traffic is very smooth, and loss rate is very low, independent of RTT, and number of flows. With a buffer size of just 10-20 packets we can gain high throughput. Routers with Small Buffers

19. Leaky Bucket – Paced vs. Reno Bucket drains with a constant rate. Load is 90% for both cases. Routers with Small Buffers

20. TCP Reno TCP Reno sends packets in a burst  High drop rate Routers with Small Buffers

21. Paced TCP Spacing packets  Much lower drop rate Routers with Small Buffers

22. TCP With Pacing O(log W) Buffers Regular TCP Routers with Small Buffers

23. O(log W) Buffers Routers with Small Buffers

24. Ideal Experiment • Highly loaded link, with real/realistic traffic • Precisely controlled router buffers • Packet traces with precise timestamps • Work in progress: Sprint, Verizion, Telcordia, Lucent, … Packet Trace Monitor Packet Trace Monitor Routers with Small Buffers

25. Conclusion and Future Work • Theory: • Reducing buffer sizes by a factor of sqrt(N) does not affect the network performance. • Reducing the buffer sizes to O(logW) does not affect the network performance if: • The network is over provisioned; and • We use Paced TCP; or • Have slow access links • Experimental Validation: • Thousands of ns2 simulations • Stanford dorm, University of Wisconsin Testbed, Internet2, Level 3 Communications, … • Ongoing work and need your help Routers with Small Buffers

26. Thanks!More Info?yganjali@stanford.eduhttp://www.stanford.edu/~yganjalihttp://yuba.stanford.edu/~yganjali/research/bsizing/ Routers with Small Buffers

27. O(log W) Buffers With a large ratio between core and access link bandwidth Bottleneck Bandwidth = 1Gb/s Routers with Small Buffers