Web100

1. Web100 Wendy Huntoon - PSC Jim Ferguson - NCSA I2 Members Meeting May 2002

2. Outline • Project Overview • Motivation: What is the problem • Web100 Collaboration • Progress to Date • Standardization Process • Code Release • Code Capabilities • Overview of Users • Web100 Resources

3. Motivations: What’s the Problem? • High performance flows slower than line rate • Delays continue/increase even with higher bandwidth • TCP tuning issues are non-trivial • Poorly conceived stacks • Router/switch buffer queues inadequate • Slow start and AIMD algorithm • Eliminate/dramatically reduce the “wizard gap” • Need for kernel instrumentation set for TCP variables

4. The Wizard Gap TCP over a long haul path Year Wizards Non-wizards Ratio • 1Mb/s 300kb/s 3:1 • 10Mb/s 1995 100Mb/s • 1Gb/s 3Mb/s 300:1 Scientists/researchers not happy with this

6. TCP tuning is painful debugging • All problems limit performance • IP routing, long round trip times • Improper MSS negotiations or path MTU discovery • IP Packet reordering • Packet losses, congestion, lame hardware • TCP sender or receive buffer space • Inefficient applications • Any one problem can mask all the others and confound all but the best (and few) tuning gurus • Need for better diagnostics and visibility into problems

7. Goal and Method • Make it “easy” (transparent) for non-experts to achieve higher throughput performance • Enhance TCP capabilities with better (finer grain) kernel instrumentation and automatic controls • Real time triage capability determines sender, receiver, and/or network bottlenecks

8. Why Focus on TCP • TCP has an ideal vantage point into throughput problem space • TCP can identify bottleneck subsystem(s) • TCP already measures the network (some) • TCP can measure the application • TCP can adjust itself (auto-tuning feedback)

9. Web100 Collaboration • Funded by the NSF • Currently Year 2 of a 3 Year grant. • Cisco URP for initial seed funding. • Collaborators • PSC (Matt Mathis, R. Reddy, Janet Brown, John Heffner) • NCAR (Peter O’Neil, Marla Meehl) • NCSA (John Estabrook, Tanya Brethour, Stephen Engelhardt, Jim Ferguson)

10. What is in the code • Web100 software consists of: • TCP Kernel Instrument Set (TPC-KIS) • Instruments coded directly in to the Operating System kernel. • Derived Instrument Set (DIS) • Information that is collected based on KIS parameters. • Application Code • Tools, applications, etc. that use the information provided by the KIS and DIS.

11. Kernel Instrument Set • Definition • Set of instruments designed to collect as much of the information as possible to enable a user to isolate the performance problems of a TCP connection. • How it is implemented • Each instrument is a variable in a "stats" structure that is linked through the kernel socket structure. • The Linux /proc interface is used to expose these instruments outside the kernel.

12. What is the TCP-KIS? • TCP-KIS instruments group naturally into categories. • Currently roughly 19 categories. • Already more than 125 instruments have been developed. • For each instrument: • Precise (standards ready) definition. • Instrument code in the kernel • Implementation verification tests • Does the kernel implementation meet the definition. • Prototype diagnostic tool(s) to demonstrate functionality and effectiveness.

13. TCP-KIS • Basic instrumentation examples • Connection ID: 5-tuple that uniquely identifies a connection. • State: determines what protocol features or algorithms are enabled. • Traffic out: statistics aggregate packets and traffic sent out on a connection.

14. Local Sender Triage • Group of instruments associated with the local sender. • Determine what subsystems are throttling TCP data transmission. • Three parallel sets of instruments that measure: • Receiver Window • Network Congestion • Senders Availability

15. Local Sender Groups • Other groups of instruments associated with the Local Sender: • Local Sender Congestion Model • Local Sender Loss Model • Local Sender Re-order Model • Local Sender RTT • Local Sender Segment Size • Local Sender Bottlenecks • Local Sender Tuning

16. Other Instruments • Similar instruments for the Local Receiver. • Observed Receiver instruments • Often inferred from the data stream. • E.g, Observed Receiver - receivers state is inferred from the ACK stream. • Application Interface • Future instruments to collect statistics on how the application is using the network.

17. Userland Distribution • Released asynchronously with kernel distribution • Currently at Alpha 1.1 • Version 1.2 release imminent • Consists of • The web100 library • Command line utilities • GUI utilities

18. Web100 Library • Web100 kernel exposes critical TCP variables/instruments through /proc • Web100 library provides the necessary access functions to access these variables/instruments • Functions • Read the value of a variable/instrument • Snap shot of a group (facilitates atomic reading of a group of variables) • Modify tunable variables (ex. send buffer size) • Etc …

19. Utilities • Command line utilities • Useful in batch scripts • Serve as demo codes for the usage of web100 library • GUI utilities • Based on GTK+ • Useful for troubleshooting network applications • Serve as examples for application developers

20. GUI Sample Screens – DTB

21. Connection Selector

22. Looking at a Variable

23. Timeline - Year 1 • Alpha code development • Establish User Support • www.web100.org • Initial User Community • Very limited to begin with. • Knowledgeable users, expected to provide technical input on the code. • Understand and develop applications.

24. Timeline - Year 2 • Began standardization process. • Develop MIB • Submit to IETF • Develop public code • Fix bugs in alpha versions • Add instrumentation • Code release • Continue code development • Identify and add new instruments

25. Code Releases - To date • Initial Release • Alpha0.2, released May 23, 2001 • Alpha0.3, released Sept. 19, 2001 • Alpha 1.0-Separation of Kernel and Userland code • Kernel Patch: • Alpha 1.1 for Linux 2.4.16, released March 18, 2002 • Alpha 1.0, released March 1, 2002 • Alpha 1.0, released February 26,2002 • Userland: • Alpha 1.1, released February 28, 2002 • Alpha 1.0, released February 26,2002

26. Timeline - Year 3 • New pathprobe diagnostic tool (wip, unreleased). • Add another 10-12 instruments. • Review instruments and code with other wizards. • Gain vendor support for ideas and code. • Finalize IETF draft by December IETF meeting.

27. Milestones • Over a year of ~ 30 alpha testers • Including: SLAC, ORNL, LBNL, and universities • www.net100.org • Modified Linux kernel supports 2.4.16 • Separation between KIS and library functions • draft-ietf-tsvwg-tcp-mib-extension-00.txt • draft-ietf-ipngwg-rfc2012-update-01.txt

28. Web100 Collaborator Activity • Rich Carlson, ANL • Tom Dunnigan, ORNL • Tom Hacker, U. of Michigan • Doug Chang, SLAC • Andreas Burkhardt & Matt Grob, Qualcomm • Larry Dunn & Scott Dier, Cisco/U. of Minnesota • Jason Lee, LBL

29. Collaborator Assistance • Bugs! • Kernel • Utilities • Release • Request new features • Review and criticize documentation • Way too easy on us

30. Collaborator Activity • Carlson/ANL working on a troubleshooting guide for LANs. • Set up network of 13 identically equipped PIII connected via Cisco 5500 network switch, running Web100-enabled Linux. • Introduces typical network faults (duplex mismatches, other config errors) and analyzes data for “signatures” of these faults. • Modified Iperf 1.2 to collect variables and reverse flow.

31. Collaborator Activity • Dunnigan/ORNL has found web100 helpful in seeing losses/retransmission and congestion avoidance parameters of individual TCP flows, and for tuning flows • Has developed a Web100-enabled ttcp • Has developed a daemon that logs web100 variables for designated paths when a flow closes • Has developed an autotuning daemon that uses web100 to tune flows, including modifications to web100 to support "event notification", so the daemon knows when a new flow/socket is opened

32. Collaborator Activity • Hacker/U.Michigan has been using the web100 software to help tune and diagnose end-to-end network performance problems across the U-M campus network as well as across Abilene for the Visible Human and Atlas projects at U-M. • Chang/SLAC is looking to fix performance problem between Linux and Solaris machines.

33. Collaborator Activity • Qualcomm is using Web100 to measure TCP performance over certain types of high speed wireless links under development. Web100 is partially integrated into some other tools - in the sense that output reports are published automatically in a format similar to other tools Qualcomm uses. • Dunn/Cisco currently using Web100 for a class at U.Minnesota. Includes accounts on test machine at NCSA.

34. Collaborator Activity • Lee/LBL has obtained accounts at SLAC and ANL for WAN testing, and have co-located one of our machines in Washington D.C. to do testing over SuperNet. Still in the process of testing all this out. • Keith Jackson at LBL has written Python wrappers to the Web100 calls using swing.

35. Web100 Summary • Main WWW site: www.web100.org • Freely available software distribution • www.web100.org/download • hundreds of downloads • Please be cognizant of impacts on others • Please use, test, provide feedback, contribute code • IETF standards process to benefit all • Attention turning to working with OS vendors to incorporate standards enhancements into their stacks