Data Plane Testing on USN, ESnet, and Internet2 Networks

1. Hybrid MLN Data Plane Testing on USN, ESnet, and Internet2 Networks ESCC/Internet2 Joint Techs Summer Meeting July 16, 2007 Fermi Lab Batavia, Illinois Tom Lehman (USC/ISI) Nasir Ghani (Tennessee Tech) Chin Guok (ESnet) Nagi Rao (ORNL) John Vollbrecht (Internet2) John Moore (MCNC) US Dept. of Energy Office of Science

2. Outline • Dataplane testing configuration • Dataplane test data • Dataplane simulation • Dataplane test conclusions • Future Work

3. Hybrid Network Data Planes • Basic premise of hybrid networks is the availability of both best effort routed service and deterministic dedicated resource paths, i.e., circuits • There are many technologies available over which to construct these circuits • IP router-based Multiprotocol Label Switching (MPLS) Label Switched Paths (LSPs) “circuits” • Ethernet VLAN based “circuits” • SONET/SDH TDM “circuits” • Wavelength Division Multiplexing (WDM) “circuits”

4. What Data Plane Technologies to Use? • What do you want to do with your circuits? • Dedicated bandwidth connections for deterministic file transfers? • Dedicated bandwidth & low jitter for instrument control or interactive applications? • Connector backhaul to your IP Network? • Traffic engineering of your IP Network? • Dynamic router-to-router circuits for traffic cut thru? • Computer to Computer communications? • Processor to memory? block data storage system access? • Setting up application specific topologies to create & optimize distributed application or data storage systems?

5. Data Plane Testing • Test characteristics/performances of “circuits” constructed via different technologies; and also “end-to-end paths” constructed via concatenation of individual circuits • Questions • What is difference between the different technologies? • How well does the concatenation/stitching work? • How well does policing/shaping work at the edge? • What happens to a flow that is policed/shaped at the ingress edge by the time it exits the egress edge?

6. Data Plane Testing • Data Paths Across: • ESnet • USN • Abilene • Internet2 Network • DRAGON http://hybrid.east.isi.edu

7. Test Equipment • Spirent AX4000 - Hardware based Traffic Source and Sink • External CDMA Clock allows for synchronized timestamps • Spirent AX4000 • 10 Gbps with OC192 • POS / BERT / 10GbE • Two Gigabit Ethernet

8. Data Collection • Approximately 75 individual tests (generally a unique path) • tests were sourced from 1 Gbps interfaces, some from 10 Gbps • some of the measured flows had cross traffic introduced • Tests generally included 9 measured data collection runs • 64, 500, 8000 byte MTU • 100, 500, 800 Mbps for 1 Gbps paths • 1, 5, 8 Gbps for 10 Gbps paths • For each test run, the following data was collected: • average datarate • total packet loss • average latency • jitter profile (histogram) • transfer delay (histogram)

9. HOPI-Abilene-UltraScience Net-ESnet Test Histograms packet inter-arrival packet end-to-end delay Spirent source  Washington-Force 10  Washington-Juniper T640  Chicago-Juniper T640 Chicago-Force 10 Chicago-Glimmer Glass  Chicago-Force 10 Chicago-Cisco 6509  Seattle-Juniper T640  Sunnyvale-Juniper T640 Sunnyvale-Force 10  Sunnyvale-CDCI  Seattle-CDCI  Chicago-CDCI  Chicago-Force 10  Chicago-Juniper T640  Washington-Juniper T640 Washington-Force 10  Spirent receiver

10. Circuit Description Example • The formal description of this extended inter-network path: • Circuit type: • usn [ethernet:tdm:ethernet]:i2dsn [ethernet:tdm:ethernet]:esnet [ethernet:pscq:ethernet]:usn[ethernet:tdm:ethernet] • Circuit path: • usn [ORNL:CHIN]:i2dsn [CHIN:WASH]:esnet [WASH:CHIN]:usn [CHIN:STTL:SUNV]

11. Modeling & Simulation OPNET ModelerTM Environment • Overview • Discrete event simulation • GUI interface, high re-use • Full C/C++ interface • Hierarchical modeling: • Subnet-node-link-process • “In-House” Development • MPLS/GMPLS control: • RSVP-TE, OSPF-TE, PCE • Layer 2/3 data plane: • IP/MPLS, VLAN • Full Layer 1 support: • DWDM, SONET,GFP • Model any networks

12. Test Plans, Reports, and Data Repositories • Test Report and Plans • http://hybrid.east.isi.edu  DataPlane Testing and Analysis • Raw data repositories • http://www.csm.ornl.gov/ultranet/SpirentMeasurements/ • http://hpn.east.isi.edu/dataplane/sprint-test-data/

13. Summary/Conclusions • All of the tested networking technologies (PSC, L2SC, TDM, LSC) and networks (ESnet, USN, Abilene, HOPI, DRAGON) performed well both individually and when concatenated together • There are some key differences observed between the various networking layer technologies when driven at or close to bottleneck capacity • QoS techniques applied to router MPLS or Ethernet switched paths exhibited notably different delay behaviors versus dedicated circuit-paths (TDM) • TDM-based infrastructures is most germane for applications requiring stringent guarantees on latency, jitter, and bandwidth protection • Inter-layer cross-connections can be achieved in a reasonable manner by “stitching” together different network layer technologies. • Ethernet VLANs presents the least problematic demarc (automated techniques needed to coordinate VLAN tag space) • Future Work • impact of ingress traffic "burstiness” on end-to-end delay and loss profiles, i.e., both for reference and interfering cross-traffic streams. • best techniques for ingress policing and transit node QoS • vendor interoperability testing • additional network testing

14. Thank-You Questions & Comments ?Tom Lehmantlehman@east.isi.edu