Operational Experience with

1. Operational Experience with MPLS Dave Siegel Vice President IP Engineering

2. Table of Contents • Overview of hub architecture • History of Network architecture • Early challenges and how MPLS solved them • Challenges with MPLS today • VPN Deployment • Architecture • Capabilities (RFC2547, l2VPN) • Provisioning aspects • Customer experiences • IPv6 deployment w/ MPLS • How the network would behave without MPLS

3. WR1 WR2 OC-48/OC-192 OC-12/OC-48 OC-3/OC-12 CR1 CR2 AR1 AR2 PR1 RR1 Modems ADMs Ethernet Switches GBLX

12. Early Challenges • Hop Count • Network diameter ranged from 14-18 hops • GRE tunnels were not supported on GSR images • Traffic Engineering • Large numbers of DS3’s and OC-3c’s in metro regions proved difficult to manage with IS-IS metrics • Future VPN Product

13. MPLS Solutions • Hop Count • MPLS had no-decrement-ttl option • MPLS tunnels were in implementation phase for GSR images • Traffic Engineering • MPLS provided for much more efficient utilization of metro bandwidth

14. MPLS Solutions • Hop Count • Established Cross-country tunnels to mask main hops normally encountered in the core • Traffic Engineering • Established regional meshes of LSPs between devices

15. Multi-vendor networks • Theory: having multiple suppliers gives you best-of-breed, plus contingency plans if you have major problems with your primary supplier • Reality: Once a vendor is entrenched in your network, replacing them completely is simply too capital intensive • Reality: you have worst-of-breed, because you must wait for both of your vendors to have a compatible implementation of a feature before deployment.

16. Multi-vendor networks • Early interoperability issues (circa 1999-2000) • Penultimate hop (NULL label vs. strip) • No-decrement-ttl issues (it’s a one-hop network!) • Current Issues • Fast Re-route • Secondary LSP • Auto-bw

17. Current Stats • MPLS core LSP mesh • 9900 tunnels make up the core mesh (100 core routers) • 1200 tunnels between PR’s that make up the IP-VPN Express route Product • 11,100 tunnels total in the core • Complexity requires automated management tools

18. MPLSrobot • Bot components • High-speed snmp poller • Tunnel resize script w/ tons of knobs • Graphing capability • Path database • Configuration push scripts • Day-to-day challenges involve conditioning of collected data • Run daily but configs pushed weekly

19. WANDL

20. Getting Started • Remove roadblocks • Look for features of your network design that increase complexity or introduce roadblocks to implementing MPLS • Multiple AS’s • Multiple levels/areas in your IGP • Lack TE support in your IGP

21. Getting Started • Choose reasonable RSVP bandwidth • Set bandwidth values on new tunnels to 0 Mbps, and then measure over 24 hours. • Set tunnel bandwidth to observed peak + some fudge factor (e.g. 95th %tile peak + 10%) • Do tunnel implementations slowly over time…don’t introduce too much churn in the network • Tune link utilization with RSVP bandwidth values during transition

22. Follow our Roadmap! • Q4 1998 MPLS lab trials begin • Q1 1999 MPLS limited production trial begins (regional mesh + ttl masking hack) • Q2 1999 national LSP mesh between all CR’s complete • Q2 2000 global LSP mesh complete • Q2 2001 RFC 2547 IP VPN’s and L2-VPN’s with DiffServ (2 CoS’s)

23. Operational issues uncovered • Through 1999, MPLS was blamed for a variety of outages and/or performance degradation issues, including • High latency • Loss • Reachability • Workarounds included bouncing LSP’s • Most of the time, CEF bugs were to blame!

24. Operational issues uncovered • Except when WRED was to blame

25. Operational issues uncovered • Training, Training, Training • Cannot be underestimated • Experience, Experience, Experience • GX had 2 full years of experience with MPLS operationally before adding MPLS-based VPN’s to the network

26. IP-VPN (ExpressRoute) architecture • Objective is to provide as much isolation from the Internet as possible • Separate ASN (not AS3549) • Private Routers (PR’s) not reachable from outside gblx.net (non-advertised address space) • Full mesh of LSP’s between all PR’s • Full iBGP mesh among all PR’s

27. IP-VPN (ExpressRoute) architecture • Secondary Objective is to provide as high a class of service as possible. • LSP’s have higher priority than LSP’s for Internet Service so they always get the best (lowest latency) routes. • ToS Bits are painted into a Business Class (vs. Best Effort for Internet Service) which is re-written into the EXP field

28. IP-VPN Customers • Connected: Even mix of RFC 2547 and l2-vpn • Sales Funnel: Majority (60%) want L3-VPN • Sales Funnel: good mix between carrier and enterprise • Largest customer is RFC 2547 with approximately 50 circuits • Market interest is still gaining momentum for this product set (ISP provided IP-VPN)

29. IP-VPN Provisioning pros/cons • L2-VPN’s are the easiest to engineer for the customer, but adds/deletes/moves require updating configs on every PR where the customer is connected. • L3-VPN’s require less configuration on the ISP side, but are preferred less due to the high level of CPE engineering coordination required. • L3-VPN’s were designed as a complete out-source of a customer’s routing, but in reality customers use this service in conjunction with another VPN

30. IPv6 deployment using MPLS • GX has 3 routers located at native IPv6 exchange points • Sure, you could use GRE tunnels to interconnect them over IPv4, but MPLS gives you: • Per tunnel utilization statistics • Path info • Scalability (as the product grows, you can add devices as an overlay network without impacting stability on existing platforms)

31. How the network would behave without MPLS • WANDL simulations show that there would be no congestion in the network based on IGP TE with IS-IS, so MPLS is not needed today for TE. • Bandwidth reservations for MPLS-based VPNs would not be as meaningful with large amounts of native IP traffic on backbone trunks.

32. Questions

33. THANK YOU Additional questions may be sent to dsiegel@gblx.net