Use of BGP and MPLS VPNs: A Case Study

1. Use of BGP and MPLS VPNs: A Case Study Fred P. Baker CCIE#3555

2. Contents • Current Network • The MPLS VPN project • Routing Objectives • What we did • How we tested

3. Current Network

4. Current Environment • Hub and spoke to 4 data centers • Sites do not in general connect to 2 data centers due to cost and OSPF issues • Generally place servers by geography • You servers are in the data center your links are in • Mostly Frame Relay to ATM interworking with some private lines • 70 of some 350 remote sites have 2 links • ATM PVC dual mesh between the data centers • 12000 agent location network done by MCI with combination of DSL and Fractional T1

5. Address Space • 10.0.0.0/8 • Mostly inside • Some BP • 192.168.0.0/16 • Used all over • 172.16.0.0/12 • Extranet • 167.127.0.0/16 • Public address space • Used mostly by extranet • Some legacy inside

6. Core • ATM PVCs • 2 10meg between each pair of data centers • 2 routers on the core • So 2 meshes

7. Allstate Core

8. 10.0.0.0 address allocation/11 for core 1 per data center

9. Allstate Data Center

10. Routing Protocol • Single OSPF AS • Cisco and OS/390 based routers only • Firewalls now static routed • Peer authentication soon

11. Remote sites • AT&T frame relay at the site • ATM into the data center • Some ISDN backup • A remote site is connected to a single data center (for now) • Servers and applications tend to have geographic affinity

12. Remote Site

13. Remote Site Switch Layer

14. Agent Broadband • 10,000 locations • Connected via IPSEC VPN • WorldCom managed routers • NO split tunneling • IPSec Transport with GRE tunnel to Dallas and Hudson • Agent PCs are 10.*.*.* • Agent access is via Allstate Internet Proxy

15. Overview

16. Agent Broadband in Data Center

17. Agent office

18. Internet/Extranet • We do not use the default route • There are 3 data center with ISP connections • We code static routes to the firewalls (we don’t trust firewalls running dynamic routing protocols) and redist to OSPF

19. The project

20. The project • We use a single data network provider • This is a single point of failure of that providers ATM/Frame networks • Add a second data provider • Initially to use for the dual attached sites • Then convert 1 of the core ATM meshes to the second provider

21. Layer 2 vs Layer 3 provider • Frame Relay is layer 2 connectivity • The routers have a direct peering relationship • Many providers are offering Layer 3 • Costs are the same or even less • MPLS VPN is the data transport • Many providers are using MPLS to move even layer 2 networks • You have a routing relationships with the provider not with yourself • So More complex to configure and fix • Not a simple OSPF network anymore

22. Which one we picked • Layer 3… • DR becomes free do not need to run more PVCs to a DR data center • The data center placement of servers assumption is changing • Apps are being put to 1 DC • Also there is more site to site traffic than we expect • So we can reduce traffic on the ATM core • And increase response time • Do dual homed sites first convert 1 link to L3 • Single homed late

23. VPN A/Site 2 10.2/16 VPN B/Site 1 10.2/16 CEA2 CE1B1 10.1/16 CEB2 VPN B/Site 2 P1 PE2 CE2B1 P2 PE1 PE3 CEA3 CEA1 P3 10.3/16 CEB3 10.1/16 VPN A/Site 3 10.4/16 VPN A/Site 1 VPN B/Site 3 MPLS VPN

24. Route types • CE customer Edge • your router • run BGP to provider • Knows nothing about other customers or provider routes • PE provider Edge • Knows about all local customer VPNS • Has multiple routing tables • P providers • Transport only • No customer routes

25. Routing objectives • Support load share from the home DC • Remote site goes direct to non home DC over L3 • Remote site directly to remote site • Reduce transit of the core • Support a L3 provider in the core replacing 1 ATM mesh • Do not use remote sites to transit traffic

26. Technical Objectives • Limit the number of bgp attributes used • Keep the remote site configuration simple • Do not inject the default route unless you must • How to inject the Internet routes

27. Routing protocol design

28. Don’t forget the 3 rules of routing • Longest subnet mask • Lowest distance • Best metric

29. BGP features we used • As path • Path length filters • No export • Backdoor • If AS Paths are equal then router uses eBGP route

30. How to route • Must look at the routes going BOTH ways • Routes to • Routes from • The routes you advertise drags traffic to you • The routes you take in is how you route back • We load share by having each router use a different path, then send equal cost into IGP

31. Result • Use MPLS VPN based L3 provider • Remote sites 2nd link to L3 • Each data center connects to L3 • Will not use L3 to route between DCs due to QoS concerns

32. Routing • Use BGP at remote sites • Can use OSPF with SOME providers but not all • BGP works much better • Each site is 1 AS • EACH data center is 1 AS • This allows us to put an L3 provider in later • BGP routes BETWEEN ASes • Address ASes from private space • This is ok because provider is a VPN

33. Route injection to/from BGP • Allstate Data Center • Explicit network statements to BGP • Redist BGP to OSPF • Remote site routes • Redist from OSPF • Decided that using network statements to complex • BGP routers send just default route to any switches • We will accept the extra LAN transit • Internet routes • Redist static

34. Internet routes • There will be non BGP L3 switches between Inet and allstate core • Redist static into OSPF already • So just redist into BGP also • Put internet router in same AS as datacenter (have to as no direct path) • Use sync • Send to L3 provider and to sites over L3

35. BGP to L3 provider (and then remote sites • Data center side • Send data center /11s • Send internet routes • Take routes from L3 provider • Do not forward other eBGP learned routes • Remote site side • Send all local routes • do not forward other learned eBGP routes • Remember the no export to kill transit • Receive all routes • Want to take L3 when I can

36. DC to Remote site FR • Send all bgp derived routes • Do as prepend of the data center AS • This makes AS path =2 for DC on FR and L3 paths • This makes AS Path=3 for DC to DC via ATM core so site to remote DC traffic over L3

37. Remote site to DC on FR • Do as prepend of 1 AS at remote end • Need this so FR and L3 paths have AS Path=2 so we load share • Filter routes with AS Path >1 • I only want to send the local site routes up the FR link • Do not want DC to send transit traffic to site

38. IBGP in the remote site • Set next hop self • Routers must have a shared Enet • No redist of BGP to OSPF • So cant use sync so cant transit a L3 switch • Do not forward routes I learn via FR • Do not want a transit from L3 up the FR link • Do not want a transit to L3 from FR link • Set no export attribute on routes from DC over the FR link • This prevents site from passing them to L3 • Cannot AS path filter on IBGP because I want to pass the DC route via iBGP • Why I use no export

39. Results

40. DC to DC • Each site learns over ATM network with AS Path = 1 • Cannot route over L3 provider

41. Remote site to non home dc • Non home DC sent via L3 AS Path = 2 • Home data sends via FR AS Path = 3 due to prepend • Use if L3 down

42. non home dc to remote site • Non Home DC learns remote site routes from L3 • Home data center sends only the /11 summary • so longest match says L3

43. home dc to remote site • Load share • Routes from L3 have AS Path = 2 • Routes from FR have AS Path = 2 due to prepend • So each router uses eBGP route

44. remote site to home dc • Don’t care as much about load share • Routes from L3 have AS Path = 2 • Routes from FR have AS Path = 2 due to prepend • So each router uses eBGP route

45. remote site to remote site • Use L3 network • Learn site specific routes directly from site • Learn /11 summaries from DCs

46. Agent routes • Only dual DC connected things that don’t use BGP • Many routes summarized as /19s • I get these from MCI as OSPF externals • Have not decided how to inject them • They go to two data centers for redundancy • So I need to send them via BGP • So a router will get an OSPF external from the local MCI connection and the other data center via BGP • eBGP < OSPF so BOOM • Use backdoor on core routers to set distance on the agent routes to > than OSPF • So if local MCI connection up use it, else transit core

47. Testing

48. Local Testing • Use 7 routers • 1 remote site OSPF route not shown • Paths • iBGP at remote • L3 • FR to home DC • Inter DC

49. CPOC • Cisco Proof Of Concept • In Raleigh and San Jose • Lab use is free (if you are big enough) • Send in specific test plan • Your SE goes in a week ahead of time • Lab is all setup when you arrive

50. Testing • Test migrations • Test routing • based on our policies • failovers • Measure convergence • Test a migration of a core ATM mesh to L3 • Get some data and experience on the MPLS side • Try multicast over MPLS/VPN