Interdomain Routing Correctness and Stability

1. Interdomain RoutingCorrectness and Stability Nick Feamster

2. Is correctness really that important?

3. Is correctness really that important? • The Internet is increasingly becoming part of the mission-critical Infrastructure (a public utility!). Big problem: Very poor understanding of how to manage it.

4. Why does routing go wrong? • Complex policies • Competing / cooperating networks • Each with only limited visibility • Large scale • Tens of thousands networks • …each with hundreds of routers • …each routing to hundreds of thousands of IP prefixes

5. What can go wrong? Some things are out of the hands of networking research But… Two-thirds of the problems are caused by configuration of the routing protocol

6. Session Destination Next-hop AS Path 18.0.0.0/8 192.5.89.89 1 3356 3 66.250.252.44 18.0.0.0/8 174 3 Review: Simple operation… Autonomous Systems (ASes) Route Advertisement MIT

7. …but complex configuration! Flexibility for realizing goals in complex business landscape • Which neighboring networks can send traffic • Where traffic enters and leaves the network • How routers within the network learn routes to external destinations Traffic No Route Route Flexibility Complexity

8. Filtering: route advertisement Dissemination: internal route advertisement Configuration Semantics Ranking: route selection Customer Primary Competitor Backup

9. What types of problems does configuration cause? • Persistent oscillation (today’s reading) • Forwarding loops • Partitions • “Blackholes” • Route instability • …

10. UUNet Sprint These problems are real “…a glitch at a small ISP… triggered a major outage in Internet access across the country. The problem started when MAI Network Services...passed bad router information from one of its customers onto Sprint.” -- news.com, April 25, 1997 Florida Internet Barn

11. These problems are real “…a glitch at a small ISP… triggered a major outage in Internet access across the country. The problem started when MAI Network Services...passed bad router information from one of its customers onto Sprint.” -- news.com, April 25, 1997 “Microsoft's websites were offline for up to 23 hours...because of a [router] misconfiguration…it took nearly a day to determine what was wrong and undo the changes.” -- wired.com, January 25, 2001 “WorldCom Inc…suffered a widespread outage on its Internet backbone that affected roughly 20 percent of its U.S. customer base. The network problems…affected millions of computer users worldwide. A spokeswoman attributed the outage to "a route table issue." -- cnn.com, October 3, 2002 "A number of Covad customers went out from 5pm today due to, supposedly, a DDOS (distributed denial of service attack) on a key Level3 data center, which later was described as a route leak (misconfiguration).” -- dslreports.com, February 23, 2004

12. Several “Big” Problems a Week

13. Why is routing hard to get right? • Defining correctness is hard • Interactions cause unintended consequences • Each network independently configured • Unintended policy interactions • Operators make mistakes • Configuration is difficult • Complex policies, distributed configuration

14. The protocol does not oscillate Correctness Specification Safety The protocol converges to a stable path assignment for every possible initial state and message ordering

15. 1 2 3 Safety: No Persistent Oscillation 1 3 0 1 0 0 2 1 0 2 0 3 2 0 3 0 Varadhan, Govindan, & Estrin, “Persistent Route Oscillations in Interdomain Routing”, 1996

16. Strawman: Global Policy Check • Require each AS to publish its policies • Detect and resolve conflicts Problems: • ASes typically unwilling to reveal policies • Checking for convergence is NP-complete • Failures may still cause oscillations

17. Think Globally, Act Locally • Key features of a good solution • Safety: guaranteed convergence • Expressiveness: allow diverse policies for each AS • Autonomy: do not require revelation/coordination • Backwards-compatibility: no changes to BGP • Local restrictions on configuration semantics • Ranking • Filtering

18. Main Idea of Today’s Paper • Permit only two business arrangements • Customer-provider • Peering • Constrain both filtering and ranking based on these arrangements to guarantee safety • Surprising result: these arrangements correspond to today’s (common) behavior Gao & Rexford, “Stable Internet Routing without Global Coordination”, IEEE/ACM ToN, 2001

19. customer customer Relationship #1: Customer-Provider Filtering • Routes from customer: to everyone • Routes from provider: only to customers From the customer To other destinations From other destinations To the customer providers providers advertisements traffic

20. traffic customer customer Relationship #2: Peering Filtering • Routes from peer: only to customers • No routes from other peers or providers advertisements peer peer

21. provider peer customer Rankings • Routes from customers over routes from peers • Routes from peers over routes from providers

22. Additional Assumption: Hierarchy Disallowed!

23. Safety: Proof Sketch • System state: the current route at each AS • Activation sequence: revisit some router’s selection based on those of neighboring ASes

24. Activation Sequence: Intuition • Activation: emulates a message ordering • Activated router has received and processed all messages corresponding to the system state • “Fair” activation:all routers receive and process outstanding messages

25. Safety: Proof Sketch • State: the current route at each AS • Activation sequence: revisit some router’s selection based on those of neighboring ASes • Goal: find an activation sequence that leads to a stable state • Safety: satisfied if that activation sequence is contained within any “fair” activation sequence

26. Proof, Step 1: Customer Routes • Activate ASes from customer to provider • AS picks a customer route if one exists • Decision of one AS cannot cause an earlier AS to change its mind An AS picks a customer route when one exists

27. Proof, Step 2: Peer & Provider Routes • Activate remaining ASes from provider to customer • Decision of one Step-2 AS cannot cause an earlier Step-2 AS to change its mind • Decision of Step-2 AS cannot affect a Step-1 AS AS picks a peer or provider route when no customer route is available

28. Ranking and Filtering Interactions • Allowing more flexibility in ranking • Allow same preference for peer and customer routes • Never choose a peer route over a shorter customer route • … at the expense of stricter AS graph assumptions • Hierarchical provider-customer relationship (as before) • No private peering with (direct or indirect) providers Peering

29. Some problems • Requires acyclic hierarchy (global condition) • Cannot express many business relationships Sprint Abovenet Verio Customer PSINet Question: Can we relax the constraints on filtering? What happens to rankings?

30. 3*,2*,0* 1 2 3 1*, 3*, 0* 2*,1*,0* Other Possible Local Rankings Accept only next-hop rankings • Captures most routing policies • Generalizes customer/peer/provider • Problem: system not safe Accept only shortest hop count rankings • Guarantees safety under filtering • Problem: not expressive Feamster, Johari, & Balakrishnan, “Implications of Autonomy for the Expressiveness of Policy Routing”, SIGCOMM 2005

31. What Rankings Violate Safety? Theorem. Permitting paths of length n+2 over paths of length n will violate safety under filtering. Theorem. Permitting paths of length n+1 over paths of length n will result in a dispute wheel. Feamster, Johari, & Balakrishnan, “Implications of Autonomy for the Expressiveness of Policy Routing”, SIGCOMM 2005

32. What about properties of resulting paths, after the protocol has converged? We need additional correctness properties.

33. The protocol does not oscillate Correctness Specification Safety The protocol converges to a stable path assignment for every possible initial state and message ordering Path Visibility Every destination with a usable path has a route advertisement If there exists a path, then there exists a route Example violation: Network partition Route Validity Every route advertisement corresponds to a usable path If there exists a route, then there exists a path Example violation: Routing loop

34. “iBGP” Path Visibility: Internal BGP (iBGP) Default: “Full mesh” iBGP. Doesn’t scale. Large ASes use “Route reflection” Route reflector: non-client routes over client sessions; client routes over all sessions Client: don’t re-advertise iBGP routes.

35. iBGP Signaling: Static Check Theorem. Suppose the iBGP reflector-client relationship graph contains no cycles. Then, path visibility is satisfied if, and only if, the set of routers that are not route reflector clients forms a clique. Condition is easy to check with static analysis.

36. How do we guarantee these additional properties in practice?

37. Today: Reactive Operation What happens if I tweak this policy…? • Problems cause downtime • Problems often not immediately apparent Revert No Yes Desired Effect? Wait for Next Problem Configure Observe

38. rcc Configure Detect Faults Deploy Goal: Proactive Operation • Idea: Analyze configuration before deployment Many faults can be detected with static analysis. Feamster & Balakrishnan, “Detecting BGP Configuration Faults with Static Analysis”, NSDI 2005

39. rcc Overview Distributed router configurations (Single AS) • Analyzing complex, distributed configuration • Defining a correctness specification • Mapping specification to constraints “rcc” Correctness Specification Constraints Faults Normalized Representation Challenges Feamster & Balakrishnan, “Detecting BGP Configuration Faults with Static Analysis”, NSDI 2005

40. rcc Implementation Preprocessor Parser Distributed router configurations Relational Database (mySQL) (Cisco, Avici, Juniper, Procket, etc.) Constraints Verifier Faults Feamster & Balakrishnan, “Detecting BGP Configuration Faults with Static Analysis”, NSDI 2005

41. Summary: Faults across 17 ASes Every AS had faults, regardless of network size Most faults can be attributed to distributed configuration Route Validity Path Visibility

42. rcc: Take-home lessons • Static configuration analysis uncovers many errors • Major causes of error: • Distributed configuration • Intra-AS dissemination is too complex • Mechanistic expression of policy • http://nms.csail.mit.edu/rcc/ • About 100 downloads (70 network operators)

43. Two Philosophies • This lecture: Accept the Internet as is. Devise “band-aids”. • Another direction: Redesign Internet routing to guarantee safety, route validity, and path visibility

44. RCP iBGP Preventing Errors in the First Place Before: conventional iBGP eBGP iBGP After: RCP gets “best” iBGP routes (and IGP topology) Feamster et al., “The Case for Separating Routing from Routers”, SIGCOMM FDNA, 2004 Caesar et al., “Design and Implementation of a Routing Control Platform”, NSDI, 2005

46. Dissemination Ranking Configuration Syntax (Example) router bgp 7018 neighbor 192.0.2.10 remote-as 65000 neighbor 192.0.2.10 route-map IMPORT in neighbor 192.0.2.20 remote-as 7018 neighbor 192.0.2.20 route-reflector-client ! route-map IMPORT permit 1 match ip address 199 set local-preference 80 ! route-map IMPORT permit 2 match as-path 99 set local-preference 110 ! route-map IMPORT permit 3 set community 7018:1000 ! ip as-path access-list 99 permit ^65000$ access-list 199 permit ip host 192.0.2.0 host 255.255.255.0 access-list 199 permit ip host 10.0.0.0 host 255.0.0.0

47. Why is Routing Hard to Get Right? • Defining correctness is hard • Operators make mistakes • Configuration is difficult • Complex policies, distributed configuration • Interactions cause unintended consequences • Each network independently configured • Unintended policy interactions

48. Which faults does rcc detect? Faults found by rcc Latent faults Potentially active faults End-to-end failures

49. Filters Rankings Normalizing Router Configuration Challenge: • Hundreds of routers distributed across an AS • Thousands to tens of thousands of lines per router • Many ways to express identical policy Solution: • Express configuration with centralized tables • Check constraints by issuing queries on tables Sessions

50. Export Clause Prepend Neighbor AS Export 1 1 123 10.1.2.3 456 1 2 1 123 123 10.4.5.6 456 2 Policy normalization makes comparison easy. Route Validity: Consistent Export Possible Causes • Malice/deception • iBGP signaling partition • Inconsistent export policy neighbor 10.4.5.6 route-map PEER permit 10 set prepend 123 123 neighbor 10.1.2.3 route-map PEER permit 10 set prepend 123