Traffic Engineering for ISP Networks

1. Traffic Engineering for ISP Networks Jennifer Rexford Computer Science Department Princeton University http://www.cs.princeton.edu/~jrex

2. Outline • Internet routing • Overview of the Internet routing architecture • Shortest-path link-state routing between edge routers • Optimization: Tune routing to the traffic • Optimizing routing given a topology and traffic matrix • Local search to select the integer link weights • Design for optimizability: Design routing protocol • Optimal traffic engineering with link-state routing • Tomography: Infer the traffic matrix • Estimating traffic matrix from routing and link load

3. Internet Routing

4. Autonomous Systems (ASes) • Internet is divided into Autonomous Systems • Distinct regions of administrative control • Routers/links managed by a single “institution” • Service provider, company, university, … • Hierarchy of Autonomous Systems • Large, tier-1 provider with a nationwide backbone • Medium-sized regional provider with smaller backbone • Small network run by a single company or university • Cooperate to ensure end-to-end reachability

5. Interdomain Routing • AS-level topology • Destinations are IP prefixes (e.g., 12.0.0.0/8) • Nodes are Autonomous Systems (ASes) • Edges are links and business relationships 4 3 5 2 6 7 1 Web server Client

6. Example Backbone: Abilene Internet2 Newtork

7. Points-of-Presence (PoPs) • Inter-PoP links • Long distances • High bandwidth • Intra-PoP links • Short cables between racks or floors • Aggregated bandwidth • Links to other networks • Wide range of media and bandwidth Inter-PoP Intra-PoP Other networks

8. Intradomain Routing: Shortest-Path Routing • Path-selection model • Destination-based • Load-insensitive (e.g., static link weights) • Minimum hop count or sum of link weights 2 1 3 1 4 2 1 5 4 3

9. Computing Shortest Paths: Link-State Routing • Topology discovery • Routers flood information to learn the topology • Each router constructs a link-state database 2 1 3 1 3 2 1 5 4 3

10. Computing Shortest Paths: Link-State Routing • Shortest-path computation • Each router runs Dijkstra’s shortest-path algorithm • Computes the “next hop” to reach other routers 2 1 3 1 3 2 1 5 4 3

11. Computing Shortest Paths: Link-State Routing • Packet forwarding • Each router maintains a forwarding table • To forward incoming packets to the right next-hop link 2 1 3 1 3 2 1 5 4 3

12. Our Focus: Traffic Engineering (TE) • Adjusting routing to the flow of traffic • How should network administrators run their networks? • Specifically, how should they set the link weights? • Designing protocols for better traffic engineering • How should future routing protocols be designed? • Specifically, how to make TE efficient and easy? • Collecting measurements of the offered traffic • How should administrators learn the traffic matrix? • Specifically, how to infer the matrix from link loads?

13. Optimization: Tuning Routing to the Traffic Joint work with Bernard Fortz and Mikkel Thorup http://www.cs.princeton.edu/~jrex/papers/ieeecomm02.pdf http://www.cs.princeton.edu/~jrex/papers/opthand04.pdf

14. Link Weights Control the Flow of Traffic • Routers compute paths • Shortest paths as sum of link weights • Operators set the link weights • To control where the traffic goes 2 1 3 1 3 2 3 1 5 4 3

15. Heuristics for Setting the Link Weights • Proportional to physical distance • Cross-country links have higher weights than local ones • Minimizes end-to-end propagation delay • Inversely proportional to link capacity • Smaller weights for higher-bandwidth links • Attracts more traffic to links with more capacity • Tuned based on the offered traffic • Network-wide optimization of weights based on traffic • Directly minimizes key metrics like max link utilization

16. Why Are the Link Weights Static? • Strawman alternative: load-sensitive routing • Link metrics based on traffic load • Flood dynamic metrics as they change • Adapt automatically to changes in offered load • Reasons why this is typically not done • Delay-based routing unsuccessful in the early days • Oscillation as routers adapt to out-of-date information • Most Internet transfers are very short-lived • Research and standards work continues… • … but operators have to do what they can today

17. Big Picture: Measure, Model, and Control Network-wide “what if” model Offered traffic Changes to the network Topology/ Configuration measure control Operational network

18. Traffic Engineering in an ISP Backbone • Topology • Connectivity and capacity of routers and links • Traffic matrix • Offered load between points in the network • Link weights • Configurable parameters for Interior Gateway Protocol • Performance objective • Balanced load, low latency, service level agreements … • Question: Given the topology and traffic matrix in an IP network, which link weights should be used?

19. Key Ingredients of Our Approach • Measurement • Topology: monitoring of the routing protocols • Traffic matrix: widely deployed traffic measurement • Network-wide models • Representations of topology and traffic • “What-if” models of shortest-path routing • Network optimization • Efficient algorithms to find good configurations • Operational experience to identify key constraints

20. Formalizing the Optimization Problem • Input: graph G(R,L) • R is the set of routers • L is the set of unidirectional links • cl is the capacity of link l • Input: traffic matrix • Mi,j is traffic load from router i to j • Output: setting of the link weights • wlis weight on unidirectional link l • Pi,j,lis fraction of traffic from i to j traversing link l

21. 0.25 0.25 0.5 1.0 1.0 0.25 0.25 0.5 0.5 0.5 Multiple Shortest Paths With Even Splitting Values of Pi,j,l

22. f(x) x 1 Defining the Objective Function • Computing the link utilization • Link load:ul = Si,j Mi,j Pi,j,l • Utilization: ul/cl • Objective functions • min(maxl(ul/cl)) • min(Slf(ul/cl))

23. Complexity of the Optimization Problem • NP-hard optimization problem • No efficient algorithm to find the link weights • Even for the simple convex objective functions • Why can’t we just do multi-commodity flow? • E.g., solve the multi-commodity flow problem… • … and the link weights pop out as the dual • Because IP routers cannot split arbitrarily over ties • What are the implications? • Have to resort to searching through weight settings

24. Optimization Based on Local Search • Start with an initial setting of the link weights • E.g., same integer weight on every link • E.g., weights inversely proportional to link capacity • E.g., existing weights in the operational network • Compute the objective function • Compute the all-pairs shortest paths to get Pi,j,l • Apply the traffic matrix Mi,j to get link loads ul • Evaluate the objective function from the ul/cl • Generate a new setting of the link weights repeat

25. Making the Search Efficient • Avoid repeating the same weight setting • Keep track of past values of the weight setting • … or keep a small signature (e.g., a hash) of past values • Do not evaluate a weight setting if signatures match • Avoid computing the shortest paths from scratch • Explore weight settings that changes just one weight • Apply fast incremental shortest-path algorithms • Limit the number of unique values of link weights • Do not explore all 216 possible values for each weight • Stop early, before exploring the whole search space

26. Incorporating Operational Realities • Minimize number of changes to the network • Changing just 1 or 2 link weights is often enough • Tolerate failure of network equipment • Weights settings usually remain good after failure • … or can be fixed by changing one or two weights • Limit dependence on measurement accuracy • Good weights remain good, despite random noise • Limit frequency of changes to the weights • Joint optimization for day and night traffic matrices

27. Application to AT&T’s Backbone Network • Performance of the optimized weights • Search finds a good solution within a few minutes • Much better than link capacity or physical distance • Competitive with multi-commodity flow solution • How AT&T changes the link weights • Maintenance done every night from midnight to 6am • Predict effects of removing link(s) from the network • Reoptimize the link weights to avoid congestion • Configure new weights before disabling equipment

28. Example from My Visit to AT&T’s Operations Center • Amtrak repairing/moving part of the train track • Need to move some of the fiber optic cables • Or, heightened risk of the cables being cut • Amtrak notifies us of the time the work will be done • AT&T engineers model the effects • Determine which IP links go over the affected fiber • Pretend the network no longer has these links • Evaluate the new shortest paths and traffic flow • Identify whether link loads will be too high

29. Example Continued • If load will be too high • Reoptimize the weights on the remaining links • Schedule the time for the new weights to be configured • Roll back to the old weight setting after Amtrak is done • Same process applied to other cases • Assessing the network’s risk to possible failures • Planning for maintenance of existing equipment • Adapting the link weights to installation of new links • Adapting the link weights in response to traffic shifts

30. Conclusions on Traffic Engineering • IP networks do not adapt on their own • Routers compute shortest paths based on static weights • Service providers need to adapt the weights • Due to failures, congestion, or planned maintenance • Leads to an interesting optimization problems • Optimize link weights based on topology and traffic • Optimization problem is computationally difficult • Forces the use of efficient local-search techniques • Results of the local search are pretty good • Near-optimal solutions that minimize disruptions

31. Ongoing Work • Robust link-weight assignments • Link/node failures • Range of traffic matrices • More complex routing models • Hot-potato routing • BGP routing policies • Interaction between ASes • Inter-AS negotiation for joint optimization • Grappling with scalability and trust issues

32. Design for Optimizability: Optimal Link-State Routing Protocol Joint work with Dahai Xu and Mung Chiang http://www.cs.princeton.edu/~jrex/papers/pefti.pdf

33. Revisiting TE With Link-State Routing Protocols • Advantages of link weights • One parameter for each unidirectional link • Hop-by-hop forwarding (no tunneling, no per-flow state) • New routes computed automatically after failure • Changing just a few weights can alleviate congestion • Disadvantages of link weights • Computationally expensive optimization • Suboptimal distribution of traffic • (Disruptions when changing the link weights)

34. Example of Inefficient TE • Simple topology • Demand of 300 units: • All on top path: 300% utilization of top path • All on bottom path: 150% utilization of bottom path • Even splitting: 150% on top path, 75% on bottom c1 = 100 t s c2 = 200

35. Stepping Back: Design for Optimizability • Two research approaches • Bottom up: do the best with what you have • Top down: design systems that are easier to manage • Design for manage-ability • “If you are both the professor and the student, you create exam questions that are easy to answer.” – Mung Chiang • Knowing what we know now… • How should intradomain routing protocols work… • … to make TE more efficient and hopefully easier?

36. Optimal TE With Multicommodity Flow • Problem with shortest-path routing • Inflexible even splitting over shortest paths • Optimal distribution of traffic • Send traffic over any paths in any proportions • Using tunneling to force traffic on the paths • Realizable with MultiProtocol Label Switching (MPLS) • Disadvantage of MPLS: high overhead • Large number of paths between pairs of routers • Must adapt the splitting ratios after each failure

37. Can We Have Link-State Routing and Optimal TE? • Link-state routing and hop-by-hop forwarding • Single weight on each link • Local rule to compute splitting over paths • Each router forwards based only on the destination • Link-state routing != shortest-path routing • Routers could use other traffic-splitting rules • … as long as they are locally computable • … only from the link weights

38. Forward Packet Based on Link Weights • Available information at router u • wu,v :weights for all links • dut : shortest distance from u to t • hu,vt : distance gap (dvt + wu,v – dut) distance gap of 1 2 1 3 1 3 2 1 5 4 3 distance gap of 0

39. Traffic-Splitting Function • Relative flow distributed on outgoing links • G(hu,vt): proportion sent out link v toward t • Split traffic to t in proportion • Even splitting • G(hu,vt) is 1 if hu,vt = 0 (all traffic on shortest paths) • G(hu,vt) is 0 if hu,vt > 0 (no traffic on longer paths) G(hu,vt) SG(hu,jt)

40. 2 1 3 1 3 2 1 5 4 3 Exponential Splitting • Exponentially diminishing traffic on longer paths • Proportion on path i proportional to exp(-pi) • … where pi is the cost of path i

41. Optimal TE • A surprising result • This kind of link-state routing can achieve optimal TE • Optimality • Can realize the multicommodity flow traffic distribution • Expressible in terms of settings of link weights • Efficient algorithm • Computationally tractable to compute optimal weights • … for a given traffic matrix and capacitated topology

42. Intuition Behind the Theory Realizable with link-state routing Optimal flow routing Feasible flow routing

43. Finding Link-State Protocols That Achieve Optimal TE • Need an additional objective function • To find solutions expressible in terms of link weights • But we already have an objective function • So, how can we add another one? • First, solve the original optimization problem • To determine the load on each link at optimality • … i.e., the “necessary capacity” of each link • Then, solve a second optimization problem • On this new topology, with our new objective

44. TE Optimization Problem: Compute Necessary Capacity • Convex objective • Min sum f() over all links • Constraints • Flow conservation: must carry the traffic matrix • Capacity constraint: cannot exceed link capacity • Variables • Flow along each path • Given • Traffic matrix and link capacities

45. New Optimization Problem • Necessary link capacity • Flow on link u,v in the multicommdity-flow solution • … becomes the capacity of the link in the new problem • In the new optimization problem • Any feasible solution is “optimal” • … relative to the original optimization problem • So, now we can pick a new objective • Key intuition: maximizing “entropy”

46. Entropy Maximization • Assume we could enumerate all paths from s to t • (Though in practice this wouldn’t be practical) • Entropy • xks,t : fraction of traffic from s to t put on path k • z(x) = - x * log(x): entropy function • New objective: maximize entropy • S Mi,j(Sz(xks,t))

47. High-Level Overview of the Details • NEM problem always has a solution • Earlier multicommodity flow solution • Solving directly is not efficient • Need to avoid enumerating all the paths • Solving with dual decomposition • Derivation leads to the exponential function • … for splitting traffic over the multiple paths • Derivation also leads to weight-setting algorithm • Computationally efficient, better than local search

48. Conclusions • Protocols induce optimization problems • E.g., setting link weights to do traffic engineering • Complexity of the optimization problems • A symptom that the protocol is not quite right • E.g., NP-hard problem and suboptimal traffic flow • Design for optimizability • Design the protocol to be easy to optimize • … using optimization theory as a protocol design tool

49. Tomography: Inferring the Traffic Matrix Work by Yin Zhang, Matthew Roughan, Nick Duffield, and Albert Greenberg http://www.cs.utexas.edu/~yzhang/papers/tomogravity-sigm03.pdf

50. Computing the Traffic Matrix Mi,j • Hard to measure the traffic matrix • IP networks transmit data as individual packets • Routers do not keep traffic statistics, except link utilization on (say) a five-minute time scale • Need to infer the traffic matrix Mi,j from • Current topology G(R,L) • Current routing Pi,j,l • Current link load ul • Link capacity cl