Andy Lake, John Vollbrecht Internet2 Nasir Ghani University of New Mexico Nagi Rao

1. ESCC/Internet2 Joint Techs Winter Meeting January 22, 2008 University of Hawaii Honolulu, Hawaii Multi-Layer, Multi-Domain Control Plane Hybrid Networks ArchitectureCurrent Status and Future Issues Andy Lake, John Vollbrecht Internet2 Nasir Ghani University of New Mexico Nagi Rao Oak Ridge National Laboratory Tom Lehman, Xi Yang Information Sciences Institute East University of Southern California Chin Guok Network Engineering Services Group ESnet

2. Multi-Domain, Multi-Layer Hybrid Networks • Hybrid networks are intended to provide a flexible mix of IP routed service and dedicated capacity “circuits” • The “Multi-Layer” is meant to identify several items regarding how hybrid networks may be built. In this context it includes the following: • Multi-Technology - MPLS, Ethernet, Ethernet PBB-TE, SONET, NG-SONET, T-MPLS, WDM • Multi-Level - domains or network regions may operate in different routing areas/regions, and maybe be presented in an abstracted manner across area/region boundaries • Multi-Domain indicates that we want to allow hybrid network service instantiation across multiple domains • And of course all this implies that this will be a Multi-Vendor environment.

3. Multi-Layer, Multi-Domain Control Plane Hybrid Networks Control Plane Where are today? What is interesting to think about for the future?

4. Hybrid Networks Control Plane - Today • Currently an early deployment of a Hybrid Network Control plane on ESnet SDN and Internet2 DCN • and some regional's such as DRAGON, NYSERNet, others • This evolved out of collaborations between several projects including ESNet OSCARS, I2 BRUW, NSF DRAGON, and the DICE Group • It is expected that this will evolve as standards bodies and other groups work on developing interface specifications/agreements with the larger community • Key features of the current architecture are noted in the following bullets. • One InterDomain Controller (IDC) per domain which is responsible for inter-domain messaging • A “Domain Controller” (DC) which takes care of provisioning inside the domain. • The DC is really an internal domain concern • The DC design will vary by domain (based on technology types, vendor capabilities, etc), and in some instances may be a very minimal set of functions • The IDC/DC combination provides the basis for a two-level hierarchical network view. Where the DC will have knowledge of the real local topology and the IDC may have a reduced or abstracted view.

5. Hybrid Networks Control Plane - Today Four distinct phases are identified for IDC communications: User Request, Topology Exchange, Resource Scheduling, Signaling Topology Exchange: currently based on abstracted link states, with little to no dynamic information. We are also planning to investigate use of a path vector style of inter-domain information exchange. Resource Scheduling: multi-domain, multi-stage path computation process where the specific resources get identified and reserved for a specific signaling event. The Signaling Phase is where specific network elements are provisioned. This phase may be initiated by the user, or by the domains. The Signaling Phase actions are based on resources identified in the Resource Scheduling phase. User Request Phase provides a message set for users to request multi-domain circuits Current security and authentication models are based on signed soap messages and X509 Certificates (User to local IDC messaging; IDC to IDC messaging)

6. Hybrid Networks Control Plane - Today • Four Primary Web Services Areas: • Topology Exchange, Resource Scheduling, Signaling, User Request

7. Hybrid Networks Control Plane - Today MetaScheduler Topology Topology Scheduling Scheduling Signaling Signaling • Meta-Scheduler Approach • Same set of Web Services used for linear instantiation model can be used by a high level process to build services: • Topology Exchange, Resource Scheduling, Signaling, User Request • A key issue is that this requires a trust relationship between the “meta-scheduler” and all the domains with which it needs to talk

8. Hybrid Network Control PlaneWhat can we do today? Dynamic provision of end to end (circuits) across multiple domains. Specify a few basic parameters regarding a single circuit request: edge technology/configuration, bandwidth, end points, domain sequence, specific start/stop times

9. Hybrid Network Control PlaneWhat is interesting to think about for the future? Enhanced Circuit Parameters and User Request Mechanisms Richer set of flexible circuit request constraints such as technology type, flexible time period specifications, latency, jitter, adjustments to in-service circuits, arbitrary business /administrative/security constraints, flexible user requests mechanisms. Topology Building: combining multiple individual circuits together into a user specified topology. Network Virtualization - True Multi-Level, Multi-Technology, Multi-Vendor network control and provisioning Only talking about network resources here; correlation with application domain resources is considered a separate activity.

10. Enhanced User Options • User has increased number of parameters to specify such as technology type, adjustments to in-service circuits, flexible time period specifications, latency, jitter, arbitrary business /administrative/security constraints, flexible user requests mechanisms • Green Path might be chosen in response to user specifying domain preference, latency/jitter requirement, or something else

11. Multi-Level, Multi-Technology, Multi-Vendor Infrastructures • Multiple Options, most will have vendor proprietary control and management mechanisms which only work across single vendor regions Ethernet Layer Routers Switched WDM Optical Layer Ethernet PBB-TE Switched WDM Optical Layer Ethernet Layer Switched SONET Layer (vcat, lcas)

12. Multi-Level, Multi-Technology, Multi-Vendor Infrastructures • Current dynamic provisioning environment can be described as: • Static Topology, Dynamic Provisioning • Next we want to enable: • Dynamic Topology, Dynamic Provisioning

13. Multi-Level, Multi-Technology, Multi-Vendor – Network Virtualization • Network Virtualization and Topology Building in Multi-Level, Multi-Technology, Multi-Vendor Infrastructures Bandwidth Request was smaller, so provision Ethernet, then router connection Same Result with Either Approach Provisioned Topologies Routers Ethernet PBB-TE Bandwidth Request was large enough to justify provisioning at WDM layer Switched WDM Optical Layer End Points might attach at different levels: How do flexibly provision at what ever level an end point might appear?

14. What are the major issues looking forward? A key requirement for the architecture is to be able to handle the reality that the underlying networks will be very heterogeneous in terms of technology, control mechanisms, and vendors. In the current architecture this is abstracted out by the DC to IDC interface. Four types of underlying domain types have been identified in terms of how the DC interacts with them: GMPLS (I2 DCN is an example, regional networks based on ethernet switch dynamic provisioning is another example) MPLS (ESNet SDN is an example) Management Plane Controlled (USN is an example) Vendor Control Plane (I2 DCN also has a component of this)

15. Dealing with Heterogeneous Network Technologies and Vendor Equipment Adding regions of new technologies and vendors is not too difficult from the provisioning perspective The difficult issue is in terms of the routing exchange between/from the technology/vendor regions and path computation (intra and multi-domain) with multiple constraints. IDC IDC IDC DC DC DC GMPLS MPLS Management Plane • As an Example, DRAGON is used as the DOMAIN Controller for I2 DCN Ciena Core Directors IDC to other domain IDCs to other domain IDCs DRAGON GMPLS Control Plane GMPLS to other domains GMPLS to other domains DRAGON uni, tl1 uni, tl1 Ciena Region subnet signaling flow CD_a CD_z

16. Multi-Constraint Path Computation IntraDomain provisioning requires a path computation process to determine a path across the local network If the domain consists of multiple technologies, multiple levels, and multiple vendors this problem can be complex In order to realize the advanced control plane features multi-domain path computation needs to be augmented to operate in these environments. This will likely include addition of the following constraints to the path computation process: time domain flexible set of AAA and other user defined constraints Ability to look for paths as a group in the context of a entire topology build. These scheduling and flexible policy processing mechanisms will need to be tightly integrated/coupled with path computation and selection processes

17. Flexible and Policy Based Multiple Constraint Path Computation with Filtering/Pruning Processes Data source (raw link states from intra- and inter-domain flooding) and 3D constraints Snapshot of topology reduced by policy filters Constraint based path computation algorithm - CSPF heuristics

18. Path Computation with Multiple Dimensions • Resource dimension • Link availability, bandwidth capability & resource interdependence • TE constraints, e.g. switching cap. • AAA policy dimension • User privileges • App. specific requirements (SLA) • Administration policies • Time schedule dimension • Integrate and translate network resource states and policies into shared control plane intelligence. • Synergize AAA policy decision with TE based provisioning decision, resulting in fast, precise and simplified control process.

19. Control Plane Scalability and Performance – Simulations and Testing In order to collect some more data on the (current and future) control plane performance, we are planning to run some simulations on OPNET and/or User Mode Linux (UML) environments. This will allow us to evaluate the scalability/stability of inter-domain information exchange, the success/blocking probability/performance of Resource Scheduling and Signaling. User Mode Linux (UML) based simulation will also allow us to connect simulated networks to real networks, since the UML will run the same software as current networks. Current and future designs will be evaluated

20. Questions/Comments? • Tom Lehman • tlehman at east.isi.edu • Thank You