Scalable Edge Bridge FDB For Datacenter Networks

1. Scalable Edge Bridge FDBFor Datacenter Networks July-2012

2. Agenda • Problem statement and related work • Protocol properties, concepts and operation • Proposal for data and control planes • Summary & discussion Edge-Bridge OverlayNetwork End-Station

3. Problem Statement and Related Work

4. Problem Statement and Related Work • Problem statement • Large # of VMs in datacenters (>1M)  large address table in datacenter bridges • Support for hot VM migration  VM address must not change  address table scaling techniques based on address aggregation limit migration options • For example, IP stations can migrate within the same VLAN • Overlay networks solve address scaling problem in Core Bridges • Core Bridge address table ~= # Edge Bridges << # of VMs in the network • Lot’s of work on overlay protocols: SPB, PBB, VPLS, TRILL, VXLAN, NVGRE • How to scale the address table in Edge Bridges (EB)? • VXLAN/NVGRE – specific solutions for IP overlay • SPB/TRILL – none (July-2012) • Objective: provide a solution to address scaling in SPB Edge Bridges • The solution must complement (not replace) overlay network protocols • Preferably, one solution should fit many overlay network protocols, so it can be easily adapted to work with other overlay protocols

5. Bridge FDB Scaling(BFS)Protocol Concepts and Operation

6. Bridge FDB Scaling (BFS) Concepts • BFS defines a handshake between the EB and the End-Station(An End-Station may host 1 or more VMs) • Capabilities exchange use control-plane • Dynamic operation uses the data-plane • EB operation in a nutshell • Learns addresses of local VMs & remote EBs (but not remote VMs) • Uses data-plane signaling to informs the End-Station of the path in the overlay network • Uses the path signaled by the End-Station to forward traffic to remote VMs over the overlay network • End-Station operation in a nutshell • Sends data traffic to EB with path indication • Updates its path database (Path$) using the indications received from the EB

7. BFS Databases and Signaling Edge Bridge S D LocalFDB Overlay FDB S S D D End-Station VM2 VM1 T.Path S.Path S S D D Path$ OverlayNetwork Generated by VM Rx byVM ServerEB EBServer

8. EB Operation • Overlay FDB learning • Control plane triggered as specified by the overlay protocol (e.g. IS-IS for SPB) • Address learning process (Local FDB) • Data-plane learning • Don’t learn on overlay ports • Learn on local ports • Forwarding packets received on local ports • If packet has no T.Path indication Lookup in local FDB using DA if found  forward accordingly, don’t assign S.Path to traffic to local ports else flood to local and overlay portselse // packet has T.Path indication Obtain the overlay path attributes using T.Path Remove T.Path, add ovelay tunnel Send to overlay • Forward packets received on overlay ports • Lookup overlay FDB with the overlay header, obtain S.PathRemove overlay header, assign S.PathLookup local FDB with DAif found, forward accordinglyelse flood to local ports

9. End-Station Operation • Forwarding packets received from VM • Lookup Path$ with DAIf found, assign T.Path to the packet and forward to EBelse forward to EB w/o T.Path • Forward packets received from EB • Use DA or 802.1Qbg/802.1BR indication to forward to the VM • Path$ update policy (packets received from EB) • If packet has no S.Path, don’t update Path$else // packet has S.Path update Path$ if any of the following is met DA indicates a VM hosted by this End-Station, OR DA=BC and L3-DA indicates a VM hosted by this End-Station

10. BFS Operation Example #1 • VM1VM2 flooded Unicast forwarding Learn only in B.1 S S D D A A BC BC 1 2 1 2 S D VM2 1 2 VM1 SPB Overlay s.Path s.Path s.Path s.Path S S S S D D D D 1 1 1 1 2 2 2 2 1 1 1 1 1 1 1 A.1 Dataplane learning  EB table size = # of local VMs + # of EBs in the network

11. BFS Operation Example #2 • VM2VM1 reply S D 1 2 A B D D S S VM2 1 2 1 2 VM1 SPB Overlay S.Path T.Path S S D D 2 1 1 1 2 2 1 1 2 2 A.1 2 1 B.1 Dataplane learning  EB table size = # of local VMs + # of EBs in the network

12. BFS Data and Control Planes(A Proposal)

13. BFS Data and Control Planes - A Proposal • Control protocol • Capabilities negotiation between the End-Station and the Edge Bridge • Modify 802.1Qaz (DCBx) • Data-plane protocol (2 options) • Add Path-ID Tag (P-Tag) • S-channel/E-Tag is outer • P-Tag is inner: • 16b source/target-path-id • Source/target depends on direction • Modify BPE E-Tag • End-StationEB • Ingress-ECID – identical use to BPE • E-CID – target-path-id • EBEnd-Station • Ingress-ECID • Ingress-ECID < 4K local virtual port (identical to BPE) • Ingress-ECID =>4K source-path-id • E-CID – identical use to BPE

14. Summary

15. Summary of BFS Properties • Complements SPB towards scaling the EB FDB • A generic solution that can be considered for additional overlay protocols • Small Path$ in End-Station • Holds active sessions only – comparable in size to the ARP$ • Easy to implement • Local scope: end-station to edge-bridge protocol • Simple control-plane – only need to negotiate capabilities, no dynamic operation • Extend DCBX 802.1Qaz • Simple extension of existing data-plane protocols • Extends 802.1BR/802.1Qbg with a P-Tag or modifies 802.1BR E-Tag • Easy to deploy • Co-exists with 802.1Qbg/802.1BR protocols • Support for incremental upgrade per EB granularity

16. Thank youContact: Carmi Arad, carmi@marvell.com