Scaling The Edge Bridge Address Table In Datacenter Networks

1. Scaling The Edge Bridge Address TableIn Datacenter Networks June-2012

2. Agenda • Motivation • Protocol properties, concepts and operation • Protocol details

3. Motivation

4. Problem Statement • Address learning methods • Control plane learning • Data plane learning • Data-plane learning is simpler than control plane learningyet, it leads to bad scaling of forwarding tables • Question: can we have both data-plane learning simplicity and forwarding tables scaling?

5. Dataplane Learning On Edge Bridges (EB) • VM1BC (e.g. ARP Request) S S D D A A BC BC 1 BC 1 BC S S S S S D D D D D VM2 1 1 BC BC 1 1 BC BC 1 BC VM1 OverlayNetwork 1 1 1 A A A.1 Dataplane learning  EB table size = # of VMs in the VLAN/Tenant Domain Severe FDB Scaling Problem in EB

6. Protocol Concepts and Operation

7. Properties of The Proposed Solution • Bridge address table scaling for data-center networks with support for hot VM migration • FDB size = # of EBs in the network + # of locally attached VMs • Layer-2 only • No higher layers awareness • End point (Hypervisor) is blind to overlay network protocol • Can work with any overlay protocol

8. Protocol Concepts • The protocol defines • Data-plane format between the hypervisor and the Edge-Bridge • Modify 802.1BR or extend 802.1Qbg • Control-plane negotiates the protocol capabilities between the EB and the hypervisor • Extend DCBX 802.1Qaz • Protocol concepts • A handshake between the EB and the hypervisor • Capabilities exchange using control-plane • Dynamic operation uses the data-plane • EB • Learns addresses of local VMs & remote EBs • Uses data-plane signaling to informs the hypervisor of the path in the overlay network • Uses the path signaled by the hypervisor to forward traffic to remote VMs over the overlay network • Hypervisor • Sends data traffic to EB with path indication • Updates its path database (Path$) using the indications received from the EB

9. Protocol Databases and Signaling EB S D LocalFDB Overlay FDB S S D D Hypervisor VM2 VM1 T.Path S.Path S S D D Path$ OverlayNetwork Generated by VM Rx byVM ServerEB EBServer

10. Protocol Operation #1 • VM1VM2 flooded Unicast forwarding Learn only in B.1 S S D D A A BC BC 1 2 1 2 S D VM2 1 2 VM1 s.Path s.Path s.Path s.Path S S S S D D D D 1 1 1 1 2 2 2 2 1 1 1 1 1 1 A.1 1 Dataplane learning  EB table size = # of local VMs + # of EBs in the network

11. Protocol Operation #2 • VM2VM1 reply S D 1 2 A B D D S S VM2 1 2 1 2 VM1 T.Path S.Path S S D D 2 1 1 1 2 2 2 1 2 1 2 B.1 1 A.1 Dataplane learning  EB table size = # of local VMs + # of EBs in the network

12. Properties Of Hypervisor Path$ • Acts like ARP$ - holds active sessions only • Inactive entries are aged-out • Not contaminated by ARP-BC received from the network • Path$ entry insert/update • ETH DA is UC/MC and conforms to a VM hosted by this hypervisor, OR • ETH DA is BC and the Layer-3 DA conforms to a VM hosted by this hypervisor • Path$ entry delete/refresh • Using an activity timer

13. Protocol Details

14. Protocol Details • Control protocol • Capabilities negotiation between the Hypervisor and the Edge Bridge • Modify 802.1Qaz (DCBx) • Data-plane protocol (2 options) • Add Path-ID Tag (P-Tag) • S-channel/E-Tag is outer • P-Tag is inner: • 16b source/target-path-id • Source/target depends on direction • Modify BPE E-Tag • HypervisorEB • I-ECID – identical use to BPE • E-CID – target-path-id • EBHypervisor • I-ECID • I-ECID < 4K local virtual port (identical to BPE) • I-ECID =>4K source-path-id • E-CID – identical use to BPE

15. Summary of Protocol Properties • Bridge address table scaling for data-center networks with support for hot VM migration • FDB size = # of EBs in the network + # of locally attached VMs • Layer-2 only • No higher layers awareness • Hypervisor is blind to overlay network protocol • Can work with any overlay protocol • Easy to implement • Local scope: hypervisor to edge-bridge protocol • Simple control-plane – only need to negotiate capabilities • Extend DCBX 802.1Qaz • Simple extension of existing data-plane protocols • Modifies 802.1BR E-Tag or extends 802.1BR/802.1Qbg with a P-Tag • Easy to deploy • Co-exists with 802.1Qbg/802.1BR protocols • Support for incremental upgrade in per EB granularity

16. Detailed Packet Walkthrough Identical To The Animation

17. Walkthrough in a Nutshell (VM1VM2) #1 VM2 • VM1VM2 (VM2 ETH address is known to VM1) and back • Initial state: all FDBs are empty • Hypervisor hosting VM1 • Receive packet from VM1 • If VM2 is registered in Path$, forward with the registered T.PathElse forward with T.Path=BC • EB-A • Learn on FDB-A (VM1,A.1) • T.Path=BCFlood to Overlay and to local ports • EB-B • Replace tunnel-header with S.Path=A • Forward to VM1 if VM1 is registered in FDB-BElse flood to local ports • Hypervisor hosting VM2 • Receive the packet and update Path$ (VM1,Path=A) if: • ETH DA conforms to a VM hosted by this hypervisor, OR • ETH DA is BC and the Layer-3 DA conforms to a VM hosted by this hypervisor • Pass packet to VM2 if any of the above conditions is true VM1

18. Walkthrough in a Nutshell (VM2VM1) #2 VM2 • Hypervisor hosting VM2 • Receive packet from VM2 • VM1 is registered in Path$  send with T.Path=A • EB-B • Learn on FDB-B (VM2,B.1) • Send over Path A to EB-A • EB-A • Replace tunnel-header with S.Path=B • VM1 is registered in FDB-A (thanks to VM1VM2 path)  Forward to VM1 • Hypervisor hosting VM2 • Receive the packet and update Path$ (VM2,Path=B) if: • ETH DA conforms to a VM hosted by this hypervisor, OR • The Layer-3 DA conforms to a VM hosted by this hypervisor • Pass packet to VM1 if any of the above conditions is true VM1

19. Thank you