Oracle Training Materials – Usage Agreement

1. The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remain at the sole discretion of Oracle.

2. Oracle Training Materials – Usage Agreement Use of this Site (“Site”) or Materials constitutes agreement with the following terms and conditions: 1. Oracle Corporation (“Oracle”) is pleased to allow its business partner (“Partner”) to download and copy the information, documents, and the online training courses (collectively, “Materials") found on this Site. The use of the Materials is restricted to the non-commercial, internal training of the Partner’s employees only. The Materials may not be used for training, promotion, or sales to customers or other partners or third parties. 2. All the Materials are trademarks of Oracle and are proprietary information of Oracle. Partner or other third party at no time has any right to resell, redistribute or create derivative works from the Materials. 3. Oracle disclaims any warranties or representations as to the accuracy or completeness of any Materials.  Materials are provided "as is" without warranty of any kind, either express or implied, including without limitation warranties of merchantability, fitness for a particular purpose, and non-infringement. 4. Under no circumstances shall Oracle or the Oracle Authorized Delivery Partner be liable for any loss, damage, liability or expense incurred or suffered which is claimed to have resulted from use of this Site of Materials. As a condition of use of the Materials, Partner agrees to indemnify Oracle from and against any and all actions, claims, losses, damages, liabilities and expenses (including reasonable attorneys' fees) arising out of Partner’s use of the Materials. 5. Reference materials including but not limited to those identified in the Boot Camp manifest can not be redistributed in any format without Oracle written consent.

4. ORACLE PRODUCT LOGO Exalogic Elastic Cloud INetworking <presenter’s name> <presenter’s info>

5. Agenda • Overview of Exalogic Networking • Internal Management Networking • Internal Networking • Ethernet Gateway • Network Connection Requirements • Network Deploment Considerations • Summary

6. Overview of Exalogic Networking

7. Exalogic Networking • Private InfiniBand Network • Primary path for all traffic inside rack • Inter-Compute Node communication • Compute Node to ZFS Storage Appliance • Compute Node to Exadata DB Server node • Client Access (10GbE Datacenter Interconnect) • Bridged from InfiniBand fabric to Ethernet • No 10GbE switching in the rack, presented as NICs to datacenter • Management Network (Ethernet) • 1GbE network connects all devices through built-in switch • Three Networks – Physical Environment

8. Exalogic Application Network Protocols Example on Encapsulation: IPoIB

9. Exalogic Application Network Protocols EoIB SDP IPoIB Native IB • Providing Performance and Compatibility Application User Applications OTD JDBC Exabus Sockets API Exabus Java/C++ APIs Exabus Sockets API TCP TCP IP IP Kernel Kernel Bypass RDMA Semantics NIC Driver IP over IB SDP IB Core IB Core Std. NW Stack HCA Driver HCA Driver IB HCA IB HCA IB HCA Ethernet NIC High-speed IB fabric, no application change required Bypass TCP/IP layer to get even higher performance, minimal app changes Highest performance, Native IB support Hardware

10. Exalogic Networking • This example illustrates a physical OS runninng on the Compute Node • Three network interfaces (Physical Environment) • bond0 for Private IB Network • bond1 for Client Access Network • eth0 for Management Network • bond1 uses the virtual network devices (vNICs) provisioned from the NM2-GW switches • Virtual network interfaces in virtualized Exalogic environment(vNet) • Three Network Interfaces –ComputeNode Compute Node (Physical OS) NM2-GW02 NM2-GW01 IPoIB, SDP, Native IB bond0 HCA ethAB ib0 bond1 EoIB ib1 ethCD createvnic on NM2-GW to provision vNICs TCP/IP eth0 Datacenter 10GbE Switch Datacenter 10GbE Switch MGMT NET3 NET2 NET1 NET0 Cisco Mgmt Switch

11. Exalogic Networking Exalogic Private IB Network Exalogic Control • Virtualized Environment

12. Exalogic Internal Management Network

13. Exalogic Integration IInternal Management Net. Exalogic Rack • Standard GbE Switch • Cat 5 UTP (no fiber) • Not HA • ILOM access • Firmware updates • Remote monitoring • HW configuration management Compute Nodes InfiniBand Gateways Storage System GbE Management Switch Compute Nodes InfiniBand Spine Switch

14. Exalogic Internal Network

15. Exalogic Integration IExabus (InfiniBand) Exalogic Rack • Internal I/O Back-plane • 40 Gb/s per compute node • Built on QDR InfiniBand • Fully redundant • Built in security and QoS • Unique Software/Firmware • Ethernet over InfiniBand bridging • Exabus RDMA-based APIs Compute Nodes InfiniBand Gateways Exabus Storage System Management Switch Compute Nodes InfiniBand Spine Switch

16. Exabus • New C++ and JAVA Async APIs for maximum applicationperformance • Avoid user/kernel-level context switches • Avoid interrupts • Avoid thread switches • Use Native OF-UV IB RDMA API • Significantperformanceimprovements in throughput and latency • Optimized Application Platform Weblogic OTD Coherence Tuxedo Any Linux or Solaris Application Java APIs Exalogic SDP Messagebus TCP/IP ExabusRDMA(C++ API) IPoIB EoIB ExabusInfiniBand Core (OFED) Stack Hardware and Firmware Exalogic Optimization Standard

17. Private InfiniBand Network Compute Node Compute Node NOTE: • In “Physical” Exalogic, Compute Nodes utilize network aliases for bond0 for VIP or Floating IP addressess for application needs • In “Virtualized” Exalogic, Compute Nodes host vServers where virtual network devices are provisioned (e.g. vnetNN) • Application Communications Compute Node Compute Node ISV App OTD ISV App OTD Logical View Oracle App FMW/WLS Oracle App FMW/WLS HCA HCA IPoIB SDP Native IB bond0 bond0 ib0 ib0 Tuxedo Coherence Tuxedo Coherence ib1 ib1 Exabus Storage Node Exadata DB

18. Private InfiniBand Network SDP IPoIB Native IB Apps, FMW, WLS, OTD, Coherence, Tuxedo, etc. • Application Protocols User Applications OTD JDBC Exabus Sockets API Exabus Java/C++ APIs Exabus Sockets API TCP IP Kernel Kernel Bypass RDMA Semantics IP over IB SDP IB Core IB Core HCA Driver HCA Driver IB HCA IB HCA IB HCA High-speed IB fabric, no application change required Bypass TCP/IP layer to get even higher performance, minimal app changes Highest performance, Native IB support Hardware

19. InfiniBand Partitions Application A Network • Define a group of InfiniBand nodes (e.g. Compute Nodes, ZFSSA) that are allowed to communite with one another • IB partitions provide Network Isolation • Similar concept as Ethernet VLANs • Infiniband Header has PKeyinformation • Packet inspections: Discard or Forward rules enforced at switch port and HCA • Partition keys (PKey) used for security • Physical interface is ib0 (default PKey 0xffff) • Virtual interface Examples on a compute node • vNIC eth6 and eth7 with Pkey 8007 and IP Address 192.168.10.17 • ib0.8008 and ib1.8008 => bond7 (172.123.170.21) • Network Isolation and Security [Ethernet] 192.168.10.17/24 (VLAN 110) [Infiniband] 192.168.10.17/24 (PKey 8007)

20. InfiniBand Partitions • IB partitions provide Network Consolidation • Same physical InfiniBand port • Same physical InfiniBand cable • Same physical InfiniBand switch port • Multiple PKeystranslate into multiple Layer-3 IPoIBdatalinks • Network Consolidation bond0 bond5 bond6 bond7 bond8 NM2-GW 1 Port1 DataCenter Switch01 Compute Node NN (IB HCA) eth7 ib1.8001 ib1.8006 ib1.8007 ib1.8008 DataCenter Switch02 Port2 ib0.8006 ib0.8007 ib0.8008 eth6 ib0.8001 NM2-GW 2

21. InfiniBand Partitions • Infiniband Memberships • Memberships are partition key based • Compute nodes can participate in multiple PKeyswith varying roles • Two types of Partition Membership • Limited • Full • Partition Memberships

22. InfiniBand Partitions • Limited Members • Can not communicate to other limited members • Can communicate with full members in same partition • One to many • e.g. ZFSSA NFS server as Full and Nodes as Limited • Full Members • Can communicate with limited members in same partition • Can communicate with full members in same partition • All to All • e.g. Cell nodes in Exadata • Partition Memberships

23. Partition Memberships TrafficDirector Application A Storage Head Application B EL CN1 EL CN2 EL SH1 EL CN3 Linux Linux Linux Full Member View BOND1(vNIC 1+2) BOND0(IPoIB) BOND0(IPoIB) ipmp0(IPoIB) BOND1(vNIC 3+4) BOND0(IPoIB) LM LM LM FM 10.0.11.0 / 27 IB Fabric TrafficDirector Application A Storage Head Application B EL CN1 EL CN2 EL SH1 EL CN3 Linux Linux Linux “Application A” View BOND1(vNIC 1+2) BOND0(IPoIB) BOND0(IPoIB) ipmp0(IPoIB) BOND1(vNIC 3+4) BOND0(IPoIB) LM FM 10.0.11.0 / 27 IB Fabric

24. Exalogic Control Networks (Configured during ECU)

25. Legend Eth Management Network Guest VM User created networks DB Eth Management Customer VMs IPoIB-default ECs IPoIB-admin Compute Nodes (OVS) IPoIB-storage IPoIB-virt-admin IPoIB-ovm-mgmt PCs IPoIB-vserver-shared-storage ILOM EoIB-external-mgmt Membership1 F : Full L : Limited B : Both2 Storage Appliance OVMM Storage Heads Eth Management network has no membership “Both” Membership refers to the way partitions are implemented for the virtual platform. vServers on top of OVS can have limited and full memberships for different PKeys. ILOM All Switches & PDUs

26. Legend Default IPoIBNetwork (PKey: 0xFFFF) Guest VM User created networks DB Eth Management Customer VMs IPoIB-default ECs IPoIB-admin Compute Nodes (OVS) IPoIB-storage IPoIB-virt-admin IPoIB-ovm-mgmt PCs F IPoIB-vserver-shared-storage ILOM EoIB-external-mgmt Membership1 F : Full L : Limited B : Both2 Storage Appliance OVMM F Storage Heads Eth Management network has no membership “Both” Membership refers to the way partitions are implemented for the virtual platform. vServers on top of OVS can have limited and full memberships for different PKeys. ILOM All Switches & PDUs

27. Legend Admin IPoIBNetwork (PKey: 8001) Guest VM User created networks DB F Eth Management Customer VMs L IPoIB-default ECs F IPoIB-admin Compute Nodes (OVS) IPoIB-storage IPoIB-virt-admin IPoIB-ovm-mgmt B PCs IPoIB-vserver-shared-storage F ILOM EoIB-external-mgmt Membership1 F : Full L : Limited B : Both2 Storage Appliance OVMM F F Storage Heads Eth Management network has no membership “Both” Membership refers to the way partitions are implemented for the virtual platform. vServers on top of OVS can have limited and full memberships for different PKeys. ILOM All Switches & PDUs

28. Legend Storage IPoIBNetwork (PKey: 8002) Guest VM User created networks DB L Eth Management Customer VMs IPoIB-default ECs IPoIB-admin L Compute Nodes (OVS) IPoIB-storage IPoIB-virt-admin B IPoIB-ovm-mgmt PCs IPoIB-vserver-shared-storage L ILOM EoIB-external-mgmt Membership1 F : Full L : Limited B : Both2 Storage Appliance OVMM L F Storage Heads Eth Management network has no membership “Both” Membership refers to the way partitions are implemented for the virtual platform. vServers on top of OVS can have limited and full memberships for different PKeys. ILOM All Switches & PDUs

29. Legend Virt-Admin IPoIBNetwork (PKey: 8003) Guest VM User created networks DB Eth Management Customer VMs F L IPoIB-default ECs IPoIB-admin Compute Nodes (OVS) F IPoIB-storage IPoIB-virt-admin B IPoIB-ovm-mgmt PCs IPoIB-vserver-shared-storage ILOM EoIB-external-mgmt F Membership1 F : Full L : Limited B : Both2 Storage Appliance OVMM F Storage Heads Eth Management network has no membership “Both” Membership refers to the way partitions are implemented for the virtual platform. vServers on top of OVS can have limited and full memberships for different PKeys. ILOM All Switches & PDUs

30. Legend OVM Mgmt IPoIBNetwork (PKey: 8004) Guest VM User created networks DB Eth Management Customer VMs F IPoIB-default L ECs IPoIB-admin Compute Nodes (OVS) IPoIB-storage F B IPoIB-virt-admin IPoIB-ovm-mgmt PCs IPoIB-vserver-shared-storage ILOM EoIB-external-mgmt F Membership1 F : Full L : Limited B : Both2 Storage Appliance OVMM F Storage Heads Eth Management network has no membership “Both” Membership refers to the way partitions are implemented for the virtual platform. vServers on top of OVS can have limited and full memberships for different PKeys. ILOM All Switches & PDUs

31. Legend vServer Shared Storage IPoIB (PKey: 8005) Guest VM User created networks DB Eth Management Customer VMs IPoIB-default ECs L IPoIB-admin Compute Nodes (OVS) IPoIB-storage B F IPoIB-virt-admin IPoIB-ovm-mgmt PCs IPoIB-vserver-shared-storage ILOM EoIB-external-mgmt F Membership1 F : Full L : Limited B : Both2 Storage Appliance OVMM F Storage Heads Eth Management network has no membership “Both” Membership refers to the way partitions are implemented for the virtual platform. vServers on top of OVS can have limited and full memberships for different PKeys. ILOM All Switches & PDUs

32. Legend External Mgmt EoIBNetwork (PKey: 8006) Guest VM User created networks DB Eth Management Customer VMs IPoIB-default ECs IPoIB-admin Compute Nodes (OVS) IPoIB-storage IPoIB-virt-admin F IPoIB-ovm-mgmt PCs IPoIB-vserver-shared-storage ILOM EoIB-external-mgmt Membership1 F : Full L : Limited B : Both2 Storage Appliance OVMM Storage Heads F Eth Management network has no membership “Both” Membership refers to the way partitions are implemented for the virtual platform. vServers on top of OVS can have limited and full memberships for different PKeys. ILOM All Switches & PDUs

33. Legend Network Mapping Diagram Guest VM User created networks DB F L Eth Management Customer VMs F L F L IPoIB-default L ECs L F IPoIB-admin L Compute Nodes (OVS) F IPoIB-storage B F B F IPoIB-virt-admin B F B IPoIB-ovm-mgmt B PCs F IPoIB-vserver-shared-storage F L ILOM EoIB-external-mgmt F F F Membership1 F : Full L : Limited B : Both2 Storage Appliance OVMM F F L F F F F F Storage Heads F Eth Management network has no membership “Both” Membership refers to the way partitions are implemented for the virtual platform. vServers on top of OVS can have limited and full memberships for different PKeys. ILOM All Switches & PDUs

34. Network Mapping Table

35. Ethernet Gateway

36. Exalogic Networking Client Access Network Requires 10GbE Connection • Client Access Network for Ingress/Egress

37. Ethernet over InfiniBand Exalogic NOTE: • EoIBNetwork Interfaces must be provisioned on NM2-GW (createVNIC) and configured them on desired compute nodes • Chapter 10 Exalogic Machine Owner’s Guide • Network Interfaces Client Network Compute Node Compute Node NM2-GW ISV App OTD 10GbE Switch HCA Oracle App FMW/WLS ethAB ib0 EoIB bond1 ethCD ib1 10GbE Switch Tuxedo Coherence NM2-GW

38. Ethernet over IB • Used with the Sun NM2-GW(Leaf) switch for bridging IB and Ethernet networks • Protocol defines • the procedure for tunneling (encapsulating) both unicast and multicast Ethernet packets over IB using the Unreliable Datagram (UD) transport • The Ethernet packet includes the standard header, VLAN tags, if appropriate, and the payload • control messages used by Gateway management software to associate IB connected servers with Ethernet ports on Gateway • EoIB Protocol

39. Ethernet over IB • Networking stack works seamlessly over EoIB NICs • EoIBprotocol server-side endpoint in OS is implemented as standard network driver (NIC) • EoIB NICs behave like regular Ethernet NICs • For example, ifconfig bond1 • EoIB Protocol

40. Exalogic Networking • Purpose of the Gateway • Allow hosts on the InfiniBand fabric to share one or more 10 GbE links to an external LAN • Allow the hosts sharing a 10GbE link to communicate between each other as well as with nodes on the external LAN as if they all had a private NIC connected to the external LAN • Implementation of the Gateway • Shared NIC Model – The GW is an endpoint on both InfiniBand and Ethernet side (i.e. not a switch or router etc.) • Hosts on the IB fabric communicate with nodes on the Ethernet LAN via the GW, but with other hosts on the InfiniBand fabric directly. However, logically they are all part of the same L2 Ethernet subnet seen from the external LAN • Ethernet Gateway for Exalogic – NM2-GW Gateway Switch

41. Exalogic Networking • NM2-GW support eight 10GbE ports • Shared NIC model for each 10GbE port • Each vNIC is bound to a single 10GbE port • 10GbE port is accessible from a server through IB using EoIB • Multiple VLANs per port supported (4k) • Each server (compute node) has access to two 10GbE ports on separate Gateways for HA. For each external physical LAN, Exalogic is connected to: • Each 10GbE port represented as an EoIBvNIC at the server • The EoIBvNIC is viewed as a normal Ethernet NIC by the OS • Two Bonded vNICs per server – Active/Passive – for HA • vNIC MAC address derived from IB HCA Port GUID • Number of servers sharing a 10GbE port is configurable • Up to 8 servers per 10GbE port • Ethernet Gateway for Exalogic – NM2-GW Gateway Switch 0 1 2 3

42. Exalogic Networking [root@el01sw-gw1 ~]# listlinkup | grep Bridge Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled) Bridge-0 Port 0A-ETH-2 (Bridge-0-2) up (Enabled) Bridge-0 Port 0A-ETH-3 (Bridge-0-1) up (Enabled) Bridge-0 Port 0A-ETH-4 (Bridge-0-1) up (Enabled) • NM2-GW Gateway Switch – 10GbE Ports • Eight 10GbE ports per NM2-GW • Each server has access to two 10GbE ports on separate NM2-GW switches for HA • Upto 8 servers per 10GbE port • Configurationdepends on • #LANs • Bandwidthrequirements

43. Exalogic Networking • Owner‘s Guide, Section 6.4.1 • #LANs: How many LANs to connect to? • EIS Hardware Installation: 1 LAN • Bandwidth requirements: How many 10GbE ports per LAN (how many compute nodes share one 10GbE port)? • Which 10GbE ports to use on each NM2-GW switch, for which compute nodes? • Configuration determines • Quantity of cables and transceivers • Ports assignment on NM2-GW switches and customer‘s switch(es) • Configuration needs to be planned by Sales Account Team with customer before order entry • NM2-GW Gateway Switch – 10GbE Ports

44. This 1 LAN configuration willbe deployed as part of the EIS Hardware Installation Exalogic Networking • Two bonded VNICs per server are connected to different NM2-GW (and corresponding 10 GbE ports) • Exalogic Edge Network HA – 1 LAN

45. Multiple LANs will NOTbe deployed as part of the EIS Hardware Installation Exalogic Networking • Two Bonds per Server – one Bond per LAN (Bond 1 and Bond 2) • Exalogic Edge Network HA – 2 LANs

46. VLANs will NOTbe deployed as part of the EIS Hardware Installation Exalogic Networking • Two Bonds per Server – one Bond per VLAN (Bond 1 and Bond 2) • Exalogic Edge Network HA – 2 VLANs

47. InfiniBand Terminology MPO-to-MPO Cable MPO-to-4LC Cable LC Connectors (Male) QSFP SR Transceivers (Female)

48. Exalogic Networking • For each Exalogic system additional 10GbE networking components must be ordered from Oracle and also provided from customer or third party • No singlestandard configuration for all deployments • Specific 10GbE network configuration needs to be planned by Sales Account Team (PreSales) with customer depending on customer requirements before order entry • Exalogic Elastic Cloud: Datacenter Network Integration http://www.oracle.com/us/products/middleware/exalogic/eec-datacenter-network-integration-1405367.pdf • Required External 10GbE Connectivity

49. Exalogic Networking • Two basic network deployment options • “Optimal” Deployment: Datacenters with existing 10GbE L2 infrastructure • “General” Deployment: Datacenters without an existing 10GbE L2 infrastructure or with unique connectivity requirements • Based on these two options, there are numerous specific network configurations possible depending on customer infrastructure • Each configuration requires different transceivers, cables, and switches • All details are published in the Exalogic Network Integration Guide • Required External 10GbE Connectivity

50. NM2-GW Deployment Options Exalogic IB GW Switch Sun Top of the Rack (ToR) Switch DataCenter Switches