High-Performance Networking With NDIS 6.0, TCP Chimney Offload, and RSS

1. High-Performance Networking With NDIS 6.0, TCP Chimney Offload, and RSS Vik DesaiProgram ManagerWindows NetworkingMicrosoft Corporation

2. Appropriate Audience Who should attend this session? • Networking product builders • Product decision makers • Hardware and software engineers • Architects • Network designers and deployers • IT Managers • IT Consultants • Venture Capitalists and Private Investors • Industry analysts

3. Agenda • Networking stack challenges • Scalable networking goals • Scalable networking architecture • Receive Side Scaling (RSS) • TCP Chimney Offload • Scalable networking demo • NetXen Demo – Vikram Karvat • Broadcom Demo – Uri Elzur • Offload roadmap • Summary and Call to Action

4. Networking Challenges • Receive processing limited to a single CPU on a multi-processor system • CPU utilized in Protocol Processing increases with Physical layer speeds • Data Movement between network and application buffers is a bottleneck • Large number of Interrupts even with Interrupt Moderation

5. Scalable Networking Goals • Boost application scalability on 1 GB and 10 GB Ethernet with an integrated architecture • That preserves standard infrastructure (1500b MTU) • That maintains standard network and server management practices • That does not compromise security, server reliability, and application compatibility • Enable Ethernet fabric convergence • Robustly support new class of protocol offload NICs in Microsoft Windows

6. Receive Side Scaling • Networking Challenge • Receive processing limited to a single CPU on a multi-processor system • Solution • Parallelize receive processing by Queuing incoming packets to multiple CPUs • Implementing Solution via RSS • NIC manages multiple hardware queues • NIC hashes incoming TCP segments to different hardware queues • NIC driver requests DPCs on appropriate CPUs

7. ReceiveFIFO InterruptLogic Regular NIC RSS Description –Non RSS Capable NIC Processor 0 D P C APP TCPIP ISR NDIS Incoming Packet

8. RSS Description –RSS Capable NIC Processor 0 Processor 1 Processor 2 D P C D P C D P C APP APP APP TCPIP TCPIP TCPIP NDIS NDIS NDIS ISR RSS CapableNIC InterruptLogic Incoming Packet ToeplitzHash ReceiveFIFOs

9. TCP Chimney Offload • Networking Challenges • Data Movement between networkand application buffers is a bottleneck • Large number of Interrupts even with Interrupt Moderation • CPU utilized in Protocol Processing increaseswith Physical layer speeds • Solution • Provide Zero Copy solution for pre posted buffers • Change interrupts from a per packet basis to a per segment basis • Offload Protocol Processing to hardware

10. TCP Chimney Architecture Applications Other Misc. Layers Switch TCP Chimney Interfaces Transport Layer (TCP) State Updates Path Layer IPv4 or IPv6 Data Transfer Framing Layer (Ethernet) NDIS 5.2 / 6.0 NDIS Miniport Driver TCP Chimney Offload Capable Hardware

11. TCP Chimney Interface Details • TCP/IP States Divided into • Const State – Does not change for connection lifetime • Cached State – Controlled by host stack and updated appropriately to offload target • Delegated State – Controlled by Offload Target • NDIS Supports • Offload Capability Advertisement • Interface to transfer and update state information • Interface to query statistics • Interface to transfer data

12. TCP Chimney Initialization • Offload Manager determines suitabilityof connection for offload • State from each layer is captured and transferred to offload target • Incoming Data packets/outgoing sends are queued • Data packets will be replayed to offload targets for successful offload attempts • Data packets will be processed by stackfor unsuccessful offload attempts • Data Transfer Begins

13. TCP Chimney Data Transfer • Sends • Segment passed to offload target for completion • Send Completions after end-to-end TCP Ack • Receive • If no receive buffers posted indicate data • If receive buffers are posted indication occurs as appropriate • OOB/Urgent Data passed to Host Stack

14. TCP Chimney Connection Teardown • Connections can be uploaded/offloaded at any time • Heuristics Manager tracks connections appropriate for upload/offload • Half Closed Connections are not uploaded • Upload request initiated by offload target • Offload target to provide delegated state to host stack • Offload target keeps connection state till host sends upload call

15. TCP Chimney Implications • IPsec Chimney required for IPsec traffic • Will not work with • IM drivers incapable of understanding Chimney interfaces • Hooking Firewalls • Best benefits for • Long Lived Connections • Pre-posted Receive Buffers • Large Application IO Sizes

16. 10GbE Chimney Offload Vikram Karvat VP Marketing vkarvat@netxen.com Faisal Latif Principal Software Engineer faisal@netxen.com

17. NetXen • Next generation Ethernet silicon provider focused on server OEMs • Chips, Boards, S/W • Founded February 2002 • Top tier investors • Accel, Benchmark, Integral Capital • Expertise in semiconductor, software, systems and servers • Intelligent NIC™ product line • Launched March 27, 2006 • REAL products, REAL customers

18. Single-Chip Dual 10GbE Quad GbE Protocol Features TCP/IP RDMA iSCSI Virtualization Security Native 8X PCI-express 1X/4X/8X Intelligent NIC Architecture DDR QDR Protocol Processing Engine 10GE QM 10GbE Flow Classifier GbE CORE INTERCONNECT FABRIC CAM L2 Caches PCI-E 8X

19. NetXen 10GbE Chimney Rx 3.4 GHz Xeon Tx 10GbE Switch Windows Server 2003 SP1 with SNP Windows Server 2003 SP1 with SNP

20. 60% Throughput 800% Processor Efficiency 10GbE Chimney Results Throughput CPU Utilization Configuration: DP Xeon, 3.4GHz, HT off, 2GB

21. Demo Conclusion • 10GbE is happening NOW • Chimney enables • Scalability with balanced system design • Increased datacenter power efficiency • The Agile Datacenter requires • Adaptability, Scalability, Intelligence

22. Broadcom Uri ElzurDirector, Advanced TechnologyBroadcom Gururaj AnanthateertaSenior Staff EngineerBroadcom

23. Scalable TCP Chimney enables Convergence Over Ethernet • Scalable TCP Chimney - basis for Convergence over Ethernet • TCP based - Socket applications, iSCSI, iSCSI boot, iWARP (RDMA) • Microsoft’s SNP enable convergence over Ethernet • Secure (Network based security), robust and standard compliant implementation is required • Ethernet requires • Layer 2 functionality – VLAN, WoL, power management • Integrated Management Sockets Applications Windows Sockets Windows Socket Switch Storage Applications RDMA Provider User Mode KernelMode File System Partition TCP/IP Class Driver NDIS iSCSI Port Driver NDIS IM Driver (iscsiprt sys) . iSCSI RDMA Driver NDIS Miniport Miniport C-NIC HBA NIC RNIC

24. Broadcom’s C-NIC 2.5G/SNTTCP over 2.5 GB/s TCP Chimney NTTTCPs C-NIC Perfmon BCM5708S BCM5708S fiber cable S2 (TX/RX) S1 (TX/RX) HP DL 380G4 server 3.4GHz Intel Xeon CPU 1 GB RAM Windows Server 2003SP1-SNP build 2670 Two BCM5708S NICsBroadcom Miniport driver v 2.6.14* Broadcom 2.5G Switch BCM56580 StrataXGS III HP DL 380G4 server 3.4GHz Intel Xeon CPU 1GB RAM Windows Server 2003SP1-SNP build 2670 Two BCM5708S NICsBroadcom Miniport driver v 2.6.14*

25. TCP Chimney scales… Higher is better Lower is better TOE L2 TOE L2 BW improvement TOE vs. L2 CPU Utilization reduction TOE vs. L2 • 2.5G/S offers more BW than non-TOE, at 1/6 of the CPU utilization • Microsoft’s SNP combined with BCM5708 provides 7.5 times better P/E • Performance Efficiency (P/E) is network throughput divided by CPU Utilization • At Gigabit and beyond, TCP Chimney is critical to free up cycles for the applications Demo: NTTTCP

26. RSS Improves SMP Scalability Demo: Web Bench 5.0 With RSS web traffic is more evenly distributed on multiple CPUs Web Bench delivers up to 50% more requests/sec

27. Demo Conclusion • Broadcom’s C-NIC with Microsoft’s TCP Chimney is here TODAY • TCP Chimney scales to accommodate the needs of the server and applications • TCP Chimney is the basis for the future of Networking in Windows • Architecture allows for IPsec based security • RSS provides for a better load spreading on SMP servers

28. Scalable Networking Pack Partners

29. Future Chimney Offloads • IPsec Chimney • RDMA Chimney • SSL Chimney

30. Call To Action • Develop low cost TCP Chimney Offload and RSS hardware for Windows Vista and Windows Server codenamed “Longhorn” • Deploy TCP Chimney Offload and RSS hardware in enterprise and personal computing environments

31. Additional Resources • Web Resources • Documentation, White Papers, and software bits availabletoday for TCP Chimney Offload and RSS: http://support.microsoft.com/?kbid=912222 • Specs: DDK and Documentation will available on: www.microsoft.com/whdc • White Paper: http://www.microsoft.com/whdc/device/network/scale.mspx • Other Resources:www.microsoft.com/snphttp://www.microsoft.com/whdc/device/network/netintro.mspx • Related Sessions • Net088 – Technical Overview of Microsoft’s NetDMA Architecture • Please send e-mail to with questions ndis6fb @ microsoft.com

32. © 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.