Virtual Machine Queue Driver Development

2. Virtual Machine Queue Driver Development Sambhrama Mundkur Sr. Software Design Engineer Core Networking sammun@microsoft.com

3. Agenda • Overview of Virtual Machine Queue (VMQ) • VMQ capability advertisement • VM Queue allocation and deletion • Setting and clearing VMQ filters • Shared memory resources allocation • VMQ receive path • Embedded NIC switch • Diagnostics and Tools • Questions

4. VMQ Architecture Parent Partition VM1 VM2 Network Virtual Service Provider Ethernet TCP/IP TCP/IP Routing VLAN filtering Data Copy Port 2 • Port 1 VM NIC 1 VM NIC 2 Miniport Driver Q1 Q2 Default Queue VM BUS Switch/Routing unit NIC

5. Overview of Virtual Machine Queue • Performance bottlenecks in receive processing in virtualized environment • Each incoming packet has to be examined to identify the target virtual machine (VM) • Received data has to be copied from the parent partition’s memory to the child partition’s memory address space • Lack of concurrency of interrupts and DPCs

6. Overview of Virtual Machine Queue (contd.) • How does VMQ help? • Determine the target VM by MAC filtering in hardware • DMA received packets directly to the child partition’s memory address space • Provide interrupt and DPC concurrency by indicating received packets based on the target VM • Use switch capabilities of the NIC to provide VM to VM loopback functionality

7. VMQ Capability Advertisement • Receive filter capabilities advertised during miniport adapter initialization • Use standardized keywords to enable or disable VMQ functionality • Capabilities advertised • Enabled filters and queue types • Number of queues and filters • Supported queue properties • Lookahead-split support • NIC switch capabilities

8. Queue State Transitions

9. Queue Allocation • Protocol driver issues an OID to allocate a queue and sets the parameters for the queue • Per-queue receive indication flag • Processor affinity • Queue name • Lookahead-split parameters • NDIS generates a unique Queue ID and sets it before issuing the OID to the miniport driver • Protocol driver can set filters after allocating the queue • Protocol driver issues an OID to complete the queue allocation • Miniport driver can allocate shared memory and other resources • Default queue (Queue ID 0) is always allocated

10. Queue Deletion • Protocol driver clears all filters before freeing the queue • Protocol driver issues an OID to free the queue • Miniport driver • Must immediately stop DMA to shared memory resources allocated for this queue • Must generate status indication to indicate that the DMA has been stopped • Waits for all outstanding NET_BUFFER_LIST (NBLs) for this queue to be returned • Frees the allocated shared memory and hardware resources • Default queue can never be freed

11. Updating Queue Parameters • Protocol driver can issue an OID to update the queue parameters • Queue parameters that can be updated are • Processor affinity • Queue name

12. VMQ Filter Operations

13. Setting and Clearing Filters • Filters can be set and cleared any time after the queue has been allocated • Protocol issues an OID to set the filter and the filter’s parameters • Queue ID • Array of header field test parameters. • For VMQ two header field tests can be set: • Destination MAC address in the packet == specified MAC address • VLAN ID in the packet == specified VLAN ID • NDIS generates a unique filter ID and issues the OID to the miniport driver

14. Setting and Clearing Filters • Miniport driver programs the hardware based on the filters • All header field tests of a particular filter must match to indicate the packet to that queue • Multiple filters can be set on a queue • Packets must be indicated from the queue if any of the filters pass • Protocol driver can set filters on the default queue, too • Protocol driver issues an OID to clear a filter • Provides the Filter ID of the filter

15. Shared Memory Resource Allocation • Miniport driver calls the new DDI, NdisAllocateSharedMemory, to allocate shared memory resources for a queue • Initial allocation occurs when processing the queue allocation complete OID • Miniport driver can allocate shared memory as long as the queue has not been freed • Miniport driver specifies: • Queue ID • Length • Usage – Lookahead, post-Lookahead etc. • NdisAllocateSharedMemory returns: • Virtual address of the allocated memory • Scatter/gather List • Shared memory handle (used in receive indications) • Allocation handle (used to free the memory)

16. Shared Memory Resources • If lookahead-split is not enabled, specify that the memory will be used for receive buffer • If lookahead-split has been enabled for the queue, allocate two shared memory buffers • Specify that usage is for the lookahead buffer • Specify that usage is for the post-lookahead (data) buffer • Call NdisFreeSharedMemory to free the memory • Driver frees the resources when the queue is freed • Driver must wait for all NBLs using the shared memory to be returned before calling this DDI

17. VMQ Packet Structures

18. VMQ Receive Path • Indicate packets on a queue only if it passes all the filter field tests for a particular filter • If per-queue receive indication flag is set: • NBLs from this queue must not be mixed with others • Must set ‘single queue indication’ flag in the Receive Flags • Must set flag to indicate that the shared memory information is valid • Out-of-band (OOB)information • Specify the Queue ID in packet OOB information • For VMQ, set the Filter ID in the OOB to zero

19. VMQ Receive Path • Specify the SharedMemoryInfo in the NET_BUFFER • Set SharedMemoryHandle returned by NdisAllocateSharedMemory • Offset is the offset from the beginning of the shared memory buffer • If lookahead split is enabled for the queue, each NET_BUFFER: • Must have two MDLs and corresponding SharedMemoryInfo structures • The post-lookahead buffer must have a backfill • The protocol driver copies the contents of the lookahead buffer to the backfill • Backfill must exist even if the packet is entirely in the lookahead buffer • MDL Length and DataLength in the packet must be set as before

20. Queue Statistics • NIC must maintain following per-queue statistics: • Number of transmitted and received packets • Number of transmitted and received bytes • NIC can maintain per-packet type statistics • The statistics can be queried using the new statistics OID • This is a mandatory Direct OID request • VMQ miniport driver must provide a Direct OID request handler • Return statistics for a particular queue based on Queue ID • Default queue can be queried, too

21. Embedded NIC Switch • Embedded switch on the NIC can assist in advanced features • Advertise support during miniport adapter initialization • Loopback support • Protocol driver issues an OID to enable these features • Loopback • On transmit, Queue ID of the outgoing queue is set in packet OOB data • NIC indicates the packet if it matches filters set on any other queue • If no match is found, NIC transmits the packet on the wire

22. Diagnostics • OIDs and WMI GUIDs • Enumerate the queues allocated on a miniport adapter • Query the parameters of a particular queue • Enumerate the filters allocated on a particular queue • Query the parameters of a particular filter • Handled by NDIS • NDIS kernel debugger extensions • Performance counters exposed by NDIS

23. Call to Action • Update drivers to NDIS 6.20 to take advantage of Windows 7 functionality • Implement VMQ support in the miniport driver • Attend other related sessions in the DDC

24. Resources • Windows 7 WDK will be available as of WinHEC • Windows Logo Program Web site: http://www.microsoft.com/whdc/winlogo/default.mspx • NDIS 6 feedback alias:ndis6fb@microsoft.com

25. Related Sessions

26. Questions?