NetFPGA Summer Camp Day 1

1. Presented by: Adam Covington, Glen Gibb (Stanford University) Stanford, CA August 9 - 13, 2010 http://NetFPGA.org NetFPGA Summer CampDay 1

2. Tutorial Outline Background Introduction The NetFPGA Platform The Stanford Base Reference Router Motivation: Basic IP review Demo1: Reference Router running on the NetFPGA The Enhanced Reference Router Motivation: Understanding buffer size requirements in a router Demo 2: Observing and controlling the queue size How does the NetFPGA work Utilities Reference Designs Inside the NetFPGA Hardware The Life of a Packet Through the NetFPGA Hardware Datapath Interface to software: Exceptions and Host I/O Exercise: Drop Nth Packet Concluding Remarks Using NetFPGA for research and teaching

3. Section I: Motivation

4. What is the NetFPGA? A line-rate, flexible, open networking platform for teaching and research

5. NetFPGA consists of… NetFPGA Board Four elements: • NetFPGA board • Tools + reference designs • Contributed projects • Community

6. NetFPGA board PC with NetFPGA 1GE FPGA 1GE 1GE Memory 1GE NetFPGA Board NetworkingSoftware running on a standard PC CPU Memory PCI A hardware accelerator built with Field Programmable Gate Arraydriving Gigabit network links

7. Tools + reference designs Tools: • Compile designs • Verify designs • Interact with hardware Reference designs: • Router (HW) • Switch (HW) • Network Interface Card (HW) • Router Kit (SW) • SCONE (SW)

8. Contributed Projects More projects: http://netfpga.org/foswiki/NetFPGA/OneGig/ProjectTable

9. Community Wiki • Documentation (slowly growing) • Encourage users to contribute Forums • Support by users for users • Active community – 10s to 100s of posts per week

10. NetFPGA’s Defining Characteristics • Line-Rate • Processes back-to-back packets • Without dropping packets • At full rate of Gigabit Ethernet Links • Operating on packet headers • For switching, routing, and firewall rules • And packet payloads • For content processing and intrusion prevention • Open-sourceHardware • Similar to open-source software • Full source code available • BSD-Style License • But harder, because • Hardware modules must meeting timing • Verilog & VHDL Components have more complex interfaces • Hardware designers need high confidence in specification of modules

11. Test-Driven Design • Regression tests • Have repeatable results • Define the supported features • Provide clear expectation on functionality • Example: Internet Router • Drops packets with bad IP checksum • Performs Longest Prefix Matching on destination address • Forwards IPv4 packets of length 64-1500 bytes • Generates ICMP message for packets with TTL <= 1 • Defines how packets with IP options or non IPv4 … and dozens more … Every feature is defined by a regression test

12. Who, How, Why Who uses the NetFPGA? • Teachers • Students • Researchers How do they use the NetFPGA? • To run the Router Kit • To build modular reference designs • IPv4 router • 4-port NIC • Ethernet switch, … Why do they use the NetFPGA? • To measure performance of Internet systems • To prototype new networking systems

13. What you will learn • Overall picture of NetFPGA • How reference designs work • How you can work on a project • NetFPGA Design Flow • Directory Structure, library modules and projects • How to utilize contributed projects • Interface/Registers • How to verify a design (Simulation and Regression Tests) • Things to do when you get stuck AND… You can start your own projects!

14. Section II: Demo Basic Use

15. Basic Uses of NetFPGA • Recap Internet Protocol and Routing • Demonstrate • How you can use the NetFPGA as a router • See routing in action

16. What is IP? • IP (Internet Protocol) • Protocol used for communicating data across packet-switched networks • Divides data into a number of packets (IP packet) • IP Packet • Header (IP Header) including: • Source IP address • Destination IP address

17. IP Header Data Data Hdr Data Hdr Data Hdr 1 4 16 32 Ver HLen T.Service Total Packet Length Fragment ID Flags Fragment Offset TTL Protocol Header Checksum Source Address Destination Address Options (if any) 20 bytes Data

18. IP Address • Used to uniquely identify a device (such as a computer) from all other devices on a network • Two parts • Identifier of a particular network on the Internet • Identifier of a particular device within a network All packets, except the ones for the same network, first go to their gateway (router) and are transferred to the destination via routers.

19. Basic Operation of an IP Router D R3 R1 R4 D A B E R2 C R5 Destination Next Hop F D R3 E R3 F R5

20. What does a router do? R3 R1 R4 D A 1 4 16 32 D Ver HLen T.Service Total Packet Length Fragment ID Flags Fragment Offset B E TTL Protocol Header Checksum 20 bytes Source Address R2 C R5 Destination Address Destination Next Hop F D R3 Options (if any) E R3 Data F R5

21. What does a router do? R3 R1 R4 D A B E R2 C R5 F

22. Basic Components of an IP Router Software Hardware Management & CLI Routing Protocols Control Plane Routing Table Datapath per-packet processing Forwarding Table Switching

23. Per-packet processing in an IP Router 1. Accept packet arriving on an incoming link. 2. Lookup packet destination address in the forwarding table to identify outgoing port(s). 3. Manipulate IP header: e.g., decrement TTL, update header checksum. 5. Buffer packet in the output queue. 6. Transmit packet onto outgoing link.

24. Generic Datapath Architecture Data Hdr Data Hdr IP Address Next Hop Forwarding Table Buffer Memory Header Processing Lookup IP Address Update Header Queue Packet

25. CIDR and Longest Prefix Matches 142.12/19 • The IP address space is broken into line segments. • Each line segment is described by a prefix. • A prefix is of the form x/y where x indicates the prefix of all addresses in the line segment, and y indicates the length of the segment. • e.g. The prefix 128.9/16 represents the line segment containing addresses in the range: 128.9.0.0 … 128.9.255.255. 128.9.0.0 65/8 128.9/16 0 232-1 216 128.9.16.14

26. Classless Interdomain Routing (CIDR) 128.9.19/24 128.9.25/24 128.9.16/20 128.9.176/20 Most specific route = “longest matching prefix” 128.9/16 0 232-1 128.9.16.14

27. Techniques for LPM in hardware • Linear search • Slow • Direct lookup • Currently requires too much memory • Updating a prefix leads to many changes • Tries • Deterministic lookup time • Easily pipelined but require multiple memories/references • TCAM (Ternary CAM) • Simple and widely used but havelower density than RAM and need more power • Gradually being replaced by algorithmic methods

28. An IP Router on NetFPGA Software Hardware Management & CLI Linux user-level processes Routing Protocols Exception Processing Routing Table Verilog on NetFPGA PCI board Forwarding Table Switching

29. NetFPGA Router Function • 4 Gigabit Ethernet ports Fully programmable • FPGA hardware Low cost Open-source FPGA hardware • Verilog base design Open-souce Software • Drivers in C and C++

30. Demo 1 Reference Router running on the NetFPGA

31. Hardware Setup for Demo #1 Demo 1 CPU x2 NIC Video Server PCI-e PCI-e GE GE Net-FPGA GE PCI Internet Router Hardware GE GE Server deliversstreaming HD video through a chain of NetFPGA Routers GE Net-FPGA GE Internet Router Hardware GE GE GE … CPU x2 NIC PCI-e GE GE Net-FPGA GE Video Display PCI Internet Router Hardware GE GE CAD Tools GE

32. Topology Demo 1 .1.1 .4.1 .7.1 .10.1 .13.1 .16.1 .1.2 .4.2 .7.2 .10.2 .13.2 .16.2 .3.1 .6.2 .9.2 .12.2 .15.2 .17.1 .3.2 .2.1 .6.1 .9.1 .12.1 .15.1 .30.2 .5.1 .8.1 .11.1 .14.1 .18.1 .30.1 .26.1 .23.1 .18.2 .27.2 .24.2 .21.2 .20.1 .29.1 .24.1 .21.1 .27.1 .28.2 .25.2 .22.2 .19.2 .28.1 .25.1 .22.1 .19.1 Video Server Video Client Shortest Path

33. Working IP Router Objectives Become familiar with Stanford Reference Router Observe PW-OSPF re-routing traffic around a failure Demo 1

34. Step 1 – Observe the Routing Tables The router is already configured and running on your machines The routing table has converged to the routing decisions with minimum number of hops Next, break a link … Demo 1

35. Step 2 - Dynamic Re-routing Break the link between video server and video client Routers re-route traffic around the broken link and video continues playing Demo 1 .1.1 .4.1 .7.1 .10.1 .13.1 .16.1 .1.2 .4.2 .7.2 .10.2 .13.2 .16.2 .3.1 .6.2 .9.2 .12.2 .15.2 .17.1 .2.1 .3.2 .15.1 .6.1 .9.1 .12.1 .30.2 .5.1 .8.1 .11.1 .14.1 .18.1 .30.1 .26.1 .23.1 .18.2 .21.2 .27.2 .24.2 .20.1 .29.1 .24.1 .21.1 .27.1 .28.2 .25.2 .22.2 .19.2 .28.1 .25.1 .22.1 .19.1

36. Section III: Demo Advanced Use

37. Advanced Uses of NetFPGA • Introduction on TCP and Buffer Sizes • Demonstrate • NetFPGA used for real time measurement • See TCP Saw tooth in real time

38. Buffer Requirements in a Router Buffer size matters: Small queues reduce delay Large buffers are expensive Theoretical tools predict requirements Queuing theory Large deviation theory Mean field theory Yet, there is no direct answer Flows have a closed-loop nature Question arises on whether focus should be on equilibrium state or transient state

39. Universally applied rule-of-thumb: A router needs a buffer size: 2T is the two-way propagation delay (or just 250ms) C is capacity of bottleneck link Context Mandated in backbone and edge routers Appears in RFPs and IETF architectural guidelines Already known by inventors of TCP [Van Jacobson, 1988] Has major consequences for router design Rule-of-thumb Source Destination Router C 2T

40. The Story So Far # packets at 10Gb/s 1,000,000 20 10,000 • Assume: Large number of desynchronized flows; 100% utilization • Assume: Large number of desynchronized flows; <100% utilization

41. Exploring Buffer Sizes Need to reduce buffer size and measure occupancy Not possible in commercial routers So, we will use the NetFPGA instead Objective: Use the NetFPGA to understand how large a buffer we need for a single TCP flow.

42. Why 2TxC for a single TCP Flow? • Rule for adjusting W • If an ACK is received: W ← W+1/W • If a packet is lost: W ← W/2 Only Wpackets may be outstanding http://guido.appenzeller.net/anims/

43. Time Evolution of a Single TCP Flow Time evolution of a single TCP flow through a router. Buffer is 2T*C Time evolution of a single TCP flow through a router. Buffer is < 2T*C

44. Demo 2Buffer Sizing Experimentsusing the NetFPGA Router

45. Hardware Setup for Demo #2 Demo 2 … CPU x2 NIC PCI-e GE GE Net-FPGA GE Video Client PCI Server deliversstreaming HD video to adjacent client Internet Router Hardware GE GE GE CPU x2 NIC Video Server PCI-e PCI-e GE GE

46. Topology Demo 2 • eth1 connects your host to your NetFPGA Router • nf2c2 routes to nf2c1 (your adjacent server) • eth2 serves web and video traffic to your neighbor • nf2c0 & nf2c3 (the network ring) are unused .2.1 .5.1 .8.1 .11.1 .4.1 .7.1 .10.1 .13.1 .1.2 .10.2 .1.1 .4.2 .7.2 .13.2 .2.2 .5.2 .8.2 .11.2 .14.2 .29.1 .29.2 .14.1 .26.2 .20.2 .23.2 .17.2 .19.2 .16.2 .28.2 .25.2 .22.2 .16.1 .28.1 .25.1 .22.1 .19.1 .26.1 .23.1 .20.1 .17.1 This configuration allows you to modify and test your router without affecting others

47. Enhanced Router Demo 2 Objectives • Observe router with new modules • New modules: rate limiting, event capture Execution • Run event capture router • Look at routing tables • Explore details pane • Start tcp transfer, look at queue occupancy • Change rate, look at queue occupancy

48. Step 1 - Run Pre-made Enhanced Router Start terminal and cd to “netfpga/projects/ tutorial_router/sw/” Type “./tut_adv_router_gui.pl” A familiar GUI should start Demo 2

49. Step 2 - Explore Enhanced Router Click on the Details tab A similar pipeline to the one seen previously shown with some additions Demo 2

50. Enhanced Router Pipeline Two modules added Event Capture to capture output queue events (writes, reads, drops) Rate Limiter to create a bottleneck Demo 2 MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ Input Arbiter Output Port Lookup Event Capture Output Queues MAC TxQ CPU TxQ Rate Limiter CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ