Extensible Routers Using Network Processors

1. Extensible Routers Using Network Processors Larry Peterson Scott Karlin Tammo Spalink Yitzchak Gottlieb

2. Running Code On Routers • Edge Routers • NAT, Firewalls, DiffServ, etc. • Home Routers • Media gateway • Scaleable servers • Level n switch Princeton University

3. Technology Forces • Commodity switching fabrics • e.g., ATM switches, Infiniband SAN • Programmable network processors • e.g., Intel IXP1200, Sitera PRISM • Commodity processors • e.g., Intel IA32, IA64 Princeton University

4. Research Problems • Scheduling • SIGMETRICS 2001 • Programming • JSAC, March 2001 • Implementation • SOSP 2001 • Architecture • OpenARCH 2001 Princeton University

5. Packet Flows Forwarding Paths Switching Paths VERA Architecture . . . Network Services . . . Virtual Router . . . Hardware Configurations . . . Princeton University

6. C F S Virtual Router Virtual Router • Classifiers • Schedulers • Forwarders Princeton University

7. IP-- F IPSEC F IP S F C Proxy F Active Protocol F Simple Example Princeton University

8. Control and Data Plane Layered Video Analysis (control plane) Shared State Smart Dropper (data plane) Princeton University

9. Processor Hierarchy Pentium StrongArm MicroEngines Princeton University

10. Lab Setup • IXP 1200 • On Pentium III Motherboard • Linux with driver • PCI Bus • Serial I/O • 4 Pentium IIIs • Packet Sources/Sinks • 2 Kingston DEC 21143 ethernet cards Princeton University

11. Development Environment • Linux • GCC/Binutil • SA programs in C • Commands and Libraries • sgo, pcirestore • Embedded C, LibCII, Runtime • Windows • EEPROM Burner • Assembler VxWorks Princeton University

12. Scratch 6 Micro- Engines DRAM SRAM Intel IXP MAC Ports FIFOs StrongARM IX Bus IXP1200 Chip PCI Bus Princeton University

13. MicroEngines DRAM (buffers) 16 Output FIFO Slots 16 Input FIFO Slots SRAM (queues, state) 16 Input Contexts 8 Output Contexts Princeton University

14. Evaluation • Maximum forwarding rate • 3.47 Mpps (64-byte packets) • Independent of ports and IXBus InstructionCounts Princeton University

15. 16 contexts input loop: until_rcv move FIFO-to-DRAM lookup route enqueue 8 contexts output loop: dequeue move DRAM-to-FIFO init_xmit Main Loops Princeton University

16. 16 contexts input loop: until_rcv move FIFO-to-DRAM lookup route nop nop ... nop nop enqueue 8 contexts output loop: dequeue move DRAM-to-FIFO init_xmit Main Loops Princeton University

17. Virtual Router Processor • Fixed Infrastructure (green code) • Can forward 1.13Mpps for 8 x 100Mbps ports • Programmable VRP (red code) • Per 64-byte chunk • All in registers • 24 x 32-bit SRAM transfers (flow state) • 240 register operations • 3 hashes with hardware support • Total of 650 instructions per MicroEngine Princeton University

18. StrongArm • Handle exceptional packets • Problems • Shares DRAM capacity with MicroEngines • On the critical path to Pentium • Solution • Limit the role the SA can play • Programmable bridge to Pentium • µEngine manager Princeton University

19. Pentium • Runs protocols in the control plane • e.g., BGP, OSPF, RSVP • Run other router extensions • e.g., proxies, active protocols, overlays • Scheduling is important • Packets can arrive too fast Princeton University

20. Performance Linux Pentium 310 Kpps with 1510 cycles/packet VERA StrongArm 3.47 Mpps w/ no VRP or 1.13 Mpps w/ VRP budget VRP MicroEngines Princeton University

21. Summary • Extensible Routers are feasible • 240 Operations per 64-byte packet fragment • 24 SRAM Operations • The processor hierarchy works • Use Line Cards (µEngines) for small tasks • IP Forwarding • Extensions • Use Main Processor (Pentium) for large tasks Princeton University

22. http://www.cs.princeton.edu/nsg Implementation paper available July 2001