Net- REpLAY : A NEW NETWORK PRIMITIVE

1. Net-REpLAY: A NEW NETWORK PRIMITIVE Ashok Anand AdityaAkella University of Wisconsin, Madison

2. Network is a black box • This keeps network simple and efficient • However, end hosts have to resort to complicated logic to infer the nature of glitches Network Black box view End hosts Packet lost or delayed No standardized way for network to inform, where and why glitch occurred

3. Current tools for locating glitches • Probing tools • Tulip[SOSP 2003] sends multiple probes to routers on the path, and use their response to infer the nature of glitches • Issues with such approach • Out-of-band troubleshooting • Probe packets can be treated differently • Transient failures hard to detect

4. What if network could tell end-hosts about glitches? • End hosts can take better actions • No need to reduce flow rate, if packet loss was not due to congestion • Route around glitches using alternate routes via multi-homing or recent routing protocol enhancements like path-splicing etc • Benefits emerging applications e.g. gaming, video streaming etc to achieve more robust network performance Where and why glitch occurred Network End hosts

5. Network-assisted troubleshooting • Design requirement • Keep the network as simple as possible • While enabling end hosts to determine where and why glitches occurred • Our design: Net-Replay • Router remember the packets they have forwarded • Routers annotate packet (e.g. with their identifier) that they see for the first time • When some glitch occurs, sender replays those packets who had experienced the glitch • Based on annotations, receiver determines the nature of glitch experienced by the original packet

6. Using Net-Replay to characterize loss • Receiver infers that packet was dropped at A-B link Packet already present at router A Replayed packet was seen for the first time at B Remember the green packet and annotate before forwarding Re-play the lost packet A B C Sender Receiver A A A B B A

7. Outline • Supporting Net-Replay functionality in network • End hosts using Net-Replay • Discussion

8. Outline • Supporting Net-Replay functionality in network • End hosts using Net-Replay • Discussion

9. Basic support at router Compute hash • Compute hash • Exclude mutable fields (e.g. TTL) • Finding if new packet was already seen by the router • Look-up hash in Hashstore • Remember new packet as seen • Store hash pointing to packet in Hashstore • Evict the oldest packet, if Hashstore becomes full • Simple hash table implementation in DRAM for speeds like 2.4 Gbps • SRAM for higher speed (40 Gbps) • 16 MB SRAM currently available Packet Hash Hash Packet Hashstore

10. High speed Hashstore implementation • Use bloom-filters • What about false positives? • Can probabilistically report the location of glitches • What about packet eviction? • Use 2 bloom filters: primary and secondary • When primary is half filled, start using both • When primary is fully filled, copy secondary to primary and clear out secondary • How much time worth packets are stored in 16 MB SRAM? • Up to 3s at 40 Gbps with average packet size of 600 bytes • Greater than10 RTTs assuming RTT < 250 ms • Sufficient enough for end-applications to react

11. Outline • Supporting Net-Replay functionality in network • End hosts using Net-Replay • Deployment discussion, cheating Issues and new applications enabled by Net-Replay

12. How end hosts can use Net-Replay? • Characterizing glitches • Packet loss • Replay lost packet • Delay • Router remembers which packets were delayed • Replay delayed packet • Reordering • If it happened due to route changes, sender could know the first router where route changed • Replay reordered packet

13. End host protocol stack • Higher layer should decide the policy of handling glitches • TCP layer can tell higher layer the nature of glitches e.g. loss • After loss, TCP layer retransmits packet (in current TCP protocols) • Uses retransmitted packet to find the nature of loss • Receiver sends the information about loss back to sender along with ACK Application or higher layer Decide how to overcome glitches MIRO/ Path splicing Or no action Nature of glitches TCP layer

14. Outline • Supporting Net-Replay functionality in network • End hosts using Net-Replay • Discussion

15. Deployment discussion • Partial deployment • Net-replay can be deployed on few routers and can be used to find the nature of glitches in path segments • Border routers of ISP and information per domain • Avoiding device modifications • Can be deployed in 2-port hardware switches as bumps in the wire Net-Replay agnostic devices Net-Replay aware bumps in the wire

16. Other applications • Network tomography uses complicated logic to infer link loss rates • With Net-Replay, location of loss can be precisely determined • Simplifies network tomography • Packets can be moved from fast memory to disks in batches • Can be used for debugging distributed applications • Useful for network operators to find the performance at fine grained level

17. Conclusion • Net-Replay helps applications perform in-band characterization of glitches • Net-Replay requires simple support from network infrastructure • End hosts can get robust network performance using Net-Replay

18. Questions • Thank you

19. Backup

20. Hashstore implementation • Simple hash table in DRAM (50 ns latency) good enough at 2.5 Gbps • Lookup and store: 100 ns per packet • 40B packets arrive every 128ns at 2.5 Gbps • However DRAM latency can’t match 40Gbps; requires faster memory like SRAM • Current SRAM up to 16MB only; Need space-efficient data structure

21. Cheating issues • ISP inserts wrong annotations to ensure that it is never considered accountable for glitches • Chances are that ISP is caught • ISP modifies ACK packet, if it finds its router is causing glitches • Use encrypted ACK • Possibly other issues and need to investigate