Optimizing TCP/UDP Protocols for Efficient Data Transmission

1. Evolution of TCP / UDP

2. User Datagram Protocol (UDP) • Strengths • Lightweight Protocol • No overhead • Weaknesses • No reliability in UDP Client Server rqst resp

3. Transmission Control Protocol (TCP) • Strengths • Reliable Data Transmission • Stream Protocol • Weaknesses • Connection Overhead • Transaction time: • 2 RTT + • Server Processing time (SPT) + • 1RTT for each additional data packet • Connection resources held until any possible lost packets have expired ( 2 * RTTmax = 240 sec) Client Server syn syn, ack ack data fin ack data fin ack

4. Minimum TCP Transaction • Protocol allows sending data with syn • Not generally done • Must hold data until 3WHS complete • Minimum transmission • 5 packets • 2 RTT + SPT Client Server syn, fin, data syn, ack ack, fin ack, fin data ack

5. The Problem: • Some applications (particularly Web apps) need both reliability and speed • Many connections (especially page loads) need only a few data packets • Early studies showed that median HTTP reply between 1770 and 958 bytes • May have many connections per session. • Overhead of connection (syn, syn-ack, ack) increases the transaction time • Google study showed 4 to 40% improvement in page load time possible • Because ports cannot be reused for 240 sec, maximum transaction rate between a client and server is ~64000 ports/ 240 sec = 267 trans/sec • Problem is most pronounced when RTT is large

6. One Solution:TCP for Transactions (T/TCP) • Originally proposed in rfc 1644 (1994) • Include data in original syn packet. • Maintain a Connection Counter (cc) in server that is passed by client to server as TCP option. Server uses this value to validate client, bypassing the need for 3-way handshake (3WHS) • Because server can now validate each connection, connection delay reduced from 240 to 12 sec. Client Server syn, cc, data, fin syn, ack, cc, data, fin ack

7. T/TCP Initial Connection Setup • Protocol requires setting up the connection count between each client and the server. Initial 3WHS includes CC request. • Each succeeding connection uses CC and avoids 3WHS • Protocol requires that CC increment monotonically.

8. T/TCP Protocol • Initial 3WHS • Succeeding data calls Client Server Client Server syn, CCnew, data syn, cc, data, fin syn, ack, cc, data, fin syn, ack, CCecho ack fin, ack ack Data, fin ack ack

9. T/TCP Problems ! • For most implementations, server initializes CC at 1 and increments by 1 for each new connection. By sniffing the connection, it is easy to anticipate and spoof the CC. Even without access to the path, picking a large number will usually succeed (CC is 32 bit number) • This makes it very easy for an attacker to implement a syn attack (DoS). It is made more effective since server must store the data associated with the syn, using up more server resources with each new packet. • Protocol moved to historical status in 2011.

10. TCP Fast Open • Researched originally at Google in 2011. • Published at CoNEXT 2011 • Objective was to improve web server performance • Offered as IETF draft first in 2012. • Currently in draft 08 (expires August, 2014) • Implemented in Linux (since Kernel 3.6/3.7)

11. TCP Fast Open • Use initial connection to establish relationship. • Client requests TCP fast open cookie and sends data in original syn packet • Server creates cookie and returns to client in syn, ack packet. Cookie is an encrypted hash of client IP address and other data. • Client ack packet completes 3WHS. • Server processes client request and returns response. • Subsequent requests do not need 3WHS • Client request includes server cookie, data • Server returns syn, ack • Server returns response • Client completes handshake.

12. Fast Open Initial connect

13. Fast Open Subsequent Packets

14. Linux Implementation • Requires current kernel (at least 3.7 for client and server) • Requires setting /proc/sys/net/ipv4/tcp_fastopen • Enable TCP Fast Open feature (draft-ietf-tcpm-fastopen) to send data in the opening SYN packet. To use this feature, the client application must use sendmsg() or sendto() with MSG_FASTOPEN flag rather than connect() to perform a TCP handshake automatically. The values (bitmap) are 1: Enables sending data in the opening SYN on the client w/ MSG_FASTOPEN. 2: Enables TCP Fast Open on the server side, i.e., allowing data in a SYN packet to be accepted and passed to the application before 3-way hand shake finishes. 4: Send data in the opening SYN regardless of cookie availability and without a cookie option. 0x100: Accept SYN data w/o validating the cookie. 0x200: Accept data-in-SYN w/o any cookie option present. 0x400/0x800: Enable Fast Open on all listeners regardless of the TCP_FASTOPEN socket option. The two different flags designate two different ways of setting max_qlen without the TCP_FASTOPEN socket option. Default: 1 Note that the client & server side Fast Open flags (1 and 2 respectively) must be also enabled before the rest of flags can take effect.

15. Linux TCP Fast Open client (TFO_client.cpp) void UDPecho(const char *host, const char *service) { char buf[BUFFSIZE][LINELEN+1] = {"This is the first message", "This is thesecond message", "This is the third message", "This is the fourth message", "This is the last message" }; char inbuf[LINELEN +1]; ints, nchars, rchars; /* socket descriptor, character counters */ : for (index = 0; index < BUFFSIZE; index++) { s = socket(AF_INET, SOCK_STREAM, 0); : nchars = strlen(buf[index]);

16. Linux TCP Fast Open client (TFO_client.cpp) status = sendto(s, &buf[index], nchars, MSG_FASTOPEN, (conststructsockaddr *) &sin, sizeof(sin)); : cout << "We just sent a datagram" << endl; rchars = recvfrom(s, inbuf, LINELEN, 0, NULL, NULL); : inbuf[rchars] = '\0'; //Add a null termination : cout << "We got back the following " << rchars << " characters: " << inbuf << endl; close (s); sleep(1); } /* end of while */ }/* end of UDPecho () */

17. Linux TCP Fast Open server (TFO_server.cpp) int main(intargc, char *argv[]) { intqlen = 5; : sock = passiveTCP(service, qlen); ans = setsockopt(sock, IPPROTO_TCP, TCP_FASTOPEN, &qlen, sizeof(qlen)); while (keep_looping == YES) { ssock = accept (sock, (sockaddr *)&fsin, (socklen_t *)&fsinsize); sprintf (log_msg, "Just got a request from %s\n", inet_ntoa(fsin.sin_addr)); logData(log_msg); cc = recv (ssock, buf, LINELEN, 0); buf[cc] = '\0'; sprintf (log_msg, "Msg: %s\n", buf); logData(log_msg); strcat (buf, " answer"); send (ssock, buf, cc +7, 0); memset (buf, 0, LINELEN); close (ssock); }//end of while loop }

18. Wireshark Trace – Initial connect (1)cookie request

19. Wireshark Trace – Initial connect (2)cookie

20. Wireshark Trace – Initial connect (3)data

21. Wireshark Trace – Initial connect (4)ack

22. Wireshark Trace – Initial connect (4)response

23. Wireshark Trace – Next connect (1)syn, cookie, data

24. Wireshark Trace – Next connect (2)syn,ack

25. Wireshark Trace – Next connect (3)response

26. Wireshark Trace – Next connect (4)fin, ack

27. Fast Open Client side Output [bob@localhost client]$ g++ -o tfocTFO_client.cpp [bob@localhost client]$ ./tfoc 192.168.1.25 23456 Our target server is at address 192.168.1.25 Enter data to send... We just sent a datagram We got back the following 32 characters: This is the first message answer We just sent a datagram We got back the following 33 characters: This is the second message answer We just sent a datagram We got back the following 32 characters: This is the third message answer We just sent a datagram We got back the following 33 characters: This is the fourth message answer We just sent a datagram We got back the following 31 characters: This is the last message answer [bob@localhost client]$

28. Fast Open Server Log ### Starting Server on port 23456 at Wed Apr 2 11:26:17 2014 Just got a request from 192.168.1.117 Msg: This is the first message Just got a request from 192.168.1.117 Msg: This is the first message Just got a request from 192.168.1.117 Msg: This is the second message Just got a request from 192.168.1.117 Msg: This is the third message Just got a request from 192.168.1.117 Msg: This is the fourth message Just got a request from 192.168.1.117 Msg: This is the last message

29. Extended Fast Open Test • Build Standard TCP Client • Send 100 packets, receive 100 responses • Build TCP Fast Open Client • Send 100 packets, receive 100 responses • Test Conditions • Client and server on adjacent machines on same LAN • Same code structure, same data sent and received. • Compare Results • How many Packets required for each • Standard: 1000 packets • TFO: 801 • How much time required for each? • Standard: .173333 sec • TFO: .119198 sec

30. Summary • Network Protocols continue to evolve • TCP Fast Open offers the speed of UDP (after the first connection) with the reliability of TCP. • Protocol is still in experimental stage.

31. References • TCP/IP Illustrated Vol 3, Richard Stevens • Chapters 1-12 (T/TCP) • Original TCP Fast Open Paper • http://conferences.sigcomm.org/co-next/2011/papers/1569470463.pdf • http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/37517.pdf • IETF draft RFC • https://datatracker.ietf.org/doc/draft-ietf-tcpm-fastopen/ • lwn.net (Linux Weekly News?) Article • https://lwn.net/Articles/508865/ • Packet Pushers Article • http://packetpushers.net/tcp-fast-curious-a-look-at-tcp-fast-open/ • Admin Magazine Article • http://www.admin-magazine.com/Articles/TCP-Fast-Open • Linux Kernel documentation • https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt