Whence a description of how to enable TCP FASTOPEN in a net-next kernel?

Is there a writeup describing the steps needed to enable TCP_FASTOPEN in a net-next kernel? (pulled earlier today)

I am looking to debug netperf's support for enabling the feature and I want to make sure I've enabled things correctly in the kernel. Thusfar I've set the tcp_fastopen sysctl to one, and I see the "client" side of netperf making the appropriate sendto() call, and I see what appears to be the correct setsockopt being set on the server side, but my tcpdump traces of the traffic flowing over loopback in my test setup, while showing the client including the experimental option, do not show the server side responding:

13:10:23.870202 IP localhost.5923 > localhost.54363: Flags [S], seq 935361110, win 43690, options [mss 65495,sackOK,TS val 889762 ecr 0,nop,wscale 7,Unknown Option 254f989], length 0 13:10:23.870214 IP localhost.54363 > localhost.5923: Flags [S.], seq 4210640362, ack 935361111, win 43690, options [mss 65495,sackOK,TS val 889762 ecr 889762,nop,wscale 7], length 0

The netserver side strace snippet:

3861  socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 8
3861  getsockopt(8, SOL_SOCKET, SO_SNDBUF, [16384], [4]) = 0
3861  getsockopt(8, SOL_SOCKET, SO_RCVBUF, [87380], [4]) = 0
3861  setsockopt(8, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
3861  bind(8, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
3861  setsockopt(8, SOL_TCP, 0x17 /* TCP_??? */, [5], 4) = 0
3861  listen(8, 5)                      = 0
...
3861  accept(8, {sa_family=AF_INET, sin_port=htons(5923), sin_addr=inet_addr("127.0.0.1")}, [16]) = 9
3861  recvfrom(9, "n", 1, 0, NULL, NULL) = 1
3861  sendto(9, "n", 1, 0, NULL, 0)     = 1
3861  getsockopt(9, SOL_SOCKET, SO_RCVBUF, [262030], [4]) = 0
3861  getsockopt(9, SOL_SOCKET, SO_SNDBUF, [663750], [4]) = 0
3861  close(9)                          = 0

lather, rinse, repeat the accept sequence off that listen endpoint.

happy benchmarking,

rick jones

I guess you need to enable fastopen both for client and server :

echo 3 >/proc/sys//net/ipv4/tcp_fastopen

vi +475 Documentation/networking/ip-sysctl.txt

tcp_fastopen - INTEGER
Enable TCP Fast Open feature (draft-ietf-tcpm-fastopen) to send data in the opening SYN packet. To use this feature, the client application must use sendmsg() or sendto() with MSG_FASTOPEN flag rather than connect() to perform a TCP handshake automatically.

The values (bitmap) are
1: Enables sending data in the opening SYN on the client. 2: Enables TCP Fast Open on the server side, i.e., allowing data in
a SYN packet to be accepted and passed to the application before 3-way hand shake finishes.
4: Send data in the opening SYN regardless of cookie availability and
without a cookie option.
0x100: Accept SYN data w/o validating the cookie.
0x200: Accept data-in-SYN w/o any cookie option present. 0x400/0x800: Enable Fast Open on all listeners regardless of the
TCP_FASTOPEN socket option. The two different flags designate two different ways of setting max_qlen without the TCP_FASTOPEN socket option.

Default: 0

Note that the client & server side Fast Open flags (1 and 2 respectively) must be also enabled before the rest of flags can take effect.

See include/net/tcp.h and the code for more details.

Looks like I'm good now:

raj@tardy-ubuntu-1204:~$ cat /proc/sys/net/ipv4/tcp_fastopen
3
raj@tardy-ubuntu-1204:~$ sudo tcpdump -c 30 -i lo 'not port 12865'
[sudo] password for raj:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
14:28:43.536166 IP localhost.29105 > localhost.srvr: Flags [S], seq 378007190, win 43690, options [mss 65495,sackOK,TS val 4294942097 ecr 0,nop,wscale 7,Unknown Option 254f989], length 0
14:28:43.536191 IP localhost.srvr > localhost.29105: Flags [S.], seq 2030806688, ack 378007191, win 43690, options [mss 65495,sackOK,TS val 4294942097 ecr 4294942097,nop,wscale 7,Unknown Option 254f989e73dc061f14d850e], length 0
14:28:43.537421 IP localhost.29105 > localhost.srvr: Flags [P.], seq 1:2, ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942097], length 1
14:28:43.537445 IP localhost.srvr > localhost.29105: Flags [.], ack 2, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.537525 IP localhost.srvr > localhost.29105: Flags [P.], seq 1:2, ack 2, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 1
14:28:43.537542 IP localhost.srvr > localhost.29105: Flags [F.], seq 2, ack 2, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.537727 IP localhost.29105 > localhost.srvr: Flags [F.], seq 2, ack 3, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.537741 IP localhost.srvr > localhost.29105: Flags [.], ack 3, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.537895 IP localhost.29106 > localhost.srvr: Flags [S], seq 1735077945:1735077946, win 43690, options [mss 65495,sackOK,TS val 4294942098 ecr 0,nop,wscale 7,Unknown Option 254f989e73dc061f14d850e], length 1
14:28:43.537909 IP localhost.srvr > localhost.29106: Flags [S.], seq 1983728126, ack 1735077947, win 43690, options [mss 65495,sackOK,TS val 4294942098 ecr 4294942098,nop,wscale 7], length 0
14:28:43.537924 IP localhost.29106 > localhost.srvr: Flags [.], ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538014 IP localhost.srvr > localhost.29106: Flags [P.], seq 1:2, ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 1
14:28:43.538028 IP localhost.srvr > localhost.29106: Flags [F.], seq 2, ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538196 IP localhost.29106 > localhost.srvr: Flags [.], ack 2, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538288 IP localhost.29106 > localhost.srvr: Flags [F.], seq 1, ack 3, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538300 IP localhost.srvr > localhost.29106: Flags [.], ack 2, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538417 IP localhost.29107 > localhost.srvr: Flags [S], seq 3902541042:3902541043, win 43690, options [mss 65495,sackOK,TS val 4294942098 ecr 0,nop,wscale 7,Unknown Option 254f989e73dc061f14d850e], length 1
14:28:43.538431 IP localhost.srvr > localhost.29107: Flags [S.], seq 941945820, ack 3902541044, win 43690, options [mss 65495,sackOK,TS val 4294942098 ecr 4294942098,nop,wscale 7], length 0
14:28:43.538445 IP localhost.29107 > localhost.srvr: Flags [.], ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538518 IP localhost.srvr > localhost.29107: Flags [P.], seq 1:2, ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 1
14:28:43.538531 IP localhost.srvr > localhost.29107: Flags [F.], seq 2, ack 1, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538654 IP localhost.29107 > localhost.srvr: Flags [.], ack 2, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538730 IP localhost.29107 > localhost.srvr: Flags [F.], seq 1, ack 3, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0
14:28:43.538742 IP localhost.srvr > localhost.29107: Flags [.], ack 2, win 342, options [nop,nop,TS val 4294942098 ecr 4294942098], length 0

in which case I suppose that means that netperf top-of-trunk does indeed have client and server side support for TCP_FASTOPEN. Enabled via the test-specific -F option though in the loopback test (in a 1 VCPU VM) I don't see much of a difference (don't suppose I should really):

raj@tardy-ubuntu-1204:~/netperf2_trunk/src$ ./netperf -t TCP_CRR -l 30 -i 30,3 -- -F -P ,12345
MIGRATED TCP Connect/Request/Response TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost () port 12345 AF_INET : +/-2.500% @ 99% conf.  : demo
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate
bytes  Bytes  bytes    bytes   secs.    per sec

16384 87380 1 1 30.00 15909.07
16384 87380

raj@tardy-ubuntu-1204:~/netperf2_trunk/src$ ./netperf -t TCP_CRR -l 30 -i 30,3 --  -P ,12345
MIGRATED TCP Connect/Request/Response TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost () port 12345 AF_INET : +/-2.500% @ 99% conf.  : demo
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate
bytes  Bytes  bytes    bytes   secs.    per sec

16384 87380 1 1 30.00 15574.37
16384 87380

happy benchmarking,

rick jones