Skip to contentSkip to conversation

TCP: Fast-Open observability

PATCH RESEND In review Jeremy Harris 5 participants 12 messages Apr 16, 2025 – Apr 23, 2025

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover TCP: Fast-Open observability

Whether TCP Fast Open was used for a connection is not reliably observable by an accepting application when the SYN passed no data.

Fix this by noting during SYN receive processing that an acceptable Fast Open option was used, and provide this to userland via getsockopt TCP_INFO.

Jeremy Harris (2):
TCP: note received valid-cookie Fast Open option
TCP: pass accepted-TFO indication through getsockopt

 include/linux/tcp.h      | 3 ++-
 include/uapi/linux/tcp.h | 1 +
 net/ipv4/tcp.c           | 2 ++
 net/ipv4/tcp_fastopen.c  | 1 +
 4 files changed, 6 insertions(+), 1 deletion(-)

base-commit: 24e31e4747698dc50d408f0082d7eb9b9520d2f6

Signature · 1 line
2.49.0
#1 lore raw

PATCH RESEND 1/2 TCP: note received valid-cookie Fast Open option

---
 include/linux/tcp.h     | 3 ++-
 net/ipv4/tcp_fastopen.c | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)
include/linux/tcp.h +2 -1
index 1669d95bb0f9..a96c38574bce 100644
@@ -385,7 +385,8 @@ struct tcp_sock {
 		syn_fastopen:1,	/* SYN includes Fast Open option */
 		syn_fastopen_exp:1,/* SYN includes Fast Open exp. option */
 		syn_fastopen_ch:1, /* Active TFO re-enabling probe */
-		syn_data_acked:1;/* data in SYN is acked by SYN-ACK */
+		syn_data_acked:1,/* data in SYN is acked by SYN-ACK */
+		syn_fastopen_in:1; /* Received SYN includes Fast Open option */
 
 	u8	keepalive_probes; /* num of allowed keep alive probes	*/
 	u32	tcp_tx_delay;	/* delay (in usec) added to TX packets */
net/ipv4/tcp_fastopen.c +1 -0
index 1a6b1bc54245..004d0024cd98 100644
@@ -401,6 +401,7 @@ struct sock *tcp_try_fastopen(struct sock *sk, struct sk_buff *skb,
 				}
 				NET_INC_STATS(sock_net(sk),
 					      LINUX_MIB_TCPFASTOPENPASSIVE);
+				tcp_sk(child)->syn_fastopen_in = 1;
 				return child;
 			}
 			NET_INC_STATS(sock_net(sk),
Signature · 1 line
2.49.0
maintainer
#2 lore raw

Re: [RESEND PATCH 1/2] TCP: note received valid-cookie Fast Open option

Quoted 1 line · Jeremy Harris

The commit description is missing.

Quoted 10 lines
---
 include/linux/tcp.h     | 3 ++-
 net/ipv4/tcp_fastopen.c | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)
include/linux/tcp.h +0 -0
index 1669d95bb0f9..a96c38574bce 100644

include/linux/tcp.h struct tcp_sock { · line 389

Quoted 6 lines
include/linux/tcp.h +2 -1
@@ -385,7 +385,8 @@ struct tcp_sock {
 		syn_fastopen:1,	/* SYN includes Fast Open option */
 		syn_fastopen_exp:1,/* SYN includes Fast Open exp. option */
 		syn_fastopen_ch:1, /* Active TFO re-enabling probe */
-		syn_data_acked:1;/* data in SYN is acked by SYN-ACK */
+		syn_data_acked:1,/* data in SYN is acked by SYN-ACK */
+		syn_fastopen_in:1; /* Received SYN includes Fast Open option */

Worth mentioning in the commit message that this will fill a bit hole.

net/ipv4/tcp_fastopen.c struct sock *tcp_try_fastopen(struct sock *sk, struct sk_buff *skb, · line 404

Quoted 12 lines
include/linux/tcp.h +0 -0
 	u8	keepalive_probes; /* num of allowed keep alive probes	*/
 	u32	tcp_tx_delay;	/* delay (in usec) added to TX packets */
net/ipv4/tcp_fastopen.c +1 -0
index 1a6b1bc54245..004d0024cd98 100644
@@ -401,6 +401,7 @@ struct sock *tcp_try_fastopen(struct sock *sk, struct sk_buff *skb,
 				}
 				NET_INC_STATS(sock_net(sk),
 					      LINUX_MIB_TCPFASTOPENPASSIVE);
+				tcp_sk(child)->syn_fastopen_in = 1;

Likely you need to reset the bit on disconnect().

/P

#3 lore raw

Re: [RESEND PATCH 1/2] TCP: note received valid-cookie Fast Open option

Thanks for the review. I'm preparing a v2, and a patch to "ss" in iproute2.

Quoted 1 line · Paolo Abeni

The commit description is missing.

Ack.

Quoted 3 lines

include/linux/tcp.h struct tcp_sock { · line 389

Quoted 1 line
include/linux/tcp.h +1 -0
+		syn_fastopen_in:1; /* Received SYN includes Fast Open option */

Worth mentioning in the commit message that this will fill a bit hole.

Will do.

Quoted 7 lines

net/ipv4/tcp_fastopen.c struct sock *tcp_try_fastopen(struct sock *sk, struct sk_buff *skb, · line 404

Quoted 5 lines
net/ipv4/tcp_fastopen.c +1 -0
@@ -401,6 +401,7 @@ struct sock *tcp_try_fastopen(struct sock *sk, struct sk_buff *skb,
  				}
  				NET_INC_STATS(sock_net(sk),
  					      LINUX_MIB_TCPFASTOPENPASSIVE);
+				tcp_sk(child)->syn_fastopen_in = 1;

Likely you need to reset the bit on disconnect().

Added.

Signature · 2 lines
Cheers,
  Jeremy
maintainer
#4 lore raw

Re: [RESEND PATCH 1/2] TCP: note received valid-cookie Fast Open option

In reply to Jeremy Harris
Quoted 2 lines · Jeremy Harris

The suggested commit title is:

TCP: note received valid-cookie Fast Open option

The "TCP:" prefix is not the typical prefix for Linux TCP changes. A "tcp:" is much more common.

Please follow the convention that we try to adhere to for TCP TFO changes by using something like:

tcp: fastopen: note received valid-cookie Fast Open option

Quoted 10 lines
---
 include/linux/tcp.h     | 3 ++-
 net/ipv4/tcp_fastopen.c | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)
include/linux/tcp.h +0 -0
index 1669d95bb0f9..a96c38574bce 100644

include/linux/tcp.h struct tcp_sock { · line 389

Quoted 6 lines
include/linux/tcp.h +2 -1
@@ -385,7 +385,8 @@ struct tcp_sock {
                syn_fastopen:1, /* SYN includes Fast Open option */
                syn_fastopen_exp:1,/* SYN includes Fast Open exp. option */
                syn_fastopen_ch:1, /* Active TFO re-enabling probe */
-               syn_data_acked:1;/* data in SYN is acked by SYN-ACK */
+               syn_data_acked:1,/* data in SYN is acked by SYN-ACK */
+               syn_fastopen_in:1; /* Received SYN includes Fast Open option */

IMHO this field name and comment are slightly misleading.

Sometimes when a SYN is received with a TFO option the server will fail to create a child because the TFO cookie is incorrect.

When this bit is set, we know not only that the "Received SYN includes Fast Open option", but we also know that the TFO cookie was correct and a child socket was created.

So I would suggest a more specific comment and field name, like:

syn_fastopen_child:1; /* created TFO passive child socket */

thanks,
neal

#5 lore raw

Re: [RESEND PATCH 1/2] TCP: note received valid-cookie Fast Open option

Thanks for the review. I'm preparing a v2, and a patch to "ss" in iproute2.

Quoted 7 lines

The "TCP:" prefix is not the typical prefix for Linux TCP changes. A "tcp:" is much more common.

Please follow the convention that we try to adhere to for TCP TFO changes by using something like:

tcp: fastopen: note received valid-cookie Fast Open option

Will change for v2.

Quoted 9 lines

include/linux/tcp.h struct tcp_sock { · line 389

Quoted 3 lines
include/linux/tcp.h +2 -1
-               syn_data_acked:1;/* data in SYN is acked by SYN-ACK */
+               syn_data_acked:1,/* data in SYN is acked by SYN-ACK */
+               syn_fastopen_in:1; /* Received SYN includes Fast Open option */

IMHO this field name and comment are slightly misleading.

Sometimes when a SYN is received with a TFO option the server will fail to create a child because the TFO cookie is incorrect.

Quoted 7 lines

When this bit is set, we know not only that the "Received SYN includes Fast Open option", but we also know that the TFO cookie was correct and a child socket was created.

So I would suggest a more specific comment and field name, like:

syn_fastopen_child:1; /* created TFO passive child socket */

Done.

Signature · 2 lines
Cheers,
  Jeremy
#6 lore raw

PATCH RESEND 2/2 TCP: pass accepted-TFO indication through getsockopt

In reply to Jeremy Harris
---
 include/uapi/linux/tcp.h | 1 +
 net/ipv4/tcp.c           | 2 ++
 2 files changed, 3 insertions(+)
include/uapi/linux/tcp.h +1 -0
index dc8fdc80e16b..ae8c5a8af0e5 100644
@@ -184,6 +184,7 @@ enum tcp_fastopen_client_fail {
 #define TCPI_OPT_ECN_SEEN	16 /* we received at least one packet with ECT */
 #define TCPI_OPT_SYN_DATA	32 /* SYN-ACK acked data in SYN sent or rcvd */
 #define TCPI_OPT_USEC_TS	64 /* usec timestamps */
+#define TCPI_OPT_TFO_SEEN	128 /* we accepted a Fast Open option on SYN */
 
 /*
  * Sender's congestion state indicating normal or abnormal situations
net/ipv4/tcp.c +2 -0
index e0e96f8fd47c..b45eb7cb2909 100644
@@ -4164,6 +4164,8 @@ void tcp_get_info(struct sock *sk, struct tcp_info *info)
 		info->tcpi_options |= TCPI_OPT_SYN_DATA;
 	if (tp->tcp_usec_ts)
 		info->tcpi_options |= TCPI_OPT_USEC_TS;
+	if (tp->syn_fastopen_in)
+		info->tcpi_options |= TCPI_OPT_TFO_SEEN;
 
 	info->tcpi_rto = jiffies_to_usecs(icsk->icsk_rto);
 	info->tcpi_ato = jiffies_to_usecs(min_t(u32, icsk->icsk_ack.ato,
Signature · 1 line
2.49.0
#7 lore raw

Re: [RESEND PATCH 2/2] TCP: pass accepted-TFO indication through getsockopt

From: Jeremy Harris <[email protected]>
Date: Wed, 16 Apr 2025 10:15:38 +0100

Quoted 11 lines
---
 include/uapi/linux/tcp.h | 1 +
 net/ipv4/tcp.c           | 2 ++
 2 files changed, 3 insertions(+)
include/uapi/linux/tcp.h +0 -0
index dc8fdc80e16b..ae8c5a8af0e5 100644

include/uapi/linux/tcp.h enum tcp_fastopen_client_fail { · line 187

Quoted 4 lines
include/uapi/linux/tcp.h +1 -0
@@ -184,6 +184,7 @@ enum tcp_fastopen_client_fail {
 #define TCPI_OPT_ECN_SEEN	16 /* we received at least one packet with ECT */
 #define TCPI_OPT_SYN_DATA	32 /* SYN-ACK acked data in SYN sent or rcvd */
 #define TCPI_OPT_USEC_TS	64 /* usec timestamps */
+#define TCPI_OPT_TFO_SEEN	128 /* we accepted a Fast Open option on SYN */

This is the last bit of tcpi_options.

We can add another field at the end of the struct if necessary in the future, but from the cover letter and commit message, I didn't get why this info is needed at per-socket granurality.

Also, This can be retrieved with BPF.

Quoted 18 lines
include/uapi/linux/tcp.h +0 -0
 /*
  * Sender's congestion state indicating normal or abnormal situations
net/ipv4/tcp.c +2 -0
index e0e96f8fd47c..b45eb7cb2909 100644
@@ -4164,6 +4164,8 @@ void tcp_get_info(struct sock *sk, struct tcp_info *info)
 		info->tcpi_options |= TCPI_OPT_SYN_DATA;
 	if (tp->tcp_usec_ts)
 		info->tcpi_options |= TCPI_OPT_USEC_TS;
+	if (tp->syn_fastopen_in)
+		info->tcpi_options |= TCPI_OPT_TFO_SEEN;
 
 	info->tcpi_rto = jiffies_to_usecs(icsk->icsk_rto);
 	info->tcpi_ato = jiffies_to_usecs(min_t(u32, icsk->icsk_ack.ato,
Signature · 1 line
2.49.0
#8 lore raw

Re: [RESEND PATCH 2/2] TCP: pass accepted-TFO indication through getsockopt

Quoted 7 lines · Kuniyuki Iwashima
Quoted 1 line

+#define TCPI_OPT_TFO_SEEN 128 /* we accepted a Fast Open option on SYN */

This is the last bit of tcpi_options.

We can add another field at the end of the struct if necessary in the future, but from the cover letter and commit message, I didn't get why this info is needed at per-socket granurality.

An application is operating at socket granularity. The provision is so that an application can tell what is going on with each one of its comms channels independently, rather than (for instance) the rolled-up information on the entire system that the existing statistics counter offers.

Quoted 1 line

Also, This can be retrieved with BPF.

If I understand correctly, BPF is aimed at sysadminss probing or modifying system operations rather than production use by an application. While your point is true, I don't think BPF is a good fit.

Signature · 2 lines
Cheers,
  Jeremy
maintainer
#9 lore raw

Re: [RESEND PATCH 2/2] TCP: pass accepted-TFO indication through getsockopt

In reply to Jeremy Harris
Quoted 1 line · Jeremy Harris

The commit message is missing.

Quoted 22 lines
---
 include/uapi/linux/tcp.h | 1 +
 net/ipv4/tcp.c           | 2 ++
 2 files changed, 3 insertions(+)
include/uapi/linux/tcp.h +1 -0
index dc8fdc80e16b..ae8c5a8af0e5 100644
@@ -184,6 +184,7 @@ enum tcp_fastopen_client_fail {
 #define TCPI_OPT_ECN_SEEN	16 /* we received at least one packet with ECT */
 #define TCPI_OPT_SYN_DATA	32 /* SYN-ACK acked data in SYN sent or rcvd */
 #define TCPI_OPT_USEC_TS	64 /* usec timestamps */
+#define TCPI_OPT_TFO_SEEN	128 /* we accepted a Fast Open option on SYN */
 
 /*
  * Sender's congestion state indicating normal or abnormal situations
net/ipv4/tcp.c +0 -0
index e0e96f8fd47c..b45eb7cb2909 100644

net/ipv4/tcp.c void tcp_get_info(struct sock *sk, struct tcp_info *info) · line 4168

Quoted 5 lines
net/ipv4/tcp.c +2 -0
@@ -4164,6 +4164,8 @@ void tcp_get_info(struct sock *sk, struct tcp_info *info)
 		info->tcpi_options |= TCPI_OPT_SYN_DATA;
 	if (tp->tcp_usec_ts)
 		info->tcpi_options |= TCPI_OPT_USEC_TS;
+	if (tp->syn_fastopen_in)
+		info->tcpi_options |= TCPI_OPT_TFO_SEEN;

I guess a paired iproute2 change is needed to really observe the new info.

And it would be nice to leverage such thing to later add a paired self-test for this feature.

Thanks,

Paolo

maintainer
#10 lore raw

Re: [RESEND PATCH 2/2] TCP: pass accepted-TFO indication through getsockopt

In reply to Jeremy Harris
Quoted 2 lines · Jeremy Harris

Regarding the proposed commit title:

TCP: pass accepted-TFO indication through getsockopt

Please use something more like:

tcp: fastopen: pass TFO child indication through getsockopt

Quoted 10 lines
---
 include/uapi/linux/tcp.h | 1 +
 net/ipv4/tcp.c           | 2 ++
 2 files changed, 3 insertions(+)
include/uapi/linux/tcp.h +0 -0
index dc8fdc80e16b..ae8c5a8af0e5 100644

include/uapi/linux/tcp.h enum tcp_fastopen_client_fail { · line 187

Quoted 4 lines
include/uapi/linux/tcp.h +1 -0
@@ -184,6 +184,7 @@ enum tcp_fastopen_client_fail {
 #define TCPI_OPT_ECN_SEEN      16 /* we received at least one packet with ECT */
 #define TCPI_OPT_SYN_DATA      32 /* SYN-ACK acked data in SYN sent or rcvd */
 #define TCPI_OPT_USEC_TS       64 /* usec timestamps */
+#define TCPI_OPT_TFO_SEEN      128 /* we accepted a Fast Open option on SYN */

IMHO this bit name is slightly misleading, and does not match the comment.

Sometimes when a SYN is received with a TFO option the server will fail to create a child because the TFO cookie is incorrect. In such a case, a TFO option is "seen", but TFO is not *used* because the TFO cookie is incorrect; so this "TFO_SEEN" bit would be 0 even though a TFO option was "seen". IMHO that is confusing/misleading.

When this bit is set, we know not only that the "Received SYN includes Fast Open option" (comment from the previous patch), but we also know that the TFO cookie was correct and a child socket was created.

So I would suggest a more specific bit name, something like:

+#define TCPI_OPT_TFO_CHILD 128 /* child from a Fast Open option on SYN */

+ if (tp->syn_fastopen_child)
+ info->tcpi_options |= TCPI_OPT_TFO_CHILD;

thanks,
neal

#11 lore raw

Re: [RESEND PATCH 2/2] TCP: pass accepted-TFO indication through getsockopt

Thanks for the review. I'm preparing a v2, and a patch to "ss" in iproute2.

Quoted 5 lines · Neal Cardwell

TCP: pass accepted-TFO indication through getsockopt

Please use something more like:

tcp: fastopen: pass TFO child indication through getsockopt

Will change for v2.

Quoted 16 lines
Quoted 1 line

+#define TCPI_OPT_TFO_SEEN 128 /* we accepted a Fast Open option on SYN */

IMHO this bit name is slightly misleading, and does not match the comment.

Sometimes when a SYN is received with a TFO option the server will fail to create a child because the TFO cookie is incorrect. In such a case, a TFO option is "seen", but TFO is not *used* because the TFO cookie is incorrect; so this "TFO_SEEN" bit would be 0 even though a TFO option was "seen". IMHO that is confusing/misleading.

When this bit is set, we know not only that the "Received SYN includes Fast Open option" (comment from the previous patch), but we also know that the TFO cookie was correct and a child socket was created.

So I would suggest a more specific bit name, something like:

Review

Quoted 4 lines
(quoted diff) +3 -0
+#define TCPI_OPT_TFO_CHILD      128 /* child from a Fast Open option on SYN */
 
+       if (tp->syn_fastopen_child)
+               info->tcpi_options |= TCPI_OPT_TFO_CHILD;

Done.

Signature · 2 lines
Cheers,
  Jeremy