Skip to contentSkip to conversation

tcp: add a new TW_PAWS drop reason

PATCH RESEND Superseded Jiayuan Chen 3 participants 8 messages Apr 7, 2025 – Apr 8, 2025

Subsystems (5)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover tcp: add a new TW_PAWS drop reason

PAWS is a long-standing issue, especially when there are upstream network devices, making it more prone to occur.

Currently, packet loss statistics for PAWS can only be viewed through MIB, which is a global metric and cannot be precisely obtained through tracing to get the specific 4-tuple of the dropped packet. In the past, we had to use kprobe ret to retrieve relevant skb information from tcp_timewait_state_process().

---
Re-sending the patch after merge window.

v2 -> v3: use new SNMP counter and drop reason suggested by Eric.
https://lore.kernel.org/netdev/[email protected]/T/#t

I didn't provide a packetdrill script.
I struggled for a long time to get packetdrill to fix the client port, but ultimately failed to do so...

Instead, I wrote my own program to trigger PAWS, which can be found at
https://github.com/mrpre/nettrigger/tree/main
'''
//assume nginx running on 172.31.75.114:9999, current host is 172.31.75.115 iptables -t filter -I OUTPUT -p tcp --sport 12345 --tcp-flags RST RST -j DROP ./nettrigger -i eth0 -s 172.31.75.115:12345 -d 172.31.75.114:9999 -action paws '''

Jiayuan Chen (2):
tcp: add TCP_RFC7323_TW_PAWS drop reason
tcp: add LINUX_MIB_PAWS_TW_REJECTED counter

 Documentation/networking/net_cachelines/snmp.rst | 2 ++
 include/net/dropreason-core.h                    | 7 +++++++
 include/net/tcp.h                                | 3 ++-
 include/uapi/linux/snmp.h                        | 1 +
 net/ipv4/proc.c                                  | 1 +
 net/ipv4/tcp_ipv4.c                              | 3 ++-
 net/ipv4/tcp_minisocks.c                         | 9 ++++++---
 net/ipv6/tcp_ipv6.c                              | 3 ++-
 8 files changed, 23 insertions(+), 6 deletions(-)
Signature · 1 line
2.47.1
#1 lore raw

PATCH RESEND v3 1/2 tcp: add TCP_RFC7323_TW_PAWS drop reason

PAWS is a long-standing issue, especially when there are upstream network devices, making it more prone to occur.

Currently, packet loss statistics for PAWS can only be viewed through MIB, which is a global metric and cannot be precisely obtained through tracing to get the specific 4-tuple of the dropped packet. In the past, we had to use kprobe ret to retrieve relevant skb information from tcp_timewait_state_process().

We add a drop_reason pointer, similar to what previous commit does:
commit e34100c2ecbb ("tcp: add a drop_reason pointer to tcp_check_req()")

This commit addresses the PAWSESTABREJECTED case and also sets the corresponding drop reason.

We use 'pwru' to test.

Before this commit:
''''
./pwru 'port 9999'
2025/04/07 13:40:19 Listening for events..
TUPLE FUNC
172.31.75.115:12345->172.31.75.114:9999(tcp) sk_skb_reason_drop(SKB_DROP_REASON_NOT_SPECIFIED) '''

After this commit:
'''
./pwru 'port 9999'
2025/04/07 13:51:34 Listening for events..
TUPLE FUNC
172.31.75.115:12345->172.31.75.114:9999(tcp) sk_skb_reason_drop(SKB_DROP_REASON_TCP_RFC7323_TW_PAWS) '''

---
 include/net/dropreason-core.h | 6 ++++++
 include/net/tcp.h             | 3 ++-
 net/ipv4/tcp_ipv4.c           | 3 ++-
 net/ipv4/tcp_minisocks.c      | 7 +++++--
 net/ipv6/tcp_ipv6.c           | 3 ++-
 5 files changed, 17 insertions(+), 5 deletions(-)
include/net/dropreason-core.h +6 -0
index e4fdc6b54cef..9701d7f936f6 100644
@@ -40,6 +40,7 @@
 	FN(TCP_OFOMERGE)		\
 	FN(TCP_RFC7323_PAWS)		\
 	FN(TCP_RFC7323_PAWS_ACK)	\
+	FN(TCP_RFC7323_TW_PAWS)		\
 	FN(TCP_RFC7323_TSECR)		\
 	FN(TCP_LISTEN_OVERFLOW)		\
 	FN(TCP_OLD_SEQUENCE)		\
@@ -283,6 +284,11 @@ enum skb_drop_reason {
 	 * Corresponds to LINUX_MIB_PAWS_OLD_ACK.
 	 */
 	SKB_DROP_REASON_TCP_RFC7323_PAWS_ACK,
+	/**
+	 * @SKB_DROP_REASON_TCP_RFC7323_TW_PAWS: PAWS check, socket is in
+	 * TIME_WAIT state.
+	 */
+	SKB_DROP_REASON_TCP_RFC7323_TW_PAWS,
 	/**
 	 * @SKB_DROP_REASON_TCP_RFC7323_TSECR: PAWS check, invalid TSEcr.
 	 * Corresponds to LINUX_MIB_TSECRREJECTED.
include/net/tcp.h +2 -1
index 4450c384ef17..5078ad868fee 100644
@@ -427,7 +427,8 @@ enum tcp_tw_status {
 enum tcp_tw_status tcp_timewait_state_process(struct inet_timewait_sock *tw,
 					      struct sk_buff *skb,
 					      const struct tcphdr *th,
-					      u32 *tw_isn);
+					      u32 *tw_isn,
+					      enum skb_drop_reason *drop_reason);
 struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
 			   struct request_sock *req, bool fastopen,
 			   bool *lost_race, enum skb_drop_reason *drop_reason);
net/ipv4/tcp_ipv4.c +2 -1
index 8cce0d5489da..d5b5c32115d2 100644
@@ -2417,7 +2417,8 @@ int tcp_v4_rcv(struct sk_buff *skb)
 		goto csum_error;
 	}
 
-	tw_status = tcp_timewait_state_process(inet_twsk(sk), skb, th, &isn);
+	tw_status = tcp_timewait_state_process(inet_twsk(sk), skb, th, &isn,
+					       &drop_reason);
 	switch (tw_status) {
 	case TCP_TW_SYN: {
 		struct sock *sk2 = inet_lookup_listener(net,
net/ipv4/tcp_minisocks.c +5 -2
index fb9349be36b8..27511bf58c0f 100644
@@ -97,7 +97,8 @@ static void twsk_rcv_nxt_update(struct tcp_timewait_sock *tcptw, u32 seq,
  */
 enum tcp_tw_status
 tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
-			   const struct tcphdr *th, u32 *tw_isn)
+			   const struct tcphdr *th, u32 *tw_isn,
+			   enum skb_drop_reason *drop_reason)
 {
 	struct tcp_timewait_sock *tcptw = tcp_twsk((struct sock *)tw);
 	u32 rcv_nxt = READ_ONCE(tcptw->tw_rcv_nxt);
@@ -245,8 +246,10 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
 		return TCP_TW_SYN;
 	}
 
-	if (paws_reject)
+	if (paws_reject) {
+		*drop_reason = SKB_DROP_REASON_TCP_RFC7323_TW_PAWS;
 		__NET_INC_STATS(twsk_net(tw), LINUX_MIB_PAWSESTABREJECTED);
+	}
 
 	if (!th->rst) {
 		/* In this case we must reset the TIMEWAIT timer.
net/ipv6/tcp_ipv6.c +2 -1
index b03c223eda4f..7dcb33f879ee 100644
@@ -1970,7 +1970,8 @@ INDIRECT_CALLABLE_SCOPE int tcp_v6_rcv(struct sk_buff *skb)
 		goto csum_error;
 	}
 
-	tw_status = tcp_timewait_state_process(inet_twsk(sk), skb, th, &isn);
+	tw_status = tcp_timewait_state_process(inet_twsk(sk), skb, th, &isn,
+					       &drop_reason);
 	switch (tw_status) {
 	case TCP_TW_SYN:
 	{
Signature · 1 line
2.47.1
Reviewed-by on v3 1/2
#2 lore raw

Re: [PATCH RESEND net-next v3 1/2] tcp: add TCP_RFC7323_TW_PAWS drop reason

Quoted 3 lines · Jiayuan Chen

PAWS is a long-standing issue, especially when there are upstream network devices, making it more prone to occur.

This sentence is quite confusing. I do not think PAWS is an issue at all.

Quoted 10 lines

Currently, packet loss statistics for PAWS can only be viewed through MIB, which is a global metric and cannot be precisely obtained through tracing to get the specific 4-tuple of the dropped packet. In the past, we had to use kprobe ret to retrieve relevant skb information from tcp_timewait_state_process().

#3 lore raw

PATCH RESEND v3 2/2 tcp: add LINUX_MIB_PAWS_TW_REJECTED counter

In reply to Jiayuan Chen

When TCP is in TIME_WAIT state, PAWS verification uses LINUX_PAWSESTABREJECTED, which is ambiguous and cannot be distinguished from other PAWS verification processes.

Moreover, when PAWS occurs in TIME_WAIT, we typically need to pay special attention to upstream network devices, so we added a new counter, like the existing PAWS_OLD_ACK one.

Also we update the doc with previously missing PAWS_OLD_ACK.

usage:
'''
nstat -az | grep PAWSTimewait
TcpExtPAWSTimewait 1 0.0
'''

---
 Documentation/networking/net_cachelines/snmp.rst | 2 ++
 include/net/dropreason-core.h                    | 1 +
 include/uapi/linux/snmp.h                        | 1 +
 net/ipv4/proc.c                                  | 1 +
 net/ipv4/tcp_minisocks.c                         | 2 +-
 5 files changed, 6 insertions(+), 1 deletion(-)
Documentation/networking/net_cachelines/snmp.rst +2 -0
index bc96efc92cf5..bd44b3eebbef 100644
@@ -37,6 +37,8 @@ unsigned_long  LINUX_MIB_TIMEWAITKILLED
 unsigned_long  LINUX_MIB_PAWSACTIVEREJECTED
 unsigned_long  LINUX_MIB_PAWSESTABREJECTED
 unsigned_long  LINUX_MIB_TSECR_REJECTED
+unsigned_long  LINUX_MIB_PAWS_OLD_ACK
+unsigned_long  LINUX_MIB_PAWS_TW_REJECTED
 unsigned_long  LINUX_MIB_DELAYEDACKLOST
 unsigned_long  LINUX_MIB_LISTENOVERFLOWS
 unsigned_long  LINUX_MIB_LISTENDROPS
include/net/dropreason-core.h +1 -0
index 9701d7f936f6..bea77934a235 100644
@@ -287,6 +287,7 @@ enum skb_drop_reason {
 	/**
 	 * @SKB_DROP_REASON_TCP_RFC7323_TW_PAWS: PAWS check, socket is in
 	 * TIME_WAIT state.
+	 * Corresponds to LINUX_MIB_PAWS_TW_REJECTED.
 	 */
 	SKB_DROP_REASON_TCP_RFC7323_TW_PAWS,
 	/**
include/uapi/linux/snmp.h +1 -0
index ec47f9b68a1b..1d234d7e1892 100644
@@ -188,6 +188,7 @@ enum
 	LINUX_MIB_PAWSESTABREJECTED,		/* PAWSEstabRejected */
 	LINUX_MIB_TSECRREJECTED,		/* TSEcrRejected */
 	LINUX_MIB_PAWS_OLD_ACK,			/* PAWSOldAck */
+	LINUX_MIB_PAWS_TW_REJECTED,		/* PAWSTimewait */
 	LINUX_MIB_DELAYEDACKS,			/* DelayedACKs */
 	LINUX_MIB_DELAYEDACKLOCKED,		/* DelayedACKLocked */
 	LINUX_MIB_DELAYEDACKLOST,		/* DelayedACKLost */
net/ipv4/proc.c +1 -0
index 10cbeb76c274..ea2f01584379 100644
@@ -191,6 +191,7 @@ static const struct snmp_mib snmp4_net_list[] = {
 	SNMP_MIB_ITEM("PAWSEstab", LINUX_MIB_PAWSESTABREJECTED),
 	SNMP_MIB_ITEM("TSEcrRejected", LINUX_MIB_TSECRREJECTED),
 	SNMP_MIB_ITEM("PAWSOldAck", LINUX_MIB_PAWS_OLD_ACK),
+	SNMP_MIB_ITEM("PAWSTimewait", LINUX_MIB_PAWS_TW_REJECTED),
 	SNMP_MIB_ITEM("DelayedACKs", LINUX_MIB_DELAYEDACKS),
 	SNMP_MIB_ITEM("DelayedACKLocked", LINUX_MIB_DELAYEDACKLOCKED),
 	SNMP_MIB_ITEM("DelayedACKLost", LINUX_MIB_DELAYEDACKLOST),
net/ipv4/tcp_minisocks.c +1 -1
index 27511bf58c0f..43d7852ce07e 100644
@@ -248,7 +248,7 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
 
 	if (paws_reject) {
 		*drop_reason = SKB_DROP_REASON_TCP_RFC7323_TW_PAWS;
-		__NET_INC_STATS(twsk_net(tw), LINUX_MIB_PAWSESTABREJECTED);
+		__NET_INC_STATS(twsk_net(tw), LINUX_MIB_PAWS_TW_REJECTED);
 	}
 
 	if (!th->rst) {
Signature · 1 line
2.47.1
Reviewed-by on v3 2/2
#4 lore raw

Re: [PATCH RESEND net-next v3 2/2] tcp: add LINUX_MIB_PAWS_TW_REJECTED counter

Quoted 8 lines · Jiayuan Chen

When TCP is in TIME_WAIT state, PAWS verification uses LINUX_PAWSESTABREJECTED, which is ambiguous and cannot be distinguished from other PAWS verification processes.

Moreover, when PAWS occurs in TIME_WAIT, we typically need to pay special attention to upstream network devices, so we added a new counter, like the existing PAWS_OLD_ACK one.

I really dislike the repetition of "upstream network devices".

Is it mentioned in some RFC ?

Quoted 11 lines

Also we update the doc with previously missing PAWS_OLD_ACK.

usage:
'''
nstat -az | grep PAWSTimewait
TcpExtPAWSTimewait 1 0.0
'''

#5 lore raw

Re: [PATCH RESEND net-next v3 2/2] tcp: add LINUX_MIB_PAWS_TW_REJECTED counter

Quoted 12 lines · Eric Dumazet
Quoted 8 lines · Jiayuan Chen

When TCP is in TIME_WAIT state, PAWS verification uses
LINUX_PAWSESTABREJECTED, which is ambiguous and cannot be distinguished from other PAWS verification processes.
Moreover, when PAWS occurs in TIME_WAIT, we typically need to pay special attention to upstream network devices, so we added a new counter, like the existing PAWS_OLD_ACK one.

Quoted 2 lines

I really dislike the repetition of "upstream network devices". Is it mentioned in some RFC ?

I used this term to refer to devices that are located in the path of the TCP connection, such as firewalls, NATs, or routers, which can perform SNAT or DNAT and these network devices use addresses from their own limited address pools to masquerade the source address during forwarding, this can cause PAWS verification to fail more easily.

You are right that this term is not mentioned in RFC but it's commonly used in IT infrastructure contexts. Sorry to have caused misunderstandings.

Thanks.

Quoted 15 lines
Quoted 4 lines

Also we update the doc with previously missing PAWS_OLD_ACK.
usage:

Quoted 8 lines

'''
nstat -az | grep PAWSTimewait
TcpExtPAWSTimewait 1 0.0
'''

maintainer
#6 lore raw

Re: [PATCH RESEND net-next v3 2/2] tcp: add LINUX_MIB_PAWS_TW_REJECTED counter

Quoted 8 lines · On Tue, 08 Apr 2025 14:57:29 +0000 Jiayuan Chen wrote:
Quoted 8 lines
Quoted 7 lines

When TCP is in TIME_WAIT state, PAWS verification uses
LINUX_PAWSESTABREJECTED, which is ambiguous and cannot be distinguished from other PAWS verification processes.
Moreover, when PAWS occurs in TIME_WAIT, we typically need to pay special attention to upstream network devices, so we added a new counter, like the existing PAWS_OLD_ACK one.

Quoted 5 lines
Quoted 2 lines

I really dislike the repetition of "upstream network devices". Is it mentioned in some RFC ?

I used this term to refer to devices that are located in the path of the TCP connection

Could we use some form of: "devices that are located in the path of the TCP connection" ? Maybe just "devices in the networking path" ? I hope that will be sufficiently clear in all contexts. Upstream devices sounds a little like devices which have drivers in upstream Linux kernel :(

Quoted 7 lines

such as firewalls, NATs, or routers, which can perform
SNAT or DNAT and these network devices use addresses from their own limited address pools to masquerade the source address during forwarding, this can cause PAWS verification to fail more easily.

You are right that this term is not mentioned in RFC but it's commonly used in IT infrastructure contexts. Sorry to have caused misunderstandings.

Signature · 1 line
pw-bot: cr
#7 lore raw

Re: [PATCH RESEND net-next v3 2/2] tcp: add LINUX_MIB_PAWS_TW_REJECTED counter

Quoted 19 lines · Jakub Kicinski
Quoted 10 lines · On Tue, 08 Apr 2025 14:57:29 +0000 Jiayuan Chen wrote:
Quoted 8 lines

When TCP is in TIME_WAIT state, PAWS verification uses LINUX_PAWSESTABREJECTED, which is ambiguous and cannot be distinguished from other PAWS verification processes.
Moreover, when PAWS occurs in TIME_WAIT, we typically need to pay special attention to upstream network devices, so we added a new counter, like the existing PAWS_OLD_ACK one.

Quoted 6 lines

I really dislike the repetition of "upstream network devices". Is it mentioned in some RFC ?

I used this term to refer to devices that are located in the path of the TCP connection

Quoted 7 lines

Could we use some form of: "devices that are located in the path of the TCP connection" ? Maybe just "devices in the networking path" ? I hope that will be sufficiently clear in all contexts.

Upstream devices sounds a little like devices which have drivers in

upstream Linux kernel :(

That makes sense :).

Thanks.

Quoted 13 lines
Quoted 8 lines

such as firewalls, NATs, or routers, which can perform
SNAT or DNAT and these network devices use addresses from their own limited address pools to masquerade the source address during forwarding, this can cause PAWS verification to fail more easily.

You are right that this term is not mentioned in RFC but it's commonly used in IT infrastructure contexts. Sorry to have caused misunderstandings.

Signature · 1 line
pw-bot: cr