tcp: add a new TW_PAWS drop reason
Subsystems (5)
v3 cover tcp: add a new TW_PAWS drop reason
PAWS is a long-standing issue, especially when there are upstream network devices, making it more prone to occur.
Currently, packet loss statistics for PAWS can only be viewed through MIB, which is a global metric and cannot be precisely obtained through tracing to get the specific 4-tuple of the dropped packet. In the past, we had to use kprobe ret to retrieve relevant skb information from tcp_timewait_state_process().
---
Re-sending the patch after merge window.
v2 -> v3: use new SNMP counter and drop reason suggested by Eric.
https://lore.kernel.org/netdev/[email protected]/T/#t
I didn't provide a packetdrill script.
I struggled for a long time to get packetdrill to fix the client port, but ultimately failed to do so...
Instead, I wrote my own program to trigger PAWS, which can be found at
https://github.com/mrpre/nettrigger/tree/main
'''
//assume nginx running on 172.31.75.114:9999, current host is 172.31.75.115 iptables -t filter -I OUTPUT -p tcp --sport 12345 --tcp-flags RST RST -j DROP ./nettrigger -i eth0 -s 172.31.75.115:12345 -d 172.31.75.114:9999 -action paws '''
Jiayuan Chen (2):
tcp: add TCP_RFC7323_TW_PAWS drop reason
tcp: add LINUX_MIB_PAWS_TW_REJECTED counter
Documentation/networking/net_cachelines/snmp.rst | 2 ++ include/net/dropreason-core.h | 7 +++++++ include/net/tcp.h | 3 ++- include/uapi/linux/snmp.h | 1 + net/ipv4/proc.c | 1 + net/ipv4/tcp_ipv4.c | 3 ++- net/ipv4/tcp_minisocks.c | 9 ++++++--- net/ipv6/tcp_ipv6.c | 3 ++- 8 files changed, 23 insertions(+), 6 deletions(-)
Signature · 1 line
2.47.1
PATCH RESEND v3 1/2 tcp: add TCP_RFC7323_TW_PAWS drop reason
PAWS is a long-standing issue, especially when there are upstream network devices, making it more prone to occur.
Currently, packet loss statistics for PAWS can only be viewed through MIB, which is a global metric and cannot be precisely obtained through tracing to get the specific 4-tuple of the dropped packet. In the past, we had to use kprobe ret to retrieve relevant skb information from tcp_timewait_state_process().
We add a drop_reason pointer, similar to what previous commit does:
commit e34100c2ecbb ("tcp: add a drop_reason pointer to tcp_check_req()")
This commit addresses the PAWSESTABREJECTED case and also sets the corresponding drop reason.
We use 'pwru' to test.
Before this commit:
''''
./pwru 'port 9999'
2025/04/07 13:40:19 Listening for events..
TUPLE FUNC
172.31.75.115:12345->172.31.75.114:9999(tcp) sk_skb_reason_drop(SKB_DROP_REASON_NOT_SPECIFIED) '''
After this commit:
'''
./pwru 'port 9999'
2025/04/07 13:51:34 Listening for events..
TUPLE FUNC
172.31.75.115:12345->172.31.75.114:9999(tcp) sk_skb_reason_drop(SKB_DROP_REASON_TCP_RFC7323_TW_PAWS) '''
--- include/net/dropreason-core.h | 6 ++++++ include/net/tcp.h | 3 ++- net/ipv4/tcp_ipv4.c | 3 ++- net/ipv4/tcp_minisocks.c | 7 +++++-- net/ipv6/tcp_ipv6.c | 3 ++- 5 files changed, 17 insertions(+), 5 deletions(-)
include/net/dropreason-core.h +6 -0
@@ -40,6 +40,7 @@ FN(TCP_OFOMERGE) \ FN(TCP_RFC7323_PAWS) \ FN(TCP_RFC7323_PAWS_ACK) \ + FN(TCP_RFC7323_TW_PAWS) \ FN(TCP_RFC7323_TSECR) \ FN(TCP_LISTEN_OVERFLOW) \ FN(TCP_OLD_SEQUENCE) \
@@ -283,6 +284,11 @@ enum skb_drop_reason {* Corresponds to LINUX_MIB_PAWS_OLD_ACK. */ SKB_DROP_REASON_TCP_RFC7323_PAWS_ACK , +/** +* @SKB_DROP_REASON_TCP_RFC7323_TW_PAWS: PAWS check, socket is in +* TIME_WAIT state. +*/ + SKB_DROP_REASON_TCP_RFC7323_TW_PAWS,/** * @SKB_DROP_REASON_TCP_RFC7323_TSECR: PAWS check, invalid TSEcr. * Corresponds to LINUX_MIB_TSECRREJECTED.
include/net/tcp.h +2 -1
@@ -427,7 +427,8 @@ enum tcp_tw_status {enum tcp_tw_status tcp_timewait_state_process (struct inet_timewait_sock *tw,struct sk_buff *skb,const struct tcphdr *th, -u32 *tw_isn); +u32 *tw_isn, +enum skb_drop_reason *drop_reason);struct sock *tcp_check_req (struct sock *sk,struct sk_buff *skb,struct request_sock *req,bool fastopen,bool *lost_race,enum skb_drop_reason *drop_reason);
net/ipv4/tcp_ipv4.c +2 -1
@@ -2417,7 +2417,8 @@ int tcp_v4_rcv(struct sk_buff *skb) goto csum_error; } - tw_status =tcp_timewait_state_process (inet_twsk (sk), skb, th, &isn); + tw_status =tcp_timewait_state_process (inet_twsk (sk), skb, th, &isn, + &drop_reason);switch (tw_status) {case TCP_TW_SYN : { struct sock *sk2 = inet_lookup_listener(net,
net/ipv4/tcp_minisocks.c +5 -2
@@ -97,7 +97,8 @@ static void twsk_rcv_nxt_update(struct tcp_timewait_sock *tcptw, u32 seq,*/ enum tcp_tw_status tcp_timewait_state_process (struct inet_timewait_sock *tw,struct sk_buff *skb, -const struct tcphdr *th,u32 *tw_isn) +const struct tcphdr *th,u32 *tw_isn, +enum skb_drop_reason *drop_reason) {struct tcp_timewait_sock *tcptw =tcp_twsk ((struct sock *)tw);u32 rcv_nxt =READ_ONCE (tcptw->tw_rcv_nxt);
@@ -245,8 +246,10 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,return TCP_TW_SYN ; } -if (paws_reject) +if (paws_reject) { + *drop_reason =SKB_DROP_REASON_TCP_RFC7323_TW_PAWS ;__NET_INC_STATS (twsk_net (tw),LINUX_MIB_PAWSESTABREJECTED ); + }if (!th->rst) {/* In this case we must reset the TIMEWAIT timer.
net/ipv6/tcp_ipv6.c +2 -1
@@ -1970,7 +1970,8 @@ INDIRECT_CALLABLE_SCOPE int tcp_v6_rcv(struct sk_buff *skb) goto csum_error; } - tw_status =tcp_timewait_state_process (inet_twsk (sk), skb, th, &isn); + tw_status =tcp_timewait_state_process (inet_twsk (sk), skb, th, &isn, + &drop_reason);switch (tw_status) {case TCP_TW_SYN : {
Signature · 1 line
2.47.1
Re: [PATCH RESEND net-next v3 1/2] tcp: add TCP_RFC7323_TW_PAWS drop reason
Quoted 3 lines · Jiayuan Chen
PAWS is a long-standing issue, especially when there are upstream network devices, making it more prone to occur.
This sentence is quite confusing. I do not think PAWS is an issue at all.
Quoted 10 lines
Currently, packet loss statistics for PAWS can only be viewed through MIB, which is a global metric and cannot be precisely obtained through tracing to get the specific 4-tuple of the dropped packet. In the past, we had to use kprobe ret to retrieve relevant skb information from
tcp_timewait_state_process().Suggested-by Eric Dumazet <[email protected]>Signed-off-by Jiayuan Chen <[email protected]>
PATCH RESEND v3 2/2 tcp: add LINUX_MIB_PAWS_TW_REJECTED counter
In reply to Jiayuan ChenWhen TCP is in TIME_WAIT state, PAWS verification uses LINUX_PAWSESTABREJECTED, which is ambiguous and cannot be distinguished from other PAWS verification processes.
Moreover, when PAWS occurs in TIME_WAIT, we typically need to pay special attention to upstream network devices, so we added a new counter, like the existing PAWS_OLD_ACK one.
Also we update the doc with previously missing PAWS_OLD_ACK.
usage:
'''
nstat -az | grep PAWSTimewait
TcpExtPAWSTimewait 1 0.0
'''
--- Documentation/networking/net_cachelines/snmp.rst | 2 ++ include/net/dropreason-core.h | 1 + include/uapi/linux/snmp.h | 1 + net/ipv4/proc.c | 1 + net/ipv4/tcp_minisocks.c | 2 +- 5 files changed, 6 insertions(+), 1 deletion(-)
Documentation/networking/net_cachelines/snmp.rst +2 -0
@@ -37,6 +37,8 @@ unsigned_long LINUX_MIB_TIMEWAITKILLED unsigned_long LINUX_MIB_PAWSACTIVEREJECTED unsigned_long LINUX_MIB_PAWSESTABREJECTED unsigned_long LINUX_MIB_TSECR_REJECTED +unsigned_long LINUX_MIB_PAWS_OLD_ACK +unsigned_long LINUX_MIB_PAWS_TW_REJECTED unsigned_long LINUX_MIB_DELAYEDACKLOST unsigned_long LINUX_MIB_LISTENOVERFLOWS unsigned_long LINUX_MIB_LISTENDROPS
include/net/dropreason-core.h +1 -0
@@ -287,6 +287,7 @@ enum skb_drop_reason {/** * @SKB_DROP_REASON_TCP_RFC7323_TW_PAWS: PAWS check, socket is in * TIME_WAIT state. +* Corresponds to LINUX_MIB_PAWS_TW_REJECTED. */ SKB_DROP_REASON_TCP_RFC7323_TW_PAWS,/**
include/uapi/linux/snmp.h +1 -0
@@ -188,6 +188,7 @@ enumLINUX_MIB_PAWSESTABREJECTED ,/* PAWSEstabRejected */ LINUX_MIB_TSECRREJECTED ,/* TSEcrRejected */ LINUX_MIB_PAWS_OLD_ACK ,/* PAWSOldAck */ +LINUX_MIB_PAWS_TW_REJECTED ,/* PAWSTimewait */ LINUX_MIB_DELAYEDACKS ,/* DelayedACKs */ LINUX_MIB_DELAYEDACKLOCKED ,/* DelayedACKLocked */ LINUX_MIB_DELAYEDACKLOST, /* DelayedACKLost */
net/ipv4/proc.c +1 -0
@@ -191,6 +191,7 @@ static const struct snmp_mib snmp4_net_list[] = {SNMP_MIB_ITEM ("PAWSEstab" ,LINUX_MIB_PAWSESTABREJECTED ),SNMP_MIB_ITEM ("TSEcrRejected" ,LINUX_MIB_TSECRREJECTED ),SNMP_MIB_ITEM ("PAWSOldAck" ,LINUX_MIB_PAWS_OLD_ACK ), +SNMP_MIB_ITEM ("PAWSTimewait" ,LINUX_MIB_PAWS_TW_REJECTED ),SNMP_MIB_ITEM ("DelayedACKs" ,LINUX_MIB_DELAYEDACKS ),SNMP_MIB_ITEM ("DelayedACKLocked" ,LINUX_MIB_DELAYEDACKLOCKED ), SNMP_MIB_ITEM("DelayedACKLost", LINUX_MIB_DELAYEDACKLOST),
net/ipv4/tcp_minisocks.c +1 -1
@@ -248,7 +248,7 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,if (paws_reject) { *drop_reason =SKB_DROP_REASON_TCP_RFC7323_TW_PAWS ; -__NET_INC_STATS (twsk_net (tw),LINUX_MIB_PAWSESTABREJECTED ); +__NET_INC_STATS (twsk_net (tw),LINUX_MIB_PAWS_TW_REJECTED ); }if (!th->rst) {
Signature · 1 line
2.47.1
Re: [PATCH RESEND net-next v3 2/2] tcp: add LINUX_MIB_PAWS_TW_REJECTED counter
Quoted 8 lines · Jiayuan Chen
When TCP is in
TIME_WAITstate, PAWS verification usesLINUX_PAWSESTABREJECTED, which is ambiguous and cannot be distinguished from other PAWS verification processes.Moreover, when PAWS occurs in
TIME_WAIT, we typically need to pay special attention to upstream network devices, so we added a new counter, like the existingPAWS_OLD_ACKone.
I really dislike the repetition of "upstream network devices".
Is it mentioned in some RFC ?
Quoted 11 lines
Also we update the doc with previously missing
PAWS_OLD_ACK.usage:
'''
nstat -az | grep PAWSTimewait
TcpExtPAWSTimewait 1 0.0
'''Suggested-by Eric Dumazet <[email protected]>Signed-off-by Jiayuan Chen <[email protected]>
Re: [PATCH RESEND net-next v3 2/2] tcp: add LINUX_MIB_PAWS_TW_REJECTED counter
Quoted 12 lines · Eric Dumazet
Quoted 8 lines · Jiayuan Chen
When TCP is in
TIME_WAITstate, PAWS verification uses
LINUX_PAWSESTABREJECTED, which is ambiguous and cannot be distinguished from other PAWS verification processes.
Moreover, when PAWS occurs inTIME_WAIT, we typically need to pay special attention to upstream network devices, so we added a new counter, like the existingPAWS_OLD_ACKone.
Quoted 2 lines
I really dislike the repetition of "upstream network devices". Is it mentioned in some RFC ?
I used this term to refer to devices that are located in the path of the TCP connection, such as firewalls, NATs, or routers, which can perform SNAT or DNAT and these network devices use addresses from their own limited address pools to masquerade the source address during forwarding, this can cause PAWS verification to fail more easily.
You are right that this term is not mentioned in RFC but it's commonly used in IT infrastructure contexts. Sorry to have caused misunderstandings.
Thanks.
Quoted 15 lines
Quoted 4 lines
Also we update the doc with previously missing
PAWS_OLD_ACK.
usage:Quoted 8 lines
'''
nstat -az | grep PAWSTimewait
TcpExtPAWSTimewait 1 0.0
'''Suggested-by Eric Dumazet <[email protected]>Signed-off-by Jiayuan Chen <[email protected]>Reviewed-by Eric Dumazet <[email protected]>
Jakub Kicinski
maintainerRe: [PATCH RESEND net-next v3 2/2] tcp: add LINUX_MIB_PAWS_TW_REJECTED counter
Quoted 8 lines · On Tue, 08 Apr 2025 14:57:29 +0000 Jiayuan Chen wrote:
Quoted 8 lines
Quoted 7 lines
When TCP is in
TIME_WAITstate, PAWS verification uses
LINUX_PAWSESTABREJECTED, which is ambiguous and cannot be distinguished from other PAWS verification processes.
Moreover, when PAWS occurs inTIME_WAIT, we typically need to pay special attention to upstream network devices, so we added a new counter, like the existingPAWS_OLD_ACKone.
Quoted 5 lines
Quoted 2 lines
I really dislike the repetition of "upstream network devices". Is it mentioned in some RFC ?
I used this term to refer to devices that are located in the path of the TCP connection
Could we use some form of: "devices that are located in the path of the TCP connection" ? Maybe just "devices in the networking path" ? I hope that will be sufficiently clear in all contexts. Upstream devices sounds a little like devices which have drivers in upstream Linux kernel :(
Quoted 7 lines
such as firewalls, NATs, or routers, which can perform
SNAT or DNAT and these network devices use addresses from their own limited address pools to masquerade the source address during forwarding, this can cause PAWS verification to fail more easily.You are right that this term is not mentioned in RFC but it's commonly used in IT infrastructure contexts. Sorry to have caused misunderstandings.
Signature · 1 line
pw-bot: cr
Re: [PATCH RESEND net-next v3 2/2] tcp: add LINUX_MIB_PAWS_TW_REJECTED counter
Quoted 19 lines · Jakub Kicinski
Quoted 10 lines · On Tue, 08 Apr 2025 14:57:29 +0000 Jiayuan Chen wrote:
Quoted 8 lines
When TCP is in
TIME_WAITstate, PAWS verification usesLINUX_PAWSESTABREJECTED, which is ambiguous and cannot be distinguished from other PAWS verification processes.
Moreover, when PAWS occurs inTIME_WAIT, we typically need to pay special attention to upstream network devices, so we added a new counter, like the existingPAWS_OLD_ACKone.Quoted 6 lines
I really dislike the repetition of "upstream network devices". Is it mentioned in some RFC ?
I used this term to refer to devices that are located in the path of the TCP connection
Quoted 7 lines
Could we use some form of: "devices that are located in the path of the TCP connection" ? Maybe just "devices in the networking path" ? I hope that will be sufficiently clear in all contexts.
Upstream devices sounds a little like devices which have drivers in
upstream Linux kernel :(
That makes sense :).
Thanks.
Quoted 13 lines
Quoted 8 lines
such as firewalls, NATs, or routers, which can perform
SNAT or DNAT and these network devices use addresses from their own limited address pools to masquerade the source address during forwarding, this can cause PAWS verification to fail more easily.You are right that this term is not mentioned in RFC but it's commonly used in IT infrastructure contexts. Sorry to have caused misunderstandings.
Signature · 1 line
pw-bot: cr