tcp: add a new PAWS_ACK drop reason
Subsystems (3)
v2 cover tcp: add a new PAWS_ACK drop reason
Current TCP_RFC7323_PAWS drop reason is too generic and can cause confusion.
One common source for these drops are ACK packets coming too late.
A prior packet with payload already changed tp->rcv_nxt.
Add TCP_RFC7323_PAWS_ACK new drop reason, and do not generate a DUPACK for such old ACK.
v2:
- Also add matching LINUX_MIB_PAWS_OLD_ACK SNMP counter.
- fix a typo in one changelog.
Eric Dumazet (3):
tcp: add drop_reason support to tcp_disordered_ack()
tcp: add TCP_RFC7323_PAWS_ACK drop reason
tcp: add LINUX_MIB_PAWS_OLD_ACK SNMP counter
include/net/dropreason-core.h | 6 +++ include/uapi/linux/snmp.h | 1 + net/ipv4/proc.c | 1 + net/ipv4/tcp_input.c | 84 +++++++++++++++++++++-------------- 4 files changed, 59 insertions(+), 33 deletions(-)
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
PATCH v2 1/3 tcp: add drop_reason support to tcp_disordered_ack()
Following patch is adding a new drop_reason to tcp_validate_incoming().
Change tcp_disordered_ack() to not return a boolean anymore, but a drop reason.
Change its name to tcp_disordered_ack_check()
Refactor tcp_validate_incoming() to ease the code review of the following patch, and reduce indentation level.
This patch is a refactor, with no functional change.
--- net/ipv4/tcp_input.c | 79 ++++++++++++++++++++++++-------------------- 1 file changed, 44 insertions(+), 35 deletions(-)
net/ipv4/tcp_input.c +44 -35
@@ -4450,34 +4450,38 @@ static u32 tcp_tsval_replay(const struct sock *sk)return inet_csk (sk)->icsk_rto *1200 /HZ ; } -static int tcp_disordered_ack (const struct sock *sk,const struct sk_buff *skb) +static enum skb_drop_reason tcp_disordered_ack_check (const struct sock *sk, +const struct sk_buff *skb) {const struct tcp_sock *tp =tcp_sk (sk);const struct tcphdr *th =tcp_hdr (skb); -u32 seq =TCP_SKB_CB (skb)->seq; +SKB_DR_INIT (reason,TCP_RFC7323_PAWS );u32 ack =TCP_SKB_CB (skb)->ack_seq; +u32 seq =TCP_SKB_CB (skb)->seq; -return /* 1. Pure ACK with correct sequence number. */ - (th->ack && seq ==TCP_SKB_CB (skb)->end_seq && seq == tp->rcv_nxt) && +/* 1. Is this not a pure ACK ? */ +if (!th->ack || seq !=TCP_SKB_CB (skb)->end_seq) +return reason; -/* 2. ... and duplicate ACK. */ - ack == tp->snd_una && +/* 2. Is its sequence not the expected one ? */ +if (seq != tp->rcv_nxt) +return reason; -/* 3. ... and does not update window. */ - !tcp_may_update_window (tp, ack, seq,ntohs (th->window) << tp->rx_opt.snd_wscale) && +/* 3. Is this not a duplicate ACK ? */ +if (ack != tp->snd_una) +return reason; -/* 4. ... and sits in replay window. */ - (s32)(tp->rx_opt.ts_recent - tp->rx_opt.rcv_tsval) <= -tcp_tsval_replay (sk); -} +/* 4. Is this updating the window ? */ +if (tcp_may_update_window (tp, ack, seq,ntohs (th->window) << + tp->rx_opt.snd_wscale)) +return reason; -static inline bool tcp_paws_discard (const struct sock *sk, -const struct sk_buff *skb) -{ -const struct tcp_sock *tp =tcp_sk (sk); +/* 5. Is this not in the replay window ? */ +if ((s32)(tp->rx_opt.ts_recent - tp->rx_opt.rcv_tsval) > +tcp_tsval_replay (sk)) +return reason; -return !tcp_paws_check (&tp->rx_opt,TCP_PAWS_WINDOW ) && - !tcp_disordered_ack (sk, skb); +return 0 ; }/* Check segment sequence number for validity.
@@ -5949,23 +5953,28 @@ static bool tcp_validate_incoming(struct sock *sk, struct sk_buff *skb,SKB_DR (reason);/* RFC1323: H1. Apply PAWS check first. */ -if (tcp_fast_parse_options (sock_net (sk), skb, th, tp) && - tp->rx_opt.saw_tstamp && -tcp_paws_discard (sk, skb)) { -if (!th->rst) { -if (unlikely (th->syn)) - goto syn_challenge; -NET_INC_STATS (sock_net (sk),LINUX_MIB_PAWSESTABREJECTED ); -if (!tcp_oow_rate_limited (sock_net (sk), skb, -LINUX_MIB_TCPACKSKIPPEDPAWS , - &tp->last_oow_ack_time)) -tcp_send_dupack (sk, skb); -SKB_DR_SET (reason,TCP_RFC7323_PAWS ); - goto discard; - } -/* Reset is accepted even if it did not pass PAWS. */ - } - +if (!tcp_fast_parse_options (sock_net (sk), skb, th, tp) || + !tp->rx_opt.saw_tstamp || +tcp_paws_check (&tp->rx_opt,TCP_PAWS_WINDOW )) + goto step1; + + reason =tcp_disordered_ack_check (sk, skb); +if (!reason) + goto step1; +/* Reset is accepted even if it did not pass PAWS. */ +if (th->rst) + goto step1; +if (unlikely (th->syn)) + goto syn_challenge; + +NET_INC_STATS (sock_net (sk),LINUX_MIB_PAWSESTABREJECTED ); +if (!tcp_oow_rate_limited (sock_net (sk), skb, +LINUX_MIB_TCPACKSKIPPEDPAWS , + &tp->last_oow_ack_time)) +tcp_send_dupack (sk, skb); + goto discard; + +step1:/* Step 1: check sequence number */ reason =tcp_sequence (tp,TCP_SKB_CB (skb)->seq,TCP_SKB_CB (skb)->end_seq);if (reason) {
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
PATCH v2 2/3 tcp: add TCP_RFC7323_PAWS_ACK drop reason
In reply to Eric DumazetXPS can cause reorders because of the relaxed OOO conditions for pure ACK packets.
For hosts not using RFS, what can happpen is that ACK packets are sent on behalf of the cpu processing NIC interrupts, selecting TX queue A for ACK packet P1.
Then a subsequent sendmsg() can run on another cpu. TX queue selection uses the socket hash and can choose another queue B for packets P2 (with payload).
If queue A is more congested than queue B, the ACK packet P1 could be sent on the wire after P2.
A linux receiver when processing P1 (after P2) currently increments LINUX_MIB_PAWSESTABREJECTED (TcpExtPAWSEstab)
and use TCP_RFC7323_PAWS drop reason.
It might also send a DUPACK if not rate limited.
In order to better understand this pattern, this patch adds a new drop_reason : TCP_RFC7323_PAWS_ACK.
For old ACKS like these, we no longer increment LINUX_MIB_PAWSESTABREJECTED and no longer sends a DUPACK, keeping credit for other more interesting DUPACK.
perf record -e skb:kfree_skb -a
perf script
...
swapper 0 [148] 27475.438637: skb:kfree_skb: ... location=tcp_validate_incoming+0x4f0 reason: TCP_RFC7323_PAWS_ACK
swapper 0 [208] 27475.438706: skb:kfree_skb: ... location=tcp_validate_incoming+0x4f0 reason: TCP_RFC7323_PAWS_ACK
swapper 0 [208] 27475.438908: skb:kfree_skb: ... location=tcp_validate_incoming+0x4f0 reason: TCP_RFC7323_PAWS_ACK
swapper 0 [148] 27475.439010: skb:kfree_skb: ... location=tcp_validate_incoming+0x4f0 reason: TCP_RFC7323_PAWS_ACK
swapper 0 [148] 27475.439214: skb:kfree_skb: ... location=tcp_validate_incoming+0x4f0 reason: TCP_RFC7323_PAWS_ACK
swapper 0 [208] 27475.439286: skb:kfree_skb: ... location=tcp_validate_incoming+0x4f0 reason: TCP_RFC7323_PAWS_ACK...
--- include/net/dropreason-core.h | 5 +++++ net/ipv4/tcp_input.c | 10 +++++++++- 2 files changed, 14 insertions(+), 1 deletion(-)
include/net/dropreason-core.h +5 -0
@@ -36,6 +36,7 @@ FN(TCP_OVERWINDOW) \ FN(TCP_OFOMERGE) \ FN(TCP_RFC7323_PAWS) \ + FN(TCP_RFC7323_PAWS_ACK) \ FN(TCP_OLD_SEQUENCE) \ FN(TCP_INVALID_SEQUENCE) \ FN(TCP_INVALID_ACK_SEQUENCE) \
@@ -259,6 +260,10 @@ enum skb_drop_reason {* LINUX_MIB_PAWSESTABREJECTED, LINUX_MIB_PAWSACTIVEREJECTED */ SKB_DROP_REASON_TCP_RFC7323_PAWS , +/** +* @SKB_DROP_REASON_TCP_RFC7323_PAWS_ACK: PAWS check, old ACK packet. +*/ +SKB_DROP_REASON_TCP_RFC7323_PAWS_ACK ,/** @SKB_DROP_REASON_TCP_OLD_SEQUENCE: Old SEQ field (duplicate packet) */ SKB_DROP_REASON_TCP_OLD_SEQUENCE,/** @SKB_DROP_REASON_TCP_INVALID_SEQUENCE: Not acceptable SEQ field */
net/ipv4/tcp_input.c +9 -1
@@ -4465,7 +4465,9 @@ static enum skb_drop_reason tcp_disordered_ack_check(const struct sock *sk,/* 2. Is its sequence not the expected one ? */ if (seq != tp->rcv_nxt) -return reason; +return before (seq, tp->rcv_nxt) ? +SKB_DROP_REASON_TCP_RFC7323_PAWS_ACK : + reason;/* 3. Is this not a duplicate ACK ? */ if (ack != tp->snd_una)
@@ -5967,6 +5969,12 @@ static bool tcp_validate_incoming(struct sock *sk, struct sk_buff *skb,if (unlikely (th->syn)) goto syn_challenge; +/* Old ACK are common, do not change PAWSESTABREJECTED +* and do not send a dupack. +*/ +if (reason ==SKB_DROP_REASON_TCP_RFC7323_PAWS_ACK ) + goto discard; +NET_INC_STATS (sock_net (sk),LINUX_MIB_PAWSESTABREJECTED ); if (!tcp_oow_rate_limited(sock_net(sk), skb, LINUX_MIB_TCPACKSKIPPEDPAWS,
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Re: [PATCH v2 net-next 2/3] tcp: add TCP_RFC7323_PAWS_ACK drop reason
Quoted 37 lines · Eric Dumazet
XPS can cause reorders because of the relaxed OOO conditions for pure ACK packets.
For hosts not using RFS, what can happpen is that ACK packets are sent on behalf of the cpu processing NIC interrupts, selecting TX queue A for ACK packet P1.
Then a subsequent
sendmsg()can run on another cpu. TX queue selection uses the socket hash and can choose another queue B for packets P2 (with payload).If queue A is more congested than queue B, the ACK packet P1 could be sent on the wire after P2.
A linux receiver when processing P1 (after P2) currently increments
LINUX_MIB_PAWSESTABREJECTED(TcpExtPAWSEstab)
and useTCP_RFC7323_PAWSdrop reason.
It might also send a DUPACK if not rate limited.In order to better understand this pattern, this patch adds a new
drop_reason:TCP_RFC7323_PAWS_ACK.For old ACKS like these, we no longer increment
LINUX_MIB_PAWSESTABREJECTEDand no longer sends a DUPACK, keeping credit for other more interesting DUPACK.perf record -e skb:kfree_skb -a
perf script
...swapper 0 [148] 27475.438637: skb:kfree_skb: ... location=tcp_validate_incoming+0x4f0 reason: TCP_RFC7323_PAWS_ACK swapper 0 [208] 27475.438706: skb:kfree_skb: ... location=tcp_validate_incoming+0x4f0 reason: TCP_RFC7323_PAWS_ACK swapper 0 [208] 27475.438908: skb:kfree_skb: ... location=tcp_validate_incoming+0x4f0 reason: TCP_RFC7323_PAWS_ACK swapper 0 [148] 27475.439010: skb:kfree_skb: ... location=tcp_validate_incoming+0x4f0 reason: TCP_RFC7323_PAWS_ACK swapper 0 [148] 27475.439214: skb:kfree_skb: ... location=tcp_validate_incoming+0x4f0 reason: TCP_RFC7323_PAWS_ACK swapper 0 [208] 27475.439286: skb:kfree_skb: ... location=tcp_validate_incoming+0x4f0 reason: TCP_RFC7323_PAWS_ACK
Quoted 5 lines
...
Signed-off-by Eric Dumazet <[email protected]>Reviewed-by Neal Cardwell <[email protected]>Reviewed-by Kuniyuki Iwashima <[email protected]>
Thank you, Eric.
PATCH v2 3/3 tcp: add LINUX_MIB_PAWS_OLD_ACK SNMP counter
In reply to Eric DumazetPrior patch in the series added TCP_RFC7323_PAWS_ACK drop reason.
This patch adds the corresponding SNMP counter, for folks using nstat instead of tracing for TCP diagnostics.
nstat -az | grep PAWSOldAck
--- include/net/dropreason-core.h | 1 + include/uapi/linux/snmp.h | 1 + net/ipv4/proc.c | 1 + net/ipv4/tcp_input.c | 7 ++++--- 4 files changed, 7 insertions(+), 3 deletions(-)
include/net/dropreason-core.h +1 -0
@@ -262,6 +262,7 @@ enum skb_drop_reason {SKB_DROP_REASON_TCP_RFC7323_PAWS ,/** * @SKB_DROP_REASON_TCP_RFC7323_PAWS_ACK: PAWS check, old ACK packet. +* Corresponds to LINUX_MIB_PAWS_OLD_ACK. */ SKB_DROP_REASON_TCP_RFC7323_PAWS_ACK,/** @SKB_DROP_REASON_TCP_OLD_SEQUENCE: Old SEQ field (duplicate packet) */
include/uapi/linux/snmp.h +1 -0
@@ -186,6 +186,7 @@ enumLINUX_MIB_TIMEWAITKILLED ,/* TimeWaitKilled */ LINUX_MIB_PAWSACTIVEREJECTED ,/* PAWSActiveRejected */ LINUX_MIB_PAWSESTABREJECTED ,/* PAWSEstabRejected */ +LINUX_MIB_PAWS_OLD_ACK ,/* PAWSOldAck */ LINUX_MIB_DELAYEDACKS ,/* DelayedACKs */ LINUX_MIB_DELAYEDACKLOCKED ,/* DelayedACKLocked */ LINUX_MIB_DELAYEDACKLOST, /* DelayedACKLost */
net/ipv4/proc.c +1 -0
@@ -189,6 +189,7 @@ static const struct snmp_mib snmp4_net_list[] = {SNMP_MIB_ITEM ("TWKilled" ,LINUX_MIB_TIMEWAITKILLED ),SNMP_MIB_ITEM ("PAWSActive" ,LINUX_MIB_PAWSACTIVEREJECTED ),SNMP_MIB_ITEM ("PAWSEstab" ,LINUX_MIB_PAWSESTABREJECTED ), +SNMP_MIB_ITEM ("PAWSOldAck" ,LINUX_MIB_PAWS_OLD_ACK ),SNMP_MIB_ITEM ("DelayedACKs" ,LINUX_MIB_DELAYEDACKS ),SNMP_MIB_ITEM ("DelayedACKLocked" ,LINUX_MIB_DELAYEDACKLOCKED ), SNMP_MIB_ITEM("DelayedACKLost", LINUX_MIB_DELAYEDACKLOST),
net/ipv4/tcp_input.c +4 -3
@@ -5969,12 +5969,13 @@ static bool tcp_validate_incoming(struct sock *sk, struct sk_buff *skb,if (unlikely (th->syn)) goto syn_challenge; -/* Old ACK are common, do not change PAWSESTABREJECTED +/* Old ACK are common, increment PAWS_OLD_ACK * and do not send a dupack. */ -if (reason ==SKB_DROP_REASON_TCP_RFC7323_PAWS_ACK ) +if (reason ==SKB_DROP_REASON_TCP_RFC7323_PAWS_ACK ) { +NET_INC_STATS (sock_net (sk),LINUX_MIB_PAWS_OLD_ACK ); goto discard; - + }NET_INC_STATS (sock_net (sk),LINUX_MIB_PAWSESTABREJECTED ); if (!tcp_oow_rate_limited(sock_net(sk), skb, LINUX_MIB_TCPACKSKIPPEDPAWS,
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Re: [PATCH v2 net-next 3/3] tcp: add LINUX_MIB_PAWS_OLD_ACK SNMP counter
Quoted 8 lines · Eric Dumazet
Prior patch in the series added
TCP_RFC7323_PAWS_ACKdrop reason.This patch adds the corresponding SNMP counter, for folks using nstat instead of tracing for TCP diagnostics.
nstat -az | grep PAWSOldAck
Quoted 8 lines
Suggested-by Neal Cardwell <[email protected]>Signed-off-by Eric Dumazet <[email protected]>--- include/net/dropreason-core.h | 1 + include/uapi/linux/snmp.h | 1 + net/ipv4/proc.c | 1 + net/ipv4/tcp_input.c | 7 ++++--- 4 files changed, 7 insertions(+), 3 deletions(-)
Looks great to me. Thanks, Eric!
neal
Re: [PATCH v2 net-next 3/3] tcp: add LINUX_MIB_PAWS_OLD_ACK SNMP counter
Quoted 19 lines · Neal Cardwell
Quoted 8 lines · Eric Dumazet
Prior patch in the series added
TCP_RFC7323_PAWS_ACKdrop reason.This patch adds the corresponding SNMP counter, for folks using nstat instead of tracing for TCP diagnostics.
nstat -az | grep PAWSOldAck
Quoted 8 lines
Suggested-by Neal Cardwell <[email protected]>Signed-off-by Eric Dumazet <[email protected]>--- include/net/dropreason-core.h | 1 + include/uapi/linux/snmp.h | 1 + net/ipv4/proc.c | 1 + net/ipv4/tcp_input.c | 7 ++++--- 4 files changed, 7 insertions(+), 3 deletions(-)
Quoted 3 lines
Looks great to me. Thanks, Eric!
Reviewed-by Neal Cardwell <[email protected]>
Wrote a little packetdrill test for this, and it passes, as expected...
neal
ps: the packetdrill test, which I'm tentatively calling gtests/net/tcp/paws/paws-disordered-ack-old-seq-discard.pkt :
---
// Test PAWS processing for reordered pure ACK packets with old sequence // numbers. These are common due to the following common case:
// ACKs being generated on different CPUs than data segments, // causing ACKs to be transmitted on different tx queues than data segments // causing reordering of older ACKs behind newer data segments // if the ACKs end up in a longer queue than the data segments. // For these packets, we simply discard them, since this 2024 commit:
// tcp: add TCP_RFC7323_PAWS_ACK drop reason
// Check outgoing TCP timestamp values.
--tcp_ts_tick_usecs=1000
// Set up config.../common/defaults.sh
// Establish a connection.
0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 +0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < S 0:0(0) win 32792 <mss 1012,sackOK,TS val 2000 ecr 0,nop,wscale 7> +0 > S. 0:0(0) ack 1 <mss 1460,sackOK,TS val 1000 ecr 100,nop,wscale 8>
+.010 < . 1:1(0) ack 1 win 257 <nop, nop, TS val 2010 ecr 1000>
+0 accept(3, ..., ...) = 4
+0 %{ TCP_INFINITE_SSTHRESH = 0x7fffffff }%
// Send a request.
+0 write(4, ..., 1000) = 1000
+0 > P. 1:1001(1000) ack 1 <nop, nop, TS val 1010 ecr 2010>
+0 %{ assert tcpi_ca_state == TCP_CA_Open }%
+0 %{ assert tcpi_snd_cwnd == 10, tcpi_snd_cwnd }%
+0 %{ assert tcpi_snd_ssthresh == TCP_INFINITE_SSTHRESH, tcpi_snd_ssthresh }%
// The peer received our request and ACKed it immediately, and then // sent a data segment in reply. However, the ACK is reordered // behind the reply data segment.
// First we receive a response packet.
+.010 < . 1:1001(1000) ack 1001 win 257 <nop, nop, TS val 2022 ecr 1010>
+0 > . 1001:1001(0) ack 1001 <nop, nop, TS val 1020 ecr 2022>
// Reset nstat counters, in case we're not in a new network namespce:
+0 nstat -n
// Then we receive a disordered pure ACK packet with old sequence number.
+0 < . 1:1(0) ack 1001 win 257 <nop, nop, TS val 2020 ecr 1010>
// Because it fails PAWS but is an expected kind of pure ACK with // an old sequence number, we don't send a dupack.
// But verify that we increment TcpExtPAWSOldAck.
+0 nstat | grep TcpExtPAWSOldAck | grep -q " 1 "
// Wait a while to verify we don't send a dupack for that disoordered ACK.
// Expect another transmit so that packetdrill will sniff that // and see any erroneous dupack instead if we sent that. +.300 write(4, ..., 1000) = 1000
+0 > P. 1001:2001(1000) ack 1001 <nop, nop, TS val 1320 ecr 2021>
Re: [PATCH v2 net-next 3/3] tcp: add LINUX_MIB_PAWS_OLD_ACK SNMP counter
In reply to Eric DumazetQuoted 10 lines · Eric Dumazet
Prior patch in the series added
TCP_RFC7323_PAWS_ACKdrop reason.This patch adds the corresponding SNMP counter, for folks using nstat instead of tracing for TCP diagnostics.
nstat -az | grep PAWSOldAck
Suggested-by Neal Cardwell <[email protected]>Signed-off-by Eric Dumazet <[email protected]>
Really good suggestion so that we'll not miss this one :)
Re: [PATCH v2 net-next 3/3] tcp: add LINUX_MIB_PAWS_OLD_ACK SNMP counter
In reply to Eric DumazetFrom: Eric Dumazet <[email protected]>
Date: Mon, 13 Jan 2025 13:55:58 +0000
Quoted 9 lines
Prior patch in the series added
TCP_RFC7323_PAWS_ACKdrop reason.This patch adds the corresponding SNMP counter, for folks using nstat instead of tracing for TCP diagnostics.
nstat -az | grep PAWSOldAck
Suggested-by Neal Cardwell <[email protected]>Signed-off-by Eric Dumazet <[email protected]>
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 8 lines · On Mon, 13 Jan 2025 13:55:55 +0000 you wrote:
Current
TCP_RFC7323_PAWSdrop reason is too generic and can cause confusion.One common source for these drops are ACK packets coming too late.
A prior packet with payload already changed
tp->rcv_nxt.[...]
Here is the summary with links:
- [v2,net-next,1/3] tcp: add drop_reason support to tcp_disordered_ack()
https://git.kernel.org/netdev/net-next/c/ea98b61bddf4 - [v2,net-next,2/3] tcp: add TCP_RFC7323_PAWS_ACK drop reason https://git.kernel.org/netdev/net-next/c/124c4c32e9f3 - [v2,net-next,3/3] tcp: add LINUX_MIB_PAWS_OLD_ACK SNMP counter https://git.kernel.org/netdev/net-next/c/d16b34479064
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html