tcp: a couple of fixes
Subsystems (4)
Paolo Abeni
maintainercover tcp: a couple of fixes
This series includes a couple of follow-up for the recent tcp receiver changes, addressing issues outlined by the nipa CI and the mptcp self-tests.
Note that despite the affected self-tests where MPTCP ones, the issues are really in the TCP code, see patch 1 for the details.
Paolo Abeni (2):
tcp: do not set a zero size receive buffer
tcp: do not increment BeyondWindow MIB for old seq
net/ipv4/tcp_input.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-)
Signature · 1 line
2.50.0
Paolo Abeni
maintainerPATCH 1/2 tcp: do not set a zero size receive buffer
The nipa CI is reporting frequent failures in the mptcp_connect self-tests.
In the failing scenarios (TCP -> MPTCP) the involved sockets are actually plain TCP ones, as fallback for passive socket at 2whs time cause the MPTCP listener to actually create a TCP socket.
The transfer is stuck due to the receiver buffer being zero. With the stronger check in place, tcp_clamp_window() can be invoked while the TCP socket has sk_rmem_alloc == 0, and the receive buffer will be zeroed, too.
Pass to tcp_clamp_window() even the current skb truesize, so that such helper could compute and use the actual limit enforced by the stack.
--- net/ipv4/tcp_input.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
net/ipv4/tcp_input.c +6 -6
@@ -610,24 +610,24 @@ static void tcp_init_buffer_space(struct sock *sk) }/* 4. Recalculate window clamp after socket hit its memory bounds. */ -static void tcp_clamp_window (struct sock *sk) +static void tcp_clamp_window (struct sock *sk,int truesize) {struct tcp_sock *tp =tcp_sk (sk);struct inet_connection_sock *icsk =inet_csk (sk);struct net *net =sock_net (sk); -int rmem2; +int rmem2, needed; icsk->icsk_ack.quick =0 ; rmem2 =READ_ONCE (net->ipv4.sysctl_tcp_rmem[2 ]); + needed =atomic_read (&sk->sk_rmem_alloc) + truesize;if (sk->sk_rcvbuf < rmem2 && !(sk->sk_userlocks &SOCK_RCVBUF_LOCK ) && !tcp_under_memory_pressure (sk) &&sk_memory_allocated (sk) <sk_prot_mem_limits (sk,0 )) { -WRITE_ONCE (sk->sk_rcvbuf, -min (atomic_read (&sk->sk_rmem_alloc), rmem2)); +WRITE_ONCE (sk->sk_rcvbuf,min (needed, rmem2)); } -if (atomic_read (&sk->sk_rmem_alloc) > sk->sk_rcvbuf) +if (needed > sk->sk_rcvbuf) tp->rcv_ssthresh =min (tp->window_clamp,2U * tp->advmss); }
@@ -5552,7 +5552,7 @@ static int tcp_prune_queue(struct sock *sk, const struct sk_buff *in_skb)NET_INC_STATS (sock_net (sk),LINUX_MIB_PRUNECALLED );if (!tcp_can_ingest (sk, in_skb)) -tcp_clamp_window (sk); +tcp_clamp_window (sk, in_skb->truesize);else if (tcp_under_memory_pressure (sk))tcp_adjust_rcv_ssthresh (sk);
Signature · 1 line
2.50.0
Re: [PATCH net-next 1/2] tcp: do not set a zero size receive buffer
Hi Paolo,
Quoted 7 lines · Paolo Abeni
The nipa CI is reporting frequent failures in the
mptcp_connectself-tests.In the failing scenarios (TCP -> MPTCP) the involved sockets are actually plain TCP ones, as fallback for passive socket at 2whs time cause the MPTCP listener to actually create a TCP socket.
Quoted 8 lines
The transfer is stuck due to the receiver buffer being zero. With the stronger check in place,
tcp_clamp_window()can be invoked while the TCP socket hassk_rmem_alloc== 0, and the receive buffer will be zeroed, too.Pass to
tcp_clamp_window()even the current skb truesize, so that such helper could compute and use the actual limit enforced by the stack.
Thank you for this fix!
I guess we can drop "mptcp-connect-sh" from the ignored tests then :)
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.
Re: [PATCH net-next 1/2] tcp: do not set a zero size receive buffer
In reply to Paolo AbeniQuoted 28 lines · Paolo Abeni
The nipa CI is reporting frequent failures in the
mptcp_connectself-tests.In the failing scenarios (TCP -> MPTCP) the involved sockets are actually plain TCP ones, as fallback for passive socket at 2whs time cause the MPTCP listener to actually create a TCP socket.
The transfer is stuck due to the receiver buffer being zero. With the stronger check in place,
tcp_clamp_window()can be invoked while the TCP socket hassk_rmem_alloc== 0, and the receive buffer will be zeroed, too.Pass to
tcp_clamp_window()even the current skb truesize, so that such helper could compute and use the actual limit enforced by the stack.Fixes 1d2fbaad7cd8 ("tcp: stronger sk_rcvbuf checks")Signed-off-by Paolo Abeni <[email protected]>--- net/ipv4/tcp_input.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)net/ipv4/tcp_input.c +0
-0
net/ipv4/tcp_input.c static void tcp_init_buffer_space(struct sock *sk) · line 613
Quoted 5 lines
net/ipv4/tcp_input.c +1
-1@@ -610,24 +610,24 @@ static void tcp_init_buffer_space(struct sock *sk) }/* 4. Recalculate window clamp after socket hit its memory bounds. */ -static void tcp_clamp_window (struct sock *sk) +static void tcp_clamp_window (struct sock *sk,int truesize)
I am unsure about this one. truesize can be 1MB here, do we want that in general ?
I am unsure why MPTCP ends up with this path.
LINUX_MIB_PRUNECALLED being called in normal MPTCP operations seems
strange to me.
Quoted 36 lines
net/ipv4/tcp_input.c +5
-6{struct tcp_sock *tp =tcp_sk (sk);struct inet_connection_sock *icsk =inet_csk (sk);struct net *net =sock_net (sk); -int rmem2; +int rmem2, needed; icsk->icsk_ack.quick =0 ; rmem2 =READ_ONCE (net->ipv4.sysctl_tcp_rmem[2 ]); + needed =atomic_read (&sk->sk_rmem_alloc) + truesize;if (sk->sk_rcvbuf < rmem2 && !(sk->sk_userlocks &SOCK_RCVBUF_LOCK ) && !tcp_under_memory_pressure (sk) &&sk_memory_allocated (sk) <sk_prot_mem_limits (sk,0 )) { -WRITE_ONCE (sk->sk_rcvbuf, -min (atomic_read (&sk->sk_rmem_alloc), rmem2)); +WRITE_ONCE (sk->sk_rcvbuf,min (needed, rmem2)); } -if (atomic_read (&sk->sk_rmem_alloc) > sk->sk_rcvbuf) +if (needed > sk->sk_rcvbuf) tp->rcv_ssthresh =min (tp->window_clamp,2U * tp->advmss); }@@ -5552,7 +5552,7 @@ static int tcp_prune_queue(struct sock *sk, const struct sk_buff *in_skb)NET_INC_STATS (sock_net (sk),LINUX_MIB_PRUNECALLED );if (!tcp_can_ingest (sk, in_skb)) -tcp_clamp_window (sk); +tcp_clamp_window (sk, in_skb->truesize);else if (tcp_under_memory_pressure (sk))tcp_adjust_rcv_ssthresh (sk); --2.50.0
Paolo Abeni
maintainerRe: [PATCH net-next 1/2] tcp: do not set a zero size receive buffer
Quoted 31 lines · Eric Dumazet
Quoted 30 lines · Paolo Abeni
The nipa CI is reporting frequent failures in the
mptcp_connectself-tests.In the failing scenarios (TCP -> MPTCP) the involved sockets are actually plain TCP ones, as fallback for passive socket at 2whs time cause the MPTCP listener to actually create a TCP socket.
The transfer is stuck due to the receiver buffer being zero. With the stronger check in place,
tcp_clamp_window()can be invoked while the TCP socket hassk_rmem_alloc== 0, and the receive buffer will be zeroed, too.Pass to
tcp_clamp_window()even the current skb truesize, so that such helper could compute and use the actual limit enforced by the stack.Fixes 1d2fbaad7cd8 ("tcp: stronger sk_rcvbuf checks")Signed-off-by Paolo Abeni <[email protected]>--- net/ipv4/tcp_input.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)net/ipv4/tcp_input.c +0
-0@@ -610,24 +610,24 @@ static void tcp_init_buffer_space(struct sock *sk) }
Quoted 7 lines
net/ipv4/tcp_input.c static void tcp_init_buffer_space(struct sock *sk) · line 613
Quoted 3 lines
net/ipv4/tcp_input.c +1
-1/* 4. Recalculate window clamp after socket hit its memory bounds. */ -static void tcp_clamp_window (struct sock *sk) +static void tcp_clamp_window (struct sock *sk,int truesize)I am unsure about this one. truesize can be 1MB here, do we want that in general ?
I'm unsure either. But I can't think of a different approach?!? If the incoming truesize is 1M the socket should allow for at least 1M rcvbuf size to accept it, right?
Quoted 4 lines
I am unsure why MPTCP ends up with this path.
LINUX_MIB_PRUNECALLEDbeing called in normal MPTCP operations seems
strange to me.
The code path hit:
!tcp_can_ingest(sk, skb)
in tcp_try_rmem_schedule(). Perhaps the scaling_ratio is a bit optimistic due to unfortunate packet layouts?
Let me check if I can grab more data.
Thanks,
Paolo
Re: [PATCH net-next 1/2] tcp: do not set a zero size receive buffer
Quoted 41 lines · Paolo Abeni
Quoted 31 lines · Eric Dumazet
Quoted 30 lines · Paolo Abeni
The nipa CI is reporting frequent failures in the
mptcp_connectself-tests.In the failing scenarios (TCP -> MPTCP) the involved sockets are actually plain TCP ones, as fallback for passive socket at 2whs time cause the MPTCP listener to actually create a TCP socket.
The transfer is stuck due to the receiver buffer being zero. With the stronger check in place,
tcp_clamp_window()can be invoked while the TCP socket hassk_rmem_alloc== 0, and the receive buffer will be zeroed, too.Pass to
tcp_clamp_window()even the current skb truesize, so that such helper could compute and use the actual limit enforced by the stack.Fixes 1d2fbaad7cd8 ("tcp: stronger sk_rcvbuf checks")Signed-off-by Paolo Abeni <[email protected]>--- net/ipv4/tcp_input.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)net/ipv4/tcp_input.c +0
-0@@ -610,24 +610,24 @@ static void tcp_init_buffer_space(struct sock *sk) }Quoted 7 lines
net/ipv4/tcp_input.c static void tcp_init_buffer_space(struct sock *sk) · line 613
Quoted 3 lines
net/ipv4/tcp_input.c +1
-1/* 4. Recalculate window clamp after socket hit its memory bounds. */ -static void tcp_clamp_window (struct sock *sk) +static void tcp_clamp_window (struct sock *sk,int truesize)I am unsure about this one. truesize can be 1MB here, do we want that in general ?
Quoted 3 lines
I'm unsure either. But I can't think of a different approach?!? If the incoming truesize is 1M the socket should allow for at least 1M rcvbuf size to accept it, right?
What I meant was :
This is the generic point, accepting skb->truesize as additional input here would make us more vulnerable, or we could risk other regressions.
The question is : why does MPTCP end up here in the first place. Perhaps an older issue with an incorrectly sized sk_rcvbuf ?
Or maybe the test about the receive queue being empty (currently done in tcp_data_queue()) should be moved to a more strategic place.
Re: [PATCH net-next 1/2] tcp: do not set a zero size receive buffer
Quoted 53 lines · Eric Dumazet
Quoted 41 lines · Paolo Abeni
Quoted 31 lines · Eric Dumazet
Quoted 30 lines · Paolo Abeni
The nipa CI is reporting frequent failures in the
mptcp_connectself-tests.In the failing scenarios (TCP -> MPTCP) the involved sockets are actually plain TCP ones, as fallback for passive socket at 2whs time cause the MPTCP listener to actually create a TCP socket.
The transfer is stuck due to the receiver buffer being zero. With the stronger check in place,
tcp_clamp_window()can be invoked while the TCP socket hassk_rmem_alloc== 0, and the receive buffer will be zeroed, too.Pass to
tcp_clamp_window()even the current skb truesize, so that such helper could compute and use the actual limit enforced by the stack.Fixes 1d2fbaad7cd8 ("tcp: stronger sk_rcvbuf checks")Signed-off-by Paolo Abeni <[email protected]>--- net/ipv4/tcp_input.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)net/ipv4/tcp_input.c +0
-0@@ -610,24 +610,24 @@ static void tcp_init_buffer_space(struct sock *sk) }Quoted 7 lines
net/ipv4/tcp_input.c static void tcp_init_buffer_space(struct sock *sk) · line 613
Quoted 3 lines
net/ipv4/tcp_input.c +1
-1/* 4. Recalculate window clamp after socket hit its memory bounds. */ -static void tcp_clamp_window (struct sock *sk) +static void tcp_clamp_window (struct sock *sk,int truesize)I am unsure about this one. truesize can be 1MB here, do we want that in general ?
Quoted 3 lines
I'm unsure either. But I can't think of a different approach?!? If the incoming truesize is 1M the socket should allow for at least 1M rcvbuf size to accept it, right?
What I meant was :
This is the generic point, accepting
skb->truesizeas additional input here would make us more vulnerable, or we could risk other regressions.
Quoted 5 lines
The question is : why does MPTCP end up here in the first place. Perhaps an older issue with an incorrectly sized
sk_rcvbuf?Or maybe the test about the receive queue being empty (currently done in
tcp_data_queue()) should be moved to a more strategic place.
If the MPTCP part can not be easily resolved, perhaps :
net/ipv4/tcp_input.c +0 -0
100644
net/ipv4/tcp_input.c +0 -0
@@ -5549,6 +5549,10 @@ static int tcp_prune_queue(struct sock *sk,
const struct sk_buff *in_skb)
{
struct tcp_sock *tp = tcp_sk(sk);+ /* Do nothing if our queues are empty. */
+ if (!atomic_read(&sk->sk_rmem_alloc))
+ return -1;
+
NET_INC_STATS(sock_net(sk), LINUX_MIB_PRUNECALLED);
if (!tcp_can_ingest(sk, in_skb))Paolo Abeni
maintainerRe: [PATCH net-next 1/2] tcp: do not set a zero size receive buffer
In reply to Eric DumazetQuoted 45 lines · Eric Dumazet
Quoted 40 lines · Paolo Abeni
Quoted 31 lines · Eric Dumazet
Quoted 30 lines · Paolo Abeni
The nipa CI is reporting frequent failures in the
mptcp_connectself-tests.In the failing scenarios (TCP -> MPTCP) the involved sockets are actually plain TCP ones, as fallback for passive socket at 2whs time cause the MPTCP listener to actually create a TCP socket.
The transfer is stuck due to the receiver buffer being zero. With the stronger check in place,
tcp_clamp_window()can be invoked while the TCP socket hassk_rmem_alloc== 0, and the receive buffer will be zeroed, too.Pass to
tcp_clamp_window()even the current skb truesize, so that such helper could compute and use the actual limit enforced by the stack.Fixes 1d2fbaad7cd8 ("tcp: stronger sk_rcvbuf checks")Signed-off-by Paolo Abeni <[email protected]>--- net/ipv4/tcp_input.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)net/ipv4/tcp_input.c +0
-0@@ -610,24 +610,24 @@ static void tcp_init_buffer_space(struct sock *sk) }Quoted 7 lines
net/ipv4/tcp_input.c static void tcp_init_buffer_space(struct sock *sk) · line 613
Quoted 3 lines
net/ipv4/tcp_input.c +1
-1/* 4. Recalculate window clamp after socket hit its memory bounds. */ -static void tcp_clamp_window (struct sock *sk) +static void tcp_clamp_window (struct sock *sk,int truesize)I am unsure about this one. truesize can be 1MB here, do we want that in general ?
Quoted 3 lines
I'm unsure either. But I can't think of a different approach?!? If the incoming truesize is 1M the socket should allow for at least 1M rcvbuf size to accept it, right?
Quoted 5 lines
What I meant was :
This is the generic point, accepting
skb->truesizeas additional input here would make us more vulnerable, or we could risk other regressions.
Understood, thanks for the clarification.
Quoted 2 lines
The question is : why does MPTCP end up here in the first place. Perhaps an older issue with an incorrectly sized
sk_rcvbuf?
I collected a few more data. The issue happens even with plain TCP sockets[1].
The relevant transfer is on top of the loopback device. The scaling_rate rapidly grows to 254 - that is truesize and len are very near.
The stall happens when the received get in a packet with a slightly less 'efficient' layout (in the experiment I have handy len is 71424, truesize 72320) (almost) filling the receiver window.
On such input, tcp_clamp_window() shrinks the receiver buffer to the current rmem usage. The same happens on retransmissions until rcvbuf becomes 0.
I *think* that catching only the !sk_rmem_alloc case would avoid the stall, but I think it's a bit 'late'. I'm unsure if we could preventing/forbidding 'too high' values of scaling_rate? (also I'm unsure where to draw the line exactly.
Cheers,
Paolo
[1] You can run the relevant test by adding '-t' on the mptcp_connect.sh command line, but it will take a lot of time to run the 10-20 iterations I need to observe the issue. To make it faster I manually trimmed the not relevant test-cases.
Re: [PATCH net-next 1/2] tcp: do not set a zero size receive buffer
Quoted 68 lines · Paolo Abeni
Quoted 45 lines · Eric Dumazet
Quoted 40 lines · Paolo Abeni
Quoted 31 lines · Eric Dumazet
Quoted 30 lines · Paolo Abeni
The nipa CI is reporting frequent failures in the
mptcp_connectself-tests.In the failing scenarios (TCP -> MPTCP) the involved sockets are actually plain TCP ones, as fallback for passive socket at 2whs time cause the MPTCP listener to actually create a TCP socket.
The transfer is stuck due to the receiver buffer being zero. With the stronger check in place,
tcp_clamp_window()can be invoked while the TCP socket hassk_rmem_alloc== 0, and the receive buffer will be zeroed, too.Pass to
tcp_clamp_window()even the current skb truesize, so that such helper could compute and use the actual limit enforced by the stack.Fixes 1d2fbaad7cd8 ("tcp: stronger sk_rcvbuf checks")Signed-off-by Paolo Abeni <[email protected]>--- net/ipv4/tcp_input.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)net/ipv4/tcp_input.c +0
-0@@ -610,24 +610,24 @@ static void tcp_init_buffer_space(struct sock *sk) }Quoted 7 lines
net/ipv4/tcp_input.c static void tcp_init_buffer_space(struct sock *sk) · line 613
Quoted 3 lines
net/ipv4/tcp_input.c +1
-1/* 4. Recalculate window clamp after socket hit its memory bounds. */ -static void tcp_clamp_window (struct sock *sk) +static void tcp_clamp_window (struct sock *sk,int truesize)I am unsure about this one. truesize can be 1MB here, do we want that in general ?
Quoted 3 lines
I'm unsure either. But I can't think of a different approach?!? If the incoming truesize is 1M the socket should allow for at least 1M rcvbuf size to accept it, right?
Quoted 5 lines
What I meant was :
This is the generic point, accepting
skb->truesizeas additional input here would make us more vulnerable, or we could risk other regressions.Understood, thanks for the clarification.
Quoted 2 lines
The question is : why does MPTCP end up here in the first place. Perhaps an older issue with an incorrectly sized
sk_rcvbuf?I collected a few more data. The issue happens even with plain TCP sockets[1].
The relevant transfer is on top of the loopback device. The
scaling_raterapidly grows to 254 - that istruesizeandlenare very near.The stall happens when the received get in a packet with a slightly less 'efficient' layout (in the experiment I have handy len is 71424, truesize 72320) (almost) filling the receiver window.
Quoted 6 lines
On such input,
tcp_clamp_window()shrinks the receiver buffer to the current rmem usage. The same happens on retransmissions until rcvbuf becomes 0.I *think* that catching only the !sk_rmem_alloc case would avoid the stall, but I think it's a bit 'late'.
A packetdrill test here would help understanding your concern.
Quoted 3 lines
I'm unsure if we could
preventing/forbidding 'too high' values ofscaling_rate? (also I'm unsure where to draw the line exactly.
Indeed we need to account for a possible variation (ie reduction) of skb->len/skb->truesize ratio in future packets.
Note that whatever conservative change we make, it will always be possible to feed packets until RWIN 0 is sent back, eventually after a bump caused by a prune operation.
Imagine wscale is 8, and a prior RWIN 1 was sent.
Normally we should be able to receive a packet with 256 bytes of payload, but typical skb->truesize
will be 256+512+4096 (for a NIC driver using 4K pages).
This is one of the reasons we force wscale to 12 in Google DC :)
Quoted 11 lines
Cheers,
Paolo
[1] You can run the relevant test by adding '-t' on the
mptcp_connect.shcommand line, but it will take a lot of time to run the 10-20 iterations I need to observe the issue. To make it faster I manually trimmed the not relevant test-cases.
Paolo Abeni
maintainerRe: [PATCH net-next 1/2] tcp: do not set a zero size receive buffer
Quoted 68 lines · Eric Dumazet
Quoted 67 lines · Paolo Abeni
Quoted 45 lines · Eric Dumazet
Quoted 40 lines · Paolo Abeni
Quoted 31 lines · Eric Dumazet
Quoted 30 lines · Paolo Abeni
The nipa CI is reporting frequent failures in the
mptcp_connectself-tests.In the failing scenarios (TCP -> MPTCP) the involved sockets are actually plain TCP ones, as fallback for passive socket at 2whs time cause the MPTCP listener to actually create a TCP socket.
The transfer is stuck due to the receiver buffer being zero. With the stronger check in place,
tcp_clamp_window()can be invoked while the TCP socket hassk_rmem_alloc== 0, and the receive buffer will be zeroed, too.Pass to
tcp_clamp_window()even the current skb truesize, so that such helper could compute and use the actual limit enforced by the stack.Fixes 1d2fbaad7cd8 ("tcp: stronger sk_rcvbuf checks")Signed-off-by Paolo Abeni <[email protected]>--- net/ipv4/tcp_input.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)net/ipv4/tcp_input.c +0
-0@@ -610,24 +610,24 @@ static void tcp_init_buffer_space(struct sock *sk) }Quoted 7 lines
net/ipv4/tcp_input.c static void tcp_init_buffer_space(struct sock *sk) · line 613
Quoted 3 lines
net/ipv4/tcp_input.c +1
-1/* 4. Recalculate window clamp after socket hit its memory bounds. */ -static void tcp_clamp_window (struct sock *sk) +static void tcp_clamp_window (struct sock *sk,int truesize)I am unsure about this one. truesize can be 1MB here, do we want that in general ?
Quoted 3 lines
I'm unsure either. But I can't think of a different approach?!? If the incoming truesize is 1M the socket should allow for at least 1M rcvbuf size to accept it, right?
Quoted 5 lines
What I meant was :
This is the generic point, accepting
skb->truesizeas additional input here would make us more vulnerable, or we could risk other regressions.Understood, thanks for the clarification.
Quoted 2 lines
The question is : why does MPTCP end up here in the first place. Perhaps an older issue with an incorrectly sized
sk_rcvbuf?I collected a few more data. The issue happens even with plain TCP sockets[1].
The relevant transfer is on top of the loopback device. The
scaling_raterapidly grows to 254 - that istruesizeandlenare very near.The stall happens when the received get in a packet with a slightly less 'efficient' layout (in the experiment I have handy len is 71424, truesize 72320) (almost) filling the receiver window.
Quoted 8 lines
Quoted 6 lines
On such input,
tcp_clamp_window()shrinks the receiver buffer to the current rmem usage. The same happens on retransmissions until rcvbuf becomes 0.I *think* that catching only the !sk_rmem_alloc case would avoid the stall, but I think it's a bit 'late'.
A packetdrill test here would help understanding your concern.
I fear like a complete working script would take a lot of time, let me try to sketch just the relevant part:
# receiver state is: # rmem=110592 rcvbuf=174650 scaling_ratio=253 rwin=63232 # no OoO data, no memory pressure,
# the incoming packet is in sequence +0 > P. 109297:172528(63232) ack 1
With just the 0 rmem check in tcp_prune_queue(), such function will still invoke tcp_clamp_window() that will shrink the receive buffer to 110592.tcp_collapse() can't make enough room and the incoming packet will be dropped. I think we should instead accept such packet.
Side note: the above data are taken from an actual reproduction of the issue
Please LMK if the above clarifies a bit my doubt or if a full pktdrill is needed.
Thanks,
Paolo
Re: [PATCH net-next 1/2] tcp: do not set a zero size receive buffer
Quoted 86 lines · Paolo Abeni
Quoted 68 lines · Eric Dumazet
Quoted 67 lines · Paolo Abeni
Quoted 45 lines · Eric Dumazet
Quoted 40 lines · Paolo Abeni
Quoted 31 lines · Eric Dumazet
Quoted 30 lines · Paolo Abeni
The nipa CI is reporting frequent failures in the
mptcp_connectself-tests.In the failing scenarios (TCP -> MPTCP) the involved sockets are actually plain TCP ones, as fallback for passive socket at 2whs time cause the MPTCP listener to actually create a TCP socket.
The transfer is stuck due to the receiver buffer being zero. With the stronger check in place,
tcp_clamp_window()can be invoked while the TCP socket hassk_rmem_alloc== 0, and the receive buffer will be zeroed, too.Pass to
tcp_clamp_window()even the current skb truesize, so that such helper could compute and use the actual limit enforced by the stack.Fixes 1d2fbaad7cd8 ("tcp: stronger sk_rcvbuf checks")Signed-off-by Paolo Abeni <[email protected]>--- net/ipv4/tcp_input.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)net/ipv4/tcp_input.c +0
-0@@ -610,24 +610,24 @@ static void tcp_init_buffer_space(struct sock *sk) }Quoted 7 lines
net/ipv4/tcp_input.c static void tcp_init_buffer_space(struct sock *sk) · line 613
Quoted 3 lines
net/ipv4/tcp_input.c +1
-1/* 4. Recalculate window clamp after socket hit its memory bounds. */ -static void tcp_clamp_window (struct sock *sk) +static void tcp_clamp_window (struct sock *sk,int truesize)I am unsure about this one. truesize can be 1MB here, do we want that in general ?
Quoted 3 lines
I'm unsure either. But I can't think of a different approach?!? If the incoming truesize is 1M the socket should allow for at least 1M rcvbuf size to accept it, right?
Quoted 5 lines
What I meant was :
This is the generic point, accepting
skb->truesizeas additional input here would make us more vulnerable, or we could risk other regressions.Understood, thanks for the clarification.
Quoted 2 lines
The question is : why does MPTCP end up here in the first place. Perhaps an older issue with an incorrectly sized
sk_rcvbuf?I collected a few more data. The issue happens even with plain TCP sockets[1].
The relevant transfer is on top of the loopback device. The
scaling_raterapidly grows to 254 - that istruesizeandlenare very near.The stall happens when the received get in a packet with a slightly less 'efficient' layout (in the experiment I have handy len is 71424, truesize 72320) (almost) filling the receiver window.
Quoted 8 lines
Quoted 6 lines
On such input,
tcp_clamp_window()shrinks the receiver buffer to the current rmem usage. The same happens on retransmissions until rcvbuf becomes 0.I *think* that catching only the !sk_rmem_alloc case would avoid the stall, but I think it's a bit 'late'.
A packetdrill test here would help understanding your concern.
I fear like a complete working script would take a lot of time, let me try to sketch just the relevant part:
# receiver state is: # rmem=110592 rcvbuf=174650 scaling_ratio=253 rwin=63232 # no OoO data, no memory pressure,
Quoted 6 lines
# the incoming packet is in sequence +0 > P. 109297:172528(63232) ack 1With just the 0 rmem check in
tcp_prune_queue(), such function will still invoketcp_clamp_window()that will shrink the receive buffer to 110592.
As long as an ACK is sent back with a smaller RWIN, I think this would be reasonable in this case.
Quoted 2 lines
tcp_collapse()can't make enough room and the incoming packet will be dropped. I think we should instead accept such packet.
Only if not completely off-the-limits...
packetdrill test :
cat eric.pkt
// Test the calculation of receive window values by a bulk data receiver.
--mss=1000
// Set up config.
// Need to set tcp_rmem[1] to what tcp_fixup_rcvbuf() would have set to // make sure the same kernel behavior after removing tcp_fixup_rcvbuf() `../common/defaults.sh../common/set_sysctls.py /proc/sys/net/ipv4/tcp_rmem="4096 131072 15728640" `
// Create a socket.
0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
// Verify that the receive buffer is the tcp_rmem default.
+0 getsockopt(3, SOL_SOCKET, SO_RCVBUF, [131072], [4]) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
// Establish a connection.
+.01 < S 0:0(0) win 65535 <mss 1000,nop,nop,sackOK,nop,wscale 6>
+0 > S. 0:0(0) ack 1 win 64240 <mss 1460,nop,nop,sackOK,nop,wscale 8>
+.01 < . 1:1(0) ack 1 win 457
+0 accept(3, ..., ...) = 4
// Verify that the receive buffer is the tcp_rmem default.
+0 getsockopt(3, SOL_SOCKET, SO_RCVBUF, [131072], [4]) = 0
// Check first outgoing window after SYN
+.01 write(4, ..., 1000) = 1000
+0 > P. 1:1001(1000) ack 1 win 251
// Phase 1: Data arrives but app doesn't read from the socket buffer.
+.01 < . 1:60001(60000) ack 1001 win 457
+0 > . 1001:1001(0) ack 60001 win 263
+.01 < . 60001:120001(60000) ack 1001 win 457
+0~+.04 > . 1001:1001(0) ack 120001 win 29
// Incoming packet has a too big skb->truesize, lets send a lower RWIN
+.01 < . 120001:127425(7424) ack 1001 win 457
+0~+.04 > . 1001:1001(0) ack 120001 win 0
// Reset to sysctls defaults/tmp/sysctl_restore_${PPID}.sh
Paolo Abeni
maintainerRe: [PATCH net-next 1/2] tcp: do not set a zero size receive buffer
Quoted 89 lines · Eric Dumazet
Quoted 88 lines · Paolo Abeni
Quoted 68 lines · Eric Dumazet
Quoted 67 lines · Paolo Abeni
Quoted 45 lines · Eric Dumazet
Quoted 40 lines · Paolo Abeni
Quoted 31 lines · Eric Dumazet
Quoted 30 lines · Paolo Abeni
The nipa CI is reporting frequent failures in the
mptcp_connectself-tests.In the failing scenarios (TCP -> MPTCP) the involved sockets are actually plain TCP ones, as fallback for passive socket at 2whs time cause the MPTCP listener to actually create a TCP socket.
The transfer is stuck due to the receiver buffer being zero. With the stronger check in place,
tcp_clamp_window()can be invoked while the TCP socket hassk_rmem_alloc== 0, and the receive buffer will be zeroed, too.Pass to
tcp_clamp_window()even the current skb truesize, so that such helper could compute and use the actual limit enforced by the stack.Fixes 1d2fbaad7cd8 ("tcp: stronger sk_rcvbuf checks")Signed-off-by Paolo Abeni <[email protected]>--- net/ipv4/tcp_input.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)net/ipv4/tcp_input.c +0
-0@@ -610,24 +610,24 @@ static void tcp_init_buffer_space(struct sock *sk) }Quoted 7 lines
net/ipv4/tcp_input.c static void tcp_init_buffer_space(struct sock *sk) · line 613
Quoted 3 lines
net/ipv4/tcp_input.c +1
-1/* 4. Recalculate window clamp after socket hit its memory bounds. */ -static void tcp_clamp_window (struct sock *sk) +static void tcp_clamp_window (struct sock *sk,int truesize)I am unsure about this one. truesize can be 1MB here, do we want that in general ?
Quoted 3 lines
I'm unsure either. But I can't think of a different approach?!? If the incoming truesize is 1M the socket should allow for at least 1M rcvbuf size to accept it, right?
Quoted 5 lines
What I meant was :
This is the generic point, accepting
skb->truesizeas additional input here would make us more vulnerable, or we could risk other regressions.Understood, thanks for the clarification.
Quoted 2 lines
The question is : why does MPTCP end up here in the first place. Perhaps an older issue with an incorrectly sized
sk_rcvbuf?I collected a few more data. The issue happens even with plain TCP sockets[1].
The relevant transfer is on top of the loopback device. The
scaling_raterapidly grows to 254 - that istruesizeandlenare very near.The stall happens when the received get in a packet with a slightly less 'efficient' layout (in the experiment I have handy len is 71424, truesize 72320) (almost) filling the receiver window.
Quoted 8 lines
Quoted 6 lines
On such input,
tcp_clamp_window()shrinks the receiver buffer to the current rmem usage. The same happens on retransmissions until rcvbuf becomes 0.I *think* that catching only the !sk_rmem_alloc case would avoid the stall, but I think it's a bit 'late'.
A packetdrill test here would help understanding your concern.
I fear like a complete working script would take a lot of time, let me try to sketch just the relevant part:
# receiver state is: # rmem=110592 rcvbuf=174650 scaling_ratio=253 rwin=63232 # no OoO data, no memory pressure,# the incoming packet is in sequence +0 > P. 109297:172528(63232) ack 1
Quoted 6 lines
Quoted 3 lines
With just the 0 rmem check in
tcp_prune_queue(), such function will still invoketcp_clamp_window()that will shrink the receive buffer to 110592.As long as an ACK is sent back with a smaller RWIN, I think this would be reasonable in this case.
I fear some possible regression, as the sender will see some unexpected drops, even on loopback and while not misbehaving.
But I tested your proposed code here and AFAICS solves the issue and I could not spot anything suspicious so far.
So I'll send a v2 using that.
Thanks!
Paolo
Jakub Kicinski
maintainerRe: [PATCH net-next 1/2] tcp: do not set a zero size receive buffer
In reply to Paolo AbeniQuoted 21 lines · On Mon, 21 Jul 2025 16:56:06 +0200 Paolo Abeni wrote:
Quoted 4 lines
Quoted 2 lines
I *think* that catching only the !sk_rmem_alloc case would avoid the stall, but I think it's a bit 'late'.
A packetdrill test here would help understanding your concern.
I fear like a complete working script would take a lot of time, let me try to sketch just the relevant part:
# receiver state is: # rmem=110592 rcvbuf=174650 scaling_ratio=253 rwin=63232 # no OoO data, no memory pressure,# the incoming packet is in sequence +0 > P. 109297:172528(63232) ack 1With just the 0 rmem check in
tcp_prune_queue(), such function will still invoketcp_clamp_window()that will shrink the receive buffer to 110592.tcp_collapse()can't make enough room and the incoming packet will be dropped. I think we should instead accept such packet.
Quoted 4 lines
Side note: the above data are taken from an actual reproduction of the issue
Please LMK if the above clarifies a bit my doubt or if a full pktdrill is needed.
Not the first time we stumble on packetdrills for scaling ratio. Solving it is probably outside the scope of this discussion but I wonder what would be the best way to do it. My go to is to integrate packetdrill with netdevsim and have an option for netdevsim to inflate truesize on demand. But perhaps there's a clever way we can force something like tap to give us the ratio we desire. Other ideas?
Jakub Kicinski
maintainerRe: [PATCH net-next 1/2] tcp: do not set a zero size receive buffer
Quoted 11 lines · On Mon, 21 Jul 2025 08:27:28 -0700 Jakub Kicinski wrote:
Quoted 10 lines
With just the 0 rmem check in
tcp_prune_queue(), such function will still invoketcp_clamp_window()that will shrink the receive buffer to 110592.tcp_collapse()can't make enough room and the incoming packet will be dropped. I think we should instead accept such packet.Side note: the above data are taken from an actual reproduction of the issue
Please LMK if the above clarifies a bit my doubt or if a full pktdrill is needed.
Quoted 6 lines
Not the first time we stumble on packetdrills for scaling ratio. Solving it is probably outside the scope of this discussion but I wonder what would be the best way to do it. My go to is to integrate packetdrill with netdevsim and have an option for netdevsim to inflate truesize on demand. But perhaps there's a clever way we can force something like tap to give us the ratio we desire. Other ideas?
FWIW didn't see Eric's reply before hitting send..
Re: [PATCH net-next 1/2] tcp: do not set a zero size receive buffer
In reply to Jakub KicinskiQuoted 28 lines · Jakub Kicinski
Quoted 21 lines · On Mon, 21 Jul 2025 16:56:06 +0200 Paolo Abeni wrote:
Quoted 4 lines
Quoted 2 lines
I *think* that catching only the !sk_rmem_alloc case would avoid the stall, but I think it's a bit 'late'.
A packetdrill test here would help understanding your concern.
I fear like a complete working script would take a lot of time, let me try to sketch just the relevant part:
# receiver state is: # rmem=110592 rcvbuf=174650 scaling_ratio=253 rwin=63232 # no OoO data, no memory pressure,# the incoming packet is in sequence +0 > P. 109297:172528(63232) ack 1With just the 0 rmem check in
tcp_prune_queue(), such function will still invoketcp_clamp_window()that will shrink the receive buffer to 110592.tcp_collapse()can't make enough room and the incoming packet will be dropped. I think we should instead accept such packet.Quoted 4 lines
Side note: the above data are taken from an actual reproduction of the issue
Please LMK if the above clarifies a bit my doubt or if a full pktdrill is needed.
Quoted 6 lines
Not the first time we stumble on packetdrills for scaling ratio. Solving it is probably outside the scope of this discussion but I wonder what would be the best way to do it. My go to is to integrate packetdrill with netdevsim and have an option for netdevsim to inflate truesize on demand. But perhaps there's a clever way we can force something like tap to give us the ratio we desire. Other ideas?
Adding a TUN option (ioctl) to be able to increase skb->truesize by a given amount / percentage is doable I think.
Then in packetdrill we could add the ability to use this option for a given scenario or packet.
Paolo Abeni
maintainerPATCH 2/2 tcp: do not increment BeyondWindow MIB for old seq
In reply to Paolo AbeniThe mentioned MIB is currently incremented even when a packet with an old sequence number (i.e. a zero window probe) is received, which is IMHO misleading.
Explicitly restrict such MIB increment at the relevant events.
--- net/ipv4/tcp_input.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
net/ipv4/tcp_input.c +5 -1
@@ -5911,7 +5911,11 @@ static bool tcp_validate_incoming(struct sock *sk, struct sk_buff *skb,if (!th->rst) {if (th->syn) goto syn_challenge; -NET_INC_STATS (sock_net (sk),LINUX_MIB_BEYOND_WINDOW ); + +if (reason ==SKB_DROP_REASON_TCP_INVALID_SEQUENCE || + reason ==SKB_DROP_REASON_TCP_INVALID_END_SEQUENCE ) +NET_INC_STATS (sock_net (sk), +LINUX_MIB_BEYOND_WINDOW );if (!tcp_oow_rate_limited (sock_net (sk), skb,LINUX_MIB_TCPACKSKIPPEDSEQ , &tp->last_oow_ack_time))
Signature · 1 line
2.50.0
Re: [PATCH net-next 2/2] tcp: do not increment BeyondWindow MIB for old seq
Hi Paolo,
Quoted 5 lines · Paolo Abeni
The mentioned MIB is currently incremented even when a packet with an old sequence number (i.e. a zero window probe) is received, which is IMHO misleading.
Explicitly restrict such MIB increment at the relevant events.
Good idea, it is less confusing!
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.
Re: [PATCH net-next 2/2] tcp: do not increment BeyondWindow MIB for old seq
In reply to Paolo AbeniQuoted 9 lines · Paolo Abeni
The mentioned MIB is currently incremented even when a packet with an old sequence number (i.e. a zero window probe) is received, which is IMHO misleading.
Explicitly restrict such MIB increment at the relevant events.
Fixes 6c758062c64d ("tcp: add LINUX_MIB_BEYOND_WINDOW")Signed-off-by Paolo Abeni <[email protected]>
Thanks !
Matthieu Baerts
maintainerHi Paolo,
Quoted 6 lines · Paolo Abeni
This series includes a couple of follow-up for the recent tcp receiver changes, addressing issues outlined by the nipa CI and the mptcp self-tests.
Note that despite the affected self-tests where MPTCP ones, the issues are really in the TCP code, see patch 1 for the details.
FYI, the two patches have been applied in the MPTCP tree on Friday. (see below).
Today, the CI complained about issues with mptcp_connect.sh, with the "disconnect" tests:
- export:
https://github.com/multipath-tcp/mptcp_net-next/actions/runs/16409726187
Quoted 4 lines
# INFO: disconnect
# 63 ns1 MPTCP -> ns1 (10.0.1.1:20001 ) MPTCP (duration 4052ms) [ OK ] # 64 ns1 MPTCP -> ns1 (10.0.1.1:20002 ) TCP (duration 963ms) [ OK ] # 65 ns1 TCP -> ns1 (10.0.1.1:20003 ) MPTCP Unexpected revents: POLLERR/POLLNVAL(19)
Quoted 9 lines
# (duration 1002ms) [FAIL] file received by server does not match (in, out):
# -rw-r--r-- 1 root root 14170836 Jul 21 06:12/tmp/tmp.RA63zyhMfZ.disconnect# Trailing bytes are:
# �[�����05}�����Oka�<���aH�1-rw------- 1 root root 14029930 Jul 21 2025/tmp/tmp.BIcGXjzpdJ# Trailing bytes are:
# �[�����05}�����Oka�<���aH�166 ns1 MPTCP -> ns1 (dead:beef:1::1:20004) MPTCP (duration 1062ms) [ OK ]
# 67 ns1 MPTCP -> ns1 (dead:beef:1::1:20005) TCP (duration 1062ms) [ OK ]
# 68 ns1 TCP -> ns1 (dead:beef:1::1:20006) MPTCP (duration 998ms) [ OK ]
# [FAIL] Tests of the full disconnection have failed
Do you think this could be related?
- export-net:
https://github.com/multipath-tcp/mptcp_net-next/actions/runs/16409727049
Quoted 10 lines
# INFO: disconnect
# 63 ns1 MPTCP -> ns1 (10.0.1.1:20001 ) MPTCP (duration 3797ms) [ OK ] # 64 ns1 MPTCP -> ns1 (10.0.1.1:20002 ) TCP (duration 1259ms) [ OK ] # 65 ns1 TCP -> ns1 (10.0.1.1:20003 ) MPTCP read: Connection reset by peer# (duration 1317ms) [FAIL] file received by server does not match (in, out):
# -rw-r--r-- 1 root root 19806768 Jul 21 06:13/tmp/tmp.bjbDDrTOHl.disconnect# Trailing bytes are:
# �Z�V;*�49�������#1�0��>3�;-rw------- 1 root root 19591200 Jul 21 2025/tmp/tmp.lJwoEj0wut# Trailing bytes are:
# �Z�V;*�49�������#1�0��>3�;66 ns1 MPTCP -> ns1 (dead:beef:1::1:20004) MPTCP (duration 1420ms) [ OK ]
Quoted 3 lines
# 67 ns1 MPTCP -> ns1 (dead:beef:1::1:20005) TCP (duration 1440ms) [ OK ] # 68 ns1 TCP -> ns1 (dead:beef:1::1:20006) MPTCP (duration 1370ms) [ OK ]# [FAIL] Tests of the full disconnection have failed
This one is probably unrelated.
---------------------------
New patches for t/upstream:
- ebfd9a78da4b: tcp: do not set a zero size receive buffer
- 038166f66b80: tcp: do not increment BeyondWindow MIB for old seq
- Results: de29714b04e4..132bb933cdf0 (export)
Tests are now in progress:
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.
Paolo Abeni
maintainerQuoted 54 lines · Matthieu Baerts
CC -netdev, +mptcpQuoted 6 lines · Paolo Abeni
This series includes a couple of follow-up for the recent tcp receiver changes, addressing issues outlined by the nipa CI and the mptcp self-tests.
Note that despite the affected self-tests where MPTCP ones, the issues are really in the TCP code, see patch 1 for the details.
FYI, the two patches have been applied in the MPTCP tree on Friday. (see below).
Today, the CI complained about issues with
mptcp_connect.sh, with the "disconnect" tests:- export:
https://github.com/multipath-tcp/mptcp_net-next/actions/runs/16409726187
Quoted 4 lines
# INFO: disconnect
# 63 ns1 MPTCP -> ns1 (10.0.1.1:20001 ) MPTCP (duration 4052ms) [ OK ] # 64 ns1 MPTCP -> ns1 (10.0.1.1:20002 ) TCP (duration 963ms) [ OK ] # 65 ns1 TCP -> ns1 (10.0.1.1:20003 ) MPTCP Unexpected revents: POLLERR/POLLNVAL(19)Quoted 9 lines
# (duration 1002ms) [FAIL] file received by server does not match (in, out):
# -rw-r--r-- 1 root root 14170836 Jul 21 06:12/tmp/tmp.RA63zyhMfZ.disconnect# Trailing bytes are:
# �[�����05}�����Oka�<���aH�1-rw------- 1 root root 14029930 Jul 21 2025/tmp/tmp.BIcGXjzpdJ# Trailing bytes are:
# �[�����05}�����Oka�<���aH�166 ns1 MPTCP -> ns1 (dead:beef:1::1:20004) MPTCP (duration 1062ms) [ OK ]
# 67 ns1 MPTCP -> ns1 (dead:beef:1::1:20005) TCP (duration 1062ms) [ OK ]
# 68 ns1 TCP -> ns1 (dead:beef:1::1:20006) MPTCP (duration 998ms) [ OK ]
# [FAIL] Tests of the full disconnection have failedDo you think this could be related?
- export-net:
https://github.com/multipath-tcp/mptcp_net-next/actions/runs/16409727049
Quoted 10 lines
# INFO: disconnect
# 63 ns1 MPTCP -> ns1 (10.0.1.1:20001 ) MPTCP (duration 3797ms) [ OK ] # 64 ns1 MPTCP -> ns1 (10.0.1.1:20002 ) TCP (duration 1259ms) [ OK ] # 65 ns1 TCP -> ns1 (10.0.1.1:20003 ) MPTCP read: Connection reset by peer# (duration 1317ms) [FAIL] file received by server does not match (in, out):
# -rw-r--r-- 1 root root 19806768 Jul 21 06:13/tmp/tmp.bjbDDrTOHl.disconnect# Trailing bytes are:
# �Z�V;*�49�������#1�0��>3�;-rw------- 1 root root 19591200 Jul 21 2025/tmp/tmp.lJwoEj0wut# Trailing bytes are:
# �Z�V;*�49�������#1�0��>3�;66 ns1 MPTCP -> ns1 (dead:beef:1::1:20004) MPTCP (duration 1420ms) [ OK ]Quoted 3 lines
# 67 ns1 MPTCP -> ns1 (dead:beef:1::1:20005) TCP (duration 1440ms) [ OK ] # 68 ns1 TCP -> ns1 (dead:beef:1::1:20006) MPTCP (duration 1370ms) [ OK ]# [FAIL] Tests of the full disconnection have failed
Quoted 1 line
This one is probably unrelated.
Yep, I'm observing something similar here, too, with a considerably low frequency that makes investigation hard :/
/P