Skip to contentSkip to conversation

tcp: Prevent low rmem stalls with SO_RCVLOWAT.

PATCH Applied Arjun Roy 4 participants 4 messages Oct 23, 2020 – Oct 24, 2020

Subsystems (3)
Networking Network drivers TCP
lore
Share & tools
lore

Arjun Roy

author
#0 lore raw

PATCH v2 1/1 tcp: Prevent low rmem stalls with SO_RCVLOWAT.

From: Arjun Roy <[email protected]>

With SO_RCVLOWAT, under memory pressure,
it is possible to enter a state where:

1. We have not received enough bytes to satisfy SO_RCVLOWAT.
2. We have not entered buffer pressure (see tcp_rmem_pressure()).
3. But, we do not have enough buffer space to accept more packets.

In this case, we advertise 0 rwnd (due to #3) but the application does not drain the receive queue (no wakeup because of #1 and #2) so the flow stalls.

Modify the heuristic for SO_RCVLOWAT so that, if we are advertising rwnd<=rcv_mss, force a wakeup to prevent a stall.

Without this patch, setting tcp_rmem to 6143 and disabling TCP autotune causes a stalled flow. With this patch, no stall occurs. This is with RPC-style traffic with large messages.

Fixes 03f45c883c6f ("tcp: avoid extra wakeups for SO_RCVLOWAT users")
Signed-off-by Arjun Roy <[email protected]>
Acked-by Soheil Hassas Yeganeh <[email protected]>
Acked-by Neal Cardwell <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>
---
 net/ipv4/tcp.c       | 2 ++
 net/ipv4/tcp_input.c | 3 ++-
 2 files changed, 4 insertions(+), 1 deletion(-)
net/ipv4/tcp.c +2 -0
index bae4284bf542..b2bc3d7fe9e8 100644
@@ -485,6 +485,8 @@ static inline bool tcp_stream_is_readable(const struct tcp_sock *tp,
 			return true;
 		if (tcp_rmem_pressure(sk))
 			return true;
+		if (tcp_receive_window(tp) <= inet_csk(sk)->icsk_ack.rcv_mss)
+			return true;
 	}
 	if (sk->sk_prot->stream_memory_read)
 		return sk->sk_prot->stream_memory_read(sk);
net/ipv4/tcp_input.c +2 -1
index fc445833b5e5..389d1b340248 100644
@@ -4908,7 +4908,8 @@ void tcp_data_ready(struct sock *sk)
 	int avail = tp->rcv_nxt - tp->copied_seq;
 
 	if (avail < sk->sk_rcvlowat && !tcp_rmem_pressure(sk) &&
-	    !sock_flag(sk, SOCK_DONE))
+	    !sock_flag(sk, SOCK_DONE) &&
+	    tcp_receive_window(tp) > inet_csk(sk)->icsk_ack.rcv_mss)
 		return;
 
 	sk->sk_data_ready(sk);
Signature · 1 line
2.29.0.rc2.309.g374f81d7ae-goog
maintainer Applied
#1 lore raw
Quoted 20 lines · On Fri, 23 Oct 2020 11:47:09 -0700 Arjun Roy wrote:

From: Arjun Roy <[email protected]>

With SO_RCVLOWAT, under memory pressure,
it is possible to enter a state where:

1. We have not received enough bytes to satisfy SO_RCVLOWAT.
2. We have not entered buffer pressure (see tcp_rmem_pressure()).
3. But, we do not have enough buffer space to accept more packets.

In this case, we advertise 0 rwnd (due to #3) but the application does not drain the receive queue (no wakeup because of #1 and #2) so the flow stalls.

Modify the heuristic for SO_RCVLOWAT so that, if we are advertising rwnd<=rcv_mss, force a wakeup to prevent a stall.

Without this patch, setting tcp_rmem to 6143 and disabling TCP autotune causes a stalled flow. With this patch, no stall occurs. This is with RPC-style traffic with large messages.

Quoted 5 lines
Fixes 03f45c883c6f ("tcp: avoid extra wakeups for SO_RCVLOWAT users")
Signed-off-by Arjun Roy <[email protected]>
Acked-by Soheil Hassas Yeganeh <[email protected]>
Acked-by Neal Cardwell <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>

Applied, thank you!

#2 lore raw
Quoted 22 lines · Jakub Kicinski
Quoted 20 lines · On Fri, 23 Oct 2020 11:47:09 -0700 Arjun Roy wrote:

From: Arjun Roy <[email protected]>

With SO_RCVLOWAT, under memory pressure,
it is possible to enter a state where:

1. We have not received enough bytes to satisfy SO_RCVLOWAT.
2. We have not entered buffer pressure (see tcp_rmem_pressure()).
3. But, we do not have enough buffer space to accept more packets.

In this case, we advertise 0 rwnd (due to #3) but the application does not drain the receive queue (no wakeup because of #1 and #2) so the flow stalls.

Modify the heuristic for SO_RCVLOWAT so that, if we are advertising rwnd<=rcv_mss, force a wakeup to prevent a stall.

Without this patch, setting tcp_rmem to 6143 and disabling TCP autotune causes a stalled flow. With this patch, no stall occurs. This is with RPC-style traffic with large messages.

Quoted 7 lines
Quoted 5 lines
Fixes 03f45c883c6f ("tcp: avoid extra wakeups for SO_RCVLOWAT users")
Signed-off-by Arjun Roy <[email protected]>
Acked-by Soheil Hassas Yeganeh <[email protected]>
Acked-by Neal Cardwell <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>

Applied, thank you!

Ack, thanks for the quick review!

-Arjun

CI Applied
#3 lore raw
In reply to Arjun Roy
This patch was applied to netdev/net.git (refs/heads/master):

Hello:

This patch was applied to netdev/net.git (refs/heads/master):

Quoted 10 lines · On Fri, 23 Oct 2020 11:47:09 -0700 you wrote:

From: Arjun Roy <[email protected]>

With SO_RCVLOWAT, under memory pressure,
it is possible to enter a state where:

1. We have not received enough bytes to satisfy SO_RCVLOWAT.
2. We have not entered buffer pressure (see tcp_rmem_pressure()).
3. But, we do not have enough buffer space to accept more packets.

[...]

Here is the summary with links:
- [net,v2] tcp: Prevent low rmem stalls with SO_RCVLOWAT.

    https://git.kernel.org/netdev/net/c/435ccfa894e3

You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html