Skip to contentSkip to conversation

net: tcp: support to probe tcp receiver OOM

PATCH In review [email protected] 3 participants 5 messages Jul 13, 2023 – Jul 19, 2023

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 net: tcp: support to probe tcp receiver OOM

From: Menglong Dong <[email protected]>

For now, skb will be dropped directly if rmem schedule fails, which means tcp_try_rmem_schedule() returns an error. This can happen on following cases:

1. The total memory allocated for TCP protocol is up to tcp_mem[2], and the receive queue of the tcp socket is not empty.
2. The receive buffer of the tcp socket is full, which can happen on small packet cases.

If the user hangs and doesn't take away the packet in the receive queue with recv() or read() for a long time, the sender will keep retransmitting until timeout, and the tcp connection will break.

In order to handle such case, we introduce the tcp protocol OOM detection in following steps, as Neal Cardwell suggested:

1. reply a pure ack, which doesn't contain SACK blocks, if we fail to alloc memory to queue the skb on the receiving path. We only do this for the next wanted sequence
2. we assume that the receiver is OOM if we have retransmitted the first skb in the retransmission queue for N times and every retransmission has a corresponding pure ack, which doesn't contain SACK blocks. Meanwhile, SACK should be enabled on the connection, too.
3. the sender enter OOM state, and use the probe timer instead of the retrans timer. In tcp_send_probe0(), we retransmit the first skb in rtx queue instead of zero-window probe packet if the socket is in OOM state.
4. we exit the OOM state if snd_una was changed, which means FLAG_SND_UNA_ADVANCED is set.

And we introduce the sysctl "tcp_oom_retries" for the "N". By default, its value is 0, which means the OOM detection is disabled, to reduce the impact to the users who don't need such function.

---
v2:
- disable the OOM detection by default to reduce the impact
- don't impact the fast path

---
 Documentation/networking/ip-sysctl.rst | 14 +++++
 include/linux/tcp.h                    |  1 +
 include/net/inet_connection_sock.h     |  3 +-
 include/net/netns/ipv4.h               |  1 +
 include/net/sock.h                     |  1 +
 net/ipv4/sysctl_net_ipv4.c             |  7 +++
 net/ipv4/tcp_input.c                   | 71 +++++++++++++++++++++++++-
 net/ipv4/tcp_ipv4.c                    |  1 +
 net/ipv4/tcp_output.c                  | 13 ++++-
 net/ipv4/tcp_timer.c                   |  7 ++-
 10 files changed, 114 insertions(+), 5 deletions(-)
Documentation/networking/ip-sysctl.rst +14 -0
index 4a010a7cde7f..3de2dce74037 100644
@@ -694,6 +694,20 @@ tcp_retries2 - INTEGER
 	RFC 1122 recommends at least 100 seconds for the timeout,
 	which corresponds to a value of at least 8.
 
+tcp_oom_retries - INTEGER
+	RTO retransmissions count of the receiver is recognised as OOM.
+	Given a value N, a hypothetical TCP connection will enter probe
+	state if N times RTO retransmissions performed and every
+	retransmission gets a pure ack, who doesn't contain SACK block.
+
+	The default value is 0, which means disable the OOM detection.
+	If disabled, skb will be dropped directly, without sending an
+	ACK, when rmem schedule fails. The function is valid only if sack
+	is enabled.
+
+	3 is the suggested value to enable this function. Do't make it
+	greater than tcp_retries2.
+
 tcp_rfc1337 - BOOLEAN
 	If set, the TCP stack behaves conforming to RFC1337. If unset,
 	we are not conforming to RFC, but prevent TCP TIME_WAIT
include/linux/tcp.h +1 -0
index b4c08ac86983..0801119a94ea 100644
@@ -337,6 +337,7 @@ struct tcp_sock {
 	u32	pushed_seq;	/* Last pushed seq, required to talk to windows */
 	u32	lost_out;	/* Lost packets			*/
 	u32	sacked_out;	/* SACK'd packets			*/
+	u32	oom_ack_rcv;	/* Number of received pure ACK for OOM */
 
 	struct hrtimer	pacing_timer;
 	struct hrtimer	compressed_ack_timer;
include/net/inet_connection_sock.h +2 -1
index c2b15f7e5516..2c0d51a241b3 100644
@@ -164,7 +164,8 @@ enum inet_csk_ack_state_t {
 	ICSK_ACK_TIMER  = 2,
 	ICSK_ACK_PUSHED = 4,
 	ICSK_ACK_PUSHED2 = 8,
-	ICSK_ACK_NOW = 16	/* Send the next ACK immediately (once) */
+	ICSK_ACK_NOW    = 16,	/* Send the next ACK immediately (once) */
+	ICSK_ACK_PURE   = 32,	/* Send ACK without sack */
 };
 
 void inet_csk_init_xmit_timers(struct sock *sk,
include/net/netns/ipv4.h +1 -0
index f00374718159..378656328b68 100644
@@ -196,6 +196,7 @@ struct netns_ipv4 {
 
 	u8 sysctl_fib_notify_on_flag_change;
 	u8 sysctl_tcp_syn_linear_timeouts;
+	u8 sysctl_tcp_oom_retries;
 
 #ifdef CONFIG_NET_L3_MASTER_DEV
 	u8 sysctl_udp_l3mdev_accept;
include/net/sock.h +1 -0
index 2eb916d1ff64..ecc5ba2f8ddf 100644
@@ -959,6 +959,7 @@ enum sock_flags {
 	SOCK_XDP, /* XDP is attached */
 	SOCK_TSTAMP_NEW, /* Indicates 64 bit timestamps always */
 	SOCK_RCVMARK, /* Receive SO_MARK  ancillary data with packet */
+	SOCK_RCV_OOM, /* the receiver is in OOM state */
 };
 
 #define SK_FLAGS_TIMESTAMP ((1UL << SOCK_TIMESTAMP) | (1UL << SOCK_TIMESTAMPING_RX_SOFTWARE))
net/ipv4/sysctl_net_ipv4.c +7 -0
index 2afb0870648b..61143d0864d7 100644
@@ -1489,6 +1489,13 @@ static struct ctl_table ipv4_net_table[] = {
 		.extra1		= SYSCTL_ZERO,
 		.extra2		= SYSCTL_ONE,
 	},
+	{
+		.procname	= "tcp_oom_retries",
+		.data		= &init_net.ipv4.sysctl_tcp_oom_retries,
+		.maxlen		= sizeof(u8),
+		.mode		= 0644,
+		.proc_handler	= proc_dou8vec_minmax,
+	},
 	{ }
 };
 
net/ipv4/tcp_input.c +70 -1

89 lines not on this page · raw message

net/ipv4/tcp_ipv4.c +1 -0
index cecd5a135e64..26a2fc6ae574 100644
@@ -3216,6 +3216,7 @@ static int __net_init tcp_sk_init(struct net *net)
 	net->ipv4.sysctl_tcp_notsent_lowat = UINT_MAX;
 	net->ipv4.sysctl_tcp_tw_reuse = 2;
 	net->ipv4.sysctl_tcp_no_ssthresh_metrics_save = 1;
+	net->ipv4.sysctl_tcp_oom_retries = 0;
 
 	refcount_set(&net->ipv4.tcp_death_row.tw_refcount, 1);
 	tcp_set_hashinfo(net);
net/ipv4/tcp_output.c +11 -2
index 2cb39b6dad02..6a30ce7e649f 100644
@@ -959,7 +959,8 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 	}
 
 	eff_sacks = tp->rx_opt.num_sacks + tp->rx_opt.dsack;
-	if (unlikely(eff_sacks)) {
+	if (!(inet_csk(sk)->icsk_ack.pending & ICSK_ACK_PURE) &&
+	    unlikely(eff_sacks)) {
 		const unsigned int remaining = MAX_TCP_OPTION_SPACE - size;
 		if (unlikely(remaining < TCPOLEN_SACK_BASE_ALIGNED +
 					 TCPOLEN_SACK_PERBLOCK))
@@ -4130,6 +4131,13 @@ int tcp_write_wakeup(struct sock *sk, int mib)
 	if (sk->sk_state == TCP_CLOSE)
 		return -1;
 
+	if (sock_flag(sk, SOCK_RCV_OOM)) {
+		skb = tcp_rtx_queue_head(sk);
+		WARN_ON_ONCE(!skb);
+		if (skb)
+			return __tcp_retransmit_skb(sk, skb, 1);
+	}
+
 	skb = tcp_send_head(sk);
 	if (skb && before(TCP_SKB_CB(skb)->seq, tcp_wnd_end(tp))) {
 		int err;
@@ -4178,7 +4186,8 @@ void tcp_send_probe0(struct sock *sk)
 
 	err = tcp_write_wakeup(sk, LINUX_MIB_TCPWINPROBE);
 
-	if (tp->packets_out || tcp_write_queue_empty(sk)) {
+	if (!sock_flag(sk, SOCK_RCV_OOM) &&
+	    (tp->packets_out || tcp_write_queue_empty(sk))) {
 		/* Cancel probe timer, if it is not required. */
 		icsk->icsk_probes_out = 0;
 		icsk->icsk_backoff = 0;
net/ipv4/tcp_timer.c +6 -1
index 470f581eedd4..41dbe52101b8 100644
@@ -369,7 +369,7 @@ static void tcp_probe_timer(struct sock *sk)
 	struct tcp_sock *tp = tcp_sk(sk);
 	int max_probes;
 
-	if (tp->packets_out || !skb) {
+	if (!sock_flag(sk, SOCK_RCV_OOM) && (tp->packets_out || !skb)) {
 		icsk->icsk_probes_out = 0;
 		icsk->icsk_probes_tstamp = 0;
 		return;
@@ -544,6 +544,11 @@ void tcp_retransmit_timer(struct sock *sk)
 		}
 		if (mib_idx)
 			__NET_INC_STATS(sock_net(sk), mib_idx);
+
+		/* initialize oom_ack_rcv with 0 when the first
+		 * retransmission happen
+		 */
+		tp->oom_ack_rcv = 0;
 	}
 
 	tcp_enter_loss(sk);
Signature · 1 line
2.40.1
reviewer
#1 lore raw

Documentation/networking/ip-sysctl.rst tcp_retries2 - INTEGER · line 701

Quoted 13 lines · On 7/13/23 04:24, [email protected] wrote:
Documentation/networking/ip-sysctl.rst +5 -0
index 4a010a7cde7f..3de2dce74037 100644
@@ -694,6 +694,20 @@ tcp_retries2 - INTEGER
 	RFC 1122 recommends at least 100 seconds for the timeout,
 	which corresponds to a value of at least 8.
 
+tcp_oom_retries - INTEGER
+	RTO retransmissions count of the receiver is recognised as OOM.
+	Given a value N, a hypothetical TCP connection will enter probe
+	state if N times RTO retransmissions performed and every
+	retransmission gets a pure ack, who doesn't contain SACK block.

ACK

Documentation/networking/ip-sysctl.rst tcp_retries2 - INTEGER · line 705

Quoted 4 lines
Documentation/networking/ip-sysctl.rst +4 -0
+
+	The default value is 0, which means disable the OOM detection.
+	If disabled, skb will be dropped directly, without sending an
+	ACK, when rmem schedule fails. The function is valid only if sack

SACK

Documentation/networking/ip-sysctl.rst tcp_retries2 - INTEGER · line 708

Quoted 3 lines
Documentation/networking/ip-sysctl.rst +3 -0
+	is enabled.
+
+	3 is the suggested value to enable this function. Do't make it

Don't

Quoted 1 line
Documentation/networking/ip-sysctl.rst +1 -0
+	greater than tcp_retries2.
Signature · 1 line
~Randy
#2 lore raw
Quoted 24 lines · Randy Dunlap

Documentation/networking/ip-sysctl.rst tcp_retries2 - INTEGER · line 701

Quoted 13 lines · On 7/13/23 04:24, [email protected] wrote:
Documentation/networking/ip-sysctl.rst +5 -0
index 4a010a7cde7f..3de2dce74037 100644
@@ -694,6 +694,20 @@ tcp_retries2 - INTEGER
      RFC 1122 recommends at least 100 seconds for the timeout,
      which corresponds to a value of at least 8.
 
+tcp_oom_retries - INTEGER
+     RTO retransmissions count of the receiver is recognised as OOM.
+     Given a value N, a hypothetical TCP connection will enter probe
+     state if N times RTO retransmissions performed and every
+     retransmission gets a pure ack, who doesn't contain SACK block.

ACK

Documentation/networking/ip-sysctl.rst tcp_retries2 - INTEGER · line 705

Quoted 4 lines
Documentation/networking/ip-sysctl.rst +4 -0
+
+     The default value is 0, which means disable the OOM detection.
+     If disabled, skb will be dropped directly, without sending an
+     ACK, when rmem schedule fails. The function is valid only if sack
Quoted 10 lines

SACK

Documentation/networking/ip-sysctl.rst tcp_retries2 - INTEGER · line 708

Quoted 3 lines
Documentation/networking/ip-sysctl.rst +3 -0
+     is enabled.
+
+     3 is the suggested value to enable this function. Do't make it

Don't

Quoted 1 line
Documentation/networking/ip-sysctl.rst +1 -0
+     greater than tcp_retries2.

Thanks! I'll fix them in the next version.

Quoted 2 lines

--
~Randy

#3 lore raw
In reply to [email protected]
Quoted 12 lines · On Thu, Jul 13, 2023 at 1:24 PM <[email protected]> wrote:

From: Menglong Dong <[email protected]>

For now, skb will be dropped directly if rmem schedule fails, which means tcp_try_rmem_schedule() returns an error. This can happen on following cases:

1. The total memory allocated for TCP protocol is up to tcp_mem[2], and the receive queue of the tcp socket is not empty.
2. The receive buffer of the tcp socket is full, which can happen on small packet cases.

Quoted 7 lines

If the user hangs and doesn't take away the packet in the receive queue with recv() or read() for a long time, the sender will keep retransmitting until timeout, and the tcp connection will break.

In order to handle such case, we introduce the tcp protocol OOM detection in following steps, as Neal Cardwell suggested:

For the record, I dislike this patch. I am not sure what Neal had in mind.

I suggested instead to send an ACK RWIN 0, whenever we were under extreme memory pressure,
and we only could queue one skb in the receive queue.

For details, look at the points we call sk_forced_mem_schedule(). This would be a matter of refactoring code around it, in tcp_data_queue()

The patch would be much simpler. Nothing changed at the sender side :/

#4 lore raw
Quoted 27 lines · Eric Dumazet
Quoted 12 lines · On Thu, Jul 13, 2023 at 1:24 PM <[email protected]> wrote:

From: Menglong Dong <[email protected]>

For now, skb will be dropped directly if rmem schedule fails, which means tcp_try_rmem_schedule() returns an error. This can happen on following cases:

1. The total memory allocated for TCP protocol is up to tcp_mem[2], and the receive queue of the tcp socket is not empty.
2. The receive buffer of the tcp socket is full, which can happen on small packet cases.

Quoted 7 lines

If the user hangs and doesn't take away the packet in the receive queue with recv() or read() for a long time, the sender will keep retransmitting until timeout, and the tcp connection will break.

In order to handle such case, we introduce the tcp protocol OOM detection in following steps, as Neal Cardwell suggested:

For the record, I dislike this patch. I am not sure what Neal had in mind.

Quoted 8 lines

I suggested instead to send an ACK RWIN 0, whenever we were under extreme memory pressure,
and we only could queue one skb in the receive queue.

For details, look at the points we call sk_forced_mem_schedule(). This would be a matter of refactoring code around it, in tcp_data_queue()

The patch would be much simpler. Nothing changed at the sender side :/

I think you are right. I misunderstood the code in tcp_retransmit_timer(). It seems that it already handle the window shrink case properly.

Let me do more testing first.

Thanks!
Menglong Dong