Skip to contentSkip to conversation

tcp: introduce rto_max_us sysctl knob

PATCH In review Jason Xing 2 participants 7 messages Jul 15, 2024 – Jul 16, 2024

Subsystems (4)
lore
Share & tools
lore

Jason Xing

author
#0 lore raw

PATCH 1/1 tcp: introduce rto_max_us sysctl knob

From: Jason Xing <[email protected]>

As we all know, the algorithm of rto is exponential backoff as RFC defined long time ago. After several rounds of repeatedly transmitting a lost skb, the expiry of rto timer could reach above 1 second within the upper bound (120s).

Waiting more than one second to retransmit for some latency-sensitive application is a little bit unacceptable nowadays, so I decided to introduce a sysctl knob to allow users to tune it. Still, the maximum value is 120 seconds.

---
 Documentation/networking/ip-sysctl.rst | 10 ++++++++++
 include/net/inet_connection_sock.h     | 10 +++++++++-
 include/net/netns/ipv4.h               |  1 +
 net/ipv4/sysctl_net_ipv4.c             |  8 ++++++++
 net/ipv4/tcp_ipv4.c                    |  1 +
 5 files changed, 29 insertions(+), 1 deletion(-)
Documentation/networking/ip-sysctl.rst +10 -0
index 3616389c8c2d..32a1907ca95d 100644
@@ -1223,6 +1223,16 @@ tcp_rto_min_us - INTEGER
 
 	Default: 200000
 
+tcp_rto_max_us - INTEGER
+	Maximum TCP retransmission timeout (in microseconds).
+
+	The recommended practice is to use a value less or equal to 120000000
+	microseconds.
+
+	Possible Values: 1 - INT_MAX
+
+	Default: 120000000
+
 UDP variables
 =============
 
include/net/inet_connection_sock.h +9 -1
index c0deaafebfdc..a0abbafcab9e 100644
@@ -217,10 +217,18 @@ static inline void inet_csk_clear_xmit_timer(struct sock *sk, const int what)
  */
 static inline void inet_csk_reset_xmit_timer(struct sock *sk, const int what,
 					     unsigned long when,
-					     const unsigned long max_when)
+					     unsigned long max_when)
 {
 	struct inet_connection_sock *icsk = inet_csk(sk);
 
+	if (what == ICSK_TIME_RETRANS) {
+		int rto_max_us = READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_rto_max_us);
+		unsigned int rto_max = usecs_to_jiffies(rto_max_us);
+
+		if (rto_max < max_when)
+			max_when = rto_max;
+	}
+
 	if (when > max_when) {
 		pr_debug("reset_xmit_timer: sk=%p %d when=0x%lx, caller=%p\n",
 			 sk, what, when, (void *)_THIS_IP_);
include/net/netns/ipv4.h +1 -0
index 5fcd61ada622..09a28a5c94d2 100644
@@ -178,6 +178,7 @@ struct netns_ipv4 {
 	u8 sysctl_tcp_window_scaling;
 	u8 sysctl_tcp_timestamps;
 	int sysctl_tcp_rto_min_us;
+	int sysctl_tcp_rto_max_us;
 	u8 sysctl_tcp_recovery;
 	u8 sysctl_tcp_thin_linear_timeouts;
 	u8 sysctl_tcp_slow_start_after_idle;
net/ipv4/sysctl_net_ipv4.c +8 -0
index 9140d20eb2d4..304f173837bc 100644
@@ -1573,6 +1573,14 @@ static struct ctl_table ipv4_net_table[] = {
 		.proc_handler	= proc_dointvec_minmax,
 		.extra1		= SYSCTL_ONE,
 	},
+	{
+		.procname	= "tcp_rto_max_us",
+		.data		= &init_net.ipv4.sysctl_tcp_rto_max_us,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= SYSCTL_ONE,
+	},
 };
 
 static __net_init int ipv4_sysctl_init_net(struct net *net)
net/ipv4/tcp_ipv4.c +1 -0
index fd17f25ff288..f06859be5942 100644
@@ -3506,6 +3506,7 @@ static int __net_init tcp_sk_init(struct net *net)
 
 	net->ipv4.sysctl_tcp_pingpong_thresh = 1;
 	net->ipv4.sysctl_tcp_rto_min_us = jiffies_to_usecs(TCP_RTO_MIN);
+	net->ipv4.sysctl_tcp_rto_max_us = jiffies_to_usecs(TCP_RTO_MAX);
 
 	return 0;
 }
Signature · 1 line
2.37.3
#1 lore raw
Quoted 5 lines · Jason Xing

From: Jason Xing <[email protected]>

As we all know, the algorithm of rto is exponential backoff as RFC defined long time ago.

This is weak sentence. Please provide RFC numbers instead.

Quoted 3 lines

After several rounds of repeatedly transmitting
a lost skb, the expiry of rto timer could reach above 1 second within the upper bound (120s).

This is confusing. What do you mean exactly ?

Quoted 5 lines

Waiting more than one second to retransmit for some latency-sensitive application is a little bit unacceptable nowadays, so I decided to introduce a sysctl knob to allow users to tune it. Still, the maximum value is 120 seconds.

I do not think this sysctl needs usec resolution.

Also storing this sysctl once, and converting it millions of times per second to jiffies is not good.
I suggest you use proc_dointvec_jiffies() instead in the sysctl handler.

Minimal value of one jiffies makes also no sense. We can not predict if some distros/users
might (ab)use this sysctl.

You forgot to update Documentation/networking/net_cachelines/netns_ipv4_sysctl.rst This means the location you chose for the new sysctl is pretty much random and not reflcting
this is used in one fast path.

I suggest you wait for net-next being reopened, we are all busy attending netdev 0x18 conference.

author
#2 lore raw

Hello Eric,

Quoted 9 lines · Eric Dumazet
Quoted 5 lines · Jason Xing

From: Jason Xing <[email protected]>

As we all know, the algorithm of rto is exponential backoff as RFC defined long time ago.

This is weak sentence. Please provide RFC numbers instead.

RFC 6298. I will update it.

Quoted 6 lines
Quoted 4 lines

After several rounds of repeatedly transmitting
a lost skb, the expiry of rto timer could reach above 1 second within the upper bound (120s).

This is confusing. What do you mean exactly ?

I will rewrite this part. What I was trying to say is that waiting more than 1 second is not very friendly to some applications, especially the expiry time can reach 120 seconds which is too long.

Quoted 8 lines
Quoted 6 lines

Waiting more than one second to retransmit for some latency-sensitive application is a little bit unacceptable nowadays, so I decided to introduce a sysctl knob to allow users to tune it. Still, the maximum value is 120 seconds.

I do not think this sysctl needs usec resolution.

Are you suggesting using jiffies is enough? But I have two reasons:
1) Keep the consistency with rto_min_us
2) If rto_min_us can be set to a very small value, why not rto_max?

What do you think?

Quoted 8 lines

Also storing this sysctl once, and converting it millions of times per second to jiffies is not good.
I suggest you use proc_dointvec_jiffies() instead in the sysctl handler.

Minimal value of one jiffies makes also no sense. We can not predict if some distros/users
might (ab)use this sysctl.

Okay. If the final solution is using jiffies, I will accordingly adjust it.

Quoted 3 lines

You forgot to update Documentation/networking/net_cachelines/netns_ipv4_sysctl.rst

Oh sorry, I forgot.

Quoted 3 lines

This means the location you chose for the new sysctl is pretty much random and not reflcting
this is used in one fast path.

I will investigate its proper location...

Quoted 3 lines

I suggest you wait for net-next being reopened, we are all busy attending netdev 0x18 conference.

Roger that. Thanks for your suggestions.

Thanks,
Jason

#3 lore raw
Quoted 21 lines · Jason Xing

Hello Eric,

Quoted 9 lines · Eric Dumazet
Quoted 5 lines · Jason Xing

From: Jason Xing <[email protected]>

As we all know, the algorithm of rto is exponential backoff as RFC defined long time ago.

This is weak sentence. Please provide RFC numbers instead.

RFC 6298. I will update it.

Quoted 5 lines
Quoted 4 lines

After several rounds of repeatedly transmitting
a lost skb, the expiry of rto timer could reach above 1 second within the upper bound (120s).

Quoted 5 lines
Quoted 1 line

This is confusing. What do you mean exactly ?

I will rewrite this part. What I was trying to say is that waiting more than 1 second is not very friendly to some applications, especially the expiry time can reach 120 seconds which is too long.

Says who ? I think this needs IETF approval.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

Waiting more than one second to retransmit for some latency-sensitive application is a little bit unacceptable nowadays, so I decided to introduce a sysctl knob to allow users to tune it. Still, the maximum value is 120 seconds.

Quoted 5 lines
Quoted 1 line

I do not think this sysctl needs usec resolution.

Are you suggesting using jiffies is enough? But I have two reasons:
1) Keep the consistency with rto_min_us
2) If rto_min_us can be set to a very small value, why not rto_max?

rto_max is usually 3 order of magnitude higher than rto_min

For HZ=100, using jiffies for rto_min would not allow rto_min < 10 ms. Some of us use 5 msec rto_min.

jiffies is plain enough for rto_max.

Quoted 2 lines

What do you think?

I think you missed many many details really.

Look at all TCP_RTO_MAX instances in net/ipv4/tcp_timer.c and ask yourself how many things
will break if we allow a sysctl value with 1 second for rto_max.

Quoted 32 lines
Quoted 9 lines

Also storing this sysctl once, and converting it millions of times per second to jiffies is not good.
I suggest you use proc_dointvec_jiffies() instead in the sysctl handler.

Minimal value of one jiffies makes also no sense. We can not predict if some distros/users
might (ab)use this sysctl.

Okay. If the final solution is using jiffies, I will accordingly adjust it.

Quoted 3 lines

You forgot to update Documentation/networking/net_cachelines/netns_ipv4_sysctl.rst

Oh sorry, I forgot.

Quoted 3 lines

This means the location you chose for the new sysctl is pretty much random and not reflcting
this is used in one fast path.

I will investigate its proper location...

Quoted 3 lines

I suggest you wait for net-next being reopened, we are all busy attending netdev 0x18 conference.

Roger that. Thanks for your suggestions.

Thanks,
Jason

author
#4 lore raw
Quoted 23 lines · Eric Dumazet
Quoted 21 lines · Jason Xing

Hello Eric,

Quoted 9 lines · Eric Dumazet
Quoted 5 lines · Jason Xing

From: Jason Xing <[email protected]>

As we all know, the algorithm of rto is exponential backoff as RFC defined long time ago.

This is weak sentence. Please provide RFC numbers instead.

RFC 6298. I will update it.

Quoted 5 lines
Quoted 4 lines

After several rounds of repeatedly transmitting
a lost skb, the expiry of rto timer could reach above 1 second within the upper bound (120s).

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

This is confusing. What do you mean exactly ?

I will rewrite this part. What I was trying to say is that waiting more than 1 second is not very friendly to some applications, especially the expiry time can reach 120 seconds which is too long.

Says who ? I think this needs IETF approval.

Did you get me wrong? I mean this rto_max is the same as rto_min_us, which can be tuned by users.

Quoted 15 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

Waiting more than one second to retransmit for some latency-sensitive application is a little bit unacceptable nowadays, so I decided to introduce a sysctl knob to allow users to tune it. Still, the maximum value is 120 seconds.

Quoted 5 lines
Quoted 1 line

I do not think this sysctl needs usec resolution.

Are you suggesting using jiffies is enough? But I have two reasons:
1) Keep the consistency with rto_min_us
2) If rto_min_us can be set to a very small value, why not rto_max?

Quoted 6 lines

rto_max is usually 3 order of magnitude higher than rto_min

For HZ=100, using jiffies for rto_min would not allow rto_min < 10 ms. Some of us use 5 msec rto_min.

jiffies is plain enough for rto_max.

I got it. Thanks.

Quoted 10 lines
Quoted 4 lines

What do you think?

I think you missed many many details really.

Look at all TCP_RTO_MAX instances in net/ipv4/tcp_timer.c and ask yourself how many things
will break if we allow a sysctl value with 1 second for rto_max.

I'm not modifying the TCP_RTO_MAX value which is tooooo complicated. Instead, I'm trying to control the maximum expiry time in the ICSK_TIME_RETRANS timer. So it's only involved in three cases:
1) syn retrans
2) synack retrans
3) data retrans

Thanks,
Jason

author
#5 lore raw
Quoted 71 lines · Jason Xing
Quoted 23 lines · Eric Dumazet
Quoted 21 lines · Jason Xing

Hello Eric,

Quoted 9 lines · Eric Dumazet
Quoted 5 lines · Jason Xing

From: Jason Xing <[email protected]>

As we all know, the algorithm of rto is exponential backoff as RFC defined long time ago.

This is weak sentence. Please provide RFC numbers instead.

RFC 6298. I will update it.

Quoted 5 lines
Quoted 4 lines

After several rounds of repeatedly transmitting
a lost skb, the expiry of rto timer could reach above 1 second within the upper bound (120s).

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

This is confusing. What do you mean exactly ?

I will rewrite this part. What I was trying to say is that waiting more than 1 second is not very friendly to some applications, especially the expiry time can reach 120 seconds which is too long.

Says who ? I think this needs IETF approval.

Did you get me wrong? I mean this rto_max is the same as rto_min_us, which can be tuned by users.

Quoted 15 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

Waiting more than one second to retransmit for some latency-sensitive application is a little bit unacceptable nowadays, so I decided to introduce a sysctl knob to allow users to tune it. Still, the maximum value is 120 seconds.

Quoted 5 lines
Quoted 1 line

I do not think this sysctl needs usec resolution.

Are you suggesting using jiffies is enough? But I have two reasons:
1) Keep the consistency with rto_min_us
2) If rto_min_us can be set to a very small value, why not rto_max?

Quoted 6 lines

rto_max is usually 3 order of magnitude higher than rto_min

For HZ=100, using jiffies for rto_min would not allow rto_min < 10 ms. Some of us use 5 msec rto_min.

jiffies is plain enough for rto_max.

I got it. Thanks.

Quoted 10 lines
Quoted 4 lines

What do you think?

I think you missed many many details really.

Look at all TCP_RTO_MAX instances in net/ipv4/tcp_timer.c and ask yourself how many things
will break if we allow a sysctl value with 1 second for rto_max.

Quoted 6 lines

I'm not modifying the TCP_RTO_MAX value which is tooooo complicated. Instead, I'm trying to control the maximum expiry time in the ICSK_TIME_RETRANS timer. So it's only involved in three cases:
1) syn retrans
2) synack retrans
3) data retrans

To be clearer, my initial goal is to accelerate the speed of retransmitting data. There is a simpler way to implement it only in tcp_retransmit_timer().

author
#6 lore raw
Quoted 80 lines · Jason Xing
Quoted 71 lines · Jason Xing
Quoted 23 lines · Eric Dumazet
Quoted 21 lines · Jason Xing

Hello Eric,

Quoted 9 lines · Eric Dumazet
Quoted 5 lines · Jason Xing

From: Jason Xing <[email protected]>

As we all know, the algorithm of rto is exponential backoff as RFC defined long time ago.

This is weak sentence. Please provide RFC numbers instead.

RFC 6298. I will update it.

Quoted 5 lines
Quoted 4 lines

After several rounds of repeatedly transmitting
a lost skb, the expiry of rto timer could reach above 1 second within the upper bound (120s).

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

This is confusing. What do you mean exactly ?

I will rewrite this part. What I was trying to say is that waiting more than 1 second is not very friendly to some applications, especially the expiry time can reach 120 seconds which is too long.

Says who ? I think this needs IETF approval.

Did you get me wrong? I mean this rto_max is the same as rto_min_us, which can be tuned by users.

Quoted 15 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

Waiting more than one second to retransmit for some latency-sensitive application is a little bit unacceptable nowadays, so I decided to introduce a sysctl knob to allow users to tune it. Still, the maximum value is 120 seconds.

Quoted 5 lines
Quoted 1 line

I do not think this sysctl needs usec resolution.

Are you suggesting using jiffies is enough? But I have two reasons:
1) Keep the consistency with rto_min_us
2) If rto_min_us can be set to a very small value, why not rto_max?

Quoted 6 lines

rto_max is usually 3 order of magnitude higher than rto_min

For HZ=100, using jiffies for rto_min would not allow rto_min < 10 ms. Some of us use 5 msec rto_min.

jiffies is plain enough for rto_max.

I got it. Thanks.

Quoted 10 lines
Quoted 4 lines

What do you think?

I think you missed many many details really.

Look at all TCP_RTO_MAX instances in net/ipv4/tcp_timer.c and ask yourself how many things
will break if we allow a sysctl value with 1 second for rto_max.

Quoted 6 lines

I'm not modifying the TCP_RTO_MAX value which is tooooo complicated. Instead, I'm trying to control the maximum expiry time in the ICSK_TIME_RETRANS timer. So it's only involved in three cases:
1) syn retrans
2) synack retrans
3) data retrans

Quoted 3 lines

To be clearer, my initial goal is to accelerate the speed of retransmitting data. There is a simpler way to implement it only in tcp_retransmit_timer().

Sorry to keep replying to this thread so frequently. I have to admit tuning rto max is much more complicated than rto min. I think simply reducing the time of retrans is not a feasible way.