tcp: Make pingpong threshold tunable
Subsystems (6)
Haiyang Zhang
maintainerPATCH 1/1 tcp: Make pingpong threshold tunable
TCP pingpong threshold is 1 by default. But some applications, like SQL DB may prefer a higher pingpong threshold to activate delayed acks in quick ack mode for better performance.
The pingpong threshold and related code were changed to 3 in the year 2019, and reverted to 1 in the year 2022. There is no single value that fits all applications.
Add net.core.tcp_pingpong_thresh sysctl tunable, so it can be tuned for optimal performance based on the application needs.
--- Documentation/admin-guide/sysctl/net.rst | 8 ++++++++ include/net/inet_connection_sock.h | 14 +++++++++++--- net/core/sysctl_net_core.c | 9 +++++++++ net/ipv4/tcp.c | 2 ++ net/ipv4/tcp_output.c | 17 +++++++++++++++-- 5 files changed, 45 insertions(+), 5 deletions(-)
Documentation/admin-guide/sysctl/net.rst +8 -0
@@ -413,6 +413,14 @@ historical importance. Default: 0 +tcp_pingpong_thresh +------------------- + +TCP pingpong threshold is 1 by default, but some application may need a higher +threshold for optimal performance. + +Default: 1, min: 1, max: 3 + 2. /proc/sys/net/unix - Parameters for Unix domain sockets ----------------------------------------------------------
include/net/inet_connection_sock.h +11 -3
@@ -324,11 +324,11 @@ void inet_csk_update_fastreuse(struct inet_bind_bucket *tb,struct dst_entry *inet_csk_update_pmtu (struct sock *sk,u32 mtu); -#define TCP_PINGPONG_THRESH 1 +extern int tcp_pingpong_thresh;static inline void inet_csk_enter_pingpong_mode (struct sock *sk) { -inet_csk (sk)->icsk_ack.pingpong =TCP_PINGPONG_THRESH ; +inet_csk (sk)->icsk_ack.pingpong = tcp_pingpong_thresh; }static inline void inet_csk_exit_pingpong_mode (struct sock *sk)
@@ -338,7 +338,15 @@ static inline void inet_csk_exit_pingpong_mode(struct sock *sk)static inline bool inet_csk_in_pingpong_mode (struct sock *sk) { -return inet_csk (sk)->icsk_ack.pingpong >=TCP_PINGPONG_THRESH ; +return inet_csk (sk)->icsk_ack.pingpong >= tcp_pingpong_thresh; +} + +static inline void inet_csk_inc_pingpong_cnt (struct sock *sk) +{ +struct inet_connection_sock *icsk =inet_csk (sk); + +if (icsk->icsk_ack.pingpong <U8_MAX ) + icsk->icsk_ack.pingpong++; }static inline bool inet_csk_has_ulp (struct sock *sk)
net/core/sysctl_net_core.c +9 -0
@@ -653,6 +653,15 @@ static struct ctl_table net_core_table[] = { .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ZERO, }, + { + .procname = "tcp_pingpong_thresh", + .data = &tcp_pingpong_thresh, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ONE, + .extra2 = SYSCTL_THREE, + }, { } };
net/ipv4/tcp.c +2 -0
@@ -308,6 +308,8 @@ EXPORT_SYMBOL(tcp_have_smc);struct percpu_counter tcp_sockets_allocated____cacheline_aligned_in_smp ;EXPORT_SYMBOL (tcp_sockets_allocated); +int tcp_pingpong_thresh__read_mostly =1 ; +/* * TCP splice context */
net/ipv4/tcp_output.c +15 -2
@@ -167,12 +167,25 @@ static void tcp_event_data_sent(struct tcp_sock *tp,if (tcp_packets_in_flight (tp) ==0 )tcp_ca_event (sk,CA_EVENT_TX_START ); +/* If tcp_pingpong_thresh > 1, and +* this is the first data packet sent in response to the +* previous received data, +* and it is a reply for ato after last received packet, +* increase pingpong count. +*/ +if (tcp_pingpong_thresh >1 && +before (tp->lsndtime, icsk->icsk_ack.lrcvtime) && + (u32)(now - icsk->icsk_ack.lrcvtime) < icsk->icsk_ack.ato) +inet_csk_inc_pingpong_cnt (sk); + tp->lsndtime = now; -/* If it is a reply for ato after last received +/* If tcp_pingpong_thresh == 1, and +* it is a reply for ato after last received * packet, enter pingpong mode. */ -if ((u32)(now - icsk->icsk_ack.lrcvtime) < icsk->icsk_ack.ato) +if (tcp_pingpong_thresh ==1 && + (u32)(now - icsk->icsk_ack.lrcvtime) < icsk->icsk_ack.ato)inet_csk_enter_pingpong_mode (sk); }
Signature · 1 line
2.25.1
Neal Cardwell
maintainerOn Fri, Jun 9, 2023 at 12:26 PM Haiyang Zhang <[email protected]> wrote:
Regarding the patch title:
Quoted 1 line
[PATCH net-next] tcp: Make pingpong threshold tunable
There are many ways to make something tunable these days, including BPF, setsockopt(), and sysctl. :-) This patch only uses sysctl. Please consider a more clear/specific title, like:
[PATCH net-next] tcp: set pingpong threshold via sysctl
Quoted 6 lines
TCP pingpong threshold is 1 by default. But some applications, like SQL DB may prefer a higher pingpong threshold to activate delayed acks in quick ack mode for better performance.
The pingpong threshold and related code were changed to 3 in the year 2019, and reverted to 1 in the year 2022.
Please include the specific commit, like:
The pingpong threshold and related code were changed to 3 in the year
2019 in:
commit 4a41f453bedf ("tcp: change pingpong threshold to 3")
and reverted to 1 in the year 2022 in:
commit 4d8f24eeedc5 ("Revert "tcp: change pingpong threshold to 3"")
Then please make sure to use scripts/checkpatch.pl on your resulting patch to check the formatting of the commit references, among other things.
Quoted 4 lines
There is no single value that
fits all applications.Add net.core.tcp_pingpong_thresh sysctl tunable,
For consistency, TCP sysctls should be in net.ipv4 rather than net.core. Yes, that is awkward, given IPv6 support. But consistency is very important here. :-)
Quoted 17 lines
so it can be tuned for
optimal performance based on the application needs.Signed-off-by Haiyang Zhang <[email protected]>--- Documentation/admin-guide/sysctl/net.rst | 8 ++++++++ include/net/inet_connection_sock.h | 14 +++++++++++--- net/core/sysctl_net_core.c | 9 +++++++++ net/ipv4/tcp.c | 2 ++ net/ipv4/tcp_output.c | 17 +++++++++++++++-- 5 files changed, 45 insertions(+), 5 deletions(-)Documentation/admin-guide/sysctl/net.rst +0
-0
Documentation/admin-guide/sysctl/net.rst historical importance. · line 422
Quoted 10 lines
Documentation/admin-guide/sysctl/net.rst +7
-0@@ -413,6 +413,14 @@ historical importance. Default: 0 +tcp_pingpong_thresh +------------------- + +TCP pingpong threshold is 1 by default, but some application may need a higher +threshold for optimal performance. + +Default: 1, min: 1, max: 3
If we want to make this tunable, it seems sad to make the max 3. I'd suggest making the max 255, since we have 8 bits of space anyway in the inet_csk(sk)->icsk_ack.pingpong field.
Quoted 9 lines
Documentation/admin-guide/sysctl/net.rst +1
-0+ 2. /proc/sys/net/unix - Parameters for Unix domain sockets ----------------------------------------------------------include/net/inet_connection_sock.h +0
-0
include/net/inet_connection_sock.h void inet_csk_update_fastreuse(struct inet_bind_bucket *tb, · line 327
Quoted 5 lines
include/net/inet_connection_sock.h +1
-1@@ -324,11 +324,11 @@ void inet_csk_update_fastreuse(struct inet_bind_bucket *tb,struct dst_entry *inet_csk_update_pmtu (struct sock *sk,u32 mtu); -#define TCP_PINGPONG_THRESH 1 +extern int tcp_pingpong_thresh;
To match most TCP sysctls, this should be per-namespace, rather than global.
Please follow a recent example by Eric, perhaps:
65466904b015f6eeb9225b51aeb29b01a1d4b59c
tcp: adjust TSO packet sizes based on min_rtt
include/net/inet_connection_sock.h void inet_csk_update_fastreuse(struct inet_bind_bucket *tb, · line 332
Quoted 6 lines
include/net/inet_connection_sock.h +1
-1static inline void inet_csk_enter_pingpong_mode (struct sock *sk) { -inet_csk (sk)->icsk_ack.pingpong =TCP_PINGPONG_THRESH ; +inet_csk (sk)->icsk_ack.pingpong = tcp_pingpong_thresh; }
inet_csk(sk)->icsk_ack.pingpong = sock_net(sk)->sysctl_tcp_pingpong_thresh;
include/net/inet_connection_sock.h static inline void inet_csk_exit_pingpong_mode(struct sock *sk) · line 342
Quoted 8 lines
include/net/inet_connection_sock.h +2
-1static inline void inet_csk_exit_pingpong_mode (struct sock *sk)@@ -338,7 +338,15 @@ static inline void inet_csk_exit_pingpong_mode(struct sock *sk)static inline bool inet_csk_in_pingpong_mode (struct sock *sk) { -return inet_csk (sk)->icsk_ack.pingpong >=TCP_PINGPONG_THRESH ; +return inet_csk (sk)->icsk_ack.pingpong >= tcp_pingpong_thresh; +}
Again, sock_net(sk)->sysctl_tcp_pingpong_thresh rather than tcp_pingpong_thresh.
net/core/sysctl_net_core.c static struct ctl_table net_core_table[] = {
Quoted 14 lines
include/net/inet_connection_sock.h +6
-0+static inline void inet_csk_inc_pingpong_cnt (struct sock *sk) +{ +struct inet_connection_sock *icsk =inet_csk (sk); + +if (icsk->icsk_ack.pingpong <U8_MAX ) + icsk->icsk_ack.pingpong++; }static inline bool inet_csk_has_ulp (struct sock *sk)net/core/sysctl_net_core.c +0
-0@@ -653,6 +653,15 @@ static struct ctl_table net_core_table[] = {
Again, in net.ipv4, not net.core.
net/core/sysctl_net_core.c static struct ctl_table net_core_table[] = { · line 663
Quoted 11 lines
net/core/sysctl_net_core.c +8
-0.proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ZERO, }, + { + .procname = "tcp_pingpong_thresh", + .data = &tcp_pingpong_thresh, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ONE, + .extra2 = SYSCTL_THREE,
Please make the max U8_MAX to allow more flexibility (since we have 8 bits of space anyway in the inet_csk(sk)->icsk_ack.pingpong field).
Quoted 9 lines
net/core/sysctl_net_core.c +1
-0+ }, { } };net/ipv4/tcp.c +0
-0
net/ipv4/tcp.c EXPORT_SYMBOL(tcp_have_smc); · line 312
Quoted 5 lines
net/ipv4/tcp.c +2
-0@@ -308,6 +308,8 @@ EXPORT_SYMBOL(tcp_have_smc);struct percpu_counter tcp_sockets_allocated____cacheline_aligned_in_smp ;EXPORT_SYMBOL (tcp_sockets_allocated); +int tcp_pingpong_thresh__read_mostly =1 ; +
Again, per-network-namespace. You will need to initialize the per-netns value in tcp_sk_init(). Again, see Eric's 65466904b015f6eeb9225b51aeb29b01a1d4b59c commit for an example.
Quoted 11 lines
net/ipv4/tcp.c +0
-0* TCP splice context */ net/ipv4/tcp_output.c +1
-0@@ -167,12 +167,25 @@ static void tcp_event_data_sent(struct tcp_sock *tp,if (tcp_packets_in_flight (tp) ==0 )tcp_ca_event (sk,CA_EVENT_TX_START ); +/* If tcp_pingpong_thresh > 1, and
net/ipv4/tcp_output.c static void tcp_event_data_sent(struct tcp_sock *tp, · line 180
Quoted 10 lines
net/ipv4/tcp_output.c +10
-0+* this is the first data packet sent in response to the +* previous received data, +* and it is a reply for ato after last received packet, +* increase pingpong count. +*/ +if (tcp_pingpong_thresh >1 && +before (tp->lsndtime, icsk->icsk_ack.lrcvtime) && + (u32)(now - icsk->icsk_ack.lrcvtime) < icsk->icsk_ack.ato) +inet_csk_inc_pingpong_cnt (sk); +
Introducing this new code re-introduces a bug fixed in 4d8f24eeedc5. As that commit description noted:
This to-be-reverted commit was meant to apply a stricter rule for the stack to enter pingpong mode. However, the condition used to check for interactive session "before(tp->lsndtime, icsk->icsk_ack.lrcvtime)" is jiffy based and might be too coarse, which delays the stack entering pingpong mode.
We revert this patch so that we no longer use the above condition to determine interactive session,
net/ipv4/tcp_output.c static void tcp_event_data_sent(struct tcp_sock *tp, · line 183
Quoted 4 lines
net/ipv4/tcp_output.c +1
-1tp->lsndtime = now; -/* If it is a reply for ato after last received +/* If tcp_pingpong_thresh == 1, and
Please remove the "If tcp_pingpong_thresh == 1, and" part, since this is the correct code path no matter the value of the threshold.
net/ipv4/tcp_output.c static void tcp_event_data_sent(struct tcp_sock *tp, · line 187
Quoted 5 lines
net/ipv4/tcp_output.c +2
-1+* it is a reply for ato after last received * packet, enter pingpong mode. */ -if ((u32)(now - icsk->icsk_ack.lrcvtime) < icsk->icsk_ack.ato) + if (tcp_pingpong_thresh == 1 &&
Please remove the "if (tcp_pingpong_thresh == 1 &&" part, since this is the correct code path no matter the value of the threshold.
net/ipv4/tcp_output.c static void tcp_event_data_sent(struct tcp_sock *tp, · line 189
Quoted 2 lines
net/ipv4/tcp_output.c +1
-0+ (u32)(now - icsk->icsk_ack.lrcvtime) < icsk->icsk_ack.ato)inet_csk_enter_pingpong_mode (sk);
Please make this call inet_csk_inc_pingpong_cnt(), since this is the correct code path no matter the value of the threshold.
thanks,
neal
From: Neal Cardwell <[email protected]>
Date: Fri, 9 Jun 2023 15:16:00 -0400
Quoted 81 lines · Haiyang Zhang
Regarding the patch title:
Quoted 1 line
[PATCH net-next] tcp: Make pingpong threshold tunable
There are many ways to make something tunable these days, including BPF,
setsockopt(), and sysctl. :-) This patch only uses sysctl. Please consider a more clear/specific title, like:[PATCH net-next] tcp: set pingpong threshold via sysctl
Quoted 6 lines
TCP pingpong threshold is 1 by default. But some applications, like SQL DB may prefer a higher pingpong threshold to activate delayed acks in quick ack mode for better performance.
The pingpong threshold and related code were changed to 3 in the year 2019, and reverted to 1 in the year 2022.
Please include the specific commit, like:
The pingpong threshold and related code were changed to 3 in the year
2019 in:
commit4a41f453bedf("tcp: change pingpong threshold to 3")
and reverted to 1 in the year 2022 in:
commit4d8f24eeedc5("Revert "tcp: change pingpong threshold to 3"")Then please make sure to use
scripts/checkpatch.plon your resulting patch to check the formatting of the commit references, among other things.Quoted 4 lines
There is no single value that
fits all applications.Add net.core.tcp_pingpong_thresh sysctl tunable,
For consistency, TCP sysctls should be in net.ipv4 rather than net.core. Yes, that is awkward, given IPv6 support. But consistency is very important here. :-)
Quoted 17 lines
so it can be tuned for
optimal performance based on the application needs.Signed-off-by Haiyang Zhang <[email protected]>--- Documentation/admin-guide/sysctl/net.rst | 8 ++++++++ include/net/inet_connection_sock.h | 14 +++++++++++--- net/core/sysctl_net_core.c | 9 +++++++++ net/ipv4/tcp.c | 2 ++ net/ipv4/tcp_output.c | 17 +++++++++++++++-- 5 files changed, 45 insertions(+), 5 deletions(-)Documentation/admin-guide/sysctl/net.rst +0
-0Documentation/admin-guide/sysctl/net.rst historical importance. · line 422
Quoted 10 lines
Documentation/admin-guide/sysctl/net.rst +7
-0@@ -413,6 +413,14 @@ historical importance. Default: 0 +tcp_pingpong_thresh +------------------- + +TCP pingpong threshold is 1 by default, but some application may need a higher +threshold for optimal performance. + +Default: 1, min: 1, max: 3If we want to make this tunable, it seems sad to make the max 3. I'd suggest making the max 255, since we have 8 bits of space anyway in the inet_csk(sk)->icsk_ack.pingpong field.
Quoted 10 lines
Documentation/admin-guide/sysctl/net.rst +1
-0+ 2. /proc/sys/net/unix - Parameters for Unix domain sockets ----------------------------------------------------------include/net/inet_connection_sock.h +0
-0@@ -324,11 +324,11 @@ void inet_csk_update_fastreuse(struct inet_bind_bucket *tb,
Quoted 6 lines
include/net/inet_connection_sock.h void inet_csk_update_fastreuse(struct inet_bind_bucket *tb, · line 327
Quoted 4 lines
include/net/inet_connection_sock.h +1
-1struct dst_entry *inet_csk_update_pmtu (struct sock *sk,u32 mtu); -#define TCP_PINGPONG_THRESH 1 +extern int tcp_pingpong_thresh;To match most TCP sysctls, this should be per-namespace, rather than global.
Also, please change int to u8.
Quoted 7 lines
Please follow a recent example by Eric, perhaps:
65466904b015f6eeb9225b51aeb29b01a1d4b59c
tcp: adjust TSO packet sizes based onmin_rttQuoted 1 line
Quoted 7 lines
include/net/inet_connection_sock.h void inet_csk_update_fastreuse(struct inet_bind_bucket *tb, · line 332
Quoted 5 lines
static inline void
inet_csk_enter_pingpong_mode(struct sock *sk){include/net/inet_connection_sock.h +1
-1-inet_csk (sk)->icsk_ack.pingpong =TCP_PINGPONG_THRESH ; +inet_csk (sk)->icsk_ack.pingpong = tcp_pingpong_thresh; }inet_csk(sk)->icsk_ack.pingpong = sock_net(sk)->sysctl_tcp_pingpong_thresh;
Let's use READ_ONCE(sock_net(sk)->sysctl_tcp_pingpong_thresh). Same for other sysctl reads.
Quoted 41 lines
include/net/inet_connection_sock.h static inline void inet_csk_exit_pingpong_mode(struct sock *sk) · line 342
Quoted 9 lines
static inline void
inet_csk_exit_pingpong_mode(struct sock *sk)include/net/inet_connection_sock.h +2
-1@@ -338,7 +338,15 @@ static inline void inet_csk_exit_pingpong_mode(struct sock *sk)static inline bool inet_csk_in_pingpong_mode (struct sock *sk) { -return inet_csk (sk)->icsk_ack.pingpong >=TCP_PINGPONG_THRESH ; +return inet_csk (sk)->icsk_ack.pingpong >= tcp_pingpong_thresh; +}Again, sock_net(sk)->sysctl_tcp_pingpong_thresh rather than
tcp_pingpong_thresh.net/core/sysctl_net_core.c static struct ctl_table net_core_table[] = {
Quoted 14 lines
include/net/inet_connection_sock.h +6
-0+static inline void inet_csk_inc_pingpong_cnt (struct sock *sk) +{ +struct inet_connection_sock *icsk =inet_csk (sk); + +if (icsk->icsk_ack.pingpong <U8_MAX ) + icsk->icsk_ack.pingpong++; }static inline bool inet_csk_has_ulp (struct sock *sk)net/core/sysctl_net_core.c +0
-0@@ -653,6 +653,15 @@ static struct ctl_table net_core_table[] = {Again, in net.ipv4, not net.core.
net/core/sysctl_net_core.c static struct ctl_table net_core_table[] = { · line 663
Quoted 11 lines
net/core/sysctl_net_core.c +8
-0.proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ZERO, }, + { + .procname = "tcp_pingpong_thresh", + .data = &tcp_pingpong_thresh, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ONE, + .extra2 = SYSCTL_THREE,
Quoted 2 lines
Please make the max
U8_MAXto allow more flexibility (since we have 8 bits of space anyway in the inet_csk(sk)->icsk_ack.pingpong field).
Please use proc_dou8vec_minmax(), then you can drop .extra2.
.maxlen = sizeof(u8), .mode = 0644, .proc_handler = proc_dou8vec_minmax, .extra1 = SYSCTL_ONE,
Thanks,
Kuniyuki
Quoted 75 lines
Quoted 10 lines
net/core/sysctl_net_core.c +1
-0+ }, { } };net/ipv4/tcp.c +0
-0net/ipv4/tcp.c EXPORT_SYMBOL(tcp_have_smc); · line 312
Quoted 5 lines
net/ipv4/tcp.c +2
-0@@ -308,6 +308,8 @@ EXPORT_SYMBOL(tcp_have_smc);struct percpu_counter tcp_sockets_allocated____cacheline_aligned_in_smp ;EXPORT_SYMBOL (tcp_sockets_allocated); +int tcp_pingpong_thresh__read_mostly =1 ; +Again, per-network-namespace. You will need to initialize the per-netns value in
tcp_sk_init(). Again, see Eric's65466904b015f6eeb9225b51aeb29b01a1d4b59ccommit for an example.Quoted 11 lines
net/ipv4/tcp.c +0
-0* TCP splice context */ net/ipv4/tcp_output.c +1
-0@@ -167,12 +167,25 @@ static void tcp_event_data_sent(struct tcp_sock *tp,if (tcp_packets_in_flight (tp) ==0 )tcp_ca_event (sk,CA_EVENT_TX_START ); +/* If tcp_pingpong_thresh > 1, and net/ipv4/tcp_output.c static void tcp_event_data_sent(struct tcp_sock *tp, · line 180
Quoted 10 lines
net/ipv4/tcp_output.c +10
-0+* this is the first data packet sent in response to the +* previous received data, +* and it is a reply for ato after last received packet, +* increase pingpong count. +*/ +if (tcp_pingpong_thresh >1 && +before (tp->lsndtime, icsk->icsk_ack.lrcvtime) && + (u32)(now - icsk->icsk_ack.lrcvtime) < icsk->icsk_ack.ato) +inet_csk_inc_pingpong_cnt (sk); +Introducing this new code re-introduces a bug fixed in
4d8f24eeedc5. As that commit description noted:This to-be-reverted commit was meant to apply a stricter rule for the stack to enter pingpong mode. However, the condition used to check for interactive session "
before(tp->lsndtime, icsk->icsk_ack.lrcvtime)" is jiffy based and might be too coarse, which delays the stack entering pingpong mode.
We revert this patch so that we no longer use the above condition to determine interactive session,net/ipv4/tcp_output.c static void tcp_event_data_sent(struct tcp_sock *tp, · line 183
Quoted 4 lines
net/ipv4/tcp_output.c +1
-1tp->lsndtime = now; -/* If it is a reply for ato after last received +/* If tcp_pingpong_thresh == 1, and Please remove the "If
tcp_pingpong_thresh== 1, and" part, since this is the correct code path no matter the value of the threshold.net/ipv4/tcp_output.c static void tcp_event_data_sent(struct tcp_sock *tp, · line 187
Quoted 5 lines
net/ipv4/tcp_output.c +2
-1+* it is a reply for ato after last received * packet, enter pingpong mode. */ -if ((u32)(now - icsk->icsk_ack.lrcvtime) < icsk->icsk_ack.ato) + if (tcp_pingpong_thresh == 1 &&Please remove the "if (
tcp_pingpong_thresh== 1 &&" part, since this is the correct code path no matter the value of the threshold.net/ipv4/tcp_output.c static void tcp_event_data_sent(struct tcp_sock *tp, · line 189
Quoted 2 lines
net/ipv4/tcp_output.c +1
-0+ (u32)(now - icsk->icsk_ack.lrcvtime) < icsk->icsk_ack.ato)inet_csk_enter_pingpong_mode (sk);Please make this call
inet_csk_inc_pingpong_cnt(), since this is the correct code path no matter the value of the threshold.
Haiyang Zhang
maintainerQuoted 217 lines
-----Original Message-----
From: Kuniyuki Iwashima <[email protected]>
Sent: Friday, June 9, 2023 3:54 PM
To: Haiyang Zhang <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; KY Srinivasan <[email protected]>; [email protected]; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Tim Gardner <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH net-next] tcp: Make pingpong threshold tunable[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
From: Neal Cardwell <[email protected]>
Date: Fri, 9 Jun 2023 15:16:00 -0400Quoted 1 line
On Fri, Jun 9, 2023 at 12:26 PM Haiyang Zhang <[email protected]>
Quoted 50 lines · wrote:
Regarding the patch title:
Quoted 1 line
[PATCH net-next] tcp: Make pingpong threshold tunable
There are many ways to make something tunable these days, including BPF,
setsockopt(), and sysctl. :-) This patch only uses sysctl. Please consider a more clear/specific title, like:[PATCH net-next] tcp: set pingpong threshold via sysctl
Quoted 6 lines
TCP pingpong threshold is 1 by default. But some applications, like SQL DB may prefer a higher pingpong threshold to activate delayed acks in quick ack mode for better performance.
The pingpong threshold and related code were changed to 3 in the year 2019, and reverted to 1 in the year 2022.
Please include the specific commit, like:
The pingpong threshold and related code were changed to 3 in the year
2019 in:
commit4a41f453bedf("tcp: change pingpong threshold to 3")
and reverted to 1 in the year 2022 in:
commit4d8f24eeedc5("Revert "tcp: change pingpong threshold to 3"")Then please make sure to use
scripts/checkpatch.plon your resulting patch to check the formatting of the commit references, among other things.Quoted 4 lines
There is no single value that
fits all applications.Add net.core.tcp_pingpong_thresh sysctl tunable,
For consistency, TCP sysctls should be in net.ipv4 rather than net.core. Yes, that is awkward, given IPv6 support. But consistency is very important here. :-)
Quoted 12 lines
so it can be tuned for
optimal performance based on the application needs.Signed-off-by Haiyang Zhang <[email protected]>--- Documentation/admin-guide/sysctl/net.rst | 8 ++++++++ include/net/inet_connection_sock.h | 14 +++++++++++--- net/core/sysctl_net_core.c | 9 +++++++++ net/ipv4/tcp.c | 2 ++ net/ipv4/tcp_output.c | 17 +++++++++++++++-- 5 files changed, 45 insertions(+), 5 deletions(-)Quoted 1 line
Review
Quoted 1 line
(quoted diff) +0
-0
b/Documentation/admin-guide/sysctl/net.rstQuoted 11 lines
Documentation/admin-guide/sysctl/net.rst historical importance. · line 419
Quoted 11 lines
index
4877563241f3..16f54be9461f100644Documentation/admin-guide/sysctl/net.rst +4
-0@@ -413,6 +413,14 @@ historical importance. Default: 0 +tcp_pingpong_thresh +------------------- + +TCP pingpong threshold is 1 by default, but some application may need ahigher
Quoted 8 lines
Documentation/admin-guide/sysctl/net.rst historical importance. · line 422
Quoted 3 lines
Documentation/admin-guide/sysctl/net.rst +3
-0+threshold for optimal performance. + +Default: 1, min: 1, max: 3If we want to make this tunable, it seems sad to make the max 3. I'd suggest making the max 255, since we have 8 bits of space anyway in the inet_csk(sk)->icsk_ack.pingpong field.
Quoted 5 lines
Review
Quoted 5 lines
Documentation/admin-guide/sysctl/net.rst +1
-0+ 2. /proc/sys/net/unix - Parameters for Unix domain sockets ----------------------------------------------------------(quoted diff) +0
-0
b/include/net/inet_connection_sock.hQuoted 4 lines
include/net/inet_connection_sock.h void inet_csk_update_fastreuse(struct
Quoted 4 lines
index
c2b15f7e5516..e84e33ddae49100644include/net/inet_connection_sock.h +0
-0@@ -324,11 +324,11 @@ void inet_csk_update_fastreuse(struct
inet_bind_bucket*tb,Quoted 7 lines
include/net/inet_connection_sock.h void inet_csk_update_fastreuse(struct inet_bind_bucket *tb, · line 327
Quoted 5 lines
include/net/inet_connection_sock.h +1
-1struct dst_entry *inet_csk_update_pmtu (struct sock *sk,u32 mtu); -#define TCP_PINGPONG_THRESH 1 +extern int tcp_pingpong_thresh;To match most TCP sysctls, this should be per-namespace, rather than
global.
Also, please change int to u8.
Quoted 7 lines
Please follow a recent example by Eric, perhaps:
65466904b015f6eeb9225b51aeb29b01a1d4b59c
tcp: adjust TSO packet sizes based onmin_rttQuoted 1 line
Quoted 8 lines
include/net/inet_connection_sock.h void inet_csk_update_fastreuse(struct inet_bind_bucket *tb, · line 332
Quoted 5 lines
static inline void
inet_csk_enter_pingpong_mode(struct sock *sk){include/net/inet_connection_sock.h +1
-1-inet_csk (sk)->icsk_ack.pingpong =TCP_PINGPONG_THRESH ; +inet_csk (sk)->icsk_ack.pingpong = tcp_pingpong_thresh; }inet_csk(sk)->icsk_ack.pingpong = sock_net(sk)-
sysctl_tcp_pingpong_thresh;Let's use READ_ONCE(sock_net(sk)->sysctl_tcp_pingpong_thresh). Same for other sysctl reads.
Quoted 3 lines
include/net/inet_connection_sock.h static inline void
Quoted 3 lines
static inline void
inet_csk_exit_pingpong_mode(struct sock *sk)include/net/inet_connection_sock.h +0
-0@@ -338,7 +338,15 @@ static inline void
inet_csk_exit_pingpong_mode(struct sock *sk)Quoted 8 lines
include/net/inet_connection_sock.h static inline void inet_csk_exit_pingpong_mode(struct sock *sk) · line 342
Quoted 6 lines
include/net/inet_connection_sock.h +2
-1static inline bool inet_csk_in_pingpong_mode (struct sock *sk) { -return inet_csk (sk)->icsk_ack.pingpong >=TCP_PINGPONG_THRESH ; +return inet_csk (sk)->icsk_ack.pingpong >= tcp_pingpong_thresh; +}Again, sock_net(sk)->sysctl_tcp_pingpong_thresh rather than
tcp_pingpong_thresh.Quoted 30 lines
net/core/sysctl_net_core.c static struct ctl_table net_core_table[] = {
Quoted 15 lines
include/net/inet_connection_sock.h +6
-0+static inline void inet_csk_inc_pingpong_cnt (struct sock *sk) +{ +struct inet_connection_sock *icsk =inet_csk (sk); + +if (icsk->icsk_ack.pingpong <U8_MAX ) + icsk->icsk_ack.pingpong++; }static inline bool inet_csk_has_ulp (struct sock *sk)net/core/sysctl_net_core.c +0
-0@@ -653,6 +653,15 @@ static struct ctl_table net_core_table[] = {Again, in net.ipv4, not net.core.
net/core/sysctl_net_core.c static struct ctl_table net_core_table[] = { · line 663
Quoted 11 lines
net/core/sysctl_net_core.c +8
-0.proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ZERO, }, + { + .procname = "tcp_pingpong_thresh", + .data = &tcp_pingpong_thresh, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ONE, + .extra2 = SYSCTL_THREE,Quoted 2 lines
Please make the max
U8_MAXto allow more flexibility (since we have 8 bits of space anyway in the inet_csk(sk)->icsk_ack.pingpong field).Please use
proc_dou8vec_minmax(), then you can drop .extra2..maxlen = sizeof(u8), .mode = 0644, .proc_handler = proc_dou8vec_minmax, .extra1 = SYSCTL_ONE,Thanks,
KuniyukiQuoted 11 lines
net/ipv4/tcp.c EXPORT_SYMBOL(tcp_have_smc); · line 308
Quoted 11 lines
net/core/sysctl_net_core.c +1
-0+ }, { } };net/ipv4/tcp.c +0
-0@@ -308,6 +308,8 @@ EXPORT_SYMBOL(tcp_have_smc);struct percpu_counter tcp_sockets_allocated
____cacheline_aligned_in_smp;Quoted 9 lines
net/ipv4/tcp.c EXPORT_SYMBOL(tcp_have_smc); · line 312
Quoted 4 lines
net/ipv4/tcp.c +2
-0EXPORT_SYMBOL (tcp_sockets_allocated); +int tcp_pingpong_thresh__read_mostly =1 ; +Again, per-network-namespace. You will need to initialize the per-netns value in
tcp_sk_init(). Again, see Eric's65466904b015f6eeb9225b51aeb29b01a1d4b59ccommit for an example.
Quoted 56 lines
Quoted 7 lines
net/ipv4/tcp_output.c static void tcp_event_data_sent(struct
Quoted 7 lines
net/ipv4/tcp.c +0
-0* TCP splice context */ net/ipv4/tcp_output.c +0
-0@@ -167,12 +167,25 @@ static void tcp_event_data_sent(struct
tcp_sock*tp,Quoted 48 lines
Quoted 4 lines
net/ipv4/tcp_output.c +1
-0if (tcp_packets_in_flight (tp) ==0 )tcp_ca_event (sk,CA_EVENT_TX_START ); +/* If tcp_pingpong_thresh > 1, and net/ipv4/tcp_output.c static void tcp_event_data_sent(struct tcp_sock *tp, · line 180
Quoted 10 lines
net/ipv4/tcp_output.c +10
-0+* this is the first data packet sent in response to the +* previous received data, +* and it is a reply for ato after last received packet, +* increase pingpong count. +*/ +if (tcp_pingpong_thresh >1 && +before (tp->lsndtime, icsk->icsk_ack.lrcvtime) && + (u32)(now - icsk->icsk_ack.lrcvtime) < icsk->icsk_ack.ato) +inet_csk_inc_pingpong_cnt (sk); +Introducing this new code re-introduces a bug fixed in
4d8f24eeedc5. As that commit description noted:This to-be-reverted commit was meant to apply a stricter rule for the stack to enter pingpong mode. However, the condition used to check for interactive session "
before(tp->lsndtime, icsk->icsk_ack.lrcvtime)" is jiffy based and might be too coarse, which delays the stack entering pingpong mode.
We revert this patch so that we no longer use the above condition to determine interactive session,net/ipv4/tcp_output.c static void tcp_event_data_sent(struct tcp_sock *tp, · line 183
Quoted 4 lines
net/ipv4/tcp_output.c +1
-1tp->lsndtime = now; -/* If it is a reply for ato after last received +/* If tcp_pingpong_thresh == 1, and Please remove the "If
tcp_pingpong_thresh== 1, and" part, since this is the correct code path no matter the value of the threshold.net/ipv4/tcp_output.c static void tcp_event_data_sent(struct tcp_sock *tp, · line 187
Quoted 5 lines
net/ipv4/tcp_output.c +2
-1+* it is a reply for ato after last received * packet, enter pingpong mode. */ -if ((u32)(now - icsk->icsk_ack.lrcvtime) < icsk->icsk_ack.ato) + if (tcp_pingpong_thresh == 1 &&Please remove the "if (
tcp_pingpong_thresh== 1 &&" part, since this is the correct code path no matter the value of the threshold.net/ipv4/tcp_output.c static void tcp_event_data_sent(struct · line 189
Quoted 2 lines
net/ipv4/tcp_output.c +1
-0+ (u32)(now - icsk->icsk_ack.lrcvtime) < icsk->icsk_ack.ato)inet_csk_enter_pingpong_mode (sk);Please make this call
inet_csk_inc_pingpong_cnt(), since this is the correct code path no matter the value of the threshold.
Thank Neal and Kuniyuki for the detailed reviews! I will update the code accordingly.
- Haiyang