tcp: update window_clamp together with scaling_ratio
Subsystems (3)
PATCH 1/1 tcp: update window_clamp together with scaling_ratio
After commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale"), we noticed an application-level timeout due to reduced throughput. This can be reproduced by the following minimal client and server program.
server:
int main(int argc, char *argv[]) {
int sockfd;
char buffer[256];
struct sockaddr_in srv_addr;
// Create socket
sockfd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if (sockfd < 0) {
perror("server: socket()");
return -1;
}
bzero((char *) &srv_addr, sizeof(srv_addr));
srv_addr.sin_family = AF_INET;
srv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
srv_addr.sin_port = htons(8080);
// Bind socket
if (bind(sockfd, (struct sockaddr *) &srv_addr,
sizeof(srv_addr)) < 0) {
perror("server: bind()");
close(sockfd);
return -1;
}
// Listen for connections
listen(sockfd,5);
while(1) {
int filefd = -1, newsockfd = -1;
struct sockaddr_in cli_addr;
socklen_t cli_len = sizeof(cli_addr);
// Accept connection
newsockfd = accept(sockfd, (struct sockaddr *)&cli_addr, &cli_len);
if (newsockfd < 0) {
perror("server: accept()");
goto end;
}
// Read filename from client
bzero(buffer, sizeof(buffer));
ssize_t n = read(newsockfd,buffer,sizeof(buffer)-1);
if (n < 0) {
perror("server: read()");
goto end;
}
// Open file
filefd = open(buffer, O_RDONLY);
if (filefd < 0) {
perror("server: read()");
goto end;
}
// Get file size
struct stat file_stat;
if(fstat(filefd, &file_stat) < 0) {
perror("server: fstat()");
goto end;
}
// Send file
off_t offset = 0;
ssize_t bytes_sent = 0, bytes_left = file_stat.st_size;
while ((bytes_sent = sendfile(newsockfd, filefd,
&offset, bytes_left)) > 0) {
bytes_left -= bytes_sent;
}end:
// Close file and client socket
if (filefd > 0) {
close(filefd);
}
if (newsockfd > 0) {
close(newsockfd);
}
}
close(sockfd);
return 0;
}client:
int main(int argc, char *argv[]) {
int sockfd, filefd;
char *server_addr = argv[1];
char *filename = argv[2];
struct sockaddr_in sockaddr;
char buffer[256];
ssize_t n;
if ((sockfd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)) == -1) {
perror("client: socket()");
return -1;
}
sockaddr.sin_family = AF_INET;
inet_pton(AF_INET, server_addr, &sockaddr.sin_addr);
sockaddr.sin_port = htons(8080);
int val = 65536;
if (setsockopt(sockfd, SOL_SOCKET, SO_RCVBUF,
&val, sizeof(val)) < 0) {
perror("client: setockopt(SO_RCVBUF)");
return -1;
}
if (connect(sockfd, (struct sockaddr*)&sockaddr,
sizeof(sockaddr)) == -1) {
close(sockfd);
perror("client: connect()");
return -1;
}
// Send filename to server
n = write(sockfd, filename, strlen(filename));
if (n < 0) {
perror("client: write()");
return -1;
}
// Open file
filefd = open(filename, O_WRONLY | O_CREAT, 0666);
if(filefd < 0) {
perror("client: open()");
return -1;
}
// Read file from server
while((n = read(sockfd, buffer, sizeof(buffer))) > 0) {
write(filefd, buffer, n);
}
// Close file and socket
close(filefd);
close(sockfd);
return 0;
}Before the commit, it takes around 22 seconds to transfer 10M data. After the commit, it takes 40 seconds. Because our application has a 30-second timeout, this regression broke the application.
The reason that it takes longer to transfer data is that tp->scaling_ratio is initialized to a value that results in ~0.25 of rcvbuf. In our case, SO_RCVBUF is set to 65536 by the application, which translates to 2 * 65536 = 131,072 bytes in rcvbuf and hence a ~28k initial receive window.
Later, even though the scaling_ratio is updated to a more accurate skb->len/skb->truesize, which is ~0.66 in our environment, the window stays at ~0.25 * rcvbuf. This is because tp->window_clamp does not change together with the tp->scaling_ratio update. As a result, the window size is capped at the initial window_clamp, which is also ~0.25 * rcvbuf, and never grows bigger.
This patch updates window_clamp along with scaling_ratio. It changes the calculation of the initial rcv_wscale as well to make sure the scale factor is also not capped by the initial window_clamp.
A comment from Tycho Andersen <[email protected]> is "What happens if someone has done setsockopt(sk, TCP_WINDOW_CLAMP) explicitly; will this and the above not violate userspace's desire to clamp the window size?". This comment is not addressed in this patch because the existing code also updates window_clamp at several places without checking if TCP_WINDOW_CLAMP is set by user space. Adding this check now may break certain user space assumption (similar to how the original patch broke the assumption of buffer overhead being 50%). For example, if a user space program sets TCP_WINDOW_CLAMP but the applicaiton behavior relies on window_clamp adjusted by the kernel as of today.
--- net/ipv4/tcp_input.c | 6 +++++- net/ipv4/tcp_output.c | 2 +- 2 files changed, 6 insertions(+), 2 deletions(-)
net/ipv4/tcp_input.c +5 -1
@@ -237,9 +237,13 @@ static void tcp_measure_rcv_mss(struct sock *sk, const struct sk_buff *skb)*/ if (unlikely (len != icsk->icsk_ack.rcv_mss)) {u64 val = (u64 )skb->len <<TCP_RMEM_TO_WIN_SCALE ; +struct tcp_sock *tp =tcp_sk (sk);do_div (val, skb->truesize); -tcp_sk (sk)->scaling_ratio = val ? val :1 ; + tp->scaling_ratio = val ? val :1 ; + +/* Make the window_clamp follow along. */ + tp->window_clamp =tcp_full_space (sk); } icsk->icsk_ack.rcv_mss = min_t(unsigned int, len,tcp_sk (sk)->advmss);
net/ipv4/tcp_output.c +1 -1
@@ -239,7 +239,7 @@ void tcp_select_initial_window(const struct sock *sk, int __space, __u32 mss,/* Set window scaling on max possible window */ space =max_t (u32, space,READ_ONCE (sock_net (sk)->ipv4.sysctl_tcp_rmem[2 ])); space =max_t (u32, space,READ_ONCE (sysctl_rmem_max)); - space =min_t (u32, space, *window_clamp); + space =min_t (u32, space, sk->sk_rcvbuf); *rcv_wscale =clamp_t (int,ilog2 (space) -15 ,0 ,TCP_MAX_WSCALE ); }
Signature · 1 line
2.34.1
Jakub Kicinski
maintainerQuoted 3 lines · On Tue, 2 Apr 2024 14:54:06 -0700 Hechao Li wrote:
After commit
dfa2f0483360("tcp: get rid ofsysctl_tcp_adv_win_scale"), we noticed an application-level timeout due to reduced throughput. This can be reproduced by the following minimal client and server program.
Hi Hechao, nice to e-meet you :)
I suspect Eric may say that SO_RCVBUF = 64k is not very reasonable. But I'll leave the technical review to him.
What I noticed is that our cryptic CI system appears to point at this change as breaking BPF tests:
https://netdev.bots.linux.dev/flakes.html?min-flip=0&pw-n=0&tn-needle=gh-bpf
We accumulate all outstanding patches and test together. BPF broke at net-next-2024-04-03--00-00, and:
$ cidiff origin/net-next-2024-04-02--21-00 \
origin/net-next-2024-04-03--00-00+tcp: update window_clamp together with scaling_ratio
+tools: ynl: ethtool.py: Output timestamping statistics from tsinfo-get operation +netlink: specs: ethtool: add header-flags enumeration
+net/mlx5e: Implement ethtool hardware timestamping statistics +net/mlx5e: Introduce timestamps statistic counter for Tx DMA layer +net/mlx5e: Introduce lost_cqe statistic counter for PTP Tx port timestamping CQ +ethtool: add interface to read Tx hardware timestamping statistics
The other patches are all driver stuff..
Here's the BPF CI output:
https://github.com/kernel-patches/bpf/actions/runs/8538300303
Quoted 7 lines · Hechao Li
After commit
dfa2f0483360("tcp: get rid ofsysctl_tcp_adv_win_scale"), we noticed an application-level timeout due to reduced throughput. This can be reproduced by the following minimal client and server program.server:
...
Quoted 10 lines
Before the commit, it takes around 22 seconds to transfer 10M data. After the commit, it takes 40 seconds. Because our application has a 30-second timeout, this regression broke the application.
The reason that it takes longer to transfer data is that
tp->scaling_ratiois initialized to a value that results in ~0.25 of rcvbuf. In our case,SO_RCVBUFis set to 65536 by the application, which translates to 2 * 65536 = 131,072 bytes in rcvbuf and hence a ~28k initial receive window.
What driver are you using, what MTU is set ?
If you get a 0.25 ratio, that is because a driver is oversizing rx skbs.
SO_RCVBUF 65536 would map indeed to 32768 bytes of payload.
Quoted 11 lines
Later, even though the
scaling_ratiois updated to a more accurate skb->len/skb->truesize, which is ~0.66 in our environment, the window stays at ~0.25 * rcvbuf. This is becausetp->window_clampdoes not change together with thetp->scaling_ratioupdate. As a result, the window size is capped at the initialwindow_clamp, which is also ~0.25 * rcvbuf, and never grows bigger.This patch updates
window_clampalong withscaling_ratio. It changes the calculation of the initialrcv_wscaleas well to make sure the scale factor is also not capped by the initialwindow_clamp.
This is very suspicious.
Quoted 11 lines
A comment from Tycho Andersen <[email protected]> is "What happens if someone has done
setsockopt(sk, TCP_WINDOW_CLAMP)explicitly; will this and the above not violate userspace's desire to clamp the window size?". This comment is not addressed in this patch because the existing code also updateswindow_clampat several places without checking ifTCP_WINDOW_CLAMPis set by user space. Adding this check now may break certain user space assumption (similar to how the original patch broke the assumption of buffer overhead being 50%). For example, if a user space program setsTCP_WINDOW_CLAMPbut the applicaiton behavior relies onwindow_clampadjusted by the kernel as of today.
Quite frankly I would prefer we increase tcp_rmem[] sysctls, instead of trying to accomodate
with too small SO_RCVBUF values.
This would benefit old applications that were written 20 years ago.
Quoted 21 lines · Eric Dumazet
Quoted 7 lines · Hechao Li
After commit
dfa2f0483360("tcp: get rid ofsysctl_tcp_adv_win_scale"), we noticed an application-level timeout due to reduced throughput. This can be reproduced by the following minimal client and server program.server:
...
Quoted 10 lines
Before the commit, it takes around 22 seconds to transfer 10M data. After the commit, it takes 40 seconds. Because our application has a 30-second timeout, this regression broke the application.
The reason that it takes longer to transfer data is that
tp->scaling_ratiois initialized to a value that results in ~0.25 of rcvbuf. In our case,SO_RCVBUFis set to 65536 by the application, which translates to 2 * 65536 = 131,072 bytes in rcvbuf and hence a ~28k initial receive window.
Quoted 13 lines
What driver are you using, what MTU is set ?
If you get a 0.25 ratio, that is because a driver is oversizing rx skbs.
SO_RCVBUF65536 would map indeed to 32768 bytes of payload.Quoted 7 lines
Later, even though the
scaling_ratiois updated to a more accurate skb->len/skb->truesize, which is ~0.66 in our environment, the window stays at ~0.25 * rcvbuf. This is becausetp->window_clampdoes not change together with thetp->scaling_ratioupdate. As a result, the window size is capped at the initialwindow_clamp, which is also ~0.25 * rcvbuf, and never grows bigger.
Sorry I missed this part. I understand better.
I wonder if we should at least test (sk->sk_userlocks & SOCK_RCVBUF_LOCK) or something...
For autotuned flows (majority of the cases), tp->window_clamp is changed from tcp_rcv_space_adjust()
I think we need to audit a bit more all tp->window_clamp changes.
Quoted 24 lines
Quoted 4 lines
This patch updates
window_clampalong withscaling_ratio. It changes the calculation of the initialrcv_wscaleas well to make sure the scale factor is also not capped by the initialwindow_clamp.This is very suspicious.
Quoted 11 lines
A comment from Tycho Andersen <[email protected]> is "What happens if someone has done
setsockopt(sk, TCP_WINDOW_CLAMP)explicitly; will this and the above not violate userspace's desire to clamp the window size?". This comment is not addressed in this patch because the existing code also updateswindow_clampat several places without checking ifTCP_WINDOW_CLAMPis set by user space. Adding this check now may break certain user space assumption (similar to how the original patch broke the assumption of buffer overhead being 50%). For example, if a user space program setsTCP_WINDOW_CLAMPbut the applicaiton behavior relies onwindow_clampadjusted by the kernel as of today.Quite frankly I would prefer we increase tcp_rmem[] sysctls, instead of trying to accomodate
with too smallSO_RCVBUFvalues.This would benefit old applications that were written 20 years ago.
Quoted 16 lines · Eric Dumazet
Quoted 15 lines · Eric Dumazet
Quoted 7 lines · Hechao Li
After commit
dfa2f0483360("tcp: get rid ofsysctl_tcp_adv_win_scale"), we noticed an application-level timeout due to reduced throughput. This can be reproduced by the following minimal client and server program.server:
...
Quoted 5 lines
Before the commit, it takes around 22 seconds to transfer 10M data. After the commit, it takes 40 seconds. Because our application has a 30-second timeout, this regression broke the application.
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
The reason that it takes longer to transfer data is that
tp->scaling_ratiois initialized to a value that results in ~0.25 of rcvbuf. In our case,SO_RCVBUFis set to 65536 by the application, which translates to 2 * 65536 = 131,072 bytes in rcvbuf and hence a ~28k initial receive window.What driver are you using, what MTU is set ?
The driver is AWS ENA driver. This is cross-region/internet traffic, so the MTU is 1500.
Quoted 5 lines
Quoted 5 lines
If you get a 0.25 ratio, that is because a driver is oversizing rx skbs.
SO_RCVBUF65536 would map indeed to 32768 bytes of payload.
The 0.25 ratio is the initial default ratio calculated using
#define TCP_DEFAULT_SCALING_RATIO ((1200 << TCP_RMEM_TO_WIN_SCALE) / \
SKB_TRUESIZE(4096))I think this is a constant 0.25, no?
Later with skb->len/skb->truesize, we get 0.66. However, the window can't grow to this ratio because window_clamp stays at the initial value, which is the initial tcp_full_space(sk), which is roughly 0.25 * rcvbuf.
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Later, even though the
scaling_ratiois updated to a more accurate skb->len/skb->truesize, which is ~0.66 in our environment, the window stays at ~0.25 * rcvbuf. This is becausetp->window_clampdoes not change together with thetp->scaling_ratioupdate. As a result, the window size is capped at the initialwindow_clamp, which is also ~0.25 * rcvbuf, and never grows bigger.
Quoted 4 lines
Sorry I missed this part. I understand better.
I wonder if we should at least test (
sk->sk_userlocks&SOCK_RCVBUF_LOCK) or something...
In our case, the application does set SOCK_RCVBUF_LOCK. But meanwhile, we also want the window_clamp to grow according to the ratio, so that the window can grow beyond the original 0.25 * rcvbuf.
Quoted 30 lines
For autotuned flows (majority of the cases),
tp->window_clampis changed fromtcp_rcv_space_adjust()I think we need to audit a bit more all
tp->window_clampchanges.Quoted 24 lines
Quoted 4 lines
This patch updates
window_clampalong withscaling_ratio. It changes the calculation of the initialrcv_wscaleas well to make sure the scale factor is also not capped by the initialwindow_clamp.This is very suspicious.
Quoted 11 lines
A comment from Tycho Andersen <[email protected]> is "What happens if someone has done
setsockopt(sk, TCP_WINDOW_CLAMP)explicitly; will this and the above not violate userspace's desire to clamp the window size?". This comment is not addressed in this patch because the existing code also updateswindow_clampat several places without checking ifTCP_WINDOW_CLAMPis set by user space. Adding this check now may break certain user space assumption (similar to how the original patch broke the assumption of buffer overhead being 50%). For example, if a user space program setsTCP_WINDOW_CLAMPbut the applicaiton behavior relies onwindow_clampadjusted by the kernel as of today.Quite frankly I would prefer we increase tcp_rmem[] sysctls, instead of trying to accomodate
with too smallSO_RCVBUFvalues.This would benefit old applications that were written 20 years ago.
The application is kafka and it has a default config of 64KB SO_RCVBUF
(https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#receive-buffer-bytes)
so in this case it's limitted by SO_RCVBUF and not tcp_rmem. It also has a default request timeout 30 seconds
(https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#request-timeout-ms)
The combination of these two configs requires the certain amount of app data (in our case 10M) to be transfer within 30 seconds. But a 32k window size can't achieve this, causing app timeout. Our goal was to upgrade the kernel without having to update applications if possible.
Quoted 35 lines · Hechao Li
Quoted 16 lines · Eric Dumazet
Quoted 15 lines · Eric Dumazet
Quoted 7 lines · Hechao Li
After commit
dfa2f0483360("tcp: get rid ofsysctl_tcp_adv_win_scale"), we noticed an application-level timeout due to reduced throughput. This can be reproduced by the following minimal client and server program.server:
...
Quoted 5 lines
Before the commit, it takes around 22 seconds to transfer 10M data. After the commit, it takes 40 seconds. Because our application has a 30-second timeout, this regression broke the application.
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
The reason that it takes longer to transfer data is that
tp->scaling_ratiois initialized to a value that results in ~0.25 of rcvbuf. In our case,SO_RCVBUFis set to 65536 by the application, which translates to 2 * 65536 = 131,072 bytes in rcvbuf and hence a ~28k initial receive window.What driver are you using, what MTU is set ?
The driver is AWS ENA driver. This is cross-region/internet traffic, so the MTU is 1500.
Quoted 5 lines
Quoted 5 lines
If you get a 0.25 ratio, that is because a driver is oversizing rx skbs.
SO_RCVBUF65536 would map indeed to 32768 bytes of payload.
Quoted 6 lines
The 0.25 ratio is the initial default ratio calculated using
#define TCP_DEFAULT_SCALING_RATIO ((1200 << TCP_RMEM_TO_WIN_SCALE) / \ SKB_TRUESIZE(4096))I think this is a constant 0.25, no?
This depends on skb metadata size, which changes over time.
With MAX_SKB_FRAGS == 17, this is .25390625
With MAX_SKB_FRAGS == 45, this is .234375
Quoted 5 lines
Later with skb->len/skb->truesize, we get 0.66. However, the window can't grow to this ratio because
window_clampstays at the initial value, which is the initialtcp_full_space(sk), which is roughly 0.25 * rcvbuf.
Sure. Please address Jakub feedback about tests.
Quoted 8 lines · Eric Dumazet
Quoted 6 lines
Later with skb->len/skb->truesize, we get 0.66. However, the window can't grow to this ratio because
window_clampstays at the initial value, which is the initialtcp_full_space(sk), which is roughly 0.25 * rcvbuf.Sure. Please address Jakub feedback about tests.
I think a less risky patch would be the following one.
If you agree, send a V2 of the patch.
Also remove from the changelog all the C code, this has no real value there. changelog should be not too small, not too big.
If you want to capture this test code, please cook a separate patch for net-next to add it in tools/testing/selftests/net
(But I guess iperf3 could be used instead, iperf3 is already used in tools/testing/selftests)
Thanks !
include/net/tcp.h +0 -0
100644
include/net/tcp.h +0 -0
@@ -1539,11 +1539,10 @@ static inline int tcp_space_from_win(const
struct sock *sk, int win)
return __tcp_space_from_win(tcp_sk(sk)->scaling_ratio, win);
}
-/* Assume a conservative default of 1200 bytes of payload per 4K page. +/* Assume a 50% default for skb->len/skb->truesize ratio.
* This may be adjusted later in tcp_measure_rcv_mss().
*/
-#define TCP_DEFAULT_SCALING_RATIO ((1200 << TCP_RMEM_TO_WIN_SCALE) / \
- SKB_TRUESIZE(4096))
+#define TCP_DEFAULT_SCALING_RATIO (1 << (TCP_RMEM_TO_WIN_SCALE - 1))
static inline void tcp_scaling_ratio_init(struct sock *sk) {
PATCH v2 1/1 tcp: increase the default TCP scaling ratio
After commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale"), we noticed an application-level timeout due to reduced throughput.
Before the commit, for a client that sets SO_RCVBUF to 65k, it takes around 22 seconds to transfer 10M data. After the commit, it takes 40 seconds. Because our application has a 30-second timeout, this regression broke the application.
The reason that it takes longer to transfer data is that tp->scaling_ratio is initialized to a value that results in ~0.25 of rcvbuf. In our case, SO_RCVBUF is set to 65536 by the application, which translates to 2 * 65536 = 131,072 bytes in rcvbuf and hence a ~28k initial receive window.
Later, even though the scaling_ratio is updated to a more accurate skb->len/skb->truesize, which is ~0.66 in our environment, the window stays at ~0.25 * rcvbuf. This is because tp->window_clamp does not change together with the tp->scaling_ratio update. As a result, the window size is capped at the initial window_clamp, which is also ~0.25 * rcvbuf, and never grows bigger.
This patch increases the initial scaling_ratio from ~25% to 50% in order to be backward compatible with the original default sysctl_tcp_adv_win_scale.
---v1->v2: increase the default tcp scaling ratio instead of updating window_clamp and update the commit message
--- include/net/tcp.h | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
include/net/tcp.h +2 -3
@@ -1539,11 +1539,10 @@ static inline int tcp_space_from_win(const struct sock *sk, int win)return __tcp_space_from_win (tcp_sk (sk)->scaling_ratio, win); } -/* Assume a conservative default of 1200 bytes of payload per 4K page. +/* Assume a 50% default for skb->len/skb->truesize ratio. * This may be adjusted later in tcp_measure_rcv_mss(). */ -#define TCP_DEFAULT_SCALING_RATIO ((1200 << TCP_RMEM_TO_WIN_SCALE) / \ - SKB_TRUESIZE(4096)) +#define TCP_DEFAULT_SCALING_RATIO (1 << (TCP_RMEM_TO_WIN_SCALE - 1))static inline void tcp_scaling_ratio_init (struct sock *sk) {
Signature · 1 line
2.34.1
Re: [PATCH net-next v2] tcp: increase the default TCP scaling ratio
Quoted 15 lines · Hechao Li
After commit
dfa2f0483360("tcp: get rid ofsysctl_tcp_adv_win_scale"), we noticed an application-level timeout due to reduced throughput.Before the commit, for a client that sets
SO_RCVBUFto 65k, it takes around 22 seconds to transfer 10M data. After the commit, it takes 40 seconds. Because our application has a 30-second timeout, this regression broke the application.The reason that it takes longer to transfer data is that
tp->scaling_ratiois initialized to a value that results in ~0.25 of rcvbuf. In our case,SO_RCVBUFis set to 65536 by the application, which translates to 2 * 65536 = 131,072 bytes in rcvbuf and hence a ~28k initial receive window.
Quoted 4 lines
Later, even though the
scaling_ratiois updated to a more accurate skb->len/skb->truesize, which is ~0.66 in our environment, the window stays at ~0.25 * rcvbuf. This is becausetp->window_clampdoes not change together with thetp->scaling_ratioupdate.
< when autotuning is disabled because of SO_RCVBUF >
Most modern applications let the kernel do autotuning, and benefit from the increased scaling_ratio.
As a result, the
Quoted 10 lines
window size is capped at the initial
window_clamp, which is also ~0.25 * rcvbuf, and never grows bigger.This patch increases the initial
scaling_ratiofrom ~25% to 50% in order to be backward compatible with the original defaultsysctl_tcp_adv_win_scale.Fixes dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale")Signed-off-by Hechao Li <[email protected]>
This tag is not standard, please use one line per reviewer. Also please include the link to V1, for further reference.
Thanks.
PATCH v3 1/1 tcp: increase the default TCP scaling ratio
After commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale"), we noticed an application-level timeout due to reduced throughput.
Before the commit, for a client that sets SO_RCVBUF to 65k, it takes around 22 seconds to transfer 10M data. After the commit, it takes 40 seconds. Because our application has a 30-second timeout, this regression broke the application.
The reason that it takes longer to transfer data is that tp->scaling_ratio is initialized to a value that results in ~0.25 of rcvbuf. In our case, SO_RCVBUF is set to 65536 by the application, which translates to 2 * 65536 = 131,072 bytes in rcvbuf and hence a ~28k initial receive window.
Later, even though the scaling_ratio is updated to a more accurate skb->len/skb->truesize, which is ~0.66 in our environment, the window stays at ~0.25 * rcvbuf. This is because tp->window_clamp does not change together with the tp->scaling_ratio update when autotuning is disabled due to SO_RCVBUF. As a result, the window size is capped at the initial window_clamp, which is also ~0.25 * rcvbuf, and never grows bigger.
Most modern applications let the kernel do autotuning, and benefit from the increased scaling_ratio. But there are applications such as kafka that has a default setting of SO_RCVBUF=64k.
This patch increases the initial scaling_ratio from ~25% to 50% in order to make it backward compatible with the original default sysctl_tcp_adv_win_scale for applications setting SO_RCVBUF.
---v1->v2: increase the default tcp scaling ratio instead of updating window_clamp and update the commit messagev2->v3: Update commit message and add the link to v1
--- include/net/tcp.h | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
include/net/tcp.h +2 -3
@@ -1539,11 +1539,10 @@ static inline int tcp_space_from_win(const struct sock *sk, int win)return __tcp_space_from_win (tcp_sk (sk)->scaling_ratio, win); } -/* Assume a conservative default of 1200 bytes of payload per 4K page. +/* Assume a 50% default for skb->len/skb->truesize ratio. * This may be adjusted later in tcp_measure_rcv_mss(). */ -#define TCP_DEFAULT_SCALING_RATIO ((1200 << TCP_RMEM_TO_WIN_SCALE) / \ - SKB_TRUESIZE(4096)) +#define TCP_DEFAULT_SCALING_RATIO (1 << (TCP_RMEM_TO_WIN_SCALE - 1))static inline void tcp_scaling_ratio_init (struct sock *sk) {
Signature · 1 line
2.34.1
Re: [PATCH net-next v3] tcp: increase the default TCP scaling ratio
This patch was applied to netdev/net-next.git (main) by David S. Miller <[email protected]>:
Hello:
This patch was applied to netdev/net-next.git (main) by David S. Miller <[email protected]>:
Quoted 9 lines · On Tue, 9 Apr 2024 09:43:55 -0700 you wrote:
After commit
dfa2f0483360("tcp: get rid ofsysctl_tcp_adv_win_scale"), we noticed an application-level timeout due to reduced throughput.Before the commit, for a client that sets
SO_RCVBUFto 65k, it takes around 22 seconds to transfer 10M data. After the commit, it takes 40 seconds. Because our application has a 30-second timeout, this regression broke the application.[...]
Here is the summary with links:
- [net-next,v3] tcp: increase the default TCP scaling ratio
https://git.kernel.org/netdev/net-next/c/697a6c8cec03
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Neal Cardwell
maintainerQuoted 9 lines · Hechao Li
The application is kafka and it has a default config of 64KB
SO_RCVBUF
(https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#receive-buffer-bytes)
so in this case it's limitted bySO_RCVBUFand nottcp_rmem. It also has a default request timeout 30 seconds
(https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#request-timeout-ms)
The combination of these two configs requires the certain amount of app data (in our case 10M) to be transfer within 30 seconds. But a 32k window size can't achieve this, causing app timeout. Our goal was to upgrade the kernel without having to update applications if possible.
Hechao, can you please file a bug against Kafka to get them to stop using SO_RCVBUF, and allow receive buffer autotuning? This default value of 64 Kbytes will cripple performance in many scenarios, especially for WAN traffic.
I guess that would boil down to asking for the default receive.buffer.bytes to be -1 rather than 64 Kbytes.
Looks like you can file bugs here:
https://issues.apache.org/jira/browse/KAFKA
thanks,
neal
Quoted 16 lines · Neal Cardwell
Quoted 9 lines · Hechao Li
The application is kafka and it has a default config of 64KB
SO_RCVBUF
(https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#receive-buffer-bytes)
so in this case it's limitted bySO_RCVBUFand nottcp_rmem. It also has a default request timeout 30 seconds
(https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#request-timeout-ms)
The combination of these two configs requires the certain amount of app data (in our case 10M) to be transfer within 30 seconds. But a 32k window size can't achieve this, causing app timeout. Our goal was to upgrade the kernel without having to update applications if possible.Hechao, can you please file a bug against Kafka to get them to stop using
SO_RCVBUF, and allow receive buffer autotuning? This default value of 64 Kbytes will cripple performance in many scenarios, especially for WAN traffic.
Quoted 8 lines
I guess that would boil down to asking for the default receive.buffer.bytes to be -1 rather than 64 Kbytes.
Looks like you can file bugs here:
https://issues.apache.org/jira/browse/KAFKAthanks,
neal
Makes sense. Filed: https://issues.apache.org/jira/browse/KAFKA-16496
I don't know the reason why kafka set 64k as the default in the first place. But I was wondering if there is any use case that requires the user to set SO_RCVBUF rather than auto tuning? Eventually do we want to depreacte the support of setting SO_RCVBUF at all?
Thank you.
Hechao