TCP: increase default initial receive window.
Subsystems (3)
PATCH 1/1 TCP: increase default initial receive window.
This patch changes the default initial receive window to 10 mss (defined constant). The default window is limited to the maximum of 10*1460 and 2*mss (when mss > 1460).
--- include/net/tcp.h | 3 +++ net/ipv4/tcp_output.c | 11 ++++++++--- 2 files changed, 11 insertions(+), 3 deletions(-)
include/net/tcp.h +3 -0
@@ -60,6 +60,9 @@ extern void tcp_time_wait(struct sock *sk, int state, int timeo);*/ #define MAX_TCP_WINDOW 32767U +/* Offer an initial receive window of 10 mss. */ +#define TCP_DEFAULT_INIT_RCVWND 10 +/* Minimal accepted MSS. It is (60+60+8) - (20+20). */ #define TCP_MIN_MSS 88U
net/ipv4/tcp_output.c +8 -3
@@ -228,10 +228,15 @@ void tcp_select_initial_window(int __space, __u32 mss, } } -/* Set initial window to value enough for senders, following RFC5681. */ +/* Set initial window to a value enough for senders starting with +* initial congestion window of TCP_DEFAULT_INIT_RCVWND. Place +* a limit on the initial window when mss is larger than 1460. +*/ if (mss > (1 << *rcv_wscale)) { -int init_cwnd =rfc3390_bytes_to_packets (mss); - +int init_cwnd =TCP_DEFAULT_INIT_RCVWND ; +if (mss >1460 ) + init_cwnd = +max_t (u32, (1460 *TCP_DEFAULT_INIT_RCVWND ) / mss,2 );/* when initializing use the value from init_rcv_wnd * rather than the default from above */
Signature · 1 line
1.7.3.1
Quoted 5 lines · On Fri, 17 Dec 2010 19:20:51 -0800 Nandita Dukkipati <[email protected]>…
This patch changes the default initial receive window to 10 mss (defined constant). The default window is limited to the maximum of 10*1460 and 2*mss (when mss > 1460).
Signed-off-by Nandita Dukkipati <[email protected]>
This needs way more discussion because it makes Linux non-RFC breaking the RFC behavior should require explicit user override. I also wonder if the magic value 10 should be a sysctl.
David Miller
maintainerFrom: Stephen Hemminger <[email protected]>
Date: Fri, 17 Dec 2010 19:59:17 -0800
Quoted 7 lines
Quoted 4 lines · On Fri, 17 Dec 2010 19:20:51 -0800 Nandita Dukkipati <[email protected]>…
This patch changes the default initial receive window to 10 mss (defined constant). The default window is limited to the maximum of 10*1460 and 2*mss (when mss > 1460).
Quoted 5 lines
Quoted 1 line
Signed-off-by Nandita Dukkipati <[email protected]>This needs way more discussion because it makes Linux non-RFC breaking the RFC behavior should require explicit user override. I also wonder if the magic value 10 should be a sysctl.
He's changing the default receive window not the transmit congestion window, so there is nothing RFC about this.
But yes he does have to explain himself why this change is being made, in exhaustive detail.
David Miller
maintainerFrom: Nandita Dukkipati <[email protected]>
Date: Fri, 17 Dec 2010 19:20:51 -0800
Quoted 5 lines
This patch changes the default initial receive window to 10 mss (defined constant). The default window is limited to the maximum of 10*1460 and 2*mss (when mss > 1460).
Signed-off-by Nandita Dukkipati <[email protected]>
That's an incredibly terse explanation for a very non-trivial change with very non-trivial implications.
What analysis have you performed to lead you to decide that this was a reasonable change to make? Where can people see that analysis and look over it to see if they agree with your assesment of the data?
We can't apply a patch like that without any form of analysis or reasoning.
You don't say "why" you're doing this, and frankly that really ticks me off.
resend in plain text.
Quoted 9 lines · David Miller
From: Nandita Dukkipati <[email protected]>
Date: Fri, 17 Dec 2010 19:20:51 -0800Quoted 5 lines
This patch changes the default initial receive window to 10 mss (defined constant). The default window is limited to the maximum of 10*1460 and 2*mss (when mss > 1460).
Signed-off-by Nandita Dukkipati <[email protected]>
Quoted 7 lines
That's an incredibly terse explanation for a very non-trivial change with very non-trivial implications.
What analysis have you performed to lead you to decide that this was a reasonable change to make? Where can people see that analysis and look over it to see if they agree with your assesment of the data?
Apologies for the terse comment. Here's a longer explanation.
Background:
A recent proposal to the IETF [Ref: 5] recommends increasing TCP's initial congestion window to 10 mss or about 15KB. This proposal, backed with data from several large-scale live experiments as well as controlled testbed experiments, is under active discussion in the TCPM working group.
Analysis performed:
Leading up to this proposal were several large-scale Internet experiments [Ref: 2] with an initial congestion window of 10 mss (IW10), where we showed that the average latency of HTTP responses improved by approximately 10%. This was accompanied by a slight increase in retransmission rate (0.5%), most of which is coming from applications opening multiple simultaneous connections. To understand the extreme
worst case scenarios, as well as fairness issues with IW10 versus IW3 traffic, we further conducted controlled testbed experiments (end-hosts are all Linux based). We came away finding minimal negative impact even under low link bandwidths (dial-ups) and small buffers [Ref: 3]. These results are extremely encouraging to adopting IW10.
But obviously, an initial congestion window of 10 mss is useless unless a TCP receiver advertises an initial receive window of at least 10 mss. Fortunately, in the large-scale Internet experiments we found that most of the operating systems advertised a large enough initial receive window, allowing us to experiment with various values of initial congestion windows. Linux systems were among the few exceptions that advertised a small receive window. This patch intends to fix that.
References:
1. This site has a comprehensive list of all IW10 references to date.
http://code.google.com/speed/protocols/tcpm-IW10.html
2. Paper describing results from large-scale Internet experiments with IW10.
http://ccr.sigcomm.org/drupal/?q=node/621
3. Controlled testbed experiments with IW10 under worst case scenarios
http://www.ietf.org/proceedings/79/slides/tcpm-0.pdf
4. Raw test data from testbed experiments (Linux senders/receivers) with initial congestion window and initial receive window of 10 mss.
http://research.csc.ncsu.edu/netsrv/?q=content/iw10
5. Internet-Draft. Increasing TCP's Initial Window.
https://datatracker.ietf.org/doc/draft-ietf-tcpm-initcwnd/
Quoted 63 lines · On Sat, 18 Dec 2010 01:08:33 -0800 Nandita Dukkipati <[email protected]>…
resend in plain text.
Quoted 9 lines · David Miller
From: Nandita Dukkipati <[email protected]>
Date: Fri, 17 Dec 2010 19:20:51 -0800Quoted 5 lines
This patch changes the default initial receive window to 10 mss (defined constant). The default window is limited to the maximum of 10*1460 and 2*mss (when mss > 1460).
Signed-off-by Nandita Dukkipati <[email protected]>Quoted 7 lines
That's an incredibly terse explanation for a very non-trivial change with very non-trivial implications.
What analysis have you performed to lead you to decide that this was a reasonable change to make? Where can people see that analysis and look over it to see if they agree with your assesment of the data?
Apologies for the terse comment. Here's a longer explanation.
Background:
A recent proposal to the IETF [Ref: 5] recommends increasing TCP's initial congestion window to 10 mss or about 15KB. This proposal, backed with data from several large-scale live experiments as well as controlled testbed experiments, is under active discussion in the TCPM working group.Analysis performed:
Leading up to this proposal were several large-scale Internet experiments [Ref: 2] with an initial congestion window of 10 mss (IW10), where we showed that the average latency of HTTP responses improved by approximately 10%. This was accompanied by a slight increase in retransmission rate (0.5%), most of which is coming from applications opening multiple simultaneous connections. To understand the extreme
worst case scenarios, as well as fairness issues with IW10 versus IW3 traffic, we further conducted controlled testbed experiments (end-hosts are all Linux based). We came away finding minimal negative impact even under low link bandwidths (dial-ups) and small buffers [Ref: 3]. These results are extremely encouraging to adopting IW10.But obviously, an initial congestion window of 10 mss is useless unless a TCP receiver advertises an initial receive window of at least 10 mss. Fortunately, in the large-scale Internet experiments we found that most of the operating systems advertised a large enough initial receive window, allowing us to experiment with various values of initial congestion windows. Linux systems were among the few exceptions that advertised a small receive window. This patch intends to fix that.
References:
1. This site has a comprehensive list of all IW10 references to date.
http://code.google.com/speed/protocols/tcpm-IW10.html2. Paper describing results from large-scale Internet experiments with IW10.
http://ccr.sigcomm.org/drupal/?q=node/6213. Controlled testbed experiments with IW10 under worst case scenarios
http://www.ietf.org/proceedings/79/slides/tcpm-0.pdf
Quoted 6 lines
4. Raw test data from testbed experiments (Linux senders/receivers) with initial congestion window and initial receive window of 10 mss.
http://research.csc.ncsu.edu/netsrv/?q=content/iw105. Internet-Draft. Increasing TCP's Initial Window.
https://datatracker.ietf.org/doc/draft-ietf-tcpm-initcwnd/
Agree this is a good idea, but some further notes:
* The control of receive window is a local function not covered by RFC.
* Linux manipulates receive window automatically, unlike some other implementations.
But any change to TCP risks breaking other broken implementations and users need a good way to recover. Therefore I recommend this be made a sysctl to allow for quick workaround for the user who has to connect to some Elbonian printer. Doing it per route is okay, but for the worst case, it needs to be a sysctl.
The default value of the sysctl should be your new value (10), and it should allow the old rfc usage if zero.
Quoted 8 lines
Agree this is a good idea, but some further notes:
* The control of receive window is a local function not covered by RFC.
* Linux manipulates receive window automatically, unlike some other implementations.But any change to TCP risks breaking other broken implementations and users need a good way to recover.
Always good to be careful, but break in what way? Many stacks have been advertising an initial receive window of well above 10 segments going back literally decades.
HP-UX systems have been advertising a default/initial recieve window of 32768 bytes since the mid 1990s, Solaris systems have been advertising a default receive window of 49152 for ages. I cannot speak to Windows' default advertised window. While that sound a bit like "But MOM! All my friends are doing it." it does seem to suggest that advertising an initial receive window of 10 segments is unlikely to uncover anything new.
rick jones
David Miller
maintainerFrom: Rick Jones <[email protected]>
Date: Mon, 20 Dec 2010 10:26:20 -0800
Quoted 11 lines
Quoted 7 lines
Agree this is a good idea, but some further notes:
* The control of receive window is a local function not covered by RFC.
* Linux manipulates receive window automatically, unlike some other implementations.
But any change to TCP risks breaking other broken implementations and users need a good way to recover.Always good to be careful, but break in what way? Many stacks have been advertising an initial receive window of well above 10 segments going back literally decades.
Agreed.
PATCH v2 1/1 TCP: increase default initial receive window.
In reply to Nandita DukkipatiThis patch changes the default initial receive window to 10 mss (defined constant). The default window is limited to the maximum of 10*1460 and 2*mss (when mss > 1460).
draft-ietf-tcpm-initcwnd-00 is a proposal to the IETF that recommends increasing TCP's initial congestion window to 10 mss or about 15KB. Leading up to this proposal were several large-scale live Internet experiments with an initial congestion window of 10 mss (IW10), where we showed that the average latency of HTTP responses improved by approximately 10%. This was accompanied by a slight increase in retransmission rate (0.5%), most of which is coming from applications opening multiple simultaneous connections. To understand the extreme worst case scenarios, and fairness issues (IW10 versus IW3), we further conducted controlled testbed experiments. We came away finding minimal negative impact even under low link bandwidths (dial-ups) and small buffers. These results are extremely encouraging to adopting IW10.
However, an initial congestion window of 10 mss is useless unless a TCP receiver advertises an initial receive window of at least 10 mss. Fortunately, in the large-scale Internet experiments we found that most widely used operating systems advertised large initial receive windows of 64KB, allowing us to experiment with a wide range of initial congestion windows. Linux systems were among the few exceptions that advertised a small receive window of 6KB. The purpose of this patch is to fix this shortcoming.
References:
1. A comprehensive list of all IW10 references to date.
http://code.google.com/speed/protocols/tcpm-IW10.html
2. Paper describing results from large-scale Internet experiments with IW10.
http://ccr.sigcomm.org/drupal/?q=node/621
3. Controlled testbed experiments under worst case scenarios and a fairness study.
http://www.ietf.org/proceedings/79/slides/tcpm-0.pdf
4. Raw test data from testbed experiments (Linux senders/receivers) with initial congestion and receive windows of both 10 mss.
http://research.csc.ncsu.edu/netsrv/?q=content/iw10
5. Internet-Draft. Increasing TCP's Initial Window.
https://datatracker.ietf.org/doc/draft-ietf-tcpm-initcwnd/
---
Changelog since v1:
- A longer and better commit message.
include/net/tcp.h | 3 +++ net/ipv4/tcp_output.c | 11 ++++++++--- 2 files changed, 11 insertions(+), 3 deletions(-)
include/net/tcp.h +3 -0
@@ -60,6 +60,9 @@ extern void tcp_time_wait(struct sock *sk, int state, int timeo);*/ #define MAX_TCP_WINDOW 32767U +/* Offer an initial receive window of 10 mss. */ +#define TCP_DEFAULT_INIT_RCVWND 10 +/* Minimal accepted MSS. It is (60+60+8) - (20+20). */ #define TCP_MIN_MSS 88U
net/ipv4/tcp_output.c +8 -3
@@ -228,10 +228,15 @@ void tcp_select_initial_window(int __space, __u32 mss, } } -/* Set initial window to value enough for senders, following RFC5681. */ +/* Set initial window to a value enough for senders starting with +* initial congestion window of TCP_DEFAULT_INIT_RCVWND. Place +* a limit on the initial window when mss is larger than 1460. +*/ if (mss > (1 << *rcv_wscale)) { -int init_cwnd =rfc3390_bytes_to_packets (mss); - +int init_cwnd =TCP_DEFAULT_INIT_RCVWND ; +if (mss >1460 ) + init_cwnd = +max_t (u32, (1460 *TCP_DEFAULT_INIT_RCVWND ) / mss,2 );/* when initializing use the value from init_rcv_wnd * rather than the default from above */
Signature · 1 line
1.7.3.1
As per the comments, I have resubmitted the patch with a longer explanation. Please let me know if the preference is to have a sysctl for the initial default receive window, as I am ok either way.
Thanks,
-Nandita
Quoted 94 lines · Nandita Dukkipati
This patch changes the default initial receive window to 10 mss (defined constant). The default window is limited to the maximum of 10*1460 and 2*mss (when mss > 1460).
draft-ietf-tcpm-initcwnd-00 is a proposal to the IETF that recommends increasing TCP's initial congestion window to 10 mss or about 15KB. Leading up to this proposal were several large-scale live Internet experiments with an initial congestion window of 10 mss (IW10), where we showed that the average latency of HTTP responses improved by approximately 10%. This was accompanied by a slight increase in retransmission rate (0.5%), most of which is coming from applications opening multiple simultaneous connections. To understand the extreme worst case scenarios, and fairness issues (IW10 versus IW3), we further conducted controlled testbed experiments. We came away finding minimal negative impact even under low link bandwidths (dial-ups) and small buffers. These results are extremely encouraging to adopting IW10.
However, an initial congestion window of 10 mss is useless unless a TCP receiver advertises an initial receive window of at least 10 mss. Fortunately, in the large-scale Internet experiments we found that most widely used operating systems advertised large initial receive windows of 64KB, allowing us to experiment with a wide range of initial congestion windows. Linux systems were among the few exceptions that advertised a small receive window of 6KB. The purpose of this patch is to fix this shortcoming.
References:
1. A comprehensive list of all IW10 references to date.
http://code.google.com/speed/protocols/tcpm-IW10.html2. Paper describing results from large-scale Internet experiments with IW10.
http://ccr.sigcomm.org/drupal/?q=node/6213. Controlled testbed experiments under worst case scenarios and a fairness study.
http://www.ietf.org/proceedings/79/slides/tcpm-0.pdf4. Raw test data from testbed experiments (Linux senders/receivers) with initial congestion and receive windows of both 10 mss.
http://research.csc.ncsu.edu/netsrv/?q=content/iw105. Internet-Draft. Increasing TCP's Initial Window.
https://datatracker.ietf.org/doc/draft-ietf-tcpm-initcwnd/Signed-off-by Nandita Dukkipati <[email protected]>---
Changelog since v1:
- A longer and better commit message.include/net/tcp.h | 3 +++
net/ipv4/tcp_output.c | 11 ++++++++---2 files changed, 11 insertions(+), 3 deletions(-)include/net/tcp.h +3
-0@@ -60,6 +60,9 @@ extern void tcp_time_wait(struct sock *sk, int state, int timeo);*/ #define MAX_TCP_WINDOW 32767U +/* Offer an initial receive window of 10 mss. */ +#define TCP_DEFAULT_INIT_RCVWND 10 + /* Minimal accepted MSS. It is (60+60+8) - (20+20). */ #define TCP_MIN_MSS 88Unet/ipv4/tcp_output.c +8
-4@@ -228,10 +228,15 @@ void tcp_select_initial_window(int __space, __u32 mss, } } - /* Set initial window to value enough for senders, following RFC5681. */ + /* Set initial window to a value enough for senders starting with + * initial congestion window of TCP_DEFAULT_INIT_RCVWND. Place + * a limit on the initial window when mss is larger than 1460. + */ if (mss > (1 << *rcv_wscale)) { - int init_cwnd = rfc3390_bytes_to_packets(mss); - + int init_cwnd = TCP_DEFAULT_INIT_RCVWND; + if (mss > 1460) + init_cwnd = + max_t(u32, (1460 * TCP_DEFAULT_INIT_RCVWND) / mss, 2); /* when initializing use the value from init_rcv_wnd* rather than the default from above */ --1.7.3.1
Quoted 37 lines · Nandita Dukkipati
This patch changes the default initial receive window to 10 mss (defined constant). The default window is limited to the maximum of 10*1460 and 2*mss (when mss > 1460).
draft-ietf-tcpm-initcwnd-00 is a proposal to the IETF that recommends increasing TCP's initial congestion window to 10 mss or about 15KB. Leading up to this proposal were several large-scale live Internet experiments with an initial congestion window of 10 mss (IW10), where we showed that the average latency of HTTP responses improved by approximately 10%. This was accompanied by a slight increase in retransmission rate (0.5%), most of which is coming from applications opening multiple simultaneous connections. To understand the extreme worst case scenarios, and fairness issues (IW10 versus IW3), we further conducted controlled testbed experiments. We came away finding minimal negative impact even under low link bandwidths (dial-ups) and small buffers. These results are extremely encouraging to adopting IW10.
However, an initial congestion window of 10 mss is useless unless a TCP receiver advertises an initial receive window of at least 10 mss. Fortunately, in the large-scale Internet experiments we found that most widely used operating systems advertised large initial receive windows of 64KB, allowing us to experiment with a wide range of initial congestion windows. Linux systems were among the few exceptions that advertised a small receive window of 6KB. The purpose of this patch is to fix this shortcoming.
References:
1. A comprehensive list of all IW10 references to date.
http://code.google.com/speed/protocols/tcpm-IW10.html2. Paper describing results from large-scale Internet experiments with IW10.
http://ccr.sigcomm.org/drupal/?q=node/6213. Controlled testbed experiments under worst case scenarios and a fairness study.
http://www.ietf.org/proceedings/79/slides/tcpm-0.pdf
Quoted 8 lines
4. Raw test data from testbed experiments (Linux senders/receivers) with initial congestion and receive windows of both 10 mss.
http://research.csc.ncsu.edu/netsrv/?q=content/iw105. Internet-Draft. Increasing TCP's Initial Window.
https://datatracker.ietf.org/doc/draft-ietf-tcpm-initcwnd/Signed-off-by Nandita Dukkipati <[email protected]>
From: Eric Dumazet <[email protected]>
Date: Tue, 21 Dec 2010 06:00:06 +0100
Quoted 4 lines · Nandita Dukkipati
Quoted 1 line
Signed-off-by Nandita Dukkipati <[email protected]>Acked-by Eric Dumazet <[email protected]>
Applied to net-next-2.6
I know this has already been applied, but one thing to think about:
Linux announces a small initial window to prevent overflowing the receive buffer when receiving segments smaller than the link MTU. Increasing this even to 10 segments might have some negative consequences. I recall, for instance, some drivers when configured with a 9000 byte MTU, have a single pool of receive buffers all 16k (the next highest power of 2). So each received segment will get 16k of allocated memory accounted to it, even if the incoming segments are <=1460 bytes long. The default initial rcvbuf of 87380 bytes is less than the 160k of memory that the initial window might consume, so we're going to start hitting the very slow path of coalescing segments to get back under memory bounds.
Some drivers are smarter about having multiple pools of receive buffers with different sizes, so it might not be so easy to hit this condition. I haven't looked at any of them for a while. Is this still a real concern?
Thanks,
-John
Quoted 55 lines · Nandita Dukkipati
This patch changes the default initial receive window to 10 mss (defined constant). The default window is limited to the maximum of 10*1460 and 2*mss (when mss > 1460).
Signed-off-by Nandita Dukkipati <[email protected]>---
include/net/tcp.h | 3 +++
net/ipv4/tcp_output.c | 11 ++++++++---2 files changed, 11 insertions(+), 3 deletions(-)include/net/tcp.h +3
-0@@ -60,6 +60,9 @@ extern void tcp_time_wait(struct sock *sk, int state, int timeo);*/ #define MAX_TCP_WINDOW 32767U +/* Offer an initial receive window of 10 mss. */ +#define TCP_DEFAULT_INIT_RCVWND 10 + /* Minimal accepted MSS. It is (60+60+8) - (20+20). */ #define TCP_MIN_MSS 88Unet/ipv4/tcp_output.c +8
-4@@ -228,10 +228,15 @@ void tcp_select_initial_window(int __space, __u32 mss, } } - /* Set initial window to value enough for senders, following RFC5681. */ + /* Set initial window to a value enough for senders starting with + * initial congestion window of TCP_DEFAULT_INIT_RCVWND. Place + * a limit on the initial window when mss is larger than 1460. + */ if (mss > (1 << *rcv_wscale)) { - int init_cwnd = rfc3390_bytes_to_packets(mss); - + int init_cwnd = TCP_DEFAULT_INIT_RCVWND; + if (mss > 1460) + init_cwnd = + max_t(u32, (1460 * TCP_DEFAULT_INIT_RCVWND) / mss, 2); /* when initializing use the value from init_rcv_wnd* rather than the default from above */ --1.7.3.1
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · John Heffner
I know this has already been applied, but one thing to think about:
Linux announces a small initial window to prevent overflowing the receive buffer when receiving segments smaller than the link MTU.
Overflowing receive buffer ? Which one ? Do you mean NIC RX ring buffer ?
Quoted 9 lines
Increasing this even to 10 segments might have some negative consequences. I recall, for instance, some drivers when configured with a 9000 byte MTU, have a single pool of receive buffers all 16k (the next highest power of 2). So each received segment will get 16k of allocated memory accounted to it, even if the incoming segments are <=1460 bytes long. The default initial rcvbuf of 87380 bytes is less than the 160k of memory that the initial window might consume, so we're going to start hitting the very slow path of coalescing segments to get back under memory bounds.
Patch is not allowing 87380 bytes, but 10 segments, limited to 14600 bytes. Its very conservative IMHO.
Quoted 5 lines
Some drivers are smarter about having multiple pools of receive buffers with different sizes, so it might not be so easy to hit this condition. I haven't looked at any of them for a while. Is this still a real concern?
I dont think so. You would have problem anyway, since the patch changes only _initial_ receive window. After some kbytes of data exchanged, window is probably larger.