3 packet TCP window limit?
Subsystems (2)
Hey,
Noticed in Linux that no matter what sysctl variable I twiddle, or what TCP congestion algorithm is running, TCP will wait for remote acks after sending the first 3 packets. After that it's normal.
Apologies, it's hard ot describe:
Linux server listening.
Remote -> SYN
(RTT wait)
Linux -> SYN/ACK
Remote -> ACK
Remote -> Packet (small HTTP request)
(RTT wait)
Linux -> Packet (x 3)
Remote -> (returning acks per packet)
(RTT wait)
Linux -> More packets (up to window size)
If the request response fits in 3 packets or less, that third RTT wait never happens. The remote client gets all its data, and sends back all the FIN/ACK packets for closing the connection.
What's bizarre is that this 3 packet/4 packet barrier is regardless of how much data there is to send. I can cause the extra RTT to flip on or off by sending exactly +/- 1 byte to cause an extra packet.
Holding the connection open and repeating the request any number of times runs just fine, after the initial request.
You can pretty easily see this by:
tc qdisc add dev eth0 root netem delay 100ms
... then fetching a 3k file, then 4k file from an http server running linux. Well. at least I can see this easily. I tried on a half dozen boxes (2.6.11 through 2.6.32).
I'm trying to track down where in the code this is, or why my sysctl tuning isn't affecting it. I can't discern its purpose. The lag it causes is pretty awful for far away clients; adding 300ms of latency will make a small request take a full second, instead of 600ms.
I'm slugging through the code but any insight would be greatly appreciated!
-Dormando
Quoted 38 lines · dormando
Hey,
Noticed in Linux that no matter what sysctl variable I twiddle, or what TCP congestion algorithm is running, TCP will wait for remote acks after sending the first 3 packets. After that it's normal.
Apologies, it's hard ot describe:
Linux server listening.
Remote -> SYN
(RTT wait)
Linux -> SYN/ACK
Remote -> ACK
Remote -> Packet (small HTTP request)
(RTT wait)
Linux -> Packet (x 3)
Remote -> (returning acks per packet)
(RTT wait)
Linux -> More packets (up to window size)If the request response fits in 3 packets or less, that third RTT wait never happens. The remote client gets all its data, and sends back all the FIN/ACK packets for closing the connection.
What's bizarre is that this 3 packet/4 packet barrier is regardless of how much data there is to send. I can cause the extra RTT to flip on or off by sending exactly +/- 1 byte to cause an extra packet.
Holding the connection open and repeating the request any number of times runs just fine, after the initial request.
You can pretty easily see this by:
tc qdisc add dev eth0 root netem delay 100ms
... then fetching a 3k file, then 4k file from an http server running linux. Well. at least I can see this easily. I tried on a half dozen boxes (2.6.11 through 2.6.32).
Quoted 7 lines
I'm trying to track down where in the code this is, or why my sysctl tuning isn't affecting it. I can't discern its purpose. The lag it causes is pretty awful for far away clients; adding 300ms of latency will make a small request take a full second, instead of 600ms.
I'm slugging through the code but any insight would be greatly appreciated!
This sounds like TCP slow start.
http://en.wikipedia.org/wiki/Slow-start
As far as tunables you might want to play with the initcwnd route flag (see "ip route help")
Quoted 7 lines
-Dormando
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines
This sounds like TCP slow start.
http://en.wikipedia.org/wiki/Slow-start
As far as tunables you might want to play with the initcwnd route flag (see "ip route help")
Ah, yes, initcwnd was it. I'm well aware of TCP Congestion control / slow start / etc. However I couldn't find the damn tunable for it :) ssthresh/tso/etc didn't seem to unwedge it. Felt like describing it in the most generic way possible would help :)
Other OS's appear to have a larger initcwnd. As do commercial load balancers. The default of 3 seems to be tuned for 56k dialup modems. I'm a little surprised that none of the pluggable TCP congestion control algorithms changed this value. I went through all of them except for tcp_yeah.
Anyway, thanks and sorry for the nearly off-topic post here. I see some google papers on bumping initcwnd to 10... but I guess that's not linux's deal yet.
Quoted 9 lines · dormando
Quoted 6 lines
This sounds like TCP slow start.
http://en.wikipedia.org/wiki/Slow-start
As far as tunables you might want to play with the initcwnd route flag (see "ip route help")
Ah, yes, initcwnd was it. I'm well aware of TCP Congestion control / slow start / etc. However I couldn't find the damn tunable for it :)
I don't believe linux as yet has a damn tunable for it :)
Quoted 1 line
ssthresh/tso/etc didn't seem to unwedge it.
If they did, it would be a bug. In fact there *was* a bug "way back when" where TSO being enabled caused the stack to ignore initcwd, but that was fixed circa 2.6.14. Until it was fixed (it was difficult to notice unless one was speaking to a non-Linux reciever, since Linux receivers autotune the receive window) it did some very nice things for SPECweb benchmark results :)
Quoted 3 lines
Felt like describing it in the most generic way possible would help :)
Other OS's appear to have a larger initcwnd.
Names? Values?
Quoted 1 line
As do commercial load balancers.
Names? Values?
Quoted 4 lines
The default of 3 seems to be tuned for 56k dialup modems. I'm a little surprised that none of the pluggable TCP congestion control algorithms changed this value. I went through all of them except for
tcp_yeah.
The initcwnd comes from IETF RFCs and their "thou shalts" and "thou shalt nots."
As you note below, Google et al seek to alter/extend the RFCs. That is an
ongoing discussion in some of the ietf related mailing lists.
rick jones
Quoted 7 lines
Anyway, thanks and sorry for the nearly off-topic post here. I see some google papers on bumping initcwnd to 10... but I guess that's not linux's deal yet.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line
I don't believe linux as yet has a damn tunable for it :)
ip route initcwnd sure does it :)
Quoted 3 lines
Quoted 1 line
Other OS's appear to have a larger initcwnd.
Names? Values?
OpenBSD 4.6 jumps between a ~5k fetch to a ~6k fetch
Quoted 3 lines
Quoted 1 line
As do commercial load balancers.
Names? Values?
An older Big/IP appears to be between 5k and 6k as well. I remember a sales meeting with netscaler (pre-NDA) back in 2004 or 2005 where they claimed to have opened up slow start. There might be others but I can't remember which side of the NDA I was informed of their TCP tunning.
Linux is consistently between 3k and 4k. Just the distinction from having the RTT in the ~4k or the ~6k range makes our latency graphs go nutty. I've been testing a subset of traffic at an initcwnd of 10 for the last few hours and latency has dropped even more, though I see some bad outliers.
Quoted 8 lines
Quoted 4 lines
The default of 3 seems to be tuned for 56k dialup modems. I'm a little surprised that none of the pluggable TCP congestion control algorithms changed this value. I went through all of them except for
tcp_yeah.The initcwnd comes from IETF RFCs and their "thou shalts" and "thou shalt nots." As you note below, Google et al seek to alter/extend the RFCs. That is an ongoing discussion in some of the ietf related mailing lists.
The RFC clearly states "around 4k", but these other OS's/products have an extra kilobyte snuck in? Could this be on purpose via rfc interpretation, or an off by one on the initcwnd estimator? :)
Hi,
Quoted 1 line · dormando
The RFC clearly states "around 4k",
no, it doesn't. RFC3390 gives a very precise formula for calculating the initial window:
min (4*MSS, max (2*MSS, 4380 bytes))
Please see the RFC for why. More reading at http://www.icir.org/floyd/tcp_init_win.html I believe that Linux implements behavior this pretty faithfully.
Quoted 3 lines
but these other OS's/products have an
extra kilobyte snuck in? Could this be on purpose via rfc interpretation, or an off by one on the initcwnd estimator? :)
I'm surprised to hear that OpenBSD doesn't follow the RFC. Can you share a measurement? Are you sure the box you are measuring is using the default configuration?
I don't think the RFC can be misread (it's pretty clear), and the formula is also not exactly complicated. My guess would be that some vendors have convinced themselves that using a slightly larger value is OK, esp. if they can show customers that "their" TCP is "faster" than some competitors' TCPs. An arms race between vendors in this space would really not be good for anyone - it's clear that at some point, problems due to overshoot will occur.
(We can definitely argue about whether the current RFC-recommended value is too low, and Google and others are gathering data in support of making a convincing and backed-up argument for increasing the initial window to the IETF. Which is exactly the correct way of going about this.)
Lars
Quoted 8 lines · dormando
Quoted 1 line
The RFC clearly states "around 4k",
no, it doesn't. RFC3390 gives a very precise formula for calculating the initial window:
min (4*MSS, max (2*MSS, 4380 bytes))Please see the RFC for why. More reading at http://www.icir.org/floyd/tcp_init_win.html I believe that Linux implements behavior this pretty faithfully.
Sorry, paraphrasing :) Web nerds have been working around this for a long time now. Google talks about using HTTP chunked encoding responses to send an initial "frame" of a webpage in under 3 packets. Which immediately gives the browser something to render and primes the TCP connection for more web junk.
Quoted 1 line
I'm surprised to hear that OpenBSD doesn't follow the RFC. Can you share a measurement? Are you sure the box you are measuring is using the default configuration?
Yeah, default config. OBSD was giving me back 4 packets in the first window, while linux always gives back 3. The Big/IP is based on linux 2.4.21. If that kernel didn't have it wrong, they tuned it.
Already nuked my dumps. If you're curious I'll re-create.
Quoted 7 lines
I don't think the RFC can be misread (it's pretty clear), and the formula is also not exactly complicated. My guess would be that some vendors have convinced themselves that using a slightly larger value is OK, esp. if they can show customers that "their" TCP is "faster" than some competitors' TCPs. An arms race between vendors in this space would really not be good for anyone - it's clear that at some point, problems due to overshoot will occur.
I clearly remember some vendors bragging about doing this. That was a long time ago? Perhaps they stopped? If it's true they've been doing it for half a decade or more, and haven't broken anything someone would notice.
The only reason why I set about tuning this is because our latency jumped while moving traffic from a commercial machine to a linux machine, and I had to figure out what they changed to do that. I've since turned the setting *back* to the standard, having confirmed what they did.
Almost tempted to test this against a bunch of websites...
Quoted 5 lines
(We can definitely argue about whether the current RFC-recommended value is too low, and Google and others are gathering data in support of making a convincing and backed-up argument for increasing the initial window to the IETF. Which is exactly the correct way of going about this.)
This sounds like fun. We have some diverse traffic, so I'm hoping we can contribute to that conversation. Still have a lot of reading to catch up with first :)
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
From: dormando <[email protected]>
Quoted 56 lines
Date: Thu, May 6, 2010 at 1:51 AM
Subject: Re: 3 packet TCP window limit?
To: Lars Eggert <[email protected]>
Cc: Rick Jones <[email protected]>, Brian Bloniarz <[email protected]>, "[email protected]" <[email protected]>Quoted 8 lines · dormando
Quoted 1 line
The RFC clearly states "around 4k",
no, it doesn't. RFC3390 gives a very precise formula for calculating the initial window:
min (4*MSS, max (2*MSS, 4380 bytes))
Please see the RFC for why. More reading at http://www.icir.org/floyd/tcp_init_win.html I believe that Linux implements behavior this pretty faithfully.
Sorry, paraphrasing :) Web nerds have been working around this for a long time now. Google talks about using HTTP chunked encoding responses to send an initial "frame" of a webpage in under 3 packets. Which immediately gives the browser something to render and primes the TCP connection for more web junk.
Quoted 1 line
I'm surprised to hear that OpenBSD doesn't follow the RFC. Can you share a measurement? Are you sure the box you are measuring is using the default configuration?
Yeah, default config. OBSD was giving me back 4 packets in the first window, while linux always gives back 3. The Big/IP is based on linux 2.4.21. If that kernel didn't have it wrong, they tuned it.
Already nuked my dumps. If you're curious I'll re-create.
Quoted 7 lines
I don't think the RFC can be misread (it's pretty clear), and the formula is also not exactly complicated. My guess would be that some vendors have convinced themselves that using a slightly larger value is OK, esp. if they can show customers that "their" TCP is "faster" than some competitors' TCPs. An arms race between vendors in this space would really not be good for anyone - it's clear that at some point, problems due to overshoot will occur.
I clearly remember some vendors bragging about doing this. That was a long time ago? Perhaps they stopped? If it's true they've been doing it for half a decade or more, and haven't broken anything someone would notice.
The only reason why I set about tuning this is because our latency jumped while moving traffic from a commercial machine to a linux machine, and I had to figure out what they changed to do that. I've since turned the setting *back* to the standard, having confirmed what they did.
Almost tempted to test this against a bunch of websites...
Quoted 5 lines
(We can definitely argue about whether the current RFC-recommended value is too low, and Google and others are gathering data in support of making a convincing and backed-up argument for increasing the initial window to the IETF. Which is exactly the correct way of going about this.)
Quoted 3 lines
This sounds like fun. We have some diverse traffic, so I'm hoping we can contribute to that conversation. Still have a lot of reading to catch up with first :)
Yes please do. Our presentation at Anaheim IETF can be found at
http://www.ietf.org/proceedings/10mar/slides/tcpm-4.pdf, with a paper describing the details of our experiments at
http://code.google.com/speed/articles/tcp_initcwnd_paper.pdf.
We've gotten a lot of feedback from IETF and are planning to collect more data to
justify the proposal. But at this point we really need help from others as the scope of
the work is certainly not a one-company job. Help can be in the form of more experiments/tests and/or simulations to study the effect of a larger initcwnd. Please
contact me directly or send your data to IETF's TCPM WG list
(http://www.ietf.org/mail-archive/web/tcpm/current/maillist.html).
Thanks,
Jerry
Quoted 9 lines · dormando
Quoted 6 lines
This sounds like TCP slow start.
http://en.wikipedia.org/wiki/Slow-start
As far as tunables you might want to play with the initcwnd route flag (see "ip route help")
Ah, yes, initcwnd was it. I'm well aware of TCP Congestion control / slow start / etc. However I couldn't find the damn tunable for it :)
Documenting the flag in ip(8) might increase its visibility a little. I don't see it documented in the iproute2 git head, though it shows up on http://linux.die.net/man/8/ip somehow.
Stephen, do you know why that is?
Quoted 8 lines · On Wed, 05 May 2010 16:56:34 -0400 Brian Bloniarz <[email protected]> wrote:
Quoted 7 lines · dormando
Quoted 6 lines
This sounds like TCP slow start.
http://en.wikipedia.org/wiki/Slow-start
As far as tunables you might want to play with the initcwnd route flag (see "ip route help")
Quoted 8 lines
Quoted 2 lines
Ah, yes, initcwnd was it. I'm well aware of TCP Congestion control / slow start / etc. However I couldn't find the damn tunable for it :)
Documenting the flag in ip(8) might increase its visibility a little. I don't see it documented in the iproute2 git head, though it shows up on http://linux.die.net/man/8/ip somehow.
Stephen, do you know why that is?
No one sent me an official patch to change it?
PATCH 1/1 document initcwnd
Quoted 12 lines · Stephen Hemminger
Quoted 9 lines · On Wed, 05 May 2010 16:56:34 -0400 Brian Bloniarz <[email protected]> wrote:
Quoted 8 lines · dormando
Quoted 6 lines
This sounds like TCP slow start.
http://en.wikipedia.org/wiki/Slow-start
As far as tunables you might want to play with the initcwnd route flag (see "ip route help")
Ah, yes, initcwnd was it. I'm well aware of TCP Congestion control / slow start / etc. However I couldn't find the damn tunable for it :)
Quoted 7 lines
Quoted 5 lines
Documenting the flag in ip(8) might increase its visibility a little. I don't see it documented in the iproute2 git head, though it shows up on http://linux.die.net/man/8/ip somehow.
Stephen, do you know why that is?
No one sent me an official patch to change it?
Mention initcwnd in ip(8). Text taken from doc/ip-cref.tex.
man/man8/ip.8 +10 -1
@@ -211,7 +211,9 @@ replace " | " monitor " } " .B realms .IR REALM " ] [ " .B rto_min -.IR TIME " ]" +.IR TIME " ] [ " +.B initcwnd +.IR NUMBER " ]" .ti -8 .IR TYPE " := [ "
@@ -1561,6 +1563,13 @@ the clamp for congestion window. It is ignored if the flag is not used. .TP +.BI initcwnd " NUMBER " "(2.5.70+ only)" +Initial congestion window size for connections to this destination. +Actual window size is this value multiplied by the MSS +(``Maximal Segment Size'') for same connection. The default is +zero, meaning to use the values specified in RFC2414. + +.TP .BI advmss " NUMBER " "(2.3.15+ only)" the MSS ('Maximal Segment Size') to advertise to these destinations when establishing TCP connections. If it is not given,
Re: [PATCH iproute2] document initcwnd
Quoted 17 lines · On Wed, 05 May 2010 21:37:40 -0400 Brian Bloniarz <[email protected]> wrote:
Quoted 16 lines · Stephen Hemminger
Quoted 13 lines · On Wed, 05 May 2010 16:56:34 -0400 Brian Bloniarz <[email protected]> wrote:
Quoted 8 lines · dormando
Quoted 6 lines
This sounds like TCP slow start.
http://en.wikipedia.org/wiki/Slow-start
As far as tunables you might want to play with the initcwnd route flag (see "ip route help")
Ah, yes, initcwnd was it. I'm well aware of TCP Congestion control / slow start / etc. However I couldn't find the damn tunable for it :)
Documenting the flag in ip(8) might increase its visibility a little. I don't see it documented in the iproute2 git head, though it shows up on http://linux.die.net/man/8/ip somehow.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Stephen, do you know why that is?
No one sent me an official patch to change it?
Mention initcwnd in ip(8). Text taken from doc/ip-cref.tex.
Signed-off-by Brian Bloniarz <[email protected]>
Ok, I will add it with an explicit caution about not doing this on public networks.
Stephen Hemminger
AppliedRe: [PATCH iproute2] document initcwnd
In reply to Brian BloniarzQuoted 17 lines · On Wed, 05 May 2010 21:37:40 -0400 Brian Bloniarz <[email protected]> wrote:
Quoted 16 lines · Stephen Hemminger
Quoted 13 lines · On Wed, 05 May 2010 16:56:34 -0400 Brian Bloniarz <[email protected]> wrote:
Quoted 8 lines · dormando
Quoted 6 lines
This sounds like TCP slow start.
http://en.wikipedia.org/wiki/Slow-start
As far as tunables you might want to play with the initcwnd route flag (see "ip route help")
Ah, yes, initcwnd was it. I'm well aware of TCP Congestion control / slow start / etc. However I couldn't find the damn tunable for it :)
Documenting the flag in ip(8) might increase its visibility a little. I don't see it documented in the iproute2 git head, though it shows up on http://linux.die.net/man/8/ip somehow.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Stephen, do you know why that is?
No one sent me an official patch to change it?
Mention initcwnd in ip(8). Text taken from doc/ip-cref.tex.
Applied