tcp wifi upload performance and lots of ACKs
Subsystems (2)
I'm going some TCP performance testing on wifi -> LAN interface connections. With UDP, we can get around 250Mbps of payload throughput. With TCP, max is about 80Mbps.
I think the problem is that there are way too many ACK packets, and bi-directional traffic on wifi interfaces really slows things down. (About 7000 pkts per second in upload direction, 2000 pps download. And the vast majority of the download pkts are 66 byte ACK pkts from what I can tell.)
Kernel is 3.3.7+
Anyone know of any tuning parameters that would let the receiving socket wait a bit longer and send more ACK data in fewer packets?
Packet traces and other info available if anyone wants to take a look.
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 13 lines · Ben Greear
I'm going some TCP performance testing on wifi -> LAN interface connections. With
UDP, we can get around 250Mbps of payload throughput. With TCP, max is about 80Mbps.I think the problem is that there are way too many ACK packets, and bi-directional
traffic on wifi interfaces really slows things down. (About 7000 pkts per second in
upload direction, 2000 pps download. And the vast majority of the download pkts
are 66 byte ACK pkts from what I can tell.)
Quoted 5 lines
Kernel is 3.3.7+
Anyone know of any tuning parameters that would let the receiving socket wait a
bit longer and send more ACK data in fewer packets?
An ACK is generated after every second full sized segment or a timeout expires.
Currently, there is no way to tune these parameters. Here is an experimental patch [1]. If anyone, thinks that this patch has a chance to get accepted I will be happily try to further improve it.
Quoted 2 lines
Packet traces and other info available if anyone wants to take a look.
thanks,
Daniel.
Quoted 20 lines · Daniel Baluta
Quoted 13 lines · Ben Greear
I'm going some TCP performance testing on wifi -> LAN interface connections.
With
UDP, we can get around 250Mbps of payload throughput. With TCP, max is about 80Mbps.I think the problem is that there are way too many ACK packets, and bi-directional
traffic on wifi interfaces really slows things down. (About 7000 pkts per second in
upload direction, 2000 pps download. And the vast majority of the download pkts
are 66 byte ACK pkts from what I can tell.)Quoted 5 lines
Kernel is 3.3.7+
Anyone know of any tuning parameters that would let the receiving socket wait a
bit longer and send more ACK data in fewer packets?
Quoted 6 lines
An ACK is generated after every second full sized segment or a timeout expires.
Currently, there is no way to tune these parameters. Here is an experimental patch [1]. If anyone, thinks that this patch has a chance to get accepted I will be happily try to further improve it.
It looks like it could be useful for my case, but I would also want per-socket options to set the min/max ack delay so that the settings are not just system-wide.
I can at least test this, and perhaps even hack on the code if you are not interested in the per-socket settings...
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 37 lines · Ben Greear
Quoted 27 lines · Daniel Baluta
Quoted 17 lines · Ben Greear
I'm going some TCP performance testing on wifi -> LAN interface connections.
With
UDP, we can get around 250Mbps of payload throughput. With TCP, max is about 80Mbps.I think the problem is that there are way too many ACK packets, and bi-directional
traffic on wifi interfaces really slows things down. (About 7000 pkts per
second in
upload direction, 2000 pps download. And the vast majority of the download
pkts
are 66 byte ACK pkts from what I can tell.)Quoted 5 lines
Kernel is 3.3.7+
Anyone know of any tuning parameters that would let the receiving socket wait a
bit longer and send more ACK data in fewer packets?Quoted 7 lines
An ACK is generated after every second full sized segment or a timeout expires.
Currently, there is no way to tune these parameters. Here is an experimental
patch [1]. If anyone, thinks that this patch has a chance to get accepted I will be happily try to further improve it.
Quoted 6 lines
It looks like it could be useful for my case, but I would also want per-socket options to set the min/max ack delay so that the settings are not just system-wide.
I can at least test this, and perhaps even hack on the code if you are not interested in the per-socket settings...
Yes, indeed this should be a per socket option. I will be back with a patch in the next days.
Meanwhile, it would be great if you could test my previous patch.
thanks,
Daniel.
Quoted 11 lines · Daniel Baluta
Currently, there is no way to tune these parameters. Here is an experimental patch [1]. If anyone, thinks that this patch has a chance to get accepted I will be happily try to further improve it.
Quoted 2 lines
Packet traces and other info available if anyone wants to take a look.
thanks,
Daniel.
I tried your patch in 3.5.0-rc1.
Doesn't seem to help any...do you have suggested settings for the proc values? (I tried increasing tcp_delack_segs up to 10, but no significant increase...)
I'll keep poking around.
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 14 lines · Ben Greear
Quoted 11 lines · Daniel Baluta
Currently, there is no way to tune these parameters. Here is an experimental patch [1]. If anyone, thinks that this patch has a chance to get accepted I will be happily try to further improve it.
Quoted 2 lines
Packet traces and other info available if anyone wants to take a look.
thanks,
Daniel.
Quoted 7 lines
I tried your patch in 3.5.0-rc1.
Doesn't seem to help any...do you have suggested settings for the proc values? (I tried increasing
tcp_delack_segsup to 10, but no significant increase...)I'll keep poking around.
Ahh, forcing send-buffer to be large helps significantly..wonder if that was my original problem.
Will get some more useful results tomorrow...
Thanks,
Ben
Quoted 4 lines
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 14 lines · Daniel Baluta
Quoted 5 lines · Ben Greear
I'm going some TCP performance testing on wifi -> LAN interface connections.
With
UDP, we can get around 250Mbps of payload throughput. With TCP, max is about 80Mbps.Quoted 7 lines
I think the problem is that there are way too many ACK packets, and bi-directional
traffic on wifi interfaces really slows things down. (About 7000 pkts per second in
upload direction, 2000 pps download. And the vast majority of the download pkts
are 66 byte ACK pkts from what I can tell.)
Quoted 1 line
After a bit more playing, I did notice a reliable 5% increase in traffic (200Mbps -> 210Mbps) from changing the delack segments to 20 from the default of 1. That is enough to be useful to me, and there may be more significant gains to be found... I haven't done a full matrix of testing yet.
I read through the original thread, and to summarize:
* Need to make the values per-socket.
* No multiplication in hot path.
* Would be nice to make it automatic.
The first seems fairly trivial..just add a new set of socket-opts (or maybe just one that can take all 3 values?) and store the settings in the socket structs.
As for getting rid of the multiply..I think you cannot just use shifts. That does not give the needed granularity. An alternative is to update a cached computation every time the mss or socket-opt changes.
As for a magic heuristic to figure this out, I think that would be quite tricky to do right. So, maybe add that in the future, but for the present, just allowing applications to set the value seems enough.
We could support configurable system-wide defaults so that users of programs that do not know this new sockopt can still take advantage of the feature.
Does this sound like a reasonable solution?
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 12 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Ben Greear
Sent: 07 June 2012 05:16
To: Daniel Baluta
Cc: netdev
Subject: Re: tcp wifi upload performance and lots of ACKsQuoted 1 line · Daniel Baluta
On Mon, Jun 4, 2012 at 9:29 PM, Ben Greear<[email protected]>
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
I'm going some TCP performance testing on wifi -> LAN interface
connections.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
With
UDP, we can get around 250Mbps of payload throughput. With TCP,
max is
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
about 80Mbps.
I think the problem is that there are way too many ACK packets, and bi-directional
traffic on wifi interfaces really slows things down. (About 7000
pkts per
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
second in
upload direction, 2000 pps download. And the vast majority of the
download
Quoted 10 lines
Quoted 4 lines
Quoted 2 lines
pkts
are 66 byte ACK pkts from what I can tell.)After a bit more playing, I did notice a reliable 5% increase in traffic (200Mbps -> 210Mbps) from changing the delack segments to 20 from the default of 1. That is enough to be useful to me, and there may be more significant gains to be found... I haven't done a full matrix of testing yet.
Does this delaying of acks have a detrimental effect on the sending end?
I've seen very bad interactions between delayed acks and (I believe) the 'slow start' code on connections with one-directional data, Nagle disabled and very low RTT.
What I saw was the sender sending 4 data packets, then sitting waiting for an ack - in spite of accumulating several kB of data to send.
Delaying acks further will only make this worse.
David
Quoted 44 lines · David Laight
Quoted 12 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Ben Greear
Sent: 07 June 2012 05:16
To: Daniel Baluta
Cc: netdev
Subject: Re: tcp wifi upload performance and lots of ACKsQuoted 1 line · Daniel Baluta
On Mon, Jun 4, 2012 at 9:29 PM, Ben Greear<[email protected]>
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
I'm going some TCP performance testing on wifi -> LAN interface
connections.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
With
UDP, we can get around 250Mbps of payload throughput. With TCP,max is
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
about 80Mbps.
I think the problem is that there are way too many ACK packets, and bi-directional
traffic on wifi interfaces really slows things down. (About 7000pkts per
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
second in
upload direction, 2000 pps download. And the vast majority of thedownload
Quoted 10 lines
Quoted 4 lines
Quoted 2 lines
pkts
are 66 byte ACK pkts from what I can tell.)After a bit more playing, I did notice a reliable 5% increase in traffic (200Mbps -> 210Mbps) from changing the delack segments to 20 from the default of 1. That is enough to be useful to me, and there may be more significant gains to be found... I haven't done a full matrix of testing yet.
Does this delaying of acks have a detrimental effect on the sending end?
I've seen very bad interactions between delayed acks and (I believe) the 'slow start' code on connections with one-directional data, Nagle disabled and very low RTT.
Quoted 5 lines
What I saw was the sender sending 4 data packets, then sitting waiting for an ack - in spite of accumulating several kB of data to send.
Delaying acks further will only make this worse.
I'm sure it's not for everyone in all cases. In my case, I'm sending long-term bulk transfer, at high speeds, over wifi network which has some latency. Tested one-way traffic so far.
With the patch and delayed acks, I get more sender throughput than without (200Mbps -> 210Mbps).
Thanks,
Ben
Quoted 4 lines
David
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 11 lines
Does this delaying of acks have a detrimental effect on the sending end?
I've seen very bad interactions between delayed acks and (I believe) the 'slow start' code on connections with one-directional data, Nagle disabled and very low RTT.What I saw was the sender sending 4 data packets, then sitting waiting for an ack - in spite of accumulating several kB of data to send.
Delaying acks further will only make this worse.
At least two stacks have a reasonable ACK avoidance heuristic. Those would be HP-UX and Solaris (Mac OS 9 had one as well, IIRC). The heuristics are rather similar because the two TCP stacks share a common ancestor. I used to interact with HP-UX's regularly, my statements will be based on that, and an assumption Solaris is similar.
Both attempt to divine what the senders' congestion window happens to be and be certain to send an ACK before that is exhausted. So, at the start of a connection, there will be the usual, more rapid ACKnowledgement. As things happen "normally" then the number of segments per ACK increases until it hits a configurable limit. There are conditions which will cut the limit in half on a given connection - one is the sending of an ACK via the standalone ACK timer (this is from memory, so may be a bit off). There are probably a few other conditions that drop the limit by half. The heuristic attempts to learn in each connection what the reasonable limit on ACK avoidance might be so there isn't a per-connection control, just the global controls via ndd. As conditions causing the limit to be cut in half arise, the connection naturally and irrevocably falls back to the usual "ack-every-other" behaviour.
When there is little to no packet loss, and a rather regular stream of data, this works rather well indeed. For example in a LAN or Data Center. You can run netperf TCP_STREAM with the limit at the default, and with the limit set to two and see the considerable difference in service demand on either side.
This may not work well when the sender has a congestion window growth heuristic different from what the ACK avoidance heuristic assumes. If I recall correctly, the heuristic in HP-UX assumes the sender grows cwnd by the number of bytes/segments ACKnowledged. If the sender grows the cwnd by only one segment per ACK rather than by the bytes ACKnowldeged by the ACK the growth of the cwnd will be slowed. In a LAN that may be papered-over a bit, but it will become quite noticable in a higher RTT environment. Probably not as noticable for a sufficiently short connection, or a long one, but will be for ones in the middle. The short connection doesn't need much cwnd in the first place, and the heuristic works its way up to avoiding ACKs, and the long one will be long enough to have the ACK avoidance heuristic gravitate down to ack-every-other.
rick jones
David Miller
maintainerFrom: Rick Jones <[email protected]>
Date: Thu, 07 Jun 2012 10:51:20 -0700
Quoted 4 lines
This may not work well when the sender has a congestion window growth heuristic different from what the ACK avoidance heuristic assumes. If I recall correctly, the heuristic in HP-UX assumes the sender grows cwnd by the number of bytes/segments ACKnowledged.
Linux violates this assumption.
Quoted 8 lines · Ben Greear
I'm going some TCP performance testing on wifi -> LAN interface connections. With UDP, we can get around 250Mbps of payload throughput. With TCP, max is about 80Mbps.
I think the problem is that there are way too many ACK packets, and bi-directional traffic on wifi interfaces really slows things down. (About 7000 pkts per second in upload direction, 2000 pps download. And the vast majority of the download pkts are 66 byte ACK pkts from what I can tell.)
Quoted 4 lines
Kernel is 3.3.7+
Anyone know of any tuning parameters that would let the receiving socket wait a bit longer and send more ACK data in fewer packets?
Well, thats half duplex days...
There is the ACK every 2 packets rule of thumb, so that tcp sender can increase its cwnd.
Then, some folks tried submitting patches to make '2' more like 10 or 15, but this went nowhere.
Other idea is to arm a timer and defer ACK sending, in the hope we receive another packet very soon.
That could be done with a special qdisc, sort of netem...
Quoted 9 lines · Eric Dumazet
Quoted 8 lines · Ben Greear
I'm going some TCP performance testing on wifi -> LAN interface connections. With UDP, we can get around 250Mbps of payload throughput. With TCP, max is about 80Mbps.
I think the problem is that there are way too many ACK packets, and bi-directional traffic on wifi interfaces really slows things down. (About 7000 pkts per second in upload direction, 2000 pps download. And the vast majority of the download pkts are 66 byte ACK pkts from what I can tell.)
Quoted 6 lines
Quoted 4 lines
Kernel is 3.3.7+
Anyone know of any tuning parameters that would let the receiving socket wait a bit longer and send more ACK data in fewer packets?
Well, thats half duplex days...
WiFi is still half-duplex, and isn't changing any time soon I think.
Quoted 8 lines
There is the ACK every 2 packets rule of thumb, so that tcp sender can increase its cwnd.
Then, some folks tried submitting patches to make '2' more like 10 or 15, but this went nowhere.
Other idea is to arm a timer and defer ACK sending, in the hope we receive another packet very soon.
From a quick read of Daniel's patch..it seems there is already a timer that could be used for this?
Quoted 1 line
That could be done with a special qdisc, sort of netem...
That sounds like a horrible hack :)
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 2 lines · Ben Greear
That sounds like a horrible hack :)
Yep, but could be usefull on a gateway/router, if you cant change the TCP stack of the clients.
Quoted 3 lines · Ben Greear
Anyone know of any tuning parameters that would let the receiving socket wait a bit longer and send more ACK data in fewer packets?
It's perhaps better to think of Acks as offering an opportunity for the transmitting TCP algorithm to sample the congestion of the path. That's particularly important in Slow Start (when the transmitting algorithm is trying to send data exponentially faster) and both traditional TCP and Linux's CUBIC+Hystart will not ramp up transmission as quickly as they otherwise would when starved of samples.
Of course you've got to weigh slower ramp up in transmission rate against the loss in throughput due to the turn-around time on the half-duplex wireless media. But I don't see the proposed patch doing any weighing of considerations at all, but punting that to user space which has even less information about the condition of TCP (particularly the receiver's guess as to the slow-start/congestion-avoidance mode of the transmitter) and less information about the attributes of the media (particularly the turn-around time of half-duplex media on the path) to make a decision with.
There's a nice summary of the tradeoffs of delayed Acks in RFC2525 section 2.13, although the discussion is dated (pre-ABC, pre-HyStart).
Note the firm requirement in RFC2581 section 4.2 that an Ack be sent within 500ms. There are implementations in the field which have delayed Acks out to 2s, these are notoriously painful and many modern algorithms drop back into a low performance mode when facing those receivers.
- Glen