TCP-Protection is really a pain...
Subsystems (2)
Hi.
Your new dynamically adjusted socket-buffer in 2.6.10 is really a pain for big servers. PLEASE tell me a way how to disable it.
Thank you.
Best regards,
Christian Schmid
Quoted 4 lines · On Thu, 03 Feb 2005 05:07:30 +0100 Christian Schmid <[email protected]>…
Hi.
Your new dynamically adjusted socket-buffer in 2.6.10 is really a pain for big servers. PLEASE tell me a way how to disable it.
sysctl -w net.ipv4.tcp_wmem="4096 8192 16384"
Your single stream will be slower, but the memory footprint will be smaller.
Actually, thats my problem. Single streams are too slow! Before I had buffers up to 500 KB. This was very nice to CPU because I only needed to "push" more data once in 5 seconds. I am doing this every second now... *sigh* well maybe you might just want to add a /proc file in order to configure this behaviour.
btw: Another problem I am experiencing is that downloads suddenly break in speed from 360 kb/sec to 8-12 kb/sec. 5 seconds later they stall completely. But the interesting part is, that the send-queue is completely full (checked with a grep in netstat). This looks like as if the receiver is just too slow. But this is not the case. That makes it rather funny. The receiver is waiting with an empty pipe but linux doesn't send. What could this be?
Quoted 14 lines · Stephen Hemminger
Quoted 3 lines · On Thu, 03 Feb 2005 05:07:30 +0100 Christian Schmid <[email protected]>…
Hi.
Your new dynamically adjusted socket-buffer in 2.6.10 is really a pain for big servers. PLEASE tell me a way how to disable it.
sysctl -w net.ipv4.tcp_wmem="4096 8192 16384"
Your single stream will be slower, but the memory footprint will be smaller.
Quoted 1 line · Christian Schmid
btw: Another problem I am experiencing is that downloads suddenly break in speed from 360 kb/sec to 8-12 kb/sec. 5 seconds later they stall completely. But the interesting part is, that the send-queue is completely full (checked with a grep in netstat). This looks like as if the receiver is just too slow. But this is not the case. That makes it rather funny. The receiver is waiting with an empty pipe but linux doesn't send. What could this be?
With recent kernels, I've noticed that my browser progress bar will download about 98% of a page, then stall for several seconds, then complete.
Very strange. Haven't investigated it at all, though.
Jeff
Several users complain about downloads stalling at 99% as well... I was unable to solve this issue.
Quoted 15 lines · Jeff Garzik
Quoted 1 line · Christian Schmid
btw: Another problem I am experiencing is that downloads suddenly break in speed from 360 kb/sec to 8-12 kb/sec. 5 seconds later they stall completely. But the interesting part is, that the send-queue is completely full (checked with a grep in netstat). This looks like as if the receiver is just too slow. But this is not the case. That makes it rather funny. The receiver is waiting with an empty pipe but linux doesn't send. What could this be?
With recent kernels, I've noticed that my browser progress bar will download about 98% of a page, then stall for several seconds, then complete.
Very strange. Haven't investigated it at all, though.
Jeff
Quoted 11 lines · On Thu, 03 Feb 2005 06:21:46 +0100 Christian Schmid <[email protected]>…
Actually, thats my problem. Single streams are too slow! Before I had buffers up to 500 KB. This was very nice to CPU because I only needed to "push" more data once in 5 seconds. I am doing this every second now... *sigh* well maybe you might just want to add a /proc file in order to configure this behaviour.
btw: Another problem I am experiencing is that downloads suddenly break in speed from 360 kb/sec to 8-12 kb/sec. 5 seconds later they stall completely. But the interesting part is, that the send-queue is completely full (checked with a grep in netstat). This looks like as if the receiver is just too slow. But this is not the case. That makes it rather funny. The receiver is waiting with an empty pipe but linux doesn't send. What could this be?
Are you using a board that support TCP Segmentation Offload. The problem may well be that before we were not doing congestion control properly with TSO. A pre-2.6.8 host with TSO was violating all sorts of RFC's and unfairly monopolizing bandwidth.
Signature · 1 line
Stephen Hemminger <[email protected]>
Hm how can I check that? and what do you mean with "board"? Mainboard? NIC? its an onboard-nic on this mainboard: http://www.supermicro.com/products/motherboard/Xeon/E7501/X5DP8-G2.cfm
The worst thing is the stalled downloads. It drops to 8-12 kb-sec. sometimes after a few seconds it goes up to full speed again, but sometimes it suddenly stops completely. After a few seconds it continues with full speed or it stalls forever. send-queue is always full.....
Thank you for your help in this matter.
Chris
Quoted 14 lines · Stephen Hemminger
Quoted 4 lines · On Thu, 03 Feb 2005 06:21:46 +0100 Christian Schmid <[email protected]>…
Actually, thats my problem. Single streams are too slow! Before I had buffers up to 500 KB. This was very nice to CPU because I only needed to "push" more data once in 5 seconds. I am doing this every second now... *sigh* well maybe you might just want to add a /proc file in order to configure this behaviour.
btw: Another problem I am experiencing is that downloads suddenly break in speed from 360 kb/sec to 8-12 kb/sec. 5 seconds later they stall completely. But the interesting part is, that the send-queue is completely full (checked with a grep in netstat). This looks like as if the receiver is just too slow. But this is not the case. That makes it rather funny. The receiver is waiting with an empty pipe but linux doesn't send. What could this be?
Are you using a board that support TCP Segmentation Offload. The problem may well be that before we were not doing congestion control properly with TSO. A pre-2.6.8 host with TSO was violating all sorts of RFC's and unfairly monopolizing bandwidth.
On Thu, 03 Feb 2005 20:58:48 +0100 Christian Schmid <[email protected]> wrote:
E1000 support TSO. To check if it is enabled do:
ethtool -k eth0
To turn it off use.
ethtool -K eth0 tso off
You get the idea..
Signature · 1 line
Stephen Hemminger <[email protected]>
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
What exactly is tcp segmentation offload? Where can I read more about it? Should I disable it or is this not a good idea?
Quoted 10 lines · Stephen Hemminger
On Thu, 03 Feb 2005 20:58:48 +0100 Christian Schmid <[email protected]> wrote:
E1000 support TSO. To check if it is enabled do:
ethtool -k eth0
To turn it off use.
ethtool -K eth0 tso offYou get the idea..
Quoted 7 lines · Christian Schmid
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: onWhat exactly is tcp segmentation offload?
TCP Segmentation Offload, aka TSO or in other contexts "large send" can be thought of as a "poor man's" Jumbo Frame. The NIC advertises to the host that it can resegment larger TCP segments into sizes apropriate for the network.
This allows the host CPU stack to make fewer trips down the protocol stack to transfer a given quantity of data, thus reducing the service demand (quantity of CPU consumed per quantity of work). The reduction in service demand can result in an increase in throughput if the transfer was previously CPU-bound.
It has the advantage over JumboFrame in that it does not require support in the rest of the infrastructure (switches, routers, receivers etc). I call it "poor man's" JumboFrame because it does not reduce the overheads on the receiver - the receiver still sees just as many "normally" sized segments as before, and sends just as many ACKs per KB transferred as before.
If a NIC is implemented with a microprocessor (bletch - personal opinion) the additional demands of TSO resegmenation may actually reduce throughput even as it greatly reduces server CPU utilization. That was particularly evident in old Tigon2 NICs. I am not sure if that is noticable in current generation BMC570X's or not - most of the systems I have at my disposal have BCM5701's in them and I'm not sure the drivers allow TSO to be enabled on that rev? I've not seen a drop in maximum throughput on the "e1000" NICs I've played with, which have been some dual-port cards HP sells. YMMV.
Quoted 1 line
Where can I read more about it?
In addition to whatever there may be in Linux documentation...
TSO or large send is also a long-standing feature of TCP in AIX and (dare I say it here?-) Windows. As such IBM and Microsoft may have writeups - I'm pretty sure that Microsoft had a write-up about it on their website - talking about the NDIS interface at least. It is also a fairly recent addition to HP-UX, but I am not sure if there are any writeups about it on HP's websites yet.
Quoted 1 line
Should I disable it or is this not a good idea?
That depends. If you are concerned about proper slow-start behaviour at the beginning of a connection you should disable it until you are on a later rev.
I've always wondered if it wouldn't be an interesting experiment with a "real" server to see if dishonouring slow-start at connection establishment (and there only, keep it after RTO) made all that much a difference in the host's retransmission rates...
rick jones
Quoted 1 line · Christian Schmid
What exactly is tcp segmentation offload? Where can I read more about it? Should I disable it or is this not a good idea?
It's an optimization designed to reduce CPU usage for zero-copy apps (those that use sendfile(2)).
It would be a useful datapoint to disable TSO, and see if that improves things for you.
Jeff
I turned it off and without it, I was unable to reproduce any stalls anymore.
Quoted 14 lines · Jeff Garzik
Quoted 1 line · Christian Schmid
What exactly is tcp segmentation offload? Where can I read more about it? Should I disable it or is this not a good idea?
It's an optimization designed to reduce CPU usage for zero-copy apps (those that use sendfile(2)).
It would be a useful datapoint to disable TSO, and see if that improves things for you.
Jeff