Skip to contentSkip to conversation

TCP-Protection is really a pain...

DISCUSSION Christian Schmid 4 participants 12 messages Feb 3, 2005

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi.

Your new dynamically adjusted socket-buffer in 2.6.10 is really a pain for big servers. PLEASE tell me a way how to disable it.

Thank you.

Best regards,
Christian Schmid

#1 lore raw
Quoted 4 lines · On Thu, 03 Feb 2005 05:07:30 +0100 Christian Schmid <[email protected]>…

Hi.

Your new dynamically adjusted socket-buffer in 2.6.10 is really a pain for big servers. PLEASE tell me a way how to disable it.

sysctl -w net.ipv4.tcp_wmem="4096 8192 16384"

Your single stream will be slower, but the memory footprint will be smaller.

#2 lore raw

Actually, thats my problem. Single streams are too slow! Before I had buffers up to 500 KB. This was very nice to CPU because I only needed to "push" more data once in 5 seconds. I am doing this every second now... *sigh* well maybe you might just want to add a /proc file in order to configure this behaviour.

btw: Another problem I am experiencing is that downloads suddenly break in speed from 360 kb/sec to 8-12 kb/sec. 5 seconds later they stall completely. But the interesting part is, that the send-queue is completely full (checked with a grep in netstat). This looks like as if the receiver is just too slow. But this is not the case. That makes it rather funny. The receiver is waiting with an empty pipe but linux doesn't send. What could this be?

Quoted 14 lines · Stephen Hemminger
Quoted 3 lines · On Thu, 03 Feb 2005 05:07:30 +0100 Christian Schmid <[email protected]>…

Hi.

Your new dynamically adjusted socket-buffer in 2.6.10 is really a pain for big servers. PLEASE tell me a way how to disable it.

sysctl -w net.ipv4.tcp_wmem="4096 8192 16384"

Your single stream will be slower, but the memory footprint will be smaller.

#3 lore raw
Quoted 1 line · Christian Schmid

btw: Another problem I am experiencing is that downloads suddenly break in speed from 360 kb/sec to 8-12 kb/sec. 5 seconds later they stall completely. But the interesting part is, that the send-queue is completely full (checked with a grep in netstat). This looks like as if the receiver is just too slow. But this is not the case. That makes it rather funny. The receiver is waiting with an empty pipe but linux doesn't send. What could this be?

With recent kernels, I've noticed that my browser progress bar will download about 98% of a page, then stall for several seconds, then complete.

Very strange. Haven't investigated it at all, though.

Jeff

#4 lore raw

Several users complain about downloads stalling at 99% as well... I was unable to solve this issue.

Quoted 15 lines · Jeff Garzik
Quoted 1 line · Christian Schmid

btw: Another problem I am experiencing is that downloads suddenly break in speed from 360 kb/sec to 8-12 kb/sec. 5 seconds later they stall completely. But the interesting part is, that the send-queue is completely full (checked with a grep in netstat). This looks like as if the receiver is just too slow. But this is not the case. That makes it rather funny. The receiver is waiting with an empty pipe but linux doesn't send. What could this be?

With recent kernels, I've noticed that my browser progress bar will download about 98% of a page, then stall for several seconds, then complete.

Very strange. Haven't investigated it at all, though.

Jeff

#5 lore raw
In reply to Christian Schmid
Quoted 11 lines · On Thu, 03 Feb 2005 06:21:46 +0100 Christian Schmid <[email protected]>…

Actually, thats my problem. Single streams are too slow! Before I had buffers up to 500 KB. This was very nice to CPU because I only needed to "push" more data once in 5 seconds. I am doing this every second now... *sigh* well maybe you might just want to add a /proc file in order to configure this behaviour.

btw: Another problem I am experiencing is that downloads suddenly break in speed from 360 kb/sec to 8-12 kb/sec. 5 seconds later they stall completely. But the interesting part is, that the send-queue is completely full (checked with a grep in netstat). This looks like as if the receiver is just too slow. But this is not the case. That makes it rather funny. The receiver is waiting with an empty pipe but linux doesn't send. What could this be?

Are you using a board that support TCP Segmentation Offload. The problem may well be that before we were not doing congestion control properly with TSO. A pre-2.6.8 host with TSO was violating all sorts of RFC's and unfairly monopolizing bandwidth.

Signature · 1 line
Stephen Hemminger	<[email protected]>
#6 lore raw

Hm how can I check that? and what do you mean with "board"? Mainboard? NIC? its an onboard-nic on this mainboard: http://www.supermicro.com/products/motherboard/Xeon/E7501/X5DP8-G2.cfm

The worst thing is the stalled downloads. It drops to 8-12 kb-sec. sometimes after a few seconds it goes up to full speed again, but sometimes it suddenly stops completely. After a few seconds it continues with full speed or it stalls forever. send-queue is always full.....

Thank you for your help in this matter.

Chris

Quoted 14 lines · Stephen Hemminger
Quoted 4 lines · On Thu, 03 Feb 2005 06:21:46 +0100 Christian Schmid <[email protected]>…

Actually, thats my problem. Single streams are too slow! Before I had buffers up to 500 KB. This was very nice to CPU because I only needed to "push" more data once in 5 seconds. I am doing this every second now... *sigh* well maybe you might just want to add a /proc file in order to configure this behaviour.

btw: Another problem I am experiencing is that downloads suddenly break in speed from 360 kb/sec to 8-12 kb/sec. 5 seconds later they stall completely. But the interesting part is, that the send-queue is completely full (checked with a grep in netstat). This looks like as if the receiver is just too slow. But this is not the case. That makes it rather funny. The receiver is waiting with an empty pipe but linux doesn't send. What could this be?

Are you using a board that support TCP Segmentation Offload. The problem may well be that before we were not doing congestion control properly with TSO. A pre-2.6.8 host with TSO was violating all sorts of RFC's and unfairly monopolizing bandwidth.

#7 lore raw

On Thu, 03 Feb 2005 20:58:48 +0100 Christian Schmid <[email protected]> wrote:

E1000 support TSO. To check if it is enabled do:
ethtool -k eth0
To turn it off use.
ethtool -K eth0 tso off

You get the idea..

Signature · 1 line
Stephen Hemminger	<[email protected]>
#8 lore raw

Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on

What exactly is tcp segmentation offload? Where can I read more about it? Should I disable it or is this not a good idea?

Quoted 10 lines · Stephen Hemminger

On Thu, 03 Feb 2005 20:58:48 +0100 Christian Schmid <[email protected]> wrote:

E1000 support TSO. To check if it is enabled do:
ethtool -k eth0
To turn it off use.
ethtool -K eth0 tso off

You get the idea..

#9 lore raw
Quoted 7 lines · Christian Schmid

Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on

What exactly is tcp segmentation offload?

TCP Segmentation Offload, aka TSO or in other contexts "large send" can be thought of as a "poor man's" Jumbo Frame. The NIC advertises to the host that it can resegment larger TCP segments into sizes apropriate for the network.

This allows the host CPU stack to make fewer trips down the protocol stack to transfer a given quantity of data, thus reducing the service demand (quantity of CPU consumed per quantity of work). The reduction in service demand can result in an increase in throughput if the transfer was previously CPU-bound.

It has the advantage over JumboFrame in that it does not require support in the rest of the infrastructure (switches, routers, receivers etc). I call it "poor man's" JumboFrame because it does not reduce the overheads on the receiver - the receiver still sees just as many "normally" sized segments as before, and sends just as many ACKs per KB transferred as before.

If a NIC is implemented with a microprocessor (bletch - personal opinion) the additional demands of TSO resegmenation may actually reduce throughput even as it greatly reduces server CPU utilization. That was particularly evident in old Tigon2 NICs. I am not sure if that is noticable in current generation BMC570X's or not - most of the systems I have at my disposal have BCM5701's in them and I'm not sure the drivers allow TSO to be enabled on that rev? I've not seen a drop in maximum throughput on the "e1000" NICs I've played with, which have been some dual-port cards HP sells. YMMV.

Quoted 1 line

Where can I read more about it?

In addition to whatever there may be in Linux documentation...
TSO or large send is also a long-standing feature of TCP in AIX and (dare I say it here?-) Windows. As such IBM and Microsoft may have writeups - I'm pretty sure that Microsoft had a write-up about it on their website - talking about the NDIS interface at least. It is also a fairly recent addition to HP-UX, but I am not sure if there are any writeups about it on HP's websites yet.

Quoted 1 line

Should I disable it or is this not a good idea?

That depends. If you are concerned about proper slow-start behaviour at the beginning of a connection you should disable it until you are on a later rev.

I've always wondered if it wouldn't be an interesting experiment with a "real" server to see if dishonouring slow-start at connection establishment (and there only, keep it after RTO) made all that much a difference in the host's retransmission rates...

rick jones

#10 lore raw
In reply to Christian Schmid
Quoted 1 line · Christian Schmid

What exactly is tcp segmentation offload? Where can I read more about it? Should I disable it or is this not a good idea?

It's an optimization designed to reduce CPU usage for zero-copy apps (those that use sendfile(2)).

It would be a useful datapoint to disable TSO, and see if that improves things for you.

Jeff

#11 lore raw

I turned it off and without it, I was unable to reproduce any stalls anymore.

Quoted 14 lines · Jeff Garzik
Quoted 1 line · Christian Schmid

What exactly is tcp segmentation offload? Where can I read more about it? Should I disable it or is this not a good idea?

It's an optimization designed to reduce CPU usage for zero-copy apps (those that use sendfile(2)).

It would be a useful datapoint to disable TSO, and see if that improves things for you.

Jeff