Socket buffer sizes with autotuning

One of the issues with netperf and linux is that netperf only snaps the socket buffer size at the beginning of the connection. This of course does not catch what the socket buffer size might become over the lifetime of the connection. So, in the in-development "omni" tests I've added code that when running on Linux will snap the socket buffer sizes at both the beginning and end of the data connection. I was a triffle surprised at some of what I saw with a 1G connection between systems - when autoscaling/ranging/tuning/whatever was active (netperf taking defaults and not calling setsockopt()) I was seeing the socket buffer size at the end of the connection up at 4MB:

sut34:~/netperf2_trunk# netperf -l 1 -t omni -H oslowest -- -d 4 -o bar -s -1 -S -1 -m ,16K
OMNI TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to oslowest.raj (10.208.0.1) port 0 AF_INET
Throughput,Direction,Local Release,Local Recv Socket Size Requested,Local Recv Socket Size Initial,Local Recv Socket Size Final,Remote Release,Remote Send Socket Size Requested,Remote Send Socket Size Initial,Remote Send Socket Size Final
940.52,Receive,2.6.25-raj,-1,87380,4194304,2.6.18-5-mckinley,-1,16384,4194304

Which was the limit of the autotuning:

net.ipv4.tcp_wmem = 16384       16384   4194304
net.ipv4.tcp_rmem = 16384       87380   4194304

The test above is basically the omni version of a TCP_MAERTS test from a 2.6.18 system to a 2.6.25 system (kernel bits grabbed about 40 minutes ago from http://www.kernel.org/hg/linux-2.6. The receiving system on which the 2.6.25 bits were compiled and run started life as a Debian Lenny/Testing system. The sender is iirc Debian Etch.

It seemed odd to me that one would need a 4MB socket buffer to get link-rate on gigabit, so I ran a quick set of tests to confirm in my mind that indeed, a much smaller socket buffer was sufficient:

sut34:~/netperf2_trunk# HDR="-P 1"; for i in -1 32K 64K 128K 256K 512K; do netperf -l 20 -t omni -H oslowest $HDR -- -d 4 -o bar -s $i -S $i -m ,16K; HDR="-P 0"; done
OMNI TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to oslowest.raj (10.208.0.1) port 0 AF_INET
Throughput,Direction,Local Release,Local Recv Socket Size Requested,Local Recv Socket Size Initial,Local Recv Socket Size Final,Remote Release,Remote Send Socket Size Requested,Remote Send Socket Size Initial,Remote Send Socket Size Final
941.38,Receive,2.6.25-raj,-1,87380,4194304,2.6.18-5-mckinley,-1,16384,4194304
939.29,Receive,2.6.25-raj,32768,65536,65536,2.6.18-5-mckinley,32768,65536,65536
940.28,Receive,2.6.25-raj,65536,131072,131072,2.6.18-5-mckinley,65536,131072,131072
940.96,Receive,2.6.25-raj,131072,262142,262142,2.6.18-5-mckinley,131072,253952,253952
940.99,Receive,2.6.25-raj,262144,262142,262142,2.6.18-5-mckinley,262144,253952,253952
940.98,Receive,2.6.25-raj,524288,262142,262142,2.6.18-5-mckinley,524288,253952,253952

And then I decided to let the receiver autotune while the sender was either autotune or fixed (simulating something other than Linux sending I suppose):

sut34:~/netperf2_trunk# HDR="-P 1"; for i in -1 32K 64K 128K 256K 512K; do netperf -l 20 -t omni -H oslowest $HDR -- -d 4 -o bar -s -1 -S $i -m ,16K; HDR="-P 0"; done
OMNI TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to oslowest.raj (10.208.0.1) port 0 AF_INET
Throughput,Direction,Local Release,Local Recv Socket Size Requested,Local Recv Socket Size Initial,Local Recv Socket Size Final,Remote Release,Remote Send Socket Size Requested,Remote Send Socket Size Initial,Remote Send Socket Size Final
941.38,Receive,2.6.25-raj,-1,87380,4194304,2.6.18-5-mckinley,-1,16384,4194304
941.34,Receive,2.6.25-raj,-1,87380,1337056,2.6.18-5-mckinley,32768,65536,65536
941.35,Receive,2.6.25-raj,-1,87380,1814576,2.6.18-5-mckinley,65536,131072,131072
941.38,Receive,2.6.25-raj,-1,87380,2645664,2.6.18-5-mckinley,131072,253952,253952
941.39,Receive,2.6.25-raj,-1,87380,2649728,2.6.18-5-mckinley,262144,253952,253952
941.38,Receive,2.6.25-raj,-1,87380,2653792,2.6.18-5-mckinley,524288,253952,253952

Finally to see what was going on the wire (in case it was simply the socket buffer getting larger and not also the window) I took a packet trace on the sender to look at the window updates coming back, and sure enough, by the end of the connection (wscale = 7) the advertised window was huge:

17:10:00.522200 IP sut34.raj.53459 > oslowest.raj.37322: S 3334965237:3334965237(0) win 5840 <mss 1460,sackOK,timestamp 4294921737 0,nop,wscale 7> 17:10:00.522214 IP oslowest.raj.37322 > sut34.raj.53459: S 962695631:962695631(0) ack 3334965238 win 5792 <mss 1460,sackOK,timestamp 3303630187 4294921737,nop,wscale 7>

...

17:10:01.554698 IP sut34.raj.53459 > oslowest.raj.37322: . ack 121392225 win 24576 <nop,nop,timestamp 4294921995 3303630438> 17:10:01.554706 IP sut34.raj.53459 > oslowest.raj.37322: . ack 121395121 win 24576 <nop,nop,timestamp 4294921995 3303630438>

I also checked (during a different connection, autotuning at both ends) how much was actually queued at the sender, and it was indeed rather large:

oslowest:~# netstat -an | grep ESTAB
...
tcp    0 2760560 10.208.0.1:40500    10.208.0.45:42049   ESTABLISHED
...

Is this expected behaviour?

rick jones

What is your interface txqueuelen and mtu? If you have a very large interface queue, TCP will happily fill it up unless you are using a delay-based congestion controller.

-John

From: "John Heffner" <[email protected]>
Date: Tue, 22 Apr 2008 19:17:39 -0700

Yes, that's the fundamental problem with loss based congestion control. If there are any queues in the path, TCP will fill them up.

Vegas and other similar techniques are able to avoid this, but come with the fundamental flaw that it's easy to get them into situations where they do not respond to increases in pipe space adequately, and thus underperform compared to loss based algorithms.

1000, MTU 1500 and TSO is enabled.

At the same time, the sender will not try to send any more than the receiver is willing to advertise as window.

I can see that for the sending side being willing to send into the receiver's ever increasing window, but is autotuning supposed to keep growing and growing the receive window the way it seems to be?

rick jones

Receive-side autotuning by design will attempt to grow the rcvbuf (adjusting for overhead) to twice the observed cwnd[1]. When the sender keeps growing its window to fill up your interface queue, the receiver will continue to grow its window to let the sender do what it wants. It's not the receiver's job to do congestion control.

One interesting observation is that when timestamps are turned off the receiver autotuning actually has the property that if the RTT is growing (in this case due to a queue filling), it will not grow the window since it's not able to update its RTT estimate. This property was described as a feature of the Dynamic Right-Sizing algorithm
(http://public.lanl.gov/radiant/software/drs.html), and obviously in some cases it is. However, in general it has the same types of problems that delay-based congestion control has. And, it's not the receiver's job to do congestion control. :-)

One thing you will note if you run many flows is that the aggregate buffer space used should be much less than n*2MB, since each flow is competing for the same queue space. This has good scaling properties.

-John

[1] It's not exactly accurate that it tries to set rcvbuf to 2*cwnd. A subtle but important distinction is that it tries to set rcvbuf to twice the data read by the application in any RTT.

Then why is it starting small and growing the advertised window?

Then why is it starting small and growing the advertised window?-)

I'll see about getting another system on which to test (the one I got 2.6.25 onto yesterday is being pulled-back for other things, grrr...) and see what happens when I run multiple netperfs.

I guess that explains why it grew but didn't keep growing when the sender's SO_SNDBUF was fixed.

What concerns me is that we tell folks to rely on autotuning because it will set their socket buffers to the size they need, but it seems to be setting their socket buffers to very much more than they need. Given that netperf couldn't have been receiving any more data in an RTT if it was always going at ~940 Mbit/s I'm still perplexed unless there is a positive feedback here - allow a greater window, this allows more data to be queued, which increases the RTT, which allows more data to be recevied in an "RTT" which allows a greater window...

rick jones

Because the receiver tracks the sender's behavior. (Also because of the algorithm for figuring out the buffer overhead ratio, but that's a different story.)

It's true that careful hand-tuning can do better than autotuning in some cases, due to the sub-optimality of congestion control. This is one such case, where you have an overbuffered bottleneck. Another case is where you have an underbuffered bottleneck, especially with a large BDP. You can calculate exactly the window size you need to fill the bottleneck and set your send or receive buffer to throttle you to exactly that window, so that you will get 100% utilization. If you let autotuning work, it will keep increasing your buffer as congestion control asks for more until you overflow the buffer and take a loss. Then, congestion control will have to back off, likely resulting in under-utilization.

In these cases, hand-tuning can do better because you have out-of-band knowledge of the path. Also, smarter congestion control algorithms can do better than traditional AIMD, but it's a very hard problem to do this well in general. Autotuning's job is mainly just to grow the buffer sizes to as much as congestion control needs to do its thing.

-John

It seems very, well, trusting in that regard.

rick jones

That just means Linux does too much queueing by default. Perhaps that should be fixed. On Ethernet hardware the NIC TX queue should be usually sufficient anyways I would guess. Do we really need the long qdisc queue too?

-Andi

The default on most ethernet devices used to be 100 packets. This is pretty short when your flow's window size is thousands of packets. It's especially a killer for large BDP flows because it causes slow-start to cut out early and you have to ramp up slowly with congestion avoidance. (This effect it mitigated somewhat by cubic, etc, or even better by limited slow-start, but it's still very significant.)

I have in the past used a hack that suppressed congestion notification when overflowing the local interface queue. This is a very simple approach, and works fairly well, but doesn't have the property of converging toward fairness if multiple flows are competing for bandwidth at the local interface. It might be possible to cook something up that's smarter.

-John

From: Andi Kleen <[email protected]>
Date: Fri, 25 Apr 2008 00:21:08 +0200

It is definitely an area that deserves attention and a real investigation, that's for sure.

I spent a lot of time looking at the behavior in this area while writing the NIU 10G driver. As usual, I got distracted before I could make any real experiments.

I'll try to get back to this if someone else doesn't beat me to it. We also have the TX multiqueue stuff to move forward, and it's deeply related to these issues.

From: "Tom Herbert" <[email protected]>
Date: Thu, 24 Apr 2008 22:34:50 -0700

This work is interesting, but doesn't it make more sense to limit by number of packets instead of bytes?

No intermediate node that I know of drops bytes, they drop packets instead :-)

If NIC is interrupt driven, enough data must be queued to span consecutive interrupts. So for instance if a 10G NIC is generating an interrupt every 100us, about 125,000 bytes needs to queued at each interrupt to prevent starvation-- this doesn't translate to a fixed number of packets. Limiting by packets seems somewhat ad hoc; if the limit is to small and the link will be starved, too big and over-queuing results.

From: "Tom Herbert" <[email protected]>
Date: Fri, 25 Apr 2008 00:42:03 -0700

Yes, however when packets are small the limiting factor becomes per-transfer transaction related overhead.

The gist of our algorithm is to determine when the device has been starved (ie. when all packets queued to the device have been sent, and there are packets ready in host queues). When the device is starved, the byte limit is increased by some amount; if the device is not starved, the byte limit is decreased by an amount subject to some mechanisms to prevent hysteresis. If throughput is less than line rate, because of small packets for example, the byte limit will decrease accordingly since fewer packets are needed to be queued to prevent starvation. This algorithm requires no a priori knowledge of link speed or interrupt rate, it will adjust the byte queue limit to such variable parameters.