TCP funny-ness when over-driving a 1Gbps link.
Subsystems (2)
I noticed something that struck me as a bit weird today, but perhaps it's normal.
I was using our application to create 3 TCP streams from one port to another (1Gbps, igb driver), running through a network emulator. Traffic is flowing bi-directional in each connection.
I am doing 24k byte writes per system call. I tried 100ms, 10ms, and 1ms latency (one-way) in the emulator, but behaviour is similar in each case. The rest of this info was gathered with 1ms delay in the emulator.
If I ask all 3 connections to run 1Gbps, netstat shows 30+GB in the sending queues and 1+ second latency (user-space to user-space). Aggregate throughput is around 700Mbps in each direction.
But, if I ask each of the connections to run at 300Mbps, latency averages 2ms and each connection runs right at 300Mbps (950Mbps or so on the wire).
It seems that when you over-drive the link, things back up and perform quite badly over-all.
This is a core-i7 3.2Ghz with 12GB RAM, Fedora 14, 2.6.38.6 kernel (with some hacks), 64-bit OS and user-space app. Quick testing on 2.6.36.3 showed similar results, so I don't think it's a regression.
I am curious if others see similar results?
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 21 lines · On Thu, 19 May 2011 15:47:14 -0700 Ben Greear <[email protected]> wrote:
I noticed something that struck me as a bit weird today, but perhaps it's normal.
I was using our application to create 3 TCP streams from one port to another (1Gbps, igb driver), running through a network emulator. Traffic is flowing bi-directional in each connection.
I am doing 24k byte writes per system call. I tried 100ms, 10ms, and 1ms latency (one-way) in the emulator, but behaviour is similar in each case. The rest of this info was gathered with 1ms delay in the emulator.
If I ask all 3 connections to run 1Gbps, netstat shows 30+GB in the sending queues and 1+ second latency (user-space to user-space). Aggregate throughput is around 700Mbps in each direction.
But, if I ask each of the connections to run at 300Mbps, latency averages 2ms and each connection runs right at 300Mbps (950Mbps or so on the wire).
It seems that when you over-drive the link, things back up and perform quite badly over-all.
Quoted 9 lines
This is a core-i7 3.2Ghz with 12GB RAM, Fedora 14, 2.6.38.6 kernel (with some hacks), 64-bit OS and user-space app. Quick testing on 2.6.36.3 showed similar results, so I don't think it's a regression.
I am curious if others see similar results?
Thanks,
Ben
If you overdrive, TCP expects your network emulator to have a some but limited queueing (like a real router).
Quoted 28 lines · Stephen Hemminger
Quoted 25 lines · On Thu, 19 May 2011 15:47:14 -0700 Ben Greear<[email protected]> wrote:
I noticed something that struck me as a bit weird today, but perhaps it's normal.
I was using our application to create 3 TCP streams from one port to another (1Gbps, igb driver), running through a network emulator. Traffic is flowing bi-directional in each connection.
I am doing 24k byte writes per system call. I tried 100ms, 10ms, and 1ms latency (one-way) in the emulator, but behaviour is similar in each case. The rest of this info was gathered with 1ms delay in the emulator.
If I ask all 3 connections to run 1Gbps, netstat shows 30+GB in the sending queues and 1+ second latency (user-space to user-space). Aggregate throughput is around 700Mbps in each direction.
But, if I ask each of the connections to run at 300Mbps, latency averages 2ms and each connection runs right at 300Mbps (950Mbps or so on the wire).
It seems that when you over-drive the link, things back up and perform quite badly over-all.
This is a core-i7 3.2Ghz with 12GB RAM, Fedora 14, 2.6.38.6 kernel (with some hacks), 64-bit OS and user-space app. Quick testing on 2.6.36.3 showed similar results, so I don't think it's a regression.
Quoted 8 lines
Quoted 5 lines
I am curious if others see similar results?
Thanks,
BenIf you overdrive, TCP expects your network emulator to have a some but limited queueing (like a real router).
The emulator is fine, it's not being over-driven (and has limited queueing if it was being over-driven). The queues that are backing up are in the tcp sockets on the sending machine.
But, just to make sure, I'll re-run the test with a looped back cable...
Ben
Quoted 1 line
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 5 lines · Ben Greear
Quoted 2 lines · Stephen Hemminger
If you overdrive, TCP expects your network emulator to have a some but limited queueing (like a real router).
Quoted 7 lines
The emulator is fine, it's not being over-driven (and has limited queueing if it was
being over-driven). The queues that are backing up are in the tcp sockets on the
sending machine.But, just to make sure, I'll re-run the test with a looped back cable...
Well, with looped back cable, it isn't so bad. I still see a small drop in aggregate throughput (around 900Mbps instead of 950Mbps), and latency goes above 600ms, but it still performs better than when going through the emulator.
At 950+Mbps, the emulator is going to impart 1-2 ms of latency even when configured for wide-open.
If I use a bridge in place of the emulator, it seems to settle on around 450Mbps in one direction and 945Mbps in the other (on the wire), with round-trip latencies often over 5 seconds (user-space to user-space), and a consistent large chunk of data in the socket send buffers:
[root@i7-965-1 igb]# netstat -an|grep tcp|grep 8.1.1
tcp 0 0 8.1.1.1:33038 0.0.0.0:* LISTEN tcp 0 0 8.1.1.1:33040 0.0.0.0:* LISTEN tcp 0 0 8.1.1.1:33042 0.0.0.0:* LISTEN tcp 0 9328612 8.1.1.2:33039 8.1.1.1:33040 ESTABLISHED tcp 0 17083176 8.1.1.1:33038 8.1.1.2:33037 ESTABLISHED tcp 0 9437340 8.1.1.2:33037 8.1.1.1:33038 ESTABLISHED tcp 0 17024620 8.1.1.1:33040 8.1.1.2:33039 ESTABLISHED tcp 0 19557040 8.1.1.1:33042 8.1.1.2:33041 ESTABLISHED tcp 0 9416600 8.1.1.2:33041 8.1.1.1:33042 ESTABLISHED
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 28 lines · Ben Greear
Quoted 5 lines · Ben Greear
Quoted 2 lines · Stephen Hemminger
If you overdrive, TCP expects your network emulator to have a some but limited queueing (like a real router).
Quoted 7 lines
The emulator is fine, it's not being over-driven (and has limited queueing if it was
being over-driven). The queues that are backing up are in the tcp sockets on the
sending machine.But, just to make sure, I'll re-run the test with a looped back cable...
Well, with looped back cable, it isn't so bad. I still see a small drop in aggregate throughput (around 900Mbps instead of 950Mbps), and latency goes above 600ms, but it still performs better than when going through the emulator.
At 950+Mbps, the emulator is going to impart 1-2 ms of latency even when configured for wide-open.
If I use a bridge in place of the emulator, it seems to settle on around 450Mbps in one direction and 945Mbps in the other (on the wire), with round-trip latencies often over 5 seconds (user-space to user-space), and a consistent large chunk of data in the socket send buffers:
[root@i7-965-1 igb]# netstat -an|grep tcp|grep 8.1.1
Quoted 9 lines
tcp 0 0 8.1.1.1:33038 0.0.0.0:* LISTEN tcp 0 0 8.1.1.1:33040 0.0.0.0:* LISTEN tcp 0 0 8.1.1.1:33042 0.0.0.0:* LISTEN tcp 0 9328612 8.1.1.2:33039 8.1.1.1:33040 ESTABLISHED tcp 0 17083176 8.1.1.1:33038 8.1.1.2:33037 ESTABLISHED tcp 0 9437340 8.1.1.2:33037 8.1.1.1:33038 ESTABLISHED tcp 0 17024620 8.1.1.1:33040 8.1.1.2:33039 ESTABLISHED tcp 0 19557040 8.1.1.1:33042 8.1.1.2:33041 ESTABLISHED tcp 0 9416600 8.1.1.2:33041 8.1.1.1:33042 ESTABLISHED
I take it your system has higher values for the tcp_wmem value:
net.ipv4.tcp_wmem = 4096 16384 4194304
and whatever is creating the TCP connections is not making explicit setsockopt() calls to set SO_*BUF.
rick jones
Quoted 39 lines · Rick Jones
Quoted 28 lines · Ben Greear
Quoted 5 lines · Ben Greear
Quoted 2 lines · Stephen Hemminger
If you overdrive, TCP expects your network emulator to have a some but limited queueing (like a real router).
Quoted 7 lines
The emulator is fine, it's not being over-driven (and has limited queueing if it was
being over-driven). The queues that are backing up are in the tcp sockets on the
sending machine.But, just to make sure, I'll re-run the test with a looped back cable...
Well, with looped back cable, it isn't so bad. I still see a small drop in aggregate throughput (around 900Mbps instead of 950Mbps), and latency goes above 600ms, but it still performs better than when going through the emulator.
At 950+Mbps, the emulator is going to impart 1-2 ms of latency even when configured for wide-open.
If I use a bridge in place of the emulator, it seems to settle on around 450Mbps in one direction and 945Mbps in the other (on the wire), with round-trip latencies often over 5 seconds (user-space to user-space), and a consistent large chunk of data in the socket send buffers:
[root@i7-965-1 igb]# netstat -an|grep tcp|grep 8.1.1
Quoted 9 lines
tcp 0 0 8.1.1.1:33038 0.0.0.0:* LISTEN tcp 0 0 8.1.1.1:33040 0.0.0.0:* LISTEN tcp 0 0 8.1.1.1:33042 0.0.0.0:* LISTEN tcp 0 9328612 8.1.1.2:33039 8.1.1.1:33040 ESTABLISHED tcp 0 17083176 8.1.1.1:33038 8.1.1.2:33037 ESTABLISHED tcp 0 9437340 8.1.1.2:33037 8.1.1.1:33038 ESTABLISHED tcp 0 17024620 8.1.1.1:33040 8.1.1.2:33039 ESTABLISHED tcp 0 19557040 8.1.1.1:33042 8.1.1.2:33041 ESTABLISHED tcp 0 9416600 8.1.1.2:33041 8.1.1.1:33042 ESTABLISHED
Quoted 3 lines
I take it your system has higher values for the
tcp_wmemvalue:net.ipv4.tcp_wmem = 4096 16384 4194304
Yes:
[root@i7-965-1 igb]# cat /proc/sys/net/ipv4/tcp_wmem
4096 16384 50000000
Quoted 2 lines
and whatever is creating the TCP connections is not making explicit
setsockopt()calls to set SO_*BUF.
It is configured not to, but if you know of an independent way to verify that, I'm interested.
Thanks,
Ben
Quoted 2 lines
rick jones
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
[root@i7-965-1 igb]# netstat -an|grep tcp|grep 8.1.1
tcp 0 0 8.1.1.1:33038 0.0.0.0:* LISTEN tcp 0 0 8.1.1.1:33040 0.0.0.0:* LISTEN tcp 0 0 8.1.1.1:33042 0.0.0.0:* LISTEN tcp 0 9328612 8.1.1.2:33039 8.1.1.1:33040 ESTABLISHED tcp 0 17083176 8.1.1.1:33038 8.1.1.2:33037 ESTABLISHED tcp 0 9437340 8.1.1.2:33037 8.1.1.1:33038 ESTABLISHED tcp 0 17024620 8.1.1.1:33040 8.1.1.2:33039 ESTABLISHED tcp 0 19557040 8.1.1.1:33042 8.1.1.2:33041 ESTABLISHED tcp 0 9416600 8.1.1.2:33041 8.1.1.1:33042 ESTABLISHED
Quoted 7 lines
Quoted 3 lines
I take it your system has higher values for the
tcp_wmemvalue:net.ipv4.tcp_wmem = 4096 16384 4194304
Yes:
[root@i7-965-1 igb]# cat/proc/sys/net/ipv4/tcp_wmem
4096 16384 50000000
Why?!? Are you trying to get link-rate to Mars or something? (I assume tcp_rmem is similarly set...) If you are indeed doing one 1 GbE, and no more than 100ms then the default (?) of 4194304 should have been more than sufficient.
Quoted 5 lines
Quoted 2 lines
and whatever is creating the TCP connections is not making explicit
setsockopt()calls to set SO_*BUF.It is configured not to, but if you know of an independent way to verify that, I'm interested.
You could always strace the code.
rick
Quoted 15 lines · Rick Jones
Quoted 15 lines
Quoted 14 lines
Quoted 10 lines
[root@i7-965-1 igb]# netstat -an|grep tcp|grep 8.1.1
tcp 0 0 8.1.1.1:33038 0.0.0.0:* LISTEN tcp 0 0 8.1.1.1:33040 0.0.0.0:* LISTEN tcp 0 0 8.1.1.1:33042 0.0.0.0:* LISTEN tcp 0 9328612 8.1.1.2:33039 8.1.1.1:33040 ESTABLISHED tcp 0 17083176 8.1.1.1:33038 8.1.1.2:33037 ESTABLISHED tcp 0 9437340 8.1.1.2:33037 8.1.1.1:33038 ESTABLISHED tcp 0 17024620 8.1.1.1:33040 8.1.1.2:33039 ESTABLISHED tcp 0 19557040 8.1.1.1:33042 8.1.1.2:33041 ESTABLISHED tcp 0 9416600 8.1.1.2:33041 8.1.1.1:33042 ESTABLISHEDI take it your system has higher values for the
tcp_wmemvalue:net.ipv4.tcp_wmem = 4096 16384 4194304
Quoted 8 lines
Quoted 3 lines
Yes:
[root@i7-965-1 igb]# cat/proc/sys/net/ipv4/tcp_wmem
4096 16384 50000000Why?!? Are you trying to get link-rate to Mars or something? (I assume
tcp_rmemis similarly set...) If you are indeed doing one 1 GbE, and no more than 100ms then the default (?) of 4194304 should have been more than sufficient.
Well, we occasionally do tests over emulated links that have several seconds of delay and may be running multiple Gbps. Either way, I'd hope that offering extra RAM to a subsystem wouldn't cause it to go nuts. Assuming this isn't some magical 1Gbps issue, you could probably hit the same problem with a wifi link and default tcp_wmem settings...
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
and whatever is creating the TCP connections is not making explicit
setsockopt()calls to set SO_*BUF.It is configured not to, but if you know of an independent way to verify that, I'm interested.
You could always strace the code.
Yeah...might be easier in this case to just comment out all those calls and do a quick test. Will be tomorrow before I can get to that, however..
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 25 lines · Ben Greear
Quoted 15 lines · Rick Jones
Quoted 15 lines
Quoted 14 lines
Quoted 10 lines
[root@i7-965-1 igb]# netstat -an|grep tcp|grep 8.1.1
tcp 0 0 8.1.1.1:33038 0.0.0.0:* LISTEN tcp 0 0 8.1.1.1:33040 0.0.0.0:* LISTEN tcp 0 0 8.1.1.1:33042 0.0.0.0:* LISTEN tcp 0 9328612 8.1.1.2:33039 8.1.1.1:33040 ESTABLISHED tcp 0 17083176 8.1.1.1:33038 8.1.1.2:33037 ESTABLISHED tcp 0 9437340 8.1.1.2:33037 8.1.1.1:33038 ESTABLISHED tcp 0 17024620 8.1.1.1:33040 8.1.1.2:33039 ESTABLISHED tcp 0 19557040 8.1.1.1:33042 8.1.1.2:33041 ESTABLISHED tcp 0 9416600 8.1.1.2:33041 8.1.1.1:33042 ESTABLISHEDI take it your system has higher values for the
tcp_wmemvalue:net.ipv4.tcp_wmem = 4096 16384 4194304
Quoted 8 lines
Quoted 3 lines
Yes:
[root@i7-965-1 igb]# cat/proc/sys/net/ipv4/tcp_wmem
4096 16384 50000000Why?!? Are you trying to get link-rate to Mars or something? (I assume
tcp_rmemis similarly set...) If you are indeed doing one 1 GbE, and no more than 100ms then the default (?) of 4194304 should have been more than sufficient.
Quoted 4 lines
Well, we occasionally do tests over emulated links that have several seconds of delay and may be running multiple Gbps. Either way, I'd hope that offering extra RAM to a subsystem wouldn't cause it to go nuts.
It has been my experience that the autotuning tends to grow things beyond the bandwidthXdelay product.
As for several seconds of delay and multiple Gbps - unless you are shooting the Moon, sounds like bufferbloat?-)
Quoted 3 lines
Assuming this isn't some magical 1Gbps issue, you could probably hit the same problem with a wifi link and default
tcp_wmemsettings...
Do you also increase tx queue's for the NIC(s)?
rick
Quoted 26 lines · Rick Jones
Quoted 25 lines · Ben Greear
Quoted 23 lines · Rick Jones
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
[root@i7-965-1 igb]# netstat -an|grep tcp|grep 8.1.1
tcp 0 0 8.1.1.1:33038 0.0.0.0:* LISTEN tcp 0 0 8.1.1.1:33040 0.0.0.0:* LISTEN tcp 0 0 8.1.1.1:33042 0.0.0.0:* LISTEN tcp 0 9328612 8.1.1.2:33039 8.1.1.1:33040 ESTABLISHED tcp 0 17083176 8.1.1.1:33038 8.1.1.2:33037 ESTABLISHED tcp 0 9437340 8.1.1.2:33037 8.1.1.1:33038 ESTABLISHED tcp 0 17024620 8.1.1.1:33040 8.1.1.2:33039 ESTABLISHED tcp 0 19557040 8.1.1.1:33042 8.1.1.2:33041 ESTABLISHED tcp 0 9416600 8.1.1.2:33041 8.1.1.1:33042 ESTABLISHEDQuoted 7 lines
Quoted 3 lines
I take it your system has higher values for the
tcp_wmemvalue:net.ipv4.tcp_wmem = 4096 16384 4194304
Yes:
[root@i7-965-1 igb]# cat/proc/sys/net/ipv4/tcp_wmem
4096 16384 50000000Why?!? Are you trying to get link-rate to Mars or something? (I assume
tcp_rmemis similarly set...) If you are indeed doing one 1 GbE, and no more than 100ms then the default (?) of 4194304 should have been more than sufficient.
Quoted 7 lines
Quoted 4 lines
Well, we occasionally do tests over emulated links that have several seconds of delay and may be running multiple Gbps. Either way, I'd hope that offering extra RAM to a subsystem wouldn't cause it to go nuts.
It has been my experience that the autotuning tends to grow things beyond the bandwidthXdelay product.
Seems a likely culprit, or somehow it's not detecting round-trip-time correctly, or maybe the timestamp is calculated when the pkt goes into the send queue, and not when it's actually sent to the NIC?
Quoted 3 lines
As for several seconds of delay and multiple Gbps - unless you are shooting the Moon, sounds like bufferbloat?-)
We try to test our stuff in all sorts of strange cases. Maybe some users really are emulating lunar traffic, or even beyond. We also can emulate buffer bloat..but in this particular case, real round-trip time is about 1-2ms, so if the socket is queuing up a second's worth of bytes on the xmit buffer, then it's not the network's fault...it's the sender.
Quoted 5 lines
Quoted 3 lines
Assuming this isn't some magical 1Gbps issue, you could probably hit the same problem with a wifi link and default
tcp_wmemsettings...Do you also increase tx queue's for the NIC(s)?
No, they are at the default (1000, I think). That's only a few ms at 1Gbps speed, so the problem is mostly higher in the stack.
Thanks,
Ben
Quoted 2 lines
rick
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Re: TCP funny-ness when over-driving a 1Gbps link (and wifi)
Quoted 14 lines · Ben Greear
Quoted 13 lines · Rick Jones
Quoted 12 lines · Ben Greear
Quoted 11 lines · Rick Jones
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
[root@i7-965-1 igb]# netstat -an|grep tcp|grep 8.1.1 tcp 0 0 8.1.1.1:33038 0.0.0.0:* LISTEN
tcp 0 0 8.1.1.1:33040 0.0.0.0:* LISTEN
tcp 0 0 8.1.1.1:33042 0.0.0.0:* LISTEN
tcp 0 9328612 8.1.1.2:33039 8.1.1.1:33040 ESTABLISHED tcp 0 17083176 8.1.1.1:33038 8.1.1.2:33037 ESTABLISHED tcp 0 9437340 8.1.1.2:33037 8.1.1.1:33038 ESTABLISHED tcp 0 17024620 8.1.1.1:33040 8.1.1.2:33039 ESTABLISHED tcp 0 19557040 8.1.1.1:33042 8.1.1.2:33041 ESTABLISHED tcp 0 9416600 8.1.1.2:33041 8.1.1.1:33042 ESTABLISHED
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I take it your system has higher values for the
tcp_wmemvalue:
I tried a different test today: 3 TCP connections between two wifi station interfaces (using ath9k). Each connection is endpoint configured to send 100Mbps of traffic to the peer.
With a single connection, it does OK (maybe 250ms round-trip time max). With 3 of them running, round-trip user-space to user-space latency often goes above 3 seconds.
I had set tcp_wmem smaller for this test, and I verified that the socket SND/RCV buffer setsockopt was not being called.
[root@lec2010-ath9k-1 lanforge]# netstat -an|grep tcp|grep 33
tcp 0 0 12.12.12.4:33040 0.0.0.0:* LISTEN tcp 0 0 12.12.12.4:33042 0.0.0.0:* LISTEN tcp 0 0 12.12.12.4:33044 0.0.0.0:* LISTEN tcp 0 556072 12.12.12.4:33040 12.12.12.3:33039 ESTABLISHED tcp 0 274916 12.12.12.3:33043 12.12.12.4:33044 ESTABLISHED tcp 0 0 192.168.100.138:33738 192.168.100.3:2049 ESTABLISHED tcp 0 205156 12.12.12.4:33042 12.12.12.3:33041 ESTABLISHED tcp 0 217184 12.12.12.3:33041 12.12.12.4:33042 ESTABLISHED tcp 0 436552 12.12.12.3:33039 12.12.12.4:33040 ESTABLISHED tcp 0 288820 12.12.12.4:33044 12.12.12.3:33043 ESTABLISHED
[root@lec2010-ath9k-1 lanforge]# cat /proc/sys/net/ipv4/tcp_wmem
4096 16384 4000000
This is 2.6.39-wl+ kernel.
So, seems a general issue with over-driving links with multiple TCP connections. Doesn't seem like a regression, and probably not really a bug, but maybe the buffer-bloat project will help this sort of thing...
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Re: TCP funny-ness when over-driving a 1Gbps link (and wifi)
Quoted 7 lines · Ben Greear
I tried a different test today: 3 TCP connections between two wifi station interfaces (using ath9k). Each connection is endpoint configured to send 100Mbps of traffic to the peer.
With a single connection, it does OK (maybe 250ms round-trip time max). With 3 of them running, round-trip user-space to user-space latency often goes above 3 seconds.
<snip>
Quoted 4 lines
So, seems a general issue with over-driving links with multiple TCP connections. Doesn't seem like a regression, and probably not really a bug, but maybe the buffer-bloat project will help this sort of thing...
Given that one rule of thumb for the send buffer size is twice the bandwidth delay product, it seems clear that on a wifi connection 3 seconds worth of buffering is excessive. I think I'd classify that as a bug.
Chris
Signature · 5 lines
Chris Friesen Software Developer GENBAND [email protected] www.genband.com