Network Performance?
This may not be the right forum to ask this. If not, please let me know where to ask.
I have a Debian box with 2 NICs. Both 100/full duplex. This machine is running as a ftp proxy (T.Rex suite). As part of the traffic going through the box, some streams have 1000k window size for a certain reason. How do I tune the NICs to handle the streams better? There are ways of doing this on other OSs. Right now, the box only does about 1.8Mb when it should be doing 80+Mb.
Thanks,
Tim
PS: This is really something to do with the window size and WAN latency. The ultimate source and destination points are either Solaris or AIX boxes. The files being sent are > 1GB in size.
The box does well when traffic goes in one NIC and out the other, as long as the end point is local When it hits the WAN, it all dies. Traffic not going through the box just flies right along, as long as both the end points have the large tcp window size. Putting the Linux box in the middle is a severe choke point. :(
Signature · 6 lines
Tim Sailer <[email protected]> Cyber Security Operations Brookhaven National Laboratory (631) 344-3001 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/
Quoted 13 lines · Tim Sailer
This may not be the right forum to ask this. If not, please let me know where to ask.
I have a Debian box with 2 NICs. Both 100/full duplex. This machine is running as a ftp proxy (T.Rex suite). As part of the traffic going through the box, some streams have 1000k window size for a certain reason. How do I tune the NICs to handle the streams better? There are ways of doing this on other OSs. Right now, the box only does about 1.8Mb when it should be doing 80+Mb.
Thanks,
Tim
Quoted 8 lines
PS: This is really something to do with the window size and WAN latency. The ultimate source and destination points are either Solaris or AIX boxes. The files being sent are > 1GB in size.
The box does well when traffic goes in one NIC and out the other, as long as the end point is local When it hits the WAN, it all dies. Traffic not going through the box just flies right along, as long as both the end points have the large tcp window size. Putting the Linux box in the middle is a severe choke point. :(
I have followed the suggestions in http://www.psc.edu/networking/perf_tune.html
but I still can not get any kind of real throughput. 250kB is all I can get from the Linux box. Setting [r|w]mem_[default|max] larger than 16k makes no difference, smaller slows things down even more. Has anyone else ran across this and fixed it? I can't be the only one with a Linux box on a fat pipe looking for maximum throughput...
Tim
Signature · 6 lines
Tim Sailer <[email protected]> Cyber Security Operations Brookhaven National Laboratory (631) 344-3001 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/
Quoted 24 lines · Tim Sailer
Quoted 13 lines · Tim Sailer
This may not be the right forum to ask this. If not, please let me know where to ask.
I have a Debian box with 2 NICs. Both 100/full duplex. This machine is running as a ftp proxy (T.Rex suite). As part of the traffic going through the box, some streams have 1000k window size for a certain reason. How do I tune the NICs to handle the streams better? There are ways of doing this on other OSs. Right now, the box only does about 1.8Mb when it should be doing 80+Mb.
Thanks,
TimQuoted 8 lines
PS: This is really something to do with the window size and WAN latency. The ultimate source and destination points are either Solaris or AIX boxes. The files being sent are > 1GB in size.
The box does well when traffic goes in one NIC and out the other, as long as the end point is local When it hits the WAN, it all dies. Traffic not going through the box just flies right along, as long as both the end points have the large tcp window size. Putting the Linux box in the middle is a severe choke point. :(
Quoted 6 lines
I have followed the suggestions in http://www.psc.edu/networking/perf_tune.html
but I still can not get any kind of real throughput. 250kB is all I can get from the Linux box. Setting [r|w]mem_[default|max] larger than 16k makes no difference, smaller slows things down even more. Has anyone else ran across this and fixed it? I can't be the only one with a Linux box on a fat pipe looking for maximum throughput...
Tim,
this issue was discussed on the netdev mailing list a few weeks back.
It's very unfortunate that the web archives of netdev stopped working several months ago and there now appears to be no web archive of [email protected].
Go to http://oss.sgi.com/projects/netdev/archive/ and pull down the November and December archives.
The subject was "linux to solaris tcp issues on WAN".
The conclusion was "The problem is also fixed with 2.4.0-test12pre3". Dunno about kernel 2.2 though.
-
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 2 lines
The conclusion was "The problem is also fixed with 2.4.0-test12pre3". Dunno about kernel 2.2 though.
DaveM sent me a patch to address the problem its in 2.2.19pre3 -
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 7 lines · Andrew Morton
this issue was discussed on the netdev mailing list a few weeks back.
It's very unfortunate that the web archives of netdev stopped working several months ago and there now appears to be no web archive of [email protected].
Quoted 7 lines
Go to http://oss.sgi.com/projects/netdev/archive/ and pull down the November and December archives.
The subject was "linux to solaris tcp issues on WAN".
The conclusion was "The problem is also fixed with 2.4.0-test12pre3". Dunno about kernel 2.2 though.
Well, on Friday, we pulled down the 'official' 2.4.0, and had the same experience... nothing better. Should I get the -test12-pre3 kernel and try that one specifically?
Tim
Signature · 6 lines
Tim Sailer <[email protected]> Cyber Security Operations Brookhaven National Laboratory (631) 344-3001 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/
Quoted 12 lines · Tim Sailer
Quoted 10 lines · Andrew Morton
this issue was discussed on the netdev mailing list a few weeks back.
It's very unfortunate that the web archives of netdev stopped working several months ago and there now appears to be no web archive of [email protected].
Go to http://oss.sgi.com/projects/netdev/archive/ and pull down the November and December archives.
Quoted 8 lines
Quoted 4 lines
The subject was "linux to solaris tcp issues on WAN".
The conclusion was "The problem is also fixed with 2.4.0-test12pre3". Dunno about kernel 2.2 though.
Well, on Friday, we pulled down the 'official' 2.4.0, and had the same experience... nothing better. Should I get the -test12-pre3 kernel and try that one specifically?
I doubt if that would help.
I claim no expertise in this area, but perhaps we can get some protocol gurus interested.
To recap:
You're sending and receiving FTP/TCP/IP4 to Solaris and AIX hosts You have a 1000kbyte window size
You have an 80 megabit/sec pipe.
You're getting 1.8 megabits/sec.
What is the round-trip time on the WAN?
Packet loss?
Does the problem occur in both directions?
Are you _sure_ the window size is being set correctly? How is it being set?
Are you able to generate TCP dumps when the problem is happening?
-
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 22 lines · Andrew Morton
Quoted 12 lines · Tim Sailer
Quoted 10 lines · Andrew Morton
this issue was discussed on the netdev mailing list a few weeks back.
It's very unfortunate that the web archives of netdev stopped working several months ago and there now appears to be no web archive of [email protected].
Go to http://oss.sgi.com/projects/netdev/archive/ and pull down the November and December archives.
Quoted 8 lines
Quoted 4 lines
The subject was "linux to solaris tcp issues on WAN".
The conclusion was "The problem is also fixed with 2.4.0-test12pre3". Dunno about kernel 2.2 though.
Well, on Friday, we pulled down the 'official' 2.4.0, and had the same experience... nothing better. Should I get the -test12-pre3 kernel and try that one specifically?
Quoted 8 lines
I doubt if that would help.
I claim no expertise in this area, but perhaps we can get some protocol gurus interested.
To recap:
You're sending and receiving FTP/TCP/IP4 to Solaris and AIX hosts
Yup
Quoted 1 line
You have a 1000kbyte window size
Yup
Quoted 1 line
You have an 80 megabit/sec pipe.
Actually, 100 to the router, and then the WAN connection to the remote system is OC-3
Quoted 1 line
You're getting 1.8 megabits/sec.
Yup
Quoted 3 lines
What is the round-trip time on the WAN?
Packet loss?
101 packets transmitted, 101 packets received, 0% packet loss round-trip min/avg/max = 109.6/110.3/112.2 ms
Quoted 1 line
Does the problem occur in both directions?
Good question. I'll find out.
Quoted 2 lines
Are you _sure_ the window size is being set correctly? How is it being set?
I'm fairly sure. We echo the value to the file. catting it back shows the correct value. If we go lower than default, it slows down even more.
Quoted 1 line
Are you able to generate TCP dumps when the problem is happening?
We can, if it will help.
Tim
Signature · 6 lines
Tim Sailer <[email protected]> Cyber Security Operations Brookhaven National Laboratory (631) 344-3001 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/
Quoted 5 lines
101 packets transmitted, 101 packets received, 0% packet loss round-trip min/avg/max = 109.6/110.3/112.2 ms
Quoted 1 line
Does the problem occur in both directions?
Quoted 8 lines
Good question. I'll find out.
Quoted 2 lines
Are you _sure_ the window size is being set correctly? How is it being set?
I'm fairly sure. We echo the value to the file. catting it back shows the correct value. If we go lower than default, it slows down even more.
what are you setting it to on the solaris machine? what window sizes have you tried?
Your pipe looks like it will have quite a few bits in flight due to its latency. From my quick guess math, which sucks, it appears that you can fit 1.2 to 1.5 megabytes on the wire (100mbit machine<-> machine) times 100-120ms wire time. This is a rather large number, so you may want to see what hosts really support, perhaps starting with 64k or 128k and work up. Make sure that you have window scaling turned on if you go with very large windows.
Also, have you upped your socket buffers to match your window sizes?
Last, solaris tends to have poorly tuned tcp values out of the box, look at this link and tune the solaris stack to better reflect reality.
http://www.google.com/search?q=cache:www.rvs.uni-hannover.de/people/voeckler/tune/EN/tune.html+%2Bwan+%2Bwindow+%2Bscale+%2Bsize+%2Bnetwork&hl=en
linux tuning has a decent amount of data in the docs section of the kernel sources.
-- craig
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 6 lines · Tim Sailer
Quoted 3 lines
What is the round-trip time on the WAN?
Packet loss?
101 packets transmitted, 101 packets received, 0% packet loss round-trip min/avg/max = 109.6/110.3/112.2 ms
Packet loss and RTT can be greatly affected by how much data you're sending through a path, so a simple ping is probably not adequate.
Also, even a much smaller packet loss rate than 1/100 could really kill your throughput.
Quoted 10 lines
Quoted 1 line
Does the problem occur in both directions?
Good question. I'll find out.
Quoted 2 lines
Are you _sure_ the window size is being set correctly? How is it being set?
I'm fairly sure. We echo the value to the file. catting it back shows the correct value. If we go lower than default, it slows down even more.
You have to be certain you restart any applications currently running, but it seems like you're probably already doing that, since you observe a slow-down with smaller windows.
Is window scaling enabled in the kernel?
(/proc/sys/net/ipv4/tcp_window_scaling)
You said you're using this as a proxy. To rule out the real ftp server machine as a problem, are you doing an FTP directly from the Linux machine to your destination?
Also, your destination machine must have an appropriate receive window/buffer.
Quoted 3 lines
Quoted 1 line
Are you able to generate TCP dumps when the problem is happening?
We can, if it will help.
Viewing a trace of the connection is often the most informative and effective means of debugging a connection. One note, traces should probably be collected from the sender. Receiver traces rarely contain useful information.
Other questions: What kind of router(s) are you using? How big are its buffers? What drop algorithm does it use? Have you successfully had any bulk TCP flows (from other OS's) go through this particular path at near the full 80Mb?
There are a number of difficulties that may occur when trying to get full bandwidth out of a long fat pipe (LFP). You certainly aren't the only one having problems. Unfortunately it often takes a TCP expert to diagnose the problem (or problems), since there are so many possible problems.
-John
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 10 lines · Craig I. Hagan
Quoted 10 lines
101 packets transmitted, 101 packets received, 0% packet loss round-trip min/avg/max = 109.6/110.3/112.2 ms
Quoted 1 line
Does the problem occur in both directions?
Good question. I'll find out.
Quoted 2 lines
Are you _sure_ the window size is being set correctly? How is it being set?
Quoted 8 lines
Quoted 3 lines
I'm fairly sure. We echo the value to the file. catting it back shows the correct value. If we go lower than default, it slows down even more.
what are you setting it to on the solaris machine? what window sizes have you tried?
Your pipe looks like it will have quite a few bits in flight due to its
Yup. That's why the tuning. WAN performance here is very important.
Quoted 2 lines
latency. From my quick guess math, which sucks, it appears that you can fit 1.2 to 1.5 megabytes on the wire (100mbit machine<-> machine) times 100-120ms wire
Hmm. 100/8 is about 12, no?
Quoted 3 lines
time. This is a rather large number, so you may want to see what hosts really support, perhaps starting with 64k or 128k and work up. Make sure that you have window scaling turned on if you go with very large windows.
Yes, we have that enabled too.
Quoted 1 line
Also, have you upped your socket buffers to match your window sizes?
We are using straight ftp for the testing.
Quoted 6 lines
Last, solaris tends to have poorly tuned tcp values out of the box, look at this link and tune the solaris stack to better reflect reality.
http://www.google.com/search?q=cache:www.rvs.uni-hannover.de/people/voeckler/tune/EN/tune.html+%2Bwan+%2Bwindow+%2Bscale+%2Bsize+%2Bnetwork&hl=enlinux tuning has a decent amount of data in the docs section of the kernel sources.
I'll take a look. THanks.
Tim
Signature · 6 lines
Tim Sailer <[email protected]> Cyber Security Operations Brookhaven National Laboratory (631) 344-3001 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/
Quoted 16 lines · Tim Sailer
Quoted 1 line · Andrew Morton
You're sending and receiving FTP/TCP/IP4 to Solaris and AIX hosts
Yup
Quoted 1 line
You have a 1000kbyte window size
Yup
Quoted 1 line
You have an 80 megabit/sec pipe.
Actually, 100 to the router, and then the WAN connection to the remote system is OC-3
Quoted 1 line
You're getting 1.8 megabits/sec.
Quoted 8 lines
Yup
Quoted 3 lines
What is the round-trip time on the WAN?
Packet loss?
101 packets transmitted, 101 packets received, 0% packet loss round-trip min/avg/max = 109.6/110.3/112.2 ms
I had similar problems two weeks ago. Turned out the connection between two switches: one of them was hard wired to 100Mbit/s full duplex, the other one to 100Mbit/s half duplex. Just to rule out the obvious...
Erik
Signature · 9 lines
J.A.K. (Erik) Mouw, Information and Communication Theory Group, Department of Electrical Engineering, Faculty of Information Technology and Systems, Delft University of Technology, PO BOX 5031, 2600 GA Delft, The Netherlands Phone: +31-15-2783635 Fax: +31-15-2781843 Email: [email protected] WWW: http://www-ict.its.tudelft.nl/~erik/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/
Quoted 3 lines · Erik Mouw
I had similar problems two weeks ago. Turned out the connection between two switches: one of them was hard wired to 100Mbit/s full duplex, the other one to 100Mbit/s half duplex. Just to rule out the obvious...
We check that as the first thing. Both are set the same. No collisions out of the ordinary.
Tim
Signature · 6 lines
Tim Sailer <[email protected]> Cyber Security Operations Brookhaven National Laboratory (631) 344-3001 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/
Quoted 7 lines · Tim Sailer
Quoted 3 lines · Erik Mouw
I had similar problems two weeks ago. Turned out the connection between two switches: one of them was hard wired to 100Mbit/s full duplex, the other one to 100Mbit/s half duplex. Just to rule out the obvious...
We check that as the first thing. Both are set the same. No collisions out of the ordinary.
Are you using netfilter? And if so, does netfilter support window-scaling without the tcp-window-tracking patch?
/Martin
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 11 lines · Martin Josefsson
Quoted 7 lines · Tim Sailer
Quoted 3 lines · Erik Mouw
I had similar problems two weeks ago. Turned out the connection between two switches: one of them was hard wired to 100Mbit/s full duplex, the other one to 100Mbit/s half duplex. Just to rule out the obvious...
We check that as the first thing. Both are set the same. No collisions out of the ordinary.
Are you using netfilter? And if so, does netfilter support window-scaling
Nope. We have it stripped down bare at this point to try to pinpoint the problem.
Tim
Quoted 3 lines
without the tcp-window-tracking patch?
/Martin
Signature · 6 lines
Tim Sailer <[email protected]> Cyber Security Operations Brookhaven National Laboratory (631) 344-3001 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/
Here is some more data:
Inbound = 99.66 kB/s
Outbound= 151 kB/s--------------------------------------------
ports:/home/ftp# sysctl -a | fgrep net/core net/core/optmem_max = 10240 net/core/message_burst = 50 net/core/message_cost = 5 net/core/netdev_max_backlog = 300 net/core/rmem_default = 32767 net/core/wmem_default = 32767 net/core/rmem_max = 2097152 net/core/wmem_max = 2097152 --------------------------------------------
Tim
Signature · 6 lines
Tim Sailer <[email protected]> Cyber Security Operations Brookhaven National Laboratory (631) 344-3001 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/
Quoted 7 lines · Tim Sailer
Here is some more data:
Inbound = 99.66 kB/s Outbound= 151 kB/s--------------------------------------------
Quoted 10 lines
ports:/home/ftp# sysctl -a | fgrep net/core net/core/optmem_max = 10240 net/core/message_burst = 50 net/core/message_cost = 5 net/core/netdev_max_backlog = 300 net/core/rmem_default = 32767 <<<<<<<<< net/core/wmem_default = 32767 <<<<<<<<< net/core/rmem_max = 2097152 net/core/wmem_max = 2097152 --------------------------------------------
The defaults must be large unless your application calls setsockopt() to set the buffers itself. (Some FTP clients and servers can do this, but for testing, your're still probably better always having the _max's and _default's the same.)
-John
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 12 lines · John Heffner
Quoted 11 lines
--------------------------------------------
ports:/home/ftp# sysctl -a | fgrep net/core net/core/optmem_max = 10240 net/core/message_burst = 50 net/core/message_cost = 5 net/core/netdev_max_backlog = 300 net/core/rmem_default = 32767 <<<<<<<<< net/core/wmem_default = 32767 <<<<<<<<< net/core/rmem_max = 2097152 net/core/wmem_max = 2097152 --------------------------------------------
Quoted 4 lines
The defaults must be large unless your application calls
setsockopt()to set the buffers itself. (Some FTP clients and servers can do this, but for testing, your're still probably better always having the _max's and _default's the same.)
Hm.. OK. I think we tried that, but I'll check again.
Thanks,
Tim
Signature · 6 lines
Tim Sailer <[email protected]> Cyber Security Operations Brookhaven National Laboratory (631) 344-3001 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/
Quoted 6 lines · Tim Sailer
Quoted 4 lines
The defaults must be large unless your application calls
setsockopt()to set the buffers itself. (Some FTP clients and servers can do this, but for testing, your're still probably better always having the _max's and _default's the same.)Hm.. OK. I think we tried that, but I'll check again.
And make sure your ftp client/server isn't resetting it to something small afterwards. For testing this, I'd use a real IP benchmarking program like iperf/netperf/ttcp, as they'll let you test different buffer sizes easily (and in the case of iperf tell you what you're actually using if you hit the limit) For a fast WAN you want something like 512k-1M buffers easily.
Signature · 8 lines
Pekka Pietikainen - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/