Annoying bug with many sockets.
Subsystems (2)
Hi.
This is really annoying. With 3500 sockets onwards, linux 2.6.10 completely lags. This is a bug and I am not willing to buy new servers just because linux has a BUG. tcp_mem _rmem and _wmem have been set to 1024000 for testing but this doesnt help as well. so whats WRONG there... please?
Best regards,
Chris
Quoted 6 lines · Christian Schmid
Hi.
This is really annoying. With 3500 sockets onwards, linux 2.6.10 completely lags. This is a bug and I am not willing to buy new servers just because linux has a BUG.
tcp_mem_rmemand_wmemhave been set to 1024000 for testing but this doesnt help as well. so whats WRONG there... please?Best regards,
Chris
You have not actually said what the problem is - do new connections not get made? Or existing connections slow down?
You are trying to run many simultaneous connections, so bumping up the individual socket buffer allocation will not necessarily help - you need to bump up the global TCP limit (tcp_mem[]) - it's a 3-tuple - if you have the memory in your system, bump it way up. netstat -tan will tell you if there is unread data in the queues..
Are you running into memory pressure? Or aborts? netstat -s might give you some info on what is happening.
Bump up the port space (/proc/sys/net/ipv4/ip_local_port_range) available - typical default is 32K - 61000 (can lower min to 4K)
Are they all receiving data or sending? Are they talking to different hosts?
You can increase tcp_max_syn_backlog, core/netdev_max_backlog, for a start.
But it would help if you looked at the stats and ifconfig
to see who's dropping packets, how many retransmissions there are, memory failures, or the bottleneck is some other issue altogether...
thanks,
Nivedita
Quoted 6 lines · Nivedita Singhvi
Quoted 4 lines · Christian Schmid
Hi.
This is really annoying. With 3500 sockets onwards, linux 2.6.10 completely lags. This is a bug and I am not willing to buy new servers just because linux has a BUG.
tcp_mem_rmemand_wmemhave been set to 1024000 for testing but this doesnt help as well. so whats WRONG there... please?
Quoted 7 lines
Quoted 2 lines
Best regards,
ChrisYou have not actually said what the problem is - do new connections not get made? Or existing connections slow down?
New connections get made without any problems. Just existing connections slow down painfully.
Quoted 6 lines
You are trying to run many simultaneous connections, so bumping up the individual socket buffer allocation will not necessarily help - you need to bump up the global TCP limit (tcp_mem[]) - it's a 3-tuple - if you have the memory in your system, bump it way up. netstat -tan will tell you if there is unread data in the queues..
I already set it to 1024000 1025000 1026000 (just to be sure). Its a 8 GB system with 2/2 split, so 2 GB of low memory.
Quoted 11 lines
Are you running into memory pressure? Or aborts? netstat -s might give you some info on what is happening.
Bump up the port space (
/proc/sys/net/ipv4/ip_local_port_range) available - typical default is 32K - 61000 (can lower min to 4K)Are they all receiving data or sending? Are they talking to different hosts?
You can increase
tcp_max_syn_backlog, core/netdev_max_backlog, for a start.
netdev_max_backlog has been raised from 300 to 3000 without any result. syn_backlog is normal but its no problem to create new connections. Just existing connections slow down suddenly. Like this:
3000 sockets = no slowdown at all (500 MBit in use)
3300 sockets = 10% slowdown
3600 sockets = 30% slowdown
4000 sockets = 60% slowdown (i aborted here, as it only uses 200 MBit for sending... catastrophy!)
They are all receiving data. Its a download-service. receive-buffer is set to 24 KB and send-buffer set to 224 KB. I don't see a problem with port-space. I only have 3500 sockets when the problem appears but it appears suddenly.
Quoted 3 lines
But it would help if you looked at the stats and ifconfig
to see who's dropping packets, how many retransmissions there are, memory failures, or the bottleneck is some other issue altogether...
No way. Doing 30000 packets per second and your stats are 32 bit integers ;)
Chris
many outgoing tcp sockets are slower than a few
Quoted 2 lines · Christian Schmid
New connections get made without any problems. Just existing connections slow down painfully.
Incoming our outgoing data mostly?
Quoted 10 lines
3000 sockets = no slowdown at all (500 MBit in use)
3300 sockets = 10% slowdown
3600 sockets = 30% slowdown
4000 sockets = 60% slowdown (i aborted here, as it only uses 200 MBit for sending... catastrophy!)They are all receiving data. Its a download-service. receive-buffer is set to 24 KB and send-buffer set to 224 KB. I don't see a problem with port-space. I only have 3500 sockets when the problem appears but it appears suddenly.
I'm a bit confused, it is a download service so you are probably *sending* data?
Quoted 5 lines
Quoted 3 lines
But it would help if you looked at the stats and ifconfig
to see who's dropping packets, how many retransmissions there are, memory failures, or the bottleneck is some other issue altogether...No way. Doing 30000 packets per second and your stats are 32 bit integers ;)
So? You could be a bit more helpful. Sample over 5 seconds, you won't overflow that.
Also, can you tcpdump a bit? Are you using iptables? The conntrack table might be slowing you down.
I have a hunch this problem has to do with high-mem issues though.
Good luck!
Signature · 2 lines
http://www.PowerDNS.com Open source, database driven DNS Software http://netherlabs.nl Open and Closed source services
Re: many outgoing tcp sockets are slower than a few
Quoted 7 lines · bert hubert
Quoted 1 line · Christian Schmid
New connections get made without any problems. Just existing connections slow down painfully.
Incoming our outgoing data mostly?
Outgoing data. I am using sendfile() to send the file on a non-blocking socket but the call blocks for 100 ms per socket if I get over 3000 sockets. Thats causing the massive slowdown in sum. I first thought its a disk-issue but I tried with pure-cache data as well and it still blocks.
Quoted 10 lines
Quoted 6 lines
3000 sockets = no slowdown at all (500 MBit in use)
3300 sockets = 10% slowdown
3600 sockets = 30% slowdown
4000 sockets = 60% slowdown (i aborted here, as it only uses 200 MBit for sending... catastrophy!)They are all receiving data. Its a download-service. receive-buffer is set to 24 KB and send-buffer set to 224 KB. I don't see a problem with port-space. I only have 3500 sockets when the problem appears but it appears suddenly.
I'm a bit confused, it is a download service so you are probably *sending* data?
Only sending. Receiving ACKs of course.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
But it would help if you looked at the stats and ifconfig
to see who's dropping packets, how many retransmissions there are, memory failures, or the bottleneck is some other issue altogether...No way. Doing 30000 packets per second and your stats are 32 bit integers ;)
So? You could be a bit more helpful. Sample over 5 seconds, you won't overflow that.
5 seconds, here you go:
eth0 Link encap:Ethernet HWaddr 00:30:48:27:84:28
inet addr:80.237.244.12 Bcast:80.237.244.63 Mask:255.255.255.192
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2861390424 errors:5991038 dropped:5991056 overruns:38681 frame:4294967287
TX packets:2906616171 errors:4294967279 dropped:0 overruns:0 carrier:4294967278
collisions:4294967289 txqueuelen:1000
RX bytes:1613084024 (1538.3 Mb) TX bytes:2528808392 (2411.6 Mb)
Base address:0x4000 Memory:fc000000-fc020000eth0 Link encap:Ethernet HWaddr 00:30:48:27:84:28
inet addr:80.237.244.12 Bcast:80.237.244.63 Mask:255.255.255.192
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2861511619 errors:5991038 dropped:5991056 overruns:38681 frame:4294967287
TX packets:2906804392 errors:4294967279 dropped:0 overruns:0 carrier:4294967278
collisions:4294967289 txqueuelen:1000
RX bytes:1625289307 (1549.9 Mb) TX bytes:2802520805 (2672.6 Mb)
Base address:0x4000 Memory:fc000000-fc020000Quoted 2 lines
Also, can you tcpdump a bit? Are you using iptables? The conntrack table might be slowing you down.
Hmmm tcpdump what exactly? I am using iptables and the conntrack-problem has been solved in the past already by disabling conn-tracking. The table overran and it dropped packets massively. I disabled conn-tracking and the problem was gone. This is a different problem though.
Quoted 1 line
I have a hunch this problem has to do with high-mem issues though.
Well, 6 GB of high-mem, 2 GB of low-mem... at least there is not too less memory.
MemTotal: 8314392 kB MemFree: 13936 kB Buffers: 28732 kB Cached: 7926792 kB SwapCached: 0 kB Active: 3129028 kB Inactive: 5031240 kB HighTotal: 6421952 kB HighFree: 640 kB LowTotal: 1892440 kB LowFree: 13296 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 54160 kB Writeback: 0 kB Mapped: 210108 kB Slab: 128176 kB CommitLimit: 4157196 kB Committed_AS: 674072 kB PageTables: 1604 kB VmallocTotal: 114680 kB VmallocUsed: 1160 kB VmallocChunk: 113416 kB
Best regards,
Chris
Re: many outgoing tcp sockets are slower than a few
Quoted 4 lines · Christian Schmid
Outgoing data. I am using
sendfile()to send the file on a non-blocking socket but the call blocks for 100 ms per socket if I get over 3000 sockets. Thats causing the massive slowdown in sum. I first thought its a disk-issue but I tried with pure-cache data as well and it still blocks.
O_NONBLOCK send() is really nonblocking, but O_NONBLOCK sendfile() really isn't, as it still does the disk read (if any) synchronously.
How are you making sure that you're sending "pure-cache data"?
--L
Re: many outgoing tcp sockets are slower than a few
Quoted 1 line · Lennert Buytenhek
O_NONBLOCKsend()is really nonblocking, butO_NONBLOCKsendfile()
It is? So it returns EAGAIN if the data to be sent is not in the page cache?
Signature · 2 lines
http://www.PowerDNS.com Open source, database driven DNS Software http://netherlabs.nl Open and Closed source services
Re: many outgoing tcp sockets are slower than a few
Quoted 4 lines · bert hubert
Quoted 1 line
O_NONBLOCKsend()is really nonblocking, butO_NONBLOCKsendfile()It is? So it returns EAGAIN if the data to be sent is not in the page cache?
No, it just blocks. It only returns EAGAIN if the data is in the page cache but the socket TX queue doesn't have enough space. (At least, when I last tried it.)
--L
Re: many outgoing tcp sockets are slower than a few
In reply to Lennert BuytenhekQuoted 10 lines · Lennert Buytenhek
Quoted 1 line · Christian Schmid
Outgoing data. I am using
sendfile()to send the file on a non-blocking socket but the call blocks for 100 ms per socket if I get over 3000 sockets. Thats causing the massive slowdown in sum. I first thought its a disk-issue but I tried with pure-cache data as well and it still blocks.
O_NONBLOCKsend()is really nonblocking, butO_NONBLOCKsendfile()really isn't, as it still does the disk read (if any) synchronously.How are you making sure that you're sending "pure-cache data"?
Because thats the first I excluded. I changed the program by replacing sendfile with a caching-routine plus syswrite. And it was really interesting that the syswrite was the one which needs most of the real-time, not the caching-routine. syswrite blocked 100 ms per socket.
Quoted 5 lines
--L
Re: many outgoing tcp sockets are slower than a few
Quoted 4 lines · Christian Schmid
Because thats the first I excluded. I changed the program by replacing sendfile with a caching-routine plus syswrite. And it was really interesting that the syswrite was the one which needs most of the real-time, not the caching-routine. syswrite blocked 100 ms per socket.
Do you poll/select for socket readiness before doing write() with EAGAIN? (shouldn't matter all that much, but could narrow down debugging)
100ms is a bit of an odd number - how big is your write (in bytes)?
Do you have 3500 threads or processes, or one big one?
What does 'id' say in vmstat 1 typically, both below and above 3500 sockets?
Thanks.
Signature · 2 lines
http://www.PowerDNS.com Open source, database driven DNS Software http://netherlabs.nl Open and Closed source services
Re: many outgoing tcp sockets are slower than a few
Quoted 8 lines · bert hubert
Quoted 1 line · Christian Schmid
Because thats the first I excluded. I changed the program by replacing sendfile with a caching-routine plus syswrite. And it was really interesting that the syswrite was the one which needs most of the real-time, not the caching-routine. syswrite blocked 100 ms per socket.
Do you poll/select for socket readiness before doing
write()with EAGAIN? (shouldn't matter all that much, but could narrow down debugging)
Yes. Using the IO::Poll in Perl.
Quoted 1 line
100ms is a bit of an odd number - how big is your write (in bytes)?
Its an estimation. Its a non-blocking write with sendfile() so it writes around 90 KB if the buffer is set to 128 KB. It writes not the full because POLLOUT comes active before all is empty on the queue.
Quoted 1 line
Do you have 3500 threads or processes, or one big one?
Tried several types. 2 processes with 2000 connections each and 6 processes with 500 connections each. Both versions slow down if the system-wide socket-count goes over 3500.
Quoted 1 line
What does 'id' say in vmstat 1 typically, both below and above 3500 sockets?
(s02) [19:51:33] root:~# vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa 2 2 0 8196 27192 7994020 0 0 2383 275 1540 650 16 19 41 24 0 3 0 12888 27172 7989144 0 0 36872 0 6542 2501 14 20 37 28 1 2 0 8004 27248 7993828 0 0 34988 448 6876 2270 13 19 38 31 0 2 0 17928 27280 7984208 0 0 30660 0 6571 2999 15 20 36 30 3 2 0 9480 27316 7991108 0 0 37168 0 6349 2476 16 22 33 29 2 1 0 9244 27360 7992492 0 0 37616 0 6397 2934 16 23 33 28 0 2 0 8988 27332 7991840 0 0 36628 0 6187 3010 17 21 34 28 3 1 0 9400 27484 7989648 0 0 35556 5708 6072 2696 18 28 27 27
Re: many outgoing tcp sockets are slower than a few
In reply to Christian SchmidQuoted 8 lines · Christian Schmid
Quoted 8 lines
Quoted 4 lines
Outgoing data. I am using
sendfile()to send the file on a non-blocking socket but the call blocks for 100 ms per socket if I get over 3000 sockets. Thats causing the massive slowdown in sum. I first thought its a disk-issue but I tried with pure-cache data as well and it still blocks.
O_NONBLOCKsend()is really nonblocking, butO_NONBLOCKsendfile()really isn't, as it still does the disk read (if any) synchronously.
Quoted 6 lines
Quoted 1 line
How are you making sure that you're sending "pure-cache data"?
Because thats the first I excluded. I changed the program by replacing sendfile with a caching-routine plus syswrite. And it was really interesting that the syswrite was the one which needs most of the real-time, not the caching-routine. syswrite blocked 100 ms per socket.
'syswrite'.
Is your application written in C or perl?
--L
Re: many outgoing tcp sockets are slower than a few
Quoted 8 lines · Lennert Buytenhek
Quoted 5 lines · Christian Schmid
Quoted 5 lines
Quoted 1 line
Outgoing data. I am using
sendfile()to send the file on a non-blocking socket but the call blocks for 100 ms per socket if I get over 3000 sockets. Thats causing the massive slowdown in sum. I first thought its a disk-issue but I tried with pure-cache data as well and it still blocks.
O_NONBLOCKsend()is really nonblocking, butO_NONBLOCKsendfile()really isn't, as it still does the disk read (if any) synchronously.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
How are you making sure that you're sending "pure-cache data"?
Because thats the first I excluded. I changed the program by replacing sendfile with a caching-routine plus syswrite. And it was really interesting that the syswrite was the one which needs most of the real-time, not the caching-routine. syswrite blocked 100 ms per socket.
'syswrite'.
Is your application written in C or perl?
Perl. But this is not the source because as stated in another mail, I tried with 6 processes when it slows down at 500 connections each and with 2 processes where it only starts to slowdown at 2000 processes each.
Chris
Re: many outgoing tcp sockets are slower than a few
In reply to Christian SchmidQuoted 10 lines · Christian Schmid
Quoted 1 line · bert hubert
On Mon, Feb 21, 2005 at 01:35:33AM +0100, Christian Schmid wrote:
Outgoing data. I am using
sendfile()to send the file on a non-blocking socket but the call blocks for 100 ms per socket if I get over 3000 sockets. Thats causing the massive slowdown in sum. I first thought its a disk-issue but I tried with pure-cache data as well and it still blocks.Quoted 5 lines
Quoted 5 lines
3000 sockets = no slowdown at all (500 MBit in use)
3300 sockets = 10% slowdown
3600 sockets = 30% slowdown
4000 sockets = 60% slowdown (i aborted here, as it only uses 200 MBit for sending... catastrophy!)
[snip]
Quoted 4 lines
Quoted 2 lines
I'm a bit confused, it is a download service so you are probably *sending* data?
Only sending. Receiving ACKs of course.
I've been doing some work to improve ACK performance, you can find some patches for Linux 2.6.6 at http://hamilton.ie/net/
I've only tested these patches for a single (or few) very fast connections, but I'd expect the problem might manifest itself for a very large number of connections as well. Though then you might hit other bottlenecks (memory access for different structures).
Baruch