Bug-hunting
Subsystems (2)
Hello.
I am still trying to hunt down the bug with the slowdown-on-many-sockets. Is there any way I can see how much tcp-memory is used right now? Why did you change the behaviour? In 2.6.10-rc2 I was able to see the amount by looking in slabinfo but now the buffers are gone. And where did you introduce the buffer-limit? It seems its now globally limited to xxx MB. I want to disable this in order to check if thats the reason.
Chris
David S. Miller
maintainerQuoted 5 lines · On Tue, 22 Feb 2005 16:07:35 +0100 Christian Schmid <[email protected]>…
I am still trying to hunt down the bug with the slowdown-on-many-sockets. Is there any way I can see how much tcp-memory is used right now? Why did you change the behaviour? In 2.6.10-rc2 I was able to see the amount by looking in slabinfo but now the buffers are gone. And where did you introduce the buffer-limit? It seems its now globally limited to xxx MB. I want to disable this in order to check if thats the reason.
The global TCP memory limit, controllable by sysctl()'s, has been there for at least 3 years.
No. This is a new thing and this wasnt there before. In 2.6.10-rc2 the kernel aborted programs with "Out of memory" when too many buffers are allocated and low memory was full. NOW it just shrinks the buffers dynamically. I don't want that. I have a 2/2 system and I want 1600 MB for buffers but you only allow around 700 MB for buffers. This is definetly NEW.
Quoted 11 lines · David S. Miller
Quoted 1 line · On Tue, 22 Feb 2005 16:07:35 +0100 Christian Schmid <[email protected]>…
I am still trying to hunt down the bug with the slowdown-on-many-sockets. Is there any way I can see how much tcp-memory is used right now? Why did you change the behaviour? In 2.6.10-rc2 I was able to see the amount by looking in slabinfo but now the buffers are gone. And where did you introduce the buffer-limit? It seems its now globally limited to xxx MB. I want to disable this in order to check if thats the reason.
The global TCP memory limit, controllable by sysctl()'s, has been there for at least 3 years.
David S. Miller
maintainerQuoted 4 lines · On Tue, 22 Feb 2005 20:17:55 +0100 Christian Schmid <[email protected]>…
No. This is a new thing and this wasnt there before. In 2.6.10-rc2 the kernel aborted programs with "Out of memory" when too many buffers are allocated and low memory was full. NOW it just shrinks the buffers dynamically. I don't want that. I have a 2/2 system and I want 1600 MB for buffers but you only allow around 700 MB for buffers. This is definetly NEW.
It always shrinks the TCP socket buffer usage when the total socket memory used by TCP is over the threshold. This code has been there for 3 years at least.
If the behavior has changed, that's interesting and probably due to some other change. It could even be a MM layer change that is causing things to behave differently now for you, and because things are being tweaked in the memory management all the time (particularly the handling of lowmem vs. highmem) this would not surprise me at all.
But the basic framework for shrinking socket buffers when total TCP memory usage crosses some threshold has been there and has been enabled for a long time.
Anyways, we're stalled on figuring out exactly what is wrong due to lack of information and difficulty in reproducing. The ball is still in your court.
Why don't you put together a very simple test case that others can use to analyze and reproduce your bug? I bet we'll fix it or figure out what is wrong with your app within 24 hours once you do that. :-)
Hm can you tell me how to change the limit? It seems the limit is good for 3/1 systems but not for 2/2 systems.
To reproduce this, you would have to create a server which is sending data to 4000 non-blocking sockets. It doesnt matter what you send. You should realize a slow-down the more sockets you use. You need Gigabit for this to appear.
Quoted 31 lines · David S. Miller
Quoted 1 line · On Tue, 22 Feb 2005 20:17:55 +0100 Christian Schmid <[email protected]>…
No. This is a new thing and this wasnt there before. In 2.6.10-rc2 the kernel aborted programs with "Out of memory" when too many buffers are allocated and low memory was full. NOW it just shrinks the buffers dynamically. I don't want that. I have a 2/2 system and I want 1600 MB for buffers but you only allow around 700 MB for buffers. This is definetly NEW.
It always shrinks the TCP socket buffer usage when the total socket memory used by TCP is over the threshold. This code has been there for 3 years at least.
If the behavior has changed, that's interesting and probably due to some other change. It could even be a MM layer change that is causing things to behave differently now for you, and because things are being tweaked in the memory management all the time (particularly the handling of lowmem vs. highmem) this would not surprise me at all.
But the basic framework for shrinking socket buffers when total TCP memory usage crosses some threshold has been there and has been enabled for a long time.
Anyways, we're stalled on figuring out exactly what is wrong due to lack of information and difficulty in reproducing. The ball is still in your court.
Why don't you put together a very simple test case that others can use to analyze and reproduce your bug? I bet we'll fix it or figure out what is wrong with your app within 24 hours once you do that. :-)
David S. Miller
maintainerQuoted 3 lines · On Wed, 23 Feb 2005 00:21:31 +0100 Christian Schmid <[email protected]>…
To reproduce this, you would have to create a server which is sending data to 4000 non-blocking sockets. It doesnt matter what you send. You should realize a slow-down the more sockets you use. You need Gigabit for this to appear.
Please send us the exact test programs to run, not some general description of what "one should do" to reproduce the problem.
This is exactly what is stalling this bug getting diagnosed properly and fixed.
I would have to write a test-program first. This needs time. Before I spend hours into coding this, first just tell me how I can raise the global-limit from around 600-700 MB to 1500 MB.
Quoted 14 lines · David S. Miller
Quoted 1 line · On Wed, 23 Feb 2005 00:21:31 +0100 Christian Schmid <[email protected]>…
To reproduce this, you would have to create a server which is sending data to 4000 non-blocking sockets. It doesnt matter what you send. You should realize a slow-down the more sockets you use. You need Gigabit for this to appear.
Please send us the exact test programs to run, not some general description of what "one should do" to reproduce the problem.
This is exactly what is stalling this bug getting diagnosed properly and fixed.
David S. Miller
maintainerQuoted 2 lines · On Wed, 23 Feb 2005 11:28:24 +0100 Christian Schmid <[email protected]>…
I would have to write a test-program first. This needs time. Before I spend hours into coding this, first just tell me how I can raise the global-limit from around 600-700 MB to 1500 MB.
The limit is in the tcp_mem sysctl, I believe we told you this before.
/proc/sys/net/ipv4/tcp_mem?
Its set to 98304 131072 196608
I have changed this to 1024000 1025000 1026000
It didnt change anything. The send-buffer still keeps being shrinked down.
Quoted 11 lines · David S. Miller
Quoted 1 line · On Wed, 23 Feb 2005 11:28:24 +0100 Christian Schmid <[email protected]>…
I would have to write a test-program first. This needs time. Before I spend hours into coding this, first just tell me how I can raise the global-limit from around 600-700 MB to 1500 MB.
The limit is in the
tcp_memsysctl, I believe we told you this before.
David S. Miller
maintainerQuoted 6 lines · On Wed, 23 Feb 2005 20:41:39 +0100 Christian Schmid <[email protected]>…
/proc/sys/net/ipv4/tcp_mem?Its set to 98304 131072 196608
I have changed this to 1024000 1025000 1026000It didnt change anything. The send-buffer still keeps being shrinked down.
That's why I felt silly telling you again where the limit is configured.
Please write the test program already.
Can I see somewhere how much memory tcp needs right now so to see if that changes effect something? Maybe the error is somewhere else so I need to check if memory-usage raises if I change the values.
Quoted 11 lines · David S. Miller
Quoted 1 line · On Wed, 23 Feb 2005 11:28:24 +0100 Christian Schmid <[email protected]>…
I would have to write a test-program first. This needs time. Before I spend hours into coding this, first just tell me how I can raise the global-limit from around 600-700 MB to 1500 MB.
The limit is in the
tcp_memsysctl, I believe we told you this before.
Christian,
Try looking at /proc/net/netstat while running. If TCPMemoryPressures is increasing, you are running out of TCP memory, and you would expect to see your socket buffers shrinking.
-John
More track-downs:
I have watched /proc/net/netstat and it didnt change. I experimented more with vm-parameters as I have understood that Linux is complex enough that bugs aren't obvious enough to simply say its the net-code just because a non-blocking socket suddenly blocks.
I found out that changing /proc/sys/vm/min_free_kbytes gives the most drastic differences. I changed it to 1024000 and it suddenly speeded up drastically. I changed back to around 16000 and it suddenly slowed down immediately. Changed back to 1024000 and it speeded up immediately. Does someone know more about this? What does this parameter do? I watched /proc/meminfo but it didnt change anything really.MemFree is always around 10 MB:
MemTotal: 8314392 kB MemFree: 10284 kB Buffers: 25644 kB Cached: 7724180 kB SwapCached: 0 kB Active: 991312 kB Inactive: 6977476 kB HighTotal: 6421952 kB HighFree: 640 kB LowTotal: 1892440 kB LowFree: 9644 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 116272 kB Writeback: 0 kB Mapped: 223940 kB Slab: 322652 kB CommitLimit: 4157196 kB Committed_AS: 794200 kB PageTables: 1788 kB VmallocTotal: 114680 kB VmallocUsed: 1200 kB VmallocChunk: 113392 kB
Quoted 9 lines · John Heffner
Christian,
Try looking at
/proc/net/netstatwhile running. If TCPMemoryPressures is increasing, you are running out of TCP memory, and you would expect to see your socket buffers shrinking.-John