VFS nr_files accounting
I just wanted to report that I am hitting the "VFS: file-max limit xxx reached" problem quite easily on my 32-cpu Niagara machine with 16GB of ram with current 2.6.x GIT.
It seems far too easy to get a box into this state due to SLAB fragmentation and RCU. And once you get a machine into this state it is totally unusable.
Our test case is usually a "make -j8192" kernel build along with a parallel bootstrap of gcc. That puts about 256 processes on each cpu's runqueue, I doubt ksoftirqd can run much at all.
I think part of what helps trigger it might be ccache, which we are using on this machine. ccache seems to open up a ton of files each build invocation.
Usually within an hour of that load you'll hit the nr_files limit and you can't run anything and have to power-cycle.
I think we need to think seriously about this problem.
Quoted 5 lines · David S. Miller
I just wanted to report that I am hitting the "VFS: file-max limit xxx reached" problem quite easily on my 32-cpu Niagara machine with 16GB of ram with current 2.6.x GIT.
Quoted 7 lines
It seems far too easy to get a box into this state due to SLAB fragmentation and RCU. And once you get a machine into this state it is totally unusable.
Our test case is usually a "make -j8192" kernel build along with a parallel bootstrap of gcc. That puts about 256 processes on each cpu's runqueue, I doubt ksoftirqd can run much at all.
Dave, there is a set of patches in -mm that may handle this better -
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.16-rc5/2.6.16-rc5-mm2/broken-out/rcu-batch-tuning.patch
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.16-rc5/2.6.16-rc5-mm2/broken-out/fix-file-counting.patch
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.16-rc5/2.6.16-rc5-mm2/broken-out/fix-file-counting-fixes.patch
Could you please try this in your setup ?
The rcu-batch tuning patch provides automatic switching to process as many RCUs as possible if too many of them are queued. The file counting fixes count the file structures correctly.
Thanks
Dipankar
From: Dipankar Sarma <[email protected]>
Date: Sat, 4 Mar 2006 19:47:17 +0530
Quoted 7 lines
Dave, there is a set of patches in -mm that may handle this better -
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.16-rc5/2.6.16-rc5-mm2/broken-out/rcu-batch-tuning.patch
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.16-rc5/2.6.16-rc5-mm2/broken-out/fix-file-counting.patch
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.16-rc5/2.6.16-rc5-mm2/broken-out/fix-file-counting-fixes.patch
Quoted 5 lines
Could you please try this in your setup ?
The rcu-batch tuning patch provides automatic switching to process as many RCUs as possible if too many of them are queued. The file counting fixes count the file structures correctly.
Thanks, I'll give these patches a spin.
From: "David S. Miller" <[email protected]>
Date: Sat, 04 Mar 2006 14:22:02 -0800 (PST)
Quoted 10 lines
From: Dipankar Sarma <[email protected]>
Date: Sat, 4 Mar 2006 19:47:17 +0530Quoted 7 lines
Dave, there is a set of patches in -mm that may handle this better -
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.16-rc5/2.6.16-rc5-mm2/broken-out/rcu-batch-tuning.patch
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.16-rc5/2.6.16-rc5-mm2/broken-out/fix-file-counting.patch
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.16-rc5/2.6.16-rc5-mm2/broken-out/fix-file-counting-fixes.patch
Quoted 7 lines
Quoted 5 lines
Could you please try this in your setup ?
The rcu-batch tuning patch provides automatic switching to process as many RCUs as possible if too many of them are queued. The file counting fixes count the file structures correctly.
Thanks, I'll give these patches a spin.
Sigh, this is going to take a while, because there are -mm dependencies in these patches such as percpu_counter_sum().
I'll have to fish those out of -mm before I can start testing this.
From: "David S. Miller" <[email protected]>
Date: Sat, 04 Mar 2006 14:28:21 -0800 (PST)
Quoted 5 lines
Sigh, this is going to take a while, because there are -mm dependencies in these patches such as
percpu_counter_sum().I'll have to fish those out of -mm before I can start testing this.
And now that I've sucked in percpu_counter_sum.patch, the rcu-batch-tuning.patch gets a bunch of rejects.
Sorry, I really can't test this. Can you by chance put together a patch against vanilla 2.6.16-GIT? We'll need that to put a fix for this bug into Linus's tree anyways.
Thanks.
Quoted 9 lines · David S. Miller
From: "David S. Miller" <[email protected]>
Date: Sat, 04 Mar 2006 14:28:21 -0800 (PST)And now that I've sucked in percpu_counter_sum.patch, the rcu-batch-tuning.patch gets a bunch of rejects.
Sorry, I really can't test this. Can you by chance put together a patch against vanilla 2.6.16-GIT? We'll need that to put a fix for this bug into Linus's tree anyways.
Dave,
Can you check if the following patchset applies to the latest git ? These were against 2.6.16-rc3.
http://www.hill9.org/linux/kernel/patches/2.6.16-rc3/rcu-batch-tuning.patch
http://www.hill9.org/linux/kernel/patches/2.6.16-rc3/percpu-counter-sum.patch
http://www.hill9.org/linux/kernel/patches/2.6.16-rc3/fix-file-counting.patch
Thanks
Dipankar
From: Dipankar Sarma <[email protected]>
Date: Sun, 5 Mar 2006 12:35:38 +0530
Quoted 6 lines
Can you check if the following patchset applies to the latest git ? These were against 2.6.16-rc3.
http://www.hill9.org/linux/kernel/patches/2.6.16-rc3/rcu-batch-tuning.patch
http://www.hill9.org/linux/kernel/patches/2.6.16-rc3/percpu-counter-sum.patch
http://www.hill9.org/linux/kernel/patches/2.6.16-rc3/fix-file-counting.patch
Applies with some fuzz to kernel/sysctl.c
Thanks a lot!
Quoted 11 lines · David S. Miller
From: Dipankar Sarma <[email protected]>
Date: Sun, 5 Mar 2006 12:35:38 +0530Quoted 6 lines
Can you check if the following patchset applies to the latest git ? These were against 2.6.16-rc3.
http://www.hill9.org/linux/kernel/patches/2.6.16-rc3/rcu-batch-tuning.patch
http://www.hill9.org/linux/kernel/patches/2.6.16-rc3/percpu-counter-sum.patch
http://www.hill9.org/linux/kernel/patches/2.6.16-rc3/fix-file-counting.patchApplies with some fuzz to
kernel/sysctl.c
Great. I look forward to hearing from you about the results with your test case.
Thanks
Dipankar
From: Dipankar Sarma <[email protected]>
Date: Sun, 5 Mar 2006 17:08:47 +0530
Quoted 2 lines
Great. I look forward to hearing from you about the results with your test case.
It works quite fine so far, I haven't seen the filp exhaustion nor a highly fragmented filp SLAB.
Instead, I'm not hitting other bugs that are of my own doing on Niagara, which is what I wanted to accomplish with these stress tests in the first place :-)
I think we should seriously consider these patches for 2.6.16
Quoted 8 lines · David S. Miller
From: Dipankar Sarma <[email protected]>
Date: Sun, 5 Mar 2006 17:08:47 +0530Quoted 2 lines
Great. I look forward to hearing from you about the results with your test case.
It works quite fine so far, I haven't seen the filp exhaustion nor a highly fragmented filp SLAB.
Good to hear that.
Quoted 3 lines
Instead, I'm not hitting other bugs that are of my own doing on Niagara, which is what I wanted to accomplish with these stress tests in the first place :-)
Not good :)
Quoted 1 line
I think we should seriously consider these patches for 2.6.16
Isn't it a little too late in the 2.6.16 cycle ? I would have liked a little more time in -mm. Anyway, it is Linus' call. I can refresh the patches and submit against latest mainline if Linus and Andrew want.
Thanks
Dipankar
From: Dipankar Sarma <[email protected]>
Date: Tue, 7 Mar 2006 12:11:20 +0530
Quoted 7 lines · David S. Miller
Quoted 1 line
I think we should seriously consider these patches for 2.6.16
Isn't it a little too late in the 2.6.16 cycle ? I would have liked a little more time in -mm. Anyway, it is Linus' call. I can refresh the patches and submit against latest mainline if Linus and Andrew want.
Users can run widely published programs to make one's system run out of file descriptors and make the machine totaly unusable.
If that doesn't qualify for something to fix for 2.6.16 I don't know what does. :-)
Quoted 11 lines · David S. Miller
From: Dipankar Sarma <[email protected]>
Date: Tue, 7 Mar 2006 12:11:20 +0530Quoted 7 lines · David S. Miller
Quoted 1 line
I think we should seriously consider these patches for 2.6.16
Isn't it a little too late in the 2.6.16 cycle ? I would have liked a little more time in -mm. Anyway, it is Linus' call. I can refresh the patches and submit against latest mainline if Linus and Andrew want.
Quoted 5 lines
Users can run widely published programs to make one's system run out of file descriptors and make the machine totaly unusable.
If that doesn't qualify for something to fix for 2.6.16 I don't know what does. :-)
Not many people have access to shiny new 8-core 32-thread CPUs (/me turns green saying this) :-)
To be honest I thought Linus' earlier fix that increased RCU maximum batch size to 1000 had more or less fixed the issue for most people. I haven't seen it in my testing, but I agree that we have to take OOMs seriously. I am just being paranoic here.
Thanks
Dipankar
Quoted 9 lines · Dipankar Sarma
Quoted 1 line · David S. Miller
I think we should seriously consider these patches for 2.6.16
Isn't it a little too late in the 2.6.16 cycle ? I would have
liked a little more time in -mm. Anyway, it is Linus' call. I can refresh the patches and submit against latest mainline if Linus and Andrew want.
If it is making the machine unusable then it is a bug rather than just poor behaviour, and definitely a regression.
I think this is very good grounds to get into 2.6.16, even if it does mean pushing the release back.
The other thing that is typically done for regressions like these close to release time is to revert the offending changes. I figure that in this case, such an option is probably _more_ risky.
Signature · 2 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
From: Nick Piggin <[email protected]>
Date: Tue, 07 Mar 2006 17:53:14 +1100
Quoted 3 lines
The other thing that is typically done for regressions like these close to release time is to revert the offending changes. I figure that in this case, such an option is probably _more_ risky.
Especially since we're talking about something that went into 2.6.14
Quoted 9 lines · David S. Miller
From: Nick Piggin <[email protected]>
Date: Tue, 07 Mar 2006 17:53:14 +1100Quoted 3 lines
The other thing that is typically done for regressions like these close to release time is to revert the offending changes. I figure that in this case, such an option is probably _more_ risky.
Especially since we're talking about something that went into 2.6.14
Without a doubt, yes. Waaaay more risky. Compared to that the rcu-batch-tuning and fix-file-counting are a cakewalk. It would however be nice if Christoph or Viro gives the file counting patch a look-over just so that I didn't break any sysctl stuff.
Thanks
Dipankar
Quoted 10 lines · Dipankar Sarma
Quoted 8 lines · David S. Miller
From: Nick Piggin <[email protected]>
Date: Tue, 07 Mar 2006 17:53:14 +1100Quoted 3 lines
The other thing that is typically done for regressions like these close to release time is to revert the offending changes. I figure that in this case, such an option is probably _more_ risky.
Especially since we're talking about something that went into 2.6.14
Quoted 8 lines
Without a doubt, yes. Waaaay more risky. Compared to that the rcu-batch-tuning and fix-file-counting are a cakewalk. It would however be nice if Christoph or Viro gives the file counting patch a look-over just so that I didn't break any sysctl stuff.
Thanks
Dipankar
Considering the impact of the problem, wouldn't be wise to push the fix to -stable as well?
Thanks
Fabio
Signature · 1 line
I'm going to make him an offer he can't refuse.
Quoted 7 lines · Dipankar Sarma
Quoted 2 lines
I think we should seriously consider these patches for 2.6.16
Isn't it a little too late in the 2.6.16 cycle ? I would have liked a little more time in -mm. Anyway, it is Linus' call. I can refresh the patches and submit against latest mainline if Linus and Andrew want.
I'd view a 2.6.16 merge as relatively low-risk. My main concern would be possible breakage of those whacky route-cache workloads.
(I've consoldidated the patches and rebased them against mainline).
Quoted 11 lines · Andrew Morton
Quoted 7 lines · Dipankar Sarma
Quoted 2 lines
I think we should seriously consider these patches for 2.6.16
Isn't it a little too late in the 2.6.16 cycle ? I would have liked a little more time in -mm. Anyway, it is Linus' call. I can refresh the patches and submit against latest mainline if Linus and Andrew want.
I'd view a 2.6.16 merge as relatively low-risk. My main concern would be possible breakage of those whacky route-cache workloads.
Yes, I was hoping that more time in -mm would bring out those whacky corner case OOM/latency problems.
Anyway, here is the kernel paramenter documentation patch. I am not sure if I got the restrictions in square bracket right.
Thanks
Dipankar
Update kernel paramenters documentation for new RCU tuning paramenters.
---
Documentation/kernel-parameters.txt | 13 +++++++++++++ 1 files changed, 13 insertions(+)
Documentation/kernel-parameters.txt renamed from Documentation/kernel-parameters.txt~rcu-tuning-parm-doc +13 -0
@@ -1280,6 +1280,19 @@ running once the system is up. New name for the ramdisk parameter. See Documentation/ramdisk.txt. + rcu.blimit= [KNL,BOOT] Set maximum number of finished + RCU callbacks to process in one batch. + + rcu.qhimark= [KNL,BOOT] Set threshold of queued + RCU callbacks over which batch limiting is disabled. + + rcu.qlowmark= [KNL,BOOT] Set threshold of queued + RCU callbacks below which batch limiting is re-enabled. + + rcu.rsinterval= [KNL,BOOT,SMP] Set the number of additional + RCU callbacks to queued before forcing reschedule + on all cpus. + rdinit= [KNL] Format: <full_path> Run specified binary instead of /init from the ramdisk,
_