two 2.6.13-rc3-mm3 oddities
A message that is not in the archive (sent off-list or not yet received): [email protected]
I am ccing this to linux-kernel for a wider audience.
Quoted 8 lines · Andrew Morton
Subject: two 2.6.13-rc3-mm3 oddities
Just wanted to record a couple of oddities I noticed with 2.6.13-rc3-mm3 (maybe there before: I hardly tested -mm1 and didn't even download -mm2), which have gone away in 2.6.13-rc4-mm1 - so of no great importance, but perhaps worth noting in case they resurface later.
Quoted 4 lines
One time my tmpfs-and-looped-tmpfs-kernel-builds collapsed with lots of
VFS: file-max limit 49778 reached
messages, which I imagine was a side-effect of the struct file RCU patches you've dropped from -rc4-mm1, perhaps a grace period problem.
Hugh, could you please try this with the experimental patch below ? Manfred, is it safe to decrement nr_files in file_free() instead of the destructor ? I can't see any problem.
Quoted 8 lines
And repeatably (after a couple of hours or so), on both i386 and
x86_64, those tests hung waiting on page locks, as if there was a missingunlock_page(or a spurious SetPageLocked) somewhere. I didn't have time to do any bisection, but studying source diffs showed no likely candidates whatsoever (I did wonder about Nick's__ClearPageDirty, but reproduced the problem with that backed out). As I say, gone away (same tests on -rc4-mm1 have run two days), but curious.
Hugh, can you mail me the exact test you run ? I would like to run it myself and see if I can reproduce it.
Thanks
Dipankar
nr_files need to decremented before the file structure is handed over to RCU.
---
fs/file_table.c | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-)
fs/file_table.c renamed from fs/file_table.c~files-fix-nr-files +4 -4
@@ -48,10 +48,6 @@ void filp_ctor(void * objp, struct kmem_void filp_dtor (void * objp,struct kmem_cache_s *cachep,unsigned long dflags) { -unsigned long flags; -spin_lock_irqsave (&filp_count_lock, flags); - files_stat.nr_files--; -spin_unlock_irqrestore (&filp_count_lock, flags); }static inline void file_free_rcu (struct rcu_head *head)
@@ -62,6 +58,10 @@ static inline void file_free_rcu(structstatic inline void file_free (struct file *f) { +unsigned long flags; +spin_lock_irqsave (&filp_count_lock, flags); + files_stat.nr_files--; +spin_unlock_irqrestore (&filp_count_lock, flags);call_rcu (&f->f_rcuhead, file_free_rcu); }
_
Quoted 10 lines · Dipankar Sarma
On Wed, Aug 03, 2005 at 09:56:44AM +1000, Andrew Morton forwarded from Hugh:
Quoted 7 lines
Subject: two 2.6.13-rc3-mm3 oddities
One time my tmpfs-and-looped-tmpfs-kernel-builds collapsed with lots of
VFS: file-max limit 49778 reached
messages, which I imagine was a side-effect of the struct file RCU patches you've dropped from -rc4-mm1, perhaps a grace period problem.Hugh, could you please try this with the experimental patch below ?
I'll give it a go. But first I ought to try to reproduce the problem more easily - it happened after about two hours of a test which had run for two or more hours (before hitting the apparently unrelated problem) at least twice each on at least two machines. So until I can reproduce the problem more easily, it'll take some while to have confidence that your patch addresses it. Therefore worth a little effort at my end to cut it down (and if I can't cut it down, I may have to solve the odder locked page issue to get the original test to run long enough to hit the file-max problem with any likelihood).
Quoted 2 lines
Hugh, can you mail me the exact test you run ? I would like to run it myself and see if I can reproduce it.
It'll waste less of both our time if I at least try to reproduce it more quickly first. Hold on...
Hugh
Quoted 6 lines · Dipankar Sarma
Hugh, could you please try this with the experimental patch below ? Manfred, is it safe to decrement
nr_filesinfile_free()instead of the destructor ? I can't see any problem.
The ctor/dtor are only called when new objects are created, not on every kmem_cache_alloc/kmem_cache_free. Thus I would expect that the counter becomes negative on builds without CONFIG_DEBUG_SLAB. Thus increase in the ctor and decrease in file_free() is the wrong thing. If you want to move the decrease from the dtor to file_free, then you must move the increase, too.
But: IIRC the counters were moved to the ctor/dtor for performance reasons, I'd guess mbligh ran into cache line trashing on the filp_count_lock spinlock with reaim or something like that.
--
Manfred
Quoted 8 lines · Manfred Spraul
Quoted 6 lines · Dipankar Sarma
Hugh, could you please try this with the experimental patch below ? Manfred, is it safe to decrement
nr_filesinfile_free()instead of the destructor ? I can't see any problem.
Quoted 9 lines
The ctor/dtor are only called when new objects are created, not on every kmem_cache_alloc/kmem_cache_free. Thus I would expect that the counter becomes negative on builds without
CONFIG_DEBUG_SLAB.
Thus increase in the ctor and decrease infile_free()is the wrong thing. If you want to move the decrease from the dtor tofile_free, then you must move the increase, too.
But: IIRC the counters were moved to the ctor/dtor for performance reasons, I'd guess mbligh ran into cache line trashing on thefilp_count_lockspinlock with reaim or something like that.
Ah, so the whole idea was to inc/dec nr_files less often so that we reduce contention on filp_count_lock, right ? This however causes skews nr_files by the size of the slab array, AFAICS. Since we check nr_files before we allocate files from slab, the check seems inaccurate.
Anyway, I guess, I need to look at scaling the file counting first.
Thanks
Dipankar
Quoted 20 lines · Dipankar Sarma
Quoted 8 lines · Manfred Spraul
Quoted 6 lines · Dipankar Sarma
Hugh, could you please try this with the experimental patch below ? Manfred, is it safe to decrement
nr_filesinfile_free()instead of the destructor ? I can't see any problem.Quoted 9 lines
The ctor/dtor are only called when new objects are created, not on every kmem_cache_alloc/kmem_cache_free. Thus I would expect that the counter becomes negative on builds without
CONFIG_DEBUG_SLAB.
Thus increase in the ctor and decrease infile_free()is the wrong thing. If you want to move the decrease from the dtor tofile_free, then you must move the increase, too.
But: IIRC the counters were moved to the ctor/dtor for performance reasons, I'd guess mbligh ran into cache line trashing on thefilp_count_lockspinlock with reaim or something like that.
Quoted 8 lines
Ah, so the whole idea was to inc/dec
nr_filesless often so that we reduce contention onfilp_count_lock, right ? This however causes skewsnr_filesby the size of the slab array, AFAICS. Since we checknr_filesbefore we allocate files from slab, the check seems inaccurate.Anyway, I guess, I need to look at scaling the file counting first.
Something like vm_acct_memory() or percpu_counter would suit.
Quoted 12 lines · Andrew Morton
Quoted 11 lines · Dipankar Sarma
Quoted 4 lines
But: IIRC the counters were moved to the ctor/dtor for performance reasons, I'd guess mbligh ran into cache line trashing on the
filp_count_lockspinlock with reaim or something like that.Ah, so the whole idea was to inc/dec
nr_filesless often so that we reduce contention onfilp_count_lock, right ? This however causes skewsnr_filesby the size of the slab array, AFAICS. Since we checknr_filesbefore we allocate files from slab, the check seems inaccurate.
Quoted 4 lines
Quoted 2 lines
Anyway, I guess, I need to look at scaling the file counting first.
Something like
vm_acct_memory()orpercpu_counterwould suit.
Yes, that is what I am doing, except that because of the sysctl stuff, I now have to wallow myself in /proc code.
Thanks
Dipankar