Skip to contentSkip to conversation

two 2.6.13-rc3-mm3 oddities

DISCUSSION Dipankar Sarma 4 participants 6 messages Aug 8, 2005 1 message not in the archive VFS

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

I am ccing this to linux-kernel for a wider audience.

Quoted 8 lines · Andrew Morton

Subject: two 2.6.13-rc3-mm3 oddities

Just wanted to record a couple of oddities I noticed with 2.6.13-rc3-mm3 (maybe there before: I hardly tested -mm1 and didn't even download -mm2), which have gone away in 2.6.13-rc4-mm1 - so of no great importance, but perhaps worth noting in case they resurface later.

Quoted 4 lines

One time my tmpfs-and-looped-tmpfs-kernel-builds collapsed with lots of
VFS: file-max limit 49778 reached
messages, which I imagine was a side-effect of the struct file RCU patches you've dropped from -rc4-mm1, perhaps a grace period problem.

Hugh, could you please try this with the experimental patch below ? Manfred, is it safe to decrement nr_files in file_free() instead of the destructor ? I can't see any problem.

Quoted 8 lines

And repeatably (after a couple of hours or so), on both i386 and x86_64, those tests hung waiting on page locks, as if there was a missing unlock_page (or a spurious SetPageLocked) somewhere. I didn't have time to do any bisection, but studying source diffs showed no likely candidates whatsoever (I did wonder about Nick's __ClearPageDirty, but reproduced the problem with that backed out). As I say, gone away (same tests on -rc4-mm1 have run two days), but curious.

Hugh, can you mail me the exact test you run ? I would like to run it myself and see if I can reproduce it.

Thanks
Dipankar

nr_files need to decremented before the file structure is handed over to RCU.

---

 fs/file_table.c |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)
fs/file_table.c renamed from fs/file_table.c~files-fix-nr-files +4 -4
diff -puN fs/file_table.c~files-fix-nr-files fs/file_table.c
@@ -48,10 +48,6 @@ void filp_ctor(void * objp, struct kmem_
 
 void filp_dtor(void * objp, struct kmem_cache_s *cachep, unsigned long dflags)
 {
-	unsigned long flags;
-	spin_lock_irqsave(&filp_count_lock, flags);
-	files_stat.nr_files--;
-	spin_unlock_irqrestore(&filp_count_lock, flags);
 }
 
 static inline void file_free_rcu(struct rcu_head *head)
@@ -62,6 +58,10 @@ static inline void file_free_rcu(struct 
 
 static inline void file_free(struct file *f)
 {
+	unsigned long flags;
+	spin_lock_irqsave(&filp_count_lock, flags);
+	files_stat.nr_files--;
+	spin_unlock_irqrestore(&filp_count_lock, flags);
 	call_rcu(&f->f_rcuhead, file_free_rcu);
 }
 

_

#2 lore raw
Quoted 10 lines · Dipankar Sarma

On Wed, Aug 03, 2005 at 09:56:44AM +1000, Andrew Morton forwarded from Hugh:

Quoted 7 lines

Subject: two 2.6.13-rc3-mm3 oddities

One time my tmpfs-and-looped-tmpfs-kernel-builds collapsed with lots of
VFS: file-max limit 49778 reached
messages, which I imagine was a side-effect of the struct file RCU patches you've dropped from -rc4-mm1, perhaps a grace period problem.

Hugh, could you please try this with the experimental patch below ?

I'll give it a go. But first I ought to try to reproduce the problem more easily - it happened after about two hours of a test which had run for two or more hours (before hitting the apparently unrelated problem) at least twice each on at least two machines. So until I can reproduce the problem more easily, it'll take some while to have confidence that your patch addresses it. Therefore worth a little effort at my end to cut it down (and if I can't cut it down, I may have to solve the odder locked page issue to get the original test to run long enough to hit the file-max problem with any likelihood).

Quoted 2 lines

Hugh, can you mail me the exact test you run ? I would like to run it myself and see if I can reproduce it.

It'll waste less of both our time if I at least try to reproduce it more quickly first. Hold on...

Hugh

#3 lore raw
In reply to Dipankar Sarma
Quoted 6 lines · Dipankar Sarma

Hugh, could you please try this with the experimental patch below ? Manfred, is it safe to decrement nr_files in file_free() instead of the destructor ? I can't see any problem.

The ctor/dtor are only called when new objects are created, not on every kmem_cache_alloc/kmem_cache_free. Thus I would expect that the counter becomes negative on builds without CONFIG_DEBUG_SLAB. Thus increase in the ctor and decrease in file_free() is the wrong thing. If you want to move the decrease from the dtor to file_free, then you must move the increase, too.
But: IIRC the counters were moved to the ctor/dtor for performance reasons, I'd guess mbligh ran into cache line trashing on the filp_count_lock spinlock with reaim or something like that.

--
Manfred

#4 lore raw
Quoted 8 lines · Manfred Spraul
Quoted 6 lines · Dipankar Sarma

Hugh, could you please try this with the experimental patch below ? Manfred, is it safe to decrement nr_files in file_free() instead of the destructor ? I can't see any problem.

Quoted 9 lines

The ctor/dtor are only called when new objects are created, not on every kmem_cache_alloc/kmem_cache_free. Thus I would expect that the counter becomes negative on builds without CONFIG_DEBUG_SLAB.
Thus increase in the ctor and decrease in file_free() is the wrong thing. If you want to move the decrease from the dtor to file_free, then you must move the increase, too.
But: IIRC the counters were moved to the ctor/dtor for performance reasons, I'd guess mbligh ran into cache line trashing on the filp_count_lock spinlock with reaim or something like that.

Ah, so the whole idea was to inc/dec nr_files less often so that we reduce contention on filp_count_lock, right ? This however causes skews nr_files by the size of the slab array, AFAICS. Since we check nr_files before we allocate files from slab, the check seems inaccurate.

Anyway, I guess, I need to look at scaling the file counting first.

Thanks
Dipankar

#5 lore raw
Quoted 20 lines · Dipankar Sarma
Quoted 8 lines · Manfred Spraul
Quoted 6 lines · Dipankar Sarma

Hugh, could you please try this with the experimental patch below ? Manfred, is it safe to decrement nr_files in file_free() instead of the destructor ? I can't see any problem.

Quoted 9 lines

The ctor/dtor are only called when new objects are created, not on every kmem_cache_alloc/kmem_cache_free. Thus I would expect that the counter becomes negative on builds without CONFIG_DEBUG_SLAB.
Thus increase in the ctor and decrease in file_free() is the wrong thing. If you want to move the decrease from the dtor to file_free, then you must move the increase, too.
But: IIRC the counters were moved to the ctor/dtor for performance reasons, I'd guess mbligh ran into cache line trashing on the filp_count_lock spinlock with reaim or something like that.

Quoted 8 lines

Ah, so the whole idea was to inc/dec nr_files less often so that we reduce contention on filp_count_lock, right ? This however causes skews nr_files by the size of the slab array, AFAICS. Since we check nr_files before we allocate files from slab, the check seems inaccurate.

Anyway, I guess, I need to look at scaling the file counting first.

Something like vm_acct_memory() or percpu_counter would suit.

#6 lore raw
Quoted 12 lines · Andrew Morton
Quoted 11 lines · Dipankar Sarma
Quoted 4 lines

But: IIRC the counters were moved to the ctor/dtor for performance reasons, I'd guess mbligh ran into cache line trashing on the filp_count_lock spinlock with reaim or something like that.

Ah, so the whole idea was to inc/dec nr_files less often so that we reduce contention on filp_count_lock, right ? This however causes skews nr_files by the size of the slab array, AFAICS. Since we check nr_files before we allocate files from slab, the check seems inaccurate.

Quoted 4 lines
Quoted 2 lines

Anyway, I guess, I need to look at scaling the file counting first.

Something like vm_acct_memory() or percpu_counter would suit.

Yes, that is what I am doing, except that because of the sysctl stuff, I now have to wallow myself in /proc code.

Thanks
Dipankar