Skip to contentSkip to conversation

negative dentry_stat.nr_unused causes aggressive dcache pruning

DISCUSSION Steve Lord 3 participants 7 messages Dec 8, 2004 – Dec 9, 2004

lore
Share & tools
lore

Steve Lord

author
#0 lore raw

I have seen this stat go negative (just from booting up a multi cpu box), and looking at the code, it is manipulated without locking in a number of places. I have only seen this in real life on a 2.4 kernel, but 2.6 also looks vulnerable.

In 2.4 this can cause shrink_dcache_memory to attempt to push the whole dcache out by calling prune_dcache with a negative parameter, prune_dcache just keeps going until count hits zero or the dentry_unused list empties.

In 2.6 kernels I do not see a real use of the variable which would cause harm, although is someone was to start relying on more than -1 coming back from shrink_dcache_memory there might be problems.

The problem code is dput which is the only place it is manipulated without the dcache lock.

nr_dentry is not mp safe either, but no one depends on that.

Steve

#1 lore raw
Quoted 5 lines · Steve Lord

I have seen this stat go negative (just from booting up a multi cpu box), and looking at the code, it is manipulated without locking in a number of places. I have only seen this in real life on a 2.4 kernel, but 2.6 also looks vulnerable.

On early 2.6.x, a heavy streaming NFS load was a great way to trigger this. I haven't seen it happen since

http://linus.bkbits.net:8080/linux-2.5/cset@40b8cf606MV-gl6VpDyWKzzW1jaIJw

Greg.

Signature · 2 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
I don't speak for SGI.
#2 lore raw
In reply to Steve Lord
Quoted 5 lines · Steve Lord

I have seen this stat go negative (just from booting up a multi cpu box),
and looking at the code, it is manipulated without locking in a number of places. I have only seen this in real life on a 2.4 kernel, but 2.6 also looks vulnerable.

In 2.6, both dentry_stat.nr_unused and dentry_stat.nr_dentry are covered by dcache_lock. I just double-checked and all seems well.

author
#3 lore raw
Quoted 11 lines · Andrew Morton
Quoted 4 lines · Steve Lord

I have seen this stat go negative (just from booting up a multi cpu box), and looking at the code, it is manipulated without locking in a number of places. I have only seen this in real life on a 2.4 kernel, but 2.6 also looks vulnerable.

In 2.6, both dentry_stat.nr_unused and dentry_stat.nr_dentry are covered by dcache_lock. I just double-checked and all seems well.

Looked again, you are right, looks like a 2.4 only problem and even there I am darned if I can see how it leaks now, still digging as to where it is coming from. I have a box which goes negative shortly after bootup, this seems to put it in a feedback loop and it just keeps heading south from there pushing on the dcache. This is running on a fedora legacy kernel for redhat 9.0 which I added kdb to. Starting to wonder if the compiler screwed up somewhere. I have an external fiberchannel driver loaded, but it does not have any dentry related code in it.

Steve

author
#4 lore raw
In reply to Andrew Morton
Quoted 11 lines · Andrew Morton
Quoted 4 lines · Steve Lord

I have seen this stat go negative (just from booting up a multi cpu box), and looking at the code, it is manipulated without locking in a number of places. I have only seen this in real life on a 2.4 kernel, but 2.6 also looks vulnerable.

In 2.6, both dentry_stat.nr_unused and dentry_stat.nr_dentry are covered by dcache_lock. I just double-checked and all seems well.

I still do not know exactly how the count gets negative, but I tracked it down to a user space app from emulex called HBAanywhere. The only thing I can see this doing which might be related is attempting to open a lot of non-existant /proc entries:

	/proc/scsi//120
	/proc/scsi//121
	etc...

Yes there is a // in there.

I ran with a BUG call if we manipulate nr_unused without the dcache lock and it never tripped. All very wierd.

Steve

#5 lore raw
Quoted 19 lines · Steve Lord
Quoted 11 lines · Andrew Morton
Quoted 4 lines · Steve Lord

I have seen this stat go negative (just from booting up a multi cpu box), and looking at the code, it is manipulated without locking in a number of places. I have only seen this in real life on a 2.4 kernel, but 2.6 also looks vulnerable.

In 2.6, both dentry_stat.nr_unused and dentry_stat.nr_dentry are covered by dcache_lock. I just double-checked and all seems well.

I still do not know exactly how the count gets negative, but I tracked it down to a user space app from emulex called HBAanywhere. The only thing I can see this doing which might be related is attempting to open a lot of non-existant /proc entries:

Quoted 9 lines
	/proc/scsi//120
	/proc/scsi//121
	etc...

Yes there is a // in there.

I ran with a BUG call if we manipulate nr_unused without the dcache lock and it never tripped. All very wierd.

Is that 2.4 or 2.6?

I'd be expecting a systematic counting bug. After all, nr_unused would normally be in the thousands and it'd take a lot of races to get that down to zero.

author
#6 lore raw
Quoted 19 lines · Andrew Morton
Quoted 6 lines · Steve Lord

I still do not know exactly how the count gets negative, but I tracked it down to a user space app from emulex called HBAanywhere. The only thing I can see this doing which might be related is attempting to open a lot of non-existant /proc entries:

Quoted 9 lines
	/proc/scsi//120
	/proc/scsi//121
	etc...

Yes there is a // in there.

I ran with a BUG call if we manipulate nr_unused without the dcache lock and it never tripped. All very wierd.

Quoted 6 lines

Is that 2.4 or 2.6?

I'd be expecting a systematic counting bug. After all, nr_unused would normally be in the thousands and it'd take a lot of races to get that down to zero.

2.4.20-31.9 (aka Yea Olde Kernel), it is a redhat 9.0 update from the fedora legacy project.

Something is pushing down hard on the dcache, I can watch the numbers drop after boot up. They start out a few hundred but within a few minutes they go negative.

I have a fiber channel driver from out the tree loaded (emulex lpfcdd) but it does not appear to mess with dcache entries itself. Nothing else from outside the tree. I have not had a chance to try 2.6 yet, not even sure this code would run on 2.6.

I don't have the source of this app, so all I can do is strace it and watch that. Actually I don't even need the app, so I can make the problem go away, but it seems very odd that a user space program can do this.

If I start up the program and watch /proc/sys/fs/dentry-state every second or so I see a sequence something like this:
slord@k4> cat /proc/sys/fs/dentry-state
368 22 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
364 18 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
363 16 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
361 14 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
361 14 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
359 13 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
359 13 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
359 13 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
359 13 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
348 5 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
332 -16 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
332 -16 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
332 -16 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
332 -16 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
332 -16 45 0 0 0

Its not like I am low on memory though:
slord@k4> free
total used free shared buffers cached
Mem: 1030108 244932 785176 0 15012 147808
-/+ buffers/cache: 82112 947996
Swap: 1574360 76 1574284

All I can see the program doing is attempting to open these non-existent /proc entries and doing an ioctl FIBMAP on a /dev/ device bound to the fiber channel driver. Of course, FIBMAP is not FIBMAP in this driver.

It looks like a steaming pile in there, and I think I am just going to assume the driver is guilty until proven innocent.

Steve