negative dentry_stat.nr_unused causes aggressive dcache pruning
I have seen this stat go negative (just from booting up a multi cpu box), and looking at the code, it is manipulated without locking in a number of places. I have only seen this in real life on a 2.4 kernel, but 2.6 also looks vulnerable.
In 2.4 this can cause shrink_dcache_memory to attempt to push the whole dcache out by calling prune_dcache with a negative parameter, prune_dcache just keeps going until count hits zero or the dentry_unused list empties.
In 2.6 kernels I do not see a real use of the variable which would cause harm, although is someone was to start relying on more than -1 coming back from shrink_dcache_memory there might be problems.
The problem code is dput which is the only place it is manipulated without the dcache lock.
nr_dentry is not mp safe either, but no one depends on that.
Steve
Quoted 5 lines · Steve Lord
I have seen this stat go negative (just from booting up a multi cpu box), and looking at the code, it is manipulated without locking in a number of places. I have only seen this in real life on a 2.4 kernel, but 2.6 also looks vulnerable.
On early 2.6.x, a heavy streaming NFS load was a great way to trigger this. I haven't seen it happen since
http://linus.bkbits.net:8080/linux-2.5/cset@40b8cf606MV-gl6VpDyWKzzW1jaIJw
Greg.
Signature · 2 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI.
Quoted 5 lines · Steve Lord
I have seen this stat go negative (just from booting up a multi cpu box),
and looking at the code, it is manipulated without locking in a number of places. I have only seen this in real life on a 2.4 kernel, but 2.6 also looks vulnerable.
In 2.6, both dentry_stat.nr_unused and dentry_stat.nr_dentry are covered by dcache_lock. I just double-checked and all seems well.
Quoted 11 lines · Andrew Morton
Quoted 4 lines · Steve Lord
I have seen this stat go negative (just from booting up a multi cpu box), and looking at the code, it is manipulated without locking in a number of places. I have only seen this in real life on a 2.4 kernel, but 2.6 also looks vulnerable.
In 2.6, both dentry_stat.nr_unused and dentry_stat.nr_dentry are covered by
dcache_lock. I just double-checked and all seems well.
Looked again, you are right, looks like a 2.4 only problem and even there I am darned if I can see how it leaks now, still digging as to where it is coming from. I have a box which goes negative shortly after bootup, this seems to put it in a feedback loop and it just keeps heading south from there pushing on the dcache. This is running on a fedora legacy kernel for redhat 9.0 which I added kdb to. Starting to wonder if the compiler screwed up somewhere. I have an external fiberchannel driver loaded, but it does not have any dentry related code in it.
Steve
Quoted 11 lines · Andrew Morton
Quoted 4 lines · Steve Lord
I have seen this stat go negative (just from booting up a multi cpu box), and looking at the code, it is manipulated without locking in a number of places. I have only seen this in real life on a 2.4 kernel, but 2.6 also looks vulnerable.
In 2.6, both dentry_stat.nr_unused and dentry_stat.nr_dentry are covered by
dcache_lock. I just double-checked and all seems well.
I still do not know exactly how the count gets negative, but I tracked it down to a user space app from emulex called HBAanywhere. The only thing I can see this doing which might be related is attempting to open a lot of non-existant /proc entries:
/proc/scsi//120 /proc/scsi//121 etc...
Yes there is a // in there.
I ran with a BUG call if we manipulate nr_unused without the dcache lock and it never tripped. All very wierd.
Steve
Quoted 19 lines · Steve Lord
Quoted 11 lines · Andrew Morton
Quoted 4 lines · Steve Lord
I have seen this stat go negative (just from booting up a multi cpu box), and looking at the code, it is manipulated without locking in a number of places. I have only seen this in real life on a 2.4 kernel, but 2.6 also looks vulnerable.
In 2.6, both dentry_stat.nr_unused and dentry_stat.nr_dentry are covered by
dcache_lock. I just double-checked and all seems well.I still do not know exactly how the count gets negative, but I tracked it down to a user space app from emulex called HBAanywhere. The only thing I can see this doing which might be related is attempting to open a lot of non-existant /proc entries:
Quoted 9 lines
/proc/scsi//120 /proc/scsi//121 etc...Yes there is a // in there.
I ran with a BUG call if we manipulate
nr_unusedwithout the dcache lock and it never tripped. All very wierd.
Is that 2.4 or 2.6?
I'd be expecting a systematic counting bug. After all, nr_unused would normally be in the thousands and it'd take a lot of races to get that down to zero.
Quoted 19 lines · Andrew Morton
Quoted 6 lines · Steve Lord
I still do not know exactly how the count gets negative, but I tracked it down to a user space app from emulex called HBAanywhere. The only thing I can see this doing which might be related is attempting to open a lot of non-existant /proc entries:
Quoted 9 lines
/proc/scsi//120 /proc/scsi//121 etc...Yes there is a // in there.
I ran with a BUG call if we manipulate
nr_unusedwithout the dcache lock and it never tripped. All very wierd.
Quoted 6 lines
Is that 2.4 or 2.6?
I'd be expecting a systematic counting bug. After all,
nr_unusedwould normally be in the thousands and it'd take a lot of races to get that down to zero.
2.4.20-31.9 (aka Yea Olde Kernel), it is a redhat 9.0 update from the fedora legacy project.
Something is pushing down hard on the dcache, I can watch the numbers drop after boot up. They start out a few hundred but within a few minutes they go negative.
I have a fiber channel driver from out the tree loaded (emulex lpfcdd) but it does not appear to mess with dcache entries itself. Nothing else from outside the tree. I have not had a chance to try 2.6 yet, not even sure this code would run on 2.6.
I don't have the source of this app, so all I can do is strace it and watch that. Actually I don't even need the app, so I can make the problem go away, but it seems very odd that a user space program can do this.
If I start up the program and watch /proc/sys/fs/dentry-state every second or so I see a sequence something like this:
slord@k4> cat /proc/sys/fs/dentry-state
368 22 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
364 18 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
363 16 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
361 14 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
361 14 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
359 13 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
359 13 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
359 13 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
359 13 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
348 5 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
332 -16 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
332 -16 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
332 -16 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
332 -16 45 0 0 0
slord@k4> cat /proc/sys/fs/dentry-state
332 -16 45 0 0 0
Its not like I am low on memory though:
slord@k4> free
total used free shared buffers cached
Mem: 1030108 244932 785176 0 15012 147808
-/+ buffers/cache: 82112 947996
Swap: 1574360 76 1574284
All I can see the program doing is attempting to open these non-existent /proc entries and doing an ioctl FIBMAP on a /dev/ device bound to the fiber channel driver. Of course, FIBMAP is not FIBMAP in this driver.
It looks like a steaming pile in there, and I think I am just going to assume the driver is guilty until proven innocent.
Steve