Busy inodes after unmount, be more verbose in generic_shutdown_super
PATCH 1/1 Busy inodes after unmount, be more verbose in generic_shutdown_super
I get 'Busy inodes after umount' very often, even with recent kernels. Usually it remains unnoticed for a while. A bit more info about what superblock had problems would be helpful.
bdevname() doesnt seem to be the best way for pretty-printing NFS mounts for example. Should it just print the major:minor pair?
Are there scripts or something that parse such kernel messages, should the extra info go somewhere else?
fs/super.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-)
fs/super.c +4 -2
@@ -227,6 +227,7 @@ void generic_shutdown_super(struct super {struct dentry *root = sb->s_root;struct super_operations *sop = sb->s_op; +char b[BDEVNAME_SIZE ];if (root) { sb->s_root =NULL ;
@@ -247,8 +248,9 @@ void generic_shutdown_super(struct super/* Forget any remaining inodes */ if (invalidate_inodes (sb)) { -printk ("VFS: Busy inodes after unmount. " -"Self-destruct in 5 seconds. Have a nice day...\n" ); +printk ("VFS: (%s) Busy inodes after unmount. " +"Self-destruct in 5 seconds. Have a nice day...\n" , +bdevname (sb->s_bdev, b)); }unlock_kernel ();
Signature · 2 lines
short story of a lazy sysadmin: alias appserv=wotan
Olaf, can you please check if my patch for busy inodes from -mm tree helps you? Patch name is fix-of-dcache-race-leading-to-busy-inodes-on-umount.patch
http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.15/2.6.15-mm4/broken-out/fix-of-dcache-race-leading-to-busy-inodes-on-umount.patch
I can also provide you a more sophisticated debug patch for this if required.
Kirill
Quoted 35 lines
I get 'Busy inodes after umount' very often, even with recent kernels. Usually it remains unnoticed for a while. A bit more info about what superblock had problems would be helpful.
bdevname()doesnt seem to be the best way for pretty-printing NFS mounts
for example. Should it just print the major:minor pair? Are there scripts or something that parse such kernel messages, should the extra info go somewhere else?fs/super.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-)Index:
linux-2.6/fs/super.c
===================================================================fs/super.c +4
-2@@ -227,6 +227,7 @@ void generic_shutdown_super(struct super {struct dentry *root = sb->s_root;struct super_operations *sop = sb->s_op; +char b[BDEVNAME_SIZE ];if (root) { sb->s_root =NULL ;@@ -247,8 +248,9 @@ void generic_shutdown_super(struct super/* Forget any remaining inodes */ if (invalidate_inodes (sb)) { -printk ("VFS: Busy inodes after unmount. " -"Self-destruct in 5 seconds. Have a nice day...\n" ); +printk ("VFS: (%s) Busy inodes after unmount. " +"Self-destruct in 5 seconds. Have a nice day...\n" , +bdevname (sb->s_bdev, b)); }unlock_kernel ();
Quoted 2 lines · Kirill Korotaev
Olaf, can you please check if my patch for busy inodes from -mm tree helps you?
I cant reprpoduce it at will, thats the thing. It likely happens with NFS mounts. [email protected] did some work recently. But I remember even with these changes (for a 2.6.13), the busy inodes did not disappear.
Merging your patch into our cvs will give it more testing, I will do that tomorrow if noone disagrees.
Signature · 2 lines
short story of a lazy sysadmin: alias appserv=wotan
Quoted 6 lines · Olaf Hering
Quoted 2 lines · Kirill Korotaev
Olaf, can you please check if my patch for busy inodes from -mm tree helps you?
Quoted 7 lines
I cant reprpoduce it at will, thats the thing. It likely happens with NFS mounts. [email protected] did some work recently. But I remember even with these changes (for a 2.6.13), the busy inodes did not disappear.
Merging your patch into our cvs will give it more testing, I will do that tomorrow if noone disagrees.
The patch is certainly safe and stable. But it's so huge and complex and ugly that I was hoping that a better fix would turn up. The bug itself takes quite some ingenuity to hit.
Quoted 8 lines
Quoted 7 lines · Olaf Hering
Quoted 1 line · Kirill Korotaev
Olaf, can you please check if my patch for busy inodes from -mm tree helps you?
I cant reprpoduce it at will, thats the thing. It likely happens with NFS mounts. [email protected] did some work recently. But I remember even with these changes (for a 2.6.13), the busy inodes did not disappear.
Quoted 7 lines
Quoted 3 lines
Merging your patch into our cvs will give it more testing, I will do that tomorrow if noone disagrees.
The patch is certainly safe and stable. But it's so huge and complex and ugly that I was hoping that a better fix would turn up. The bug itself takes quite some ingenuity to hit.
We have another idea how to reimplement it via refcounters instead of lists. But I'm not sure when this will happen, due to lack of time :(
Kirill
Quoted 4 lines · Kirill Korotaev
Olaf, can you please check if my patch for busy inodes from -mm tree helps you?
Patch name is fix-of-dcache-race-leading-to-busy-inodes-on-umount.patch
http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.15/2.6.15-mm4/broken-out/fix-of-dcache-race-leading-to-busy-inodes-on-umount.patch
This patch is just wrong. It is hiding bugs in file systems. The problem is that somewhere the reference counting on the vfsmount objects is wrong. The file system is unmounted before the last dentry is dereferenced. Either you didn't hold a reference to the proper vfsmount objects at all or you dereference it too early. See Al Viros patch series (search for "namei fixes") on how to fix this issues.
Regards,
Jan
Signature · 4 lines
Jan Blunck [email protected] SuSE LINUX AG - A Novell company Maxfeldstr. 5 +49-911-74053-608 D-90409 Nürnberg http://www.suse.de
Quoted 8 lines · Jan Blunck
Quoted 4 lines · Kirill Korotaev
Olaf, can you please check if my patch for busy inodes from -mm tree helps you?
Patch name is fix-of-dcache-race-leading-to-busy-inodes-on-umount.patch
http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.15/2.6.15-mm4/broken-out/fix-of-dcache-race-leading-to-busy-inodes-on-umount.patch
Quoted 7 lines
This patch is just wrong. It is hiding bugs in file systems. The problem is that somewhere the reference counting on the vfsmount objects is wrong. The file system is unmounted before the last dentry is dereferenced. Either you didn't hold a reference to the proper vfsmount objects at all or you dereference it too early. See Al Viros patch series (search for "namei fixes") on how to fix this issues.
The only reason I've been carrying that patch is as a reminder that there's a bug that we need to fix. It'd be good news if that bug had been fixed by other means.
Kirill, do you know whether the bug is still present in 2.6.16-rc1?
Quoted 11 lines
Quoted 3 lines
Olaf, can you please check if my patch for busy inodes from -mm tree helps you? Patch name is fix-of-dcache-race-leading-to-busy-inodes-on-umount.patch
http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.15/2.6.15-mm4/broken-out/fix-of-dcache-race-leading-to-busy-inodes-on-umount.patchThis patch is just wrong. It is hiding bugs in file systems. The problem is that somewhere the reference counting on the vfsmount objects is wrong. The file system is unmounted before the last dentry is dereferenced. Either you didn't hold a reference to the proper vfsmount objects at all or you dereference it too early. See Al Viros patch series (search for "namei fixes") on how to fix this issues.
This patch has nothing to do with vfsmount references and doesn't hide anything. It just adds syncronization barrier between do_umount() and shrink_dcache() since the latter can work with dentries/inodes without holding locks.
So if you think there is something wrong with it, please, be more specific.
Kirill
Quoted 7 lines · Kirill Korotaev
This patch has nothing to do with vfsmount references and doesn't hide anything. It just adds syncronization barrier between
do_umount()andshrink_dcache()since the latter can work with dentries/inodes without holding locks.So if you think there is something wrong with it, please, be more specific.
You can only unmount a file system if there are no references to the vfsmount object anymore. Since shrink_dcache*() is called after checking the refcount of vfsmount while unmounting the file system, it isn't possible to hold a reference to a dentry (and therefore call dput()) after this point in time. Therefore your reference counting on the vfsmount is wrong which is the root case for your problem of busy inodes.
Regards,
Jan
Signature · 4 lines
Jan Blunck [email protected] SuSE LINUX AG - A Novell company Maxfeldstr. 5 +49-911-74053-608 D-90409 Nürnberg http://www.suse.de
Quoted 15 lines
Quoted 7 lines · Kirill Korotaev
Quoted 5 lines
Olaf, can you please check if my patch for busy inodes from -mm tree helps you? Patch name is fix-of-dcache-race-leading-to-busy-inodes-on-umount.patch
http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.15/2.6.15-mm4/broken-out/fix-of-dcache-race-leading-to-busy-inodes-on-umount.patchQuoted 6 lines
This patch is just wrong. It is hiding bugs in file systems. The problem is that somewhere the reference counting on the vfsmount objects is wrong. The file system is unmounted before the last dentry is dereferenced. Either you didn't hold a reference to the proper vfsmount objects at all or you dereference it too early. See Al Viros patch series (search for "namei fixes") on how to fix this issues.
Quoted 5 lines
The only reason I've been carrying that patch is as a reminder that there's a bug that we need to fix. It'd be good news if that bug had been fixed by other means.
Kirill, do you know whether the bug is still present in 2.6.16-rc1?
it exists in 2.6.15 and I see no changes in 2.6.16-rc1 except for cosmetics :( checked the git tree, dput() etc. the bug is definetely still here - nothing changed in this area.
Sorry for bad news.
The patch can be probably remade via introducing "notlocked_refs" counter on dentry. if shrinker()/umount() see such a dentry() with non-zero refcnt it can sleep as it is done in current patch. It would be a little bit cleaner/simpler. What do you think? If someone suggest any brilliant/helpfull idea I would be happy to improve it.
Kirill
Quoted 6 lines
Quoted 4 lines
This patch has nothing to do with vfsmount references and doesn't hide anything. It just adds syncronization barrier between
do_umount()andshrink_dcache()since the latter can work with dentries/inodes without holding locks.So if you think there is something wrong with it, please, be more specific.
Quoted 6 lines
You can only unmount a file system if there are no references to the vfsmount object anymore. Since shrink_dcache*() is called after checking the refcount of vfsmount while unmounting the file system, it isn't possible to hold a reference to a dentry (and therefore call
dput()) after this point in time. Therefore your reference counting on the vfsmount is wrong which is the root case for your problem of busy inodes.
You didn't take into account shrink_dcache*() on memory pressure. It works when it works. And when it calls dput() it detaches dentry from the whole tree and starts to work with inode. do_umount() can successfully shrink the other part of the tree, since dentry in question is detached, complain about busy inode (it is really being put on another CPU, but still busy) and destroy super block.
another scenario from patch comment:
CPU 1 CPU 2
~~~~~ ~~~~~
umount /dev/sda1
generic_shutdown_super shrink_dcache_memory() shrink_dcache_parent dput dentry select_parent prune_one_dentry()
<<<< child is dead, locks are released,
but parent is still referenced!!! >>>>
skip dentry->parent,
since it's d_count > 0
message: BUSY inodes after umount...
<<< parent is left on dentry_unused list,
referencing freed super block >>>
Kirill
Quoted 6 lines · Kirill Korotaev
CPU 1 CPU 2
~~~~~ ~~~~~
umount/dev/sda1generic_shutdown_super shrink_dcache_memory() shrink_dcache_parent dput dentry select_parent prune_one_dentry()
Quoted 8 lines
<<<< child is dead, locks are released,
but parent is still referenced!!! >>>>
skipdentry->parent,
since it'sd_count> 0message: BUSY inodes after umount...
<<< parent is left ondentry_unusedlist,
referencing freed super block >>>
I see. The problem is that dcache_lock is given up before dereferencing the parent. But your patch seems to be wrong anyway IMHO. I'll post patches in a seperate thread.
Regards,
Jan
Signature · 4 lines
Jan Blunck [email protected] SuSE LINUX AG - A Novell company Maxfeldstr. 5 +49-911-74053-608 D-90409 Nürnberg http://www.suse.de
Quoted 11 lines
Quoted 11 lines
CPU 1 CPU 2
~~~~~ ~~~~~
umount/dev/sda1generic_shutdown_super shrink_dcache_memory() shrink_dcache_parent dput dentry select_parent prune_one_dentry()<<<< child is dead, locks are released,
but parent is still referenced!!! >>>>
skipdentry->parent,
since it'sd_count> 0
Quoted 8 lines
Quoted 3 lines
message: BUSY inodes after umount...
<<< parent is left ondentry_unusedlist,
referencing freed super block >>>I see. The problem is that
dcache_lockis given up before dereferencing the parent. But your patch seems to be wrong anyway IMHO. I'll post patches in a seperate thread.
Jan, I still have not heard a single comment about what's wrong with it... I would really appreciate if you provide me one.
Kirill
Quoted 3 lines · Kirill Korotaev
Jan, I still have not heard a single comment about what's wrong with it... I would really appreciate if you provide me one.
Sorry for the delay. I had to fix a totally bogus patch (mine ;).
The problem with your patch is that it hides too early mntput's. Think about following situation:
mntput(path->mnt); // too early mntput()
dput(path->dentry);
Assuming that in-between this sequence someone unmounts the file system, your patch will wait for this dput() to finish before it proceeds with unmounting the file system. I think this isn't what we want.
Regards,
Jan
Signature · 4 lines
Jan Blunck [email protected] SuSE LINUX AG - A Novell company Maxfeldstr. 5 +49-911-74053-608 D-90409 Nürnberg http://www.suse.de
Hello Jan,
Quoted 9 lines
Quoted 2 lines
Jan, I still have not heard a single comment about what's wrong with it... I would really appreciate if you provide me one.
Sorry for the delay. I had to fix a totally bogus patch (mine ;).
The problem with your patch is that it hides too early mntput's. Think about following situation:
Quoted 6 lines
mntput(path->mnt); // too earlymntput()
dput(path->dentry);Assuming that in-between this sequence someone unmounts the file system, your patch will wait for this
dput()to finish before it proceeds with unmounting the file system. I think this isn't what we want.
No, it won't wait for anything, because if umount happened between mntput/dput, dentry is not in s_dshrinkers list.
if umount happens in parallell with dput() (where shrinker operations are), then it will behave ok - will wait for dput() and then umount. It was intended behaviour!
Also, please, note that such early mntput()'s are bugs!!! because such dentries can reference freed memory after last mntput(). And I remember some patches in 2.4.x/2.6.x which fixed this sequence everywhere.
Kirill
Quoted 9 lines · Kirill Korotaev
Quoted 9 lines
mntput(path->mnt); // too earlymntput()dput(path->dentry);Assuming that in-between this sequence someone unmounts the file system, your
patch will wait for thisdput()to finish before it proceeds with unmounting
the file system. I think this isn't what we want.
Quoted 5 lines
No, it won't wait for anything, because if umount happened between mntput/dput, dentry is not in
s_dshrinkerslist.
if umount happens in parallell withdput()(where shrinker operations are), then it will behave ok - will wait fordput()and then umount. It was intended behaviour!
It should not wait.
Quoted 4 lines
Also, please, note that such early mntput()'s are bugs!!! because such dentries can reference freed memory after last
mntput(). And I remember some patches in 2.4.x/2.6.x which fixed this sequence everywhere.
Thats why I'm complaining ...
Regards,
Jan
Signature · 4 lines
Jan Blunck [email protected] SuSE LINUX AG - A Novell company Maxfeldstr. 5 +49-911-74053-608 D-90409 Nürnberg http://www.suse.de
Hello Jan,
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
mntput(path->mnt); // too earlymntput()dput(path->dentry);Assuming that in-between this sequence someone unmounts the file system, your patch will wait for this
dput()to finish before it proceeds with unmounting the file system. I think this isn't what we want.
Quoted 5 lines
Quoted 2 lines
No, it won't wait for anything, because if umount happened between mntput/dput, dentry is not in
s_dshrinkerslist.
if umount happens in parallell withdput()(where shrinker operations are), then it will behave ok - will wait fordput()and then umount. It was intended behaviour!It should not wait.
why?! it makes sure, that dentries/inodes are gone _before_ super block destroyed.
Quoted 4 lines
Quoted 1 line
Also, please, note that such early mntput()'s are bugs!!! because such dentries can reference freed memory after last
mntput(). And I remember some patches in 2.4.x/2.6.x which fixed this sequence everywhere.Thats why I'm complaining ...
about what?
my patch doesn't hide this bug, nor helps it anyhow.
Kirill