VFS: Busy inodes after unmount on 2 way SMP
Subsystems (4)
We've seen a few "VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day.." messages on a dual processor NFS client. Here's the use case:
- Due to network load issues, the NFS server becomes unreachable for some time
- The automounter tries to expire the mount
- The unmount finds a couple of busy inodes. Putting some debug printks shows that typically two inodes are busy i.e. have inode->i_count == 1. But they don't have any waiters on inode->i_wait. Further, the inodes that are busy are/mnt/foo -> autofs mount point/mnt/foo/bar -> bar is a symbolic link
It's not clear if this is a NFS issue or a autofs issue, but it's seen often with autofs. Are there any known race conditions that have been fixed after 2.4.20 ? The two calls I'm worried about are:
fs/autofs/root.c:305: d_instantiate(dentry, iget(dir->i_sb,ent->ino));fs/autofs/root.c:416: d_instantiate(dentry, iget(dir->i_sb,ino));
-Arun
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 15 lines · Arun Sharma
We've seen a few "VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day.." messages on a dual processor NFS client. Here's the use case:
- Due to network load issues, the NFS server becomes unreachable for some time
- The automounter tries to expire the mount
- The unmount finds a couple of busy inodes. Putting some debug printks shows that typically two inodes are busy i.e. haveinode->i_count== 1. But they don't have any waiters oninode->i_wait. Further, the inodes that are busy are/mnt/foo-> autofs mount point/mnt/foo/bar-> bar is a symbolic link
Quoted 7 lines
It's not clear if this is a NFS issue or a autofs issue, but it's seen often with autofs. Are there any known race conditions that have been fixed after 2.4.20 ? The two calls I'm worried about are:
fs/autofs/root.c:305:d_instantiate(dentry, iget(dir->i_sb,ent->ino));fs/autofs/root.c:416:d_instantiate(dentry, iget(dir->i_sb,ino));
Which version of autofs are you running? What version of mount(8)?
It's possible NFS lets you umount a mount which has busy inodes under certain conditions. autofs stresses mounting/umounting NFS in ways that sometimes exposes bugs that otherwise wouldn't be seen.
-hpa
Hello,
We have seen the same message on a box with vanilla 2.4.18 kernel with rmap12h patch. Mount version is 2.11b and autofs4-4.0.0pre10-10 (SuSe package). If it matters, we are using only nisplus to get our maps.
I know the above setup is kinda old. We are still having some other issues moving to a newer release.
<< ryan
Quoted 37 lines · H. Peter Anvin
Quoted 15 lines · Arun Sharma
We've seen a few "VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day.." messages on a dual processor NFS client. Here's the use case:
- Due to network load issues, the NFS server becomes unreachable for some time
- The automounter tries to expire the mount
- The unmount finds a couple of busy inodes. Putting some debug printks shows that typically two inodes are busy i.e. haveinode->i_count== 1. But they don't have any waiters oninode->i_wait. Further, the inodes that are busy are/mnt/foo-> autofs mount point/mnt/foo/bar-> bar is a symbolic linkQuoted 7 lines
It's not clear if this is a NFS issue or a autofs issue, but it's seen often with autofs. Are there any known race conditions that have been fixed after 2.4.20 ? The two calls I'm worried about are:
fs/autofs/root.c:305:d_instantiate(dentry, iget(dir->i_sb,ent->ino));fs/autofs/root.c:416:d_instantiate(dentry, iget(dir->i_sb,ino));Which version of autofs are you running? What version of mount(8)?
It's possible NFS lets you umount a mount which has busy inodes under certain conditions. autofs stresses mounting/umounting NFS in ways that sometimes exposes bugs that otherwise wouldn't be seen.
-hpa
_______________________________________________
autofs mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/autofs
Quoted 3 lines · H. Peter Anvin
Which version of autofs are you running? What version of mount(8)?
autofs-3.1.7 and mount-2.11g
Quoted 2 lines
It's possible NFS lets you umount a mount which has busy inodes under certain conditions.
Are you thinking of the MNT_FORCE flag ? I have reasons to believe that it won't help. Mainly because if it was a RPC task that marked the inode busy, it'd have put itself on the waitq of the inode. However I found that waitqueue_active(&inode->i_wait) == 0.
-Arun
Quoted 8 lines · Arun Sharma
Quoted 3 lines · H. Peter Anvin
Which version of autofs are you running? What version of mount(8)?
autofs-3.1.7 and mount-2.11g
Quoted 9 lines
Quoted 2 lines
It's possible NFS lets you umount a mount which has busy inodes under certain conditions.
Are you thinking of the
MNT_FORCEflag ? I have reasons to believe that it won't help. Mainly because if it was a RPC task that marked the inode busy, it'd have put itself on the waitq of the inode. However I found thatwaitqueue_active(&inode->i_wait)== 0.
No, I'm thinking about the possibility that there is a bug.
autofs depends critically on a filesystem *not* allowing itself to be unmounted when there are busy inodes; however, this is something that things other than autofs doesn't really stress a whole lot.
-hpa
Quoted 8 lines · Arun Sharma
We've seen a few "VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day.." messages on a dual processor NFS client. Here's the use case:
- Due to network load issues, the NFS server becomes unreachable for some time
- The automounter tries to expire the mount
- The unmount finds a couple of busy inodes. Putting some debug printks shows that typically two inodes are busy i.e. haveinode->i_count== 1. But they don't have any waiters oninode->i_wait. Further, the inodes that are busy are/mnt/foo-> autofs mount point/mnt/foo/bar-> bar is a symbolic link
Quoted 4 lines
It's not clear if this is a NFS issue or a autofs issue, but it's seen often with autofs. Are there any known race conditions that have been fixed after 2.4.20 ? The two calls I'm worried about are:
fs/autofs/root.c:305:d_instantiate(dentry, iget(dir->i_sb,ent->ino));fs/autofs/root.c:416:d_instantiate(dentry, iget(dir->i_sb,ino));
I don't know if it's the same issue, but I've seen it many many times while testing amd's autofs functionality. It occurs when a non-empty autofs filesystem is unmounted, e.g. if it has some leftover subdirectories from previous mounts. Therefore I'd say it's quite clearly an autofs issue, at least in my case.
Do you see it when shutting down the automounter, or in other cases as well?
Ion
Signature · 11 lines
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfsQuoted 2 lines · Ion Badulescu
I don't know if it's the same issue, but I've seen it many many times while testing amd's autofs functionality. It occurs when a non-empty autofs filesystem is unmounted, e.g. if it has some leftover subdirectories from previous mounts. Therefore I'd say it's quite clearly an autofs issue, at least in my case.
Do you have a case that can deterministically reproduce the issue ?
Quoted 1 line
Do you see it when shutting down the automounter, or in other cases as well?
Everytime I've seen it, it's during an unmount by /usr/sbin/automount after the mount has expired. It's usually coupled with a temporarily unreachable NFS server.
-Arun
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 9 lines · Arun Sharma
Quoted 6 lines · Ion Badulescu
I don't know if it's the same issue, but I've seen it many many times while testing amd's autofs functionality. It occurs when a non-empty autofs filesystem is unmounted, e.g. if it has some leftover subdirectories from previous mounts. Therefore I'd say it's quite clearly an autofs issue, at least in my case.
Do you have a case that can deterministically reproduce the issue ?
Oh yeah, easily. Kill the automounter while some things are mounted, unmount everything by hand but do not rmdir the mountpoints. When you unmount the autofs filesystem itself, you will get the self destruct message.
Quoted 3 lines
Everytime I've seen it, it's during an unmount by
/usr/sbin/automountafter the mount has expired. It's usually coupled with a temporarily unreachable NFS server.
So maybe this is a different issue than the one I am describing...
Ion
Signature · 12 lines
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfsQuoted 7 lines · Ion Badulescu
Quoted 4 lines · Arun Sharma
Quoted 1 line · Ion Badulescu
I don't know if it's the same issue, but I've seen it many many times while testing amd's autofs functionality. It occurs when a non-empty autofs filesystem is unmounted, e.g. if it has some leftover subdirectories from previous mounts. Therefore I'd say it's quite clearly an autofs issue, at least in my case.
Quoted 8 lines
Quoted 1 line
Do you have a case that can deterministically reproduce the issue ?
Oh yeah, easily. Kill the automounter while some things are mounted, unmount everything by hand but do not rmdir the mountpoints. When you unmount the autofs filesystem itself, you will get the self destruct message.
Thanks. This reproduces the problem I'm chasing very well. I think this is clearly an unacceptable kernel bug.
-Arun
Sep 15 13:58:57 arun-desktop kernel: VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day...
<snip>
Sep 15 14:00:00 arun-desktop kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000004 Sep 15 14:00:00 arun-desktop kernel: printing eip: Sep 15 14:00:00 arun-desktop kernel: c015c738 Sep 15 14:00:00 arun-desktop kernel: *pde = 00000000 Sep 15 14:00:00 arun-desktop kernel: Oops: 0000 Sep 15 14:00:00 arun-desktop kernel: ide-cd cdrom emu10k1 ac97_codec sound soundcore nfs lockd sunrpc parport_pc lp parport autofs iptable_filter ip_tables e100 keybdev mousedev hid input usb-uhc Sep 15 14:00:00 arun-desktop kernel: CPU: 0 Sep 15 14:00:00 arun-desktop kernel: EIP: 0060:[<c015c738>] Not tainted Sep 15 14:00:00 arun-desktop kernel: EFLAGS: 00010246 Sep 15 14:00:00 arun-desktop kernel: Sep 15 14:00:00 arun-desktop kernel: EIP is at destroy_inode [kernel] 0x28 (2.4.20-8) Sep 15 14:00:00 arun-desktop kernel: eax: 00000000 ebx: cee44c80 ecx: 00000000 edx: cee44c80 Sep 15 14:00:00 arun-desktop kernel: esi: cb32ee00 edi: cee44c80 ebp: 000003c6 esp: cff87f84 Sep 15 14:00:01 arun-desktop kernel: ds: 0068 es: 0068 ss: 0068 Sep 15 14:00:01 arun-desktop kernel: Process kswapd (pid: 5, stackpage=cff87000) Sep 15 14:00:01 arun-desktop kernel: Stack: cee44c80 cb32ee00 cb32ee18 c015b563 cee44c80 cb0ff180 cff86000 00000000 Sep 15 14:00:01 arun-desktop kernel: 000001d0 00000000 c015b904 0000057c c013cd5b 00000006 000001d0 cff86000 Sep 15 14:00:01 arun-desktop kernel: cff86000 00000000 cff86305 00000000 c013d023 000001d0 00000002 000001d0 Sep 15 14:00:01 arun-desktop kernel: Call Trace: [<c015b563>] prune_dcache [kernel] 0xc3 (0xcff87f90)) Sep 15 14:00:01 arun-desktop kernel: [<c015b904>] shrink_dcache_memory [kernel] 0x24 (0xcff87fac)) Sep 15 14:00:01 arun-desktop kernel: [<c013cd5b>] do_try_to_free_pages_kswapd [kernel] 0x1b (0xcff87fb4)) Sep 15 14:00:01 arun-desktop kernel: [<c013d023>] kswapd [kernel] 0x83 (0xcff87fd4)) Sep 15 14:00:01 arun-desktop kernel: [<c013cfa0>] kswapd [kernel] 0x0 (0xcff87fe4)) Sep 15 14:00:01 arun-desktop kernel: [<c010742d>] kernel_thread_helper [kernel] 0x5 (0xcff87ff0))
Quoted 9 lines · Arun Sharma
Quoted 5 lines
Oh yeah, easily. Kill the automounter while some things are mounted, unmount everything by hand but do not rmdir the mountpoints. When you unmount the autofs filesystem itself, you will get the self destruct message.
Thanks. This reproduces the problem I'm chasing very well. I think this is clearly an unacceptable kernel bug.
Indeed. This is on autofs 3?
-hpa
Quoted 2 lines · H. Peter Anvin
Indeed. This is on autofs 3?
Yes.
-Arun
Quoted 11 lines · H. Peter Anvin
Quoted 9 lines · Arun Sharma
Quoted 5 lines
Oh yeah, easily. Kill the automounter while some things are mounted, unmount everything by hand but do not rmdir the mountpoints. When you unmount the autofs filesystem itself, you will get the self destruct message.
Thanks. This reproduces the problem I'm chasing very well. I think this is clearly an unacceptable kernel bug.
Indeed. This is on autofs 3?
Both 3 and 4, at least as far as the "self-destruct" message goes.
Ion
Signature · 2 lines
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.Don't know if it helps, but we also run a few hundred autofs4 clients, with a mix of 2.4.21 and 2.4.18. The 2.4.21 machines are seeing quite a few of these messages, but not on the 2.4.18 machines. We haven't seen any of the oopses described below, though.
-matt
Quoted 9 lines · Ion Badulescu
Quoted 7 lines · H. Peter Anvin
Quoted 6 lines · Arun Sharma
Quoted 5 lines
Oh yeah, easily. Kill the automounter while some things are mounted, unmount everything by hand but do not rmdir the mountpoints. When you unmount the autofs filesystem itself, you will get the self destruct message.
Quoted 10 lines
Quoted 4 lines
Quoted 3 lines
Thanks. This reproduces the problem I'm chasing very well. I think this is clearly an unacceptable kernel bug.
Indeed. This is on autofs 3?
Both 3 and 4, at least as far as the "self-destruct" message goes.
Ion
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 12 lines · Matt C
Quoted 11 lines · Ion Badulescu
Quoted 10 lines · H. Peter Anvin
Quoted 9 lines · Arun Sharma
Quoted 5 lines
Oh yeah, easily. Kill the automounter while some things are mounted, unmount everything by hand but do not rmdir the mountpoints. When you unmount the autofs filesystem itself, you will get the self destruct message.
Thanks. This reproduces the problem I'm chasing very well. I think this is clearly an unacceptable kernel bug.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Indeed. This is on autofs 3?
Both 3 and 4, at least as far as the "self-destruct" message goes.
Don't know if it helps, but we also run a few hundred autofs4 clients, with a mix of 2.4.21 and 2.4.18. The 2.4.21 machines are seeing quite a few of these messages, but not on the 2.4.18 machines. We haven't seen any of the oopses described below, though.
Same here, 2.4.22, autofs 3 and 4. I'll be playing with 2.6.0 next week.
I haven't tried to reproduce it with your method, but I do see the bug; no oops though.
/fc
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 3 lines · H. Peter Anvin
Indeed. This is on autofs 3?
Correction: the problem was primarily seen on autofs4. However the oops that I reported yesterday was autofs3. So the problem affects both.
-Arun
Quoted 10 lines · Arun Sharma
Quoted 3 lines · H. Peter Anvin
Indeed. This is on autofs 3?
Correction: the problem was primarily seen on autofs4. However the oops that I reported yesterday was autofs3. So the problem affects both.
-Arun
Thanks. That's useful information.
-hpa
Ian Kent
maintainerQuoted 9 lines · Arun Sharma
We've seen a few "VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day.." messages on a dual processor NFS client. Here's the use case:
- Due to network load issues, the NFS server becomes unreachable for some time
- The automounter tries to expire the mount
- The unmount finds a couple of busy inodes. Putting some debug printks shows that typically two inodes are busy i.e. haveinode->i_count== 1. But they don't have any waiters oninode->i_wait. Further, the inodes that are busy are/mnt/foo-> autofs mount point/mnt/foo/bar-> bar is a symbolic link
Quoted 5 lines
It's not clear if this is a NFS issue or a autofs issue, but it's seen often with autofs. Are there any known race conditions that have been fixed after 2.4.20 ? The two calls I'm worried about are:
fs/autofs/root.c:305:d_instantiate(dentry, iget(dir->i_sb,ent->ino));fs/autofs/root.c:416:d_instantiate(dentry, iget(dir->i_sb,ino));
This is in the autofs (aka v3) kernel module not autofs4.
Please specify which version of the userspace daemon and kernel module?
Signature · 4 lines
,-._|\ Ian Kent / \ Perth, Western Australia *_.--._/ E-mail: [email protected] v Web: http://themaw.net/
Quoted 9 lines · Ian Kent
Quoted 5 lines
It's not clear if this is a NFS issue or a autofs issue, but it's seen often with autofs. Are there any known race conditions that have been fixed after 2.4.20 ? The two calls I'm worried about are:
fs/autofs/root.c:305:d_instantiate(dentry, iget(dir->i_sb,ent->ino));fs/autofs/root.c:416:d_instantiate(dentry, iget(dir->i_sb,ino));This is in the autofs (aka v3) kernel module not autofs4.
Please ignore the above lines. I corrected myself in an earlier message. The problem was seen on the autofs4 kernel module in 2.4.20 + autofs-3.1.7 user daemon.
-Arun
Quoted 11 lines · Arun Sharma
Quoted 7 lines · Ian Kent
Quoted 5 lines
It's not clear if this is a NFS issue or a autofs issue, but it's seen often with autofs. Are there any known race conditions that have been fixed after 2.4.20 ? The two calls I'm worried about are:
fs/autofs/root.c:305:d_instantiate(dentry, iget(dir->i_sb,ent->ino));fs/autofs/root.c:416:d_instantiate(dentry, iget(dir->i_sb,ino));This is in the autofs (aka v3) kernel module not autofs4.
Please ignore the above lines. I corrected myself in an earlier message. The problem was seen on the autofs4 kernel module in 2.4.20 + autofs-3.1.7 user daemon.
Please don't mix the v3 and v4 stuff (this was a misguided change on Linus' part which I still very much disagree with.)
-hpa
Ian Kent
maintainerQuoted 4 lines · H. Peter Anvin
Please don't mix the v3 and v4 stuff (this was a misguided change on Linus' part which I still very much disagree with.)
Yes.
It should work but if it doesn't it becomes very difficult to support.
Signature · 4 lines
,-._|\ Ian Kent / \ Perth, Western Australia *_.--._/ E-mail: [email protected] v Web: http://themaw.net/
Ian Kent
maintainerQuoted 3 lines · Arun Sharma
Please ignore the above lines. I corrected myself in an earlier message. The problem was seen on the autofs4 kernel module in 2.4.20 + autofs-3.1.7 user daemon.
Vanilla 2.4.20?
Signature · 4 lines
,-._|\ Ian Kent / \ Perth, Western Australia *_.--._/ E-mail: [email protected] v Web: http://themaw.net/
Quoted 10 lines · Ian Kent
Quoted 3 lines · Arun Sharma
Please ignore the above lines. I corrected myself in an earlier message. The problem was seen on the autofs4 kernel module in 2.4.20 + autofs-3.1.7 user daemon.
Vanilla 2.4.20?
Yes, vanilla 2.4.20.
-Arun
Okay... let me ask this so I get it straight...
Has anyone seen this (busy inodes after stopping the automounter) using autofs v3 kernel module and daemon?
-hpa
Quoted 4 lines · H. Peter Anvin
Okay... let me ask this so I get it straight...
Has anyone seen this (busy inodes after stopping the automounter) using autofs v3 kernel module and daemon?
I'm not sure, but everyone who has seen this should fiddle this patch into the kernel and see how it goes.
The whole thing started very recently (post 2.4.21) for us.
This patch is untested, any feedback appreciated.
This patch will hopefully fix the "VFS: Busy inodes after unmount. Self-destruct in 5 seconds" oopses generated by autofs+nfs.
fs/nfs/inode.c 2003-09-17 21:24:57.000000000 +0200 renamed from fs/nfs/inode.c 2003-09-17 17:51:53.000000000 +0200 +10 -0
@@ -976,9 +976,19 @@ int nfs_release(struct inode *inode, struct file *filp) { + struct dentry *dentry; struct rpc_cred *cred; lock_kernel(); + + /* Do the silly unlink here as long as we still have a reference + * on the vfsmount. Otherwise the unlink cache may hold a reference + * to the parent dentry across umount() calls. */ + if ((dentry = filp->f_dentry) && (dentry->d_flags & DCACHE_NFSFS_RENAMED)) { + /* Just trigger the NFS call */ + nfs_complete_unlink(dentry); + } + cred = nfs_file_cred(filp); if (cred) put_rpccred(cred);
Signature · 3 lines
USB is for mice, FireWire is for men! sUse lINUX ag, nÜRNBERG
Ian Kent
maintainerQuoted 6 lines · Olaf Hering
This patch is untested, any feedback appreciated.
This patch will hopefully fix the "VFS: Busy inodes after unmount. Self-destruct in 5 seconds" oopses generated by autofs+nfs.
Could you elaborate on reasoning behind the patch a bit please.
Signature · 4 lines
,-._|\ Ian Kent / \ Perth, Western Australia *_.--._/ E-mail: [email protected] v Web: http://themaw.net/
Quoted 10 lines · Ian Kent
Quoted 6 lines · Olaf Hering
This patch is untested, any feedback appreciated.
This patch will hopefully fix the "VFS: Busy inodes after unmount. Self-destruct in 5 seconds" oopses generated by autofs+nfs.
Could you elaborate on reasoning behind the patch a bit please.
probably that:
...
Additional Comment #10 From Olaf Kirch 2003-09-16 21:13 The way I understood it the code is supposed to work like
umount(8) calls umount(2) with MNT_FORCE
umount(2) calls NFS to signal all RPC tasks
later notices there are still buys inodes and returns to user space (-EBUSY)
umount(8) retries umount(2) call
If that assumption is incorrect, we indeed to wait for the rpc tasks to complete. Nasty :-(
...
Signature · 3 lines
USB is for mice, FireWire is for men! sUse lINUX ag, nÜRNBERG
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Olaf Hering <[email protected]> writes:
> On Wed, Sep 17, H. Peter Anvin wrote:
>> Okay... let me ask this so I get it straight... >>
>> Has anyone seen this (busy inodes after stopping the >> automounter) using autofs v3 kernel module and daemon?
> I'm not sure, but everyone who has seen this should fiddle this > patch into the kernel and see how it goes. The whole thing > started very recently (post 2.4.21) for us.
> This patch is untested, any feedback appreciated.
...and is indeed wrong... It does the exact opposite of what sillydelete should do. Instead of causing the last application that closes the file to perform the sillydelete, you are asking the *first* application that closes it to do so.
Sillydelete *has* to be tied to dentries. Not files, and not inodes. It is purely a namespace operation...
So exactly what are you trying to do, and why?
Cheers,
Trond
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 4 lines · Trond Myklebust
...and is indeed wrong... It does the exact opposite of what sillydelete should do. Instead of causing the last application that closes the file to perform the sillydelete, you are asking the *first* application that closes it to do so.
I know. I was not claiming it's perfectly right, what I'm interested in whether it cures the oopses we're seeing. If that were true, one could still look for The Right Solution...
Quoted 4 lines
Sillydelete *has* to be tied to dentries. Not files, and not inodes. It is purely a namespace operation...
So exactly what are you trying to do, and why?
We have seen several reports of autofs+nfs leading to oopses, preceded by "Busy inodes on umount, self-destruct in 5 seconds".
It's somewhat hard to reproduce, so I'm currently trying to come up with possible patches by looking at the source.
You cannot umount if the vfsmount mnt_count is != 2. So either something hosed the mnt_count completely (by doing too many mntput() calls for instance), or something holds dentry references without bumping mnt_count along with it. Do you agree?
The only place in the nfs client code that actually does a dget is the sillydelete stuff in unlink.c. So my idea in generating this patch was that the nfs_complete_unlink was happening too late, when the file is already closed and the vfsmount reference is gone.
It's a long shot admittedly...
Have a nice day,
Olaf
Signature · 12 lines
Olaf Kirch | Anyone who has had to work with X.509 has probably [email protected] | experienced what can best be described as ---------------+ ISO water torture. -- Peter Gutmann ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 11 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Olaf Hering <[email protected]> writes:
> On Wed, Sep 17, H. Peter Anvin wrote:
>> Okay... let me ask this so I get it straight... >>
>> Has anyone seen this (busy inodes after stopping the >> automounter) using autofs v3 kernel module and daemon?> I'm not sure, but everyone who has seen this should fiddle this > patch into the kernel and see how it goes. The whole thing > started very recently (post 2.4.21) for us.
After thinking more about the conditions, I got a crash with 2.4.19 client. the codebase was unchanged for a long time. But the server side was updated to 2.4.21 with lots of nfs patches. Possible that the newer server side causes trouble for the client.
Signature · 3 lines
USB is for mice, FireWire is for men! sUse lINUX ag, nÜRNBERG
Hi All-
We've been having the 'VFS: Busy inodes after unmount' problems like crazy on our desktop and server linux boxes running autofs under 2.4.21 (vanilla). We're using the autofs-3.1.7 userspace tools, but with the autofs4 kernel module.
I applied the patch that was attached earlier in this thread to our kernel and have been running it for the past few days on my desktop. These errors have completely disappeared, and autofs is behaving as expected so far.
So, while it might be the wrong thing to do, it does fix our problems.
-matt
Quoted 27 lines · Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Olaf Hering <[email protected]> writes:
> On Wed, Sep 17, H. Peter Anvin wrote:
>> Okay... let me ask this so I get it straight... >>
>> Has anyone seen this (busy inodes after stopping the >> automounter) using autofs v3 kernel module and daemon?> I'm not sure, but everyone who has seen this should fiddle this > patch into the kernel and see how it goes. The whole thing > started very recently (post 2.4.21) for us.
> This patch is untested, any feedback appreciated.
...and is indeed wrong... It does the exact opposite of what sillydelete should do. Instead of causing the last application that closes the file to perform the sillydelete, you are asking the *first* application that closes it to do so.
Sillydelete *has* to be tied to dentries. Not files, and not inodes. It is purely a namespace operation...So exactly what are you trying to do, and why?
Cheers,
Trond
Quoted 8 lines
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Matt C <Matt> writes:
> Hi All- We've been having the 'VFS: Busy inodes after unmount' > problems like crazy on our desktop and server linux boxes > running autofs under 2.4.21 (vanilla). We're using the > autofs-3.1.7 userspace tools, but with the autofs4 kernel > module.
> I applied the patch that was attached earlier in this thread to > our kernel and have been running it for the past few days on my > desktop. These errors have completely disappeared, and autofs > is behaving as expected so far.
> So, while it might be the wrong thing to do, it does fix our > problems.
That changes nothing. It is still not a "fix" that I want to see go into the main kernel.
The problem here is that sillyrename needs to hold onto the parent directory's inode until it is done, and yet it cannot know which vfsmount struct to hold onto because that information is not passed down by the VFS.
I would rather then have something like the appended patch go in to which force the last process to close the file to actually wait on completion (unless someone signals it with a 'kill -9' - in which case they are knowingly breaking things).
Cheers,
Trond
fs/nfs/dir.c +7 -1
@@ -994,7 +994,7 @@struct inode *old_inode = old_dentry->d_inode;struct inode *new_inode = new_dentry->d_inode;struct dentry *dentry =NULL , *rehash =NULL ; -int error = -EBUSY ; +int error;/* * To prevent any new references to the target during the rename,
@@ -1020,6 +1020,12 @@*/ if (!new_inode) goto go_ahead; +/* If target is a hard link to the source, then noop */ + error =0 ; +if (NFS_FILEID (new_inode) ==NFS_FILEID (old_inode)) + goto out; + + error = -EBUSY ;if (S_ISDIR (new_inode->i_mode)) goto out;else if (atomic_read (&new_dentry->d_count) >1 ) {
fs/nfs/unlink.c +12 -1
@@ -12,6 +12,7 @@#include <linux/sunrpc/sched.h> #include <linux/sunrpc/clnt.h> #include <linux/nfs_fs.h> +#include <linux/wait.h> struct nfs_unlinkdata {
@@ -21,6 +22,9 @@struct rpc_task task;struct rpc_cred *cred;unsigned int count; + +wait_queue_head_t waitq; +int completed; };static struct nfs_unlinkdata *nfs_deletes;
@@ -133,6 +137,8 @@put_rpccred (data->cred); data->cred =NULL ;dput (dir); + data->completed =1 ; +wake_up (&data->waitq); }/**
@@ -175,6 +181,8 @@ nfs_deletes = data; data->count =1 ; +init_waitqueue_head (&data->waitq); + task = &data->task;rpc_init_task (task, clnt, nfs_async_unlink_done ,RPC_TASK_ASYNC ); task->tk_calldata = data;
@@ -212,7 +220,10 @@ data->count++;nfs_copy_dname (dentry, data); dentry->d_flags &= ~DCACHE_NFSFS_RENAMED ; -if (data->task.tk_rpcwait == &nfs_delete_queue) +if (data->task.tk_rpcwait == &nfs_delete_queue) { +struct rpc_clnt *clnt = data->task.tk_client;rpc_wake_up_task (&data->task); +nfs_wait_event (clnt, data->waitq, data->completed ==1 ); + }nfs_put_unlinkdata (data); }
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 4 lines · Trond Myklebust
I would rather then have something like the appended patch go in to which force the last process to close the file to actually wait on completion (unless someone signals it with a 'kill -9' - in which case they are knowingly breaking things).
Ugh, that's ugly -- and potentially a DoS issue as well. Since you don't need any privileges to issue the 'kill -9' for your own processes, it becomes fairly easy to write some code that exploits this and causes kernel crashes across the board...
Not that the current situation is any better, of course.
Perhaps making that wait an uninterruptable wait would be enough to take care of the problem?
Ion
Signature · 2 lines
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Ion Badulescu <[email protected]> writes:
> Ugh, that's ugly -- and potentially a DoS issue as well. Since > you don't need any privileges to issue the 'kill -9' for your > own processes, it becomes fairly easy to write some code that > exploits this and causes kernel crashes across the board...
Then don't set the "intr" flag.
> Not that the current situation is any better, of course.
> Perhaps making that wait an uninterruptable wait would be > enough to take care of the problem?
People expect the "intr" flag to work if they set it.
Cheers,
Trond
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 6 lines · Trond Myklebust
> Not that the current situation is any better, of course.
> Perhaps making that wait an uninterruptable wait would be > enough to take care of the problem?
People expect the "intr" flag to work if they set it.
In this case, not removing the sillyrenamed file instead of oopsing may be the preferred action.
Olaf
Signature · 12 lines
Olaf Kirch | Anyone who has had to work with X.509 has probably [email protected] | experienced what can best be described as ---------------+ ISO water torture. -- Peter Gutmann ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 2 lines · Olaf Kirch
In this case, not removing the sillyrenamed file instead of oopsing may be the preferred action.
Would it be possibly to combine both ideas, i.e. have the last process wait, but if the user attempt to kill that process and it will be allowed because intr is set the sillyrenamed file is not removed.
Surely it is reasonable for people who kill -9 things to expect to do some tidying up, but not to expect a kernel panic.
Steve.
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 4 lines · Trond Myklebust
> Perhaps making that wait an uninterruptable wait would be > enough to take care of the problem?
People expect the "intr" flag to work if they set it.
People also expect the supported "intr" flag not to cause security problems and would gladly take an unkillable process over a kernel oops...
Ion
Signature · 13 lines
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfsQuoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Ion Badulescu <[email protected]> writes:
People expect the "intr" flag to work if they set it.
> People also expect the supported "intr" flag not to cause > security problems and would gladly take an unkillable process > over a kernel oops...
You'd have to be extremely unlucky to kill the process and hit the window for the Oops. I defy you to come up with an exploit for it.
That said, I agree that a full fix would be preferable. I'm working on other projects right now, that's why I'm being slow about this issue (plus the fact that it's not exactly easy to reproduce). I'll get onto it soon...
Cheers,
Trond
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 7 lines · Trond Myklebust
You'd have to be extremely unlucky to kill the process and hit the window for the Oops. I defy you to come up with an exploit for it.
That said, I agree that a full fix would be preferable. I'm working on other projects right now, that's why I'm being slow about this issue (plus the fact that it's not exactly easy to reproduce). I'll get onto it soon...
I've had no luck reproducing it either in a controlled environment. However the bug is common enough to crash a few of our build machines over a weekend...
I took your patch, Trond, and modified it a little to not crash if the user ctrl-c's the unlink. Still not pretty - maybe it should zap the cached attributes:
@@ -212,7 +222,15 @@
data->count++;
nfs_copy_dname(dentry, data);
dentry->d_flags &= ~DCACHE_NFSFS_RENAMED;
- if (data->task.tk_rpcwait == &nfs_delete_queue)
+ if (data->task.tk_rpcwait == &nfs_delete_queue) {
+ struct rpc_clnt *clnt = data->task.tk_client;
rpc_wake_up_task(&data->task);
+ nfs_wait_event(clnt, data->waitq, data->completed == 1);
+ /* This is safe as we hold the BKL */
+ if (!data->completed) {+ dput(data->dir);
+ data->dir = NULL;
+ }
+ }
nfs_put_unlinkdata(data);
}
Olaf
Signature · 3 lines
Olaf Kirch | Anyone who has had to work with X.509 has probably [email protected] | experienced what can best be described as ---------------+ ISO water torture. -- Peter Gutmann
fs/nfs/dir.c +7 -1
@@ -1144,7 +1144,7 @@struct inode *old_inode = old_dentry->d_inode;struct inode *new_inode = new_dentry->d_inode;struct dentry *dentry =NULL , *rehash =NULL ; -int error = -EBUSY ; +int error;/* * To prevent any new references to the target during the rename,
@@ -1170,6 +1170,12 @@*/ if (!new_inode) goto go_ahead; +/* If target is a hard link to the source, then noop */ + error =0 ; +if (NFS_FILEID (new_inode) ==NFS_FILEID (old_inode)) + goto out; + + error = -EBUSY ;if (S_ISDIR (new_inode->i_mode)) goto out;else if (atomic_read (&new_dentry->d_count) >1 ) {
fs/nfs/unlink.c +19 -1
@@ -12,6 +12,7 @@#include <linux/sunrpc/sched.h> #include <linux/sunrpc/clnt.h> #include <linux/nfs_fs.h> +#include <linux/wait.h> struct nfs_unlinkdata {
@@ -21,6 +22,9 @@struct rpc_task task;struct rpc_cred *cred;unsigned int count; + +wait_queue_head_t waitq; +int completed; };static struct nfs_unlinkdata *nfs_deletes;
@@ -54,6 +58,8 @@nfs_detach_unlinkdata (data);if (data->name.name !=NULL )kfree (data->name.name); +if (data->cred) +put_rpccred (data->cred);kfree (data); } }
@@ -133,6 +139,8 @@put_rpccred (data->cred); data->cred =NULL ;dput (dir); + data->completed =1 ; +wake_up (&data->waitq); }/**
@@ -175,6 +183,8 @@ nfs_deletes = data; data->count =1 ; +init_waitqueue_head (&data->waitq); + task = &data->task;rpc_init_task (task, clnt, nfs_async_unlink_done ,RPC_TASK_ASYNC ); task->tk_calldata = data;
@@ -212,7 +222,15 @@ data->count++;nfs_copy_dname (dentry, data); dentry->d_flags &= ~DCACHE_NFSFS_RENAMED ; -if (data->task.tk_rpcwait == &nfs_delete_queue) +if (data->task.tk_rpcwait == &nfs_delete_queue) { +struct rpc_clnt *clnt = data->task.tk_client;rpc_wake_up_task (&data->task); +nfs_wait_event (clnt, data->waitq, data->completed ==1 ); +/* This is safe as we hold the BKL */ +if (!data->completed) { +dput (data->dir); + data->dir =NULL ; + } + }nfs_put_unlinkdata (data); }
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Olaf Kirch <[email protected]> writes:
> On Mon, Sep 29, 2003 at 10:22:40AM -0700, Trond Myklebust
>> That said, I agree that a full fix would be preferable. I'm >> working on other projects right now, that's why I'm being slow >> about this issue (plus the fact that it's not exactly easy to >> reproduce). I'll get onto it soon...
> I've had no luck reproducing it either in a controlled > environment. However the bug is common enough to crash a few > of our build machines over a weekend...
> I took your patch, Trond, and modified it a little to not crash > if the user ctrl-c's the unlink. Still not pretty - maybe it > should zap the cached attributes:
Alternatively, we should perhaps make fsync_super() sync out all dirty inodes and readaheads, then wait on all pending RPC ops. That would be cleaner, I think...
Cheers,
Trond
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Does this patch help?
fs/namei.c +7 -2
@@ -893,6 +893,8 @@ static inline int check_sticky(struct in* 7. If we were asked to remove a directory and victim isn't one - ENOTDIR. * 8. If we were asked to remove a non-directory and victim isn't one - EISDIR. * 9. We can't remove a root or mountpoint. +* 10. We don't allow removal of NFS sillyrenamed files; it's handled by +* nfs_async_unlink(). */ static inline int may_delete (struct inode *dir,struct dentry *victim,int isdir) {
@@ -916,6 +918,8 @@ static inline int may_delete(struct inodreturn -EISDIR ;if (IS_DEADDIR (dir))return -ENOENT ; +if (victim->d_flags &DCACHE_NFSFS_RENAMED ) +return -EBUSY ;return 0 ; }
@@ -1484,13 +1488,14 @@ int vfs_unlink(struct inode *dir, structlock_kernel (); error = dir->i_op->unlink (dir, dentry);unlock_kernel (); -if (!error) +if (!error && + !(dentry->d_flags &DCACHE_NFSFS_RENAMED ))d_delete (dentry); } } }up (&dir->i_zombie); -if (!error) +if (!error && !(dentry->d_flags &DCACHE_NFSFS_RENAMED ))inode_dir_notify (dir,DN_DELETE );return error; }
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 4 lines · H. Peter Anvin
Okay... let me ask this so I get it straight...
Has anyone seen this (busy inodes after stopping the automounter) using autofs v3 kernel module and daemon?
Why do you bring the daemon into the picture, it is irrelevant here. Sure it probably does things differently which might avoid the error, but the kernel issue exists nonetheless.
To answer your question again: yes, the issue is present in both v3 and v4 kernel modules. You can always replicate it yourself if you don't believe me, using the little recipe I posted earlier:
# lsmod | grep autofs autofs 11236 4 (autoclean) # cat /proc/mounts [snip] boxter:(pid29327) /bar autofs rw 0 0 boxter:(pid29327) /net autofs rw 0 0 boxter:(pid29327) /src autofs rw 0 0 boxter:(pid29327) /misc autofs rw 0 0 buggy:/n/buggy/misc/homes /misc/homes nfs rw,v3,rsize=8192,wsize=8192,hard,intr,tcp,lock,addr=buggy 0 0 # umount /misc/homes # umount /misc # dmesg |tail VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day...
Ion
Signature · 2 lines
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.Quoted 7 lines · Ion Badulescu
Quoted 3 lines · H. Peter Anvin
Okay... let me ask this so I get it straight...
Has anyone seen this (busy inodes after stopping the automounter) using autofs v3 kernel module and daemon?
Why do you bring the daemon into the picture, it is irrelevant here.
Because it makes it a lot harder to debug the problem.
Quoted 6 lines
Sure it probably does things differently which might avoid the error, but the kernel issue exists nonetheless.
To answer your question again: yes, the issue is present in both v3 and v4 kernel modules. You can always replicate it yourself if you don't believe me, using the little recipe I posted earlier:
Quoted 13 lines
# lsmod | grep autofs autofs 11236 4 (autoclean) # cat /proc/mounts [snip] boxter:(pid29327) /bar autofs rw 0 0 boxter:(pid29327) /net autofs rw 0 0 boxter:(pid29327) /src autofs rw 0 0 boxter:(pid29327) /misc autofs rw 0 0 buggy:/n/buggy/misc/homes /misc/homes nfs rw,v3,rsize=8192,wsize=8192,hard,intr,tcp,lock,addr=buggy 0 0 # umount /misc/homes # umount /misc # dmesg |tail VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day...
OK. Did you try Olaf's NFS patch?
-hpa
Quoted 3 lines · H. Peter Anvin
Quoted 1 line
Why do you bring the daemon into the picture, it is irrelevant here.
Because it makes it a lot harder to debug the problem.
Ok, fair enough.
The daemon version doesn't matter, any daemon will do. All you need is to kill -9 automountd while something is mounted, and then unmount everything by hand without removing the mountpoint directories.
Quoted 12 lines
Quoted 11 lines
# cat /proc/mounts [snip] boxter:(pid29327) /bar autofs rw 0 0 boxter:(pid29327) /net autofs rw 0 0 boxter:(pid29327) /src autofs rw 0 0 boxter:(pid29327) /misc autofs rw 0 0 buggy:/n/buggy/misc/homes /misc/homes nfs rw,v3,rsize=8192,wsize=8192,hard,intr,tcp,lock,addr=buggy 0 0 # umount /misc/homes # umount /misc # dmesg |tail VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day...
Quoted 1 line
OK. Did you try Olaf's NFS patch?
This is not an NFS problem. The message occurs regardless of what is mounted on /misc/homes (NFS, local ext2, bind mount, whatever).
Ion
Signature · 2 lines
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.Ian Kent
maintainerQuoted 4 lines · Ion Badulescu
Quoted 2 lines · H. Peter Anvin
Quoted 1 line
Why do you bring the daemon into the picture, it is irrelevant here.
Quoted 8 lines
Quoted 1 line
Because it makes it a lot harder to debug the problem.
Ok, fair enough.
The daemon version doesn't matter, any daemon will do. All you need is to kill -9 automountd while something is mounted, and then unmount everything by hand without removing the mountpoint directories.
Yes, your'e exactly right. That's expected behaviour as the directory dentries still exist and there is no daemon to remove them. Hence the message. Seeing this message has always been a signal to me that the daemon has a bug.
I have been thinking that implementing a umount_begin routine in the super_operations for autofs4 would remedy the problem. I believe that, correct me if I'm wrong, the umount must be done with a force option in this case, so the umount_begin would be called.
snip ...
Quoted 5 lines
Quoted 2 lines
OK. Did you try Olaf's NFS patch?
This is not an NFS problem. The message occurs regardless of what is mounted on /misc/homes (NFS, local ext2, bind mount, whatever).
I must admit, reading through the thread, I was wondering how it could be an NFS problem.
Signature · 4 lines
,-._|\ Ian Kent / \ Perth, Western Australia *_.--._/ E-mail: [email protected] v Web: http://themaw.net/
Quoted 11 lines · Ian Kent
Quoted 6 lines
Quoted 3 lines
OK. Did you try Olaf's NFS patch?
This is not an NFS problem. The message occurs regardless of what is mounted on /misc/homes (NFS, local ext2, bind mount, whatever).
I must admit, reading through the thread, I was wondering how it could be an NFS problem.
Inodes are only initialized once, when they are first allocated from the slab. Any reused inodes will continue having a bad refcount, causing similar behaviour on different filesystems.
Can this be reproduced on a filesystem other than NFS for which the system has never made an NFS mount?
Mike Waychison
Ian Kent
maintainerQuoted 8 lines · Mike Waychison
Quoted 6 lines · Ian Kent
I must admit, reading through the thread, I was wondering how it could be an NFS problem.
Quoted 7 lines
Inodes are only initialized once, when they are first allocated from the slab. Any reused inodes will continue having a bad refcount, causing similar behaviour on different filesystems.
Can this be reproduced on a filesystem other than NFS for which the system has never made an NFS mount?
All my usage is NFS so I can't say.
Signature · 4 lines
,-._|\ Ian Kent / \ Perth, Western Australia *_.--._/ E-mail: [email protected] v Web: http://themaw.net/
Hi Ian,
Quoted 8 lines · Ian Kent
Quoted 4 lines · Ion Badulescu
The daemon version doesn't matter, any daemon will do. All you need is to kill -9 automountd while something is mounted, and then unmount everything by hand without removing the mountpoint directories.
Yes, your'e exactly right. That's expected behaviour as the directory dentries still exist and there is no daemon to remove them.
Hmm. Expected behaviour, you say? Maybe, maybe not, it's hard to tell what's to be expected here, since the autofs umount semantics are so different from a regular filesystems.
Normally a filesystem is allowed to be unmounted if it is not busy, right? But with autofs it's different, it should be unmounted only if it is completely empty, with no entries at all -- and I'm not convinced the VFS layer is up to the task of dealing with this requirement. Alternatively the autofs code should be prepared to clean up after itself, but that might not be possible in all circumstances.
Quoted 2 lines
Hence the message. Seeing this message has always been a signal to me that the daemon has a bug.
Oh, you're absolutely right here (although we're talking about different daemons, I'm using--and expanding, and debugging--amd's autofs support). :-) Nonetheless, it's a bug in the kernel code that should be remedied.
Quoted 4 lines
I have been thinking that implementing a
umount_beginroutine in thesuper_operationsfor autofs4 would remedy the problem. I believe that, correct me if I'm wrong, the umount must be done with a force option in this case, so theumount_beginwould be called.
No, you're right, you need MNT_FORCE for umount_begin to be called.
That, however, would be a reasonable solution if the filesystem also prevented its own unmounting when it is not empty (by having each entry hold a reference to the vfsmount in its private data, perhaps, I'm not sure). Then, inside umount_begin, and only if the filesystem is catatonic, it would garbage-collect any entries not mounted upon and not busy. That should work, I think...
Thanks,
Ion
Signature · 2 lines
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.Ian Kent
maintainerQuoted 10 lines · Ion Badulescu
Hi Ian,
Quoted 6 lines · Ian Kent
Quoted 4 lines · Ion Badulescu
The daemon version doesn't matter, any daemon will do. All you need is to kill -9 automountd while something is mounted, and then unmount everything by hand without removing the mountpoint directories.
Quoted 6 lines
Quoted 2 lines
Yes, your'e exactly right. That's expected behaviour as the directory dentries still exist and there is no daemon to remove them.
Hmm. Expected behaviour, you say? Maybe, maybe not, it's hard to tell what's to be expected here, since the autofs umount semantics are so different from a regular filesystems.
For autofs v4 anyway possibly v3 also. Cause of the nature of it.
Quoted 20 lines
Normally a filesystem is allowed to be unmounted if it is not busy, right? But with autofs it's different, it should be unmounted only if it is completely empty, with no entries at all -- and I'm not convinced the VFS layer is up to the task of dealing with this requirement. Alternatively the autofs code should be prepared to clean up after itself, but that might not be possible in all circumstances.
Quoted 2 lines
Hence the message. Seeing this message has always been a signal to me that the daemon has a bug.
Oh, you're absolutely right here (although we're talking about different daemons, I'm using--and expanding, and debugging--amd's autofs support). :-) Nonetheless, it's a bug in the kernel code that should be remedied.
Quoted 4 lines
I have been thinking that implementing a
umount_beginroutine in thesuper_operationsfor autofs4 would remedy the problem. I believe that, correct me if I'm wrong, the umount must be done with a force option in this case, so theumount_beginwould be called.
Quoted 8 lines
No, you're right, you need
MNT_FORCEforumount_beginto be called.That, however, would be a reasonable solution if the filesystem also prevented its own unmounting when it is not empty (by having each entry hold a reference to the vfsmount in its private data, perhaps, I'm not sure). Then, inside
umount_begin, and only if the filesystem is catatonic, it would garbage-collect any entries not mounted upon and not busy. That should work, I think...
As you point out I am talking about autofs not amd.
I think it's a lot simpler than it sounds above. Granted if the tree is busy with mounts you can probably only give up but much of the time it's not. The logic to traverse the filesystem would be similar to what I have used in my rewrite of the expire module in autofs4. Check it out at ftp://ftp.kernel.org/pub/linux/daemons/autofs/v4 and let me know if you think the idea has merit.
Signature · 4 lines
,-._|\ Ian Kent / \ Perth, Western Australia *_.--._/ E-mail: [email protected] v Web: http://themaw.net/
Hello Ian,
We have also seen the self-destruct message on boxes with autofs4 in kernel 2.4.18 (w/ rmap12h patch) and autofs4-4.0.0pre10-10 userspace daemon.
Thanks,
<< ryan
Quoted 33 lines · Ian Kent
Quoted 9 lines · Arun Sharma
We've seen a few "VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day.." messages on a dual processor NFS client. Here's the use case:
- Due to network load issues, the NFS server becomes unreachable for some time
- The automounter tries to expire the mount
- The unmount finds a couple of busy inodes. Putting some debug printks shows that typically two inodes are busy i.e. haveinode->i_count== 1. But they don't have any waiters oninode->i_wait. Further, the inodes that are busy are/mnt/foo-> autofs mount point/mnt/foo/bar-> bar is a symbolic linkQuoted 5 lines
It's not clear if this is a NFS issue or a autofs issue, but it's seen often with autofs. Are there any known race conditions that have been fixed after 2.4.20 ? The two calls I'm worried about are:
fs/autofs/root.c:305:d_instantiate(dentry, iget(dir->i_sb,ent->ino));fs/autofs/root.c:416:d_instantiate(dentry, iget(dir->i_sb,ino));This is in the autofs (aka v3) kernel module not autofs4.
Please specify which version of the userspace daemon and kernel module?
--
,-._|\ Ian Kent / \ Perth, Western Australia *_.--._/ E-mail: [email protected] v Web: http://themaw.net/_______________________________________________
autofs mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/autofs