automount in status D
Hi all,
we have a weird problem here and I fear I need to reboot the box (painful, many users involved), but maybe there is a way out:
I'm automounting many nodes and usually everything seems to work nicely. However, I'm seeing this here right now:
# ps aux|grep auto.node root 2022 0.0 0.0 30304 1240 ? Ds Jun19 0:21 /usr/sbin/automount --pid-file=/var/run/autofs/_atlas_node.pid --timeout=5 /atlas/node yp auto.node_local root 11489 0.0 0.0 30516 1108 ? D Jun26 0:00 /usr/sbin/automount --pid-file=/var/run/autofs/_atlas_node.pid --timeout=5 /atlas/node yp auto.node_local root 28610 0.0 0.0 2736 656 pts/31 S+ 10:03 0:00 grep auto.node
As you can see there are two automounter processes in D (or Ds) state sitting there for the same mount point (the PID-file contains 2022).
My questions:
Any idea how this happened?
Any idea how to get this resolved (possibly without a reboot)?
TIA
Carsten
PS:
# ypcat auto.node_local -fstype=nfs,nfsvers=3,soft,intr,rsize=32768,wsize=32768,tcp &:/local
Ian Kent
maintainerQuoted 14 lines · Carsten Aulbert
Hi all,
we have a weird problem here and I fear I need to reboot the box (painful, many users involved), but maybe there is a way out:
I'm automounting many nodes and usually everything seems to work nicely. However, I'm seeing this here right now:
# ps aux|grep auto.node root 2022 0.0 0.0 30304 1240 ? Ds Jun19 0:21 /usr/sbin/automount --pid-file=/var/run/autofs/_atlas_node.pid --timeout=5 /atlas/node yp auto.node_local root 11489 0.0 0.0 30516 1108 ? D Jun26 0:00 /usr/sbin/automount --pid-file=/var/run/autofs/_atlas_node.pid --timeout=5 /atlas/node yp auto.node_local root 28610 0.0 0.0 2736 656 pts/31 S+ 10:03 0:00 grep auto.node
Quoted 5 lines
As you can see there are two automounter processes in D (or Ds) state sitting there for the same mount point (the PID-file contains 2022).
My questions:
Any idea how this happened?
It's been a long time since I've seen this reported. What distro and kernel?
Quoted 2 lines
Any idea how to get this resolved (possibly without a reboot)?
There is nothing you can do if the process is stuck in an uninterruptible state in the kernel.
Ian
Quoted 5 lines · Ian Kent
Quoted 1 line
Any idea how this happened?
It's been a long time since I've seen this reported. What distro and kernel?
Sorry, Debian Etch with self compiled kernel:
h2:~# uname -a Linux h2 2.6.24.4-storage #1 SMP Sun Apr 20 09:33:42 CEST 2008 x86_64 x86_64 x86_64 GNU/Linux h2:~# apt-cache show autofs|grep Version Version: 4.1.4-13
Quoted 4 lines
Quoted 1 line
Any idea how to get this resolved (possibly without a reboot)?
There is nothing you can do if the process is stuck in an uninterruptible state in the kernel.
Well, as bad as I feared it to be :(
Thanks already, reboot is already announced to the users. Anything I might help you with to corner that bug?
Cheers
Carsten
Signature · 9 lines
Dr. Carsten Aulbert - Max Planck Institut für Gravitationsphysik Callinstraße 38, 30167 Hannover, Germany Fon: +49 511 762 17185, Fax: +49 511 762 17193 http://www.top500.org/system/9234 | http://www.top500.org/connfam/6/list/31 _______________________________________________ autofs mailing list [email protected] http://linux.kernel.org/mailman/listinfo/autofs
Ian Kent
maintainerQuoted 18 lines · Carsten Aulbert
Quoted 5 lines · Ian Kent
Quoted 1 line
Any idea how this happened?
It's been a long time since I've seen this reported. What distro and kernel?
Sorry, Debian Etch with self compiled kernel:
h2:~# uname -a Linux h2 2.6.24.4-storage #1 SMP Sun Apr 20 09:33:42 CEST 2008 x86_64 x86_64 x86_64 GNU/Linux h2:~# apt-cache show autofs|grep Version Version: 4.1.4-13Quoted 2 lines
Quoted 1 line
Any idea how to get this resolved (possibly without a reboot)?
Quoted 7 lines
Quoted 2 lines
There is nothing you can do if the process is stuck in an uninterruptible state in the kernel.
Well, as bad as I feared it to be :(
Thanks already, reboot is already announced to the users. Anything I might help you with to corner that bug?
Aggh .. it's been soooo long since I've heard of this happening I can't remember.
One thing that is interesting is that the autofs4 kernel module doesn't do an uninterruptible sleep "anywhere". It hasn't since long before 2.6.24. So if user space can't recover by doing a control-c (maybe even a couple of times) then it's possibly not actually autofs.
Ian
Hi Ian,
Quoted 4 lines · Ian Kent
One thing that is interesting is that the autofs4 kernel module doesn't do an uninterruptible sleep "anywhere". It hasn't since long before 2.6.24. So if user space can't recover by doing a control-c (maybe even a couple of times) then it's possibly not actually autofs.
Well, when I tried to umount it manually (with options -f -l -i) stopped indefinitely when doing a readlink on the mount point itself and never returned from its own account.
I was able to CTRL-C this one but not getting the mount point removed. A nasty side effect was that df never returned either.
I did a bit more log reading and it seems that the node got a mount as well as a umount request yesterday evening and from the nodes perspective everything looked fine.
In the end I guess this falls into the category of: Hopefully it won't happen again too soon ;)
If you come up with an idea, please let me know. I'll stay on the list, reading quietly.
Cheers
Carsten
Quoted 22 lines · Ian Kent
Quoted 21 lines · Carsten Aulbert
Quoted 5 lines · Ian Kent
Quoted 1 line
Any idea how this happened?
It's been a long time since I've seen this reported. What distro and kernel?
Sorry, Debian Etch with self compiled kernel:
h2:~# uname -a Linux h2 2.6.24.4-storage #1 SMP Sun Apr 20 09:33:42 CEST 2008 x86_64 x86_64 x86_64 GNU/Linux h2:~# apt-cache show autofs|grep Version Version: 4.1.4-13Quoted 4 lines
Quoted 1 line
Any idea how to get this resolved (possibly without a reboot)?
There is nothing you can do if the process is stuck in an uninterruptible state in the kernel.
Quoted 7 lines
Quoted 4 lines
Well, as bad as I feared it to be :(
Thanks already, reboot is already announced to the users. Anything I might help you with to corner that bug?
Aggh .. it's been soooo long since I've heard of this happening I can't remember.
A sysrq-t would be a good starting point.
-Jeff