two /net paths to the same local mount?
When I use /net/host and /net/host.domain to access a file system that is local to the current host, the second flavor of access will hang.
We have a complex build environment running on multiple hosts, so directory names often need to work either locally or remotely. That's why we're using /net paths for the local host.
Is this a known bug?
I can supply more details, if necessary.
Rebuilding from source is not really practical in my environment.
--chris
<reply inline>
Quoted 3 lines · --On Monday, June 21, 2010 12:17 PM -0700 Chris Quenelle…
When I use /net/host and /net/host.domain to access a file system that is local to the current host, the second flavor of access will hang.
Check the /etc/hosts file on the machine having the problem. Usually this means that your'e expecting it to contact from/to one address (say the loopback in the case of hostname) but the hosts file has another for the hostname.domain case, or doesn't have one, so it's getting the external address via DNS.
Quoted 18 lines
We have a complex build environment running on multiple hosts, so directory names often need to work either locally or remotely. That's why we're using /net paths for the local host.
Is this a known bug?
I can supply more details, if necessary.
Rebuilding from source is not really practical in my environment.--chris
_______________________________________________
autofs mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/autofs
Thanks for your time guys!
It turns out it's not a general bug. I was just applying a pessimistic interpretation without checking.
After more testing, it turns out it's just problem on one machine, for two mount points under one host alias.
I'm interested in knowing more about how I could
debug the situation without just rebooting and
hoping it doesn't happen again.
Here's my scenario:
I have three exported filesystems.
I can use three host aliases to refer to my own machine. (carabas, carabas.sfbay, carabas.sfbay.sun.com)
/etc/hosts says:
[IP addr] carabas.sfbay.sun.com carabas
All filesystems work under all /net/foo aliases
except two of them hang when accessed via one
of the host aliases.
% showmount -e localhost
Export list for localhost:
/export/home3 *
/export/home2 *
/export/home1 *
% df -kl
Filesystem 1K-blocks Used Available Use% Mounted on/dev/sda2 69575344 3408900 66166444 5% /
udev 8222568 148 8222420 1% /dev
/dev/sdb1 71671728 26507132 45164596 37% /export/home1 /dev/sdc1 71671728 8545196 63126532 12% /export/home2 /dev/sdd1 71671728 5375672 66296056 8% /export/home3 /export/home1 71671728 26507128 45164600 37% /net/carabas/export/home1 /export/home2 71671728 8545196 63126532 12% /net/carabas.sfbay/export/home2 /export/home1 71671728 26507128 45164600 37% /net/carabas.sfbay/export/home1 /export/home3 71671728 5375672 66296056 8% /net/carabas.sfbay/export/home3 /export/home1 71671728 26507128 45164600 37% /net/carabas.sfbay.sun.com/export/home1 /export/home2 71671728 8545196 63126532 12% /net/carabas.sfbay.sun.com/export/home2 /export/home3 71671728 5375672 66296056 8% /net/carabas.sfbay.sun.com/export/home3
% strace ls /net/carabas/export/home2
...
...mmap(NULL, 217016, PROT_READ, MAP_SHARED, 4, 0) = 0x2b2388370000
close(4) = 0
close(3) = 0
open("/net/carabas/export/home2", O_RDONLY|O_NONBLOCK|O_DIRECTORY <unfinished ...>
I hit ctrl-C when the ls command hangs. Above is the tail of the strace output. It doesn't tell you much, the interesting stuff happens in the automounter.
Is there a error log file for the automounter that is enabled by default? Is there an easy recipe for restarting the automounter with debugging output on?
--chris
Quoted 37 lines · Michael Loftis
<reply inline>
--On Monday, June 21, 2010 12:17 PM -0700 Chris Quenelle
Quoted 3 lines · <[email protected]> wrote:
When I use /net/host and /net/host.domain to access a file system that is local to the current host, the second flavor of access will hang.
Check the
/etc/hostsfile on the machine having the problem. Usually this means that your'e expecting it to contact from/to one address (say the loopback in the case of hostname) but the hosts file has another for the hostname.domain case, or doesn't have one, so it's getting the external address via DNS.Quoted 18 lines
We have a complex build environment running on multiple hosts, so directory names often need to work either locally or remotely. That's why we're using /net paths for the local host.
Is this a known bug?
I can supply more details, if necessary.
Rebuilding from source is not really practical in my environment.--chris
_______________________________________________
autofs mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/autofs
Ian Kent
maintainerQuoted 19 lines · Chris Quenelle
Thanks for your time guys!
It turns out it's not a general bug. I was just applying a pessimistic interpretation without checking.
After more testing, it turns out it's just problem on one machine, for two mount points under one host alias.
I'm interested in knowing more about how I could
debug the situation without just rebooting and
hoping it doesn't happen again.Here's my scenario:
I have three exported filesystems.
I can use three host aliases to refer to my own machine. (carabas, carabas.sfbay, carabas.sfbay.sun.com)
Quoted 7 lines
/etc/hostssays:
[IP addr] carabas.sfbay.sun.com carabasAll filesystems work under all /net/foo aliases
except two of them hang when accessed via one
of the host aliases.
Are you sure the address lookup is returning what you think it does? What's in /etc/host.conf?
Quoted 24 lines
% showmount -e localhost
Export list for localhost:
/export/home3 *
/export/home2 *
/export/home1 *% df -kl
Filesystem 1K-blocks Used Available Use% Mounted on/dev/sda269575344 3408900 66166444 5% /
udev 8222568 148 8222420 1% /dev/dev/sdb1 71671728 26507132 45164596 37% /export/home1 /dev/sdc1 71671728 8545196 63126532 12% /export/home2 /dev/sdd1 71671728 5375672 66296056 8% /export/home3 /export/home1 71671728 26507128 45164600 37% /net/carabas/export/home1 /export/home2 71671728 8545196 63126532 12% /net/carabas.sfbay/export/home2 /export/home1 71671728 26507128 45164600 37% /net/carabas.sfbay/export/home1 /export/home3 71671728 5375672 66296056 8% /net/carabas.sfbay/export/home3 /export/home1 71671728 26507128 45164600 37% /net/carabas.sfbay.sun.com/export/home1 /export/home2 71671728 8545196 63126532 12% /net/carabas.sfbay.sun.com/export/home2 /export/home3 71671728 5375672 66296056 8% /net/carabas.sfbay.sun.com/export/home3
Quoted 7 lines
% strace ls /net/carabas/export/home2
...
...mmap(NULL, 217016, PROT_READ, MAP_SHARED, 4, 0)=0x2b2388370000
close(4) = 0
close(3) = 0
open("/net/carabas/export/home2", O_RDONLY|O_NONBLOCK|O_DIRECTORY <unfinished ...>
What does a ps listing give you when it is blocked? Any mount.nfs processes?
That's sign that automount thinks that the mount isn't local and is probably trying to mount from a possibly non-existent ip address.
Quoted 6 lines
I hit ctrl-C when the ls command hangs. Above is the tail of the strace output. It doesn't tell you much, the interesting stuff happens in the automounter.
Is there a error log file for the automounter that is enabled by default? Is there an easy recipe for restarting the automounter with debugging output on?
You need to ensure that debug logging is being recorded in syslog. Ensure you are sending daemon.* somewhere.
Then
automount -l debug /net
will start the debug logging and
automount -l err /net
will set it back to no logging.
That is depending on the version of autofs you are using.
If you don't see this option in the automount(8) man page you don't have this option and all you can do is enable debug logging in the configuration.
Ian
Ian,
Thanks again for your help.
I have syslog-ng installed, and I can't figure out how to tell if "daemon.*" is being sent somewhere or not. "daemon" doesn't appear anywhere in /etc/syslog-ng/syslog-ng.conf
But when I grep /var/log/messages I see:
% grep automount messages
Jun 29 02:41:14 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 02:41:14 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 03:17:45 carabas automount[11786]: lookup_mount: lookup(yp): key "opt" not found in map Jun 29 03:24:27 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 03:24:27 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 09:04:46 carabas automount[11786]: Debug logging set for /net Jun 29 09:09:22 carabas automount[11786]: get_pkt: message pending on control fifo. Jun 29 09:09:22 carabas automount[11786]: Basic logging set for /net
I didn't get the get_pkg message until I turned off debug logging.
While logging was set to debug, I tried the hanging ls command again, and I also tried listing a different file system that was actually remote. I didn't see any log messages for those operations.
Notes:
1. /etc/host.conf says only:
order hosts, bind
multi on
2. No mount.nfs processes are running when the ls command hangs on the local mount.
Quoted 96 lines · Ian Kent
Quoted 19 lines · Chris Quenelle
Thanks for your time guys!
It turns out it's not a general bug. I was just applying a pessimistic interpretation without checking.
After more testing, it turns out it's just problem on one machine, for two mount points under one host alias.
I'm interested in knowing more about how I could
debug the situation without just rebooting and
hoping it doesn't happen again.Here's my scenario:
I have three exported filesystems.
I can use three host aliases to refer to my own machine. (carabas, carabas.sfbay, carabas.sfbay.sun.com)Quoted 7 lines
/etc/hostssays:
[IP addr] carabas.sfbay.sun.com carabasAll filesystems work under all /net/foo aliases
except two of them hang when accessed via one
of the host aliases.Are you sure the address lookup is returning what you think it does? What's in
/etc/host.conf?Quoted 23 lines
% showmount -e localhost
Export list for localhost:
/export/home3 *
/export/home2 *
/export/home1 *% df -kl
Filesystem 1K-blocks Used Available Use% Mounted on/dev/sda269575344 3408900 66166444 5% /
udev 8222568 148 8222420 1% /dev/dev/sdb1 71671728 26507132 45164596 37% /export/home1 /dev/sdc1 71671728 8545196 63126532 12% /export/home2 /dev/sdd1 71671728 5375672 66296056 8% /export/home3 /export/home1 71671728 26507128 45164600 37% /net/carabas/export/home1 /export/home2 71671728 8545196 63126532 12% /net/carabas.sfbay/export/home2 /export/home1 71671728 26507128 45164600 37% /net/carabas.sfbay/export/home1 /export/home3 71671728 5375672 66296056 8% /net/carabas.sfbay/export/home3 /export/home1 71671728 26507128 45164600 37% /net/carabas.sfbay.sun.com/export/home1 /export/home2 71671728 8545196 63126532 12% /net/carabas.sfbay.sun.com/export/home2 /export/home3 71671728 5375672 66296056 8% /net/carabas.sfbay.sun.com/export/home3Quoted 7 lines
% strace ls /net/carabas/export/home2
...
...mmap(NULL, 217016, PROT_READ, MAP_SHARED, 4, 0)=0x2b2388370000
close(4) = 0
close(3) = 0
open("/net/carabas/export/home2", O_RDONLY|O_NONBLOCK|O_DIRECTORY <unfinished ...>What does a ps listing give you when it is blocked? Any mount.nfs processes?
That's sign that automount thinks that the mount isn't local and is probably trying to mount from a possibly non-existent ip address.
Quoted 5 lines
I hit ctrl-C when the ls command hangs. Above is the tail of the strace output. It doesn't tell you much, the interesting stuff happens in the automounter.
Is there a error log file for the automounter that is enabled by default? Is there an easy recipe for restarting the automounter with debugging output on?
You need to ensure that debug logging is being recorded in syslog. Ensure you are sending daemon.* somewhere.
Then
automount -l debug /net
will start the debug logging and
automount -l err /net
will set it back to no logging.
That is depending on the version of autofs you are using.
If you don't see this option in the automount(8) man page you don't have this option and all you can do is enable debug logging in the configuration.
Ian
Ian Kent
maintainerQuoted 6 lines · Chris Quenelle
Ian,
Thanks again for your help.
I have syslog-ng installed, and I can't figure out how to tell if "daemon.*" is being sent somewhere or not. "daemon" doesn't appear anywhere in/etc/syslog-ng/syslog-ng.conf
Well, you need to consult the man page for the configuration file then don't you.
For me, when syslog was in use and now with rsyslog, I just add a new line to send all the facility daemon output to a log file.
I add the line:
daemon.* /var/log/debug
to the configuration and then
touch /var/log/debug
and finally restart logging service.
You probably should post your autofs maps as well.
Quoted 126 lines
But when I grep
/var/log/messagesI see:% grep automount messages
Jun 29 02:41:14 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 02:41:14 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 03:17:45 carabas automount[11786]: lookup_mount: lookup(yp): key "opt" not found in map Jun 29 03:24:27 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 03:24:27 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 09:04:46 carabas automount[11786]: Debug logging set for /net Jun 29 09:09:22 carabas automount[11786]: get_pkt: message pending on control fifo. Jun 29 09:09:22 carabas automount[11786]: Basic logging set for /netI didn't get the
get_pkgmessage until I turned off debug logging.While logging was set to debug, I tried the hanging ls command again, and I also tried listing a different file system that was actually remote. I didn't see any log messages for those operations.
Notes:
1.
/etc/host.confsays only:
order hosts, bind
multi on2. No mount.nfs processes are running when the ls command hangs on the local mount.
Quoted 96 lines · Ian Kent
Quoted 19 lines · Chris Quenelle
Thanks for your time guys!
It turns out it's not a general bug. I was just applying a pessimistic interpretation without checking.
After more testing, it turns out it's just problem on one machine, for two mount points under one host alias.
I'm interested in knowing more about how I could
debug the situation without just rebooting and
hoping it doesn't happen again.Here's my scenario:
I have three exported filesystems.
I can use three host aliases to refer to my own machine. (carabas, carabas.sfbay, carabas.sfbay.sun.com)Quoted 7 lines
/etc/hostssays:
[IP addr] carabas.sfbay.sun.com carabasAll filesystems work under all /net/foo aliases
except two of them hang when accessed via one
of the host aliases.Are you sure the address lookup is returning what you think it does? What's in
/etc/host.conf?Quoted 23 lines
% showmount -e localhost
Export list for localhost:
/export/home3 *
/export/home2 *
/export/home1 *% df -kl
Filesystem 1K-blocks Used Available Use% Mounted on/dev/sda269575344 3408900 66166444 5% /
udev 8222568 148 8222420 1% /dev/dev/sdb1 71671728 26507132 45164596 37% /export/home1 /dev/sdc1 71671728 8545196 63126532 12% /export/home2 /dev/sdd1 71671728 5375672 66296056 8% /export/home3 /export/home1 71671728 26507128 45164600 37% /net/carabas/export/home1 /export/home2 71671728 8545196 63126532 12% /net/carabas.sfbay/export/home2 /export/home1 71671728 26507128 45164600 37% /net/carabas.sfbay/export/home1 /export/home3 71671728 5375672 66296056 8% /net/carabas.sfbay/export/home3 /export/home1 71671728 26507128 45164600 37% /net/carabas.sfbay.sun.com/export/home1 /export/home2 71671728 8545196 63126532 12% /net/carabas.sfbay.sun.com/export/home2 /export/home3 71671728 5375672 66296056 8% /net/carabas.sfbay.sun.com/export/home3Quoted 7 lines
% strace ls /net/carabas/export/home2
...
...mmap(NULL, 217016, PROT_READ, MAP_SHARED, 4, 0)=0x2b2388370000
close(4) = 0
close(3) = 0
open("/net/carabas/export/home2", O_RDONLY|O_NONBLOCK|O_DIRECTORY <unfinished ...>What does a ps listing give you when it is blocked? Any mount.nfs processes?
That's sign that automount thinks that the mount isn't local and is probably trying to mount from a possibly non-existent ip address.
Quoted 5 lines
I hit ctrl-C when the ls command hangs. Above is the tail of the strace output. It doesn't tell you much, the interesting stuff happens in the automounter.
Is there a error log file for the automounter that is enabled by default? Is there an easy recipe for restarting the automounter with debugging output on?
You need to ensure that debug logging is being recorded in syslog. Ensure you are sending daemon.* somewhere.
Then
automount -l debug /net
will start the debug logging and
automount -l err /net
will set it back to no logging.
That is depending on the version of autofs you are using.
If you don't see this option in the automount(8) man page you don't have this option and all you can do is enable debug logging in the configuration.
Ian
Quoted 10 lines · Ian Kent
Quoted 6 lines · Chris Quenelle
Ian,
Thanks again for your help.
I have syslog-ng installed, and I can't figure out how to tell if "daemon.*" is being sent somewhere or not. "daemon" doesn't appear anywhere in/etc/syslog-ng/syslog-ng.confWell, you need to consult the man page for the configuration file then don't you.
Sorry for sounding lazy there. I've never used syslog-ng before, and I read the man page for a while without figuring how the old style of names/categories mapped to the new system. I'll continue to fiddle around and see if I can find out what's going on.
Thanks for helping me get started with the debugging output.
My autofs map looks like this:
% ypcat -k auto.master | grep /net
/net -hosts -intr,nosuid,retrans=10,retry=3,nobrowse
But I'm not sure that helps.
--chris
Quoted 144 lines
For me, when syslog was in use and now with rsyslog, I just add a new line to send all the facility daemon output to a log file.
I add the line:
daemon.*
/var/log/debugto the configuration and then
touch
/var/log/debugand finally restart logging service.
You probably should post your autofs maps as well.
Quoted 126 lines
But when I grep
/var/log/messagesI see:% grep automount messages
Jun 29 02:41:14 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 02:41:14 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 03:17:45 carabas automount[11786]: lookup_mount: lookup(yp): key "opt" not found in map Jun 29 03:24:27 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 03:24:27 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 09:04:46 carabas automount[11786]: Debug logging set for /net Jun 29 09:09:22 carabas automount[11786]: get_pkt: message pending on control fifo. Jun 29 09:09:22 carabas automount[11786]: Basic logging set for /netI didn't get the
get_pkgmessage until I turned off debug logging.While logging was set to debug, I tried the hanging ls command again, and I also tried listing a different file system that was actually remote. I didn't see any log messages for those operations.
Notes:
1.
/etc/host.confsays only:
order hosts, bind
multi on2. No mount.nfs processes are running when the ls command hangs on the local mount.
Quoted 96 lines · Ian Kent
Quoted 19 lines · Chris Quenelle
Thanks for your time guys!
It turns out it's not a general bug. I was just applying a pessimistic interpretation without checking.
After more testing, it turns out it's just problem on one machine, for two mount points under one host alias.
I'm interested in knowing more about how I could
debug the situation without just rebooting and
hoping it doesn't happen again.Here's my scenario:
I have three exported filesystems.
I can use three host aliases to refer to my own machine. (carabas, carabas.sfbay, carabas.sfbay.sun.com)Quoted 7 lines
/etc/hostssays:
[IP addr] carabas.sfbay.sun.com carabasAll filesystems work under all /net/foo aliases
except two of them hang when accessed via one
of the host aliases.Are you sure the address lookup is returning what you think it does? What's in
/etc/host.conf?Quoted 23 lines
% showmount -e localhost
Export list for localhost:
/export/home3 *
/export/home2 *
/export/home1 *% df -kl
Filesystem 1K-blocks Used Available Use% Mounted on/dev/sda269575344 3408900 66166444 5% /
udev 8222568 148 8222420 1% /dev/dev/sdb1 71671728 26507132 45164596 37% /export/home1 /dev/sdc1 71671728 8545196 63126532 12% /export/home2 /dev/sdd1 71671728 5375672 66296056 8% /export/home3 /export/home1 71671728 26507128 45164600 37% /net/carabas/export/home1 /export/home2 71671728 8545196 63126532 12% /net/carabas.sfbay/export/home2 /export/home1 71671728 26507128 45164600 37% /net/carabas.sfbay/export/home1 /export/home3 71671728 5375672 66296056 8% /net/carabas.sfbay/export/home3 /export/home1 71671728 26507128 45164600 37% /net/carabas.sfbay.sun.com/export/home1 /export/home2 71671728 8545196 63126532 12% /net/carabas.sfbay.sun.com/export/home2 /export/home3 71671728 5375672 66296056 8% /net/carabas.sfbay.sun.com/export/home3Quoted 7 lines
% strace ls /net/carabas/export/home2
...
...mmap(NULL, 217016, PROT_READ, MAP_SHARED, 4, 0)=0x2b2388370000
close(4) = 0
close(3) = 0
open("/net/carabas/export/home2", O_RDONLY|O_NONBLOCK|O_DIRECTORY<unfinished ...>What does a ps listing give you when it is blocked? Any mount.nfs processes?
That's sign that automount thinks that the mount isn't local and is probably trying to mount from a possibly non-existent ip address.
Quoted 5 lines
I hit ctrl-C when the ls command hangs. Above is the tail of the strace output. It doesn't tell you much, the interesting stuff happens in the automounter.
Is there a error log file for the automounter that is enabled by default? Is there an easy recipe for restarting the automounter with debugging output on?
You need to ensure that debug logging is being recorded in syslog. Ensure you are sending daemon.* somewhere.
Then
automount -l debug /net
will start the debug logging and
automount -l err /net
will set it back to no logging.
That is depending on the version of autofs you are using.
If you don't see this option in the automount(8) man page you don't have this option and all you can do is enable debug logging in the configuration.
Ian
Ian Kent
maintainerQuoted 17 lines · Chris Quenelle
Quoted 10 lines · Ian Kent
Quoted 6 lines · Chris Quenelle
Ian,
Thanks again for your help.
I have syslog-ng installed, and I can't figure out how to tell if "daemon.*" is being sent somewhere or not. "daemon" doesn't appear anywhere in/etc/syslog-ng/syslog-ng.confWell, you need to consult the man page for the configuration file then don't you.
Sorry for sounding lazy there. I've never used syslog-ng before, and I read the man page for a while without figuring how the old style of names/categories mapped to the new system. I'll continue to fiddle around and see if I can find out what's going on.
Quoted 7 lines
Thanks for helping me get started with the debugging output.
My autofs map looks like this:
% ypcat -k auto.master | grep /net
/net -hosts -intr,nosuid,retrans=10,retry=3,nobrowseBut I'm not sure that helps.
Is automount crashing, ie. does it show up in a ps listing after you see it hang?
Quoted 150 lines
--chris
Quoted 144 lines
For me, when syslog was in use and now with rsyslog, I just add a new line to send all the facility daemon output to a log file.
I add the line:
daemon.*
/var/log/debugto the configuration and then
touch
/var/log/debugand finally restart logging service.
You probably should post your autofs maps as well.
Quoted 126 lines
But when I grep
/var/log/messagesI see:% grep automount messages
Jun 29 02:41:14 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 02:41:14 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 03:17:45 carabas automount[11786]: lookup_mount: lookup(yp): key "opt" not found in map Jun 29 03:24:27 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 03:24:27 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 09:04:46 carabas automount[11786]: Debug logging set for /net Jun 29 09:09:22 carabas automount[11786]: get_pkt: message pending on control fifo. Jun 29 09:09:22 carabas automount[11786]: Basic logging set for /netI didn't get the
get_pkgmessage until I turned off debug logging.While logging was set to debug, I tried the hanging ls command again, and I also tried listing a different file system that was actually remote. I didn't see any log messages for those operations.
Notes:
1.
/etc/host.confsays only:
order hosts, bind
multi on2. No mount.nfs processes are running when the ls command hangs on the local mount.
Quoted 96 lines · Ian Kent
Quoted 19 lines · Chris Quenelle
Thanks for your time guys!
It turns out it's not a general bug. I was just applying a pessimistic interpretation without checking.
After more testing, it turns out it's just problem on one machine, for two mount points under one host alias.
I'm interested in knowing more about how I could
debug the situation without just rebooting and
hoping it doesn't happen again.Here's my scenario:
I have three exported filesystems.
I can use three host aliases to refer to my own machine. (carabas, carabas.sfbay, carabas.sfbay.sun.com)Quoted 7 lines
/etc/hostssays:
[IP addr] carabas.sfbay.sun.com carabasAll filesystems work under all /net/foo aliases
except two of them hang when accessed via one
of the host aliases.Are you sure the address lookup is returning what you think it does? What's in
/etc/host.conf?Quoted 23 lines
% showmount -e localhost
Export list for localhost:
/export/home3 *
/export/home2 *
/export/home1 *% df -kl
Filesystem 1K-blocks Used Available Use% Mounted on/dev/sda269575344 3408900 66166444 5% /
udev 8222568 148 8222420 1% /dev/dev/sdb1 71671728 26507132 45164596 37% /export/home1 /dev/sdc1 71671728 8545196 63126532 12% /export/home2 /dev/sdd1 71671728 5375672 66296056 8% /export/home3 /export/home1 71671728 26507128 45164600 37% /net/carabas/export/home1 /export/home2 71671728 8545196 63126532 12% /net/carabas.sfbay/export/home2 /export/home1 71671728 26507128 45164600 37% /net/carabas.sfbay/export/home1 /export/home3 71671728 5375672 66296056 8% /net/carabas.sfbay/export/home3 /export/home1 71671728 26507128 45164600 37% /net/carabas.sfbay.sun.com/export/home1 /export/home2 71671728 8545196 63126532 12% /net/carabas.sfbay.sun.com/export/home2 /export/home3 71671728 5375672 66296056 8% /net/carabas.sfbay.sun.com/export/home3Quoted 7 lines
% strace ls /net/carabas/export/home2
...
...mmap(NULL, 217016, PROT_READ, MAP_SHARED, 4, 0)=0x2b2388370000
close(4) = 0
close(3) = 0
open("/net/carabas/export/home2", O_RDONLY|O_NONBLOCK|O_DIRECTORY<unfinished ...>What does a ps listing give you when it is blocked? Any mount.nfs processes?
That's sign that automount thinks that the mount isn't local and is probably trying to mount from a possibly non-existent ip address.
Quoted 5 lines
I hit ctrl-C when the ls command hangs. Above is the tail of the strace output. It doesn't tell you much, the interesting stuff happens in the automounter.
Is there a error log file for the automounter that is enabled by default? Is there an easy recipe for restarting the automounter with debugging output on?
You need to ensure that debug logging is being recorded in syslog. Ensure you are sending daemon.* somewhere.
Then
automount -l debug /net
will start the debug logging and
automount -l err /net
will set it back to no logging.
That is depending on the version of autofs you are using.
If you don't see this option in the automount(8) man page you don't have this option and all you can do is enable debug logging in the configuration.
Ian
Quoted 18 lines · Ian Kent
Quoted 17 lines · Chris Quenelle
Quoted 9 lines · Ian Kent
Quoted 6 lines · Chris Quenelle
Ian,
Thanks again for your help.
I have syslog-ng installed, and I can't figure out how to tell if "daemon.*" is being sent somewhere or not. "daemon" doesn't appear anywhere in/etc/syslog-ng/syslog-ng.confWell, you need to consult the man page for the configuration file then don't you.
Sorry for sounding lazy there. I've never used syslog-ng before, and I read the man page for a while without figuring how the old style of names/categories mapped to the new system. I'll continue to fiddle around and see if I can find out what's going on.
Thanks for helping me get started with the debugging output.
Quoted 8 lines
Quoted 5 lines
My autofs map looks like this:
% ypcat -k auto.master | grep /net
/net -hosts -intr,nosuid,retrans=10,retry=3,nobrowseBut I'm not sure that helps.
Is automount crashing, ie. does it show up in a ps listing after you see it hang?
Nope. The ps command doesn't show any change in the status of the automount process.
In fact, when I attach strace to the automount process I don't see any sign of activity when I do an "ls" on the unhealthy path.
It sounds like I would need to look at kernel tracing/debugging if I want to pursue this further.
Quoted 145 lines
Quoted 143 lines
--chris
Quoted 138 lines
For me, when syslog was in use and now with rsyslog, I just add a new line to send all the facility daemon output to a log file.
I add the line:
daemon.*
/var/log/debugto the configuration and then
touch
/var/log/debugand finally restart logging service.
You probably should post your autofs maps as well.
Quoted 122 lines
But when I grep
/var/log/messagesI see:% grep automount messages
Jun 29 02:41:14 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 02:41:14 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 03:17:45 carabas automount[11786]: lookup_mount: lookup(yp): key "opt" not found in map Jun 29 03:24:27 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 03:24:27 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 09:04:46 carabas automount[11786]: Debug logging set for /net Jun 29 09:09:22 carabas automount[11786]: get_pkt: message pending on control fifo. Jun 29 09:09:22 carabas automount[11786]: Basic logging set for /netI didn't get the
get_pkgmessage until I turned off debug logging.While logging was set to debug, I tried the hanging ls command again, and I also tried listing a different file system that was actually remote. I didn't see any log messages for those operations.
Notes:
1.
/etc/host.confsays only:
order hosts, bind
multi on2. No mount.nfs processes are running when the ls command hangs on the local mount.
Quoted 93 lines · Ian Kent
Quoted 19 lines · Chris Quenelle
Thanks for your time guys!
It turns out it's not a general bug. I was just applying a pessimistic interpretation without checking.
After more testing, it turns out it's just problem on one machine, for two mount points under one host alias.
I'm interested in knowing more about how I could
debug the situation without just rebooting and
hoping it doesn't happen again.Here's my scenario:
I have three exported filesystems.
I can use three host aliases to refer to my own machine. (carabas, carabas.sfbay, carabas.sfbay.sun.com)Quoted 7 lines
/etc/hostssays:
[IP addr] carabas.sfbay.sun.com carabasAll filesystems work under all /net/foo aliases
except two of them hang when accessed via one
of the host aliases.Are you sure the address lookup is returning what you think it does? What's in
/etc/host.conf?Quoted 23 lines
% showmount -e localhost
Export list for localhost:
/export/home3 *
/export/home2 *
/export/home1 *% df -kl
Filesystem 1K-blocks Used Available Use% Mounted on/dev/sda269575344 3408900 66166444 5% /
udev 8222568 148 8222420 1% /dev/dev/sdb1 71671728 26507132 45164596 37% /export/home1 /dev/sdc1 71671728 8545196 63126532 12% /export/home2 /dev/sdd1 71671728 5375672 66296056 8% /export/home3 /export/home1 71671728 26507128 45164600 37% /net/carabas/export/home1 /export/home2 71671728 8545196 63126532 12% /net/carabas.sfbay/export/home2 /export/home1 71671728 26507128 45164600 37% /net/carabas.sfbay/export/home1 /export/home3 71671728 5375672 66296056 8% /net/carabas.sfbay/export/home3 /export/home1 71671728 26507128 45164600 37% /net/carabas.sfbay.sun.com/export/home1 /export/home2 71671728 8545196 63126532 12% /net/carabas.sfbay.sun.com/export/home2 /export/home3 71671728 5375672 66296056 8% /net/carabas.sfbay.sun.com/export/home3Quoted 7 lines
% strace ls /net/carabas/export/home2
...
...mmap(NULL, 217016, PROT_READ, MAP_SHARED, 4, 0)=0x2b2388370000
close(4) = 0
close(3) = 0
open("/net/carabas/export/home2", O_RDONLY|O_NONBLOCK|O_DIRECTORY<unfinished ...>What does a ps listing give you when it is blocked? Any mount.nfs processes?
That's sign that automount thinks that the mount isn't local and is probably trying to mount from a possibly non-existent ip address.
Quoted 5 lines
I hit ctrl-C when the ls command hangs. Above is the tail of the strace output. It doesn't tell you much, the interesting stuff happens in the automounter.
Is there a error log file for the automounter that is enabled by default? Is there an easy recipe for restarting the automounter with debugging output on?
You need to ensure that debug logging is being recorded in syslog. Ensure you are sending daemon.* somewhere.
Then
automount -l debug /net
will start the debug logging and
automount -l err /net
will set it back to no logging.
That is depending on the version of autofs you are using.
If you don't see this option in the automount(8) man page you don't have this option and all you can do is enable debug logging in the configuration.
Ian
Ian Kent
maintainerQuoted 29 lines · Chris Quenelle
Quoted 18 lines · Ian Kent
Quoted 17 lines · Chris Quenelle
Quoted 9 lines · Ian Kent
Quoted 6 lines · Chris Quenelle
Ian,
Thanks again for your help.
I have syslog-ng installed, and I can't figure out how to tell if "daemon.*" is being sent somewhere or not. "daemon" doesn't appear anywhere in/etc/syslog-ng/syslog-ng.confWell, you need to consult the man page for the configuration file then don't you.
Sorry for sounding lazy there. I've never used syslog-ng before, and I read the man page for a while without figuring how the old style of names/categories mapped to the new system. I'll continue to fiddle around and see if I can find out what's going on.
Thanks for helping me get started with the debugging output.
Quoted 8 lines
Quoted 5 lines
My autofs map looks like this:
% ypcat -k auto.master | grep /net
/net -hosts -intr,nosuid,retrans=10,retry=3,nobrowseBut I'm not sure that helps.
Is automount crashing, ie. does it show up in a ps listing after you see it hang?
Quoted 7 lines
Nope. The ps command doesn't show any change in the status of the automount process.
In fact, when I attach strace to the automount process I don't see any sign of activity when I do an "ls" on the unhealthy path.
It sounds like I would need to look at kernel tracing/debugging if I want to pursue this further.
strace output is often not very useful.
If you think there is some sort of deadlock going on get a sysreq-t dump to syslog. We still haven't seen a debug log?
Quoted 152 lines
Quoted 151 lines
Quoted 149 lines
--chris
Quoted 138 lines
For me, when syslog was in use and now with rsyslog, I just add a new line to send all the facility daemon output to a log file.
I add the line:
daemon.*
/var/log/debugto the configuration and then
touch
/var/log/debugand finally restart logging service.
You probably should post your autofs maps as well.
Quoted 122 lines
But when I grep
/var/log/messagesI see:% grep automount messages
Jun 29 02:41:14 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 02:41:14 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 03:17:45 carabas automount[11786]: lookup_mount: lookup(yp): key "opt" not found in map Jun 29 03:24:27 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 03:24:27 carabas automount[11786]: lookup_mount: lookup(yp): key "Codemgr_wsdata" not found in map Jun 29 09:04:46 carabas automount[11786]: Debug logging set for /net Jun 29 09:09:22 carabas automount[11786]: get_pkt: message pending on control fifo. Jun 29 09:09:22 carabas automount[11786]: Basic logging set for /netI didn't get the
get_pkgmessage until I turned off debug logging.While logging was set to debug, I tried the hanging ls command again, and I also tried listing a different file system that was actually remote. I didn't see any log messages for those operations.
Notes:
1.
/etc/host.confsays only:
order hosts, bind
multi on2. No mount.nfs processes are running when the ls command hangs on the local mount.
Quoted 93 lines · Ian Kent
Quoted 19 lines · Chris Quenelle
Thanks for your time guys!
It turns out it's not a general bug. I was just applying a pessimistic interpretation without checking.
After more testing, it turns out it's just problem on one machine, for two mount points under one host alias.
I'm interested in knowing more about how I could
debug the situation without just rebooting and
hoping it doesn't happen again.Here's my scenario:
I have three exported filesystems.
I can use three host aliases to refer to my own machine. (carabas, carabas.sfbay, carabas.sfbay.sun.com)Quoted 7 lines
/etc/hostssays:
[IP addr] carabas.sfbay.sun.com carabasAll filesystems work under all /net/foo aliases
except two of them hang when accessed via one
of the host aliases.Are you sure the address lookup is returning what you think it does? What's in
/etc/host.conf?Quoted 23 lines
% showmount -e localhost
Export list for localhost:
/export/home3 *
/export/home2 *
/export/home1 *% df -kl
Filesystem 1K-blocks Used Available Use% Mounted on/dev/sda269575344 3408900 66166444 5% /
udev 8222568 148 8222420 1% /dev/dev/sdb1 71671728 26507132 45164596 37% /export/home1 /dev/sdc1 71671728 8545196 63126532 12% /export/home2 /dev/sdd1 71671728 5375672 66296056 8% /export/home3 /export/home1 71671728 26507128 45164600 37% /net/carabas/export/home1 /export/home2 71671728 8545196 63126532 12% /net/carabas.sfbay/export/home2 /export/home1 71671728 26507128 45164600 37% /net/carabas.sfbay/export/home1 /export/home3 71671728 5375672 66296056 8% /net/carabas.sfbay/export/home3 /export/home1 71671728 26507128 45164600 37% /net/carabas.sfbay.sun.com/export/home1 /export/home2 71671728 8545196 63126532 12% /net/carabas.sfbay.sun.com/export/home2 /export/home3 71671728 5375672 66296056 8% /net/carabas.sfbay.sun.com/export/home3Quoted 7 lines
% strace ls /net/carabas/export/home2
...
...mmap(NULL, 217016, PROT_READ, MAP_SHARED, 4, 0)=0x2b2388370000
close(4) = 0
close(3) = 0
open("/net/carabas/export/home2", O_RDONLY|O_NONBLOCK|O_DIRECTORY<unfinished ...>What does a ps listing give you when it is blocked? Any mount.nfs processes?
That's sign that automount thinks that the mount isn't local and is probably trying to mount from a possibly non-existent ip address.
Quoted 5 lines
I hit ctrl-C when the ls command hangs. Above is the tail of the strace output. It doesn't tell you much, the interesting stuff happens in the automounter.
Is there a error log file for the automounter that is enabled by default? Is there an easy recipe for restarting the automounter with debugging output on?
You need to ensure that debug logging is being recorded in syslog. Ensure you are sending daemon.* somewhere.
Then
automount -l debug /net
will start the debug logging and
automount -l err /net
will set it back to no logging.
That is depending on the version of autofs you are using.
If you don't see this option in the automount(8) man page you don't have this option and all you can do is enable debug logging in the configuration.
Ian
Quoted 4 lines · Ian Kent
strace output is often not very useful.
If you think there is some sort of deadlock going on get a sysreq-t dump to syslog. We still haven't seen a debug log?
I've had reports that my emails are being delayed when they go out to the list. If anyone is following along and you'd like me to add you to my cc:
lines so you get the email directly, let me know, and I'll do that.
I'm getting close to my limits of what this problem is worth to me. I suspect the two broken paths will get unwedged if I reboot the system. But I'd love to know how to prevent it from happening again.
I saw these lines in /var/log/messages:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Jun 29 09:04:46 carabas automount[11786]: Debug logging set for /net Jun 29 09:09:22 carabas automount[11786]: get_pkt: message pending on control fifo. Jun 29 09:09:22 carabas automount[11786]: Basic logging set for /net
Does that mean that all debugging output from automount should be going to that file? Or could the debug output still be going someplace else (or into /dev/null?) In between the first line of that log output and the last line, I provoked a correctly functioning automount of a local file system, and I also tried to access the "broken" path to the local filesystem.
So that in combination with strace/automount not giving any output when I access the broken path, makes me think the control path is not getting out of the kernel.
Can you point me to an explanation of what a "sysreq-t dump" is and how to get it? I don't have access to the console of this machine, hopefully it's something I can do from a root term window.
To summarize my problem, I have a test set of paths to access a local filesystem, 7 work and 2 don't.
/net/carabas/export/home1
/net/carabas/export/home2 <-- fails
/net/carabas/export/home3 <-- fails
/net/carabas.sfbay/export/home1
/net/carabas.sfbay/export/home2
/net/carabas.sfbay/export/home3
/net/carabas.sfbay.sun.com/export/home1
/net/carabas.sfbay.sun.com/export/home2
/net/carabas.sfbay.sun.com/export/home3
I don't see anythign suspicious in the output of:
showmount
df/etc/host.conf
strace automount
automount -l debug /net
--chris
Ian Kent
maintainerQuoted 10 lines · Chris Quenelle
Quoted 4 lines · Ian Kent
strace output is often not very useful.
If you think there is some sort of deadlock going on get a sysreq-t dump to syslog. We still haven't seen a debug log?
I've had reports that my emails are being delayed when they go out to the list. If anyone is following along and you'd like me to add you to my cc:
lines so you get the email directly, let me know, and I'll do that.
That's going to happen if you post to a subscribers only list without subscribing to it.
Quoted 2 lines
I'm getting close to my limits of what this problem is worth to me.
And yet you haven't really provided the information requested?
I don't remember but did we get the distribution and autofs version your using?
Quoted 9 lines
I suspect the two broken paths will get unwedged if I reboot the system. But I'd love to know how to prevent it from happening again.
I saw these lines in
/var/log/messages:Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Jun 29 09:04:46 carabas automount[11786]: Debug logging set for /net Jun 29 09:09:22 carabas automount[11786]: get_pkt: message pending on control fifo. Jun 29 09:09:22 carabas automount[11786]: Basic logging set for /net
Quoted 6 lines
Does that mean that all debugging output from automount should be going to that file? Or could the debug output still be going someplace else (or into
/dev/null?) In between the first line of that log output and the last line, I provoked a correctly functioning automount of a local file system, and I also tried to access the "broken" path to the local filesystem.
What file, I don't understand what you mean?
But you don't mention what you have done to tell syslog to actually send "all" facility daemon messages to the syslog.
Try having a look at Jeffs page http://people.redhat.com/jmoyer for a description debug logging setup.
Quoted 4 lines
So that in combination with strace/automount not giving any output when I access the broken path, makes me think the control path is not getting out of the kernel.
Maybe.
Quoted 4 lines
Can you point me to an explanation of what a "sysreq-t dump" is and how to get it? I don't have access to the console of this machine, hopefully it's something I can do from a root term window.
Wherever your distribution's has kernel documentation (or a package that contains the documentation) look at Documentation/sysrq.txt.
Often, you will find you can:
echo "t" > /proc/sysrq-trigger
to get a trace dump, which is what I'm asking for.
Quoted 26 lines
To summarize my problem, I have a test set of paths to access a local filesystem, 7 work and 2 don't.
/net/carabas/export/home1
/net/carabas/export/home2 <-- fails
/net/carabas/export/home3 <-- fails
/net/carabas.sfbay/export/home1
/net/carabas.sfbay/export/home2
/net/carabas.sfbay/export/home3
/net/carabas.sfbay.sun.com/export/home1
/net/carabas.sfbay.sun.com/export/home2
/net/carabas.sfbay.sun.com/export/home3I don't see anythign suspicious in the output of:
showmount
df/etc/host.conf
strace automount
automount -l debug /net--chris
I went through this thread and collected all the information in a problem description. I also included sysrq dump output before and during the problem. It's 300k. I can send it to the list in email if you prefer. For now it's available here:
http://quenelle.org/unix/wp-content/uploads/2010/07/linux-log.txt
Again, I want to thank you guys for your time. I've learned a lot.
From the dump output I can see that there is one additional "automount" thread when the problem is happening. I think the new one has the number 5603. But that number seems to be in the "father" column, not the "pid" column. I'm not sure what that means.
automount S 0000555555686e00 0 5603 1 4054 (NOTLB)ffff810366a07e88 0000000000000086 0000000005f5e100 000000000000000a
ffff810417dc62d8 ffff810417dc6080 ffff810001033700 001082fb301a703a
0000000000000653 0000000001037030
Call Trace: <ffffffff8014a06b>{enqueue_hrtimer+90} <ffffffff802ea159>{schedule_hrtimer+41}
<ffffffff8014a5af>{hrtimer_nanosleep+130} <ffffffff8014a6a5>{sys_nanosleep+76}
<ffffffff8010ae42>{system_call+126}Anyway, the full dumps are included in the log I pointed at above.
--chris
Quoted 90 lines · Ian Kent
Quoted 9 lines · Chris Quenelle
Quoted 4 lines · Ian Kent
strace output is often not very useful.
If you think there is some sort of deadlock going on get a sysreq-t dump to syslog. We still haven't seen a debug log?
I've had reports that my emails are being delayed when they go out to the list. If anyone is following along and you'd like me to add you to my cc:
lines so you get the email directly, let me know, and I'll do that.That's going to happen if you post to a subscribers only list without subscribing to it.
Quoted 1 line
I'm getting close to my limits of what this problem is worth to me.
And yet you haven't really provided the information requested?
I don't remember but did we get the distribution and autofs version your using?
Quoted 8 lines
I suspect the two broken paths will get unwedged if I reboot the system. But I'd love to know how to prevent it from happening again.
I saw these lines in
/var/log/messages:Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Jun 29 09:04:46 carabas automount[11786]: Debug logging set for /net Jun 29 09:09:22 carabas automount[11786]: get_pkt: message pending on control fifo. Jun 29 09:09:22 carabas automount[11786]: Basic logging set for /netQuoted 6 lines
Does that mean that all debugging output from automount should be going to that file? Or could the debug output still be going someplace else (or into
/dev/null?) In between the first line of that log output and the last line, I provoked a correctly functioning automount of a local file system, and I also tried to access the "broken" path to the local filesystem.What file, I don't understand what you mean?
But you don't mention what you have done to tell syslog to actually send "all" facility daemon messages to the syslog.
Try having a look at Jeffs page http://people.redhat.com/jmoyer for a description debug logging setup.
Quoted 3 lines
So that in combination with strace/automount not giving any output when I access the broken path, makes me think the control path is not getting out of the kernel.
Maybe.
Quoted 3 lines
Can you point me to an explanation of what a "sysreq-t dump" is and how to get it? I don't have access to the console of this machine, hopefully it's something I can do from a root term window.
Wherever your distribution's has kernel documentation (or a package that contains the documentation) look at
Documentation/sysrq.txt.Often, you will find you can:
echo "t" >
/proc/sysrq-triggerto get a trace dump, which is what I'm asking for.
Quoted 25 lines
To summarize my problem, I have a test set of paths to access a local filesystem, 7 work and 2 don't.
/net/carabas/export/home1
/net/carabas/export/home2 <-- fails
/net/carabas/export/home3 <-- fails
/net/carabas.sfbay/export/home1
/net/carabas.sfbay/export/home2
/net/carabas.sfbay/export/home3
/net/carabas.sfbay.sun.com/export/home1
/net/carabas.sfbay.sun.com/export/home2
/net/carabas.sfbay.sun.com/export/home3I don't see anythign suspicious in the output of:
showmount
df/etc/host.conf
strace automount
automount -l debug /net--chris
Ian Kent
maintainerQuoted 7 lines · Chris Quenelle
I went through this thread and collected all the information in a problem description. I also included sysrq dump output before and during the problem. It's 300k. I can send it to the list in email if you prefer. For now it's available here:
http://quenelle.org/unix/wp-content/uploads/2010/07/linux-log.txt
This doesn't look like a deadlock in the kernel.
We still need a full debug log, which would have been useful to relate to the srsrq-t dump.
You might be seeing a thread create synchronization problem. I've fixed some problems in that area since 5.0.2 (but then we don't know what patches the SuSE folks have applied). Information about that possibility can be obtained by getting a gdb backtrace of the main automount process. This isn't much use unless debug symbols are available. In Fedora we have debuginfo packages that correspond to each package. They can be installed along with the package so that gdb has access to the program symbols.
In any case once the debug symbols are available you can use:
gdb -p <automount pid> /usr/sbin/automount
gdb> thr a a bt
(assuming automount is actuall in /usr/sbin) and capture the output of this so we can see what the automount threads are doing, or not doing, as the case may be.
Quoted 115 lines
Again, I want to thank you guys for your time. I've learned a lot.
From the dump output I can see that there is one additional "automount" thread when the problem is happening. I think the new one has the number 5603. But that number seems to be in the "father" column, not the "pid" column. I'm not sure what that means.
automount S
0000555555686e000 5603 1 4054 (NOTLB)ffff810366a07e8800000000000000860000000005f5e100000000000000000affff810417dc62d8 ffff810417dc6080 ffff810001033700 001082fb301a703a 0000000000000653 0000000001037030 Call Trace: <ffffffff8014a06b>{enqueue_hrtimer+90} <ffffffff802ea159>{schedule_hrtimer+41} <ffffffff8014a5af>{hrtimer_nanosleep+130} <ffffffff8014a6a5>{sys_nanosleep+76} <ffffffff8010ae42>{system_call+126}Anyway, the full dumps are included in the log I pointed at above.
--chris
Quoted 90 lines · Ian Kent
Quoted 9 lines · Chris Quenelle
Quoted 4 lines · Ian Kent
strace output is often not very useful.
If you think there is some sort of deadlock going on get a sysreq-t dump to syslog. We still haven't seen a debug log?
I've had reports that my emails are being delayed when they go out to the list. If anyone is following along and you'd like me to add you to my cc:
lines so you get the email directly, let me know, and I'll do that.That's going to happen if you post to a subscribers only list without subscribing to it.
Quoted 1 line
I'm getting close to my limits of what this problem is worth to me.
And yet you haven't really provided the information requested?
I don't remember but did we get the distribution and autofs version your using?
Quoted 8 lines
I suspect the two broken paths will get unwedged if I reboot the system. But I'd love to know how to prevent it from happening again.
I saw these lines in
/var/log/messages:Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Jun 29 09:04:46 carabas automount[11786]: Debug logging set for /net Jun 29 09:09:22 carabas automount[11786]: get_pkt: message pending on control fifo. Jun 29 09:09:22 carabas automount[11786]: Basic logging set for /netQuoted 6 lines
Does that mean that all debugging output from automount should be going to that file? Or could the debug output still be going someplace else (or into
/dev/null?) In between the first line of that log output and the last line, I provoked a correctly functioning automount of a local file system, and I also tried to access the "broken" path to the local filesystem.What file, I don't understand what you mean?
But you don't mention what you have done to tell syslog to actually send "all" facility daemon messages to the syslog.
Try having a look at Jeffs page http://people.redhat.com/jmoyer for a description debug logging setup.
Quoted 3 lines
So that in combination with strace/automount not giving any output when I access the broken path, makes me think the control path is not getting out of the kernel.
Maybe.
Quoted 3 lines
Can you point me to an explanation of what a "sysreq-t dump" is and how to get it? I don't have access to the console of this machine, hopefully it's something I can do from a root term window.
Wherever your distribution's has kernel documentation (or a package that contains the documentation) look at
Documentation/sysrq.txt.Often, you will find you can:
echo "t" >
/proc/sysrq-triggerto get a trace dump, which is what I'm asking for.
Quoted 25 lines
To summarize my problem, I have a test set of paths to access a local filesystem, 7 work and 2 don't.
/net/carabas/export/home1
/net/carabas/export/home2 <-- fails
/net/carabas/export/home3 <-- fails
/net/carabas.sfbay/export/home1
/net/carabas.sfbay/export/home2
/net/carabas.sfbay/export/home3
/net/carabas.sfbay.sun.com/export/home1
/net/carabas.sfbay.sun.com/export/home2
/net/carabas.sfbay.sun.com/export/home3I don't see anythign suspicious in the output of:
showmount
df/etc/host.conf
strace automount
automount -l debug /net--chris
A message that is not in the archive (sent off-list or not yet received): [email protected]
Ian Kent
maintainerQuoted 12 lines · Chris Quenelle
Quoted 11 lines · Ian Kent
Quoted 7 lines · Chris Quenelle
I went through this thread and collected all the information in a problem description. I also included sysrq dump output before and during the problem. It's 300k. I can send it to the list in email if you prefer. For now it's available here:
http://quenelle.org/unix/wp-content/uploads/2010/07/linux-log.txt
This doesn't look like a deadlock in the kernel.
Quoted 7 lines
Quoted 2 lines
We still need a full debug log, which would have been useful to relate to the srsrq-t dump.
There is no debug output when I access the problematic path. I verified that I'm actually getting all the debug output by accessing an unmounted /net location on another host. This surprised me.
I understand that may be the case but a full debug log is usually the staring point for automount debugging. A full debug log means, a debug log from the start of running automount (with the syslog facility daemon.* being recorded) until the problem occurs, with autofs in a clean state at the start. Most people don't get the "in a clean state at the start" so don't worry too much about that, but if we can't see anything in the log then I start talking about the state of autofs when it was started and move on from there.
The whole point is that, often times a problem is seen that happened quite some time before, and sometimes there are messages in the log that help focus efforts, which leads to a solution.
But even more importantly, with such an old version, we might have seen the problem before and the problem signature in the log might just "turn on that light". Of course, we may have something different or a previous problem with slightly different symptoms.
This is so important I'll say it again, a full debug log is pretty much always the starting point in trying to resolve autofs problems.
Quoted 2 lines
I'm not really in a position to get debug symbols right now.
That's a bummer.
Quoted 5 lines
You've done two important things for me so far.
1. You've showed me there is a place I can get help with my autofs problems.
Sure. But your distribution autofs maintainer should be able to go through the debugging exercise with you and bring his view to the list. That is actually important because the maintainer should be keeping an eye on what is happening and what new patches are appearing on kernel.org and be aware of the patches they have applied in their distribution version. This last point is the most important as I have no idea what the SuSE folks have applied to the package and I really shouldn't have to go through the exercise of finding out, although I have done so from time to time.
Oh ... I can't resist ... a shameless plug.
If your a heavy autofs user the best distribution to use is RedHat Enterpise Linux because I am up with what is happening with autofs (obviously).
Quoted 4 lines
2. You've helped me come up to speed on Linux/automount debugging so that I can do an initial evaluation for any future problems I run into.
I'm sure you will pick up a bit more too as time passes.
Quoted 5 lines
These things are very useful to me, even if I didn't get to root of my problem. When I get more time I will look for debug symbol packages for SUSE, and/or I'll try building automount from source with debugging enabled.
Yes, it is a bit hard, since a lot has happened since 5.2.
Quoted 135 lines
--chris
Quoted 128 lines
You might be seeing a thread create synchronization problem. I've fixed some problems in that area since 5.0.2 (but then we don't know what patches the SuSE folks have applied). Information about that possibility can be obtained by getting a gdb backtrace of the main automount process. This isn't much use unless debug symbols are available. In Fedora we have debuginfo packages that correspond to each package. They can be installed along with the package so that gdb has access to the program symbols.
In any case once the debug symbols are available you can use:
gdb -p <automount pid>
/usr/sbin/automount
gdb> thr a a bt(assuming automount is actuall in
/usr/sbin) and capture the output of this so we can see what the automount threads are doing, or not doing, as the case may be.Quoted 107 lines
Again, I want to thank you guys for your time. I've learned a lot.
From the dump output I can see that there is one additional "automount" thread when the problem is happening. I think the new one has the number 5603. But that number seems to be in the "father" column, not the "pid" column. I'm not sure what that means.
automount S
0000555555686e000 5603 1 4054 (NOTLB)ffff810366a07e8800000000000000860000000005f5e100000000000000000affff810417dc62d8 ffff810417dc6080 ffff810001033700 001082fb301a703a 0000000000000653 0000000001037030 Call Trace: <ffffffff8014a06b>{enqueue_hrtimer+90} <ffffffff802ea159>{schedule_hrtimer+41} <ffffffff8014a5af>{hrtimer_nanosleep+130} <ffffffff8014a6a5>{sys_nanosleep+76} <ffffffff8010ae42>{system_call+126}Anyway, the full dumps are included in the log I pointed at above.
--chris
Quoted 84 lines · Ian Kent
Quoted 9 lines · Chris Quenelle
Quoted 4 lines · Ian Kent
strace output is often not very useful.
If you think there is some sort of deadlock going on get a sysreq-t dump to syslog. We still haven't seen a debug log?
I've had reports that my emails are being delayed when they go out to the list. If anyone is following along and you'd like me to add you to my cc:
lines so you get the email directly, let me know, and I'll do that.That's going to happen if you post to a subscribers only list without subscribing to it.
Quoted 1 line
I'm getting close to my limits of what this problem is worth to me.
And yet you haven't really provided the information requested?
I don't remember but did we get the distribution and autofs version your using?
Quoted 8 lines
I suspect the two broken paths will get unwedged if I reboot the system. But I'd love to know how to prevent it from happening again.
I saw these lines in
/var/log/messages:Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Jun 29 09:04:46 carabas automount[11786]: Debug logging set for /net Jun 29 09:09:22 carabas automount[11786]: get_pkt: message pending on control fifo. Jun 29 09:09:22 carabas automount[11786]: Basic logging set for /netQuoted 6 lines
Does that mean that all debugging output from automount should be going to that file? Or could the debug output still be going someplace else (or into
/dev/null?) In between the first line of that log output and the last line, I provoked a correctly functioning automount of a local file system, and I also tried to access the "broken" path to the local filesystem.What file, I don't understand what you mean?
But you don't mention what you have done to tell syslog to actually send "all" facility daemon messages to the syslog.
Try having a look at Jeffs page http://people.redhat.com/jmoyer for a description debug logging setup.
Quoted 3 lines
So that in combination with strace/automount not giving any output when I access the broken path, makes me think the control path is not getting out of the kernel.
Maybe.
Quoted 3 lines
Can you point me to an explanation of what a "sysreq-t dump" is and how to get it? I don't have access to the console of this machine, hopefully it's something I can do from a root term window.
Wherever your distribution's has kernel documentation (or a package that contains the documentation) look at
Documentation/sysrq.txt.Often, you will find you can:
echo "t" >
/proc/sysrq-triggerto get a trace dump, which is what I'm asking for.
Quoted 25 lines
To summarize my problem, I have a test set of paths to access a local filesystem, 7 work and 2 don't.
/net/carabas/export/home1
/net/carabas/export/home2 <-- fails
/net/carabas/export/home3 <-- fails
/net/carabas.sfbay/export/home1
/net/carabas.sfbay/export/home2
/net/carabas.sfbay/export/home3
/net/carabas.sfbay.sun.com/export/home1
/net/carabas.sfbay.sun.com/export/home2
/net/carabas.sfbay.sun.com/export/home3I don't see anythign suspicious in the output of:
showmount
df/etc/host.conf
strace automount
automount -l debug /net--chris
Quoted 32 lines · Ian Kent
Quoted 9 lines · Chris Quenelle
Quoted 8 lines · Ian Kent
Quoted 7 lines · Chris Quenelle
I went through this thread and collected all the information in a problem description. I also included sysrq dump output before and during the problem. It's 300k. I can send it to the list in email if you prefer. For now it's available here:
http://quenelle.org/unix/wp-content/uploads/2010/07/linux-log.txt
Quoted 8 lines
Quoted 4 lines
This doesn't look like a deadlock in the kernel.
We still need a full debug log, which would have been useful to relate to the srsrq-t dump.
There is no debug output when I access the problematic path. I verified that I'm actually getting all the debug output by accessing an unmounted /net location on another host. This surprised me.
I understand that may be the case but a full debug log is usually the staring point for automount debugging. A full debug log means, a debug log from the start of running automount (with the syslog facility daemon.* being recorded) until the problem occurs, with autofs in a clean state at the start. Most people don't get the "in a clean state at the start" so don't worry too much about that, but if we can't see anything in the log then I start talking about the state of autofs when it was started and move on from there.
The whole point is that, often times a problem is seen that happened quite some time before, and sometimes there are messages in the log that help focus efforts, which leads to a solution.
Quoted 7 lines
But even more importantly, with such an old version, we might have seen the problem before and the problem signature in the log might just "turn on that light". Of course, we may have something different or a previous problem with slightly different symptoms.
This is so important I'll say it again, a full debug log is pretty much always the starting point in trying to resolve autofs problems.
Point taken about the importance of the full debug log. If the problem turns out to be reproducible after rebooting the machine (which I haven't done yet) then I'll post a debug log from the time of boot up until the problem happens.
Quoted 10 lines
Quoted 2 lines
I'm not really in a position to get debug symbols right now.
That's a bummer.
Quoted 4 lines
You've done two important things for me so far.
1. You've showed me there is a place I can get help with my autofs problems.
Quoted 9 lines
Sure. But your distribution autofs maintainer should be able to go through the debugging exercise with you and bring his view to the list. That is actually important because the maintainer should be keeping an eye on what is happening and what new patches are appearing on kernel.org and be aware of the patches they have applied in their distribution version. This last point is the most important as I have no idea what the SuSE folks have applied to the package and I really shouldn't have to go through the exercise of finding out, although I have done so from time to time.
Okay.
Quoted 6 lines
Oh ... I can't resist ... a shameless plug.
If your a heavy autofs user the best distribution to use is RedHat Enterpise Linux because I am up with what is happening with autofs (obviously).
If I have an RHEL system and I want to set it up to automatically pull down the latest patches, is there a web page describing how I can do that? Or the same for SUSE for that matter?
I realize this isn't an automount-specific question. I'm using an ocean of poorly updated lab machines with different distros and versions of Linux. Any tips for keeping some or all of them updated with official patches would be great. Like I said, I know it's off-topic, but pointers to RTFM would make my life much easier.
Quoted 146 lines
Quoted 4 lines
2. You've helped me come up to speed on Linux/automount debugging so that I can do an initial evaluation for any future problems I run into.
I'm sure you will pick up a bit more too as time passes.
Quoted 4 lines
These things are very useful to me, even if I didn't get to root of my problem. When I get more time I will look for debug symbol packages for SUSE, and/or I'll try building automount from source with debugging enabled.
Yes, it is a bit hard, since a lot has happened since 5.2.
Quoted 130 lines
--chris
Quoted 125 lines
You might be seeing a thread create synchronization problem. I've fixed some problems in that area since 5.0.2 (but then we don't know what patches the SuSE folks have applied). Information about that possibility can be obtained by getting a gdb backtrace of the main automount process. This isn't much use unless debug symbols are available. In Fedora we have debuginfo packages that correspond to each package. They can be installed along with the package so that gdb has access to the program symbols.
In any case once the debug symbols are available you can use:
gdb -p <automount pid>
/usr/sbin/automount
gdb> thr a a bt(assuming automount is actuall in
/usr/sbin) and capture the output of this so we can see what the automount threads are doing, or not doing, as the case may be.Quoted 106 lines
Again, I want to thank you guys for your time. I've learned a lot.
From the dump output I can see that there is one additional "automount" thread when the problem is happening. I think the new one has the number 5603. But that number seems to be in the "father" column, not the "pid" column. I'm not sure what that means.
automount S
0000555555686e000 5603 1 4054 (NOTLB)ffff810366a07e8800000000000000860000000005f5e100000000000000000affff810417dc62d8 ffff810417dc6080 ffff810001033700 001082fb301a703a 0000000000000653 0000000001037030 Call Trace: <ffffffff8014a06b>{enqueue_hrtimer+90} <ffffffff802ea159>{schedule_hrtimer+41} <ffffffff8014a5af>{hrtimer_nanosleep+130} <ffffffff8014a6a5>{sys_nanosleep+76} <ffffffff8010ae42>{system_call+126}Anyway, the full dumps are included in the log I pointed at above.
--chris
Quoted 83 lines · Ian Kent
Quoted 9 lines · Chris Quenelle
Quoted 4 lines · Ian Kent
strace output is often not very useful.
If you think there is some sort of deadlock going on get a sysreq-t dump to syslog. We still haven't seen a debug log?
I've had reports that my emails are being delayed when they go out to the list. If anyone is following along and you'd like me to add you to my cc:
lines so you get the email directly, let me know, and I'll do that.That's going to happen if you post to a subscribers only list without subscribing to it.
Quoted 1 line
I'm getting close to my limits of what this problem is worth to me.
And yet you haven't really provided the information requested?
I don't remember but did we get the distribution and autofs version your using?
Quoted 8 lines
I suspect the two broken paths will get unwedged if I reboot the system. But I'd love to know how to prevent it from happening again.
I saw these lines in
/var/log/messages:Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Jun 29 09:04:46 carabas automount[11786]: Debug logging set for /net Jun 29 09:09:22 carabas automount[11786]: get_pkt: message pending on control fifo. Jun 29 09:09:22 carabas automount[11786]: Basic logging set for /netQuoted 6 lines
Does that mean that all debugging output from automount should be going to that file? Or could the debug output still be going someplace else (or into
/dev/null?) In between the first line of that log output and the last line, I provoked a correctly functioning automount of a local file system, and I also tried to access the "broken" path to the local filesystem.What file, I don't understand what you mean?
But you don't mention what you have done to tell syslog to actually send "all" facility daemon messages to the syslog.
Try having a look at Jeffs page http://people.redhat.com/jmoyer for a description debug logging setup.
Quoted 3 lines
So that in combination with strace/automount not giving any output when I access the broken path, makes me think the control path is not getting out of the kernel.
Maybe.
Quoted 3 lines
Can you point me to an explanation of what a "sysreq-t dump" is and how to get it? I don't have access to the console of this machine, hopefully it's something I can do from a root term window.
Wherever your distribution's has kernel documentation (or a package that contains the documentation) look at
Documentation/sysrq.txt.Often, you will find you can:
echo "t" >
/proc/sysrq-triggerto get a trace dump, which is what I'm asking for.
Quoted 25 lines
To summarize my problem, I have a test set of paths to access a local filesystem, 7 work and 2 don't.
/net/carabas/export/home1
/net/carabas/export/home2 <-- fails
/net/carabas/export/home3 <-- fails
/net/carabas.sfbay/export/home1
/net/carabas.sfbay/export/home2
/net/carabas.sfbay/export/home3
/net/carabas.sfbay.sun.com/export/home1
/net/carabas.sfbay.sun.com/export/home2
/net/carabas.sfbay.sun.com/export/home3I don't see anythign suspicious in the output of:
showmount
df/etc/host.conf
strace automount
automount -l debug /net--chris