A weird autofs problem
I'm trying to debug a very weird problem I've started seeing with autofs over the last couple of days. The machines I'm seeing this on are all running Fedora Core 5 i386, fully updated (autofs 4.1.4-33 and kernel-smp 2.6.20-1.2320.fc5). Yes, I'm in the process of rolling out F7, but that will still take at least a couple of weeks and in the meantime I'd like to get to the bottom of this if possible.
We use autofs to automount user home directories under /home. Some users are reporting that their home directories are just disappearing out from under them; attempts to write to them result in "permission denied". Only once have I seen a system in this state (users keep rebooting the machines) and I don't have a screen capture, but I recall that df showed the user directory as being "there" but showed dashes instead of numbers for the capacity figures. Unfortunately I don't recall what showed up in the Filesystem column.
I can't think of anything which has changed on the affected systems. Since they're running FC5 nothing is getting updated. The automount maps are served via LDAP and as far as I can tell the queries aren't failing. The server doesn't report any problems. Nothing has changed recently on either the LDAP servers or the NFS servers.
So I'm at a complete loss here. If I can access a system while the problem is manifesting I'm sure I can find some more useful information but until then I was hoping that either someone might have seen something liks this and had an idea of what might be causing it or someone might have some idea of how I might go about debugging this. I have debugging turned on but haven't seen anything useful yet.
- J<
Ian Kent
maintainerQuoted 7 lines · Jason L Tibbitts III
I'm trying to debug a very weird problem I've started seeing with autofs over the last couple of days. The machines I'm seeing this on are all running Fedora Core 5 i386, fully updated (autofs 4.1.4-33 and kernel-smp 2.6.20-1.2320.fc5). Yes, I'm in the process of rolling out F7, but that will still take at least a couple of weeks and in the meantime I'd like to get to the bottom of this if possible.
Quoted 8 lines
We use autofs to automount user home directories under /home. Some users are reporting that their home directories are just disappearing out from under them; attempts to write to them result in "permission denied". Only once have I seen a system in this state (users keep rebooting the machines) and I don't have a screen capture, but I recall that df showed the user directory as being "there" but showed dashes instead of numbers for the capacity figures. Unfortunately I don't recall what showed up in the Filesystem column.
That NFS problem was fixed in 2.6.18 I think.
Quoted 6 lines
I can't think of anything which has changed on the affected systems. Since they're running FC5 nothing is getting updated. The automount maps are served via LDAP and as far as I can tell the queries aren't failing. The server doesn't report any problems. Nothing has changed recently on either the LDAP servers or the NFS servers.
Something has changed?
Quoted 8 lines
So I'm at a complete loss here. If I can access a system while the problem is manifesting I'm sure I can find some more useful information but until then I was hoping that either someone might have seen something liks this and had an idea of what might be causing it or someone might have some idea of how I might go about debugging this. I have debugging turned on but haven't seen anything useful yet.
We'll need you to get a first hand look at a broken system and have a look around and see what really is wrong. And you know that a debug log would be good too.
Ian
Quoted 17 lines · Ian Kent
Quoted 7 lines · Jason L Tibbitts III
I'm trying to debug a very weird problem I've started seeing with autofs over the last couple of days. The machines I'm seeing this on are all running Fedora Core 5 i386, fully updated (autofs 4.1.4-33 and kernel-smp 2.6.20-1.2320.fc5). Yes, I'm in the process of rolling out F7, but that will still take at least a couple of weeks and in the meantime I'd like to get to the bottom of this if possible.
Quoted 8 lines
We use autofs to automount user home directories under /home. Some users are reporting that their home directories are just disappearing out from under them; attempts to write to them result in "permission denied". Only once have I seen a system in this state (users keep rebooting the machines) and I don't have a screen capture, but I recall that df showed the user directory as being "there" but showed dashes instead of numbers for the capacity figures. Unfortunately I don't recall what showed up in the Filesystem column.
Quoted 2 lines
That NFS problem was fixed in 2.6.18 I think.
when i read this, nfs was the first thing that popped into my mind also. second would be a network outage. i have experienced the exact symptoms, but on static nfs mounts, after i have brought down the server for extended periods, or the net is out for long periods, then a client will do this. i simply recover by umounting then doing a mount -a.
hth
Quoted 27 lines
Quoted 5 lines
I can't think of anything which has changed on the affected systems. Since they're running FC5 nothing is getting updated. The automount maps are served via LDAP and as far as I can tell the queries aren't failing. The server doesn't report any problems. Nothing has changed recently on either the LDAP servers or the NFS servers.
Something has changed?
Quoted 7 lines
So I'm at a complete loss here. If I can access a system while the problem is manifesting I'm sure I can find some more useful information but until then I was hoping that either someone might have seen something liks this and had an idea of what might be causing it or someone might have some idea of how I might go about debugging this. I have debugging turned on but haven't seen anything useful yet.
We'll need you to get a first hand look at a broken system and have a look around and see what really is wrong. And you know that a debug log would be good too.
Ian
_______________________________________________
autofs mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/autofs
Signature · 3 lines
Jim Summers School of Computer Science-University of Oklahoma -------------------------------------------------
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"IK" == Ian Kent <[email protected]> writes:
IK> That NFS problem was fixed in 2.6.18 I think.
Well, I'm at 2.6.20, so....
IK> Something has changed?
I honestly can't think of anything.
IK> We'll need you to get a first hand look at a broken system and IK> have a look around and see what really is wrong. And you know that IK> a debug log would be good too.
OK, here's a machine that's gone bad. df shows:
nas01:/export/users-00/boyarkin
- - - - /home/boyarkin
Attempting to unmount the directory results in "device is busy" as the user is currently logged in and has files open on the filesystem.
Unfortunately because the user has not logged out of this machine, I haven't been able to restart the automounter so debugging is not enabled.
Simply running "umount -l /home/boyarkin" as roott causes the mount to appear properly.
I've kicked the user out of the machine and restarted the automounter with debugging on. I'll continue to watch this machine and see if the problem reoccurs.
- J<
Ian Kent
maintainerQuoted 14 lines · Jason L Tibbitts III
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"IK" == Ian Kent <[email protected]> writes:
IK> That NFS problem was fixed in 2.6.18 I think.
Well, I'm at 2.6.20, so....
IK> Something has changed?
I honestly can't think of anything.
IK> We'll need you to get a first hand look at a broken system and IK> have a look around and see what really is wrong. And you know that IK> a debug log would be good too.
Quoted 7 lines
OK, here's a machine that's gone bad. df shows:
nas01:/export/users-00/boyarkin
- - - -/home/boyarkinAttempting to unmount the directory results in "device is busy" as the user is currently logged in and has files open on the filesystem.
What about the output of an ls as you referred bogus entries being listed. Is that actually the case? Is there anything in the log that might give us a clue as to what's going on?
Quoted 7 lines
Unfortunately because the user has not logged out of this machine, I haven't been able to restart the automounter so debugging is not enabled.
Simply running "umount -l
/home/boyarkin" as roott causes the mount to appear properly.
Yes, that would do the trick.
Quoted 6 lines
I've kicked the user out of the machine and restarted the automounter with debugging on. I'll continue to watch this machine and see if the problem reoccurs.
- J<
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"IK" == Ian Kent <[email protected]> writes:
IK> What about the output of an ls as you referred bogus entries being IK> listed.
No, the filesystem is completely inaccessible at this point.
However, I have another hint which seems to let autofs off the hook:
These happened on the NFS server around the time of the first report of problems:
Aug 15 11:34:36 nas01 nscd: 14741 Error getting context of nscd Aug 15 11:34:47 nas01 last message repeated 18 times Aug 15 11:34:47 nas01 nscd: nscd shutdown succeeded Aug 15 11:34:47 nas01 nscd: 21046 Access Vector Cache (AVC) started Aug 15 11:34:47 nas01 nscd: nscd startup succeeded
(actually the first line has been appearing in the logs for ages; I assume it's just more selinux stuff which I still don't understand.)
And then this morning:
Aug 17 10:16:12 nas01 kernel: nscd[14745]: segfault at 0000002afed9e933 rip 000000552aab7cf2 rsp 0000000040c00750 error 4 Aug 17 10:16:27 nas01 kernel: nscd[21050]: segfault at 0000002b99607280 rip 000000552aab7d08 rsp 0000000040c00750 error 4
so nscd definitely screwed up in some manner. I restarted it:
Aug 17 10:30:27 nas01 nscd: 22346 Access Vector Cache (AVC) started Aug 17 10:30:27 nas01 nscd: nscd startup succeeded Aug 17 10:30:27 nas01 nscd: 22346 invalid persistent database file "/var/db/nscd/passwd": verification failed
and so far no reports of problems.
Sorry to try to lay any blame on autofs. Just another tribute to how impenetrable NFS can be.
- J<