ENOENT on first reference to an automounted file
I have what looks like an automount race condition, and am very puzzled. Any suggestions would be appreciated.
The first time I reference an automounted file, it is not there (ENOENT). On the second and later try, the file is there. For instance:
$ cat /net/fileserver/fs/somefile
cat: /net/fileserver/fs/somefile: No such file or directory
$ cat /net/fileserver/fs/somefile
Contents of somefile.I watched the log on fileserver, and the automount request is logged seemingly immediately after the first "cat" prints its error.
This causes havoc with our applications, which expect files to be there the first time they look for them.
I can repeat the problem after umounting the fileystem.
I see this problem on a CentOS 4.x system running their standard autofs-4.1.3-199.3. I do NOT see it on CentOS 5.x, using autofs-5.0.1-0.rc2.43.0.2. Instead I see a slight pause before "cat" prints the contents of the file, presumably as the automount completes. Both the CentOS4 and CentOS5 systems are completely up-to-date.
I also only see this problem with our Linux NFS servers (FC5 and FC6), but not with a non-Fedora NAS server we have.
So I am not sure this is an automount problem, per se. Perhaps it's some kind of NFS version problem?
The automount options include --ghost. At first I thought it might be due to --ghost, because the very first time I reference the file, say after a reboot or restarting autofs, I don't get an ENOENT. The first time, the mountpoint dir does not yet exist. But removing --ghost from the automount options does not seem to fix it.
Gory details about the automount maps are below.
Thanks for any help,
Dan Halbert
---------------
More details:
Our automount maps are stored in ldap. The entry in auto.master for fileserver (for cn=/net/fileserver) is:
ldap:ou=auto.fileserver,ou=autofs,dc=example,dc=com --timeout=86400
--ghost -o rw,hard,async,noatime,intr,retrans=4,timeo=100,rsize=8192,wsize=8192
The auto.fileserver is (for cn=*):
fileserver.example.com:/export/&
We are not using the fancy executable /net maps that come with these systems.
Ian Kent
maintainerQuoted 28 lines · Dan Halbert
I have what looks like an automount race condition, and am very puzzled. Any suggestions would be appreciated.
The first time I reference an automounted file, it is not there (ENOENT). On the second and later try, the file is there. For instance:
$ cat /net/fileserver/fs/somefile cat: /net/fileserver/fs/somefile: No such file or directory $ cat /net/fileserver/fs/somefile Contents of somefile.I watched the log on fileserver, and the automount request is logged seemingly immediately after the first "cat" prints its error.
This causes havoc with our applications, which expect files to be there the first time they look for them.
I can repeat the problem after umounting the fileystem.
I see this problem on a CentOS 4.x system running their standard autofs-4.1.3-199.3. I do NOT see it on CentOS 5.x, using autofs-5.0.1-0.rc2.43.0.2. Instead I see a slight pause before "cat" prints the contents of the file, presumably as the automount completes. Both the CentOS4 and CentOS5 systems are completely up-to-date.
I also only see this problem with our Linux NFS servers (FC5 and FC6), but not with a non-Fedora NAS server we have.
Quoted 8 lines
So I am not sure this is an automount problem, per se. Perhaps it's some kind of NFS version problem?
The automount options include --ghost. At first I thought it might be due to --ghost, because the very first time I reference the file, say after a reboot or restarting autofs, I don't get an ENOENT. The first time, the mountpoint dir does not yet exist. But removing --ghost from the automount options does not seem to fix it.
We've seen this from time to time for various reasons but to be honest I have trouble remembering so we'll need to check through a debug log.
Jeff may recall this?
Also, you don't mention the kernel versions?
Quoted 12 lines
Gory details about the automount maps are below.
Thanks for any help,
Dan Halbert---------------
More details:Our automount maps are stored in ldap. The entry in auto.master for fileserver (for cn=/net/fileserver) is:
Quoted 8 lines
ldap:ou=auto.fileserver,ou=autofs,dc=example,dc=com --timeout=86400--ghost -o rw,hard,async,noatime,intr,retrans=4,timeo=100,rsize=8192,wsize=8192
The auto.fileserver is (for cn=*):
fileserver.example.com:/export/&
We really must have a debug log, include everything and give some indication of when the problem occurred. See
http://people.redhat.com/jmoyer for info.
Ian
Quoted 38 lines · Ian Kent
Quoted 28 lines · Dan Halbert
I have what looks like an automount race condition, and am very puzzled. Any suggestions would be appreciated.
The first time I reference an automounted file, it is not there (ENOENT). On the second and later try, the file is there. For instance:
$ cat /net/fileserver/fs/somefile cat: /net/fileserver/fs/somefile: No such file or directory $ cat /net/fileserver/fs/somefile Contents of somefile.I watched the log on fileserver, and the automount request is logged seemingly immediately after the first "cat" prints its error.
This causes havoc with our applications, which expect files to be there the first time they look for them.
I can repeat the problem after umounting the fileystem.
I see this problem on a CentOS 4.x system running their standard autofs-4.1.3-199.3. I do NOT see it on CentOS 5.x, using autofs-5.0.1-0.rc2.43.0.2. Instead I see a slight pause before "cat" prints the contents of the file, presumably as the automount completes. Both the CentOS4 and CentOS5 systems are completely up-to-date.
I also only see this problem with our Linux NFS servers (FC5 and FC6), but not with a non-Fedora NAS server we have.
Quoted 8 lines
So I am not sure this is an automount problem, per se. Perhaps it's some kind of NFS version problem?
The automount options include --ghost. At first I thought it might be due to --ghost, because the very first time I reference the file, say after a reboot or restarting autofs, I don't get an ENOENT. The first time, the mountpoint dir does not yet exist. But removing --ghost from the automount options does not seem to fix it.
Quoted 4 lines
We've seen this from time to time for various reasons but to be honest I have trouble remembering so we'll need to check through a debug log.
Jeff may recall this?
I think that the last time we looked at this, the problem was that there was a replicated server entry, and the first picked entry failed to mount. Then, the second succeeded, but we returned the wrong dentry from lookup. This resulted in a reported failure, even though the mount was successful.
I'm not convinced this is the same problem. I'll try to reproduce it.
Cheers,
Jeff
Quoted 10 lines · Dan Halbert
Quoted 8 lines · Ian Kent
Quoted 4 lines · Dan Halbert
I have what looks like an automount race condition, and am very puzzled...
The first time I reference an automounted file, it is not there (ENOENT). On the second and later try, the file is there...
We really must have a debug log, include everything and give some indication of when the problem occurred....
Quoted 8 lines
Details you requested follow.
autofs-4.1.3-199.3(I also reproduced the problem with autofs-4.1.3-214, the latest version
at jmoyer's webpage.)kernel:
2.6.9-55.0.9.ELsmp x86_64
I can't reproduce the problem. Would you be willing to enable debugging in the kernel module. This will generate oodles of output. Alternatively, I could try to come up with some pointed kprobes to get the information we most likely need. The first option will definitely be quicker.
Cheers,
Jeff
Quoted 3 lines · Jeff Moyer
I can't reproduce the problem. Would you be willing to enable debugging in the kernel module.
Sure, happy to do this, though you'll need to tell me how to turn that on. I'm testing on a machine I have complete control over.
Dan
Quoted 6 lines · Dan Halbert
Quoted 3 lines · Jeff Moyer
I can't reproduce the problem. Would you be willing to enable debugging in the kernel module.
Sure, happy to do this, though you'll need to tell me how to turn that on. I'm testing on a machine I have complete control over.
Apply the following patch and rebuild the kernel module.
Cheers,
Jeff
fs/autofs4/autofs_i.h renamed from fs/autofs4/autofs_i.h.orig +1 -1
@@ -30,7 +30,7 @@#include <asm/current.h> #include <asm/uaccess.h> -/* #define DEBUG */ +#define DEBUG #ifdef DEBUG #define DPRINTK(fmt,args...) do { printk(KERN_DEBUG "pid %d: %s: " fmt "\n" , current->pid , __FUNCTION__ , ##args); } while(0)
Quoted 6 lines · Dan Halbert
Quoted 3 lines · Jeff Moyer
I can't reproduce the problem. Would you be willing to enable debugging in the kernel module.
Sure, happy to do this, though you'll need to tell me how to turn that on. I'm testing on a machine I have complete control over.
I was able to reproduce it. It turns out that I had a kernel installed that had a fix for the following bug:
Bugzilla Bug 248126: autofs problem with symbolic links
When I moved to the exact kernel you were running, I hit the problem. So, it's a known issue, and it had better be addressed in the next update (kernel 2.6.9-61.EL).
Cheers,
Jeff
Fedora kernel question (was Re: ENOENT on first reference to an automounted file)
Quoted 8 lines · Jeff Moyer
I was able to reproduce it. It turns out that I had a kernel installed that had a fix for the following bug:
Bugzilla Bug 248126: autofs problem with symbolic linksWhen I moved to the exact kernel you were running, I hit the problem. So, it's a known issue, and it had better be addressed in the next update (kernel 2.6.9-61.EL).
Would you have any info on the status of these fixes into Fedora kernels ? Everything I read in bugzilla seems specific to RHEL.
Much Thanks,
Jimmy
Quoted 8 lines · Jeff Moyer
I was able to reproduce it. It turns out that I had a kernel installed that had a fix for the following bug:
Bugzilla Bug 248126: autofs problem with symbolic linksWhen I moved to the exact kernel you were running, I hit the problem. So, it's a known issue, and it had better be addressed in the next update (kernel 2.6.9-61.EL).
Thanks! You saved me from a module rebuild, which I have not done in quite a few years.
I had searched the existing bugs and seen 248126, but it did not seem to me that I could have had a simultaneous expire, since we have such long timeouts on the automounts (usually --timeout=86400). But I must have misunderstood what "expire" means in this case.
Also, from 248126 and the bug it references, 174821, it appeared that these patches were already incorporated into my kernel 2.6.9-55.0.9-smp (248126 comment #24 mentions 55.0.7, for instance). But apparently not!
Is this bug non-existent in the latest updated RHEL5.0? I am trying to think of a workaround until 2.6.9-61 comes out. We have a tried cron job to provoke the automount more often than its timeout, but I am not sure that would solve the problem.
Dan
Re: Fedora kernel question (was Re: ENOENT on first reference to an automounted file)
In reply to Jimmy DorffQuoted 5 lines · Jimmy Dorff
Quoted 4 lines · Jeff Moyer
I was able to reproduce it. It turns out that I had a kernel installed that had a fix for the following bug:
Bugzilla Bug 248126: autofs problem with symbolic links
Quoted 7 lines
Quoted 4 lines
When I moved to the exact kernel you were running, I hit the problem. So, it's a known issue, and it had better be addressed in the next update (kernel 2.6.9-61.EL).
Would you have any info on the status of these fixes into Fedora kernels ? Everything I read in bugzilla seems specific to RHEL.
Are you seeing this problem in Fedora? If so, just run a newer kernel. In general, Fedora gets kernel fixes much more quickly than RHEL (due to the constant rebasing that goes on in Fedora).
Cheers,
Jeff
Quoted 9 lines · Dan Halbert
Quoted 8 lines · Jeff Moyer
I was able to reproduce it. It turns out that I had a kernel installed that had a fix for the following bug:
Bugzilla Bug 248126: autofs problem with symbolic linksWhen I moved to the exact kernel you were running, I hit the problem. So, it's a known issue, and it had better be addressed in the next update (kernel 2.6.9-61.EL).
Quoted 7 lines
Thanks! You saved me from a module rebuild, which I have not done in quite a few years.
I had searched the existing bugs and seen 248126, but it did not seem to me that I could have had a simultaneous expire, since we have such long timeouts on the automounts (usually --timeout=86400). But I must have misunderstood what "expire" means in this case.
Yes, that's a bit puzzling.
Quoted 4 lines
Also, from 248126 and the bug it references, 174821, it appeared that these patches were already incorporated into my kernel 2.6.9-55.0.9-smp (248126 comment #24 mentions 55.0.7, for instance). But apparently not!
The patch was reverted in this revision. I'm not sure why.
Quoted 5 lines
Is this bug non-existent in the latest updated RHEL5.0? I am trying to think of a workaround until 2.6.9-61 comes out. We have a tried cron job to provoke the automount more often than its timeout, but I am not sure that would solve the problem.
I'd need to check but I believe the patch is present in the RHEL5 kernel (but I think there are a couple of corrections missing).
You could apply the patch(es) yourself.
Ian
Ian Kent
maintainerRe: Fedora kernel question (was Re: ENOENT on first reference to an automounted file)
In reply to Jimmy DorffQuoted 5 lines · Jimmy Dorff
Quoted 4 lines · Jeff Moyer
I was able to reproduce it. It turns out that I had a kernel installed that had a fix for the following bug:
Bugzilla Bug 248126: autofs problem with symbolic links
Quoted 7 lines
Quoted 4 lines
When I moved to the exact kernel you were running, I hit the problem. So, it's a known issue, and it had better be addressed in the next update (kernel 2.6.9-61.EL).
Would you have any info on the status of these fixes into Fedora kernels ? Everything I read in bugzilla seems specific to RHEL.
I think it was 2.6.21 that got the patch discussed here. The corrections I mentioned are included in another patch that is in 2.6.23, so obviously not in a Fedora release kernel.
Ian