Skip to contentSkip to conversation

ENOENT on first reference to an automounted file

DISCUSSION Dan Halbert 5 participants 12 messages Oct 5, 2007 – Oct 9, 2007 1 message not in the archive Kernel automounter

lore
Share & tools
lore
#0 lore raw

I have what looks like an automount race condition, and am very puzzled. Any suggestions would be appreciated.

The first time I reference an automounted file, it is not there (ENOENT). On the second and later try, the file is there. For instance:

     $ cat /net/fileserver/fs/somefile
     cat: /net/fileserver/fs/somefile: No such file or directory
     $ cat /net/fileserver/fs/somefile
     Contents of somefile.

I watched the log on fileserver, and the automount request is logged seemingly immediately after the first "cat" prints its error.

This causes havoc with our applications, which expect files to be there the first time they look for them.

I can repeat the problem after umounting the fileystem.

I see this problem on a CentOS 4.x system running their standard autofs-4.1.3-199.3. I do NOT see it on CentOS 5.x, using autofs-5.0.1-0.rc2.43.0.2. Instead I see a slight pause before "cat" prints the contents of the file, presumably as the automount completes. Both the CentOS4 and CentOS5 systems are completely up-to-date.

I also only see this problem with our Linux NFS servers (FC5 and FC6), but not with a non-Fedora NAS server we have.

So I am not sure this is an automount problem, per se. Perhaps it's some kind of NFS version problem?

The automount options include --ghost. At first I thought it might be due to --ghost, because the very first time I reference the file, say after a reboot or restarting autofs, I don't get an ENOENT. The first time, the mountpoint dir does not yet exist. But removing --ghost from the automount options does not seem to fix it.

Gory details about the automount maps are below.

Thanks for any help,
Dan Halbert

---------------
More details:

Our automount maps are stored in ldap. The entry in auto.master for fileserver (for cn=/net/fileserver) is:

     ldap:ou=auto.fileserver,ou=autofs,dc=example,dc=com --timeout=86400 

--ghost -o rw,hard,async,noatime,intr,retrans=4,timeo=100,rsize=8192,wsize=8192

The auto.fileserver is (for cn=*):

fileserver.example.com:/export/&

We are not using the fancy executable /net maps that come with these systems.

maintainer
#1 lore raw
Quoted 28 lines · Dan Halbert

I have what looks like an automount race condition, and am very puzzled. Any suggestions would be appreciated.

The first time I reference an automounted file, it is not there (ENOENT). On the second and later try, the file is there. For instance:

     $ cat /net/fileserver/fs/somefile
     cat: /net/fileserver/fs/somefile: No such file or directory
     $ cat /net/fileserver/fs/somefile
     Contents of somefile.

I watched the log on fileserver, and the automount request is logged seemingly immediately after the first "cat" prints its error.

This causes havoc with our applications, which expect files to be there the first time they look for them.

I can repeat the problem after umounting the fileystem.

I see this problem on a CentOS 4.x system running their standard autofs-4.1.3-199.3. I do NOT see it on CentOS 5.x, using autofs-5.0.1-0.rc2.43.0.2. Instead I see a slight pause before "cat" prints the contents of the file, presumably as the automount completes. Both the CentOS4 and CentOS5 systems are completely up-to-date.

I also only see this problem with our Linux NFS servers (FC5 and FC6), but not with a non-Fedora NAS server we have.

Quoted 8 lines

So I am not sure this is an automount problem, per se. Perhaps it's some kind of NFS version problem?

The automount options include --ghost. At first I thought it might be due to --ghost, because the very first time I reference the file, say after a reboot or restarting autofs, I don't get an ENOENT. The first time, the mountpoint dir does not yet exist. But removing --ghost from the automount options does not seem to fix it.

We've seen this from time to time for various reasons but to be honest I have trouble remembering so we'll need to check through a debug log.

Jeff may recall this?

Also, you don't mention the kernel versions?

Quoted 12 lines

Gory details about the automount maps are below.

Thanks for any help,
Dan Halbert

---------------
More details:

Our automount maps are stored in ldap. The entry in auto.master for fileserver (for cn=/net/fileserver) is:

Quoted 8 lines
     ldap:ou=auto.fileserver,ou=autofs,dc=example,dc=com --timeout=86400 

--ghost -o rw,hard,async,noatime,intr,retrans=4,timeo=100,rsize=8192,wsize=8192

The auto.fileserver is (for cn=*):

fileserver.example.com:/export/&

We really must have a debug log, include everything and give some indication of when the problem occurred. See
http://people.redhat.com/jmoyer for info.

Ian

#2 lore raw
Quoted 38 lines · Ian Kent
Quoted 28 lines · Dan Halbert

I have what looks like an automount race condition, and am very puzzled. Any suggestions would be appreciated.

The first time I reference an automounted file, it is not there (ENOENT). On the second and later try, the file is there. For instance:

     $ cat /net/fileserver/fs/somefile
     cat: /net/fileserver/fs/somefile: No such file or directory
     $ cat /net/fileserver/fs/somefile
     Contents of somefile.

I watched the log on fileserver, and the automount request is logged seemingly immediately after the first "cat" prints its error.

This causes havoc with our applications, which expect files to be there the first time they look for them.

I can repeat the problem after umounting the fileystem.

I see this problem on a CentOS 4.x system running their standard autofs-4.1.3-199.3. I do NOT see it on CentOS 5.x, using autofs-5.0.1-0.rc2.43.0.2. Instead I see a slight pause before "cat" prints the contents of the file, presumably as the automount completes. Both the CentOS4 and CentOS5 systems are completely up-to-date.

I also only see this problem with our Linux NFS servers (FC5 and FC6), but not with a non-Fedora NAS server we have.

Quoted 8 lines

So I am not sure this is an automount problem, per se. Perhaps it's some kind of NFS version problem?

The automount options include --ghost. At first I thought it might be due to --ghost, because the very first time I reference the file, say after a reboot or restarting autofs, I don't get an ENOENT. The first time, the mountpoint dir does not yet exist. But removing --ghost from the automount options does not seem to fix it.

Quoted 4 lines

We've seen this from time to time for various reasons but to be honest I have trouble remembering so we'll need to check through a debug log.

Jeff may recall this?

I think that the last time we looked at this, the problem was that there was a replicated server entry, and the first picked entry failed to mount. Then, the second succeeded, but we returned the wrong dentry from lookup. This resulted in a reported failure, even though the mount was successful.

I'm not convinced this is the same problem. I'll try to reproduce it.

Cheers,

Jeff

#4 lore raw
In reply to a message not in the archive
Quoted 10 lines · Dan Halbert
Quoted 8 lines · Ian Kent
Quoted 4 lines · Dan Halbert

I have what looks like an automount race condition, and am very puzzled...

The first time I reference an automounted file, it is not there (ENOENT). On the second and later try, the file is there...

We really must have a debug log, include everything and give some indication of when the problem occurred....

Quoted 8 lines

Details you requested follow.

	autofs-4.1.3-199.3

(I also reproduced the problem with autofs-4.1.3-214, the latest version
at jmoyer's webpage.)

kernel:

	2.6.9-55.0.9.ELsmp x86_64

I can't reproduce the problem. Would you be willing to enable debugging in the kernel module. This will generate oodles of output. Alternatively, I could try to come up with some pointed kprobes to get the information we most likely need. The first option will definitely be quicker.

Cheers,

Jeff

#5 lore raw
Quoted 3 lines · Jeff Moyer

I can't reproduce the problem. Would you be willing to enable debugging in the kernel module.

Sure, happy to do this, though you'll need to tell me how to turn that on. I'm testing on a machine I have complete control over.

Dan

#6 lore raw
Quoted 6 lines · Dan Halbert
Quoted 3 lines · Jeff Moyer

I can't reproduce the problem. Would you be willing to enable debugging in the kernel module.

Sure, happy to do this, though you'll need to tell me how to turn that on. I'm testing on a machine I have complete control over.

Apply the following patch and rebuild the kernel module.

Cheers,

Jeff

fs/autofs4/autofs_i.h renamed from fs/autofs4/autofs_i.h.orig +1 -1
@@ -30,7 +30,7 @@
 #include <asm/current.h>
 #include <asm/uaccess.h>
 
-/* #define DEBUG */
+#define DEBUG
 
 #ifdef DEBUG
 #define DPRINTK(fmt,args...) do { printk(KERN_DEBUG "pid %d: %s: " fmt "\n" , current->pid , __FUNCTION__ , ##args); } while(0)
#7 lore raw
In reply to Dan Halbert
Quoted 6 lines · Dan Halbert
Quoted 3 lines · Jeff Moyer

I can't reproduce the problem. Would you be willing to enable debugging in the kernel module.

Sure, happy to do this, though you'll need to tell me how to turn that on. I'm testing on a machine I have complete control over.

I was able to reproduce it. It turns out that I had a kernel installed that had a fix for the following bug:
Bugzilla Bug 248126: autofs problem with symbolic links

When I moved to the exact kernel you were running, I hit the problem. So, it's a known issue, and it had better be addressed in the next update (kernel 2.6.9-61.EL).

Cheers,

Jeff

#8 lore raw

Fedora kernel question (was Re: ENOENT on first reference to an automounted file)

Quoted 8 lines · Jeff Moyer

I was able to reproduce it. It turns out that I had a kernel installed that had a fix for the following bug:
Bugzilla Bug 248126: autofs problem with symbolic links

When I moved to the exact kernel you were running, I hit the problem. So, it's a known issue, and it had better be addressed in the next update (kernel 2.6.9-61.EL).

Would you have any info on the status of these fixes into Fedora kernels ? Everything I read in bugzilla seems specific to RHEL.

Much Thanks,
Jimmy

#11 lore raw
In reply to Jeff Moyer
Quoted 8 lines · Jeff Moyer

I was able to reproduce it. It turns out that I had a kernel installed that had a fix for the following bug:
Bugzilla Bug 248126: autofs problem with symbolic links

When I moved to the exact kernel you were running, I hit the problem. So, it's a known issue, and it had better be addressed in the next update (kernel 2.6.9-61.EL).

Thanks! You saved me from a module rebuild, which I have not done in quite a few years.

I had searched the existing bugs and seen 248126, but it did not seem to me that I could have had a simultaneous expire, since we have such long timeouts on the automounts (usually --timeout=86400). But I must have misunderstood what "expire" means in this case.

Also, from 248126 and the bug it references, 174821, it appeared that these patches were already incorporated into my kernel 2.6.9-55.0.9-smp (248126 comment #24 mentions 55.0.7, for instance). But apparently not!

Is this bug non-existent in the latest updated RHEL5.0? I am trying to think of a workaround until 2.6.9-61 comes out. We have a tried cron job to provoke the automount more often than its timeout, but I am not sure that would solve the problem.

Dan

#9 lore raw

Re: Fedora kernel question (was Re: ENOENT on first reference to an automounted file)

In reply to Jimmy Dorff
Quoted 5 lines · Jimmy Dorff
Quoted 4 lines · Jeff Moyer

I was able to reproduce it. It turns out that I had a kernel installed that had a fix for the following bug:
Bugzilla Bug 248126: autofs problem with symbolic links

Quoted 7 lines
Quoted 4 lines

When I moved to the exact kernel you were running, I hit the problem. So, it's a known issue, and it had better be addressed in the next update (kernel 2.6.9-61.EL).

Would you have any info on the status of these fixes into Fedora kernels ? Everything I read in bugzilla seems specific to RHEL.

Are you seeing this problem in Fedora? If so, just run a newer kernel. In general, Fedora gets kernel fixes much more quickly than RHEL (due to the constant rebasing that goes on in Fedora).

Cheers,

Jeff

#12 lore raw
In reply to Dan Halbert
Quoted 9 lines · Dan Halbert
Quoted 8 lines · Jeff Moyer

I was able to reproduce it. It turns out that I had a kernel installed that had a fix for the following bug:
Bugzilla Bug 248126: autofs problem with symbolic links

When I moved to the exact kernel you were running, I hit the problem. So, it's a known issue, and it had better be addressed in the next update (kernel 2.6.9-61.EL).

Quoted 7 lines

Thanks! You saved me from a module rebuild, which I have not done in quite a few years.

I had searched the existing bugs and seen 248126, but it did not seem to me that I could have had a simultaneous expire, since we have such long timeouts on the automounts (usually --timeout=86400). But I must have misunderstood what "expire" means in this case.

Yes, that's a bit puzzling.

Quoted 4 lines

Also, from 248126 and the bug it references, 174821, it appeared that these patches were already incorporated into my kernel 2.6.9-55.0.9-smp (248126 comment #24 mentions 55.0.7, for instance). But apparently not!

The patch was reverted in this revision. I'm not sure why.

Quoted 5 lines

Is this bug non-existent in the latest updated RHEL5.0? I am trying to think of a workaround until 2.6.9-61 comes out. We have a tried cron job to provoke the automount more often than its timeout, but I am not sure that would solve the problem.

I'd need to check but I believe the patch is present in the RHEL5 kernel (but I think there are a couple of corrections missing).

You could apply the patch(es) yourself.

Ian

maintainer
#10 lore raw

Re: Fedora kernel question (was Re: ENOENT on first reference to an automounted file)

In reply to Jimmy Dorff
Quoted 5 lines · Jimmy Dorff
Quoted 4 lines · Jeff Moyer

I was able to reproduce it. It turns out that I had a kernel installed that had a fix for the following bug:
Bugzilla Bug 248126: autofs problem with symbolic links

Quoted 7 lines
Quoted 4 lines

When I moved to the exact kernel you were running, I hit the problem. So, it's a known issue, and it had better be addressed in the next update (kernel 2.6.9-61.EL).

Would you have any info on the status of these fixes into Fedora kernels ? Everything I read in bugzilla seems specific to RHEL.

I think it was 2.6.21 that got the patch discussed here. The corrections I mentioned are included in another patch that is in 2.6.23, so obviously not in a Fedora release kernel.

Ian