Skip to contentSkip to conversation

statfs() and autofs

DISCUSSION Chris Walker 5 participants 18 messages Jul 15, 2008 – Aug 4, 2008 Kernel automounter

lore
Share & tools
lore
#0 lore raw

autofs 5.0.3
linux 2.6.18

When statfs() is called on an automounted directory, it returns before the automount occurs. This results in output like this initially:

% df /home/XXX
Filesystem 1K-blocks Used Available Use% Mounted on
- 0 0 0 - /home

If df is run a second later,

% df /home/XXX
Filesystem 1K-blocks Used Available Use% Mounted on
XXX:/home/XXX

                      80634688  77952736   2681952  97% /home/XXX

Perhaps there are options of which I'm not aware to mitigate this behavior.

Apologies if the answer to my query lies elsewhere in the kernel.

Signature · 1 line
Chris Walker -- [email protected]
maintainer
#1 lore raw
Quoted 5 lines · Chris Walker

autofs 5.0.3
linux 2.6.18

When statfs() is called on an automounted directory, it returns before the automount occurs. This results in output like this initially:

That may be because there are a group of system calls that don't trigger a mount because we have to avoid mount storms for map entries that use the browse option. In particular stat(2) and calls that use the same kernel path lookup flags, such as statfs(2).

Basically, if the mount point directory exists in the autofs file system but doesn't currently have a mount for some reason (perhaps it wasn't removed at expire or the entry uses the browse option), these calls will return the attributes of the autofs directory without performing a mount.

Quoted 17 lines

% df /home/XXX
Filesystem 1K-blocks Used Available Use% Mounted on
- 0 0 0 - /home

If df is run a second later,

% df /home/XXX
Filesystem 1K-blocks Used Available Use% Mounted on
XXX:/home/XXX

                      80634688  77952736   2681952  97% /home/XXX

Perhaps there are options of which I'm not aware to mitigate this behavior.

Apologies if the answer to my query lies elsewhere in the kernel.

#2 lore raw
Quoted 14 lines · Ian Kent
Quoted 6 lines · Chris Walker

autofs 5.0.3
linux 2.6.18

When statfs() is called on an automounted directory, it returns before the automount occurs. This results in output like this initially:

That may be because there are a group of system calls that don't trigger a mount because we have to avoid mount storms for map entries that use the browse option. In particular stat(2) and calls that use the same kernel path lookup flags, such as statfs(2).

Quoted 7 lines

Basically, if the mount point directory exists in the autofs file system but doesn't currently have a mount for some reason (perhaps it wasn't removed at expire or the entry uses the browse option), these calls will return the attributes of the autofs directory without performing a mount.

Hi Ian,

I need to say that this basically has to be declared a bug. I have been silent on this until now because I am so over committed with other tasks, and haven't had time to test it so cleanly as Chris has.

At my lab we find that on machines that do large batch processing, automounted directories often aren't there at the time that they are stat-ed. The workaround has been to put in some nanosleeps to give the mount point a chance to mount, but that is very much a hack, since now the code authors need to know the point where an automount is joined, or they at least have to find it out empirically. The code they write has become unportable and for all we know, incompatible on other systems if the autofs mount points are different.

To throw this in the mix, somehow, our Solaris systems, which share identical automount maps, simply never have this issue. Is there an architectural reason for this? And if so, can we learn how this is so, and apply similar logic to the Linux autofs/mount path?

Thanks,

Jon

_______________________________________________
autofs mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/autofs

#3 lore raw
Quoted 6 lines · Jonathan Loran
Quoted 4 lines · Ian Kent
Quoted 3 lines · Chris Walker

When statfs() is called on an automounted directory, it returns before the automount occurs. This results in output like this initially:

Quoted 6 lines
Quoted 4 lines

That may be because there are a group of system calls that don't trigger a mount because we have to avoid mount storms for map entries that use the browse option. In particular stat(2) and calls that use the same kernel path lookup flags, such as statfs(2).

I need to say that this basically has to be declared a bug.

I see this two ways. First, for the folks who have giant NAS boxes with a thousand exported filesysems, if ghost mounts (visibility on the client of non-mounted filesystems) are turned on and someone does "ls -F /net/nasbox" the resulting mount storm, when it stats every exported directory, is a serious pain in the butt, and when ghost mounts (browsing) were introduced there was a lot of howling until the mount storms were tamed.

On the other hand, the issue is Not Autofs' Problem, and if stat() or statfs() is directed at a nonlocal directory, information about its referent should be returned, not about the soon-to-be-underlying mount point. "ls", Nautilus, etc. are supposed to use mental telepathy to avoid causing mount storms; it is not autofs' place to interpret why the information is wanted and whether major resources should be expended to make it available.

Actually, only the sysop knows which is more important: doing the mount storm to respond to stat() or statfs(), or avoiding the storm.

Could we have two kinds of browsing, with or without mount storms, i.e. with or without a response to stat() or statfs()? This would be configurable the same way browsing is, individually per map row. The default should be to do the mount, because most sites export a reasonable number of filesystems from each server and so the mount storm is not a big deal, but a sysop who exports lots of directories from one NAS box and also turns on browsing in the client is advised to pick the non-mounting browse variant.

James F. Carter Voice 310 825 2897 FAX 310 206 6673
UCLA-Mathnet; 6115 MSA; 520 Portola Plaza; Los Angeles, CA, USA 90095-1555
Email: [email protected] http://www.math.ucla.edu/~jimc (q.v. for PGP key)

#4 lore raw

From jimc [07/20 15:31]:

Quoted 2 lines

Actually, only the sysop knows which is more important: doing the mount storm to respond to stat() or statfs(), or avoiding the storm.

Agreed. In our case, we have 'heavily discouraged' use of /net on our big servers and we would prefer that statfs()/stat() waited for the automount to complete.

Signature · 1 line
Chris Walker -- [email protected]
maintainer
#5 lore raw
Quoted 7 lines · Chris Walker

From jimc [07/20 15:31]:

Quoted 2 lines

Actually, only the sysop knows which is more important: doing the mount storm to respond to stat() or statfs(), or avoiding the storm.

Agreed. In our case, we have 'heavily discouraged' use of /net on our big servers and we would prefer that statfs()/stat() waited for the automount to complete.

Just to be absolutely clear.

The "browse" or "--ghost option isn't being used, in oder to avoid mount storms, but we still see stat(2) and statfs(2) (et. al.) not wait for a mount to complete.

Is that right?
Is anyone in a position to test a kernel patch?

Ian

maintainer
#6 lore raw
Quoted 9 lines · Ian Kent
Quoted 7 lines · Chris Walker

From jimc [07/20 15:31]:

Quoted 2 lines

Actually, only the sysop knows which is more important: doing the mount storm to respond to stat() or statfs(), or avoiding the storm.

Agreed. In our case, we have 'heavily discouraged' use of /net on our big servers and we would prefer that statfs()/stat() waited for the automount to complete.

Quoted 8 lines

Just to be absolutely clear.

The "browse" or "--ghost option isn't being used, in oder to avoid mount storms, but we still see stat(2) and statfs(2) (et. al.) not wait for a mount to complete.

Is that right?
Is anyone in a position to test a kernel patch?

We could add a third case to the description of the recently posted "[autofs] [PATCH 3/7] autofs4 - fix pending checks" patch but it's already been posted.

This can also happen via ->lookup() when only the LOOKUP_FOLLOW flag is present, as is the case for stat(2) and statfs(2) and the like.

Ian

#7 lore raw
In reply to Ian Kent

From raven [07/20 21:43]:

Quoted 8 lines

Just to be absolutely clear.

The "browse" or "--ghost option isn't being used, in oder to avoid mount storms, but we still see stat(2) and statfs(2) (et. al.) not wait for a mount to complete.

Is that right?
Is anyone in a position to test a kernel patch?

This is what I'm seeing and I'm only running automount with the --debug option. If I don't pass --ghost to automount, is there any other way browsing is enabled?

Signature · 1 line
Chris Walker -- [email protected]
maintainer
#8 lore raw
Quoted 7 lines · Chris Walker

From raven [07/20 21:43]:

Quoted 6 lines

Just to be absolutely clear.

The "browse" or "--ghost option isn't being used, in oder to avoid mount storms, but we still see stat(2) and statfs(2) (et. al.) not wait for a mount to complete.

Quoted 6 lines
Quoted 2 lines

Is that right?
Is anyone in a position to test a kernel patch?

This is what I'm seeing and I'm only running automount with the --debug option. If I don't pass --ghost to automount, is there any other way browsing is enabled?

Depends on version.

Version 5 will honor the "browse" option in shared maps, essentially causing indirect mounts to have "--ghost" option.

Ian

#9 lore raw
In reply to Ian Kent
Quoted 11 lines · Ian Kent
Quoted 8 lines · Chris Walker

From jimc [07/20 15:31]:

Quoted 2 lines

Actually, only the sysop knows which is more important: doing the mount storm to respond to stat() or statfs(), or avoiding the storm.

Agreed. In our case, we have 'heavily discouraged' use of /net on our big servers and we would prefer that statfs()/stat() waited for the automount to complete.

Quoted 10 lines

Just to be absolutely clear.

The "browse" or "--ghost option isn't being used, in oder to avoid mount storms, but we still see stat(2) and statfs(2) (et. al.) not wait for a mount to complete.

Is that right?
Is anyone in a position to test a kernel patch?

I may be willing to try the patch, but first I need to see if I can recreate the problem reliably. If not, there isn't much use. I just know that on our "faster" servers, the programmers I work with have demonstrated the issue. I will need their help. I'll let know if I make head way. Thanks,

Jon

P.S. Chris, can you reliably demonstrate the not waiting for the mount problem?

_______________________________________________
autofs mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/autofs

#10 lore raw

From jonathan [07/20 22:08]:

Quoted 2 lines

P.S. Chris, can you reliably demonstrate the not waiting for the mount problem?

I can, but I'm a few revs back. I'll see if I can replicate on current revs of linux and autofs.

Signature · 1 line
Chris Walker -- [email protected]
maintainer
#11 lore raw
Quoted 6 lines · Chris Walker

From jonathan [07/20 22:08]:

Quoted 2 lines

P.S. Chris, can you reliably demonstrate the not waiting for the mount problem?

I can, but I'm a few revs back. I'll see if I can replicate on current revs of linux and autofs.

It should be reproducible on any version of autofs and any kernel, if I understand correctly what is causing it. But, the real gotcha is stat(2) or statfs(2) are often accompanied by other system calls that will wait for the mount to complete. The mount can complete sometimes before but also sometimes after, and possibly independent of system call order (due to scheduling) so the problem should be intermittent and appear unpredictable.

Ian

#12 lore raw

From raven [07/21 00:39]:

Quoted 9 lines · Chris Walker
Quoted 6 lines

From jonathan [07/20 22:08]:

Quoted 2 lines

P.S. Chris, can you reliably demonstrate the not waiting for the mount problem?

I can, but I'm a few revs back. I'll see if I can replicate on current revs of linux and autofs.

Quoted 7 lines

It should be reproducible on any version of autofs and any kernel, if I understand correctly what is causing it. But, the real gotcha is stat(2) or statfs(2) are often accompanied by other system calls that will wait for the mount to complete. The mount can complete sometimes before but also sometimes after, and possibly independent of system call order (due to scheduling) so the problem should be intermittent and appear unpredictable.

I cannot reproduce this on fedora 9 (kernel: 2.6.25.6, autofs: autofs-5.0.3). Maybe I'll do some git bisecting to see where it got fixed.

Signature · 1 line
Chris Walker -- [email protected]
#13 lore raw

From cwalker [07/22 10:20]:

Quoted 18 lines

From raven [07/21 00:39]:

Quoted 9 lines · Chris Walker
Quoted 6 lines

From jonathan [07/20 22:08]:

Quoted 2 lines

P.S. Chris, can you reliably demonstrate the not waiting for the mount problem?

I can, but I'm a few revs back. I'll see if I can replicate on current revs of linux and autofs.

Quoted 7 lines

It should be reproducible on any version of autofs and any kernel, if I understand correctly what is causing it. But, the real gotcha is stat(2) or statfs(2) are often accompanied by other system calls that will wait for the mount to complete. The mount can complete sometimes before but also sometimes after, and possibly independent of system call order (due to scheduling) so the problem should be intermittent and appear unpredictable.

Quoted 2 lines

I cannot reproduce this on fedora 9 (kernel: 2.6.25.6, autofs: autofs-5.0.3). Maybe I'll do some git bisecting to see where it got fixed.

Looks like git bisect only helps to find introduced bugs, but not where something got fixed. Oh well. I think we were going to upgrade to 2.6.25 anyway... :)

Signature · 1 line
Chris Walker -- [email protected]
#14 lore raw
In reply to Chris Walker

From cwalker [07/22 10:20]:

Quoted 18 lines

From raven [07/21 00:39]:

Quoted 9 lines · Chris Walker
Quoted 6 lines

From jonathan [07/20 22:08]:

Quoted 2 lines

P.S. Chris, can you reliably demonstrate the not waiting for the mount problem?

I can, but I'm a few revs back. I'll see if I can replicate on current revs of linux and autofs.

Quoted 7 lines

It should be reproducible on any version of autofs and any kernel, if I understand correctly what is causing it. But, the real gotcha is stat(2) or statfs(2) are often accompanied by other system calls that will wait for the mount to complete. The mount can complete sometimes before but also sometimes after, and possibly independent of system call order (due to scheduling) so the problem should be intermittent and appear unpredictable.

Quoted 2 lines

I cannot reproduce this on fedora 9 (kernel: 2.6.25.6, autofs: autofs-5.0.3). Maybe I'll do some git bisecting to see where it got fixed.

Just closing the loop on this one--it turns out browsing was on. Disabling browsing (-nobrowse for each map in auto.master) corrected this behavior.

Signature · 1 line
Chris Walker -- [email protected]
#15 lore raw
In reply to Ian Kent
Quoted 5 lines · Ian Kent

Just to be absolutely clear.

The "browse" or "--ghost option isn't being used, in oder to avoid mount storms, but we still see stat(2) and statfs(2) (et. al.) not wait for a mount to complete.

I see this problem quite often. I have a CentOS 5.2 x86_64 system with autofs-5.0.1-0.rc2.88, kernel 2.6.18-92.1.6.el5xen. Automount maps are distributed by NIS, and I have explicitly set BROWSE_MODE=no. I see that:

* after an inactive mount times out, an umount is performed, but the mount point that was created is never removed, leading to:

* if the NIS map is changed to point the same key to a different location, a "df /foo/bar" _always_ returns /foo, and then /foo/bar (correctly) a few moments later.

Steve

#16 lore raw
In reply to Ian Kent

Thanks for your response.

In this case, it appears that statsfs triggers the mount, but returns before the mount completes. stat waits for the automount to complete before returning.

I would like statfs() to wait--is there a way to make it?

Thanks for all your hard work on this critical piece of software.

From raven [07/15 20:47]:

Quoted 37 lines · Chris Walker
Quoted 5 lines

autofs 5.0.3
linux 2.6.18

When statfs() is called on an automounted directory, it returns before the automount occurs. This results in output like this initially:

That may be because there are a group of system calls that don't trigger a mount because we have to avoid mount storms for map entries that use the browse option. In particular stat(2) and calls that use the same kernel path lookup flags, such as statfs(2).

Basically, if the mount point directory exists in the autofs file system but doesn't currently have a mount for some reason (perhaps it wasn't removed at expire or the entry uses the browse option), these calls will return the attributes of the autofs directory without performing a mount.

Quoted 17 lines

% df /home/XXX
Filesystem 1K-blocks Used Available Use% Mounted on
- 0 0 0 - /home

If df is run a second later,

% df /home/XXX
Filesystem 1K-blocks Used Available Use% Mounted on
XXX:/home/XXX

                      80634688  77952736   2681952  97% /home/XXX

Perhaps there are options of which I'm not aware to mitigate this behavior.

Apologies if the answer to my query lies elsewhere in the kernel.

Signature · 1 line
Chris Walker -- [email protected]
maintainer
#17 lore raw
Quoted 5 lines · Chris Walker

Thanks for your response.

In this case, it appears that statsfs triggers the mount, but returns before the mount completes. stat waits for the automount to complete before returning.

If statfs really is triggering the mount (not something else you haven't noticed, maybe another command or program system call) then it should wait. If your map doesn't use the "browse" option or the --ghost option then stat and statfs will trigger mounts and wait for them to complete. That's not to say there aren't any races relating to multiple concurrent processes accessing the mount points. I'm working on that right now. But that isn't the situation your describing by the sound of it.

So something else is going on that we're not yet aware of.

Quoted 10 lines

I would like statfs() to wait--is there a way to make it?

Thanks for all your hard work on this critical piece of software.

From raven [07/15 20:47]:

Quoted 4 lines · Chris Walker
Quoted 2 lines

autofs 5.0.3
linux 2.6.18

What patches have you applied to this kernel.
I have a bunch of maintenance and bug fixes that aren't in this kernel.

Quoted 34 lines
Quoted 33 lines
Quoted 3 lines

When statfs() is called on an automounted directory, it returns before the automount occurs. This results in output like this initially:

That may be because there are a group of system calls that don't trigger a mount because we have to avoid mount storms for map entries that use the browse option. In particular stat(2) and calls that use the same kernel path lookup flags, such as statfs(2).

Basically, if the mount point directory exists in the autofs file system but doesn't currently have a mount for some reason (perhaps it wasn't removed at expire or the entry uses the browse option), these calls will return the attributes of the autofs directory without performing a mount.

Quoted 17 lines

% df /home/XXX
Filesystem 1K-blocks Used Available Use% Mounted on
- 0 0 0 - /home

If df is run a second later,

% df /home/XXX
Filesystem 1K-blocks Used Available Use% Mounted on
XXX:/home/XXX

                      80634688  77952736   2681952  97% /home/XXX

Perhaps there are options of which I'm not aware to mitigate this behavior.

Apologies if the answer to my query lies elsewhere in the kernel.