mount.nfs: chk_mountpoint()

Hi all-

The recent addition of the chk_mountpoint() function in utils/mount/mount.c in nfs-utils commit 3b55934b has broken a particular behavior of background mounts.

nfs(5) states that, if the "bg" option is specified, "A missing mount point is treated as a timeout, to allow for nested NFS mounts."

If I try mounting an NFS share onto a non-existent directory while using the "bg" option, I now get an immediate failure:

mount.nfs: mount point /mnt/nothere does not exist

instead of the mount backgrounding itself to wait for /mnt/nothere to show up. This is because chk_mountpoint() is causing the mount request to fail immediately.

Is the documented bg retry behavior still desirable?

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Isn't this what autofs is for? To be able to handle hierarchies of mounts?

    Thanx...

       ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

I think the purpose of this feature is to allow a sysadmin to specify a set of mount points in /etc/fstab, some possibly nested. "mount -a -tnfs" should work no matter what order the mounts in /etc/fstab are specified.

After all, some servers may be unresponsive when the client boots -- the mounting order is nondeterministic; it can't be depended on, in any event.

We also don't know if the automounter itself depends on this feature.

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Autofs depending upon this feature would be a large mistake. IMHO, of course. :-)

I don't think that it does.

But your explanation makes sense, although we should be moving people away from static mounts in fstab and towards dynamic mounting via autofs. Ian and Jeff have made autofs much, much better in recent times. Improving autofs further to make it only mount file systems which are actually referenced would make it even better.

How do we find out whether we need to continue supporting this semantic or whether we can do away with it? Clearly, if it was busted, then not many people were depending upon it because there didn't seem to be any hue and cry about it not working.

    Thanx...

       ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

I'm in great favor of autoconfiguration. Anything that will make NFS "just work" is goodness, in my book.

Well, that change went into nfs-utils in May of 2007, only 3 months ago.
Depending on when nfs-utils-1.1.0 got into Fedora 7, I don't think the
change has had wide exposure quite yet.

Considering there hasn't been much "hue and cry" about "bg" not working in Fedora, however, that may not be much of a standard by which to measure customer dissatisfaction.

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Is this an argument for or against autofs or these changes?

I would guess that not so many people are using the "bg" option, period. Many of Linux's customers are ex-Sun customers and they were educated to use autofs and to move away from and stay away from static mounts via fstab or vfstab.

The "bg" option was a hack added to speed up system booting. A much better solution to the problem was autofs because it delayed the mounting until the file system was actually needed.

The "bg" option can lead to applications not working correctly because the file system may or may not be mounted when they need it to be there and there is no automatic synchronization to block them until it is. Autofs supplies this synchronization, thus once again, making it a vastly superior solution.

    Thanx...

       ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Making autofs a reliable and useful facility is a good thing. Kudos to Ian and Jeff for their effort.

I don't disagree with that assessment.

Whether or not it is a kludge, I don't think we have enough information about who is using it for what to blithely remove it without fanfare or documentation.

What alarms me more, though, is that we don't have any unit tests that caught this change before it went into the git repo. This change, in itself, may not be terribly harmful. But think of a minor and unintended change that might go in without notice, and break a lot of environments.

Some people find such enforced "synchronization" to be painful and annoying. They would prefer a solution where the application is free to take its own recourse rather than hang indefinitely.

I think it is valid to want to have either type of behavior.

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

I would agree completely.

Well, then we'd hear pretty quickly, I think...

But yes, a test would be a better way to do it. :-)

This, I would agree with. I still think that autofs is the right place to put the appropriate synchronization, whether it be to block the application or fail whatever system call that it was that attempted to access the not yet mounted file system.

    Thanx...

       ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

No, it is indispensable to recover properly from a power outage:
servers tend to boot slower than clients. Also, it is not unusual to have some minor network/server problems after an outage causing the mount to fail.

Without the bg option a temporary power outage may render all client systems unusable.

Frank

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

And a better solution to this problem is still to use autofs.

That said, what use are the clients _until_ the servers are up? The applications on them can't run correctly because the file systems that they depend upon may or may not be there yet. With autofs, you would have a chance of getting the synchronization right.

You also get all sorts of benefits such as decreased resource usage (by not having inactive file systems mounted), reduced hangs (by not having inactive file systems from servers which go down still mounted), in addition to the situation described above and other benefits as well.

I do recognize that we can't get rid of the bg option, but I would request that people using it consider different alternatives to solving their problems.

ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

For the record, one downside to using automounter is the mount storm that is caused when a distributed application starts up on multiple clients requiring many NFS mount points on each client. This is one reason some sites choose not to use automounter. "bg"s retry behavior, though a kludge, is somewhat more friendly.

From my experience, generally mountd (on most any server implementation) has been a scalability problem in these scenarios. It can't handle more than a few requests per second.

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

If the application on each client is going to need many mount points, then how does "bg" do anything but increase the number of concurrent mount requests coming from each client, thus increasing the load?

Autofs supporting dynamic mounting of individual file systems within a hierarchy would help to reduce the overhead on the network and server as much as seems possible to me...

I suspect that this is on Ian's and Jeff's lists... :-)

Perhaps we need to look at multithreading mountd? Ala Solaris?

    Thanx...

       ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Does nfs-utils commit 11d34d1 (below) do the job?

--b.

commit 11d34d11153df198103a57291937ea9ff8b7356e

Author: Greg Banks <[email protected]>
Date:   Wed Jun 14 22:48:10 2006 +1000

multiple threads for mountd

How about the attached patch against nfs-utils tot? It adds a -t option to set the number of forked workers. Default is 1 thread, i.e. the old behaviour.

I've verified that showmount -e, the Ogata mount client, and a real mount from Linux and IRIX boxes work with and without the new option.

I've verified that you can manually kill any of the workers without the portmap registration going away, that killing all the workers causes the manager process to wake up and unregister, and killing the manager process causes the workers to be killed and portmap unregistered.

I've verified that all the workers have file descriptors for the udp socket and the tcp rendezvous socket, that connections are balanced across all the workers if service times are sufficiently long, and that performance is improved by that parallelism, at least for small numbers of threads. For example, with 60 parallel MOUNT calls and a testing patch to make DNS lookups take 100 milliseconds time to perform all mounts (averaged over 5 runs) is:

num elapsed
threads time (sec)
------ ----------
1 13.125
2 6.859
3 4.836
4 3.841
5 3.303
6 3.100
7 3.078
8 3.018

Greg.
--
Greg Banks, R&D Software Engineer, SGI Australian Software Group. I don't speak for SGI.

index a3cb7ce..888c915 100644
@@ -22,6 +22,7 @@
 #include <netdb.h>
 #include <rpc/rpc.h>
 #include <sys/socket.h>
+#include <sys/fcntl.h>
 #include <errno.h>
 
 #ifdef _LIBC

@@ -112,6 +113,26 @@ svc_socket (u_long number, int type, int protocol, int reuse)
 	}
     }
 
+  if (sock >= 0 && protocol == IPPROTO_TCP)
+    {
+	/* Make the TCP rendezvous socket non-block to avoid
+	 * problems with blocking in accept() after a spurious
+	 * wakeup from the kernel */
+	int flags;
+	if ((flags = fcntl(sock, F_GETFL)) < 0)
+	  {
+	      perror (_("svc_socket: can't get socket flags"));
+	      (void) __close (sock);
+	      sock = -1;
+	  }
+	else if (fcntl(sock, F_SETFL, flags|O_NONBLOCK) < 0)
+	  {
+	      perror (_("svc_socket: can't set socket flags"));
+	      (void) __close (sock);
+	      sock = -1;
+	  }
+    }
+
   return sock;
 }
 

index bac4421..70166c1 100644
@@ -125,6 +125,13 @@ If this option is not specified the default of
 .BR /var/lib/nfs
 is used.
 .TP
+.BR "\-t N" " or " "\-\-num\-threads=N"
+This option specifies the number of worker threads that rpc.mountd
+spawns.  The default is 1 thread, which is probably enough.  More
+threads are usually only needed for NFS servers which need to handle
+mount storms of hundreds of NFS mounts in a few seconds, or when
+your DNS server is slow or unreliable.
+.TP
 .B \-V " or " \-\-nfs-version
 This option can be used to request that
 .B rpc.mountd

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

This solves part of the problem. If mountd depends on external services like DNS and netgroup lookups, multithreading will help only somewhat.

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

"bg" has an exponential backoff, so the load increase isn't terribly bothersome. It's the "bg" recovery mechanism that's useful here for getting all the mount requests to be successful in a nondeterministic environment.

I'm not arguing against automounter here; I'm just presenting a typical use case that is common enough for us to pay some heed.

Yes, although I was thinking of ways to make the client side more friendly, say, by serializing mount requests to the same server, since we don't have any influence on the scalability of mountd in non-Linux NFS implementations.

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

"bg" also tries synchronously before going into the background (and backing off). So, by itself "bg" does not generate a storm. It only slightly raises the mount traffic after a failure, by giving way to the next filesystem in line - which may well be on another server.

Done right, "bg" is a reasonable approach, IMO.

Tom.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

I will belabor this just a little more and then move on.

Yes, if everything is working correctly, then the "bg" option adds insignificant overhead and that is in the argument processing for the mount command.

However, if it backgrounds, there can be multiple mounts running at the same time. This could add up to quite a bit for very many file systems.

And, why are they getting mounted? Probably not because they are doing to needed immediately, but because they are being statically mounted and the only way to do that is via fstab at mount time.

All of these file systems are sitting, in the namespace, waiting to cause problems for applications looking at the namespace if the slightest problem occurs out on the network or at one of the servers which was mounted from. They add overhead to applications which do look at the namespace in the meantime because they add to the number of file systems that these applications need to keep track of.

I wish that I had a nickel for each time that I heard a customer complain because some dead server caused his application or the df command to hang and he wasn't interested in that server. I'd be retired and typing this from some fun location. :-)

Oh well, "bg" is a solution, but I think that there are better.

    Thanx...

       ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Sometimes, but not always.

The point is, you would have to re-issue mount -a whenever a server starts to honour mount requests again: manual intervention on all client systems is not really practical with many clients.

No, actually the automounter _costs_ resources depending on the situation:
The automatic umount flushes all kinds of caches on the client, requiring network bandwith after mounting them again.

There is no perfect solution. autofs and statically configured mounts have both advantages and disadvantages.

Frank

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs