Status of mount.nfs

Hi,

util-linux 2.13 drops support for NFS mounts:

Release highlights:
------------------
mount(8) doesn't include NFS client code anymore. Don't forget to install nfs-utils 1.1.0 or newer with /sbin/[u]mount.{nfs,nfs4}.

However, the last I can find in nfs-utils changelog about this is:

commit 99414bd3eecf93f23c378d3bb3d45bc98f364abc
Author: Neil Brown <[email protected]>
Date: Sat Jul 8 09:41:58 2006 +1000

Disable building/installing mount.nfs by default.

mount.nfs does not yet support 'user' option and some others. To make it support this we need to make it setuid-root, and some security isses need to be resolved before that can be done safely.

What's the current recommendation for distributions with regard to NFS mounting?

/* Steinar */

Homepage: http://www.sesse.net/

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Hmm, that's news! :-( Sounds like your only choice is to stay with older releases of util-linux in order to provide NFS support in the normal mount command binary.

I'm trying to build a prototype for in-kernel mount option string processing, and I thought mount.nfs would be the place to start. I didn't realize that it was in such an "alpha" state.

However, as I've hacked on the code, I've come up with a number of fixes to some of the functional issues Neil mentions, although not the security ones.

I'll post here as soon as I can test them, since I think we want to get mount.nfs fixed and off the ground ASAP. It's a good idea, although perhaps the current implementation still needs work.

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA email;internet:chuck dot lever at nospam oracle dot com title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

I wonder why you didn't find:
commit b3b111b1bd5fbc678419bf1964b6093045081139

Author: Neil Brown <[email protected]>
Date:   Tue Mar 20 14:18:41 2007 +1100

Build mount.nfs by default, and install setuid

Also fix a few bugs that came up in initial testing.

nfs-utils-1.1.0 builds and install mount.nfs by default, and should be used with the new util-linux.

NeilBrown

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

There's a simple explanation for that -- I checked the file ChangeLog, which doesn't seem to have been updated since September, now that I look at it. I guess you stopped updating it at some point...

Thanks for the clarification; we'll make the switch pretty shortly.

/* Steinar */

Homepage: http://www.sesse.net/

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

I just noticed this morning that "./configure --help" in the latest git repo still says this:

--enable-mount Create mount.nfs and don't use the util-linux
mount(8) functionality. [default=no]
^^^^^^^^^^^^

If building and installing mount.nfs is the default, perhaps this should be updated.

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel/
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Thanks for noticing that. It is now fixed in current git.

NeilBrown

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Just a few experiences with this after a week or so with the new default in Debian:

- The new mount is really more picky in several aspects. Most notably, it refuses to mount anything unless rpc.statd is running, which broke our NFS mounting on boot. Also, it seems to be less forgiving about portmapper registration; cfs (an encrypted filesystem based off NFS, it seems) sets up a server on port 3049 and expects "-o port=3049,nfsvers=2" to work. The new mount.nfs searches for a portmapper unless the udp option is also correctly set. I'm slightly surprised at this increase in strictness, given that I thought it was essentially the same code.
- If you forget to install it suid (we strip suid bits automatically) you will of course break user mounts. :-)
- The umount exit status is broken, which will cause odd failures on umount from the GNOME drive manager (basically, an empty error dialog box). Apply the patch I posted here earlier.

I guess that's all the traps I've fell into for now; I hope they'll make it easier to make the move for other distributions.

/* Steinar */

Homepage: http://www.sesse.net/

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Thanks for sharing.

Is this simply because you were mounting NFS filesystems before running statd? In that case, maybe we should say that it "highlighted that your NFS mounting on boot was already broken" ?? :-) If you need to nfsmount '/' or '/var', you should mount them with "-o nolock" and then statd won't be needed.

I'm a bit confused here. I would expect that to avoid portmap being used, you would need to set both 'port' and 'mountport' on the command line. It shouldn't have anything to do with whether UDP is set. Can you say a bit more about what you discovered here?

It is based on the same code but with quite a number of fixes and "improvements". It is entirely possible that some regressions were introduced, though I tried to do a reasonable amount of testing and review.

and if you forget to install the new man pages, you will have incomplete documentation :-)
nfs-common 1:1.1.0-9 does not have mount.nfs and umount.nfs man pages.

Thanks for this fix.

Thanks a lot!

NeilBrown

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Yes, we were. I'm in the process of patching the boot process now, so you could probably shed light on a related issue: Do we need to start portmap before statd or not? The comments in the old code is slightly cryptic :-)

Well, I have no idea how these things work at all, I just traced through the code and saw what would make it work :-) cfs runs as its own NFS daemon, on port 3049, but does not seem to register with the portmapper. The mount command given, without "-o udp", gives:

mount: mount to NFS server 'localhost' failed: RPC Error: Success.

The error seems to originate from probe_nfsport(), which also has a line saying

  if (pmap->pm_vers && pmap->pm_prot && pmap->pm_port)
          return 1;

which indicates that if version+protocol+port are all set beforehand, don't probe -- and indeed, that works. I stopped debugging the issue there.

Thanks, nice catch. I'll be sure to fix that in the next upload.

/* Steinar */

Homepage: http://www.sesse.net/

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Yes. Portmap should be one of the first things started after basic networking is up. You might even be able to start it before that, but I'm not sure. You need it before any RPC service which includes statd, lockd, nfsd, ypbind, ...
You can mount an NFS filesystem with '-o nolocks' before portmap, but that it about all.

Hmmm.... That code looks a little odd. If you say you want udp, it will still probe for tcp first. That doesn't hurt as UDP and TCP always use the same port: 2049. But it is still odd. I wonder if it is needed at all.

NeilBrown

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

It seems it broke for a more normal situation as well:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=433881

Giving the "udp" option helped the user, though, but I don't really see why the behavior changed.

/* Steinar */

Homepage: http://www.sesse.net/

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

The default _had_ to change to TCP because UDP is just too unreliable for NFS running with large (> 8k) r/w sizes, or on nonhomogeneous networks (mixed 10/100/1000Gbit), or with fast clients.

Trond

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Are you saying it changed between util-linux mount and nfs-utils mount.nfs? I thought that change happened a lot earlier, and that this was something else.

/* Steinar */

Homepage: http://www.sesse.net/

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

The private versions of util-linux maintained by RedHat and SuSE (and Debian?) transitioned earlier, but I don't believe we ever sent those patches to the upstream maintainers of util-linux.

Trond

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

It was quite simple to test this myself. I started the kernel server on a machine, then shut down portmap. First I did:

fugl:~> sudo mount -t nfs -o port=2049,mountport=901,nfsvers=3 192.168.0.101:/ /mnt
mount: mount to NFS server '192.168.0.101' failed: System Error: Connection refused.

The dump is attached as "default.dump". Then I did

fugl:~> sudo mount -t nfs -o port=2049,mountport=901,nfsvers=3,udp 192.168.0.101:/ /mnt

which is attached as "udp.dump".

Note that in default.dump, UDP is simply never tried at all. I believe that to be a bug.

/* Steinar */

Homepage: http://www.sesse.net/

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Nope. Nowhere in the documentation will you find a promise to fall back to UDP.

Trond

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

OK, but in that case the regression should still be documented, as it seems this worked before.

/* Steinar */

Homepage: http://www.sesse.net/

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

It looks like, in the UDP dump you sent, there is no attempt to contact the portmapper at all. The mount request is the first thing in the dump, and the request goes right to the port specified on the command line.

The TCP case fails because mount.nfs is using the portmapper even though the user has specified the ports on the command line. Could that be the root cause of the failure?

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

If the user specifies a port, then there is no real good reason to use the portmapper:

* For the case of the mount protocol, we should just try an RPC call and then look at the returned RPC error values to figure out which versions of the protocol are supported if it fails (or alternatively, fall back to UDP if the TCP connection fails).
* If the mount call has succeeded and we have a port for the NFS server, we should probably just try the mount call for a sufficiently recent kernel, then look at the returned error codes. For older kernels (pre 2.6.13?) which don't return decent error values, then ping first in userland and look at the RPC return values. Retry with UDP if TCP doesn't work...

Trond

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Exactly.

I'm wondering how much of this is already implemented in mount.nfs, but is just broken. I'm reviewing the code now.

I cleaned this up a little with recent patches so it should be easier to figure out what's going wrong.

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Hi Steve-

I was trying out the mount.nfs test case for another bug (see below). The test case didn't work against a Fedora 7 server. Trying to mount with UDP against a specific port just hangs. So I tried an rpcinfo against it to see what the current rocbind configuration was.

Um. Ok, where are the IPv4 entries?

I've now completely shut off IPv6 initialization and autoconfiguration on the only network interface on the system and rebooted several times (I wasn't using the IPv6 networking stuff yet anyway). I still get *no* udp4 or tcp4 entries in the rpcbind database. The NFS service on this system is all IPv4 (it's a Linux NFS server).

How are these getting registered?

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Hi Steinar-

I'm looking at probe_nfsport() and probe_mntport() and I see that the portmapper call is avoided iff the protocol version, transport protocol, and the port number are all specified in advance.

So if you specify:

-o mountport=650,port=2049

mount will still contact the server's portmapper to determine which transport protocols are available (and this breaks through a firewall that doesn't pass portmapper requests). Only if you specify:

-o tcp,mountport=650,port=2049

or

-o udp,mountport=650,port=2049

then the portmapper calls are avoided entirely. For some reason probe_bothports() sets the default NFS version to 3 but does not set a default transport protocol.

The transport protocols are probed differently for NFS and MNT: for MNT, UDP is probed first then TCP; for NFS, the opposite is true. The mount command is supposed to try both transport protocol types both for NFS and MNT, but it appears that it is failing to try the other type if the first fails... I see this is also problematic for umount.nfs.

So yes, I think the logic is there, but it's broken. I recall that the transport protocol probing logic was added recently.

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

The protocol is set by probe_port().

The I idea here was to using UDP to probing for both rpc.mountd and the NFS server so as not to put (tcp) ports in TIMEWAIT, basically making them unavailable for the actual mount. The allowed many more tcp mounts to happen during autofs mount storms.

Note: if the protocol was explicitly specified (i.e. proto=tcp) only that protocol was used during the probing and mounting. I have been asked to change that as well. Meaning if when proto=tcp is specified, they still want the udp probing to occur.

steved.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Right. I don't see GETPORT using UDP as often as perhaps it should, though. I need to go back and check it.

And umount.nfs always uses TCP for the mountd request. I have a patch that fixes that to behave more like mount.nfs does, which I will forward in the next day or two.

I notice some problems if a share is mounted with TCP, but the server later disables TCP -- umount.nfs hiccups on that when it tries to umount using the same protocol as listed in /etc/mtab. Perhaps relying on /etc/mtab for setting the umount protocol is unnecessary.

I would like to spell out exactly what behavior we want to solder into the community nfs-utils here, at least so I can approximate it with the string-ified mount implementation, but also because it seems to be quite heuristic and totally undocumented. If we tune for a particular use case, it will break others, so we need to agree on a default that works for most cases.

(As part of this exercise it would also be lovely to assemble some nice regression test cases, but that may be too much to hope for ;-)

We have three requests that need to be made:

1. GETPORT -- I think this should UDP all the time unless proto=tcp is explicitly specified;

2. MNT -- likewise, UDP unless proto=tcp is specified or GETPORT says UDP is not supported;

3. NFS -- this should be TCP all the time unless proto=udp is specified or GETPORT says TCP is not supported.

Even better would be to use RPCB_DUMP instead of RPCB_GETPORT. That way we only need a single rpcbind call for both protocols, and can get transport protocol information as well, and make an "informed" choice.

Also, can we get rid of the clnt_ping()? If not, can we document why it is there? It adds two extra round trips to the whole process. If error reporting is the problem, maybe we can try the pings only if the kernel part of the mount process fails?

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA email;internet:chuck dot lever at nospam oracle dot com title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

thats a bug... umount should use the protocol the mount did... I thought I had fixed that... :-\

I think I was using /proc/mounts...

Some people have asked that we first try UDP all the time... which I have resisted but it might make sense...

What about rollbacks... meaning if tcp is not supported do we try udp? if v4 is not supported to we try v3 and the v2 or just fail the mount?

Good point... but note, a while back I got a request to use GETPORT instead of DUMP because some Cisco router actually use the GETPORTs to punch wholes in their firewalls.

How do we avoid hang down deep in RPC land (governed by uncontrollable timeout) when either mountd or nfsd are not up?

That was the main reason for the ping. Since neither portmapper or rpcbind ping their services before they hand out the ports, there is really no way of telling where the server is up? So to avoid the hang, we ping them... Sure its costly network wise, but hanging during a boot because a server is not responding is a bit more costly... imho...

steved.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Right, but recent kernels both can and will ping the NFS service for you.

Cheers
Trond

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Good point... but that's just as costly (wrt network traffic) as the mount command doing the pinging... plus the status of the remote mountd is also needed.

steved.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

The kernel ping is sent on the same connection to the server than the NFS client will use, so we don't waste any extra TCP ports.

I agree that you need to figure out the mountd parameters, but the right thing to do there is simply to try the command. The RPC return values will tell you if the service or version you tried isn't supported.

Cheers
Trond

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

After how long of a wait? If all the timeouts are controllable then I agrees, but if we have to wait undefined-able amount of time for every RPC retransmit, then I think we should do a ping...

steved.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

You should be able to set the timeout either on a per-RPC call basis by using clnt_call(), or by changing the default timeout on the CLIENT object using clnt_control() (man 3 rpc).

Cheers
Trond

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

The mount command uses a "standard" timeout structure to set a 25 second timeout for both UDP and TCP, but I've noticed that TCP connections for most RPC requests from the mount command to unavailable servers appear to hang indefinitely. I'll take a look at this.

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

True... but I was thinking of when clnt_cnt() call pmap_getport() which does not take a timeout value... In that case you are stuck with a 60 sec hard coded timeout, regardless of the time out you pass in...

steved.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

In the version of mount.nfs from the linux-nfs.org git tree, we call our own private implementation of pmap_getport() instead of the one from glibc.

Cheers
Trond

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

I was looking at this yesterday. The stock timeout for TCP connects on Linux is 75 seconds. The version of getport() used in the mount command might control the TCP connect timeout by using a non-blocking connect() with a select(). The select() then times out if the connection doesn't complete.

But I'm wondering if we really want to continue using TCP for GETPORT calls. Solaris mount appears to use only UDP for GETPORT, for example.

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

As as long as the GETPORTs don't use privilege ports I don't think its a problem... plus I don't think one size fixes all.. meaning due to different firewalls requirements both udp and tcp GETPORTS will be needed... imho...

steved.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Not sure what you mean. Yesterday you said the TCP connect timeout *was* a problem. I've recommended two ways to address it.

The ephemeral port space is limited too, don't forget. It's simply a somewhat larger space than the privileged port space. If a large network application (say, a web server) is running on the system, that space can shrink fairly rapidly, and we're in nearly the same boat as with privileged ports. Using a TCP connection from an ephemeral port only mitigates the port space problem, it doesn't really correct it entirely.

We say "firewall!" a lot, but I would like to see typical use cases for mounting through a firewall so I understand what kind of implementation we're aiming for (and maybe even what kind of test cases to build!). Do our users really expect to mount NFS shares through any firewall with "-o defaults" ?

I'd like to hear from the distributors what you consider are the use cases that absolutely must be supported. Otherwise we will end up standing on our left big toenail to support stuff that isn't worth the pain or is never used.

And can anyone shed some light on why Solaris uses only UDP for GETPORT requests at mount time? Does it ever fall back to try GETPORT on a TCP connection? How do Solaris users mount through a firewall? Come on all you Solaris NFS guys out there, I know you're lurking.

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

TCP timeouts are a problem if you can't control them... But point taken... UPD is probably the best way to query a portmapper or rpcbinder to get the needed info...

Only mitigates the problem for a short time and you'll always run out of privileged port before running out of non-privileged but again... point taken... eliminating the problem is probably the answer...

Yes! Mostly on the server side... meaning people wanted to set the port the daemons listen on (via the initscripts) so clients can access the server through a firewall... Is this a common setup? No. But there are people that want a firewall between the server and client.. Also I can only assume the reason for the 'mountport=" option was to work better with firewalls... but that is only speculation...

In the end, I think we need to be able to control the ports and protocol mounts uses, allowing people to punch holes in firewalls.

steved.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

OK, I have a patch that shortens the TCP connect timeout for mount.nfs.
Will post a follow-up; please take a look.

Yes, and you've suggested a mount connection cache to help with this... that might be something reasonable to try in the kernel mount implementation at some point.

I'm not suggesting that we don't support mounting through a firewall. I'm wondering, though, how people expect it to work. Is it acceptable to require a few extra mount options on clients to mount successfully through a firewall, or should a mount with no options whatsoever always work in this case?

And, does anyone have real and precise test cases to make sure we don't break mounting through a firewall when changes are made to the mount infrastructure?

I agree that the mount{prog,vers,port}= options are very likely for mounting through firewalls.

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

The strategy should be to make the _default_ behaviour safe. If you want to add optimisations that need switching on/off then those may take extra mount options.

Trond

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Exactly: is mounting through a firewall an optimization, or is it something that users expect, by default, to work without using extra options?

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

If the user has opened the firewall for outgoing TCP connections, then that is one of the things that I think should just work.

It doesn't need to work with maximum efficiency (so if you want to try sending UDP portmap requests before falling back to TCP, then fine) but opening UDP ports in the firewall or specifying extra mount options should not be a requirement.

Trond

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

I think I agree with Trond, although I I'm not sure what he means by safe mount behavior... but I am of the opinion that mounts which need to go through a firewall will require extra options will be needed...

We don't have explicit test for this... but I can guarantee if we break it we'll hear about very quickly... ;-)

steved.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Sorry about all the questions... and thanks for providing the history. The good news is that, now that [u]mount.nfs resides in nfs-utils, we can easily make this work a whole lot better.

Nope... umount.nfs sets the transport protocol to TCP explicitly before doing the umount call. Check out utils/mount/nfsumount.c:_nfsumount() .

umount.nfs uses getmntdirbackward(), which probes /etc/mtab, as far as I can tell. One problem with this is that often the effective transport protocol isn't listed in /etc/mtab at all, if, say, the user requests TCP and the server supports only UDP.

I can't see why we need to refer back to either file to determine the transport protocol for a umount request. Whatever transport mountd is advertising at the moment is what should be used, right?

[ Steve, since you have a different recollection of how all this mount stuff works, I wonder if Amit took an older version of mount when he split out the new mount.nfs helper... Can you verify this? Maybe there are some fixes you made that need to be ported over. ]

So far I've only found one reason to allow GETPORT over TCP, and that's to go through firewalls. That's why I proposed allowing proto=tcp to override the default... I'm very interested to hear other use cases that might fail with UDP.

I think breaking back can be supported by grabbing all data about the interesting services from portmapper at the start of a mount request. That way mount.nfs can build the correct request based on the list of services advertised on the server, and the list of options from the mount command line, then make a single set of requests. No retry logic is needed except for handling "bg".

Sigh. ;-)

I guess I don't see how a NULL RPC is different than sending a real request, when we're talking about a single MNT request from a user space application. If the service is down, it fails either way.

My feeling is we should then fix the kernel to behave more reasonably. I recently changed the kernel's rpcbind client to use "intr" instead of "nointr" for its requests, for example. Is it practical to track down the hangs and fix them? Is it just the long time waiting for a failure, or do the mount processes actually get totally stuck?

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

This got lost in the translation... In older mount code (i.e. the one in utils-linux) /proc/mounts is used which is a much simpler way of dealing with this... imho..

Well for firewall reasons you generally want to use the protocol that the mount used...

No... I pretty sure I had Amit use the latest and greatest... I just think there was some decisions made or liberties taken without a complete understand of what the ramification were...

right...

As long as the request does not get caught up in some unreasonably long timeout in the RPC code... there is no difference... Waiting 60sec for each retry or to find out some service is down would not be a good thing when a machine is coming up...

In the kernel yes, in glibc no because that code will not change, period!

Its a long time wait that can not be controlled...

steved.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Miklos seems intent on eliminating /etc/mtab anyway...

That could have been a very long time ago, even months, and the server settings may have changed. Thus sending what mount used seems inherently unreliable. The race window is enormous!

Thanks for checking on this. I worried we may have missed some important bug fixes.

Well, if libtirpc is added to nfs-utils, the mount command could use that instead. We'd be able to fix any bugs in libtirpc quite easily. That seems like an excellent way to address every problem with glibc's RPC implementation, and immediately have a "simple" use case for testing libtirpc (or whatever we have to replace the RPC functionality in glibc).

Ok.

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Good...

hmm... I must be missing something... Why is umount-ing with the same network protocol that mount used unreliable and racy?

A while back I did a patch dump of all the bugs we found when we added the new code to Fedora... Neil's tree has all the patch we have..

I can't agree with you more... At this point both rpcbind and libtirpc are now fully supported by both Bull and yours truly... Both tarballs are available on sourceforge:

http://sourceforge.net/projects/libtirpc/
http://sourceforge.net/projects/rpcbind/
Git trees are at:
http://git.infradead.org/?p=users/steved/libtirpc.git
http://git.infradead.org/?p=users/steved/rpcbind.git

And of course the rpms are available from Fedora mirrors

At this point everything is not quite synced up but that will change very shortly... and new release will be coming because the code is being used and bugs are being fixed...

The next major step would be to port nfs-utils to use libtirpc which is on my todo list along with a ton of other things... :-\

In the end, I think it would be a very good move for our community to own the entire stack... including the RPC library code...

steved.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Well, I exaggerated a bit, but technically speaking it is a race window because the client is caching information about the server. When a mount succeeds, the client remembers the transport protocol it used for the mount in /etc/mtab.

A long time later, it uses that cached information to set the transport protocol for the umount request.

During the time the share is mounted (ie the window), some system administrator can change the server's settings -- for example, by disabling the TCP mountd service, or by changing the iptables on the server. At that point, the information in the client's /etc/mtab becomes stale, and, in fact, useless as a hint.

I'm simply arguing that /etc/mtab should be ignored by umount entirely when it comes to determining what transport protocol to use because that cached information in not reliable. Just do a fresh GETPORT before the umount, and you will be all set.

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA email;internet:chuck dot lever at nospam oracle dot com title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Here's a new one: mount doesn't pass -s (--sloppy) on to mount.nfs. This causes odd behavior when people give all sorts of weird options, especially (it seems) to automount. I've asked the util-linux maintainer in Debian to fix the bug (and probably send the patch upstream), but I guess you should be aware of the issue until it's fixed.

/* Steinar */

Homepage: http://www.sesse.net/

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs