Skip to contentSkip to conversation

Text-based mount interface breaking non-UDP mounts

DISCUSSION Steinar H. Gunderson 4 participants 18 messages Apr 23, 2008 – May 5, 2008 8 messages not in the archive

Subsystems (2)
NFS client NFS server
lore
Share & tools
lore

Steinar H. Gunderson

author
#0 lore raw

Hi,

I've gotten a few bug reports lately that if UDP is blocked at the server, NFS mounts (even with -o tcp) fail. git bisect shows that the culprit is the change "Get rid of the "-i" option for mount.nfs[4] and always use the text-based mount(2) system call for kernel version 2.6.23 and later." -- in other words, the bug is specific to the text interface. Does anyone know why this would be the case?

/* Steinar */

Signature · 1 line
Homepage: http://www.sesse.net/
#2 lore raw
In reply to a message not in the archive

Hi Steinar-

Quoted 6 lines · Steinar H. Gunderson

I've gotten a few bug reports lately that if UDP is blocked at the server, NFS mounts (even with -o tcp) fail. git bisect shows that the culprit is the change "Get rid of the "-i" option for mount.nfs[4] and always use the text-based mount(2) system call for kernel version 2.6.23 and later." -- in other words, the bug is specific to the text interface. Does anyone know why this would be the case?

I specifically tested that when developing the text-based mount interface, so if it doesn't work, it's a bug or a missed corner case.

Start with "mount.nfs -v" to see if the mount command tells you anything interesting. Then do a raw network trace to see what's failing during the mount version/protocol negotiation.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

author
#3 lore raw
Quoted 3 lines · Chuck Lever

Start with "mount.nfs -v" to see if the mount command tells you anything interesting. Then do a raw network trace to see what's failing during the mount version/protocol negotiation.

There's nothing in -v except that it adds an addr= field. As for the packet dump, that's the most interesting part -- for my part, the server doesn't have iptables, so I'm blocking on the client. In that case, exactly zero packets enter the write before mount gives up...

The other submitters have iptables blocking on the server, though, and it doesn't work for them either (one is using NFSv3, the other NFSv4). Is this really working for you?

/* Steinar */

Signature · 1 line
Homepage: http://www.sesse.net/
#5 lore raw
In reply to a message not in the archive
Quoted 8 lines · Steinar H. Gunderson
Quoted 3 lines · Chuck Lever

Start with "mount.nfs -v" to see if the mount command tells you anything interesting. Then do a raw network trace to see what's failing during the mount version/protocol negotiation.

There's nothing in -v except that it adds an addr= field. As for the
packet dump, that's the most interesting part -- for my part, the server
doesn't have iptables, so I'm blocking on the client. In that case, exactly zero packets enter the write before mount gives up...

The kernel may be returning an incorrect return code to the mount command.

Set the NFSDBG_MOUNT flag before you try your mount, then look in /var/log/messages. That will show you the kernel part of the mount process. And/or use strace with the mount command.

Quoted 2 lines

The other submitters have iptables blocking on the server, though, and it
doesn't work for them either (one is using NFSv3, the other NFSv4). Is this really working for you?

I can't say until you post a complete description of a specific test case.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

author
#6 lore raw
Quoted 5 lines · Chuck Lever
Quoted 3 lines

The other submitters have iptables blocking on the server, though, and it doesn't work for them either (one is using NFSv3, the other NFSv4). Is this really working for you?

I can't say until you post a complete description of a specific test case.

On the client:

iptables -A OUTPUT -d 10.0.0.10 -p udp -j DROP
mount -t nfs 10.0.0.10:/foo /bar

Substitute 10.0.0.10 with the server, of course.

/* Steinar */

Signature · 1 line
Homepage: http://www.sesse.net/
#8 lore raw
In reply to a message not in the archive
Quoted 7 lines · Steinar H. Gunderson
Quoted 5 lines · Chuck Lever
Quoted 3 lines

The other submitters have iptables blocking on the server, though, and it doesn't work for them either (one is using NFSv3, the other NFSv4). Is this really working for you?

I can't say until you post a complete description of a specific test case.

Quoted 6 lines

On the client:

iptables -A OUTPUT -d 10.0.0.10 -p udp -j DROP
mount -t nfs 10.0.0.10:/foo /bar

Substitute 10.0.0.10 with the server, of course.

I just tried this specific use case.

Because of the local packet filtering on the client, the kernel's RPC client is getting -EPERM when trying to send the initial rpcbind request.

As far as I can see, nothing in the RPC client knows how to deal specifically with that error code, so it punts the request, and the mount fails.

When I originally tested mount protocol/version negotiation, I used only server-side filtering.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

author
#9 lore raw
Quoted 6 lines · Chuck Lever

I just tried this specific use case.

Because of the local packet filtering on the client, the kernel's RPC client is getting -EPERM when trying to send the initial rpcbind request.

Quoted 6 lines

As far as I can see, nothing in the RPC client knows how to deal specifically with that error code, so it punts the request, and the mount fails.

When I originally tested mount protocol/version negotiation, I used only server-side filtering.

I checked the original bug report, and indeed it seems that the blocking was on the client there as well. I'll ask the user to check what happens when the filtering is done server-side instead.

There's also http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=476128, but I believe that's a different bug.

/* Steinar */

Signature · 1 line
Homepage: http://www.sesse.net/
author
#11 lore raw
In reply to a message not in the archive
Quoted 3 lines · Steinar H. Gunderson

I checked the original bug report, and indeed it seems that the blocking was on the client there as well. I'll ask the user to check what happens when the filtering is done server-side instead.

Hm, no, there's also reports of blocking on a firewall in the middle; see

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=473171

/* Steinar */

Signature · 1 line
Homepage: http://www.sesse.net/
#17 lore raw
In reply to a message not in the archive
Quoted 8 lines · Steinar H. Gunderson

Hi,

I've gotten a few bug reports lately that if UDP is blocked at the server, NFS mounts (even with -o tcp) fail. git bisect shows that the culprit is the change "Get rid of the "-i" option for mount.nfs[4] and always use the text-based mount(2) system call for kernel version 2.6.23 and later." -- in other words, the bug is specific to the text interface. Does anyone know why this would be the case?

This also fails for me. My setup is through an ssh tunnel
ssh -L2049:srv:2049 -L2050:srv:2050 -L2051:srv:2051ser@gateway

and I follow that with

   strace mount localhost:/export /mountpoint \
             -o nfsvers=3,port=2049,mountport=2050,...

This hangs in this kernel call
mount("localhost:/export", "/mountpoint", "nfs",
MS_NOSUID|MS_NOATIME, "tcp,intr,fg,soft,nfsvers=3,port=2049,mountport=2050,nolock,addr=127.0.0.1"

This hack makes it work for me again:

utils/mount/mount.c +1 -1
diff -ru nfs-utils-1.1.2/utils/mount/mount.c nfs-utils-1.1.2-changed/utils/mount/mount.c
@@ -191,7 +191,8 @@
 		nfs_mount_data_version = NFS_MOUNT_VERSION;
 	else
 		if (kernel_version > MAKE_VERSION(2, 6, 22))
-			string++;
+			string=0;
 }
 
 static void print_one(char *spec, char *node, char *type, char *opts)

Frank

#20 lore raw
In reply to a message not in the archive
Quoted 6 lines · Steinar H. Gunderson

I've gotten a few bug reports lately that if UDP is blocked at the server, NFS mounts (even with -o tcp) fail. git bisect shows that the culprit is the change "Get rid of the "-i" option for mount.nfs[4] and always use the text-based mount(2) system call for kernel version 2.6.23 and later." -- in other words, the bug is specific to the text interface. Does anyone know why this would be the case?

The NFS client code (nfs-utils-1.1.2) tests for
if (kernel_version > MAKE_VERSION(2, 6, 22))

yet the kernel patch for NFS did not go un until 2.6.25-rc2

Quoted 4 lines

From: Linus Torvalds <torvalds@...>
Subject: Linux 2.6.25-rc2
Date: Feb 15, 5:23 pm 2008

Quoted 8 lines

Ok,
this kernel is a winner.

[...]

Chuck Lever (1):
NFS: Allow text-based mounts via compat_sys_mount

So the test should at least be > MAKE_VERSION(2, 6, 24).

Frank

#22 lore raw
In reply to a message not in the archive
Quoted 22 lines · Frank A. Kingswood
Quoted 6 lines · Steinar H. Gunderson

I've gotten a few bug reports lately that if UDP is blocked at the server, NFS mounts (even with -o tcp) fail. git bisect shows that the culprit is the change "Get rid of the "-i" option for mount.nfs[4] and always use the text-based mount(2) system call for kernel version 2.6.23 and later." -- in other words, the bug is specific to the text interface. Does anyone know why this would be the case?

The NFS client code (nfs-utils-1.1.2) tests for
if (kernel_version > MAKE_VERSION(2, 6, 22))

yet the kernel patch for NFS did not go un until 2.6.25-rc2

Quoted 8 lines

From: Linus Torvalds <torvalds@...>
Subject: Linux 2.6.25-rc2
Date: Feb 15, 5:23 pm 2008
Ok,
this kernel is a winner.
[...]
Chuck Lever (1):
NFS: Allow text-based mounts via compat_sys_mount

Quoted 1 line

So the test should at least be > MAKE_VERSION(2, 6, 24).

Frank, the patch you refer to fixes a bug with the compat_sys_mount interface, which is separate from the sys_mount interface, and used only on a handful of hardware platforms (SPARC?).

The original NFS text-based mount support was added in 2.6.23.

I don't have a problem with adjusting MAKE_VERSION in the master nfs-utils repo as you describe, since that bug does break text-based mounts on those few hardware platforms that use compat_sys_mount instead of sys_mount. It might affect some early adopters of NFS over RDMA or FS cache, however.

On distributions that don't support NFS over RDMA or the FS cache facility, it should be harmless for now to make the mount command use the legacy mount API even on newer kernels. Simply adjust the MAKE_VERSION() macro as needed, or disable it entirely as you did with your "string = 0" hack.

New features will be added only to the text-based mount interface, however, so eventually everyone will need to use the text-based mount interface on new kernels.

Because we have little documented history and only a handful of use cases and unit tests for the mount command, it's important for everyone to test the new API and report problems here so we can address them. NFS mount is complex and has many subtle and hidden historical behaviors.

Even if 70-80% of the common use cases are working properly, there are going to be outliers that will experience issues until we can get the corner cases fixed.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#15 lore raw
In reply to a message not in the archive
Quoted 11 lines · Steinar H. Gunderson
Quoted 10 lines · Chuck Lever

I just tried this specific use case.

Because of the local packet filtering on the client, the kernel's RPC client is getting -EPERM when trying to send the initial rpcbind request.

As far as I can see, nothing in the RPC client knows how to deal specifically with that error code, so it punts the request, and the mount fails.

Quoted 7 lines
Quoted 2 lines

When I originally tested mount protocol/version negotiation, I used only server-side filtering.

I checked the original bug report, and indeed it seems that the blocking was
on the client there as well. I'll ask the user to check what happens when the filtering is done server-side instead.

There's also http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=476128, but I believe that's a different bug.

FYI: Steve Dickson (nfs-utils maintainer) is already looking at removing EACCES from the list of temporary errors, so yes, we're aware of that issue.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#13 lore raw
In reply to a message not in the archive
Quoted 8 lines · Steinar H. Gunderson
Quoted 3 lines · Steinar H. Gunderson

I checked the original bug report, and indeed it seems that the blocking was on the client there as well. I'll ask the user to check what happens when the filtering is done server-side instead.

Hm, no, there's also reports of blocking on a firewall in the middle; see

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=473171

The user is correct that the mount process for NFSv4 should not attempt to use the portmapper unless the port=0 option is specified. I will look into the problem.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#23 lore raw
In reply to Chuck Lever
Quoted 7 lines · Chuck Lever
Quoted 6 lines

The NFS client code (nfs-utils-1.1.2) tests for
if (kernel_version > MAKE_VERSION(2, 6, 22))

yet the kernel patch for NFS did not go un until 2.6.25-rc2 So the test should at least be > MAKE_VERSION(2, 6, 24).

The original NFS text-based mount support was added in 2.6.23.

OK.

Quoted 1 line

New features will be added only to the text-based mount interface, however, so eventually everyone will need to use the text-based mount interface on new kernels.

That sounds like a good idea.

Quoted 1 line

Because we have little documented history and only a handful of use cases and unit tests for the mount command, it's important for everyone to test the new API and report problems here so we can address them. NFS mount is complex and has many subtle and hidden historical behaviors.

My system fails with 2.6.23 and 2.6.25 kernels. As I described in my original mail, I only have TCP access to the server, and only through an ssh tunnel.

Would you like me to try and produce some debug traces, or do you think you can find the problem with the information you have already? I've seen quite a few reports of this now.

Frank

Signature · 3 lines
------------------------------------------------------------------------
Frank A. Kingswood                      frank-YzJE6wl5KmBjuoFY8sQqqY5Lr7LD32fptUK59QYPAWc@public.gmane.org
Cambridge, United Kingdom                               +44-7943-217 571
#25 lore raw
In reply to a message not in the archive
Quoted 7 lines · Frank A. Kingswood
Quoted 6 lines · Chuck Lever
Quoted 6 lines

The NFS client code (nfs-utils-1.1.2) tests for
if (kernel_version > MAKE_VERSION(2, 6, 22))

yet the kernel patch for NFS did not go un until 2.6.25-rc2 So the test should at least be > MAKE_VERSION(2, 6, 24).

Quoted 8 lines
Quoted 1 line

The original NFS text-based mount support was added in 2.6.23.

OK.

Quoted 1 line

New features will be added only to the text-based mount interface, however, so eventually everyone will need to use the text-based mount interface on new kernels.

That sounds like a good idea.

Quoted 1 line

Because we have little documented history and only a handful of use cases and unit tests for the mount command, it's important for everyone to test the new API and report problems here so we can address them. NFS mount is complex and has many subtle and hidden historical behaviors.

My system fails with 2.6.23 and 2.6.25 kernels. As I described in my original mail, I only have TCP access to the server, and only through an ssh tunnel.

Would you like me to try and produce some debug traces, or do you think you can find the problem with the information you have already? I've seen quite a few reports of this now.

Thanks for the offer. I tested against a server configuration that dropped UDP packets, but I never tested through an ssh tunnel.

Network traces might be a little hard with a tunnel, but give it a shot. The best information comes from a full-packet raw trace. Use the "-s0" option on tcpdump.

You can also trace NFS mount processing with:

# sysctl -w sunrpc.nfs_debug = 128

Or trace all RPC activity (including rpcbind and RPC network transport errors) with:

# sysctl -w sunrpc.rpc_debug = 32767

You can set both of these at the same time. Trace output will appear in /var/log/messages. To disable tracing when you're done, set these sysctls to zero.

You can post these to me privately.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#14 lore raw
In reply to Chuck Lever
Quoted 11 lines · Chuck Lever
Quoted 8 lines · Steinar H. Gunderson
Quoted 3 lines · Steinar H. Gunderson

I checked the original bug report, and indeed it seems that the blocking was on the client there as well. I'll ask the user to check what happens when the filtering is done server-side instead.

Hm, no, there's also reports of blocking on a firewall in the middle; see

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=473171

The user is correct that the mount process for NFSv4 should not attempt to use the portmapper unless the port=0 option is specified. I will look into the problem.

This issue may be addressed as a side-effect of:

commit f22d6d79fe227245363a8849ea8c85fe6c6598c3

Author: Chuck Lever <[email protected]>
Date:   Fri Mar 14 14:10:22 2008 -0400

NFS: Save the value of the "port=" mount option

During a remount based on the mount options displayed in /proc/mounts, we want to preserve the original behavior of the mount request. Let's save the original setting of the "port=" mount option in the mount's nfs_server structure.

Please give 2.6.25 final a try to see if it prevents the noted extraneous rpcbind activity during an NFSv4 mount.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#16 lore raw
In reply to Chuck Lever
Quoted 20 lines · Chuck Lever
Quoted 19 lines · Steinar H. Gunderson
Quoted 6 lines · Chuck Lever

I just tried this specific use case.

Because of the local packet filtering on the client, the kernel's RPC client is getting -EPERM when trying to send the initial rpcbind request.

Quoted 6 lines

As far as I can see, nothing in the RPC client knows how to deal specifically with that error code, so it punts the request, and the mount fails.

When I originally tested mount protocol/version negotiation, I used only server-side filtering.

I checked the original bug report, and indeed it seems that the blocking was
on the client there as well. I'll ask the user to check what happens when the filtering is done server-side instead.

Quoted 7 lines
Quoted 2 lines

There's also http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=476128,
but I believe that's a different bug.

FYI: Steve Dickson (nfs-utils maintainer) is already looking at removing EACCES from the list of temporary errors, so yes, we're aware of that issue.

Actually it has been removed already...

steved.