NFS mount hang (no response to getattr)

PATCH New Jan Kasprzak 1 participant 4 messages Oct 15, 2004 – Oct 20, 2004 1 message not in the archive

Subsystems (3)

Networking NFS client NFS server

lore

Share & tools

lore

Jan Kasprzak

author

2004-10-15 14:48 UTC #0 lore raw

Hello, world!\n

I have ran into the following problem with Linux NFS code. Following
examples are the simplest test (meaningless) cases, the real /etc/exports is more complicated, of course.

The problem is, that in some cases (depending on the contents of
/etc/exports) mounting the NFS volume hangs. The NFS client gets filehandle from mount-daemon, but then it keeps retransmitting the NFS getattr call for the root directory of the to-be-mounted volume, while NFS server (nfsd on port 2049) does not respond. The tcpdump looks like this (first packets omitted, they look the same for both cases).

The non working case:
[...]
15:38:09.512977 IP erinys2.fi.muni.cz.xxx > scylla.fi.muni.cz.nfs: 104 getattr fh 0,1/16777984 15:38:10.622833 IP erinys2.fi.muni.cz.xxx > scylla.fi.muni.cz.nfs: 104 getattr fh 0,1/16777984 [ and the getattr call gets retransmitted ad infinitum ]

The working case:
[...]
15:41:13.010098 IP erinys2.fi.muni.cz.xxx > scylla.fi.muni.cz.nfs: 104 getattr fh 0,1/16777984 15:41:13.010481 IP scylla.fi.muni.cz.nfs > erinys2.fi.muni.cz.xxx: reply ok 112 getattr DIR 40755 ids 0/0 sz 0x1000 15:41:13.010829 IP erinys2.fi.muni.cz.xxx > scylla.fi.muni.cz.nfs: 104 fsinfo fh 0,1/16777984
[ the rest of traffic snipped ]

Note that in both cases, the getattr refers to the same fh. As I said, the difference is in /etc/exports. I have a single non-working case:

/tmp/1                          erinys2.fi.muni.cz(ro)
/tmp/2                          erinys*.fi.muni.cz(ro)
/tmp/3                          *.fi.muni.cz(ro)

(in tcpdump above, I am trying to mount scylla:/tmp/2 to erinys2:/mnt1). I can get mount of /tmp/2 working by any of the three modifications to /etc/exports on scylla (the last one is most strange one):

# Modification 1 - comment out the first line:

#/tmp/1                         erinys2.fi.muni.cz(ro)
/tmp/2                          erinys*.fi.muni.cz(ro)
/tmp/3                          *.fi.muni.cz(ro)

# Modification 2 - comment out the last line:

/tmp/1                          erinys2.fi.muni.cz(ro)
/tmp/2                          erinys*.fi.muni.cz(ro)
#/tmp/3                         *.fi.muni.cz(ro)

# Modification 3 - change "erinys*" to "eriny*" in the middle line:

/tmp/1                          erinys2.fi.muni.cz(ro)
/tmp/2                          eriny*.fi.muni.cz(ro)
/tmp/3                          *.fi.muni.cz(ro)

Any of the above versions of /etc/exports makes mount of /tmp/2 on erinys2 working. The problem may be that all exported directories are on the same physical volume on scylla. I have the following contents of /proc/fs/nfs/exports:

# Version 1.1
# Path Client(Flags) # IPs
/tmp/2 *.fi.muni.cz,erinys*.fi.muni.cz,erinys2.fi.muni.cz(ro,root_squash,sync,wdelay)

So maybe mount-daemon tells kernel something different than is written in /etc/exports.

The NFS server is 2.6.8.1-mm2 running on Fedora core 3 test3 on single-cpu x86-64 box, but I have tried i386 box running FC2 an x86_64 SMP box running RHEL3 too, with different kernels around 2.6.8.1 (vanilla 2.6.8.1, 2.6.7, 2.6.9-rc4). So the problem is probably not dependent on the distribution, compiler, SMPness or architecture. I have not tried 2.4 kernel.

I have tried two NFS clients - one is IRIX 6.5.22, the other is Linux i386 box with FC2 and 2.6.7 kernel. IRIX box uses NFS v2 and v3 over UDP (I have tried -o vers=2 and -o vers=3), Linux box uses NFS v3, but I have tried v2 as well.

I can provide more detailed information on demand. There is definitely a problem either with parsing wild-card export lists in rpc.mountd, or with kernel NFS-damon not responding to legitimate requests.

Does anybody see the problem here? Thanks,

-Yenya

P.S.: I have come with another strange case of /etc/exports: I am exporting
only one directory for multiple wild-carded clients:

/tmp/2 erinys2.fi.muni.cz(ro) *.fi.muni.cz(ro) *.muni.cz(ro) eriny*.fi.muni.cz(ro)

This does not work (retransmits of getattr call), but by deleting
_any_ of the four client descriptions in the above line it is possible to make it working.

Signature · 5 lines

| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
btw, David, I'm wondering about this loop: [...]  Is this
a busy-wait-until-someone-plugs-in-more-ram-chips thing? ;)  --Andrew Morton

A message that is not in the archive (sent off-list or not yet received): [email protected]

Jan Kasprzak

author

2004-10-19 13:36 UTC #2 lore raw

Hello,

I've sent the following to the LKML on Friday, but forgot to
explicitly Cc: the NFS maintainers. So I am forwarding this to the NFS list.

Thanks,

-Yenya

Jan Kasprzak wrote:
: Hello, world!\n
:
: I have ran into the following problem with Linux NFS code. Following
: examples are the simplest test (meaningless) cases, the real /etc/exports
: is more complicated, of course.
:
: The problem is, that in some cases (depending on the contents of
: /etc/exports) mounting the NFS volume hangs. The NFS client gets filehandle
: from mount-daemon, but then it keeps retransmitting the NFS getattr call
: for the root directory of the to-be-mounted volume, while NFS server
: (nfsd on port 2049) does not respond. The tcpdump looks like this (first
: packets omitted, they look the same for both cases).
:
: The non working case:
: [...]
: 15:38:09.512977 IP erinys2.fi.muni.cz.xxx > scylla.fi.muni.cz.nfs: 104 getattr fh 0,1/16777984
: 15:38:10.622833 IP erinys2.fi.muni.cz.xxx > scylla.fi.muni.cz.nfs: 104 getattr fh 0,1/16777984
: [ and the getattr call gets retransmitted ad infinitum ]
:
: The working case:
: [...]
: 15:41:13.010098 IP erinys2.fi.muni.cz.xxx > scylla.fi.muni.cz.nfs: 104 getattr fh 0,1/16777984
: 15:41:13.010481 IP scylla.fi.muni.cz.nfs > erinys2.fi.muni.cz.xxx: reply ok 112 getattr DIR 40755 ids 0/0 sz 0x1000 : 15:41:13.010829 IP erinys2.fi.muni.cz.xxx > scylla.fi.muni.cz.nfs: 104 fsinfo fh 0,1/16777984
: [ the rest of traffic snipped ]
:
: Note that in both cases, the getattr refers to the same fh.
: As I said, the difference is in /etc/exports. I have a single
: non-working case:
:

: /tmp/1                          erinys2.fi.muni.cz(ro)
: /tmp/2                          erinys*.fi.muni.cz(ro)
: /tmp/3                          *.fi.muni.cz(ro)

:
: (in tcpdump above, I am trying to mount scylla:/tmp/2 to erinys2:/mnt1). : I can get mount of /tmp/2 working by any of the three modifications to : /etc/exports on scylla (the last one is most strange one):
:
: # Modification 1 - comment out the first line:

: #/tmp/1                         erinys2.fi.muni.cz(ro)
: /tmp/2                          erinys*.fi.muni.cz(ro)
: /tmp/3                          *.fi.muni.cz(ro)
:                                                                                 
: # Modification 2 - comment out the last line:
: /tmp/1                          erinys2.fi.muni.cz(ro)
: /tmp/2                          erinys*.fi.muni.cz(ro)
: #/tmp/3                         *.fi.muni.cz(ro)
:                                                                                 
: # Modification 3 - change "erinys*" to "eriny*" in the middle line:
: /tmp/1                          erinys2.fi.muni.cz(ro)
: /tmp/2                          eriny*.fi.muni.cz(ro)
: /tmp/3                          *.fi.muni.cz(ro)

:
: Any of the above versions of /etc/exports makes mount of /tmp/2 on erinys2 : working. The problem may be that all exported directories are on the
: same physical volume on scylla. I have the following contents of
: /proc/fs/nfs/exports:
:
: # Version 1.1
: # Path Client(Flags) # IPs
: /tmp/2 *.fi.muni.cz,erinys*.fi.muni.cz,erinys2.fi.muni.cz(ro,root_squash,sync,wdelay) :
: So maybe mount-daemon tells kernel something different than is written
: in /etc/exports.
:
: The NFS server is 2.6.8.1-mm2 running on Fedora core 3 test3 on single-cpu : x86-64 box, but I have tried i386 box running FC2 an x86_64 SMP box running : RHEL3 too, with different kernels around 2.6.8.1 (vanilla 2.6.8.1, 2.6.7, : 2.6.9-rc4). So the problem is probably not dependent on the distribution, : compiler, SMPness or architecture. I have not tried 2.4 kernel.
:
: I have tried two NFS clients - one is IRIX 6.5.22, the other is Linux i386 : box with FC2 and 2.6.7 kernel. IRIX box uses NFS v2 and v3 over UDP (I have : tried -o vers=2 and -o vers=3), Linux box uses NFS v3, but I have tried v2 : as well.
:
: I can provide more detailed information on demand. There is definitely
: a problem either with parsing wild-card export lists in rpc.mountd,
: or with kernel NFS-damon not responding to legitimate requests.
:
: Does anybody see the problem here? Thanks,
:
: -Yenya
:
: P.S.: I have come with another strange case of /etc/exports: I am exporting
: only one directory for multiple wild-carded clients:
:
: /tmp/2 erinys2.fi.muni.cz(ro) *.fi.muni.cz(ro) *.muni.cz(ro) eriny*.fi.muni.cz(ro) :
: This does not work (retransmits of getattr call), but by deleting
: _any_ of the four client descriptions in the above line it is possible
: to make it working.
:
: --

: | Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
: | GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
: | http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |

: btw, David, I'm wondering about this loop: [...] Is this
: a busy-wait-until-someone-plugs-in-more-ram-chips thing? ;) --Andrew Morton

Signature · 22 lines

| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
btw, David, I'm wondering about this loop: [...]  Is this
a busy-wait-until-someone-plugs-in-more-ram-chips thing? ;)  --Andrew Morton

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
btw, David, I'm wondering about this loop: [...]  Is this
a busy-wait-until-someone-plugs-in-more-ram-chips thing? ;)  --Andrew Morton


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Jan Kasprzak

author

2004-10-19 13:42 UTC #3 lore raw

Another similar problem may be the one described in Red Hat
bugzilla id 127521:

	http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=127521

It is also related to wildcard/subnet exports as the one I posted today.

-Yenya

Signature · 15 lines

| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
btw, David, I'm wondering about this loop: [...]  Is this
a busy-wait-until-someone-plugs-in-more-ram-chips thing? ;)  --Andrew Morton


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Jan Kasprzak

author

2004-10-20 14:09 UTC #4 lore raw

PATCH 1/1 : NFS mount hang (no response to getattr)

In reply to Jan Kasprzak

Jan Kasprzak wrote:
: I have ran into the following problem with Linux NFS code. Following
: examples are the simplest test (meaningless) cases, the real /etc/exports : is more complicated, of course.
[...]

I have found why this is happening to me, but I am not sure
about the solution. The problem is, that mountd joins all the host/subnet/wild-card names which match the IP address from all /etc/exports (not only from the appropriate line in exports), and then sends this as a ip->name translation to the kernel via /proc/net/rpc/auth.unix.ip/channel. Now the problem occurs when the "FQDN+aliases+wildcards" part is 50 characters or more. Kernel then replies to write() with EINVAL, which mountd ignores.

So the bug is probably in linux/net/sunrpc/svcauth_unix.c
- the number 50 is hardcoded there several times (ugly, ugly, ...) and it is too low anyway - the _single_componet_ of FQDN can be up to 63 chars, FQDN can be up to 255 chars, and I am not counting possible aliases/wildcards/subnets which mountd decides to add there.

And the bug is in mountd as well, because it should at least
report to syslog when the write to /proc/.../channel returns EINVAL.

I propose the following patch to svcauth_unix.c. It is
relative to 2.6.8.1-mm2. I have additionally made the "class" and "buf" variables static, thus shaving 100 bytes off the kernel stack. NFS maintaners, please verify this and apply.

-Yenya

net/sunrpc/svcauth_unix.c renamed from net/sunrpc/svcauth_unix.c.orig +6 -4

@@ -151,11 +151,13 @@
 }
 
 static struct ip_map *ip_map_lookup(struct ip_map *, int);
+#define DOMAINNAME_MAX  1024    /* FQDN + possible aliases/subnets/wildcards */
+#define CLASS_MAX	50
 static int ip_map_parse(struct cache_detail *cd,
 			  char *mesg, int mlen)
 {
 	/* class ipaddress [domainname] */
-	char class[50], buf[50];
+	static char class[CLASS_MAX], buf[DOMAINNAME_MAX];
 	int len;
 	int b1,b2,b3,b4;
 	char c;

@@ -168,11 +170,11 @@
 	mesg[mlen-1] = 0;
 
 	/* class */
-	len = qword_get(&mesg, class, 50);
+	len = qword_get(&mesg, class, CLASS_MAX);
 	if (len <= 0) return -EINVAL;
 
 	/* ip address */
-	len = qword_get(&mesg, buf, 50);
+	len = qword_get(&mesg, buf, DOMAINNAME_MAX);
 	if (len <= 0) return -EINVAL;
 
 	if (sscanf(buf, "%u.%u.%u.%u%c", &b1, &b2, &b3, &b4, &c) != 4)

@@ -183,7 +185,7 @@
 		return -EINVAL;
 
 	/* domainname, or empty for NEGATIVE */
-	len = qword_get(&mesg, buf, 50);
+	len = qword_get(&mesg, buf, DOMAINNAME_MAX);
 	if (len < 0) return -EINVAL;
 
 	if (len) {

Signature · 5 lines

| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/   Czech Linux Homepage: http://www.linux.cz/ |
btw, David, I'm wondering about this loop: [...]  Is this
a busy-wait-until-someone-plugs-in-more-ram-chips thing? ;)  --Andrew Morton