rpc.mountd stops functioning

Hello,

For a long time we've been experiencing problems with rpc.mountd stopping functioning.

I'm finding it hard to pinpoint the precise circumstances where we encounter a problem, but here is what we have observed:

- A mount request, or a showmount -e, to the server will hang

- The server will print a log message authenticating the mount request

- The server doesn't print log messages that it has authenticated unmount requests.

- Killing rpc.mountd and then restarting it fixes the problem (I've not tried a HUP)

- There is a correlation between the number of mounts being handled (i.e. the number of entries in rmtab) and the frequency of the problem.

- The problem only occurs on servers that are mounted by a significant number of hosts (500-700).

- The more exports on the server, the more frequently the problem occurs.

- It should be noted that clients are mounting subdirectories of a single exported filesystem, so number of rmtab entries > number of xtab entries.

- Clearing the rmtab and restarting nfs (service nfs restart) seems to provide the longest time between failures.

I clicked send a bit fast. Other info:

Running RHEL 3 with 2.4.25 vanilla kernel. nfs-utils-1.0.5

Experienced the problem in the past with Red Hat 7.3 with same 2.4.25 kernel and 2.4.20 kernel + XFS patches + Trond's all patch for 2.4.20. Tried nfs-utils 0.3.3 and 1.0.6

Quoted 30 lines · Johan van den Dorpe

Hello,

For a long time we've been experiencing problems with rpc.mountd stopping functioning.

I'm finding it hard to pinpoint the precise circumstances where we encounter a problem, but here is what we have observed:

- A mount request, or a showmount -e, to the server will hang

- The server will print a log message authenticating the mount request

- The server doesn't print log messages that it has authenticated unmount requests.

- Killing rpc.mountd and then restarting it fixes the problem (I've not tried a HUP)

- There is a correlation between the number of mounts being handled (i.e. the number of entries in rmtab) and the frequency of the problem.

- The problem only occurs on servers that are mounted by a significant number of hosts (500-700).

- The more exports on the server, the more frequently the problem occurs.

- It should be noted that clients are mounting subdirectories of a single exported filesystem, so number of rmtab entries > number of xtab entries.

- Clearing the rmtab and restarting nfs (service nfs restart) seems to provide the longest time between failures.

For a long time we've been experiencing problems with rpc.mountd stopping functioning.

From Garrick Staples <[email protected]>:

Quoted 1 line · Johan van den Dorpe

For a long time we've been experiencing problems with rpc.mountd stopping functioning.

Maybe you need the following?

(Neil or someone, could you please apply this? Last I checked it wasn't in nfs-utils cvs.)

--Bruce Fields

From Garrick Staples <[email protected]>:

After mountd handles a cache upcall, we should clear the relevant bits in the fd_set.

Quoted 1 line · Johan van den Dorpe

For a long time we've been experiencing problems with rpc.mountd stopping functioning.

Maybe you need the following?

(Neil or someone, could you please apply this? Last I checked it wasn't in nfs-utils cvs.)

--Bruce Fields

From Garrick Staples <[email protected]>:

After mountd handles a cache upcall, we should clear the relevant bits in the fd_set.

Quoted 2 lines · Johan van den Dorpe

For a long time we've been experiencing problems with rpc.mountd stopping functioning.

Maybe you need the following?

(Neil or someone, could you please apply this? Last I checked it wasn't in nfs-utils cvs.)

--Bruce Fields

Quoted 1 line

From Garrick Staples <[email protected]>:

After mountd handles a cache upcall, we should clear the relevant bits in the fd_set.

---

 nfs-utils-1.0.6-bfields/utils/mountd/cache.c |    1 +
 1 files changed, 1 insertion(+)

diff -puN utils/mountd/cache.c~cache_select_bugfix utils/mountd/cache.c

utils/mountd/cache.c renamed from utils/mountd/cache.c~cache_select_bugfix +1 -0

@@ -315,6 +315,7 @@ int cache_process_req(fd_set *readfds) 
 		    FD_ISSET(fileno(cachelist[i].f), readfds)) {
 			cnt++;
 			cachelist[i].cache_handle(cachelist[i].f);
+			FD_CLR(fileno(cachelist[i].f), readfds);
 		}
 	}
 	return cnt;

_

-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now, one more big change to announce. We are now OSTG- Open Source Technology Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

I don't think this patch is needed (though it wouldn't actually break anything). The loop in svc_run.c starts:

		readfds = svc_fdset;
		cache_set_fds(&readfds);

so readfds is completely initialised at the top of the loop, so clearing something at the bottom (which is essentially what this patch does) should be a no-op.

Am I wrong?

Quoted 24 lines

Quoted 1 line

From Garrick Staples <[email protected]>:

After mountd handles a cache upcall, we should clear the relevant bits in the fd_set.

---

 nfs-utils-1.0.6-bfields/utils/mountd/cache.c |    1 +
 1 files changed, 1 insertion(+)

diff -puN utils/mountd/cache.c~cache_select_bugfix utils/mountd/cache.c

utils/mountd/cache.c renamed from utils/mountd/cache.c~cache_select_bugfix +1 -0

@@ -315,6 +315,7 @@ int cache_process_req(fd_set *readfds) 
 		    FD_ISSET(fileno(cachelist[i].f), readfds)) {
 			cnt++;
 			cachelist[i].cache_handle(cachelist[i].f);
+			FD_CLR(fileno(cachelist[i].f), readfds);
 		}
 	}
 	return cnt;

_

Quoted 8 lines

-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now, one more big change to announce. We are now OSTG- Open Source Technology Group. Come see the changes on the new OSTG site. www.ostg.com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Quoted 4 lines

I don't think this patch is needed (though it wouldn't actually break anything). The loop in svc_run.c starts:

Quoted 8 lines

		readfds = svc_fdset;
		cache_set_fds(&readfds);

so readfds is completely initialised at the top of the loop, so clearing something at the bottom (which is essentially what this patch does) should be a no-op.

Am I wrong?

Note that cache_process_req() (which the patch below modifies) is not all the way at the bottom of the loop; there's still an importan call to svc_getreqset() after it.

If I understand the problem correctly, what happens is that if select() returns with a set that includes two file descriptors, the first for a cache channel and the second for an rpc socket, and if we don't clear the first file descriptor from that set before passing it to svc_getreqset(), then the rpc code will try to read an rpc request from a cache channel file, and will get very confused.

The symptom is that mountd may die occasionally on a server that's using the new interface (hence mountd is handling upcalls), and that's getting a lot of mount requests.

Quoted 1 line · Neil Brown

I don't think this patch is needed (though it wouldn't actually bre=

Quoted 8 lines

The loop in svc_run.c starts:
=20
=20
=09=09readfds =3D svc_fdset; =09=09cache_set_fds(&readfds);
=20
so readfds is completely initialised at the top of the loop, so clearing something at the bottom (which is essentially what this pa=

Quoted 3 lines

does) should be a no-op.
=20
Am I wrong?

=20
Note that cache_process_req() (which the patch below modifies) is not=

all the way at the bottom of the loop; there's still an importan call=

svc_getreqset() after it.
=20
If I understand the problem correctly, what happens is that if select=

returns with a set that includes two file descriptors, the first for =

cache channel and the second for an rpc socket, and if we don't clear=

the first file descriptor from that set before passing it to svc_getreqset(), then the rpc code will try to read an rpc request fr=

a cache channel file, and will get very confused.