nfsd: terminating on error 104 problem

Hi all

We are currently using quite a number of HP DL380 servers within our company that use the 2.4.25 kernel. These are primarily used for heavy NFS access, so we keep a large number of nfsd processes concurrently running. We have noticed over time however that single instances of nfsd processes periodically die. From inspection of the system logs, we get numerous entries:

Feb 22 12:25:24 ps29 kernel: nfsd: recvfrom returned errno 104
Feb 22 12:25:24 ps29 kernel: nfsd: terminating on error 104

At the moment we cron a script that counts the number of nfsds and restart rpc.nfsd if they drop below a threshold. Although this is a working solution, it's not ideal and we would really like to get his problem patched up properly.

So from my limited knowledge of the kernel source I can see that "terminating on error 104" corresponds to line 221 of /usr/src/linux-2.4.25/fs/nfsd/nfssvc.c. So svc_recv on line 191 is obviously returning -104.

I've noticed that in the 2.6 kernel there are quite a few changes to nfssvc.c, and I wondered if they dealt with this situation.

In the mean time, are there any quick hacks I could add to nfssvc.c to make it tolerate error -104? Could I safely alter the main request loop to simply continue execution if svc_recv returns this code?

Any help would be much appreciated.

Many thanks,

Johan van den Dorpe


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

104 is ECONNRESET, this means the client reset the connection. The server should really clean up the socket in this case.

Does the attached patch help?

Beware - totally untested, better not try this on a production machine :)

Olaf

Olaf Kirch     |  Stop wasting entropy - start using predictable
[email protected]   |  tempfile names today!
---------------+ 

@@ -904,6 +904,10 @@
 	if (len == -EAGAIN) {
 		dprintk("RPC: TCP recvfrom got EAGAIN\n");
 		svc_sock_received(svsk);
+	} else if (len == -ECONNRESET) {
+		dprintk("RPC: TCP recvfrom got ECONNRESET\n");
+		svc_sock_received(svsk);
+		set_bit(SK_DEAD, &svsk->sk_flags);
 	} else {
 		printk(KERN_NOTICE "%s: recvfrom returned errno %d\n",
 					svsk->sk_server->sv_name, -len);

Thanks very much for this... I'm going to run a test with this patch and get back to you with results after.

Johan van den Dorpe


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

I tried out the patch today, rebooted with the new kernel at 9:42am

Then, we started getting these messages (which we were having before but didn't cause any noticable problems)

Mar  5 10:05:00 ps30 kernel: nfsd: peername failed (err 107)!

After a few hours, we started getting the terminating on error 104 errors:

Mar  5 13:56:49 ps30 kernel: nfsd: terminating on error 104

Until finally the server crashed, refusing input from the console. There were no messages on the console apart from the above nfsd errors. This is the last syslog entry

Mar  5 17:10:10 ps30 rpc.mountd: Caught signal 15, un-registering and exiting.
Mar  5 17:10:10 ps30 nfs: rpc.mountd shutdown succeeded

While the server was up we recieved 112 peername failed (err 107) messages, and 5 terminating on error 104 messages.

If you want any more info please tell me what it is you need.

Otherwise have a good weekend!

Thanks,
Johan.

Johan van den Dorpe


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs