Client Lockups

DISCUSSION IHE Lists 2 participants 2 messages Dec 13, 2004

Subsystems (2)

NFS client NFS server

lore

Share & tools

lore

IHE Lists

author

2004-12-13 06:08 UTC #0 lore raw

Here's the situation:

We have a multithreaded Java application that does a great deal of IO over NFS. Almost daily, sometimes several times within a single day, we get an application deadlock. The error in the kernel syslog message is:

Dec 11 11:09:05 <hostname> kernel: RPC: error 512 connecting to server

This error repeats numerous times.

The client mount options are:

rw,hard,intr,rsize=8192,wsize=8192,noatime,nfsvers=3,tcp,timeo=600,retrans=2

We have a deadlock monitor that attempts to restart the application by first sending the process a "kill -9", then running the init.d script. Of course, the threads stuck in the D state do not die, so the restart fails. This cycle continues for almost 25 minutes (almost consistently) before it succeeds.

I believe the eventual total deadlock is a problem in the Java virtual machine, but I cannot determine what is causing the problem to begin with. Other applications on the same system do not lockup accessing other NFS mounts.

I've seen this problem on a custom kernel running 2.4.20 with the -aa1 patches and all the other applicable 2.4.20 patches from Trond and Neil. I've also seen it occur on SLES8 kernels 2.4.21-241, 2.4.21-251, and 2.4.21-261. Our systems are somewhat custom, so they are all running nfs-utils 1.0.6, but the base is currently Red Hat 7.3.

I guess my primary question is: this a kernel problem, an nfs-utils problem, or does the problem lie somewhere else?

Any help is greatly appreciated!

-Brian

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Trond Myklebust

2004-12-13 15:51 UTC #1 lore raw

m=E5 den 13.12.2004 Klokka 00:08 (-0600) skreiv IHE Lists:

Quoted 9 lines

Here's the situation:
=20
We have a multithreaded Java application that does a great deal of IO over NFS. Almost daily, sometimes several times within a single day, we get an application deadlock. The error in the kernel syslog message is:
=20
Dec 11 11:09:05 <hostname> kernel: RPC: error 512 connecting to server
<NFS server>

error 512 =3D=3D ERESTARTSYS. That just means that someone interrupted the RPC call: your deadlock monitor would trigger this error when it sends the kill -9.

Quoted 4 lines

I believe the eventual total deadlock is a problem in the Java virtual machine, but I cannot determine what is causing the problem to begin with. Other applications on the same system do not lockup accessing other NFS mounts.

Have you tried using "strace" to figure out which system call is hanging?

Also, which server are you using? The basic RedHat 7.3 certainly would not support tcp on the server side.

Cheers,
Trond

--=20
Trond Myklebust <[email protected]>