Need help with NFSD hang in 2.4.20+NFS_ALL
Subsystems (2)
Everyone,
This morning I arrived at the office to find an NFS server hung up and users whining at me.
It appeared that an updatedb launched from a cron job had gotten hung up. Perhaps it caused some sort of overload, I'm not really sure. System load was over 130, which is about what I expected given that we had 128 nfs daemon threads, all of which were presumably waiting.
Since nothing was responding (couldn't touch the affected disk, couldn't
successfully sync), I tried to start a "graceful" shutdown, and the
exportfs -ua step hung. A strace -p of that process showed it hung up in
nfsservctl(0x4, <some address>, 0)
on the first mount listed in /var/lib/nfs/xtab. Nothing I could do would make it move, and it wouldn't get any closer to shutdown so I had to cycle the power. Joy.
Now, some background info: the NFS shared partition is a loopback-mounted reiserfs partition, the file underlying which rests on a big SW-raid volume. (It's every bit as awful as it sounds.) I don't think NFS is necessarily the culprit here but it did seize up in the most painful way.
There were some messages that looked like they were from lockd in the ring buffer but (I see now) they never got written to the messages file.
Damn.
Does it sound feasible to anyone who might know that the system might have just hiccuped under the load of the updatedb process? That's not exactly good, but I can easily prevent it from running again...
More germane to this list: if I find this hung up again, is there anything I can do to diagnose the problem? I don't know now if changing the value of /proc/sys/sunrpc/nfsd_debug would have any effect, but if someone suggests a good value I will try it.
This is a SMP box.
I'd greatly appreciate any help or suggestions or even questions to try to figure out what is going on.
Signature · 14 lines
Matthew Mitchell Systems Programmer/Administrator [email protected] Geophysical Development Corporation phone 713 782 1234 1 Riverway Suite 2100, Houston, TX 77056 fax 713 782 1829 ------------------------------------------------------- This SF.net email is sponsored by: Etnus, makers of TotalView, The best thread debugger on the planet. Designed with thread debugging features you've never dreamed of, try TotalView 6 free at www.etnus.com. _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs