100% repeatable NFS v3 client hang

DISCUSSION John McCorquodale 2 participants 3 messages Jun 14, 2007

Subsystems (2)

NFS client NFS server

lore

Share & tools

lore

John McCorquodale

author

2007-06-14 16:55 UTC #0 lore raw

Guys,

I've been having a reliable problem with NFS v3 since 2.6.19 (possibly earlier; I didn't test), and just confirmed still present in 2.6.21.5 (x86_64). The problem is that issuing the command:

  $ dd if=/dev/zero of=biggy bs=1G count=50

100% reliably results in a hung client and completely idle server after a few (different each time, usually betwene 5 and 10) GB have been written. No output via dmesg at all. Sometimes, however, when not writing large files, I'll see an occasional dmesg "NFS: desynchronized value of nfs_i.ncommit." It proceeds at about 30MB/s until the client eats it, and then it never wakes up again. The roof fs is mounted via nfs v3, so once it eats it I can't log on or anything. Serial console outputs no information during/after the eat-it condition, so it's not a panic. Pings continue. Feels deadlocky.

I saw an old dicussion/patch about something that sounded like my problem (but the old kernel+patch didn't fix my problem) here:
http://lkml.org/lkml/2007/4/21/112

This problem has been reliable and present for months for me, and I assumed that it was so obvious that anybody would see it immediately, and thus concluded that 'nobody was working on v3 anymore'. Enough eyebrows were raised by that comment by my most on the v4 list this morning:
http://linux-nfs.org/pipermail/nfsv4/2007-June/006183.html
that I am forced to re-evaluate that conclusion -- it appears likely that this may be something unique to my hardware.

The only thing at all weird about my platform is that the servers (not the clients) are nvidia forcedeth gigE, which has never inspired my confidence. The forcedeth in 2.6.21 seems fine, 'tho, as it regularly sends >800GB via nc for backups without hiccup or incident.

Anyway, last bit of data is that I just switched to nfs4 and the problem is gone. I have successfully dd'd four 50G files-of-zeros simultaneously with no ill effects.

Anyway, I have a shadow server/client that I could conceivably bring v3 up on and run tests with if you guys would like to use me as a debugging agent or would like something to log in to.

I was going to ignore this, but when Trond's eyebrows went up on the v4 list, I figured I ought to get off my duff and report it.

Cheers,

-mcq

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Trond Myklebust

2007-06-14 18:03 UTC #1 lore raw

Quoted 38 lines · John McCorquodale

Guys,
I've been having a reliable problem with NFS v3 since 2.6.19 (possibly earlier; I didn't test), and just confirmed still present in 2.6.21.5 (x86_64). The problem is that issuing the command:
  $ dd if=/dev/zero of=biggy bs=1G count=50
100% reliably results in a hung client and completely idle server after a few (different each time, usually betwene 5 and 10) GB have been written. No output via dmesg at all. Sometimes, however, when not writing large files, I'll see an occasional dmesg "NFS: desynchronized value of nfs_i.ncommit." It proceeds at about 30MB/s until the client eats it, and then it never wakes up again. The roof fs is mounted via nfs v3, so once it eats it I can't log on or anything. Serial console outputs no information during/after the eat-it condition, so it's not a panic. Pings continue. Feels deadlocky.
I saw an old dicussion/patch about something that sounded like my problem (but the old kernel+patch didn't fix my problem) here:
http://lkml.org/lkml/2007/4/21/112
This problem has been reliable and present for months for me, and I assumed that it was so obvious that anybody would see it immediately, and thus concluded that 'nobody was working on v3 anymore'. Enough eyebrows were raised by that comment by my most on the v4 list this morning:
http://linux-nfs.org/pipermail/nfsv4/2007-June/006183.html
that I am forced to re-evaluate that conclusion -- it appears likely that this may be something unique to my hardware.
The only thing at all weird about my platform is that the servers (not the clients) are nvidia forcedeth gigE, which has never inspired my confidence. The forcedeth in 2.6.21 seems fine, 'tho, as it regularly sends >800GB via nc for backups without hiccup or incident.
Anyway, last bit of data is that I just switched to nfs4 and the problem is gone. I have successfully dd'd four 50G files-of-zeros simultaneously with no ill effects.

Quoted 6 lines

Anyway, I have a shadow server/client that I could conceivably bring v3 up on and run tests with if you guys would like to use me as a debugging agent or would like something to log in to.
I was going to ignore this, but when Trond's eyebrows went up on the v4 list, I figured I ought to get off my duff and report it.

The write() code had to go through emergency surgery in the 2.6.20 timeframe, since the VM/MM code was becoming increasingly intolerant of our aging asynchronous write design. This didn't affect just NFSv3, it affected NFSv2 and v4 too since it was in generic code.

2.6.21 still had some problems, but 2.6.22-rcX has fixes for all reported bugs and should be stable. At least I haven't heard of any further issues.

If you can reproduce the hang, and send me the output from an 'echo t

Quoted 1 line

/proc/sysrq-trigger', then I'd be happy to look at the resulting dump.

Don't forget to specify a large buffer size when you use dmesg: I tend to use 'dmesg -s 90000'.
It would also be nice if you could try to reproduce the problem on a 2.6.22-rc4 kernel, to see if it is still there.

Cheers
Trond

John McCorquodale

author

2007-06-14 18:09 UTC #2 lore raw

Quoted 10 lines

2.6.21 still had some problems, but 2.6.22-rcX has fixes for all reported bugs and should be stable. At least I haven't heard of any further issues.
If you can reproduce the hang, and send me the output from an 'echo t
Quoted 1 line
/proc/sysrq-trigger', then I'd be happy to look at the resulting dump.
Don't forget to specify a large buffer size when you use dmesg: I tend to use 'dmesg -s 90000'.
It would also be nice if you could try to reproduce the problem on a 2.6.22-rc4 kernel, to see if it is still there.

Will do as you instruct. It'll probably be the weekend before I can get the shadow systems running for our experiments. I was running 2.6.22-rc3 at one point and saw the problem, so we'll probably get something interesting from the sysrq.

Thanks for the precise insturctions; that's about the right level of detail for my inexperience doing this sort of thing. :)

Cheers,

-mcq