Skip to contentSkip to conversation

nfs stalls on cp with certain file sizes with jumbo-gigabit

DISCUSSION Martin L. Purschke 2 participants 2 messages Dec 28, 2003

Subsystems (2)
NFS client NFS server
lore
Share & tools
lore

Martin L. Purschke

author
#0 lore raw

Hello,

I have a few linux nfs servers, all identical, dual gigabit NIC's, one on mtu 1500 and one on another jumbo-frame network. I have dozens of nfs clients on the regular network and never had a problem. I recently started to move machines to the jumbo-frame network. I came across a strange problem that a nfs client stalls when cp'ing a file with a certain file size. (There are many other ways to get the nfs client to stall that has so far prevented me from deploying machines on the J-network, but this one is easily reproducible).

I tracked down a file that would cause the problem. I made another file with the exact same size and could reproduce the error (so it's not the particular file, but the size that does it). The size happens to be 4698 bytes, here is a strace of the cp command of the special-made file with that exact length:

Quoted 15 lines

[root@va033 tmp]# strace cp /root/4698.dat . execve("/bin/cp", ["cp", "/root/4698.dat", "."], [/* 45 vars */]) = 0 uname({sys="Linux", node="va033.phenix.bnl.local", ...}) = 0

....

lstat64(".", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0 stat64(".", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0 stat64("/root/4698.dat", {st_mode=S_IFREG|0644, st_size=4698, ...}) = 0 stat64("./4698.dat", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 open("/root/4698.dat", O_RDONLY|O_LARGEFILE) = 3

fstat64(3, {st_mode=S_IFREG|0644, st_size=4698, ...}) = 0
open("./4698.dat", O_WRONLY|O_TRUNC|O_LARGEFILE) = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=4698, ...}) = 0
Quoted 6 lines

read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 32768) = 4698
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4698) = 4698
read(3, "", 32768) = 0
close(4

and there is hangs indefinitely. I have no problem doing the exact same thing on the mtu=1500 network, in fact, I dropped the jumbo network down to 1500 and the same operation works.

Both client and server run kernel 2.4.22 with SMP and high-mem nfs is: type nfs (rw,nfsvers=3,hard,wsize=32768,rsize=32768,addr=192.168.30.6) I played with the sizes, no effect.

Does this ring a bell? I looked in this list and saw some of Trond's patches, but the errors they address were different, and I'm not sure if I need to look in the network drivers instead. I'm planning to install the patches as advised in
http://sourceforge.net/mailarchive/message.php?msg_id=6283173
but these are production machines, and the patches are tagged "experimental". Any advice is highly appreciated.

Thanks,
Martin

Signature · 19 lines
Martin L. Purschke               ;   [email protected]
                                 ;   http://www.phenix.bnl.gov/~purschke
                                 ;
Brookhaven National Laboratory   ;   phone: +1-631-344-5244
Physics Department Bldg 510 C    ;   fax:   +1-631-344-3253
Upton, NY 11973-5000             ;
-----------------------------------------------------------------------




-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#1 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Martin L Purschke <[email protected]> writes:

> Hello,

> I have a few linux nfs servers, all identical, dual gigabit > NIC's, one on mtu 1500 and one on another jumbo-frame > network. I have dozens of nfs clients on the regular network > and never had a problem. I recently started to move machines to > the jumbo-frame network. I came across a strange problem that a > nfs client stalls when cp'ing a file with a certain file > size. (There are many other ways to get the nfs client to stall > that has so far prevented me from deploying machines on the > J-network, but this one is easily reproducible).

This is not an NFS problem. It sounds rather as if jumbo frames is poorly supported for your hardware/driver combination (a common problem).

You'll have better luck reporting it to the networking mailing lists.

Cheers,
Trond

-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs