2.4.20rc1 NFS V3 UDP client problems

I'm seeing some 2.4.20rc1 client problems.

25 seconds to write a 1G file
13 minutes to read that file back

CLIENT 2.4.20rc1 NFSV3 UDP HIGHMEM SUPPORT ON
gigE card with jumbo frames enabled
Tested with both netgear and intel -- same result

SERVER
Netapps 825 ontap 6.3
gigE card with jumbo frames enabled

private point-to-point connection.

Is this a HIGHMEM problem? Or is it related to jumbo frames? I'll test with both disabled.

I'm seeing network errors on the client -- dropped and overruns are both the same.

Anything I should try?

eric

# time dd if=/dev/zero of=file1 bs=1G count=1
1+0 records in
1+0 records out

real    0m25.881s
user    0m0.000s
sys     0m4.220s

(run a program to calloc 1G of RAM and clear some buffer cache)

# time dd if=file1  of=/dev/null

2097152+0 records in
2097152+0 records out

real    13m7.445s
user    0m0.370s
sys     0m2.160s

 # ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:02:B3:AC:52:F5
          inet addr:10.0.0.2  Bcast:10.255.255.255  Mask:255.0.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:1534585 errors:10240 dropped:10240 overruns:10240
frame:0
          TX packets:5332748 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:2069590604 (1973.7 Mb)  TX bytes:587855320 (560.6 Mb)
          Interrupt:12 Base address:0xc000 Memory:ea020000-ea040000

-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing your web site with SSL, click here to get a FREE TRIAL of a Thawte Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

I disabled jumbo frames and I'm seeing 70M/s reads (with a clean cache)..

There must be a UDP or NFS client problem with the 9000 mtu packets.

eric

-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing your web site with SSL, click here to get a FREE TRIAL of a Thawte Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

> I disabled jumbo frames and I'm seeing 70M/s reads (with a > clean cache)..

> There must be a UDP or NFS client problem with the 9000 mtu > packets.

NFS neither knows nor cares what the MTU is: all it knows is that it has a socket. I've suspected a bug in the UDP layer for a long time as you are far from being the first person to report this problem when enabling jumbo frames.

Cheers,
Trond

-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing your web site with SSL, click here to get a FREE TRIAL of a Thawte Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

We had to set our MTUs to 8960 for things to work. We've never followed through to see what the issue was though.

We are using SysKonnect cards and 2.4.18 kernel. Note, we haven't recently verified that the problem still exists.

--Shane

-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing your web site with SSL, click here to get a FREE TRIAL of a Thawte Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

I tried various MTU values headed gradually towards 9000. I found that a MTUs between 1500 and 7500 give me the expected performance. MTU values of anything much more than that causes problems given my setup.

eric

-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing your web site with SSL, click here to get a FREE TRIAL of a Thawte Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Which NIC? Some chipsets (like ns83820) only have an 8KB TX fifo and end up doing two passes over the data for larger packet sizes. Also, running gige with a standard 32bit/33MHz bus is pretty close to the limits.

-ben

-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing your web site with SSL, click here to get a FREE TRIAL of a Thawte Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Both of these NICs seemed to have the same behavior. <6>eth0: Intel(R) PRO/1000 Network Connection <6>eth1: NetGear GA620 Gigabit Ethernet at 0xea060000, irq 11

Yes -- I'm running on a 32bit/33Mhz bus right now. I do have a Sun 280R with a 64bit/66Mhz card in a 66Mhz slot that doesn't even seem to support jumbo frames at all.

eric

-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing your web site with SSL, click here to get a FREE TRIAL of a Thawte Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

I'd guess you're hitting the PCI bandwidth limitations then, as the larger packets permit greater throughput, which in turn drives up bus utilizations. If you've got any disk controllers on the same bus, that would definately explain the problem (or any other devices on the bus that are in use).

-ben

-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing your web site with SSL, click here to get a FREE TRIAL of a Thawte Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Ben,

I found a 64Bit/66Mhz PCI bus and the jumbo frames are working fine. Same
kernel, same network cards, same Netapp 825 filer -- simply a different motherboard.

Here are the NFS client read/write speeds for my setup. The write speeds appear to be somewhat limited by the netapps 825 disk/cpu speed. I wish I had a netapps 9xx to test.

TCP UDP
READ 99MB/s 104MB/s
WRITE 36MB/s 49MB/s

UDP looks good with my current setup -- but with the gigE running jumbo frames in the 32 bit PCI bus the UDP was sick on the reads -- 1MB/s.

Read testing is:
while(true);do mount blaa;dd if=path of=/dev/null bs=x count=y;umount blaa;done
where bs/count equal a 500M file -- to use the netapps cache.

Write testing is:
dd if=/dev/zero of=file bs=1M

Network rates monitored using netapps syststat and the time results.

eric

-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing your web site with SSL, click here to get a FREE TRIAL of a Thawte Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs