Skip to contentSkip to conversation

ReasmFails increases / NFS performance on Linux Cluster

DISCUSSION Patrice Seyed 2 participants 3 messages Aug 14, 2004 – Aug 15, 2004

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

We have 1 ibm x345 as a management node, and 1 ibm x35 as a storage = node,
(both dual cpu 2.8 xeon and 2.5gb memory) which is fiber channeled to a = Raid
5 array with ~900 GB usable. Then we have 10 IBM BladeCenters which = contain
14 blades per chassis except the last one (has 8). So 134 nodes with = dual
xeon at 2.8Ghz, mostly with 1GB ram and 20 with 2GB.

All nodes including x345s are set to 1000/full and cabled into a 3750 Cataylst. Bear in mind each BladeCenter chassis include a switch module, which is also set to 1000/full.

In testing dd writes from /dev/zero to an nfs mount and back in a large number of batch jobs, I'm seeing high load on the storage node and heavy slowdowns for example for ssh login, df, or ls on the head node.=20 In my attempt to tune the storage node, I have tried 32k and now 8k (rsize,wsize in autofs) with no improvement in the slowness. Other parameters I am using are hard,intr,noatime,retrans=3D20,timeo=3D25. I = am
currently running 64 nfsd daemons. Also I now have ipfrag_low_thresh and ipfrag_high_thread both set to 1045876. When I had doubled the default values for these settings a few weeks ago, it appeared to solve I/O = errors
that were appearing in the logs for many of the nodes. Still now the ReasmFails value in /proc/net/snmp steadily increases (for 8k or 32k) = when I
submit a moderate number of jobs (20-40) that are heavy on I/O.

More info (on the storage node):

$ netstat -s | less
Ip:
    226750504 total packets received
    0 forwarded
    501200 incoming packets discarded
    164658619 incoming packets delivered
    161895650 requests sent out
    5006 fragments dropped after timeout
    80794698 reassemblies required
    13723023 packets reassembled ok
    126665 packet reassembles failed
    2858536 fragments received ok
    14006786 fragments created

Udp:
159845714 packets received
142 packets to unknown port received.
501200 packet receive errors
152909713 packets sent

I welcome any suggestions or recommendations.

Regards,
=20
Patrice Seyed
Linux System Administrator - SIG
RHCE, SCSA
Boston University School of Medicine

-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#1 lore raw
Quoted 1 line · Patrice Seyed

I welcome any suggestions or recommendations.

Try using NFS over TCP, or, if all your NICs support it, use jumbograms to avoid fragmentation.

Olaf

Signature · 13 lines
Olaf Kirch     |  The Hardware Gods hate me.
[email protected]   |
---------------+ 


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#2 lore raw

Thanks Olaf, when I tried nfs over tcp on the clients the ReasmFails = value
increased much significantly slower. At this time of testing the value = is
not increasing at all.

Still, slowness on logging in (or running ls on that mount) continues, = which
is not good since many users will be logging into the headnode during production.

In addition, I have enabled jumbo packets on the cisco, and set all = nodes on
the cluster network to mtu 9000. Also I have set read and write size to = 32k
as a document suggested with tcp and mtu at 9000 for better thoroughput. Still I see the slowness when logging in.

I am not sure if there is now any performance improvement as I am not
currently using any intelligent benchmark or mbit/sec analysis. Any recommendation on doing these measurements and benchmarking is = appreciated
greatly. Also I think I should consider the setup and the feasibility of putting this type of load on this cluster (I/O intensive usually calls = for
pvfs or multiple nfs servers, or both). E.g. one of my tests uses dd to write 256mb files to nfs from 28 nodes at once (all going to the storage node), and also running 2 jobs from each node (134 total node) that use = dd
to write 4MB files and also run nbench. I.e. Should an nfs server be = able to
handle this amount of load anyway?

Like I said I'm now running 64 nfs daemons, and the th line looks like = this
now:

th 64 2380739 168.120 128.860 68.350 261.730 184.990 121.680 145.930 = 122.090
178.390 1749.090

Thanks Olaf for your suggestions on tcp and jumbo packets. Now I'm = wondering
what I can use to determine if actually speed performance was improved, = and
if the "bottleneck" I'm seeing now should be expected for my setup. Any comments, further suggestions, appreciated.

Cheers,
-Patrice

-----Original Message-----
From: Olaf Kirch [mailto:[email protected]]=20
Sent: Saturday, August 14, 2004 5:24 AM
To: Patrice Seyed
Cc: [email protected]
Subject: Re: [NFS] ReasmFails increases / NFS performance on Linux = Cluster

Quoted 1 line · Patrice Seyed

I welcome any suggestions or recommendations.

Try using NFS over TCP, or, if all your NICs support it, use jumbograms to avoid fragmentation.

Olaf
--=20
Olaf Kirch | The Hardware Gods hate me.
[email protected] |
---------------+=20

-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs