Performance Difference Between Linux NFS Server and Netapp
Subsystems (2)
I've been doing a little limited benchmarking of NFS performance, and doing a comparison between Linux NFS servers and a Netapp. All tests done over gigabit, clients are Linux (i386 and Opteron), OS is Fedora Core 2,
kernel is 2.6.10-1.771_FC2smp, filesystem is Reiser. I basically copy a 205
megabyte file to the server three times and average the results. Sure, limited,
but it seems to have real world relevance here.
The difference in performance is kind of startling. I had originally put the Linux-based servers (with a Nexsan Atabeast backend) in as a replacement for an older EMC Celerra and a few older Suns. On both the Suns and Celerra I would see single-client performance topping out at between 120 and 150 megabits per second.
The Linux setup (Supermicro Dual 2.6 Ghz Xeons with 4 gigabytes of RAM, recently boosted to 8 gigabytes of RAM) regularly returns about 300 megabits per second. Given the difference in price between the Linux boxes and the Celerra, everyone's pretty happy ;)
However, recently I've been trying the same testing between Linux clients and a Netapp, and can achieve between 500 and 600 megabits per second! Darned impressive.
I realize that the disk subsystem of the Netapp is going to stomp on my Atabeast, but neither 300 nor 600 megabits is anywhere near the speed I can get if I run the same test locally on the Linux server boxes, which can regularly top 2000 megabits per second, so I really don't think that the differences in disk architecture should make this much difference.
I guess my question is, are my Linux NFS servers seriously underperforming? Could I expect to achieve better? I realize that my "benchmark" is pretty limited, but the difference is pretty big. Since the same client
on all the Linux boxen gets these results, I'm guessing that the client end has little to do with it.
How does Netapp do it? Don't suppose they'd tell me ...
Hugh
--
Hugh Caley | Unix Systems Administrator | CIS AFFYMETRIX INC. | 6550 Vallejo St. Ste 100 | Emeryville, CA 94608
Tel: 510-428-8537 | [email protected]
Signature · 12 lines
"Brain-eating mutants are bad for business" - Battle Angel Batmensch <[email protected]> ------------------------------------------------------- This SF.Net email is sponsored by the 'Do More With Dual!' webinar happening July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual core and dual graphics technology at this free one hour event hosted by HP, AMD, and NVIDIA. To register visit http://www.hp.com/go/dualwebinar _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Locally disk cache will cause the results to be junk, locally the cache will be used, with NFS the cache will not be used for writes.
With Suse 9 Enterprise and a disk subsystem that will sustain rates of 125MiB/second I can get write rates of 95MiB/second and read rates of 115MiB/second, this is with sustained over 12 minutes of writes.
2000mbit/second is a rate that no ide disk subsystem will obtain, so double check your local performance, if it is fast single disk it will get 50mb/second, if you are using a built-in ata controller, with all channels having fast disks you won't be able exceed 90mb/second no matter what disk you use or how many or how you stripe them, as the build-in mb controllers are not the best, and this is on the more expensive dual-cpu mbs, it may be worse on the single cpu machines.
You will need to use a test larger than the ram size of the machine to get a realistic rate, at least 2x, and probably 4x, and maybe both just to see how things fall over.
Roger
Quoted 64 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Hugh Caley
Sent: Monday, July 11, 2005 5:56 PM
To: [email protected]
Subject: [NFS] Performance Difference Between Linux NFS Server and NetappI've been doing a little limited benchmarking of NFS performance, and doing a comparison between Linux NFS servers and a Netapp. All tests done over gigabit, clients are Linux (i386 and Opteron), OS is Fedora Core 2, kernel is 2.6.10-1.771_FC2smp, filesystem is Reiser. I basically copy a 205
megabyte file to the server three times and average the results. Sure, limited, but it seems to have real world relevance here.The difference in performance is kind of startling. I had originally put the Linux-based servers (with a Nexsan Atabeast backend) in as a replacement for an older EMC Celerra and a few older Suns. On both the Suns and Celerra I would see single-client performance topping out at between 120 and 150 megabits per second.
The Linux setup (Supermicro Dual 2.6 Ghz Xeons with 4 gigabytes of RAM, recently boosted to 8 gigabytes of RAM) regularly returns about 300 megabits per second. Given the difference in price between the Linux boxes and the Celerra, everyone's pretty happy ;)
However, recently I've been trying the same testing between Linux clients and a Netapp, and can achieve between 500 and 600 megabits per second! Darned impressive.
I realize that the disk subsystem of the Netapp is going to stomp on my Atabeast, but neither 300 nor 600 megabits is anywhere near the speed I can get if I run the same test locally on the Linux server boxes, which can regularly top 2000 megabits per second, so I really don't think that the differences in disk architecture should make this much difference.
I guess my question is, are my Linux NFS servers seriously underperforming? Could I expect to achieve better? I realize that my "benchmark" is pretty limited, but the difference is pretty big. Since the same client on all the Linux boxen gets these results, I'm guessing that the client end has little to do with it.
How does Netapp do it? Don't suppose they'd tell me ...
Hugh
--
Hugh Caley | Unix Systems Administrator | CIS AFFYMETRIX INC.| 6550 Vallejo St. Ste 100 | Emeryville, CA 94608 Tel: 510-428-8537 | [email protected]--
"Brain-eating mutants are bad for business" - Battle Angel Batmensch <[email protected]>
Quoted 12 lines
-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!' webinar happening
July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual
core and dual graphics technology at this free one hour event hosted by HP,
AMD, and NVIDIA. To register visit http://www.hp.com/go/dualwebinar
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!' webinar happening July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual core and dual graphics technology at this free one hour event hosted by HP, AMD, and NVIDIA. To register visit http://www.hp.com/go/dualwebinar
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Tue, 12 Jul 2005 at 11:24am, Roger Heflin wrote
Quoted 2 lines
2000mbit/second is a rate that no ide disk subsystem will obtain, so double check your local performance,
I have a 3ware based server that begs to differ with you -- over 150MB/s writing, over 300MB/s reading. That's on a software RAID0 stripe across 2 12 port controllers, with each controller doing hardware RAID5.
Quoted 4 lines
You will need to use a test larger than the ram size of the machine to get a realistic rate, at least 2x, and probably 4x, and maybe both just to see how things fall over.
Absolutely.
Signature · 13 lines
Joshua Baker-LePain Department of Biomedical Engineering Duke University ------------------------------------------------------- This SF.Net email is sponsored by the 'Do More With Dual!' webinar happening July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual core and dual graphics technology at this free one hour event hosted by HP, AMD, and NVIDIA. To register visit http://www.hp.com/go/dualwebinar _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
The Atabeast is an ATA-raid array, but as far as the server knows it is a 2 Gb fibre-attached SCSI disk. I get 800-900 Mb/sec sequential writes using ext3; with Reiser or JFS it is much faster.
Regardless, I don't think this is very relevant. My only point was that the disk subsystem is capable of much better performance than I'm currently seeing over NFS. 300 Mb/sec seems slow to me when I have a Dual-2.6 Ghz Linux box running kernel 2.6.10 (Fedora core 2) attached to a 10 disk RAID5, and the box is doing nothing else but serving this single NFS server.
Hugh
Quoted 23 lines · Joshua Baker-LePain
On Tue, 12 Jul 2005 at 11:24am, Roger Heflin wrote
Quoted 4 lines
2000mbit/second is a rate that no ide disk subsystem will obtain, so double check your local performance,
I have a 3ware based server that begs to differ with you -- over 150MB/s writing, over 300MB/s reading. That's on a software RAID0 stripe across 2 12 port controllers, with each controller doing hardware RAID5.
Quoted 5 lines
You will need to use a test larger than the ram size of the machine to get a realistic rate, at least 2x, and probably 4x, and maybe both just to see how things fall
over.Absolutely.
Signature · 12 lines
"Take off your pants and get in the reptile tank." - Joe Rogan, Fear Factor Hugh Caley [email protected] ------------------------------------------------------- SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line · Hugh Caley
How does Netapp do it? Don't suppose they'd tell me ...
Quite simple really, you pay them loads of money, they give part of it to engineers doing work on nfs tuning.
They generally have 3 performance advantages:
- battery backed cache with full os support, which should make it perform better than battery backed cache on normal raid cards. They use this cache for journaling. I suppose one could try something similar with pci-ram cards with battery and store the journal on these.
- special filesystem which allows writing data on any disk, this allows distributing the workload across the available disks, more disks more performance ( if the fibrechannel can handle it ).
- Highly tuned nfs inplementation with dedicated operating system.
The bruteforce linux approach is quite good for the money spent, but don't expect the features/performance of a way more expensive enterprise solution.
Signature · 14 lines
Sten Spans "There is a crack in everything, that's how the light gets in." Leonard Cohen - Anthem ------------------------------------------------------- SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 8 lines · Sten Spans
Quoted 1 line · Hugh Caley
How does Netapp do it? Don't suppose they'd tell me ...
Quite simple really, you pay them loads of money, they give part of it to engineers doing work on nfs tuning.
Quoted 7 lines
They generally have 3 performance advantages:
- battery backed cache with full os support,
...- special filesystem which allows writing data on any
...
I actually don't think filesystem type is relevant to my question. I tested this out pretty thoroughly with all the journaling filesystems available from the Fedora Core 2 kernels (Ext3, Reiser, XFS, JFS), and although Ext3 had by far the worst local performance, it's performance over NFS with my simple sequential write test was exactly the same as the others. I mean, I'm talking about a single sequential write to a 10-disk RAID5 attached with 2 Gb fibre channel; whichever file system I use on the RAID should be a whole lot faster than over NFS, and indeed they all are, Ext3 about 3 times, the others much more.
So, unless there are hooks between the filesystem and the NFSd that are not obvious to a user (and the fact that changing filesystems doesn't seem to make much difference seems to rule that out) and the fact that the NFS client seems capable of much better performance when used against a Netapp, it makes me wonder what's going on with the server portion.
Quoted 6 lines
- Highly tuned nfs inplementation with dedicated operating system.
The bruteforce linux approach is quite good for the money spent, but don't expect the features/performance of a way more expensive enterprise solution.
A valid point, of course, but I don't think I'm actually expecting a single NFSd to act like an expensive Netapp. I do think that wondering why the Netapp is twice as fast for a sequential write is a valid question, even if the OS and NFS server subsystem are free. I was kind of hoping someone would just say "you're getting what you should expect to get" or "wow, that's slow, try this and this and this".
And I did mention that moving to the Linux/NFS/Atabeast solution was quite a bit better than what we had before ;) I'm not really complaining, just wondering here.
Hugh
Signature · 13 lines
Whatsoever hath no fins nor scales in the waters, that shall be an abomination unto you. - Leviticus 11:9-12 http://godhatesshrimp.com Hugh Caley [email protected] ------------------------------------------------------- SF.Net email is sponsored by: Discover Easy Linux Migration Strategies from IBM. Find simple to follow Roadmaps, straightforward articles, informative Webcasts and more! Get everything you need to get up to speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 7 lines · Hugh Caley
A valid point, of course, but I don't think I'm actually expecting a single NFSd to act like an expensive Netapp. I do think that wondering why the Netapp is twice as fast for a sequential write is a valid question, even if the OS and NFS server subsystem are free. I was kind of hoping someone would just say "you're getting what you should expect to get" or "wow, that's slow, try this and this and this".
You referenced that you were getting 300 megabits (or 37MB/s). I have
several SLES 9 nfs servers (using self compiled 2.6.11.5
<http://2.6.11.5>kernel) running on IBM x345 hardware (dual cpu pentum 4, 2gb ram, dual
qlogic hbas) connected to a single LSI storage array that presents four luns (two from controller A and two from B). Each lun is 1TB and made from hardware raid 8+1. Luns are merged together using device mapper.
It's not uncommon with my setup to get a sustained write speed of 75MB/s on
one of our SLES 9 compute systems (AMD Opterons) when doing a sequential write of an 8GB file. With two systems writing at the same time I get aggregate bandwidth better than 75MB/s (can't recall what it is).
I use tcp/nfs3 and for write testing I use 'iozone -c -e -s 8192m -i 0'. I
use 128 nfsds, export with 'rw,sync,no_subtree_check,no_root_squash' and add the following to sysctl.conf:
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 65536 8388608
net.ipv4.tcp_mem = 8388608 8388608 8388608
On nfs clients (Sun, Linux, IRIX) I use the mount options:
nosuid,rw,bg,hard,intr,vers=3,proto=tcp,rsize=32768,wsize=32768. On AIX I use the same options, but also add the critical 'combehind' (without it writes of large files [ie. close to the size of physical mem] is just horrid).
Chris