Linux NFS vs NetApp
Hello!
I am trying to understand how NetApp can be so much better at NFS servicing than my quad Opteron 250 SAN attached machine. So I need some help and some pointers to understand how I can make my opteron machine come on par (or within 70% NFS performance range) as that of my NetApp R200. I have run through the NFS-how-to's and have heard "that is why they cost so much more", but I really have to consider that probably most of the ideas that are in the NetApp are common knowldge (just not in my head).
Can anyone shed some light on this?
TIA!
Phy
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Quoted 12 lines
I am trying to understand how NetApp can be so much better at NFS servicing than my quad Opteron 250 SAN attached machine. So I need some help and some pointers to understand how I can make my opteron machine come on par (or within 70% NFS performance range) as that of my NetApp R200. I have run through the NFS-how-to's and have heard "that is why they cost so much more", but I really have to consider that probably most of the ideas that are in the NetApp are common knowldge (just not in my head).
Quoted 1 line
Can anyone shed some light on this?
Definitely sounds like something is wrong. You can do your own comparisons of Linux 2.6 vs Netapp here (the OpenPower 720 is a ppc64 Linux box):
http://www.spec.org/sfs97r1/results/sfs97r1.html
Anton
Quoted 12 lines · Anton Blanchard
Quoted 12 lines
I am trying to understand how NetApp can be so much better at NFS servicing than my quad Opteron 250 SAN attached machine. So I need some help and some pointers to understand how I can make my opteron machine come on par (or within 70% NFS performance range) as that of my NetApp R200. I have run through the NFS-how-to's and have heard "that is why they cost so much more", but I really have to consider that probably most of the ideas that are in the NetApp are common knowldge (just not in my head).
Quoted 1 line
Quoted 1 line
Can anyone shed some light on this?
you have to quantify what sort of hardware you're benchmarking in either case and how its configured before you can reasonably conclude to much... I spent quite a bit of time benchmarking filers and linux configurations recently and while I can say with some certainty that while netapp makes some very fast and well balanced filers they don't by any means have a lock on building a high-performance nfs box.
Quoted 5 lines
Definitely sounds like something is wrong. You can do your own comparisons of Linux 2.6 vs Netapp here (the OpenPower 720 is a ppc64 Linux box):
In actually using sfs97r1 published benchmarks to compare to hardware I was benchmarking (from emc, netapp and several roll-your own linux boxes) I found the published benchmark information alsmost entirely useless given that vendors tend to provide wildly silly hardware configurations. In the case of the openpower 720 (to use that for an example) the benchmarked machine has 70 15k rpm disks spread across 12 fibre channel controllers, 64GB of ram, 12GB of nvram and 7 network interfaces...
Quoted 7 lines
Anton
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 1 line
-------------------------------------------------------------------------- Joel Jaeggli Unix Consulting [email protected] GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2
Quoted 7 lines · Joel Jaeggli
In actually using sfs97r1 published benchmarks to compare to hardware I was benchmarking (from emc, netapp and several roll-your own linux boxes) I found the published benchmark information alsmost entirely useless given that vendors tend to provide wildly silly hardware configurations. In the case of the openpower 720 (to use that for an example) the benchmarked machine has 70 15k rpm disks spread across 12 fibre channel controllers, 64GB of ram, 12GB of nvram and 7 network interfaces...
If you threw that much hardware at a Linux system, and then tuned it so that it didn't really care about userspace performance (oh.. say.. by giving the knfsd thread a RT priority ;), and tuned things like the filesystem, the slab allocator and the networking stack to NFS requirements, it probably would be screaming fast too.. ;)
Quoted 11 lines · On Tue, Jan 11, 2005 at 04:19:57AM -0500, [email protected] wrote:
Quoted 7 lines · Joel Jaeggli
In actually using sfs97r1 published benchmarks to compare to hardware I was benchmarking (from emc, netapp and several roll-your own linux boxes) I found the published benchmark information alsmost entirely useless given that vendors tend to provide wildly silly hardware configurations. In the case of the openpower 720 (to use that for an example) the benchmarked machine has 70 15k rpm disks spread across 12 fibre channel controllers, 64GB of ram, 12GB of nvram and 7 network interfaces...
If you threw that much hardware at a Linux system,
... theory ... or have you actually tried?
Quoted 5 lines
and then tuned it so that it
didn't really care about userspace performance (oh.. say.. by giving the knfsd thread a RT priority ;), and tuned things like the filesystem, the slab allocator and the networking stack to NFS requirements, it probably would be screaming fast too.. ;)
You'd need to run a 2.4 kernel.
Current problems with 2.6:
1 ext3 causes kjournald oops on load
2 xfs has bad NFS/SMP/dcache interactions (you end up with undeletable
directories)
3 knfsd will give you stale handles (can be worked around by stat'ing
all your directories constantly on the server side)
The SGI XFS kernel from CVS actually almost solved (2) above, but not entirely - I was going to report on that again to LKML. The other problems are still, as far as I know, unsolved.
Not trying to flame anyone here, just trying to be realistic ;)
Signature · 1 line
/ jakob
Quoted 2 lines · Jakob Oestergaard
3 knfsd will give you stale handles (can be worked around by stat'ing
all your directories constantly on the server side)
This should be fixed now. Bug reports to the contrary welcomed. --Bruce Fields
Quoted 5 lines · J. Bruce Fields
Quoted 2 lines · Jakob Oestergaard
3 knfsd will give you stale handles (can be worked around by stat'ing
all your directories constantly on the server side)This should be fixed now. Bug reports to the contrary welcomed.
Excellent!
It seems SGI has merged their XFS kernel up to 2.6.10 - I'll give that a try and see what happens.
Thanks,
Signature · 1 line
/ jakob
Quoted 3 lines · Jakob Oestergaard
Quoted 1 line
If you threw that much hardware at a Linux system,
... theory ... or have you actually tried?
Merely indicating a method to approach the problem space...
Quoted 6 lines
Current problems with 2.6:
1 ext3 causes kjournald oops on load
2 xfs has bad NFS/SMP/dcache interactions (you end up with undeletable
directories)
3 knfsd will give you stale handles (can be worked around by stat'ing
all your directories constantly on the server side)
A mere matter of debugging. ;)
Quoted 13 lines · On Monday January 10, [email protected] wrote:
Hello!
I am trying to understand how NetApp can be so much better at NFS servicing than my quad Opteron 250 SAN attached machine. So I need some help and some pointers to understand how I can make my opteron machine come on par (or within 70% NFS performance range) as that of my NetApp R200. I have run through the NFS-how-to's and have heard "that is why they cost so much more", but I really have to consider that probably most of the ideas that are in the NetApp are common knowldge (just not in my head).
Quoted 1 line
Can anyone shed some light on this?
If you want to come anything close to comparable with a Netapp, get a few hundred Megabytes of NVRAM (e.g. www.umem.com), and configure it as an external journal for your filesystem (I know this can be done for ext3, I don't know about other filesystems). Then make sure your filesystem journals all data, not just metadata (data=journal option to ext3).
If you use a dedicated drive (or mirrored pair) in place of the NVRAM, you will come reasonably close.
NeilBrown
Quoted 14 lines · Neil Brown
Quoted 13 lines · On Monday January 10, [email protected] wrote:
Hello!
I am trying to understand how NetApp can be so much better at NFS servicing than my quad Opteron 250 SAN attached machine. So I need some help and some pointers to understand how I can make my opteron machine come on par (or within 70% NFS performance range) as that of my NetApp R200. I have run through the NFS-how-to's and have heard "that is why they cost so much more", but I really have to consider that probably most of the ideas that are in the NetApp are common knowldge (just not in my head).
Quoted 8 lines
Quoted 1 line
Can anyone shed some light on this?
If you want to come anything close to comparable with a Netapp, get a few hundred Megabytes of NVRAM (e.g. www.umem.com), and configure it as an external journal for your filesystem (I know this can be done for ext3, I don't know about other filesystems). Then make sure your filesystem journals all data, not just metadata (data=journal option to ext3).
NetApp's WAFL only journals metadata in NVRAM ...
(one of the primary reasons its called WAFL is that the data-write only happens once..).
cheers,
lincoln.
Quoted 8 lines · On Tuesday January 11, [email protected] wrote:
Quoted 6 lines · Neil Brown
If you want to come anything close to comparable with a Netapp, get a few hundred Megabytes of NVRAM (e.g. www.umem.com), and configure it as an external journal for your filesystem (I know this can be done for ext3, I don't know about other filesystems). Then make sure your filesystem journals all data, not just metadata (data=journal option to ext3).
Quoted 4 lines
NetApp's WAFL only journals metadata in NVRAM ...
(one of the primary reasons its called WAFL is that the data-write only happens once..).
That may be, though it doesn't fit with my (admittedly limitted) understanding of WAFL.
However Linux NFS definitely runs faster over ext3 if data=journal is selected.
NeilBrown
Quoted 6 lines · In article <[email protected]> you wrote:
Quoted 3 lines
NetApp's WAFL only journals metadata in NVRAM ...
(one of the primary reasons its called WAFL is that the data-write only happens once..).That may be, though it doesn't fit with my (admittedly limitted) understanding of WAFL.
Yes, AFAIK the NVRAM is used for the RAID-4, independend of WAFL and as a write-back cache.
However since also the read performance of Linux NFS is bad (at least not very well selftuning) the Hardware is not really the reason for the fast NFS implementation.
Greetings
Bernd
on den 12.01.2005 Klokka 00:36 (+0100) skreiv Bernd Eckenfels:
Quoted 3 lines
However since also the read performance of Linux NFS is bad (at least not very well selftuning) the Hardware is not really the reason for the fast NFS implementation.
Indeed: NFS readahead requests are often processed out of order by the server (due to the basic unordered nature of RPC calls, the lack of ordering between nfsd server threads, use of UDP, etc) and so I suspect the generic readahead algorithm will tend to default to the random access mode in many cases where it should really be doing sequential access.
Cheers,
Trond
Signature · 1 line
Trond Myklebust <[email protected]>