Skip to contentSkip to conversation

OT: RAID perf doldrums: advice/recommendations?

DISCUSSION email builder 7 participants 13 messages Oct 28, 2005 – Nov 10, 2005

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello,

Recently we turned on a RAID-backed NFS server that fell flat on its fa=
ce
as soon as it started to see even moderate activity (particularly IMAP traffic from Courier-IMAP). Through help on the Courier mailing list, th= e
problem appears to be very poor throughput to our hard drives (although systems is not my strong point, so I hope there is not something else I'v= e
overlooked).

Our choice to go with consumer-grade hardware seems to have bit us hard=
.=20
We are using the oft-maligned 3Ware 85xx series RAID5 controller and thre= e
Hitachi Deskstar 7k250 (250GB) SATA drives. The NFS serves up the mailst= ore
for a system that does about 60 mail messages/min, plus IMAP against the mailstore (which is what brings the hurt), and some moderately (much more than Joe Bob's homepage, but not Slashdot) trafficed web content. We wer= e
able to buy some time by moving our (ext3) journal off of the same device= ,
fiddle with a couple NFS settings (nodiratime,noatime), but it is still woefully slow.

Are we completely wrong to give up on this hardware? If anyone is
interested in giving advice in that realm, I am happy to provide more sta= ts
(like the fact that the machine sits at near 100% iowait all day long).

But instead of sticking it out with that hardware, we are thinking abou=
t a
couple possibilities, but a look around Google just found too many "shopp= ing"
sites, and not enough real information and comparisons. We are wanting advice for proven performers: RAID controller cards and drives. We had b= een
hoping not to have to pay the hefty costs that SCSI incurrs, but at this point we are considering the possibility, so advice on both sides of the fence is sought.

We are also quite curious about just running with software raid instead=
,
since there seems to be a growing movement that claims there is no reason=
it
cannot be as (or more) reliable and fast -- especially compared to the pi= ece
of $@^!% that we apparently got. :)

Thoguhts and links highly appreciated!

=09

	=09

__________________________________=20
Yahoo! Mail - PC Magazine Editors' Choice 2005=20
http://mail.yahoo.com

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#1 lore raw
Quoted 9 lines · On Thursday October 27, [email protected] wrote:

Hello,

Recently we turned on a RAID-backed NFS server that fell flat on its face
as soon as it started to see even moderate activity (particularly IMAP traffic from Courier-IMAP). Through help on the Courier mailing list, the problem appears to be very poor throughput to our hard drives (although systems is not my strong point, so I hope there is not something else I've overlooked).

Quoted 9 lines

Our choice to go with consumer-grade hardware seems to have bit us hard.
We are using the oft-maligned 3Ware 85xx series RAID5 controller and three Hitachi Deskstar 7k250 (250GB) SATA drives. The NFS serves up the mailstore for a system that does about 60 mail messages/min, plus IMAP against the mailstore (which is what brings the hurt), and some moderately (much more than Joe Bob's homepage, but not Slashdot) trafficed web content. We were able to buy some time by moving our (ext3) journal off of the same device, fiddle with a couple NFS settings (nodiratime,noatime), but it is still woefully slow.

mounting the ext3 partition with 'data=journal' can help NFS service, especially with the journal on a separate device.

Quoted 5 lines

We are also quite curious about just running with software raid instead,
since there seems to be a growing movement that claims there is no reason it cannot be as (or more) reliable and fast -- especially compared to the piece of $@^!% that we apparently got. :)

Certainly worth a try. It may not be as fast as some high-end raid cards, it is unlikely to be slower than what you have.

NeilBrown

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#2 lore raw
Quoted 4 lines

were

Quoted 1 line

able to buy some time by moving our (ext3) journal off of the same

device,

Quoted 1 line

fiddle with a couple NFS settings (nodiratime,noatime), but it is sti=

ll

Quoted 4 lines
Quoted 1 line

woefully slow.

=20
mounting the ext3 partition with 'data=3Djournal' can help NFS service, especially with the journal on a separate device.

OK, we must have missed this one in the noise. I just added it to the mo= unt
where all the NFS server's exports are (ext3 defaults,data=3Djournal). E= ven
rebooted, but don't see evidence that it took (cat /proc/mounts just show= s
ext3 rw but nothing about the journaling mode, should it?). Where can I verify what mode it is in?

The other thing that came as a strong recommendation to us was to put "no_acl" on all of our NFS exports, but on our Fedora system (with up to = date
nfs-utils), "no_acl" is treated as an unrecognized keyword. Is there any support for no_acl under redhat at all???

=09

	=09

__________________________________=20
Yahoo! Mail - PC Magazine Editors' Choice 2005=20
http://mail.yahoo.com

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#3 lore raw
In reply to email builder
Quoted 19 lines · email builder

Hello,

Recently we turned on a RAID-backed NFS server that fell flat on its face
as soon as it started to see even moderate activity (particularly IMAP traffic from Courier-IMAP). Through help on the Courier mailing list, the problem appears to be very poor throughput to our hard drives (although systems is not my strong point, so I hope there is not something else I've overlooked).

Our choice to go with consumer-grade hardware seems to have bit us hard.
We are using the oft-maligned 3Ware 85xx series RAID5 controller and three Hitachi Deskstar 7k250 (250GB) SATA drives. The NFS serves up the mailstore for a system that does about 60 mail messages/min, plus IMAP against the mailstore (which is what brings the hurt), and some moderately (much more than Joe Bob's homepage, but not Slashdot) trafficed web content. We were able to buy some time by moving our (ext3) journal off of the same device, fiddle with a couple NFS settings (nodiratime,noatime), but it is still woefully slow.

Quoted 3 lines

Are we completely wrong to give up on this hardware? If anyone is
interested in giving advice in that realm, I am happy to provide more stats (like the fact that the machine sits at near 100% iowait all day long).

Hi

We noticed similar issues which we never tracked down, even after help from this list. There is a huge bug report at Redhat entitled:

"Extremely high iowait with 3Ware array and moderate disk activity"

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=121434

which may be of some help, but it never quite gets to the bottom
of things. 3ware cards/drivers seem to the common factor in most if
not all of the "me too's", but of course everyone with a 3ware problem would find this bug report...

Cheers

Ian

Signature · 14 lines
Ian Thurlbeck                http://www.stams.strath.ac.uk/
Statistics and Modelling Science, University of Strathclyde
Livingstone Tower, 26 Richmond Street, Glasgow, UK,  G1 1XH
Tel: +44 (0)141 548 3667           Fax: +44 (0)141 552 2079


-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course
Free Certification Exam for All Training Attendees Through End of 2005
Visit http://www.jboss.com/services/certification for more information
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#4 lore raw

Hi List,

I was having some performance problems too, a while back. I don't know that performance is now as good as it could possibly be, but it is substantially better than it was. I'm not sure the problems I faced are related to those described below, but I'll share them anyway in the hope that someone finds them useful.

I have two NFS servers which are basically clones of each other, from a hardware perspective. One was set up with Red Hat 8.0, and the other was running Red Hat Enterprise 3. That becomes important later. Both have Intel SE7500CW2 motherboards and 3Ware 7506 RAID controllers.

When I set up the first system, I noticed that I was having all sorts of trouble with the gigabit card. I'll just paste in some of my notes:

# This next line comes to us from Intel. The SE7500CW2 motherboard which we

# are using has a bit of a problem, in combination with its Intel
# Gigabit card.  When using any version of the e1000 driver newer than 4.3.15,
# we would get all sorts of messages in the syslog saying, "NETDEV WATCHDOG:
# eth0: transmit timed out".  The card would then go offline for a number of
# seconds (10-15) while it reset itself.  This happened quite frequently and
# was highly undesirable behavior in an NFS server.  After much conferring with
# Intel ([email protected]), it was determined that there was a cache
# coherency issue with the ICH (Southbridge) device on our SE7500CW2, and that
# setting bit 13 in the 0x50 register of the ICH would alter its behavior with
# regard to prefetch flushing of the cache.  Notes from Intel indicated that
# this device would probably be at Bus 0, Device 0x1E, Func 0, Vendor ID 8086,
# Device ID 244E.  This was confirmed with "lspci" and "pcitweak -l".  The
# original value in register 0x50 was 0x00001402.  A bitwise OR with 0x00002000
# resulted in the new value, 0x00003402.  This fix does not persist across
# reboots, which is why it had to be entered into this file.  Special thanks
# goes to Colleen of Intel Support, who put in a lot of effort to help me get
# this fixed.

The other problem we noticed is that the server with RH8 had pretty decent NFS performance, where the RHEL3 system seemed very slow. After a lot of conferring with Red Hat on this, I was told that it was because of the change from async to sync as the default for exports and that I should specify async when exporting. I asked them about the safety of this (FAQ: B6), but don't recall getting much by way of response; however, since performance *was* better with async and since the complaints about performance were very loud, I had little choice but to stick with that.

The other thing I've done to (hopefully) improve performance was to increase the number of NFS threads being spawned, and give them each a bunch of memory ([rw]mem_{default,max}). I figure these boxes are NFS servers and they've got the memory to spare, so I may as well dedicate that memory to what they're supposed to be doing. I can't speak to the "Extremely high iowait with 3Ware array and moderate disk activity", but if anyone hears anything about that, I'd certainly be interested in hearing as well.

James

As always, standard disclaimers apply. I don't speak for my company, what works for me may not work for you, etc.

Quoted 33 lines · Ian Thurlbeck
Quoted 19 lines · email builder

Hello,

Recently we turned on a RAID-backed NFS server that fell flat on its face
as soon as it started to see even moderate activity (particularly IMAP traffic from Courier-IMAP). Through help on the Courier mailing list, the problem appears to be very poor throughput to our hard drives (although systems is not my strong point, so I hope there is not something else I've overlooked).

Our choice to go with consumer-grade hardware seems to have bit us hard.
We are using the oft-maligned 3Ware 85xx series RAID5 controller and three Hitachi Deskstar 7k250 (250GB) SATA drives. The NFS serves up the mailstore for a system that does about 60 mail messages/min, plus IMAP against the mailstore (which is what brings the hurt), and some moderately (much more than Joe Bob's homepage, but not Slashdot) trafficed web content. We were able to buy some time by moving our (ext3) journal off of the same device, fiddle with a couple NFS settings (nodiratime,noatime), but it is still woefully slow.

Quoted 3 lines

Are we completely wrong to give up on this hardware? If anyone is
interested in giving advice in that realm, I am happy to provide more stats (like the fact that the machine sits at near 100% iowait all day long).

Hi

We noticed similar issues which we never tracked down, even after help from this list. There is a huge bug report at Redhat entitled:

"Extremely high iowait with 3Ware array and moderate disk activity"

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=121434

Quoted 8 lines

which may be of some help, but it never quite gets to the bottom
of things. 3ware cards/drivers seem to the common factor in most if
not all of the "me too's", but of course everyone with a 3ware problem would find this bug report...

Cheers

Ian

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#5 lore raw
Quoted 1 line

I have two NFS servers which are basically clones of each other, from a=

=20

Quoted 1 line

hardware perspective. One was set up with Red Hat 8.0, and the other w=

as=20

Quoted 5 lines

running Red Hat Enterprise 3. That becomes important later. Both have Intel=20
SE7500CW2 motherboards and 3Ware 7506 RAID controllers.
=20
When I set up the first system, I noticed that I was having all sorts o=

f=20

Quoted 1 line

trouble with the gigabit card. I'll just paste in some of my notes:

<snip> we don't seem to have any actual eth problems =20

Quoted 1 line

The other problem we noticed is that the server with RH8 had pretty dec=

ent=20

Quoted 1 line

NFS performance, where the RHEL3 system seemed very slow. After a lot =

of=20

Quoted 5 lines

conferring with Red Hat on this, I was told that it was because of the change=20
from async to sync as the default for exports and that I should specify async=20
when exporting. I asked them about the safety of this (FAQ: B6), but d=

on't

Quoted 2 lines

=20
recall getting much by way of response; however, since performance *was=

*=20

Quoted 3 lines

better with async and since the complaints about performance were very loud,=20
I had little choice but to stick with that.

Ouch. I have spent many hours reading about this, and general consensus seems to say this is not a good idea. This to me (along with their appar= ent
inability to solve the issues with the 3Ware card) indicates that RedHat = is a
bad choice for NFS servers.... I mean, if they just say you should do something dangerous because it is the only way to get better performance = but
then not back it up or have any kind of real response to the fact that th= is
should not be the appropriate thing to do.... well, you get the picture.= =20
Maybe we'll try FreeBSD before we change anything else or buy new hardwar= e.

Quoted 3 lines

The other thing I've done to (hopefully) improve performance was to increase=20
the number of NFS threads being spawned, and give them each a bunch of

Yeah, we bumped ours to 128 just to be safe (beside the severe iowait, ou= r
machine is lightly used). The default 8 were being absolutely swamped.

Quoted 2 lines

memory=20
([rw]mem_{default,max}). I figure these boxes are NFS servers and they=

've=20

Where do you set their memory and what are people's recommended changes f= or
getting more "umph" out of NFS?

Thanks a ton for sharing!

Quoted 2 lines

got the memory to spare, so I may as well dedicate that memory to what=20 they're supposed to be doing. I can't speak to the "Extremely high iow=

ait=20

Quoted 1 line

with 3Ware array and moderate disk activity", but if anyone hears anyth=

ing=20

Quoted 1 line

about that, I'd certainly be interested in hearing as well.

=09

	=09

__________________________________=20
Yahoo! Mail - PC Magazine Editors' Choice 2005=20
http://mail.yahoo.com

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#6 lore raw
In reply to Ian Thurlbeck
Quoted 9 lines

We noticed similar issues which we never tracked down, even after help=20 from this list. There is a huge bug report at Redhat entitled:
=20
"Extremely high iowait with 3Ware array and moderate disk activity" =20
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=3D121434
=20
which may be of some help, but it never quite gets to the bottom of things. 3ware cards/drivers seem to the common factor in most if

Yes, we've combed this thread, much to our shagrin, none of the tips work= ed
for us. I suppose I should have said that one of our other possibilities=
is
to go with FreeBSD to see if this issue is the source of the problem. Wh= at
can you say after reading this thread that's been open for WELL over a ye= ar?=20
RedHat loses a lot of points on this one.

Quoted 2 lines

not all of the "me too's", but of course everyone with a 3ware problem=20 would find this bug report...

=09

	=09

__________________________________=20
Yahoo! Mail - PC Magazine Editors' Choice 2005=20
http://mail.yahoo.com

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#7 lore raw
In reply to email builder

If you stay with RAID5, get something with a battery backed write cache. That will make a huge difference in RAID5 performance.

Michael Haverkamp

[email protected] wrote on 10/27/2005 10:21:49 PM:

Quoted 3 lines

Hello,

Recently we turned on a RAID-backed NFS server that fell flat on its

face

Quoted 2 lines

as soon as it started to see even moderate activity (particularly IMAP traffic from Courier-IMAP). Through help on the Courier mailing list,

the

Quoted 2 lines

problem appears to be very poor throughput to our hard drives (although systems is not my strong point, so I hope there is not something else

I've

Quoted 3 lines

overlooked).

Our choice to go with consumer-grade hardware seems to have bit us

hard.

Quoted 1 line

We are using the oft-maligned 3Ware 85xx series RAID5 controller and

three

Quoted 1 line

Hitachi Deskstar 7k250 (250GB) SATA drives. The NFS serves up the

mailstore

Quoted 2 lines

for a system that does about 60 mail messages/min, plus IMAP against the mailstore (which is what brings the hurt), and some moderately (much

more

Quoted 1 line

than Joe Bob's homepage, but not Slashdot) trafficed web content. We

were

Quoted 1 line

able to buy some time by moving our (ext3) journal off of the same

device,

Quoted 5 lines

fiddle with a couple NFS settings (nodiratime,noatime), but it is still woefully slow.

Are we completely wrong to give up on this hardware? If anyone is
interested in giving advice in that realm, I am happy to provide more

stats

Quoted 3 lines

(like the fact that the machine sits at near 100% iowait all day long).

But instead of sticking it out with that hardware, we are thinking

about a

Quoted 1 line

couple possibilities, but a look around Google just found too many

"shopping"

Quoted 2 lines

sites, and not enough real information and comparisons. We are wanting advice for proven performers: RAID controller cards and drives. We had

been

Quoted 5 lines

hoping not to have to pay the hefty costs that SCSI incurrs, but at this point we are considering the possibility, so advice on both sides of the fence is sought.

We are also quite curious about just running with software raid

instead,

Quoted 1 line

since there seems to be a growing movement that claims there is no

reason it

Quoted 1 line

cannot be as (or more) reliable and fast -- especially compared to the

piece

Quoted 14 lines

of $@^!% that we apparently got. :)

Thoguhts and links highly appreciated!

__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005
http://mail.yahoo.com

Quoted 8 lines

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#8 lore raw
Quoted 2 lines

If you stay with RAID5, get something with a battery backed write cache=

.=20

Quoted 1 line

That will make a huge difference in RAID5 performance.

Really? I was under the impression that this was just a safety precautio= n or
similar... it is called a "backup" after all. :) How exactly does it increase performance? =20

(And, yes, I have seen this mentioned in many places, so this would be a criteria for our next purchase. :) )

=09

	=09

__________________________________=20
Yahoo! Mail - PC Magazine Editors' Choice 2005=20
http://mail.yahoo.com

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#9 lore raw

[email protected] wrote on 10/28/2005 05:22:38 PM:

Quoted 2 lines
Quoted 2 lines

If you stay with RAID5, get something with a battery backed write

cache.

Quoted 3 lines
Quoted 1 line

That will make a huge difference in RAID5 performance.

Really? I was under the impression that this was just a safety

precaution or

Quoted 2 lines

similar... it is called a "backup" after all. :) How exactly does it increase performance?

Without one, writes must be flushed to disk before the RAID controller can report them as complete. This includes even trivial things like updating file access times. With one, writes only need to get into the controller's write cache before they are reported as complete. This allows the controller to schedule the actual write to disk at an optimal time.

A good way to show this is to untar a tar file full of small files (e.g. kernel source). On RAID5 without write cache, this takes at least 10 times longer than even a single ATA disk.

Quoted 12 lines

(And, yes, I have seen this mentioned in many places, so this would be a criteria for our next purchase. :) )

__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005
http://mail.yahoo.com

Quoted 8 lines

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#10 lore raw
Quoted 7 lines
Quoted 1 line
Quoted 1 line

If you stay with RAID5, get something with a battery backed write=20

cache.=20

Quoted 3 lines
Quoted 1 line

That will make a huge difference in RAID5 performance.

=20
Really? I was under the impression that this was just a safety=20

precaution or

Quoted 1 line

similar... it is called a "backup" after all. :) How exactly does i=

t

Quoted 3 lines
Quoted 1 line

increase performance?=20

=20
Without one, writes must be flushed to disk before the RAID controller =

can=20

Quoted 1 line

report them as complete. This includes even trivial things like updati=

ng=20

Quoted 3 lines

file access times. With one, writes only need to get into the=20 controller's write cache before they are reported as complete. This=20 allows the controller to schedule the actual write to disk at an optima=

l=20

Quoted 1 line

time.

Thank you sincerely for this excellent explanation. We are currently loo= king
at the new 3Ware SATA II card, the 9550SX:

http://3ware.com/products/serial_ata2-9000.asp

What is strange about these is that the battery backup is sold separately= .=20
Does anyone have experience with the BBUs (battery backup units) on the 3= Ware
9000 series cards?

- Do they really provide the bang for the extra $100?
- What kind of configuration/installation mess are the BBUs?

Quoted 1 line

A good way to show this is to untar a tar file full of small files (e.g=

.=20

Quoted 5 lines

kernel source). On RAID5 without write cache, this takes at least 10=20 times longer than even a single ATA disk.
=20

Quoted 2 lines

=20
(And, yes, I have seen this mentioned in many places, so this would b=

e a

Quoted 1 line
Quoted 1 line

criteria for our next purchase. :) )

	=09

__________________________________=20
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Downl= oad
it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#11 lore raw
In reply to email builder

Most, if not all, of my notes pertaining to large storage are here:

http://dcs.nac.uci.edu/~strombrg/tech-tidbits-by-category.html#Large%20Storage

I think commodity hardware is generally a good way to go when you can, but we also had bad experiences with some 3Ware RAID cards, though in our case, it wasn't clear if the problem was best attributed to the Maxtor disks, the 3Ware RAID cards, the Promicrosystems PC's, or the GFS and Lustre we tried overtop of this hardware.

We eventually replaced all this with a Sun 3511 with a Sun V440 out front of it as a NAS head, with a QFS filesystem to piece together the slices (no SVM or other extra tack-them-together layer required, since QFS does that -and- the filesystem). It's served out via a Solaris NFS server to a handful of AIX NFS clients, and it works well. But that's because we needed 15T of storage - had we required only 2T, we would've had many more choices.

You -can- get good performance out of commodity stuff. As a simple thought-experiment, consider what would happen if you had a huge number of small disks accessed via distinct 100BaseT links, with your data being striped across them all. Because of the huge amount of striping, your aggregate throughput should be quite good despite the 100BaseT.

I believe Hitachi bought the Deskstar line of disks from IBM, shortly after IBM had to replace tons of Deskstars being they were dying left and right. The problems may or may not have been thoroughly worked out since then.

Sometimes under linux and other OS's, a single process that's pound a storage resource can pretty well mess up the performance for the other processes requiring access to that resource. Sometimes deprioritising that I/O hungry process can help the other processes on the system A LOT:

http://dcs.nac.uci.edu/~strombrg/Keeping-backups-from-pounding-a-system-as-hard.html

HTH!

Quoted 48 lines · email builder

Hello,

Recently we turned on a RAID-backed NFS server that fell flat on its face
as soon as it started to see even moderate activity (particularly IMAP traffic from Courier-IMAP). Through help on the Courier mailing list, the problem appears to be very poor throughput to our hard drives (although systems is not my strong point, so I hope there is not something else I've overlooked).

Our choice to go with consumer-grade hardware seems to have bit us hard.
We are using the oft-maligned 3Ware 85xx series RAID5 controller and three Hitachi Deskstar 7k250 (250GB) SATA drives. The NFS serves up the mailstore for a system that does about 60 mail messages/min, plus IMAP against the mailstore (which is what brings the hurt), and some moderately (much more than Joe Bob's homepage, but not Slashdot) trafficed web content. We were able to buy some time by moving our (ext3) journal off of the same device, fiddle with a couple NFS settings (nodiratime,noatime), but it is still woefully slow.

Are we completely wrong to give up on this hardware? If anyone is
interested in giving advice in that realm, I am happy to provide more stats (like the fact that the machine sits at near 100% iowait all day long).

But instead of sticking it out with that hardware, we are thinking about a
couple possibilities, but a look around Google just found too many "shopping" sites, and not enough real information and comparisons. We are wanting advice for proven performers: RAID controller cards and drives. We had been hoping not to have to pay the hefty costs that SCSI incurrs, but at this point we are considering the possibility, so advice on both sides of the fence is sought.

We are also quite curious about just running with software raid instead,
since there seems to be a growing movement that claims there is no reason it cannot be as (or more) reliable and fast -- especially compared to the piece of $@^!% that we apparently got. :)

Thoguhts and links highly appreciated!

__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005
http://mail.yahoo.com

Quoted 9 lines

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course Free Certification Exam for All Training Attendees Through End of 2005 Visit http://www.jboss.com/services/certification for more information
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#12 lore raw
Quoted 4 lines · Dan Stromberg

Most, if not all, of my notes pertaining to large storage are here:

http://dcs.nac.uci.edu/~strombrg/tech-tidbits-by-category.html#Large%20St=

orage

Quoted 6 lines

I think commodity hardware is generally a good way to go when you can, but we also had bad experiences with some 3Ware RAID cards, though in our case, it wasn't clear if the problem was best attributed to the Maxtor disks, the 3Ware RAID cards, the Promicrosystems PC's, or the GFS and Lustre we tried overtop of this hardware.

I agree that commodity hardware is a good way to go if possible. I've had great experience with 3ware cards on Redhat systems if they are running the latest firmware/driver - horrible experiences otherwise.=20 Also had horrible luck with maxtor drives on 3ware controllers and maxtor drives in generally - you should think twice about using them.

We see 12MB/sec from multiple clients all day long on RH systems with this configuration - this is just ata raid on 75xx controllers, nothing fancy.

Obviously the following options are huge for performance on clients as well, tcp,rsize=3D8192,wsize=3D8192

The sync option also has a rather large impact on client write performance and I have left it async for that reason.

-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs