3ware escalade vs software raid, from a different jeff
I'm building an nfs server to export home directories to a few login servers used somewhat heavily by a few hundred people. Additionally, on the array will be a bunch of web content and mysql data files.
I'm currently thinking of a few different configurations, all using four drives in the 180-250gb range. I could get sata or udma drives, i could go raid5 or raid10, and i could do this in linux software raid (using, say, a pair of promise ultra100s to give me four udma ports, or with an SATA card), using 3ware's UDMA four-port escalade, or with 3ware's new sata escalade.
What are the compelling reasons for using sata over udma? I've had really good experiences with linux software raid, but always in small workgroup or personal file servers. I'm wondering if this sort of big, random, heavy load will drag down an otherwise speedy box. Is it worth it to just drop the extra $300 and go with hardware raid? as quality as linux software raid is, i'm somehow still more comfortable just leaving it to hardware.
what sort of ideas or experiences can you guys share?
i'm going for cost first (i've got a $1k budget or so), reliability and stability second, and speed third.
-j
--
--------------------------------------------------------
Rev. Jeffrey Paul -datavibe- [email protected]
aim:x736e65616b pgp:0x15FA257E phone:8777483467
70E0 B896 D5F3 8BF4 4BEE 2CCF EF2F BA28 15FA 257E
--------------------------------------------------------
Quoted 8 lines · Rev. Jeffrey Paul
I'm building an nfs server to export home directories to a few login servers used somewhat heavily by a few hundred people. Additionally, on the array will be a bunch of web content and mysql data files.
I'm currently thinking of a few different configurations, all using four drives in the 180-250gb range. I could get sata or udma drives, i could go raid5 or raid10, and i could do this in linux software raid (using, say, a pair of promise ultra100s to give me four udma ports, or with an SATA card), using 3ware's UDMA four-port escalade, or with 3ware's new sata escalade.
3ware hardware is really nice, so that would be a good choice.
But... I am a bit biased: I almost always prefer software RAID over hardware RAID. Much more flexibility to move disks between systems, choose controller(s) you like, etc.
In theory performance of hardware RAID can be better than software RAID.
In practice, only a few hardware RAID controllers actually achieve this.
Jeff
Quoted 5 lines · Rev. Jeffrey Paul
I'm building an nfs server to export home directories to a few login servers used somewhat heavily by a few hundred people. Additionally, on the array will be a bunch of web content and mysql data files.
Quoted 9 lines
I'm currently thinking of a few different configurations, all using four drives in the 180-250gb range. I could get sata or udma drives, i could go raid5 or raid10, and i could do this in linux software raid (using, say, a pair of promise ultra100s to give me four udma ports, or with an SATA card), using 3ware's UDMA four-port escalade, or with 3ware's new sata escalade.
What are the compelling reasons for using sata over udma?
For now none, really. Only that in future it will be more and more difficult to get UDMA drives. But this might take a long time before this really becomes a problem. SATA II will bring a real advantage.
Quoted 4 lines
I've had really good experiences with linux software raid, but always in small workgroup or personal file servers. I'm wondering if this sort of big, random, heavy load will drag down an otherwise speedy box.
I have a system running for nearly three years distributing some 2.3 million files with 200GB daily. This is with linux software raid and have encountered absolutly no problems. During the same period another system (not linux) with a similar workload but with hardware raid has failed twice, once makeing all data useless.
Quoted 2 lines
Is it worth it to just drop the extra $300 and go with hardware raid?
In my opinion no. Most of the cheaper hardware raids are really just software raid solutions. For the more expansive once always remember you will always need drivers and there is no guarentee that you will get them in two or three years. Some vendors do no longer exist or no longer support that product.
Also don't worry about the slightly higher CPU usage of software raid. The time spend by most application is by waiting for IO from disk/raid, so any time gained here (software raid is faster then hardware raid) will easily make up the lost CPU time.
Regards,
Holger
On Thu, 19 Feb 2004 at 11:49am, Holger Kiehl wrote
Quoted 5 lines
In my opinion no. Most of the cheaper hardware raids are really just software raid solutions. For the more expansive once always remember you will always need drivers and there is no guarentee that you will get them in two or three years. Some vendors do no longer exist or no longer support that product.
I fully agree on avoiding cheap "hardware" RAID cards. Regarding drivers going away, though, the 3ware drivers are open source and have been in the kernel for a long time. If 3ware were to disappear (unlikely), I'd be willing to bet that the drivers stay supported by the community.
Signature · 3 lines
Joshua Baker-LePain Department of Biomedical Engineering Duke University
Quoted 6 lines · Joshua Baker-LePain
I fully agree on avoiding cheap "hardware" RAID cards. Regarding drivers going away, though, the 3ware drivers are open source and have been in the kernel for a long time. If 3ware were to disappear (unlikely), I'd be willing to bet that the drivers stay supported by the community.
I have always avoided the promise fasttrak cards for exactly that reason ("hardware" being in quotes). It's my understanding that they're not much more than their udma cards with drivers that do most of the RAIDing.
It was also my understanding that the 3ware cards are true hardware raid and are up to the task of something like this, which would also explain why they're 4x the cost.
I am looking at a four disk raid5 or raid10, and it seems like the interrupt load from four drives on four channels might be a bit excesive. I'm going to be running critical services on the machine that the drives are in (namely mysql and nfs) and don't want to worry about performance.
Am I thinking about this the wrong way?
-j
--
--------------------------------------------------------
Rev. Jeffrey Paul -datavibe- [email protected]
aim:x736e65616b pgp:0x15FA257E phone:8777483467
70E0 B896 D5F3 8BF4 4BEE 2CCF EF2F BA28 15FA 257E
--------------------------------------------------------
Quoted 8 lines · Joshua Baker-LePain
Quoted 6 lines
I fully agree on avoiding cheap "hardware" RAID cards. Regarding drivers going away, though, the 3ware drivers are open source and have been in the kernel for a long time. If 3ware were to disappear (unlikely), I'd be willing to bet that the drivers stay supported by the community.
Quoted 9 lines
I have always avoided the promise fasttrak cards for exactly that reason ("hardware" being in quotes). It's my understanding that they're not much more than their udma cards with drivers that do most of the RAIDing.
It was also my understanding that the 3ware cards are true hardware raid and are up to the task of something like this, which would also explain why they're 4x the cost.
Bear in mind that what you are calling "true hardware raid" is really a microprocessor programmed to do the raid algorithims. Usually these microprocessors are stretched to the limit to handle the throughput of modern udam drives. I don't know but I suspect that the small overhead use in the mmu for software raid has far more and faster throughput than any of these dedicated microprocessors..... and, you can see the code and know it is bug free or will be if you report the bug. I am the unhappy owner of several Adaptec raid cards that have onboard processors to handle not only raid, but command processing for the scsi bus. These turkeys have micro-code bugs that cause a variety of problem for which there is no workaround or solution other than trashing the cards. Don't get me wrong, I thing the 3ware product is exceptionally good, I just wouldn't use the raid code given the choice of linux software raid.
Currently running 10 linux software raid boxes -- mix of raid 1 and raid 5. Yes, I'm biased :-)
Michael
[email protected]
Quoted 2 lines · Michael
Bear in mind that what you are calling "true hardware raid" is really a microprocessor programmed to do the raid algorithims.
3ware has custom designed matrix switch chips to handle each IDE drive. This *alone* is worth the cost of the card. The RAID parity calculations are also done in hardware (not a bunch of CPU xor's.) You'd be surprised how much data a small "slow" processor can move when properly programmed.
Quoted 3 lines
These turkeys have micro-code bugs that cause a
variety of problem for which there is no workaround or solution other than trashing the cards.
Exactly what bugs? I've used a number of hardware SCSI RAID cards (none by Adaptec however -- some that are now owned by Adapatec) and all of them work properly -- as long as they aren't physically damaged (I had one with a bad DIMM)
--Ricky
Scsi adapters and software raid...
It's been interesting reading these comments about adapters to avoid.
My question is... what SCSI adapter works WELL with software raid (mdadm) and doesn't get in the way?
Thanks, BobH
Quoted 25 lines · Ricky Beam
Quoted 2 lines · Michael
Bear in mind that what you are calling "true hardware raid" is really a microprocessor programmed to do the raid algorithims.
3ware has custom designed matrix switch chips to handle each IDE drive. This *alone* is worth the cost of the card. The RAID parity calculations are also done in hardware (not a bunch of CPU xor's.) You'd be surprised how much data a small "slow" processor can move when properly programmed.
Quoted 3 lines
These turkeys have micro-code bugs that cause a
variety of problem for which there is no workaround or solution other than trashing the cards.Exactly what bugs? I've used a number of hardware SCSI RAID cards (none by Adaptec however -- some that are now owned by Adapatec) and all of them work properly -- as long as they aren't physically damaged (I had one with a bad DIMM)
--Ricky
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 1 line
Bob Hillegas <[email protected]>
Re: Scsi adapters and software raid...
Bob,
All the Adaptec non-RAID SCSI controllers work nicely. The ones based on the 79xx and newer chips require the newer kernel module. The 78xx and older ones are all supported by the old module. They are generally stable, compatible and offer good performance.
Their SCSI _hardware_ RAID controllers are a different story. Steer clear of those.
-Kanoa
Quoted 4 lines · Bob Hillegas
It's been interesting reading these comments about adapters to avoid.
My question is... what SCSI adapter works WELL with software raid (mdadm) and doesn't get in the way?
Re: Scsi adapters and software raid...
In reply to Bob HillegasQuoted 4 lines · Bob Hillegas
It's been interesting reading these comments about adapters to avoid.
My question is... what SCSI adapter works WELL with software raid (mdadm) and doesn't get in the way?
For parallel SCSI?
IMO Adaptec is really the only big vendor in the game anymore. I think QLogic dumped their parallel SCSI hardware (correct me if I'm wrong), even though they still support their parallel SCSI driver.
Personally, though, I would skip SCSI and go straight to Serial ATA :)
Jeff
Quoted 9 lines · Ricky Beam
Quoted 2 lines · Michael
Bear in mind that what you are calling "true hardware raid" is really a microprocessor programmed to do the raid algorithims.
3ware has custom designed matrix switch chips to handle each IDE drive. This *alone* is worth the cost of the card. The RAID parity calculations are also done in hardware (not a bunch of CPU xor's.) You'd be surprised how much data a small "slow" processor can move when properly programmed.
Are you talking about host-processor XOR or adapter card XOR? The i960 processor used in practically every RAID card known to man right now has an XOR engine in the memory controller. It's not quite as efficient as some higher-end implementations, but it's a heck of a lot better than having that i960 core spin through an XOR software loop.
Scott
I have 14 18Gig disks in a RAID5. When it re-builds I get just over 6Meg/sec. That's 6M per disk, or 84Meg total I/O per second. The CPU load is less than 5%. My system is a P3-500. If I had a real computer I would not be able to measure the CPU usage, it would be too low! :)
I have a 2.4 Kernel.
My SCSI buses are the limiting factor.
40MB bus with 6 disks, 80MB with 7, 80MB with 1.
I have only used a few hardware RAID cards, they are not this fast!
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Scott Long
Sent: Thursday, February 19, 2004 5:34 PM
To: Ricky Beam
Cc: Michael; [email protected]
Subject: Re: 3ware escalade vs software raid, from a different jeff
Quoted 10 lines · Ricky Beam
Quoted 2 lines · Michael
Bear in mind that what you are calling "true hardware raid" is really a microprocessor programmed to do the raid algorithims.
3ware has custom designed matrix switch chips to handle each IDE drive. This *alone* is worth the cost of the card. The RAID parity calculations are also done in hardware (not a bunch of CPU xor's.) You'd be surprised how
much data a small "slow" processor can move when properly programmed.
Are you talking about host-processor XOR or adapter card XOR? The i960 processor used in practically every RAID card known to man right now has an XOR engine in the memory controller. It's not quite as efficient as some higher-end implementations, but it's a heck of a lot better than having that i960 core spin through an XOR software loop.
Scott
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Michael
Bear in mind that what you are calling "true hardware raid" is really a microprocessor programmed to do the raid algorithims. Usually these microprocessors are stretched to the limit to handle the throughput of modern udam drives. I don't know but I suspect that the small overhead use in the mmu for software raid has far more and faster throughput than any of these dedicated microprocessors..... and, you can see the code and know it is bug free or will be if you report the bug. I am the unhappy owner of several Adaptec raid cards that have onboard processors to handle not only raid, but command processing for the scsi bus.
What exactly are you talking about here? Are you using a multi-channel RAID card to do RAID on one channel and SCSI-passthru on the other? Please explain.
Quoted 3 lines
These turkeys have micro-code bugs that cause a
variety of problem for which there is no workaround or solution other than trashing the cards.
What 'micro-code' bugs are you talking about? What problems are you
talking about. If you could provide some details to back up these claims, there might be some recourse.
Quoted 9 lines
Don't get me wrong, I thing the 3ware
product is exceptionally good, I just wouldn't use the raid code given the choice of linux software raid.Currently running 10 linux software raid boxes -- mix of raid 1 and raid 5. Yes, I'm biased :-)
Michael
[email protected]
Scott
Quoted 1 line · Michael
Bear in mind that what you are calling "true hardware raid" is really a microprocessor programmed to do the raid algorithims. Usually these microprocessors are stretched to the limit to handle the throughput of modern udam drives. I don't know but I suspect that the small overhead use in the mmu for software raid has far more and faster throughput than any of these dedicated microprocessors..... and, you can see the code and know it is bug free or will be if you report the
Nod... many hardware RAIDs are turning out this way. A certain vendor whose name does -not- start with 'A' manages to make their hardware RAID perform so poorly, it is _half_ the speed of a software RAID using the same drives, on a single non-RAID Adaptec SCSI controller.
Most hardware RAID isn't 100% ASIC, but rather a general ASIC and firmware with the RAID code in it.
OTOH, hardware RAID really wins for situations like RAID-1, where you can -halve- the amount of data going across the PCI bus versus software RAID.
Jeff
Quoted 9 lines · Jeff Garzik
Quoted 7 lines · Michael
Bear in mind that what you are calling "true hardware raid" is really a microprocessor programmed to do the raid algorithims. Usually these microprocessors are stretched to the limit to handle the throughput of modern udam drives. I don't know but I suspect that the small overhead use in the mmu for software raid has far more and faster throughput than any of these dedicated microprocessors..... and, you can see the code and know it is bug free or will be if you report the
Quoted 5 lines
Nod... many hardware RAIDs are turning out this way. A certain vendor whose name does -not- start with 'A' manages to make their hardware RAID perform so poorly, it is _half_ the speed of a software RAID using the same drives, on a single non-RAID Adaptec SCSI controller.
Whoever could that be????
Quoted 3 lines
Most hardware RAID isn't 100% ASIC, but rather a general ASIC and firmware with the RAID code in it.
Well, the 8030x (i960) chips are really tailored for RAID. That's why they have a PCI-PCI bridge, XOR accelerator, and DRAM controller built into the package.
Quoted 3 lines
OTOH, hardware RAID really wins for situations like RAID-1, where you can -halve- the amount of data going across the PCI bus versus software RAID.
If performance is how you measure the quality of RAID, then you can't forget the benefit of a large write cache on the controller and the intelligence in the raid stack to use it well. However, the real benefit to hardware-accelerated RAID is the ability to guarantee that your reads and writes always succeed, regardless of the failure thrown at it. Of course, the number of cards out there that actually meet this goal is quite small, even from companies whose names start with 'A'.
Scott
Quoted 1 line · Rev. Jeffrey Paul
I have always avoided the promise fasttrak cards for exactly that reason ("hardware" being in quotes). It's my understanding that they're not much more than their udma cards with drivers that do most of the RAIDing.
Actually, Promise is one of the very few companies that are doing something innovative in RAID... Their hardware is not 100% software RAID nor 100% hardware RAID. They instead follow the model of network cards -- perform all key operations on the board, and let the host CPU handle the rest.
Quoted 1 line
It was also my understanding that the 3ware cards are true hardware raid and are up to the task of something like this, which would also explain why they're 4x the cost.
Correct.
Quoted 1 line
I am looking at a four disk raid5 or raid10, and it seems like the interrupt load from four drives on four channels might be a bit excesive. I'm going to be running critical services on the machine that the drives are in (namely mysql and nfs) and don't want to worry about performance.
Once your spindles can max out your PCI bus bandwidth, -then- you can start worrying about PCI bandwidth and interrupt load ;-)
Jeff
On Thu, 19 Feb 2004 at 7:38pm, Jeff Garzik wrote
Quoted 10 lines
Quoted 5 lines · Rev. Jeffrey Paul
I am looking at a four disk raid5 or raid10, and it seems like the interrupt load from four drives on four channels might be a bit excesive. I'm going to be running critical services on the machine that the drives are in (namely mysql and nfs) and don't want to worry about performance.
Once your spindles can max out your PCI bus bandwidth, -then- you can start worrying about PCI bandwidth and interrupt load ;-)
With some boards, it's actually pretty easy to do that. ;) Not with 4 disks, as the OP is doing, but 8 disks on one 64bit/33MHz bus (3ware 7500-8) can start to bump up against the PCI bandwidth. I've got a couple servers with 2 3wares and 16 disks, and I need to put the 3wares on separate PCI busses to make sure I get the full speed out of the combined array (hardware RAID5, software RAID0).
Signature · 3 lines
Joshua Baker-LePain Department of Biomedical Engineering Duke University
Quoted 5 lines · Holger Kiehl
In my opinion no. Most of the cheaper hardware raids are really just software raid solutions. For the more expansive once always remember you will always need drivers and there is no guarentee that you will get them in two or three years. Some vendors do no longer exist or no longer support that product.
Wasn't there some talk recently about standardizing the on-disk format for both hardware and software RAID?
Signature · 7 lines
Måns Rullgård [email protected] - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Måns Rullgård
Quoted 5 lines · Holger Kiehl
In my opinion no. Most of the cheaper hardware raids are really just software raid solutions. For the more expansive once always remember you will always need drivers and there is no guarentee that you will get them in two or three years. Some vendors do no longer exist or no longer support that product.
Wasn't there some talk recently about standardizing the on-disk format for both hardware and software RAID?
Correct, I read this with great interest and hope that all vendors do agree on a standart.
Holger
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Måns Rullgård
Quoted 5 lines · Holger Kiehl
In my opinion no. Most of the cheaper hardware raids are really just software raid solutions. For the more expansive once always remember you will always need drivers and there is no guarentee that you will get them in two or three years. Some vendors do no longer exist or no longer support that product.
Quoted 2 lines
Wasn't there some talk recently about standardizing the on-disk format for both hardware and software RAID?
It's happening.
"SNIA" has created an on-disk format "DDF", which both software and hardware RAID vendors have pretty much all agreed to use.
Of course, the spec is not public yet, so who knows what they have agreed to...
Jeff
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html