Software RAID5 with 2.6.0-test
Is software RAID5 stable in Linux 2.6.0-test7? A while back I tried running a software RAID5 with a 2.6.0-test kernel, and had to spend the evening running fsck. The corruption could have been caused by something other than the RAID layer. So, is it considered safe to use RAID5 in 2.6.0 kernels? I sort of dislike the try and see approach with matters like this.
Signature · 2 lines
Måns Rullgård [email protected]
My experience:
I'm running 2.6.0-test6 on a dual pentium 3 with software raid-5 across 5 disks on two different IDE hardware controllers (VIA and Promise). I've got a 224 GB reiserfs partition on that.
After 8 days uptime, it doesn't seem to have blown up yet. However I don't stress it heavily - just a nightly rsync or two which does a lot of reading and writing, and I export my music collection on it via NFS, which is a low level of read activity.
I mirror the RAID to an external firewire drive nightly for backup, since I don't trust it 100% either, yet.
I'd also be interested to know if there are known problems with software RAID5 in 2.6.
Hope that helps,
Torrey Hoffman
[email protected]
Quoted 6 lines · Måns Rullgård
Is software RAID5 stable in Linux 2.6.0-test7? A while back I tried running a software RAID5 with a 2.6.0-test kernel, and had to spend the evening running fsck. The corruption could have been caused by something other than the RAID layer. So, is it considered safe to use RAID5 in 2.6.0 kernels? I sort of dislike the try and see approach with matters like this.
Signature · 1 line
Torrey Hoffman <[email protected]>
Quoted 10 lines · Torrey Hoffman
My experience:
I'm running 2.6.0-test6 on a dual pentium 3 with software raid-5 across 5 disks on two different IDE hardware controllers (VIA and Promise). I've got a 224 GB reiserfs partition on that.
After 8 days uptime, it doesn't seem to have blown up yet. However I don't stress it heavily - just a nightly rsync or two which does a lot of reading and writing, and I export my music collection on it via NFS, which is a low level of read activity.
When I tried it, I was running 2.6.0-test4. The RAID5 was 4 120 GB Seagate disks on a Highpoint controller. On top of that, I had LVM, with ext3 fs. After just minutes, strange things started happening to files. Some had random bits changed in the inode, others were just trashed. e2fsck complained a great deal.
I went back to 2.4.21, which is working OK. A couple of things bother me, though. In the dmesg output there are many of these:
raid5: switching cache buffer size, 8192 --> 1024
raid5: switching cache buffer size, 1024 --> 4096
raid5: switching cache buffer size, 4096 --> 1024
raid5: switching cache buffer size, 1024 --> 4096
raid5: switching cache buffer size, 4096 --> 1024
ISTR reading somewhere, that this has a bad impact on performance.
The other thing that I don't like, is the performance of the RAID array. The disks individually give ~40 MB/s read speed, but the array only measures 25 MB/s. I was of the impression, that RAID5 would give read speeds at least equal to the underlying disks. Is this incorrect?
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 2 lines · Måns Rullgård
When I tried it, I was running 2.6.0-test4. The RAID5 was 4 120 GB Seagate disks on a Highpoint controller.
<snip>
Quoted 5 lines
The other thing that I don't like, is the performance of the RAID array. The disks individually give ~40 MB/s read speed, but the array only measures 25 MB/s. I was of the impression, that RAID5 would give read speeds at least equal to the underlying disks. Is this incorrect?
Was this a 4 port or 2 port HPT controller? Keep in mind, two disks on the same IDE channel severely degrades performance, *especially* with RAID.
Signature · 3 lines
Mvh, André Tomt [email protected]
Quoted 11 lines · Andre Tomt
Quoted 2 lines
When I tried it, I was running 2.6.0-test4. The RAID5 was 4 120 GB Seagate disks on a Highpoint controller.
<snip>
Quoted 5 lines
The other thing that I don't like, is the performance of the RAID array. The disks individually give ~40 MB/s read speed, but the array only measures 25 MB/s. I was of the impression, that RAID5 would give read speeds at least equal to the underlying disks. Is this incorrect?
Quoted 3 lines
Was this a 4 port or 2 port HPT controller? Keep in mind, two disks on the same IDE channel severely degrades performance, *especially* with RAID.
It's a four port SATA controller. I'd never even think about placing two disks on the same cable.
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 6 lines · Måns Rullgård
Quoted 3 lines
Was this a 4 port or 2 port HPT controller? Keep in mind, two disks on the same IDE channel severely degrades performance, *especially* with RAID.
It's a four port SATA controller. I'd never even think about placing two disks on the same cable.
You can't either, considering it is SATA :-)
However, I wouln't count on superior performance from software based RAID 5 (ata/fakeraid or otherwise), that is whats real raid controllers are for.
Out of couriosity, how well is it performing in lets say.. a RAID10 or RAID1 setup?
Signature · 3 lines
Mvh, André Tomt [email protected]
Quoted 4 lines · Andre Tomt
Quoted 4 lines
Quoted 3 lines
Was this a 4 port or 2 port HPT controller? Keep in mind, two disks on the same IDE channel severely degrades performance, *especially* with RAID.
Quoted 8 lines
Quoted 2 lines
It's a four port SATA controller. I'd never even think about placing two disks on the same cable.
You can't either, considering it is SATA :-)
However, I wouln't count on superior performance from software based RAID 5 (ata/fakeraid or otherwise), that is whats real raid controllers are for.
I've seen reports of people obtaining nearly 100 MB/s from software RAID5.
Quoted 2 lines
Out of couriosity, how well is it performing in lets say.. a RAID10 or RAID1 setup?
I didn't try. I might, of course.
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 5 lines · Andre Tomt
Quoted 4 lines · Måns Rullgård
Quoted 3 lines
Was this a 4 port or 2 port HPT controller? Keep in mind, two disks on the same IDE channel severely degrades performance, *especially* with RAID.
Quoted 8 lines
Quoted 2 lines
It's a four port SATA controller. I'd never even think about placing two disks on the same cable.
You can't either, considering it is SATA :-)
However, I wouln't count on superior performance from software based RAID 5 (ata/fakeraid or otherwise), that is whats real raid controllers are for.
While an overloaded system may benefit from offloaded the CPU requirements of RAID, unless you go to a very expensive external unit the kernel RAID will usually outperform the inexpensive RAID embedded on a controller. The kernel simply knows more about the data needs and can can do things a controller can't.
You can also use only part of a drive, by partition, in a RAID group, leaving the rest to be used as dumb disk. Very cost effective, if only a part of the data justified mirroring, you don't need to mirror whole drives. Yes, that's a cost vs. speed tradeoff, but an effective one where reliabiity is needed, but performance is not critical.
Signature · 3 lines
bill davidsen <[email protected]> CTO, TMR Associates, Inc Doing interesting things with little computers since 1979.
Quoted 9 lines · Bill Davidsen
Quoted 3 lines
However, I wouln't count on superior performance from software based RAID 5 (ata/fakeraid or otherwise), that is whats real raid controllers are for.
While an overloaded system may benefit from offloaded the CPU requirements of RAID, unless you go to a very expensive external unit the kernel RAID will usually outperform the inexpensive RAID embedded on a controller. The kernel simply knows more about the data needs and can can do things a controller can't.
What about the RAID controllers in the $400 category? Surely, they must be doing something better than the $50 fakeraid controllers.
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 14 lines · Måns Rullgård
Quoted 9 lines · Bill Davidsen
Quoted 3 lines
However, I wouln't count on superior performance from software based RAID 5 (ata/fakeraid or otherwise), that is whats real raid controllers are for.
While an overloaded system may benefit from offloaded the CPU requirements of RAID, unless you go to a very expensive external unit the kernel RAID will usually outperform the inexpensive RAID embedded on a controller. The kernel simply knows more about the data needs and can can do things a controller can't.
Quoted 3 lines
What about the RAID controllers in the $400 category? Surely, they must be doing something better than the $50 fakeraid controllers.
Yes, but follow this logic.
1)You are willing to devote 10% of 2Ghz xeon to software raid. 2)A $500+ controller has a 100Mhz proccessor.
Thus just from this you could guess that software raid has x2 as many
clock cycles availble to it. It's even worse when you realize the 2Ghz xeon is a better proccessor in many more ways than just clock cycles.
The reasons to use that 500-1000 raid controller are due the features
the controller gives you. Things like gracefull recovery from bad sectors, and SAF-TE support.
Signature · 3 lines
Once you have their hardware. Never give it back. (The First Rule of Hardware Acquisition) Sam Flory <[email protected]>
Quoted 6 lines · Samuel Flory
Quoted 3 lines
What about the RAID controllers in the $400 category? Surely, they must be doing something better than the $50 fakeraid controllers.
Yes, but follow this logic.
Quoted 7 lines
1)You are willing to devote 10% of 2Ghz xeon to software raid. 2)A $500+ controller has a 100Mhz proccessor.
Thus just from this you could guess that software raid has x2 as many clock cycles availble to it. It's even worse when you realize the 2Ghz xeon is a better proccessor in many more ways than just clock cycles.
How about this logic:
1) If the processor on the RAID controller can handle the full bandwidth of the disks, it's fast enough.
2) If someone else does the 10% work, the CPU can do 10% more work.
Signature · 2 lines
Måns Rullgård [email protected]
On Fri, Oct 17, 2003 at 07:44:31PM +0200, Måns Rullgård wrote:
...
Quoted 3 lines
What about the RAID controllers in the $400 category? Surely, they must be doing something better than the $50 fakeraid controllers.
The RAID controllers in the $400 category are exactly the ones who include (inferior) processors, and add an extra layer in the 'chain of command'.
The inexpensive 'fakeraid' controllers include some form of SW RAID in their driver, and could therefore perform as well as Linux SW RAID (if the driver's RAID code is as clever and efficient as the Linux SW RAID code - which it may not be).
Ironic, maybe ;)
I have never tried using one of the 'fakeraid' controllers, so I can't speak for their actual real-world performance. They could include a crap IDE controller, or they could have crap drivers - both could give poor performance.
Could, would, should, don't know...
However, the "hardware RAID" controllers that were discussed here would be the ones in the $400->$(obscene) price range.
Now that I'm posting anyway - I thought of a plus for the HW RAID controllers (hey, they're way behind on the scoreboard so far, so I might as well be a gentleman and give them a point or two):
*) Battery backed write cache
This will allow the controller to say 'ok I'm done with your sync()', way before the data actually reaches the disk platters. For some workloads this can be a big win. For some odd reason, it seems that 'most' (read: all that I could find) HW RAID controllers are unable to control more than 256 MB of memory - which is odd. That amound of memory is almost so low that it defeats the purpose of having it in the first place. But again, some workloads may well benefit. And this is something you just can't do with SW RAID (at least not before someone implements support for an NVRAM buffer store in the Linux I/O layer).
While on that topic: SDRAM PCI cards with battery backup can be had relatively cheap up to at least a few gigabytes. It'd be pretty cool to have the kernel recognize that for buffer storage. I could see the fun in doing half a million random writes and a sync(), and having the system tell me it's done the microsecond I issue the sync().
Enough with the daydreaming already ;)
Signature · 6 lines
................................................................ : [email protected] : And I see the elder races, : :.........................: putrid forms of man : : Jakob Østergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............:
Quoted 1 line · Måns Rullgård
Samuel Flory <[email protected]> writes:
...
Quoted 9 lines
Quoted 3 lines
many clock cycles availble to it. It's even worse when you realize the 2Ghz xeon is a better proccessor in many more ways than just clock cycles.
How about this logic:
1) If the processor on the RAID controller can handle the full bandwidth of the disks, it's fast enough.
2) If someone else does the 10% work, the CPU can do 10% more work.
3) You have a four year old machine - one day the RAID controller dies. The company that produced it has been acquired by someone else, and the product is no longer availble. Can you get a new adapter with
firmware that can actually read your disks? Or are your data lost?
Can you find a replacement controller on e-bay? And would you want to?
Anyway, not wanting to spread more FUD than stricly necessary: It's a
matter of cost/benefit and risk management. Everyone has their
personal preferences on this - and even if it wasn't so, let's not begin to pretend that there is a simple answer to what's "best".
Signature · 6 lines
................................................................ : [email protected] : And I see the elder races, : :.........................: putrid forms of man : : Jakob Østergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............:
Quoted 10 lines · Jakob Oestergaard
Quoted 9 lines
Quoted 3 lines
many clock cycles availble to it. It's even worse when you realize the 2Ghz xeon is a better proccessor in many more ways than just clock cycles.
How about this logic:
1) If the processor on the RAID controller can handle the full bandwidth of the disks, it's fast enough.
2) If someone else does the 10% work, the CPU can do 10% more work.
Quoted 6 lines
3) You have a four year old machine - one day the RAID controller dies. The company that produced it has been acquired by someone else, and the product is no longer availble. Can you get a new adapter with
firmware that can actually read your disks? Or are your data lost?
Can you find a replacement controller on e-bay? And would you want to?
What are backups for? That argument applies to any controller, for anything. What if you wanted to read those old 8" floppies? Or that hard disk from the PDP/11.
If a four year old RAID controller breaks, maybe it's time to get new disks anyway.
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 16 lines · Måns Rullgård
Quoted 6 lines · Samuel Flory
Quoted 3 lines
What about the RAID controllers in the $400 category? Surely, they must be doing something better than the $50 fakeraid controllers.
Yes, but follow this logic.
Quoted 7 lines
1)You are willing to devote 10% of 2Ghz xeon to software raid. 2)A $500+ controller has a 100Mhz proccessor.
Thus just from this you could guess that software raid has x2 as many clock cycles availble to it. It's even worse when you realize the 2Ghz xeon is a better proccessor in many more ways than just clock cycles.
Quoted 5 lines
How about this logic:
1) If the processor on the RAID controller can handle the full bandwidth of the disks, it's fast enough.
2) If someone else does the 10% work, the CPU can do 10% more work.
And as has been addressed on this list before:
3) If the additional I/O traffic of the RAID can be kept off of the system busses the overall system throughput goes up.
Once the CPU reaches a certain level of performance it is the I/O and memory that limit things. Do you really want to pollute L1 cache with RAID-5? When the $400 RAID server card can saturate the PCI buss it doesn't matter how much spare CPU you have, SW RAID will not be able to match the performance.
Signature · 5 lines
________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: [email protected] Remember Cernan and Schmitt
In article <[email protected]>, =?iso-8859-1?q?M=E5ns_Rullg=E5rd?= <[email protected]> wrote:
| Bill Davidsen <[email protected]> writes: | | >> However, I wouln't count on superior performance from software based | >> RAID 5 (ata/fakeraid or otherwise), that is whats real raid controllers | >> are for. | > | > While an overloaded system may benefit from offloaded the CPU | > requirements of RAID, unless you go to a very expensive external unit | > the kernel RAID will usually outperform the inexpensive RAID embedded on | > a controller. The kernel simply knows more about the data needs and can | > can do things a controller can't. | | What about the RAID controllers in the $400 category? Surely, they | must be doing something better than the $50 fakeraid controllers.
Unfortunately my experience is with either the cheap motherboard ones I get for free and the IBM ServRAID controllers which are kind of in the "more than that" category. The benefit of the m/b ones is that with RAID-1 you will get a boot if the boot drive fails. Period. Unlike optimal RAID-1 which will read mirrored data from any non-busy drive, the cheap ones seem to do fail-over only, and read the second drive only if the first fails, or even require both drives to be on the same cable, ensuring that the bus will be busy when either drive is busy. They do buy reliability, however, so there are places where they are very cost effective.
The IBM controllers are everything you ever wanted in a controller. It supports four SCSI busses for bandwidth, all the RAID types including 5e, and it has Linux config software so you can do most reconfigures on a running system. I would choose them over anything else I've personally used, mainly Adaptec and PERC. They are really great for multi-TB storage systems.
But in between is where software RAID wins. In many medium applications, the limit isn't CPU, and it isn't bus bandwidth, it's the base speed of the disk access counted as either access time or bandwidth depending on the application. So the major benefits of midprice RAID for ATAPI/SATA aren't realized, performance is the issue.
It's obvious that the o/s has information which a RAID controller doesn't in terms of ordering i/o. And similarly true that adding a GB to the main system is going to help overall more than a GB on a controller (as noted, 256MB seems the limit on some), because the o/s will use it more effectively for buffers, and because it will eliminate some i/o entirely, particularly swap. And nothing speeds a disk operation more than avoiding the need to do it.
Anyone who says that any one solution is best for everything is clearly misguided, but I do think that for many midsize applications that software RAID is the most cost effective solution, and vs. moderately priced controllers very likely to be the higher performance solution as well.
Signature · 3 lines
bill davidsen <[email protected]> CTO, TMR Associates, Inc Doing interesting things with little computers since 1979.
In article <[email protected]>,
Jakob Oestergaard <[email protected]> wrote:
| Now that I'm posting anyway - I thought of a plus for the HW RAID | controllers (hey, they're way behind on the scoreboard so far, so I | might as well be a gentleman and give them a point or two): | *) Battery backed write cache | | This will allow the controller to say 'ok I'm done with your sync()', | way before the data actually reaches the disk platters. For some | workloads this can be a big win.
Unless the drives are battery backed up as well, I'm not sure that this is a good thing, or at least a safe thing. And if a write error happens after the controller tells you the sync() is done? That's a question, not a comment, I'm not sure Linux software RAID would relocate if the drive was out of spare sectors, either, but it at least could.
I don't question your statement that caching helps performance, but I think there is some loss of reliability. I have no numbers to estimate the effect, so take it as a comment only.
Signature · 3 lines
bill davidsen <[email protected]> CTO, TMR Associates, Inc Doing interesting things with little computers since 1979.
Quoted 4 lines · bill davidsen
In article <[email protected]>, =?iso-8859-1?q?M=E5ns_Rullg=E5rd?= <[email protected]> wrote:
| Bill Davidsen <[email protected]> writes:
Quoted 10 lines
Unfortunately my experience is with either the cheap motherboard ones I get for free and the IBM ServRAID controllers which are kind of in the "more than that" category. The benefit of the m/b ones is that with RAID-1 you will get a boot if the boot drive fails. Period. Unlike optimal RAID-1 which will read mirrored data from any non-busy drive, the cheap ones seem to do fail-over only, and read the second drive only if the first fails, or even require both drives to be on the same cable, ensuring that the bus will be busy when either drive is busy. They do buy reliability, however, so there are places where they are very cost effective.
In this situation that you describe, how does one recover from this? Do these cheap motherboard RAID-1's do an autorebuild when the failed disk is replaced? (Obviously not hot-swap, but on reboot to replace the drive).
Quoted 7 lines
The IBM controllers are everything you ever wanted in a controller. It supports four SCSI busses for bandwidth, all the RAID types including 5e, and it has Linux config software so you can do most reconfigures on a running system. I would choose them over anything else I've personally used, mainly Adaptec and PERC. They are really great for multi-TB storage systems.
Do they compare favorable against 3Ware as well?
Quoted 5 lines
Anyone who says that any one solution is best for everything is clearly misguided, but I do think that for many midsize applications that software RAID is the most cost effective solution, and vs. moderately priced controllers very likely to be the higher performance solution as well.
Is this statement true, in your opinion, for RAID-5 as well?
-chuck
Hi!
Quoted 7 lines
Now that I'm posting anyway - I thought of a plus for the HW RAID controllers (hey, they're way behind on the scoreboard so far, so I might as well be a gentleman and give them a point or two):
*) Battery backed write cacheThis will allow the controller to say 'ok I'm done with your
sync()', way before the data actually reaches the disk platters. For some
Well, this would mean not only battery-backed controller, but battery-backed drives, too. (Unless you want to assume that memory-backup-battery never goes empty).
Quoted 5 lines
While on that topic: SDRAM PCI cards with battery backup can be had relatively cheap up to at least a few gigabytes. It'd be pretty cool to have the kernel recognize that for buffer storage. I could see the fun in doing half a million random writes and a
sync(), and having the system tell me it's done the microsecond I issue thesync().
Sync would still take some time as it still makes sense to do writeback... Main memory is still faster than PCI.
Pavel
Signature · 2 lines
Pavel Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...