RAID5E
Where I was working most recently some systems were using RAID5E (RAID5 with both the parity and hot spare distributed). This seems to be highly desirable for small arrays, where spreading head motion over one more drive will improve performance, and in all cases where a rebuild to the hot spare will avoid a bottleneck on a single drive.
Is there any plan to add this capability?
Signature · 3 lines
bill davidsen <[email protected]> CTO TMR Associates, Inc Doing interesting things with small computers since 1979
Quoted 3 lines · Bill Davidsen
Where I was working most recently some systems were using RAID5E (RAID5 with both the parity and hot spare distributed). This seems to be highly desirable for small arrays, where spreading head motion over one more drive will improve performance, and in all cases where a rebuild to the hot spare will avoid a bottleneck on a single drive.
Is there any plan to add this capability?
What advantage does that have over raid6? You use exactly as many drives (n+2), with the disadvantage of having to do a rebuild without parity when a drive fails and a raid failure at a double disk failure.
/Mattias Wadenstein
Quoted 7 lines · Mattias Wadenstein
Quoted 6 lines · Bill Davidsen
Where I was working most recently some systems were using RAID5E (RAID5 with both the parity and hot spare distributed). This seems to be highly desirable for small arrays, where spreading head motion over one more drive will improve performance, and in all cases where a rebuild to the hot spare will avoid a bottleneck on a single drive.
Quoted 5 lines
Quoted 1 line
Is there any plan to add this capability?
What advantage does that have over raid6? You use exactly as many drives (n+2), with the disadvantage of having to do a rebuild without parity when a drive fails and a raid failure at a double disk failure.
Advantage:
- Easier to calculate the checksum (RAID5 XOR instead of a generator polynome with RAID6)
- Higher throughput compared to standard RAID5
- Actually uses the hot spare
Disadvantage:
- Doesn't protect against double disk failures, but RAID5 also doesn't
Note that you could also do RAID6E.
Erik
Signature · 2 lines
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 -- | Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
Quoted 8 lines · Mattias Wadenstein
Quoted 3 lines · Bill Davidsen
Where I was working most recently some systems were using RAID5E (RAID5 with both the parity and hot spare distributed). This seems to be highly desirable for small arrays, where spreading head motion over one more drive will improve performance, and in all cases where a rebuild to the hot spare will avoid a bottleneck on a single drive.
Is there any plan to add this capability?
What advantage does that have over raid6? You use exactly as many drives (n+2), with the disadvantage of having to do a rebuild without parity when a drive fails and a raid failure at a double disk failure.
The write overhead of RAID-6 vs. RAID-5 is much higher, both in terms of CPU and disk operations generated. See below re double drive failure and RAID-5 with spare.
As a starting point consider RAID-4, and four drives, one of which is a hot spare (I did clearly note small arrays). For reads you only have striping over two drives, for writes the parity drive has double the i/o of the data drives. The spare may spin down or not, but if it fails it will be just when you need it most.
With RAID-5 the load is evenly spread over the entire array of active drives. This improves both read and write performance, but the hot spare is still not used in normal operation. Note that in the two drive failure mode the only critical time is during the rebuild to hot spare, after that a second drive failure is tolerated without loss of data.
With RAID-5E the i/o is spread over all four drives, which gives a further improvement in the read and write performance, particularly if large reads and writes (relative to clunk size) are common. There is no hot spare which is mostly unused, and rebuild to the hot spare is distributed over all remaining drives.
Drawbacks: you can't share a single hot spare between multiple arrays. That's not really common, and I mention it for completeness. The other issue is that when a failed drive is replaced the rebuild appears to be somewhat complex because the new drive doesn't just become the new hot spare. The same issues apply regarding two drives failing, but the rebuild time is usually shorter, so the exposure is less.
And finally, with RAID-6 and the same number of drives, if you have a failure there's no hot spare, and the arrray runs in degraded mode until the failed drive is replaced. That is a different balance between reliability and performance after failure, and needs to be a per-instance choice.
Signature · 3 lines
bill davidsen <[email protected]> CTO TMR Associates, Inc Doing interesting things with small computers since 1979
Quoted 7 lines · On Wednesday May 31, [email protected] wrote:
Where I was working most recently some systems were using RAID5E (RAID5 with both the parity and hot spare distributed). This seems to be highly desirable for small arrays, where spreading head motion over one more drive will improve performance, and in all cases where a rebuild to the hot spare will avoid a bottleneck on a single drive.
Is there any plan to add this capability?
I thought about it briefly....
As I understand it, the layout of raid5e when non-degraded is very similar to raid6 - however the 'Q' block is simply not used. This would be trivial to implement.
The interesting bit comes when a device fails and you want to rebuild that distributed spare.
There are two possible ways that you could do this:
1/ Leave the spare where it is and write the correct data into each
spare. This would be fairly easy but would leave an array with an very ... interesting layout of data.
When you add a replacement you just move everything back.
2/ reshape the array to be a regular raid5 layout. This would be hard
to do well without NVRAM as you are moving live data, but would result in a neat and tidy array. Ofcourse adding a drive back in would be interesting again...
I had previously only thought of option '2', and so discarded the idea as not worth the effort. The more I think about it, the more possible option 1 sounds.
I've put it back on my todo list, but I don't expect to get to it this year. Ofcourse if someone else wants to give it a try, I'm happy to make suggestions and review code.
NeilBrown
Quoted 29 lines · Neil Brown
Quoted 5 lines · On Wednesday May 31, [email protected] wrote:
Where I was working most recently some systems were using RAID5E (RAID5 with both the parity and hot spare distributed). This seems to be highly desirable for small arrays, where spreading head motion over one more drive will improve performance, and in all cases where a rebuild to the hot spare will avoid a bottleneck on a single drive.
Is there any plan to add this capability?
I thought about it briefly....
As I understand it, the layout of raid5e when non-degraded is very similar to raid6 - however the 'Q' block is simply not used. This would be trivial to implement.
The interesting bit comes when a device fails and you want to rebuild that distributed spare.
There are two possible ways that you could do this:1/ Leave the spare where it is and write the correct data into each spare. This would be fairly easy but would leave an array with an very ... interesting layout of data.
When you add a replacement you just move everything back.2/ reshape the array to be a regular raid5 layout. This would be hard to do well without NVRAM as you are moving live data, but would result in a neat and tidy array. Ofcourse adding a drive back in would be interesting again...
Quoted 7 lines
I had previously only thought of option '2', and so discarded the idea as not worth the effort. The more I think about it, the more possible option 1 sounds.
I've put it back on my todo list, but I don't expect to get to it this year. Ofcourse if someone else wants to give it a try, I'm happy to make suggestions and review code.
I do appreciate being too busy, I'm just glad I have been able to clarify the tradeoffs of RAID5e, and get it on your list at all. I did look at the code a bit, and it would seem that if the "rebuild to hot spare" code is modified to handle a distributed spare, then it looks as if RAID6e might pretty much fall out. Feel free to tell me I'm dreaming.
Signature · 3 lines
bill davidsen <[email protected]> CTO TMR Associates, Inc Doing interesting things with small computers since 1979