Skip to contentSkip to conversation

Distributed spares

DISCUSSION Bill Davidsen 9 participants 16 messages Oct 13, 2008 – Oct 20, 2008 MD RAID

lore
Share & tools
lore
#0 lore raw

Over a year ago I mentioned RAID-5e, a RAID-5 with the spare(s) distributed over multiple drives. This has come up again, so I thought I'd just mention why, and what advantages it offers.

By spreading the spare over multiple drives the head motion of normal access is spread over one (or several) more drives. This reduces seeks, improves performance, etc. The benefit reduces as the number of drives in the array gets larger, obviously with four drives using only three for normal operation is slower than four, etc. And by using all the drives all the time, the chance of a spare being undetected after going bad is reduced.

This becomes important as array drive counts shrink. Lower cost for drives ($100/TB!), and attempts to drop power use by using fewer drives, result in an overall drop in drive count, important in serious applications.

All that said, I would really like to bring this up one more time, even if the answer is "no interest."

Signature · 3 lines
Bill Davidsen <[email protected]>
 "Woe unto the statesman who makes war without a reason that will still
 be valid when the war is over..." Otto von Bismark 
#1 lore raw
Quoted 18 lines · Bill Davidsen

Over a year ago I mentioned RAID-5e, a RAID-5 with the spare(s) distributed over multiple drives. This has come up again, so I thought I'd just mention why, and what advantages it offers.

By spreading the spare over multiple drives the head motion of normal access is spread over one (or several) more drives. This reduces seeks, improves performance, etc. The benefit reduces as the number of drives in the array gets larger, obviously with four drives using only three for normal operation is slower than four, etc. And by using all the drives all the time, the chance of a spare being undetected after going bad is reduced.

This becomes important as array drive counts shrink. Lower cost for drives ($100/TB!), and attempts to drop power use by using fewer drives, result in an overall drop in drive count, important in serious applications.

All that said, I would really like to bring this up one more time, even if the answer is "no interest."

Signature · 8 lines
Bill Davidsen <[email protected]>
"Woe unto the statesman who makes war without a reason that will still
be valid when the war is over..." Otto von Bismark 

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Bill,

Not a bad idea; however, can the same not be acheived (somewhat) by performing daily/smart, weekly/long tests on the drive to validate its health? I find this to work fairly well on a large scale.

Justin.

#2 lore raw

Just my two cents.... Those daily smart tests or regularly running badblocks are fine, but they're not 'real' load. A test can't prove everything is right, it can at best only prove it didn't find anything wrong. Distributed spare would exert 'real' load on the spare because the spare disks ARE the live disks.

On a side note, it would be handy to have a daemon that could run in the background on large raid1's, or raid6', and once a month, pull each disk out of the array sequentially, completely overwrite it, check it with badblocks several times, do the smart tests, etc..., then rejoin it, reinstall grub, wait an hour and move on. The point being, of course, to kill weak drives off early and in a controlled manor. It would be even nicer if there were a way to hot-transfer one raid component to another without setting anything faulty. I suppose you could make all the components of the real array be single disk raid1 arrays for that purpose. Then you could have one extra disk set aside for this sort of scrubbing, and never even be down one of your parities. I guess I should add that onto my todo list....

-Billy

Quoted 44 lines · Justin Piszcz
Quoted 18 lines · Bill Davidsen

Over a year ago I mentioned RAID-5e, a RAID-5 with the spare(s) distributed over multiple drives. This has come up again, so I thought I'd just mention why, and what advantages it offers.

By spreading the spare over multiple drives the head motion of normal access is spread over one (or several) more drives. This reduces seeks, improves performance, etc. The benefit reduces as the number of drives in the array gets larger, obviously with four drives using only three for normal operation is slower than four, etc. And by using all the drives all the time, the chance of a spare being undetected after going bad is reduced.

This becomes important as array drive counts shrink. Lower cost for drives ($100/TB!), and attempts to drop power use by using fewer drives, result in an overall drop in drive count, important in serious applications.

All that said, I would really like to bring this up one more time, even if the answer is "no interest."

Quoted 9 lines

--
Bill Davidsen <[email protected]>
"Woe unto the statesman who makes war without a reason that will still be valid when the war is over..." Otto von Bismark
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Bill,

Not a bad idea; however, can the same not be acheived (somewhat) by performing daily/smart, weekly/long tests on the drive to validate its health? I find this to work fairly well on a large scale.

Justin.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
Quoted 7 lines · Billy Crook

Just my two cents.... Those daily smart tests or regularly running badblocks are fine, but they're not 'real' load. A test can't prove everything is right, it can at best only prove it didn't find anything wrong. Distributed spare would exert 'real' load on the spare because the spare disks ARE the live disks.

Quoted 12 lines

On a side note, it would be handy to have a daemon that could run in the background on large raid1's, or raid6', and once a month, pull each disk out of the array sequentially, completely overwrite it, check it with badblocks several times, do the smart tests, etc..., then rejoin it, reinstall grub, wait an hour and move on. The point being, of course, to kill weak drives off early and in a controlled manor. It would be even nicer if there were a way to hot-transfer one raid component to another without setting anything faulty. I suppose you could make all the components of the real array be single disk raid1 arrays for that purpose. Then you could have one extra disk set aside for this sort of scrubbing, and never even be down one of your parities. I guess I should add that onto my todo list....

I have also been thinking a little on this. My idea is that if bit errors develop on disks, then there is first maybe one bit error, and the crc check on the disk sectors then finds and corrects these.

If you rewrite such bit errors, then that bit error will be corrected, and you prevent the one-bit error from developing to a two-bit error that is not correctable by the CRC.

Is there some merit to this idea?

Furthermore, if bad luck has striken, then in the case of mirrored RAIDs you could - when crc fails, then see that this is the block in error and recreate it from the redundant info, Would be good for raid1, raid10, raid5, raid6. If the block then could not be written without errors, then it could be added to a bad blocks list and remapped.

I think there is nothing novel in a scheme like this, but I would like to know if it is implemented somewhere. Articles say that bit errors on disks are becoming more and more frequent, so schemes like this may help the scary scenarion somewhat.

best regards
keld

#10 lore raw
In reply to Bill Davidsen
Quoted 12 lines · On Monday October 13, [email protected] wrote:

Over a year ago I mentioned RAID-5e, a RAID-5 with the spare(s) distributed over multiple drives. This has come up again, so I thought I'd just mention why, and what advantages it offers.

By spreading the spare over multiple drives the head motion of normal access is spread over one (or several) more drives. This reduces seeks, improves performance, etc. The benefit reduces as the number of drives in the array gets larger, obviously with four drives using only three for normal operation is slower than four, etc. And by using all the drives all the time, the chance of a spare being undetected after going bad is reduced.

Quoted 6 lines

This becomes important as array drive counts shrink. Lower cost for drives ($100/TB!), and attempts to drop power use by using fewer drives, result in an overall drop in drive count, important in serious applications.

All that said, I would really like to bring this up one more time, even if the answer is "no interest."

How are your coding skills?

The tricky bit is encoding the new state.
We can not longer tell the difference between "optimal" and "degraded" based on the number of in-sync devices. We also need some state flag to say that the "distributed spare" has been constructed. Maybe that could be encoded in the "layout".

We would also need to allow a "recovery" pass to happen without having actually added any spares, or having any non-insync devices. That probably means passing the decision "is a recovery pending" down into the personality rather than making it in common code. Maybe have some field in the mddev structure which the personality sets if a recovery is worth trying. Or maybe just try it anyway after any significant change and if the personality finds nothing can be done it aborts.

I'm happy to advise on, review, and eventually accept patches.

NeilBrown

#4 lore raw
In reply to Keld Jørn Simonsen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Keld" == Keld Jørn Simonsen <[email protected]> writes:

Keld> I have also been thinking a little on this. My idea is that if Keld> bit errors develop on disks, then there is first maybe one bit Keld> error, and the crc check on the disk sectors then finds and Keld> corrects these.

Keld> If you rewrite such bit errors, then that bit error will be Keld> corrected, and you prevent the one-bit error from developing to Keld> a two-bit error that is not correctable by the CRC.

I think you are assuming that disks are much simpler than they actually are.

A modern disk drive protects a 512-byte sector with a pretty strong ECC that's capable of correcting errors up to ~50 bytes. Yes, that's bytes.

Also, many drive firmwares will internally keep track of problematic media areas and rewrite or reallocate affected blocks. That includes stuff like rewriting sectors that are susceptible to bleed due to being adjacent to write hot spots.

Signature · 5 lines
Martin K. Petersen	Oracle Linux Engineering
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#7 lore raw

non-degraded component replacement was Re: Distributed spares

In reply to Billy Crook
Quoted 6 lines · Billy Crook

It would be even nicer if there were a way to hot-transfer one raid component to another without setting anything faulty. I suppose you could make all the components of the real array be single disk raid1 arrays for that purpose. Then you could have one extra disk set aside for this sort of scrubbing, and never even be down one of your parities. I guess I should add that onto my todo list....

IMHO This one should be high on the todo list. Especially if it's a pre-requisite for other improvements to resilience.

Right now, if a drive fails or shows signs of going bad then you get into a very
risky situation.

I'm sure most here know that the risk is because removing the failing drive and installing a good one to re-sync puts you in a very vulnerable position; if another drive fails (even one bad block) then you lose data.

The solution involves raid1 - but it needs a twist of raid5/6.

http://arctic.org/~dean/proactive-raid5-disk-replacement.txt

I think this is what was discussed:

Assume md0 has drives A B C D
D is failing
E is new

* add E as spare
* set E to mirror 'failing' drive D (with bitmap?)
* subsequent writes go to both D+E
* recover 99+% of data from D to E by simple mirroring
* any md0 or D->E read failures on D are recovered from reading ABC not E unless E is in sync. D is not failed out. (and it's these tricks that stops users from doing all this manually)
* any md0 sector read failure on ABC can still (hopefully) be read from D even if not yet mirrored to E (also not possible
* once E is mirrored, D is removed and the job is done

Personally I think this feature is more important than the reshaping requests; of course that's just one opinion :)

David

#5 lore raw
In reply to Martin K. Petersen
Quoted 14 lines · Martin K. Petersen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Keld" == Keld Jørn Simonsen <[email protected]> writes:

Keld> I have also been thinking a little on this. My idea is that if Keld> bit errors develop on disks, then there is first maybe one bit Keld> error, and the crc check on the disk sectors then finds and Keld> corrects these.

Keld> If you rewrite such bit errors, then that bit error will be Keld> corrected, and you prevent the one-bit error from developing to Keld> a two-bit error that is not correctable by the CRC.

I think you are assuming that disks are much simpler than they actually are.

Quoted 8 lines

A modern disk drive protects a 512-byte sector with a pretty strong ECC that's capable of correcting errors up to ~50 bytes. Yes, that's bytes.

Also, many drive firmwares will internally keep track of problematic media areas and rewrite or reallocate affected blocks. That includes stuff like rewriting sectors that are susceptible to bleed due to being adjacent to write hot spots.

Good to know. Could yo tell me if this is actually true for normal state-of-the art SATA disks, or only true for more expensive disks? Do you have a good reference for it.

best regards
keld
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#8 lore raw

Re: non-degraded component replacement was Re: Distributed spares

In reply to David Greaves
Quoted 10 lines · David Greaves
Quoted 6 lines · Billy Crook

It would be even nicer if there were a way to hot-transfer one raid component to another without setting anything faulty. I suppose you could make all the components of the real array be single disk raid1 arrays for that purpose. Then you could have one extra disk set aside for this sort of scrubbing, and never even be down one of your parities. I guess I should add that onto my todo list....

IMHO This one should be high on the todo list. Especially if it's a pre-requisite for other improvements to resilience.

Here's the process as I thought it out. I'm sure it can be improved upon:

Component C will be the current drive that one wishes to take out of service. Component N will be the new drive that one wishes to put in service in place of component C.

Redirect incoming writes from component C to component C AND N. Check to make sure component N is same size or larger than C. Create counter curBlock to store position of the drive copy, (initialize counter at 0).
While curBlock < component C's block count:
Copy block curBlock from component C to curBlock on component N.
If copy fails, then try to construct that block from the other
disks using parity and apply that to component N.
Increment curBlock.
Once the copy is complete, optionally verify by comparing both components. Set curBlock to 0 again.
While curBlock < component C's block count:
Compare curBlock on component C to curBlock on component N.
If compare fails, then terminate with error and stop mirroring writes to N.
Increment curBlock.
Redirect reads from component N only.
Stop writing to component C, and only write to N.
Present some notification that this process is done.

At all points during the process, redundancy should be as good as or better than before the process. The process can be aborted at any time without disruption to the array. This could be represented IMHO with some different status designator character in /proc/mdstat like M (for migrating), and the name for this capability, I'd call "Hot raid component migration". Just so long as people realise its an option for replacing raid components more safely. I bet the majority of the code needed is already in the raid1 personality. You could accomplish the same thing by building your 'real' array ontop of single-disk raid1 arrays, but oh that would be messy to look at!

#6 lore raw
In reply to Martin K. Petersen
Quoted 33 lines

-----Original Message-----
From: [email protected] [mailto:linux-raid- [email protected]] On Behalf Of Martin K. Petersen
Sent: Tuesday, October 14, 2008 5:12 AM
To: Keld Jørn Simonsen
Cc: Billy Crook; Justin Piszcz; Bill Davidsen; Neil Brown; Linux RAID
Subject: Re: Distributed spares

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Keld" == Keld Jørn Simonsen <[email protected]> writes:

Keld> I have also been thinking a little on this. My idea is that if Keld> bit errors develop on disks, then there is first maybe one bit Keld> error, and the crc check on the disk sectors then finds and Keld> corrects these.

Keld> If you rewrite such bit errors, then that bit error will be Keld> corrected, and you prevent the one-bit error from developing to Keld> a two-bit error that is not correctable by the CRC.

I think you are assuming that disks are much simpler than they actually are.

A modern disk drive protects a 512-byte sector with a pretty strong ECC that's capable of correcting errors up to ~50 bytes. Yes, that's bytes.

Also, many drive firmwares will internally keep track of problematic media areas and rewrite or reallocate affected blocks. That includes stuff like rewriting sectors that are susceptible to bleed due to being adjacent to write hot spots.

Quoted 7 lines

--
Martin K. Petersen Oracle Linux Engineering
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Martin is absolutely correct. Enterprise class drives have come a long way. They will scan and fix blocks (but certainly not 100% of them) in background. The $99 disk drives you get at the local computer retailer now even have limited BGMS / repair capability.

If you run the built-in diags on disk drives, you can be presented with a list of known bad blocks, or when you boot a disk drive, sometimes you can get a bad block display in POST.

How about a baby step? When you run offline or online tests, or even when you run media scans, you get a list of known defects. How about a program that rewrites a RAID1/3/5/6 stripe, and you just pass it the physical device name and known block number?

As for checking out a disk ..

The prior poster's idea about putting the RAID in degraded mode for purposes of checking out a disk is,
Frankly, nuts. NEVER degrade anything. Just use the hotspare and do a hot clone of the disk in question
to the hotspare, then make that disk the new hot spare and repeat..

Equate this to a "Rotating the Tires" mode.

David @ santools com
http://www.santools.com/smart/unix/manual

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#9 lore raw
In reply to Justin Piszcz
Quoted 22 lines · Justin Piszcz
Quoted 17 lines · Bill Davidsen

Over a year ago I mentioned RAID-5e, a RAID-5 with the spare(s) distributed over multiple drives. This has come up again, so I thought I'd just mention why, and what advantages it offers.

By spreading the spare over multiple drives the head motion of normal access is spread over one (or several) more drives. This reduces seeks, improves performance, etc. The benefit reduces as the number of drives in the array gets larger, obviously with four drives using only three for normal operation is slower than four, etc. And by using all the drives all the time, the chance of a spare being undetected after going bad is reduced.

This becomes important as array drive counts shrink. Lower cost for drives ($100/TB!), and attempts to drop power use by using fewer drives, result in an overall drop in drive count, important in serious applications.

All that said, I would really like to bring this up one more time, even if the answer is "no interest."

Signature · 7 lines
Bill Davidsen <[email protected]>
"Woe unto the statesman who makes war without a reason that will still
be valid when the war is over..." Otto von Bismark
-- 
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Quoted 3 lines

Bill,

Not a bad idea; however, can the same not be acheived (somewhat) by performing daily/smart, weekly/long tests on the drive to validate its health? I find this to work fairly well on a large scale.

Not really, the performance benefit comes from spreading head motion to (at least) one more drive. You can get a check on basic functionality with SMART, but it doesn't beat the drive the way real load does. Add to that the unfortunate problem that more realistic testing also takes up i/o bandwidth for non-productive transfers. Better to be doing actual live data transfers to those drives if you can.

Signature · 3 lines
Bill Davidsen <[email protected]>
 "Woe unto the statesman who makes war without a reason that will still
 be valid when the war is over..." Otto von Bismark 
#11 lore raw
In reply to Neil Brown
Quoted 18 lines · Neil Brown
Quoted 8 lines · On Monday October 13, [email protected] wrote:

Over a year ago I mentioned RAID-5e, a RAID-5 with the spare(s) distributed over multiple drives. This has come up again, so I thought I'd just mention why, and what advantages it offers.

By spreading the spare over multiple drives the head motion of normal access is spread over one (or several) more drives. This reduces seeks, improves performance, etc. The benefit reduces as the number of drives in the array gets larger, obviously with four drives using only three for normal operation is slower than four, etc. And by using all the drives all the time, the chance of a spare being undetected after going bad is reduced.

This becomes important as array drive counts shrink. Lower cost for drives ($100/TB!), and attempts to drop power use by using fewer drives, result in an overall drop in drive count, important in serious applications.

All that said, I would really like to bring this up one more time, even if the answer is "no interest."

How are your coding skills?

The tricky bit is encoding the new state.
We can not longer tell the difference between "optimal" and "degraded"
based on the number of in-sync devices. We also need some state flag
to say that the "distributed spare" has been constructed. Maybe that could be encoded in the "layout".

Quoted 9 lines

We would also need to allow a "recovery" pass to happen without having actually added any spares, or having any non-insync devices. That probably means passing the decision "is a recovery pending" down into the personality rather than making it in common code. Maybe have some field in the mddev structure which the personality sets if a recovery is worth trying. Or maybe just try it anyway after any significant change and if the personality finds nothing can be done it aborts.

My coding skills are fine, here, but I have to do a lot of planning before even considering this.
Here's why:
Say you have a five drive RAID-5e, and you are running happily. A
drive fails! Now you can rebuild on the spare drive, but the spare drive must be created on the parts from the remaining functional drives, so it can't be done pre-failure, the allocation has to be defined after you see what you have left. Does that sound ugly and complex? Does to me, too. So I'm thinking about this, and doing some reading, but it's not quite as simple as I thought.

Quoted 2 lines

I'm happy to advise on, review, and eventually accept patches.

Actually what I think I would do is want to build a test bed in software before trying this in the kernel, then run the kernel part in a virtual machine. I have another idea, which has about 75% of the benefit with 10% of the complexity. Since it sounds too good to be true it probably is, I'll get back to you after I think about the simpler solution, I distrust free lunch algorithms.

Quoted 1 line

NeilBrown

Signature · 3 lines
Bill Davidsen <[email protected]>
 "Woe unto the statesman who makes war without a reason that will still
 be valid when the war is over..." Otto von Bismark 
#12 lore raw
Quoted 39 lines

-----Original Message-----
From: [email protected] [mailto:linux-raid- [email protected]] On Behalf Of Bill Davidsen
Sent: Thursday, October 16, 2008 6:50 PM
To: Neil Brown
Cc: Linux RAID
Subject: Re: Distributed spares

Quoted 4 lines · Neil Brown
Quoted 2 lines · On Monday October 13, [email protected] wrote:

Over a year ago I mentioned RAID-5e, a RAID-5 with the spare(s) distributed over multiple drives. This has come up again, so I

thought

Quoted 3 lines
Quoted 3 lines

I'd just mention why, and what advantages it offers.

By spreading the spare over multiple drives the head motion of

normal

Quoted 1 line
Quoted 1 line

access is spread over one (or several) more drives. This reduces

seeks,

Quoted 1 line
Quoted 1 line

improves performance, etc. The benefit reduces as the number of

drives

Quoted 1 line
Quoted 1 line

in the array gets larger, obviously with four drives using only

three

Quoted 2 lines
Quoted 2 lines

for normal operation is slower than four, etc. And by using all the drives all the time, the chance of a spare being undetected after

going

Quoted 4 lines
Quoted 4 lines

bad is reduced.

This becomes important as array drive counts shrink. Lower cost for drives ($100/TB!), and attempts to drop power use by using fewer

drives,

Quoted 1 line
Quoted 1 line

result in an overall drop in drive count, important in serious

applications.

Quoted 2 lines
Quoted 2 lines

All that said, I would really like to bring this up one more time,

even

Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

if the answer is "no interest."

How are your coding skills?

The tricky bit is encoding the new state.
We can not longer tell the difference between "optimal" and

"degraded"

Quoted 1 line

based on the number of in-sync devices. We also need some state

flag

Quoted 7 lines
Quoted 4 lines

to say that the "distributed spare" has been constructed. Maybe that could be encoded in the "layout".

We would also need to allow a "recovery" pass to happen without

having

Quoted 2 lines

actually added any spares, or having any non-insync devices. That probably means passing the decision "is a recovery pending" down

into

Quoted 3 lines
Quoted 1 line

the personality rather than making it in common code. Maybe have

some

Quoted 1 line

field in the mddev structure which the personality sets if a

recovery

Quoted 16 lines
Quoted 4 lines

is worth trying. Or maybe just try it anyway after any significant change and if the personality finds nothing can be done it aborts.

My coding skills are fine, here, but I have to do a lot of planning before even considering this.
Here's why:
Say you have a five drive RAID-5e, and you are running happily. A
drive fails! Now you can rebuild on the spare drive, but the spare drive
must be created on the parts from the remaining functional drives, so it
can't be done pre-failure, the allocation has to be defined after you see what you have left. Does that sound ugly and complex? Does to me, too. So I'm thinking about this, and doing some reading, but it's not quite as simple as I thought.

Quoted 6 lines
Quoted 2 lines

I'm happy to advise on, review, and eventually accept patches.

Actually what I think I would do is want to build a test bed in software
before trying this in the kernel, then run the kernel part in a

virtual

Quoted 13 lines

machine. I have another idea, which has about 75% of the benefit with 10% of the complexity. Since it sounds too good to be true it probably is, I'll get back to you after I think about the simpler solution, I distrust free lunch algorithms.

Quoted 1 line

NeilBrown

--

Bill Davidsen <[email protected]>
"Woe unto the statesman who makes war without a reason that will
still
be valid when the war is over..." Otto von Bismark

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

With all due respect, RAID5E isn't practical. Too many corner cases dealing
With performance implications, and where you even just put the parity block, to insure
That when a disk fails you didn't put yourself into situation where the Hot spare chunk
is located on the disk drive that just died.

What about all of those dozens of utilities that need to know about RAID5E to work properly?
The cynic in me says if they're still having to recall patches (like today) that deals with mdadm on
Established RAID levels, then RAID5E is going to be much worse. Algorithms dealing
with drive failures, unrecoverable read/write errors on normal operations as well as rebuilds, expansions,
and journalization/optimization are not well understood. It is new territory.

If you want multiple distributed spares, just do RAID6, it is better than RAID5 in that respect, and nobody
Has to re-invent the wheel. Your "hot spare" is still distributed in all of the disks, and you can survive multiple
Drive failures. If your motivation is performance, then buy faster disks, additional controller(s),
optimize your storage pools; and tune your md settings to be more compatible with your filesystem parameters.
Or even look at your application and see if anything can be done to reduce the I/O count.

The fastest I/Os are the ones you eliminate.

David

#15 lore raw
In reply to Neil Brown
Quoted 5 lines · Neil Brown

The tricky bit is encoding the new state.
We can not longer tell the difference between "optimal" and "degraded" based on the number of in-sync devices. We also need some state flag to say that the "distributed spare" has been constructed. Maybe that could be encoded in the "layout".

Or you need to add a "virtual" spare that does not have an actual block device behind it. Or rather it could be a virtual disk constructed from the spare chunks on the data disks; maybe device mapper could be used here? If you could create such a virtual disk then maybe the normal RAID5 code could just do the rest. Of course the mapping of stripes to disk locations has to be changed to account for the "black holes" now belonging to the virtual spare device.

Hmm, if you create DM devices for all the disks that just leave out the proper holes from the mapping, and you also create a DM device from the holes, and you create an MD RAID5+spare on top of these DM devices, then you could have RAID5e right now completely from userspace. The superblock should be changed so nobody ever tries to build a RAID5 from the raw disks and there may be some other details...

Gabor

Signature · 4 lines
     ---------------------------------------------------------
     MTA SZTAKI Computer and Automation Research Institute
                Hungarian Academy of Sciences
     ---------------------------------------------------------
#13 lore raw
In reply to David Lethe
Quoted 140 lines · David Lethe
Quoted 60 lines

-----Original Message-----
From: [email protected] [mailto:linux-raid- [email protected]] On Behalf Of Bill Davidsen
Sent: Thursday, October 16, 2008 6:50 PM
To: Neil Brown
Cc: Linux RAID
Subject: Re: Distributed spares

Quoted 6 lines · Neil Brown
Quoted 3 lines · On Monday October 13, [email protected] wrote:

Over a year ago I mentioned RAID-5e, a RAID-5 with the spare(s) distributed over multiple drives. This has come up again, so I

thought

Quoted 4 lines
Quoted 4 lines

I'd just mention why, and what advantages it offers.

By spreading the spare over multiple drives the head motion of

normal

Quoted 2 lines
Quoted 2 lines

access is spread over one (or several) more drives. This reduces

seeks,

Quoted 2 lines
Quoted 2 lines

improves performance, etc. The benefit reduces as the number of

drives

Quoted 2 lines
Quoted 2 lines

in the array gets larger, obviously with four drives using only

three

Quoted 3 lines
Quoted 3 lines

for normal operation is slower than four, etc. And by using all the drives all the time, the chance of a spare being undetected after

going

Quoted 5 lines
Quoted 5 lines

bad is reduced.

This becomes important as array drive counts shrink. Lower cost for drives ($100/TB!), and attempts to drop power use by using fewer

drives,

Quoted 2 lines
Quoted 2 lines

result in an overall drop in drive count, important in serious

applications.

Quoted 2 lines
Quoted 2 lines

All that said, I would really like to bring this up one more time,

even

Quoted 3 lines
Quoted 3 lines

if the answer is "no interest."

Quoted 9 lines
Quoted 5 lines

How are your coding skills?

The tricky bit is encoding the new state.
We can not longer tell the difference between "optimal" and

"degraded"

Quoted 2 lines

based on the number of in-sync devices. We also need some state

flag

Quoted 10 lines
Quoted 5 lines

to say that the "distributed spare" has been constructed. Maybe that could be encoded in the "layout".

We would also need to allow a "recovery" pass to happen without

having

Quoted 3 lines

actually added any spares, or having any non-insync devices. That probably means passing the decision "is a recovery pending" down

into

Quoted 6 lines
Quoted 2 lines

the personality rather than making it in common code. Maybe have

some

Quoted 2 lines

field in the mddev structure which the personality sets if a

recovery

Quoted 18 lines
Quoted 5 lines

is worth trying. Or maybe just try it anyway after any significant change and if the personality finds nothing can be done it aborts.

My coding skills are fine, here, but I have to do a lot of planning before even considering this.
Here's why:
Say you have a five drive RAID-5e, and you are running happily. A
drive fails! Now you can rebuild on the spare drive, but the spare drive
must be created on the parts from the remaining functional drives, so it
can't be done pre-failure, the allocation has to be defined after you see what you have left. Does that sound ugly and complex? Does to me, too. So I'm thinking about this, and doing some reading, but it's not quite as simple as I thought.

Quoted 7 lines
Quoted 3 lines

I'm happy to advise on, review, and eventually accept patches.

Actually what I think I would do is want to build a test bed in software
before trying this in the kernel, then run the kernel part in a

virtual

Quoted 15 lines

machine. I have another idea, which has about 75% of the benefit with 10% of the complexity. Since it sounds too good to be true it probably is, I'll get back to you after I think about the simpler solution, I distrust free lunch algorithms.

Quoted 2 lines

NeilBrown

--

Bill Davidsen <[email protected]>
"Woe unto the statesman who makes war without a reason that will
still
be valid when the war is over..." Otto von Bismark

Quoted 6 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 9 lines

With all due respect, RAID5E isn't practical. Too many corner cases dealing
With performance implications, and where you even just put the parity block, to insure
That when a disk fails you didn't put yourself into situation where the Hot spare chunk
is located on the disk drive that just died.

Having run 38 multi-TB machines for an ISP using RAID5e in the SCSI controller, I feel pretty sure that the practicality is established, and only the ability to reinvent that particular wheel is in question. The complexity is that the hot spare drive needs to be defined after the 1st drive failure, using the spare sectors on the functional drives.

Quoted 3 lines

What about all of those dozens of utilities that need to know about RAID5E to work properly?

What did you have in mind? Virtually all utilities working at the filesystem level have no need to know anything, and treat any array as a drive in a black box fashion.

Quoted 4 lines

The cynic in me says if they're still having to recall patches (like today) that deals with mdadm on
Established RAID levels, then RAID5E is going to be much worse.

Let's definitely not add anything to the kernel, then, as a feature-static kernel is much more stable. Features like the software RAID-10 (not 1+0) are not established in some standard I've seen, but they work just fine. Any this is not unexplored territory, distributed spare is called RAID5e on the IBM servers I used, and I believe Storage Computer (in NH) has a similar feature they call "RAID-7" and trademark.

Quoted 5 lines

Algorithms dealing
with drive failures, unrecoverable read/write errors on normal
operations as well as rebuilds, expansions, and journalization/optimization are not well understood. It is new territory.

That's why I'm being quite cautious about saying I can do this, the coding is easy, it's finding out what to code that's hard. It appears that configuration decisions need to be made after the failure event, before the rebuild. Yes, it's complex. But from experience I can say that performance during rebuild is far better with a distributed spare than beating the snot out of one newly added spare with other RAID levels. So there's a performance benefit for both the normal case and the rebuild case, and a side benefit of faster rebuild time.

The full recovery after replacing the failed drive is also an interesting time. :-(

Quoted 10 lines

If you want multiple distributed spares, just do RAID6, it is better
than RAID5 in that respect, and nobody
Has to re-invent the wheel. Your "hot spare" is still distributed in
all of the disks, and you can survive multiple
Drive failures. If your motivation is performance, then buy faster
disks, additional controller(s), optimize your storage pools; and tune your md settings to be more compatible with your filesystem parameters. Or even look at your application and see if anything can be done to reduce the I/O count.

The motivation is to get the best performance from the hardware you have. Adding hardware cost so you can use storage hardware inefficiently is *really* not practical. Neither power, cooling, drives, nor floor space are cheap enough to use poorly.

Quoted 2 lines

The fastest I/Os are the ones you eliminate.

And the fastest seeks are the ones you don't do because you spread head motion over more drives. But once you have distributed spare in the kernel, you have a free performance gain. Or as free as using either more CPU or more memory for mapping will allow. Most people will trade a little of either for better performance.

Signature · 3 lines
Bill Davidsen <[email protected]>
 "Woe unto the statesman who makes war without a reason that will still
 be valid when the war is over..." Otto von Bismark 
#14 lore raw
Quoted 11 lines · On Friday October 17, [email protected] wrote:
Quoted 10 lines · David Lethe

With all due respect, RAID5E isn't practical. Too many corner cases dealing
With performance implications, and where you even just put the parity block, to insure
That when a disk fails you didn't put yourself into situation where the Hot spare chunk
is located on the disk drive that just died.

Quoted 5 lines

Having run 38 multi-TB machines for an ISP using RAID5e in the SCSI controller, I feel pretty sure that the practicality is established, and only the ability to reinvent that particular wheel is in question. The complexity is that the hot spare drive needs to be defined after the 1st drive failure, using the spare sectors on the functional drives.

I don't think that will be particularly complex. It will just be a bit of code in raid5_compute_sector. The detail of 'which device has failed' would be stored in ->algorithm somehow.

There is an interesting question of how general do we want the code to be.
e.g do we want to be able to configure an array with 2 distributed spares? I suspect that people would rarely want 2, and never want 3, so it would be worth making 2 work if the code didn't get too complex, which I don't think it would (but I'm not certain).

Quoted 7 lines
Quoted 6 lines

Algorithms dealing
with drive failures, unrecoverable read/write errors on normal operations as well as rebuilds, expansions,
and journalization/optimization are not well understood. It is new territory.

Quoted 8 lines

That's why I'm being quite cautious about saying I can do this, the coding is easy, it's finding out what to code that's hard. It appears that configuration decisions need to be made after the failure event, before the rebuild. Yes, it's complex. But from experience I can say that performance during rebuild is far better with a distributed spare than beating the snot out of one newly added spare with other RAID levels. So there's a performance benefit for both the normal case and the rebuild case, and a side benefit of faster rebuild time.

I cannot see why rebuilding a raid5e would be faster than rebuilding a raid5 to a fresh device.
In each case, you need to read from n-1 devices, and write to 1 device. So all devices are constantly doing IO at the same rate. In the raid5 case you could get better streaming as each device is either "always reading" or "always writing", where as in a raid5e rebuild, devices will sometimes read and sometimes write. So if anything, I would expect raid5e to rebuild more slowly, but you would probably only notice this with small chunk sizes.

I agree that (with suitably large chunk sizes) you should be able to get better throughput on raid5e.

NeilBrown