Skip to contentSkip to conversation

RAID-5 implementation questions

DISCUSSION Phil Karn 3 participants 4 messages Dec 3, 2010 MD RAID

lore
Share & tools
lore

Phil Karn

author
#0 lore raw

Are there any papers documenting the implementation of the Linux RAID subsystem? I'm interested in some of the details of how RAID-5 works.

I've never seen a virgin disk drive from the factory that wasn't all 0's. Creating a RAID array on a set of such drives triggers an initial rebuild that simply writes lots zeroes on lots of zeroes. With disks now pushing past 2 TB, this can easily take half a day.

Except for the admittedly somewhat useful side effect of scanning the disks for bad sectors, all this activity seems rather unnecessary. Is there a way to create a RAID-5 (or any other RAID level) array so that it will immediately come up without an initial rebuild?

File systems generally don't read disk blocks that they haven't already written. So even when you build a RAID array from drives with old data, I can't see how skipping the initial rebuild can cause any real harm. The first write to any block causes the RAID system to initialize the parity in that stripe, thus making it possible to regenerate that block in case of a drive failure.

During the initial rebuild of a RAID-5 array, /proc/mdstat suggests that the array is operating in degraded mode and the last drive in the array is being rebuilt. Is this true, i.e., are all the rebuild writes going to that last drive?

How does a rebuilding RAID-5 array handle a read or write operation when it lands on the "broken" drive? Does it depend on whether the block is before or after the rebuild pointer?

Thanks much,

Phil

#1 lore raw
Quoted 1 line · Phil Karn

Except for the admittedly somewhat useful side effect of scanning the disks for bad sectors, all this activity seems rather unnecessary. Is there a way to create a RAID-5 (or any other RAID level) array so that it will immediately come up without an initial rebuild?

"--assume-clean".

Quoted 6 lines

File systems generally don't read disk blocks that they haven't already written. So even when you build a RAID array from drives with old data, I can't see how skipping the initial rebuild can cause any real harm. The first write to any block causes the RAID system to initialize the parity in that stripe, thus making it possible to regenerate that block in case of a drive failure.

Some raid implementations won't read/write to all drives, but might instead read the block being written to, and the parity block, then write the new block and recalculate the parity, thus not read/writing to all blocks. If this is the case, if the parity is wrong, it'll still be wrong after the operation, thus you don't have any redundancy.

Doing a rebuild when creating the array is something I'd only skip if I was doing lab work, never in production. I use raid for redundancy, thus I want to make sure everything is ok and it doesn't matter to me if it takes half a day.

Signature · 1 line
Mikael Abrahamsson    email: [email protected]
#3 lore raw
In reply to Phil Karn
Quoted 2 lines · On Fri, 03 Dec 2010 00:49:52 -0800 Phil Karn <[email protected]> wrote:

Are there any papers documenting the implementation of the Linux RAID subsystem? I'm interested in some of the details of how RAID-5 works.

I would suggest
man mdadm
and
man md

That should answer at least some of your questions.

Then try http://raid.wiki.kernel.org/

If you have further questions after that, please ask.

NeilBrown

Quoted 34 lines

I've never seen a virgin disk drive from the factory that wasn't all 0's. Creating a RAID array on a set of such drives triggers an initial rebuild that simply writes lots zeroes on lots of zeroes. With disks now pushing past 2 TB, this can easily take half a day.

Except for the admittedly somewhat useful side effect of scanning the disks for bad sectors, all this activity seems rather unnecessary. Is there a way to create a RAID-5 (or any other RAID level) array so that it will immediately come up without an initial rebuild?

File systems generally don't read disk blocks that they haven't already written. So even when you build a RAID array from drives with old data, I can't see how skipping the initial rebuild can cause any real harm. The first write to any block causes the RAID system to initialize the parity in that stripe, thus making it possible to regenerate that block in case of a drive failure.

During the initial rebuild of a RAID-5 array, /proc/mdstat suggests that the array is operating in degraded mode and the last drive in the array is being rebuilt. Is this true, i.e., are all the rebuild writes going to that last drive?

How does a rebuilding RAID-5 array handle a read or write operation when it lands on the "broken" drive? Does it depend on whether the block is before or after the rebuild pointer?

Thanks much,

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#2 lore raw
In reply to Mikael Abrahamsson
Quoted 1 line · Mikael Abrahamsson

"--assume-clean".

Thanks.

Quoted 5 lines

Some raid implementations won't read/write to all drives, but might instead read the block being written to, and the parity block, then write the new block and recalculate the parity, thus not read/writing to all blocks. If this is the case, if the parity is wrong, it'll still be wrong after the operation, thus you don't have any redundancy.

Good point. That had occurred to me too but I didn't know if Linux did that. I can see how one might dynamically pick one way or the other depending on how much of the stripe is already in the buffer cache.

Quoted 4 lines

Doing a rebuild when creating the array is something I'd only skip if I was doing lab work, never in production. I use raid for redundancy, thus I want to make sure everything is ok and it doesn't matter to me if it takes half a day.

I hear you. But I think an important special case is when you're initially loading a new RAID-5 array from an existing (typically smaller) file system that will then be replaced by the new array.

Why not let the new array work something like a RAID-0, leaving the parity blocks unwritten until you're finished loading the array? Then pass through the array writing all the parity blocks with the final data. If a drive fails in the new array before you're done, you still have all your original data; you haven't lost anything.

Ultimately, RAID-5 in software is always going to be at least somewhat vulnerable because of the lack of an atomic (all or none) committed write of all the blocks in a stripe. This might silently corrupt an old, stable file in a way that you won't notice until a drive fails and you don't have the redundancy you thought you had to reconstruct it. can accept losing whatever files I was writing at the time of a crash, but silent corruption of an old and stable file seems far more insidious. I do periodically run checkarray to ensure that the parities are consistent, but this takes a long time and seems inelegant somehow. Maybe we need software ECC on all data so that one doesn't have to rely on the drive itself to detect errors.

Thanks,

Phil