RAID-6: help wanted
Okay, it's well overdue to do this...
There is currently a data-corruption bug in the RAID-6 md layer (raid6main.c). I have so far not been successful in locating it, although it is easily reproducible, thanks to a set of scripts by Jim Paris. I suspect it is a race condition between raid6d and the rest of the kernel.
The problem is that I already have way too many projects in flight, and I *really* would like to focus the time I have on getting klibc/early userspace integrated. Thus, I don't expect to have any time at chasing the RAID-6 corruption bug any time soon.
What can be ruled out is errors in the RAID-6 algorithm codes, however. Those are easily testable in isolation and have a clean bill of health.
-hpa
Quoted 8 lines · H. Peter Anvin
Okay, it's well overdue to do this...
There is currently a data-corruption bug in the RAID-6 md layer (
raid6main.c). I have so far not been successful in locating it, although it is easily reproducible, thanks to a set of scripts by Jim Paris. I suspect it is a race condition between raid6d and the rest of the kernel.
Quoted 8 lines
The problem is that I already have way too many projects in flight, and I *really* would like to focus the time I have on getting klibc/early userspace integrated. Thus, I don't expect to have any time at chasing the RAID-6 corruption bug any time soon.
What can be ruled out is errors in the RAID-6 algorithm codes, however. Those are easily testable in isolation and have a clean bill of health.
Great, can we get a copy of the scripts to try and assist?
Brad
Quoted 7 lines
Quoted 5 lines
There is currently a data-corruption bug in the RAID-6 md layer (
raid6main.c). I have so far not been successful in locating it, although it is easily reproducible, thanks to a set of scripts by Jim Paris. I suspect it is a race condition between raid6d and the rest of the kernel.Great, can we get a copy of the scripts to try and assist?
Sure:
---
Date: Fri, 6 Aug 2004 00:04:39 -0400
From: Jim Paris <[email protected]>
To: "H. Peter Anvin" <[email protected]>
Cc: [email protected]
Subject: Re: Kernel panic, FS corruption Was: Re: Call for RAID-6 users
Quoted 2 lines
If you can reproduce it with ext2/3 it would make debugging simpler, because I understand the ext code and data structures a lot better.
This demonstrates it on ext2. I can't seem to reproduce it with just simple use of 'dd', but it shows up if I untar a ton of data.
This script:
- creates five 100MB "disks" through loopback
- puts them in a six-disk RAID-6 array (resulting size=400MB, degraded)
- untars about 350MB of data to the array
- runs e2fsck, which shows filesystem errors
Usage:
- put r6ext.sh and big.tar.bz2 in a directory
- run r6ext.sh as root
Sorry for the huge files, but e2fsck didn't show any problems when I scaled everything down by a factor of 10. You could probably make your own big.tar.bz2 and see the same problem, as there's nothing special about this data.
http://stonewall.mit.edu/~jim/r6ext.sh
http://stonewall.mit.edu/~jim/big.tar.bz2 (77MB)
-jim
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines · H. Peter Anvin
Quoted 4 lines · Brad Campbell
Great, can we get a copy of the scripts to try and assist?
Brad
Quoted 8 lines
In the process of being uploaded to:
http://www.zytor.com/~hpa/raid6-bug/
It'll be about 40 minutes until it's uploaded, because of the 80 MB data file.
This will create a RAID-6 volume on a loopback device, fail a unit, and then write a bunch of stuff, which will corrupt the filesystem. The corruption only happens when writing to a volume that is running in degraded mode; it seems to happen both with 1 and 2 disks missing.
Changed to:
http://userweb.kernel.org/~hpa/raid6-bug/
(It's a faster server, but it'll still take about 40 minutes from now until it's uploaded, but then it can at least be a quicker download.)
-hpa
Quoted 5 lines · H. Peter Anvin
Quoted 5 lines
It'll be about 40 minutes until it's uploaded, because of the 80 MB data file.
This will create a RAID-6 volume on a loopback device, fail a unit, and then write a bunch of stuff, which will corrupt the filesystem. The corruption only happens when writing to a volume that is running in degraded mode; it seems to happen both with 1 and 2 disks missing.
For all I know, too, the data file can be any large collection of small-to-medium size files (I think it currently contains a Linux kernel build tree), so downloading it all might not be necessary.
-hpa
Quoted 5 lines · H. Peter Anvin
Changed to:
http://userweb.kernel.org/~hpa/raid6-bug/
(It's a faster server, but it'll still take about 40 minutes from now until it's uploaded, but then it can at least be a quicker download.)
It will make no difference to me. I'm in the desert in the clutches of a monopolistic telecommunications carrier that seems to use brine soaked string as its backbone :p)
Ta for the headsup.
Regards,
Brad
Quoted 7 lines · On Saturday October 23, [email protected] wrote:
Okay, it's well overdue to do this...
There is currently a data-corruption bug in the RAID-6 md layer (
raid6main.c). I have so far not been successful in locating it, although it is easily reproducible, thanks to a set of scripts by Jim Paris. I suspect it is a race condition between raid6d and the rest of the kernel.
Does this fix the problem?
It is definitely wrong as it is.
NeilBrown
----------- Diffstat output ------------
./drivers/md/raid6main.c | 1 - 1 files changed, 1 deletion(-)
drivers/md/raid6main.c renamed from drivers/md/raid6main.c~current~ +0 -1
@@ -734,7 +734,6 @@ static void compute_parity(struct stripecase READ_MODIFY_WRITE :BUG ();/* READ_MODIFY_WRITE N/A for RAID-6 */ case RECONSTRUCT_WRITE : -case UPDATE_PARITY :/* Is this right? */ for (i= disks; i-- ;)if ( i != pd_idx && i != qd_idx && sh->dev[i].towrite ) { chosen = sh->dev[i].towrite;
Quoted 3 lines
Does this fix the problem?
It is definitely wrong as it is.
..
drivers/md/raid6main.c static void compute_parity(struct stripe · line 737
Quoted 2 lines
drivers/md/raid6main.c renamed from drivers/md/raid6main.c~current~ +0
-1case RECONSTRUCT_WRITE : -case UPDATE_PARITY :/* Is this right? */
No, it does not. Same sort of corruption with that line removed.
-jim
Quoted 9 lines · On Monday October 25, [email protected] wrote:
Quoted 3 lines
Does this fix the problem?
It is definitely wrong as it is...
drivers/md/raid6main.c static void compute_parity(struct stripe · line 737
Quoted 2 lines
drivers/md/raid6main.c renamed from drivers/md/raid6main.c~current~ +0
-1case RECONSTRUCT_WRITE : -case UPDATE_PARITY :/* Is this right? */ No, it does not. Same sort of corruption with that line removed.
Thanks. I'll try harder...
Looking at the script, the corruption happens while resync is still going on. If you wait for resync to finish, do you still get corruption?
NeilBrown
Quoted 9 lines
Quoted 3 lines
drivers/md/raid6main.c static void compute_parity(struct stripe · line 737
Quoted 1 line
drivers/md/raid6main.c renamed from drivers/md/raid6main.c~current~ +0
-1-case UPDATE_PARITY :/* Is this right? */ No, it does not. Same sort of corruption with that line removed.
Thanks. I'll try harder...
Looking at the script, the corruption happens while resync is still going on. If you wait for resync to finish, do you still get corruption?
The problem shows up when the array is degraded (either 1 or 2 disks missing), regardless of resync. If the array is not degraded, everything works fine, again regardless of resync.
-jim
Followup to: <[email protected]> By author: Neil Brown <[email protected]> In newsgroup: linux.dev.raid
Quoted 12 lines · On Monday October 25, [email protected] wrote:
Quoted 9 lines
Quoted 3 lines
Does this fix the problem?
It is definitely wrong as it is...
drivers/md/raid6main.c static void compute_parity(struct stripe · line 737
Quoted 2 lines
drivers/md/raid6main.c renamed from drivers/md/raid6main.c~current~ +0
-1case RECONSTRUCT_WRITE : -case UPDATE_PARITY :/* Is this right? */ No, it does not. Same sort of corruption with that line removed.
Quoted 6 lines
Thanks. I'll try harder...
Looking at the script, the corruption happens while resync is still going on. If you wait for resync to finish, do you still get corruption?
The version of the script I posted does wait for the resync to finish before writing; it still exhibits the same problem.
-hpa
Quoted 8 lines · On Monday October 25, [email protected] wrote:
Quoted 3 lines
Does this fix the problem?
It is definitely wrong as it is...
Review
Quoted 2 lines
(quoted diff) +0
-1case RECONSTRUCT_WRITE: - case UPDATE_PARITY: /* Is this right? */No, it does not. Same sort of corruption with that line removed.
Ok, take-2. I've tested this one myself and it does seem to fix it.
The problem is that it is sometimes using parity to reconstruct a block, when not all of the blocks have been read in.
In raid5, there are two choices for write - reconstruct-write or read-modify-write.
If there are any failed drives, it always chooses read-modify-write and so only has to read data from good drives.
raid6 only allows for reconstruct-write, so if it ever writes to an array with a failed drive, it must read all blocks and reconstruct the missing blocks before allowing the write.
As this is something that raid5 didn't have to care about, and as the raid6 code was based on the raid5 code, it is easy to see how this case was missed.
The following patch added a bit of tracing to track other cases (hopefully non-existent) where calculations are done using non-existent data, and make sure the required blocks are pre-read. Possible this code (in handle_stripe) needs a substantial clean up...
I'll wait for comments and further testing before I forward it to Andrew.
NeilBrown
========================================================
Fix raid6 problem
Sometimes it didn't read all (working) drives before a parity calculation.
### Diffstat output
./drivers/md/raid6main.c | 17 +++++++++-------- 1 files changed, 9 insertions(+), 8 deletions(-)
drivers/md/raid6main.c renamed from drivers/md/raid6main.c~current~ +9 -8
@@ -734,7 +734,6 @@ static void compute_parity(struct stripecase READ_MODIFY_WRITE :BUG ();/* READ_MODIFY_WRITE N/A for RAID-6 */ case RECONSTRUCT_WRITE : -case UPDATE_PARITY :/* Is this right? */ for (i= disks; i-- ;)if ( i != pd_idx && i != qd_idx && sh->dev[i].towrite ) { chosen = sh->dev[i].towrite;
@@ -770,7 +769,8 @@ static void compute_parity(struct stripe i = d0_idx;do { ptrs[count++] =page_address (sh->dev[i].page); - +if (count <= disks-2 && !test_bit (R5_UPTODATE , &sh->dev[i].flags)) +printk ("block %d/%d not uptodate on parity calc\n" , i,count); i =raid6_next_disk (i, disks); }while ( i != d0_idx );// break;
@@ -818,7 +818,7 @@ static void compute_block_1(struct stripif (test_bit (R5_UPTODATE , &sh->dev[i].flags)) ptr[count++] = p;else -PRINTK ("compute_block() %d, stripe %llu, %d" +printk ("compute_block() %d, stripe %llu, %d" " not present\n" , dd_idx, (unsigned long long )sh->sector, i);
@@ -875,6 +875,9 @@ static void compute_block_2(struct stripdo { ptrs[count++] =page_address (sh->dev[i].page); i =raid6_next_disk (i, disks); +if (i != dd_idx1 && i != dd_idx2 && + !test_bit (R5_UPTODATE , &sh->dev[i].flags)) +printk ("compute_2 with missing block %d/%d\n" , count, i); }while ( i != d0_idx );if ( failb == disks-2 ) {
@@ -1157,17 +1160,15 @@ static void handle_stripe(struct stripe_* parity, or to satisfy requests * or to load a block that is being partially written. */ -if (to_read || non_overwrite || (syncing && (uptodate < disks))) { +if (to_read || non_overwrite || (to_write && failed) || (syncing && (uptodate < disks))) {for (i=disks; i--;) { dev = &sh->dev[i];if (!test_bit (R5_LOCKED , &dev->flags) && !test_bit (R5_UPTODATE , &dev->flags) && (dev->toread || (dev->towrite && !test_bit (R5_OVERWRITE , &dev->flags)) || syncing || - (failed >=1 && (sh->dev[failed_num[0 ]].toread || - (sh->dev[failed_num[0 ]].towrite && !test_bit (R5_OVERWRITE , &sh->dev[failed_num[0 ]].flags)))) || - (failed >=2 && (sh->dev[failed_num[1 ]].toread || - (sh->dev[failed_num[1 ]].towrite && !test_bit (R5_OVERWRITE , &sh->dev[failed_num[1 ]].flags)))) + (failed >=1 && (sh->dev[failed_num[0 ]].toread || to_write)) || + (failed >=2 && (sh->dev[failed_num[1 ]].toread || to_write)) ) ) {/* we would like to get this block, possibly
Quoted 19 lines · Neil Brown
Ok, take-2. I've tested this one myself and it does seem to fix it.
The problem is that it is sometimes using parity to reconstruct a block, when not all of the blocks have been read in.
In raid5, there are two choices for write - reconstruct-write or read-modify-write.
If there are any failed drives, it always chooses read-modify-write and so only has to read data from good drives.
raid6 only allows for reconstruct-write, so if it ever writes to an array with a failed drive, it must read all blocks and reconstruct the missing blocks before allowing the write.
As this is something that raid5 didn't have to care about, and as the raid6 code was based on the raid5 code, it is easy to see how this case was missed.
Quoted 8 lines
The following patch added a bit of tracing to track other cases (hopefully non-existent) where calculations are done using non-existent data, and make sure the required blocks are pre-read. Possible this code (in
handle_stripe) needs a substantial clean up...I'll wait for comments and further testing before I forward it to Andrew.
That makes sense (and definitely explains why I didn't find the problem.)
I tried it out, and it seems much better now. It does, however, still seem to have a problem:
+ e2fsck -nf /dev/md6
e2fsck 1.35 (28-Feb-2004)
Pass 1: Checking inodes, blocks, and sizes
Inode 7 has illegal block(s). Clear? no
Illegal block #-1 (33619968) in inode 7. IGNORED.
Error while iterating over blocks in inode 7: Illegal indirect block found
e2fsck: aborted
Inode 7 is a special-use inode:
#define EXT3_RESIZE_INO 7 /* Reserved group descriptors inode */
This is running the version of the r6ext.sh script that I posted, with the same datafile, on a PowerMac.
-hpa
Quoted 5 lines
That makes sense (and definitely explains why I didn't find the problem.)
I tried it out, and it seems much better now. It does, however, still seem to have a problem:
Quoted 8 lines
+ e2fsck -nf
/dev/md6
e2fsck 1.35 (28-Feb-2004)
Pass 1: Checking inodes, blocks, and sizes
Inode 7 has illegal block(s). Clear? noIllegal block #-1 (33619968) in inode 7. IGNORED.
Error while iterating over blocks in inode 7: Illegal indirect block found
e2fsck: aborted
The patch (thanks, Neil!) seems to work fine for me with both the ReiserFS and ext2 test scripts, on an x86, both with and without waiting for resync.
-jim
Quoted 9 lines · Jim Paris
Quoted 9 lines
That makes sense (and definitely explains why I didn't find the problem.)
I tried it out, and it seems much better now. It does, however, still seem to have a problem:
+ e2fsck -nf
/dev/md6
e2fsck 1.35 (28-Feb-2004)
Pass 1: Checking inodes, blocks, and sizes
Inode 7 has illegal block(s). Clear? no
Quoted 9 lines
Quoted 3 lines
Illegal block #-1 (33619968) in inode 7. IGNORED.
Error while iterating over blocks in inode 7: Illegal indirect block found
e2fsck: abortedThe patch (thanks, Neil!) seems to work fine for me with both the ReiserFS and ext2 test scripts, on an x86, both with and without waiting for resync.
Right, see previous; it seems to be an unrelated ppc64 problem that happens even without RAID of any kind. I'm building an i386 kernel with the patch now to try it out.
FWIW, I also hacked up Altivec support for ppc/ppc64; it took all of a whopping half-hour to make work, since gcc can generate Altivec code and it's actually quite good at it. The resulting code runs at a whopping 6.1 GB/s on a 2.5 GHz 970.
-hpa
Quoted 5 lines · Jim Paris
The patch (thanks, Neil!) seems to work fine for me with both the ReiserFS and ext2 test scripts, on an x86, both with and without waiting for resync.
Works here too, once I tried it on i386. I'll look separately into why ppc64 seems to have problems, but that's not an issue for RAID-6.
-hpa
Quoted 2 lines
Works here too, once I tried it on i386. I'll look separately into why ppc64 seems to have problems, but that's not an issue for RAID-6.
Great. I'll continue testing as I start using it on one of my file servers.
Another issue: If I create a 6-disk RAID-6 array ...
... with 2 missing, no resync happens.
... with 1 missing, no resync happens. (???) ... with 0 missing, resync happens.
... with 2 missing, then add 1, recovery happens. ... with 0 missing, then fail 1, resync continues.
Shouldn't resync happen in the created-with-1-disk-missing case?
-jim
Quoted 14 lines · Jim Paris
Quoted 1 line
Works here too, once I tried it on i386. I'll look separately into why ppc64 seems to have problems, but that's not an issue for RAID-6.
Great. I'll continue testing as I start using it on one of my file servers.
Another issue: If I create a 6-disk RAID-6 array ...
... with 2 missing, no resync happens.
... with 1 missing, no resync happens. (???) ... with 0 missing, resync happens.
... with 2 missing, then add 1, recovery happens. ... with 0 missing, then fail 1, resync continues.
Quoted 3 lines
Shouldn't resync happen in the created-with-1-disk-missing case?
-jim
No, why would it? You don't have anything to resync to.
-hpa
Quoted 11 lines · Jim Paris
Another issue: If I create a 6-disk RAID-6 array ...
... with 2 missing, no resync happens.
... with 1 missing, no resync happens. (???) ... with 0 missing, resync happens.
... with 2 missing, then add 1, recovery happens. ... with 0 missing, then fail 1, resync continues.Shouldn't resync happen in the created-with-1-disk-missing case?
Nevermind, I guess it probably should, since there is still redundancy and therefore it can be inconsistent.
-hpa
Quoted 10 lines · On Thursday October 28, [email protected] wrote:
Quoted 9 lines · Jim Paris
Another issue: If I create a 6-disk RAID-6 array ...
... with 2 missing, no resync happens.
... with 1 missing, no resync happens. (???) ... with 0 missing, resync happens.
... with 2 missing, then add 1, recovery happens. ... with 0 missing, then fail 1, resync continues.
Quoted 7 lines
Quoted 2 lines
Shouldn't resync happen in the created-with-1-disk-missing case?
Nevermind, I guess it probably should, since there is still redundancy and therefore it can be inconsistent.
-hpa
I have a patch to mdadm to make it resync when there is one failure, but I'm no longer convinced that it is needed.
In fact, the initial resync isn't really needed for raid6 (or raid1) at all. The first write to any stripe will make the redundancy for that stripe correct regardless of what it was, and before the first write, the content of the array is meaningless anyway.
Note that this is different to raid5 which, if using a read-modify-write cycle, depends on the parity block being correct.
There would be an issue once we start doing background scans of the arrays as the first scan could find lots of errors. But maybe that isn't a problem....
I'll probably include the patch in the next mdadm release, and revisit the whole idea when (if) I implement background array scans.
NeilBrown
Quoted 4 lines
I have a patch to mdadm to make it resync when there is one failure, but I'm no longer convinced that it is needed.
In fact, the initial resync isn't really needed for raid6 (or raid1) at all.
I see. I figured raid6 should just behave the same as raid5, but didn't realize that raid5 only did it because of the read-modify-write.
However, I do still think that the data should always be synchronized. Just because it's holding completely meaningless values at the moment doesn't mean it should change each time you read it (which can easily happen if the data gets read from different disks). Other things might eventually depend on the "meaningless" value.
Consider running raid5 on top of unsynced raid1 devices /dev/md[012]:
1. Do an initial resync of the raid5 so that D1 + D2 = P.
2. Write a real value to D1, and update P.
At this point, D1 and P are synced between disks of their raid1s, since they have been written.
3. Now lose the disk holding D1, so you need to reconstruct it from D2 and P. But you can't do that, because D2's value changes every time you read it!
-jim
I should read the messages in reverse order!
I just posted a similar email, but I could not think of a good example, or a bad example. :)
As you said (in my words):
RAID5 on top of RAID1 arrays would be bad if the RAID1 arrays were not synced and one RAID1 array was to fail.
You also said:
"3. Now lose the disk holding D1, so you need to reconstruct it from D2 and P."
But loosing D1 would require 2 disk failures since a single disk failure would not take a RAID1 array offline. But other things can fail that could cause the array to go offline. So your point is still valid.
But it does not require any failures to corrupt data. Using your example, once the RAID5 array is synced, if you were to modify a block the old block and the parity would be read, then the new parity would be computed based on the old block and the new block, then both block written. You never know which disk a RAID1 array will read from. Since the RAID1 was never synced you can get different data on the read of the old data which will cause the new parity to be wrong.
1. Do an initial resync of the raid5 so that D1 + D2 = P.
2. Write a real value to D1, and update P.
Since this requires a read from D1 you don't know which disk will be used, if a different disk is used than the one used during the initial sync it will corrupt P.
Guy
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jim Paris
Sent: Friday, October 29, 2004 7:48 AM
To: Neil Brown
Cc: H. Peter Anvin; [email protected]
Subject: Re: RAID-6: help wanted
Quoted 4 lines
I have a patch to mdadm to make it resync when there is one failure, but I'm no longer convinced that it is needed.
In fact, the initial resync isn't really needed for raid6 (or raid1) at all.
I see. I figured raid6 should just behave the same as raid5, but didn't realize that raid5 only did it because of the read-modify-write.
However, I do still think that the data should always be synchronized. Just because it's holding completely meaningless values at the moment doesn't mean it should change each time you read it (which can easily happen if the data gets read from different disks). Other things might eventually depend on the "meaningless" value.
Consider running raid5 on top of unsynced raid1 devices /dev/md[012]:
1. Do an initial resync of the raid5 so that D1 + D2 = P.
2. Write a real value to D1, and update P.
At this point, D1 and P are synced between disks of their raid1s, since they have been written.
3. Now lose the disk holding D1, so you need to reconstruct it from D2 and P.
But you can't do that, because D2's value changes every time you read it!
-jim
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines
I just posted a similar email, but I could not think of a good example, or a bad example. :)
It's hard coming up with specific examples. The RAID5-on-RAID1 is probably the best one I can think of. But there are other cases:
let's say I have a small RAID1 partition used for booting my system, always mounted read-only, and to back it up, I do a "dd" of the entire /dev/md0 (since exact block positions matter to boot-loaders). If uninitialized areas of the disk change every time I read them, my scripts might conclude that the backups have changed and need to get burned to CD when nothing actually changed on the array.
Quoted 1 line
But it does not require any failures to corrupt data.
Right. Having uninitialized portions appear to randomly change violates assumptions that other drivers (like raid5) and applications (like my fictional backup script) make about a block device.
Quoted 2 lines
If you insist to add this feature, please make it an option that defaults to sync everything.
For now, to force RAID6 to sync, start it with n-2 disks and add 1, rather than starting with n-1 disks:
mdadm --create /dev/md1 -l 6 -n 6 missing missing /dev/hd[gikm]2 mdadm --add /dev/md1 /dev/hdo2
Quoted 3 lines
You say RAID6 requires 100% of the stripe to be read to modify the strip. Is this due to the math of RAID6, or was it done this way because it was easier?
I think they can both be updated with read-modify-write:
P' = P + D_n + D_n'
Q' = Q + g^n * D_n + g^n * D_n'
However, the multiplications by g^n for computing Q' could be killer on your CPU, so it's a tradeoff. Since we're updating e.g. 128k at once for a single value of n, it's possible that it could be done in such a way that it's not too intensive (or cache-thrashing). Or perhaps you have so many disks that it really is worth the time to do the computation rather than read from all of them.
-jim
Quoted 12 lines · Jim Paris
I think they can both be updated with read-modify-write:
P' = P +
D_n+D_n'
Q' = Q + g^n *D_n+ g^n *D_n'However, the multiplications by g^n for computing Q' could be killer
on your CPU, so it's a tradeoff. Since we're updating e.g. 128k at
once for a single value of n, it's possible that it could be done in such a way that it's not too intensive (or cache-thrashing). Or perhaps you have so many disks that it really is worth the time
to do the computation rather than read from all of them.
It's not a matter of cache-trashing, it's a matter of the fact that very few CPUs have any form of parallel table lookup. It could be done with dynamic code generation, but that's a whole ball of wax on its own.
-hpa
Quoted 8 lines
Quoted 4 lines
I think they can both be updated with read-modify-write:
P' = P +
D_n+D_n'
Q' = Q + g^n *D_n+ g^n *D_n'It's not a matter of cache-trashing, it's a matter of the fact that very few CPUs have any form of parallel table lookup. It could be done with dynamic code generation, but that's a whole ball of wax on its own.
Why dynamic? Are there problems with just pregenerating all 256 optimized codepaths?
-jim
Quoted 4 lines · Jim Paris
Why dynamic? Are there problems with just pregenerating all 256 optimized codepaths?
Right, you could do that, of course, but it still amounts to 256 code paths that have to be generated, a framework to do them with (since gcc doesn't generate SSE-2 code properly, for example), and then figure out if it was all worth it at the end.
Given that the RAID-6 module is already quite large, I'm not so sure it's a win.
-hpa
Could you translate this into English?
P' = P + D_n + D_n'
Q' = Q + g^n * D_n + g^n * D_n'
I understand this much:
P =old parity D_n =old data block(s) P' =new parity
D_n' =new data block(s)
But is "+" = xor?
I am lost on this one:
Q' = Q + g^n * D_n + g^n * D_n'
With the parity (xor) it can be done by the bit, so an example is easy. Can Q be done by the bit, and if so, could you give an example?
If it takes more than 10 minutes, just tell me it is magic! :)
Also, on a related subject....
I have a 14 disk RAID5 with 1 spare.
Once RAID6 seems safe and stable I had hoped to convert to a 15 disk RAID6. Is 15 disks too much for RAID6?
Any idea what a reasonable limit would be?
Thanks,
Guy
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jim Paris
Sent: Friday, October 29, 2004 3:22 PM
To: H. Peter Anvin
Cc: Guy; 'Neil Brown'; [email protected]
Subject: Re: RAID-6: help wanted
Quoted 8 lines
Quoted 4 lines
I think they can both be updated with read-modify-write:
P' = P +
D_n+D_n'
Q' = Q + g^n *D_n+ g^n *D_n'It's not a matter of cache-trashing, it's a matter of the fact that very few CPUs have any form of parallel table lookup. It could be done with dynamic code generation, but that's a whole ball of wax on its own.
Why dynamic? Are there problems with just pregenerating all 256 optimized codepaths?
-jim
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 12 lines · Guy
Could you translate this into English?
P' = P +D_n+D_n'
Q' = Q + g^n *D_n+ g^n *D_n'I understand this much:
P =old parity D_n =old data block(s) P' =new parity
D_n' =new data block(s)But is "+" = xor?
Quoted 7 lines
I am lost on this one:
Q' = Q + g^n *D_n+ g^n *D_n'With the parity (xor) it can be done by the bit, so an example is easy. Can Q be done by the bit, and if so, could you give an example?
If it takes more than 10 minutes, just tell me it is magic! :)
See my paper on the subject:
http://www.kernel.org/pub/linux/kernel/people/hpa/raid6.pdf
Quoted 6 lines
Also, on a related subject....
I have a 14 disk RAID5 with 1 spare.
Once RAID6 seems safe and stable I had hoped to convert to a 15 disk RAID6. Is 15 disks too much for RAID6?
Any idea what a reasonable limit would be?
The limit is 27 disks total, it's imposed by the md system rather than RAID-6; RAID-6's inherent limit is 254+2 (see, again, the paper.)
-hpa
I took a look at your paper. It's magic! :)
My head hurts. :(
Ok, I understand that 27 disks is the limit. But is that usable? Would it be too slow? The disk IO seems unreasonable with 27 disks.
Guy
-----Original Message-----
From: H. Peter Anvin [mailto:[email protected]]
Sent: Friday, October 29, 2004 4:32 PM
To: Guy
Cc: 'Jim Paris'; 'Neil Brown'; [email protected]
Subject: Re: RAID-6: help wanted
Quoted 12 lines · Guy
Could you translate this into English?
P' = P +D_n+D_n'
Q' = Q + g^n *D_n+ g^n *D_n'I understand this much:
P =old parity D_n =old data block(s) P' =new parity
D_n' =new data block(s)But is "+" = xor?
Quoted 7 lines
I am lost on this one:
Q' = Q + g^n *D_n+ g^n *D_n'With the parity (xor) it can be done by the bit, so an example is easy. Can Q be done by the bit, and if so, could you give an example?
If it takes more than 10 minutes, just tell me it is magic! :)
See my paper on the subject:
http://www.kernel.org/pub/linux/kernel/people/hpa/raid6.pdf
Quoted 3 lines
Also, on a related subject....
I have a 14 disk RAID5 with 1 spare.
Once RAID6 seems safe and stable I had hoped to convert to a 15 disk
RAID6.
Quoted 3 lines
Is 15 disks too much for RAID6?
Any idea what a reasonable limit would be?
The limit is 27 disks total, it's imposed by the md system rather than RAID-6;
RAID-6's inherent limit is 254+2 (see, again, the paper.)
-hpa
Quoted 5 lines · Guy
I took a look at your paper. It's magic! :)
My head hurts. :(Ok, I understand that 27 disks is the limit. But is that usable? Would it be too slow? The disk IO seems unreasonable with 27 disks.
Shouldn't be too different from RAID-5, at least as long as you have reasonably large I/O transactions.
However, unless someone can actually test it out, it's hard to say.
Note that supporting RMW on RAID-6 is definitely a possibility; I tried it once on a 6-disk configuration (the only one I have) and it was slower.
-hpa
Neil,
You said:
"the initial resync isn't really needed for raid6 (or raid1) at all"
I understand your logic, but, I would prefer the data to be synced. I can't think of any examples of how it could make a difference, but if I read block x, then ten minutes later I read the same block again, I want it to be the same unless I changed it. With RAID1 you never know which disk will be read, RAID6 would only change if a disk failed. If you insist to add this feature, please make it an option that defaults to sync everything. This way someone that knows what they are doing can use the option, others will get the safer (IMHO) default.
I also want an integrity checker that does not require the array to be stopped. :)
I know you did not write the RAID6 code, but:
You say RAID6 requires 100% of the stripe to be read to modify the strip. Is this due to the math of RAID6, or was it done this way because it was easier? When doing random disk writes, any idea how this effects the performance of RAID6 compared to RAID5? Does the performance of RAID6 get worse as the number of disks is increased? Until now, I assumed a disk write would require read, read, read, modify, write, write, write. Compared to RAID5 with read, read, modify, write, write (for small updates).
Thanks for your time,
Guy
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Neil Brown
Sent: Thursday, October 28, 2004 8:44 PM
To: H. Peter Anvin
Cc: Jim Paris; [email protected]
Subject: Re: RAID-6: help wanted
Quoted 10 lines · On Thursday October 28, [email protected] wrote:
Quoted 9 lines · Jim Paris
Another issue: If I create a 6-disk RAID-6 array ...
... with 2 missing, no resync happens.
... with 1 missing, no resync happens. (???) ... with 0 missing, resync happens.
... with 2 missing, then add 1, recovery happens. ... with 0 missing, then fail 1, resync continues.
Quoted 7 lines
Quoted 2 lines
Shouldn't resync happen in the created-with-1-disk-missing case?
Nevermind, I guess it probably should, since there is still redundancy and therefore it can be inconsistent.
-hpa
I have a patch to mdadm to make it resync when there is one failure, but I'm no longer convinced that it is needed.
In fact, the initial resync isn't really needed for raid6 (or raid1) at all. The first write to any stripe will make the redundancy for that stripe correct regardless of what it was, and before the first write, the content of the array is meaningless anyway.
Note that this is different to raid5 which, if using a read-modify-write cycle, depends on the parity block being correct.
There would be an issue once we start doing background scans of the arrays as the first scan could find lots of errors. But maybe that isn't a problem....
I'll probably include the patch in the next mdadm release, and revisit the whole idea when (if) I implement background array scans.
NeilBrown
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Okay, I think the "inode 7" problem is unrelated, because it happens without any disk failures, it happens on RAID-0, RAID-5, and even on a plain loop device. I suspect there is something wrong in the ppc64 kernel; I'll try it again on i386 and report back.
-hpa