detection/correction of corruption with raid6
Hello.
I was looking at the PDFs linked to from the wiki, and found this:
http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdf
More specifically, section 4, starting on page 8.
Am I understanding this correctly, in that with raid6, linux is capable of detecting if the content on 1 disk is corrupted, and reconstruct it from the remaining disks?
mvh.
Kasper Sandberg
Quoted 10 lines · Redeeman
Hello.
I was looking at the PDFs linked to from the wiki, and found this:
http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdfMore specifically, section 4, starting on page 8.
Am I understanding this correctly, in that with raid6, linux is capable of detecting if the content on 1 disk is corrupted, and reconstruct it from the remaining disks?
I ran md/raid6 for awhile, do you mean remap the bad sector on the fly? Linux/md raid does not do this afaik.
But it can recover from a single or double disk failure, I have had both happen.
Justin.
Quoted 8 lines · Justin Piszcz
Quoted 5 lines · Redeeman
Hello.
I was looking at the PDFs linked to from the wiki, and found this:
http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdf
Quoted 8 lines
Quoted 5 lines
More specifically, section 4, starting on page 8.
Am I understanding this correctly, in that with raid6, linux is capable of detecting if the content on 1 disk is corrupted, and reconstruct it from the remaining disks?
I ran md/raid6 for awhile, do you mean remap the bad sector on the fly? Linux/md raid does not do this afaik.
No, i mean, if one disk does silent corruption
Quoted 9 lines
But it can recover from a single or double disk failure, I have had both happen.
Justin.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Redeeman
Quoted 10 lines · Justin Piszcz
Quoted 7 lines · Redeeman
Hello.
I was looking at the PDFs linked to from the wiki, and found this:
http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdfMore specifically, section 4, starting on page 8.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Am I understanding this correctly, in that with raid6, linux is capable of detecting if the content on 1 disk is corrupted, and reconstruct it from the remaining disks?
I ran md/raid6 for awhile, do you mean remap the bad sector on the fly? Linux/md raid does not do this afaik.
No, i mean, if one disk does silent corruption
What would the error look like? Both md/Linux & in the 3ware manual recommend you run a 'check' across the raid at least once a week (3ware/raid-verify) and md/Linux in Debian runs a check once a month I believe to eliminate these issues.
If you are asking whether a read error of a latent sector from the one disk will result it reading the data from the second disk that is a good question.
Justin.
Quoted 23 lines · Justin Piszcz
Quoted 11 lines · Redeeman
Quoted 10 lines · Justin Piszcz
Quoted 7 lines · Redeeman
Hello.
I was looking at the PDFs linked to from the wiki, and found this:
http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdfMore specifically, section 4, starting on page 8.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Am I understanding this correctly, in that with raid6, linux is capable of detecting if the content on 1 disk is corrupted, and reconstruct it from the remaining disks?
I ran md/raid6 for awhile, do you mean remap the bad sector on the fly? Linux/md raid does not do this afaik.
No, i mean, if one disk does silent corruption
Quoted 8 lines
What would the error look like? Both md/Linux & in the 3ware manual recommend you run a 'check' across the raid at least once a week (3ware/raid-verify) and md/Linux in Debian runs a check once a month I believe to eliminate these issues.
If you are asking whether a read error of a latent sector from the one disk will result it reading the data from the second disk that is a good question.
im asking, if one disk in a raid6 setup suddenly decides to flip a few bits in some bytes, will it be able to detect that in a scan, and correct it? i cant see how it can do it on raid5, but maybe raid6?
Quoted 6 lines
Justin.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 29 lines · Redeeman
Quoted 28 lines · Justin Piszcz
Quoted 11 lines · Redeeman
Quoted 10 lines · Justin Piszcz
Quoted 7 lines · Redeeman
Hello.
I was looking at the PDFs linked to from the wiki, and found this:
http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdfMore specifically, section 4, starting on page 8.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Am I understanding this correctly, in that with raid6, linux is capable of detecting if the content on 1 disk is corrupted, and reconstruct it from the remaining disks?
I ran md/raid6 for awhile, do you mean remap the bad sector on the fly? Linux/md raid does not do this afaik.
No, i mean, if one disk does silent corruption
What would the error look like? Both md/Linux & in the 3ware manual recommend you run a 'check' across the raid at least once a week (3ware/raid-verify) and md/Linux in Debian runs a check once a month I believe to eliminate these issues.
Quoted 7 lines
Quoted 3 lines
If you are asking whether a read error of a latent sector from the one disk will result it reading the data from the second disk that is a good question.
im asking, if one disk in a raid6 setup suddenly decides to flip a few bits in some bytes, will it be able to detect that in a scan, and correct it? i cant see how it can do it on raid5, but maybe raid6?
I have never seen any kernel messages showing md/raid: fixed sector 29383 for example and I have had 8-9 RMA's with Western Digital for bad/failed velociraptors in RAID6, I had go with raid6 because it is so bad that sometimes 2 drives at a time would drop out of the array. However, for the instances where there were just problems with one drive, it kicked it out of the array and the raid6 became degraded.
Hi,
2008/12/5 Redeeman <[email protected]>:
Quoted 29 lines · Justin Piszcz
Quoted 28 lines · Redeeman
Quoted 12 lines · Justin Piszcz
Quoted 10 lines · Redeeman
Quoted 8 lines
Hello.
I was looking at the PDFs linked to from the wiki, and found this:
http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdfMore specifically, section 4, starting on page 8.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Am I understanding this correctly, in that with raid6, linux is capable of detecting if the content on 1 disk is corrupted, and reconstruct it from the remaining disks?
I ran md/raid6 for awhile, do you mean remap the bad sector on the fly? Linux/md raid does not do this afaik.
No, i mean, if one disk does silent corruption
What would the error look like? Both md/Linux & in the 3ware manual recommend you run a 'check' across the raid at least once a week (3ware/raid-verify) and md/Linux in Debian runs a check once a month I believe to eliminate these issues.
Quoted 7 lines
Quoted 3 lines
If you are asking whether a read error of a latent sector from the one disk will result it reading the data from the second disk that is a good question.
im asking, if one disk in a raid6 setup suddenly decides to flip a few bits in some bytes, will it be able to detect that in a scan, and correct it? i cant see how it can do it on raid5, but maybe raid6?
No, not really.
I've been investigating silent corruption for a quite a while now, and it looks more or less like this.
During a "check" action it'll be detected. During normal operation - it won't be detected.
Normal (non-degraded) raid5/6 reads don't read parity (or Q syndrome), they just read data. So they have no idea that something went bad. Now, worse news is that you cannot really fix it automagically, even after detecting by a "check" procedure. A "repair" will overwrite parity and Q syndrome, with new values (new = calculated from what it seems to be data blocks).
It is possible (by the theory of Q syndrome, per the article you linked) to detect which drive is doing a silent corruption with raid6 (and with some extra assumption, that just one drive is doing that). But it's not implemented.
Greets,
Mike
Quoted 46 lines · Michał Przyłuski
Hi,
2008/12/5 Redeeman <[email protected]>:
Quoted 24 lines · Justin Piszcz
Quoted 23 lines · Redeeman
Quoted 9 lines · Justin Piszcz
Quoted 7 lines · Redeeman
Quoted 6 lines
Hello.
I was looking at the PDFs linked to from the wiki, and found this:
http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdfQuoted 8 lines
Quoted 7 lines
Quoted 5 lines
More specifically, section 4, starting on page 8.
Am I understanding this correctly, in that with raid6, linux is capable of detecting if the content on 1 disk is corrupted, and reconstruct it from the remaining disks?
I ran md/raid6 for awhile, do you mean remap the bad sector on the fly? Linux/md raid does not do this afaik.
No, i mean, if one disk does silent corruption
What would the error look like? Both md/Linux & in the 3ware manual recommend you run a 'check' across the raid at least once a week (3ware/raid-verify) and md/Linux in Debian runs a check once a month I believe to eliminate these issues.
Quoted 6 lines
Quoted 3 lines
If you are asking whether a read error of a latent sector from the one disk will result it reading the data from the second disk that is a good question.
im asking, if one disk in a raid6 setup suddenly decides to flip a few bits in some bytes, will it be able to detect that in a scan, and correct it? i cant see how it can do it on raid5, but maybe raid6?
No, not really.
I've been investigating silent corruption for a quite a while now, and it looks more or less like this.
During a "check" action it'll be detected. During normal operation - it won't be detected.
Normal (non-degraded) raid5/6 reads don't read parity (or Q syndrome), they just read data. So they have no idea that something went bad. Now, worse news is that you cannot really fix it automagically, even after detecting by a "check" procedure. A "repair" will overwrite parity and Q syndrome, with new values (new = calculated from what it seems to be data blocks).
Quoted 5 lines
It is possible (by the theory of Q syndrome, per the article you linked) to detect which drive is doing a silent corruption with raid6 (and with some extra assumption, that just one drive is doing that). But it's not implemented.
I'd like to shamelessly bring in an older related thread:
http://marc.info/?l=linux-raid&m=120605458309825
http://marc.info/?l=linux-raid&m=120618020817057
Maybe someone will get inspired, and will actually write the damned thing :)
Cheers
Peter
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
2008/12/5 Peter Rabbitson <[email protected]>:
Quoted 53 lines
Quoted 46 lines · Michał Przyłuski
Hi,
2008/12/5 Redeeman <[email protected]>:
Quoted 24 lines · Justin Piszcz
Quoted 23 lines · Redeeman
Quoted 9 lines · Justin Piszcz
Quoted 7 lines · Redeeman
Quoted 5 lines
Hello.
I was looking at the PDFs linked to from the wiki, and found this:
http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdfQuoted 8 lines
Quoted 7 lines
Quoted 5 lines
More specifically, section 4, starting on page 8.
Am I understanding this correctly, in that with raid6, linux is capable of detecting if the content on 1 disk is corrupted, and reconstruct it from the remaining disks?
I ran md/raid6 for awhile, do you mean remap the bad sector on the fly? Linux/md raid does not do this afaik.
No, i mean, if one disk does silent corruption
What would the error look like? Both md/Linux & in the 3ware manual recommend you run a 'check' across the raid at least once a week (3ware/raid-verify) and md/Linux in Debian runs a check once a month I believe to eliminate these issues.
Quoted 6 lines
Quoted 3 lines
If you are asking whether a read error of a latent sector from the one disk will result it reading the data from the second disk that is a good question.
im asking, if one disk in a raid6 setup suddenly decides to flip a few bits in some bytes, will it be able to detect that in a scan, and correct it? i cant see how it can do it on raid5, but maybe raid6?
No, not really.
I've been investigating silent corruption for a quite a while now, and it looks more or less like this.
During a "check" action it'll be detected. During normal operation - it won't be detected.
Normal (non-degraded) raid5/6 reads don't read parity (or Q syndrome), they just read data. So they have no idea that something went bad. Now, worse news is that you cannot really fix it automagically, even after detecting by a "check" procedure. A "repair" will overwrite parity and Q syndrome, with new values (new = calculated from what it seems to be data blocks).Quoted 5 lines
It is possible (by the theory of Q syndrome, per the article you linked) to detect which drive is doing a silent corruption with raid6 (and with some extra assumption, that just one drive is doing that). But it's not implemented.
Quoted 5 lines
I'd like to shamelessly bring in an older related thread:
http://marc.info/?l=linux-raid&m=120605458309825
http://marc.info/?l=linux-raid&m=120618020817057Maybe someone will get inspired, and will actually write the damned thing :)
I concur. Even without a "fix", just printing information which disk is suspected of doing silent corruption will be helpful. One can at least, fail the disk, and get rid of it. Still better than taking wild guesses what went wrong. I'm a silent corruption maniac myself, keeping md5's of most bigger/more important files, so my judgment might not be fair.
Also, it seems the feature is being asked about about 3-4 times a year, which is probably the second most requested feature after numerous reshape variations.
Regards,
Mike
2008/12/5 Michał Przyłuski <[email protected]>:
Quoted 67 lines
2008/12/5 Peter Rabbitson <[email protected]>:
Quoted 53 lines
Quoted 46 lines · Michał Przyłuski
Hi,
2008/12/5 Redeeman <[email protected]>:
Quoted 24 lines · Justin Piszcz
Quoted 23 lines · Redeeman
Quoted 9 lines · Justin Piszcz
Quoted 7 lines · Redeeman
Quoted 5 lines
Hello.
I was looking at the PDFs linked to from the wiki, and found this:
http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdfQuoted 8 lines
Quoted 7 lines
Quoted 5 lines
More specifically, section 4, starting on page 8.
Am I understanding this correctly, in that with raid6, linux is capable of detecting if the content on 1 disk is corrupted, and reconstruct it from the remaining disks?
I ran md/raid6 for awhile, do you mean remap the bad sector on the fly? Linux/md raid does not do this afaik.
No, i mean, if one disk does silent corruption
What would the error look like? Both md/Linux & in the 3ware manual recommend you run a 'check' across the raid at least once a week (3ware/raid-verify) and md/Linux in Debian runs a check once a month I believe to eliminate these issues.
Quoted 6 lines
Quoted 3 lines
If you are asking whether a read error of a latent sector from the one disk will result it reading the data from the second disk that is a good question.
im asking, if one disk in a raid6 setup suddenly decides to flip a few bits in some bytes, will it be able to detect that in a scan, and correct it? i cant see how it can do it on raid5, but maybe raid6?
No, not really.
I've been investigating silent corruption for a quite a while now, and it looks more or less like this.
During a "check" action it'll be detected. During normal operation - it won't be detected.
Normal (non-degraded) raid5/6 reads don't read parity (or Q syndrome), they just read data. So they have no idea that something went bad. Now, worse news is that you cannot really fix it automagically, even after detecting by a "check" procedure. A "repair" will overwrite parity and Q syndrome, with new values (new = calculated from what it seems to be data blocks).Quoted 5 lines
It is possible (by the theory of Q syndrome, per the article you linked) to detect which drive is doing a silent corruption with raid6 (and with some extra assumption, that just one drive is doing that). But it's not implemented.
Quoted 5 lines
I'd like to shamelessly bring in an older related thread:
http://marc.info/?l=linux-raid&m=120605458309825
http://marc.info/?l=linux-raid&m=120618020817057Maybe someone will get inspired, and will actually write the damned thing :)
I concur. Even without a "fix", just printing information which disk is suspected of doing silent corruption will be helpful. One can at least, fail the disk, and get rid of it. Still better than taking wild guesses what went wrong. I'm a silent corruption maniac myself, keeping md5's of most bigger/more important files, so my judgment might not be fair.
Quoted 6 lines
Also, it seems the feature is being asked about about 3-4 times a year, which is probably the second most requested feature after numerous reshape variations.
Regards,
Mike
I'm also very concerned about silent corruption and we often "verify" our critical large files by performing MD5 verifies against a known good value. Especially when we make copies or move them from one media to another.
But in all the cases of silent corruption I've seen, it was never the disk. Instead I've seen it be the cable, the controller, bad memory, bad power supply, but never the disk itself. Not to say the disk controller could not be the cause, just that I have not seen it.
I did not read the relevant threads, but do they cover all of these sources of silent corruption, or just if a disk is the source?
Thanks
Greg
Signature · 9 lines
Greg Freemyer Litigation Triage Solutions Specialist http://www.linkedin.com/in/gregfreemyer First 99 Days Litigation White Paper - http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf The Norcross Group The Intersection of Evidence & Technology http://www.norcrossgroup.com
Quoted 10 lines · Greg Freemyer
I'm also very concerned about silent corruption and we often "verify" our critical large files by performing MD5 verifies against a known good value. Especially when we make copies or move them from one media to another.
But in all the cases of silent corruption I've seen, it was never the disk. Instead I've seen it be the cable, the controller, bad memory, bad power supply, but never the disk itself. Not to say the disk controller could not be the cause, just that I have not seen it.
Quoted 5 lines
I did not read the relevant threads, but do they cover all of these sources of silent corruption, or just if a disk is the source?
Thanks
Greg
I will second what Greg says, I have debugged a number of corruptions related to filesystems. I have never seen it be the disk, I have seen 3-4 different controllers corrupt (bad PCI/MB interaction-2 different manufacturers controllers, and a bad controller).
And then the #1 issue is actual bad memory or bad power supply in the machine. None of the actual cases I saw actually affected *ONLY* a single disk=they affected all of the disks on the controller, so whatever has to be done would almost have to be done a the filesystem level or the application level. The typical corruption is not data off of the disk, the platters themselves (and the internals of the disk) appear to have very very good corruption detection and correction, it is really really unlikely for a bad sector read to not get caught. The PCI bus only has parity (and likely parity errors on the PCI bus are not being monitored-unless you installed the edac_mc module) so 50% of the errors that happen get missed. This was one of the bad PCI/MB interactions, one of the slots on a certain MB (all of the specific MB with a couple of different companies card) *HAD* to be throttled to not produce corrupt data every 1GB of reads or so.
And internally the controllers often have poor checking, and will miss things if the controller goes bad. The disks themselves appear to have very good internal controls-I have never seen disk electronics screw up and corrupt data either.
Basically don't waste time worrying about the single disk corrupting data silently, worry about everything after the disk first as that is the weakest link of everything and is far far more likely to bite you.
Quoted 52 lines · Michał Przyłuski
Hi,
2008/12/5 Redeeman <[email protected]>:
Quoted 29 lines · Justin Piszcz
Quoted 28 lines · Redeeman
Quoted 12 lines · Justin Piszcz
Quoted 10 lines · Redeeman
Quoted 8 lines
Hello.
I was looking at the PDFs linked to from the wiki, and found this:
http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdfMore specifically, section 4, starting on page 8.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Am I understanding this correctly, in that with raid6, linux is capable of detecting if the content on 1 disk is corrupted, and reconstruct it from the remaining disks?
I ran md/raid6 for awhile, do you mean remap the bad sector on the fly? Linux/md raid does not do this afaik.
No, i mean, if one disk does silent corruption
What would the error look like? Both md/Linux & in the 3ware manual recommend you run a 'check' across the raid at least once a week (3ware/raid-verify) and md/Linux in Debian runs a check once a month I believe to eliminate these issues.
Quoted 7 lines
Quoted 3 lines
If you are asking whether a read error of a latent sector from the one disk will result it reading the data from the second disk that is a good question.
im asking, if one disk in a raid6 setup suddenly decides to flip a few bits in some bytes, will it be able to detect that in a scan, and correct it? i cant see how it can do it on raid5, but maybe raid6?
No, not really.
I've been investigating silent corruption for a quite a while now, and it looks more or less like this.
During a "check" action it'll be detected. During normal operation - it won't be detected.
Normal (non-degraded) raid5/6 reads don't read parity (or Q syndrome), they just read data. So they have no idea that something went bad. Now, worse news is that you cannot really fix it automagically, even after detecting by a "check" procedure. A "repair" will overwrite parity and Q syndrome, with new values (new = calculated from what it seems to be data blocks).
Quoted 4 lines
It is possible (by the theory of Q syndrome, per the article you linked) to detect which drive is doing a silent corruption with raid6 (and with some extra assumption, that just one drive is doing that). But it's not implemented.
thats a shame, it seems like a KILLER feature, but i guess its not too simple to do, or it would have been done already :)
Quoted 7 lines
Greets,
Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · On Friday December 12, [email protected] wrote:
Quoted 5 lines
It is possible (by the theory of Q syndrome, per the article you linked) to detect which drive is doing a silent corruption with raid6 (and with some extra assumption, that just one drive is doing that). But it's not implemented.
thats a shame, it seems like a KILLER feature, but i guess its not too simple to do, or it would have been done already :)
The reason that it hasn't been done is not that it is difficult. Certainly it is not trivial, but more complicated things have been implemented.
The reason that it is not even on my TODO list is that I don't think it is justifiable.
As has been said elsewhere in this thread, silent corruption is rarely if ever caused by the storage device. They tend to have strong CRCs etc which detect bit-flips with greater reliability than the RAID6 algorithm would detect them.
If the silent corruption comes from anywhere else in the system, it is not clear what if anything should be done.
e.g. if the corruption was due to bad memory, there is no behaviour that will reliably do the "right" thing.
In that case, the best that can be done is simply to log any error that is found and let some human figure it out. That is part of the motivation for a monthly 'check'.
I like to think about raid in a similar way to thinking about security issues (after all, we are dealing with data security).
So before implementing any mechanism that might enhance security, I need to have a clear understanding of what the threat model is. In this case, what is the source of corruption.
Then I need a clear understanding on how the enhancement neutralises or logs the threat, and a credible explanation of why it won't increase the risk from some other threat.
If silent corruption is an issue for you then you really need to be doing checks at a much higher level than the md level. A filesystem that does checksums on all blocks (e.g. btrfs), or an application that does them an all files (tripwire) are much more likely to be beneficial than trying to leverage a side-effect of raid6.
I have a similar attitude to 3-way raid1 and voting on the result. I simply don't think it is the right solution.
NeilBrown
Quoted 22 lines · Neil Brown
Quoted 8 lines · On Friday December 12, [email protected] wrote:
Quoted 5 lines
It is possible (by the theory of Q syndrome, per the article you linked) to detect which drive is doing a silent corruption with raid6 (and with some extra assumption, that just one drive is doing that). But it's not implemented.
thats a shame, it seems like a KILLER feature, but i guess its not too simple to do, or it would have been done already :)
The reason that it hasn't been done is not that it is difficult. Certainly it is not trivial, but more complicated things have been implemented.
The reason that it is not even on my TODO list is that I don't think it is justifiable.
As has been said elsewhere in this thread, silent corruption is rarely if ever caused by the storage device. They tend to have strong CRCs etc which detect bit-flips with greater reliability than the RAID6 algorithm would detect them.
Quoted 4 lines
If the silent corruption comes from anywhere else in the system, it is not clear what if anything should be done.
e.g. if the corruption was due to bad memory, there is no behaviour that will reliably do the "right" thing.
I respectfully disagree. Consider this example(and please correct me if my assumptions are wrong)
you have a raid1 setup, with 3 disks, or just for the example, say... 5 disks.
You then force a check, which detects that 4 disks have identical data, while 1 disk differs. Chance would dictate, that the data is SOMEHOW wrong on 1 disk, now, that may well be the fault of the pci bus, ram, or anything, but in my mind, it is very reasonable to assume that the right thing, is to duplicate the data which is identical on 4/5 disks, onto the next disk.
I would also argue, that in the case of raid1, if you 2/5 disks had identical data, and the remaining 3 had differing data, that it still makes for the best choice to "restore" the data which is identical most times, and certainly, i cannot see any reason why it would be a worse thing to do, than just randomly selecting a dataset to "trust".
Granted, if these instances occurs, its something to be very concerned about, and surely requires a human to figure out what is causing it, but that still doesnt mean it shouldnt try to do all it can to keep the users data intact.
As for raid6, as i understand it, you have the ability, with parity and Q syndrome etc, to arrive at the final data in 3 ways, involving different disks, this still allows it to be 2/3 versus 1/3 with correct data, and i would still argue that its much more reasonable to conclude that 1 disk SOMEHOW has wrong data, versus two disks having the SAME wrong data.
I do however get your point, if the corruption is in the controller etc that may actually occur, that 2 disks have same corruption, however, i would still argue that in general, this scheme would be better on average, and since its not possibly to know 100% what causes stuff, i'd say this is the most logical and reasonable action to take.
Am i wrong?
Quoted 29 lines
In that case, the best that can be done is simply to log any error that is found and let some human figure it out. That is part of the motivation for a monthly 'check'.
I like to think about raid in a similar way to thinking about security issues (after all, we are dealing with data security).
So before implementing any mechanism that might enhance security, I need to have a clear understanding of what the threat model is. In this case, what is the source of corruption.
Then I need a clear understanding on how the enhancement neutralises or logs the threat, and a credible explanation of why it won't increase the risk from some other threat.If silent corruption is an issue for you then you really need to be doing checks at a much higher level than the md level. A filesystem that does checksums on all blocks (e.g. btrfs), or an application that does them an all files (tripwire) are much more likely to be beneficial than trying to leverage a side-effect of raid6.
I have a similar attitude to 3-way raid1 and voting on the result. I simply don't think it is the right solution.
NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 14 lines · Neil Brown
Quoted 8 lines · On Friday December 12, [email protected] wrote:
Quoted 5 lines
It is possible (by the theory of Q syndrome, per the article you linked) to detect which drive is doing a silent corruption with raid6 (and with some extra assumption, that just one drive is doing that). But it's not implemented.
thats a shame, it seems like a KILLER feature, but i guess its not too simple to do, or it would have been done already :)
The reason that it hasn't been done is not that it is difficult. Certainly it is not trivial, but more complicated things have been implemented.
Quoted 7 lines
The reason that it is not even on my TODO list is that I don't think it is justifiable.
As has been said elsewhere in this thread, silent corruption is rarely if ever caused by the storage device. They tend to have strong CRCs etc which detect bit-flips with greater reliability than the RAID6 algorithm would detect them.
If by storage device, you mean the actual disk, then yes, it seems that way. At least in practice. In theory hdd manufacturers only guarantee you get a bitflip less often than once every 10^14 or 10^15 bits. Which is quite often. 10^14 bits is roughly the ammount of data read during a resync of a large:ish raidset.
In general, I agree that a checksumming filesystem is more important for data integrity. This is why all new fileservers around here are running Solaris+ZFS instead of Linux.
/Mattias Wadenstein
Quoted 14 lines · Mattias Wadenstein
Quoted 6 lines · Neil Brown
On Friday December 12, [email protected] wrote:
As has been said elsewhere in this thread, silent corruption is rarely if ever caused by the storage device. They tend to have strong CRCs etc which detect bit-flips with greater reliability than the RAID6 algorithm would detect them.
If by storage device, you mean the actual disk, then yes, it seems that way. At least in practice. In theory hdd manufacturers only guarantee you get a bitflip less often than once every 10^14 or 10^15 bits. Which is quite often. 10^14 bits is roughly the ammount of data read during a resync of a large:ish raidset.
Quoted 3 lines
In general, I agree that a checksumming filesystem is more important for data integrity. This is why all new fileservers around here are running Solaris+ZFS instead of Linux.
I've seen many high-end arrays, such as those done by DDN (www.datadirectnet.com), that spend idle time scouring the drives for corruption. They don't think it's insignificant.
I've had Linux RAID5 MD arrays with silent corrupt data, that could have detected/corrected by idle scouring (before a disk goes bad altogether, when the silent/corrupted data on a "good" disk becomes totally unrecoverable).
SSD's like FusionIO have strong ECC built into every write (11 bits per 240 bytes) that extend the usable life of NAND storage from 100K writes per cell (using the industry standard 1 bit per 512 bytes) into decades (they constantly scour too).
Chris