Software RAID 5 and crashes
Hello,
I have been extensively searching for documentation and mailing lists, but was yet unable to answer this question:
Does Linux software RAID 5 (or RAID 4) do ordered writes? (data stripes first, then parity stripes)
Because if the writes are not ordered, parity stripes could be written before data stripes. If the system crashes at this time, reconstruction will try to reconstruct the parity stripes by using the wrong (old) data stripes.
If the writes are ordered, crashes after the write of the data stripe but before the write to the parity stripe do not harm.
Thanks,
Xuân Baldauf.
Quoted 7 lines · On Sunday August 1, [email protected]…
Hello,
I have been extensively searching for documentation and mailing lists, but was yet unable to answer this question:
Does Linux software RAID 5 (or RAID 4) do ordered writes? (data stripes first, then parity stripes)
No, it doesn't impose any ordering between writes of parity and data in the same stripe, and it would not have any material effect on any outcomes if it did.
Quoted 8 lines
Because if the writes are not ordered, parity stripes could be written before data stripes. If the system crashes at this time, reconstruction will try to reconstruct the parity stripes by using the wrong (old) data stripes.
If the writes are ordered, crashes after the write of the data stripe but before the write to the parity stripe do not harm.
When the system crashes, the RAID5 manager assume that all data blocks are correct and all parity blocks are suspect. It checks all parity blocks against corresponding data and corrects those that don't match.
If a write is "in progress" - i.e. it has started but not all data and parity has been written, then either the "old" data or the "new" data are equally correct. The only thing that needs to be guaranteed after a crash, and the only thing that can be guaranteed, is that any data that has been reported as "safe-in-storage" really is safe. That is all journalling filesystems, or anything else, assume.
Hope that helps.
NeilBrown
Quoted 7 lines · Neil Brown
Quoted 4 lines · On Sunday August 1, [email protected]…
Hello,
I have been extensively searching for documentation and mailing lists, but was yet unable to answer this question:
Quoted 9 lines
Quoted 3 lines
Does Linux software RAID 5 (or RAID 4) do ordered writes? (data stripes first, then parity stripes)
No, it doesn't impose any ordering between writes of parity and data in the same stripe, and it would not have any material effect on any outcomes if it did.
I disagree. :-)
Quoted 19 lines
Quoted 7 lines
Because if the writes are not ordered, parity stripes could be written before data stripes. If the system crashes at this time, reconstruction will try to reconstruct the parity stripes by using the wrong (old) data stripes.
If the writes are ordered, crashes after the write of the data stripe but before the write to the parity stripe do not harm.
When the system crashes, the RAID5 manager assume that all data blocks are correct and all parity blocks are suspect. It checks all parity blocks against corresponding data and corrects those that don't match.
If a write is "in progress" - i.e. it has started but not all data and parity has been written, then either the "old" data or the "new" data are equally correct. The only thing that needs to be guaranteed after a crash, and the only thing that can be guaranteed, is that any data that has been reported as "safe-in-storage" really is safe. That is all journalling filesystems, or anything else, assume.
Quoted 3 lines
Hope that helps.
Yes, thank you, the clear statement, that writes are not ordered, helps. :-) It is also relieving to read that data blocks are always preferred to parity blocks, so that data blocks never can become scrambled by unmatching other data blocks and parity blocks (at least in non-degraded mode).
Unfortunately, it still does not make me satisfied, because: The asymmetry of "all data blocks are correct, all parity blocks are suspect" should be exploited. Consider 4 disks joined as RAID 5. There are 4 stripes (s0, s1, s2, s3), where s3 is the parity stripe.
* <>Example 0: s3,s2,s1 are written to disk, while s0 is not written to disk for some reason. The system crashes. What happens at reconstruction? s3 gets replaced by s0 XOR s1 XOR s2. s0 contains old (read: wrong) data.
* Example 1: s0,s1,s2 are written to disk, while s3 is not written to disk for some reason. The system crashes. What happens at reconstruction? s3 gets replaced by s0 XOR s1 XOR s2. s0 does not contain old data. The stripe which contains the old data (s3) is replaced anyway during reconstruction.
If we now consider, that for each disk (as member of a RAID 5), there are parity stripes and there are data stripes. Doesn't it make sense to prefer data blocks over partiy blocks when writing, just to get more cases of "example 1" against "example 0" than without this preference?
One could even imagine to intensionally postpone the parity block writing for some time in favour of peak throughput. The RAID 5 device looses its rendundancy for some bounded time at a bounded region of its space, but this may be acceptable for certain applications, I think.
Quoted 4 lines
NeilBrown
Xuân Baldauf.
Quoted 1 line · In article <[email protected]> you wrote:
Unfortunately, it still does not make me satisfied, because: The
IMHO the current Raid5 implementation can be better in terms of crash recovery, BUT one should not forget that RAID5 is simply pretty bad even if ideally coded with ordered write and transaction sequence numbers. Therefore it is perhaps better to spend more time on alternatives or at least in communicating the inherent problems of raid5 to the users which want more (like you seem to need?)
Quoted 2 lines
* <>Example 0: s3,s2,s1 are written to disk, while s0 is not written
* Example 1: s0,s1,s2 are written to disk, while s3 is not written
One has to state clearly state that the failure to commit raid stripes in a crash result in guranteed data loss even when the data is written redundant. This is one of the raid5 problems, and it is even worse in a degregated scenario.
Quoted 4 lines
If we now consider, that for each disk (as member of a RAID 5), there are parity stripes and there are data stripes. Doesn't it make sense to prefer data blocks over partiy blocks when writing, just to get more cases of "example 1" against "example 0" than without this preference?
I guess it depends on what "prefer" mean? Do you think about write ordering with a performance impact or about some minor tweaking with possibly no use (ie. because sorting the requests will be reordered in the controller and device anyway)
Quoted 2 lines
One could even imagine to intensionally postpone the parity block writing for some time in favour of peak throughput.
That would only help, if you defer the decision when the data is stable to that delayed time, which is for sure a performance killer.
Quoted 3 lines
The RAID 5 device
looses its rendundancy for some bounded time at a bounded region of its space, but this may be acceptable for certain applications, I think.
Well, I cant imagine applications which are sensitive to losing random file appends without a transaction protocol, but are not sensitive to degregaded redundancy?
BTW: I dont think hardware raid5 of any vendor performs much better?
Greetings
Bernd
Signature · 2 lines
eckes privat - http://www.eckes.org/ Project Freefire - http://www.freefire.org/
Quoted 11 lines · On Sunday August 1, [email protected]…
Unfortunately, it still does not make me satisfied, because: The asymmetry of "all data blocks are correct, all parity blocks are suspect" should be exploited.
Consider 4 disks joined as RAID 5. There are 4 stripes (s0, s1, s2, s3), where s3 is the parity stripe.* <>Example 0: s3,s2,s1 are written to disk, while s0 is not written to disk for some reason. The system crashes. What happens at reconstruction? s3 gets replaced by s0 XOR s1 XOR s2. s0 contains old (read: wrong) data.
Define "wrong"....
Supposing it had actually crashed a couple of milliseconds earlier, when none of the blocks had been written. Is that any more wrong? or less?
When ever a machine crashes, it is wrong. Whenever a machine crashes
you lose data. A little more or a little bit less data being "lost"
in neither here nor there. The only thing that it really makes sense to worry about is consistency. RAID5 provides all the consistency it can, and leaves the rest up to higher layers.
Quoted 11 lines
* Example 1: s0,s1,s2 are written to disk, while s3 is not written to disk for some reason. The system crashes. What happens at reconstruction? s3 gets replaced by s0 XOR s1 XOR s2. s0 does not contain old data. The stripe which contains the old data (s3) is replaced anyway during reconstruction.
If we now consider, that for each disk (as member of a RAID 5), there are parity stripes and there are data stripes. Doesn't it make sense to prefer data blocks over partiy blocks when writing, just to get more cases of "example 1" against "example 0" than without this preference?
No, and definitely not at the cost of delaying any writes or complicating the code at all.
NeilBrown