Skip to contentSkip to conversation

critical bugs in md raid5

DISCUSSION Marc Lehmann 4 participants 10 messages Jan 27, 2005 – Jan 29, 2005

lore
Share & tools
lore
#0 lore raw

Hi,

I want to report a number of problems in the current raid5 code, some of which are pretty annoying, some of which require a superblock reformat.

Here's my setup:

- dual AMD opteron with 64-bit kernel, 2.6.10/2.6.8.1
- 5 raid disks, 4 standard ide on hda..hdd, one sata-device (that setup gives a much higher performance than putting every device on it's own ide port).

First, the really bad ones (happened with 2.6.10):

Today, I had a crash that required a hard reset. After the next start, the raid started to rebuild as expected (and, as usually, at 5 times the speed that I get when reading from the raid array, but that's an old problem that I had on 2.4, too).

I only remember that I had all all five disks with an up-to-date sign [UUUUU] during the whole rebuild, which made sense to me, as the data should be up-to-date, despite the raid being out-of-sync. I rebooted during the time, at around 46%. After the next reboot, resyncing properly continued.

Then the rebuild finished:

md: md0: sync done.

*immediately* after that line, I got an ide error:

hda: dma_intr: status=0x59 { DriveReady SeekComplete DataRequest Error }
hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=312581110, high=18, low=10591222, sector=312580370
ide: failed opcode was: unknown
end_request: I/O error, dev hda, sector 312580370

I did verify that the disk block indeed was unreadable. The raid did react:

raid5: Disk failure on hda1, disabling device. Operation continuing on 4 devices RAID5 conf printout:

    --- rd:5 wd:4 fd:1
    disk 0, o:0, dev:hda1
    disk 1, o:1, dev:sda1
    disk 2, o:1, dev:hdd1
    disk 3, o:1, dev:hdc1
    disk 4, o:1, dev:hdb1
   RAID5 conf printout:
    --- rd:5 wd:4 fd:1
    disk 1, o:1, dev:sda1
    disk 2, o:1, dev:hdd1
    disk 3, o:1, dev:hdc1
    disk 4, o:1, dev:hdb1

Interestingly, it immediately started to rebuild the array, with only for disks (/proc/mdstat showed hda1 as F (faulty)):

.<6>md: syncing RAID array md0
md: minimum _guaranteed_ reconstruction speed: 0 KB/sec/disc.
md: using maximum available idle IO bandwith (but not more than 99999 KB/sec) for reconstruction.
md: using 128k window, over a total of 156290816 blocks.
md: resuming recovery of md0 from checkpoint.

Indeed, /proc/mdstat showed that it had 46% (last checkpoint) already in sync, while at the same time only showing 4 disks.

I did:

raidhotremove /dev/md0 /dev/hda1

which worked (rebuild continued, though), then

raidhotadd /dev/md0 /dev/hda1

which worked (rebuilt continued, though). I then decided to reboot.

After the reboot, I no longer had a raid, because the array was dirty and degraded.

This is an important bug, IMHO, as it required me to manually mdadm -C the array, which I am fairly proficient with, but it's still rather risky, because of the following issues:

Sometimes, when the machine crashes, it seems that a dirty array is just as unsafe as a degraded array: when the rebuild starts and hits a bad block somewhere on the disks it chose to read from (bad blocks are very common during hard crashes, as the disk might not have time to properly write the block), it will mark the disk as faulty. Unfortunately, as the array seems to be in some artifical degraded mode, it seems to think yet another disk is faulty, and then stop working, as it thinks it has lost two disks - reformat required.

Then, the reformat might be more difficult than necessary, as, of course, you need to know the "index" numbers of your disks, I mean these numbers:

md0 : active raid5 hda1[5] sda1[1] hdd1[2] hdc1[3] hdb1[4]

mdadm can be made to print the same numbers. Unfortunately, neither mkraid nor mdadm have any waay of working with disk indices that are outside the normal range (i.e. only 0..4 are valid on my 5 disk-array when building, but every raid failure will give the disks a different number, which is in no relation to the original order).

That means that every replacement disk will need paper and pencil work, as the required data is not readily available with user-level commands.

In one case, I ahd two unavailable disks (generated by the problem above) with indices 5 and 6. Simple guesswork (formatting the array superblocks in degraded mode - not a very documented thing, readonly-mount, retry with different order) gave me the correct order, but it's a tedious and error-prone task. I'd say many admins might just format the superblocks in non-degraded mode, and when they detect a problem, it'S already too late because the sync has already started (although it could be that a sync will not damage the disks in any way, after all, it's simple xor).

The summary seems to be that the linux raid driver only protects your data as long as all disks are fine and the machine never crashes.

The last annoying issue is not a bug per se, and has been reported in much larger detail earlier: my array can rebuild at a top speed of 165MB/s, but dd or other tools never read more than about 25-35MB/s top, which is much less than the speed of a single disk - dd'ing from a single disk gives a speed of >50MB/s, and dd'ing from, say, 4 or 5 disks gives me wlel over 200MB/s).

Of course, this last issue is not critical at all - I am working with this problem since 2.4 days :)

Thanks for all the good work that alraedy went into linux, though!

Hope this helps,

Signature · 6 lines
                The choice of a
      -----==-     _GNU_
      ----==-- _       generation     Marc Lehmann
      ---==---(_)__  __ ____  __      [email protected]
      --==---/ / _ \/ // /\ \/ /      http://schmorp.de/
      -=====/_/_//_/\_,_/ /_/\_\      XX11-RIPE
#1 lore raw
Quoted 3 lines · Marc Lehmann

The summary seems to be that the linux raid driver only protects your data as long as all disks are fine and the machine never crashes.

"as long as the machine never crashes". That's correct. If you think about how RAID 5 works there is no way around it. When a write to a single stripe is interrupted (machine crash) and you lose a disk during the recovery a lot of data (even unrelated to the data just written) is lost. That is because there is no way to figure out what part of the data on the stripe belonged to the old and what part to the new write.

But that's nothing inherent in Linux RAID5. It's a generic problem. Pretty much all Software RAID5 implementations have it.

The only way around it is to journal all writes, to make stripe updates atomic, but in general that's too slow unless you have a battery backed up journal device.

There are some tricks to avoid this (e.g. always write to a new disk location and update an disk index atomically), but they tend to be heavily patented and are slower too. They also go far beyond RAID-5 (use disk space less efficiently etc.) and typically need support from the file system to be efficient.

RAID-1 helps a bit, because you either get the old or the new data, but not some corruption. In practice even old data can be a big problem though (e.g. when file system metadata is affected)

Morale: if you really care about your data backup very often and use RAID-1 or get an expensive hardware RAID with battery backup (all the cheap "hardware RAIDs" are equally useless for this)

-Andi

#2 lore raw
Quoted 6 lines · Andi Kleen
Quoted 2 lines · Marc Lehmann

The summary seems to be that the linux raid driver only protects your data as long as all disks are fine and the machine never crashes.

"as long as the machine never crashes". That's correct. If you think about how RAID 5 works there is no way around it. When a write to

I disagree. When not working in degraded mode, it's absolutely reasonable to e.g. use only the non-parity data. A crash with raid5 is in no way different to a crash without raid5 then: either the old data is on the disk, the new data is on the disk, or you had some catastrophic disk event and no data is on the disk.

The case I reported was not a catastrophic failure: either the old or new data was on the disk, and the filesystem journaling (which is ext3) will take care of it. Even if the parity information is not in sync, either old or new data is on the disk.

Quoted 3 lines

a single stripe is interrupted (machine crash) and you lose a disk during the recovery a lot of data (even unrelated to the data just written) is lost.

This is not what I described, in fact, I haven't lost any data, despite having had a number of such problems (I did verify that afterwards, and found no differences. Maybe this is luck, but it seems to happen in the majority of cases, and I ahd a similar problem at least 5 or 6 times because I didn't encounter the bug I reported).

Quoted 2 lines

But that's nothing inherent in Linux RAID5. It's a generic problem. Pretty much all Software RAID5 implementations have it.

Indeed, but I think linux' behaviour is especially poor. For example, the renumbering of the devices or the strange rebuild-restart behaviour (which is definitely a bug) will make recovery unnecessarily complicated.

Quoted 2 lines

RAID-1 helps a bit, because you either get the old or the new data, but not some corruption.

You don't get any magical corruption with RAID5 either... the data contents will either be old, or new. The differnce is that you cannot trust parity.

Quoted 2 lines

In practice even old data can be a big
problem though (e.g. when file system metadata is affected)

Of course, but that's supposed to be worked around by using a journaling file system, right?

Quoted 3 lines

Morale: if you really care about your data backup very often and use RAID-1 or get an expensive hardware RAID with battery backup (all the cheap "hardware RAIDs" are equally useless for this)

Yes, I am thinking of that for some time now, but always had a problem because the affordable ones have low performance. But given linux' effective slower-than-a-single-disk performance it shouldn't be hard to beat nowadays.

There is, however, at least the resyncing with only 4 out of 5 disks, that is doubtlessly a bug somewhere.

Signature · 6 lines
                The choice of a
      -----==-     _GNU_
      ----==-- _       generation     Marc Lehmann
      ---==---(_)__  __ ____  __      [email protected]
      --==---/ / _ \/ // /\ \/ /      http://schmorp.de/
      -=====/_/_//_/\_,_/ /_/\_\      XX11-RIPE
#9 lore raw
In reply to Andi Kleen
Quoted 6 lines · Andi Kleen
Quoted 3 lines · Marc Lehmann

The summary seems to be that the linux raid driver only protects your data as long as all disks are fine and the machine never crashes.

"as long as the machine never crashes". That's correct. If you think

Thanks for your thoughts, btw :)

I forgot to mention that even if data is known to be lost it's much better to return, say, EIO to higher levels than to completely shut down the device (after all, this is no differnce to what other block devices behave).

Also, it's still likely that some old error can be repaired, as the broken non-parity block might be old. This is probably better to be handled in userspace, though, with special tools. But for them it might be vital to get the correct disk index, to be able to detect the stripe layout.

It's usually much faster to repair and verify, as opposed to format and restore, of course.

Signature · 6 lines
                The choice of a
      -----==-     _GNU_
      ----==-- _       generation     Marc Lehmann
      ---==---(_)__  __ ____  __      [email protected]
      --==---/ / _ \/ // /\ \/ /      http://schmorp.de/
      -=====/_/_//_/\_,_/ /_/\_\      XX11-RIPE
#3 lore raw
In reply to Marc Lehmann
Quoted 2 lines

I disagree. When not working in degraded mode, it's absolutely reasonable to e.g. use only the non-parity data. A crash with raid5 is in no way

Yep. But when you go into degraded mode during the crash recovery (before the RAID is fully synced again) you lose.

Quoted 3 lines

different to a crash without raid5 then: either the old data is on the disk, the new data is on the disk, or you had some catastrophic disk event and no data is on the disk.

No, that's not how RAID-5 works. For its redundancy it requires coordinated writes of full stripes (= bigger than fs block) over multiple disks. When you crash in the middle of a write and you lose a disk during crash recovery there is no way to fully reconstruct all the data because the XOR data recovery requires valid data on all disks.

The nasty part there is that it can affect completely unrelated data too (on a traditional disk you normally only lose the data that is currently being written) because of of the relationship between stripes on different disks.

Quoted 5 lines

The case I reported was not a catastrophic failure: either the old or new data was on the disk, and the filesystem journaling (which is ext3) will take care of it. Even if the parity information is not in sync, either old or new data is on the disk.

But you lost a disk in the middle of recovery (any IO error is a lost disk)

Quoted 3 lines

Indeed, but I think linux' behaviour is especially poor. For example, the renumbering of the devices or the strange rebuild-restart behaviour (which is definitely a bug) will make recovery unnecessarily complicated.

There were some suggestions in the past
to be a bit nicer on read IO errors - often if a read fails and you rewrite the block from the reconstructed data the disk would allocate a new block and then be error free again.

The problem is just that when there are user visible IO errors on a modern disk something is very wrong and it will likely run quickly out of replacement blocks, and will eventually fail. That is why Linux "forces" early replacement of the disk on any error - it is the safest thing to do.

Quoted 4 lines
Quoted 1 line

problem though (e.g. when file system metadata is affected)

Of course, but that's supposed to be worked around by using a journaling file system, right?

Nope, journaling is no magical fix for meta data corruption.

-Andi

#4 lore raw

Re: critical bugs in md raid5 and ATA disk failure/recovery modes

Quoted 5 lines · Andi Kleen
Quoted 2 lines

I disagree. When not working in degraded mode, it's absolutely reasonable to e.g. use only the non-parity data. A crash with raid5 is in no way

Yep. But when you go into degraded mode during the crash recovery (before the RAID is fully synced again) you lose.

Hi, see below.

Quoted 6 lines
Quoted 4 lines

different to a crash without raid5 then: either the old data is on the disk, the new data is on the disk, or you had some catastrophic disk event and no data is on the disk.

No, that's not how RAID-5 works. For its redundancy it requires

Hi, I think we might have misunderstood each other.

In fact, I fully agree with you on how raid5 works. However, the current linux raid behaviour is highy suboptimal, and offers much less than your description of raid5 would enable it to do:

- it shouldn't do any kind of re-syncing with two failed disks (as it did in the case I described). That makes no sense and possibly destroys more data.

- it should still satisfy read requests whenever possible (right now, the device is often fully dead, despite the data being there in maybe 100% of the cases).

The thing is, the md raid5 driver itself is able to satisfy most read requests and even some write requests in >= two-failed-disk mode, but it is not so when the failure happens during reconstruction, so in a case were much *more* data can safely be provided is offers *less* then when two disks have failed.

- Vital information about the disk order that might be required for repairing is being destroyed.

I think all of these points are valid, despite the deficiencies in raid5 protection in theory, the linux raid behaviour is much worse in practise.

Quoted 4 lines

The nasty part there is that it can affect completely unrelated data too (on a traditional disk you normally only lose the data that is currently being written) because of of the relationship between stripes on different disks.

Hmm.. indeed, I do not understand this. My reasoning would be as follows:

If I had a bad block, I either lose parity (== no data loss) or I lose a data block (== this data block is lost when the machine crashes).

If unrelated data can get lost (as it is right now, as the device basically is lost), then this seems like a deficiency in the driver.

Quoted 7 lines
Quoted 4 lines

The case I reported was not a catastrophic failure: either the old or new data was on the disk, and the filesystem journaling (which is ext3) will take care of it. Even if the parity information is not in sync, either old or new data is on the disk.

But you lost a disk in the middle of recovery (any IO error is a lost disk)

Yes, and I hit a bug, which I reported.

Quoted 8 lines
Quoted 3 lines

Indeed, but I think linux' behaviour is especially poor. For example, the renumbering of the devices or the strange rebuild-restart behaviour (which is definitely a bug) will make recovery unnecessarily complicated.

There were some suggestions in the past
to be a bit nicer on read IO errors - often if a read fails and you rewrite the block from the reconstructed data the disk would allocate a new block and then be error free again.

(I am not asking for this kind fo automatic recovery, but here are some thoughts on the above):

With modern IDE drives, trying to correct is actually the right thing to do.

At least when the device indicates that it is still working fine, as opposed to be being in pre-failure mode for example due to lack of replacement blocks.

Quoted 2 lines

The problem is just that when there are user visible IO errors on a modern disk something is very wrong and it will likely run quickly out

No, the disk will likely just re-write the block. There are different failure modes on IDE drives, the most likely failure on a crash is that some block couldn't be written due to, say, a power outage, or a hard or soft reset in the middle of a write (sad, but true, many ide disks act like that, there have been discussions about this on LK).

No replacement block will need to be allocated in that case, just the currently written data is lost. And nothing at all will be wrong with the disk in that case, either. So I dispute the "on a modern disk something is very wrong" because that is normal operation of a ATA disk, mandated by standards.

Also, the number of used replacement blocks can be queried on basically all modern ATA disks, and there is a method in place to warn about possible failures, namely SMART.

Please note that most disks can be made to regularly scan their surface and replace blocks, so the device might run out of replacement blocks without any write access from the driver.

So this kind of danger is already possible and likely without linux trying to repair the block, so repairing the block is just normal operation for the drive.

What the drive in many failures is simply tag the block as unreadable (mostly because the checksum/ecc data does not match) and correct this on write. Most drivers will also check the surface and allocate a replacement block automatically if required.

Quoted 1 line

of replacement blocks, and will eventually fail. That is why

Then the drive would be very buggy. If it runs out of replacement blocks it will not suddenly fail, but only be unable to repair the block.

Quoted 2 lines

Linux "forces" early replacement of the disk on any error - it is the safest thing to do.

That is certainly untrue. The safest thing to do would doubtlessly be to make a warning that the disk needs to be replaced but still provide the data as long as possible, instead of killing the device.

It would certainly make sense to no touch the disk in write mode, or, if one is paranoid, in read mode, but right now the device is simply lost.

Quoted 4 lines
Quoted 2 lines

Of course, but that's supposed to be worked around by using a journaling file system, right?

Nope, journaling is no magical fix for meta data corruption.

Meta data corruption of what? The raid device, then yes, the filesystem, then no.

raid5 works by relying on error detetcion of the underlying device. it will suffer form the same kind of corruption that a normal device suffers, i.e. if data gets corrupted silently it's gone. However, in other cases (loud error reporting), the raid device will not corrupt data, as it can always know which data is there and which isn't, juts as with a normal disk.

What raid provides is just more redundant data in normal operation - it doens't suffer from silent data corruption more than a normal disk.

Signature · 6 lines
                The choice of a
      -----==-     _GNU_
      ----==-- _       generation     Marc Lehmann
      ---==---(_)__  __ ____  __      [email protected]
      --==---/ / _ \/ // /\ \/ /      http://schmorp.de/
      -=====/_/_//_/\_,_/ /_/\_\      XX11-RIPE
#8 lore raw
In reply to Andi Kleen
Quoted 4 lines · Andi Kleen

The nasty part there is that it can affect completely unrelated data too (on a traditional disk you normally only lose the data that is currently being written) because of of the relationship between stripes on different disks.

Sorry, I must be a bit dense at times I understood that now, you meant in the case where parity is lost and you have an I/O error in other cases.

Quoted 7 lines

There were some suggestions in the past
to be a bit nicer on read IO errors - often if a read fails and you rewrite the block from the reconstructed data the disk would allocate a new block and then be error free again.

The problem is just that when there are user visible IO errors on a modern disk something is very wrong and it will likely run quickly out

Also, linux already does re-write failed parity blocks automatically on a crash, so whatever damage you might think might be done to the disk will already be done at numerous occasions, as linux in general nor the raid driver in particular checks for bad blocks before rewriting (I don't suggets that it does, just that linux already rewrites failed blocks if it doesn't know about them, and this hasn't been a particular bad problem).

Signature · 6 lines
                The choice of a
      -----==-     _GNU_
      ----==-- _       generation     Marc Lehmann
      ---==---(_)__  __ ____  __      [email protected]
      --==---/ / _ \/ // /\ \/ /      http://schmorp.de/
      -=====/_/_//_/\_,_/ /_/\_\      XX11-RIPE
#5 lore raw

Re: critical bugs in md raid5 and ATA disk failure/recovery modes

In reply to Marc Lehmann

Hi!

Quoted 4 lines
Quoted 4 lines

The nasty part there is that it can affect completely unrelated data too (on a traditional disk you normally only lose the data that is currently being written) because of of the relationship between stripes on different disks.

Well, you could set stripe size to 512B; that way, RAID-5 would be *very* slow, but it should have same characteristics as normal disc w.r.t. crash. Unrelated data would not be lost, and you'd either get old data or new data...

Nasty part might be that if it went to degraded mode (before resync is done), data on disk might silently change; that's bad I guess.

Performance would not be good, also.
Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
#6 lore raw

Re: critical bugs in md raid5 and ATA disk failure/recovery modes

Quoted 4 lines

Well, you could set stripe size to 512B; that way, RAID-5 would be *very* slow, but it should have same characteristics as normal disc w.r.t. crash. Unrelated data would not be lost, and you'd either get old data or new data...

When you lose a disk during recovery you can still lose unrelated data (any "sibling" in a stripe set because its parity information is incomplete). RAID-1 doesn't have this problem though.

-Andi

#7 lore raw

Re: critical bugs in md raid5 and ATA disk failure/recovery modes

Hi!

Quoted 8 lines
Quoted 4 lines

Well, you could set stripe size to 512B; that way, RAID-5 would be *very* slow, but it should have same characteristics as normal disc w.r.t. crash. Unrelated data would not be lost, and you'd either get old data or new data...

When you lose a disk during recovery you can still lose unrelated data (any "sibling" in a stripe set because its parity information is incomplete). RAID-1 doesn't have this problem though.

You are right, I'd have to do soething very special... Like if I know it is 4K filesystem, raid-5 from 5 disks could do the trick. Like

Disk1 Disk2 Disk3 Disk4 Disk5
bytes0-511 512-1023 1024-1535 1536-2048 parity
....

....no, even that does not work. You could add single bit for each 4K saying "this stripe is being written" (with barriers etc) and return read errors if bit is set might actually do the trick, but that's no longer raid-5... (Can ext3 handle error in journal?)
Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!