end to end error recovery musings

In the IO/FS workshop, one idea we kicked around is the need to provide better and more specific error messages between the IO stack and the file system layer.

My group has been working to stabilize a relatively up to date libata + MD based box, so I can try to lay out at least one "appliance like" typical configuration to help frame the issue. We are working on a relatively large appliance, but you can buy similar home appliance (or build them) that use linux to provide a NAS in a box for end users.

The use case that we have is on an ICH6R/AHCI box with 4 large (500+ GB) drives, with some of the small system partitions on a 4-way RAID1 device. The libata version we have is back port of 2.6.18 onto SLES10, so the error handling at the libata level is a huge improvement over what we had before.

Each box has a watchdog timer that can be set to fire after at most 2 minutes.

(We have a second flavor of this box with an ICH5 and P-ATA drives using the non-libata drivers that has a similar use case).

Using the patches that Mark sent around recently for error injection, we inject media errors into one or more drives and try to see how smoothly error handling runs and, importantly, whether or not the error handling will complete before the watchdog fires and reboots the box. If you want to be especially mean, inject errors into the RAID superblocks on 3 out of the 4 drives.

We still have the following challenges:

(1) read-ahead often means that we will retry every bad sector at
least twice from the file system level. The first time, the fs read ahead request triggers a speculative read that includes the bad sector (triggering the error handling mechanisms) right before the real application triggers a read does the same thing. Not sure what the answer is here since read-ahead is obviously a huge win in the normal case.

(2) the patches that were floating around on how to make sure that
we effectively handle single sector errors in a large IO request are critical. On one hand, we want to combine adjacent IO requests into larger IO's whenever possible. On the other hand, when the combined IO fails, we need to isolate the error to the correct range, avoid reissuing a request that touches that sector again and communicate up the stack to file system/MD what really failed. All of this needs to complete in tens of seconds, not multiple minutes.

(3) The timeout values on the failed IO's need to be tuned well (as
was discussed in an earlier linux-ide thread). We cannot afford to hang for 30 seconds, especially in the MD case, since you might need to fail more than one device for a single IO. Prompt error prorogation (say that 4 times quickly!) can allow MD to mask the underlying errors as you would hope, hanging on too long will almost certainly cause a watchdog reboot...

(4) The newish libata+SCSI stack is pretty good at handling disk
errors, but adding in MD actually can reduce the reliability of your system unless you tune the error handling correctly.

We will follow up with specific issues as they arise, but I wanted to lay out a use case that can help frame part of the discussion. I also want to encourage people to inject real disk errors with the Mark patches so we can share the pain ;-)

ric

Probably the only sane thing to do is to remember the bad sectors and avoid attempting reading them; that would mean marking "automatic" versus "explicitly requested" requests to determine whether or not to filter them against a list of discovered bad blocks.

-hpa

And clearing this list when the sector is overwritten, as it will almost certainly be relocated at the disk level. For that matter, a huge win would be to have the MD RAID layer rewrite only the bad sector (in hopes of the disk relocating it) instead of failing the whiole disk. Otherwise, a few read errors on different disks in a RAID set can take the whole system offline. Apologies if this is already done in recent kernels...

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Certainly if the overwrite is successful.

-hpa

And having a way of making this list available to both the filesystem and to a userspace utility, so they can more easily deal with doing a forced rewrite of the bad sector, after determining which file is involved and perhaps doing something intelligent (up to and including automatically requesting a backup system to fetch a backup version of the file, and if it can be determined that the file shouldn't have been changed since the last backup, automatically fixing up the corrupted data block :-).

- Ted

i had a small c program + perl script that would take a badblocks list and figure out which files on an xfs filesystem were trashed, though in the case of xfs it's somewhat easier because you can dump the extents for a file something more generic wouldn't be hard to make work, it also wouldn't be hard to extend this to inodes in some cases though im not sure that there is much you can do there beyond fsck

Yes, current md does this.

So we want a clear path for media read errors from the device up to user-space. Stacked devices (like md) would do appropriate mappings maybe (for raid0/linear at least. Other levels wouldn't tolerate errors).
There would need to be a limit on the number of 'bad blocks' that is recorded. Maybe a mechanism to clear old bad blocks from the list is needed.

Maybe if generic make request gets a request for a block which overlaps a 'bad-block' it returns an error immediately.

Do we want a path in the other direction to handle write errors? The file system could say "Don't worry to much if this block cannot be written, just return an error and I will write it somewhere else"? This might allow md not to fail a whole drive if there is a single write error.
Or is that completely un-necessary as all modern devices do bad-block relocation for us?
Is there any need for a bad-block-relocating layer in md or dm?

What about corrected-error counts? Drives provide them with SMART. The SCSI layer could provide some as well. Md can do a similar thing to some extent. Where these are actually useful predictors of pending failure is unclear, but there could be some value.
e.g. after a certain number of recovered errors raid5 could trigger a background consistency check, or a filesystem could trigger a background fsck should it support that.

Lots of interesting questions... not so many answers.

NeilBrown

Can someone with knowledge of current disk drive behavior confirm that for all drives that support bad block sparing, if an attempt to write to a particular spot on disk results in an error due to bad media at that spot, the disk drive will automatically rewrite the sector to a sector in its spare pool, and automatically redirect that sector to the new location. I believe this should be always true, so presumably with all modern disk drives a write error should mean something very serious has happend.

(Or that someone was in the middle of reconfiguring a FC network and they're running a kernel that doesn't understand why short-duration FC timeouts should be retried. :-)

That's the question. It wouldn't be that hard for filesystems to be able to remap a data block, but (a) it would be much more difficult for fundamental metadata (for example, the inode table), and (b) it's unnecessary complexity if the lower levels in the storage stack should always be doing this for us in the case of media errors anyway.

Somewhat off-topic, but my one big regret with how the dm vs. evms competition settled out was that evms had the ability to perform block device snapshots using a non-LVM volume as the base --- and that EVMS allowed a single drive to be partially managed by the LVM layer, and partially managed by evms.

What this allowed is the ability to do device snapshots and therefore background fsck's without needing to convert the entire laptop disk to using a LVM solution (since to this day I still don't trust initrd's to always do the right thing when I am constantly replacing the kernel for kernel development).

I know, I'm weird, distro users have initrd that seem to mostly work, and it's only wierd developers that try to use bleeding edge kernels with a RHEL4 userspace that suffer, but it's one of the reasons why I've avoided initrd's like the plague --- I've wasted entire days trying to debug problems with the userspace-provided initrd being too old to support newer 2.6 development kernels.

In any case, the reason why I bring this up is that it would be really nice if there was a way with a single laptop drive to be able to do snapshots and background fsck's without having to use initrd's with device mapper.

- Ted

Not quite that simple.

If you write a block aligned size the same size as the physical media block size maybe this is true. If you write a sector on a device with physical sector size larger than logical block size (as allowed by say ATA7) then it's less clear what happens. I don't know if the drive firmware implements multiple "tails" in this case.

On a read error it is worth trying the other parts of the I/O.

Alan

I think that write errors are normally quite serious, but there are exceptions which might be able to be worked around with retries. To Ted's point, in general, a write to a bad spot on the media will cause a remapping which should be transparent (if a bit slow) to us.

I think that this is mostly true, but we also need to balance this against the need for higher levels to get a timely response. In a really large IO, a naive retry of a very large write could lead to a non-responsive system for a very large time...

ric

And losing the I/O could result in a system that is non responsive until the tape restore completes two days later....

Which brings us back to a recent discussion at the file system workshop on being more repair oriented in file system design so we can survive situations like this a bit more reliably ;-)

ric

If all you want is a snapshot, md can do this today ... you just create a RAID-1 resync it and then break it ... of course, you have to have the filesystem mounted above an md device initially ...

James

This is a major part of why I've been trying to push integrated klibc to have all that stuff as a unified "kernel" deliverable. Unfortunately, as you know, Linus apparently rejected the concept "at least for now" at LKS last year.

With klibc this stuff could still be in one single wrapper without funny dependencies, but wouldn't have to be ported to kernel space.

-hpa

This is what will /probably/ happen. The drive should indeed find a spare sector and remap it, if the write attempt encounters a bad spot on the media.

However, with a large enough write, large enough bad-spot-on-media, and a firmware programmed to never take more than X seconds to complete their enterprise customers' I/O, it might just fail.

IMO, somewhere in the kernel, when we receive a read-op or write-op media error, we should immediately try to plaster that area with small writes. Sure, if it's a read-op you lost data, but this method will maximize the chance that you can refresh/reuse the logical sectors in question.

Jeff

One interesting counter example is a smaller write than a full page - say 512 bytes out of 4k.

If we need to do a read-modify-write and it just so happens that 1 of the 7 sectors we need to read is flaky, will this "look" like a write failure?

ric

The current core kernel code can't handle propogating sub-page sized errors up to the file system layers (there is nowhere in the page cache to store 'part of this page is missing'). This is a long standing (four year plus) problem with CD-RW support as well.

For ATA we can at least retrieve the true media sector size now, which may be helpful at the physical layer but the page cache would need to grow some brains to do anything with it.

Some disks are doing their own "read-ahead" in the form of a background media scan. Scans are done on request or periodically (e.g. once per day or once per week) and we have tools that can fetch the scan results from a disk (e.g. a list of unreadable sectors). What we don't have is any way to feed such information to a file system that may be impacted.

Doug Gilbert