Are linux-fs's drive-fault-tolerant by concept?
Quoted 6 lines · arjan
Quoted 2 lines
You will if it writes and fails to read back. The disk can't invent a sector that is gone.
but linux can if you use an raid1 mirror... maybe we should teach the md layer to write back the data from the other disk on a "bad sector" error.
I have some ugly code that forces all reads from a mirror set to
a specific copy, set via a global sysctl. This lets you do things like make a backup from disk 0, then verify against disk 1 and take action if something is wrong.
------
Chuck
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
You will if it writes and fails to read back. The disk can't invent a sector that is gone.
but linux can if you use an raid1 mirror... maybe we should teach the md layer to write back the data from the other disk on a "bad sector" error.
Quoted 4 lines
I have some ugly code that forces all reads from a mirror set to
a specific copy, set via a global sysctl. This lets you do things like make a backup from disk 0, then verify against disk 1 and take action if something is wrong.
That's interesting. Have you thought of making it read from _both_ disks and check that the data matches, before passing it back?
RAID1 mirrors guard against drive failiure, but if a drive returns bad data, but doesn't report an error, that will usually go unnoticed.
By reading from both disks, and checking that the data was the same, we could guard against broken firmware.
Of course, this would reduce performane quite a bit, but it might have some uses.
John.
Verifying a RAID device (Was: Are linux-fs's drive-fault-tolerant by concept?)
Quoted 11 lines · John Bradford
[Chuck Ebbert writes]
Quoted 4 lines
I have some ugly code that forces all reads from a mirror set to
a specific copy, set via a global sysctl. This lets you do things like make a backup from disk 0, then verify against disk 1 and take action if something is wrong.That's interesting. Have you thought of making it read from _both_ disks and check that the data matches, before passing it back?
RAID1 mirrors guard against drive failiure, but if a drive returns bad data, but doesn't report an error, that will usually go unnoticed.
Checking the disks periodically guards against another important failure mode. Consider this scenario:
Start with a RAID1 mirror using two apparently working disks. The first disk develops a media fault, however, this goes unnoticed because there happens to be no data stored on that part of the media.
Later, a fault is detected during a read of the second disk; the disk is marked off off-line, but all the data is still readable on the first disk so there's no need to panic yet.
You replace the known faulty disk with a new one. The md driver automatically reconstructs the array from the first disk. Since the md driver doesn't know about the filesystem, it reads every disk block, regardless of whether it contains data or not.
The latent error on the first disk is now discovered, and the first disk is now marked off-line. The replacement is only partially reconstructed, so it remains inactive. Oops, there are no active disks left - the md device has failed!
To guard against this, it is a good idea to periodically read all of every disk in a RAID to detect faults early. Ideally, this should be done as another background task, like reconstruction. Checking that the data is valid is then a trivial extra step. If a read fails, you mark the disk relevant disk off-line, as usual. If the parity check fails, you just return a bad blocks list.
Peter