Monitoring for disk failures

Hi all!

I am trying to create a very simple script that would alert in case of disk failures from a RAID Btrfs.

Digging into the code, I have noticed that the "btrfs fi sh" command should display a warning if there is a missing disk. However, testing in a Qemu, I used "drive_del" via QMP to remove a "live" SCSI drive, already mounted as part of a RAID10 array, the "fi sh" command still gave no indication that the drive is missing. Then, I tried removing a scsi disk from the host via "echo 1 >/sys/block/sdX/device/delete" to actually make the kernel SCSI host forget about it, and "fi sh" still doesn't show anything.

I have tested using btrfs-progs v3.12 and kernel 3.13.0.

Do you guys know what's wrong with the setup explained above or do you have any indication on how to detect if there is a failing disk, part of a Btrfs RAID?

Cheers,
Alin.

Alin Dobre posted on Mon, 27 Jan 2014 11:43:33 +0000 as excerpted:

Without actually trying it here... I believe by default that'd update only when there was an I/O error.

Did you try btrfs filesystem show --all-devices? That scans differently.

If that doesn't work try btrfs device scan first as that updates the in- kernel list, then filesystem show. Alternatively, monitor the kernel log for output as the scanned devices show up there.

And if /that/ doesn't work, try show, followed by a probe of all the devices listed by show. But I strongly suspect a device scan will force the update you're looking for.

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

That will show missing with its own probes but kernel does not know that disk is missing.

Nope. Scan does not remove the old (missing disk) entries, I am writing patch(es)..

Thats the best choice as of now OR btrfs fi show -d will show missing

Thanks, Anand

Alin,

[bug] its messy when missing device reappears after its been replaced in RAID1

I am aware of it and working on it. I also reported a more critical bug earlier as below.

[bug] its messy when missing device reappears after its been replaced
in RAID1

We see IO errors when disk goes missing. But NO as of now a mounted FS will never report a missing disk unless you unmount and mount (not remount) then kernel will realize the missing disk

Note that don;t be happy about
btrfs fi show -d <all-devices>
reporting missing disk (when fs is mounted), since its not inline with kernel. Here with -d option btrfs-progs is adding its 'own' intelligence to show disk as missing (what is not what end user want, end user would want to know how btrfs kernel is managing the missing disk and they want to do it by using btrfs-progs. At many places btrfs-progs is way to intelligent than what actually needed. That's wrong).

More to come.

Thanks, Anand