Skip to contentSkip to conversation

disk failure but no alert

DISCUSSION Daniel Pocock 2 participants 2 messages Aug 19, 2015 – Aug 20, 2015 Btrfs file system

lore
Share & tools
lore
#0 lore raw

There are two large disks, part of the disks partitioned for MD RAID1 and the rest of the disks partitioned for BtrFs RAID1

One of the disks (/dev/sdd) appears to have failed, there were plenty of alerts from MD (including dmesg and emails) but nothing from the BtrFs filesystem

Could this just be a problem on a sector within the MD RAID1 partition (/dev/sdd2) or is BtrFs failing to alert? If there is a failure on another partition on the same disk, should BtrFs be notified by the kernel in some way and should it consider the filesystem to be at risk?

Should I do anything proactively to stop BtrFs using the /dev/sdd3 partition now? Unfortunately it is not possible to get a new disk to this server in the same day and it may just be shut down until the disk can be replaced.

# uname -a
Linux - 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04)
x86_64 GNU/Linux
# btrfs fi show /dev/sdd3
Label: none  uuid: -----------------------------
    Total devices 2 FS bytes used 1.74TiB
    devid    1 size 4.55TiB used 1.75TiB path /dev/sdd3
    devid    2 size 4.55TiB used 1.75TiB path /dev/sda3

Btrfs v3.17

Here is the dmesg output:

[996932.734999] sd 0:0:3:0: [sdd] 
[996932.735039] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[996932.735047] sd 0:0:3:0: [sdd] 
[996932.735053] Sense Key : Illegal Request [current]
[996932.735062] Info fld=0x80808
[996932.735069] sd 0:0:3:0: [sdd] 
[996932.735078] Add. Sense: Logical block address out of range
[996932.735085] sd 0:0:3:0: [sdd] CDB:
[996932.735089] Write(16): 8a 00 00 00 00 00 00 08 08 08 00 00 00 02 00 00
[996932.735110] end_request: critical target error, dev sdd, sector 526344
[996932.735280] md: super_written gets error=-121, uptodate=0
[996932.735290] md/raid1:md2: Disk failure on sdd2, disabling device.
md/raid1:md2: Operation continuing on 1 devices.
[996932.777853] RAID1 conf printout:
[996932.777917]  --- wd:1 rd:2
[996932.777925]  disk 0, wo:0, o:1, dev:sda2
[996932.777931]  disk 1, wo:1, o:0, dev:sdd2
[996932.794052] RAID1 conf printout:
[996932.794063]  --- wd:1 rd:2
[996932.794069]  disk 0, wo:0, o:1, dev:sda2
#1 lore raw
Quoted 3 lines

is BtrFs failing to alert?

Yes. Btrfs does not do that. as of now.

Only action that it takes is to put FS into readonly mode. That may be fine for ext4 kind of FS but its not correct from the btrfs Volume Manager perspective.

A work in progress at my end to fix that.

Quoted 4 lines
[996932.735110] end_request: critical target error, dev sdd, sector 526344
[996932.735280] md: super_written gets error=-121, uptodate=0
[996932.735290] md/raid1:md2: Disk failure on sdd2, disabling device.

md/raid1:md2: Operation continuing on 1 devices.

MD is a complete volume manager, Btrfs isn't yet.

My recent patch (below), is a step towards that. Its not yet integrated.

[PATCH 3/3] Btrfs: introduce function to handle device offline

HTH, Anand