Skip to contentSkip to conversation

Unmountable 4-drive btrfs RAID after drive failure

DISCUSSION gara 1 participant 1 message Dec 1, 2023 Btrfs file system

lore
Share & tools
lore

gara

author
#0 lore raw

tl;dr: btrfs RAID doesn't mount after single drive failure, btrfs restore also can't make progress - pls help save my data

I am currently running btrfs rescue chunk-recover and will for another day or two, but before I ask it to rebuild anything (didnt get a prompt to yet) I would be very thankful for any pointers on what else I could do/check to get as much data as possible off of the FS. Data loss would not be critical but it would hurt.

The situation / process so far:

I run a 4-drive NAS that with LUKS->BTRFS-RAID setup over all disks (don't remember if it's a RAID5/6 or RAID10, is there a way to check?), my rootfs is on a different drive so i can still boot into my box

Quoted 1 line

uname -a

Linux helios4 5.15.93-mvebu #23.02.2 SMP Fri Feb 17 23:47:24 UTC 2023 armv7l GNU/Linux

Quoted 1 line

btrfs --version

btrfs-progs v5.10.1

I noticed yesterday that my SSHFS mounts to this NAS are not writable.

After getting Input/Output Errors reading the mountpoint from bash directly on the device, I rebooted.

After reboot /dev/sdd** is missing (it's clicking, so probably ded), SMART info for the 3 remaining drives looks healthy. Normal luksOpen works on all 3 drives, but mount fails and dmesg prints

[  465.182744] BTRFS error (device dm-2): devid 4 uuid e79df2b2-5ad0-43b3-a280-3e240b9a2cff is missing
[  465.182759] BTRFS error (device dm-2): failed to read the system array: -2
[  465.183063] BTRFS error (device dm-2): open_ctree failed
Quoted 1 line

btrfs fi show

Label: none uuid: 90f61aa9-e02c-4c47-ac31-559f08083565
Total devices 1 FS bytes used 384.00KiB
devid 1 size 1.82TiB used 2.02GiB path /dev/sdd

warning, device 4 is missing
parent transid verify failed on 92405825536 wanted 40448 found 31913
parent transid verify failed on 92405825536 wanted 40448 found 31913
Ignoring transid failure
parent transid verify failed on 92405841920 wanted 33513 found 31913
parent transid verify failed on 92405841920 wanted 33513 found 31913
Ignoring transid failure
parent transid verify failed on 92405858304 wanted 35618 found 31923
parent transid verify failed on 92405858304 wanted 35618 found 31923
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=92405809152 item=6 parent level=1 child bytenr=92405858304 child level=1 Couldn't read chunk tree
Label: none uuid: 4a957a95-f784-41c3-a46c-13e975867a9f
Total devices 4 FS bytes used 2.50TiB
devid 1 size 5.46TiB used 1.25TiB path /dev/mapper/redA
devid 2 size 5.46TiB used 389.03GiB path /dev/mapper/blueB
devid 3 size 5.46TiB used 1.25TiB path /dev/mapper/redC
*** Some devices missing

mounting degraded fails too

Quoted 1 line

mount -t btrfs -o ro,degraded /dev/mapper/redA /mnt/nas/

mount: /mnt/nas: wrong fs type, bad option, bad superblock on /dev/mapper/redA, missing codepage or helper program, or other error.

and dmesg prints after mounting:

[56750.747574] BTRFS info (device dm-2): allowing degraded mounts
[56750.747590] BTRFS info (device dm-2): disk space caching is enabled
[56750.747594] BTRFS info (device dm-2): has skinny extents
[56750.932131] BTRFS error (device dm-2): parent transid verify failed on 92405825536 wanted 40448 found 31913
[56750.939473] BTRFS error (device dm-2): failed to read chunk tree: -5
[56750.994132] BTRFS error (device dm-2): open_ctree failed

I spent a good day googling and trying all nondestructive things i could find to get the FS mounted or the data off, i.e.

Quoted 1 line

btrfs restore -si /dev/mapper/redA /mnt/rec/

warning, device 4 is missing
parent transid verify failed on 92405825536 wanted 40448 found 31913
parent transid verify failed on 92405825536 wanted 40448 found 31913
Ignoring transid failure
parent transid verify failed on 92405841920 wanted 33513 found 31913
parent transid verify failed on 92405841920 wanted 33513 found 31913
Ignoring transid failure
parent transid verify failed on 92405858304 wanted 35618 found 31923
parent transid verify failed on 92405858304 wanted 35618 found 31923
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=92405809152 item=6 parent level=1 child bytenr=92405858304 child level=1 Couldn't read chunk tree
Could not open root, trying backup super
warning, device 1 is missing
warning, device 2 is missing
warning, device 4 is missing
bad tree block 92405825536, bytenr mismatch, want=92405825536, have=0
Couldn't read chunk tree
Could not open root, trying backup super
warning, device 1 is missing
warning, device 2 is missing
warning, device 4 is missing
bad tree block 92405825536, bytenr mismatch, want=92405825536, have=0
Couldn't read chunk tree
Could not open root, trying backup super

Quoted 1 line

btrfs check --readonly --force /dev/mapper/redA

Opening filesystem to check...
warning, device 4 is missing
parent transid verify failed on 92405825536 wanted 40448 found 31913
parent transid verify failed on 92405825536 wanted 40448 found 31913
Ignoring transid failure
parent transid verify failed on 92405841920 wanted 33513 found 31913
parent transid verify failed on 92405841920 wanted 33513 found 31913
Ignoring transid failure
parent transid verify failed on 92405858304 wanted 35618 found 31923
parent transid verify failed on 92405858304 wanted 35618 found 31923
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=92405809152 item=6 parent level=1 child bytenr=92405858304 child level=1 Couldn't read chunk tree
ERROR: cannot open file system

Quoted 1 line

mount -t btrfs -o ro,degraded,rescue=all /dev/mapper/redA /mnt/nas/

mount: /mnt/nas: wrong fs type, bad option, bad superblock on /dev/mapper/redA, missing codepage or helper program, or other error.

Quoted 1 line

mount -t btrfs -o degraded,rescue=skip_bg /dev/mapper/redA /mnt/nas/

mount: /mnt/nas: wrong fs type, bad option, bad superblock on /dev/mapper/redA, missing codepage or helper program, or other error.

Quoted 1 line

mount -t btrfs -o degraded,ro,recovery /dev/mapper/redA /mnt/nas/

mount: /mnt/nas: wrong fs type, bad option, bad superblock on /dev/mapper/redA, missing codepage or helper program, or other error.

superblock dump in case it helps:

Quoted 1 line

btrfs inspect-internal dump-super /dev/mapper/redA

superblock: bytenr=65536, device=/dev/mapper/redA
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x7f35aa3d [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 4a957a95-f784-41c3-a46c-13e975867a9f
metadata_uuid 4a957a95-f784-41c3-a46c-13e975867a9f
label
generation 54015
root 3023220178944
sys_array_size 193
chunk_root_generation 40448
root_level 1
chunk_root 92405809152
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 24004633395200
bytes_used 2744322326528
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 4
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x161
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation 54015
uuid_tree_generation 54015
dev_item.uuid d64ab077-6466-4c54-8a89-7b60f86cacb9
dev_item.fsid 4a957a95-f784-41c3-a46c-13e975867a9f [match]
dev_item.type 0
dev_item.total_bytes 6001158348800
dev_item.bytes_used 1373349347328
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0