Missing Device Blocks Mount After Replace
Hello,
I've got issues following an apparently successful replace of a failed drive in RAID6.
The replace was interrupted a couple of times. First by motherboard failure. Subsequent tries aborted due uncorrectable errors caused by MB fail. Corrupted files were identified and deleted, and the replace restarted.
Replace and a subsequent balance finished successfully. Now on reboot, mount attempts fail and return errors indicating array is still looking for replaced disk.
Attached are relevant checks I thought to run, as well as the errors and dmesg output following attempts to mount with or without the degraded option
There's a line in dmesg about "btrfs device scan --forget" but running that on the new device does not result in a successful mount. It can't be run on the failed device, as that is no longer in the system.
My Best,
Nathan
###Kernel Version
uname -r
6.12.48+deb13-amd64
###BTRFS version
btrfs-progs v6.14-EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=builtin
###BTRFS Show
btrfs fi show
Label: 'AC5B-btrfs' uuid: 383d6a7d-0825-4ef9-9441-76c4a951b111
Total devices 2 FS bytes used 83.47GiB
devid 1 size 109.91GiB used 94.03GiB path /dev/mapper/crypt-AC5B
devid 2 size 109.91GiB used 94.03GiB path /dev/mapper/crypt-AC5ALabel: '839E-12TB' uuid: 84dbefd6-bf87-4a8e-927d-a3b4c060b326
Total devices 4 FS bytes used 5.59TiB
devid 0 size 10.91TiB used 3.18TiB path /dev/mapper/crypt-J9WE-12TB
devid 1 size 10.91TiB used 3.15TiB path /dev/mapper/crypt-839E-12TB
devid 3 size 10.91TiB used 3.16TiB path /dev/mapper/crypt-9J4E-12TB
devid 4 size 10.91TiB used 3.16TiB path /dev/mapper/crypt-L21H-12TB###BTRFS Check
btrfs check /dev/disk/by-label/839E-12TB
Opening filesystem to check...
warning, device 2 is missing
Checking filesystem on /dev/disk/by-label/839E-12TB
UUID: 84dbefd6-bf87-4a8e-927d-a3b4c060b326
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
ERROR: super total bytes 48000479002624 smaller than real device(s) size 60000598753280
ERROR: mounting this fs may fail for newer kernels
ERROR: this can be fixed by 'btrfs rescue fix-device-size'
[4/8] checking free space tree
[5/8] checking fs roots
[6/8] checking only csums items (without verifying data)
[7/8] checking root refs
[8/8] checking quota groups skipped (not enabled on this FS)
found 6142215630848 bytes used, error(s) found
total csum bytes: 11949613464
total tree bytes: 24013537280
total fs tree bytes: 9576693760
total extent tree bytes: 1187840000
btree space waste bytes: 3086616977
file data blocks allocated: 133676934918144
referenced 10970653450240
### Mount w/o degraded
mount -o subvolid=262 /dev/mapper/crypt-J9WE-12TB /media/12-pool/
mount: /media/12-pool: fsconfig() failed: No such file or directory.
dmesg(1) may have more information after failed mount system call.
dmesg
first mount of filesystem 84dbefd6-bf87-4a8e-927d-a3b4c060b326
[ 8066.864554] BTRFS info (device dm-2): using xxhash64 (xxhash64-generic) checksum algorithm [ 8066.869923] BTRFS error (device dm-2): devid 2 uuid bdfff555-c9cb-4d2e-9b00-3fe194c98930 is missing [ 8066.870023] BTRFS error (device dm-2): failed to read chunk tree: -2
###Mount w/ degraded
mount -t btrfs -o subvolid=262,compress=zstd:5,degraded /dev/disk/by-label/839E-12TB /root/media/12-pool/^C
mount: mounting /dev/disk/by-label/839E-12TB on /root/media/12-pool/ failed: Structure needs cleaning
dmesg
[ 1503.922480] BTRFS info (device dm-2): first mount of filesystem 84dbefd6-bf87-4a8e-927d-a3b4c060b326 [ 1503.922510] BTRFS info (device dm-2): using xxhash64 (xxhash64-generic) checksum algorithm [ 1503.925489] BTRFS warning (device dm-2): devid 2 uuid bdfff555-c9cb-4d2e-9b00-3fe194c98930 is missing [ 1503.944105] BTRFS warning (device dm-2): devid 2 uuid bdfff555-c9cb-4d2e-9b00-3fe194c98930 is missing [ 1504.626893] BTRFS info (device dm-2): bdev <missing disk> errs: wr 3399257, rd 30, flush 479, corrupt 0, gen 0 [ 1504.626921] BTRFS info (device dm-2): bdev /dev/dm-5 errs: wr 3397353, rd 0, flush 470, corrupt 0, gen 0 [ 1504.626936] BTRFS info (device dm-2): bdev /dev/dm-2 errs: wr 0, rd 0, flush 0, corrupt 47, gen 0 [ 1504.626949] BTRFS info (device dm-2): bdev /dev/dm-3 errs: wr 0, rd 0, flush 0, corrupt 335, gen 0 [ 1504.626963] BTRFS info (device dm-2): bdev /dev/dm-4 errs: wr 22390, rd 3124, flush 68, corrupt 32774, gen 0 [ 1504.626979] BTRFS error (device dm-2): replace without active item, run 'device scan --forget' on the target device [ 1504.626988] BTRFS error (device dm-2): failed to init dev_replace: -117 [ 1504.638249] BTRFS error (device dm-2): open_ctree failed: -117