How to fix btrfs in inconsistent state after temp. disk removal
I have a btrfs configured as raid1 on top of several device-mapped LUKS volumes, which live on full HDDs.
There used to be 4 devices. Then I added a fifth disk, wiped and LUKS-encrypted it, unlocked and added it to the filesystem.
While the balance was running, one of the original four disks was disconnected from the system. To my surprise, the device file for the LUKS volume on that disk was not removed. So btrfs didn't notice anything and the balance finished "successfully".
The filesystem now has this structure:
# btrfs filesystem show /mnt2/Main
Label: 'Main' uuid: 6a40d64e-ff83-4f08-a5b2-1236dd4add01
Total devices 5 FS bytes used 8.69TiB
devid 1 size 5.46TiB used 3.49TiB path /dev/mapper/luks-b52cf99d-e8f4-4193-9fe0-694ce5b2b6d7
devid 2 size 5.46TiB used 3.49TiB path /dev/mapper/luks-06f1927d-7ace-4e73-ab1f-efa06970318b
devid 3 size 5.46TiB used 3.49TiB path /dev/mapper/luks-e35e3681-c01d-44aa-9305-a5171f6da24f
devid 4 size 5.46TiB used 3.49TiB path /dev/mapper/luks-d3295051-2128-4282-98ec-d138622a8c09
devid 5 size 5.46TiB used 3.49TiB path /dev/mapper/luks-b3956446-40bd-4546-a103-0b12ba71a9e6But obviously, one of the devices, luks-06f1927d-7ace-4e73-ab1f-efa06970318b, is not really there and btrfs can neither read from nor write to it.
This can be seen from the device stats:
# btrfs device stats -T /mnt2/Main Id Path Write errors Read errors Flush errors Corruption errors Generation errors -- ----------------------------------------------------- ------------ ----------- ------------ ----------------- ----------------- 1 /dev/mapper/luks-b52cf99d-e8f4-4193-9fe0-694ce5b2b6d7 0 0 0 0 0 2 /dev/mapper/luks-06f1927d-7ace-4e73-ab1f-efa06970318b 175662803 116845708 0 0 0 3 /dev/mapper/luks-e35e3681-c01d-44aa-9305-a5171f6da24f 0 0 0 0 0 4 /dev/mapper/luks-d3295051-2128-4282-98ec-d138622a8c09 0 0 0 0 0 5 /dev/mapper/luks-b3956446-40bd-4546-a103-0b12ba71a9e6 0 0 0 0 0
The underlying disk has been reconnected, but I have not unlocked the LUKS volume, because it is bound to be in an inconsistent state.
My question is how this can best be fixed.
I got the advice on the openSUSE forums to wipe the disk an re-encrypt it, creating a fresh new LUKS volume (with a different UUID) and run btrfs replace with the old, fault device as source and the new one as target.
It seems like that should work, but is it the best way to go about it? Or would you recommend something else?
Best,
gabe
Signature · 11 lines
Gabriel Niebler (he/him/his) Senior Full-Stack Web Developer [email protected] SUSE Software Solutions Germany GmbH Frankenstr. 146 90461 Nuernberg Germany www.suse.com #ThePowerOfMany #HaveALotOfFun
Quoted 41 lines · Gabriel Niebler
I have a btrfs configured as raid1 on top of several device-mapped LUKS volumes, which live on full HDDs.
There used to be 4 devices. Then I added a fifth disk, wiped and LUKS- encrypted it, unlocked and added it to the filesystem.
While the balance was running, one of the original four disks was disconnected from the system. To my surprise, the device file for the LUKS volume on that disk was not removed. So btrfs didn't notice anything and the balance finished "successfully".
The filesystem now has this structure:
# btrfs filesystem show /mnt2/Main Label: 'Main' uuid: 6a40d64e-ff83-4f08-a5b2-1236dd4add01 Total devices 5 FS bytes used 8.69TiB devid 1 size 5.46TiB used 3.49TiB path /dev/mapper/luks- b52cf99d-e8f4-4193-9fe0-694ce5b2b6d7 devid 2 size 5.46TiB used 3.49TiB path /dev/mapper/ luks-06f1927d-7ace-4e73-ab1f-efa06970318b devid 3 size 5.46TiB used 3.49TiB path /dev/mapper/luks- e35e3681-c01d-44aa-9305-a5171f6da24f devid 4 size 5.46TiB used 3.49TiB path /dev/mapper/luks- d3295051-2128-4282-98ec-d138622a8c09 devid 5 size 5.46TiB used 3.49TiB path /dev/mapper/luks- b3956446-40bd-4546-a103-0b12ba71a9e6But obviously, one of the devices, luks-06f1927d-7ace-4e73-ab1f-
efa06970318b, is not really there and btrfs can neither read from nor write to it.This can be seen from the device stats:
# btrfs device stats -T /mnt2/Main Id Path Write errors Read errors Flush errors Corruption errors Generation errors -- ----------------------------------------------------- ------------ ----------- ------------ ----------------- ----------------- 1 /dev/mapper/luks-b52cf99d-e8f4-4193-9fe0-694ce5b2b6d7 0 0 0 0 0 2 /dev/mapper/luks-06f1927d-7ace-4e73-ab1f-efa06970318b 175662803 116845708 0 0 0 3 /dev/mapper/luks-e35e3681-c01d-44aa-9305-a5171f6da24f 0 0 0 0 0 4 /dev/mapper/luks-d3295051-2128-4282-98ec-d138622a8c09 0 0 0 0 0 5 /dev/mapper/luks-b3956446-40bd-4546-a103-0b12ba71a9e6 0 0 0 0 0
Quoted 7 lines
The underlying disk has been reconnected, but I have not unlocked the LUKS volume, because it is bound to be in an inconsistent state.
My question is how this can best be fixed.
I got the advice on the openSUSE forums to wipe the disk an re-encrypt it, creating a fresh new LUKS volume (with a different UUID) and run btrfs replace with the old, fault device as source and the new one as target.
It seems like that should work, but is it the best way to go about it?
Yes.
Quoted 1 line
Or would you recommend something else?
I can not come up with any better solution than replace.
Thanks,
Qu
Quoted 4 lines
Best,
gabe