btrfs degraded raid1 gone read-only
Hi,
this looks a bit like the warning on the wiki[1], but since that falls under old kernels it's probably slightly different.
[1] https://btrfs.wiki.kernel.org/index.php/Gotchas#raid1_volumes_only_mountable_once_RW_if_degraded
Kernel 5.11.16-300.fc34.x86_64
How to get there (didn't try to reproduce yet, I'll keep the broken device around a few days if requested to get more traces just in case but it looks reproductible):
- two disks in raid1 profile
- at some point I was lacking slotting space, took one disk out and
mounted just the other. I didn't take any special care so it was most likely mounted rw, but I didn't write any data on it as I expected to put the drives back together later.
- put the disks back together, time passed and tons of data was
written...
- this time I had to "sacrifice" a disk to a silly appliance which
wouldn't let me ssh into it without formatting a disk first, so I backed up data as well as I could, did a full scrub and picked a disk of the two to feed the monster.
The whole volume was also brought down during the operation (no hotplug), so I didn't take any special care in separating the disks e.g. didn't do anything like btrfs device remove...
That's done and I'd like to add the disk back to the raid1, but the original volume can no longer be mounted read-write, so all disk operations fail saying the volume is read-only.
dmesg, first mounting with -o degraded,ro
[ 3324.135037] BTRFS: device fsid c3f4826a-6936-445d-8bae-98233ddda48f devid 1 transid 22031 /dev/dm-1 scanned by systemd-udevd (4812) [ 3340.207257] BTRFS info (device dm-1): allowing degraded mounts [ 3340.207273] BTRFS info (device dm-1): using free space tree [ 3340.207278] BTRFS info (device dm-1): has skinny extents [ 3340.208125] BTRFS warning (device dm-1): devid 2 uuid 0bab0bad-2cd2-42f4-96e8-8dff89d44552 is missing [ 3340.218286] BTRFS warning (device dm-1): devid 2 uuid 0bab0bad-2cd2-42f4-96e8-8dff89d44552 is missing [ 3340.516655] BTRFS info (device dm-1): bdev /dev/mapper/data1 errs: wr 0, rd 0, flush 0, corrupt 0, gen 1799
then mount -o remount,rw failing:
[ 3512.420569] BTRFS info (device dm-1): allowing degraded mounts [ 3512.420615] BTRFS info (device dm-1): using free space tree [ 3512.421180] BTRFS warning (device dm-1): chunk 3475900465152 missing 1 devices, max tolerance is 0 for writable mount [ 3512.421189] BTRFS warning (device dm-1): too many missing devices, writable remount is not allowed [ 3512.421206] BTRFS info (device dm-1): allowing degraded mounts [ 3512.421218] BTRFS info (device dm-1): using free space tree
and btrfs ins dump-tree -t chunk /dev/mapper/data1 does show that chunk is indeed missing, from what I can see only these two are (num_stripes 1):
item 109 key (FIRST_CHUNK_TREE CHUNK_ITEM 3475900465152) itemoff 3995 itemsize 80 length 1073741824 owner 2 stripe_len 65536 type METADATA io_align 65536 io_width 65536 sector_size 4096 num_stripes 1 sub_stripes 1 stripe 0 devid 2 offset 1752381259776 dev_uuid 0bab0bad-2cd2-42f4-96e8-8dff89d44552 item 110 key (FIRST_CHUNK_TREE CHUNK_ITEM 3476974206976) itemoff 3915 itemsize 80 length 33554432 owner 2 stripe_len 65536 type SYSTEM io_align 65536 io_width 65536 sector_size 4096 num_stripes 1 sub_stripes 1 stripe 0 devid 2 offset 1271311368192 dev_uuid 0bab0bad-2cd2-42f4-96e8-8dff89d44552
This can also be confirmed with btrfs fi df which prints a warning:
Data, RAID1: total=2.81TiB, used=2.81TiB
System, RAID1: total=32.00MiB, used=416.00KiB
System, single: total=32.00MiB, used=0.00B
Metadata, RAID1: total=4.00GiB, used=3.61GiB
Metadata, single: total=1.00GiB, used=0.00B GlobalReserve, single: total=512.00MiB, used=0.00B
WARNING: Multiple block group profiles detected, see 'man btrfs(5)'. WARNING: Metadata: single, raid1 WARNING: System: single, raid1
So my guess is that when I mounted the other disk a while ago, these two chunks got created to accomodate e.g. atime updates or whatever it is that it wanted to accomodate.
Later when I remounted both arrays I didn't notice there were single extents in the pool, and scrub only checks that each extent is coherent so didn't reveal any problem. Data later written with two disks correctly got written as raid1, so in practice there is no problem with any of the data and as far as I can see all files are readable and intact.
If I had never put the disks back together or if when they were together I had cleanly removed the disk first I suspect things would have worked fine and the rebalance to single would have copied whatever there was to copy from there just fine... But that doesn't really help me now!
As it mounts read-only fine I just created a new btrfs device and copied data from it overnight, but while this was possible for me I think it's worth addressing.
So, two questions:
- first for my culture, is there a way to tell btrfs to ignore/remove
these chunks? "trust me there's no data I care about in there"? At which point rw mount would probably work again.
I don't think anything automated can be done at this point.
- for other people (read: future me), if like me they don't check btrfs
fi df output, would it make sense to be a bit more verbose about the mixed profile and suggest running a rebalance?
Things were really working well when I put the two disks back together so I think most people wouldn't notice it either, but I'm not sure what to suggest... For all I know there could already have been warnings in dmesg when I put the two disks back together before my check.
Thanks,
Signature · 1 line
Dominique