Skip to contentSkip to conversation

btrfs: volumes: Allow missing devices to be writeable

PATCH In review Qu Wenruo 4 participants 5 messages Aug 29, 2019 – Sep 12, 2019 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

PATCH 1/1 btrfs: volumes: Allow missing devices to be writeable

[BUG]
There is a long existing bug that degraded mounted btrfs can allocate new SINGLE/DUP chunks on a RAID1 fs:
#!/bin/bash

dev1=/dev/test/scratch1
dev2=/dev/test/scratch2
mnt=/mnt/btrfs

umount $mnt &> /dev/null
umount $dev1 &> /dev/null
umount $dev2 &> /dev/null

dmesg -C
mkfs.btrfs -f -m raid1 -d raid1 $dev1 $dev2

wipefs -fa $dev2

mount -o degraded $dev1 $mnt
btrfs balance start --full $mnt
umount $mnt
echo "=== chunk after degraded mount ==="
btrfs ins dump-tree -t chunk $dev1 | grep stripe_len.*type

The result fs will have chunks with SINGLE and DUP only:
=== chunk after degraded mount ===

                  length 33554432 owner 2 stripe_len 65536 type SYSTEM
                  length 1073741824 owner 2 stripe_len 65536 type DATA
                  length 1073741824 owner 2 stripe_len 65536 type DATA|DUP
                  length 219676672 owner 2 stripe_len 65536 type METADATA|DUP
                  length 33554432 owner 2 stripe_len 65536 type SYSTEM|DUP

This behavior greatly breaks the RAID1 tolerance.

Even with missing device replaced, if the device with DUP/SINGLE chunks on them get missing, the whole fs can't be mounted RW any more. And we already have reports that user even can't mount the fs as some essential tree blocks got written to those DUP chunks.

[CAUSE]
The cause is pretty simple, we treat missing devices as non-writable. Thus when we need to allocate chunks, we can only fall back to single device profiles (SINGLE and DUP).

[FIX]
Just consider the missing devices as WRITABLE, so we allocate new chunks on them to maintain old profiles.

---
 fs/btrfs/volumes.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
fs/btrfs/volumes.c +12 -0
index 56f751192a6c..cc30b1fa9306 100644
@@ -7002,6 +7002,18 @@ static int read_one_dev(struct extent_buffer *leaf,
 
 	fill_device_from_item(leaf, dev_item, device);
 	set_bit(BTRFS_DEV_STATE_IN_FS_METADATA, &device->dev_state);
+
+	/*
+	 * We treat missing devices as writable, so that we can maintain
+	 * the existing profiles without degrading to DUP/SINGLE.
+	 */
+	if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state)) {
+		set_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state);
+		list_add(&device->dev_alloc_list,
+			 &fs_devices->alloc_list);
+		fs_devices->rw_devices++;
+	}
+
 	if (test_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state) &&
 	   !test_bit(BTRFS_DEV_STATE_REPLACE_TGT, &device->dev_state)) {
 		device->fs_devices->total_rw_bytes += device->total_bytes;
Signature · 1 line
2.23.0
#1 lore raw
Quoted 39 lines · Qu Wenruo

[BUG]
There is a long existing bug that degraded mounted btrfs can allocate new SINGLE/DUP chunks on a RAID1 fs:
#!/bin/bash

dev1=/dev/test/scratch1
dev2=/dev/test/scratch2
mnt=/mnt/btrfs

umount $mnt &> /dev/null
umount $dev1 &> /dev/null
umount $dev2 &> /dev/null

dmesg -C
mkfs.btrfs -f -m raid1 -d raid1 $dev1 $dev2

wipefs -fa $dev2

mount -o degraded $dev1 $mnt
btrfs balance start --full $mnt
umount $mnt
echo "=== chunk after degraded mount ==="
btrfs ins dump-tree -t chunk $dev1 | grep stripe_len.*type

The result fs will have chunks with SINGLE and DUP only:
=== chunk after degraded mount ===

                  length 33554432 owner 2 stripe_len 65536 type SYSTEM
                  length 1073741824 owner 2 stripe_len 65536 type DATA
                  length 1073741824 owner 2 stripe_len 65536 type DATA|DUP
                  length 219676672 owner 2 stripe_len 65536 type METADATA|DUP
                  length 33554432 owner 2 stripe_len 65536 type SYSTEM|DUP

This behavior greatly breaks the RAID1 tolerance.

Even with missing device replaced, if the device with DUP/SINGLE chunks on them get missing, the whole fs can't be mounted RW any more. And we already have reports that user even can't mount the fs as some essential tree blocks got written to those DUP chunks.

Quoted 8 lines

[CAUSE]
The cause is pretty simple, we treat missing devices as non-writable. Thus when we need to allocate chunks, we can only fall back to single device profiles (SINGLE and DUP).

[FIX]
Just consider the missing devices as WRITABLE, so we allocate new chunks on them to maintain old profiles.

I'm not sure this is the best way to fix it, it makes the meaning of rw_devices ambiguous. A missing device is by definition not readable nor writeable.

This should be tracked separatelly, ie. counting real devices that can be written and devices that can be considered for allocation (with a documented meaning that even missing devices are included).

#2 lore raw
Quoted 49 lines · David Sterba
Quoted 39 lines · Qu Wenruo

[BUG]
There is a long existing bug that degraded mounted btrfs can allocate new SINGLE/DUP chunks on a RAID1 fs:
#!/bin/bash

dev1=/dev/test/scratch1
dev2=/dev/test/scratch2
mnt=/mnt/btrfs

umount $mnt &> /dev/null
umount $dev1 &> /dev/null
umount $dev2 &> /dev/null

dmesg -C
mkfs.btrfs -f -m raid1 -d raid1 $dev1 $dev2

wipefs -fa $dev2

mount -o degraded $dev1 $mnt
btrfs balance start --full $mnt
umount $mnt
echo "=== chunk after degraded mount ==="
btrfs ins dump-tree -t chunk $dev1 | grep stripe_len.*type

The result fs will have chunks with SINGLE and DUP only:
=== chunk after degraded mount ===

                  length 33554432 owner 2 stripe_len 65536 type SYSTEM
                  length 1073741824 owner 2 stripe_len 65536 type DATA
                  length 1073741824 owner 2 stripe_len 65536 type DATA|DUP
                  length 219676672 owner 2 stripe_len 65536 type METADATA|DUP
                  length 33554432 owner 2 stripe_len 65536 type SYSTEM|DUP

This behavior greatly breaks the RAID1 tolerance.

Even with missing device replaced, if the device with DUP/SINGLE chunks on them get missing, the whole fs can't be mounted RW any more. And we already have reports that user even can't mount the fs as some essential tree blocks got written to those DUP chunks.

Quoted 8 lines

[CAUSE]
The cause is pretty simple, we treat missing devices as non-writable. Thus when we need to allocate chunks, we can only fall back to single device profiles (SINGLE and DUP).

[FIX]
Just consider the missing devices as WRITABLE, so we allocate new chunks on them to maintain old profiles.

Quoted 8 lines

I'm not sure this is the best way to fix it, it makes the meaning of rw_devices ambiguous. A missing device is by definition not readable nor writeable.

This should be tracked separatelly, ie. counting real devices that can be written and devices that can be considered for allocation (with a documented meaning that even missing devices are included).

Indeed this sounds much better.

I'd go that direction.

Thanks,
Qu

#3 lore raw
In reply to Qu Wenruo

There is previous work [1]

[1]
https://lore.kernel.org/linux-btrfs/[email protected]/

I guess it was on purpose that missing device is not part of alloc chunk, so to have lesser impact due to writehole bug. My target is to fix the writehole first, and then this and other bugs.

Quoted 3 lines

[FIX]
Just consider the missing devices as WRITABLE, so we allocate new chunks on them to maintain old profiles.

IMO. In a 3-disks raid1 when one of the disk fails, we still need the _new writes_ not to be degraded. Just use two available disks. This fix fails that idea which is being followed now.

Thanks, Anand

author
#4 lore raw
Quoted 14 lines · Anand Jain

There is previous work [1]

[1]
https://lore.kernel.org/linux-btrfs/[email protected]/

I guess it was on purpose that missing device is not part of alloc chunk, so to have lesser impact due to writehole bug. My target is to fix the writehole first, and then this and other bugs.

Quoted 7 lines
Quoted 3 lines

[FIX]
Just consider the missing devices as WRITABLE, so we allocate new chunks on them to maintain old profiles.

 IMO. In a 3-disks raid1 when one of the disk fails, we still need the _new writes_ not to be degraded. Just use two available disks. This fix fails that idea which is being followed now.

So priority comes, use existing, and only when it's impossible, consider degraded/missing device?

Quoted 3 lines

Thanks, Anand