Skip to contentSkip to conversation

btrfs: More intelligent degraded chunk allocator

PATCH In review Qu Wenruo 7 participants 26 messages Nov 7, 2019 – Dec 2, 2019 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

cover btrfs: More intelligent degraded chunk allocator

This patchset will make btrfs degraded mount more intelligent and provide more consistent profile keeping function.

One of the most problematic aspect of degraded mount is, btrfs may create unwanted profiles.

 # mkfs.btrfs -f /dev/test/scratch[12] -m raid1 -d raid1
 # wipefs -fa /dev/test/scratch2
 # mount -o degraded /dev/test/scratch1 /mnt/btrfs
 # fallocate -l 1G /mnt/btrfs/foobar
 # btrfs ins dump-tree -t chunk /dev/test/scratch1
        item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 1674575872) itemoff 15511 itemsize 80
                length 536870912 owner 2 stripe_len 65536 type DATA
 New data chunk will fallback to SINGLE or DUP.

The cause is pretty simple, when mounted degraded, missing devices can't be used for chunk allocation.
Thus btrfs has to fall back to SINGLE profile.

This patchset will make btrfs to consider missing devices as last resort if current rw devices can't fulfil the profile request.

This should provide a good balance between considering all missing device as RW and completely ruling out missing devices (current mainline behavior).

Qu Wenruo (3):
btrfs: volumes: Refactor device holes gathering into a separate
function
btrfs: volumes: Add btrfs_fs_devices::missing_list to collect missing
devices
btrfs: volumes: Allocate degraded chunks if rw devices can't fullfil a
chunk

 fs/btrfs/block-group.c |  10 ++-
 fs/btrfs/volumes.c     | 170 ++++++++++++++++++++++++++---------------
 fs/btrfs/volumes.h     |   6 ++
 3 files changed, 121 insertions(+), 65 deletions(-)
Signature · 1 line
2.24.0
author
#1 lore raw

PATCH 1/3 btrfs: volumes: Refactor device holes gathering into a separate function

In __btrfs_alloc_chunk() we need to iterate through all rw devices and gather the hole sizes of them.

This function can be refactor into a new function, gather_dev_holes(), to gather holes info from a list_head.

This provides the basis for later degraded chunk feature.

---
 fs/btrfs/volumes.c | 129 ++++++++++++++++++++++++++-------------------
 1 file changed, 74 insertions(+), 55 deletions(-)
fs/btrfs/volumes.c +74 -55
index cdd7af424033..eee5fc1d11f0 100644
@@ -4898,17 +4898,84 @@ static void check_raid56_incompat_flag(struct btrfs_fs_info *info, u64 type)
 	btrfs_set_fs_incompat(info, RAID56);
 }
 
+static int gather_dev_holes(struct btrfs_fs_info *info,
+			    struct btrfs_device_info *devices_info,
+			    int *index, struct list_head *list,
+			    int max_nr_devs, u64 stripe_size, int dev_stripes)
+{
+	struct btrfs_device *device;
+	int ret;
+	int ndevs = 0;
+
+	list_for_each_entry(device, list, dev_alloc_list) {
+		u64 max_avail;
+		u64 dev_offset;
+		u64 total_avail;
+
+		if (!test_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state) &&
+		    !test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state)) {
+			WARN(1, KERN_ERR
+			       "BTRFS: read-only device in alloc_list\n");
+			continue;
+		}
+
+		if (!test_bit(BTRFS_DEV_STATE_IN_FS_METADATA,
+					&device->dev_state) ||
+		    test_bit(BTRFS_DEV_STATE_REPLACE_TGT, &device->dev_state))
+			continue;
+
+		if (device->total_bytes > device->bytes_used)
+			total_avail = device->total_bytes - device->bytes_used;
+		else
+			total_avail = 0;
+
+		/* If there is no space on this device, skip it. */
+		if (total_avail == 0)
+			continue;
+
+		ret = find_free_dev_extent(device,
+					   stripe_size * dev_stripes,
+					   &dev_offset, &max_avail);
+		if (ret && ret != -ENOSPC)
+			break;
+
+		if (ret == 0)
+			max_avail = stripe_size * dev_stripes;
+
+		if (max_avail < BTRFS_STRIPE_LEN * dev_stripes) {
+			if (btrfs_test_opt(info, ENOSPC_DEBUG))
+				btrfs_debug(info,
+			"%s: devid %llu has no free space, have=%llu want=%u",
+					    __func__, device->devid, max_avail,
+					    BTRFS_STRIPE_LEN * dev_stripes);
+			continue;
+		}
+
+		if (ndevs == max_nr_devs) {
+			WARN(1, "%s: found more than %u devices\n", __func__,
+			     max_nr_devs);
+			break;
+		}
+		ret = 0;
+		devices_info[ndevs].dev_offset = dev_offset;
+		devices_info[ndevs].max_avail = max_avail;
+		devices_info[ndevs].total_avail = total_avail;
+		devices_info[ndevs].dev = device;
+		++ndevs;
+	}
+	*index += ndevs;
+	return 0;
+}
+
 static int __btrfs_alloc_chunk(struct btrfs_trans_handle *trans,
 			       u64 start, u64 type)
 {
 	struct btrfs_fs_info *info = trans->fs_info;
 	struct btrfs_fs_devices *fs_devices = info->fs_devices;
-	struct btrfs_device *device;
 	struct map_lookup *map = NULL;
 	struct extent_map_tree *em_tree;
 	struct extent_map *em;
 	struct btrfs_device_info *devices_info = NULL;
-	u64 total_avail;
 	int num_stripes;	/* total number of stripes to allocate */
 	int data_stripes;	/* number of stripes that count for
 				   block group size */
@@ -4983,59 +5050,11 @@ static int __btrfs_alloc_chunk(struct btrfs_trans_handle *trans,
 	 * about the available holes on each device.
 	 */
 	ndevs = 0;
-	list_for_each_entry(device, &fs_devices->alloc_list, dev_alloc_list) {
-		u64 max_avail;
-		u64 dev_offset;
-
-		if (!test_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state)) {
-			WARN(1, KERN_ERR
-			       "BTRFS: read-only device in alloc_list\n");
-			continue;
-		}
-
-		if (!test_bit(BTRFS_DEV_STATE_IN_FS_METADATA,
-					&device->dev_state) ||
-		    test_bit(BTRFS_DEV_STATE_REPLACE_TGT, &device->dev_state))
-			continue;
-
-		if (device->total_bytes > device->bytes_used)
-			total_avail = device->total_bytes - device->bytes_used;
-		else
-			total_avail = 0;
-
-		/* If there is no space on this device, skip it. */
-		if (total_avail == 0)
-			continue;
-
-		ret = find_free_dev_extent(device,
-					   max_stripe_size * dev_stripes,
-					   &dev_offset, &max_avail);
-		if (ret && ret != -ENOSPC)
-			goto error;
-
-		if (ret == 0)
-			max_avail = max_stripe_size * dev_stripes;
-
-		if (max_avail < BTRFS_STRIPE_LEN * dev_stripes) {
-			if (btrfs_test_opt(info, ENOSPC_DEBUG))
-				btrfs_debug(info,
-			"%s: devid %llu has no free space, have=%llu want=%u",
-					    __func__, device->devid, max_avail,
-					    BTRFS_STRIPE_LEN * dev_stripes);
-			continue;
-		}
-
-		if (ndevs == fs_devices->rw_devices) {
-			WARN(1, "%s: found more than %llu devices\n",
-			     __func__, fs_devices->rw_devices);
-			break;
-		}
-		devices_info[ndevs].dev_offset = dev_offset;
-		devices_info[ndevs].max_avail = max_avail;
-		devices_info[ndevs].total_avail = total_avail;
-		devices_info[ndevs].dev = device;
-		++ndevs;
-	}
+	ret = gather_dev_holes(info, devices_info, &ndevs,
+			&fs_devices->alloc_list, fs_devices->rw_devices,
+			max_stripe_size, dev_stripes);
+	if (ret < 0)
+		goto error;
 
 	/*
 	 * now sort the devices by hole size / available space
Signature · 1 line
2.24.0
#2 lore raw

Re: [PATCH 1/3] btrfs: volumes: Refactor device holes gathering into a separate function

fs/btrfs/volumes.c static void check_raid56_incompat_flag(struct btrfs_fs_info *info, u64 type) · line 4905

Quoted 5 lines · Qu Wenruo
fs/btrfs/volumes.c +5 -0
+static int gather_dev_holes(struct btrfs_fs_info *info,
+			    struct btrfs_device_info *devices_info,
+			    int *index, struct list_head *list,
+			    int max_nr_devs, u64 stripe_size, int dev_stripes)
+{

Hi Qu,

What do you think of

static int gather_dev_holes(struct btrfs_fs_info *info,
int *index, u64 stripe_size,
int dev_stripes)

{
	struct btrfs_fs_devices *fs_devices = info->fs_devices;
	struct btrfs_device *device;
	int ret;
	int ndevs = 0;

	list_for_each_entry(device, &fs_devices->alloc_list,
			    dev_alloc_list) {

[...]

btrfs_device_info can be derived from btrfs_fs_info, and *list and max_nr_devs can be derived from btrfs_device_info.

This reduces the number of arguments by 3 and we don't need to pass that odd 'struct list_head *list' which isn't really clear from the type what it is referring to.

Byte,
Johannes

Signature · 9 lines
Johannes Thumshirn                            SUSE Labs Filesystems
[email protected]                                +49 911 74053 689
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5
90409 Nürnberg
Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Felix Imendörffer
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
#3 lore raw

Re: [PATCH 1/3] btrfs: volumes: Refactor device holes gathering into a separate function

Quoted 26 lines · Johannes Thumshirn

fs/btrfs/volumes.c static void check_raid56_incompat_flag(struct btrfs_fs_info *info, u64 type) · line 4905

Quoted 5 lines · Qu Wenruo
fs/btrfs/volumes.c +5 -0
+static int gather_dev_holes(struct btrfs_fs_info *info,
+			    struct btrfs_device_info *devices_info,
+			    int *index, struct list_head *list,
+			    int max_nr_devs, u64 stripe_size, int dev_stripes)
+{

Hi Qu,

What do you think of

static int gather_dev_holes(struct btrfs_fs_info *info,
int *index, u64 stripe_size,
int dev_stripes)

{
	struct btrfs_fs_devices *fs_devices = info->fs_devices;
	struct btrfs_device *device;
	int ret;
	int ndevs = 0;

	list_for_each_entry(device, &fs_devices->alloc_list,
			    dev_alloc_list) {

[...]

Quoted 6 lines

btrfs_device_info can be derived from btrfs_fs_info, and *list and max_nr_devs can be derived from btrfs_device_info.

This reduces the number of arguments by 3 and we don't need to pass that odd 'struct list_head *list' which isn't really clear from the type what it is referring to.

The objective of this refactor is to accept different list, not only fs_devices->alloc_list, but also later fs_devices->missing_list.

So I'm afraid it's not that easy to reduce the number of paramters.

Thanks,
Qu

Quoted 5 lines

Byte,
Johannes

#4 lore raw

Re: [PATCH 1/3] btrfs: volumes: Refactor device holes gathering into a separate function

Quoted 32 lines · Qu Wenruo
Quoted 29 lines · Johannes Thumshirn

fs/btrfs/volumes.c static void check_raid56_incompat_flag(struct btrfs_fs_info *info, u64 type) · line 4905

Quoted 5 lines · Qu Wenruo
fs/btrfs/volumes.c +5 -0
+static int gather_dev_holes(struct btrfs_fs_info *info,
+			    struct btrfs_device_info *devices_info,
+			    int *index, struct list_head *list,
+			    int max_nr_devs, u64 stripe_size, int dev_stripes)
+{

Hi Qu,

What do you think of

static int gather_dev_holes(struct btrfs_fs_info *info,
int *index, u64 stripe_size,
int dev_stripes)

{
	struct btrfs_fs_devices *fs_devices = info->fs_devices;
	struct btrfs_device *device;
	int ret;
	int ndevs = 0;

	list_for_each_entry(device, &fs_devices->alloc_list,
			    dev_alloc_list) {

[...]

btrfs_device_info can be derived from btrfs_fs_info, and *list and max_nr_devs can be derived from btrfs_device_info.

Quoted 8 lines
Quoted 3 lines

This reduces the number of arguments by 3 and we don't need to pass that odd 'struct list_head *list' which isn't really clear from the type what it is referring to.

The objective of this refactor is to accept different list, not only fs_devices->alloc_list, but also later fs_devices->missing_list.

So I'm afraid it's not that easy to reduce the number of paramters.

hmm this makes the api of gather_dev_holes() even more intransparent.

Looking at patch 3/3 I see why you're doing it though.

Signature · 9 lines
Johannes Thumshirn                            SUSE Labs Filesystems
[email protected]                                +49 911 74053 689
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5
90409 Nürnberg
Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Felix Imendörffer
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
author
#5 lore raw

PATCH 2/3 btrfs: volumes: Add btrfs_fs_devices::missing_list to collect missing devices

In reply to Qu Wenruo

This enables btrfs to iterate missing devices separately, without iterating all fs_devices.

This provides the basis for later degraded chunk enhancement.

The change includes:
- Add missing devices to btrfs_fs_devices::missing_list This happens at add_missing_dev() and other locations where missing_devices get increased.

- Remove missing devices from btrfs_fs_devices::missing_list This needs to cover all locations where missing_devices get decreased.

---
 fs/btrfs/volumes.c | 27 +++++++++++++++++++++------
 fs/btrfs/volumes.h |  6 ++++++
 2 files changed, 27 insertions(+), 6 deletions(-)
fs/btrfs/volumes.c +21 -6
index eee5fc1d11f0..a462d8de5d2a 100644
@@ -324,6 +324,7 @@ static struct btrfs_fs_devices *alloc_fs_devices(const u8 *fsid,
 
 	INIT_LIST_HEAD(&fs_devs->devices);
 	INIT_LIST_HEAD(&fs_devs->alloc_list);
+	INIT_LIST_HEAD(&fs_devs->missing_list);
 	INIT_LIST_HEAD(&fs_devs->fs_list);
 	if (fsid)
 		memcpy(fs_devs->fsid, fsid, BTRFS_FSID_SIZE);
@@ -1089,6 +1090,7 @@ static noinline struct btrfs_device *device_list_add(const char *path,
 		if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state)) {
 			fs_devices->missing_devices--;
 			clear_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state);
+			list_del_init(&device->dev_alloc_list);
 		}
 	}
 
@@ -1250,11 +1252,10 @@ static void btrfs_close_one_device(struct btrfs_device *device)
 	if (device->bdev)
 		fs_devices->open_devices--;
 
+	list_del_init(&device->dev_alloc_list);
 	if (test_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state) &&
-	    device->devid != BTRFS_DEV_REPLACE_DEVID) {
-		list_del_init(&device->dev_alloc_list);
+	    device->devid != BTRFS_DEV_REPLACE_DEVID)
 		fs_devices->rw_devices--;
-	}
 
 	if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state))
 		fs_devices->missing_devices--;
@@ -2140,6 +2141,12 @@ int btrfs_rm_device(struct btrfs_fs_info *fs_info, const char *device_path,
 		device->fs_devices->rw_devices--;
 		mutex_unlock(&fs_info->chunk_mutex);
 	}
+	if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state)) {
+		mutex_lock(&fs_info->chunk_mutex);
+		list_del_init(&device->dev_alloc_list);
+		device->fs_devices->missing_devices--;
+		mutex_unlock(&fs_info->chunk_mutex);
+	}
 
 	mutex_unlock(&uuid_mutex);
 	ret = btrfs_shrink_device(device, 0);
@@ -2184,9 +2191,6 @@ int btrfs_rm_device(struct btrfs_fs_info *fs_info, const char *device_path,
 	if (cur_devices != fs_devices)
 		fs_devices->total_devices--;
 
-	if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state))
-		cur_devices->missing_devices--;
-
 	btrfs_assign_next_active_device(device, NULL);
 
 	if (device->bdev) {
@@ -2236,6 +2240,13 @@ int btrfs_rm_device(struct btrfs_fs_info *fs_info, const char *device_path,
 		device->fs_devices->rw_devices++;
 		mutex_unlock(&fs_info->chunk_mutex);
 	}
+	if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state)) {
+		mutex_lock(&fs_info->chunk_mutex);
+		list_add(&device->dev_alloc_list,
+			 &fs_devices->missing_list);
+		device->fs_devices->missing_devices++;
+		mutex_unlock(&fs_info->chunk_mutex);
+	}
 	goto out;
 }
 
@@ -2438,6 +2449,7 @@ static int btrfs_prepare_sprout(struct btrfs_fs_info *fs_info)
 	seed_devices->opened = 1;
 	INIT_LIST_HEAD(&seed_devices->devices);
 	INIT_LIST_HEAD(&seed_devices->alloc_list);
+	INIT_LIST_HEAD(&seed_devices->missing_list);
 	mutex_init(&seed_devices->device_list_mutex);
 
 	mutex_lock(&fs_devices->device_list_mutex);
@@ -6640,6 +6652,7 @@ static struct btrfs_device *add_missing_dev(struct btrfs_fs_devices *fs_devices,
 	fs_devices->num_devices++;
 
 	set_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state);
+	list_add(&device->dev_alloc_list, &fs_devices->missing_list);
 	fs_devices->missing_devices++;
 
 	return device;
@@ -6979,6 +6992,7 @@ static int read_one_dev(struct extent_buffer *leaf,
 			 */
 			device->fs_devices->missing_devices++;
 			set_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state);
+			list_add(&device->dev_alloc_list, &fs_devices->missing_list);
 		}
 
 		/* Move the device to its own fs_devices */
@@ -6992,6 +7006,7 @@ static int read_one_dev(struct extent_buffer *leaf,
 
 			device->fs_devices->missing_devices--;
 			fs_devices->missing_devices++;
+			list_move(&device->dev_alloc_list, &fs_devices->missing_list);
 
 			device->fs_devices = fs_devices;
 		}
fs/btrfs/volumes.h +6 -0
index a7da1f3e3627..9cef4dc4b5be 100644
@@ -253,6 +253,12 @@ struct btrfs_fs_devices {
 	 */
 	struct list_head alloc_list;
 
+	/*
+	 * Devices which can't be found. Projected by chunk_mutex.
+	 * This acts as a fallback allocation list for certain degraded mount.
+	 */
+	struct list_head missing_list;
+
 	struct btrfs_fs_devices *seed;
 	int seeding;
 
Signature · 1 line
2.24.0
#6 lore raw

Re: [PATCH 2/3] btrfs: volumes: Add btrfs_fs_devices::missing_list to collect missing devices

fs/btrfs/volumes.h struct btrfs_fs_devices { · line 260

Quoted 5 lines · Qu Wenruo
fs/btrfs/volumes.h +5 -0
+	/*
+	 * Devices which can't be found. Projected by chunk_mutex.
+	 * This acts as a fallback allocation list for certain degraded mount.
+	 */
+	struct list_head missing_list;

From a quick glance, s/Projected/Protected/

Signature · 9 lines
Johannes Thumshirn                            SUSE Labs Filesystems
[email protected]                                +49 911 74053 689
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5
90409 Nürnberg
Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Felix Imendörffer
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
#7 lore raw

Re: [PATCH 2/3] btrfs: volumes: Add btrfs_fs_devices::missing_list to collect missing devices

In reply to Qu Wenruo
Quoted 2 lines · Qu Wenruo

This enables btrfs to iterate missing devices separately, without iterating all fs_devices.

IMO.
We don't need another list to maintain the missing device. We already have good enough device lists.
The way its been implemented is
Allo_list is the only list from which we shall alloc the chunks. Missing is a state of the device.
A device in the alloc list can be in the Missing state.

If there is missing_list that means the device in the missing list is also possible candidate for the alloc that's messy. Also its not typical to have a larger number of missing devices to constitute its own list.

Thanks, Anand

Quoted 137 lines

This provides the basis for later degraded chunk enhancement.

The change includes:
- Add missing devices to btrfs_fs_devices::missing_list This happens at add_missing_dev() and other locations where missing_devices get increased.

- Remove missing devices from btrfs_fs_devices::missing_list This needs to cover all locations where missing_devices get decreased.

---
  fs/btrfs/volumes.c | 27 +++++++++++++++++++++------
  fs/btrfs/volumes.h |  6 ++++++
  2 files changed, 27 insertions(+), 6 deletions(-)
fs/btrfs/volumes.c +21 -6
index eee5fc1d11f0..a462d8de5d2a 100644
@@ -324,6 +324,7 @@ static struct btrfs_fs_devices *alloc_fs_devices(const u8 *fsid,
 
  	INIT_LIST_HEAD(&fs_devs->devices);
  	INIT_LIST_HEAD(&fs_devs->alloc_list);
+	INIT_LIST_HEAD(&fs_devs->missing_list);
  	INIT_LIST_HEAD(&fs_devs->fs_list);
  	if (fsid)
  		memcpy(fs_devs->fsid, fsid, BTRFS_FSID_SIZE);
@@ -1089,6 +1090,7 @@ static noinline struct btrfs_device *device_list_add(const char *path,
  		if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state)) {
  			fs_devices->missing_devices--;
  			clear_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state);
+			list_del_init(&device->dev_alloc_list);
  		}
  	}
 
@@ -1250,11 +1252,10 @@ static void btrfs_close_one_device(struct btrfs_device *device)
  	if (device->bdev)
  		fs_devices->open_devices--;
 
+	list_del_init(&device->dev_alloc_list);
  	if (test_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state) &&
-	    device->devid != BTRFS_DEV_REPLACE_DEVID) {
-		list_del_init(&device->dev_alloc_list);
+	    device->devid != BTRFS_DEV_REPLACE_DEVID)
  		fs_devices->rw_devices--;
-	}
 
  	if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state))
  		fs_devices->missing_devices--;
@@ -2140,6 +2141,12 @@ int btrfs_rm_device(struct btrfs_fs_info *fs_info, const char *device_path,
  		device->fs_devices->rw_devices--;
  		mutex_unlock(&fs_info->chunk_mutex);
  	}
+	if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state)) {
+		mutex_lock(&fs_info->chunk_mutex);
+		list_del_init(&device->dev_alloc_list);
+		device->fs_devices->missing_devices--;
+		mutex_unlock(&fs_info->chunk_mutex);
+	}
 
  	mutex_unlock(&uuid_mutex);
  	ret = btrfs_shrink_device(device, 0);
@@ -2184,9 +2191,6 @@ int btrfs_rm_device(struct btrfs_fs_info *fs_info, const char *device_path,
  	if (cur_devices != fs_devices)
  		fs_devices->total_devices--;
 
-	if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state))
-		cur_devices->missing_devices--;
-
  	btrfs_assign_next_active_device(device, NULL);
 
  	if (device->bdev) {
@@ -2236,6 +2240,13 @@ int btrfs_rm_device(struct btrfs_fs_info *fs_info, const char *device_path,
  		device->fs_devices->rw_devices++;
  		mutex_unlock(&fs_info->chunk_mutex);
  	}
+	if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state)) {
+		mutex_lock(&fs_info->chunk_mutex);
+		list_add(&device->dev_alloc_list,
+			 &fs_devices->missing_list);
+		device->fs_devices->missing_devices++;
+		mutex_unlock(&fs_info->chunk_mutex);
+	}
  	goto out;
  }
 
@@ -2438,6 +2449,7 @@ static int btrfs_prepare_sprout(struct btrfs_fs_info *fs_info)
  	seed_devices->opened = 1;
  	INIT_LIST_HEAD(&seed_devices->devices);
  	INIT_LIST_HEAD(&seed_devices->alloc_list);
+	INIT_LIST_HEAD(&seed_devices->missing_list);
  	mutex_init(&seed_devices->device_list_mutex);
 
  	mutex_lock(&fs_devices->device_list_mutex);
@@ -6640,6 +6652,7 @@ static struct btrfs_device *add_missing_dev(struct btrfs_fs_devices *fs_devices,
  	fs_devices->num_devices++;
 
  	set_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state);
+	list_add(&device->dev_alloc_list, &fs_devices->missing_list);
  	fs_devices->missing_devices++;
 
  	return device;
@@ -6979,6 +6992,7 @@ static int read_one_dev(struct extent_buffer *leaf,
  			 */
  			device->fs_devices->missing_devices++;
  			set_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state);
+			list_add(&device->dev_alloc_list, &fs_devices->missing_list);
  		}
 
  		/* Move the device to its own fs_devices */
@@ -6992,6 +7006,7 @@ static int read_one_dev(struct extent_buffer *leaf,
 
  			device->fs_devices->missing_devices--;
  			fs_devices->missing_devices++;
+			list_move(&device->dev_alloc_list, &fs_devices->missing_list);
 
  			device->fs_devices = fs_devices;
  		}
fs/btrfs/volumes.h +6 -0
index a7da1f3e3627..9cef4dc4b5be 100644
@@ -253,6 +253,12 @@ struct btrfs_fs_devices {
  	 */
  	struct list_head alloc_list;
 
+	/*
+	 * Devices which can't be found. Projected by chunk_mutex.
+	 * This acts as a fallback allocation list for certain degraded mount.
+	 */
+	struct list_head missing_list;
+
  	struct btrfs_fs_devices *seed;
  	int seeding;
 
#8 lore raw

Re: [PATCH 2/3] btrfs: volumes: Add btrfs_fs_devices::missing_list to collect missing devices

Quoted 11 lines · Anand Jain
Quoted 2 lines · Qu Wenruo

This enables btrfs to iterate missing devices separately, without iterating all fs_devices.

 IMO.
 We don't need another list to maintain the missing device. We already have good enough device lists.
 The way its been implemented is
 Allo_list is the only list from which we shall alloc the chunks. Missing is a state of the device.
 A device in the alloc list can be in the Missing state.

That would cause problem, especially when you only want to use missing device as last resort method.

IIRC it's you mentioned this is a problem in my original design (which put all missing deviecs into alloc_list). Or it's David?

Quoted 3 lines

 If there is missing_list that means the device in the missing list is also possible candidate for the alloc that's messy.

But when you want to avoid missing device, alloc_list and missing_list makes sense.

E.g. 6 devices RAID5, with one missing device, we should *avoid* using missing devices as we have enough (5) devices to allocate from.

Quoted 2 lines

 Also its not typical to have a larger number of missing devices to constitute its own list.

That's just for now.

If we're going to allow RAID10 to lost half of its devices, then it would be a problem.

Thanks,
Qu

Quoted 137 lines

Thanks, Anand

Quoted 132 lines

This provides the basis for later degraded chunk enhancement.

The change includes:
- Add missing devices to btrfs_fs_devices::missing_list This happens at add_missing_dev() and other locations where missing_devices get increased.

- Remove missing devices from btrfs_fs_devices::missing_list This needs to cover all locations where missing_devices get decreased.

---
  fs/btrfs/volumes.c | 27 +++++++++++++++++++++------ fs/btrfs/volumes.h |  6 ++++++

  2 files changed, 27 insertions(+), 6 deletions(-)
fs/btrfs/volumes.c +2 -0
index eee5fc1d11f0..a462d8de5d2a 100644
@@ -324,6 +324,7 @@ static struct btrfs_fs_devices
 *alloc_fs_devices(const u8 *fsid,
         INIT_LIST_HEAD(&fs_devs->devices);
       INIT_LIST_HEAD(&fs_devs->alloc_list);
+    INIT_LIST_HEAD(&fs_devs->missing_list);
       INIT_LIST_HEAD(&fs_devs->fs_list);
       if (fsid)
           memcpy(fs_devs->fsid, fsid, BTRFS_FSID_SIZE);
@@ -1089,6 +1090,7 @@ static noinline struct btrfs_device
 *device_list_add(const char *path,
           if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state)) {
               fs_devices->missing_devices--;
               clear_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state);
+            list_del_init(&device->dev_alloc_list);
           }
       }

  @@ -1250,11 +1252,10 @@ static void btrfs_close_one_device(struct btrfs_device *device)
      if (device->bdev)
          fs_devices->open_devices--;
  +    list_del_init(&device->dev_alloc_list);
      if (test_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state) && -        device->devid != BTRFS_DEV_REPLACE_DEVID) {
-        list_del_init(&device->dev_alloc_list);
+        device->devid != BTRFS_DEV_REPLACE_DEVID) fs_devices->rw_devices--;
-    }
        if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state)) fs_devices->missing_devices--;

fs/btrfs/volumes.c +13 -2
@@ -2140,6 +2141,12 @@ int btrfs_rm_device(struct btrfs_fs_info
 *fs_info, const char *device_path,
           device->fs_devices->rw_devices--;
           mutex_unlock(&fs_info->chunk_mutex);
       }
+    if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state)) {
+        mutex_lock(&fs_info->chunk_mutex);
+        list_del_init(&device->dev_alloc_list);
+        device->fs_devices->missing_devices--;
+        mutex_unlock(&fs_info->chunk_mutex);
+    }
         mutex_unlock(&uuid_mutex);
       ret = btrfs_shrink_device(device, 0);
@@ -2184,9 +2191,6 @@ int btrfs_rm_device(struct btrfs_fs_info
 *fs_info, const char *device_path,
       if (cur_devices != fs_devices)
           fs_devices->total_devices--;
   -    if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state))
-        cur_devices->missing_devices--;
-
       btrfs_assign_next_active_device(device, NULL);
         if (device->bdev) {
@@ -2236,6 +2240,13 @@ int btrfs_rm_device(struct btrfs_fs_info
 *fs_info, const char *device_path,
           device->fs_devices->rw_devices++;
           mutex_unlock(&fs_info->chunk_mutex);
       }
+    if (test_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state)) {
+        mutex_lock(&fs_info->chunk_mutex);
+        list_add(&device->dev_alloc_list,
+             &fs_devices->missing_list);
+        device->fs_devices->missing_devices++;
+        mutex_unlock(&fs_info->chunk_mutex);
+    }
       goto out;
   }

  @@ -2438,6 +2449,7 @@ static int btrfs_prepare_sprout(struct btrfs_fs_info *fs_info)
      seed_devices->opened = 1; INIT_LIST_HEAD(&seed_devices->devices); INIT_LIST_HEAD(&seed_devices->alloc_list);
+    INIT_LIST_HEAD(&seed_devices->missing_list); mutex_init(&seed_devices->device_list_mutex); mutex_lock(&fs_devices->device_list_mutex);

fs/btrfs/volumes.c +3 -0
@@ -6640,6 +6652,7 @@ static struct btrfs_device
 *add_missing_dev(struct btrfs_fs_devices *fs_devices,
       fs_devices->num_devices++;
         set_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state);
+    list_add(&device->dev_alloc_list, &fs_devices->missing_list);
       fs_devices->missing_devices++;
         return device;
@@ -6979,6 +6992,7 @@ static int read_one_dev(struct extent_buffer *leaf,
                */
               device->fs_devices->missing_devices++;
               set_bit(BTRFS_DEV_STATE_MISSING, &device->dev_state);
+            list_add(&device->dev_alloc_list,
 &fs_devices->missing_list);
           }
             /* Move the device to its own fs_devices */
@@ -6992,6 +7006,7 @@ static int read_one_dev(struct extent_buffer *leaf,
                 device->fs_devices->missing_devices--;
               fs_devices->missing_devices++;
+            list_move(&device->dev_alloc_list,
 &fs_devices->missing_list);
                 device->fs_devices = fs_devices;
           }
fs/btrfs/volumes.h +0 -0
index a7da1f3e3627..9cef4dc4b5be 100644
@@ -253,6 +253,12 @@ struct btrfs_fs_devices {
        */
       struct list_head alloc_list;

  +    /*
+     * Devices which can't be found. Projected by chunk_mutex. +     * This acts as a fallback allocation list for certain degraded mount.
+     */
+    struct list_head missing_list;
+
      struct btrfs_fs_devices *seed;
      int seeding;

#9 lore raw

Re: [PATCH 2/3] btrfs: volumes: Add btrfs_fs_devices::missing_list to collect missing devices

Quoted 20 lines · Qu Wenruo
Quoted 11 lines · Anand Jain
Quoted 2 lines · Qu Wenruo

This enables btrfs to iterate missing devices separately, without iterating all fs_devices.

 IMO.
 We don't need another list to maintain the missing device. We already have good enough device lists.
 The way its been implemented is
 Allo_list is the only list from which we shall alloc the chunks. Missing is a state of the device.
 A device in the alloc list can be in the Missing state.

That would cause problem, especially when you only want to use missing device as last resort method.

IIRC it's you mentioned this is a problem in my original design (which put all missing deviecs into alloc_list). Or it's David?

Quoted 1 line
Quoted 8 lines
Quoted 2 lines

 If there is missing_list that means the device in the missing list is also possible candidate for the alloc that's messy.

But when you want to avoid missing device, alloc_list and missing_list makes sense.

E.g. 6 devices RAID5, with one missing device, we should *avoid* using missing devices as we have enough (5) devices to allocate from.

I tend to agree that adding more lists would make things messy. This needs to keep the missing state bit and presence in the list in sync, there's the counter of missing devices. And that there are typically only very few missing devices is also something to consider.

The device selection in __btrfs_alloc_chunk can avoid that. There's an array allocated, with some size related data then it's passed to qsort so the first N drives will be used for the chunk.

In case the degraded allocation is allowed (as mentioned in the other mail, only for the mirrored profiles)

* add the missing device to the array
* update the comparison function btrfs_cmp_device_info to order missing devices to the end

Then the same logic "first N" would work here.

author
#10 lore raw

PATCH 3/3 btrfs: volumes: Allocate degraded chunks if rw devices can't fullfil a chunk

In reply to Qu Wenruo

[PROBLEM]
Btrfs degraded mount will fallback to SINGLE profile if there are not enough devices:

 # mkfs.btrfs -f /dev/test/scratch[12] -m raid1 -d raid1
 # wipefs -fa /dev/test/scratch2
 # mount -o degraded /dev/test/scratch1 /mnt/btrfs
 # fallocate -l 1G /mnt/btrfs/foobar
 # btrfs ins dump-tree -t chunk /dev/test/scratch1
        item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 1674575872) itemoff 15511 itemsize 80
                length 536870912 owner 2 stripe_len 65536 type DATA
 New data chunk will fallback to SINGLE.

If user doesn't balance those SINGLE chunks, even with missing devices replaced, the fs is no longer full RAID1, and a missing device can break the tolerance.

[CAUSE]
The cause is pretty simple, when mounted degraded, missing devices can't be used for chunk allocation.
Thus btrfs has to fall back to SINGLE profile.

[ENHANCEMENT]
To avoid such problem, this patch will:
- Make all profiler reducer/updater to consider missing devices as part of num_devices
- Make chunk allocator to fallback to missing_list as last resort

If we have enough rw_devices, then go regular chunk allocation code. This can avoid allocating degraded chunks.
E.g. for 3 devices RAID1 degraded mount, we will use the 2 existing devices to allocate chunk, avoid degraded chunk.

But if we don't have enough rw_devices, then we check missing devices to allocate degraded chunks.
E.g. for 2 devices RAID1 degraded mount, we have to allocate degraded chunks to keep the RAID1 profile.

---
 fs/btrfs/block-group.c | 10 +++++++---
 fs/btrfs/volumes.c     | 18 +++++++++++++++---
 2 files changed, 22 insertions(+), 6 deletions(-)
fs/btrfs/block-group.c +7 -3
index bf7e3f23bba7..1686fd31679b 100644
@@ -52,11 +52,13 @@ static u64 get_restripe_target(struct btrfs_fs_info *fs_info, u64 flags)
  */
 static u64 btrfs_reduce_alloc_profile(struct btrfs_fs_info *fs_info, u64 flags)
 {
-	u64 num_devices = fs_info->fs_devices->rw_devices;
+	u64 num_devices;
 	u64 target;
 	u64 raid_type;
 	u64 allowed = 0;
 
+	num_devices = fs_info->fs_devices->rw_devices +
+		      fs_info->fs_devices->missing_devices;
 	/*
 	 * See if restripe for this chunk_type is in progress, if so try to
 	 * reduce to the target profile
@@ -1986,7 +1988,8 @@ static u64 update_block_group_flags(struct btrfs_fs_info *fs_info, u64 flags)
 	if (stripped)
 		return extended_to_chunk(stripped);
 
-	num_devices = fs_info->fs_devices->rw_devices;
+	num_devices = fs_info->fs_devices->rw_devices +
+		      fs_info->fs_devices->missing_devices;
 
 	stripped = BTRFS_BLOCK_GROUP_RAID0 | BTRFS_BLOCK_GROUP_RAID56_MASK |
 		BTRFS_BLOCK_GROUP_RAID1_MASK | BTRFS_BLOCK_GROUP_RAID10;
@@ -2981,7 +2984,8 @@ static u64 get_profile_num_devs(struct btrfs_fs_info *fs_info, u64 type)
 
 	num_dev = btrfs_raid_array[btrfs_bg_flags_to_raid_index(type)].devs_max;
 	if (!num_dev)
-		num_dev = fs_info->fs_devices->rw_devices;
+		num_dev = fs_info->fs_devices->rw_devices +
+			  fs_info->fs_devices->missing_devices;
 
 	return num_dev;
 }
fs/btrfs/volumes.c +15 -3
index a462d8de5d2a..4dee1974ceb7 100644
@@ -5052,8 +5052,9 @@ static int __btrfs_alloc_chunk(struct btrfs_trans_handle *trans,
 	max_chunk_size = min(div_factor(fs_devices->total_rw_bytes, 1),
 			     max_chunk_size);
 
-	devices_info = kcalloc(fs_devices->rw_devices, sizeof(*devices_info),
-			       GFP_NOFS);
+	devices_info = kcalloc(fs_devices->rw_devices +
+			       fs_devices->missing_devices,
+			       sizeof(*devices_info), GFP_NOFS);
 	if (!devices_info)
 		return -ENOMEM;
 
@@ -5067,7 +5068,18 @@ static int __btrfs_alloc_chunk(struct btrfs_trans_handle *trans,
 			max_stripe_size, dev_stripes);
 	if (ret < 0)
 		goto error;
-
+	/*
+	 * If rw devices can't fullfil the request, fallback to missing devices
+	 * as last resort.
+	 */
+	if (ndevs < devs_min) {
+		ret = gather_dev_holes(info, devices_info + ndevs, &ndevs,
+				&fs_devices->missing_list,
+				fs_devices->missing_devices,
+				max_stripe_size, dev_stripes);
+		if (ret < 0)
+			goto error;
+	}
 	/*
 	 * now sort the devices by hole size / available space
 	 */
Signature · 1 line
2.24.0
#11 lore raw

Re: [PATCH 3/3] btrfs: volumes: Allocate degraded chunks if rw devices can't fullfil a chunk

Quoted 3 lines · Qu Wenruo

[PROBLEM]
Btrfs degraded mount will fallback to SINGLE profile if there are not enough devices:

Its better to keep it like this for now until there is a fix for the write hole. Otherwise hitting the write hole bug in case of degraded raid1 will be more prevalent.

I proposed a RFC a long time before [1] (also in there, there is a commit id which turned the degraded raid1 profile into single profile (without much write-up on it)).

[1] [PATCH 0/2] [RFC] btrfs: create degraded-RAID1 chunks

Similarly the patch related to the reappearing missing device [2] falls under the same category. Will push for the integration after the write hole fix.

[2] [PATCH] btrfs: handle dynamically reappearing missing device (test case 154).

If you look close enough the original author has quite nicely made sure write hole bug will be very difficultly to hit. These fixes shall make it easy to hit. So its better to work on the write hole first.

I am trying to fix write hole. First attempt has limited success (works fine in two disk raid1 only). Now trying other ways to fix.

Quoted 9 lines
  # mkfs.btrfs -f /dev/test/scratch[12] -m raid1 -d raid1
  # wipefs -fa /dev/test/scratch2
  # mount -o degraded /dev/test/scratch1 /mnt/btrfs
  # fallocate -l 1G /mnt/btrfs/foobar
  # btrfs ins dump-tree -t chunk /dev/test/scratch1
         item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 1674575872) itemoff 15511 itemsize 80
                 length 536870912 owner 2 stripe_len 65536 type DATA
  New data chunk will fallback to SINGLE.
Quoted 3 lines

If user doesn't balance those SINGLE chunks, even with missing devices replaced, the fs is no longer full RAID1, and a missing device can break the tolerance.

As its been discussed quite a lot of time before, the current re-silver/recovery approach for degraded-raid1 (with offload to Single) requires balance. Its kind of known.

Thanks, Anand

Quoted 104 lines

[CAUSE]
The cause is pretty simple, when mounted degraded, missing devices can't be used for chunk allocation.
Thus btrfs has to fall back to SINGLE profile.

[ENHANCEMENT]
To avoid such problem, this patch will:
- Make all profiler reducer/updater to consider missing devices as part of num_devices
- Make chunk allocator to fallback to missing_list as last resort

If we have enough rw_devices, then go regular chunk allocation code.

This can avoid allocating degraded chunks.
E.g. for 3 devices RAID1 degraded mount, we will use the 2 existing devices to allocate chunk, avoid degraded chunk.

But if we don't have enough rw_devices, then we check missing devices to allocate degraded chunks.
E.g. for 2 devices RAID1 degraded mount, we have to allocate degraded chunks to keep the RAID1 profile.

---
  fs/btrfs/block-group.c | 10 +++++++---
  fs/btrfs/volumes.c     | 18 +++++++++++++++---
  2 files changed, 22 insertions(+), 6 deletions(-)
fs/btrfs/block-group.c +7 -3
index bf7e3f23bba7..1686fd31679b 100644
@@ -52,11 +52,13 @@ static u64 get_restripe_target(struct btrfs_fs_info *fs_info, u64 flags)
   */
  static u64 btrfs_reduce_alloc_profile(struct btrfs_fs_info *fs_info, u64 flags)
  {
-	u64 num_devices = fs_info->fs_devices->rw_devices;
+	u64 num_devices;
  	u64 target;
  	u64 raid_type;
  	u64 allowed = 0;
 
+	num_devices = fs_info->fs_devices->rw_devices +
+		      fs_info->fs_devices->missing_devices;
  	/*
  	 * See if restripe for this chunk_type is in progress, if so try to
  	 * reduce to the target profile
@@ -1986,7 +1988,8 @@ static u64 update_block_group_flags(struct btrfs_fs_info *fs_info, u64 flags)
  	if (stripped)
  		return extended_to_chunk(stripped);
 
-	num_devices = fs_info->fs_devices->rw_devices;
+	num_devices = fs_info->fs_devices->rw_devices +
+		      fs_info->fs_devices->missing_devices;
 
  	stripped = BTRFS_BLOCK_GROUP_RAID0 | BTRFS_BLOCK_GROUP_RAID56_MASK |
  		BTRFS_BLOCK_GROUP_RAID1_MASK | BTRFS_BLOCK_GROUP_RAID10;
@@ -2981,7 +2984,8 @@ static u64 get_profile_num_devs(struct btrfs_fs_info *fs_info, u64 type)
 
  	num_dev = btrfs_raid_array[btrfs_bg_flags_to_raid_index(type)].devs_max;
  	if (!num_dev)
-		num_dev = fs_info->fs_devices->rw_devices;
+		num_dev = fs_info->fs_devices->rw_devices +
+			  fs_info->fs_devices->missing_devices;
 
  	return num_dev;
  }
fs/btrfs/volumes.c +15 -3
index a462d8de5d2a..4dee1974ceb7 100644
@@ -5052,8 +5052,9 @@ static int __btrfs_alloc_chunk(struct btrfs_trans_handle *trans,
  	max_chunk_size = min(div_factor(fs_devices->total_rw_bytes, 1),
  			     max_chunk_size);
 
-	devices_info = kcalloc(fs_devices->rw_devices, sizeof(*devices_info),
-			       GFP_NOFS);
+	devices_info = kcalloc(fs_devices->rw_devices +
+			       fs_devices->missing_devices,
+			       sizeof(*devices_info), GFP_NOFS);
  	if (!devices_info)
  		return -ENOMEM;
 
@@ -5067,7 +5068,18 @@ static int __btrfs_alloc_chunk(struct btrfs_trans_handle *trans,
  			max_stripe_size, dev_stripes);
  	if (ret < 0)
  		goto error;
-
+	/*
+	 * If rw devices can't fullfil the request, fallback to missing devices
+	 * as last resort.
+	 */
+	if (ndevs < devs_min) {
+		ret = gather_dev_holes(info, devices_info + ndevs, &ndevs,
+				&fs_devices->missing_list,
+				fs_devices->missing_devices,
+				max_stripe_size, dev_stripes);
+		if (ret < 0)
+			goto error;
+	}
  	/*
  	 * now sort the devices by hole size / available space
  	 */
#12 lore raw

Re: [PATCH 3/3] btrfs: volumes: Allocate degraded chunks if rw devices can't fullfil a chunk

Quoted 8 lines · Anand Jain
Quoted 3 lines · Qu Wenruo

[PROBLEM]
Btrfs degraded mount will fallback to SINGLE profile if there are not enough devices:

 Its better to keep it like this for now until there is a fix for the write hole. Otherwise hitting the write hole bug in case of degraded raid1 will be more prevalent.

Write hole should be a problem for RAID5/6, not the degraded chunk feature itself.

Furthermore, this design will try to avoid allocating chunks using missing devices.
So even for 3 devices RAID5, new chunks will be allocated by using existing devices (2 devices RAID5), so no new write hole is introduced.

Quoted 6 lines

 I proposed a RFC a long time before [1] (also in there, there is a commit id which turned the degraded raid1 profile into single profile (without much write-up on it)).

   [1] [PATCH 0/2] [RFC] btrfs: create degraded-RAID1 chunks

My point for this patchset is:
- Create regular chunk if we have enough devices
- Create degraded chunk only when we have not enough devices

I guess since you didn't get the point of my preparation patches, your patches aren't that good to avoid missing devices.

Quoted 7 lines

 Similarly the patch related to the reappearing missing device [2] falls under the same category. Will push for the integration after the write hole fix.

   [2] [PATCH] btrfs: handle dynamically reappearing missing device (test case 154).

That's another case, and I didn't see how it affects this feature.

Quoted 5 lines

 If you look close enough the original author has quite nicely made sure write hole bug will be very difficultly to hit. These fixes shall make it easy to hit. So its better to work on the write hole first.

If you're talking about RAID5/6, you are talking at the wrong thread. Go implement some write-a-head log for RAID5/6, or mark all degraded RAID5/6 chunks read-only at mount time.

Quoted 14 lines

 I am trying to fix write hole. First attempt has limited success (works fine in two disk raid1 only). Now trying other ways to fix.

Quoted 10 lines
  # mkfs.btrfs -f /dev/test/scratch[12] -m raid1 -d raid1
  # wipefs -fa /dev/test/scratch2
  # mount -o degraded /dev/test/scratch1 /mnt/btrfs
  # fallocate -l 1G /mnt/btrfs/foobar
  # btrfs ins dump-tree -t chunk /dev/test/scratch1
         item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 1674575872) itemoff
15511 itemsize 80
                 length 536870912 owner 2 stripe_len 65536 type DATA
  New data chunk will fallback to SINGLE.
Quoted 7 lines
Quoted 3 lines

If user doesn't balance those SINGLE chunks, even with missing devices replaced, the fs is no longer full RAID1, and a missing device can break the tolerance.

 As its been discussed quite a lot of time before, the current re-silver/recovery approach for degraded-raid1 (with offload to Single) requires balance. Its kind of known.

I'd call such "well-known" behavior BS.

All other raid1 implementation can accept single device RAID1 and resilver itself with more device into a full RAID1 setup.

But for BTRFS you're calling SINGLE profile "well-known"? It's "well-known" because it's not working properly, that's why I'm trying to fix it.

Quoted 112 lines

Thanks, Anand

Quoted 107 lines

[CAUSE]
The cause is pretty simple, when mounted degraded, missing devices can't be used for chunk allocation.
Thus btrfs has to fall back to SINGLE profile.

[ENHANCEMENT]
To avoid such problem, this patch will:
- Make all profiler reducer/updater to consider missing devices as part of num_devices
- Make chunk allocator to fallback to missing_list as last resort

If we have enough rw_devices, then go regular chunk allocation code.

This can avoid allocating degraded chunks.
E.g. for 3 devices RAID1 degraded mount, we will use the 2 existing devices to allocate chunk, avoid degraded chunk.

But if we don't have enough rw_devices, then we check missing devices to allocate degraded chunks.
E.g. for 2 devices RAID1 degraded mount, we have to allocate degraded chunks to keep the RAID1 profile.

---
  fs/btrfs/block-group.c | 10 +++++++---
  fs/btrfs/volumes.c     | 18 +++++++++++++++---

  2 files changed, 22 insertions(+), 6 deletions(-)
fs/btrfs/block-group.c +1 -1
index bf7e3f23bba7..1686fd31679b 100644
@@ -52,11 +52,13 @@ static u64 get_restripe_target(struct
 btrfs_fs_info *fs_info, u64 flags)
    */
   static u64 btrfs_reduce_alloc_profile(struct btrfs_fs_info *fs_info,
 u64 flags)
   {
-    u64 num_devices = fs_info->fs_devices->rw_devices;
+    u64 num_devices;
       u64 target;
       u64 raid_type;
       u64 allowed = 0;

  +    num_devices = fs_info->fs_devices->rw_devices +
+              fs_info->fs_devices->missing_devices;
      /*
       * See if restripe for this chunk_type is in progress, if so try to
       * reduce to the target profile

fs/btrfs/block-group.c +2 -0
@@ -1986,7 +1988,8 @@ static u64 update_block_group_flags(struct
 btrfs_fs_info *fs_info, u64 flags)
       if (stripped)
           return extended_to_chunk(stripped);
   -    num_devices = fs_info->fs_devices->rw_devices;
+    num_devices = fs_info->fs_devices->rw_devices +
+              fs_info->fs_devices->missing_devices;

        stripped = BTRFS_BLOCK_GROUP_RAID0 | BTRFS_BLOCK_GROUP_RAID56_MASK | BTRFS_BLOCK_GROUP_RAID1_MASK | BTRFS_BLOCK_GROUP_RAID10;

fs/btrfs/block-group.c +0 -0
@@ -2981,7 +2984,8 @@ static u64 get_profile_num_devs(struct
 btrfs_fs_info *fs_info, u64 type)

        num_dev = btrfs_raid_array[btrfs_bg_flags_to_raid_index(type)].devs_max; if (!num_dev)
-        num_dev = fs_info->fs_devices->rw_devices; +        num_dev = fs_info->fs_devices->rw_devices + +              fs_info->fs_devices->missing_devices; return num_dev;
  }

fs/btrfs/volumes.c +3 -1
index a462d8de5d2a..4dee1974ceb7 100644
@@ -5052,8 +5052,9 @@ static int __btrfs_alloc_chunk(struct
 btrfs_trans_handle *trans,
       max_chunk_size = min(div_factor(fs_devices->total_rw_bytes, 1),
                    max_chunk_size);
   -    devices_info = kcalloc(fs_devices->rw_devices,
 sizeof(*devices_info),
-                   GFP_NOFS);
+    devices_info = kcalloc(fs_devices->rw_devices +
+                   fs_devices->missing_devices,
+                   sizeof(*devices_info), GFP_NOFS);
       if (!devices_info)

          return -ENOMEM;
  @@ -5067,7 +5068,18 @@ static int __btrfs_alloc_chunk(struct btrfs_trans_handle *trans,
              max_stripe_size, dev_stripes);
      if (ret < 0)
          goto error;
-
+    /*
+     * If rw devices can't fullfil the request, fallback to missing devices
+     * as last resort.
+     */
+    if (ndevs < devs_min) {
+        ret = gather_dev_holes(info, devices_info + ndevs, &ndevs, +                &fs_devices->missing_list,
+                fs_devices->missing_devices,
+                max_stripe_size, dev_stripes);
+        if (ret < 0)
+            goto error;
+    }
      /*
       * now sort the devices by hole size / available space */

#13 lore raw

Re: [PATCH 3/3] btrfs: volumes: Allocate degraded chunks if rw devices can't fullfil a chunk

Quoted 10 lines · Qu Wenruo
Quoted 8 lines · Anand Jain
Quoted 3 lines · Qu Wenruo

[PROBLEM]
Btrfs degraded mount will fallback to SINGLE profile if there are not enough devices:

 Its better to keep it like this for now until there is a fix for the write hole. Otherwise hitting the write hole bug in case of degraded raid1 will be more prevalent.

Quoted 7 lines

Write hole should be a problem for RAID5/6, not the degraded chunk feature itself.

Furthermore, this design will try to avoid allocating chunks using missing devices.
So even for 3 devices RAID5, new chunks will be allocated by using existing devices (2 devices RAID5), so no new write hole is introduced.

That this would allow a 2 device raid5 (from expected 3) is similar to the reduced chunks, but now hidden because we don't have a detailed report for stripes on devices. And rebalance would be needed to make sure that's the filesystem is again 3 devices (and 1 device lost tolerant).

This is different to the 1 device missing for raid1, where scrub can fix that (expected), but the balance is IMHO not.

I'd suggest to allow allocation from missing devices only from the profiles with redundancy. For now.

#14 lore raw

Re: [PATCH 3/3] btrfs: volumes: Allocate degraded chunks if rw devices can't fullfil a chunk

Quoted 25 lines · David Sterba
Quoted 10 lines · Qu Wenruo
Quoted 8 lines · Anand Jain
Quoted 3 lines · Qu Wenruo

[PROBLEM]
Btrfs degraded mount will fallback to SINGLE profile if there are not enough devices:

 Its better to keep it like this for now until there is a fix for the write hole. Otherwise hitting the write hole bug in case of degraded raid1 will be more prevalent.

Quoted 7 lines

Write hole should be a problem for RAID5/6, not the degraded chunk feature itself.

Furthermore, this design will try to avoid allocating chunks using missing devices.
So even for 3 devices RAID5, new chunks will be allocated by using existing devices (2 devices RAID5), so no new write hole is introduced.

That this would allow a 2 device raid5 (from expected 3) is similar to the reduced chunks, but now hidden because we don't have a detailed report for stripes on devices. And rebalance would be needed to make sure that's the filesystem is again 3 devices (and 1 device lost tolerant).

Quoted 5 lines

This is different to the 1 device missing for raid1, where scrub can fix that (expected), but the balance is IMHO not.

I'd suggest to allow allocation from missing devices only from the profiles with redundancy. For now.

But RAID5 itself supports 2 devices, right?
And even 2 devices RAID5 can still tolerant 1 missing device.

The tolerance hasn't changed in that case, just unbalanced disk usage then.

Thanks,
Qu

#15 lore raw

Re: [PATCH 3/3] btrfs: volumes: Allocate degraded chunks if rw devices can't fullfil a chunk

Quoted 29 lines · Qu Wenruo
Quoted 28 lines · David Sterba
Quoted 10 lines · Qu Wenruo
Quoted 8 lines · Anand Jain
Quoted 3 lines · Qu Wenruo

[PROBLEM]
Btrfs degraded mount will fallback to SINGLE profile if there are not enough devices:

 Its better to keep it like this for now until there is a fix for the write hole. Otherwise hitting the write hole bug in case of degraded raid1 will be more prevalent.

Quoted 7 lines

Write hole should be a problem for RAID5/6, not the degraded chunk feature itself.

Furthermore, this design will try to avoid allocating chunks using missing devices.
So even for 3 devices RAID5, new chunks will be allocated by using existing devices (2 devices RAID5), so no new write hole is introduced.

That this would allow a 2 device raid5 (from expected 3) is similar to the reduced chunks, but now hidden because we don't have a detailed report for stripes on devices. And rebalance would be needed to make sure that's the filesystem is again 3 devices (and 1 device lost tolerant).

This is different to the 1 device missing for raid1, where scrub can fix that (expected), but the balance is IMHO not.

Quoted 7 lines
Quoted 2 lines

I'd suggest to allow allocation from missing devices only from the profiles with redundancy. For now.

But RAID5 itself supports 2 devices, right?
And even 2 devices RAID5 can still tolerant 1 missing device.

The tolerance hasn't changed in that case, just unbalanced disk usage then.

Ah right, the constraints are still fine. That the usage is unbalanced is something I'd still consider a problem because it's silently changing the layout from the one that was set by user.

As there are two conflicting ways to continue from the missing device state:

- try to use remaining devices to allow writes but change the layout
- don't allow writes, let user/admin sort it out

I'd rather have more time to understand the implications and try to experiment with that.

#16 lore raw

Re: [PATCH 3/3] btrfs: volumes: Allocate degraded chunks if rw devices can't fullfil a chunk

Quoted 33 lines · David Sterba
Quoted 32 lines · Qu Wenruo
Quoted 30 lines · David Sterba
Quoted 10 lines · Qu Wenruo
Quoted 8 lines · Anand Jain
Quoted 3 lines · Qu Wenruo

[PROBLEM]
Btrfs degraded mount will fallback to SINGLE profile if there are not enough devices:

 Its better to keep it like this for now until there is a fix for the write hole. Otherwise hitting the write hole bug in case of degraded raid1 will be more prevalent.

Quoted 7 lines

Write hole should be a problem for RAID5/6, not the degraded chunk feature itself.

Furthermore, this design will try to avoid allocating chunks using missing devices.
So even for 3 devices RAID5, new chunks will be allocated by using existing devices (2 devices RAID5), so no new write hole is introduced.

That this would allow a 2 device raid5 (from expected 3) is similar to the reduced chunks, but now hidden because we don't have a detailed report for stripes on devices. And rebalance would be needed to make sure that's the filesystem is again 3 devices (and 1 device lost tolerant).

This is different to the 1 device missing for raid1, where scrub can fix that (expected), but the balance is IMHO not.

I'd suggest to allow allocation from missing devices only from the profiles with redundancy. For now.

Quoted 8 lines
Quoted 4 lines

But RAID5 itself supports 2 devices, right?
And even 2 devices RAID5 can still tolerant 1 missing device.

The tolerance hasn't changed in that case, just unbalanced disk usage then.

Ah right, the constraints are still fine. That the usage is unbalanced is something I'd still consider a problem because it's silently changing the layout from the one that was set by user.

Then it's the trade off between:
- Completely the same layout
- Just tolerance

I'd say, even without missing device, btrfs by its nature, it can't ensure all chunks are allocated using the same device layout.

E.g. 4 disk raid5, 1T + 2T + 2T + 2T.
There will be a point where btrfs can only go 3 disks RAID5 other than 4.

Quoted 8 lines

As there are two conflicting ways to continue from the missing device state:

- try to use remaining devices to allow writes but change the layout
- don't allow writes, let user/admin sort it out

I'd rather have more time to understand the implications and try to experiment with that.

Sure, no problem.

Thanks,
Qu

author Reviewed-by
#17 lore raw

Re: [PATCH 3/3] btrfs: volumes: Allocate degraded chunks if rw devices can't fullfil a chunk

In reply to David Sterba
Quoted 42 lines · David Sterba
Quoted 29 lines · Qu Wenruo
Quoted 28 lines · David Sterba
Quoted 10 lines · Qu Wenruo
Quoted 8 lines · Anand Jain
Quoted 3 lines · Qu Wenruo

[PROBLEM]
Btrfs degraded mount will fallback to SINGLE profile if there are not enough devices:

 Its better to keep it like this for now until there is a fix for the write hole. Otherwise hitting the write hole bug in case of degraded raid1 will be more prevalent.

Quoted 7 lines

Write hole should be a problem for RAID5/6, not the degraded chunk feature itself.

Furthermore, this design will try to avoid allocating chunks using missing devices.
So even for 3 devices RAID5, new chunks will be allocated by using existing devices (2 devices RAID5), so no new write hole is introduced.

That this would allow a 2 device raid5 (from expected 3) is similar to the reduced chunks, but now hidden because we don't have a detailed report for stripes on devices. And rebalance would be needed to make sure that's the filesystem is again 3 devices (and 1 device lost tolerant).

This is different to the 1 device missing for raid1, where scrub can fix that (expected), but the balance is IMHO not.

Quoted 7 lines
Quoted 2 lines

I'd suggest to allow allocation from missing devices only from the profiles with redundancy. For now.

But RAID5 itself supports 2 devices, right?
And even 2 devices RAID5 can still tolerant 1 missing device.

The tolerance hasn't changed in that case, just unbalanced disk usage then.

Ah right, the constraints are still fine. That the usage is unbalanced is something I'd still consider a problem because it's silently changing the layout from the one that was set by user.

Quoted 8 lines

As there are two conflicting ways to continue from the missing device state:

- try to use remaining devices to allow writes but change the layout
- don't allow writes, let user/admin sort it out

I'd rather have more time to understand the implications and try to experiment with that.

Ah, makes sense.

So no need for a new version.

Thanks,
Qu

#18 lore raw

Re: [PATCH 3/3] btrfs: volumes: Allocate degraded chunks if rw devices can't fullfil a chunk

Quoted 53 lines · Qu WenRuo
Quoted 42 lines · David Sterba
Quoted 29 lines · Qu Wenruo
Quoted 28 lines · David Sterba
Quoted 10 lines · Qu Wenruo
Quoted 8 lines · Anand Jain
Quoted 3 lines · Qu Wenruo

[PROBLEM]
Btrfs degraded mount will fallback to SINGLE profile if there are not enough devices:

 Its better to keep it like this for now until there is a fix for the write hole. Otherwise hitting the write hole bug in case of degraded raid1 will be more prevalent.

Quoted 7 lines

Write hole should be a problem for RAID5/6, not the degraded chunk feature itself.

Furthermore, this design will try to avoid allocating chunks using missing devices.
So even for 3 devices RAID5, new chunks will be allocated by using existing devices (2 devices RAID5), so no new write hole is introduced.

That this would allow a 2 device raid5 (from expected 3) is similar to the reduced chunks, but now hidden because we don't have a detailed report for stripes on devices. And rebalance would be needed to make sure that's the filesystem is again 3 devices (and 1 device lost tolerant).

This is different to the 1 device missing for raid1, where scrub can fix that (expected), but the balance is IMHO not.

Quoted 7 lines
Quoted 2 lines

I'd suggest to allow allocation from missing devices only from the profiles with redundancy. For now.

But RAID5 itself supports 2 devices, right?
And even 2 devices RAID5 can still tolerant 1 missing device.

The tolerance hasn't changed in that case, just unbalanced disk usage then.

Ah right, the constraints are still fine. That the usage is unbalanced is something I'd still consider a problem because it's silently changing the layout from the one that was set by user.

Quoted 8 lines

As there are two conflicting ways to continue from the missing device state:

- try to use remaining devices to allow writes but change the layout
- don't allow writes, let user/admin sort it out

I'd rather have more time to understand the implications and try to experiment with that.

Quoted 9 lines

Ah, makes sense.

So no need for a new version.

Thanks,
Qu

Facepalm, that's for another thread....

Reviewing patch from myself, WTF....

#19 lore raw
In reply to Qu Wenruo
Quoted 20 lines · Qu Wenruo

This patchset will make btrfs degraded mount more intelligent and provide more consistent profile keeping function.

One of the most problematic aspect of degraded mount is, btrfs may create unwanted profiles.

 # mkfs.btrfs -f /dev/test/scratch[12] -m raid1 -d raid1
 # wipefs -fa /dev/test/scratch2
 # mount -o degraded /dev/test/scratch1 /mnt/btrfs
 # fallocate -l 1G /mnt/btrfs/foobar
 # btrfs ins dump-tree -t chunk /dev/test/scratch1
        item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 1674575872) itemoff 15511 itemsize 80
                length 536870912 owner 2 stripe_len 65536 type DATA
 New data chunk will fallback to SINGLE or DUP.

The cause is pretty simple, when mounted degraded, missing devices can't be used for chunk allocation.
Thus btrfs has to fall back to SINGLE profile.

Quoted 6 lines

This patchset will make btrfs to consider missing devices as last resort if current rw devices can't fulfil the profile request.

This should provide a good balance between considering all missing device as RW and completely ruling out missing devices (current mainline behavior).

Thanks. This is going to change the behaviour with a missing device, so the question is if we should make this configurable first and then switch the default.

How does this work with scrub? Eg. if there are 2 devices in RAID1, one goes missing and then scrub is started. It makes no sense to try to repair the missing blocks, but given the logic in the patches all the data will be rewritten, right?

#20 lore raw
Quoted 24 lines · David Sterba
Quoted 23 lines · Qu Wenruo

This patchset will make btrfs degraded mount more intelligent and provide more consistent profile keeping function.

One of the most problematic aspect of degraded mount is, btrfs may create unwanted profiles.

 # mkfs.btrfs -f /dev/test/scratch[12] -m raid1 -d raid1
 # wipefs -fa /dev/test/scratch2
 # mount -o degraded /dev/test/scratch1 /mnt/btrfs
 # fallocate -l 1G /mnt/btrfs/foobar
 # btrfs ins dump-tree -t chunk /dev/test/scratch1
        item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 1674575872) itemoff 15511 itemsize 80
                length 536870912 owner 2 stripe_len 65536 type DATA
 New data chunk will fallback to SINGLE or DUP.

The cause is pretty simple, when mounted degraded, missing devices can't be used for chunk allocation.
Thus btrfs has to fall back to SINGLE profile.

This patchset will make btrfs to consider missing devices as last resort if current rw devices can't fulfil the profile request.

Quoted 7 lines
Quoted 3 lines

This should provide a good balance between considering all missing device as RW and completely ruling out missing devices (current mainline behavior).

Thanks. This is going to change the behaviour with a missing device, so the question is if we should make this configurable first and then switch the default.

Configurable then switch makes sense for most cases, but for this degraded chunk case, IIRC the new behavior is superior in all cases.

For 2 devices RAID1 with one missing device (the main concern), old behavior will create SINGLE/DUP chunk, which has no tolerance for extra missing devices.

The new behavior will create degraded RAID1, which still lacks tolerance for extra missing devices.

The difference is, for degraded chunk, if we have the device back, and do proper scrub, then we're completely back to proper RAID1. No need to do extra balance/convert, only scrub is needed.

So the new behavior is kinda of a super set of old behavior, using the new behavior by default should not cause extra concern.

Quoted 5 lines

How does this work with scrub? Eg. if there are 2 devices in RAID1, one goes missing and then scrub is started. It makes no sense to try to repair the missing blocks, but given the logic in the patches all the data will be rewritten, right?

Scrub is unchanged at all.

Missing device will not go through scrub at all, as scrub is per-device based, missing device will be ruled out at very beginning of scrub.

Thanks,
Qu

Quoted 1 line
#21 lore raw
Quoted 42 lines · Qu Wenruo
Quoted 24 lines · David Sterba
Quoted 23 lines · Qu Wenruo

This patchset will make btrfs degraded mount more intelligent and provide more consistent profile keeping function.

One of the most problematic aspect of degraded mount is, btrfs may create unwanted profiles.

  # mkfs.btrfs -f /dev/test/scratch[12] -m raid1 -d raid1
  # wipefs -fa /dev/test/scratch2
  # mount -o degraded /dev/test/scratch1 /mnt/btrfs
  # fallocate -l 1G /mnt/btrfs/foobar
  # btrfs ins dump-tree -t chunk /dev/test/scratch1
         item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 1674575872) itemoff 15511 itemsize 80
                 length 536870912 owner 2 stripe_len 65536 type DATA
  New data chunk will fallback to SINGLE or DUP.

The cause is pretty simple, when mounted degraded, missing devices can't be used for chunk allocation.
Thus btrfs has to fall back to SINGLE profile.

This patchset will make btrfs to consider missing devices as last resort if current rw devices can't fulfil the profile request.

Quoted 6 lines
Quoted 3 lines

This should provide a good balance between considering all missing device as RW and completely ruling out missing devices (current mainline behavior).

Thanks. This is going to change the behaviour with a missing device, so the question is if we should make this configurable first and then switch the default.

Configurable then switch makes sense for most cases, but for this degraded chunk case, IIRC the new behavior is superior in all cases.

For 2 devices RAID1 with one missing device (the main concern), old behavior will create SINGLE/DUP chunk, which has no tolerance for extra missing devices.

The new behavior will create degraded RAID1, which still lacks tolerance for extra missing devices.

Quoted 6 lines

The difference is, for degraded chunk, if we have the device back, and do proper scrub, then we're completely back to proper RAID1. No need to do extra balance/convert, only scrub is needed.

So the new behavior is kinda of a super set of old behavior, using the new behavior by default should not cause extra concern.

I think most users will see this as a bug fix, as the current behavior of creating

SINGLE chunks is very annoying and can cause confusion as it is NOT an

expected behavior for a classic (mdadm or hardware) degraded RAID array.

-Alberto

#22 lore raw
Quoted 20 lines · Alberto Bursi
Quoted 13 lines
Quoted 3 lines

Thanks. This is going to change the behaviour with a missing device, so the question is if we should make this configurable first and then switch the default.

Configurable then switch makes sense for most cases, but for this degraded chunk case, IIRC the new behavior is superior in all cases.

For 2 devices RAID1 with one missing device (the main concern), old behavior will create SINGLE/DUP chunk, which has no tolerance for extra missing devices.

The new behavior will create degraded RAID1, which still lacks tolerance for extra missing devices.

Quoted 6 lines

The difference is, for degraded chunk, if we have the device back, and do proper scrub, then we're completely back to proper RAID1. No need to do extra balance/convert, only scrub is needed.

So the new behavior is kinda of a super set of old behavior, using the new behavior by default should not cause extra concern.

Quoted 6 lines

I think most users will see this as a bug fix, as the current behavior of creating

SINGLE chunks is very annoying and can cause confusion as it is NOT an

expected behavior for a classic (mdadm or hardware) degraded RAID array.

Thanks for the feedback, I agree with that. It's good to have a confirmation from somebody outside of developer group.

#23 lore raw
In reply to Qu Wenruo
Quoted 42 lines · Qu Wenruo
Quoted 24 lines · David Sterba
Quoted 23 lines · Qu Wenruo

This patchset will make btrfs degraded mount more intelligent and provide more consistent profile keeping function.

One of the most problematic aspect of degraded mount is, btrfs may create unwanted profiles.

 # mkfs.btrfs -f /dev/test/scratch[12] -m raid1 -d raid1
 # wipefs -fa /dev/test/scratch2
 # mount -o degraded /dev/test/scratch1 /mnt/btrfs
 # fallocate -l 1G /mnt/btrfs/foobar
 # btrfs ins dump-tree -t chunk /dev/test/scratch1
        item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 1674575872) itemoff 15511 itemsize 80
                length 536870912 owner 2 stripe_len 65536 type DATA
 New data chunk will fallback to SINGLE or DUP.

The cause is pretty simple, when mounted degraded, missing devices can't be used for chunk allocation.
Thus btrfs has to fall back to SINGLE profile.

This patchset will make btrfs to consider missing devices as last resort if current rw devices can't fulfil the profile request.

Quoted 7 lines
Quoted 3 lines

This should provide a good balance between considering all missing device as RW and completely ruling out missing devices (current mainline behavior).

Thanks. This is going to change the behaviour with a missing device, so the question is if we should make this configurable first and then switch the default.

Configurable then switch makes sense for most cases, but for this degraded chunk case, IIRC the new behavior is superior in all cases.

For 2 devices RAID1 with one missing device (the main concern), old behavior will create SINGLE/DUP chunk, which has no tolerance for extra missing devices.

Quoted 6 lines

The new behavior will create degraded RAID1, which still lacks tolerance for extra missing devices.

The difference is, for degraded chunk, if we have the device back, and do proper scrub, then we're completely back to proper RAID1. No need to do extra balance/convert, only scrub is needed.

I think you meant to say "replace" instead of "scrub" above.

Quoted 2 lines

So the new behavior is kinda of a super set of old behavior, using the new behavior by default should not cause extra concern.

It sounds OK to me, provided that the missing device is going away permanently, and a new device replaces it.

If the missing device comes back, we end up relying on scrub and 32-bit CRCs to figure out which disk has correct data, and it will be wrong 1/2^32 of the time. For nodatasum files there are no CRCs so the data will be wrong much more often. This patch doesn't change that, but maybe another patch should.

Quoted 14 lines
Quoted 4 lines

How does this work with scrub? Eg. if there are 2 devices in RAID1, one goes missing and then scrub is started. It makes no sense to try to repair the missing blocks, but given the logic in the patches all the data will be rewritten, right?

Scrub is unchanged at all.

Missing device will not go through scrub at all, as scrub is per-device based, missing device will be ruled out at very beginning of scrub.

Thanks,
Qu

Quoted 1 line
#24 lore raw
Quoted 43 lines · Zygo Blaxell
Quoted 42 lines · Qu Wenruo
Quoted 24 lines · David Sterba
Quoted 23 lines · Qu Wenruo

This patchset will make btrfs degraded mount more intelligent and provide more consistent profile keeping function.

One of the most problematic aspect of degraded mount is, btrfs may create unwanted profiles.

 # mkfs.btrfs -f /dev/test/scratch[12] -m raid1 -d raid1
 # wipefs -fa /dev/test/scratch2
 # mount -o degraded /dev/test/scratch1 /mnt/btrfs
 # fallocate -l 1G /mnt/btrfs/foobar
 # btrfs ins dump-tree -t chunk /dev/test/scratch1
        item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 1674575872) itemoff 15511 itemsize 80
                length 536870912 owner 2 stripe_len 65536 type DATA
 New data chunk will fallback to SINGLE or DUP.

The cause is pretty simple, when mounted degraded, missing devices can't be used for chunk allocation.
Thus btrfs has to fall back to SINGLE profile.

This patchset will make btrfs to consider missing devices as last resort if current rw devices can't fulfil the profile request.

Quoted 7 lines
Quoted 3 lines

This should provide a good balance between considering all missing device as RW and completely ruling out missing devices (current mainline behavior).

Thanks. This is going to change the behaviour with a missing device, so the question is if we should make this configurable first and then switch the default.

Configurable then switch makes sense for most cases, but for this degraded chunk case, IIRC the new behavior is superior in all cases.

For 2 devices RAID1 with one missing device (the main concern), old behavior will create SINGLE/DUP chunk, which has no tolerance for extra missing devices.

Quoted 8 lines
Quoted 6 lines

The new behavior will create degraded RAID1, which still lacks tolerance for extra missing devices.

The difference is, for degraded chunk, if we have the device back, and do proper scrub, then we're completely back to proper RAID1. No need to do extra balance/convert, only scrub is needed.

I think you meant to say "replace" instead of "scrub" above.

"scrub" for missing-then-back case.

As at the time of write, I didn't even take the replace case into consideration...

Quoted 4 lines
Quoted 3 lines

So the new behavior is kinda of a super set of old behavior, using the new behavior by default should not cause extra concern.

Quoted 8 lines

It sounds OK to me, provided that the missing device is going away permanently, and a new device replaces it.

If the missing device comes back, we end up relying on scrub and 32-bit CRCs to figure out which disk has correct data, and it will be wrong 1/2^32 of the time. For nodatasum files there are no CRCs so the data will be wrong much more often. This patch doesn't change that, but maybe another patch should.

Yep, the patchset won't change it.

But this also remind me, so far we are all talking about "degraded" mount option.
Under most case, user is only using "degraded" when they completely understands that device is missing, not using that option as a daily option.

So that shouldn't be a big problem so far.

Thanks,
Qu

Quoted 18 lines
Quoted 15 lines
Quoted 5 lines

How does this work with scrub? Eg. if there are 2 devices in RAID1, one goes missing and then scrub is started. It makes no sense to try to repair the missing blocks, but given the logic in the patches all the data will be rewritten, right?

Scrub is unchanged at all.

Missing device will not go through scrub at all, as scrub is per-device based, missing device will be ruled out at very beginning of scrub.

Thanks,
Qu

Quoted 1 line
#25 lore raw
Quoted 53 lines · Qu Wenruo
Quoted 50 lines · Zygo Blaxell
Quoted 48 lines · Qu Wenruo
Quoted 24 lines · David Sterba
Quoted 23 lines · Qu Wenruo

This patchset will make btrfs degraded mount more intelligent and provide more consistent profile keeping function.

One of the most problematic aspect of degraded mount is, btrfs may create unwanted profiles.

 # mkfs.btrfs -f /dev/test/scratch[12] -m raid1 -d raid1
 # wipefs -fa /dev/test/scratch2
 # mount -o degraded /dev/test/scratch1 /mnt/btrfs
 # fallocate -l 1G /mnt/btrfs/foobar
 # btrfs ins dump-tree -t chunk /dev/test/scratch1
        item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 1674575872) itemoff 15511 itemsize 80
                length 536870912 owner 2 stripe_len 65536 type DATA
 New data chunk will fallback to SINGLE or DUP.

The cause is pretty simple, when mounted degraded, missing devices can't be used for chunk allocation.
Thus btrfs has to fall back to SINGLE profile.

This patchset will make btrfs to consider missing devices as last resort if current rw devices can't fulfil the profile request.

Quoted 7 lines
Quoted 3 lines

This should provide a good balance between considering all missing device as RW and completely ruling out missing devices (current mainline behavior).

Thanks. This is going to change the behaviour with a missing device, so the question is if we should make this configurable first and then switch the default.

Configurable then switch makes sense for most cases, but for this degraded chunk case, IIRC the new behavior is superior in all cases.

For 2 devices RAID1 with one missing device (the main concern), old behavior will create SINGLE/DUP chunk, which has no tolerance for extra missing devices.

The new behavior will create degraded RAID1, which still lacks tolerance for extra missing devices.

The difference is, for degraded chunk, if we have the device back, and do proper scrub, then we're completely back to proper RAID1. No need to do extra balance/convert, only scrub is needed.

Quoted 6 lines
Quoted 1 line

I think you meant to say "replace" instead of "scrub" above.

"scrub" for missing-then-back case.

As at the time of write, I didn't even take the replace case into consideration...

Using scrub for that purpose has some pretty bad failure modes, e.g.

	https://www.mail-archive.com/[email protected]/msg78254.html

mdadm raid1 has many problems, but this is not one of them. Drives that exit a mdadm raid1 array are marked of date on the remaining drives' superblocks (using sequence numbers and time stamps to avoid colliding sequence numbers from split-brain raid1 cases). If the drive returns, it cannot simply reenter the array, it has to be resynced using online drives as a replacement data source (partial resyncs are possible with mdadm bitmaps, a similar thing could be done with btrfs block groups). mdadm will do all this for you automatically.

Quoted 12 lines
Quoted 11 lines
Quoted 2 lines

So the new behavior is kinda of a super set of old behavior, using the new behavior by default should not cause extra concern.

It sounds OK to me, provided that the missing device is going away permanently, and a new device replaces it.

If the missing device comes back, we end up relying on scrub and 32-bit CRCs to figure out which disk has correct data, and it will be wrong 1/2^32 of the time. For nodatasum files there are no CRCs so the data will be wrong much more often. This patch doesn't change that, but maybe another patch should.

Quoted 6 lines

Yep, the patchset won't change it.

But this also remind me, so far we are all talking about "degraded" mount option.
Under most case, user is only using "degraded" when they completely understands that device is missing, not using that option as a daily option.

That makes certain high-availability setups (e.g. when btrfs is /) difficult in practice. To do it by the book, you'd need to:

	1.  Experience a total disk failure
	2.  Reconfigure bootloader to mount -o degraded
	3.  Shut system down
	4.  Remove old drive, add 
	5.  Boot system
	6.  btrfs replace
	7.  Reconfigure bootloader to mount without degraded

That's hard (or just expensive and risky) to do in headless remote setups, especially if step 1 causes a crash, step 2 gets skipped, and the system has neither a console to reconfigure the bootloader nor will btrfs boot without reconfiguring the bootloader.

(Note: I'm assuming that for whatever reason you can't just hotswap the drive and run replace immediately--or maybe you did hotswap the drive, then there was a crash or power failure and you had to go back to mounting with -o degraded to finish the replace anyway.)

In practice, what happens is more like:

	1.  Bootloader configured to mount -o degraded all the time
	2.  Hard disk has a partial failure, drops off bus or firmware crash
	3.  Reboot (due to crash, power failure, adding replacement drive, etc)
	4.  System comes back up with bad disk present
	5.  Lots of csum failures, 0.0000004% of which are not recovered

in scrub with CRC32C csums, and nodatacow files damaged
6. Repeat from step 2 several times

If users on IRC are representative, most of them do not know that step #2 is a data-losing failure (btrfs does, but btrfs currently doesn't act on the information as long as only one drive has failed). Users also don't know that scrub is not replace (resync in mdadm terms), and will keep using the drive and filesystem until something more visible goes wrong. After several loops they show up on IRC asking for help, but by that point there has been unrecoverable data loss.

Quoted 1 line

So that shouldn't be a big problem so far.

It isn't a big problem if you follow exactly the single correct procedure, and know precisely which tools to use in what order.

This is mostly a usability concern: other RAID systems cope with this kind of event automatically, without requiring all the explicit operator intervention.

Quoted 22 lines

Thanks,
Qu

Quoted 18 lines
Quoted 15 lines
Quoted 5 lines

How does this work with scrub? Eg. if there are 2 devices in RAID1, one goes missing and then scrub is started. It makes no sense to try to repair the missing blocks, but given the logic in the patches all the data will be rewritten, right?

Scrub is unchanged at all.

Missing device will not go through scrub at all, as scrub is per-device based, missing device will be ruled out at very beginning of scrub.

Thanks,
Qu

Quoted 1 line