Skip to contentSkip to conversation

raid1 balancing methods

PATCH Superseded Anand Jain 3 participants 10 messages Sep 27, 2024 – Oct 11, 2024 Btrfs file system

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Anand Jain

author
#0 lore raw

cover raid1 balancing methods

The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the CONFIG_BTRFS_DEBUG config option and are on top of the previously added /sys/fs/btrfs/<UUID>/read_policy interface to configure the desired RAID1 read balancing method.

I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from /sys/devices/<..>/stat for each device. Below is a summary of the results, with each result the average of three iterations.

A typical generic random rw workload:

$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \
  --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \
  --group_reporting --name=iops-test-job --eta-newline=1
|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 29.4MiB/s  | 29.5MiB/s  | 456548 | 447975 |
| rotation| 29.3MiB/s  | 29.3MiB/s  | 450105 | 450055 |
| latency | 21.9MiB/s  | 21.9MiB/s  | 672387 | 0      |
| devid:1 | 22.0MiB/s  | 22.0MiB/s  | 674788 | 0      |

Defragmentation with compression workload:

$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo
$ sync
$ echo 3 > /proc/sys/vm/drop_caches
$ btrfs filesystem defrag -f -c /btrfs/foo
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 21.61s| 3810   | 0      |
| rotation| 11.55s| 1905   | 1905   |
| latency | 20.99s| 0      | 3810   |
| devid:2 | 21.41s| 0      | 3810   |

. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.

Anand Jain (3):
btrfs: introduce RAID1 round-robin read balancing
btrfs: use the path with the lowest latency for RAID1 reads
btrfs: add RAID1 preferred read device feature

 fs/btrfs/sysfs.c   |  94 ++++++++++++++++++++++++++++++-------
 fs/btrfs/volumes.c | 113 +++++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/volumes.h |  14 ++++++
 3 files changed, 205 insertions(+), 16 deletions(-)
Signature · 1 line
2.46.0
author
#1 lore raw

PATCH 1/3 btrfs: introduce RAID1 round-robin read balancing

This feature balances I/O across the striped devices when reading from RAID1 blocks.

---
 fs/btrfs/sysfs.c   |  4 ++++
 fs/btrfs/volumes.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/volumes.h |  7 ++++++
 3 files changed, 64 insertions(+)
fs/btrfs/sysfs.c +4 -0
index 03926ad467c9..18fb35a887c6 100644
@@ -1305,7 +1305,11 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj,
 }
 BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);
 
+#ifdef CONFIG_BTRFS_DEBUG
+static const char * const btrfs_read_policy_name[] = { "pid", "rotation" };
+#else
 static const char * const btrfs_read_policy_name[] = { "pid" };
+#endif
 
 static ssize_t btrfs_read_policy_show(struct kobject *kobj,
 				      struct kobj_attribute *a, char *buf)
fs/btrfs/volumes.c +53 -0

65 lines not on this page · raw message

fs/btrfs/volumes.h +7 -0
index 4481575dd70f..81701217dbb9 100644
@@ -303,6 +303,10 @@ enum btrfs_chunk_allocation_policy {
 enum btrfs_read_policy {
 	/* Use process PID to choose the stripe */
 	BTRFS_READ_POLICY_PID,
+#ifdef CONFIG_BTRFS_DEBUG
+	/* Balancing raid1 reads across all striped devices */
+	BTRFS_READ_POLICY_ROTATION,
+#endif
 	BTRFS_NR_READ_POLICY,
 };
 
@@ -431,6 +435,9 @@ struct btrfs_fs_devices {
 	enum btrfs_read_policy read_policy;
 
 #ifdef CONFIG_BTRFS_DEBUG
+	/* read counter for the filesystem */ 
+	atomic_t total_reads;
+
 	/* Checksum mode - offload it or do it synchronously. */
 	enum btrfs_offload_csum_mode offload_csum_mode;
 #endif
Signature · 1 line
2.46.0
author
#4 lore raw

PATCH 2/3 btrfs: use the path with the lowest latency for RAID1 reads

In reply to Anand Jain

This feature aims to direct the read I/O to the device with the lowest known latency for reading RAID1 blocks.

echo "latency" > /sys/fs/btrfs/<UUID>/read_policy

---
 fs/btrfs/sysfs.c   |  2 +-
 fs/btrfs/volumes.c | 40 ++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/volumes.h |  2 ++
 3 files changed, 43 insertions(+), 1 deletion(-)
fs/btrfs/sysfs.c +1 -1
index 18fb35a887c6..15abf931726c 100644
@@ -1306,7 +1306,7 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj,
 BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);
 
 #ifdef CONFIG_BTRFS_DEBUG
-static const char * const btrfs_read_policy_name[] = { "pid", "rotation" };
+static const char * const btrfs_read_policy_name[] = { "pid", "rotation", "latency" };
 #else
 static const char * const btrfs_read_policy_name[] = { "pid" };
 #endif
fs/btrfs/volumes.c +40 -0
index c130a27386a7..20bc62d85b3b 100644
@@ -12,6 +12,9 @@
 #include <linux/uuid.h>
 #include <linux/list_sort.h>
 #include <linux/namei.h>
+#ifdef CONFIG_BTRFS_DEBUG
+#include <linux/part_stat.h>
+#endif
 #include "misc.h"
 #include "ctree.h"
 #include "disk-io.h"
@@ -5860,6 +5863,39 @@ int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len)
 }
 
 #ifdef CONFIG_BTRFS_DEBUG
+static int btrfs_best_stripe(struct btrfs_fs_info *fs_info,
+			     struct btrfs_chunk_map *map, int first,
+			     int num_stripe)
+{
+	u64 est_wait = 0;
+	int best_stripe = 0;
+	int index;
+
+	for (index = first; index < first + num_stripe; index++) {
+		u64 read_wait;
+		u64 avg_wait = 0;
+		unsigned long read_ios;
+		struct btrfs_device *device = map->stripes[index].dev;
+
+		read_wait = part_stat_read(device->bdev, nsecs[READ]);
+		read_ios = part_stat_read(device->bdev, ios[READ]);
+
+		if (read_wait && read_ios && read_wait >= read_ios)
+			avg_wait = div_u64(read_wait, read_ios);
+		else
+			btrfs_debug_rl(fs_info,
+			"devid: %llu avg_wait ZERO read_wait %llu read_ios %lu",
+				       device->devid, read_wait, read_ios);
+
+		if (est_wait == 0 || est_wait > avg_wait) {
+			est_wait = avg_wait;
+			best_stripe = index;
+		}
+	}
+
+	return best_stripe;
+}
+
 struct stripe_mirror {
 	u64 devid;
 	int map;
@@ -5940,6 +5976,10 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,
 	case BTRFS_READ_POLICY_ROTATION:
 		preferred_mirror = btrfs_read_rotation(map, first, num_stripes);
 		break;
+	case BTRFS_READ_POLICY_LATENCY:
+		preferred_mirror = btrfs_best_stripe(fs_info, map, first,
+								num_stripes);
+		break;
 #endif
 	}
 
fs/btrfs/volumes.h +2 -0
index 81701217dbb9..09920ef76a9b 100644
@@ -306,6 +306,8 @@ enum btrfs_read_policy {
 #ifdef CONFIG_BTRFS_DEBUG
 	/* Balancing raid1 reads across all striped devices */
 	BTRFS_READ_POLICY_ROTATION,
+	/* Use the lowest-latency device dynamically */
+	BTRFS_READ_POLICY_LATENCY,
 #endif
 	BTRFS_NR_READ_POLICY,
 };
Signature · 1 line
2.46.0
author
#7 lore raw

PATCH 3/3 btrfs: add RAID1 preferred read device feature

In reply to Anand Jain

When there's stale data on a mirrored device, this feature lets you choose which device to read from. Mainly used for testing.

echo "devid:2" > /sys/fs/btrfs/<UUID>/read_policy

---
 fs/btrfs/sysfs.c   | 92 +++++++++++++++++++++++++++++++++++++---------
 fs/btrfs/volumes.c | 20 ++++++++++
 fs/btrfs/volumes.h |  5 +++
 3 files changed, 100 insertions(+), 17 deletions(-)
fs/btrfs/sysfs.c +75 -17
index 15abf931726c..e32999ea761d 100644
@@ -1306,7 +1306,7 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj,
 BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);
 
 #ifdef CONFIG_BTRFS_DEBUG
-static const char * const btrfs_read_policy_name[] = { "pid", "rotation", "latency" };
+static const char * const btrfs_read_policy_name[] = { "pid", "rotation", "latency", "devid" };
 #else
 static const char * const btrfs_read_policy_name[] = { "pid" };
 #endif
@@ -1320,14 +1320,22 @@ static ssize_t btrfs_read_policy_show(struct kobject *kobj,
 	int i;
 
 	for (i = 0; i < BTRFS_NR_READ_POLICY; i++) {
-		if (policy == i)
-			ret += sysfs_emit_at(buf, ret, "%s[%s]",
-					 (ret == 0 ? "" : " "),
-					 btrfs_read_policy_name[i]);
-		else
-			ret += sysfs_emit_at(buf, ret, "%s%s",
-					 (ret == 0 ? "" : " "),
-					 btrfs_read_policy_name[i]);
+		if (ret != 0)
+			ret += sysfs_emit_at(buf, ret, " ");
+
+		if (i == policy)
+			ret += sysfs_emit_at(buf, ret, "[");
+
+		ret += sysfs_emit_at(buf, ret, "%s", btrfs_read_policy_name[i]);
+
+#ifdef CONFIG_BTRFS_DEBUG
+		if (i == BTRFS_READ_POLICY_DEVID)
+			ret += sysfs_emit_at(buf, ret, ":%llu",
+							fs_devices->read_devid);
+#endif
+
+		if (i == policy)
+			ret += sysfs_emit_at(buf, ret, "]");
 	}
 
 	ret += sysfs_emit_at(buf, ret, "\n");
@@ -1340,21 +1348,71 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj,
 				       const char *buf, size_t len)
 {
 	struct btrfs_fs_devices *fs_devices = to_fs_devs(kobj);
+	char *value;
+#ifdef CONFIG_BTRFS_DEBUG
+	u64 devid = 0;
+#endif
+	int index = -1;
 	int i;
+	bool changed = false;
+
+	value = strchr(buf, ':');
+	if (value) {
+		*value = '\0';
+		value = value + 1;
+	}
 
 	for (i = 0; i < BTRFS_NR_READ_POLICY; i++) {
 		if (sysfs_streq(buf, btrfs_read_policy_name[i])) {
-			if (i != READ_ONCE(fs_devices->read_policy)) {
-				WRITE_ONCE(fs_devices->read_policy, i);
-				btrfs_info(fs_devices->fs_info,
-					   "read policy set to '%s'",
-					   btrfs_read_policy_name[i]);
-			}
-			return len;
+			index = i;
+			break;
+		}
+	}
+
+	if (index == -1)
+		return -EINVAL;
+
+#ifdef CONFIG_BTRFS_DEBUG
+	/* Extract values from input in devid:value format */
+	if (index == BTRFS_READ_POLICY_DEVID) {
+		BTRFS_DEV_LOOKUP_ARGS(args);
+
+		if (value == NULL || kstrtou64(value, 10, &devid))
+			return -EINVAL;
+
+		args.devid = devid;
+		if (btrfs_find_device(fs_devices, &args) == NULL)
+			return -EINVAL;
+
+		if (READ_ONCE(fs_devices->read_devid) != devid) {
+			WRITE_ONCE(fs_devices->read_devid, devid);
+			changed = true;
 		}
 	}
+#endif
+
+	if (index != READ_ONCE(fs_devices->read_policy)) {
+		WRITE_ONCE(fs_devices->read_policy, index);
+		changed = true;
+	}
+
+	if (changed) {
+#ifdef CONFIG_BTRFS_DEBUG
+		if (devid)
+			btrfs_info(fs_devices->fs_info,
+				   "read policy set to '%s:%llu'",
+				   btrfs_read_policy_name[index], devid);
+		else
+			btrfs_info(fs_devices->fs_info,
+				   "read policy set to '%s'",
+				   btrfs_read_policy_name[index]);
+#else
+		btrfs_info(fs_devices->fs_info, "read policy set to '%s'",
+			   btrfs_read_policy_name[index]);
+#endif
+	}
 
-	return -EINVAL;
+	return len;
 }
 BTRFS_ATTR_RW(, read_policy, btrfs_read_policy_show, btrfs_read_policy_store);
 
fs/btrfs/volumes.c +20 -0
index 20bc62d85b3b..c49ca48e7b2e 100644
@@ -5863,6 +5863,23 @@ int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len)
 }
 
 #ifdef CONFIG_BTRFS_DEBUG
+static int btrfs_read_preferred(struct btrfs_chunk_map *map, int first,
+				int num_stripe)
+{
+	int last = first + num_stripe;
+	int stripe_index;
+
+	for (stripe_index = first; stripe_index < last; stripe_index++) {
+		struct btrfs_device *device = map->stripes[stripe_index].dev;
+
+		if (device->devid == READ_ONCE(device->fs_devices->read_devid))
+			return stripe_index;
+	}
+
+	/* If no read-preferred device, use first stripe */
+	return first;
+}
+
 static int btrfs_best_stripe(struct btrfs_fs_info *fs_info,
 			     struct btrfs_chunk_map *map, int first,
 			     int num_stripe)
@@ -5980,6 +5997,9 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,
 		preferred_mirror = btrfs_best_stripe(fs_info, map, first,
 								num_stripes);
 		break;
+	case BTRFS_READ_POLICY_DEVID:
+		preferred_mirror = btrfs_read_preferred(map, first, num_stripes);
+		break;
 #endif
 	}
 
fs/btrfs/volumes.h +5 -0
index 09920ef76a9b..9850edaafe8c 100644
@@ -308,6 +308,8 @@ enum btrfs_read_policy {
 	BTRFS_READ_POLICY_ROTATION,
 	/* Use the lowest-latency device dynamically */
 	BTRFS_READ_POLICY_LATENCY,
+	/* Read from the specific device */
+	BTRFS_READ_POLICY_DEVID,
 #endif
 	BTRFS_NR_READ_POLICY,
 };
@@ -440,6 +442,9 @@ struct btrfs_fs_devices {
 	/* read counter for the filesystem */ 
 	atomic_t total_reads;
 
+	/* Device to be used for reading in case of RAID1 */
+	u64 read_devid;
+
 	/* Checksum mode - offload it or do it synchronously. */
 	enum btrfs_offload_csum_mode offload_csum_mode;
 #endif
Signature · 1 line
2.46.0
#2 lore raw

Re: [PATCH 1/3] btrfs: introduce RAID1 round-robin read balancing

In reply to Anand Jain
Quoted 31 lines · Anand Jain

This feature balances I/O across the striped devices when reading from RAID1 blocks.

---
  fs/btrfs/sysfs.c   |  4 ++++
  fs/btrfs/volumes.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++
  fs/btrfs/volumes.h |  7 ++++++
  3 files changed, 64 insertions(+)
fs/btrfs/sysfs.c +4 -0
index 03926ad467c9..18fb35a887c6 100644
@@ -1305,7 +1305,11 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj,
  }
  BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);
 
+#ifdef CONFIG_BTRFS_DEBUG
+static const char * const btrfs_read_policy_name[] = { "pid", "rotation" };
+#else
  static const char * const btrfs_read_policy_name[] = { "pid" };
+#endif
 
  static ssize_t btrfs_read_policy_show(struct kobject *kobj,
  				      struct kobj_attribute *a, char *buf)
fs/btrfs/volumes.c +0 -0
index 995b0647f538..c130a27386a7 100644

fs/btrfs/volumes.c int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len) · line 5862

Quoted 4 lines
fs/btrfs/volumes.c +1 -0
@@ -5859,6 +5859,54 @@ int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len)
  	return ret;
  }
 
+#ifdef CONFIG_BTRFS_DEBUG

It would be much better to utilize CONFIG_BTRFS_EXPERIMENTAL. CONFIG_BTRFS_DEBUG is now for pure debug purposes.

Thanks,
Qu

Quoted 87 lines
fs/btrfs/volumes.c +52 -0
+struct stripe_mirror {
+	u64 devid;
+	int map;
+};
+
+static int btrfs_cmp_devid(const void *a, const void *b)
+{
+	struct stripe_mirror *s1 = (struct stripe_mirror *)a;
+	struct stripe_mirror *s2 = (struct stripe_mirror *)b;
+
+	if (s1->devid < s2->devid)
+		return -1;
+	if (s1->devid > s2->devid)
+		return 1;
+	return 0;
+}
+
+static int btrfs_read_rotation(struct btrfs_chunk_map *map, int first,
+			       int num_stripe)
+{
+	struct stripe_mirror stripes[4] = {0}; //4: for testing, works for now.
+	struct btrfs_fs_devices *fs_devices;
+	u64 devid;
+	int index, j, cnt;
+	int next_stripe;
+
+	index = 0;
+	for (j = first; j < first + num_stripe; j++) {
+		devid = map->stripes[j].dev->devid;
+
+		stripes[index].devid = devid;
+		stripes[index].map = j;
+
+		index++;
+	}
+
+	sort(stripes, num_stripe, sizeof(struct stripe_mirror),
+	     btrfs_cmp_devid, NULL);
+
+	fs_devices = map->stripes[first].dev->fs_devices;
+	cnt = atomic_inc_return(&fs_devices->total_reads);
+	next_stripe = stripes[cnt % num_stripe].map;
+
+	return next_stripe;
+}
+#endif
+
  static int find_live_mirror(struct btrfs_fs_info *fs_info,
  			    struct btrfs_chunk_map *map, int first,
  			    int dev_replace_is_ongoing)
@@ -5888,6 +5936,11 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,
  	case BTRFS_READ_POLICY_PID:
  		preferred_mirror = first + (current->pid % num_stripes);
  		break;
+#ifdef CONFIG_BTRFS_DEBUG
+	case BTRFS_READ_POLICY_ROTATION:
+		preferred_mirror = btrfs_read_rotation(map, first, num_stripes);
+		break;
+#endif
  	}
 
  	if (dev_replace_is_ongoing &&
fs/btrfs/volumes.h +7 -0
index 4481575dd70f..81701217dbb9 100644
@@ -303,6 +303,10 @@ enum btrfs_chunk_allocation_policy {
  enum btrfs_read_policy {
  	/* Use process PID to choose the stripe */
  	BTRFS_READ_POLICY_PID,
+#ifdef CONFIG_BTRFS_DEBUG
+	/* Balancing raid1 reads across all striped devices */
+	BTRFS_READ_POLICY_ROTATION,
+#endif
  	BTRFS_NR_READ_POLICY,
  };
 
@@ -431,6 +435,9 @@ struct btrfs_fs_devices {
  	enum btrfs_read_policy read_policy;
 
  #ifdef CONFIG_BTRFS_DEBUG
+	/* read counter for the filesystem */
+	atomic_t total_reads;
+
  	/* Checksum mode - offload it or do it synchronously. */
  	enum btrfs_offload_csum_mode offload_csum_mode;
  #endif
#5 lore raw

Re: [PATCH 2/3] btrfs: use the path with the lowest latency for RAID1 reads

In reply to Anand Jain
Quoted 40 lines · Anand Jain

This feature aims to direct the read I/O to the device with the lowest known latency for reading RAID1 blocks.

echo "latency" > /sys/fs/btrfs/<UUID>/read_policy

---
  fs/btrfs/sysfs.c   |  2 +-
  fs/btrfs/volumes.c | 40 ++++++++++++++++++++++++++++++++++++++++
  fs/btrfs/volumes.h |  2 ++
  3 files changed, 43 insertions(+), 1 deletion(-)
fs/btrfs/sysfs.c +1 -1
index 18fb35a887c6..15abf931726c 100644
@@ -1306,7 +1306,7 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj,
  BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);
 
  #ifdef CONFIG_BTRFS_DEBUG
-static const char * const btrfs_read_policy_name[] = { "pid", "rotation" };
+static const char * const btrfs_read_policy_name[] = { "pid", "rotation", "latency" };
  #else
  static const char * const btrfs_read_policy_name[] = { "pid" };
  #endif
fs/btrfs/volumes.c +3 -0
index c130a27386a7..20bc62d85b3b 100644
@@ -12,6 +12,9 @@
  #include <linux/uuid.h>
  #include <linux/list_sort.h>
  #include <linux/namei.h>
+#ifdef CONFIG_BTRFS_DEBUG
+#include <linux/part_stat.h>
+#endif
  #include "misc.h"
  #include "ctree.h"
  #include "disk-io.h"

fs/btrfs/volumes.c int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len) · line 5870

Quoted 8 lines
fs/btrfs/volumes.c +5 -0
@@ -5860,6 +5863,39 @@ int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len)
  }
 
  #ifdef CONFIG_BTRFS_DEBUG
+static int btrfs_best_stripe(struct btrfs_fs_info *fs_info,
+			     struct btrfs_chunk_map *map, int first,
+			     int num_stripe)
+{
+	u64 est_wait = 0;

Is this a typo of best_wait? Or do you mean estimated wait?

Quoted 8 lines
fs/btrfs/volumes.c +8 -0
+	int best_stripe = 0;
+	int index;
+
+	for (index = first; index < first + num_stripe; index++) {
+		u64 read_wait;
+		u64 avg_wait = 0;
+		unsigned long read_ios;
+		struct btrfs_device *device = map->stripes[index].dev;

fs/btrfs/volumes.c int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len) · line 5888

Quoted 10 lines
fs/btrfs/volumes.c +10 -0
+
+		read_wait = part_stat_read(device->bdev, nsecs[READ]);
+		read_ios = part_stat_read(device->bdev, ios[READ]);
+
+		if (read_wait && read_ios && read_wait >= read_ios)
+			avg_wait = div_u64(read_wait, read_ios);
+		else
+			btrfs_debug_rl(fs_info,
+			"devid: %llu avg_wait ZERO read_wait %llu read_ios %lu",
+				       device->devid, read_wait, read_ios);

I do not think we need this debug messages.

The device can have no read so far.

fs/btrfs/volumes.c int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len) · line 5890

Quoted 2 lines
fs/btrfs/volumes.c +2 -0
+
+		if (est_wait == 0 || est_wait > avg_wait) {

You can give @est_wait a default value of U64_MAX, so that you do not need to check for 0, just est_wait > avg_wait is enough.

Thanks,
Qu

Quoted 35 lines
fs/btrfs/volumes.c +12 -0
+			est_wait = avg_wait;
+			best_stripe = index;
+		}
+	}
+
+	return best_stripe;
+}
+
  struct stripe_mirror {
  	u64 devid;
  	int map;
@@ -5940,6 +5976,10 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,
  	case BTRFS_READ_POLICY_ROTATION:
  		preferred_mirror = btrfs_read_rotation(map, first, num_stripes);
  		break;
+	case BTRFS_READ_POLICY_LATENCY:
+		preferred_mirror = btrfs_best_stripe(fs_info, map, first,
+								num_stripes);
+		break;
  #endif
  	}
 
fs/btrfs/volumes.h +2 -0
index 81701217dbb9..09920ef76a9b 100644
@@ -306,6 +306,8 @@ enum btrfs_read_policy {
  #ifdef CONFIG_BTRFS_DEBUG
  	/* Balancing raid1 reads across all striped devices */
  	BTRFS_READ_POLICY_ROTATION,
+	/* Use the lowest-latency device dynamically */
+	BTRFS_READ_POLICY_LATENCY,
  #endif
  	BTRFS_NR_READ_POLICY,
  };
#8 lore raw
In reply to Anand Jain
Quoted 6 lines · Anand Jain

The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the CONFIG_BTRFS_DEBUG config option and are on top of the previously added /sys/fs/btrfs/<UUID>/read_policy interface to configure the desired RAID1 read balancing method.

I am currently testing this on 6.12-rc1 with policy rotation, seems good for now. Would be better if policy can be set in mount options.

HAN Yuwei

Quoted 58 lines

I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from /sys/devices/<..>/stat for each device. Below is a summary of the results, with each result the average of three iterations.

A typical generic random rw workload:

$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \
   --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \
   --group_reporting --name=iops-test-job --eta-newline=1
|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 29.4MiB/s  | 29.5MiB/s  | 456548 | 447975 |
| rotation| 29.3MiB/s  | 29.3MiB/s  | 450105 | 450055 |
| latency | 21.9MiB/s  | 21.9MiB/s  | 672387 | 0      |
| devid:1 | 22.0MiB/s  | 22.0MiB/s  | 674788 | 0      |

Defragmentation with compression workload:

$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo
$ sync
$ echo 3 > /proc/sys/vm/drop_caches
$ btrfs filesystem defrag -f -c /btrfs/foo
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 21.61s| 3810   | 0      |
| rotation| 11.55s| 1905   | 1905   |
| latency | 20.99s| 0      | 3810   |
| devid:2 | 21.41s| 0      | 3810   |

. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.

Anand Jain (3):
btrfs: introduce RAID1 round-robin read balancing
btrfs: use the path with the lowest latency for RAID1 reads
btrfs: add RAID1 preferred read device feature

  fs/btrfs/sysfs.c   |  94 ++++++++++++++++++++++++++++++-------
  fs/btrfs/volumes.c | 113 +++++++++++++++++++++++++++++++++++++++++++++
  fs/btrfs/volumes.h |  14 ++++++
  3 files changed, 205 insertions(+), 16 deletions(-)
author
#9 lore raw
Quoted 9 lines · Yuwei Han
Quoted 6 lines · Anand Jain

The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the CONFIG_BTRFS_DEBUG config option and are on top of the previously added /sys/fs/btrfs/<UUID>/read_policy interface to configure the desired RAID1 read balancing method.

I am currently testing this on 6.12-rc1 with policy rotation, seems good for now.

Thanks for testing and reviewing.

Quoted 1 line

Would be better if policy can be set in mount options.

I think it is a good idea. However, we should also consolidate our sysfs knobs, mount options, and btrfs properties all together where applicable and make it easy to use.

V2 is in the ML.

Thanks,
-Anand

Quoted 61 lines

HAN Yuwei

Quoted 58 lines

I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from /sys/devices/<..>/stat for each device. Below is a summary of the results, with each result the average of three iterations.

A typical generic random rw workload:

$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \
   --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 -- time_based \
   --group_reporting --name=iops-test-job --eta-newline=1
|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 29.4MiB/s  | 29.5MiB/s  | 456548 | 447975 |
| rotation| 29.3MiB/s  | 29.3MiB/s  | 450105 | 450055 |
| latency | 21.9MiB/s  | 21.9MiB/s  | 672387 | 0      |
| devid:1 | 22.0MiB/s  | 22.0MiB/s  | 674788 | 0      |

Defragmentation with compression workload:

$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo
$ sync
$ echo 3 > /proc/sys/vm/drop_caches
$ btrfs filesystem defrag -f -c /btrfs/foo
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 21.61s| 3810   | 0      |
| rotation| 11.55s| 1905   | 1905   |
| latency | 20.99s| 0      | 3810   |
| devid:2 | 21.41s| 0      | 3810   |

. The PID-based balancing method works well for the generic random rw fio workload.
. The rotation method is ideal when you want to keep both devices active, and it boosts performance in sequential defragmentation scenarios. . The latency-based method work well when we have mixed device types or when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and testing RAID1 mirror synchronizations.

Anand Jain (3):
   btrfs: introduce RAID1 round-robin read balancing
   btrfs: use the path with the lowest latency for RAID1 reads
   btrfs: add RAID1 preferred read device feature

  fs/btrfs/sysfs.c   |  94 ++++++++++++++++++++++++++++++------- fs/btrfs/volumes.c | 113 +++++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h |  14 ++++++

  3 files changed, 205 insertions(+), 16 deletions(-)
author
#6 lore raw

Re: [PATCH 2/3] btrfs: use the path with the lowest latency for RAID1 reads

In reply to Qu Wenruo
Quoted 20 lines
Quoted 20 lines
fs/btrfs/volumes.c +3 -0
index c130a27386a7..20bc62d85b3b 100644
@@ -12,6 +12,9 @@
   #include <linux/uuid.h>
   #include <linux/list_sort.h>
   #include <linux/namei.h>
+#ifdef CONFIG_BTRFS_DEBUG
+#include <linux/part_stat.h>
+#endif
   #include "misc.h"
   #include "ctree.h"
   #include "disk-io.h"
@@ -5860,6 +5863,39 @@ int btrfs_is_parity_mirror(struct btrfs_fs_info 
 *fs_info, u64 logical, u64 len)
   }
 
Quoted 9 lines

fs/btrfs/volumes.c int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len) · line 5870

Quoted 6 lines
fs/btrfs/volumes.c +5 -0
   #ifdef CONFIG_BTRFS_DEBUG
+static int btrfs_best_stripe(struct btrfs_fs_info *fs_info,
+                 struct btrfs_chunk_map *map, int first,
+                 int num_stripe)
+{
+    u64 est_wait = 0;

Is this a typo of best_wait? Or do you mean estimated wait?

It is best_wait. Fixed in v2.

Quoted 21 lines
Quoted 8 lines
fs/btrfs/volumes.c +8 -0
+    int best_stripe = 0;
+    int index;
+
+    for (index = first; index < first + num_stripe; index++) {
+        u64 read_wait;
+        u64 avg_wait = 0;
+        unsigned long read_ios;
+        struct btrfs_device *device = map->stripes[index].dev;

fs/btrfs/volumes.c int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len) · line 5888

Quoted 10 lines
fs/btrfs/volumes.c +10 -0
+
+        read_wait = part_stat_read(device->bdev, nsecs[READ]);
+        read_ios = part_stat_read(device->bdev, ios[READ]);
+
+        if (read_wait && read_ios && read_wait >= read_ios)
+            avg_wait = div_u64(read_wait, read_ios);
+        else
+            btrfs_debug_rl(fs_info,
+            "devid: %llu avg_wait ZERO read_wait %llu read_ios %lu",
+                       device->devid, read_wait, read_ios);

I do not think we need this debug messages.

Quoted 2 lines

The device can have no read so far.

Um. Yeah, we can remove it.

Quoted 6 lines

fs/btrfs/volumes.c int btrfs_is_parity_mirror(struct btrfs_fs_info · line 5891

Quoted 2 lines
fs/btrfs/volumes.c +2 -0
+
+        if (est_wait == 0 || est_wait > avg_wait) {

You can give @est_wait a default value of U64_MAX, so that you do not need to check for 0, just est_wait > avg_wait is enough.

Fixed in v2.

Thanks for the review.

- Anand

Quoted 2 lines

Thanks,
Qu

author
#3 lore raw

Re: [PATCH 1/3] btrfs: introduce RAID1 round-robin read balancing

In reply to Qu Wenruo
Quoted 5 lines
Quoted 2 lines

+#ifdef CONFIG_BTRFS_DEBUG

It would be much better to utilize CONFIG_BTRFS_EXPERIMENTAL. CONFIG_BTRFS_DEBUG is now for pure debug purposes.

Yes, I noticed the recent patch that changed that. Fixed in v2.

Thanks, Anand