Skip to contentSkip to conversation

raid1 balancing methods

PATCH In review Anand Jain 4 participants 26 messages Jan 1, 2025 – Apr 10, 2025 Btrfs file system

lore
Share & tools
lore

Anand Jain

author
#0 lore raw

v5 cover raid1 balancing methods

v5:
Fixes based on review comments:
. Rewrite btrfs_read_policy_to_enum() using sysfs_match_string()

    and `strncpy()`.
  . Rewrite the round-robin method based on read counts.
  . Fix the smatch indentation warning.
 - Change the default minimum contiguous device read size for round-robin
   from 256K to 192K, as the latter performs slightly better.
 - Introduce a framework to track filesystem read counts. (New patch)
 - Reran defrag performance numbers
      $ xfs_io -f -d -c 'pwrite -S 0xab 0 1000000000' /btrfs/P6B
      $ time -p btrfs filesystem defrag -r -f -c /btrfs
|         | Time  | Read I/O Count  | gain  |
|         | Real  | devid1 | devid2 | w-PID |
|---------|-------|--------|--------|-------|
| pid     | 11.14s| 3808   | 0      |  -    |
| rotation|       |        |        |       |
|   196608|  6.54s| 2532   | 1276   | 41.29%|
|   262144|  8.42s| 1907   | 1901   | 24.41%|
| devid:1 | 10.95s| 3807   | 0      | 1.70% |

v4:
Fixes based on review comments:
3/10: Use == 0 to check strlen(str); drop dynamic alloc for %param.
4/10: Add __maybe_unused for %value_str in btrfs_read_policy_to_enum().
Return int instead of enum btrfs_read_policy.
5/10: Fix change-log (: is part of optional [ ]).
Wrap btrfs_read_policy_name[] with ifdef for new methods.
Use IS_ALIGNED() for sector-size alignment check.
Roundup misaligned %value.
Use named constants: BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ, BTRFS_RAID1_MAX_MIRRORS. Mark %s1 and %s2 in btrfs_cmp_devid() as const.
Add comments to btrfs_read_rr();
Use loop-local %i. Add space around /.
Use >> for sector-size division.
Prefix %min_contiguous_read with rr.
7/10: Move experimental to the top of the feature list.
Use experiment=on. Skip printing when off.

v3:
1. Removed the latency-based RAID1 balancing patch. (Per David's review)
2. Renamed "rotation" to "round-robin" and set the per-set min_contiguous_read to 256k. (Per David's review)
3. Added raid1-balancing module configuration for fstests testing. raid1-balancing can now be configured through both module load parameters and sysfs.

The logic of individual methods remains unchanged, and performance metrics are consistent with v2.

-----
v2:
1. Move new features to CONFIG_BTRFS_EXPERIMENTAL instead of CONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait to U64_MAX and remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:

echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy

The default value is the sector size, and the min_contiguous_read value must be a multiple of the sector size.

5. Tested FIO random read/write and defrag compression workloads with min_contiguous_read set to sector size, 192k, and 256k.

RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.

     $ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
        --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
        --time_based --group_reporting --name=iops-test-job --eta-newline=1
|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 20.3MiB/s  | 20.5MiB/s  | 313895 | 313895 |
| rotation|            |            |        |        |
|     4096| 20.4MiB/s  | 20.5MiB/s  | 313895 | 313895 |
|   196608| 20.2MiB/s  | 20.2MiB/s  | 310152 | 310175 |
|   262144| 20.3MiB/s  | 20.4MiB/s  | 312180 | 312191 |
|  latency| 18.4MiB/s  | 18.4MiB/s  | 272980 | 291683 |
| devid:1 | 14.8MiB/s  | 14.9MiB/s  | 456376 | 0      |

rotation RAID1 balancing technique performs more than 2x better for single-process defrag.

      $ time -p btrfs filesystem defrag -r -f -c /btrfs
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 18.00s| 3800   | 0      |
| rotation|       |        |        |
|     4096|  8.95s| 1900   | 1901   |
|   196608|  8.50s| 1881   | 1919   |
|   262144|  8.80s| 1881   | 1919   |
| latency | 17.18s| 3800   | 0      |
| devid:2 | 17.48s| 0      | 3800   |

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.

Latency is smarter with unstable block layer transport.

Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.

Devid is a hands-on approach, provides manual or user-space script control.

These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.

Thx.

--------- original v1 ------------

The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the CONFIG_BTRFS_DEBUG config option and are on top of the previously added /sys/fs/btrfs/<UUID>/read_policy interface to configure the desired RAID1 read balancing method.

I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from /sys/devices/<..>/stat for each device. Below is a summary of the results, with each result the average of three iterations.

A typical generic random rw workload:

$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \
  --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \
  --group_reporting --name=iops-test-job --eta-newline=1
|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 29.4MiB/s  | 29.5MiB/s  | 456548 | 447975 |
| rotation| 29.3MiB/s  | 29.3MiB/s  | 450105 | 450055 |
| latency | 21.9MiB/s  | 21.9MiB/s  | 672387 | 0      |
| devid:1 | 22.0MiB/s  | 22.0MiB/s  | 674788 | 0      |

Defragmentation with compression workload:

$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo
$ sync
$ echo 3 > /proc/sys/vm/drop_caches
$ btrfs filesystem defrag -f -c /btrfs/foo
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 21.61s| 3810   | 0      |
| rotation| 11.55s| 1905   | 1905   |
| latency | 20.99s| 0      | 3810   |
| devid:2 | 21.41s| 0      | 3810   |

. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.

Anand Jain (10):
btrfs: initialize fs_devices->fs_info earlier
btrfs: simplify output formatting in btrfs_read_policy_show
btrfs: add btrfs_read_policy_to_enum helper and refactor read policy
store
btrfs: handle value associated with raid1 balancing parameter
btrfs: add read count tracking for filesystem stats
btrfs: introduce RAID1 round-robin read balancing
btrfs: add RAID1 preferred read device
btrfs: expose experimental mode in module information
btrfs: enable RAID1 balancing configuration via modprobe parameter
btrfs: modload to print RAID1 balancing status

 fs/btrfs/bio.c     |   8 +++
 fs/btrfs/disk-io.c |   4 ++
 fs/btrfs/super.c   |  18 +++++
 fs/btrfs/sysfs.c   | 168 ++++++++++++++++++++++++++++++++++++++++-----
 fs/btrfs/sysfs.h   |   5 ++
 fs/btrfs/volumes.c | 115 ++++++++++++++++++++++++++++++-
 fs/btrfs/volumes.h |  22 +++++-
 7 files changed, 317 insertions(+), 23 deletions(-)
Signature · 1 line
2.47.0
author
#1 lore raw

PATCH v5 1/10 btrfs: initialize fs_devices->fs_info earlier

Currently, fs_devices->fs_info is initialized in btrfs_init_devices_late(), but this occurs too late for find_live_mirror(), which is invoked by load_super_root() much earlier than btrfs_init_devices_late().

Fix this by moving the initialization to open_ctree(), before load_super_root().

---
 fs/btrfs/disk-io.c | 1 +
 fs/btrfs/volumes.c | 2 --
 2 files changed, 1 insertion(+), 2 deletions(-)
fs/btrfs/disk-io.c +1 -0
index 814320948645..ab45b02df957 100644
@@ -3321,6 +3321,7 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device
 	fs_info->sectors_per_page = (PAGE_SIZE >> fs_info->sectorsize_bits);
 	fs_info->csums_per_leaf = BTRFS_MAX_ITEM_SIZE(fs_info) / fs_info->csum_size;
 	fs_info->stripesize = stripesize;
+	fs_info->fs_devices->fs_info = fs_info;
 
 	/*
 	 * Handle the space caching options appropriately now that we have the
fs/btrfs/volumes.c +0 -2
index 1cccaf9c2b0d..fe5ceea2ba0b 100644
@@ -7568,8 +7568,6 @@ int btrfs_init_devices_late(struct btrfs_fs_info *fs_info)
 	struct btrfs_device *device;
 	int ret = 0;
 
-	fs_devices->fs_info = fs_info;
-
 	mutex_lock(&fs_devices->device_list_mutex);
 	list_for_each_entry(device, &fs_devices->devices, dev_list)
 		device->fs_info = fs_info;
Signature · 1 line
2.47.0
author
#2 lore raw

PATCH v5 2/10 btrfs: simplify output formatting in btrfs_read_policy_show

In reply to Anand Jain

Refactor the logic in btrfs_read_policy_show() to streamline the formatting of read policies output. Streamline the space and bracket handling around the active policy without altering the functional output. This is in preparation to add more methods.

---
 fs/btrfs/sysfs.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
fs/btrfs/sysfs.c +10 -8
index b843308e2bc6..fd3c49c6c3c5 100644
@@ -1316,14 +1316,16 @@ static ssize_t btrfs_read_policy_show(struct kobject *kobj,
 	int i;
 
 	for (i = 0; i < BTRFS_NR_READ_POLICY; i++) {
-		if (policy == i)
-			ret += sysfs_emit_at(buf, ret, "%s[%s]",
-					 (ret == 0 ? "" : " "),
-					 btrfs_read_policy_name[i]);
-		else
-			ret += sysfs_emit_at(buf, ret, "%s%s",
-					 (ret == 0 ? "" : " "),
-					 btrfs_read_policy_name[i]);
+		if (ret != 0)
+			ret += sysfs_emit_at(buf, ret, " ");
+
+		if (i == policy)
+			ret += sysfs_emit_at(buf, ret, "[");
+
+		ret += sysfs_emit_at(buf, ret, "%s", btrfs_read_policy_name[i]);
+
+		if (i == policy)
+			ret += sysfs_emit_at(buf, ret, "]");
 	}
 
 	ret += sysfs_emit_at(buf, ret, "\n");
Signature · 1 line
2.47.0
author
#3 lore raw

PATCH v5 3/10 btrfs: add btrfs_read_policy_to_enum helper and refactor read policy store

In reply to Anand Jain

Introduce the btrfs_read_policy_to_enum helper function to simplify the conversion of a string read policy to its corresponding enum value. This reduces duplication and improves code clarity in btrfs_read_policy_store. The btrfs_read_policy_store function has been refactored to use the new helper.

The parameter is copied locally to allow modification, enabling the separation of the method and its value. This prepares for the addition of more functionality in subsequent patches.

---
 fs/btrfs/sysfs.c | 34 ++++++++++++++++++++++------------
 1 file changed, 22 insertions(+), 12 deletions(-)
fs/btrfs/sysfs.c +22 -12
index fd3c49c6c3c5..3b0325259c02 100644
@@ -1307,6 +1307,18 @@ BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);
 
 static const char * const btrfs_read_policy_name[] = { "pid" };
 
+static int btrfs_read_policy_to_enum(const char *str)
+{
+	char param[32] = {'\0'};
+
+	if (!str || strlen(str) == 0)
+		return 0;
+
+	strncpy(param, str, sizeof(param) - 1);
+
+	return sysfs_match_string(btrfs_read_policy_name, param);
+}
+
 static ssize_t btrfs_read_policy_show(struct kobject *kobj,
 				      struct kobj_attribute *a, char *buf)
 {
@@ -1338,21 +1350,19 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj,
 				       const char *buf, size_t len)
 {
 	struct btrfs_fs_devices *fs_devices = to_fs_devs(kobj);
-	int i;
+	int index;
 
-	for (i = 0; i < BTRFS_NR_READ_POLICY; i++) {
-		if (sysfs_streq(buf, btrfs_read_policy_name[i])) {
-			if (i != READ_ONCE(fs_devices->read_policy)) {
-				WRITE_ONCE(fs_devices->read_policy, i);
-				btrfs_info(fs_devices->fs_info,
-					   "read policy set to '%s'",
-					   btrfs_read_policy_name[i]);
-			}
-			return len;
-		}
+	index = btrfs_read_policy_to_enum(buf);
+	if (index < 0)
+		return -EINVAL;
+
+	if (index != READ_ONCE(fs_devices->read_policy)) {
+		WRITE_ONCE(fs_devices->read_policy, index);
+		btrfs_info(fs_devices->fs_info, "read policy set to '%s'",
+			   btrfs_read_policy_name[index]);
 	}
 
-	return -EINVAL;
+	return len;
 }
 BTRFS_ATTR_RW(, read_policy, btrfs_read_policy_show, btrfs_read_policy_store);
 
Signature · 1 line
2.47.0
author
#4 lore raw

PATCH v5 4/10 btrfs: handle value associated with raid1 balancing parameter

In reply to Anand Jain

This change enables specifying additional configuration values alongside the raid1 balancing / read policy in a single input string.

Updated btrfs_read_policy_to_enum() to parse and handle a value associated with the policy in the format policy:value, the value part if present is converted 64-bit integer. Update btrfs_read_policy_store() to accommodate the new parameter.

---
 fs/btrfs/sysfs.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
fs/btrfs/sysfs.c +14 -2
index 3b0325259c02..cf6e5322621f 100644
@@ -1307,15 +1307,26 @@ BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);
 
 static const char * const btrfs_read_policy_name[] = { "pid" };
 
-static int btrfs_read_policy_to_enum(const char *str)
+static int btrfs_read_policy_to_enum(const char *str, s64 *value)
 {
 	char param[32] = {'\0'};
+	char *__maybe_unused value_str;
 
 	if (!str || strlen(str) == 0)
 		return 0;
 
 	strncpy(param, str, sizeof(param) - 1);
 
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+	/* Separate value from input in policy:value format. */
+	if ((value_str = strchr(param, ':'))) {
+		*value_str = '\0';
+		value_str++;
+		if (value && kstrtou64(value_str, 10, value) != 0)
+			return -EINVAL;
+	}
+#endif
+
 	return sysfs_match_string(btrfs_read_policy_name, param);
 }
 
@@ -1351,8 +1362,9 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj,
 {
 	struct btrfs_fs_devices *fs_devices = to_fs_devs(kobj);
 	int index;
+	s64 value = -1;
 
-	index = btrfs_read_policy_to_enum(buf);
+	index = btrfs_read_policy_to_enum(buf, &value);
 	if (index < 0)
 		return -EINVAL;
 
Signature · 1 line
2.47.0
#5 lore raw

Re: [PATCH v5 04/10] btrfs: handle value associated with raid1 balancing parameter

Quoted 29 lines · Anand Jain

This change enables specifying additional configuration values alongside the raid1 balancing / read policy in a single input string.

Updated btrfs_read_policy_to_enum() to parse and handle a value associated with the policy in the format policy:value, the value part if present is converted 64-bit integer. Update btrfs_read_policy_store() to accommodate the new parameter.

---
 fs/btrfs/sysfs.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
fs/btrfs/sysfs.c +2 -1
index 3b0325259c02..cf6e5322621f 100644
@@ -1307,15 +1307,26 @@ BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);
 
 static const char * const btrfs_read_policy_name[] = { "pid" };
 
-static int btrfs_read_policy_to_enum(const char *str)
+static int btrfs_read_policy_to_enum(const char *str, s64 *value)
 {
 	char param[32] = {'\0'};
+	char *__maybe_unused value_str;
 
 	if (!str || strlen(str) == 0)
 		return 0;

fs/btrfs/sysfs.c BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show); · line 1326

Quoted 10 lines
fs/btrfs/sysfs.c +7 -0
 
 	strncpy(param, str, sizeof(param) - 1);
 
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+	/* Separate value from input in policy:value format. */
+	if ((value_str = strchr(param, ':'))) {
+		*value_str = '\0';
+		value_str++;
+		if (value && kstrtou64(value_str, 10, value) != 0)
+			return -EINVAL;

I think I've mentioned that before, this lacks validation of the 'value', there should be some lower and upper bound check. Minimum can be the sectorsize and maximum maybe 1G or 2G, so the u32 type is sufficient. The silent conversion from u64 to s64 should be avoided.

author
#6 lore raw

Re: [PATCH v5 04/10] btrfs: handle value associated with raid1 balancing parameter

Quoted 33 lines · David Sterba
Quoted 32 lines · Anand Jain

This change enables specifying additional configuration values alongside the raid1 balancing / read policy in a single input string.

Updated btrfs_read_policy_to_enum() to parse and handle a value associated with the policy in the format policy:value, the value part if present is converted 64-bit integer. Update btrfs_read_policy_store() to accommodate the new parameter.

---
  fs/btrfs/sysfs.c | 16 ++++++++++++++--
  1 file changed, 14 insertions(+), 2 deletions(-)
fs/btrfs/sysfs.c +2 -1
index 3b0325259c02..cf6e5322621f 100644
@@ -1307,15 +1307,26 @@ BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);
 
  static const char * const btrfs_read_policy_name[] = { "pid" };
 
-static int btrfs_read_policy_to_enum(const char *str)
+static int btrfs_read_policy_to_enum(const char *str, s64 *value)
  {
  	char param[32] = {'\0'};
+	char *__maybe_unused value_str;
 
  	if (!str || strlen(str) == 0)
  		return 0;
 
  	strncpy(param, str, sizeof(param) - 1);
 
Quoted 12 lines

fs/btrfs/sysfs.c BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show); · line 1326

Quoted 7 lines
fs/btrfs/sysfs.c +7 -0
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+	/* Separate value from input in policy:value format. */
+	if ((value_str = strchr(param, ':'))) {
+		*value_str = '\0';
+		value_str++;
+		if (value && kstrtou64(value_str, 10, value) != 0)
+			return -EINVAL;

I think I've mentioned that before, this lacks validation of the 'value', there should be some lower and upper bound check. Minimum can be the sectorsize and maximum maybe 1G or 2G, so the u32 type is sufficient.

Regarding the lack of validation, as I replied earlier, only the minimum needs validation, not the upper limit. A 1TB disk or SAN storage can be connected to a 32-bit host, and we can still read the full 1TB from a single mirror. IMO, it depends on the use case, or we can revisit it based on feedback.?

The patch already handles the lower limit.

$ echo round-robin:1024 > /sys/fs/btrfs/b03580af-900e-4940-a7f8-3f04f9981a12/read_policy
$ cat /sys/fs/btrfs/b03580af-900e-4940-a7f8-3f04f9981a12/read_policy
pid [round-robin:4096] devid:1
$ dmesg -k | tail -1
BTRFS warning (device sda): read_policy: min contiguous read 1024 should be multiples of the sectorsize 4096, rounded to 4096
Quoted 1 line

The silent conversion from u64 to s64 should be avoided.

Do you mean from s64 to u64?

Thanks, Anand

#7 lore raw

Re: [PATCH v5 04/10] btrfs: handle value associated with raid1 balancing parameter

Quoted 13 lines · Anand Jain
Quoted 4 lines
Quoted 4 lines
fs/btrfs/sysfs.c +4 -0
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+	/* Separate value from input in policy:value format. */
+	if ((value_str = strchr(param, ':'))) {
+		*value_str = '\0';
Quoted 8 lines

fs/btrfs/sysfs.c BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show); · line 1326

Quoted 3 lines
fs/btrfs/sysfs.c +3 -0
+		value_str++;
+		if (value && kstrtou64(value_str, 10, value) != 0)
+			return -EINVAL;

I think I've mentioned that before, this lacks validation of the 'value', there should be some lower and upper bound check. Minimum can be the sectorsize and maximum maybe 1G or 2G, so the u32 type is sufficient.

Quoted 5 lines

Regarding the lack of validation, as I replied earlier, only the minimum needs validation, not the upper limit. A 1TB disk or SAN storage can be connected to a 32-bit host, and we can still read the full 1TB from a single mirror. IMO, it depends on the use case, or we can revisit it based on feedback.?

SAN is not a typical storage, what the load balancing should assume is a set of individual rotational or ssd/nvme disks. Also 32bit host is not something we're expecting by default.

Quoted 11 lines

The patch already handles the lower limit.

$ echo round-robin:1024 > 
/sys/fs/btrfs/b03580af-900e-4940-a7f8-3f04f9981a12/read_policy
$ cat /sys/fs/btrfs/b03580af-900e-4940-a7f8-3f04f9981a12/read_policy
pid [round-robin:4096] devid:1
$ dmesg -k | tail -1
BTRFS warning (device sda): read_policy: min contiguous read 1024 should 
be multiples of the sectorsize 4096, rounded to 4096

Right.

Quoted 3 lines
Quoted 1 line

The silent conversion from u64 to s64 should be avoided.

Do you mean from s64 to u64?

Yes, I've used kstrtos64 and reject negative values, then the types match. I'm not sure about the upper bound, the idea was like 1G or 2G but a fast NVME can do a few gigabytes per second so we may need more than a 32bit type can hold. But it's counted in blocks it's 4G * 4K so it's safe.

author
#8 lore raw

PATCH v5 5/10 btrfs: add read count tracking for filesystem stats

In reply to Anand Jain

Add fs_devices::read_cnt_blocks to track read blocks, initialize it in open_fs_devices() and clean it up in close_fs_devices(). btrfs_submit_dev_bio() increments it for reads when stats tracking is enabled. Stats tracking is disabled by default and is enabled through fs_devices::fs_stats when required.

The code is not under the EXPERIMENTAL define, as stats can be expanded to include write counts and other performance counters, with the user interface independent of its internal use.

This is an in-memory-only feature, different to the dev error stats.

---
 fs/btrfs/bio.c     | 8 ++++++++
 fs/btrfs/disk-io.c | 3 +++
 fs/btrfs/volumes.c | 8 +++++++-
 fs/btrfs/volumes.h | 7 ++++++-
 4 files changed, 24 insertions(+), 2 deletions(-)
fs/btrfs/bio.c +8 -0
index 1f216d07eff6..faefb18010fc 100644
@@ -450,6 +450,14 @@ static void btrfs_submit_dev_bio(struct btrfs_device *dev, struct bio *bio)
 		(unsigned long)dev->bdev->bd_dev, btrfs_dev_name(dev),
 		dev->devid, bio->bi_iter.bi_size);
 
+	/*
+	 * Track reads if tracking is enabled; ignore I/O operations before
+	 * fully initialized.
+	 */
+	if (dev->fs_devices->fs_stats && bio_op(bio) == REQ_OP_READ && dev->fs_info)
+		percpu_counter_add(&dev->fs_devices->read_cnt_blocks,
+				   bio->bi_iter.bi_size >> dev->fs_info->sectorsize_bits);
+
 	if (bio->bi_opf & REQ_BTRFS_CGROUP_PUNT)
 		blkcg_punt_bio_submit(bio);
 	else
fs/btrfs/disk-io.c +3 -0
index ab45b02df957..79b859790e8c 100644
@@ -3481,6 +3481,9 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device
 		goto fail_sysfs;
 	}
 
+	/* Disable filesystem stats tracking unless required by a feature. */
+	fs_devices->fs_stats = false;
+
 	ret = btrfs_read_block_groups(fs_info);
 	if (ret) {
 		btrfs_err(fs_info, "failed to read block groups: %d", ret);
fs/btrfs/volumes.c +7 -1
index fe5ceea2ba0b..1fa40bf6f708 100644
@@ -1254,6 +1254,7 @@ static void close_fs_devices(struct btrfs_fs_devices *fs_devices)
 	list_for_each_entry_safe(device, tmp, &fs_devices->devices, dev_list)
 		btrfs_close_one_device(device);
 
+	percpu_counter_destroy(&fs_devices->read_cnt_blocks);
 	WARN_ON(fs_devices->open_devices);
 	WARN_ON(fs_devices->rw_devices);
 	fs_devices->opened = 0;
@@ -1300,6 +1301,11 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices,
 	struct btrfs_device *tmp_device;
 	int ret = 0;
 
+	/* Initialize the in-memory record of filesystem read count */
+	ret = percpu_counter_init(&fs_devices->read_cnt_blocks, 0, GFP_KERNEL);
+	if (ret)
+		return ret;
+
 	list_for_each_entry_safe(device, tmp_device, &fs_devices->devices,
 				 dev_list) {
 		int ret2;
@@ -7669,7 +7675,7 @@ int btrfs_init_dev_stats(struct btrfs_fs_info *fs_info)
 	list_for_each_entry(device, &fs_devices->devices, dev_list) {
 		ret = btrfs_device_init_dev_stats(device, path);
 		if (ret)
-			goto out;
+			return ret;
 	}
 	list_for_each_entry(seed_devs, &fs_devices->seed_list, seed_list) {
 		list_for_each_entry(device, &seed_devs->devices, dev_list) {
fs/btrfs/volumes.h +6 -1
index 3a416b1bc24c..45d0eb3429c6 100644
@@ -185,7 +185,7 @@ struct btrfs_device {
 	 * enum btrfs_dev_stat_values in ioctl.h */
 	int dev_stats_valid;
 
-	/* Counter to record the change of device stats */
+	/* Counter to record of the change of device stats */
 	atomic_t dev_stats_ccnt;
 	atomic_t dev_stat_values[BTRFS_DEV_STAT_VALUES_MAX];
 
@@ -417,6 +417,8 @@ struct btrfs_fs_devices {
 	bool seeding;
 	/* The mount needs to use a randomly generated fsid. */
 	bool temp_fsid;
+	/* Enable/disable the filesystem stats tracking */
+	bool fs_stats;
 
 	struct btrfs_fs_info *fs_info;
 	/* sysfs kobjects */
@@ -427,6 +429,9 @@ struct btrfs_fs_devices {
 
 	enum btrfs_chunk_allocation_policy chunk_alloc_policy;
 
+	/* Tracks the number of blocks (sectors) read from the filesystem. */
+	struct percpu_counter read_cnt_blocks;
+
 	/* Policy used to read the mirrored stripes. */
 	enum btrfs_read_policy read_policy;
 
Signature · 1 line
2.47.0
author
#9 lore raw

PATCH 1/1 fixup: btrfs: add read count tracking for filesystem stats

fs_devices::fs_stats is disabled by default during allocation. Explicitly disabling it in open_ctree() overrides this setting during device open. So we dont' have to disable it in the open_ctree().

This patch should be merged with in the ML.
btrfs: add read count tracking for filesystem stats

---
 fs/btrfs/disk-io.c | 3 ---
 1 file changed, 3 deletions(-)
fs/btrfs/disk-io.c +0 -3
index 79b859790e8c..ab45b02df957 100644
@@ -3481,9 +3481,6 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device
 		goto fail_sysfs;
 	}
 
-	/* Disable filesystem stats tracking unless required by a feature. */
-	fs_devices->fs_stats = false;
-
 	ret = btrfs_read_block_groups(fs_info);
 	if (ret) {
 		btrfs_err(fs_info, "failed to read block groups: %d", ret);
Signature · 1 line
2.47.0
#10 lore raw

Re: [PATCH v5 05/10] btrfs: add read count tracking for filesystem stats

In reply to Anand Jain

Hi Anand,

kernel test robot noticed the following build warnings:

https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Anand-Jain/btrfs-initialize-fs_devices-fs_info-earlier/20250102-021904
base:   v6.13-rc5
patch link:    https://lore.kernel.org/r/c4010cee5398e35a695def3ad97d4de6f136ae2c.1735748715.git.anand.jain%40oracle.com
patch subject: [PATCH v5 05/10] btrfs: add read count tracking for filesystem stats
config: i386-randconfig-r072-20250103 (https://download.01.org/0day-ci/archive/20250104/[email protected]/config)
compiler: clang version 19.1.3 (https://github.com/llvm/llvm-project ab51eccf88f5321e7c60591c5546b254b6afab99)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags

| Reported-by: kernel test robot <[email protected]>
| Reported-by: Dan Carpenter <[email protected]>
| Closes: https://lore.kernel.org/r/[email protected]/

New smatch warnings:
fs/btrfs/volumes.c:7691 btrfs_init_dev_stats() warn: inconsistent returns '&fs_devices->device_list_mutex'.

vim +7691 fs/btrfs/volumes.c

124604eb50f88e Josef Bacik    2020-09-18  7663  int btrfs_init_dev_stats(struct btrfs_fs_info *fs_info)
124604eb50f88e Josef Bacik    2020-09-18  7664  {
124604eb50f88e Josef Bacik    2020-09-18  7665  	struct btrfs_fs_devices *fs_devices = fs_info->fs_devices, *seed_devs;
124604eb50f88e Josef Bacik    2020-09-18  7666  	struct btrfs_device *device;
124604eb50f88e Josef Bacik    2020-09-18  7667  	struct btrfs_path *path = NULL;
92e26df43b1a97 Josef Bacik    2020-09-18  7668  	int ret = 0;
124604eb50f88e Josef Bacik    2020-09-18  7669  
124604eb50f88e Josef Bacik    2020-09-18  7670  	path = btrfs_alloc_path();
124604eb50f88e Josef Bacik    2020-09-18  7671  	if (!path)
124604eb50f88e Josef Bacik    2020-09-18  7672  		return -ENOMEM;
124604eb50f88e Josef Bacik    2020-09-18  7673  
124604eb50f88e Josef Bacik    2020-09-18  7674  	mutex_lock(&fs_devices->device_list_mutex);
92e26df43b1a97 Josef Bacik    2020-09-18  7675  	list_for_each_entry(device, &fs_devices->devices, dev_list) {
92e26df43b1a97 Josef Bacik    2020-09-18  7676  		ret = btrfs_device_init_dev_stats(device, path);
92e26df43b1a97 Josef Bacik    2020-09-18  7677  		if (ret)
ec90aa75ef29fb Anand Jain     2025-01-02  7678  			return ret;

mutex_unlock(&fs_devices->device_list_mutex);

92e26df43b1a97 Josef Bacik    2020-09-18  7679  	}
124604eb50f88e Josef Bacik    2020-09-18  7680  	list_for_each_entry(seed_devs, &fs_devices->seed_list, seed_list) {
92e26df43b1a97 Josef Bacik    2020-09-18  7681  		list_for_each_entry(device, &seed_devs->devices, dev_list) {
92e26df43b1a97 Josef Bacik    2020-09-18  7682  			ret = btrfs_device_init_dev_stats(device, path);
92e26df43b1a97 Josef Bacik    2020-09-18  7683  			if (ret)
92e26df43b1a97 Josef Bacik    2020-09-18  7684  				goto out;
124604eb50f88e Josef Bacik    2020-09-18  7685  		}
92e26df43b1a97 Josef Bacik    2020-09-18  7686  	}
92e26df43b1a97 Josef Bacik    2020-09-18  7687  out:
733f4fbbc1083a Stefan Behrens 2012-05-25  7688  	mutex_unlock(&fs_devices->device_list_mutex);
733f4fbbc1083a Stefan Behrens 2012-05-25  7689  
733f4fbbc1083a Stefan Behrens 2012-05-25  7690  	btrfs_free_path(path);

92e26df43b1a97 Josef Bacik 2020-09-18 @7691 return ret;
733f4fbbc1083a Stefan Behrens 2012-05-25 7692 }

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author
#11 lore raw

Re: [PATCH v5 05/10] btrfs: add read count tracking for filesystem stats

Hi Dan,

This was fixed earlier. Thanks for the report.

-Anand

Quoted 58 lines · Dan Carpenter

Hi Anand,

kernel test robot noticed the following build warnings:

https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Anand-Jain/btrfs-initialize-fs_devices-fs_info-earlier/20250102-021904
base:   v6.13-rc5
patch link:    https://lore.kernel.org/r/c4010cee5398e35a695def3ad97d4de6f136ae2c.1735748715.git.anand.jain%40oracle.com
patch subject: [PATCH v5 05/10] btrfs: add read count tracking for filesystem stats
config: i386-randconfig-r072-20250103 (https://download.01.org/0day-ci/archive/20250104/[email protected]/config)
compiler: clang version 19.1.3 (https://github.com/llvm/llvm-project ab51eccf88f5321e7c60591c5546b254b6afab99)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags

| Reported-by: kernel test robot <[email protected]>
| Reported-by: Dan Carpenter <[email protected]>
| Closes: https://lore.kernel.org/r/[email protected]/

New smatch warnings:
fs/btrfs/volumes.c:7691 btrfs_init_dev_stats() warn: inconsistent returns '&fs_devices->device_list_mutex'.

vim +7691 fs/btrfs/volumes.c

124604eb50f88e Josef Bacik    2020-09-18  7663  int btrfs_init_dev_stats(struct btrfs_fs_info *fs_info)
124604eb50f88e Josef Bacik    2020-09-18  7664  {
124604eb50f88e Josef Bacik    2020-09-18  7665  	struct btrfs_fs_devices *fs_devices = fs_info->fs_devices, *seed_devs;
124604eb50f88e Josef Bacik    2020-09-18  7666  	struct btrfs_device *device;
124604eb50f88e Josef Bacik    2020-09-18  7667  	struct btrfs_path *path = NULL;
92e26df43b1a97 Josef Bacik    2020-09-18  7668  	int ret = 0;
124604eb50f88e Josef Bacik    2020-09-18  7669
124604eb50f88e Josef Bacik    2020-09-18  7670  	path = btrfs_alloc_path();
124604eb50f88e Josef Bacik    2020-09-18  7671  	if (!path)
124604eb50f88e Josef Bacik    2020-09-18  7672  		return -ENOMEM;
124604eb50f88e Josef Bacik    2020-09-18  7673
124604eb50f88e Josef Bacik    2020-09-18  7674  	mutex_lock(&fs_devices->device_list_mutex);
92e26df43b1a97 Josef Bacik    2020-09-18  7675  	list_for_each_entry(device, &fs_devices->devices, dev_list) {
92e26df43b1a97 Josef Bacik    2020-09-18  7676  		ret = btrfs_device_init_dev_stats(device, path);
92e26df43b1a97 Josef Bacik    2020-09-18  7677  		if (ret)
ec90aa75ef29fb Anand Jain     2025-01-02  7678  			return ret;

mutex_unlock(&fs_devices->device_list_mutex);

92e26df43b1a97 Josef Bacik    2020-09-18  7679  	}
124604eb50f88e Josef Bacik    2020-09-18  7680  	list_for_each_entry(seed_devs, &fs_devices->seed_list, seed_list) {
92e26df43b1a97 Josef Bacik    2020-09-18  7681  		list_for_each_entry(device, &seed_devs->devices, dev_list) {
92e26df43b1a97 Josef Bacik    2020-09-18  7682  			ret = btrfs_device_init_dev_stats(device, path);
92e26df43b1a97 Josef Bacik    2020-09-18  7683  			if (ret)
92e26df43b1a97 Josef Bacik    2020-09-18  7684  				goto out;
124604eb50f88e Josef Bacik    2020-09-18  7685  		}
92e26df43b1a97 Josef Bacik    2020-09-18  7686  	}
92e26df43b1a97 Josef Bacik    2020-09-18  7687  out:
733f4fbbc1083a Stefan Behrens 2012-05-25  7688  	mutex_unlock(&fs_devices->device_list_mutex);
733f4fbbc1083a Stefan Behrens 2012-05-25  7689
733f4fbbc1083a Stefan Behrens 2012-05-25  7690  	btrfs_free_path(path);

92e26df43b1a97 Josef Bacik 2020-09-18 @7691 return ret;
733f4fbbc1083a Stefan Behrens 2012-05-25 7692 }

author
#12 lore raw

PATCH v5 6/10 btrfs: introduce RAID1 round-robin read balancing

In reply to Anand Jain

This feature balances I/O across the striped devices when reading from RAID1 blocks.

echo round-robin[:min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy

The min_contiguous_read parameter defines the minimum read size before switching to the next mirrored device. This setting is optional, with a default value of 192KiB.

---
 fs/btrfs/sysfs.c   | 49 ++++++++++++++++++++++++++++++-
 fs/btrfs/volumes.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/volumes.h | 10 +++++++
 3 files changed, 130 insertions(+), 1 deletion(-)
fs/btrfs/sysfs.c +48 -1
index cf6e5322621f..70f89d1adfbc 100644
@@ -1305,7 +1305,12 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj,
 }
 BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);
 
-static const char * const btrfs_read_policy_name[] = { "pid" };
+static const char *btrfs_read_policy_name[] = {
+	"pid",
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+	"round-robin",
+#endif
+};
 
 static int btrfs_read_policy_to_enum(const char *str, s64 *value)
 {
@@ -1347,6 +1352,12 @@ static ssize_t btrfs_read_policy_show(struct kobject *kobj,
 
 		ret += sysfs_emit_at(buf, ret, "%s", btrfs_read_policy_name[i]);
 
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+		if (i == BTRFS_READ_POLICY_RR)
+			ret += sysfs_emit_at(buf, ret, ":%d",
+				 READ_ONCE(fs_devices->rr_min_contiguous_read));
+#endif
+
 		if (i == policy)
 			ret += sysfs_emit_at(buf, ret, "]");
 	}
@@ -1368,6 +1379,42 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj,
 	if (index < 0)
 		return -EINVAL;
 
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+	/* If moving out of RR then disable fs_stats */
+	if (fs_devices->read_policy == BTRFS_READ_POLICY_RR &&
+	    index != BTRFS_READ_POLICY_RR)
+		fs_devices->fs_stats = false;
+
+	if (index == BTRFS_READ_POLICY_RR) {
+		if (value != -1) {
+			u32 sectorsize = fs_devices->fs_info->sectorsize;
+
+			if (!IS_ALIGNED(value, sectorsize)) {
+				u64 temp_value = round_up(value, sectorsize);
+
+				btrfs_warn(fs_devices->fs_info,
+"read_policy: min contiguous read %lld should be multiples of the sectorsize %u, rounded to %llu",
+					  value, sectorsize, temp_value);
+				value = temp_value;
+			}
+		} else {
+			value = BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ;
+		}
+
+		if (index != READ_ONCE(fs_devices->read_policy) ||
+		    value != READ_ONCE(fs_devices->rr_min_contiguous_read)) {
+			WRITE_ONCE(fs_devices->read_policy, index);
+			WRITE_ONCE(fs_devices->rr_min_contiguous_read, value);
+
+			btrfs_info(fs_devices->fs_info, "read policy set to '%s:%lld'",
+				   btrfs_read_policy_name[index], value);
+		}
+
+		fs_devices->fs_stats = true;
+
+		return len;
+	}
+#endif
 	if (index != READ_ONCE(fs_devices->read_policy)) {
 		WRITE_ONCE(fs_devices->read_policy, index);
 		btrfs_info(fs_devices->fs_info, "read policy set to '%s'",
fs/btrfs/volumes.c +72 -0

90 lines not on this page · raw message

fs/btrfs/volumes.h +10 -0
index 45d0eb3429c6..5728b8717317 100644
@@ -296,6 +296,9 @@ enum btrfs_chunk_allocation_policy {
 	BTRFS_CHUNK_ALLOC_ZONED,
 };
 
+/* SZ_192K = 192 * 1024 = 196608 */
+#define BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ	(196608)
+#define BTRFS_RAID1_MAX_MIRRORS			(4)
 /*
  * Read policies for mirrored block group profiles, read picks the stripe based
  * on these policies.
@@ -303,6 +306,10 @@ enum btrfs_chunk_allocation_policy {
 enum btrfs_read_policy {
 	/* Use process PID to choose the stripe */
 	BTRFS_READ_POLICY_PID,
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+	/* Balancing raid1 reads across all striped devices (round-robin) */
+	BTRFS_READ_POLICY_RR,
+#endif
 	BTRFS_NR_READ_POLICY,
 };
 
@@ -436,6 +443,9 @@ struct btrfs_fs_devices {
 	enum btrfs_read_policy read_policy;
 
 #ifdef CONFIG_BTRFS_EXPERIMENTAL
+	/* Min contiguous reads before switching to next device. */
+	int rr_min_contiguous_read;
+
 	/* Checksum mode - offload it or do it synchronously. */
 	enum btrfs_offload_csum_mode offload_csum_mode;
 #endif
Signature · 1 line
2.47.0
author
#13 lore raw

PATCH 1/1 fixup: btrfs: introduce RAID1 round-robin read balancing

For now, let's keep the more typical storage device cache size at 256 KB, which can be updated once we have more widespread tests.

This patch should be merged with in the ML.
btrfs: introduce RAID1 round-robin read balancing

---
 fs/btrfs/volumes.h | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
fs/btrfs/volumes.h +1 -2
index 18521aebc484..48b38315672e 100644
@@ -296,8 +296,7 @@ enum btrfs_chunk_allocation_policy {
 	BTRFS_CHUNK_ALLOC_ZONED,
 };
 
-/* SZ_192K = 192 * 1024 = 196608 */
-#define BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ	(196608)
+#define BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ	(SZ_256K)
 #define BTRFS_RAID1_MAX_MIRRORS			(4)
 /*
  * Read policies for mirrored block group profiles, read picks the stripe based
Signature · 1 line
2.47.0
author
#14 lore raw

PATCH v5 7/10 btrfs: add RAID1 preferred read device

In reply to Anand Jain

When there's stale data on a mirrored device, this feature lets you choose which device to read from. Mainly used for testing.

echo "devid:<devid-value>" > /sys/fs/btrfs/<UUID>/read_policy

---
 fs/btrfs/sysfs.c   | 33 ++++++++++++++++++++++++++++++++-
 fs/btrfs/volumes.c | 21 +++++++++++++++++++++
 fs/btrfs/volumes.h |  5 +++++
 3 files changed, 58 insertions(+), 1 deletion(-)
fs/btrfs/sysfs.c +32 -1
index 70f89d1adfbc..eb23f29995e4 100644
@@ -1309,6 +1309,7 @@ static const char *btrfs_read_policy_name[] = {
 	"pid",
 #ifdef CONFIG_BTRFS_EXPERIMENTAL
 	"round-robin",
+	"devid",
 #endif
 };
 
@@ -1356,8 +1357,11 @@ static ssize_t btrfs_read_policy_show(struct kobject *kobj,
 		if (i == BTRFS_READ_POLICY_RR)
 			ret += sysfs_emit_at(buf, ret, ":%d",
 				 READ_ONCE(fs_devices->rr_min_contiguous_read));
-#endif
 
+		if (i == BTRFS_READ_POLICY_DEVID)
+			ret += sysfs_emit_at(buf, ret, ":%llu",
+					     READ_ONCE(fs_devices->read_devid));
+#endif
 		if (i == policy)
 			ret += sysfs_emit_at(buf, ret, "]");
 	}
@@ -1414,6 +1418,33 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj,
 
 		return len;
 	}
+
+	if (index == BTRFS_READ_POLICY_DEVID) {
+
+		if (value != -1) {
+			BTRFS_DEV_LOOKUP_ARGS(args);
+
+			/* Validate input devid */
+			args.devid = value;
+			if (btrfs_find_device(fs_devices, &args) == NULL)
+				return -EINVAL;
+		} else {
+			/* Set default devid to the devid of the latest device */
+			value = fs_devices->latest_dev->devid;
+		}
+
+		if (index != READ_ONCE(fs_devices->read_policy) ||
+		    (value != READ_ONCE(fs_devices->read_devid))) {
+			WRITE_ONCE(fs_devices->read_policy, index);
+			WRITE_ONCE(fs_devices->read_devid, value);
+
+			btrfs_info(fs_devices->fs_info, "read policy set to '%s:%llu'",
+				   btrfs_read_policy_name[index], value);
+
+		}
+
+		return len;
+	}
 #endif
 	if (index != READ_ONCE(fs_devices->read_policy)) {
 		WRITE_ONCE(fs_devices->read_policy, index);
fs/btrfs/volumes.c +21 -0
index ab2e970dd6bf..e8cccfad6ad3 100644
@@ -1336,6 +1336,7 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices,
 	fs_devices->read_policy = BTRFS_READ_POLICY_PID;
 #ifdef CONFIG_BTRFS_EXPERIMENTAL
 	fs_devices->rr_min_contiguous_read = BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ;
+	fs_devices->read_devid = latest_dev->devid;
 #endif
 
 	return 0;
@@ -5969,6 +5970,23 @@ unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info,
 }
 
 #ifdef CONFIG_BTRFS_EXPERIMENTAL
+static int btrfs_read_preferred(struct btrfs_chunk_map *map, int first,
+				int num_stripe)
+{
+	int last = first + num_stripe;
+	int stripe_index;
+
+	for (stripe_index = first; stripe_index < last; stripe_index++) {
+		struct btrfs_device *device = map->stripes[stripe_index].dev;
+
+		if (device->devid == READ_ONCE(device->fs_devices->read_devid))
+			return stripe_index;
+	}
+
+	/* If no read-preferred device, use first stripe */
+	return first;
+}
+
 struct stripe_mirror {
 	u64 devid;
 	int num;
@@ -6065,6 +6083,9 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,
 	case BTRFS_READ_POLICY_RR:
 		preferred_mirror = btrfs_read_rr(map, first, num_stripes);
 		break;
+	case BTRFS_READ_POLICY_DEVID:
+		preferred_mirror = btrfs_read_preferred(map, first, num_stripes);
+		break;
 #endif
 	}
 
fs/btrfs/volumes.h +5 -0
index 5728b8717317..18521aebc484 100644
@@ -309,6 +309,8 @@ enum btrfs_read_policy {
 #ifdef CONFIG_BTRFS_EXPERIMENTAL
 	/* Balancing raid1 reads across all striped devices (round-robin) */
 	BTRFS_READ_POLICY_RR,
+	/* Read from the specific device */
+	BTRFS_READ_POLICY_DEVID,
 #endif
 	BTRFS_NR_READ_POLICY,
 };
@@ -446,6 +448,9 @@ struct btrfs_fs_devices {
 	/* Min contiguous reads before switching to next device. */
 	int rr_min_contiguous_read;
 
+	/* Device to be used for reading in case of RAID1. */
+	u64 read_devid;
+
 	/* Checksum mode - offload it or do it synchronously. */
 	enum btrfs_offload_csum_mode offload_csum_mode;
 #endif
Signature · 1 line
2.47.0
author
#15 lore raw

PATCH v5 8/10 btrfs: expose experimental mode in module information

In reply to Anand Jain

Commit c9c49e8f157e ("btrfs: split out CONFIG_BTRFS_EXPERIMENTAL from CONFIG_BTRFS_DEBUG") introduces a way to enable or disable experimental features, print its status during module load, like so:

Btrfs loaded, experimental=on, debug=on, assert=on, zoned=yes, fsverity=yes

---
 fs/btrfs/super.c | 3 +++
 1 file changed, 3 insertions(+)
fs/btrfs/super.c +3 -0
index 3381389ab93a..fb6a009c72ae 100644
@@ -2457,6 +2457,9 @@ static __cold void btrfs_interface_exit(void)
 static int __init btrfs_print_mod_info(void)
 {
 	static const char options[] = ""
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+			", experimental=on"
+#endif
 #ifdef CONFIG_BTRFS_DEBUG
 			", debug=on"
 #endif
Signature · 1 line
2.47.0
author
#16 lore raw

PATCH v5 9/10 btrfs: enable RAID1 balancing configuration via modprobe parameter

In reply to Anand Jain

This update allows configuring the raid1-balancing methods using a modprobe parameter when experimental mode CONFIG_BTRFS_EXPERIMENTAL is enabled.

Examples:

- Set the RAID1 balancing method to round-robin with a custom min_contiguous_read of 4k:

  $ modprobe btrfs raid1-balancing=round-robin:4096

- Set the round-robin balancing method with the default min_contiguous_read:

  $ modprobe btrfs raid1-balancing=round-robin

- Set the devid balancing method, defaulting to the latest device:

  $ modprobe btrfs raid1-balancing=devid
---
 fs/btrfs/super.c   |  5 +++++
 fs/btrfs/sysfs.c   | 30 +++++++++++++++++++++++++++++-
 fs/btrfs/sysfs.h   |  5 +++++
 fs/btrfs/volumes.c | 14 +++++++++++++-
 4 files changed, 52 insertions(+), 2 deletions(-)
fs/btrfs/super.c +5 -0
index fb6a009c72ae..58190989a29d 100644
@@ -2538,6 +2538,11 @@ static const struct init_sequence mod_init_seq[] = {
 	}, {
 		.init_func = extent_map_init,
 		.exit_func = extent_map_exit,
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+	}, {
+		.init_func = btrfs_raid1_balancing_init,
+		.exit_func = NULL,
+#endif
 	}, {
 		.init_func = ordered_data_init,
 		.exit_func = ordered_data_exit,
fs/btrfs/sysfs.c +29 -1
index eb23f29995e4..ac1a32af2442 100644
@@ -1313,7 +1313,21 @@ static const char *btrfs_read_policy_name[] = {
 #endif
 };
 
-static int btrfs_read_policy_to_enum(const char *str, s64 *value)
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+/* Global module configuration parameters */
+static char *raid1_balancing;
+char *btrfs_get_raid1_balancing(void)
+{
+	return raid1_balancing;
+}
+
+/* Set perm 0, disable sys/module/btrfs/parameter/raid1_balancing interface */
+module_param(raid1_balancing, charp, 0);
+MODULE_PARM_DESC(raid1_balancing,
+"Global read policy; pid (default), round-robin[:min_contiguous_read], devid[[:devid]|[:latest-gen]|[:oldest-gen]]");
+#endif
+
+int btrfs_read_policy_to_enum(const char *str, s64 *value)
 {
 	char param[32] = {'\0'};
 	char *__maybe_unused value_str;
@@ -1336,6 +1350,20 @@ static int btrfs_read_policy_to_enum(const char *str, s64 *value)
 	return sysfs_match_string(btrfs_read_policy_name, param);
 }
 
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+int __init btrfs_raid1_balancing_init(void)
+{
+	s64 value;
+
+	if (btrfs_read_policy_to_enum(raid1_balancing, &value) == -EINVAL) {
+		btrfs_err(NULL, "Invalid raid1_balancing %s", raid1_balancing);
+		return -EINVAL;
+	}
+
+	return 0;
+}
+#endif
+
 static ssize_t btrfs_read_policy_show(struct kobject *kobj,
 				      struct kobj_attribute *a, char *buf)
 {
fs/btrfs/sysfs.h +5 -0
index e6a284c59809..e97d383b9ffc 100644
@@ -47,5 +47,10 @@ void btrfs_sysfs_del_qgroups(struct btrfs_fs_info *fs_info);
 int btrfs_sysfs_add_qgroups(struct btrfs_fs_info *fs_info);
 void btrfs_sysfs_del_one_qgroup(struct btrfs_fs_info *fs_info,
 				struct btrfs_qgroup *qgroup);
+int btrfs_read_policy_to_enum(const char *str, s64 *value);
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+int __init btrfs_raid1_balancing_init(void);
+char *btrfs_get_raid1_balancing(void);
+#endif
 
 #endif
fs/btrfs/volumes.c +13 -1
index e8cccfad6ad3..e5e9b33837b8 100644
@@ -1299,6 +1299,7 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices,
 	struct btrfs_device *device;
 	struct btrfs_device *latest_dev = NULL;
 	struct btrfs_device *tmp_device;
+	s64 __maybe_unused value = 0;
 	int ret = 0;
 
 	/* Initialize the in-memory record of filesystem read count */
@@ -1333,10 +1334,21 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices,
 	fs_devices->latest_dev = latest_dev;
 	fs_devices->total_rw_bytes = 0;
 	fs_devices->chunk_alloc_policy = BTRFS_CHUNK_ALLOC_REGULAR;
-	fs_devices->read_policy = BTRFS_READ_POLICY_PID;
 #ifdef CONFIG_BTRFS_EXPERIMENTAL
 	fs_devices->rr_min_contiguous_read = BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ;
 	fs_devices->read_devid = latest_dev->devid;
+	fs_devices->read_policy =
+		btrfs_read_policy_to_enum(btrfs_get_raid1_balancing(), &value);
+	if (fs_devices->read_policy == BTRFS_READ_POLICY_RR)
+		fs_devices->fs_stats = true;
+	if (value) {
+		if (fs_devices->read_policy == BTRFS_READ_POLICY_RR)
+			fs_devices->rr_min_contiguous_read = value;
+		if (fs_devices->read_policy == BTRFS_READ_POLICY_DEVID)
+			fs_devices->read_devid = value;
+	}
+#else
+	fs_devices->read_policy = BTRFS_READ_POLICY_PID;
 #endif
 
 	return 0;
Signature · 1 line
2.47.0
#17 lore raw

Re: [PATCH v5 09/10] btrfs: enable RAID1 balancing configuration via modprobe parameter

Hi Anand!

Am Mi., 1. Jan. 2025 um 19:10 Uhr schrieb Anand Jain <[email protected]>:

Quoted 54 lines

This update allows configuring the raid1-balancing methods using a modprobe parameter when experimental mode CONFIG_BTRFS_EXPERIMENTAL is enabled.

Examples:

- Set the RAID1 balancing method to round-robin with a custom min_contiguous_read of 4k:

  $ modprobe btrfs raid1-balancing=round-robin:4096

- Set the round-robin balancing method with the default min_contiguous_read:

  $ modprobe btrfs raid1-balancing=round-robin

- Set the devid balancing method, defaulting to the latest device:

  $ modprobe btrfs raid1-balancing=devid
---
 fs/btrfs/super.c   |  5 +++++
 fs/btrfs/sysfs.c   | 30 +++++++++++++++++++++++++++++-
 fs/btrfs/sysfs.h   |  5 +++++
 fs/btrfs/volumes.c | 14 +++++++++++++-
 4 files changed, 52 insertions(+), 2 deletions(-)
fs/btrfs/super.c +5 -0
index fb6a009c72ae..58190989a29d 100644
@@ -2538,6 +2538,11 @@ static const struct init_sequence mod_init_seq[] = {
        }, {
                .init_func = extent_map_init,
                .exit_func = extent_map_exit,
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+       }, {
+               .init_func = btrfs_raid1_balancing_init,
+               .exit_func = NULL,
+#endif
        }, {
                .init_func = ordered_data_init,
                .exit_func = ordered_data_exit,
fs/btrfs/sysfs.c +2 -1
index eb23f29995e4..ac1a32af2442 100644
@@ -1313,7 +1313,21 @@ static const char *btrfs_read_policy_name[] = {
 #endif
 };
 
-static int btrfs_read_policy_to_enum(const char *str, s64 *value)
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+/* Global module configuration parameters */

fs/btrfs/sysfs.c static const char *btrfs_read_policy_name[] = { · line 1327

Quoted 10 lines
fs/btrfs/sysfs.c +10 -0
+static char *raid1_balancing;
+char *btrfs_get_raid1_balancing(void)
+{
+       return raid1_balancing;
+}
+
+/* Set perm 0, disable sys/module/btrfs/parameter/raid1_balancing interface */
+module_param(raid1_balancing, charp, 0);
+MODULE_PARM_DESC(raid1_balancing,
+"Global read policy; pid (default), round-robin[:min_contiguous_read], devid[[:devid]|[:latest-gen]|[:oldest-gen]]");

I don't see where "[:latest-gen]|[:oldest-gen]" would be used in the code.

Quoted 78 lines
fs/btrfs/sysfs.c +17 -0
+#endif
+
+int btrfs_read_policy_to_enum(const char *str, s64 *value)
 {
        char param[32] = {'\0'};
        char *__maybe_unused value_str;
@@ -1336,6 +1350,20 @@ static int btrfs_read_policy_to_enum(const char *str, s64 *value)
        return sysfs_match_string(btrfs_read_policy_name, param);
 }
 
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+int __init btrfs_raid1_balancing_init(void)
+{
+       s64 value;
+
+       if (btrfs_read_policy_to_enum(raid1_balancing, &value) == -EINVAL) {
+               btrfs_err(NULL, "Invalid raid1_balancing %s", raid1_balancing);
+               return -EINVAL;
+       }
+
+       return 0;
+}
+#endif
+
 static ssize_t btrfs_read_policy_show(struct kobject *kobj,
                                      struct kobj_attribute *a, char *buf)
 {
fs/btrfs/sysfs.h +5 -0
index e6a284c59809..e97d383b9ffc 100644
@@ -47,5 +47,10 @@ void btrfs_sysfs_del_qgroups(struct btrfs_fs_info *fs_info);
 int btrfs_sysfs_add_qgroups(struct btrfs_fs_info *fs_info);
 void btrfs_sysfs_del_one_qgroup(struct btrfs_fs_info *fs_info,
                                struct btrfs_qgroup *qgroup);
+int btrfs_read_policy_to_enum(const char *str, s64 *value);
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+int __init btrfs_raid1_balancing_init(void);
+char *btrfs_get_raid1_balancing(void);
+#endif
 
 #endif
fs/btrfs/volumes.c +13 -2
index e8cccfad6ad3..e5e9b33837b8 100644
@@ -1299,6 +1299,7 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices,
        struct btrfs_device *device;
        struct btrfs_device *latest_dev = NULL;
        struct btrfs_device *tmp_device;
+       s64 __maybe_unused value = 0;
        int ret = 0;
 
        /* Initialize the in-memory record of filesystem read count */
@@ -1333,10 +1334,21 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices,
        fs_devices->latest_dev = latest_dev;
        fs_devices->total_rw_bytes = 0;
        fs_devices->chunk_alloc_policy = BTRFS_CHUNK_ALLOC_REGULAR;
-       fs_devices->read_policy = BTRFS_READ_POLICY_PID;
 #ifdef CONFIG_BTRFS_EXPERIMENTAL
        fs_devices->rr_min_contiguous_read = BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ;
        fs_devices->read_devid = latest_dev->devid;
+       fs_devices->read_policy =
+               btrfs_read_policy_to_enum(btrfs_get_raid1_balancing(), &value);
+       if (fs_devices->read_policy == BTRFS_READ_POLICY_RR)
+               fs_devices->fs_stats = true;
+       if (value) {
+               if (fs_devices->read_policy == BTRFS_READ_POLICY_RR)
+                       fs_devices->rr_min_contiguous_read = value;
+               if (fs_devices->read_policy == BTRFS_READ_POLICY_DEVID)
+                       fs_devices->read_devid = value;
+       }
+#else
+       fs_devices->read_policy = BTRFS_READ_POLICY_PID;
 #endif
 
        return 0;
--
Quoted 1 line

2.47.0

Thanks
Kai

#18 lore raw

Re: [PATCH v5 09/10] btrfs: enable RAID1 balancing configuration via modprobe parameter

Hello!

Am Mi., 9. Apr. 2025 um 18:59 Uhr schrieb Kai Krakow <[email protected]>:

Quoted 55 lines

Hi Anand!

Am Mi., 1. Jan. 2025 um 19:10 Uhr schrieb Anand Jain <[email protected]>:

Quoted 51 lines

This update allows configuring the raid1-balancing methods using a modprobe parameter when experimental mode CONFIG_BTRFS_EXPERIMENTAL is enabled.

Examples:

- Set the RAID1 balancing method to round-robin with a custom min_contiguous_read of 4k:

  $ modprobe btrfs raid1-balancing=round-robin:4096

- Set the round-robin balancing method with the default min_contiguous_read:

  $ modprobe btrfs raid1-balancing=round-robin

- Set the devid balancing method, defaulting to the latest device:

  $ modprobe btrfs raid1-balancing=devid
---
 fs/btrfs/super.c   |  5 +++++
 fs/btrfs/sysfs.c   | 30 +++++++++++++++++++++++++++++-
 fs/btrfs/sysfs.h   |  5 +++++
 fs/btrfs/volumes.c | 14 +++++++++++++-
 4 files changed, 52 insertions(+), 2 deletions(-)
fs/btrfs/super.c +5 -0
index fb6a009c72ae..58190989a29d 100644
@@ -2538,6 +2538,11 @@ static const struct init_sequence mod_init_seq[] = {
        }, {
                .init_func = extent_map_init,
                .exit_func = extent_map_exit,
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+       }, {
+               .init_func = btrfs_raid1_balancing_init,
+               .exit_func = NULL,
+#endif
        }, {
                .init_func = ordered_data_init,
                .exit_func = ordered_data_exit,
fs/btrfs/sysfs.c +0 -0
index eb23f29995e4..ac1a32af2442 100644
@@ -1313,7 +1313,21 @@ static const char *btrfs_read_policy_name[] = {
 #endif
 };
 
Quoted 15 lines

fs/btrfs/sysfs.c static const char *btrfs_read_policy_name[] = { · line 1327

Quoted 13 lines
fs/btrfs/sysfs.c +12 -1
-static int btrfs_read_policy_to_enum(const char *str, s64 *value)
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+/* Global module configuration parameters */
+static char *raid1_balancing;
+char *btrfs_get_raid1_balancing(void)
+{
+       return raid1_balancing;
+}
+
+/* Set perm 0, disable sys/module/btrfs/parameter/raid1_balancing interface */
+module_param(raid1_balancing, charp, 0);
+MODULE_PARM_DESC(raid1_balancing,
+"Global read policy; pid (default), round-robin[:min_contiguous_read], devid[[:devid]|[:latest-gen]|[:oldest-gen]]");

I don't see where "[:latest-gen]|[:oldest-gen]" would be used in the code.

I just realized this has been fixed in the final merged patchset. Sorry for the noise.

Quoted 82 lines
Quoted 78 lines
fs/btrfs/sysfs.c +17 -0
+#endif
+
+int btrfs_read_policy_to_enum(const char *str, s64 *value)
 {
        char param[32] = {'\0'};
        char *__maybe_unused value_str;
@@ -1336,6 +1350,20 @@ static int btrfs_read_policy_to_enum(const char *str, s64 *value)
        return sysfs_match_string(btrfs_read_policy_name, param);
 }
 
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+int __init btrfs_raid1_balancing_init(void)
+{
+       s64 value;
+
+       if (btrfs_read_policy_to_enum(raid1_balancing, &value) == -EINVAL) {
+               btrfs_err(NULL, "Invalid raid1_balancing %s", raid1_balancing);
+               return -EINVAL;
+       }
+
+       return 0;
+}
+#endif
+
 static ssize_t btrfs_read_policy_show(struct kobject *kobj,
                                      struct kobj_attribute *a, char *buf)
 {
fs/btrfs/sysfs.h +5 -0
index e6a284c59809..e97d383b9ffc 100644
@@ -47,5 +47,10 @@ void btrfs_sysfs_del_qgroups(struct btrfs_fs_info *fs_info);
 int btrfs_sysfs_add_qgroups(struct btrfs_fs_info *fs_info);
 void btrfs_sysfs_del_one_qgroup(struct btrfs_fs_info *fs_info,
                                struct btrfs_qgroup *qgroup);
+int btrfs_read_policy_to_enum(const char *str, s64 *value);
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+int __init btrfs_raid1_balancing_init(void);
+char *btrfs_get_raid1_balancing(void);
+#endif
 
 #endif
fs/btrfs/volumes.c +13 -2
index e8cccfad6ad3..e5e9b33837b8 100644
@@ -1299,6 +1299,7 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices,
        struct btrfs_device *device;
        struct btrfs_device *latest_dev = NULL;
        struct btrfs_device *tmp_device;
+       s64 __maybe_unused value = 0;
        int ret = 0;
 
        /* Initialize the in-memory record of filesystem read count */
@@ -1333,10 +1334,21 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices,
        fs_devices->latest_dev = latest_dev;
        fs_devices->total_rw_bytes = 0;
        fs_devices->chunk_alloc_policy = BTRFS_CHUNK_ALLOC_REGULAR;
-       fs_devices->read_policy = BTRFS_READ_POLICY_PID;
 #ifdef CONFIG_BTRFS_EXPERIMENTAL
        fs_devices->rr_min_contiguous_read = BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ;
        fs_devices->read_devid = latest_dev->devid;
+       fs_devices->read_policy =
+               btrfs_read_policy_to_enum(btrfs_get_raid1_balancing(), &value);
+       if (fs_devices->read_policy == BTRFS_READ_POLICY_RR)
+               fs_devices->fs_stats = true;
+       if (value) {
+               if (fs_devices->read_policy == BTRFS_READ_POLICY_RR)
+                       fs_devices->rr_min_contiguous_read = value;
+               if (fs_devices->read_policy == BTRFS_READ_POLICY_DEVID)
+                       fs_devices->read_devid = value;
+       }
+#else
+       fs_devices->read_policy = BTRFS_READ_POLICY_PID;
 #endif
 
        return 0;
--
Quoted 1 line

2.47.0

Thanks
Kai

author
#19 lore raw

PATCH v5 10/10 btrfs: modload to print RAID1 balancing status

In reply to Anand Jain

Modified the Btrfs loading message to include the RAID1 balancing status if the experimental feature is enabled.

---
 fs/btrfs/super.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
fs/btrfs/super.c +10 -0
index 58190989a29d..236eec7c19cf 100644
@@ -2480,7 +2480,17 @@ static int __init btrfs_print_mod_info(void)
 			", fsverity=no"
 #endif
 			;
+
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+	if (btrfs_get_raid1_balancing() == NULL)
+		pr_info("Btrfs loaded%s\n", options);
+	else
+		pr_info("Btrfs loaded%s, raid1_balancing=%s\n",
+			 options, btrfs_get_raid1_balancing());
+#else
 	pr_info("Btrfs loaded%s\n", options);
+#endif
+
 	return 0;
 }
 
Signature · 1 line
2.47.0
#20 lore raw
In reply to Anand Jain
Quoted 8 lines · Anand Jain

v5:
Fixes based on review comments:
. Rewrite btrfs_read_policy_to_enum() using sysfs_match_string()

    and `strncpy()`.

. Rewrite the round-robin method based on read counts.
. Fix the smatch indentation warning.
- Change the default minimum contiguous device read size for round-robin from 256K to 192K, as the latter performs slightly better.

This depends on the device and load and the read pattern so any number is fine, I'd rather stick to something that looks sensible if the difference is slight. Changing that later based on extensive benchmarks.

Quoted 14 lines

- Introduce a framework to track filesystem read counts. (New patch)
- Reran defrag performance numbers

      $ xfs_io -f -d -c 'pwrite -S 0xab 0 1000000000' /btrfs/P6B
      $ time -p btrfs filesystem defrag -r -f -c /btrfs
|         | Time  | Read I/O Count  | gain  |
|         | Real  | devid1 | devid2 | w-PID |
|---------|-------|--------|--------|-------|
| pid     | 11.14s| 3808   | 0      |  -    |
| rotation|       |        |        |       |
|   196608|  6.54s| 2532   | 1276   | 41.29%|
|   262144|  8.42s| 1907   | 1901   | 24.41%|
| devid:1 | 10.95s| 3807   | 0      | 1.70% |
Quoted 1 line

v4:

As we're at rc5 we need to get this series to for-next soon, before rc6 is released this Sunday. I'll do one review pass and then move the patches to for-next. I'm not convinced we should do the module parameter but as this is to ease testing let's do it.

author
#21 lore raw
Quoted 10 lines · David Sterba
Quoted 8 lines · Anand Jain

v5:
Fixes based on review comments:
. Rewrite btrfs_read_policy_to_enum() using sysfs_match_string()

     and `strncpy()`.

. Rewrite the round-robin method based on read counts.
. Fix the smatch indentation warning.
- Change the default minimum contiguous device read size for round-robin from 256K to 192K, as the latter performs slightly better.

Quoted 4 lines

This depends on the device and load and the read pattern so any number is fine, I'd rather stick to something that looks sensible if the difference is slight. Changing that later based on extensive benchmarks.

The optimization should target the generic read-write workload, where I find 256k or more reasonable, though I am unable to demonstrate this at the moment. It's a good idea to gather more feedback. I've sent a fix with SZ_256K and for a missed bug, hoping both are folded into their patches. Thx.

Quoted 14 lines
Quoted 14 lines

- Introduce a framework to track filesystem read counts. (New patch)
- Reran defrag performance numbers

       $ xfs_io -f -d -c 'pwrite -S 0xab 0 1000000000' /btrfs/P6B
       $ time -p btrfs filesystem defrag -r -f -c /btrfs
|         | Time  | Read I/O Count  | gain  |
|         | Real  | devid1 | devid2 | w-PID |
|---------|-------|--------|--------|-------|
| pid     | 11.14s| 3808   | 0      |  -    |
| rotation|       |        |        |       |
|   196608|  6.54s| 2532   | 1276   | 41.29%|
|   262144|  8.42s| 1907   | 1901   | 24.41%|
| devid:1 | 10.95s| 3807   | 0      | 1.70% |
Quoted 6 lines
Quoted 1 line

v4:

As we're at rc5 we need to get this series to for-next soon, before rc6 is released this Sunday. I'll do one review pass and then move the patches to for-next. I'm not convinced we should do the module parameter but as this is to ease testing let's do it.

Thanks, Anand

#22 lore raw
Quoted 16 lines · Anand Jain
Quoted 10 lines · David Sterba
Quoted 8 lines · Anand Jain

v5:
Fixes based on review comments:
. Rewrite btrfs_read_policy_to_enum() using sysfs_match_string()

     and `strncpy()`.

. Rewrite the round-robin method based on read counts.
. Fix the smatch indentation warning.
- Change the default minimum contiguous device read size for round-robin from 256K to 192K, as the latter performs slightly better.

Quoted 4 lines

This depends on the device and load and the read pattern so any number is fine, I'd rather stick to something that looks sensible if the difference is slight. Changing that later based on extensive benchmarks.

Quoted 5 lines

The optimization should target the generic read-write workload, where I find 256k or more reasonable, though I am unable to demonstrate this at the moment. It's a good idea to gather more feedback. I've sent a fix with SZ_256K and for a missed bug, hoping both are folded into their patches. Thx.

As discussed, the min contig size will be tuned. Per current testing the balancing seems to work so we can keep it as is for now.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

- Introduce a framework to track filesystem read counts. (New patch)
- Reran defrag performance numbers

       $ xfs_io -f -d -c 'pwrite -S 0xab 0 1000000000' /btrfs/P6B
       $ time -p btrfs filesystem defrag -r -f -c /btrfs
|         | Time  | Read I/O Count  | gain  |
|         | Real  | devid1 | devid2 | w-PID |
|---------|-------|--------|--------|-------|
| pid     | 11.14s| 3808   | 0      |  -    |
| rotation|       |        |        |       |
|   196608|  6.54s| 2532   | 1276   | 41.29%|
|   262144|  8.42s| 1907   | 1901   | 24.41%|
| devid:1 | 10.95s| 3807   | 0      | 1.70% |
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

v4:

As we're at rc5 we need to get this series to for-next soon, before rc6 is released this Sunday. I'll do one review pass and then move the patches to for-next. I'm not convinced we should do the module parameter but as this is to ease testing let's do it.

The testing delayed it but now the series is in for-next, with some cleanups and fixups I did. The remaining thing is to accept suffixed sizes as sysfs (ie. use memparse), I did not implement it as the feature is usable without that and we need to get it merged now.

author
#23 lore raw
Quoted 47 lines · David Sterba
Quoted 16 lines · Anand Jain
Quoted 10 lines · David Sterba
Quoted 8 lines · Anand Jain

v5:
Fixes based on review comments:
. Rewrite btrfs_read_policy_to_enum() using sysfs_match_string()
and strncpy().
. Rewrite the round-robin method based on read counts.
. Fix the smatch indentation warning.
- Change the default minimum contiguous device read size for round-robin from 256K to 192K, as the latter performs slightly better.

Quoted 4 lines

This depends on the device and load and the read pattern so any number is fine, I'd rather stick to something that looks sensible if the difference is slight. Changing that later based on extensive benchmarks.

Quoted 5 lines

The optimization should target the generic read-write workload, where I find 256k or more reasonable, though I am unable to demonstrate this at the moment. It's a good idea to gather more feedback. I've sent a fix with SZ_256K and for a missed bug, hoping both are folded into their patches. Thx.

As discussed, the min contig size will be tuned. Per current testing the balancing seems to work so we can keep it as is for now.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

- Introduce a framework to track filesystem read counts. (New patch)
- Reran defrag performance numbers

        $ xfs_io -f -d -c 'pwrite -S 0xab 0 1000000000' /btrfs/P6B
        $ time -p btrfs filesystem defrag -r -f -c /btrfs
|         | Time  | Read I/O Count  | gain  |
|         | Real  | devid1 | devid2 | w-PID |
|---------|-------|--------|--------|-------|
| pid     | 11.14s| 3808   | 0      |  -    |
| rotation|       |        |        |       |
|   196608|  6.54s| 2532   | 1276   | 41.29%|
|   262144|  8.42s| 1907   | 1901   | 24.41%|
| devid:1 | 10.95s| 3807   | 0      | 1.70% |
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

v4:

As we're at rc5 we need to get this series to for-next soon, before rc6 is released this Sunday. I'll do one review pass and then move the patches to for-next. I'm not convinced we should do the module parameter but as this is to ease testing let's do it.

Quoted 4 lines

The testing delayed it but now the series is in for-next, with some cleanups and fixups I did. The remaining thing is to accept suffixed sizes as sysfs (ie. use memparse), I did not implement it as the feature is usable without that and we need to get it merged now.

The for-next patches match misc-next, reviewed, fixed, and ready to go.

Thanks!
Anand

#24 lore raw
In reply to Anand Jain

Hi Anand!

Am Mi., 1. Jan. 2025 um 19:10 Uhr schrieb Anand Jain <[email protected]>:

Quoted 206 lines

v5:
Fixes based on review comments:
. Rewrite btrfs_read_policy_to_enum() using sysfs_match_string()

    and `strncpy()`.
  . Rewrite the round-robin method based on read counts.
  . Fix the smatch indentation warning.
 - Change the default minimum contiguous device read size for round-robin
   from 256K to 192K, as the latter performs slightly better.
 - Introduce a framework to track filesystem read counts. (New patch)
 - Reran defrag performance numbers
      $ xfs_io -f -d -c 'pwrite -S 0xab 0 1000000000' /btrfs/P6B
      $ time -p btrfs filesystem defrag -r -f -c /btrfs
|         | Time  | Read I/O Count  | gain  |
|         | Real  | devid1 | devid2 | w-PID |
|---------|-------|--------|--------|-------|
| pid     | 11.14s| 3808   | 0      |  -    |
| rotation|       |        |        |       |
|   196608|  6.54s| 2532   | 1276   | 41.29%|
|   262144|  8.42s| 1907   | 1901   | 24.41%|
| devid:1 | 10.95s| 3807   | 0      | 1.70% |

v4:
Fixes based on review comments:
3/10: Use == 0 to check strlen(str); drop dynamic alloc for %param.
4/10: Add __maybe_unused for %value_str in btrfs_read_policy_to_enum().
Return int instead of enum btrfs_read_policy.
5/10: Fix change-log (: is part of optional [ ]).
Wrap btrfs_read_policy_name[] with ifdef for new methods.
Use IS_ALIGNED() for sector-size alignment check.
Roundup misaligned %value.
Use named constants: BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ, BTRFS_RAID1_MAX_MIRRORS. Mark %s1 and %s2 in btrfs_cmp_devid() as const.
Add comments to btrfs_read_rr();
Use loop-local %i. Add space around /.
Use >> for sector-size division.
Prefix %min_contiguous_read with rr.
7/10: Move experimental to the top of the feature list.
Use experiment=on. Skip printing when off.

v3:
1. Removed the latency-based RAID1 balancing patch. (Per David's review)
2. Renamed "rotation" to "round-robin" and set the per-set min_contiguous_read to 256k. (Per David's review)
3. Added raid1-balancing module configuration for fstests testing. raid1-balancing can now be configured through both module load parameters and sysfs.

The logic of individual methods remains unchanged, and performance metrics are consistent with v2.

-----
v2:
1. Move new features to CONFIG_BTRFS_EXPERIMENTAL instead of CONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait to U64_MAX and remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:

echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy

The default value is the sector size, and the min_contiguous_read value must be a multiple of the sector size.

5. Tested FIO random read/write and defrag compression workloads with min_contiguous_read set to sector size, 192k, and 256k.

   RAID1 balancing method rotation is better for multi-process workloads
   such as fio and also single-process workload such as defragmentation.

     $ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
        --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
        --time_based --group_reporting --name=iops-test-job --eta-newline=1
|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 20.3MiB/s  | 20.5MiB/s  | 313895 | 313895 |
| rotation|            |            |        |        |
|     4096| 20.4MiB/s  | 20.5MiB/s  | 313895 | 313895 |
|   196608| 20.2MiB/s  | 20.2MiB/s  | 310152 | 310175 |
|   262144| 20.3MiB/s  | 20.4MiB/s  | 312180 | 312191 |
|  latency| 18.4MiB/s  | 18.4MiB/s  | 272980 | 291683 |
| devid:1 | 14.8MiB/s  | 14.9MiB/s  | 456376 | 0      |

rotation RAID1 balancing technique performs more than 2x better for single-process defrag.

      $ time -p btrfs filesystem defrag -r -f -c /btrfs
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 18.00s| 3800   | 0      |
| rotation|       |        |        |
|     4096|  8.95s| 1900   | 1901   |
|   196608|  8.50s| 1881   | 1919   |
|   262144|  8.80s| 1881   | 1919   |
| latency | 17.18s| 3800   | 0      |
| devid:2 | 17.48s| 0      | 3800   |

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.

Latency is smarter with unstable block layer transport.

Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.

Devid is a hands-on approach, provides manual or user-space script control.

These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.

Thx.

--------- original v1 ------------

The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the CONFIG_BTRFS_DEBUG config option and are on top of the previously added /sys/fs/btrfs/<UUID>/read_policy interface to configure the desired RAID1 read balancing method.

I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from /sys/devices/<..>/stat for each device. Below is a summary of the results, with each result the average of three iterations.

A typical generic random rw workload:

$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \
  --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \
  --group_reporting --name=iops-test-job --eta-newline=1
|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 29.4MiB/s  | 29.5MiB/s  | 456548 | 447975 |
| rotation| 29.3MiB/s  | 29.3MiB/s  | 450105 | 450055 |
| latency | 21.9MiB/s  | 21.9MiB/s  | 672387 | 0      |
| devid:1 | 22.0MiB/s  | 22.0MiB/s  | 674788 | 0      |

Defragmentation with compression workload:

$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo
$ sync
$ echo 3 > /proc/sys/vm/drop_caches
$ btrfs filesystem defrag -f -c /btrfs/foo
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 21.61s| 3810   | 0      |
| rotation| 11.55s| 1905   | 1905   |
| latency | 20.99s| 0      | 3810   |
| devid:2 | 21.41s| 0      | 3810   |

. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.

Anand Jain (10):
btrfs: initialize fs_devices->fs_info earlier
btrfs: simplify output formatting in btrfs_read_policy_show
btrfs: add btrfs_read_policy_to_enum helper and refactor read policy
store
btrfs: handle value associated with raid1 balancing parameter
btrfs: add read count tracking for filesystem stats
btrfs: introduce RAID1 round-robin read balancing
btrfs: add RAID1 preferred read device
btrfs: expose experimental mode in module information
btrfs: enable RAID1 balancing configuration via modprobe parameter
btrfs: modload to print RAID1 balancing status

 fs/btrfs/bio.c     |   8 +++
 fs/btrfs/disk-io.c |   4 ++
 fs/btrfs/super.c   |  18 +++++
 fs/btrfs/sysfs.c   | 168 ++++++++++++++++++++++++++++++++++++++++-----
 fs/btrfs/sysfs.h   |   5 ++
 fs/btrfs/volumes.c | 115 ++++++++++++++++++++++++++++++-
 fs/btrfs/volumes.h |  22 +++++-
 7 files changed, 317 insertions(+), 23 deletions(-)
Quoted 2 lines

--
2.47.0

I've backported your patches to my LTS kernel to experiment with it a bit. I've also included the latency-based policy for that reason. Here's what I currently got:
https://github.com/kakra/linux/pull/36

I've added an idea to use a hybrid policy which combines round-robin and latency into one combined policy. But it seems like over time, it will just acts like round-robin. This is probably because the average latency is calculated over the full history of requests. I think using an EMA (exponential moving average) with an alpha of 1/8 or 1/16 could work better. But this would require to sample each bio individually and add fields to the device structs. Something like this:

s64 current_ema = atomic64_read(&device->avg_latency_ema);
s64 difference = (s64)current_latency - current_ema; // important:
signed difference!
s64 new_ema = current_ema + (difference >> N); // N = 3 or 4 (alpha 1/8 or 1/16) atomic64_cmpxchg(&device->avg_latency_ema, current_ema, new_ema);

What do you think?

I'm currently not posting my patch series here because

(a) it's based/backported on LTS
(b) I'm not yet very familiar with doing that on the mailing list
(c) the code is not ready and contains some duplication

I hope that's okay.

So far, both the latency and round-robin policies work well in my setup. The latency policy actually filters out a low-end desktop HDD which is quite slow (according to fio) for individual short requests and long sequential reading (acts more like 5400rpm in that case) but is actually not that bad for typical random IO with larger block sizes (actually competes well with my other 7200rpm disks). Combined with bcache, this automatically optimizes the fastest disks into the cache, avoiding the slower ones when using the latency policy.

Overall, loading times in some applications with random but massive IO thus seem faster with the latency policy while sequential IO is faster with the round-robin policy.

I think the hybrid approach could fit both scenarios but it currently suffers from not adapting well to changes in IO patterns - which EMA could solve.

Thanks
Kai

author
#25 lore raw

Hi Kai,

Quoted 13 lines

I've backported your patches to my LTS kernel to experiment with it a bit. I've also included the latency-based policy for that reason. Here's what I currently got:
https://github.com/kakra/linux/pull/36

I've added an idea to use a hybrid policy which combines round-robin and latency into one combined policy. But it seems like over time, it will just acts like round-robin. This is probably because the average latency is calculated over the full history of requests. I think using an EMA (exponential moving average) with an alpha of 1/8 or 1/16 could work better. But this would require to sample each bio individually and add fields to the device structs. Something like this:

Quoted 8 lines

s64 current_ema = atomic64_read(&device->avg_latency_ema);
s64 difference = (s64)current_latency - current_ema; // important:
signed difference!
s64 new_ema = current_ema + (difference >> N); // N = 3 or 4 (alpha 1/8 or 1/16) atomic64_cmpxchg(&device->avg_latency_ema, current_ema, new_ema);

What do you think?

You can assume both devices are of the same type; otherwise, the logic gets sidetracked, and some performance numbers may vary. I couldn’t figure out how are you calculating EMA in the kernel or how well it prioritizes current I/O latency over historical values (which is a problem should be avoided), or if it gradually fades out the impact of device's stale latency numbers.

Quoted 17 lines

I'm currently not posting my patch series here because

(a) it's based/backported on LTS
(b) I'm not yet very familiar with doing that on the mailing list
(c) the code is not ready and contains some duplication

I hope that's okay.

So far, both the latency and round-robin policies work well in my setup. The latency policy actually filters out a low-end desktop HDD which is quite slow (according to fio) for individual short requests and long sequential reading (acts more like 5400rpm in that case) but is actually not that bad for typical random IO with larger block sizes (actually competes well with my other 7200rpm disks). Combined with bcache, this automatically optimizes the fastest disks into the cache, avoiding the slower ones when using the latency policy.

Quoted 7 lines

Overall, loading times in some applications with random but massive IO thus seem faster with the latency policy while sequential IO is faster with the round-robin policy.

I think the hybrid approach could fit both scenarios but it currently suffers from not adapting well to changes in IO patterns - which EMA could solve.

To try it out better, you can take the latency code from this series in v2 and add EMA to it. For now, keep both devices of the same type. Test it with a defrag workload. To make it more interesting, add some noise to the blk-layer transport of one device to increase its latency, and see if the algorithm dynamically switches to the other device. Then compare it with round-robin. That algorithm will need fine-tuning based on how adaptable it should be to intermittent block layer bottlenecks

Thanks, Anand