Skip to contentSkip to conversation

raid1 balancing methods

PATCH Superseded Anand Jain 5 participants 15 messages Oct 11, 2024 – Oct 24, 2024 Btrfs file system

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Anand Jain

author
#0 lore raw

v2 cover raid1 balancing methods

v2:
1. Move new features to CONFIG_BTRFS_EXPERIMENTAL instead of CONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait to U64_MAX and remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:

echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy

The default value is the sector size, and the min_contiguous_read value must be a multiple of the sector size.

5. Tested FIO random read/write and defrag compression workloads with min_contiguous_read set to sector size, 192k, and 256k.

RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.

     $ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
        --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
        --time_based --group_reporting --name=iops-test-job --eta-newline=1
|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 20.3MiB/s  | 20.5MiB/s  | 313895 | 313895 |
| rotation|            |            |        |        |
|     4096| 20.4MiB/s  | 20.5MiB/s  | 313895 | 313895 |
|   196608| 20.2MiB/s  | 20.2MiB/s  | 310152 | 310175 |
|   262144| 20.3MiB/s  | 20.4MiB/s  | 312180 | 312191 |
|  latency| 18.4MiB/s  | 18.4MiB/s  | 272980 | 291683 |
| devid:1 | 14.8MiB/s  | 14.9MiB/s  | 456376 | 0      |

rotation RAID1 balancing technique performs more than 2x better for single-process defrag.

      $ time -p btrfs filesystem defrag -r -f -c /btrfs
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 18.00s| 3800   | 0      |
| rotation|       |        |        |
|     4096|  8.95s| 1900   | 1901   |
|   196608|  8.50s| 1881   | 1919   |
|   262144|  8.80s| 1881   | 1919   |
| latency | 17.18s| 3800   | 0      |
| devid:2 | 17.48s| 0      | 3800   |

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.

Latency is smarter with unstable block layer transport.

Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.

Devid is a hands-on approach, provides manual or user-space script control.

These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.

Thx.

--------- original v1 ------------

The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the CONFIG_BTRFS_DEBUG config option and are on top of the previously added /sys/fs/btrfs/<UUID>/read_policy interface to configure the desired RAID1 read balancing method.

I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from /sys/devices/<..>/stat for each device. Below is a summary of the results, with each result the average of three iterations.

A typical generic random rw workload:

$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \
  --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \
  --group_reporting --name=iops-test-job --eta-newline=1
|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 29.4MiB/s  | 29.5MiB/s  | 456548 | 447975 |
| rotation| 29.3MiB/s  | 29.3MiB/s  | 450105 | 450055 |
| latency | 21.9MiB/s  | 21.9MiB/s  | 672387 | 0      |
| devid:1 | 22.0MiB/s  | 22.0MiB/s  | 674788 | 0      |

Defragmentation with compression workload:

$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo
$ sync
$ echo 3 > /proc/sys/vm/drop_caches
$ btrfs filesystem defrag -f -c /btrfs/foo
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 21.61s| 3810   | 0      |
| rotation| 11.55s| 1905   | 1905   |
| latency | 20.99s| 0      | 3810   |
| devid:2 | 21.41s| 0      | 3810   |

. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.

Anand Jain (3):
btrfs: introduce RAID1 round-robin read balancing
btrfs: use the path with the lowest latency for RAID1 reads
btrfs: add RAID1 preferred read device

 fs/btrfs/disk-io.c |   4 ++
 fs/btrfs/sysfs.c   | 116 +++++++++++++++++++++++++++++++++++++++------
 fs/btrfs/volumes.c | 109 ++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/volumes.h |  16 +++++++
 4 files changed, 230 insertions(+), 15 deletions(-)
Signature · 1 line
2.46.1
author
#1 lore raw

PATCH v2 1/3 btrfs: introduce RAID1 round-robin read balancing

This feature balances I/O across the striped devices when reading from RAID1 blocks.

echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy

Default value of min_contiguous_read is equal to the sectorsize.

---
 fs/btrfs/disk-io.c |  3 ++
 fs/btrfs/sysfs.c   | 88 ++++++++++++++++++++++++++++++++++++++--------
 fs/btrfs/volumes.c | 53 ++++++++++++++++++++++++++++
 fs/btrfs/volumes.h |  9 +++++
 4 files changed, 138 insertions(+), 15 deletions(-)
fs/btrfs/disk-io.c +3 -0
index 4ad5db619b00..5b157f407e0a 100644
@@ -3320,6 +3320,9 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device
 
 	fs_info->nodesize = nodesize;
 	fs_info->sectorsize = sectorsize;
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+	fs_info->fs_devices->min_contiguous_read = sectorsize;
+#endif
 	fs_info->sectorsize_bits = ilog2(sectorsize);
 	fs_info->sectors_per_page = (PAGE_SIZE >> fs_info->sectorsize_bits);
 	fs_info->csums_per_leaf = BTRFS_MAX_ITEM_SIZE(fs_info) / fs_info->csum_size;
fs/btrfs/sysfs.c +73 -15

115 lines not on this page · raw message

fs/btrfs/volumes.c +53 -0

65 lines not on this page · raw message

fs/btrfs/volumes.h +9 -0
index 3a416b1bc24c..0db754a4b13d 100644
@@ -303,6 +303,10 @@ enum btrfs_chunk_allocation_policy {
 enum btrfs_read_policy {
 	/* Use process PID to choose the stripe */
 	BTRFS_READ_POLICY_PID,
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+	/* Balancing raid1 reads across all striped devices */
+	BTRFS_READ_POLICY_ROTATION,
+#endif
 	BTRFS_NR_READ_POLICY,
 };
 
@@ -431,6 +435,11 @@ struct btrfs_fs_devices {
 	enum btrfs_read_policy read_policy;
 
 #ifdef CONFIG_BTRFS_EXPERIMENTAL
+	/* IO stat, read counter. */
+	atomic_t total_reads;
+	/* Min contiguous reads before switching to next device. */
+	int min_contiguous_read;
+
 	/* Checksum mode - offload it or do it synchronously. */
 	enum btrfs_offload_csum_mode offload_csum_mode;
 #endif
Signature · 1 line
2.46.1
author
#2 lore raw

PATCH v2 2/3 btrfs: use the path with the lowest latency for RAID1 reads

In reply to Anand Jain

This feature aims to direct the read I/O to the device with the lowest known latency for reading RAID1 blocks.

echo "latency" > /sys/fs/btrfs/<UUID>/read_policy

---
 fs/btrfs/sysfs.c   |  2 +-
 fs/btrfs/volumes.c | 36 ++++++++++++++++++++++++++++++++++++
 fs/btrfs/volumes.h |  2 ++
 3 files changed, 39 insertions(+), 1 deletion(-)
fs/btrfs/sysfs.c +1 -1
index bacb2871109b..9f506d46a94c 100644
@@ -1306,7 +1306,7 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj,
 BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);
 
 #ifdef CONFIG_BTRFS_EXPERIMENTAL
-static const char * const btrfs_read_policy_name[] = { "pid", "rotation" };
+static const char * const btrfs_read_policy_name[] = { "pid", "rotation", "latency" };
 #else
 static const char * const btrfs_read_policy_name[] = { "pid" };
 #endif
fs/btrfs/volumes.c +36 -0
index ec5dbe69ba2c..8912ee1d8b54 100644
@@ -12,6 +12,9 @@
 #include <linux/uuid.h>
 #include <linux/list_sort.h>
 #include <linux/namei.h>
+#ifdef CONFIG_BTRFS_EXPERIMENTAL
+#include <linux/part_stat.h>
+#endif
 #include "misc.h"
 #include "ctree.h"
 #include "disk-io.h"
@@ -5963,6 +5966,35 @@ unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info,
 }
 
 #ifdef CONFIG_BTRFS_EXPERIMENTAL
+static int btrfs_best_stripe(struct btrfs_fs_info *fs_info,
+			     struct btrfs_chunk_map *map, int first,
+			     int num_stripe)
+{
+	u64 best_wait = U64_MAX;
+	int best_stripe = 0;
+	int index;
+
+	for (index = first; index < first + num_stripe; index++) {
+		u64 read_wait;
+		u64 avg_wait = 0;
+		unsigned long read_ios;
+		struct btrfs_device *device = map->stripes[index].dev;
+
+		read_wait = part_stat_read(device->bdev, nsecs[READ]);
+		read_ios = part_stat_read(device->bdev, ios[READ]);
+
+		if (read_wait && read_ios && read_wait >= read_ios)
+			avg_wait = div_u64(read_wait, read_ios);
+
+		if (best_wait > avg_wait) {
+			best_wait = avg_wait;
+			best_stripe = index;
+		}
+	}
+
+	return best_stripe;
+}
+
 struct stripe_mirror {
 	u64 devid;
 	int num;
@@ -6043,6 +6075,10 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,
 	case BTRFS_READ_POLICY_ROTATION:
 		preferred_mirror = btrfs_read_rotation(map, first, num_stripes);
 		break;
+	case BTRFS_READ_POLICY_LATENCY:
+		preferred_mirror = btrfs_best_stripe(fs_info, map, first,
+								num_stripes);
+		break;
 #endif
 	}
 
fs/btrfs/volumes.h +2 -0
index 0db754a4b13d..f9c744b87b61 100644
@@ -306,6 +306,8 @@ enum btrfs_read_policy {
 #ifdef CONFIG_BTRFS_EXPERIMENTAL
 	/* Balancing raid1 reads across all striped devices */
 	BTRFS_READ_POLICY_ROTATION,
+	/* Use the lowest-latency device dynamically */
+	BTRFS_READ_POLICY_LATENCY,
 #endif
 	BTRFS_NR_READ_POLICY,
 };
Signature · 1 line
2.46.1
author
#3 lore raw

PATCH v2 3/3 btrfs: add RAID1 preferred read device

In reply to Anand Jain

When there's stale data on a mirrored device, this feature lets you choose which device to read from. Mainly used for testing.

echo "devid:2" > /sys/fs/btrfs/<UUID>/read_policy

---
 fs/btrfs/disk-io.c |  1 +
 fs/btrfs/sysfs.c   | 32 ++++++++++++++++++++++++++++++--
 fs/btrfs/volumes.c | 20 ++++++++++++++++++++
 fs/btrfs/volumes.h |  5 +++++
 4 files changed, 56 insertions(+), 2 deletions(-)
fs/btrfs/disk-io.c +1 -0
index 5b157f407e0a..0e7b29282136 100644
@@ -3322,6 +3322,7 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device
 	fs_info->sectorsize = sectorsize;
 #ifdef CONFIG_BTRFS_EXPERIMENTAL
 	fs_info->fs_devices->min_contiguous_read = sectorsize;
+	fs_info->fs_devices->read_devid = fs_info->fs_devices->latest_dev->devid;
 #endif
 	fs_info->sectorsize_bits = ilog2(sectorsize);
 	fs_info->sectors_per_page = (PAGE_SIZE >> fs_info->sectorsize_bits);
fs/btrfs/sysfs.c +30 -2
index 9f506d46a94c..aa4c9cbaa61f 100644
@@ -1306,7 +1306,7 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj,
 BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);
 
 #ifdef CONFIG_BTRFS_EXPERIMENTAL
-static const char * const btrfs_read_policy_name[] = { "pid", "rotation", "latency" };
+static const char * const btrfs_read_policy_name[] = { "pid", "rotation", "latency", "devid" };
 #else
 static const char * const btrfs_read_policy_name[] = { "pid" };
 #endif
@@ -1332,8 +1332,11 @@ static ssize_t btrfs_read_policy_show(struct kobject *kobj,
 		if (i == BTRFS_READ_POLICY_ROTATION)
 			ret += sysfs_emit_at(buf, ret, ":%d",
 					     fs_devices->min_contiguous_read);
-#endif
 
+		if (i == BTRFS_READ_POLICY_DEVID)
+			ret += sysfs_emit_at(buf, ret, ":%llu",
+							fs_devices->read_devid);
+#endif
 		if (i == policy)
 			ret += sysfs_emit_at(buf, ret, "]");
 	}
@@ -1401,7 +1404,32 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj,
 
 		return len;
 	}
+
+	if (index == BTRFS_READ_POLICY_DEVID) {
+		u64 value_devid;
+		BTRFS_DEV_LOOKUP_ARGS(args);
+
+		if (value == NULL || kstrtou64(value, 10, &value_devid))
+			return -EINVAL;
+
+		args.devid = value_devid;
+		if (btrfs_find_device(fs_devices, &args) == NULL)
+			return -EINVAL;
+
+		if (index != READ_ONCE(fs_devices->read_policy) ||
+		    (value_devid != READ_ONCE(fs_devices->read_devid))) {
+			WRITE_ONCE(fs_devices->read_policy, index);
+			WRITE_ONCE(fs_devices->read_devid, value_devid);
+
+			btrfs_info(fs_devices->fs_info, "read policy set to '%s:%llu'",
+				   btrfs_read_policy_name[index], value_devid);
+
+		}
+
+		return len;
+	}
 #endif
+
 	if (index != READ_ONCE(fs_devices->read_policy)) {
 		WRITE_ONCE(fs_devices->read_policy, index);
 		btrfs_info(fs_devices->fs_info, "read policy set to '%s'",
fs/btrfs/volumes.c +20 -0
index 8912ee1d8b54..87a072fa9be4 100644
@@ -5966,6 +5966,23 @@ unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info,
 }
 
 #ifdef CONFIG_BTRFS_EXPERIMENTAL
+static int btrfs_read_preferred(struct btrfs_chunk_map *map, int first,
+				int num_stripe)
+{
+	int last = first + num_stripe;
+	int stripe_index;
+
+	for (stripe_index = first; stripe_index < last; stripe_index++) {
+		struct btrfs_device *device = map->stripes[stripe_index].dev;
+
+		if (device->devid == READ_ONCE(device->fs_devices->read_devid))
+			return stripe_index;
+	}
+
+	/* If no read-preferred device, use first stripe */
+	return first;
+}
+
 static int btrfs_best_stripe(struct btrfs_fs_info *fs_info,
 			     struct btrfs_chunk_map *map, int first,
 			     int num_stripe)
@@ -6079,6 +6096,9 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,
 		preferred_mirror = btrfs_best_stripe(fs_info, map, first,
 								num_stripes);
 		break;
+	case BTRFS_READ_POLICY_DEVID:
+		preferred_mirror = btrfs_read_preferred(map, first, num_stripes);
+		break;
 #endif
 	}
 
fs/btrfs/volumes.h +5 -0
index f9c744b87b61..b5ade9d41fe7 100644
@@ -308,6 +308,8 @@ enum btrfs_read_policy {
 	BTRFS_READ_POLICY_ROTATION,
 	/* Use the lowest-latency device dynamically */
 	BTRFS_READ_POLICY_LATENCY,
+	/* Read from the specific device */
+	BTRFS_READ_POLICY_DEVID,
 #endif
 	BTRFS_NR_READ_POLICY,
 };
@@ -442,6 +444,9 @@ struct btrfs_fs_devices {
 	/* Min contiguous reads before switching to next device. */
 	int min_contiguous_read;
 
+	/* Device to be used for reading in case of RAID1. */
+	u64 read_devid;
+
 	/* Checksum mode - offload it or do it synchronously. */
 	enum btrfs_offload_csum_mode offload_csum_mode;
 #endif
Signature · 1 line
2.46.1
author
#4 lore raw
In reply to Anand Jain
Quoted 40 lines · Anand Jain

v2:
1. Move new features to CONFIG_BTRFS_EXPERIMENTAL instead of CONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait to U64_MAX and remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:

echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy

The default value is the sector size, and the min_contiguous_read value must be a multiple of the sector size.

5. Tested FIO random read/write and defrag compression workloads with min_contiguous_read set to sector size, 192k, and 256k.

RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.

$ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
--ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
--time_based --group_reporting --name=iops-test-job --eta-newline=1

|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 20.3MiB/s  | 20.5MiB/s  | 313895 | 313895 |
| rotation|            |            |        |        |
|     4096| 20.4MiB/s  | 20.5MiB/s  | 313895 | 313895 |
|   196608| 20.2MiB/s  | 20.2MiB/s  | 310152 | 310175 |
|   262144| 20.3MiB/s  | 20.4MiB/s  | 312180 | 312191 |
|  latency| 18.4MiB/s  | 18.4MiB/s  | 272980 | 291683 |
| devid:1 | 14.8MiB/s  | 14.9MiB/s  | 456376 | 0      |

rotation RAID1 balancing technique performs more than 2x better for single-process defrag.

$ time -p btrfs filesystem defrag -r -f -c /btrfs

Quoted 11 lines
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 18.00s| 3800   | 0      |
| rotation|       |        |        |
|     4096|  8.95s| 1900   | 1901   |
|   196608|  8.50s| 1881   | 1919   |
|   262144|  8.80s| 1881   | 1919   |
| latency | 17.18s| 3800   | 0      |
| devid:2 | 17.48s| 0      | 3800   |

Copy and paste error. Please ignore the below paragraph. Thx. ---vvv--- ignore ---vvv----

Quoted 7 lines

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.

---^^^------------^^^------

Quoted 83 lines

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.

Latency is smarter with unstable block layer transport.

Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.

Devid is a hands-on approach, provides manual or user-space script control.

These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.

Thx.

--------- original v1 ------------

The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the CONFIG_BTRFS_DEBUG config option and are on top of the previously added /sys/fs/btrfs/<UUID>/read_policy interface to configure the desired RAID1 read balancing method.

I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from /sys/devices/<..>/stat for each device. Below is a summary of the results, with each result the average of three iterations.

A typical generic random rw workload:

$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \
   --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \
   --group_reporting --name=iops-test-job --eta-newline=1
|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 29.4MiB/s  | 29.5MiB/s  | 456548 | 447975 |
| rotation| 29.3MiB/s  | 29.3MiB/s  | 450105 | 450055 |
| latency | 21.9MiB/s  | 21.9MiB/s  | 672387 | 0      |
| devid:1 | 22.0MiB/s  | 22.0MiB/s  | 674788 | 0      |

Defragmentation with compression workload:

$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo
$ sync
$ echo 3 > /proc/sys/vm/drop_caches
$ btrfs filesystem defrag -f -c /btrfs/foo
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 21.61s| 3810   | 0      |
| rotation| 11.55s| 1905   | 1905   |
| latency | 20.99s| 0      | 3810   |
| devid:2 | 21.41s| 0      | 3810   |

. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.

Anand Jain (3):
btrfs: introduce RAID1 round-robin read balancing
btrfs: use the path with the lowest latency for RAID1 reads
btrfs: add RAID1 preferred read device

  fs/btrfs/disk-io.c |   4 ++
  fs/btrfs/sysfs.c   | 116 +++++++++++++++++++++++++++++++++++++++------
  fs/btrfs/volumes.c | 109 ++++++++++++++++++++++++++++++++++++++++++
  fs/btrfs/volumes.h |  16 +++++++
  4 files changed, 230 insertions(+), 15 deletions(-)
#5 lore raw
In reply to Anand Jain
Quoted 11 lines · Anand Jain

v2:
1. Move new features to CONFIG_BTRFS_EXPERIMENTAL instead of CONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait to U64_MAX and remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:

echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy

The default value is the sector size, and the min_contiguous_read value must be a multiple of the sector size.

Overall, I'm fine with the latency and preferred device policies.

Meanwhile I'd prefer the previous version of round-robin, without the min_contiguous_read.

That looks a little overkilled, and I think we should keep the policy as simple as possible for now.

Mind to share why the min_contiguous_read is introduced in this update?

In the future, we should go the same method as sched_ext, by pushing the complex policies to eBPF programs.

Another future improvement is the interface, I'm fine with the sysfs knob for an experimental feature.

But from my last drop_subtree_threshold experience, sysfs is not going to be a user-friendly interface. It really relies on some user space daemon to set.

I'd prefer something more persistent, like some XATTR but inside root tree, and go with prop interfaces.
But that can all be done in the future.

Thanks,
Qu

Quoted 131 lines

5. Tested FIO random read/write and defrag compression workloads with min_contiguous_read set to sector size, 192k, and 256k.

RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.

$ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
--ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
--time_based --group_reporting --name=iops-test-job --eta-newline=1

|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 20.3MiB/s  | 20.5MiB/s  | 313895 | 313895 |
| rotation|            |            |        |        |
|     4096| 20.4MiB/s  | 20.5MiB/s  | 313895 | 313895 |
|   196608| 20.2MiB/s  | 20.2MiB/s  | 310152 | 310175 |
|   262144| 20.3MiB/s  | 20.4MiB/s  | 312180 | 312191 |
|  latency| 18.4MiB/s  | 18.4MiB/s  | 272980 | 291683 |
| devid:1 | 14.8MiB/s  | 14.9MiB/s  | 456376 | 0      |

rotation RAID1 balancing technique performs more than 2x better for single-process defrag.

$ time -p btrfs filesystem defrag -r -f -c /btrfs

|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 18.00s| 3800   | 0      |
| rotation|       |        |        |
|     4096|  8.95s| 1900   | 1901   |
|   196608|  8.50s| 1881   | 1919   |
|   262144|  8.80s| 1881   | 1919   |
| latency | 17.18s| 3800   | 0      |
| devid:2 | 17.48s| 0      | 3800   |

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.

Latency is smarter with unstable block layer transport.

Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.

Devid is a hands-on approach, provides manual or user-space script control.

These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.

Thx.

--------- original v1 ------------

The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the CONFIG_BTRFS_DEBUG config option and are on top of the previously added /sys/fs/btrfs/<UUID>/read_policy interface to configure the desired RAID1 read balancing method.

I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from /sys/devices/<..>/stat for each device. Below is a summary of the results, with each result the average of three iterations.

A typical generic random rw workload:

$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \
   --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \
   --group_reporting --name=iops-test-job --eta-newline=1
|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 29.4MiB/s  | 29.5MiB/s  | 456548 | 447975 |
| rotation| 29.3MiB/s  | 29.3MiB/s  | 450105 | 450055 |
| latency | 21.9MiB/s  | 21.9MiB/s  | 672387 | 0      |
| devid:1 | 22.0MiB/s  | 22.0MiB/s  | 674788 | 0      |

Defragmentation with compression workload:

$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo
$ sync
$ echo 3 > /proc/sys/vm/drop_caches
$ btrfs filesystem defrag -f -c /btrfs/foo
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 21.61s| 3810   | 0      |
| rotation| 11.55s| 1905   | 1905   |
| latency | 20.99s| 0      | 3810   |
| devid:2 | 21.41s| 0      | 3810   |

. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.

Anand Jain (3):
btrfs: introduce RAID1 round-robin read balancing
btrfs: use the path with the lowest latency for RAID1 reads
btrfs: add RAID1 preferred read device

  fs/btrfs/disk-io.c |   4 ++
  fs/btrfs/sysfs.c   | 116 +++++++++++++++++++++++++++++++++++++++------
  fs/btrfs/volumes.c | 109 ++++++++++++++++++++++++++++++++++++++++++
  fs/btrfs/volumes.h |  16 +++++++
  4 files changed, 230 insertions(+), 15 deletions(-)
author
#6 lore raw
Quoted 17 lines · Qu Wenruo
Quoted 11 lines · Anand Jain

v2:
1. Move new features to CONFIG_BTRFS_EXPERIMENTAL instead of CONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait to U64_MAX and remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:

         echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/ read_policy

    The default value is the sector size, and the min_contiguous_read value must be a multiple of the sector size.

Overall, I'm fine with the latency and preferred device policies.

Quoted 8 lines

Meanwhile I'd prefer the previous version of round-robin, without the min_contiguous_read.

That looks a little overkilled, and I think we should keep the policy as simple as possible for now.

Mind to share why the min_contiguous_read is introduced in this update?

The reason for adding min_contiguous_read:
The block layer optimizes with bio merging to improve HDD performance. David mentioned on Slack that 192k to 256k co-reads can performance better, though I haven't seen this in my setup but it
may work in others.

Quoted 2 lines

In the future, we should go the same method as sched_ext, by pushing the complex policies to eBPF programs.

External scripts for RAID1 balancing are achievable with BPF, though it require writable BPF, which is disabled in some cases. That said, still we should prioritize adding support and provide choice to the use-case to decide.

Quoted 2 lines

Another future improvement is the interface, I'm fine with the sysfs knob for an experimental feature.

Yes, we need to review tunables - mount options, sysfs, and btrfs properties to have a clear guidelines/consolidation.

Quoted 4 lines

But from my last drop_subtree_threshold experience, sysfs is not going to be a user-friendly interface. It really relies on some user space daemon to set.

Agreed. However, for Btrfs, sysfs has been the most comprehensive so far.

Quoted 4 lines

I'd prefer something more persistent, like some XATTR but inside root tree, and go with prop interfaces.
But that can all be done in the future.

Absolutely. I included that in earlier experiments, but it was removed due to review comments. Now isn't the right time to reintroduce it; we can update the on-disk formats and xattrs once the in-memory graduates address specific use cases.

Thanks, Anand

Quoted 134 lines

Thanks,
Qu

Quoted 131 lines

5. Tested FIO random read/write and defrag compression workloads with min_contiguous_read set to sector size, 192k, and 256k.

    RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.

      $ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw -- bs=4k \
         --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
         --time_based --group_reporting --name=iops-test-job --eta- newline=1
|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 20.3MiB/s  | 20.5MiB/s  | 313895 | 313895 |
| rotation|            |            |        |        |
|     4096| 20.4MiB/s  | 20.5MiB/s  | 313895 | 313895 |
|   196608| 20.2MiB/s  | 20.2MiB/s  | 310152 | 310175 |
|   262144| 20.3MiB/s  | 20.4MiB/s  | 312180 | 312191 |
|  latency| 18.4MiB/s  | 18.4MiB/s  | 272980 | 291683 |
| devid:1 | 14.8MiB/s  | 14.9MiB/s  | 456376 | 0      |

    rotation RAID1 balancing technique performs more than 2x better for single-process defrag.

       $ time -p btrfs filesystem defrag -r -f -c /btrfs
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 18.00s| 3800   | 0      |
| rotation|       |        |        |
|     4096|  8.95s| 1900   | 1901   |
|   196608|  8.50s| 1881   | 1919   |
|   262144|  8.80s| 1881   | 1919   |
| latency | 17.18s| 3800   | 0      |
| devid:2 | 17.48s| 0      | 3800   |

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.

Latency is smarter with unstable block layer transport.

Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.

Devid is a hands-on approach, provides manual or user-space script control.

These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.

Thx.

--------- original v1 ------------

The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the CONFIG_BTRFS_DEBUG config option and are on top of the previously added /sys/fs/btrfs/<UUID>/read_policy interface to configure the desired RAID1 read balancing method.

I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from /sys/devices/<..>/stat for each device. Below is a summary of the results, with each result the average of three iterations.

A typical generic random rw workload:

$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \
   --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 -- time_based \
   --group_reporting --name=iops-test-job --eta-newline=1
|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 29.4MiB/s  | 29.5MiB/s  | 456548 | 447975 |
| rotation| 29.3MiB/s  | 29.3MiB/s  | 450105 | 450055 |
| latency | 21.9MiB/s  | 21.9MiB/s  | 672387 | 0      |
| devid:1 | 22.0MiB/s  | 22.0MiB/s  | 674788 | 0      |

Defragmentation with compression workload:

$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo
$ sync
$ echo 3 > /proc/sys/vm/drop_caches
$ btrfs filesystem defrag -f -c /btrfs/foo
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 21.61s| 3810   | 0      |
| rotation| 11.55s| 1905   | 1905   |
| latency | 20.99s| 0      | 3810   |
| devid:2 | 21.41s| 0      | 3810   |

. The PID-based balancing method works well for the generic random rw fio workload.
. The rotation method is ideal when you want to keep both devices active, and it boosts performance in sequential defragmentation scenarios. . The latency-based method work well when we have mixed device types or when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and testing RAID1 mirror synchronizations.

Anand Jain (3):
   btrfs: introduce RAID1 round-robin read balancing
   btrfs: use the path with the lowest latency for RAID1 reads
   btrfs: add RAID1 preferred read device

  fs/btrfs/disk-io.c |   4 ++
  fs/btrfs/sysfs.c   | 116 +++++++++++++++++++++++++++++++++++++++------ fs/btrfs/volumes.c | 109 ++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h |  16 +++++++

  4 files changed, 230 insertions(+), 15 deletions(-)
#7 lore raw
In reply to Anand Jain
Quoted 11 lines · Anand Jain

v2:
1. Move new features to CONFIG_BTRFS_EXPERIMENTAL instead of CONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait to U64_MAX and remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:

echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy

The default value is the sector size, and the min_contiguous_read value must be a multiple of the sector size.

I think it's safe to start with the round-round robin policy, but the syntax is strange, why the [ ] are mandatory? Also please call it round-robin, or 'rr' for short.

The default of sector size is IMHO a wrong value, switching devices so often will drop the performance just because of the io request overhead. From what I rememer values around 200k were reasonable, so either 192k or 256k should be the default. We may also drop the configurable value at all and provide a few hard coded sizes like rr-256k, rr-512k, rr-1m, if not only to drop parsing of user strings.

Quoted 39 lines

5. Tested FIO random read/write and defrag compression workloads with min_contiguous_read set to sector size, 192k, and 256k.

   RAID1 balancing method rotation is better for multi-process workloads
   such as fio and also single-process workload such as defragmentation.

     $ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
        --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
        --time_based --group_reporting --name=iops-test-job --eta-newline=1
|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 20.3MiB/s  | 20.5MiB/s  | 313895 | 313895 |
| rotation|            |            |        |        |
|     4096| 20.4MiB/s  | 20.5MiB/s  | 313895 | 313895 |
|   196608| 20.2MiB/s  | 20.2MiB/s  | 310152 | 310175 |
|   262144| 20.3MiB/s  | 20.4MiB/s  | 312180 | 312191 |
|  latency| 18.4MiB/s  | 18.4MiB/s  | 272980 | 291683 |
| devid:1 | 14.8MiB/s  | 14.9MiB/s  | 456376 | 0      |

rotation RAID1 balancing technique performs more than 2x better for single-process defrag.

      $ time -p btrfs filesystem defrag -r -f -c /btrfs
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 18.00s| 3800   | 0      |
| rotation|       |        |        |
|     4096|  8.95s| 1900   | 1901   |
|   196608|  8.50s| 1881   | 1919   |
|   262144|  8.80s| 1881   | 1919   |
| latency | 17.18s| 3800   | 0      |
| devid:2 | 17.48s| 0      | 3800   |
Quoted 3 lines

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.

Yeah round-robin will be a good defalt, we only need to verify the chunk size and then do the switch in the next release.

Quoted 4 lines

While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.

This sounds like he latency is good for a specific case and maybe a fallback if the device becomes faulty, but once the layer below becomes unstable we may need to skip reading from the device. This is also a different mode of operation than balancing reads.

Quoted 6 lines

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.

Latency is smarter with unstable block layer transport.

Quoted 7 lines

Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.

Devid is a hands-on approach, provides manual or user-space script control.

These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.

To move forward with the feature I think the round robin and preferred device id can be merged. I'm not sure about the latency but if it's under experimental we can take it as is and tune later.

I'll check my notes from the last time Michal attempted to implement the policies if we haven't missed something.

author
#8 lore raw

Thanks for commenting.

Quoted 8 lines · David Sterba
Quoted 7 lines · Anand Jain

v2:
1. Move new features to CONFIG_BTRFS_EXPERIMENTAL instead of CONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait to U64_MAX and remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:

Quoted 9 lines
Quoted 4 lines

echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy

The default value is the sector size, and the min_contiguous_read value must be a multiple of the sector size.

I think it's safe to start with the round-round robin policy, but the syntax is strange, why the [ ] are mandatory? Also please call it round-robin, or 'rr' for short.

I'm fine with round-robin. The [ ] part is not mandatory; if the min_contiguous_read value is not specified, it will default to a predefined value.

Quoted 7 lines

The default of sector size is IMHO a wrong value, switching devices so often will drop the performance just because of the io request overhead.

From what I rememer values around 200k were reasonable, so either 192k
or 256k should be the default. We may also drop the configurable value at all and provide a few hard coded sizes like rr-256k, rr-512k, rr-1m, if not only to drop parsing of user strings.

I'm okay with a default value of 256k. For the experimental feature, we can keep it configurable, allowing the opportunity to experiment with other values as well

Quoted 40 lines
Quoted 40 lines

5. Tested FIO random read/write and defrag compression workloads with min_contiguous_read set to sector size, 192k, and 256k.

RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.

$ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
--ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
--time_based --group_reporting --name=iops-test-job --eta-newline=1

|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 20.3MiB/s  | 20.5MiB/s  | 313895 | 313895 |
| rotation|            |            |        |        |
|     4096| 20.4MiB/s  | 20.5MiB/s  | 313895 | 313895 |
|   196608| 20.2MiB/s  | 20.2MiB/s  | 310152 | 310175 |
|   262144| 20.3MiB/s  | 20.4MiB/s  | 312180 | 312191 |
|  latency| 18.4MiB/s  | 18.4MiB/s  | 272980 | 291683 |
| devid:1 | 14.8MiB/s  | 14.9MiB/s  | 456376 | 0      |

rotation RAID1 balancing technique performs more than 2x better for single-process defrag.

$ time -p btrfs filesystem defrag -r -f -c /btrfs

|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 18.00s| 3800   | 0      |
| rotation|       |        |        |
|     4096|  8.95s| 1900   | 1901   |
|   196608|  8.50s| 1881   | 1919   |
|   262144|  8.80s| 1881   | 1919   |
| latency | 17.18s| 3800   | 0      |
| devid:2 | 17.48s| 0      | 3800   |
Quoted 7 lines
Quoted 3 lines

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.

Yeah round-robin will be a good defalt, we only need to verify the chunk size and then do the switch in the next release.

Yes..

Quoted 10 lines
Quoted 4 lines

While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.

This sounds like he latency is good for a specific case and maybe a fallback if the device becomes faulty, but once the layer below becomes unstable we may need to skip reading from the device. This is also a different mode of operation than balancing reads.

If the latency on the faulty path is so high that it shouldn't pick that path at all, so it works. However, the round-robin balancing is unaware of dynamic faults on the device path. IMO, a round-robin method that is latency aware (with ~20% variance) would be better.

Quoted 9 lines
Quoted 9 lines

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.

Latency is smarter with unstable block layer transport.

Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.

Quoted 8 lines
Quoted 4 lines

Devid is a hands-on approach, provides manual or user-space script control.

These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.

To move forward with the feature I think the round robin and preferred device id can be merged. I'm not sure about the latency but if it's under experimental we can take it as is and tune later.

I hope the experimental feature also means we can change the name of the balancing method at any time. Once we have tested a fair combination of block device types, we'll definitely need a method that can automatically tune based on the device type, which will require adding or dropping balancing methods accordingly.

Quoted 2 lines

I'll check my notes from the last time Michal attempted to implement the policies if we haven't missed something.

Thanks, Anand

#9 lore raw
Quoted 11 lines · Anand Jain
Quoted 3 lines

I think it's safe to start with the round-round robin policy, but the syntax is strange, why the [ ] are mandatory? Also please call it round-robin, or 'rr' for short.

I'm fine with round-robin. The [ ] part is not mandatory; if the min_contiguous_read value is not specified, it will default to a predefined value.

Quoted 2 lines

The default of sector size is IMHO a wrong value, switching devices so often will drop the performance just because of the io request overhead.

Quoted 8 lines
Quoted 4 lines

From what I rememer values around 200k were reasonable, so either 192k
or 256k should be the default. We may also drop the configurable value at all and provide a few hard coded sizes like rr-256k, rr-512k, rr-1m, if not only to drop parsing of user strings.

I'm okay with a default value of 256k. For the experimental feature, we can keep it configurable, allowing the opportunity to experiment with other values as well

Yeah, for experimenting it makes sense to make it flexible, no need to patch and reboot the kernel. For final we should settle on some reasonable values.

Quoted 60 lines
Quoted 39 lines
Quoted 39 lines

5. Tested FIO random read/write and defrag compression workloads with min_contiguous_read set to sector size, 192k, and 256k.

RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.

$ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
--ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
--time_based --group_reporting --name=iops-test-job --eta-newline=1

|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 20.3MiB/s  | 20.5MiB/s  | 313895 | 313895 |
| rotation|            |            |        |        |
|     4096| 20.4MiB/s  | 20.5MiB/s  | 313895 | 313895 |
|   196608| 20.2MiB/s  | 20.2MiB/s  | 310152 | 310175 |
|   262144| 20.3MiB/s  | 20.4MiB/s  | 312180 | 312191 |
|  latency| 18.4MiB/s  | 18.4MiB/s  | 272980 | 291683 |
| devid:1 | 14.8MiB/s  | 14.9MiB/s  | 456376 | 0      |

rotation RAID1 balancing technique performs more than 2x better for single-process defrag.

$ time -p btrfs filesystem defrag -r -f -c /btrfs

|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 18.00s| 3800   | 0      |
| rotation|       |        |        |
|     4096|  8.95s| 1900   | 1901   |
|   196608|  8.50s| 1881   | 1919   |
|   262144|  8.80s| 1881   | 1919   |
| latency | 17.18s| 3800   | 0      |
| devid:2 | 17.48s| 0      | 3800   |
Quoted 7 lines
Quoted 3 lines

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.

Yeah round-robin will be a good defalt, we only need to verify the chunk size and then do the switch in the next release.

Yes..

Quoted 10 lines
Quoted 4 lines

While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.

This sounds like he latency is good for a specific case and maybe a fallback if the device becomes faulty, but once the layer below becomes unstable we may need to skip reading from the device. This is also a different mode of operation than balancing reads.

Quoted 4 lines

If the latency on the faulty path is so high that it shouldn't pick that path at all, so it works. However, the round-robin balancing is unaware of dynamic faults on the device path. IMO, a round-robin method that is latency aware (with ~20% variance) would be better.

We should not mix the faulty device handling mode to the read balancing, at least for now. A back off algorithm that checks number of failed io requests should precede the balancing.

Quoted 18 lines
Quoted 9 lines
Quoted 9 lines

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.

Latency is smarter with unstable block layer transport.

Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.

Quoted 8 lines
Quoted 4 lines

Devid is a hands-on approach, provides manual or user-space script control.

These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.

To move forward with the feature I think the round robin and preferred device id can be merged. I'm not sure about the latency but if it's under experimental we can take it as is and tune later.

Quoted 5 lines

I hope the experimental feature also means we can change the name of the balancing method at any time. Once we have tested a fair combination of block device types, we'll definitely need a method that can automatically tune based on the device type, which will require adding or dropping balancing methods accordingly.

Yes we can change the names. The automatic tuning would need some feedback that measures the load and tries to improve the throughput, this is where we got stuck last time. So for now let's do some starightforward policy that on average works better than the current pid policy. I hope that tha round-robin-256k can be a good default, but of course we need more data for that.

author
#10 lore raw
Quoted 118 lines · David Sterba
Quoted 11 lines · Anand Jain
Quoted 3 lines

I think it's safe to start with the round-round robin policy, but the syntax is strange, why the [ ] are mandatory? Also please call it round-robin, or 'rr' for short.

I'm fine with round-robin. The [ ] part is not mandatory; if the min_contiguous_read value is not specified, it will default to a predefined value.

Quoted 2 lines

The default of sector size is IMHO a wrong value, switching devices so often will drop the performance just because of the io request overhead.

Quoted 8 lines
Quoted 4 lines

From what I rememer values around 200k were reasonable, so either 192k
or 256k should be the default. We may also drop the configurable value at all and provide a few hard coded sizes like rr-256k, rr-512k, rr-1m, if not only to drop parsing of user strings.

I'm okay with a default value of 256k. For the experimental feature, we can keep it configurable, allowing the opportunity to experiment with other values as well

Yeah, for experimenting it makes sense to make it flexible, no need to patch and reboot the kernel. For final we should settle on some reasonable values.

Quoted 60 lines
Quoted 39 lines
Quoted 39 lines

5. Tested FIO random read/write and defrag compression workloads with min_contiguous_read set to sector size, 192k, and 256k.

RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.

$ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
--ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
--time_based --group_reporting --name=iops-test-job --eta-newline=1

|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 20.3MiB/s  | 20.5MiB/s  | 313895 | 313895 |
| rotation|            |            |        |        |
|     4096| 20.4MiB/s  | 20.5MiB/s  | 313895 | 313895 |
|   196608| 20.2MiB/s  | 20.2MiB/s  | 310152 | 310175 |
|   262144| 20.3MiB/s  | 20.4MiB/s  | 312180 | 312191 |
|  latency| 18.4MiB/s  | 18.4MiB/s  | 272980 | 291683 |
| devid:1 | 14.8MiB/s  | 14.9MiB/s  | 456376 | 0      |

rotation RAID1 balancing technique performs more than 2x better for single-process defrag.

$ time -p btrfs filesystem defrag -r -f -c /btrfs

|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 18.00s| 3800   | 0      |
| rotation|       |        |        |
|     4096|  8.95s| 1900   | 1901   |
|   196608|  8.50s| 1881   | 1919   |
|   262144|  8.80s| 1881   | 1919   |
| latency | 17.18s| 3800   | 0      |
| devid:2 | 17.48s| 0      | 3800   |
Quoted 7 lines
Quoted 3 lines

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.

Yeah round-robin will be a good defalt, we only need to verify the chunk size and then do the switch in the next release.

Yes..

Quoted 10 lines
Quoted 4 lines

While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.

This sounds like he latency is good for a specific case and maybe a fallback if the device becomes faulty, but once the layer below becomes unstable we may need to skip reading from the device. This is also a different mode of operation than balancing reads.

Quoted 4 lines

If the latency on the faulty path is so high that it shouldn't pick that path at all, so it works. However, the round-robin balancing is unaware of dynamic faults on the device path. IMO, a round-robin method that is latency aware (with ~20% variance) would be better.

We should not mix the faulty device handling mode to the read balancing, at least for now. A back off algorithm that checks number of failed io requests should precede the balancing.

Quoted 18 lines
Quoted 9 lines
Quoted 9 lines

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.

Latency is smarter with unstable block layer transport.

Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.

Quoted 8 lines
Quoted 4 lines

Devid is a hands-on approach, provides manual or user-space script control.

These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.

To move forward with the feature I think the round robin and preferred device id can be merged. I'm not sure about the latency but if it's under experimental we can take it as is and tune later.

Quoted 5 lines

I hope the experimental feature also means we can change the name of the balancing method at any time. Once we have tested a fair combination of block device types, we'll definitely need a method that can automatically tune based on the device type, which will require adding or dropping balancing methods accordingly.

Quoted 6 lines

Yes we can change the names. The automatic tuning would need some feedback that measures the load and tries to improve the throughput, this is where we got stuck last time. So for now let's do some starightforward policy that on average works better than the current pid policy. I hope that tha round-robin-256k can be a good default, but of course we need more data for that.

Sending v3 with rotation renamed to round-robin. Code review appreciated; I'll wait a day.

Thanks, Anand

#11 lore raw
In reply to Anand Jain
Quoted 9 lines · Anand Jain

v2:
1. Move new features to CONFIG_BTRFS_EXPERIMENTAL instead of CONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait to U64_MAX and remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:

echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy

Quoted 8 lines

The default value is the sector size, and the min_contiguous_read value must be a multiple of the sector size.

5. Tested FIO random read/write and defrag compression workloads with min_contiguous_read set to sector size, 192k, and 256k.

RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.

With this functionality added, would it not also make sense to add a RAID0/10 profile that limits the stripe width, so a stripe does not spawn more than n disk (for example n=4).

On systems with for example 24 disks in RAID10, a read may activate 12 disks at the same time which could easily saturate the bus.

Therefore if a storage profile that limits the number of devices a stripe occupy existed, it seems like there might be posibillities for RAID0/10 as well.

Note that as of writing this I believe that RAID0/10/5/6 make the stripe as wide as the number of storage devices available for the filesystem. If I am wrong about this please ignore my jabbering and move on.

author
#12 lore raw
Quoted 19 lines · waxhead
Quoted 9 lines · Anand Jain

v2:
1. Move new features to CONFIG_BTRFS_EXPERIMENTAL instead of CONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait to U64_MAX and remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:

         echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/ read_policy

Quoted 8 lines

    The default value is the sector size, and the min_contiguous_read value must be a multiple of the sector size.

5. Tested FIO random read/write and defrag compression workloads with min_contiguous_read set to sector size, 192k, and 256k.

    RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.

Quoted 7 lines

With this functionality added, would it not also make sense to add a RAID0/10 profile that limits the stripe width, so a stripe does not spawn more than n disk (for example n=4).

On systems with for example 24 disks in RAID10, a read may activate 12 > disks at the same time which could easily saturate the bus.

Therefore if a storage profile that limits the number of devices a stripe occupy existed, it seems like there might be posibillities for RAID0/10 as well.

Note that as of writing this I believe that RAID0/10/5/6 make the stripe as wide as the number of storage devices available for the filesystem. If I am wrong about this please ignore my jabbering and move on.

That's correct. I previously attempted to come up with a fix using the device grouping method. If there's a convincing and more generic way to specify how the devices should be grouped, we could consider that.

Thanks, Anand

#13 lore raw
In reply to waxhead
Quoted 3 lines · waxhead

Note that as of writing this I believe that RAID0/10/5/6 make the stripe as wide as the number of storage devices available for the filesystem. If I am wrong about this please ignore my jabbering and move on.

Nope, you're correct and this is a huge problem for bigger (in numbers of drives) arrays.

But it's also on my list of things I want to change in how we handle RAID with the RAID stripe-tree. This way we can do declustered RAID and ease on rebuild times. Also we can drastically enhance write parallelism to an array by directing different write streams to different sets of stripes. Which btw atm isn't even done for RAID1, as we're picking a block-group at a time until it's full which then gets written, instead of creating new block groups on new drive sets for different write streams (i.e. different files, etc..)

Reviewed-by on the v2 series
#14 lore raw
In reply to Anand Jain
Quoted 15 lines · Anand Jain

v2:
1. Move new features to CONFIG_BTRFS_EXPERIMENTAL instead of CONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait to U64_MAX and remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:

echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy

The default value is the sector size, and the min_contiguous_read value must be a multiple of the sector size.

5. Tested FIO random read/write and defrag compression workloads with min_contiguous_read set to sector size, 192k, and 256k.

Quoted 6 lines

RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.

$ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
--ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
--time_based --group_reporting --name=iops-test-job --eta-newline=1

Although not 100% happy with the min_contiguous_read setting, since it's an optional one and still experimental, I'm fine with series so far.

Just want to express my concern about going mount option.

I know sysfs is not a good way to setup a lot of features, but mount option is way too committed to me, even under experimental features.

But I also understand without mount option it can be pretty hard to setup the read policy for fstests runs.

So I'd prefer to have some on-disk solution (XATTR or temporary items) to save the read policy.
It's less committed compared to mount option (aka, much easier to revert the change with breaking any compatibility), and can help for future features.

Thanks,
Qu

Quoted 121 lines

|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 20.3MiB/s  | 20.5MiB/s  | 313895 | 313895 |
| rotation|            |            |        |        |
|     4096| 20.4MiB/s  | 20.5MiB/s  | 313895 | 313895 |
|   196608| 20.2MiB/s  | 20.2MiB/s  | 310152 | 310175 |
|   262144| 20.3MiB/s  | 20.4MiB/s  | 312180 | 312191 |
|  latency| 18.4MiB/s  | 18.4MiB/s  | 272980 | 291683 |
| devid:1 | 14.8MiB/s  | 14.9MiB/s  | 456376 | 0      |

rotation RAID1 balancing technique performs more than 2x better for single-process defrag.

$ time -p btrfs filesystem defrag -r -f -c /btrfs

|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 18.00s| 3800   | 0      |
| rotation|       |        |        |
|     4096|  8.95s| 1900   | 1901   |
|   196608|  8.50s| 1881   | 1919   |
|   262144|  8.80s| 1881   | 1919   |
| latency | 17.18s| 3800   | 0      |
| devid:2 | 17.48s| 0      | 3800   |

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.

Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.

Latency is smarter with unstable block layer transport.

Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.

Devid is a hands-on approach, provides manual or user-space script control.

These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.

Thx.

--------- original v1 ------------

The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the CONFIG_BTRFS_DEBUG config option and are on top of the previously added /sys/fs/btrfs/<UUID>/read_policy interface to configure the desired RAID1 read balancing method.

I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from /sys/devices/<..>/stat for each device. Below is a summary of the results, with each result the average of three iterations.

A typical generic random rw workload:

$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \
   --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \
   --group_reporting --name=iops-test-job --eta-newline=1
|         |            |            | Read I/O count  |
|         | Read       | Write      | devid1 | devid2 |
|---------|------------|------------|--------|--------|
| pid     | 29.4MiB/s  | 29.5MiB/s  | 456548 | 447975 |
| rotation| 29.3MiB/s  | 29.3MiB/s  | 450105 | 450055 |
| latency | 21.9MiB/s  | 21.9MiB/s  | 672387 | 0      |
| devid:1 | 22.0MiB/s  | 22.0MiB/s  | 674788 | 0      |

Defragmentation with compression workload:

$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo
$ sync
$ echo 3 > /proc/sys/vm/drop_caches
$ btrfs filesystem defrag -f -c /btrfs/foo
|         | Time  | Read I/O Count  |
|         | Real  | devid1 | devid2 |
|---------|-------|--------|--------|
| pid     | 21.61s| 3810   | 0      |
| rotation| 11.55s| 1905   | 1905   |
| latency | 20.99s| 0      | 3810   |
| devid:2 | 21.41s| 0      | 3810   |

. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.

Anand Jain (3):
btrfs: introduce RAID1 round-robin read balancing
btrfs: use the path with the lowest latency for RAID1 reads
btrfs: add RAID1 preferred read device

  fs/btrfs/disk-io.c |   4 ++
  fs/btrfs/sysfs.c   | 116 +++++++++++++++++++++++++++++++++++++++------
  fs/btrfs/volumes.c | 109 ++++++++++++++++++++++++++++++++++++++++++
  fs/btrfs/volumes.h |  16 +++++++
  4 files changed, 230 insertions(+), 15 deletions(-)