raid1 balancing methods
v2 cover raid1 balancing methods
v2:
1. Move new features to CONFIG_BTRFS_EXPERIMENTAL instead of CONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait to U64_MAX and remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:
echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy
The default value is the sector size, and the min_contiguous_read value must be a multiple of the sector size.
5. Tested FIO random read/write and defrag compression workloads with min_contiguous_read set to sector size, 192k, and 256k.
RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.
$ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
--ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
--time_based --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 20.3MiB/s | 20.5MiB/s | 313895 | 313895 | | rotation| | | | | | 4096| 20.4MiB/s | 20.5MiB/s | 313895 | 313895 | | 196608| 20.2MiB/s | 20.2MiB/s | 310152 | 310175 | | 262144| 20.3MiB/s | 20.4MiB/s | 312180 | 312191 | | latency| 18.4MiB/s | 18.4MiB/s | 272980 | 291683 | | devid:1 | 14.8MiB/s | 14.9MiB/s | 456376 | 0 |
rotation RAID1 balancing technique performs more than 2x better for single-process defrag.
$ time -p btrfs filesystem defrag -r -f -c /btrfs
| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 18.00s| 3800 | 0 | | rotation| | | | | 4096| 8.95s| 1900 | 1901 | | 196608| 8.50s| 1881 | 1919 | | 262144| 8.80s| 1881 | 1919 | | latency | 17.18s| 3800 | 0 | | devid:2 | 17.48s| 0 | 3800 |
Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.
Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.
Latency is smarter with unstable block layer transport.
Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.
Devid is a hands-on approach, provides manual or user-space script control.
These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.
Thx.
--------- original v1 ------------
The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the CONFIG_BTRFS_DEBUG config option and are on top of the previously added /sys/fs/btrfs/<UUID>/read_policy interface to configure the desired RAID1 read balancing method.
I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from /sys/devices/<..>/stat for each device. Below is a summary of the results, with each result the average of three iterations.
A typical generic random rw workload:
$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \ --group_reporting --name=iops-test-job --eta-newline=1
| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 29.4MiB/s | 29.5MiB/s | 456548 | 447975 | | rotation| 29.3MiB/s | 29.3MiB/s | 450105 | 450055 | | latency | 21.9MiB/s | 21.9MiB/s | 672387 | 0 | | devid:1 | 22.0MiB/s | 22.0MiB/s | 674788 | 0 |
Defragmentation with compression workload:
$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo $ sync $ echo 3 > /proc/sys/vm/drop_caches $ btrfs filesystem defrag -f -c /btrfs/foo
| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 21.61s| 3810 | 0 | | rotation| 11.55s| 1905 | 1905 | | latency | 20.99s| 0 | 3810 | | devid:2 | 21.41s| 0 | 3810 |
. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.
Anand Jain (3):
btrfs: introduce RAID1 round-robin read balancing
btrfs: use the path with the lowest latency for RAID1 reads
btrfs: add RAID1 preferred read device
fs/btrfs/disk-io.c | 4 ++ fs/btrfs/sysfs.c | 116 +++++++++++++++++++++++++++++++++++++++------ fs/btrfs/volumes.c | 109 ++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 16 +++++++ 4 files changed, 230 insertions(+), 15 deletions(-)
Signature · 1 line
2.46.1
PATCH v2 1/3 btrfs: introduce RAID1 round-robin read balancing
This feature balances I/O across the striped devices when reading from RAID1 blocks.
echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy
Default value of min_contiguous_read is equal to the sectorsize.
--- fs/btrfs/disk-io.c | 3 ++ fs/btrfs/sysfs.c | 88 ++++++++++++++++++++++++++++++++++++++-------- fs/btrfs/volumes.c | 53 ++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 9 +++++ 4 files changed, 138 insertions(+), 15 deletions(-)
fs/btrfs/disk-io.c +3 -0
@@ -3320,6 +3320,9 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device fs_info->nodesize = nodesize; fs_info->sectorsize = sectorsize; +#ifdef CONFIG_BTRFS_EXPERIMENTAL + fs_info->fs_devices->min_contiguous_read = sectorsize; +#endif fs_info->sectorsize_bits =ilog2 (sectorsize); fs_info->sectors_per_page = (PAGE_SIZE >> fs_info->sectorsize_bits); fs_info->csums_per_leaf =BTRFS_MAX_ITEM_SIZE (fs_info) / fs_info->csum_size;
fs/btrfs/sysfs.c +73 -15
115 lines not on this page · raw message
fs/btrfs/volumes.c +53 -0
65 lines not on this page · raw message
fs/btrfs/volumes.h +9 -0
@@ -303,6 +303,10 @@ enum btrfs_chunk_allocation_policy {enum btrfs_read_policy {/* Use process PID to choose the stripe */ BTRFS_READ_POLICY_PID , +#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Balancing raid1 reads across all striped devices */ +BTRFS_READ_POLICY_ROTATION , +#endif BTRFS_NR_READ_POLICY , };
@@ -431,6 +435,11 @@ struct btrfs_fs_devices {enum btrfs_read_policy read_policy;#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* IO stat, read counter. */ +atomic_t total_reads; +/* Min contiguous reads before switching to next device. */ +int min_contiguous_read; +/* Checksum mode - offload it or do it synchronously. */ enum btrfs_offload_csum_mode offload_csum_mode;#endif
Signature · 1 line
2.46.1
PATCH v2 2/3 btrfs: use the path with the lowest latency for RAID1 reads
In reply to Anand JainThis feature aims to direct the read I/O to the device with the lowest known latency for reading RAID1 blocks.
echo "latency" > /sys/fs/btrfs/<UUID>/read_policy
--- fs/btrfs/sysfs.c | 2 +- fs/btrfs/volumes.c | 36 ++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 2 ++ 3 files changed, 39 insertions(+), 1 deletion(-)
fs/btrfs/sysfs.c +1 -1
@@ -1306,7 +1306,7 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj, BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);#ifdef CONFIG_BTRFS_EXPERIMENTAL -static const char *const btrfs_read_policy_name[] = {"pid" ,"rotation" }; +static const char *const btrfs_read_policy_name[] = {"pid" ,"rotation" ,"latency" };#else static const char *const btrfs_read_policy_name[] = {"pid" };#endif
fs/btrfs/volumes.c +36 -0
@@ -12,6 +12,9 @@#include <linux/uuid.h> #include <linux/list_sort.h> #include <linux/namei.h> +#ifdef CONFIG_BTRFS_EXPERIMENTAL +#include <linux/part_stat.h> +#endif #include "misc.h" #include "ctree.h" #include "disk-io.h"
@@ -5963,6 +5966,35 @@ unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, } #ifdef CONFIG_BTRFS_EXPERIMENTAL +static int btrfs_best_stripe(struct btrfs_fs_info *fs_info, + struct btrfs_chunk_map *map, int first, + int num_stripe) +{ + u64 best_wait = U64_MAX; + int best_stripe = 0; + int index; + + for (index = first; index < first + num_stripe; index++) { + u64 read_wait; + u64 avg_wait = 0; + unsigned long read_ios; + struct btrfs_device *device = map->stripes[index].dev; + + read_wait = part_stat_read(device->bdev, nsecs[READ]); + read_ios = part_stat_read(device->bdev, ios[READ]); + + if (read_wait && read_ios && read_wait >= read_ios) + avg_wait = div_u64(read_wait, read_ios); + + if (best_wait > avg_wait) { + best_wait = avg_wait; + best_stripe = index; + } + } + + return best_stripe; +} + struct stripe_mirror { u64 devid; int num;
@@ -6043,6 +6075,10 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,case BTRFS_READ_POLICY_ROTATION : preferred_mirror =btrfs_read_rotation (map, first, num_stripes);break ; +case BTRFS_READ_POLICY_LATENCY : + preferred_mirror =btrfs_best_stripe (fs_info, map, first, + num_stripes); +break ;#endif }
fs/btrfs/volumes.h +2 -0
@@ -306,6 +306,8 @@ enum btrfs_read_policy {#ifdef CONFIG_BTRFS_EXPERIMENTAL /* Balancing raid1 reads across all striped devices */ BTRFS_READ_POLICY_ROTATION , +/* Use the lowest-latency device dynamically */ +BTRFS_READ_POLICY_LATENCY , #endif BTRFS_NR_READ_POLICY, };
Signature · 1 line
2.46.1
PATCH v2 3/3 btrfs: add RAID1 preferred read device
In reply to Anand JainWhen there's stale data on a mirrored device, this feature lets you choose which device to read from. Mainly used for testing.
echo "devid:2" > /sys/fs/btrfs/<UUID>/read_policy
--- fs/btrfs/disk-io.c | 1 + fs/btrfs/sysfs.c | 32 ++++++++++++++++++++++++++++++-- fs/btrfs/volumes.c | 20 ++++++++++++++++++++ fs/btrfs/volumes.h | 5 +++++ 4 files changed, 56 insertions(+), 2 deletions(-)
fs/btrfs/disk-io.c +1 -0
@@ -3322,6 +3322,7 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device fs_info->sectorsize = sectorsize;#ifdef CONFIG_BTRFS_EXPERIMENTAL fs_info->fs_devices->min_contiguous_read = sectorsize; + fs_info->fs_devices->read_devid = fs_info->fs_devices->latest_dev->devid;#endif fs_info->sectorsize_bits =ilog2 (sectorsize); fs_info->sectors_per_page = (PAGE_SIZE >> fs_info->sectorsize_bits);
fs/btrfs/sysfs.c +30 -2
@@ -1306,7 +1306,7 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj, BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);#ifdef CONFIG_BTRFS_EXPERIMENTAL -static const char *const btrfs_read_policy_name[] = {"pid" ,"rotation" ,"latency" }; +static const char *const btrfs_read_policy_name[] = {"pid" ,"rotation" ,"latency" ,"devid" };#else static const char *const btrfs_read_policy_name[] = {"pid" };#endif
@@ -1332,8 +1332,11 @@ static ssize_t btrfs_read_policy_show(struct kobject *kobj,if (i ==BTRFS_READ_POLICY_ROTATION ) ret +=sysfs_emit_at (buf, ret,":%d" , fs_devices->min_contiguous_read); -#endif +if (i ==BTRFS_READ_POLICY_DEVID ) + ret +=sysfs_emit_at (buf, ret,":%llu" , + fs_devices->read_devid); +#endif if (i == policy) ret +=sysfs_emit_at (buf, ret,"]" ); }
@@ -1401,7 +1404,32 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj,return len; } + +if (index ==BTRFS_READ_POLICY_DEVID ) { +u64 value_devid; +BTRFS_DEV_LOOKUP_ARGS (args); + +if (value ==NULL ||kstrtou64 (value,10 , &value_devid)) +return -EINVAL ; + + args.devid = value_devid; +if (btrfs_find_device (fs_devices, &args) ==NULL ) +return -EINVAL ; + +if (index !=READ_ONCE (fs_devices->read_policy) || + (value_devid !=READ_ONCE (fs_devices->read_devid))) { +WRITE_ONCE (fs_devices->read_policy, index); +WRITE_ONCE (fs_devices->read_devid, value_devid); + +btrfs_info (fs_devices->fs_info,"read policy set to '%s:%llu'" , + btrfs_read_policy_name[index], value_devid); + + } + +return len; + }#endif +if (index !=READ_ONCE (fs_devices->read_policy)) {WRITE_ONCE (fs_devices->read_policy, index); btrfs_info(fs_devices->fs_info, "read policy set to '%s'",
fs/btrfs/volumes.c +20 -0
@@ -5966,6 +5966,23 @@ unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, }#ifdef CONFIG_BTRFS_EXPERIMENTAL +static int btrfs_read_preferred (struct btrfs_chunk_map *map,int first, +int num_stripe) +{ +int last = first + num_stripe; +int stripe_index; + +for (stripe_index = first; stripe_index < last; stripe_index++) { +struct btrfs_device *device = map->stripes[stripe_index].dev; + +if (device->devid ==READ_ONCE (device->fs_devices->read_devid)) +return stripe_index; + } + +/* If no read-preferred device, use first stripe */ +return first; +} +static int btrfs_best_stripe (struct btrfs_fs_info *fs_info,struct btrfs_chunk_map *map,int first,int num_stripe)
@@ -6079,6 +6096,9 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info, preferred_mirror =btrfs_best_stripe (fs_info, map, first, num_stripes);break ; +case BTRFS_READ_POLICY_DEVID : + preferred_mirror =btrfs_read_preferred (map, first, num_stripes); +break ;#endif }
fs/btrfs/volumes.h +5 -0
@@ -308,6 +308,8 @@ enum btrfs_read_policy {BTRFS_READ_POLICY_ROTATION ,/* Use the lowest-latency device dynamically */ BTRFS_READ_POLICY_LATENCY , +/* Read from the specific device */ +BTRFS_READ_POLICY_DEVID , #endif BTRFS_NR_READ_POLICY, };
@@ -442,6 +444,9 @@ struct btrfs_fs_devices {/* Min contiguous reads before switching to next device. */ int min_contiguous_read; +/* Device to be used for reading in case of RAID1. */ +u64 read_devid; +/* Checksum mode - offload it or do it synchronously. */ enum btrfs_offload_csum_mode offload_csum_mode;#endif
Signature · 1 line
2.46.1
Quoted 40 lines · Anand Jain
v2:
1. Move new features toCONFIG_BTRFS_EXPERIMENTALinstead ofCONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait toU64_MAXand remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy
The default value is the sector size, and the
min_contiguous_readvalue must be a multiple of the sector size.5. Tested FIO random read/write and defrag compression workloads with
min_contiguous_readset to sector size, 192k, and 256k.RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.
$ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
--ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
--time_based --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 20.3MiB/s | 20.5MiB/s | 313895 | 313895 | | rotation| | | | | | 4096| 20.4MiB/s | 20.5MiB/s | 313895 | 313895 | | 196608| 20.2MiB/s | 20.2MiB/s | 310152 | 310175 | | 262144| 20.3MiB/s | 20.4MiB/s | 312180 | 312191 | | latency| 18.4MiB/s | 18.4MiB/s | 272980 | 291683 | | devid:1 | 14.8MiB/s | 14.9MiB/s | 456376 | 0 |rotation RAID1 balancing technique performs more than 2x better for single-process defrag.
$ time -p btrfs filesystem defrag -r -f -c /btrfs
Quoted 11 lines
| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 18.00s| 3800 | 0 | | rotation| | | | | 4096| 8.95s| 1900 | 1901 | | 196608| 8.50s| 1881 | 1919 | | 262144| 8.80s| 1881 | 1919 | | latency | 17.18s| 3800 | 0 | | devid:2 | 17.48s| 0 | 3800 |
Copy and paste error. Please ignore the below paragraph. Thx. ---vvv--- ignore ---vvv----
Quoted 7 lines
Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.
---^^^------------^^^------
Quoted 83 lines
Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.
Latency is smarter with unstable block layer transport.
Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.
Devid is a hands-on approach, provides manual or user-space script control.
These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.
Thx.
--------- original v1 ------------
The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the
CONFIG_BTRFS_DEBUGconfig option and are on top of the previously added/sys/fs/btrfs/<UUID>/read_policyinterface to configure the desired RAID1 read balancing method.I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from
/sys/devices/<..>/statfor each device. Below is a summary of the results, with each result the average of three iterations.A typical generic random rw workload:
$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \ --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 29.4MiB/s | 29.5MiB/s | 456548 | 447975 | | rotation| 29.3MiB/s | 29.3MiB/s | 450105 | 450055 | | latency | 21.9MiB/s | 21.9MiB/s | 672387 | 0 | | devid:1 | 22.0MiB/s | 22.0MiB/s | 674788 | 0 |Defragmentation with compression workload:
$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo $ sync $ echo 3 > /proc/sys/vm/drop_caches $ btrfs filesystem defrag -f -c /btrfs/foo| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 21.61s| 3810 | 0 | | rotation| 11.55s| 1905 | 1905 | | latency | 20.99s| 0 | 3810 | | devid:2 | 21.41s| 0 | 3810 |. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.Anand Jain (3):
btrfs: introduce RAID1 round-robin read balancing
btrfs: use the path with the lowest latency for RAID1 reads
btrfs: add RAID1 preferred read devicefs/btrfs/disk-io.c | 4 ++ fs/btrfs/sysfs.c | 116 +++++++++++++++++++++++++++++++++++++++------ fs/btrfs/volumes.c | 109 ++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 16 +++++++ 4 files changed, 230 insertions(+), 15 deletions(-)
Quoted 11 lines · Anand Jain
v2:
1. Move new features toCONFIG_BTRFS_EXPERIMENTALinstead ofCONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait toU64_MAXand remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy
The default value is the sector size, and the
min_contiguous_readvalue must be a multiple of the sector size.
Overall, I'm fine with the latency and preferred device policies.
Meanwhile I'd prefer the previous version of round-robin, without the min_contiguous_read.
That looks a little overkilled, and I think we should keep the policy as simple as possible for now.
Mind to share why the min_contiguous_read is introduced in this update?
In the future, we should go the same method as sched_ext, by pushing the complex policies to eBPF programs.
Another future improvement is the interface, I'm fine with the sysfs knob for an experimental feature.
But from my last drop_subtree_threshold experience, sysfs is not going to be a user-friendly interface. It really relies on some user space daemon to set.
I'd prefer something more persistent, like some XATTR but inside root tree, and go with prop interfaces.
But that can all be done in the future.
Thanks,
Qu
Quoted 131 lines
5. Tested FIO random read/write and defrag compression workloads with
min_contiguous_readset to sector size, 192k, and 256k.RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.
$ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
--ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
--time_based --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 20.3MiB/s | 20.5MiB/s | 313895 | 313895 | | rotation| | | | | | 4096| 20.4MiB/s | 20.5MiB/s | 313895 | 313895 | | 196608| 20.2MiB/s | 20.2MiB/s | 310152 | 310175 | | 262144| 20.3MiB/s | 20.4MiB/s | 312180 | 312191 | | latency| 18.4MiB/s | 18.4MiB/s | 272980 | 291683 | | devid:1 | 14.8MiB/s | 14.9MiB/s | 456376 | 0 |rotation RAID1 balancing technique performs more than 2x better for single-process defrag.
$ time -p btrfs filesystem defrag -r -f -c /btrfs
| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 18.00s| 3800 | 0 | | rotation| | | | | 4096| 8.95s| 1900 | 1901 | | 196608| 8.50s| 1881 | 1919 | | 262144| 8.80s| 1881 | 1919 | | latency | 17.18s| 3800 | 0 | | devid:2 | 17.48s| 0 | 3800 |Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.
Latency is smarter with unstable block layer transport.
Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.
Devid is a hands-on approach, provides manual or user-space script control.
These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.
Thx.
--------- original v1 ------------
The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the
CONFIG_BTRFS_DEBUGconfig option and are on top of the previously added/sys/fs/btrfs/<UUID>/read_policyinterface to configure the desired RAID1 read balancing method.I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from
/sys/devices/<..>/statfor each device. Below is a summary of the results, with each result the average of three iterations.A typical generic random rw workload:
$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \ --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 29.4MiB/s | 29.5MiB/s | 456548 | 447975 | | rotation| 29.3MiB/s | 29.3MiB/s | 450105 | 450055 | | latency | 21.9MiB/s | 21.9MiB/s | 672387 | 0 | | devid:1 | 22.0MiB/s | 22.0MiB/s | 674788 | 0 |Defragmentation with compression workload:
$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo $ sync $ echo 3 > /proc/sys/vm/drop_caches $ btrfs filesystem defrag -f -c /btrfs/foo| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 21.61s| 3810 | 0 | | rotation| 11.55s| 1905 | 1905 | | latency | 20.99s| 0 | 3810 | | devid:2 | 21.41s| 0 | 3810 |. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.Anand Jain (3):
btrfs: introduce RAID1 round-robin read balancing
btrfs: use the path with the lowest latency for RAID1 reads
btrfs: add RAID1 preferred read devicefs/btrfs/disk-io.c | 4 ++ fs/btrfs/sysfs.c | 116 +++++++++++++++++++++++++++++++++++++++------ fs/btrfs/volumes.c | 109 ++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 16 +++++++ 4 files changed, 230 insertions(+), 15 deletions(-)
Quoted 17 lines · Qu Wenruo
Quoted 11 lines · Anand Jain
v2:
1. Move new features toCONFIG_BTRFS_EXPERIMENTALinstead ofCONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait toU64_MAXand remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/
read_policyThe default value is the sector size, and the
min_contiguous_readvalue must be a multiple of the sector size.Overall, I'm fine with the latency and preferred device policies.
Quoted 8 lines
Meanwhile I'd prefer the previous version of round-robin, without the
min_contiguous_read.That looks a little overkilled, and I think we should keep the policy as simple as possible for now.
Mind to share why the
min_contiguous_readis introduced in this update?
The reason for adding min_contiguous_read:
The block layer optimizes with bio merging to improve HDD performance. David mentioned on Slack that 192k to 256k co-reads can performance better, though I haven't seen this in my setup but it
may work in others.
Quoted 2 lines
In the future, we should go the same method as
sched_ext, by pushing the complex policies to eBPF programs.
External scripts for RAID1 balancing are achievable with BPF, though it require writable BPF, which is disabled in some cases. That said, still we should prioritize adding support and provide choice to the use-case to decide.
Quoted 2 lines
Another future improvement is the interface, I'm fine with the sysfs knob for an experimental feature.
Yes, we need to review tunables - mount options, sysfs, and btrfs properties to have a clear guidelines/consolidation.
Quoted 4 lines
But from my last
drop_subtree_thresholdexperience, sysfs is not going to be a user-friendly interface. It really relies on some user space daemon to set.
Agreed. However, for Btrfs, sysfs has been the most comprehensive so far.
Quoted 4 lines
I'd prefer something more persistent, like some XATTR but inside root tree, and go with prop interfaces.
But that can all be done in the future.
Absolutely. I included that in earlier experiments, but it was removed due to review comments. Now isn't the right time to reintroduce it; we can update the on-disk formats and xattrs once the in-memory graduates address specific use cases.
Thanks, Anand
Quoted 134 lines
Thanks,
QuQuoted 131 lines
5. Tested FIO random read/write and defrag compression workloads with
min_contiguous_readset to sector size, 192k, and 256k.RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.
$ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw -- bs=4k \ --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \ --time_based --group_reporting --name=iops-test-job --eta- newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 20.3MiB/s | 20.5MiB/s | 313895 | 313895 | | rotation| | | | | | 4096| 20.4MiB/s | 20.5MiB/s | 313895 | 313895 | | 196608| 20.2MiB/s | 20.2MiB/s | 310152 | 310175 | | 262144| 20.3MiB/s | 20.4MiB/s | 312180 | 312191 | | latency| 18.4MiB/s | 18.4MiB/s | 272980 | 291683 | | devid:1 | 14.8MiB/s | 14.9MiB/s | 456376 | 0 |rotation RAID1 balancing technique performs more than 2x better for single-process defrag.
$ time -p btrfs filesystem defrag -r -f -c /btrfs| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 18.00s| 3800 | 0 | | rotation| | | | | 4096| 8.95s| 1900 | 1901 | | 196608| 8.50s| 1881 | 1919 | | 262144| 8.80s| 1881 | 1919 | | latency | 17.18s| 3800 | 0 | | devid:2 | 17.48s| 0 | 3800 |Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.
Latency is smarter with unstable block layer transport.
Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.
Devid is a hands-on approach, provides manual or user-space script control.
These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.
Thx.
--------- original v1 ------------
The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the
CONFIG_BTRFS_DEBUGconfig option and are on top of the previously added/sys/fs/btrfs/<UUID>/read_policyinterface to configure the desired RAID1 read balancing method.I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from
/sys/devices/<..>/statfor each device. Below is a summary of the results, with each result the average of three iterations.A typical generic random rw workload:
$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 -- time_based \ --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 29.4MiB/s | 29.5MiB/s | 456548 | 447975 | | rotation| 29.3MiB/s | 29.3MiB/s | 450105 | 450055 | | latency | 21.9MiB/s | 21.9MiB/s | 672387 | 0 | | devid:1 | 22.0MiB/s | 22.0MiB/s | 674788 | 0 |Defragmentation with compression workload:
$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo $ sync $ echo 3 > /proc/sys/vm/drop_caches $ btrfs filesystem defrag -f -c /btrfs/foo| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 21.61s| 3810 | 0 | | rotation| 11.55s| 1905 | 1905 | | latency | 20.99s| 0 | 3810 | | devid:2 | 21.41s| 0 | 3810 |. The PID-based balancing method works well for the generic random rw fio workload.
. The rotation method is ideal when you want to keep both devices active, and it boosts performance in sequential defragmentation scenarios. . The latency-based method work well when we have mixed device types or when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and testing RAID1 mirror synchronizations.Anand Jain (3):
btrfs: introduce RAID1 round-robin read balancing
btrfs: use the path with the lowest latency for RAID1 reads
btrfs: add RAID1 preferred read device
fs/btrfs/disk-io.c| 4 ++
fs/btrfs/sysfs.c | 116 +++++++++++++++++++++++++++++++++++++++------fs/btrfs/volumes.c| 109 ++++++++++++++++++++++++++++++++++++++++++fs/btrfs/volumes.h| 16 +++++++4 files changed, 230 insertions(+), 15 deletions(-)
Quoted 11 lines · Anand Jain
v2:
1. Move new features toCONFIG_BTRFS_EXPERIMENTALinstead ofCONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait toU64_MAXand remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy
The default value is the sector size, and the
min_contiguous_readvalue must be a multiple of the sector size.
I think it's safe to start with the round-round robin policy, but the syntax is strange, why the [ ] are mandatory? Also please call it round-robin, or 'rr' for short.
The default of sector size is IMHO a wrong value, switching devices so often will drop the performance just because of the io request overhead. From what I rememer values around 200k were reasonable, so either 192k or 256k should be the default. We may also drop the configurable value at all and provide a few hard coded sizes like rr-256k, rr-512k, rr-1m, if not only to drop parsing of user strings.
Quoted 39 lines
5. Tested FIO random read/write and defrag compression workloads with
min_contiguous_readset to sector size, 192k, and 256k.RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation. $ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \ --time_based --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 20.3MiB/s | 20.5MiB/s | 313895 | 313895 | | rotation| | | | | | 4096| 20.4MiB/s | 20.5MiB/s | 313895 | 313895 | | 196608| 20.2MiB/s | 20.2MiB/s | 310152 | 310175 | | 262144| 20.3MiB/s | 20.4MiB/s | 312180 | 312191 | | latency| 18.4MiB/s | 18.4MiB/s | 272980 | 291683 | | devid:1 | 14.8MiB/s | 14.9MiB/s | 456376 | 0 |rotation RAID1 balancing technique performs more than 2x better for single-process defrag.
$ time -p btrfs filesystem defrag -r -f -c /btrfs| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 18.00s| 3800 | 0 | | rotation| | | | | 4096| 8.95s| 1900 | 1901 | | 196608| 8.50s| 1881 | 1919 | | 262144| 8.80s| 1881 | 1919 | | latency | 17.18s| 3800 | 0 | | devid:2 | 17.48s| 0 | 3800 |
Quoted 3 lines
Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
Yeah round-robin will be a good defalt, we only need to verify the chunk size and then do the switch in the next release.
Quoted 4 lines
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.
This sounds like he latency is good for a specific case and maybe a fallback if the device becomes faulty, but once the layer below becomes unstable we may need to skip reading from the device. This is also a different mode of operation than balancing reads.
Quoted 6 lines
Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.
Latency is smarter with unstable block layer transport.
Quoted 7 lines
Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.
Devid is a hands-on approach, provides manual or user-space script control.
These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.
To move forward with the feature I think the round robin and preferred device id can be merged. I'm not sure about the latency but if it's under experimental we can take it as is and tune later.
I'll check my notes from the last time Michal attempted to implement the policies if we haven't missed something.
Quoted 9 lines · Anand Jain
v2:
1. Move new features toCONFIG_BTRFS_EXPERIMENTALinstead ofCONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait toU64_MAXand remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy
Quoted 8 lines
The default value is the sector size, and the
min_contiguous_readvalue must be a multiple of the sector size.5. Tested FIO random read/write and defrag compression workloads with
min_contiguous_readset to sector size, 192k, and 256k.RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.
With this functionality added, would it not also make sense to add a RAID0/10 profile that limits the stripe width, so a stripe does not spawn more than n disk (for example n=4).
On systems with for example 24 disks in RAID10, a read may activate 12 disks at the same time which could easily saturate the bus.
Therefore if a storage profile that limits the number of devices a stripe occupy existed, it seems like there might be posibillities for RAID0/10 as well.
Note that as of writing this I believe that RAID0/10/5/6 make the stripe as wide as the number of storage devices available for the filesystem. If I am wrong about this please ignore my jabbering and move on.
Thanks for commenting.
Quoted 8 lines · David Sterba
Quoted 7 lines · Anand Jain
v2:
1. Move new features toCONFIG_BTRFS_EXPERIMENTALinstead ofCONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait toU64_MAXand remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:
Quoted 9 lines
Quoted 4 lines
echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy
The default value is the sector size, and the
min_contiguous_readvalue must be a multiple of the sector size.I think it's safe to start with the round-round robin policy, but the syntax is strange, why the [ ] are mandatory? Also please call it round-robin, or 'rr' for short.
I'm fine with round-robin. The [ ] part is not mandatory; if the min_contiguous_read value is not specified, it will default to a predefined value.
Quoted 7 lines
The default of sector size is IMHO a wrong value, switching devices so often will drop the performance just because of the io request overhead.
From what I rememer values around 200k were reasonable, so either 192k
or 256k should be the default. We may also drop the configurable value at all and provide a few hard coded sizes like rr-256k, rr-512k, rr-1m, if not only to drop parsing of user strings.
I'm okay with a default value of 256k. For the experimental feature, we can keep it configurable, allowing the opportunity to experiment with other values as well
Quoted 40 lines
Quoted 40 lines
5. Tested FIO random read/write and defrag compression workloads with
min_contiguous_readset to sector size, 192k, and 256k.RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.
$ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
--ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
--time_based --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 20.3MiB/s | 20.5MiB/s | 313895 | 313895 | | rotation| | | | | | 4096| 20.4MiB/s | 20.5MiB/s | 313895 | 313895 | | 196608| 20.2MiB/s | 20.2MiB/s | 310152 | 310175 | | 262144| 20.3MiB/s | 20.4MiB/s | 312180 | 312191 | | latency| 18.4MiB/s | 18.4MiB/s | 272980 | 291683 | | devid:1 | 14.8MiB/s | 14.9MiB/s | 456376 | 0 |rotation RAID1 balancing technique performs more than 2x better for single-process defrag.
$ time -p btrfs filesystem defrag -r -f -c /btrfs
| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 18.00s| 3800 | 0 | | rotation| | | | | 4096| 8.95s| 1900 | 1901 | | 196608| 8.50s| 1881 | 1919 | | 262144| 8.80s| 1881 | 1919 | | latency | 17.18s| 3800 | 0 | | devid:2 | 17.48s| 0 | 3800 |
Quoted 7 lines
Quoted 3 lines
Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
Yeah round-robin will be a good defalt, we only need to verify the chunk size and then do the switch in the next release.
Yes..
Quoted 10 lines
Quoted 4 lines
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.
This sounds like he latency is good for a specific case and maybe a fallback if the device becomes faulty, but once the layer below becomes unstable we may need to skip reading from the device. This is also a different mode of operation than balancing reads.
If the latency on the faulty path is so high that it shouldn't pick that path at all, so it works. However, the round-robin balancing is unaware of dynamic faults on the device path. IMO, a round-robin method that is latency aware (with ~20% variance) would be better.
Quoted 9 lines
Quoted 9 lines
Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.
Latency is smarter with unstable block layer transport.
Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.
Quoted 8 lines
Quoted 4 lines
Devid is a hands-on approach, provides manual or user-space script control.
These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.
To move forward with the feature I think the round robin and preferred device id can be merged. I'm not sure about the latency but if it's under experimental we can take it as is and tune later.
I hope the experimental feature also means we can change the name of the balancing method at any time. Once we have tested a fair combination of block device types, we'll definitely need a method that can automatically tune based on the device type, which will require adding or dropping balancing methods accordingly.
Quoted 2 lines
I'll check my notes from the last time Michal attempted to implement the policies if we haven't missed something.
Thanks, Anand
Quoted 19 lines · waxhead
Quoted 9 lines · Anand Jain
v2:
1. Move new features toCONFIG_BTRFS_EXPERIMENTALinstead ofCONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait toU64_MAXand remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/
read_policyQuoted 8 lines
The default value is the sector size, and the
min_contiguous_readvalue must be a multiple of the sector size.5. Tested FIO random read/write and defrag compression workloads with
min_contiguous_readset to sector size, 192k, and 256k.RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.
Quoted 7 lines
With this functionality added, would it not also make sense to add a RAID0/10 profile that limits the stripe width, so a stripe does not spawn more than n disk (for example n=4).
On systems with for example 24 disks in RAID10, a read may activate 12 > disks at the same time which could easily saturate the bus.
Therefore if a storage profile that limits the number of devices a stripe occupy existed, it seems like there might be posibillities for RAID0/10 as well.
Note that as of writing this I believe that RAID0/10/5/6 make the stripe as wide as the number of storage devices available for the filesystem. If I am wrong about this please ignore my jabbering and move on.
That's correct. I previously attempted to come up with a fix using the device grouping method. If there's a convincing and more generic way to specify how the devices should be grouped, we could consider that.
Thanks, Anand
Quoted 11 lines · Anand Jain
Quoted 3 lines
I think it's safe to start with the round-round robin policy, but the syntax is strange, why the [ ] are mandatory? Also please call it round-robin, or 'rr' for short.
I'm fine with round-robin. The [ ] part is not mandatory; if the
min_contiguous_readvalue is not specified, it will default to a predefined value.Quoted 2 lines
The default of sector size is IMHO a wrong value, switching devices so often will drop the performance just because of the io request overhead.
Quoted 8 lines
Quoted 4 lines
From what I rememer values around 200k were reasonable, so either 192k
or 256k should be the default. We may also drop the configurable value at all and provide a few hard coded sizes like rr-256k, rr-512k, rr-1m, if not only to drop parsing of user strings.I'm okay with a default value of 256k. For the experimental feature, we can keep it configurable, allowing the opportunity to experiment with other values as well
Yeah, for experimenting it makes sense to make it flexible, no need to patch and reboot the kernel. For final we should settle on some reasonable values.
Quoted 60 lines
Quoted 39 lines
Quoted 39 lines
5. Tested FIO random read/write and defrag compression workloads with
min_contiguous_readset to sector size, 192k, and 256k.RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.
$ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
--ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
--time_based --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 20.3MiB/s | 20.5MiB/s | 313895 | 313895 | | rotation| | | | | | 4096| 20.4MiB/s | 20.5MiB/s | 313895 | 313895 | | 196608| 20.2MiB/s | 20.2MiB/s | 310152 | 310175 | | 262144| 20.3MiB/s | 20.4MiB/s | 312180 | 312191 | | latency| 18.4MiB/s | 18.4MiB/s | 272980 | 291683 | | devid:1 | 14.8MiB/s | 14.9MiB/s | 456376 | 0 |rotation RAID1 balancing technique performs more than 2x better for single-process defrag.
$ time -p btrfs filesystem defrag -r -f -c /btrfs
| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 18.00s| 3800 | 0 | | rotation| | | | | 4096| 8.95s| 1900 | 1901 | | 196608| 8.50s| 1881 | 1919 | | 262144| 8.80s| 1881 | 1919 | | latency | 17.18s| 3800 | 0 | | devid:2 | 17.48s| 0 | 3800 |Quoted 7 lines
Quoted 3 lines
Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
Yeah round-robin will be a good defalt, we only need to verify the chunk size and then do the switch in the next release.
Yes..
Quoted 10 lines
Quoted 4 lines
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.
This sounds like he latency is good for a specific case and maybe a fallback if the device becomes faulty, but once the layer below becomes unstable we may need to skip reading from the device. This is also a different mode of operation than balancing reads.
Quoted 4 lines
If the latency on the faulty path is so high that it shouldn't pick that path at all, so it works. However, the round-robin balancing is unaware of dynamic faults on the device path. IMO, a round-robin method that is latency aware (with ~20% variance) would be better.
We should not mix the faulty device handling mode to the read balancing, at least for now. A back off algorithm that checks number of failed io requests should precede the balancing.
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.
Latency is smarter with unstable block layer transport.
Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.
Quoted 8 lines
Quoted 4 lines
Devid is a hands-on approach, provides manual or user-space script control.
These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.
To move forward with the feature I think the round robin and preferred device id can be merged. I'm not sure about the latency but if it's under experimental we can take it as is and tune later.
Quoted 5 lines
I hope the experimental feature also means we can change the name of the balancing method at any time. Once we have tested a fair combination of block device types, we'll definitely need a method that can automatically tune based on the device type, which will require adding or dropping balancing methods accordingly.
Yes we can change the names. The automatic tuning would need some feedback that measures the load and tries to improve the throughput, this is where we got stuck last time. So for now let's do some starightforward policy that on average works better than the current pid policy. I hope that tha round-robin-256k can be a good default, but of course we need more data for that.
Quoted 118 lines · David Sterba
Quoted 11 lines · Anand Jain
Quoted 3 lines
I think it's safe to start with the round-round robin policy, but the syntax is strange, why the [ ] are mandatory? Also please call it round-robin, or 'rr' for short.
I'm fine with round-robin. The [ ] part is not mandatory; if the
min_contiguous_readvalue is not specified, it will default to a predefined value.Quoted 2 lines
The default of sector size is IMHO a wrong value, switching devices so often will drop the performance just because of the io request overhead.
Quoted 8 lines
Quoted 4 lines
From what I rememer values around 200k were reasonable, so either 192k
or 256k should be the default. We may also drop the configurable value at all and provide a few hard coded sizes like rr-256k, rr-512k, rr-1m, if not only to drop parsing of user strings.I'm okay with a default value of 256k. For the experimental feature, we can keep it configurable, allowing the opportunity to experiment with other values as well
Yeah, for experimenting it makes sense to make it flexible, no need to patch and reboot the kernel. For final we should settle on some reasonable values.
Quoted 60 lines
Quoted 39 lines
Quoted 39 lines
5. Tested FIO random read/write and defrag compression workloads with
min_contiguous_readset to sector size, 192k, and 256k.RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.
$ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
--ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
--time_based --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 20.3MiB/s | 20.5MiB/s | 313895 | 313895 | | rotation| | | | | | 4096| 20.4MiB/s | 20.5MiB/s | 313895 | 313895 | | 196608| 20.2MiB/s | 20.2MiB/s | 310152 | 310175 | | 262144| 20.3MiB/s | 20.4MiB/s | 312180 | 312191 | | latency| 18.4MiB/s | 18.4MiB/s | 272980 | 291683 | | devid:1 | 14.8MiB/s | 14.9MiB/s | 456376 | 0 |rotation RAID1 balancing technique performs more than 2x better for single-process defrag.
$ time -p btrfs filesystem defrag -r -f -c /btrfs
| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 18.00s| 3800 | 0 | | rotation| | | | | 4096| 8.95s| 1900 | 1901 | | 196608| 8.50s| 1881 | 1919 | | 262144| 8.80s| 1881 | 1919 | | latency | 17.18s| 3800 | 0 | | devid:2 | 17.48s| 0 | 3800 |Quoted 7 lines
Quoted 3 lines
Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
Yeah round-robin will be a good defalt, we only need to verify the chunk size and then do the switch in the next release.
Yes..
Quoted 10 lines
Quoted 4 lines
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.
This sounds like he latency is good for a specific case and maybe a fallback if the device becomes faulty, but once the layer below becomes unstable we may need to skip reading from the device. This is also a different mode of operation than balancing reads.
Quoted 4 lines
If the latency on the faulty path is so high that it shouldn't pick that path at all, so it works. However, the round-robin balancing is unaware of dynamic faults on the device path. IMO, a round-robin method that is latency aware (with ~20% variance) would be better.
We should not mix the faulty device handling mode to the read balancing, at least for now. A back off algorithm that checks number of failed io requests should precede the balancing.
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.
Latency is smarter with unstable block layer transport.
Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.
Quoted 8 lines
Quoted 4 lines
Devid is a hands-on approach, provides manual or user-space script control.
These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.
To move forward with the feature I think the round robin and preferred device id can be merged. I'm not sure about the latency but if it's under experimental we can take it as is and tune later.
Quoted 5 lines
I hope the experimental feature also means we can change the name of the balancing method at any time. Once we have tested a fair combination of block device types, we'll definitely need a method that can automatically tune based on the device type, which will require adding or dropping balancing methods accordingly.
Quoted 6 lines
Yes we can change the names. The automatic tuning would need some feedback that measures the load and tries to improve the throughput, this is where we got stuck last time. So for now let's do some starightforward policy that on average works better than the current pid policy. I hope that tha round-robin-256k can be a good default, but of course we need more data for that.
Sending v3 with rotation renamed to round-robin. Code review appreciated; I'll wait a day.
Thanks, Anand
Quoted 3 lines · waxhead
Note that as of writing this I believe that RAID0/10/5/6 make the stripe as wide as the number of storage devices available for the filesystem. If I am wrong about this please ignore my jabbering and move on.
Nope, you're correct and this is a huge problem for bigger (in numbers of drives) arrays.
But it's also on my list of things I want to change in how we handle RAID with the RAID stripe-tree. This way we can do declustered RAID and ease on rebuild times. Also we can drastically enhance write parallelism to an array by directing different write streams to different sets of stripes. Which btw atm isn't even done for RAID1, as we're picking a block-group at a time until it's full which then gets written, instead of creating new block groups on new drive sets for different write streams (i.e. different files, etc..)
Quoted 15 lines · Anand Jain
v2:
1. Move new features toCONFIG_BTRFS_EXPERIMENTALinstead ofCONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait toU64_MAXand remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy
The default value is the sector size, and the
min_contiguous_readvalue must be a multiple of the sector size.5. Tested FIO random read/write and defrag compression workloads with
min_contiguous_readset to sector size, 192k, and 256k.
Quoted 6 lines
RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.
$ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
--ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
--time_based --group_reporting --name=iops-test-job --eta-newline=1
Although not 100% happy with the min_contiguous_read setting, since it's an optional one and still experimental, I'm fine with series so far.
Just want to express my concern about going mount option.
I know sysfs is not a good way to setup a lot of features, but mount option is way too committed to me, even under experimental features.
But I also understand without mount option it can be pretty hard to setup the read policy for fstests runs.
So I'd prefer to have some on-disk solution (XATTR or temporary items) to save the read policy.
It's less committed compared to mount option (aka, much easier to revert the change with breaking any compatibility), and can help for future features.
Thanks,
Qu
Quoted 121 lines
| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 20.3MiB/s | 20.5MiB/s | 313895 | 313895 | | rotation| | | | | | 4096| 20.4MiB/s | 20.5MiB/s | 313895 | 313895 | | 196608| 20.2MiB/s | 20.2MiB/s | 310152 | 310175 | | 262144| 20.3MiB/s | 20.4MiB/s | 312180 | 312191 | | latency| 18.4MiB/s | 18.4MiB/s | 272980 | 291683 | | devid:1 | 14.8MiB/s | 14.9MiB/s | 456376 | 0 |rotation RAID1 balancing technique performs more than 2x better for single-process defrag.
$ time -p btrfs filesystem defrag -r -f -c /btrfs
| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 18.00s| 3800 | 0 | | rotation| | | | | 4096| 8.95s| 1900 | 1901 | | 196608| 8.50s| 1881 | 1919 | | 262144| 8.80s| 1881 | 1919 | | latency | 17.18s| 3800 | 0 | | devid:2 | 17.48s| 0 | 3800 |Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.
Latency is smarter with unstable block layer transport.
Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.
Devid is a hands-on approach, provides manual or user-space script control.
These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.
Thx.
--------- original v1 ------------
The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the
CONFIG_BTRFS_DEBUGconfig option and are on top of the previously added/sys/fs/btrfs/<UUID>/read_policyinterface to configure the desired RAID1 read balancing method.I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from
/sys/devices/<..>/statfor each device. Below is a summary of the results, with each result the average of three iterations.A typical generic random rw workload:
$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \ --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 29.4MiB/s | 29.5MiB/s | 456548 | 447975 | | rotation| 29.3MiB/s | 29.3MiB/s | 450105 | 450055 | | latency | 21.9MiB/s | 21.9MiB/s | 672387 | 0 | | devid:1 | 22.0MiB/s | 22.0MiB/s | 674788 | 0 |Defragmentation with compression workload:
$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo $ sync $ echo 3 > /proc/sys/vm/drop_caches $ btrfs filesystem defrag -f -c /btrfs/foo| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 21.61s| 3810 | 0 | | rotation| 11.55s| 1905 | 1905 | | latency | 20.99s| 0 | 3810 | | devid:2 | 21.41s| 0 | 3810 |. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.Anand Jain (3):
btrfs: introduce RAID1 round-robin read balancing
btrfs: use the path with the lowest latency for RAID1 reads
btrfs: add RAID1 preferred read devicefs/btrfs/disk-io.c | 4 ++ fs/btrfs/sysfs.c | 116 +++++++++++++++++++++++++++++++++++++++------ fs/btrfs/volumes.c | 109 ++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 16 +++++++ 4 files changed, 230 insertions(+), 15 deletions(-)