raid1 balancing methods
cover raid1 balancing methods
The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the CONFIG_BTRFS_DEBUG config option and are on top of the previously added /sys/fs/btrfs/<UUID>/read_policy interface to configure the desired RAID1 read balancing method.
I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from /sys/devices/<..>/stat for each device. Below is a summary of the results, with each result the average of three iterations.
A typical generic random rw workload:
$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \ --group_reporting --name=iops-test-job --eta-newline=1
| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 29.4MiB/s | 29.5MiB/s | 456548 | 447975 | | rotation| 29.3MiB/s | 29.3MiB/s | 450105 | 450055 | | latency | 21.9MiB/s | 21.9MiB/s | 672387 | 0 | | devid:1 | 22.0MiB/s | 22.0MiB/s | 674788 | 0 |
Defragmentation with compression workload:
$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo $ sync $ echo 3 > /proc/sys/vm/drop_caches $ btrfs filesystem defrag -f -c /btrfs/foo
| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 21.61s| 3810 | 0 | | rotation| 11.55s| 1905 | 1905 | | latency | 20.99s| 0 | 3810 | | devid:2 | 21.41s| 0 | 3810 |
. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.
Anand Jain (3):
btrfs: introduce RAID1 round-robin read balancing
btrfs: use the path with the lowest latency for RAID1 reads
btrfs: add RAID1 preferred read device feature
fs/btrfs/sysfs.c | 94 ++++++++++++++++++++++++++++++------- fs/btrfs/volumes.c | 113 +++++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 14 ++++++ 3 files changed, 205 insertions(+), 16 deletions(-)
Signature · 1 line
2.46.0
PATCH 1/3 btrfs: introduce RAID1 round-robin read balancing
This feature balances I/O across the striped devices when reading from RAID1 blocks.
--- fs/btrfs/sysfs.c | 4 ++++ fs/btrfs/volumes.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 7 ++++++ 3 files changed, 64 insertions(+)
fs/btrfs/sysfs.c +4 -0
@@ -1305,7 +1305,11 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj, } BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show); +#ifdef CONFIG_BTRFS_DEBUG +static const char *const btrfs_read_policy_name[] = {"pid" ,"rotation" }; +#else static const char *const btrfs_read_policy_name[] = {"pid" }; +#endif static ssize_t btrfs_read_policy_show (struct kobject *kobj,struct kobj_attribute *a,char *buf)
fs/btrfs/volumes.c +53 -0
65 lines not on this page · raw message
fs/btrfs/volumes.h +7 -0
@@ -303,6 +303,10 @@ enum btrfs_chunk_allocation_policy {enum btrfs_read_policy {/* Use process PID to choose the stripe */ BTRFS_READ_POLICY_PID , +#ifdef CONFIG_BTRFS_DEBUG +/* Balancing raid1 reads across all striped devices */ +BTRFS_READ_POLICY_ROTATION , +#endif BTRFS_NR_READ_POLICY , };
@@ -431,6 +435,9 @@ struct btrfs_fs_devices {enum btrfs_read_policy read_policy;#ifdef CONFIG_BTRFS_DEBUG +/* read counter for the filesystem */ +atomic_t total_reads; +/* Checksum mode - offload it or do it synchronously. */ enum btrfs_offload_csum_mode offload_csum_mode;#endif
Signature · 1 line
2.46.0
Re: [PATCH 1/3] btrfs: introduce RAID1 round-robin read balancing
Quoted 31 lines · Anand Jain
This feature balances I/O across the striped devices when reading from RAID1 blocks.
Signed-off-by Anand Jain <[email protected]>--- fs/btrfs/sysfs.c | 4 ++++ fs/btrfs/volumes.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 7 ++++++ 3 files changed, 64 insertions(+)fs/btrfs/sysfs.c +4
-0@@ -1305,7 +1305,11 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj, } BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show); +#ifdef CONFIG_BTRFS_DEBUG +static const char *const btrfs_read_policy_name[] = {"pid" ,"rotation" }; +#else static const char *const btrfs_read_policy_name[] = {"pid" }; +#endif static ssize_t btrfs_read_policy_show (struct kobject *kobj,struct kobj_attribute *a,char *buf)fs/btrfs/volumes.c +0
-0
fs/btrfs/volumes.c int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len) · line 5862
Quoted 4 lines
fs/btrfs/volumes.c +1
-0@@ -5859,6 +5859,54 @@ int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len)return ret; } +#ifdef CONFIG_BTRFS_DEBUG
It would be much better to utilize CONFIG_BTRFS_EXPERIMENTAL. CONFIG_BTRFS_DEBUG is now for pure debug purposes.
Thanks,
Qu
Quoted 87 lines
fs/btrfs/volumes.c +52
-0+struct stripe_mirror { +u64 devid; +int map; +}; + +static int btrfs_cmp_devid (const void *a,const void *b) +{ +struct stripe_mirror *s1 = (struct stripe_mirror *)a; +struct stripe_mirror *s2 = (struct stripe_mirror *)b; + +if (s1->devid < s2->devid) +return -1 ; +if (s1->devid > s2->devid) +return 1 ; +return 0 ; +} + +static int btrfs_read_rotation (struct btrfs_chunk_map *map,int first, +int num_stripe) +{ +struct stripe_mirror stripes[4 ] = {0 };//4: for testing, works for now. +struct btrfs_fs_devices *fs_devices; +u64 devid; +int index, j, cnt; +int next_stripe; + + index =0 ; +for (j = first; j < first + num_stripe; j++) { + devid = map->stripes[j].dev->devid; + + stripes[index].devid = devid; + stripes[index].map = j; + + index++; + } + +sort (stripes, num_stripe,sizeof (struct stripe_mirror ), + btrfs_cmp_devid,NULL ); + + fs_devices = map->stripes[first].dev->fs_devices; + cnt =atomic_inc_return (&fs_devices->total_reads); + next_stripe = stripes[cnt % num_stripe].map; + +return next_stripe; +} +#endif +static int find_live_mirror (struct btrfs_fs_info *fs_info,struct btrfs_chunk_map *map,int first,int dev_replace_is_ongoing)@@ -5888,6 +5936,11 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,case BTRFS_READ_POLICY_PID : preferred_mirror = first + (current->pid % num_stripes);break ; +#ifdef CONFIG_BTRFS_DEBUG +case BTRFS_READ_POLICY_ROTATION : + preferred_mirror =btrfs_read_rotation (map, first, num_stripes); +break ; +#endif } if (dev_replace_is_ongoing &&fs/btrfs/volumes.h +7
-0@@ -303,6 +303,10 @@ enum btrfs_chunk_allocation_policy {enum btrfs_read_policy {/* Use process PID to choose the stripe */ BTRFS_READ_POLICY_PID , +#ifdef CONFIG_BTRFS_DEBUG +/* Balancing raid1 reads across all striped devices */ +BTRFS_READ_POLICY_ROTATION , +#endif BTRFS_NR_READ_POLICY , };@@ -431,6 +435,9 @@ struct btrfs_fs_devices {enum btrfs_read_policy read_policy;#ifdef CONFIG_BTRFS_DEBUG +/* read counter for the filesystem */ +atomic_t total_reads; +/* Checksum mode - offload it or do it synchronously. */ enum btrfs_offload_csum_mode offload_csum_mode;#endif
Re: [PATCH 1/3] btrfs: introduce RAID1 round-robin read balancing
Quoted 5 lines
Quoted 2 lines
+#ifdef
CONFIG_BTRFS_DEBUGIt would be much better to utilize
CONFIG_BTRFS_EXPERIMENTAL.CONFIG_BTRFS_DEBUGis now for pure debug purposes.
Yes, I noticed the recent patch that changed that. Fixed in v2.
Thanks, Anand
PATCH 2/3 btrfs: use the path with the lowest latency for RAID1 reads
In reply to Anand JainThis feature aims to direct the read I/O to the device with the lowest known latency for reading RAID1 blocks.
echo "latency" > /sys/fs/btrfs/<UUID>/read_policy
--- fs/btrfs/sysfs.c | 2 +- fs/btrfs/volumes.c | 40 ++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 2 ++ 3 files changed, 43 insertions(+), 1 deletion(-)
fs/btrfs/sysfs.c +1 -1
@@ -1306,7 +1306,7 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj, BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);#ifdef CONFIG_BTRFS_DEBUG -static const char *const btrfs_read_policy_name[] = {"pid" ,"rotation" }; +static const char *const btrfs_read_policy_name[] = {"pid" ,"rotation" ,"latency" };#else static const char *const btrfs_read_policy_name[] = {"pid" };#endif
fs/btrfs/volumes.c +40 -0
@@ -12,6 +12,9 @@#include <linux/uuid.h> #include <linux/list_sort.h> #include <linux/namei.h> +#ifdef CONFIG_BTRFS_DEBUG +#include <linux/part_stat.h> +#endif #include "misc.h" #include "ctree.h" #include "disk-io.h"
@@ -5860,6 +5863,39 @@ int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len) } #ifdef CONFIG_BTRFS_DEBUG +static int btrfs_best_stripe(struct btrfs_fs_info *fs_info, + struct btrfs_chunk_map *map, int first, + int num_stripe) +{ + u64 est_wait = 0; + int best_stripe = 0; + int index; + + for (index = first; index < first + num_stripe; index++) { + u64 read_wait; + u64 avg_wait = 0; + unsigned long read_ios; + struct btrfs_device *device = map->stripes[index].dev; + + read_wait = part_stat_read(device->bdev, nsecs[READ]); + read_ios = part_stat_read(device->bdev, ios[READ]); + + if (read_wait && read_ios && read_wait >= read_ios) + avg_wait = div_u64(read_wait, read_ios); + else + btrfs_debug_rl(fs_info, + "devid: %llu avg_wait ZERO read_wait %llu read_ios %lu", + device->devid, read_wait, read_ios); + + if (est_wait == 0 || est_wait > avg_wait) { + est_wait = avg_wait; + best_stripe = index; + } + } + + return best_stripe; +} + struct stripe_mirror { u64 devid; int map;
@@ -5940,6 +5976,10 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,case BTRFS_READ_POLICY_ROTATION : preferred_mirror =btrfs_read_rotation (map, first, num_stripes);break ; +case BTRFS_READ_POLICY_LATENCY : + preferred_mirror =btrfs_best_stripe (fs_info, map, first, + num_stripes); +break ;#endif }
fs/btrfs/volumes.h +2 -0
@@ -306,6 +306,8 @@ enum btrfs_read_policy {#ifdef CONFIG_BTRFS_DEBUG /* Balancing raid1 reads across all striped devices */ BTRFS_READ_POLICY_ROTATION , +/* Use the lowest-latency device dynamically */ +BTRFS_READ_POLICY_LATENCY , #endif BTRFS_NR_READ_POLICY, };
Signature · 1 line
2.46.0
Re: [PATCH 2/3] btrfs: use the path with the lowest latency for RAID1 reads
Quoted 40 lines · Anand Jain
This feature aims to direct the read I/O to the device with the lowest known latency for reading RAID1 blocks.
echo "latency" > /sys/fs/btrfs/<UUID>/read_policy
Signed-off-by Anand Jain <[email protected]>--- fs/btrfs/sysfs.c | 2 +- fs/btrfs/volumes.c | 40 ++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 2 ++ 3 files changed, 43 insertions(+), 1 deletion(-)fs/btrfs/sysfs.c +1
-1@@ -1306,7 +1306,7 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj, BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);#ifdef CONFIG_BTRFS_DEBUG -static const char *const btrfs_read_policy_name[] = {"pid" ,"rotation" }; +static const char *const btrfs_read_policy_name[] = {"pid" ,"rotation" ,"latency" };#else static const char *const btrfs_read_policy_name[] = {"pid" };#endif fs/btrfs/volumes.c +3
-0@@ -12,6 +12,9 @@#include <linux/uuid.h> #include <linux/list_sort.h> #include <linux/namei.h> +#ifdef CONFIG_BTRFS_DEBUG +#include <linux/part_stat.h> +#endif #include "misc.h" #include "ctree.h" #include "disk-io.h"
fs/btrfs/volumes.c int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len) · line 5870
Quoted 8 lines
fs/btrfs/volumes.c +5
-0@@ -5860,6 +5863,39 @@ int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len) }#ifdef CONFIG_BTRFS_DEBUG +static int btrfs_best_stripe (struct btrfs_fs_info *fs_info, +struct btrfs_chunk_map *map,int first, +int num_stripe) +{ +u64 est_wait =0 ;
Is this a typo of best_wait? Or do you mean estimated wait?
Quoted 8 lines
fs/btrfs/volumes.c +8
-0+int best_stripe =0 ; +int index; + +for (index = first; index < first + num_stripe; index++) { +u64 read_wait; +u64 avg_wait =0 ; +unsigned long read_ios; +struct btrfs_device *device = map->stripes[index].dev;
fs/btrfs/volumes.c int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len) · line 5888
Quoted 10 lines
fs/btrfs/volumes.c +10
-0+ + read_wait =part_stat_read (device->bdev, nsecs[READ ]); + read_ios =part_stat_read (device->bdev, ios[READ ]); + +if (read_wait && read_ios && read_wait >= read_ios) + avg_wait =div_u64 (read_wait, read_ios); +else +btrfs_debug_rl (fs_info, +"devid: %llu avg_wait ZERO read_wait %llu read_ios %lu" , + device->devid, read_wait, read_ios);
I do not think we need this debug messages.
The device can have no read so far.
fs/btrfs/volumes.c int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len) · line 5890
Quoted 2 lines
fs/btrfs/volumes.c +2
-0+ +if (est_wait ==0 || est_wait > avg_wait) {
You can give @est_wait a default value of U64_MAX, so that you do not need to check for 0, just est_wait > avg_wait is enough.
Thanks,
Qu
Quoted 35 lines
fs/btrfs/volumes.c +12
-0+ est_wait = avg_wait; + best_stripe = index; + } + } + +return best_stripe; +} +struct stripe_mirror {u64 devid;int map;@@ -5940,6 +5976,10 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,case BTRFS_READ_POLICY_ROTATION : preferred_mirror =btrfs_read_rotation (map, first, num_stripes);break ; +case BTRFS_READ_POLICY_LATENCY : + preferred_mirror =btrfs_best_stripe (fs_info, map, first, + num_stripes); +break ;#endif }fs/btrfs/volumes.h +2
-0@@ -306,6 +306,8 @@ enum btrfs_read_policy {#ifdef CONFIG_BTRFS_DEBUG /* Balancing raid1 reads across all striped devices */ BTRFS_READ_POLICY_ROTATION , +/* Use the lowest-latency device dynamically */ +BTRFS_READ_POLICY_LATENCY , #endif BTRFS_NR_READ_POLICY, };
Re: [PATCH 2/3] btrfs: use the path with the lowest latency for RAID1 reads
Quoted 20 lines
Quoted 20 lines
fs/btrfs/volumes.c +3
-0@@ -12,6 +12,9 @@ #include <linux/uuid.h> #include <linux/list_sort.h> #include <linux/namei.h> +#ifdef CONFIG_BTRFS_DEBUG +#include <linux/part_stat.h> +#endif #include "misc.h" #include "ctree.h" #include "disk-io.h"@@ -5860,6 +5863,39 @@ int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len) }
Quoted 9 lines
fs/btrfs/volumes.c int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len) · line 5870
Quoted 6 lines
fs/btrfs/volumes.c +5
-0#ifdef CONFIG_BTRFS_DEBUG +static int btrfs_best_stripe (struct btrfs_fs_info *fs_info, + struct btrfs_chunk_map *map, int first, + int num_stripe) +{ + u64 est_wait = 0;Is this a typo of
best_wait? Or do you mean estimated wait?
It is best_wait. Fixed in v2.
Quoted 21 lines
Quoted 8 lines
fs/btrfs/volumes.c +8
-0+ int best_stripe = 0; + int index; + + for (index = first; index < first + num_stripe; index++) { + u64 read_wait; + u64 avg_wait = 0; + unsigned long read_ios; + struct btrfs_device *device = map->stripes[index].dev;fs/btrfs/volumes.c int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len) · line 5888
Quoted 10 lines
fs/btrfs/volumes.c +10
-0+ + read_wait = part_stat_read(device->bdev, nsecs[READ]); + read_ios = part_stat_read(device->bdev, ios[READ]); + + if (read_wait && read_ios && read_wait >= read_ios) + avg_wait = div_u64(read_wait, read_ios); + else + btrfs_debug_rl(fs_info, + "devid: %llu avg_wait ZERO read_wait %llu read_ios %lu", + device->devid, read_wait, read_ios);I do not think we need this debug messages.
Quoted 2 lines
The device can have no read so far.
Um. Yeah, we can remove it.
Quoted 6 lines
fs/btrfs/volumes.c int btrfs_is_parity_mirror(struct btrfs_fs_info · line 5891
Quoted 2 lines
fs/btrfs/volumes.c +2
-0+ + if (est_wait == 0 || est_wait > avg_wait) {You can give @est_wait a default value of
U64_MAX, so that you do not need to check for 0, justest_wait>avg_waitis enough.
Fixed in v2.
Thanks for the review.
- Anand
Quoted 2 lines
Thanks,
Qu
PATCH 3/3 btrfs: add RAID1 preferred read device feature
In reply to Anand JainWhen there's stale data on a mirrored device, this feature lets you choose which device to read from. Mainly used for testing.
echo "devid:2" > /sys/fs/btrfs/<UUID>/read_policy
--- fs/btrfs/sysfs.c | 92 +++++++++++++++++++++++++++++++++++++--------- fs/btrfs/volumes.c | 20 ++++++++++ fs/btrfs/volumes.h | 5 +++ 3 files changed, 100 insertions(+), 17 deletions(-)
fs/btrfs/sysfs.c +75 -17
@@ -1306,7 +1306,7 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj, BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);#ifdef CONFIG_BTRFS_DEBUG -static const char *const btrfs_read_policy_name[] = {"pid" ,"rotation" ,"latency" }; +static const char *const btrfs_read_policy_name[] = {"pid" ,"rotation" ,"latency" ,"devid" };#else static const char *const btrfs_read_policy_name[] = {"pid" };#endif
@@ -1320,14 +1320,22 @@ static ssize_t btrfs_read_policy_show(struct kobject *kobj,int i;for (i =0 ; i <BTRFS_NR_READ_POLICY ; i++) { -if (policy == i) - ret +=sysfs_emit_at (buf, ret,"%s[%s]" , - (ret ==0 ?"" :" " ), - btrfs_read_policy_name[i]); -else - ret +=sysfs_emit_at (buf, ret,"%s%s" , - (ret ==0 ?"" :" " ), - btrfs_read_policy_name[i]); +if (ret !=0 ) + ret +=sysfs_emit_at (buf, ret," " ); + +if (i == policy) + ret +=sysfs_emit_at (buf, ret,"[" ); + + ret +=sysfs_emit_at (buf, ret,"%s" , btrfs_read_policy_name[i]); + +#ifdef CONFIG_BTRFS_DEBUG +if (i ==BTRFS_READ_POLICY_DEVID ) + ret +=sysfs_emit_at (buf, ret,":%llu" , + fs_devices->read_devid); +#endif + +if (i == policy) + ret +=sysfs_emit_at (buf, ret,"]" ); } ret +=sysfs_emit_at (buf, ret,"\n" );
@@ -1340,21 +1348,71 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj, const char *buf, size_t len) {struct btrfs_fs_devices *fs_devices =to_fs_devs (kobj); +char *value; +#ifdef CONFIG_BTRFS_DEBUG +u64 devid =0 ; +#endif +int index =-1 ;int i; +bool changed = false; + + value =strchr (buf,':' ); +if (value) { + *value ='\0' ; + value = value +1 ; + }for (i =0 ; i <BTRFS_NR_READ_POLICY ; i++) {if (sysfs_streq (buf, btrfs_read_policy_name[i])) { -if (i !=READ_ONCE (fs_devices->read_policy)) { -WRITE_ONCE (fs_devices->read_policy, i); -btrfs_info (fs_devices->fs_info, -"read policy set to '%s'" , - btrfs_read_policy_name[i]); - } -return len; + index = i; +break ; + } + } + +if (index ==-1 ) +return -EINVAL ; + +#ifdef CONFIG_BTRFS_DEBUG +/* Extract values from input in devid:value format */ +if (index ==BTRFS_READ_POLICY_DEVID ) { +BTRFS_DEV_LOOKUP_ARGS (args); + +if (value ==NULL ||kstrtou64 (value,10 , &devid)) +return -EINVAL ; + + args.devid = devid; +if (btrfs_find_device (fs_devices, &args) ==NULL ) +return -EINVAL ; + +if (READ_ONCE (fs_devices->read_devid) != devid) { +WRITE_ONCE (fs_devices->read_devid, devid); + changed = true; } } +#endif + +if (index !=READ_ONCE (fs_devices->read_policy)) { +WRITE_ONCE (fs_devices->read_policy, index); + changed = true; + } + +if (changed) { +#ifdef CONFIG_BTRFS_DEBUG +if (devid) +btrfs_info (fs_devices->fs_info, +"read policy set to '%s:%llu'" , + btrfs_read_policy_name[index], devid); +else +btrfs_info (fs_devices->fs_info, +"read policy set to '%s'" , + btrfs_read_policy_name[index]); +#else +btrfs_info (fs_devices->fs_info,"read policy set to '%s'" , + btrfs_read_policy_name[index]); +#endif + } -return -EINVAL ; +return len; } BTRFS_ATTR_RW(, read_policy, btrfs_read_policy_show, btrfs_read_policy_store);
fs/btrfs/volumes.c +20 -0
@@ -5863,6 +5863,23 @@ int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len) }#ifdef CONFIG_BTRFS_DEBUG +static int btrfs_read_preferred (struct btrfs_chunk_map *map,int first, +int num_stripe) +{ +int last = first + num_stripe; +int stripe_index; + +for (stripe_index = first; stripe_index < last; stripe_index++) { +struct btrfs_device *device = map->stripes[stripe_index].dev; + +if (device->devid ==READ_ONCE (device->fs_devices->read_devid)) +return stripe_index; + } + +/* If no read-preferred device, use first stripe */ +return first; +} +static int btrfs_best_stripe (struct btrfs_fs_info *fs_info,struct btrfs_chunk_map *map,int first,int num_stripe)
@@ -5980,6 +5997,9 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info, preferred_mirror =btrfs_best_stripe (fs_info, map, first, num_stripes);break ; +case BTRFS_READ_POLICY_DEVID : + preferred_mirror =btrfs_read_preferred (map, first, num_stripes); +break ;#endif }
fs/btrfs/volumes.h +5 -0
@@ -308,6 +308,8 @@ enum btrfs_read_policy {BTRFS_READ_POLICY_ROTATION ,/* Use the lowest-latency device dynamically */ BTRFS_READ_POLICY_LATENCY , +/* Read from the specific device */ +BTRFS_READ_POLICY_DEVID , #endif BTRFS_NR_READ_POLICY, };
@@ -440,6 +442,9 @@ struct btrfs_fs_devices {/* read counter for the filesystem */ atomic_t total_reads; +/* Device to be used for reading in case of RAID1 */ +u64 read_devid; +/* Checksum mode - offload it or do it synchronously. */ enum btrfs_offload_csum_mode offload_csum_mode;#endif
Signature · 1 line
2.46.0
Quoted 6 lines · Anand Jain
The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the
CONFIG_BTRFS_DEBUGconfig option and are on top of the previously added/sys/fs/btrfs/<UUID>/read_policyinterface to configure the desired RAID1 read balancing method.
I am currently testing this on 6.12-rc1 with policy rotation, seems good for now. Would be better if policy can be set in mount options.
HAN Yuwei
Quoted 58 lines
I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from
/sys/devices/<..>/statfor each device. Below is a summary of the results, with each result the average of three iterations.A typical generic random rw workload:
$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \ --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 29.4MiB/s | 29.5MiB/s | 456548 | 447975 | | rotation| 29.3MiB/s | 29.3MiB/s | 450105 | 450055 | | latency | 21.9MiB/s | 21.9MiB/s | 672387 | 0 | | devid:1 | 22.0MiB/s | 22.0MiB/s | 674788 | 0 |Defragmentation with compression workload:
$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo $ sync $ echo 3 > /proc/sys/vm/drop_caches $ btrfs filesystem defrag -f -c /btrfs/foo| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 21.61s| 3810 | 0 | | rotation| 11.55s| 1905 | 1905 | | latency | 20.99s| 0 | 3810 | | devid:2 | 21.41s| 0 | 3810 |. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.Anand Jain (3):
btrfs: introduce RAID1 round-robin read balancing
btrfs: use the path with the lowest latency for RAID1 reads
btrfs: add RAID1 preferred read device featurefs/btrfs/sysfs.c | 94 ++++++++++++++++++++++++++++++------- fs/btrfs/volumes.c | 113 +++++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 14 ++++++ 3 files changed, 205 insertions(+), 16 deletions(-)
Quoted 9 lines · Yuwei Han
Quoted 6 lines · Anand Jain
The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the
CONFIG_BTRFS_DEBUGconfig option and are on top of the previously added/sys/fs/btrfs/<UUID>/read_policyinterface to configure the desired RAID1 read balancing method.I am currently testing this on 6.12-rc1 with policy rotation, seems good for now.
Thanks for testing and reviewing.
Quoted 1 line
Would be better if policy can be set in mount options.
I think it is a good idea. However, we should also consolidate our sysfs knobs, mount options, and btrfs properties all together where applicable and make it easy to use.
V2 is in the ML.
Thanks,
-Anand
Quoted 61 lines
HAN Yuwei
Quoted 58 lines
I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from
/sys/devices/<..>/statfor each device. Below is a summary of the results, with each result the average of three iterations.A typical generic random rw workload:
$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 -- time_based \ --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 29.4MiB/s | 29.5MiB/s | 456548 | 447975 | | rotation| 29.3MiB/s | 29.3MiB/s | 450105 | 450055 | | latency | 21.9MiB/s | 21.9MiB/s | 672387 | 0 | | devid:1 | 22.0MiB/s | 22.0MiB/s | 674788 | 0 |Defragmentation with compression workload:
$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo $ sync $ echo 3 > /proc/sys/vm/drop_caches $ btrfs filesystem defrag -f -c /btrfs/foo| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 21.61s| 3810 | 0 | | rotation| 11.55s| 1905 | 1905 | | latency | 20.99s| 0 | 3810 | | devid:2 | 21.41s| 0 | 3810 |. The PID-based balancing method works well for the generic random rw fio workload.
. The rotation method is ideal when you want to keep both devices active, and it boosts performance in sequential defragmentation scenarios. . The latency-based method work well when we have mixed device types or when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and testing RAID1 mirror synchronizations.Anand Jain (3):
btrfs: introduce RAID1 round-robin read balancing
btrfs: use the path with the lowest latency for RAID1 reads
btrfs: add RAID1 preferred read device featurefs/btrfs/sysfs.c | 94 ++++++++++++++++++++++++++++++-------
fs/btrfs/volumes.c| 113 +++++++++++++++++++++++++++++++++++++++++++++fs/btrfs/volumes.h| 14 ++++++3 files changed, 205 insertions(+), 16 deletions(-)