raid1 balancing methods
v3 cover raid1 balancing methods
v3:
1. Removed the latency-based RAID1 balancing patch. (Per David's review)
2. Renamed "rotation" to "round-robin" and set the per-set min_contiguous_read to 256k. (Per David's review)
3. Added raid1-balancing module configuration for fstests testing. raid1-balancing can now be configured through both module load parameters and sysfs.
The logic of individual methods remains unchanged, and performance metrics are consistent with v2.
-----
v2:
1. Move new features to CONFIG_BTRFS_EXPERIMENTAL instead of CONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait to U64_MAX and remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:
echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy
The default value is the sector size, and the min_contiguous_read value must be a multiple of the sector size.
5. Tested FIO random read/write and defrag compression workloads with min_contiguous_read set to sector size, 192k, and 256k.
RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.
$ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
--ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
--time_based --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 20.3MiB/s | 20.5MiB/s | 313895 | 313895 | | rotation| | | | | | 4096| 20.4MiB/s | 20.5MiB/s | 313895 | 313895 | | 196608| 20.2MiB/s | 20.2MiB/s | 310152 | 310175 | | 262144| 20.3MiB/s | 20.4MiB/s | 312180 | 312191 | | latency| 18.4MiB/s | 18.4MiB/s | 272980 | 291683 | | devid:1 | 14.8MiB/s | 14.9MiB/s | 456376 | 0 |
rotation RAID1 balancing technique performs more than 2x better for single-process defrag.
$ time -p btrfs filesystem defrag -r -f -c /btrfs
| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 18.00s| 3800 | 0 | | rotation| | | | | 4096| 8.95s| 1900 | 1901 | | 196608| 8.50s| 1881 | 1919 | | 262144| 8.80s| 1881 | 1919 | | latency | 17.18s| 3800 | 0 | | devid:2 | 17.48s| 0 | 3800 |
Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.
Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.
Latency is smarter with unstable block layer transport.
Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.
Devid is a hands-on approach, provides manual or user-space script control.
These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.
Thx.
--------- original v1 ------------
The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the CONFIG_BTRFS_DEBUG config option and are on top of the previously added /sys/fs/btrfs/<UUID>/read_policy interface to configure the desired RAID1 read balancing method.
I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from /sys/devices/<..>/stat for each device. Below is a summary of the results, with each result the average of three iterations.
A typical generic random rw workload:
$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \ --group_reporting --name=iops-test-job --eta-newline=1
| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 29.4MiB/s | 29.5MiB/s | 456548 | 447975 | | rotation| 29.3MiB/s | 29.3MiB/s | 450105 | 450055 | | latency | 21.9MiB/s | 21.9MiB/s | 672387 | 0 | | devid:1 | 22.0MiB/s | 22.0MiB/s | 674788 | 0 |
Defragmentation with compression workload:
$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo $ sync $ echo 3 > /proc/sys/vm/drop_caches $ btrfs filesystem defrag -f -c /btrfs/foo
| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 21.61s| 3810 | 0 | | rotation| 11.55s| 1905 | 1905 | | latency | 20.99s| 0 | 3810 | | devid:2 | 21.41s| 0 | 3810 |
. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.
Anand Jain (10):
btrfs: initialize fs_devices->fs_info earlier
btrfs: simplify output formatting in btrfs_read_policy_show
btrfs: add btrfs_read_policy_to_enum helper and refactor read policy
store
btrfs: handle value associated with raid1 balancing parameter
btrfs: introduce RAID1 round-robin read balancing
btrfs: add RAID1 preferred read device
btrfs: pr CONFIG_BTRFS_EXPERIMENTAL status
btrfs: fix CONFIG_BTRFS_EXPERIMENTAL migration
btrfs: enable RAID1 balancing configuration via modprobe parameter
btrfs: modload to print RAID1 balancing status
fs/btrfs/disk-io.c | 1 + fs/btrfs/super.c | 22 +++++- fs/btrfs/sysfs.c | 181 ++++++++++++++++++++++++++++++++++++++++----- fs/btrfs/sysfs.h | 5 ++ fs/btrfs/volumes.c | 86 ++++++++++++++++++++- fs/btrfs/volumes.h | 14 ++++ 6 files changed, 286 insertions(+), 23 deletions(-)
Signature · 1 line
2.46.1
PATCH v3 1/10 btrfs: initialize fs_devices->fs_info earlier
Currently, fs_devices->fs_info is initialized in btrfs_init_devices_late(), but this occurs too late for find_live_mirror(), which is invoked by load_super_root() much earlier than btrfs_init_devices_late().
Fix this by moving the initialization to open_ctree(), before load_super_root().
--- fs/btrfs/disk-io.c | 1 + fs/btrfs/volumes.c | 2 -- 2 files changed, 1 insertion(+), 2 deletions(-)
fs/btrfs/disk-io.c +1 -0
@@ -3321,6 +3321,7 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device fs_info->sectors_per_page = (PAGE_SIZE >> fs_info->sectorsize_bits); fs_info->csums_per_leaf =BTRFS_MAX_ITEM_SIZE (fs_info) / fs_info->csum_size; fs_info->stripesize = stripesize; + fs_info->fs_devices->fs_info = fs_info;/* * Handle the space caching options appropriately now that we have the
fs/btrfs/volumes.c +0 -2
@@ -7568,8 +7568,6 @@ int btrfs_init_devices_late(struct btrfs_fs_info *fs_info)struct btrfs_device *device;int ret =0 ; - fs_devices->fs_info = fs_info; -mutex_lock (&fs_devices->device_list_mutex);list_for_each_entry (device, &fs_devices->devices, dev_list) device->fs_info = fs_info;
Signature · 1 line
2.46.1
PATCH v3 2/10 btrfs: simplify output formatting in btrfs_read_policy_show
In reply to Anand JainRefactor the logic in btrfs_read_policy_show() to streamline the formatting of read policies output. Streamline the space and bracket handling around the active policy without altering the functional output. This is in preparation to add more methods.
--- fs/btrfs/sysfs.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)
fs/btrfs/sysfs.c +10 -8
@@ -1316,14 +1316,16 @@ static ssize_t btrfs_read_policy_show(struct kobject *kobj,int i;for (i =0 ; i <BTRFS_NR_READ_POLICY ; i++) { -if (policy == i) - ret +=sysfs_emit_at (buf, ret,"%s[%s]" , - (ret ==0 ?"" :" " ), - btrfs_read_policy_name[i]); -else - ret +=sysfs_emit_at (buf, ret,"%s%s" , - (ret ==0 ?"" :" " ), - btrfs_read_policy_name[i]); +if (ret !=0 ) + ret +=sysfs_emit_at (buf, ret," " ); + +if (i == policy) + ret +=sysfs_emit_at (buf, ret,"[" ); + + ret +=sysfs_emit_at (buf, ret,"%s" , btrfs_read_policy_name[i]); + +if (i == policy) + ret +=sysfs_emit_at (buf, ret,"]" ); } ret +=sysfs_emit_at (buf, ret,"\n" );
Signature · 1 line
2.46.1
PATCH v3 3/10 btrfs: add btrfs_read_policy_to_enum helper and refactor read policy store
In reply to Anand JainIntroduce the btrfs_read_policy_to_enum helper function to simplify the conversion of a string read policy to its corresponding enum value. This reduces duplication and improves code clarity in btrfs_read_policy_store. The btrfs_read_policy_store function has been refactored to use the new helper.
The parameter is copied locally to allow modification, enabling the separation of the method and its value. This prepares for the addition of more functionality in subsequent patches.
--- fs/btrfs/sysfs.c | 50 ++++++++++++++++++++++++++++++++++++------------ 1 file changed, 38 insertions(+), 12 deletions(-)
fs/btrfs/sysfs.c +38 -12
@@ -1307,6 +1307,34 @@ BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);static const char *const btrfs_read_policy_name[] = {"pid" }; +static enum btrfs_read_policy btrfs_read_policy_to_enum (const char *str) +{ +bool found = false; +enum btrfs_read_policy index; +char *param; + +if (!str || !strlen (str)) +return 0 ; + + param =kstrdup (str,GFP_KERNEL ); +if (!param) +return -ENOMEM ; + +for (index =0 ; index <BTRFS_NR_READ_POLICY ; index++) { +if (sysfs_streq (param, btrfs_read_policy_name[index])) { + found = true; +break ; + } + } + +kfree (param); + +if (found) +return index; + +return -EINVAL ; +} +static ssize_t btrfs_read_policy_show (struct kobject *kobj,struct kobj_attribute *a,char *buf) {
@@ -1338,21 +1366,19 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj, const char *buf, size_t len) {struct btrfs_fs_devices *fs_devices =to_fs_devs (kobj); -int i; +enum btrfs_read_policy index; -for (i =0 ; i <BTRFS_NR_READ_POLICY ; i++) { -if (sysfs_streq (buf, btrfs_read_policy_name[i])) { -if (i !=READ_ONCE (fs_devices->read_policy)) { -WRITE_ONCE (fs_devices->read_policy, i); -btrfs_info (fs_devices->fs_info, -"read policy set to '%s'" , - btrfs_read_policy_name[i]); - } -return len; - } + index =btrfs_read_policy_to_enum (buf); +if (index == -EINVAL ) +return -EINVAL ; + +if (index !=READ_ONCE (fs_devices->read_policy)) { +WRITE_ONCE (fs_devices->read_policy, index); +btrfs_info (fs_devices->fs_info,"read policy set to '%s'" , + btrfs_read_policy_name[index]); } -return -EINVAL ; +return len; } BTRFS_ATTR_RW(, read_policy, btrfs_read_policy_show, btrfs_read_policy_store);
Signature · 1 line
2.46.1
PATCH v3 4/10 btrfs: handle value associated with raid1 balancing parameter
In reply to Anand JainThis change enables specifying additional configuration values alongside the raid1 balancing / read policy in a single input string.
Updated btrfs_read_policy_to_enum() to parse and handle a value associated with the policy in the format policy:value, the value part if present is converted 64-bit integer. Update btrfs_read_policy_store() to accommodate the new parameter.
--- fs/btrfs/sysfs.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)
fs/btrfs/sysfs.c +18 -2
@@ -1307,8 +1307,11 @@ BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);static const char *const btrfs_read_policy_name[] = {"pid" }; -static enum btrfs_read_policy btrfs_read_policy_to_enum (const char *str) +static enum btrfs_read_policy btrfs_read_policy_to_enum (const char *str,s64 *value) { +#ifdef CONFIG_BTRFS_EXPERIMENTAL +char *value_str; +#endif bool found = false;enum btrfs_read_policy index;char *param;
@@ -1320,6 +1323,18 @@ static enum btrfs_read_policy btrfs_read_policy_to_enum(const char *str)if (!param)return -ENOMEM ; +#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Separate value from input in policy:value format. */ +if ((value_str =strchr (param,':' ))) { + *value_str ='\0' ; + value_str++; +if (value &&kstrtou64 (value_str,10 , value) !=0 ) { +kfree (param); +return -EINVAL ; + } + } +#endif +for (index =0 ; index <BTRFS_NR_READ_POLICY ; index++) {if (sysfs_streq (param, btrfs_read_policy_name[index])) { found = true;
@@ -1367,8 +1382,9 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj, {struct btrfs_fs_devices *fs_devices =to_fs_devs (kobj);enum btrfs_read_policy index; +s64 value =-1 ; - index =btrfs_read_policy_to_enum (buf); + index =btrfs_read_policy_to_enum (buf, &value);if (index == -EINVAL )return -EINVAL ;
Signature · 1 line
2.46.1
PATCH v3 5/10 btrfs: introduce RAID1 round-robin read balancing
In reply to Anand JainThis feature balances I/O across the striped devices when reading from RAID1 blocks.
echo round-robin:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy
The min_contiguous_read parameter defines the minimum read size before switching to the next mirrored device. This setting is optional, with a default value of 256 KiB.
--- fs/btrfs/sysfs.c | 38 +++++++++++++++++++++++++++++ fs/btrfs/volumes.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 9 +++++++ 3 files changed, 107 insertions(+)
fs/btrfs/sysfs.c +38 -0
@@ -1305,7 +1305,11 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj, } BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show); +#ifdef CONFIG_BTRFS_EXPERIMENTAL +static const char *const btrfs_read_policy_name[] = {"pid" ,"round-robin" }; +#else static const char *const btrfs_read_policy_name[] = {"pid" }; +#endif static enum btrfs_read_policy btrfs_read_policy_to_enum (const char *str,s64 *value) {
@@ -1367,6 +1371,12 @@ static ssize_t btrfs_read_policy_show(struct kobject *kobj, ret +=sysfs_emit_at (buf, ret,"%s" , btrfs_read_policy_name[i]); +#ifdef CONFIG_BTRFS_EXPERIMENTAL +if (i ==BTRFS_READ_POLICY_RR ) + ret +=sysfs_emit_at (buf, ret,":%d" , + fs_devices->min_contiguous_read); +#endif +if (i == policy) ret +=sysfs_emit_at (buf, ret,"]" ); }
@@ -1388,6 +1398,34 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj,if (index == -EINVAL )return -EINVAL ; +#ifdef CONFIG_BTRFS_EXPERIMENTAL +if (index ==BTRFS_READ_POLICY_RR ) { +if (value !=-1 ) { +if ((value % fs_devices->fs_info->sectorsize) !=0 ) { +btrfs_err (fs_devices->fs_info, +"read_policy: min_contiguous_read %lld should be multiples of the sectorsize %u" , + value, fs_devices->fs_info->sectorsize); +return -EINVAL ; + } + }else { +/* value is not provided, set it to the default 256k */ + value =256 *1024 ; + } + +if (index !=READ_ONCE (fs_devices->read_policy) || + value !=READ_ONCE (fs_devices->min_contiguous_read)) { +WRITE_ONCE (fs_devices->read_policy, index); +WRITE_ONCE (fs_devices->min_contiguous_read, value); +atomic_set (&fs_devices->total_reads,0 ); + +btrfs_info (fs_devices->fs_info,"read policy set to '%s:%lld'" , + btrfs_read_policy_name[index], value); + + } + +return len; + } +#endif if (index !=READ_ONCE (fs_devices->read_policy)) {WRITE_ONCE (fs_devices->read_policy, index); btrfs_info(fs_devices->fs_info, "read policy set to '%s'",
fs/btrfs/volumes.c +60 -0
78 lines not on this page · raw message
fs/btrfs/volumes.h +9 -0
@@ -303,6 +303,10 @@ enum btrfs_chunk_allocation_policy {enum btrfs_read_policy {/* Use process PID to choose the stripe */ BTRFS_READ_POLICY_PID , +#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Balancing raid1 reads across all striped devices (round-robin) */ +BTRFS_READ_POLICY_RR , +#endif BTRFS_NR_READ_POLICY , };
@@ -431,6 +435,11 @@ struct btrfs_fs_devices {enum btrfs_read_policy read_policy;#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* IO stat, read counter. */ +atomic_t total_reads; +/* Min contiguous reads before switching to next device. */ +int min_contiguous_read; +/* Checksum mode - offload it or do it synchronously. */ enum btrfs_offload_csum_mode offload_csum_mode;#endif
Signature · 1 line
2.46.1
PATCH v3 6/10 btrfs: add RAID1 preferred read device
In reply to Anand JainWhen there's stale data on a mirrored device, this feature lets you choose which device to read from. Mainly used for testing.
echo "devid:<devid-value>" > /sys/fs/btrfs/<UUID>/read_policy
--- fs/btrfs/sysfs.c | 34 ++++++++++++++++++++++++++++++++-- fs/btrfs/volumes.c | 21 +++++++++++++++++++++ fs/btrfs/volumes.h | 5 +++++ 3 files changed, 58 insertions(+), 2 deletions(-)
fs/btrfs/sysfs.c +32 -2
@@ -1306,7 +1306,7 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj, BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);#ifdef CONFIG_BTRFS_EXPERIMENTAL -static const char *const btrfs_read_policy_name[] = {"pid" ,"round-robin" }; +static const char *const btrfs_read_policy_name[] = {"pid" ,"round-robin" ,"devid" };#else static const char *const btrfs_read_policy_name[] = {"pid" };#endif
@@ -1375,8 +1375,11 @@ static ssize_t btrfs_read_policy_show(struct kobject *kobj,if (i ==BTRFS_READ_POLICY_RR ) ret +=sysfs_emit_at (buf, ret,":%d" , fs_devices->min_contiguous_read); -#endif +if (i ==BTRFS_READ_POLICY_DEVID ) + ret +=sysfs_emit_at (buf, ret,":%llu" , + fs_devices->read_devid); +#endif if (i == policy) ret +=sysfs_emit_at (buf, ret,"]" ); }
@@ -1425,6 +1428,33 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj,return len; } + +if (index ==BTRFS_READ_POLICY_DEVID ) { + +if (value !=-1 ) { +BTRFS_DEV_LOOKUP_ARGS (args); + +/* Validate input devid */ + args.devid = value; +if (btrfs_find_device (fs_devices, &args) ==NULL ) +return -EINVAL ; + }else { +/* Set default devid to the devid of the latest device */ + value = fs_devices->latest_dev->devid; + } + +if (index !=READ_ONCE (fs_devices->read_policy) || + (value !=READ_ONCE (fs_devices->read_devid))) { +WRITE_ONCE (fs_devices->read_policy, index); +WRITE_ONCE (fs_devices->read_devid, value); + +btrfs_info (fs_devices->fs_info,"read policy set to '%s:%llu'" , + btrfs_read_policy_name[index], value); + + } + +return len; + }#endif if (index !=READ_ONCE (fs_devices->read_policy)) {WRITE_ONCE (fs_devices->read_policy, index);
fs/btrfs/volumes.c +21 -0
@@ -1331,6 +1331,7 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices,#ifdef CONFIG_BTRFS_EXPERIMENTAL /* Set min_contiguous_read to a default 256kib */ fs_devices->min_contiguous_read =256 *1024 ; + fs_devices->read_devid = latest_dev->devid;#endif return 0 ;
@@ -5964,6 +5965,23 @@ unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, } #ifdef CONFIG_BTRFS_EXPERIMENTAL +static int btrfs_read_preferred(struct btrfs_chunk_map *map, int first, + int num_stripe) +{ + int last = first + num_stripe; + int stripe_index; + + for (stripe_index = first; stripe_index < last; stripe_index++) { + struct btrfs_device *device = map->stripes[stripe_index].dev; + + if (device->devid == READ_ONCE(device->fs_devices->read_devid)) + return stripe_index; + } + + /* If no read-preferred device, use first stripe */ + return first; +} + struct stripe_mirror { u64 devid; int num;
@@ -6047,6 +6065,9 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,case BTRFS_READ_POLICY_RR : preferred_mirror =btrfs_read_rr (map, first, num_stripes);break ; +case BTRFS_READ_POLICY_DEVID : + preferred_mirror =btrfs_read_preferred (map, first, num_stripes); +break ;#endif }
fs/btrfs/volumes.h +5 -0
@@ -306,6 +306,8 @@ enum btrfs_read_policy {#ifdef CONFIG_BTRFS_EXPERIMENTAL /* Balancing raid1 reads across all striped devices (round-robin) */ BTRFS_READ_POLICY_RR , +/* Read from the specific device */ +BTRFS_READ_POLICY_DEVID , #endif BTRFS_NR_READ_POLICY, };
@@ -440,6 +442,9 @@ struct btrfs_fs_devices {/* Min contiguous reads before switching to next device. */ int min_contiguous_read; +/* Device to be used for reading in case of RAID1. */ +u64 read_devid; +/* Checksum mode - offload it or do it synchronously. */ enum btrfs_offload_csum_mode offload_csum_mode;#endif
Signature · 1 line
2.46.1
PATCH v3 7/10 btrfs: pr CONFIG_BTRFS_EXPERIMENTAL status
In reply to Anand JainCommit c9c49e8f157e ("btrfs: split out CONFIG_BTRFS_EXPERIMENTAL from CONFIG_BTRFS_DEBUG") introduces a way to enable or disable experimental features, print its status during module load, like so:
Btrfs loaded, debug=on, assert=on, zoned=yes, fsverity=yes
--- fs/btrfs/super.c | 5 +++++ 1 file changed, 5 insertions(+)
fs/btrfs/super.c +5 -0
@@ -2487,6 +2487,11 @@ static int __init btrfs_print_mod_info(void)", fsverity=yes" #else ", fsverity=no" +#endif +#ifdef CONFIG_BTRFS_EXPERIMENTAL +", experimental=yes" +#else + ", experimental=no"#endif ;pr_info ("Btrfs loaded%s\n" , options);
Signature · 1 line
2.46.1
PATCH v3 8/10 btrfs: fix CONFIG_BTRFS_EXPERIMENTAL migration
In reply to Anand JainCommit c9c49e8f157e ("btrfs: split out CONFIG_BTRFS_EXPERIMENTAL from CONFIG_BTRFS_DEBUG") migrated some features from CONFIG_BTRFS_DEBUG to CONFIG_BTRFS_EXPERIMENTAL. We could also move the corresponding sysfs entries for these features, as there is no point in retaining the sysfs interfaces once the feature is moved.
--- fs/btrfs/sysfs.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/btrfs/sysfs.c +2 -2
@@ -295,7 +295,7 @@ BTRFS_FEAT_ATTR_INCOMPAT(simple_quota, SIMPLE_QUOTA);#ifdef CONFIG_BLK_DEV_ZONED BTRFS_FEAT_ATTR_INCOMPAT (zoned,ZONED );#endif -#ifdef CONFIG_BTRFS_DEBUG +#ifdef CONFIG_BTRFS_EXPERIMENTAL /* Remove once support for extent tree v2 is feature complete */ BTRFS_FEAT_ATTR_INCOMPAT (extent_tree_v2,EXTENT_TREE_V2 );/* Remove once support for raid stripe tree is feature complete. */
@@ -329,7 +329,7 @@ static struct attribute *btrfs_supported_feature_attrs[] = {#ifdef CONFIG_BLK_DEV_ZONED BTRFS_FEAT_ATTR_PTR(zoned),#endif -#ifdef CONFIG_BTRFS_DEBUG +#ifdef CONFIG_BTRFS_EXPERIMENTAL BTRFS_FEAT_ATTR_PTR(extent_tree_v2), BTRFS_FEAT_ATTR_PTR(raid_stripe_tree),#endif
Signature · 1 line
2.46.1
PATCH v3 9/10 btrfs: enable RAID1 balancing configuration via modprobe parameter
In reply to Anand JainThis update allows configuring the raid1-balancing methods using a modprobe parameter when experimental mode CONFIG_BTRFS_EXPERIMENTAL is enabled.
Examples:
- Set the RAID1 balancing method to round-robin with a custom min_contiguous_read of 192k:
$ modprobe btrfs raid1-balancing=round-robin:196608
- Set the round-robin balancing method with the default min_contiguous_read of 256k:
$ modprobe btrfs raid1-balancing=round-robin
- Set the devid balancing method, defaulting to the latest device:
$ modprobe btrfs raid1-balancing=devid
--- fs/btrfs/super.c | 5 +++++ fs/btrfs/sysfs.c | 29 +++++++++++++++++++++++++++++ fs/btrfs/sysfs.h | 5 +++++ fs/btrfs/volumes.c | 5 ++++- 4 files changed, 43 insertions(+), 1 deletion(-)
fs/btrfs/super.c +5 -0
@@ -2552,6 +2552,11 @@ static const struct init_sequence mod_init_seq[] = { }, { .init_func = extent_map_init, .exit_func = extent_map_exit, +#ifdef CONFIG_BTRFS_EXPERIMENTAL + }, { + .init_func = btrfs_raid1_balancing_init, + .exit_func = NULL, +#endif }, { .init_func = ordered_data_init, .exit_func = ordered_data_exit,
fs/btrfs/sysfs.c +29 -0
@@ -1307,11 +1307,28 @@ BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);#ifdef CONFIG_BTRFS_EXPERIMENTAL static const char *const btrfs_read_policy_name[] = {"pid" ,"round-robin" ,"devid" }; + +/* Global module configuration parameters */ +static char *raid1_balancing; +char *btrfs_get_raid1_balancing (void ) +{ +return raid1_balancing; +} + +/* Set perm 0, disable sys/module/btrfs/parameter/raid1_balancing interface */ +module_param (raid1_balancing, charp,0 ); +MODULE_PARM_DESC (raid1_balancing, +"Global read policy; pid (default), round-robin:[min_contiguous_read], devid:[[devid]|[latest-gen]|[oldest-gen]]" ); +#else static const char *const btrfs_read_policy_name[] = {"pid" };#endif +#ifdef CONFIG_BTRFS_EXPERIMENTAL +enum btrfs_read_policy btrfs_read_policy_to_enum(const char *str, s64 *value) +#else static enum btrfs_read_policy btrfs_read_policy_to_enum(const char *str, s64 *value) +#endif {#ifdef CONFIG_BTRFS_EXPERIMENTAL char *value_str;
@@ -1354,6 +1371,18 @@ static enum btrfs_read_policy btrfs_read_policy_to_enum(const char *str, s64 *vareturn -EINVAL ; } +#ifdef CONFIG_BTRFS_EXPERIMENTAL +int __init btrfs_raid1_balancing_init (void ) +{ +if (btrfs_read_policy_to_enum (raid1_balancing,NULL ) == -EINVAL ) { +btrfs_err (NULL ,"Invalid raid1_balancing %s" , raid1_balancing); +return -EINVAL ; + } + +return 0 ; +} +#endif +static ssize_t btrfs_read_policy_show (struct kobject *kobj,struct kobj_attribute *a,char *buf) {
fs/btrfs/sysfs.h +5 -0
@@ -47,5 +47,10 @@ void btrfs_sysfs_del_qgroups(struct btrfs_fs_info *fs_info);int btrfs_sysfs_add_qgroups (struct btrfs_fs_info *fs_info);void btrfs_sysfs_del_one_qgroup (struct btrfs_fs_info *fs_info,struct btrfs_qgroup *qgroup); +#ifdef CONFIG_BTRFS_EXPERIMENTAL +enum btrfs_read_policy btrfs_read_policy_to_enum (const char *str,s64 *value); +int __init btrfs_raid1_balancing_init (void ); +char *btrfs_get_raid1_balancing (void ); +#endif #endif
fs/btrfs/volumes.c +4 -1
@@ -1327,11 +1327,14 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices, fs_devices->latest_dev = latest_dev; fs_devices->total_rw_bytes =0 ; fs_devices->chunk_alloc_policy =BTRFS_CHUNK_ALLOC_REGULAR ; - fs_devices->read_policy =BTRFS_READ_POLICY_PID ;#ifdef CONFIG_BTRFS_EXPERIMENTAL /* Set min_contiguous_read to a default 256kib */ fs_devices->min_contiguous_read =256 *1024 ; fs_devices->read_devid = latest_dev->devid; + fs_devices->read_policy = +btrfs_read_policy_to_enum (btrfs_get_raid1_balancing (),NULL ); +#else + fs_devices->read_policy =BTRFS_READ_POLICY_PID ;#endif return 0 ;
Signature · 1 line
2.46.1
PATCH v3 10/10 btrfs: modload to print RAID1 balancing status
In reply to Anand JainModified the Btrfs loading message to include the RAID1 balancing status if the experimental feature is enabled.
--- fs/btrfs/super.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
fs/btrfs/super.c +11 -1
@@ -2494,7 +2494,17 @@ static int __init btrfs_print_mod_info(void) ", experimental=no"#endif ; -pr_info ("Btrfs loaded%s\n" , options); + +#ifdef CONFIG_BTRFS_EXPERIMENTAL +if (btrfs_get_raid1_balancing () ==NULL ) +pr_info ("Btrfs loaded%s\n" , options); +else +pr_info ("Btrfs loaded%s, raid1_balancing=%s\n" , + options,btrfs_get_raid1_balancing ()); +#else +pr_info ("Btrfs loaded%s\n" , options); +#endif +return 0 ; }
Signature · 1 line
2.46.1
cover fstests: refine filesystem module reload handling
In reply to Anand JainThese changes refine module reload control in testing. Patch 1 reloads the module earlier in run_section, ensuring each section's first test starts with a reloaded module. Patch 2 adds FS_MODULE_RELOAD_OPTIONS to pass module options during reload, useful for the Btrfs pre-mount configurations that aren’t available as mount options.
Anand Jain (2):
fstests: move fs-module reload to earlier in the run_section function
fstests: FS_MODULE_RELOAD_OPTIONS to control filesystem module reload
options
check | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)
Signature · 1 line
2.46.1
PATCH 1/2 fstests: move fs-module reload to earlier in the run_section function
Reload the module before each test, instead of later in run_section.
--- check | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
check +9 -9
@@ -935,6 +935,15 @@ function run_section() continue fi + # Reload the module after each test to check for leaks or + # other problems. + if [ -n "${TEST_FS_MODULE_RELOAD}" ]; then + _test_unmount 2> /dev/null + _scratch_unmount 2> /dev/null + modprobe -r fs-$FSTYP + modprobe fs-$FSTYP + fi + # record that we really tried to run this test. if ((!${#loop_status[*]})); then try+=("$seqnum")
@@ -1033,15 +1042,6 @@ function run_section() done fi - # Reload the module after each test to check for leaks or - # other problems. - if [ -n "${TEST_FS_MODULE_RELOAD}" ]; then - _test_unmount 2> /dev/null - _scratch_unmount 2> /dev/null - modprobe -r fs-$FSTYP - modprobe fs-$FSTYP - fi - # Scan for memory leaks after every test so that associating # a leak to a particular test will be as accurate as possible. _check_kmemleak || tc_status="fail"
Signature · 1 line
2.46.1
PATCH 2/2 fstests: FS_MODULE_RELOAD_OPTIONS to control filesystem module reload options
In reply to Anand JainExtend module reload logic to allow passing additional options via FS_MODULE_RELOAD_OPTIONS. This enhancement enables more flexible configuration during module reloads, which can be useful for testing specific module parameters.
Maintains existing behavior for TEST_FS_MODULE_RELOAD.
--- check | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
check +3 -2
@@ -937,11 +937,12 @@ function run_section() # Reload the module after each test to check for leaks or # other problems. - if [ -n "${TEST_FS_MODULE_RELOAD}" ]; then + if [[ -n "${TEST_FS_MODULE_RELOAD}" || + -n "${FS_MODULE_RELOAD_OPTIONS}" ]]; then _test_unmount 2> /dev/null _scratch_unmount 2> /dev/null modprobe -r fs-$FSTYP - modprobe fs-$FSTYP + modprobe fs-$FSTYP ${FS_MODULE_RELOAD_OPTIONS} fi # record that we really tried to run this test.
Signature · 1 line
2.46.1
Re: [PATCH 2/2] fstests: FS_MODULE_RELOAD_OPTIONS to control filesystem module reload options
Quoted 4 lines · Anand Jain
Extend module reload logic to allow passing additional options via
FS_MODULE_RELOAD_OPTIONS. This enhancement enables more flexible configuration during module reloads, which can be useful for testing specific module parameters.
You might want to document the existence of this knob in the README.
Quoted 13 lines
Maintains existing behavior for
TEST_FS_MODULE_RELOAD.Signed-off-by Anand Jain <[email protected]>--- check | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)check +0
-0@@ -937,11 +937,12 @@ function run_section()
check function run_section() · line 945
Quoted 10 lines
check +3
-2# Reload the module after each test to check for leaks or # other problems. - if [ -n "${TEST_FS_MODULE_RELOAD}" ]; then + if [[ -n "${TEST_FS_MODULE_RELOAD}" || + -n "${FS_MODULE_RELOAD_OPTIONS}" ]]; then _test_unmount 2> /dev/null _scratch_unmount 2> /dev/null modprobe -r fs-$FSTYP - modprobe fs-$FSTYP + modprobe fs-$FSTYP ${FS_MODULE_RELOAD_OPTIONS}
What happens if the test itself decides to reload the $FSTYP module? Do you want these options to propagate to those reloadings as well? AFAICT there are some btrfs/ group tests that do such things.
(Perhaps you'd be better off injecting FS_MODULE_RELOAD_OPTIONS into /etc/modprobe.d/<somefile> to catch all these cases?)
--D
Quoted 7 lines
check +0
-0fi # record that we really tried to run this test.Signature · 1 line
2.46.1
Re: [PATCH 1/2] fstests: move fs-module reload to earlier in the run_section function
In reply to Anand JainQuoted 12 lines · Anand Jain
Reload the module before each test, instead of later in
run_section.Signed-off-by Anand Jain <[email protected]>--- check | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)check +0
-0
check function run_section() · line 939
Quoted 5 lines
check +2
-0@@ -935,6 +935,15 @@ function run_section() continue fi + # Reload the module after each test to check for leaks or + # other problems.
Hrmm. The nice thing about doing the reload /after/ each test is that unloading the module will purge any per-fs slab caches that the module creates. The slab teardown logs any forgotten objects while $seq still points to the test that leaked those objects. Granted it's not a 100% solution (that job falls to kmemcheck) but it's a cheap check.
This change makes it so that if test N leaks something, fstests reports them for test N+1.
--D
Quoted 30 lines
check +7
-9+ if [ -n "${TEST_FS_MODULE_RELOAD}" ]; then + _test_unmount 2> /dev/null + _scratch_unmount 2> /dev/null + modprobe -r fs-$FSTYP + modprobe fs-$FSTYP + fi + # record that we really tried to run this test. if ((!${#loop_status[*]})); then try+=("$seqnum")@@ -1033,15 +1042,6 @@ function run_section() done fi - # Reload the module after each test to check for leaks or - # other problems. - if [ -n "${TEST_FS_MODULE_RELOAD}" ]; then - _test_unmount 2> /dev/null - _scratch_unmount 2> /dev/null - modprobe -r fs-$FSTYP - modprobe fs-$FSTYP - fi - # Scan for memory leaks after every test so that associating # a leak to a particular test will be as accurate as possible. _check_kmemleak || tc_status="fail"Signature · 1 line
2.46.1
Re: [PATCH 1/2] fstests: move fs-module reload to earlier in the run_section function
In reply to Anand JainQuoted 2 lines · Anand Jain
Reload the module before each test, instead of later in
run_section.
Why?
The change log should give a justification for the change.
It's just stating what the code changes are doing but without any reasoning.
Thanks.
Quoted 46 lines
Signed-off-by Anand Jain <[email protected]>--- check | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)check +9
-10@@ -935,6 +935,15 @@ function run_section() continue fi + # Reload the module after each test to check for leaks or + # other problems. + if [ -n "${TEST_FS_MODULE_RELOAD}" ]; then + _test_unmount 2> /dev/null + _scratch_unmount 2> /dev/null + modprobe -r fs-$FSTYP + modprobe fs-$FSTYP + fi + # record that we really tried to run this test. if ((!${#loop_status[*]})); then try+=("$seqnum")@@ -1033,15 +1042,6 @@ function run_section() done fi - # Reload the module after each test to check for leaks or - # other problems. - if [ -n "${TEST_FS_MODULE_RELOAD}" ]; then - _test_unmount 2> /dev/null - _scratch_unmount 2> /dev/null - modprobe -r fs-$FSTYP - modprobe fs-$FSTYP - fi - # Scan for memory leaks after every test so that associating # a leak to a particular test will be as accurate as possible. _check_kmemleak || tc_status="fail" --2.46.1
Quoted 144 lines · Anand Jain
v3:
1. Removed the latency-based RAID1 balancing patch. (Per David's review)
2. Renamed "rotation" to "round-robin" and set the per-setmin_contiguous_readto 256k. (Per David's review)
3. Added raid1-balancing module configuration for fstests testing. raid1-balancing can now be configured through both module load parameters and sysfs.The logic of individual methods remains unchanged, and performance metrics are consistent with v2.
-----
v2:
1. Move new features toCONFIG_BTRFS_EXPERIMENTALinstead ofCONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait toU64_MAXand remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy
The default value is the sector size, and the
min_contiguous_readvalue must be a multiple of the sector size.5. Tested FIO random read/write and defrag compression workloads with
min_contiguous_readset to sector size, 192k, and 256k.RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation. $ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \ --time_based --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 20.3MiB/s | 20.5MiB/s | 313895 | 313895 | | rotation| | | | | | 4096| 20.4MiB/s | 20.5MiB/s | 313895 | 313895 | | 196608| 20.2MiB/s | 20.2MiB/s | 310152 | 310175 | | 262144| 20.3MiB/s | 20.4MiB/s | 312180 | 312191 | | latency| 18.4MiB/s | 18.4MiB/s | 272980 | 291683 | | devid:1 | 14.8MiB/s | 14.9MiB/s | 456376 | 0 |rotation RAID1 balancing technique performs more than 2x better for single-process defrag.
$ time -p btrfs filesystem defrag -r -f -c /btrfs| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 18.00s| 3800 | 0 | | rotation| | | | | 4096| 8.95s| 1900 | 1901 | | 196608| 8.50s| 1881 | 1919 | | 262144| 8.80s| 1881 | 1919 | | latency | 17.18s| 3800 | 0 | | devid:2 | 17.48s| 0 | 3800 |Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.
Latency is smarter with unstable block layer transport.
Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.
Devid is a hands-on approach, provides manual or user-space script control.
These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.
Thx.
--------- original v1 ------------
The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the
CONFIG_BTRFS_DEBUGconfig option and are on top of the previously added/sys/fs/btrfs/<UUID>/read_policyinterface to configure the desired RAID1 read balancing method.I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from
/sys/devices/<..>/statfor each device. Below is a summary of the results, with each result the average of three iterations.A typical generic random rw workload:
$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \ --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 29.4MiB/s | 29.5MiB/s | 456548 | 447975 | | rotation| 29.3MiB/s | 29.3MiB/s | 450105 | 450055 | | latency | 21.9MiB/s | 21.9MiB/s | 672387 | 0 | | devid:1 | 22.0MiB/s | 22.0MiB/s | 674788 | 0 |Defragmentation with compression workload:
$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo $ sync $ echo 3 > /proc/sys/vm/drop_caches $ btrfs filesystem defrag -f -c /btrfs/foo| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 21.61s| 3810 | 0 | | rotation| 11.55s| 1905 | 1905 | | latency | 20.99s| 0 | 3810 | | devid:2 | 21.41s| 0 | 3810 |. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.
Quoted 10 lines
Anand Jain (10):
btrfs: initializefs_devices->fs_infoearlier
btrfs: simplify output formatting inbtrfs_read_policy_show
btrfs: addbtrfs_read_policy_to_enumhelper and refactor read policy
store
btrfs: handle value associated with raid1 balancing parameter
btrfs: introduce RAID1 round-robin read balancing
btrfs: add RAID1 preferred read device
btrfs: prCONFIG_BTRFS_EXPERIMENTALstatus
btrfs: fixCONFIG_BTRFS_EXPERIMENTALmigration
Why are these two patches, which are fixes unrelated to the raid balancing feature, in the middle of this patchset? These should go as a separate patchset...
Also for the second patch, there's already a fix from yesterday and in for-next:
Thanks.
Quoted 15 lines
btrfs: enable RAID1 balancing configuration via modprobe parameter
btrfs: modload to print RAID1 balancing statusfs/btrfs/disk-io.c | 1 + fs/btrfs/super.c | 22 +++++- fs/btrfs/sysfs.c | 181 ++++++++++++++++++++++++++++++++++++++++----- fs/btrfs/sysfs.h | 5 ++ fs/btrfs/volumes.c | 86 ++++++++++++++++++++- fs/btrfs/volumes.h | 14 ++++ 6 files changed, 286 insertions(+), 23 deletions(-)--
2.46.1
Re: [PATCH 1/2] fstests: move fs-module reload to earlier in the run_section function
In reply to Darrick J. WongQuoted 25 lines · Darrick J. Wong
Quoted 12 lines · Anand Jain
Reload the module before each test, instead of later in
run_section.Signed-off-by Anand Jain <[email protected]>--- check | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)check +0
-0check function run_section() · line 939
Quoted 5 lines
check +2
-0@@ -935,6 +935,15 @@ function run_section() continue fi + # Reload the module after each test to check for leaks or + # other problems.Hrmm. The nice thing about doing the reload /after/ each test is that unloading the module will purge any per-fs slab caches that the module creates. The slab teardown logs any forgotten objects while $seq still points to the test that leaked those objects. Granted it's not a 100% solution (that job falls to kmemcheck) but it's a cheap check.
Quoted 3 lines
This change makes it so that if test N leaks something, fstests reports them for test N+1.
Agreed. It's better to report the issue at test N. I'll add this comment to the code in v2.
Thanks!
Anand
Quoted 32 lines
--D
Quoted 30 lines
check +7
-9+ if [ -n "${TEST_FS_MODULE_RELOAD}" ]; then + _test_unmount 2> /dev/null + _scratch_unmount 2> /dev/null + modprobe -r fs-$FSTYP + modprobe fs-$FSTYP + fi + # record that we really tried to run this test. if ((!${#loop_status[*]})); then try+=("$seqnum")@@ -1033,15 +1042,6 @@ function run_section() done fi - # Reload the module after each test to check for leaks or - # other problems. - if [ -n "${TEST_FS_MODULE_RELOAD}" ]; then - _test_unmount 2> /dev/null - _scratch_unmount 2> /dev/null - modprobe -r fs-$FSTYP - modprobe fs-$FSTYP - fi - # Scan for memory leaks after every test so that associating # a leak to a particular test will be as accurate as possible. _check_kmemleak || tc_status="fail"Signature · 1 line
2.46.1
Quoted 155 lines · Filipe Manana
Quoted 152 lines · Anand Jain
v3:
1. Removed the latency-based RAID1 balancing patch. (Per David's review)
2. Renamed "rotation" to "round-robin" and set the per-setmin_contiguous_readto 256k. (Per David's review)
3. Added raid1-balancing module configuration for fstests testing. raid1-balancing can now be configured through both module load parameters and sysfs.The logic of individual methods remains unchanged, and performance metrics are consistent with v2.
-----
v2:
1. Move new features toCONFIG_BTRFS_EXPERIMENTALinstead ofCONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait toU64_MAXand remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy
The default value is the sector size, and the
min_contiguous_readvalue must be a multiple of the sector size.5. Tested FIO random read/write and defrag compression workloads with
min_contiguous_readset to sector size, 192k, and 256k.RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.
$ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
--ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
--time_based --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 20.3MiB/s | 20.5MiB/s | 313895 | 313895 | | rotation| | | | | | 4096| 20.4MiB/s | 20.5MiB/s | 313895 | 313895 | | 196608| 20.2MiB/s | 20.2MiB/s | 310152 | 310175 | | 262144| 20.3MiB/s | 20.4MiB/s | 312180 | 312191 | | latency| 18.4MiB/s | 18.4MiB/s | 272980 | 291683 | | devid:1 | 14.8MiB/s | 14.9MiB/s | 456376 | 0 |rotation RAID1 balancing technique performs more than 2x better for single-process defrag.
$ time -p btrfs filesystem defrag -r -f -c /btrfs
| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 18.00s| 3800 | 0 | | rotation| | | | | 4096| 8.95s| 1900 | 1901 | | 196608| 8.50s| 1881 | 1919 | | 262144| 8.80s| 1881 | 1919 | | latency | 17.18s| 3800 | 0 | | devid:2 | 17.48s| 0 | 3800 |Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.
Latency is smarter with unstable block layer transport.
Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.
Devid is a hands-on approach, provides manual or user-space script control.
These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.
Thx.
--------- original v1 ------------
The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the
CONFIG_BTRFS_DEBUGconfig option and are on top of the previously added/sys/fs/btrfs/<UUID>/read_policyinterface to configure the desired RAID1 read balancing method.I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from
/sys/devices/<..>/statfor each device. Below is a summary of the results, with each result the average of three iterations.A typical generic random rw workload:
$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \ --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 29.4MiB/s | 29.5MiB/s | 456548 | 447975 | | rotation| 29.3MiB/s | 29.3MiB/s | 450105 | 450055 | | latency | 21.9MiB/s | 21.9MiB/s | 672387 | 0 | | devid:1 | 22.0MiB/s | 22.0MiB/s | 674788 | 0 |Defragmentation with compression workload:
$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo $ sync $ echo 3 > /proc/sys/vm/drop_caches $ btrfs filesystem defrag -f -c /btrfs/foo| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 21.61s| 3810 | 0 | | rotation| 11.55s| 1905 | 1905 | | latency | 20.99s| 0 | 3810 | | devid:2 | 21.41s| 0 | 3810 |. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.Anand Jain (10):
btrfs: initializefs_devices->fs_infoearlier
btrfs: simplify output formatting inbtrfs_read_policy_show
btrfs: addbtrfs_read_policy_to_enumhelper and refactor read policy
store
btrfs: handle value associated with raid1 balancing parameter
btrfs: introduce RAID1 round-robin read balancing
btrfs: add RAID1 preferred read device
Quoted 4 lines
Quoted 4 lines
btrfs: pr
CONFIG_BTRFS_EXPERIMENTALstatusbtrfs: fix
CONFIG_BTRFS_EXPERIMENTALmigration
This patch can be dropped, as Filipe's patch below already fixes it.
Quoted 2 lines
Why are these two patches, which are fixes unrelated to the raid
This patchset includes the last patch, which depends on the patch (btrfs: pr CONFIG_BTRFS_EXPERIMENTAL status). I would prefer to keep it here.
But the question is there a more effective way to indicate experimental features with the raid1-balancing as the use-case? The current issue with the patchset is that dmesg doesn't clearly mark raid1_balancing as experimental. Would it be better to add "(experimental: )" or simply use "()" to highlight this?
---
[ 3663.742802] Btrfs loaded, debug=on, assert=on, zoned=yes, fsverity=yes, experimental=yes, raid1_balancing=round-robin
---
or
---
[ 3663.742802] Btrfs loaded, debug=on, assert=on, zoned=yes, fsverity=yes, (experimental: raid1_balancing=round-robin)
---
or
---
[ 3663.742802] Btrfs loaded, debug=on, assert=on, zoned=yes, fsverity=yes, (raid1_balancing=round-robin)
---
Quoted 7 lines
balancing feature, in the middle of this patchset? These should go as a separate patchset...
Also for the second patch, there's already a fix from yesterday and in for-next:
Thanks for the review.
-Anand
Quoted 17 lines
Thanks.
Quoted 15 lines
btrfs: enable RAID1 balancing configuration via modprobe parameter
btrfs: modload to print RAID1 balancing statusfs/btrfs/disk-io.c | 1 + fs/btrfs/super.c | 22 +++++- fs/btrfs/sysfs.c | 181 ++++++++++++++++++++++++++++++++++++++++----- fs/btrfs/sysfs.h | 5 ++ fs/btrfs/volumes.c | 86 ++++++++++++++++++++- fs/btrfs/volumes.h | 14 ++++ 6 files changed, 286 insertions(+), 23 deletions(-)--
2.46.1
kernel test robot
build botRe: [PATCH v3 05/10] btrfs: introduce RAID1 round-robin read balancing
In reply to Anand Jainkernel test robot noticed the following build errors:
Hi Anand,
kernel test robot noticed the following build errors:
[auto build test ERROR on kdave/for-next]
[also build test ERROR on next-20241115]
[cannot apply to linus/master v6.12]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Anand-Jain/btrfs-initialize-fs_devices-fs_info-earlier/20241116-230928 base: https://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-next patch link: https://lore.kernel.org/r/995d4a9dd9f553825805efdac24dec4a9de20ef3.1731076425.git.anand.jain%40oracle.com patch subject: [PATCH v3 05/10] btrfs: introduce RAID1 round-robin read balancing config: m68k-randconfig-r113-20241117 (https://download.01.org/0day-ci/archive/20241118/[email protected]/config) compiler: m68k-linux-gcc (GCC) 14.2.0 reproduce: (https://download.01.org/0day-ci/archive/20241118/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
m68k-linux-ld: fs/btrfs/sysfs.o: in function `btrfs_read_policy_store':
>> sysfs.c:(.text+0x197a): undefined reference to `__moddi3'
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v3 03/10] btrfs: add btrfs_read_policy_to_enum helper and refactor read policy store
In reply to Anand JainQuoted 20 lines · Anand Jain
Introduce the
btrfs_read_policy_to_enumhelper function to simplify the conversion of a string read policy to its corresponding enum value. This reduces duplication and improves code clarity inbtrfs_read_policy_store. Thebtrfs_read_policy_storefunction has been refactored to use the new helper.The parameter is copied locally to allow modification, enabling the separation of the method and its value. This prepares for the addition of more functionality in subsequent patches.
Signed-off-by Anand Jain <[email protected]>--- fs/btrfs/sysfs.c | 50 ++++++++++++++++++++++++++++++++++++------------ 1 file changed, 38 insertions(+), 12 deletions(-)fs/btrfs/sysfs.c +0
-0
fs/btrfs/sysfs.c BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show); · line 1316
Quoted 10 lines
fs/btrfs/sysfs.c +7
-0@@ -1307,6 +1307,34 @@ BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);static const char *const btrfs_read_policy_name[] = {"pid" }; +static enum btrfs_read_policy btrfs_read_policy_to_enum (const char *str) +{ +bool found = false; +enum btrfs_read_policy index; +char *param; + +if (!str || !strlen (str))
For ineger values please use '== 0'
fs/btrfs/sysfs.c BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show); · line 1319
Quoted 3 lines
fs/btrfs/sysfs.c +3
-0+return 0 ; + + param =kstrdup (str,GFP_KERNEL );
Why do you need the dynamic allocation? The policy strings won't be long, so a 32 bytes on stack can be sufficient.
Quoted 17 lines
fs/btrfs/sysfs.c +17
-0+if (!param) +return -ENOMEM ; + +for (index =0 ; index <BTRFS_NR_READ_POLICY ; index++) { +if (sysfs_streq (param, btrfs_read_policy_name[index])) { + found = true; +break ; + } + } + +kfree (param); + +if (found) +return index; + +return -EINVAL ; +}
Re: [PATCH v3 04/10] btrfs: handle value associated with raid1 balancing parameter
In reply to Anand JainQuoted 18 lines · Anand Jain
This change enables specifying additional configuration values alongside the raid1 balancing / read policy in a single input string.
Updated
btrfs_read_policy_to_enum()to parse and handle a value associated with the policy in the formatpolicy:value, the value part if present is converted 64-bit integer. Updatebtrfs_read_policy_store()to accommodate the new parameter.Signed-off-by Anand Jain <[email protected]>--- fs/btrfs/sysfs.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)fs/btrfs/sysfs.c +0
-0
fs/btrfs/sysfs.c BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show); · line 1310
Quoted 5 lines
fs/btrfs/sysfs.c +1
-1@@ -1307,8 +1307,11 @@ BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);static const char *const btrfs_read_policy_name[] = {"pid" }; -static enum btrfs_read_policy btrfs_read_policy_to_enum (const char *str) +static enum btrfs_read_policy btrfs_read_policy_to_enum (const char *str,s64 *value)
Why is value signed type?
fs/btrfs/sysfs.c BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show); · line 1314
Quoted 4 lines
fs/btrfs/sysfs.c +3
-0{ +#ifdef CONFIG_BTRFS_EXPERIMENTAL +char *value_str; +#endif
Please keep the inline #ifdefs to minimum, with __maybe_unused there won't be any warning if experimental build is disabled.
fs/btrfs/sysfs.c static enum btrfs_read_policy btrfs_read_policy_to_enum(const char *str) · line 1331
Quoted 13 lines
fs/btrfs/sysfs.c +6
-0bool found = false;enum btrfs_read_policy index;char *param;@@ -1320,6 +1323,18 @@ static enum btrfs_read_policy btrfs_read_policy_to_enum(const char *str)if (!param)return -ENOMEM ; +#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Separate value from input in policy:value format. */ +if ((value_str =strchr (param,':' ))) { + *value_str ='\0' ; + value_str++; +if (value &&kstrtou64 (value_str,10 , value) !=0 ) {
You can assume that the 'value' is always non zero and assert it at the beginning of the function.
fs/btrfs/sysfs.c static enum btrfs_read_policy btrfs_read_policy_to_enum(const char *str) · line 1333
Quoted 2 lines
fs/btrfs/sysfs.c +2
-0+kfree (param); +return -EINVAL ;
A negative errno in a function returning enum looks quite strange, either add an enum type for invalid value or use 'int'.
Quoted 21 lines
fs/btrfs/sysfs.c +6
-1+ } + } +#endif +for (index =0 ; index <BTRFS_NR_READ_POLICY ; index++) {if (sysfs_streq (param, btrfs_read_policy_name[index])) { found = true;@@ -1367,8 +1382,9 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj, {struct btrfs_fs_devices *fs_devices =to_fs_devs (kobj);enum btrfs_read_policy index; +s64 value =-1 ; - index =btrfs_read_policy_to_enum (buf); + index =btrfs_read_policy_to_enum (buf, &value);if (index == -EINVAL )return -EINVAL ;Signature · 1 line
2.46.1
Re: [PATCH v3 05/10] btrfs: introduce RAID1 round-robin read balancing
In reply to Anand JainQuoted 4 lines · Anand Jain
This feature balances I/O across the striped devices when reading from RAID1 blocks.
echo round-robin:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy
The ":" is part of the optional value, so the example should be "round-robin[:min]"
Quoted 16 lines
The
min_contiguous_readparameter defines the minimum read size before switching to the next mirrored device. This setting is optional, with a default value of 256 KiB.Signed-off-by Anand Jain <[email protected]>--- fs/btrfs/sysfs.c | 38 +++++++++++++++++++++++++++++ fs/btrfs/volumes.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 9 +++++++ 3 files changed, 107 insertions(+)fs/btrfs/sysfs.c +0
-0
fs/btrfs/sysfs.c static ssize_t btrfs_temp_fsid_show(struct kobject *kobj, · line 1312
Quoted 8 lines
fs/btrfs/sysfs.c +4
-0@@ -1305,7 +1305,11 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj, } BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show); +#ifdef CONFIG_BTRFS_EXPERIMENTAL +static const char *const btrfs_read_policy_name[] = {"pid" ,"round-robin" }; +#else static const char *const btrfs_read_policy_name[] = {"pid" }; +#endif
Instead of the duplication of the definition please put the #ifdef around the experimental strings.
fs/btrfs/sysfs.c static ssize_t btrfs_read_policy_show(struct kobject *kobj, · line 1377
Quoted 11 lines
fs/btrfs/sysfs.c +4
-0static enum btrfs_read_policy btrfs_read_policy_to_enum (const char *str,s64 *value) {@@ -1367,6 +1371,12 @@ static ssize_t btrfs_read_policy_show(struct kobject *kobj, ret +=sysfs_emit_at (buf, ret,"%s" , btrfs_read_policy_name[i]); +#ifdef CONFIG_BTRFS_EXPERIMENTAL +if (i ==BTRFS_READ_POLICY_RR ) + ret +=sysfs_emit_at (buf, ret,":%d" , + fs_devices->min_contiguous_read);
Maybe this should be printed with the suffix, "rr:256k".
fs/btrfs/sysfs.c static ssize_t btrfs_read_policy_store(struct kobject *kobj, · line 1404
Quoted 13 lines
fs/btrfs/sysfs.c +6
-0+#endif +if (i == policy) ret +=sysfs_emit_at (buf, ret,"]" ); }@@ -1388,6 +1398,34 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj,if (index == -EINVAL )return -EINVAL ; +#ifdef CONFIG_BTRFS_EXPERIMENTAL +if (index ==BTRFS_READ_POLICY_RR ) { +if (value !=-1 ) { +if ((value % fs_devices->fs_info->sectorsize) !=0 ) {
I think 64bit types and % could lead to the division which fails on 32bit platforms. As it's the s64 type for no apparent reason a u32 would be a better fit.
fs/btrfs/sysfs.c static ssize_t btrfs_read_policy_store(struct kobject *kobj, · line 1408
Quoted 4 lines
fs/btrfs/sysfs.c +4
-0+btrfs_err (fs_devices->fs_info, +"read_policy: min_contiguous_read %lld should be multiples of the sectorsize %u" , + value, fs_devices->fs_info->sectorsize); +return -EINVAL ;
This does not need to fail hard, we can simply round it up to the next sector size multiple.
Also this does not validate the upper bound of value.
fs/btrfs/sysfs.c static ssize_t btrfs_read_policy_store(struct kobject *kobj, · line 1412
Quoted 4 lines
fs/btrfs/sysfs.c +4
-0+ } + }else { +/* value is not provided, set it to the default 256k */ + value =256 *1024 ;
Please use a named constant, then the comment is not necessary.
Quoted 24 lines
fs/btrfs/sysfs.c +16
-0+ } + +if (index !=READ_ONCE (fs_devices->read_policy) || + value !=READ_ONCE (fs_devices->min_contiguous_read)) { +WRITE_ONCE (fs_devices->read_policy, index); +WRITE_ONCE (fs_devices->min_contiguous_read, value); +atomic_set (&fs_devices->total_reads,0 ); + +btrfs_info (fs_devices->fs_info,"read policy set to '%s:%lld'" , + btrfs_read_policy_name[index], value); + + } + +return len; + } +#endif if (index !=READ_ONCE (fs_devices->read_policy)) {WRITE_ONCE (fs_devices->read_policy, index); btrfs_info(fs_devices->fs_info, "read policy set to '%s'",fs/btrfs/volumes.c +0
-0
fs/btrfs/volumes.c static int open_fs_devices(struct btrfs_fs_devices *fs_devices, · line 1333
Quoted 6 lines
fs/btrfs/volumes.c +3
-0@@ -1328,6 +1328,10 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices, fs_devices->total_rw_bytes =0 ; fs_devices->chunk_alloc_policy =BTRFS_CHUNK_ALLOC_REGULAR ; fs_devices->read_policy =BTRFS_READ_POLICY_PID ; +#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Set min_contiguous_read to a default 256kib */ + fs_devices->min_contiguous_read =256 *1024 ;
Named constant.
Quoted 8 lines
fs/btrfs/volumes.c +1
-0+#endif return 0 ; }@@ -5959,6 +5963,57 @@ unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info,return len; }
fs/btrfs/volumes.c unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, · line 5975
Quoted 10 lines
fs/btrfs/volumes.c +10
-0+#ifdef CONFIG_BTRFS_EXPERIMENTAL +struct stripe_mirror { +u64 devid; +int num; +}; + +static int btrfs_cmp_devid (const void *a,const void *b) +{ +struct stripe_mirror *s1 = (struct stripe_mirror *)a; +struct stripe_mirror *s2 = (struct stripe_mirror *)b;
Please keep the const qualifier.
fs/btrfs/volumes.c unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, · line 5984
Quoted 9 lines
fs/btrfs/volumes.c +9
-0+ +if (s1->devid < s2->devid) +return -1 ; +if (s1->devid > s2->devid) +return 1 ; +return 0 ; +} + +static int btrfs_read_rr (struct btrfs_chunk_map *map,int first,int num_stripe)
Please add a comment explaining what the function does or how it selects the device.
fs/btrfs/volumes.c unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, · line 5986
Quoted 2 lines
fs/btrfs/volumes.c +2
-0+{ +struct stripe_mirror stripes[4 ] = {0 };//4: max possible mirrors
Another magic constant. We probably won't have more than 4 but it still needs to be defined elsewhre. Also the comment format is wrong.
fs/btrfs/volumes.c unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, · line 5989
Quoted 3 lines
fs/btrfs/volumes.c +3
-0+struct btrfs_fs_devices *fs_devices; +struct btrfs_device *device; +int j;
This is used in the for loop only, so you can declare it as for (int j = ...)
fs/btrfs/volumes.c unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, · line 5999
Quoted 10 lines
fs/btrfs/volumes.c +10
-0+int read_cycle; +int index; +int ret_stripe; +int total_reads; +int reads_per_dev =0 ; + + device = map->stripes[first].dev; + + fs_devices = device->fs_devices; + reads_per_dev = fs_devices->min_contiguous_read/fs_devices->fs_info->sectorsize;
Missing space around '/' and we have the sectorsize_bits so division can be replaced by >>
fs/btrfs/volumes.c unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, · line 6001
Quoted 2 lines
fs/btrfs/volumes.c +2
-0+ index =0 ; +for (j = first; j < first + num_stripe; j++) {
Actually you can use 'i' for iteration.
fs/btrfs/volumes.c unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, · line 6010
Quoted 9 lines
fs/btrfs/volumes.c +9
-0+ stripes[index].devid = map->stripes[j].dev->devid; + stripes[index].num = j; + index++; + } +sort (stripes, num_stripe,sizeof (struct stripe_mirror ), + btrfs_cmp_devid,NULL ); + + total_reads =atomic_inc_return (&fs_devices->total_reads); + read_cycle = total_reads/reads_per_dev;
Missing spaces around "/"
Quoted 37 lines
fs/btrfs/volumes.c +11
-0+ ret_stripe = stripes[read_cycle % num_stripe].num; + + return ret_stripe; +} +#endif +static int find_live_mirror (struct btrfs_fs_info *fs_info,struct btrfs_chunk_map *map,int first,int dev_replace_is_ongoing)@@ -5988,6 +6043,11 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,case BTRFS_READ_POLICY_PID : preferred_mirror = first + (current->pid % num_stripes);break ; +#ifdef CONFIG_BTRFS_EXPERIMENTAL +case BTRFS_READ_POLICY_RR : + preferred_mirror =btrfs_read_rr (map, first, num_stripes); +break ; +#endif } if (dev_replace_is_ongoing &&fs/btrfs/volumes.h +4
-0@@ -303,6 +303,10 @@ enum btrfs_chunk_allocation_policy {enum btrfs_read_policy {/* Use process PID to choose the stripe */ BTRFS_READ_POLICY_PID , +#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Balancing raid1 reads across all striped devices (round-robin) */ +BTRFS_READ_POLICY_RR , +#endif BTRFS_NR_READ_POLICY , };
fs/btrfs/volumes.h struct btrfs_fs_devices { · line 441
Quoted 7 lines
fs/btrfs/volumes.h +4
-0@@ -431,6 +435,11 @@ struct btrfs_fs_devices {enum btrfs_read_policy read_policy;#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* IO stat, read counter. */ +atomic_t total_reads; +/* Min contiguous reads before switching to next device. */ +int min_contiguous_read;
As this is in the fs_devices, there should be some prefix that it's related to the read policy or round robin.
Quoted 7 lines
fs/btrfs/volumes.h +1
-0+/* Checksum mode - offload it or do it synchronously. */ enum btrfs_offload_csum_mode offload_csum_mode;#endif Signature · 1 line
2.46.1
Re: [PATCH v3 07/10] btrfs: pr CONFIG_BTRFS_EXPERIMENTAL status
In reply to Anand JainQuoted 16 lines · Anand Jain
Commit
c9c49e8f157e("btrfs: split outCONFIG_BTRFS_EXPERIMENTALfromCONFIG_BTRFS_DEBUG") introduces a way to enable or disable experimental features, print its status during module load, like so:Btrfs loaded, debug=on, assert=on, zoned=yes, fsverity=yes
Signed-off-by Anand Jain <[email protected]>--- fs/btrfs/super.c | 5 +++++ 1 file changed, 5 insertions(+)fs/btrfs/super.c +0
-0
fs/btrfs/super.c static int __init btrfs_print_mod_info(void) · line 2494
Quoted 8 lines
fs/btrfs/super.c +5
-0@@ -2487,6 +2487,11 @@ static int __init btrfs_print_mod_info(void)", fsverity=yes" #else ", fsverity=no" +#endif +#ifdef CONFIG_BTRFS_EXPERIMENTAL +", experimental=yes" +#else +", experimental=no"
I think this should be the first in the list so it's obvious. Also I'm not sure if we need to print it when it's disabled.
Quoted 6 lines
fs/btrfs/super.c +0
-0#endif ;pr_info ("Btrfs loaded%s\n" , options);Signature · 1 line
2.46.1
Re: [PATCH v3 07/10] btrfs: pr CONFIG_BTRFS_EXPERIMENTAL status
In reply to Anand JainQuoted 7 lines · Anand Jain
Commit
c9c49e8f157e("btrfs: split outCONFIG_BTRFS_EXPERIMENTALfromCONFIG_BTRFS_DEBUG") introduces a way to enable or disable experimental features, print its status during module load, like so:Btrfs loaded, debug=on, assert=on, zoned=yes, fsverity=yes
Signed-off-by Anand Jain <[email protected]>
Oh and please change the subject so it's actually understandable, 'pr' is not a verb.
Re: [PATCH v3 05/10] btrfs: introduce RAID1 round-robin read balancing
In reply to David SterbaQuoted 9 lines · David Sterba
Quoted 4 lines · Anand Jain
This feature balances I/O across the striped devices when reading from RAID1 blocks.
echo round-robin:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy
The ":" is part of the optional value, so the example should be "round-robin[:min]"
fixed.
Quoted 19 lines
Quoted 19 lines
The
min_contiguous_readparameter defines the minimum read size before switching to the next mirrored device. This setting is optional, with a default value of 256 KiB.Signed-off-by Anand Jain <[email protected]>--- fs/btrfs/sysfs.c | 38 +++++++++++++++++++++++++++++ fs/btrfs/volumes.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 9 +++++++ 3 files changed, 107 insertions(+)fs/btrfs/sysfs.c +0
-0@@ -1305,7 +1305,11 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj, } BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);
Quoted 9 lines
fs/btrfs/sysfs.c static ssize_t btrfs_temp_fsid_show(struct kobject *kobj, · line 1312
Quoted 5 lines
fs/btrfs/sysfs.c +4
-0+#ifdef CONFIG_BTRFS_EXPERIMENTAL +static const char *const btrfs_read_policy_name[] = {"pid" ,"round-robin" }; +#else static const char *const btrfs_read_policy_name[] = {"pid" }; +#endif Instead of the duplication of the definition please put the #ifdef around the experimental strings.
Now fixed.
Quoted 5 lines
Quoted 5 lines
fs/btrfs/sysfs.c +0
-0static enum btrfs_read_policy btrfs_read_policy_to_enum (const char *str,s64 *value) {@@ -1367,6 +1371,12 @@ static ssize_t btrfs_read_policy_show(struct kobject *kobj,
Quoted 9 lines
fs/btrfs/sysfs.c static ssize_t btrfs_read_policy_show(struct kobject *kobj, · line 1377
Quoted 6 lines
fs/btrfs/sysfs.c +4
-0ret +=sysfs_emit_at (buf, ret,"%s" , btrfs_read_policy_name[i]); +#ifdef CONFIG_BTRFS_EXPERIMENTAL +if (i ==BTRFS_READ_POLICY_RR ) + ret +=sysfs_emit_at (buf, ret,":%d" , + fs_devices->min_contiguous_read);Maybe this should be printed with the suffix, "rr:256k".
If the output has a suffix, we should also accept an input suffix. Perhaps I could do that separately after this patch set?
Quoted 9 lines
Quoted 9 lines
fs/btrfs/sysfs.c +2
-0+#endif +if (i == policy) ret +=sysfs_emit_at (buf, ret,"]" ); }@@ -1388,6 +1398,34 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj,if (index == -EINVAL )return -EINVAL ;
Quoted 9 lines
fs/btrfs/sysfs.c static ssize_t btrfs_read_policy_store(struct kobject *kobj, · line 1404
Quoted 4 lines
fs/btrfs/sysfs.c +4
-0+#ifdef CONFIG_BTRFS_EXPERIMENTAL +if (index ==BTRFS_READ_POLICY_RR ) { +if (value !=-1 ) { +if ((value % fs_devices->fs_info->sectorsize) !=0 ) {I think 64bit types and % could lead to the division which fails on 32bit platforms. As it's the s64 type for no apparent reason a u32 would be a better fit.
Yeah, that will be a problem. In v4, I'm using IS_ALIGNED(), and we need the value in the argument as a u64 because it also represents a devid.
Quoted 8 lines
fs/btrfs/sysfs.c static ssize_t btrfs_read_policy_store(struct kobject *kobj, · line 1408
Quoted 4 lines
fs/btrfs/sysfs.c +4
-0+btrfs_err (fs_devices->fs_info, +"read_policy: min_contiguous_read %lld should be multiples of the sectorsize %u" , + value, fs_devices->fs_info->sectorsize); +return -EINVAL ;This does not need to fail hard, we can simply round it up to the next sector size multiple.
Oh yeah that will be nice. Now fixed.
Quoted 2 lines
Also this does not validate the upper bound of value.
Um, IMO, there's no need to validate the upper bound as long as it is sector-size aligned, because it always upto the use case to decide. And, there is no theoretical limit we could set in the code.
Quoted 7 lines
fs/btrfs/sysfs.c static ssize_t btrfs_read_policy_store(struct kobject *kobj, · line 1412
Quoted 4 lines
fs/btrfs/sysfs.c +4
-0+ } + }else { +/* value is not provided, set it to the default 256k */ + value =256 *1024 ;Please use a named constant, then the comment is not necessary.
Done in v4.
Quoted 24 lines
Quoted 24 lines
fs/btrfs/sysfs.c +16
-0+ } + +if (index !=READ_ONCE (fs_devices->read_policy) || + value !=READ_ONCE (fs_devices->min_contiguous_read)) { +WRITE_ONCE (fs_devices->read_policy, index); +WRITE_ONCE (fs_devices->min_contiguous_read, value); +atomic_set (&fs_devices->total_reads,0 ); + +btrfs_info (fs_devices->fs_info,"read policy set to '%s:%lld'" , + btrfs_read_policy_name[index], value); + + } + +return len; + } +#endif if (index !=READ_ONCE (fs_devices->read_policy)) {WRITE_ONCE (fs_devices->read_policy, index); btrfs_info(fs_devices->fs_info, "read policy set to '%s'",fs/btrfs/volumes.c +0
-0@@ -1328,6 +1328,10 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices,
Quoted 9 lines
fs/btrfs/volumes.c static int open_fs_devices(struct btrfs_fs_devices *fs_devices, · line 1333
Quoted 6 lines
fs/btrfs/volumes.c +3
-0fs_devices->total_rw_bytes =0 ; fs_devices->chunk_alloc_policy =BTRFS_CHUNK_ALLOC_REGULAR ; fs_devices->read_policy =BTRFS_READ_POLICY_PID ; +#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Set min_contiguous_read to a default 256kib */ + fs_devices->min_contiguous_read =256 *1024 ;Named constant.
Fixed.
Quoted 8 lines
Quoted 8 lines
fs/btrfs/volumes.c +1
-0+#endif return 0 ; }@@ -5959,6 +5963,57 @@ unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info,return len; }
Quoted 13 lines
fs/btrfs/volumes.c unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, · line 5975
Quoted 10 lines
fs/btrfs/volumes.c +10
-0+#ifdef CONFIG_BTRFS_EXPERIMENTAL +struct stripe_mirror { +u64 devid; +int num; +}; + +static int btrfs_cmp_devid (const void *a,const void *b) +{ +struct stripe_mirror *s1 = (struct stripe_mirror *)a; +struct stripe_mirror *s2 = (struct stripe_mirror *)b;Please keep the const qualifier.
Fixed.
Quoted 4 lines
Quoted 4 lines
fs/btrfs/volumes.c +4
-0+ +if (s1->devid < s2->devid) +return -1 ; +if (s1->devid > s2->devid)
Quoted 9 lines
fs/btrfs/volumes.c unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, · line 5984
Quoted 5 lines
fs/btrfs/volumes.c +5
-0+return 1 ; +return 0 ; +} + +static int btrfs_read_rr (struct btrfs_chunk_map *map,int first,int num_stripe)Please add a comment explaining what the function does or how it selects the device.
Added.
Quoted 6 lines
fs/btrfs/volumes.c unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, · line 5986
Quoted 2 lines
fs/btrfs/volumes.c +2
-0+{ +struct stripe_mirror stripes[4 ] = {0 };//4: max possible mirrors Another magic constant. We probably won't have more than 4 but it still needs to be defined elsewhre. Also the comment format is wrong.
Fixed in v4 (in volume.h).
Quoted 7 lines
fs/btrfs/volumes.c unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, · line 5989
Quoted 3 lines
fs/btrfs/volumes.c +3
-0+struct btrfs_fs_devices *fs_devices; +struct btrfs_device *device; +int j;This is used in the for loop only, so you can declare it as for (int j = ...)
Fixed.
Quoted 5 lines
Quoted 5 lines
fs/btrfs/volumes.c +5
-0+int read_cycle; +int index; +int ret_stripe; +int total_reads; +int reads_per_dev =0 ;
Quoted 9 lines
fs/btrfs/volumes.c unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, · line 5999
Quoted 5 lines
fs/btrfs/volumes.c +5
-0+ + device = map->stripes[first].dev; + + fs_devices = device->fs_devices; + reads_per_dev = fs_devices->min_contiguous_read/fs_devices->fs_info->sectorsize;Missing space around '/' and we have the
sectorsize_bitsso division can be replaced by >>
Fixed.
Quoted 4 lines
fs/btrfs/volumes.c unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, · line 6001
Quoted 2 lines
fs/btrfs/volumes.c +2
-0+ index =0 ; +for (j = first; j < first + num_stripe; j++) {Actually you can use 'i' for iteration.
Fixed.
Quoted 13 lines
fs/btrfs/volumes.c unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, · line 6010
Quoted 10 lines
fs/btrfs/volumes.c +9
-0+ stripes[index].devid = map->stripes[j].dev->devid; + stripes[index].num = j; + index++; + } +sort (stripes, num_stripe,sizeof (struct stripe_mirror ), + btrfs_cmp_devid,NULL ); + + total_reads =atomic_inc_return (&fs_devices->total_reads); + read_cycle = total_reads/reads_per_dev;Missing spaces around "/"
Fixed.
Quoted 39 lines
Quoted 39 lines
fs/btrfs/volumes.c +11
-0+ ret_stripe = stripes[read_cycle % num_stripe].num; + + return ret_stripe; +} +#endif +static int find_live_mirror (struct btrfs_fs_info *fs_info,struct btrfs_chunk_map *map,int first,int dev_replace_is_ongoing)@@ -5988,6 +6043,11 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,case BTRFS_READ_POLICY_PID : preferred_mirror = first + (current->pid % num_stripes);break ; +#ifdef CONFIG_BTRFS_EXPERIMENTAL +case BTRFS_READ_POLICY_RR : + preferred_mirror =btrfs_read_rr (map, first, num_stripes); +break ; +#endif } if (dev_replace_is_ongoing &&fs/btrfs/volumes.h +4
-0@@ -303,6 +303,10 @@ enum btrfs_chunk_allocation_policy {enum btrfs_read_policy {/* Use process PID to choose the stripe */ BTRFS_READ_POLICY_PID , +#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Balancing raid1 reads across all striped devices (round-robin) */ +BTRFS_READ_POLICY_RR , +#endif BTRFS_NR_READ_POLICY , };@@ -431,6 +435,11 @@ struct btrfs_fs_devices {enum btrfs_read_policy read_policy;
Quoted 9 lines
fs/btrfs/volumes.h struct btrfs_fs_devices { · line 441
Quoted 5 lines
fs/btrfs/volumes.h +4
-0#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* IO stat, read counter. */ +atomic_t total_reads; +/* Min contiguous reads before switching to next device. */ +int min_contiguous_read;As this is in the
fs_devices, there should be some prefix that it's related to the read policy or round robin.
Naming is bit challenging given that it is long already, I have renamed to rr_min_contiguous_read.
Thanks, Anand
Quoted 7 lines
Quoted 7 lines
fs/btrfs/volumes.h +1
-0+/* Checksum mode - offload it or do it synchronously. */ enum btrfs_offload_csum_mode offload_csum_mode;#endif Signature · 1 line
2.46.1
Re: [PATCH v3 04/10] btrfs: handle value associated with raid1 balancing parameter
In reply to David SterbaQuoted 20 lines · David Sterba
Quoted 19 lines · Anand Jain
This change enables specifying additional configuration values alongside the raid1 balancing / read policy in a single input string.
Updated
btrfs_read_policy_to_enum()to parse and handle a value associated with the policy in the formatpolicy:value, the value part if present is converted 64-bit integer. Updatebtrfs_read_policy_store()to accommodate the new parameter.Signed-off-by Anand Jain <[email protected]>--- fs/btrfs/sysfs.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)fs/btrfs/sysfs.c +0
-0@@ -1307,8 +1307,11 @@ BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);
Quoted 7 lines
fs/btrfs/sysfs.c BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show); · line 1310
Quoted 4 lines
fs/btrfs/sysfs.c +1
-1static const char *const btrfs_read_policy_name[] = {"pid" }; -static enum btrfs_read_policy btrfs_read_policy_to_enum (const char *str) +static enum btrfs_read_policy btrfs_read_policy_to_enum (const char *str,s64 *value)Why is value signed type?
The signed type is intended for future enhancements.. For the "Device" raid1-balancing method, we need to specify either the actual device ID (as of now) or a negative number (in future) representing a device with lowest generation number. Perhaps -1? I'm ok to make it u64, if there is a better way.
Quoted 8 lines
fs/btrfs/sysfs.c BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show); · line 1314
Quoted 4 lines
fs/btrfs/sysfs.c +3
-0{ +#ifdef CONFIG_BTRFS_EXPERIMENTAL +char *value_str; +#endif Please keep the inline #ifdefs to minimum, with
__maybe_unusedthere won't be any warning if experimental build is disabled.
Oh, right. Will fix.
Quoted 7 lines
Quoted 7 lines
fs/btrfs/sysfs.c +0
-0bool found = false;enum btrfs_read_policy index;char *param;@@ -1320,6 +1323,18 @@ static enum btrfs_read_policy btrfs_read_policy_to_enum(const char *str)if (!param)return -ENOMEM ;
Quoted 10 lines
fs/btrfs/sysfs.c static enum btrfs_read_policy btrfs_read_policy_to_enum(const char *str) · line 1331
Quoted 6 lines
fs/btrfs/sysfs.c +6
-0+#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Separate value from input in policy:value format. */ +if ((value_str =strchr (param,':' ))) { + *value_str ='\0' ; + value_str++; +if (value &&kstrtou64 (value_str,10 , value) !=0 ) {You can assume that the 'value' is always non zero and assert it at the beginning of the function.
In v3 I had to check because patch 8/10 sends out arg with NULL. In v4 I have revised with a common helper in which the assert will help.
Quoted 6 lines
fs/btrfs/sysfs.c static enum btrfs_read_policy btrfs_read_policy_to_enum(const char *str) · line 1333
Quoted 2 lines
fs/btrfs/sysfs.c +2
-0+kfree (param); +return -EINVAL ;A negative errno in a function returning enum looks quite strange, either add an enum type for invalid value or use 'int'.
now declared it is int. It was simpler to implement.
Thanks.
Quoted 21 lines
Quoted 21 lines
fs/btrfs/sysfs.c +6
-1+ } + } +#endif +for (index =0 ; index <BTRFS_NR_READ_POLICY ; index++) {if (sysfs_streq (param, btrfs_read_policy_name[index])) { found = true;@@ -1367,8 +1382,9 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj, {struct btrfs_fs_devices *fs_devices =to_fs_devs (kobj);enum btrfs_read_policy index; +s64 value =-1 ; - index =btrfs_read_policy_to_enum (buf); + index =btrfs_read_policy_to_enum (buf, &value);if (index == -EINVAL )return -EINVAL ;Signature · 1 line
2.46.1
Re: [PATCH v3 07/10] btrfs: pr CONFIG_BTRFS_EXPERIMENTAL status
In reply to David SterbaQuoted 29 lines · David Sterba
Quoted 16 lines · Anand Jain
Commit
c9c49e8f157e("btrfs: split outCONFIG_BTRFS_EXPERIMENTALfromCONFIG_BTRFS_DEBUG") introduces a way to enable or disable experimental features, print its status during module load, like so:Btrfs loaded, debug=on, assert=on, zoned=yes, fsverity=yes
Signed-off-by Anand Jain <[email protected]>--- fs/btrfs/super.c | 5 +++++ 1 file changed, 5 insertions(+)fs/btrfs/super.c +0
-0fs/btrfs/super.c static int __init btrfs_print_mod_info(void) · line 2494
Quoted 8 lines
fs/btrfs/super.c +5
-0@@ -2487,6 +2487,11 @@ static int __init btrfs_print_mod_info(void)", fsverity=yes" #else ", fsverity=no" +#endif +#ifdef CONFIG_BTRFS_EXPERIMENTAL +", experimental=yes" +#else +", experimental=no" I think this should be the first in the list so it's obvious.
Quoted 2 lines
Also I'm
not sure if we need to print it when it's disabled.
It should be fine. It matches with debug and asset; they aren't printed when disabled.
So, currently:
Btrfs loaded, debug=on, assert=on, zoned=yes, fsverity=yes,
experimental=yes
Fixed to:
Btrfs loaded, experimental=on, debug=on, assert=on, zoned=yes,
fsverity=yes
And when disabled.
Btrfs loaded, debug=on, assert=on, zoned=yes, fsverity=yes
Thx.
Quoted 6 lines
Quoted 6 lines
fs/btrfs/super.c +0
-0#endif ;pr_info ("Btrfs loaded%s\n" , options);Signature · 1 line
2.46.1