raid1 balancing methods
v5 cover raid1 balancing methods
v5:
Fixes based on review comments:
. Rewrite btrfs_read_policy_to_enum() using sysfs_match_string()
and `strncpy()`.
. Rewrite the round-robin method based on read counts.
. Fix the smatch indentation warning.
- Change the default minimum contiguous device read size for round-robin
from 256K to 192K, as the latter performs slightly better.
- Introduce a framework to track filesystem read counts. (New patch)
- Reran defrag performance numbers
$ xfs_io -f -d -c 'pwrite -S 0xab 0 1000000000' /btrfs/P6B
$ time -p btrfs filesystem defrag -r -f -c /btrfs| | Time | Read I/O Count | gain | | | Real | devid1 | devid2 | w-PID | |---------|-------|--------|--------|-------| | pid | 11.14s| 3808 | 0 | - | | rotation| | | | | | 196608| 6.54s| 2532 | 1276 | 41.29%| | 262144| 8.42s| 1907 | 1901 | 24.41%| | devid:1 | 10.95s| 3807 | 0 | 1.70% |
v4:
Fixes based on review comments:
3/10: Use == 0 to check strlen(str); drop dynamic alloc for %param.
4/10: Add __maybe_unused for %value_str in btrfs_read_policy_to_enum().
Return int instead of enum btrfs_read_policy.
5/10: Fix change-log (: is part of optional [ ]).
Wrap btrfs_read_policy_name[] with ifdef for new methods.
Use IS_ALIGNED() for sector-size alignment check.
Roundup misaligned %value.
Use named constants: BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ, BTRFS_RAID1_MAX_MIRRORS. Mark %s1 and %s2 in btrfs_cmp_devid() as const.
Add comments to btrfs_read_rr();
Use loop-local %i. Add space around /.
Use >> for sector-size division.
Prefix %min_contiguous_read with rr.
7/10: Move experimental to the top of the feature list.
Use experiment=on. Skip printing when off.
v3:
1. Removed the latency-based RAID1 balancing patch. (Per David's review)
2. Renamed "rotation" to "round-robin" and set the per-set min_contiguous_read to 256k. (Per David's review)
3. Added raid1-balancing module configuration for fstests testing. raid1-balancing can now be configured through both module load parameters and sysfs.
The logic of individual methods remains unchanged, and performance metrics are consistent with v2.
-----
v2:
1. Move new features to CONFIG_BTRFS_EXPERIMENTAL instead of CONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait to U64_MAX and remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:
echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy
The default value is the sector size, and the min_contiguous_read value must be a multiple of the sector size.
5. Tested FIO random read/write and defrag compression workloads with min_contiguous_read set to sector size, 192k, and 256k.
RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation.
$ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \
--ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \
--time_based --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 20.3MiB/s | 20.5MiB/s | 313895 | 313895 | | rotation| | | | | | 4096| 20.4MiB/s | 20.5MiB/s | 313895 | 313895 | | 196608| 20.2MiB/s | 20.2MiB/s | 310152 | 310175 | | 262144| 20.3MiB/s | 20.4MiB/s | 312180 | 312191 | | latency| 18.4MiB/s | 18.4MiB/s | 272980 | 291683 | | devid:1 | 14.8MiB/s | 14.9MiB/s | 456376 | 0 |
rotation RAID1 balancing technique performs more than 2x better for single-process defrag.
$ time -p btrfs filesystem defrag -r -f -c /btrfs
| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 18.00s| 3800 | 0 | | rotation| | | | | 4096| 8.95s| 1900 | 1901 | | 196608| 8.50s| 1881 | 1919 | | 262144| 8.80s| 1881 | 1919 | | latency | 17.18s| 3800 | 0 | | devid:2 | 17.48s| 0 | 3800 |
Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.
Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.
Latency is smarter with unstable block layer transport.
Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.
Devid is a hands-on approach, provides manual or user-space script control.
These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.
Thx.
--------- original v1 ------------
The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the CONFIG_BTRFS_DEBUG config option and are on top of the previously added /sys/fs/btrfs/<UUID>/read_policy interface to configure the desired RAID1 read balancing method.
I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from /sys/devices/<..>/stat for each device. Below is a summary of the results, with each result the average of three iterations.
A typical generic random rw workload:
$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \ --group_reporting --name=iops-test-job --eta-newline=1
| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 29.4MiB/s | 29.5MiB/s | 456548 | 447975 | | rotation| 29.3MiB/s | 29.3MiB/s | 450105 | 450055 | | latency | 21.9MiB/s | 21.9MiB/s | 672387 | 0 | | devid:1 | 22.0MiB/s | 22.0MiB/s | 674788 | 0 |
Defragmentation with compression workload:
$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo $ sync $ echo 3 > /proc/sys/vm/drop_caches $ btrfs filesystem defrag -f -c /btrfs/foo
| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 21.61s| 3810 | 0 | | rotation| 11.55s| 1905 | 1905 | | latency | 20.99s| 0 | 3810 | | devid:2 | 21.41s| 0 | 3810 |
. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.
Anand Jain (10):
btrfs: initialize fs_devices->fs_info earlier
btrfs: simplify output formatting in btrfs_read_policy_show
btrfs: add btrfs_read_policy_to_enum helper and refactor read policy
store
btrfs: handle value associated with raid1 balancing parameter
btrfs: add read count tracking for filesystem stats
btrfs: introduce RAID1 round-robin read balancing
btrfs: add RAID1 preferred read device
btrfs: expose experimental mode in module information
btrfs: enable RAID1 balancing configuration via modprobe parameter
btrfs: modload to print RAID1 balancing status
fs/btrfs/bio.c | 8 +++ fs/btrfs/disk-io.c | 4 ++ fs/btrfs/super.c | 18 +++++ fs/btrfs/sysfs.c | 168 ++++++++++++++++++++++++++++++++++++++++----- fs/btrfs/sysfs.h | 5 ++ fs/btrfs/volumes.c | 115 ++++++++++++++++++++++++++++++- fs/btrfs/volumes.h | 22 +++++- 7 files changed, 317 insertions(+), 23 deletions(-)
Signature · 1 line
2.47.0
PATCH v5 1/10 btrfs: initialize fs_devices->fs_info earlier
Currently, fs_devices->fs_info is initialized in btrfs_init_devices_late(), but this occurs too late for find_live_mirror(), which is invoked by load_super_root() much earlier than btrfs_init_devices_late().
Fix this by moving the initialization to open_ctree(), before load_super_root().
--- fs/btrfs/disk-io.c | 1 + fs/btrfs/volumes.c | 2 -- 2 files changed, 1 insertion(+), 2 deletions(-)
fs/btrfs/disk-io.c +1 -0
@@ -3321,6 +3321,7 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device fs_info->sectors_per_page = (PAGE_SIZE >> fs_info->sectorsize_bits); fs_info->csums_per_leaf =BTRFS_MAX_ITEM_SIZE (fs_info) / fs_info->csum_size; fs_info->stripesize = stripesize; + fs_info->fs_devices->fs_info = fs_info;/* * Handle the space caching options appropriately now that we have the
fs/btrfs/volumes.c +0 -2
@@ -7568,8 +7568,6 @@ int btrfs_init_devices_late(struct btrfs_fs_info *fs_info)struct btrfs_device *device;int ret =0 ; - fs_devices->fs_info = fs_info; -mutex_lock (&fs_devices->device_list_mutex);list_for_each_entry (device, &fs_devices->devices, dev_list) device->fs_info = fs_info;
Signature · 1 line
2.47.0
PATCH v5 2/10 btrfs: simplify output formatting in btrfs_read_policy_show
In reply to Anand JainRefactor the logic in btrfs_read_policy_show() to streamline the formatting of read policies output. Streamline the space and bracket handling around the active policy without altering the functional output. This is in preparation to add more methods.
--- fs/btrfs/sysfs.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)
fs/btrfs/sysfs.c +10 -8
@@ -1316,14 +1316,16 @@ static ssize_t btrfs_read_policy_show(struct kobject *kobj,int i;for (i =0 ; i <BTRFS_NR_READ_POLICY ; i++) { -if (policy == i) - ret +=sysfs_emit_at (buf, ret,"%s[%s]" , - (ret ==0 ?"" :" " ), - btrfs_read_policy_name[i]); -else - ret +=sysfs_emit_at (buf, ret,"%s%s" , - (ret ==0 ?"" :" " ), - btrfs_read_policy_name[i]); +if (ret !=0 ) + ret +=sysfs_emit_at (buf, ret," " ); + +if (i == policy) + ret +=sysfs_emit_at (buf, ret,"[" ); + + ret +=sysfs_emit_at (buf, ret,"%s" , btrfs_read_policy_name[i]); + +if (i == policy) + ret +=sysfs_emit_at (buf, ret,"]" ); } ret +=sysfs_emit_at (buf, ret,"\n" );
Signature · 1 line
2.47.0
PATCH v5 3/10 btrfs: add btrfs_read_policy_to_enum helper and refactor read policy store
In reply to Anand JainIntroduce the btrfs_read_policy_to_enum helper function to simplify the conversion of a string read policy to its corresponding enum value. This reduces duplication and improves code clarity in btrfs_read_policy_store. The btrfs_read_policy_store function has been refactored to use the new helper.
The parameter is copied locally to allow modification, enabling the separation of the method and its value. This prepares for the addition of more functionality in subsequent patches.
--- fs/btrfs/sysfs.c | 34 ++++++++++++++++++++++------------ 1 file changed, 22 insertions(+), 12 deletions(-)
fs/btrfs/sysfs.c +22 -12
@@ -1307,6 +1307,18 @@ BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);static const char *const btrfs_read_policy_name[] = {"pid" }; +static int btrfs_read_policy_to_enum (const char *str) +{ +char param[32 ] = {'\0' }; + +if (!str ||strlen (str) ==0 ) +return 0 ; + +strncpy (param, str,sizeof (param) -1 ); + +return sysfs_match_string (btrfs_read_policy_name, param); +} +static ssize_t btrfs_read_policy_show (struct kobject *kobj,struct kobj_attribute *a,char *buf) {
@@ -1338,21 +1350,19 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj, const char *buf, size_t len) {struct btrfs_fs_devices *fs_devices =to_fs_devs (kobj); -int i; +int index; -for (i =0 ; i <BTRFS_NR_READ_POLICY ; i++) { -if (sysfs_streq (buf, btrfs_read_policy_name[i])) { -if (i !=READ_ONCE (fs_devices->read_policy)) { -WRITE_ONCE (fs_devices->read_policy, i); -btrfs_info (fs_devices->fs_info, -"read policy set to '%s'" , - btrfs_read_policy_name[i]); - } -return len; - } + index =btrfs_read_policy_to_enum (buf); +if (index <0 ) +return -EINVAL ; + +if (index !=READ_ONCE (fs_devices->read_policy)) { +WRITE_ONCE (fs_devices->read_policy, index); +btrfs_info (fs_devices->fs_info,"read policy set to '%s'" , + btrfs_read_policy_name[index]); } -return -EINVAL ; +return len; } BTRFS_ATTR_RW(, read_policy, btrfs_read_policy_show, btrfs_read_policy_store);
Signature · 1 line
2.47.0
PATCH v5 4/10 btrfs: handle value associated with raid1 balancing parameter
In reply to Anand JainThis change enables specifying additional configuration values alongside the raid1 balancing / read policy in a single input string.
Updated btrfs_read_policy_to_enum() to parse and handle a value associated with the policy in the format policy:value, the value part if present is converted 64-bit integer. Update btrfs_read_policy_store() to accommodate the new parameter.
--- fs/btrfs/sysfs.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
fs/btrfs/sysfs.c +14 -2
@@ -1307,15 +1307,26 @@ BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);static const char *const btrfs_read_policy_name[] = {"pid" }; -static int btrfs_read_policy_to_enum (const char *str) +static int btrfs_read_policy_to_enum (const char *str,s64 *value) {char param[32 ] = {'\0' }; +char *__maybe_unused value_str;if (!str ||strlen (str) ==0 )return 0 ;strncpy (param, str,sizeof (param) -1 ); +#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Separate value from input in policy:value format. */ +if ((value_str =strchr (param,':' ))) { + *value_str ='\0' ; + value_str++; +if (value &&kstrtou64 (value_str,10 , value) !=0 ) +return -EINVAL ; + } +#endif +return sysfs_match_string (btrfs_read_policy_name, param); }
@@ -1351,8 +1362,9 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj, {struct btrfs_fs_devices *fs_devices =to_fs_devs (kobj);int index; +s64 value =-1 ; - index =btrfs_read_policy_to_enum (buf); + index =btrfs_read_policy_to_enum (buf, &value);if (index <0 )return -EINVAL ;
Signature · 1 line
2.47.0
Re: [PATCH v5 04/10] btrfs: handle value associated with raid1 balancing parameter
Quoted 29 lines · Anand Jain
This change enables specifying additional configuration values alongside the raid1 balancing / read policy in a single input string.
Updated
btrfs_read_policy_to_enum()to parse and handle a value associated with the policy in the formatpolicy:value, the value part if present is converted 64-bit integer. Updatebtrfs_read_policy_store()to accommodate the new parameter.Signed-off-by Anand Jain <[email protected]>--- fs/btrfs/sysfs.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)fs/btrfs/sysfs.c +2
-1@@ -1307,15 +1307,26 @@ BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);static const char *const btrfs_read_policy_name[] = {"pid" }; -static int btrfs_read_policy_to_enum (const char *str) +static int btrfs_read_policy_to_enum (const char *str,s64 *value) {char param[32 ] = {'\0' }; +char *__maybe_unused value_str;if (!str ||strlen (str) ==0 )return 0 ;
fs/btrfs/sysfs.c BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show); · line 1326
Quoted 10 lines
fs/btrfs/sysfs.c +7
-0strncpy (param, str,sizeof (param) -1 ); +#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Separate value from input in policy:value format. */ +if ((value_str =strchr (param,':' ))) { + *value_str ='\0' ; + value_str++; +if (value &&kstrtou64 (value_str,10 , value) !=0 ) +return -EINVAL ;
I think I've mentioned that before, this lacks validation of the 'value', there should be some lower and upper bound check. Minimum can be the sectorsize and maximum maybe 1G or 2G, so the u32 type is sufficient. The silent conversion from u64 to s64 should be avoided.
Re: [PATCH v5 04/10] btrfs: handle value associated with raid1 balancing parameter
Quoted 33 lines · David Sterba
Quoted 32 lines · Anand Jain
This change enables specifying additional configuration values alongside the raid1 balancing / read policy in a single input string.
Updated
btrfs_read_policy_to_enum()to parse and handle a value associated with the policy in the formatpolicy:value, the value part if present is converted 64-bit integer. Updatebtrfs_read_policy_store()to accommodate the new parameter.Signed-off-by Anand Jain <[email protected]>--- fs/btrfs/sysfs.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)fs/btrfs/sysfs.c +2
-1@@ -1307,15 +1307,26 @@ BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show);static const char *const btrfs_read_policy_name[] = {"pid" }; -static int btrfs_read_policy_to_enum (const char *str) +static int btrfs_read_policy_to_enum (const char *str,s64 *value) {char param[32 ] = {'\0' }; +char *__maybe_unused value_str;if (!str ||strlen (str) ==0 )return 0 ;strncpy (param, str,sizeof (param) -1 );
Quoted 12 lines
fs/btrfs/sysfs.c BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show); · line 1326
Quoted 7 lines
fs/btrfs/sysfs.c +7
-0+#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Separate value from input in policy:value format. */ +if ((value_str =strchr (param,':' ))) { + *value_str ='\0' ; + value_str++; +if (value &&kstrtou64 (value_str,10 , value) !=0 ) +return -EINVAL ;I think I've mentioned that before, this lacks validation of the 'value', there should be some lower and upper bound check. Minimum can be the sectorsize and maximum maybe 1G or 2G, so the u32 type is sufficient.
Regarding the lack of validation, as I replied earlier, only the minimum needs validation, not the upper limit. A 1TB disk or SAN storage can be connected to a 32-bit host, and we can still read the full 1TB from a single mirror. IMO, it depends on the use case, or we can revisit it based on feedback.?
The patch already handles the lower limit.
$ echo round-robin:1024 > /sys/fs/btrfs/b03580af-900e-4940-a7f8-3f04f9981a12/read_policy
$ cat /sys/fs/btrfs/b03580af-900e-4940-a7f8-3f04f9981a12/read_policy pid [round-robin:4096] devid:1
$ dmesg -k | tail -1 BTRFS warning (device sda): read_policy: min contiguous read 1024 should be multiples of the sectorsize 4096, rounded to 4096
Quoted 1 line
The silent conversion from u64 to s64 should be avoided.
Do you mean from s64 to u64?
Thanks, Anand
Re: [PATCH v5 04/10] btrfs: handle value associated with raid1 balancing parameter
Quoted 13 lines · Anand Jain
Quoted 4 lines
Quoted 4 lines
fs/btrfs/sysfs.c +4
-0+#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Separate value from input in policy:value format. */ +if ((value_str =strchr (param,':' ))) { + *value_str ='\0' ;Quoted 8 lines
fs/btrfs/sysfs.c BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show); · line 1326
Quoted 3 lines
fs/btrfs/sysfs.c +3
-0+ value_str++; +if (value &&kstrtou64 (value_str,10 , value) !=0 ) +return -EINVAL ;I think I've mentioned that before, this lacks validation of the 'value', there should be some lower and upper bound check. Minimum can be the sectorsize and maximum maybe 1G or 2G, so the u32 type is sufficient.
Quoted 5 lines
Regarding the lack of validation, as I replied earlier, only the minimum needs validation, not the upper limit. A 1TB disk or SAN storage can be connected to a 32-bit host, and we can still read the full 1TB from a single mirror. IMO, it depends on the use case, or we can revisit it based on feedback.?
SAN is not a typical storage, what the load balancing should assume is a set of individual rotational or ssd/nvme disks. Also 32bit host is not something we're expecting by default.
Quoted 11 lines
The patch already handles the lower limit.
$ echo round-robin:1024 > /sys/fs/btrfs/b03580af-900e-4940-a7f8-3f04f9981a12/read_policy$ cat /sys/fs/btrfs/b03580af-900e-4940-a7f8-3f04f9981a12/read_policy pid [round-robin:4096] devid:1$ dmesg -k | tail -1 BTRFS warning (device sda): read_policy: min contiguous read 1024 should be multiples of the sectorsize 4096, rounded to 4096
Right.
Quoted 3 lines
Quoted 1 line
The silent conversion from u64 to s64 should be avoided.
Do you mean from s64 to u64?
Yes, I've used kstrtos64 and reject negative values, then the types match. I'm not sure about the upper bound, the idea was like 1G or 2G but a fast NVME can do a few gigabytes per second so we may need more than a 32bit type can hold. But it's counted in blocks it's 4G * 4K so it's safe.
PATCH v5 5/10 btrfs: add read count tracking for filesystem stats
In reply to Anand JainAdd fs_devices::read_cnt_blocks to track read blocks, initialize it in open_fs_devices() and clean it up in close_fs_devices(). btrfs_submit_dev_bio() increments it for reads when stats tracking is enabled. Stats tracking is disabled by default and is enabled through fs_devices::fs_stats when required.
The code is not under the EXPERIMENTAL define, as stats can be expanded to include write counts and other performance counters, with the user interface independent of its internal use.
This is an in-memory-only feature, different to the dev error stats.
--- fs/btrfs/bio.c | 8 ++++++++ fs/btrfs/disk-io.c | 3 +++ fs/btrfs/volumes.c | 8 +++++++- fs/btrfs/volumes.h | 7 ++++++- 4 files changed, 24 insertions(+), 2 deletions(-)
fs/btrfs/bio.c +8 -0
@@ -450,6 +450,14 @@ static void btrfs_submit_dev_bio(struct btrfs_device *dev, struct bio *bio) (unsigned long )dev->bdev->bd_dev,btrfs_dev_name (dev), dev->devid, bio->bi_iter.bi_size); +/* +* Track reads if tracking is enabled; ignore I/O operations before +* fully initialized. +*/ +if (dev->fs_devices->fs_stats &&bio_op (bio) ==REQ_OP_READ && dev->fs_info) +percpu_counter_add (&dev->fs_devices->read_cnt_blocks, + bio->bi_iter.bi_size >> dev->fs_info->sectorsize_bits); +if (bio->bi_opf &REQ_BTRFS_CGROUP_PUNT )blkcg_punt_bio_submit (bio);else
fs/btrfs/disk-io.c +3 -0
@@ -3481,6 +3481,9 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device goto fail_sysfs; } +/* Disable filesystem stats tracking unless required by a feature. */ + fs_devices->fs_stats = false; + ret =btrfs_read_block_groups (fs_info);if (ret) {btrfs_err (fs_info,"failed to read block groups: %d" , ret);
fs/btrfs/volumes.c +7 -1
@@ -1254,6 +1254,7 @@ static void close_fs_devices(struct btrfs_fs_devices *fs_devices)list_for_each_entry_safe (device, tmp, &fs_devices->devices, dev_list)btrfs_close_one_device (device); +percpu_counter_destroy (&fs_devices->read_cnt_blocks);WARN_ON (fs_devices->open_devices);WARN_ON (fs_devices->rw_devices); fs_devices->opened =0 ;
@@ -1300,6 +1301,11 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices,struct btrfs_device *tmp_device;int ret =0 ; +/* Initialize the in-memory record of filesystem read count */ + ret =percpu_counter_init (&fs_devices->read_cnt_blocks,0 ,GFP_KERNEL ); +if (ret) +return ret; +list_for_each_entry_safe (device, tmp_device, &fs_devices->devices, dev_list) {int ret2;
@@ -7669,7 +7675,7 @@ int btrfs_init_dev_stats(struct btrfs_fs_info *fs_info)list_for_each_entry (device, &fs_devices->devices, dev_list) { ret =btrfs_device_init_dev_stats (device, path);if (ret) - goto out; +return ret; }list_for_each_entry (seed_devs, &fs_devices->seed_list, seed_list) {list_for_each_entry (device, &seed_devs->devices, dev_list) {
fs/btrfs/volumes.h +6 -1
@@ -185,7 +185,7 @@ struct btrfs_device {* enum btrfs_dev_stat_values in ioctl.h */ int dev_stats_valid; -/* Counter to record the change of device stats */ +/* Counter to record of the change of device stats */ atomic_t dev_stats_ccnt;atomic_t dev_stat_values[BTRFS_DEV_STAT_VALUES_MAX ];
@@ -417,6 +417,8 @@ struct btrfs_fs_devices {bool seeding;/* The mount needs to use a randomly generated fsid. */ bool temp_fsid; +/* Enable/disable the filesystem stats tracking */ +bool fs_stats;struct btrfs_fs_info *fs_info;/* sysfs kobjects */
@@ -427,6 +429,9 @@ struct btrfs_fs_devices {enum btrfs_chunk_allocation_policy chunk_alloc_policy; +/* Tracks the number of blocks (sectors) read from the filesystem. */ +struct percpu_counter read_cnt_blocks; +/* Policy used to read the mirrored stripes. */ enum btrfs_read_policy read_policy;
Signature · 1 line
2.47.0
PATCH 1/1 fixup: btrfs: add read count tracking for filesystem stats
fs_devices::fs_stats is disabled by default during allocation. Explicitly disabling it in open_ctree() overrides this setting during device open. So we dont' have to disable it in the open_ctree().
This patch should be merged with in the ML.
btrfs: add read count tracking for filesystem stats
--- fs/btrfs/disk-io.c | 3 --- 1 file changed, 3 deletions(-)
fs/btrfs/disk-io.c +0 -3
@@ -3481,9 +3481,6 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device goto fail_sysfs; } -/* Disable filesystem stats tracking unless required by a feature. */ - fs_devices->fs_stats = false; - ret =btrfs_read_block_groups (fs_info);if (ret) {btrfs_err (fs_info,"failed to read block groups: %d" , ret);
Signature · 1 line
2.47.0
Re: [PATCH v5 05/10] btrfs: add read count tracking for filesystem stats
In reply to Anand JainHi Anand,
kernel test robot noticed the following build warnings:
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Anand-Jain/btrfs-initialize-fs_devices-fs_info-earlier/20250102-021904 base: v6.13-rc5 patch link: https://lore.kernel.org/r/c4010cee5398e35a695def3ad97d4de6f136ae2c.1735748715.git.anand.jain%40oracle.com patch subject: [PATCH v5 05/10] btrfs: add read count tracking for filesystem stats config: i386-randconfig-r072-20250103 (https://download.01.org/0day-ci/archive/20250104/[email protected]/config) compiler: clang version 19.1.3 (https://github.com/llvm/llvm-project ab51eccf88f5321e7c60591c5546b254b6afab99)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]> | Reported-by: Dan Carpenter <[email protected]> | Closes: https://lore.kernel.org/r/[email protected]/
New smatch warnings:fs/btrfs/volumes.c:7691 btrfs_init_dev_stats() warn: inconsistent returns '&fs_devices->device_list_mutex'.
vim +7691 fs/btrfs/volumes.c
124604eb50f88e Josef Bacik 2020-09-18 7663 int btrfs_init_dev_stats(struct btrfs_fs_info *fs_info)
124604eb50f88e Josef Bacik 2020-09-18 7664 {
124604eb50f88e Josef Bacik 2020-09-18 7665 struct btrfs_fs_devices *fs_devices = fs_info->fs_devices, *seed_devs;
124604eb50f88e Josef Bacik 2020-09-18 7666 struct btrfs_device *device;
124604eb50f88e Josef Bacik 2020-09-18 7667 struct btrfs_path *path = NULL;
92e26df43b1a97 Josef Bacik 2020-09-18 7668 int ret = 0;
124604eb50f88e Josef Bacik 2020-09-18 7669
124604eb50f88e Josef Bacik 2020-09-18 7670 path = btrfs_alloc_path();
124604eb50f88e Josef Bacik 2020-09-18 7671 if (!path)
124604eb50f88e Josef Bacik 2020-09-18 7672 return -ENOMEM;
124604eb50f88e Josef Bacik 2020-09-18 7673
124604eb50f88e Josef Bacik 2020-09-18 7674 mutex_lock(&fs_devices->device_list_mutex);
92e26df43b1a97 Josef Bacik 2020-09-18 7675 list_for_each_entry(device, &fs_devices->devices, dev_list) {
92e26df43b1a97 Josef Bacik 2020-09-18 7676 ret = btrfs_device_init_dev_stats(device, path);
92e26df43b1a97 Josef Bacik 2020-09-18 7677 if (ret)
ec90aa75ef29fb Anand Jain 2025-01-02 7678 return ret;mutex_unlock(&fs_devices->device_list_mutex);
92e26df43b1a97 Josef Bacik 2020-09-18 7679 }
124604eb50f88e Josef Bacik 2020-09-18 7680 list_for_each_entry(seed_devs, &fs_devices->seed_list, seed_list) {
92e26df43b1a97 Josef Bacik 2020-09-18 7681 list_for_each_entry(device, &seed_devs->devices, dev_list) {
92e26df43b1a97 Josef Bacik 2020-09-18 7682 ret = btrfs_device_init_dev_stats(device, path);
92e26df43b1a97 Josef Bacik 2020-09-18 7683 if (ret)
92e26df43b1a97 Josef Bacik 2020-09-18 7684 goto out;
124604eb50f88e Josef Bacik 2020-09-18 7685 }
92e26df43b1a97 Josef Bacik 2020-09-18 7686 }
92e26df43b1a97 Josef Bacik 2020-09-18 7687 out:
733f4fbbc1083a Stefan Behrens 2012-05-25 7688 mutex_unlock(&fs_devices->device_list_mutex);
733f4fbbc1083a Stefan Behrens 2012-05-25 7689
733f4fbbc1083a Stefan Behrens 2012-05-25 7690 btrfs_free_path(path);92e26df43b1a97 Josef Bacik 2020-09-18 @7691 return ret;733f4fbbc1083a Stefan Behrens 2012-05-25 7692 }
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v5 05/10] btrfs: add read count tracking for filesystem stats
Hi Dan,
This was fixed earlier. Thanks for the report.
-Anand
Quoted 58 lines · Dan Carpenter
Hi Anand,
kernel test robot noticed the following build warnings:
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Anand-Jain/btrfs-initialize-fs_devices-fs_info-earlier/20250102-021904 base: v6.13-rc5 patch link: https://lore.kernel.org/r/c4010cee5398e35a695def3ad97d4de6f136ae2c.1735748715.git.anand.jain%40oracle.com patch subject: [PATCH v5 05/10] btrfs: add read count tracking for filesystem stats config: i386-randconfig-r072-20250103 (https://download.01.org/0day-ci/archive/20250104/[email protected]/config) compiler: clang version 19.1.3 (https://github.com/llvm/llvm-project ab51eccf88f5321e7c60591c5546b254b6afab99)If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]> | Reported-by: Dan Carpenter <[email protected]> | Closes: https://lore.kernel.org/r/[email protected]/New smatch warnings:
fs/btrfs/volumes.c:7691btrfs_init_dev_stats()warn: inconsistent returns '&fs_devices->device_list_mutex'.vim +7691
fs/btrfs/volumes.c124604eb50f88e Josef Bacik 2020-09-18 7663 int btrfs_init_dev_stats(struct btrfs_fs_info *fs_info) 124604eb50f88e Josef Bacik 2020-09-18 7664 { 124604eb50f88e Josef Bacik 2020-09-18 7665 struct btrfs_fs_devices *fs_devices = fs_info->fs_devices, *seed_devs; 124604eb50f88e Josef Bacik 2020-09-18 7666 struct btrfs_device *device; 124604eb50f88e Josef Bacik 2020-09-18 7667 struct btrfs_path *path = NULL; 92e26df43b1a97 Josef Bacik 2020-09-18 7668 int ret = 0; 124604eb50f88e Josef Bacik 2020-09-18 7669 124604eb50f88e Josef Bacik 2020-09-18 7670 path = btrfs_alloc_path(); 124604eb50f88e Josef Bacik 2020-09-18 7671 if (!path) 124604eb50f88e Josef Bacik 2020-09-18 7672 return -ENOMEM; 124604eb50f88e Josef Bacik 2020-09-18 7673 124604eb50f88e Josef Bacik 2020-09-18 7674 mutex_lock(&fs_devices->device_list_mutex); 92e26df43b1a97 Josef Bacik 2020-09-18 7675 list_for_each_entry(device, &fs_devices->devices, dev_list) { 92e26df43b1a97 Josef Bacik 2020-09-18 7676 ret = btrfs_device_init_dev_stats(device, path); 92e26df43b1a97 Josef Bacik 2020-09-18 7677 if (ret) ec90aa75ef29fb Anand Jain 2025-01-02 7678 return ret;
mutex_unlock(&fs_devices->device_list_mutex);92e26df43b1a97 Josef Bacik 2020-09-18 7679 } 124604eb50f88e Josef Bacik 2020-09-18 7680 list_for_each_entry(seed_devs, &fs_devices->seed_list, seed_list) { 92e26df43b1a97 Josef Bacik 2020-09-18 7681 list_for_each_entry(device, &seed_devs->devices, dev_list) { 92e26df43b1a97 Josef Bacik 2020-09-18 7682 ret = btrfs_device_init_dev_stats(device, path); 92e26df43b1a97 Josef Bacik 2020-09-18 7683 if (ret) 92e26df43b1a97 Josef Bacik 2020-09-18 7684 goto out; 124604eb50f88e Josef Bacik 2020-09-18 7685 } 92e26df43b1a97 Josef Bacik 2020-09-18 7686 } 92e26df43b1a97 Josef Bacik 2020-09-18 7687 out: 733f4fbbc1083a Stefan Behrens 2012-05-25 7688 mutex_unlock(&fs_devices->device_list_mutex); 733f4fbbc1083a Stefan Behrens 2012-05-25 7689 733f4fbbc1083a Stefan Behrens 2012-05-25 7690 btrfs_free_path(path);
92e26df43b1a97Josef Bacik 2020-09-18 @7691 return ret;733f4fbbc1083aStefan Behrens 2012-05-25 7692 }
PATCH v5 6/10 btrfs: introduce RAID1 round-robin read balancing
In reply to Anand JainThis feature balances I/O across the striped devices when reading from RAID1 blocks.
echo round-robin[:min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy
The min_contiguous_read parameter defines the minimum read size before switching to the next mirrored device. This setting is optional, with a default value of 192KiB.
--- fs/btrfs/sysfs.c | 49 ++++++++++++++++++++++++++++++- fs/btrfs/volumes.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.h | 10 +++++++ 3 files changed, 130 insertions(+), 1 deletion(-)
fs/btrfs/sysfs.c +48 -1
@@ -1305,7 +1305,12 @@ static ssize_t btrfs_temp_fsid_show(struct kobject *kobj, } BTRFS_ATTR(, temp_fsid, btrfs_temp_fsid_show); -static const char *const btrfs_read_policy_name[] = {"pid" }; +static const char *btrfs_read_policy_name[] = { +"pid" , +#ifdef CONFIG_BTRFS_EXPERIMENTAL +"round-robin" , +#endif +};static int btrfs_read_policy_to_enum (const char *str,s64 *value) {
@@ -1347,6 +1352,12 @@ static ssize_t btrfs_read_policy_show(struct kobject *kobj, ret +=sysfs_emit_at (buf, ret,"%s" , btrfs_read_policy_name[i]); +#ifdef CONFIG_BTRFS_EXPERIMENTAL +if (i ==BTRFS_READ_POLICY_RR ) + ret +=sysfs_emit_at (buf, ret,":%d" , +READ_ONCE (fs_devices->rr_min_contiguous_read)); +#endif +if (i == policy) ret +=sysfs_emit_at (buf, ret,"]" ); }
@@ -1368,6 +1379,42 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj,if (index <0 )return -EINVAL ; +#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* If moving out of RR then disable fs_stats */ +if (fs_devices->read_policy ==BTRFS_READ_POLICY_RR && + index !=BTRFS_READ_POLICY_RR ) + fs_devices->fs_stats = false; + +if (index ==BTRFS_READ_POLICY_RR ) { +if (value !=-1 ) { +u32 sectorsize = fs_devices->fs_info->sectorsize; + +if (!IS_ALIGNED (value, sectorsize)) { +u64 temp_value =round_up (value, sectorsize); + +btrfs_warn (fs_devices->fs_info, +"read_policy: min contiguous read %lld should be multiples of the sectorsize %u, rounded to %llu" , + value, sectorsize, temp_value); + value = temp_value; + } + }else { + value =BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ ; + } + +if (index !=READ_ONCE (fs_devices->read_policy) || + value !=READ_ONCE (fs_devices->rr_min_contiguous_read)) { +WRITE_ONCE (fs_devices->read_policy, index); +WRITE_ONCE (fs_devices->rr_min_contiguous_read, value); + +btrfs_info (fs_devices->fs_info,"read policy set to '%s:%lld'" , + btrfs_read_policy_name[index], value); + } + + fs_devices->fs_stats = true; + +return len; + } +#endif if (index !=READ_ONCE (fs_devices->read_policy)) {WRITE_ONCE (fs_devices->read_policy, index); btrfs_info(fs_devices->fs_info, "read policy set to '%s'",
fs/btrfs/volumes.c +72 -0
90 lines not on this page · raw message
fs/btrfs/volumes.h +10 -0
@@ -296,6 +296,9 @@ enum btrfs_chunk_allocation_policy { BTRFS_CHUNK_ALLOC_ZONED, }; +/* SZ_192K = 192 * 1024 = 196608 */ +#define BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ (196608) +#define BTRFS_RAID1_MAX_MIRRORS (4)/* * Read policies for mirrored block group profiles, read picks the stripe based * on these policies.
@@ -303,6 +306,10 @@ enum btrfs_chunk_allocation_policy {enum btrfs_read_policy {/* Use process PID to choose the stripe */ BTRFS_READ_POLICY_PID , +#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Balancing raid1 reads across all striped devices (round-robin) */ +BTRFS_READ_POLICY_RR , +#endif BTRFS_NR_READ_POLICY , };
@@ -436,6 +443,9 @@ struct btrfs_fs_devices {enum btrfs_read_policy read_policy;#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Min contiguous reads before switching to next device. */ +int rr_min_contiguous_read; +/* Checksum mode - offload it or do it synchronously. */ enum btrfs_offload_csum_mode offload_csum_mode;#endif
Signature · 1 line
2.47.0
PATCH 1/1 fixup: btrfs: introduce RAID1 round-robin read balancing
For now, let's keep the more typical storage device cache size at 256 KB, which can be updated once we have more widespread tests.
This patch should be merged with in the ML.
btrfs: introduce RAID1 round-robin read balancing
--- fs/btrfs/volumes.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
fs/btrfs/volumes.h +1 -2
@@ -296,8 +296,7 @@ enum btrfs_chunk_allocation_policy { BTRFS_CHUNK_ALLOC_ZONED, }; -/* SZ_192K = 192 * 1024 = 196608 */ -#define BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ (196608) +#define BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ (SZ_256K)#define BTRFS_RAID1_MAX_MIRRORS (4)/* * Read policies for mirrored block group profiles, read picks the stripe based
Signature · 1 line
2.47.0
PATCH v5 7/10 btrfs: add RAID1 preferred read device
In reply to Anand JainWhen there's stale data on a mirrored device, this feature lets you choose which device to read from. Mainly used for testing.
echo "devid:<devid-value>" > /sys/fs/btrfs/<UUID>/read_policy
--- fs/btrfs/sysfs.c | 33 ++++++++++++++++++++++++++++++++- fs/btrfs/volumes.c | 21 +++++++++++++++++++++ fs/btrfs/volumes.h | 5 +++++ 3 files changed, 58 insertions(+), 1 deletion(-)
fs/btrfs/sysfs.c +32 -1
@@ -1309,6 +1309,7 @@ static const char *btrfs_read_policy_name[] = {"pid" , #ifdef CONFIG_BTRFS_EXPERIMENTAL"round-robin" , + "devid", #endif };
@@ -1356,8 +1357,11 @@ static ssize_t btrfs_read_policy_show(struct kobject *kobj,if (i ==BTRFS_READ_POLICY_RR ) ret +=sysfs_emit_at (buf, ret,":%d" ,READ_ONCE (fs_devices->rr_min_contiguous_read)); -#endif +if (i ==BTRFS_READ_POLICY_DEVID ) + ret +=sysfs_emit_at (buf, ret,":%llu" , +READ_ONCE (fs_devices->read_devid)); +#endif if (i == policy) ret +=sysfs_emit_at (buf, ret,"]" ); }
@@ -1414,6 +1418,33 @@ static ssize_t btrfs_read_policy_store(struct kobject *kobj,return len; } + +if (index ==BTRFS_READ_POLICY_DEVID ) { + +if (value !=-1 ) { +BTRFS_DEV_LOOKUP_ARGS (args); + +/* Validate input devid */ + args.devid = value; +if (btrfs_find_device (fs_devices, &args) ==NULL ) +return -EINVAL ; + }else { +/* Set default devid to the devid of the latest device */ + value = fs_devices->latest_dev->devid; + } + +if (index !=READ_ONCE (fs_devices->read_policy) || + (value !=READ_ONCE (fs_devices->read_devid))) { +WRITE_ONCE (fs_devices->read_policy, index); +WRITE_ONCE (fs_devices->read_devid, value); + +btrfs_info (fs_devices->fs_info,"read policy set to '%s:%llu'" , + btrfs_read_policy_name[index], value); + + } + +return len; + }#endif if (index !=READ_ONCE (fs_devices->read_policy)) {WRITE_ONCE (fs_devices->read_policy, index);
fs/btrfs/volumes.c +21 -0
@@ -1336,6 +1336,7 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices, fs_devices->read_policy =BTRFS_READ_POLICY_PID ;#ifdef CONFIG_BTRFS_EXPERIMENTAL fs_devices->rr_min_contiguous_read =BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ ; + fs_devices->read_devid = latest_dev->devid;#endif return 0 ;
@@ -5969,6 +5970,23 @@ unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, } #ifdef CONFIG_BTRFS_EXPERIMENTAL +static int btrfs_read_preferred(struct btrfs_chunk_map *map, int first, + int num_stripe) +{ + int last = first + num_stripe; + int stripe_index; + + for (stripe_index = first; stripe_index < last; stripe_index++) { + struct btrfs_device *device = map->stripes[stripe_index].dev; + + if (device->devid == READ_ONCE(device->fs_devices->read_devid)) + return stripe_index; + } + + /* If no read-preferred device, use first stripe */ + return first; +} + struct stripe_mirror { u64 devid; int num;
@@ -6065,6 +6083,9 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,case BTRFS_READ_POLICY_RR : preferred_mirror =btrfs_read_rr (map, first, num_stripes);break ; +case BTRFS_READ_POLICY_DEVID : + preferred_mirror =btrfs_read_preferred (map, first, num_stripes); +break ;#endif }
fs/btrfs/volumes.h +5 -0
@@ -309,6 +309,8 @@ enum btrfs_read_policy {#ifdef CONFIG_BTRFS_EXPERIMENTAL /* Balancing raid1 reads across all striped devices (round-robin) */ BTRFS_READ_POLICY_RR , +/* Read from the specific device */ +BTRFS_READ_POLICY_DEVID , #endif BTRFS_NR_READ_POLICY, };
@@ -446,6 +448,9 @@ struct btrfs_fs_devices {/* Min contiguous reads before switching to next device. */ int rr_min_contiguous_read; +/* Device to be used for reading in case of RAID1. */ +u64 read_devid; +/* Checksum mode - offload it or do it synchronously. */ enum btrfs_offload_csum_mode offload_csum_mode;#endif
Signature · 1 line
2.47.0
PATCH v5 8/10 btrfs: expose experimental mode in module information
In reply to Anand JainCommit c9c49e8f157e ("btrfs: split out CONFIG_BTRFS_EXPERIMENTAL from CONFIG_BTRFS_DEBUG") introduces a way to enable or disable experimental features, print its status during module load, like so:
Btrfs loaded, experimental=on, debug=on, assert=on, zoned=yes, fsverity=yes
--- fs/btrfs/super.c | 3 +++ 1 file changed, 3 insertions(+)
fs/btrfs/super.c +3 -0
@@ -2457,6 +2457,9 @@ static __cold void btrfs_interface_exit(void)static int __init btrfs_print_mod_info (void ) { static const char options[] = "" +#ifdef CONFIG_BTRFS_EXPERIMENTAL + ", experimental=on" +#endif #ifdef CONFIG_BTRFS_DEBUG", debug=on" #endif
Signature · 1 line
2.47.0
PATCH v5 9/10 btrfs: enable RAID1 balancing configuration via modprobe parameter
In reply to Anand JainThis update allows configuring the raid1-balancing methods using a modprobe parameter when experimental mode CONFIG_BTRFS_EXPERIMENTAL is enabled.
Examples:
- Set the RAID1 balancing method to round-robin with a custom min_contiguous_read of 4k:
$ modprobe btrfs raid1-balancing=round-robin:4096
- Set the round-robin balancing method with the default min_contiguous_read:
$ modprobe btrfs raid1-balancing=round-robin
- Set the devid balancing method, defaulting to the latest device:
$ modprobe btrfs raid1-balancing=devid
--- fs/btrfs/super.c | 5 +++++ fs/btrfs/sysfs.c | 30 +++++++++++++++++++++++++++++- fs/btrfs/sysfs.h | 5 +++++ fs/btrfs/volumes.c | 14 +++++++++++++- 4 files changed, 52 insertions(+), 2 deletions(-)
fs/btrfs/super.c +5 -0
@@ -2538,6 +2538,11 @@ static const struct init_sequence mod_init_seq[] = { }, { .init_func = extent_map_init, .exit_func = extent_map_exit, +#ifdef CONFIG_BTRFS_EXPERIMENTAL + }, { + .init_func = btrfs_raid1_balancing_init, + .exit_func = NULL, +#endif }, { .init_func = ordered_data_init, .exit_func = ordered_data_exit,
fs/btrfs/sysfs.c +29 -1
@@ -1313,7 +1313,21 @@ static const char *btrfs_read_policy_name[] = {#endif }; -static int btrfs_read_policy_to_enum (const char *str,s64 *value) +#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Global module configuration parameters */ +static char *raid1_balancing; +char *btrfs_get_raid1_balancing (void ) +{ +return raid1_balancing; +} + +/* Set perm 0, disable sys/module/btrfs/parameter/raid1_balancing interface */ +module_param (raid1_balancing, charp,0 ); +MODULE_PARM_DESC (raid1_balancing, +"Global read policy; pid (default), round-robin[:min_contiguous_read], devid[[:devid]|[:latest-gen]|[:oldest-gen]]" ); +#endif + +int btrfs_read_policy_to_enum (const char *str,s64 *value) {char param[32 ] = {'\0' };char *__maybe_unused value_str;
@@ -1336,6 +1350,20 @@ static int btrfs_read_policy_to_enum(const char *str, s64 *value)return sysfs_match_string (btrfs_read_policy_name, param); } +#ifdef CONFIG_BTRFS_EXPERIMENTAL +int __init btrfs_raid1_balancing_init (void ) +{ +s64 value; + +if (btrfs_read_policy_to_enum (raid1_balancing, &value) == -EINVAL ) { +btrfs_err (NULL ,"Invalid raid1_balancing %s" , raid1_balancing); +return -EINVAL ; + } + +return 0 ; +} +#endif +static ssize_t btrfs_read_policy_show (struct kobject *kobj,struct kobj_attribute *a,char *buf) {
fs/btrfs/sysfs.h +5 -0
@@ -47,5 +47,10 @@ void btrfs_sysfs_del_qgroups(struct btrfs_fs_info *fs_info);int btrfs_sysfs_add_qgroups (struct btrfs_fs_info *fs_info);void btrfs_sysfs_del_one_qgroup (struct btrfs_fs_info *fs_info,struct btrfs_qgroup *qgroup); +int btrfs_read_policy_to_enum (const char *str,s64 *value); +#ifdef CONFIG_BTRFS_EXPERIMENTAL +int __init btrfs_raid1_balancing_init (void ); +char *btrfs_get_raid1_balancing (void ); +#endif #endif
fs/btrfs/volumes.c +13 -1
@@ -1299,6 +1299,7 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices,struct btrfs_device *device;struct btrfs_device *latest_dev =NULL ;struct btrfs_device *tmp_device; +s64 __maybe_unused value =0 ;int ret =0 ;/* Initialize the in-memory record of filesystem read count */
@@ -1333,10 +1334,21 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices, fs_devices->latest_dev = latest_dev; fs_devices->total_rw_bytes =0 ; fs_devices->chunk_alloc_policy =BTRFS_CHUNK_ALLOC_REGULAR ; - fs_devices->read_policy =BTRFS_READ_POLICY_PID ;#ifdef CONFIG_BTRFS_EXPERIMENTAL fs_devices->rr_min_contiguous_read =BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ ; fs_devices->read_devid = latest_dev->devid; + fs_devices->read_policy = +btrfs_read_policy_to_enum (btrfs_get_raid1_balancing (), &value); +if (fs_devices->read_policy ==BTRFS_READ_POLICY_RR ) + fs_devices->fs_stats = true; +if (value) { +if (fs_devices->read_policy ==BTRFS_READ_POLICY_RR ) + fs_devices->rr_min_contiguous_read = value; +if (fs_devices->read_policy ==BTRFS_READ_POLICY_DEVID ) + fs_devices->read_devid = value; + } +#else + fs_devices->read_policy =BTRFS_READ_POLICY_PID ;#endif return 0 ;
Signature · 1 line
2.47.0
Re: [PATCH v5 09/10] btrfs: enable RAID1 balancing configuration via modprobe parameter
Hi Anand!
Am Mi., 1. Jan. 2025 um 19:10 Uhr schrieb Anand Jain <[email protected]>:
Quoted 54 lines
This update allows configuring the
raid1-balancingmethods using a modprobe parameter when experimental modeCONFIG_BTRFS_EXPERIMENTALis enabled.Examples:
- Set the RAID1 balancing method to round-robin with a custom
min_contiguous_readof 4k:$ modprobe btrfs raid1-balancing=round-robin:4096- Set the round-robin balancing method with the default
min_contiguous_read:$ modprobe btrfs raid1-balancing=round-robin- Set the
devidbalancing method, defaulting to the latest device:$ modprobe btrfs raid1-balancing=devidSigned-off-by Anand Jain <[email protected]>--- fs/btrfs/super.c | 5 +++++ fs/btrfs/sysfs.c | 30 +++++++++++++++++++++++++++++- fs/btrfs/sysfs.h | 5 +++++ fs/btrfs/volumes.c | 14 +++++++++++++- 4 files changed, 52 insertions(+), 2 deletions(-)fs/btrfs/super.c +5
-0@@ -2538,6 +2538,11 @@ static const struct init_sequence mod_init_seq[] = { }, { .init_func = extent_map_init, .exit_func = extent_map_exit, +#ifdef CONFIG_BTRFS_EXPERIMENTAL + }, { + .init_func = btrfs_raid1_balancing_init, + .exit_func = NULL, +#endif }, { .init_func = ordered_data_init, .exit_func = ordered_data_exit,fs/btrfs/sysfs.c +2
-1@@ -1313,7 +1313,21 @@ static const char *btrfs_read_policy_name[] = {#endif }; -static int btrfs_read_policy_to_enum (const char *str,s64 *value) +#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Global module configuration parameters */
fs/btrfs/sysfs.c static const char *btrfs_read_policy_name[] = { · line 1327
Quoted 10 lines
fs/btrfs/sysfs.c +10
-0+static char *raid1_balancing; +char *btrfs_get_raid1_balancing (void ) +{ +return raid1_balancing; +} + +/* Set perm 0, disable sys/module/btrfs/parameter/raid1_balancing interface */ +module_param (raid1_balancing, charp,0 ); +MODULE_PARM_DESC (raid1_balancing, +"Global read policy; pid (default), round-robin[:min_contiguous_read], devid[[:devid]|[:latest-gen]|[:oldest-gen]]" );
I don't see where "[:latest-gen]|[:oldest-gen]" would be used in the code.
Quoted 78 lines
fs/btrfs/sysfs.c +17
-0+#endif + +int btrfs_read_policy_to_enum (const char *str,s64 *value) {char param[32 ] = {'\0' };char *__maybe_unused value_str;@@ -1336,6 +1350,20 @@ static int btrfs_read_policy_to_enum(const char *str, s64 *value)return sysfs_match_string (btrfs_read_policy_name, param); } +#ifdef CONFIG_BTRFS_EXPERIMENTAL +int __init btrfs_raid1_balancing_init (void ) +{ +s64 value; + +if (btrfs_read_policy_to_enum (raid1_balancing, &value) == -EINVAL ) { +btrfs_err (NULL ,"Invalid raid1_balancing %s" , raid1_balancing); +return -EINVAL ; + } + +return 0 ; +} +#endif +static ssize_t btrfs_read_policy_show (struct kobject *kobj,struct kobj_attribute *a,char *buf) {fs/btrfs/sysfs.h +5
-0@@ -47,5 +47,10 @@ void btrfs_sysfs_del_qgroups(struct btrfs_fs_info *fs_info);int btrfs_sysfs_add_qgroups (struct btrfs_fs_info *fs_info);void btrfs_sysfs_del_one_qgroup (struct btrfs_fs_info *fs_info,struct btrfs_qgroup *qgroup); +int btrfs_read_policy_to_enum (const char *str,s64 *value); +#ifdef CONFIG_BTRFS_EXPERIMENTAL +int __init btrfs_raid1_balancing_init (void ); +char *btrfs_get_raid1_balancing (void ); +#endif #endif fs/btrfs/volumes.c +13
-2@@ -1299,6 +1299,7 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices,struct btrfs_device *device;struct btrfs_device *latest_dev =NULL ;struct btrfs_device *tmp_device; +s64 __maybe_unused value =0 ;int ret =0 ;/* Initialize the in-memory record of filesystem read count */ @@ -1333,10 +1334,21 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices, fs_devices->latest_dev = latest_dev; fs_devices->total_rw_bytes =0 ; fs_devices->chunk_alloc_policy =BTRFS_CHUNK_ALLOC_REGULAR ; - fs_devices->read_policy =BTRFS_READ_POLICY_PID ;#ifdef CONFIG_BTRFS_EXPERIMENTAL fs_devices->rr_min_contiguous_read =BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ ; fs_devices->read_devid = latest_dev->devid; + fs_devices->read_policy = +btrfs_read_policy_to_enum (btrfs_get_raid1_balancing (), &value); +if (fs_devices->read_policy ==BTRFS_READ_POLICY_RR ) + fs_devices->fs_stats = true; +if (value) { +if (fs_devices->read_policy ==BTRFS_READ_POLICY_RR ) + fs_devices->rr_min_contiguous_read = value; +if (fs_devices->read_policy ==BTRFS_READ_POLICY_DEVID ) + fs_devices->read_devid = value; + } +#else + fs_devices->read_policy =BTRFS_READ_POLICY_PID ;#endif return 0 ; --
Quoted 1 line
2.47.0
Thanks
Kai
Re: [PATCH v5 09/10] btrfs: enable RAID1 balancing configuration via modprobe parameter
Hello!
Am Mi., 9. Apr. 2025 um 18:59 Uhr schrieb Kai Krakow <[email protected]>:
Quoted 55 lines
Hi Anand!
Am Mi., 1. Jan. 2025 um 19:10 Uhr schrieb Anand Jain <[email protected]>:
Quoted 51 lines
This update allows configuring the
raid1-balancingmethods using a modprobe parameter when experimental modeCONFIG_BTRFS_EXPERIMENTALis enabled.Examples:
- Set the RAID1 balancing method to round-robin with a custom
min_contiguous_readof 4k:$ modprobe btrfs raid1-balancing=round-robin:4096- Set the round-robin balancing method with the default
min_contiguous_read:$ modprobe btrfs raid1-balancing=round-robin- Set the
devidbalancing method, defaulting to the latest device:$ modprobe btrfs raid1-balancing=devidSigned-off-by Anand Jain <[email protected]>--- fs/btrfs/super.c | 5 +++++ fs/btrfs/sysfs.c | 30 +++++++++++++++++++++++++++++- fs/btrfs/sysfs.h | 5 +++++ fs/btrfs/volumes.c | 14 +++++++++++++- 4 files changed, 52 insertions(+), 2 deletions(-)fs/btrfs/super.c +5
-0@@ -2538,6 +2538,11 @@ static const struct init_sequence mod_init_seq[] = { }, { .init_func = extent_map_init, .exit_func = extent_map_exit, +#ifdef CONFIG_BTRFS_EXPERIMENTAL + }, { + .init_func = btrfs_raid1_balancing_init, + .exit_func = NULL, +#endif }, { .init_func = ordered_data_init, .exit_func = ordered_data_exit,fs/btrfs/sysfs.c +0
-0@@ -1313,7 +1313,21 @@ static const char *btrfs_read_policy_name[] = {#endif };
Quoted 15 lines
fs/btrfs/sysfs.c static const char *btrfs_read_policy_name[] = { · line 1327
Quoted 13 lines
fs/btrfs/sysfs.c +12
-1-static int btrfs_read_policy_to_enum (const char *str,s64 *value) +#ifdef CONFIG_BTRFS_EXPERIMENTAL +/* Global module configuration parameters */ +static char *raid1_balancing; +char *btrfs_get_raid1_balancing (void ) +{ +return raid1_balancing; +} + +/* Set perm 0, disable sys/module/btrfs/parameter/raid1_balancing interface */ +module_param (raid1_balancing, charp,0 ); +MODULE_PARM_DESC (raid1_balancing, +"Global read policy; pid (default), round-robin[:min_contiguous_read], devid[[:devid]|[:latest-gen]|[:oldest-gen]]" );I don't see where "[:latest-gen]|[:oldest-gen]" would be used in the code.
I just realized this has been fixed in the final merged patchset. Sorry for the noise.
Quoted 82 lines
Quoted 78 lines
fs/btrfs/sysfs.c +17
-0+#endif + +int btrfs_read_policy_to_enum (const char *str,s64 *value) {char param[32 ] = {'\0' };char *__maybe_unused value_str;@@ -1336,6 +1350,20 @@ static int btrfs_read_policy_to_enum(const char *str, s64 *value)return sysfs_match_string (btrfs_read_policy_name, param); } +#ifdef CONFIG_BTRFS_EXPERIMENTAL +int __init btrfs_raid1_balancing_init (void ) +{ +s64 value; + +if (btrfs_read_policy_to_enum (raid1_balancing, &value) == -EINVAL ) { +btrfs_err (NULL ,"Invalid raid1_balancing %s" , raid1_balancing); +return -EINVAL ; + } + +return 0 ; +} +#endif +static ssize_t btrfs_read_policy_show (struct kobject *kobj,struct kobj_attribute *a,char *buf) {fs/btrfs/sysfs.h +5
-0@@ -47,5 +47,10 @@ void btrfs_sysfs_del_qgroups(struct btrfs_fs_info *fs_info);int btrfs_sysfs_add_qgroups (struct btrfs_fs_info *fs_info);void btrfs_sysfs_del_one_qgroup (struct btrfs_fs_info *fs_info,struct btrfs_qgroup *qgroup); +int btrfs_read_policy_to_enum (const char *str,s64 *value); +#ifdef CONFIG_BTRFS_EXPERIMENTAL +int __init btrfs_raid1_balancing_init (void ); +char *btrfs_get_raid1_balancing (void ); +#endif #endif fs/btrfs/volumes.c +13
-2@@ -1299,6 +1299,7 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices,struct btrfs_device *device;struct btrfs_device *latest_dev =NULL ;struct btrfs_device *tmp_device; +s64 __maybe_unused value =0 ;int ret =0 ;/* Initialize the in-memory record of filesystem read count */ @@ -1333,10 +1334,21 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices, fs_devices->latest_dev = latest_dev; fs_devices->total_rw_bytes =0 ; fs_devices->chunk_alloc_policy =BTRFS_CHUNK_ALLOC_REGULAR ; - fs_devices->read_policy =BTRFS_READ_POLICY_PID ;#ifdef CONFIG_BTRFS_EXPERIMENTAL fs_devices->rr_min_contiguous_read =BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ ; fs_devices->read_devid = latest_dev->devid; + fs_devices->read_policy = +btrfs_read_policy_to_enum (btrfs_get_raid1_balancing (), &value); +if (fs_devices->read_policy ==BTRFS_READ_POLICY_RR ) + fs_devices->fs_stats = true; +if (value) { +if (fs_devices->read_policy ==BTRFS_READ_POLICY_RR ) + fs_devices->rr_min_contiguous_read = value; +if (fs_devices->read_policy ==BTRFS_READ_POLICY_DEVID ) + fs_devices->read_devid = value; + } +#else + fs_devices->read_policy =BTRFS_READ_POLICY_PID ;#endif return 0 ; --Quoted 1 line
2.47.0
Thanks
Kai
PATCH v5 10/10 btrfs: modload to print RAID1 balancing status
In reply to Anand JainModified the Btrfs loading message to include the RAID1 balancing status if the experimental feature is enabled.
--- fs/btrfs/super.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
fs/btrfs/super.c +10 -0
@@ -2480,7 +2480,17 @@ static int __init btrfs_print_mod_info(void) ", fsverity=no"#endif ; + +#ifdef CONFIG_BTRFS_EXPERIMENTAL +if (btrfs_get_raid1_balancing () ==NULL ) +pr_info ("Btrfs loaded%s\n" , options); +else +pr_info ("Btrfs loaded%s, raid1_balancing=%s\n" , + options,btrfs_get_raid1_balancing ()); +#else pr_info ("Btrfs loaded%s\n" , options); +#endif +return 0 ; }
Signature · 1 line
2.47.0
Quoted 8 lines · Anand Jain
v5:
Fixes based on review comments:
. Rewritebtrfs_read_policy_to_enum()usingsysfs_match_string()and `strncpy()`.. Rewrite the round-robin method based on read counts.
. Fix the smatch indentation warning.
- Change the default minimum contiguous device read size for round-robin from 256K to 192K, as the latter performs slightly better.
This depends on the device and load and the read pattern so any number is fine, I'd rather stick to something that looks sensible if the difference is slight. Changing that later based on extensive benchmarks.
Quoted 14 lines
- Introduce a framework to track filesystem read counts. (New patch)
- Reran defrag performance numbers$ xfs_io -f -d -c 'pwrite -S 0xab 0 1000000000' /btrfs/P6B $ time -p btrfs filesystem defrag -r -f -c /btrfs| | Time | Read I/O Count | gain | | | Real | devid1 | devid2 | w-PID | |---------|-------|--------|--------|-------| | pid | 11.14s| 3808 | 0 | - | | rotation| | | | | | 196608| 6.54s| 2532 | 1276 | 41.29%| | 262144| 8.42s| 1907 | 1901 | 24.41%| | devid:1 | 10.95s| 3807 | 0 | 1.70% |
Quoted 1 line
v4:
As we're at rc5 we need to get this series to for-next soon, before rc6 is released this Sunday. I'll do one review pass and then move the patches to for-next. I'm not convinced we should do the module parameter but as this is to ease testing let's do it.
Quoted 10 lines · David Sterba
Quoted 8 lines · Anand Jain
v5:
Fixes based on review comments:
. Rewritebtrfs_read_policy_to_enum()usingsysfs_match_string()and `strncpy()`.. Rewrite the round-robin method based on read counts.
. Fix the smatch indentation warning.
- Change the default minimum contiguous device read size for round-robin from 256K to 192K, as the latter performs slightly better.
Quoted 4 lines
This depends on the device and load and the read pattern so any number is fine, I'd rather stick to something that looks sensible if the difference is slight. Changing that later based on extensive benchmarks.
The optimization should target the generic read-write workload, where I find 256k or more reasonable, though I am unable to demonstrate this at the moment. It's a good idea to gather more feedback. I've sent a fix with SZ_256K and for a missed bug, hoping both are folded into their patches. Thx.
Quoted 14 lines
Quoted 14 lines
- Introduce a framework to track filesystem read counts. (New patch)
- Reran defrag performance numbers$ xfs_io -f -d -c 'pwrite -S 0xab 0 1000000000' /btrfs/P6B $ time -p btrfs filesystem defrag -r -f -c /btrfs| | Time | Read I/O Count | gain | | | Real | devid1 | devid2 | w-PID | |---------|-------|--------|--------|-------| | pid | 11.14s| 3808 | 0 | - | | rotation| | | | | | 196608| 6.54s| 2532 | 1276 | 41.29%| | 262144| 8.42s| 1907 | 1901 | 24.41%| | devid:1 | 10.95s| 3807 | 0 | 1.70% |
Quoted 6 lines
Quoted 1 line
v4:
As we're at rc5 we need to get this series to for-next soon, before rc6 is released this Sunday. I'll do one review pass and then move the patches to for-next. I'm not convinced we should do the module parameter but as this is to ease testing let's do it.
Thanks, Anand
Quoted 16 lines · Anand Jain
Quoted 10 lines · David Sterba
Quoted 8 lines · Anand Jain
v5:
Fixes based on review comments:
. Rewritebtrfs_read_policy_to_enum()usingsysfs_match_string()and `strncpy()`.. Rewrite the round-robin method based on read counts.
. Fix the smatch indentation warning.
- Change the default minimum contiguous device read size for round-robin from 256K to 192K, as the latter performs slightly better.Quoted 4 lines
This depends on the device and load and the read pattern so any number is fine, I'd rather stick to something that looks sensible if the difference is slight. Changing that later based on extensive benchmarks.
Quoted 5 lines
The optimization should target the generic read-write workload, where I find 256k or more reasonable, though I am unable to demonstrate this at the moment. It's a good idea to gather more feedback. I've sent a fix with
SZ_256Kand for a missed bug, hoping both are folded into their patches. Thx.
As discussed, the min contig size will be tuned. Per current testing the balancing seems to work so we can keep it as is for now.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
- Introduce a framework to track filesystem read counts. (New patch)
- Reran defrag performance numbers$ xfs_io -f -d -c 'pwrite -S 0xab 0 1000000000' /btrfs/P6B $ time -p btrfs filesystem defrag -r -f -c /btrfs| | Time | Read I/O Count | gain | | | Real | devid1 | devid2 | w-PID | |---------|-------|--------|--------|-------| | pid | 11.14s| 3808 | 0 | - | | rotation| | | | | | 196608| 6.54s| 2532 | 1276 | 41.29%| | 262144| 8.42s| 1907 | 1901 | 24.41%| | devid:1 | 10.95s| 3807 | 0 | 1.70% |
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
v4:
As we're at rc5 we need to get this series to for-next soon, before rc6 is released this Sunday. I'll do one review pass and then move the patches to for-next. I'm not convinced we should do the module parameter but as this is to ease testing let's do it.
The testing delayed it but now the series is in for-next, with some cleanups and fixups I did. The remaining thing is to accept suffixed sizes as sysfs (ie. use memparse), I did not implement it as the feature is usable without that and we need to get it merged now.
Quoted 47 lines · David Sterba
Quoted 16 lines · Anand Jain
Quoted 10 lines · David Sterba
Quoted 8 lines · Anand Jain
v5:
Fixes based on review comments:
. Rewritebtrfs_read_policy_to_enum()usingsysfs_match_string()
andstrncpy().
. Rewrite the round-robin method based on read counts.
. Fix the smatch indentation warning.
- Change the default minimum contiguous device read size for round-robin from 256K to 192K, as the latter performs slightly better.Quoted 4 lines
This depends on the device and load and the read pattern so any number is fine, I'd rather stick to something that looks sensible if the difference is slight. Changing that later based on extensive benchmarks.
Quoted 5 lines
The optimization should target the generic read-write workload, where I find 256k or more reasonable, though I am unable to demonstrate this at the moment. It's a good idea to gather more feedback. I've sent a fix with
SZ_256Kand for a missed bug, hoping both are folded into their patches. Thx.As discussed, the min contig size will be tuned. Per current testing the balancing seems to work so we can keep it as is for now.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
- Introduce a framework to track filesystem read counts. (New patch)
- Reran defrag performance numbers$ xfs_io -f -d -c 'pwrite -S 0xab 0 1000000000' /btrfs/P6B $ time -p btrfs filesystem defrag -r -f -c /btrfs| | Time | Read I/O Count | gain | | | Real | devid1 | devid2 | w-PID | |---------|-------|--------|--------|-------| | pid | 11.14s| 3808 | 0 | - | | rotation| | | | | | 196608| 6.54s| 2532 | 1276 | 41.29%| | 262144| 8.42s| 1907 | 1901 | 24.41%| | devid:1 | 10.95s| 3807 | 0 | 1.70% |Quoted 6 lines
Quoted 6 lines
Quoted 1 line
v4:
As we're at rc5 we need to get this series to for-next soon, before rc6 is released this Sunday. I'll do one review pass and then move the patches to for-next. I'm not convinced we should do the module parameter but as this is to ease testing let's do it.
Quoted 4 lines
The testing delayed it but now the series is in for-next, with some cleanups and fixups I did. The remaining thing is to accept suffixed sizes as sysfs (ie. use memparse), I did not implement it as the feature is usable without that and we need to get it merged now.
The for-next patches match misc-next, reviewed, fixed, and ready to go.
Thanks!
Anand
Hi Anand!
Am Mi., 1. Jan. 2025 um 19:10 Uhr schrieb Anand Jain <[email protected]>:
Quoted 206 lines
v5:
Fixes based on review comments:
. Rewritebtrfs_read_policy_to_enum()usingsysfs_match_string()and `strncpy()`. . Rewrite the round-robin method based on read counts. . Fix the smatch indentation warning. - Change the default minimum contiguous device read size for round-robin from 256K to 192K, as the latter performs slightly better. - Introduce a framework to track filesystem read counts. (New patch) - Reran defrag performance numbers $ xfs_io -f -d -c 'pwrite -S 0xab 0 1000000000' /btrfs/P6B $ time -p btrfs filesystem defrag -r -f -c /btrfs| | Time | Read I/O Count | gain | | | Real | devid1 | devid2 | w-PID | |---------|-------|--------|--------|-------| | pid | 11.14s| 3808 | 0 | - | | rotation| | | | | | 196608| 6.54s| 2532 | 1276 | 41.29%| | 262144| 8.42s| 1907 | 1901 | 24.41%| | devid:1 | 10.95s| 3807 | 0 | 1.70% |v4:
Fixes based on review comments:
3/10: Use == 0 to check strlen(str); drop dynamic alloc for %param.
4/10: Add__maybe_unusedfor %value_str inbtrfs_read_policy_to_enum().
Return int instead of enumbtrfs_read_policy.
5/10: Fix change-log (: is part of optional [ ]).
Wrap btrfs_read_policy_name[] with ifdef for new methods.
UseIS_ALIGNED()for sector-size alignment check.
Roundup misaligned %value.
Use named constants:BTRFS_DEFAULT_RR_MIN_CONTIGUOUS_READ,BTRFS_RAID1_MAX_MIRRORS. Mark %s1 and %s2 inbtrfs_cmp_devid()as const.
Add comments tobtrfs_read_rr();
Use loop-local %i. Add space around /.
Use >> for sector-size division.
Prefix %min_contiguous_read with rr.
7/10: Move experimental to the top of the feature list.
Use experiment=on. Skip printing when off.v3:
1. Removed the latency-based RAID1 balancing patch. (Per David's review)
2. Renamed "rotation" to "round-robin" and set the per-setmin_contiguous_readto 256k. (Per David's review)
3. Added raid1-balancing module configuration for fstests testing. raid1-balancing can now be configured through both module load parameters and sysfs.The logic of individual methods remains unchanged, and performance metrics are consistent with v2.
-----
v2:
1. Move new features toCONFIG_BTRFS_EXPERIMENTALinstead ofCONFIG_BTRFS_DEBUG.
2. Correct the typo from %est_wait to %best_wait.
3. Initialize %best_wait toU64_MAXand remove the check for 0.
4. Implement rotation with a minimum contiguous read threshold before switching to the next stripe. Configure this, using:echo rotation:[min_contiguous_read] > /sys/fs/btrfs/<uuid>/read_policy
The default value is the sector size, and the
min_contiguous_readvalue must be a multiple of the sector size.5. Tested FIO random read/write and defrag compression workloads with
min_contiguous_readset to sector size, 192k, and 256k.RAID1 balancing method rotation is better for multi-process workloads such as fio and also single-process workload such as defragmentation. $ fio --filename=/btrfs/foo --size=5Gi --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 \ --time_based --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 20.3MiB/s | 20.5MiB/s | 313895 | 313895 | | rotation| | | | | | 4096| 20.4MiB/s | 20.5MiB/s | 313895 | 313895 | | 196608| 20.2MiB/s | 20.2MiB/s | 310152 | 310175 | | 262144| 20.3MiB/s | 20.4MiB/s | 312180 | 312191 | | latency| 18.4MiB/s | 18.4MiB/s | 272980 | 291683 | | devid:1 | 14.8MiB/s | 14.9MiB/s | 456376 | 0 |rotation RAID1 balancing technique performs more than 2x better for single-process defrag.
$ time -p btrfs filesystem defrag -r -f -c /btrfs| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 18.00s| 3800 | 0 | | rotation| | | | | 4096| 8.95s| 1900 | 1901 | | 196608| 8.50s| 1881 | 1919 | | 262144| 8.80s| 1881 | 1919 | | latency | 17.18s| 3800 | 0 | | devid:2 | 17.48s| 0 | 3800 |Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method is preferable as default. More workload testing is needed while the code is EXPERIMENTAL.
While Latency is better during the failing/unstable block layer transport. As of now these two techniques, are needed to be further independently tested with different worloads, and in the long term we should be merge these technique to a unified heuristic.Rotation keeps all devices active, and for now, the Rotation RAID1 balancing method should be the default. More workload testing is needed while the code is EXPERIMENTAL.
Latency is smarter with unstable block layer transport.
Both techniques need independent testing across workloads, with the goal of eventually merging them into a unified approach? for the long term.
Devid is a hands-on approach, provides manual or user-space script control.
These RAID1 balancing methods are tunable via the sysfs knob. The mount -o option and btrfs properties are under consideration.
Thx.
--------- original v1 ------------
The RAID1-balancing methods helps distribute read I/O across devices, and this patch introduces three balancing methods: rotation, latency, and devid. These methods are enabled under the
CONFIG_BTRFS_DEBUGconfig option and are on top of the previously added/sys/fs/btrfs/<UUID>/read_policyinterface to configure the desired RAID1 read balancing method.I've tested these patches using fio and filesystem defragmentation workloads on a two-device RAID1 setup (with both data and metadata mirrored across identical devices). I tracked device read counts by extracting stats from
/sys/devices/<..>/statfor each device. Below is a summary of the results, with each result the average of three iterations.A typical generic random rw workload:
$ fio --filename=/btrfs/foo --size=10Gi --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based \ --group_reporting --name=iops-test-job --eta-newline=1| | | | Read I/O count | | | Read | Write | devid1 | devid2 | |---------|------------|------------|--------|--------| | pid | 29.4MiB/s | 29.5MiB/s | 456548 | 447975 | | rotation| 29.3MiB/s | 29.3MiB/s | 450105 | 450055 | | latency | 21.9MiB/s | 21.9MiB/s | 672387 | 0 | | devid:1 | 22.0MiB/s | 22.0MiB/s | 674788 | 0 |Defragmentation with compression workload:
$ xfs_io -f -d -c 'pwrite -S 0xab 0 1G' /btrfs/foo $ sync $ echo 3 > /proc/sys/vm/drop_caches $ btrfs filesystem defrag -f -c /btrfs/foo| | Time | Read I/O Count | | | Real | devid1 | devid2 | |---------|-------|--------|--------| | pid | 21.61s| 3810 | 0 | | rotation| 11.55s| 1905 | 1905 | | latency | 20.99s| 0 | 3810 | | devid:2 | 21.41s| 0 | 3810 |. The PID-based balancing method works well for the generic random rw fio
workload.
. The rotation method is ideal when you want to keep both devices active,
and it boosts performance in sequential defragmentation scenarios.
. The latency-based method work well when we have mixed device types or
when one device experiences intermittent I/O failures the latency increases and it automatically picks the other device for further Read IOs.
. The devid method is a more hands-on approach, useful for diagnosing and
testing RAID1 mirror synchronizations.Anand Jain (10):
btrfs: initializefs_devices->fs_infoearlier
btrfs: simplify output formatting inbtrfs_read_policy_show
btrfs: addbtrfs_read_policy_to_enumhelper and refactor read policy
store
btrfs: handle value associated with raid1 balancing parameter
btrfs: add read count tracking for filesystem stats
btrfs: introduce RAID1 round-robin read balancing
btrfs: add RAID1 preferred read device
btrfs: expose experimental mode in module information
btrfs: enable RAID1 balancing configuration via modprobe parameter
btrfs: modload to print RAID1 balancing statusfs/btrfs/bio.c | 8 +++ fs/btrfs/disk-io.c | 4 ++ fs/btrfs/super.c | 18 +++++ fs/btrfs/sysfs.c | 168 ++++++++++++++++++++++++++++++++++++++++----- fs/btrfs/sysfs.h | 5 ++ fs/btrfs/volumes.c | 115 ++++++++++++++++++++++++++++++- fs/btrfs/volumes.h | 22 +++++- 7 files changed, 317 insertions(+), 23 deletions(-)
Quoted 2 lines
--
2.47.0
I've backported your patches to my LTS kernel to experiment with it a bit. I've also included the latency-based policy for that reason. Here's what I currently got:
https://github.com/kakra/linux/pull/36
I've added an idea to use a hybrid policy which combines round-robin and latency into one combined policy. But it seems like over time, it will just acts like round-robin. This is probably because the average latency is calculated over the full history of requests. I think using an EMA (exponential moving average) with an alpha of 1/8 or 1/16 could work better. But this would require to sample each bio individually and add fields to the device structs. Something like this:
s64 current_ema = atomic64_read(&device->avg_latency_ema);
s64 difference = (s64)current_latency - current_ema; // important:
signed difference!
s64 new_ema = current_ema + (difference >> N); // N = 3 or 4 (alpha 1/8 or 1/16) atomic64_cmpxchg(&device->avg_latency_ema, current_ema, new_ema);
What do you think?
I'm currently not posting my patch series here because
(a) it's based/backported on LTS
(b) I'm not yet very familiar with doing that on the mailing list
(c) the code is not ready and contains some duplication
I hope that's okay.
So far, both the latency and round-robin policies work well in my setup. The latency policy actually filters out a low-end desktop HDD which is quite slow (according to fio) for individual short requests and long sequential reading (acts more like 5400rpm in that case) but is actually not that bad for typical random IO with larger block sizes (actually competes well with my other 7200rpm disks). Combined with bcache, this automatically optimizes the fastest disks into the cache, avoiding the slower ones when using the latency policy.
Overall, loading times in some applications with random but massive IO thus seem faster with the latency policy while sequential IO is faster with the round-robin policy.
I think the hybrid approach could fit both scenarios but it currently suffers from not adapting well to changes in IO patterns - which EMA could solve.
Thanks
Kai
Hi Kai,
Quoted 13 lines
I've backported your patches to my LTS kernel to experiment with it a bit. I've also included the latency-based policy for that reason. Here's what I currently got:
https://github.com/kakra/linux/pull/36I've added an idea to use a hybrid policy which combines round-robin and latency into one combined policy. But it seems like over time, it will just acts like round-robin. This is probably because the average latency is calculated over the full history of requests. I think using an EMA (exponential moving average) with an alpha of 1/8 or 1/16 could work better. But this would require to sample each bio individually and add fields to the device structs. Something like this:
Quoted 8 lines
s64
current_ema=atomic64_read(&device->avg_latency_ema);
s64 difference = (s64)current_latency -current_ema; // important:
signed difference!
s64new_ema=current_ema+ (difference >> N); // N = 3 or 4 (alpha 1/8 or 1/16)atomic64_cmpxchg(&device->avg_latency_ema, current_ema, new_ema);What do you think?
You can assume both devices are of the same type; otherwise, the logic gets sidetracked, and some performance numbers may vary. I couldn’t figure out how are you calculating EMA in the kernel or how well it prioritizes current I/O latency over historical values (which is a problem should be avoided), or if it gradually fades out the impact of device's stale latency numbers.
Quoted 17 lines
I'm currently not posting my patch series here because
(a) it's based/backported on LTS
(b) I'm not yet very familiar with doing that on the mailing list
(c) the code is not ready and contains some duplicationI hope that's okay.
So far, both the latency and round-robin policies work well in my setup. The latency policy actually filters out a low-end desktop HDD which is quite slow (according to fio) for individual short requests and long sequential reading (acts more like 5400rpm in that case) but is actually not that bad for typical random IO with larger block sizes (actually competes well with my other 7200rpm disks). Combined with bcache, this automatically optimizes the fastest disks into the cache, avoiding the slower ones when using the latency policy.
Quoted 7 lines
Overall, loading times in some applications with random but massive IO thus seem faster with the latency policy while sequential IO is faster with the round-robin policy.
I think the hybrid approach could fit both scenarios but it currently suffers from not adapting well to changes in IO patterns - which EMA could solve.
To try it out better, you can take the latency code from this series in v2 and add EMA to it. For now, keep both devices of the same type. Test it with a defrag workload. To make it more interesting, add some noise to the blk-layer transport of one device to increase its latency, and see if the algorithm dynamically switches to the other device. Then compare it with round-robin. That algorithm will need fine-tuning based on how adaptable it should be to intermittent block layer bottlenecks
Thanks, Anand