readmirror feature (read_policy sysfs and in-memory only approach)
v10 cover readmirror feature (read_policy sysfs and in-memory only approach)
Resend and based on misc-next (as dependency on genhd.c is gone).
v10: patch1/3 (fix btrfs_strmatch) and patch 3/3 (use scnprintf instead of snprintf)
and add rb.
v9: C coding style fixes in 1/3 and 3/3
v8:
Separate the sysfs framework and the %pid read_policy into a separate patchset here, so that the new read policies can be in its own patch set.
A latency based read_policy is being prepared to send it in a separate patchset as it depends on a few changes in the block layer as well.
__ Original email: __
v7:
Fix switch's fall through warning. Changle logs updates where necessary.
v6:
Patch 4/5 - If there is no change in device's read prefer then don't log Patch 4/5 - Add pid to the logs
Patch 5/5 - If there isn't read preferred device in the chunk don't reset read policy to default, instead just use stripe 0. As this is in the read path it avoids going through the device list to find read preferred device. So inline to this drop to check if there is read preferred device before setting read policy to device.
v5:
Worked on review comments as received in its previous version. Please refer to individual patches for the specific changes. Introduces the new read_policy 'device'.
v4:
Rename readmirror attribute to read_policy. Drop separate kobj for readmirror instead create read_policy attribute in fsid kobj. merge v2:2/3 and v2:3/3 into v4:2/2. Patch titles have changed.
v3:
v2:
Mainly fixes the fs_devices::readmirror declaration type from atomic_t to u8. (Thanks Josef).
v1:
As of now we use only %pid method to read stripped mirrored data. So application's process id determines the stripe id to be read. This type of routing typically helps in a system with many small independent applications tying to read random data. On the other hand the %pid based read IO distribution policy is inefficient if there is a single application trying to read large data as because the overall disk bandwidth would remains under utilized.
One type of readmirror policy isn't good enough and other choices are routing the IO based on device's waitqueue or manual when we have a read-preferred device or a readmirror policy based on the target storage caching. So this patch-set introduces a framework where we could add more readmirror policies.
This policy is a filesystem wide policy as of now, and though the readmirror policy at the subvolume level is a novel approach as it provides maximum flexibility in the data center, but as of now its not practical to implement such a granularity as you can't really ensure reflinked extents will be read from the stripe of its desire and so there will be more limitations and it can be assessed separately.
The approach in this patch-set is sys interface with in-memory policy. And does not add any new readmirror type in this set, which can be add once we are ok with the framework. Also the default policy remains %pid.
Previous works:
----------------------------------------------------------------------
There were few RFCs [1] before, mainly to figure out storage (or in memory only) for the readmirror policy and the interface needed.
[1]
https://www.mail-archive.com/[email protected]/msg86368.html
https://lore.kernel.org/linux-btrfs/[email protected]/
https://lore.kernel.org/linux-btrfs/[email protected]/
https://patchwork.kernel.org/cover/10859213/
Mount -o:
In the first trial it was attempted to use the mount -o option to carry the readmirror policy, this is good for debugging which can make sure even the mount thread metadata tree blocks are read from the disk desired. It was very effective in testing radi1/raid10 write-holes.
Extended attribute:
As extended attribute is associated with the inode, to implement this there is bit of extended attribute abuse or else makes it mandatory to mount the rootid 5. Its messy unless readmirror policy is applied at the subvol level which is not possible as of now.
An item type:
The proposed patch was to create an item to hold the readmirror policy, it makes sense when compared to the abusive extended attribute approach but introduces a new item and so no backward compatibility.
-----------------------------------------------------------------------
Anand Jain (3):
btrfs: add btrfs_strmatch helper
btrfs: create read policy framework
btrfs: create read policy sysfs attribute, pid
fs/btrfs/sysfs.c | 67 ++++++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/volumes.c | 15 ++++++++++- fs/btrfs/volumes.h | 14 ++++++++++ 3 files changed, 95 insertions(+), 1 deletion(-)
Signature · 1 line
2.25.1
PATCH v10 1/3 btrfs: add btrfs_strmatch helper
Add a generic helper to match the golden-string in the given-string, and ignore the leading and trailing whitespaces if any.
---
v10: return bool instead of int.
drop unnecessary local variable and ( ) with in if.
v9: use Josef suggested C coding style, using single if statement.
v5: born
fs/btrfs/sysfs.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
fs/btrfs/sysfs.c +18 -0
@@ -888,6 +888,24 @@ static ssize_t btrfs_generation_show(struct kobject *kobj, } BTRFS_ATTR(, generation, btrfs_generation_show); +/* +* Match the %golden in the %given. Ignore the leading and trailing whitespaces +* if any. +*/ +static bool btrfs_strmatch (const char *given,const char *golden) +{ +size_t len =strlen (golden); + +/* skip leading whitespace */ + given =skip_spaces (given); + +if (strncmp (given, golden, len) ==0 && + !strlen (skip_spaces (given + len))) +return true; + +return false; +} +static const struct attribute *btrfs_attrs[] = { BTRFS_ATTR_PTR(, label), BTRFS_ATTR_PTR(, nodesize),
Signature · 1 line
2.25.1
Re: [PATCH v10 1/3] btrfs: add btrfs_strmatch helper
Quoted 21 lines · Anand Jain
Add a generic helper to match the golden-string in the given-string, and ignore the leading and trailing whitespaces if any.
Signed-off-by Anand Jain <[email protected]>Suggested-by David Sterba <[email protected]>Reviewed-by Josef Bacik <[email protected]>---
v10: return bool instead of int.
drop unnecessary local variable and ( ) with in if.
v9: use Josef suggested C coding style, using single if statement.
v5: bornfs/btrfs/sysfs.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)fs/btrfs/sysfs.c +0
-0@@ -888,6 +888,24 @@ static ssize_t btrfs_generation_show(struct kobject *kobj, } BTRFS_ATTR(, generation, btrfs_generation_show);
fs/btrfs/sysfs.c static ssize_t btrfs_generation_show(struct kobject *kobj, · line 899
Quoted 10 lines
fs/btrfs/sysfs.c +9
-0+/* +* Match the %golden in the %given. Ignore the leading and trailing whitespaces +* if any. +*/ +static bool btrfs_strmatch (const char *given,const char *golden) +{ +size_t len =strlen (golden); + +/* skip leading whitespace */
Comments should start with uppercase unless it's an identifier.
fs/btrfs/sysfs.c static ssize_t btrfs_generation_show(struct kobject *kobj, · line 903
Quoted 4 lines
fs/btrfs/sysfs.c +4
-0+ given =skip_spaces (given); + +if (strncmp (given, golden, len) ==0 && + !strlen (skip_spaces (given + len)))
You had strlen() == 0 instead of !strlen in v9, the == 0 is better here.
I'll fix that locally and push to misc-next so we can proceed with the actual policies.
PATCH v10 2/3 btrfs: create read policy framework
In reply to Anand JainAs of now, we use the %pid method to read striped mirrored data, which means process id determines the stripe id to read. This type of routing typically helps in a system with many small independent processes tying to read random data. On the other hand, the %pid based read IO policy is inefficient because if there is a single process trying to read a large file, the overall disk bandwidth remains under-utilized.
So this patch introduces a read policy framework so that we could add more read policies, such as IO routing based on the device's wait-queue or manual when we have a read-preferred device or a policy based on the target storage caching.
---
v10: -
v9: -
v8: use fallthrough;
v7: Fix missing /* fall through */ in the switch
Removed Reviewed-by: Josef Bacik <[email protected]>
v6:-
v5: Title renamed from:- btrfs: add read_policy framework
Change log updated.
Unnecessary comment dropped, added more where necessary. Optimize code in the switch remove duplicate code. Define BTRFS_READ_POLICY_DEFAULT dropped.
Rename enum btrfs_read_policy_type to enum btrfs_read_policy. Rename BTRFS_READ_BY_PID to BTRFS_READ_POLICY_PID.
(As its mainly renames. Reviewed-by retained).
v4: -
v3: Declare fs_devices::readmirror as enum btrfs_readmirror_policy_type
v2: Declare fs_devices::readmirror as u8 instead of atomic_t
A small change in comment and change log wordings.
fs/btrfs/volumes.c | 15 ++++++++++++++- fs/btrfs/volumes.h | 14 ++++++++++++++ 2 files changed, 28 insertions(+), 1 deletion(-)
fs/btrfs/volumes.c +14 -1
@@ -1223,6 +1223,7 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices, fs_devices->latest_bdev = latest_dev->bdev; fs_devices->total_rw_bytes =0 ; fs_devices->chunk_alloc_policy =BTRFS_CHUNK_ALLOC_REGULAR ; + fs_devices->read_policy =BTRFS_READ_POLICY_PID ;return 0 ; }
@@ -5485,7 +5486,19 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info, else num_stripes = map->num_stripes; - preferred_mirror = first + current->pid % num_stripes; +switch (fs_info->fs_devices->read_policy) { +default : +/* +* Shouldn't happen, just warn and use pid instead of failing. +*/ +btrfs_warn_rl (fs_info, +"unknown read_policy type %u, fallback to pid" , + fs_info->fs_devices->read_policy); + fallthrough; +case BTRFS_READ_POLICY_PID : + preferred_mirror = first + current->pid % num_stripes; +break ; + }if (dev_replace_is_ongoing && fs_info->dev_replace.cont_reading_from_srcdev_mode ==
fs/btrfs/volumes.h +14 -0
@@ -211,6 +211,15 @@ enum btrfs_chunk_allocation_policy { BTRFS_CHUNK_ALLOC_REGULAR, }; +/* +* Read policies for the mirrored block groups, read picks the stripe based +* on these policies. +*/ +enum btrfs_read_policy { +BTRFS_READ_POLICY_PID , +BTRFS_NR_READ_POLICY , +}; +struct btrfs_fs_devices {u8 fsid[BTRFS_FSID_SIZE ];/* FS specific uuid */ u8 metadata_uuid[BTRFS_FSID_SIZE ];
@@ -264,6 +273,11 @@ struct btrfs_fs_devices {struct completion kobj_unregister;enum btrfs_chunk_allocation_policy chunk_alloc_policy; + +/* +* policy used to read the mirrored stripes +*/ +enum btrfs_read_policy read_policy; };#define BTRFS_BIO_INLINE_CSUM_SIZE 64
Signature · 1 line
2.25.1
Re: [PATCH v10 2/3] btrfs: create read policy framework
fs/btrfs/volumes.c static int find_live_mirror(struct btrfs_fs_info *fs_info, · line 5496
Quoted 13 lines · Anand Jain
fs/btrfs/volumes.c +8
-1@@ -5485,7 +5486,19 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,else num_stripes = map->num_stripes; - preferred_mirror = first + current->pid % num_stripes; +switch (fs_info->fs_devices->read_policy) { +default : +/* +* Shouldn't happen, just warn and use pid instead of failing. +*/ +btrfs_warn_rl (fs_info, +"unknown read_policy type %u, fallback to pid" , + fs_info->fs_devices->read_policy);
So this would flood the system log with useless messages once this happens. We should really reset it to pid or whatever default we choose. Fixed.
fs/btrfs/volumes.c static int find_live_mirror(struct btrfs_fs_info *fs_info, · line 5499
Quoted 3 lines
fs/btrfs/volumes.c +3
-0+ fallthrough; +case BTRFS_READ_POLICY_PID : + preferred_mirror = first + current->pid % num_stripes;
This is in the original code, but this should really use ( ) to make the precedence clear.
PATCH v10 3/3 btrfs: create read policy sysfs attribute, pid
In reply to Anand JainAdd
/sys/fs/btrfs/UUID/read_policy
attribute so that the read policy for the raid1, raid1c34 and raid10 can be tuned.
When this attribute is read, it shall show all available policies, with active policy being with in [ ]. The read_policy attribute can be written using one of the items listed in there.
For example:
$cat /sys/fs/btrfs/UUID/read_policy
[pid]
$echo pid > /sys/fs/btrfs/UUID/read_policy
---
v10: use scnprintf instead of snprintf
v9: fix C coding style, static const char*
v5:
Title rename: old: btrfs: sysfs, add read_policy attribute Uses the btrfs_strmatch() helper (BTRFS_READ_POLICY_NAME_MAX dropped). Use the table for the policy names.
Rename len to ret.
Use a simple logic to prefix space in btrfs_read_policy_show()
v4:-
v3: rename [by_pid] to [pid]
v2: v2: check input len before strip and kstrdup
fs/btrfs/sysfs.c | 49 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+)
fs/btrfs/sysfs.c +49 -0
@@ -906,6 +906,54 @@ static bool btrfs_strmatch(const char *given, const char *golden)return false; } +static const char *const btrfs_read_policy_name[] = {"pid" }; + +static ssize_t btrfs_read_policy_show (struct kobject *kobj, +struct kobj_attribute *a,char *buf) +{ +int i; +ssize_t ret =0 ; +struct btrfs_fs_devices *fs_devices =to_fs_devs (kobj); + +for (i =0 ; i <BTRFS_NR_READ_POLICY ; i++) { +if (fs_devices->read_policy == i) + ret +=scnprintf (buf + ret,PAGE_SIZE - ret,"%s[%s]" , + (ret ==0 ?"" :" " ), + btrfs_read_policy_name[i]); +else + ret +=scnprintf (buf + ret,PAGE_SIZE - ret,"%s%s" , + (ret ==0 ?"" :" " ), + btrfs_read_policy_name[i]); + } + + ret +=scnprintf (buf + ret,PAGE_SIZE - ret,"\n" ); + +return ret; +} + +static ssize_t btrfs_read_policy_store (struct kobject *kobj, +struct kobj_attribute *a, +const char *buf,size_t len) +{ +int i; +struct btrfs_fs_devices *fs_devices =to_fs_devs (kobj); + +for (i =0 ; i <BTRFS_NR_READ_POLICY ; i++) { +if (btrfs_strmatch (buf, btrfs_read_policy_name[i])) { +if (i != fs_devices->read_policy) { + fs_devices->read_policy = i; +btrfs_info (fs_devices->fs_info, +"read policy set to '%s'" , + btrfs_read_policy_name[i]); + } +return len; + } + } + +return -EINVAL ; +} +BTRFS_ATTR_RW(, read_policy, btrfs_read_policy_show, btrfs_read_policy_store); +static const struct attribute *btrfs_attrs[] = { BTRFS_ATTR_PTR(, label), BTRFS_ATTR_PTR(, nodesize),
@@ -916,6 +964,7 @@ static const struct attribute *btrfs_attrs[] = { BTRFS_ATTR_PTR(, checksum), BTRFS_ATTR_PTR(, exclusive_operation), BTRFS_ATTR_PTR(, generation), + BTRFS_ATTR_PTR(, read_policy), NULL, };
Signature · 1 line
2.25.1
Re: [PATCH v10 3/3] btrfs: create read policy sysfs attribute, pid
fs/btrfs/sysfs.c static bool btrfs_strmatch(const char *given, const char *golden) · line 942
Quoted 9 lines · Anand Jain
fs/btrfs/sysfs.c +9
-0+static ssize_t btrfs_read_policy_store (struct kobject *kobj, +struct kobj_attribute *a, +const char *buf,size_t len) +{ +int i; +struct btrfs_fs_devices *fs_devices =to_fs_devs (kobj); + +for (i =0 ; i <BTRFS_NR_READ_POLICY ; i++) { +if (btrfs_strmatch (buf, btrfs_read_policy_name[i])) {
Does sysfs guarantee that the buf is nul terminated string or that it contains a null at all? Because if not, the skip_space step of strmatch could run out of the buffer.
Re: [PATCH v10 3/3] btrfs: create read policy sysfs attribute, pid
Quoted 15 lines · David Sterba
fs/btrfs/sysfs.c static bool btrfs_strmatch(const char *given, const char *golden) · line 942
Quoted 9 lines · Anand Jain
fs/btrfs/sysfs.c +9
-0+static ssize_t btrfs_read_policy_store (struct kobject *kobj, +struct kobj_attribute *a, +const char *buf,size_t len) +{ +int i; +struct btrfs_fs_devices *fs_devices =to_fs_devs (kobj); + +for (i =0 ; i <BTRFS_NR_READ_POLICY ; i++) { +if (btrfs_strmatch (buf, btrfs_read_policy_name[i])) {Does sysfs guarantee that the buf is nul terminated string or that it contains a null at all? Because if not, the
skip_spacestep of strmatch could run out of the buffer.
It does
[ 173.555507] ? btrfs_read_policy_store+0x3e/0x12d [btrfs] [ 173.555541] ? kobj_attr_store+0x16/0x30 [ 173.555562] ? sysfs_kf_write+0x54/0x80 [ 173.555582] ? kernfs_fop_write+0xfa/0x290 [ 173.555611] ? vfs_write+0xee/0x2f0 [ 173.555641] ? ksys_write+0x80/0x170 [ 173.555671] ? __x64_sys_write+0x1e/0x30 [ 173.555692] ? do_syscall_64+0x4b/0x80 [ 173.555708] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
static ssize_t kernfs_fop_write(struct file *file, const char __user *user_buf,
size_t count, loff_t *ppos)
{
::
buf[len] = '\0'; /* guarantee string termination */
Thanks, Anand
Re: [PATCH v10 3/3] btrfs: create read policy sysfs attribute, pid
Quoted 32 lines · Anand Jain
Quoted 15 lines · David Sterba
fs/btrfs/sysfs.c static bool btrfs_strmatch(const char *given, const char *golden) · line 942
Quoted 9 lines · Anand Jain
fs/btrfs/sysfs.c +9
-0+static ssize_t btrfs_read_policy_store (struct kobject *kobj, +struct kobj_attribute *a, +const char *buf,size_t len) +{ +int i; +struct btrfs_fs_devices *fs_devices =to_fs_devs (kobj); + +for (i =0 ; i <BTRFS_NR_READ_POLICY ; i++) { +if (btrfs_strmatch (buf, btrfs_read_policy_name[i])) {Does sysfs guarantee that the buf is nul terminated string or that it contains a null at all? Because if not, the
skip_spacestep of strmatch could run out of the buffer.It does
[ 173.555507] ? btrfs_read_policy_store+0x3e/0x12d [btrfs] [ 173.555541] ? kobj_attr_store+0x16/0x30 [ 173.555562] ? sysfs_kf_write+0x54/0x80 [ 173.555582] ? kernfs_fop_write+0xfa/0x290 [ 173.555611] ? vfs_write+0xee/0x2f0 [ 173.555641] ? ksys_write+0x80/0x170 [ 173.555671] ? __x64_sys_write+0x1e/0x30 [ 173.555692] ? do_syscall_64+0x4b/0x80 [ 173.555708] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
Quoted 6 lines
static
ssize_tkernfs_fop_write(struct file *file, const char__user*user_buf,
size_tcount,loff_t*ppos)
{
::
buf[len] = '\0'; /* guarantee string termination */
Ok thanks.