btrfs: Refactor delayed ref parameter list
cover btrfs: Refactor delayed ref parameter list
This patchset can be fetched from github:
https://github.com/adam900710/linux/tree/refactor_delayed_ref_parameter
Which is based on David's misc-next branch, the base commit is:
commit 56d46f96de92ec69963acb7b1d9aed83d2a56a7b (david/misc-next-with-write-checks, david/misc-next)
Author: Nikolay Borisov <[email protected]> Date: Wed Mar 27 14:24:18 2019 +0200
btrfs: Switch btrfs_trim_free_extents to find_first_clear_extent_bit
Current delayed ref interface has several problems:
- Longer and longer parameter lists
bytenr
num_bytes
parent
---- So far so good
ref_root
owner
offset
---- I don't feel well now
for_reloc
^^^^ This parameter only makes sense for qgroup code, but we need
to pass the parameter a long way down.
This makes later parameter list add more and more tricky.
- Different interpretation for the same parameter
Above @owner for data ref is inode who owns this extent, while for tree ref, it's level. They are even in different size range.
For level we only need 0~8, while for ino it's BTRFS_FIRST_FREE_OBJECTID~BTRFS_LAST_FREE_OBJECTID, so it's still possible to distinguish them, but it's never a straight-forward thing to grasp.
And @offset doesn't even makes sense for tree ref.
Such parameter reuse may look clever as an hidden union, but it destroys code readability.
This patchset will change the way how we pass parameters for delayed ref.
Instead of calling delayed ref interface like:
ret = btrfs_inc_extent_ref(trans, root, bytenr, num_bytes, parent,
ref_root, owner, offset); Or ret = btrfs_inc_extent_ref(trans, root, bytenr, nodesize, parent, level, ref_root, 0);
We now call like:
btrfs_init_generic_ref(&ref, bytenr, num_bytes,
root->root_key.objectid, parent);
btrfs_init_data_ref(&ref, ref_root, owner, offset); ret = btrfs_inc_extent_ref(trans, &ref);
Or
btrfs_init_generic_ref(&ref, bytenr, num_bytes,
root->root_key.objectid, parent);
btrfs_init_tree_ref(&ref, level, ref_root);
ret = btrfs_inc_extent_ref(trans, &ref);
To determine if a ref is tree or data, instead of calling like:
if (owner < BTRFS_FIRST_FREE_OBJECTID) {
} else {
}
We do it straight-forward:
if (ref->type == BTRFS_REF_METADATA) {
} else {
}
And for new members determining some minor behavior, we don't need to add a new parameter to btrfs_add_delayed_tree|data_ref() or btrfs_inc_extent_ref(), we just assign them after generic/data/tree init, like:
btrfs_init_generic_ref(&ref, bytenr, num_bytes,
root->root_key.objectid, parent);
ref->real_root = root->root_key.objectid; ref->skip_qgroup = true; btrfs_init_data_ref(&ref, ref_root, owner, offset);
ret = btrfs_inc_extent_ref(trans, &ref);
This should improve the code readability and make later code easier to write.
Furthermore, with the help of btrfs_ref::real_root parameter, qgroup can skip quit a lot of delayed tree/data ref for reloc tree, which makes qgroup + balance as fast as quota disabled:
Test VM:
- vRAM 8G
- vCPU 8
- block dev vitrio-blk, 'unsafe' cache mode
- host block 850evo
Test workload
- Copy 4G data from /usr/ to one subvolume
- Create 16 snapshots of that subvolume, and modify 3 files in each snapshot
- Enable quota, rescan
- Time "btrfs balance start -m"
| base | w/ patchset | no qgroups |
-------------------------------------------------------------
relocated | 23765 | 23772 | 23811 | qgroup dirty | 124498 | 70 | 0 | time (sec) | 23.353 | 3.505 | 3.421 |
Changelog:
v2:
- Better documentation for btrfs_ref declaration
- Rebase to newer delayed subtree rescan patchset
- Add reviewed-by tags
- Remove unnecessary ASSERT() for NULL pointer.
v3:
- Rebase to misc-next branch as that branch has all prerequisite now.
- Update benchmark result, compare with qgroups disabled case directly.
v3.1:
- Rebase to misc-next branch.
Qu Wenruo (9):
btrfs: delayed-ref: Introduce better documented delayed ref structures
btrfs: extent-tree: Open-code process_func in __btrfs_mod_ref
btrfs: delayed-ref: Use btrfs_ref to refactor
btrfs_add_delayed_tree_ref()
btrfs: delayed-ref: Use btrfs_ref to refactor
btrfs_add_delayed_data_ref()
btrfs: ref-verify: Use btrfs_ref to refactor btrfs_ref_tree_mod()
btrfs: extent-tree: Use btrfs_ref to refactor add_pinned_bytes()
btrfs: extent-tree: Use btrfs_ref to refactor btrfs_inc_extent_ref()
btrfs: extent-tree: Use btrfs_ref to refactor btrfs_free_extent()
btrfs: qgroup: Don't scan leaf if we're modifying reloc tree
fs/btrfs/ctree.h | 10 +-- fs/btrfs/delayed-ref.c | 40 ++++++--- fs/btrfs/delayed-ref.h | 126 ++++++++++++++++++++++++-- fs/btrfs/extent-tree.c | 199 ++++++++++++++++++++--------------------- fs/btrfs/file.c | 39 ++++---- fs/btrfs/inode.c | 23 +++-- fs/btrfs/ioctl.c | 15 ++-- fs/btrfs/ref-verify.c | 53 ++++++----- fs/btrfs/ref-verify.h | 10 +-- fs/btrfs/relocation.c | 67 +++++++++----- fs/btrfs/tree-log.c | 11 ++- 11 files changed, 380 insertions(+), 213 deletions(-)
Signature · 1 line
2.21.0
PATCH 1/9 btrfs: delayed-ref: Introduce better documented delayed ref structures
Current delayed ref interface has several problems:
- Longer and longer parameter lists
bytenr
num_bytes
parent
---------- so far so good
ref_root
owner
offset
---------- I don't feel good now
- Different interpretation for the same parameter Above @owner for data ref is inode number (u64), while for tree ref, it's level (int).
They are even in different size range.
For level we only need 0~8, while for ino it's BTRFS_FIRST_FREE_OBJECTID~BTRFS_LAST_FREE_OBJECTID.
And @offset doesn't even makes sense for tree ref.
Such parameter reuse may look clever as an hidden union, but it destroys code readability.
To solve both problems, we introduce a new structure, btrfs_ref to solve them:
- Structure instead of long parameter list
This makes later expansion easier, and better documented.
- Use btrfs_ref::type to distinguish data and tree ref
- Use proper union to store data/tree ref specific structures.
- Use separate functions to fill data/tree ref data, with a common generic function to fill common bytenr/num_bytes members.
All parameters will find its place in btrfs_ref, and an extra member, @real_root, inspired by ref-verify code, is newly introduced for later qgroup code, to record which tree is triggered this extent modification.
This patch doesn't touch any code, but provides the basis for incoming refactors.
--- fs/btrfs/delayed-ref.h | 116 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+)
fs/btrfs/delayed-ref.h +116 -0
@@ -176,6 +176,90 @@ struct btrfs_delayed_ref_root {u64 qgroup_to_skip; }; +enum btrfs_ref_type { +BTRFS_REF_NOT_SET , +BTRFS_REF_DATA , +BTRFS_REF_METADATA , +BTRFS_REF_LAST , +}; + +struct btrfs_data_ref { +/* For EXTENT_DATA_REF */ + +/* Root who refers to this data extent */ +u64 ref_root; + +/* Inode who refers to this data extent */ +u64 ino; + +/* +* file_offset - extent_offset +* +* file_offset is the key.offset of the EXTENT_DATA key. +* extent_offset is btrfs_file_extent_offset() of the EXTENT_DATA data. +*/ +u64 offset; +}; + +struct btrfs_tree_ref { +/* +* Level of this tree block +* +* Shared for skinny (TREE_BLOCK_REF) and normal tree ref. +*/ +int level; + +/* +* Root who refers to this tree block. +* +* For TREE_BLOCK_REF (skinny metadata, either inline or keyed) +*/ +u64 root; + +/* For non-skinny metadata, no special member needed */ +}; + +struct btrfs_ref { +enum btrfs_ref_type type; +int action; + +/* +* Only use parent pointers as backref (SHARED_BLOCK_REF or +* SHARED_DATA_REF) for this extent and its children. +* Set for reloc trees. +*/ +bool only_backreferences:1 ; + +/* +* Whether this extent should go through qgroup record. +* +* Normally false, but for certain case like delayed subtree scan, +* setting this flag can hugely reduce qgroup overhead. +*/ +bool skip_qgroup:1 ; + +/* +* Optional. To which root this modification is for. +* Mostly used for qgroup optimization. +* +* When unset, data/tree ref init code will populate it. +* In certain case, we're modifying reference for a different root. +* E.g. Cow fs tree blocks for balance. +* In that case, tree_ref::root will be fs tree, but we're doing this +* for reloc tree, then we should set @real_root to reloc tree. +*/ +u64 real_root; +u64 bytenr; +u64 len; + +/* Bytenr of the parent tree block */ +u64 parent; +union { +struct btrfs_data_ref data_ref; +struct btrfs_tree_ref tree_ref; + }; +}; +extern struct kmem_cache *btrfs_delayed_ref_head_cachep;extern struct kmem_cache *btrfs_delayed_tree_ref_cachep;extern struct kmem_cache *btrfs_delayed_data_ref_cachep;
@@ -184,6 +268,38 @@ extern struct kmem_cache *btrfs_delayed_extent_op_cachep;int __init btrfs_delayed_ref_init (void );void __cold btrfs_delayed_ref_exit (void ); +static inline void btrfs_init_generic_ref (struct btrfs_ref *generic_ref, +int action,u64 bytenr,u64 len,u64 parent) +{ + generic_ref->action = action; + generic_ref->bytenr = bytenr; + generic_ref->len = len; + generic_ref->parent = parent; +} + +static inline void btrfs_init_tree_ref (struct btrfs_ref *generic_ref, +int level,u64 root) +{ +/* If @real_root not set, use @root as fallback */ +if (!generic_ref->real_root) + generic_ref->real_root = root; + generic_ref->tree_ref.level = level; + generic_ref->tree_ref.root = root; + generic_ref->type =BTRFS_REF_METADATA ; +} + +static inline void btrfs_init_data_ref (struct btrfs_ref *generic_ref, +u64 ref_root,u64 ino,u64 offset) +{ +/* If @real_root not set, use @root as fallback */ +if (!generic_ref->real_root) + generic_ref->real_root = ref_root; + generic_ref->data_ref.ref_root = ref_root; + generic_ref->data_ref.ino = ino; + generic_ref->data_ref.offset = offset; + generic_ref->type =BTRFS_REF_DATA ; +} +static inline struct btrfs_delayed_extent_op *btrfs_alloc_delayed_extent_op (void ) {
Signature · 1 line
2.21.0
Re: [PATCH v3.1 1/9] btrfs: delayed-ref: Introduce better documented delayed ref structures
Quoted 91 lines · Qu Wenruo
Current delayed ref interface has several problems:
- Longer and longer parameter lists
bytenr
num_bytes
parent
---------- so far so good
ref_root
owner
offset
---------- I don't feel good now- Different interpretation for the same parameter Above @owner for data ref is inode number (u64), while for tree ref, it's level (int).
They are even in different size range.
For level we only need 0~8, while for ino it's BTRFS_FIRST_FREE_OBJECTID~BTRFS_LAST_FREE_OBJECTID.And @offset doesn't even makes sense for tree ref.
Such parameter reuse may look clever as an hidden union, but it destroys code readability.
To solve both problems, we introduce a new structure,
btrfs_refto solve them:- Structure instead of long parameter list
This makes later expansion easier, and better documented.- Use
btrfs_ref::typeto distinguish data and tree ref- Use proper union to store data/tree ref specific structures.
- Use separate functions to fill data/tree ref data, with a common generic function to fill common bytenr/num_bytes members.
All parameters will find its place in
btrfs_ref, and an extra member, @real_root, inspired by ref-verify code, is newly introduced for later qgroup code, to record which tree is triggered this extent modification.This patch doesn't touch any code, but provides the basis for incoming refactors.
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/delayed-ref.h | 116 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+)fs/btrfs/delayed-ref.h +34
-0@@ -176,6 +176,90 @@ struct btrfs_delayed_ref_root {u64 qgroup_to_skip; }; +enum btrfs_ref_type { +BTRFS_REF_NOT_SET , +BTRFS_REF_DATA , +BTRFS_REF_METADATA , +BTRFS_REF_LAST , +}; + +struct btrfs_data_ref { +/* For EXTENT_DATA_REF */ + +/* Root who refers to this data extent */ +u64 ref_root; + +/* Inode who refers to this data extent */ +u64 ino; + +/* +* file_offset - extent_offset +* +* file_offset is the key.offset of the EXTENT_DATA key. +* extent_offset is btrfs_file_extent_offset() of the EXTENT_DATA data. +*/ +u64 offset; +}; + +struct btrfs_tree_ref { +/* +* Level of this tree block +* +* Shared for skinny (TREE_BLOCK_REF) and normal tree ref. +*/ +int level; + +/*
fs/btrfs/delayed-ref.h struct btrfs_delayed_ref_root { · line 222
Quoted 10 lines
fs/btrfs/delayed-ref.h +10
-0+* Root who refers to this tree block. +* +* For TREE_BLOCK_REF (skinny metadata, either inline or keyed) +*/ +u64 root; + +/* For non-skinny metadata, no special member needed */ +}; + +struct btrfs_ref {
The structure name sounds a bit generic, but I think we can keep it short. There are no other btrfs-specific references that could be confused, there are 'backrefs', 'delayed-refs' all refering to the b-tree references.
fs/btrfs/delayed-ref.h struct btrfs_delayed_ref_root { · line 231
Quoted 9 lines
fs/btrfs/delayed-ref.h +9
-0+enum btrfs_ref_type type; +int action; + +/* +* Only use parent pointers as backref (SHARED_BLOCK_REF or +* SHARED_DATA_REF) for this extent and its children. +* Set for reloc trees. +*/ + bool only_backreferences:1;
No bool bitfields please, wasn't this mentioned last time?
Re: [PATCH v3.1 1/9] btrfs: delayed-ref: Introduce better documented delayed ref structures
Quoted 108 lines · David Sterba
Quoted 91 lines · Qu Wenruo
Current delayed ref interface has several problems:
- Longer and longer parameter lists
bytenr
num_bytes
parent
---------- so far so good
ref_root
owner
offset
---------- I don't feel good now- Different interpretation for the same parameter Above @owner for data ref is inode number (u64), while for tree ref, it's level (int).
They are even in different size range.
For level we only need 0~8, while for ino it's BTRFS_FIRST_FREE_OBJECTID~BTRFS_LAST_FREE_OBJECTID.And @offset doesn't even makes sense for tree ref.
Such parameter reuse may look clever as an hidden union, but it destroys code readability.
To solve both problems, we introduce a new structure,
btrfs_refto solve them:- Structure instead of long parameter list
This makes later expansion easier, and better documented.- Use
btrfs_ref::typeto distinguish data and tree ref- Use proper union to store data/tree ref specific structures.
- Use separate functions to fill data/tree ref data, with a common generic function to fill common bytenr/num_bytes members.
All parameters will find its place in
btrfs_ref, and an extra member, @real_root, inspired by ref-verify code, is newly introduced for later qgroup code, to record which tree is triggered this extent modification.This patch doesn't touch any code, but provides the basis for incoming refactors.
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/delayed-ref.h | 116 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+)fs/btrfs/delayed-ref.h +34
-0@@ -176,6 +176,90 @@ struct btrfs_delayed_ref_root {u64 qgroup_to_skip; }; +enum btrfs_ref_type { +BTRFS_REF_NOT_SET , +BTRFS_REF_DATA , +BTRFS_REF_METADATA , +BTRFS_REF_LAST , +}; + +struct btrfs_data_ref { +/* For EXTENT_DATA_REF */ + +/* Root who refers to this data extent */ +u64 ref_root; + +/* Inode who refers to this data extent */ +u64 ino; + +/* +* file_offset - extent_offset +* +* file_offset is the key.offset of the EXTENT_DATA key. +* extent_offset is btrfs_file_extent_offset() of the EXTENT_DATA data. +*/ +u64 offset; +}; + +struct btrfs_tree_ref { +/* +* Level of this tree block +* +* Shared for skinny (TREE_BLOCK_REF) and normal tree ref. +*/ +int level; + +/* fs/btrfs/delayed-ref.h struct btrfs_delayed_ref_root { · line 222
Quoted 10 lines
fs/btrfs/delayed-ref.h +10
-0+* Root who refers to this tree block. +* +* For TREE_BLOCK_REF (skinny metadata, either inline or keyed) +*/ +u64 root; + +/* For non-skinny metadata, no special member needed */ +}; + +struct btrfs_ref {The structure name sounds a bit generic, but I think we can keep it short. There are no other btrfs-specific references that could be confused, there are 'backrefs', 'delayed-refs' all refering to the b-tree references.
Quoted 12 lines
fs/btrfs/delayed-ref.h struct btrfs_delayed_ref_root { · line 231
Quoted 9 lines
fs/btrfs/delayed-ref.h +9
-0+enum btrfs_ref_type type; +int action; + +/* +* Only use parent pointers as backref (SHARED_BLOCK_REF or +* SHARED_DATA_REF) for this extent and its children. +* Set for reloc trees. +*/ + bool only_backreferences:1;No bool bitfields please, wasn't this mentioned last time?
Oh, I forgot that one.
Do I need to resend or just edit that commit in my github branch?
Thanks,
Qu
Re: [PATCH v3.1 1/9] btrfs: delayed-ref: Introduce better documented delayed ref structures
Quoted 123 lines · Qu Wenruo
Quoted 108 lines · David Sterba
Quoted 91 lines · Qu Wenruo
Current delayed ref interface has several problems:
- Longer and longer parameter lists
bytenr
num_bytes
parent
---------- so far so good
ref_root
owner
offset
---------- I don't feel good now- Different interpretation for the same parameter Above @owner for data ref is inode number (u64), while for tree ref, it's level (int).
They are even in different size range.
For level we only need 0~8, while for ino it's BTRFS_FIRST_FREE_OBJECTID~BTRFS_LAST_FREE_OBJECTID.And @offset doesn't even makes sense for tree ref.
Such parameter reuse may look clever as an hidden union, but it destroys code readability.
To solve both problems, we introduce a new structure,
btrfs_refto solve them:- Structure instead of long parameter list
This makes later expansion easier, and better documented.- Use
btrfs_ref::typeto distinguish data and tree ref- Use proper union to store data/tree ref specific structures.
- Use separate functions to fill data/tree ref data, with a common generic function to fill common bytenr/num_bytes members.
All parameters will find its place in
btrfs_ref, and an extra member, @real_root, inspired by ref-verify code, is newly introduced for later qgroup code, to record which tree is triggered this extent modification.This patch doesn't touch any code, but provides the basis for incoming refactors.
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/delayed-ref.h | 116 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+)fs/btrfs/delayed-ref.h +34
-0@@ -176,6 +176,90 @@ struct btrfs_delayed_ref_root {u64 qgroup_to_skip; }; +enum btrfs_ref_type { +BTRFS_REF_NOT_SET , +BTRFS_REF_DATA , +BTRFS_REF_METADATA , +BTRFS_REF_LAST , +}; + +struct btrfs_data_ref { +/* For EXTENT_DATA_REF */ + +/* Root who refers to this data extent */ +u64 ref_root; + +/* Inode who refers to this data extent */ +u64 ino; + +/* +* file_offset - extent_offset +* +* file_offset is the key.offset of the EXTENT_DATA key. +* extent_offset is btrfs_file_extent_offset() of the EXTENT_DATA data. +*/ +u64 offset; +}; + +struct btrfs_tree_ref { +/* +* Level of this tree block +* +* Shared for skinny (TREE_BLOCK_REF) and normal tree ref. +*/ +int level; + +/* fs/btrfs/delayed-ref.h struct btrfs_delayed_ref_root { · line 222
Quoted 10 lines
fs/btrfs/delayed-ref.h +10
-0+* Root who refers to this tree block. +* +* For TREE_BLOCK_REF (skinny metadata, either inline or keyed) +*/ +u64 root; + +/* For non-skinny metadata, no special member needed */ +}; + +struct btrfs_ref {The structure name sounds a bit generic, but I think we can keep it short. There are no other btrfs-specific references that could be confused, there are 'backrefs', 'delayed-refs' all refering to the b-tree references.
Quoted 12 lines
fs/btrfs/delayed-ref.h struct btrfs_delayed_ref_root { · line 231
Quoted 9 lines
fs/btrfs/delayed-ref.h +9
-0+enum btrfs_ref_type type; +int action; + +/* +* Only use parent pointers as backref (SHARED_BLOCK_REF or +* SHARED_DATA_REF) for this extent and its children. +* Set for reloc trees. +*/ + bool only_backreferences:1;No bool bitfields please, wasn't this mentioned last time?
Quoted 3 lines
Oh, I forgot that one.
Do I need to resend or just edit that commit in my github branch?
No need to resend, I'll edit that as it's a trivial change. I'll have another look at the whole structure if the layout could be optimized, eg. the enum takes 4 bytes but we'd be fine with a byte.
Re: [PATCH v3.1 1/9] btrfs: delayed-ref: Introduce better documented delayed ref structures
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
fs/btrfs/delayed-ref.h struct btrfs_delayed_ref_root { · line 222
Quoted 5 lines
fs/btrfs/delayed-ref.h +4
-0+/* For non-skinny metadata, no special member needed */ +}; + +struct btrfs_ref {The structure name sounds a bit generic, but I think we can keep it short. There are no other btrfs-specific references that could be confused, there are 'backrefs', 'delayed-refs' all refering to the b-tree references.
fs/btrfs/delayed-ref.h struct btrfs_delayed_ref_root { · line 231
Quoted 9 lines
fs/btrfs/delayed-ref.h +9
-0+enum btrfs_ref_type type; +int action; + +/* +* Only use parent pointers as backref (SHARED_BLOCK_REF or +* SHARED_DATA_REF) for this extent and its children. +* Set for reloc trees. +*/ + bool only_backreferences:1;No bool bitfields please, wasn't this mentioned last time?
Quoted 8 lines
Quoted 3 lines
Oh, I forgot that one.
Do I need to resend or just edit that commit in my github branch?
No need to resend, I'll edit that as it's a trivial change. I'll have another look at the whole structure if the layout could be optimized, eg. the enum takes 4 bytes but we'd be fine with a byte.
Maybe I have already said this before, but I still really hope either we have some external tool to do that for us, or some attribute to do it.
Thanks,
Qu
Re: [PATCH v3.1 1/9] btrfs: delayed-ref: Introduce better documented delayed ref structures
Quoted 25 lines · Qu Wenruo
Quoted 25 lines
Quoted 25 lines
Quoted 11 lines
fs/btrfs/delayed-ref.h struct btrfs_delayed_ref_root { · line 222
Quoted 5 lines
fs/btrfs/delayed-ref.h +4
-0+/* For non-skinny metadata, no special member needed */ +}; + +struct btrfs_ref {The structure name sounds a bit generic, but I think we can keep it short. There are no other btrfs-specific references that could be confused, there are 'backrefs', 'delayed-refs' all refering to the b-tree references.
Quoted 12 lines
fs/btrfs/delayed-ref.h struct btrfs_delayed_ref_root { · line 231
Quoted 9 lines
fs/btrfs/delayed-ref.h +9
-0+enum btrfs_ref_type type; +int action; + +/* +* Only use parent pointers as backref (SHARED_BLOCK_REF or +* SHARED_DATA_REF) for this extent and its children. +* Set for reloc trees. +*/ + bool only_backreferences:1;No bool bitfields please, wasn't this mentioned last time?
Oh, I forgot that one.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Do I need to resend or just edit that commit in my github branch?
No need to resend, I'll edit that as it's a trivial change. I'll have another look at the whole structure if the layout could be optimized, eg. the enum takes 4 bytes but we'd be fine with a byte.
Maybe I have already said this before, but I still really hope either we have some external tool to do that for us, or some attribute to do it.
Pahole[0] does show the layout of structures. Not entirely automatic but better than nothing.
[0]https://git.kernel.org/pub/scm/devel/pahole/pahole.git
Quoted 4 lines
Thanks,
Qu
Re: [PATCH v3.1 1/9] btrfs: delayed-ref: Introduce better documented delayed ref structures
In reply to Qu Wenruofs/btrfs/delayed-ref.h struct btrfs_delayed_ref_root { · line 231
Quoted 10 lines · Qu Wenruo
fs/btrfs/delayed-ref.h +10
-0+struct btrfs_ref { +enum btrfs_ref_type type; +int action; + +/* +* Only use parent pointers as backref (SHARED_BLOCK_REF or +* SHARED_DATA_REF) for this extent and its children. +* Set for reloc trees. +*/ +bool only_backreferences:1 ;
I renamed this to only_backrefs and was surprised that there were no compilation errors, ie. this member is not used at all ...
Re: [PATCH v3.1 1/9] btrfs: delayed-ref: Introduce better documented delayed ref structures
Quoted 14 lines · David Sterba
fs/btrfs/delayed-ref.h struct btrfs_delayed_ref_root { · line 231
Quoted 10 lines · Qu Wenruo
fs/btrfs/delayed-ref.h +10
-0+struct btrfs_ref { +enum btrfs_ref_type type; +int action; + +/* +* Only use parent pointers as backref (SHARED_BLOCK_REF or +* SHARED_DATA_REF) for this extent and its children. +* Set for reloc trees. +*/ +bool only_backreferences:1 ;I renamed this to
only_backrefsand was surprised that there were no compilation errors, ie. this member is not used at all ...
Yep, for callers who really uses this member, they just set @parent, and that's all.
Thanks,
Qu
Re: [PATCH v3.1 1/9] btrfs: delayed-ref: Introduce better documented delayed ref structures
Quoted 18 lines · Qu Wenruo
Quoted 14 lines · David Sterba
fs/btrfs/delayed-ref.h struct btrfs_delayed_ref_root { · line 231
Quoted 10 lines · Qu Wenruo
fs/btrfs/delayed-ref.h +10
-0+struct btrfs_ref { +enum btrfs_ref_type type; +int action; + +/* +* Only use parent pointers as backref (SHARED_BLOCK_REF or +* SHARED_DATA_REF) for this extent and its children. +* Set for reloc trees. +*/ +bool only_backreferences:1 ;I renamed this to
only_backrefsand was surprised that there were no compilation errors, ie. this member is not used at all ...
Quoted 2 lines
Yep, for callers who really uses this member, they just set @parent, and that's all.
So there's nothing in the old and new code that uses it, then why do you add it? If this is for some existing patchset then ok, keep it there but otherwise remove it.
Re: [PATCH v3.1 1/9] btrfs: delayed-ref: Introduce better documented delayed ref structures
Quoted 19 lines · David Sterba
Quoted 18 lines · Qu Wenruo
Quoted 14 lines · David Sterba
fs/btrfs/delayed-ref.h struct btrfs_delayed_ref_root { · line 231
Quoted 10 lines · Qu Wenruo
fs/btrfs/delayed-ref.h +10
-0+struct btrfs_ref { +enum btrfs_ref_type type; +int action; + +/* +* Only use parent pointers as backref (SHARED_BLOCK_REF or +* SHARED_DATA_REF) for this extent and its children. +* Set for reloc trees. +*/ +bool only_backreferences:1 ;I renamed this to
only_backrefsand was surprised that there were no compilation errors, ie. this member is not used at all ...
Quoted 6 lines
Quoted 2 lines
Yep, for callers who really uses this member, they just set @parent, and that's all.
So there's nothing in the old and new code that uses it, then why do you add it? If this is for some existing patchset then ok, keep it there but otherwise remove it.
It should be removed.
My bad, at the time of writing, I didn't get the point that @parent is enough to info to use SHARED_BLOCK_REF_KEY.
Would you mind to fold this removal?
Thanks,
Qu
Re: [PATCH v3.1 1/9] btrfs: delayed-ref: Introduce better documented delayed ref structures
Quoted 29 lines · Qu Wenruo
Quoted 19 lines · David Sterba
Quoted 18 lines · Qu Wenruo
Quoted 14 lines · David Sterba
fs/btrfs/delayed-ref.h struct btrfs_delayed_ref_root { · line 231
Quoted 10 lines · Qu Wenruo
fs/btrfs/delayed-ref.h +10
-0+struct btrfs_ref { +enum btrfs_ref_type type; +int action; + +/* +* Only use parent pointers as backref (SHARED_BLOCK_REF or +* SHARED_DATA_REF) for this extent and its children. +* Set for reloc trees. +*/ +bool only_backreferences:1 ;I renamed this to
only_backrefsand was surprised that there were no compilation errors, ie. this member is not used at all ...Quoted 6 lines
Quoted 2 lines
Yep, for callers who really uses this member, they just set @parent, and that's all.
So there's nothing in the old and new code that uses it, then why do you add it? If this is for some existing patchset then ok, keep it there but otherwise remove it.
Quoted 5 lines
It should be removed.
My bad, at the time of writing, I didn't get the point that @parent is enough to info to useSHARED_BLOCK_REF_KEY.Would you mind to fold this removal?
If it's sufficient to remove the struct member then I'll do that, no need to resend.
PATCH 2/9 btrfs: extent-tree: Open-code process_func in __btrfs_mod_ref
In reply to Qu WenruoThe process_func function pointer is local to __btrfs_mod_ref() and points to either btrfs_inc_extent_ref() or btrfs_free_extent().
Open code it to make later delayed ref refactor easier, so we can refactor btrfs_inc_extent_ref() and btrfs_free_extent() in different patches.
--- fs/btrfs/extent-tree.c | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-)
fs/btrfs/extent-tree.c +16 -14
@@ -3242,10 +3242,6 @@ static int __btrfs_mod_ref(struct btrfs_trans_handle *trans,int i;int level;int ret =0 ; -int (*process_func)(struct btrfs_trans_handle *, -struct btrfs_root *, -u64 ,u64 ,u64 ,u64 ,u64 ,u64 ); -if (btrfs_is_testing (fs_info))return 0 ;
@@ -3257,11 +3253,6 @@ static int __btrfs_mod_ref(struct btrfs_trans_handle *trans,if (!test_bit (BTRFS_ROOT_REF_COWS , &root->state) && level ==0 )return 0 ; -if (inc) - process_func = btrfs_inc_extent_ref; -else - process_func = btrfs_free_extent; -if (full_backref) parent = buf->start;else
@@ -3283,16 +3274,27 @@ static int __btrfs_mod_ref(struct btrfs_trans_handle *trans, num_bytes =btrfs_file_extent_disk_num_bytes (buf, fi); key.offset -=btrfs_file_extent_offset (buf, fi); - ret =process_func (trans, root, bytenr, num_bytes, - parent, ref_root, key.objectid, - key.offset); +if (inc) + ret =btrfs_inc_extent_ref (trans, root, bytenr, + num_bytes, parent, ref_root, + key.objectid, key.offset); +else + ret =btrfs_free_extent (trans, root, bytenr, + num_bytes, parent, ref_root, + key.objectid, key.offset);if (ret) goto fail; }else { bytenr =btrfs_node_blockptr (buf, i); num_bytes = fs_info->nodesize; - ret =process_func (trans, root, bytenr, num_bytes, - parent, ref_root, level -1 ,0 ); +if (inc) + ret =btrfs_inc_extent_ref (trans, root, bytenr, + num_bytes, parent, ref_root, + level -1 ,0 ); +else + ret =btrfs_free_extent (trans, root, bytenr, + num_bytes, parent, ref_root, + level -1 ,0 );if (ret) goto fail; }
Signature · 1 line
2.21.0
PATCH 3/9 btrfs: delayed-ref: Use btrfs_ref to refactor btrfs_add_delayed_tree_ref()
In reply to Qu Wenruobtrfs_add_delayed_tree_ref() has a longer and longer parameter list, and some caller like btrfs_inc_extent_ref() are using @owner as level for delayed tree ref.
Instead of making the parameter list longer and longer, use btrfs_ref to refactor it, so each parameter assignment should be self-explaining without dirty level/owner trick, and provides the basis for later refactor.
--- fs/btrfs/delayed-ref.c | 22 ++++++++++++++------- fs/btrfs/delayed-ref.h | 3 +-- fs/btrfs/extent-tree.c | 44 +++++++++++++++++++++++++----------------- 3 files changed, 42 insertions(+), 27 deletions(-)
fs/btrfs/delayed-ref.c +15 -7
@@ -735,8 +735,7 @@ static void init_delayed_ref_common(struct btrfs_fs_info *fs_info,* transaction commits. */ int btrfs_add_delayed_tree_ref (struct btrfs_trans_handle *trans, -u64 bytenr,u64 num_bytes,u64 parent, -u64 ref_root,int level,int action, +struct btrfs_ref *generic_ref,struct btrfs_delayed_extent_op *extent_op,int *old_ref_mod,int *new_ref_mod) {
@@ -746,10 +745,16 @@ int btrfs_add_delayed_tree_ref(struct btrfs_trans_handle *trans,struct btrfs_delayed_ref_root *delayed_refs;struct btrfs_qgroup_extent_record *record =NULL ;int qrecord_inserted; -bool is_system = (ref_root ==BTRFS_CHUNK_TREE_OBJECTID ); +bool is_system = (generic_ref->real_root ==BTRFS_CHUNK_TREE_OBJECTID ); +int action = generic_ref->action; +int level = generic_ref->tree_ref.level;int ret; +u64 bytenr = generic_ref->bytenr; +u64 num_bytes = generic_ref->len; +u64 parent = generic_ref->parent;u8 ref_type; +ASSERT (generic_ref->type ==BTRFS_REF_METADATA && generic_ref->action);BUG_ON (extent_op && extent_op->is_data); ref =kmem_cache_alloc (btrfs_delayed_tree_ref_cachep,GFP_NOFS );if (!ref)
@@ -762,7 +767,9 @@ int btrfs_add_delayed_tree_ref(struct btrfs_trans_handle *trans, }if (test_bit (BTRFS_FS_QUOTA_ENABLED , &fs_info->flags) && -is_fstree (ref_root)) { +is_fstree (generic_ref->real_root) && +is_fstree (generic_ref->tree_ref.root) && + !generic_ref->skip_qgroup) { record =kzalloc (sizeof (*record),GFP_NOFS );if (!record) {kmem_cache_free (btrfs_delayed_tree_ref_cachep, ref);
@@ -777,13 +784,14 @@ int btrfs_add_delayed_tree_ref(struct btrfs_trans_handle *trans, ref_type =BTRFS_TREE_BLOCK_REF_KEY ;init_delayed_ref_common (fs_info, &ref->node, bytenr, num_bytes, - ref_root, action, ref_type); - ref->root = ref_root; + generic_ref->tree_ref.root, action, ref_type); + ref->root = generic_ref->tree_ref.root; ref->parent = parent; ref->level = level;init_delayed_ref_head (head_ref, record, bytenr, num_bytes, - ref_root,0 , action, false, is_system); + generic_ref->tree_ref.root,0 , action, false, + is_system); head_ref->extent_op = extent_op; delayed_refs = &trans->transaction->delayed_refs;
fs/btrfs/delayed-ref.h +1 -2
@@ -340,8 +340,7 @@ static inline void btrfs_put_delayed_ref_head(struct btrfs_delayed_ref_head *hea }int btrfs_add_delayed_tree_ref (struct btrfs_trans_handle *trans, -u64 bytenr,u64 num_bytes,u64 parent, -u64 ref_root,int level,int action, +struct btrfs_ref *generic_ref,struct btrfs_delayed_extent_op *extent_op,int *old_ref_mod,int *new_ref_mod); int btrfs_add_delayed_data_ref(struct btrfs_trans_handle *trans,
fs/btrfs/extent-tree.c +26 -18
102 lines not on this page · raw message
Signature · 1 line
2.21.0
PATCH 4/9 btrfs: delayed-ref: Use btrfs_ref to refactor btrfs_add_delayed_data_ref()
In reply to Qu WenruoJust like btrfs_add_delayed_tree_ref(), use btrfs_ref to refactor btrfs_add_delayed_data_ref().
--- fs/btrfs/delayed-ref.c | 18 +++++++++++++----- fs/btrfs/delayed-ref.h | 7 +++---- fs/btrfs/extent-tree.c | 23 ++++++++++------------- 3 files changed, 26 insertions(+), 22 deletions(-)
fs/btrfs/delayed-ref.c +13 -5
@@ -830,10 +830,9 @@ int btrfs_add_delayed_tree_ref(struct btrfs_trans_handle *trans,* add a delayed data ref. it's similar to btrfs_add_delayed_tree_ref. */ int btrfs_add_delayed_data_ref (struct btrfs_trans_handle *trans, -u64 bytenr,u64 num_bytes, -u64 parent,u64 ref_root, -u64 owner,u64 offset,u64 reserved,int action, -int *old_ref_mod,int *new_ref_mod) +struct btrfs_ref *generic_ref, +u64 reserved,int *old_ref_mod, +int *new_ref_mod) {struct btrfs_fs_info *fs_info = trans->fs_info;struct btrfs_delayed_data_ref *ref;
@@ -841,9 +840,17 @@ int btrfs_add_delayed_data_ref(struct btrfs_trans_handle *trans,struct btrfs_delayed_ref_root *delayed_refs;struct btrfs_qgroup_extent_record *record =NULL ;int qrecord_inserted; +int action = generic_ref->action;int ret; +u64 bytenr = generic_ref->bytenr; +u64 num_bytes = generic_ref->len; +u64 parent = generic_ref->parent; +u64 ref_root = generic_ref->data_ref.ref_root; +u64 owner = generic_ref->data_ref.ino; +u64 offset = generic_ref->data_ref.offset;u8 ref_type; +ASSERT (generic_ref->type ==BTRFS_REF_DATA && action); ref =kmem_cache_alloc (btrfs_delayed_data_ref_cachep,GFP_NOFS );if (!ref)return -ENOMEM ;
@@ -867,7 +874,8 @@ int btrfs_add_delayed_data_ref(struct btrfs_trans_handle *trans, }if (test_bit (BTRFS_FS_QUOTA_ENABLED , &fs_info->flags) && -is_fstree (ref_root)) { +is_fstree (ref_root) &&is_fstree (generic_ref->real_root) && + !generic_ref->skip_qgroup) { record =kzalloc (sizeof (*record),GFP_NOFS );if (!record) {kmem_cache_free (btrfs_delayed_data_ref_cachep, ref);
fs/btrfs/delayed-ref.h +3 -4
@@ -344,10 +344,9 @@ int btrfs_add_delayed_tree_ref(struct btrfs_trans_handle *trans,struct btrfs_delayed_extent_op *extent_op, int *old_ref_mod, int *new_ref_mod);int btrfs_add_delayed_data_ref (struct btrfs_trans_handle *trans, -u64 bytenr,u64 num_bytes, -u64 parent,u64 ref_root, -u64 owner,u64 offset,u64 reserved,int action, -int *old_ref_mod,int *new_ref_mod); +struct btrfs_ref *generic_ref, +u64 reserved,int *old_ref_mod, +int *new_ref_mod); int btrfs_add_delayed_extent_op(struct btrfs_fs_info *fs_info, struct btrfs_trans_handle *trans, u64 bytenr, u64 num_bytes,
fs/btrfs/extent-tree.c +10 -13
@@ -2065,10 +2065,8 @@ int btrfs_inc_extent_ref(struct btrfs_trans_handle *trans, ret =btrfs_add_delayed_tree_ref (trans, &generic_ref,NULL , &old_ref_mod, &new_ref_mod); }else { - ret =btrfs_add_delayed_data_ref (trans, bytenr, - num_bytes, parent, - root_objectid, owner, offset, -0 ,BTRFS_ADD_DELAYED_REF , +btrfs_init_data_ref (&generic_ref, root_objectid, owner, offset); + ret =btrfs_add_delayed_data_ref (trans, &generic_ref,0 , &old_ref_mod, &new_ref_mod); }
@@ -7368,10 +7366,8 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans, ret =btrfs_add_delayed_tree_ref (trans, &generic_ref,NULL , &old_ref_mod, &new_ref_mod); }else { - ret =btrfs_add_delayed_data_ref (trans, bytenr, - num_bytes, parent, - root_objectid, owner, offset, -0 ,BTRFS_DROP_DELAYED_REF , +btrfs_init_data_ref (&generic_ref, root_objectid, owner, offset); + ret =btrfs_add_delayed_data_ref (trans, &generic_ref,0 , &old_ref_mod, &new_ref_mod); }
@@ -8479,6 +8475,7 @@ int btrfs_alloc_reserved_file_extent(struct btrfs_trans_handle *trans, u64 offset, u64 ram_bytes, struct btrfs_key *ins) { + struct btrfs_ref generic_ref = { 0 };int ret;BUG_ON (root->root_key.objectid ==BTRFS_TREE_LOG_OBJECTID );
@@ -8487,11 +8484,11 @@ int btrfs_alloc_reserved_file_extent(struct btrfs_trans_handle *trans, root->root_key.objectid, owner, offset, BTRFS_ADD_DELAYED_EXTENT); - ret =btrfs_add_delayed_data_ref (trans, ins->objectid, - ins->offset,0 , - root->root_key.objectid, owner, - offset, ram_bytes, -BTRFS_ADD_DELAYED_EXTENT ,NULL ,NULL ); +btrfs_init_generic_ref (&generic_ref,BTRFS_ADD_DELAYED_EXTENT , + ins->objectid, ins->offset,0 ); +btrfs_init_data_ref (&generic_ref, root->root_key.objectid, owner, offset); + ret =btrfs_add_delayed_data_ref (trans, &generic_ref, + ram_bytes,NULL ,NULL );return ret; }
Signature · 1 line
2.21.0
PATCH 5/9 btrfs: ref-verify: Use btrfs_ref to refactor btrfs_ref_tree_mod()
In reply to Qu WenruoIt's a perfect match for btrfs_ref_tree_mod() to use btrfs_ref, as btrfs_ref describes a metadata/data reference update comprehensively.
Now we have one less function use confusing owner/level trick.
--- fs/btrfs/extent-tree.c | 27 +++++++-------------- fs/btrfs/ref-verify.c | 53 ++++++++++++++++++++++++------------------ fs/btrfs/ref-verify.h | 10 ++++---- 3 files changed, 42 insertions(+), 48 deletions(-)
fs/btrfs/extent-tree.c +8 -19
@@ -2054,9 +2054,6 @@ int btrfs_inc_extent_ref(struct btrfs_trans_handle *trans,BUG_ON (owner <BTRFS_FIRST_FREE_OBJECTID && root_objectid ==BTRFS_TREE_LOG_OBJECTID ); -btrfs_ref_tree_mod (root, bytenr, num_bytes, parent, root_objectid, - owner, offset,BTRFS_ADD_DELAYED_REF ); -btrfs_init_generic_ref (&generic_ref,BTRFS_ADD_DELAYED_REF , bytenr, num_bytes, parent); generic_ref.real_root = root->root_key.objectid;
@@ -2070,6 +2067,8 @@ int btrfs_inc_extent_ref(struct btrfs_trans_handle *trans, &old_ref_mod, &new_ref_mod); } +btrfs_ref_tree_mod (fs_info, &generic_ref); +if (ret ==0 && old_ref_mod <0 && new_ref_mod >=0 ) {bool metadata = owner <BTRFS_FIRST_FREE_OBJECTID ;
@@ -7279,10 +7278,7 @@ void btrfs_free_tree_block(struct btrfs_trans_handle *trans,if (root->root_key.objectid !=BTRFS_TREE_LOG_OBJECTID ) {int old_ref_mod, new_ref_mod; -btrfs_ref_tree_mod (root, buf->start, buf->len, parent, - root->root_key.objectid, -btrfs_header_level (buf),0 , -BTRFS_DROP_DELAYED_REF ); +btrfs_ref_tree_mod (fs_info, &generic_ref); ret =btrfs_add_delayed_tree_ref (trans, &generic_ref,NULL , &old_ref_mod, &new_ref_mod);BUG_ON (ret);/* -ENOMEM */
@@ -7343,11 +7339,6 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans,if (btrfs_is_testing (fs_info))return 0 ; -if (root_objectid !=BTRFS_TREE_LOG_OBJECTID ) -btrfs_ref_tree_mod (root, bytenr, num_bytes, parent, - root_objectid, owner, offset, -BTRFS_DROP_DELAYED_REF ); -btrfs_init_generic_ref (&generic_ref,BTRFS_DROP_DELAYED_REF , bytenr, num_bytes, parent); generic_ref.real_root = root->root_key.objectid;
@@ -7371,6 +7362,9 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans, &old_ref_mod, &new_ref_mod); } +if (root_objectid !=BTRFS_TREE_LOG_OBJECTID ) +btrfs_ref_tree_mod (fs_info, &generic_ref); +if (ret ==0 && old_ref_mod >=0 && new_ref_mod <0 ) {bool metadata = owner <BTRFS_FIRST_FREE_OBJECTID ;
@@ -8480,13 +8474,10 @@ int btrfs_alloc_reserved_file_extent(struct btrfs_trans_handle *trans,BUG_ON (root->root_key.objectid ==BTRFS_TREE_LOG_OBJECTID ); -btrfs_ref_tree_mod (root, ins->objectid, ins->offset,0 , - root->root_key.objectid, owner, offset, -BTRFS_ADD_DELAYED_EXTENT ); -btrfs_init_generic_ref (&generic_ref,BTRFS_ADD_DELAYED_EXTENT , ins->objectid, ins->offset,0 );btrfs_init_data_ref (&generic_ref, root->root_key.objectid, owner, offset); +btrfs_ref_tree_mod (root->fs_info, &generic_ref); ret =btrfs_add_delayed_data_ref (trans, &generic_ref, ram_bytes,NULL ,NULL );return ret;
@@ -8735,13 +8726,11 @@ struct extent_buffer *btrfs_alloc_tree_block(struct btrfs_trans_handle *trans, extent_op->is_data = false; extent_op->level = level; -btrfs_ref_tree_mod (root, ins.objectid, ins.offset, parent, - root_objectid, level,0 , -BTRFS_ADD_DELAYED_EXTENT );btrfs_init_generic_ref (&generic_ref,BTRFS_ADD_DELAYED_EXTENT , ins.objectid, ins.offset, parent); generic_ref.real_root = root->root_key.objectid;btrfs_init_tree_ref (&generic_ref, level, root_objectid); +btrfs_ref_tree_mod (fs_info, &generic_ref); ret =btrfs_add_delayed_tree_ref (trans, &generic_ref, extent_op,NULL ,NULL );if (ret)
fs/btrfs/ref-verify.c +30 -23
107 lines not on this page · raw message
fs/btrfs/ref-verify.h +4 -6
@@ -9,9 +9,8 @@#ifdef CONFIG_BTRFS_FS_REF_VERIFY int btrfs_build_ref_tree (struct btrfs_fs_info *fs_info);void btrfs_free_ref_cache (struct btrfs_fs_info *fs_info); -int btrfs_ref_tree_mod (struct btrfs_root *root,u64 bytenr,u64 num_bytes, -u64 parent,u64 ref_root,u64 owner,u64 offset, -int action); +int btrfs_ref_tree_mod (struct btrfs_fs_info *fs_info, +struct btrfs_ref *generic_ref);void btrfs_free_ref_tree_range (struct btrfs_fs_info *fs_info,u64 start,u64 len);
@@ -30,9 +29,8 @@ static inline void btrfs_free_ref_cache(struct btrfs_fs_info *fs_info) { } -static inline int btrfs_ref_tree_mod (struct btrfs_root *root,u64 bytenr, -u64 num_bytes,u64 parent,u64 ref_root, -u64 owner,u64 offset,int action) +static inline int btrfs_ref_tree_mod (struct btrfs_fs_info *fs_info, +struct btrfs_ref *generic_ref) {return 0 ; }
Signature · 1 line
2.21.0
PATCH 6/9 btrfs: extent-tree: Use btrfs_ref to refactor add_pinned_bytes()
In reply to Qu WenruoSince add_pinned_bytes() only needs to know if the extent is metadata and if it's a chunk tree extent, btrfs_ref is a perfect match for it, as we don't need various owner/level trick to determine extent type.
--- fs/btrfs/extent-tree.c | 26 ++++++++++---------------- 1 file changed, 10 insertions(+), 16 deletions(-)
fs/btrfs/extent-tree.c +10 -16
@@ -756,14 +756,15 @@ static struct btrfs_space_info *__find_space_info(struct btrfs_fs_info *info,return NULL ; } -static void add_pinned_bytes (struct btrfs_fs_info *fs_info,s64 num_bytes, -bool metadata,u64 root_objectid) +static void add_pinned_bytes (struct btrfs_fs_info *fs_info, +struct btrfs_ref *ref) {struct btrfs_space_info *space_info; +s64 num_bytes = -ref->len;u64 flags; -if (metadata) { -if (root_objectid ==BTRFS_CHUNK_TREE_OBJECTID ) +if (ref->type ==BTRFS_REF_METADATA ) { +if (ref->tree_ref.root ==BTRFS_CHUNK_TREE_OBJECTID ) flags =BTRFS_BLOCK_GROUP_SYSTEM ;else flags =BTRFS_BLOCK_GROUP_METADATA ;
@@ -2069,11 +2070,8 @@ int btrfs_inc_extent_ref(struct btrfs_trans_handle *trans,btrfs_ref_tree_mod (fs_info, &generic_ref); -if (ret ==0 && old_ref_mod <0 && new_ref_mod >=0 ) { -bool metadata = owner <BTRFS_FIRST_FREE_OBJECTID ; - -add_pinned_bytes (fs_info, -num_bytes, metadata, root_objectid); - } +if (ret ==0 && old_ref_mod <0 && new_ref_mod >=0 ) +add_pinned_bytes (fs_info, &generic_ref);return ret; }
@@ -7313,8 +7311,7 @@ void btrfs_free_tree_block(struct btrfs_trans_handle *trans, } out:if (pin) -add_pinned_bytes (fs_info, buf->len, true, - root->root_key.objectid); +add_pinned_bytes (fs_info, &generic_ref);if (last_ref) {/*
@@ -7365,11 +7362,8 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans,if (root_objectid !=BTRFS_TREE_LOG_OBJECTID )btrfs_ref_tree_mod (fs_info, &generic_ref); -if (ret ==0 && old_ref_mod >=0 && new_ref_mod <0 ) { -bool metadata = owner <BTRFS_FIRST_FREE_OBJECTID ; - -add_pinned_bytes (fs_info, num_bytes, metadata, root_objectid); - } +if (ret ==0 && old_ref_mod >=0 && new_ref_mod <0 ) +add_pinned_bytes (fs_info, &generic_ref);return ret; }
Signature · 1 line
2.21.0
PATCH 7/9 btrfs: extent-tree: Use btrfs_ref to refactor btrfs_inc_extent_ref()
In reply to Qu WenruoNow we don't need to play the dirty game of reusing @owner for tree block level.
--- fs/btrfs/ctree.h | 5 ++-- fs/btrfs/extent-tree.c | 57 ++++++++++++++++++++++++------------------ fs/btrfs/file.c | 17 +++++++++---- fs/btrfs/inode.c | 10 +++++--- fs/btrfs/ioctl.c | 15 ++++++----- fs/btrfs/relocation.c | 42 ++++++++++++++++++++----------- fs/btrfs/tree-log.c | 11 +++++--- 7 files changed, 96 insertions(+), 61 deletions(-)
fs/btrfs/ctree.h +2 -3
@@ -41,6 +41,7 @@ extern struct kmem_cache *btrfs_bit_radix_cachep;extern struct kmem_cache *btrfs_path_cachep;extern struct kmem_cache *btrfs_free_space_cachep;struct btrfs_ordered_sum ; +struct btrfs_ref ;#define BTRFS_MAGIC 0x4D5F53665248425FULL/* ascii _BHRfS_M, no null */
@@ -2762,9 +2763,7 @@ int btrfs_free_and_pin_reserved_extent(struct btrfs_fs_info *fs_info,void btrfs_prepare_extent_commit (struct btrfs_fs_info *fs_info);int btrfs_finish_extent_commit (struct btrfs_trans_handle *trans);int btrfs_inc_extent_ref (struct btrfs_trans_handle *trans, -struct btrfs_root *root, -u64 bytenr,u64 num_bytes,u64 parent, -u64 root_objectid,u64 owner,u64 offset); +struct btrfs_ref *generic_ref);int btrfs_start_dirty_block_groups (struct btrfs_trans_handle *trans); int btrfs_write_dirty_block_groups(struct btrfs_trans_handle *trans,
fs/btrfs/extent-tree.c +32 -25
100 lines not on this page · raw message
fs/btrfs/file.c +12 -5
@@ -754,6 +754,7 @@ int __btrfs_drop_extents(struct btrfs_trans_handle *trans,struct btrfs_fs_info *fs_info = root->fs_info;struct extent_buffer *leaf;struct btrfs_file_extent_item *fi; +struct btrfs_ref ref = {0 };struct btrfs_key key;struct btrfs_key new_key;u64 ino =btrfs_ino (BTRFS_I (inode));
@@ -909,11 +910,14 @@ int __btrfs_drop_extents(struct btrfs_trans_handle *trans,btrfs_mark_buffer_dirty (leaf);if (update_refs && disk_bytenr >0 ) { - ret =btrfs_inc_extent_ref (trans, root, - disk_bytenr, num_bytes,0 , +btrfs_init_generic_ref (&ref, +BTRFS_ADD_DELAYED_REF , + disk_bytenr, num_bytes,0 ); +btrfs_init_data_ref (&ref, root->root_key.objectid, new_key.objectid, start - extent_offset); + ret =btrfs_inc_extent_ref (trans, &ref);BUG_ON (ret);/* -ENOMEM */ } key.offset = start;
@@ -1142,6 +1146,7 @@ int btrfs_mark_extent_written(struct btrfs_trans_handle *trans,struct extent_buffer *leaf;struct btrfs_path *path;struct btrfs_file_extent_item *fi; +struct btrfs_ref ref = {0 };struct btrfs_key key;struct btrfs_key new_key;u64 bytenr;
@@ -1287,9 +1292,11 @@ int btrfs_mark_extent_written(struct btrfs_trans_handle *trans, extent_end - split);btrfs_mark_buffer_dirty (leaf); - ret =btrfs_inc_extent_ref (trans, root, bytenr, num_bytes, -0 , root->root_key.objectid, - ino, orig_offset); +btrfs_init_generic_ref (&ref,BTRFS_ADD_DELAYED_REF , bytenr, + num_bytes,0 ); +btrfs_init_data_ref (&ref, root->root_key.objectid, ino, + orig_offset); + ret =btrfs_inc_extent_ref (trans, &ref);if (ret) {btrfs_abort_transaction (trans, ret); goto out;
fs/btrfs/inode.c +6 -4
@@ -2520,6 +2520,7 @@ static noinline int relink_extent_backref(struct btrfs_path *path,struct btrfs_file_extent_item *item;struct btrfs_ordered_extent *ordered;struct btrfs_trans_handle *trans; +struct btrfs_ref ref = {0 };struct btrfs_root *root;struct btrfs_key key;struct extent_buffer *leaf;
@@ -2690,10 +2691,11 @@ static noinline int relink_extent_backref(struct btrfs_path *path,inode_add_bytes (inode, len);btrfs_release_path (path); - ret =btrfs_inc_extent_ref (trans, root, new->bytenr, - new->disk_len,0 , - backref->root_id, backref->inum, - new->file_pos);/* start - extent_offset */ +btrfs_init_generic_ref (&ref,BTRFS_ADD_DELAYED_REF , new->bytenr, + new->disk_len,0 ); +btrfs_init_data_ref (&ref, backref->root_id, backref->inum, + new->file_pos);/* start - extent_offset */ + ret =btrfs_inc_extent_ref (trans, &ref);if (ret) {btrfs_abort_transaction (trans, ret); goto out_free_path;
fs/btrfs/ioctl.c +9 -6
@@ -3737,13 +3737,16 @@ static int btrfs_clone(struct inode *src, struct inode *inode, datal);if (disko) { +struct btrfs_ref ref = {0 };inode_add_bytes (inode, datal); - ret =btrfs_inc_extent_ref (trans, - root, - disko, diskl,0 , - root->root_key.objectid, -btrfs_ino (BTRFS_I (inode)), - new_key.offset - datao); +btrfs_init_generic_ref (&ref, +BTRFS_ADD_DELAYED_REF , disko, + diskl,0 ); +btrfs_init_data_ref (&ref, + root->root_key.objectid, +btrfs_ino (BTRFS_I (inode)), + new_key.offset - datao); + ret =btrfs_inc_extent_ref (trans, &ref);if (ret) {btrfs_abort_transaction (trans, ret);
fs/btrfs/relocation.c +27 -15
79 lines not on this page · raw message
fs/btrfs/tree-log.c +8 -3
@@ -705,9 +705,11 @@ static noinline int replay_one_extent(struct btrfs_trans_handle *trans, goto out;if (ins.objectid >0 ) { +struct btrfs_ref ref = {0 };u64 csum_start;u64 csum_end;LIST_HEAD (ordered_sums); +/* * is this extent already allocated in the extent * allocation tree? If so, just add a reference
@@ -715,10 +717,13 @@ static noinline int replay_one_extent(struct btrfs_trans_handle *trans, ret =btrfs_lookup_data_extent (fs_info, ins.objectid, ins.offset);if (ret ==0 ) { - ret =btrfs_inc_extent_ref (trans, root, - ins.objectid, ins.offset, -0 , root->root_key.objectid, +btrfs_init_generic_ref (&ref, +BTRFS_ADD_DELAYED_REF , + ins.objectid, ins.offset,0 ); +btrfs_init_data_ref (&ref, + root->root_key.objectid, key->objectid, offset); + ret =btrfs_inc_extent_ref (trans, &ref);if (ret) goto out; }else {
Signature · 1 line
2.21.0
PATCH 8/9 btrfs: extent-tree: Use btrfs_ref to refactor btrfs_free_extent()
In reply to Qu WenruoSimilar to btrfs_inc_extent_ref(), just use btrfs_ref to replace the long parameter list and the confusing @owner parameter.
--- fs/btrfs/ctree.h | 5 +--- fs/btrfs/extent-tree.c | 52 +++++++++++++++++++----------------------- fs/btrfs/file.c | 22 ++++++++++-------- fs/btrfs/inode.c | 13 +++++++---- fs/btrfs/relocation.c | 25 ++++++++++++-------- 5 files changed, 61 insertions(+), 56 deletions(-)
fs/btrfs/ctree.h +1 -4
@@ -2751,10 +2751,7 @@ int btrfs_set_disk_extent_flags(struct btrfs_trans_handle *trans,struct btrfs_fs_info *fs_info, u64 bytenr, u64 num_bytes, u64 flags, int level, int is_data); -int btrfs_free_extent (struct btrfs_trans_handle *trans, -struct btrfs_root *root, -u64 bytenr,u64 num_bytes,u64 parent,u64 root_objectid, -u64 owner,u64 offset); +int btrfs_free_extent (struct btrfs_trans_handle *trans,struct btrfs_ref *ref);int btrfs_free_reserved_extent (struct btrfs_fs_info *fs_info,u64 start,u64 len,int delalloc);
fs/btrfs/extent-tree.c +23 -29
103 lines not on this page · raw message
fs/btrfs/file.c +12 -10
@@ -997,11 +997,14 @@ int __btrfs_drop_extents(struct btrfs_trans_handle *trans, extent_end =ALIGN (extent_end, fs_info->sectorsize); }else if (update_refs && disk_bytenr >0 ) { - ret =btrfs_free_extent (trans, root, - disk_bytenr, num_bytes,0 , +btrfs_init_generic_ref (&ref, +BTRFS_DROP_DELAYED_REF , + disk_bytenr, num_bytes,0 ); +btrfs_init_data_ref (&ref, root->root_key.objectid, - key.objectid, key.offset - - extent_offset); + key.objectid, + key.offset - extent_offset); + ret =btrfs_free_extent (trans, &ref);BUG_ON (ret);/* -ENOMEM */ inode_sub_bytes (inode, extent_end - key.offset);
@@ -1318,6 +1321,9 @@ int btrfs_mark_extent_written(struct btrfs_trans_handle *trans, other_start = end; other_end =0 ; +btrfs_init_generic_ref (&ref,BTRFS_DROP_DELAYED_REF , bytenr, + num_bytes,0 ); +btrfs_init_data_ref (&ref, root->root_key.objectid, ino, orig_offset);if (extent_mergeable (leaf, path->slots[0 ] +1 , ino, bytenr, orig_offset, &other_start, &other_end)) {
@@ -1328,9 +1334,7 @@ int btrfs_mark_extent_written(struct btrfs_trans_handle *trans, extent_end = other_end; del_slot = path->slots[0 ] +1 ; del_nr++; - ret =btrfs_free_extent (trans, root, bytenr, num_bytes, -0 , root->root_key.objectid, - ino, orig_offset); + ret =btrfs_free_extent (trans, &ref);if (ret) {btrfs_abort_transaction (trans, ret); goto out;
@@ -1348,9 +1352,7 @@ int btrfs_mark_extent_written(struct btrfs_trans_handle *trans, key.offset = other_start; del_slot = path->slots[0 ]; del_nr++; - ret =btrfs_free_extent (trans, root, bytenr, num_bytes, -0 , root->root_key.objectid, - ino, orig_offset); + ret =btrfs_free_extent (trans, &ref);if (ret) {btrfs_abort_transaction (trans, ret); goto out;
fs/btrfs/inode.c +9 -4
@@ -4709,12 +4709,17 @@ int btrfs_truncate_inode_items(struct btrfs_trans_handle *trans,if (found_extent && (test_bit (BTRFS_ROOT_REF_COWS , &root->state) || root == fs_info->tree_root)) { +struct btrfs_ref ref = {0 }; +btrfs_set_path_blocking (path); bytes_deleted += extent_num_bytes; - ret =btrfs_free_extent (trans, root, extent_start, - extent_num_bytes,0 , -btrfs_header_owner (leaf), - ino, extent_offset); + +btrfs_init_generic_ref (&ref,BTRFS_DROP_DELAYED_REF , + extent_start, extent_num_bytes,0 ); + ref.real_root = root->root_key.objectid; +btrfs_init_data_ref (&ref,btrfs_header_owner (leaf), + ino, extent_offset); + ret =btrfs_free_extent (trans, &ref);if (ret) {btrfs_abort_transaction (trans, ret);break ;
fs/btrfs/relocation.c +16 -9
@@ -1716,9 +1716,12 @@ int replace_file_extents(struct btrfs_trans_handle *trans,break ; } - ret =btrfs_free_extent (trans, root, bytenr, num_bytes, - parent,btrfs_header_owner (leaf), - key.objectid, key.offset); +btrfs_init_generic_ref (&ref,BTRFS_DROP_DELAYED_REF , bytenr, + num_bytes, parent); + ref.real_root = root->root_key.objectid; +btrfs_init_data_ref (&ref,btrfs_header_owner (leaf), + key.objectid, key.offset); + ret =btrfs_free_extent (trans, &ref);if (ret) {btrfs_abort_transaction (trans, ret);break ;
@@ -1934,14 +1937,18 @@ int replace_path(struct btrfs_trans_handle *trans, struct reloc_control *rc, ret =btrfs_inc_extent_ref (trans, &ref);BUG_ON (ret); - ret =btrfs_free_extent (trans, src, new_bytenr, blocksize, - path->nodes[level]->start, - src->root_key.objectid, level -1 ,0 ); +btrfs_init_generic_ref (&ref,BTRFS_DROP_DELAYED_REF , new_bytenr, + blocksize, path->nodes[level]->start); +btrfs_init_tree_ref (&ref, level -1 , src->root_key.objectid); + ref.skip_qgroup = true; + ret =btrfs_free_extent (trans, &ref);BUG_ON (ret); - ret =btrfs_free_extent (trans, dest, old_bytenr, blocksize, -0 , dest->root_key.objectid, level -1 , -0 ); +btrfs_init_generic_ref (&ref,BTRFS_DROP_DELAYED_REF , old_bytenr, + blocksize,0 ); +btrfs_init_tree_ref (&ref, level -1 , dest->root_key.objectid); + ref.skip_qgroup = true; + ret =btrfs_free_extent (trans, &ref);BUG_ON (ret);btrfs_unlock_up_safe (path,0 );
Signature · 1 line
2.21.0
PATCH 9/9 btrfs: qgroup: Don't scan leaf if we're modifying reloc tree
In reply to Qu WenruoSince reloc tree doesn't contribute to qgroup numbers, just skip them.
This should catch the final cause of unnecessary data refs for qgroup + metadata balance.
The 4G data 16 snapshots test (*) should explain it pretty well:
| delayed subtree | refactor delayed ref | this patch
---------------------------------------------------------------------
relocated | 22653 | 22673 | 22744 qgroup dirty | 122792 | 48360 | 70 time | 24.494 | 11.606 | 3.944
Finally, we're at the stage where qgroup + metadata balance cost no obvious overhead.
*: Test environment
Test VM:
- vRAM 8G
- vCPU 8
- block dev vitrio-blk, 'unsafe' cache mode
- host block 850evo
Test workload
- Copy 4G data from /usr/ to one subvolume
- Create 16 snapshots of that subvolume, and modify 3 files in each snapshot
- Enable quota, rescan
- Time "btrfs balance start -m"
--- fs/btrfs/extent-tree.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
fs/btrfs/extent-tree.c +6 -4
@@ -9235,11 +9235,13 @@ static noinline int walk_up_proc(struct btrfs_trans_handle *trans,else ret =btrfs_dec_ref (trans, root, eb,0 );BUG_ON (ret);/* -ENOMEM */ - ret =btrfs_qgroup_trace_leaf_items (trans, eb); -if (ret) { -btrfs_err_rl (fs_info, -"error %d accounting leaf items. Quota is out of sync, rescan required." , +if (is_fstree (root->root_key.objectid)) { + ret =btrfs_qgroup_trace_leaf_items (trans, eb); +if (ret) { +btrfs_err_rl (fs_info, +"error %d accounting leaf items. Quota is out of sync, rescan required." , ret); + } } }/* make block locked assertion in btrfs_clean_tree_block happy */
Signature · 1 line
2.21.0
Quoted 116 lines · Qu Wenruo
This patchset can be fetched from github:
https://github.com/adam900710/linux/tree/refactor_delayed_ref_parameterWhich is based on David's misc-next branch, the base commit is:
commit56d46f96de92ec69963acb7b1d9aed83d2a56a7b(david/misc-next-with-write-checks, david/misc-next)Author: Nikolay Borisov <[email protected]> Date: Wed Mar 27 14:24:18 2019 +0200btrfs: Switch
btrfs_trim_free_extentstofind_first_clear_extent_bitCurrent delayed ref interface has several problems:
- Longer and longer parameter lists
bytenr
num_bytes
parent
---- So far so good
ref_root
owner
offset
---- I don't feel well now
for_reloc
^^^^ This parameter only makes sense for qgroup code, but we need
to pass the parameter a long way down.This makes later parameter list add more and more tricky.
- Different interpretation for the same parameter
Above @owner for data ref is inode who owns this extent, while for tree ref, it's level. They are even in different size range.For level we only need 0~8, while for ino it's BTRFS_FIRST_FREE_OBJECTID~BTRFS_LAST_FREE_OBJECTID, so it's still possible to distinguish them, but it's never a straight-forward thing to grasp.
And @offset doesn't even makes sense for tree ref.
Such parameter reuse may look clever as an hidden union, but it destroys code readability.
This patchset will change the way how we pass parameters for delayed ref.
Instead of calling delayed ref interface like:
ret = btrfs_inc_extent_ref(trans, root, bytenr,num_bytes, parent,ref_root, owner, offset); Or ret = btrfs_inc_extent_ref(trans, root, bytenr, nodesize, parent, level, ref_root, 0);We now call like:
btrfs_init_generic_ref(&ref, bytenr,num_bytes,root->root_key.objectid, parent);
btrfs_init_data_ref(&ref, ref_root, owner, offset); ret =btrfs_inc_extent_ref(trans, &ref);
Or
btrfs_init_generic_ref(&ref, bytenr,num_bytes,root->root_key.objectid, parent);
btrfs_init_tree_ref(&ref, level, ref_root);
ret =btrfs_inc_extent_ref(trans, &ref);To determine if a ref is tree or data, instead of calling like:
if (owner <BTRFS_FIRST_FREE_OBJECTID) {
} else {
}
We do it straight-forward:
if (ref->type==BTRFS_REF_METADATA) {
} else {
}And for new members determining some minor behavior, we don't need to add a new parameter to btrfs_add_delayed_tree|data_ref() or
btrfs_inc_extent_ref(), we just assign them after generic/data/tree init, like:btrfs_init_generic_ref(&ref, bytenr,
num_bytes,root->root_key.objectid, parent);
ref->real_root=root->root_key.objectid;ref->skip_qgroup= true;btrfs_init_data_ref(&ref, ref_root, owner, offset);ret = btrfs_inc_extent_ref(trans, &ref);This should improve the code readability and make later code easier to write.
Furthermore, with the help of
btrfs_ref::real_rootparameter, qgroup can skip quit a lot of delayed tree/data ref for reloc tree, which makes qgroup + balance as fast as quota disabled:Test VM:
- vRAM 8G
- vCPU 8
- block dev vitrio-blk, 'unsafe' cache mode
- host block 850evoTest workload
- Copy 4G data from/usr/to one subvolume
- Create 16 snapshots of that subvolume, and modify 3 files in each snapshot
- Enable quota, rescan
- Time "btrfs balance start -m"| base | w/ patchset | no qgroups |
-------------------------------------------------------------relocated | 23765 | 23772 | 23811 | qgroup dirty | 124498 | 70 | 0 | time (sec) | 23.353 | 3.505 | 3.421 |Changelog:
v2:
- Better documentation forbtrfs_refdeclaration
- Rebase to newer delayed subtree rescan patchset
- Add reviewed-by tags
- Remove unnecessaryASSERT()for NULL pointer.
Quoted 6 lines
v3:
- Rebase to misc-next branch as that branch has all prerequisite now.
- Update benchmark result, compare with qgroups disabled case directly.v3.1:
- Rebase to misc-next branch.
This does not seem to exhibit the problems I recall from previous testing, so I'll queue it for merge to misc-next unless there are more review comments.
Quoted 18 lines · David Sterba
Quoted 18 lines
| base | w/ patchset | no qgroups |
-------------------------------------------------------------relocated | 23765 | 23772 | 23811 | qgroup dirty | 124498 | 70 | 0 | time (sec) | 23.353 | 3.505 | 3.421 |Changelog:
v2:
- Better documentation forbtrfs_refdeclaration
- Rebase to newer delayed subtree rescan patchset
- Add reviewed-by tags
- Remove unnecessaryASSERT()for NULL pointer.v3:
- Rebase to misc-next branch as that branch has all prerequisite now.
- Update benchmark result, compare with qgroups disabled case directly.
Quoted 6 lines
Quoted 2 lines
v3.1:
- Rebase to misc-next branch.This does not seem to exhibit the problems I recall from previous testing, so I'll queue it for merge to misc-next unless there are more review comments.
Added to misc-next, ie. the argument cleanup and there's the qgroup+metadata balance improvement. Thanks.