btrfs: qgroup: detect and fix leaked data reserved space
cover btrfs: qgroup: detect and fix leaked data reserved space
There is an internal report complaining that qgroup is only half of the limit, but they still get EDQUOT errors.
With some extra debugging patch added, it turns out that even fsstress with 15 steps can sometimes cause qgroup reserved data space to leak.
This patch set is going to:
- Fix the reserved data space leakage
Mostly caused by missing btrfs_qgroup_free_data() call in release page.
As I thought a dirty page either goes through finish_ordered_io(), or get invalidated directly.
But due to the designed of delayed finish_ordered_io(), we can still get dirty page get released directly.
- Add extra safenet to catch qgroup reserved space leakage.
The existing test case btrfs/022 can already catch the bug pretty reliably.
I will add a specific case for fstests if needed.
Qu Wenruo (2):
btrfs: extent_io: fix qgroup reserved data space leakage when
releasing a page
btrfs: qgroup: catch reserved space leakage at unmount time
fs/btrfs/disk-io.c | 6 ++++++ fs/btrfs/extent_io.c | 34 +++++++++++++++++++++++++++------- fs/btrfs/qgroup.c | 43 +++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/qgroup.h | 2 +- 4 files changed, 77 insertions(+), 8 deletions(-)
Signature · 1 line
2.26.2
PATCH 1/2 btrfs: extent_io: fix qgroup reserved data space leakage when releasing a page
[BUG]
The following simple workload from fsstress can lead to qgroup reserved
data space leakage:
0/0: creat f0 x:0 0 0
0/0: creat add id=0,parent=-1
0/1: write f0[259 1 0 0 0 0] [600030,27288] 0
0/4: dwrite - xfsctl(XFS_IOC_DIOINFO) f0[259 1 0 0 64 627318] return 25, fallback to stat() 0/4: dwrite f0[259 1 0 0 64 627318] [610304,106496] 0
This would cause btrfs qgroup to leak 20480 bytes for data reserved space.
If btrfs qgroup limit is enabled, such leakage can lead to unexpected early EDQUOT and unusable space.
[CAUSE]
When doing direct IO, kernel will try to writeback existing buffered page cache, then invalidate them:
iomap_dio_rw()
|- filemap_write_and_wait_range();
|- invalidate_inode_pages2_range();
However for btrfs, the bi_end_io hook doesn't finish all its heavy work right after bio ends.
In fact, it delays its work further:
submit_extent_page(end_io_func=end_bio_extent_writepage); end_bio_extent_writepage()
|- btrfs_writepage_endio_finish_ordered()
|- btrfs_init_work(finish_ordered_fn);
<<< Work queue execution >>>
finish_ordered_fn()
|- btrfs_finish_ordered_io();
|- Clear qgroup bitsThis means, when filemap_write_and_wait_range() returns, btrfs_finish_ordered_io() is not ensured to be executed, thus the qgroup bits for related range is not cleared.
Now into how the leakage happens, this will only focus on the overlapping part of buffered and direct IO part.
1. After buffered write
The inode had the following range with QGROUP_RESERVED bit:
596 616K
|///////////////|
Qgroup reserved data space: 20K
2. Writeback part for range [596K, 616K)
Write back finished, but btrfs_finish_ordered_io() not get called yet.
So we still have:
596K 616K
|///////////////|
Qgroup reserved data space: 20K
3. Pages for range [596K, 616K) get released
This will clear all qgroup bits, but don't update the reserved data space.
So we have:
596K 616K
| |
Qgroup reserved data space: 20K
That number doesn't match with the qgroup bit range anymore.
4. Dio prepare space for range [596K, 700K) Qgroup reserved data space for that range, we got:
596K 616K 700K
|///////////////|///////////////////////|
Qgroup reserved data space: 20K + 104K = 124K
5. btrfs_finish_ordered_range() get executed for range [596K, 616K) Qgroup free reserved space for that range, we got:
596K 616K 700K
| |///////////////////////|
We need to free that range of reserved space.
Qgroup reserved data space: 124K - 20K = 104K
6. btrfs_finish_ordered_range() get executed for range [596K, 700K) However qgroup bit for range [596K, 616K) is already cleared in previous step, so we only free 84K for qgroup reserved space.
596K 616K 700K
| | |
We need to free that range of reserved space.
Qgroup reserved data space: 104K - 84K = 20K
Now there is no way to release that 20K unless disabling qgroup or unmount the fs.
[FIX]
This patch will fix the problem by calling btrfs_qgroup_free_data() when a page is released.
So that even a dirty page is released, its qgroup reserved data space will get freed along with it.
--- fs/btrfs/extent_io.c | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-)
fs/btrfs/extent_io.c +27 -7
@@ -24,6 +24,7 @@#include "rcu-string.h" #include "backref.h" #include "disk-io.h" +#include "qgroup.h" static struct kmem_cache *extent_state_cache;static struct kmem_cache *extent_buffer_cache;
@@ -4476,21 +4477,40 @@ static int try_release_extent_state(struct extent_io_tree *tree,if (test_range_bit (tree, start, end,EXTENT_LOCKED ,0 ,NULL )) { ret =0 ; }else { +int tmp; + +/* +* When releasepage is called, some page can still be dirty and +* has qgroup reserved bit. +* +* This is caused by delayed endio work, the real endio work, +* finish_ordered_io(), is queued into another workqueue in +* bio endio function. +* Thus even writeback finishes, we do not clear dirty and +* qgroup bits immediately. +* +* So here we still need to clear qgroup bits at release page +* time, or we may leak qgroup reserved data space. +*/ + tmp =btrfs_qgroup_free_data (page->mapping->host,NULL , start, +PAGE_SIZE ); +if (tmp <0 ) + ret =0 ; +/* -* at this point we can safely clear everything except the -* locked bit and the nodatasum bit +* At this point we can safely clear everything except the +* locked bit and the nodatasum bit. */ - ret =__clear_extent_bit (tree, start, end, + tmp =__clear_extent_bit (tree, start, end, ~(EXTENT_LOCKED |EXTENT_NODATASUM ),0 ,0 ,NULL , mask,NULL ); -/* if clear_extent_bit failed for enomem reasons, +/* +* If clear_extent_bit failed for enomem reasons, * we can't allow the release to continue. */ -if (ret <0 ) +if (tmp <0 ) ret =0 ; -else - ret =1 ; }return ret; }
Signature · 1 line
2.26.2
Re: [PATCH 1/2] btrfs: extent_io: fix qgroup reserved data space leakage when releasing a page
Quoted 92 lines · Qu Wenruo
[BUG]
The following simple workload from fsstress can lead to qgroup reserved data space leakage:0/0: creat f0 x:0 0 0 0/0: creat add id=0,parent=-1 0/1: write f0[259 1 0 0 0 0] [600030,27288] 0 0/4: dwrite - xfsctl(XFS_IOC_DIOINFO) f0[259 1 0 0 64 627318] return 25, fallback to stat() 0/4: dwrite f0[259 1 0 0 64 627318] [610304,106496] 0This would cause btrfs qgroup to leak 20480 bytes for data reserved space.
If btrfs qgroup limit is enabled, such leakage can lead to unexpected early EDQUOT and unusable space.[CAUSE]
When doing direct IO, kernel will try to writeback existing buffered page cache, then invalidate them:iomap_dio_rw() |- filemap_write_and_wait_range(); |- invalidate_inode_pages2_range();However for btrfs, the
bi_end_iohook doesn't finish all its heavy work right after bio ends.
In fact, it delays its work further:submit_extent_page(end_io_func=end_bio_extent_writepage); end_bio_extent_writepage() |- btrfs_writepage_endio_finish_ordered() |- btrfs_init_work(finish_ordered_fn); <<< Work queue execution >>> finish_ordered_fn() |- btrfs_finish_ordered_io(); |- Clear qgroup bitsThis means, when
filemap_write_and_wait_range()returns,btrfs_finish_ordered_io()is not ensured to be executed, thus the qgroup bits for related range is not cleared.Now into how the leakage happens, this will only focus on the overlapping part of buffered and direct IO part.
1. After buffered write
The inode had the following range withQGROUP_RESERVEDbit:
596 616K
|///////////////|
Qgroup reserved data space: 20K2. Writeback part for range [596K, 616K)
Write back finished, butbtrfs_finish_ordered_io()not get called yet.
So we still have:
596K 616K
|///////////////|
Qgroup reserved data space: 20K3. Pages for range [596K, 616K) get released
This will clear all qgroup bits, but don't update the reserved data space.
So we have:
596K 616K
| |
Qgroup reserved data space: 20K
That number doesn't match with the qgroup bit range anymore.4. Dio prepare space for range [596K, 700K) Qgroup reserved data space for that range, we got:
596K 616K 700K
|///////////////|///////////////////////|
Qgroup reserved data space: 20K + 104K = 124K5.
btrfs_finish_ordered_range()get executed for range [596K, 616K) Qgroup free reserved space for that range, we got:
596K 616K 700K
| |///////////////////////|
We need to free that range of reserved space.
Qgroup reserved data space: 124K - 20K = 104K6.
btrfs_finish_ordered_range()get executed for range [596K, 700K) However qgroup bit for range [596K, 616K) is already cleared in previous step, so we only free 84K for qgroup reserved space.
596K 616K 700K
| | |
We need to free that range of reserved space.
Qgroup reserved data space: 104K - 84K = 20KNow there is no way to release that 20K unless disabling qgroup or unmount the fs.
[FIX]
This patch will fix the problem by callingbtrfs_qgroup_free_data()when a page is released.
Quoted 5 lines
So that even a dirty page is released, its qgroup reserved data space will get freed along with it.
Fixes f695fdcef83a ("btrfs: qgroup: Introduce functions to release/free qgroup reserve data space")Signed-off-by Qu Wenruo <[email protected]>
This seems backwards to me, and not in keeping with the actual lifetime of the changes. At the point that the ordered extent is created it is now in charge of the qgroup reservation, so it should be the ultimate arbiter of what is done with that qgroup reservation. So fix try_release_extent_state to not remove EXTENT_QGROUP_RESERVED, because it's going to get dropped elsewhere. Thanks,
Josef
Re: [PATCH 1/2] btrfs: extent_io: fix qgroup reserved data space leakage when releasing a page
Quoted 101 lines · Josef Bacik
Quoted 93 lines · Qu Wenruo
[BUG]
The following simple workload from fsstress can lead to qgroup reserved data space leakage:
0/0: creat f0 x:0 0 0
0/0: creat add id=0,parent=-1
0/1: write f0[259 1 0 0 0 0] [600030,27288] 0
0/4: dwrite -xfsctl(XFS_IOC_DIOINFO)f0[259 1 0 0 64 627318] return 25, fallback tostat()
0/4: dwrite f0[259 1 0 0 64 627318] [610304,106496] 0This would cause btrfs qgroup to leak 20480 bytes for data reserved space.
If btrfs qgroup limit is enabled, such leakage can lead to unexpected early EDQUOT and unusable space.[CAUSE]
When doing direct IO, kernel will try to writeback existing buffered page cache, then invalidate them:
iomap_dio_rw()
|-filemap_write_and_wait_range();
|-invalidate_inode_pages2_range();However for btrfs, the
bi_end_iohook doesn't finish all its heavy work right after bio ends.
In fact, it delays its work further:
submit_extent_page(end_io_func=end_bio_extent_writepage);end_bio_extent_writepage()
|-btrfs_writepage_endio_finish_ordered()
|-btrfs_init_work(finish_ordered_fn);<<< Work queue execution >>>
finish_ordered_fn()
|-btrfs_finish_ordered_io();
|- Clear qgroup bitsThis means, when
filemap_write_and_wait_range()returns,btrfs_finish_ordered_io()is not ensured to be executed, thus the qgroup bits for related range is not cleared.Now into how the leakage happens, this will only focus on the overlapping part of buffered and direct IO part.
1. After buffered write
The inode had the following range withQGROUP_RESERVEDbit:
596 616K
|///////////////|
Qgroup reserved data space: 20K2. Writeback part for range [596K, 616K)
Write back finished, butbtrfs_finish_ordered_io()not get called yet.
So we still have:
596K 616K
|///////////////|
Qgroup reserved data space: 20K3. Pages for range [596K, 616K) get released
This will clear all qgroup bits, but don't update the reserved data space.
So we have:
596K 616K
| |
Qgroup reserved data space: 20K
That number doesn't match with the qgroup bit range anymore.4. Dio prepare space for range [596K, 700K) Qgroup reserved data space for that range, we got:
596K 616K 700K |///////////////|///////////////////////| Qgroup reserved data space: 20K + 104K = 124K5.
btrfs_finish_ordered_range()get executed for range [596K, 616K) Qgroup free reserved space for that range, we got:
596K 616K 700K
| |///////////////////////|
We need to free that range of reserved space.
Qgroup reserved data space: 124K - 20K = 104K6.
btrfs_finish_ordered_range()get executed for range [596K, 700K) However qgroup bit for range [596K, 616K) is already cleared in previous step, so we only free 84K for qgroup reserved space. 596K 616K 700K
| | |
We need to free that range of reserved space.
Qgroup reserved data space: 104K - 84K = 20KNow there is no way to release that 20K unless disabling qgroup or unmount the fs.
[FIX]
This patch will fix the problem by callingbtrfs_qgroup_free_data()when a page is released.Quoted 6 lines
So that even a dirty page is released, its qgroup reserved data space will get freed along with it.
Fixes f695fdcef83a ("btrfs: qgroup: Introduce functions torelease/free qgroup reserve data space")
Signed-off-by Qu Wenruo <[email protected]>
Quoted 6 lines
This seems backwards to me, and not in keeping with the actual lifetime of the changes. At the point that the ordered extent is created it is now in charge of the qgroup reservation, so it should be the ultimate arbiter of what is done with that qgroup reservation. So fix
try_release_extent_stateto not removeEXTENT_QGROUP_RESERVED, because it's going to get dropped elsewhere. Thanks,
Indeed, doing the qgroup rsv work in ordered extent looks more reasonable.
Although that change would make a lot of timing completely different, and won't go as smooth in the first run, it still looks like a more proper fix.
Thanks for the advice,
Qu
Quoted 2 lines
Josef
PATCH 2/2 btrfs: qgroup: catch reserved space leakage at unmount time
In reply to Qu Wenruo--- fs/btrfs/disk-io.c | 6 ++++++ fs/btrfs/qgroup.c | 43 +++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/qgroup.h | 2 +- 3 files changed, 50 insertions(+), 1 deletion(-)
fs/btrfs/disk-io.c +6 -0
@@ -4058,6 +4058,12 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)ASSERT (list_empty (&fs_info->delayed_iputs));set_bit (BTRFS_FS_CLOSING_DONE , &fs_info->flags); +if (btrfs_qgroup_has_leak (fs_info)) { +WARN (IS_ENABLED (CONFIG_BTRFS_DEBUG ), +KERN_ERR "BTRFS: qgroup reserved space leaked\n" ); +btrfs_err (fs_info,"qgroup reserved space leaked\n" ); + } +btrfs_free_qgroup_config (fs_info);ASSERT (list_empty (&fs_info->delalloc_roots));
fs/btrfs/qgroup.c +43 -0
49 lines not on this page · raw message
fs/btrfs/qgroup.h +1 -1
@@ -415,5 +415,5 @@ int btrfs_qgroup_add_swapped_blocks(struct btrfs_trans_handle *trans,int btrfs_qgroup_trace_subtree_after_cow (struct btrfs_trans_handle *trans,struct btrfs_root *root,struct extent_buffer *eb);void btrfs_qgroup_destroy_extent_records (struct btrfs_transaction *trans); - +bool btrfs_qgroup_has_leak (struct btrfs_fs_info *fs_info);#endif
Signature · 1 line
2.26.2
Re: [PATCH 2/2] btrfs: qgroup: catch reserved space leakage at unmount time
Quoted 12 lines · Qu Wenruo
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/disk-io.c | 6 ++++++ fs/btrfs/qgroup.c | 43 +++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/qgroup.h | 2 +- 3 files changed, 50 insertions(+), 1 deletion(-)fs/btrfs/disk-io.c +0
-0
fs/btrfs/disk-io.c void __cold close_ctree(struct btrfs_fs_info *fs_info) · line 4064
Quoted 7 lines
fs/btrfs/disk-io.c +4
-0@@ -4058,6 +4058,12 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)ASSERT (list_empty (&fs_info->delayed_iputs));set_bit (BTRFS_FS_CLOSING_DONE , &fs_info->flags); +if (btrfs_qgroup_has_leak (fs_info)) { +WARN (IS_ENABLED (CONFIG_BTRFS_DEBUG ), +KERN_ERR "BTRFS: qgroup reserved space leaked\n" ); +btrfs_err (fs_info,"qgroup reserved space leaked\n" );
I don't think the message from the WARN() brings any value, it's simply duplicated by the btrfs_err. IMO it's more concise to do:
WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));btrfs_err(qgroup reserved space leaked);
without losing any information whatsoever.
fs/btrfs/disk-io.c void __cold close_ctree(struct btrfs_fs_info *fs_info) · line 4065
Quoted 1 line
fs/btrfs/disk-io.c +1
-0+ }
Is it safe calling this code here? workqueues are being destroyed after it in btrfs_stop_all_workers so it's possible that they have some lingering work which in turn might cause false positive in this check ?
Quoted 16 lines
fs/btrfs/disk-io.c +1
-0+btrfs_free_qgroup_config (fs_info);ASSERT (list_empty (&fs_info->delalloc_roots));fs/btrfs/qgroup.c +4
-0@@ -505,6 +505,49 @@ int btrfs_read_qgroup_config(struct btrfs_fs_info *fs_info)return ret <0 ? ret :0 ; } +static u64 btrfs_qgroup_subvolid (u64 qgroupid) +{ +return (qgroupid & ((1ULL <<BTRFS_QGROUP_LEVEL_SHIFT ) -1 )); +}
fs/btrfs/qgroup.c int btrfs_read_qgroup_config(struct btrfs_fs_info *fs_info) · line 521
Quoted 10 lines
fs/btrfs/qgroup.c +10
-0+/* +* Get called for close_ctree() when quota is still enabled. +* This verifies we don't leak some reserved space. +* +* Return false if no reserved space is left. +* Return true if some reserved space is leaked. +*/ +bool btrfs_qgroup_has_leak (struct btrfs_fs_info *fs_info) +{ +struct btrfs_qgroup *qgroup;
nit:This variable is used only in the loop below just define it there to reduce its scope.
Quoted 44 lines
fs/btrfs/qgroup.c +29
-0+struct rb_node *node; +bool ret = false; + +if (!test_bit (BTRFS_FS_QUOTA_ENABLED , &fs_info->flags)) +return ret; +/* +* Since we're unmounting, there is no race and no need to grab +* qgroup lock. +* And here we don't go post order to provide a more user friendly +* sorted result. +*/ +for (node =rb_first (&fs_info->qgroup_tree); node; node =rb_next (node)) { +int i; + + qgroup = rb_entry(node, struct btrfs_qgroup, node); +for (i =0 ; i <BTRFS_QGROUP_RSV_LAST ; i++) { +if (qgroup->rsv.values[i]) { + ret = true; +btrfs_warn (fs_info, +"qgroup %llu/%llu has unreleased space, type=%d rsv=%llu" , +btrfs_qgroup_level (qgroup->qgroupid), +btrfs_qgroup_subvolid (qgroup->qgroupid), + i, qgroup->rsv.values[i]); + } + } + } +return ret; +} +/* * This is called from close_ctree() or open_ctree() or btrfs_quota_disable(), * first two are in single-threaded paths.And for the third one, we have set fs/btrfs/qgroup.h +1
-1@@ -415,5 +415,5 @@ int btrfs_qgroup_add_swapped_blocks(struct btrfs_trans_handle *trans,int btrfs_qgroup_trace_subtree_after_cow (struct btrfs_trans_handle *trans,struct btrfs_root *root,struct extent_buffer *eb);void btrfs_qgroup_destroy_extent_records (struct btrfs_transaction *trans); - +bool btrfs_qgroup_has_leak (struct btrfs_fs_info *fs_info);#endif
Re: [PATCH 2/2] btrfs: qgroup: catch reserved space leakage at unmount time
Quoted 25 lines · Nikolay Borisov
Quoted 12 lines · Qu Wenruo
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/disk-io.c | 6 ++++++ fs/btrfs/qgroup.c | 43 +++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/qgroup.h | 2 +- 3 files changed, 50 insertions(+), 1 deletion(-)fs/btrfs/disk-io.c +0
-0fs/btrfs/disk-io.c void __cold close_ctree(struct btrfs_fs_info *fs_info) · line 4064
Quoted 7 lines
fs/btrfs/disk-io.c +4
-0@@ -4058,6 +4058,12 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)ASSERT (list_empty (&fs_info->delayed_iputs));set_bit (BTRFS_FS_CLOSING_DONE , &fs_info->flags); +if (btrfs_qgroup_has_leak (fs_info)) { +WARN (IS_ENABLED (CONFIG_BTRFS_DEBUG ), +KERN_ERR "BTRFS: qgroup reserved space leaked\n" ); +btrfs_err (fs_info,"qgroup reserved space leaked\n" );I don't think the message from the
WARN()brings any value, it's simply duplicated by thebtrfs_err. IMO it's more concise to do:
Quoted 4 lines
WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));
btrfs_err(qgroup reserved space leaked);without losing any information whatsoever.
Makes sense, also another cleanup item for existing similar cases.
Quoted 6 lines
fs/btrfs/disk-io.c void __cold close_ctree(struct btrfs_fs_info *fs_info) · line 4065
Quoted 2 lines
fs/btrfs/disk-io.c +1
-0+ }Is it safe calling this code here? workqueues are being destroyed after it in
btrfs_stop_all_workersso it's possible that they have some lingering work which in turn might cause false positive in this check ?
The safety here is as safe as other calls in close_ctree(), we expect no new trans started nor existing running trans (finish ordered io call), and no dirty pages (invalidate/release page call)
So at this timing there should be nothing to modify qgroup and we're safe. Or did I miss something for the close_ctree() context?
Quoted 21 lines
Quoted 21 lines
fs/btrfs/disk-io.c +1
-0+btrfs_free_qgroup_config (fs_info);ASSERT (list_empty (&fs_info->delalloc_roots));fs/btrfs/qgroup.c +9
-0@@ -505,6 +505,49 @@ int btrfs_read_qgroup_config(struct btrfs_fs_info *fs_info)return ret <0 ? ret :0 ; } +static u64 btrfs_qgroup_subvolid (u64 qgroupid) +{ +return (qgroupid & ((1ULL <<BTRFS_QGROUP_LEVEL_SHIFT ) -1 )); +} +/* +* Get called for close_ctree() when quota is still enabled. +* This verifies we don't leak some reserved space. +* +* Return false if no reserved space is left.
Quoted 8 lines
fs/btrfs/qgroup.c int btrfs_read_qgroup_config(struct btrfs_fs_info *fs_info) · line 521
Quoted 5 lines
fs/btrfs/qgroup.c +5
-0+* Return true if some reserved space is leaked. +*/ +bool btrfs_qgroup_has_leak (struct btrfs_fs_info *fs_info) +{ +struct btrfs_qgroup *qgroup;nit:This variable is used only in the loop below just define it there to reduce its scope.
Sure.
Thanks,
Qu
Quoted 45 lines
Quoted 45 lines
fs/btrfs/qgroup.c +29
-0+struct rb_node *node; +bool ret = false; + +if (!test_bit (BTRFS_FS_QUOTA_ENABLED , &fs_info->flags)) +return ret; +/* +* Since we're unmounting, there is no race and no need to grab +* qgroup lock. +* And here we don't go post order to provide a more user friendly +* sorted result. +*/ +for (node =rb_first (&fs_info->qgroup_tree); node; node =rb_next (node)) { +int i; + + qgroup = rb_entry(node, struct btrfs_qgroup, node); +for (i =0 ; i <BTRFS_QGROUP_RSV_LAST ; i++) { +if (qgroup->rsv.values[i]) { + ret = true; +btrfs_warn (fs_info, +"qgroup %llu/%llu has unreleased space, type=%d rsv=%llu" , +btrfs_qgroup_level (qgroup->qgroupid), +btrfs_qgroup_subvolid (qgroup->qgroupid), + i, qgroup->rsv.values[i]); + } + } + } +return ret; +} +/* * This is called from close_ctree() or open_ctree() or btrfs_quota_disable(), * first two are in single-threaded paths.And for the third one, we have set fs/btrfs/qgroup.h +1
-1@@ -415,5 +415,5 @@ int btrfs_qgroup_add_swapped_blocks(struct btrfs_trans_handle *trans,int btrfs_qgroup_trace_subtree_after_cow (struct btrfs_trans_handle *trans,struct btrfs_root *root,struct extent_buffer *eb);void btrfs_qgroup_destroy_extent_records (struct btrfs_transaction *trans); - +bool btrfs_qgroup_has_leak (struct btrfs_fs_info *fs_info);#endif
Re: [PATCH 2/2] btrfs: qgroup: catch reserved space leakage at unmount time
In reply to Qu WenruoQuoted 12 lines · Qu Wenruo
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/disk-io.c | 6 ++++++ fs/btrfs/qgroup.c | 43 +++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/qgroup.h | 2 +- 3 files changed, 50 insertions(+), 1 deletion(-)fs/btrfs/disk-io.c +0
-0
fs/btrfs/disk-io.c void __cold close_ctree(struct btrfs_fs_info *fs_info) · line 4065
Quoted 8 lines
fs/btrfs/disk-io.c +5
-0@@ -4058,6 +4058,12 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)ASSERT (list_empty (&fs_info->delayed_iputs));set_bit (BTRFS_FS_CLOSING_DONE , &fs_info->flags); +if (btrfs_qgroup_has_leak (fs_info)) { +WARN (IS_ENABLED (CONFIG_BTRFS_DEBUG ), +KERN_ERR "BTRFS: qgroup reserved space leaked\n" ); +btrfs_err (fs_info,"qgroup reserved space leaked\n" ); + }
This looks like debugging aid, so:
if (IS_ENABLED(CONFIG_BTRFS_DEBUG))
btrfs_check_qgroup_leak(fs_info);
would be more readable (WARN() pushed to the function).
Best Regards,
Michał Mirosław
Re: [PATCH 2/2] btrfs: qgroup: catch reserved space leakage at unmount time
Quoted 24 lines · Michał Mirosław
Quoted 12 lines · Qu Wenruo
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/disk-io.c | 6 ++++++ fs/btrfs/qgroup.c | 43 +++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/qgroup.h | 2 +- 3 files changed, 50 insertions(+), 1 deletion(-)fs/btrfs/disk-io.c +0
-0fs/btrfs/disk-io.c void __cold close_ctree(struct btrfs_fs_info *fs_info) · line 4065
Quoted 8 lines
fs/btrfs/disk-io.c +5
-0@@ -4058,6 +4058,12 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)ASSERT (list_empty (&fs_info->delayed_iputs));set_bit (BTRFS_FS_CLOSING_DONE , &fs_info->flags); +if (btrfs_qgroup_has_leak (fs_info)) { +WARN (IS_ENABLED (CONFIG_BTRFS_DEBUG ), +KERN_ERR "BTRFS: qgroup reserved space leaked\n" ); +btrfs_err (fs_info,"qgroup reserved space leaked\n" ); + }This looks like debugging aid, so:
Quoted 4 lines
if (
IS_ENABLED(CONFIG_BTRFS_DEBUG))btrfs_check_qgroup_leak(fs_info);would be more readable (
WARN()pushed to the function).
We want to check to be executed even on production system, but just less noisy (no kernel backtrace dump).
Just like tree-checker and EXTENT_QUOTA_RESERVED check.
Thanks,
Qu
Quoted 4 lines
Best Regards,
Michał Mirosław
Re: [PATCH 2/2] btrfs: qgroup: catch reserved space leakage at unmount time
Quoted 25 lines · Qu Wenruo
Quoted 24 lines · Michał Mirosław
Quoted 12 lines · Qu Wenruo
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/disk-io.c | 6 ++++++ fs/btrfs/qgroup.c | 43 +++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/qgroup.h | 2 +- 3 files changed, 50 insertions(+), 1 deletion(-)fs/btrfs/disk-io.c +0
-0fs/btrfs/disk-io.c void __cold close_ctree(struct btrfs_fs_info *fs_info) · line 4065
Quoted 8 lines
fs/btrfs/disk-io.c +5
-0@@ -4058,6 +4058,12 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)ASSERT (list_empty (&fs_info->delayed_iputs));set_bit (BTRFS_FS_CLOSING_DONE , &fs_info->flags); +if (btrfs_qgroup_has_leak (fs_info)) { +WARN (IS_ENABLED (CONFIG_BTRFS_DEBUG ), +KERN_ERR "BTRFS: qgroup reserved space leaked\n" ); +btrfs_err (fs_info,"qgroup reserved space leaked\n" ); + }This looks like debugging aid, so:
Quoted 8 lines
Quoted 4 lines
if (
IS_ENABLED(CONFIG_BTRFS_DEBUG))btrfs_check_qgroup_leak(fs_info);would be more readable (
WARN()pushed to the function).We want to check to be executed even on production system, but just less noisy (no kernel backtrace dump).
Just like tree-checker andEXTENT_QUOTA_RESERVEDcheck.
In that case I suggest:
btrfs_err(...);
WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));
as I expect people look for messages before the Oops for more information.
Best Regards,
Michał Mirosław
Re: [PATCH 2/2] btrfs: qgroup: catch reserved space leakage at unmount time
Quoted 35 lines · Michał Mirosław
Quoted 25 lines · Qu Wenruo
Quoted 24 lines · Michał Mirosław
Quoted 12 lines · Qu Wenruo
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/disk-io.c | 6 ++++++ fs/btrfs/qgroup.c | 43 +++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/qgroup.h | 2 +- 3 files changed, 50 insertions(+), 1 deletion(-)fs/btrfs/disk-io.c +0
-0fs/btrfs/disk-io.c void __cold close_ctree(struct btrfs_fs_info *fs_info) · line 4065
Quoted 8 lines
fs/btrfs/disk-io.c +5
-0@@ -4058,6 +4058,12 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)ASSERT (list_empty (&fs_info->delayed_iputs));set_bit (BTRFS_FS_CLOSING_DONE , &fs_info->flags); +if (btrfs_qgroup_has_leak (fs_info)) { +WARN (IS_ENABLED (CONFIG_BTRFS_DEBUG ), +KERN_ERR "BTRFS: qgroup reserved space leaked\n" ); +btrfs_err (fs_info,"qgroup reserved space leaked\n" ); + }This looks like debugging aid, so:
Quoted 8 lines
Quoted 4 lines
if (
IS_ENABLED(CONFIG_BTRFS_DEBUG))btrfs_check_qgroup_leak(fs_info);would be more readable (
WARN()pushed to the function).We want to check to be executed even on production system, but just less noisy (no kernel backtrace dump).
Just like tree-checker andEXTENT_QUOTA_RESERVEDcheck.
Quoted 6 lines
In that case I suggest:
btrfs_err(...);
WARN_ON(IS_ENABLED(CONFIG_BTRFS_DEBUG));as I expect people look for messages before the Oops for more information.
Yep, exactly what Nik suggested and what I would do in next version.
Thanks,
Qu
Quoted 4 lines
Best Regards,
Michał Mirosław
Re: [PATCH 2/2] btrfs: qgroup: catch reserved space leakage at unmount time
In reply to Qu WenruoOn Sun, Jun 07, 2020 at 03:25:12PM +0800, Qu Wenruo wrote:
Please write some changelog.
Quoted 12 lines
Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/disk-io.c | 6 ++++++ fs/btrfs/qgroup.c | 43 +++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/qgroup.h | 2 +- 3 files changed, 50 insertions(+), 1 deletion(-)fs/btrfs/disk-io.c +0
-0
fs/btrfs/disk-io.c void __cold close_ctree(struct btrfs_fs_info *fs_info) · line 4064
Quoted 7 lines
fs/btrfs/disk-io.c +4
-0@@ -4058,6 +4058,12 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)ASSERT (list_empty (&fs_info->delayed_iputs));set_bit (BTRFS_FS_CLOSING_DONE , &fs_info->flags); +if (btrfs_qgroup_has_leak (fs_info)) { +WARN (IS_ENABLED (CONFIG_BTRFS_DEBUG ), +KERN_ERR "BTRFS: qgroup reserved space leaked\n" ); +btrfs_err (fs_info,"qgroup reserved space leaked\n" );
No newline in the btrfs_err strings.
Quoted 10 lines
fs/btrfs/disk-io.c +2
-0+ } +btrfs_free_qgroup_config (fs_info);ASSERT (list_empty (&fs_info->delalloc_roots));fs/btrfs/qgroup.c +0
-0
fs/btrfs/qgroup.c int btrfs_read_qgroup_config(struct btrfs_fs_info *fs_info) · line 511
Quoted 7 lines
fs/btrfs/qgroup.c +4
-0@@ -505,6 +505,49 @@ int btrfs_read_qgroup_config(struct btrfs_fs_info *fs_info)return ret <0 ? ret :0 ; } +static u64 btrfs_qgroup_subvolid (u64 qgroupid) +{ +return (qgroupid & ((1ULL <<BTRFS_QGROUP_LEVEL_SHIFT ) -1 )); +}
Missing newline.
fs/btrfs/qgroup.c int btrfs_read_qgroup_config(struct btrfs_fs_info *fs_info) · line 519
Quoted 8 lines
fs/btrfs/qgroup.c +8
-0+/* +* Get called for close_ctree() when quota is still enabled. +* This verifies we don't leak some reserved space. +* +* Return false if no reserved space is left. +* Return true if some reserved space is leaked. +*/ +bool btrfs_qgroup_has_leak (struct btrfs_fs_info *fs_info)
I think we've been naming such functions with 'check' eg. btrfs_check_quota_leak, and return true/false.
Quoted 12 lines
fs/btrfs/qgroup.c +12
-0+{ +struct btrfs_qgroup *qgroup; +struct rb_node *node; +bool ret = false; + +if (!test_bit (BTRFS_FS_QUOTA_ENABLED , &fs_info->flags)) +return ret; +/* +* Since we're unmounting, there is no race and no need to grab +* qgroup lock. +* And here we don't go post order to provide a more user friendly +* sorted result.
fs/btrfs/qgroup.c int btrfs_read_qgroup_config(struct btrfs_fs_info *fs_info) · line 541
Quoted 10 lines
fs/btrfs/qgroup.c +10
-0+*/ +for (node =rb_first (&fs_info->qgroup_tree); node; node =rb_next (node)) { +int i; + + qgroup = rb_entry(node, struct btrfs_qgroup, node); +for (i =0 ; i <BTRFS_QGROUP_RSV_LAST ; i++) { +if (qgroup->rsv.values[i]) { + ret = true; + btrfs_warn(fs_info, + "qgroup %llu/%llu has unreleased space, type=%d rsv=%llu",
If this could be potentially noisy, the ratelimited version would be more suitable.
The message wording sounds ok, as it points to the qgroups and there's one global message printed from close_ctree that says it's a 'leak'.
Quoted 17 lines
fs/btrfs/qgroup.c +9
-0+btrfs_qgroup_level (qgroup->qgroupid), +btrfs_qgroup_subvolid (qgroup->qgroupid), + i, qgroup->rsv.values[i]); + } + } + } +return ret; +} +/* * This is called from close_ctree() or open_ctree() or btrfs_quota_disable(), * first two are in single-threaded paths.And for the third one, we have set fs/btrfs/qgroup.h +0
-0
fs/btrfs/qgroup.h int btrfs_qgroup_add_swapped_blocks(struct btrfs_trans_handle *trans, · line 418
Quoted 5 lines
fs/btrfs/qgroup.h +1
-1@@ -415,5 +415,5 @@ int btrfs_qgroup_add_swapped_blocks(struct btrfs_trans_handle *trans,int btrfs_qgroup_trace_subtree_after_cow (struct btrfs_trans_handle *trans,struct btrfs_root *root,struct extent_buffer *eb);void btrfs_qgroup_destroy_extent_records (struct btrfs_transaction *trans); - +bool btrfs_qgroup_has_leak (struct btrfs_fs_info *fs_info);
So when I'm pointing out newlines, please keep one between the text and the last #endif.
Thanks.
Quoted 3 lines
fs/btrfs/qgroup.h +0
-0#endif Signature · 1 line
2.26.2