btrfs: tree-checker: Make write time tree checker verify the content of leaves
PATCH 1/2 btrfs: tree-checker: Make write time tree checker verify the content of leaves
Commit "btrfs: Do mandatory tree block check before submitting bio" is designed to check leaves contents, just as read time check.
However due to the confusing parameter list, "false" is passed to where it is supposed to be "true".
--- fs/btrfs/tree-checker.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/tree-checker.c +1 -1
@@ -999,7 +999,7 @@ int btrfs_check_leaf_relaxed(struct btrfs_fs_info *fs_info,int btrfs_check_leaf_write (struct btrfs_fs_info *fs_info,struct extent_buffer *leaf) { -return check_leaf (leaf, false, false); +return check_leaf (leaf, true, false); }int btrfs_check_node (struct btrfs_fs_info *fs_info,struct extent_buffer *node)
Signature · 1 line
2.21.0
PATCH 2/2 btrfs: tree-checker: Remove comprehensive root owner check
Commit 1ba98d086fe3 ("Btrfs: detect corruption when non-root leaf has zero item") introduced comprehensive root owner checker.
However it's pretty expensive tree search to locate the owner root, especially when it get reused by mandatory read and write time tree-checker.
This patch will remove that check, and completely rely on owner based empty leaf check, which is much faster and still works fine for most case.
And since we skip the old root owner check, now write time tree check can be merged with btrfs_check_leaf_full().
--- fs/btrfs/disk-io.c | 2 +- fs/btrfs/tree-checker.c | 51 +++-------------------------------------- fs/btrfs/tree-checker.h | 2 -- 3 files changed, 4 insertions(+), 51 deletions(-)
fs/btrfs/disk-io.c +1 -1
@@ -539,7 +539,7 @@ static int csum_dirty_buffer(struct btrfs_fs_info *fs_info, struct page *page)if (btrfs_header_level (eb)) err =btrfs_check_node (fs_info, eb);else - err =btrfs_check_leaf_write (fs_info, eb); + err =btrfs_check_leaf_full (fs_info, eb);if (err <0 ) { btrfs_err(fs_info,
fs/btrfs/tree-checker.c +3 -48
80 lines not on this page · raw message
fs/btrfs/tree-checker.h +0 -2
@@ -23,8 +23,6 @@ int btrfs_check_leaf_full(struct btrfs_fs_info *fs_info,*/ int btrfs_check_leaf_relaxed (struct btrfs_fs_info *fs_info,struct extent_buffer *leaf); -int btrfs_check_leaf_write (struct btrfs_fs_info *fs_info, -struct extent_buffer *leaf);int btrfs_check_node (struct btrfs_fs_info *fs_info,struct extent_buffer *node); int btrfs_check_chunk_valid(struct btrfs_fs_info *fs_info,
Signature · 1 line
2.21.0
Re: [PATCH 2/2] btrfs: tree-checker: Remove comprehensive root owner check
Quoted 2 lines · Qu Wenruo
Commit
1ba98d086fe3("Btrfs: detect corruption when non-root leaf has zero item") introduced comprehensive root owner checker.
That's 4.7, so we might want to do stable backport to 4.9+.
Quoted 11 lines
However it's pretty expensive tree search to locate the owner root, especially when it get reused by mandatory read and write time tree-checker.
This patch will remove that check, and completely rely on owner based empty leaf check, which is much faster and still works fine for most case.
And since we skip the old root owner check, now write time tree check can be merged with
btrfs_check_leaf_full().
And for that reason the change should be minimal, this patch depends on current development queue so it's not suitable for the backport.
I'm thinking what to do here, I want to avoid patches that effectively revert changes that are still in the development branch and haven't been released.
This means to rework "btrfs: Do mandatory tree block check before submitting bio", though it's deep in the branch, the scope of changes is only 2 files so this should not cause any conflicts.
Re: [PATCH 2/2] btrfs: tree-checker: Remove comprehensive root owner check
Quoted 5 lines · David Sterba
Quoted 2 lines · Qu Wenruo
Commit
1ba98d086fe3("Btrfs: detect corruption when non-root leaf has zero item") introduced comprehensive root owner checker.That's 4.7, so we might want to do stable backport to 4.9+.
It's just a performance optimization.
I'm not sure if we should backport such things even if we can solve the conflicts.
Quoted 12 lines
Quoted 11 lines
However it's pretty expensive tree search to locate the owner root, especially when it get reused by mandatory read and write time tree-checker.
This patch will remove that check, and completely rely on owner based empty leaf check, which is much faster and still works fine for most case.
And since we skip the old root owner check, now write time tree check can be merged with
btrfs_check_leaf_full().
Quoted 6 lines
And for that reason the change should be minimal, this patch depends on current development queue so it's not suitable for the backport.
I'm thinking what to do here, I want to avoid patches that effectively revert changes that are still in the development branch and haven't been released.
OK, I'll resend write time tree checker patchset.
Quoted 4 lines
This means to rework "btrfs: Do mandatory tree block check before submitting bio", though it's deep in the branch, the scope of changes is only 2 files so this should not cause any conflicts.
BTW, I'm a little interesting in the workflow on your side.
In this case, what's the correct way to handle them?
Just remove those two related patches from misc-next branch and re-apply the newer version?
Or remove all those write time tree checker and later enhancement patches and reapply them all?
Either way, I hope to experience the same workflow so I could reduce your workload.
Thanks,
Qu
Re: [PATCH 2/2] btrfs: tree-checker: Remove comprehensive root owner check
Quoted 11 lines · Qu Wenruo
Quoted 5 lines · David Sterba
Quoted 2 lines · Qu Wenruo
Commit
1ba98d086fe3("Btrfs: detect corruption when non-root leaf has zero item") introduced comprehensive root owner checker.That's 4.7, so we might want to do stable backport to 4.9+.
It's just a performance optimization.
I'm not sure if we should backport such things even if we can solve the conflicts.
That's for the consideration, if the performance improvement is significant, but due to other changes in the codebases it might not be safe.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
However it's pretty expensive tree search to locate the owner root, especially when it get reused by mandatory read and write time tree-checker.
This patch will remove that check, and completely rely on owner based empty leaf check, which is much faster and still works fine for most case.
And since we skip the old root owner check, now write time tree check can be merged with
btrfs_check_leaf_full().
Quoted 8 lines
Quoted 6 lines
And for that reason the change should be minimal, this patch depends on current development queue so it's not suitable for the backport.
I'm thinking what to do here, I want to avoid patches that effectively revert changes that are still in the development branch and haven't been released.
OK, I'll resend write time tree checker patchset.
Wait, it's just one patch that needs update, "btrfs: Do mandatory tree block check before submitting bio".
If the above patch is dropped, then the expensive checks in check_leaf can be removed as you do in this patch, without the updates to parameters.
Quoted 9 lines
Quoted 3 lines
This means to rework "btrfs: Do mandatory tree block check before submitting bio", though it's deep in the branch, the scope of changes is only 2 files so this should not cause any conflicts.
BTW, I'm a little interesting in the workflow on your side.
In this case, what's the correct way to handle them?
Just remove those two related patches from misc-next branch and re-apply the newer version?
If you mean these two
btrfs: disk-io: Show the timing of corrupted tree block explicitly
btrfs: Do mandatory tree block check before submitting bio
we don't need to remove both, just the second one.
Quoted 2 lines
Or remove all those write time tree checker and later enhancement patches and reapply them all?
No, why removing them? That's independent enhancement of the tree checker. What needs to be removed is the call at write time.
Quoted 2 lines
Either way, I hope to experience the same workflow so I could reduce your workload.
This is an infrequent case when a patch deep in the branch needs update so the solution depends on the impact on the patches above it. Here it's only minor conflict with a cleanup patch removing redundant fs_info from check_leaf.
So that's trivial and we don't need to update the patches in place. Further updates can be applied on top of misc-next (with the patch removed).
The ideal workflow is:
1) patches are developed, tested by the developer
2) posted to mailinglist
3) after review added to a topic branch in for-next for wider testing
- patchset revisions are easy to update as it's in a topic branch
4) after some time the patches get merged to misc-next and no updates are expected, beyond minor updates like coding style or changelog
5) patch goes to release branch and is effectively frozen, all fixes go as separate patches
So if you want to reduce my workload, spend more time on 1-3 so we don't have to be creative when fixing problems we discover at 4-5.
I know this cannot be followed 100% times, but so far we've been able to resolve all the situations and the number of times is overall low. So it's bearable.
Re: [PATCH 2/2] btrfs: tree-checker: Remove comprehensive root owner check
In reply to Qu WenruoQuoted 7 lines · Qu Wenruo
Commit
1ba98d086fe3("Btrfs: detect corruption when non-root leaf has zero item") introduced comprehensive root owner checker.However it's pretty expensive tree search to locate the owner root, especially when it get reused by mandatory read and write time tree-checker.
Quoted 8 lines
This patch will remove that check, and completely rely on owner based empty leaf check, which is much faster and still works fine for most case.
And since we skip the old root owner check, now write time tree check can be merged with
btrfs_check_leaf_full().Signed-off-by Qu Wenruo <[email protected]>
Quoted 5 lines · Qu Wenruo
Commit "btrfs: Do mandatory tree block check before submitting bio" is designed to check leaves contents, just as read time check.
However due to the confusing parameter list, "false" is passed to where it is supposed to be "true".
And this seems to invalidate all testing since false->true will do additional expensive checks. I thought I'd fold the fix to the original patch but in this case it'd be the wrong thing to do due to the visible effects of the change.
Quoted 9 lines · David Sterba
Quoted 5 lines · Qu Wenruo
Commit "btrfs: Do mandatory tree block check before submitting bio" is designed to check leaves contents, just as read time check.
However due to the confusing parameter list, "false" is passed to where it is supposed to be "true".
And this seems to invalidate all testing since
false->truewill do additional expensive checks.
That's the case.
However with the next patch, btrfs_check_leaf_full() is less expensive. Just 3~5 times expensive as csum_tree_block(), reduced from 20 times.
My full fstests run is coming to the end and so far no regression.
Quoted 3 lines
I thought I'd fold the fix to the original
patch but in this case it'd be the wrong thing to do due to the visible effects of the change.
If that's the case, I'd add new Fixes: tag, but that can only happen after write time patch get merged upstream.
Thanks,
Qu
Quoted 18 lines · Qu Wenruo
Quoted 9 lines · David Sterba
Quoted 5 lines · Qu Wenruo
Commit "btrfs: Do mandatory tree block check before submitting bio" is designed to check leaves contents, just as read time check.
However due to the confusing parameter list, "false" is passed to where it is supposed to be "true".
And this seems to invalidate all testing since
false->truewill do additional expensive checks.That's the case.
However with the next patch,
btrfs_check_leaf_full()is less expensive. Just 3~5 times expensive ascsum_tree_block(), reduced from 20 times.
Quoted 8 lines
My full fstests run is coming to the end and so far no regression.
Quoted 3 lines
I thought I'd fold the fix to the original
patch but in this case it'd be the wrong thing to do due to the visible effects of the change.If that's the case, I'd add new Fixes: tag, but that can only happen after write time patch get merged upstream.
Fixes: to development patches should raise an alert that something's wrong.
Quoted 7 lines · Qu Wenruo
Commit "btrfs: Do mandatory tree block check before submitting bio" is designed to check leaves contents, just as read time check.
However due to the confusing parameter list, "false" is passed to where it is supposed to be "true".
Signed-off-by Qu Wenruo <[email protected]>
The subject of the patch could be a bit more precise, by passing true you are doing further validation of the body of individual items in the leaves. Alternatively you are sanity checking the keys and the boundaries of item body. So how about:
"Perform item body validation during write"
Quoted 18 lines
--- fs/btrfs/tree-checker.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/btrfs/tree-checker.c +1
-1@@ -999,7 +999,7 @@ int btrfs_check_leaf_relaxed(struct btrfs_fs_info *fs_info,int btrfs_check_leaf_write (struct btrfs_fs_info *fs_info,struct extent_buffer *leaf) { -return check_leaf (leaf, false, false); +return check_leaf (leaf, true, false); }int btrfs_check_node (struct btrfs_fs_info *fs_info,struct extent_buffer *node)
Quoted 9 lines · Nikolay Borisov
Quoted 6 lines · Qu Wenruo
Commit "btrfs: Do mandatory tree block check before submitting bio" is designed to check leaves contents, just as read time check.
However due to the confusing parameter list, "false" is passed to where it is supposed to be "true".
Quoted 8 lines
Quoted 1 line
Signed-off-by Qu Wenruo <[email protected]>The subject of the patch could be a bit more precise, by passing true you are doing further validation of the body of individual items in the leaves. Alternatively you are sanity checking the keys and the boundaries of item body. So how about:
"Perform item body validation during write"
That's much better, I'd take that tile if I need to resend.
Thanks,
Qu
Quoted 22 lines
Reviewed-by Nikolay Borisov <[email protected]>Quoted 18 lines
--- fs/btrfs/tree-checker.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/btrfs/tree-checker.c +1
-1@@ -999,7 +999,7 @@ int btrfs_check_leaf_relaxed(struct btrfs_fs_info *fs_info,int btrfs_check_leaf_write (struct btrfs_fs_info *fs_info,struct extent_buffer *leaf) { -return check_leaf (leaf, false, false); +return check_leaf (leaf, true, false); }int btrfs_check_node (struct btrfs_fs_info *fs_info,struct extent_buffer *node)