Skip to contentSkip to conversation

btrfs: tree-checker: Make write time tree checker verify the content of leaves

PATCH In review Qu Wenruo 4 participants 11 messages Apr 3, 2019 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

PATCH 1/2 btrfs: tree-checker: Make write time tree checker verify the content of leaves

Commit "btrfs: Do mandatory tree block check before submitting bio" is designed to check leaves contents, just as read time check.

However due to the confusing parameter list, "false" is passed to where it is supposed to be "true".

---
 fs/btrfs/tree-checker.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/tree-checker.c +1 -1
index 4318e3e67443..204fe53c90aa 100644
@@ -999,7 +999,7 @@ int btrfs_check_leaf_relaxed(struct btrfs_fs_info *fs_info,
 int btrfs_check_leaf_write(struct btrfs_fs_info *fs_info,
 			   struct extent_buffer *leaf)
 {
-	return check_leaf(leaf, false, false);
+	return check_leaf(leaf, true, false);
 }
 
 int btrfs_check_node(struct btrfs_fs_info *fs_info, struct extent_buffer *node)
Signature · 1 line
2.21.0
author
#1 lore raw

PATCH 2/2 btrfs: tree-checker: Remove comprehensive root owner check

Commit 1ba98d086fe3 ("Btrfs: detect corruption when non-root leaf has zero item") introduced comprehensive root owner checker.

However it's pretty expensive tree search to locate the owner root, especially when it get reused by mandatory read and write time tree-checker.

This patch will remove that check, and completely rely on owner based empty leaf check, which is much faster and still works fine for most case.

And since we skip the old root owner check, now write time tree check can be merged with btrfs_check_leaf_full().

---
 fs/btrfs/disk-io.c      |  2 +-
 fs/btrfs/tree-checker.c | 51 +++--------------------------------------
 fs/btrfs/tree-checker.h |  2 --
 3 files changed, 4 insertions(+), 51 deletions(-)
fs/btrfs/disk-io.c +1 -1
index c3b89584dad6..fddcf7d7c30c 100644
@@ -539,7 +539,7 @@ static int csum_dirty_buffer(struct btrfs_fs_info *fs_info, struct page *page)
 	if (btrfs_header_level(eb))
 		err = btrfs_check_node(fs_info, eb);
 	else
-		err = btrfs_check_leaf_write(fs_info, eb);
+		err = btrfs_check_leaf_full(fs_info, eb);
 
 	if (err < 0) {
 		btrfs_err(fs_info,
fs/btrfs/tree-checker.c +3 -48

80 lines not on this page · raw message

fs/btrfs/tree-checker.h +0 -2
index 281eb7136bc1..4df45e8a6659 100644
@@ -23,8 +23,6 @@ int btrfs_check_leaf_full(struct btrfs_fs_info *fs_info,
  */
 int btrfs_check_leaf_relaxed(struct btrfs_fs_info *fs_info,
 			     struct extent_buffer *leaf);
-int btrfs_check_leaf_write(struct btrfs_fs_info *fs_info,
-			   struct extent_buffer *leaf);
 int btrfs_check_node(struct btrfs_fs_info *fs_info, struct extent_buffer *node);
 
 int btrfs_check_chunk_valid(struct btrfs_fs_info *fs_info,
Signature · 1 line
2.21.0
#2 lore raw

Re: [PATCH 2/2] btrfs: tree-checker: Remove comprehensive root owner check

Quoted 2 lines · Qu Wenruo

Commit 1ba98d086fe3 ("Btrfs: detect corruption when non-root leaf has zero item") introduced comprehensive root owner checker.

That's 4.7, so we might want to do stable backport to 4.9+.

Quoted 11 lines

However it's pretty expensive tree search to locate the owner root, especially when it get reused by mandatory read and write time tree-checker.

This patch will remove that check, and completely rely on owner based empty leaf check, which is much faster and still works fine for most case.

And since we skip the old root owner check, now write time tree check can be merged with btrfs_check_leaf_full().

And for that reason the change should be minimal, this patch depends on current development queue so it's not suitable for the backport.

I'm thinking what to do here, I want to avoid patches that effectively revert changes that are still in the development branch and haven't been released.

This means to rework "btrfs: Do mandatory tree block check before submitting bio", though it's deep in the branch, the scope of changes is only 2 files so this should not cause any conflicts.

#3 lore raw

Re: [PATCH 2/2] btrfs: tree-checker: Remove comprehensive root owner check

Quoted 5 lines · David Sterba
Quoted 2 lines · Qu Wenruo

Commit 1ba98d086fe3 ("Btrfs: detect corruption when non-root leaf has zero item") introduced comprehensive root owner checker.

That's 4.7, so we might want to do stable backport to 4.9+.

It's just a performance optimization.

I'm not sure if we should backport such things even if we can solve the conflicts.

Quoted 12 lines
Quoted 11 lines

However it's pretty expensive tree search to locate the owner root, especially when it get reused by mandatory read and write time tree-checker.

This patch will remove that check, and completely rely on owner based empty leaf check, which is much faster and still works fine for most case.

And since we skip the old root owner check, now write time tree check can be merged with btrfs_check_leaf_full().

Quoted 6 lines

And for that reason the change should be minimal, this patch depends on current development queue so it's not suitable for the backport.

I'm thinking what to do here, I want to avoid patches that effectively revert changes that are still in the development branch and haven't been released.

OK, I'll resend write time tree checker patchset.

Quoted 4 lines

This means to rework "btrfs: Do mandatory tree block check before submitting bio", though it's deep in the branch, the scope of changes is only 2 files so this should not cause any conflicts.

BTW, I'm a little interesting in the workflow on your side.

In this case, what's the correct way to handle them?
Just remove those two related patches from misc-next branch and re-apply the newer version?

Or remove all those write time tree checker and later enhancement patches and reapply them all?

Either way, I hope to experience the same workflow so I could reduce your workload.

Thanks,
Qu

#4 lore raw

Re: [PATCH 2/2] btrfs: tree-checker: Remove comprehensive root owner check

Quoted 11 lines · Qu Wenruo
Quoted 5 lines · David Sterba
Quoted 2 lines · Qu Wenruo

Commit 1ba98d086fe3 ("Btrfs: detect corruption when non-root leaf has zero item") introduced comprehensive root owner checker.

That's 4.7, so we might want to do stable backport to 4.9+.

It's just a performance optimization.

I'm not sure if we should backport such things even if we can solve the conflicts.

That's for the consideration, if the performance improvement is significant, but due to other changes in the codebases it might not be safe.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

However it's pretty expensive tree search to locate the owner root, especially when it get reused by mandatory read and write time tree-checker.

This patch will remove that check, and completely rely on owner based empty leaf check, which is much faster and still works fine for most case.

And since we skip the old root owner check, now write time tree check can be merged with btrfs_check_leaf_full().

Quoted 8 lines
Quoted 6 lines

And for that reason the change should be minimal, this patch depends on current development queue so it's not suitable for the backport.

I'm thinking what to do here, I want to avoid patches that effectively revert changes that are still in the development branch and haven't been released.

OK, I'll resend write time tree checker patchset.

Wait, it's just one patch that needs update, "btrfs: Do mandatory tree block check before submitting bio".

If the above patch is dropped, then the expensive checks in check_leaf can be removed as you do in this patch, without the updates to parameters.

Quoted 9 lines
Quoted 3 lines

This means to rework "btrfs: Do mandatory tree block check before submitting bio", though it's deep in the branch, the scope of changes is only 2 files so this should not cause any conflicts.

BTW, I'm a little interesting in the workflow on your side.

In this case, what's the correct way to handle them?
Just remove those two related patches from misc-next branch and re-apply the newer version?

If you mean these two

btrfs: disk-io: Show the timing of corrupted tree block explicitly
btrfs: Do mandatory tree block check before submitting bio

we don't need to remove both, just the second one.

Quoted 2 lines

Or remove all those write time tree checker and later enhancement patches and reapply them all?

No, why removing them? That's independent enhancement of the tree checker. What needs to be removed is the call at write time.

Quoted 2 lines

Either way, I hope to experience the same workflow so I could reduce your workload.

This is an infrequent case when a patch deep in the branch needs update so the solution depends on the impact on the patches above it. Here it's only minor conflict with a cleanup patch removing redundant fs_info from check_leaf.

So that's trivial and we don't need to update the patches in place. Further updates can be applied on top of misc-next (with the patch removed).

The ideal workflow is:

1) patches are developed, tested by the developer
2) posted to mailinglist
3) after review added to a topic branch in for-next for wider testing
- patchset revisions are easy to update as it's in a topic branch
4) after some time the patches get merged to misc-next and no updates are expected, beyond minor updates like coding style or changelog
5) patch goes to release branch and is effectively frozen, all fixes go as separate patches

So if you want to reduce my workload, spend more time on 1-3 so we don't have to be creative when fixing problems we discover at 4-5.

I know this cannot be followed 100% times, but so far we've been able to resolve all the situations and the number of times is overall low. So it's bearable.

Reviewed-by on v1 2/2
#5 lore raw

Re: [PATCH 2/2] btrfs: tree-checker: Remove comprehensive root owner check

In reply to Qu Wenruo
Quoted 7 lines · Qu Wenruo

Commit 1ba98d086fe3 ("Btrfs: detect corruption when non-root leaf has zero item") introduced comprehensive root owner checker.

However it's pretty expensive tree search to locate the owner root, especially when it get reused by mandatory read and write time tree-checker.

Quoted 8 lines

This patch will remove that check, and completely rely on owner based empty leaf check, which is much faster and still works fine for most case.

And since we skip the old root owner check, now write time tree check can be merged with btrfs_check_leaf_full().

#6 lore raw
In reply to Qu Wenruo
Quoted 5 lines · Qu Wenruo

Commit "btrfs: Do mandatory tree block check before submitting bio" is designed to check leaves contents, just as read time check.

However due to the confusing parameter list, "false" is passed to where it is supposed to be "true".

And this seems to invalidate all testing since false->true will do additional expensive checks. I thought I'd fold the fix to the original patch but in this case it'd be the wrong thing to do due to the visible effects of the change.

#7 lore raw
Quoted 9 lines · David Sterba
Quoted 5 lines · Qu Wenruo

Commit "btrfs: Do mandatory tree block check before submitting bio" is designed to check leaves contents, just as read time check.

However due to the confusing parameter list, "false" is passed to where it is supposed to be "true".

And this seems to invalidate all testing since false->true will do additional expensive checks.

That's the case.

However with the next patch, btrfs_check_leaf_full() is less expensive. Just 3~5 times expensive as csum_tree_block(), reduced from 20 times.

My full fstests run is coming to the end and so far no regression.

Quoted 3 lines

I thought I'd fold the fix to the original
patch but in this case it'd be the wrong thing to do due to the visible effects of the change.

If that's the case, I'd add new Fixes: tag, but that can only happen after write time patch get merged upstream.

Thanks,
Qu

#8 lore raw
Quoted 18 lines · Qu Wenruo
Quoted 9 lines · David Sterba
Quoted 5 lines · Qu Wenruo

Commit "btrfs: Do mandatory tree block check before submitting bio" is designed to check leaves contents, just as read time check.

However due to the confusing parameter list, "false" is passed to where it is supposed to be "true".

And this seems to invalidate all testing since false->true will do additional expensive checks.

That's the case.

However with the next patch, btrfs_check_leaf_full() is less expensive. Just 3~5 times expensive as csum_tree_block(), reduced from 20 times.

Quoted 8 lines

My full fstests run is coming to the end and so far no regression.

Quoted 3 lines

I thought I'd fold the fix to the original
patch but in this case it'd be the wrong thing to do due to the visible effects of the change.

If that's the case, I'd add new Fixes: tag, but that can only happen after write time patch get merged upstream.

Fixes: to development patches should raise an alert that something's wrong.

Reviewed-by on v1 1/2
#9 lore raw
In reply to Qu Wenruo
Quoted 7 lines · Qu Wenruo

Commit "btrfs: Do mandatory tree block check before submitting bio" is designed to check leaves contents, just as read time check.

However due to the confusing parameter list, "false" is passed to where it is supposed to be "true".

The subject of the patch could be a bit more precise, by passing true you are doing further validation of the body of individual items in the leaves. Alternatively you are sanity checking the keys and the boundaries of item body. So how about:

"Perform item body validation during write"

Quoted 18 lines
---
 fs/btrfs/tree-checker.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/tree-checker.c +1 -1
index 4318e3e67443..204fe53c90aa 100644
@@ -999,7 +999,7 @@ int btrfs_check_leaf_relaxed(struct btrfs_fs_info *fs_info,
 int btrfs_check_leaf_write(struct btrfs_fs_info *fs_info,
 			   struct extent_buffer *leaf)
 {
-	return check_leaf(leaf, false, false);
+	return check_leaf(leaf, true, false);
 }
 
 int btrfs_check_node(struct btrfs_fs_info *fs_info, struct extent_buffer *node)
#10 lore raw
Quoted 9 lines · Nikolay Borisov
Quoted 6 lines · Qu Wenruo

Commit "btrfs: Do mandatory tree block check before submitting bio" is designed to check leaves contents, just as read time check.

However due to the confusing parameter list, "false" is passed to where it is supposed to be "true".

Quoted 8 lines
Quoted 1 line

The subject of the patch could be a bit more precise, by passing true you are doing further validation of the body of individual items in the leaves. Alternatively you are sanity checking the keys and the boundaries of item body. So how about:

"Perform item body validation during write"

That's much better, I'd take that tile if I need to resend.

Thanks,
Qu

Quoted 22 lines
Quoted 18 lines
---
 fs/btrfs/tree-checker.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/tree-checker.c +1 -1
index 4318e3e67443..204fe53c90aa 100644
@@ -999,7 +999,7 @@ int btrfs_check_leaf_relaxed(struct btrfs_fs_info *fs_info,
 int btrfs_check_leaf_write(struct btrfs_fs_info *fs_info,
 			   struct extent_buffer *leaf)
 {
-	return check_leaf(leaf, false, false);
+	return check_leaf(leaf, true, false);
 }
 
 int btrfs_check_node(struct btrfs_fs_info *fs_info, struct extent_buffer *node)