btrfs-progs: make subpage warnings more strict
cover btrfs-progs: make subpage warnings more strict
For the incoming support of more page sizes for subpage RW mount, we will require tree blocks to be nodesize aligned.
This patch prepare such restrict warnings for btrfs-check and update self-tests to handle them.
Currently all convert/fsck/misc can pass except one unrelated regression, fsck/025, which is caused by "btrfs-progs: Drop the type check in init_alloc_chunk_ctl_policy_regular".
All fsck images except fsck/018 are newer enough to have all their tree blocks nodesize aligned, so they won't cause any new warnings. But still, for read-only tests, we will skip the subpage warnings as we only want to ensure our writes from btrfs-progs won't cause new subpage warnings.
Qu Wenruo (3):
btrfs-progs: tests: also check subpage warning for type 2 test cases
btrfs-progs: tests: don't check subpage related warnings for fsck type
1 tests
btrfs-progs: require full nodesize alignement for subpage support
check/main.c | 3 ++- check/mode-common.h | 22 +++++++++++++--------- check/mode-lowmem.c | 3 ++- tests/common | 16 +++++++++++++--- tests/fsck-tests.sh | 9 ++++++++- 5 files changed, 38 insertions(+), 15 deletions(-)
Signature · 1 line
2.32.0
PATCH 1/3 btrfs-progs: tests: also check subpage warning for type 2 test cases
There are two types of test cases:
- Type 1 (without test.sh)
- Type 2 (test.sh, mostly will override check_image())
For Type 2 tests, we check subpage related warnings of btrfs-check, but didn't check it for Type 1 test cases.
In fact, Type 1 test cases are more important, as they involve repair, which can generate new tree blocks, and we want to make sure such new tree blocks won't cause subpage related warnings.
This patch will add the extra check for Type 1 test cases.
And it will make sure the subpage related warnings are really from this test case, to prevent false alerts.
--- tests/common | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
tests/common +12 -2
@@ -423,13 +423,23 @@ check_image() { local image + tmp_output=$(mktemp --tmpdir btrfs-progs-test-check-image.XXXXXX) + image=$1 echo "testing image $(basename $image)" >> "$RESULTS" - "$TOP/btrfs" check "$image" >> "$RESULTS" 2>&1 + "$TOP/btrfs" check "$image" &> "$tmp_output" [ $? -eq 0 ] && _fail "btrfs check should have detected corruption" + cat "$tmp_output" >> "$RESULTS" + # Also make sure no subpage related warnings + check_test_results "$tmp_output" "$testname" + run_check "$TOP/btrfs" check --repair --force "$image" - run_check "$TOP/btrfs" check "$image" + run_check_stdout "$TOP/btrfs" check "$image" &> "$tmp_output" + + # Also make sure no subpage related warnings for the repaired image + check_test_results "$tmp_output" "$testname" + rm -f "$tmp_output" } # Extract a usable image from packed formats
Signature · 1 line
2.32.0
Re: [PATCH 1/3] btrfs-progs: tests: also check subpage warning for type 2 test cases
Quoted 27 lines · Qu Wenruo
There are two types of test cases:
- Type 1 (without
test.sh)
- Type 2 (test.sh, mostly will overridecheck_image())For Type 2 tests, we check subpage related warnings of btrfs-check, but didn't check it for Type 1 test cases.
In fact, Type 1 test cases are more important, as they involve repair, which can generate new tree blocks, and we want to make sure such new tree blocks won't cause subpage related warnings.
This patch will add the extra check for Type 1 test cases.
And it will make sure the subpage related warnings are really from this test case, to prevent false alerts.
Signed-off-by Qu Wenruo <[email protected]>--- tests/common | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)tests/common +0
-0
tests/common check_image() · line 430
Quoted 9 lines
tests/common +3
-1@@ -423,13 +423,23 @@ check_image() { local image + tmp_output=$(mktemp --tmpdir btrfs-progs-test-check-image.XXXXXX) + image=$1 echo "testing image $(basename $image)" >> "$RESULTS" - "$TOP/btrfs" check "$image" >> "$RESULTS" 2>&1 + "$TOP/btrfs" check "$image" &> "$tmp_output"
So this captures the output but does not store it into the results file
tests/common check_image() · line 431
Quoted 1 line
tests/common +0
-0[ $? -eq 0 ] && _fail "btrfs check should have detected corruption"
and when this fails, we lack that information.
tests/common check_image() · line 433
Quoted 3 lines
tests/common +1
-0+ cat "$tmp_output" >> "$RESULTS"
So this should probably go before the 'check' call.
Quoted 15 lines
tests/common +8
-1+ # Also make sure no subpage related warnings + check_test_results "$tmp_output" "$testname" + run_check "$TOP/btrfs" check --repair --force "$image" - run_check "$TOP/btrfs" check "$image" + run_check_stdout "$TOP/btrfs" check "$image" &> "$tmp_output" + + # Also make sure no subpage related warnings for the repaired image + check_test_results "$tmp_output" "$testname" + rm -f "$tmp_output" } # Extract a usable image from packed formatsSignature · 1 line
2.32.0
Re: [PATCH 1/3] btrfs-progs: tests: also check subpage warning for type 2 test cases
In reply to Qu WenruoQuoted 27 lines · Qu Wenruo
And it will make sure the subpage related warnings are really from this test case, to prevent false alerts.
Signed-off-by Qu Wenruo <[email protected]>--- tests/common | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)tests/common +5
-1@@ -423,13 +423,23 @@ check_image() { local image + tmp_output=$(mktemp --tmpdir btrfs-progs-test-check-image.XXXXXX) + image=$1 echo "testing image $(basename $image)" >> "$RESULTS" - "$TOP/btrfs" check "$image" >> "$RESULTS" 2>&1 + "$TOP/btrfs" check "$image" &> "$tmp_output" [ $? -eq 0 ] && _fail "btrfs check should have detected corruption" + cat "$tmp_output" >> "$RESULTS" + # Also make sure no subpage related warnings
tests/common check_image() · line 443
Quoted 10 lines
tests/common +7
-1+ check_test_results "$tmp_output" "$testname" + run_check "$TOP/btrfs" check --repair --force "$image" - run_check "$TOP/btrfs" check "$image" + run_check_stdout "$TOP/btrfs" check "$image" &> "$tmp_output" + + # Also make sure no subpage related warnings for the repaired image + check_test_results "$tmp_output" "$testname" + rm -f "$tmp_output" }
Applied with the following fixup
tests/common +5 -3
@@ -422,15 +422,17 @@ check_dm_target_support() check_image() { local image + local tmp_output tmp_output=$(mktemp --tmpdir btrfs-progs-test-check-image.XXXXXX) image=$1 echo "testing image $(basename $image)" >> "$RESULTS" "$TOP/btrfs" check "$image" &> "$tmp_output" - [ $? -eq 0 ] && _fail "btrfs check should have detected corruption" - + ret=$? cat "$tmp_output" >> "$RESULTS" + [ "$ret" -eq 0 ] && _fail "btrfs check should have detected corruption" + # Also make sure no subpage related warnings check_test_results "$tmp_output" "$testname"
@@ -439,7 +441,7 @@ check_image() # Also make sure no subpage related warnings for the repaired image check_test_results "$tmp_output" "$testname" - rm -f "$tmp_output" + rm -f -- "$tmp_output" } # Extract a usable image from packed formats
PATCH 2/3 btrfs-progs: tests: don't check subpage related warnings for fsck type 1 tests
In reply to Qu WenruoFor fsck tests, we check the subpage warnings for each type 1 test, but such type 1 tests are mostly read-only tests, and one of the test will trigger new subpage related warnings (fsck/018).
For subpage related warnings, what we really care are write operations, including mkfs, btrfs-convert and repair, not those read-only tests.
So skip the subpage warning check for fsck type 1 tests to prevent false alert of later more strict subpage warnings.
--- tests/fsck-tests.sh | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
tests/fsck-tests.sh +8 -1
@@ -64,7 +64,14 @@ run_one_test() { fi _fail "test failed for case $(basename $testname)" fi - check_test_results "$RESULTS" "$testname" + # These tests have overriden check_image() and their + # images may have intentional unaligned metadata to trigger + # subpage warnings (like fsck/018), skip the check for their + # subpage warnings. + # + # We care subpage related warnings for write operations + # (mkfs/convert/repair), not those read-only checks on + # pre-crafted images. else # Type 1 check_all_images
Signature · 1 line
2.32.0
PATCH 3/3 btrfs-progs: require full nodesize alignement for subpage support
In reply to Qu WenruoFor the incoming extra page size support for subpage (sectorsize < PAGE_SIZE) cases, the support for metadata will be a critical point.
Currently for subpage support, we require 64K page size, so that no matter whatever the nodesize is, it will be contained inside one page. And we will reject any tree block which crosses page boundary.
But for other page size, especially 16K page size, we must support nodesize differently.
For nodesize < PAGE_SIZE, we will have the same requirement (tree blocks can't cross page boundary).
While for nodesize >= PAGE_SIZE, we will require the tree blocks to be page aligned.
To support such feature, we will make btrfs-check to reports more subpage related warnings for metadata.
This patch will report any tree block which is not nodesize aligned as a warning.
Existing mkfs/convert has already make sure all new tree blocks are nodesize aligned, this is just for older converted filesystems.
--- check/main.c | 3 ++- check/mode-common.h | 22 +++++++++++++--------- check/mode-lowmem.c | 3 ++- tests/common | 2 +- 4 files changed, 18 insertions(+), 12 deletions(-)
check/main.c +2 -1
@@ -5410,7 +5410,8 @@ static int process_extent_item(struct btrfs_root *root,return -EIO ; }if (metadata) -btrfs_check_subpage_eb_alignment (key.objectid, num_bytes); +btrfs_check_subpage_eb_alignment (gfs_info, key.objectid, + num_bytes);memset (&tmpl,0 ,sizeof (tmpl)); tmpl.start = key.objectid;
check/mode-common.h +13 -9
@@ -175,20 +175,24 @@ static inline u32 btrfs_type_to_imode(u8 type)int get_extent_item_generation (u64 bytenr,u64 *gen_ret);/* -* Check tree block alignment for future subpage support on 64K page system. +* Check tree block alignment for future subpage support. * -* Subpage support on 64K page size require one eb to be completely contained -* by a page. Not allowing a tree block to cross 64K page boundary. +* For subpage support, either nodesize is smaller than PAGE_SIZE, then tree +* block should not cross page boundary. (A) +* Or nodesize >= PAGE_SIZE, then it should be page aligned. (B) * -* Since subpage support is still under development, this check only provides -* warning. +* But here we have no idea the PAGE_SIZE could be, so here we play safe by +* requiring all tree blocks to be nodesize aligned. +* +* For 4K page size system, it always meets condtion (B), thus we don't need +* to bother that much. */ -static inline void btrfs_check_subpage_eb_alignment (u64 start,u32 len) +static inline void btrfs_check_subpage_eb_alignment (struct btrfs_fs_info *info, +u64 start,u32 len) { -if (start /BTRFS_MAX_METADATA_BLOCKSIZE != - (start + len -1 ) /BTRFS_MAX_METADATA_BLOCKSIZE ) +if (!IS_ALIGNED (start, info->nodesize))warning ( -"tree block [%llu, %llu) crosses 64K page boundary, may cause problem for 64K page system" , +"tree block [%llu, %llu) is not nodesize aligned, may cause problem for 64K page system" , start, start + len); }
check/mode-lowmem.c +2 -1
@@ -4265,7 +4265,8 @@ static int check_extent_item(struct btrfs_path *path) err |=CROSSING_STRIPE_BOUNDARY ; }if (metadata) -btrfs_check_subpage_eb_alignment (key.objectid, nodesize); +btrfs_check_subpage_eb_alignment (gfs_info, key.objectid, + nodesize); ptr = (unsigned long )(ei +1 );
tests/common +1 -1
@@ -850,7 +850,7 @@ check_test_results() local testname="$2" # Check subpage related warning - if grep -q "crrosses 64K page boundary" "$results"; then + if grep -q "not nodesize aligned" "$results"; then _fail "found subpage related warning for case $testname" fi }
Signature · 1 line
2.32.0
Quoted 9 lines · Qu Wenruo
For the incoming support of more page sizes for subpage RW mount, we will require tree blocks to be nodesize aligned.
This patch prepare such restrict warnings for btrfs-check and update self-tests to handle them.
Currently all convert/fsck/misc can pass except one unrelated regression, fsck/025, which is caused by "btrfs-progs: Drop the type check in
init_alloc_chunk_ctl_policy_regular".
I've dropped the test 025 for now so we can do the tests.
Quoted 11 lines
All fsck images except fsck/018 are newer enough to have all their tree blocks nodesize aligned, so they won't cause any new warnings. But still, for read-only tests, we will skip the subpage warnings as we only want to ensure our writes from btrfs-progs won't cause new subpage warnings.
Qu Wenruo (3):
btrfs-progs: tests: also check subpage warning for type 2 test cases
btrfs-progs: tests: don't check subpage related warnings for fsck type
1 tests
btrfs-progs: require full nodesize alignement for subpage support
Added to devel, thanks.