btrfs: More test fixes for large block/node sizes
v2 cover btrfs: More test fixes for large block/node sizes
More btrfs test fixes. These tests were ran with 64k block size and they failed. Individual patches have the details. A previous patch series with some other test fixes was posted[1].
[v1] -> v2
1. Removed the patch for generic/371 (will send it in a separate series)
2. Added RB by Filipe in generic/562
3. Added RBs by Filipe and Zorro in btrfs/200
4. For btrfs/290 - Commit message changes and changing "196608" and "65536" to "192k" and "64k" respectively in the test script to make it more readable (Suggested by Filipe).
[1] https://lore.kernel.org/all/[email protected]/
[v1] https://lore.kernel.org/all/[email protected]/
Nirjhar Roy (IBM) (3):
generic/562: Make test compatible with block sizes till 64k btrfs/200: Make the test compatible with all supported block sizes btrfs/290: Make the test compatible with all supported block sizes
tests/btrfs/200 | 8 ++++---- tests/btrfs/200.out | 8 ++++---- tests/btrfs/290 | 16 ++++++++-------- tests/generic/562 | 2 +- 4 files changed, 17 insertions(+), 17 deletions(-)
--
2.34.1
PATCH v2 1/3 generic/562: Make test compatible with block sizes till 64k
This test fails with 64k sector size in btrfs. The reason for this is the need for additional space because of COW. When the reflink/clone of file bar into file foo is done, there is no additional space left for COW - the reason is that the metadata space usage is much higher with 64k node size. In order to verify this, I instrumented the test script and disabled COW for file foo and bar and the test passes in 64k (and runs faster too).
With 64k sector and node size (COW enabled) After pwriting foo and bar and before filling up the fs
Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 324M 3.0M 100% /mnt1/scratch After filling up the fs Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 441M 3.0M 100% /mnt1/scratch
With 64k sector and node size (COW disabled) After pwriting foo and bar and before filling up the fs
Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 224M 231M 50% /mnt1/scratch After filling up the fs Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 424M 31M 94% /mnt1/scratch
As we can see, with COW, the fs is completely full after filling up the fs but with COW disabled, we have some space left.
Fix this by increasing the fs size to 590M so that even with 64k node size and COW enabled, reflink has enough space to continue.
--- tests/generic/562 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/562 +1 -1
@@ -22,7 +22,7 @@ _require_scratch_reflink _require_test_program "punch-alternating" _require_xfs_io_command "fpunch" -_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1 +_scratch_mkfs_sized $((590 * 1024 * 1024)) >>$seqres.full 2>&1 _scratch_mount file_size=$(( 200 * 1024 * 1024 )) # 200Mb
Signature · 1 line
2.34.1
PATCH v2 2/3 btrfs/200: Make the test compatible with all supported block sizes
In reply to Nirjhar Roy (IBM)This test fails on 64k block size with the following error:
At snapshot incr
OK
OK
+File foo does not have 2 shared extents in the base snapshot +/mnt/scratch/base/foo:
+ 0: [0..255]: 26624..26879
+File foo does not have 2 shared extents in the incr snapshot
So, basically after btrfs receive, the file /mnt/scratch/base/foo doesn't have any shared extents in the base snapshot. The reason is that during btrfs send, the extents are not cloned and instead they are written.
The following condition is responsible for the above behavior
in function clone_range():
/*
* Prevent cloning from a zero offset with a length matching the sector
* size because in some scenarios this will make the receiver fail. *
* For example, if in the source filesystem the extent at offset 0
* has a length of sectorsize and it was written using direct IO, then
* it can never be an inline extent (even if compression is enabled).
* Then this extent can be cloned in the original filesystem to a non
* zero file offset, but it may not be possible to clone in the
* destination filesystem because it can be inlined due to compression
* on the destination filesystem (as the receiver's write operations are
* always done using buffered IO). The same happens when the original
* filesystem does not have compression enabled but the destination
* filesystem has.
*/
if (clone_root->offset == 0 &&
len == sctx->send_root->fs_info->sectorsize)
return send_extent_data(sctx, dst_path, offset, len);
Since we are cloning from the first half [0 to 64k), clone_root->offset = 0 and len = 64k which is the sectorsize (sctx->send_root->fs_info->sectorsize). Fix this by increasing the file size and offsets to 128k so that len == sctx->send_root->fs_info->sectorsize is not true.
--- tests/btrfs/200 | 8 ++++---- tests/btrfs/200.out | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-)
tests/btrfs/200 +4 -4
@@ -38,14 +38,14 @@ _scratch_mount # Create our first test file, which has an extent that is shared only with # itself and no other files. We want to verify a full send operation will # clone the extent. -$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \ +$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 128K 0 128K" $SCRATCH_MNT/foo \ | _filter_xfs_io -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \ +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 128K 128K" $SCRATCH_MNT/foo \ | _filter_xfs_io # Create out second test file which initially, for the first send operation, # only has a single extent that is not shared. -$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \ +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 128K 0 128K" $SCRATCH_MNT/bar \ | _filter_xfs_io _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base
@@ -56,7 +56,7 @@ $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \ # Now clone the existing extent in file bar to itself at a different offset. # We want to verify the incremental send operation below will issue a clone # operation instead of a write operation. -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \ +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 128K 128K" $SCRATCH_MNT/bar \ | _filter_xfs_io _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incr
tests/btrfs/200.out +4 -4
@@ -1,12 +1,12 @@ QA output created by 200 -wrote 65536/65536 bytes at offset 0 +wrote 131072/131072 bytes at offset 0 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -linked 65536/65536 bytes at offset 65536 +linked 131072/131072 bytes at offset 131072 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 65536/65536 bytes at offset 0 +wrote 131072/131072 bytes at offset 0 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/base -linked 65536/65536 bytes at offset 65536 +linked 131072/131072 bytes at offset 131072 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/incr At subvol base
Signature · 1 line
2.34.1
PATCH v2 3/3 btrfs/290: Make the test compatible with all supported block sizes
In reply to Nirjhar Roy (IBM)This test fails with 64k block size with the following error:
punch
pread: Input/output error
pread: Input/output error
+ERROR: couldn't find extent 4096 for inode 261
plug
-pread: Input/output error
-pread: Input/output error
...
The reason is that, some of the subtests are written with 4k blocksize in mind. Fix the test by making the offsets and sizes to multiples of 64k so that it becomes compatible/aligned with all supported block sizes.
--- tests/btrfs/290 | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
tests/btrfs/290 +8 -8
@@ -106,15 +106,15 @@ corrupt_reg_to_prealloc() { # corrupt a file by punching a hole corrupt_punch_hole() { local f=$SCRATCH_MNT/punch - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 192k" $f local ino=$(get_ino $f) # make a new extent in the middle, sync so the writes don't coalesce $XFS_IO_PROG -c sync $SCRATCH_MNT - $XFS_IO_PROG -fc "pwrite -q -S 0x59 4096 4096" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x59 64k 64k" $f _fsv_enable $f _scratch_unmount # change disk_bytenr to 0, representing a hole - $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr --value 0 \ + $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr --value 0 \ $SCRATCH_DEV _scratch_mount validate $f
@@ -123,14 +123,14 @@ corrupt_punch_hole() { # plug hole corrupt_plug_hole() { local f=$SCRATCH_MNT/plug - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 192k" $f local ino=$(get_ino $f) - $XFS_IO_PROG -fc "falloc 4k 4k" $f + $XFS_IO_PROG -fc "falloc 64k 64k" $f _fsv_enable $f _scratch_unmount # change disk_bytenr to some value, plugging the hole - $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr \ - --value 13639680 $SCRATCH_DEV + $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr \ + --value 218234880 $SCRATCH_DEV _scratch_mount validate $f }
@@ -166,7 +166,7 @@ corrupt_root_hash() { # corrupt the Merkle tree data itself corrupt_merkle_tree() { local f=$SCRATCH_MNT/merkle - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 192k" $f local ino=$(get_ino $f) _fsv_enable $f _scratch_unmount
Signature · 1 line
2.34.1
Re: [PATCH v2 3/3] btrfs/290: Make the test compatible with all supported block sizes
Quoted 11 lines · Nirjhar Roy (IBM)
This test fails with 64k block size with the following error:
punch
pread: Input/output error
pread: Input/output error
+ERROR: couldn't find extent 4096 for inode 261
plug
-pread: Input/output error
-pread: Input/output error
...
Quoted 3 lines
The reason is that, some of the subtests are written with 4k blocksize in mind. Fix the test by making the offsets and sizes to multiples of 64k so that it becomes compatible/aligned with all supported block sizes.
Hi Filipe,
Do you have any additional feedback on this? I have made the changes suggested by you in [v1]
[v1] https://lore.kernel.org/all/CAL3q7H58hDCrYMqDwdO_Lf7B2J+Wdv5FpAw6u5NkDK0ExZ8K0A@mail.gmail.com/
--NR
Quoted 58 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/btrfs/290 | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)tests/btrfs/290 +8
-8@@ -106,15 +106,15 @@ corrupt_reg_to_prealloc() { # corrupt a file by punching a hole corrupt_punch_hole() { local f=$SCRATCH_MNT/punch - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 192k" $f local ino=$(get_ino $f) # make a new extent in the middle, sync so the writes don't coalesce $XFS_IO_PROG -c sync $SCRATCH_MNT - $XFS_IO_PROG -fc "pwrite -q -S 0x59 4096 4096" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x59 64k 64k" $f _fsv_enable $f _scratch_unmount # change disk_bytenr to 0, representing a hole - $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr --value 0 \ + $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr --value 0 \ $SCRATCH_DEV _scratch_mount validate $f@@ -123,14 +123,14 @@ corrupt_punch_hole() { # plug hole corrupt_plug_hole() { local f=$SCRATCH_MNT/plug - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 192k" $f local ino=$(get_ino $f) - $XFS_IO_PROG -fc "falloc 4k 4k" $f + $XFS_IO_PROG -fc "falloc 64k 64k" $f _fsv_enable $f _scratch_unmount # change disk_bytenr to some value, plugging the hole - $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr \ - --value 13639680 $SCRATCH_DEV + $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr \ + --value 218234880 $SCRATCH_DEV _scratch_mount validate $f }@@ -166,7 +166,7 @@ corrupt_root_hash() { # corrupt the Merkle tree data itself corrupt_merkle_tree() { local f=$SCRATCH_MNT/merkle - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 192k" $f local ino=$(get_ino $f) _fsv_enable $f _scratch_unmount
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Filipe Manana
Reviewed-byRe: [PATCH v2 3/3] btrfs/290: Make the test compatible with all supported block sizes
Quoted 14 lines · Nirjhar Roy (IBM)
Quoted 11 lines · Nirjhar Roy (IBM)
This test fails with 64k block size with the following error:
punch
pread: Input/output error
pread: Input/output error
+ERROR: couldn't find extent 4096 for inode 261
plug
-pread: Input/output error
-pread: Input/output error
...
Quoted 8 lines
Quoted 3 lines
The reason is that, some of the subtests are written with 4k blocksize in mind. Fix the test by making the offsets and sizes to multiples of 64k so that it becomes compatible/aligned with all supported block sizes.
Hi Filipe,
Do you have any additional feedback on this? I have made the changes suggested by you in [v1]
Looks good now, thanks.
Quoted 70 lines
[v1]
https://lore.kernel.org/all/CAL3q7H58hDCrYMqDwdO_Lf7B2J+Wdv5FpAw6u5NkDK0ExZ8K0A@mail.gmail.com/--NR
Quoted 50 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/btrfs/290 | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)tests/btrfs/290 +7
-7@@ -106,15 +106,15 @@ corrupt_reg_to_prealloc() { # corrupt a file by punching a hole corrupt_punch_hole() { local f=$SCRATCH_MNT/punch - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 192k" $f local ino=$(get_ino $f) # make a new extent in the middle, sync so the writes don't coalesce $XFS_IO_PROG -c sync $SCRATCH_MNT - $XFS_IO_PROG -fc "pwrite -q -S 0x59 4096 4096" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x59 64k 64k" $f _fsv_enable $f _scratch_unmount # change disk_bytenr to 0, representing a hole - $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr --value 0 \ + $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr --value 0 \ $SCRATCH_DEV _scratch_mount validate $f@@ -123,14 +123,14 @@ corrupt_punch_hole() { # plug hole corrupt_plug_hole() { local f=$SCRATCH_MNT/plug - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 192k" $f local ino=$(get_ino $f) - $XFS_IO_PROG -fc "falloc 4k 4k" $f + $XFS_IO_PROG -fc "falloc 64k 64k" $f _fsv_enable $f _scratch_unmount # change disk_bytenr to some value, plugging the hole - $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr \ - --value 13639680 $SCRATCH_DEV + $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr \ + --value 218234880 $SCRATCH_DEV _scratch_mount validate $f }tests/btrfs/290 corrupt_root_hash() { · line 172
Quoted 8 lines
tests/btrfs/290 +1
-1@@ -166,7 +166,7 @@ corrupt_root_hash() { # corrupt the Merkle tree data itself corrupt_merkle_tree() { local f=$SCRATCH_MNT/merkle - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 192k" $f local ino=$(get_ino $f) _fsv_enable $f _scratch_unmount--
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore