btrfs: Misc test fixes for large block/node sizes
cover btrfs: Misc test fixes for large block/node sizes
Some of the btrfs and generic tests are written with 4k block/node size in mind. This patch fixes some of those tests to be scalable with other block/node sizes too. We caught this bugs while running the auto group tests on btrfs with large block/node sizes.
Nirjhar Roy (IBM) (7):
common/filter: Add a helper function to filter offsets and sizes common/btrfs: Add a helper function to get the nodesize btrfs/137: Make this compatible with all block sizes btrfs/200: Make this test scale with the block size generic/563: Increase the write tolerance to 6% for larger nodesize btrfs/301: Make this test compatible with all block sizes. generic/274: Make the test compatible with all blocksizes.
common/btrfs | 9 +++ common/filter | 14 +++++ tests/btrfs/137 | 135 +++++++++++++++++++++++++++++--------------- tests/btrfs/137.out | 59 ++----------------- tests/btrfs/200 | 24 +++++--- tests/btrfs/200.out | 8 +-- tests/btrfs/301 | 8 ++- tests/generic/274 | 21 +++---- tests/generic/563 | 17 +++++- 9 files changed, 171 insertions(+), 124 deletions(-)
--
2.34.1
PATCH 1/7 common/filter: Add a helper function to filter offsets and sizes
_filter_xfs_io_size_offset() filters out the size and the offset emitted by various subcommands of xfs_io like pwrite.
--- common/filter | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
common/filter +14 -0
@@ -115,6 +115,20 @@ _filter_date() -e 's/[A-Z][a-z][a-z] [A-z][a-z][a-z] *[0-9][0-9]* [0-9][0-9]:[0-9][0-9]:[0-9][0-9] [0-9][0-9][0-9][0-9]$/DATE/' } +# This filters out the offsets and bytes written with +# various subcommands of xfs_io. For example +# wrote 4096/4096 bytes at offset 1052672 will be +# converted to +# wrote SIZE/SIZE bytes at offset OFFSET. +# usage: __filter_xfs_io_size_offset <offset> <size> +_filter_xfs_io_size_offset() +{ + local offset="$1" + local size="$2" + sed -e "s#${size}/${size}#SIZE/SIZE#g" \ + -e "s#offset ${offset}#offset OFFSET#g" +} + # prints filtered output on stdout, values (use eval) on stderr # Non XFS filesystems always return a 4k block size and a 256 byte inode. _filter_mkfs()
Signature · 1 line
2.34.1
PATCH 2/7 common/btrfs: Add a helper function to get the nodesize
In reply to Nirjhar Roy (IBM)Introduce a helper function to get the nodesize of the btrfs filesystem.
--- common/btrfs | 9 +++++++++ 1 file changed, 9 insertions(+)
common/btrfs +9 -0
@@ -18,6 +18,15 @@ _btrfs_get_subvolid() $BTRFS_UTIL_PROG subvolume list $mnt | grep -E "\s$name$" | $AWK_PROG '{ print $2 }' } +# returns the node size of the filesystem. +# usage _get_btrfs_node_size <device_name> +_get_btrfs_node_size() +{ + local dev=$1 + $BTRFS_UTIL_PROG inspect-internal dump-super -f "$dev"\ + | awk '/nodesize/ { print $2 }' +} + # _require_btrfs_command <command> [<subcommand>|<option>] # We check for btrfs and (optionally) features of the btrfs command # This function support both subfunction like "inspect-internal dump-tree" and
Signature · 1 line
2.34.1
PATCH 3/7 btrfs/137: Make this compatible with all block sizes
In reply to Nirjhar Roy (IBM)For large blocksizes like 64k on powerpc with 64k pagesize it failed simply because this test was written with 4k block size in mind.
The first few lines of the error logs are as follows:
d3dc847171f9081bd75d7a2d3b53d322 SCRATCH_MNT/snap2/bar
File snap1/foo fiemap results in the original filesystem:
-0: [0..7]: data
+0: [0..127]: data
File snap1/bar fiemap results in the original filesystem:
...Fix this by making the test choose offsets based on the blocksize. Also, now that the file hashes and the extent/block numbers will change depending on the blocksize, calculate the hashes and the block mappings, store them in temporary files and then calculate their diff between the new and the original filesystem.
This allows us to remove all the block mapping and hashes from the .out file.
--- tests/btrfs/137 | 135 +++++++++++++++++++++++++++++--------------- tests/btrfs/137.out | 59 ++----------------- 2 files changed, 94 insertions(+), 100 deletions(-)
tests/btrfs/137 +89 -46
@@ -27,53 +27,74 @@ _require_xfs_io_command "fiemap" send_files_dir=$TEST_DIR/btrfs-test-$seq rm -fr $send_files_dir -mkdir $send_files_dir +mkdir $send_files_dir $tmp _scratch_mkfs >>$seqres.full 2>&1 _scratch_mount +blksz=`_get_block_size $SCRATCH_MNT` +echo "block size = $blksz" >> $seqres.full + # Create the first test file. -$XFS_IO_PROG -f -c "pwrite -S 0xaa 0 4K" $SCRATCH_MNT/foo | _filter_xfs_io +$XFS_IO_PROG -f -c "pwrite -S 0xaa -b $blksz 0 $blksz" $SCRATCH_MNT/foo | _filter_xfs_io | \ + _filter_xfs_io_size_offset 0 $blksz # Create a second test file with a 1Mb hole. $XFS_IO_PROG -f \ - -c "pwrite -S 0xaa 0 4K" \ - -c "pwrite -S 0xbb 1028K 4K" \ - $SCRATCH_MNT/bar | _filter_xfs_io + -c "pwrite -S 0xaa -b $blksz 0 $blksz" \ + -c "pwrite -S 0xbb -b $blksz $(( 1024 * 1024 + blksz )) $blksz" \ + $SCRATCH_MNT/bar | _filter_xfs_io | \ + _filter_xfs_io_size_offset "$(( 1024 * 1024 + blksz ))" $blksz | \ + _filter_xfs_io_size_offset 0 $blksz $BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \ $SCRATCH_MNT/snap1 >/dev/null # Now add one new extent to our first test file, increasing its size and leaving # a 1Mb hole between the first extent and this new extent. -$XFS_IO_PROG -c "pwrite -S 0xbb 1028K 4K" $SCRATCH_MNT/foo | _filter_xfs_io +$XFS_IO_PROG -c "pwrite -S 0xbb -b $blksz $(( 1024 * 1024 + blksz )) $blksz" $SCRATCH_MNT/foo \ + | _filter_xfs_io | _filter_xfs_io_size_offset "$(( 1024 * 1024 + blksz ))" $blksz # Now overwrite the last extent of our second test file. -$XFS_IO_PROG -c "pwrite -S 0xcc 1028K 4K" $SCRATCH_MNT/bar | _filter_xfs_io +$XFS_IO_PROG -c "pwrite -S 0xcc -b $blksz $(( 1024 * 1024 + blksz )) $blksz" $SCRATCH_MNT/bar \ + | _filter_xfs_io | _filter_xfs_io_size_offset "$(( 1024 * 1024 + blksz ))" $blksz $BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \ $SCRATCH_MNT/snap2 >/dev/null -echo -echo "File digests in the original filesystem:" -md5sum $SCRATCH_MNT/snap1/foo | _filter_scratch -md5sum $SCRATCH_MNT/snap1/bar | _filter_scratch -md5sum $SCRATCH_MNT/snap2/foo | _filter_scratch -md5sum $SCRATCH_MNT/snap2/bar | _filter_scratch - -echo -echo "File snap1/foo fiemap results in the original filesystem:" -$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/foo | _filter_fiemap -echo -echo "File snap1/bar fiemap results in the original filesystem:" -$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/bar | _filter_fiemap -echo -echo "File snap2/foo fiemap results in the original filesystem:" -$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/foo | _filter_fiemap -echo -echo "File snap2/bar fiemap results in the original filesystem:" -$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/bar | _filter_fiemap -echo +echo >> $seqres.full + +echo "File digests in the original filesystem:" >> $seqres.full +md5sum $SCRATCH_MNT/snap1/foo | _filter_scratch >> $tmp/snap1_foo.original.hash +cat $tmp/snap1_foo.original.hash >> $seqres.full +md5sum $SCRATCH_MNT/snap1/bar | _filter_scratch > $tmp/snap1_bar.original.hash +cat $tmp/snap1_bar.original.hash >> $seqres.full +md5sum $SCRATCH_MNT/snap2/foo | _filter_scratch >> $tmp/snap2_foo.original.hash +cat $tmp/snap2_foo.original.hash >> $seqres.full +md5sum $SCRATCH_MNT/snap2/bar | _filter_scratch > $tmp/snap2_bar.original.hash +cat $tmp/snap2_bar.original.hash >> $seqres.full + +echo >> $seqres.full + +echo "File snap1/foo fiemap results in the original filesystem:" >> $seqres.full +$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/foo | _filter_fiemap > $tmp/snap1_foo.original.map +cat $tmp/snap1_foo.original.map >> $seqres.full +echo >> $seqres.full + +echo "File snap1/bar fiemap results in the original filesystem:" >> $seqres.full +$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/bar | _filter_fiemap > $tmp/snap1_bar.original.map +cat $tmp/snap1_bar.original.map >> $seqres.full +echo >> $seqres.full + +echo "File snap2/foo fiemap results in the original filesystem:" >> $seqres.full +$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/foo | _filter_fiemap > $tmp/snap2_foo.original.map +cat $tmp/snap2_foo.original.map >> $seqres.full +echo >> $seqres.full + +echo "File snap2/bar fiemap results in the original filesystem:" >> $seqres.full +$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/bar | _filter_fiemap > $tmp/snap2_bar.original.map +cat $tmp/snap2_bar.original.map >> $seqres.full +echo >> $seqres.full # Create the send streams to apply later on a new filesystem. $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/snap1 2>&1 \
@@ -90,25 +111,47 @@ _scratch_mount $BTRFS_UTIL_PROG receive -f $send_files_dir/1.snap $SCRATCH_MNT >/dev/null $BTRFS_UTIL_PROG receive -f $send_files_dir/2.snap $SCRATCH_MNT >/dev/null -echo -echo "File digests in the new filesystem:" -md5sum $SCRATCH_MNT/snap1/foo | _filter_scratch -md5sum $SCRATCH_MNT/snap1/bar | _filter_scratch -md5sum $SCRATCH_MNT/snap2/foo | _filter_scratch -md5sum $SCRATCH_MNT/snap2/bar | _filter_scratch - -echo -echo "File snap1/foo fiemap results in the new filesystem:" -$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/foo | _filter_fiemap -echo -echo "File snap1/bar fiemap results in the new filesystem:" -$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/bar | _filter_fiemap -echo -echo "File snap2/foo fiemap results in the new filesystem:" -$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/foo | _filter_fiemap -echo -echo "File snap2/bar fiemap results in the new filesystem:" -$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/bar | _filter_fiemap +echo >> $seqres.full +echo "File digests in the new filesystem:" >> $seqres.full +md5sum $SCRATCH_MNT/snap1/foo | _filter_scratch > $tmp/snap1_foo.new.hash +cat $tmp/snap1_foo.new.hash >> $seqres.full +md5sum $SCRATCH_MNT/snap1/bar | _filter_scratch > $tmp/snap1_bar.new.hash +cat $tmp/snap1_bar.new.hash >> $seqres.full +md5sum $SCRATCH_MNT/snap2/foo | _filter_scratch > $tmp/snap2_foo.new.hash +cat $tmp/snap2_foo.new.hash >> $seqres.full +md5sum $SCRATCH_MNT/snap2/bar | _filter_scratch > $tmp/snap2_bar.new.hash +cat $tmp/snap2_bar.new.hash >> $seqres.full + +diff $tmp/snap1_foo.new.hash $tmp/snap1_foo.original.hash +diff $tmp/snap1_bar.new.hash $tmp/snap1_bar.original.hash +diff $tmp/snap2_foo.new.hash $tmp/snap2_foo.original.hash +diff $tmp/snap2_bar.new.hash $tmp/snap2_bar.original.hash + +echo >> $seqres.full + +echo "File snap1/foo fiemap results in the new filesystem:" >> $seqres.full +$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/foo | _filter_fiemap > $tmp/snap1_foo.new.map +cat $tmp/snap1_foo.new.map >> $seqres.full +echo >> $seqres.full + +echo "File snap1/bar fiemap results in the new filesystem:" >> $seqres.full +$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/bar | _filter_fiemap > $tmp/snap1_bar.new.map +cat $tmp/snap1_bar.new.map >> $seqres.full +echo >> $seqres.full + +echo "File snap2/foo fiemap results in the new filesystem:" >> $seqres.full +$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/foo | _filter_fiemap > $tmp/snap2_foo.new.map +cat $tmp/snap2_foo.new.map >> $seqres.full +echo >> $seqres.full + +echo "File snap2/bar fiemap results in the new filesystem:" >> $seqres.full +$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/bar | _filter_fiemap > $tmp/snap2_bar.new.map +cat $tmp/snap2_bar.new.map >> $seqres.full + +diff $tmp/snap1_foo.new.map $tmp/snap1_foo.original.map +diff $tmp/snap1_bar.new.map $tmp/snap1_bar.original.map +diff $tmp/snap2_foo.new.map $tmp/snap2_foo.original.map +diff $tmp/snap2_bar.new.map $tmp/snap2_bar.original.map status=0 exit
tests/btrfs/137.out +5 -54
68 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH 3/7] btrfs/137: Make this compatible with all block sizes
Quoted 14 lines · Nirjhar Roy (IBM)
For large blocksizes like 64k on powerpc with 64k pagesize it failed simply because this test was written with 4k block size in mind.
The first few lines of the error logs are as follows:d3dc847171f9081bd75d7a2d3b53d322 SCRATCH_MNT/snap2/bar File snap1/foo fiemap results in the original filesystem: -0: [0..7]: data +0: [0..127]: data File snap1/bar fiemap results in the original filesystem: ...
Quoted 2 lines
Fix this by making the test choose offsets based on the blocksize.
I'm wondering, why not just use a fixed 64K block size?
So that all supported btrfs block sizes can result the same file contents.
Quoted 7 lines
Also, now that the file hashes and
the extent/block numbers will change depending on the blocksize, calculate the hashes and the block mappings, store them in temporary files and then calculate their diff between the new and the original filesystem.
This allows us to remove all the block mapping and hashes from the .out file.
Although I agree we should remove the block mappings from the golden output, as compression can add extra flags and pollute the golden output.
But that can also be done with _require_btrfs_no_compress() helper.
Quoted 13 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/btrfs/137 | 135 +++++++++++++++++++++++++++++--------------- tests/btrfs/137.out | 59 ++----------------- 2 files changed, 94 insertions(+), 100 deletions(-)tests/btrfs/137 +0
-0
tests/btrfs/137 _require_xfs_io_command "fiemap" · line 30
Quoted 5 lines
tests/btrfs/137 +1
-1@@ -27,53 +27,74 @@ _require_xfs_io_command "fiemap" send_files_dir=$TEST_DIR/btrfs-test-$seq rm -fr $send_files_dir -mkdir $send_files_dir +mkdir $send_files_dir $tmp
Just a small nitpick, it's more common to use $tmp.<suffix>, that's why the default _cleanup() template goes with "rm -f $tmp.*"
Thanks,
Qu
Re: [PATCH 3/7] btrfs/137: Make this compatible with all block sizes
Quoted 14 lines · Qu Wenruo
Quoted 11 lines · Nirjhar Roy (IBM)
For large blocksizes like 64k on powerpc with 64k pagesize it failed simply because this test was written with 4k block size in mind.
The first few lines of the error logs are as follows:d3dc847171f9081bd75d7a2d3b53d322 SCRATCH_MNT/snap2/bar File snap1/foo fiemap results in the original filesystem: -0: [0..7]: data +0: [0..127]: data
Quoted 7 lines
Quoted 5 lines
File snap1/bar fiemap results in the original filesystem:
...Fix this by making the test choose offsets based on the blocksize.
I'm wondering, why not just use a fixed 64K block size?
Hi Qu,
It will definitely be simpler to just use 64k io size but I feel it might be better to not hard code it for future proofing the tests. I know right now we don't have bs > ps in btrfs but maybe we get it in the future and we might start seeing funky block sizes > 64k.
Same goes for not hardcoding block mappings in the golden output.
Regards,
ojaswin
Re: [PATCH 3/7] btrfs/137: Make this compatible with all block sizes
Quoted 21 lines · Ojaswin Mujoo
Quoted 20 lines · Qu Wenruo
Quoted 16 lines · Nirjhar Roy (IBM)
For large blocksizes like 64k on powerpc with 64k pagesize it failed simply because this test was written with 4k block size in mind.
The first few lines of the error logs are as follows:d3dc847171f9081bd75d7a2d3b53d322 SCRATCH_MNT/snap2/bar File snap1/foo fiemap results in the original filesystem: -0: [0..7]: data +0: [0..127]: data File snap1/bar fiemap results in the original filesystem: ...Fix this by making the test choose offsets based on the blocksize.
Quoted 8 lines
Quoted 1 line
I'm wondering, why not just use a fixed 64K block size?
Hi Qu,
It will definitely be simpler to just use 64k io size but I feel it might be better to not hard code it for future proofing the tests. I know right now we don't have bs > ps in btrfs but maybe we get it in the future and we might start seeing funky block sizes > 64k.
And in fact I'm going to add that bs > ps support very soon, since we already have large folio support for data, it will be just a simple mapping_set_folio_order_range() with a min order.
But still for btrfs, we're only going to support 4K, 8K, 16K, 32K, 64K block sizes, thus I don't think we need to bother larger bs yet.
Thanks,
Qu
Quoted 7 lines
Same goes for not hardcoding block mappings in the golden output.
Regards,
ojaswin
Re: [PATCH 3/7] btrfs/137: Make this compatible with all block sizes
Quoted 37 lines · Qu Wenruo
Quoted 21 lines · Ojaswin Mujoo
Quoted 20 lines · Qu Wenruo
Quoted 16 lines · Nirjhar Roy (IBM)
For large blocksizes like 64k on powerpc with 64k pagesize it failed simply because this test was written with 4k block size in mind.
The first few lines of the error logs are as follows:d3dc847171f9081bd75d7a2d3b53d322 SCRATCH_MNT/snap2/bar File snap1/foo fiemap results in the original filesystem: -0: [0..7]: data +0: [0..127]: data File snap1/bar fiemap results in the original filesystem: ...Fix this by making the test choose offsets based on the blocksize.
Quoted 8 lines
Quoted 1 line
I'm wondering, why not just use a fixed 64K block size?
Hi Qu,
It will definitely be simpler to just use 64k io size but I feel it might be better to not hard code it for future proofing the tests. I know right now we don't have bs > ps in btrfs but maybe we get it in the future and we might start seeing funky block sizes > 64k.
And in fact I'm going to add that bs > ps support very soon, since we already have large folio support for data, it will be just a simple
mapping_set_folio_order_range()with a min order.
Quoted 5 lines
But still for btrfs, we're only going to support 4K, 8K, 16K, 32K, 64K block sizes, thus I don't think we need to bother larger bs yet.
Thanks,
Qu
Okay sure, if you feel 64k should be okay for considerable future then lets hardcode it for simplicity.
Regards,
ojaswin
Quoted 9 lines
Quoted 8 lines
Same goes for not hardcoding block mappings in the golden output.
Regards,
ojaswin
Re: [PATCH 3/7] btrfs/137: Make this compatible with all block sizes
In reply to Ojaswin MujooQuoted 24 lines · Ojaswin Mujoo
Quoted 14 lines · Qu Wenruo
Quoted 11 lines · Nirjhar Roy (IBM)
For large blocksizes like 64k on powerpc with 64k pagesize it failed simply because this test was written with 4k block size in mind.
The first few lines of the error logs are as follows:d3dc847171f9081bd75d7a2d3b53d322 SCRATCH_MNT/snap2/bar File snap1/foo fiemap results in the original filesystem: -0: [0..7]: data +0: [0..127]: dataQuoted 7 lines
Quoted 5 lines
File snap1/bar fiemap results in the original filesystem:
...Fix this by making the test choose offsets based on the blocksize.
I'm wondering, why not just use a fixed 64K block size?
Quoted 8 lines
Hi Qu,
It will definitely be simpler to just use 64k io size but I feel it might be better to not hard code it for future proofing the tests. I know right now we don't have bs > ps in btrfs but maybe we get it in the future and we might start seeing funky block sizes > 64k.
Same goes for not hardcoding block mappings in the golden output.
Please keep it simple and use fixed 64K aligned sizes and offsets. It's very unlikely we will support sector sizes larger than 64K, so keeping fixed sizes is a lot simpler to understand, maintain and debug tests.
Thanks.
Quoted 4 lines
Regards,
ojaswin
Re: [PATCH 3/7] btrfs/137: Make this compatible with all block sizes
Quoted 28 lines · Filipe Manana
Quoted 27 lines · Ojaswin Mujoo
Quoted 13 lines · Qu Wenruo
Quoted 11 lines · Nirjhar Roy (IBM)
For large blocksizes like 64k on powerpc with 64k pagesize it failed simply because this test was written with 4k block size in mind.
The first few lines of the error logs are as follows:d3dc847171f9081bd75d7a2d3b53d322 SCRATCH_MNT/snap2/bar File snap1/foo fiemap results in the original filesystem: -0: [0..7]: data +0: [0..127]: dataQuoted 6 lines
Quoted 5 lines
File snap1/bar fiemap results in the original filesystem:
...Fix this by making the test choose offsets based on the blocksize.
I'm wondering, why not just use a fixed 64K block size?
Hi Qu,
It will definitely be simpler to just use 64k io size but I feel it might be better to not hard code it for future proofing the tests. I know right now we don't have bs > ps in btrfs but maybe we get it in the future and we might start seeing funky block sizes > 64k.
Quoted 5 lines
Quoted 1 line
Same goes for not hardcoding block mappings in the golden output.
Please keep it simple and use fixed 64K aligned sizes and offsets. It's very unlikely we will support sector sizes larger than 64K, so keeping fixed sizes is a lot simpler to understand, maintain and debug tests.
Okay, noted.
--NR
Quoted 6 lines
Thanks.
Quoted 3 lines
Regards,
ojaswin
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Re: [PATCH 3/7] btrfs/137: Make this compatible with all block sizes
In reply to Qu WenruoQuoted 14 lines · Qu Wenruo
Quoted 11 lines · Nirjhar Roy (IBM)
For large blocksizes like 64k on powerpc with 64k pagesize it failed simply because this test was written with 4k block size in mind.
The first few lines of the error logs are as follows:d3dc847171f9081bd75d7a2d3b53d322 SCRATCH_MNT/snap2/bar
File snap1/foo fiemap results in the original filesystem:
-0: [0..7]: data
+0: [0..127]: data
Quoted 7 lines
Quoted 5 lines
File snap1/bar fiemap results in the original filesystem:
...Fix this by making the test choose offsets based on the blocksize.
I'm wondering, why not just use a fixed 64K block size?
Sorry for delayed response. Yes, if you feel keeping it simple by keeping the values aligned to 64k, I can make the change.
Quoted 11 lines
So that all supported btrfs block sizes can result the same file contents.
Quoted 7 lines
Also, now that the file hashes and
the extent/block numbers will change depending on the blocksize, calculate the hashes and the block mappings, store them in temporary files and then calculate their diff between the new and the original filesystem.
This allows us to remove all the block mapping and hashes from the .out file.
Quoted 3 lines
Although I agree we should remove the block mappings from the golden output, as compression can add extra flags and pollute the golden output.
But that can also be done with
_require_btrfs_no_compress()helper.
Okay. In that case, I will have the above pre-condition too.
Quoted 9 lines
Quoted 9 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>---
tests/btrfs/137 | 135 +++++++++++++++++++++++++++++--------------- tests/btrfs/137.out | 59 ++-----------------2 files changed, 94 insertions(+), 100 deletions(-)
Quoted 11 lines
tests/btrfs/137 _require_xfs_io_command "fiemap" · line 30
Quoted 9 lines
tests/btrfs/137 +0
-0@@ -27,53 +27,74 @@ _require_xfs_io_command "fiemap"send_files_dir=$TEST_DIR/btrfs-test-$seq
rm -fr $send_files_dirtests/btrfs/137 +1
-1-mkdir $send_files_dir +mkdir $send_files_dir $tmpJust a small nitpick, it's more common to use $tmp.<suffix>, that's why the default
_cleanup()template goes with "rm -f $tmp.*"
Noted. Thanks.
--NR
Quoted 4 lines
Thanks,
Qu
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
PATCH 4/7 btrfs/200: Make this test scale with the block size
In reply to Nirjhar Roy (IBM)For large block sizes like 64k on powerpc with 64k pagesize it failed because this test was hardcoded
to work with 4k blocksize.
With blocksize 4k and the existing file lengths,
we are getting 2 extents but with 64k page size
number of extents is not exceeding 1(due to lower
file size).
The first few lines of the error message is as follows:
At snapshot incr
OK
OK
+File foo does not have 2 shared extents in the base snapshot +/mnt/scratch/base/foo:
+ 0: [0..255]: 26624..26879
+File foo does not have 2 shared extents in the incr snapshot
...
Fix this by scaling the size and offsets to scale with the block size by a factor of (blocksize/4k).
--- tests/btrfs/200 | 24 ++++++++++++++++-------- tests/btrfs/200.out | 8 ++++---- 2 files changed, 20 insertions(+), 12 deletions(-)
tests/btrfs/200 +16 -8
@@ -35,18 +35,26 @@ mkdir $send_files_dir _scratch_mkfs >>$seqres.full 2>&1 _scratch_mount +blksz=`_get_block_size $SCRATCH_MNT` +echo "block size = $blksz" >> $seqres.full + +# Scale the test with any block size starting from 1k +scale=$(( blksz / 1024 )) +offset=$(( 16 * 1024 * scale )) +size=$(( 16 * 1024 * scale )) + # Create our first test file, which has an extent that is shared only with # itself and no other files. We want to verify a full send operation will # clone the extent. -$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \ - | _filter_xfs_io -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \ - | _filter_xfs_io +$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b $size 0 $size" $SCRATCH_MNT/foo \ + | _filter_xfs_io | _filter_xfs_io_size_offset 0 $size +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 $offset $size" $SCRATCH_MNT/foo \ + | _filter_xfs_io | _filter_xfs_io_size_offset $offset $size # Create out second test file which initially, for the first send operation, # only has a single extent that is not shared. -$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \ - | _filter_xfs_io +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $size 0 $size" $SCRATCH_MNT/bar \ + | _filter_xfs_io | _filter_xfs_io_size_offset 0 $size _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base
@@ -56,8 +64,8 @@ $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \ # Now clone the existing extent in file bar to itself at a different offset. # We want to verify the incremental send operation below will issue a clone # operation instead of a write operation. -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \ - | _filter_xfs_io +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 $offset $size" $SCRATCH_MNT/bar \ + | _filter_xfs_io | _filter_xfs_io_size_offset $offset $size _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incr
tests/btrfs/200.out +4 -4
@@ -1,12 +1,12 @@ QA output created by 200 -wrote 65536/65536 bytes at offset 0 +wrote SIZE/SIZE bytes at offset OFFSET XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -linked 65536/65536 bytes at offset 65536 +linked SIZE/SIZE bytes at offset OFFSET XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 65536/65536 bytes at offset 0 +wrote SIZE/SIZE bytes at offset OFFSET XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/base -linked 65536/65536 bytes at offset 65536 +linked SIZE/SIZE bytes at offset OFFSET XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/incr At subvol base
Signature · 1 line
2.34.1
Re: [PATCH 4/7] btrfs/200: Make this test scale with the block size
Quoted 4 lines · Nirjhar Roy (IBM)
For large block sizes like 64k on powerpc with 64k pagesize it failed because this test was hardcoded to work with 4k blocksize.
Where exactly is it hardcoded with 4K blocksize expectations?
The test does 64K writes and reflinks at offsets multiples of 64K (0 and 64K). In fact that's why the test is doing 64K writes and using only the file offsets 0 and 64K, so that it works with any block size.
Quoted 4 lines
With blocksize 4k and the existing file lengths,
we are getting 2 extents but with 64k page size
number of extents is not exceeding 1(due to lower
file size).
Due to lower file size? How?
The file sizes should be independent of the block size, and be 64K and 128K everywhere.
Please provide more details in the changelog.
Thanks.
Quoted 92 lines
The first few lines of the error message is as follows:
At snapshot incr
OK
OK
+File foo does not have 2 shared extents in the base snapshot +/mnt/scratch/base/foo:
+ 0: [0..255]: 26624..26879
+File foo does not have 2 shared extents in the incr snapshot
...Fix this by scaling the size and offsets to scale with the block size by a factor of (blocksize/4k).
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/btrfs/200 | 24 ++++++++++++++++-------- tests/btrfs/200.out | 8 ++++---- 2 files changed, 20 insertions(+), 12 deletions(-)tests/btrfs/200 +16
-8@@ -35,18 +35,26 @@ mkdir $send_files_dir _scratch_mkfs >>$seqres.full 2>&1 _scratch_mount +blksz=`_get_block_size $SCRATCH_MNT` +echo "block size = $blksz" >> $seqres.full + +# Scale the test with any block size starting from 1k +scale=$(( blksz / 1024 )) +offset=$(( 16 * 1024 * scale )) +size=$(( 16 * 1024 * scale )) + # Create our first test file, which has an extent that is shared only with # itself and no other files. We want to verify a full send operation will # clone the extent. -$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \ - | _filter_xfs_io -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \ - | _filter_xfs_io +$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b $size 0 $size" $SCRATCH_MNT/foo \ + | _filter_xfs_io | _filter_xfs_io_size_offset 0 $size +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 $offset $size" $SCRATCH_MNT/foo \ + | _filter_xfs_io | _filter_xfs_io_size_offset $offset $size # Create out second test file which initially, for the first send operation, # only has a single extent that is not shared. -$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \ - | _filter_xfs_io +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $size 0 $size" $SCRATCH_MNT/bar \ + | _filter_xfs_io | _filter_xfs_io_size_offset 0 $size _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base@@ -56,8 +64,8 @@ $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \ # Now clone the existing extent in file bar to itself at a different offset. # We want to verify the incremental send operation below will issue a clone # operation instead of a write operation. -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \ - | _filter_xfs_io +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 $offset $size" $SCRATCH_MNT/bar \ + | _filter_xfs_io | _filter_xfs_io_size_offset $offset $size _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incrtests/btrfs/200.out +4
-5@@ -1,12 +1,12 @@ QA output created by 200 -wrote 65536/65536 bytes at offset 0 +wrote SIZE/SIZE bytes at offset OFFSET XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -linked 65536/65536 bytes at offset 65536 +linked SIZE/SIZE bytes at offset OFFSET XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 65536/65536 bytes at offset 0 +wrote SIZE/SIZE bytes at offset OFFSET XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/base -linked 65536/65536 bytes at offset 65536 +linked SIZE/SIZE bytes at offset OFFSET XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/incr At subvol base --2.34.1
Re: [PATCH 4/7] btrfs/200: Make this test scale with the block size
Quoted 15 lines · Filipe Manana
Quoted 3 lines · Nirjhar Roy (IBM)
For large block sizes like 64k on powerpc with 64k pagesize it failed because this test was hardcoded to work with 4k blocksize.
Where exactly is it hardcoded with 4K blocksize expectations?
The test does 64K writes and reflinks at offsets multiples of 64K (0 and 64K). In fact that's why the test is doing 64K writes and using only the file offsets 0 and 64K, so that it works with any block size.
Quoted 4 lines
With blocksize 4k and the existing file lengths,
we are getting 2 extents but with 64k page size
number of extents is not exceeding 1(due to lower
file size).
Quoted 6 lines
Due to lower file size? How?
The file sizes should be independent of the block size, and be 64K and 128K everywhere.Please provide more details in the changelog.
Thanks.
Yes, I think I mis-interpreted the actual issue. I am looking into this. For now, I will remove this patch in the next version and once I am aware of the actual root cause, I will re-send with a proper fix and an explanation.
--NR
Quoted 93 lines
Quoted 93 lines
The first few lines of the error message is as follows:
At snapshot incr
OK
OK
+File foo does not have 2 shared extents in the base snapshot +/mnt/scratch/base/foo:
+ 0: [0..255]: 26624..26879
+File foo does not have 2 shared extents in the incr snapshot
...Fix this by scaling the size and offsets to scale with the block size by a factor of (blocksize/4k).
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/btrfs/200 | 24 ++++++++++++++++-------- tests/btrfs/200.out | 8 ++++---- 2 files changed, 20 insertions(+), 12 deletions(-)tests/btrfs/200 +16
-8@@ -35,18 +35,26 @@ mkdir $send_files_dir _scratch_mkfs >>$seqres.full 2>&1 _scratch_mount +blksz=`_get_block_size $SCRATCH_MNT` +echo "block size = $blksz" >> $seqres.full + +# Scale the test with any block size starting from 1k +scale=$(( blksz / 1024 )) +offset=$(( 16 * 1024 * scale )) +size=$(( 16 * 1024 * scale )) + # Create our first test file, which has an extent that is shared only with # itself and no other files. We want to verify a full send operation will # clone the extent. -$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \ - | _filter_xfs_io -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \ - | _filter_xfs_io +$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b $size 0 $size" $SCRATCH_MNT/foo \ + | _filter_xfs_io | _filter_xfs_io_size_offset 0 $size +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 $offset $size" $SCRATCH_MNT/foo \ + | _filter_xfs_io | _filter_xfs_io_size_offset $offset $size # Create out second test file which initially, for the first send operation, # only has a single extent that is not shared. -$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \ - | _filter_xfs_io +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $size 0 $size" $SCRATCH_MNT/bar \ + | _filter_xfs_io | _filter_xfs_io_size_offset 0 $size _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base@@ -56,8 +64,8 @@ $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \ # Now clone the existing extent in file bar to itself at a different offset. # We want to verify the incremental send operation below will issue a clone # operation instead of a write operation. -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \ - | _filter_xfs_io +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 $offset $size" $SCRATCH_MNT/bar \ + | _filter_xfs_io | _filter_xfs_io_size_offset $offset $size _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incrtests/btrfs/200.out +4
-5@@ -1,12 +1,12 @@ QA output created by 200 -wrote 65536/65536 bytes at offset 0 +wrote SIZE/SIZE bytes at offset OFFSET XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -linked 65536/65536 bytes at offset 65536 +linked SIZE/SIZE bytes at offset OFFSET XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 65536/65536 bytes at offset 0 +wrote SIZE/SIZE bytes at offset OFFSET XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/base -linked 65536/65536 bytes at offset 65536 +linked SIZE/SIZE bytes at offset OFFSET XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/incr At subvol base --2.34.1
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Re: [PATCH 4/7] btrfs/200: Make this test scale with the block size
In reply to Nirjhar Roy (IBM)Quoted 17 lines · Nirjhar Roy (IBM)
For large block sizes like 64k on powerpc with 64k pagesize it failed because this test was hardcoded
to work with 4k blocksize.
With blocksize 4k and the existing file lengths,
we are getting 2 extents but with 64k page size
number of extents is not exceeding 1(due to lower
file size).
The first few lines of the error message is as follows:
At snapshot incr
OK
OK
+File foo does not have 2 shared extents in the base snapshot +/mnt/scratch/base/foo:
+ 0: [0..255]: 26624..26879
+File foo does not have 2 shared extents in the incr snapshot
...
Quoted 2 lines
Fix this by scaling the size and offsets to scale with the block size by a factor of (blocksize/4k).
Although I can reproduce the bug on aarch64 with 64K page size, the changelog doesn't seem to explain the problem.
The problem is after receive, the file base/foo is a single extent, not the original reflinked two extent.
And the original fs is indeed have the correct two extents layout.
That different extent layout is definitely not explained properly in the commit message.
Furthermore the test case is already using 64K block size for extents creation, thus all supported block size should work.
So I think the change doesn't really touch the core of the failure.
Thanks,
Qu
Quoted 77 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/btrfs/200 | 24 ++++++++++++++++-------- tests/btrfs/200.out | 8 ++++---- 2 files changed, 20 insertions(+), 12 deletions(-)tests/btrfs/200 +16
-8@@ -35,18 +35,26 @@ mkdir $send_files_dir _scratch_mkfs >>$seqres.full 2>&1 _scratch_mount +blksz=`_get_block_size $SCRATCH_MNT` +echo "block size = $blksz" >> $seqres.full + +# Scale the test with any block size starting from 1k +scale=$(( blksz / 1024 )) +offset=$(( 16 * 1024 * scale )) +size=$(( 16 * 1024 * scale )) + # Create our first test file, which has an extent that is shared only with # itself and no other files. We want to verify a full send operation will # clone the extent. -$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \ - | _filter_xfs_io -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \ - | _filter_xfs_io +$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b $size 0 $size" $SCRATCH_MNT/foo \ + | _filter_xfs_io | _filter_xfs_io_size_offset 0 $size +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 $offset $size" $SCRATCH_MNT/foo \ + | _filter_xfs_io | _filter_xfs_io_size_offset $offset $size # Create out second test file which initially, for the first send operation, # only has a single extent that is not shared. -$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \ - | _filter_xfs_io +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $size 0 $size" $SCRATCH_MNT/bar \ + | _filter_xfs_io | _filter_xfs_io_size_offset 0 $size _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base@@ -56,8 +64,8 @@ $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \ # Now clone the existing extent in file bar to itself at a different offset. # We want to verify the incremental send operation below will issue a clone # operation instead of a write operation. -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \ - | _filter_xfs_io +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 $offset $size" $SCRATCH_MNT/bar \ + | _filter_xfs_io | _filter_xfs_io_size_offset $offset $size _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incrtests/btrfs/200.out +4
-4@@ -1,12 +1,12 @@ QA output created by 200 -wrote 65536/65536 bytes at offset 0 +wrote SIZE/SIZE bytes at offset OFFSET XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -linked 65536/65536 bytes at offset 65536 +linked SIZE/SIZE bytes at offset OFFSET XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 65536/65536 bytes at offset 0 +wrote SIZE/SIZE bytes at offset OFFSET XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/base -linked 65536/65536 bytes at offset 65536 +linked SIZE/SIZE bytes at offset OFFSET XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/incr At subvol base
PATCH 5/7 generic/563: Increase the write tolerance to 6% for larger nodesize
In reply to Nirjhar Roy (IBM)When tested with blocksize/nodesize 64K on powerpc with 64k pagesize on btrfs, then the test fails with the folllowing error:
QA output created by 563
read/write
read is in range
-write is in range
+write has value of 8855552
+write is NOT in range 7969177.6 .. 8808038.4
write -> read/write
...The slight increase in the amount of bytes that
are written is because of the increase in the
the nodesize(metadata) and hence it exceeds the tolerance limit slightly. Fix this by increasing the write tolerance limit from 5% from 6% for 64k blocksize btrfs.
--- tests/generic/563 | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
tests/generic/563 +16 -1
@@ -119,7 +119,22 @@ $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \ $SCRATCH_MNT/file >> $seqres.full 2>&1 switch_cg $cgdir $XFS_IO_PROG -c fsync $SCRATCH_MNT/file -check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% +blksz=`_get_block_size $SCRATCH_MNT` + +# On higher node sizes on btrfs, we observed slightly more +# writes, due to increased metadata sizes. +# Hence have a higher write tolerance for btrfs and when +# node size is greater than 4k. +if [[ "$FSTYP" == "btrfs" ]]; then + nodesz=$(_get_btrfs_node_size "$SCRATCH_DEV") + if [[ "$nodesz" -gt 4096 ]]; then + check_cg $cgdir/$seq-cg $iosize $iosize 5% 6% + else + check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% + fi +else + check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% +fi # Write from one cgroup then read and write from a second. Writes are charged to # the first group and nothing to the second.
Signature · 1 line
2.34.1
Re: [PATCH 5/7] generic/563: Increase the write tolerance to 6% for larger nodesize
tests/generic/563 $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \ · line 137
Quoted 13 lines · Nirjhar Roy (IBM)
tests/generic/563 +13
-0+# writes, due to increased metadata sizes. +# Hence have a higher write tolerance for btrfs and when +# node size is greater than 4k. +if [[ "$FSTYP" == "btrfs" ]]; then + nodesz=$(_get_btrfs_node_size "$SCRATCH_DEV") + if [[ "$nodesz" -gt 4096 ]]; then + check_cg $cgdir/$seq-cg $iosize $iosize 5% 6% + else + check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% + fi +else + check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% +fi
Everyone please stop hacking file system specific things into generic tests. Add proper core helpers for this instead.
Re: [PATCH 5/7] generic/563: Increase the write tolerance to 6% for larger nodesize
Quoted 16 lines · Christoph Hellwig
tests/generic/563 $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \ · line 137
Quoted 13 lines · Nirjhar Roy (IBM)
tests/generic/563 +13
-0+# writes, due to increased metadata sizes. +# Hence have a higher write tolerance for btrfs and when +# node size is greater than 4k. +if [[ "$FSTYP" == "btrfs" ]]; then + nodesz=$(_get_btrfs_node_size "$SCRATCH_DEV") + if [[ "$nodesz" -gt 4096 ]]; then + check_cg $cgdir/$seq-cg $iosize $iosize 5% 6% + else + check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% + fi +else + check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% +fiEveryone please stop hacking file system specific things into generic tests. Add proper core helpers for this instead.
Okay. I will try to work something out. Thank you.
--NR
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Re: [PATCH 5/7] generic/563: Increase the write tolerance to 6% for larger nodesize
In reply to Nirjhar Roy (IBM)Quoted 29 lines · Nirjhar Roy (IBM)
When tested with blocksize/nodesize 64K on powerpc with 64k pagesize on btrfs, then the test fails with the folllowing error:
QA output created by 563 read/write read is in range -write is in range +write has value of 8855552 +write is NOT in range 7969177.6 .. 8808038.4 write -> read/write ...The slight increase in the amount of bytes that
are written is because of the increase in the
the nodesize(metadata) and hence it exceeds the tolerance limit slightly. Fix this by increasing the write tolerance limit from 5% from 6% for 64k blocksize btrfs.Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/generic/563 | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)tests/generic/563 +0
-0
tests/generic/563 $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \ · line 122
Quoted 5 lines
tests/generic/563 +1
-1@@ -119,7 +119,22 @@ $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \ $SCRATCH_MNT/file >> $seqres.full 2>&1 switch_cg $cgdir $XFS_IO_PROG -c fsync $SCRATCH_MNT/file -check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% +blksz=`_get_block_size $SCRATCH_MNT`
Here the block size is captured, but then it's not used anywhere in the test...
Thanks.
Quoted 21 lines
tests/generic/563 +15
-1+ +# On higher node sizes on btrfs, we observed slightly more +# writes, due to increased metadata sizes. +# Hence have a higher write tolerance for btrfs and when +# node size is greater than 4k. +if [[ "$FSTYP" == "btrfs" ]]; then + nodesz=$(_get_btrfs_node_size "$SCRATCH_DEV") + if [[ "$nodesz" -gt 4096 ]]; then + check_cg $cgdir/$seq-cg $iosize $iosize 5% 6% + else + check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% + fi +else + check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% +fi # Write from one cgroup then read and write from a second. Writes are charged to # the first group and nothing to the second. --2.34.1
Re: [PATCH 5/7] generic/563: Increase the write tolerance to 6% for larger nodesize
Quoted 25 lines · Filipe Manana
Quoted 23 lines · Nirjhar Roy (IBM)
When tested with blocksize/nodesize 64K on powerpc with 64k pagesize on btrfs, then the test fails with the folllowing error:
QA output created by 563 read/write read is in range -write is in range +write has value of 8855552 +write is NOT in range 7969177.6 .. 8808038.4 write -> read/write ...The slight increase in the amount of bytes that
are written is because of the increase in the
the nodesize(metadata) and hence it exceeds the tolerance limit slightly. Fix this by increasing the write tolerance limit from 5% from 6% for 64k blocksize btrfs.Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/generic/563 | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
Quoted 11 lines
tests/generic/563 $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \ · line 122
Quoted 10 lines
tests/generic/563 +1
-1@@ -119,7 +119,22 @@ $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \ $SCRATCH_MNT/file >> $seqres.full 2>&1 switch_cg $cgdir $XFS_IO_PROG -c fsync $SCRATCH_MNT/file -check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% +blksz=`_get_block_size $SCRATCH_MNT`Here the block size is captured, but then it's not used anywhere in the test...
Yeah. I will remove this. Thanks.
--NR
Quoted 24 lines
Thanks.
Quoted 21 lines
tests/generic/563 +15
-1+ +# On higher node sizes on btrfs, we observed slightly more +# writes, due to increased metadata sizes. +# Hence have a higher write tolerance for btrfs and when +# node size is greater than 4k. +if [[ "$FSTYP" == "btrfs" ]]; then + nodesz=$(_get_btrfs_node_size "$SCRATCH_DEV") + if [[ "$nodesz" -gt 4096 ]]; then + check_cg $cgdir/$seq-cg $iosize $iosize 5% 6% + else + check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% + fi +else + check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% +fi # Write from one cgroup then read and write from a second. Writes are charged to # the first group and nothing to the second. --2.34.1
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Re: [PATCH 5/7] generic/563: Increase the write tolerance to 6% for larger nodesize
In reply to Nirjhar Roy (IBM)Quoted 9 lines · Nirjhar Roy (IBM)
When tested with blocksize/nodesize 64K on powerpc with 64k pagesize on btrfs, then the test fails with the folllowing error:
QA output created by 563
read/write
read is in range
-write is in range
+write has value of 8855552
+write is NOT in range 7969177.6 .. 8808038.4
I can reproduce the failure, although it's not 100% reliable, and indeed with one tree block's size removed, it's back into the tolerance range.
Quoted 29 lines
write -> read/write ...The slight increase in the amount of bytes that
are written is because of the increase in the
the nodesize(metadata) and hence it exceeds the tolerance limit slightly. Fix this by increasing the write tolerance limit from 5% from 6% for 64k blocksize btrfs.Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/generic/563 | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)tests/generic/563 +6
-1@@ -119,7 +119,22 @@ $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \ $SCRATCH_MNT/file >> $seqres.full 2>&1 switch_cg $cgdir $XFS_IO_PROG -c fsync $SCRATCH_MNT/file -check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% +blksz=`_get_block_size $SCRATCH_MNT` + +# On higher node sizes on btrfs, we observed slightly more +# writes, due to increased metadata sizes. +# Hence have a higher write tolerance for btrfs and when +# node size is greater than 4k.
tests/generic/563 $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \ · line 137
Quoted 10 lines
tests/generic/563 +10
-0+if [[ "$FSTYP" == "btrfs" ]]; then + nodesz=$(_get_btrfs_node_size "$SCRATCH_DEV") + if [[ "$nodesz" -gt 4096 ]]; then + check_cg $cgdir/$seq-cg $iosize $iosize 5% 6% + else + check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% + fi +else + check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% +fi
Instead of the btrfs specific hack, I'd recommend to just enlarge iosize.
Double the iosize will easily make it to cover the tolerance of even btrfs, but you still need a proper explanation of the change.
Thanks,
Qu
Quoted 3 lines
tests/generic/563 +0
-0# Write from one cgroup then read and write from a second. Writes are charged to # the first group and nothing to the second.
Re: [PATCH 5/7] generic/563: Increase the write tolerance to 6% for larger nodesize
Quoted 58 lines · Qu Wenruo
Quoted 9 lines · Nirjhar Roy (IBM)
When tested with blocksize/nodesize 64K on powerpc with 64k pagesize on btrfs, then the test fails with the folllowing error:
QA output created by 563
read/write
read is in range
-write is in range
+write has value of 8855552
+write is NOT in range 7969177.6 .. 8808038.4I can reproduce the failure, although it's not 100% reliable, and indeed with one tree block's size removed, it's back into the tolerance range.
Quoted 30 lines
write -> read/write
...
The slight increase in the amount of bytes that
are written is because of the increase in the
the nodesize(metadata) and hence it exceeds the tolerance limit slightly. Fix this by increasing the write tolerance limit from 5% from 6% for 64k blocksize btrfs.Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>---
tests/generic/563 | 17 ++++++++++++++++-1 file changed, 16 insertions(+), 1 deletion(-)tests/generic/563 +0
-0@@ -119,7 +119,22 @@ $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \$SCRATCH_MNT/file >> $seqres.full 2>&1
switch_cg$cgdir
$XFS_IO_PROG -c fsync $SCRATCH_MNT/filetests/generic/563 +6
-1-check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% +blksz=`_get_block_size $SCRATCH_MNT` + +# On higher node sizes on btrfs, we observed slightly more +# writes, due to increased metadata sizes. +# Hence have a higher write tolerance for btrfs and when +# node size is greater than 4k.tests/generic/563 $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \ · line 137
Quoted 10 lines
tests/generic/563 +10
-0+if [[ "$FSTYP" == "btrfs" ]]; then + nodesz=$(_get_btrfs_node_size "$SCRATCH_DEV") + if [[ "$nodesz" -gt 4096 ]]; then + check_cg $cgdir/$seq-cg $iosize $iosize 5% 6% + else + check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% + fi +else + check_cg $cgdir/$seq-cg $iosize $iosize 5% 5% +fiInstead of the btrfs specific hack, I'd recommend to just enlarge iosize.
Quoted 1 line
Double the iosize will easily make it to cover the tolerance of even btrfs, but you still need a proper explanation of the change.
Okay. I can try the above and will come up with more detailed explantion.
--NR
Quoted 7 lines
Thanks,
QuQuoted 2 lines
# Write from one cgroup then read and write from a second. Writes are charged to
# the first group and nothing to the second.
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
PATCH 6/7 btrfs/301: Make this test compatible with all block sizes.
In reply to Nirjhar Roy (IBM)With large block sizes like 64k on powerpc with 64k pagesize the test failed with the following logs:
QA output created by 301
basic accounting
+subvol 256 mismatched usage 33947648 vs 4587520 \
(expected data 4194304 expected meta 393216 diff 29360128)
+subvol 256 mismatched usage 168165376 vs 138805248 \
(expected data 138412032 expected meta 393216 diff 29360128)
+subvol 256 mismatched usage 33947648 vs 4587520 \
(expected data 4194304 expected meta 393216 diff 29360128)
+subvol 256 mismatched usage 33947648 vs 4587520 \
(expected data 4194304 expected meta 393216 diff 29360128)
fallocate: Disk quota exceeded(Please note that the above ouptut had to be modified a bit since the number of characters in each line was much greater than the 72 characters.)
The test creates nr_fill files each of size 8k i.e, 2x4k(stored in fill_sz). Now with 64k blocksize, 8k sized files occupy more than expected sizes (i.e, 8k) due to internal fragmentation since 1 file
will occupy at least 1 block. Fix this by scaling the file size (fill_sz) with the blocksize.
--- tests/btrfs/301 | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
tests/btrfs/301 +7 -1
@@ -23,7 +23,13 @@ subv=$SCRATCH_MNT/subv nested=$SCRATCH_MNT/subv/nested snap=$SCRATCH_MNT/snap nr_fill=512 -fill_sz=$((8 * 1024)) + +_scratch_mkfs >> $seqres.full +_scratch_mount +blksz=`_get_block_size $SCRATCH_MNT` +_scratch_unmount +fill_sz=$(( 2 * blksz )) + total_fill=$(($nr_fill * $fill_sz)) nodesize=$($BTRFS_UTIL_PROG inspect-internal dump-super $SCRATCH_DEV | \ grep nodesize | $AWK_PROG '{print $2}')
Signature · 1 line
2.34.1
Re: [PATCH 6/7] btrfs/301: Make this test compatible with all block sizes.
Quoted 14 lines · Nirjhar Roy (IBM)
With large block sizes like 64k on powerpc with 64k pagesize the test failed with the following logs:
QA output created by 301 basic accounting +subvol 256 mismatched usage 33947648 vs 4587520 \ (expected data 4194304 expected meta 393216 diff 29360128) +subvol 256 mismatched usage 168165376 vs 138805248 \ (expected data 138412032 expected meta 393216 diff 29360128) +subvol 256 mismatched usage 33947648 vs 4587520 \ (expected data 4194304 expected meta 393216 diff 29360128) +subvol 256 mismatched usage 33947648 vs 4587520 \ (expected data 4194304 expected meta 393216 diff 29360128) fallocate: Disk quota exceeded
Quoted 3 lines
(Please note that the above ouptut had to be modified a bit since the number of characters in each line was much greater than the 72 characters.)
You don't need to break the line for raw output.
Quoted 6 lines
The test creates
nr_fillfiles each of size 8k i.e, 2x4k(stored infill_sz). Now with 64k blocksize, 8k sized files occupy more than expected sizes (i.e, 8k) due to internal fragmentation since 1 file
will occupy at least 1 block. Fix this by scaling the file size (fill_sz) with the blocksize.
You can just replace the fill_sz to 64K so that all block sizes will work.
Just tested with 64K fill_sz, it works for both 4K and 64K block size with 64K page size.
Thanks,
Qu
Quoted 26 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/btrfs/301 | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)tests/btrfs/301 +7
-1@@ -23,7 +23,13 @@ subv=$SCRATCH_MNT/subv nested=$SCRATCH_MNT/subv/nested snap=$SCRATCH_MNT/snap nr_fill=512 -fill_sz=$((8 * 1024)) + +_scratch_mkfs >> $seqres.full +_scratch_mount +blksz=`_get_block_size $SCRATCH_MNT` +_scratch_unmount +fill_sz=$(( 2 * blksz )) + total_fill=$(($nr_fill * $fill_sz)) nodesize=$($BTRFS_UTIL_PROG inspect-internal dump-super $SCRATCH_DEV | \ grep nodesize | $AWK_PROG '{print $2}')
Re: [PATCH 6/7] btrfs/301: Make this test compatible with all block sizes.
Quoted 21 lines · Qu Wenruo
Quoted 17 lines · Nirjhar Roy (IBM)
With large block sizes like 64k on powerpc with 64k pagesize the test failed with the following logs:
QA output created by 301
basic accounting
+subvol 256 mismatched usage 33947648 vs 4587520 \
(expected data 4194304 expected meta 393216 diff 29360128) +subvol 256 mismatched usage 168165376 vs 138805248 \
(expected data 138412032 expected meta 393216 diff 29360128) +subvol 256 mismatched usage 33947648 vs 4587520 \
(expected data 4194304 expected meta 393216 diff 29360128) +subvol 256 mismatched usage 33947648 vs 4587520 \
(expected data 4194304 expected meta 393216 diff 29360128)
fallocate: Disk quota exceeded
(Please note that the above ouptut had to be modified a bit since the number of characters in each line was much greater than the 72 characters.)
Quoted 1 line
You don't need to break the line for raw output.
Noted.
Quoted 11 lines
Quoted 7 lines
The test creates
nr_fillfiles each of size 8k i.e, 2x4k(stored infill_sz). Now with 64k blocksize, 8k sized files occupy more than expected sizes (i.e, 8k) due to internal fragmentation since 1 file
will occupy at least 1 block. Fix this by scaling the file size (fill_sz) with the blocksize.You can just replace the
fill_szto 64K so that all block sizes will work.Just tested with 64K
fill_sz, it works for both 4K and 64K block size with 64K page size.
Okay, I will make the test work with 64k aligned values (similar to suggestions to the previous fixes in this patch series).
--NR
Quoted 31 lines
Thanks,
QuQuoted 26 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>---
tests/btrfs/301 | 8 +++++++-1 file changed, 7 insertions(+), 1 deletion(-)tests/btrfs/301 +0
-0@@ -23,7 +23,13 @@ subv=$SCRATCH_MNT/subvnested=$SCRATCH_MNT/subv/nested
snap=$SCRATCH_MNT/snap
nr_fill=512tests/btrfs/301 +7
-1-fill_sz=$((8 * 1024)) + +_scratch_mkfs >> $seqres.full +_scratch_mount +blksz=`_get_block_size $SCRATCH_MNT` +_scratch_unmount +fill_sz=$(( 2 * blksz )) +total_fill=$(($nr_fill * $fill_sz)) nodesize=$($BTRFS_UTIL_PROG inspect-internal dump-super $SCRATCH_DEV | \
grep nodesize | $AWK_PROG '{print $2}')
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
PATCH 7/7 generic/274: Make the test compatible with all blocksizes.
In reply to Nirjhar Roy (IBM)On btrfs with 64k blocksize on powerpc with 64k pagesize it failed with the following error:
------------------------------
preallocation test
------------------------------
-done
+failed to write to test file
+(see /home/xfstests-dev/results//btrfs_64k/generic/274.full for details)
...So, this test is written with 4K block size in mind. As we can see, it first creates a file of size 4K and then fallocates 4MB beyond the EOF.
Then there are 2 loops - one that fragments at alternate blocks and the other punches holes in the remaining alternate blocks. Hence, the test fails in 64k block size due to incorrect calculations.
Fix this test by making the test scale with the block size, that is the offset, filesize and the assumed blocksize matches/scales with the actual blocksize of the underlying filesystem.
--- tests/generic/274 | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-)
tests/generic/274 +11 -10
@@ -40,30 +40,31 @@ _scratch_unmount 2>/dev/null _scratch_mkfs_sized $((2 * 1024 * 1024 * 1024)) >>$seqres.full 2>&1 _scratch_mount -# Create a 4k file and Allocate 4M past EOF on that file -$XFS_IO_PROG -f -c "pwrite 0 4k" -c "falloc -k 4k 4m" $SCRATCH_MNT/test \ - >>$seqres.full 2>&1 || _fail "failed to create test file" +blksz=`_get_block_size $SCRATCH_MNT` +scale=$(( blksz / 1024 )) +# Create a blocksize worth file and Allocate a large file past EOF on that file +$XFS_IO_PROG -f -c "pwrite -b $blksz 0 $blksz" -c "falloc -k $blksz $(( 1 * 1024 * 1024 * scale ))" \ + $SCRATCH_MNT/test >>$seqres.full 2>&1 || _fail "failed to create test file" # Fill the rest of the fs completely # Note, this will show ENOSPC errors in $seqres.full, that's ok. echo "Fill fs with 1M IOs; ENOSPC expected" >> $seqres.full dd if=/dev/zero of=$SCRATCH_MNT/tmp1 bs=1M >>$seqres.full 2>&1 -echo "Fill fs with 4K IOs; ENOSPC expected" >> $seqres.full -dd if=/dev/zero of=$SCRATCH_MNT/tmp2 bs=4K >>$seqres.full 2>&1 +echo "Fill fs with $blksz K IOs; ENOSPC expected" >> $seqres.full +dd if=/dev/zero of=$SCRATCH_MNT/tmp2 bs=$blksz >>$seqres.full 2>&1 _scratch_sync # Last effort, use O_SYNC -echo "Fill fs with 4K DIOs; ENOSPC expected" >> $seqres.full -dd if=/dev/zero of=$SCRATCH_MNT/tmp3 bs=4K oflag=sync >>$seqres.full 2>&1 +echo "Fill fs with $blksz DIOs; ENOSPC expected" >> $seqres.full +dd if=/dev/zero of=$SCRATCH_MNT/tmp3 bs=$blksz oflag=sync >>$seqres.full 2>&1 # Save space usage info echo "Post-fill space:" >> $seqres.full df $SCRATCH_MNT >>$seqres.full 2>&1 - # Now attempt a write into all of the preallocated space - # in a very nasty way, badly fragmenting it and then filling it in. echo "Fill in prealloc space; fragment at offsets:" >> $seqres.full for i in `seq 1 2 1023`; do echo -n "$i " >> $seqres.full - dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=4K count=1 conv=notrunc \ + dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=$blksz count=1 conv=notrunc \ >>$seqres.full 2>/dev/null || _fail "failed to write to test file" done _scratch_sync
@@ -71,7 +72,7 @@ echo >> $seqres.full echo "Fill in prealloc space; fill holes at offsets:" >> $seqres.full for i in `seq 2 2 1023`; do echo -n "$i " >> $seqres.full - dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=4K count=1 conv=notrunc \ + dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=$blksz count=1 conv=notrunc \ >>$seqres.full 2>/dev/null || _fail "failed to fill test file" done _scratch_sync
Signature · 1 line
2.34.1
Re: [PATCH 7/7] generic/274: Make the test compatible with all blocksizes.
Quoted 17 lines · Nirjhar Roy (IBM)
On btrfs with 64k blocksize on powerpc with 64k pagesize it failed with the following error:
------------------------------ preallocation test ------------------------------ -done +failed to write to test file +(see /home/xfstests-dev/results//btrfs_64k/generic/274.full for details) ...So, this test is written with 4K block size in mind. As we can see, it first creates a file of size 4K and then fallocates 4MB beyond the EOF.
Then there are 2 loops - one that fragments at alternate blocks and the other punches holes in the remaining alternate blocks. Hence, the test fails in 64k block size due to incorrect calculations.
Quoted 3 lines
Fix this test by making the test scale with the block size, that is the offset, filesize and the assumed blocksize matches/scales with the actual blocksize of the underlying filesystem.
Again, just enlarge the block size from 4K to 64K, then all block size will work.
Thanks,
Qu
Quoted 61 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/generic/274 | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-)tests/generic/274 +11
-10@@ -40,30 +40,31 @@ _scratch_unmount 2>/dev/null _scratch_mkfs_sized $((2 * 1024 * 1024 * 1024)) >>$seqres.full 2>&1 _scratch_mount -# Create a 4k file and Allocate 4M past EOF on that file -$XFS_IO_PROG -f -c "pwrite 0 4k" -c "falloc -k 4k 4m" $SCRATCH_MNT/test \ - >>$seqres.full 2>&1 || _fail "failed to create test file" +blksz=`_get_block_size $SCRATCH_MNT` +scale=$(( blksz / 1024 )) +# Create a blocksize worth file and Allocate a large file past EOF on that file +$XFS_IO_PROG -f -c "pwrite -b $blksz 0 $blksz" -c "falloc -k $blksz $(( 1 * 1024 * 1024 * scale ))" \ + $SCRATCH_MNT/test >>$seqres.full 2>&1 || _fail "failed to create test file" # Fill the rest of the fs completely # Note, this will show ENOSPC errors in $seqres.full, that's ok. echo "Fill fs with 1M IOs; ENOSPC expected" >> $seqres.full dd if=/dev/zero of=$SCRATCH_MNT/tmp1 bs=1M >>$seqres.full 2>&1 -echo "Fill fs with 4K IOs; ENOSPC expected" >> $seqres.full -dd if=/dev/zero of=$SCRATCH_MNT/tmp2 bs=4K >>$seqres.full 2>&1 +echo "Fill fs with $blksz K IOs; ENOSPC expected" >> $seqres.full +dd if=/dev/zero of=$SCRATCH_MNT/tmp2 bs=$blksz >>$seqres.full 2>&1 _scratch_sync # Last effort, use O_SYNC -echo "Fill fs with 4K DIOs; ENOSPC expected" >> $seqres.full -dd if=/dev/zero of=$SCRATCH_MNT/tmp3 bs=4K oflag=sync >>$seqres.full 2>&1 +echo "Fill fs with $blksz DIOs; ENOSPC expected" >> $seqres.full +dd if=/dev/zero of=$SCRATCH_MNT/tmp3 bs=$blksz oflag=sync >>$seqres.full 2>&1 # Save space usage info echo "Post-fill space:" >> $seqres.full df $SCRATCH_MNT >>$seqres.full 2>&1 - # Now attempt a write into all of the preallocated space - # in a very nasty way, badly fragmenting it and then filling it in. echo "Fill in prealloc space; fragment at offsets:" >> $seqres.full for i in `seq 1 2 1023`; do echo -n "$i " >> $seqres.full - dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=4K count=1 conv=notrunc \ + dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=$blksz count=1 conv=notrunc \ >>$seqres.full 2>/dev/null || _fail "failed to write to test file" done _scratch_sync@@ -71,7 +72,7 @@ echo >> $seqres.full echo "Fill in prealloc space; fill holes at offsets:" >> $seqres.full for i in `seq 2 2 1023`; do echo -n "$i " >> $seqres.full - dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=4K count=1 conv=notrunc \ + dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=$blksz count=1 conv=notrunc \ >>$seqres.full 2>/dev/null || _fail "failed to fill test file" done _scratch_sync
Re: [PATCH 7/7] generic/274: Make the test compatible with all blocksizes.
Quoted 20 lines · Qu Wenruo
Quoted 17 lines · Nirjhar Roy (IBM)
On btrfs with 64k blocksize on powerpc with 64k pagesize it failed with the following error:
------------------------------
preallocation test
------------------------------
-done
+failed to write to test file
+(see/home/xfstests-dev/results//btrfs_64k/generic/274.fullfor details)
...
So, this test is written with 4K block size in mind. As we can see, it first creates a file of size 4K and then fallocates 4MB beyond the EOF.
Then there are 2 loops - one that fragments at alternate blocks and the other punches holes in the remaining alternate blocks. Hence, the test fails in 64k block size due to incorrect calculations.
Quoted 5 lines
Quoted 3 lines
Fix this test by making the test scale with the block size, that is the offset, filesize and the assumed blocksize matches/scales with the actual blocksize of the underlying filesystem.
Again, just enlarge the block size from 4K to 64K, then all block size will work.
Okay.
--NR
Quoted 74 lines
Thanks,
QuQuoted 69 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>---
tests/generic/274 | 21 +++++++++++----------1 file changed, 11 insertions(+), 10 deletions(-)tests/generic/274 +0
-0@@ -40,30 +40,31 @@ _scratch_unmount 2>/dev/null
_scratch_mkfs_sized$((2 * 1024 * 1024 * 1024)) >>$seqres.full 2>&1_scratch_mount
-# Create a 4k file and Allocate 4M past EOF on that file -$XFS_IO_PROG -f -c "pwrite 0 4k" -c "falloc -k 4k 4m" $SCRATCH_MNT/test \
- >>$seqres.full 2>&1 ||_fail"failed to create test file"tests/generic/274 +3
-0+blksz=`_get_block_size $SCRATCH_MNT` +scale=$(( blksz / 1024 )) +# Create a blocksize worth file and Allocate a large file past EOFon that file
+$XFS_IO_PROG -f -c "pwrite -b $blksz 0 $blksz" -c "falloc -k $blksz $(( 1 * 1024 * 1024 * scale ))" \
+ $SCRATCH_MNT/test >>$seqres.full 2>&1 ||_fail"failed to create test file"
# Fill the rest of the fs completely
# Note, this will show ENOSPC errors in $seqres.full, that's ok. echo "Fill fs with 1M IOs; ENOSPC expected" >> $seqres.full dd if=/dev/zero of=$SCRATCH_MNT/tmp1 bs=1M >>$seqres.full 2>&1tests/generic/274 +2
-2-echo "Fill fs with 4K IOs; ENOSPC expected" >> $seqres.full -dd if=/dev/zero of=$SCRATCH_MNT/tmp2 bs=4K >>$seqres.full 2>&1 +echo "Fill fs with $blksz K IOs; ENOSPC expected" >> $seqres.full +dd if=/dev/zero of=$SCRATCH_MNT/tmp2 bs=$blksz >>$seqres.full 2>&1
_scratch_sync
# Last effort, useO_SYNCtests/generic/274 +0
-2-echo "Fill fs with 4K DIOs; ENOSPC expected" >> $seqres.full -dd if=/dev/zero of=$SCRATCH_MNT/tmp3 bs=4K oflag=sync >>$seqres.full2>&1
tests/generic/274 +2
-0+echo "Fill fs with $blksz DIOs; ENOSPC expected" >> $seqres.full +dd if=/dev/zero of=$SCRATCH_MNT/tmp3 bs=$blksz oflag=syncQuoted 1 line
Quoted 1 line
$seqres.full 2>&1
# Save space usage info
echo "Post-fill space:" >> $seqres.full
df $SCRATCH_MNT >>$seqres.full 2>&1
-
# Now attempt a write into all of the preallocated space -# in a very nasty way, badly fragmenting it and then filling it in. echo "Fill in prealloc space; fragment at offsets:" >> $seqres.full for i in `seq 1 2 1023`; do echo -n "$i " >> $seqres.full - dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=4K count=1 conv=notrunc \ + dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=$blksz count=1 conv=notrunc \ >>$seqres.full 2>/dev/null || _fail "failed to write to test file" done _scratch_synctests/generic/274 +0
-0@@ -71,7 +72,7 @@ echo >> $seqres.fullecho "Fill in prealloc space; fill holes at offsets:" >> $seqres.full for i in
seq 2 2 1023; do
echo -n "$i " >> $seqres.full
- dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=4K count=1 conv=notrunc \
+ dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=$blksz count=1 conv=notrunc \
>>$seqres.full 2>/dev/null ||_fail"failed to fill test file" done
_scratch_sync
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore