Skip to contentSkip to conversation

btrfs: Misc test fixes for large block/node sizes

PATCH Superseded Nirjhar Roy (IBM) 5 participants 28 messages Jul 29, 2025 – Aug 12, 2025 Btrfs file system

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

cover btrfs: Misc test fixes for large block/node sizes

Some of the btrfs and generic tests are written with 4k block/node size in mind. This patch fixes some of those tests to be scalable with other block/node sizes too. We caught this bugs while running the auto group tests on btrfs with large block/node sizes.

Nirjhar Roy (IBM) (7):
common/filter: Add a helper function to filter offsets and sizes common/btrfs: Add a helper function to get the nodesize btrfs/137: Make this compatible with all block sizes btrfs/200: Make this test scale with the block size generic/563: Increase the write tolerance to 6% for larger nodesize btrfs/301: Make this test compatible with all block sizes. generic/274: Make the test compatible with all blocksizes.

 common/btrfs        |   9 +++
 common/filter       |  14 +++++
 tests/btrfs/137     | 135 +++++++++++++++++++++++++++++---------------
 tests/btrfs/137.out |  59 ++-----------------
 tests/btrfs/200     |  24 +++++---
 tests/btrfs/200.out |   8 +--
 tests/btrfs/301     |   8 ++-
 tests/generic/274   |  21 +++----
 tests/generic/563   |  17 +++++-
 9 files changed, 171 insertions(+), 124 deletions(-)

--
2.34.1

#1 lore raw

PATCH 1/7 common/filter: Add a helper function to filter offsets and sizes

_filter_xfs_io_size_offset() filters out the size and the offset emitted by various subcommands of xfs_io like pwrite.

---
 common/filter | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
common/filter +14 -0
index bbe13f4c..45188a5a 100644
@@ -115,6 +115,20 @@ _filter_date()
 	-e 's/[A-Z][a-z][a-z] [A-z][a-z][a-z]  *[0-9][0-9]* [0-9][0-9]:[0-9][0-9]:[0-9][0-9] [0-9][0-9][0-9][0-9]$/DATE/'
 }
 
+# This filters out the offsets and bytes written with
+# various subcommands of xfs_io. For example
+# wrote 4096/4096 bytes at offset 1052672 will be
+# converted to
+# wrote SIZE/SIZE bytes at offset OFFSET.
+# usage: __filter_xfs_io_size_offset <offset> <size>
+_filter_xfs_io_size_offset()
+{
+	local offset="$1"
+	local size="$2"
+	sed -e "s#${size}/${size}#SIZE/SIZE#g" \
+		-e "s#offset ${offset}#offset OFFSET#g"
+}
+
 # prints filtered output on stdout, values (use eval) on stderr
 # Non XFS filesystems always return a 4k block size and a 256 byte inode.
 _filter_mkfs()
Signature · 1 line
2.34.1
#2 lore raw

PATCH 2/7 common/btrfs: Add a helper function to get the nodesize

In reply to Nirjhar Roy (IBM)

Introduce a helper function to get the nodesize of the btrfs filesystem.

---
 common/btrfs | 9 +++++++++
 1 file changed, 9 insertions(+)
common/btrfs +9 -0
index 6a1095ff..2922eb8e 100644
@@ -18,6 +18,15 @@ _btrfs_get_subvolid()
 	$BTRFS_UTIL_PROG subvolume list $mnt | grep -E "\s$name$" | $AWK_PROG '{ print $2 }'
 }
 
+# returns the node size of the filesystem.
+# usage _get_btrfs_node_size <device_name>
+_get_btrfs_node_size()
+{
+        local dev=$1
+        $BTRFS_UTIL_PROG inspect-internal dump-super -f "$dev"\
+                 | awk '/nodesize/ { print $2 }'
+}
+
 # _require_btrfs_command <command> [<subcommand>|<option>]
 # We check for btrfs and (optionally) features of the btrfs command
 # This function support both subfunction like "inspect-internal dump-tree" and
Signature · 1 line
2.34.1
#3 lore raw

PATCH 3/7 btrfs/137: Make this compatible with all block sizes

In reply to Nirjhar Roy (IBM)

For large blocksizes like 64k on powerpc with 64k pagesize it failed simply because this test was written with 4k block size in mind.
The first few lines of the error logs are as follows:

     d3dc847171f9081bd75d7a2d3b53d322  SCRATCH_MNT/snap2/bar

     File snap1/foo fiemap results in the original filesystem:
    -0: [0..7]: data
    +0: [0..127]: data

     File snap1/bar fiemap results in the original filesystem:
    ...

Fix this by making the test choose offsets based on the blocksize. Also, now that the file hashes and the extent/block numbers will change depending on the blocksize, calculate the hashes and the block mappings, store them in temporary files and then calculate their diff between the new and the original filesystem.
This allows us to remove all the block mapping and hashes from the .out file.

---
 tests/btrfs/137     | 135 +++++++++++++++++++++++++++++---------------
 tests/btrfs/137.out |  59 ++-----------------
 2 files changed, 94 insertions(+), 100 deletions(-)
tests/btrfs/137 +89 -46
index 7710dc18..61e983cb 100755
@@ -27,53 +27,74 @@ _require_xfs_io_command "fiemap"
 send_files_dir=$TEST_DIR/btrfs-test-$seq
 
 rm -fr $send_files_dir
-mkdir $send_files_dir
+mkdir $send_files_dir $tmp
 
 _scratch_mkfs >>$seqres.full 2>&1
 _scratch_mount
 
+blksz=`_get_block_size $SCRATCH_MNT`
+echo "block size = $blksz" >> $seqres.full
+
 # Create the first test file.
-$XFS_IO_PROG -f -c "pwrite -S 0xaa 0 4K" $SCRATCH_MNT/foo | _filter_xfs_io
+$XFS_IO_PROG -f -c "pwrite -S 0xaa -b $blksz 0 $blksz" $SCRATCH_MNT/foo | _filter_xfs_io | \
+	_filter_xfs_io_size_offset 0 $blksz
 
 # Create a second test file with a 1Mb hole.
 $XFS_IO_PROG -f \
-     -c "pwrite -S 0xaa 0 4K" \
-     -c "pwrite -S 0xbb 1028K 4K" \
-     $SCRATCH_MNT/bar | _filter_xfs_io
+ 	-c "pwrite -S 0xaa -b $blksz 0 $blksz" \
+ 	-c "pwrite -S 0xbb -b $blksz $(( 1024 * 1024 + blksz )) $blksz" \
+ 	$SCRATCH_MNT/bar | _filter_xfs_io | \
+	_filter_xfs_io_size_offset "$(( 1024 * 1024 + blksz ))" $blksz | \
+ 	_filter_xfs_io_size_offset 0 $blksz
 
 $BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \
 	$SCRATCH_MNT/snap1 >/dev/null
 
 # Now add one new extent to our first test file, increasing its size and leaving
 # a 1Mb hole between the first extent and this new extent.
-$XFS_IO_PROG -c "pwrite -S 0xbb 1028K 4K" $SCRATCH_MNT/foo | _filter_xfs_io
+$XFS_IO_PROG -c "pwrite -S 0xbb -b $blksz $(( 1024 * 1024 + blksz )) $blksz" $SCRATCH_MNT/foo \
+	| _filter_xfs_io | _filter_xfs_io_size_offset "$(( 1024 * 1024 + blksz ))" $blksz
 
 # Now overwrite the last extent of our second test file.
-$XFS_IO_PROG -c "pwrite -S 0xcc 1028K 4K" $SCRATCH_MNT/bar | _filter_xfs_io
+$XFS_IO_PROG -c "pwrite -S 0xcc -b $blksz $(( 1024 * 1024 + blksz )) $blksz" $SCRATCH_MNT/bar \
+	| _filter_xfs_io | _filter_xfs_io_size_offset "$(( 1024 * 1024 + blksz ))" $blksz
 
 $BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \
 		 $SCRATCH_MNT/snap2 >/dev/null
 
-echo
-echo "File digests in the original filesystem:"
-md5sum $SCRATCH_MNT/snap1/foo | _filter_scratch
-md5sum $SCRATCH_MNT/snap1/bar | _filter_scratch
-md5sum $SCRATCH_MNT/snap2/foo | _filter_scratch
-md5sum $SCRATCH_MNT/snap2/bar | _filter_scratch
-
-echo
-echo "File snap1/foo fiemap results in the original filesystem:"
-$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/foo | _filter_fiemap
-echo
-echo "File snap1/bar fiemap results in the original filesystem:"
-$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/bar | _filter_fiemap
-echo
-echo "File snap2/foo fiemap results in the original filesystem:"
-$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/foo | _filter_fiemap
-echo
-echo "File snap2/bar fiemap results in the original filesystem:"
-$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/bar | _filter_fiemap
-echo
+echo >> $seqres.full
+
+echo "File digests in the original filesystem:" >> $seqres.full
+md5sum $SCRATCH_MNT/snap1/foo | _filter_scratch >> $tmp/snap1_foo.original.hash
+cat $tmp/snap1_foo.original.hash >> $seqres.full
+md5sum $SCRATCH_MNT/snap1/bar | _filter_scratch > $tmp/snap1_bar.original.hash
+cat $tmp/snap1_bar.original.hash >> $seqres.full
+md5sum $SCRATCH_MNT/snap2/foo | _filter_scratch >> $tmp/snap2_foo.original.hash
+cat $tmp/snap2_foo.original.hash >> $seqres.full
+md5sum $SCRATCH_MNT/snap2/bar | _filter_scratch > $tmp/snap2_bar.original.hash
+cat $tmp/snap2_bar.original.hash >> $seqres.full
+
+echo >> $seqres.full
+
+echo "File snap1/foo fiemap results in the original filesystem:" >> $seqres.full
+$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/foo | _filter_fiemap > $tmp/snap1_foo.original.map
+cat $tmp/snap1_foo.original.map >> $seqres.full
+echo >> $seqres.full
+
+echo "File snap1/bar fiemap results in the original filesystem:" >> $seqres.full
+$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/bar | _filter_fiemap > $tmp/snap1_bar.original.map
+cat $tmp/snap1_bar.original.map >> $seqres.full
+echo >> $seqres.full
+
+echo "File snap2/foo fiemap results in the original filesystem:" >> $seqres.full
+$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/foo | _filter_fiemap > $tmp/snap2_foo.original.map
+cat $tmp/snap2_foo.original.map >> $seqres.full
+echo >> $seqres.full
+
+echo "File snap2/bar fiemap results in the original filesystem:" >> $seqres.full
+$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/bar | _filter_fiemap > $tmp/snap2_bar.original.map
+cat $tmp/snap2_bar.original.map >> $seqres.full
+echo >> $seqres.full
 
 # Create the send streams to apply later on a new filesystem.
 $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/snap1 2>&1 \
@@ -90,25 +111,47 @@ _scratch_mount
 $BTRFS_UTIL_PROG receive -f $send_files_dir/1.snap $SCRATCH_MNT >/dev/null
 $BTRFS_UTIL_PROG receive -f $send_files_dir/2.snap $SCRATCH_MNT >/dev/null
 
-echo
-echo "File digests in the new filesystem:"
-md5sum $SCRATCH_MNT/snap1/foo | _filter_scratch
-md5sum $SCRATCH_MNT/snap1/bar | _filter_scratch
-md5sum $SCRATCH_MNT/snap2/foo | _filter_scratch
-md5sum $SCRATCH_MNT/snap2/bar | _filter_scratch
-
-echo
-echo "File snap1/foo fiemap results in the new filesystem:"
-$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/foo | _filter_fiemap
-echo
-echo "File snap1/bar fiemap results in the new filesystem:"
-$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/bar | _filter_fiemap
-echo
-echo "File snap2/foo fiemap results in the new filesystem:"
-$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/foo | _filter_fiemap
-echo
-echo "File snap2/bar fiemap results in the new filesystem:"
-$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/bar | _filter_fiemap
+echo >> $seqres.full
+echo "File digests in the new filesystem:" >> $seqres.full
+md5sum $SCRATCH_MNT/snap1/foo | _filter_scratch > $tmp/snap1_foo.new.hash
+cat $tmp/snap1_foo.new.hash >> $seqres.full
+md5sum $SCRATCH_MNT/snap1/bar | _filter_scratch > $tmp/snap1_bar.new.hash
+cat $tmp/snap1_bar.new.hash >> $seqres.full
+md5sum $SCRATCH_MNT/snap2/foo | _filter_scratch > $tmp/snap2_foo.new.hash
+cat $tmp/snap2_foo.new.hash >> $seqres.full
+md5sum $SCRATCH_MNT/snap2/bar | _filter_scratch > $tmp/snap2_bar.new.hash
+cat $tmp/snap2_bar.new.hash >> $seqres.full
+
+diff $tmp/snap1_foo.new.hash $tmp/snap1_foo.original.hash
+diff $tmp/snap1_bar.new.hash $tmp/snap1_bar.original.hash
+diff $tmp/snap2_foo.new.hash $tmp/snap2_foo.original.hash
+diff $tmp/snap2_bar.new.hash $tmp/snap2_bar.original.hash
+
+echo >> $seqres.full
+
+echo "File snap1/foo fiemap results in the new filesystem:" >> $seqres.full
+$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/foo | _filter_fiemap > $tmp/snap1_foo.new.map
+cat $tmp/snap1_foo.new.map >> $seqres.full
+echo >> $seqres.full
+
+echo "File snap1/bar fiemap results in the new filesystem:" >> $seqres.full
+$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap1/bar | _filter_fiemap > $tmp/snap1_bar.new.map
+cat $tmp/snap1_bar.new.map >> $seqres.full
+echo >> $seqres.full
+
+echo "File snap2/foo fiemap results in the new filesystem:" >> $seqres.full
+$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/foo | _filter_fiemap > $tmp/snap2_foo.new.map
+cat $tmp/snap2_foo.new.map >> $seqres.full
+echo >> $seqres.full
+
+echo "File snap2/bar fiemap results in the new filesystem:" >> $seqres.full
+$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/snap2/bar | _filter_fiemap > $tmp/snap2_bar.new.map
+cat $tmp/snap2_bar.new.map >> $seqres.full
+
+diff $tmp/snap1_foo.new.map $tmp/snap1_foo.original.map
+diff $tmp/snap1_bar.new.map $tmp/snap1_bar.original.map
+diff $tmp/snap2_foo.new.map $tmp/snap2_foo.original.map
+diff $tmp/snap2_bar.new.map $tmp/snap2_bar.original.map
 
 status=0
 exit
tests/btrfs/137.out +5 -54

68 lines not on this page · raw message

Signature · 1 line
2.34.1
#4 lore raw

Re: [PATCH 3/7] btrfs/137: Make this compatible with all block sizes

Quoted 14 lines · Nirjhar Roy (IBM)

For large blocksizes like 64k on powerpc with 64k pagesize it failed simply because this test was written with 4k block size in mind.
The first few lines of the error logs are as follows:

      d3dc847171f9081bd75d7a2d3b53d322  SCRATCH_MNT/snap2/bar

      File snap1/foo fiemap results in the original filesystem:
     -0: [0..7]: data
     +0: [0..127]: data

      File snap1/bar fiemap results in the original filesystem:
     ...
Quoted 2 lines

Fix this by making the test choose offsets based on the blocksize.

I'm wondering, why not just use a fixed 64K block size?

So that all supported btrfs block sizes can result the same file contents.

Quoted 7 lines

Also, now that the file hashes and
the extent/block numbers will change depending on the blocksize, calculate the hashes and the block mappings, store them in temporary files and then calculate their diff between the new and the original filesystem.
This allows us to remove all the block mapping and hashes from the .out file.

Although I agree we should remove the block mappings from the golden output, as compression can add extra flags and pollute the golden output.

But that can also be done with _require_btrfs_no_compress() helper.

Quoted 13 lines
---
  tests/btrfs/137     | 135 +++++++++++++++++++++++++++++---------------
  tests/btrfs/137.out |  59 ++-----------------
  2 files changed, 94 insertions(+), 100 deletions(-)
tests/btrfs/137 +0 -0
index 7710dc18..61e983cb 100755

tests/btrfs/137 _require_xfs_io_command "fiemap" · line 30

Quoted 5 lines
tests/btrfs/137 +1 -1
@@ -27,53 +27,74 @@ _require_xfs_io_command "fiemap"
  send_files_dir=$TEST_DIR/btrfs-test-$seq
 
  rm -fr $send_files_dir
-mkdir $send_files_dir
+mkdir $send_files_dir $tmp

Just a small nitpick, it's more common to use $tmp.<suffix>, that's why the default _cleanup() template goes with "rm -f $tmp.*"

Thanks,
Qu

#5 lore raw

Re: [PATCH 3/7] btrfs/137: Make this compatible with all block sizes

Quoted 14 lines · Qu Wenruo
Quoted 11 lines · Nirjhar Roy (IBM)

For large blocksizes like 64k on powerpc with 64k pagesize it failed simply because this test was written with 4k block size in mind.
The first few lines of the error logs are as follows:

      d3dc847171f9081bd75d7a2d3b53d322  SCRATCH_MNT/snap2/bar

      File snap1/foo fiemap results in the original filesystem:
     -0: [0..7]: data
     +0: [0..127]: data
Quoted 7 lines
Quoted 5 lines

File snap1/bar fiemap results in the original filesystem:
...

Fix this by making the test choose offsets based on the blocksize.

I'm wondering, why not just use a fixed 64K block size?

Hi Qu,

It will definitely be simpler to just use 64k io size but I feel it might be better to not hard code it for future proofing the tests. I know right now we don't have bs > ps in btrfs but maybe we get it in the future and we might start seeing funky block sizes > 64k.

Same goes for not hardcoding block mappings in the golden output.

Regards,
ojaswin

#6 lore raw

Re: [PATCH 3/7] btrfs/137: Make this compatible with all block sizes

Quoted 21 lines · Ojaswin Mujoo
Quoted 20 lines · Qu Wenruo
Quoted 16 lines · Nirjhar Roy (IBM)

For large blocksizes like 64k on powerpc with 64k pagesize it failed simply because this test was written with 4k block size in mind.
The first few lines of the error logs are as follows:

       d3dc847171f9081bd75d7a2d3b53d322  SCRATCH_MNT/snap2/bar

       File snap1/foo fiemap results in the original filesystem:
      -0: [0..7]: data
      +0: [0..127]: data

       File snap1/bar fiemap results in the original filesystem:
      ...

Fix this by making the test choose offsets based on the blocksize.

Quoted 8 lines
Quoted 1 line

I'm wondering, why not just use a fixed 64K block size?

Hi Qu,

It will definitely be simpler to just use 64k io size but I feel it might be better to not hard code it for future proofing the tests. I know right now we don't have bs > ps in btrfs but maybe we get it in the future and we might start seeing funky block sizes > 64k.

And in fact I'm going to add that bs > ps support very soon, since we already have large folio support for data, it will be just a simple mapping_set_folio_order_range() with a min order.

But still for btrfs, we're only going to support 4K, 8K, 16K, 32K, 64K block sizes, thus I don't think we need to bother larger bs yet.

Thanks,
Qu

Quoted 7 lines

Same goes for not hardcoding block mappings in the golden output.

Regards,
ojaswin

#7 lore raw

Re: [PATCH 3/7] btrfs/137: Make this compatible with all block sizes

Quoted 37 lines · Qu Wenruo
Quoted 21 lines · Ojaswin Mujoo
Quoted 20 lines · Qu Wenruo
Quoted 16 lines · Nirjhar Roy (IBM)

For large blocksizes like 64k on powerpc with 64k pagesize it failed simply because this test was written with 4k block size in mind.
The first few lines of the error logs are as follows:

       d3dc847171f9081bd75d7a2d3b53d322  SCRATCH_MNT/snap2/bar

       File snap1/foo fiemap results in the original filesystem:
      -0: [0..7]: data
      +0: [0..127]: data

       File snap1/bar fiemap results in the original filesystem:
      ...

Fix this by making the test choose offsets based on the blocksize.

Quoted 8 lines
Quoted 1 line

I'm wondering, why not just use a fixed 64K block size?

Hi Qu,

It will definitely be simpler to just use 64k io size but I feel it might be better to not hard code it for future proofing the tests. I know right now we don't have bs > ps in btrfs but maybe we get it in the future and we might start seeing funky block sizes > 64k.

And in fact I'm going to add that bs > ps support very soon, since we already have large folio support for data, it will be just a simple mapping_set_folio_order_range() with a min order.

Quoted 5 lines

But still for btrfs, we're only going to support 4K, 8K, 16K, 32K, 64K block sizes, thus I don't think we need to bother larger bs yet.

Thanks,
Qu

Okay sure, if you feel 64k should be okay for considerable future then lets hardcode it for simplicity.

Regards,
ojaswin

Quoted 9 lines
Quoted 8 lines

Same goes for not hardcoding block mappings in the golden output.

Regards,
ojaswin

#8 lore raw

Re: [PATCH 3/7] btrfs/137: Make this compatible with all block sizes

In reply to Ojaswin Mujoo
Quoted 24 lines · Ojaswin Mujoo
Quoted 14 lines · Qu Wenruo
Quoted 11 lines · Nirjhar Roy (IBM)

For large blocksizes like 64k on powerpc with 64k pagesize it failed simply because this test was written with 4k block size in mind.
The first few lines of the error logs are as follows:

      d3dc847171f9081bd75d7a2d3b53d322  SCRATCH_MNT/snap2/bar

      File snap1/foo fiemap results in the original filesystem:
     -0: [0..7]: data
     +0: [0..127]: data
Quoted 7 lines
Quoted 5 lines

File snap1/bar fiemap results in the original filesystem:
...

Fix this by making the test choose offsets based on the blocksize.

I'm wondering, why not just use a fixed 64K block size?

Quoted 8 lines

Hi Qu,

It will definitely be simpler to just use 64k io size but I feel it might be better to not hard code it for future proofing the tests. I know right now we don't have bs > ps in btrfs but maybe we get it in the future and we might start seeing funky block sizes > 64k.

Same goes for not hardcoding block mappings in the golden output.

Please keep it simple and use fixed 64K aligned sizes and offsets. It's very unlikely we will support sector sizes larger than 64K, so keeping fixed sizes is a lot simpler to understand, maintain and debug tests.

Thanks.

Quoted 4 lines

Regards,
ojaswin

#9 lore raw

Re: [PATCH 3/7] btrfs/137: Make this compatible with all block sizes

Quoted 28 lines · Filipe Manana
Quoted 27 lines · Ojaswin Mujoo
Quoted 13 lines · Qu Wenruo
Quoted 11 lines · Nirjhar Roy (IBM)

For large blocksizes like 64k on powerpc with 64k pagesize it failed simply because this test was written with 4k block size in mind.
The first few lines of the error logs are as follows:

       d3dc847171f9081bd75d7a2d3b53d322  SCRATCH_MNT/snap2/bar

       File snap1/foo fiemap results in the original filesystem:
      -0: [0..7]: data
      +0: [0..127]: data
Quoted 6 lines
Quoted 5 lines

File snap1/bar fiemap results in the original filesystem:
...

Fix this by making the test choose offsets based on the blocksize.

I'm wondering, why not just use a fixed 64K block size?

Hi Qu,

It will definitely be simpler to just use 64k io size but I feel it might be better to not hard code it for future proofing the tests. I know right now we don't have bs > ps in btrfs but maybe we get it in the future and we might start seeing funky block sizes > 64k.

Quoted 5 lines
Quoted 1 line

Same goes for not hardcoding block mappings in the golden output.

Please keep it simple and use fixed 64K aligned sizes and offsets. It's very unlikely we will support sector sizes larger than 64K, so keeping fixed sizes is a lot simpler to understand, maintain and debug tests.

Okay, noted.

--NR

Quoted 6 lines

Thanks.

Quoted 3 lines

Regards,
ojaswin

Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
#10 lore raw

Re: [PATCH 3/7] btrfs/137: Make this compatible with all block sizes

In reply to Qu Wenruo
Quoted 14 lines · Qu Wenruo
Quoted 11 lines · Nirjhar Roy (IBM)

For large blocksizes like 64k on powerpc with 64k pagesize it failed simply because this test was written with 4k block size in mind.
The first few lines of the error logs are as follows:

      d3dc847171f9081bd75d7a2d3b53d322  SCRATCH_MNT/snap2/bar

      File snap1/foo fiemap results in the original filesystem:
     -0: [0..7]: data
     +0: [0..127]: data

Quoted 7 lines
Quoted 5 lines

      File snap1/bar fiemap results in the original filesystem:
     ...

Fix this by making the test choose offsets based on the blocksize.

I'm wondering, why not just use a fixed 64K block size?

Sorry for delayed response. Yes, if you feel keeping it simple by keeping the values aligned to 64k, I can make the change.

Quoted 11 lines

So that all supported btrfs block sizes can result the same file contents.

Quoted 7 lines

Also, now that the file hashes and
the extent/block numbers will change depending on the blocksize, calculate the hashes and the block mappings, store them in temporary files and then calculate their diff between the new and the original filesystem.
This allows us to remove all the block mapping and hashes from the .out file.

Quoted 3 lines

Although I agree we should remove the block mappings from the golden output, as compression can add extra flags and pollute the golden output.

But that can also be done with _require_btrfs_no_compress() helper.

Okay. In that case, I will have the above pre-condition too.

Quoted 9 lines
Quoted 9 lines

---
  tests/btrfs/137     | 135 +++++++++++++++++++++++++++++--------------- tests/btrfs/137.out |  59 ++-----------------

  2 files changed, 94 insertions(+), 100 deletions(-)
Quoted 11 lines

tests/btrfs/137 _require_xfs_io_command "fiemap" · line 30

Quoted 9 lines
tests/btrfs/137 +0 -0
index 7710dc18..61e983cb 100755
@@ -27,53 +27,74 @@ _require_xfs_io_command "fiemap"

  send_files_dir=$TEST_DIR/btrfs-test-$seq
    rm -fr $send_files_dir

tests/btrfs/137 +1 -1
-mkdir $send_files_dir
+mkdir $send_files_dir $tmp

Just a small nitpick, it's more common to use $tmp.<suffix>, that's why the default _cleanup() template goes with "rm -f $tmp.*"

Noted. Thanks.

--NR

Quoted 4 lines

Thanks,
Qu

Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
#11 lore raw

PATCH 4/7 btrfs/200: Make this test scale with the block size

In reply to Nirjhar Roy (IBM)

For large block sizes like 64k on powerpc with 64k pagesize it failed because this test was hardcoded
to work with 4k blocksize.
With blocksize 4k and the existing file lengths,
we are getting 2 extents but with 64k page size
number of extents is not exceeding 1(due to lower
file size).
The first few lines of the error message is as follows:
At snapshot incr
OK
OK
+File foo does not have 2 shared extents in the base snapshot +/mnt/scratch/base/foo:
+ 0: [0..255]: 26624..26879
+File foo does not have 2 shared extents in the incr snapshot
...

Fix this by scaling the size and offsets to scale with the block size by a factor of (blocksize/4k).

---
 tests/btrfs/200     | 24 ++++++++++++++++--------
 tests/btrfs/200.out |  8 ++++----
 2 files changed, 20 insertions(+), 12 deletions(-)
tests/btrfs/200 +16 -8
index e62937a4..fd2c2026 100755
@@ -35,18 +35,26 @@ mkdir $send_files_dir
 _scratch_mkfs >>$seqres.full 2>&1
 _scratch_mount
 
+blksz=`_get_block_size $SCRATCH_MNT`
+echo "block size = $blksz" >> $seqres.full
+
+# Scale the test with any block size starting from 1k
+scale=$(( blksz / 1024 ))
+offset=$(( 16 * 1024 * scale ))
+size=$(( 16 * 1024 * scale ))
+
 # Create our first test file, which has an extent that is shared only with
 # itself and no other files. We want to verify a full send operation will
 # clone the extent.
-$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \
-	| _filter_xfs_io
-$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \
-	| _filter_xfs_io
+$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b $size 0 $size" $SCRATCH_MNT/foo \
+	| _filter_xfs_io | _filter_xfs_io_size_offset 0 $size
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 $offset $size" $SCRATCH_MNT/foo \
+	| _filter_xfs_io | _filter_xfs_io_size_offset $offset $size
 
 # Create out second test file which initially, for the first send operation,
 # only has a single extent that is not shared.
-$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \
-	| _filter_xfs_io
+$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $size 0 $size" $SCRATCH_MNT/bar \
+	| _filter_xfs_io | _filter_xfs_io_size_offset 0 $size
 
 _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base
 
@@ -56,8 +64,8 @@ $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \
 # Now clone the existing extent in file bar to itself at a different offset.
 # We want to verify the incremental send operation below will issue a clone
 # operation instead of a write operation.
-$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \
-	| _filter_xfs_io
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 $offset $size" $SCRATCH_MNT/bar \
+	| _filter_xfs_io | _filter_xfs_io_size_offset $offset $size
 
 _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incr
 
tests/btrfs/200.out +4 -4
index 306d9b24..4a10e506 100644
@@ -1,12 +1,12 @@
 QA output created by 200
-wrote 65536/65536 bytes at offset 0
+wrote SIZE/SIZE bytes at offset OFFSET
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-linked 65536/65536 bytes at offset 65536
+linked SIZE/SIZE bytes at offset OFFSET
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-wrote 65536/65536 bytes at offset 0
+wrote SIZE/SIZE bytes at offset OFFSET
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 At subvol SCRATCH_MNT/base
-linked 65536/65536 bytes at offset 65536
+linked SIZE/SIZE bytes at offset OFFSET
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 At subvol SCRATCH_MNT/incr
 At subvol base
Signature · 1 line
2.34.1
#12 lore raw

Re: [PATCH 4/7] btrfs/200: Make this test scale with the block size

Quoted 4 lines · Nirjhar Roy (IBM)

For large block sizes like 64k on powerpc with 64k pagesize it failed because this test was hardcoded to work with 4k blocksize.

Where exactly is it hardcoded with 4K blocksize expectations?

The test does 64K writes and reflinks at offsets multiples of 64K (0 and 64K). In fact that's why the test is doing 64K writes and using only the file offsets 0 and 64K, so that it works with any block size.

Quoted 4 lines

With blocksize 4k and the existing file lengths,
we are getting 2 extents but with 64k page size
number of extents is not exceeding 1(due to lower
file size).

Due to lower file size? How?
The file sizes should be independent of the block size, and be 64K and 128K everywhere.

Please provide more details in the changelog.
Thanks.

Quoted 92 lines

The first few lines of the error message is as follows:
At snapshot incr
OK
OK
+File foo does not have 2 shared extents in the base snapshot +/mnt/scratch/base/foo:
+ 0: [0..255]: 26624..26879
+File foo does not have 2 shared extents in the incr snapshot
...

Fix this by scaling the size and offsets to scale with the block size by a factor of (blocksize/4k).

---
 tests/btrfs/200     | 24 ++++++++++++++++--------
 tests/btrfs/200.out |  8 ++++----
 2 files changed, 20 insertions(+), 12 deletions(-)
tests/btrfs/200 +16 -8
index e62937a4..fd2c2026 100755
@@ -35,18 +35,26 @@ mkdir $send_files_dir
 _scratch_mkfs >>$seqres.full 2>&1
 _scratch_mount
 
+blksz=`_get_block_size $SCRATCH_MNT`
+echo "block size = $blksz" >> $seqres.full
+
+# Scale the test with any block size starting from 1k
+scale=$(( blksz / 1024 ))
+offset=$(( 16 * 1024 * scale ))
+size=$(( 16 * 1024 * scale ))
+
 # Create our first test file, which has an extent that is shared only with
 # itself and no other files. We want to verify a full send operation will
 # clone the extent.
-$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \
-       | _filter_xfs_io
-$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \
-       | _filter_xfs_io
+$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b $size 0 $size" $SCRATCH_MNT/foo \
+       | _filter_xfs_io | _filter_xfs_io_size_offset 0 $size
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 $offset $size" $SCRATCH_MNT/foo \
+       | _filter_xfs_io | _filter_xfs_io_size_offset $offset $size
 
 # Create out second test file which initially, for the first send operation,
 # only has a single extent that is not shared.
-$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \
-       | _filter_xfs_io
+$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $size 0 $size" $SCRATCH_MNT/bar \
+       | _filter_xfs_io | _filter_xfs_io_size_offset 0 $size
 
 _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base
 
@@ -56,8 +64,8 @@ $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \
 # Now clone the existing extent in file bar to itself at a different offset.
 # We want to verify the incremental send operation below will issue a clone
 # operation instead of a write operation.
-$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \
-       | _filter_xfs_io
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 $offset $size" $SCRATCH_MNT/bar \
+       | _filter_xfs_io | _filter_xfs_io_size_offset $offset $size
 
 _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incr
 
tests/btrfs/200.out +4 -5
index 306d9b24..4a10e506 100644
@@ -1,12 +1,12 @@
 QA output created by 200
-wrote 65536/65536 bytes at offset 0
+wrote SIZE/SIZE bytes at offset OFFSET
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-linked 65536/65536 bytes at offset 65536
+linked SIZE/SIZE bytes at offset OFFSET
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-wrote 65536/65536 bytes at offset 0
+wrote SIZE/SIZE bytes at offset OFFSET
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 At subvol SCRATCH_MNT/base
-linked 65536/65536 bytes at offset 65536
+linked SIZE/SIZE bytes at offset OFFSET
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 At subvol SCRATCH_MNT/incr
 At subvol base
--

2.34.1

#13 lore raw

Re: [PATCH 4/7] btrfs/200: Make this test scale with the block size

Quoted 15 lines · Filipe Manana
Quoted 3 lines · Nirjhar Roy (IBM)

For large block sizes like 64k on powerpc with 64k pagesize it failed because this test was hardcoded to work with 4k blocksize.

Where exactly is it hardcoded with 4K blocksize expectations?

The test does 64K writes and reflinks at offsets multiples of 64K (0 and 64K). In fact that's why the test is doing 64K writes and using only the file offsets 0 and 64K, so that it works with any block size.

Quoted 4 lines

With blocksize 4k and the existing file lengths,
we are getting 2 extents but with 64k page size
number of extents is not exceeding 1(due to lower
file size).

Quoted 6 lines

Due to lower file size? How?
The file sizes should be independent of the block size, and be 64K and 128K everywhere.

Please provide more details in the changelog.
Thanks.

Yes, I think I mis-interpreted the actual issue. I am looking into this. For now, I will remove this patch in the next version and once I am aware of the actual root cause, I will re-send with a proper fix and an explanation.

--NR

Quoted 93 lines
Quoted 93 lines

The first few lines of the error message is as follows:
At snapshot incr
OK
OK
+File foo does not have 2 shared extents in the base snapshot +/mnt/scratch/base/foo:
+ 0: [0..255]: 26624..26879
+File foo does not have 2 shared extents in the incr snapshot
...

Fix this by scaling the size and offsets to scale with the block size by a factor of (blocksize/4k).

---
  tests/btrfs/200     | 24 ++++++++++++++++--------
  tests/btrfs/200.out |  8 ++++----
  2 files changed, 20 insertions(+), 12 deletions(-)
tests/btrfs/200 +16 -8
index e62937a4..fd2c2026 100755
@@ -35,18 +35,26 @@ mkdir $send_files_dir
  _scratch_mkfs >>$seqres.full 2>&1
  _scratch_mount
 
+blksz=`_get_block_size $SCRATCH_MNT`
+echo "block size = $blksz" >> $seqres.full
+
+# Scale the test with any block size starting from 1k
+scale=$(( blksz / 1024 ))
+offset=$(( 16 * 1024 * scale ))
+size=$(( 16 * 1024 * scale ))
+
  # Create our first test file, which has an extent that is shared only with
  # itself and no other files. We want to verify a full send operation will
  # clone the extent.
-$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \
-       | _filter_xfs_io
-$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \
-       | _filter_xfs_io
+$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b $size 0 $size" $SCRATCH_MNT/foo \
+       | _filter_xfs_io | _filter_xfs_io_size_offset 0 $size
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 $offset $size" $SCRATCH_MNT/foo \
+       | _filter_xfs_io | _filter_xfs_io_size_offset $offset $size
 
  # Create out second test file which initially, for the first send operation,
  # only has a single extent that is not shared.
-$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \
-       | _filter_xfs_io
+$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $size 0 $size" $SCRATCH_MNT/bar \
+       | _filter_xfs_io | _filter_xfs_io_size_offset 0 $size
 
  _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base
 
@@ -56,8 +64,8 @@ $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \
  # Now clone the existing extent in file bar to itself at a different offset.
  # We want to verify the incremental send operation below will issue a clone
  # operation instead of a write operation.
-$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \
-       | _filter_xfs_io
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 $offset $size" $SCRATCH_MNT/bar \
+       | _filter_xfs_io | _filter_xfs_io_size_offset $offset $size
 
  _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incr
 
tests/btrfs/200.out +4 -5
index 306d9b24..4a10e506 100644
@@ -1,12 +1,12 @@
  QA output created by 200
-wrote 65536/65536 bytes at offset 0
+wrote SIZE/SIZE bytes at offset OFFSET
  XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-linked 65536/65536 bytes at offset 65536
+linked SIZE/SIZE bytes at offset OFFSET
  XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-wrote 65536/65536 bytes at offset 0
+wrote SIZE/SIZE bytes at offset OFFSET
  XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  At subvol SCRATCH_MNT/base
-linked 65536/65536 bytes at offset 65536
+linked SIZE/SIZE bytes at offset OFFSET
  XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  At subvol SCRATCH_MNT/incr
  At subvol base
--

2.34.1

Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
#14 lore raw

Re: [PATCH 4/7] btrfs/200: Make this test scale with the block size

In reply to Nirjhar Roy (IBM)
Quoted 17 lines · Nirjhar Roy (IBM)

For large block sizes like 64k on powerpc with 64k pagesize it failed because this test was hardcoded
to work with 4k blocksize.
With blocksize 4k and the existing file lengths,
we are getting 2 extents but with 64k page size
number of extents is not exceeding 1(due to lower
file size).
The first few lines of the error message is as follows:
At snapshot incr
OK
OK
+File foo does not have 2 shared extents in the base snapshot +/mnt/scratch/base/foo:
+ 0: [0..255]: 26624..26879
+File foo does not have 2 shared extents in the incr snapshot
...

Quoted 2 lines

Fix this by scaling the size and offsets to scale with the block size by a factor of (blocksize/4k).

Although I can reproduce the bug on aarch64 with 64K page size, the changelog doesn't seem to explain the problem.

The problem is after receive, the file base/foo is a single extent, not the original reflinked two extent.

And the original fs is indeed have the correct two extents layout.

That different extent layout is definitely not explained properly in the commit message.

Furthermore the test case is already using 64K block size for extents creation, thus all supported block size should work.

So I think the change doesn't really touch the core of the failure.

Thanks,
Qu

Quoted 77 lines
---
  tests/btrfs/200     | 24 ++++++++++++++++--------
  tests/btrfs/200.out |  8 ++++----
  2 files changed, 20 insertions(+), 12 deletions(-)
tests/btrfs/200 +16 -8
index e62937a4..fd2c2026 100755
@@ -35,18 +35,26 @@ mkdir $send_files_dir
  _scratch_mkfs >>$seqres.full 2>&1
  _scratch_mount
 
+blksz=`_get_block_size $SCRATCH_MNT`
+echo "block size = $blksz" >> $seqres.full
+
+# Scale the test with any block size starting from 1k
+scale=$(( blksz / 1024 ))
+offset=$(( 16 * 1024 * scale ))
+size=$(( 16 * 1024 * scale ))
+
  # Create our first test file, which has an extent that is shared only with
  # itself and no other files. We want to verify a full send operation will
  # clone the extent.
-$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \
-	| _filter_xfs_io
-$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \
-	| _filter_xfs_io
+$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b $size 0 $size" $SCRATCH_MNT/foo \
+	| _filter_xfs_io | _filter_xfs_io_size_offset 0 $size
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 $offset $size" $SCRATCH_MNT/foo \
+	| _filter_xfs_io | _filter_xfs_io_size_offset $offset $size
 
  # Create out second test file which initially, for the first send operation,
  # only has a single extent that is not shared.
-$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \
-	| _filter_xfs_io
+$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b $size 0 $size" $SCRATCH_MNT/bar \
+	| _filter_xfs_io | _filter_xfs_io_size_offset 0 $size
 
  _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base
 
@@ -56,8 +64,8 @@ $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \
  # Now clone the existing extent in file bar to itself at a different offset.
  # We want to verify the incremental send operation below will issue a clone
  # operation instead of a write operation.
-$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \
-	| _filter_xfs_io
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 $offset $size" $SCRATCH_MNT/bar \
+	| _filter_xfs_io | _filter_xfs_io_size_offset $offset $size
 
  _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incr
 
tests/btrfs/200.out +4 -4
index 306d9b24..4a10e506 100644
@@ -1,12 +1,12 @@
  QA output created by 200
-wrote 65536/65536 bytes at offset 0
+wrote SIZE/SIZE bytes at offset OFFSET
  XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-linked 65536/65536 bytes at offset 65536
+linked SIZE/SIZE bytes at offset OFFSET
  XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-wrote 65536/65536 bytes at offset 0
+wrote SIZE/SIZE bytes at offset OFFSET
  XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  At subvol SCRATCH_MNT/base
-linked 65536/65536 bytes at offset 65536
+linked SIZE/SIZE bytes at offset OFFSET
  XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  At subvol SCRATCH_MNT/incr
  At subvol base
#15 lore raw

PATCH 5/7 generic/563: Increase the write tolerance to 6% for larger nodesize

In reply to Nirjhar Roy (IBM)

When tested with blocksize/nodesize 64K on powerpc with 64k pagesize on btrfs, then the test fails with the folllowing error:

     QA output created by 563
     read/write
     read is in range
    -write is in range
    +write has value of 8855552
    +write is NOT in range 7969177.6 .. 8808038.4
     write -> read/write
    ...

The slight increase in the amount of bytes that
are written is because of the increase in the
the nodesize(metadata) and hence it exceeds the tolerance limit slightly. Fix this by increasing the write tolerance limit from 5% from 6% for 64k blocksize btrfs.

---
 tests/generic/563 | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
tests/generic/563 +16 -1
index 89a71aa4..efcac1ec 100755
@@ -119,7 +119,22 @@ $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \
 	$SCRATCH_MNT/file >> $seqres.full 2>&1
 switch_cg $cgdir
 $XFS_IO_PROG -c fsync $SCRATCH_MNT/file
-check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+blksz=`_get_block_size $SCRATCH_MNT`
+
+# On higher node sizes on btrfs, we observed slightly more 
+# writes, due to increased metadata sizes.
+# Hence have a higher write tolerance for btrfs and when
+# node size is greater than 4k.
+if [[ "$FSTYP" == "btrfs" ]]; then
+	nodesz=$(_get_btrfs_node_size "$SCRATCH_DEV")
+	if [[ "$nodesz" -gt 4096 ]]; then
+		check_cg $cgdir/$seq-cg $iosize $iosize 5% 6%
+	else
+		check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+	fi
+else
+	check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+fi
 
 # Write from one cgroup then read and write from a second. Writes are charged to
 # the first group and nothing to the second.
Signature · 1 line
2.34.1
#16 lore raw

Re: [PATCH 5/7] generic/563: Increase the write tolerance to 6% for larger nodesize

tests/generic/563 $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \ · line 137

Quoted 13 lines · Nirjhar Roy (IBM)
tests/generic/563 +13 -0
+# writes, due to increased metadata sizes.
+# Hence have a higher write tolerance for btrfs and when
+# node size is greater than 4k.
+if [[ "$FSTYP" == "btrfs" ]]; then
+	nodesz=$(_get_btrfs_node_size "$SCRATCH_DEV")
+	if [[ "$nodesz" -gt 4096 ]]; then
+		check_cg $cgdir/$seq-cg $iosize $iosize 5% 6%
+	else
+		check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+	fi
+else
+	check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+fi

Everyone please stop hacking file system specific things into generic tests. Add proper core helpers for this instead.

#17 lore raw

Re: [PATCH 5/7] generic/563: Increase the write tolerance to 6% for larger nodesize

Quoted 16 lines · Christoph Hellwig

tests/generic/563 $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \ · line 137

Quoted 13 lines · Nirjhar Roy (IBM)
tests/generic/563 +13 -0
+# writes, due to increased metadata sizes.
+# Hence have a higher write tolerance for btrfs and when
+# node size is greater than 4k.
+if [[ "$FSTYP" == "btrfs" ]]; then
+	nodesz=$(_get_btrfs_node_size "$SCRATCH_DEV")
+	if [[ "$nodesz" -gt 4096 ]]; then
+		check_cg $cgdir/$seq-cg $iosize $iosize 5% 6%
+	else
+		check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+	fi
+else
+	check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+fi

Everyone please stop hacking file system specific things into generic tests. Add proper core helpers for this instead.

Okay. I will try to work something out. Thank you.

--NR

Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
#18 lore raw

Re: [PATCH 5/7] generic/563: Increase the write tolerance to 6% for larger nodesize

In reply to Nirjhar Roy (IBM)
Quoted 29 lines · Nirjhar Roy (IBM)

When tested with blocksize/nodesize 64K on powerpc with 64k pagesize on btrfs, then the test fails with the folllowing error:

     QA output created by 563
     read/write
     read is in range
    -write is in range
    +write has value of 8855552
    +write is NOT in range 7969177.6 .. 8808038.4
     write -> read/write
    ...

The slight increase in the amount of bytes that
are written is because of the increase in the
the nodesize(metadata) and hence it exceeds the tolerance limit slightly. Fix this by increasing the write tolerance limit from 5% from 6% for 64k blocksize btrfs.

---
 tests/generic/563 | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
tests/generic/563 +0 -0
index 89a71aa4..efcac1ec 100755

tests/generic/563 $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \ · line 122

Quoted 5 lines
tests/generic/563 +1 -1
@@ -119,7 +119,22 @@ $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \
        $SCRATCH_MNT/file >> $seqres.full 2>&1
 switch_cg $cgdir
 $XFS_IO_PROG -c fsync $SCRATCH_MNT/file
-check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+blksz=`_get_block_size $SCRATCH_MNT`

Here the block size is captured, but then it's not used anywhere in the test...

Thanks.

Quoted 21 lines
tests/generic/563 +15 -1
+
+# On higher node sizes on btrfs, we observed slightly more
+# writes, due to increased metadata sizes.
+# Hence have a higher write tolerance for btrfs and when
+# node size is greater than 4k.
+if [[ "$FSTYP" == "btrfs" ]]; then
+       nodesz=$(_get_btrfs_node_size "$SCRATCH_DEV")
+       if [[ "$nodesz" -gt 4096 ]]; then
+               check_cg $cgdir/$seq-cg $iosize $iosize 5% 6%
+       else
+               check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+       fi
+else
+       check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+fi
 
 # Write from one cgroup then read and write from a second. Writes are charged to
 # the first group and nothing to the second.
--

2.34.1

#19 lore raw

Re: [PATCH 5/7] generic/563: Increase the write tolerance to 6% for larger nodesize

Quoted 25 lines · Filipe Manana
Quoted 23 lines · Nirjhar Roy (IBM)

When tested with blocksize/nodesize 64K on powerpc with 64k pagesize on btrfs, then the test fails with the folllowing error:

      QA output created by 563
      read/write
      read is in range
     -write is in range
     +write has value of 8855552
     +write is NOT in range 7969177.6 .. 8808038.4
      write -> read/write
     ...

The slight increase in the amount of bytes that
are written is because of the increase in the
the nodesize(metadata) and hence it exceeds the tolerance limit slightly. Fix this by increasing the write tolerance limit from 5% from 6% for 64k blocksize btrfs.

---
  tests/generic/563 | 17 ++++++++++++++++-
  1 file changed, 16 insertions(+), 1 deletion(-)
Quoted 11 lines

tests/generic/563 $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \ · line 122

Quoted 10 lines
tests/generic/563 +1 -1
index 89a71aa4..efcac1ec 100755
@@ -119,7 +119,22 @@ $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \
         $SCRATCH_MNT/file >> $seqres.full 2>&1
  switch_cg $cgdir
  $XFS_IO_PROG -c fsync $SCRATCH_MNT/file
-check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+blksz=`_get_block_size $SCRATCH_MNT`

Here the block size is captured, but then it's not used anywhere in the test...

Yeah. I will remove this. Thanks.

--NR

Quoted 24 lines

Thanks.

Quoted 21 lines
tests/generic/563 +15 -1
+
+# On higher node sizes on btrfs, we observed slightly more
+# writes, due to increased metadata sizes.
+# Hence have a higher write tolerance for btrfs and when
+# node size is greater than 4k.
+if [[ "$FSTYP" == "btrfs" ]]; then
+       nodesz=$(_get_btrfs_node_size "$SCRATCH_DEV")
+       if [[ "$nodesz" -gt 4096 ]]; then
+               check_cg $cgdir/$seq-cg $iosize $iosize 5% 6%
+       else
+               check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+       fi
+else
+       check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+fi
 
  # Write from one cgroup then read and write from a second. Writes are charged to
  # the first group and nothing to the second.
--

2.34.1

Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
#20 lore raw

Re: [PATCH 5/7] generic/563: Increase the write tolerance to 6% for larger nodesize

In reply to Nirjhar Roy (IBM)
Quoted 9 lines · Nirjhar Roy (IBM)

When tested with blocksize/nodesize 64K on powerpc with 64k pagesize on btrfs, then the test fails with the folllowing error:
QA output created by 563
read/write
read is in range
-write is in range
+write has value of 8855552
+write is NOT in range 7969177.6 .. 8808038.4

I can reproduce the failure, although it's not 100% reliable, and indeed with one tree block's size removed, it's back into the tolerance range.

Quoted 29 lines
      write -> read/write
     ...

The slight increase in the amount of bytes that
are written is because of the increase in the
the nodesize(metadata) and hence it exceeds the tolerance limit slightly. Fix this by increasing the write tolerance limit from 5% from 6% for 64k blocksize btrfs.

---
  tests/generic/563 | 17 ++++++++++++++++-
  1 file changed, 16 insertions(+), 1 deletion(-)
tests/generic/563 +6 -1
index 89a71aa4..efcac1ec 100755
@@ -119,7 +119,22 @@ $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \
  	$SCRATCH_MNT/file >> $seqres.full 2>&1
  switch_cg $cgdir
  $XFS_IO_PROG -c fsync $SCRATCH_MNT/file
-check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+blksz=`_get_block_size $SCRATCH_MNT`
+
+# On higher node sizes on btrfs, we observed slightly more
+# writes, due to increased metadata sizes.
+# Hence have a higher write tolerance for btrfs and when
+# node size is greater than 4k.

tests/generic/563 $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \ · line 137

Quoted 10 lines
tests/generic/563 +10 -0
+if [[ "$FSTYP" == "btrfs" ]]; then
+	nodesz=$(_get_btrfs_node_size "$SCRATCH_DEV")
+	if [[ "$nodesz" -gt 4096 ]]; then
+		check_cg $cgdir/$seq-cg $iosize $iosize 5% 6%
+	else
+		check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+	fi
+else
+	check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+fi

Instead of the btrfs specific hack, I'd recommend to just enlarge iosize.

Double the iosize will easily make it to cover the tolerance of even btrfs, but you still need a proper explanation of the change.

Thanks,
Qu

Quoted 3 lines
tests/generic/563 +0 -0
  # Write from one cgroup then read and write from a second. Writes are charged to
  # the first group and nothing to the second.
#21 lore raw

Re: [PATCH 5/7] generic/563: Increase the write tolerance to 6% for larger nodesize

Quoted 58 lines · Qu Wenruo
Quoted 9 lines · Nirjhar Roy (IBM)

When tested with blocksize/nodesize 64K on powerpc with 64k  pagesize on btrfs, then the test fails with the folllowing error:
      QA output created by 563
      read/write
      read is in range
     -write is in range
     +write has value of 8855552
     +write is NOT in range 7969177.6 .. 8808038.4

I can reproduce the failure, although it's not 100% reliable, and indeed with one tree block's size removed, it's back into the tolerance range.

Quoted 30 lines

      write -> read/write
     ...
The slight increase in the amount of bytes that
are written is because of the increase in the
the nodesize(metadata) and hence it exceeds the tolerance limit slightly. Fix this by increasing the write tolerance limit from 5% from 6% for 64k blocksize btrfs.

---
  tests/generic/563 | 17 ++++++++++++++++-

  1 file changed, 16 insertions(+), 1 deletion(-)
tests/generic/563 +0 -0
index 89a71aa4..efcac1ec 100755
@@ -119,7 +119,22 @@ $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b 
 $blksize 0 $iosize" -c fsync \

      $SCRATCH_MNT/file >> $seqres.full 2>&1
  switch_cg $cgdir
  $XFS_IO_PROG -c fsync $SCRATCH_MNT/file

tests/generic/563 +6 -1
-check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+blksz=`_get_block_size $SCRATCH_MNT`
+
+# On higher node sizes on btrfs, we observed slightly more
+# writes, due to increased metadata sizes.
+# Hence have a higher write tolerance for btrfs and when
+# node size is greater than 4k.

tests/generic/563 $XFS_IO_PROG -c "pread 0 $iosize" -c "pwrite -b $blksize 0 $iosize" -c fsync \ · line 137

Quoted 10 lines
tests/generic/563 +10 -0
+if [[ "$FSTYP" == "btrfs" ]]; then
+    nodesz=$(_get_btrfs_node_size "$SCRATCH_DEV")
+    if [[ "$nodesz" -gt 4096 ]]; then
+        check_cg $cgdir/$seq-cg $iosize $iosize 5% 6%
+    else
+        check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+    fi
+else
+    check_cg $cgdir/$seq-cg $iosize $iosize 5% 5%
+fi

Instead of the btrfs specific hack, I'd recommend to just enlarge iosize.

Quoted 1 line

Double the iosize will easily make it to cover the tolerance of even btrfs, but you still need a proper explanation of the change.

Okay. I can try the above and will come up with more detailed explantion.

--NR

Quoted 7 lines

Thanks,
Qu

Quoted 2 lines

    # Write from one cgroup then read and write from a second. Writes are charged to

  # the first group and nothing to the second.
Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
#22 lore raw

PATCH 6/7 btrfs/301: Make this test compatible with all block sizes.

In reply to Nirjhar Roy (IBM)

With large block sizes like 64k on powerpc with 64k pagesize the test failed with the following logs:

     QA output created by 301
     basic accounting
    +subvol 256 mismatched usage 33947648 vs 4587520 \
         (expected data 4194304 expected meta 393216 diff 29360128)
    +subvol 256 mismatched usage 168165376 vs 138805248 \
	(expected data 138412032 expected meta 393216 diff 29360128)
    +subvol 256 mismatched usage 33947648 vs 4587520 \
	(expected data 4194304 expected meta 393216 diff 29360128)
    +subvol 256 mismatched usage 33947648 vs 4587520 \
	(expected data 4194304 expected meta 393216 diff 29360128)
     fallocate: Disk quota exceeded

(Please note that the above ouptut had to be modified a bit since the number of characters in each line was much greater than the 72 characters.)

The test creates nr_fill files each of size 8k i.e, 2x4k(stored in fill_sz). Now with 64k blocksize, 8k sized files occupy more than expected sizes (i.e, 8k) due to internal fragmentation since 1 file
will occupy at least 1 block. Fix this by scaling the file size (fill_sz) with the blocksize.

---
 tests/btrfs/301 | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
tests/btrfs/301 +7 -1
index 6b59749d..7547ff0e 100755
@@ -23,7 +23,13 @@ subv=$SCRATCH_MNT/subv
 nested=$SCRATCH_MNT/subv/nested
 snap=$SCRATCH_MNT/snap
 nr_fill=512
-fill_sz=$((8 * 1024))
+
+_scratch_mkfs >> $seqres.full
+_scratch_mount
+blksz=`_get_block_size $SCRATCH_MNT`
+_scratch_unmount
+fill_sz=$(( 2 * blksz ))
+
 total_fill=$(($nr_fill * $fill_sz))
 nodesize=$($BTRFS_UTIL_PROG inspect-internal dump-super $SCRATCH_DEV | \
 					grep nodesize | $AWK_PROG '{print $2}')
Signature · 1 line
2.34.1
#23 lore raw

Re: [PATCH 6/7] btrfs/301: Make this test compatible with all block sizes.

Quoted 14 lines · Nirjhar Roy (IBM)

With large block sizes like 64k on powerpc with 64k pagesize the test failed with the following logs:

      QA output created by 301
      basic accounting
     +subvol 256 mismatched usage 33947648 vs 4587520 \
          (expected data 4194304 expected meta 393216 diff 29360128)
     +subvol 256 mismatched usage 168165376 vs 138805248 \
	(expected data 138412032 expected meta 393216 diff 29360128)
     +subvol 256 mismatched usage 33947648 vs 4587520 \
	(expected data 4194304 expected meta 393216 diff 29360128)
     +subvol 256 mismatched usage 33947648 vs 4587520 \
	(expected data 4194304 expected meta 393216 diff 29360128)
      fallocate: Disk quota exceeded
Quoted 3 lines

(Please note that the above ouptut had to be modified a bit since the number of characters in each line was much greater than the 72 characters.)

You don't need to break the line for raw output.

Quoted 6 lines

The test creates nr_fill files each of size 8k i.e, 2x4k(stored in fill_sz). Now with 64k blocksize, 8k sized files occupy more than expected sizes (i.e, 8k) due to internal fragmentation since 1 file
will occupy at least 1 block. Fix this by scaling the file size (fill_sz) with the blocksize.

You can just replace the fill_sz to 64K so that all block sizes will work.

Just tested with 64K fill_sz, it works for both 4K and 64K block size with 64K page size.

Thanks,
Qu

Quoted 26 lines
---
  tests/btrfs/301 | 8 +++++++-
  1 file changed, 7 insertions(+), 1 deletion(-)
tests/btrfs/301 +7 -1
index 6b59749d..7547ff0e 100755
@@ -23,7 +23,13 @@ subv=$SCRATCH_MNT/subv
  nested=$SCRATCH_MNT/subv/nested
  snap=$SCRATCH_MNT/snap
  nr_fill=512
-fill_sz=$((8 * 1024))
+
+_scratch_mkfs >> $seqres.full
+_scratch_mount
+blksz=`_get_block_size $SCRATCH_MNT`
+_scratch_unmount
+fill_sz=$(( 2 * blksz ))
+
  total_fill=$(($nr_fill * $fill_sz))
  nodesize=$($BTRFS_UTIL_PROG inspect-internal dump-super $SCRATCH_DEV | \
  					grep nodesize | $AWK_PROG '{print $2}')
#24 lore raw

Re: [PATCH 6/7] btrfs/301: Make this test compatible with all block sizes.

Quoted 21 lines · Qu Wenruo
Quoted 17 lines · Nirjhar Roy (IBM)

With large block sizes like 64k on powerpc with 64k pagesize the test failed with the following logs:

      QA output created by 301
      basic accounting
     +subvol 256 mismatched usage 33947648 vs 4587520 \
          (expected data 4194304 expected meta 393216 diff 29360128) +subvol 256 mismatched usage 168165376 vs 138805248 \
    (expected data 138412032 expected meta 393216 diff 29360128) +subvol 256 mismatched usage 33947648 vs 4587520 \
    (expected data 4194304 expected meta 393216 diff 29360128) +subvol 256 mismatched usage 33947648 vs 4587520 \
    (expected data 4194304 expected meta 393216 diff 29360128)
      fallocate: Disk quota exceeded
(Please note that the above ouptut had to be modified a bit since the number of characters in each line was much greater than the 72 characters.)

Quoted 1 line

You don't need to break the line for raw output.

Noted.

Quoted 11 lines
Quoted 7 lines

The test creates nr_fill files each of size 8k i.e, 2x4k(stored in fill_sz). Now with 64k blocksize, 8k sized files occupy more than expected sizes (i.e, 8k) due to internal fragmentation since 1 file
will occupy at least 1 block. Fix this by scaling the file size (fill_sz) with the blocksize.

You can just replace the fill_sz to 64K so that all block sizes will work.

Just tested with 64K fill_sz, it works for both 4K and 64K block size with 64K page size.

Okay, I will make the test work with 64k aligned values (similar to suggestions to the previous fixes in this patch series).

--NR

Quoted 31 lines

Thanks,
Qu

Quoted 26 lines

---
  tests/btrfs/301 | 8 +++++++-

  1 file changed, 7 insertions(+), 1 deletion(-)
tests/btrfs/301 +0 -0
index 6b59749d..7547ff0e 100755
@@ -23,7 +23,13 @@ subv=$SCRATCH_MNT/subv

  nested=$SCRATCH_MNT/subv/nested
  snap=$SCRATCH_MNT/snap
  nr_fill=512

tests/btrfs/301 +7 -1
-fill_sz=$((8 * 1024))
+
+_scratch_mkfs >> $seqres.full
+_scratch_mount
+blksz=`_get_block_size $SCRATCH_MNT`
+_scratch_unmount
+fill_sz=$(( 2 * blksz ))
+

  total_fill=$(($nr_fill * $fill_sz)) nodesize=$($BTRFS_UTIL_PROG inspect-internal dump-super $SCRATCH_DEV | \
                      grep nodesize | $AWK_PROG '{print $2}')

Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
#25 lore raw

PATCH 7/7 generic/274: Make the test compatible with all blocksizes.

In reply to Nirjhar Roy (IBM)

On btrfs with 64k blocksize on powerpc with 64k pagesize it failed with the following error:

     ------------------------------
     preallocation test
     ------------------------------
    -done
    +failed to write to test file
    +(see /home/xfstests-dev/results//btrfs_64k/generic/274.full for details)
    ...

So, this test is written with 4K block size in mind. As we can see, it first creates a file of size 4K and then fallocates 4MB beyond the EOF.
Then there are 2 loops - one that fragments at alternate blocks and the other punches holes in the remaining alternate blocks. Hence, the test fails in 64k block size due to incorrect calculations.

Fix this test by making the test scale with the block size, that is the offset, filesize and the assumed blocksize matches/scales with the actual blocksize of the underlying filesystem.

---
 tests/generic/274 | 21 +++++++++++----------
 1 file changed, 11 insertions(+), 10 deletions(-)
tests/generic/274 +11 -10
index 916c7173..4ea42f30 100755
@@ -40,30 +40,31 @@ _scratch_unmount 2>/dev/null
 _scratch_mkfs_sized $((2 * 1024 * 1024 * 1024)) >>$seqres.full 2>&1
 _scratch_mount
 
-# Create a 4k file and Allocate 4M past EOF on that file
-$XFS_IO_PROG -f -c "pwrite 0 4k" -c "falloc -k 4k 4m" $SCRATCH_MNT/test \
-	>>$seqres.full 2>&1 || _fail "failed to create test file"
+blksz=`_get_block_size $SCRATCH_MNT`
+scale=$(( blksz / 1024 ))
+# Create a blocksize worth file and Allocate a large file past EOF on that file
+$XFS_IO_PROG -f -c "pwrite -b $blksz 0 $blksz" -c "falloc -k $blksz $(( 1 * 1024 * 1024 * scale ))" \
+	$SCRATCH_MNT/test >>$seqres.full 2>&1 || _fail "failed to create test file"
 
 # Fill the rest of the fs completely
 # Note, this will show ENOSPC errors in $seqres.full, that's ok.
 echo "Fill fs with 1M IOs; ENOSPC expected" >> $seqres.full
 dd if=/dev/zero of=$SCRATCH_MNT/tmp1 bs=1M >>$seqres.full 2>&1
-echo "Fill fs with 4K IOs; ENOSPC expected" >> $seqres.full
-dd if=/dev/zero of=$SCRATCH_MNT/tmp2 bs=4K >>$seqres.full 2>&1
+echo "Fill fs with $blksz K IOs; ENOSPC expected" >> $seqres.full
+dd if=/dev/zero of=$SCRATCH_MNT/tmp2 bs=$blksz >>$seqres.full 2>&1
 _scratch_sync
 # Last effort, use O_SYNC
-echo "Fill fs with 4K DIOs; ENOSPC expected" >> $seqres.full
-dd if=/dev/zero of=$SCRATCH_MNT/tmp3 bs=4K oflag=sync >>$seqres.full 2>&1
+echo "Fill fs with $blksz DIOs; ENOSPC expected" >> $seqres.full
+dd if=/dev/zero of=$SCRATCH_MNT/tmp3 bs=$blksz oflag=sync >>$seqres.full 2>&1
 # Save space usage info
 echo "Post-fill space:" >> $seqres.full
 df $SCRATCH_MNT >>$seqres.full 2>&1
-
 # Now attempt a write into all of the preallocated space -
 # in a very nasty way, badly fragmenting it and then filling it in.
 echo "Fill in prealloc space; fragment at offsets:" >> $seqres.full
 for i in `seq 1 2 1023`; do
 	echo -n "$i " >> $seqres.full
-	dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=4K count=1 conv=notrunc \
+	dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=$blksz count=1 conv=notrunc \
 		>>$seqres.full 2>/dev/null || _fail "failed to write to test file"
 done
 _scratch_sync
@@ -71,7 +72,7 @@ echo >> $seqres.full
 echo "Fill in prealloc space; fill holes at offsets:" >> $seqres.full
 for i in `seq 2 2 1023`; do
 	echo -n "$i " >> $seqres.full
-	dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=4K count=1 conv=notrunc \
+	dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=$blksz count=1 conv=notrunc \
 		>>$seqres.full 2>/dev/null || _fail "failed to fill test file"
 done
 _scratch_sync
Signature · 1 line
2.34.1
#26 lore raw

Re: [PATCH 7/7] generic/274: Make the test compatible with all blocksizes.

Quoted 17 lines · Nirjhar Roy (IBM)

On btrfs with 64k blocksize on powerpc with 64k pagesize it failed with the following error:

      ------------------------------
      preallocation test
      ------------------------------
     -done
     +failed to write to test file
     +(see /home/xfstests-dev/results//btrfs_64k/generic/274.full for details)
     ...

So, this test is written with 4K block size in mind. As we can see, it first creates a file of size 4K and then fallocates 4MB beyond the EOF.
Then there are 2 loops - one that fragments at alternate blocks and the other punches holes in the remaining alternate blocks. Hence, the test fails in 64k block size due to incorrect calculations.

Quoted 3 lines

Fix this test by making the test scale with the block size, that is the offset, filesize and the assumed blocksize matches/scales with the actual blocksize of the underlying filesystem.

Again, just enlarge the block size from 4K to 64K, then all block size will work.

Thanks,
Qu

Quoted 61 lines
---
  tests/generic/274 | 21 +++++++++++----------
  1 file changed, 11 insertions(+), 10 deletions(-)
tests/generic/274 +11 -10
index 916c7173..4ea42f30 100755
@@ -40,30 +40,31 @@ _scratch_unmount 2>/dev/null
  _scratch_mkfs_sized $((2 * 1024 * 1024 * 1024)) >>$seqres.full 2>&1
  _scratch_mount
 
-# Create a 4k file and Allocate 4M past EOF on that file
-$XFS_IO_PROG -f -c "pwrite 0 4k" -c "falloc -k 4k 4m" $SCRATCH_MNT/test \
-	>>$seqres.full 2>&1 || _fail "failed to create test file"
+blksz=`_get_block_size $SCRATCH_MNT`
+scale=$(( blksz / 1024 ))
+# Create a blocksize worth file and Allocate a large file past EOF on that file
+$XFS_IO_PROG -f -c "pwrite -b $blksz 0 $blksz" -c "falloc -k $blksz $(( 1 * 1024 * 1024 * scale ))" \
+	$SCRATCH_MNT/test >>$seqres.full 2>&1 || _fail "failed to create test file"
 
  # Fill the rest of the fs completely
  # Note, this will show ENOSPC errors in $seqres.full, that's ok.
  echo "Fill fs with 1M IOs; ENOSPC expected" >> $seqres.full
  dd if=/dev/zero of=$SCRATCH_MNT/tmp1 bs=1M >>$seqres.full 2>&1
-echo "Fill fs with 4K IOs; ENOSPC expected" >> $seqres.full
-dd if=/dev/zero of=$SCRATCH_MNT/tmp2 bs=4K >>$seqres.full 2>&1
+echo "Fill fs with $blksz K IOs; ENOSPC expected" >> $seqres.full
+dd if=/dev/zero of=$SCRATCH_MNT/tmp2 bs=$blksz >>$seqres.full 2>&1
  _scratch_sync
  # Last effort, use O_SYNC
-echo "Fill fs with 4K DIOs; ENOSPC expected" >> $seqres.full
-dd if=/dev/zero of=$SCRATCH_MNT/tmp3 bs=4K oflag=sync >>$seqres.full 2>&1
+echo "Fill fs with $blksz DIOs; ENOSPC expected" >> $seqres.full
+dd if=/dev/zero of=$SCRATCH_MNT/tmp3 bs=$blksz oflag=sync >>$seqres.full 2>&1
  # Save space usage info
  echo "Post-fill space:" >> $seqres.full
  df $SCRATCH_MNT >>$seqres.full 2>&1
-
  # Now attempt a write into all of the preallocated space -
  # in a very nasty way, badly fragmenting it and then filling it in.
  echo "Fill in prealloc space; fragment at offsets:" >> $seqres.full
  for i in `seq 1 2 1023`; do
  	echo -n "$i " >> $seqres.full
-	dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=4K count=1 conv=notrunc \
+	dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=$blksz count=1 conv=notrunc \
  		>>$seqres.full 2>/dev/null || _fail "failed to write to test file"
  done
  _scratch_sync
@@ -71,7 +72,7 @@ echo >> $seqres.full
  echo "Fill in prealloc space; fill holes at offsets:" >> $seqres.full
  for i in `seq 2 2 1023`; do
  	echo -n "$i " >> $seqres.full
-	dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=4K count=1 conv=notrunc \
+	dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=$blksz count=1 conv=notrunc \
  		>>$seqres.full 2>/dev/null || _fail "failed to fill test file"
  done
  _scratch_sync
#27 lore raw

Re: [PATCH 7/7] generic/274: Make the test compatible with all blocksizes.

Quoted 20 lines · Qu Wenruo
Quoted 17 lines · Nirjhar Roy (IBM)

On btrfs with 64k blocksize on powerpc with 64k pagesize it failed with the following error:

      ------------------------------
      preallocation test
      ------------------------------
     -done
     +failed to write to test file
     +(see /home/xfstests-dev/results//btrfs_64k/generic/274.full for details)
     ...
So, this test is written with 4K block size in mind. As we can see, it first creates a file of size 4K and then fallocates 4MB beyond the EOF.
Then there are 2 loops - one that fragments at alternate blocks and the other punches holes in the remaining alternate blocks. Hence, the test fails in 64k block size due to incorrect calculations.

Quoted 5 lines
Quoted 3 lines

Fix this test by making the test scale with the block size, that is the offset, filesize and the assumed blocksize matches/scales with the actual blocksize of the underlying filesystem.

Again, just enlarge the block size from 4K to 64K, then all block size will work.

Okay.

--NR

Quoted 74 lines

Thanks,
Qu

Quoted 69 lines

---
  tests/generic/274 | 21 +++++++++++----------

  1 file changed, 11 insertions(+), 10 deletions(-)
tests/generic/274 +0 -0
index 916c7173..4ea42f30 100755
@@ -40,30 +40,31 @@ _scratch_unmount 2>/dev/null

  _scratch_mkfs_sized $((2 * 1024 * 1024 * 1024)) >>$seqres.full 2>&1 _scratch_mount
  -# Create a 4k file and Allocate 4M past EOF on that file -$XFS_IO_PROG -f -c "pwrite 0 4k" -c "falloc -k 4k 4m" $SCRATCH_MNT/test \
-    >>$seqres.full 2>&1 || _fail "failed to create test file"

tests/generic/274 +3 -0
+blksz=`_get_block_size $SCRATCH_MNT`
+scale=$(( blksz / 1024 ))
+# Create a blocksize worth file and Allocate a large file past EOF 

on that file
+$XFS_IO_PROG -f -c "pwrite -b $blksz 0 $blksz" -c "falloc -k $blksz $(( 1 * 1024 * 1024 * scale ))" \
+    $SCRATCH_MNT/test >>$seqres.full 2>&1 || _fail "failed to create test file"
    # Fill the rest of the fs completely
  # Note, this will show ENOSPC errors in $seqres.full, that's ok. echo "Fill fs with 1M IOs; ENOSPC expected" >> $seqres.full dd if=/dev/zero of=$SCRATCH_MNT/tmp1 bs=1M >>$seqres.full 2>&1

tests/generic/274 +2 -2
-echo "Fill fs with 4K IOs; ENOSPC expected" >> $seqres.full
-dd if=/dev/zero of=$SCRATCH_MNT/tmp2 bs=4K >>$seqres.full 2>&1
+echo "Fill fs with $blksz K IOs; ENOSPC expected" >> $seqres.full
+dd if=/dev/zero of=$SCRATCH_MNT/tmp2 bs=$blksz >>$seqres.full 2>&1

  _scratch_sync
  # Last effort, use O_SYNC

tests/generic/274 +0 -2
-echo "Fill fs with 4K DIOs; ENOSPC expected" >> $seqres.full
-dd if=/dev/zero of=$SCRATCH_MNT/tmp3 bs=4K oflag=sync >>$seqres.full 

2>&1

tests/generic/274 +2 -0
+echo "Fill fs with $blksz DIOs; ENOSPC expected" >> $seqres.full
+dd if=/dev/zero of=$SCRATCH_MNT/tmp3 bs=$blksz oflag=sync 
Quoted 1 line
Quoted 1 line

$seqres.full 2>&1

  # Save space usage info
  echo "Post-fill space:" >> $seqres.full
  df $SCRATCH_MNT >>$seqres.full 2>&1
-
  # Now attempt a write into all of the preallocated space -

  # in a very nasty way, badly fragmenting it and then filling it in.
  echo "Fill in prealloc space; fragment at offsets:" >> $seqres.full
  for i in `seq 1 2 1023`; do
      echo -n "$i " >> $seqres.full
-    dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=4K count=1 
conv=notrunc \
+    dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=$blksz count=1 
conv=notrunc \
          >>$seqres.full 2>/dev/null || _fail "failed to write to test file"
  done
  _scratch_sync
tests/generic/274 +0 -0
@@ -71,7 +72,7 @@ echo >> $seqres.full

  echo "Fill in prealloc space; fill holes at offsets:" >> $seqres.full for i in seq 2 2 1023; do
      echo -n "$i " >> $seqres.full
-    dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=4K count=1 conv=notrunc \
+    dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=$blksz count=1 conv=notrunc \
          >>$seqres.full 2>/dev/null || _fail "failed to fill test file" done
  _scratch_sync

Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore