Skip to contentSkip to conversation

btrfs: More test fixes for large block/node sizes

PATCH Superseded Nirjhar Roy (IBM) 3 participants 24 messages Sep 16, 2025 – Jan 21 Btrfs file system

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover btrfs: More test fixes for large block/node sizes

More btrfs test fixes. These tests were ran with 64k block size and they failed. Individual patches have the details. A previous patch series with some other test fixes was posted[1].

[1] https://lore.kernel.org/all/[email protected]/

Nirjhar Roy (IBM) (4):
generic/371: Fix the test to be compatible block sizes upto 64k generic/562: Make test compatible with block sizes till 64k btrfs/200: Make the test compatible with all supported block sizes btrfs/290: Make the test compatible with all supported block sizes

 tests/btrfs/200     |  8 ++++----
 tests/btrfs/200.out |  8 ++++----
 tests/btrfs/290     | 16 ++++++++--------
 tests/generic/371   |  2 +-
 tests/generic/562   |  2 +-
 5 files changed, 18 insertions(+), 18 deletions(-)

--
2.34.1

#1 lore raw

PATCH 1/4 generic/371: Fix the test to be compatible block sizes upto 64k

When this test was ran with btrfs with 64k sector/block size, it failed with

QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...

This is what is going on:

Let us see the following set of operations:

--- With 4k sector size ---

$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   82M   95M  47% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  137M   40M  78% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  162M   15M  92% /mnt1/scratch

Now let us repeat with 64k sector size
--- With 64k sector size ---

$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec)
$
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  104M   95M  53% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
pwrite: No space left on device

Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.

---
 tests/generic/371 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/371 +1 -1
index b312c450..95af308e 100755
@@ -19,7 +19,7 @@ _require_scratch
 _require_xfs_io_command "falloc"
 test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize"
 
-_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1
+_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1
 _scratch_mount
 
 # Disable speculative post-EOF preallocation on XFS, which can grow fast enough
Signature · 1 line
2.34.1
#12 lore raw

PATCH 2/4 generic/562: Make test compatible with block sizes till 64k

In reply to Nirjhar Roy (IBM)

This test fails with 64k sector size in btrfs. The reason for this is the need for additional space because of COW. When the reflink/clone of file bar into file foo is done, there is no additional space left for COW - the reason is that the metadata space usage is much higher with 64k node size. In order to verify this, I instrumented the test script and disabled COW for file foo and bar and the test passes in 64k (and runs faster too).

With 64k sector and node size (COW enabled) After pwriting foo and bar and before filling up the fs

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  324M  3.0M 100% /mnt1/scratch
After filling up the fs
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  441M  3.0M 100% /mnt1/scratch

With 64k sector and node size (COW disabled) After pwriting foo and bar and before filling up the fs

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  224M  231M  50% /mnt1/scratch
After filling up the fs
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  424M   31M  94% /mnt1/scratch

As we can see, with COW, the fs is completely full after filling up the fs but with COW disabled, we have some space left.

Fix this by increasing the fs size to 590M so that even with 64k node size and COW enabled, reflink has enough space to continue.

---
 tests/generic/562 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/562 +1 -1
index 03a66ff2..b9562730 100755
@@ -22,7 +22,7 @@ _require_scratch_reflink
 _require_test_program "punch-alternating"
 _require_xfs_io_command "fpunch"
 
-_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1
+_scratch_mkfs_sized $((590 * 1024 * 1024)) >>$seqres.full 2>&1
 _scratch_mount
 
 file_size=$(( 200 * 1024 * 1024 )) # 200Mb
Signature · 1 line
2.34.1
#17 lore raw

PATCH 3/4 btrfs/200: Make the test compatible with all supported block sizes

In reply to Nirjhar Roy (IBM)

This test fails on 64k block size with the following error:

At snapshot incr
OK
OK
+File foo does not have 2 shared extents in the base snapshot +/mnt/scratch/base/foo:
+ 0: [0..255]: 26624..26879
+File foo does not have 2 shared extents in the incr snapshot

So, basically after btrfs receive, the file /mnt/scratch/base/foo doesn't have any shared extents in the base snapshot. The reason is that during btrfs send, the extents are not cloned and instead they are written.
The following condition is responsible for the above behavior

in function clone_range():

/*
* Prevent cloning from a zero offset with a length matching the sector
* size because in some scenarios this will make the receiver fail. *
* For example, if in the source filesystem the extent at offset 0
* has a length of sectorsize and it was written using direct IO, then
* it can never be an inline extent (even if compression is enabled).
* Then this extent can be cloned in the original filesystem to a non
* zero file offset, but it may not be possible to clone in the
* destination filesystem because it can be inlined due to compression
* on the destination filesystem (as the receiver's write operations are
* always done using buffered IO). The same happens when the original
* filesystem does not have compression enabled but the destination
* filesystem has.
*/
if (clone_root->offset == 0 &&
len == sctx->send_root->fs_info->sectorsize)
return send_extent_data(sctx, dst_path, offset, len);

Since we are cloning from the first half [0 to 64k), clone_root->offset = 0 and len = 64k which is the sectorsize (sctx->send_root->fs_info->sectorsize). Fix this by increasing the file size and offsets to 128k so that len == sctx->send_root->fs_info->sectorsize is not true.

---
 tests/btrfs/200     | 8 ++++----
 tests/btrfs/200.out | 8 ++++----
 2 files changed, 8 insertions(+), 8 deletions(-)
tests/btrfs/200 +4 -4
index e62937a4..b53955ce 100755
@@ -38,14 +38,14 @@ _scratch_mount
 # Create our first test file, which has an extent that is shared only with
 # itself and no other files. We want to verify a full send operation will
 # clone the extent.
-$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \
+$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 128K 0 128K" $SCRATCH_MNT/foo \
 	| _filter_xfs_io
-$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 128K 128K" $SCRATCH_MNT/foo \
 	| _filter_xfs_io
 
 # Create out second test file which initially, for the first send operation,
 # only has a single extent that is not shared.
-$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \
+$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 128K 0 128K" $SCRATCH_MNT/bar \
 	| _filter_xfs_io
 
 _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base
@@ -56,7 +56,7 @@ $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \
 # Now clone the existing extent in file bar to itself at a different offset.
 # We want to verify the incremental send operation below will issue a clone
 # operation instead of a write operation.
-$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 128K 128K" $SCRATCH_MNT/bar \
 	| _filter_xfs_io
 
 _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incr
tests/btrfs/200.out +4 -4
index 306d9b24..a33b3c1e 100644
@@ -1,12 +1,12 @@
 QA output created by 200
-wrote 65536/65536 bytes at offset 0
+wrote 131072/131072 bytes at offset 0
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-linked 65536/65536 bytes at offset 65536
+linked 131072/131072 bytes at offset 131072
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-wrote 65536/65536 bytes at offset 0
+wrote 131072/131072 bytes at offset 0
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 At subvol SCRATCH_MNT/base
-linked 65536/65536 bytes at offset 65536
+linked 131072/131072 bytes at offset 131072
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 At subvol SCRATCH_MNT/incr
 At subvol base
Signature · 1 line
2.34.1
#21 lore raw

PATCH 4/4 btrfs/290: Make the test compatible with all supported block sizes

In reply to Nirjhar Roy (IBM)

This test fails with 64k block size with the following error:

punch
pread: Input/output error
pread: Input/output error
+ERROR: couldn't find extent 4096 for inode 261
plug
-pread: Input/output error
-pread: Input/output error
...

The reason is that, some of the subtests are written with 4k blocksize in mind. Fix the test by making the offsets and sizes to 64k so that it becomes compatible/aligned with all supported block sizes.

---
 tests/btrfs/290 | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
tests/btrfs/290 +8 -8
index 04563dfe..fecec473 100755
@@ -106,15 +106,15 @@ corrupt_reg_to_prealloc() {
 # corrupt a file by punching a hole
 corrupt_punch_hole() {
 	local f=$SCRATCH_MNT/punch
-	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
+	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $f
 	local ino=$(get_ino $f)
 	# make a new extent in the middle, sync so the writes don't coalesce
 	$XFS_IO_PROG -c sync $SCRATCH_MNT
-	$XFS_IO_PROG -fc "pwrite -q -S 0x59 4096 4096" $f
+	$XFS_IO_PROG -fc "pwrite -q -S 0x59 64k 64k" $f
 	_fsv_enable $f
 	_scratch_unmount
 	# change disk_bytenr to 0, representing a hole
-	$BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr --value 0 \
+	$BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr --value 0 \
 								    $SCRATCH_DEV
 	_scratch_mount
 	validate $f
@@ -123,14 +123,14 @@ corrupt_punch_hole() {
 # plug hole
 corrupt_plug_hole() {
 	local f=$SCRATCH_MNT/plug
-	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
+	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $f
 	local ino=$(get_ino $f)
-	$XFS_IO_PROG -fc "falloc 4k 4k" $f
+	$XFS_IO_PROG -fc "falloc 64k 64k" $f
 	_fsv_enable $f
 	_scratch_unmount
 	# change disk_bytenr to some value, plugging the hole
-	$BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr \
-						   --value 13639680 $SCRATCH_DEV
+	$BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr \
+						   --value 218234880 $SCRATCH_DEV
 	_scratch_mount
 	validate $f
 }
@@ -166,7 +166,7 @@ corrupt_root_hash() {
 # corrupt the Merkle tree data itself
 corrupt_merkle_tree() {
 	local f=$SCRATCH_MNT/merkle
-	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
+	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $f
 	local ino=$(get_ino $f)
 	_fsv_enable $f
 	_scratch_unmount
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k

In reply to Nirjhar Roy (IBM)
Quoted 73 lines · Nirjhar Roy (IBM)

When this test was ran with btrfs with 64k sector/block size, it failed with

QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...

This is what is going on:

Let us see the following set of operations:

--- With 4k sector size ---

$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   82M   95M  47% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  137M   40M  78% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  162M   15M  92% /mnt1/scratch

Now let us repeat with 64k sector size
--- With 64k sector size ---

$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec)
$
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  104M   95M  53% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
pwrite: No space left on device
Quoted 7 lines

Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.

Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).

Thanks,
Zorro

Quoted 23 lines
---
 tests/generic/371 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/371 +1 -1
index b312c450..95af308e 100755
@@ -19,7 +19,7 @@ _require_scratch
 _require_xfs_io_command "falloc"
 test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize"
 
-_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1
+_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1
 _scratch_mount
 
 # Disable speculative post-EOF preallocation on XFS, which can grow fast enough
Signature · 1 line
2.34.1
#13 lore raw

Re: [PATCH 2/4] generic/562: Make test compatible with block sizes till 64k

In reply to Nirjhar Roy (IBM)
Quoted 44 lines · Nirjhar Roy (IBM)

This test fails with 64k sector size in btrfs. The reason for this is the need for additional space because of COW. When the reflink/clone of file bar into file foo is done, there is no additional space left for COW - the reason is that the metadata space usage is much higher with 64k node size. In order to verify this, I instrumented the test script and disabled COW for file foo and bar and the test passes in 64k (and runs faster too).

With 64k sector and node size (COW enabled) After pwriting foo and bar and before filling up the fs

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  324M  3.0M 100% /mnt1/scratch
After filling up the fs
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  441M  3.0M 100% /mnt1/scratch

With 64k sector and node size (COW disabled) After pwriting foo and bar and before filling up the fs

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  224M  231M  50% /mnt1/scratch
After filling up the fs
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  424M   31M  94% /mnt1/scratch

As we can see, with COW, the fs is completely full after filling up the fs but with COW disabled, we have some space left.

Fix this by increasing the fs size to 590M so that even with 64k node size and COW enabled, reflink has enough space to continue.

---
 tests/generic/562 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/562 +0 -0
index 03a66ff2..b9562730 100755

tests/generic/562 _require_scratch_reflink · line 25

Quoted 5 lines
tests/generic/562 +1 -1
@@ -22,7 +22,7 @@ _require_scratch_reflink
 _require_test_program "punch-alternating"
 _require_xfs_io_command "fpunch"
 
-_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1
+_scratch_mkfs_sized $((590 * 1024 * 1024)) >>$seqres.full 2>&1

Filipe is the author of this test case, if he can help to make sure the 512MiB fs size isn't the necessary condition to reproduce the original bug he tried to uncover, I'm good to have this change, even change to bigger size :)

Thanks,
Zorro

Quoted 7 lines
tests/generic/562 +0 -0
 _scratch_mount
 
 file_size=$(( 200 * 1024 * 1024 )) # 200Mb
Signature · 1 line
2.34.1
Reviewed-by on v1 3/4
#18 lore raw

Re: [PATCH 3/4] btrfs/200: Make the test compatible with all supported block sizes

In reply to Nirjhar Roy (IBM)
Quoted 54 lines · Nirjhar Roy (IBM)

This test fails on 64k block size with the following error:

At snapshot incr
OK
OK
+File foo does not have 2 shared extents in the base snapshot +/mnt/scratch/base/foo:
+ 0: [0..255]: 26624..26879
+File foo does not have 2 shared extents in the incr snapshot

So, basically after btrfs receive, the file /mnt/scratch/base/foo doesn't have any shared extents in the base snapshot. The reason is that during btrfs send, the extents are not cloned and instead they are written.
The following condition is responsible for the above behavior

in function clone_range():

/*
* Prevent cloning from a zero offset with a length matching the sector
* size because in some scenarios this will make the receiver fail. *
* For example, if in the source filesystem the extent at offset 0
* has a length of sectorsize and it was written using direct IO, then
* it can never be an inline extent (even if compression is enabled).
* Then this extent can be cloned in the original filesystem to a non
* zero file offset, but it may not be possible to clone in the
* destination filesystem because it can be inlined due to compression
* on the destination filesystem (as the receiver's write operations are
* always done using buffered IO). The same happens when the original
* filesystem does not have compression enabled but the destination
* filesystem has.
*/
if (clone_root->offset == 0 &&
len == sctx->send_root->fs_info->sectorsize)
return send_extent_data(sctx, dst_path, offset, len);

Since we are cloning from the first half [0 to 64k), clone_root->offset = 0 and len = 64k which is the sectorsize (sctx->send_root->fs_info->sectorsize). Fix this by increasing the file size and offsets to 128k so that len == sctx->send_root->fs_info->sectorsize is not true.

---
 tests/btrfs/200     | 8 ++++----
 tests/btrfs/200.out | 8 ++++----
 2 files changed, 8 insertions(+), 8 deletions(-)
tests/btrfs/200 +0 -0
index e62937a4..b53955ce 100755

tests/btrfs/200 _scratch_mount · line 41

Quoted 5 lines
tests/btrfs/200 +1 -1
@@ -38,14 +38,14 @@ _scratch_mount
 # Create our first test file, which has an extent that is shared only with
 # itself and no other files. We want to verify a full send operation will
 # clone the extent.
-$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \
+$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 128K 0 128K" $SCRATCH_MNT/foo \

This change and explanation makes sense to me, and I think 128k won't break the "an extent" requirement (compare with 64k). So,

Quoted 45 lines
tests/btrfs/200 +3 -3
 	| _filter_xfs_io
-$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 128K 128K" $SCRATCH_MNT/foo \
 	| _filter_xfs_io
 
 # Create out second test file which initially, for the first send operation,
 # only has a single extent that is not shared.
-$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \
+$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 128K 0 128K" $SCRATCH_MNT/bar \
 	| _filter_xfs_io
 
 _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base
@@ -56,7 +56,7 @@ $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \
 # Now clone the existing extent in file bar to itself at a different offset.
 # We want to verify the incremental send operation below will issue a clone
 # operation instead of a write operation.
-$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 128K 128K" $SCRATCH_MNT/bar \
 	| _filter_xfs_io
 
 _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incr
tests/btrfs/200.out +4 -4
index 306d9b24..a33b3c1e 100644
@@ -1,12 +1,12 @@
 QA output created by 200
-wrote 65536/65536 bytes at offset 0
+wrote 131072/131072 bytes at offset 0
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-linked 65536/65536 bytes at offset 65536
+linked 131072/131072 bytes at offset 131072
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-wrote 65536/65536 bytes at offset 0
+wrote 131072/131072 bytes at offset 0
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 At subvol SCRATCH_MNT/base
-linked 65536/65536 bytes at offset 65536
+linked 131072/131072 bytes at offset 131072
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 At subvol SCRATCH_MNT/incr
 At subvol base
Signature · 1 line
2.34.1
#19 lore raw

Re: [PATCH 3/4] btrfs/200: Make the test compatible with all supported block sizes

Quoted 63 lines · Zorro Lang
Quoted 54 lines · Nirjhar Roy (IBM)

This test fails on 64k block size with the following error:

At snapshot incr
OK
OK
+File foo does not have 2 shared extents in the base snapshot +/mnt/scratch/base/foo:
+ 0: [0..255]: 26624..26879
+File foo does not have 2 shared extents in the incr snapshot

So, basically after btrfs receive, the file /mnt/scratch/base/foo doesn't have any shared extents in the base snapshot. The reason is that during btrfs send, the extents are not cloned and instead they are written.
The following condition is responsible for the above behavior

in function clone_range():

/*
* Prevent cloning from a zero offset with a length matching the sector
* size because in some scenarios this will make the receiver fail. *
* For example, if in the source filesystem the extent at offset 0
* has a length of sectorsize and it was written using direct IO, then
* it can never be an inline extent (even if compression is enabled).
* Then this extent can be cloned in the original filesystem to a non
* zero file offset, but it may not be possible to clone in the
* destination filesystem because it can be inlined due to compression
* on the destination filesystem (as the receiver's write operations are
* always done using buffered IO). The same happens when the original
* filesystem does not have compression enabled but the destination
* filesystem has.
*/
if (clone_root->offset == 0 &&
len == sctx->send_root->fs_info->sectorsize)
return send_extent_data(sctx, dst_path, offset, len);

Since we are cloning from the first half [0 to 64k), clone_root->offset = 0 and len = 64k which is the sectorsize (sctx->send_root->fs_info->sectorsize). Fix this by increasing the file size and offsets to 128k so that len == sctx->send_root->fs_info->sectorsize is not true.

---
  tests/btrfs/200     | 8 ++++----
  tests/btrfs/200.out | 8 ++++----
  2 files changed, 8 insertions(+), 8 deletions(-)
tests/btrfs/200 +0 -0
index e62937a4..b53955ce 100755

tests/btrfs/200 _scratch_mount · line 41

Quoted 5 lines
tests/btrfs/200 +1 -1
@@ -38,14 +38,14 @@ _scratch_mount
  # Create our first test file, which has an extent that is shared only with
  # itself and no other files. We want to verify a full send operation will
  # clone the extent.
-$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \
+$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 128K 0 128K" $SCRATCH_MNT/foo \

This change and explanation makes sense to me, and I think 128k won't break the "an extent" requirement (compare with 64k). So,

Quoted 1 line

Thank you.

--NR

Quoted 45 lines
Quoted 45 lines
tests/btrfs/200 +3 -3
  	| _filter_xfs_io
-$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 128K 128K" $SCRATCH_MNT/foo \
  	| _filter_xfs_io
 
  # Create out second test file which initially, for the first send operation,
  # only has a single extent that is not shared.
-$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \
+$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 128K 0 128K" $SCRATCH_MNT/bar \
  	| _filter_xfs_io
 
  _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base
@@ -56,7 +56,7 @@ $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \
  # Now clone the existing extent in file bar to itself at a different offset.
  # We want to verify the incremental send operation below will issue a clone
  # operation instead of a write operation.
-$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 128K 128K" $SCRATCH_MNT/bar \
  	| _filter_xfs_io
 
  _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incr
tests/btrfs/200.out +4 -4
index 306d9b24..a33b3c1e 100644
@@ -1,12 +1,12 @@
  QA output created by 200
-wrote 65536/65536 bytes at offset 0
+wrote 131072/131072 bytes at offset 0
  XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-linked 65536/65536 bytes at offset 65536
+linked 131072/131072 bytes at offset 131072
  XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-wrote 65536/65536 bytes at offset 0
+wrote 131072/131072 bytes at offset 0
  XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  At subvol SCRATCH_MNT/base
-linked 65536/65536 bytes at offset 65536
+linked 131072/131072 bytes at offset 131072
  XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
  At subvol SCRATCH_MNT/incr
  At subvol base
Signature · 1 line
2.34.1
Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
#14 lore raw

Re: [PATCH 2/4] generic/562: Make test compatible with block sizes till 64k

In reply to Zorro Lang
Quoted 48 lines · Zorro Lang
Quoted 47 lines · Nirjhar Roy (IBM)

This test fails with 64k sector size in btrfs. The reason for this is the need for additional space because of COW. When the reflink/clone of file bar into file foo is done, there is no additional space left for COW - the reason is that the metadata space usage is much higher with 64k node size. In order to verify this, I instrumented the test script and disabled COW for file foo and bar and the test passes in 64k (and runs faster too).

With 64k sector and node size (COW enabled) After pwriting foo and bar and before filling up the fs

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  324M  3.0M 100% /mnt1/scratch
After filling up the fs
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  441M  3.0M 100% /mnt1/scratch

With 64k sector and node size (COW disabled) After pwriting foo and bar and before filling up the fs

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  224M  231M  50% /mnt1/scratch
After filling up the fs
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  424M   31M  94% /mnt1/scratch

As we can see, with COW, the fs is completely full after filling up the fs but with COW disabled, we have some space left.

Fix this by increasing the fs size to 590M so that even with 64k node size and COW enabled, reflink has enough space to continue.

---
  tests/generic/562 | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/562 +0 -0
index 03a66ff2..b9562730 100755
@@ -22,7 +22,7 @@ _require_scratch_reflink
  _require_test_program "punch-alternating"
  _require_xfs_io_command "fpunch"
 
Quoted 6 lines

tests/generic/562 _require_scratch_reflink · line 25

Quoted 2 lines
tests/generic/562 +1 -1
-_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1
+_scratch_mkfs_sized $((590 * 1024 * 1024)) >>$seqres.full 2>&1

Filipe is the author of this test case, if he can help to make sure the 512MiB fs size isn't the necessary condition to reproduce the original bug he tried to uncover, I'm good to have this change, even change to bigger size :)

Okay, I will wait for Filipe's comment on this change. Thank you.

--NR

Quoted 11 lines

Thanks,
Zorro

Quoted 7 lines
tests/generic/562 +0 -0
  _scratch_mount
 
  file_size=$(( 200 * 1024 * 1024 )) # 200Mb
Signature · 1 line
2.34.1
Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
Reviewed-by
#15 lore raw

Re: [PATCH 2/4] generic/562: Make test compatible with block sizes till 64k

Quoted 51 lines · Nirjhar Roy (IBM)
Quoted 48 lines · Zorro Lang
Quoted 47 lines · Nirjhar Roy (IBM)

This test fails with 64k sector size in btrfs. The reason for this is the need for additional space because of COW. When the reflink/clone of file bar into file foo is done, there is no additional space left for COW - the reason is that the metadata space usage is much higher with 64k node size. In order to verify this, I instrumented the test script and disabled COW for file foo and bar and the test passes in 64k (and runs faster too).

With 64k sector and node size (COW enabled) After pwriting foo and bar and before filling up the fs

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  324M  3.0M 100% /mnt1/scratch
After filling up the fs
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  441M  3.0M 100% /mnt1/scratch

With 64k sector and node size (COW disabled) After pwriting foo and bar and before filling up the fs

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  224M  231M  50% /mnt1/scratch
After filling up the fs
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  424M   31M  94% /mnt1/scratch

As we can see, with COW, the fs is completely full after filling up the fs but with COW disabled, we have some space left.

Fix this by increasing the fs size to 590M so that even with 64k node size and COW enabled, reflink has enough space to continue.

---
  tests/generic/562 | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/562 +0 -0
index 03a66ff2..b9562730 100755
@@ -22,7 +22,7 @@ _require_scratch_reflink
  _require_test_program "punch-alternating"
  _require_xfs_io_command "fpunch"
 
Quoted 8 lines
Quoted 6 lines

tests/generic/562 _require_scratch_reflink · line 25

Quoted 2 lines
tests/generic/562 +1 -1
-_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1
+_scratch_mkfs_sized $((590 * 1024 * 1024)) >>$seqres.full 2>&1

Filipe is the author of this test case, if he can help to make sure the 512MiB fs size isn't the necessary condition to reproduce the original bug he tried to uncover, I'm good to have this change, even change to bigger size :)

Okay, I will wait for Filipe's comment on this change. Thank you.

This is probably ok.
This test was to exercise a bug fixed in 2019, commit 690a5dbfc513 ("Btrfs: fix ENOSPC errors, leading to transaction aborts, when cloning extents").
It's hard to confirm nowadays since the fix landed in kernel 5.4, trying a revert of that commit is not practical since there are tons of changes done on top of it, the alternative would be to test a 5.3 kernel or older.

The while loop below should ensure we try to fill all metadata space with files that have only inline extents, so in theory it should be ok for any fs size.

Btw, when you do a patchset that modifies btrfs tests, or even generic tests that were motivated by btrfs bugs, please cc the btrfs mailing list... I haven't noticed this before because the list was not cc'ed.

Quoted 19 lines

--NR

Quoted 11 lines

Thanks,
Zorro

Quoted 7 lines
tests/generic/562 +0 -1
  _scratch_mount
 
  file_size=$(( 200 * 1024 * 1024 )) # 200Mb
--

2.34.1

--
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore

Reviewed-by on v1 3/4
#20 lore raw

Re: [PATCH 3/4] btrfs/200: Make the test compatible with all supported block sizes

In reply to Nirjhar Roy (IBM)
Quoted 38 lines · Nirjhar Roy (IBM)

This test fails on 64k block size with the following error:

At snapshot incr
OK
OK
+File foo does not have 2 shared extents in the base snapshot +/mnt/scratch/base/foo:
+ 0: [0..255]: 26624..26879
+File foo does not have 2 shared extents in the incr snapshot

So, basically after btrfs receive, the file /mnt/scratch/base/foo doesn't have any shared extents in the base snapshot. The reason is that during btrfs send, the extents are not cloned and instead they are written.
The following condition is responsible for the above behavior

in function clone_range():

/*
* Prevent cloning from a zero offset with a length matching the sector
* size because in some scenarios this will make the receiver fail. *
* For example, if in the source filesystem the extent at offset 0
* has a length of sectorsize and it was written using direct IO, then
* it can never be an inline extent (even if compression is enabled).
* Then this extent can be cloned in the original filesystem to a non
* zero file offset, but it may not be possible to clone in the
* destination filesystem because it can be inlined due to compression
* on the destination filesystem (as the receiver's write operations are
* always done using buffered IO). The same happens when the original
* filesystem does not have compression enabled but the destination
* filesystem has.
*/
if (clone_root->offset == 0 &&
len == sctx->send_root->fs_info->sectorsize)
return send_extent_data(sctx, dst_path, offset, len);

Quoted 7 lines

Since we are cloning from the first half [0 to 64k), clone_root->offset = 0 and len = 64k which is the sectorsize (sctx->send_root->fs_info->sectorsize). Fix this by increasing the file size and offsets to 128k so that len == sctx->send_root->fs_info->sectorsize is not true.

Please always cc the btrfs mailing list when doing changes to btrfs test cases.

Quoted 60 lines
---
 tests/btrfs/200     | 8 ++++----
 tests/btrfs/200.out | 8 ++++----
 2 files changed, 8 insertions(+), 8 deletions(-)
tests/btrfs/200 +4 -4
index e62937a4..b53955ce 100755
@@ -38,14 +38,14 @@ _scratch_mount
 # Create our first test file, which has an extent that is shared only with
 # itself and no other files. We want to verify a full send operation will
 # clone the extent.
-$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \
+$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 128K 0 128K" $SCRATCH_MNT/foo \
        | _filter_xfs_io
-$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 128K 128K" $SCRATCH_MNT/foo \
        | _filter_xfs_io
 
 # Create out second test file which initially, for the first send operation,
 # only has a single extent that is not shared.
-$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \
+$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 128K 0 128K" $SCRATCH_MNT/bar \
        | _filter_xfs_io
 
 _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base
@@ -56,7 +56,7 @@ $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \
 # Now clone the existing extent in file bar to itself at a different offset.
 # We want to verify the incremental send operation below will issue a clone
 # operation instead of a write operation.
-$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \
+$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 128K 128K" $SCRATCH_MNT/bar \
        | _filter_xfs_io
 
 _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incr
tests/btrfs/200.out +4 -5
index 306d9b24..a33b3c1e 100644
@@ -1,12 +1,12 @@
 QA output created by 200
-wrote 65536/65536 bytes at offset 0
+wrote 131072/131072 bytes at offset 0
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-linked 65536/65536 bytes at offset 65536
+linked 131072/131072 bytes at offset 131072
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-wrote 65536/65536 bytes at offset 0
+wrote 131072/131072 bytes at offset 0
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 At subvol SCRATCH_MNT/base
-linked 65536/65536 bytes at offset 65536
+linked 131072/131072 bytes at offset 131072
 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 At subvol SCRATCH_MNT/incr
 At subvol base
--

2.34.1

#16 lore raw

Re: [PATCH 2/4] generic/562: Make test compatible with block sizes till 64k

In reply to Filipe Manana
Quoted 72 lines · Filipe Manana
Quoted 50 lines · Nirjhar Roy (IBM)
Quoted 48 lines · Zorro Lang
Quoted 47 lines · Nirjhar Roy (IBM)

This test fails with 64k sector size in btrfs. The reason for this is the need for additional space because of COW. When the reflink/clone of file bar into file foo is done, there is no additional space left for COW - the reason is that the metadata space usage is much higher with 64k node size. In order to verify this, I instrumented the test script and disabled COW for file foo and bar and the test passes in 64k (and runs faster too).

With 64k sector and node size (COW enabled) After pwriting foo and bar and before filling up the fs

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  324M  3.0M 100% /mnt1/scratch
After filling up the fs
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  441M  3.0M 100% /mnt1/scratch

With 64k sector and node size (COW disabled) After pwriting foo and bar and before filling up the fs

Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  224M  231M  50% /mnt1/scratch
After filling up the fs
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      512M  424M   31M  94% /mnt1/scratch

As we can see, with COW, the fs is completely full after filling up the fs but with COW disabled, we have some space left.

Fix this by increasing the fs size to 590M so that even with 64k node size and COW enabled, reflink has enough space to continue.

---
   tests/generic/562 | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/562 +0 -0
index 03a66ff2..b9562730 100755
@@ -22,7 +22,7 @@ _require_scratch_reflink
   _require_test_program "punch-alternating"
   _require_xfs_io_command "fpunch"
 
Quoted 7 lines
Quoted 6 lines

tests/generic/562 _require_scratch_reflink · line 25

Quoted 2 lines
tests/generic/562 +1 -1
-_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1
+_scratch_mkfs_sized $((590 * 1024 * 1024)) >>$seqres.full 2>&1

Filipe is the author of this test case, if he can help to make sure the 512MiB fs size isn't the necessary condition to reproduce the original bug he tried to uncover, I'm good to have this change, even change to bigger size :)

Okay, I will wait for Filipe's comment on this change. Thank you.

This is probably ok.
This test was to exercise a bug fixed in 2019, commit 690a5dbfc513 ("Btrfs: fix ENOSPC errors, leading to transaction aborts, when cloning extents").
It's hard to confirm nowadays since the fix landed in kernel 5.4, trying a revert of that commit is not practical since there are tons of changes done on top of it, the alternative would be to test a 5.3 kernel or older.

The while loop below should ensure we try to fill all metadata space with files that have only inline extents, so in theory it should be ok for any fs size.

Quoted 5 lines

Btw, when you do a patchset that modifies btrfs tests, or even generic tests that were motivated by btrfs bugs, please cc the btrfs mailing list... I haven't noticed this before because the list was not cc'ed.

Yes, I usually cc the btrfs mailing list but I missed it this time, sorry about that.

--NR

Quoted 19 lines
Quoted 19 lines

--NR

Quoted 10 lines

Thanks,
Zorro

Quoted 7 lines
tests/generic/562 +0 -1
   _scratch_mount
 
   file_size=$(( 200 * 1024 * 1024 )) # 200Mb
--

2.34.1

--
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore

Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
#22 lore raw

Re: [PATCH 4/4] btrfs/290: Make the test compatible with all supported block sizes

In reply to Nirjhar Roy (IBM)
Quoted 12 lines · Nirjhar Roy (IBM)

This test fails with 64k block size with the following error:

punch
pread: Input/output error
pread: Input/output error
+ERROR: couldn't find extent 4096 for inode 261
plug
-pread: Input/output error
-pread: Input/output error
...

Quoted 2 lines

The reason is that, some of the subtests are written with 4k blocksize in mind. Fix the test by making the offsets and sizes to 64k so

"... offsets and sizes to 64k..." -> "... offsets and sizes multiples of 64K..."

Quoted 13 lines

that it becomes compatible/aligned with all supported block sizes.

---
 tests/btrfs/290 | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
tests/btrfs/290 +0 -0
index 04563dfe..fecec473 100755

tests/btrfs/290 corrupt_reg_to_prealloc() { · line 109

Quoted 5 lines
tests/btrfs/290 +1 -1
@@ -106,15 +106,15 @@ corrupt_reg_to_prealloc() {
 # corrupt a file by punching a hole
 corrupt_punch_hole() {
        local f=$SCRATCH_MNT/punch
-       $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
+       $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $f

Can you please make this more readable and type 192K instead of 196608?

tests/btrfs/290 corrupt_reg_to_prealloc() { · line 113

Quoted 5 lines
tests/btrfs/290 +1 -1
        local ino=$(get_ino $f)
        # make a new extent in the middle, sync so the writes don't coalesce
        $XFS_IO_PROG -c sync $SCRATCH_MNT
-       $XFS_IO_PROG -fc "pwrite -q -S 0x59 4096 4096" $f
+       $XFS_IO_PROG -fc "pwrite -q -S 0x59 64k 64k" $f

Here you use 64k instead of 65536, which is more readable.

Quoted 9 lines
tests/btrfs/290 +1 -1
        _fsv_enable $f
        _scratch_unmount
        # change disk_bytenr to 0, representing a hole
-       $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr --value 0 \
+       $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr --value 0 \
                                                                    $SCRATCH_DEV
        _scratch_mount
        validate $f

tests/btrfs/290 corrupt_punch_hole() { · line 126

Quoted 5 lines
tests/btrfs/290 +1 -1
@@ -123,14 +123,14 @@ corrupt_punch_hole() {
 # plug hole
 corrupt_plug_hole() {
        local f=$SCRATCH_MNT/plug
-       $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
+       $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $f

Same here, 192K.

Quoted 14 lines
tests/btrfs/290 +3 -3
        local ino=$(get_ino $f)
-       $XFS_IO_PROG -fc "falloc 4k 4k" $f
+       $XFS_IO_PROG -fc "falloc 64k 64k" $f
        _fsv_enable $f
        _scratch_unmount
        # change disk_bytenr to some value, plugging the hole
-       $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr \
-                                                  --value 13639680 $SCRATCH_DEV
+       $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr \
+                                                  --value 218234880 $SCRATCH_DEV
        _scratch_mount
        validate $f
 }

tests/btrfs/290 corrupt_root_hash() { · line 169

Quoted 5 lines
tests/btrfs/290 +1 -1
@@ -166,7 +166,7 @@ corrupt_root_hash() {
 # corrupt the Merkle tree data itself
 corrupt_merkle_tree() {
        local f=$SCRATCH_MNT/merkle
-       $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
+       $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $f

Same here, 192K.

Also please always cc the btrfs mailing list for changes to btrfs tests.

Quoted 6 lines
tests/btrfs/290 +0 -1
        local ino=$(get_ino $f)
        _fsv_enable $f
        _scratch_unmount
--

2.34.1

#23 lore raw

Re: [PATCH 4/4] btrfs/290: Make the test compatible with all supported block sizes

Quoted 13 lines · Filipe Manana
Quoted 11 lines · Nirjhar Roy (IBM)

This test fails with 64k block size with the following error:

punch
pread: Input/output error
pread: Input/output error
+ERROR: couldn't find extent 4096 for inode 261
plug
-pread: Input/output error
-pread: Input/output error
...

Quoted 3 lines
Quoted 2 lines

The reason is that, some of the subtests are written with 4k blocksize in mind. Fix the test by making the offsets and sizes to 64k so

"... offsets and sizes to 64k..." -> "... offsets and sizes multiples of 64K..."

Noted.

Quoted 65 lines
Quoted 14 lines

that it becomes compatible/aligned with all supported block sizes.

---
  tests/btrfs/290 | 16 ++++++++--------
  1 file changed, 8 insertions(+), 8 deletions(-)
tests/btrfs/290 +0 -0
index 04563dfe..fecec473 100755

tests/btrfs/290 corrupt_reg_to_prealloc() { · line 109

Quoted 5 lines
tests/btrfs/290 +1 -1
@@ -106,15 +106,15 @@ corrupt_reg_to_prealloc() {
  # corrupt a file by punching a hole
  corrupt_punch_hole() {
         local f=$SCRATCH_MNT/punch
-       $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
+       $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $f

Can you please make this more readable and type 192K instead of 196608?

tests/btrfs/290 corrupt_reg_to_prealloc() { · line 113

Quoted 5 lines
tests/btrfs/290 +1 -1
         local ino=$(get_ino $f)
         # make a new extent in the middle, sync so the writes don't coalesce
         $XFS_IO_PROG -c sync $SCRATCH_MNT
-       $XFS_IO_PROG -fc "pwrite -q -S 0x59 4096 4096" $f
+       $XFS_IO_PROG -fc "pwrite -q -S 0x59 64k 64k" $f

Here you use 64k instead of 65536, which is more readable.

Quoted 9 lines
tests/btrfs/290 +1 -1
         _fsv_enable $f
         _scratch_unmount
         # change disk_bytenr to 0, representing a hole
-       $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr --value 0 \
+       $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr --value 0 \
                                                                     $SCRATCH_DEV
         _scratch_mount
         validate $f

tests/btrfs/290 corrupt_punch_hole() { · line 126

Quoted 5 lines
tests/btrfs/290 +1 -1
@@ -123,14 +123,14 @@ corrupt_punch_hole() {
  # plug hole
  corrupt_plug_hole() {
         local f=$SCRATCH_MNT/plug
-       $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
+       $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $f

Same here, 192K.

Quoted 14 lines
tests/btrfs/290 +3 -3
         local ino=$(get_ino $f)
-       $XFS_IO_PROG -fc "falloc 4k 4k" $f
+       $XFS_IO_PROG -fc "falloc 64k 64k" $f
         _fsv_enable $f
         _scratch_unmount
         # change disk_bytenr to some value, plugging the hole
-       $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr \
-                                                  --value 13639680 $SCRATCH_DEV
+       $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr \
+                                                  --value 218234880 $SCRATCH_DEV
         _scratch_mount
         validate $f
  }

tests/btrfs/290 corrupt_root_hash() { · line 169

Quoted 5 lines
tests/btrfs/290 +1 -1
@@ -166,7 +166,7 @@ corrupt_root_hash() {
  # corrupt the Merkle tree data itself
  corrupt_merkle_tree() {
         local f=$SCRATCH_MNT/merkle
-       $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
+       $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $f

Same here, 192K.

Quoted 1 line

Also please always cc the btrfs mailing list for changes to btrfs tests.

Sure. I will change all the 196608 to 192k and send an updated version.

--NR

Quoted 8 lines
Quoted 8 lines
tests/btrfs/290 +0 -1
         local ino=$(get_ino $f)
         _fsv_enable $f
         _scratch_unmount
--

2.34.1

Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
#3 lore raw

Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k

In reply to Zorro Lang
Quoted 81 lines · Zorro Lang
Quoted 73 lines · Nirjhar Roy (IBM)

When this test was ran with btrfs with 64k sector/block size, it failed with

QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...

This is what is going on:

Let us see the following set of operations:

--- With 4k sector size ---

$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   82M   95M  47% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  137M   40M  78% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  162M   15M  92% /mnt1/scratch

Now let us repeat with 64k sector size
--- With 64k sector size ---

$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec)
$
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  104M   95M  53% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
pwrite: No space left on device
Quoted 7 lines

Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.

Quoted 4 lines

Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).

Hi Filipe, Zorro, Wang,

The original commit message of generic/371 says that this test catches some excess space usage issues. Is(Are) there any patch(es) that fix this issue - so that I can remove the commits and check if the test expectedly fails with slightly large fssize i.e, 330M?

I did find some related commits [1] and I ran the test with 330M and 256M(the default size) after removing the commits[1] but the test passes with both the filesystem sizes. So I am guessing, this is not the patch that can test generic/371.

[1] https://lore.kernel.org/linux-btrfs/[email protected]/

--NR

Quoted 26 lines

Thanks,
Zorro

Quoted 22 lines
---
  tests/generic/371 | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/371 +1 -1
index b312c450..95af308e 100755
@@ -19,7 +19,7 @@ _require_scratch
  _require_xfs_io_command "falloc"
  test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize"
 
-_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1
+_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1
  _scratch_mount
 
  # Disable speculative post-EOF preallocation on XFS, which can grow fast enough
Signature · 1 line
2.34.1
Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
#10 lore raw

Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k

In reply to Zorro Lang
Quoted 81 lines · Zorro Lang
Quoted 73 lines · Nirjhar Roy (IBM)

When this test was ran with btrfs with 64k sector/block size, it failed with

QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...

This is what is going on:

Let us see the following set of operations:

--- With 4k sector size ---

$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   82M   95M  47% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  137M   40M  78% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  162M   15M  92% /mnt1/scratch

Now let us repeat with 64k sector size
--- With 64k sector size ---

$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec)
$
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  104M   95M  53% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
pwrite: No space left on device
Quoted 7 lines

Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.

Quoted 4 lines

Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).

Hi Filipe, Zorro

The original commit message of generic/371 says that this test catches some excess space usage issues. Is(Are) there any patch(es) that fix this issue - so that I can remove the commits and check if the test expectedly fails with slightly large fssize i.e, 330M?

I did find some related commits [1] and I ran the test with 330M and 256M(the default size) after removing the commits[1] but the test passes with both the filesystem sizes. So I am guessing, this is not the patch that can test generic/371.

[1] https://lore.kernel.org/linux-btrfs/[email protected]/

--NR

Quoted 26 lines

Thanks,
Zorro

Quoted 22 lines
---
  tests/generic/371 | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/371 +1 -1
index b312c450..95af308e 100755
@@ -19,7 +19,7 @@ _require_scratch
  _require_xfs_io_command "falloc"
  test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize"
 
-_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1
+_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1
  _scratch_mount
 
  # Disable speculative post-EOF preallocation on XFS, which can grow fast enough
Signature · 1 line
2.34.1
Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
#4 lore raw

Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k

In reply to Nirjhar Roy (IBM)
Quoted 95 lines · Nirjhar Roy (IBM)
Quoted 81 lines · Zorro Lang
Quoted 73 lines · Nirjhar Roy (IBM)

When this test was ran with btrfs with 64k sector/block size, it failed with

QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...

This is what is going on:

Let us see the following set of operations:

--- With 4k sector size ---

$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   82M   95M  47% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  137M   40M  78% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  162M   15M  92% /mnt1/scratch

Now let us repeat with 64k sector size
--- With 64k sector size ---

$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec)
$
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  104M   95M  53% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
pwrite: No space left on device
Quoted 7 lines

Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.

Quoted 4 lines

Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).

Hi Filipe, Zorro, Wang,

The original commit message of generic/371 says that this test catches some excess space usage issues. Is(Are) there any patch(es) that fix this issue - so that I can remove the commits and check if the test expectedly fails with slightly large fssize i.e, 330M?

Quoted 6 lines

I did find some related commits [1] and I ran the test with 330M and 256M(the default size) after removing the commits[1] but the test passes with both the filesystem sizes. So I am guessing, this is not the patch that can test generic/371.

[1] https://lore.kernel.org/linux-btrfs/[email protected]/

Hi,

I'm not familiar with btrfs patches, but from the commit history, I suspect it *might* be related with below commit:

commit 18513091af9483ba84328d42092bd4d42a3c958f

Author: Wang Xiaoguang <[email protected]>
Date:   Mon Jul 25 15:51:40 2016 +0800

btrfs: update btrfs_space_info's bytes_may_use timely

Or "c0d2f6104e8ab2eb75e58e72494ad4b69c5227f8" :)

Thanks,
Zorro

Quoted 32 lines

--NR

Quoted 24 lines

Thanks,
Zorro

Quoted 20 lines
---
  tests/generic/371 | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/371 +1 -1
index b312c450..95af308e 100755
@@ -19,7 +19,7 @@ _require_scratch
  _require_xfs_io_command "falloc"
  test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize"
-_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1
+_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1
  _scratch_mount
  # Disable speculative post-EOF preallocation on XFS, which can grow fast enough
Signature · 1 line
2.34.1
Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
#5 lore raw

Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k

Quoted 105 lines · Zorro Lang
Quoted 93 lines · Nirjhar Roy (IBM)
Quoted 81 lines · Zorro Lang
Quoted 73 lines · Nirjhar Roy (IBM)

When this test was ran with btrfs with 64k sector/block size, it failed with

QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...

This is what is going on:

Let us see the following set of operations:

--- With 4k sector size ---

$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   82M   95M  47% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  137M   40M  78% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  162M   15M  92% /mnt1/scratch

Now let us repeat with 64k sector size
--- With 64k sector size ---

$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec)
$
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  104M   95M  53% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
pwrite: No space left on device
Quoted 7 lines

Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.

Quoted 4 lines

Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).

Hi Filipe, Zorro, Wang,

The original commit message of generic/371 says that this test catches some excess space usage issues. Is(Are) there any patch(es) that fix this issue - so that I can remove the commits and check if the test expectedly fails with slightly large fssize i.e, 330M?

Quoted 6 lines

I did find some related commits [1] and I ran the test with 330M and 256M(the default size) after removing the commits[1] but the test passes with both the filesystem sizes. So I am guessing, this is not the patch that can test generic/371.

[1] https://lore.kernel.org/linux-btrfs/[email protected]/

Hi,

I'm not familiar with btrfs patches, but from the commit history, I suspect it *might* be related with below commit:

Quoted 7 lines

commit 18513091af9483ba84328d42092bd4d42a3c958f

Author: Wang Xiaoguang <[email protected]>
Date:   Mon Jul 25 15:51:40 2016 +0800

btrfs: update btrfs_space_info's bytes_may_use timely

Or "c0d2f6104e8ab2eb75e58e72494ad4b69c5227f8" :)

Thank you for the pointers. I tried running the test by checking out the commits before the above 2 mentioned commits, but the test seems to pass with the existing fs size.

--NR

Quoted 34 lines

Thanks,
Zorro

Quoted 30 lines

--NR

Quoted 23 lines

Thanks,
Zorro

Quoted 20 lines
---
   tests/generic/371 | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/371 +1 -1
index b312c450..95af308e 100755
@@ -19,7 +19,7 @@ _require_scratch
   _require_xfs_io_command "falloc"
   test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize"
-_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1
+_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1
   _scratch_mount
   # Disable speculative post-EOF preallocation on XFS, which can grow fast enough
Signature · 1 line
2.34.1
Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
#6 lore raw

Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k

In reply to Zorro Lang
Quoted 105 lines · Zorro Lang
Quoted 93 lines · Nirjhar Roy (IBM)
Quoted 81 lines · Zorro Lang
Quoted 73 lines · Nirjhar Roy (IBM)

When this test was ran with btrfs with 64k sector/block size, it failed with

QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...

This is what is going on:

Let us see the following set of operations:

--- With 4k sector size ---

$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   82M   95M  47% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  137M   40M  78% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  162M   15M  92% /mnt1/scratch

Now let us repeat with 64k sector size
--- With 64k sector size ---

$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec)
$
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  104M   95M  53% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
pwrite: No space left on device
Quoted 7 lines

Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.

Quoted 4 lines

Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).

Hi Filipe, Zorro, Wang,

The original commit message of generic/371 says that this test catches some excess space usage issues. Is(Are) there any patch(es) that fix this issue - so that I can remove the commits and check if the test expectedly fails with slightly large fssize i.e, 330M?

Quoted 6 lines

I did find some related commits [1] and I ran the test with 330M and 256M(the default size) after removing the commits[1] but the test passes with both the filesystem sizes. So I am guessing, this is not the patch that can test generic/371.

[1] https://lore.kernel.org/linux-btrfs/[email protected]/

Hi,

I'm not familiar with btrfs patches, but from the commit history, I suspect it *might* be related with below commit:

Quoted 7 lines

commit 18513091af9483ba84328d42092bd4d42a3c958f

Author: Wang Xiaoguang <[email protected]>
Date:   Mon Jul 25 15:51:40 2016 +0800

btrfs: update btrfs_space_info's bytes_may_use timely

Or "c0d2f6104e8ab2eb75e58e72494ad4b69c5227f8" :)

Thank you for the pointers. I tried running the test by checking out the commits before the above 2 mentioned commits, but the test seems to pass with the existing fs size.

--NR

Quoted 34 lines

Thanks,
Zorro

Quoted 30 lines

--NR

Quoted 23 lines

Thanks,
Zorro

Quoted 20 lines
---
   tests/generic/371 | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/371 +1 -1
index b312c450..95af308e 100755
@@ -19,7 +19,7 @@ _require_scratch
   _require_xfs_io_command "falloc"
   test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize"
-_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1
+_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1
   _scratch_mount
   # Disable speculative post-EOF preallocation on XFS, which can grow fast enough
Signature · 1 line
2.34.1
Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
#7 lore raw

Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k

In reply to Zorro Lang
Quoted 109 lines · Zorro Lang
Quoted 93 lines · Nirjhar Roy (IBM)
Quoted 81 lines · Zorro Lang
Quoted 73 lines · Nirjhar Roy (IBM)

When this test was ran with btrfs with 64k sector/block size, it failed with

QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...

This is what is going on:

Let us see the following set of operations:

--- With 4k sector size ---

$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   82M   95M  47% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  137M   40M  78% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  162M   15M  92% /mnt1/scratch

Now let us repeat with 64k sector size
--- With 64k sector size ---

$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec)
$
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  104M   95M  53% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
pwrite: No space left on device
Quoted 7 lines

Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.

Quoted 4 lines

Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).

Hi Filipe, Zorro, Wang,

The original commit message of generic/371 says that this test catches some excess space usage issues. Is(Are) there any patch(es) that fix this issue - so that I can remove the commits and check if the test expectedly fails with slightly large fssize i.e, 330M?

Quoted 6 lines

I did find some related commits [1] and I ran the test with 330M and 256M(the default size) after removing the commits[1] but the test passes with both the filesystem sizes. So I am guessing, this is not the patch that can test generic/371.

[1] https://lore.kernel.org/linux-btrfs/[email protected]/

Hi,

I'm not familiar with btrfs patches, but from the commit history, I suspect it *might* be related with below commit:

commit 18513091af9483ba84328d42092bd4d42a3c958f

Author: Wang Xiaoguang <[email protected]>
Date:   Mon Jul 25 15:51:40 2016 +0800
Quoted 6 lines

btrfs: update btrfs_space_info's bytes_may_use timely

Or "c0d2f6104e8ab2eb75e58e72494ad4b69c5227f8" :)

Thanks,
Zorro

Hi Zorro, Filipe and Qu

Thank you for the pointers. I tried running the test reverting the above 2 mentioned commits, but the test seems to pass without these commits, so probably these are not the ones. I am not able to find any other relevant commits and the test case is also very old. Any further ideas?

--NR

Quoted 31 lines
Quoted 31 lines

--NR

Quoted 23 lines

Thanks,
Zorro

Quoted 20 lines
---
   tests/generic/371 | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/371 +1 -1
index b312c450..95af308e 100755
@@ -19,7 +19,7 @@ _require_scratch
   _require_xfs_io_command "falloc"
   test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize"
-_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1
+_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1
   _scratch_mount
   # Disable speculative post-EOF preallocation on XFS, which can grow fast enough
Signature · 1 line
2.34.1
Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
#8 lore raw

Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k

Quoted 118 lines · Nirjhar Roy (IBM)
Quoted 109 lines · Zorro Lang
Quoted 93 lines · Nirjhar Roy (IBM)
Quoted 81 lines · Zorro Lang
Quoted 73 lines · Nirjhar Roy (IBM)

When this test was ran with btrfs with 64k sector/block size, it failed with

QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...

This is what is going on:

Let us see the following set of operations:

--- With 4k sector size ---

$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   82M   95M  47% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  137M   40M  78% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  162M   15M  92% /mnt1/scratch

Now let us repeat with 64k sector size
--- With 64k sector size ---

$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec)
$
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  104M   95M  53% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
pwrite: No space left on device
Quoted 7 lines

Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.

Quoted 4 lines

Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).

Hi Filipe, Zorro, Wang,

The original commit message of generic/371 says that this test catches some excess space usage issues. Is(Are) there any patch(es) that fix this issue - so that I can remove the commits and check if the test expectedly fails with slightly large fssize i.e, 330M?

Quoted 6 lines

I did find some related commits [1] and I ran the test with 330M and 256M(the default size) after removing the commits[1] but the test passes with both the filesystem sizes. So I am guessing, this is not the patch that can test generic/371.

[1] https://lore.kernel.org/linux-btrfs/[email protected]/

Hi,

I'm not familiar with btrfs patches, but from the commit history, I suspect it *might* be related with below commit:

commit 18513091af9483ba84328d42092bd4d42a3c958f

Author: Wang Xiaoguang <[email protected]>
Date:   Mon Jul 25 15:51:40 2016 +0800
Quoted 6 lines

btrfs: update btrfs_space_info's bytes_may_use timely

Or "c0d2f6104e8ab2eb75e58e72494ad4b69c5227f8" :)

Thanks,
Zorro

Quoted 6 lines

Hi Zorro, Filipe and Qu

Thank you for the pointers. I tried running the test reverting the above 2 mentioned commits, but the test seems to pass without these commits, so probably these are not the ones. I am not able to find any other relevant commits and the test case is also very old. Any further ideas?

Thanks, I don't have any objection on this patch now :) As this patchset still has unresolved review points from btrfs list, I think you can change and resend this patchset to get further reviewing.

Thanks,
Zorro

Quoted 39 lines

--NR

Quoted 31 lines
Quoted 31 lines

--NR

Quoted 23 lines

Thanks,
Zorro

Quoted 20 lines
---
   tests/generic/371 | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/371 +1 -1
index b312c450..95af308e 100755
@@ -19,7 +19,7 @@ _require_scratch
   _require_xfs_io_command "falloc"
   test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize"
-_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1
+_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1
   _scratch_mount
   # Disable speculative post-EOF preallocation on XFS, which can grow fast enough
Signature · 1 line
2.34.1
Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
#9 lore raw

Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k

Quoted 119 lines · Zorro Lang
Quoted 118 lines · Nirjhar Roy (IBM)
Quoted 109 lines · Zorro Lang
Quoted 93 lines · Nirjhar Roy (IBM)
Quoted 81 lines · Zorro Lang
Quoted 73 lines · Nirjhar Roy (IBM)

When this test was ran with btrfs with 64k sector/block size, it failed with

QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...

This is what is going on:

Let us see the following set of operations:

--- With 4k sector size ---

$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   82M   95M  47% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  137M   40M  78% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  162M   15M  92% /mnt1/scratch

Now let us repeat with 64k sector size
--- With 64k sector size ---

$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec)
$
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  104M   95M  53% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
pwrite: No space left on device
Quoted 7 lines

Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.

Quoted 4 lines

Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).

Hi Filipe, Zorro, Wang,

The original commit message of generic/371 says that this test catches some excess space usage issues. Is(Are) there any patch(es) that fix this issue - so that I can remove the commits and check if the test expectedly fails with slightly large fssize i.e, 330M?

Quoted 6 lines

I did find some related commits [1] and I ran the test with 330M and 256M(the default size) after removing the commits[1] but the test passes with both the filesystem sizes. So I am guessing, this is not the patch that can test generic/371.

[1] https://lore.kernel.org/linux-btrfs/[email protected]/

Hi,

I'm not familiar with btrfs patches, but from the commit history, I suspect it *might* be related with below commit:

commit 18513091af9483ba84328d42092bd4d42a3c958f

Author: Wang Xiaoguang <[email protected]>
Date:   Mon Jul 25 15:51:40 2016 +0800
Quoted 6 lines

btrfs: update btrfs_space_info's bytes_may_use timely

Or "c0d2f6104e8ab2eb75e58e72494ad4b69c5227f8" :)

Thanks,
Zorro

Hi Zorro, Filipe and Qu

Quoted 7 lines
Quoted 4 lines

Thank you for the pointers. I tried running the test reverting the above 2 mentioned commits, but the test seems to pass without these commits, so probably these are not the ones. I am not able to find any other relevant commits and the test case is also very old. Any further ideas?

Thanks, I don't have any objection on this patch now :) As this patchset still has unresolved review points from btrfs list, I think you can change and resend this patchset to get further reviewing.

I have already re-sent this patchset excluding this particular patch, and the rest of patches are merged (i.e, the changes for generic/562, btrfs/{200, 290}. Only this one is left.

--NR

Quoted 41 lines

Thanks,
Zorro

Quoted 37 lines

--NR

Quoted 30 lines
Quoted 30 lines

--NR

Quoted 23 lines

Thanks,
Zorro

Quoted 20 lines
---
    tests/generic/371 | 2 +-
    1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/371 +1 -1
index b312c450..95af308e 100755
@@ -19,7 +19,7 @@ _require_scratch
    _require_xfs_io_command "falloc"
    test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize"
-_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1
+_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1
    _scratch_mount
    # Disable speculative post-EOF preallocation on XFS, which can grow fast enough
Signature · 1 line
2.34.1
Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
Signature · 3 lines
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
Reviewed-by on v1 1/4
#11 lore raw

Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k

In reply to Nirjhar Roy (IBM)
Quoted 81 lines · Nirjhar Roy (IBM)

When this test was ran with btrfs with 64k sector/block size, it failed with

QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...

This is what is going on:

Let us see the following set of operations:

--- With 4k sector size ---

$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  1.5M  175M   1% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   82M   95M  47% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  137M   40M  78% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  162M   15M  92% /mnt1/scratch

Now let us repeat with 64k sector size
--- With 64k sector size ---

$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0
$ mount /dev/loop0 /mnt1/scratch/
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1
wrote 83886080/83886080 bytes at offset 0
80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec)
$
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M   24M  175M  12% /mnt1/scratch
$ sync
$ df -h /dev/loop0
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      256M  104M   95M  53% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2
pwrite: No space left on device

Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.

Quoted 3 lines

---

Quoted 19 lines
 tests/generic/371 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/371 +1 -1
index b312c450..95af308e 100755
@@ -19,7 +19,7 @@ _require_scratch
 _require_xfs_io_command "falloc"
 test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize"
 
-_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1
+_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1
 _scratch_mount
 
 # Disable speculative post-EOF preallocation on XFS, which can grow fast enough
Signature · 1 line
2.34.1