btrfs: More test fixes for large block/node sizes
cover btrfs: More test fixes for large block/node sizes
More btrfs test fixes. These tests were ran with 64k block size and they failed. Individual patches have the details. A previous patch series with some other test fixes was posted[1].
[1] https://lore.kernel.org/all/[email protected]/
Nirjhar Roy (IBM) (4):
generic/371: Fix the test to be compatible block sizes upto 64k generic/562: Make test compatible with block sizes till 64k btrfs/200: Make the test compatible with all supported block sizes btrfs/290: Make the test compatible with all supported block sizes
tests/btrfs/200 | 8 ++++---- tests/btrfs/200.out | 8 ++++---- tests/btrfs/290 | 16 ++++++++-------- tests/generic/371 | 2 +- tests/generic/562 | 2 +- 5 files changed, 18 insertions(+), 18 deletions(-)
--
2.34.1
PATCH 1/4 generic/371: Fix the test to be compatible block sizes upto 64k
When this test was ran with btrfs with 64k sector/block size, it failed with
QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...
This is what is going on:
Let us see the following set of operations:
--- With 4k sector size ---
$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)
$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch
$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 82M 95M 47% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)
$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 137M 40M 78% /mnt1/scratch
$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 162M 15M 92% /mnt1/scratch
Now let us repeat with 64k sector size
--- With 64k sector size ---
$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec) $ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch
$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 104M 95M 53% /mnt1/scratch
$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 pwrite: No space left on device
Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.
--- tests/generic/371 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/371 +1 -1
@@ -19,7 +19,7 @@ _require_scratch _require_xfs_io_command "falloc" test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize" -_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1 _scratch_mount # Disable speculative post-EOF preallocation on XFS, which can grow fast enough
Signature · 1 line
2.34.1
PATCH 2/4 generic/562: Make test compatible with block sizes till 64k
In reply to Nirjhar Roy (IBM)This test fails with 64k sector size in btrfs. The reason for this is the need for additional space because of COW. When the reflink/clone of file bar into file foo is done, there is no additional space left for COW - the reason is that the metadata space usage is much higher with 64k node size. In order to verify this, I instrumented the test script and disabled COW for file foo and bar and the test passes in 64k (and runs faster too).
With 64k sector and node size (COW enabled) After pwriting foo and bar and before filling up the fs
Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 324M 3.0M 100% /mnt1/scratch After filling up the fs Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 441M 3.0M 100% /mnt1/scratch
With 64k sector and node size (COW disabled) After pwriting foo and bar and before filling up the fs
Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 224M 231M 50% /mnt1/scratch After filling up the fs Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 424M 31M 94% /mnt1/scratch
As we can see, with COW, the fs is completely full after filling up the fs but with COW disabled, we have some space left.
Fix this by increasing the fs size to 590M so that even with 64k node size and COW enabled, reflink has enough space to continue.
--- tests/generic/562 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/562 +1 -1
@@ -22,7 +22,7 @@ _require_scratch_reflink _require_test_program "punch-alternating" _require_xfs_io_command "fpunch" -_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1 +_scratch_mkfs_sized $((590 * 1024 * 1024)) >>$seqres.full 2>&1 _scratch_mount file_size=$(( 200 * 1024 * 1024 )) # 200Mb
Signature · 1 line
2.34.1
PATCH 3/4 btrfs/200: Make the test compatible with all supported block sizes
In reply to Nirjhar Roy (IBM)This test fails on 64k block size with the following error:
At snapshot incr
OK
OK
+File foo does not have 2 shared extents in the base snapshot +/mnt/scratch/base/foo:
+ 0: [0..255]: 26624..26879
+File foo does not have 2 shared extents in the incr snapshot
So, basically after btrfs receive, the file /mnt/scratch/base/foo doesn't have any shared extents in the base snapshot. The reason is that during btrfs send, the extents are not cloned and instead they are written.
The following condition is responsible for the above behavior
in function clone_range():
/*
* Prevent cloning from a zero offset with a length matching the sector
* size because in some scenarios this will make the receiver fail. *
* For example, if in the source filesystem the extent at offset 0
* has a length of sectorsize and it was written using direct IO, then
* it can never be an inline extent (even if compression is enabled).
* Then this extent can be cloned in the original filesystem to a non
* zero file offset, but it may not be possible to clone in the
* destination filesystem because it can be inlined due to compression
* on the destination filesystem (as the receiver's write operations are
* always done using buffered IO). The same happens when the original
* filesystem does not have compression enabled but the destination
* filesystem has.
*/
if (clone_root->offset == 0 &&
len == sctx->send_root->fs_info->sectorsize)
return send_extent_data(sctx, dst_path, offset, len);
Since we are cloning from the first half [0 to 64k), clone_root->offset = 0 and len = 64k which is the sectorsize (sctx->send_root->fs_info->sectorsize). Fix this by increasing the file size and offsets to 128k so that len == sctx->send_root->fs_info->sectorsize is not true.
--- tests/btrfs/200 | 8 ++++---- tests/btrfs/200.out | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-)
tests/btrfs/200 +4 -4
@@ -38,14 +38,14 @@ _scratch_mount # Create our first test file, which has an extent that is shared only with # itself and no other files. We want to verify a full send operation will # clone the extent. -$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \ +$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 128K 0 128K" $SCRATCH_MNT/foo \ | _filter_xfs_io -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \ +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 128K 128K" $SCRATCH_MNT/foo \ | _filter_xfs_io # Create out second test file which initially, for the first send operation, # only has a single extent that is not shared. -$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \ +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 128K 0 128K" $SCRATCH_MNT/bar \ | _filter_xfs_io _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base
@@ -56,7 +56,7 @@ $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \ # Now clone the existing extent in file bar to itself at a different offset. # We want to verify the incremental send operation below will issue a clone # operation instead of a write operation. -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \ +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 128K 128K" $SCRATCH_MNT/bar \ | _filter_xfs_io _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incr
tests/btrfs/200.out +4 -4
@@ -1,12 +1,12 @@ QA output created by 200 -wrote 65536/65536 bytes at offset 0 +wrote 131072/131072 bytes at offset 0 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -linked 65536/65536 bytes at offset 65536 +linked 131072/131072 bytes at offset 131072 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 65536/65536 bytes at offset 0 +wrote 131072/131072 bytes at offset 0 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/base -linked 65536/65536 bytes at offset 65536 +linked 131072/131072 bytes at offset 131072 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/incr At subvol base
Signature · 1 line
2.34.1
PATCH 4/4 btrfs/290: Make the test compatible with all supported block sizes
In reply to Nirjhar Roy (IBM)This test fails with 64k block size with the following error:
punch
pread: Input/output error
pread: Input/output error
+ERROR: couldn't find extent 4096 for inode 261
plug
-pread: Input/output error
-pread: Input/output error
...
The reason is that, some of the subtests are written with 4k blocksize in mind. Fix the test by making the offsets and sizes to 64k so that it becomes compatible/aligned with all supported block sizes.
--- tests/btrfs/290 | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
tests/btrfs/290 +8 -8
@@ -106,15 +106,15 @@ corrupt_reg_to_prealloc() { # corrupt a file by punching a hole corrupt_punch_hole() { local f=$SCRATCH_MNT/punch - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $f local ino=$(get_ino $f) # make a new extent in the middle, sync so the writes don't coalesce $XFS_IO_PROG -c sync $SCRATCH_MNT - $XFS_IO_PROG -fc "pwrite -q -S 0x59 4096 4096" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x59 64k 64k" $f _fsv_enable $f _scratch_unmount # change disk_bytenr to 0, representing a hole - $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr --value 0 \ + $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr --value 0 \ $SCRATCH_DEV _scratch_mount validate $f
@@ -123,14 +123,14 @@ corrupt_punch_hole() { # plug hole corrupt_plug_hole() { local f=$SCRATCH_MNT/plug - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $f local ino=$(get_ino $f) - $XFS_IO_PROG -fc "falloc 4k 4k" $f + $XFS_IO_PROG -fc "falloc 64k 64k" $f _fsv_enable $f _scratch_unmount # change disk_bytenr to some value, plugging the hole - $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr \ - --value 13639680 $SCRATCH_DEV + $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr \ + --value 218234880 $SCRATCH_DEV _scratch_mount validate $f }
@@ -166,7 +166,7 @@ corrupt_root_hash() { # corrupt the Merkle tree data itself corrupt_merkle_tree() { local f=$SCRATCH_MNT/merkle - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $f local ino=$(get_ino $f) _fsv_enable $f _scratch_unmount
Signature · 1 line
2.34.1
Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k
In reply to Nirjhar Roy (IBM)Quoted 73 lines · Nirjhar Roy (IBM)
When this test was ran with btrfs with 64k sector/block size, it failed with
QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...This is what is going on:
Let us see the following set of operations:
--- With 4k sector size ---
$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 82M 95M 47% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 137M 40M 78% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 162M 15M 92% /mnt1/scratchNow let us repeat with 64k sector size
--- With 64k sector size ---$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec) $ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 104M 95M 53% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 pwrite: No space left on device
Quoted 7 lines
Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.
Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).
Thanks,
Zorro
Quoted 23 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/generic/371 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/371 +1
-1@@ -19,7 +19,7 @@ _require_scratch _require_xfs_io_command "falloc" test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize" -_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1 _scratch_mount # Disable speculative post-EOF preallocation on XFS, which can grow fast enoughSignature · 1 line
2.34.1
Re: [PATCH 2/4] generic/562: Make test compatible with block sizes till 64k
In reply to Nirjhar Roy (IBM)Quoted 44 lines · Nirjhar Roy (IBM)
This test fails with 64k sector size in btrfs. The reason for this is the need for additional space because of COW. When the reflink/clone of file bar into file foo is done, there is no additional space left for COW - the reason is that the metadata space usage is much higher with 64k node size. In order to verify this, I instrumented the test script and disabled COW for file foo and bar and the test passes in 64k (and runs faster too).
With 64k sector and node size (COW enabled) After pwriting foo and bar and before filling up the fs
Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 324M 3.0M 100% /mnt1/scratch After filling up the fs Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 441M 3.0M 100% /mnt1/scratchWith 64k sector and node size (COW disabled) After pwriting foo and bar and before filling up the fs
Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 224M 231M 50% /mnt1/scratch After filling up the fs Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 424M 31M 94% /mnt1/scratchAs we can see, with COW, the fs is completely full after filling up the fs but with COW disabled, we have some space left.
Fix this by increasing the fs size to 590M so that even with 64k node size and COW enabled, reflink has enough space to continue.
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/generic/562 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/562 +0
-0
tests/generic/562 _require_scratch_reflink · line 25
Quoted 5 lines
tests/generic/562 +1
-1@@ -22,7 +22,7 @@ _require_scratch_reflink _require_test_program "punch-alternating" _require_xfs_io_command "fpunch" -_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1 +_scratch_mkfs_sized $((590 * 1024 * 1024)) >>$seqres.full 2>&1
Filipe is the author of this test case, if he can help to make sure the 512MiB fs size isn't the necessary condition to reproduce the original bug he tried to uncover, I'm good to have this change, even change to bigger size :)
Thanks,
Zorro
Quoted 7 lines
tests/generic/562 +0
-0_scratch_mount file_size=$(( 200 * 1024 * 1024 )) # 200MbSignature · 1 line
2.34.1
Re: [PATCH 3/4] btrfs/200: Make the test compatible with all supported block sizes
In reply to Nirjhar Roy (IBM)Quoted 54 lines · Nirjhar Roy (IBM)
This test fails on 64k block size with the following error:
At snapshot incr
OK
OK
+File foo does not have 2 shared extents in the base snapshot +/mnt/scratch/base/foo:
+ 0: [0..255]: 26624..26879
+File foo does not have 2 shared extents in the incr snapshotSo, basically after btrfs receive, the file
/mnt/scratch/base/foodoesn't have any shared extents in the base snapshot. The reason is that during btrfs send, the extents are not cloned and instead they are written.
The following condition is responsible for the above behaviorin function
clone_range():/*
* Prevent cloning from a zero offset with a length matching the sector
* size because in some scenarios this will make the receiver fail. *
* For example, if in the source filesystem the extent at offset 0
* has a length of sectorsize and it was written using direct IO, then
* it can never be an inline extent (even if compression is enabled).
* Then this extent can be cloned in the original filesystem to a non
* zero file offset, but it may not be possible to clone in the
* destination filesystem because it can be inlined due to compression
* on the destination filesystem (as the receiver's write operations are
* always done using buffered IO). The same happens when the original
* filesystem does not have compression enabled but the destination
* filesystem has.
*/
if (clone_root->offset== 0 &&
len ==sctx->send_root->fs_info->sectorsize)
returnsend_extent_data(sctx, dst_path, offset, len);Since we are cloning from the first half [0 to 64k),
clone_root->offset= 0 and len = 64k which is the sectorsize (sctx->send_root->fs_info->sectorsize). Fix this by increasing the file size and offsets to 128k so that len ==sctx->send_root->fs_info->sectorsizeis not true.Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/btrfs/200 | 8 ++++---- tests/btrfs/200.out | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-)tests/btrfs/200 +0
-0
tests/btrfs/200 _scratch_mount · line 41
Quoted 5 lines
tests/btrfs/200 +1
-1@@ -38,14 +38,14 @@ _scratch_mount # Create our first test file, which has an extent that is shared only with # itself and no other files. We want to verify a full send operation will # clone the extent. -$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \ +$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 128K 0 128K" $SCRATCH_MNT/foo \
This change and explanation makes sense to me, and I think 128k won't break the "an extent" requirement (compare with 64k). So,
Quoted 45 lines
tests/btrfs/200 +3
-3| _filter_xfs_io -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \ +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 128K 128K" $SCRATCH_MNT/foo \ | _filter_xfs_io # Create out second test file which initially, for the first send operation, # only has a single extent that is not shared. -$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \ +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 128K 0 128K" $SCRATCH_MNT/bar \ | _filter_xfs_io _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base@@ -56,7 +56,7 @@ $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \ # Now clone the existing extent in file bar to itself at a different offset. # We want to verify the incremental send operation below will issue a clone # operation instead of a write operation. -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \ +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 128K 128K" $SCRATCH_MNT/bar \ | _filter_xfs_io _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incrtests/btrfs/200.out +4
-4@@ -1,12 +1,12 @@ QA output created by 200 -wrote 65536/65536 bytes at offset 0 +wrote 131072/131072 bytes at offset 0 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -linked 65536/65536 bytes at offset 65536 +linked 131072/131072 bytes at offset 131072 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 65536/65536 bytes at offset 0 +wrote 131072/131072 bytes at offset 0 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/base -linked 65536/65536 bytes at offset 65536 +linked 131072/131072 bytes at offset 131072 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/incr At subvol baseSignature · 1 line
2.34.1
Re: [PATCH 3/4] btrfs/200: Make the test compatible with all supported block sizes
Quoted 63 lines · Zorro Lang
Quoted 54 lines · Nirjhar Roy (IBM)
This test fails on 64k block size with the following error:
At snapshot incr
OK
OK
+File foo does not have 2 shared extents in the base snapshot +/mnt/scratch/base/foo:
+ 0: [0..255]: 26624..26879
+File foo does not have 2 shared extents in the incr snapshotSo, basically after btrfs receive, the file
/mnt/scratch/base/foodoesn't have any shared extents in the base snapshot. The reason is that during btrfs send, the extents are not cloned and instead they are written.
The following condition is responsible for the above behaviorin function
clone_range():/*
* Prevent cloning from a zero offset with a length matching the sector
* size because in some scenarios this will make the receiver fail. *
* For example, if in the source filesystem the extent at offset 0
* has a length of sectorsize and it was written using direct IO, then
* it can never be an inline extent (even if compression is enabled).
* Then this extent can be cloned in the original filesystem to a non
* zero file offset, but it may not be possible to clone in the
* destination filesystem because it can be inlined due to compression
* on the destination filesystem (as the receiver's write operations are
* always done using buffered IO). The same happens when the original
* filesystem does not have compression enabled but the destination
* filesystem has.
*/
if (clone_root->offset== 0 &&
len ==sctx->send_root->fs_info->sectorsize)
returnsend_extent_data(sctx, dst_path, offset, len);Since we are cloning from the first half [0 to 64k),
clone_root->offset= 0 and len = 64k which is the sectorsize (sctx->send_root->fs_info->sectorsize). Fix this by increasing the file size and offsets to 128k so that len ==sctx->send_root->fs_info->sectorsizeis not true.Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/btrfs/200 | 8 ++++---- tests/btrfs/200.out | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-)tests/btrfs/200 +0
-0tests/btrfs/200 _scratch_mount · line 41
Quoted 5 lines
tests/btrfs/200 +1
-1@@ -38,14 +38,14 @@ _scratch_mount # Create our first test file, which has an extent that is shared only with # itself and no other files. We want to verify a full send operation will # clone the extent. -$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \ +$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 128K 0 128K" $SCRATCH_MNT/foo \This change and explanation makes sense to me, and I think 128k won't break the "an extent" requirement (compare with 64k). So,
Quoted 1 line
Reviewed-by Zorro Lang <[email protected]>
Thank you.
--NR
Quoted 45 lines
Quoted 45 lines
tests/btrfs/200 +3
-3| _filter_xfs_io -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \ +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 128K 128K" $SCRATCH_MNT/foo \ | _filter_xfs_io # Create out second test file which initially, for the first send operation, # only has a single extent that is not shared. -$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \ +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 128K 0 128K" $SCRATCH_MNT/bar \ | _filter_xfs_io _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base@@ -56,7 +56,7 @@ $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \ # Now clone the existing extent in file bar to itself at a different offset. # We want to verify the incremental send operation below will issue a clone # operation instead of a write operation. -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \ +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 128K 128K" $SCRATCH_MNT/bar \ | _filter_xfs_io _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incrtests/btrfs/200.out +4
-4@@ -1,12 +1,12 @@ QA output created by 200 -wrote 65536/65536 bytes at offset 0 +wrote 131072/131072 bytes at offset 0 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -linked 65536/65536 bytes at offset 65536 +linked 131072/131072 bytes at offset 131072 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 65536/65536 bytes at offset 0 +wrote 131072/131072 bytes at offset 0 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/base -linked 65536/65536 bytes at offset 65536 +linked 131072/131072 bytes at offset 131072 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/incr At subvol baseSignature · 1 line
2.34.1
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Re: [PATCH 2/4] generic/562: Make test compatible with block sizes till 64k
In reply to Zorro LangQuoted 48 lines · Zorro Lang
Quoted 47 lines · Nirjhar Roy (IBM)
This test fails with 64k sector size in btrfs. The reason for this is the need for additional space because of COW. When the reflink/clone of file bar into file foo is done, there is no additional space left for COW - the reason is that the metadata space usage is much higher with 64k node size. In order to verify this, I instrumented the test script and disabled COW for file foo and bar and the test passes in 64k (and runs faster too).
With 64k sector and node size (COW enabled) After pwriting foo and bar and before filling up the fs
Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 324M 3.0M 100% /mnt1/scratch After filling up the fs Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 441M 3.0M 100% /mnt1/scratchWith 64k sector and node size (COW disabled) After pwriting foo and bar and before filling up the fs
Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 224M 231M 50% /mnt1/scratch After filling up the fs Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 424M 31M 94% /mnt1/scratchAs we can see, with COW, the fs is completely full after filling up the fs but with COW disabled, we have some space left.
Fix this by increasing the fs size to 590M so that even with 64k node size and COW enabled, reflink has enough space to continue.
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/generic/562 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/562 +0
-0@@ -22,7 +22,7 @@ _require_scratch_reflink _require_test_program "punch-alternating" _require_xfs_io_command "fpunch"
Quoted 6 lines
tests/generic/562 _require_scratch_reflink · line 25
Quoted 2 lines
tests/generic/562 +1
-1-_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1 +_scratch_mkfs_sized $((590 * 1024 * 1024)) >>$seqres.full 2>&1Filipe is the author of this test case, if he can help to make sure the 512MiB fs size isn't the necessary condition to reproduce the original bug he tried to uncover, I'm good to have this change, even change to bigger size :)
Okay, I will wait for Filipe's comment on this change. Thank you.
--NR
Quoted 11 lines
Thanks,
ZorroQuoted 7 lines
tests/generic/562 +0
-0_scratch_mount file_size=$(( 200 * 1024 * 1024 )) # 200MbSignature · 1 line
2.34.1
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Filipe Manana
Reviewed-byRe: [PATCH 2/4] generic/562: Make test compatible with block sizes till 64k
Quoted 51 lines · Nirjhar Roy (IBM)
Quoted 48 lines · Zorro Lang
Quoted 47 lines · Nirjhar Roy (IBM)
This test fails with 64k sector size in btrfs. The reason for this is the need for additional space because of COW. When the reflink/clone of file bar into file foo is done, there is no additional space left for COW - the reason is that the metadata space usage is much higher with 64k node size. In order to verify this, I instrumented the test script and disabled COW for file foo and bar and the test passes in 64k (and runs faster too).
With 64k sector and node size (COW enabled) After pwriting foo and bar and before filling up the fs
Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 324M 3.0M 100% /mnt1/scratch After filling up the fs Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 441M 3.0M 100% /mnt1/scratchWith 64k sector and node size (COW disabled) After pwriting foo and bar and before filling up the fs
Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 224M 231M 50% /mnt1/scratch After filling up the fs Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 424M 31M 94% /mnt1/scratchAs we can see, with COW, the fs is completely full after filling up the fs but with COW disabled, we have some space left.
Fix this by increasing the fs size to 590M so that even with 64k node size and COW enabled, reflink has enough space to continue.
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/generic/562 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/562 +0
-0@@ -22,7 +22,7 @@ _require_scratch_reflink _require_test_program "punch-alternating" _require_xfs_io_command "fpunch"
Quoted 8 lines
Quoted 6 lines
tests/generic/562 _require_scratch_reflink · line 25
Quoted 2 lines
tests/generic/562 +1
-1-_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1 +_scratch_mkfs_sized $((590 * 1024 * 1024)) >>$seqres.full 2>&1Filipe is the author of this test case, if he can help to make sure the 512MiB fs size isn't the necessary condition to reproduce the original bug he tried to uncover, I'm good to have this change, even change to bigger size :)
Okay, I will wait for Filipe's comment on this change. Thank you.
This is probably ok.
This test was to exercise a bug fixed in 2019, commit 690a5dbfc513 ("Btrfs: fix ENOSPC errors, leading to transaction aborts, when cloning extents").
It's hard to confirm nowadays since the fix landed in kernel 5.4, trying a revert of that commit is not practical since there are tons of changes done on top of it, the alternative would be to test a 5.3 kernel or older.
The while loop below should ensure we try to fill all metadata space with files that have only inline extents, so in theory it should be ok for any fs size.
Btw, when you do a patchset that modifies btrfs tests, or even generic tests that were motivated by btrfs bugs, please cc the btrfs mailing list... I haven't noticed this before because the list was not cc'ed.
Quoted 19 lines
--NR
Quoted 11 lines
Thanks,
ZorroQuoted 7 lines
tests/generic/562 +0
-1_scratch_mount file_size=$(( 200 * 1024 * 1024 )) # 200Mb --2.34.1
--
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
Re: [PATCH 3/4] btrfs/200: Make the test compatible with all supported block sizes
In reply to Nirjhar Roy (IBM)Quoted 38 lines · Nirjhar Roy (IBM)
This test fails on 64k block size with the following error:
At snapshot incr
OK
OK
+File foo does not have 2 shared extents in the base snapshot +/mnt/scratch/base/foo:
+ 0: [0..255]: 26624..26879
+File foo does not have 2 shared extents in the incr snapshotSo, basically after btrfs receive, the file
/mnt/scratch/base/foodoesn't have any shared extents in the base snapshot. The reason is that during btrfs send, the extents are not cloned and instead they are written.
The following condition is responsible for the above behaviorin function
clone_range():/*
* Prevent cloning from a zero offset with a length matching the sector
* size because in some scenarios this will make the receiver fail. *
* For example, if in the source filesystem the extent at offset 0
* has a length of sectorsize and it was written using direct IO, then
* it can never be an inline extent (even if compression is enabled).
* Then this extent can be cloned in the original filesystem to a non
* zero file offset, but it may not be possible to clone in the
* destination filesystem because it can be inlined due to compression
* on the destination filesystem (as the receiver's write operations are
* always done using buffered IO). The same happens when the original
* filesystem does not have compression enabled but the destination
* filesystem has.
*/
if (clone_root->offset== 0 &&
len ==sctx->send_root->fs_info->sectorsize)
returnsend_extent_data(sctx, dst_path, offset, len);
Quoted 7 lines
Since we are cloning from the first half [0 to 64k),
clone_root->offset= 0 and len = 64k which is the sectorsize (sctx->send_root->fs_info->sectorsize). Fix this by increasing the file size and offsets to 128k so that len ==sctx->send_root->fs_info->sectorsizeis not true.Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>
Please always cc the btrfs mailing list when doing changes to btrfs test cases.
Quoted 60 lines
--- tests/btrfs/200 | 8 ++++---- tests/btrfs/200.out | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-)tests/btrfs/200 +4
-4@@ -38,14 +38,14 @@ _scratch_mount # Create our first test file, which has an extent that is shared only with # itself and no other files. We want to verify a full send operation will # clone the extent. -$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 64K 0 64K" $SCRATCH_MNT/foo \ +$XFS_IO_PROG -f -c "pwrite -S 0xb1 -b 128K 0 128K" $SCRATCH_MNT/foo \ | _filter_xfs_io -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 64K 64K" $SCRATCH_MNT/foo \ +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foo 0 128K 128K" $SCRATCH_MNT/foo \ | _filter_xfs_io # Create out second test file which initially, for the first send operation, # only has a single extent that is not shared. -$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 64K 0 64K" $SCRATCH_MNT/bar \ +$XFS_IO_PROG -f -c "pwrite -S 0xc7 -b 128K 0 128K" $SCRATCH_MNT/bar \ | _filter_xfs_io _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base@@ -56,7 +56,7 @@ $BTRFS_UTIL_PROG send -f $send_files_dir/1.snap $SCRATCH_MNT/base 2>&1 \ # Now clone the existing extent in file bar to itself at a different offset. # We want to verify the incremental send operation below will issue a clone # operation instead of a write operation. -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 64K 64K" $SCRATCH_MNT/bar \ +$XFS_IO_PROG -c "reflink $SCRATCH_MNT/bar 0 128K 128K" $SCRATCH_MNT/bar \ | _filter_xfs_io _btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incrtests/btrfs/200.out +4
-5@@ -1,12 +1,12 @@ QA output created by 200 -wrote 65536/65536 bytes at offset 0 +wrote 131072/131072 bytes at offset 0 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -linked 65536/65536 bytes at offset 65536 +linked 131072/131072 bytes at offset 131072 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 65536/65536 bytes at offset 0 +wrote 131072/131072 bytes at offset 0 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/base -linked 65536/65536 bytes at offset 65536 +linked 131072/131072 bytes at offset 131072 XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) At subvol SCRATCH_MNT/incr At subvol base --2.34.1
Re: [PATCH 2/4] generic/562: Make test compatible with block sizes till 64k
In reply to Filipe MananaQuoted 72 lines · Filipe Manana
Quoted 50 lines · Nirjhar Roy (IBM)
Quoted 48 lines · Zorro Lang
Quoted 47 lines · Nirjhar Roy (IBM)
This test fails with 64k sector size in btrfs. The reason for this is the need for additional space because of COW. When the reflink/clone of file bar into file foo is done, there is no additional space left for COW - the reason is that the metadata space usage is much higher with 64k node size. In order to verify this, I instrumented the test script and disabled COW for file foo and bar and the test passes in 64k (and runs faster too).
With 64k sector and node size (COW enabled) After pwriting foo and bar and before filling up the fs
Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 324M 3.0M 100% /mnt1/scratch After filling up the fs Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 441M 3.0M 100% /mnt1/scratchWith 64k sector and node size (COW disabled) After pwriting foo and bar and before filling up the fs
Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 224M 231M 50% /mnt1/scratch After filling up the fs Filesystem Size Used Avail Use% Mounted on /dev/loop1 512M 424M 31M 94% /mnt1/scratchAs we can see, with COW, the fs is completely full after filling up the fs but with COW disabled, we have some space left.
Fix this by increasing the fs size to 590M so that even with 64k node size and COW enabled, reflink has enough space to continue.
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/generic/562 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/562 +0
-0@@ -22,7 +22,7 @@ _require_scratch_reflink _require_test_program "punch-alternating" _require_xfs_io_command "fpunch"Quoted 7 lines
Quoted 6 lines
tests/generic/562 _require_scratch_reflink · line 25
Quoted 2 lines
tests/generic/562 +1
-1-_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1 +_scratch_mkfs_sized $((590 * 1024 * 1024)) >>$seqres.full 2>&1Filipe is the author of this test case, if he can help to make sure the 512MiB fs size isn't the necessary condition to reproduce the original bug he tried to uncover, I'm good to have this change, even change to bigger size :)
Okay, I will wait for Filipe's comment on this change. Thank you.
This is probably ok.
This test was to exercise a bug fixed in 2019, commit690a5dbfc513("Btrfs: fix ENOSPC errors, leading to transaction aborts, when cloning extents").
It's hard to confirm nowadays since the fix landed in kernel 5.4, trying a revert of that commit is not practical since there are tons of changes done on top of it, the alternative would be to test a 5.3 kernel or older.The while loop below should ensure we try to fill all metadata space with files that have only inline extents, so in theory it should be ok for any fs size.
Quoted 5 lines
Reviewed-by Filipe Manana <[email protected]>Btw, when you do a patchset that modifies btrfs tests, or even generic tests that were motivated by btrfs bugs, please cc the btrfs mailing list... I haven't noticed this before because the list was not cc'ed.
Yes, I usually cc the btrfs mailing list but I missed it this time, sorry about that.
--NR
Quoted 19 lines
Quoted 19 lines
--NR
Quoted 10 lines
Thanks,
ZorroQuoted 7 lines
tests/generic/562 +0
-1_scratch_mount file_size=$(( 200 * 1024 * 1024 )) # 200Mb --2.34.1
--
Nirjhar Roy
Linux Kernel Developer
IBM, Bangalore
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Re: [PATCH 4/4] btrfs/290: Make the test compatible with all supported block sizes
In reply to Nirjhar Roy (IBM)Quoted 12 lines · Nirjhar Roy (IBM)
This test fails with 64k block size with the following error:
punch
pread: Input/output error
pread: Input/output error
+ERROR: couldn't find extent 4096 for inode 261
plug
-pread: Input/output error
-pread: Input/output error
...
Quoted 2 lines
The reason is that, some of the subtests are written with 4k blocksize in mind. Fix the test by making the offsets and sizes to 64k so
"... offsets and sizes to 64k..." -> "... offsets and sizes multiples of 64K..."
Quoted 13 lines
that it becomes compatible/aligned with all supported block sizes.
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/btrfs/290 | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)tests/btrfs/290 +0
-0
tests/btrfs/290 corrupt_reg_to_prealloc() { · line 109
Quoted 5 lines
tests/btrfs/290 +1
-1@@ -106,15 +106,15 @@ corrupt_reg_to_prealloc() { # corrupt a file by punching a hole corrupt_punch_hole() { local f=$SCRATCH_MNT/punch - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $f
Can you please make this more readable and type 192K instead of 196608?
tests/btrfs/290 corrupt_reg_to_prealloc() { · line 113
Quoted 5 lines
tests/btrfs/290 +1
-1local ino=$(get_ino $f) # make a new extent in the middle, sync so the writes don't coalesce $XFS_IO_PROG -c sync $SCRATCH_MNT - $XFS_IO_PROG -fc "pwrite -q -S 0x59 4096 4096" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x59 64k 64k" $f
Here you use 64k instead of 65536, which is more readable.
Quoted 9 lines
tests/btrfs/290 +1
-1_fsv_enable $f _scratch_unmount # change disk_bytenr to 0, representing a hole - $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr --value 0 \ + $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr --value 0 \ $SCRATCH_DEV _scratch_mount validate $f
tests/btrfs/290 corrupt_punch_hole() { · line 126
Quoted 5 lines
tests/btrfs/290 +1
-1@@ -123,14 +123,14 @@ corrupt_punch_hole() { # plug hole corrupt_plug_hole() { local f=$SCRATCH_MNT/plug - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $f
Same here, 192K.
Quoted 14 lines
tests/btrfs/290 +3
-3local ino=$(get_ino $f) - $XFS_IO_PROG -fc "falloc 4k 4k" $f + $XFS_IO_PROG -fc "falloc 64k 64k" $f _fsv_enable $f _scratch_unmount # change disk_bytenr to some value, plugging the hole - $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr \ - --value 13639680 $SCRATCH_DEV + $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr \ + --value 218234880 $SCRATCH_DEV _scratch_mount validate $f }
tests/btrfs/290 corrupt_root_hash() { · line 169
Quoted 5 lines
tests/btrfs/290 +1
-1@@ -166,7 +166,7 @@ corrupt_root_hash() { # corrupt the Merkle tree data itself corrupt_merkle_tree() { local f=$SCRATCH_MNT/merkle - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $f
Same here, 192K.
Also please always cc the btrfs mailing list for changes to btrfs tests.
Quoted 6 lines
tests/btrfs/290 +0
-1local ino=$(get_ino $f) _fsv_enable $f _scratch_unmount --2.34.1
Re: [PATCH 4/4] btrfs/290: Make the test compatible with all supported block sizes
Quoted 13 lines · Filipe Manana
Quoted 11 lines · Nirjhar Roy (IBM)
This test fails with 64k block size with the following error:
punch
pread: Input/output error
pread: Input/output error
+ERROR: couldn't find extent 4096 for inode 261
plug
-pread: Input/output error
-pread: Input/output error
...
Quoted 3 lines
Quoted 2 lines
The reason is that, some of the subtests are written with 4k blocksize in mind. Fix the test by making the offsets and sizes to 64k so
"... offsets and sizes to 64k..." -> "... offsets and sizes multiples of 64K..."
Noted.
Quoted 65 lines
Quoted 14 lines
that it becomes compatible/aligned with all supported block sizes.
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/btrfs/290 | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)tests/btrfs/290 +0
-0tests/btrfs/290 corrupt_reg_to_prealloc() { · line 109
Quoted 5 lines
tests/btrfs/290 +1
-1@@ -106,15 +106,15 @@ corrupt_reg_to_prealloc() { # corrupt a file by punching a hole corrupt_punch_hole() { local f=$SCRATCH_MNT/punch - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $fCan you please make this more readable and type 192K instead of 196608?
tests/btrfs/290 corrupt_reg_to_prealloc() { · line 113
Quoted 5 lines
tests/btrfs/290 +1
-1local ino=$(get_ino $f) # make a new extent in the middle, sync so the writes don't coalesce $XFS_IO_PROG -c sync $SCRATCH_MNT - $XFS_IO_PROG -fc "pwrite -q -S 0x59 4096 4096" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x59 64k 64k" $fHere you use 64k instead of 65536, which is more readable.
Quoted 9 lines
tests/btrfs/290 +1
-1_fsv_enable $f _scratch_unmount # change disk_bytenr to 0, representing a hole - $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr --value 0 \ + $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr --value 0 \ $SCRATCH_DEV _scratch_mount validate $ftests/btrfs/290 corrupt_punch_hole() { · line 126
Quoted 5 lines
tests/btrfs/290 +1
-1@@ -123,14 +123,14 @@ corrupt_punch_hole() { # plug hole corrupt_plug_hole() { local f=$SCRATCH_MNT/plug - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $fSame here, 192K.
Quoted 14 lines
tests/btrfs/290 +3
-3local ino=$(get_ino $f) - $XFS_IO_PROG -fc "falloc 4k 4k" $f + $XFS_IO_PROG -fc "falloc 64k 64k" $f _fsv_enable $f _scratch_unmount # change disk_bytenr to some value, plugging the hole - $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr \ - --value 13639680 $SCRATCH_DEV + $BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 65536 -f disk_bytenr \ + --value 218234880 $SCRATCH_DEV _scratch_mount validate $f }tests/btrfs/290 corrupt_root_hash() { · line 169
Quoted 5 lines
tests/btrfs/290 +1
-1@@ -166,7 +166,7 @@ corrupt_root_hash() { # corrupt the Merkle tree data itself corrupt_merkle_tree() { local f=$SCRATCH_MNT/merkle - $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f + $XFS_IO_PROG -fc "pwrite -q -S 0x58 0 196608" $fSame here, 192K.
Quoted 1 line
Also please always cc the btrfs mailing list for changes to btrfs tests.
Sure. I will change all the 196608 to 192k and send an updated version.
--NR
Quoted 8 lines
Quoted 8 lines
tests/btrfs/290 +0
-1local ino=$(get_ino $f) _fsv_enable $f _scratch_unmount --2.34.1
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k
In reply to Zorro LangQuoted 81 lines · Zorro Lang
Quoted 73 lines · Nirjhar Roy (IBM)
When this test was ran with btrfs with 64k sector/block size, it failed with
QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...This is what is going on:
Let us see the following set of operations:
--- With 4k sector size ---
$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 82M 95M 47% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 137M 40M 78% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 162M 15M 92% /mnt1/scratchNow let us repeat with 64k sector size
--- With 64k sector size ---$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec) $ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 104M 95M 53% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 pwrite: No space left on deviceQuoted 7 lines
Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.
Quoted 4 lines
Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).
Hi Filipe, Zorro, Wang,
The original commit message of generic/371 says that this test catches some excess space usage issues. Is(Are) there any patch(es) that fix this issue - so that I can remove the commits and check if the test expectedly fails with slightly large fssize i.e, 330M?
I did find some related commits [1] and I ran the test with 330M and 256M(the default size) after removing the commits[1] but the test passes with both the filesystem sizes. So I am guessing, this is not the patch that can test generic/371.
[1] https://lore.kernel.org/linux-btrfs/[email protected]/
--NR
Quoted 26 lines
Thanks,
ZorroQuoted 22 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/generic/371 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/371 +1
-1@@ -19,7 +19,7 @@ _require_scratch _require_xfs_io_command "falloc" test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize" -_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1 _scratch_mount # Disable speculative post-EOF preallocation on XFS, which can grow fast enoughSignature · 1 line
2.34.1
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k
In reply to Zorro LangQuoted 81 lines · Zorro Lang
Quoted 73 lines · Nirjhar Roy (IBM)
When this test was ran with btrfs with 64k sector/block size, it failed with
QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...This is what is going on:
Let us see the following set of operations:
--- With 4k sector size ---
$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 82M 95M 47% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 137M 40M 78% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 162M 15M 92% /mnt1/scratchNow let us repeat with 64k sector size
--- With 64k sector size ---$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec) $ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 104M 95M 53% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 pwrite: No space left on deviceQuoted 7 lines
Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.
Quoted 4 lines
Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).
Hi Filipe, Zorro
The original commit message of generic/371 says that this test catches some excess space usage issues. Is(Are) there any patch(es) that fix this issue - so that I can remove the commits and check if the test expectedly fails with slightly large fssize i.e, 330M?
I did find some related commits [1] and I ran the test with 330M and 256M(the default size) after removing the commits[1] but the test passes with both the filesystem sizes. So I am guessing, this is not the patch that can test generic/371.
[1] https://lore.kernel.org/linux-btrfs/[email protected]/
--NR
Quoted 26 lines
Thanks,
ZorroQuoted 22 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/generic/371 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/371 +1
-1@@ -19,7 +19,7 @@ _require_scratch _require_xfs_io_command "falloc" test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize" -_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1 _scratch_mount # Disable speculative post-EOF preallocation on XFS, which can grow fast enoughSignature · 1 line
2.34.1
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k
In reply to Nirjhar Roy (IBM)Quoted 95 lines · Nirjhar Roy (IBM)
Quoted 81 lines · Zorro Lang
Quoted 73 lines · Nirjhar Roy (IBM)
When this test was ran with btrfs with 64k sector/block size, it failed with
QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...This is what is going on:
Let us see the following set of operations:
--- With 4k sector size ---
$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 82M 95M 47% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 137M 40M 78% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 162M 15M 92% /mnt1/scratchNow let us repeat with 64k sector size
--- With 64k sector size ---$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec) $ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 104M 95M 53% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 pwrite: No space left on deviceQuoted 7 lines
Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.Quoted 4 lines
Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).
Hi Filipe, Zorro, Wang,
The original commit message of generic/371 says that this test catches some excess space usage issues. Is(Are) there any patch(es) that fix this issue - so that I can remove the commits and check if the test expectedly fails with slightly large fssize i.e, 330M?
Quoted 6 lines
I did find some related commits [1] and I ran the test with 330M and 256M(the default size) after removing the commits[1] but the test passes with both the filesystem sizes. So I am guessing, this is not the patch that can test generic/371.
Hi,
I'm not familiar with btrfs patches, but from the commit history, I suspect it *might* be related with below commit:
commit 18513091af9483ba84328d42092bd4d42a3c958f
Author: Wang Xiaoguang <[email protected]> Date: Mon Jul 25 15:51:40 2016 +0800
btrfs: update btrfs_space_info's bytes_may_use timely
Or "c0d2f6104e8ab2eb75e58e72494ad4b69c5227f8" :)
Thanks,
Zorro
Quoted 32 lines
--NR
Quoted 24 lines
Thanks,
ZorroQuoted 20 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/generic/371 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/371 +1
-1@@ -19,7 +19,7 @@ _require_scratch _require_xfs_io_command "falloc" test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize" -_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1 _scratch_mount # Disable speculative post-EOF preallocation on XFS, which can grow fast enoughSignature · 1 line
2.34.1Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k
Quoted 105 lines · Zorro Lang
Quoted 93 lines · Nirjhar Roy (IBM)
Quoted 81 lines · Zorro Lang
Quoted 73 lines · Nirjhar Roy (IBM)
When this test was ran with btrfs with 64k sector/block size, it failed with
QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...This is what is going on:
Let us see the following set of operations:
--- With 4k sector size ---
$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 82M 95M 47% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 137M 40M 78% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 162M 15M 92% /mnt1/scratchNow let us repeat with 64k sector size
--- With 64k sector size ---$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec) $ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 104M 95M 53% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 pwrite: No space left on deviceQuoted 7 lines
Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.Quoted 4 lines
Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).
Hi Filipe, Zorro, Wang,
The original commit message of generic/371 says that this test catches some excess space usage issues. Is(Are) there any patch(es) that fix this issue - so that I can remove the commits and check if the test expectedly fails with slightly large fssize i.e, 330M?
Quoted 6 lines
I did find some related commits [1] and I ran the test with 330M and 256M(the default size) after removing the commits[1] but the test passes with both the filesystem sizes. So I am guessing, this is not the patch that can test generic/371.
Hi,
I'm not familiar with btrfs patches, but from the commit history, I suspect it *might* be related with below commit:
Quoted 7 lines
commit
18513091af9483ba84328d42092bd4d42a3c958fAuthor: Wang Xiaoguang <[email protected]> Date: Mon Jul 25 15:51:40 2016 +0800btrfs: update btrfs_space_info's
bytes_may_usetimelyOr "
c0d2f6104e8ab2eb75e58e72494ad4b69c5227f8" :)
Thank you for the pointers. I tried running the test by checking out the commits before the above 2 mentioned commits, but the test seems to pass with the existing fs size.
--NR
Quoted 34 lines
Thanks,
ZorroQuoted 30 lines
--NR
Quoted 23 lines
Thanks,
ZorroQuoted 20 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/generic/371 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/371 +1
-1@@ -19,7 +19,7 @@ _require_scratch _require_xfs_io_command "falloc" test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize" -_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1 _scratch_mount # Disable speculative post-EOF preallocation on XFS, which can grow fast enoughSignature · 1 line
2.34.1Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k
In reply to Zorro LangQuoted 105 lines · Zorro Lang
Quoted 93 lines · Nirjhar Roy (IBM)
Quoted 81 lines · Zorro Lang
Quoted 73 lines · Nirjhar Roy (IBM)
When this test was ran with btrfs with 64k sector/block size, it failed with
QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...This is what is going on:
Let us see the following set of operations:
--- With 4k sector size ---
$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 82M 95M 47% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 137M 40M 78% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 162M 15M 92% /mnt1/scratchNow let us repeat with 64k sector size
--- With 64k sector size ---$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec) $ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 104M 95M 53% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 pwrite: No space left on deviceQuoted 7 lines
Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.Quoted 4 lines
Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).
Hi Filipe, Zorro, Wang,
The original commit message of generic/371 says that this test catches some excess space usage issues. Is(Are) there any patch(es) that fix this issue - so that I can remove the commits and check if the test expectedly fails with slightly large fssize i.e, 330M?
Quoted 6 lines
I did find some related commits [1] and I ran the test with 330M and 256M(the default size) after removing the commits[1] but the test passes with both the filesystem sizes. So I am guessing, this is not the patch that can test generic/371.
Hi,
I'm not familiar with btrfs patches, but from the commit history, I suspect it *might* be related with below commit:
Quoted 7 lines
commit
18513091af9483ba84328d42092bd4d42a3c958fAuthor: Wang Xiaoguang <[email protected]> Date: Mon Jul 25 15:51:40 2016 +0800btrfs: update btrfs_space_info's
bytes_may_usetimelyOr "
c0d2f6104e8ab2eb75e58e72494ad4b69c5227f8" :)
Thank you for the pointers. I tried running the test by checking out the commits before the above 2 mentioned commits, but the test seems to pass with the existing fs size.
--NR
Quoted 34 lines
Thanks,
ZorroQuoted 30 lines
--NR
Quoted 23 lines
Thanks,
ZorroQuoted 20 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/generic/371 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/371 +1
-1@@ -19,7 +19,7 @@ _require_scratch _require_xfs_io_command "falloc" test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize" -_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1 _scratch_mount # Disable speculative post-EOF preallocation on XFS, which can grow fast enoughSignature · 1 line
2.34.1Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k
In reply to Zorro LangQuoted 109 lines · Zorro Lang
Quoted 93 lines · Nirjhar Roy (IBM)
Quoted 81 lines · Zorro Lang
Quoted 73 lines · Nirjhar Roy (IBM)
When this test was ran with btrfs with 64k sector/block size, it failed with
QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...This is what is going on:
Let us see the following set of operations:
--- With 4k sector size ---
$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 82M 95M 47% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 137M 40M 78% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 162M 15M 92% /mnt1/scratchNow let us repeat with 64k sector size
--- With 64k sector size ---$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec) $ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 104M 95M 53% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 pwrite: No space left on deviceQuoted 7 lines
Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.Quoted 4 lines
Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).
Hi Filipe, Zorro, Wang,
The original commit message of generic/371 says that this test catches some excess space usage issues. Is(Are) there any patch(es) that fix this issue - so that I can remove the commits and check if the test expectedly fails with slightly large fssize i.e, 330M?
Quoted 6 lines
I did find some related commits [1] and I ran the test with 330M and 256M(the default size) after removing the commits[1] but the test passes with both the filesystem sizes. So I am guessing, this is not the patch that can test generic/371.
Hi,
I'm not familiar with btrfs patches, but from the commit history, I suspect it *might* be related with below commit:
commit
18513091af9483ba84328d42092bd4d42a3c958fAuthor: Wang Xiaoguang <[email protected]> Date: Mon Jul 25 15:51:40 2016 +0800
Quoted 6 lines
btrfs: update btrfs_space_info's
bytes_may_usetimelyOr "
c0d2f6104e8ab2eb75e58e72494ad4b69c5227f8" :)Thanks,
Zorro
Hi Zorro, Filipe and Qu
Thank you for the pointers. I tried running the test reverting the above 2 mentioned commits, but the test seems to pass without these commits, so probably these are not the ones. I am not able to find any other relevant commits and the test case is also very old. Any further ideas?
--NR
Quoted 31 lines
Quoted 31 lines
--NR
Quoted 23 lines
Thanks,
ZorroQuoted 20 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/generic/371 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/371 +1
-1@@ -19,7 +19,7 @@ _require_scratch _require_xfs_io_command "falloc" test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize" -_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1 _scratch_mount # Disable speculative post-EOF preallocation on XFS, which can grow fast enoughSignature · 1 line
2.34.1Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k
Quoted 118 lines · Nirjhar Roy (IBM)
Quoted 109 lines · Zorro Lang
Quoted 93 lines · Nirjhar Roy (IBM)
Quoted 81 lines · Zorro Lang
Quoted 73 lines · Nirjhar Roy (IBM)
When this test was ran with btrfs with 64k sector/block size, it failed with
QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...This is what is going on:
Let us see the following set of operations:
--- With 4k sector size ---
$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 82M 95M 47% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 137M 40M 78% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 162M 15M 92% /mnt1/scratchNow let us repeat with 64k sector size
--- With 64k sector size ---$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec) $ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 104M 95M 53% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 pwrite: No space left on deviceQuoted 7 lines
Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.Quoted 4 lines
Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).
Hi Filipe, Zorro, Wang,
The original commit message of generic/371 says that this test catches some excess space usage issues. Is(Are) there any patch(es) that fix this issue - so that I can remove the commits and check if the test expectedly fails with slightly large fssize i.e, 330M?
Quoted 6 lines
I did find some related commits [1] and I ran the test with 330M and 256M(the default size) after removing the commits[1] but the test passes with both the filesystem sizes. So I am guessing, this is not the patch that can test generic/371.
Hi,
I'm not familiar with btrfs patches, but from the commit history, I suspect it *might* be related with below commit:
commit
18513091af9483ba84328d42092bd4d42a3c958fAuthor: Wang Xiaoguang <[email protected]> Date: Mon Jul 25 15:51:40 2016 +0800Quoted 6 lines
btrfs: update btrfs_space_info's
bytes_may_usetimelyOr "
c0d2f6104e8ab2eb75e58e72494ad4b69c5227f8" :)Thanks,
Zorro
Quoted 6 lines
Hi Zorro, Filipe and Qu
Thank you for the pointers. I tried running the test reverting the above 2 mentioned commits, but the test seems to pass without these commits, so probably these are not the ones. I am not able to find any other relevant commits and the test case is also very old. Any further ideas?
Thanks, I don't have any objection on this patch now :) As this patchset still has unresolved review points from btrfs list, I think you can change and resend this patchset to get further reviewing.
Thanks,
Zorro
Quoted 39 lines
--NR
Quoted 31 lines
Quoted 31 lines
--NR
Quoted 23 lines
Thanks,
ZorroQuoted 20 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/generic/371 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/371 +1
-1@@ -19,7 +19,7 @@ _require_scratch _require_xfs_io_command "falloc" test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize" -_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1 _scratch_mount # Disable speculative post-EOF preallocation on XFS, which can grow fast enoughSignature · 1 line
2.34.1Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, BangaloreSignature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k
Quoted 119 lines · Zorro Lang
Quoted 118 lines · Nirjhar Roy (IBM)
Quoted 109 lines · Zorro Lang
Quoted 93 lines · Nirjhar Roy (IBM)
Quoted 81 lines · Zorro Lang
Quoted 73 lines · Nirjhar Roy (IBM)
When this test was ran with btrfs with 64k sector/block size, it failed with
QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...This is what is going on:
Let us see the following set of operations:
--- With 4k sector size ---
$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 82M 95M 47% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 137M 40M 78% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 162M 15M 92% /mnt1/scratchNow let us repeat with 64k sector size
--- With 64k sector size ---$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec) $ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 104M 95M 53% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 pwrite: No space left on deviceQuoted 7 lines
Now, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.Quoted 4 lines
Thanks for this detailed explanation. As this's a ENOSPC test, so we must make sure this case still can uncover the original bug, before increasing the fs size. Can you make sure that? Or maybe we can replace that "80M" with a variable (according to "Avail" size).
Hi Filipe, Zorro, Wang,
The original commit message of generic/371 says that this test catches some excess space usage issues. Is(Are) there any patch(es) that fix this issue - so that I can remove the commits and check if the test expectedly fails with slightly large fssize i.e, 330M?
Quoted 6 lines
I did find some related commits [1] and I ran the test with 330M and 256M(the default size) after removing the commits[1] but the test passes with both the filesystem sizes. So I am guessing, this is not the patch that can test generic/371.
Hi,
I'm not familiar with btrfs patches, but from the commit history, I suspect it *might* be related with below commit:
commit
18513091af9483ba84328d42092bd4d42a3c958fAuthor: Wang Xiaoguang <[email protected]> Date: Mon Jul 25 15:51:40 2016 +0800Quoted 6 lines
btrfs: update btrfs_space_info's
bytes_may_usetimelyOr "
c0d2f6104e8ab2eb75e58e72494ad4b69c5227f8" :)Thanks,
ZorroHi Zorro, Filipe and Qu
Quoted 7 lines
Quoted 4 lines
Thank you for the pointers. I tried running the test reverting the above 2 mentioned commits, but the test seems to pass without these commits, so probably these are not the ones. I am not able to find any other relevant commits and the test case is also very old. Any further ideas?
Thanks, I don't have any objection on this patch now :) As this patchset still has unresolved review points from btrfs list, I think you can change and resend this patchset to get further reviewing.
I have already re-sent this patchset excluding this particular patch, and the rest of patches are merged (i.e, the changes for generic/562, btrfs/{200, 290}. Only this one is left.
--NR
Quoted 41 lines
Thanks,
ZorroQuoted 37 lines
--NR
Quoted 30 lines
Quoted 30 lines
--NR
Quoted 23 lines
Thanks,
ZorroQuoted 20 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>--- tests/generic/371 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/371 +1
-1@@ -19,7 +19,7 @@ _require_scratch _require_xfs_io_command "falloc" test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize" -_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1 _scratch_mount # Disable speculative post-EOF preallocation on XFS, which can grow fast enoughSignature · 1 line
2.34.1Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, BangaloreSignature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Signature · 3 lines
Nirjhar Roy Linux Kernel Developer IBM, Bangalore
Re: [PATCH 1/4] generic/371: Fix the test to be compatible block sizes upto 64k
In reply to Nirjhar Roy (IBM)Quoted 81 lines · Nirjhar Roy (IBM)
When this test was ran with btrfs with 64k sector/block size, it failed with
QA output created by 371
Silence is golden
+fallocate: No space left on device
+pwrite: No space left on device
+fallocate: No space left on device
+pwrite: No space left on device
+pwrite: No space left on device
...This is what is going on:
Let us see the following set of operations:
--- With 4k sector size ---
$ mkfs.btrfs -f -b 256m -s 4k -n 4k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.4378 sec (182.693 MiB/sec and 46769.3095 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 1.5M 175M 1% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 82M 95M 47% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0:00:01.25 (63.881 MiB/sec and 16353.4648 ops/sec)$ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 137M 40M 78% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 162M 15M 92% /mnt1/scratchNow let us repeat with 64k sector size
--- With 64k sector size ---$ mkfs.btrfs -f -b 256m -s 64k -n 64k /dev/loop0 $ mount /dev/loop0 /mnt1/scratch/ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t1 wrote 83886080/83886080 bytes at offset 0 80 MiB, 20480 ops; 0.8460 sec (94.553 MiB/sec and 24205.4914 ops/sec) $ $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 24M 175M 12% /mnt1/scratch$ sync $ df -h /dev/loop0 Filesystem Size Used Avail Use% Mounted on /dev/loop0 256M 104M 95M 53% /mnt1/scratch$ xfs_io -f -c "pwrite 0 80M" /mnt1/scratch/t2 pwrite: No space left on deviceNow, we can see that with 64k node size, 256M is not sufficient to hold 2 files worth 80M. For 64k, we can also see that the initial space usage on a fresh filesystem is 24M and for 4k its 1.5M. So because of higher node size, more metadata space is getting used. This test requires the size of the filesystem to be at least capable to hold 2 80M files.
Fix this by increasing the fs size from 256M to 330M.
Quoted 3 lines
Reported-by Disha Goel <[email protected]>Signed-off-by Nirjhar Roy (IBM) <[email protected]>---
Quoted 19 lines
tests/generic/371 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/371 +1
-1@@ -19,7 +19,7 @@ _require_scratch _require_xfs_io_command "falloc" test "$FSTYP" = "xfs" && _require_xfs_io_command "extsize" -_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1 +_scratch_mkfs_sized $((330 * 1024 * 1024)) >> $seqres.full 2>&1 _scratch_mount # Disable speculative post-EOF preallocation on XFS, which can grow fast enoughSignature · 1 line
2.34.1