nfs: test files written size as expected
Subsystems (2)
PATCH v2 1/1 nfs: test files written size as expected
Test nfs and its underlying fs, make sure file size as expected after writting a file, and the speculative allocation space can be shrunken.
---
Last year I sent a patch to fstests@, but it sometimes fails on the upstream kernel that year:
https://lore.kernel.org/fstests/Y3vTbHqT64gsQ573@magnolia/
And we didn't get a proper reason for that, so that patch was blocked. Now I found this case test passed on current upstream linux [1] (after loop running it a whole night). So I think it's time to rebase and re-send this patch to get review.
Thanks,
Zorro
[1]
FSTYP -- nfs PLATFORM -- Linux/x86_64 xxxx 6.5.0-rc4 #1 SMP PREEMPT_DYNAMIC Tue Aug 1 15:32:55 EDT 2023 MKFS_OPTIONS -- xxxx.redhat.com:/mnt/xfstests/scratch/nfs-server
MOUNT_OPTIONS -- -o vers=4.2 -o context=system_u:object_r:root_t:s0 xxxx.redhat.com:/mnt/xfstests/scratch/nfs-server /mnt/xfstests/scratch/nfs-client
nfs/002 4s ... 4s
Ran: nfs/002
Passed all 1 tests
tests/nfs/002 | 46 ++++++++++++++++++++++++++++++++++++++++++++++ tests/nfs/002.out | 2 ++ 2 files changed, 48 insertions(+) create mode 100755 tests/nfs/002 create mode 100644 tests/nfs/002.out
tests/nfs/002 new file +46 -0
@@ -0,0 +1,46 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Red Hat, Inc. All Rights Reserved. +# +# FS QA Test 002 +# +# Make sure nfs gets expected file size after writting a big sized file. It's +# not only testing nfs, test its underlying fs too. For example a known old bug +# on xfs (underlying fs) caused nfs get larger file size (e.g. 16M) after +# writting 10M data to a file. It's fixed by a series of patches around +# 579b62faa5fb16 ("xfs: add background scanning to clear eofblocks inodes") +# +. ./common/preamble +_begin_fstest auto quick rw + +# real QA test starts here +_supported_fs nfs +# Need a series of patches related with this patch +_fixed_by_kernel_commit 579b62faa5fb16 \ + "xfs: add background scanning to clear eofblocks inodes" +_require_test + +localfile=$TEST_DIR/testfile.$seq +rm -rf $localfile + +$XFS_IO_PROG -f -t -c "pwrite 0 10m" -c "fsync" $localfile >>$seqres.full 2>&1 +block_size=`stat -c '%B' $localfile` +iblocks_expected=$((10 * 1024 * 1024 / $block_size)) +# Try several times for the speculative allocated file size can be shrunken +res=1 +for ((i=0; i<10; i++));do + iblocks_real=`stat -c '%b' $localfile` + if [ "$iblocks_expected" = "$iblocks_real" ];then + res=0 + break + fi + sleep 10 +done +if [ $res -ne 0 ];then + echo "Write $iblocks_expected blocks, but get $iblocks_real blocks" +fi + +echo "Silence is golden" +# success, all done +status=0 +exit
tests/nfs/002.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 002 +Silence is golden
Signature · 1 line
2.40.1
Quoted 43 lines · Zorro Lang
Test nfs and its underlying fs, make sure file size as expected after writting a file, and the speculative allocation space can be shrunken.
Signed-off-by Zorro Lang <[email protected]>---
Last year I sent a patch to fstests@, but it sometimes fails on the upstream kernel that year:
https://lore.kernel.org/fstests/Y3vTbHqT64gsQ573@magnolia/
And we didn't get a proper reason for that, so that patch was blocked. Now I found this case test passed on current upstream linux [1] (after loop running it a whole night). So I think it's time to rebase and re-send this patch to get review.
Thanks,
Zorro[1]
FSTYP -- nfs PLATFORM -- Linux/x86_64 xxxx 6.5.0-rc4 #1 SMP PREEMPT_DYNAMIC Tue Aug 1 15:32:55 EDT 2023 MKFS_OPTIONS -- xxxx.redhat.com:/mnt/xfstests/scratch/nfs-server
MOUNT_OPTIONS-- -o vers=4.2 -o context=system_u:object_r:root_t:s0 xxxx.redhat.com:/mnt/xfstests/scratch/nfs-server/mnt/xfstests/scratch/nfs-clientnfs/002 4s ... 4s
Ran: nfs/002
Passed all 1 teststests/nfs/002 | 46 ++++++++++++++++++++++++++++++++++++++++++++++ tests/nfs/002.out | 2 ++ 2 files changed, 48 insertions(+) create mode 100755 tests/nfs/002 create mode 100644 tests/nfs/002.outtests/nfs/002 new file +1
-0@@ -0,0 +1,46 @@ +#! /bin/bash
tests/nfs/002 line 11
Quoted 10 lines
tests/nfs/002 new file +10
-0+# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Red Hat, Inc. All Rights Reserved. +# +# FS QA Test 002 +# +# Make sure nfs gets expected file size after writting a big sized file. It's +# not only testing nfs, test its underlying fs too. For example a known old bug +# on xfs (underlying fs) caused nfs get larger file size (e.g. 16M) after +# writting 10M data to a file. It's fixed by a series of patches around +# 579b62faa5fb16 ("xfs: add background scanning to clear eofblocks inodes")
Er... has this been banging around in the trunk for 11 years? ;)
Quoted 12 lines
tests/nfs/002 new file +12
-0+# +. ./common/preamble +_begin_fstest auto quick rw + +# real QA test starts here +_supported_fs nfs +# Need a series of patches related with this patch +_fixed_by_kernel_commit 579b62faa5fb16 \ + "xfs: add background scanning to clear eofblocks inodes" +_require_test + +localfile=$TEST_DIR/testfile.$seq
tests/nfs/002 line 33
Quoted 10 lines
tests/nfs/002 new file +10
-0+rm -rf $localfile + +$XFS_IO_PROG -f -t -c "pwrite 0 10m" -c "fsync" $localfile >>$seqres.full 2>&1 +block_size=`stat -c '%B' $localfile` +iblocks_expected=$((10 * 1024 * 1024 / $block_size)) +# Try several times for the speculative allocated file size can be shrunken +res=1 +for ((i=0; i<10; i++));do + iblocks_real=`stat -c '%b' $localfile` + if [ "$iblocks_expected" = "$iblocks_real" ];then
What happens if real < expected? Should there be some sort of bail out for unexpected things like that?
tests/nfs/002 line 38
Quoted 5 lines
tests/nfs/002 new file +5
-0+ res=0 + break + fi + sleep 10 +done
Though I guess the runtime is capped at ~100s so maybe it doesn't matter practically.
(What happens if xfs blockgc only runs every 5 minutes?)
--D
Quoted 19 lines
tests/nfs/002 new file +8
-0+if [ $res -ne 0 ];then + echo "Write $iblocks_expected blocks, but get $iblocks_real blocks" +fi + +echo "Silence is golden" +# success, all done +status=0 +exittests/nfs/002.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 002 +Silence is goldenSignature · 1 line
2.40.1
Quoted 48 lines · Darrick J. Wong
Quoted 47 lines · Zorro Lang
Test nfs and its underlying fs, make sure file size as expected after writting a file, and the speculative allocation space can be shrunken.
Signed-off-by Zorro Lang <[email protected]>---
Last year I sent a patch to fstests@, but it sometimes fails on the upstream kernel that year:
https://lore.kernel.org/fstests/Y3vTbHqT64gsQ573@magnolia/
And we didn't get a proper reason for that, so that patch was blocked. Now I found this case test passed on current upstream linux [1] (after loop running it a whole night). So I think it's time to rebase and re-send this patch to get review.
Thanks,
Zorro[1]
FSTYP -- nfs PLATFORM -- Linux/x86_64 xxxx 6.5.0-rc4 #1 SMP PREEMPT_DYNAMIC Tue Aug 1 15:32:55 EDT 2023 MKFS_OPTIONS -- xxxx.redhat.com:/mnt/xfstests/scratch/nfs-server
MOUNT_OPTIONS-- -o vers=4.2 -o context=system_u:object_r:root_t:s0 xxxx.redhat.com:/mnt/xfstests/scratch/nfs-server/mnt/xfstests/scratch/nfs-clientnfs/002 4s ... 4s
Ran: nfs/002
Passed all 1 teststests/nfs/002 | 46 ++++++++++++++++++++++++++++++++++++++++++++++ tests/nfs/002.out | 2 ++ 2 files changed, 48 insertions(+) create mode 100755 tests/nfs/002 create mode 100644 tests/nfs/002.outtests/nfs/002 new file +5
-0@@ -0,0 +1,46 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Red Hat, Inc. All Rights Reserved. +# +# FS QA Test 002
Quoted 8 lines
tests/nfs/002 line 11
Quoted 6 lines
tests/nfs/002 new file +6
-0+# +# Make sure nfs gets expected file size after writting a big sized file. It's +# not only testing nfs, test its underlying fs too. For example a known old bug +# on xfs (underlying fs) caused nfs get larger file size (e.g. 16M) after +# writting 10M data to a file. It's fixed by a series of patches around +# 579b62faa5fb16 ("xfs: add background scanning to clear eofblocks inodes")Er... has this been banging around in the trunk for 11 years? ;)
Yeah, that's an old enough test case :-D I tried to tidy our internal test cases, felt this case can be in fstests.
Quoted 18 lines
Quoted 18 lines
tests/nfs/002 new file +17
-0+# +. ./common/preamble +_begin_fstest auto quick rw + +# real QA test starts here +_supported_fs nfs +# Need a series of patches related with this patch +_fixed_by_kernel_commit 579b62faa5fb16 \ + "xfs: add background scanning to clear eofblocks inodes" +_require_test + +localfile=$TEST_DIR/testfile.$seq +rm -rf $localfile + +$XFS_IO_PROG -f -t -c "pwrite 0 10m" -c "fsync" $localfile >>$seqres.full 2>&1 +block_size=`stat -c '%B' $localfile` +iblocks_expected=$((10 * 1024 * 1024 / $block_size))
Quoted 8 lines
tests/nfs/002 line 33
Quoted 5 lines
tests/nfs/002 new file +5
-0+# Try several times for the speculative allocated file size can be shrunken +res=1 +for ((i=0; i<10; i++));do + iblocks_real=`stat -c '%b' $localfile` + if [ "$iblocks_expected" = "$iblocks_real" ];thenWhat happens if real < expected? Should there be some sort of bail out for unexpected things like that?
Hmm... I never thought that. I saw the real >= expected, is there any chance to get real < expected?
Quoted 9 lines
tests/nfs/002 line 38
Quoted 6 lines
tests/nfs/002 new file +5
-0+ res=0 + break + fi + sleep 10 +doneThough I guess the runtime is capped at ~100s so maybe it doesn't matter practically.
Mostly the test done in several seconds in my testing:
FSTYP -- nfs PLATFORM -- Linux/x86_64 hp-dl360g9-06 6.5.0-rc4 #1 SMP PREEMPT_DYNAMIC Tue Aug 1 15:32:55 EDT 2023 MKFS_OPTIONS -- hp-dl360g9-06.rhts.eng.pek2.redhat.com:/mnt/xfstests/scratch/nfs-server
MOUNT_OPTIONS -- -o vers=4.2 -o context=system_u:object_r:root_t:s0 hp-dl360g9-06.rhts.eng.pek2.redhat.com:/mnt/xfstests/scratch/nfs-server /mnt/xfstests/scratch/nfs-client
nfs/002 5s ... 4s
Ran: nfs/002
Passed all 1 tests
Quoted 2 lines
(What happens if xfs blockgc only runs every 5 minutes?)
How can can make that happen? If the 100s isn't enough, is there an upper limit, or how to make an upper limit?
Thanks,
Zorro
Quoted 23 lines
--D
Quoted 19 lines
tests/nfs/002 new file +8
-0+if [ $res -ne 0 ];then + echo "Write $iblocks_expected blocks, but get $iblocks_real blocks" +fi + +echo "Silence is golden" +# success, all done +status=0 +exittests/nfs/002.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 002 +Silence is goldenSignature · 1 line
2.40.1
Quoted 88 lines · Zorro Lang
Quoted 48 lines · Darrick J. Wong
Quoted 47 lines · Zorro Lang
Test nfs and its underlying fs, make sure file size as expected after writting a file, and the speculative allocation space can be shrunken.
Signed-off-by Zorro Lang <[email protected]>---
Last year I sent a patch to fstests@, but it sometimes fails on the upstream kernel that year:
https://lore.kernel.org/fstests/Y3vTbHqT64gsQ573@magnolia/
And we didn't get a proper reason for that, so that patch was blocked. Now I found this case test passed on current upstream linux [1] (after loop running it a whole night). So I think it's time to rebase and re-send this patch to get review.
Thanks,
Zorro[1]
FSTYP -- nfs PLATFORM -- Linux/x86_64 xxxx 6.5.0-rc4 #1 SMP PREEMPT_DYNAMIC Tue Aug 1 15:32:55 EDT 2023 MKFS_OPTIONS -- xxxx.redhat.com:/mnt/xfstests/scratch/nfs-server
MOUNT_OPTIONS-- -o vers=4.2 -o context=system_u:object_r:root_t:s0 xxxx.redhat.com:/mnt/xfstests/scratch/nfs-server/mnt/xfstests/scratch/nfs-clientnfs/002 4s ... 4s
Ran: nfs/002
Passed all 1 teststests/nfs/002 | 46 ++++++++++++++++++++++++++++++++++++++++++++++ tests/nfs/002.out | 2 ++ 2 files changed, 48 insertions(+) create mode 100755 tests/nfs/002 create mode 100644 tests/nfs/002.outtests/nfs/002 new file +5
-0@@ -0,0 +1,46 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Red Hat, Inc. All Rights Reserved. +# +# FS QA Test 002Quoted 8 lines
tests/nfs/002 line 11
Quoted 6 lines
tests/nfs/002 new file +6
-0+# +# Make sure nfs gets expected file size after writting a big sized file. It's +# not only testing nfs, test its underlying fs too. For example a known old bug +# on xfs (underlying fs) caused nfs get larger file size (e.g. 16M) after +# writting 10M data to a file. It's fixed by a series of patches around +# 579b62faa5fb16 ("xfs: add background scanning to clear eofblocks inodes")Er... has this been banging around in the trunk for 11 years? ;)
Yeah, that's an old enough test case :-D I tried to tidy our internal test cases, felt this case can be in fstests.
Quoted 18 lines
Quoted 18 lines
tests/nfs/002 new file +17
-0+# +. ./common/preamble +_begin_fstest auto quick rw + +# real QA test starts here +_supported_fs nfs +# Need a series of patches related with this patch +_fixed_by_kernel_commit 579b62faa5fb16 \ + "xfs: add background scanning to clear eofblocks inodes" +_require_test + +localfile=$TEST_DIR/testfile.$seq +rm -rf $localfile + +$XFS_IO_PROG -f -t -c "pwrite 0 10m" -c "fsync" $localfile >>$seqres.full 2>&1 +block_size=`stat -c '%B' $localfile` +iblocks_expected=$((10 * 1024 * 1024 / $block_size))Quoted 8 lines
tests/nfs/002 line 33
Quoted 5 lines
tests/nfs/002 new file +5
-0+# Try several times for the speculative allocated file size can be shrunken +res=1 +for ((i=0; i<10; i++));do + iblocks_real=`stat -c '%b' $localfile` + if [ "$iblocks_expected" = "$iblocks_real" ];thenWhat happens if real < expected? Should there be some sort of bail out for unexpected things like that?
Quoted 2 lines
Hmm... I never thought that. I saw the real >= expected, is there any chance to get real < expected?
<shrug> Suppose the NFS server is running on top of a filesystem that supports compression and i_blocks as returned by stat reflects that?
--D
Quoted 12 lines
Quoted 9 lines
tests/nfs/002 line 38
Quoted 6 lines
tests/nfs/002 new file +5
-0+ res=0 + break + fi + sleep 10 +doneThough I guess the runtime is capped at ~100s so maybe it doesn't matter practically.
Mostly the test done in several seconds in my testing:
Quoted 8 lines
FSTYP -- nfs PLATFORM -- Linux/x86_64 hp-dl360g9-06 6.5.0-rc4 #1 SMP PREEMPT_DYNAMIC Tue Aug 1 15:32:55 EDT 2023 MKFS_OPTIONS -- hp-dl360g9-06.rhts.eng.pek2.redhat.com:/mnt/xfstests/scratch/nfs-server
MOUNT_OPTIONS-- -o vers=4.2 -o context=system_u:object_r:root_t:s0 hp-dl360g9-06.rhts.eng.pek2.redhat.com:/mnt/xfstests/scratch/nfs-server/mnt/xfstests/scratch/nfs-clientnfs/002 5s ... 4s
Ran: nfs/002
Passed all 1 tests
Doesn't xfs remove the speculative preallocations every time a write fd is closed?
Yes, it does do that:
https://lore.kernel.org/linux-xfs/155259894034.30230.7188877605950498518.stgit@magnolia/
IOWs, how is this test actually checking the behavior of background blockgc clearing out speculative preallocations?
Quoted 4 lines
Quoted 1 line
(What happens if xfs blockgc only runs every 5 minutes?)
How can can make that happen? If the 100s isn't enough, is there an upper limit, or how to make an upper limit?
There's no way to tell over NFS...
--D
Quoted 28 lines
Thanks,
ZorroQuoted 23 lines
--D
Quoted 19 lines
tests/nfs/002 new file +8
-0+if [ $res -ne 0 ];then + echo "Write $iblocks_expected blocks, but get $iblocks_real blocks" +fi + +echo "Silence is golden" +# success, all done +status=0 +exittests/nfs/002.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 002 +Silence is goldenSignature · 1 line
2.40.1
Quoted 118 lines · Darrick J. Wong
Quoted 88 lines · Zorro Lang
Quoted 48 lines · Darrick J. Wong
Quoted 47 lines · Zorro Lang
Test nfs and its underlying fs, make sure file size as expected after writting a file, and the speculative allocation space can be shrunken.
Signed-off-by Zorro Lang <[email protected]>---
Last year I sent a patch to fstests@, but it sometimes fails on the upstream kernel that year:
https://lore.kernel.org/fstests/Y3vTbHqT64gsQ573@magnolia/
And we didn't get a proper reason for that, so that patch was blocked. Now I found this case test passed on current upstream linux [1] (after loop running it a whole night). So I think it's time to rebase and re-send this patch to get review.
Thanks,
Zorro[1]
FSTYP -- nfs PLATFORM -- Linux/x86_64 xxxx 6.5.0-rc4 #1 SMP PREEMPT_DYNAMIC Tue Aug 1 15:32:55 EDT 2023 MKFS_OPTIONS -- xxxx.redhat.com:/mnt/xfstests/scratch/nfs-server
MOUNT_OPTIONS-- -o vers=4.2 -o context=system_u:object_r:root_t:s0 xxxx.redhat.com:/mnt/xfstests/scratch/nfs-server/mnt/xfstests/scratch/nfs-clientnfs/002 4s ... 4s
Ran: nfs/002
Passed all 1 teststests/nfs/002 | 46 ++++++++++++++++++++++++++++++++++++++++++++++ tests/nfs/002.out | 2 ++ 2 files changed, 48 insertions(+) create mode 100755 tests/nfs/002 create mode 100644 tests/nfs/002.outtests/nfs/002 new file +5
-0@@ -0,0 +1,46 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2023 Red Hat, Inc. All Rights Reserved. +# +# FS QA Test 002Quoted 8 lines
tests/nfs/002 line 11
Quoted 6 lines
tests/nfs/002 new file +6
-0+# +# Make sure nfs gets expected file size after writting a big sized file. It's +# not only testing nfs, test its underlying fs too. For example a known old bug +# on xfs (underlying fs) caused nfs get larger file size (e.g. 16M) after +# writting 10M data to a file. It's fixed by a series of patches around +# 579b62faa5fb16 ("xfs: add background scanning to clear eofblocks inodes")Er... has this been banging around in the trunk for 11 years? ;)
Yeah, that's an old enough test case :-D I tried to tidy our internal test cases, felt this case can be in fstests.
Quoted 18 lines
Quoted 18 lines
tests/nfs/002 new file +17
-0+# +. ./common/preamble +_begin_fstest auto quick rw + +# real QA test starts here +_supported_fs nfs +# Need a series of patches related with this patch +_fixed_by_kernel_commit 579b62faa5fb16 \ + "xfs: add background scanning to clear eofblocks inodes" +_require_test + +localfile=$TEST_DIR/testfile.$seq +rm -rf $localfile + +$XFS_IO_PROG -f -t -c "pwrite 0 10m" -c "fsync" $localfile >>$seqres.full 2>&1 +block_size=`stat -c '%B' $localfile` +iblocks_expected=$((10 * 1024 * 1024 / $block_size))Quoted 8 lines
tests/nfs/002 line 33
Quoted 5 lines
tests/nfs/002 new file +5
-0+# Try several times for the speculative allocated file size can be shrunken +res=1 +for ((i=0; i<10; i++));do + iblocks_real=`stat -c '%b' $localfile` + if [ "$iblocks_expected" = "$iblocks_real" ];thenWhat happens if real < expected? Should there be some sort of bail out for unexpected things like that?
Quoted 2 lines
Hmm... I never thought that. I saw the real >= expected, is there any chance to get real < expected?
<shrug> Suppose the NFS server is running on top of a filesystem that supports compression and
i_blocksas returned by stat reflects that?--D
Quoted 12 lines
Quoted 9 lines
tests/nfs/002 line 38
Quoted 6 lines
tests/nfs/002 new file +5
-0+ res=0 + break + fi + sleep 10 +doneThough I guess the runtime is capped at ~100s so maybe it doesn't matter practically.
Mostly the test done in several seconds in my testing:
Quoted 8 lines
FSTYP -- nfs PLATFORM -- Linux/x86_64 hp-dl360g9-06 6.5.0-rc4 #1 SMP PREEMPT_DYNAMIC Tue Aug 1 15:32:55 EDT 2023 MKFS_OPTIONS -- hp-dl360g9-06.rhts.eng.pek2.redhat.com:/mnt/xfstests/scratch/nfs-server
MOUNT_OPTIONS-- -o vers=4.2 -o context=system_u:object_r:root_t:s0 hp-dl360g9-06.rhts.eng.pek2.redhat.com:/mnt/xfstests/scratch/nfs-server/mnt/xfstests/scratch/nfs-clientnfs/002 5s ... 4s
Ran: nfs/002
Passed all 1 tests
Quoted 8 lines
Doesn't xfs remove the speculative preallocations every time a write fd is closed?
Yes, it does do that:
https://lore.kernel.org/linux-xfs/155259894034.30230.7188877605950498518.stgit@magnolia/IOWs, how is this test actually checking the behavior of background blockgc clearing out speculative preallocations?
OK, looks like this case is invalid. I'll keep it :)
Thanks,
Zorro
Quoted 39 lines
Quoted 5 lines
Quoted 2 lines
(What happens if xfs blockgc only runs every 5 minutes?)
How can can make that happen? If the 100s isn't enough, is there an upper limit, or how to make an upper limit?
There's no way to tell over NFS...
--D
Quoted 28 lines
Thanks,
ZorroQuoted 23 lines
--D
Quoted 19 lines
tests/nfs/002 new file +8
-0+if [ $res -ne 0 ];then + echo "Write $iblocks_expected blocks, but get $iblocks_real blocks" +fi + +echo "Silence is golden" +# success, all done +status=0 +exittests/nfs/002.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 002 +Silence is goldenSignature · 1 line
2.40.1