Improvements for NFS
Subsystems (2)
Anna Schumaker
maintainercover Improvements for NFS
From: Anna Schumaker <[email protected]>
This is a collection of fixes that I've had in my private xfstests instance on my NFS client VM for testing.
I've been using the "-R xunit" option lately, so the first patch fixes up
the PLATFORM and CHECK_OPTIONS output in the resulting xunit file. I wouldn't be surprised if there is a better way to do this, so please let me know!
The second patch moves a very long running test (generic/531) out of the quick group because I'm tired of killing it after a half hour. Maybe it's quick on non networked filesystems, though? I had some other ideas on how to fix my issue that I included in the patch description.
The last two patches add some extra checks for functionality that NFS doesn't support, otherwise the tests are getting marked as "failed" with "whatever isn't supposted" in the test output log. After these patches the tests are simply skipped.
Thoughts?
Anna
Anna Schumaker (4):
check: Export CHECK_OPTIONS and PLATFORM for Xunit Reporting generic/531: Move test from 'quick' group to 'stress' generic/578: Test that filefrag is supported before running generic/633: Check if idmapped mounts are supported before running
check | 2 ++ common/rc | 14 ++++++++++++++ tests/generic/531 | 2 +- tests/generic/578 | 2 +- tests/generic/633 | 1 + 5 files changed, 19 insertions(+), 2 deletions(-)
Signature · 1 line
2.35.1
Anna Schumaker
maintainerPATCH 1/4 check: Export CHECK_OPTIONS and PLATFORM for Xunit Reporting
From: Anna Schumaker <[email protected]>
I found that running ./check by hand with Xunit reporting enabled caused the PLATFORM variable to be unset, and the CHECK_OPTIONS was the default "-g auto" instead of the "-g quick" that I also passed in. This patch sets both variables so Xunit reporting is accurate.
---
Maybe there is a better way to do this? If so please let me know!
--- check | 2 ++ 1 file changed, 2 insertions(+)
check +2 -0
@@ -273,6 +273,7 @@ _prepare_test_list() } # Process command arguments first. +export CHECK_OPTIONS="$*" while [ $# -gt 0 ]; do case "$1" in -\? | -h | --help) usage ;;
@@ -360,6 +361,7 @@ if ! . ./common/rc; then echo "check: failed to source common/rc" exit 1 fi +export PLATFORM=$(_full_platform_details) if [ -n "$subdir_xfile" ]; then for d in $SRC_GROUPS $FSTYP; do
Signature · 1 line
2.35.1
Anna Schumaker
maintainerPATCH 2/4 generic/531: Move test from 'quick' group to 'stress'
In reply to Anna SchumakerFrom: Anna Schumaker <[email protected]>
The comment up top says this is a stress test, so at the very least it should be added to this group. As for removing it from the quick group, making this test variable on the number of CPUs means this test could take a very long time to finish (I'm unsure exactly how long on NFS v4.1 because I usually kill it after a half hour or so)
---
I have thought of two alternatives to this patch that would work for me:
1) Could we add an _unsupported_fs function which is the opposite of _supported_fs to prevent tests from running on specific filesystems?
2) Would it be okay to check if $FSTYP == "nfs" when setting nr_cpus, and set it to 1 instead? Perhaps through a function in common/rc that other tests can use if they scale work based on cpu-count?
--- tests/generic/531 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/531 +1 -1
@@ -12,7 +12,7 @@ # Use every CPU possible to stress the filesystem. # . ./common/preamble -_begin_fstest auto quick unlink +_begin_fstest auto stress unlink testfile=$TEST_DIR/$seq.txt # Import common functions.
Signature · 1 line
2.35.1
Re: [PATCH 2/4] generic/531: Move test from 'quick' group to 'stress'
Quoted 8 lines · Anna Schumaker
From: Anna Schumaker <[email protected]>
The comment up top says this is a stress test, so at the very least it should be added to this group. As for removing it from the quick group, making this test variable on the number of CPUs means this test could take a very long time to finish (I'm unsure exactly how long on NFS v4.1 because I usually kill it after a half hour or so)
Quoted 8 lines
Signed-off-by Anna Schumaker <[email protected]>---
I have thought of two alternatives to this patch that would work for me:
1) Could we add an_unsupported_fsfunction which is the opposite of_supported_fsto prevent tests from running on specific filesystems?
2) Would it be okay to check if $FSTYP == "nfs" when settingnr_cpus, and set it to 1 instead? Perhaps through a function in common/rc that other tests can use if they scale work based on cpu-count?
How about we create a function to estimate fs threading scalability? There are probably (simple) filesystems out there with a Big Filesystem Lock that won't benefit from more CPUs pounding on it...
# Estimate how many writer threads we should start to stress test this
# type of filesystem.
_estimate_threading_factor() {
case "$FSTYP" in
"nfs")
echo 1;;
*)
echo $((2 * $(getconf _NPROCESSORS_ONLN) ));;
esac
}and later:
nr_cpus=$(_estimate_threading_factor)
Once something like this is landed, we can customize for each FSTYP. I suspect that XFS on spinning rust might actually want "2" here, not nr_cpus*2, given the sporadic complaints about this test taking much longer for a few people.
Quoted 9 lines
--- tests/generic/531 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/531 +0
-0
tests/generic/531 line 15
Quoted 5 lines
tests/generic/531 +1
-1@@ -12,7 +12,7 @@ # Use every CPU possible to stress the filesystem. # . ./common/preamble -_begin_fstest auto quick unlink +_begin_fstest auto stress unlink
As for this change itself,
--D
Quoted 6 lines
tests/generic/531 +0
-0testfile=$TEST_DIR/$seq.txt # Import common functions.Signature · 1 line
2.35.1
Anna Schumaker
maintainerRe: [PATCH 2/4] generic/531: Move test from 'quick' group to 'stress'
Quoted 34 lines · Darrick J. Wong
Quoted 8 lines · Anna Schumaker
From: Anna Schumaker <[email protected]>
The comment up top says this is a stress test, so at the very least it should be added to this group. As for removing it from the quick group, making this test variable on the number of CPUs means this test could take a very long time to finish (I'm unsure exactly how long on NFS v4.1 because I usually kill it after a half hour or so)
Quoted 8 lines
Signed-off-by Anna Schumaker <[email protected]>---
I have thought of two alternatives to this patch that would work for me:
1) Could we add an_unsupported_fsfunction which is the opposite of_supported_fsto prevent tests from running on specific filesystems?
2) Would it be okay to check if $FSTYP == "nfs" when settingnr_cpus, and set it to 1 instead? Perhaps through a function in common/rc that other tests can use if they scale work based on cpu-count?How about we create a function to estimate fs threading scalability? There are probably (simple) filesystems out there with a Big Filesystem Lock that won't benefit from more CPUs pounding on it...
# Estimate how many writer threads we should start to stress test this
# type of filesystem. _estimate_threading_factor() { case "$FSTYP" in "nfs") echo 1;; *) echo $((2 * $(getconf _NPROCESSORS_ONLN) ));; esac }
Quoted 8 lines
and later:
nr_cpus=$(_estimate_threading_factor)
Once something like this is landed, we can customize for each FSTYP. I suspect that XFS on spinning rust might actually want "2" here, not nr_cpus*2, given the sporadic complaints about this test taking much longer for a few people.
Sure. Should I do a git grep for "nr_cpus" on the other tests and update them all at the same time, or just leave it with this one file to start?
Anna
Quoted 28 lines
Quoted 10 lines
--- tests/generic/531 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/531 +0
-0tests/generic/531 line 15
Quoted 5 lines
tests/generic/531 +1
-1@@ -12,7 +12,7 @@ # Use every CPU possible to stress the filesystem. # . ./common/preamble -_begin_fstest auto quick unlink +_begin_fstest auto stress unlinkAs for this change itself,
Reviewed-by Darrick J. Wong <[email protected]>--D
Quoted 6 lines
tests/generic/531 +0
-1testfile=$TEST_DIR/$seq.txt # Import common functions. --2.35.1
Re: [PATCH 2/4] generic/531: Move test from 'quick' group to 'stress'
Quoted 39 lines · Anna Schumaker
Quoted 38 lines · Darrick J. Wong
Quoted 8 lines · Anna Schumaker
From: Anna Schumaker <[email protected]>
The comment up top says this is a stress test, so at the very least it should be added to this group. As for removing it from the quick group, making this test variable on the number of CPUs means this test could take a very long time to finish (I'm unsure exactly how long on NFS v4.1 because I usually kill it after a half hour or so)
Quoted 8 lines
Signed-off-by Anna Schumaker <[email protected]>---
I have thought of two alternatives to this patch that would work for me:
1) Could we add an_unsupported_fsfunction which is the opposite of_supported_fsto prevent tests from running on specific filesystems?
2) Would it be okay to check if $FSTYP == "nfs" when settingnr_cpus, and set it to 1 instead? Perhaps through a function in common/rc that other tests can use if they scale work based on cpu-count?How about we create a function to estimate fs threading scalability? There are probably (simple) filesystems out there with a Big Filesystem Lock that won't benefit from more CPUs pounding on it...
# Estimate how many writer threads we should start to stress test this
# type of filesystem. _estimate_threading_factor() { case "$FSTYP" in "nfs") echo 1;; *) echo $((2 * $(getconf _NPROCESSORS_ONLN) ));; esac }and later:
nr_cpus=$(_estimate_threading_factor)
Quoted 8 lines
Quoted 4 lines
Once something like this is landed, we can customize for each FSTYP. I suspect that XFS on spinning rust might actually want "2" here, not nr_cpus*2, given the sporadic complaints about this test taking much longer for a few people.
Sure. Should I do a
git grepfor "nr_cpus" on the other tests and update them all at the same time, or just leave it with this one file to start?
Ugh, what a mess...
Unbeknownst to me, "$here/src/feature -o" also returns the CPU count. If you want to get started on fixing other tests, I'd say ... add the new helper here for this test, and attach any other conversions as new patches at the end of the series.
--D
Quoted 29 lines
Anna
Quoted 28 lines
Quoted 10 lines
--- tests/generic/531 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/531 +0
-0tests/generic/531 line 15
Quoted 5 lines
tests/generic/531 +1
-1@@ -12,7 +12,7 @@ # Use every CPU possible to stress the filesystem. # . ./common/preamble -_begin_fstest auto quick unlink +_begin_fstest auto stress unlinkAs for this change itself,
Reviewed-by Darrick J. Wong <[email protected]>--D
Quoted 6 lines
tests/generic/531 +0
-1testfile=$TEST_DIR/$seq.txt # Import common functions. --2.35.1
Anna Schumaker
maintainerPATCH 3/4 generic/578: Test that filefrag is supported before running
In reply to Anna SchumakerFrom: Anna Schumaker <[email protected]>
NFS does not support FIBMAP/FIEMAP, so the check for non-shared extents on NFS v4.2 always fails with the message: "FIBMAP/FIEMAP unsupported". I added the _require_filefrag check for NFS and other filesystems that don't have FIEMAP or FIBMAP support.
--- common/rc | 14 ++++++++++++++ tests/generic/578 | 2 +- 2 files changed, 15 insertions(+), 1 deletion(-)
common/rc +14 -0
@@ -4673,6 +4673,20 @@ _require_inode_limits() fi } +_require_filefrag() +{ + _require_command "$FILEFRAG_PROG" filefrag + + local file="$TEST_DIR/filefrag_testfile" + + echo "XX" > $file + ${FILEFRAG_PROG} $file 2>&1 | grep -q "FIBMAP/FIEMAP[[:space:]]*unsupported" + if [ $? -eq 0 ]; then + _notrun "FIBMAP/FIEMAP not supported by this filesystem" + fi + rm -f $file +} + _require_filefrag_options() { _require_command "$FILEFRAG_PROG" filefrag
tests/generic/578 +1 -1
@@ -23,7 +23,7 @@ _cleanup() # real QA test starts here _supported_fs generic _require_test_program "mmap-write-concurrent" -_require_command "$FILEFRAG_PROG" filefrag +_require_filefrag _require_test_reflink _require_cp_reflink
Signature · 1 line
2.35.1
Re: [PATCH 3/4] generic/578: Test that filefrag is supported before running
Quoted 18 lines · Anna Schumaker
From: Anna Schumaker <[email protected]>
NFS does not support FIBMAP/FIEMAP, so the check for non-shared extents on NFS v4.2 always fails with the message: "FIBMAP/FIEMAP unsupported". I added the
_require_filefragcheck for NFS and other filesystems that don't have FIEMAP or FIBMAP support.Signed-off-by Anna Schumaker <[email protected]>--- common/rc | 14 ++++++++++++++ tests/generic/578 | 2 +- 2 files changed, 15 insertions(+), 1 deletion(-)common/rc +0
-0
common/rc _require_inode_limits() · line 4682
Quoted 10 lines
common/rc +7
-0@@ -4673,6 +4673,20 @@ _require_inode_limits() fi } +_require_filefrag() +{ + _require_command "$FILEFRAG_PROG" filefrag + + local file="$TEST_DIR/filefrag_testfile" + + echo "XX" > $file
Nit: You might want to rm -f $file before echoing into it, just in case some future knave ;) sets up that pathname to point to a named pipe or something that will hang fstests...
common/rc _require_inode_limits() · line 4684
Quoted 2 lines
common/rc +2
-0+ ${FILEFRAG_PROG} $file 2>&1 | grep -q "FIBMAP/FIEMAP[[:space:]]*unsupported" + if [ $? -eq 0 ]; then
...and rm it again here to avoid leaving test files around.
Otherwise this looks ok to me.
--D
Quoted 24 lines
common/rc +5
-0+ _notrun "FIBMAP/FIEMAP not supported by this filesystem" + fi + rm -f $file +} + _require_filefrag_options() { _require_command "$FILEFRAG_PROG" filefragtests/generic/578 +1
-1@@ -23,7 +23,7 @@ _cleanup() # real QA test starts here _supported_fs generic _require_test_program "mmap-write-concurrent" -_require_command "$FILEFRAG_PROG" filefrag +_require_filefrag _require_test_reflink _require_cp_reflinkSignature · 1 line
2.35.1
Anna Schumaker
maintainerRe: [PATCH 3/4] generic/578: Test that filefrag is supported before running
Quoted 35 lines · Darrick J. Wong
Quoted 18 lines · Anna Schumaker
From: Anna Schumaker <[email protected]>
NFS does not support FIBMAP/FIEMAP, so the check for non-shared extents on NFS v4.2 always fails with the message: "FIBMAP/FIEMAP unsupported". I added the
_require_filefragcheck for NFS and other filesystems that don't have FIEMAP or FIBMAP support.Signed-off-by Anna Schumaker <[email protected]>--- common/rc | 14 ++++++++++++++ tests/generic/578 | 2 +- 2 files changed, 15 insertions(+), 1 deletion(-)common/rc +0
-0common/rc _require_inode_limits() · line 4682
Quoted 10 lines
common/rc +7
-0@@ -4673,6 +4673,20 @@ _require_inode_limits() fi } +_require_filefrag() +{ + _require_command "$FILEFRAG_PROG" filefrag + + local file="$TEST_DIR/filefrag_testfile" + + echo "XX" > $fileNit: You might want to rm -f $file before echoing into it, just in case some future knave ;) sets up that pathname to point to a named pipe or something that will hang fstests...
Quoted 4 lines
common/rc _require_inode_limits() · line 4684
Quoted 2 lines
common/rc +2
-0+ ${FILEFRAG_PROG} $file 2>&1 | grep -q "FIBMAP/FIEMAP[[:space:]]*unsupported" + if [ $? -eq 0 ]; then...and rm it again here to avoid leaving test files around.
Sure, I can make that change. Should I update _require_filefrag_options() and _require_fibmap() to follow the same pattern while I'm at it? The new function was based off of those.
Anna
Quoted 29 lines
Otherwise this looks ok to me.
--D
Quoted 24 lines
common/rc +5
-0+ _notrun "FIBMAP/FIEMAP not supported by this filesystem" + fi + rm -f $file +} + _require_filefrag_options() { _require_command "$FILEFRAG_PROG" filefragtests/generic/578 +1
-2@@ -23,7 +23,7 @@ _cleanup() # real QA test starts here _supported_fs generic _require_test_program "mmap-write-concurrent" -_require_command "$FILEFRAG_PROG" filefrag +_require_filefrag _require_test_reflink _require_cp_reflink --2.35.1
Re: [PATCH 3/4] generic/578: Test that filefrag is supported before running
Quoted 36 lines · Anna Schumaker
Quoted 35 lines · Darrick J. Wong
Quoted 18 lines · Anna Schumaker
From: Anna Schumaker <[email protected]>
NFS does not support FIBMAP/FIEMAP, so the check for non-shared extents on NFS v4.2 always fails with the message: "FIBMAP/FIEMAP unsupported". I added the
_require_filefragcheck for NFS and other filesystems that don't have FIEMAP or FIBMAP support.Signed-off-by Anna Schumaker <[email protected]>--- common/rc | 14 ++++++++++++++ tests/generic/578 | 2 +- 2 files changed, 15 insertions(+), 1 deletion(-)common/rc +0
-0common/rc _require_inode_limits() · line 4682
Quoted 10 lines
common/rc +7
-0@@ -4673,6 +4673,20 @@ _require_inode_limits() fi } +_require_filefrag() +{ + _require_command "$FILEFRAG_PROG" filefrag + + local file="$TEST_DIR/filefrag_testfile" + + echo "XX" > $fileNit: You might want to rm -f $file before echoing into it, just in case some future knave ;) sets up that pathname to point to a named pipe or something that will hang fstests...
Quoted 8 lines
Quoted 4 lines
common/rc _require_inode_limits() · line 4684
Quoted 2 lines
common/rc +2
-0+ ${FILEFRAG_PROG} $file 2>&1 | grep -q "FIBMAP/FIEMAP[[:space:]]*unsupported" + if [ $? -eq 0 ]; then...and rm it again here to avoid leaving test files around.
Sure, I can make that change. Should I update
_require_filefrag_options()and_require_fibmap()to follow the same pattern while I'm at it? The new function was based off of those.
Yes please, and thank you! :)
--D
Quoted 31 lines
Anna
Quoted 29 lines
Otherwise this looks ok to me.
--D
Quoted 24 lines
common/rc +5
-0+ _notrun "FIBMAP/FIEMAP not supported by this filesystem" + fi + rm -f $file +} + _require_filefrag_options() { _require_command "$FILEFRAG_PROG" filefragtests/generic/578 +1
-2@@ -23,7 +23,7 @@ _cleanup() # real QA test starts here _supported_fs generic _require_test_program "mmap-write-concurrent" -_require_command "$FILEFRAG_PROG" filefrag +_require_filefrag _require_test_reflink _require_cp_reflink --2.35.1
Anna Schumaker
maintainerPATCH 4/4 generic/633: Check if idmapped mounts are supported before running
In reply to Anna SchumakerFrom: Anna Schumaker <[email protected]>
This appears to have been missed when the test was added.
--- tests/generic/633 | 1 + 1 file changed, 1 insertion(+)
tests/generic/633 +1 -0
@@ -15,6 +15,7 @@ _begin_fstest auto quick atime attr cap idmapped io_uring mount perms rw unlink # real QA test starts here _supported_fs generic +_require_idmapped_mounts _require_test echo "Silence is golden"
Signature · 1 line
2.35.1
Re: [PATCH 4/4] generic/633: Check if idmapped mounts are supported before running
Quoted 4 lines · Anna Schumaker
From: Anna Schumaker <[email protected]>
This appears to have been missed when the test was added.
It's removed on purpose to also verify idmapped mounts unrelated tests.
Generic/633 fails on NFS because of setgid inheritance behavior of NFS. Already reported it to the community[1].
--
Su
Quoted 18 lines
Signed-off-by Anna Schumaker <[email protected]>--- tests/generic/633 | 1 + 1 file changed, 1 insertion(+)tests/generic/633 +0
-0@@ -15,6 +15,7 @@ _begin_fstest auto quick atime attr capidmapped
io_uringmount perms rw unlink# real QA test starts here
_supported_fsgeneric
+_require_idmapped_mounts
_require_testecho "Silence is golden"