fstests: disable generic duperemove tests for NFS and others
PATCH 1/1 fstests: disable generic duperemove tests for NFS and others
From: Chuck Lever <[email protected]>
On NFS mounts, at least, generic/559, 560, and 561 run for a very long time, and usually fail.
The above tests already gate on whether duperemove is installed on the test system, but when fstests is installed as part of an automated workflow designed to handle many filesystem types, duperemove is installed by default.
duperemove(8) states:
Deduplication is currently only supported by the btrfs and xfs filesystem.
Ensure that the generic dedupe tests are run on only filesystems where duperemove is known to work.
--- tests/generic/559 | 1 + tests/generic/560 | 1 + tests/generic/561 | 1 + 3 files changed, 3 insertions(+)
tests/generic/559 +1 -0
@@ -13,6 +13,7 @@ _begin_fstest auto stress dedupe . ./common/filter . ./common/reflink +_supported_fs btrfs xfs _require_scratch_duperemove fssize=$((2 * 1024 * 1024 * 1024))
tests/generic/560 +1 -0
@@ -15,6 +15,7 @@ _begin_fstest auto stress dedupe . ./common/filter . ./common/reflink +_supported_fs btrfs xfs _require_scratch_duperemove _scratch_mkfs > $seqres.full 2>&1
tests/generic/561 +1 -0
@@ -28,6 +28,7 @@ _cleanup() . ./common/filter . ./common/reflink +_supported_fs btrfs xfs _require_scratch_duperemove _scratch_mkfs > $seqres.full 2>&1
Signature · 1 line
2.47.0
Quoted 10 lines · On Sun, Dec 08, 2024 at 01:07:18PM -0500, [email protected] wrote:
From: Chuck Lever <[email protected]>
On NFS mounts, at least, generic/559, 560, and 561 run for a very long time, and usually fail.
The above tests already gate on whether duperemove is installed on the test system, but when fstests is installed as part of an automated workflow designed to handle many filesystem types, duperemove is installed by default.
Quoted 7 lines
duperemove(8) states:
Deduplication is currently only supported by the btrfs and xfs filesystem.
Ensure that the generic dedupe tests are run on only filesystems where duperemove is known to work.
Perhaps we should just remove this test. duperemove seems to be quite flakey unreliable and keeps failing on my XFS test systems (e.g. getting stuck in endless fiemap loops). I've set these tests to be 'unreliable_in_parallel' because a 75% failure rate when the system is under heavy load due to duperemove bugs is unacceptible in a tool being used for a regression test.
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 1 line
+_supported_fs btrfs xfs
This is not how generic tests works. They need a helper in common
to check if a feature is present or not, which then probes for the feature. We should figure out what the issue is here first before doing hacks like that as well.
Quoted 5 lines · Christoph Hellwig
Quoted 1 line
+_supported_fs btrfs xfs
This is not how generic tests works. They need a helper in common
to check if a feature is present or not, which then probes for the feature.
Someone who has knowledge of the interfaces and facilities that duperemove depends on will need to construct such a helper. I can only guess.
Quoted 2 lines
We should figure out what the issue is here first before doing hacks like that as well.
To be clear, there are two issues:
1. Dave's report of unreliability on filesystems where it still makes sense to use duperemove, and
2. How to disable this test on filesystems (like NFS) where duperemove is not supported or where the test is not meaningful. The current check for the presence of the duperemove executable is IMO inadequate.
For now I've expunged these three tests from the NFSD CI workflow.
Signature · 1 line
Chuck Lever
Quoted 3 lines · Chuck Lever
2. How to disable this test on filesystems (like NFS) where duperemove is not supported or where the test is not meaningful. The current check for the presence of the duperemove executable is IMO inadequate.
I've not looked at dupremove, but the name suggested it's de-duplicating and thus needs a working FIDEDUPERANGE ioctl. So the test should check for that using the _require_test_dedupe / _require_scratch_dedupe helpers. generic/{559,560,561} do that, so something either in these helpers is probably broken given that NFS rejects remap_file_range with REMAP_FILE_DEDUP, although a bit of tracing would be useful if there actually is something broken in NFS or the VFS.
Quoted 10 lines · On Sun, Dec 08, 2024 at 01:07:18PM -0500, [email protected] wrote:
From: Chuck Lever <[email protected]>
On NFS mounts, at least, generic/559, 560, and 561 run for a very long time, and usually fail.
The above tests already gate on whether duperemove is installed on the test system, but when fstests is installed as part of an automated workflow designed to handle many filesystem types, duperemove is installed by default.
Quoted 4 lines
duperemove(8) states:
Deduplication is currently only supported by the btrfs and xfs filesystem.
If so, I'm good to limit this test on btrfs and xfs. It might be better to add this comment to "_supported_fs btrfs xfs". Anyway,
(This's a fstests patch, send to [email protected].)
Thanks,
Zorro
Quoted 51 lines
Ensure that the generic dedupe tests are run on only filesystems where duperemove is known to work.
Signed-off-by Chuck Lever <[email protected]>--- tests/generic/559 | 1 + tests/generic/560 | 1 + tests/generic/561 | 1 + 3 files changed, 3 insertions(+)tests/generic/559 +1
-0@@ -13,6 +13,7 @@ _begin_fstest auto stress dedupe . ./common/filter . ./common/reflink +_supported_fs btrfs xfs _require_scratch_duperemove fssize=$((2 * 1024 * 1024 * 1024))tests/generic/560 +1
-0@@ -15,6 +15,7 @@ _begin_fstest auto stress dedupe . ./common/filter . ./common/reflink +_supported_fs btrfs xfs _require_scratch_duperemove _scratch_mkfs > $seqres.full 2>&1tests/generic/561 +1
-0@@ -28,6 +28,7 @@ _cleanup() . ./common/filter . ./common/reflink +_supported_fs btrfs xfs _require_scratch_duperemove _scratch_mkfs > $seqres.full 2>&1Signature · 1 line
2.47.0
Quoted 16 lines · Zorro Lang
Quoted 10 lines · On Sun, Dec 08, 2024 at 01:07:18PM -0500, [email protected] wrote:
From: Chuck Lever <[email protected]>
On NFS mounts, at least, generic/559, 560, and 561 run for a very long time, and usually fail.
The above tests already gate on whether duperemove is installed on the test system, but when fstests is installed as part of an automated workflow designed to handle many filesystem types, duperemove is installed by default.
Quoted 4 lines
duperemove(8) states:
Deduplication is currently only supported by the btrfs and xfs filesystem.
Quoted 6 lines
If so, I'm good to limit this test on btrfs and xfs. It might be better to add this comment to "
_supported_fsbtrfs xfs". Anyway,Reviewed-by Zorro Lang <[email protected]>(This's a fstests patch, send to [email protected].)
Noted.
Christoph requested additional root cause analysis to see why the existing feature check utilities are not blocking these tests on NFS. I plan to look into that and repost if needed.
Quoted 55 lines
Thanks,
ZorroQuoted 51 lines
Ensure that the generic dedupe tests are run on only filesystems where duperemove is known to work.
Signed-off-by Chuck Lever <[email protected]>--- tests/generic/559 | 1 + tests/generic/560 | 1 + tests/generic/561 | 1 + 3 files changed, 3 insertions(+)tests/generic/559 +1
-0@@ -13,6 +13,7 @@ _begin_fstest auto stress dedupe . ./common/filter . ./common/reflink +_supported_fs btrfs xfs _require_scratch_duperemove fssize=$((2 * 1024 * 1024 * 1024))tests/generic/560 +1
-0@@ -15,6 +15,7 @@ _begin_fstest auto stress dedupe . ./common/filter . ./common/reflink +_supported_fs btrfs xfs _require_scratch_duperemove _scratch_mkfs > $seqres.full 2>&1tests/generic/561 +1
-0@@ -28,6 +28,7 @@ _cleanup() . ./common/filter . ./common/reflink +_supported_fs btrfs xfs _require_scratch_duperemove _scratch_mkfs > $seqres.full 2>&1Signature · 1 line
2.47.0
Signature · 1 line
Chuck Lever
Quoted 5 lines · Christoph Hellwig
Quoted 3 lines · Chuck Lever
2. How to disable this test on filesystems (like NFS) where duperemove is not supported or where the test is not meaningful. The current check for the presence of the duperemove executable is IMO inadequate.
Quoted 7 lines
I've not looked at dupremove, but the name suggested it's de-duplicating and thus needs a working FIDEDUPERANGE ioctl. So the test should check for that using the
_require_test_dedupe/_require_scratch_dedupehelpers. generic/{559,560,561} do that, so something either in these helpers is probably broken given that NFS rejectsremap_file_rangewithREMAP_FILE_DEDUP, although a bit of tracing would be useful if there actually is something broken in NFS or the VFS.
It doesn't appear to be an NFS problem; duperemove is not even issuing the ioctl as a toe-test to see if it will work.
I opened https://github.com/markfasheh/duperemove/issues/363
Signature · 1 line
Chuck Lever