Skip to contentSkip to conversation

fstests: disable generic duperemove tests for NFS and others

PATCH In review [email protected] 5 participants 8 messages Dec 8, 2024 – Dec 9, 2024 VFS

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fstests: disable generic duperemove tests for NFS and others

From: Chuck Lever <[email protected]>

On NFS mounts, at least, generic/559, 560, and 561 run for a very long time, and usually fail.

The above tests already gate on whether duperemove is installed on the test system, but when fstests is installed as part of an automated workflow designed to handle many filesystem types, duperemove is installed by default.

duperemove(8) states:

Deduplication is currently only supported by the btrfs and xfs filesystem.

Ensure that the generic dedupe tests are run on only filesystems where duperemove is known to work.

---
 tests/generic/559 | 1 +
 tests/generic/560 | 1 +
 tests/generic/561 | 1 +
 3 files changed, 3 insertions(+)
tests/generic/559 +1 -0
index 28cf2e1a32c2..cf80be92142d 100755
@@ -13,6 +13,7 @@ _begin_fstest auto stress dedupe
 . ./common/filter
 . ./common/reflink
 
+_supported_fs btrfs xfs
 _require_scratch_duperemove
 
 fssize=$((2 * 1024 * 1024 * 1024))
tests/generic/560 +1 -0
index 067d3ec0049e..a94b512efda1 100755
@@ -15,6 +15,7 @@ _begin_fstest auto stress dedupe
 . ./common/filter
 . ./common/reflink
 
+_supported_fs btrfs xfs
 _require_scratch_duperemove
 
 _scratch_mkfs > $seqres.full 2>&1
tests/generic/561 +1 -0
index afe727ac56cb..da5f111c5b23 100755
@@ -28,6 +28,7 @@ _cleanup()
 . ./common/filter
 . ./common/reflink
 
+_supported_fs btrfs xfs
 _require_scratch_duperemove
 
 _scratch_mkfs > $seqres.full 2>&1
Signature · 1 line
2.47.0
#1 lore raw
Quoted 10 lines · On Sun, Dec 08, 2024 at 01:07:18PM -0500, [email protected] wrote:

From: Chuck Lever <[email protected]>

On NFS mounts, at least, generic/559, 560, and 561 run for a very long time, and usually fail.

The above tests already gate on whether duperemove is installed on the test system, but when fstests is installed as part of an automated workflow designed to handle many filesystem types, duperemove is installed by default.

Quoted 7 lines

duperemove(8) states:

Deduplication is currently only supported by the btrfs and xfs filesystem.

Ensure that the generic dedupe tests are run on only filesystems where duperemove is known to work.

Perhaps we should just remove this test. duperemove seems to be quite flakey unreliable and keeps failing on my XFS test systems (e.g. getting stuck in endless fiemap loops). I've set these tests to be 'unreliable_in_parallel' because a 75% failure rate when the system is under heavy load due to duperemove bugs is unacceptible in a tool being used for a regression test.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#2 lore raw
In reply to [email protected]
Quoted 1 line

+_supported_fs btrfs xfs

This is not how generic tests works. They need a helper in common
to check if a feature is present or not, which then probes for the feature. We should figure out what the issue is here first before doing hacks like that as well.

#3 lore raw
Quoted 5 lines · Christoph Hellwig
Quoted 1 line

+_supported_fs btrfs xfs

This is not how generic tests works. They need a helper in common
to check if a feature is present or not, which then probes for the feature.

Someone who has knowledge of the interfaces and facilities that duperemove depends on will need to construct such a helper. I can only guess.

Quoted 2 lines

We should figure out what the issue is here first before doing hacks like that as well.

To be clear, there are two issues:

1. Dave's report of unreliability on filesystems where it still makes sense to use duperemove, and

2. How to disable this test on filesystems (like NFS) where duperemove is not supported or where the test is not meaningful. The current check for the presence of the duperemove executable is IMO inadequate.

For now I've expunged these three tests from the NFSD CI workflow.

Signature · 1 line
Chuck Lever
#4 lore raw
Quoted 3 lines · Chuck Lever

2. How to disable this test on filesystems (like NFS) where duperemove is not supported or where the test is not meaningful. The current check for the presence of the duperemove executable is IMO inadequate.

I've not looked at dupremove, but the name suggested it's de-duplicating and thus needs a working FIDEDUPERANGE ioctl. So the test should check for that using the _require_test_dedupe / _require_scratch_dedupe helpers. generic/{559,560,561} do that, so something either in these helpers is probably broken given that NFS rejects remap_file_range with REMAP_FILE_DEDUP, although a bit of tracing would be useful if there actually is something broken in NFS or the VFS.

Reviewed-by on v1 1/1
#6 lore raw
In reply to [email protected]
Quoted 10 lines · On Sun, Dec 08, 2024 at 01:07:18PM -0500, [email protected] wrote:

From: Chuck Lever <[email protected]>

On NFS mounts, at least, generic/559, 560, and 561 run for a very long time, and usually fail.

The above tests already gate on whether duperemove is installed on the test system, but when fstests is installed as part of an automated workflow designed to handle many filesystem types, duperemove is installed by default.

Quoted 4 lines

duperemove(8) states:

Deduplication is currently only supported by the btrfs and xfs filesystem.

If so, I'm good to limit this test on btrfs and xfs. It might be better to add this comment to "_supported_fs btrfs xfs". Anyway,

(This's a fstests patch, send to [email protected].)

Thanks,
Zorro

Quoted 51 lines

Ensure that the generic dedupe tests are run on only filesystems where duperemove is known to work.

---
 tests/generic/559 | 1 +
 tests/generic/560 | 1 +
 tests/generic/561 | 1 +
 3 files changed, 3 insertions(+)
tests/generic/559 +1 -0
index 28cf2e1a32c2..cf80be92142d 100755
@@ -13,6 +13,7 @@ _begin_fstest auto stress dedupe
 . ./common/filter
 . ./common/reflink
 
+_supported_fs btrfs xfs
 _require_scratch_duperemove
 
 fssize=$((2 * 1024 * 1024 * 1024))
tests/generic/560 +1 -0
index 067d3ec0049e..a94b512efda1 100755
@@ -15,6 +15,7 @@ _begin_fstest auto stress dedupe
 . ./common/filter
 . ./common/reflink
 
+_supported_fs btrfs xfs
 _require_scratch_duperemove
 
 _scratch_mkfs > $seqres.full 2>&1
tests/generic/561 +1 -0
index afe727ac56cb..da5f111c5b23 100755
@@ -28,6 +28,7 @@ _cleanup()
 . ./common/filter
 . ./common/reflink
 
+_supported_fs btrfs xfs
 _require_scratch_duperemove
 
 _scratch_mkfs > $seqres.full 2>&1
Signature · 1 line
2.47.0
#7 lore raw
Quoted 16 lines · Zorro Lang
Quoted 10 lines · On Sun, Dec 08, 2024 at 01:07:18PM -0500, [email protected] wrote:

From: Chuck Lever <[email protected]>

On NFS mounts, at least, generic/559, 560, and 561 run for a very long time, and usually fail.

The above tests already gate on whether duperemove is installed on the test system, but when fstests is installed as part of an automated workflow designed to handle many filesystem types, duperemove is installed by default.

Quoted 4 lines

duperemove(8) states:

Deduplication is currently only supported by the btrfs and xfs filesystem.

Quoted 6 lines

If so, I'm good to limit this test on btrfs and xfs. It might be better to add this comment to "_supported_fs btrfs xfs". Anyway,

(This's a fstests patch, send to [email protected].)

Noted.

Christoph requested additional root cause analysis to see why the existing feature check utilities are not blocking these tests on NFS. I plan to look into that and repost if needed.

Quoted 55 lines

Thanks,
Zorro

Quoted 51 lines

Ensure that the generic dedupe tests are run on only filesystems where duperemove is known to work.

---
  tests/generic/559 | 1 +
  tests/generic/560 | 1 +
  tests/generic/561 | 1 +
  3 files changed, 3 insertions(+)
tests/generic/559 +1 -0
index 28cf2e1a32c2..cf80be92142d 100755
@@ -13,6 +13,7 @@ _begin_fstest auto stress dedupe
  . ./common/filter
  . ./common/reflink
 
+_supported_fs btrfs xfs
  _require_scratch_duperemove
 
  fssize=$((2 * 1024 * 1024 * 1024))
tests/generic/560 +1 -0
index 067d3ec0049e..a94b512efda1 100755
@@ -15,6 +15,7 @@ _begin_fstest auto stress dedupe
  . ./common/filter
  . ./common/reflink
 
+_supported_fs btrfs xfs
  _require_scratch_duperemove
 
  _scratch_mkfs > $seqres.full 2>&1
tests/generic/561 +1 -0
index afe727ac56cb..da5f111c5b23 100755
@@ -28,6 +28,7 @@ _cleanup()
  . ./common/filter
  . ./common/reflink
 
+_supported_fs btrfs xfs
  _require_scratch_duperemove
 
  _scratch_mkfs > $seqres.full 2>&1
Signature · 1 line
2.47.0
Signature · 1 line
Chuck Lever
#5 lore raw
In reply to Christoph Hellwig
Quoted 5 lines · Christoph Hellwig
Quoted 3 lines · Chuck Lever

2. How to disable this test on filesystems (like NFS) where duperemove is not supported or where the test is not meaningful. The current check for the presence of the duperemove executable is IMO inadequate.

Quoted 7 lines

I've not looked at dupremove, but the name suggested it's de-duplicating and thus needs a working FIDEDUPERANGE ioctl. So the test should check for that using the _require_test_dedupe / _require_scratch_dedupe helpers. generic/{559,560,561} do that, so something either in these helpers is probably broken given that NFS rejects remap_file_range with REMAP_FILE_DEDUP, although a bit of tracing would be useful if there actually is something broken in NFS or the VFS.

It doesn't appear to be an NFS problem; duperemove is not even issuing the ioctl as a toe-test to see if it will work.

I opened https://github.com/markfasheh/duperemove/issues/363

Signature · 1 line
Chuck Lever