Skip to contentSkip to conversation

fstests overlay fixes for v2025.05.25

PATCH In review Amir Goldstein 3 participants 33 messages Jun 3, 2025 – Jun 9, 2025

lore
Share & tools
lore
#0 lore raw

v2 cover fstests overlay fixes for v2025.05.25

Zorro,

My first batch of fixes [1] was addressing overlayfs test failures after upgrade of test machine to a newer distro (trixie) with libmount v1.41.

After a closer look, I found some tests that regressed into _notrun, which is harder to notice at first glance.

Tests generic/{633,697,697} were regressed in v2025.03.17. Test generic/699 always had a problem AFAICT, but I only just tested it with custom MOUNT_OPTIONS.

Lastly, following the review discussion between Miklos and Karel Zak, I changed the workaround to the regressions observed after upgrade to newer libmount.

Thanks,
Amir.

Changes since v1 [1]:
- Change workaround from LIBMOUNT_FORCE_MOUNT2 to --options-mode=ignore
- Add two commits to fix tests that were not running by mistake

[1] https://lore.kernel.org/fstests/[email protected]/

Amir Goldstein (6):
overlay: workaround libmount failure to remount,ro
overlay: fix regression in _repair_overlay_scratch_fs generic/604: do not run with overlayfs generic/623: do not run with overlayfs
generic: remove incorrect _require_idmapped_mounts checks generic/699: fix failure with MOUNT_OPTIONS

 common/overlay    |  7 ++++++-
 common/rc         | 10 +++++++++-
 tests/generic/330 |  2 +-
 tests/generic/604 | 11 ++++++-----
 tests/generic/623 |  1 +
 tests/generic/633 |  1 -
 tests/generic/696 |  1 -
 tests/generic/697 |  1 -
 tests/generic/699 | 17 ++++++++++-------
 tests/overlay/035 |  2 +-
 10 files changed, 34 insertions(+), 19 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

PATCH v2 1/6 overlay: workaround libmount failure to remount,ro

libmount >= v1.39 calls several unneeded fsconfig() calls to reconfigure lowerdir/upperdir when user requests only -o remount,ro.

Those calls fail because overlayfs does not allow making any config changes with new mount api, besides MS_RDONLY.

We workaround this problem with --options-mode ignore.

---

Changes since v1 [1]:
- Change workaround from LIBMOUNT_FORCE_MOUNT2 to --options-mode=ignore

[1] https://lore.kernel.org/fstests/[email protected]/

 common/overlay    | 5 ++++-
 tests/overlay/035 | 2 +-
 2 files changed, 5 insertions(+), 2 deletions(-)
common/overlay +4 -1
index 01b6622f..0fad6e70 100644
@@ -127,7 +127,10 @@ _overlay_base_scratch_mount()
 _overlay_scratch_mount()
 {
 	if echo "$*" | grep -q remount; then
-		$MOUNT_PROG $SCRATCH_MNT $*
+		# By default, libmount merges remount options with old mount options.
+		# overlayfs does not support re-configuring the same mount options.
+		# We workaround this problem with --options-mode ignore.
+		$MOUNT_PROG $SCRATCH_MNT --options-mode ignore $*
 		return
 	fi
 
tests/overlay/035 +1 -1
index 0b3257c4..2a4df99a 100755
@@ -42,7 +42,7 @@ mkdir -p $lowerdir1 $lowerdir2 $upperdir $workdir
 # Verify that overlay is mounted read-only and that it cannot be remounted rw.
 _overlay_scratch_mount_opts -o"lowerdir=$lowerdir2:$lowerdir1"
 touch $SCRATCH_MNT/foo 2>&1 | _filter_scratch
-$MOUNT_PROG -o remount,rw $SCRATCH_MNT 2>&1 | _filter_ro_mount
+_scratch_remount rw 2>&1 | _filter_ro_mount
 $UMOUNT_PROG $SCRATCH_MNT
 
 # Make workdir immutable to prevent workdir re-create on mount
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH v2 1/6] overlay: workaround libmount failure to remount,ro

Quoted 13 lines · Amir Goldstein

libmount >= v1.39 calls several unneeded fsconfig() calls to reconfigure lowerdir/upperdir when user requests only -o remount,ro.

Those calls fail because overlayfs does not allow making any config changes with new mount api, besides MS_RDONLY.

We workaround this problem with --options-mode ignore.

Quoted 6 lines

---

Changes since v1 [1]:
- Change workaround from LIBMOUNT_FORCE_MOUNT2 to --options-mode=ignore

[1] https://lore.kernel.org/fstests/[email protected]/

I'm not sure if I understand clearly. Does overlay list are fixing this issue on kernel side, then providing a workaround to fstests to avoid the issue be triggered too?

Thanks,
Zorro

Quoted 37 lines
 common/overlay    | 5 ++++-
 tests/overlay/035 | 2 +-
 2 files changed, 5 insertions(+), 2 deletions(-)
common/overlay +4 -1
index 01b6622f..0fad6e70 100644
@@ -127,7 +127,10 @@ _overlay_base_scratch_mount()
 _overlay_scratch_mount()
 {
 	if echo "$*" | grep -q remount; then
-		$MOUNT_PROG $SCRATCH_MNT $*
+		# By default, libmount merges remount options with old mount options.
+		# overlayfs does not support re-configuring the same mount options.
+		# We workaround this problem with --options-mode ignore.
+		$MOUNT_PROG $SCRATCH_MNT --options-mode ignore $*
 		return
 	fi
 
tests/overlay/035 +1 -1
index 0b3257c4..2a4df99a 100755
@@ -42,7 +42,7 @@ mkdir -p $lowerdir1 $lowerdir2 $upperdir $workdir
 # Verify that overlay is mounted read-only and that it cannot be remounted rw.
 _overlay_scratch_mount_opts -o"lowerdir=$lowerdir2:$lowerdir1"
 touch $SCRATCH_MNT/foo 2>&1 | _filter_scratch
-$MOUNT_PROG -o remount,rw $SCRATCH_MNT 2>&1 | _filter_ro_mount
+_scratch_remount rw 2>&1 | _filter_ro_mount
 $UMOUNT_PROG $SCRATCH_MNT
 
 # Make workdir immutable to prevent workdir re-create on mount
Signature · 1 line
2.34.1
#3 lore raw

Re: [PATCH v2 1/6] overlay: workaround libmount failure to remount,ro

Quoted 17 lines · Zorro Lang
Quoted 15 lines · Amir Goldstein

libmount >= v1.39 calls several unneeded fsconfig() calls to reconfigure lowerdir/upperdir when user requests only -o remount,ro.

Those calls fail because overlayfs does not allow making any config changes with new mount api, besides MS_RDONLY.

We workaround this problem with --options-mode ignore.

---

Quoted 9 lines
Quoted 4 lines

Changes since v1 [1]:
- Change workaround from LIBMOUNT_FORCE_MOUNT2 to --options-mode=ignore

[1] https://lore.kernel.org/fstests/[email protected]/

I'm not sure if I understand clearly. Does overlay list are fixing this issue on kernel side, then providing a workaround to fstests to avoid the issue be triggered too?

Noone agreed to fix it on the kernel side.
At least not yet.

Thanks,
Amir.

#4 lore raw

Re: [PATCH v2 1/6] overlay: workaround libmount failure to remount,ro

Quoted 23 lines · Amir Goldstein
Quoted 22 lines · Zorro Lang
Quoted 19 lines · Amir Goldstein

libmount >= v1.39 calls several unneeded fsconfig() calls to reconfigure lowerdir/upperdir when user requests only -o remount,ro.

Those calls fail because overlayfs does not allow making any config changes with new mount api, besides MS_RDONLY.

We workaround this problem with --options-mode ignore.

---

Changes since v1 [1]:
- Change workaround from LIBMOUNT_FORCE_MOUNT2 to --options-mode=ignore

[1] https://lore.kernel.org/fstests/[email protected]/

Quoted 7 lines
Quoted 4 lines

I'm not sure if I understand clearly. Does overlay list are fixing this issue on kernel side, then providing a workaround to fstests to avoid the issue be triggered too?

Noone agreed to fix it on the kernel side.
At least not yet.

If so, I have two questions:)
1) Will overlay fix it on kernel or mount util side?
2) Do you plan to keep this workaround until the issue be fixed in one day? Then revert this workaround?

Thanks,
Zorro

Quoted 4 lines

Thanks,
Amir.

#5 lore raw

Re: [PATCH v2 1/6] overlay: workaround libmount failure to remount,ro

Quoted 30 lines · Zorro Lang
Quoted 28 lines · Amir Goldstein
Quoted 26 lines · Zorro Lang
Quoted 13 lines · Amir Goldstein

libmount >= v1.39 calls several unneeded fsconfig() calls to reconfigure lowerdir/upperdir when user requests only -o remount,ro.

Those calls fail because overlayfs does not allow making any config changes with new mount api, besides MS_RDONLY.

We workaround this problem with --options-mode ignore.

Quoted 6 lines

---

Changes since v1 [1]:
- Change workaround from LIBMOUNT_FORCE_MOUNT2 to --options-mode=ignore

[1] https://lore.kernel.org/fstests/[email protected]/

I'm not sure if I understand clearly. Does overlay list are fixing this issue on kernel side, then providing a workaround to fstests to avoid the issue be triggered too?

Quoted 5 lines
Quoted 2 lines

Noone agreed to fix it on the kernel side.
At least not yet.

If so, I have two questions:)
1) Will overlay fix it on kernel or mount util side?

This is not known at this time.

Quoted 2 lines

2) Do you plan to keep this workaround until the issue be fixed in one day? Then revert this workaround?

Maybe, but keep in mind that the workaround is simply telling the library what we want to do.

We want to remount overlay ro and nothing else and that is exactly what "--options-mode ignore" tells the library to do.

I could have just as well written a test helper src/remount_rdonly.c and not have to deal with the question of which libmount version the test machine is using.

Note that the tests in question are not intended to test the remount,ro functionality itself, they are intended to test the behavior of fs in some scenarios involving a rdonly mount.

I do not want to lose important test coverage of these scenarios because of regressions in the kernel/libmount API.

We can add a new test that ONLY tests remount,ro and let that test fail on overlayfs to keep us reminded of the real regresion that needs to be fixed, but the "workaround" or as I prefer to call it "using the right tool for the test case" has to stay for those other tests.

Thanks,
Amir.

#6 lore raw

Re: [PATCH v2 1/6] overlay: workaround libmount failure to remount,ro

Quoted 31 lines · Amir Goldstein
Quoted 30 lines · Zorro Lang
Quoted 28 lines · Amir Goldstein
Quoted 26 lines · Zorro Lang
Quoted 13 lines · Amir Goldstein

libmount >= v1.39 calls several unneeded fsconfig() calls to reconfigure lowerdir/upperdir when user requests only -o remount,ro.

Those calls fail because overlayfs does not allow making any config changes with new mount api, besides MS_RDONLY.

We workaround this problem with --options-mode ignore.

Quoted 6 lines

---

Changes since v1 [1]:
- Change workaround from LIBMOUNT_FORCE_MOUNT2 to --options-mode=ignore

[1] https://lore.kernel.org/fstests/[email protected]/

I'm not sure if I understand clearly. Does overlay list are fixing this issue on kernel side, then providing a workaround to fstests to avoid the issue be triggered too?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Noone agreed to fix it on the kernel side.
At least not yet.

If so, I have two questions:)
1) Will overlay fix it on kernel or mount util side?

This is not known at this time.

Oh, I thought it's getting fix :-D

Quoted 18 lines
Quoted 3 lines

2) Do you plan to keep this workaround until the issue be fixed in one day? Then revert this workaround?

Maybe, but keep in mind that the workaround is simply telling the library what we want to do.

We want to remount overlay ro and nothing else and that is exactly what "--options-mode ignore" tells the library to do.

I could have just as well written a test helper src/remount_rdonly.c and not have to deal with the question of which libmount version the test machine is using.

Note that the tests in question are not intended to test the remount,ro functionality itself, they are intended to test the behavior of fs in some scenarios involving a rdonly mount.

Quoted 7 lines

I do not want to lose important test coverage of these scenarios because of regressions in the kernel/libmount API.

We can add a new test that ONLY tests remount,ro and let that test fail on overlayfs to keep us reminded of the real regresion that needs to be fixed, but the "workaround" or as I prefer to call it "using the right tool for the test case" has to stay for those other tests.

OK, I just tried to figure out if "hide this error output on new mount APIs" is what overlay list wants. If overlay list (or vfs) acks this patch, and will track this issue. I'm good to merge this workaround for testing :)

Thanks,
Zorro

Quoted 4 lines

Thanks,
Amir.

#7 lore raw

Re: [PATCH v2 1/6] overlay: workaround libmount failure to remount,ro

Quoted 64 lines · Zorro Lang
Quoted 31 lines · Amir Goldstein
Quoted 30 lines · Zorro Lang
Quoted 28 lines · Amir Goldstein
Quoted 26 lines · Zorro Lang
Quoted 13 lines · Amir Goldstein

libmount >= v1.39 calls several unneeded fsconfig() calls to reconfigure lowerdir/upperdir when user requests only -o remount,ro.

Those calls fail because overlayfs does not allow making any config changes with new mount api, besides MS_RDONLY.

We workaround this problem with --options-mode ignore.

Quoted 6 lines

---

Changes since v1 [1]:
- Change workaround from LIBMOUNT_FORCE_MOUNT2 to --options-mode=ignore

[1] https://lore.kernel.org/fstests/[email protected]/

I'm not sure if I understand clearly. Does overlay list are fixing this issue on kernel side, then providing a workaround to fstests to avoid the issue be triggered too?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Noone agreed to fix it on the kernel side.
At least not yet.

If so, I have two questions:)
1) Will overlay fix it on kernel or mount util side?

This is not known at this time.

Oh, I thought it's getting fix :-D

Quoted 21 lines
Quoted 3 lines

2) Do you plan to keep this workaround until the issue be fixed in one day? Then revert this workaround?

Maybe, but keep in mind that the workaround is simply telling the library what we want to do.

We want to remount overlay ro and nothing else and that is exactly what "--options-mode ignore" tells the library to do.

I could have just as well written a test helper src/remount_rdonly.c and not have to deal with the question of which libmount version the test machine is using.

Note that the tests in question are not intended to test the remount,ro functionality itself, they are intended to test the behavior of fs in some scenarios involving a rdonly mount.

I do not want to lose important test coverage of these scenarios because of regressions in the kernel/libmount API.

Quoted 9 lines
Quoted 4 lines

We can add a new test that ONLY tests remount,ro and let that test fail on overlayfs to keep us reminded of the real regresion that needs to be fixed, but the "workaround" or as I prefer to call it "using the right tool for the test case" has to stay for those other tests.

OK, I just tried to figure out if "hide this error output on new mount APIs" is what overlay list wants. If overlay list (or vfs) acks this patch, and will track this issue. I'm good to merge this workaround for testing :)

This workaround in v2 was suggested by libmount maintainer and approved by overlayfs maintainer:

"> So, you do not need LIBMOUNT_FORCE_MOUNT2= workaround, use

Quoted 1 line

"--options-mode ignore" or source and target ;-)

Yeah, that's definitely a better workaround.

I wouldn't call it a fix, since "mount -oremount,ro /overlay" still doesn't work the way it is supposed to, and the thought of adding code to the kernel to work around the current libmount behavior makes me go bleah."

Thanks,
Amir.

[1] https://lore.kernel.org/linux-unionfs/CAJfpegtJ3SDKmC80B4AfWiC3JmtWdW2+78fRZVtsuhe-wSRPvg@mail.gmail.com/

Reviewed-by
#8 lore raw

Re: [PATCH v2 1/6] overlay: workaround libmount failure to remount,ro

Quoted 79 lines · Amir Goldstein
Quoted 64 lines · Zorro Lang
Quoted 31 lines · Amir Goldstein
Quoted 30 lines · Zorro Lang
Quoted 28 lines · Amir Goldstein
Quoted 26 lines · Zorro Lang
Quoted 13 lines · Amir Goldstein

libmount >= v1.39 calls several unneeded fsconfig() calls to reconfigure lowerdir/upperdir when user requests only -o remount,ro.

Those calls fail because overlayfs does not allow making any config changes with new mount api, besides MS_RDONLY.

We workaround this problem with --options-mode ignore.

Quoted 6 lines

---

Changes since v1 [1]:
- Change workaround from LIBMOUNT_FORCE_MOUNT2 to --options-mode=ignore

[1] https://lore.kernel.org/fstests/[email protected]/

I'm not sure if I understand clearly. Does overlay list are fixing this issue on kernel side, then providing a workaround to fstests to avoid the issue be triggered too?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Noone agreed to fix it on the kernel side.
At least not yet.

If so, I have two questions:)
1) Will overlay fix it on kernel or mount util side?

This is not known at this time.

Oh, I thought it's getting fix :-D

Quoted 21 lines
Quoted 3 lines

2) Do you plan to keep this workaround until the issue be fixed in one day? Then revert this workaround?

Maybe, but keep in mind that the workaround is simply telling the library what we want to do.

We want to remount overlay ro and nothing else and that is exactly what "--options-mode ignore" tells the library to do.

I could have just as well written a test helper src/remount_rdonly.c and not have to deal with the question of which libmount version the test machine is using.

Note that the tests in question are not intended to test the remount,ro functionality itself, they are intended to test the behavior of fs in some scenarios involving a rdonly mount.

I do not want to lose important test coverage of these scenarios because of regressions in the kernel/libmount API.

Quoted 9 lines
Quoted 4 lines

We can add a new test that ONLY tests remount,ro and let that test fail on overlayfs to keep us reminded of the real regresion that needs to be fixed, but the "workaround" or as I prefer to call it "using the right tool for the test case" has to stay for those other tests.

OK, I just tried to figure out if "hide this error output on new mount APIs" is what overlay list wants. If overlay list (or vfs) acks this patch, and will track this issue. I'm good to merge this workaround for testing :)

This workaround in v2 was suggested by libmount maintainer and approved by overlayfs maintainer:

"> So, you do not need LIBMOUNT_FORCE_MOUNT2= workaround, use

Quoted 8 lines
Quoted 1 line

"--options-mode ignore" or source and target ;-)

Yeah, that's definitely a better workaround.

I wouldn't call it a fix, since "mount -oremount,ro /overlay" still doesn't work the way it is supposed to, and the thought of adding code to the kernel to work around the current libmount behavior makes me go bleah."

OK, just to make sure overlay/vfs folks know why these failures gone :)

With this patch, I've merged 5 patches of this patchset. Now only 4/6 still need more reviewing (I've provided my suggestion about that). If patch 4/6 can't catch up the release of this week, don't worry, I'll push these 5 patches at first, feel free to send patch 4/6 singly later :)

Thanks,
Zorro

Quoted 6 lines

Thanks,
Amir.

[1] https://lore.kernel.org/linux-unionfs/CAJfpegtJ3SDKmC80B4AfWiC3JmtWdW2+78fRZVtsuhe-wSRPvg@mail.gmail.com/

#9 lore raw

Re: [PATCH v2 1/6] overlay: workaround libmount failure to remount,ro

In reply to Amir Goldstein

Hi Amir,

Quoted 27 lines · Amir Goldstein
Quoted 26 lines · Zorro Lang
Quoted 25 lines · Amir Goldstein
Quoted 24 lines · Zorro Lang
Quoted 13 lines · Amir Goldstein

libmount >= v1.39 calls several unneeded fsconfig() calls to reconfigure lowerdir/upperdir when user requests only -o remount,ro.

Those calls fail because overlayfs does not allow making any config changes with new mount api, besides MS_RDONLY.

We workaround this problem with --options-mode ignore.

Quoted 6 lines

---

Changes since v1 [1]:
- Change workaround from LIBMOUNT_FORCE_MOUNT2 to --options-mode=ignore

[1] https://lore.kernel.org/fstests/[email protected]/

I'm not sure if I understand clearly. Does overlay list are fixing this issue on kernel side, then providing a workaround to fstests to avoid the issue be triggered too?

Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

Noone agreed to fix it on the kernel side.
At least not yet.

If so, I have two questions:)
1) Will overlay fix it on kernel or mount util side?

This is not known at this time.

Do you know how calling fsconfig() in a "redundant" way behaves in other filesystems? Do they allow to call fsconfig() calls that doesn't change the state, like a noop?

#10 lore raw

Re: [PATCH v2 1/6] overlay: workaround libmount failure to remount,ro

Quoted 33 lines · André Almeida

Hi Amir,

Quoted 29 lines · Amir Goldstein
Quoted 28 lines · Zorro Lang
Quoted 27 lines · Amir Goldstein
Quoted 16 lines · Zorro Lang
Quoted 15 lines · Amir Goldstein

libmount >= v1.39 calls several unneeded fsconfig() calls to reconfigure lowerdir/upperdir when user requests only -o remount,ro.

Those calls fail because overlayfs does not allow making any config changes with new mount api, besides MS_RDONLY.

We workaround this problem with --options-mode ignore.

---

Quoted 8 lines
Quoted 4 lines

Changes since v1 [1]:
- Change workaround from LIBMOUNT_FORCE_MOUNT2 to --options-mode=ignore

[1] https://lore.kernel.org/fstests/[email protected]/

I'm not sure if I understand clearly. Does overlay list are fixing this issue on kernel side, then providing a workaround to fstests to avoid the issue be triggered too?

Noone agreed to fix it on the kernel side.
At least not yet.

Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

If so, I have two questions:)
1) Will overlay fix it on kernel or mount util side?

This is not known at this time.

Do you know how calling fsconfig() in a "redundant" way behaves in other filesystems? Do they allow to call fsconfig() calls that doesn't change the state, like a noop?

I don't know. didn't do a survey.
"We did this until now"
Is not a good enough reason to keep doing the same workarounds in the kernel IMO.

Thanks,
Amir.

#11 lore raw

PATCH v2 2/6 overlay: fix regression in _repair_overlay_scratch_fs

In reply to Amir Goldstein

_repair_overlay_scratch_fs assumed that the base fs is mounted. This was a wrong assumption to make, and that was exposed by commit 4c6bc456 ("fstests: clean up mount and unmount operations") that converted open coded umount in generic/332 to _scratch_unmount.

After this change, there errors were observed when running generic/332 if fsck.overlay is installed:

Check for damage
+fsck.overlay:[Error]: Faile to resolve upperdir:/vdf/ovl-upper:
No such file or directory
+fsck.overlay failed, err=8
+umount: /vdf: not mounted.

Fix this by making sure that base fs is mounted before running the layers check and fix test generic/330 to conform with the umount conversion patch.

Fixes 4c6bc456 ("fstests: clean up mount and unmount operations")
---
 common/overlay    | 2 ++
 tests/generic/330 | 2 +-
 2 files changed, 3 insertions(+), 1 deletion(-)
common/overlay +2 -0
index 0fad6e70..0be943b1 100644
@@ -434,6 +434,8 @@ _check_overlay_scratch_fs()
 
 _repair_overlay_scratch_fs()
 {
+	# Base fs needs to be mounted for overlayfs check
+	_overlay_base_scratch_mount
 	_overlay_fsck_dirs $OVL_BASE_SCRATCH_MNT/$OVL_LOWER \
 		$OVL_BASE_SCRATCH_MNT/$OVL_UPPER \
 		$OVL_BASE_SCRATCH_MNT/$OVL_WORK -y
tests/generic/330 +1 -1
index c67defc4..901b17b1 100755
@@ -61,7 +61,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "Check for damage"
-umount $SCRATCH_MNT
+_scratch_unmount
 _repair_scratch_fs >> $seqres.full
 
 # success, all done
Signature · 1 line
2.34.1
Reviewed-by on v2 2/6
#12 lore raw

Re: [PATCH v2 2/6] overlay: fix regression in _repair_overlay_scratch_fs

Quoted 14 lines · Amir Goldstein

_repair_overlay_scratch_fs assumed that the base fs is mounted. This was a wrong assumption to make, and that was exposed by commit 4c6bc456 ("fstests: clean up mount and unmount operations") that converted open coded umount in generic/332 to _scratch_unmount.

After this change, there errors were observed when running generic/332 if fsck.overlay is installed:

Check for damage
+fsck.overlay:[Error]: Faile to resolve upperdir:/vdf/ovl-upper:
No such file or directory
+fsck.overlay failed, err=8
+umount: /vdf: not mounted.

Quoted 8 lines

Fix this by making sure that base fs is mounted before running the layers check and fix test generic/330 to conform with the umount conversion patch.

Fixes 4c6bc456 ("fstests: clean up mount and unmount operations")

---

Thanks for this regression fix from overlay.

Quoted 33 lines
 common/overlay    | 2 ++
 tests/generic/330 | 2 +-
 2 files changed, 3 insertions(+), 1 deletion(-)
common/overlay +2 -0
index 0fad6e70..0be943b1 100644
@@ -434,6 +434,8 @@ _check_overlay_scratch_fs()
 
 _repair_overlay_scratch_fs()
 {
+	# Base fs needs to be mounted for overlayfs check
+	_overlay_base_scratch_mount
 	_overlay_fsck_dirs $OVL_BASE_SCRATCH_MNT/$OVL_LOWER \
 		$OVL_BASE_SCRATCH_MNT/$OVL_UPPER \
 		$OVL_BASE_SCRATCH_MNT/$OVL_WORK -y
tests/generic/330 +1 -1
index c67defc4..901b17b1 100755
@@ -61,7 +61,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "Check for damage"
-umount $SCRATCH_MNT
+_scratch_unmount
 _repair_scratch_fs >> $seqres.full
 
 # success, all done
Signature · 1 line
2.34.1
#13 lore raw

PATCH v2 3/6 generic/604: do not run with overlayfs

In reply to Amir Goldstein

Overlayfs does not allow mounting over again with the same layers until umount is fully completed, so is not appropriate for this test which tries to mount in parallel to umount.

This is manifested as the test failure below when overlayfs strict mount checks are enabled by enabling the index feature:

$ echo Y > /sys/module/overlay/parameters/index
...
    +mount: /vdf/ovl-mnt: /vdf already mounted or mount point busy.
    +       dmesg(1) may have more information after failed mount system call.
    +mount /vdf /vdf/ovl-mnt failed

Opt-out of this test with overlayfs and remove the hacks that were placed by commit 06cee932 ("generic/604: Fix for overlayfs") to make the test pass with overlayfs in the first place.

---
 tests/generic/604 | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
tests/generic/604 +6 -5
index 744d3456..481250fd 100755
@@ -13,6 +13,9 @@ _begin_fstest auto quick mount
 # Import common functions.
 . ./common/filter
 
+# Overlayfs does not allow mounting over again with the same layers
+# until umount is fully completed, so is not appropriate for this test.
+_exclude_fs overlay
 
 # Modify as appropriate.
 _require_scratch
@@ -22,11 +25,9 @@ _scratch_mount
 for i in $(seq 0 500); do
 	$XFS_IO_PROG -f -c "pwrite 0 4K" $SCRATCH_MNT/$i >/dev/null
 done
-# For overlayfs, avoid unmounting the base fs after _scratch_mount tries to
-# mount the base fs.  Delay the mount attempt by a small amount in the hope
-# that the mount() call will try to lock s_umount /after/ umount has already
-# taken it.
-_unmount $SCRATCH_MNT &
+# Delay the mount attempt by a small amount in the hope that the mount() call
+# will try to lock s_umount /after/ umount has already taken it.
+_scratch_unmount &
 sleep 0.01s ; _scratch_mount
 wait
 
Signature · 1 line
2.34.1
Reviewed-by on v2 3/6
#14 lore raw

Re: [PATCH v2 3/6] generic/604: do not run with overlayfs

Quoted 13 lines · Amir Goldstein

Overlayfs does not allow mounting over again with the same layers until umount is fully completed, so is not appropriate for this test which tries to mount in parallel to umount.

This is manifested as the test failure below when overlayfs strict mount checks are enabled by enabling the index feature:

$ echo Y > /sys/module/overlay/parameters/index
...
    +mount: /vdf/ovl-mnt: /vdf already mounted or mount point busy.
    +       dmesg(1) may have more information after failed mount system call.
    +mount /vdf /vdf/ovl-mnt failed
Quoted 7 lines

Opt-out of this test with overlayfs and remove the hacks that were placed by commit 06cee932 ("generic/604: Fix for overlayfs") to make the test pass with overlayfs in the first place.

---

Good to me,

Quoted 35 lines
 tests/generic/604 | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
tests/generic/604 +6 -5
index 744d3456..481250fd 100755
@@ -13,6 +13,9 @@ _begin_fstest auto quick mount
 # Import common functions.
 . ./common/filter
 
+# Overlayfs does not allow mounting over again with the same layers
+# until umount is fully completed, so is not appropriate for this test.
+_exclude_fs overlay
 
 # Modify as appropriate.
 _require_scratch
@@ -22,11 +25,9 @@ _scratch_mount
 for i in $(seq 0 500); do
 	$XFS_IO_PROG -f -c "pwrite 0 4K" $SCRATCH_MNT/$i >/dev/null
 done
-# For overlayfs, avoid unmounting the base fs after _scratch_mount tries to
-# mount the base fs.  Delay the mount attempt by a small amount in the hope
-# that the mount() call will try to lock s_umount /after/ umount has already
-# taken it.
-_unmount $SCRATCH_MNT &
+# Delay the mount attempt by a small amount in the hope that the mount() call
+# will try to lock s_umount /after/ umount has already taken it.
+_scratch_unmount &
 sleep 0.01s ; _scratch_mount
 wait
 
Signature · 1 line
2.34.1
#15 lore raw

PATCH v2 4/6 generic/623: do not run with overlayfs

In reply to Amir Goldstein

This test performs shutdown via xfs_io -c shutdown.

Overlayfs tests can use _scratch_shutdown, but they cannot use "-c shutdown" xfs_io command without jumping through hoops, so by default we do not support it.

Add this condition to _require_xfs_io_command and add the require statement to test generic/623 so it wont run with overlayfs.

---
 common/rc         | 8 ++++++++
 tests/generic/623 | 1 +
 2 files changed, 9 insertions(+)
common/rc +8 -0
index d8ee8328..bffd576a 100644
@@ -3033,6 +3033,14 @@ _require_xfs_io_command()
 		touch $testfile
 		testio=`$XFS_IO_PROG -c "syncfs" $testfile 2>&1`
 		;;
+	"shutdown")
+		if [ $FSTYP = "overlay" ]; then
+			# Overlayfs tests can use _scratch_shutdown, but they
+			# cannot use "-c shutdown" xfs_io command without jumping
+			# through hoops, so by default we do not support it.
+			_notrun "xfs_io $command not supported on $FSTYP"
+		fi
+		;;
 	*)
 		testio=`$XFS_IO_PROG -c "help $command" 2>&1`
 	esac
tests/generic/623 +1 -0
index b97e2adb..4e36daaf 100755
@@ -16,6 +16,7 @@ _begin_fstest auto quick shutdown mmap
 
 _require_scratch_nocheck
 _require_scratch_shutdown
+_require_xfs_io_command "shutdown"
 
 _scratch_mkfs &>> $seqres.full
 _scratch_mount
Signature · 1 line
2.34.1
#16 lore raw

Re: [PATCH v2 4/6] generic/623: do not run with overlayfs

Quoted 24 lines · Amir Goldstein

This test performs shutdown via xfs_io -c shutdown.

Overlayfs tests can use _scratch_shutdown, but they cannot use "-c shutdown" xfs_io command without jumping through hoops, so by default we do not support it.

Add this condition to _require_xfs_io_command and add the require statement to test generic/623 so it wont run with overlayfs.

---
 common/rc         | 8 ++++++++
 tests/generic/623 | 1 +
 2 files changed, 9 insertions(+)
common/rc +0 -0
index d8ee8328..bffd576a 100644
@@ -3033,6 +3033,14 @@ _require_xfs_io_command()
 		touch $testfile

common/rc _require_xfs_io_command() · line 3043

Quoted 10 lines
common/rc +8 -0
 		testio=`$XFS_IO_PROG -c "syncfs" $testfile 2>&1`
 		;;
+	"shutdown")
+		if [ $FSTYP = "overlay" ]; then
+			# Overlayfs tests can use _scratch_shutdown, but they
+			# cannot use "-c shutdown" xfs_io command without jumping
+			# through hoops, so by default we do not support it.
+			_notrun "xfs_io $command not supported on $FSTYP"
+		fi
+		;;

Hmm... I'm not sure this's a good way.
For example, overlay/087 does xfs_io shutdown too, generally it should calls _require_xfs_io_command "shutdown" although it doesn't. If someone overlay test case hope to test as o/087 does, and it calls _require_xfs_io_command "shutdown", then it'll be _notrun.

If g/623 is not suitable for overlay, how about skip it for overlay clearly, by _exclude_fs overlay ?

Thanks,
Zorro

Quoted 18 lines
common/rc +0 -0
 	*)
 		testio=`$XFS_IO_PROG -c "help $command" 2>&1`
 	esac
tests/generic/623 +1 -0
index b97e2adb..4e36daaf 100755
@@ -16,6 +16,7 @@ _begin_fstest auto quick shutdown mmap
 
 _require_scratch_nocheck
 _require_scratch_shutdown
+_require_xfs_io_command "shutdown"
 
 _scratch_mkfs &>> $seqres.full
 _scratch_mount
Signature · 1 line
2.34.1
#17 lore raw

Re: [PATCH v2 4/6] generic/623: do not run with overlayfs

Quoted 31 lines · Zorro Lang
Quoted 29 lines · Amir Goldstein

This test performs shutdown via xfs_io -c shutdown.

Overlayfs tests can use _scratch_shutdown, but they cannot use "-c shutdown" xfs_io command without jumping through hoops, so by default we do not support it.

Add this condition to _require_xfs_io_command and add the require statement to test generic/623 so it wont run with overlayfs.

---
 common/rc         | 8 ++++++++
 tests/generic/623 | 1 +
 2 files changed, 9 insertions(+)
common/rc +3 -0
index d8ee8328..bffd576a 100644
@@ -3033,6 +3033,14 @@ _require_xfs_io_command()
              touch $testfile
              testio=`$XFS_IO_PROG -c "syncfs" $testfile 2>&1`
              ;;
+     "shutdown")
+             if [ $FSTYP = "overlay" ]; then
+                     # Overlayfs tests can use _scratch_shutdown, but they
Quoted 8 lines

common/rc _require_xfs_io_command() · line 3043

Quoted 5 lines
common/rc +5 -0
+                     # cannot use "-c shutdown" xfs_io command without jumping
+                     # through hoops, so by default we do not support it.
+                     _notrun "xfs_io $command not supported on $FSTYP"
+             fi
+             ;;

Hmm... I'm not sure this's a good way.
For example, overlay/087 does xfs_io shutdown too,

Yes it does but look at the effort needed to do that properly:

$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f

' -c close -c syncfs $SCRATCH_MNT | \
        grep -vF '[00'
Quoted 4 lines

generally it should calls
_require_xfs_io_command "shutdown" although it doesn't. If someone overlay test case hope to test as o/087 does, and it calls _require_xfs_io_command "shutdown", then it'll be _notrun.

If someone knows enough to perform the dance above with _scratch_shutdown_handle then that someone should know enough not to call _require_xfs_io_command "shutdown".
OTOH, if someone doesn't know then default is to not run.

Quoted 4 lines

If g/623 is not suitable for overlay, how about skip it for overlay clearly, by _exclude_fs overlay ?

I do not personally mind doing this _exclude_fs overlay, but it is usually prefered to require what the test needs.

Whatever you prefer is fine by me.

Thanks,
Amir.

#18 lore raw

Re: [PATCH v2 4/6] generic/623: do not run with overlayfs

Quoted 52 lines · Amir Goldstein
Quoted 31 lines · Zorro Lang
Quoted 29 lines · Amir Goldstein

This test performs shutdown via xfs_io -c shutdown.

Overlayfs tests can use _scratch_shutdown, but they cannot use "-c shutdown" xfs_io command without jumping through hoops, so by default we do not support it.

Add this condition to _require_xfs_io_command and add the require statement to test generic/623 so it wont run with overlayfs.

---
 common/rc         | 8 ++++++++
 tests/generic/623 | 1 +
 2 files changed, 9 insertions(+)
common/rc +3 -0
index d8ee8328..bffd576a 100644
@@ -3033,6 +3033,14 @@ _require_xfs_io_command()
              touch $testfile
              testio=`$XFS_IO_PROG -c "syncfs" $testfile 2>&1`
              ;;
+     "shutdown")
+             if [ $FSTYP = "overlay" ]; then
+                     # Overlayfs tests can use _scratch_shutdown, but they
Quoted 8 lines

common/rc _require_xfs_io_command() · line 3043

Quoted 5 lines
common/rc +5 -0
+                     # cannot use "-c shutdown" xfs_io command without jumping
+                     # through hoops, so by default we do not support it.
+                     _notrun "xfs_io $command not supported on $FSTYP"
+             fi
+             ;;

Hmm... I'm not sure this's a good way.
For example, overlay/087 does xfs_io shutdown too,

Yes it does but look at the effort needed to do that properly:

$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f

' -c close -c syncfs $SCRATCH_MNT | \
        grep -vF '[00'
Quoted 4 lines

generally it should calls
_require_xfs_io_command "shutdown" although it doesn't. If someone overlay test case hope to test as o/087 does, and it calls _require_xfs_io_command "shutdown", then it'll be _notrun.

Quoted 4 lines

If someone knows enough to perform the dance above with _scratch_shutdown_handle then that someone should know enough not to call _require_xfs_io_command "shutdown".
OTOH, if someone doesn't know then default is to not run.

Sure, I can understand that, just this logic is a bit *obscure* :) It sounds like:
"If an overlay test case wants to do xfs_io shutdown, it shouldn't call _require_xfs_io_command "shutdown". Or call that to skip a shutdown test on overlay :)"

And the expected result of _require_xfs_io_command "shutdown" will be totally opposite with _require_scratch_shutdown on overlay, that might be confused. Can we have a clearer way to deal with that?

Quoted 10 lines
Quoted 5 lines

If g/623 is not suitable for overlay, how about skip it for overlay clearly, by _exclude_fs overlay ?

I do not personally mind doing this _exclude_fs overlay, but it is usually prefered to require what the test needs.

Whatever you prefer is fine by me.

I don't perfer "_exclude_fs overlay", just before we have a clear way to deal with overlay shutdown, this might be simple and forthright :)

Thanks,
Zorro

Quoted 4 lines

Thanks,
Amir.

#19 lore raw

Re: [PATCH v2 4/6] generic/623: do not run with overlayfs

Quoted 59 lines · Zorro Lang
Quoted 52 lines · Amir Goldstein
Quoted 31 lines · Zorro Lang
Quoted 29 lines · Amir Goldstein

This test performs shutdown via xfs_io -c shutdown.

Overlayfs tests can use _scratch_shutdown, but they cannot use "-c shutdown" xfs_io command without jumping through hoops, so by default we do not support it.

Add this condition to _require_xfs_io_command and add the require statement to test generic/623 so it wont run with overlayfs.

---
 common/rc         | 8 ++++++++
 tests/generic/623 | 1 +
 2 files changed, 9 insertions(+)
common/rc +3 -0
index d8ee8328..bffd576a 100644
@@ -3033,6 +3033,14 @@ _require_xfs_io_command()
              touch $testfile
              testio=`$XFS_IO_PROG -c "syncfs" $testfile 2>&1`
              ;;
+     "shutdown")
+             if [ $FSTYP = "overlay" ]; then
+                     # Overlayfs tests can use _scratch_shutdown, but they
Quoted 8 lines

common/rc _require_xfs_io_command() · line 3043

Quoted 5 lines
common/rc +5 -0
+                     # cannot use "-c shutdown" xfs_io command without jumping
+                     # through hoops, so by default we do not support it.
+                     _notrun "xfs_io $command not supported on $FSTYP"
+             fi
+             ;;

Hmm... I'm not sure this's a good way.
For example, overlay/087 does xfs_io shutdown too,

Yes it does but look at the effort needed to do that properly:

$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f

' -c close -c syncfs $SCRATCH_MNT | \
        grep -vF '[00'
Quoted 4 lines

generally it should calls
_require_xfs_io_command "shutdown" although it doesn't. If someone overlay test case hope to test as o/087 does, and it calls _require_xfs_io_command "shutdown", then it'll be _notrun.

Quoted 4 lines

If someone knows enough to perform the dance above with _scratch_shutdown_handle then that someone should know enough not to call _require_xfs_io_command "shutdown".
OTOH, if someone doesn't know then default is to not run.

Quoted 9 lines

Sure, I can understand that, just this logic is a bit *obscure* :) It sounds like:
"If an overlay test case wants to do xfs_io shutdown, it shouldn't call _require_xfs_io_command "shutdown". Or call that to skip a shutdown test on overlay :)"

And the expected result of _require_xfs_io_command "shutdown" will be totally opposite with _require_scratch_shutdown on overlay, that might be confused. Can we have a clearer way to deal with that?

I don't really understand the confusion.

_require_xfs_io_command "shutdown"

Like any other _require statement
requires support for what this test does -
meaning that a test does xfs_io -c shutdown, just like test generic/623 does

and _require_scratch_shutdown implies that the test does _scratch_shutdown

FSTYP overlay happens to be able to do _scratch_shutdown but not able to do xfs_io -c shutdown $SCRATCH_MNT

The different _require statements simply reflect reality as it is.

We can solve the confused about o/087 not having _require_xfs_io_command "shutdown"
by moving the special hand crafted xfs_io command in o/087 to a helper _scratch_shutdown_and_syncfs to hide those internal implementation details from test writers.
See attached patch.

Thanks,
Amir.

From 3b4d6ff3ce909fa4b385a8d075bbac1f32b82cb8 Mon Sep 17 00:00:00 2001
From: Amir Goldstein <[email protected]>
Date: Fri, 6 Jun 2025 09:11:17 +0200
Subject: [PATCH] fstests: add helper _scratch_shutdown_and_syncfs

Test xfs/546 has to chain syncfs after shutdown and cannot use the _scratch_shitdown helper, because after shutdown a fd cannot be opened to execute syncfs on.

The xfs_io command of chaining syncfs after shutdown is rather more complex to execute in the derived overlayfs test overlay/087.

Add a helper to abstract this complexity from test writers. Add a _require statement to match.

---
 common/rc         | 27 +++++++++++++++++++++++++++
 tests/overlay/087 | 13 +++----------
 tests/xfs/546     |  5 ++---
 3 files changed, 32 insertions(+), 13 deletions(-)
common/rc +27 -0
index 96d65d1c..77683b82 100644
@@ -595,6 +595,27 @@ _scratch_shutdown_handle()
 	fi
 }
 
+_scratch_shutdown_and_syncfs()
+{
+	if [ $FSTYP = "overlay" ]; then
+		# In lagacy overlay usage, it may specify directory as
+		# SCRATCH_DEV, in this case OVL_BASE_SCRATCH_DEV
+		# will be null, so check OVL_BASE_SCRATCH_DEV before
+		# running shutdown to avoid shutting down base fs accidently.
+		if [ -z $OVL_BASE_SCRATCH_DEV ]; then
+			_fail "_scratch_shutdown: call _require_scratch_shutdown first in test"
+		fi
+		# This command is complicated a bit because in the case of overlayfs the
+		# syncfs fd needs to be opened before shutdown and it is different from the
+		# shutdown fd, so we cannot use the _scratch_shutdown() helper.
+		# Filter out xfs_io output of active fds.
+		$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' \
+				-c close -c syncfs $SCRATCH_MNT | grep -vF '[00'
+	else
+		$XFS_IO_PROG -x -c 'shutdown -f ' -c syncfs $SCRATCH_MNT
+	fi
+}
+
 _move_mount()
 {
 	local mnt=$1
@@ -4110,6 +4131,12 @@ _require_scratch_shutdown()
 	_scratch_unmount
 }
 
+_require_scratch_shutdown_and_syncfs()
+{
+	_require_xfs_io_command syncfs
+	_require_scratch_shutdown
+}
+
 _check_s_dax()
 {
 	local target=$1
tests/overlay/087 +3 -10
index a5afb0d5..2ad069db 100755
@@ -32,9 +32,8 @@ _begin_fstest auto quick mount shutdown
 
 
 # Modify as appropriate.
-_require_xfs_io_command syncfs
 _require_scratch_nocheck
-_require_scratch_shutdown
+_require_scratch_shutdown_and_syncfs
 
 [ "$OVL_BASE_FSTYP" == "xfs" ] || \
 	_notrun "base fs $OVL_BASE_FSTYP has unknown behavior with syncfs after shutdown"
@@ -43,19 +42,13 @@ _require_scratch_shutdown
 # bother checking the filesystem afterwards since we never wrote anything.
 echo "=== syncfs after shutdown"
 _scratch_mount
-# This command is complicated a bit because in the case of overlayfs the
-# syncfs fd needs to be opened before shutdown and it is different from the
-# shutdown fd, so we cannot use the _scratch_shutdown() helper.
-# Filter out xfs_io output of active fds.
-$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
-	grep -vF '[00'
+_scratch_shutdown_and_syncfs
 
 # Now repeat the same test with a volatile overlayfs mount and expect no error
 _scratch_unmount
 echo "=== syncfs after shutdown (volatile)"
 _scratch_mount -o volatile
-$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
-	grep -vF '[00'
+_scratch_shutdown_and_syncfs
 
 # success, all done
 status=0
tests/xfs/546 +2 -3
index 316ffc50..c50d41a6 100755
@@ -27,14 +27,13 @@ _begin_fstest auto quick shutdown
 
 
 # Modify as appropriate.
-_require_xfs_io_command syncfs
 _require_scratch_nocheck
-_require_scratch_shutdown
+_require_scratch_shutdown_and_syncfs
 
 # Reuse the fs formatted when we checked for the shutdown ioctl, and don't
 # bother checking the filesystem afterwards since we never wrote anything.
 _scratch_mount
-$XFS_IO_PROG -x -c 'shutdown -f ' -c syncfs $SCRATCH_MNT
+_scratch_shutdown_and_syncfs
 
 # success, all done
 status=0
Signature · 1 line
2.34.1
#20 lore raw

Re: [PATCH v2 4/6] generic/623: do not run with overlayfs

Quoted 84 lines · Amir Goldstein
Quoted 59 lines · Zorro Lang
Quoted 52 lines · Amir Goldstein
Quoted 31 lines · Zorro Lang
Quoted 29 lines · Amir Goldstein

This test performs shutdown via xfs_io -c shutdown.

Overlayfs tests can use _scratch_shutdown, but they cannot use "-c shutdown" xfs_io command without jumping through hoops, so by default we do not support it.

Add this condition to _require_xfs_io_command and add the require statement to test generic/623 so it wont run with overlayfs.

---
 common/rc         | 8 ++++++++
 tests/generic/623 | 1 +
 2 files changed, 9 insertions(+)
common/rc +3 -0
index d8ee8328..bffd576a 100644
@@ -3033,6 +3033,14 @@ _require_xfs_io_command()
              touch $testfile
              testio=`$XFS_IO_PROG -c "syncfs" $testfile 2>&1`
              ;;
+     "shutdown")
+             if [ $FSTYP = "overlay" ]; then
+                     # Overlayfs tests can use _scratch_shutdown, but they
Quoted 8 lines

common/rc _require_xfs_io_command() · line 3043

Quoted 5 lines
common/rc +5 -0
+                     # cannot use "-c shutdown" xfs_io command without jumping
+                     # through hoops, so by default we do not support it.
+                     _notrun "xfs_io $command not supported on $FSTYP"
+             fi
+             ;;

Hmm... I'm not sure this's a good way.
For example, overlay/087 does xfs_io shutdown too,

Yes it does but look at the effort needed to do that properly:

$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f

' -c close -c syncfs $SCRATCH_MNT | \
        grep -vF '[00'
Quoted 4 lines

generally it should calls
_require_xfs_io_command "shutdown" although it doesn't. If someone overlay test case hope to test as o/087 does, and it calls _require_xfs_io_command "shutdown", then it'll be _notrun.

Quoted 4 lines

If someone knows enough to perform the dance above with _scratch_shutdown_handle then that someone should know enough not to call _require_xfs_io_command "shutdown".
OTOH, if someone doesn't know then default is to not run.

Quoted 9 lines

Sure, I can understand that, just this logic is a bit *obscure* :) It sounds like:
"If an overlay test case wants to do xfs_io shutdown, it shouldn't call _require_xfs_io_command "shutdown". Or call that to skip a shutdown test on overlay :)"

And the expected result of _require_xfs_io_command "shutdown" will be totally opposite with _require_scratch_shutdown on overlay, that might be confused. Can we have a clearer way to deal with that?

I don't really understand the confusion.

_require_xfs_io_command "shutdown"

Like any other _require statement
requires support for what this test does -
meaning that a test does xfs_io -c shutdown, just like test generic/623 does

and _require_scratch_shutdown implies that the test does _scratch_shutdown

FSTYP overlay happens to be able to do _scratch_shutdown but not able to do xfs_io -c shutdown $SCRATCH_MNT

Quoted 8 lines

The different _require statements simply reflect reality as it is.

We can solve the confused about o/087 not having _require_xfs_io_command "shutdown"
by moving the special hand crafted xfs_io command in o/087 to a helper _scratch_shutdown_and_syncfs to hide those internal implementation details from test writers.
See attached patch.

Hmm... give me a moment to order my thoughts step by step :)

There're only 2 cases tend to do xfs_io shutdown on overlay currently (others are xfs specific test cases):

  $ grep -rsn shutdown tests/|grep -- "-c"
  tests/generic/623:29:$XFS_IO_PROG -x -c "mmap 0 4k" -c "mwrite 0 4k" -c shutdown -c fsync \
  tests/overlay/087:50:$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
  tests/overlay/087:57:$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
  ...

others shutdown cases nearly all use *_scratch_shutdown* with *_require_scratch_shutdown*, these two functions are consistent in code logic. And no one calls "_require_xfs_io_command shutdown" currently.

So g/623 and o/087 are specifal, actually they call _require_scratch_shutdown too, that makes sense for o/087. Now only g/623 doesn't make sense. Now we need to help it to make sense.

I think the key is in _require_scratch_shutdown function [1], how about add an argument to clearly tell it we need to check shutdown "only on the top layer $SCRATCH_MNT" or "try the lowest layer $BASE_SCRATCH_MNT if there is".

For example:

common/rc +4 -2
index c3af8485c..5f30143e4 100644
@@ -4075,15 +4075,17 @@ _require_exportfs()
        _require_open_by_handle
 }
 
-# Does shutdown work on this fs?
+# Does shutdown work on this [lower|top] layer fs?
 _require_scratch_shutdown()
 {
+       local layer="${1:-lower}"
+
        [ -x $here/src/godown ] || _notrun "src/godown executable not found"
 
        _scratch_mkfs > /dev/null 2>&1 || _notrun "_scratch_mkfs failed on $SCRATCH_DEV"
        _scratch_mount
 
-       if [ $FSTYP = "overlay" ]; then
+       if [ $FSTYP = "overlay" -a "$level" = "lower" ]; then
                if [ -z $OVL_BASE_SCRATCH_DEV ]; then
                        # In lagacy overlay usage, it may specify directory as
                        # SCRATCH_DEV, in this case OVL_BASE_SCRATCH_DEV
tests/generic/623 +1 -1
index b97e2adbe..af0f55397 100755
@@ -15,7 +15,7 @@ _begin_fstest auto quick shutdown mmap
        "xfs: restore shutdown check in mapped write fault path"
 
 _require_scratch_nocheck
-_require_scratch_shutdown
+_require_scratch_shutdown top
 
 _scratch_mkfs &>> $seqres.full
 _scratch_mount

Thanks,
Zorro

[1]
_require_scratch_shutdown()
{
[ -x $here/src/godown ] || _notrun "src/godown executable not found"

_scratch_mkfs > /dev/null 2>&1 || _notrun "_scratch_mkfs failed on $SCRATCH_DEV" _scratch_mount

if [ $FSTYP = "overlay" ]; then
if [ -z $OVL_BASE_SCRATCH_DEV ]; then
# In lagacy overlay usage, it may specify directory as
# SCRATCH_DEV, in this case OVL_BASE_SCRATCH_DEV
# will be null, so check OVL_BASE_SCRATCH_DEV before
# running shutdown to avoid shutting down base fs accidently. _notrun "This test requires a valid $OVL_BASE_SCRATCH_DEV as ovl base fs"
else
$here/src/godown -f $OVL_BASE_SCRATCH_MNT 2>&1 \
|| _notrun "Underlying filesystem does not support shutdown"
fi
else
$here/src/godown -f $SCRATCH_MNT 2>&1 \
|| _notrun "$FSTYP does not support shutdown"
fi

_scratch_unmount
}

Quoted 3 lines

Thanks,
Amir.

#21 lore raw

Re: [PATCH v2 4/6] generic/623: do not run with overlayfs

Quoted 149 lines · Zorro Lang
Quoted 84 lines · Amir Goldstein
Quoted 59 lines · Zorro Lang
Quoted 52 lines · Amir Goldstein
Quoted 31 lines · Zorro Lang
Quoted 29 lines · Amir Goldstein

This test performs shutdown via xfs_io -c shutdown.

Overlayfs tests can use _scratch_shutdown, but they cannot use "-c shutdown" xfs_io command without jumping through hoops, so by default we do not support it.

Add this condition to _require_xfs_io_command and add the require statement to test generic/623 so it wont run with overlayfs.

---
 common/rc         | 8 ++++++++
 tests/generic/623 | 1 +
 2 files changed, 9 insertions(+)
common/rc +3 -0
index d8ee8328..bffd576a 100644
@@ -3033,6 +3033,14 @@ _require_xfs_io_command()
              touch $testfile
              testio=`$XFS_IO_PROG -c "syncfs" $testfile 2>&1`
              ;;
+     "shutdown")
+             if [ $FSTYP = "overlay" ]; then
+                     # Overlayfs tests can use _scratch_shutdown, but they
Quoted 8 lines

common/rc _require_xfs_io_command() · line 3043

Quoted 5 lines
common/rc +5 -0
+                     # cannot use "-c shutdown" xfs_io command without jumping
+                     # through hoops, so by default we do not support it.
+                     _notrun "xfs_io $command not supported on $FSTYP"
+             fi
+             ;;

Hmm... I'm not sure this's a good way.
For example, overlay/087 does xfs_io shutdown too,

Yes it does but look at the effort needed to do that properly:

$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f

' -c close -c syncfs $SCRATCH_MNT | \
        grep -vF '[00'
Quoted 4 lines

generally it should calls
_require_xfs_io_command "shutdown" although it doesn't. If someone overlay test case hope to test as o/087 does, and it calls _require_xfs_io_command "shutdown", then it'll be _notrun.

Quoted 4 lines

If someone knows enough to perform the dance above with _scratch_shutdown_handle then that someone should know enough not to call _require_xfs_io_command "shutdown".
OTOH, if someone doesn't know then default is to not run.

Quoted 9 lines

Sure, I can understand that, just this logic is a bit *obscure* :) It sounds like:
"If an overlay test case wants to do xfs_io shutdown, it shouldn't call _require_xfs_io_command "shutdown". Or call that to skip a shutdown test on overlay :)"

And the expected result of _require_xfs_io_command "shutdown" will be totally opposite with _require_scratch_shutdown on overlay, that might be confused. Can we have a clearer way to deal with that?

I don't really understand the confusion.

_require_xfs_io_command "shutdown"

Like any other _require statement
requires support for what this test does -
meaning that a test does xfs_io -c shutdown, just like test generic/623 does

and _require_scratch_shutdown implies that the test does _scratch_shutdown

FSTYP overlay happens to be able to do _scratch_shutdown but not able to do xfs_io -c shutdown $SCRATCH_MNT

Quoted 8 lines

The different _require statements simply reflect reality as it is.

We can solve the confused about o/087 not having _require_xfs_io_command "shutdown"
by moving the special hand crafted xfs_io command in o/087 to a helper _scratch_shutdown_and_syncfs to hide those internal implementation details from test writers.
See attached patch.

Hmm... give me a moment to order my thoughts step by step :)

There're only 2 cases tend to do xfs_io shutdown on overlay currently (others are xfs specific test cases):

  $ grep -rsn shutdown tests/|grep -- "-c"
  tests/generic/623:29:$XFS_IO_PROG -x -c "mmap 0 4k" -c "mwrite 0 4k" -c shutdown -c fsync \
  tests/overlay/087:50:$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
  tests/overlay/087:57:$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
  ...

others shutdown cases nearly all use *_scratch_shutdown* with *_require_scratch_shutdown*, these two functions are consistent in code logic. And no one calls "_require_xfs_io_command shutdown" currently.

So g/623 and o/087 are specifal, actually they call _require_scratch_shutdown too, that makes sense for o/087. Now only g/623 doesn't make sense. Now we need to help it to make sense.

I think the key is in _require_scratch_shutdown function [1], how about add an argument to clearly tell it we need to check shutdown "only on the top layer $SCRATCH_MNT" or "try the lowest layer $BASE_SCRATCH_MNT if there is".

For example:

common/rc +4 -2
index c3af8485c..5f30143e4 100644
@@ -4075,15 +4075,17 @@ _require_exportfs()
        _require_open_by_handle
 }
 
-# Does shutdown work on this fs?
+# Does shutdown work on this [lower|top] layer fs?
 _require_scratch_shutdown()
 {
+       local layer="${1:-lower}"
+
        [ -x $here/src/godown ] || _notrun "src/godown executable not found"
 
        _scratch_mkfs > /dev/null 2>&1 || _notrun "_scratch_mkfs failed on $SCRATCH_DEV"
        _scratch_mount
 
-       if [ $FSTYP = "overlay" ]; then
+       if [ $FSTYP = "overlay" -a "$level" = "lower" ]; then
                if [ -z $OVL_BASE_SCRATCH_DEV ]; then
                        # In lagacy overlay usage, it may specify directory as
                        # SCRATCH_DEV, in this case OVL_BASE_SCRATCH_DEV
tests/generic/623 +0 -0
index b97e2adbe..af0f55397 100755

tests/generic/623 _begin_fstest auto quick shutdown mmap · line 18

Quoted 5 lines
tests/generic/623 +1 -1
@@ -15,7 +15,7 @@ _begin_fstest auto quick shutdown mmap
        "xfs: restore shutdown check in mapped write fault path"
 
 _require_scratch_nocheck
-_require_scratch_shutdown
+_require_scratch_shutdown top

Sorry I find this utterly confusing.

Think of all the 95% of fstests developers that do not care about overlayfs what does this top mean to them and why should they use it for tests that do xfs_io -c shutdown and not for tests that do _scratch_shutdown?

The test author and reviewers should be able to look at the tests and easily derive what the test requirements should be according to simple rules. For example:

1. A test that calls _scrash_shutdown needs to _require_scratch_shutdown
2. A test that calls _scratch_shutdown_and_syncfs needs to _require_scratch_shutdown_and_syncfs
3. A test that calls xfs_io -c shutdown needs to _require_xfs_io_shutdown

I completely understand why you do not like my hack of _require_xfs_io_command "shutdown"

Would you approve if it was an explicit _require_xfs_io_shutdown helper?

# Requirements for tests that call xfs_io -c shutdown instead of using the # _scratch_shutdown helper
_require_xfs_io_shutdown()
{
if [ $FSTYP = "overlay" ]; then
# Overlayfs tests can use _scratch_shutdown, but they # cannot use "xfs_io -c shutdown" command
without jumping

                        # through hoops, so by default we do not support it.
                        _notrun "xfs_io -c shutdown not supported on $FSTYP"
                fi
                _require_xfs_io_command "shutdown"
                _require_scratch_shutdown
}

Thanks,
Amir

#22 lore raw

Re: [PATCH v2 4/6] generic/623: do not run with overlayfs

Quoted 171 lines · Amir Goldstein
Quoted 149 lines · Zorro Lang
Quoted 84 lines · Amir Goldstein
Quoted 59 lines · Zorro Lang
Quoted 52 lines · Amir Goldstein
Quoted 31 lines · Zorro Lang
Quoted 29 lines · Amir Goldstein

This test performs shutdown via xfs_io -c shutdown.

Overlayfs tests can use _scratch_shutdown, but they cannot use "-c shutdown" xfs_io command without jumping through hoops, so by default we do not support it.

Add this condition to _require_xfs_io_command and add the require statement to test generic/623 so it wont run with overlayfs.

---
 common/rc         | 8 ++++++++
 tests/generic/623 | 1 +
 2 files changed, 9 insertions(+)
common/rc +3 -0
index d8ee8328..bffd576a 100644
@@ -3033,6 +3033,14 @@ _require_xfs_io_command()
              touch $testfile
              testio=`$XFS_IO_PROG -c "syncfs" $testfile 2>&1`
              ;;
+     "shutdown")
+             if [ $FSTYP = "overlay" ]; then
+                     # Overlayfs tests can use _scratch_shutdown, but they
Quoted 8 lines

common/rc _require_xfs_io_command() · line 3043

Quoted 5 lines
common/rc +5 -0
+                     # cannot use "-c shutdown" xfs_io command without jumping
+                     # through hoops, so by default we do not support it.
+                     _notrun "xfs_io $command not supported on $FSTYP"
+             fi
+             ;;

Hmm... I'm not sure this's a good way.
For example, overlay/087 does xfs_io shutdown too,

Yes it does but look at the effort needed to do that properly:

$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f

' -c close -c syncfs $SCRATCH_MNT | \
        grep -vF '[00'
Quoted 4 lines

generally it should calls
_require_xfs_io_command "shutdown" although it doesn't. If someone overlay test case hope to test as o/087 does, and it calls _require_xfs_io_command "shutdown", then it'll be _notrun.

Quoted 4 lines

If someone knows enough to perform the dance above with _scratch_shutdown_handle then that someone should know enough not to call _require_xfs_io_command "shutdown".
OTOH, if someone doesn't know then default is to not run.

Quoted 9 lines

Sure, I can understand that, just this logic is a bit *obscure* :) It sounds like:
"If an overlay test case wants to do xfs_io shutdown, it shouldn't call _require_xfs_io_command "shutdown". Or call that to skip a shutdown test on overlay :)"

And the expected result of _require_xfs_io_command "shutdown" will be totally opposite with _require_scratch_shutdown on overlay, that might be confused. Can we have a clearer way to deal with that?

I don't really understand the confusion.

_require_xfs_io_command "shutdown"

Like any other _require statement
requires support for what this test does -
meaning that a test does xfs_io -c shutdown, just like test generic/623 does

and _require_scratch_shutdown implies that the test does _scratch_shutdown

FSTYP overlay happens to be able to do _scratch_shutdown but not able to do xfs_io -c shutdown $SCRATCH_MNT

Quoted 8 lines

The different _require statements simply reflect reality as it is.

We can solve the confused about o/087 not having _require_xfs_io_command "shutdown"
by moving the special hand crafted xfs_io command in o/087 to a helper _scratch_shutdown_and_syncfs to hide those internal implementation details from test writers.
See attached patch.

Hmm... give me a moment to order my thoughts step by step :)

There're only 2 cases tend to do xfs_io shutdown on overlay currently (others are xfs specific test cases):

  $ grep -rsn shutdown tests/|grep -- "-c"
  tests/generic/623:29:$XFS_IO_PROG -x -c "mmap 0 4k" -c "mwrite 0 4k" -c shutdown -c fsync \
  tests/overlay/087:50:$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
  tests/overlay/087:57:$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
  ...

others shutdown cases nearly all use *_scratch_shutdown* with *_require_scratch_shutdown*, these two functions are consistent in code logic. And no one calls "_require_xfs_io_command shutdown" currently.

So g/623 and o/087 are specifal, actually they call _require_scratch_shutdown too, that makes sense for o/087. Now only g/623 doesn't make sense. Now we need to help it to make sense.

I think the key is in _require_scratch_shutdown function [1], how about add an argument to clearly tell it we need to check shutdown "only on the top layer $SCRATCH_MNT" or "try the lowest layer $BASE_SCRATCH_MNT if there is".

For example:

common/rc +4 -2
index c3af8485c..5f30143e4 100644
@@ -4075,15 +4075,17 @@ _require_exportfs()
        _require_open_by_handle
 }
 
-# Does shutdown work on this fs?
+# Does shutdown work on this [lower|top] layer fs?
 _require_scratch_shutdown()
 {
+       local layer="${1:-lower}"
+
        [ -x $here/src/godown ] || _notrun "src/godown executable not found"
 
        _scratch_mkfs > /dev/null 2>&1 || _notrun "_scratch_mkfs failed on $SCRATCH_DEV"
        _scratch_mount
 
-       if [ $FSTYP = "overlay" ]; then
+       if [ $FSTYP = "overlay" -a "$level" = "lower" ]; then
                if [ -z $OVL_BASE_SCRATCH_DEV ]; then
                        # In lagacy overlay usage, it may specify directory as
                        # SCRATCH_DEV, in this case OVL_BASE_SCRATCH_DEV
tests/generic/623 +0 -0
index b97e2adbe..af0f55397 100755

tests/generic/623 _begin_fstest auto quick shutdown mmap · line 18

Quoted 5 lines
tests/generic/623 +1 -1
@@ -15,7 +15,7 @@ _begin_fstest auto quick shutdown mmap
        "xfs: restore shutdown check in mapped write fault path"
 
 _require_scratch_nocheck
-_require_scratch_shutdown
+_require_scratch_shutdown top

Sorry I find this utterly confusing.

Think of all the 95% of fstests developers that do not care about overlayfs what does this top mean to them and why should they use it for tests that do xfs_io -c shutdown and not for tests that do _scratch_shutdown?

The test author and reviewers should be able to look at the tests and easily derive what the test requirements should be according to simple rules. For example:

1. A test that calls _scrash_shutdown needs to _require_scratch_shutdown
2. A test that calls _scratch_shutdown_and_syncfs needs to _require_scratch_shutdown_and_syncfs
3. A test that calls xfs_io -c shutdown needs to _require_xfs_io_shutdown

Quoted 7 lines

I completely understand why you do not like my hack of _require_xfs_io_command "shutdown"

Would you approve if it was an explicit _require_xfs_io_shutdown helper?

# Requirements for tests that call xfs_io -c shutdown instead of using the # _scratch_shutdown helper

OK, but you might metion that it's better not be used if _scratch_shutdown_handle is called for xfs_io, as we hope the lower layer fs supports shutdown at that time:)

Actually I'm wondering if we should help xfstests to support BASE_FSTYP and FSTYP for more upper layer fs, likes nfs, cifs, and so on. If so, overlay will not be the only one fs who uses BASE_FSTYP and BASE_SCRATCH_DEV things, then we need to differentiate if a feature (e.g. shutdown) is needed by upper layer fs or underlying fs in a case.

...
BTW, a question which isn't belong to this patch:) There're also some failures from those xfstests overlay cases which run unionmount-testsuite (can't rememember all, maybe o/102~109, o/144~117). The error (diff) output are similar as [1]. Is there a fix for that too? Or I missed the fix?

Thanks,
Zorro

[1]

103.out.bad renamed from dev/fd/63 +15 -0
@@ -1,2 +1,17 @@
 QA output created by 103
+mount: /mnt/fstests/SCRATCH_DIR/union/m: wrong fs type, bad option, bad superblock on overlay, missing codepage or helper program, or other error.
+       dmesg(1) may have more information after failed mount system call.
+Traceback (most recent call last):
+  File "/mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.tar.bz2/filesystems/xfstests/unionmount-testsuite/./run", line 362, in <module>
+    func(ctx)
+  File "/mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.tar.bz2/filesystems/xfstests/unionmount-testsuite/tests/rename-file.py", line 71, in subtest_5
+    ctx.rename(f, f2)
+  File "/mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.tar.bz2/filesystems/xfstests/unionmount-testsuite/context.py", line 1254, in rename
+    remount_union(self, rotate_upper=True)
+  File "/mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.tar.bz2/filesystems/xfstests/unionmount-testsuite/remount_union.py", line 35, in remount_union
+    system(cmd)
+  File "/mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.tar.bz2/filesystems/xfstests/unionmount-testsuite/tool_box.py", line 25, in system
+    raise RuntimeError("Command failed: " + command)
+RuntimeError: Command failed: mount -t overlay overlay /mnt/fstests/SCRATCH_DIR/union/m -orw,xino=on -olowerdir=/mnt/fstests/SCRATCH_DIR/union/6/u:/mnt/fstests/SCRATCH_DIR/union/5/u:/mnt/fstests/SCRATCH_DIR/union/4/u:/mnt/fstests/SCRATCH_DIR/union/3/u:/mnt/fstests/SCRATCH_DIR/union/2/u:/mnt/fstests/SCRATCH_DIR/union/1/u:/mnt/fstests/SCRATCH_DIR/union/0/u:/mnt/fstests/SCRATCH_DIR/union/l,upperdir=/mnt/fstests/SCRATCH_DIR/union/7/u,workdir=/mnt/fstests/SCRATCH_DIR/union/7/w
+unionmount testsuite failed! see /var/lib/xfstests/results//overlay/103.full for details.
 Silence is golden
Quoted 19 lines

_require_xfs_io_shutdown()
{
if [ $FSTYP = "overlay" ]; then
# Overlayfs tests can use _scratch_shutdown, but they # cannot use "xfs_io -c shutdown" command
without jumping

                        # through hoops, so by default we do not support it.
                        _notrun "xfs_io -c shutdown not supported on $FSTYP"
                fi
                _require_xfs_io_command "shutdown"
                _require_scratch_shutdown
}

Thanks,
Amir

#23 lore raw

Re: [PATCH v2 4/6] generic/623: do not run with overlayfs

Quoted 176 lines · Zorro Lang
Quoted 174 lines · Amir Goldstein
Quoted 149 lines · Zorro Lang
Quoted 84 lines · Amir Goldstein
Quoted 59 lines · Zorro Lang
Quoted 52 lines · Amir Goldstein
Quoted 31 lines · Zorro Lang
Quoted 29 lines · Amir Goldstein

This test performs shutdown via xfs_io -c shutdown.

Overlayfs tests can use _scratch_shutdown, but they cannot use "-c shutdown" xfs_io command without jumping through hoops, so by default we do not support it.

Add this condition to _require_xfs_io_command and add the require statement to test generic/623 so it wont run with overlayfs.

---
 common/rc         | 8 ++++++++
 tests/generic/623 | 1 +
 2 files changed, 9 insertions(+)
common/rc +3 -0
index d8ee8328..bffd576a 100644
@@ -3033,6 +3033,14 @@ _require_xfs_io_command()
              touch $testfile
              testio=`$XFS_IO_PROG -c "syncfs" $testfile 2>&1`
              ;;
+     "shutdown")
+             if [ $FSTYP = "overlay" ]; then
+                     # Overlayfs tests can use _scratch_shutdown, but they
Quoted 8 lines

common/rc _require_xfs_io_command() · line 3043

Quoted 5 lines
common/rc +5 -0
+                     # cannot use "-c shutdown" xfs_io command without jumping
+                     # through hoops, so by default we do not support it.
+                     _notrun "xfs_io $command not supported on $FSTYP"
+             fi
+             ;;

Hmm... I'm not sure this's a good way.
For example, overlay/087 does xfs_io shutdown too,

Yes it does but look at the effort needed to do that properly:

$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f

' -c close -c syncfs $SCRATCH_MNT | \
        grep -vF '[00'
Quoted 4 lines

generally it should calls
_require_xfs_io_command "shutdown" although it doesn't. If someone overlay test case hope to test as o/087 does, and it calls _require_xfs_io_command "shutdown", then it'll be _notrun.

Quoted 4 lines

If someone knows enough to perform the dance above with _scratch_shutdown_handle then that someone should know enough not to call _require_xfs_io_command "shutdown".
OTOH, if someone doesn't know then default is to not run.

Quoted 9 lines

Sure, I can understand that, just this logic is a bit *obscure* :) It sounds like:
"If an overlay test case wants to do xfs_io shutdown, it shouldn't call _require_xfs_io_command "shutdown". Or call that to skip a shutdown test on overlay :)"

And the expected result of _require_xfs_io_command "shutdown" will be totally opposite with _require_scratch_shutdown on overlay, that might be confused. Can we have a clearer way to deal with that?

I don't really understand the confusion.

_require_xfs_io_command "shutdown"

Like any other _require statement
requires support for what this test does -
meaning that a test does xfs_io -c shutdown, just like test generic/623 does

and _require_scratch_shutdown implies that the test does _scratch_shutdown

FSTYP overlay happens to be able to do _scratch_shutdown but not able to do xfs_io -c shutdown $SCRATCH_MNT

Quoted 8 lines

The different _require statements simply reflect reality as it is.

We can solve the confused about o/087 not having _require_xfs_io_command "shutdown"
by moving the special hand crafted xfs_io command in o/087 to a helper _scratch_shutdown_and_syncfs to hide those internal implementation details from test writers.
See attached patch.

Hmm... give me a moment to order my thoughts step by step :)

There're only 2 cases tend to do xfs_io shutdown on overlay currently (others are xfs specific test cases):

  $ grep -rsn shutdown tests/|grep -- "-c"
  tests/generic/623:29:$XFS_IO_PROG -x -c "mmap 0 4k" -c "mwrite 0 4k" -c shutdown -c fsync \
  tests/overlay/087:50:$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
  tests/overlay/087:57:$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
  ...

others shutdown cases nearly all use *_scratch_shutdown* with *_require_scratch_shutdown*, these two functions are consistent in code logic. And no one calls "_require_xfs_io_command shutdown" currently.

So g/623 and o/087 are specifal, actually they call _require_scratch_shutdown too, that makes sense for o/087. Now only g/623 doesn't make sense. Now we need to help it to make sense.

I think the key is in _require_scratch_shutdown function [1], how about add an argument to clearly tell it we need to check shutdown "only on the top layer $SCRATCH_MNT" or "try the lowest layer $BASE_SCRATCH_MNT if there is".

For example:

common/rc +4 -2
index c3af8485c..5f30143e4 100644
@@ -4075,15 +4075,17 @@ _require_exportfs()
        _require_open_by_handle
 }
 
-# Does shutdown work on this fs?
+# Does shutdown work on this [lower|top] layer fs?
 _require_scratch_shutdown()
 {
+       local layer="${1:-lower}"
+
        [ -x $here/src/godown ] || _notrun "src/godown executable not found"
 
        _scratch_mkfs > /dev/null 2>&1 || _notrun "_scratch_mkfs failed on $SCRATCH_DEV"
        _scratch_mount
 
-       if [ $FSTYP = "overlay" ]; then
+       if [ $FSTYP = "overlay" -a "$level" = "lower" ]; then
                if [ -z $OVL_BASE_SCRATCH_DEV ]; then
                        # In lagacy overlay usage, it may specify directory as
                        # SCRATCH_DEV, in this case OVL_BASE_SCRATCH_DEV
tests/generic/623 +0 -0
index b97e2adbe..af0f55397 100755

tests/generic/623 _begin_fstest auto quick shutdown mmap · line 18

Quoted 5 lines
tests/generic/623 +1 -1
@@ -15,7 +15,7 @@ _begin_fstest auto quick shutdown mmap
        "xfs: restore shutdown check in mapped write fault path"
 
 _require_scratch_nocheck
-_require_scratch_shutdown
+_require_scratch_shutdown top

Sorry I find this utterly confusing.

Think of all the 95% of fstests developers that do not care about overlayfs what does this top mean to them and why should they use it for tests that do xfs_io -c shutdown and not for tests that do _scratch_shutdown?

The test author and reviewers should be able to look at the tests and easily derive what the test requirements should be according to simple rules. For example:

1. A test that calls _scrash_shutdown needs to _require_scratch_shutdown
2. A test that calls _scratch_shutdown_and_syncfs needs to _require_scratch_shutdown_and_syncfs
3. A test that calls xfs_io -c shutdown needs to _require_xfs_io_shutdown

I completely understand why you do not like my hack of _require_xfs_io_command "shutdown"

Quoted 8 lines
Quoted 4 lines

Would you approve if it was an explicit _require_xfs_io_shutdown helper?

# Requirements for tests that call xfs_io -c shutdown instead of using the # _scratch_shutdown helper

OK, but you might metion that it's better not be used if _scratch_shutdown_handle is called for xfs_io, as we hope the lower layer fs supports shutdown at that time:)

Sure I'll add that

Quoted 2 lines

Actually I'm wondering if we should help xfstests to support BASE_FSTYP and FSTYP for more upper layer fs, likes nfs, cifs, and so on.

I think that could be very useful, but will require cifs/nfs to implement more complicated _mount/umount/remount helpers like overlay.

Quoted 5 lines

If so, overlay will not be
the only one fs who uses BASE_FSTYP and BASE_SCRATCH_DEV things, then we need to differentiate if a feature (e.g. shutdown) is needed by upper layer fs or underlying fs in a case.

First of all, terminology. Many get this wrong.
In overlayfs, the "upper" and "lower" refer to the different underlying layers. In fstests BASE_SCRATCH_DEV is always the same for both OVL_UPPER and OVL_LOWER layers.

Referring to the BASE_SCRATCH_DEV as "underlying" or "base" fs is unambiguous in all cases of overlay/nfs/cifs.

I do not have a good terminology to offer for referring to the "fs under test" be it overlay/cifs/nfs. You are welcome to offer your suggestions.

Quoted 5 lines

...
BTW, a question which isn't belong to this patch:) There're also some failures from those xfstests overlay cases which run unionmount-testsuite (can't rememember all, maybe o/102~109, o/144~117). The error (diff) output are similar as [1]. Is there a fix for that too? Or I missed the fix?

I do not get these errors.
I suppose you are using the latest unionmount-testsuite, although it hasn't been changed for a while anyway.

103.out.bad 2025-06-07 05:16:59.445716549 -0400 line 3

Quoted 11 lines

Thanks,
Zorro

[1]

103.out.bad 2025-06-07 05:16:59.445716549 -0400 renamed from dev/fd/63 2025-06-07 05:16:59.489929312 -0400 +2 -0
@@ -1,2 +1,17 @@
 QA output created by 103
+mount: /mnt/fstests/SCRATCH_DIR/union/m: wrong fs type, bad option, bad superblock on overlay, missing codepage or helper program, or other error.
+       dmesg(1) may have more information after failed mount system call.

Reason for failure to mount is likely to be found in dmesg. Please try to see if it is there.

103.out.bad 2025-06-07 05:16:59.445716549 -0400 line 15

Quoted 12 lines
103.out.bad 2025-06-07 05:16:59.445716549 -0400 renamed from dev/fd/63 2025-06-07 05:16:59.489929312 -0400 +12 -0
+Traceback (most recent call last):
+  File "/mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.tar.bz2/filesystems/xfstests/unionmount-testsuite/./run", line 362, in <module>
+    func(ctx)
+  File "/mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.tar.bz2/filesystems/xfstests/unionmount-testsuite/tests/rename-file.py", line 71, in subtest_5
+    ctx.rename(f, f2)
+  File "/mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.tar.bz2/filesystems/xfstests/unionmount-testsuite/context.py", line 1254, in rename
+    remount_union(self, rotate_upper=True)
+  File "/mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.tar.bz2/filesystems/xfstests/unionmount-testsuite/remount_union.py", line 35, in remount_union
+    system(cmd)
+  File "/mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.tar.bz2/filesystems/xfstests/unionmount-testsuite/tool_box.py", line 25, in system
+    raise RuntimeError("Command failed: " + command)
+RuntimeError: Command failed: mount -t overlay overlay /mnt/fstests/SCRATCH_DIR/union/m -orw,xino=on -olowerdir=/mnt/fstests/SCRATCH_DIR/union/6/u:/mnt/fstests/SCRATCH_DIR/union/5/u:/mnt/fstests/SCRATCH_DIR/union/4/u:/mnt/fstests/SCRATCH_DIR/union/3/u:/mnt/fstests/SCRATCH_DIR/union/2/u:/mnt/fstests/SCRATCH_DIR/union/1/u:/mnt/fstests/SCRATCH_DIR/union/0/u:/mnt/fstests/SCRATCH_DIR/union/l,upperdir=/mnt/fstests/SCRATCH_DIR/union/7/u,workdir=/mnt/fstests/SCRATCH_DIR/union/7/w

Nothing looks obviously wrong in the mount command above. Was this a regression for you?
with kernel upgrade? libmount upgrade?

Thanks,
Amir.

#24 lore raw

Re: [PATCH v2 4/6] generic/623: do not run with overlayfs

Quoted 17 lines
Quoted 2 lines

Actually I'm wondering if we should help xfstests to support BASE_FSTYP and FSTYP for more upper layer fs, likes nfs, cifs, and so on.

I think that could be very useful, but will require cifs/nfs to implement more complicated _mount/umount/remount helpers like overlay.

Quoted 5 lines

If so, overlay will not be
the only one fs who uses BASE_FSTYP and BASE_SCRATCH_DEV things, then we need to differentiate if a feature (e.g. shutdown) is needed by upper layer fs or underlying fs in a case.

First of all, terminology. Many get this wrong.
In overlayfs, the "upper" and "lower" refer to the different underlying layers. In fstests BASE_SCRATCH_DEV is always the same for both OVL_UPPER and OVL_LOWER layers.

Quoted 6 lines

Referring to the BASE_SCRATCH_DEV as "underlying" or "base" fs is unambiguous in all cases of overlay/nfs/cifs.

I do not have a good terminology to offer for referring to the "fs under test" be it overlay/cifs/nfs. You are welcome to offer your suggestions.

Sorry, I forgot to answer the question.

So far, with overlayfs there was no need for anything other than _require_scratch_shutdown and _scratch_shutdown and _scratch_remount for generic tests that do shutdown and remount to test consistency after a crash.

overlay tests that are aware of the underlying layers and perform operations explicitly on underlying layers are not generic tests they are overlay/* tests, so the generic helpers are not for them.

Bottom line is that I do not think that _require_scratch_shutdown should refer to fs under test or base fs, until I see a generic test case that suggests otherwise.

What we could do is a re-factoring:

_require_shutdown()
{
local dev=$1
local mnt=$2
...
}

_require_scratch_shutdown()
{
_require_shutdown $SCRATCH_DEV $SCRATCH_MNT
}

Then overlay,cifs,nfs/* tests can call _require_shutdown on base fs if they want to, but first, let's see a test case that needs it.

Thanks,
Amir.

#25 lore raw

Re: [PATCH v2 4/6] generic/623: do not run with overlayfs

Quoted 48 lines · Amir Goldstein
Quoted 17 lines
Quoted 2 lines

Actually I'm wondering if we should help xfstests to support BASE_FSTYP and FSTYP for more upper layer fs, likes nfs, cifs, and so on.

I think that could be very useful, but will require cifs/nfs to implement more complicated _mount/umount/remount helpers like overlay.

Quoted 5 lines

If so, overlay will not be
the only one fs who uses BASE_FSTYP and BASE_SCRATCH_DEV things, then we need to differentiate if a feature (e.g. shutdown) is needed by upper layer fs or underlying fs in a case.

First of all, terminology. Many get this wrong.
In overlayfs, the "upper" and "lower" refer to the different underlying layers. In fstests BASE_SCRATCH_DEV is always the same for both OVL_UPPER and OVL_LOWER layers.

Quoted 6 lines

Referring to the BASE_SCRATCH_DEV as "underlying" or "base" fs is unambiguous in all cases of overlay/nfs/cifs.

I do not have a good terminology to offer for referring to the "fs under test" be it overlay/cifs/nfs. You are welcome to offer your suggestions.

Sorry, I forgot to answer the question.

So far, with overlayfs there was no need for anything other than _require_scratch_shutdown and _scratch_shutdown and _scratch_remount for generic tests that do shutdown and remount to test consistency after a crash.

overlay tests that are aware of the underlying layers and perform operations explicitly on underlying layers are not generic tests they are overlay/* tests, so the generic helpers are not for them.

Bottom line is that I do not think that _require_scratch_shutdown should refer to fs under test or base fs, until I see a generic test case that suggests otherwise.

What we could do is a re-factoring:

_require_shutdown()
{
local dev=$1
local mnt=$2
...
}

Quoted 7 lines

_require_scratch_shutdown()
{
_require_shutdown $SCRATCH_DEV $SCRATCH_MNT
}

Then overlay,cifs,nfs/* tests can call _require_shutdown on base fs if they want to, but first, let's see a test case that needs it.

Actually I thought about the _require_shutdown when I reviewed this patch. Something likes:

g/623: _require_shutdown $SCRATCH_MNT ...
o/087: _require_shutdown $BASE_SCRATCH_MNT ...

But I dropped this idea for two reasons:
1) _require_shutdown needs to do real "shutdown" on a directory, a flexible argument to be "shutdown" is dangerous. If someone would like to check "shutdown" is supported on a $mnt, but if some exceptions cause the $mnt isn't mounted the expected fs, then the "/" might be down.
2) shutdown checking need to shutdown the $mnt and remount it. It's not only about $mnt and $dev, but also about mount options. _require_scratch_shutdown can use _scratch_mount, but how to deal with a flexible $mnt?

The 1) problem might can be fixed by checking the $mnt is a mountpoint, or return. The 2) problem is difficult to me, except we let _require_shutdown accept the mount options list as the 3rd argument. I think that looks ugly for a _require_ function :)

If you have good idea, it can replace this patch 4/6 :)

Thanks,
Zorro

Quoted 4 lines

Thanks,
Amir.

#26 lore raw

Re: [PATCH v2 4/6] generic/623: do not run with overlayfs

Quoted 73 lines · Zorro Lang
Quoted 48 lines · Amir Goldstein
Quoted 17 lines
Quoted 2 lines

Actually I'm wondering if we should help xfstests to support BASE_FSTYP and FSTYP for more upper layer fs, likes nfs, cifs, and so on.

I think that could be very useful, but will require cifs/nfs to implement more complicated _mount/umount/remount helpers like overlay.

Quoted 5 lines

If so, overlay will not be
the only one fs who uses BASE_FSTYP and BASE_SCRATCH_DEV things, then we need to differentiate if a feature (e.g. shutdown) is needed by upper layer fs or underlying fs in a case.

First of all, terminology. Many get this wrong.
In overlayfs, the "upper" and "lower" refer to the different underlying layers. In fstests BASE_SCRATCH_DEV is always the same for both OVL_UPPER and OVL_LOWER layers.

Quoted 6 lines

Referring to the BASE_SCRATCH_DEV as "underlying" or "base" fs is unambiguous in all cases of overlay/nfs/cifs.

I do not have a good terminology to offer for referring to the "fs under test" be it overlay/cifs/nfs. You are welcome to offer your suggestions.

Sorry, I forgot to answer the question.

So far, with overlayfs there was no need for anything other than _require_scratch_shutdown and _scratch_shutdown and _scratch_remount for generic tests that do shutdown and remount to test consistency after a crash.

overlay tests that are aware of the underlying layers and perform operations explicitly on underlying layers are not generic tests they are overlay/* tests, so the generic helpers are not for them.

Bottom line is that I do not think that _require_scratch_shutdown should refer to fs under test or base fs, until I see a generic test case that suggests otherwise.

What we could do is a re-factoring:

_require_shutdown()
{
local dev=$1
local mnt=$2
...
}

Quoted 7 lines

_require_scratch_shutdown()
{
_require_shutdown $SCRATCH_DEV $SCRATCH_MNT
}

Then overlay,cifs,nfs/* tests can call _require_shutdown on base fs if they want to, but first, let's see a test case that needs it.

Actually I thought about the _require_shutdown when I reviewed this patch. Something likes:

g/623: _require_shutdown $SCRATCH_MNT ...
o/087: _require_shutdown $BASE_SCRATCH_MNT ...

But I dropped this idea for two reasons:
1) _require_shutdown needs to do real "shutdown" on a directory, a flexible argument to be "shutdown" is dangerous. If someone would like to check "shutdown" is supported on a $mnt, but if some exceptions cause the $mnt isn't mounted the expected fs, then the "/" might be down.
2) shutdown checking need to shutdown the $mnt and remount it. It's not only about $mnt and $dev, but also about mount options. _require_scratch_shutdown can use _scratch_mount, but how to deal with a flexible $mnt?

Quoted 4 lines

The 1) problem might can be fixed by checking the $mnt is a mountpoint, or return. The 2) problem is difficult to me, except we let _require_shutdown accept the mount options list as the 3rd argument. I think that looks ugly for a _require_ function :)

I don't like it either

Quoted 2 lines

If you have good idea, it can replace this patch 4/6 :)

Already sent a replacement patch per our earlier discussion
https://lore.kernel.org/fstests/[email protected]/

Thanks,
Amir.

#27 lore raw

Re: [PATCH v2 4/6] generic/623: do not run with overlayfs

In reply to Amir Goldstein
Quoted 236 lines · Amir Goldstein
Quoted 176 lines · Zorro Lang
Quoted 174 lines · Amir Goldstein
Quoted 149 lines · Zorro Lang
Quoted 84 lines · Amir Goldstein
Quoted 59 lines · Zorro Lang
Quoted 52 lines · Amir Goldstein
Quoted 31 lines · Zorro Lang
Quoted 29 lines · Amir Goldstein

This test performs shutdown via xfs_io -c shutdown.

Overlayfs tests can use _scratch_shutdown, but they cannot use "-c shutdown" xfs_io command without jumping through hoops, so by default we do not support it.

Add this condition to _require_xfs_io_command and add the require statement to test generic/623 so it wont run with overlayfs.

---
 common/rc         | 8 ++++++++
 tests/generic/623 | 1 +
 2 files changed, 9 insertions(+)
common/rc +3 -0
index d8ee8328..bffd576a 100644
@@ -3033,6 +3033,14 @@ _require_xfs_io_command()
              touch $testfile
              testio=`$XFS_IO_PROG -c "syncfs" $testfile 2>&1`
              ;;
+     "shutdown")
+             if [ $FSTYP = "overlay" ]; then
+                     # Overlayfs tests can use _scratch_shutdown, but they
Quoted 8 lines

common/rc _require_xfs_io_command() · line 3043

Quoted 5 lines
common/rc +5 -0
+                     # cannot use "-c shutdown" xfs_io command without jumping
+                     # through hoops, so by default we do not support it.
+                     _notrun "xfs_io $command not supported on $FSTYP"
+             fi
+             ;;

Hmm... I'm not sure this's a good way.
For example, overlay/087 does xfs_io shutdown too,

Yes it does but look at the effort needed to do that properly:

$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f

' -c close -c syncfs $SCRATCH_MNT | \
        grep -vF '[00'
Quoted 4 lines

generally it should calls
_require_xfs_io_command "shutdown" although it doesn't. If someone overlay test case hope to test as o/087 does, and it calls _require_xfs_io_command "shutdown", then it'll be _notrun.

Quoted 4 lines

If someone knows enough to perform the dance above with _scratch_shutdown_handle then that someone should know enough not to call _require_xfs_io_command "shutdown".
OTOH, if someone doesn't know then default is to not run.

Quoted 9 lines

Sure, I can understand that, just this logic is a bit *obscure* :) It sounds like:
"If an overlay test case wants to do xfs_io shutdown, it shouldn't call _require_xfs_io_command "shutdown". Or call that to skip a shutdown test on overlay :)"

And the expected result of _require_xfs_io_command "shutdown" will be totally opposite with _require_scratch_shutdown on overlay, that might be confused. Can we have a clearer way to deal with that?

I don't really understand the confusion.

_require_xfs_io_command "shutdown"

Like any other _require statement
requires support for what this test does -
meaning that a test does xfs_io -c shutdown, just like test generic/623 does

and _require_scratch_shutdown implies that the test does _scratch_shutdown

FSTYP overlay happens to be able to do _scratch_shutdown but not able to do xfs_io -c shutdown $SCRATCH_MNT

Quoted 8 lines

The different _require statements simply reflect reality as it is.

We can solve the confused about o/087 not having _require_xfs_io_command "shutdown"
by moving the special hand crafted xfs_io command in o/087 to a helper _scratch_shutdown_and_syncfs to hide those internal implementation details from test writers.
See attached patch.

Hmm... give me a moment to order my thoughts step by step :)

There're only 2 cases tend to do xfs_io shutdown on overlay currently (others are xfs specific test cases):

  $ grep -rsn shutdown tests/|grep -- "-c"
  tests/generic/623:29:$XFS_IO_PROG -x -c "mmap 0 4k" -c "mwrite 0 4k" -c shutdown -c fsync \
  tests/overlay/087:50:$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
  tests/overlay/087:57:$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
  ...

others shutdown cases nearly all use *_scratch_shutdown* with *_require_scratch_shutdown*, these two functions are consistent in code logic. And no one calls "_require_xfs_io_command shutdown" currently.

So g/623 and o/087 are specifal, actually they call _require_scratch_shutdown too, that makes sense for o/087. Now only g/623 doesn't make sense. Now we need to help it to make sense.

I think the key is in _require_scratch_shutdown function [1], how about add an argument to clearly tell it we need to check shutdown "only on the top layer $SCRATCH_MNT" or "try the lowest layer $BASE_SCRATCH_MNT if there is".

For example:

common/rc +4 -2
index c3af8485c..5f30143e4 100644
@@ -4075,15 +4075,17 @@ _require_exportfs()
        _require_open_by_handle
 }
 
-# Does shutdown work on this fs?
+# Does shutdown work on this [lower|top] layer fs?
 _require_scratch_shutdown()
 {
+       local layer="${1:-lower}"
+
        [ -x $here/src/godown ] || _notrun "src/godown executable not found"
 
        _scratch_mkfs > /dev/null 2>&1 || _notrun "_scratch_mkfs failed on $SCRATCH_DEV"
        _scratch_mount
 
-       if [ $FSTYP = "overlay" ]; then
+       if [ $FSTYP = "overlay" -a "$level" = "lower" ]; then
                if [ -z $OVL_BASE_SCRATCH_DEV ]; then
                        # In lagacy overlay usage, it may specify directory as
                        # SCRATCH_DEV, in this case OVL_BASE_SCRATCH_DEV
tests/generic/623 +0 -0
index b97e2adbe..af0f55397 100755

tests/generic/623 _begin_fstest auto quick shutdown mmap · line 18

Quoted 5 lines
tests/generic/623 +1 -1
@@ -15,7 +15,7 @@ _begin_fstest auto quick shutdown mmap
        "xfs: restore shutdown check in mapped write fault path"
 
 _require_scratch_nocheck
-_require_scratch_shutdown
+_require_scratch_shutdown top

Sorry I find this utterly confusing.

Think of all the 95% of fstests developers that do not care about overlayfs what does this top mean to them and why should they use it for tests that do xfs_io -c shutdown and not for tests that do _scratch_shutdown?

The test author and reviewers should be able to look at the tests and easily derive what the test requirements should be according to simple rules. For example:

1. A test that calls _scrash_shutdown needs to _require_scratch_shutdown
2. A test that calls _scratch_shutdown_and_syncfs needs to _require_scratch_shutdown_and_syncfs
3. A test that calls xfs_io -c shutdown needs to _require_xfs_io_shutdown

I completely understand why you do not like my hack of _require_xfs_io_command "shutdown"

Quoted 8 lines
Quoted 4 lines

Would you approve if it was an explicit _require_xfs_io_shutdown helper?

# Requirements for tests that call xfs_io -c shutdown instead of using the # _scratch_shutdown helper

OK, but you might metion that it's better not be used if _scratch_shutdown_handle is called for xfs_io, as we hope the lower layer fs supports shutdown at that time:)

Sure I'll add that

Quoted 2 lines

Actually I'm wondering if we should help xfstests to support BASE_FSTYP and FSTYP for more upper layer fs, likes nfs, cifs, and so on.

I think that could be very useful, but will require cifs/nfs to implement more complicated _mount/umount/remount helpers like overlay.

Quoted 5 lines

If so, overlay will not be
the only one fs who uses BASE_FSTYP and BASE_SCRATCH_DEV things, then we need to differentiate if a feature (e.g. shutdown) is needed by upper layer fs or underlying fs in a case.

First of all, terminology. Many get this wrong.
In overlayfs, the "upper" and "lower" refer to the different underlying layers. In fstests BASE_SCRATCH_DEV is always the same for both OVL_UPPER and OVL_LOWER layers.

Referring to the BASE_SCRATCH_DEV as "underlying" or "base" fs is unambiguous in all cases of overlay/nfs/cifs.

I do not have a good terminology to offer for referring to the "fs under test" be it overlay/cifs/nfs. You are welcome to offer your suggestions.

Quoted 5 lines

...
BTW, a question which isn't belong to this patch:) There're also some failures from those xfstests overlay cases which run unionmount-testsuite (can't rememember all, maybe o/102~109, o/144~117). The error (diff) output are similar as [1]. Is there a fix for that too? Or I missed the fix?

I do not get these errors.
I suppose you are using the latest unionmount-testsuite, although it hasn't been changed for a while anyway.

103.out.bad 2025-06-07 05:16:59.445716549 -0400 line 3

Quoted 11 lines

Thanks,
Zorro

[1]

103.out.bad 2025-06-07 05:16:59.445716549 -0400 renamed from dev/fd/63 2025-06-07 05:16:59.489929312 -0400 +2 -0
@@ -1,2 +1,17 @@
 QA output created by 103
+mount: /mnt/fstests/SCRATCH_DIR/union/m: wrong fs type, bad option, bad superblock on overlay, missing codepage or helper program, or other error.
+       dmesg(1) may have more information after failed mount system call.

Reason for failure to mount is likely to be found in dmesg. Please try to see if it is there.

Quoted 16 lines

103.out.bad 2025-06-07 05:16:59.445716549 -0400 line 15

Quoted 12 lines
103.out.bad 2025-06-07 05:16:59.445716549 -0400 renamed from dev/fd/63 2025-06-07 05:16:59.489929312 -0400 +12 -0
+Traceback (most recent call last):
+  File "/mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.tar.bz2/filesystems/xfstests/unionmount-testsuite/./run", line 362, in <module>
+    func(ctx)
+  File "/mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.tar.bz2/filesystems/xfstests/unionmount-testsuite/tests/rename-file.py", line 71, in subtest_5
+    ctx.rename(f, f2)
+  File "/mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.tar.bz2/filesystems/xfstests/unionmount-testsuite/context.py", line 1254, in rename
+    remount_union(self, rotate_upper=True)
+  File "/mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.tar.bz2/filesystems/xfstests/unionmount-testsuite/remount_union.py", line 35, in remount_union
+    system(cmd)
+  File "/mnt/tests/gitlab.cee.redhat.com/kernel-qe/kernel/-/archive/master/kernel-master.tar.bz2/filesystems/xfstests/unionmount-testsuite/tool_box.py", line 25, in system
+    raise RuntimeError("Command failed: " + command)
+RuntimeError: Command failed: mount -t overlay overlay /mnt/fstests/SCRATCH_DIR/union/m -orw,xino=on -olowerdir=/mnt/fstests/SCRATCH_DIR/union/6/u:/mnt/fstests/SCRATCH_DIR/union/5/u:/mnt/fstests/SCRATCH_DIR/union/4/u:/mnt/fstests/SCRATCH_DIR/union/3/u:/mnt/fstests/SCRATCH_DIR/union/2/u:/mnt/fstests/SCRATCH_DIR/union/1/u:/mnt/fstests/SCRATCH_DIR/union/0/u:/mnt/fstests/SCRATCH_DIR/union/l,upperdir=/mnt/fstests/SCRATCH_DIR/union/7/u,workdir=/mnt/fstests/SCRATCH_DIR/union/7/w

Nothing looks obviously wrong in the mount command above. Was this a regression for you?
with kernel upgrade? libmount upgrade?

Thanks Amir, I'll check more and send another email to talk about this failure :)

Quoted 4 lines

Thanks,
Amir.

#28 lore raw

PATCH v2 5/6 generic: remove incorrect _require_idmapped_mounts checks

In reply to Amir Goldstein

commit f5661920 ("generic: add missed _require_idmapped_mounts check") wrongly adds _require_idmapped_mounts to tests that do not require idmapped mounts support.

The added _require_idmapped_mounts in test generic/633 goes against commit d8dee122 ("idmapped-mounts: always run generic vfs tests") that intentionally removed this requirement from the generic tests.

The added _require_idmapped_mounts in tests generic/69{6,7} causes those tests not to run with overlayfs, which does not support idmapped mounts. However, those tests are regression tests to kernel commit 1639a49ccdce ("fs: move S_ISGID stripping into the vfs_*() helpers") which is documented as also solving a correction issue with overlayfs, so removing this test converage is very much undesired.

Remove the incorrectly added _require_idmapped_mounts checks. Also fix the log in _require_idmapped_mounts to say that "idmapped mounts not support by $FSTYP", which is what the helper checks instead of "vfstests not support by $FSTYP" which is incorrect.

Cc Yang Xu <[email protected]>
Cc Anthony Iliopoulos <[email protected]>
Cc David Disseldorp <[email protected]>
Fixes commit f5661920 ("generic: add missed _require_idmapped_mounts check")
---
 common/rc         | 2 +-
 tests/generic/633 | 1 -
 tests/generic/696 | 1 -
 tests/generic/697 | 1 -
 4 files changed, 1 insertion(+), 4 deletions(-)
common/rc +1 -1
index bffd576a..96d65d1c 100644
@@ -2639,7 +2639,7 @@ _require_idmapped_mounts()
 		--fstype "$FSTYP"
 
 	if [ $? -ne 0 ]; then
-		_notrun "vfstest not support by $FSTYP"
+		_notrun "idmapped mounts not support by $FSTYP"
 	fi
 }
 
tests/generic/633 +0 -1
index f58dbbf5..b683c427 100755
@@ -12,7 +12,6 @@ _begin_fstest auto quick atime attr cap idmapped io_uring mount perms rw unlink
 # Import common functions.
 . ./common/filter
 
-_require_idmapped_mounts
 _require_test
 
 echo "Silence is golden"
tests/generic/696 +0 -1
index d2e86c96..48b3aea0 100755
@@ -17,7 +17,6 @@ _begin_fstest auto quick cap idmapped mount perms rw unlink
 # Import common functions.
 . ./common/filter
 
-_require_idmapped_mounts
 _require_test
 _require_scratch
 _fixed_by_kernel_commit ac6800e279a2 \
tests/generic/697 +0 -1
index 1ce673f7..66444a95 100755
@@ -17,7 +17,6 @@ _begin_fstest auto quick cap acl idmapped mount perms rw unlink
 . ./common/filter
 . ./common/attr
 
-_require_idmapped_mounts
 _require_test
 _require_acls
 _fixed_by_kernel_commit 1639a49ccdce \
Signature · 1 line
2.34.1
#29 lore raw

Re: [PATCH v2 5/6] generic: remove incorrect _require_idmapped_mounts checks

Quoted 20 lines · Amir Goldstein

commit f5661920 ("generic: add missed _require_idmapped_mounts check") wrongly adds _require_idmapped_mounts to tests that do not require idmapped mounts support.

The added _require_idmapped_mounts in test generic/633 goes against commit d8dee122 ("idmapped-mounts: always run generic vfs tests") that intentionally removed this requirement from the generic tests.

The added _require_idmapped_mounts in tests generic/69{6,7} causes those tests not to run with overlayfs, which does not support idmapped mounts. However, those tests are regression tests to kernel commit 1639a49ccdce ("fs: move S_ISGID stripping into the vfs_*() helpers") which is documented as also solving a correction issue with overlayfs, so removing this test converage is very much undesired.

Remove the incorrectly added _require_idmapped_mounts checks. Also fix the log in _require_idmapped_mounts to say that "idmapped mounts not support by $FSTYP", which is what the helper checks instead of "vfstests not support by $FSTYP" which is incorrect.

Quoted 6 lines
Cc Yang Xu <[email protected]>
Cc Anthony Iliopoulos <[email protected]>
Cc David Disseldorp <[email protected]>
Fixes commit f5661920 ("generic: add missed _require_idmapped_mounts check")

---

Is this one changed anything from this ?
https://lore.kernel.org/fstests/[email protected]/

Due to above link has been reviewed by Christian Brauner, do you want to add his RVB to this version?

Thanks,
Zorro

Quoted 58 lines
 common/rc         | 2 +-
 tests/generic/633 | 1 -
 tests/generic/696 | 1 -
 tests/generic/697 | 1 -
 4 files changed, 1 insertion(+), 4 deletions(-)
common/rc +1 -1
index bffd576a..96d65d1c 100644
@@ -2639,7 +2639,7 @@ _require_idmapped_mounts()
 		--fstype "$FSTYP"
 
 	if [ $? -ne 0 ]; then
-		_notrun "vfstest not support by $FSTYP"
+		_notrun "idmapped mounts not support by $FSTYP"
 	fi
 }
 
tests/generic/633 +0 -1
index f58dbbf5..b683c427 100755
@@ -12,7 +12,6 @@ _begin_fstest auto quick atime attr cap idmapped io_uring mount perms rw unlink
 # Import common functions.
 . ./common/filter
 
-_require_idmapped_mounts
 _require_test
 
 echo "Silence is golden"
tests/generic/696 +0 -1
index d2e86c96..48b3aea0 100755
@@ -17,7 +17,6 @@ _begin_fstest auto quick cap idmapped mount perms rw unlink
 # Import common functions.
 . ./common/filter
 
-_require_idmapped_mounts
 _require_test
 _require_scratch
 _fixed_by_kernel_commit ac6800e279a2 \
tests/generic/697 +0 -1
index 1ce673f7..66444a95 100755
@@ -17,7 +17,6 @@ _begin_fstest auto quick cap acl idmapped mount perms rw unlink
 . ./common/filter
 . ./common/attr
 
-_require_idmapped_mounts
 _require_test
 _require_acls
 _fixed_by_kernel_commit 1639a49ccdce \
Signature · 1 line
2.34.1
#30 lore raw

Re: [PATCH v2 5/6] generic: remove incorrect _require_idmapped_mounts checks

Quoted 29 lines · Zorro Lang
Quoted 20 lines · Amir Goldstein

commit f5661920 ("generic: add missed _require_idmapped_mounts check") wrongly adds _require_idmapped_mounts to tests that do not require idmapped mounts support.

The added _require_idmapped_mounts in test generic/633 goes against commit d8dee122 ("idmapped-mounts: always run generic vfs tests") that intentionally removed this requirement from the generic tests.

The added _require_idmapped_mounts in tests generic/69{6,7} causes those tests not to run with overlayfs, which does not support idmapped mounts. However, those tests are regression tests to kernel commit 1639a49ccdce ("fs: move S_ISGID stripping into the vfs_*() helpers") which is documented as also solving a correction issue with overlayfs, so removing this test converage is very much undesired.

Remove the incorrectly added _require_idmapped_mounts checks. Also fix the log in _require_idmapped_mounts to say that "idmapped mounts not support by $FSTYP", which is what the helper checks instead of "vfstests not support by $FSTYP" which is incorrect.

Quoted 6 lines
Cc Yang Xu <[email protected]>
Cc Anthony Iliopoulos <[email protected]>
Cc David Disseldorp <[email protected]>
Fixes commit f5661920 ("generic: add missed _require_idmapped_mounts check")

---

Quoted 2 lines

Is this one changed anything from this ?
https://lore.kernel.org/fstests/[email protected]/

There is no change. I just forgot to add the RVB.

Quoted 4 lines

Due to above link has been reviewed by Christian Brauner, do you want to add his RVB to this version?

Yes please.

Thanks,
Amir.

#31 lore raw

PATCH v2 6/6 generic/699: fix failure with MOUNT_OPTIONS

In reply to Amir Goldstein

generic/699 uses overalyfs helper _overlay_mount_dirs, which is meant to be used by overlayfs tests, where MOUNT_OPTIONS refer to overalyfs mount options.

Using this helper from a generic test when FSTYP is not overlay is causing undesired results. For example, when MOUNT_OPTIONS is defined and includes a mount option not supported by overalyfs (e.g. 'acl'), the test is notrun because of:

mount: /vdc/ovl-merge: fsconfig() failed: overlay: Unknown parameter 'acl'.

There is no other generic test that includes the common/overlay helpers and uses them, so remove this practice from generic/699 as well.

---
 tests/generic/699 | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)
tests/generic/699 +10 -7
index 620a40aa..2a6f857d 100755
@@ -8,7 +8,6 @@
 # mounts specifically.
 #
 . ./common/preamble
-. ./common/overlay
 _begin_fstest auto quick perms attr idmapped mount
 
 # Override the default cleanup function.
@@ -96,20 +95,24 @@ reset_ownership()
 	stat -c '%u:%g' $path
 }
 
+setup_overlayfs()
+{
+	mkdir -p $upper $work $merge
+	_mount -t overlay -o lowerdir=$lower,upperdir=$upper,workdir=$work \
+		overlay $merge $*
+}
+
 # Prepare overlayfs with metacopy turned off.
 setup_overlayfs_idmapped_lower_metacopy_off()
 {
-	mkdir -p $upper $work $merge
-	_overlay_mount_dirs $lower $upper $work \
-			    overlay $merge -ometacopy=off || \
-			    _notrun "overlayfs doesn't support idmappped layers"
+	setup_overlayfs -ometacopy=off || \
+	    _notrun "overlayfs doesn't support idmappped layers"
 }
 
 # Prepare overlayfs with metacopy turned on.
 setup_overlayfs_idmapped_lower_metacopy_on()
 {
-	mkdir -p $upper $work $merge
-	_overlay_mount_dirs $lower $upper $work overlay $merge -ometacopy=on
+	setup_overlayfs -ometacopy=on
 }
 
 reset_overlayfs()
Signature · 1 line
2.34.1
Reviewed-by on v2 6/6
#32 lore raw

Re: [PATCH v2 6/6] generic/699: fix failure with MOUNT_OPTIONS

Quoted 9 lines · Amir Goldstein

generic/699 uses overalyfs helper _overlay_mount_dirs, which is meant to be used by overlayfs tests, where MOUNT_OPTIONS refer to overalyfs mount options.

Using this helper from a generic test when FSTYP is not overlay is causing undesired results. For example, when MOUNT_OPTIONS is defined and includes a mount option not supported by overalyfs (e.g. 'acl'), the test is notrun because of:

Quoted 7 lines

mount: /vdc/ovl-merge: fsconfig() failed: overlay: Unknown parameter 'acl'.

There is no other generic test that includes the common/overlay helpers and uses them, so remove this practice from generic/699 as well.

---

Good to me,

Quoted 49 lines
 tests/generic/699 | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)
tests/generic/699 +10 -7
index 620a40aa..2a6f857d 100755
@@ -8,7 +8,6 @@
 # mounts specifically.
 #
 . ./common/preamble
-. ./common/overlay
 _begin_fstest auto quick perms attr idmapped mount
 
 # Override the default cleanup function.
@@ -96,20 +95,24 @@ reset_ownership()
 	stat -c '%u:%g' $path
 }
 
+setup_overlayfs()
+{
+	mkdir -p $upper $work $merge
+	_mount -t overlay -o lowerdir=$lower,upperdir=$upper,workdir=$work \
+		overlay $merge $*
+}
+
 # Prepare overlayfs with metacopy turned off.
 setup_overlayfs_idmapped_lower_metacopy_off()
 {
-	mkdir -p $upper $work $merge
-	_overlay_mount_dirs $lower $upper $work \
-			    overlay $merge -ometacopy=off || \
-			    _notrun "overlayfs doesn't support idmappped layers"
+	setup_overlayfs -ometacopy=off || \
+	    _notrun "overlayfs doesn't support idmappped layers"
 }
 
 # Prepare overlayfs with metacopy turned on.
 setup_overlayfs_idmapped_lower_metacopy_on()
 {
-	mkdir -p $upper $work $merge
-	_overlay_mount_dirs $lower $upper $work overlay $merge -ometacopy=on
+	setup_overlayfs -ometacopy=on
 }
 
 reset_overlayfs()
Signature · 1 line
2.34.1