Skip to contentSkip to conversation

fstests overlay updates for 6.16-rc1

PATCH In review Amir Goldstein 4 participants 16 messages Jun 9, 2025 – Jun 18, 2025

lore
Share & tools
lore
#0 lore raw

cover fstests overlay updates for 6.16-rc1

Zorro,

Please find two new tests by Miklos to cover ian overlayfs feature merged to 6.16-rc1. Those tests do notrun on older kernels.

Adding two new helpers to sort out the shutdown test requiremetns w.r.t overlayfs following our discussion on generic/623 patch review [1].

Thanks,
Amir.

[1] https://lore.kernel.org/fstests/[email protected]/

Amir Goldstein (2):
fstests: add helper _scratch_shutdown_and_syncfs
fstests: add helper _require_xfs_io_shutdown

Miklos Szeredi (1):
overlay/08[89]: add tests for data-only redirect with userxattr

 common/overlay        |  29 +++++
 common/rc             |  48 +++++++
 tests/generic/623     |   2 +-
 tests/overlay/087     |  13 +-
 tests/overlay/088     | 296 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/088.out |  39 ++++++
 tests/overlay/089     | 272 ++++++++++++++++++++++++++++++++++++++
 tests/overlay/089.out |   5 +
 tests/xfs/546         |   5 +-
 9 files changed, 695 insertions(+), 14 deletions(-)
 create mode 100755 tests/overlay/088
 create mode 100644 tests/overlay/088.out
 create mode 100755 tests/overlay/089
 create mode 100644 tests/overlay/089.out
Signature · 1 line
2.34.1
#1 lore raw

PATCH 1/3 fstests: add helper _scratch_shutdown_and_syncfs

Test xfs/546 has to chain syncfs after shutdown and cannot use the _scratch_shitdown helper, because after shutdown a fd cannot be opened to execute syncfs on.

The xfs_io command of chaining syncfs after shutdown is rather more complex to execute in the derived overlayfs test overlay/087.

Add a helper to abstract this complexity from test writers. Add a _require statement to match.

---
 common/rc         | 27 +++++++++++++++++++++++++++
 tests/overlay/087 | 13 +++----------
 tests/xfs/546     |  5 ++---
 3 files changed, 32 insertions(+), 13 deletions(-)
common/rc +27 -0
index f71cc8f0..d9a8b52e 100644
@@ -595,6 +595,27 @@ _scratch_shutdown_handle()
 	fi
 }
 
+_scratch_shutdown_and_syncfs()
+{
+	if [ $FSTYP = "overlay" ]; then
+		# In lagacy overlay usage, it may specify directory as
+		# SCRATCH_DEV, in this case OVL_BASE_SCRATCH_DEV
+		# will be null, so check OVL_BASE_SCRATCH_DEV before
+		# running shutdown to avoid shutting down base fs accidently.
+		if [ -z $OVL_BASE_SCRATCH_DEV ]; then
+			_fail "_scratch_shutdown: call _require_scratch_shutdown first in test"
+		fi
+		# This command is complicated a bit because in the case of overlayfs the
+		# syncfs fd needs to be opened before shutdown and it is different from the
+		# shutdown fd, so we cannot use the _scratch_shutdown() helper.
+		# Filter out xfs_io output of active fds.
+		$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' \
+				-c close -c syncfs $SCRATCH_MNT | grep -vF '[00'
+	else
+		$XFS_IO_PROG -x -c 'shutdown -f ' -c syncfs $SCRATCH_MNT
+	fi
+}
+
 _move_mount()
 {
 	local mnt=$1
@@ -4102,6 +4123,12 @@ _require_scratch_shutdown()
 	_scratch_unmount
 }
 
+_require_scratch_shutdown_and_syncfs()
+{
+	_require_xfs_io_command syncfs
+	_require_scratch_shutdown
+}
+
 _check_s_dax()
 {
 	local target=$1
tests/overlay/087 +3 -10
index a5afb0d5..2ad069db 100755
@@ -32,9 +32,8 @@ _begin_fstest auto quick mount shutdown
 
 
 # Modify as appropriate.
-_require_xfs_io_command syncfs
 _require_scratch_nocheck
-_require_scratch_shutdown
+_require_scratch_shutdown_and_syncfs
 
 [ "$OVL_BASE_FSTYP" == "xfs" ] || \
 	_notrun "base fs $OVL_BASE_FSTYP has unknown behavior with syncfs after shutdown"
@@ -43,19 +42,13 @@ _require_scratch_shutdown
 # bother checking the filesystem afterwards since we never wrote anything.
 echo "=== syncfs after shutdown"
 _scratch_mount
-# This command is complicated a bit because in the case of overlayfs the
-# syncfs fd needs to be opened before shutdown and it is different from the
-# shutdown fd, so we cannot use the _scratch_shutdown() helper.
-# Filter out xfs_io output of active fds.
-$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
-	grep -vF '[00'
+_scratch_shutdown_and_syncfs
 
 # Now repeat the same test with a volatile overlayfs mount and expect no error
 _scratch_unmount
 echo "=== syncfs after shutdown (volatile)"
 _scratch_mount -o volatile
-$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
-	grep -vF '[00'
+_scratch_shutdown_and_syncfs
 
 # success, all done
 status=0
tests/xfs/546 +2 -3
index 316ffc50..c50d41a6 100755
@@ -27,14 +27,13 @@ _begin_fstest auto quick shutdown
 
 
 # Modify as appropriate.
-_require_xfs_io_command syncfs
 _require_scratch_nocheck
-_require_scratch_shutdown
+_require_scratch_shutdown_and_syncfs
 
 # Reuse the fs formatted when we checked for the shutdown ioctl, and don't
 # bother checking the filesystem afterwards since we never wrote anything.
 _scratch_mount
-$XFS_IO_PROG -x -c 'shutdown -f ' -c syncfs $SCRATCH_MNT
+_scratch_shutdown_and_syncfs
 
 # success, all done
 status=0
Signature · 1 line
2.34.1
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] fstests: add helper _scratch_shutdown_and_syncfs

Quoted 2 lines · Amir Goldstein

Test xfs/546 has to chain syncfs after shutdown and cannot use the _scratch_shitdown helper, because after shutdown a fd

shutdown

Though even this typo will age better than the original ioctl... ;)

Quoted 20 lines

cannot be opened to execute syncfs on.

The xfs_io command of chaining syncfs after shutdown is rather more complex to execute in the derived overlayfs test overlay/087.

Add a helper to abstract this complexity from test writers. Add a _require statement to match.

---
 common/rc         | 27 +++++++++++++++++++++++++++
 tests/overlay/087 | 13 +++----------
 tests/xfs/546     |  5 ++---
 3 files changed, 32 insertions(+), 13 deletions(-)
common/rc +0 -0
index f71cc8f0..d9a8b52e 100644

common/rc _scratch_shutdown_handle() · line 601

Quoted 7 lines
common/rc +4 -0
@@ -595,6 +595,27 @@ _scratch_shutdown_handle()
 	fi
 }
 
+_scratch_shutdown_and_syncfs()
+{
+	if [ $FSTYP = "overlay" ]; then
+		# In lagacy overlay usage, it may specify directory as

legacy

(Note there's a similar typo in _require_scratch_shutdown itself)

With the typos fixed this looks like a reasonable cleanup

--D

Quoted 95 lines
common/rc +23 -0
+		# SCRATCH_DEV, in this case OVL_BASE_SCRATCH_DEV
+		# will be null, so check OVL_BASE_SCRATCH_DEV before
+		# running shutdown to avoid shutting down base fs accidently.
+		if [ -z $OVL_BASE_SCRATCH_DEV ]; then
+			_fail "_scratch_shutdown: call _require_scratch_shutdown first in test"
+		fi
+		# This command is complicated a bit because in the case of overlayfs the
+		# syncfs fd needs to be opened before shutdown and it is different from the
+		# shutdown fd, so we cannot use the _scratch_shutdown() helper.
+		# Filter out xfs_io output of active fds.
+		$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' \
+				-c close -c syncfs $SCRATCH_MNT | grep -vF '[00'
+	else
+		$XFS_IO_PROG -x -c 'shutdown -f ' -c syncfs $SCRATCH_MNT
+	fi
+}
+
 _move_mount()
 {
 	local mnt=$1
@@ -4102,6 +4123,12 @@ _require_scratch_shutdown()
 	_scratch_unmount
 }
 
+_require_scratch_shutdown_and_syncfs()
+{
+	_require_xfs_io_command syncfs
+	_require_scratch_shutdown
+}
+
 _check_s_dax()
 {
 	local target=$1
tests/overlay/087 +3 -10
index a5afb0d5..2ad069db 100755
@@ -32,9 +32,8 @@ _begin_fstest auto quick mount shutdown
 
 
 # Modify as appropriate.
-_require_xfs_io_command syncfs
 _require_scratch_nocheck
-_require_scratch_shutdown
+_require_scratch_shutdown_and_syncfs
 
 [ "$OVL_BASE_FSTYP" == "xfs" ] || \
 	_notrun "base fs $OVL_BASE_FSTYP has unknown behavior with syncfs after shutdown"
@@ -43,19 +42,13 @@ _require_scratch_shutdown
 # bother checking the filesystem afterwards since we never wrote anything.
 echo "=== syncfs after shutdown"
 _scratch_mount
-# This command is complicated a bit because in the case of overlayfs the
-# syncfs fd needs to be opened before shutdown and it is different from the
-# shutdown fd, so we cannot use the _scratch_shutdown() helper.
-# Filter out xfs_io output of active fds.
-$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
-	grep -vF '[00'
+_scratch_shutdown_and_syncfs
 
 # Now repeat the same test with a volatile overlayfs mount and expect no error
 _scratch_unmount
 echo "=== syncfs after shutdown (volatile)"
 _scratch_mount -o volatile
-$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
-	grep -vF '[00'
+_scratch_shutdown_and_syncfs
 
 # success, all done
 status=0
tests/xfs/546 +2 -3
index 316ffc50..c50d41a6 100755
@@ -27,14 +27,13 @@ _begin_fstest auto quick shutdown
 
 
 # Modify as appropriate.
-_require_xfs_io_command syncfs
 _require_scratch_nocheck
-_require_scratch_shutdown
+_require_scratch_shutdown_and_syncfs
 
 # Reuse the fs formatted when we checked for the shutdown ioctl, and don't
 # bother checking the filesystem afterwards since we never wrote anything.
 _scratch_mount
-$XFS_IO_PROG -x -c 'shutdown -f ' -c syncfs $SCRATCH_MNT
+_scratch_shutdown_and_syncfs
 
 # success, all done
 status=0
Signature · 1 line
2.34.1
Reviewed-by on v1 1/3
#3 lore raw

Re: [PATCH 1/3] fstests: add helper _scratch_shutdown_and_syncfs

In reply to Amir Goldstein
Quoted 4 lines · Amir Goldstein

Test xfs/546 has to chain syncfs after shutdown and cannot use the _scratch_shitdown helper, because after shutdown a fd cannot be opened to execute syncfs on.

Quoted 8 lines

The xfs_io command of chaining syncfs after shutdown is rather more complex to execute in the derived overlayfs test overlay/087.

Add a helper to abstract this complexity from test writers. Add a _require statement to match.

---

As we talked, I think this version is good to me, I'll merge it with the typo fix which Darrick metioned :)

Thanks,
Zorro

Quoted 111 lines
 common/rc         | 27 +++++++++++++++++++++++++++
 tests/overlay/087 | 13 +++----------
 tests/xfs/546     |  5 ++---
 3 files changed, 32 insertions(+), 13 deletions(-)
common/rc +27 -0
index f71cc8f0..d9a8b52e 100644
@@ -595,6 +595,27 @@ _scratch_shutdown_handle()
 	fi
 }
 
+_scratch_shutdown_and_syncfs()
+{
+	if [ $FSTYP = "overlay" ]; then
+		# In lagacy overlay usage, it may specify directory as
+		# SCRATCH_DEV, in this case OVL_BASE_SCRATCH_DEV
+		# will be null, so check OVL_BASE_SCRATCH_DEV before
+		# running shutdown to avoid shutting down base fs accidently.
+		if [ -z $OVL_BASE_SCRATCH_DEV ]; then
+			_fail "_scratch_shutdown: call _require_scratch_shutdown first in test"
+		fi
+		# This command is complicated a bit because in the case of overlayfs the
+		# syncfs fd needs to be opened before shutdown and it is different from the
+		# shutdown fd, so we cannot use the _scratch_shutdown() helper.
+		# Filter out xfs_io output of active fds.
+		$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' \
+				-c close -c syncfs $SCRATCH_MNT | grep -vF '[00'
+	else
+		$XFS_IO_PROG -x -c 'shutdown -f ' -c syncfs $SCRATCH_MNT
+	fi
+}
+
 _move_mount()
 {
 	local mnt=$1
@@ -4102,6 +4123,12 @@ _require_scratch_shutdown()
 	_scratch_unmount
 }
 
+_require_scratch_shutdown_and_syncfs()
+{
+	_require_xfs_io_command syncfs
+	_require_scratch_shutdown
+}
+
 _check_s_dax()
 {
 	local target=$1
tests/overlay/087 +3 -10
index a5afb0d5..2ad069db 100755
@@ -32,9 +32,8 @@ _begin_fstest auto quick mount shutdown
 
 
 # Modify as appropriate.
-_require_xfs_io_command syncfs
 _require_scratch_nocheck
-_require_scratch_shutdown
+_require_scratch_shutdown_and_syncfs
 
 [ "$OVL_BASE_FSTYP" == "xfs" ] || \
 	_notrun "base fs $OVL_BASE_FSTYP has unknown behavior with syncfs after shutdown"
@@ -43,19 +42,13 @@ _require_scratch_shutdown
 # bother checking the filesystem afterwards since we never wrote anything.
 echo "=== syncfs after shutdown"
 _scratch_mount
-# This command is complicated a bit because in the case of overlayfs the
-# syncfs fd needs to be opened before shutdown and it is different from the
-# shutdown fd, so we cannot use the _scratch_shutdown() helper.
-# Filter out xfs_io output of active fds.
-$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
-	grep -vF '[00'
+_scratch_shutdown_and_syncfs
 
 # Now repeat the same test with a volatile overlayfs mount and expect no error
 _scratch_unmount
 echo "=== syncfs after shutdown (volatile)"
 _scratch_mount -o volatile
-$XFS_IO_PROG -x -c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close -c syncfs $SCRATCH_MNT | \
-	grep -vF '[00'
+_scratch_shutdown_and_syncfs
 
 # success, all done
 status=0
tests/xfs/546 +2 -3
index 316ffc50..c50d41a6 100755
@@ -27,14 +27,13 @@ _begin_fstest auto quick shutdown
 
 
 # Modify as appropriate.
-_require_xfs_io_command syncfs
 _require_scratch_nocheck
-_require_scratch_shutdown
+_require_scratch_shutdown_and_syncfs
 
 # Reuse the fs formatted when we checked for the shutdown ioctl, and don't
 # bother checking the filesystem afterwards since we never wrote anything.
 _scratch_mount
-$XFS_IO_PROG -x -c 'shutdown -f ' -c syncfs $SCRATCH_MNT
+_scratch_shutdown_and_syncfs
 
 # success, all done
 status=0
Signature · 1 line
2.34.1
#4 lore raw

PATCH 2/3 fstests: add helper _require_xfs_io_shutdown

In reply to Amir Goldstein

Requirements for tests that shutdown fs using "xfs_io -c shutdown". The requirements are stricter than the requirement for tests that shutdown fs using _scratch_shutdown helper.

Generally, with overlay fs, tests can do _scratch_shutdown, but not xfs_io -c shutdown.

Encode this stricter requirement in helper _require_xfs_io_shutdown and use it in test generic/623, to express that it cannot run on overalyfs.

---
 common/rc         | 21 +++++++++++++++++++++
 tests/generic/623 |  2 +-
 2 files changed, 22 insertions(+), 1 deletion(-)
common/rc +21 -0
index d9a8b52e..21899a4a 100644
@@ -616,6 +616,27 @@ _scratch_shutdown_and_syncfs()
 	fi
 }
 
+# Requirements for tests that shutdown fs using "xfs_io -c shutdown".
+# The requirements are stricter than the requirement for tests that
+# shutdown fs using _scratch_shutdown helper.
+# Generally, with overlay fs, test can do _scratch_shutdown, but not
+# xfs_io -c shutdown.
+# It is possible, but not trivial, to execute "xfs_io -c shutdown" as part
+# of a command sequence when shutdown ioctl is to be performed on the base fs
+# (i.e. on an alternative _scratch_shutdown_handle path) as the example code
+# in _scratch_shutdown_and_syncfs() does.
+# A test that open codes this pattern can relax the _require_xfs_io_shutdown
+# requirement down to _require_scratch_shutdown.
+_require_xfs_io_shutdown()
+{
+	if [ _scratch_shutdown_handle != $SCRATCH_MNT ]; then
+		# Most likely overlayfs
+		_notrun "xfs_io -c shutdown not supported on $FSTYP"
+	fi
+	_require_xfs_io_command "shutdown"
+	_require_scratch_shutdown
+}
+
 _move_mount()
 {
 	local mnt=$1
tests/generic/623 +1 -1
index b97e2adb..f546d529 100755
@@ -15,7 +15,7 @@ _begin_fstest auto quick shutdown mmap
 	"xfs: restore shutdown check in mapped write fault path"
 
 _require_scratch_nocheck
-_require_scratch_shutdown
+_require_xfs_io_shutdown
 
 _scratch_mkfs &>> $seqres.full
 _scratch_mount
Signature · 1 line
2.34.1
Tested-by on v1 2/3
#5 lore raw

Re: [PATCH 2/3] fstests: add helper _require_xfs_io_shutdown

Hi Amir,

Em 09/06/2025 12:19, Amir Goldstein escreveu:

Quoted 7 lines

Requirements for tests that shutdown fs using "xfs_io -c shutdown". The requirements are stricter than the requirement for tests that shutdown fs using _scratch_shutdown helper.

Generally, with overlay fs, tests can do _scratch_shutdown, but not xfs_io -c shutdown.

Quoted 8 lines

Encode this stricter requirement in helper _require_xfs_io_shutdown and use it in test generic/623, to express that it cannot run on overalyfs.

---

Thanks for the fix!

#6 lore raw

Re: [PATCH 2/3] fstests: add helper _require_xfs_io_shutdown

In reply to Amir Goldstein
Quoted 7 lines · Amir Goldstein

Requirements for tests that shutdown fs using "xfs_io -c shutdown". The requirements are stricter than the requirement for tests that shutdown fs using _scratch_shutdown helper.

Generally, with overlay fs, tests can do _scratch_shutdown, but not xfs_io -c shutdown.

Quoted 7 lines

Encode this stricter requirement in helper _require_xfs_io_shutdown and use it in test generic/623, to express that it cannot run on overalyfs.

Makes sense to me, assuming you don't want to try to integrate the 'open shutdown handle' dance into this test.

ioargs=(-x -c "mmap 0 4k" -c "mwrite 0 4k")
case "$FSTYP" in
overlayfs)

	ioargs+=(-c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close)
	;;
*)
	ioargs+=(-c shutdown)
	;;

esac
ioargs+=(-c fsync -c "mwrite 0 4k" $file)

$XFS_IO_PROG "${ioargs[@]}" | _filter_xfs_io

(Though I don't know if you actually tried that and it didn't work, or maybe overlayfs mmap is weird, etc...)

--D

Quoted 54 lines
---
 common/rc         | 21 +++++++++++++++++++++
 tests/generic/623 |  2 +-
 2 files changed, 22 insertions(+), 1 deletion(-)
common/rc +21 -0
index d9a8b52e..21899a4a 100644
@@ -616,6 +616,27 @@ _scratch_shutdown_and_syncfs()
 	fi
 }
 
+# Requirements for tests that shutdown fs using "xfs_io -c shutdown".
+# The requirements are stricter than the requirement for tests that
+# shutdown fs using _scratch_shutdown helper.
+# Generally, with overlay fs, test can do _scratch_shutdown, but not
+# xfs_io -c shutdown.
+# It is possible, but not trivial, to execute "xfs_io -c shutdown" as part
+# of a command sequence when shutdown ioctl is to be performed on the base fs
+# (i.e. on an alternative _scratch_shutdown_handle path) as the example code
+# in _scratch_shutdown_and_syncfs() does.
+# A test that open codes this pattern can relax the _require_xfs_io_shutdown
+# requirement down to _require_scratch_shutdown.
+_require_xfs_io_shutdown()
+{
+	if [ _scratch_shutdown_handle != $SCRATCH_MNT ]; then
+		# Most likely overlayfs
+		_notrun "xfs_io -c shutdown not supported on $FSTYP"
+	fi
+	_require_xfs_io_command "shutdown"
+	_require_scratch_shutdown
+}
+
 _move_mount()
 {
 	local mnt=$1
tests/generic/623 +1 -1
index b97e2adb..f546d529 100755
@@ -15,7 +15,7 @@ _begin_fstest auto quick shutdown mmap
 	"xfs: restore shutdown check in mapped write fault path"
 
 _require_scratch_nocheck
-_require_scratch_shutdown
+_require_xfs_io_shutdown
 
 _scratch_mkfs &>> $seqres.full
 _scratch_mount
Signature · 1 line
2.34.1
#7 lore raw

Re: [PATCH 2/3] fstests: add helper _require_xfs_io_shutdown

Quoted 28 lines · Darrick J. Wong
Quoted 7 lines · Amir Goldstein

Requirements for tests that shutdown fs using "xfs_io -c shutdown". The requirements are stricter than the requirement for tests that shutdown fs using _scratch_shutdown helper.

Generally, with overlay fs, tests can do _scratch_shutdown, but not xfs_io -c shutdown.

Quoted 7 lines

Encode this stricter requirement in helper _require_xfs_io_shutdown and use it in test generic/623, to express that it cannot run on overalyfs.

Makes sense to me, assuming you don't want to try to integrate the 'open shutdown handle' dance into this test.

ioargs=(-x -c "mmap 0 4k" -c "mwrite 0 4k")
case "$FSTYP" in
overlayfs)

        ioargs+=(-c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close)
        ;;
*)
        ioargs+=(-c shutdown)
        ;;
Quoted 7 lines

esac
ioargs+=(-c fsync -c "mwrite 0 4k" $file)

$XFS_IO_PROG "${ioargs[@]}" | _filter_xfs_io

(Though I don't know if you actually tried that and it didn't work, or maybe overlayfs mmap is weird, etc...)

I did not try it.
I had briefly considered trying and decided it's not worth it. overlayfs doesn't have an aops of its own, and mmap results in a memory map of the underlying file directly, so the test will essentially testing the base via overlayfs which does not add much test coverage.

I do not object to making this test run on overlayfs, but I do wish to keep the helper around for future tests that will not do the dance.

Thanks,
Amir.

Reviewed-by
#8 lore raw

Re: [PATCH 2/3] fstests: add helper _require_xfs_io_shutdown

Quoted 37 lines · Amir Goldstein
Quoted 28 lines · Darrick J. Wong
Quoted 7 lines · Amir Goldstein

Requirements for tests that shutdown fs using "xfs_io -c shutdown". The requirements are stricter than the requirement for tests that shutdown fs using _scratch_shutdown helper.

Generally, with overlay fs, tests can do _scratch_shutdown, but not xfs_io -c shutdown.

Quoted 7 lines

Encode this stricter requirement in helper _require_xfs_io_shutdown and use it in test generic/623, to express that it cannot run on overalyfs.

Makes sense to me, assuming you don't want to try to integrate the 'open shutdown handle' dance into this test.

ioargs=(-x -c "mmap 0 4k" -c "mwrite 0 4k")
case "$FSTYP" in
overlayfs)

        ioargs+=(-c "open $(_scratch_shutdown_handle)" -c 'shutdown -f ' -c close)
        ;;
*)
        ioargs+=(-c shutdown)
        ;;
Quoted 7 lines

esac
ioargs+=(-c fsync -c "mwrite 0 4k" $file)

$XFS_IO_PROG "${ioargs[@]}" | _filter_xfs_io

(Though I don't know if you actually tried that and it didn't work, or maybe overlayfs mmap is weird, etc...)

Quoted 8 lines

I did not try it.
I had briefly considered trying and decided it's not worth it. overlayfs doesn't have an aops of its own, and mmap results in a memory map of the underlying file directly, so the test will essentially testing the base via overlayfs which does not add much test coverage.

I do not object to making this test run on overlayfs, but I do wish to keep the helper around for future tests that will not do the dance.

<shrug> If fstests were written in a modern hipster language then passing around a lambda would be easy and trivial...

...but this is bash.

--D

Quoted 2 lines

Thanks,
Amir.

Reviewed-by on v1 2/3
#9 lore raw

Re: [PATCH 2/3] fstests: add helper _require_xfs_io_shutdown

In reply to Amir Goldstein
Quoted 36 lines · Amir Goldstein

Requirements for tests that shutdown fs using "xfs_io -c shutdown". The requirements are stricter than the requirement for tests that shutdown fs using _scratch_shutdown helper.

Generally, with overlay fs, tests can do _scratch_shutdown, but not xfs_io -c shutdown.

Encode this stricter requirement in helper _require_xfs_io_shutdown and use it in test generic/623, to express that it cannot run on overalyfs.

---
 common/rc         | 21 +++++++++++++++++++++
 tests/generic/623 |  2 +-
 2 files changed, 22 insertions(+), 1 deletion(-)
common/rc +9 -0
index d9a8b52e..21899a4a 100644
@@ -616,6 +616,27 @@ _scratch_shutdown_and_syncfs()
 	fi
 }
 
+# Requirements for tests that shutdown fs using "xfs_io -c shutdown".
+# The requirements are stricter than the requirement for tests that
+# shutdown fs using _scratch_shutdown helper.
+# Generally, with overlay fs, test can do _scratch_shutdown, but not
+# xfs_io -c shutdown.
+# It is possible, but not trivial, to execute "xfs_io -c shutdown" as part
+# of a command sequence when shutdown ioctl is to be performed on the base fs
+# (i.e. on an alternative _scratch_shutdown_handle path) as the example code
+# in _scratch_shutdown_and_syncfs() does.

common/rc _scratch_shutdown_and_syncfs() · line 637

Quoted 10 lines
common/rc +10 -0
+# A test that open codes this pattern can relax the _require_xfs_io_shutdown
+# requirement down to _require_scratch_shutdown.
+_require_xfs_io_shutdown()
+{
+	if [ _scratch_shutdown_handle != $SCRATCH_MNT ]; then
+		# Most likely overlayfs
+		_notrun "xfs_io -c shutdown not supported on $FSTYP"
+	fi
+	_require_xfs_io_command "shutdown"
+	_require_scratch_shutdown

Thanks, I think this's good to me now,

Quoted 21 lines
common/rc +2 -0
+}
+
 _move_mount()
 {
 	local mnt=$1
tests/generic/623 +1 -1
index b97e2adb..f546d529 100755
@@ -15,7 +15,7 @@ _begin_fstest auto quick shutdown mmap
 	"xfs: restore shutdown check in mapped write fault path"
 
 _require_scratch_nocheck
-_require_scratch_shutdown
+_require_xfs_io_shutdown
 
 _scratch_mkfs &>> $seqres.full
 _scratch_mount
Signature · 1 line
2.34.1
#10 lore raw

PATCH 3/3 overlay/08[89]: add tests for data-only redirect with userxattr

In reply to Amir Goldstein

From: Miklos Szeredi <[email protected]>

New kernel feature (target release is v6.16) allows data-only redirect to be enabled without metacopy and redirect_dir turned on. This works with or without verity enabled.

Tests are done with the userxattr option, to verify that it will work in a user namespace.

---
 common/overlay        |  29 +++++
 tests/overlay/088     | 296 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/088.out |  39 ++++++
 tests/overlay/089     | 272 ++++++++++++++++++++++++++++++++++++++
 tests/overlay/089.out |   5 +
 5 files changed, 641 insertions(+)
 create mode 100755 tests/overlay/088
 create mode 100644 tests/overlay/088.out
 create mode 100755 tests/overlay/089
 create mode 100644 tests/overlay/089.out
common/overlay +29 -0
index 0be943b1..d02d40b1 100644
@@ -271,6 +271,22 @@ _require_scratch_overlay_lowerdir_add_layers()
 	_scratch_unmount
 }
 
+# Check kernel support for datadir+=<datadir> without "metacopy=on" option
+_require_scratch_overlay_datadir_without_metacopy()
+{
+	local lowerdir="$OVL_BASE_SCRATCH_MNT/$OVL_UPPER"
+	local datadir="$OVL_BASE_SCRATCH_MNT/$OVL_LOWER"
+
+	_scratch_mkfs > /dev/null 2>&1
+	_overlay_scratch_mount_opts \
+		-o"lowerdir+=$lowerdir,datadir+=$datadir" > /dev/null 2>&1 || \
+	        _notrun "overlay datadir+ without metacopy not supported on ${SCRATCH_DEV}"
+
+	_scratch_unmount
+
+}
+
+
 # Helper function to check underlying dirs of overlay filesystem
 _overlay_fsck_dirs()
 {
@@ -472,6 +488,19 @@ _require_unionmount_testsuite()
 		_notrun "newer version of unionmount testsuite required to support OVERLAY_MOUNT_OPTIONS."
 }
 
+# transform overlay xattrs (trusted.overlay -> user.overlay)
+_overlay_trusted_to_user()
+{
+	local dir=$1
+
+	for file in `find $dir`; do
+		_getfattr --absolute-names -d -m '^trusted.overlay.(redirect|metacopy)$' $file  | sed 's/^trusted/user/' | setfattr --restore=-
+		for xattr in `_getfattr --absolute-names -d -m '^trusted.overlay.' $file  | tail -n +2 | cut -d= -f1`; do
+			setfattr -x $xattr $file;
+		done
+	done
+}
+
 _unionmount_testsuite_run()
 {
 	[ "$FSTYP" = overlay ] || \
tests/overlay/088 new file +296 -0

296 lines not on this page · raw message

tests/overlay/088.out new file +39 -0
new file mode 100644
index 00000000..b587b874
@@ -0,0 +1,39 @@
+QA output created by 088
+
+== Check no follow to lowerdata layer without redirect ==
+No access to lowerdata layer datafile
+Unmount and Mount rw
+No access to lowerdata layer datafile
+No access to lowerdata layer datafile2
+Unmount and Mount rw
+No access to lowerdata layer datafile2
+No access to lowerdata layer shared
+Unmount and Mount rw
+No access to lowerdata layer shared
+
+== Check no follow to lowerdata layer with relative redirect ==
+No access to lowerdata layer datafile
+Unmount and Mount rw
+No access to lowerdata layer datafile
+No access to lowerdata layer datafile2
+Unmount and Mount rw
+No access to lowerdata layer datafile2
+No access to lowerdata layer shared
+Unmount and Mount rw
+No access to lowerdata layer shared
+
+== Check follow to lowerdata layer with absolute redirect ==
+Mount ro
+check properties of copied up file datafile
+Unmount and Mount rw
+check properties of copied up file datafile
+Mount ro
+check properties of copied up file datafile2
+Unmount and Mount rw
+check properties of copied up file datafile2
+Mount ro
+check properties of copied up file shared
+Unmount and Mount rw
+check properties of copied up file shared
+
+== Check lazy follow to lowerdata layer ==
tests/overlay/089 new file +272 -0

272 lines not on this page · raw message

tests/overlay/089.out new file +5 -0
new file mode 100644
index 00000000..0c3eee71
@@ -0,0 +1,5 @@
+QA output created by 089
+
+== Check fsverity validation ==
+
+== Check fsverity require ==
Signature · 1 line
2.34.1
Reviewed-by on v1 3/3
#11 lore raw

Re: [PATCH 3/3] overlay/08[89]: add tests for data-only redirect with userxattr

Quoted 53 lines · Amir Goldstein

From: Miklos Szeredi <[email protected]>

New kernel feature (target release is v6.16) allows data-only redirect to be enabled without metacopy and redirect_dir turned on. This works with or without verity enabled.

Tests are done with the userxattr option, to verify that it will work in a user namespace.

---

 common/overlay        |  29 +++++
 tests/overlay/088     | 296 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/088.out |  39 ++++++
 tests/overlay/089     | 272 ++++++++++++++++++++++++++++++++++++++
 tests/overlay/089.out |   5 +
 5 files changed, 641 insertions(+)
 create mode 100755 tests/overlay/088
 create mode 100644 tests/overlay/088.out
 create mode 100755 tests/overlay/089
 create mode 100644 tests/overlay/089.out
common/overlay +16 -0
index 0be943b1..d02d40b1 100644
@@ -271,6 +271,22 @@ _require_scratch_overlay_lowerdir_add_layers()
 	_scratch_unmount
 }
 
+# Check kernel support for datadir+=<datadir> without "metacopy=on" option
+_require_scratch_overlay_datadir_without_metacopy()
+{
+	local lowerdir="$OVL_BASE_SCRATCH_MNT/$OVL_UPPER"
+	local datadir="$OVL_BASE_SCRATCH_MNT/$OVL_LOWER"
+
+	_scratch_mkfs > /dev/null 2>&1
+	_overlay_scratch_mount_opts \
+		-o"lowerdir+=$lowerdir,datadir+=$datadir" > /dev/null 2>&1 || \
+	        _notrun "overlay datadir+ without metacopy not supported on ${SCRATCH_DEV}"
+
+	_scratch_unmount
+
+}
+
+
 # Helper function to check underlying dirs of overlay filesystem
 _overlay_fsck_dirs()
 {

common/overlay _require_unionmount_testsuite() · line 497

Quoted 10 lines
common/overlay +7 -0
@@ -472,6 +488,19 @@ _require_unionmount_testsuite()
 		_notrun "newer version of unionmount testsuite required to support OVERLAY_MOUNT_OPTIONS."
 }
 
+# transform overlay xattrs (trusted.overlay -> user.overlay)
+_overlay_trusted_to_user()
+{
+	local dir=$1
+
+	for file in `find $dir`; do
+		_getfattr --absolute-names -d -m '^trusted.overlay.(redirect|metacopy)$' $file  | sed 's/^trusted/user/' | setfattr --restore=-
                                                                                                                           ^^^^^^^^

common/overlay _require_unionmount_testsuite() · line 499

Quoted 2 lines
common/overlay +2 -0
+		for xattr in `_getfattr --absolute-names -d -m '^trusted.overlay.' $file  | tail -n +2 | cut -d= -f1`; do
+			setfattr -x $xattr $file;
                        ^^^^^^^^
                      $SETFATTR_PROG

common/overlay _require_unionmount_testsuite() · line 502

Quoted 3 lines
common/overlay +3 -0
+		done
+	done
+}

So o/088 and o/089 need _require_attrs trusted? And they all belong to "attr" test group.

Others look good to me, and test passed.

If you agree with above, I'll merge this patch with these changes.

Quoted 13 lines
common/overlay +1 -0
+
 _unionmount_testsuite_run()
 {
 	[ "$FSTYP" = overlay ] || \
tests/overlay/088 new file +3 -0
new file mode 100755
index 00000000..c774e816
@@ -0,0 +1,296 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2018 Red Hat, Inc. All Rights Reserved.

tests/overlay/088 line 13

Quoted 10 lines
tests/overlay/088 new file +10 -0
+# Copyright (C) 2023 CTERA Networks. All Rights Reserved.
+#
+# FS QA Test No. 088
+#
+# Test data-only layers functionality.
+# This is a variant of test overlay/085 with userxattr and without
+# redirect_dir/metacopy options
+#
+. ./common/preamble
+_begin_fstest auto quick metacopy redirect prealloc
                                                       ^^^^
                                                       attr
Quoted 337 lines
tests/overlay/088 new file +283 -0
+
+# Import common functions.
+. ./common/filter
+. ./common/attr
+
+# We use non-default scratch underlying overlay dirs, we need to check
+# them explicity after test.
+_require_scratch_nocheck
+_require_scratch_overlay_features redirect_dir metacopy
+_require_scratch_overlay_lowerdir_add_layers
+_require_scratch_overlay_datadir_without_metacopy
+_require_xfs_io_command "falloc"
+
+# remove all files from previous tests
+_scratch_mkfs
+
+# File size on lower
+dataname="datafile"
+sharedname="shared"
+datacontent="data"
+dataname2="datafile2"
+datacontent2="data2"
+datasize="4096"
+
+# Number of blocks allocated by filesystem on lower. Will be queried later.
+datarblocks=""
+datarblocksize=""
+estimated_datablocks=""
+
+udirname="pureupper"
+ufile="upperfile"
+
+
+# Check redirect xattr
+check_redirect()
+{
+	local target=$1
+	local expect=$2
+
+	value=$(_getfattr --absolute-names --only-values -n \
+		user.overlay.redirect $target)
+
+	[[ "$value" == "$expect" ]] || echo "Redirect xattr incorrect. Expected=\"$expect\", actual=\"$value\""
+}
+
+# Check size
+check_file_size()
+{
+	local target=$1 expected_size=$2 actual_size
+
+	actual_size=$(_get_filesize $target)
+
+	[ "$actual_size" == "$expected_size" ] || echo "Expected file size $expected_size but actual size is $actual_size"
+}
+
+check_file_blocks()
+{
+	local target=$1 expected_blocks=$2 nr_blocks
+
+	nr_blocks=$(stat -c "%b" $target)
+
+	[ "$nr_blocks" == "$expected_blocks" ] || echo "Expected $expected_blocks blocks but actual number of blocks is ${nr_blocks}."
+}
+
+check_file_contents()
+{
+	local target=$1 expected=$2
+	local actual target_f
+
+	target_f=`echo $target | _filter_scratch`
+
+	read actual<$target
+
+	[ "$actual" == "$expected" ] || echo "Expected file $target_f contents to be \"$expected\" but actual contents are \"$actual\""
+}
+
+check_no_file_contents()
+{
+	local target=$1
+	local actual target_f out_f
+
+	target_f=`echo $target | _filter_scratch`
+	out_f=`cat $target 2>&1 | _filter_scratch`
+	msg="cat: $target_f: No such file or directory"
+
+	[ "$out_f" == "$msg" ] && return
+
+	echo "$target_f unexpectedly has content"
+}
+
+
+check_file_size_contents()
+{
+	local target=$1 expected_size=$2 expected_content=$3
+
+	check_file_size $target $expected_size
+	check_file_contents $target $expected_content
+}
+
+mount_overlay()
+{
+	local _lowerdir=$1 _datadir2=$2 _datadir=$3
+
+	_overlay_scratch_mount_opts \
+		-o"lowerdir+=$_lowerdir,datadir+=$_datadir2,datadir+=$_datadir" \
+		-o"upperdir=$upperdir,workdir=$workdir" \
+		-o userxattr
+}
+
+mount_ro_overlay()
+{
+	local _lowerdir=$1 _datadir2=$2 _datadir=$3
+
+	_overlay_scratch_mount_opts \
+		-o"lowerdir+=$_lowerdir,datadir+=$_datadir2,datadir+=$_datadir" \
+		-o userxattr
+}
+
+umount_overlay()
+{
+	$UMOUNT_PROG $SCRATCH_MNT
+}
+
+test_no_access()
+{
+	local _target=$1
+
+	mount_ro_overlay "$lowerdir" "$datadir2" "$datadir"
+
+	stat $SCRATCH_MNT/$_target >> $seqres.full 2>&1 || \
+		echo "No access to lowerdata layer $_target"
+
+	echo "Unmount and Mount rw"
+	umount_overlay
+	mount_overlay "$lowerdir" "$datadir2" "$datadir"
+	stat $SCRATCH_MNT/$_target >> $seqres.full 2>&1 || \
+		echo "No access to lowerdata layer $_target"
+	umount_overlay
+}
+
+test_common()
+{
+	local _lowerdir=$1 _datadir2=$2 _datadir=$3
+	local _target=$4 _size=$5 _blocks=$6 _data="$7"
+	local _redirect=$8
+
+	echo "Mount ro"
+	mount_ro_overlay $_lowerdir $_datadir2 $_datadir
+
+	# Check redirect xattr to lowerdata
+	[ -n "$_redirect" ] && check_redirect $lowerdir/$_target "$_redirect"
+
+	echo "check properties of copied up file $_target"
+	check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"
+	check_file_blocks $SCRATCH_MNT/$_target $_blocks
+
+	# Do a mount cycle and check size and contents again.
+	echo "Unmount and Mount rw"
+	umount_overlay
+	mount_overlay $_lowerdir $_datadir2 $_datadir
+	echo "check properties of copied up file $_target"
+	check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"
+	check_file_blocks $SCRATCH_MNT/$_target $_blocks
+
+	# Trigger copy up and check upper file properties.
+	chmod 400 $SCRATCH_MNT/$_target
+	umount_overlay
+	check_file_size_contents $upperdir/$_target $_size "$_data"
+}
+
+test_lazy()
+{
+	local _target=$1
+
+	mount_overlay "$lowerdir" "$datadir2" "$datadir"
+
+	# Metadata should be valid
+	check_file_size $SCRATCH_MNT/$_target $datasize
+	check_file_blocks $SCRATCH_MNT/$_target $estimated_datablocks
+
+	# But have no content
+	check_no_file_contents $SCRATCH_MNT/$_target
+
+	umount_overlay
+}
+
+create_basic_files()
+{
+	_scratch_mkfs
+	mkdir -p $datadir/subdir $datadir2/subdir $lowerdir $lowerdir2 $upperdir $workdir $workdir2
+	mkdir -p $upperdir/$udirname
+	echo "$datacontent" > $datadir/$dataname
+	chmod 600 $datadir/$dataname
+	echo "$datacontent2" > $datadir2/$dataname2
+	chmod 600 $datadir2/$dataname2
+
+	echo "$datacontent" > $datadir/$sharedname
+	echo "$datacontent2" > $datadir2/$sharedname
+	chmod 600 $datadir/$sharedname  $datadir2/$sharedname
+
+	# Create files of size datasize.
+	for f in $datadir/$dataname $datadir2/$dataname2 $datadir/$sharedname $datadir2/$sharedname; do
+		$XFS_IO_PROG -c "falloc 0 $datasize" $f
+		$XFS_IO_PROG -c "fsync" $f
+	done
+
+	# Query number of block
+	datablocks=$(stat -c "%b" $datadir/$dataname)
+
+	# For lazy lookup file the block count is estimated based on size and block size
+	datablocksize=$(stat -c "%B" $datadir/$dataname)
+	estimated_datablocks=$(( ($datasize + $datablocksize - 1)/$datablocksize ))
+}
+
+prepare_midlayer()
+{
+	local _redirect=$1
+
+	_scratch_mkfs
+	create_basic_files
+	if [ -n "$_redirect" ]; then
+		mv "$datadir/$dataname" "$datadir/$_redirect"
+		mv "$datadir2/$dataname2" "$datadir2/$_redirect.2"
+		mv "$datadir/$sharedname" "$datadir/$_redirect.shared"
+		mv "$datadir2/$sharedname" "$datadir2/$_redirect.shared"
+	fi
+	# Create midlayer
+	_overlay_scratch_mount_dirs $datadir2:$datadir $lowerdir $workdir2 -o redirect_dir=on,index=on,metacopy=on
+	# Trigger a metacopy with or without redirect
+	if [ -n "$_redirect" ]; then
+		mv "$SCRATCH_MNT/$_redirect" "$SCRATCH_MNT/$dataname"
+		mv "$SCRATCH_MNT/$_redirect.2" "$SCRATCH_MNT/$dataname2"
+		mv "$SCRATCH_MNT/$_redirect.shared" "$SCRATCH_MNT/$sharedname"
+	else
+		chmod 400 $SCRATCH_MNT/$dataname
+		chmod 400 $SCRATCH_MNT/$dataname2
+		chmod 400 $SCRATCH_MNT/$sharedname
+	fi
+	umount_overlay
+
+	_overlay_trusted_to_user $lowerdir
+}
+
+# Create test directories
+datadir=$OVL_BASE_SCRATCH_MNT/data
+datadir2=$OVL_BASE_SCRATCH_MNT/data2
+lowerdir=$OVL_BASE_SCRATCH_MNT/lower
+upperdir=$OVL_BASE_SCRATCH_MNT/upper
+workdir=$OVL_BASE_SCRATCH_MNT/workdir
+workdir2=$OVL_BASE_SCRATCH_MNT/workdir2
+
+echo -e "\n== Check no follow to lowerdata layer without redirect =="
+prepare_midlayer
+test_no_access "$dataname"
+test_no_access "$dataname2"
+test_no_access "$sharedname"
+
+echo -e "\n== Check no follow to lowerdata layer with relative redirect =="
+prepare_midlayer "$dataname.renamed"
+test_no_access "$dataname"
+test_no_access "$dataname2"
+test_no_access "$sharedname"
+
+echo -e "\n== Check follow to lowerdata layer with absolute redirect =="
+prepare_midlayer "/subdir/$dataname"
+test_common "$lowerdir" "$datadir2" "$datadir" "$dataname" $datasize $datablocks \
+		"$datacontent" "/subdir/$dataname"
+test_common "$lowerdir" "$datadir2" "$datadir" "$dataname2" $datasize $datablocks \
+		"$datacontent2" "/subdir/$dataname.2"
+# Shared file should be picked from upper datadir
+test_common "$lowerdir" "$datadir2" "$datadir" "$sharedname" $datasize $datablocks \
+		"$datacontent2" "/subdir/$dataname.shared"
+
+echo -e "\n== Check lazy follow to lowerdata layer =="
+
+prepare_midlayer "/subdir/$dataname"
+rm $datadir/subdir/$dataname
+test_lazy $dataname
+
+
+# success, all done
+status=0
+exit
tests/overlay/088.out new file +39 -0
new file mode 100644
index 00000000..b587b874
@@ -0,0 +1,39 @@
+QA output created by 088
+
+== Check no follow to lowerdata layer without redirect ==
+No access to lowerdata layer datafile
+Unmount and Mount rw
+No access to lowerdata layer datafile
+No access to lowerdata layer datafile2
+Unmount and Mount rw
+No access to lowerdata layer datafile2
+No access to lowerdata layer shared
+Unmount and Mount rw
+No access to lowerdata layer shared
+
+== Check no follow to lowerdata layer with relative redirect ==
+No access to lowerdata layer datafile
+Unmount and Mount rw
+No access to lowerdata layer datafile
+No access to lowerdata layer datafile2
+Unmount and Mount rw
+No access to lowerdata layer datafile2
+No access to lowerdata layer shared
+Unmount and Mount rw
+No access to lowerdata layer shared
+
+== Check follow to lowerdata layer with absolute redirect ==
+Mount ro
+check properties of copied up file datafile
+Unmount and Mount rw
+check properties of copied up file datafile
+Mount ro
+check properties of copied up file datafile2
+Unmount and Mount rw
+check properties of copied up file datafile2
+Mount ro
+check properties of copied up file shared
+Unmount and Mount rw
+check properties of copied up file shared
+
+== Check lazy follow to lowerdata layer ==
tests/overlay/089 new file +3 -0
new file mode 100755
index 00000000..2259f917
@@ -0,0 +1,272 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2023 Red Hat, Inc. All Rights Reserved.

tests/overlay/089 line 13

Quoted 10 lines
tests/overlay/089 new file +10 -0
+# Copyright (C) 2023 CTERA Networks. All Rights Reserved.
+#
+# FS QA Test No. 089
+#
+# Test fs-verity functionallity
+# This is a variant of test overlay/080 with userxattr and without
+# redirect_dir/metacopy options
+#
+. ./common/preamble
+_begin_fstest auto quick metacopy redirect verity
                                                       ^^^^
                                                       attr
Quoted 273 lines
tests/overlay/089 new file +259 -0
+
+# Import common functions.
+. ./common/filter
+. ./common/attr
+. ./common/verity
+
+# We use non-default scratch underlying overlay dirs, we need to check
+# them explicity after test.
+_require_scratch_nocheck
+_require_scratch_overlay_features redirect_dir metacopy
+_require_scratch_overlay_lowerdata_layers
+_require_scratch_overlay_datadir_without_metacopy
+_require_scratch_overlay_verity
+
+# remove all files from previous tests
+_scratch_mkfs
+
+verityname="verityfile"
+noverityname="noverityfile"
+wrongverityname="wrongverityfile"
+missingverityname="missingverityfile"
+lowerdata="data1"
+lowerdata2="data2"
+lowerdata3="data3"
+lowerdata4="data4"
+lowersize="5"
+
+# Create test directories
+lowerdir=$OVL_BASE_SCRATCH_MNT/lower
+lowerdir2=$OVL_BASE_SCRATCH_MNT/lower2
+upperdir=$OVL_BASE_SCRATCH_MNT/upper
+workdir=$OVL_BASE_SCRATCH_MNT/workdir
+workdir2=$OVL_BASE_SCRATCH_MNT/workdir2
+
+# Check metacopy xattr
+check_metacopy()
+{
+	local target=$1 exist=$2 dataonlybase=$3
+	local out_f target_f
+	local msg
+
+	out_f=$( { _getfattr --absolute-names --only-values -n \
+		"user.overlay.metacopy" $target 2>&3 | od -A n -t x1 -w256 ; } 3>&1 | _filter_scratch)
+        has_version0=`echo $out_f | awk 'NR==1{print $1 == 0}'`
+
+	if [ "$exist" == "y" ];then
+		[ "$out_f" == "" -o "$has_version0" == "1" ] && return
+		echo "Metacopy xattr does not exist on ${target}. stdout=$out_f"
+		return
+	fi
+
+	if [ "$out_f" == ""  -o "$has_version0" == "1" ];then
+		echo "Metacopy xattr exists on ${target} unexpectedly."
+		return
+	fi
+
+	target_f=`echo $target | _filter_scratch`
+	msg="$target_f: user.overlay.metacopy: No such attribute"
+
+	[ "$out_f" == "$msg" ] && return
+
+	echo "Error while checking xattr on ${target}. stdout=$out"
+}
+
+# Check verity set in metacopy
+check_verity()
+{
+	local target=$1 exist=$2
+	local out_f target_f
+	local msg
+
+	out_f=$( { _getfattr --absolute-names --only-values -n "user.overlay.metacopy" $target 2>&3 | od -A n -t x1 -w256 ; } 3>&1 | _filter_scratch)
+
+	target_f=`echo $target | _filter_scratch`
+	msg="$target_f: user.overlay.metacopy: No such attribute"
+	has_digest=`echo $out_f | awk 'NR==1{print $4 == 1}'`
+
+	if [ "$exist" == "y" ]; then
+		[ "$out_f" == "$msg" -o "$has_digest" == "0" ] && echo "No verity on ${target}. stdout=$out_f"
+		return
+	fi
+
+	[ "$out_f" == "$msg" -o "$has_digest" == "0" ] && return
+	echo "Verity xattr exists on ${target} unexpectedly. stdout=$out_f"
+}
+
+# Check redirect xattr
+check_redirect()
+{
+	local target=$1
+	local expect=$2
+
+	value=$(_getfattr --absolute-names --only-values -n \
+		"user.overlay.redirect" $target)
+
+	[[ "$value" == "$expect" ]] || echo "Redirect xattr incorrect. Expected=\"$expect\", actual=\"$value\""
+}
+
+# Check size
+check_file_size()
+{
+	local target=$1 expected_size=$2 actual_size
+
+	actual_size=$(_get_filesize $target)
+
+	[ "$actual_size" == "$expected_size" ] || echo "Expected file size of $target $expected_size but actual size is $actual_size"
+}
+
+check_file_contents()
+{
+	local target=$1 expected=$2
+	local actual target_f
+
+	target_f=`echo $target | _filter_scratch`
+
+	read actual<$target
+
+	[ "$actual" == "$expected" ] || echo "Expected file $target_f contents to be \"$expected\" but actual contents are \"$actual\""
+}
+
+check_file_size_contents()
+{
+	local target=$1 expected_size=$2 expected_content=$3
+
+	check_file_size $target $expected_size
+	check_file_contents $target $expected_content
+}
+
+check_io_error()
+{
+	local target=$1
+	local actual target_f out_f
+
+	target_f=`echo $target | _filter_scratch`
+	out_f=`cat $target 2>&1 | _filter_scratch`
+	msg="cat: $target_f: Input/output error"
+
+	[ "$out_f" == "$msg" ] && return
+
+	echo "$target_f unexpectedly has no I/O error"
+}
+
+create_basic_files()
+{
+	local subdir=$1
+
+	_scratch_mkfs
+	mkdir -p $lowerdir $lowerdir2 $upperdir $workdir $workdir2
+
+	if [ "$subdir" != "" ]; then
+	    mkdir $lowerdir/$subdir
+	fi
+
+	echo -n "$lowerdata" > $lowerdir/$subdir$verityname
+	echo -n "$lowerdata2" > $lowerdir/$subdir$noverityname
+	echo -n "$lowerdata3" > $lowerdir/$subdir$wrongverityname
+	echo -n "$lowerdata4" > $lowerdir/$subdir$missingverityname
+
+	for f in $verityname $noverityname $wrongverityname $missingverityname; do
+		chmod 600 $lowerdir/$subdir$f
+
+		if [ "$f" != "$noverityname" ]; then
+			_fsv_enable $lowerdir/$subdir$f
+		fi
+        done
+}
+
+prepare_midlayer()
+{
+	subdir="base/"
+
+	create_basic_files "$subdir"
+	# Create midlayer
+	_overlay_scratch_mount_dirs $lowerdir $lowerdir2 $workdir2 -o redirect_dir=on,index=on,verity=on,metacopy=on
+	for f in $verityname $noverityname $wrongverityname $missingverityname; do
+		mv $SCRATCH_MNT/base/$f $SCRATCH_MNT/$f
+	done
+	umount_overlay
+
+	_overlay_trusted_to_user $lowerdir2
+
+	rm -rf $lowerdir2/base
+
+	for f in $verityname $noverityname $wrongverityname $missingverityname; do
+		# Ensure we have right metacopy and verity xattrs
+		check_metacopy $lowerdir2/$f "y"
+
+		if [ "$f" == "$noverityname" ]; then
+		    check_verity $lowerdir2/$f "n"
+		else
+		    check_verity $lowerdir2/$f "y"
+		fi
+
+		check_redirect $lowerdir2/$f "/base/$f"
+
+		check_file_size_contents $lowerdir2/$f $lowersize ""
+	done
+
+	# Fixup missing and wrong verity in lowerdir
+	rm -f $lowerdir/$subdir$wrongverityname $lowerdir/$subdir$missingverityname
+	echo -n "changed" > $lowerdir/$subdir$wrongverityname
+	_fsv_enable $lowerdir/$subdir$wrongverityname
+	echo "$lowerdata4" > $lowerdir/$subdir$missingverityname
+}
+
+test_common()
+{
+	local verity=$1
+
+	mount_overlay "$lowerdir2::$lowerdir" $verity
+
+	check_file_size_contents $SCRATCH_MNT/$verityname $lowersize "$lowerdata"
+
+	if [ "$verity" == "require" ]; then
+		check_io_error $SCRATCH_MNT/$noverityname
+	else
+		check_file_size_contents $SCRATCH_MNT/$noverityname $lowersize "$lowerdata2"
+	fi
+
+	if [ "$verity" == "off" ]; then
+		check_file_size_contents $SCRATCH_MNT/$wrongverityname $lowersize "changed"
+		check_file_size_contents $SCRATCH_MNT/$missingverityname $lowersize "$lowerdata4"
+	else
+		check_io_error $SCRATCH_MNT/$missingverityname
+		check_io_error $SCRATCH_MNT/$wrongverityname
+	fi
+
+	umount_overlay
+}
+
+mount_overlay()
+{
+	local _lowerdir=$1
+	local _verity=$2
+
+	_overlay_scratch_mount_dirs "$_lowerdir" $upperdir $workdir -o userxattr,verity=$_verity
+}
+
+umount_overlay()
+{
+	$UMOUNT_PROG $SCRATCH_MNT
+}
+
+
+echo -e "\n== Check fsverity validation =="
+
+prepare_midlayer
+test_common "off"
+prepare_midlayer
+test_common "on"
+
+echo -e "\n== Check fsverity require =="
+
+prepare_midlayer
+test_common "require"
+
+# success, all done
+status=0
+exit
tests/overlay/089.out new file +5 -0
new file mode 100644
index 00000000..0c3eee71
@@ -0,0 +1,5 @@
+QA output created by 089
+
+== Check fsverity validation ==
+
+== Check fsverity require ==
Signature · 1 line
2.34.1
#12 lore raw

Re: [PATCH 3/3] overlay/08[89]: add tests for data-only redirect with userxattr

Quoted 68 lines · Zorro Lang
Quoted 53 lines · Amir Goldstein

From: Miklos Szeredi <[email protected]>

New kernel feature (target release is v6.16) allows data-only redirect to be enabled without metacopy and redirect_dir turned on. This works with or without verity enabled.

Tests are done with the userxattr option, to verify that it will work in a user namespace.

---

 common/overlay        |  29 +++++
 tests/overlay/088     | 296 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/088.out |  39 ++++++
 tests/overlay/089     | 272 ++++++++++++++++++++++++++++++++++++++
 tests/overlay/089.out |   5 +
 5 files changed, 641 insertions(+)
 create mode 100755 tests/overlay/088
 create mode 100644 tests/overlay/088.out
 create mode 100755 tests/overlay/089
 create mode 100644 tests/overlay/089.out
common/overlay +16 -0
index 0be943b1..d02d40b1 100644
@@ -271,6 +271,22 @@ _require_scratch_overlay_lowerdir_add_layers()
      _scratch_unmount
 }
 
+# Check kernel support for datadir+=<datadir> without "metacopy=on" option
+_require_scratch_overlay_datadir_without_metacopy()
+{
+     local lowerdir="$OVL_BASE_SCRATCH_MNT/$OVL_UPPER"
+     local datadir="$OVL_BASE_SCRATCH_MNT/$OVL_LOWER"
+
+     _scratch_mkfs > /dev/null 2>&1
+     _overlay_scratch_mount_opts \
+             -o"lowerdir+=$lowerdir,datadir+=$datadir" > /dev/null 2>&1 || \
+             _notrun "overlay datadir+ without metacopy not supported on ${SCRATCH_DEV}"
+
+     _scratch_unmount
+
+}
+
+
 # Helper function to check underlying dirs of overlay filesystem
 _overlay_fsck_dirs()
 {

common/overlay _require_unionmount_testsuite() · line 497

Quoted 10 lines
common/overlay +7 -0
@@ -472,6 +488,19 @@ _require_unionmount_testsuite()
              _notrun "newer version of unionmount testsuite required to support OVERLAY_MOUNT_OPTIONS."
 }
 
+# transform overlay xattrs (trusted.overlay -> user.overlay)
+_overlay_trusted_to_user()
+{
+     local dir=$1
+
+     for file in `find $dir`; do
+             _getfattr --absolute-names -d -m '^trusted.overlay.(redirect|metacopy)$' $file  | sed 's/^trusted/user/' | setfattr --restore=-
                                                                                                                           ^^^^^^^^

common/overlay _require_unionmount_testsuite() · line 499

Quoted 2 lines
common/overlay +2 -0
+             for xattr in `_getfattr --absolute-names -d -m '^trusted.overlay.' $file  | tail -n +2 | cut -d= -f1`; do
+                     setfattr -x $xattr $file;
Quoted 8 lines
                        ^^^^^^^^
                      $SETFATTR_PROG

common/overlay _require_unionmount_testsuite() · line 502

Quoted 3 lines
common/overlay +3 -0
+             done
+     done
+}

So o/088 and o/089 need _require_attrs trusted? And they all belong to "attr" test group.

Sure, fine by me.

Thanks,
Amir.

Quoted 655 lines

Others look good to me, and test passed.

If you agree with above, I'll merge this patch with these changes.

Quoted 13 lines
common/overlay +1 -0
+
 _unionmount_testsuite_run()
 {
      [ "$FSTYP" = overlay ] || \
tests/overlay/088 new file +3 -0
new file mode 100755
index 00000000..c774e816
@@ -0,0 +1,296 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2018 Red Hat, Inc. All Rights Reserved.

tests/overlay/088 line 13

Quoted 10 lines
tests/overlay/088 new file +10 -0
+# Copyright (C) 2023 CTERA Networks. All Rights Reserved.
+#
+# FS QA Test No. 088
+#
+# Test data-only layers functionality.
+# This is a variant of test overlay/085 with userxattr and without
+# redirect_dir/metacopy options
+#
+. ./common/preamble
+_begin_fstest auto quick metacopy redirect prealloc
                                                       ^^^^
                                                       attr
Quoted 337 lines
tests/overlay/088 new file +283 -0
+
+# Import common functions.
+. ./common/filter
+. ./common/attr
+
+# We use non-default scratch underlying overlay dirs, we need to check
+# them explicity after test.
+_require_scratch_nocheck
+_require_scratch_overlay_features redirect_dir metacopy
+_require_scratch_overlay_lowerdir_add_layers
+_require_scratch_overlay_datadir_without_metacopy
+_require_xfs_io_command "falloc"
+
+# remove all files from previous tests
+_scratch_mkfs
+
+# File size on lower
+dataname="datafile"
+sharedname="shared"
+datacontent="data"
+dataname2="datafile2"
+datacontent2="data2"
+datasize="4096"
+
+# Number of blocks allocated by filesystem on lower. Will be queried later.
+datarblocks=""
+datarblocksize=""
+estimated_datablocks=""
+
+udirname="pureupper"
+ufile="upperfile"
+
+
+# Check redirect xattr
+check_redirect()
+{
+     local target=$1
+     local expect=$2
+
+     value=$(_getfattr --absolute-names --only-values -n \
+             user.overlay.redirect $target)
+
+     [[ "$value" == "$expect" ]] || echo "Redirect xattr incorrect. Expected=\"$expect\", actual=\"$value\""
+}
+
+# Check size
+check_file_size()
+{
+     local target=$1 expected_size=$2 actual_size
+
+     actual_size=$(_get_filesize $target)
+
+     [ "$actual_size" == "$expected_size" ] || echo "Expected file size $expected_size but actual size is $actual_size"
+}
+
+check_file_blocks()
+{
+     local target=$1 expected_blocks=$2 nr_blocks
+
+     nr_blocks=$(stat -c "%b" $target)
+
+     [ "$nr_blocks" == "$expected_blocks" ] || echo "Expected $expected_blocks blocks but actual number of blocks is ${nr_blocks}."
+}
+
+check_file_contents()
+{
+     local target=$1 expected=$2
+     local actual target_f
+
+     target_f=`echo $target | _filter_scratch`
+
+     read actual<$target
+
+     [ "$actual" == "$expected" ] || echo "Expected file $target_f contents to be \"$expected\" but actual contents are \"$actual\""
+}
+
+check_no_file_contents()
+{
+     local target=$1
+     local actual target_f out_f
+
+     target_f=`echo $target | _filter_scratch`
+     out_f=`cat $target 2>&1 | _filter_scratch`
+     msg="cat: $target_f: No such file or directory"
+
+     [ "$out_f" == "$msg" ] && return
+
+     echo "$target_f unexpectedly has content"
+}
+
+
+check_file_size_contents()
+{
+     local target=$1 expected_size=$2 expected_content=$3
+
+     check_file_size $target $expected_size
+     check_file_contents $target $expected_content
+}
+
+mount_overlay()
+{
+     local _lowerdir=$1 _datadir2=$2 _datadir=$3
+
+     _overlay_scratch_mount_opts \
+             -o"lowerdir+=$_lowerdir,datadir+=$_datadir2,datadir+=$_datadir" \
+             -o"upperdir=$upperdir,workdir=$workdir" \
+             -o userxattr
+}
+
+mount_ro_overlay()
+{
+     local _lowerdir=$1 _datadir2=$2 _datadir=$3
+
+     _overlay_scratch_mount_opts \
+             -o"lowerdir+=$_lowerdir,datadir+=$_datadir2,datadir+=$_datadir" \
+             -o userxattr
+}
+
+umount_overlay()
+{
+     $UMOUNT_PROG $SCRATCH_MNT
+}
+
+test_no_access()
+{
+     local _target=$1
+
+     mount_ro_overlay "$lowerdir" "$datadir2" "$datadir"
+
+     stat $SCRATCH_MNT/$_target >> $seqres.full 2>&1 || \
+             echo "No access to lowerdata layer $_target"
+
+     echo "Unmount and Mount rw"
+     umount_overlay
+     mount_overlay "$lowerdir" "$datadir2" "$datadir"
+     stat $SCRATCH_MNT/$_target >> $seqres.full 2>&1 || \
+             echo "No access to lowerdata layer $_target"
+     umount_overlay
+}
+
+test_common()
+{
+     local _lowerdir=$1 _datadir2=$2 _datadir=$3
+     local _target=$4 _size=$5 _blocks=$6 _data="$7"
+     local _redirect=$8
+
+     echo "Mount ro"
+     mount_ro_overlay $_lowerdir $_datadir2 $_datadir
+
+     # Check redirect xattr to lowerdata
+     [ -n "$_redirect" ] && check_redirect $lowerdir/$_target "$_redirect"
+
+     echo "check properties of copied up file $_target"
+     check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"
+     check_file_blocks $SCRATCH_MNT/$_target $_blocks
+
+     # Do a mount cycle and check size and contents again.
+     echo "Unmount and Mount rw"
+     umount_overlay
+     mount_overlay $_lowerdir $_datadir2 $_datadir
+     echo "check properties of copied up file $_target"
+     check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"
+     check_file_blocks $SCRATCH_MNT/$_target $_blocks
+
+     # Trigger copy up and check upper file properties.
+     chmod 400 $SCRATCH_MNT/$_target
+     umount_overlay
+     check_file_size_contents $upperdir/$_target $_size "$_data"
+}
+
+test_lazy()
+{
+     local _target=$1
+
+     mount_overlay "$lowerdir" "$datadir2" "$datadir"
+
+     # Metadata should be valid
+     check_file_size $SCRATCH_MNT/$_target $datasize
+     check_file_blocks $SCRATCH_MNT/$_target $estimated_datablocks
+
+     # But have no content
+     check_no_file_contents $SCRATCH_MNT/$_target
+
+     umount_overlay
+}
+
+create_basic_files()
+{
+     _scratch_mkfs
+     mkdir -p $datadir/subdir $datadir2/subdir $lowerdir $lowerdir2 $upperdir $workdir $workdir2
+     mkdir -p $upperdir/$udirname
+     echo "$datacontent" > $datadir/$dataname
+     chmod 600 $datadir/$dataname
+     echo "$datacontent2" > $datadir2/$dataname2
+     chmod 600 $datadir2/$dataname2
+
+     echo "$datacontent" > $datadir/$sharedname
+     echo "$datacontent2" > $datadir2/$sharedname
+     chmod 600 $datadir/$sharedname  $datadir2/$sharedname
+
+     # Create files of size datasize.
+     for f in $datadir/$dataname $datadir2/$dataname2 $datadir/$sharedname $datadir2/$sharedname; do
+             $XFS_IO_PROG -c "falloc 0 $datasize" $f
+             $XFS_IO_PROG -c "fsync" $f
+     done
+
+     # Query number of block
+     datablocks=$(stat -c "%b" $datadir/$dataname)
+
+     # For lazy lookup file the block count is estimated based on size and block size
+     datablocksize=$(stat -c "%B" $datadir/$dataname)
+     estimated_datablocks=$(( ($datasize + $datablocksize - 1)/$datablocksize ))
+}
+
+prepare_midlayer()
+{
+     local _redirect=$1
+
+     _scratch_mkfs
+     create_basic_files
+     if [ -n "$_redirect" ]; then
+             mv "$datadir/$dataname" "$datadir/$_redirect"
+             mv "$datadir2/$dataname2" "$datadir2/$_redirect.2"
+             mv "$datadir/$sharedname" "$datadir/$_redirect.shared"
+             mv "$datadir2/$sharedname" "$datadir2/$_redirect.shared"
+     fi
+     # Create midlayer
+     _overlay_scratch_mount_dirs $datadir2:$datadir $lowerdir $workdir2 -o redirect_dir=on,index=on,metacopy=on
+     # Trigger a metacopy with or without redirect
+     if [ -n "$_redirect" ]; then
+             mv "$SCRATCH_MNT/$_redirect" "$SCRATCH_MNT/$dataname"
+             mv "$SCRATCH_MNT/$_redirect.2" "$SCRATCH_MNT/$dataname2"
+             mv "$SCRATCH_MNT/$_redirect.shared" "$SCRATCH_MNT/$sharedname"
+     else
+             chmod 400 $SCRATCH_MNT/$dataname
+             chmod 400 $SCRATCH_MNT/$dataname2
+             chmod 400 $SCRATCH_MNT/$sharedname
+     fi
+     umount_overlay
+
+     _overlay_trusted_to_user $lowerdir
+}
+
+# Create test directories
+datadir=$OVL_BASE_SCRATCH_MNT/data
+datadir2=$OVL_BASE_SCRATCH_MNT/data2
+lowerdir=$OVL_BASE_SCRATCH_MNT/lower
+upperdir=$OVL_BASE_SCRATCH_MNT/upper
+workdir=$OVL_BASE_SCRATCH_MNT/workdir
+workdir2=$OVL_BASE_SCRATCH_MNT/workdir2
+
+echo -e "\n== Check no follow to lowerdata layer without redirect =="
+prepare_midlayer
+test_no_access "$dataname"
+test_no_access "$dataname2"
+test_no_access "$sharedname"
+
+echo -e "\n== Check no follow to lowerdata layer with relative redirect =="
+prepare_midlayer "$dataname.renamed"
+test_no_access "$dataname"
+test_no_access "$dataname2"
+test_no_access "$sharedname"
+
+echo -e "\n== Check follow to lowerdata layer with absolute redirect =="
+prepare_midlayer "/subdir/$dataname"
+test_common "$lowerdir" "$datadir2" "$datadir" "$dataname" $datasize $datablocks \
+             "$datacontent" "/subdir/$dataname"
+test_common "$lowerdir" "$datadir2" "$datadir" "$dataname2" $datasize $datablocks \
+             "$datacontent2" "/subdir/$dataname.2"
+# Shared file should be picked from upper datadir
+test_common "$lowerdir" "$datadir2" "$datadir" "$sharedname" $datasize $datablocks \
+             "$datacontent2" "/subdir/$dataname.shared"
+
+echo -e "\n== Check lazy follow to lowerdata layer =="
+
+prepare_midlayer "/subdir/$dataname"
+rm $datadir/subdir/$dataname
+test_lazy $dataname
+
+
+# success, all done
+status=0
+exit
tests/overlay/088.out new file +39 -0
new file mode 100644
index 00000000..b587b874
@@ -0,0 +1,39 @@
+QA output created by 088
+
+== Check no follow to lowerdata layer without redirect ==
+No access to lowerdata layer datafile
+Unmount and Mount rw
+No access to lowerdata layer datafile
+No access to lowerdata layer datafile2
+Unmount and Mount rw
+No access to lowerdata layer datafile2
+No access to lowerdata layer shared
+Unmount and Mount rw
+No access to lowerdata layer shared
+
+== Check no follow to lowerdata layer with relative redirect ==
+No access to lowerdata layer datafile
+Unmount and Mount rw
+No access to lowerdata layer datafile
+No access to lowerdata layer datafile2
+Unmount and Mount rw
+No access to lowerdata layer datafile2
+No access to lowerdata layer shared
+Unmount and Mount rw
+No access to lowerdata layer shared
+
+== Check follow to lowerdata layer with absolute redirect ==
+Mount ro
+check properties of copied up file datafile
+Unmount and Mount rw
+check properties of copied up file datafile
+Mount ro
+check properties of copied up file datafile2
+Unmount and Mount rw
+check properties of copied up file datafile2
+Mount ro
+check properties of copied up file shared
+Unmount and Mount rw
+check properties of copied up file shared
+
+== Check lazy follow to lowerdata layer ==
tests/overlay/089 new file +3 -0
new file mode 100755
index 00000000..2259f917
@@ -0,0 +1,272 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2023 Red Hat, Inc. All Rights Reserved.

tests/overlay/089 line 13

Quoted 10 lines
tests/overlay/089 new file +10 -0
+# Copyright (C) 2023 CTERA Networks. All Rights Reserved.
+#
+# FS QA Test No. 089
+#
+# Test fs-verity functionallity
+# This is a variant of test overlay/080 with userxattr and without
+# redirect_dir/metacopy options
+#
+. ./common/preamble
+_begin_fstest auto quick metacopy redirect verity
                                                       ^^^^
                                                       attr
Quoted 273 lines
tests/overlay/089 new file +259 -0
+
+# Import common functions.
+. ./common/filter
+. ./common/attr
+. ./common/verity
+
+# We use non-default scratch underlying overlay dirs, we need to check
+# them explicity after test.
+_require_scratch_nocheck
+_require_scratch_overlay_features redirect_dir metacopy
+_require_scratch_overlay_lowerdata_layers
+_require_scratch_overlay_datadir_without_metacopy
+_require_scratch_overlay_verity
+
+# remove all files from previous tests
+_scratch_mkfs
+
+verityname="verityfile"
+noverityname="noverityfile"
+wrongverityname="wrongverityfile"
+missingverityname="missingverityfile"
+lowerdata="data1"
+lowerdata2="data2"
+lowerdata3="data3"
+lowerdata4="data4"
+lowersize="5"
+
+# Create test directories
+lowerdir=$OVL_BASE_SCRATCH_MNT/lower
+lowerdir2=$OVL_BASE_SCRATCH_MNT/lower2
+upperdir=$OVL_BASE_SCRATCH_MNT/upper
+workdir=$OVL_BASE_SCRATCH_MNT/workdir
+workdir2=$OVL_BASE_SCRATCH_MNT/workdir2
+
+# Check metacopy xattr
+check_metacopy()
+{
+     local target=$1 exist=$2 dataonlybase=$3
+     local out_f target_f
+     local msg
+
+     out_f=$( { _getfattr --absolute-names --only-values -n \
+             "user.overlay.metacopy" $target 2>&3 | od -A n -t x1 -w256 ; } 3>&1 | _filter_scratch)
+        has_version0=`echo $out_f | awk 'NR==1{print $1 == 0}'`
+
+     if [ "$exist" == "y" ];then
+             [ "$out_f" == "" -o "$has_version0" == "1" ] && return
+             echo "Metacopy xattr does not exist on ${target}. stdout=$out_f"
+             return
+     fi
+
+     if [ "$out_f" == ""  -o "$has_version0" == "1" ];then
+             echo "Metacopy xattr exists on ${target} unexpectedly."
+             return
+     fi
+
+     target_f=`echo $target | _filter_scratch`
+     msg="$target_f: user.overlay.metacopy: No such attribute"
+
+     [ "$out_f" == "$msg" ] && return
+
+     echo "Error while checking xattr on ${target}. stdout=$out"
+}
+
+# Check verity set in metacopy
+check_verity()
+{
+     local target=$1 exist=$2
+     local out_f target_f
+     local msg
+
+     out_f=$( { _getfattr --absolute-names --only-values -n "user.overlay.metacopy" $target 2>&3 | od -A n -t x1 -w256 ; } 3>&1 | _filter_scratch)
+
+     target_f=`echo $target | _filter_scratch`
+     msg="$target_f: user.overlay.metacopy: No such attribute"
+     has_digest=`echo $out_f | awk 'NR==1{print $4 == 1}'`
+
+     if [ "$exist" == "y" ]; then
+             [ "$out_f" == "$msg" -o "$has_digest" == "0" ] && echo "No verity on ${target}. stdout=$out_f"
+             return
+     fi
+
+     [ "$out_f" == "$msg" -o "$has_digest" == "0" ] && return
+     echo "Verity xattr exists on ${target} unexpectedly. stdout=$out_f"
+}
+
+# Check redirect xattr
+check_redirect()
+{
+     local target=$1
+     local expect=$2
+
+     value=$(_getfattr --absolute-names --only-values -n \
+             "user.overlay.redirect" $target)
+
+     [[ "$value" == "$expect" ]] || echo "Redirect xattr incorrect. Expected=\"$expect\", actual=\"$value\""
+}
+
+# Check size
+check_file_size()
+{
+     local target=$1 expected_size=$2 actual_size
+
+     actual_size=$(_get_filesize $target)
+
+     [ "$actual_size" == "$expected_size" ] || echo "Expected file size of $target $expected_size but actual size is $actual_size"
+}
+
+check_file_contents()
+{
+     local target=$1 expected=$2
+     local actual target_f
+
+     target_f=`echo $target | _filter_scratch`
+
+     read actual<$target
+
+     [ "$actual" == "$expected" ] || echo "Expected file $target_f contents to be \"$expected\" but actual contents are \"$actual\""
+}
+
+check_file_size_contents()
+{
+     local target=$1 expected_size=$2 expected_content=$3
+
+     check_file_size $target $expected_size
+     check_file_contents $target $expected_content
+}
+
+check_io_error()
+{
+     local target=$1
+     local actual target_f out_f
+
+     target_f=`echo $target | _filter_scratch`
+     out_f=`cat $target 2>&1 | _filter_scratch`
+     msg="cat: $target_f: Input/output error"
+
+     [ "$out_f" == "$msg" ] && return
+
+     echo "$target_f unexpectedly has no I/O error"
+}
+
+create_basic_files()
+{
+     local subdir=$1
+
+     _scratch_mkfs
+     mkdir -p $lowerdir $lowerdir2 $upperdir $workdir $workdir2
+
+     if [ "$subdir" != "" ]; then
+         mkdir $lowerdir/$subdir
+     fi
+
+     echo -n "$lowerdata" > $lowerdir/$subdir$verityname
+     echo -n "$lowerdata2" > $lowerdir/$subdir$noverityname
+     echo -n "$lowerdata3" > $lowerdir/$subdir$wrongverityname
+     echo -n "$lowerdata4" > $lowerdir/$subdir$missingverityname
+
+     for f in $verityname $noverityname $wrongverityname $missingverityname; do
+             chmod 600 $lowerdir/$subdir$f
+
+             if [ "$f" != "$noverityname" ]; then
+                     _fsv_enable $lowerdir/$subdir$f
+             fi
+        done
+}
+
+prepare_midlayer()
+{
+     subdir="base/"
+
+     create_basic_files "$subdir"
+     # Create midlayer
+     _overlay_scratch_mount_dirs $lowerdir $lowerdir2 $workdir2 -o redirect_dir=on,index=on,verity=on,metacopy=on
+     for f in $verityname $noverityname $wrongverityname $missingverityname; do
+             mv $SCRATCH_MNT/base/$f $SCRATCH_MNT/$f
+     done
+     umount_overlay
+
+     _overlay_trusted_to_user $lowerdir2
+
+     rm -rf $lowerdir2/base
+
+     for f in $verityname $noverityname $wrongverityname $missingverityname; do
+             # Ensure we have right metacopy and verity xattrs
+             check_metacopy $lowerdir2/$f "y"
+
+             if [ "$f" == "$noverityname" ]; then
+                 check_verity $lowerdir2/$f "n"
+             else
+                 check_verity $lowerdir2/$f "y"
+             fi
+
+             check_redirect $lowerdir2/$f "/base/$f"
+
+             check_file_size_contents $lowerdir2/$f $lowersize ""
+     done
+
+     # Fixup missing and wrong verity in lowerdir
+     rm -f $lowerdir/$subdir$wrongverityname $lowerdir/$subdir$missingverityname
+     echo -n "changed" > $lowerdir/$subdir$wrongverityname
+     _fsv_enable $lowerdir/$subdir$wrongverityname
+     echo "$lowerdata4" > $lowerdir/$subdir$missingverityname
+}
+
+test_common()
+{
+     local verity=$1
+
+     mount_overlay "$lowerdir2::$lowerdir" $verity
+
+     check_file_size_contents $SCRATCH_MNT/$verityname $lowersize "$lowerdata"
+
+     if [ "$verity" == "require" ]; then
+             check_io_error $SCRATCH_MNT/$noverityname
+     else
+             check_file_size_contents $SCRATCH_MNT/$noverityname $lowersize "$lowerdata2"
+     fi
+
+     if [ "$verity" == "off" ]; then
+             check_file_size_contents $SCRATCH_MNT/$wrongverityname $lowersize "changed"
+             check_file_size_contents $SCRATCH_MNT/$missingverityname $lowersize "$lowerdata4"
+     else
+             check_io_error $SCRATCH_MNT/$missingverityname
+             check_io_error $SCRATCH_MNT/$wrongverityname
+     fi
+
+     umount_overlay
+}
+
+mount_overlay()
+{
+     local _lowerdir=$1
+     local _verity=$2
+
+     _overlay_scratch_mount_dirs "$_lowerdir" $upperdir $workdir -o userxattr,verity=$_verity
+}
+
+umount_overlay()
+{
+     $UMOUNT_PROG $SCRATCH_MNT
+}
+
+
+echo -e "\n== Check fsverity validation =="
+
+prepare_midlayer
+test_common "off"
+prepare_midlayer
+test_common "on"
+
+echo -e "\n== Check fsverity require =="
+
+prepare_midlayer
+test_common "require"
+
+# success, all done
+status=0
+exit
tests/overlay/089.out new file +5 -1
new file mode 100644
index 00000000..0c3eee71
@@ -0,0 +1,5 @@
+QA output created by 089
+
+== Check fsverity validation ==
+
+== Check fsverity require ==
--

2.34.1

#13 lore raw
In reply to Amir Goldstein

Hi Amir,

Em 09/06/2025 12:19, Amir Goldstein escreveu:

Quoted 4 lines

Zorro,

Please find two new tests by Miklos to cover ian overlayfs feature merged to 6.16-rc1. Those tests do notrun on older kernels.

Using this patchset on top of 6.16-rc1 I still found that overlay/012 still doesn't work in my setup:

$ sudo FSTYPE=ext4 TEST_DIR=/tmp/dir1 TEST_DEV=/dev/vdb SCRATCH_DEV=/dev/vdc SCRATCH_MNT=/tmp/dir2 ./check -overlay overlay/012
...
    -rm: cannot remove 'SCRATCH_MNT/test': Stale file handle
    +rm: cannot remove 'SCRATCH_MNT/test': Is a directory

Here's a smaller reproducer for 012:

```
mkdir -p /tmp/dir2/ovl-lower/test /tmp/dir2/ovl-upper /tmp/dir2/ovl-work tmp/dir2/ovl-mnt

sudo mount -t overlay -olowerdir=/tmp/dir2/ovl-lower,upperdir=/tmp/dir2/ovl-upper,workdir=/tmp/dir2/ovl-work /tmp/dir2 /tmp/dir2/ovl-mnt

rmdir /tmp/dir2/ovl-mnt/test
touch /tmp/dir2/ovl-mnt/test
rm /tmp/dir2/ovl-upper/test
# rm /tmp/dir2/ovl-mnt/test (bug happens here)
```

Before executing the last line, if you run ls you can see that test is a file:

$ ls -la dir2/ovl-mnt/test
-rw-r--r-- 0 user user 0 Jun 10 17:54 dir2/ovl-mnt/test

After trying to remove the file, the previous ls command is empty now:

$ rm dir2/ovl-mnt/test
rm: cannot remove 'dir2/ovl-mnt/test': Is a directory
$ ls -la dir2/ovl-mnt/test
total 0
drwxr-xr-x 2 user user 40 Jun 10 17:54 .
drwxr-xr-x 1 user user 40 Jun 10 17:55 ..

But running it in a upper level shows the file as a directory:

$ ls -la dir2/ovl-mnt/
total 0
drwxr-xr-x 1 user user  40 Jun 10 17:55 .
drwxr-xr-x 6 user user 120 Jun 10 17:54 ..
drwxr-xr-x 2 user user  40 Jun 10 17:54 test
#14 lore raw
Quoted 49 lines · André Almeida

Hi Amir,

Em 09/06/2025 12:19, Amir Goldstein escreveu:

Quoted 4 lines

Zorro,

Please find two new tests by Miklos to cover ian overlayfs feature merged to 6.16-rc1. Those tests do notrun on older kernels.

Using this patchset on top of 6.16-rc1 I still found that overlay/012 still doesn't work in my setup:

$ sudo FSTYPE=ext4 TEST_DIR=/tmp/dir1 TEST_DEV=/dev/vdb
SCRATCH_DEV=/dev/vdc SCRATCH_MNT=/tmp/dir2 ./check -overlay overlay/012
...
     -rm: cannot remove 'SCRATCH_MNT/test': Stale file handle
     +rm: cannot remove 'SCRATCH_MNT/test': Is a directory

Here's a smaller reproducer for 012:

```
mkdir -p /tmp/dir2/ovl-lower/test /tmp/dir2/ovl-upper /tmp/dir2/ovl-work
tmp/dir2/ovl-mnt

sudo mount -t overlay
-olowerdir=/tmp/dir2/ovl-lower,upperdir=/tmp/dir2/ovl-upper,workdir=/tmp/dir2/ovl-work
/tmp/dir2 /tmp/dir2/ovl-mnt

rmdir /tmp/dir2/ovl-mnt/test
touch /tmp/dir2/ovl-mnt/test
rm /tmp/dir2/ovl-upper/test
# rm /tmp/dir2/ovl-mnt/test (bug happens here)
```

Before executing the last line, if you run ls you can see that test is a file:

$ ls -la dir2/ovl-mnt/test
-rw-r--r-- 0 user user 0 Jun 10 17:54 dir2/ovl-mnt/test

After trying to remove the file, the previous ls command is empty now:

$ rm dir2/ovl-mnt/test
rm: cannot remove 'dir2/ovl-mnt/test': Is a directory
$ ls -la dir2/ovl-mnt/test
total 0
drwxr-xr-x 2 user user 40 Jun 10 17:54 .
drwxr-xr-x 1 user user 40 Jun 10 17:55 ..
Quoted 9 lines

But running it in a upper level shows the file as a directory:

$ ls -la dir2/ovl-mnt/
total 0
drwxr-xr-x 1 user user  40 Jun 10 17:55 .
drwxr-xr-x 6 user user 120 Jun 10 17:54 ..
drwxr-xr-x 2 user user  40 Jun 10 17:54 test

This is a kernel 6.16-rc1 regression.

See https://lore.kernel.org/linux-fsdevel/[email protected]/

Thanks,
Amir.

#15 lore raw
In reply to Amir Goldstein
Quoted 4 lines · Amir Goldstein

Zorro,

Please find two new tests by Miklos to cover ian overlayfs feature merged to 6.16-rc1. Those tests do notrun on older kernels.

Thanks for the testing and feedback for older kernels :)

Quoted 3 lines

Adding two new helpers to sort out the shutdown test requiremetns w.r.t overlayfs following our discussion on generic/623 patch review [1].

Sure, I'll merge this version (after testing) as we talked.

Thanks,
Zorro

Quoted 31 lines

Thanks,
Amir.

[1] https://lore.kernel.org/fstests/[email protected]/

Amir Goldstein (2):
fstests: add helper _scratch_shutdown_and_syncfs
fstests: add helper _require_xfs_io_shutdown

Miklos Szeredi (1):
overlay/08[89]: add tests for data-only redirect with userxattr

 common/overlay        |  29 +++++
 common/rc             |  48 +++++++
 tests/generic/623     |   2 +-
 tests/overlay/087     |  13 +-
 tests/overlay/088     | 296 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/088.out |  39 ++++++
 tests/overlay/089     | 272 ++++++++++++++++++++++++++++++++++++++
 tests/overlay/089.out |   5 +
 tests/xfs/546         |   5 +-
 9 files changed, 695 insertions(+), 14 deletions(-)
 create mode 100755 tests/overlay/088
 create mode 100644 tests/overlay/088.out
 create mode 100755 tests/overlay/089
 create mode 100644 tests/overlay/089.out
Signature · 1 line
2.34.1