Skip to contentSkip to conversation

Overlayfs tests for 6.7-rc1

PATCH Superseded Amir Goldstein 2 participants 13 messages Dec 4, 2023 – Dec 11, 2023

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover Overlayfs tests for 6.7-rc1

Zorro,

This update contains 3 new overlayfs tests for new features added in v6.7-rc1.

overlay/084, written by Alexander, tests the new nested xattrs feature. overlay/{085,086} test the new lowerdir+,datadir+ mount options.

overlay/086 was partly forked from overlay/083, but overlay/083 is not sensitive to libmount version, because the escaped commas test is not related to any specific mount option, so it wasn't copied over.

All the new tests do not run on older kernels.

Thanks,
Amir.

Changed since v1:
- Helper _require_scratch_overlay_xattr_escapes() already added by "overlay/026: Fix test expectation for newer kernels"

Amir Goldstein (4):
overlay: Add tests for nesting private xattrs
overlay: prepare for new lowerdir+,datadir+ tests
overlay: test data-only lowerdirs with datadir+ mount option
overlay: test parsing of lowerdir+,datadir+ mount options

 common/overlay        |  15 ++
 tests/overlay/079     |  36 +++--
 tests/overlay/084     | 169 +++++++++++++++++++++
 tests/overlay/084.out |  61 ++++++++
 tests/overlay/085     | 332 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/085.out |  42 ++++++
 tests/overlay/086     |  81 +++++++++++
 tests/overlay/086.out |   2 +
 8 files changed, 723 insertions(+), 15 deletions(-)
 create mode 100755 tests/overlay/084
 create mode 100644 tests/overlay/084.out
 create mode 100755 tests/overlay/085
 create mode 100644 tests/overlay/085.out
 create mode 100755 tests/overlay/086
 create mode 100644 tests/overlay/086.out
Signature · 1 line
2.34.1
#1 lore raw

PATCH v2 1/4 overlay: Add tests for nesting private xattrs

If overlayfs xattr escaping is supported, ensure:
* We can create "overlay.*" xattrs on a file in the overlayfs
* We can create an xwhiteout file in the overlayfs

We check for nesting support by trying to getattr an "overlay.*" xattr in an overlayfs mount, which will return ENOSUPP in older kernels.

---
 tests/overlay/084     | 169 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/084.out |  61 +++++++++++++++
 2 files changed, 230 insertions(+)
 create mode 100755 tests/overlay/084
 create mode 100644 tests/overlay/084.out
tests/overlay/084 new file +169 -0
new file mode 100755
index 00000000..ff451f38
@@ -0,0 +1,169 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2023 Red Hat, Inc. All Rights Reserved.
+# Copyright (C) 2023 CTERA Networks. All Rights Reserved.
+#
+# FS QA Test No. 084
+#
+# Test advanded nesting functionallity
+#
+. ./common/preamble
+_begin_fstest auto quick nested
+
+# Override the default cleanup function.
+_cleanup()
+{
+	cd /
+	# Unmount nested mounts if things fail
+	$UMOUNT_PROG $OVL_BASE_SCRATCH_MNT/nested  2>/dev/null
+	rm -rf $tmp
+}
+
+# Import common functions.
+. ./common/filter
+. ./common/attr
+
+# real QA test starts here
+_supported_fs overlay
+# We use non-default scratch underlying overlay dirs, we need to check
+# them explicity after test.
+_require_scratch_nocheck
+_require_scratch_overlay_xattr_escapes
+
+# remove all files from previous tests
+_scratch_mkfs
+
+lowerdir=$OVL_BASE_SCRATCH_MNT/lower
+middir=$OVL_BASE_SCRATCH_MNT/mid
+upperdir=$OVL_BASE_SCRATCH_MNT/upper
+workdir=$OVL_BASE_SCRATCH_MNT/workdir
+nesteddir=$OVL_BASE_SCRATCH_MNT/nested
+
+umount_overlay()
+{
+	$UMOUNT_PROG $SCRATCH_MNT
+}
+
+test_escape()
+{
+	local prefix=$1
+
+	echo -e "\n== Check xattr escape $prefix =="
+
+	local extra_options=""
+	if [ "$prefix" == "user" ]; then
+            extra_options="-o userxattr"
+	fi
+
+	_scratch_mkfs
+	mkdir -p $lowerdir $middir $upperdir $workdir $nesteddir
+
+	_overlay_scratch_mount_dirs $lowerdir $middir $workdir $extra_options
+
+	mkdir -p $SCRATCH_MNT/layer1/dir/ $SCRATCH_MNT/layer2/dir
+
+	touch $SCRATCH_MNT/layer1/dir/file
+
+	# Make layer2/dir an opaque file
+	# Only one of these will be escaped, but both should succeed
+	setfattr -n user.overlay.opaque -v "y" $SCRATCH_MNT/layer2/dir
+	setfattr -n trusted.overlay.opaque -v "y" $SCRATCH_MNT/layer2/dir
+
+	getfattr -m "overlay\\." --absolute-names -d $SCRATCH_MNT/layer2/dir | _filter_scratch
+
+	umount_overlay
+
+	getfattr -m "overlay\\." --absolute-names -d $middir/layer2/dir | _filter_scratch
+
+	# Remount as lower and try again
+	_overlay_scratch_mount_dirs $middir:$lowerdir $upperdir $workdir $extra_options
+
+	getfattr -m "overlay\\." --absolute-names -d $SCRATCH_MNT/layer2/dir | _filter_scratch
+
+	# Recursively mount and ensure the opaque dir is working with both trusted and user xattrs
+	echo "nested xattr mount with trusted.overlay"
+	_overlay_mount_dirs $SCRATCH_MNT/layer2:$SCRATCH_MNT/layer1 - - overlayfs $nesteddir
+	stat $nesteddir/dir/file  2>&1 | _filter_scratch
+	$UMOUNT_PROG $nesteddir
+
+	echo "nested xattr mount with user.overlay"
+	_overlay_mount_dirs $SCRATCH_MNT/layer2:$SCRATCH_MNT/layer1 - - -o userxattr overlayfs $nesteddir
+	stat $nesteddir/dir/file  2>&1 | _filter_scratch
+	$UMOUNT_PROG $nesteddir
+
+	# Also ensure propagate the escaped xattr when we copy-up layer2/dir
+	echo "copy-up of escaped xattrs"
+	touch $SCRATCH_MNT/layer2/dir/other_file
+	getfattr -m "$prefix.overlay\\.overlay" --absolute-names -d $upperdir/layer2/dir | _filter_scratch
+
+	umount_overlay
+}
+
+test_escape trusted
+test_escape user
+
+do_test_xwhiteout()
+{
+	local prefix=$1
+	local basedir=$2
+
+	local extra_options=""
+	if [ "$prefix" == "user" ]; then
+            extra_options="-o userxattr"
+	fi
+
+	mkdir -p $basedir/lower $basedir/upper $basedir/work
+	touch $basedir/lower/regular $basedir/lower/hidden  $basedir/upper/hidden
+	setfattr -n $prefix.overlay.whiteouts -v "y" $basedir/upper
+	setfattr -n $prefix.overlay.whiteout -v "y" $basedir/upper/hidden
+
+	# Test the hidden is invisible
+	_overlay_scratch_mount_dirs $basedir/upper:$basedir/lower - - $extra_options
+	ls $SCRATCH_MNT
+	stat $SCRATCH_MNT/hidden 2>&1 | _filter_scratch
+	umount_overlay
+}
+
+# Validate that xwhiteouts work like whiteouts
+test_xwhiteout()
+{
+	local prefix=$1
+
+	echo -e "\n== Check xwhiteout $prefix =="
+
+	_scratch_mkfs
+
+	do_test_xwhiteout $prefix $OVL_BASE_SCRATCH_MNT
+}
+
+test_xwhiteout trusted
+test_xwhiteout user
+
+# Validate that (escaped) xwhiteouts work inside a nested overlayfs mount
+test_escaped_xwhiteout()
+{
+	local prefix=$1
+
+	echo -e "\n== Check escaped xwhiteout $prefix =="
+
+	local extra_options=""
+	if [ "$prefix" == "user" ]; then
+            extra_options="-o userxattr"
+	fi
+
+	_scratch_mkfs
+	mkdir -p $lowerdir $upperdir $workdir $nesteddir
+
+	_overlay_mount_dirs $lowerdir $upperdir $workdir $extra_options overlayfs $nesteddir
+
+	do_test_xwhiteout $prefix $nesteddir
+
+	$UMOUNT_PROG $nesteddir
+}
+
+test_escaped_xwhiteout trusted
+test_escaped_xwhiteout user
+
+# success, all done
+status=0
+exit
tests/overlay/084.out new file +61 -0

61 lines not on this page · raw message

Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH v2 1/4] overlay: Add tests for nesting private xattrs

Quoted 10 lines · Amir Goldstein

If overlayfs xattr escaping is supported, ensure:
* We can create "overlay.*" xattrs on a file in the overlayfs
* We can create an xwhiteout file in the overlayfs

We check for nesting support by trying to getattr an "overlay.*" xattr in an overlayfs mount, which will return ENOSUPP in older kernels.

---

Hi Amir,

This test passed with below kernel configuration at first:
CONFIG_OVERLAY_FS=m
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set
# CONFIG_OVERLAY_FS_XINO_AUTO is not set
# CONFIG_OVERLAY_FS_METACOPY is not set

But then I found it fails if I enabled below configurations:
CONFIG_OVERLAY_FS_REDIRECT_DIR=y CONFIG_OVERLAY_FS_INDEX=y CONFIG_OVERLAY_FS_XINO_AUTO=y CONFIG_OVERLAY_FS_METACOPY=y

Without these configures, this test passed. But with them, it fails as [1]. The underlying fs is xfs (with default mkfs options), there're not specific MOUNT_OPTIONS and MKFS_OPTIONS to use.

I'll delay merging this patchset temporarily, please check.

Thanks,
Zorro

[1]
QA output created by 084

== Check xattr escape trusted ==
# file: SCRATCH_MNT/layer2/dir
trusted.overlay.opaque="y"
user.overlay.opaque="y"

# file: SCRATCH_DEV/mid/layer2/dir
trusted.overlay.overlay.opaque="y"
user.overlay.opaque="y"

mount: /mnt/fstests/SCRATCH_DIR/ovl-mnt: mount(2) system call failed: Stale file handle.
getfattr: /mnt/fstests/SCRATCH_DIR/ovl-mnt/layer2/dir: No such file or directory nested xattr mount with trusted.overlay
mount: /mnt/fstests/SCRATCH_DIR/nested: special device overlayfs does not exist.
stat: cannot statx 'SCRATCH_DEV/nested/dir/file': No such file or directory
umount: /mnt/fstests/SCRATCH_DIR/nested: not mounted.
nested xattr mount with user.overlay
mount: /mnt/fstests/SCRATCH_DIR/nested: special device overlayfs does not exist.
stat: cannot statx 'SCRATCH_DEV/nested/dir/file': No such file or directory
umount: /mnt/fstests/SCRATCH_DIR/nested: not mounted.
copy-up of escaped xattrs
touch: cannot touch '/mnt/fstests/SCRATCH_DIR/ovl-mnt/layer2/dir/other_file': No such file or directory
getfattr: /mnt/fstests/SCRATCH_DIR/upper/layer2/dir: No such file or directory
umount: /mnt/fstests/SCRATCH_DIR/ovl-mnt: not mounted.

== Check xattr escape user ==
# file: SCRATCH_MNT/layer2/dir
trusted.overlay.opaque="y"
user.overlay.opaque="y"

# file: SCRATCH_DEV/mid/layer2/dir
trusted.overlay.opaque="y"
user.overlay.overlay.opaque="y"

mount: /mnt/fstests/SCRATCH_DIR/ovl-mnt: mount(2) system call failed: Stale file handle.
getfattr: /mnt/fstests/SCRATCH_DIR/ovl-mnt/layer2/dir: No such file or directory nested xattr mount with trusted.overlay
mount: /mnt/fstests/SCRATCH_DIR/nested: special device overlayfs does not exist.
stat: cannot statx 'SCRATCH_DEV/nested/dir/file': No such file or directory
umount: /mnt/fstests/SCRATCH_DIR/nested: not mounted.
nested xattr mount with user.overlay
mount: /mnt/fstests/SCRATCH_DIR/nested: special device overlayfs does not exist.
stat: cannot statx 'SCRATCH_DEV/nested/dir/file': No such file or directory
umount: /mnt/fstests/SCRATCH_DIR/nested: not mounted.
copy-up of escaped xattrs
touch: cannot touch '/mnt/fstests/SCRATCH_DIR/ovl-mnt/layer2/dir/other_file': No such file or directory
getfattr: /mnt/fstests/SCRATCH_DIR/upper/layer2/dir: No such file or directory
umount: /mnt/fstests/SCRATCH_DIR/ovl-mnt: not mounted.

== Check xwhiteout trusted ==
regular
stat: cannot statx 'SCRATCH_MNT/hidden': No such file or directory

== Check xwhiteout user ==
regular
stat: cannot statx 'SCRATCH_MNT/hidden': No such file or directory

== Check escaped xwhiteout trusted ==
regular
stat: cannot statx 'SCRATCH_MNT/hidden': No such file or directory

== Check escaped xwhiteout user ==
regular
stat: cannot statx 'SCRATCH_MNT/hidden': No such file or directory

Quoted 251 lines
 tests/overlay/084     | 169 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/084.out |  61 +++++++++++++++
 2 files changed, 230 insertions(+)
 create mode 100755 tests/overlay/084
 create mode 100644 tests/overlay/084.out
tests/overlay/084 new file +169 -0
new file mode 100755
index 00000000..ff451f38
@@ -0,0 +1,169 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2023 Red Hat, Inc. All Rights Reserved.
+# Copyright (C) 2023 CTERA Networks. All Rights Reserved.
+#
+# FS QA Test No. 084
+#
+# Test advanded nesting functionallity
+#
+. ./common/preamble
+_begin_fstest auto quick nested
+
+# Override the default cleanup function.
+_cleanup()
+{
+	cd /
+	# Unmount nested mounts if things fail
+	$UMOUNT_PROG $OVL_BASE_SCRATCH_MNT/nested  2>/dev/null
+	rm -rf $tmp
+}
+
+# Import common functions.
+. ./common/filter
+. ./common/attr
+
+# real QA test starts here
+_supported_fs overlay
+# We use non-default scratch underlying overlay dirs, we need to check
+# them explicity after test.
+_require_scratch_nocheck
+_require_scratch_overlay_xattr_escapes
+
+# remove all files from previous tests
+_scratch_mkfs
+
+lowerdir=$OVL_BASE_SCRATCH_MNT/lower
+middir=$OVL_BASE_SCRATCH_MNT/mid
+upperdir=$OVL_BASE_SCRATCH_MNT/upper
+workdir=$OVL_BASE_SCRATCH_MNT/workdir
+nesteddir=$OVL_BASE_SCRATCH_MNT/nested
+
+umount_overlay()
+{
+	$UMOUNT_PROG $SCRATCH_MNT
+}
+
+test_escape()
+{
+	local prefix=$1
+
+	echo -e "\n== Check xattr escape $prefix =="
+
+	local extra_options=""
+	if [ "$prefix" == "user" ]; then
+            extra_options="-o userxattr"
+	fi
+
+	_scratch_mkfs
+	mkdir -p $lowerdir $middir $upperdir $workdir $nesteddir
+
+	_overlay_scratch_mount_dirs $lowerdir $middir $workdir $extra_options
+
+	mkdir -p $SCRATCH_MNT/layer1/dir/ $SCRATCH_MNT/layer2/dir
+
+	touch $SCRATCH_MNT/layer1/dir/file
+
+	# Make layer2/dir an opaque file
+	# Only one of these will be escaped, but both should succeed
+	setfattr -n user.overlay.opaque -v "y" $SCRATCH_MNT/layer2/dir
+	setfattr -n trusted.overlay.opaque -v "y" $SCRATCH_MNT/layer2/dir
+
+	getfattr -m "overlay\\." --absolute-names -d $SCRATCH_MNT/layer2/dir | _filter_scratch
+
+	umount_overlay
+
+	getfattr -m "overlay\\." --absolute-names -d $middir/layer2/dir | _filter_scratch
+
+	# Remount as lower and try again
+	_overlay_scratch_mount_dirs $middir:$lowerdir $upperdir $workdir $extra_options
+
+	getfattr -m "overlay\\." --absolute-names -d $SCRATCH_MNT/layer2/dir | _filter_scratch
+
+	# Recursively mount and ensure the opaque dir is working with both trusted and user xattrs
+	echo "nested xattr mount with trusted.overlay"
+	_overlay_mount_dirs $SCRATCH_MNT/layer2:$SCRATCH_MNT/layer1 - - overlayfs $nesteddir
+	stat $nesteddir/dir/file  2>&1 | _filter_scratch
+	$UMOUNT_PROG $nesteddir
+
+	echo "nested xattr mount with user.overlay"
+	_overlay_mount_dirs $SCRATCH_MNT/layer2:$SCRATCH_MNT/layer1 - - -o userxattr overlayfs $nesteddir
+	stat $nesteddir/dir/file  2>&1 | _filter_scratch
+	$UMOUNT_PROG $nesteddir
+
+	# Also ensure propagate the escaped xattr when we copy-up layer2/dir
+	echo "copy-up of escaped xattrs"
+	touch $SCRATCH_MNT/layer2/dir/other_file
+	getfattr -m "$prefix.overlay\\.overlay" --absolute-names -d $upperdir/layer2/dir | _filter_scratch
+
+	umount_overlay
+}
+
+test_escape trusted
+test_escape user
+
+do_test_xwhiteout()
+{
+	local prefix=$1
+	local basedir=$2
+
+	local extra_options=""
+	if [ "$prefix" == "user" ]; then
+            extra_options="-o userxattr"
+	fi
+
+	mkdir -p $basedir/lower $basedir/upper $basedir/work
+	touch $basedir/lower/regular $basedir/lower/hidden  $basedir/upper/hidden
+	setfattr -n $prefix.overlay.whiteouts -v "y" $basedir/upper
+	setfattr -n $prefix.overlay.whiteout -v "y" $basedir/upper/hidden
+
+	# Test the hidden is invisible
+	_overlay_scratch_mount_dirs $basedir/upper:$basedir/lower - - $extra_options
+	ls $SCRATCH_MNT
+	stat $SCRATCH_MNT/hidden 2>&1 | _filter_scratch
+	umount_overlay
+}
+
+# Validate that xwhiteouts work like whiteouts
+test_xwhiteout()
+{
+	local prefix=$1
+
+	echo -e "\n== Check xwhiteout $prefix =="
+
+	_scratch_mkfs
+
+	do_test_xwhiteout $prefix $OVL_BASE_SCRATCH_MNT
+}
+
+test_xwhiteout trusted
+test_xwhiteout user
+
+# Validate that (escaped) xwhiteouts work inside a nested overlayfs mount
+test_escaped_xwhiteout()
+{
+	local prefix=$1
+
+	echo -e "\n== Check escaped xwhiteout $prefix =="
+
+	local extra_options=""
+	if [ "$prefix" == "user" ]; then
+            extra_options="-o userxattr"
+	fi
+
+	_scratch_mkfs
+	mkdir -p $lowerdir $upperdir $workdir $nesteddir
+
+	_overlay_mount_dirs $lowerdir $upperdir $workdir $extra_options overlayfs $nesteddir
+
+	do_test_xwhiteout $prefix $nesteddir
+
+	$UMOUNT_PROG $nesteddir
+}
+
+test_escaped_xwhiteout trusted
+test_escaped_xwhiteout user
+
+# success, all done
+status=0
+exit
tests/overlay/084.out new file +61 -0
new file mode 100644
index 00000000..54b890de
@@ -0,0 +1,61 @@
+QA output created by 084
+
+== Check xattr escape trusted ==
+# file: SCRATCH_MNT/layer2/dir
+trusted.overlay.opaque="y"
+user.overlay.opaque="y"
+
+# file: SCRATCH_DEV/mid/layer2/dir
+trusted.overlay.overlay.opaque="y"
+user.overlay.opaque="y"
+
+# file: SCRATCH_MNT/layer2/dir
+trusted.overlay.opaque="y"
+user.overlay.opaque="y"
+
+nested xattr mount with trusted.overlay
+stat: cannot statx 'SCRATCH_DEV/nested/dir/file': No such file or directory
+nested xattr mount with user.overlay
+stat: cannot statx 'SCRATCH_DEV/nested/dir/file': No such file or directory
+copy-up of escaped xattrs
+# file: SCRATCH_DEV/upper/layer2/dir
+trusted.overlay.overlay.opaque="y"
+
+
+== Check xattr escape user ==
+# file: SCRATCH_MNT/layer2/dir
+trusted.overlay.opaque="y"
+user.overlay.opaque="y"
+
+# file: SCRATCH_DEV/mid/layer2/dir
+trusted.overlay.opaque="y"
+user.overlay.overlay.opaque="y"
+
+# file: SCRATCH_MNT/layer2/dir
+trusted.overlay.opaque="y"
+user.overlay.opaque="y"
+
+nested xattr mount with trusted.overlay
+stat: cannot statx 'SCRATCH_DEV/nested/dir/file': No such file or directory
+nested xattr mount with user.overlay
+stat: cannot statx 'SCRATCH_DEV/nested/dir/file': No such file or directory
+copy-up of escaped xattrs
+# file: SCRATCH_DEV/upper/layer2/dir
+user.overlay.overlay.opaque="y"
+
+
+== Check xwhiteout trusted ==
+regular
+stat: cannot statx 'SCRATCH_MNT/hidden': No such file or directory
+
+== Check xwhiteout user ==
+regular
+stat: cannot statx 'SCRATCH_MNT/hidden': No such file or directory
+
+== Check escaped xwhiteout trusted ==
+regular
+stat: cannot statx 'SCRATCH_MNT/hidden': No such file or directory
+
+== Check escaped xwhiteout user ==
+regular
+stat: cannot statx 'SCRATCH_MNT/hidden': No such file or directory
Signature · 1 line
2.34.1
#3 lore raw

Re: [PATCH v2 1/4] overlay: Add tests for nesting private xattrs

Quoted 29 lines · Zorro Lang
Quoted 10 lines · Amir Goldstein

If overlayfs xattr escaping is supported, ensure:
* We can create "overlay.*" xattrs on a file in the overlayfs
* We can create an xwhiteout file in the overlayfs

We check for nesting support by trying to getattr an "overlay.*" xattr in an overlayfs mount, which will return ENOSUPP in older kernels.

---

Hi Amir,

This test passed with below kernel configuration at first:
CONFIG_OVERLAY_FS=m
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set
# CONFIG_OVERLAY_FS_XINO_AUTO is not set
# CONFIG_OVERLAY_FS_METACOPY is not set

But then I found it fails if I enabled below configurations:
CONFIG_OVERLAY_FS_REDIRECT_DIR=y CONFIG_OVERLAY_FS_INDEX=y CONFIG_OVERLAY_FS_XINO_AUTO=y CONFIG_OVERLAY_FS_METACOPY=y

Quoted 6 lines

Without these configures, this test passed. But with them, it fails as [1]. The underlying fs is xfs (with default mkfs options), there're not specific MOUNT_OPTIONS and MKFS_OPTIONS to use.

I'll delay merging this patchset temporarily, please check.

good spotting!

Here is a fix if you want to fix and test it in your tree:

tests/overlay/084 +6 -4
index ff451f38..8465caeb 100755
@@ -50,9 +50,10 @@ test_escape()
 
        echo -e "\n== Check xattr escape $prefix =="
 
-       local extra_options=""
+       # index feature would require nfs_export on $nesteddir mount
+       local extra_options="-o index=off"
        if [ "$prefix" == "user" ]; then
-            extra_options="-o userxattr"
+            extra_options+=",userxattr"
        fi
 
        _scratch_mkfs
@@ -146,9 +147,10 @@ test_escaped_xwhiteout()
 
        echo -e "\n== Check escaped xwhiteout $prefix =="
 
-       local extra_options=""
+       # index feature would require nfs_export on $nesteddir mount
+       local extra_options="-o index=off"
        if [ "$prefix" == "user" ]; then
-            extra_options="-o userxattr"
+            extra_options+=",userxattr"
        fi
 
        _scratch_mkfs

Thanks,
Amir.

#4 lore raw

Re: [PATCH v2 1/4] overlay: Add tests for nesting private xattrs

Quoted 59 lines · Amir Goldstein
Quoted 29 lines · Zorro Lang
Quoted 10 lines · Amir Goldstein

If overlayfs xattr escaping is supported, ensure:
* We can create "overlay.*" xattrs on a file in the overlayfs
* We can create an xwhiteout file in the overlayfs

We check for nesting support by trying to getattr an "overlay.*" xattr in an overlayfs mount, which will return ENOSUPP in older kernels.

---

Hi Amir,

This test passed with below kernel configuration at first:
CONFIG_OVERLAY_FS=m
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set
# CONFIG_OVERLAY_FS_XINO_AUTO is not set
# CONFIG_OVERLAY_FS_METACOPY is not set

But then I found it fails if I enabled below configurations:
CONFIG_OVERLAY_FS_REDIRECT_DIR=y CONFIG_OVERLAY_FS_INDEX=y CONFIG_OVERLAY_FS_XINO_AUTO=y CONFIG_OVERLAY_FS_METACOPY=y

Quoted 6 lines

Without these configures, this test passed. But with them, it fails as [1]. The underlying fs is xfs (with default mkfs options), there're not specific MOUNT_OPTIONS and MKFS_OPTIONS to use.

I'll delay merging this patchset temporarily, please check.

good spotting!

Here is a fix if you want to fix and test it in your tree:

tests/overlay/084 +3 -2
index ff451f38..8465caeb 100755
@@ -50,9 +50,10 @@ test_escape()
 
        echo -e "\n== Check xattr escape $prefix =="
 
-       local extra_options=""
+       # index feature would require nfs_export on $nesteddir mount
+       local extra_options="-o index=off"
        if [ "$prefix" == "user" ]; then
-            extra_options="-o userxattr"
+            extra_options+=",userxattr"
        fi
 
        _scratch_mkfs

tests/overlay/084 test_escaped_xwhiteout() · line 153

Quoted 9 lines
tests/overlay/084 +3 -2
@@ -146,9 +147,10 @@ test_escaped_xwhiteout()
 
        echo -e "\n== Check escaped xwhiteout $prefix =="
 
-       local extra_options=""
+       # index feature would require nfs_export on $nesteddir mount
+       local extra_options="-o index=off"
        if [ "$prefix" == "user" ]; then
-            extra_options="-o userxattr"
+            extra_options+=",userxattr"

It works, so it's about the CONFIG_OVERLAY_FS_INDEX=y. I've releated fstests v2023.12.10 version, this patchset will be in next release. Will send a new version with this change?

Thanks,
Zorro

Quoted 8 lines
tests/overlay/084 +0 -0
        fi
 
        _scratch_mkfs

Thanks,
Amir.

#5 lore raw

Re: [PATCH v2 1/4] overlay: Add tests for nesting private xattrs

Quoted 64 lines · Zorro Lang
Quoted 62 lines · Amir Goldstein
Quoted 29 lines · Zorro Lang
Quoted 10 lines · Amir Goldstein

If overlayfs xattr escaping is supported, ensure:
* We can create "overlay.*" xattrs on a file in the overlayfs
* We can create an xwhiteout file in the overlayfs

We check for nesting support by trying to getattr an "overlay.*" xattr in an overlayfs mount, which will return ENOSUPP in older kernels.

---

Hi Amir,

This test passed with below kernel configuration at first:
CONFIG_OVERLAY_FS=m
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set
# CONFIG_OVERLAY_FS_XINO_AUTO is not set
# CONFIG_OVERLAY_FS_METACOPY is not set

But then I found it fails if I enabled below configurations:
CONFIG_OVERLAY_FS_REDIRECT_DIR=y CONFIG_OVERLAY_FS_INDEX=y CONFIG_OVERLAY_FS_XINO_AUTO=y CONFIG_OVERLAY_FS_METACOPY=y

Quoted 6 lines

Without these configures, this test passed. But with them, it fails as [1]. The underlying fs is xfs (with default mkfs options), there're not specific MOUNT_OPTIONS and MKFS_OPTIONS to use.

I'll delay merging this patchset temporarily, please check.

good spotting!

Here is a fix if you want to fix and test it in your tree:

tests/overlay/084 +3 -2
index ff451f38..8465caeb 100755
@@ -50,9 +50,10 @@ test_escape()
 
        echo -e "\n== Check xattr escape $prefix =="
 
-       local extra_options=""
+       # index feature would require nfs_export on $nesteddir mount
+       local extra_options="-o index=off"
        if [ "$prefix" == "user" ]; then
-            extra_options="-o userxattr"
+            extra_options+=",userxattr"
        fi
 
        _scratch_mkfs
@@ -146,9 +147,10 @@ test_escaped_xwhiteout()
 
        echo -e "\n== Check escaped xwhiteout $prefix =="
 
Quoted 8 lines

tests/overlay/084 test_escape() · line 56

Quoted 6 lines
tests/overlay/084 +3 -2
-       local extra_options=""
+       # index feature would require nfs_export on $nesteddir mount
+       local extra_options="-o index=off"
        if [ "$prefix" == "user" ]; then
-            extra_options="-o userxattr"
+            extra_options+=",userxattr"

It works, so it's about the CONFIG_OVERLAY_FS_INDEX=y.

Yes.

the nested overlayfs setup requires that either the inner overlayfs has nfs_export enabled, as is done in tests overlay/068,069,070,071
or that the outer overlayfs has index disabled. The latter is easier for this test, because there is no need for the index feature in these test cases.

Quoted 4 lines

I've released fstests
v2023.12.10 version, this patchset will be in next release. Will send a new version with this change?

Ok, I will send a new version of test 084.

Thanks,
Amir.

#6 lore raw

PATCH v2 2/4 overlay: prepare for new lowerdir+,datadir+ tests

In reply to Amir Goldstein

In preparation to forking tests for new lowerdir+,datadir+ mount options, prepare a helper to test kernel support and pass datadirs into mount helpers in overlay/079 test.

---
 common/overlay    | 15 +++++++++++++++
 tests/overlay/079 | 36 +++++++++++++++++++++---------------
 2 files changed, 36 insertions(+), 15 deletions(-)
common/overlay +15 -0
index 8f275228..ea1eb7b1 100644
@@ -247,6 +247,21 @@ _require_scratch_overlay_lowerdata_layers()
 	_scratch_unmount
 }
 
+# Check kernel support for lowerdir+=<lowerdir>,datadir+=<lowerdatadir> format
+_require_scratch_overlay_lowerdir_add_layers()
+{
+	local lowerdir="$OVL_BASE_SCRATCH_MNT/$OVL_UPPER"
+	local datadir="$OVL_BASE_SCRATCH_MNT/$OVL_LOWER"
+
+	_scratch_mkfs > /dev/null 2>&1
+	$MOUNT_PROG -t overlay $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT \
+		-o"lowerdir+=$lowerdir,datadir+=$datadir" \
+		-o"redirect_dir=follow,metacopy=on" > /dev/null 2>&1 || \
+	        _notrun "overlay lowerdir+,datadir+ not supported on ${SCRATCH_DEV}"
+
+	_scratch_unmount
+}
+
 # Helper function to check underlying dirs of overlay filesystem
 _overlay_fsck_dirs()
 {
tests/overlay/079 +21 -15
index 77f94598..078ee816 100755
@@ -139,16 +139,21 @@ check_file_size_contents()
 
 mount_overlay()
 {
-	local _lowerdir=$1
+	local _lowerdir=$1 _datadir2=$2 _datadir=$3
 
-	_overlay_scratch_mount_dirs "$_lowerdir" $upperdir $workdir -o redirect_dir=on,index=on,metacopy=on
+	$MOUNT_PROG -t overlay $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT \
+		-o"lowerdir=$_lowerdir::$_datadir2::$_datadir" \
+		-o"upperdir=$upperdir,workdir=$workdir" \
+		-o redirect_dir=on,metacopy=on
 }
 
 mount_ro_overlay()
 {
-	local _lowerdir=$1
+	local _lowerdir=$1 _datadir2=$2 _datadir=$3
 
-	_overlay_scratch_mount_dirs "$_lowerdir" "-" "-" -o ro,redirect_dir=follow,metacopy=on
+	$MOUNT_PROG -t overlay $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT \
+		-o"lowerdir=$_lowerdir::$_datadir2::$_datadir" \
+		-o redirect_dir=follow,metacopy=on
 }
 
 umount_overlay()
@@ -160,14 +165,14 @@ test_no_access()
 {
 	local _target=$1
 
-	mount_ro_overlay "$lowerdir::$datadir2::$datadir"
+	mount_ro_overlay "$lowerdir" "$datadir2" "$datadir"
 
 	stat $SCRATCH_MNT/$_target >> $seqres.full 2>&1 || \
 		echo "No access to lowerdata layer $_target"
 
 	echo "Unmount and Mount rw"
 	umount_overlay
-	mount_overlay "$lowerdir::$datadir2::$datadir"
+	mount_overlay "$lowerdir" "$datadir2" "$datadir"
 	stat $SCRATCH_MNT/$_target >> $seqres.full 2>&1 || \
 		echo "No access to lowerdata layer $_target"
 	umount_overlay
@@ -175,11 +180,12 @@ test_no_access()
 
 test_common()
 {
-	local _lowerdirs=$1 _target=$2 _size=$3 _blocks=$4 _data="$5"
-	local _redirect=$6
+	local _lowerdir=$1 _datadir2=$2 _datadir=$3
+	local _target=$4 _size=$5 _blocks=$6 _data="$7"
+	local _redirect=$8
 
 	echo "Mount ro"
-	mount_ro_overlay $_lowerdirs
+	mount_ro_overlay $_lowerdir $_datadir2 $_datadir
 
 	# Check redirect xattr to lowerdata
 	[ -n "$_redirect" ] && check_redirect $lowerdir/$_target "$_redirect"
@@ -191,7 +197,7 @@ test_common()
 	# Do a mount cycle and check size and contents again.
 	echo "Unmount and Mount rw"
 	umount_overlay
-	mount_overlay $_lowerdirs
+	mount_overlay $_lowerdir $_datadir2 $_datadir
 	echo "check properties of metadata copied up file $_target"
 	check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"
 	check_file_blocks $SCRATCH_MNT/$_target $_blocks
@@ -203,7 +209,7 @@ test_common()
 	check_file_size_contents $upperdir/$_target $_size ""
 
 	# Trigger data copy up and check absence of metacopy xattr.
-	mount_overlay $_lowerdirs
+	mount_overlay $_lowerdir $_datadir2 $_datadir
 	$XFS_IO_PROG -c "falloc 0 1" $SCRATCH_MNT/$_target >> $seqres.full
 	echo "check properties of data copied up file $_target"
 	check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"
@@ -216,7 +222,7 @@ test_lazy()
 {
 	local _target=$1
 
-	mount_overlay "$lowerdir::$datadir2::$datadir"
+	mount_overlay "$lowerdir" "$datadir2" "$datadir"
 
 	# Metadata should be valid
 	check_file_size $SCRATCH_MNT/$_target $datasize
@@ -305,12 +311,12 @@ test_no_access "$sharedname"
 
 echo -e "\n== Check follow to lowerdata layer with absolute redirect =="
 prepare_midlayer "/subdir/$dataname"
-test_common "$lowerdir::$datadir2::$datadir" "$dataname" $datasize $datablocks \
+test_common "$lowerdir" "$datadir2" "$datadir" "$dataname" $datasize $datablocks \
 		"$datacontent" "/subdir/$dataname"
-test_common "$lowerdir::$datadir2::$datadir" "$dataname2" $datasize $datablocks \
+test_common "$lowerdir" "$datadir2" "$datadir" "$dataname2" $datasize $datablocks \
 		"$datacontent2" "/subdir/$dataname.2"
 # Shared file should be picked from upper datadir
-test_common "$lowerdir::$datadir2::$datadir" "$sharedname" $datasize $datablocks \
+test_common "$lowerdir" "$datadir2" "$datadir" "$sharedname" $datasize $datablocks \
 		"$datacontent2" "/subdir/$dataname.shared"
 
 echo -e "\n== Check lazy follow to lowerdata layer =="
Signature · 1 line
2.34.1
#7 lore raw

Re: [PATCH v2 2/4] overlay: prepare for new lowerdir+,datadir+ tests

Quoted 19 lines · Amir Goldstein

In preparation to forking tests for new lowerdir+,datadir+ mount options, prepare a helper to test kernel support and pass datadirs into mount helpers in overlay/079 test.

---
 common/overlay    | 15 +++++++++++++++
 tests/overlay/079 | 36 +++++++++++++++++++++---------------
 2 files changed, 36 insertions(+), 15 deletions(-)
common/overlay +1 -0
index 8f275228..ea1eb7b1 100644
@@ -247,6 +247,21 @@ _require_scratch_overlay_lowerdata_layers()
 	_scratch_unmount
 }
 
+# Check kernel support for lowerdir+=<lowerdir>,datadir+=<lowerdatadir> format

common/overlay _require_scratch_overlay_lowerdata_layers() · line 260

Quoted 10 lines
common/overlay +10 -0
+_require_scratch_overlay_lowerdir_add_layers()
+{
+	local lowerdir="$OVL_BASE_SCRATCH_MNT/$OVL_UPPER"
+	local datadir="$OVL_BASE_SCRATCH_MNT/$OVL_LOWER"
+
+	_scratch_mkfs > /dev/null 2>&1
+	$MOUNT_PROG -t overlay $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT \
+		-o"lowerdir+=$lowerdir,datadir+=$datadir" \
+		-o"redirect_dir=follow,metacopy=on" > /dev/null 2>&1 || \
+	        _notrun "overlay lowerdir+,datadir+ not supported on ${SCRATCH_DEV}"

Hi Amir,

I found overlay cases don't use helpers in common/overlay recently, always use raw $MOUNT_PROG directly (not only in this patchset). Although overlay supports new mount format, can we improve the mount helpers in common/overlay to support that? It would be to good to use common helpers to do common operation.

Anyway, that can be changed in another patch, if it takes too much time or you don't want to do it at here. What do you think?

Thanks,
Zorro

Quoted 116 lines
common/overlay +4 -0
+
+	_scratch_unmount
+}
+
 # Helper function to check underlying dirs of overlay filesystem
 _overlay_fsck_dirs()
 {
tests/overlay/079 +21 -15
index 77f94598..078ee816 100755
@@ -139,16 +139,21 @@ check_file_size_contents()
 
 mount_overlay()
 {
-	local _lowerdir=$1
+	local _lowerdir=$1 _datadir2=$2 _datadir=$3
 
-	_overlay_scratch_mount_dirs "$_lowerdir" $upperdir $workdir -o redirect_dir=on,index=on,metacopy=on
+	$MOUNT_PROG -t overlay $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT \
+		-o"lowerdir=$_lowerdir::$_datadir2::$_datadir" \
+		-o"upperdir=$upperdir,workdir=$workdir" \
+		-o redirect_dir=on,metacopy=on
 }
 
 mount_ro_overlay()
 {
-	local _lowerdir=$1
+	local _lowerdir=$1 _datadir2=$2 _datadir=$3
 
-	_overlay_scratch_mount_dirs "$_lowerdir" "-" "-" -o ro,redirect_dir=follow,metacopy=on
+	$MOUNT_PROG -t overlay $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT \
+		-o"lowerdir=$_lowerdir::$_datadir2::$_datadir" \
+		-o redirect_dir=follow,metacopy=on
 }
 
 umount_overlay()
@@ -160,14 +165,14 @@ test_no_access()
 {
 	local _target=$1
 
-	mount_ro_overlay "$lowerdir::$datadir2::$datadir"
+	mount_ro_overlay "$lowerdir" "$datadir2" "$datadir"
 
 	stat $SCRATCH_MNT/$_target >> $seqres.full 2>&1 || \
 		echo "No access to lowerdata layer $_target"
 
 	echo "Unmount and Mount rw"
 	umount_overlay
-	mount_overlay "$lowerdir::$datadir2::$datadir"
+	mount_overlay "$lowerdir" "$datadir2" "$datadir"
 	stat $SCRATCH_MNT/$_target >> $seqres.full 2>&1 || \
 		echo "No access to lowerdata layer $_target"
 	umount_overlay
@@ -175,11 +180,12 @@ test_no_access()
 
 test_common()
 {
-	local _lowerdirs=$1 _target=$2 _size=$3 _blocks=$4 _data="$5"
-	local _redirect=$6
+	local _lowerdir=$1 _datadir2=$2 _datadir=$3
+	local _target=$4 _size=$5 _blocks=$6 _data="$7"
+	local _redirect=$8
 
 	echo "Mount ro"
-	mount_ro_overlay $_lowerdirs
+	mount_ro_overlay $_lowerdir $_datadir2 $_datadir
 
 	# Check redirect xattr to lowerdata
 	[ -n "$_redirect" ] && check_redirect $lowerdir/$_target "$_redirect"
@@ -191,7 +197,7 @@ test_common()
 	# Do a mount cycle and check size and contents again.
 	echo "Unmount and Mount rw"
 	umount_overlay
-	mount_overlay $_lowerdirs
+	mount_overlay $_lowerdir $_datadir2 $_datadir
 	echo "check properties of metadata copied up file $_target"
 	check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"
 	check_file_blocks $SCRATCH_MNT/$_target $_blocks
@@ -203,7 +209,7 @@ test_common()
 	check_file_size_contents $upperdir/$_target $_size ""
 
 	# Trigger data copy up and check absence of metacopy xattr.
-	mount_overlay $_lowerdirs
+	mount_overlay $_lowerdir $_datadir2 $_datadir
 	$XFS_IO_PROG -c "falloc 0 1" $SCRATCH_MNT/$_target >> $seqres.full
 	echo "check properties of data copied up file $_target"
 	check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"
@@ -216,7 +222,7 @@ test_lazy()
 {
 	local _target=$1
 
-	mount_overlay "$lowerdir::$datadir2::$datadir"
+	mount_overlay "$lowerdir" "$datadir2" "$datadir"
 
 	# Metadata should be valid
 	check_file_size $SCRATCH_MNT/$_target $datasize
@@ -305,12 +311,12 @@ test_no_access "$sharedname"
 
 echo -e "\n== Check follow to lowerdata layer with absolute redirect =="
 prepare_midlayer "/subdir/$dataname"
-test_common "$lowerdir::$datadir2::$datadir" "$dataname" $datasize $datablocks \
+test_common "$lowerdir" "$datadir2" "$datadir" "$dataname" $datasize $datablocks \
 		"$datacontent" "/subdir/$dataname"
-test_common "$lowerdir::$datadir2::$datadir" "$dataname2" $datasize $datablocks \
+test_common "$lowerdir" "$datadir2" "$datadir" "$dataname2" $datasize $datablocks \
 		"$datacontent2" "/subdir/$dataname.2"
 # Shared file should be picked from upper datadir
-test_common "$lowerdir::$datadir2::$datadir" "$sharedname" $datasize $datablocks \
+test_common "$lowerdir" "$datadir2" "$datadir" "$sharedname" $datasize $datablocks \
 		"$datacontent2" "/subdir/$dataname.shared"
 
 echo -e "\n== Check lazy follow to lowerdata layer =="
Signature · 1 line
2.34.1
#8 lore raw

Re: [PATCH v2 2/4] overlay: prepare for new lowerdir+,datadir+ tests

Quoted 34 lines · Zorro Lang
Quoted 19 lines · Amir Goldstein

In preparation to forking tests for new lowerdir+,datadir+ mount options, prepare a helper to test kernel support and pass datadirs into mount helpers in overlay/079 test.

---
 common/overlay    | 15 +++++++++++++++
 tests/overlay/079 | 36 +++++++++++++++++++++---------------
 2 files changed, 36 insertions(+), 15 deletions(-)
common/overlay +1 -0
index 8f275228..ea1eb7b1 100644
@@ -247,6 +247,21 @@ _require_scratch_overlay_lowerdata_layers()
      _scratch_unmount
 }
 
+# Check kernel support for lowerdir+=<lowerdir>,datadir+=<lowerdatadir> format

common/overlay _require_scratch_overlay_lowerdata_layers() · line 260

Quoted 10 lines
common/overlay +10 -0
+_require_scratch_overlay_lowerdir_add_layers()
+{
+     local lowerdir="$OVL_BASE_SCRATCH_MNT/$OVL_UPPER"
+     local datadir="$OVL_BASE_SCRATCH_MNT/$OVL_LOWER"
+
+     _scratch_mkfs > /dev/null 2>&1
+     $MOUNT_PROG -t overlay $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT \
+             -o"lowerdir+=$lowerdir,datadir+=$datadir" \
+             -o"redirect_dir=follow,metacopy=on" > /dev/null 2>&1 || \
+             _notrun "overlay lowerdir+,datadir+ not supported on ${SCRATCH_DEV}"

Hi Amir,

Quoted 8 lines

I found overlay cases don't use helpers in common/overlay recently, always use raw $MOUNT_PROG directly (not only in this patchset). Although overlay supports new mount format, can we improve the mount helpers in common/overlay to support that? It would be to good to use common helpers to do common operation.

Anyway, that can be changed in another patch, if it takes too much time or you don't want to do it at here. What do you think?

I agree. I wouldn't improve the existing helpers to support the new lowerdir+,datadir+ options as positional argument like in _overlay_scratch_mount_dirs(), but there is an opportunity to reduce dedupe of this common line with a helper:

# Mount with mnt/dev of scratch mount and custom mount options _overlay_scratch_mount_opts()
{
$MOUNT_PROG -t overlay $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT $*
}

I will work on this cleanup and post a patch when I get to it. No need to block this series for the cleanup.

Thanks,
Amir.

#9 lore raw

Re: [PATCH v2 2/4] overlay: prepare for new lowerdir+,datadir+ tests

Quoted 49 lines · Amir Goldstein
Quoted 34 lines · Zorro Lang
Quoted 19 lines · Amir Goldstein

In preparation to forking tests for new lowerdir+,datadir+ mount options, prepare a helper to test kernel support and pass datadirs into mount helpers in overlay/079 test.

---
 common/overlay    | 15 +++++++++++++++
 tests/overlay/079 | 36 +++++++++++++++++++++---------------
 2 files changed, 36 insertions(+), 15 deletions(-)
common/overlay +1 -0
index 8f275228..ea1eb7b1 100644
@@ -247,6 +247,21 @@ _require_scratch_overlay_lowerdata_layers()
      _scratch_unmount
 }
 
+# Check kernel support for lowerdir+=<lowerdir>,datadir+=<lowerdatadir> format

common/overlay _require_scratch_overlay_lowerdata_layers() · line 260

Quoted 10 lines
common/overlay +10 -0
+_require_scratch_overlay_lowerdir_add_layers()
+{
+     local lowerdir="$OVL_BASE_SCRATCH_MNT/$OVL_UPPER"
+     local datadir="$OVL_BASE_SCRATCH_MNT/$OVL_LOWER"
+
+     _scratch_mkfs > /dev/null 2>&1
+     $MOUNT_PROG -t overlay $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT \
+             -o"lowerdir+=$lowerdir,datadir+=$datadir" \
+             -o"redirect_dir=follow,metacopy=on" > /dev/null 2>&1 || \
+             _notrun "overlay lowerdir+,datadir+ not supported on ${SCRATCH_DEV}"

Hi Amir,

Quoted 8 lines

I found overlay cases don't use helpers in common/overlay recently, always use raw $MOUNT_PROG directly (not only in this patchset). Although overlay supports new mount format, can we improve the mount helpers in common/overlay to support that? It would be to good to use common helpers to do common operation.

Anyway, that can be changed in another patch, if it takes too much time or you don't want to do it at here. What do you think?

I agree. I wouldn't improve the existing helpers to support the new lowerdir+,datadir+ options as positional argument like in _overlay_scratch_mount_dirs(), but there is an opportunity to reduce dedupe of this common line with a helper:

Quoted 8 lines

# Mount with mnt/dev of scratch mount and custom mount options _overlay_scratch_mount_opts()
{
$MOUNT_PROG -t overlay $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT $*
}

I will work on this cleanup and post a patch when I get to it. No need to block this series for the cleanup.

Agree, thanks for doing this!

Quoted 4 lines

Thanks,
Amir.

#10 lore raw

PATCH v2 3/4 overlay: test data-only lowerdirs with datadir+ mount option

In reply to Amir Goldstein

Fork test overlay/079 to use the new lowerdir+,datadir+ mount options.

---
 tests/overlay/085     | 332 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/085.out |  42 ++++++
 2 files changed, 374 insertions(+)
 create mode 100755 tests/overlay/085
 create mode 100644 tests/overlay/085.out
tests/overlay/085 new file +332 -0
new file mode 100755
index 00000000..07a32c24
@@ -0,0 +1,332 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2018 Red Hat, Inc. All Rights Reserved.
+# Copyright (C) 2023 CTERA Networks. All Rights Reserved.
+#
+# FS QA Test No. 085
+#
+# Test data-only layers functionality.
+# This is a variant of test overlay/079 with lowerdir+,datadir+ mount options
+#
+. ./common/preamble
+_begin_fstest auto quick metacopy redirect prealloc
+
+# Import common functions.
+. ./common/filter
+. ./common/attr
+
+# real QA test starts here
+_supported_fs overlay
+# We use non-default scratch underlying overlay dirs, we need to check
+# them explicity after test.
+_require_scratch_nocheck
+_require_scratch_overlay_features redirect_dir metacopy
+_require_scratch_overlay_lowerdir_add_layers
+_require_xfs_io_command "falloc"
+
+# remove all files from previous tests
+_scratch_mkfs
+
+# File size on lower
+dataname="datafile"
+sharedname="shared"
+datacontent="data"
+dataname2="datafile2"
+datacontent2="data2"
+datasize="4096"
+
+# Number of blocks allocated by filesystem on lower. Will be queried later.
+datarblocks=""
+datarblocksize=""
+estimated_datablocks=""
+
+udirname="pureupper"
+ufile="upperfile"
+
+# Check metacopy xattr
+check_metacopy()
+{
+	local target=$1 exist=$2
+	local out_f target_f
+	local msg
+
+	out_f=$(_getfattr --absolute-names --only-values -n \
+		$OVL_XATTR_METACOPY $target 2>&1 | _filter_scratch)
+
+	if [ "$exist" == "y" ];then
+		[ "$out_f" == "" ] && return
+		echo "Metacopy xattr does not exist on ${target}. stdout=$out_f"
+		return
+	fi
+
+	if [ "$out_f" == "" ];then
+		echo "Metacopy xattr exists on ${target} unexpectedly."
+		return
+	fi
+
+	target_f=`echo $target | _filter_scratch`
+	msg="$target_f: trusted.overlay.metacopy: No such attribute"
+
+	[ "$out_f" == "$msg" ] && return
+
+	echo "Error while checking xattr on ${target}. stdout=$out"
+}
+
+# Check redirect xattr
+check_redirect()
+{
+	local target=$1
+	local expect=$2
+
+	value=$(_getfattr --absolute-names --only-values -n \
+		$OVL_XATTR_REDIRECT $target)
+
+	[[ "$value" == "$expect" ]] || echo "Redirect xattr incorrect. Expected=\"$expect\", actual=\"$value\""
+}
+
+# Check size
+check_file_size()
+{
+	local target=$1 expected_size=$2 actual_size
+
+	actual_size=$(_get_filesize $target)
+
+	[ "$actual_size" == "$expected_size" ] || echo "Expected file size $expected_size but actual size is $actual_size"
+}
+
+check_file_blocks()
+{
+	local target=$1 expected_blocks=$2 nr_blocks
+
+	nr_blocks=$(stat -c "%b" $target)
+
+	[ "$nr_blocks" == "$expected_blocks" ] || echo "Expected $expected_blocks blocks but actual number of blocks is ${nr_blocks}."
+}
+
+check_file_contents()
+{
+	local target=$1 expected=$2
+	local actual target_f
+
+	target_f=`echo $target | _filter_scratch`
+
+	read actual<$target
+
+	[ "$actual" == "$expected" ] || echo "Expected file $target_f contents to be \"$expected\" but actual contents are \"$actual\""
+}
+
+check_no_file_contents()
+{
+	local target=$1
+	local actual target_f out_f
+
+	target_f=`echo $target | _filter_scratch`
+	out_f=`cat $target 2>&1 | _filter_scratch`
+	msg="cat: $target_f: No such file or directory"
+
+	[ "$out_f" == "$msg" ] && return
+
+	echo "$target_f unexpectedly has content"
+}
+
+
+check_file_size_contents()
+{
+	local target=$1 expected_size=$2 expected_content=$3
+
+	check_file_size $target $expected_size
+	check_file_contents $target $expected_content
+}
+
+mount_overlay()
+{
+	local _lowerdir=$1 _datadir2=$2 _datadir=$3
+
+	$MOUNT_PROG -t overlay $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT \
+		-o"lowerdir+=$_lowerdir,datadir+=$_datadir2,datadir+=$_datadir" \
+		-o"upperdir=$upperdir,workdir=$workdir" \
+		-o redirect_dir=on,metacopy=on
+}
+
+mount_ro_overlay()
+{
+	local _lowerdir=$1 _datadir2=$2 _datadir=$3
+
+	$MOUNT_PROG -t overlay $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT \
+		-o"lowerdir+=$_lowerdir,datadir+=$_datadir2,datadir+=$_datadir" \
+		-o redirect_dir=follow,metacopy=on
+}
+
+umount_overlay()
+{
+	$UMOUNT_PROG $SCRATCH_MNT
+}
+
+test_no_access()
+{
+	local _target=$1
+
+	mount_ro_overlay "$lowerdir" "$datadir2" "$datadir"
+
+	stat $SCRATCH_MNT/$_target >> $seqres.full 2>&1 || \
+		echo "No access to lowerdata layer $_target"
+
+	echo "Unmount and Mount rw"
+	umount_overlay
+	mount_overlay "$lowerdir" "$datadir2" "$datadir"
+	stat $SCRATCH_MNT/$_target >> $seqres.full 2>&1 || \
+		echo "No access to lowerdata layer $_target"
+	umount_overlay
+}
+
+test_common()
+{
+	local _lowerdir=$1 _datadir2=$2 _datadir=$3
+	local _target=$4 _size=$5 _blocks=$6 _data="$7"
+	local _redirect=$8
+
+	echo "Mount ro"
+	mount_ro_overlay $_lowerdir $_datadir2 $_datadir
+
+	# Check redirect xattr to lowerdata
+	[ -n "$_redirect" ] && check_redirect $lowerdir/$_target "$_redirect"
+
+	echo "check properties of metadata copied up file $_target"
+	check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"
+	check_file_blocks $SCRATCH_MNT/$_target $_blocks
+
+	# Do a mount cycle and check size and contents again.
+	echo "Unmount and Mount rw"
+	umount_overlay
+	mount_overlay $_lowerdir $_datadir2 $_datadir
+	echo "check properties of metadata copied up file $_target"
+	check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"
+	check_file_blocks $SCRATCH_MNT/$_target $_blocks
+
+	# Trigger metadata copy up and check existence of metacopy xattr.
+	chmod 400 $SCRATCH_MNT/$_target
+	umount_overlay
+	check_metacopy $upperdir/$_target "y"
+	check_file_size_contents $upperdir/$_target $_size ""
+
+	# Trigger data copy up and check absence of metacopy xattr.
+	mount_overlay $_lowerdir $_datadir2 $_datadir
+	$XFS_IO_PROG -c "falloc 0 1" $SCRATCH_MNT/$_target >> $seqres.full
+	echo "check properties of data copied up file $_target"
+	check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"
+	umount_overlay
+	check_metacopy $upperdir/$_target "n"
+	check_file_size_contents $upperdir/$_target $_size "$_data"
+}
+
+test_lazy()
+{
+	local _target=$1
+
+	mount_overlay "$lowerdir" "$datadir2" "$datadir"
+
+	# Metadata should be valid
+	check_file_size $SCRATCH_MNT/$_target $datasize
+	check_file_blocks $SCRATCH_MNT/$_target $estimated_datablocks
+
+	# But have no content
+	check_no_file_contents $SCRATCH_MNT/$_target
+
+	umount_overlay
+}
+
+create_basic_files()
+{
+	_scratch_mkfs
+	mkdir -p $datadir/subdir $datadir2/subdir $lowerdir $lowerdir2 $upperdir $workdir $workdir2
+	mkdir -p $upperdir/$udirname
+	echo "$datacontent" > $datadir/$dataname
+	chmod 600 $datadir/$dataname
+	echo "$datacontent2" > $datadir2/$dataname2
+	chmod 600 $datadir2/$dataname2
+
+	echo "$datacontent" > $datadir/$sharedname
+	echo "$datacontent2" > $datadir2/$sharedname
+	chmod 600 $datadir/$sharedname  $datadir2/$sharedname
+
+	# Create files of size datasize.
+	for f in $datadir/$dataname $datadir2/$dataname2 $datadir/$sharedname $datadir2/$sharedname; do
+		$XFS_IO_PROG -c "falloc 0 $datasize" $f
+		$XFS_IO_PROG -c "fsync" $f
+	done
+
+	# Query number of block
+	datablocks=$(stat -c "%b" $datadir/$dataname)
+
+	# For lazy lookup file the block count is estimated based on size and block size
+	datablocksize=$(stat -c "%B" $datadir/$dataname)
+	estimated_datablocks=$(( ($datasize + $datablocksize - 1)/$datablocksize ))
+}
+
+prepare_midlayer()
+{
+	local _redirect=$1
+
+	_scratch_mkfs
+	create_basic_files
+	if [ -n "$_redirect" ]; then
+		mv "$datadir/$dataname" "$datadir/$_redirect"
+		mv "$datadir2/$dataname2" "$datadir2/$_redirect.2"
+		mv "$datadir/$sharedname" "$datadir/$_redirect.shared"
+		mv "$datadir2/$sharedname" "$datadir2/$_redirect.shared"
+	fi
+	# Create midlayer
+	_overlay_scratch_mount_dirs $datadir2:$datadir $lowerdir $workdir2 -o redirect_dir=on,index=on,metacopy=on
+	# Trigger a metacopy with or without redirect
+	if [ -n "$_redirect" ]; then
+		mv "$SCRATCH_MNT/$_redirect" "$SCRATCH_MNT/$dataname"
+		mv "$SCRATCH_MNT/$_redirect.2" "$SCRATCH_MNT/$dataname2"
+		mv "$SCRATCH_MNT/$_redirect.shared" "$SCRATCH_MNT/$sharedname"
+	else
+		chmod 400 $SCRATCH_MNT/$dataname
+		chmod 400 $SCRATCH_MNT/$dataname2
+		chmod 400 $SCRATCH_MNT/$sharedname
+	fi
+	umount_overlay
+}
+
+# Create test directories
+datadir=$OVL_BASE_SCRATCH_MNT/data
+datadir2=$OVL_BASE_SCRATCH_MNT/data2
+lowerdir=$OVL_BASE_SCRATCH_MNT/lower
+upperdir=$OVL_BASE_SCRATCH_MNT/upper
+workdir=$OVL_BASE_SCRATCH_MNT/workdir
+workdir2=$OVL_BASE_SCRATCH_MNT/workdir2
+
+echo -e "\n== Check no follow to lowerdata layer without redirect =="
+prepare_midlayer
+test_no_access "$dataname"
+test_no_access "$dataname2"
+test_no_access "$sharedname"
+
+echo -e "\n== Check no follow to lowerdata layer with relative redirect =="
+prepare_midlayer "$dataname.renamed"
+test_no_access "$dataname"
+test_no_access "$dataname2"
+test_no_access "$sharedname"
+
+echo -e "\n== Check follow to lowerdata layer with absolute redirect =="
+prepare_midlayer "/subdir/$dataname"
+test_common "$lowerdir" "$datadir2" "$datadir" "$dataname" $datasize $datablocks \
+		"$datacontent" "/subdir/$dataname"
+test_common "$lowerdir" "$datadir2" "$datadir" "$dataname2" $datasize $datablocks \
+		"$datacontent2" "/subdir/$dataname.2"
+# Shared file should be picked from upper datadir
+test_common "$lowerdir" "$datadir2" "$datadir" "$sharedname" $datasize $datablocks \
+		"$datacontent2" "/subdir/$dataname.shared"
+
+echo -e "\n== Check lazy follow to lowerdata layer =="
+
+prepare_midlayer "/subdir/$dataname"
+rm $datadir/subdir/$dataname
+test_lazy $dataname
+
+
+# success, all done
+status=0
+exit
tests/overlay/085.out new file +42 -0

42 lines not on this page · raw message

Signature · 1 line
2.34.1
#11 lore raw

PATCH v2 4/4 overlay: test parsing of lowerdir+,datadir+ mount options

In reply to Amir Goldstein

Fork test overlay/083 to test parsing of lowerdir+,datadir+ mount options.

---
 tests/overlay/086     | 81 +++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/086.out |  2 ++
 2 files changed, 83 insertions(+)
 create mode 100755 tests/overlay/086
 create mode 100644 tests/overlay/086.out
tests/overlay/086 new file +81 -0
new file mode 100755
index 00000000..b5960517
@@ -0,0 +1,81 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2023 CTERA Networks. All Rights Reserved.
+#
+# FS QA Test 086
+#
+# Test lowerdir+,datadir+ mount option restrictions.
+#
+
+. ./common/preamble
+_begin_fstest auto quick mount
+
+# Import common functions.
+. ./common/filter
+
+# real QA test starts here
+_supported_fs overlay
+
+# _overlay_check_* helpers do not handle special chars well
+_require_scratch_nocheck
+_require_scratch_overlay_lowerdir_add_layers
+
+# Remove all files from previous tests
+_scratch_mkfs
+
+# Create lowerdirs with special characters
+lowerdir_spaces="$OVL_BASE_SCRATCH_MNT/lower1 with  spaces"
+lowerdir_colons="$OVL_BASE_SCRATCH_MNT/lower2:with::colons"
+lowerdir_colons_esc="$OVL_BASE_SCRATCH_MNT/lower2\:with\:\:colons"
+lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER
+upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER
+workdir=$OVL_BASE_SCRATCH_MNT/$OVL_WORK
+mkdir -p "$lowerdir_spaces" "$lowerdir_colons"
+
+# _overlay_mount_* helpers do not handle lowerdir+,datadir+, so execute mount directly.
+
+# check illegal combinations and order of lowerdir,lowerdir+,datadir+
+$MOUNT_PROG -t overlay none $SCRATCH_MNT \
+	-o"lowerdir=$lowerdir,lowerdir+=$lowerdir_colons" \
+	2>> $seqres.full && \
+	echo "ERROR: invalid combination of lowerdir and lowerdir+ mount options"
+
+$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+
+$MOUNT_PROG -t overlay none $SCRATCH_MNT \
+	-o"lowerdir=$lowerdir,datadir+=$lowerdir_colons" \
+	-o redirect_dir=follow,metacopy=on 2>> $seqres.full && \
+	echo "ERROR: invalid combination of lowerdir and datadir+ mount options"
+
+$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+
+$MOUNT_PROG -t overlay none $SCRATCH_MNT \
+	-o"datadir+=$lowerdir,lowerdir+=$lowerdir_colons" \
+	-o redirect_dir=follow,metacopy=on 2>> $seqres.full && \
+	echo "ERROR: invalid order of lowerdir+ and datadir+ mount options"
+
+$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+
+# mount is expected to fail with escaped colons.
+$MOUNT_PROG -t overlay none $SCRATCH_MNT \
+	-o"lowerdir+=$lowerdir_colons_esc" \
+	2>> $seqres.full && \
+	echo "ERROR: incorrect parsing of escaped colons in lowerdir+ mount option"
+
+$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+
+# mount is expected to succeed without escaped colons.
+$MOUNT_PROG -t overlay ovl_esc_test $SCRATCH_MNT \
+	-o"lowerdir+=$lowerdir_colons,datadir+=$lowerdir_spaces" \
+	-o redirect_dir=follow,metacopy=on \
+	2>&1 | tee -a $seqres.full
+
+# if spaces are not escaped when showing mount options,
+# mount command will not show the word 'spaces' after the spaces
+$MOUNT_PROG -t overlay | grep ovl_esc_test | tee -a $seqres.full | \
+	grep -q 'datadir+'.*spaces || \
+	echo "ERROR: escaped spaces truncated from datadir+ mount option"
+
+echo "Silence is golden"
+status=0
+exit
tests/overlay/086.out new file +2 -0
new file mode 100644
index 00000000..b34758fd
@@ -0,0 +1,2 @@
+QA output created by 086
+Silence is golden
Signature · 1 line
2.34.1
Reviewed-by on the v2 series
#12 lore raw
In reply to Amir Goldstein
Quoted 12 lines · Amir Goldstein

Zorro,

This update contains 3 new overlayfs tests for new features added in v6.7-rc1.

overlay/084, written by Alexander, tests the new nested xattrs feature. overlay/{085,086} test the new lowerdir+,datadir+ mount options.

overlay/086 was partly forked from overlay/083, but overlay/083 is not sensitive to libmount version, because the escaped commas test is not related to any specific mount option, so it wasn't copied over.

Quoted 8 lines

All the new tests do not run on older kernels.

Thanks,
Amir.

Changed since v1:
- Helper _require_scratch_overlay_xattr_escapes() already added by "overlay/026: Fix test expectation for newer kernels"

This version looks good to me, let's have this feature test coverage at first.

Quoted 26 lines

Amir Goldstein (4):
overlay: Add tests for nesting private xattrs
overlay: prepare for new lowerdir+,datadir+ tests
overlay: test data-only lowerdirs with datadir+ mount option
overlay: test parsing of lowerdir+,datadir+ mount options

 common/overlay        |  15 ++
 tests/overlay/079     |  36 +++--
 tests/overlay/084     | 169 +++++++++++++++++++++
 tests/overlay/084.out |  61 ++++++++
 tests/overlay/085     | 332 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/085.out |  42 ++++++
 tests/overlay/086     |  81 +++++++++++
 tests/overlay/086.out |   2 +
 8 files changed, 723 insertions(+), 15 deletions(-)
 create mode 100755 tests/overlay/084
 create mode 100644 tests/overlay/084.out
 create mode 100755 tests/overlay/085
 create mode 100644 tests/overlay/085.out
 create mode 100755 tests/overlay/086
 create mode 100644 tests/overlay/086.out
Signature · 1 line
2.34.1