Skip to contentSkip to conversation

overlay/060: add test cases of follow to lowerdata

PATCH Superseded [email protected] 3 participants 9 messages Jul 6, 2023 – Jul 10, 2023 1 message not in the archive

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

PATCH 1/4 overlay: add helper for mounting rdonly overlay

From: Amir Goldstein <[email protected]>

Allow passing empty upperdir to _overlay_mount_dirs().

---
 common/overlay | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
common/overlay +6 -2
index 20cafeb1..452b3b09 100644
@@ -17,15 +17,19 @@ if [ -n "$OVL_BASE_FSTYP" ];then
 fi
 
 # helper function to do the actual overlayfs mount operation
+# accepts "-" as upperdir for non-upper overlayfs
 _overlay_mount_dirs()
 {
 	local lowerdir=$1
 	local upperdir=$2
 	local workdir=$3
 	shift 3
+	local diropts="-olowerdir=$lowerdir"
 
-	$MOUNT_PROG -t overlay -o lowerdir=$lowerdir -o upperdir=$upperdir \
-		    -o workdir=$workdir `_common_dev_mount_options $*`
+	[ -n "$upperdir" ] && [ "$upperdir" != "-" ] && \
+		diropts+=",upperdir=$upperdir,workdir=$workdir"
+
+	$MOUNT_PROG -t overlay $diropts `_common_dev_mount_options $*`
 }
 
 # Mount with same options/mnt/dev of scratch mount, but optionally
Signature · 1 line
2.40.1
#2 lore raw

Re: [PATCH 1/4] overlay: add helper for mounting rdonly overlay

Quoted 7 lines · On Thu, Jul 6, 2023 at 12:51 PM <[email protected]> wrote:

From: Amir Goldstein <[email protected]>

Allow passing empty upperdir to _overlay_mount_dirs().

---

Forgot to say - you need to sign off on patches that you submit on behalf of someone else, so please reply with SOB on those patches.

Quoted 32 lines
 common/overlay | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
common/overlay +6 -3
index 20cafeb1..452b3b09 100644
@@ -17,15 +17,19 @@ if [ -n "$OVL_BASE_FSTYP" ];then
 fi
 
 # helper function to do the actual overlayfs mount operation
+# accepts "-" as upperdir for non-upper overlayfs
 _overlay_mount_dirs()
 {
        local lowerdir=$1
        local upperdir=$2
        local workdir=$3
        shift 3
+       local diropts="-olowerdir=$lowerdir"
 
-       $MOUNT_PROG -t overlay -o lowerdir=$lowerdir -o upperdir=$upperdir \
-                   -o workdir=$workdir `_common_dev_mount_options $*`
+       [ -n "$upperdir" ] && [ "$upperdir" != "-" ] && \
+               diropts+=",upperdir=$upperdir,workdir=$workdir"
+
+       $MOUNT_PROG -t overlay $diropts `_common_dev_mount_options $*`
 }
 
 # Mount with same options/mnt/dev of scratch mount, but optionally
--

2.40.1

#3 lore raw

Re: [PATCH 1/4] overlay: add helper for mounting rdonly overlay

In reply to [email protected]
Quoted 6 lines · On Thu, Jul 6, 2023 at 11:50 AM <[email protected]> wrote:

From: Amir Goldstein <[email protected]>

Allow passing empty upperdir to _overlay_mount_dirs().

Quoted 33 lines
---
 common/overlay | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
common/overlay +6 -3
index 20cafeb1..452b3b09 100644
@@ -17,15 +17,19 @@ if [ -n "$OVL_BASE_FSTYP" ];then
 fi
 
 # helper function to do the actual overlayfs mount operation
+# accepts "-" as upperdir for non-upper overlayfs
 _overlay_mount_dirs()
 {
        local lowerdir=$1
        local upperdir=$2
        local workdir=$3
        shift 3
+       local diropts="-olowerdir=$lowerdir"
 
-       $MOUNT_PROG -t overlay -o lowerdir=$lowerdir -o upperdir=$upperdir \
-                   -o workdir=$workdir `_common_dev_mount_options $*`
+       [ -n "$upperdir" ] && [ "$upperdir" != "-" ] && \
+               diropts+=",upperdir=$upperdir,workdir=$workdir"
+
+       $MOUNT_PROG -t overlay $diropts `_common_dev_mount_options $*`
 }
 
 # Mount with same options/mnt/dev of scratch mount, but optionally
--

2.40.1

Signature · 3 lines
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                Red Hat, Inc
       [email protected]         [email protected]
#4 lore raw

PATCH 2/4 overlay/060: add test cases of follow to lowerdata

In reply to a message not in the archive

From: Amir Goldstein <[email protected]>

Add test coverage for following metacopy from lower layer to lower data with absolute, relative and no redirect.

---
 tests/overlay/060     | 43 ++++++++++++++++++++++++++++++++++++++-----
 tests/overlay/060.out | 18 ++++++++++++++++++
 2 files changed, 56 insertions(+), 5 deletions(-)
tests/overlay/060 +38 -5
index 363207ba..f37755da 100755
@@ -7,7 +7,7 @@
 # Test metadata only copy up functionality.
 #
 . ./common/preamble
-_begin_fstest auto quick metacopy prealloc
+_begin_fstest auto quick metacopy redirect prealloc
 
 # Import common functions.
 . ./common/filter
@@ -123,6 +123,13 @@ mount_overlay()
 	_overlay_scratch_mount_dirs "$_lowerdir" $upperdir $workdir -o redirect_dir=on,index=on,metacopy=on
 }
 
+mount_ro_overlay()
+{
+	local _lowerdir=$1
+
+	_overlay_scratch_mount_dirs "$_lowerdir" "-" "-" -o ro,redirect_dir=follow,metacopy=on
+}
+
 umount_overlay()
 {
 	$UMOUNT_PROG $SCRATCH_MNT
@@ -146,7 +153,8 @@ test_common()
 	check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"
 	check_file_blocks $SCRATCH_MNT/$_target $_blocks
 
-	# Make sure copied up file is a metacopy file.
+	# Trigger metadata copy up and check existence of metacopy xattr.
+	chmod 400 $SCRATCH_MNT/$_target
 	umount_overlay
 	check_metacopy $upperdir/$_target "y"
 	check_file_size_contents $upperdir/$_target $_size ""
@@ -165,7 +173,7 @@ test_common()
 create_basic_files()
 {
 	_scratch_mkfs
-	mkdir -p $lowerdir $lowerdir2 $upperdir $workdir $workdir2
+	mkdir -p $lowerdir/subdir $lowerdir2 $upperdir $workdir $workdir2
 	mkdir -p $upperdir/$udirname
 	echo "$lowerdata" > $lowerdir/$lowername
 	chmod 600 $lowerdir/$lowername
@@ -184,12 +192,19 @@ create_lower_link()
 
 prepare_midlayer()
 {
+	local _redirect=$1
+
 	_scratch_mkfs
 	create_basic_files
+	[ -n "$_redirect" ] && mv "$lowerdir/$lowername" "$lowerdir/$_redirect"
 	# Create midlayer
 	_overlay_scratch_mount_dirs $lowerdir $lowerdir2 $workdir2 -o redirect_dir=on,index=on,metacopy=on
-	# Trigger a metacopy
-	chmod 400 $SCRATCH_MNT/$lowername
+	# Trigger a metacopy with or without redirect
+	if [ -n "$_redirect" ]; then
+		mv "$SCRATCH_MNT/$_redirect" "$SCRATCH_MNT/$lowername"
+	else
+		chmod 400 $SCRATCH_MNT/$lowername
+	fi
 	umount_overlay
 }
 
@@ -254,6 +269,24 @@ mount_overlay $lowerdir
 mv $SCRATCH_MNT/$lowerlink $SCRATCH_MNT/$ufile
 test_common $lowerdir $ufile $lowersize $lowerblocks "$lowerdata" "/$lowerlink"
 
+echo -e "\n== Check follow to lowerdata without redirect =="
+prepare_midlayer
+mount_ro_overlay "$lowerdir2:$lowerdir"
+test_common "$lowerdir2:$lowerdir" $lowername $lowersize $lowerblocks \
+		"$lowerdata"
+
+echo -e "\n== Check follow to lowerdata with relative redirect =="
+prepare_midlayer "$lowername.renamed"
+mount_ro_overlay "$lowerdir2:$lowerdir"
+test_common "$lowerdir2:$lowerdir" "$lowername" $lowersize $lowerblocks \
+		"$lowerdata"
+
+echo -e "\n== Check follow to lowerdata with absolute redirect =="
+prepare_midlayer "/subdir/$lowername"
+mount_ro_overlay "$lowerdir2:$lowerdir"
+test_common "$lowerdir2:$lowerdir" "$lowername" $lowersize $lowerblocks \
+		"$lowerdata"
+
 # success, all done
 status=0
 exit
tests/overlay/060.out +18 -0
index a4790d31..f4ce0244 100644
@@ -40,3 +40,21 @@ check properties of metadata copied up file
 Unmount and Mount again
 check properties of metadata copied up file
 check properties of data copied up file
+
+== Check follow to lowerdata without redirect ==
+check properties of metadata copied up file
+Unmount and Mount again
+check properties of metadata copied up file
+check properties of data copied up file
+
+== Check follow to lowerdata with relative redirect ==
+check properties of metadata copied up file
+Unmount and Mount again
+check properties of metadata copied up file
+check properties of data copied up file
+
+== Check follow to lowerdata with absolute redirect ==
+check properties of metadata copied up file
+Unmount and Mount again
+check properties of metadata copied up file
+check properties of data copied up file
Signature · 1 line
2.40.1
#5 lore raw
Quoted 7 lines · On Thu, Jul 6, 2023 at 11:50 AM <[email protected]> wrote:

From: Amir Goldstein <[email protected]>

Add test coverage for following metacopy from lower layer to lower data with absolute, relative and no redirect.

Quoted 127 lines
---
 tests/overlay/060     | 43 ++++++++++++++++++++++++++++++++++++++-----
 tests/overlay/060.out | 18 ++++++++++++++++++
 2 files changed, 56 insertions(+), 5 deletions(-)
tests/overlay/060 +38 -5
index 363207ba..f37755da 100755
@@ -7,7 +7,7 @@
 # Test metadata only copy up functionality.
 #
 . ./common/preamble
-_begin_fstest auto quick metacopy prealloc
+_begin_fstest auto quick metacopy redirect prealloc
 
 # Import common functions.
 . ./common/filter
@@ -123,6 +123,13 @@ mount_overlay()
        _overlay_scratch_mount_dirs "$_lowerdir" $upperdir $workdir -o redirect_dir=on,index=on,metacopy=on
 }
 
+mount_ro_overlay()
+{
+       local _lowerdir=$1
+
+       _overlay_scratch_mount_dirs "$_lowerdir" "-" "-" -o ro,redirect_dir=follow,metacopy=on
+}
+
 umount_overlay()
 {
        $UMOUNT_PROG $SCRATCH_MNT
@@ -146,7 +153,8 @@ test_common()
        check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"
        check_file_blocks $SCRATCH_MNT/$_target $_blocks
 
-       # Make sure copied up file is a metacopy file.
+       # Trigger metadata copy up and check existence of metacopy xattr.
+       chmod 400 $SCRATCH_MNT/$_target
        umount_overlay
        check_metacopy $upperdir/$_target "y"
        check_file_size_contents $upperdir/$_target $_size ""
@@ -165,7 +173,7 @@ test_common()
 create_basic_files()
 {
        _scratch_mkfs
-       mkdir -p $lowerdir $lowerdir2 $upperdir $workdir $workdir2
+       mkdir -p $lowerdir/subdir $lowerdir2 $upperdir $workdir $workdir2
        mkdir -p $upperdir/$udirname
        echo "$lowerdata" > $lowerdir/$lowername
        chmod 600 $lowerdir/$lowername
@@ -184,12 +192,19 @@ create_lower_link()
 
 prepare_midlayer()
 {
+       local _redirect=$1
+
        _scratch_mkfs
        create_basic_files
+       [ -n "$_redirect" ] && mv "$lowerdir/$lowername" "$lowerdir/$_redirect"
        # Create midlayer
        _overlay_scratch_mount_dirs $lowerdir $lowerdir2 $workdir2 -o redirect_dir=on,index=on,metacopy=on
-       # Trigger a metacopy
-       chmod 400 $SCRATCH_MNT/$lowername
+       # Trigger a metacopy with or without redirect
+       if [ -n "$_redirect" ]; then
+               mv "$SCRATCH_MNT/$_redirect" "$SCRATCH_MNT/$lowername"
+       else
+               chmod 400 $SCRATCH_MNT/$lowername
+       fi
        umount_overlay
 }
 
@@ -254,6 +269,24 @@ mount_overlay $lowerdir
 mv $SCRATCH_MNT/$lowerlink $SCRATCH_MNT/$ufile
 test_common $lowerdir $ufile $lowersize $lowerblocks "$lowerdata" "/$lowerlink"
 
+echo -e "\n== Check follow to lowerdata without redirect =="
+prepare_midlayer
+mount_ro_overlay "$lowerdir2:$lowerdir"
+test_common "$lowerdir2:$lowerdir" $lowername $lowersize $lowerblocks \
+               "$lowerdata"
+
+echo -e "\n== Check follow to lowerdata with relative redirect =="
+prepare_midlayer "$lowername.renamed"
+mount_ro_overlay "$lowerdir2:$lowerdir"
+test_common "$lowerdir2:$lowerdir" "$lowername" $lowersize $lowerblocks \
+               "$lowerdata"
+
+echo -e "\n== Check follow to lowerdata with absolute redirect =="
+prepare_midlayer "/subdir/$lowername"
+mount_ro_overlay "$lowerdir2:$lowerdir"
+test_common "$lowerdir2:$lowerdir" "$lowername" $lowersize $lowerblocks \
+               "$lowerdata"
+
 # success, all done
 status=0
 exit
tests/overlay/060.out +18 -1
index a4790d31..f4ce0244 100644
@@ -40,3 +40,21 @@ check properties of metadata copied up file
 Unmount and Mount again
 check properties of metadata copied up file
 check properties of data copied up file
+
+== Check follow to lowerdata without redirect ==
+check properties of metadata copied up file
+Unmount and Mount again
+check properties of metadata copied up file
+check properties of data copied up file
+
+== Check follow to lowerdata with relative redirect ==
+check properties of metadata copied up file
+Unmount and Mount again
+check properties of metadata copied up file
+check properties of data copied up file
+
+== Check follow to lowerdata with absolute redirect ==
+check properties of metadata copied up file
+Unmount and Mount again
+check properties of metadata copied up file
+check properties of data copied up file
--

2.40.1

Signature · 3 lines
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                Red Hat, Inc
       [email protected]         [email protected]
#6 lore raw

PATCH 3/4 overlay: Add test for follow of lowerdata in data-only layers

In reply to a message not in the archive

From: Amir Goldstein <[email protected]>

Add test coverage for following metacopy from lower layer to data-only lower layers.

Data-only lower layers are configured using the syntax:
lowerdir=<lowerdir1>:<lowerdir2>::<lowerdata1>::<lowerdata2>.

Test that lowerdata files can be followed only by absolute redirect from lower layer.

Test that with two lowerdata dirs, we can lookup individual lowerdata files in both, and that a shared file is resolved from the uppermost lowerdata dir.

There is also test case for lazy-data lookups, where we remove the lowerdata file and validate that we get metadata from the metacopy file, but open fails.

---
 common/overlay        |  13 ++
 tests/overlay/079     | 325 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/079.out |  42 ++++++
 3 files changed, 380 insertions(+)
 create mode 100755 tests/overlay/079
 create mode 100644 tests/overlay/079.out
common/overlay +13 -0
index 452b3b09..816ed66d 100644
@@ -201,6 +201,19 @@ _require_scratch_overlay_features()
 	_scratch_unmount
 }
 
+# Check kernel support for <lowerdirs>::<lowerdatadir> format
+_require_scratch_overlay_lowerdata_layers()
+{
+	local lowerdirs="$OVL_BASE_SCRATCH_MNT/$OVL_UPPER::$OVL_BASE_SCRATCH_MNT/$OVL_LOWER"
+
+	_scratch_mkfs > /dev/null 2>&1
+	_overlay_scratch_mount_dirs "$lowerdirs" "-" "-" \
+		-o ro,redirect_dir=follow,metacopy=on > /dev/null 2>&1 || \
+	        _notrun "overlay data-only layers not supported on ${SCRATCH_DEV}"
+
+	_scratch_unmount
+}
+
 # Helper function to check underlying dirs of overlay filesystem
 _overlay_fsck_dirs()
 {
tests/overlay/079 new file +325 -0

325 lines not on this page · raw message

tests/overlay/079.out new file +42 -0

42 lines not on this page · raw message

Signature · 1 line
2.40.1
#7 lore raw

PATCH 4/4 overlay: Add test coverage for fs-verity support

In reply to a message not in the archive

From: Alexander Larsson <[email protected]>

This tests that the right xattrs are set during copy-up, and that we properly fail on missing of erronous fs-verity digests when validating.

We also ensure that verity=require fails if a metacopy has not fs-verity, and doesn't do a meta-coopy-up if the base file lacks verity.

---
 common/overlay        |  14 ++
 common/verity         |  10 +-
 tests/overlay/080     | 326 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/080.out |   7 +
 4 files changed, 355 insertions(+), 2 deletions(-)
 create mode 100755 tests/overlay/080
 create mode 100644 tests/overlay/080.out
common/overlay +14 -0
index 816ed66d..7004187f 100644
@@ -201,6 +201,20 @@ _require_scratch_overlay_features()
 	_scratch_unmount
 }
 
+_require_scratch_overlay_verity()
+{
+	local lowerdirs="$OVL_BASE_SCRATCH_MNT/$OVL_UPPER:$OVL_BASE_SCRATCH_MNT/$OVL_LOWER"
+
+	_require_scratch_verity "$OVL_BASE_FSTYP" "$OVL_BASE_SCRATCH_MNT"
+
+	_scratch_mkfs > /dev/null 2>&1
+	_overlay_scratch_mount_dirs "$lowerdirs" "-" "-" \
+		-o ro,redirect_dir=follow,metacopy=on,verity=on > /dev/null 2>&1 || \
+	        _notrun "overlay verity not supported on ${SCRATCH_DEV}"
+
+	_scratch_unmount
+}
+
 # Check kernel support for <lowerdirs>::<lowerdatadir> format
 _require_scratch_overlay_lowerdata_layers()
 {
common/verity +8 -2
index 77c257d3..2b663210 100644
@@ -38,9 +38,12 @@ _require_scratch_verity()
 			"or mkfs options are not compatible with verity"
 	fi
 
+	local fstyp=${1:-$FSTYP}
+	local scratch_mnt=${2:-$SCRATCH_MNT}
+
 	# The filesystem may be aware of fs-verity but have it disabled by
 	# CONFIG_FS_VERITY=n.  Detect support via sysfs.
-	if [ ! -e /sys/fs/$FSTYP/features/verity ]; then
+	if [ ! -e /sys/fs/$fstyp/features/verity ]; then
 		_notrun "kernel $FSTYP isn't configured with verity support"
 	fi
 
@@ -68,7 +71,7 @@ _require_scratch_verity()
 	# The filesystem may have fs-verity enabled but not actually usable by
 	# default.  E.g., ext4 only supports verity on extent-based files, so it
 	# doesn't work on ext3-style filesystems.  So, try actually using it.
-	if ! _fsv_can_enable $SCRATCH_MNT/tmpfile; then
+	if ! _fsv_can_enable $scratch_mnt/tmpfile; then
 		_notrun "$FSTYP verity isn't usable by default with these mkfs options"
 	fi
 
@@ -201,6 +204,9 @@ _scratch_mkfs_verity()
 	btrfs)
 		_scratch_mkfs
 		;;
+	overlay)
+		_scratch_mkfs # This relies on the scratch fs supporting verity
+                ;;
 	*)
 		_notrun "No verity support for $FSTYP"
 		;;
tests/overlay/080 new file +326 -0

326 lines not on this page · raw message

tests/overlay/080.out new file +7 -0
new file mode 100644
index 00000000..7d1f1d10
@@ -0,0 +1,7 @@
+QA output created by 080
+
+== Check fsverity validation ==
+
+== Check fsverity require ==
+
+== Check fsverity copy-up ==
Signature · 1 line
2.40.1
#8 lore raw

Re: [PATCH 4/4] overlay: Add test coverage for fs-verity support

common/verity _require_scratch_verity() · line 42

Quoted 2 lines · On Thu, Jul 06, 2023 at 11:51:01AM +0200, [email protected] wrote:
common/verity +2 -0
+	local fstyp=${1:-$FSTYP}
+	local scratch_mnt=${2:-$SCRATCH_MNT}

Some code after this still uses $FSTYP and $SCRATCH_MNT. Is that a bug?

- Eric

#9 lore raw

Re: [PATCH 4/4] overlay: Add test coverage for fs-verity support

Quoted 6 lines · Eric Biggers

common/verity _require_scratch_verity() · line 42

Quoted 2 lines · On Thu, Jul 06, 2023 at 11:51:01AM +0200, [email protected] wrote:
common/verity +2 -0
+     local fstyp=${1:-$FSTYP}
+     local scratch_mnt=${2:-$SCRATCH_MNT}

Some code after this still uses $FSTYP and $SCRATCH_MNT. Is that a bug?

Yes. Will fix.

Signature · 3 lines
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                Red Hat, Inc
       [email protected]         [email protected]