Skip to contentSkip to conversation

overlayfs: Test data-only upperdirs and fs-verity

PATCH In review [email protected] 3 participants 7 messages Jul 10, 2023 – Jul 13, 2023

lore
Share & tools
lore
#0 lore raw

v2 cover overlayfs: Test data-only upperdirs and fs-verity

From: Alexander Larsson <[email protected]>

This adds support for testing the new data-only upperdir feature which is currently in master and will be in 6.5-rc1. It also adds tests for the fs-verity validation feature which is queued in overlayfs-next for 6.6.

All new tests check for the required features before running, so having it in early will be nice for testers of linux-next.

Changes since v1:
* Consistently use $fstyp and $scratch_mnt in _require_scratch_verity (Pointed out by Eric Biggers)
* Added Signed-off-by to patches from Amir

Alexander Larsson (1):
overlay: Add test coverage for fs-verity support

Amir Goldstein (3):
overlay: add helper for mounting rdonly overlay overlay/060: add test cases of follow to lowerdata
overlay: Add test for follow of lowerdata in data-only layers

 common/overlay        |  35 ++++-
 common/verity         |  16 ++-
 tests/overlay/060     |  43 +++++-
 tests/overlay/060.out |  18 +++
 tests/overlay/079     | 325 +++++++++++++++++++++++++++++++++++++++++
 tests/overlay/079.out |  42 ++++++
 tests/overlay/080     | 326 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/080.out |   7 +
 8 files changed, 800 insertions(+), 12 deletions(-)
 create mode 100755 tests/overlay/079
 create mode 100644 tests/overlay/079.out
 create mode 100755 tests/overlay/080
 create mode 100644 tests/overlay/080.out
Signature · 1 line
2.40.1
#1 lore raw

PATCH v2 1/4 overlay: add helper for mounting rdonly overlay

From: Amir Goldstein <[email protected]>

Allow passing empty upperdir to _overlay_mount_dirs().

---
 common/overlay | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
common/overlay +6 -2
index 20cafeb1..452b3b09 100644
@@ -17,15 +17,19 @@ if [ -n "$OVL_BASE_FSTYP" ];then
 fi
 
 # helper function to do the actual overlayfs mount operation
+# accepts "-" as upperdir for non-upper overlayfs
 _overlay_mount_dirs()
 {
 	local lowerdir=$1
 	local upperdir=$2
 	local workdir=$3
 	shift 3
+	local diropts="-olowerdir=$lowerdir"
 
-	$MOUNT_PROG -t overlay -o lowerdir=$lowerdir -o upperdir=$upperdir \
-		    -o workdir=$workdir `_common_dev_mount_options $*`
+	[ -n "$upperdir" ] && [ "$upperdir" != "-" ] && \
+		diropts+=",upperdir=$upperdir,workdir=$workdir"
+
+	$MOUNT_PROG -t overlay $diropts `_common_dev_mount_options $*`
 }
 
 # Mount with same options/mnt/dev of scratch mount, but optionally
Signature · 1 line
2.40.1
#2 lore raw

PATCH v2 2/4 overlay/060: add test cases of follow to lowerdata

In reply to [email protected]

From: Amir Goldstein <[email protected]>

Add test coverage for following metacopy from lower layer to lower data with absolute, relative and no redirect.

---
 tests/overlay/060     | 43 ++++++++++++++++++++++++++++++++++++++-----
 tests/overlay/060.out | 18 ++++++++++++++++++
 2 files changed, 56 insertions(+), 5 deletions(-)
tests/overlay/060 +38 -5
index 363207ba..f37755da 100755
@@ -7,7 +7,7 @@
 # Test metadata only copy up functionality.
 #
 . ./common/preamble
-_begin_fstest auto quick metacopy prealloc
+_begin_fstest auto quick metacopy redirect prealloc
 
 # Import common functions.
 . ./common/filter
@@ -123,6 +123,13 @@ mount_overlay()
 	_overlay_scratch_mount_dirs "$_lowerdir" $upperdir $workdir -o redirect_dir=on,index=on,metacopy=on
 }
 
+mount_ro_overlay()
+{
+	local _lowerdir=$1
+
+	_overlay_scratch_mount_dirs "$_lowerdir" "-" "-" -o ro,redirect_dir=follow,metacopy=on
+}
+
 umount_overlay()
 {
 	$UMOUNT_PROG $SCRATCH_MNT
@@ -146,7 +153,8 @@ test_common()
 	check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"
 	check_file_blocks $SCRATCH_MNT/$_target $_blocks
 
-	# Make sure copied up file is a metacopy file.
+	# Trigger metadata copy up and check existence of metacopy xattr.
+	chmod 400 $SCRATCH_MNT/$_target
 	umount_overlay
 	check_metacopy $upperdir/$_target "y"
 	check_file_size_contents $upperdir/$_target $_size ""
@@ -165,7 +173,7 @@ test_common()
 create_basic_files()
 {
 	_scratch_mkfs
-	mkdir -p $lowerdir $lowerdir2 $upperdir $workdir $workdir2
+	mkdir -p $lowerdir/subdir $lowerdir2 $upperdir $workdir $workdir2
 	mkdir -p $upperdir/$udirname
 	echo "$lowerdata" > $lowerdir/$lowername
 	chmod 600 $lowerdir/$lowername
@@ -184,12 +192,19 @@ create_lower_link()
 
 prepare_midlayer()
 {
+	local _redirect=$1
+
 	_scratch_mkfs
 	create_basic_files
+	[ -n "$_redirect" ] && mv "$lowerdir/$lowername" "$lowerdir/$_redirect"
 	# Create midlayer
 	_overlay_scratch_mount_dirs $lowerdir $lowerdir2 $workdir2 -o redirect_dir=on,index=on,metacopy=on
-	# Trigger a metacopy
-	chmod 400 $SCRATCH_MNT/$lowername
+	# Trigger a metacopy with or without redirect
+	if [ -n "$_redirect" ]; then
+		mv "$SCRATCH_MNT/$_redirect" "$SCRATCH_MNT/$lowername"
+	else
+		chmod 400 $SCRATCH_MNT/$lowername
+	fi
 	umount_overlay
 }
 
@@ -254,6 +269,24 @@ mount_overlay $lowerdir
 mv $SCRATCH_MNT/$lowerlink $SCRATCH_MNT/$ufile
 test_common $lowerdir $ufile $lowersize $lowerblocks "$lowerdata" "/$lowerlink"
 
+echo -e "\n== Check follow to lowerdata without redirect =="
+prepare_midlayer
+mount_ro_overlay "$lowerdir2:$lowerdir"
+test_common "$lowerdir2:$lowerdir" $lowername $lowersize $lowerblocks \
+		"$lowerdata"
+
+echo -e "\n== Check follow to lowerdata with relative redirect =="
+prepare_midlayer "$lowername.renamed"
+mount_ro_overlay "$lowerdir2:$lowerdir"
+test_common "$lowerdir2:$lowerdir" "$lowername" $lowersize $lowerblocks \
+		"$lowerdata"
+
+echo -e "\n== Check follow to lowerdata with absolute redirect =="
+prepare_midlayer "/subdir/$lowername"
+mount_ro_overlay "$lowerdir2:$lowerdir"
+test_common "$lowerdir2:$lowerdir" "$lowername" $lowersize $lowerblocks \
+		"$lowerdata"
+
 # success, all done
 status=0
 exit
tests/overlay/060.out +18 -0
index a4790d31..f4ce0244 100644
@@ -40,3 +40,21 @@ check properties of metadata copied up file
 Unmount and Mount again
 check properties of metadata copied up file
 check properties of data copied up file
+
+== Check follow to lowerdata without redirect ==
+check properties of metadata copied up file
+Unmount and Mount again
+check properties of metadata copied up file
+check properties of data copied up file
+
+== Check follow to lowerdata with relative redirect ==
+check properties of metadata copied up file
+Unmount and Mount again
+check properties of metadata copied up file
+check properties of data copied up file
+
+== Check follow to lowerdata with absolute redirect ==
+check properties of metadata copied up file
+Unmount and Mount again
+check properties of metadata copied up file
+check properties of data copied up file
Signature · 1 line
2.40.1
#3 lore raw

PATCH v2 3/4 overlay: Add test for follow of lowerdata in data-only layers

In reply to [email protected]

From: Amir Goldstein <[email protected]>

Add test coverage for following metacopy from lower layer to data-only lower layers.

Data-only lower layers are configured using the syntax:
lowerdir=<lowerdir1>:<lowerdir2>::<lowerdata1>::<lowerdata2>.

Test that lowerdata files can be followed only by absolute redirect from lower layer.

Test that with two lowerdata dirs, we can lookup individual lowerdata files in both, and that a shared file is resolved from the uppermost lowerdata dir.

There is also test case for lazy-data lookups, where we remove the lowerdata file and validate that we get metadata from the metacopy file, but open fails.

---
 common/overlay        |  13 ++
 tests/overlay/079     | 325 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/079.out |  42 ++++++
 3 files changed, 380 insertions(+)
 create mode 100755 tests/overlay/079
 create mode 100644 tests/overlay/079.out
common/overlay +13 -0
index 452b3b09..816ed66d 100644
@@ -201,6 +201,19 @@ _require_scratch_overlay_features()
 	_scratch_unmount
 }
 
+# Check kernel support for <lowerdirs>::<lowerdatadir> format
+_require_scratch_overlay_lowerdata_layers()
+{
+	local lowerdirs="$OVL_BASE_SCRATCH_MNT/$OVL_UPPER::$OVL_BASE_SCRATCH_MNT/$OVL_LOWER"
+
+	_scratch_mkfs > /dev/null 2>&1
+	_overlay_scratch_mount_dirs "$lowerdirs" "-" "-" \
+		-o ro,redirect_dir=follow,metacopy=on > /dev/null 2>&1 || \
+	        _notrun "overlay data-only layers not supported on ${SCRATCH_DEV}"
+
+	_scratch_unmount
+}
+
 # Helper function to check underlying dirs of overlay filesystem
 _overlay_fsck_dirs()
 {
tests/overlay/079 new file +325 -0

325 lines not on this page · raw message

tests/overlay/079.out new file +42 -0

42 lines not on this page · raw message

Signature · 1 line
2.40.1
#4 lore raw

PATCH v2 4/4 overlay: Add test coverage for fs-verity support

In reply to [email protected]

From: Alexander Larsson <[email protected]>

This tests that the right xattrs are set during copy-up, and that we properly fail on missing of erronous fs-verity digests when validating.

We also ensure that verity=require fails if a metacopy has not fs-verity, and doesn't do a meta-coopy-up if the base file lacks verity.

---
 common/overlay        |  14 ++
 common/verity         |  16 ++-
 tests/overlay/080     | 326 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/080.out |   7 +
 4 files changed, 358 insertions(+), 5 deletions(-)
 create mode 100755 tests/overlay/080
 create mode 100644 tests/overlay/080.out
common/overlay +14 -0
index 816ed66d..7004187f 100644
@@ -201,6 +201,20 @@ _require_scratch_overlay_features()
 	_scratch_unmount
 }
 
+_require_scratch_overlay_verity()
+{
+	local lowerdirs="$OVL_BASE_SCRATCH_MNT/$OVL_UPPER:$OVL_BASE_SCRATCH_MNT/$OVL_LOWER"
+
+	_require_scratch_verity "$OVL_BASE_FSTYP" "$OVL_BASE_SCRATCH_MNT"
+
+	_scratch_mkfs > /dev/null 2>&1
+	_overlay_scratch_mount_dirs "$lowerdirs" "-" "-" \
+		-o ro,redirect_dir=follow,metacopy=on,verity=on > /dev/null 2>&1 || \
+	        _notrun "overlay verity not supported on ${SCRATCH_DEV}"
+
+	_scratch_unmount
+}
+
 # Check kernel support for <lowerdirs>::<lowerdatadir> format
 _require_scratch_overlay_lowerdata_layers()
 {
common/verity +11 -5
index 77c257d3..e0937717 100644
@@ -38,10 +38,13 @@ _require_scratch_verity()
 			"or mkfs options are not compatible with verity"
 	fi
 
+	local fstyp=${1:-$FSTYP}
+	local scratch_mnt=${2:-$SCRATCH_MNT}
+
 	# The filesystem may be aware of fs-verity but have it disabled by
 	# CONFIG_FS_VERITY=n.  Detect support via sysfs.
-	if [ ! -e /sys/fs/$FSTYP/features/verity ]; then
-		_notrun "kernel $FSTYP isn't configured with verity support"
+	if [ ! -e /sys/fs/$fstyp/features/verity ]; then
+		_notrun "kernel $fstyp isn't configured with verity support"
 	fi
 
 	# Select a default Merkle tree block size for when tests don't
@@ -57,7 +60,7 @@ _require_scratch_verity()
 	#
 	# Therefore, we default to merkle_tree_block_size == min(fs_block_size,
 	# page_size).  That maximizes the chance of verity actually working.
-	local fs_block_size=$(_get_block_size $SCRATCH_MNT)
+	local fs_block_size=$(_get_block_size $scratch_mnt)
 	local page_size=$(get_page_size)
 	if (( fs_block_size <= page_size )); then
 		FSV_BLOCK_SIZE=$fs_block_size
@@ -68,8 +71,8 @@ _require_scratch_verity()
 	# The filesystem may have fs-verity enabled but not actually usable by
 	# default.  E.g., ext4 only supports verity on extent-based files, so it
 	# doesn't work on ext3-style filesystems.  So, try actually using it.
-	if ! _fsv_can_enable $SCRATCH_MNT/tmpfile; then
-		_notrun "$FSTYP verity isn't usable by default with these mkfs options"
+	if ! _fsv_can_enable $scratch_mnt/tmpfile; then
+		_notrun "$fstyp verity isn't usable by default with these mkfs options"
 	fi
 
 	_scratch_unmount
@@ -201,6 +204,9 @@ _scratch_mkfs_verity()
 	btrfs)
 		_scratch_mkfs
 		;;
+	overlay)
+		_scratch_mkfs # This relies on the scratch fs supporting verity
+                ;;
 	*)
 		_notrun "No verity support for $FSTYP"
 		;;
tests/overlay/080 new file +326 -0

326 lines not on this page · raw message

tests/overlay/080.out new file +7 -0
new file mode 100644
index 00000000..7d1f1d10
@@ -0,0 +1,7 @@
+QA output created by 080
+
+== Check fsverity validation ==
+
+== Check fsverity require ==
+
+== Check fsverity copy-up ==
Signature · 1 line
2.40.1
Reviewed-by on the v2 series
#5 lore raw
In reply to [email protected]
Quoted 14 lines · On Mon, Jul 10, 2023 at 11:07:09AM +0200, [email protected] wrote:

From: Alexander Larsson <[email protected]>

This adds support for testing the new data-only upperdir feature which is currently in master and will be in 6.5-rc1. It also adds tests for the fs-verity validation feature which is queued in overlayfs-next for 6.6.

All new tests check for the required features before running, so having it in early will be nice for testers of linux-next.

Changes since v1:
* Consistently use $fstyp and $scratch_mnt in _require_scratch_verity (Pointed out by Eric Biggers)
* Added Signed-off-by to patches from Amir

Quoted 7 lines

Alexander Larsson (1):
overlay: Add test coverage for fs-verity support

Amir Goldstein (3):
overlay: add helper for mounting rdonly overlay overlay/060: add test cases of follow to lowerdata
overlay: Add test for follow of lowerdata in data-only layers

This patchset looks good to me, and I didn't regression from it. So if there's not more review points from overlayfs side, I'll merge this patchset in next fstests release.

Thanks,
Zorro

Quoted 18 lines
 common/overlay        |  35 ++++-
 common/verity         |  16 ++-
 tests/overlay/060     |  43 +++++-
 tests/overlay/060.out |  18 +++
 tests/overlay/079     | 325 +++++++++++++++++++++++++++++++++++++++++
 tests/overlay/079.out |  42 ++++++
 tests/overlay/080     | 326 ++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/080.out |   7 +
 8 files changed, 800 insertions(+), 12 deletions(-)
 create mode 100755 tests/overlay/079
 create mode 100644 tests/overlay/079.out
 create mode 100755 tests/overlay/080
 create mode 100644 tests/overlay/080.out
Signature · 1 line
2.40.1
#6 lore raw
Quoted 19 lines · Zorro Lang
Quoted 17 lines · On Mon, Jul 10, 2023 at 11:07:09AM +0200, [email protected] wrote:

From: Alexander Larsson <[email protected]>

This adds support for testing the new data-only upperdir feature which is currently in master and will be in 6.5-rc1. It also adds tests for the fs-verity validation feature which is queued in overlayfs-next for 6.6.

All new tests check for the required features before running, so having it in early will be nice for testers of linux-next.

Changes since v1:
* Consistently use $fstyp and $scratch_mnt in _require_scratch_verity (Pointed out by Eric Biggers)
* Added Signed-off-by to patches from Amir

Alexander Larsson (1):
overlay: Add test coverage for fs-verity support

Quoted 9 lines
Quoted 4 lines

Amir Goldstein (3):
overlay: add helper for mounting rdonly overlay overlay/060: add test cases of follow to lowerdata
overlay: Add test for follow of lowerdata in data-only layers

This patchset looks good to me, and I didn't regression from it. So if there's not more review points from overlayfs side, I'll merge this patchset in next fstests release.

That would be great.

Thanks,
Amir.