Skip to contentSkip to conversation

xfstest: overlay: Add tests for overlay metadata only copy up feature

PATCH Superseded Vivek Goyal 2 participants 3 messages May 29, 2018 – Jun 1, 2018

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 xfstest: overlay: Add tests for overlay metadata only copy up feature

Hi,

Find attached V3 of the patch. It now works on top of ext4 as well as xfs with reflink enabled. I removed some of nr_block checks which could not be guaranteed.

Add tests for metadata only copy up feature.

---
 common/overlay        |    1 
 tests/overlay/060     |  285 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/060.out |   42 +++++++
 tests/overlay/group   |    1 
 4 files changed, 329 insertions(+)
tests/overlay/060 new file +285 -0
Index: xfstests-dev/tests/overlay/060
===================================================================
@@ -0,0 +1,285 @@
+#! /bin/bash
+# FS QA Test No. 060
+#
+# Test metadata only copy up functionality.
+#
+#-----------------------------------------------------------------------
+# Copyright (C) 2018 Red Hat, Inc. All Rights Reserved.
+# Author: Vivek Goyal <[email protected]>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs overlay
+_supported_os Linux
+# We use non-default scratch underlying overlay dirs, we need to check
+# them explicity after test.
+_require_scratch_nocheck
+_require_scratch_overlay_features index redirect_dir metacopy
+
+# remove all files from previous tests
+_scratch_mkfs
+
+# File size on lower
+lowername="lowerfile"
+lowerlink="lowerfile-link"
+lowerdata="lower"
+lowerblocks="32"
+lowersize=$(( $lowerblocks * 512 ))
+
+udirname="pureupper"
+ufile="upperfile"
+
+# Check metacopy xattr
+check_metacopy()
+{
+        local target=$1 exist=$2
+	local out_f target_f
+	local msg
+
+        out_f=$($GETFATTR_PROG --absolute-names --only-values -n \
+		$OVL_XATTR_METACOPY $target 2>&1 | _filter_scratch)
+
+	if [ "$exist" == "y" ];then
+		[ "$out_f" == "" ] && return
+		echo "Metacopy xattr does not exist on ${target}. stdout=$out_f"
+		return
+	fi
+
+	if [ "$out_f" == "" ];then
+		echo "Metacopy xattr exists on ${target} unexpectedly."
+		return
+	fi
+
+	target_f=`echo $target | _filter_scratch`
+	msg="$target_f: trusted.overlay.metacopy: No such attribute"
+
+	[ "$out_f" == "$msg" ] && return
+
+	echo "Error while checking xattr on ${target}. stdout=$out"
+}
+
+# Check redirect xattr
+check_redirect()
+{
+	local target=$1
+	local expect=$2
+
+	value=$($GETFATTR_PROG --absolute-names --only-values -n \
+		$OVL_XATTR_REDIRECT $target)
+
+	[[ "$value" == "$expect" ]] || echo "Redirect xattr incorrect. Expected=\"$expect\", actual=\"$value\""
+}
+
+# Check size
+check_file_size()
+{
+	local target=$1 expected_size=$2 actual_size
+
+	actual_size=$(stat -c "%s" $target)
+
+	[ "$actual_size" == "$expected_size" ] || echo "Expected file size $expected_size but actual size is $actual_size"
+}
+
+check_file_blocks()
+{
+	local target=$1 expected_blocks=$2 nr_blocks
+
+	nr_blocks=$(stat -c "%b" $target)
+
+	[ "$nr_blocks" == "$expected_blocks" ] || echo "Expected $expected_blocks blocks but actual number of blocks is ${nr_blocks}."
+}
+
+check_file_contents()
+{
+	local target=$1 expected=$2
+	local actual target_f
+
+	target_f=`echo $target | _filter_scratch`
+
+	read actual<$target
+
+	[ "$actual" == "$expected" ] || echo "Expected file $target_f contents to be \"$expected\" but actual contents are \"$actual\""
+}
+
+check_file_size_contents()
+{
+	local target=$1 expected_size=$2 expected_content=$3
+
+	check_file_size $target $expected_size
+	check_file_contents $target $expected_content
+}
+
+mount_overlay()
+{
+	local _lowerdir=$1
+
+	_overlay_scratch_mount_dirs "$_lowerdir" $upperdir $workdir -o redirect_dir=on,index=on,metacopy=on
+}
+
+umount_overlay()
+{
+	$UMOUNT_PROG $SCRATCH_MNT
+}
+
+# Assumes it is called with overlay mounted.
+test_common()
+{
+	local _lowerdir=$1 _target=$2 _size=$3 _blocks=$4 _data="$5"
+	local _redirect=$6
+
+	echo "check properties of metadata copied up file"
+	check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"
+	check_file_blocks $SCRATCH_MNT/$_target $_blocks
+
+	# Do a mount cycle and check size and contents again.
+	echo "Unmount and Mount again"
+	umount_overlay
+	mount_overlay $_lowerdir
+	echo "check properties of metadata copied up file"
+	check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"
+	check_file_blocks $SCRATCH_MNT/$_target $_blocks
+
+	# Make sure copied up file is a metacopy file.
+	umount_overlay
+	check_metacopy $upperdir/$_target "y"
+	check_file_size_contents $upperdir/$_target $_size ""
+	[ -n "$_redirect" ] && check_redirect $upperdir/$_target "$_redirect"
+
+	# Trigger data copy up and check absence of metacopy xattr.
+	mount_overlay $_lowerdir
+	$XFS_IO_PROG -c "open -a $SCRATCH_MNT/$_target"
+	echo "check properties of data copied up file"
+	check_file_size_contents $SCRATCH_MNT/$_target $_size "$_data"
+	umount_overlay
+	check_metacopy $upperdir/$_target "n"
+	check_file_size_contents $upperdir/$_target $_size "$_data"
+}
+
+create_basic_files()
+{
+	_scratch_mkfs
+	mkdir -p $lowerdir $lowerdir2 $upperdir $workdir $workdir2
+	mkdir -p $upperdir/$udirname
+	echo "$lowerdata" > $lowerdir/$lowername
+	chmod 600 $lowerdir/$lowername
+	# Create a file of size lowersize.
+	$XFS_IO_PROG -c "falloc 0 $lowersize" $lowerdir/$lowername
+	$XFS_IO_PROG -c "fsync" $lowerdir/$lowername
+}
+
+create_lower_link()
+{
+	ln $lowerdir/$lowername $lowerdir/$lowerlink
+}
+
+prepare_midlayer()
+{
+	_scratch_mkfs
+	create_basic_files
+	# Create midlayer
+	_overlay_scratch_mount_dirs $lowerdir $lowerdir2 $workdir2 -o redirect_dir=on,index=on,metacopy=on
+	# Trigger a metacopy
+	chmod 400 $SCRATCH_MNT/$lowername
+	umount_overlay
+}
+
+# Create test directories
+lowerdir=$OVL_BASE_SCRATCH_MNT/lower
+lowerdir2=$OVL_BASE_SCRATCH_MNT/lower2
+upperdir=$OVL_BASE_SCRATCH_MNT/upper
+workdir=$OVL_BASE_SCRATCH_MNT/workdir
+workdir2=$OVL_BASE_SCRATCH_MNT/workdir2
+
+# Tests start here
+echo "== Check Simple Metacopy =="
+create_basic_files
+mount_overlay $lowerdir
+chmod 400 $SCRATCH_MNT/$lowername
+test_common $lowerdir $lowername $lowersize $lowerblocks "$lowerdata"
+
+# Test midlayer metacopy
+echo -e "\n== Check Midlayer Metacopy =="
+prepare_midlayer
+mount_overlay "$lowerdir2:$lowerdir"
+chmod 400 $SCRATCH_MNT/$lowername
+test_common "$lowerdir2:$lowerdir" $lowername $lowersize $lowerblocks \
+		"$lowerdata"
+
+# Test Rename Redirect
+echo -e "\n== Check Rename Redirect =="
+create_basic_files
+mount_overlay $lowerdir
+mv $SCRATCH_MNT/$lowername $SCRATCH_MNT/$ufile
+test_common $lowerdir $ufile $lowersize $lowerblocks "$lowerdata" "$lowername"
+
+# Test midlayer rename redirect
+echo -e "\n== Check Midlayer Rename Redirect =="
+prepare_midlayer
+mount_overlay "$lowerdir2:$lowerdir"
+mv $SCRATCH_MNT/$lowername $SCRATCH_MNT/$ufile
+test_common "$lowerdir2:$lowerdir" $ufile $lowersize $lowerblocks "$lowerdata" \
+		"$lowername"
+
+# Test Link Redirect
+echo -e "\n== Check Link Redirect =="
+create_basic_files
+mount_overlay $lowerdir
+ln $SCRATCH_MNT/$lowername $SCRATCH_MNT/$udirname/$ufile
+test_common $lowerdir $udirname/$ufile $lowersize $lowerblocks "$lowerdata" \
+		"/$lowername"
+
+# Test midlayer link redirect
+echo -e "\n== Check Midlayer Link Redirect =="
+prepare_midlayer
+mount_overlay "$lowerdir2:$lowerdir"
+ln $SCRATCH_MNT/$lowername $SCRATCH_MNT/$udirname/$ufile
+test_common "$lowerdir2:$lowerdir" $udirname/$ufile $lowersize $lowerblocks \
+		"$lowerdata" "/$lowername"
+
+# Test lower link file gets absolute redirect upon rename
+echo -e "\n== Check Lower Link Rename Absolute Redirect =="
+create_basic_files
+create_lower_link
+mount_overlay $lowerdir
+mv $SCRATCH_MNT/$lowerlink $SCRATCH_MNT/$ufile
+test_common $lowerdir $ufile $lowersize $lowerblocks "$lowerdata" "/$lowerlink"
+
+# success, all done
+status=0
+exit
common/overlay +1 -0
Index: xfstests-dev/common/overlay
===================================================================
@@ -10,6 +10,7 @@ export OVL_XATTR_IMPURE="trusted.overlay
 export OVL_XATTR_ORIGIN="trusted.overlay.origin"
 export OVL_XATTR_NLINK="trusted.overlay.nlink"
 export OVL_XATTR_UPPER="trusted.overlay.upper"
+export OVL_XATTR_METACOPY="trusted.overlay.metacopy"
 
 # helper function to do the actual overlayfs mount operation
 _overlay_mount_dirs()
tests/overlay/060.out new file +42 -0

42 lines not on this page · raw message

tests/overlay/group +1 -0
Index: xfstests-dev/tests/overlay/group
===================================================================
@@ -62,3 +62,4 @@
 057 auto quick redirect
 058 auto quick exportfs
 059 auto quick copyup
+060 auto quick metacopy
Reviewed-by on v3 1/1
#1 lore raw
Quoted 4 lines · Vivek Goyal

Hi,

Find attached V3 of the patch. It now works on top of ext4 as well as xfs with reflink enabled.

I can confirm that.

Quoted 6 lines

I removed some of nr_block checks which could
not be guaranteed.

Add tests for metadata only copy up feature.

You may add

Once one last comment is addressed

Quoted 66 lines
---
 common/overlay        |    1
 tests/overlay/060     |  285 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/060.out |   42 +++++++
 tests/overlay/group   |    1
 4 files changed, 329 insertions(+)

Index: xfstests-dev/tests/overlay/060
===================================================================

tests/overlay/060 2018-05-29 15:34:52.430714715 -0400 renamed from dev/null 1970-01-01 00:00:00.000000000 +0000 +54 -0
@@ -0,0 +1,285 @@
+#! /bin/bash
+# FS QA Test No. 060
+#
+# Test metadata only copy up functionality.
+#
+#-----------------------------------------------------------------------
+# Copyright (C) 2018 Red Hat, Inc. All Rights Reserved.
+# Author: Vivek Goyal <[email protected]>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1       # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+       cd /
+       rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs overlay
+_supported_os Linux
+# We use non-default scratch underlying overlay dirs, we need to check
+# them explicity after test.
+_require_scratch_nocheck
+_require_scratch_overlay_features index redirect_dir metacopy
+

tests/overlay/060 2018-05-29 15:34:52.430714715 -0400 line 64

Quoted 10 lines
tests/overlay/060 2018-05-29 15:34:52.430714715 -0400 renamed from dev/null 1970-01-01 00:00:00.000000000 +0000 +10 -0
+# remove all files from previous tests
+_scratch_mkfs
+
+# File size on lower
+lowername="lowerfile"
+lowerlink="lowerfile-link"
+lowerdata="lower"
+lowerblocks="32"
+lowersize=$(( $lowerblocks * 512 ))
+

Although the test passes now on ext4/xfs, my comment remains - it makes no sense for this test to calculate expected nr of blocks instead of using actual lower file blocks.

While fallocate $size guaranties that resulting file size is $size you cannot say the same about $blocks. There is no guarantie that file systems will use $size / 512 blocks to store size$. file systems can and do allocate more blocks for e.g. xattr and other file metadata. There is really no reason for this test to make the assumption about $blocks however probable it may be.

Thanks,
Amir.

#2 lore raw

On Fri, Jun 01, 2018 at 11:34:38AM +0300, Amir Goldstein wrote:

[..]

Quoted 12 lines

tests/overlay/060 line 64

Quoted 7 lines
tests/overlay/060 new file +7 -0
+# File size on lower
+lowername="lowerfile"
+lowerlink="lowerfile-link"
+lowerdata="lower"
+lowerblocks="32"
+lowersize=$(( $lowerblocks * 512 ))
+

Although the test passes now on ext4/xfs, my comment remains - it makes no sense for this test to calculate expected nr of blocks instead of using actual lower file blocks.

Quoted 6 lines

While fallocate $size guaranties that resulting file size is $size you cannot say the same about $blocks. There is no guarantie that file systems will use $size / 512 blocks to store size$. file systems can and do allocate more blocks for e.g. xattr and other file metadata. There is really no reason for this test to make the assumption about $blocks however probable it may be.

Ok. So instead of making assumptions about number of blocks, instead, create a file of pre-determined size, and then query the number of blocks from lower and use that as expected blocks from overlay mount?

Will make that change. Thanks.

Vivek