Skip to contentSkip to conversation

xfstest: overlay: Add tests for overlay metadata only copy up feature

PATCH Superseded Vivek Goyal 3 participants 6 messages May 9, 2018 – May 29, 2018

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 xfstest: overlay: Add tests for overlay metadata only copy up feature

Hi,

Please find attached the V2 of the patch. Took care of Amir's comments from previous version.

Add tests for metadata only copy up feature.

---
 common/overlay        |    1 
 tests/overlay/059     |  283 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/059.out |   42 +++++++
 tests/overlay/group   |    1 
 4 files changed, 327 insertions(+)
tests/overlay/059 new file +283 -0
Index: xfstests-dev/tests/overlay/059
===================================================================
@@ -0,0 +1,283 @@
+#! /bin/bash
+# FS QA Test No. 059
+#
+# Test metadata only copy up functionality.
+#
+#-----------------------------------------------------------------------
+# Copyright (C) 2018 Red Hat, Inc. All Rights Reserved.
+# Author: Vivek Goyal <[email protected]>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs overlay
+_supported_os Linux
+# We use non-default scratch underlying overlay dirs, we need to check
+# them explicity after test.
+_require_scratch_nocheck
+_require_scratch_overlay_features index redirect_dir metacopy
+
+# remove all files from previous tests
+_scratch_mkfs
+
+# File size on lower
+lowername="lowerfile"
+lowerlink="lowerfile-link"
+lowerdata="lower"
+lowerblocks="32"
+lowersize=$(( $lowerblocks * 512 ))
+
+udirname="pureupper"
+ufile="upperfile"
+
+# Check metacopy xattr
+check_metacopy()
+{
+        local target=$1 exist=$2
+	local out_f target_f
+	local msg
+
+        out_f=$($GETFATTR_PROG --absolute-names --only-values -n \
+		$OVL_XATTR_METACOPY $target 2>&1 | _filter_scratch)
+
+	if [ "$exist" == "y" ];then
+		[ "$out_f" == "" ] && return
+		echo "Metacopy xattr does not exist on ${target}. stdout=$out_f"
+		return
+	fi
+
+	if [ "$out_f" == "" ];then
+		echo "Metacopy xattr exists on ${target} unexpectedly."
+		return
+	fi
+
+	target_f=`echo $target | _filter_scratch`
+	msg="$target_f: trusted.overlay.metacopy: No such attribute"
+
+	[ "$out_f" == "$msg" ] && return
+
+	echo "Error while checking xattr on ${target}. stdout=$out"
+}
+
+# Check redirect xattr
+check_redirect()
+{
+	local target=$1
+	local expect=$2
+
+	value=$($GETFATTR_PROG --absolute-names --only-values -n \
+		$OVL_XATTR_REDIRECT $target)
+
+	[[ "$value" == "$expect" ]] || echo "Redirect xattr incorrect. Expected=\"$expect\", actual=\"$value\""
+}
+
+# Check size
+check_file_size()
+{
+	local target=$1 expected_size=$2 actual_size
+
+	actual_size=$(stat -c "%s" $target)
+
+	[ "$actual_size" == "$expected_size" ] || echo "Expected file size $expected_size but actual size is $actual_size"
+}
+
+check_file_blocks()
+{
+	local target=$1 expected_blocks=$2 nr_blocks
+
+	nr_blocks=$(stat -c "%b" $target)
+
+	[ "$nr_blocks" == "$expected_blocks" ] || echo "Expected $expected_blocks blocks but actual number of blocks is ${nr_blocks}."
+}
+
+check_file_contents()
+{
+	local target=$1 expected=$2
+	local actual target_f
+
+	target_f=`echo $target | _filter_scratch`
+
+	read actual<$target
+
+	[ "$actual" == "$expected" ] || echo "Expected file $target_f contents to be \"$expected\" but actual contents are \"$actual\""
+}
+
+check_file_properties()
+{
+	local target=$1 expected_size=$2 expected_blocks=$3 expected_content=$4
+
+	check_file_size $target $expected_size
+	check_file_blocks $target $expected_blocks
+	check_file_contents $target $expected_content
+}
+
+mount_overlay()
+{
+	local _lowerdir=$1
+
+	_overlay_scratch_mount_dirs "$_lowerdir" $upperdir $workdir -o redirect_dir=on,index=on,metacopy=on
+}
+
+umount_overlay()
+{
+	$UMOUNT_PROG $SCRATCH_MNT
+}
+
+# Assumes it is called with overlay mounted.
+test_common()
+{
+	local _lowerdir=$1 _target=$2 _size=$3 _blocks=$4 _data="$5"
+	local _redirect=$6
+
+	echo "check properties of metadata copied up file"
+	check_file_properties $SCRATCH_MNT/$_target $_size $_blocks "$_data"
+
+	# Do a mount cycle and check size and contents again.
+	echo "Unmount and Mount again"
+	umount_overlay
+	mount_overlay $_lowerdir
+	echo "check properties of metadata copied up file"
+	check_file_properties $SCRATCH_MNT/$_target $_size $_blocks "$_data"
+
+	# Make sure copied up file is a metacopy file.
+	umount_overlay
+	check_metacopy $upperdir/$_target "y"
+	check_file_properties $upperdir/$_target $_size "0" ""
+	[ -n "$_redirect" ] && check_redirect $upperdir/$_target "$_redirect"
+
+	# Trigger data copy up and check absence of metacopy xattr.
+	mount_overlay $_lowerdir
+	$XFS_IO_PROG -c "open -a $SCRATCH_MNT/$_target"
+	echo "check properties of data copied up file"
+	check_file_properties $SCRATCH_MNT/$_target $_size $_blocks "$_data"
+	umount_overlay
+	check_metacopy $upperdir/$_target "n"
+	check_file_properties $upperdir/$_target $_size $_blocks "$_data"
+}
+
+create_basic_files()
+{
+	_scratch_mkfs
+	mkdir -p $lowerdir $lowerdir2 $upperdir $workdir $workdir2
+	mkdir -p $upperdir/$udirname
+	echo "$lowerdata" > $lowerdir/$lowername
+	chmod 600 $lowerdir/$lowername
+	# Create a file of size lowersize.
+	$XFS_IO_PROG -c "falloc 0 $lowersize" $lowerdir/$lowername
+}
+
+create_lower_link()
+{
+	ln $lowerdir/$lowername $lowerdir/$lowerlink
+}
+
+prepare_midlayer()
+{
+	_scratch_mkfs
+	create_basic_files
+	# Create midlayer
+	_overlay_scratch_mount_dirs $lowerdir $lowerdir2 $workdir2 -o redirect_dir=on,index=on,metacopy=on
+	# Trigger a metacopy
+	chmod 400 $SCRATCH_MNT/$lowername
+	umount_overlay
+}
+
+# Create test directories
+lowerdir=$OVL_BASE_SCRATCH_MNT/lower
+lowerdir2=$OVL_BASE_SCRATCH_MNT/lower2
+upperdir=$OVL_BASE_SCRATCH_MNT/upper
+workdir=$OVL_BASE_SCRATCH_MNT/workdir
+workdir2=$OVL_BASE_SCRATCH_MNT/workdir2
+
+# Tests start here
+echo "== Check Simple Metacopy =="
+create_basic_files
+mount_overlay $lowerdir
+chmod 400 $SCRATCH_MNT/$lowername
+test_common $lowerdir $lowername $lowersize $lowerblocks "$lowerdata"
+
+# Test midlayer metacopy
+echo -e "\n== Check Midlayer Metacopy =="
+prepare_midlayer
+mount_overlay "$lowerdir2:$lowerdir"
+chmod 400 $SCRATCH_MNT/$lowername
+test_common "$lowerdir2:$lowerdir" $lowername $lowersize $lowerblocks \
+		"$lowerdata"
+
+# Test Rename Redirect
+echo -e "\n== Check Rename Redirect =="
+create_basic_files
+mount_overlay $lowerdir
+mv $SCRATCH_MNT/$lowername $SCRATCH_MNT/$ufile
+test_common $lowerdir $ufile $lowersize $lowerblocks "$lowerdata" "$lowername"
+
+# Test midlayer rename redirect
+echo -e "\n== Check Midlayer Rename Redirect =="
+prepare_midlayer
+mount_overlay "$lowerdir2:$lowerdir"
+mv $SCRATCH_MNT/$lowername $SCRATCH_MNT/$ufile
+test_common "$lowerdir2:$lowerdir" $ufile $lowersize $lowerblocks "$lowerdata" \
+		"$lowername"
+
+# Test Link Redirect
+echo -e "\n== Check Link Redirect =="
+create_basic_files
+mount_overlay $lowerdir
+ln $SCRATCH_MNT/$lowername $SCRATCH_MNT/$udirname/$ufile
+test_common $lowerdir $udirname/$ufile $lowersize $lowerblocks "$lowerdata" \
+		"/$lowername"
+
+# Test midlayer link redirect
+echo -e "\n== Check Midlayer Link Redirect =="
+prepare_midlayer
+mount_overlay "$lowerdir2:$lowerdir"
+ln $SCRATCH_MNT/$lowername $SCRATCH_MNT/$udirname/$ufile
+test_common "$lowerdir2:$lowerdir" $udirname/$ufile $lowersize $lowerblocks \
+		"$lowerdata" "/$lowername"
+
+# Test lower link file gets absolute redirect upon rename
+echo -e "\n== Check Lower Link Rename Absolute Redirect =="
+create_basic_files
+create_lower_link
+mount_overlay $lowerdir
+mv $SCRATCH_MNT/$lowerlink $SCRATCH_MNT/$ufile
+test_common $lowerdir $ufile $lowersize $lowerblocks "$lowerdata" "/$lowerlink"
+
+# success, all done
+status=0
+exit
common/overlay +1 -0
Index: xfstests-dev/common/overlay
===================================================================
@@ -10,6 +10,7 @@ export OVL_XATTR_IMPURE="trusted.overlay
 export OVL_XATTR_ORIGIN="trusted.overlay.origin"
 export OVL_XATTR_NLINK="trusted.overlay.nlink"
 export OVL_XATTR_UPPER="trusted.overlay.upper"
+export OVL_XATTR_METACOPY="trusted.overlay.metacopy"
 
 # helper function to do the actual overlayfs mount operation
 _overlay_mount_dirs()
tests/overlay/059.out new file +42 -0

42 lines not on this page · raw message

tests/overlay/group +1 -0
Index: xfstests-dev/tests/overlay/group
===================================================================
@@ -61,3 +61,4 @@
 056 auto quick fsck
 057 auto quick redirect
 058 auto quick exportfs
+059 auto quick metacopy
Reviewed-by on v2 1/1
#1 lore raw
Quoted 8 lines · Vivek Goyal

Hi,

Please find attached the V2 of the patch. Took care of Amir's comments from previous version.

Add tests for metadata only copy up feature.

Looks good.

Just a note to Eryu -

You may want to hold on to this test until metacopy is merged or at least staged on Miklos' next branch.

Up to you.

Thanks,
Amir.

#2 lore raw
Quoted 6 lines · Amir Goldstein
Quoted 5 lines · Vivek Goyal

Hi,

Please find attached the V2 of the patch. Took care of Amir's comments from previous version.

Quoted 7 lines
Quoted 3 lines

Add tests for metadata only copy up feature.

Looks good.

Thanks for the review!

Quoted 5 lines

Just a note to Eryu -

You may want to hold on to this test until metacopy is merged or at least staged on Miklos' next branch.

I'd wait for the patches hit Miklos' tree.

Thanks,
Eryu

#3 lore raw
Quoted 18 lines · Eryu Guan
Quoted 6 lines · Amir Goldstein
Quoted 5 lines · Vivek Goyal

Hi,

Please find attached the V2 of the patch. Took care of Amir's comments from previous version.

Quoted 7 lines
Quoted 3 lines

Add tests for metadata only copy up feature.

Looks good.

Thanks for the review!

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

Just a note to Eryu -

You may want to hold on to this test until metacopy is merged or at least staged on Miklos' next branch.

I'd wait for the patches hit Miklos' tree.

Vivek,

This test seems to fail with overlayfs-next on check_file_blocks(). With ext4 underneath, blocks count is bigger than expected by 8 sectors. With xfs+reflink the blocks check also fails with different actual nr of blocks.

I think it is generally unreliable to try to compute nr of blocks and fail the test if actual is different than computed. Maybe just check that overlay nr blocks is as the actual lower nr of blocks.

Thanks,
Amir.

#4 lore raw
Quoted 29 lines · Amir Goldstein
Quoted 18 lines · Eryu Guan
Quoted 6 lines · Amir Goldstein
Quoted 5 lines · Vivek Goyal

Hi,

Please find attached the V2 of the patch. Took care of Amir's comments from previous version.

Quoted 7 lines
Quoted 3 lines

Add tests for metadata only copy up feature.

Looks good.

Thanks for the review!

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

Just a note to Eryu -

You may want to hold on to this test until metacopy is merged or at least staged on Miklos' next branch.

I'd wait for the patches hit Miklos' tree.

Vivek,

Quoted 8 lines

This test seems to fail with overlayfs-next on check_file_blocks(). With ext4 underneath, blocks count is bigger than expected by 8 sectors. With xfs+reflink the blocks check also fails with different actual nr of blocks.

I think it is generally unreliable to try to compute nr of blocks and fail the test if actual is different than computed. Maybe just check that overlay nr blocks is as the actual lower nr of blocks.

Hi Amir,

Ok, I will change test to only make sure lower blocks are same as overlay blocks for metacopy file.

But something strange is happening on ext4. I create a file and do stat on that it reports 40 blocks. After sleeping for 10 seconds, it reports 32 blocks on same file. Is this expected? I run following bash script on ext4 and I can reproduce the issue. Is this a bug in ext4?

Vivek

!/bin/bash
ext4file="/mnt/test-ext4/testfile.txt"

rm $ext4file >> /dev/null 2>&1
echo "hello" >> $ext4file
chmod 600 $ext4file
xfs_io -c "falloc 0 16384" $ext4file
stat $ext4file
sleep 10
stat $ext4file

#5 lore raw
Quoted 41 lines · Vivek Goyal
Quoted 29 lines · Amir Goldstein
Quoted 18 lines · Eryu Guan
Quoted 6 lines · Amir Goldstein
Quoted 5 lines · Vivek Goyal

Hi,

Please find attached the V2 of the patch. Took care of Amir's comments from previous version.

Quoted 7 lines
Quoted 3 lines

Add tests for metadata only copy up feature.

Looks good.

Thanks for the review!

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

Just a note to Eryu -

You may want to hold on to this test until metacopy is merged or at least staged on Miklos' next branch.

I'd wait for the patches hit Miklos' tree.

Vivek,

Quoted 8 lines

This test seems to fail with overlayfs-next on check_file_blocks(). With ext4 underneath, blocks count is bigger than expected by 8 sectors. With xfs+reflink the blocks check also fails with different actual nr of blocks.

I think it is generally unreliable to try to compute nr of blocks and fail the test if actual is different than computed. Maybe just check that overlay nr blocks is as the actual lower nr of blocks.

Hi Amir,

Quoted 7 lines

Ok, I will change test to only make sure lower blocks are same as overlay blocks for metacopy file.

But something strange is happening on ext4. I create a file and do stat on that it reports 40 blocks. After sleeping for 10 seconds, it reports 32 blocks on same file. Is this expected? I run following bash script on ext4 and I can reproduce the issue. Is this a bug in ext4?

Adding "xfs_io -c "fsync" $ext4file" after file creation fixes the anomaly.

Vivek

Quoted 15 lines

Vivek

!/bin/bash
ext4file="/mnt/test-ext4/testfile.txt"

rm $ext4file >> /dev/null 2>&1
echo "hello" >> $ext4file
chmod 600 $ext4file
xfs_io -c "falloc 0 16384" $ext4file
stat $ext4file
sleep 10
stat $ext4file