Skip to contentSkip to conversation

fstests: various fixes

PATCH In review Darrick J. Wong 3 participants 21 messages Mar 20, 2019 – Mar 26, 2019

lore
Share & tools
lore
#0 lore raw

cover fstests: various fixes

Hi all,

Here are a bunch of fixes and some new tests.

Patches 1-2 improve the random number algorithm that is used to shuffle the test list when -r is used.

Patches 3-6 fix some minor problems with existing tests.

Patches 7-8 refactor the code that creates and maintains the cache of populated filesystem images so that we can cache multiple different configurations on the test filesystem, since it is advantageous to be able to populate different configurations without discarding perfectly good images.

Patch 9 implements a bunch more reflink corner cases when sharing into or out from a bunch of different types of blocks.

Patch 10 ensures that xfs_copy and xfs_metadump can deal with all types of xfs metadata.

Patch 11-12 wipe the scratch devices between tests and fix the tests that erroneously expected the scratch devices to contain anything.

If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=djwong-devel

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=djwong-devel

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=djwong-devel

#1 lore raw

PATCH 1/12 check: improve test list randomization

From: Darrick J. Wong <[email protected]>

awk doesn't have a particularly good random number generator -- it seeds from the Unix epoch time in seconds, which means that the run order across a bunch of VMs started at exactly the same time are unsettlingly predictable. Therefore, at least try to seed it with bash's $RANDOM, which is slightly less predictable.

---
 check         |   12 ++++++------
 randomize.awk |   12 ++++++++----
 2 files changed, 14 insertions(+), 10 deletions(-)
check +6 -6
index 20e95302..ecd1d39a 100755
@@ -242,13 +242,13 @@ _prepare_test_list()
 	done
 
 	# sort the list of tests into numeric order
-	list=`sort -n $tmp.list | uniq`
-	rm -f $tmp.list
-
-	if $randomize
-	then
-		list=`echo $list | awk -f randomize.awk`
+	if $randomize; then
+		sorter="awk -v seed=$RANDOM -f randomize.awk"
+	else
+		sorter="cat"
 	fi
+	list=`sort -n $tmp.list | uniq | $sorter`
+	rm -f $tmp.list
 }
 
 # Process command arguments first.
randomize.awk +8 -4
index 0a8ad71a..d979fb03 100644
@@ -15,10 +15,14 @@ function randomize(array, N) {
 return
 }
 
+BEGIN {
+	srand(seed)
+}
 {
-    srand()
-    for (i = 0; i < NF; i++ ) array[i] = $(i+1)
-    randomize(array, NF)
-    for (i = 0; i < NF; i++) printf("%s ", array[i])
+	array[NR - 1] = $0
+}
+END {
+    randomize(array, NR)
+    for (i = 0; i < NR; i++) printf("%s ", array[i])
 }
 
#2 lore raw

PATCH 2/12 check: really improve test list randomization

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

coreutils provides the shuf(1) utility that randomizes the order of a list and seeds its random number generator with /dev/urandom. It's a bit speedier than awk, so use it if available.

---
 check |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
check +5 -1
index ecd1d39a..a2c5ba21 100755
@@ -243,7 +243,11 @@ _prepare_test_list()
 
 	# sort the list of tests into numeric order
 	if $randomize; then
-		sorter="awk -v seed=$RANDOM -f randomize.awk"
+		if type shuf >& /dev/null; then
+			sorter="shuf"
+		else
+			sorter="awk -v seed=$RANDOM -f randomize.awk"
+		fi
 	else
 		sorter="cat"
 	fi
#3 lore raw

PATCH 3/12 generic/042: fix stale disk contents check

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

This test doesn't call fsync or sync to force writeback of the first 60k of the file, which means that we could end up with a file full of zeroes or an empty file. Since this is a regression test that looks for stale disk contents slipping through, change the test to look for the stale bytes.

---
 tests/generic/042 |   10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
tests/generic/042 +7 -3
index 1aa18f9b..6c62eb63 100755
@@ -44,7 +44,7 @@ _crashtest()
 
 	# Create an fs on a small, initialized image. The pattern is written to
 	# the image to detect stale data exposure.
-	$XFS_IO_PROG -f -c "truncate 0" -c "pwrite 0 25M" $img \
+	$XFS_IO_PROG -f -c "truncate 0" -c "pwrite -S 0xCD 0 25M" $img \
 		>> $seqres.full 2>&1
 	_mkfs_dev $img >> $seqres.full 2>&1
 
@@ -61,8 +61,12 @@ _crashtest()
 	$UMOUNT_PROG $mnt
 	_mount $img $mnt
 
-	# we generally expect a zero-sized file (this should be silent)
-	hexdump $file
+	# We should /never/ see 0xCD in the file, because we wrote that pattern
+	# to the filesystem image to expose stale data.
+	if hexdump -v -e '/1 "%02X "' $file | grep -q "CD"; then
+		echo "Saw stale data!!!"
+		hexdump $file
+	fi
 
 	$UMOUNT_PROG $mnt
 }
#4 lore raw

PATCH 4/12 generic/032: fix unwritten extent checks

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Fix the unwritten extent detector in this test to ignore post-eof allocations because those are harmless.

---
 tests/generic/032 |   17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)
tests/generic/032 +10 -7
index affde4b7..38352a21 100755
@@ -69,18 +69,21 @@ do
 
 	# preallocate the first 64k and overwite, writing past 64k to contend
 	# with writeback
+	file_len=0x100000
 	$XFS_IO_PROG \
 		-c "falloc 0 0x10000"	\
-		-c "pwrite 0 0x100000"	\
+		-c "pwrite 0 $file_len"	\
 		-c "fsync"		\
 		$SCRATCH_MNT/file >> $seqres.full 2>&1
 
-	# Check for unwritten extents. We should have none since we wrote over
-	# the entire preallocated region and ran fsync.
-	$XFS_IO_PROG -c "fiemap -v" $SCRATCH_MNT/file | \
-		tee -a $seqres.full | \
-		_filter_fiemap | grep unwritten
-	[ $? == 0 ] && _fail "Unwritten extents found!"
+	# Check for unwritten extents. We should have none before EOF since we
+	# wrote over the entire preallocated region and ran fsync.
+	eof_sector=$(( file_len / 512 ))
+	$XFS_IO_PROG -c 'fiemap -v' $SCRATCH_MNT/file | \
+		_filter_fiemap | \
+		tr '[.]:' '    ' | \
+		awk "{if (\$2 < $eof_sector) {print \$0}}" | \
+		grep -q unwritten && _fail "Unwritten extents found!"
 done
 
 echo $iters iterations
#5 lore raw

PATCH 5/12 generic/454: stop the test if we run out of space

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Certain filesystems (ext4 w/ 1k block size) can run out of space while running this test because they have very limited xattr storage capabilities. If we run out of space while setting an attr, don't bother continuing the test.

---
 tests/generic/454 |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
tests/generic/454 +6 -1
index 4a0936c5..01986d6b 100755
@@ -48,7 +48,12 @@ setf() {
 	key="$(echo -e "$1")"
 	value="$2"
 
-	$SETFATTR_PROG -n "user.${key}" -v "${value}" "${testfile}"
+	$SETFATTR_PROG -n "user.${key}" -v "${value}" "${testfile}" > $tmp.output 2>&1
+	if [ $? -ne 0 ]; then
+		grep -q 'No space left on device' $tmp.output && \
+			_notrun "ran out of space"
+		cat $tmp.output
+	fi
 	echo "Storing ${key} ($(hexbytes "${key}")) -> ${value}" >> $seqres.full
 }
 
#6 lore raw

PATCH 6/12 ext4/023: don't require scrub for ext4 populated image creation

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Don't require scrub for ext4's populated fs creation test because there is no general online scrub program for ext*.

---
 tests/ext4/023 |    1 -
 1 file changed, 1 deletion(-)
tests/ext4/023 +0 -1
index a598e2b4..1dadb29f 100755
@@ -33,7 +33,6 @@ _cleanup()
 _supported_os Linux
 _supported_fs ext4
 _require_scratch
-_require_scrub
 
 echo "Format and populate"
 _scratch_populate_cached > $seqres.full 2>&1
#7 lore raw

PATCH 7/12 common/populate: refactor _scratch_populate_cached

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Refactor _scratch_populate_cached into smaller helper functions.

---
 common/populate |   69 ++++++++++++++++++++++++++++++++++---------------------
 1 file changed, 43 insertions(+), 26 deletions(-)
common/populate +43 -26
index 95953476..1f921ac5 100644
@@ -765,20 +765,14 @@ _fill_fs()
 	done
 }
 
-# Populate a scratch FS from scratch or from a cached image.
-_scratch_populate_cached() {
-	POPULATE_METADUMP="${TEST_DIR}/__populate.${FSTYP}"
-	POPULATE_METADUMP_DESCR="${TEST_DIR}/__populate.${FSTYP}.txt"
-
-	# Don't keep metadata images cached for more 48 hours...
-	rm -rf "$(find "${POPULATE_METADUMP}" -mtime +2 2>/dev/null)"
+# Compute the fs geometry description of a populated filesystem
+_scratch_populate_cache_tag() {
+	local extra_descr=""
+	local size="$(blockdev --getsz "${SCRATCH_DEV}")"
 
-	# Throw away cached image if it doesn't match our spec.
 	case "${FSTYP}" in
 	"ext4")
 		extra_descr="LOGDEV ${SCRATCH_LOGDEV} USE_EXTERNAL ${USE_EXTERNAL}"
-		# ext4 cannot e2image external logs, so we cannot restore
-		test -n "${SCRATCH_LOGDEV}" && rm -f "${POPULATE_METADUMP}"
 		;;
 	"xfs")
 		extra_descr="LOGDEV ${SCRATCH_LOGDEV} USE_EXTERNAL ${USE_EXTERNAL} RTDEV ${SCRATCH_RTDEV}"
@@ -787,23 +781,46 @@ _scratch_populate_cached() {
 			extra_descr="${extra_descr} QUOTAS"
 		fi
 		;;
-	*)
-		extra_descr="";;
 	esac
-	meta_descr="FSTYP ${FSTYP} MKFS_OPTIONS ${MKFS_OPTIONS} SIZE $(blockdev --getsz "${SCRATCH_DEV}") ${extra_descr} ARGS $@"
-	cmp -s "${POPULATE_METADUMP_DESCR}" <(echo "${meta_descr}") || rm -rf "${POPULATE_METADUMP}"
-
-	# Do we have a cached image?
-	if [ -r "${POPULATE_METADUMP}" ]; then
-		case "${FSTYP}" in
-		"xfs")
-			xfs_mdrestore "${POPULATE_METADUMP}" "${SCRATCH_DEV}" && return
-			;;
-		"ext2"|"ext3"|"ext4")
-			e2image -r "${POPULATE_METADUMP}" "${SCRATCH_DEV}" && return
-			;;
-		esac
-	fi
+	echo "FSTYP ${FSTYP} MKFS_OPTIONS ${MKFS_OPTIONS} SIZE ${size} ${extra_descr} ARGS $@"
+}
+
+# Restore a cached populated fs from a metadata dump
+_scratch_populate_restore_cached() {
+	local metadump="$1"
+
+	case "${FSTYP}" in
+	"xfs")
+		xfs_mdrestore "${metadump}" "${SCRATCH_DEV}" && return 0
+		;;
+	"ext2"|"ext3"|"ext4")
+		# ext4 cannot e2image external logs, so we cannot restore
+		test -n "${SCRATCH_LOGDEV}" && return 1
+		e2image -r "${metadump}" "${SCRATCH_DEV}" && return 0
+		;;
+	esac
+	return 1
+}
+
+# Populate a scratch FS from scratch or from a cached image.
+_scratch_populate_cached() {
+	local meta_descr="$(_scratch_populate_cache_tag "$@")"
+
+	# These variables are shared outside this function
+	POPULATE_METADUMP="${TEST_DIR}/__populate.${FSTYP}"
+	POPULATE_METADUMP_DESCR="${TEST_DIR}/__populate.${FSTYP}.txt"
+
+	# Don't keep metadata images cached for more 48 hours...
+	rm -rf "$(find "${POPULATE_METADUMP}" -mtime +2 2>/dev/null)"
+
+	# Throw away cached image if it doesn't match our spec.
+	cmp -s "${POPULATE_METADUMP_DESCR}" <(echo "${meta_descr}") || \
+		rm -rf "${POPULATE_METADUMP}"
+
+	# Try to restore from the metadump
+	test -r "${POPULATE_METADUMP}" && \
+		_scratch_populate_restore_cached "${POPULATE_METADUMP}" && \
+		return
 
 	# Oh well, just create one from scratch
 	_scratch_mkfs
#8 lore raw

PATCH 8/12 common/populate: support multiple cached images

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Enhance the populated fs metadump image cache to support multiple configurations per filesystem so that we reduce the image creation overhead even further.

---
 common/populate |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
common/populate +4 -2
index 1f921ac5..4fa118f0 100644
@@ -805,10 +805,12 @@ _scratch_populate_restore_cached() {
 # Populate a scratch FS from scratch or from a cached image.
 _scratch_populate_cached() {
 	local meta_descr="$(_scratch_populate_cache_tag "$@")"
+	local meta_tag="$(echo "${meta_descr}" | md5sum - | cut -d ' ' -f 1)"
+	local metadump_stem="${TEST_DIR}/__populate.${FSTYP}.${meta_tag}"
 
 	# These variables are shared outside this function
-	POPULATE_METADUMP="${TEST_DIR}/__populate.${FSTYP}"
-	POPULATE_METADUMP_DESCR="${TEST_DIR}/__populate.${FSTYP}.txt"
+	POPULATE_METADUMP="${metadump_stem}.metadump"
+	POPULATE_METADUMP_DESCR="${metadump_stem}.txt"
 
 	# Don't keep metadata images cached for more 48 hours...
 	rm -rf "$(find "${POPULATE_METADUMP}" -mtime +2 2>/dev/null)"
#9 lore raw

PATCH 9/12 clonerange: test remapping the rainbow

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Add some more clone range tests that missed various "wacky" combinations of file state. Specifically, we test reflinking into and out of rainbow ranges (a mix of real, unwritten, hole, delalloc, and shared extents), and also we test that we can correctly handle double-inode locking no matter what order of inodes or the filesystem's locking rules.

---
 common/reflink        |    7 ++
 tests/generic/940     |   94 ++++++++++++++++++++++++
 tests/generic/940.out |   14 ++++
 tests/generic/941     |   99 +++++++++++++++++++++++++
 tests/generic/941.out |   16 ++++
 tests/generic/942     |   95 ++++++++++++++++++++++++
 tests/generic/942.out |   14 ++++
 tests/generic/943     |   99 +++++++++++++++++++++++++
 tests/generic/943.out |   16 ++++
 tests/generic/944     |  196 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/944.out |   62 ++++++++++++++++
 tests/generic/group   |    5 +
 12 files changed, 716 insertions(+), 1 deletion(-)
 create mode 100755 tests/generic/940
 create mode 100644 tests/generic/940.out
 create mode 100755 tests/generic/941
 create mode 100644 tests/generic/941.out
 create mode 100755 tests/generic/942
 create mode 100644 tests/generic/942.out
 create mode 100755 tests/generic/943
 create mode 100644 tests/generic/943.out
 create mode 100755 tests/generic/944
 create mode 100644 tests/generic/944.out
common/reflink +6 -1
index 11561a76..598f0877 100644
@@ -14,9 +14,14 @@ _require_cp_reflink()
 # Can we reflink between arbitrary file sets?
 # i.e. if we reflink a->b and c->d, can we later share
 # blocks between b & c?
+_supports_arbitrary_fileset_reflink()
+{
+	test "$FSTYP" != "ocfs2"
+}
+
 _require_arbitrary_fileset_reflink()
 {
-	test "$FSTYP" = "ocfs2" && \
+	_supports_arbitrary_fileset_reflink ||
 		_notrun "reflink between arbitrary file groups not supported in $FSTYP"
 }
 
tests/generic/940 new file +94 -0

94 lines not on this page · raw message

tests/generic/940.out new file +14 -0
new file mode 100644
index 00000000..d34c7b50
@@ -0,0 +1,14 @@
+QA output created by 940
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-940/file1
+5a5221017d3ab8fd7583312a14d2ba80  SCRATCH_MNT/test-940/file2
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-940/file3
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-940/file3.chk
+reflink across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-940/file1
+5a5221017d3ab8fd7583312a14d2ba80  SCRATCH_MNT/test-940/file2
+5d47954e2336b2547afde6e44d2f13cc  SCRATCH_MNT/test-940/file3
+5d47954e2336b2547afde6e44d2f13cc  SCRATCH_MNT/test-940/file3.chk
tests/generic/941 new file +99 -0

99 lines not on this page · raw message

tests/generic/941.out new file +16 -0
new file mode 100644
index 00000000..a76e6c62
@@ -0,0 +1,16 @@
+QA output created by 941
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-941/file1
+5a5221017d3ab8fd7583312a14d2ba80  SCRATCH_MNT/test-941/file2
+5a5221017d3ab8fd7583312a14d2ba80  SCRATCH_MNT/test-941/file2.chk
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-941/file3
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-941/file3.chk
+reflink across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-941/file1
+51a300aae3a4b4eaa023876a397e01ef  SCRATCH_MNT/test-941/file2
+51a300aae3a4b4eaa023876a397e01ef  SCRATCH_MNT/test-941/file2.chk
+7bf7a779a0a54647b41753206c5218b1  SCRATCH_MNT/test-941/file3
+7bf7a779a0a54647b41753206c5218b1  SCRATCH_MNT/test-941/file3.chk
tests/generic/942 new file +95 -0

95 lines not on this page · raw message

tests/generic/942.out new file +14 -0
new file mode 100644
index 00000000..529ecff2
@@ -0,0 +1,14 @@
+QA output created by 942
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-942/file1
+5a5221017d3ab8fd7583312a14d2ba80  SCRATCH_MNT/test-942/file2
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-942/file3
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-942/file3.chk
+reflink across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-942/file1
+5a5221017d3ab8fd7583312a14d2ba80  SCRATCH_MNT/test-942/file2
+52bb341f992de6ef4bf5e5d61177eddc  SCRATCH_MNT/test-942/file3
+52bb341f992de6ef4bf5e5d61177eddc  SCRATCH_MNT/test-942/file3.chk
tests/generic/943 new file +99 -0

99 lines not on this page · raw message

tests/generic/943.out new file +16 -0
new file mode 100644
index 00000000..471c3f34
@@ -0,0 +1,16 @@
+QA output created by 943
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-943/file1
+5a5221017d3ab8fd7583312a14d2ba80  SCRATCH_MNT/test-943/file2
+5a5221017d3ab8fd7583312a14d2ba80  SCRATCH_MNT/test-943/file2.chk
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-943/file3
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-943/file3.chk
+reflink across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-943/file1
+d93123af536c8c012f866ea383a905ce  SCRATCH_MNT/test-943/file2
+d93123af536c8c012f866ea383a905ce  SCRATCH_MNT/test-943/file2.chk
+7bf7a779a0a54647b41753206c5218b1  SCRATCH_MNT/test-943/file3
+7bf7a779a0a54647b41753206c5218b1  SCRATCH_MNT/test-943/file3.chk
tests/generic/944 new file +196 -0

196 lines not on this page · raw message

tests/generic/944.out new file +62 -0

62 lines not on this page · raw message

tests/generic/group +5 -0
index 78b9b45d..2e4341fb 100644
@@ -538,3 +538,8 @@
 533 auto quick attr
 534 auto quick log
 535 auto quick log
+940 auto quick clone punch
+941 auto quick clone punch
+942 auto quick clone punch
+943 auto quick clone punch
+944 auto quick clone punch
#10 lore raw

PATCH 10/12 xfs: test xfs_copy and xfs_mdrestore on the populate images

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Make sure that copy, metadump, and mdrestore work on a filesystem with all known metadata types.

---
 tests/xfs/740     |   97 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/740.out |   12 +++++++
 tests/xfs/group   |    1 +
 3 files changed, 110 insertions(+)
 create mode 100755 tests/xfs/740
 create mode 100644 tests/xfs/740.out
tests/xfs/740 new file +97 -0
new file mode 100755
index 00000000..0ff548cb
@@ -0,0 +1,97 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright (c) 2019 Oracle, Inc.  All Rights Reserved.
+#
+# FS QA Test No. 740
+#
+# Populate a XFS filesystem and ensure that metadump, mdrestore, and copy
+# all work properly.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 7 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.* $testdir
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/populate
+
+testdir=$TEST_DIR/test-$seq
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs xfs
+
+_require_scratch_nocheck
+_require_populate_commands
+
+echo "Format and populate"
+_scratch_populate_cached nofill > $seqres.full 2>&1
+
+mkdir -p $testdir
+metadump_file=$testdir/scratch.md
+metadump_file_a=${metadump_file}.a
+metadump_file_g=${metadump_file}.g
+metadump_file_ag=${metadump_file}.ag
+copy_file=$testdir/copy.img
+
+echo metadump
+_scratch_metadump $metadump_file
+
+echo metadump a
+_scratch_metadump $metadump_file_a
+
+echo metadump g
+_scratch_metadump $metadump_file_g
+
+echo metadump ag
+_scratch_metadump $metadump_file_ag
+
+echo copy
+$XFS_COPY_PROG $SCRATCH_DEV $copy_file >> $seqres.full
+_check_scratch_fs $copy_file
+
+echo recopy
+$XFS_COPY_PROG $copy_file $SCRATCH_DEV >> $seqres.full
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore
+xfs_mdrestore $metadump_file $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore a
+xfs_mdrestore $metadump_file_a $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore g
+xfs_mdrestore $metadump_file_g $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore ag
+xfs_mdrestore $metadump_file_ag $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+# success, all done
+status=0
+exit
tests/xfs/740.out new file +12 -0
new file mode 100644
index 00000000..0c979dc4
@@ -0,0 +1,12 @@
+QA output created by 740
+Format and populate
+metadump
+metadump a
+metadump g
+metadump ag
+copy
+recopy
+mdrestore
+mdrestore a
+mdrestore g
+mdrestore ag
tests/xfs/group +1 -0
index 9d9458b8..b8bd1012 100644
@@ -500,3 +500,4 @@
 500 auto quick mkfs prealloc mkfs
 501 auto quick unlink
 502 auto quick unlink
+740 auto copy metadump
#11 lore raw

Re: [PATCH 10/12] xfs: test xfs_copy and xfs_mdrestore on the populate images

Quoted 70 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Make sure that copy, metadump, and mdrestore work on a filesystem with all known metadata types.

---
 tests/xfs/740     |   97 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/740.out |   12 +++++++
 tests/xfs/group   |    1 +
 3 files changed, 110 insertions(+)
 create mode 100755 tests/xfs/740
 create mode 100644 tests/xfs/740.out
tests/xfs/740 new file +49 -0
new file mode 100755
index 00000000..0ff548cb
@@ -0,0 +1,97 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright (c) 2019 Oracle, Inc.  All Rights Reserved.
+#
+# FS QA Test No. 740
+#
+# Populate a XFS filesystem and ensure that metadump, mdrestore, and copy
+# all work properly.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 7 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.* $testdir
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/populate
+
+testdir=$TEST_DIR/test-$seq
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs xfs
+
+_require_scratch_nocheck
+_require_populate_commands
+
+echo "Format and populate"
+_scratch_populate_cached nofill > $seqres.full 2>&1
+
+mkdir -p $testdir
+metadump_file=$testdir/scratch.md
+metadump_file_a=${metadump_file}.a
+metadump_file_g=${metadump_file}.g
+metadump_file_ag=${metadump_file}.ag
+copy_file=$testdir/copy.img
+
+echo metadump

tests/xfs/740 line 59

Quoted 10 lines
tests/xfs/740 new file +10 -0
+_scratch_metadump $metadump_file
+
+echo metadump a
+_scratch_metadump $metadump_file_a
+
+echo metadump g
+_scratch_metadump $metadump_file_g
+
+echo metadump ag
+_scratch_metadump $metadump_file_ag

I may miss something here, but I'm a bit confused, what's the difference between all these metadump files? Looks like scratch device is just dumped multiple times to different dump files? And later restored multiple times?

Thanks,
Eryu

Quoted 66 lines
tests/xfs/740 new file +38 -0
+
+echo copy
+$XFS_COPY_PROG $SCRATCH_DEV $copy_file >> $seqres.full
+_check_scratch_fs $copy_file
+
+echo recopy
+$XFS_COPY_PROG $copy_file $SCRATCH_DEV >> $seqres.full
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore
+xfs_mdrestore $metadump_file $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore a
+xfs_mdrestore $metadump_file_a $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore g
+xfs_mdrestore $metadump_file_g $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore ag
+xfs_mdrestore $metadump_file_ag $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+# success, all done
+status=0
+exit
tests/xfs/740.out new file +12 -0
new file mode 100644
index 00000000..0c979dc4
@@ -0,0 +1,12 @@
+QA output created by 740
+Format and populate
+metadump
+metadump a
+metadump g
+metadump ag
+copy
+recopy
+mdrestore
+mdrestore a
+mdrestore g
+mdrestore ag
tests/xfs/group +1 -0
index 9d9458b8..b8bd1012 100644
@@ -500,3 +500,4 @@
 500 auto quick mkfs prealloc mkfs
 501 auto quick unlink
 502 auto quick unlink
+740 auto copy metadump
#12 lore raw

Re: [PATCH 10/12] xfs: test xfs_copy and xfs_mdrestore on the populate images

Quoted 78 lines · Eryu Guan
Quoted 77 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Make sure that copy, metadump, and mdrestore work on a filesystem with all known metadata types.

---
 tests/xfs/740     |   97 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/740.out |   12 +++++++
 tests/xfs/group   |    1 +
 3 files changed, 110 insertions(+)
 create mode 100755 tests/xfs/740
 create mode 100644 tests/xfs/740.out
tests/xfs/740 new file +56 -0
new file mode 100755
index 00000000..0ff548cb
@@ -0,0 +1,97 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright (c) 2019 Oracle, Inc.  All Rights Reserved.
+#
+# FS QA Test No. 740
+#
+# Populate a XFS filesystem and ensure that metadump, mdrestore, and copy
+# all work properly.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 7 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.* $testdir
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/populate
+
+testdir=$TEST_DIR/test-$seq
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs xfs
+
+_require_scratch_nocheck
+_require_populate_commands
+
+echo "Format and populate"
+_scratch_populate_cached nofill > $seqres.full 2>&1
+
+mkdir -p $testdir
+metadump_file=$testdir/scratch.md
+metadump_file_a=${metadump_file}.a
+metadump_file_g=${metadump_file}.g
+metadump_file_ag=${metadump_file}.ag
+copy_file=$testdir/copy.img
+
+echo metadump
+_scratch_metadump $metadump_file
+
+echo metadump a
+_scratch_metadump $metadump_file_a
+
+echo metadump g
+_scratch_metadump $metadump_file_g
Quoted 8 lines

tests/xfs/740 line 59

Quoted 3 lines
tests/xfs/740 new file +3 -0
+
+echo metadump ag
+_scratch_metadump $metadump_file_ag

I may miss something here, but I'm a bit confused, what's the difference between all these metadump files? Looks like scratch device is just dumped multiple times to different dump files? And later restored multiple times?

Bah, that was *supposed* to be a test of the different flavors of metadump (copy entire blocks or not; obfuscate names or not) but clearly I thinko'd the metadump commands. :(

Where is this AI programming future where I can simply express a desire to test all the major format-altering metadump variants and the computer will figure out what needs to be done, rather than me having to write that idea in triplicate in the form of a log message, a file name, and a command line switch?

It's times like these I really hate programming. <grumble>

--D

Quoted 69 lines

Thanks,
Eryu

Quoted 66 lines
tests/xfs/740 new file +38 -0
+
+echo copy
+$XFS_COPY_PROG $SCRATCH_DEV $copy_file >> $seqres.full
+_check_scratch_fs $copy_file
+
+echo recopy
+$XFS_COPY_PROG $copy_file $SCRATCH_DEV >> $seqres.full
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore
+xfs_mdrestore $metadump_file $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore a
+xfs_mdrestore $metadump_file_a $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore g
+xfs_mdrestore $metadump_file_g $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore ag
+xfs_mdrestore $metadump_file_ag $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+# success, all done
+status=0
+exit
tests/xfs/740.out new file +12 -0
new file mode 100644
index 00000000..0c979dc4
@@ -0,0 +1,12 @@
+QA output created by 740
+Format and populate
+metadump
+metadump a
+metadump g
+metadump ag
+copy
+recopy
+mdrestore
+mdrestore a
+mdrestore g
+mdrestore ag
tests/xfs/group +1 -0
index 9d9458b8..b8bd1012 100644
@@ -500,3 +500,4 @@
 500 auto quick mkfs prealloc mkfs
 501 auto quick unlink
 502 auto quick unlink
+740 auto copy metadump
#13 lore raw

PATCH v2 10/12 xfs: test xfs_copy and xfs_mdrestore on the populate images

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Make sure that copy, metadump, and mdrestore work on a filesystem with all known metadata types.

---
 tests/xfs/740     |   97 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/740.out |   12 +++++++
 tests/xfs/group   |    1 +
 3 files changed, 110 insertions(+)
 create mode 100755 tests/xfs/740
 create mode 100644 tests/xfs/740.out
tests/xfs/740 new file +97 -0
new file mode 100755
index 00000000..bd0a4577
@@ -0,0 +1,97 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0+
+# Copyright (c) 2019 Oracle, Inc.  All Rights Reserved.
+#
+# FS QA Test No. 740
+#
+# Populate a XFS filesystem and ensure that metadump, mdrestore, and copy
+# all work properly.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 7 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.* $testdir
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/populate
+
+testdir=$TEST_DIR/test-$seq
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs xfs
+
+_require_scratch_nocheck
+_require_populate_commands
+
+echo "Format and populate"
+_scratch_populate_cached nofill > $seqres.full 2>&1
+
+mkdir -p $testdir
+metadump_file=$testdir/scratch.md
+metadump_file_a=${metadump_file}.a
+metadump_file_g=${metadump_file}.g
+metadump_file_ag=${metadump_file}.ag
+copy_file=$testdir/copy.img
+
+echo metadump
+_scratch_metadump $metadump_file >> $seqres.full
+
+echo metadump a
+_scratch_metadump $metadump_file_a -a >> $seqres.full
+
+echo metadump g
+_scratch_metadump $metadump_file_g -g >> $seqres.full
+
+echo metadump ag
+_scratch_metadump $metadump_file_ag -a -g >> $seqres.full
+
+echo copy
+$XFS_COPY_PROG $SCRATCH_DEV $copy_file >> $seqres.full
+_check_scratch_fs $copy_file
+
+echo recopy
+$XFS_COPY_PROG $copy_file $SCRATCH_DEV >> $seqres.full
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore
+xfs_mdrestore $metadump_file $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore a
+xfs_mdrestore $metadump_file_a $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore g
+xfs_mdrestore $metadump_file_g $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+echo mdrestore ag
+xfs_mdrestore $metadump_file_ag $SCRATCH_DEV
+_scratch_mount
+_check_scratch_fs
+_scratch_unmount
+
+# success, all done
+status=0
+exit
tests/xfs/740.out new file +12 -0
new file mode 100644
index 00000000..0c979dc4
@@ -0,0 +1,12 @@
+QA output created by 740
+Format and populate
+metadump
+metadump a
+metadump g
+metadump ag
+copy
+recopy
+mdrestore
+mdrestore a
+mdrestore g
+mdrestore ag
tests/xfs/group +1 -0
index 9d9458b8..b8bd1012 100644
@@ -500,3 +500,4 @@
 500 auto quick mkfs prealloc mkfs
 501 auto quick unlink
 502 auto quick unlink
+740 auto copy metadump
#14 lore raw

PATCH 11/12 check: wipe scratch devices between tests

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Wipe the scratch devices in between each test to ensure that tests are formatting them and not making assumptions about previous contents.

---
 check      |    1 +
 common/rc  |    6 ++++++
 common/xfs |    1 +
 3 files changed, 8 insertions(+)
check +1 -0
index a2c5ba21..bcf08dfe 100755
@@ -737,6 +737,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
 			# _check_dmesg depends on this log in dmesg
 			touch ${RESULT_DIR}/check_dmesg
 		fi
+		_try_wipe_scratch_devs > /dev/null 2>&1
 		if [ "$DUMP_OUTPUT" = true ]; then
 			./$seq 2>&1 | tee $tmp.out
 			# Because $? would get tee's return code
common/rc +6 -0
index 1c42515f..40eef80f 100644
@@ -3942,6 +3942,12 @@ _require_fibmap()
 	rm -f $file
 }
 
+_try_wipe_scratch_devs()
+{
+	_scratch_unmount
+	test -x "$WIPEFS_PROG" && $WIPEFS_PROG -a $SCRATCH_DEV_POOL $SCRATCH_DEV $SCRATCH_LOGDEV $SCRATCH_RTDEV
+}
+
 init_rc
 
 ################################################################################
common/xfs +1 -0
index 24065813..af2b62ba 100644
@@ -295,6 +295,7 @@ _require_xfs_db_command()
 	fi
 	command=$1
 
+	_scratch_mkfs_xfs >/dev/null 2>&1
 	_scratch_xfs_db -x -c "help" | grep $command > /dev/null || \
 		_notrun "xfs_db $command support is missing"
 }
#15 lore raw

Re: [PATCH 11/12] check: wipe scratch devices between tests

Quoted 31 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Wipe the scratch devices in between each test to ensure that tests are formatting them and not making assumptions about previous contents.

---
 check      |    1 +
 common/rc  |    6 ++++++
 common/xfs |    1 +
 3 files changed, 8 insertions(+)
check +1 -0
index a2c5ba21..bcf08dfe 100755
@@ -737,6 +737,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
                        # _check_dmesg depends on this log in dmesg
                        touch ${RESULT_DIR}/check_dmesg
                fi
+               _try_wipe_scratch_devs > /dev/null 2>&1
                if [ "$DUMP_OUTPUT" = true ]; then
                        ./$seq 2>&1 | tee $tmp.out
                        # Because $? would get tee's return code
common/rc +0 -0
index 1c42515f..40eef80f 100644

common/rc _require_fibmap() · line 3947

Quoted 6 lines
common/rc +3 -0
@@ -3942,6 +3942,12 @@ _require_fibmap()
        rm -f $file
 }
 
+_try_wipe_scratch_devs()
+{
+       _scratch_unmount

So I find this change strange, not because it doesn't make sense to start every test with scratch unmounted, but because today scratch *is* mounted on test start and there is quite a bit of code to make it mounted on test start, which seems backwards.

IOW, instead of unmounting scratch just before the test runs, seems better to make sure it is unmounted at the end of each test and before starting the loop.

Note that for a test with _require_scratch_nocheck, _check_filesystems() will unmount scratch, but for a test with _require_scratch, _check_scratch_fs() will unmount+fsck+mount scratch - inconsistent.
Or am I reading this wrong?

common/rc _require_fibmap() · line 3948

Quoted 1 line
common/rc +1 -0
+       test -x "$WIPEFS_PROG" && $WIPEFS_PROG -a $SCRATCH_DEV_POOL $SCRATCH_DEV $SCRATCH_LOGDEV $SCRATCH_RTDEV

Please check that $SCRATCH_DEV is not a network name (nfs|cifs) and that it is really a blockdev (overlayfs) before trying to wipefs.

Thanks,
Amir.

#16 lore raw

Re: [PATCH 11/12] check: wipe scratch devices between tests

Quoted 35 lines · Amir Goldstein
Quoted 34 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Wipe the scratch devices in between each test to ensure that tests are formatting them and not making assumptions about previous contents.

---
 check      |    1 +
 common/rc  |    6 ++++++
 common/xfs |    1 +
 3 files changed, 8 insertions(+)
check +1 -0
index a2c5ba21..bcf08dfe 100755
@@ -737,6 +737,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
                        # _check_dmesg depends on this log in dmesg
                        touch ${RESULT_DIR}/check_dmesg
                fi
+               _try_wipe_scratch_devs > /dev/null 2>&1
                if [ "$DUMP_OUTPUT" = true ]; then
                        ./$seq 2>&1 | tee $tmp.out
                        # Because $? would get tee's return code
common/rc +0 -0
index 1c42515f..40eef80f 100644
@@ -3942,6 +3942,12 @@ _require_fibmap()
        rm -f $file
 }
 
Quoted 8 lines

common/rc _require_fibmap() · line 3947

Quoted 3 lines
common/rc +3 -0
+_try_wipe_scratch_devs()
+{
+       _scratch_unmount

So I find this change strange, not because it doesn't make sense to start every test with scratch unmounted, but because today scratch *is* mounted on test start and there is quite a bit of code to make it mounted on test start, which seems backwards.

Huh? On XFS the test device is always mounted, but the scratch device is never mounted before the test starts.

Hmmm, maybe this is one of those weird things that different with nonblock filesystems...?

Quoted 10 lines

IOW, instead of unmounting scratch just before the test runs, seems better to make sure it is unmounted at the end of each test and before starting the loop.

Note that for a test with _require_scratch_nocheck, _check_filesystems() will unmount scratch, but for a test with _require_scratch, _check_scratch_fs() will unmount+fsck+mount scratch - inconsistent.
Or am I reading this wrong?

Quoted 4 lines

common/rc _require_fibmap() · line 3948

Quoted 1 line
common/rc +1 -0
+       test -x "$WIPEFS_PROG" && $WIPEFS_PROG -a $SCRATCH_DEV_POOL $SCRATCH_DEV $SCRATCH_LOGDEV $SCRATCH_RTDEV

Please check that $SCRATCH_DEV is not a network name (nfs|cifs) and that it is really a blockdev (overlayfs) before trying to wipefs.

I wonder if maybe we only bother with wipefs for filesystems where all tests have been fixed to not stumble over unformatted scratchdev (i.e. xfs and ext* only)

--D

Quoted 2 lines

Thanks,
Amir.

#17 lore raw

Re: [PATCH 11/12] check: wipe scratch devices between tests

Quoted 45 lines · Darrick J. Wong
Quoted 35 lines · Amir Goldstein
Quoted 34 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Wipe the scratch devices in between each test to ensure that tests are formatting them and not making assumptions about previous contents.

---
 check      |    1 +
 common/rc  |    6 ++++++
 common/xfs |    1 +
 3 files changed, 8 insertions(+)
check +1 -0
index a2c5ba21..bcf08dfe 100755
@@ -737,6 +737,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
                        # _check_dmesg depends on this log in dmesg
                        touch ${RESULT_DIR}/check_dmesg
                fi
+               _try_wipe_scratch_devs > /dev/null 2>&1
                if [ "$DUMP_OUTPUT" = true ]; then
                        ./$seq 2>&1 | tee $tmp.out
                        # Because $? would get tee's return code
common/rc +0 -0
index 1c42515f..40eef80f 100644
@@ -3942,6 +3942,12 @@ _require_fibmap()
        rm -f $file
 }
 
Quoted 8 lines

common/rc _require_fibmap() · line 3947

Quoted 3 lines
common/rc +3 -0
+_try_wipe_scratch_devs()
+{
+       _scratch_unmount

So I find this change strange, not because it doesn't make sense to start every test with scratch unmounted, but because today scratch *is* mounted on test start and there is quite a bit of code to make it mounted on test start, which seems backwards.

Quoted 2 lines

Huh? On XFS the test device is always mounted, but the scratch device is never mounted before the test starts.

Yes, as _require_test always mounts TEST_DEV and _require_scratch_nocheck always umounts SCRATCH_DEV.

Quoted 12 lines

Hmmm, maybe this is one of those weird things that different with nonblock filesystems...?

Quoted 8 lines

IOW, instead of unmounting scratch just before the test runs, seems better to make sure it is unmounted at the end of each test and before starting the loop.

Note that for a test with _require_scratch_nocheck, _check_filesystems() will unmount scratch, but for a test with _require_scratch, _check_scratch_fs() will unmount+fsck+mount scratch - inconsistent.
Or am I reading this wrong?

For tests that don't call _require_scratch{_nocheck}, to make sure SCRATCH_DEV is umounted I think we could just call _scratch_unmount unconditionally in _check_filesystems(), then we could wipefs before test safely.

Quoted 6 lines
Quoted 6 lines

common/rc _require_fibmap() · line 3948

Quoted 3 lines
common/rc +1 -0
+       test -x "$WIPEFS_PROG" && $WIPEFS_PROG -a $SCRATCH_DEV_POOL $SCRATCH_DEV $SCRATCH_LOGDEV $SCRATCH_RTDEV

Please check that $SCRATCH_DEV is not a network name (nfs|cifs) and that it is really a blockdev (overlayfs) before trying to wipefs.

Yes, good catch!

Thanks,
Eryu

Quoted 9 lines

I wonder if maybe we only bother with wipefs for filesystems where all tests have been fixed to not stumble over unformatted scratchdev (i.e. xfs and ext* only)

--D

Quoted 2 lines

Thanks,
Amir.

#18 lore raw

Re: [PATCH 11/12] check: wipe scratch devices between tests

Quoted 46 lines · Eryu Guan
Quoted 45 lines · Darrick J. Wong
Quoted 43 lines · Amir Goldstein
Quoted 31 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Wipe the scratch devices in between each test to ensure that tests are formatting them and not making assumptions about previous contents.

---
 check      |    1 +
 common/rc  |    6 ++++++
 common/xfs |    1 +
 3 files changed, 8 insertions(+)
check +1 -0
index a2c5ba21..bcf08dfe 100755
@@ -737,6 +737,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
                        # _check_dmesg depends on this log in dmesg
                        touch ${RESULT_DIR}/check_dmesg
                fi
+               _try_wipe_scratch_devs > /dev/null 2>&1
                if [ "$DUMP_OUTPUT" = true ]; then
                        ./$seq 2>&1 | tee $tmp.out
                        # Because $? would get tee's return code
common/rc +0 -0
index 1c42515f..40eef80f 100644

common/rc _require_fibmap() · line 3947

Quoted 6 lines
common/rc +3 -0
@@ -3942,6 +3942,12 @@ _require_fibmap()
        rm -f $file
 }
 
+_try_wipe_scratch_devs()
+{
+       _scratch_unmount

So I find this change strange, not because it doesn't make sense to start every test with scratch unmounted, but because today scratch *is* mounted on test start and there is quite a bit of code to make it mounted on test start, which seems backwards.

Quoted 5 lines
Quoted 2 lines

Huh? On XFS the test device is always mounted, but the scratch device is never mounted before the test starts.

Yes, as _require_test always mounts TEST_DEV and _require_scratch_nocheck always umounts SCRATCH_DEV.

Heh, both of you were right and I was wrong, we format and mount the scratch device before we start the loop.

Quoted 13 lines
Quoted 12 lines

Hmmm, maybe this is one of those weird things that different with nonblock filesystems...?

Quoted 8 lines

IOW, instead of unmounting scratch just before the test runs, seems better to make sure it is unmounted at the end of each test and before starting the loop.

Note that for a test with _require_scratch_nocheck, _check_filesystems() will unmount scratch, but for a test with _require_scratch, _check_scratch_fs() will unmount+fsck+mount scratch - inconsistent.
Or am I reading this wrong?

Quoted 4 lines

For tests that don't call _require_scratch{_nocheck}, to make sure SCRATCH_DEV is umounted I think we could just call _scratch_unmount unconditionally in _check_filesystems(), then we could wipefs before test safely.

I think it does this already:

	if [ -f ${RESULT_DIR}/require_scratch ]; then
		_check_scratch_fs || err=true
		rm -f ${RESULT_DIR}/require_scratch*
	else
		_scratch_unmount 2> /dev/null
	fi

Right? So I think we're covered for unmounting the scratch device after each test, and it's just the first time through the loop where we have to unmount the scratch device.

--D

Quoted 22 lines
Quoted 7 lines
Quoted 7 lines

common/rc _require_fibmap() · line 3948

Quoted 4 lines
common/rc +1 -0
+       test -x "$WIPEFS_PROG" && $WIPEFS_PROG -a $SCRATCH_DEV_POOL $SCRATCH_DEV $SCRATCH_LOGDEV $SCRATCH_RTDEV

Please check that $SCRATCH_DEV is not a network name (nfs|cifs) and that it is really a blockdev (overlayfs) before trying to wipefs.

Yes, good catch!

Thanks,
Eryu

Quoted 9 lines

I wonder if maybe we only bother with wipefs for filesystems where all tests have been fixed to not stumble over unformatted scratchdev (i.e. xfs and ext* only)

--D

Quoted 2 lines

Thanks,
Amir.

#19 lore raw

Re: [PATCH 11/12] check: wipe scratch devices between tests

Quoted 84 lines · Darrick J. Wong
Quoted 46 lines · Eryu Guan
Quoted 45 lines · Darrick J. Wong
Quoted 43 lines · Amir Goldstein
Quoted 31 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Wipe the scratch devices in between each test to ensure that tests are formatting them and not making assumptions about previous contents.

---
 check      |    1 +
 common/rc  |    6 ++++++
 common/xfs |    1 +
 3 files changed, 8 insertions(+)
check +1 -0
index a2c5ba21..bcf08dfe 100755
@@ -737,6 +737,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
                        # _check_dmesg depends on this log in dmesg
                        touch ${RESULT_DIR}/check_dmesg
                fi
+               _try_wipe_scratch_devs > /dev/null 2>&1
                if [ "$DUMP_OUTPUT" = true ]; then
                        ./$seq 2>&1 | tee $tmp.out
                        # Because $? would get tee's return code
common/rc +0 -0
index 1c42515f..40eef80f 100644

common/rc _require_fibmap() · line 3947

Quoted 6 lines
common/rc +3 -0
@@ -3942,6 +3942,12 @@ _require_fibmap()
        rm -f $file
 }
 
+_try_wipe_scratch_devs()
+{
+       _scratch_unmount

So I find this change strange, not because it doesn't make sense to start every test with scratch unmounted, but because today scratch *is* mounted on test start and there is quite a bit of code to make it mounted on test start, which seems backwards.

Quoted 5 lines
Quoted 2 lines

Huh? On XFS the test device is always mounted, but the scratch device is never mounted before the test starts.

Yes, as _require_test always mounts TEST_DEV and _require_scratch_nocheck always umounts SCRATCH_DEV.

Heh, both of you were right and I was wrong, we format and mount the scratch device before we start the loop.

Quoted 13 lines
Quoted 12 lines

Hmmm, maybe this is one of those weird things that different with nonblock filesystems...?

Quoted 8 lines

IOW, instead of unmounting scratch just before the test runs, seems better to make sure it is unmounted at the end of each test and before starting the loop.

Note that for a test with _require_scratch_nocheck, _check_filesystems() will unmount scratch, but for a test with _require_scratch, _check_scratch_fs() will unmount+fsck+mount scratch - inconsistent.
Or am I reading this wrong?

Quoted 4 lines

For tests that don't call _require_scratch{_nocheck}, to make sure SCRATCH_DEV is umounted I think we could just call _scratch_unmount unconditionally in _check_filesystems(), then we could wipefs before test safely.

I think it does this already:

        if [ -f ${RESULT_DIR}/require_scratch ]; then
                _check_scratch_fs || err=true
                rm -f ${RESULT_DIR}/require_scratch*
        else
                _scratch_unmount 2> /dev/null
        fi
Quoted 4 lines

Right? So I think we're covered for unmounting the scratch device after each test, and it's just the first time through the loop where we have to unmount the scratch device.

Right about _require_scratch_nocheck and no _require_scratch. Test with _require_scratch will call _check_filesystems(). _check_scratch_fs() will unmount scratch; fsck; mount scratch. So as far as I can see, its not only the first time through the loop.

Thanks,
Amir.

#20 lore raw

PATCH 12/12 misc: fix broken _require_scratch usage

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

_require_scratch doesn't actually format the scratch device with anything, which means that tests are required to format them before using them. Fix tests that don't do this correctly.

---
 tests/generic/026 |    1 -
 tests/generic/100 |    3 ---
 tests/generic/506 |    2 +-
 tests/xfs/065     |    2 +-
 tests/xfs/108     |    1 +
 tests/xfs/109     |    2 +-
 tests/xfs/194     |    4 +++-
 tests/xfs/261     |    2 ++
 tests/xfs/284     |    1 +
 9 files changed, 10 insertions(+), 8 deletions(-)
tests/generic/026 +0 -1
index 6cbd2829..31cd0827 100755
@@ -32,7 +32,6 @@ _cleanup()
 _supported_fs generic
 _supported_os Linux
 _require_test
-_require_scratch
 _acl_setup_ids
 _require_acls
 _require_acl_get_max
tests/generic/100 +0 -3
index c2593333..927c4949 100755
@@ -52,9 +52,6 @@ _populate_fs -n $NDIRS -f $NFILES -d $DEPTH -r $POPULATED_DIR -s $SIZE >>$seqres
 # Then tar up the directory structure
 tar -cvf $TEMP_DIR/$TAR_FILE $POPULATED_DIR >>$seqres.full 2>&1
 
-# create f/s
-_require_scratch
-
 # untar on f/s
 cd $TEST_DIR
 tar -xvf $TEMP_DIR/$TAR_FILE >>$seqres.full 2>&1
tests/generic/506 +1 -1
index 5d419cad..7002c00c 100755
@@ -46,12 +46,12 @@ _supported_os Linux
 _require_command "$LSATTR_PROG" lsattr
 _require_command "$CHATTR_PROG" chattr
 
-_require_prjquota $SCRATCH_DEV
 _require_scratch
 _require_scratch_shutdown
 
 _scratch_mkfs >/dev/null 2>&1
 _require_metadata_journaling $SCRATCH_DEV
+_require_prjquota $SCRATCH_DEV
 
 testfile=$SCRATCH_MNT/testfile
 
tests/xfs/065 +1 -1
index c3472486..c0d1ee93 100755
@@ -55,7 +55,7 @@ _supported_os Linux
 # too much hassle to get output matching with quotas turned on
 # so don't run it
 #
-_scratch_unmount 2>/dev/null
+_scratch_mkfs_xfs >> $seqres.full
 _scratch_mount
 $here/src/feature -U $SCRATCH_DEV && \
 	_notrun "UQuota enabled, test needs controlled xfsdump output"
tests/xfs/108 +1 -0
index c47f4f37..e70a1f9a 100755
@@ -61,6 +61,7 @@ test_accounting()
 }
 
 export MOUNT_OPTIONS="-opquota"
+_scratch_mkfs_xfs >> $seqres.full
 _qmount
 _require_prjquota $SCRATCH_DEV
 
tests/xfs/109 +1 -1
index a063dc3f..df4ec157 100755
@@ -75,7 +75,7 @@ allocate()
 }
 
 # real QA test starts here
-_scratch_unmount 2>/dev/null
+_scratch_mkfs_xfs >> $seqres.full
 _scratch_mount
 rm -f $seqres.full
 
tests/xfs/194 +3 -1
index 6c1eddba..3e186528 100755
@@ -32,6 +32,9 @@ _supported_os Linux
 # real QA test starts here
 rm -f $seqres.full
 
+_require_scratch
+_scratch_mkfs_xfs >/dev/null 2>&1
+
 # For this test we use block size = 1/8 page size
 pgsize=`$here/src/feature -s`
 blksize=`expr $pgsize / 8`
@@ -70,7 +73,6 @@ _filter_od()
     sed -e "s/^[0-9A-Fa-f ]\{7,8\}//"
 }
 
-_require_scratch
 unset MKFS_OPTIONS
 unset XFS_MKFS_OPTIONS
 
tests/xfs/261 +2 -0
index 0fabcd64..92152ace 100755
@@ -47,6 +47,8 @@ _supported_os Linux
 _require_quota
 _require_scratch
 
+_scratch_mkfs >> $seqres.full 2>&1
+
 # Just use the current mount table as an example mtab file.  Odds
 # are good there's nothing wrong with it.
 _setup_my_mtab() {
tests/xfs/284 +1 -0
index 07f71031..7af77634 100755
@@ -50,6 +50,7 @@ COPY_FILE="${TEST_DIR}/${seq}_copyfile"
 
 # Test dump a mounted device
 # xfs_metadump should refuse to dump a mounted device
+_scratch_mkfs >> $seqres.full 2>&1
 _scratch_mount
 _scratch_metadump $METADUMP_FILE 2>&1 | filter_mounted
 _scratch_unmount