Skip to contentSkip to conversation

Add support for testing XFS metadump v2

PATCH Superseded Chandan Babu R 2 participants 9 messages Jan 9, 2024 – Jan 10, 2024

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover Add support for testing XFS metadump v2

Hi,

This patchset modifies existing tests to add support for testing Metadump v2 feature. Metadump v2 feature is already part of Xfsprogs' for-next branch.

Changelog:
V1 -> V2:
1. Unroll loops which execute metadump/mdrestore for both v1 and v2 variants.

Chandan Babu R (5):
common/xfs: Do not append -a and -o options to metadump common/xfs: Add function to detect support for metadump v2 _scratch_xfs_mdrestore: Pass scratch log device when applicable
xfs: Add support for testing metadump v2
xfs: Check correctness of metadump/mdrestore's ability to work with
dirty log

 common/populate   |   2 +-
 common/xfs        |  32 +++++++--
 tests/xfs/129     |  97 ++++++++++++++++++++++---
 tests/xfs/129.out |   4 +-
 tests/xfs/234     |  97 ++++++++++++++++++++++---
 tests/xfs/234.out |   4 +-
 tests/xfs/253     | 118 +++++++++++++++++++++++++-----
 tests/xfs/291     |  25 +++++--
 tests/xfs/336     |   2 +-
 tests/xfs/432     |  29 ++++++--
 tests/xfs/432.out |   3 +-
 tests/xfs/503     |  94 +++++++++++++++---------
 tests/xfs/503.out |  12 ++--
 tests/xfs/801     | 178 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/801.out |  14 ++++
 15 files changed, 601 insertions(+), 110 deletions(-)
 create mode 100755 tests/xfs/801
 create mode 100644 tests/xfs/801.out
Signature · 1 line
2.43.0
#1 lore raw

PATCH v2 1/5 common/xfs: Do not append -a and -o options to metadump

xfs/253 requires the metadump to be obfuscated. However _xfs_metadump() would append the '-o' option causing the metadump to be unobfuscated.

This commit fixes the bug by modifying _xfs_metadump() to no longer append any metadump options. The direct/indirect callers of this function now pass the required options explicitly.

---
 common/populate | 2 +-
 common/xfs      | 7 +++----
 tests/xfs/291   | 2 +-
 tests/xfs/336   | 2 +-
 tests/xfs/432   | 2 +-
 tests/xfs/503   | 2 +-
 6 files changed, 8 insertions(+), 9 deletions(-)
common/populate +1 -1
index 3d233073..cfbfd88a 100644
@@ -55,7 +55,7 @@ __populate_fail() {
 	case "$FSTYP" in
 	xfs)
 		_scratch_unmount
-		_scratch_xfs_metadump "$metadump"
+		_scratch_xfs_metadump "$metadump" -a -o
 		;;
 	ext4)
 		_scratch_unmount
common/xfs +3 -4
index f53b33fc..38094828 100644
@@ -667,7 +667,6 @@ _xfs_metadump() {
 	local compressopt="$4"
 	shift; shift; shift; shift
 	local options="$@"
-	test -z "$options" && options="-a -o"
 
 	if [ "$logdev" != "none" ]; then
 		options="$options -l $logdev"
@@ -855,7 +854,7 @@ _check_xfs_filesystem()
 	if [ "$ok" -ne 1 ] && [ "$DUMP_CORRUPT_FS" = "1" ]; then
 		local flatdev="$(basename "$device")"
 		_xfs_metadump "$seqres.$flatdev.check.md" "$device" "$logdev" \
-			compress >> $seqres.full
+			compress -a -o >> $seqres.full
 	fi
 
 	# Optionally test the index rebuilding behavior.
@@ -888,7 +887,7 @@ _check_xfs_filesystem()
 		if [ "$rebuild_ok" -ne 1 ] && [ "$DUMP_CORRUPT_FS" = "1" ]; then
 			local flatdev="$(basename "$device")"
 			_xfs_metadump "$seqres.$flatdev.rebuild.md" "$device" \
-				"$logdev" compress >> $seqres.full
+				"$logdev" compress -a -o >> $seqres.full
 		fi
 	fi
 
@@ -972,7 +971,7 @@ _check_xfs_filesystem()
 		if [ "$orebuild_ok" -ne 1 ] && [ "$DUMP_CORRUPT_FS" = "1" ]; then
 			local flatdev="$(basename "$device")"
 			_xfs_metadump "$seqres.$flatdev.orebuild.md" "$device" \
-				"$logdev" compress >> $seqres.full
+				"$logdev" compress -a -o >> $seqres.full
 		fi
 	fi
 
tests/xfs/291 +1 -1
index 600dcb2e..54448497 100755
@@ -92,7 +92,7 @@ _scratch_xfs_check >> $seqres.full 2>&1 || _fail "xfs_check failed"
 
 # Yes they can!  Now...
 # Can xfs_metadump cope with this monster?
-_scratch_xfs_metadump $tmp.metadump || _fail "xfs_metadump failed"
+_scratch_xfs_metadump $tmp.metadump -a -o || _fail "xfs_metadump failed"
 SCRATCH_DEV=$tmp.img _scratch_xfs_mdrestore $tmp.metadump || _fail "xfs_mdrestore failed"
 SCRATCH_DEV=$tmp.img _scratch_xfs_repair -f &>> $seqres.full || \
 	_fail "xfs_repair of metadump failed"
tests/xfs/336 +1 -1
index d7a074d9..43b3790c 100755
@@ -62,7 +62,7 @@ _scratch_cycle_mount
 
 echo "Create metadump file"
 _scratch_unmount
-_scratch_xfs_metadump $metadump_file
+_scratch_xfs_metadump $metadump_file -a
 
 # Now restore the obfuscated one back and take a look around
 echo "Restore metadump"
tests/xfs/432 +1 -1
index 66315b03..dae68fb2 100755
@@ -86,7 +86,7 @@ echo "qualifying extent: $extlen blocks" >> $seqres.full
 test -n "$extlen" || _notrun "could not create dir extent > 1000 blocks"
 
 echo "Try to metadump"
-_scratch_xfs_metadump $metadump_file -w
+_scratch_xfs_metadump $metadump_file -a -o -w
 SCRATCH_DEV=$metadump_img _scratch_xfs_mdrestore $metadump_file
 
 echo "Check restored metadump image"
tests/xfs/503 +1 -1
index f5710ece..8805632d 100755
@@ -46,7 +46,7 @@ metadump_file_ag=${metadump_file}.ag
 copy_file=$testdir/copy.img
 
 echo metadump
-_scratch_xfs_metadump $metadump_file >> $seqres.full
+_scratch_xfs_metadump $metadump_file -a -o >> $seqres.full
 
 echo metadump a
 _scratch_xfs_metadump $metadump_file_a -a >> $seqres.full
Signature · 1 line
2.43.0
#2 lore raw

PATCH v2 2/5 common/xfs: Add function to detect support for metadump v2

In reply to Chandan Babu R

This commit defines a new function to help detect support for metadump v2.

---
 common/xfs | 6 ++++++
 1 file changed, 6 insertions(+)
common/xfs +6 -0
index 38094828..fc744489 100644
@@ -698,6 +698,12 @@ _xfs_mdrestore() {
 	$XFS_MDRESTORE_PROG $options "${metadump}" "${device}"
 }
 
+_scratch_metadump_v2_supported()
+{
+	$XFS_DB_PROG -c "help metadump" $SCRATCH_DEV | \
+		grep -q "Metadump version to be used"
+}
+
 # Snapshot the metadata on the scratch device
 _scratch_xfs_metadump()
 {
Signature · 1 line
2.43.0
Reviewed-by on v2 2/5
#3 lore raw

Re: [PATCH V2 2/5] common/xfs: Add function to detect support for metadump v2

Quoted 12 lines · Chandan Babu R

This commit defines a new function to help detect support for metadump v2.

---
 common/xfs | 6 ++++++
 1 file changed, 6 insertions(+)
common/xfs +0 -0
index 38094828..fc744489 100644

common/xfs _xfs_mdrestore() { · line 701

Quoted 4 lines
common/xfs +1 -0
@@ -698,6 +698,12 @@ _xfs_mdrestore() {
 	$XFS_MDRESTORE_PROG $options "${metadump}" "${device}"
 }
 
+_scratch_metadump_v2_supported()

/me wonders if the "_scratch" prefix is really warranted here since metadump v2 support is a property of the tool, not the scratch fs. OTOH xfs_db won't give you subcommand help without passing in a filesystem and we probably don't want to force-feed it $TEST_DEV so if zorro doesn't have any objections,

--D

Quoted 12 lines
common/xfs +5 -0
+{
+	$XFS_DB_PROG -c "help metadump" $SCRATCH_DEV | \
+		grep -q "Metadump version to be used"
+}
+
 # Snapshot the metadata on the scratch device
 _scratch_xfs_metadump()
 {
Signature · 1 line
2.43.0
#4 lore raw

PATCH v2 3/5 _scratch_xfs_mdrestore: Pass scratch log device when applicable

In reply to Chandan Babu R

Metadump v2 supports dumping contents of an external log device. This commit modifies _scratch_xfs_mdrestore() and _xfs_mdrestore() to be able to restore metadump files which contain data from external log devices.

The callers of _scratch_xfs_mdrestore() must set the value of $SCRATCH_LOGDEV only when all of the following conditions are met:
1. Metadump is in v2 format.
2. Metadump has contents dumped from an external log device.

---
 common/xfs | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
common/xfs +17 -2
index fc744489..070680d8 100644
@@ -682,7 +682,8 @@ _xfs_metadump() {
 _xfs_mdrestore() {
 	local metadump="$1"
 	local device="$2"
-	shift; shift
+	local logdev="$3"
+	shift; shift; shift
 	local options="$@"
 
 	# If we're configured for compressed dumps and there isn't already an
@@ -695,6 +696,10 @@ _xfs_mdrestore() {
 	fi
 	test -r "$metadump" || return 1
 
+	if [ "$logdev" != "none" ]; then
+		options="$options -l $logdev"
+	fi
+
 	$XFS_MDRESTORE_PROG $options "${metadump}" "${device}"
 }
 
@@ -722,8 +727,18 @@ _scratch_xfs_mdrestore()
 {
 	local metadump=$1
 	shift
+	local logdev=none
+	local options="$@"
 
-	_xfs_mdrestore "$metadump" "$SCRATCH_DEV" "$@"
+	# $SCRATCH_LOGDEV should have a non-zero length value only when all of
+	# the following conditions are met.
+	# 1. Metadump is in v2 format.
+	# 2. Metadump has contents dumped from an external log device.
+	if [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_LOGDEV" ]; then
+		logdev=$SCRATCH_LOGDEV
+	fi
+
+	_xfs_mdrestore "$metadump" "$SCRATCH_DEV" "$logdev" "$@"
 }
 
 # Do not use xfs_repair (offline fsck) to rebuild the filesystem
Signature · 1 line
2.43.0
#5 lore raw

PATCH v2 4/5 xfs: Add support for testing metadump v2

In reply to Chandan Babu R

This commit adds the ability to test metadump v2 to existing metadump tests.

---
 tests/xfs/129     |  97 ++++++++++++++++++++++++++++++++-----
 tests/xfs/129.out |   4 +-
 tests/xfs/234     |  97 ++++++++++++++++++++++++++++++++-----
 tests/xfs/234.out |   4 +-
 tests/xfs/253     | 118 ++++++++++++++++++++++++++++++++++++++--------
 tests/xfs/291     |  25 ++++++++--
 tests/xfs/432     |  29 +++++++++---
 tests/xfs/432.out |   3 +-
 tests/xfs/503     |  94 ++++++++++++++++++++++--------------
 tests/xfs/503.out |  12 ++---
 10 files changed, 381 insertions(+), 102 deletions(-)
tests/xfs/129 +86 -11
index 6f2ef564..8a817b41 100755
@@ -16,7 +16,11 @@ _cleanup()
 {
     cd /
     _scratch_unmount > /dev/null 2>&1
-    rm -rf $tmp.* $testdir $metadump_file $TEST_DIR/image
+    [[ -n $logdev && $logdev != "none" && $logdev != $SCRATCH_LOGDEV ]] && \
+	    _destroy_loop_device $logdev
+    [[ -n $datadev ]] && _destroy_loop_device $datadev
+    rm -rf $tmp.* $testdir $metadump_file $TEST_DIR/data-image \
+       $TEST_DIR/log-image
 }
 
 # Import common functions.
@@ -29,12 +33,86 @@ _require_command "$XFS_MDRESTORE_PROG" "xfs_mdrestore"
 _require_loop
 _require_scratch_reflink
 
+metadump_file=$TEST_DIR/${seq}_metadump
+
+verify_metadump_v1()
+{
+	local max_version=$1
+	local version=""
+
+	if [[ $max_version == 2 ]]; then
+		version="-v 1"
+	fi
+
+	_scratch_xfs_metadump $metadump_file $version
+
+	SCRATCH_DEV=$TEST_DIR/data-image SCRATCH_LOGDEV="" \
+		   _scratch_xfs_mdrestore $metadump_file
+
+	datadev=$(_create_loop_device $TEST_DIR/data-image)
+
+	SCRATCH_DEV=$datadev _scratch_mount
+	SCRATCH_DEV=$datadev _scratch_unmount
+
+	logdev=$SCRATCH_LOGDEV
+	[[ -z $logdev ]] && logdev=none
+	_check_xfs_filesystem $datadev $logdev none
+
+	_destroy_loop_device $datadev
+	datadev=""
+	rm -f $TEST_DIR/data-image
+}
+
+verify_metadump_v2()
+{
+	version="-v 2"
+
+	_scratch_xfs_metadump $metadump_file $version
+
+	# Metadump v2 files can contain contents dumped from an external log
+	# device. Use a temporary file to hold the log device contents restored
+	# from such a metadump file.
+	slogdev=""
+	if [[ -n $SCRATCH_LOGDEV ]]; then
+		slogdev=$TEST_DIR/log-image
+	fi
+
+	SCRATCH_DEV=$TEST_DIR/data-image SCRATCH_LOGDEV=$slogdev \
+		   _scratch_xfs_mdrestore $metadump_file
+
+	datadev=$(_create_loop_device $TEST_DIR/data-image)
+
+	logdev=${SCRATCH_LOGDEV}
+	if [[ -s $TEST_DIR/log-image ]]; then
+		logdev=$(_create_loop_device $TEST_DIR/log-image)
+	fi
+
+	SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _scratch_mount
+	SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _scratch_unmount
+
+	[[ -z $logdev ]] && logdev=none
+	_check_xfs_filesystem $datadev $logdev none
+
+	if [[ -s $TEST_DIR/log-image ]]; then
+		_destroy_loop_device $logdev
+		logdev=""
+		rm -f $TEST_DIR/log-image
+	fi
+
+	_destroy_loop_device $datadev
+	datadev=""
+	rm -f $TEST_DIR/data-image
+}
+
 _scratch_mkfs >/dev/null 2>&1
+
+max_md_version=1
+_scratch_metadump_v2_supported && max_md_version=2
+
 _scratch_mount
 
 testdir=$SCRATCH_MNT/test-$seq
 mkdir $testdir
-metadump_file=$TEST_DIR/${seq}_metadump
 
 echo "Create the original file blocks"
 blksz="$(_get_file_block_size $testdir)"
@@ -47,18 +125,15 @@ seq 1 2 $((nr_blks - 1)) | while read nr; do
 			$testdir/file2 $((nr * blksz)) $blksz >> $seqres.full
 done
 
-echo "Create metadump file"
 _scratch_unmount
-_scratch_xfs_metadump $metadump_file
 
-# Now restore the obfuscated one back and take a look around
-echo "Restore metadump"
-SCRATCH_DEV=$TEST_DIR/image _scratch_xfs_mdrestore $metadump_file
-SCRATCH_DEV=$TEST_DIR/image _scratch_mount
-SCRATCH_DEV=$TEST_DIR/image _scratch_unmount
+echo "Create metadump file, restore it and check restored fs"
 
-echo "Check restored fs"
-_check_generic_filesystem $metadump_file
+verify_metadump_v1 $max_md_version
+
+if [[ $max_md_version == 2 ]]; then
+	verify_metadump_v2
+fi
 
 # success, all done
 status=0
tests/xfs/129.out +1 -3
index da6f43fd..0f24c431 100644
@@ -1,6 +1,4 @@
 QA output created by 129
 Create the original file blocks
 Reflink every other block
-Create metadump file
-Restore metadump
-Check restored fs
+Create metadump file, restore it and check restored fs
tests/xfs/234 +86 -11

123 lines not on this page · raw message

tests/xfs/234.out +1 -3
index 463d4660..fc2ddd77 100644
@@ -1,6 +1,4 @@
 QA output created by 234
 Create the original file blocks
 Punch every other block
-Create metadump file
-Restore metadump
-Check restored fs
+Create metadump file, restore it and check restored fs
tests/xfs/253 +99 -19

156 lines not on this page · raw message

tests/xfs/291 +21 -4
index 54448497..33193eb7 100755
@@ -92,10 +92,27 @@ _scratch_xfs_check >> $seqres.full 2>&1 || _fail "xfs_check failed"
 
 # Yes they can!  Now...
 # Can xfs_metadump cope with this monster?
-_scratch_xfs_metadump $tmp.metadump -a -o || _fail "xfs_metadump failed"
-SCRATCH_DEV=$tmp.img _scratch_xfs_mdrestore $tmp.metadump || _fail "xfs_mdrestore failed"
-SCRATCH_DEV=$tmp.img _scratch_xfs_repair -f &>> $seqres.full || \
-	_fail "xfs_repair of metadump failed"
+max_md_version=1
+_scratch_metadump_v2_supported && max_md_version=2
+
+for md_version in $(seq 1 $max_md_version); do
+	version=""
+	if [[ $max_md_version == 2 ]]; then
+		version="-v $md_version"
+	fi
+
+	_scratch_xfs_metadump $tmp.metadump -a -o $version || \
+		_fail "xfs_metadump failed"
+
+	slogdev=$SCRATCH_LOGDEV
+	if [[ -z $version || $version == "-v 1" ]]; then
+		slogdev=""
+	fi
+	SCRATCH_DEV=$tmp.img SCRATCH_LOGDEV=$slogdev _scratch_xfs_mdrestore \
+		   $tmp.metadump || _fail "xfs_mdrestore failed"
+	SCRATCH_DEV=$tmp.img _scratch_xfs_repair -f &>> $seqres.full || \
+		_fail "xfs_repair of metadump failed"
+done
 
 # Yes it can; success, all done
 status=0
tests/xfs/432 +23 -6
index dae68fb2..a215d3ce 100755
@@ -50,6 +50,7 @@ echo "Format and mount"
 # block.  8187 hashes/dablk / 248 dirents/dirblock = ~33 dirblocks per
 # dablock.  33 dirblocks * 64k mean that we can expand a directory by
 # 2112k before we have to allocate another da btree block.
+
 _scratch_mkfs -b size=1k -n size=64k > "$seqres.full" 2>&1
 _scratch_mount >> "$seqres.full" 2>&1
 
@@ -85,13 +86,29 @@ extlen="$(check_for_long_extent $dir_inum)"
 echo "qualifying extent: $extlen blocks" >> $seqres.full
 test -n "$extlen" || _notrun "could not create dir extent > 1000 blocks"
 
-echo "Try to metadump"
-_scratch_xfs_metadump $metadump_file -a -o -w
-SCRATCH_DEV=$metadump_img _scratch_xfs_mdrestore $metadump_file
+echo "Try to metadump, restore and check restored metadump image"
+max_md_version=1
+_scratch_metadump_v2_supported && max_md_version=2
 
-echo "Check restored metadump image"
-SCRATCH_DEV=$metadump_img _scratch_xfs_repair -n &>> $seqres.full || \
-	echo "xfs_repair on restored fs returned $?"
+for md_version in $(seq 1 $max_md_version); do
+	version=""
+	if [[ $max_md_version == 2 ]]; then
+		version="-v $md_version"
+	fi
+
+	_scratch_xfs_metadump $metadump_file -a -o -w $version
+
+	slogdev=$SCRATCH_LOGDEV
+	if [[ -z $version || $version == "-v 1" ]]; then
+		slogdev=""
+	fi
+
+	SCRATCH_DEV=$metadump_img SCRATCH_LOGDEV=$slogdev \
+		   _scratch_xfs_mdrestore $metadump_file
+
+	SCRATCH_DEV=$metadump_img _scratch_xfs_repair -n &>> $seqres.full || \
+		echo "xfs_repair on restored fs returned $?"
+done
 
 # success, all done
 status=0
tests/xfs/432.out +1 -2
index 1f135d16..37bac902 100644
@@ -2,5 +2,4 @@ QA output created by 432
 Format and mount
 Create huge dir
 Check for > 1000 block extent?
-Try to metadump
-Check restored metadump image
+Try to metadump, restore and check restored metadump image
tests/xfs/503 +59 -35

117 lines not on this page · raw message

tests/xfs/503.out +4 -8
index 8ef31dbe..496f2516 100644
@@ -1,12 +1,8 @@
 QA output created by 503
 Format and populate
-metadump
-metadump a
-metadump g
-metadump ag
+metadump and mdrestore
+metadump a and mdrestore
+metadump g and mdrestore
+metadump ag and mdrestore
 copy
 recopy
-mdrestore
-mdrestore a
-mdrestore g
-mdrestore ag
Signature · 1 line
2.43.0
#6 lore raw

PATCH v2 5/5 xfs: Check correctness of metadump/mdrestore's ability to work with dirty log

In reply to Chandan Babu R

Add a new test to verify if metadump/mdrestore are able to dump and restore the contents of a dirty log.

---
 tests/xfs/801     | 178 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/801.out |  14 ++++
 2 files changed, 192 insertions(+)
 create mode 100755 tests/xfs/801
 create mode 100644 tests/xfs/801.out
tests/xfs/801 new file +178 -0
new file mode 100755
index 00000000..a7866ce7
@@ -0,0 +1,178 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oracle, Inc.  All Rights Reserved.
+#
+# FS QA Test 801
+#
+# Test metadump/mdrestore's ability to dump a dirty log and restore it
+# correctly.
+#
+. ./common/preamble
+_begin_fstest auto quick metadump log logprint punch
+
+# Override the default cleanup function.
+_cleanup()
+{
+	cd /
+	rm -r -f $tmp.*
+	_scratch_unmount > /dev/null 2>&1
+	[[ -n $logdev && $logdev != "none" && $logdev != $SCRATCH_LOGDEV ]] && \
+		_destroy_loop_device $logdev
+	[[ -n $datadev ]] && _destroy_loop_device $datadev
+	rm -r -f $metadump_file $TEST_DIR/data-image \
+	   $TEST_DIR/log-image
+}
+
+# Import common functions.
+. ./common/dmflakey
+. ./common/inject
+
+# real QA test starts here
+_supported_fs xfs
+_require_scratch
+_require_test
+_require_loop
+_require_xfs_debug
+_require_xfs_io_error_injection log_item_pin
+_require_dm_target flakey
+_require_xfs_io_command "pwrite"
+_require_test_program "punch-alternating"
+
+metadump_file=${TEST_DIR}/${seq}.md
+testfile=${SCRATCH_MNT}/testfile
+
+echo "Format filesystem on scratch device"
+_scratch_mkfs >> $seqres.full 2>&1
+
+max_md_version=1
+_scratch_metadump_v2_supported && max_md_version=2
+
+external_log=0
+if [[ $USE_EXTERNAL = yes && -n "$SCRATCH_LOGDEV" ]]; then
+	external_log=1
+fi
+
+if [[ $max_md_version == 1 && $external_log == 1 ]]; then
+	_notrun "metadump v1 does not support external log device"
+fi
+
+verify_metadump_v1()
+{
+	local version=""
+	if [[ $max_md_version == 2 ]]; then
+		version="-v 1"
+	fi
+
+	_scratch_xfs_metadump $metadump_file -a -o $version
+
+	SCRATCH_DEV=$TEST_DIR/data-image _scratch_xfs_mdrestore $metadump_file
+
+	datadev=$(_create_loop_device $TEST_DIR/data-image)
+
+	SCRATCH_DEV=$datadev _scratch_mount
+	SCRATCH_DEV=$datadev _check_scratch_fs
+	SCRATCH_DEV=$datadev _scratch_unmount
+
+	_destroy_loop_device $datadev
+	datadev=""
+	rm -f $TEST_DIR/data-image
+}
+
+verify_metadump_v2()
+{
+	local version="-v 2"
+
+	_scratch_xfs_metadump $metadump_file -a -o $version
+
+	# Metadump v2 files can contain contents dumped from an external log
+	# device. Use a temporary file to hold the log device contents restored
+	# from such a metadump file.
+	slogdev=""
+	if [[ -n $SCRATCH_LOGDEV ]]; then
+		slogdev=$TEST_DIR/log-image
+	fi
+
+	SCRATCH_DEV=$TEST_DIR/data-image SCRATCH_LOGDEV=$slogdev \
+		   _scratch_xfs_mdrestore $metadump_file
+
+	datadev=$(_create_loop_device $TEST_DIR/data-image)
+
+	logdev=${SCRATCH_LOGDEV}
+	if [[ -s $TEST_DIR/log-image ]]; then
+		logdev=$(_create_loop_device $TEST_DIR/log-image)
+	fi
+
+	SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _scratch_mount
+	SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _check_scratch_fs
+	SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _scratch_unmount
+
+	if [[ -s $TEST_DIR/log-image ]]; then
+		_destroy_loop_device $logdev
+		logdev=""
+		rm -f $TEST_DIR/log-image
+	fi
+
+	_destroy_loop_device $datadev
+	datadev=""
+	rm -f $TEST_DIR/data-image
+}
+
+echo "Initialize and mount filesystem on flakey device"
+_init_flakey
+_load_flakey_table $FLAKEY_ALLOW_WRITES
+_mount_flakey
+
+echo "Create test file"
+$XFS_IO_PROG -s -f -c "pwrite 0 5M" $testfile >> $seqres.full
+
+echo "Punch alternative blocks of test file"
+$here/src/punch-alternating $testfile
+
+echo "Mount cycle the filesystem on flakey device"
+_unmount_flakey
+_mount_flakey
+
+device=$(readlink -f $FLAKEY_DEV)
+device=$(_short_dev $device)
+
+echo "Pin log items in the AIL"
+echo 1 > /sys/fs/xfs/${device}/errortag/log_item_pin
+
+echo "Create two checkpoint transactions on ondisk log"
+for ct in $(seq 1 2); do
+	offset=$($XFS_IO_PROG -c 'fiemap' $testfile | tac |  grep -v hole | \
+			 head -n 1 | awk -F '[\\[.]' '{ print $2 * 512; }')
+	$XFS_IO_PROG -c "truncate $offset" -c fsync $testfile
+done
+
+echo "Drop writes to filesystem from here onwards"
+_load_flakey_table $FLAKEY_DROP_WRITES
+
+echo "Unpin log items in AIL"
+echo 0 > /sys/fs/xfs/${device}/errortag/log_item_pin
+
+echo "Unmount filesystem on flakey device"
+_unmount_flakey
+
+echo "Clean up flakey device"
+_cleanup_flakey
+
+echo -n "Filesystem has a "
+_print_logstate
+
+echo "Create metadump file, restore it and check restored fs"
+
+if [[ $external_log == 0 ]]; then
+	verify_metadump_v1 $max_md_version
+fi
+
+if [[ $max_md_version == 2 ]]; then
+	verify_metadump_v2
+fi
+
+# Mount the fs to replay the contents from the dirty log.
+_scratch_mount
+
+# success, all done
+status=0
+exit
tests/xfs/801.out new file +14 -0
new file mode 100644
index 00000000..a2f2abca
@@ -0,0 +1,14 @@
+QA output created by 801
+Format filesystem on scratch device
+Initialize and mount filesystem on flakey device
+Create test file
+Punch alternative blocks of test file
+Mount cycle the filesystem on flakey device
+Pin log items in the AIL
+Create two checkpoint transactions on ondisk log
+Drop writes to filesystem from here onwards
+Unpin log items in AIL
+Unmount filesystem on flakey device
+Clean up flakey device
+Filesystem has a dirty log
+Create metadump file, restore it and check restored fs
Signature · 1 line
2.43.0
#7 lore raw

Re: [PATCH V2 5/5] xfs: Check correctness of metadump/mdrestore's ability to work with dirty log

Quoted 99 lines · Chandan Babu R

Add a new test to verify if metadump/mdrestore are able to dump and restore the contents of a dirty log.

---
 tests/xfs/801     | 178 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/801.out |  14 ++++
 2 files changed, 192 insertions(+)
 create mode 100755 tests/xfs/801
 create mode 100644 tests/xfs/801.out
tests/xfs/801 new file +82 -0
new file mode 100755
index 00000000..a7866ce7
@@ -0,0 +1,178 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oracle, Inc.  All Rights Reserved.
+#
+# FS QA Test 801
+#
+# Test metadump/mdrestore's ability to dump a dirty log and restore it
+# correctly.
+#
+. ./common/preamble
+_begin_fstest auto quick metadump log logprint punch
+
+# Override the default cleanup function.
+_cleanup()
+{
+	cd /
+	rm -r -f $tmp.*
+	_scratch_unmount > /dev/null 2>&1
+	[[ -n $logdev && $logdev != "none" && $logdev != $SCRATCH_LOGDEV ]] && \
+		_destroy_loop_device $logdev
+	[[ -n $datadev ]] && _destroy_loop_device $datadev
+	rm -r -f $metadump_file $TEST_DIR/data-image \
+	   $TEST_DIR/log-image
+}
+
+# Import common functions.
+. ./common/dmflakey
+. ./common/inject
+
+# real QA test starts here
+_supported_fs xfs
+_require_scratch
+_require_test
+_require_loop
+_require_xfs_debug
+_require_xfs_io_error_injection log_item_pin
+_require_dm_target flakey
+_require_xfs_io_command "pwrite"
+_require_test_program "punch-alternating"
+
+metadump_file=${TEST_DIR}/${seq}.md
+testfile=${SCRATCH_MNT}/testfile
+
+echo "Format filesystem on scratch device"
+_scratch_mkfs >> $seqres.full 2>&1
+
+max_md_version=1
+_scratch_metadump_v2_supported && max_md_version=2
+
+external_log=0
+if [[ $USE_EXTERNAL = yes && -n "$SCRATCH_LOGDEV" ]]; then
+	external_log=1
+fi
+
+if [[ $max_md_version == 1 && $external_log == 1 ]]; then
+	_notrun "metadump v1 does not support external log device"
+fi
+
+verify_metadump_v1()
+{
+	local version=""
+	if [[ $max_md_version == 2 ]]; then
+		version="-v 1"
+	fi
+
+	_scratch_xfs_metadump $metadump_file -a -o $version
+
+	SCRATCH_DEV=$TEST_DIR/data-image _scratch_xfs_mdrestore $metadump_file
+
+	datadev=$(_create_loop_device $TEST_DIR/data-image)
+
+	SCRATCH_DEV=$datadev _scratch_mount
+	SCRATCH_DEV=$datadev _check_scratch_fs
+	SCRATCH_DEV=$datadev _scratch_unmount
+
+	_destroy_loop_device $datadev
+	datadev=""
+	rm -f $TEST_DIR/data-image
+}
+
+verify_metadump_v2()
+{

tests/xfs/801 line 92

Quoted 10 lines
tests/xfs/801 new file +10 -0
+	local version="-v 2"
+
+	_scratch_xfs_metadump $metadump_file -a -o $version
+
+	# Metadump v2 files can contain contents dumped from an external log
+	# device. Use a temporary file to hold the log device contents restored
+	# from such a metadump file.
+	slogdev=""
+	if [[ -n $SCRATCH_LOGDEV ]]; then
+		slogdev=$TEST_DIR/log-image

Why not create the loopdevs here?

tests/xfs/801 line 102

Quoted 10 lines
tests/xfs/801 new file +10 -0
+	fi
+
+	SCRATCH_DEV=$TEST_DIR/data-image SCRATCH_LOGDEV=$slogdev \
+		   _scratch_xfs_mdrestore $metadump_file
+
+	datadev=$(_create_loop_device $TEST_DIR/data-image)
+
+	logdev=${SCRATCH_LOGDEV}
+	if [[ -s $TEST_DIR/log-image ]]; then
+		logdev=$(_create_loop_device $TEST_DIR/log-image)

if [[ -s $slogdev ]]; then

	logdev=$(_create_loop_device $slogdev)

fi

When would we have logdev == SCRATCH_LOGDEV at this point in the program?

--D

Quoted 100 lines
tests/xfs/801 new file +76 -0
+	fi
+
+	SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _scratch_mount
+	SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _check_scratch_fs
+	SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _scratch_unmount
+
+	if [[ -s $TEST_DIR/log-image ]]; then
+		_destroy_loop_device $logdev
+		logdev=""
+		rm -f $TEST_DIR/log-image
+	fi
+
+	_destroy_loop_device $datadev
+	datadev=""
+	rm -f $TEST_DIR/data-image
+}
+
+echo "Initialize and mount filesystem on flakey device"
+_init_flakey
+_load_flakey_table $FLAKEY_ALLOW_WRITES
+_mount_flakey
+
+echo "Create test file"
+$XFS_IO_PROG -s -f -c "pwrite 0 5M" $testfile >> $seqres.full
+
+echo "Punch alternative blocks of test file"
+$here/src/punch-alternating $testfile
+
+echo "Mount cycle the filesystem on flakey device"
+_unmount_flakey
+_mount_flakey
+
+device=$(readlink -f $FLAKEY_DEV)
+device=$(_short_dev $device)
+
+echo "Pin log items in the AIL"
+echo 1 > /sys/fs/xfs/${device}/errortag/log_item_pin
+
+echo "Create two checkpoint transactions on ondisk log"
+for ct in $(seq 1 2); do
+	offset=$($XFS_IO_PROG -c 'fiemap' $testfile | tac |  grep -v hole | \
+			 head -n 1 | awk -F '[\\[.]' '{ print $2 * 512; }')
+	$XFS_IO_PROG -c "truncate $offset" -c fsync $testfile
+done
+
+echo "Drop writes to filesystem from here onwards"
+_load_flakey_table $FLAKEY_DROP_WRITES
+
+echo "Unpin log items in AIL"
+echo 0 > /sys/fs/xfs/${device}/errortag/log_item_pin
+
+echo "Unmount filesystem on flakey device"
+_unmount_flakey
+
+echo "Clean up flakey device"
+_cleanup_flakey
+
+echo -n "Filesystem has a "
+_print_logstate
+
+echo "Create metadump file, restore it and check restored fs"
+
+if [[ $external_log == 0 ]]; then
+	verify_metadump_v1 $max_md_version
+fi
+
+if [[ $max_md_version == 2 ]]; then
+	verify_metadump_v2
+fi
+
+# Mount the fs to replay the contents from the dirty log.
+_scratch_mount
+
+# success, all done
+status=0
+exit
tests/xfs/801.out new file +14 -0
new file mode 100644
index 00000000..a2f2abca
@@ -0,0 +1,14 @@
+QA output created by 801
+Format filesystem on scratch device
+Initialize and mount filesystem on flakey device
+Create test file
+Punch alternative blocks of test file
+Mount cycle the filesystem on flakey device
+Pin log items in the AIL
+Create two checkpoint transactions on ondisk log
+Drop writes to filesystem from here onwards
+Unpin log items in AIL
+Unmount filesystem on flakey device
+Clean up flakey device
+Filesystem has a dirty log
+Create metadump file, restore it and check restored fs
Signature · 1 line
2.43.0
#8 lore raw

Re: [PATCH V2 5/5] xfs: Check correctness of metadump/mdrestore's ability to work with dirty log

Quoted 104 lines · Darrick J. Wong
Quoted 103 lines · Chandan Babu R

Add a new test to verify if metadump/mdrestore are able to dump and restore the contents of a dirty log.

---
 tests/xfs/801     | 178 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/801.out |  14 ++++
 2 files changed, 192 insertions(+)
 create mode 100755 tests/xfs/801
 create mode 100644 tests/xfs/801.out
tests/xfs/801 new file +86 -0
new file mode 100755
index 00000000..a7866ce7
@@ -0,0 +1,178 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Oracle, Inc.  All Rights Reserved.
+#
+# FS QA Test 801
+#
+# Test metadump/mdrestore's ability to dump a dirty log and restore it
+# correctly.
+#
+. ./common/preamble
+_begin_fstest auto quick metadump log logprint punch
+
+# Override the default cleanup function.
+_cleanup()
+{
+	cd /
+	rm -r -f $tmp.*
+	_scratch_unmount > /dev/null 2>&1
+	[[ -n $logdev && $logdev != "none" && $logdev != $SCRATCH_LOGDEV ]] && \
+		_destroy_loop_device $logdev
+	[[ -n $datadev ]] && _destroy_loop_device $datadev
+	rm -r -f $metadump_file $TEST_DIR/data-image \
+	   $TEST_DIR/log-image
+}
+
+# Import common functions.
+. ./common/dmflakey
+. ./common/inject
+
+# real QA test starts here
+_supported_fs xfs
+_require_scratch
+_require_test
+_require_loop
+_require_xfs_debug
+_require_xfs_io_error_injection log_item_pin
+_require_dm_target flakey
+_require_xfs_io_command "pwrite"
+_require_test_program "punch-alternating"
+
+metadump_file=${TEST_DIR}/${seq}.md
+testfile=${SCRATCH_MNT}/testfile
+
+echo "Format filesystem on scratch device"
+_scratch_mkfs >> $seqres.full 2>&1
+
+max_md_version=1
+_scratch_metadump_v2_supported && max_md_version=2
+
+external_log=0
+if [[ $USE_EXTERNAL = yes && -n "$SCRATCH_LOGDEV" ]]; then
+	external_log=1
+fi
+
+if [[ $max_md_version == 1 && $external_log == 1 ]]; then
+	_notrun "metadump v1 does not support external log device"
+fi
+
+verify_metadump_v1()
+{
+	local version=""
+	if [[ $max_md_version == 2 ]]; then
+		version="-v 1"
+	fi
+
+	_scratch_xfs_metadump $metadump_file -a -o $version
+
+	SCRATCH_DEV=$TEST_DIR/data-image _scratch_xfs_mdrestore $metadump_file
+
+	datadev=$(_create_loop_device $TEST_DIR/data-image)
+
+	SCRATCH_DEV=$datadev _scratch_mount
+	SCRATCH_DEV=$datadev _check_scratch_fs
+	SCRATCH_DEV=$datadev _scratch_unmount
+
+	_destroy_loop_device $datadev
+	datadev=""
+	rm -f $TEST_DIR/data-image
+}
+
+verify_metadump_v2()
+{
+	local version="-v 2"
+
+	_scratch_xfs_metadump $metadump_file -a -o $version
+
Quoted 9 lines

tests/xfs/801 line 92

Quoted 6 lines
tests/xfs/801 new file +6 -0
+	# Metadump v2 files can contain contents dumped from an external log
+	# device. Use a temporary file to hold the log device contents restored
+	# from such a metadump file.
+	slogdev=""
+	if [[ -n $SCRATCH_LOGDEV ]]; then
+		slogdev=$TEST_DIR/log-image

Why not create the loopdevs here?

The backing files (i.e. $TEST_DIR/data-image and $TEST_DIR/log-image) have not yet been created. They are created by the invocation of mdrestore command below.

Quoted 12 lines

tests/xfs/801 line 102

Quoted 10 lines
tests/xfs/801 new file +10 -0
+	fi
+
+	SCRATCH_DEV=$TEST_DIR/data-image SCRATCH_LOGDEV=$slogdev \
+		   _scratch_xfs_mdrestore $metadump_file
+
+	datadev=$(_create_loop_device $TEST_DIR/data-image)
+
+	logdev=${SCRATCH_LOGDEV}
+	if [[ -s $TEST_DIR/log-image ]]; then
+		logdev=$(_create_loop_device $TEST_DIR/log-image)

if [[ -s $slogdev ]]; then

Quoted 4 lines
	logdev=$(_create_loop_device $slogdev)

fi

When would we have logdev == SCRATCH_LOGDEV at this point in the program?

logdev == SCRATCH_LOGDEV only when using an internal log. I think a much cleaner way of initializing logdev would be

	logdev=""

Combining this with the change suggested by you earlier, the code now looks like the following,

	verify_metadump_v2()
	{
		local version="-v 2"
	
		_scratch_xfs_metadump $metadump_file -a -o $version
	
		# Metadump v2 files can contain contents dumped from an external log
		# device. Use a temporary file to hold the log device contents restored
		# from such a metadump file.
		slogdev=""
		if [[ -n $SCRATCH_LOGDEV ]]; then
			slogdev=$TEST_DIR/log-image
		fi
	
		SCRATCH_DEV=$TEST_DIR/data-image SCRATCH_LOGDEV=$slogdev \
			   _scratch_xfs_mdrestore $metadump_file
	
		datadev=$(_create_loop_device $TEST_DIR/data-image)
	
		logdev=""
		if [[ -s $slogdev ]]; then
			logdev=$(_create_loop_device $slogdev)
		fi
	
		SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _scratch_mount
		SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _check_scratch_fs
		SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _scratch_unmount
	
		if [[ -s $logdev ]]; then
			_destroy_loop_device $logdev
			logdev=""
			rm -f $slogdev
		fi
	
		_destroy_loop_device $datadev
		datadev=""
		rm -f $TEST_DIR/data-image
	}
Signature · 1 line
Chandan