Add support for testing XFS metadump v2
v2 cover Add support for testing XFS metadump v2
Hi,
This patchset modifies existing tests to add support for testing Metadump v2 feature. Metadump v2 feature is already part of Xfsprogs' for-next branch.
Changelog:
V1 -> V2:
1. Unroll loops which execute metadump/mdrestore for both v1 and v2 variants.
Chandan Babu R (5):
common/xfs: Do not append -a and -o options to metadump common/xfs: Add function to detect support for metadump v2 _scratch_xfs_mdrestore: Pass scratch log device when applicable
xfs: Add support for testing metadump v2
xfs: Check correctness of metadump/mdrestore's ability to work with
dirty log
common/populate | 2 +- common/xfs | 32 +++++++-- tests/xfs/129 | 97 ++++++++++++++++++++++--- tests/xfs/129.out | 4 +- tests/xfs/234 | 97 ++++++++++++++++++++++--- tests/xfs/234.out | 4 +- tests/xfs/253 | 118 +++++++++++++++++++++++++----- tests/xfs/291 | 25 +++++-- tests/xfs/336 | 2 +- tests/xfs/432 | 29 ++++++-- tests/xfs/432.out | 3 +- tests/xfs/503 | 94 +++++++++++++++--------- tests/xfs/503.out | 12 ++-- tests/xfs/801 | 178 ++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/801.out | 14 ++++ 15 files changed, 601 insertions(+), 110 deletions(-) create mode 100755 tests/xfs/801 create mode 100644 tests/xfs/801.out
Signature · 1 line
2.43.0
PATCH v2 1/5 common/xfs: Do not append -a and -o options to metadump
xfs/253 requires the metadump to be obfuscated. However _xfs_metadump() would append the '-o' option causing the metadump to be unobfuscated.
This commit fixes the bug by modifying _xfs_metadump() to no longer append any metadump options. The direct/indirect callers of this function now pass the required options explicitly.
--- common/populate | 2 +- common/xfs | 7 +++---- tests/xfs/291 | 2 +- tests/xfs/336 | 2 +- tests/xfs/432 | 2 +- tests/xfs/503 | 2 +- 6 files changed, 8 insertions(+), 9 deletions(-)
common/populate +1 -1
@@ -55,7 +55,7 @@ __populate_fail() { case "$FSTYP" in xfs) _scratch_unmount - _scratch_xfs_metadump "$metadump" + _scratch_xfs_metadump "$metadump" -a -o ;; ext4) _scratch_unmount
common/xfs +3 -4
@@ -667,7 +667,6 @@ _xfs_metadump() { local compressopt="$4" shift; shift; shift; shift local options="$@" - test -z "$options" && options="-a -o" if [ "$logdev" != "none" ]; then options="$options -l $logdev"
@@ -855,7 +854,7 @@ _check_xfs_filesystem() if [ "$ok" -ne 1 ] && [ "$DUMP_CORRUPT_FS" = "1" ]; then local flatdev="$(basename "$device")" _xfs_metadump "$seqres.$flatdev.check.md" "$device" "$logdev" \ - compress >> $seqres.full + compress -a -o >> $seqres.full fi # Optionally test the index rebuilding behavior.
@@ -888,7 +887,7 @@ _check_xfs_filesystem() if [ "$rebuild_ok" -ne 1 ] && [ "$DUMP_CORRUPT_FS" = "1" ]; then local flatdev="$(basename "$device")" _xfs_metadump "$seqres.$flatdev.rebuild.md" "$device" \ - "$logdev" compress >> $seqres.full + "$logdev" compress -a -o >> $seqres.full fi fi
@@ -972,7 +971,7 @@ _check_xfs_filesystem() if [ "$orebuild_ok" -ne 1 ] && [ "$DUMP_CORRUPT_FS" = "1" ]; then local flatdev="$(basename "$device")" _xfs_metadump "$seqres.$flatdev.orebuild.md" "$device" \ - "$logdev" compress >> $seqres.full + "$logdev" compress -a -o >> $seqres.full fi fi
tests/xfs/291 +1 -1
@@ -92,7 +92,7 @@ _scratch_xfs_check >> $seqres.full 2>&1 || _fail "xfs_check failed" # Yes they can! Now... # Can xfs_metadump cope with this monster? -_scratch_xfs_metadump $tmp.metadump || _fail "xfs_metadump failed" +_scratch_xfs_metadump $tmp.metadump -a -o || _fail "xfs_metadump failed" SCRATCH_DEV=$tmp.img _scratch_xfs_mdrestore $tmp.metadump || _fail "xfs_mdrestore failed" SCRATCH_DEV=$tmp.img _scratch_xfs_repair -f &>> $seqres.full || \ _fail "xfs_repair of metadump failed"
tests/xfs/336 +1 -1
@@ -62,7 +62,7 @@ _scratch_cycle_mount echo "Create metadump file" _scratch_unmount -_scratch_xfs_metadump $metadump_file +_scratch_xfs_metadump $metadump_file -a # Now restore the obfuscated one back and take a look around echo "Restore metadump"
tests/xfs/432 +1 -1
@@ -86,7 +86,7 @@ echo "qualifying extent: $extlen blocks" >> $seqres.full test -n "$extlen" || _notrun "could not create dir extent > 1000 blocks" echo "Try to metadump" -_scratch_xfs_metadump $metadump_file -w +_scratch_xfs_metadump $metadump_file -a -o -w SCRATCH_DEV=$metadump_img _scratch_xfs_mdrestore $metadump_file echo "Check restored metadump image"
tests/xfs/503 +1 -1
@@ -46,7 +46,7 @@ metadump_file_ag=${metadump_file}.ag copy_file=$testdir/copy.img echo metadump -_scratch_xfs_metadump $metadump_file >> $seqres.full +_scratch_xfs_metadump $metadump_file -a -o >> $seqres.full echo metadump a _scratch_xfs_metadump $metadump_file_a -a >> $seqres.full
Signature · 1 line
2.43.0
PATCH v2 2/5 common/xfs: Add function to detect support for metadump v2
In reply to Chandan Babu RThis commit defines a new function to help detect support for metadump v2.
--- common/xfs | 6 ++++++ 1 file changed, 6 insertions(+)
common/xfs +6 -0
@@ -698,6 +698,12 @@ _xfs_mdrestore() { $XFS_MDRESTORE_PROG $options "${metadump}" "${device}" } +_scratch_metadump_v2_supported() +{ + $XFS_DB_PROG -c "help metadump" $SCRATCH_DEV | \ + grep -q "Metadump version to be used" +} + # Snapshot the metadata on the scratch device _scratch_xfs_metadump() {
Signature · 1 line
2.43.0
Re: [PATCH V2 2/5] common/xfs: Add function to detect support for metadump v2
Quoted 12 lines · Chandan Babu R
This commit defines a new function to help detect support for metadump v2.
Signed-off-by Chandan Babu R <[email protected]>--- common/xfs | 6 ++++++ 1 file changed, 6 insertions(+)common/xfs +0
-0
common/xfs _xfs_mdrestore() { · line 701
Quoted 4 lines
common/xfs +1
-0@@ -698,6 +698,12 @@ _xfs_mdrestore() { $XFS_MDRESTORE_PROG $options "${metadump}" "${device}" } +_scratch_metadump_v2_supported()
/me wonders if the "_scratch" prefix is really warranted here since metadump v2 support is a property of the tool, not the scratch fs. OTOH xfs_db won't give you subcommand help without passing in a filesystem and we probably don't want to force-feed it $TEST_DEV so if zorro doesn't have any objections,
--D
Quoted 12 lines
common/xfs +5
-0+{ + $XFS_DB_PROG -c "help metadump" $SCRATCH_DEV | \ + grep -q "Metadump version to be used" +} + # Snapshot the metadata on the scratch device _scratch_xfs_metadump() {Signature · 1 line
2.43.0
PATCH v2 3/5 _scratch_xfs_mdrestore: Pass scratch log device when applicable
In reply to Chandan Babu RMetadump v2 supports dumping contents of an external log device. This commit modifies _scratch_xfs_mdrestore() and _xfs_mdrestore() to be able to restore metadump files which contain data from external log devices.
The callers of _scratch_xfs_mdrestore() must set the value of $SCRATCH_LOGDEV only when all of the following conditions are met:
1. Metadump is in v2 format.
2. Metadump has contents dumped from an external log device.
--- common/xfs | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
common/xfs +17 -2
@@ -682,7 +682,8 @@ _xfs_metadump() { _xfs_mdrestore() { local metadump="$1" local device="$2" - shift; shift + local logdev="$3" + shift; shift; shift local options="$@" # If we're configured for compressed dumps and there isn't already an
@@ -695,6 +696,10 @@ _xfs_mdrestore() { fi test -r "$metadump" || return 1 + if [ "$logdev" != "none" ]; then + options="$options -l $logdev" + fi + $XFS_MDRESTORE_PROG $options "${metadump}" "${device}" }
@@ -722,8 +727,18 @@ _scratch_xfs_mdrestore() { local metadump=$1 shift + local logdev=none + local options="$@" - _xfs_mdrestore "$metadump" "$SCRATCH_DEV" "$@" + # $SCRATCH_LOGDEV should have a non-zero length value only when all of + # the following conditions are met. + # 1. Metadump is in v2 format. + # 2. Metadump has contents dumped from an external log device. + if [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_LOGDEV" ]; then + logdev=$SCRATCH_LOGDEV + fi + + _xfs_mdrestore "$metadump" "$SCRATCH_DEV" "$logdev" "$@" } # Do not use xfs_repair (offline fsck) to rebuild the filesystem
Signature · 1 line
2.43.0
PATCH v2 4/5 xfs: Add support for testing metadump v2
In reply to Chandan Babu RThis commit adds the ability to test metadump v2 to existing metadump tests.
--- tests/xfs/129 | 97 ++++++++++++++++++++++++++++++++----- tests/xfs/129.out | 4 +- tests/xfs/234 | 97 ++++++++++++++++++++++++++++++++----- tests/xfs/234.out | 4 +- tests/xfs/253 | 118 ++++++++++++++++++++++++++++++++++++++-------- tests/xfs/291 | 25 ++++++++-- tests/xfs/432 | 29 +++++++++--- tests/xfs/432.out | 3 +- tests/xfs/503 | 94 ++++++++++++++++++++++-------------- tests/xfs/503.out | 12 ++--- 10 files changed, 381 insertions(+), 102 deletions(-)
tests/xfs/129 +86 -11
@@ -16,7 +16,11 @@ _cleanup() { cd / _scratch_unmount > /dev/null 2>&1 - rm -rf $tmp.* $testdir $metadump_file $TEST_DIR/image + [[ -n $logdev && $logdev != "none" && $logdev != $SCRATCH_LOGDEV ]] && \ + _destroy_loop_device $logdev + [[ -n $datadev ]] && _destroy_loop_device $datadev + rm -rf $tmp.* $testdir $metadump_file $TEST_DIR/data-image \ + $TEST_DIR/log-image } # Import common functions.
@@ -29,12 +33,86 @@ _require_command "$XFS_MDRESTORE_PROG" "xfs_mdrestore" _require_loop _require_scratch_reflink +metadump_file=$TEST_DIR/${seq}_metadump + +verify_metadump_v1() +{ + local max_version=$1 + local version="" + + if [[ $max_version == 2 ]]; then + version="-v 1" + fi + + _scratch_xfs_metadump $metadump_file $version + + SCRATCH_DEV=$TEST_DIR/data-image SCRATCH_LOGDEV="" \ + _scratch_xfs_mdrestore $metadump_file + + datadev=$(_create_loop_device $TEST_DIR/data-image) + + SCRATCH_DEV=$datadev _scratch_mount + SCRATCH_DEV=$datadev _scratch_unmount + + logdev=$SCRATCH_LOGDEV + [[ -z $logdev ]] && logdev=none + _check_xfs_filesystem $datadev $logdev none + + _destroy_loop_device $datadev + datadev="" + rm -f $TEST_DIR/data-image +} + +verify_metadump_v2() +{ + version="-v 2" + + _scratch_xfs_metadump $metadump_file $version + + # Metadump v2 files can contain contents dumped from an external log + # device. Use a temporary file to hold the log device contents restored + # from such a metadump file. + slogdev="" + if [[ -n $SCRATCH_LOGDEV ]]; then + slogdev=$TEST_DIR/log-image + fi + + SCRATCH_DEV=$TEST_DIR/data-image SCRATCH_LOGDEV=$slogdev \ + _scratch_xfs_mdrestore $metadump_file + + datadev=$(_create_loop_device $TEST_DIR/data-image) + + logdev=${SCRATCH_LOGDEV} + if [[ -s $TEST_DIR/log-image ]]; then + logdev=$(_create_loop_device $TEST_DIR/log-image) + fi + + SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _scratch_mount + SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _scratch_unmount + + [[ -z $logdev ]] && logdev=none + _check_xfs_filesystem $datadev $logdev none + + if [[ -s $TEST_DIR/log-image ]]; then + _destroy_loop_device $logdev + logdev="" + rm -f $TEST_DIR/log-image + fi + + _destroy_loop_device $datadev + datadev="" + rm -f $TEST_DIR/data-image +} + _scratch_mkfs >/dev/null 2>&1 + +max_md_version=1 +_scratch_metadump_v2_supported && max_md_version=2 + _scratch_mount testdir=$SCRATCH_MNT/test-$seq mkdir $testdir -metadump_file=$TEST_DIR/${seq}_metadump echo "Create the original file blocks" blksz="$(_get_file_block_size $testdir)"
@@ -47,18 +125,15 @@ seq 1 2 $((nr_blks - 1)) | while read nr; do $testdir/file2 $((nr * blksz)) $blksz >> $seqres.full done -echo "Create metadump file" _scratch_unmount -_scratch_xfs_metadump $metadump_file -# Now restore the obfuscated one back and take a look around -echo "Restore metadump" -SCRATCH_DEV=$TEST_DIR/image _scratch_xfs_mdrestore $metadump_file -SCRATCH_DEV=$TEST_DIR/image _scratch_mount -SCRATCH_DEV=$TEST_DIR/image _scratch_unmount +echo "Create metadump file, restore it and check restored fs" -echo "Check restored fs" -_check_generic_filesystem $metadump_file +verify_metadump_v1 $max_md_version + +if [[ $max_md_version == 2 ]]; then + verify_metadump_v2 +fi # success, all done status=0
tests/xfs/129.out +1 -3
@@ -1,6 +1,4 @@ QA output created by 129 Create the original file blocks Reflink every other block -Create metadump file -Restore metadump -Check restored fs +Create metadump file, restore it and check restored fs
tests/xfs/234 +86 -11
123 lines not on this page · raw message
tests/xfs/234.out +1 -3
@@ -1,6 +1,4 @@ QA output created by 234 Create the original file blocks Punch every other block -Create metadump file -Restore metadump -Check restored fs +Create metadump file, restore it and check restored fs
tests/xfs/253 +99 -19
156 lines not on this page · raw message
tests/xfs/291 +21 -4
@@ -92,10 +92,27 @@ _scratch_xfs_check >> $seqres.full 2>&1 || _fail "xfs_check failed" # Yes they can! Now... # Can xfs_metadump cope with this monster? -_scratch_xfs_metadump $tmp.metadump -a -o || _fail "xfs_metadump failed" -SCRATCH_DEV=$tmp.img _scratch_xfs_mdrestore $tmp.metadump || _fail "xfs_mdrestore failed" -SCRATCH_DEV=$tmp.img _scratch_xfs_repair -f &>> $seqres.full || \ - _fail "xfs_repair of metadump failed" +max_md_version=1 +_scratch_metadump_v2_supported && max_md_version=2 + +for md_version in $(seq 1 $max_md_version); do + version="" + if [[ $max_md_version == 2 ]]; then + version="-v $md_version" + fi + + _scratch_xfs_metadump $tmp.metadump -a -o $version || \ + _fail "xfs_metadump failed" + + slogdev=$SCRATCH_LOGDEV + if [[ -z $version || $version == "-v 1" ]]; then + slogdev="" + fi + SCRATCH_DEV=$tmp.img SCRATCH_LOGDEV=$slogdev _scratch_xfs_mdrestore \ + $tmp.metadump || _fail "xfs_mdrestore failed" + SCRATCH_DEV=$tmp.img _scratch_xfs_repair -f &>> $seqres.full || \ + _fail "xfs_repair of metadump failed" +done # Yes it can; success, all done status=0
tests/xfs/432 +23 -6
@@ -50,6 +50,7 @@ echo "Format and mount" # block. 8187 hashes/dablk / 248 dirents/dirblock = ~33 dirblocks per # dablock. 33 dirblocks * 64k mean that we can expand a directory by # 2112k before we have to allocate another da btree block. + _scratch_mkfs -b size=1k -n size=64k > "$seqres.full" 2>&1 _scratch_mount >> "$seqres.full" 2>&1
@@ -85,13 +86,29 @@ extlen="$(check_for_long_extent $dir_inum)" echo "qualifying extent: $extlen blocks" >> $seqres.full test -n "$extlen" || _notrun "could not create dir extent > 1000 blocks" -echo "Try to metadump" -_scratch_xfs_metadump $metadump_file -a -o -w -SCRATCH_DEV=$metadump_img _scratch_xfs_mdrestore $metadump_file +echo "Try to metadump, restore and check restored metadump image" +max_md_version=1 +_scratch_metadump_v2_supported && max_md_version=2 -echo "Check restored metadump image" -SCRATCH_DEV=$metadump_img _scratch_xfs_repair -n &>> $seqres.full || \ - echo "xfs_repair on restored fs returned $?" +for md_version in $(seq 1 $max_md_version); do + version="" + if [[ $max_md_version == 2 ]]; then + version="-v $md_version" + fi + + _scratch_xfs_metadump $metadump_file -a -o -w $version + + slogdev=$SCRATCH_LOGDEV + if [[ -z $version || $version == "-v 1" ]]; then + slogdev="" + fi + + SCRATCH_DEV=$metadump_img SCRATCH_LOGDEV=$slogdev \ + _scratch_xfs_mdrestore $metadump_file + + SCRATCH_DEV=$metadump_img _scratch_xfs_repair -n &>> $seqres.full || \ + echo "xfs_repair on restored fs returned $?" +done # success, all done status=0
tests/xfs/432.out +1 -2
@@ -2,5 +2,4 @@ QA output created by 432 Format and mount Create huge dir Check for > 1000 block extent? -Try to metadump -Check restored metadump image +Try to metadump, restore and check restored metadump image
tests/xfs/503 +59 -35
117 lines not on this page · raw message
tests/xfs/503.out +4 -8
@@ -1,12 +1,8 @@ QA output created by 503 Format and populate -metadump -metadump a -metadump g -metadump ag +metadump and mdrestore +metadump a and mdrestore +metadump g and mdrestore +metadump ag and mdrestore copy recopy -mdrestore -mdrestore a -mdrestore g -mdrestore ag
Signature · 1 line
2.43.0
PATCH v2 5/5 xfs: Check correctness of metadump/mdrestore's ability to work with dirty log
In reply to Chandan Babu RAdd a new test to verify if metadump/mdrestore are able to dump and restore the contents of a dirty log.
--- tests/xfs/801 | 178 ++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/801.out | 14 ++++ 2 files changed, 192 insertions(+) create mode 100755 tests/xfs/801 create mode 100644 tests/xfs/801.out
tests/xfs/801 new file +178 -0
@@ -0,0 +1,178 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024 Oracle, Inc. All Rights Reserved. +# +# FS QA Test 801 +# +# Test metadump/mdrestore's ability to dump a dirty log and restore it +# correctly. +# +. ./common/preamble +_begin_fstest auto quick metadump log logprint punch + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* + _scratch_unmount > /dev/null 2>&1 + [[ -n $logdev && $logdev != "none" && $logdev != $SCRATCH_LOGDEV ]] && \ + _destroy_loop_device $logdev + [[ -n $datadev ]] && _destroy_loop_device $datadev + rm -r -f $metadump_file $TEST_DIR/data-image \ + $TEST_DIR/log-image +} + +# Import common functions. +. ./common/dmflakey +. ./common/inject + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_test +_require_loop +_require_xfs_debug +_require_xfs_io_error_injection log_item_pin +_require_dm_target flakey +_require_xfs_io_command "pwrite" +_require_test_program "punch-alternating" + +metadump_file=${TEST_DIR}/${seq}.md +testfile=${SCRATCH_MNT}/testfile + +echo "Format filesystem on scratch device" +_scratch_mkfs >> $seqres.full 2>&1 + +max_md_version=1 +_scratch_metadump_v2_supported && max_md_version=2 + +external_log=0 +if [[ $USE_EXTERNAL = yes && -n "$SCRATCH_LOGDEV" ]]; then + external_log=1 +fi + +if [[ $max_md_version == 1 && $external_log == 1 ]]; then + _notrun "metadump v1 does not support external log device" +fi + +verify_metadump_v1() +{ + local version="" + if [[ $max_md_version == 2 ]]; then + version="-v 1" + fi + + _scratch_xfs_metadump $metadump_file -a -o $version + + SCRATCH_DEV=$TEST_DIR/data-image _scratch_xfs_mdrestore $metadump_file + + datadev=$(_create_loop_device $TEST_DIR/data-image) + + SCRATCH_DEV=$datadev _scratch_mount + SCRATCH_DEV=$datadev _check_scratch_fs + SCRATCH_DEV=$datadev _scratch_unmount + + _destroy_loop_device $datadev + datadev="" + rm -f $TEST_DIR/data-image +} + +verify_metadump_v2() +{ + local version="-v 2" + + _scratch_xfs_metadump $metadump_file -a -o $version + + # Metadump v2 files can contain contents dumped from an external log + # device. Use a temporary file to hold the log device contents restored + # from such a metadump file. + slogdev="" + if [[ -n $SCRATCH_LOGDEV ]]; then + slogdev=$TEST_DIR/log-image + fi + + SCRATCH_DEV=$TEST_DIR/data-image SCRATCH_LOGDEV=$slogdev \ + _scratch_xfs_mdrestore $metadump_file + + datadev=$(_create_loop_device $TEST_DIR/data-image) + + logdev=${SCRATCH_LOGDEV} + if [[ -s $TEST_DIR/log-image ]]; then + logdev=$(_create_loop_device $TEST_DIR/log-image) + fi + + SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _scratch_mount + SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _check_scratch_fs + SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _scratch_unmount + + if [[ -s $TEST_DIR/log-image ]]; then + _destroy_loop_device $logdev + logdev="" + rm -f $TEST_DIR/log-image + fi + + _destroy_loop_device $datadev + datadev="" + rm -f $TEST_DIR/data-image +} + +echo "Initialize and mount filesystem on flakey device" +_init_flakey +_load_flakey_table $FLAKEY_ALLOW_WRITES +_mount_flakey + +echo "Create test file" +$XFS_IO_PROG -s -f -c "pwrite 0 5M" $testfile >> $seqres.full + +echo "Punch alternative blocks of test file" +$here/src/punch-alternating $testfile + +echo "Mount cycle the filesystem on flakey device" +_unmount_flakey +_mount_flakey + +device=$(readlink -f $FLAKEY_DEV) +device=$(_short_dev $device) + +echo "Pin log items in the AIL" +echo 1 > /sys/fs/xfs/${device}/errortag/log_item_pin + +echo "Create two checkpoint transactions on ondisk log" +for ct in $(seq 1 2); do + offset=$($XFS_IO_PROG -c 'fiemap' $testfile | tac | grep -v hole | \ + head -n 1 | awk -F '[\\[.]' '{ print $2 * 512; }') + $XFS_IO_PROG -c "truncate $offset" -c fsync $testfile +done + +echo "Drop writes to filesystem from here onwards" +_load_flakey_table $FLAKEY_DROP_WRITES + +echo "Unpin log items in AIL" +echo 0 > /sys/fs/xfs/${device}/errortag/log_item_pin + +echo "Unmount filesystem on flakey device" +_unmount_flakey + +echo "Clean up flakey device" +_cleanup_flakey + +echo -n "Filesystem has a " +_print_logstate + +echo "Create metadump file, restore it and check restored fs" + +if [[ $external_log == 0 ]]; then + verify_metadump_v1 $max_md_version +fi + +if [[ $max_md_version == 2 ]]; then + verify_metadump_v2 +fi + +# Mount the fs to replay the contents from the dirty log. +_scratch_mount + +# success, all done +status=0 +exit
tests/xfs/801.out new file +14 -0
@@ -0,0 +1,14 @@ +QA output created by 801 +Format filesystem on scratch device +Initialize and mount filesystem on flakey device +Create test file +Punch alternative blocks of test file +Mount cycle the filesystem on flakey device +Pin log items in the AIL +Create two checkpoint transactions on ondisk log +Drop writes to filesystem from here onwards +Unpin log items in AIL +Unmount filesystem on flakey device +Clean up flakey device +Filesystem has a dirty log +Create metadump file, restore it and check restored fs
Signature · 1 line
2.43.0
Re: [PATCH V2 5/5] xfs: Check correctness of metadump/mdrestore's ability to work with dirty log
Quoted 99 lines · Chandan Babu R
Add a new test to verify if metadump/mdrestore are able to dump and restore the contents of a dirty log.
Signed-off-by Chandan Babu R <[email protected]>--- tests/xfs/801 | 178 ++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/801.out | 14 ++++ 2 files changed, 192 insertions(+) create mode 100755 tests/xfs/801 create mode 100644 tests/xfs/801.outtests/xfs/801 new file +82
-0@@ -0,0 +1,178 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024 Oracle, Inc. All Rights Reserved. +# +# FS QA Test 801 +# +# Test metadump/mdrestore's ability to dump a dirty log and restore it +# correctly. +# +. ./common/preamble +_begin_fstest auto quick metadump log logprint punch + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* + _scratch_unmount > /dev/null 2>&1 + [[ -n $logdev && $logdev != "none" && $logdev != $SCRATCH_LOGDEV ]] && \ + _destroy_loop_device $logdev + [[ -n $datadev ]] && _destroy_loop_device $datadev + rm -r -f $metadump_file $TEST_DIR/data-image \ + $TEST_DIR/log-image +} + +# Import common functions. +. ./common/dmflakey +. ./common/inject + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_test +_require_loop +_require_xfs_debug +_require_xfs_io_error_injection log_item_pin +_require_dm_target flakey +_require_xfs_io_command "pwrite" +_require_test_program "punch-alternating" + +metadump_file=${TEST_DIR}/${seq}.md +testfile=${SCRATCH_MNT}/testfile + +echo "Format filesystem on scratch device" +_scratch_mkfs >> $seqres.full 2>&1 + +max_md_version=1 +_scratch_metadump_v2_supported && max_md_version=2 + +external_log=0 +if [[ $USE_EXTERNAL = yes && -n "$SCRATCH_LOGDEV" ]]; then + external_log=1 +fi + +if [[ $max_md_version == 1 && $external_log == 1 ]]; then + _notrun "metadump v1 does not support external log device" +fi + +verify_metadump_v1() +{ + local version="" + if [[ $max_md_version == 2 ]]; then + version="-v 1" + fi + + _scratch_xfs_metadump $metadump_file -a -o $version + + SCRATCH_DEV=$TEST_DIR/data-image _scratch_xfs_mdrestore $metadump_file + + datadev=$(_create_loop_device $TEST_DIR/data-image) + + SCRATCH_DEV=$datadev _scratch_mount + SCRATCH_DEV=$datadev _check_scratch_fs + SCRATCH_DEV=$datadev _scratch_unmount + + _destroy_loop_device $datadev + datadev="" + rm -f $TEST_DIR/data-image +} + +verify_metadump_v2() +{
tests/xfs/801 line 92
Quoted 10 lines
tests/xfs/801 new file +10
-0+ local version="-v 2" + + _scratch_xfs_metadump $metadump_file -a -o $version + + # Metadump v2 files can contain contents dumped from an external log + # device. Use a temporary file to hold the log device contents restored + # from such a metadump file. + slogdev="" + if [[ -n $SCRATCH_LOGDEV ]]; then + slogdev=$TEST_DIR/log-image
Why not create the loopdevs here?
tests/xfs/801 line 102
Quoted 10 lines
tests/xfs/801 new file +10
-0+ fi + + SCRATCH_DEV=$TEST_DIR/data-image SCRATCH_LOGDEV=$slogdev \ + _scratch_xfs_mdrestore $metadump_file + + datadev=$(_create_loop_device $TEST_DIR/data-image) + + logdev=${SCRATCH_LOGDEV} + if [[ -s $TEST_DIR/log-image ]]; then + logdev=$(_create_loop_device $TEST_DIR/log-image)
if [[ -s $slogdev ]]; then
logdev=$(_create_loop_device $slogdev)
fi
When would we have logdev == SCRATCH_LOGDEV at this point in the program?
--D
Quoted 100 lines
tests/xfs/801 new file +76
-0+ fi + + SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _scratch_mount + SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _check_scratch_fs + SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _scratch_unmount + + if [[ -s $TEST_DIR/log-image ]]; then + _destroy_loop_device $logdev + logdev="" + rm -f $TEST_DIR/log-image + fi + + _destroy_loop_device $datadev + datadev="" + rm -f $TEST_DIR/data-image +} + +echo "Initialize and mount filesystem on flakey device" +_init_flakey +_load_flakey_table $FLAKEY_ALLOW_WRITES +_mount_flakey + +echo "Create test file" +$XFS_IO_PROG -s -f -c "pwrite 0 5M" $testfile >> $seqres.full + +echo "Punch alternative blocks of test file" +$here/src/punch-alternating $testfile + +echo "Mount cycle the filesystem on flakey device" +_unmount_flakey +_mount_flakey + +device=$(readlink -f $FLAKEY_DEV) +device=$(_short_dev $device) + +echo "Pin log items in the AIL" +echo 1 > /sys/fs/xfs/${device}/errortag/log_item_pin + +echo "Create two checkpoint transactions on ondisk log" +for ct in $(seq 1 2); do + offset=$($XFS_IO_PROG -c 'fiemap' $testfile | tac | grep -v hole | \ + head -n 1 | awk -F '[\\[.]' '{ print $2 * 512; }') + $XFS_IO_PROG -c "truncate $offset" -c fsync $testfile +done + +echo "Drop writes to filesystem from here onwards" +_load_flakey_table $FLAKEY_DROP_WRITES + +echo "Unpin log items in AIL" +echo 0 > /sys/fs/xfs/${device}/errortag/log_item_pin + +echo "Unmount filesystem on flakey device" +_unmount_flakey + +echo "Clean up flakey device" +_cleanup_flakey + +echo -n "Filesystem has a " +_print_logstate + +echo "Create metadump file, restore it and check restored fs" + +if [[ $external_log == 0 ]]; then + verify_metadump_v1 $max_md_version +fi + +if [[ $max_md_version == 2 ]]; then + verify_metadump_v2 +fi + +# Mount the fs to replay the contents from the dirty log. +_scratch_mount + +# success, all done +status=0 +exittests/xfs/801.out new file +14
-0@@ -0,0 +1,14 @@ +QA output created by 801 +Format filesystem on scratch device +Initialize and mount filesystem on flakey device +Create test file +Punch alternative blocks of test file +Mount cycle the filesystem on flakey device +Pin log items in the AIL +Create two checkpoint transactions on ondisk log +Drop writes to filesystem from here onwards +Unpin log items in AIL +Unmount filesystem on flakey device +Clean up flakey device +Filesystem has a dirty log +Create metadump file, restore it and check restored fsSignature · 1 line
2.43.0
Re: [PATCH V2 5/5] xfs: Check correctness of metadump/mdrestore's ability to work with dirty log
Quoted 104 lines · Darrick J. Wong
Quoted 103 lines · Chandan Babu R
Add a new test to verify if metadump/mdrestore are able to dump and restore the contents of a dirty log.
Signed-off-by Chandan Babu R <[email protected]>--- tests/xfs/801 | 178 ++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/801.out | 14 ++++ 2 files changed, 192 insertions(+) create mode 100755 tests/xfs/801 create mode 100644 tests/xfs/801.outtests/xfs/801 new file +86
-0@@ -0,0 +1,178 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2024 Oracle, Inc. All Rights Reserved. +# +# FS QA Test 801 +# +# Test metadump/mdrestore's ability to dump a dirty log and restore it +# correctly. +# +. ./common/preamble +_begin_fstest auto quick metadump log logprint punch + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* + _scratch_unmount > /dev/null 2>&1 + [[ -n $logdev && $logdev != "none" && $logdev != $SCRATCH_LOGDEV ]] && \ + _destroy_loop_device $logdev + [[ -n $datadev ]] && _destroy_loop_device $datadev + rm -r -f $metadump_file $TEST_DIR/data-image \ + $TEST_DIR/log-image +} + +# Import common functions. +. ./common/dmflakey +. ./common/inject + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_test +_require_loop +_require_xfs_debug +_require_xfs_io_error_injection log_item_pin +_require_dm_target flakey +_require_xfs_io_command "pwrite" +_require_test_program "punch-alternating" + +metadump_file=${TEST_DIR}/${seq}.md +testfile=${SCRATCH_MNT}/testfile + +echo "Format filesystem on scratch device" +_scratch_mkfs >> $seqres.full 2>&1 + +max_md_version=1 +_scratch_metadump_v2_supported && max_md_version=2 + +external_log=0 +if [[ $USE_EXTERNAL = yes && -n "$SCRATCH_LOGDEV" ]]; then + external_log=1 +fi + +if [[ $max_md_version == 1 && $external_log == 1 ]]; then + _notrun "metadump v1 does not support external log device" +fi + +verify_metadump_v1() +{ + local version="" + if [[ $max_md_version == 2 ]]; then + version="-v 1" + fi + + _scratch_xfs_metadump $metadump_file -a -o $version + + SCRATCH_DEV=$TEST_DIR/data-image _scratch_xfs_mdrestore $metadump_file + + datadev=$(_create_loop_device $TEST_DIR/data-image) + + SCRATCH_DEV=$datadev _scratch_mount + SCRATCH_DEV=$datadev _check_scratch_fs + SCRATCH_DEV=$datadev _scratch_unmount + + _destroy_loop_device $datadev + datadev="" + rm -f $TEST_DIR/data-image +} + +verify_metadump_v2() +{ + local version="-v 2" + + _scratch_xfs_metadump $metadump_file -a -o $version +
Quoted 9 lines
tests/xfs/801 line 92
Quoted 6 lines
tests/xfs/801 new file +6
-0+ # Metadump v2 files can contain contents dumped from an external log + # device. Use a temporary file to hold the log device contents restored + # from such a metadump file. + slogdev="" + if [[ -n $SCRATCH_LOGDEV ]]; then + slogdev=$TEST_DIR/log-imageWhy not create the loopdevs here?
The backing files (i.e. $TEST_DIR/data-image and $TEST_DIR/log-image) have not yet been created. They are created by the invocation of mdrestore command below.
Quoted 12 lines
tests/xfs/801 line 102
Quoted 10 lines
tests/xfs/801 new file +10
-0+ fi + + SCRATCH_DEV=$TEST_DIR/data-image SCRATCH_LOGDEV=$slogdev \ + _scratch_xfs_mdrestore $metadump_file + + datadev=$(_create_loop_device $TEST_DIR/data-image) + + logdev=${SCRATCH_LOGDEV} + if [[ -s $TEST_DIR/log-image ]]; then + logdev=$(_create_loop_device $TEST_DIR/log-image)if [[ -s $slogdev ]]; then
Quoted 4 lines
logdev=$(_create_loop_device $slogdev)fi
When would we have logdev ==
SCRATCH_LOGDEVat this point in the program?
logdev == SCRATCH_LOGDEV only when using an internal log. I think a much cleaner way of initializing logdev would be
logdev=""
Combining this with the change suggested by you earlier, the code now looks like the following,
verify_metadump_v2()
{
local version="-v 2"
_scratch_xfs_metadump $metadump_file -a -o $version
# Metadump v2 files can contain contents dumped from an external log
# device. Use a temporary file to hold the log device contents restored
# from such a metadump file.
slogdev=""
if [[ -n $SCRATCH_LOGDEV ]]; then
slogdev=$TEST_DIR/log-image
fi
SCRATCH_DEV=$TEST_DIR/data-image SCRATCH_LOGDEV=$slogdev \
_scratch_xfs_mdrestore $metadump_file
datadev=$(_create_loop_device $TEST_DIR/data-image)
logdev=""
if [[ -s $slogdev ]]; then
logdev=$(_create_loop_device $slogdev)
fi
SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _scratch_mount
SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _check_scratch_fs
SCRATCH_DEV=$datadev SCRATCH_LOGDEV=$logdev _scratch_unmount
if [[ -s $logdev ]]; then
_destroy_loop_device $logdev
logdev=""
rm -f $slogdev
fi
_destroy_loop_device $datadev
datadev=""
rm -f $TEST_DIR/data-image
}Signature · 1 line
Chandan