Skip to contentSkip to conversation

fstests: concurrent test execution

RFC PATCH In review Dave Chinner 3 participants 65 messages Nov 27, 2024 – Dec 10, 2024 Page 2 of 2

lore
Share & tools
lore
#20 lore raw

PATCH 13/40 fstests: clean up loop device instantiation

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Lots of tests do there own special thing with loop devices rather than using _create_loop_device() and _destroy_loop_device(). This oftens means they do not clean up after themselves properly, leaving stale loop devices around that result in unmountable test or scratch devices. This is common when tests are killed by user interrupt.

Even the tests that do use _destroy_loop_device and try to clean up often do it incorrectly, leading to spurious error messages.

Some tests try to use dynamic instantiation via "mount -o loop", but then don't clean up in the correct order or hack around to find the loop device that was instantiated because the test needs to know the instantiated device name

Clean this up by converting all the tests to use _create_loop_device() and _destroy_loop_device(). In all the tests, use the variable "loop_dev" for the device consistently. In _destroy_loop_device(), test that a device name has been passed so that we don't try to clean up the same device twice (e.g. once before test exit and again from the _cleanup() function). When we destroy a loop device, unset the variable used to hold the loop device name so that we don't try to destroy it twice.

This results in much more reliable cleanup and clean exit from fstests when killed by the user.

---
 common/metadump   | 32 ++++++++++++--------------------
 tests/generic/067 | 11 ++++++++---
 tests/generic/361 |  6 +++++-
 tests/generic/563 | 14 +++++++++-----
 tests/generic/564 | 12 ++++++------
 tests/generic/590 |  9 ++++++---
 tests/generic/744 |  4 ++--
 tests/generic/746 |  6 +++++-
 tests/xfs/014     |  7 +++++--
 tests/xfs/049     | 35 +++++++++++++++++++++--------------
 tests/xfs/073     | 24 +++++++++++++-----------
 tests/xfs/074     | 21 ++++++++++++---------
 tests/xfs/078     | 16 +++++++++-------
 tests/xfs/148     | 23 ++++++++++++++---------
 tests/xfs/149     |  4 ++++
 tests/xfs/216     | 15 ++++++++++-----
 tests/xfs/217     | 22 ++++++++++++++++------
 tests/xfs/250     | 14 +++++++++-----
 tests/xfs/259     | 13 ++++++++-----
 tests/xfs/513     | 44 ++++++++++++++++++++------------------------
 tests/xfs/521     |  8 ++++++--
 tests/xfs/528     | 10 +++++++---
 tests/xfs/530     | 11 ++++++-----
 tests/xfs/606     | 12 ++++++++----
 tests/xfs/613     | 40 ++++++++++++++--------------------------
 tests/xfs/613.out |  1 -
 26 files changed, 235 insertions(+), 179 deletions(-)
common/metadump +12 -20
index 3373edfe9..bc3038e3b 100644
@@ -24,17 +24,9 @@ _xfs_cleanup_verify_metadump()
 
 	test -n "$XFS_METADUMP_FILE" && rm -f "$XFS_METADUMP_FILE"
 
-	if [ -n "$XFS_METADUMP_IMG" ]; then
-		losetup -n -a -O BACK-FILE,NAME | grep "^$XFS_METADUMP_IMG" | while read backing ldev; do
-			losetup -d "$ldev"
-		done
-
-		# Don't call rm directly with a globbed argument here to avoid
-		# issues issues with variable expansions.
-		for img in "$XFS_METADUMP_IMG"*; do
-			test -e "$img" && rm -f "$img"
-		done
-	fi
+	[ -n "$md_data_loop_dev" ] && _destroy_loop_device $md_data_loop_dev
+	[ -n "$md_log_loop_dev" ] && _destroy_loop_device $md_log_loop_dev
+	rm -f $data_img $log_img
 }
 
 # Can xfs_metadump snapshot the fs metadata to a v1 metadump file?
@@ -122,25 +114,25 @@ _xfs_verify_metadump_v2()
 		_scratch_xfs_mdrestore $metadump_file
 
 	# Create loopdev for data device so we can mount the fs
-	data_loop=$(_create_loop_device $data_img)
+	md_data_loop_dev=$(_create_loop_device $data_img)
 
 	# Create loopdev for log device if we recovered anything
-	test -s "$log_img" && log_loop=$(_create_loop_device $log_img)
+	test -s "$log_img" && md_log_loop_dev=$(_create_loop_device $log_img)
 
 	# Mount fs, run an extra test, fsck, and unmount
-	SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop _scratch_mount
+	SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev _scratch_mount
 	if [ -n "$extra_test" ]; then
-		SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop $extra_test
+		SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev $extra_test
 	fi
-	SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop _check_xfs_scratch_fs
-	SCRATCH_DEV=$data_loop _scratch_unmount
+	SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev _check_xfs_scratch_fs
+	_unmount $md_data_loop_dev
 
 	# Tear down what we created
-	if [ -b "$log_loop" ]; then
-		_destroy_loop_device $log_loop
+	if [ -b "$md_log_loop_dev" ]; then
+		_destroy_loop_device $md_log_loop_dev
 		rm -f $log_img
 	fi
-	_destroy_loop_device $data_loop
+	_destroy_loop_device $md_data_loop_dev
 	rm -f $data_img
 }
 
tests/generic/067 +8 -3
index ccb1e3fbb..b45ae834f 100755
@@ -37,13 +37,18 @@ mount_nonexistent_mnt()
 	$MOUNT_PROG $SCRATCH_DEV $TEST_DIR/nosuchdir >>$seqres.full 2>&1
 }
 
-# fs driver should be able to handle mounting a free loop device gracefully
-# xfs ever hung, "ec53d1d xfs: don't block on buffer read errors" fixed it
+# fs driver should be able to handle mounting a free loop device gracefully xfs
+# ever hung, "ec53d1d xfs: don't block on buffer read errors" fixed it
+#
+# Using 'losetup -f' like this is racy. We might end up mounting a real loop dev
+# here, so unmount $SCRATCH_MNT (not the loop dev we might not own!) and ignore
+# any error it might return.
 mount_free_loopdev()
 {
 	echo "# mount a free loop device" >>$seqres.full
 	loopdev=`losetup -f`
-	$MOUNT_PROG -t $FSTYP $loopdev $SCRATCH_MNT >>$seqres.full 2>&1
+	_mount $loopdev $SCRATCH_MNT >>$seqres.full 2>&1
+	_unmount $SCRATCH_MNT >> /dev/null 2>&1
 }
 
 # mount with wrong fs type specified.
tests/generic/361 +5 -1
index 7273dd056..e2b798436 100755
@@ -17,7 +17,7 @@ _begin_fstest auto quick
 _cleanup()
 {
 	_unmount $fs_mnt
-	_destroy_loop_device $loop_dev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
 }
@@ -54,6 +54,10 @@ $XFS_IO_PROG -fc "pwrite 0 520m" $fs_mnt/testfile >>$seqres.full 2>&1
 # remount should not hang
 $MOUNT_PROG -o remount,ro $fs_mnt >>$seqres.full 2>&1
 
+_unmount $fs_mnt
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 echo "Silence is golden"
 status=0
tests/generic/563 +9 -5
index f0d2f404c..34d21170c 100755
@@ -87,12 +87,12 @@ reset()
 
 # cgroup I/O accounting doesn't work on partitions. Use a loop device to rule
 # that out.
-LOOP_DEV=$(_create_loop_device $SCRATCH_DEV)
-smajor=$((0x`stat -L -c %t $LOOP_DEV`))
-sminor=$((0x`stat -L -c %T $LOOP_DEV`))
+loop_dev=$(_create_loop_device $SCRATCH_DEV)
+smajor=$((0x`stat -L -c %t $loop_dev`))
+sminor=$((0x`stat -L -c %T $loop_dev`))
 
-_mkfs_dev $LOOP_DEV >> $seqres.full 2>&1
-_mount $LOOP_DEV $SCRATCH_MNT || _fail "mount failed"
+_mkfs_dev $loop_dev >> $seqres.full 2>&1
+_mount $loop_dev $SCRATCH_MNT || _fail "mount failed"
 
 blksize=$(_get_block_size "$SCRATCH_MNT")
 
@@ -147,6 +147,10 @@ if [ "$drop_io_cgroup" = 1 ]; then
 	echo "-io" > $cgdir/cgroup.subtree_control || _fail "subtree control"
 fi
 
+_unmount $SCRATCH_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/generic/564 +6 -6
index 647472d78..b66b72496 100755
@@ -19,7 +19,7 @@ _cleanup()
 {
 	cd /
 	rm -rf $tmp.*
-	[ -z "$loopdev" ] || _destroy_loop_device $loopdev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 }
 
 # Import common functions.
@@ -72,11 +72,11 @@ $XFS_IO_PROG -f -c "copy_range -l 32k $testdir" $testdir/copy
 echo
 echo source/destination as blkdev returns EINVAL
 $XFS_IO_PROG -f -c "truncate 128k" $testdir/img >> $seqres.full 2>&1
-loopdev=`_create_loop_device $testdir/img`
-$XFS_IO_PROG -c "copy_range -l 32k $testdir/file" $loopdev
-$XFS_IO_PROG -f -c "copy_range -l 32k $loopdev" $testdir/copy
-_destroy_loop_device $loopdev
-loopdev=
+loop_dev=`_create_loop_device $testdir/img`
+$XFS_IO_PROG -c "copy_range -l 32k $testdir/file" $loop_dev
+$XFS_IO_PROG -f -c "copy_range -l 32k $loop_dev" $testdir/copy
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo
 echo source/destination as chardev returns EINVAL
tests/generic/590 +6 -3
index 2b7ccfb53..2eefa2c62 100755
@@ -15,9 +15,10 @@ _begin_fstest auto prealloc preallocrw
 # Override the default cleanup function.
 _cleanup()
 {
+	_scratch_unmount
+	[ -n $loop_dev ] && _destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
-	test -n "$loop" && _destroy_loop_device "$loop"
 	rm -f "$TEST_DIR/$seq"
 }
 
@@ -56,9 +57,9 @@ if [[ $FSTYP = xfs ]]; then
 		loopsz="$((filesz + (1 << 26)))"
 		_require_fs_space "$TEST_DIR" $((loopsz / 1024))
 		$XFS_IO_PROG -c "truncate $loopsz" -f "$TEST_DIR/$seq"
-		loop="$(_create_loop_device "$TEST_DIR/$seq")"
+		loop_dev="$(_create_loop_device "$TEST_DIR/$seq")"
 		USE_EXTERNAL=yes
-		SCRATCH_RTDEV="$loop"
+		SCRATCH_RTDEV="$loop_dev"
 		disabled_features=()
 
 		# disable reflink if not supported by realtime devices
@@ -114,6 +115,8 @@ $XFS_IO_PROG -c "truncate 0" -c fsync "$SCRATCH_MNT/file"
 # We need to do this before the loop device gets torn down.
 _scratch_unmount
 _check_scratch_fs
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo "Silence is golden"
 status=0
tests/generic/744 +2 -2
index df8f6ae9b..cda10e0f6 100755
@@ -17,8 +17,8 @@ _cleanup()
 
 	_unmount $mnt2 &> /dev/null
 	_unmount $mnt1 &> /dev/null
-	[ -b "$loop_dev2" ] && losetup -d $loop_dev2
-	[ -b "$loop_dev1" ] && losetup -d $loop_dev1
+	[ -b "$loop_dev2" ] && _destroy_loop_device $loop_dev2
+	[ -b "$loop_dev1" ] && _destroy_loop_device $loop_dev1
 	[ -n "$seq" ] && rm -rf $TEST_DIR/$seq
 }
 
tests/generic/746 +5 -1
index 671910606..ba8ed25e8 100755
@@ -39,7 +39,7 @@ esac
 _cleanup()
 {
 	_unmount $loop_mnt &> /dev/null
-	_destroy_loop_device $loop_dev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	if [ $status -eq 0 ]; then
 		rm -rf $tmp
 		rm $img_file
@@ -223,5 +223,9 @@ while read line; do
 done < $fiemap_after
 echo "done."
 
+_unmount $loop_mnt
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 status=0
 exit
tests/xfs/014 +5 -2
index f638e4b16..39ea40e2a 100755
@@ -23,6 +23,7 @@ _cleanup()
 {
 	cd /
 	_unmount $LOOP_MNT 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	_scratch_unmount 2>/dev/null
 	rm -f $tmp.*
 }
@@ -166,19 +167,21 @@ LOOP_FILE=$SCRATCH_MNT/$seq.fs
 LOOP_MNT=$SCRATCH_MNT/$seq.mnt
 
 $MKFS_XFS_PROG -d "file=1,name=$LOOP_FILE,size=10g" >> $seqres.full 2>&1
+loop_dev=$(_create_loop_device $LOOP_FILE)
 
 mkdir -p $LOOP_MNT
-_mount -o uquota,gquota $LOOP_FILE $LOOP_MNT || \
+_mount -o uquota,gquota $loop_dev $LOOP_MNT || \
 	_fail "Failed to mount loop fs."
 
 _test_enospc $LOOP_MNT
 _test_edquot $LOOP_MNT
 
 _unmount $LOOP_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo $orig_sp_time > /proc/sys/fs/xfs/speculative_prealloc_lifetime
 
 _scratch_unmount
-
 status=0
 exit
tests/xfs/049 +21 -14
index 4163a144f..cdcddf764 100755
@@ -12,16 +12,17 @@ _begin_fstest rw auto quick
 # Override the default cleanup function.
 _cleanup()
 {
-    cd /
-    umount $SCRATCH_MNT/test2 > /dev/null 2>&1
-    umount $SCRATCH_MNT/test > /dev/null 2>&1
-    rm -f $tmp.*
+	cd /
+	_unmount $SCRATCH_MNT/test2 > /dev/null 2>&1
+	_unmount $SCRATCH_MNT/test > /dev/null 2>&1
+	[ -n "$loop_dev2" ] && _destroy_loop_device $loop_dev2
+	[ -n "$loop_dev1" ] && _destroy_loop_device $loop_dev1
+	rm -f $tmp.*
 
-    if [ -w $seqres.full ]
-    then
-        echo "--- mounts at end (after cleanup)" >> $seqres.full
-        mount >> $seqres.full
-    fi
+	if [ -w $seqres.full ]; then
+		echo "--- mounts at end (after cleanup)" >> $seqres.full
+		mount >> $seqres.full
+	fi
 }
 
 # Import common functions.
@@ -64,7 +65,8 @@ mkdir $SCRATCH_MNT/test $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to make mount points"
 
 _log "Mount xfs via loop"
-mount -t xfs -o loop $SCRATCH_MNT/test.xfs $SCRATCH_MNT/test >> $seqres.full 2>&1 \
+loop_dev1=$(_create_loop_device $SCRATCH_MNT/test.xfs)
+_mount $loop_dev1 $SCRATCH_MNT/test >> $seqres.full 2>&1 \
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress"
@@ -80,11 +82,12 @@ dd if=/dev/zero of=$SCRATCH_MNT/test/test.ext2 bs=1024 count=10240 >> $seqres.fu
     || _fail "!!! create file failed"
 
 _log "Create ext2 fs in file on looped xfs"
-echo y | mkfs -t ext2 $SCRATCH_MNT/test/test.ext2 >> $seqres.full 2>&1 \
+loop_dev2=$(_create_loop_device $SCRATCH_MNT/test/test.ext2)
+echo y | mkfs -t ext2 $loop_dev2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to mkfs ext2 on xfs"
 
 _log "Mount ext2 on xfs via loop"
-mount -t ext2 -o loop $SCRATCH_MNT/test/test.ext2 $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
+_mount $loop_dev2 $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress ext2 on xfs via loop"
@@ -96,12 +99,16 @@ rm -rf $SCRATCH_MNT/test/* >> $seqres.full 2>&1 \
     || _fail "!!! clean failed"
 
 _log "umount ext2 on xfs"
-umount $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
+_unmount $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! umount ext2 failed"
+_destroy_loop_device $loop_dev2
+unset loop_dev2
 
 _log "umount xfs"
-umount $SCRATCH_MNT/test >> $seqres.full 2>&1 \
+_unmount $SCRATCH_MNT/test >> $seqres.full 2>&1 \
     || _fail "!!! umount xfs failed"
+_destroy_loop_device $loop_dev1
+unset loop_dev1
 
 echo "--- mounts at end (before cleanup)" >> $seqres.full
 mount >> $seqres.full
tests/xfs/073 +13 -11
index e4b17c5e7..2274079ef 100755
@@ -23,6 +23,8 @@ _cleanup()
 	_scratch_unmount 2>/dev/null
 	_unmount $imgs.loop 2>/dev/null
 	_unmount $imgs.source_dir 2>/dev/null
+	[ -n "$loop_dev1" ] && _destroy_loop_device $loop_dev1
+	[ -n "$loop_dev2" ] && _destroy_loop_device $loop_dev2
 	[ -d $imgs.loop ] && rmdir $imgs.loop
 	[ -d $imgs.source_dir ] && rm -rf $imgs.source_dir
 	rm -f $imgs.* $tmp.* /var/tmp/xfs_copy.log.*
@@ -65,10 +67,11 @@ _verify_copy()
 	rmdir $target_dir 2>/dev/null
 	mkdir $target_dir
 
-	_mount -t xfs -o loop $target $target_dir 2>/dev/null
+	loop_dev1=$(_create_loop_device $target)
+	_mount $loop_dev1 $target_dir 2>/dev/null
 	if [ $? -ne 0 ]; then
 		echo retrying mount with nouuid option >>$seqres.full
-		_mount -t xfs -o loop -o nouuid $target $target_dir
+		_mount -o nouuid $loop_dev1 $target_dir
 		if [ $? -ne 0 ]; then
 			echo mount failed - evil!
 			return
@@ -100,6 +103,8 @@ _verify_copy()
 	echo unmounting and removing new image
 	_unmount $source_dir
 	_unmount $target_dir > /dev/null 2>&1
+	_destroy_loop_device $loop_dev1
+	unset loop_dev1
 	rm -f $target
 }
 
@@ -134,18 +139,15 @@ ${MKFS_XFS_PROG} -dfile,name=$imgs.source,size=100g \
 rmdir $imgs.source_dir 2>/dev/null
 mkdir $imgs.source_dir
 
-_mount -t xfs -o loop $imgs.source $imgs.source_dir
-loop2=`mount | grep $imgs.source | grep -o -e 'loop=.*[^),]' | grep -o -e '/.*$'`
+loop_dev2=$(_create_loop_device $imgs.source)
+_mount $loop_dev2 $imgs.source_dir
 cp -a $here $imgs.source_dir
-_mount -t xfs -o remount,ro $imgs.source $imgs.source_dir
-$XFS_COPY_PROG $imgs.source $imgs.image | _filter_copy '#' $imgs.image '#' '#'
+_mount -o remount,ro $loop_dev2 $imgs.source_dir
+$XFS_COPY_PROG $loop_dev2 $imgs.image 2> /dev/null | _filter_copy '#' $imgs.image '#' '#'
 _verify_copy $imgs.image $imgs.source $imgs.source_dir
 
-# HACK WARNING:
-#
-# We're done with the nested loop mount, now we have to clean
-# up the pieces that mount is incapable of doing.
-losetup -d $loop2 > /dev/null 2>&1
+_destroy_loop_device $loop_dev2
+unset loop_dev2
 
 echo
 echo === copying scratch device to multiple targets
tests/xfs/074 +12 -9
index 6a03c8111..5df864fad 100755
@@ -26,7 +26,7 @@ _begin_fstest quick auto prealloc rw
 _cleanup()
 {
 	cd /
-	_destroy_loop_device $LOOP_DEV
+	[ -n "$loop_dev" ] &&_destroy_loop_device $loop_dev
 	rm -f $tmp.* $LOOP_FILE
 }
 
@@ -45,10 +45,10 @@ LOOP_FILE=$TEST_DIR/$seq.img
 LOOP_MNT=$TEST_DIR/$seq.mnt
 mkdir -p $LOOP_MNT
 $XFS_IO_PROG -ft -c "truncate 1t" $LOOP_FILE >> $seqres.full
-LOOP_DEV=`_create_loop_device $LOOP_FILE`
+loop_dev=`_create_loop_device $LOOP_FILE`
 
-_mkfs_dev -d size=260g,agcount=2 $LOOP_DEV
-_mount $LOOP_DEV $LOOP_MNT
+_mkfs_dev -d size=260g,agcount=2 $loop_dev
+_mount $loop_dev $LOOP_MNT
 
 BLOCK_SIZE=$(_get_file_block_size $LOOP_MNT)
 
@@ -59,11 +59,11 @@ $XFS_IO_PROG -ft \
 	-c "falloc 0 $(($BLOCK_SIZE * 2097152))" \
 	$LOOP_MNT/foo >> $seqres.full
 
-umount $LOOP_MNT
-_check_xfs_filesystem $LOOP_DEV none none
+_unmount $LOOP_MNT
+_check_xfs_filesystem $loop_dev none none
 
-_mkfs_dev -f $LOOP_DEV
-_mount $LOOP_DEV $LOOP_MNT
+_mkfs_dev -f $loop_dev
+_mount $loop_dev $LOOP_MNT
 
 # check we trim both ends of the extent approproiately; this will fail
 # on 1k block size filesystems without the correct fixes in place.
@@ -73,7 +73,10 @@ $XFS_IO_PROG -ft \
 	$LOOP_MNT/foo >> $seqres.full
 
 _unmount $LOOP_MNT
-_check_xfs_filesystem $LOOP_DEV none none
+_check_xfs_filesystem $loop_dev none none
+
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 # success, all done
 echo "Silence is golden"
tests/xfs/078 +9 -7
index 6b325e05f..0d3c2eb23 100755
@@ -17,7 +17,7 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_unmount $LOOP_MNT 2>/dev/null
-	[ -n "$LOOP_DEV" ] && _destroy_loop_device $LOOP_DEV 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev 2>/dev/null
 	# try to keep the image file if test fails
 	[ $status -eq 0 ] && rm -f $LOOP_IMG
 	rmdir $LOOP_MNT
@@ -55,7 +55,7 @@ _grow_loop()
 	agsize=$5
 
 	$XFS_IO_PROG -f -c "truncate $original" $LOOP_IMG
-	LOOP_DEV=`_create_loop_device $LOOP_IMG`
+	loop_dev=`_create_loop_device $LOOP_IMG`
 
 	dparam=""
 	if [ -n "$agsize" ]; then
@@ -67,15 +67,15 @@ _grow_loop()
 	echo
 
 	echo "*** mkfs loop file (size=$original)"
-	$MKFS_XFS_PROG -b size=$bsize $dparam $LOOP_DEV | \
+	$MKFS_XFS_PROG -b size=$bsize $dparam $loop_dev | \
 		_filter_mkfs 2>/dev/null
 
 	echo "*** extend loop file"
-	_destroy_loop_device $LOOP_DEV
+	_destroy_loop_device $loop_dev
 	$XFS_IO_PROG -c "pwrite $new_size $bsize" $LOOP_IMG | _filter_io
-	LOOP_DEV=`_create_loop_device $LOOP_IMG`
+	loop_dev=`_create_loop_device $LOOP_IMG`
 	echo "*** mount loop filesystem"
-	_mount -t xfs $LOOP_DEV $LOOP_MNT
+	_mount $loop_dev $LOOP_MNT
 
 	echo "*** grow loop filesystem"
 	$XFS_GROWFS_PROG $LOOP_MNT 2>&1 |  _filter_growfs 2>&1
@@ -87,9 +87,11 @@ _grow_loop()
 	if [ "$check" -gt "0" ]
 	then
 		echo "*** check"
-		_check_xfs_filesystem $LOOP_IMG none none
+		_check_xfs_filesystem $loop_dev none none
 	fi
 
+	_destroy_loop_device $loop_dev
+	unset loop_dev
 	rm -f $LOOP_IMG
 }
 
tests/xfs/148 +14 -9
index c42c9b119..4d2f7a808 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_unmount $mntpt > /dev/null 2>&1
-	_destroy_loop_device $loopdev > /dev/null 2>&1
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	rm -r -f $tmp.*
 }
 
@@ -48,12 +48,12 @@ rm -f $imgfile $imgfile.old
 # even when security xattrs are present so we are always doing name matches on
 # lookup and not name hash compares as leaf/node forms will do.
 $XFS_IO_PROG -f -c 'truncate 40m' $imgfile
-loopdev=$(_create_loop_device $imgfile)
-MKFS_OPTIONS="-m crc=0 -i size=512" _mkfs_dev $loopdev >> $seqres.full
+loop_dev=$(_create_loop_device $imgfile)
+MKFS_OPTIONS="-m crc=0 -i size=512" _mkfs_dev $loop_dev >> $seqres.full
 
 # Mount image file
 mkdir -p $mntpt
-_mount $loopdev $mntpt
+_mount $loop_dev $mntpt
 
 echo "creating entries" >> $seqres.full
 
@@ -91,7 +91,8 @@ cat $tmp.log | _filter_test_dir
 
 # Corrupt the entries
 _unmount $mntpt
-_destroy_loop_device $loopdev
+_destroy_loop_device $loop_dev
+unset loop_dev
 cp $imgfile $imgfile.old
 sed -b \
 	-e "s/$nullstr/too_many\x00beans/g" \
@@ -99,8 +100,9 @@ sed -b \
 	-i $imgfile
 test "$(md5sum < $imgfile)" != "$(md5sum < $imgfile.old)" ||
 	_fail "sed failed to change the image file?"
-loopdev=$(_create_loop_device $imgfile)
-_mount $loopdev $mntpt
+
+loop_dev=$(_create_loop_device $imgfile)
+_mount $loop_dev $mntpt
 
 # Try to access the corrupt metadata
 echo "++ ACCESSING BAD METADATA" | tee -a $seqres.full
@@ -111,7 +113,7 @@ cat $tmp.log | _filter_test_dir | sed -e '/Could not list/d'
 echo "does scrub complain?" >> $seqres.full
 
 # Does scrub complain about this?
-if _supports_xfs_scrub $mntpt $loopdev; then
+if _supports_xfs_scrub $mntpt $loop_dev; then
 	$XFS_SCRUB_PROG -n $mntpt >> $seqres.full 2>&1
 	res=$?
 	test $((res & 1)) -eq 0 && \
@@ -122,11 +124,14 @@ echo "does repair complain?" >> $seqres.full
 
 # Does repair complain about this?
 _unmount $mntpt
-$XFS_REPAIR_PROG -n $loopdev >> $seqres.full 2>&1
+$XFS_REPAIR_PROG -n $loop_dev >> $seqres.full 2>&1
 res=$?
 test $res -eq 1 || \
 	echo "repair failed to report corruption ($res)"
 
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/149 +4 -0
index f2187109b..9a96f82ed 100755
@@ -84,6 +84,10 @@ $XFS_GROWFS_PROG -D 16384 $loop_symlink > /dev/null
 echo "=== xfs_growfs - check device node ==="
 $XFS_GROWFS_PROG -D 20480 $loop_dev > /dev/null
 
+_unmount $mntdir
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/216 +10 -5
index 6b8b2eb22..091c11d08 100755
@@ -15,6 +15,7 @@ _begin_fstest log metadata auto quick
 _cleanup()
 {
 	_unmount $LOOP_MNT > /dev/null 2>&1
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
 }
@@ -24,7 +25,7 @@ _scratch_mkfs_xfs >/dev/null 2>&1
 _scratch_mount
 
 _require_loop
-LOOP_DEV=$SCRATCH_MNT/test_fs
+LOOP_IMG=$SCRATCH_MNT/test_fs
 LOOP_MNT=$SCRATCH_MNT/test_fs_dir
 
 loop_mkfs_opts=
@@ -55,22 +56,26 @@ _do_mkfs()
 	for i in $*; do
 		echo -n "fssize=${i}g "
 		$MKFS_XFS_PROG -f -b size=4096 -l version=2 \
-			-d name=$LOOP_DEV,size=${i}g $loop_mkfs_opts |grep log
-		_mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
+			-d size=${i}g $loop_mkfs_opts $loop_dev |grep log
+		_mount $loop_dev $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
 		_unmount $LOOP_MNT > /dev/null 2>&1
 	done
 }
 # make large holey file
-$XFS_IO_PROG -f -c "truncate 256g" $LOOP_DEV
+$XFS_IO_PROG -f -c "truncate 256g" $LOOP_IMG
 
-choose_golden_output $0 $LOOP_DEV
+choose_golden_output $0 $LOOP_IMG
 
 #make loopback mount dir
 mkdir $LOOP_MNT
 
+loop_dev=$(_create_loop_device $LOOP_IMG)
+
 # walk over standard sizes (up to 256GB)
 _do_mkfs 1 2 4 8 16 32 64 128 256
 
+_destroy_loop_device $loop_dev
+unset loop_dev
 status=0
 exit
tests/xfs/217 +16 -6
index b2eb34490..dae6ce55f 100755
@@ -12,6 +12,12 @@ _begin_fstest log metadata auto
 # Import common functions.
 . ./common/filter
 
+_cleanup()
+{
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
+	cd /
+	rm -f $tmp.*
+}
 
 _require_scratch
 _scratch_mkfs_xfs >/dev/null 2>&1
@@ -20,7 +26,7 @@ _scratch_mount
 _require_fs_space $SCRATCH_MNT 2202000
 
 _require_loop
-LOOP_DEV=$SCRATCH_MNT/test_fs
+LOOP_IMG=$SCRATCH_MNT/test_fs
 LOOP_MNT=$SCRATCH_MNT/test_fs_dir
 
 _do_mkfs()
@@ -28,28 +34,30 @@ _do_mkfs()
 	for i in $*; do
 		echo -n "fssize=${i}g "
 		$MKFS_XFS_PROG -f -b size=4096 -l version=2 \
-			-d name=$LOOP_DEV,size=${i}g |grep log
-		_mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
+			-d size=${i}g $loop_dev |grep log
+		_mount $loop_dev $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
 		_unmount $LOOP_MNT > /dev/null 2>&1
 
 		# punch out the previous blocks so that we keep the amount of
 		# disk space the test requires down to a minimum.
-		$XFS_IO_PROG -f -c "unresvsp 0 16383g" $LOOP_DEV
+		$XFS_IO_PROG -f -c "unresvsp 0 16383g" $LOOP_IMG
 	done
 }
 # make large holey file
-$XFS_IO_PROG -f -c "truncate 16383g" $LOOP_DEV
+$XFS_IO_PROG -f -c "truncate 16383g" $LOOP_IMG
 
 #make loopback mount dir
 mkdir $LOOP_MNT
 
 # test if large logs are supported
-$MKFS_XFS_PROG -f -l size=256m -d name=$LOOP_DEV,size=10g > /dev/null 2>&1
+$MKFS_XFS_PROG -f -l size=256m -d name=$LOOP_IMG,size=10g > /dev/null 2>&1
 if [ $? -ne 0 ]; then
 	_notrun "large log sizes not supported by mkfs"
 fi
 
+loop_dev=$(_create_loop_device $LOOP_IMG)
+
 #
 # walk over "new" sizes supported by recent xfsprogs.
 # Note that the last test is for 16TB-1GB as 32bit platforms only support
@@ -57,5 +65,7 @@ fi
 #
 _do_mkfs 512 1024 2048 4096 8192 16383
 
+_destroy_loop_device $loop_dev
+unset loop_dev
 status=0
 exit
tests/xfs/250 +9 -5
index 4e3473ebc..2554e1e91 100755
@@ -14,7 +14,8 @@ _cleanup()
 {
 	cd /
 	_unmount $LOOP_MNT 2>/dev/null
-	rm -f $LOOP_DEV
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
+	rm -f $LOOP_IMG
 	rmdir $LOOP_MNT
 	rm -f $tmp.*
 }
@@ -26,7 +27,7 @@ _require_test
 _require_loop
 _require_xfs_io_command "falloc"
 
-LOOP_DEV=$TEST_DIR/$seq.fs
+LOOP_IMG=$TEST_DIR/$seq.fs
 LOOP_MNT=$TEST_DIR/$seq.mnt
 
 _filter_io()
@@ -45,7 +46,7 @@ _test_loop()
 	agsize=$2
 	fsize=$3
 
-	dparam="file,name=$LOOP_DEV,size=$size"
+	dparam="file,name=$LOOP_IMG,size=$size"
 	if [ -n "$agsize" ]; then
 		dparam="$dparam,agsize=$agsize"
 	fi
@@ -55,7 +56,8 @@ _test_loop()
 		| _filter_mkfs 2>/dev/null
 
 	echo "*** mount loop filesystem"
-	_mount -t xfs -o loop $LOOP_DEV $LOOP_MNT
+	loop_dev=$(_create_loop_device $LOOP_IMG)
+	mount $loop_dev $LOOP_MNT
 
 	echo "*** preallocate large file"
 	$XFS_IO_PROG -f -c "resvsp 0 $fsize" $LOOP_MNT/foo | _filter_io
@@ -64,7 +66,9 @@ _test_loop()
 	_unmount $LOOP_MNT > /dev/null 2>&1
 
 	echo "*** check loop filesystem"
-	 _check_xfs_filesystem $LOOP_DEV none none
+	_check_xfs_filesystem $loop_dev none none
+	_destroy_loop_device $loop_dev
+	unset loop_dev
 }
 
 _test_loop 50g 16m 40G
tests/xfs/259 +8 -5
index 0c8d6eb56..c2d26381a 100755
@@ -12,7 +12,10 @@ _begin_fstest auto quick
 # Override the default cleanup function.
 _cleanup()
 {
-    rm -f "$testfile"
+	[ -n "$loop_dev" ] && _destroy_loop_device $testfile
+	rm -f "$testfile"
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -45,13 +48,13 @@ for del in $sizes_to_check; do
 		rm -f "$testfile"
 		dd if=/dev/zero "of=$testfile" bs=1 count=0 seek=$ddseek \
 			>/dev/null 2>&1 || echo "dd failed"
-		lofile=$(losetup -f)
-		losetup $lofile "$testfile"
-		$MKFS_XFS_PROG -l size=32m -b size=$bs $lofile |  _filter_mkfs \
+		loop_dev=$(_create_loop_device $testfile)
+		$MKFS_XFS_PROG -l size=32m -b size=$bs $loop_dev |  _filter_mkfs \
 			>/dev/null 2> $tmp.mkfs || echo "mkfs failed!"
 		. $tmp.mkfs
 		sync
-		losetup -d $lofile
+		_destroy_loop_device $loop_dev
+		unset loop_dev
 	done
 done
 
tests/xfs/513 +20 -24

124 lines not on this page · raw message

tests/xfs/521 +6 -2
index 13982c440..c92c621a2 100755
@@ -21,7 +21,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -35,7 +35,7 @@ _require_no_large_scratch_dev
 
 echo "Create fake rt volume"
 truncate -s 400m $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Format and mount 100m rt volume"
 export USE_EXTERNAL=yes
@@ -69,6 +69,10 @@ cp -p $testdir/original $testdir/copy3
 echo "Check filesystem"
 _check_scratch_fs
 
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/528 +7 -3
index 6ca9a2370..a1efbbd27 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -155,11 +155,11 @@ test_ops() {
 
 echo "Create fake rt volume"
 $XFS_IO_PROG -f -c "truncate 400m" $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Make sure synth rt volume works"
 export USE_EXTERNAL=yes
-export SCRATCH_RTDEV=$rtdev
+export SCRATCH_RTDEV=$rt_loop_dev
 _scratch_mkfs > $seqres.full
 _try_scratch_mount || \
 	_notrun "Could not mount with synthetic rt volume"
@@ -170,6 +170,10 @@ test_ops 262144
 # not a power of two
 test_ops 327680
 
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/530 +6 -5
index 8a182bd6a..d0d0e2665 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -52,12 +52,12 @@ fi
 
 rtdevsz=$((nr_bits * rtextsz))
 truncate -s $rtdevsz $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Format and mount rt volume"
 
 export USE_EXTERNAL=yes
-export SCRATCH_RTDEV=$rtdev
+export SCRATCH_RTDEV=$rt_loop_dev
 _scratch_mkfs -d size=$((1024 * 1024 * 1024)) \
 	      -r size=${rtextsz},extsize=${rtextsz} >> $seqres.full
 _try_scratch_mount || _notrun "Couldn't mount fs with synthetic rt volume"
@@ -116,8 +116,9 @@ done
 echo "Check filesystem"
 _check_scratch_fs
 
-losetup -d $rtdev
-rm -f $TEST_DIR/$seq.rtvol
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
 
 # success, all done
 status=0
tests/xfs/606 +8 -4
index f958bddd8..b537ea145 100755
@@ -13,10 +13,8 @@ _begin_fstest auto quick growfs
 
 _cleanup()
 {
-	local dev
 	_unmount $LOOP_MNT 2>/dev/null
-	dev=$(losetup -j testfile | cut -d: -f1)
-	losetup -d $dev 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	rm -rf $LOOP_IMG $LOOP_MNT
 	cd /
 	rm -f $tmp.*
@@ -40,7 +38,9 @@ $MKFS_XFS_PROG -f $LOOP_IMG >$seqres.full
 # Extend by just 8K, expected to start with the last full-size AG ends of
 # above 1G block device.
 $XFS_IO_PROG -f -c "truncate 1073750016" $LOOP_IMG
-_mount -oloop $LOOP_IMG $LOOP_MNT
+
+loop_dev=$(_create_loop_device $LOOP_IMG)
+_mount $loop_dev $LOOP_MNT
 # A known bug shows "XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on
 # device" at here, refer to _fixed_by_kernel_commit above
 $XFS_GROWFS_PROG $LOOP_MNT >$seqres.full
@@ -48,6 +48,10 @@ if [ $? -ne 0 ];then
 	echo "xfs_growfs fails!"
 fi
 
+_unmount $LOOP_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 echo "Silence is golden"
 # success, all done
 status=0
tests/xfs/613 +14 -26
index 6ba3d87bf..b87cae91e 100755
@@ -15,14 +15,8 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_unmount $LOOP_MNT 2>/dev/null
-	if [ -n "$LOOP_DEV" ];then
-		_destroy_loop_device $LOOP_DEV 2>/dev/null
-	fi
-	if [ -n "$LOOP_SPARE_DEV" ];then
-		_destroy_loop_device $LOOP_SPARE_DEV 2>/dev/null
-	fi
+	[ -n $loop_dev ] &&_destroy_loop_device $loop_dev
 	rm -f $LOOP_IMG
-	rm -f $LOOP_SPARE_IMG
 	rmdir $LOOP_MNT
 }
 
@@ -38,16 +32,11 @@ _require_loop
 _require_xfs_io_command "falloc"
 
 LOOP_IMG=$TEST_DIR/$seq.dev
-LOOP_SPARE_IMG=$TEST_DIR/$seq.logdev
 LOOP_MNT=$TEST_DIR/$seq.mnt
 
 echo "** create loop device"
 $XFS_IO_PROG -f -c "truncate 32g" $LOOP_IMG
-LOOP_DEV=`_create_loop_device $LOOP_IMG`
-
-echo "** create loop log device"
-$XFS_IO_PROG -f -c "truncate 1g" $LOOP_SPARE_IMG
-LOOP_SPARE_DEV=`_create_loop_device $LOOP_SPARE_IMG`
+loop_dev=`_create_loop_device $LOOP_IMG`
 
 echo "** create loop mount point"
 rmdir $LOOP_MNT 2>/dev/null
@@ -56,8 +45,7 @@ mkdir -p $LOOP_MNT || _fail "cannot create loopback mount point"
 filter_loop()
 {
 	sed -e "s,\B$LOOP_MNT,LOOP_MNT,g" \
-	    -e "s,\B$LOOP_DEV,LOOP_DEV,g" \
-	    -e "s,\B$LOOP_SPARE_DEV,LOOP_SPARE_DEV,g"
+	    -e "s,\B$loop_dev,LOOP_DEV,g"
 }
 
 filter_xfs_opt()
@@ -70,22 +58,22 @@ MKFS_OPTIONS=""
 do_mkfs()
 {
 	echo "FORMAT: $@" | filter_loop | tee -a $seqres.full
-	$MKFS_XFS_PROG -f $* $LOOP_DEV | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
+	$MKFS_XFS_PROG -f $* $loop_dev | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
 	if [ "${PIPESTATUS[0]}" -ne 0 ]; then
-		_fail "Fails on _mkfs_dev $* $LOOP_DEV"
+		_fail "Fails on _mkfs_dev $* $loop_dev"
 	fi
 	. $tmp.mkfs
 }
 
 is_dev_mounted()
 {
-	findmnt --source $LOOP_DEV >/dev/null
+	findmnt --source $loop_dev >/dev/null
 	return $?
 }
 
 get_mount_info()
 {
-	findmnt --source $LOOP_DEV -o OPTIONS -n
+	findmnt --source $loop_dev -o OPTIONS -n
 }
 
 force_unmount()
@@ -104,29 +92,29 @@ _do_test()
 	local info
 
 	# mount test
-	_mount $LOOP_DEV $LOOP_MNT $opts 2>>$seqres.full
+	_mount $loop_dev $LOOP_MNT $opts 2>>$seqres.full
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "${mounted}" = "fail" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to fail, but it succeeded"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -ne 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs not mounted even mount return 0"
 			return 1
 		fi
 	else
 		if [ "${mounted}" = "pass" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to succeed, but it failed"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -eq 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs is mounted even mount return non-zero"
 			return 1
 		fi
@@ -142,13 +130,13 @@ _do_test()
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "$found" != "true" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expected to find \"$key\" in mount info \"$info\""
 			return 1
 		fi
 	else
 		if [ "$found" != "false" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: did not expect to find \"$key\" in \"$info\""
 			return 1
 		fi
tests/xfs/613.out +0 -1
index 1624617ee..2a693c53c 100644
@@ -1,6 +1,5 @@
 QA output created by 613
 ** create loop device
-** create loop log device
 ** create loop mount point
 ** start xfs mount testing ...
 FORMAT: -m crc=0
Signature · 1 line
2.45.2
#21 lore raw

Re: [PATCH 13/40] fstests: clean up loop device instantiation

Quoted 237 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Lots of tests do there own special thing with loop devices rather than using _create_loop_device() and _destroy_loop_device(). This oftens means they do not clean up after themselves properly, leaving stale loop devices around that result in unmountable test or scratch devices. This is common when tests are killed by user interrupt.

Even the tests that do use _destroy_loop_device and try to clean up often do it incorrectly, leading to spurious error messages.

Some tests try to use dynamic instantiation via "mount -o loop", but then don't clean up in the correct order or hack around to find the loop device that was instantiated because the test needs to know the instantiated device name

Clean this up by converting all the tests to use _create_loop_device() and _destroy_loop_device(). In all the tests, use the variable "loop_dev" for the device consistently. In _destroy_loop_device(), test that a device name has been passed so that we don't try to clean up the same device twice (e.g. once before test exit and again from the _cleanup() function). When we destroy a loop device, unset the variable used to hold the loop device name so that we don't try to destroy it twice.

This results in much more reliable cleanup and clean exit from fstests when killed by the user.

---
 common/metadump   | 32 ++++++++++++--------------------
 tests/generic/067 | 11 ++++++++---
 tests/generic/361 |  6 +++++-
 tests/generic/563 | 14 +++++++++-----
 tests/generic/564 | 12 ++++++------
 tests/generic/590 |  9 ++++++---
 tests/generic/744 |  4 ++--
 tests/generic/746 |  6 +++++-
 tests/xfs/014     |  7 +++++--
 tests/xfs/049     | 35 +++++++++++++++++++++--------------
 tests/xfs/073     | 24 +++++++++++++-----------
 tests/xfs/074     | 21 ++++++++++++---------
 tests/xfs/078     | 16 +++++++++-------
 tests/xfs/148     | 23 ++++++++++++++---------
 tests/xfs/149     |  4 ++++
 tests/xfs/216     | 15 ++++++++++-----
 tests/xfs/217     | 22 ++++++++++++++++------
 tests/xfs/250     | 14 +++++++++-----
 tests/xfs/259     | 13 ++++++++-----
 tests/xfs/513     | 44 ++++++++++++++++++++------------------------
 tests/xfs/521     |  8 ++++++--
 tests/xfs/528     | 10 +++++++---
 tests/xfs/530     | 11 ++++++-----
 tests/xfs/606     | 12 ++++++++----
 tests/xfs/613     | 40 ++++++++++++++--------------------------
 tests/xfs/613.out |  1 -
 26 files changed, 235 insertions(+), 179 deletions(-)
common/metadump +12 -20
index 3373edfe9..bc3038e3b 100644
@@ -24,17 +24,9 @@ _xfs_cleanup_verify_metadump()
 
 	test -n "$XFS_METADUMP_FILE" && rm -f "$XFS_METADUMP_FILE"
 
-	if [ -n "$XFS_METADUMP_IMG" ]; then
-		losetup -n -a -O BACK-FILE,NAME | grep "^$XFS_METADUMP_IMG" | while read backing ldev; do
-			losetup -d "$ldev"
-		done
-
-		# Don't call rm directly with a globbed argument here to avoid
-		# issues issues with variable expansions.
-		for img in "$XFS_METADUMP_IMG"*; do
-			test -e "$img" && rm -f "$img"
-		done
-	fi
+	[ -n "$md_data_loop_dev" ] && _destroy_loop_device $md_data_loop_dev
+	[ -n "$md_log_loop_dev" ] && _destroy_loop_device $md_log_loop_dev
+	rm -f $data_img $log_img
 }
 
 # Can xfs_metadump snapshot the fs metadata to a v1 metadump file?
@@ -122,25 +114,25 @@ _xfs_verify_metadump_v2()
 		_scratch_xfs_mdrestore $metadump_file
 
 	# Create loopdev for data device so we can mount the fs
-	data_loop=$(_create_loop_device $data_img)
+	md_data_loop_dev=$(_create_loop_device $data_img)
 
 	# Create loopdev for log device if we recovered anything
-	test -s "$log_img" && log_loop=$(_create_loop_device $log_img)
+	test -s "$log_img" && md_log_loop_dev=$(_create_loop_device $log_img)
 
 	# Mount fs, run an extra test, fsck, and unmount
-	SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop _scratch_mount
+	SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev _scratch_mount
 	if [ -n "$extra_test" ]; then
-		SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop $extra_test
+		SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev $extra_test
 	fi
-	SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop _check_xfs_scratch_fs
-	SCRATCH_DEV=$data_loop _scratch_unmount
+	SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev _check_xfs_scratch_fs
+	_unmount $md_data_loop_dev
 
 	# Tear down what we created
-	if [ -b "$log_loop" ]; then
-		_destroy_loop_device $log_loop
+	if [ -b "$md_log_loop_dev" ]; then
+		_destroy_loop_device $md_log_loop_dev
 		rm -f $log_img
 	fi
-	_destroy_loop_device $data_loop
+	_destroy_loop_device $md_data_loop_dev
 	rm -f $data_img
 }
 
tests/generic/067 +8 -3
index ccb1e3fbb..b45ae834f 100755
@@ -37,13 +37,18 @@ mount_nonexistent_mnt()
 	$MOUNT_PROG $SCRATCH_DEV $TEST_DIR/nosuchdir >>$seqres.full 2>&1
 }
 
-# fs driver should be able to handle mounting a free loop device gracefully
-# xfs ever hung, "ec53d1d xfs: don't block on buffer read errors" fixed it
+# fs driver should be able to handle mounting a free loop device gracefully xfs
+# ever hung, "ec53d1d xfs: don't block on buffer read errors" fixed it
+#
+# Using 'losetup -f' like this is racy. We might end up mounting a real loop dev
+# here, so unmount $SCRATCH_MNT (not the loop dev we might not own!) and ignore
+# any error it might return.
 mount_free_loopdev()
 {
 	echo "# mount a free loop device" >>$seqres.full
 	loopdev=`losetup -f`
-	$MOUNT_PROG -t $FSTYP $loopdev $SCRATCH_MNT >>$seqres.full 2>&1
+	_mount $loopdev $SCRATCH_MNT >>$seqres.full 2>&1
+	_unmount $SCRATCH_MNT >> /dev/null 2>&1
 }
 
 # mount with wrong fs type specified.
tests/generic/361 +5 -1
index 7273dd056..e2b798436 100755
@@ -17,7 +17,7 @@ _begin_fstest auto quick
 _cleanup()
 {
 	_unmount $fs_mnt
-	_destroy_loop_device $loop_dev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
 }
@@ -54,6 +54,10 @@ $XFS_IO_PROG -fc "pwrite 0 520m" $fs_mnt/testfile >>$seqres.full 2>&1
 # remount should not hang
 $MOUNT_PROG -o remount,ro $fs_mnt >>$seqres.full 2>&1
 
+_unmount $fs_mnt
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 echo "Silence is golden"
 status=0
tests/generic/563 +9 -5
index f0d2f404c..34d21170c 100755
@@ -87,12 +87,12 @@ reset()
 
 # cgroup I/O accounting doesn't work on partitions. Use a loop device to rule
 # that out.
-LOOP_DEV=$(_create_loop_device $SCRATCH_DEV)
-smajor=$((0x`stat -L -c %t $LOOP_DEV`))
-sminor=$((0x`stat -L -c %T $LOOP_DEV`))
+loop_dev=$(_create_loop_device $SCRATCH_DEV)
+smajor=$((0x`stat -L -c %t $loop_dev`))
+sminor=$((0x`stat -L -c %T $loop_dev`))
 
-_mkfs_dev $LOOP_DEV >> $seqres.full 2>&1
-_mount $LOOP_DEV $SCRATCH_MNT || _fail "mount failed"
+_mkfs_dev $loop_dev >> $seqres.full 2>&1
+_mount $loop_dev $SCRATCH_MNT || _fail "mount failed"
 
 blksize=$(_get_block_size "$SCRATCH_MNT")
 
@@ -147,6 +147,10 @@ if [ "$drop_io_cgroup" = 1 ]; then
 	echo "-io" > $cgdir/cgroup.subtree_control || _fail "subtree control"
 fi
 
+_unmount $SCRATCH_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/generic/564 +6 -6
index 647472d78..b66b72496 100755
@@ -19,7 +19,7 @@ _cleanup()
 {
 	cd /
 	rm -rf $tmp.*
-	[ -z "$loopdev" ] || _destroy_loop_device $loopdev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 }
 
 # Import common functions.
@@ -72,11 +72,11 @@ $XFS_IO_PROG -f -c "copy_range -l 32k $testdir" $testdir/copy
 echo
 echo source/destination as blkdev returns EINVAL
 $XFS_IO_PROG -f -c "truncate 128k" $testdir/img >> $seqres.full 2>&1
-loopdev=`_create_loop_device $testdir/img`
-$XFS_IO_PROG -c "copy_range -l 32k $testdir/file" $loopdev
-$XFS_IO_PROG -f -c "copy_range -l 32k $loopdev" $testdir/copy
-_destroy_loop_device $loopdev
-loopdev=
+loop_dev=`_create_loop_device $testdir/img`
+$XFS_IO_PROG -c "copy_range -l 32k $testdir/file" $loop_dev
+$XFS_IO_PROG -f -c "copy_range -l 32k $loop_dev" $testdir/copy
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo
 echo source/destination as chardev returns EINVAL
tests/generic/590 +0 -0
index 2b7ccfb53..2eefa2c62 100755

tests/generic/590 _begin_fstest auto prealloc preallocrw · line 19

Quoted 5 lines
tests/generic/590 +2 -0
@@ -15,9 +15,10 @@ _begin_fstest auto prealloc preallocrw
 # Override the default cleanup function.
 _cleanup()
 {
+	_scratch_unmount
+	[ -n $loop_dev ] && _destroy_loop_device $loop_dev

The $loop_dev need quotation marks, or the _destroy_loop_device always be run. Same issue on xfs/513 and xfs/613. I'll add "..." to them.

Thanks,
Zorro

Quoted 1,072 lines
tests/generic/590 +4 -3
 	cd /
 	rm -f $tmp.*
-	test -n "$loop" && _destroy_loop_device "$loop"
 	rm -f "$TEST_DIR/$seq"
 }
 
@@ -56,9 +57,9 @@ if [[ $FSTYP = xfs ]]; then
 		loopsz="$((filesz + (1 << 26)))"
 		_require_fs_space "$TEST_DIR" $((loopsz / 1024))
 		$XFS_IO_PROG -c "truncate $loopsz" -f "$TEST_DIR/$seq"
-		loop="$(_create_loop_device "$TEST_DIR/$seq")"
+		loop_dev="$(_create_loop_device "$TEST_DIR/$seq")"
 		USE_EXTERNAL=yes
-		SCRATCH_RTDEV="$loop"
+		SCRATCH_RTDEV="$loop_dev"
 		disabled_features=()
 
 		# disable reflink if not supported by realtime devices
@@ -114,6 +115,8 @@ $XFS_IO_PROG -c "truncate 0" -c fsync "$SCRATCH_MNT/file"
 # We need to do this before the loop device gets torn down.
 _scratch_unmount
 _check_scratch_fs
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo "Silence is golden"
 status=0
tests/generic/744 +2 -2
index df8f6ae9b..cda10e0f6 100755
@@ -17,8 +17,8 @@ _cleanup()
 
 	_unmount $mnt2 &> /dev/null
 	_unmount $mnt1 &> /dev/null
-	[ -b "$loop_dev2" ] && losetup -d $loop_dev2
-	[ -b "$loop_dev1" ] && losetup -d $loop_dev1
+	[ -b "$loop_dev2" ] && _destroy_loop_device $loop_dev2
+	[ -b "$loop_dev1" ] && _destroy_loop_device $loop_dev1
 	[ -n "$seq" ] && rm -rf $TEST_DIR/$seq
 }
 
tests/generic/746 +5 -1
index 671910606..ba8ed25e8 100755
@@ -39,7 +39,7 @@ esac
 _cleanup()
 {
 	_unmount $loop_mnt &> /dev/null
-	_destroy_loop_device $loop_dev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	if [ $status -eq 0 ]; then
 		rm -rf $tmp
 		rm $img_file
@@ -223,5 +223,9 @@ while read line; do
 done < $fiemap_after
 echo "done."
 
+_unmount $loop_mnt
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 status=0
 exit
tests/xfs/014 +5 -2
index f638e4b16..39ea40e2a 100755
@@ -23,6 +23,7 @@ _cleanup()
 {
 	cd /
 	_unmount $LOOP_MNT 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	_scratch_unmount 2>/dev/null
 	rm -f $tmp.*
 }
@@ -166,19 +167,21 @@ LOOP_FILE=$SCRATCH_MNT/$seq.fs
 LOOP_MNT=$SCRATCH_MNT/$seq.mnt
 
 $MKFS_XFS_PROG -d "file=1,name=$LOOP_FILE,size=10g" >> $seqres.full 2>&1
+loop_dev=$(_create_loop_device $LOOP_FILE)
 
 mkdir -p $LOOP_MNT
-_mount -o uquota,gquota $LOOP_FILE $LOOP_MNT || \
+_mount -o uquota,gquota $loop_dev $LOOP_MNT || \
 	_fail "Failed to mount loop fs."
 
 _test_enospc $LOOP_MNT
 _test_edquot $LOOP_MNT
 
 _unmount $LOOP_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo $orig_sp_time > /proc/sys/fs/xfs/speculative_prealloc_lifetime
 
 _scratch_unmount
-
 status=0
 exit
tests/xfs/049 +21 -14
index 4163a144f..cdcddf764 100755
@@ -12,16 +12,17 @@ _begin_fstest rw auto quick
 # Override the default cleanup function.
 _cleanup()
 {
-    cd /
-    umount $SCRATCH_MNT/test2 > /dev/null 2>&1
-    umount $SCRATCH_MNT/test > /dev/null 2>&1
-    rm -f $tmp.*
+	cd /
+	_unmount $SCRATCH_MNT/test2 > /dev/null 2>&1
+	_unmount $SCRATCH_MNT/test > /dev/null 2>&1
+	[ -n "$loop_dev2" ] && _destroy_loop_device $loop_dev2
+	[ -n "$loop_dev1" ] && _destroy_loop_device $loop_dev1
+	rm -f $tmp.*
 
-    if [ -w $seqres.full ]
-    then
-        echo "--- mounts at end (after cleanup)" >> $seqres.full
-        mount >> $seqres.full
-    fi
+	if [ -w $seqres.full ]; then
+		echo "--- mounts at end (after cleanup)" >> $seqres.full
+		mount >> $seqres.full
+	fi
 }
 
 # Import common functions.
@@ -64,7 +65,8 @@ mkdir $SCRATCH_MNT/test $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to make mount points"
 
 _log "Mount xfs via loop"
-mount -t xfs -o loop $SCRATCH_MNT/test.xfs $SCRATCH_MNT/test >> $seqres.full 2>&1 \
+loop_dev1=$(_create_loop_device $SCRATCH_MNT/test.xfs)
+_mount $loop_dev1 $SCRATCH_MNT/test >> $seqres.full 2>&1 \
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress"
@@ -80,11 +82,12 @@ dd if=/dev/zero of=$SCRATCH_MNT/test/test.ext2 bs=1024 count=10240 >> $seqres.fu
     || _fail "!!! create file failed"
 
 _log "Create ext2 fs in file on looped xfs"
-echo y | mkfs -t ext2 $SCRATCH_MNT/test/test.ext2 >> $seqres.full 2>&1 \
+loop_dev2=$(_create_loop_device $SCRATCH_MNT/test/test.ext2)
+echo y | mkfs -t ext2 $loop_dev2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to mkfs ext2 on xfs"
 
 _log "Mount ext2 on xfs via loop"
-mount -t ext2 -o loop $SCRATCH_MNT/test/test.ext2 $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
+_mount $loop_dev2 $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress ext2 on xfs via loop"
@@ -96,12 +99,16 @@ rm -rf $SCRATCH_MNT/test/* >> $seqres.full 2>&1 \
     || _fail "!!! clean failed"
 
 _log "umount ext2 on xfs"
-umount $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
+_unmount $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! umount ext2 failed"
+_destroy_loop_device $loop_dev2
+unset loop_dev2
 
 _log "umount xfs"
-umount $SCRATCH_MNT/test >> $seqres.full 2>&1 \
+_unmount $SCRATCH_MNT/test >> $seqres.full 2>&1 \
     || _fail "!!! umount xfs failed"
+_destroy_loop_device $loop_dev1
+unset loop_dev1
 
 echo "--- mounts at end (before cleanup)" >> $seqres.full
 mount >> $seqres.full
tests/xfs/073 +13 -11
index e4b17c5e7..2274079ef 100755
@@ -23,6 +23,8 @@ _cleanup()
 	_scratch_unmount 2>/dev/null
 	_unmount $imgs.loop 2>/dev/null
 	_unmount $imgs.source_dir 2>/dev/null
+	[ -n "$loop_dev1" ] && _destroy_loop_device $loop_dev1
+	[ -n "$loop_dev2" ] && _destroy_loop_device $loop_dev2
 	[ -d $imgs.loop ] && rmdir $imgs.loop
 	[ -d $imgs.source_dir ] && rm -rf $imgs.source_dir
 	rm -f $imgs.* $tmp.* /var/tmp/xfs_copy.log.*
@@ -65,10 +67,11 @@ _verify_copy()
 	rmdir $target_dir 2>/dev/null
 	mkdir $target_dir
 
-	_mount -t xfs -o loop $target $target_dir 2>/dev/null
+	loop_dev1=$(_create_loop_device $target)
+	_mount $loop_dev1 $target_dir 2>/dev/null
 	if [ $? -ne 0 ]; then
 		echo retrying mount with nouuid option >>$seqres.full
-		_mount -t xfs -o loop -o nouuid $target $target_dir
+		_mount -o nouuid $loop_dev1 $target_dir
 		if [ $? -ne 0 ]; then
 			echo mount failed - evil!
 			return
@@ -100,6 +103,8 @@ _verify_copy()
 	echo unmounting and removing new image
 	_unmount $source_dir
 	_unmount $target_dir > /dev/null 2>&1
+	_destroy_loop_device $loop_dev1
+	unset loop_dev1
 	rm -f $target
 }
 
@@ -134,18 +139,15 @@ ${MKFS_XFS_PROG} -dfile,name=$imgs.source,size=100g \
 rmdir $imgs.source_dir 2>/dev/null
 mkdir $imgs.source_dir
 
-_mount -t xfs -o loop $imgs.source $imgs.source_dir
-loop2=`mount | grep $imgs.source | grep -o -e 'loop=.*[^),]' | grep -o -e '/.*$'`
+loop_dev2=$(_create_loop_device $imgs.source)
+_mount $loop_dev2 $imgs.source_dir
 cp -a $here $imgs.source_dir
-_mount -t xfs -o remount,ro $imgs.source $imgs.source_dir
-$XFS_COPY_PROG $imgs.source $imgs.image | _filter_copy '#' $imgs.image '#' '#'
+_mount -o remount,ro $loop_dev2 $imgs.source_dir
+$XFS_COPY_PROG $loop_dev2 $imgs.image 2> /dev/null | _filter_copy '#' $imgs.image '#' '#'
 _verify_copy $imgs.image $imgs.source $imgs.source_dir
 
-# HACK WARNING:
-#
-# We're done with the nested loop mount, now we have to clean
-# up the pieces that mount is incapable of doing.
-losetup -d $loop2 > /dev/null 2>&1
+_destroy_loop_device $loop_dev2
+unset loop_dev2
 
 echo
 echo === copying scratch device to multiple targets
tests/xfs/074 +12 -9
index 6a03c8111..5df864fad 100755
@@ -26,7 +26,7 @@ _begin_fstest quick auto prealloc rw
 _cleanup()
 {
 	cd /
-	_destroy_loop_device $LOOP_DEV
+	[ -n "$loop_dev" ] &&_destroy_loop_device $loop_dev
 	rm -f $tmp.* $LOOP_FILE
 }
 
@@ -45,10 +45,10 @@ LOOP_FILE=$TEST_DIR/$seq.img
 LOOP_MNT=$TEST_DIR/$seq.mnt
 mkdir -p $LOOP_MNT
 $XFS_IO_PROG -ft -c "truncate 1t" $LOOP_FILE >> $seqres.full
-LOOP_DEV=`_create_loop_device $LOOP_FILE`
+loop_dev=`_create_loop_device $LOOP_FILE`
 
-_mkfs_dev -d size=260g,agcount=2 $LOOP_DEV
-_mount $LOOP_DEV $LOOP_MNT
+_mkfs_dev -d size=260g,agcount=2 $loop_dev
+_mount $loop_dev $LOOP_MNT
 
 BLOCK_SIZE=$(_get_file_block_size $LOOP_MNT)
 
@@ -59,11 +59,11 @@ $XFS_IO_PROG -ft \
 	-c "falloc 0 $(($BLOCK_SIZE * 2097152))" \
 	$LOOP_MNT/foo >> $seqres.full
 
-umount $LOOP_MNT
-_check_xfs_filesystem $LOOP_DEV none none
+_unmount $LOOP_MNT
+_check_xfs_filesystem $loop_dev none none
 
-_mkfs_dev -f $LOOP_DEV
-_mount $LOOP_DEV $LOOP_MNT
+_mkfs_dev -f $loop_dev
+_mount $loop_dev $LOOP_MNT
 
 # check we trim both ends of the extent approproiately; this will fail
 # on 1k block size filesystems without the correct fixes in place.
@@ -73,7 +73,10 @@ $XFS_IO_PROG -ft \
 	$LOOP_MNT/foo >> $seqres.full
 
 _unmount $LOOP_MNT
-_check_xfs_filesystem $LOOP_DEV none none
+_check_xfs_filesystem $loop_dev none none
+
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 # success, all done
 echo "Silence is golden"
tests/xfs/078 +9 -7
index 6b325e05f..0d3c2eb23 100755
@@ -17,7 +17,7 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_unmount $LOOP_MNT 2>/dev/null
-	[ -n "$LOOP_DEV" ] && _destroy_loop_device $LOOP_DEV 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev 2>/dev/null
 	# try to keep the image file if test fails
 	[ $status -eq 0 ] && rm -f $LOOP_IMG
 	rmdir $LOOP_MNT
@@ -55,7 +55,7 @@ _grow_loop()
 	agsize=$5
 
 	$XFS_IO_PROG -f -c "truncate $original" $LOOP_IMG
-	LOOP_DEV=`_create_loop_device $LOOP_IMG`
+	loop_dev=`_create_loop_device $LOOP_IMG`
 
 	dparam=""
 	if [ -n "$agsize" ]; then
@@ -67,15 +67,15 @@ _grow_loop()
 	echo
 
 	echo "*** mkfs loop file (size=$original)"
-	$MKFS_XFS_PROG -b size=$bsize $dparam $LOOP_DEV | \
+	$MKFS_XFS_PROG -b size=$bsize $dparam $loop_dev | \
 		_filter_mkfs 2>/dev/null
 
 	echo "*** extend loop file"
-	_destroy_loop_device $LOOP_DEV
+	_destroy_loop_device $loop_dev
 	$XFS_IO_PROG -c "pwrite $new_size $bsize" $LOOP_IMG | _filter_io
-	LOOP_DEV=`_create_loop_device $LOOP_IMG`
+	loop_dev=`_create_loop_device $LOOP_IMG`
 	echo "*** mount loop filesystem"
-	_mount -t xfs $LOOP_DEV $LOOP_MNT
+	_mount $loop_dev $LOOP_MNT
 
 	echo "*** grow loop filesystem"
 	$XFS_GROWFS_PROG $LOOP_MNT 2>&1 |  _filter_growfs 2>&1
@@ -87,9 +87,11 @@ _grow_loop()
 	if [ "$check" -gt "0" ]
 	then
 		echo "*** check"
-		_check_xfs_filesystem $LOOP_IMG none none
+		_check_xfs_filesystem $loop_dev none none
 	fi
 
+	_destroy_loop_device $loop_dev
+	unset loop_dev
 	rm -f $LOOP_IMG
 }
 
tests/xfs/148 +14 -9
index c42c9b119..4d2f7a808 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_unmount $mntpt > /dev/null 2>&1
-	_destroy_loop_device $loopdev > /dev/null 2>&1
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	rm -r -f $tmp.*
 }
 
@@ -48,12 +48,12 @@ rm -f $imgfile $imgfile.old
 # even when security xattrs are present so we are always doing name matches on
 # lookup and not name hash compares as leaf/node forms will do.
 $XFS_IO_PROG -f -c 'truncate 40m' $imgfile
-loopdev=$(_create_loop_device $imgfile)
-MKFS_OPTIONS="-m crc=0 -i size=512" _mkfs_dev $loopdev >> $seqres.full
+loop_dev=$(_create_loop_device $imgfile)
+MKFS_OPTIONS="-m crc=0 -i size=512" _mkfs_dev $loop_dev >> $seqres.full
 
 # Mount image file
 mkdir -p $mntpt
-_mount $loopdev $mntpt
+_mount $loop_dev $mntpt
 
 echo "creating entries" >> $seqres.full
 
@@ -91,7 +91,8 @@ cat $tmp.log | _filter_test_dir
 
 # Corrupt the entries
 _unmount $mntpt
-_destroy_loop_device $loopdev
+_destroy_loop_device $loop_dev
+unset loop_dev
 cp $imgfile $imgfile.old
 sed -b \
 	-e "s/$nullstr/too_many\x00beans/g" \
@@ -99,8 +100,9 @@ sed -b \
 	-i $imgfile
 test "$(md5sum < $imgfile)" != "$(md5sum < $imgfile.old)" ||
 	_fail "sed failed to change the image file?"
-loopdev=$(_create_loop_device $imgfile)
-_mount $loopdev $mntpt
+
+loop_dev=$(_create_loop_device $imgfile)
+_mount $loop_dev $mntpt
 
 # Try to access the corrupt metadata
 echo "++ ACCESSING BAD METADATA" | tee -a $seqres.full
@@ -111,7 +113,7 @@ cat $tmp.log | _filter_test_dir | sed -e '/Could not list/d'
 echo "does scrub complain?" >> $seqres.full
 
 # Does scrub complain about this?
-if _supports_xfs_scrub $mntpt $loopdev; then
+if _supports_xfs_scrub $mntpt $loop_dev; then
 	$XFS_SCRUB_PROG -n $mntpt >> $seqres.full 2>&1
 	res=$?
 	test $((res & 1)) -eq 0 && \
@@ -122,11 +124,14 @@ echo "does repair complain?" >> $seqres.full
 
 # Does repair complain about this?
 _unmount $mntpt
-$XFS_REPAIR_PROG -n $loopdev >> $seqres.full 2>&1
+$XFS_REPAIR_PROG -n $loop_dev >> $seqres.full 2>&1
 res=$?
 test $res -eq 1 || \
 	echo "repair failed to report corruption ($res)"
 
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/149 +4 -0
index f2187109b..9a96f82ed 100755
@@ -84,6 +84,10 @@ $XFS_GROWFS_PROG -D 16384 $loop_symlink > /dev/null
 echo "=== xfs_growfs - check device node ==="
 $XFS_GROWFS_PROG -D 20480 $loop_dev > /dev/null
 
+_unmount $mntdir
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/216 +10 -5
index 6b8b2eb22..091c11d08 100755
@@ -15,6 +15,7 @@ _begin_fstest log metadata auto quick
 _cleanup()
 {
 	_unmount $LOOP_MNT > /dev/null 2>&1
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
 }
@@ -24,7 +25,7 @@ _scratch_mkfs_xfs >/dev/null 2>&1
 _scratch_mount
 
 _require_loop
-LOOP_DEV=$SCRATCH_MNT/test_fs
+LOOP_IMG=$SCRATCH_MNT/test_fs
 LOOP_MNT=$SCRATCH_MNT/test_fs_dir
 
 loop_mkfs_opts=
@@ -55,22 +56,26 @@ _do_mkfs()
 	for i in $*; do
 		echo -n "fssize=${i}g "
 		$MKFS_XFS_PROG -f -b size=4096 -l version=2 \
-			-d name=$LOOP_DEV,size=${i}g $loop_mkfs_opts |grep log
-		_mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
+			-d size=${i}g $loop_mkfs_opts $loop_dev |grep log
+		_mount $loop_dev $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
 		_unmount $LOOP_MNT > /dev/null 2>&1
 	done
 }
 # make large holey file
-$XFS_IO_PROG -f -c "truncate 256g" $LOOP_DEV
+$XFS_IO_PROG -f -c "truncate 256g" $LOOP_IMG
 
-choose_golden_output $0 $LOOP_DEV
+choose_golden_output $0 $LOOP_IMG
 
 #make loopback mount dir
 mkdir $LOOP_MNT
 
+loop_dev=$(_create_loop_device $LOOP_IMG)
+
 # walk over standard sizes (up to 256GB)
 _do_mkfs 1 2 4 8 16 32 64 128 256
 
+_destroy_loop_device $loop_dev
+unset loop_dev
 status=0
 exit
tests/xfs/217 +16 -6
index b2eb34490..dae6ce55f 100755
@@ -12,6 +12,12 @@ _begin_fstest log metadata auto
 # Import common functions.
 . ./common/filter
 
+_cleanup()
+{
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
+	cd /
+	rm -f $tmp.*
+}
 
 _require_scratch
 _scratch_mkfs_xfs >/dev/null 2>&1
@@ -20,7 +26,7 @@ _scratch_mount
 _require_fs_space $SCRATCH_MNT 2202000
 
 _require_loop
-LOOP_DEV=$SCRATCH_MNT/test_fs
+LOOP_IMG=$SCRATCH_MNT/test_fs
 LOOP_MNT=$SCRATCH_MNT/test_fs_dir
 
 _do_mkfs()
@@ -28,28 +34,30 @@ _do_mkfs()
 	for i in $*; do
 		echo -n "fssize=${i}g "
 		$MKFS_XFS_PROG -f -b size=4096 -l version=2 \
-			-d name=$LOOP_DEV,size=${i}g |grep log
-		_mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
+			-d size=${i}g $loop_dev |grep log
+		_mount $loop_dev $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
 		_unmount $LOOP_MNT > /dev/null 2>&1
 
 		# punch out the previous blocks so that we keep the amount of
 		# disk space the test requires down to a minimum.
-		$XFS_IO_PROG -f -c "unresvsp 0 16383g" $LOOP_DEV
+		$XFS_IO_PROG -f -c "unresvsp 0 16383g" $LOOP_IMG
 	done
 }
 # make large holey file
-$XFS_IO_PROG -f -c "truncate 16383g" $LOOP_DEV
+$XFS_IO_PROG -f -c "truncate 16383g" $LOOP_IMG
 
 #make loopback mount dir
 mkdir $LOOP_MNT
 
 # test if large logs are supported
-$MKFS_XFS_PROG -f -l size=256m -d name=$LOOP_DEV,size=10g > /dev/null 2>&1
+$MKFS_XFS_PROG -f -l size=256m -d name=$LOOP_IMG,size=10g > /dev/null 2>&1
 if [ $? -ne 0 ]; then
 	_notrun "large log sizes not supported by mkfs"
 fi
 
+loop_dev=$(_create_loop_device $LOOP_IMG)
+
 #
 # walk over "new" sizes supported by recent xfsprogs.
 # Note that the last test is for 16TB-1GB as 32bit platforms only support
@@ -57,5 +65,7 @@ fi
 #
 _do_mkfs 512 1024 2048 4096 8192 16383
 
+_destroy_loop_device $loop_dev
+unset loop_dev
 status=0
 exit
tests/xfs/250 +9 -5
index 4e3473ebc..2554e1e91 100755
@@ -14,7 +14,8 @@ _cleanup()
 {
 	cd /
 	_unmount $LOOP_MNT 2>/dev/null
-	rm -f $LOOP_DEV
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
+	rm -f $LOOP_IMG
 	rmdir $LOOP_MNT
 	rm -f $tmp.*
 }
@@ -26,7 +27,7 @@ _require_test
 _require_loop
 _require_xfs_io_command "falloc"
 
-LOOP_DEV=$TEST_DIR/$seq.fs
+LOOP_IMG=$TEST_DIR/$seq.fs
 LOOP_MNT=$TEST_DIR/$seq.mnt
 
 _filter_io()
@@ -45,7 +46,7 @@ _test_loop()
 	agsize=$2
 	fsize=$3
 
-	dparam="file,name=$LOOP_DEV,size=$size"
+	dparam="file,name=$LOOP_IMG,size=$size"
 	if [ -n "$agsize" ]; then
 		dparam="$dparam,agsize=$agsize"
 	fi
@@ -55,7 +56,8 @@ _test_loop()
 		| _filter_mkfs 2>/dev/null
 
 	echo "*** mount loop filesystem"
-	_mount -t xfs -o loop $LOOP_DEV $LOOP_MNT
+	loop_dev=$(_create_loop_device $LOOP_IMG)
+	mount $loop_dev $LOOP_MNT
 
 	echo "*** preallocate large file"
 	$XFS_IO_PROG -f -c "resvsp 0 $fsize" $LOOP_MNT/foo | _filter_io
@@ -64,7 +66,9 @@ _test_loop()
 	_unmount $LOOP_MNT > /dev/null 2>&1
 
 	echo "*** check loop filesystem"
-	 _check_xfs_filesystem $LOOP_DEV none none
+	_check_xfs_filesystem $loop_dev none none
+	_destroy_loop_device $loop_dev
+	unset loop_dev
 }
 
 _test_loop 50g 16m 40G
tests/xfs/259 +8 -5
index 0c8d6eb56..c2d26381a 100755
@@ -12,7 +12,10 @@ _begin_fstest auto quick
 # Override the default cleanup function.
 _cleanup()
 {
-    rm -f "$testfile"
+	[ -n "$loop_dev" ] && _destroy_loop_device $testfile
+	rm -f "$testfile"
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -45,13 +48,13 @@ for del in $sizes_to_check; do
 		rm -f "$testfile"
 		dd if=/dev/zero "of=$testfile" bs=1 count=0 seek=$ddseek \
 			>/dev/null 2>&1 || echo "dd failed"
-		lofile=$(losetup -f)
-		losetup $lofile "$testfile"
-		$MKFS_XFS_PROG -l size=32m -b size=$bs $lofile |  _filter_mkfs \
+		loop_dev=$(_create_loop_device $testfile)
+		$MKFS_XFS_PROG -l size=32m -b size=$bs $loop_dev |  _filter_mkfs \
 			>/dev/null 2> $tmp.mkfs || echo "mkfs failed!"
 		. $tmp.mkfs
 		sync
-		losetup -d $lofile
+		_destroy_loop_device $loop_dev
+		unset loop_dev
 	done
 done
 
tests/xfs/513 +20 -24
index 5895e6e2d..0c0edc75e 100755
@@ -15,12 +15,8 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_unmount $LOOP_MNT 2>/dev/null
-	if [ -n "$LOOP_DEV" ];then
-		_destroy_loop_device $LOOP_DEV 2>/dev/null
-	fi
-	if [ -n "$LOOP_SPARE_DEV" ];then
-		_destroy_loop_device $LOOP_SPARE_DEV 2>/dev/null
-	fi
+	[ -n $loop_dev ] &&_destroy_loop_device $loop_dev
+	[ -n $loop_spare_dev ] &&_destroy_loop_device $loop_spare_dev
 	rm -f $LOOP_IMG
 	rm -f $LOOP_SPARE_IMG
 	rmdir $LOOP_MNT
@@ -42,11 +38,11 @@ LOOP_MNT=$TEST_DIR/$seq.mnt
 
 echo "** create loop device"
 $XFS_IO_PROG -f -c "truncate 32g" $LOOP_IMG
-LOOP_DEV=`_create_loop_device $LOOP_IMG`
+loop_dev=`_create_loop_device $LOOP_IMG`
 
 echo "** create loop log device"
 $XFS_IO_PROG -f -c "truncate 1g" $LOOP_SPARE_IMG
-LOOP_SPARE_DEV=`_create_loop_device $LOOP_SPARE_IMG`
+loop_spare_dev=`_create_loop_device $LOOP_SPARE_IMG`
 
 echo "** create loop mount point"
 rmdir $LOOP_MNT 2>/dev/null
@@ -55,8 +51,8 @@ mkdir -p $LOOP_MNT || _fail "cannot create loopback mount point"
 filter_loop()
 {
 	sed -e "s,\B$LOOP_MNT,LOOP_MNT,g" \
-	    -e "s,\B$LOOP_DEV,LOOP_DEV,g" \
-	    -e "s,\B$LOOP_SPARE_DEV,LOOP_SPARE_DEV,g"
+	    -e "s,\B$loop_dev,LOOP_DEV,g" \
+	    -e "s,\B$loop_spare_dev,LOOP_SPARE_DEV,g"
 }
 
 filter_xfs_opt()
@@ -69,22 +65,22 @@ MKFS_OPTIONS=""
 do_mkfs()
 {
 	echo "FORMAT: $@" | filter_loop | tee -a $seqres.full
-	$MKFS_XFS_PROG -f $* $LOOP_DEV | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
+	$MKFS_XFS_PROG -f $* $loop_dev | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
 	if [ "${PIPESTATUS[0]}" -ne 0 ]; then
-		_fail "Fails on _mkfs_dev $* $LOOP_DEV"
+		_fail "Fails on _mkfs_dev $* $loop_dev"
 	fi
 	. $tmp.mkfs
 }
 
 is_dev_mounted()
 {
-	findmnt --source $LOOP_DEV >/dev/null
+	findmnt --source $loop_dev >/dev/null
 	return $?
 }
 
 get_mount_info()
 {
-	findmnt --source $LOOP_DEV -o OPTIONS -n
+	findmnt --source $loop_dev -o OPTIONS -n
 }
 
 force_unmount()
@@ -103,29 +99,29 @@ _do_test()
 	local info
 
 	# mount test
-	_mount $LOOP_DEV $LOOP_MNT $opts 2>>$seqres.full
+	_mount $loop_dev $LOOP_MNT $opts 2>>$seqres.full
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "${mounted}" = "fail" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to fail, but it succeeded"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -ne 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs not mounted even mount return 0"
 			return 1
 		fi
 	else
 		if [ "${mounted}" = "pass" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to succeed, but it failed"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -eq 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs is mounted even mount return non-zero"
 			return 1
 		fi
@@ -141,13 +137,13 @@ _do_test()
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "$found" != "true" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expected to find \"$key\" in mount info \"$info\""
 			return 1
 		fi
 	else
 		if [ "$found" != "false" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: did not expect to find \"$key\" in \"$info\""
 			return 1
 		fi
@@ -253,9 +249,9 @@ do_test "-o logbsize=512k" fail
 # Test logdev
 do_mkfs
 do_test "" pass "logdev" "false"
-do_test "-o logdev=$LOOP_SPARE_DEV" fail
-do_mkfs -l logdev=$LOOP_SPARE_DEV
-do_test "-o logdev=$LOOP_SPARE_DEV" pass "logdev=$LOOP_SPARE_DEV" "true"
+do_test "-o logdev=$loop_spare_dev" fail
+do_mkfs -l logdev=$loop_spare_dev
+do_test "-o logdev=$loop_spare_dev" pass "logdev=$loop_spare_dev" "true"
 do_test "" fail
 
 # Test noalign
tests/xfs/521 +6 -2
index 13982c440..c92c621a2 100755
@@ -21,7 +21,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -35,7 +35,7 @@ _require_no_large_scratch_dev
 
 echo "Create fake rt volume"
 truncate -s 400m $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Format and mount 100m rt volume"
 export USE_EXTERNAL=yes
@@ -69,6 +69,10 @@ cp -p $testdir/original $testdir/copy3
 echo "Check filesystem"
 _check_scratch_fs
 
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/528 +7 -3
index 6ca9a2370..a1efbbd27 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -155,11 +155,11 @@ test_ops() {
 
 echo "Create fake rt volume"
 $XFS_IO_PROG -f -c "truncate 400m" $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Make sure synth rt volume works"
 export USE_EXTERNAL=yes
-export SCRATCH_RTDEV=$rtdev
+export SCRATCH_RTDEV=$rt_loop_dev
 _scratch_mkfs > $seqres.full
 _try_scratch_mount || \
 	_notrun "Could not mount with synthetic rt volume"
@@ -170,6 +170,10 @@ test_ops 262144
 # not a power of two
 test_ops 327680
 
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/530 +6 -5
index 8a182bd6a..d0d0e2665 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -52,12 +52,12 @@ fi
 
 rtdevsz=$((nr_bits * rtextsz))
 truncate -s $rtdevsz $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Format and mount rt volume"
 
 export USE_EXTERNAL=yes
-export SCRATCH_RTDEV=$rtdev
+export SCRATCH_RTDEV=$rt_loop_dev
 _scratch_mkfs -d size=$((1024 * 1024 * 1024)) \
 	      -r size=${rtextsz},extsize=${rtextsz} >> $seqres.full
 _try_scratch_mount || _notrun "Couldn't mount fs with synthetic rt volume"
@@ -116,8 +116,9 @@ done
 echo "Check filesystem"
 _check_scratch_fs
 
-losetup -d $rtdev
-rm -f $TEST_DIR/$seq.rtvol
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
 
 # success, all done
 status=0
tests/xfs/606 +8 -4
index f958bddd8..b537ea145 100755
@@ -13,10 +13,8 @@ _begin_fstest auto quick growfs
 
 _cleanup()
 {
-	local dev
 	_unmount $LOOP_MNT 2>/dev/null
-	dev=$(losetup -j testfile | cut -d: -f1)
-	losetup -d $dev 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	rm -rf $LOOP_IMG $LOOP_MNT
 	cd /
 	rm -f $tmp.*
@@ -40,7 +38,9 @@ $MKFS_XFS_PROG -f $LOOP_IMG >$seqres.full
 # Extend by just 8K, expected to start with the last full-size AG ends of
 # above 1G block device.
 $XFS_IO_PROG -f -c "truncate 1073750016" $LOOP_IMG
-_mount -oloop $LOOP_IMG $LOOP_MNT
+
+loop_dev=$(_create_loop_device $LOOP_IMG)
+_mount $loop_dev $LOOP_MNT
 # A known bug shows "XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on
 # device" at here, refer to _fixed_by_kernel_commit above
 $XFS_GROWFS_PROG $LOOP_MNT >$seqres.full
@@ -48,6 +48,10 @@ if [ $? -ne 0 ];then
 	echo "xfs_growfs fails!"
 fi
 
+_unmount $LOOP_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 echo "Silence is golden"
 # success, all done
 status=0
tests/xfs/613 +14 -26
index 6ba3d87bf..b87cae91e 100755
@@ -15,14 +15,8 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_unmount $LOOP_MNT 2>/dev/null
-	if [ -n "$LOOP_DEV" ];then
-		_destroy_loop_device $LOOP_DEV 2>/dev/null
-	fi
-	if [ -n "$LOOP_SPARE_DEV" ];then
-		_destroy_loop_device $LOOP_SPARE_DEV 2>/dev/null
-	fi
+	[ -n $loop_dev ] &&_destroy_loop_device $loop_dev
 	rm -f $LOOP_IMG
-	rm -f $LOOP_SPARE_IMG
 	rmdir $LOOP_MNT
 }
 
@@ -38,16 +32,11 @@ _require_loop
 _require_xfs_io_command "falloc"
 
 LOOP_IMG=$TEST_DIR/$seq.dev
-LOOP_SPARE_IMG=$TEST_DIR/$seq.logdev
 LOOP_MNT=$TEST_DIR/$seq.mnt
 
 echo "** create loop device"
 $XFS_IO_PROG -f -c "truncate 32g" $LOOP_IMG
-LOOP_DEV=`_create_loop_device $LOOP_IMG`
-
-echo "** create loop log device"
-$XFS_IO_PROG -f -c "truncate 1g" $LOOP_SPARE_IMG
-LOOP_SPARE_DEV=`_create_loop_device $LOOP_SPARE_IMG`
+loop_dev=`_create_loop_device $LOOP_IMG`
 
 echo "** create loop mount point"
 rmdir $LOOP_MNT 2>/dev/null
@@ -56,8 +45,7 @@ mkdir -p $LOOP_MNT || _fail "cannot create loopback mount point"
 filter_loop()
 {
 	sed -e "s,\B$LOOP_MNT,LOOP_MNT,g" \
-	    -e "s,\B$LOOP_DEV,LOOP_DEV,g" \
-	    -e "s,\B$LOOP_SPARE_DEV,LOOP_SPARE_DEV,g"
+	    -e "s,\B$loop_dev,LOOP_DEV,g"
 }
 
 filter_xfs_opt()
@@ -70,22 +58,22 @@ MKFS_OPTIONS=""
 do_mkfs()
 {
 	echo "FORMAT: $@" | filter_loop | tee -a $seqres.full
-	$MKFS_XFS_PROG -f $* $LOOP_DEV | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
+	$MKFS_XFS_PROG -f $* $loop_dev | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
 	if [ "${PIPESTATUS[0]}" -ne 0 ]; then
-		_fail "Fails on _mkfs_dev $* $LOOP_DEV"
+		_fail "Fails on _mkfs_dev $* $loop_dev"
 	fi
 	. $tmp.mkfs
 }
 
 is_dev_mounted()
 {
-	findmnt --source $LOOP_DEV >/dev/null
+	findmnt --source $loop_dev >/dev/null
 	return $?
 }
 
 get_mount_info()
 {
-	findmnt --source $LOOP_DEV -o OPTIONS -n
+	findmnt --source $loop_dev -o OPTIONS -n
 }
 
 force_unmount()
@@ -104,29 +92,29 @@ _do_test()
 	local info
 
 	# mount test
-	_mount $LOOP_DEV $LOOP_MNT $opts 2>>$seqres.full
+	_mount $loop_dev $LOOP_MNT $opts 2>>$seqres.full
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "${mounted}" = "fail" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to fail, but it succeeded"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -ne 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs not mounted even mount return 0"
 			return 1
 		fi
 	else
 		if [ "${mounted}" = "pass" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to succeed, but it failed"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -eq 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs is mounted even mount return non-zero"
 			return 1
 		fi
@@ -142,13 +130,13 @@ _do_test()
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "$found" != "true" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expected to find \"$key\" in mount info \"$info\""
 			return 1
 		fi
 	else
 		if [ "$found" != "false" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: did not expect to find \"$key\" in \"$info\""
 			return 1
 		fi
tests/xfs/613.out +0 -1
index 1624617ee..2a693c53c 100644
@@ -1,6 +1,5 @@
 QA output created by 613
 ** create loop device
-** create loop log device
 ** create loop mount point
 ** start xfs mount testing ...
 FORMAT: -m crc=0
Signature · 1 line
2.45.2
#22 lore raw

Re: [PATCH 13/40] fstests: clean up loop device instantiation

In reply to Dave Chinner
Quoted 174 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Lots of tests do there own special thing with loop devices rather than using _create_loop_device() and _destroy_loop_device(). This oftens means they do not clean up after themselves properly, leaving stale loop devices around that result in unmountable test or scratch devices. This is common when tests are killed by user interrupt.

Even the tests that do use _destroy_loop_device and try to clean up often do it incorrectly, leading to spurious error messages.

Some tests try to use dynamic instantiation via "mount -o loop", but then don't clean up in the correct order or hack around to find the loop device that was instantiated because the test needs to know the instantiated device name

Clean this up by converting all the tests to use _create_loop_device() and _destroy_loop_device(). In all the tests, use the variable "loop_dev" for the device consistently. In _destroy_loop_device(), test that a device name has been passed so that we don't try to clean up the same device twice (e.g. once before test exit and again from the _cleanup() function). When we destroy a loop device, unset the variable used to hold the loop device name so that we don't try to destroy it twice.

This results in much more reliable cleanup and clean exit from fstests when killed by the user.

---
 common/metadump   | 32 ++++++++++++--------------------
 tests/generic/067 | 11 ++++++++---
 tests/generic/361 |  6 +++++-
 tests/generic/563 | 14 +++++++++-----
 tests/generic/564 | 12 ++++++------
 tests/generic/590 |  9 ++++++---
 tests/generic/744 |  4 ++--
 tests/generic/746 |  6 +++++-
 tests/xfs/014     |  7 +++++--
 tests/xfs/049     | 35 +++++++++++++++++++++--------------
 tests/xfs/073     | 24 +++++++++++++-----------
 tests/xfs/074     | 21 ++++++++++++---------
 tests/xfs/078     | 16 +++++++++-------
 tests/xfs/148     | 23 ++++++++++++++---------
 tests/xfs/149     |  4 ++++
 tests/xfs/216     | 15 ++++++++++-----
 tests/xfs/217     | 22 ++++++++++++++++------
 tests/xfs/250     | 14 +++++++++-----
 tests/xfs/259     | 13 ++++++++-----
 tests/xfs/513     | 44 ++++++++++++++++++++------------------------
 tests/xfs/521     |  8 ++++++--
 tests/xfs/528     | 10 +++++++---
 tests/xfs/530     | 11 ++++++-----
 tests/xfs/606     | 12 ++++++++----
 tests/xfs/613     | 40 ++++++++++++++--------------------------
 tests/xfs/613.out |  1 -
 26 files changed, 235 insertions(+), 179 deletions(-)
common/metadump +12 -20
index 3373edfe9..bc3038e3b 100644
@@ -24,17 +24,9 @@ _xfs_cleanup_verify_metadump()
 
 	test -n "$XFS_METADUMP_FILE" && rm -f "$XFS_METADUMP_FILE"
 
-	if [ -n "$XFS_METADUMP_IMG" ]; then
-		losetup -n -a -O BACK-FILE,NAME | grep "^$XFS_METADUMP_IMG" | while read backing ldev; do
-			losetup -d "$ldev"
-		done
-
-		# Don't call rm directly with a globbed argument here to avoid
-		# issues issues with variable expansions.
-		for img in "$XFS_METADUMP_IMG"*; do
-			test -e "$img" && rm -f "$img"
-		done
-	fi
+	[ -n "$md_data_loop_dev" ] && _destroy_loop_device $md_data_loop_dev
+	[ -n "$md_log_loop_dev" ] && _destroy_loop_device $md_log_loop_dev
+	rm -f $data_img $log_img
 }
 
 # Can xfs_metadump snapshot the fs metadata to a v1 metadump file?
@@ -122,25 +114,25 @@ _xfs_verify_metadump_v2()
 		_scratch_xfs_mdrestore $metadump_file
 
 	# Create loopdev for data device so we can mount the fs
-	data_loop=$(_create_loop_device $data_img)
+	md_data_loop_dev=$(_create_loop_device $data_img)
 
 	# Create loopdev for log device if we recovered anything
-	test -s "$log_img" && log_loop=$(_create_loop_device $log_img)
+	test -s "$log_img" && md_log_loop_dev=$(_create_loop_device $log_img)
 
 	# Mount fs, run an extra test, fsck, and unmount
-	SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop _scratch_mount
+	SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev _scratch_mount
 	if [ -n "$extra_test" ]; then
-		SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop $extra_test
+		SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev $extra_test
 	fi
-	SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop _check_xfs_scratch_fs
-	SCRATCH_DEV=$data_loop _scratch_unmount
+	SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev _check_xfs_scratch_fs
+	_unmount $md_data_loop_dev
 
 	# Tear down what we created
-	if [ -b "$log_loop" ]; then
-		_destroy_loop_device $log_loop
+	if [ -b "$md_log_loop_dev" ]; then
+		_destroy_loop_device $md_log_loop_dev
 		rm -f $log_img
 	fi
-	_destroy_loop_device $data_loop
+	_destroy_loop_device $md_data_loop_dev
 	rm -f $data_img
 }
 
tests/generic/067 +8 -3
index ccb1e3fbb..b45ae834f 100755
@@ -37,13 +37,18 @@ mount_nonexistent_mnt()
 	$MOUNT_PROG $SCRATCH_DEV $TEST_DIR/nosuchdir >>$seqres.full 2>&1
 }
 
-# fs driver should be able to handle mounting a free loop device gracefully
-# xfs ever hung, "ec53d1d xfs: don't block on buffer read errors" fixed it
+# fs driver should be able to handle mounting a free loop device gracefully xfs
+# ever hung, "ec53d1d xfs: don't block on buffer read errors" fixed it
+#
+# Using 'losetup -f' like this is racy. We might end up mounting a real loop dev
+# here, so unmount $SCRATCH_MNT (not the loop dev we might not own!) and ignore
+# any error it might return.
 mount_free_loopdev()
 {
 	echo "# mount a free loop device" >>$seqres.full
 	loopdev=`losetup -f`
-	$MOUNT_PROG -t $FSTYP $loopdev $SCRATCH_MNT >>$seqres.full 2>&1
+	_mount $loopdev $SCRATCH_MNT >>$seqres.full 2>&1
+	_unmount $SCRATCH_MNT >> /dev/null 2>&1
 }
 
 # mount with wrong fs type specified.
tests/generic/361 +5 -1
index 7273dd056..e2b798436 100755
@@ -17,7 +17,7 @@ _begin_fstest auto quick
 _cleanup()
 {
 	_unmount $fs_mnt
-	_destroy_loop_device $loop_dev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
 }
@@ -54,6 +54,10 @@ $XFS_IO_PROG -fc "pwrite 0 520m" $fs_mnt/testfile >>$seqres.full 2>&1
 # remount should not hang
 $MOUNT_PROG -o remount,ro $fs_mnt >>$seqres.full 2>&1
 
+_unmount $fs_mnt
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 echo "Silence is golden"
 status=0
tests/generic/563 +0 -0
index f0d2f404c..34d21170c 100755

tests/generic/563 reset() · line 90

Quoted 7 lines
tests/generic/563 +1 -3
@@ -87,12 +87,12 @@ reset()
 
 # cgroup I/O accounting doesn't work on partitions. Use a loop device to rule
 # that out.
-LOOP_DEV=$(_create_loop_device $SCRATCH_DEV)
-smajor=$((0x`stat -L -c %t $LOOP_DEV`))
-sminor=$((0x`stat -L -c %T $LOOP_DEV`))
+loop_dev=$(_create_loop_device $SCRATCH_DEV)

This test case isn't be changed entirely, there're still some LOOP_DEV, e.g.

  _destroy_loop_device $LOOP_DEV > /dev/null 2>&1
  ...
  _mount $LOOP_DEV $SCRATCH_MNT || _fail "mount failed"

I'll change them to loop_dev.

Thanks,
Zorro

Quoted 1,133 lines
tests/generic/563 +8 -2
+smajor=$((0x`stat -L -c %t $loop_dev`))
+sminor=$((0x`stat -L -c %T $loop_dev`))
 
-_mkfs_dev $LOOP_DEV >> $seqres.full 2>&1
-_mount $LOOP_DEV $SCRATCH_MNT || _fail "mount failed"
+_mkfs_dev $loop_dev >> $seqres.full 2>&1
+_mount $loop_dev $SCRATCH_MNT || _fail "mount failed"
 
 blksize=$(_get_block_size "$SCRATCH_MNT")
 
@@ -147,6 +147,10 @@ if [ "$drop_io_cgroup" = 1 ]; then
 	echo "-io" > $cgdir/cgroup.subtree_control || _fail "subtree control"
 fi
 
+_unmount $SCRATCH_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/generic/564 +6 -6
index 647472d78..b66b72496 100755
@@ -19,7 +19,7 @@ _cleanup()
 {
 	cd /
 	rm -rf $tmp.*
-	[ -z "$loopdev" ] || _destroy_loop_device $loopdev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 }
 
 # Import common functions.
@@ -72,11 +72,11 @@ $XFS_IO_PROG -f -c "copy_range -l 32k $testdir" $testdir/copy
 echo
 echo source/destination as blkdev returns EINVAL
 $XFS_IO_PROG -f -c "truncate 128k" $testdir/img >> $seqres.full 2>&1
-loopdev=`_create_loop_device $testdir/img`
-$XFS_IO_PROG -c "copy_range -l 32k $testdir/file" $loopdev
-$XFS_IO_PROG -f -c "copy_range -l 32k $loopdev" $testdir/copy
-_destroy_loop_device $loopdev
-loopdev=
+loop_dev=`_create_loop_device $testdir/img`
+$XFS_IO_PROG -c "copy_range -l 32k $testdir/file" $loop_dev
+$XFS_IO_PROG -f -c "copy_range -l 32k $loop_dev" $testdir/copy
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo
 echo source/destination as chardev returns EINVAL
tests/generic/590 +6 -3
index 2b7ccfb53..2eefa2c62 100755
@@ -15,9 +15,10 @@ _begin_fstest auto prealloc preallocrw
 # Override the default cleanup function.
 _cleanup()
 {
+	_scratch_unmount
+	[ -n $loop_dev ] && _destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
-	test -n "$loop" && _destroy_loop_device "$loop"
 	rm -f "$TEST_DIR/$seq"
 }
 
@@ -56,9 +57,9 @@ if [[ $FSTYP = xfs ]]; then
 		loopsz="$((filesz + (1 << 26)))"
 		_require_fs_space "$TEST_DIR" $((loopsz / 1024))
 		$XFS_IO_PROG -c "truncate $loopsz" -f "$TEST_DIR/$seq"
-		loop="$(_create_loop_device "$TEST_DIR/$seq")"
+		loop_dev="$(_create_loop_device "$TEST_DIR/$seq")"
 		USE_EXTERNAL=yes
-		SCRATCH_RTDEV="$loop"
+		SCRATCH_RTDEV="$loop_dev"
 		disabled_features=()
 
 		# disable reflink if not supported by realtime devices
@@ -114,6 +115,8 @@ $XFS_IO_PROG -c "truncate 0" -c fsync "$SCRATCH_MNT/file"
 # We need to do this before the loop device gets torn down.
 _scratch_unmount
 _check_scratch_fs
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo "Silence is golden"
 status=0
tests/generic/744 +2 -2
index df8f6ae9b..cda10e0f6 100755
@@ -17,8 +17,8 @@ _cleanup()
 
 	_unmount $mnt2 &> /dev/null
 	_unmount $mnt1 &> /dev/null
-	[ -b "$loop_dev2" ] && losetup -d $loop_dev2
-	[ -b "$loop_dev1" ] && losetup -d $loop_dev1
+	[ -b "$loop_dev2" ] && _destroy_loop_device $loop_dev2
+	[ -b "$loop_dev1" ] && _destroy_loop_device $loop_dev1
 	[ -n "$seq" ] && rm -rf $TEST_DIR/$seq
 }
 
tests/generic/746 +5 -1
index 671910606..ba8ed25e8 100755
@@ -39,7 +39,7 @@ esac
 _cleanup()
 {
 	_unmount $loop_mnt &> /dev/null
-	_destroy_loop_device $loop_dev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	if [ $status -eq 0 ]; then
 		rm -rf $tmp
 		rm $img_file
@@ -223,5 +223,9 @@ while read line; do
 done < $fiemap_after
 echo "done."
 
+_unmount $loop_mnt
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 status=0
 exit
tests/xfs/014 +5 -2
index f638e4b16..39ea40e2a 100755
@@ -23,6 +23,7 @@ _cleanup()
 {
 	cd /
 	_unmount $LOOP_MNT 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	_scratch_unmount 2>/dev/null
 	rm -f $tmp.*
 }
@@ -166,19 +167,21 @@ LOOP_FILE=$SCRATCH_MNT/$seq.fs
 LOOP_MNT=$SCRATCH_MNT/$seq.mnt
 
 $MKFS_XFS_PROG -d "file=1,name=$LOOP_FILE,size=10g" >> $seqres.full 2>&1
+loop_dev=$(_create_loop_device $LOOP_FILE)
 
 mkdir -p $LOOP_MNT
-_mount -o uquota,gquota $LOOP_FILE $LOOP_MNT || \
+_mount -o uquota,gquota $loop_dev $LOOP_MNT || \
 	_fail "Failed to mount loop fs."
 
 _test_enospc $LOOP_MNT
 _test_edquot $LOOP_MNT
 
 _unmount $LOOP_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo $orig_sp_time > /proc/sys/fs/xfs/speculative_prealloc_lifetime
 
 _scratch_unmount
-
 status=0
 exit
tests/xfs/049 +21 -14
index 4163a144f..cdcddf764 100755
@@ -12,16 +12,17 @@ _begin_fstest rw auto quick
 # Override the default cleanup function.
 _cleanup()
 {
-    cd /
-    umount $SCRATCH_MNT/test2 > /dev/null 2>&1
-    umount $SCRATCH_MNT/test > /dev/null 2>&1
-    rm -f $tmp.*
+	cd /
+	_unmount $SCRATCH_MNT/test2 > /dev/null 2>&1
+	_unmount $SCRATCH_MNT/test > /dev/null 2>&1
+	[ -n "$loop_dev2" ] && _destroy_loop_device $loop_dev2
+	[ -n "$loop_dev1" ] && _destroy_loop_device $loop_dev1
+	rm -f $tmp.*
 
-    if [ -w $seqres.full ]
-    then
-        echo "--- mounts at end (after cleanup)" >> $seqres.full
-        mount >> $seqres.full
-    fi
+	if [ -w $seqres.full ]; then
+		echo "--- mounts at end (after cleanup)" >> $seqres.full
+		mount >> $seqres.full
+	fi
 }
 
 # Import common functions.
@@ -64,7 +65,8 @@ mkdir $SCRATCH_MNT/test $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to make mount points"
 
 _log "Mount xfs via loop"
-mount -t xfs -o loop $SCRATCH_MNT/test.xfs $SCRATCH_MNT/test >> $seqres.full 2>&1 \
+loop_dev1=$(_create_loop_device $SCRATCH_MNT/test.xfs)
+_mount $loop_dev1 $SCRATCH_MNT/test >> $seqres.full 2>&1 \
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress"
@@ -80,11 +82,12 @@ dd if=/dev/zero of=$SCRATCH_MNT/test/test.ext2 bs=1024 count=10240 >> $seqres.fu
     || _fail "!!! create file failed"
 
 _log "Create ext2 fs in file on looped xfs"
-echo y | mkfs -t ext2 $SCRATCH_MNT/test/test.ext2 >> $seqres.full 2>&1 \
+loop_dev2=$(_create_loop_device $SCRATCH_MNT/test/test.ext2)
+echo y | mkfs -t ext2 $loop_dev2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to mkfs ext2 on xfs"
 
 _log "Mount ext2 on xfs via loop"
-mount -t ext2 -o loop $SCRATCH_MNT/test/test.ext2 $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
+_mount $loop_dev2 $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress ext2 on xfs via loop"
@@ -96,12 +99,16 @@ rm -rf $SCRATCH_MNT/test/* >> $seqres.full 2>&1 \
     || _fail "!!! clean failed"
 
 _log "umount ext2 on xfs"
-umount $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
+_unmount $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! umount ext2 failed"
+_destroy_loop_device $loop_dev2
+unset loop_dev2
 
 _log "umount xfs"
-umount $SCRATCH_MNT/test >> $seqres.full 2>&1 \
+_unmount $SCRATCH_MNT/test >> $seqres.full 2>&1 \
     || _fail "!!! umount xfs failed"
+_destroy_loop_device $loop_dev1
+unset loop_dev1
 
 echo "--- mounts at end (before cleanup)" >> $seqres.full
 mount >> $seqres.full
tests/xfs/073 +13 -11
index e4b17c5e7..2274079ef 100755
@@ -23,6 +23,8 @@ _cleanup()
 	_scratch_unmount 2>/dev/null
 	_unmount $imgs.loop 2>/dev/null
 	_unmount $imgs.source_dir 2>/dev/null
+	[ -n "$loop_dev1" ] && _destroy_loop_device $loop_dev1
+	[ -n "$loop_dev2" ] && _destroy_loop_device $loop_dev2
 	[ -d $imgs.loop ] && rmdir $imgs.loop
 	[ -d $imgs.source_dir ] && rm -rf $imgs.source_dir
 	rm -f $imgs.* $tmp.* /var/tmp/xfs_copy.log.*
@@ -65,10 +67,11 @@ _verify_copy()
 	rmdir $target_dir 2>/dev/null
 	mkdir $target_dir
 
-	_mount -t xfs -o loop $target $target_dir 2>/dev/null
+	loop_dev1=$(_create_loop_device $target)
+	_mount $loop_dev1 $target_dir 2>/dev/null
 	if [ $? -ne 0 ]; then
 		echo retrying mount with nouuid option >>$seqres.full
-		_mount -t xfs -o loop -o nouuid $target $target_dir
+		_mount -o nouuid $loop_dev1 $target_dir
 		if [ $? -ne 0 ]; then
 			echo mount failed - evil!
 			return
@@ -100,6 +103,8 @@ _verify_copy()
 	echo unmounting and removing new image
 	_unmount $source_dir
 	_unmount $target_dir > /dev/null 2>&1
+	_destroy_loop_device $loop_dev1
+	unset loop_dev1
 	rm -f $target
 }
 
@@ -134,18 +139,15 @@ ${MKFS_XFS_PROG} -dfile,name=$imgs.source,size=100g \
 rmdir $imgs.source_dir 2>/dev/null
 mkdir $imgs.source_dir
 
-_mount -t xfs -o loop $imgs.source $imgs.source_dir
-loop2=`mount | grep $imgs.source | grep -o -e 'loop=.*[^),]' | grep -o -e '/.*$'`
+loop_dev2=$(_create_loop_device $imgs.source)
+_mount $loop_dev2 $imgs.source_dir
 cp -a $here $imgs.source_dir
-_mount -t xfs -o remount,ro $imgs.source $imgs.source_dir
-$XFS_COPY_PROG $imgs.source $imgs.image | _filter_copy '#' $imgs.image '#' '#'
+_mount -o remount,ro $loop_dev2 $imgs.source_dir
+$XFS_COPY_PROG $loop_dev2 $imgs.image 2> /dev/null | _filter_copy '#' $imgs.image '#' '#'
 _verify_copy $imgs.image $imgs.source $imgs.source_dir
 
-# HACK WARNING:
-#
-# We're done with the nested loop mount, now we have to clean
-# up the pieces that mount is incapable of doing.
-losetup -d $loop2 > /dev/null 2>&1
+_destroy_loop_device $loop_dev2
+unset loop_dev2
 
 echo
 echo === copying scratch device to multiple targets
tests/xfs/074 +12 -9
index 6a03c8111..5df864fad 100755
@@ -26,7 +26,7 @@ _begin_fstest quick auto prealloc rw
 _cleanup()
 {
 	cd /
-	_destroy_loop_device $LOOP_DEV
+	[ -n "$loop_dev" ] &&_destroy_loop_device $loop_dev
 	rm -f $tmp.* $LOOP_FILE
 }
 
@@ -45,10 +45,10 @@ LOOP_FILE=$TEST_DIR/$seq.img
 LOOP_MNT=$TEST_DIR/$seq.mnt
 mkdir -p $LOOP_MNT
 $XFS_IO_PROG -ft -c "truncate 1t" $LOOP_FILE >> $seqres.full
-LOOP_DEV=`_create_loop_device $LOOP_FILE`
+loop_dev=`_create_loop_device $LOOP_FILE`
 
-_mkfs_dev -d size=260g,agcount=2 $LOOP_DEV
-_mount $LOOP_DEV $LOOP_MNT
+_mkfs_dev -d size=260g,agcount=2 $loop_dev
+_mount $loop_dev $LOOP_MNT
 
 BLOCK_SIZE=$(_get_file_block_size $LOOP_MNT)
 
@@ -59,11 +59,11 @@ $XFS_IO_PROG -ft \
 	-c "falloc 0 $(($BLOCK_SIZE * 2097152))" \
 	$LOOP_MNT/foo >> $seqres.full
 
-umount $LOOP_MNT
-_check_xfs_filesystem $LOOP_DEV none none
+_unmount $LOOP_MNT
+_check_xfs_filesystem $loop_dev none none
 
-_mkfs_dev -f $LOOP_DEV
-_mount $LOOP_DEV $LOOP_MNT
+_mkfs_dev -f $loop_dev
+_mount $loop_dev $LOOP_MNT
 
 # check we trim both ends of the extent approproiately; this will fail
 # on 1k block size filesystems without the correct fixes in place.
@@ -73,7 +73,10 @@ $XFS_IO_PROG -ft \
 	$LOOP_MNT/foo >> $seqres.full
 
 _unmount $LOOP_MNT
-_check_xfs_filesystem $LOOP_DEV none none
+_check_xfs_filesystem $loop_dev none none
+
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 # success, all done
 echo "Silence is golden"
tests/xfs/078 +9 -7
index 6b325e05f..0d3c2eb23 100755
@@ -17,7 +17,7 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_unmount $LOOP_MNT 2>/dev/null
-	[ -n "$LOOP_DEV" ] && _destroy_loop_device $LOOP_DEV 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev 2>/dev/null
 	# try to keep the image file if test fails
 	[ $status -eq 0 ] && rm -f $LOOP_IMG
 	rmdir $LOOP_MNT
@@ -55,7 +55,7 @@ _grow_loop()
 	agsize=$5
 
 	$XFS_IO_PROG -f -c "truncate $original" $LOOP_IMG
-	LOOP_DEV=`_create_loop_device $LOOP_IMG`
+	loop_dev=`_create_loop_device $LOOP_IMG`
 
 	dparam=""
 	if [ -n "$agsize" ]; then
@@ -67,15 +67,15 @@ _grow_loop()
 	echo
 
 	echo "*** mkfs loop file (size=$original)"
-	$MKFS_XFS_PROG -b size=$bsize $dparam $LOOP_DEV | \
+	$MKFS_XFS_PROG -b size=$bsize $dparam $loop_dev | \
 		_filter_mkfs 2>/dev/null
 
 	echo "*** extend loop file"
-	_destroy_loop_device $LOOP_DEV
+	_destroy_loop_device $loop_dev
 	$XFS_IO_PROG -c "pwrite $new_size $bsize" $LOOP_IMG | _filter_io
-	LOOP_DEV=`_create_loop_device $LOOP_IMG`
+	loop_dev=`_create_loop_device $LOOP_IMG`
 	echo "*** mount loop filesystem"
-	_mount -t xfs $LOOP_DEV $LOOP_MNT
+	_mount $loop_dev $LOOP_MNT
 
 	echo "*** grow loop filesystem"
 	$XFS_GROWFS_PROG $LOOP_MNT 2>&1 |  _filter_growfs 2>&1
@@ -87,9 +87,11 @@ _grow_loop()
 	if [ "$check" -gt "0" ]
 	then
 		echo "*** check"
-		_check_xfs_filesystem $LOOP_IMG none none
+		_check_xfs_filesystem $loop_dev none none
 	fi
 
+	_destroy_loop_device $loop_dev
+	unset loop_dev
 	rm -f $LOOP_IMG
 }
 
tests/xfs/148 +14 -9
index c42c9b119..4d2f7a808 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_unmount $mntpt > /dev/null 2>&1
-	_destroy_loop_device $loopdev > /dev/null 2>&1
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	rm -r -f $tmp.*
 }
 
@@ -48,12 +48,12 @@ rm -f $imgfile $imgfile.old
 # even when security xattrs are present so we are always doing name matches on
 # lookup and not name hash compares as leaf/node forms will do.
 $XFS_IO_PROG -f -c 'truncate 40m' $imgfile
-loopdev=$(_create_loop_device $imgfile)
-MKFS_OPTIONS="-m crc=0 -i size=512" _mkfs_dev $loopdev >> $seqres.full
+loop_dev=$(_create_loop_device $imgfile)
+MKFS_OPTIONS="-m crc=0 -i size=512" _mkfs_dev $loop_dev >> $seqres.full
 
 # Mount image file
 mkdir -p $mntpt
-_mount $loopdev $mntpt
+_mount $loop_dev $mntpt
 
 echo "creating entries" >> $seqres.full
 
@@ -91,7 +91,8 @@ cat $tmp.log | _filter_test_dir
 
 # Corrupt the entries
 _unmount $mntpt
-_destroy_loop_device $loopdev
+_destroy_loop_device $loop_dev
+unset loop_dev
 cp $imgfile $imgfile.old
 sed -b \
 	-e "s/$nullstr/too_many\x00beans/g" \
@@ -99,8 +100,9 @@ sed -b \
 	-i $imgfile
 test "$(md5sum < $imgfile)" != "$(md5sum < $imgfile.old)" ||
 	_fail "sed failed to change the image file?"
-loopdev=$(_create_loop_device $imgfile)
-_mount $loopdev $mntpt
+
+loop_dev=$(_create_loop_device $imgfile)
+_mount $loop_dev $mntpt
 
 # Try to access the corrupt metadata
 echo "++ ACCESSING BAD METADATA" | tee -a $seqres.full
@@ -111,7 +113,7 @@ cat $tmp.log | _filter_test_dir | sed -e '/Could not list/d'
 echo "does scrub complain?" >> $seqres.full
 
 # Does scrub complain about this?
-if _supports_xfs_scrub $mntpt $loopdev; then
+if _supports_xfs_scrub $mntpt $loop_dev; then
 	$XFS_SCRUB_PROG -n $mntpt >> $seqres.full 2>&1
 	res=$?
 	test $((res & 1)) -eq 0 && \
@@ -122,11 +124,14 @@ echo "does repair complain?" >> $seqres.full
 
 # Does repair complain about this?
 _unmount $mntpt
-$XFS_REPAIR_PROG -n $loopdev >> $seqres.full 2>&1
+$XFS_REPAIR_PROG -n $loop_dev >> $seqres.full 2>&1
 res=$?
 test $res -eq 1 || \
 	echo "repair failed to report corruption ($res)"
 
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/149 +4 -0
index f2187109b..9a96f82ed 100755
@@ -84,6 +84,10 @@ $XFS_GROWFS_PROG -D 16384 $loop_symlink > /dev/null
 echo "=== xfs_growfs - check device node ==="
 $XFS_GROWFS_PROG -D 20480 $loop_dev > /dev/null
 
+_unmount $mntdir
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/216 +10 -5
index 6b8b2eb22..091c11d08 100755
@@ -15,6 +15,7 @@ _begin_fstest log metadata auto quick
 _cleanup()
 {
 	_unmount $LOOP_MNT > /dev/null 2>&1
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
 }
@@ -24,7 +25,7 @@ _scratch_mkfs_xfs >/dev/null 2>&1
 _scratch_mount
 
 _require_loop
-LOOP_DEV=$SCRATCH_MNT/test_fs
+LOOP_IMG=$SCRATCH_MNT/test_fs
 LOOP_MNT=$SCRATCH_MNT/test_fs_dir
 
 loop_mkfs_opts=
@@ -55,22 +56,26 @@ _do_mkfs()
 	for i in $*; do
 		echo -n "fssize=${i}g "
 		$MKFS_XFS_PROG -f -b size=4096 -l version=2 \
-			-d name=$LOOP_DEV,size=${i}g $loop_mkfs_opts |grep log
-		_mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
+			-d size=${i}g $loop_mkfs_opts $loop_dev |grep log
+		_mount $loop_dev $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
 		_unmount $LOOP_MNT > /dev/null 2>&1
 	done
 }
 # make large holey file
-$XFS_IO_PROG -f -c "truncate 256g" $LOOP_DEV
+$XFS_IO_PROG -f -c "truncate 256g" $LOOP_IMG
 
-choose_golden_output $0 $LOOP_DEV
+choose_golden_output $0 $LOOP_IMG
 
 #make loopback mount dir
 mkdir $LOOP_MNT
 
+loop_dev=$(_create_loop_device $LOOP_IMG)
+
 # walk over standard sizes (up to 256GB)
 _do_mkfs 1 2 4 8 16 32 64 128 256
 
+_destroy_loop_device $loop_dev
+unset loop_dev
 status=0
 exit
tests/xfs/217 +16 -6
index b2eb34490..dae6ce55f 100755
@@ -12,6 +12,12 @@ _begin_fstest log metadata auto
 # Import common functions.
 . ./common/filter
 
+_cleanup()
+{
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
+	cd /
+	rm -f $tmp.*
+}
 
 _require_scratch
 _scratch_mkfs_xfs >/dev/null 2>&1
@@ -20,7 +26,7 @@ _scratch_mount
 _require_fs_space $SCRATCH_MNT 2202000
 
 _require_loop
-LOOP_DEV=$SCRATCH_MNT/test_fs
+LOOP_IMG=$SCRATCH_MNT/test_fs
 LOOP_MNT=$SCRATCH_MNT/test_fs_dir
 
 _do_mkfs()
@@ -28,28 +34,30 @@ _do_mkfs()
 	for i in $*; do
 		echo -n "fssize=${i}g "
 		$MKFS_XFS_PROG -f -b size=4096 -l version=2 \
-			-d name=$LOOP_DEV,size=${i}g |grep log
-		_mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
+			-d size=${i}g $loop_dev |grep log
+		_mount $loop_dev $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
 		_unmount $LOOP_MNT > /dev/null 2>&1
 
 		# punch out the previous blocks so that we keep the amount of
 		# disk space the test requires down to a minimum.
-		$XFS_IO_PROG -f -c "unresvsp 0 16383g" $LOOP_DEV
+		$XFS_IO_PROG -f -c "unresvsp 0 16383g" $LOOP_IMG
 	done
 }
 # make large holey file
-$XFS_IO_PROG -f -c "truncate 16383g" $LOOP_DEV
+$XFS_IO_PROG -f -c "truncate 16383g" $LOOP_IMG
 
 #make loopback mount dir
 mkdir $LOOP_MNT
 
 # test if large logs are supported
-$MKFS_XFS_PROG -f -l size=256m -d name=$LOOP_DEV,size=10g > /dev/null 2>&1
+$MKFS_XFS_PROG -f -l size=256m -d name=$LOOP_IMG,size=10g > /dev/null 2>&1
 if [ $? -ne 0 ]; then
 	_notrun "large log sizes not supported by mkfs"
 fi
 
+loop_dev=$(_create_loop_device $LOOP_IMG)
+
 #
 # walk over "new" sizes supported by recent xfsprogs.
 # Note that the last test is for 16TB-1GB as 32bit platforms only support
@@ -57,5 +65,7 @@ fi
 #
 _do_mkfs 512 1024 2048 4096 8192 16383
 
+_destroy_loop_device $loop_dev
+unset loop_dev
 status=0
 exit
tests/xfs/250 +9 -5
index 4e3473ebc..2554e1e91 100755
@@ -14,7 +14,8 @@ _cleanup()
 {
 	cd /
 	_unmount $LOOP_MNT 2>/dev/null
-	rm -f $LOOP_DEV
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
+	rm -f $LOOP_IMG
 	rmdir $LOOP_MNT
 	rm -f $tmp.*
 }
@@ -26,7 +27,7 @@ _require_test
 _require_loop
 _require_xfs_io_command "falloc"
 
-LOOP_DEV=$TEST_DIR/$seq.fs
+LOOP_IMG=$TEST_DIR/$seq.fs
 LOOP_MNT=$TEST_DIR/$seq.mnt
 
 _filter_io()
@@ -45,7 +46,7 @@ _test_loop()
 	agsize=$2
 	fsize=$3
 
-	dparam="file,name=$LOOP_DEV,size=$size"
+	dparam="file,name=$LOOP_IMG,size=$size"
 	if [ -n "$agsize" ]; then
 		dparam="$dparam,agsize=$agsize"
 	fi
@@ -55,7 +56,8 @@ _test_loop()
 		| _filter_mkfs 2>/dev/null
 
 	echo "*** mount loop filesystem"
-	_mount -t xfs -o loop $LOOP_DEV $LOOP_MNT
+	loop_dev=$(_create_loop_device $LOOP_IMG)
+	mount $loop_dev $LOOP_MNT
 
 	echo "*** preallocate large file"
 	$XFS_IO_PROG -f -c "resvsp 0 $fsize" $LOOP_MNT/foo | _filter_io
@@ -64,7 +66,9 @@ _test_loop()
 	_unmount $LOOP_MNT > /dev/null 2>&1
 
 	echo "*** check loop filesystem"
-	 _check_xfs_filesystem $LOOP_DEV none none
+	_check_xfs_filesystem $loop_dev none none
+	_destroy_loop_device $loop_dev
+	unset loop_dev
 }
 
 _test_loop 50g 16m 40G
tests/xfs/259 +8 -5
index 0c8d6eb56..c2d26381a 100755
@@ -12,7 +12,10 @@ _begin_fstest auto quick
 # Override the default cleanup function.
 _cleanup()
 {
-    rm -f "$testfile"
+	[ -n "$loop_dev" ] && _destroy_loop_device $testfile
+	rm -f "$testfile"
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -45,13 +48,13 @@ for del in $sizes_to_check; do
 		rm -f "$testfile"
 		dd if=/dev/zero "of=$testfile" bs=1 count=0 seek=$ddseek \
 			>/dev/null 2>&1 || echo "dd failed"
-		lofile=$(losetup -f)
-		losetup $lofile "$testfile"
-		$MKFS_XFS_PROG -l size=32m -b size=$bs $lofile |  _filter_mkfs \
+		loop_dev=$(_create_loop_device $testfile)
+		$MKFS_XFS_PROG -l size=32m -b size=$bs $loop_dev |  _filter_mkfs \
 			>/dev/null 2> $tmp.mkfs || echo "mkfs failed!"
 		. $tmp.mkfs
 		sync
-		losetup -d $lofile
+		_destroy_loop_device $loop_dev
+		unset loop_dev
 	done
 done
 
tests/xfs/513 +20 -24
index 5895e6e2d..0c0edc75e 100755
@@ -15,12 +15,8 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_unmount $LOOP_MNT 2>/dev/null
-	if [ -n "$LOOP_DEV" ];then
-		_destroy_loop_device $LOOP_DEV 2>/dev/null
-	fi
-	if [ -n "$LOOP_SPARE_DEV" ];then
-		_destroy_loop_device $LOOP_SPARE_DEV 2>/dev/null
-	fi
+	[ -n $loop_dev ] &&_destroy_loop_device $loop_dev
+	[ -n $loop_spare_dev ] &&_destroy_loop_device $loop_spare_dev
 	rm -f $LOOP_IMG
 	rm -f $LOOP_SPARE_IMG
 	rmdir $LOOP_MNT
@@ -42,11 +38,11 @@ LOOP_MNT=$TEST_DIR/$seq.mnt
 
 echo "** create loop device"
 $XFS_IO_PROG -f -c "truncate 32g" $LOOP_IMG
-LOOP_DEV=`_create_loop_device $LOOP_IMG`
+loop_dev=`_create_loop_device $LOOP_IMG`
 
 echo "** create loop log device"
 $XFS_IO_PROG -f -c "truncate 1g" $LOOP_SPARE_IMG
-LOOP_SPARE_DEV=`_create_loop_device $LOOP_SPARE_IMG`
+loop_spare_dev=`_create_loop_device $LOOP_SPARE_IMG`
 
 echo "** create loop mount point"
 rmdir $LOOP_MNT 2>/dev/null
@@ -55,8 +51,8 @@ mkdir -p $LOOP_MNT || _fail "cannot create loopback mount point"
 filter_loop()
 {
 	sed -e "s,\B$LOOP_MNT,LOOP_MNT,g" \
-	    -e "s,\B$LOOP_DEV,LOOP_DEV,g" \
-	    -e "s,\B$LOOP_SPARE_DEV,LOOP_SPARE_DEV,g"
+	    -e "s,\B$loop_dev,LOOP_DEV,g" \
+	    -e "s,\B$loop_spare_dev,LOOP_SPARE_DEV,g"
 }
 
 filter_xfs_opt()
@@ -69,22 +65,22 @@ MKFS_OPTIONS=""
 do_mkfs()
 {
 	echo "FORMAT: $@" | filter_loop | tee -a $seqres.full
-	$MKFS_XFS_PROG -f $* $LOOP_DEV | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
+	$MKFS_XFS_PROG -f $* $loop_dev | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
 	if [ "${PIPESTATUS[0]}" -ne 0 ]; then
-		_fail "Fails on _mkfs_dev $* $LOOP_DEV"
+		_fail "Fails on _mkfs_dev $* $loop_dev"
 	fi
 	. $tmp.mkfs
 }
 
 is_dev_mounted()
 {
-	findmnt --source $LOOP_DEV >/dev/null
+	findmnt --source $loop_dev >/dev/null
 	return $?
 }
 
 get_mount_info()
 {
-	findmnt --source $LOOP_DEV -o OPTIONS -n
+	findmnt --source $loop_dev -o OPTIONS -n
 }
 
 force_unmount()
@@ -103,29 +99,29 @@ _do_test()
 	local info
 
 	# mount test
-	_mount $LOOP_DEV $LOOP_MNT $opts 2>>$seqres.full
+	_mount $loop_dev $LOOP_MNT $opts 2>>$seqres.full
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "${mounted}" = "fail" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to fail, but it succeeded"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -ne 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs not mounted even mount return 0"
 			return 1
 		fi
 	else
 		if [ "${mounted}" = "pass" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to succeed, but it failed"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -eq 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs is mounted even mount return non-zero"
 			return 1
 		fi
@@ -141,13 +137,13 @@ _do_test()
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "$found" != "true" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expected to find \"$key\" in mount info \"$info\""
 			return 1
 		fi
 	else
 		if [ "$found" != "false" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: did not expect to find \"$key\" in \"$info\""
 			return 1
 		fi
@@ -253,9 +249,9 @@ do_test "-o logbsize=512k" fail
 # Test logdev
 do_mkfs
 do_test "" pass "logdev" "false"
-do_test "-o logdev=$LOOP_SPARE_DEV" fail
-do_mkfs -l logdev=$LOOP_SPARE_DEV
-do_test "-o logdev=$LOOP_SPARE_DEV" pass "logdev=$LOOP_SPARE_DEV" "true"
+do_test "-o logdev=$loop_spare_dev" fail
+do_mkfs -l logdev=$loop_spare_dev
+do_test "-o logdev=$loop_spare_dev" pass "logdev=$loop_spare_dev" "true"
 do_test "" fail
 
 # Test noalign
tests/xfs/521 +6 -2
index 13982c440..c92c621a2 100755
@@ -21,7 +21,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -35,7 +35,7 @@ _require_no_large_scratch_dev
 
 echo "Create fake rt volume"
 truncate -s 400m $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Format and mount 100m rt volume"
 export USE_EXTERNAL=yes
@@ -69,6 +69,10 @@ cp -p $testdir/original $testdir/copy3
 echo "Check filesystem"
 _check_scratch_fs
 
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/528 +7 -3
index 6ca9a2370..a1efbbd27 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -155,11 +155,11 @@ test_ops() {
 
 echo "Create fake rt volume"
 $XFS_IO_PROG -f -c "truncate 400m" $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Make sure synth rt volume works"
 export USE_EXTERNAL=yes
-export SCRATCH_RTDEV=$rtdev
+export SCRATCH_RTDEV=$rt_loop_dev
 _scratch_mkfs > $seqres.full
 _try_scratch_mount || \
 	_notrun "Could not mount with synthetic rt volume"
@@ -170,6 +170,10 @@ test_ops 262144
 # not a power of two
 test_ops 327680
 
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/530 +6 -5
index 8a182bd6a..d0d0e2665 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -52,12 +52,12 @@ fi
 
 rtdevsz=$((nr_bits * rtextsz))
 truncate -s $rtdevsz $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Format and mount rt volume"
 
 export USE_EXTERNAL=yes
-export SCRATCH_RTDEV=$rtdev
+export SCRATCH_RTDEV=$rt_loop_dev
 _scratch_mkfs -d size=$((1024 * 1024 * 1024)) \
 	      -r size=${rtextsz},extsize=${rtextsz} >> $seqres.full
 _try_scratch_mount || _notrun "Couldn't mount fs with synthetic rt volume"
@@ -116,8 +116,9 @@ done
 echo "Check filesystem"
 _check_scratch_fs
 
-losetup -d $rtdev
-rm -f $TEST_DIR/$seq.rtvol
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
 
 # success, all done
 status=0
tests/xfs/606 +8 -4
index f958bddd8..b537ea145 100755
@@ -13,10 +13,8 @@ _begin_fstest auto quick growfs
 
 _cleanup()
 {
-	local dev
 	_unmount $LOOP_MNT 2>/dev/null
-	dev=$(losetup -j testfile | cut -d: -f1)
-	losetup -d $dev 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	rm -rf $LOOP_IMG $LOOP_MNT
 	cd /
 	rm -f $tmp.*
@@ -40,7 +38,9 @@ $MKFS_XFS_PROG -f $LOOP_IMG >$seqres.full
 # Extend by just 8K, expected to start with the last full-size AG ends of
 # above 1G block device.
 $XFS_IO_PROG -f -c "truncate 1073750016" $LOOP_IMG
-_mount -oloop $LOOP_IMG $LOOP_MNT
+
+loop_dev=$(_create_loop_device $LOOP_IMG)
+_mount $loop_dev $LOOP_MNT
 # A known bug shows "XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on
 # device" at here, refer to _fixed_by_kernel_commit above
 $XFS_GROWFS_PROG $LOOP_MNT >$seqres.full
@@ -48,6 +48,10 @@ if [ $? -ne 0 ];then
 	echo "xfs_growfs fails!"
 fi
 
+_unmount $LOOP_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 echo "Silence is golden"
 # success, all done
 status=0
tests/xfs/613 +14 -26
index 6ba3d87bf..b87cae91e 100755
@@ -15,14 +15,8 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_unmount $LOOP_MNT 2>/dev/null
-	if [ -n "$LOOP_DEV" ];then
-		_destroy_loop_device $LOOP_DEV 2>/dev/null
-	fi
-	if [ -n "$LOOP_SPARE_DEV" ];then
-		_destroy_loop_device $LOOP_SPARE_DEV 2>/dev/null
-	fi
+	[ -n $loop_dev ] &&_destroy_loop_device $loop_dev
 	rm -f $LOOP_IMG
-	rm -f $LOOP_SPARE_IMG
 	rmdir $LOOP_MNT
 }
 
@@ -38,16 +32,11 @@ _require_loop
 _require_xfs_io_command "falloc"
 
 LOOP_IMG=$TEST_DIR/$seq.dev
-LOOP_SPARE_IMG=$TEST_DIR/$seq.logdev
 LOOP_MNT=$TEST_DIR/$seq.mnt
 
 echo "** create loop device"
 $XFS_IO_PROG -f -c "truncate 32g" $LOOP_IMG
-LOOP_DEV=`_create_loop_device $LOOP_IMG`
-
-echo "** create loop log device"
-$XFS_IO_PROG -f -c "truncate 1g" $LOOP_SPARE_IMG
-LOOP_SPARE_DEV=`_create_loop_device $LOOP_SPARE_IMG`
+loop_dev=`_create_loop_device $LOOP_IMG`
 
 echo "** create loop mount point"
 rmdir $LOOP_MNT 2>/dev/null
@@ -56,8 +45,7 @@ mkdir -p $LOOP_MNT || _fail "cannot create loopback mount point"
 filter_loop()
 {
 	sed -e "s,\B$LOOP_MNT,LOOP_MNT,g" \
-	    -e "s,\B$LOOP_DEV,LOOP_DEV,g" \
-	    -e "s,\B$LOOP_SPARE_DEV,LOOP_SPARE_DEV,g"
+	    -e "s,\B$loop_dev,LOOP_DEV,g"
 }
 
 filter_xfs_opt()
@@ -70,22 +58,22 @@ MKFS_OPTIONS=""
 do_mkfs()
 {
 	echo "FORMAT: $@" | filter_loop | tee -a $seqres.full
-	$MKFS_XFS_PROG -f $* $LOOP_DEV | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
+	$MKFS_XFS_PROG -f $* $loop_dev | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
 	if [ "${PIPESTATUS[0]}" -ne 0 ]; then
-		_fail "Fails on _mkfs_dev $* $LOOP_DEV"
+		_fail "Fails on _mkfs_dev $* $loop_dev"
 	fi
 	. $tmp.mkfs
 }
 
 is_dev_mounted()
 {
-	findmnt --source $LOOP_DEV >/dev/null
+	findmnt --source $loop_dev >/dev/null
 	return $?
 }
 
 get_mount_info()
 {
-	findmnt --source $LOOP_DEV -o OPTIONS -n
+	findmnt --source $loop_dev -o OPTIONS -n
 }
 
 force_unmount()
@@ -104,29 +92,29 @@ _do_test()
 	local info
 
 	# mount test
-	_mount $LOOP_DEV $LOOP_MNT $opts 2>>$seqres.full
+	_mount $loop_dev $LOOP_MNT $opts 2>>$seqres.full
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "${mounted}" = "fail" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to fail, but it succeeded"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -ne 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs not mounted even mount return 0"
 			return 1
 		fi
 	else
 		if [ "${mounted}" = "pass" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to succeed, but it failed"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -eq 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs is mounted even mount return non-zero"
 			return 1
 		fi
@@ -142,13 +130,13 @@ _do_test()
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "$found" != "true" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expected to find \"$key\" in mount info \"$info\""
 			return 1
 		fi
 	else
 		if [ "$found" != "false" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: did not expect to find \"$key\" in \"$info\""
 			return 1
 		fi
tests/xfs/613.out +0 -1
index 1624617ee..2a693c53c 100644
@@ -1,6 +1,5 @@
 QA output created by 613
 ** create loop device
-** create loop log device
 ** create loop mount point
 ** start xfs mount testing ...
 FORMAT: -m crc=0
Signature · 1 line
2.45.2
#23 lore raw

Re: [PATCH 13/40] fstests: clean up loop device instantiation

In reply to Dave Chinner
Quoted 70 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Lots of tests do there own special thing with loop devices rather than using _create_loop_device() and _destroy_loop_device(). This oftens means they do not clean up after themselves properly, leaving stale loop devices around that result in unmountable test or scratch devices. This is common when tests are killed by user interrupt.

Even the tests that do use _destroy_loop_device and try to clean up often do it incorrectly, leading to spurious error messages.

Some tests try to use dynamic instantiation via "mount -o loop", but then don't clean up in the correct order or hack around to find the loop device that was instantiated because the test needs to know the instantiated device name

Clean this up by converting all the tests to use _create_loop_device() and _destroy_loop_device(). In all the tests, use the variable "loop_dev" for the device consistently. In _destroy_loop_device(), test that a device name has been passed so that we don't try to clean up the same device twice (e.g. once before test exit and again from the _cleanup() function). When we destroy a loop device, unset the variable used to hold the loop device name so that we don't try to destroy it twice.

This results in much more reliable cleanup and clean exit from fstests when killed by the user.

---
 common/metadump   | 32 ++++++++++++--------------------
 tests/generic/067 | 11 ++++++++---
 tests/generic/361 |  6 +++++-
 tests/generic/563 | 14 +++++++++-----
 tests/generic/564 | 12 ++++++------
 tests/generic/590 |  9 ++++++---
 tests/generic/744 |  4 ++--
 tests/generic/746 |  6 +++++-
 tests/xfs/014     |  7 +++++--
 tests/xfs/049     | 35 +++++++++++++++++++++--------------
 tests/xfs/073     | 24 +++++++++++++-----------
 tests/xfs/074     | 21 ++++++++++++---------
 tests/xfs/078     | 16 +++++++++-------
 tests/xfs/148     | 23 ++++++++++++++---------
 tests/xfs/149     |  4 ++++
 tests/xfs/216     | 15 ++++++++++-----
 tests/xfs/217     | 22 ++++++++++++++++------
 tests/xfs/250     | 14 +++++++++-----
 tests/xfs/259     | 13 ++++++++-----
 tests/xfs/513     | 44 ++++++++++++++++++++------------------------
 tests/xfs/521     |  8 ++++++--
 tests/xfs/528     | 10 +++++++---
 tests/xfs/530     | 11 ++++++-----
 tests/xfs/606     | 12 ++++++++----
 tests/xfs/613     | 40 ++++++++++++++--------------------------
 tests/xfs/613.out |  1 -
 26 files changed, 235 insertions(+), 179 deletions(-)
common/metadump +0 -3
index 3373edfe9..bc3038e3b 100644
@@ -24,17 +24,9 @@ _xfs_cleanup_verify_metadump()
 
 	test -n "$XFS_METADUMP_FILE" && rm -f "$XFS_METADUMP_FILE"
 
-	if [ -n "$XFS_METADUMP_IMG" ]; then
-		losetup -n -a -O BACK-FILE,NAME | grep "^$XFS_METADUMP_IMG" | while read backing ldev; do
-			losetup -d "$ldev"

common/metadump _xfs_cleanup_verify_metadump() · line 28

Quoted 10 lines
common/metadump +2 -8
-		done
-
-		# Don't call rm directly with a globbed argument here to avoid
-		# issues issues with variable expansions.
-		for img in "$XFS_METADUMP_IMG"*; do
-			test -e "$img" && rm -f "$img"
-		done
-	fi
+	[ -n "$md_data_loop_dev" ] && _destroy_loop_device $md_data_loop_dev
+	[ -n "$md_log_loop_dev" ] && _destroy_loop_device $md_log_loop_dev

As Darrick reported that he always hit below error:

losetup: /dev/loop0: detach failed: No such device or address Cannot destroy loop device /dev/loop0

I'm wondering we should use [ -b ] not [ -n ] at here? Or output the stderr to /dev/null directly. Due to... (see below)

Quoted 16 lines
common/metadump +3 -2
+	rm -f $data_img $log_img
 }
 
 # Can xfs_metadump snapshot the fs metadata to a v1 metadump file?
@@ -122,25 +114,25 @@ _xfs_verify_metadump_v2()
 		_scratch_xfs_mdrestore $metadump_file
 
 	# Create loopdev for data device so we can mount the fs
-	data_loop=$(_create_loop_device $data_img)
+	md_data_loop_dev=$(_create_loop_device $data_img)
 
 	# Create loopdev for log device if we recovered anything
-	test -s "$log_img" && log_loop=$(_create_loop_device $log_img)
+	test -s "$log_img" && md_log_loop_dev=$(_create_loop_device $log_img)
 
 	# Mount fs, run an extra test, fsck, and unmount

common/metadump _xfs_verify_metadump_v2() · line 128

Quoted 10 lines
common/metadump +4 -4
-	SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop _scratch_mount
+	SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev _scratch_mount
 	if [ -n "$extra_test" ]; then
-		SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop $extra_test
+		SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev $extra_test
 	fi
-	SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop _check_xfs_scratch_fs
-	SCRATCH_DEV=$data_loop _scratch_unmount
+	SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev _check_xfs_scratch_fs
+	_unmount $md_data_loop_dev

We umount loop devices we created above at here, and ...

common/metadump _xfs_verify_metadump_v2() · line 135

Quoted 10 lines
common/metadump +3 -3
 	# Tear down what we created
-	if [ -b "$log_loop" ]; then
-		_destroy_loop_device $log_loop
+	if [ -b "$md_log_loop_dev" ]; then
+		_destroy_loop_device $md_log_loop_dev
 		rm -f $log_img
 	fi
-	_destroy_loop_device $data_loop
+	_destroy_loop_device $md_data_loop_dev

we even has destroied them at here.

So that _xfs_cleanup_verify_metadump function hit "detach failed: No such device or address", due to those loop devices might have been released, but the variables are not unset.

I didn't give it a test, just did a code review. As Darrick uses this part mostly, could you please check what I said above?

Thanks,
Zorro

Quoted 1,198 lines
common/metadump +0 -0
 	rm -f $data_img
 }
 
tests/generic/067 +8 -3
index ccb1e3fbb..b45ae834f 100755
@@ -37,13 +37,18 @@ mount_nonexistent_mnt()
 	$MOUNT_PROG $SCRATCH_DEV $TEST_DIR/nosuchdir >>$seqres.full 2>&1
 }
 
-# fs driver should be able to handle mounting a free loop device gracefully
-# xfs ever hung, "ec53d1d xfs: don't block on buffer read errors" fixed it
+# fs driver should be able to handle mounting a free loop device gracefully xfs
+# ever hung, "ec53d1d xfs: don't block on buffer read errors" fixed it
+#
+# Using 'losetup -f' like this is racy. We might end up mounting a real loop dev
+# here, so unmount $SCRATCH_MNT (not the loop dev we might not own!) and ignore
+# any error it might return.
 mount_free_loopdev()
 {
 	echo "# mount a free loop device" >>$seqres.full
 	loopdev=`losetup -f`
-	$MOUNT_PROG -t $FSTYP $loopdev $SCRATCH_MNT >>$seqres.full 2>&1
+	_mount $loopdev $SCRATCH_MNT >>$seqres.full 2>&1
+	_unmount $SCRATCH_MNT >> /dev/null 2>&1
 }
 
 # mount with wrong fs type specified.
tests/generic/361 +5 -1
index 7273dd056..e2b798436 100755
@@ -17,7 +17,7 @@ _begin_fstest auto quick
 _cleanup()
 {
 	_unmount $fs_mnt
-	_destroy_loop_device $loop_dev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
 }
@@ -54,6 +54,10 @@ $XFS_IO_PROG -fc "pwrite 0 520m" $fs_mnt/testfile >>$seqres.full 2>&1
 # remount should not hang
 $MOUNT_PROG -o remount,ro $fs_mnt >>$seqres.full 2>&1
 
+_unmount $fs_mnt
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 echo "Silence is golden"
 status=0
tests/generic/563 +9 -5
index f0d2f404c..34d21170c 100755
@@ -87,12 +87,12 @@ reset()
 
 # cgroup I/O accounting doesn't work on partitions. Use a loop device to rule
 # that out.
-LOOP_DEV=$(_create_loop_device $SCRATCH_DEV)
-smajor=$((0x`stat -L -c %t $LOOP_DEV`))
-sminor=$((0x`stat -L -c %T $LOOP_DEV`))
+loop_dev=$(_create_loop_device $SCRATCH_DEV)
+smajor=$((0x`stat -L -c %t $loop_dev`))
+sminor=$((0x`stat -L -c %T $loop_dev`))
 
-_mkfs_dev $LOOP_DEV >> $seqres.full 2>&1
-_mount $LOOP_DEV $SCRATCH_MNT || _fail "mount failed"
+_mkfs_dev $loop_dev >> $seqres.full 2>&1
+_mount $loop_dev $SCRATCH_MNT || _fail "mount failed"
 
 blksize=$(_get_block_size "$SCRATCH_MNT")
 
@@ -147,6 +147,10 @@ if [ "$drop_io_cgroup" = 1 ]; then
 	echo "-io" > $cgdir/cgroup.subtree_control || _fail "subtree control"
 fi
 
+_unmount $SCRATCH_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/generic/564 +6 -6
index 647472d78..b66b72496 100755
@@ -19,7 +19,7 @@ _cleanup()
 {
 	cd /
 	rm -rf $tmp.*
-	[ -z "$loopdev" ] || _destroy_loop_device $loopdev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 }
 
 # Import common functions.
@@ -72,11 +72,11 @@ $XFS_IO_PROG -f -c "copy_range -l 32k $testdir" $testdir/copy
 echo
 echo source/destination as blkdev returns EINVAL
 $XFS_IO_PROG -f -c "truncate 128k" $testdir/img >> $seqres.full 2>&1
-loopdev=`_create_loop_device $testdir/img`
-$XFS_IO_PROG -c "copy_range -l 32k $testdir/file" $loopdev
-$XFS_IO_PROG -f -c "copy_range -l 32k $loopdev" $testdir/copy
-_destroy_loop_device $loopdev
-loopdev=
+loop_dev=`_create_loop_device $testdir/img`
+$XFS_IO_PROG -c "copy_range -l 32k $testdir/file" $loop_dev
+$XFS_IO_PROG -f -c "copy_range -l 32k $loop_dev" $testdir/copy
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo
 echo source/destination as chardev returns EINVAL
tests/generic/590 +6 -3
index 2b7ccfb53..2eefa2c62 100755
@@ -15,9 +15,10 @@ _begin_fstest auto prealloc preallocrw
 # Override the default cleanup function.
 _cleanup()
 {
+	_scratch_unmount
+	[ -n $loop_dev ] && _destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
-	test -n "$loop" && _destroy_loop_device "$loop"
 	rm -f "$TEST_DIR/$seq"
 }
 
@@ -56,9 +57,9 @@ if [[ $FSTYP = xfs ]]; then
 		loopsz="$((filesz + (1 << 26)))"
 		_require_fs_space "$TEST_DIR" $((loopsz / 1024))
 		$XFS_IO_PROG -c "truncate $loopsz" -f "$TEST_DIR/$seq"
-		loop="$(_create_loop_device "$TEST_DIR/$seq")"
+		loop_dev="$(_create_loop_device "$TEST_DIR/$seq")"
 		USE_EXTERNAL=yes
-		SCRATCH_RTDEV="$loop"
+		SCRATCH_RTDEV="$loop_dev"
 		disabled_features=()
 
 		# disable reflink if not supported by realtime devices
@@ -114,6 +115,8 @@ $XFS_IO_PROG -c "truncate 0" -c fsync "$SCRATCH_MNT/file"
 # We need to do this before the loop device gets torn down.
 _scratch_unmount
 _check_scratch_fs
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo "Silence is golden"
 status=0
tests/generic/744 +2 -2
index df8f6ae9b..cda10e0f6 100755
@@ -17,8 +17,8 @@ _cleanup()
 
 	_unmount $mnt2 &> /dev/null
 	_unmount $mnt1 &> /dev/null
-	[ -b "$loop_dev2" ] && losetup -d $loop_dev2
-	[ -b "$loop_dev1" ] && losetup -d $loop_dev1
+	[ -b "$loop_dev2" ] && _destroy_loop_device $loop_dev2
+	[ -b "$loop_dev1" ] && _destroy_loop_device $loop_dev1
 	[ -n "$seq" ] && rm -rf $TEST_DIR/$seq
 }
 
tests/generic/746 +5 -1
index 671910606..ba8ed25e8 100755
@@ -39,7 +39,7 @@ esac
 _cleanup()
 {
 	_unmount $loop_mnt &> /dev/null
-	_destroy_loop_device $loop_dev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	if [ $status -eq 0 ]; then
 		rm -rf $tmp
 		rm $img_file
@@ -223,5 +223,9 @@ while read line; do
 done < $fiemap_after
 echo "done."
 
+_unmount $loop_mnt
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 status=0
 exit
tests/xfs/014 +5 -2
index f638e4b16..39ea40e2a 100755
@@ -23,6 +23,7 @@ _cleanup()
 {
 	cd /
 	_unmount $LOOP_MNT 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	_scratch_unmount 2>/dev/null
 	rm -f $tmp.*
 }
@@ -166,19 +167,21 @@ LOOP_FILE=$SCRATCH_MNT/$seq.fs
 LOOP_MNT=$SCRATCH_MNT/$seq.mnt
 
 $MKFS_XFS_PROG -d "file=1,name=$LOOP_FILE,size=10g" >> $seqres.full 2>&1
+loop_dev=$(_create_loop_device $LOOP_FILE)
 
 mkdir -p $LOOP_MNT
-_mount -o uquota,gquota $LOOP_FILE $LOOP_MNT || \
+_mount -o uquota,gquota $loop_dev $LOOP_MNT || \
 	_fail "Failed to mount loop fs."
 
 _test_enospc $LOOP_MNT
 _test_edquot $LOOP_MNT
 
 _unmount $LOOP_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo $orig_sp_time > /proc/sys/fs/xfs/speculative_prealloc_lifetime
 
 _scratch_unmount
-
 status=0
 exit
tests/xfs/049 +21 -14
index 4163a144f..cdcddf764 100755
@@ -12,16 +12,17 @@ _begin_fstest rw auto quick
 # Override the default cleanup function.
 _cleanup()
 {
-    cd /
-    umount $SCRATCH_MNT/test2 > /dev/null 2>&1
-    umount $SCRATCH_MNT/test > /dev/null 2>&1
-    rm -f $tmp.*
+	cd /
+	_unmount $SCRATCH_MNT/test2 > /dev/null 2>&1
+	_unmount $SCRATCH_MNT/test > /dev/null 2>&1
+	[ -n "$loop_dev2" ] && _destroy_loop_device $loop_dev2
+	[ -n "$loop_dev1" ] && _destroy_loop_device $loop_dev1
+	rm -f $tmp.*
 
-    if [ -w $seqres.full ]
-    then
-        echo "--- mounts at end (after cleanup)" >> $seqres.full
-        mount >> $seqres.full
-    fi
+	if [ -w $seqres.full ]; then
+		echo "--- mounts at end (after cleanup)" >> $seqres.full
+		mount >> $seqres.full
+	fi
 }
 
 # Import common functions.
@@ -64,7 +65,8 @@ mkdir $SCRATCH_MNT/test $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to make mount points"
 
 _log "Mount xfs via loop"
-mount -t xfs -o loop $SCRATCH_MNT/test.xfs $SCRATCH_MNT/test >> $seqres.full 2>&1 \
+loop_dev1=$(_create_loop_device $SCRATCH_MNT/test.xfs)
+_mount $loop_dev1 $SCRATCH_MNT/test >> $seqres.full 2>&1 \
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress"
@@ -80,11 +82,12 @@ dd if=/dev/zero of=$SCRATCH_MNT/test/test.ext2 bs=1024 count=10240 >> $seqres.fu
     || _fail "!!! create file failed"
 
 _log "Create ext2 fs in file on looped xfs"
-echo y | mkfs -t ext2 $SCRATCH_MNT/test/test.ext2 >> $seqres.full 2>&1 \
+loop_dev2=$(_create_loop_device $SCRATCH_MNT/test/test.ext2)
+echo y | mkfs -t ext2 $loop_dev2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to mkfs ext2 on xfs"
 
 _log "Mount ext2 on xfs via loop"
-mount -t ext2 -o loop $SCRATCH_MNT/test/test.ext2 $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
+_mount $loop_dev2 $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress ext2 on xfs via loop"
@@ -96,12 +99,16 @@ rm -rf $SCRATCH_MNT/test/* >> $seqres.full 2>&1 \
     || _fail "!!! clean failed"
 
 _log "umount ext2 on xfs"
-umount $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
+_unmount $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! umount ext2 failed"
+_destroy_loop_device $loop_dev2
+unset loop_dev2
 
 _log "umount xfs"
-umount $SCRATCH_MNT/test >> $seqres.full 2>&1 \
+_unmount $SCRATCH_MNT/test >> $seqres.full 2>&1 \
     || _fail "!!! umount xfs failed"
+_destroy_loop_device $loop_dev1
+unset loop_dev1
 
 echo "--- mounts at end (before cleanup)" >> $seqres.full
 mount >> $seqres.full
tests/xfs/073 +13 -11
index e4b17c5e7..2274079ef 100755
@@ -23,6 +23,8 @@ _cleanup()
 	_scratch_unmount 2>/dev/null
 	_unmount $imgs.loop 2>/dev/null
 	_unmount $imgs.source_dir 2>/dev/null
+	[ -n "$loop_dev1" ] && _destroy_loop_device $loop_dev1
+	[ -n "$loop_dev2" ] && _destroy_loop_device $loop_dev2
 	[ -d $imgs.loop ] && rmdir $imgs.loop
 	[ -d $imgs.source_dir ] && rm -rf $imgs.source_dir
 	rm -f $imgs.* $tmp.* /var/tmp/xfs_copy.log.*
@@ -65,10 +67,11 @@ _verify_copy()
 	rmdir $target_dir 2>/dev/null
 	mkdir $target_dir
 
-	_mount -t xfs -o loop $target $target_dir 2>/dev/null
+	loop_dev1=$(_create_loop_device $target)
+	_mount $loop_dev1 $target_dir 2>/dev/null
 	if [ $? -ne 0 ]; then
 		echo retrying mount with nouuid option >>$seqres.full
-		_mount -t xfs -o loop -o nouuid $target $target_dir
+		_mount -o nouuid $loop_dev1 $target_dir
 		if [ $? -ne 0 ]; then
 			echo mount failed - evil!
 			return
@@ -100,6 +103,8 @@ _verify_copy()
 	echo unmounting and removing new image
 	_unmount $source_dir
 	_unmount $target_dir > /dev/null 2>&1
+	_destroy_loop_device $loop_dev1
+	unset loop_dev1
 	rm -f $target
 }
 
@@ -134,18 +139,15 @@ ${MKFS_XFS_PROG} -dfile,name=$imgs.source,size=100g \
 rmdir $imgs.source_dir 2>/dev/null
 mkdir $imgs.source_dir
 
-_mount -t xfs -o loop $imgs.source $imgs.source_dir
-loop2=`mount | grep $imgs.source | grep -o -e 'loop=.*[^),]' | grep -o -e '/.*$'`
+loop_dev2=$(_create_loop_device $imgs.source)
+_mount $loop_dev2 $imgs.source_dir
 cp -a $here $imgs.source_dir
-_mount -t xfs -o remount,ro $imgs.source $imgs.source_dir
-$XFS_COPY_PROG $imgs.source $imgs.image | _filter_copy '#' $imgs.image '#' '#'
+_mount -o remount,ro $loop_dev2 $imgs.source_dir
+$XFS_COPY_PROG $loop_dev2 $imgs.image 2> /dev/null | _filter_copy '#' $imgs.image '#' '#'
 _verify_copy $imgs.image $imgs.source $imgs.source_dir
 
-# HACK WARNING:
-#
-# We're done with the nested loop mount, now we have to clean
-# up the pieces that mount is incapable of doing.
-losetup -d $loop2 > /dev/null 2>&1
+_destroy_loop_device $loop_dev2
+unset loop_dev2
 
 echo
 echo === copying scratch device to multiple targets
tests/xfs/074 +12 -9
index 6a03c8111..5df864fad 100755
@@ -26,7 +26,7 @@ _begin_fstest quick auto prealloc rw
 _cleanup()
 {
 	cd /
-	_destroy_loop_device $LOOP_DEV
+	[ -n "$loop_dev" ] &&_destroy_loop_device $loop_dev
 	rm -f $tmp.* $LOOP_FILE
 }
 
@@ -45,10 +45,10 @@ LOOP_FILE=$TEST_DIR/$seq.img
 LOOP_MNT=$TEST_DIR/$seq.mnt
 mkdir -p $LOOP_MNT
 $XFS_IO_PROG -ft -c "truncate 1t" $LOOP_FILE >> $seqres.full
-LOOP_DEV=`_create_loop_device $LOOP_FILE`
+loop_dev=`_create_loop_device $LOOP_FILE`
 
-_mkfs_dev -d size=260g,agcount=2 $LOOP_DEV
-_mount $LOOP_DEV $LOOP_MNT
+_mkfs_dev -d size=260g,agcount=2 $loop_dev
+_mount $loop_dev $LOOP_MNT
 
 BLOCK_SIZE=$(_get_file_block_size $LOOP_MNT)
 
@@ -59,11 +59,11 @@ $XFS_IO_PROG -ft \
 	-c "falloc 0 $(($BLOCK_SIZE * 2097152))" \
 	$LOOP_MNT/foo >> $seqres.full
 
-umount $LOOP_MNT
-_check_xfs_filesystem $LOOP_DEV none none
+_unmount $LOOP_MNT
+_check_xfs_filesystem $loop_dev none none
 
-_mkfs_dev -f $LOOP_DEV
-_mount $LOOP_DEV $LOOP_MNT
+_mkfs_dev -f $loop_dev
+_mount $loop_dev $LOOP_MNT
 
 # check we trim both ends of the extent approproiately; this will fail
 # on 1k block size filesystems without the correct fixes in place.
@@ -73,7 +73,10 @@ $XFS_IO_PROG -ft \
 	$LOOP_MNT/foo >> $seqres.full
 
 _unmount $LOOP_MNT
-_check_xfs_filesystem $LOOP_DEV none none
+_check_xfs_filesystem $loop_dev none none
+
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 # success, all done
 echo "Silence is golden"
tests/xfs/078 +9 -7
index 6b325e05f..0d3c2eb23 100755
@@ -17,7 +17,7 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_unmount $LOOP_MNT 2>/dev/null
-	[ -n "$LOOP_DEV" ] && _destroy_loop_device $LOOP_DEV 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev 2>/dev/null
 	# try to keep the image file if test fails
 	[ $status -eq 0 ] && rm -f $LOOP_IMG
 	rmdir $LOOP_MNT
@@ -55,7 +55,7 @@ _grow_loop()
 	agsize=$5
 
 	$XFS_IO_PROG -f -c "truncate $original" $LOOP_IMG
-	LOOP_DEV=`_create_loop_device $LOOP_IMG`
+	loop_dev=`_create_loop_device $LOOP_IMG`
 
 	dparam=""
 	if [ -n "$agsize" ]; then
@@ -67,15 +67,15 @@ _grow_loop()
 	echo
 
 	echo "*** mkfs loop file (size=$original)"
-	$MKFS_XFS_PROG -b size=$bsize $dparam $LOOP_DEV | \
+	$MKFS_XFS_PROG -b size=$bsize $dparam $loop_dev | \
 		_filter_mkfs 2>/dev/null
 
 	echo "*** extend loop file"
-	_destroy_loop_device $LOOP_DEV
+	_destroy_loop_device $loop_dev
 	$XFS_IO_PROG -c "pwrite $new_size $bsize" $LOOP_IMG | _filter_io
-	LOOP_DEV=`_create_loop_device $LOOP_IMG`
+	loop_dev=`_create_loop_device $LOOP_IMG`
 	echo "*** mount loop filesystem"
-	_mount -t xfs $LOOP_DEV $LOOP_MNT
+	_mount $loop_dev $LOOP_MNT
 
 	echo "*** grow loop filesystem"
 	$XFS_GROWFS_PROG $LOOP_MNT 2>&1 |  _filter_growfs 2>&1
@@ -87,9 +87,11 @@ _grow_loop()
 	if [ "$check" -gt "0" ]
 	then
 		echo "*** check"
-		_check_xfs_filesystem $LOOP_IMG none none
+		_check_xfs_filesystem $loop_dev none none
 	fi
 
+	_destroy_loop_device $loop_dev
+	unset loop_dev
 	rm -f $LOOP_IMG
 }
 
tests/xfs/148 +14 -9
index c42c9b119..4d2f7a808 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_unmount $mntpt > /dev/null 2>&1
-	_destroy_loop_device $loopdev > /dev/null 2>&1
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	rm -r -f $tmp.*
 }
 
@@ -48,12 +48,12 @@ rm -f $imgfile $imgfile.old
 # even when security xattrs are present so we are always doing name matches on
 # lookup and not name hash compares as leaf/node forms will do.
 $XFS_IO_PROG -f -c 'truncate 40m' $imgfile
-loopdev=$(_create_loop_device $imgfile)
-MKFS_OPTIONS="-m crc=0 -i size=512" _mkfs_dev $loopdev >> $seqres.full
+loop_dev=$(_create_loop_device $imgfile)
+MKFS_OPTIONS="-m crc=0 -i size=512" _mkfs_dev $loop_dev >> $seqres.full
 
 # Mount image file
 mkdir -p $mntpt
-_mount $loopdev $mntpt
+_mount $loop_dev $mntpt
 
 echo "creating entries" >> $seqres.full
 
@@ -91,7 +91,8 @@ cat $tmp.log | _filter_test_dir
 
 # Corrupt the entries
 _unmount $mntpt
-_destroy_loop_device $loopdev
+_destroy_loop_device $loop_dev
+unset loop_dev
 cp $imgfile $imgfile.old
 sed -b \
 	-e "s/$nullstr/too_many\x00beans/g" \
@@ -99,8 +100,9 @@ sed -b \
 	-i $imgfile
 test "$(md5sum < $imgfile)" != "$(md5sum < $imgfile.old)" ||
 	_fail "sed failed to change the image file?"
-loopdev=$(_create_loop_device $imgfile)
-_mount $loopdev $mntpt
+
+loop_dev=$(_create_loop_device $imgfile)
+_mount $loop_dev $mntpt
 
 # Try to access the corrupt metadata
 echo "++ ACCESSING BAD METADATA" | tee -a $seqres.full
@@ -111,7 +113,7 @@ cat $tmp.log | _filter_test_dir | sed -e '/Could not list/d'
 echo "does scrub complain?" >> $seqres.full
 
 # Does scrub complain about this?
-if _supports_xfs_scrub $mntpt $loopdev; then
+if _supports_xfs_scrub $mntpt $loop_dev; then
 	$XFS_SCRUB_PROG -n $mntpt >> $seqres.full 2>&1
 	res=$?
 	test $((res & 1)) -eq 0 && \
@@ -122,11 +124,14 @@ echo "does repair complain?" >> $seqres.full
 
 # Does repair complain about this?
 _unmount $mntpt
-$XFS_REPAIR_PROG -n $loopdev >> $seqres.full 2>&1
+$XFS_REPAIR_PROG -n $loop_dev >> $seqres.full 2>&1
 res=$?
 test $res -eq 1 || \
 	echo "repair failed to report corruption ($res)"
 
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/149 +4 -0
index f2187109b..9a96f82ed 100755
@@ -84,6 +84,10 @@ $XFS_GROWFS_PROG -D 16384 $loop_symlink > /dev/null
 echo "=== xfs_growfs - check device node ==="
 $XFS_GROWFS_PROG -D 20480 $loop_dev > /dev/null
 
+_unmount $mntdir
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/216 +10 -5
index 6b8b2eb22..091c11d08 100755
@@ -15,6 +15,7 @@ _begin_fstest log metadata auto quick
 _cleanup()
 {
 	_unmount $LOOP_MNT > /dev/null 2>&1
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
 }
@@ -24,7 +25,7 @@ _scratch_mkfs_xfs >/dev/null 2>&1
 _scratch_mount
 
 _require_loop
-LOOP_DEV=$SCRATCH_MNT/test_fs
+LOOP_IMG=$SCRATCH_MNT/test_fs
 LOOP_MNT=$SCRATCH_MNT/test_fs_dir
 
 loop_mkfs_opts=
@@ -55,22 +56,26 @@ _do_mkfs()
 	for i in $*; do
 		echo -n "fssize=${i}g "
 		$MKFS_XFS_PROG -f -b size=4096 -l version=2 \
-			-d name=$LOOP_DEV,size=${i}g $loop_mkfs_opts |grep log
-		_mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
+			-d size=${i}g $loop_mkfs_opts $loop_dev |grep log
+		_mount $loop_dev $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
 		_unmount $LOOP_MNT > /dev/null 2>&1
 	done
 }
 # make large holey file
-$XFS_IO_PROG -f -c "truncate 256g" $LOOP_DEV
+$XFS_IO_PROG -f -c "truncate 256g" $LOOP_IMG
 
-choose_golden_output $0 $LOOP_DEV
+choose_golden_output $0 $LOOP_IMG
 
 #make loopback mount dir
 mkdir $LOOP_MNT
 
+loop_dev=$(_create_loop_device $LOOP_IMG)
+
 # walk over standard sizes (up to 256GB)
 _do_mkfs 1 2 4 8 16 32 64 128 256
 
+_destroy_loop_device $loop_dev
+unset loop_dev
 status=0
 exit
tests/xfs/217 +16 -6
index b2eb34490..dae6ce55f 100755
@@ -12,6 +12,12 @@ _begin_fstest log metadata auto
 # Import common functions.
 . ./common/filter
 
+_cleanup()
+{
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
+	cd /
+	rm -f $tmp.*
+}
 
 _require_scratch
 _scratch_mkfs_xfs >/dev/null 2>&1
@@ -20,7 +26,7 @@ _scratch_mount
 _require_fs_space $SCRATCH_MNT 2202000
 
 _require_loop
-LOOP_DEV=$SCRATCH_MNT/test_fs
+LOOP_IMG=$SCRATCH_MNT/test_fs
 LOOP_MNT=$SCRATCH_MNT/test_fs_dir
 
 _do_mkfs()
@@ -28,28 +34,30 @@ _do_mkfs()
 	for i in $*; do
 		echo -n "fssize=${i}g "
 		$MKFS_XFS_PROG -f -b size=4096 -l version=2 \
-			-d name=$LOOP_DEV,size=${i}g |grep log
-		_mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
+			-d size=${i}g $loop_dev |grep log
+		_mount $loop_dev $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
 		_unmount $LOOP_MNT > /dev/null 2>&1
 
 		# punch out the previous blocks so that we keep the amount of
 		# disk space the test requires down to a minimum.
-		$XFS_IO_PROG -f -c "unresvsp 0 16383g" $LOOP_DEV
+		$XFS_IO_PROG -f -c "unresvsp 0 16383g" $LOOP_IMG
 	done
 }
 # make large holey file
-$XFS_IO_PROG -f -c "truncate 16383g" $LOOP_DEV
+$XFS_IO_PROG -f -c "truncate 16383g" $LOOP_IMG
 
 #make loopback mount dir
 mkdir $LOOP_MNT
 
 # test if large logs are supported
-$MKFS_XFS_PROG -f -l size=256m -d name=$LOOP_DEV,size=10g > /dev/null 2>&1
+$MKFS_XFS_PROG -f -l size=256m -d name=$LOOP_IMG,size=10g > /dev/null 2>&1
 if [ $? -ne 0 ]; then
 	_notrun "large log sizes not supported by mkfs"
 fi
 
+loop_dev=$(_create_loop_device $LOOP_IMG)
+
 #
 # walk over "new" sizes supported by recent xfsprogs.
 # Note that the last test is for 16TB-1GB as 32bit platforms only support
@@ -57,5 +65,7 @@ fi
 #
 _do_mkfs 512 1024 2048 4096 8192 16383
 
+_destroy_loop_device $loop_dev
+unset loop_dev
 status=0
 exit
tests/xfs/250 +9 -5
index 4e3473ebc..2554e1e91 100755
@@ -14,7 +14,8 @@ _cleanup()
 {
 	cd /
 	_unmount $LOOP_MNT 2>/dev/null
-	rm -f $LOOP_DEV
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
+	rm -f $LOOP_IMG
 	rmdir $LOOP_MNT
 	rm -f $tmp.*
 }
@@ -26,7 +27,7 @@ _require_test
 _require_loop
 _require_xfs_io_command "falloc"
 
-LOOP_DEV=$TEST_DIR/$seq.fs
+LOOP_IMG=$TEST_DIR/$seq.fs
 LOOP_MNT=$TEST_DIR/$seq.mnt
 
 _filter_io()
@@ -45,7 +46,7 @@ _test_loop()
 	agsize=$2
 	fsize=$3
 
-	dparam="file,name=$LOOP_DEV,size=$size"
+	dparam="file,name=$LOOP_IMG,size=$size"
 	if [ -n "$agsize" ]; then
 		dparam="$dparam,agsize=$agsize"
 	fi
@@ -55,7 +56,8 @@ _test_loop()
 		| _filter_mkfs 2>/dev/null
 
 	echo "*** mount loop filesystem"
-	_mount -t xfs -o loop $LOOP_DEV $LOOP_MNT
+	loop_dev=$(_create_loop_device $LOOP_IMG)
+	mount $loop_dev $LOOP_MNT
 
 	echo "*** preallocate large file"
 	$XFS_IO_PROG -f -c "resvsp 0 $fsize" $LOOP_MNT/foo | _filter_io
@@ -64,7 +66,9 @@ _test_loop()
 	_unmount $LOOP_MNT > /dev/null 2>&1
 
 	echo "*** check loop filesystem"
-	 _check_xfs_filesystem $LOOP_DEV none none
+	_check_xfs_filesystem $loop_dev none none
+	_destroy_loop_device $loop_dev
+	unset loop_dev
 }
 
 _test_loop 50g 16m 40G
tests/xfs/259 +8 -5
index 0c8d6eb56..c2d26381a 100755
@@ -12,7 +12,10 @@ _begin_fstest auto quick
 # Override the default cleanup function.
 _cleanup()
 {
-    rm -f "$testfile"
+	[ -n "$loop_dev" ] && _destroy_loop_device $testfile
+	rm -f "$testfile"
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -45,13 +48,13 @@ for del in $sizes_to_check; do
 		rm -f "$testfile"
 		dd if=/dev/zero "of=$testfile" bs=1 count=0 seek=$ddseek \
 			>/dev/null 2>&1 || echo "dd failed"
-		lofile=$(losetup -f)
-		losetup $lofile "$testfile"
-		$MKFS_XFS_PROG -l size=32m -b size=$bs $lofile |  _filter_mkfs \
+		loop_dev=$(_create_loop_device $testfile)
+		$MKFS_XFS_PROG -l size=32m -b size=$bs $loop_dev |  _filter_mkfs \
 			>/dev/null 2> $tmp.mkfs || echo "mkfs failed!"
 		. $tmp.mkfs
 		sync
-		losetup -d $lofile
+		_destroy_loop_device $loop_dev
+		unset loop_dev
 	done
 done
 
tests/xfs/513 +20 -24
index 5895e6e2d..0c0edc75e 100755
@@ -15,12 +15,8 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_unmount $LOOP_MNT 2>/dev/null
-	if [ -n "$LOOP_DEV" ];then
-		_destroy_loop_device $LOOP_DEV 2>/dev/null
-	fi
-	if [ -n "$LOOP_SPARE_DEV" ];then
-		_destroy_loop_device $LOOP_SPARE_DEV 2>/dev/null
-	fi
+	[ -n $loop_dev ] &&_destroy_loop_device $loop_dev
+	[ -n $loop_spare_dev ] &&_destroy_loop_device $loop_spare_dev
 	rm -f $LOOP_IMG
 	rm -f $LOOP_SPARE_IMG
 	rmdir $LOOP_MNT
@@ -42,11 +38,11 @@ LOOP_MNT=$TEST_DIR/$seq.mnt
 
 echo "** create loop device"
 $XFS_IO_PROG -f -c "truncate 32g" $LOOP_IMG
-LOOP_DEV=`_create_loop_device $LOOP_IMG`
+loop_dev=`_create_loop_device $LOOP_IMG`
 
 echo "** create loop log device"
 $XFS_IO_PROG -f -c "truncate 1g" $LOOP_SPARE_IMG
-LOOP_SPARE_DEV=`_create_loop_device $LOOP_SPARE_IMG`
+loop_spare_dev=`_create_loop_device $LOOP_SPARE_IMG`
 
 echo "** create loop mount point"
 rmdir $LOOP_MNT 2>/dev/null
@@ -55,8 +51,8 @@ mkdir -p $LOOP_MNT || _fail "cannot create loopback mount point"
 filter_loop()
 {
 	sed -e "s,\B$LOOP_MNT,LOOP_MNT,g" \
-	    -e "s,\B$LOOP_DEV,LOOP_DEV,g" \
-	    -e "s,\B$LOOP_SPARE_DEV,LOOP_SPARE_DEV,g"
+	    -e "s,\B$loop_dev,LOOP_DEV,g" \
+	    -e "s,\B$loop_spare_dev,LOOP_SPARE_DEV,g"
 }
 
 filter_xfs_opt()
@@ -69,22 +65,22 @@ MKFS_OPTIONS=""
 do_mkfs()
 {
 	echo "FORMAT: $@" | filter_loop | tee -a $seqres.full
-	$MKFS_XFS_PROG -f $* $LOOP_DEV | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
+	$MKFS_XFS_PROG -f $* $loop_dev | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
 	if [ "${PIPESTATUS[0]}" -ne 0 ]; then
-		_fail "Fails on _mkfs_dev $* $LOOP_DEV"
+		_fail "Fails on _mkfs_dev $* $loop_dev"
 	fi
 	. $tmp.mkfs
 }
 
 is_dev_mounted()
 {
-	findmnt --source $LOOP_DEV >/dev/null
+	findmnt --source $loop_dev >/dev/null
 	return $?
 }
 
 get_mount_info()
 {
-	findmnt --source $LOOP_DEV -o OPTIONS -n
+	findmnt --source $loop_dev -o OPTIONS -n
 }
 
 force_unmount()
@@ -103,29 +99,29 @@ _do_test()
 	local info
 
 	# mount test
-	_mount $LOOP_DEV $LOOP_MNT $opts 2>>$seqres.full
+	_mount $loop_dev $LOOP_MNT $opts 2>>$seqres.full
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "${mounted}" = "fail" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to fail, but it succeeded"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -ne 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs not mounted even mount return 0"
 			return 1
 		fi
 	else
 		if [ "${mounted}" = "pass" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to succeed, but it failed"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -eq 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs is mounted even mount return non-zero"
 			return 1
 		fi
@@ -141,13 +137,13 @@ _do_test()
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "$found" != "true" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expected to find \"$key\" in mount info \"$info\""
 			return 1
 		fi
 	else
 		if [ "$found" != "false" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: did not expect to find \"$key\" in \"$info\""
 			return 1
 		fi
@@ -253,9 +249,9 @@ do_test "-o logbsize=512k" fail
 # Test logdev
 do_mkfs
 do_test "" pass "logdev" "false"
-do_test "-o logdev=$LOOP_SPARE_DEV" fail
-do_mkfs -l logdev=$LOOP_SPARE_DEV
-do_test "-o logdev=$LOOP_SPARE_DEV" pass "logdev=$LOOP_SPARE_DEV" "true"
+do_test "-o logdev=$loop_spare_dev" fail
+do_mkfs -l logdev=$loop_spare_dev
+do_test "-o logdev=$loop_spare_dev" pass "logdev=$loop_spare_dev" "true"
 do_test "" fail
 
 # Test noalign
tests/xfs/521 +6 -2
index 13982c440..c92c621a2 100755
@@ -21,7 +21,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -35,7 +35,7 @@ _require_no_large_scratch_dev
 
 echo "Create fake rt volume"
 truncate -s 400m $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Format and mount 100m rt volume"
 export USE_EXTERNAL=yes
@@ -69,6 +69,10 @@ cp -p $testdir/original $testdir/copy3
 echo "Check filesystem"
 _check_scratch_fs
 
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/528 +7 -3
index 6ca9a2370..a1efbbd27 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -155,11 +155,11 @@ test_ops() {
 
 echo "Create fake rt volume"
 $XFS_IO_PROG -f -c "truncate 400m" $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Make sure synth rt volume works"
 export USE_EXTERNAL=yes
-export SCRATCH_RTDEV=$rtdev
+export SCRATCH_RTDEV=$rt_loop_dev
 _scratch_mkfs > $seqres.full
 _try_scratch_mount || \
 	_notrun "Could not mount with synthetic rt volume"
@@ -170,6 +170,10 @@ test_ops 262144
 # not a power of two
 test_ops 327680
 
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/530 +6 -5
index 8a182bd6a..d0d0e2665 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -52,12 +52,12 @@ fi
 
 rtdevsz=$((nr_bits * rtextsz))
 truncate -s $rtdevsz $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Format and mount rt volume"
 
 export USE_EXTERNAL=yes
-export SCRATCH_RTDEV=$rtdev
+export SCRATCH_RTDEV=$rt_loop_dev
 _scratch_mkfs -d size=$((1024 * 1024 * 1024)) \
 	      -r size=${rtextsz},extsize=${rtextsz} >> $seqres.full
 _try_scratch_mount || _notrun "Couldn't mount fs with synthetic rt volume"
@@ -116,8 +116,9 @@ done
 echo "Check filesystem"
 _check_scratch_fs
 
-losetup -d $rtdev
-rm -f $TEST_DIR/$seq.rtvol
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
 
 # success, all done
 status=0
tests/xfs/606 +8 -4
index f958bddd8..b537ea145 100755
@@ -13,10 +13,8 @@ _begin_fstest auto quick growfs
 
 _cleanup()
 {
-	local dev
 	_unmount $LOOP_MNT 2>/dev/null
-	dev=$(losetup -j testfile | cut -d: -f1)
-	losetup -d $dev 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	rm -rf $LOOP_IMG $LOOP_MNT
 	cd /
 	rm -f $tmp.*
@@ -40,7 +38,9 @@ $MKFS_XFS_PROG -f $LOOP_IMG >$seqres.full
 # Extend by just 8K, expected to start with the last full-size AG ends of
 # above 1G block device.
 $XFS_IO_PROG -f -c "truncate 1073750016" $LOOP_IMG
-_mount -oloop $LOOP_IMG $LOOP_MNT
+
+loop_dev=$(_create_loop_device $LOOP_IMG)
+_mount $loop_dev $LOOP_MNT
 # A known bug shows "XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on
 # device" at here, refer to _fixed_by_kernel_commit above
 $XFS_GROWFS_PROG $LOOP_MNT >$seqres.full
@@ -48,6 +48,10 @@ if [ $? -ne 0 ];then
 	echo "xfs_growfs fails!"
 fi
 
+_unmount $LOOP_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 echo "Silence is golden"
 # success, all done
 status=0
tests/xfs/613 +14 -26
index 6ba3d87bf..b87cae91e 100755
@@ -15,14 +15,8 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_unmount $LOOP_MNT 2>/dev/null
-	if [ -n "$LOOP_DEV" ];then
-		_destroy_loop_device $LOOP_DEV 2>/dev/null
-	fi
-	if [ -n "$LOOP_SPARE_DEV" ];then
-		_destroy_loop_device $LOOP_SPARE_DEV 2>/dev/null
-	fi
+	[ -n $loop_dev ] &&_destroy_loop_device $loop_dev
 	rm -f $LOOP_IMG
-	rm -f $LOOP_SPARE_IMG
 	rmdir $LOOP_MNT
 }
 
@@ -38,16 +32,11 @@ _require_loop
 _require_xfs_io_command "falloc"
 
 LOOP_IMG=$TEST_DIR/$seq.dev
-LOOP_SPARE_IMG=$TEST_DIR/$seq.logdev
 LOOP_MNT=$TEST_DIR/$seq.mnt
 
 echo "** create loop device"
 $XFS_IO_PROG -f -c "truncate 32g" $LOOP_IMG
-LOOP_DEV=`_create_loop_device $LOOP_IMG`
-
-echo "** create loop log device"
-$XFS_IO_PROG -f -c "truncate 1g" $LOOP_SPARE_IMG
-LOOP_SPARE_DEV=`_create_loop_device $LOOP_SPARE_IMG`
+loop_dev=`_create_loop_device $LOOP_IMG`
 
 echo "** create loop mount point"
 rmdir $LOOP_MNT 2>/dev/null
@@ -56,8 +45,7 @@ mkdir -p $LOOP_MNT || _fail "cannot create loopback mount point"
 filter_loop()
 {
 	sed -e "s,\B$LOOP_MNT,LOOP_MNT,g" \
-	    -e "s,\B$LOOP_DEV,LOOP_DEV,g" \
-	    -e "s,\B$LOOP_SPARE_DEV,LOOP_SPARE_DEV,g"
+	    -e "s,\B$loop_dev,LOOP_DEV,g"
 }
 
 filter_xfs_opt()
@@ -70,22 +58,22 @@ MKFS_OPTIONS=""
 do_mkfs()
 {
 	echo "FORMAT: $@" | filter_loop | tee -a $seqres.full
-	$MKFS_XFS_PROG -f $* $LOOP_DEV | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
+	$MKFS_XFS_PROG -f $* $loop_dev | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
 	if [ "${PIPESTATUS[0]}" -ne 0 ]; then
-		_fail "Fails on _mkfs_dev $* $LOOP_DEV"
+		_fail "Fails on _mkfs_dev $* $loop_dev"
 	fi
 	. $tmp.mkfs
 }
 
 is_dev_mounted()
 {
-	findmnt --source $LOOP_DEV >/dev/null
+	findmnt --source $loop_dev >/dev/null
 	return $?
 }
 
 get_mount_info()
 {
-	findmnt --source $LOOP_DEV -o OPTIONS -n
+	findmnt --source $loop_dev -o OPTIONS -n
 }
 
 force_unmount()
@@ -104,29 +92,29 @@ _do_test()
 	local info
 
 	# mount test
-	_mount $LOOP_DEV $LOOP_MNT $opts 2>>$seqres.full
+	_mount $loop_dev $LOOP_MNT $opts 2>>$seqres.full
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "${mounted}" = "fail" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to fail, but it succeeded"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -ne 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs not mounted even mount return 0"
 			return 1
 		fi
 	else
 		if [ "${mounted}" = "pass" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to succeed, but it failed"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -eq 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs is mounted even mount return non-zero"
 			return 1
 		fi
@@ -142,13 +130,13 @@ _do_test()
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "$found" != "true" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expected to find \"$key\" in mount info \"$info\""
 			return 1
 		fi
 	else
 		if [ "$found" != "false" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: did not expect to find \"$key\" in \"$info\""
 			return 1
 		fi
tests/xfs/613.out +0 -1
index 1624617ee..2a693c53c 100644
@@ -1,6 +1,5 @@
 QA output created by 613
 ** create loop device
-** create loop log device
 ** create loop mount point
 ** start xfs mount testing ...
 FORMAT: -m crc=0
Signature · 1 line
2.45.2
#24 lore raw

Re: [PATCH 13/40] fstests: clean up loop device instantiation

Quoted 84 lines · Zorro Lang
Quoted 70 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Lots of tests do there own special thing with loop devices rather than using _create_loop_device() and _destroy_loop_device(). This oftens means they do not clean up after themselves properly, leaving stale loop devices around that result in unmountable test or scratch devices. This is common when tests are killed by user interrupt.

Even the tests that do use _destroy_loop_device and try to clean up often do it incorrectly, leading to spurious error messages.

Some tests try to use dynamic instantiation via "mount -o loop", but then don't clean up in the correct order or hack around to find the loop device that was instantiated because the test needs to know the instantiated device name

Clean this up by converting all the tests to use _create_loop_device() and _destroy_loop_device(). In all the tests, use the variable "loop_dev" for the device consistently. In _destroy_loop_device(), test that a device name has been passed so that we don't try to clean up the same device twice (e.g. once before test exit and again from the _cleanup() function). When we destroy a loop device, unset the variable used to hold the loop device name so that we don't try to destroy it twice.

This results in much more reliable cleanup and clean exit from fstests when killed by the user.

---
 common/metadump   | 32 ++++++++++++--------------------
 tests/generic/067 | 11 ++++++++---
 tests/generic/361 |  6 +++++-
 tests/generic/563 | 14 +++++++++-----
 tests/generic/564 | 12 ++++++------
 tests/generic/590 |  9 ++++++---
 tests/generic/744 |  4 ++--
 tests/generic/746 |  6 +++++-
 tests/xfs/014     |  7 +++++--
 tests/xfs/049     | 35 +++++++++++++++++++++--------------
 tests/xfs/073     | 24 +++++++++++++-----------
 tests/xfs/074     | 21 ++++++++++++---------
 tests/xfs/078     | 16 +++++++++-------
 tests/xfs/148     | 23 ++++++++++++++---------
 tests/xfs/149     |  4 ++++
 tests/xfs/216     | 15 ++++++++++-----
 tests/xfs/217     | 22 ++++++++++++++++------
 tests/xfs/250     | 14 +++++++++-----
 tests/xfs/259     | 13 ++++++++-----
 tests/xfs/513     | 44 ++++++++++++++++++++------------------------
 tests/xfs/521     |  8 ++++++--
 tests/xfs/528     | 10 +++++++---
 tests/xfs/530     | 11 ++++++-----
 tests/xfs/606     | 12 ++++++++----
 tests/xfs/613     | 40 ++++++++++++++--------------------------
 tests/xfs/613.out |  1 -
 26 files changed, 235 insertions(+), 179 deletions(-)
common/metadump +0 -3
index 3373edfe9..bc3038e3b 100644
@@ -24,17 +24,9 @@ _xfs_cleanup_verify_metadump()
 
 	test -n "$XFS_METADUMP_FILE" && rm -f "$XFS_METADUMP_FILE"
 
-	if [ -n "$XFS_METADUMP_IMG" ]; then
-		losetup -n -a -O BACK-FILE,NAME | grep "^$XFS_METADUMP_IMG" | while read backing ldev; do
-			losetup -d "$ldev"

common/metadump _xfs_cleanup_verify_metadump() · line 28

Quoted 10 lines
common/metadump +2 -8
-		done
-
-		# Don't call rm directly with a globbed argument here to avoid
-		# issues issues with variable expansions.
-		for img in "$XFS_METADUMP_IMG"*; do
-			test -e "$img" && rm -f "$img"
-		done
-	fi
+	[ -n "$md_data_loop_dev" ] && _destroy_loop_device $md_data_loop_dev
+	[ -n "$md_log_loop_dev" ] && _destroy_loop_device $md_log_loop_dev

As Darrick reported that he always hit below error:

Quoted 7 lines

losetup: /dev/loop0: detach failed: No such device or address Cannot destroy loop device /dev/loop0

I'm wondering we should use [ -b ] not [ -n ] at here? Or output the stderr to /dev/null directly. Due to... (see below)

common/metadump _xfs_cleanup_verify_metadump() · line 29

Quoted 1 line
common/metadump +1 -0
+	rm -f $data_img $log_img
                ^^^^^^   ^^^^^^

Besides above problems, this line doesn't work either, due to the "data_img" and "log_img" are local variables in other functions. Even they're not local, think these variable names are too common to be global variable in common helper.

I'll try to change part of above new code back to old code as below [1]. Hope it can help the test work as usual, then we can merge this patch and move on. And we can improve this part in later patch.

Thanks,
Zorro

[1]

-       [ -n "$md_data_loop_dev" ] && _destroy_loop_device $md_data_loop_dev
-       [ -n "$md_log_loop_dev" ] && _destroy_loop_device $md_log_loop_dev
-       rm -f $data_img $log_img
+       if [ -n "$XFS_METADUMP_IMG" ]; then
+               [ -b "$md_data_loop_dev" ] && _destroy_loop_device $md_data_loop_dev
+               [ -b "$md_log_loop_dev" ] && _destroy_loop_device $md_log_loop_dev
+               for img in "$XFS_METADUMP_IMG"*; do

+ test -e "$img" && rm -f "$img"
+ done
+ fi

Quoted 1,249 lines
Quoted 15 lines
common/metadump +2 -2
 }
 
 # Can xfs_metadump snapshot the fs metadata to a v1 metadump file?
@@ -122,25 +114,25 @@ _xfs_verify_metadump_v2()
 		_scratch_xfs_mdrestore $metadump_file
 
 	# Create loopdev for data device so we can mount the fs
-	data_loop=$(_create_loop_device $data_img)
+	md_data_loop_dev=$(_create_loop_device $data_img)
 
 	# Create loopdev for log device if we recovered anything
-	test -s "$log_img" && log_loop=$(_create_loop_device $log_img)
+	test -s "$log_img" && md_log_loop_dev=$(_create_loop_device $log_img)
 
 	# Mount fs, run an extra test, fsck, and unmount

common/metadump _xfs_verify_metadump_v2() · line 128

Quoted 10 lines
common/metadump +4 -4
-	SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop _scratch_mount
+	SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev _scratch_mount
 	if [ -n "$extra_test" ]; then
-		SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop $extra_test
+		SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev $extra_test
 	fi
-	SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop _check_xfs_scratch_fs
-	SCRATCH_DEV=$data_loop _scratch_unmount
+	SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev _check_xfs_scratch_fs
+	_unmount $md_data_loop_dev

We umount loop devices we created above at here, and ...

common/metadump _xfs_verify_metadump_v2() · line 135

Quoted 10 lines
common/metadump +3 -3
 	# Tear down what we created
-	if [ -b "$log_loop" ]; then
-		_destroy_loop_device $log_loop
+	if [ -b "$md_log_loop_dev" ]; then
+		_destroy_loop_device $md_log_loop_dev
 		rm -f $log_img
 	fi
-	_destroy_loop_device $data_loop
+	_destroy_loop_device $md_data_loop_dev

we even has destroied them at here.

So that _xfs_cleanup_verify_metadump function hit "detach failed: No such device or address", due to those loop devices might have been released, but the variables are not unset.

I didn't give it a test, just did a code review. As Darrick uses this part mostly, could you please check what I said above?

Thanks,
Zorro

Quoted 1,198 lines
common/metadump +0 -0
 	rm -f $data_img
 }
 
tests/generic/067 +8 -3
index ccb1e3fbb..b45ae834f 100755
@@ -37,13 +37,18 @@ mount_nonexistent_mnt()
 	$MOUNT_PROG $SCRATCH_DEV $TEST_DIR/nosuchdir >>$seqres.full 2>&1
 }
 
-# fs driver should be able to handle mounting a free loop device gracefully
-# xfs ever hung, "ec53d1d xfs: don't block on buffer read errors" fixed it
+# fs driver should be able to handle mounting a free loop device gracefully xfs
+# ever hung, "ec53d1d xfs: don't block on buffer read errors" fixed it
+#
+# Using 'losetup -f' like this is racy. We might end up mounting a real loop dev
+# here, so unmount $SCRATCH_MNT (not the loop dev we might not own!) and ignore
+# any error it might return.
 mount_free_loopdev()
 {
 	echo "# mount a free loop device" >>$seqres.full
 	loopdev=`losetup -f`
-	$MOUNT_PROG -t $FSTYP $loopdev $SCRATCH_MNT >>$seqres.full 2>&1
+	_mount $loopdev $SCRATCH_MNT >>$seqres.full 2>&1
+	_unmount $SCRATCH_MNT >> /dev/null 2>&1
 }
 
 # mount with wrong fs type specified.
tests/generic/361 +5 -1
index 7273dd056..e2b798436 100755
@@ -17,7 +17,7 @@ _begin_fstest auto quick
 _cleanup()
 {
 	_unmount $fs_mnt
-	_destroy_loop_device $loop_dev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
 }
@@ -54,6 +54,10 @@ $XFS_IO_PROG -fc "pwrite 0 520m" $fs_mnt/testfile >>$seqres.full 2>&1
 # remount should not hang
 $MOUNT_PROG -o remount,ro $fs_mnt >>$seqres.full 2>&1
 
+_unmount $fs_mnt
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 echo "Silence is golden"
 status=0
tests/generic/563 +9 -5
index f0d2f404c..34d21170c 100755
@@ -87,12 +87,12 @@ reset()
 
 # cgroup I/O accounting doesn't work on partitions. Use a loop device to rule
 # that out.
-LOOP_DEV=$(_create_loop_device $SCRATCH_DEV)
-smajor=$((0x`stat -L -c %t $LOOP_DEV`))
-sminor=$((0x`stat -L -c %T $LOOP_DEV`))
+loop_dev=$(_create_loop_device $SCRATCH_DEV)
+smajor=$((0x`stat -L -c %t $loop_dev`))
+sminor=$((0x`stat -L -c %T $loop_dev`))
 
-_mkfs_dev $LOOP_DEV >> $seqres.full 2>&1
-_mount $LOOP_DEV $SCRATCH_MNT || _fail "mount failed"
+_mkfs_dev $loop_dev >> $seqres.full 2>&1
+_mount $loop_dev $SCRATCH_MNT || _fail "mount failed"
 
 blksize=$(_get_block_size "$SCRATCH_MNT")
 
@@ -147,6 +147,10 @@ if [ "$drop_io_cgroup" = 1 ]; then
 	echo "-io" > $cgdir/cgroup.subtree_control || _fail "subtree control"
 fi
 
+_unmount $SCRATCH_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/generic/564 +6 -6
index 647472d78..b66b72496 100755
@@ -19,7 +19,7 @@ _cleanup()
 {
 	cd /
 	rm -rf $tmp.*
-	[ -z "$loopdev" ] || _destroy_loop_device $loopdev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 }
 
 # Import common functions.
@@ -72,11 +72,11 @@ $XFS_IO_PROG -f -c "copy_range -l 32k $testdir" $testdir/copy
 echo
 echo source/destination as blkdev returns EINVAL
 $XFS_IO_PROG -f -c "truncate 128k" $testdir/img >> $seqres.full 2>&1
-loopdev=`_create_loop_device $testdir/img`
-$XFS_IO_PROG -c "copy_range -l 32k $testdir/file" $loopdev
-$XFS_IO_PROG -f -c "copy_range -l 32k $loopdev" $testdir/copy
-_destroy_loop_device $loopdev
-loopdev=
+loop_dev=`_create_loop_device $testdir/img`
+$XFS_IO_PROG -c "copy_range -l 32k $testdir/file" $loop_dev
+$XFS_IO_PROG -f -c "copy_range -l 32k $loop_dev" $testdir/copy
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo
 echo source/destination as chardev returns EINVAL
tests/generic/590 +6 -3
index 2b7ccfb53..2eefa2c62 100755
@@ -15,9 +15,10 @@ _begin_fstest auto prealloc preallocrw
 # Override the default cleanup function.
 _cleanup()
 {
+	_scratch_unmount
+	[ -n $loop_dev ] && _destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
-	test -n "$loop" && _destroy_loop_device "$loop"
 	rm -f "$TEST_DIR/$seq"
 }
 
@@ -56,9 +57,9 @@ if [[ $FSTYP = xfs ]]; then
 		loopsz="$((filesz + (1 << 26)))"
 		_require_fs_space "$TEST_DIR" $((loopsz / 1024))
 		$XFS_IO_PROG -c "truncate $loopsz" -f "$TEST_DIR/$seq"
-		loop="$(_create_loop_device "$TEST_DIR/$seq")"
+		loop_dev="$(_create_loop_device "$TEST_DIR/$seq")"
 		USE_EXTERNAL=yes
-		SCRATCH_RTDEV="$loop"
+		SCRATCH_RTDEV="$loop_dev"
 		disabled_features=()
 
 		# disable reflink if not supported by realtime devices
@@ -114,6 +115,8 @@ $XFS_IO_PROG -c "truncate 0" -c fsync "$SCRATCH_MNT/file"
 # We need to do this before the loop device gets torn down.
 _scratch_unmount
 _check_scratch_fs
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo "Silence is golden"
 status=0
tests/generic/744 +2 -2
index df8f6ae9b..cda10e0f6 100755
@@ -17,8 +17,8 @@ _cleanup()
 
 	_unmount $mnt2 &> /dev/null
 	_unmount $mnt1 &> /dev/null
-	[ -b "$loop_dev2" ] && losetup -d $loop_dev2
-	[ -b "$loop_dev1" ] && losetup -d $loop_dev1
+	[ -b "$loop_dev2" ] && _destroy_loop_device $loop_dev2
+	[ -b "$loop_dev1" ] && _destroy_loop_device $loop_dev1
 	[ -n "$seq" ] && rm -rf $TEST_DIR/$seq
 }
 
tests/generic/746 +5 -1
index 671910606..ba8ed25e8 100755
@@ -39,7 +39,7 @@ esac
 _cleanup()
 {
 	_unmount $loop_mnt &> /dev/null
-	_destroy_loop_device $loop_dev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	if [ $status -eq 0 ]; then
 		rm -rf $tmp
 		rm $img_file
@@ -223,5 +223,9 @@ while read line; do
 done < $fiemap_after
 echo "done."
 
+_unmount $loop_mnt
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 status=0
 exit
tests/xfs/014 +5 -2
index f638e4b16..39ea40e2a 100755
@@ -23,6 +23,7 @@ _cleanup()
 {
 	cd /
 	_unmount $LOOP_MNT 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	_scratch_unmount 2>/dev/null
 	rm -f $tmp.*
 }
@@ -166,19 +167,21 @@ LOOP_FILE=$SCRATCH_MNT/$seq.fs
 LOOP_MNT=$SCRATCH_MNT/$seq.mnt
 
 $MKFS_XFS_PROG -d "file=1,name=$LOOP_FILE,size=10g" >> $seqres.full 2>&1
+loop_dev=$(_create_loop_device $LOOP_FILE)
 
 mkdir -p $LOOP_MNT
-_mount -o uquota,gquota $LOOP_FILE $LOOP_MNT || \
+_mount -o uquota,gquota $loop_dev $LOOP_MNT || \
 	_fail "Failed to mount loop fs."
 
 _test_enospc $LOOP_MNT
 _test_edquot $LOOP_MNT
 
 _unmount $LOOP_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo $orig_sp_time > /proc/sys/fs/xfs/speculative_prealloc_lifetime
 
 _scratch_unmount
-
 status=0
 exit
tests/xfs/049 +21 -14
index 4163a144f..cdcddf764 100755
@@ -12,16 +12,17 @@ _begin_fstest rw auto quick
 # Override the default cleanup function.
 _cleanup()
 {
-    cd /
-    umount $SCRATCH_MNT/test2 > /dev/null 2>&1
-    umount $SCRATCH_MNT/test > /dev/null 2>&1
-    rm -f $tmp.*
+	cd /
+	_unmount $SCRATCH_MNT/test2 > /dev/null 2>&1
+	_unmount $SCRATCH_MNT/test > /dev/null 2>&1
+	[ -n "$loop_dev2" ] && _destroy_loop_device $loop_dev2
+	[ -n "$loop_dev1" ] && _destroy_loop_device $loop_dev1
+	rm -f $tmp.*
 
-    if [ -w $seqres.full ]
-    then
-        echo "--- mounts at end (after cleanup)" >> $seqres.full
-        mount >> $seqres.full
-    fi
+	if [ -w $seqres.full ]; then
+		echo "--- mounts at end (after cleanup)" >> $seqres.full
+		mount >> $seqres.full
+	fi
 }
 
 # Import common functions.
@@ -64,7 +65,8 @@ mkdir $SCRATCH_MNT/test $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to make mount points"
 
 _log "Mount xfs via loop"
-mount -t xfs -o loop $SCRATCH_MNT/test.xfs $SCRATCH_MNT/test >> $seqres.full 2>&1 \
+loop_dev1=$(_create_loop_device $SCRATCH_MNT/test.xfs)
+_mount $loop_dev1 $SCRATCH_MNT/test >> $seqres.full 2>&1 \
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress"
@@ -80,11 +82,12 @@ dd if=/dev/zero of=$SCRATCH_MNT/test/test.ext2 bs=1024 count=10240 >> $seqres.fu
     || _fail "!!! create file failed"
 
 _log "Create ext2 fs in file on looped xfs"
-echo y | mkfs -t ext2 $SCRATCH_MNT/test/test.ext2 >> $seqres.full 2>&1 \
+loop_dev2=$(_create_loop_device $SCRATCH_MNT/test/test.ext2)
+echo y | mkfs -t ext2 $loop_dev2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to mkfs ext2 on xfs"
 
 _log "Mount ext2 on xfs via loop"
-mount -t ext2 -o loop $SCRATCH_MNT/test/test.ext2 $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
+_mount $loop_dev2 $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress ext2 on xfs via loop"
@@ -96,12 +99,16 @@ rm -rf $SCRATCH_MNT/test/* >> $seqres.full 2>&1 \
     || _fail "!!! clean failed"
 
 _log "umount ext2 on xfs"
-umount $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
+_unmount $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! umount ext2 failed"
+_destroy_loop_device $loop_dev2
+unset loop_dev2
 
 _log "umount xfs"
-umount $SCRATCH_MNT/test >> $seqres.full 2>&1 \
+_unmount $SCRATCH_MNT/test >> $seqres.full 2>&1 \
     || _fail "!!! umount xfs failed"
+_destroy_loop_device $loop_dev1
+unset loop_dev1
 
 echo "--- mounts at end (before cleanup)" >> $seqres.full
 mount >> $seqres.full
tests/xfs/073 +13 -11
index e4b17c5e7..2274079ef 100755
@@ -23,6 +23,8 @@ _cleanup()
 	_scratch_unmount 2>/dev/null
 	_unmount $imgs.loop 2>/dev/null
 	_unmount $imgs.source_dir 2>/dev/null
+	[ -n "$loop_dev1" ] && _destroy_loop_device $loop_dev1
+	[ -n "$loop_dev2" ] && _destroy_loop_device $loop_dev2
 	[ -d $imgs.loop ] && rmdir $imgs.loop
 	[ -d $imgs.source_dir ] && rm -rf $imgs.source_dir
 	rm -f $imgs.* $tmp.* /var/tmp/xfs_copy.log.*
@@ -65,10 +67,11 @@ _verify_copy()
 	rmdir $target_dir 2>/dev/null
 	mkdir $target_dir
 
-	_mount -t xfs -o loop $target $target_dir 2>/dev/null
+	loop_dev1=$(_create_loop_device $target)
+	_mount $loop_dev1 $target_dir 2>/dev/null
 	if [ $? -ne 0 ]; then
 		echo retrying mount with nouuid option >>$seqres.full
-		_mount -t xfs -o loop -o nouuid $target $target_dir
+		_mount -o nouuid $loop_dev1 $target_dir
 		if [ $? -ne 0 ]; then
 			echo mount failed - evil!
 			return
@@ -100,6 +103,8 @@ _verify_copy()
 	echo unmounting and removing new image
 	_unmount $source_dir
 	_unmount $target_dir > /dev/null 2>&1
+	_destroy_loop_device $loop_dev1
+	unset loop_dev1
 	rm -f $target
 }
 
@@ -134,18 +139,15 @@ ${MKFS_XFS_PROG} -dfile,name=$imgs.source,size=100g \
 rmdir $imgs.source_dir 2>/dev/null
 mkdir $imgs.source_dir
 
-_mount -t xfs -o loop $imgs.source $imgs.source_dir
-loop2=`mount | grep $imgs.source | grep -o -e 'loop=.*[^),]' | grep -o -e '/.*$'`
+loop_dev2=$(_create_loop_device $imgs.source)
+_mount $loop_dev2 $imgs.source_dir
 cp -a $here $imgs.source_dir
-_mount -t xfs -o remount,ro $imgs.source $imgs.source_dir
-$XFS_COPY_PROG $imgs.source $imgs.image | _filter_copy '#' $imgs.image '#' '#'
+_mount -o remount,ro $loop_dev2 $imgs.source_dir
+$XFS_COPY_PROG $loop_dev2 $imgs.image 2> /dev/null | _filter_copy '#' $imgs.image '#' '#'
 _verify_copy $imgs.image $imgs.source $imgs.source_dir
 
-# HACK WARNING:
-#
-# We're done with the nested loop mount, now we have to clean
-# up the pieces that mount is incapable of doing.
-losetup -d $loop2 > /dev/null 2>&1
+_destroy_loop_device $loop_dev2
+unset loop_dev2
 
 echo
 echo === copying scratch device to multiple targets
tests/xfs/074 +12 -9
index 6a03c8111..5df864fad 100755
@@ -26,7 +26,7 @@ _begin_fstest quick auto prealloc rw
 _cleanup()
 {
 	cd /
-	_destroy_loop_device $LOOP_DEV
+	[ -n "$loop_dev" ] &&_destroy_loop_device $loop_dev
 	rm -f $tmp.* $LOOP_FILE
 }
 
@@ -45,10 +45,10 @@ LOOP_FILE=$TEST_DIR/$seq.img
 LOOP_MNT=$TEST_DIR/$seq.mnt
 mkdir -p $LOOP_MNT
 $XFS_IO_PROG -ft -c "truncate 1t" $LOOP_FILE >> $seqres.full
-LOOP_DEV=`_create_loop_device $LOOP_FILE`
+loop_dev=`_create_loop_device $LOOP_FILE`
 
-_mkfs_dev -d size=260g,agcount=2 $LOOP_DEV
-_mount $LOOP_DEV $LOOP_MNT
+_mkfs_dev -d size=260g,agcount=2 $loop_dev
+_mount $loop_dev $LOOP_MNT
 
 BLOCK_SIZE=$(_get_file_block_size $LOOP_MNT)
 
@@ -59,11 +59,11 @@ $XFS_IO_PROG -ft \
 	-c "falloc 0 $(($BLOCK_SIZE * 2097152))" \
 	$LOOP_MNT/foo >> $seqres.full
 
-umount $LOOP_MNT
-_check_xfs_filesystem $LOOP_DEV none none
+_unmount $LOOP_MNT
+_check_xfs_filesystem $loop_dev none none
 
-_mkfs_dev -f $LOOP_DEV
-_mount $LOOP_DEV $LOOP_MNT
+_mkfs_dev -f $loop_dev
+_mount $loop_dev $LOOP_MNT
 
 # check we trim both ends of the extent approproiately; this will fail
 # on 1k block size filesystems without the correct fixes in place.
@@ -73,7 +73,10 @@ $XFS_IO_PROG -ft \
 	$LOOP_MNT/foo >> $seqres.full
 
 _unmount $LOOP_MNT
-_check_xfs_filesystem $LOOP_DEV none none
+_check_xfs_filesystem $loop_dev none none
+
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 # success, all done
 echo "Silence is golden"
tests/xfs/078 +9 -7
index 6b325e05f..0d3c2eb23 100755
@@ -17,7 +17,7 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_unmount $LOOP_MNT 2>/dev/null
-	[ -n "$LOOP_DEV" ] && _destroy_loop_device $LOOP_DEV 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev 2>/dev/null
 	# try to keep the image file if test fails
 	[ $status -eq 0 ] && rm -f $LOOP_IMG
 	rmdir $LOOP_MNT
@@ -55,7 +55,7 @@ _grow_loop()
 	agsize=$5
 
 	$XFS_IO_PROG -f -c "truncate $original" $LOOP_IMG
-	LOOP_DEV=`_create_loop_device $LOOP_IMG`
+	loop_dev=`_create_loop_device $LOOP_IMG`
 
 	dparam=""
 	if [ -n "$agsize" ]; then
@@ -67,15 +67,15 @@ _grow_loop()
 	echo
 
 	echo "*** mkfs loop file (size=$original)"
-	$MKFS_XFS_PROG -b size=$bsize $dparam $LOOP_DEV | \
+	$MKFS_XFS_PROG -b size=$bsize $dparam $loop_dev | \
 		_filter_mkfs 2>/dev/null
 
 	echo "*** extend loop file"
-	_destroy_loop_device $LOOP_DEV
+	_destroy_loop_device $loop_dev
 	$XFS_IO_PROG -c "pwrite $new_size $bsize" $LOOP_IMG | _filter_io
-	LOOP_DEV=`_create_loop_device $LOOP_IMG`
+	loop_dev=`_create_loop_device $LOOP_IMG`
 	echo "*** mount loop filesystem"
-	_mount -t xfs $LOOP_DEV $LOOP_MNT
+	_mount $loop_dev $LOOP_MNT
 
 	echo "*** grow loop filesystem"
 	$XFS_GROWFS_PROG $LOOP_MNT 2>&1 |  _filter_growfs 2>&1
@@ -87,9 +87,11 @@ _grow_loop()
 	if [ "$check" -gt "0" ]
 	then
 		echo "*** check"
-		_check_xfs_filesystem $LOOP_IMG none none
+		_check_xfs_filesystem $loop_dev none none
 	fi
 
+	_destroy_loop_device $loop_dev
+	unset loop_dev
 	rm -f $LOOP_IMG
 }
 
tests/xfs/148 +14 -9
index c42c9b119..4d2f7a808 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_unmount $mntpt > /dev/null 2>&1
-	_destroy_loop_device $loopdev > /dev/null 2>&1
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	rm -r -f $tmp.*
 }
 
@@ -48,12 +48,12 @@ rm -f $imgfile $imgfile.old
 # even when security xattrs are present so we are always doing name matches on
 # lookup and not name hash compares as leaf/node forms will do.
 $XFS_IO_PROG -f -c 'truncate 40m' $imgfile
-loopdev=$(_create_loop_device $imgfile)
-MKFS_OPTIONS="-m crc=0 -i size=512" _mkfs_dev $loopdev >> $seqres.full
+loop_dev=$(_create_loop_device $imgfile)
+MKFS_OPTIONS="-m crc=0 -i size=512" _mkfs_dev $loop_dev >> $seqres.full
 
 # Mount image file
 mkdir -p $mntpt
-_mount $loopdev $mntpt
+_mount $loop_dev $mntpt
 
 echo "creating entries" >> $seqres.full
 
@@ -91,7 +91,8 @@ cat $tmp.log | _filter_test_dir
 
 # Corrupt the entries
 _unmount $mntpt
-_destroy_loop_device $loopdev
+_destroy_loop_device $loop_dev
+unset loop_dev
 cp $imgfile $imgfile.old
 sed -b \
 	-e "s/$nullstr/too_many\x00beans/g" \
@@ -99,8 +100,9 @@ sed -b \
 	-i $imgfile
 test "$(md5sum < $imgfile)" != "$(md5sum < $imgfile.old)" ||
 	_fail "sed failed to change the image file?"
-loopdev=$(_create_loop_device $imgfile)
-_mount $loopdev $mntpt
+
+loop_dev=$(_create_loop_device $imgfile)
+_mount $loop_dev $mntpt
 
 # Try to access the corrupt metadata
 echo "++ ACCESSING BAD METADATA" | tee -a $seqres.full
@@ -111,7 +113,7 @@ cat $tmp.log | _filter_test_dir | sed -e '/Could not list/d'
 echo "does scrub complain?" >> $seqres.full
 
 # Does scrub complain about this?
-if _supports_xfs_scrub $mntpt $loopdev; then
+if _supports_xfs_scrub $mntpt $loop_dev; then
 	$XFS_SCRUB_PROG -n $mntpt >> $seqres.full 2>&1
 	res=$?
 	test $((res & 1)) -eq 0 && \
@@ -122,11 +124,14 @@ echo "does repair complain?" >> $seqres.full
 
 # Does repair complain about this?
 _unmount $mntpt
-$XFS_REPAIR_PROG -n $loopdev >> $seqres.full 2>&1
+$XFS_REPAIR_PROG -n $loop_dev >> $seqres.full 2>&1
 res=$?
 test $res -eq 1 || \
 	echo "repair failed to report corruption ($res)"
 
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/149 +4 -0
index f2187109b..9a96f82ed 100755
@@ -84,6 +84,10 @@ $XFS_GROWFS_PROG -D 16384 $loop_symlink > /dev/null
 echo "=== xfs_growfs - check device node ==="
 $XFS_GROWFS_PROG -D 20480 $loop_dev > /dev/null
 
+_unmount $mntdir
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/216 +10 -5
index 6b8b2eb22..091c11d08 100755
@@ -15,6 +15,7 @@ _begin_fstest log metadata auto quick
 _cleanup()
 {
 	_unmount $LOOP_MNT > /dev/null 2>&1
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
 }
@@ -24,7 +25,7 @@ _scratch_mkfs_xfs >/dev/null 2>&1
 _scratch_mount
 
 _require_loop
-LOOP_DEV=$SCRATCH_MNT/test_fs
+LOOP_IMG=$SCRATCH_MNT/test_fs
 LOOP_MNT=$SCRATCH_MNT/test_fs_dir
 
 loop_mkfs_opts=
@@ -55,22 +56,26 @@ _do_mkfs()
 	for i in $*; do
 		echo -n "fssize=${i}g "
 		$MKFS_XFS_PROG -f -b size=4096 -l version=2 \
-			-d name=$LOOP_DEV,size=${i}g $loop_mkfs_opts |grep log
-		_mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
+			-d size=${i}g $loop_mkfs_opts $loop_dev |grep log
+		_mount $loop_dev $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
 		_unmount $LOOP_MNT > /dev/null 2>&1
 	done
 }
 # make large holey file
-$XFS_IO_PROG -f -c "truncate 256g" $LOOP_DEV
+$XFS_IO_PROG -f -c "truncate 256g" $LOOP_IMG
 
-choose_golden_output $0 $LOOP_DEV
+choose_golden_output $0 $LOOP_IMG
 
 #make loopback mount dir
 mkdir $LOOP_MNT
 
+loop_dev=$(_create_loop_device $LOOP_IMG)
+
 # walk over standard sizes (up to 256GB)
 _do_mkfs 1 2 4 8 16 32 64 128 256
 
+_destroy_loop_device $loop_dev
+unset loop_dev
 status=0
 exit
tests/xfs/217 +16 -6
index b2eb34490..dae6ce55f 100755
@@ -12,6 +12,12 @@ _begin_fstest log metadata auto
 # Import common functions.
 . ./common/filter
 
+_cleanup()
+{
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
+	cd /
+	rm -f $tmp.*
+}
 
 _require_scratch
 _scratch_mkfs_xfs >/dev/null 2>&1
@@ -20,7 +26,7 @@ _scratch_mount
 _require_fs_space $SCRATCH_MNT 2202000
 
 _require_loop
-LOOP_DEV=$SCRATCH_MNT/test_fs
+LOOP_IMG=$SCRATCH_MNT/test_fs
 LOOP_MNT=$SCRATCH_MNT/test_fs_dir
 
 _do_mkfs()
@@ -28,28 +34,30 @@ _do_mkfs()
 	for i in $*; do
 		echo -n "fssize=${i}g "
 		$MKFS_XFS_PROG -f -b size=4096 -l version=2 \
-			-d name=$LOOP_DEV,size=${i}g |grep log
-		_mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
+			-d size=${i}g $loop_dev |grep log
+		_mount $loop_dev $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
 		_unmount $LOOP_MNT > /dev/null 2>&1
 
 		# punch out the previous blocks so that we keep the amount of
 		# disk space the test requires down to a minimum.
-		$XFS_IO_PROG -f -c "unresvsp 0 16383g" $LOOP_DEV
+		$XFS_IO_PROG -f -c "unresvsp 0 16383g" $LOOP_IMG
 	done
 }
 # make large holey file
-$XFS_IO_PROG -f -c "truncate 16383g" $LOOP_DEV
+$XFS_IO_PROG -f -c "truncate 16383g" $LOOP_IMG
 
 #make loopback mount dir
 mkdir $LOOP_MNT
 
 # test if large logs are supported
-$MKFS_XFS_PROG -f -l size=256m -d name=$LOOP_DEV,size=10g > /dev/null 2>&1
+$MKFS_XFS_PROG -f -l size=256m -d name=$LOOP_IMG,size=10g > /dev/null 2>&1
 if [ $? -ne 0 ]; then
 	_notrun "large log sizes not supported by mkfs"
 fi
 
+loop_dev=$(_create_loop_device $LOOP_IMG)
+
 #
 # walk over "new" sizes supported by recent xfsprogs.
 # Note that the last test is for 16TB-1GB as 32bit platforms only support
@@ -57,5 +65,7 @@ fi
 #
 _do_mkfs 512 1024 2048 4096 8192 16383
 
+_destroy_loop_device $loop_dev
+unset loop_dev
 status=0
 exit
tests/xfs/250 +9 -5
index 4e3473ebc..2554e1e91 100755
@@ -14,7 +14,8 @@ _cleanup()
 {
 	cd /
 	_unmount $LOOP_MNT 2>/dev/null
-	rm -f $LOOP_DEV
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
+	rm -f $LOOP_IMG
 	rmdir $LOOP_MNT
 	rm -f $tmp.*
 }
@@ -26,7 +27,7 @@ _require_test
 _require_loop
 _require_xfs_io_command "falloc"
 
-LOOP_DEV=$TEST_DIR/$seq.fs
+LOOP_IMG=$TEST_DIR/$seq.fs
 LOOP_MNT=$TEST_DIR/$seq.mnt
 
 _filter_io()
@@ -45,7 +46,7 @@ _test_loop()
 	agsize=$2
 	fsize=$3
 
-	dparam="file,name=$LOOP_DEV,size=$size"
+	dparam="file,name=$LOOP_IMG,size=$size"
 	if [ -n "$agsize" ]; then
 		dparam="$dparam,agsize=$agsize"
 	fi
@@ -55,7 +56,8 @@ _test_loop()
 		| _filter_mkfs 2>/dev/null
 
 	echo "*** mount loop filesystem"
-	_mount -t xfs -o loop $LOOP_DEV $LOOP_MNT
+	loop_dev=$(_create_loop_device $LOOP_IMG)
+	mount $loop_dev $LOOP_MNT
 
 	echo "*** preallocate large file"
 	$XFS_IO_PROG -f -c "resvsp 0 $fsize" $LOOP_MNT/foo | _filter_io
@@ -64,7 +66,9 @@ _test_loop()
 	_unmount $LOOP_MNT > /dev/null 2>&1
 
 	echo "*** check loop filesystem"
-	 _check_xfs_filesystem $LOOP_DEV none none
+	_check_xfs_filesystem $loop_dev none none
+	_destroy_loop_device $loop_dev
+	unset loop_dev
 }
 
 _test_loop 50g 16m 40G
tests/xfs/259 +8 -5
index 0c8d6eb56..c2d26381a 100755
@@ -12,7 +12,10 @@ _begin_fstest auto quick
 # Override the default cleanup function.
 _cleanup()
 {
-    rm -f "$testfile"
+	[ -n "$loop_dev" ] && _destroy_loop_device $testfile
+	rm -f "$testfile"
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -45,13 +48,13 @@ for del in $sizes_to_check; do
 		rm -f "$testfile"
 		dd if=/dev/zero "of=$testfile" bs=1 count=0 seek=$ddseek \
 			>/dev/null 2>&1 || echo "dd failed"
-		lofile=$(losetup -f)
-		losetup $lofile "$testfile"
-		$MKFS_XFS_PROG -l size=32m -b size=$bs $lofile |  _filter_mkfs \
+		loop_dev=$(_create_loop_device $testfile)
+		$MKFS_XFS_PROG -l size=32m -b size=$bs $loop_dev |  _filter_mkfs \
 			>/dev/null 2> $tmp.mkfs || echo "mkfs failed!"
 		. $tmp.mkfs
 		sync
-		losetup -d $lofile
+		_destroy_loop_device $loop_dev
+		unset loop_dev
 	done
 done
 
tests/xfs/513 +20 -24
index 5895e6e2d..0c0edc75e 100755
@@ -15,12 +15,8 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_unmount $LOOP_MNT 2>/dev/null
-	if [ -n "$LOOP_DEV" ];then
-		_destroy_loop_device $LOOP_DEV 2>/dev/null
-	fi
-	if [ -n "$LOOP_SPARE_DEV" ];then
-		_destroy_loop_device $LOOP_SPARE_DEV 2>/dev/null
-	fi
+	[ -n $loop_dev ] &&_destroy_loop_device $loop_dev
+	[ -n $loop_spare_dev ] &&_destroy_loop_device $loop_spare_dev
 	rm -f $LOOP_IMG
 	rm -f $LOOP_SPARE_IMG
 	rmdir $LOOP_MNT
@@ -42,11 +38,11 @@ LOOP_MNT=$TEST_DIR/$seq.mnt
 
 echo "** create loop device"
 $XFS_IO_PROG -f -c "truncate 32g" $LOOP_IMG
-LOOP_DEV=`_create_loop_device $LOOP_IMG`
+loop_dev=`_create_loop_device $LOOP_IMG`
 
 echo "** create loop log device"
 $XFS_IO_PROG -f -c "truncate 1g" $LOOP_SPARE_IMG
-LOOP_SPARE_DEV=`_create_loop_device $LOOP_SPARE_IMG`
+loop_spare_dev=`_create_loop_device $LOOP_SPARE_IMG`
 
 echo "** create loop mount point"
 rmdir $LOOP_MNT 2>/dev/null
@@ -55,8 +51,8 @@ mkdir -p $LOOP_MNT || _fail "cannot create loopback mount point"
 filter_loop()
 {
 	sed -e "s,\B$LOOP_MNT,LOOP_MNT,g" \
-	    -e "s,\B$LOOP_DEV,LOOP_DEV,g" \
-	    -e "s,\B$LOOP_SPARE_DEV,LOOP_SPARE_DEV,g"
+	    -e "s,\B$loop_dev,LOOP_DEV,g" \
+	    -e "s,\B$loop_spare_dev,LOOP_SPARE_DEV,g"
 }
 
 filter_xfs_opt()
@@ -69,22 +65,22 @@ MKFS_OPTIONS=""
 do_mkfs()
 {
 	echo "FORMAT: $@" | filter_loop | tee -a $seqres.full
-	$MKFS_XFS_PROG -f $* $LOOP_DEV | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
+	$MKFS_XFS_PROG -f $* $loop_dev | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
 	if [ "${PIPESTATUS[0]}" -ne 0 ]; then
-		_fail "Fails on _mkfs_dev $* $LOOP_DEV"
+		_fail "Fails on _mkfs_dev $* $loop_dev"
 	fi
 	. $tmp.mkfs
 }
 
 is_dev_mounted()
 {
-	findmnt --source $LOOP_DEV >/dev/null
+	findmnt --source $loop_dev >/dev/null
 	return $?
 }
 
 get_mount_info()
 {
-	findmnt --source $LOOP_DEV -o OPTIONS -n
+	findmnt --source $loop_dev -o OPTIONS -n
 }
 
 force_unmount()
@@ -103,29 +99,29 @@ _do_test()
 	local info
 
 	# mount test
-	_mount $LOOP_DEV $LOOP_MNT $opts 2>>$seqres.full
+	_mount $loop_dev $LOOP_MNT $opts 2>>$seqres.full
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "${mounted}" = "fail" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to fail, but it succeeded"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -ne 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs not mounted even mount return 0"
 			return 1
 		fi
 	else
 		if [ "${mounted}" = "pass" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to succeed, but it failed"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -eq 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs is mounted even mount return non-zero"
 			return 1
 		fi
@@ -141,13 +137,13 @@ _do_test()
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "$found" != "true" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expected to find \"$key\" in mount info \"$info\""
 			return 1
 		fi
 	else
 		if [ "$found" != "false" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: did not expect to find \"$key\" in \"$info\""
 			return 1
 		fi
@@ -253,9 +249,9 @@ do_test "-o logbsize=512k" fail
 # Test logdev
 do_mkfs
 do_test "" pass "logdev" "false"
-do_test "-o logdev=$LOOP_SPARE_DEV" fail
-do_mkfs -l logdev=$LOOP_SPARE_DEV
-do_test "-o logdev=$LOOP_SPARE_DEV" pass "logdev=$LOOP_SPARE_DEV" "true"
+do_test "-o logdev=$loop_spare_dev" fail
+do_mkfs -l logdev=$loop_spare_dev
+do_test "-o logdev=$loop_spare_dev" pass "logdev=$loop_spare_dev" "true"
 do_test "" fail
 
 # Test noalign
tests/xfs/521 +6 -2
index 13982c440..c92c621a2 100755
@@ -21,7 +21,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -35,7 +35,7 @@ _require_no_large_scratch_dev
 
 echo "Create fake rt volume"
 truncate -s 400m $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Format and mount 100m rt volume"
 export USE_EXTERNAL=yes
@@ -69,6 +69,10 @@ cp -p $testdir/original $testdir/copy3
 echo "Check filesystem"
 _check_scratch_fs
 
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/528 +7 -3
index 6ca9a2370..a1efbbd27 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -155,11 +155,11 @@ test_ops() {
 
 echo "Create fake rt volume"
 $XFS_IO_PROG -f -c "truncate 400m" $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Make sure synth rt volume works"
 export USE_EXTERNAL=yes
-export SCRATCH_RTDEV=$rtdev
+export SCRATCH_RTDEV=$rt_loop_dev
 _scratch_mkfs > $seqres.full
 _try_scratch_mount || \
 	_notrun "Could not mount with synthetic rt volume"
@@ -170,6 +170,10 @@ test_ops 262144
 # not a power of two
 test_ops 327680
 
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/530 +6 -5
index 8a182bd6a..d0d0e2665 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -52,12 +52,12 @@ fi
 
 rtdevsz=$((nr_bits * rtextsz))
 truncate -s $rtdevsz $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Format and mount rt volume"
 
 export USE_EXTERNAL=yes
-export SCRATCH_RTDEV=$rtdev
+export SCRATCH_RTDEV=$rt_loop_dev
 _scratch_mkfs -d size=$((1024 * 1024 * 1024)) \
 	      -r size=${rtextsz},extsize=${rtextsz} >> $seqres.full
 _try_scratch_mount || _notrun "Couldn't mount fs with synthetic rt volume"
@@ -116,8 +116,9 @@ done
 echo "Check filesystem"
 _check_scratch_fs
 
-losetup -d $rtdev
-rm -f $TEST_DIR/$seq.rtvol
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
 
 # success, all done
 status=0
tests/xfs/606 +8 -4
index f958bddd8..b537ea145 100755
@@ -13,10 +13,8 @@ _begin_fstest auto quick growfs
 
 _cleanup()
 {
-	local dev
 	_unmount $LOOP_MNT 2>/dev/null
-	dev=$(losetup -j testfile | cut -d: -f1)
-	losetup -d $dev 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	rm -rf $LOOP_IMG $LOOP_MNT
 	cd /
 	rm -f $tmp.*
@@ -40,7 +38,9 @@ $MKFS_XFS_PROG -f $LOOP_IMG >$seqres.full
 # Extend by just 8K, expected to start with the last full-size AG ends of
 # above 1G block device.
 $XFS_IO_PROG -f -c "truncate 1073750016" $LOOP_IMG
-_mount -oloop $LOOP_IMG $LOOP_MNT
+
+loop_dev=$(_create_loop_device $LOOP_IMG)
+_mount $loop_dev $LOOP_MNT
 # A known bug shows "XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on
 # device" at here, refer to _fixed_by_kernel_commit above
 $XFS_GROWFS_PROG $LOOP_MNT >$seqres.full
@@ -48,6 +48,10 @@ if [ $? -ne 0 ];then
 	echo "xfs_growfs fails!"
 fi
 
+_unmount $LOOP_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 echo "Silence is golden"
 # success, all done
 status=0
tests/xfs/613 +14 -26
index 6ba3d87bf..b87cae91e 100755
@@ -15,14 +15,8 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_unmount $LOOP_MNT 2>/dev/null
-	if [ -n "$LOOP_DEV" ];then
-		_destroy_loop_device $LOOP_DEV 2>/dev/null
-	fi
-	if [ -n "$LOOP_SPARE_DEV" ];then
-		_destroy_loop_device $LOOP_SPARE_DEV 2>/dev/null
-	fi
+	[ -n $loop_dev ] &&_destroy_loop_device $loop_dev
 	rm -f $LOOP_IMG
-	rm -f $LOOP_SPARE_IMG
 	rmdir $LOOP_MNT
 }
 
@@ -38,16 +32,11 @@ _require_loop
 _require_xfs_io_command "falloc"
 
 LOOP_IMG=$TEST_DIR/$seq.dev
-LOOP_SPARE_IMG=$TEST_DIR/$seq.logdev
 LOOP_MNT=$TEST_DIR/$seq.mnt
 
 echo "** create loop device"
 $XFS_IO_PROG -f -c "truncate 32g" $LOOP_IMG
-LOOP_DEV=`_create_loop_device $LOOP_IMG`
-
-echo "** create loop log device"
-$XFS_IO_PROG -f -c "truncate 1g" $LOOP_SPARE_IMG
-LOOP_SPARE_DEV=`_create_loop_device $LOOP_SPARE_IMG`
+loop_dev=`_create_loop_device $LOOP_IMG`
 
 echo "** create loop mount point"
 rmdir $LOOP_MNT 2>/dev/null
@@ -56,8 +45,7 @@ mkdir -p $LOOP_MNT || _fail "cannot create loopback mount point"
 filter_loop()
 {
 	sed -e "s,\B$LOOP_MNT,LOOP_MNT,g" \
-	    -e "s,\B$LOOP_DEV,LOOP_DEV,g" \
-	    -e "s,\B$LOOP_SPARE_DEV,LOOP_SPARE_DEV,g"
+	    -e "s,\B$loop_dev,LOOP_DEV,g"
 }
 
 filter_xfs_opt()
@@ -70,22 +58,22 @@ MKFS_OPTIONS=""
 do_mkfs()
 {
 	echo "FORMAT: $@" | filter_loop | tee -a $seqres.full
-	$MKFS_XFS_PROG -f $* $LOOP_DEV | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
+	$MKFS_XFS_PROG -f $* $loop_dev | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
 	if [ "${PIPESTATUS[0]}" -ne 0 ]; then
-		_fail "Fails on _mkfs_dev $* $LOOP_DEV"
+		_fail "Fails on _mkfs_dev $* $loop_dev"
 	fi
 	. $tmp.mkfs
 }
 
 is_dev_mounted()
 {
-	findmnt --source $LOOP_DEV >/dev/null
+	findmnt --source $loop_dev >/dev/null
 	return $?
 }
 
 get_mount_info()
 {
-	findmnt --source $LOOP_DEV -o OPTIONS -n
+	findmnt --source $loop_dev -o OPTIONS -n
 }
 
 force_unmount()
@@ -104,29 +92,29 @@ _do_test()
 	local info
 
 	# mount test
-	_mount $LOOP_DEV $LOOP_MNT $opts 2>>$seqres.full
+	_mount $loop_dev $LOOP_MNT $opts 2>>$seqres.full
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "${mounted}" = "fail" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to fail, but it succeeded"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -ne 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs not mounted even mount return 0"
 			return 1
 		fi
 	else
 		if [ "${mounted}" = "pass" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to succeed, but it failed"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -eq 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs is mounted even mount return non-zero"
 			return 1
 		fi
@@ -142,13 +130,13 @@ _do_test()
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "$found" != "true" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expected to find \"$key\" in mount info \"$info\""
 			return 1
 		fi
 	else
 		if [ "$found" != "false" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: did not expect to find \"$key\" in \"$info\""
 			return 1
 		fi
tests/xfs/613.out +0 -1
index 1624617ee..2a693c53c 100644
@@ -1,6 +1,5 @@
 QA output created by 613
 ** create loop device
-** create loop log device
 ** create loop mount point
 ** start xfs mount testing ...
 FORMAT: -m crc=0
Signature · 1 line
2.45.2
#25 lore raw

Re: [PATCH 13/40] fstests: clean up loop device instantiation

Quoted 105 lines · Zorro Lang
Quoted 84 lines · Zorro Lang
Quoted 70 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Lots of tests do there own special thing with loop devices rather than using _create_loop_device() and _destroy_loop_device(). This oftens means they do not clean up after themselves properly, leaving stale loop devices around that result in unmountable test or scratch devices. This is common when tests are killed by user interrupt.

Even the tests that do use _destroy_loop_device and try to clean up often do it incorrectly, leading to spurious error messages.

Some tests try to use dynamic instantiation via "mount -o loop", but then don't clean up in the correct order or hack around to find the loop device that was instantiated because the test needs to know the instantiated device name

Clean this up by converting all the tests to use _create_loop_device() and _destroy_loop_device(). In all the tests, use the variable "loop_dev" for the device consistently. In _destroy_loop_device(), test that a device name has been passed so that we don't try to clean up the same device twice (e.g. once before test exit and again from the _cleanup() function). When we destroy a loop device, unset the variable used to hold the loop device name so that we don't try to destroy it twice.

This results in much more reliable cleanup and clean exit from fstests when killed by the user.

---
 common/metadump   | 32 ++++++++++++--------------------
 tests/generic/067 | 11 ++++++++---
 tests/generic/361 |  6 +++++-
 tests/generic/563 | 14 +++++++++-----
 tests/generic/564 | 12 ++++++------
 tests/generic/590 |  9 ++++++---
 tests/generic/744 |  4 ++--
 tests/generic/746 |  6 +++++-
 tests/xfs/014     |  7 +++++--
 tests/xfs/049     | 35 +++++++++++++++++++++--------------
 tests/xfs/073     | 24 +++++++++++++-----------
 tests/xfs/074     | 21 ++++++++++++---------
 tests/xfs/078     | 16 +++++++++-------
 tests/xfs/148     | 23 ++++++++++++++---------
 tests/xfs/149     |  4 ++++
 tests/xfs/216     | 15 ++++++++++-----
 tests/xfs/217     | 22 ++++++++++++++++------
 tests/xfs/250     | 14 +++++++++-----
 tests/xfs/259     | 13 ++++++++-----
 tests/xfs/513     | 44 ++++++++++++++++++++------------------------
 tests/xfs/521     |  8 ++++++--
 tests/xfs/528     | 10 +++++++---
 tests/xfs/530     | 11 ++++++-----
 tests/xfs/606     | 12 ++++++++----
 tests/xfs/613     | 40 ++++++++++++++--------------------------
 tests/xfs/613.out |  1 -
 26 files changed, 235 insertions(+), 179 deletions(-)
common/metadump +0 -3
index 3373edfe9..bc3038e3b 100644
@@ -24,17 +24,9 @@ _xfs_cleanup_verify_metadump()
 
 	test -n "$XFS_METADUMP_FILE" && rm -f "$XFS_METADUMP_FILE"
 
-	if [ -n "$XFS_METADUMP_IMG" ]; then
-		losetup -n -a -O BACK-FILE,NAME | grep "^$XFS_METADUMP_IMG" | while read backing ldev; do
-			losetup -d "$ldev"

common/metadump _xfs_cleanup_verify_metadump() · line 28

Quoted 10 lines
common/metadump +2 -8
-		done
-
-		# Don't call rm directly with a globbed argument here to avoid
-		# issues issues with variable expansions.
-		for img in "$XFS_METADUMP_IMG"*; do
-			test -e "$img" && rm -f "$img"
-		done
-	fi
+	[ -n "$md_data_loop_dev" ] && _destroy_loop_device $md_data_loop_dev
+	[ -n "$md_log_loop_dev" ] && _destroy_loop_device $md_log_loop_dev

As Darrick reported that he always hit below error:

Quoted 7 lines

losetup: /dev/loop0: detach failed: No such device or address Cannot destroy loop device /dev/loop0

I'm wondering we should use [ -b ] not [ -n ] at here? Or output the stderr to /dev/null directly. Due to... (see below)

common/metadump _xfs_cleanup_verify_metadump() · line 29

Quoted 1 line
common/metadump +1 -0
+	rm -f $data_img $log_img
                ^^^^^^   ^^^^^^

Besides above problems, this line doesn't work either, due to the "data_img" and "log_img" are local variables in other functions. Even they're not local, think these variable names are too common to be global variable in common helper.

I'll try to change part of above new code back to old code as below [1]. Hope it can help the test work as usual, then we can merge this patch and move on. And we can improve this part in later patch.

Thanks,
Zorro

[1]

Review

Quoted 6 lines
(quoted diff) +3 -3
-       [ -n "$md_data_loop_dev" ] && _destroy_loop_device $md_data_loop_dev
-       [ -n "$md_log_loop_dev" ] && _destroy_loop_device $md_log_loop_dev
-       rm -f $data_img $log_img
+       if [ -n "$XFS_METADUMP_IMG" ]; then
+               [ -b "$md_data_loop_dev" ] && _destroy_loop_device $md_data_loop_dev
+               [ -b "$md_log_loop_dev" ] && _destroy_loop_device $md_log_loop_dev

Finally I can reproduce the issue which Darrick metioned, it needs a new xfsprogs feature which hasn't been merged. Then I found The [ -b ] only solves half of the problem, due to even the /dev/loop0 is deteched, the /dev/loop0 is still there, and [ -b /dev/loop0 ] return true.

To solve this problem totally, I think there're 3 ways:
1) Check if a loop device is really detected, likes the old code did:
"losetup -n -a -O BACK-FILE,NAME | grep "^$XFS_METADUMP_IMG" | while read backing ldev"
2) unset md_data_loop_dev and md_log_loop_dev properly, after _xfs_verify_metadump_v2() deteches them. Something likes this:
@@ -134,9 +134,11 @@ _xfs_verify_metadump_v2()
# Tear down what we created
if [ -b "$md_log_loop_dev" ]; then
_destroy_loop_device $md_log_loop_dev
+ unset md_log_loop_dev
rm -f $log_img
fi
_destroy_loop_device $md_data_loop_dev
+ unset md_data_loop_dev
rm -f $data_img

3) Ignore all outputs of _destroy_loop_device, e.g. _destroy_loop_device $md_data_loop_dev >/dev/null 2>&1 _destroy_loop_device $md_log_loop_dev >/dev/null 2>&1

The 1st and 3rd methods can avoid this failure totally. But the 2nd one looks smoother, and the xfs/129 test passed with it. I'm just not sure if I missed something else :)

  # ./check -s default xfs/129
  SECTION       -- default
  FSTYP         -- xfs (non-debug)
  PLATFORM      -- Linux/x86_64 dell-per750-41 6.12.0-0.rc7.58.fc42.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Nov 11 15:23:45 UTC 2024
  MKFS_OPTIONS  -- -f /dev/sda6
  MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/sda6 /mnt/scratch

xfs/129 9s ... 14s
Ran: xfs/129
Passed all 1 tests

If you all agree, I can change this part as the 2nd way when I merge it.

Thanks,
Zorro

Quoted 1,254 lines
(quoted diff) +4 -0
+               for img in "$XFS_METADUMP_IMG"*; do
+                       test -e "$img" && rm -f "$img"
+               done
+       fi
Quoted 1,250 lines
Quoted 16 lines
common/metadump +2 -2
 }
 
 # Can xfs_metadump snapshot the fs metadata to a v1 metadump file?
@@ -122,25 +114,25 @@ _xfs_verify_metadump_v2()
 		_scratch_xfs_mdrestore $metadump_file
 
 	# Create loopdev for data device so we can mount the fs
-	data_loop=$(_create_loop_device $data_img)
+	md_data_loop_dev=$(_create_loop_device $data_img)
 
 	# Create loopdev for log device if we recovered anything
-	test -s "$log_img" && log_loop=$(_create_loop_device $log_img)
+	test -s "$log_img" && md_log_loop_dev=$(_create_loop_device $log_img)
 
 	# Mount fs, run an extra test, fsck, and unmount

common/metadump _xfs_verify_metadump_v2() · line 128

Quoted 10 lines
common/metadump +4 -4
-	SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop _scratch_mount
+	SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev _scratch_mount
 	if [ -n "$extra_test" ]; then
-		SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop $extra_test
+		SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev $extra_test
 	fi
-	SCRATCH_DEV=$data_loop SCRATCH_LOGDEV=$log_loop _check_xfs_scratch_fs
-	SCRATCH_DEV=$data_loop _scratch_unmount
+	SCRATCH_DEV=$md_data_loop_dev SCRATCH_LOGDEV=$md_log_loop_dev _check_xfs_scratch_fs
+	_unmount $md_data_loop_dev

We umount loop devices we created above at here, and ...

common/metadump _xfs_verify_metadump_v2() · line 135

Quoted 10 lines
common/metadump +3 -3
 	# Tear down what we created
-	if [ -b "$log_loop" ]; then
-		_destroy_loop_device $log_loop
+	if [ -b "$md_log_loop_dev" ]; then
+		_destroy_loop_device $md_log_loop_dev
 		rm -f $log_img
 	fi
-	_destroy_loop_device $data_loop
+	_destroy_loop_device $md_data_loop_dev

we even has destroied them at here.

So that _xfs_cleanup_verify_metadump function hit "detach failed: No such device or address", due to those loop devices might have been released, but the variables are not unset.

I didn't give it a test, just did a code review. As Darrick uses this part mostly, could you please check what I said above?

Thanks,
Zorro

Quoted 1,198 lines
common/metadump +0 -0
 	rm -f $data_img
 }
 
tests/generic/067 +8 -3
index ccb1e3fbb..b45ae834f 100755
@@ -37,13 +37,18 @@ mount_nonexistent_mnt()
 	$MOUNT_PROG $SCRATCH_DEV $TEST_DIR/nosuchdir >>$seqres.full 2>&1
 }
 
-# fs driver should be able to handle mounting a free loop device gracefully
-# xfs ever hung, "ec53d1d xfs: don't block on buffer read errors" fixed it
+# fs driver should be able to handle mounting a free loop device gracefully xfs
+# ever hung, "ec53d1d xfs: don't block on buffer read errors" fixed it
+#
+# Using 'losetup -f' like this is racy. We might end up mounting a real loop dev
+# here, so unmount $SCRATCH_MNT (not the loop dev we might not own!) and ignore
+# any error it might return.
 mount_free_loopdev()
 {
 	echo "# mount a free loop device" >>$seqres.full
 	loopdev=`losetup -f`
-	$MOUNT_PROG -t $FSTYP $loopdev $SCRATCH_MNT >>$seqres.full 2>&1
+	_mount $loopdev $SCRATCH_MNT >>$seqres.full 2>&1
+	_unmount $SCRATCH_MNT >> /dev/null 2>&1
 }
 
 # mount with wrong fs type specified.
tests/generic/361 +5 -1
index 7273dd056..e2b798436 100755
@@ -17,7 +17,7 @@ _begin_fstest auto quick
 _cleanup()
 {
 	_unmount $fs_mnt
-	_destroy_loop_device $loop_dev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
 }
@@ -54,6 +54,10 @@ $XFS_IO_PROG -fc "pwrite 0 520m" $fs_mnt/testfile >>$seqres.full 2>&1
 # remount should not hang
 $MOUNT_PROG -o remount,ro $fs_mnt >>$seqres.full 2>&1
 
+_unmount $fs_mnt
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 echo "Silence is golden"
 status=0
tests/generic/563 +9 -5
index f0d2f404c..34d21170c 100755
@@ -87,12 +87,12 @@ reset()
 
 # cgroup I/O accounting doesn't work on partitions. Use a loop device to rule
 # that out.
-LOOP_DEV=$(_create_loop_device $SCRATCH_DEV)
-smajor=$((0x`stat -L -c %t $LOOP_DEV`))
-sminor=$((0x`stat -L -c %T $LOOP_DEV`))
+loop_dev=$(_create_loop_device $SCRATCH_DEV)
+smajor=$((0x`stat -L -c %t $loop_dev`))
+sminor=$((0x`stat -L -c %T $loop_dev`))
 
-_mkfs_dev $LOOP_DEV >> $seqres.full 2>&1
-_mount $LOOP_DEV $SCRATCH_MNT || _fail "mount failed"
+_mkfs_dev $loop_dev >> $seqres.full 2>&1
+_mount $loop_dev $SCRATCH_MNT || _fail "mount failed"
 
 blksize=$(_get_block_size "$SCRATCH_MNT")
 
@@ -147,6 +147,10 @@ if [ "$drop_io_cgroup" = 1 ]; then
 	echo "-io" > $cgdir/cgroup.subtree_control || _fail "subtree control"
 fi
 
+_unmount $SCRATCH_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/generic/564 +6 -6
index 647472d78..b66b72496 100755
@@ -19,7 +19,7 @@ _cleanup()
 {
 	cd /
 	rm -rf $tmp.*
-	[ -z "$loopdev" ] || _destroy_loop_device $loopdev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 }
 
 # Import common functions.
@@ -72,11 +72,11 @@ $XFS_IO_PROG -f -c "copy_range -l 32k $testdir" $testdir/copy
 echo
 echo source/destination as blkdev returns EINVAL
 $XFS_IO_PROG -f -c "truncate 128k" $testdir/img >> $seqres.full 2>&1
-loopdev=`_create_loop_device $testdir/img`
-$XFS_IO_PROG -c "copy_range -l 32k $testdir/file" $loopdev
-$XFS_IO_PROG -f -c "copy_range -l 32k $loopdev" $testdir/copy
-_destroy_loop_device $loopdev
-loopdev=
+loop_dev=`_create_loop_device $testdir/img`
+$XFS_IO_PROG -c "copy_range -l 32k $testdir/file" $loop_dev
+$XFS_IO_PROG -f -c "copy_range -l 32k $loop_dev" $testdir/copy
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo
 echo source/destination as chardev returns EINVAL
tests/generic/590 +6 -3
index 2b7ccfb53..2eefa2c62 100755
@@ -15,9 +15,10 @@ _begin_fstest auto prealloc preallocrw
 # Override the default cleanup function.
 _cleanup()
 {
+	_scratch_unmount
+	[ -n $loop_dev ] && _destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
-	test -n "$loop" && _destroy_loop_device "$loop"
 	rm -f "$TEST_DIR/$seq"
 }
 
@@ -56,9 +57,9 @@ if [[ $FSTYP = xfs ]]; then
 		loopsz="$((filesz + (1 << 26)))"
 		_require_fs_space "$TEST_DIR" $((loopsz / 1024))
 		$XFS_IO_PROG -c "truncate $loopsz" -f "$TEST_DIR/$seq"
-		loop="$(_create_loop_device "$TEST_DIR/$seq")"
+		loop_dev="$(_create_loop_device "$TEST_DIR/$seq")"
 		USE_EXTERNAL=yes
-		SCRATCH_RTDEV="$loop"
+		SCRATCH_RTDEV="$loop_dev"
 		disabled_features=()
 
 		# disable reflink if not supported by realtime devices
@@ -114,6 +115,8 @@ $XFS_IO_PROG -c "truncate 0" -c fsync "$SCRATCH_MNT/file"
 # We need to do this before the loop device gets torn down.
 _scratch_unmount
 _check_scratch_fs
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo "Silence is golden"
 status=0
tests/generic/744 +2 -2
index df8f6ae9b..cda10e0f6 100755
@@ -17,8 +17,8 @@ _cleanup()
 
 	_unmount $mnt2 &> /dev/null
 	_unmount $mnt1 &> /dev/null
-	[ -b "$loop_dev2" ] && losetup -d $loop_dev2
-	[ -b "$loop_dev1" ] && losetup -d $loop_dev1
+	[ -b "$loop_dev2" ] && _destroy_loop_device $loop_dev2
+	[ -b "$loop_dev1" ] && _destroy_loop_device $loop_dev1
 	[ -n "$seq" ] && rm -rf $TEST_DIR/$seq
 }
 
tests/generic/746 +5 -1
index 671910606..ba8ed25e8 100755
@@ -39,7 +39,7 @@ esac
 _cleanup()
 {
 	_unmount $loop_mnt &> /dev/null
-	_destroy_loop_device $loop_dev
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	if [ $status -eq 0 ]; then
 		rm -rf $tmp
 		rm $img_file
@@ -223,5 +223,9 @@ while read line; do
 done < $fiemap_after
 echo "done."
 
+_unmount $loop_mnt
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 status=0
 exit
tests/xfs/014 +5 -2
index f638e4b16..39ea40e2a 100755
@@ -23,6 +23,7 @@ _cleanup()
 {
 	cd /
 	_unmount $LOOP_MNT 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	_scratch_unmount 2>/dev/null
 	rm -f $tmp.*
 }
@@ -166,19 +167,21 @@ LOOP_FILE=$SCRATCH_MNT/$seq.fs
 LOOP_MNT=$SCRATCH_MNT/$seq.mnt
 
 $MKFS_XFS_PROG -d "file=1,name=$LOOP_FILE,size=10g" >> $seqres.full 2>&1
+loop_dev=$(_create_loop_device $LOOP_FILE)
 
 mkdir -p $LOOP_MNT
-_mount -o uquota,gquota $LOOP_FILE $LOOP_MNT || \
+_mount -o uquota,gquota $loop_dev $LOOP_MNT || \
 	_fail "Failed to mount loop fs."
 
 _test_enospc $LOOP_MNT
 _test_edquot $LOOP_MNT
 
 _unmount $LOOP_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 echo $orig_sp_time > /proc/sys/fs/xfs/speculative_prealloc_lifetime
 
 _scratch_unmount
-
 status=0
 exit
tests/xfs/049 +21 -14
index 4163a144f..cdcddf764 100755
@@ -12,16 +12,17 @@ _begin_fstest rw auto quick
 # Override the default cleanup function.
 _cleanup()
 {
-    cd /
-    umount $SCRATCH_MNT/test2 > /dev/null 2>&1
-    umount $SCRATCH_MNT/test > /dev/null 2>&1
-    rm -f $tmp.*
+	cd /
+	_unmount $SCRATCH_MNT/test2 > /dev/null 2>&1
+	_unmount $SCRATCH_MNT/test > /dev/null 2>&1
+	[ -n "$loop_dev2" ] && _destroy_loop_device $loop_dev2
+	[ -n "$loop_dev1" ] && _destroy_loop_device $loop_dev1
+	rm -f $tmp.*
 
-    if [ -w $seqres.full ]
-    then
-        echo "--- mounts at end (after cleanup)" >> $seqres.full
-        mount >> $seqres.full
-    fi
+	if [ -w $seqres.full ]; then
+		echo "--- mounts at end (after cleanup)" >> $seqres.full
+		mount >> $seqres.full
+	fi
 }
 
 # Import common functions.
@@ -64,7 +65,8 @@ mkdir $SCRATCH_MNT/test $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to make mount points"
 
 _log "Mount xfs via loop"
-mount -t xfs -o loop $SCRATCH_MNT/test.xfs $SCRATCH_MNT/test >> $seqres.full 2>&1 \
+loop_dev1=$(_create_loop_device $SCRATCH_MNT/test.xfs)
+_mount $loop_dev1 $SCRATCH_MNT/test >> $seqres.full 2>&1 \
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress"
@@ -80,11 +82,12 @@ dd if=/dev/zero of=$SCRATCH_MNT/test/test.ext2 bs=1024 count=10240 >> $seqres.fu
     || _fail "!!! create file failed"
 
 _log "Create ext2 fs in file on looped xfs"
-echo y | mkfs -t ext2 $SCRATCH_MNT/test/test.ext2 >> $seqres.full 2>&1 \
+loop_dev2=$(_create_loop_device $SCRATCH_MNT/test/test.ext2)
+echo y | mkfs -t ext2 $loop_dev2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to mkfs ext2 on xfs"
 
 _log "Mount ext2 on xfs via loop"
-mount -t ext2 -o loop $SCRATCH_MNT/test/test.ext2 $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
+_mount $loop_dev2 $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress ext2 on xfs via loop"
@@ -96,12 +99,16 @@ rm -rf $SCRATCH_MNT/test/* >> $seqres.full 2>&1 \
     || _fail "!!! clean failed"
 
 _log "umount ext2 on xfs"
-umount $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
+_unmount $SCRATCH_MNT/test2 >> $seqres.full 2>&1 \
     || _fail "!!! umount ext2 failed"
+_destroy_loop_device $loop_dev2
+unset loop_dev2
 
 _log "umount xfs"
-umount $SCRATCH_MNT/test >> $seqres.full 2>&1 \
+_unmount $SCRATCH_MNT/test >> $seqres.full 2>&1 \
     || _fail "!!! umount xfs failed"
+_destroy_loop_device $loop_dev1
+unset loop_dev1
 
 echo "--- mounts at end (before cleanup)" >> $seqres.full
 mount >> $seqres.full
tests/xfs/073 +13 -11
index e4b17c5e7..2274079ef 100755
@@ -23,6 +23,8 @@ _cleanup()
 	_scratch_unmount 2>/dev/null
 	_unmount $imgs.loop 2>/dev/null
 	_unmount $imgs.source_dir 2>/dev/null
+	[ -n "$loop_dev1" ] && _destroy_loop_device $loop_dev1
+	[ -n "$loop_dev2" ] && _destroy_loop_device $loop_dev2
 	[ -d $imgs.loop ] && rmdir $imgs.loop
 	[ -d $imgs.source_dir ] && rm -rf $imgs.source_dir
 	rm -f $imgs.* $tmp.* /var/tmp/xfs_copy.log.*
@@ -65,10 +67,11 @@ _verify_copy()
 	rmdir $target_dir 2>/dev/null
 	mkdir $target_dir
 
-	_mount -t xfs -o loop $target $target_dir 2>/dev/null
+	loop_dev1=$(_create_loop_device $target)
+	_mount $loop_dev1 $target_dir 2>/dev/null
 	if [ $? -ne 0 ]; then
 		echo retrying mount with nouuid option >>$seqres.full
-		_mount -t xfs -o loop -o nouuid $target $target_dir
+		_mount -o nouuid $loop_dev1 $target_dir
 		if [ $? -ne 0 ]; then
 			echo mount failed - evil!
 			return
@@ -100,6 +103,8 @@ _verify_copy()
 	echo unmounting and removing new image
 	_unmount $source_dir
 	_unmount $target_dir > /dev/null 2>&1
+	_destroy_loop_device $loop_dev1
+	unset loop_dev1
 	rm -f $target
 }
 
@@ -134,18 +139,15 @@ ${MKFS_XFS_PROG} -dfile,name=$imgs.source,size=100g \
 rmdir $imgs.source_dir 2>/dev/null
 mkdir $imgs.source_dir
 
-_mount -t xfs -o loop $imgs.source $imgs.source_dir
-loop2=`mount | grep $imgs.source | grep -o -e 'loop=.*[^),]' | grep -o -e '/.*$'`
+loop_dev2=$(_create_loop_device $imgs.source)
+_mount $loop_dev2 $imgs.source_dir
 cp -a $here $imgs.source_dir
-_mount -t xfs -o remount,ro $imgs.source $imgs.source_dir
-$XFS_COPY_PROG $imgs.source $imgs.image | _filter_copy '#' $imgs.image '#' '#'
+_mount -o remount,ro $loop_dev2 $imgs.source_dir
+$XFS_COPY_PROG $loop_dev2 $imgs.image 2> /dev/null | _filter_copy '#' $imgs.image '#' '#'
 _verify_copy $imgs.image $imgs.source $imgs.source_dir
 
-# HACK WARNING:
-#
-# We're done with the nested loop mount, now we have to clean
-# up the pieces that mount is incapable of doing.
-losetup -d $loop2 > /dev/null 2>&1
+_destroy_loop_device $loop_dev2
+unset loop_dev2
 
 echo
 echo === copying scratch device to multiple targets
tests/xfs/074 +12 -9
index 6a03c8111..5df864fad 100755
@@ -26,7 +26,7 @@ _begin_fstest quick auto prealloc rw
 _cleanup()
 {
 	cd /
-	_destroy_loop_device $LOOP_DEV
+	[ -n "$loop_dev" ] &&_destroy_loop_device $loop_dev
 	rm -f $tmp.* $LOOP_FILE
 }
 
@@ -45,10 +45,10 @@ LOOP_FILE=$TEST_DIR/$seq.img
 LOOP_MNT=$TEST_DIR/$seq.mnt
 mkdir -p $LOOP_MNT
 $XFS_IO_PROG -ft -c "truncate 1t" $LOOP_FILE >> $seqres.full
-LOOP_DEV=`_create_loop_device $LOOP_FILE`
+loop_dev=`_create_loop_device $LOOP_FILE`
 
-_mkfs_dev -d size=260g,agcount=2 $LOOP_DEV
-_mount $LOOP_DEV $LOOP_MNT
+_mkfs_dev -d size=260g,agcount=2 $loop_dev
+_mount $loop_dev $LOOP_MNT
 
 BLOCK_SIZE=$(_get_file_block_size $LOOP_MNT)
 
@@ -59,11 +59,11 @@ $XFS_IO_PROG -ft \
 	-c "falloc 0 $(($BLOCK_SIZE * 2097152))" \
 	$LOOP_MNT/foo >> $seqres.full
 
-umount $LOOP_MNT
-_check_xfs_filesystem $LOOP_DEV none none
+_unmount $LOOP_MNT
+_check_xfs_filesystem $loop_dev none none
 
-_mkfs_dev -f $LOOP_DEV
-_mount $LOOP_DEV $LOOP_MNT
+_mkfs_dev -f $loop_dev
+_mount $loop_dev $LOOP_MNT
 
 # check we trim both ends of the extent approproiately; this will fail
 # on 1k block size filesystems without the correct fixes in place.
@@ -73,7 +73,10 @@ $XFS_IO_PROG -ft \
 	$LOOP_MNT/foo >> $seqres.full
 
 _unmount $LOOP_MNT
-_check_xfs_filesystem $LOOP_DEV none none
+_check_xfs_filesystem $loop_dev none none
+
+_destroy_loop_device $loop_dev
+unset loop_dev
 
 # success, all done
 echo "Silence is golden"
tests/xfs/078 +9 -7
index 6b325e05f..0d3c2eb23 100755
@@ -17,7 +17,7 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_unmount $LOOP_MNT 2>/dev/null
-	[ -n "$LOOP_DEV" ] && _destroy_loop_device $LOOP_DEV 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev 2>/dev/null
 	# try to keep the image file if test fails
 	[ $status -eq 0 ] && rm -f $LOOP_IMG
 	rmdir $LOOP_MNT
@@ -55,7 +55,7 @@ _grow_loop()
 	agsize=$5
 
 	$XFS_IO_PROG -f -c "truncate $original" $LOOP_IMG
-	LOOP_DEV=`_create_loop_device $LOOP_IMG`
+	loop_dev=`_create_loop_device $LOOP_IMG`
 
 	dparam=""
 	if [ -n "$agsize" ]; then
@@ -67,15 +67,15 @@ _grow_loop()
 	echo
 
 	echo "*** mkfs loop file (size=$original)"
-	$MKFS_XFS_PROG -b size=$bsize $dparam $LOOP_DEV | \
+	$MKFS_XFS_PROG -b size=$bsize $dparam $loop_dev | \
 		_filter_mkfs 2>/dev/null
 
 	echo "*** extend loop file"
-	_destroy_loop_device $LOOP_DEV
+	_destroy_loop_device $loop_dev
 	$XFS_IO_PROG -c "pwrite $new_size $bsize" $LOOP_IMG | _filter_io
-	LOOP_DEV=`_create_loop_device $LOOP_IMG`
+	loop_dev=`_create_loop_device $LOOP_IMG`
 	echo "*** mount loop filesystem"
-	_mount -t xfs $LOOP_DEV $LOOP_MNT
+	_mount $loop_dev $LOOP_MNT
 
 	echo "*** grow loop filesystem"
 	$XFS_GROWFS_PROG $LOOP_MNT 2>&1 |  _filter_growfs 2>&1
@@ -87,9 +87,11 @@ _grow_loop()
 	if [ "$check" -gt "0" ]
 	then
 		echo "*** check"
-		_check_xfs_filesystem $LOOP_IMG none none
+		_check_xfs_filesystem $loop_dev none none
 	fi
 
+	_destroy_loop_device $loop_dev
+	unset loop_dev
 	rm -f $LOOP_IMG
 }
 
tests/xfs/148 +14 -9
index c42c9b119..4d2f7a808 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_unmount $mntpt > /dev/null 2>&1
-	_destroy_loop_device $loopdev > /dev/null 2>&1
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	rm -r -f $tmp.*
 }
 
@@ -48,12 +48,12 @@ rm -f $imgfile $imgfile.old
 # even when security xattrs are present so we are always doing name matches on
 # lookup and not name hash compares as leaf/node forms will do.
 $XFS_IO_PROG -f -c 'truncate 40m' $imgfile
-loopdev=$(_create_loop_device $imgfile)
-MKFS_OPTIONS="-m crc=0 -i size=512" _mkfs_dev $loopdev >> $seqres.full
+loop_dev=$(_create_loop_device $imgfile)
+MKFS_OPTIONS="-m crc=0 -i size=512" _mkfs_dev $loop_dev >> $seqres.full
 
 # Mount image file
 mkdir -p $mntpt
-_mount $loopdev $mntpt
+_mount $loop_dev $mntpt
 
 echo "creating entries" >> $seqres.full
 
@@ -91,7 +91,8 @@ cat $tmp.log | _filter_test_dir
 
 # Corrupt the entries
 _unmount $mntpt
-_destroy_loop_device $loopdev
+_destroy_loop_device $loop_dev
+unset loop_dev
 cp $imgfile $imgfile.old
 sed -b \
 	-e "s/$nullstr/too_many\x00beans/g" \
@@ -99,8 +100,9 @@ sed -b \
 	-i $imgfile
 test "$(md5sum < $imgfile)" != "$(md5sum < $imgfile.old)" ||
 	_fail "sed failed to change the image file?"
-loopdev=$(_create_loop_device $imgfile)
-_mount $loopdev $mntpt
+
+loop_dev=$(_create_loop_device $imgfile)
+_mount $loop_dev $mntpt
 
 # Try to access the corrupt metadata
 echo "++ ACCESSING BAD METADATA" | tee -a $seqres.full
@@ -111,7 +113,7 @@ cat $tmp.log | _filter_test_dir | sed -e '/Could not list/d'
 echo "does scrub complain?" >> $seqres.full
 
 # Does scrub complain about this?
-if _supports_xfs_scrub $mntpt $loopdev; then
+if _supports_xfs_scrub $mntpt $loop_dev; then
 	$XFS_SCRUB_PROG -n $mntpt >> $seqres.full 2>&1
 	res=$?
 	test $((res & 1)) -eq 0 && \
@@ -122,11 +124,14 @@ echo "does repair complain?" >> $seqres.full
 
 # Does repair complain about this?
 _unmount $mntpt
-$XFS_REPAIR_PROG -n $loopdev >> $seqres.full 2>&1
+$XFS_REPAIR_PROG -n $loop_dev >> $seqres.full 2>&1
 res=$?
 test $res -eq 1 || \
 	echo "repair failed to report corruption ($res)"
 
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/149 +4 -0
index f2187109b..9a96f82ed 100755
@@ -84,6 +84,10 @@ $XFS_GROWFS_PROG -D 16384 $loop_symlink > /dev/null
 echo "=== xfs_growfs - check device node ==="
 $XFS_GROWFS_PROG -D 20480 $loop_dev > /dev/null
 
+_unmount $mntdir
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/216 +10 -5
index 6b8b2eb22..091c11d08 100755
@@ -15,6 +15,7 @@ _begin_fstest log metadata auto quick
 _cleanup()
 {
 	_unmount $LOOP_MNT > /dev/null 2>&1
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
 }
@@ -24,7 +25,7 @@ _scratch_mkfs_xfs >/dev/null 2>&1
 _scratch_mount
 
 _require_loop
-LOOP_DEV=$SCRATCH_MNT/test_fs
+LOOP_IMG=$SCRATCH_MNT/test_fs
 LOOP_MNT=$SCRATCH_MNT/test_fs_dir
 
 loop_mkfs_opts=
@@ -55,22 +56,26 @@ _do_mkfs()
 	for i in $*; do
 		echo -n "fssize=${i}g "
 		$MKFS_XFS_PROG -f -b size=4096 -l version=2 \
-			-d name=$LOOP_DEV,size=${i}g $loop_mkfs_opts |grep log
-		_mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
+			-d size=${i}g $loop_mkfs_opts $loop_dev |grep log
+		_mount $loop_dev $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
 		_unmount $LOOP_MNT > /dev/null 2>&1
 	done
 }
 # make large holey file
-$XFS_IO_PROG -f -c "truncate 256g" $LOOP_DEV
+$XFS_IO_PROG -f -c "truncate 256g" $LOOP_IMG
 
-choose_golden_output $0 $LOOP_DEV
+choose_golden_output $0 $LOOP_IMG
 
 #make loopback mount dir
 mkdir $LOOP_MNT
 
+loop_dev=$(_create_loop_device $LOOP_IMG)
+
 # walk over standard sizes (up to 256GB)
 _do_mkfs 1 2 4 8 16 32 64 128 256
 
+_destroy_loop_device $loop_dev
+unset loop_dev
 status=0
 exit
tests/xfs/217 +16 -6
index b2eb34490..dae6ce55f 100755
@@ -12,6 +12,12 @@ _begin_fstest log metadata auto
 # Import common functions.
 . ./common/filter
 
+_cleanup()
+{
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
+	cd /
+	rm -f $tmp.*
+}
 
 _require_scratch
 _scratch_mkfs_xfs >/dev/null 2>&1
@@ -20,7 +26,7 @@ _scratch_mount
 _require_fs_space $SCRATCH_MNT 2202000
 
 _require_loop
-LOOP_DEV=$SCRATCH_MNT/test_fs
+LOOP_IMG=$SCRATCH_MNT/test_fs
 LOOP_MNT=$SCRATCH_MNT/test_fs_dir
 
 _do_mkfs()
@@ -28,28 +34,30 @@ _do_mkfs()
 	for i in $*; do
 		echo -n "fssize=${i}g "
 		$MKFS_XFS_PROG -f -b size=4096 -l version=2 \
-			-d name=$LOOP_DEV,size=${i}g |grep log
-		_mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
+			-d size=${i}g $loop_dev |grep log
+		_mount $loop_dev $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
 		_unmount $LOOP_MNT > /dev/null 2>&1
 
 		# punch out the previous blocks so that we keep the amount of
 		# disk space the test requires down to a minimum.
-		$XFS_IO_PROG -f -c "unresvsp 0 16383g" $LOOP_DEV
+		$XFS_IO_PROG -f -c "unresvsp 0 16383g" $LOOP_IMG
 	done
 }
 # make large holey file
-$XFS_IO_PROG -f -c "truncate 16383g" $LOOP_DEV
+$XFS_IO_PROG -f -c "truncate 16383g" $LOOP_IMG
 
 #make loopback mount dir
 mkdir $LOOP_MNT
 
 # test if large logs are supported
-$MKFS_XFS_PROG -f -l size=256m -d name=$LOOP_DEV,size=10g > /dev/null 2>&1
+$MKFS_XFS_PROG -f -l size=256m -d name=$LOOP_IMG,size=10g > /dev/null 2>&1
 if [ $? -ne 0 ]; then
 	_notrun "large log sizes not supported by mkfs"
 fi
 
+loop_dev=$(_create_loop_device $LOOP_IMG)
+
 #
 # walk over "new" sizes supported by recent xfsprogs.
 # Note that the last test is for 16TB-1GB as 32bit platforms only support
@@ -57,5 +65,7 @@ fi
 #
 _do_mkfs 512 1024 2048 4096 8192 16383
 
+_destroy_loop_device $loop_dev
+unset loop_dev
 status=0
 exit
tests/xfs/250 +9 -5
index 4e3473ebc..2554e1e91 100755
@@ -14,7 +14,8 @@ _cleanup()
 {
 	cd /
 	_unmount $LOOP_MNT 2>/dev/null
-	rm -f $LOOP_DEV
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
+	rm -f $LOOP_IMG
 	rmdir $LOOP_MNT
 	rm -f $tmp.*
 }
@@ -26,7 +27,7 @@ _require_test
 _require_loop
 _require_xfs_io_command "falloc"
 
-LOOP_DEV=$TEST_DIR/$seq.fs
+LOOP_IMG=$TEST_DIR/$seq.fs
 LOOP_MNT=$TEST_DIR/$seq.mnt
 
 _filter_io()
@@ -45,7 +46,7 @@ _test_loop()
 	agsize=$2
 	fsize=$3
 
-	dparam="file,name=$LOOP_DEV,size=$size"
+	dparam="file,name=$LOOP_IMG,size=$size"
 	if [ -n "$agsize" ]; then
 		dparam="$dparam,agsize=$agsize"
 	fi
@@ -55,7 +56,8 @@ _test_loop()
 		| _filter_mkfs 2>/dev/null
 
 	echo "*** mount loop filesystem"
-	_mount -t xfs -o loop $LOOP_DEV $LOOP_MNT
+	loop_dev=$(_create_loop_device $LOOP_IMG)
+	mount $loop_dev $LOOP_MNT
 
 	echo "*** preallocate large file"
 	$XFS_IO_PROG -f -c "resvsp 0 $fsize" $LOOP_MNT/foo | _filter_io
@@ -64,7 +66,9 @@ _test_loop()
 	_unmount $LOOP_MNT > /dev/null 2>&1
 
 	echo "*** check loop filesystem"
-	 _check_xfs_filesystem $LOOP_DEV none none
+	_check_xfs_filesystem $loop_dev none none
+	_destroy_loop_device $loop_dev
+	unset loop_dev
 }
 
 _test_loop 50g 16m 40G
tests/xfs/259 +8 -5
index 0c8d6eb56..c2d26381a 100755
@@ -12,7 +12,10 @@ _begin_fstest auto quick
 # Override the default cleanup function.
 _cleanup()
 {
-    rm -f "$testfile"
+	[ -n "$loop_dev" ] && _destroy_loop_device $testfile
+	rm -f "$testfile"
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -45,13 +48,13 @@ for del in $sizes_to_check; do
 		rm -f "$testfile"
 		dd if=/dev/zero "of=$testfile" bs=1 count=0 seek=$ddseek \
 			>/dev/null 2>&1 || echo "dd failed"
-		lofile=$(losetup -f)
-		losetup $lofile "$testfile"
-		$MKFS_XFS_PROG -l size=32m -b size=$bs $lofile |  _filter_mkfs \
+		loop_dev=$(_create_loop_device $testfile)
+		$MKFS_XFS_PROG -l size=32m -b size=$bs $loop_dev |  _filter_mkfs \
 			>/dev/null 2> $tmp.mkfs || echo "mkfs failed!"
 		. $tmp.mkfs
 		sync
-		losetup -d $lofile
+		_destroy_loop_device $loop_dev
+		unset loop_dev
 	done
 done
 
tests/xfs/513 +20 -24
index 5895e6e2d..0c0edc75e 100755
@@ -15,12 +15,8 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_unmount $LOOP_MNT 2>/dev/null
-	if [ -n "$LOOP_DEV" ];then
-		_destroy_loop_device $LOOP_DEV 2>/dev/null
-	fi
-	if [ -n "$LOOP_SPARE_DEV" ];then
-		_destroy_loop_device $LOOP_SPARE_DEV 2>/dev/null
-	fi
+	[ -n $loop_dev ] &&_destroy_loop_device $loop_dev
+	[ -n $loop_spare_dev ] &&_destroy_loop_device $loop_spare_dev
 	rm -f $LOOP_IMG
 	rm -f $LOOP_SPARE_IMG
 	rmdir $LOOP_MNT
@@ -42,11 +38,11 @@ LOOP_MNT=$TEST_DIR/$seq.mnt
 
 echo "** create loop device"
 $XFS_IO_PROG -f -c "truncate 32g" $LOOP_IMG
-LOOP_DEV=`_create_loop_device $LOOP_IMG`
+loop_dev=`_create_loop_device $LOOP_IMG`
 
 echo "** create loop log device"
 $XFS_IO_PROG -f -c "truncate 1g" $LOOP_SPARE_IMG
-LOOP_SPARE_DEV=`_create_loop_device $LOOP_SPARE_IMG`
+loop_spare_dev=`_create_loop_device $LOOP_SPARE_IMG`
 
 echo "** create loop mount point"
 rmdir $LOOP_MNT 2>/dev/null
@@ -55,8 +51,8 @@ mkdir -p $LOOP_MNT || _fail "cannot create loopback mount point"
 filter_loop()
 {
 	sed -e "s,\B$LOOP_MNT,LOOP_MNT,g" \
-	    -e "s,\B$LOOP_DEV,LOOP_DEV,g" \
-	    -e "s,\B$LOOP_SPARE_DEV,LOOP_SPARE_DEV,g"
+	    -e "s,\B$loop_dev,LOOP_DEV,g" \
+	    -e "s,\B$loop_spare_dev,LOOP_SPARE_DEV,g"
 }
 
 filter_xfs_opt()
@@ -69,22 +65,22 @@ MKFS_OPTIONS=""
 do_mkfs()
 {
 	echo "FORMAT: $@" | filter_loop | tee -a $seqres.full
-	$MKFS_XFS_PROG -f $* $LOOP_DEV | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
+	$MKFS_XFS_PROG -f $* $loop_dev | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
 	if [ "${PIPESTATUS[0]}" -ne 0 ]; then
-		_fail "Fails on _mkfs_dev $* $LOOP_DEV"
+		_fail "Fails on _mkfs_dev $* $loop_dev"
 	fi
 	. $tmp.mkfs
 }
 
 is_dev_mounted()
 {
-	findmnt --source $LOOP_DEV >/dev/null
+	findmnt --source $loop_dev >/dev/null
 	return $?
 }
 
 get_mount_info()
 {
-	findmnt --source $LOOP_DEV -o OPTIONS -n
+	findmnt --source $loop_dev -o OPTIONS -n
 }
 
 force_unmount()
@@ -103,29 +99,29 @@ _do_test()
 	local info
 
 	# mount test
-	_mount $LOOP_DEV $LOOP_MNT $opts 2>>$seqres.full
+	_mount $loop_dev $LOOP_MNT $opts 2>>$seqres.full
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "${mounted}" = "fail" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to fail, but it succeeded"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -ne 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs not mounted even mount return 0"
 			return 1
 		fi
 	else
 		if [ "${mounted}" = "pass" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to succeed, but it failed"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -eq 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs is mounted even mount return non-zero"
 			return 1
 		fi
@@ -141,13 +137,13 @@ _do_test()
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "$found" != "true" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expected to find \"$key\" in mount info \"$info\""
 			return 1
 		fi
 	else
 		if [ "$found" != "false" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: did not expect to find \"$key\" in \"$info\""
 			return 1
 		fi
@@ -253,9 +249,9 @@ do_test "-o logbsize=512k" fail
 # Test logdev
 do_mkfs
 do_test "" pass "logdev" "false"
-do_test "-o logdev=$LOOP_SPARE_DEV" fail
-do_mkfs -l logdev=$LOOP_SPARE_DEV
-do_test "-o logdev=$LOOP_SPARE_DEV" pass "logdev=$LOOP_SPARE_DEV" "true"
+do_test "-o logdev=$loop_spare_dev" fail
+do_mkfs -l logdev=$loop_spare_dev
+do_test "-o logdev=$loop_spare_dev" pass "logdev=$loop_spare_dev" "true"
 do_test "" fail
 
 # Test noalign
tests/xfs/521 +6 -2
index 13982c440..c92c621a2 100755
@@ -21,7 +21,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -35,7 +35,7 @@ _require_no_large_scratch_dev
 
 echo "Create fake rt volume"
 truncate -s 400m $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Format and mount 100m rt volume"
 export USE_EXTERNAL=yes
@@ -69,6 +69,10 @@ cp -p $testdir/original $testdir/copy3
 echo "Check filesystem"
 _check_scratch_fs
 
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/528 +7 -3
index 6ca9a2370..a1efbbd27 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -155,11 +155,11 @@ test_ops() {
 
 echo "Create fake rt volume"
 $XFS_IO_PROG -f -c "truncate 400m" $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Make sure synth rt volume works"
 export USE_EXTERNAL=yes
-export SCRATCH_RTDEV=$rtdev
+export SCRATCH_RTDEV=$rt_loop_dev
 _scratch_mkfs > $seqres.full
 _try_scratch_mount || \
 	_notrun "Could not mount with synthetic rt volume"
@@ -170,6 +170,10 @@ test_ops 262144
 # not a power of two
 test_ops 327680
 
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
+
 # success, all done
 status=0
 exit
tests/xfs/530 +6 -5
index 8a182bd6a..d0d0e2665 100755
@@ -15,7 +15,7 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount >> $seqres.full 2>&1
-	test -e "$rtdev" && losetup -d $rtdev >> $seqres.full 2>&1
+	[ -n "$rt_loop_dev" ] && _destroy_loop_device $rt_loop_dev
 	rm -f $tmp.* $TEST_DIR/$seq.rtvol
 }
 
@@ -52,12 +52,12 @@ fi
 
 rtdevsz=$((nr_bits * rtextsz))
 truncate -s $rtdevsz $TEST_DIR/$seq.rtvol
-rtdev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
+rt_loop_dev=$(_create_loop_device $TEST_DIR/$seq.rtvol)
 
 echo "Format and mount rt volume"
 
 export USE_EXTERNAL=yes
-export SCRATCH_RTDEV=$rtdev
+export SCRATCH_RTDEV=$rt_loop_dev
 _scratch_mkfs -d size=$((1024 * 1024 * 1024)) \
 	      -r size=${rtextsz},extsize=${rtextsz} >> $seqres.full
 _try_scratch_mount || _notrun "Couldn't mount fs with synthetic rt volume"
@@ -116,8 +116,9 @@ done
 echo "Check filesystem"
 _check_scratch_fs
 
-losetup -d $rtdev
-rm -f $TEST_DIR/$seq.rtvol
+_scratch_unmount
+_destroy_loop_device $rt_loop_dev
+unset rt_loop_dev
 
 # success, all done
 status=0
tests/xfs/606 +8 -4
index f958bddd8..b537ea145 100755
@@ -13,10 +13,8 @@ _begin_fstest auto quick growfs
 
 _cleanup()
 {
-	local dev
 	_unmount $LOOP_MNT 2>/dev/null
-	dev=$(losetup -j testfile | cut -d: -f1)
-	losetup -d $dev 2>/dev/null
+	[ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
 	rm -rf $LOOP_IMG $LOOP_MNT
 	cd /
 	rm -f $tmp.*
@@ -40,7 +38,9 @@ $MKFS_XFS_PROG -f $LOOP_IMG >$seqres.full
 # Extend by just 8K, expected to start with the last full-size AG ends of
 # above 1G block device.
 $XFS_IO_PROG -f -c "truncate 1073750016" $LOOP_IMG
-_mount -oloop $LOOP_IMG $LOOP_MNT
+
+loop_dev=$(_create_loop_device $LOOP_IMG)
+_mount $loop_dev $LOOP_MNT
 # A known bug shows "XFS_IOC_FSGROWFSDATA xfsctl failed: No space left on
 # device" at here, refer to _fixed_by_kernel_commit above
 $XFS_GROWFS_PROG $LOOP_MNT >$seqres.full
@@ -48,6 +48,10 @@ if [ $? -ne 0 ];then
 	echo "xfs_growfs fails!"
 fi
 
+_unmount $LOOP_MNT
+_destroy_loop_device $loop_dev
+unset loop_dev
+
 echo "Silence is golden"
 # success, all done
 status=0
tests/xfs/613 +14 -26
index 6ba3d87bf..b87cae91e 100755
@@ -15,14 +15,8 @@ _cleanup()
 	cd /
 	rm -f $tmp.*
 	_unmount $LOOP_MNT 2>/dev/null
-	if [ -n "$LOOP_DEV" ];then
-		_destroy_loop_device $LOOP_DEV 2>/dev/null
-	fi
-	if [ -n "$LOOP_SPARE_DEV" ];then
-		_destroy_loop_device $LOOP_SPARE_DEV 2>/dev/null
-	fi
+	[ -n $loop_dev ] &&_destroy_loop_device $loop_dev
 	rm -f $LOOP_IMG
-	rm -f $LOOP_SPARE_IMG
 	rmdir $LOOP_MNT
 }
 
@@ -38,16 +32,11 @@ _require_loop
 _require_xfs_io_command "falloc"
 
 LOOP_IMG=$TEST_DIR/$seq.dev
-LOOP_SPARE_IMG=$TEST_DIR/$seq.logdev
 LOOP_MNT=$TEST_DIR/$seq.mnt
 
 echo "** create loop device"
 $XFS_IO_PROG -f -c "truncate 32g" $LOOP_IMG
-LOOP_DEV=`_create_loop_device $LOOP_IMG`
-
-echo "** create loop log device"
-$XFS_IO_PROG -f -c "truncate 1g" $LOOP_SPARE_IMG
-LOOP_SPARE_DEV=`_create_loop_device $LOOP_SPARE_IMG`
+loop_dev=`_create_loop_device $LOOP_IMG`
 
 echo "** create loop mount point"
 rmdir $LOOP_MNT 2>/dev/null
@@ -56,8 +45,7 @@ mkdir -p $LOOP_MNT || _fail "cannot create loopback mount point"
 filter_loop()
 {
 	sed -e "s,\B$LOOP_MNT,LOOP_MNT,g" \
-	    -e "s,\B$LOOP_DEV,LOOP_DEV,g" \
-	    -e "s,\B$LOOP_SPARE_DEV,LOOP_SPARE_DEV,g"
+	    -e "s,\B$loop_dev,LOOP_DEV,g"
 }
 
 filter_xfs_opt()
@@ -70,22 +58,22 @@ MKFS_OPTIONS=""
 do_mkfs()
 {
 	echo "FORMAT: $@" | filter_loop | tee -a $seqres.full
-	$MKFS_XFS_PROG -f $* $LOOP_DEV | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
+	$MKFS_XFS_PROG -f $* $loop_dev | _filter_mkfs >>$seqres.full 2>$tmp.mkfs
 	if [ "${PIPESTATUS[0]}" -ne 0 ]; then
-		_fail "Fails on _mkfs_dev $* $LOOP_DEV"
+		_fail "Fails on _mkfs_dev $* $loop_dev"
 	fi
 	. $tmp.mkfs
 }
 
 is_dev_mounted()
 {
-	findmnt --source $LOOP_DEV >/dev/null
+	findmnt --source $loop_dev >/dev/null
 	return $?
 }
 
 get_mount_info()
 {
-	findmnt --source $LOOP_DEV -o OPTIONS -n
+	findmnt --source $loop_dev -o OPTIONS -n
 }
 
 force_unmount()
@@ -104,29 +92,29 @@ _do_test()
 	local info
 
 	# mount test
-	_mount $LOOP_DEV $LOOP_MNT $opts 2>>$seqres.full
+	_mount $loop_dev $LOOP_MNT $opts 2>>$seqres.full
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "${mounted}" = "fail" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to fail, but it succeeded"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -ne 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs not mounted even mount return 0"
 			return 1
 		fi
 	else
 		if [ "${mounted}" = "pass" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expect mount to succeed, but it failed"
 			return 1
 		fi
 		is_dev_mounted
 		if [ $? -eq 0 ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: fs is mounted even mount return non-zero"
 			return 1
 		fi
@@ -142,13 +130,13 @@ _do_test()
 	rc=$?
 	if [ $rc -eq 0 ];then
 		if [ "$found" != "true" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: expected to find \"$key\" in mount info \"$info\""
 			return 1
 		fi
 	else
 		if [ "$found" != "false" ];then
-			echo "[FAILED]: mount $LOOP_DEV $LOOP_MNT $opts"
+			echo "[FAILED]: mount $loop_dev $LOOP_MNT $opts"
 			echo "ERROR: did not expect to find \"$key\" in \"$info\""
 			return 1
 		fi
tests/xfs/613.out +0 -1
index 1624617ee..2a693c53c 100644
@@ -1,6 +1,5 @@
 QA output created by 613
 ** create loop device
-** create loop log device
 ** create loop mount point
 ** start xfs mount testing ...
 FORMAT: -m crc=0
Signature · 1 line
2.45.2
#26 lore raw

PATCH 14/40 fstests: xfs/227 is really slow

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

The slowest test tto run on my test VMs is xfs/227:

...
xfs/227 826
...

It is doing nested iteration on created filesets that are explicitly defined, so separate the inner loop filesets and run the outer loops in parallel.

Also reduce the number of times we have to execute setfattr and xfs_io to once per created file instead of once per xattr/extent count per file.

The result is test runtime reduction of ~60%.

....
xfs/227 336
....

---
 tests/xfs/227 | 57 ++++++++++++++++++++++++++++++++-------------------
 1 file changed, 36 insertions(+), 21 deletions(-)
tests/xfs/227 +36 -21
index 2ad910cbd..7f8ee2dba 100755
@@ -41,7 +41,7 @@ fragment_freespace()
 	# allocate inode space
 	mkdir -p $_dir
 	for i in `seq 0 1 1000`; do
-		touch $_file.$i
+		echo -n > $_file.$i
 	done
 	for i in `seq 0 63 1000`; do
 		mv $_file.$i $_dir
@@ -68,19 +68,23 @@ fragment_freespace()
 
 create_attrs()
 {
-	for foo in `seq 0 1 $1`; do
-		$SETFATTR_PROG -n user.$foo -v 0xbabe $2
-	done
+	( echo "# file: $2"
+	for i in `seq 0 1 $1`; do
+		echo "user.$foo=\"0xbabe\""
+	done ) > $tmp.$1.attrs
+	$SETFATTR_PROG --restore=$tmp.$1.attrs $2
 }
 
 create_data()
 {
-	size=`expr \( $1 + 1 \) \* 4096`
-	$XFS_IO_PROG -f -c "truncate $size" $2 > /dev/null 2>&1
-	for foo in `seq $1 -1 0`; do
-		let offset=$foo*4096
-		$XFS_IO_PROG -f -c "resvsp $offset 4096" $2 > /dev/null 2>&1
+	local blocks=$1
+	local cmd_str
+
+	for off in `seq $blocks -1 0`; do
+		cmd_str="-c \"resvsp $((off * 4096)) 4096\" $cmd_str"
 	done
+	$XFS_IO_PROG -f -c "truncate $((($blocks + 1) * 4096))" \
+			$cmd_str $2 >> seqres.full 2>&1
 }
 
 # create the designated file with a certain number of attributes and a certain
@@ -119,6 +123,27 @@ create_target_attr_last()
 	create_attrs $nattrs $target
 }
 
+do_fsr()
+{
+	local n=$1
+	local i=$2
+	local j
+
+	for j in `seq 5 1 20`; do
+		create_target_attr_first $i $j $targ.$i.$j >> $seqres.full 2>&1
+	done
+	xfs_bmap -vp $targ.$i.* >> $seqres.full 2>&1
+	FSRXFSTEST=true xfs_fsr -d -v -C $n $targ.$i.* >> $seqres.full 2>&1
+	xfs_bmap -vp $targ.$i.* >> $seqres.full 2>&1
+	for j in `seq 5 1 20`; do
+		create_target_attr_last $i $j $targ.$i.$j >> $seqres.full 2>&1
+	done
+	xfs_bmap -vp $targ.$i.* >> $seqres.full 2>&1
+	FSRXFSTEST=true xfs_fsr -d -v -C $n $targ.$i.* >> $seqres.full 2>&1
+	xfs_bmap -vp $targ.$i.* >> $seqres.full 2>&1
+}
+
+
 # use a small filesystem so we can control freespace easily
 _scratch_mkfs_sized $((50 * 1024 * 1024)) >> $seqres.full 2>&1
 _scratch_mount
@@ -154,19 +179,9 @@ targ=$SCRATCH_MNT/fsr_test_file.$$
 for n in `seq 4 1 12`; do
 	echo "*** n == $n ***" >> $seqres.full
 	for i in `seq 5 1 15`; do
-		for j in `seq 5 1 20`; do
-			create_target_attr_first $i $j $targ.$i.$j >> $seqres.full 2>&1
-		done
-		xfs_bmap -vp $targ.$i.* >> $seqres.full 2>&1
-		FSRXFSTEST=true xfs_fsr -d -v -C $n $targ.$i.* >> $seqres.full 2>&1
-		xfs_bmap -vp $targ.$i.* >> $seqres.full 2>&1
-		for j in `seq 5 1 20`; do
-			create_target_attr_last $i $j $targ.$i.$j >> $seqres.full 2>&1
-		done
-		xfs_bmap -vp $targ.$i.* >> $seqres.full 2>&1
-		FSRXFSTEST=true xfs_fsr -d -v -C $n $targ.$i.* >> $seqres.full 2>&1
-		xfs_bmap -vp $targ.$i.* >> $seqres.full 2>&1
+		do_fsr $n $i &
 	done
+	wait
 done
 
 _scratch_unmount
Signature · 1 line
2.45.2
#27 lore raw

PATCH 15/40 fstests: mark tests that are unreliable when run in parallel

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Add a group named "unreliable_in_parallel" to mark tests that do not give reliable results when multiple tests are run in parallel. Generally this happens with tests that are reliant on caching in some way, such as generating specific file layouts using buffered IO or expecting inodes to be cached in memory. These are perturbed by other tests running sync(), generating memory pressure, dropping caches, etc.

Hence whether these tests pass or fail is wholly dependent on what tests are running at the same time, and hence randomly fail when nothing has actually gone wrong. Hence they are unreliable as regression tests when running tests in parallel, so we add them to the "unreliable_in_parallel" group and a parallel check can exclude this group.

As tests are updated to be robust against external interference, they can be removed from the unreliable_in_parallel group.

---
 doc/group-names.txt | 1 +
 tests/generic/336   | 7 ++++++-
 tests/generic/561   | 8 +++++++-
 tests/xfs/177       | 8 ++++++--
 tests/xfs/232       | 6 +++++-
 tests/xfs/237       | 8 +++++++-
 tests/xfs/243       | 7 +++++--
 tests/xfs/300       | 8 ++++++--
 tests/xfs/440       | 6 +++++-
 tests/xfs/527       | 5 ++++-
 tests/xfs/631       | 7 ++++++-
 tests/xfs/802       | 7 ++++++-
 12 files changed, 64 insertions(+), 14 deletions(-)
doc/group-names.txt +1 -0
index ed886caac..f5bf79a56 100644
@@ -138,6 +138,7 @@ trim			FITRIM ioctl
 udf			UDF functionality tests
 union			tests from the unionmount test suite
 unlink			O_TMPFILE unlinked files
+unreliable_in_parallel	randomly fail when run in parallel with other tests
 unshare			fallocate FALLOC_FL_UNSHARE_RANGE
 v2log			XFS v2 log format tests
 verity			fsverity
tests/generic/336 +6 -1
index 06391a93f..c874997e4 100755
@@ -9,8 +9,13 @@
 # file F2 from directory B into directory C, fsync inode F1, power fail and
 # remount the filesystem, file F2 exists and is located only in directory C.
 #
+
+# unreliable_in_parallel: external sync operations can change what is synced to
+# the log before the flakey device drops writes. hence post-remount file
+# contents can be different to what the test expects.
+
 . ./common/preamble
-_begin_fstest auto quick metadata log
+_begin_fstest auto quick metadata log unreliable_in_parallel
 
 # Override the default cleanup function.
 _cleanup()
tests/generic/561 +7 -1
index 3e931b1a7..602c235bc 100755
@@ -7,8 +7,14 @@
 # Dedup & random I/O race test, do multi-threads fsstress and dedupe on
 # same directory/files
 #
+
+# unreliable_in_parallel: duperemove is buggy. It can get stuck in endless
+# fiemap mapping loops, and this seems to happen a *lot* when the system is
+# under heavy load. when they do this, they don't die when they are supposed to
+# and so have to be manually killed to end the test.
+
 . ./common/preamble
-_begin_fstest auto stress dedupe
+_begin_fstest auto stress dedupe unreliable_in_parallel
 
 # Override the default cleanup function.
 _cleanup()
tests/xfs/177 +6 -2
index 773049524..22719ba1c 100755
@@ -21,9 +21,13 @@
 # Regrettably, there is no way to poke /only/ XFS inode reclamation directly,
 # so we're stuck with setting xfssyncd_centisecs to a low value and sleeping
 # while watching the internal inode cache counters.
-#
+
+# unreliable_in_parallel: cache residency is affected by external drop caches
+# operations. Hence counting inodes "in cache" often does not reflect what the
+# test has actually done.
+
 . ./common/preamble
-_begin_fstest auto ioctl
+_begin_fstest auto ioctl unreliable_in_parallel
 
 _cleanup()
 {
tests/xfs/232 +5 -1
index 0eea2c098..f0f3916e7 100755
@@ -12,8 +12,12 @@
 # - Wait for the reclaim to run.
 # - Write more and see how bad fragmentation is.
 #
+
+# unreliable_in_parallel: external sync operations affect what happens while
+# the test is waiting for COW expiration.
+
 . ./common/preamble
-_begin_fstest auto quick clone fiemap prealloc
+_begin_fstest auto quick clone fiemap prealloc unreliable_in_parallel
 
 # Override the default cleanup function.
 _cleanup()
tests/xfs/237 +7 -1
index f172aaf59..91f56d6c1 100755
@@ -6,8 +6,14 @@
 #
 # Test AIO DIO CoW behavior when the write temporarily fails.
 #
+
+# unreliable_in_parallel: external drop caches can co-incide with the error
+# table being loaded, so the test being run fails with EIO trying to load the
+# inode from disk instead of whatever operation it is supposed to fail on when
+# the inode is already cached in memory.
+
 . ./common/preamble
-_begin_fstest auto quick clone eio
+_begin_fstest auto quick clone eio unreliable_in_parallel
 
 # Override the default cleanup function.
 _cleanup()
tests/xfs/243 +5 -2
index 964e94e1d..f9cc2d50f 100755
@@ -15,9 +15,12 @@
 #     5. delalloc
 #   - CoW across the halfway mark, starting with the unwritten extent.
 #   - Check that the files are now different where we say they're different.
-#
+
+# unreliable_in_parallel: external sync can affect the layout of the files being
+# created, results in unreliable detection of delalloc extents.
+
 . ./common/preamble
-_begin_fstest auto quick clone punch prealloc
+_begin_fstest auto quick clone punch prealloc unreliable_in_parallel
 
 # Import common functions.
 . ./common/filter
tests/xfs/300 +6 -2
index 3f0dbb9ac..c4c3b1ab8 100755
@@ -5,9 +5,13 @@
 # FS QA Test No. 300
 #
 # Test xfs_fsr / exchangerange management of di_forkoff w/ selinux
-#
+
+# unreliable_in_parallel: file layout appears to be perturbed by load related
+# timing issues. Not 100% sure, but the backwards write does not reliably
+# fragment the source file under heavy external load
+
 . ./common/preamble
-_begin_fstest auto fsr
+_begin_fstest auto fsr unreliable_in_parallel
 
 # Import common functions.
 . ./common/filter
tests/xfs/440 +5 -1
index 0cc679aeb..c0b6756ba 100755
@@ -8,8 +8,12 @@
 # a file that has CoW reservations and no dirty pages.  The reservations
 # should shift over to the new owner, but they do not.
 #
+
+# unreliable_in_parallel: external sync(1) and/or drop caches can reclaim inodes
+# and free post-eof space, resulting in lower than expected block counts.
+
 . ./common/preamble
-_begin_fstest auto quick clone quota
+_begin_fstest auto quick clone quota unreliable_in_parallel
 
 # Import common functions.
 . ./common/reflink
tests/xfs/527 +4 -1
index 2ef428c25..0d06b128c 100755
@@ -14,8 +14,11 @@
 # xfs: fix incorrect root dquot corruption error when switching group/project
 # quota types
 
+# unreliable_in_parallel: dmesg check can pick up corruptions from other tests.
+# Need to filter corruption reports by short scratch dev name.
+
 . ./common/preamble
-_begin_fstest auto quick quota
+_begin_fstest auto quick quota unreliable_in_parallel
 
 # Import common functions.
 . ./common/quota
tests/xfs/631 +6 -1
index 4d79b821f..319995f81 100755
@@ -7,8 +7,13 @@
 # Post-EOF preallocation defeat test for direct I/O with extent size hints.
 #
 
+# unreliable_in_parallel: external cache drops can result in the extent size
+# being truncated as the inode is evicted from cache between writes. This can
+# increase the number of extents significantly beyond what would be expected
+# from the extent size hint.
+
 . ./common/preamble
-_begin_fstest auto quick prealloc rw
+_begin_fstest auto quick prealloc rw unreliable_in_parallel
 
 . ./common/filter
 
tests/xfs/802 +6 -1
index ea09817fd..fc4767acb 100755
@@ -8,8 +8,13 @@
 # filesystem, and that we can read the health reports after the fact.  IOWs,
 # this is basic testing for the systemd background services.
 #
+
+# unreliable_in_parallel: this appears to try to run scrub services on all
+# mounted filesystems - that's aproblem when there are a hundred other test
+# filesystems mounted running other tests...
+
 . ./common/preamble
-_begin_fstest auto scrub
+_begin_fstest auto scrub unreliable_in_parallel
 
 _cleanup()
 {
Signature · 1 line
2.45.2
#28 lore raw

PATCH 16/40 fstests: use udevadm wait in preference to settle

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

When running lots of tests in parallel, there are lots of filesystems and block devices changing state. This generates a lot of udev events when means the udev event queue is rarely empty. Unfortunately, an empty event queue is what udev settling waits upon. Hence calling UDEV_SETTLE_PROG can mean waiting for a lot of time for other tests to stop generating udev events.

For the majority of cases, what we care about is that udev has performed device node addition or removal, not that there are no udev events pending. Recent(-ish) systemd releases support 'udevadm wait' to wait for a specific file to be created or unlinked rather than waiting for the event that does that work to be completed.

Hence we don't have to wait for the udev event queue to empty, just for the udev event that does the device node manipulation to complete.

Introduce detection of 'udevadm wait' support and a _udev_wait() wrapper function to use it if it is available. If it isn't, the use the existing UDEV_SETTLE_PROG behaviour.

---
 common/config     | 35 +++++++++++++++++++++++++----------
 common/rc         | 25 ++++++++++++++++---------
 tests/btrfs/291   |  5 +++--
 tests/generic/081 |  6 +++---
 tests/generic/108 |  7 +++----
 tests/generic/459 |  6 +++---
 6 files changed, 53 insertions(+), 31 deletions(-)
common/config +25 -10
index fcff0660b..41b8f29d1 100644
@@ -165,7 +165,7 @@ export XFS_MDRESTORE_PROG="$(type -P xfs_mdrestore)"
 export XFS_ADMIN_PROG="$(type -P xfs_admin)"
 export XFS_GROWFS_PROG=$(type -P xfs_growfs)
 export XFS_SPACEMAN_PROG="$(type -P xfs_spaceman)"
-export XFS_SCRUB_PROG="$(type -P xfs_scrub)"
+#export XFS_SCRUB_PROG="$(type -P xfs_scrub)"
 export XFS_PARALLEL_REPAIR_PROG="$(type -P xfs_prepair)"
 export XFS_PARALLEL_REPAIR64_PROG="$(type -P xfs_prepair64)"
 export __XFSDUMP_PROG="$(type -P xfsdump)"
@@ -236,18 +236,30 @@ export BTRFS_MAP_LOGICAL_PROG=$(type -P btrfs-map-logical)
 export PARTED_PROG="$(type -P parted)"
 export XFS_PROPERTY_PROG="$(type -P xfs_property)"
 
-# use 'udevadm settle' or 'udevsettle' to wait for lv to be settled.
-# newer systems have udevadm command but older systems like RHEL5 don't.
-# But if neither one is available, just set it to "sleep 1" to wait for lv to
-# be settled
-UDEV_SETTLE_PROG="$(type -P udevadm)"
-if [ "$UDEV_SETTLE_PROG" == "" ]; then
-	# try udevsettle command
+# udev wait functions.
+#
+# This is how we wait for udev to create or remove device nodes after running a
+# device create/remove command for logical volumes (e.g. lvm or dm).
+#
+# We can wait for the udev queue to empty via "settling". This, however, has
+# major issues when running tests in parallel - the udev queue takes a long time
+# to reach empty state. Hence if we have udev > 2.51 installed we use device
+# waiting instead. This waits for the device node to appear/disappear rather
+# than waiting for the udev queue to empty.
+#
+# If none of these methods are available, fall back to a simple delay (sleep 1)
+# and hope this is sufficient.
+UDEVADM_PROG="$(type -P udevadm)"
+if [ -z "$UDEVADM_PROG" ]; then
 	UDEV_SETTLE_PROG="$(type -P udevsettle)"
 else
-	# udevadm is available, add 'settle' as subcommand
-	UDEV_SETTLE_PROG="$UDEV_SETTLE_PROG settle"
+	UDEV_SETTLE_PROG="$UDEVADM_PROG settle"
+	$UDEVADM_PROG help | grep -q "Wait for device or device symlink"
+	if [ $? -eq 0 ]; then
+		UDEV_WAIT_PROG="$UDEVADM_PROG wait"
+	fi
 fi
+
 # neither command is available, use sleep 1
 #
 # Udev events are sent via netlink to userspace through
@@ -258,8 +270,11 @@ fi
 # exist or always be 0. We check for /proc/net to see CONFIG_NET was enabled.
 if [[ "$UDEV_SETTLE_PROG" == "" || ! -d /proc/net ]]; then
 	UDEV_SETTLE_PROG="sleep 1"
+	unset UDEV_WAIT_PROG
 fi
 export UDEV_SETTLE_PROG
+export UDEVADM_PROG
+export UDEV_WAIT_PROG
 
 # Set MODPROBE_PATIENT_RM_TIMEOUT_SECONDS to "forever" if you want the patient
 # modprobe removal to run forever trying to remove a module.
common/rc +16 -9
index 3f35da7fe..fdd18a386 100644
@@ -5191,22 +5191,29 @@ _require_label_get_max()
 	dummy=$(_label_get_max)
 }
 
+_udev_wait()
+{
+	local args="$*"
+
+	if [ -z "$UDEV_WAIT_PROG" ]; then
+		$UDEV_SETTLE_PROG >/dev/null 2>&1
+	else
+		$UDEV_WAIT_PROG $args
+	fi
+}
+
 _dmsetup_remove()
 {
-	$UDEV_SETTLE_PROG >/dev/null 2>&1
-	$DMSETUP_PROG remove --retry "$@" >>$seqres.full 2>&1
-	$UDEV_SETTLE_PROG >/dev/null 2>&1
+	[ $# -le 0 ] && return
+
+	$DMSETUP_PROG remove --deferred "$@" >>$seqres.full 2>&1
+	_udev_wait --removed /dev/mapper/$1
 }
 
 _dmsetup_create()
 {
-	# Wait for udev to settle so that the dm creation doesn't fail because
-	# some udev subprogram opened one of the block devices mentioned in the
-	# table string w/ O_EXCL.  Do it again at the end so that an immediate
-	# device open won't also fail.
-	$UDEV_SETTLE_PROG >/dev/null 2>&1
 	$DMSETUP_PROG create "$@" >>$seqres.full 2>&1 || return 1
-	$UDEV_SETTLE_PROG >/dev/null 2>&1
+	_udev_wait /dev/mapper/$1
 }
 
 _require_btime()
tests/btrfs/291 +3 -2
index c31de3a96..122aeaa5d 100755
@@ -21,6 +21,7 @@ _cleanup()
 	cd /
 	_log_writes_cleanup &> /dev/null
 	$LVM_PROG vgremove -f -y $vgname >>$seqres.full 2>&1
+	_udev_wait --removed /dev/mapper/$vgname-$lvname
 	losetup -d $loop_dev >>$seqres.full 2>&1
 	rm -f $img
 	_restore_fsverity_signatures
@@ -106,7 +107,7 @@ snap_dev=/dev/mapper/vg_replay-$snapname
 $LVM_PROG vgcreate -f $vgname $loop_dev >>$seqres.full 2>&1 || _fail "failed to vgcreate $vgname"
 $LVM_PROG lvcreate -L "$replay_bytes"B -n $lvname $vgname -y >>$seqres.full 2>&1 || \
 	_fail "failed to lvcreate $lvname"
-$UDEV_SETTLE_PROG >>$seqres.full 2>&1
+_udev_wait /dev/mapper/$vgname-$lvname
 
 replay_log_prog=$here/src/log-writes/replay-log
 num_entries=$($replay_log_prog --log $LOGWRITES_DEV --num-entries)
@@ -125,7 +126,7 @@ do
 
 	$LVM_PROG lvcreate -s -L 4M -n $snapname $vgname/$lvname >>$seqres.full 2>&1 || \
 		_fail "Failed to create snapshot"
-	$UDEV_SETTLE_PROG >>$seqres.full 2>&1
+	_udev_wait /dev/mapper/$vgname-$snapname
 
 	orphan=$(count_item $snap_dev ORPHAN)
 	[ $state -eq 0 ] && [ $orphan -gt 0 ] && state=1
tests/generic/081 +3 -3
index df17ab6c1..37137d937 100755
@@ -38,7 +38,7 @@ _cleanup()
 		$LVM_PROG vgremove -f $vgname >>$seqres.full 2>&1
 		$LVM_PROG pvremove -f $SCRATCH_DEV >>$seqres.full 2>&1
 		pv_ret=$?
-		$UDEV_SETTLE_PROG
+		_udev_wait --removed /dev/mapper/$vgname-$lvname
 		test $pv_ret -eq 0 && break
 		sleep 2
 	done
@@ -70,8 +70,8 @@ $LVM_PROG vgcreate -f $vgname $SCRATCH_DEV >>$seqres.full 2>&1
 # We use yes pipe instead of 'lvcreate --yes' because old version of lvm
 # (like 2.02.95 in RHEL6) don't support --yes option
 yes | $LVM_PROG lvcreate -L ${lvsize}M -n $lvname $vgname >>$seqres.full 2>&1
-# wait for lvcreation to fully complete
-$UDEV_SETTLE_PROG >>$seqres.full 2>&1
+_udev_wait /dev/mapper/$vgname-$lvname
+
 
 # _mkfs_dev exits the test on failure, this can make sure lv is created in
 # above vgcreate/lvcreate steps
tests/generic/108 +3 -4
index 2709472f6..f630450ec 100755
@@ -20,8 +20,8 @@ _cleanup()
 	echo running > /sys/block/`_short_dev $SCSI_DEBUG_DEV`/device/state
 	_unmount $SCRATCH_MNT >>$seqres.full 2>&1
 	$LVM_PROG vgremove -f $vgname >>$seqres.full 2>&1
-	$LVM_PROG pvremove -f $SCRATCH_DEV $SCSI_DEBUG_DEV >>$seqres.full 2>&1
-	$UDEV_SETTLE_PROG
+	 pvremove -f $SCRATCH_DEV $SCSI_DEBUG_DEV >>$seqres.full 2>&1
+	_udev_wait --removed /dev/mapper/$vgname-$lvname
 	_put_scsi_debug_dev
 	rm -f $tmp.*
 }
@@ -57,8 +57,7 @@ $LVM_PROG vgcreate -f $vgname $SCSI_DEBUG_DEV $SCRATCH_DEV >>$seqres.full 2>&1
 # (like 2.02.95 in RHEL6) don't support --yes option
 yes | $LVM_PROG lvcreate -i 2 -I 4m -L ${lvsize}m -n $lvname $vgname \
 	>>$seqres.full 2>&1
-# wait for lv creation to fully complete
-$UDEV_SETTLE_PROG >>$seqres.full 2>&1
+_udev_wait /dev/mapper/$vgname-$lvname
 
 # _mkfs_dev exits the test on failure, this makes sure test lv is created by
 # above vgcreate/lvcreate operations
tests/generic/459 +3 -3
index daccc80ce..1986c2e8f 100755
@@ -31,7 +31,7 @@ _cleanup()
 	_unmount $SCRATCH_MNT >>$seqres.full 2>&1
 	$LVM_PROG vgremove -ff $vgname >>$seqres.full 2>&1
 	$LVM_PROG pvremove -ff $SCRATCH_DEV >>$seqres.full 2>&1
-	$UDEV_SETTLE_PROG
+	_udev_wait --removed /dev/mapper/$vgname-$lvname
 }
 
 # Import common functions.
@@ -88,8 +88,7 @@ $LVM_PROG lvcreate  --thinpool $poolname  --errorwhenfull y \
 $LVM_PROG lvcreate  --virtualsize $virtsize \
 		    -T $vgname/$poolname \
 		    -n $lvname >>$seqres.full 2>&1
-
-$UDEV_SETTLE_PROG &>/dev/null
+_udev_wait /dev/mapper/$vgname-$lvname
 _mkfs_dev /dev/mapper/$vgname-$lvname >>$seqres.full 2>&1
 
 # Running the test over the original volume doesn't reproduce the problem
@@ -97,6 +96,7 @@ _mkfs_dev /dev/mapper/$vgname-$lvname >>$seqres.full 2>&1
 # reproducible, so, create a snapshot and run the test over it.
 $LVM_PROG lvcreate  -k n -s $vgname/$lvname \
 		    -n $snapname >>$seqres.full 2>&1
+_udev_wait /dev/mapper/$vgname-$snapname
 
 # Catch mount failure so we don't blindly go an freeze the root filesystem
 # instead of lvm volume.
Signature · 1 line
2.45.2
#29 lore raw

Re: [PATCH 16/40] fstests: use udevadm wait in preference to settle

Quoted 38 lines · Dave Chinner

From: Dave Chinner <[email protected]>

When running lots of tests in parallel, there are lots of filesystems and block devices changing state. This generates a lot of udev events when means the udev event queue is rarely empty. Unfortunately, an empty event queue is what udev settling waits upon. Hence calling UDEV_SETTLE_PROG can mean waiting for a lot of time for other tests to stop generating udev events.

For the majority of cases, what we care about is that udev has performed device node addition or removal, not that there are no udev events pending. Recent(-ish) systemd releases support 'udevadm wait' to wait for a specific file to be created or unlinked rather than waiting for the event that does that work to be completed.

Hence we don't have to wait for the udev event queue to empty, just for the udev event that does the device node manipulation to complete.

Introduce detection of 'udevadm wait' support and a _udev_wait() wrapper function to use it if it is available. If it isn't, the use the existing UDEV_SETTLE_PROG behaviour.

---
 common/config     | 35 +++++++++++++++++++++++++----------
 common/rc         | 25 ++++++++++++++++---------
 tests/btrfs/291   |  5 +++--
 tests/generic/081 |  6 +++---
 tests/generic/108 |  7 +++----
 tests/generic/459 |  6 +++---
 6 files changed, 53 insertions(+), 31 deletions(-)
common/config +0 -0
index fcff0660b..41b8f29d1 100644

common/config export XFS_MDRESTORE_PROG="$(type -P xfs_mdrestore)" · line 168

Quoted 5 lines
common/config +1 -1
@@ -165,7 +165,7 @@ export XFS_MDRESTORE_PROG="$(type -P xfs_mdrestore)"
 export XFS_ADMIN_PROG="$(type -P xfs_admin)"
 export XFS_GROWFS_PROG=$(type -P xfs_growfs)
 export XFS_SPACEMAN_PROG="$(type -P xfs_spaceman)"
-export XFS_SCRUB_PROG="$(type -P xfs_scrub)"
+#export XFS_SCRUB_PROG="$(type -P xfs_scrub)"

If you have problems with online fsck, please report them to the mailing list.

--D

Quoted 212 lines
common/config +24 -9
 export XFS_PARALLEL_REPAIR_PROG="$(type -P xfs_prepair)"
 export XFS_PARALLEL_REPAIR64_PROG="$(type -P xfs_prepair64)"
 export __XFSDUMP_PROG="$(type -P xfsdump)"
@@ -236,18 +236,30 @@ export BTRFS_MAP_LOGICAL_PROG=$(type -P btrfs-map-logical)
 export PARTED_PROG="$(type -P parted)"
 export XFS_PROPERTY_PROG="$(type -P xfs_property)"
 
-# use 'udevadm settle' or 'udevsettle' to wait for lv to be settled.
-# newer systems have udevadm command but older systems like RHEL5 don't.
-# But if neither one is available, just set it to "sleep 1" to wait for lv to
-# be settled
-UDEV_SETTLE_PROG="$(type -P udevadm)"
-if [ "$UDEV_SETTLE_PROG" == "" ]; then
-	# try udevsettle command
+# udev wait functions.
+#
+# This is how we wait for udev to create or remove device nodes after running a
+# device create/remove command for logical volumes (e.g. lvm or dm).
+#
+# We can wait for the udev queue to empty via "settling". This, however, has
+# major issues when running tests in parallel - the udev queue takes a long time
+# to reach empty state. Hence if we have udev > 2.51 installed we use device
+# waiting instead. This waits for the device node to appear/disappear rather
+# than waiting for the udev queue to empty.
+#
+# If none of these methods are available, fall back to a simple delay (sleep 1)
+# and hope this is sufficient.
+UDEVADM_PROG="$(type -P udevadm)"
+if [ -z "$UDEVADM_PROG" ]; then
 	UDEV_SETTLE_PROG="$(type -P udevsettle)"
 else
-	# udevadm is available, add 'settle' as subcommand
-	UDEV_SETTLE_PROG="$UDEV_SETTLE_PROG settle"
+	UDEV_SETTLE_PROG="$UDEVADM_PROG settle"
+	$UDEVADM_PROG help | grep -q "Wait for device or device symlink"
+	if [ $? -eq 0 ]; then
+		UDEV_WAIT_PROG="$UDEVADM_PROG wait"
+	fi
 fi
+
 # neither command is available, use sleep 1
 #
 # Udev events are sent via netlink to userspace through
@@ -258,8 +270,11 @@ fi
 # exist or always be 0. We check for /proc/net to see CONFIG_NET was enabled.
 if [[ "$UDEV_SETTLE_PROG" == "" || ! -d /proc/net ]]; then
 	UDEV_SETTLE_PROG="sleep 1"
+	unset UDEV_WAIT_PROG
 fi
 export UDEV_SETTLE_PROG
+export UDEVADM_PROG
+export UDEV_WAIT_PROG
 
 # Set MODPROBE_PATIENT_RM_TIMEOUT_SECONDS to "forever" if you want the patient
 # modprobe removal to run forever trying to remove a module.
common/rc +16 -9
index 3f35da7fe..fdd18a386 100644
@@ -5191,22 +5191,29 @@ _require_label_get_max()
 	dummy=$(_label_get_max)
 }
 
+_udev_wait()
+{
+	local args="$*"
+
+	if [ -z "$UDEV_WAIT_PROG" ]; then
+		$UDEV_SETTLE_PROG >/dev/null 2>&1
+	else
+		$UDEV_WAIT_PROG $args
+	fi
+}
+
 _dmsetup_remove()
 {
-	$UDEV_SETTLE_PROG >/dev/null 2>&1
-	$DMSETUP_PROG remove --retry "$@" >>$seqres.full 2>&1
-	$UDEV_SETTLE_PROG >/dev/null 2>&1
+	[ $# -le 0 ] && return
+
+	$DMSETUP_PROG remove --deferred "$@" >>$seqres.full 2>&1
+	_udev_wait --removed /dev/mapper/$1
 }
 
 _dmsetup_create()
 {
-	# Wait for udev to settle so that the dm creation doesn't fail because
-	# some udev subprogram opened one of the block devices mentioned in the
-	# table string w/ O_EXCL.  Do it again at the end so that an immediate
-	# device open won't also fail.
-	$UDEV_SETTLE_PROG >/dev/null 2>&1
 	$DMSETUP_PROG create "$@" >>$seqres.full 2>&1 || return 1
-	$UDEV_SETTLE_PROG >/dev/null 2>&1
+	_udev_wait /dev/mapper/$1
 }
 
 _require_btime()
tests/btrfs/291 +3 -2
index c31de3a96..122aeaa5d 100755
@@ -21,6 +21,7 @@ _cleanup()
 	cd /
 	_log_writes_cleanup &> /dev/null
 	$LVM_PROG vgremove -f -y $vgname >>$seqres.full 2>&1
+	_udev_wait --removed /dev/mapper/$vgname-$lvname
 	losetup -d $loop_dev >>$seqres.full 2>&1
 	rm -f $img
 	_restore_fsverity_signatures
@@ -106,7 +107,7 @@ snap_dev=/dev/mapper/vg_replay-$snapname
 $LVM_PROG vgcreate -f $vgname $loop_dev >>$seqres.full 2>&1 || _fail "failed to vgcreate $vgname"
 $LVM_PROG lvcreate -L "$replay_bytes"B -n $lvname $vgname -y >>$seqres.full 2>&1 || \
 	_fail "failed to lvcreate $lvname"
-$UDEV_SETTLE_PROG >>$seqres.full 2>&1
+_udev_wait /dev/mapper/$vgname-$lvname
 
 replay_log_prog=$here/src/log-writes/replay-log
 num_entries=$($replay_log_prog --log $LOGWRITES_DEV --num-entries)
@@ -125,7 +126,7 @@ do
 
 	$LVM_PROG lvcreate -s -L 4M -n $snapname $vgname/$lvname >>$seqres.full 2>&1 || \
 		_fail "Failed to create snapshot"
-	$UDEV_SETTLE_PROG >>$seqres.full 2>&1
+	_udev_wait /dev/mapper/$vgname-$snapname
 
 	orphan=$(count_item $snap_dev ORPHAN)
 	[ $state -eq 0 ] && [ $orphan -gt 0 ] && state=1
tests/generic/081 +3 -3
index df17ab6c1..37137d937 100755
@@ -38,7 +38,7 @@ _cleanup()
 		$LVM_PROG vgremove -f $vgname >>$seqres.full 2>&1
 		$LVM_PROG pvremove -f $SCRATCH_DEV >>$seqres.full 2>&1
 		pv_ret=$?
-		$UDEV_SETTLE_PROG
+		_udev_wait --removed /dev/mapper/$vgname-$lvname
 		test $pv_ret -eq 0 && break
 		sleep 2
 	done
@@ -70,8 +70,8 @@ $LVM_PROG vgcreate -f $vgname $SCRATCH_DEV >>$seqres.full 2>&1
 # We use yes pipe instead of 'lvcreate --yes' because old version of lvm
 # (like 2.02.95 in RHEL6) don't support --yes option
 yes | $LVM_PROG lvcreate -L ${lvsize}M -n $lvname $vgname >>$seqres.full 2>&1
-# wait for lvcreation to fully complete
-$UDEV_SETTLE_PROG >>$seqres.full 2>&1
+_udev_wait /dev/mapper/$vgname-$lvname
+
 
 # _mkfs_dev exits the test on failure, this can make sure lv is created in
 # above vgcreate/lvcreate steps
tests/generic/108 +3 -4
index 2709472f6..f630450ec 100755
@@ -20,8 +20,8 @@ _cleanup()
 	echo running > /sys/block/`_short_dev $SCSI_DEBUG_DEV`/device/state
 	_unmount $SCRATCH_MNT >>$seqres.full 2>&1
 	$LVM_PROG vgremove -f $vgname >>$seqres.full 2>&1
-	$LVM_PROG pvremove -f $SCRATCH_DEV $SCSI_DEBUG_DEV >>$seqres.full 2>&1
-	$UDEV_SETTLE_PROG
+	 pvremove -f $SCRATCH_DEV $SCSI_DEBUG_DEV >>$seqres.full 2>&1
+	_udev_wait --removed /dev/mapper/$vgname-$lvname
 	_put_scsi_debug_dev
 	rm -f $tmp.*
 }
@@ -57,8 +57,7 @@ $LVM_PROG vgcreate -f $vgname $SCSI_DEBUG_DEV $SCRATCH_DEV >>$seqres.full 2>&1
 # (like 2.02.95 in RHEL6) don't support --yes option
 yes | $LVM_PROG lvcreate -i 2 -I 4m -L ${lvsize}m -n $lvname $vgname \
 	>>$seqres.full 2>&1
-# wait for lv creation to fully complete
-$UDEV_SETTLE_PROG >>$seqres.full 2>&1
+_udev_wait /dev/mapper/$vgname-$lvname
 
 # _mkfs_dev exits the test on failure, this makes sure test lv is created by
 # above vgcreate/lvcreate operations
tests/generic/459 +3 -3
index daccc80ce..1986c2e8f 100755
@@ -31,7 +31,7 @@ _cleanup()
 	_unmount $SCRATCH_MNT >>$seqres.full 2>&1
 	$LVM_PROG vgremove -ff $vgname >>$seqres.full 2>&1
 	$LVM_PROG pvremove -ff $SCRATCH_DEV >>$seqres.full 2>&1
-	$UDEV_SETTLE_PROG
+	_udev_wait --removed /dev/mapper/$vgname-$lvname
 }
 
 # Import common functions.
@@ -88,8 +88,7 @@ $LVM_PROG lvcreate  --thinpool $poolname  --errorwhenfull y \
 $LVM_PROG lvcreate  --virtualsize $virtsize \
 		    -T $vgname/$poolname \
 		    -n $lvname >>$seqres.full 2>&1
-
-$UDEV_SETTLE_PROG &>/dev/null
+_udev_wait /dev/mapper/$vgname-$lvname
 _mkfs_dev /dev/mapper/$vgname-$lvname >>$seqres.full 2>&1
 
 # Running the test over the original volume doesn't reproduce the problem
@@ -97,6 +96,7 @@ _mkfs_dev /dev/mapper/$vgname-$lvname >>$seqres.full 2>&1
 # reproducible, so, create a snapshot and run the test over it.
 $LVM_PROG lvcreate  -k n -s $vgname/$lvname \
 		    -n $snapname >>$seqres.full 2>&1
+_udev_wait /dev/mapper/$vgname-$snapname
 
 # Catch mount failure so we don't blindly go an freeze the root filesystem
 # instead of lvm volume.
Signature · 1 line
2.45.2
#30 lore raw

Re: [PATCH 16/40] fstests: use udevadm wait in preference to settle

Quoted 34 lines · Darrick J. Wong
Quoted 33 lines · Dave Chinner

From: Dave Chinner <[email protected]>

When running lots of tests in parallel, there are lots of filesystems and block devices changing state. This generates a lot of udev events when means the udev event queue is rarely empty. Unfortunately, an empty event queue is what udev settling waits upon. Hence calling UDEV_SETTLE_PROG can mean waiting for a lot of time for other tests to stop generating udev events.

For the majority of cases, what we care about is that udev has performed device node addition or removal, not that there are no udev events pending. Recent(-ish) systemd releases support 'udevadm wait' to wait for a specific file to be created or unlinked rather than waiting for the event that does that work to be completed.

Hence we don't have to wait for the udev event queue to empty, just for the udev event that does the device node manipulation to complete.

Introduce detection of 'udevadm wait' support and a _udev_wait() wrapper function to use it if it is available. If it isn't, the use the existing UDEV_SETTLE_PROG behaviour.

---
 common/config     | 35 +++++++++++++++++++++++++----------
 common/rc         | 25 ++++++++++++++++---------
 tests/btrfs/291   |  5 +++--
 tests/generic/081 |  6 +++---
 tests/generic/108 |  7 +++----
 tests/generic/459 |  6 +++---
 6 files changed, 53 insertions(+), 31 deletions(-)
Quoted 13 lines

common/config export XFS_MDRESTORE_PROG="$(type -P xfs_mdrestore)" · line 168

Quoted 10 lines
common/config +1 -1
index fcff0660b..41b8f29d1 100644
@@ -165,7 +165,7 @@ export XFS_MDRESTORE_PROG="$(type -P xfs_mdrestore)"
 export XFS_ADMIN_PROG="$(type -P xfs_admin)"
 export XFS_GROWFS_PROG=$(type -P xfs_growfs)
 export XFS_SPACEMAN_PROG="$(type -P xfs_spaceman)"
-export XFS_SCRUB_PROG="$(type -P xfs_scrub)"
+#export XFS_SCRUB_PROG="$(type -P xfs_scrub)"

If you have problems with online fsck, please report them to the mailing list.

Thank you for finding this, Darrick, but there is absolutely no need to treat me like a n00b who doesn't know how kernel development works.

I simply forgot I did this when debugging the hundreds of busy-after-unmount failures that were occurring whilst trying to get this patchset to work.

I turned off scrub while I debugged these issues to reduce the amount of noise and variations in the end-of-test failures, and I simply never noticed that I had left this hunk in the patch.

So, nothing wrong with scrub, just a simple case of missing one line of debug changes in a patchset that touches 400+ files....

Removed.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#31 lore raw

Re: [PATCH 16/40] fstests: use udevadm wait in preference to settle

Quoted 57 lines · Dave Chinner
Quoted 34 lines · Darrick J. Wong
Quoted 33 lines · Dave Chinner

From: Dave Chinner <[email protected]>

When running lots of tests in parallel, there are lots of filesystems and block devices changing state. This generates a lot of udev events when means the udev event queue is rarely empty. Unfortunately, an empty event queue is what udev settling waits upon. Hence calling UDEV_SETTLE_PROG can mean waiting for a lot of time for other tests to stop generating udev events.

For the majority of cases, what we care about is that udev has performed device node addition or removal, not that there are no udev events pending. Recent(-ish) systemd releases support 'udevadm wait' to wait for a specific file to be created or unlinked rather than waiting for the event that does that work to be completed.

Hence we don't have to wait for the udev event queue to empty, just for the udev event that does the device node manipulation to complete.

Introduce detection of 'udevadm wait' support and a _udev_wait() wrapper function to use it if it is available. If it isn't, the use the existing UDEV_SETTLE_PROG behaviour.

---
 common/config     | 35 +++++++++++++++++++++++++----------
 common/rc         | 25 ++++++++++++++++---------
 tests/btrfs/291   |  5 +++--
 tests/generic/081 |  6 +++---
 tests/generic/108 |  7 +++----
 tests/generic/459 |  6 +++---
 6 files changed, 53 insertions(+), 31 deletions(-)
Quoted 13 lines

common/config export XFS_MDRESTORE_PROG="$(type -P xfs_mdrestore)" · line 168

Quoted 10 lines
common/config +1 -1
index fcff0660b..41b8f29d1 100644
@@ -165,7 +165,7 @@ export XFS_MDRESTORE_PROG="$(type -P xfs_mdrestore)"
 export XFS_ADMIN_PROG="$(type -P xfs_admin)"
 export XFS_GROWFS_PROG=$(type -P xfs_growfs)
 export XFS_SPACEMAN_PROG="$(type -P xfs_spaceman)"
-export XFS_SCRUB_PROG="$(type -P xfs_scrub)"
+#export XFS_SCRUB_PROG="$(type -P xfs_scrub)"

If you have problems with online fsck, please report them to the mailing list.

Thank you for finding this, Darrick, but there is absolutely no need to treat me like a n00b who doesn't know how kernel development works.

I simply forgot I did this when debugging the hundreds of busy-after-unmount failures that were occurring whilst trying to get this patchset to work.

Quoted 8 lines

I turned off scrub while I debugged these issues to reduce the amount of noise and variations in the end-of-test failures, and I simply never noticed that I had left this hunk in the patch.

So, nothing wrong with scrub, just a simple case of missing one line of debug changes in a patchset that touches 400+ files....

Removed.

Thanks Darrick and Dave, this's a big patchset, warm welcome more review/test from anyone :) I've prepared a "for-dave-check-parallel" branch which merged Dave's "check-parallel" branch onto fstests next release(this weekend), and fixed some conflict. I also removed above comment mark and re-pushed. Feel free to merge&update code or review&test on that, I'm testing on it too :)

Thanks,
Zorro

Quoted 6 lines

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#32 lore raw

PATCH 17/40 xfs/442: rescale load so it's not exponential

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

....
xfs/442 491
....

xfs/442 takes a long time to run because it is scaling the load by the number of processes it is going to run on twice. It scales the number of operations by the number of processes it is going to run, meaning that doubling the number of processes quadruples the runtime.

Reduce it to scale linearly by fixing the number of ops it runs per process.

---
 tests/xfs/442 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/xfs/442 +1 -1
index 58e56b5d2..d539fa5e0 100755
@@ -63,7 +63,7 @@ _scratch_mkfs_sized $((1600 * 1048576)) > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 
 nr_cpus=$((LOAD_FACTOR * 4))
-nr_ops=$((25000 * nr_cpus * TIME_FACTOR))
+nr_ops=$((25000 * TIME_FACTOR))
 _run_fsstress -w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus
 
 echo "Check quota before remount"
Signature · 1 line
2.45.2
#33 lore raw

PATCH 18/40 xfs/176: fix broken setup code

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

The test does not pass the mkfs output through the mkfs filter, so the inode size is not set up correctly. Hence it calculates the CHUNK_SIZE as 0, and it ends up getting stuck in an endless loop throwing ENOSPC errors because the offset never changes.

While there, use 'echo -n' rather than 'touch' to create zero length files much faster.

---
 tests/xfs/176 | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tests/xfs/176 +2 -2
index e3fa87c43..8e5951ec1 100755
@@ -21,7 +21,7 @@ _require_scratch_xfs_shrink
 _require_xfs_io_command "falloc"
 _require_xfs_io_command "fpunch"
 
-_scratch_mkfs "-d size=50m -m crc=1 -i sparse" |
+_scratch_mkfs "-d size=50m -m crc=1 -i sparse" | \
 	_filter_mkfs > /dev/null 2> $tmp.mkfs
 . $tmp.mkfs	# for isize
 cat $tmp.mkfs >> $seqres.full
@@ -69,7 +69,7 @@ _alloc_inodes()
 
 	i=0
 	while [ true ]; do
-		touch $dir/$i 2>> $seqres.full || break
+		echo -n > $dir/$i >> $seqres.full 2>&1 || break
 		i=$((i + 1))
 	done
 }
Signature · 1 line
2.45.2
#34 lore raw

PATCH 19/40 xfs/177: remove unused slab object count location checks

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Stale code; we count XFS inodes through the sysfs stats code now so remove it.

---
 tests/xfs/177 | 7 -------
 1 file changed, 7 deletions(-)
tests/xfs/177 +0 -7
index 22719ba1c..82b3ca264 100755
@@ -58,13 +58,6 @@ dump_debug_info() {
 	echo "round $1 baseline: $baseline_count high: $high_count fresh: $fresh_count post: $post_count end: $end_count" >> $seqres.full
 }
 
-# Either of these need to be available to monitor slab usage
-xfs_ino_objcount_file=/sys/kernel/slab/xfs_inode/objects
-slabinfo_file=/proc/slabinfo
-if [ ! -r "$xfs_ino_objcount_file" ] && [ ! -r "$slabinfo_file" ]; then
-	_notrun "Cannot find xfs_inode slab count?"
-fi
-
 # Background reclamation of disused xfs inodes is scheduled for ($xfssyncd / 6)
 # centiseconds after the first inode is tagged for reclamation.  It's not great
 # to encode this implementation detail in a test like this, but there isn't
Signature · 1 line
2.45.2
#35 lore raw

PATCH 20/40 fstests: remove uses of killall where possible

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

there are many unnecessary uses of killall and stale checks for it's existence. Parallel check execution means killall is considered harmful, so get rid of these unneccesary uses.

---
 common/config                |  1 -
 common/fuzzy                 |  6 +++---
 doc/requirement-checking.txt |  6 +++---
 tests/btrfs/192              |  1 -
 tests/btrfs/212              |  2 --
 tests/generic/270            |  1 -
 tests/generic/310            | 22 +++++++++++++++++-----
 tests/generic/751            |  1 -
 tests/overlay/058            | 12 ++++++++----
 tests/xfs/011                |  1 -
 tests/xfs/013                |  1 -
 tests/xfs/051                |  1 -
 tests/xfs/057                |  1 -
 tests/xfs/070                |  9 +++++----
 tests/xfs/079                |  1 -
 tests/xfs/141                |  1 -
 tests/xfs/167                |  3 ---
 tests/xfs/297                |  2 --
 tests/xfs/442                |  1 -
 19 files changed, 36 insertions(+), 37 deletions(-)
common/config +0 -1
index 41b8f29d1..6f531c43d 100644
@@ -182,7 +182,6 @@ export CHACL_PROG="$(type -P chacl)"
 export ATTR_PROG="$(type -P attr)"
 export QUOTA_PROG="$(type -P quota)"
 export XFS_QUOTA_PROG="$(type -P xfs_quota)"
-export KILLALL_PROG="$(type -P killall)"
 export INDENT_PROG="$(type -P indent)"
 export XFS_COPY_PROG="$(type -P xfs_copy)"
 export FSTRIM_PROG="$(type -P fstrim)"
common/fuzzy +3 -3
index 3a7f04aae..534e91ded 100644
@@ -1188,14 +1188,14 @@ _scratch_xfs_stress_scrub_cleanup() {
 	# a race condition that can hang fstests.
 	#
 	# If the xfs_io -c freeze process is asleep waiting for a write lock on
-	# s_umount or sb_write when the killall signal is delivered, it will
+	# s_umount or sb_write when the kill signal is delivered, it will
 	# not check for pending signals until after it has frozen the fs.  If
 	# even one thread of the stress test processes (xfs_io, fsstress, etc.)
-	# is waiting for read locks on sb_write when the killall signals are
+	# is waiting for read locks on sb_write when the kill signals are
 	# delivered, they will block in the kernel until someone thaws the fs,
 	# and the `wait' below will wait forever.
 	#
-	# Hence we issue the killall, wait for the freezer loop to exit, thaw
+	# Hence we issue the kill, wait for the freezer loop to exit, thaw
 	# the filesystem, and wait for the rest of the children.
 	if [ -n "$__SCRUB_STRESS_FREEZE_PID" ]; then
 		echo "Waiting for fs freezer $__SCRUB_STRESS_FREEZE_PID to exit at $(date)" >> $seqres.full
doc/requirement-checking.txt +3 -3
index 802bf2a3f..78990fe9e 100644
@@ -40,11 +40,11 @@ _require_command "$NAME_PROG" name
      should then be used to refer to the command when executing it.  For
      example:
 
-	_require_command "KILLALL_PROG" killall
+	_require_command "$XFS_DB_PROG" "xfs_db"
 
-     to locate the killall command and then:
+     to locate the xfs_db command and then:
 
-	$KILLALL_PROG -q $FSSTRESS_PROG
+	$XFS_DB_PROG -c "sb 0" -c "print" $SCRATCH_DEV
 
      to make use of it.
 
tests/btrfs/192 +0 -1
index cc8e1e003..0a8ab2c1b 100755
@@ -27,7 +27,6 @@ _cleanup()
 . ./common/dmlogwrites
 
 
-_require_command "$KILLALL_PROG" killall
 _require_command "$BLKDISCARD_PROG" blkdiscard
 _require_btrfs_fs_feature "no_holes"
 _require_btrfs_mkfs_feature "no-holes"
tests/btrfs/212 +0 -2
index 745b9598a..80c702720 100755
@@ -25,8 +25,6 @@ _cleanup()
 . ./common/filter
 
 _require_scratch
-_require_command "$KILLALL_PROG" killall
-
 _scratch_mkfs >> $seqres.full
 _scratch_mount
 
tests/generic/270 +0 -1
index aff379ac5..342ac8b5d 100755
@@ -56,7 +56,6 @@ _workout()
 _require_quota
 _require_user
 _require_scratch
-_require_command "$KILLALL_PROG" killall
 _require_command "$SETCAP_PROG" setcap
 _require_attrs security
 
tests/generic/310 +17 -5
index 15e87aece..1ae9e0233 100755
@@ -29,14 +29,19 @@ _begin_fstest auto
 # Override the default cleanup function.
 _cleanup()
 {
+	if [ -n "$readdir_pid" ]; then
+		pkill --parent "$readdir_pid" t_readdir > /dev/null 2>&1
+		kill -9 $readdir_pid > /dev/null 2>&1
+		wait
+	fi
 	rm -rf $TEST_DIR/tmp
+	rm -f $tmp.*
 }
 
 # Import common functions.
 . ./common/filter
 
 _require_test
-_require_command "$KILLALL_PROG" killall
 
 dmesg -c > /dev/null
 
@@ -80,9 +85,12 @@ done
 
 _test_read()
 {
-	$here/src/t_readdir_1 $SEQ_DIR &
+	$here/src/t_readdir_1 $SEQ_DIR > /dev/null 2>&1 &
+	readdir_pid=$!
 	sleep $RUN_TIME
-	$KILLALL_PROG t_readdir_1
+	kill $readdir_pid
+	unset readdir_pid
+	wait
 	check_kernel_bug
 	if [ $? -ne 0 ]; then
 		_fatal "kernel bug detected, check dmesg for more infomation."
@@ -91,9 +99,13 @@ _test_read()
 
 _test_lseek()
 {
-	$here/src/t_readdir_2 $SEQ_DIR &
+	$here/src/t_readdir_2 $SEQ_DIR > /dev/null 2>&1 &
+	readdir_pid=$!
 	sleep $RUN_TIME
-	$KILLALL_PROG t_readdir_2
+	kill $readdir_pid
+	unset readdir_pid
+	wait
+
 	check_kernel_bug
 	if [ $? -ne 0 ]; then
 		_fatal "kernel bug detected, check dmesg for more infomation."
tests/generic/751 +0 -1
index eac2d230a..892e59edf 100755
@@ -80,7 +80,6 @@ fio_err=$tmp.fio.err
 _require_test
 _require_scratch
 _require_split_huge_pages_knob
-_require_command "$KILLALL_PROG" "killall"
 _fixed_by_git_commit kernel 2a0774c2886d \
 	"XArray: set the marks correctly when splitting an entry"
 
tests/overlay/058 +8 -4
index b70e06638..b19a47e36 100755
@@ -21,8 +21,10 @@ _begin_fstest auto quick exportfs
 # Override the default cleanup function.
 _cleanup()
 {
-	$KILLALL_PROG -9 open_by_handle >/dev/null 2>&1
-	wait
+	if [ -n "$pids" ]; then
+		kill -9 $pids >/dev/null 2>&1
+		wait
+	fi
 	cd /
 	rm -f $tmp.*
 }
@@ -36,7 +38,6 @@ _require_test_program "open_by_handle"
 # We need to require all features together, because nfs_export cannot
 # be enabled when index is disabled
 _require_scratch_overlay_features index nfs_export
-_require_command "$KILLALL_PROG" killall
 
 # All overlay dirs are on scratch partition
 lower=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER
@@ -89,9 +90,11 @@ test_file_handles $SCRATCH_MNT -rnp -i $tmp.lower_file_handles
 # overlay dentries in cache
 _scratch_cycle_mount "index=on,nfs_export=on"
 test_file_handles $SCRATCH_MNT -rnps -i $tmp.upper_file_handles &
+pids=$!
 # Give the above 1 second to get to sleep loop
 sleep 1
 test_file_handles $SCRATCH_MNT -rnps -i $tmp.lower_file_handles &
+pids="$pids $!"
 # Give the above 1 second to get to sleep loop
 sleep 1
 
@@ -105,8 +108,9 @@ test_file_handles $SCRATCH_MNT -rnp -i $tmp.upper_file_handles
 test_file_handles $SCRATCH_MNT -rnp -i $tmp.lower_file_handles
 
 # SIGPIPE avoids Terminated/Killed message from bash
-$KILLALL_PROG -q -13 open_by_handle
+kill -13 $pids > /dev/null 2>&1
 wait
+unset pids
 
 status=0
 exit
tests/xfs/011 +0 -1
index ed69879c5..1192e75dd 100755
@@ -14,7 +14,6 @@ _begin_fstest auto freeze log metadata quick
 _require_scratch
 _require_freeze
 _require_xfs_sysfs $(_short_dev $TEST_DEV)/log
-_require_command "$KILLALL_PROG" killall
 
 . ./common/filter
 
tests/xfs/013 +0 -1
index c68c6ad85..fd3d8c64c 100755
@@ -74,7 +74,6 @@ _cleaner()
 _require_scratch
 _require_xfs_mkfs_finobt
 _require_xfs_finobt
-_require_command "$KILLALL_PROG" killall
 
 _scratch_mkfs_xfs "-m crc=1,finobt=1 -d agcount=2" | \
 	_filter_mkfs 2>> $seqres.full
tests/xfs/051 +0 -1
index bb9c36da8..fe3d75cab 100755
@@ -19,7 +19,6 @@ _begin_fstest shutdown auto log metadata
 _require_scratch
 _require_dm_target flakey
 _require_xfs_sysfs debug/log_recovery_delay
-_require_command "$KILLALL_PROG" killall
 
 echo "Silence is golden."
 
tests/xfs/057 +0 -1
index 62eb8b93c..da583e621 100755
@@ -42,7 +42,6 @@ _cleanup()
 _require_xfs_io_error_injection log_item_pin
 _require_xfs_io_error_injection log_bad_crc
 _require_scratch
-_require_command "$KILLALL_PROG" killall
 
 echo "Silence is golden."
 
tests/xfs/070 +5 -4
index d2bd7e2c4..143f56888 100755
@@ -24,10 +24,10 @@ _begin_fstest auto quick repair
 # Override the default cleanup function.
 _cleanup()
 {
+	[ -n "$repair_pid" ] && kill -9 $repair_pid > /dev/null 2>&1
+	wait > /dev/null 2>&1
 	cd /
 	rm -f $tmp.*
-	$KILLALL_PROG -9 $XFS_REPAIR_PROG > /dev/null 2>&1
-	wait > /dev/null 2>&1
 }
 
 # Start and monitor an xfs_repair of the scratch device. This test can induce a
@@ -45,7 +45,8 @@ _xfs_repair_noscan()
 	repair_pid=$!
 
 	# monitor progress for as long as it is running
-	while [ `pgrep xfs_repair` ]; do
+	running=`pgrep xfs_repair`
+	while [ -n "$running" ]; do
 		grep "couldn't verify primary superblock" $tmp.repair \
 			> /dev/null 2>&1
 		if [ $? == 0 ]; then
@@ -58,6 +59,7 @@ _xfs_repair_noscan()
 		fi
 
 		sleep 1
+		running=`pgrep xfs_repair`
 	done
 
 	wait
@@ -72,7 +74,6 @@ _xfs_repair_noscan()
 
 # Modify as appropriate.
 _require_scratch_nocheck
-_require_command "$KILLALL_PROG" killall
 
 _scratch_mkfs | _filter_mkfs > /dev/null 2> $tmp.mkfs
 
tests/xfs/079 +0 -1
index 794d2db49..4a1999bb8 100755
@@ -23,7 +23,6 @@ _begin_fstest shutdown auto log quick
 # Modify as appropriate.
 _require_scratch
 _require_v2log
-_require_command "$KILLALL_PROG" killall
 
 echo "Silence is golden."
 
tests/xfs/141 +0 -1
index b630ba10d..6aa844825 100755
@@ -20,7 +20,6 @@ _begin_fstest auto log metadata
 # Modify as appropriate.
 _require_xfs_io_error_injection "log_bad_crc"
 _require_scratch
-_require_command "$KILLALL_PROG" killall
 
 echo "Silence is golden."
 
tests/xfs/167 +0 -3
index 5ef2aa2ea..2a6e6b8d0 100755
@@ -21,10 +21,7 @@ workout()
 # Import common functions.
 . ./common/filter
 
-
-_require_command "$KILLALL_PROG" killall
 _require_xfs_io_command "falloc"
-
 _require_scratch
 _scratch_mkfs_xfs >/dev/null 2>&1
 _scratch_mount
tests/xfs/297 +0 -2
index 66c5d0cc7..f9cd2ff12 100755
@@ -24,10 +24,8 @@ _cleanup()
 # Import common functions.
 . ./common/filter
 
-
 _require_scratch
 _require_freeze
-_require_command "$KILLALL_PROG" killall
 
 logblks=$(_scratch_find_xfs_min_logblocks -d agcount=16,su=256k,sw=12 -l su=256k)
 _scratch_mkfs_xfs -d agcount=16,su=256k,sw=12 -l su=256k,size=${logblks}b >/dev/null 2>&1
tests/xfs/442 +0 -1
index d539fa5e0..08f0aac40 100755
@@ -21,7 +21,6 @@ _begin_fstest auto stress clone quota
 
 _require_scratch_reflink
 _require_quota
-_require_command "$KILLALL_PROG" "killall"
 
 report_quota_blocks() {
 	$XFS_QUOTA_PROG -x -c "report $1" $SCRATCH_MNT | \
Signature · 1 line
2.45.2
#36 lore raw

PATCH 21/40 generic/127: reduce runtime

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

...
generic/127 684
...

This takes a long time to run because it runs 6 individual invocations of fsx sequentially. Make them run concurrently as they can operate on separate files.

...
generic/127 168
...

---
 tests/generic/127     | 67 ++++++++++++++++++++++++++-----------------
 tests/generic/127.out |  6 ----
 2 files changed, 41 insertions(+), 32 deletions(-)
tests/generic/127 +41 -26
index c8984f273..985c99cfb 100755
@@ -16,7 +16,10 @@ _begin_fstest rw auto
 # Override the default cleanup function.
 _cleanup()
 {
-    rm -f $tmp.output
+	[ -n "$fsx_pids" ] && kill $fsx_pids 2> /dev/null
+	wait
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -27,7 +30,6 @@ FSX_ARGS="-q -l $FSX_FILE_SIZE -o 65536 -S 191110531 -N 100000"
 
 _fsx_lite_nommap()
 {
-    echo "=== FSX Light Mode, No Memory Mapping ==="
     dd if=/dev/zero of=$TEST_DIR/fsx_lite_nommap bs=${FSX_FILE_SIZE} count=1 > /dev/null 2>&1
     if ! ltp/fsx $FSX_ARGS -L -R -W $FSX_AVOID $TEST_DIR/fsx_lite_nommap > $tmp.output 2>&1
     then
@@ -41,7 +43,6 @@ _fsx_lite_nommap()
 
 _fsx_lite_mmap()
 {
-    echo "=== FSX Light Mode, Memory Mapping ==="
     dd if=/dev/zero of=$TEST_DIR/fsx_lite_mmap bs=${FSX_FILE_SIZE} count=1 > /dev/null 2>&1
     if ! ltp/fsx $FSX_ARGS -L $FSX_AVOID $TEST_DIR/fsx_lite_mmap > $tmp.output 2>&1
     then
@@ -55,42 +56,56 @@ _fsx_lite_mmap()
 
 _fsx_std_nommap()
 {
-    echo "=== FSX Standard Mode, No Memory Mapping ==="
-    if ! ltp/fsx $FSX_ARGS -R -W $FSX_AVOID $TEST_DIR/fsx_std_nommap > $tmp.output 2>&1
-    then
-        echo "ltp/fsx $FSX_ARGS -R -W fsx_std_nommap"
-        cat $tmp.output
-        return 1
-    fi
-    cat $tmp.output
-    return 0
+	local fname="$TEST_DIR/$1"
+
+	if ! ltp/fsx $FSX_ARGS -R -W $FSX_AVOID $fname > $tmp.output 2>&1
+	then
+		echo "ltp/fsx $FSX_ARGS -R -W fsx_std_nommap"
+		cat $tmp.output
+		return 1
+	fi
+	cat $tmp.output
+	return 0
 }
 
 _fsx_std_mmap()
 {
-    echo "=== FSX Standard Mode, Memory Mapping ==="
-    if ! ltp/fsx $FSX_ARGS $FSX_AVOID $TEST_DIR/fsx_std_mmap > $tmp.output 2>&1
-    then
-    	echo "ltp/fsx $FSX_ARGS fsx_std_mmap"
+	local fname="$TEST_DIR/$1"
+
+	if ! ltp/fsx $FSX_ARGS $FSX_AVOID $fname > $tmp.output 2>&1
+	then
+		echo "ltp/fsx $FSX_ARGS fsx_std_mmap"
+		cat $tmp.output
+		return 1
+	fi
 	cat $tmp.output
-	return 1
-    fi
-    cat $tmp.output
-    return 0
+	return 0
 }
 
 _require_test
 
-_fsx_lite_nommap || exit
-_fsx_lite_mmap || exit
+_fsx_lite_nommap &
+fsx_pids=$!
 
-_fsx_std_nommap || exit
-_fsx_std_mmap || exit
+_fsx_lite_mmap &
+fsx_pids="$! $fsx_pids"
+
+_fsx_std_nommap fsx_std_nommap &
+fsx_pids="$! $fsx_pids"
+
+_fsx_std_mmap fsx_std_mmap &
+fsx_pids="$! $fsx_pids"
 
 #flush cache after write
 FSX_ARGS="-f $FSX_ARGS"
-_fsx_std_nommap || exit
-_fsx_std_mmap || exit
+_fsx_std_nommap fsx_std_nommap_flush &
+fsx_pids="$! $fsx_pids"
+
+_fsx_std_mmap fsx_std_mmap_flush &
+fsx_pids="$! $fsx_pids"
+
+wait $fsx_pids
+unset fsx_pids
 
 status=0
 _cleanup
tests/generic/127.out +0 -6
index 9459dc897..99fb0fddc 100644
@@ -1,13 +1,7 @@
 QA output created by 127
-=== FSX Light Mode, No Memory Mapping ===
 All 100000 operations completed A-OK!
-=== FSX Light Mode, Memory Mapping ===
 All 100000 operations completed A-OK!
-=== FSX Standard Mode, No Memory Mapping ===
 All 100000 operations completed A-OK!
-=== FSX Standard Mode, Memory Mapping ===
 All 100000 operations completed A-OK!
-=== FSX Standard Mode, No Memory Mapping ===
 All 100000 operations completed A-OK!
-=== FSX Standard Mode, Memory Mapping ===
 All 100000 operations completed A-OK!
Signature · 1 line
2.45.2
#37 lore raw

PATCH 22/40 quota: system project quota files need to be shared

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Tests that treat them as exclusively owned end up tripping over other tests that do the same. Fix this by using append and filter techniques to update the files, then using different project quota ids for each test.

---
 common/quota      | 35 +++++++----------------------------
 tests/generic/603 |  8 +++++---
 tests/generic/691 |  6 ++++--
 3 files changed, 16 insertions(+), 33 deletions(-)
common/quota +7 -28
index e87489192..8135b7c04 100644
@@ -274,7 +274,7 @@ _choose_prid()
 
 _qmount()
 {
-    _scratch_unmount >/dev/null 2>&1
+    _scratch_unmount
     _try_scratch_mount || _fail "qmount failed"
     # xfs doesn't need these setups and quotacheck even fails on xfs
     # redirect the output to $seqres.full for debug purpose and ignore results
@@ -432,39 +432,18 @@ _create_project_quota()
 	mkdir $prjdir
 	chmod ugo+rwx $prjdir
 
-	if [ -f /etc/projects -a ! -f $tmp.projects.bk ];then
-		cat /etc/projects > $tmp.projects.bk
-		echo >/etc/projects
-	fi
-	if [ -f /etc/projid -a ! -f $tmp.projid.bk ];then
-		cat /etc/projid > $tmp.projid.bk
-		echo >/etc/projid
-	fi
-
-	cat >>/etc/projects <<EOF
-$id:$prjdir
-EOF
-	cat >>/etc/projid <<EOF
-$name:$id
-EOF
+	echo "$id:$prjdir" >> /etc/project
+	echo "$name:$id" >> /etc/projid
 	$XFS_IO_PROG -r -c "chproj $id" -c "chattr +P" $prjdir
 }
 
 # If you've called _create_project_quota, then use this function in _cleanup
 _restore_project_quota()
 {
-	if [ -f $tmp.projects.bk ];then
-		cat $tmp.projects.bk > /etc/projects && \
-			rm -f $tmp.projects.bk
-	else
-		rm -f /etc/projects
-	fi
-	if [ -f $tmp.projid.bk ];then
-		cat $tmp.projid.bk > /etc/projid && \
-			rm -f $tmp.projid.bk
-	else
-		rm -f /etc/projid
-	fi
+	local id=$1
+
+	sed -i "/^$id:/d" /etc/project
+	sed -i "/:$id$/d" /etc/projid
 }
 
 # make sure this script returns success
tests/generic/603 +5 -3
index 32dcaeb94..a84695831 100755
@@ -10,10 +10,12 @@
 . ./common/preamble
 _begin_fstest auto quick quota
 
+projid=$seq
+
 # Override the default cleanup function.
 _cleanup()
 {
-	_restore_project_quota
+	_restore_project_quota $projid
 	cd /
 	rm -f $tmp.*
 }
@@ -122,8 +124,8 @@ _require_prjquota $SCRATCH_DEV
 BLOCK_SIZE=$(_get_file_block_size $SCRATCH_MNT)
 rm -rf $SCRATCH_MNT/t
 mkdir $SCRATCH_MNT/t
-$XFS_IO_PROG -r -c "chproj 100" -c "chattr +P" $SCRATCH_MNT/t
-_create_project_quota $SCRATCH_MNT/t 100 $qa_user
+$XFS_IO_PROG -r -c "chproj $projid" -c "chattr +P" $SCRATCH_MNT/t
+_create_project_quota $SCRATCH_MNT/t $projid $qa_user
 
 echo "### Set up different grace timers to each type of quota"
 UBGRACE=12
tests/generic/691 +4 -2
index 25ce92bc4..f33d6edf1 100755
@@ -16,10 +16,12 @@
 . ./common/preamble
 _begin_fstest auto quota
 
+projid=$seq
+
 # Override the default cleanup function.
 _cleanup()
 {
-	_restore_project_quota
+	_restore_project_quota $projid
 	cd /
 	rm -r -f $tmp.*
 }
@@ -65,7 +67,7 @@ exercise()
 	fi
 	_qmount
 	if [ "$type" = "P" ];then
-		_create_project_quota $SCRATCH_MNT/t 100 $qa_user
+		_create_project_quota $SCRATCH_MNT/t $projid $qa_user
 		file=$SCRATCH_MNT/t/testfile
 	fi
 
Signature · 1 line
2.45.2
#38 lore raw

PATCH 23/40 dmesg: reduce noise from other tests

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

dmesg records everything from every test concurrently running, so noise from other tests can cause multiple other tests to fail because they detect something from another test. Update the filter behaviour to minimise this crosstalk problem.

`

---
 common/rc         | 13 +++++++------
 tests/generic/310 |  5 +----
 tests/xfs/079     |  2 +-
 3 files changed, 9 insertions(+), 11 deletions(-)
common/rc +7 -6
index fdd18a386..3b4b9a59f 100644
@@ -4591,6 +4591,7 @@ _check_dmesg_filter()
 
 	grep -E -v -e "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low" \
 		-e "BUG: MAX_STACK_TRACE_ENTRIES too low" \
+		-e "WARNING: Reset corrupted AGFL" \
 		$extra_filter
 }
 
@@ -4613,18 +4614,18 @@ _check_dmesg()
 	fi
 	rm -f ${RESULT_DIR}/check_dmesg
 
-	# default filter is a simple cat command, caller could provide a
-	# customized filter and pass the name through the first argument, to
-	# filter out intentional WARNINGs or Oopses
-	local filter=${1:-_check_dmesg_filter}
+	# we always filter dmesg for global things that should not cause
+	# test failures. Custom filters are then applied afterwards.
+	_dmesg_since_test_start | _check_dmesg_filter > $seqres.dmesg
+	if [ -n "$1" ]; then
+		cat $seqres.dmesg | $1 > $seqres.dmesg
+	fi
 
-	_dmesg_since_test_start | $filter >$seqres.dmesg
 	grep -E -q -e "kernel BUG at" \
 	     -e "WARNING:" \
 	     -e "\bBUG:" \
 	     -e "Oops:" \
 	     -e "possible recursive locking detected" \
-	     -e "Internal error" \
 	     -e "(INFO|ERR): suspicious RCU usage" \
 	     -e "INFO: possible circular locking dependency detected" \
 	     -e "general protection fault:" \
tests/generic/310 +1 -4
index 1ae9e0233..1ab81dd2e 100755
@@ -46,7 +46,6 @@ _require_test
 dmesg -c > /dev/null
 
 nr_bug=`dmesg | grep -c "kernel BUG"`
-nr_error=`dmesg | grep -wc "error"`
 nr_null=`dmesg | grep -c "kernel NULL pointer dereference"`
 nr_warning=`dmesg | grep -c "^WARNING"`
 nr_lockdep=`dmesg | grep -c "possible recursive locking detected"`
@@ -55,20 +54,18 @@ nr_lockdep=`dmesg | grep -c "possible recursive locking detected"`
 check_kernel_bug()
 {
 	new_bug=`dmesg | grep -c "kernel BUG"`
-	new_error=`dmesg | grep -wc "error"`
 	new_null=`dmesg | grep -c "kernel NULL pointer dereference"`
 	new_warning=`dmesg | grep -c "^WARNING"`
 	new_lockdep=`dmesg | grep -c "possible recursive locking detected"`
 
 	# no kernel bug is detected
-	if [ $new_bug -eq $nr_bug -a $new_error -eq $nr_error -a \
+	if [ $new_bug -eq $nr_bug -a \
 	     $new_null -eq $nr_null -a $new_warning -eq $nr_warning -a \
 	     $new_lockdep -eq $nr_lockdep ]; then
 		return 0
 	fi
 
 	nr_bug=$new_bug
-	nr_error=$new_error
 	nr_null=$new_null
 	nr_warning=$new_warning
 	nr_lockdep=$new_lockdep
tests/xfs/079 +1 -1
index 4a1999bb8..3287a20fb 100755
@@ -45,7 +45,7 @@ _try_scratch_mount "-o logbsize=64k"
 
 # The mount may not fail on v4 filesystems. Check for CRC mismatch warning
 # messages to detect failure in this case.
-dmesg -c | grep XFS | grep CRC
+_dmesg_since_test_start | grep "XFS $(_short_dev $SCRATCH_DEV)" | grep CRC
 
 # success, all done
 status=0
Signature · 1 line
2.45.2
#39 lore raw

PATCH 24/40 fstests: stop using /tmp directly

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Tests should be using $tmp, not /tmp. this causes problems when multiple tests all use /tmp/foo as a temporary test state file and then step on each other.

Note that there are some tests that use /tmp to store "test stop" files for background processes. Those that have proven to be unreliable at stopping tests when interrupted by ctrl-c are also updated to track and kill background processes in the cleanup function.

---
 tests/generic/162 |  4 ++--
 tests/generic/163 |  4 ++--
 tests/generic/164 |  4 ++--
 tests/generic/165 |  4 ++--
 tests/generic/166 |  6 +++---
 tests/generic/167 |  4 ++--
 tests/generic/168 |  7 +++++--
 tests/generic/170 |  7 +++++--
 tests/generic/333 |  6 +++---
 tests/generic/334 |  6 +++---
 tests/generic/373 |  2 +-
 tests/generic/374 |  2 +-
 tests/generic/670 |  2 +-
 tests/generic/671 |  2 +-
 tests/generic/672 |  2 +-
 tests/generic/722 | 15 ++++-----------
 tests/xfs/601     |  2 +-
 17 files changed, 39 insertions(+), 40 deletions(-)
tests/generic/162 +2 -2
index e23014f03..25d2da7e0 100755
@@ -15,7 +15,7 @@ _register_cleanup "_cleanup" BUS
 _cleanup()
 {
     cd /
-    rm -rf $tmp.* $finished_file
+    rm -rf $tmp.*
     wait
 }
 
@@ -30,7 +30,7 @@ _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 
 testdir=$SCRATCH_MNT/test-$seq
-finished_file=/tmp/finished
+finished_file=$tmp.finished
 rm -rf $finished_file
 mkdir $testdir
 
tests/generic/163 +2 -2
index 51a00e4b7..f80d699ac 100755
@@ -15,7 +15,7 @@ _register_cleanup "_cleanup" BUS
 _cleanup()
 {
     cd /
-    rm -rf $tmp.* $finished_file
+    rm -rf $tmp.*
     wait
 }
 
@@ -30,7 +30,7 @@ _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 
 testdir=$SCRATCH_MNT/test-$seq
-finished_file=/tmp/finished
+finished_file=$tmp.finished
 rm -rf $finished_file
 mkdir $testdir
 
tests/generic/164 +2 -2
index e5c8acd02..f38e39868 100755
@@ -16,7 +16,7 @@ _register_cleanup "_cleanup" BUS
 _cleanup()
 {
     cd /
-    rm -rf $tmp.* $finished_file
+    rm -rf $tmp.*
     wait
 }
 
@@ -32,7 +32,7 @@ _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 
 testdir=$SCRATCH_MNT/test-$seq
-finished_file=/tmp/finished
+finished_file=$tmp.finished
 rm -rf $finished_file
 mkdir $testdir
 
tests/generic/165 +2 -2
index 0e565eb1d..128afbeaa 100755
@@ -16,7 +16,7 @@ _register_cleanup "_cleanup" BUS
 _cleanup()
 {
     cd /
-    rm -rf $tmp.* $finished_file
+    rm -rf $tmp.*
     wait
 }
 
@@ -33,7 +33,7 @@ _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 
 testdir=$SCRATCH_MNT/test-$seq
-finished_file=/tmp/finished
+finished_file=$tmp.finished
 rm -rf $finished_file
 mkdir $testdir
 
tests/generic/166 +3 -3
index 16157571f..4e0b19170 100755
@@ -16,7 +16,7 @@ _register_cleanup "_cleanup" BUS
 _cleanup()
 {
     cd /
-    rm -rf $tmp.* $finished_file
+    rm -rf $tmp.* 
     wait
 }
 
@@ -33,8 +33,8 @@ _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 
 testdir=$SCRATCH_MNT/test-$seq
-finished_file=/tmp/finished
-do_snapshot=/tmp/snapshot
+finished_file=$tmp.finished
+do_snapshot=$tmp.snapshot
 rm -rf $finished_file
 mkdir $testdir
 
tests/generic/167 +2 -2
index 0c3e20fe2..ceb53963f 100755
@@ -16,7 +16,7 @@ _register_cleanup "_cleanup" BUS
 _cleanup()
 {
     cd /
-    rm -rf $tmp.* $finished_file
+    rm -rf $tmp.*
     wait
 }
 
@@ -32,7 +32,7 @@ _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 
 testdir=$SCRATCH_MNT/test-$seq
-finished_file=/tmp/finished
+finished_file=$tmp.finished
 rm -rf $finished_file
 mkdir $testdir
 
tests/generic/168 +5 -2
index bd9859f73..d4b63cec8 100755
@@ -16,7 +16,8 @@ _register_cleanup "_cleanup" BUS
 _cleanup()
 {
     cd /
-    rm -rf $tmp.* $finished_file
+    rm -rf $tmp.*
+    [ -n "$write_pid" ] && kill -9 $write_pid > /dev/null 2>&1
     wait
 }
 
@@ -31,7 +32,7 @@ _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 
 testdir=$SCRATCH_MNT/test-$seq
-finished_file=/tmp/finished
+finished_file=$tmp.finished
 rm -rf $finished_file
 mkdir $testdir
 
@@ -57,6 +58,7 @@ overwrite() {
 
 echo "Reflink and write the target"
 overwrite &
+write_pid=$!
 seq 1 10 | while read j; do
 	seq 0 $nr_loops | while read i; do
 		_reflink_range  $testdir/file1 $((i * blksz)) \
@@ -66,6 +68,7 @@ seq 1 10 | while read j; do
 done
 touch $finished_file
 wait
+unset write_pid
 
 # success, all done
 status=0
tests/generic/170 +5 -2
index 35b278b9f..be9cf7a18 100755
@@ -16,7 +16,8 @@ _register_cleanup "_cleanup" BUS
 _cleanup()
 {
     cd /
-    rm -rf $tmp.* $finished_file
+    rm -rf $tmp.*
+    [ -n "$write_pid" ] && kill -9 $write_pid > /dev/null 2>&1
     wait
 }
 
@@ -32,7 +33,7 @@ _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 
 testdir=$SCRATCH_MNT/test-$seq
-finished_file=/tmp/finished
+finished_file=$tmp.finished
 rm -rf $finished_file
 mkdir $testdir
 
@@ -58,6 +59,7 @@ overwrite() {
 
 echo "Reflink and dio write the target"
 overwrite &
+write_pid=$!
 seq 1 10 | while read j; do
 	seq 0 $nr_loops | while read i; do
 		_reflink_range  $testdir/file1 $((i * blksz)) \
@@ -67,6 +69,7 @@ seq 1 10 | while read j; do
 done
 touch $finished_file
 wait
+unset write_pid
 
 # success, all done
 status=0
tests/generic/333 +3 -3
index 03a7bcd0e..220614147 100755
@@ -16,7 +16,7 @@ _register_cleanup "_cleanup" BUS
 _cleanup()
 {
     cd /
-    rm -rf $tmp.* $finished_file $abort_file
+    rm -rf $tmp.*
     wait
 }
 
@@ -33,9 +33,9 @@ _scratch_mkfs_sized $((400 * 1048576)) > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 
 testdir=$SCRATCH_MNT/test-$seq
-finished_file=/tmp/finished
+finished_file=$tmp.finished
 rm -rf $finished_file
-abort_file=/tmp/abort
+abort_file=$tmp.abort
 rm -rf $abort_file
 mkdir $testdir
 
tests/generic/334 +3 -3
index 3f9718476..0faa5d696 100755
@@ -16,7 +16,7 @@ _register_cleanup "_cleanup" BUS
 _cleanup()
 {
     cd /
-    rm -rf $tmp.* $finished_file $abort_file
+    rm -rf $tmp.*
     wait
 }
 
@@ -32,9 +32,9 @@ _scratch_mkfs_sized $((400 * 1048576)) > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 
 testdir=$SCRATCH_MNT/test-$seq
-finished_file=/tmp/finished
+finished_file=$tmp.finished
 rm -rf $finished_file
-abort_file=/tmp/abort
+abort_file=$tmp.abort
 rm -rf $abort_file
 mkdir $testdir
 
tests/generic/373 +1 -1
index e6334398f..04ec64251 100755
@@ -32,7 +32,7 @@ _scratch_mount >> $seqres.full 2>&1
 
 testdir=$SCRATCH_MNT/test-$seq
 mkdir $testdir
-otherdir=/tmp/m.$seq
+otherdir=$tmp.m.$seq
 othertestdir=$otherdir/test-$seq
 rm -rf $otherdir
 mkdir $otherdir
tests/generic/374 +1 -1
index e56521aa0..9a85091e2 100755
@@ -31,7 +31,7 @@ _scratch_mount >> $seqres.full 2>&1
 
 testdir=$SCRATCH_MNT/test-$seq
 mkdir $testdir
-otherdir=/tmp/m.$seq
+otherdir=$tmp.m.$seq
 othertestdir=$otherdir/test-$seq
 rm -rf $otherdir
 mkdir $otherdir
tests/generic/670 +1 -1
index 67a8f6fe2..f32199e70 100755
@@ -23,7 +23,7 @@ _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 
 testdir=$SCRATCH_MNT/test-$seq
-finished_file=/tmp/finished
+finished_file=$tmp.finished
 rm -rf $finished_file
 mkdir $testdir
 
tests/generic/671 +1 -1
index f20069113..3abe12274 100755
@@ -23,7 +23,7 @@ _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 
 testdir=$SCRATCH_MNT/test-$seq
-finished_file=/tmp/finished
+finished_file=$tmp.finished
 rm -rf $finished_file
 mkdir $testdir
 
tests/generic/672 +1 -1
index 832907de3..a5e65042e 100755
@@ -22,7 +22,7 @@ _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 
 testdir=$SCRATCH_MNT/test-$seq
-finished_file=/tmp/finished
+finished_file=$tmp.finished
 rm -rf $finished_file
 mkdir $testdir
 
tests/generic/722 +4 -11
index 5542c045f..b9c608713 100755
@@ -10,13 +10,6 @@
 . ./common/preamble
 _begin_fstest auto quick fiexchange
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	rm -r -f $tmp.* $dir
-}
-
 # Import common functions.
 . ./common/filter
 . ./common/reflink
@@ -38,8 +31,8 @@ old_a=$(md5sum $SCRATCH_MNT/a | awk '{print $1}')
 old_b=$(md5sum $SCRATCH_MNT/b | awk '{print $1}')
 echo "md5 a: $old_a md5 b: $old_b" >> $seqres.full
 
-od -tx1 -Ad -c $SCRATCH_MNT/a > /tmp/a0
-od -tx1 -Ad -c $SCRATCH_MNT/b > /tmp/b0
+od -tx1 -Ad -c $SCRATCH_MNT/a > $tmp.a0
+od -tx1 -Ad -c $SCRATCH_MNT/b > $tmp.b0
 
 echo swap >> $seqres.full
 $XFS_IO_PROG -c "exchangerange -f $SCRATCH_MNT/a" $SCRATCH_MNT/b
@@ -53,8 +46,8 @@ echo "md5 a: $new_a md5 b: $new_b" >> $seqres.full
 test $old_a = $new_b || echo "scratch file B doesn't match old file A"
 test $old_b = $new_a || echo "scratch file A doesn't match old file B"
 
-od -tx1 -Ad -c $SCRATCH_MNT/a > /tmp/a1
-od -tx1 -Ad -c $SCRATCH_MNT/b > /tmp/b1
+od -tx1 -Ad -c $SCRATCH_MNT/a > $tmp.a1
+od -tx1 -Ad -c $SCRATCH_MNT/b > $tmp.b1
 
 # success, all done
 echo Silence is golden
tests/xfs/601 +1 -1
index bc8fa719b..df382402b 100755
@@ -15,7 +15,7 @@ _register_cleanup "_cleanup" BUS
 _cleanup()
 {
 	cd /
-	rm -rf $tmp.* $testdir
+	rm -rf $tmp.*
 }
 
 # Import common functions.
Signature · 1 line
2.45.2
#40 lore raw

PATCH 25/40 fstests: scale some tests for high CPU count sanity

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Several tests use lots of processes to stress the filesystem. many of them haven't really considered what this means for running the test on high CPU machines (e.g. >32p) and the potential contention and performance issues this might trigger.

Some of these tests simply need to increase the size of the journal. Some need to run on filesystems with high inherent concurrency (e.g. larger AG count). Some need more efficient/faster file creation. And so on.

This commit is a collection of those sorts of changes to improve runtimes on high CPU count machines.

---
 src/aio-dio-regress/aio-last-ref-held-by-io.c | 5 ++++-
 tests/generic/251                             | 5 ++++-
 tests/generic/323                             | 7 +++++--
 tests/generic/530                             | 2 +-
 tests/generic/531                             | 8 +++++++-
 tests/xfs/013                                 | 4 ++--
 tests/xfs/076                                 | 6 +++---
 tests/xfs/176                                 | 6 +++---
 tests/xfs/297                                 | 4 +++-
 tests/xfs/501                                 | 2 +-
 tests/xfs/502                                 | 2 +-
 11 files changed, 34 insertions(+), 17 deletions(-)
src/aio-dio-regress/aio-last-ref-held-by-io.c +4 -1
index a70f2a9b7..7106e30a9 100644
@@ -85,11 +85,14 @@ aio_test_thread(void *data)
 	/*
 	 * Problems have been easier to trigger when spreading the
 	 * workload over the available CPUs.
+	 *
+	 * If CPU hotplug is active, this can randomly fail so dump the error
+	 * to stderror so it can be filtered out easily by the caller.
 	 */
 	CPU_ZERO(&cpuset);
 	CPU_SET(mycpu, &cpuset);
 	if (sched_setaffinity(mytid, sizeof(cpuset), &cpuset)) {
-		printf("FAILED to set thread %d to run on cpu %ld\n",
+		fprintf(stderr, "FAILED to set thread %d to run on cpu %ld\n",
 		       mytid, mycpu);
 	}
 
tests/generic/251 +4 -1
index b432fb119..98986469e 100755
@@ -175,9 +175,12 @@ nproc=20
 # Copy $here to the scratch fs and make coipes of the replica.  The fstests
 # output (and hence $seqres.full) could be in $here, so we need to snapshot
 # $here before computing file checksums.
+#
+# $here/* as the files to copy so we avoid any .git directory that might be
+# much, much larger than the rest of the fstests source tree we are copying.
 content=$SCRATCH_MNT/orig
 mkdir -p $content
-cp -axT $here/ $content/
+cp -ax $here/* $content/
 
 mkdir -p $tmp
 
tests/generic/323 +5 -2
index 457253fee..2dde04d06 100755
@@ -23,12 +23,15 @@ _require_aiodio aio-last-ref-held-by-io
 testfile=$TEST_DIR/aio-testfile
 $XFS_IO_PROG -ftc "pwrite 0 10m" $testfile | _filter_xfs_io
 
-$AIO_TEST 0 100 $testfile
+# This can emit cpu affinity setting failures that aren't considered test
+# failures but cause golden image failures. Redirect the test output to
+# $seqres.full so that it is captured but doesn't directly cause test failures.
+$AIO_TEST 0 100 $testfile 2>> $seqres.full
 if [ $? -ne 0 ]; then
 	exit $status
 fi
 
-$AIO_TEST 1 100 $testfile
+$AIO_TEST 1 100 $testfile 2>> $seqres.full
 if [ $? -ne 0 ]; then
 	exit $status
 fi
tests/generic/530 +1 -1
index 2e47c3e0c..18256b870 100755
@@ -22,7 +22,7 @@ _require_scratch_shutdown
 _require_metadata_journaling
 _require_test_program "t_open_tmpfiles"
 
-_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mkfs "-l size=256m" >> $seqres.full 2>&1
 _scratch_mount
 
 # Set ULIMIT_NOFILE to min(file-max / 2, 50000 files per LOAD_FACTOR)
tests/generic/531 +7 -1
index 0e3564fd4..ed6c3f911 100755
@@ -21,7 +21,13 @@ _require_scratch
 _require_xfs_io_command "-T"
 _require_test_program "t_open_tmpfiles"
 
-_scratch_mkfs >> $seqres.full 2>&1
+# On high CPU count machines, this runs a -lot- of create and unlink
+# concurrency. Set the filesytsem up to handle this.
+if [ $FSTYP = "xfs" ]; then
+	_scratch_mkfs "-d agcount=32" >> $seqres.full 2>&1
+else
+	_scratch_mkfs >> $seqres.full 2>&1
+fi
 _scratch_mount
 
 # Try to load up all the CPUs, two threads per CPU.
tests/xfs/013 +2 -2
index fd3d8c64c..5a92ef084 100755
@@ -28,7 +28,7 @@ _create()
 	mkdir -p $dir
 	for i in $(seq 0 $count)
 	do
-		touch $dir/$i 2>&1 | filter_enospc
+		echo -n > $dir/$i 2>&1 | filter_enospc
 	done
 }
 
@@ -42,7 +42,7 @@ _rand_replace()
 	do
 		file=$((RANDOM % count))
 		rm -f $dir/$file
-		touch $dir/$file 2>&1 | filter_enospc
+		echo -n > $dir/$file 2>&1 | filter_enospc
 	done
 }
 
tests/xfs/076 +3 -3
index 840617ccb..e315a067c 100755
@@ -47,10 +47,10 @@ _alloc_inodes()
 	dir=$1
 
 	i=0
-	while [ true ]; do
-		touch $dir/$i 2>> $seqres.full || break
+	( while [ true ]; do
+		echo -n > $dir/$i || break
 		i=$((i + 1))
-	done
+	done ) >> $seqres.full 2>&1
 }
 
 
tests/xfs/176 +3 -3
index 8e5951ec1..1aa8cde38 100755
@@ -68,10 +68,10 @@ _alloc_inodes()
 	dir=$1
 
 	i=0
-	while [ true ]; do
-		echo -n > $dir/$i >> $seqres.full 2>&1 || break
+	( while [ true ]; do
+		echo -n > $dir/$i || break
 		i=$((i + 1))
-	done
+	done ) >> $seqres.full 2>&1
 }
 
 # Find a sparse inode cluster after logend_agno/logend_agino.
tests/xfs/297 +3 -1
index f9cd2ff12..af6af601a 100755
@@ -34,7 +34,9 @@ _scratch_mount
 STRESS_DIR="$SCRATCH_MNT/testdir"
 mkdir -p $STRESS_DIR
 
-_run_fsstress_bg -d $STRESS_DIR -n 1000 -p 1000 $FSSTRESS_AVOID
+# turn off sync as this can lead to near deadlock conditions due to every
+# fsstress process lockstepping against freeze on large CPU count machines
+_run_fsstress_bg -d $STRESS_DIR -f sync=0 -n 1000 -p 1000 $FSSTRESS_AVOID
 
 # Freeze/unfreeze file system randomly
 echo "Start freeze/unfreeze randomly" | tee -a $seqres.full
tests/xfs/501 +1 -1
index 1da4cbf92..678c51b52 100755
@@ -33,7 +33,7 @@ _require_xfs_sysfs debug/log_recovery_delay
 _require_scratch
 _require_test_program "t_open_tmpfiles"
 
-_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mkfs "-l size=256m" >> $seqres.full 2>&1
 _scratch_mount
 
 # Set ULIMIT_NOFILE to min(file-max / 2, 30000 files per LOAD_FACTOR)
tests/xfs/502 +1 -1
index 52b8e95a2..10b0017f6 100755
@@ -23,7 +23,7 @@ _require_xfs_io_error_injection "iunlink_fallback"
 _require_scratch
 _require_test_program "t_open_tmpfiles"
 
-_scratch_mkfs | _filter_mkfs 2> $tmp.mkfs > /dev/null
+_scratch_mkfs "-l size=256m" | _filter_mkfs 2> $tmp.mkfs > /dev/null
 cat $tmp.mkfs >> $seqres.full
 . $tmp.mkfs
 
Signature · 1 line
2.45.2
#41 lore raw

Re: [PATCH 25/40] fstests: scale some tests for high CPU count sanity

Quoted 55 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Several tests use lots of processes to stress the filesystem. many of them haven't really considered what this means for running the test on high CPU machines (e.g. >32p) and the potential contention and performance issues this might trigger.

Some of these tests simply need to increase the size of the journal. Some need to run on filesystems with high inherent concurrency (e.g. larger AG count). Some need more efficient/faster file creation. And so on.

This commit is a collection of those sorts of changes to improve runtimes on high CPU count machines.

---
 src/aio-dio-regress/aio-last-ref-held-by-io.c | 5 ++++-
 tests/generic/251                             | 5 ++++-
 tests/generic/323                             | 7 +++++--
 tests/generic/530                             | 2 +-
 tests/generic/531                             | 8 +++++++-
 tests/xfs/013                                 | 4 ++--
 tests/xfs/076                                 | 6 +++---
 tests/xfs/176                                 | 6 +++---
 tests/xfs/297                                 | 4 +++-
 tests/xfs/501                                 | 2 +-
 tests/xfs/502                                 | 2 +-
 11 files changed, 34 insertions(+), 17 deletions(-)
src/aio-dio-regress/aio-last-ref-held-by-io.c +4 -1
index a70f2a9b7..7106e30a9 100644
@@ -85,11 +85,14 @@ aio_test_thread(void *data)
 	/*
 	 * Problems have been easier to trigger when spreading the
 	 * workload over the available CPUs.
+	 *
+	 * If CPU hotplug is active, this can randomly fail so dump the error
+	 * to stderror so it can be filtered out easily by the caller.
 	 */
 	CPU_ZERO(&cpuset);
 	CPU_SET(mycpu, &cpuset);
 	if (sched_setaffinity(mytid, sizeof(cpuset), &cpuset)) {
-		printf("FAILED to set thread %d to run on cpu %ld\n",
+		fprintf(stderr, "FAILED to set thread %d to run on cpu %ld\n",
 		       mytid, mycpu);
 	}
 
tests/generic/251 +0 -0
index b432fb119..98986469e 100755

tests/generic/251 nproc=20 · line 183

Quoted 10 lines
tests/generic/251 +4 -1
@@ -175,9 +175,12 @@ nproc=20
 # Copy $here to the scratch fs and make coipes of the replica.  The fstests
 # output (and hence $seqres.full) could be in $here, so we need to snapshot
 # $here before computing file checksums.
+#
+# $here/* as the files to copy so we avoid any .git directory that might be
+# much, much larger than the rest of the fstests source tree we are copying.
 content=$SCRATCH_MNT/orig
 mkdir -p $content
-cp -axT $here/ $content/
+cp -ax $here/* $content/

Hi Dave,

Darrick sent a patch to review this line:
https://lore.kernel.org/fstests/173258395238.4031902.16373799205312238046.stgit@frogsfrogsfrogs/T/#u

Please help to review, if you don't need this change anymore, I'll fix this conflict (by removing your above change on g/251) when I merge this patch.

Thanks,
Zorro

Quoted 160 lines
tests/generic/251 +0 -0
 mkdir -p $tmp
 
tests/generic/323 +5 -2
index 457253fee..2dde04d06 100755
@@ -23,12 +23,15 @@ _require_aiodio aio-last-ref-held-by-io
 testfile=$TEST_DIR/aio-testfile
 $XFS_IO_PROG -ftc "pwrite 0 10m" $testfile | _filter_xfs_io
 
-$AIO_TEST 0 100 $testfile
+# This can emit cpu affinity setting failures that aren't considered test
+# failures but cause golden image failures. Redirect the test output to
+# $seqres.full so that it is captured but doesn't directly cause test failures.
+$AIO_TEST 0 100 $testfile 2>> $seqres.full
 if [ $? -ne 0 ]; then
 	exit $status
 fi
 
-$AIO_TEST 1 100 $testfile
+$AIO_TEST 1 100 $testfile 2>> $seqres.full
 if [ $? -ne 0 ]; then
 	exit $status
 fi
tests/generic/530 +1 -1
index 2e47c3e0c..18256b870 100755
@@ -22,7 +22,7 @@ _require_scratch_shutdown
 _require_metadata_journaling
 _require_test_program "t_open_tmpfiles"
 
-_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mkfs "-l size=256m" >> $seqres.full 2>&1
 _scratch_mount
 
 # Set ULIMIT_NOFILE to min(file-max / 2, 50000 files per LOAD_FACTOR)
tests/generic/531 +7 -1
index 0e3564fd4..ed6c3f911 100755
@@ -21,7 +21,13 @@ _require_scratch
 _require_xfs_io_command "-T"
 _require_test_program "t_open_tmpfiles"
 
-_scratch_mkfs >> $seqres.full 2>&1
+# On high CPU count machines, this runs a -lot- of create and unlink
+# concurrency. Set the filesytsem up to handle this.
+if [ $FSTYP = "xfs" ]; then
+	_scratch_mkfs "-d agcount=32" >> $seqres.full 2>&1
+else
+	_scratch_mkfs >> $seqres.full 2>&1
+fi
 _scratch_mount
 
 # Try to load up all the CPUs, two threads per CPU.
tests/xfs/013 +2 -2
index fd3d8c64c..5a92ef084 100755
@@ -28,7 +28,7 @@ _create()
 	mkdir -p $dir
 	for i in $(seq 0 $count)
 	do
-		touch $dir/$i 2>&1 | filter_enospc
+		echo -n > $dir/$i 2>&1 | filter_enospc
 	done
 }
 
@@ -42,7 +42,7 @@ _rand_replace()
 	do
 		file=$((RANDOM % count))
 		rm -f $dir/$file
-		touch $dir/$file 2>&1 | filter_enospc
+		echo -n > $dir/$file 2>&1 | filter_enospc
 	done
 }
 
tests/xfs/076 +3 -3
index 840617ccb..e315a067c 100755
@@ -47,10 +47,10 @@ _alloc_inodes()
 	dir=$1
 
 	i=0
-	while [ true ]; do
-		touch $dir/$i 2>> $seqres.full || break
+	( while [ true ]; do
+		echo -n > $dir/$i || break
 		i=$((i + 1))
-	done
+	done ) >> $seqres.full 2>&1
 }
 
 
tests/xfs/176 +3 -3
index 8e5951ec1..1aa8cde38 100755
@@ -68,10 +68,10 @@ _alloc_inodes()
 	dir=$1
 
 	i=0
-	while [ true ]; do
-		echo -n > $dir/$i >> $seqres.full 2>&1 || break
+	( while [ true ]; do
+		echo -n > $dir/$i || break
 		i=$((i + 1))
-	done
+	done ) >> $seqres.full 2>&1
 }
 
 # Find a sparse inode cluster after logend_agno/logend_agino.
tests/xfs/297 +3 -1
index f9cd2ff12..af6af601a 100755
@@ -34,7 +34,9 @@ _scratch_mount
 STRESS_DIR="$SCRATCH_MNT/testdir"
 mkdir -p $STRESS_DIR
 
-_run_fsstress_bg -d $STRESS_DIR -n 1000 -p 1000 $FSSTRESS_AVOID
+# turn off sync as this can lead to near deadlock conditions due to every
+# fsstress process lockstepping against freeze on large CPU count machines
+_run_fsstress_bg -d $STRESS_DIR -f sync=0 -n 1000 -p 1000 $FSSTRESS_AVOID
 
 # Freeze/unfreeze file system randomly
 echo "Start freeze/unfreeze randomly" | tee -a $seqres.full
tests/xfs/501 +1 -1
index 1da4cbf92..678c51b52 100755
@@ -33,7 +33,7 @@ _require_xfs_sysfs debug/log_recovery_delay
 _require_scratch
 _require_test_program "t_open_tmpfiles"
 
-_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mkfs "-l size=256m" >> $seqres.full 2>&1
 _scratch_mount
 
 # Set ULIMIT_NOFILE to min(file-max / 2, 30000 files per LOAD_FACTOR)
tests/xfs/502 +1 -1
index 52b8e95a2..10b0017f6 100755
@@ -23,7 +23,7 @@ _require_xfs_io_error_injection "iunlink_fallback"
 _require_scratch
 _require_test_program "t_open_tmpfiles"
 
-_scratch_mkfs | _filter_mkfs 2> $tmp.mkfs > /dev/null
+_scratch_mkfs "-l size=256m" | _filter_mkfs 2> $tmp.mkfs > /dev/null
 cat $tmp.mkfs >> $seqres.full
 . $tmp.mkfs
 
Signature · 1 line
2.45.2
#42 lore raw

PATCH 26/40 generic/310: cleanup killing background processes

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Use the trick we used with fsstress of copying the binary to a test specific name so that we can simply use pkill to reliably kill the background processes this test runs. Also use SIGPIPE to avoid bash from throwing out "Killed" errors.

---
 tests/generic/310 | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)
tests/generic/310 +9 -12
index 1ab81dd2e..52babfdc8 100755
@@ -29,11 +29,8 @@ _begin_fstest auto
 # Override the default cleanup function.
 _cleanup()
 {
-	if [ -n "$readdir_pid" ]; then
-		pkill --parent "$readdir_pid" t_readdir > /dev/null 2>&1
-		kill -9 $readdir_pid > /dev/null 2>&1
-		wait
-	fi
+	pkill -9 $seq.t_readdir > /dev/null 2>&1
+	wait
 	rm -rf $TEST_DIR/tmp
 	rm -f $tmp.*
 }
@@ -74,6 +71,8 @@ check_kernel_bug()
 
 RUN_TIME=$((30 * $TIME_FACTOR))
 
+cp $here/src/t_readdir_1 $TEST_DIR/$seq.t_readdir_1
+cp $here/src/t_readdir_2 $TEST_DIR/$seq.t_readdir_2
 SEQ_DIR=$TEST_DIR/$seq
 mkdir -p $SEQ_DIR
 for n in {1..4096}; do
@@ -82,12 +81,11 @@ done
 
 _test_read()
 {
-	$here/src/t_readdir_1 $SEQ_DIR > /dev/null 2>&1 &
-	readdir_pid=$!
+	 $TEST_DIR/$seq.t_readdir_1 $SEQ_DIR > /dev/null 2>&1 &
 	sleep $RUN_TIME
-	kill $readdir_pid
-	unset readdir_pid
+	pkill -PIPE $seq.t_readdir_1
 	wait
+
 	check_kernel_bug
 	if [ $? -ne 0 ]; then
 		_fatal "kernel bug detected, check dmesg for more infomation."
@@ -96,11 +94,10 @@ _test_read()
 
 _test_lseek()
 {
-	$here/src/t_readdir_2 $SEQ_DIR > /dev/null 2>&1 &
+	$TEST_DIR/$seq.t_readdir_2 $SEQ_DIR > /dev/null 2>&1 &
 	readdir_pid=$!
 	sleep $RUN_TIME
-	kill $readdir_pid
-	unset readdir_pid
+	pkill -PIPE $seq.t_readdir_2
 	wait
 
 	check_kernel_bug
Signature · 1 line
2.45.2
#43 lore raw

PATCH 27/40 filter: handle mount errors from CONFIG_BLK_DEV_WRITE_MOUNTED=y

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Kernels post 6.x may have CONFIG_BLK_DEV_WRITE_MOUNTED=y which prevents mount from opening the block device on a mounted filesystem. This results in an error such as:

mount: <dev>: Can't open blockdev

which is not the error that callers of _filter_error_mount() are looking for. It is, however, a direct result of the test trying to mount an alreayd mounted filesystem, so it is reflecting the same error case. Hence this mismatch in errors should not fail the test.

Catch this mount error and convert it to the expected "already mounted" error for the tests that exercise this behaviour.

There is also a minor test change here to push mount failure information to $seqres.full in the cases where mount errors occur.

---
 common/filter | 8 +++++++-
 tests/xfs/154 | 1 +
 2 files changed, 8 insertions(+), 1 deletion(-)
common/filter +7 -1
index 36d51bd95..6a509d8b5 100644
@@ -430,6 +430,10 @@ _filter_ro_mount() {
 # v2.38 and later:
 # dmesg(1) may have more information after failed mount mount system call
 #
+# Kernels post 6.x may have CONFIG_BLK_DEV_WRITE_MOUNTED=y which prevents mount
+# from opening the block device on a mounted filesytsem:
+# mount: <dev>: Can't open blockdev
+#
 # This is also true for ESTALE error. So let's remove all the changing parts
 # and keep the 'prior to v2.21' format:
 # mount: Structure needs cleaning
@@ -437,7 +441,9 @@ _filter_ro_mount() {
 _filter_error_mount()
 {
 	grep -v "dmesg(1) may have more information after failed mount" | \
-		sed -e "s/mount:\(.*failed:\)/mount:/" | _filter_ending_dot
+		sed -e "s/mount:\(.*failed:\)/mount:/" \
+		    -e "s/mount: SCRATCH_DEV: Can't open blockdev/mount: TEST_DIR\/extra_mnt: SCRATCH_DEV already mounted or mount point busy/" |\
+		    _filter_ending_dot
 }
 
 # Similar to _filter_error_mount, filter a busy mount output.
tests/xfs/154 +1 -0
index e8494bbd6..674d06784 100755
@@ -46,6 +46,7 @@ test $? -eq 137 || echo "repair should have been killed??"
 _check_scratch_xfs_features NEEDSREPAIR
 _try_scratch_mount &> $tmp.mount
 res=$?
+cat $tmp.mount >> $seqres.full
 _filter_error_mount < $tmp.mount
 if [ $res -eq 0 ]; then
 	echo "Should not be able to mount after needsrepair crash"
Signature · 1 line
2.45.2
#44 lore raw

PATCH 28/40 filters: add a filter that accepts EIO instead of other errors

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Running a dm-flakey or dm-error test that loads a table that returns EIO to all IO and then running a command that is expected to fail with a specific error is racy.

If there is memory pressure at the same time that the table is loaded, cached inodes can be turfed from memory and the command then needs to read the inode it is about to act on from disk again. This results in the inode read getting EIO and failing (e.g. xfs_io will return a stat() error) rather than having the desired operation fail.

This results in spurious test failures that look like this:

generic/331 - output mismatch (see /mnt/xfs/runner-41/results-2024-11-20-10:57:31/xfs/generic/331.out.bad)

    --- tests/generic/331.out   2022-12-21 15:53:25.487044098 +1100
    +++ /mnt/xfs/runner-41/results-2024-11-20-10:57:31/xfs/generic/331.out.bad  2024-11-20 11:02:12.123572607 +1100
    @@ -5,7 +5,8 @@
     1886e67cf8783e89ce6ddc5bb09a3944  SCRATCH_MNT/test-331/file1
     1886e67cf8783e89ce6ddc5bb09a3944  SCRATCH_MNT/test-331/file2
     CoW and unmount
    -fdatasync: Input/output error
    +/mnt/xfs/runner-41/scratch/test-331/file2: Input/output error
    +stat: Input/output error
     Compare files
    ...

Add a new "flakey EIO filter" that will catch -any- EIO error from the command and change it to the error we expected to see.

---
 common/filter     | 9 +++++++++
 tests/btrfs/160   | 3 ++-
 tests/generic/252 | 3 ++-
 tests/generic/329 | 3 ++-
 tests/generic/331 | 5 +++--
 tests/generic/484 | 3 ++-
 tests/generic/743 | 4 +++-
 tests/xfs/237     | 3 ++-
 tests/xfs/240     | 5 +++--
 9 files changed, 28 insertions(+), 10 deletions(-)
common/filter +9 -0
index 6a509d8b5..7e02ded37 100644
@@ -662,5 +662,14 @@ _filter_trailing_whitespace()
 	sed -E -e "s/\s+$//"
 }
 
+# Catch -any- EIO error regardless of it's source and replace it with the
+# supplied error message.
+_filter_flakey_EIO()
+{
+	local message="$*"
+
+	sed -e "s#.*: Input\/output error#$message#"
+}
+
 # make sure this script returns success
 /bin/true
tests/btrfs/160 +2 -1
index 04ed1f176..c4a01d330 100755
@@ -66,7 +66,8 @@ _dmerror_load_working_table
 
 # open again and call fsync
 echo "The following fsync should fail with EIO:"
-$XFS_IO_PROG -c fsync $testfile
+$XFS_IO_PROG -c fsync $testfile |& \
+		_filter_flakey_EIO "fsync: Input/output error"
 echo "done"
 
 # close file
tests/generic/252 +2 -1
index ddf950e34..2ad0524c2 100755
@@ -66,7 +66,8 @@ $XFS_IO_PROG -f -c "pwrite -S 0x63 $bufsize 1" $testdir/file2 >> $seqres.full
 $XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $TEST_DIR/moo >> $seqres.full
 _scratch_sync
 _dmerror_load_error_table
-$AIO_TEST -a $alignment -f DIRECT -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full
+$AIO_TEST -a $alignment -f DIRECT -b $bufsize $TEST_DIR/moo $testdir/file2 |& \
+		_filter_flakey_EIO "write missed bytes expect 8388608 got 0"
 _dmerror_load_working_table
 _dmerror_unmount
 _dmerror_mount
tests/generic/329 +2 -1
index e4300f92c..96a5ad54c 100755
@@ -59,7 +59,8 @@ $XFS_IO_PROG -f -c "pwrite -S 0x63 $bufsize 1" $testdir/file2 >> $seqres.full
 $XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $TEST_DIR/moo >> $seqres.full
 _scratch_sync
 _dmerror_load_error_table
-$AIO_TEST -a $alignment -f DIRECT -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full
+$AIO_TEST -a $alignment -f DIRECT -b $bufsize $TEST_DIR/moo $testdir/file2 |& \
+		_filter_flakey_EIO "write missed bytes expect 8388608 got 0"
 _dmerror_load_working_table
 _dmerror_unmount
 _dmerror_mount
tests/generic/331 +3 -2
index fe12ec4eb..704bb1283 100755
@@ -59,8 +59,9 @@ $XFS_IO_PROG -f -c "pwrite -S 0x63 $bufsize 1" $testdir/file2 >> $seqres.full
 $XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $TEST_DIR/moo >> $seqres.full
 _scratch_sync
 _dmerror_load_error_table
-$AIO_TEST -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full
-$XFS_IO_PROG -c "fdatasync" $testdir/file2
+$AIO_TEST -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full 2>&1
+$XFS_IO_PROG -c "fdatasync" $testdir/file2 |& \
+		_filter_flakey_EIO "fdatasync: Input/output error"
 _dmerror_load_working_table
 _dmerror_unmount
 _dmerror_mount
tests/generic/484 +2 -1
index 09c2c5598..ec50735a5 100755
@@ -54,7 +54,8 @@ $XFS_IO_PROG -c "pwrite -W -q 0 $datalen" $testfile
 _dmerror_load_error_table
 
 # rewrite the data and call fdatasync
-$XFS_IO_PROG -c "pwrite -w -q 0 $datalen" $testfile
+$XFS_IO_PROG -c "pwrite -w -q 0 $datalen" $testfile |& \
+		_filter_flakey_EIO "fdatasync: Input/output error"
 
 # heal the device error
 _dmerror_load_working_table
tests/generic/743 +3 -1
index 228ba764e..efdeec82d 100755
@@ -21,6 +21,7 @@ _cleanup()
 }
 
 # Import common functions.
+. ./common/filter
 . ./common/dmerror
 
 _fixed_by_kernel_commit 631426ba1d45 \
@@ -55,7 +56,8 @@ _dmerror_mount
 stat "$SCRATCH_MNT/a" >> $seqres.full
 echo read with IO errors
 _dmerror_load_error_table
-$TIMEOUT_PROG -s KILL 10s $XFS_IO_PROG -c "mmap -r 0 $filesz" -c "madvise -R 0 $filesz" "$SCRATCH_MNT/a"
+$TIMEOUT_PROG -s KILL 10s $XFS_IO_PROG -c "mmap -r 0 $filesz" -c "madvise -R 0 $filesz" "$SCRATCH_MNT/a" |& \
+		_filter_flakey_EIO "madvise: Bad address"
 _dmerror_load_working_table
 
 # success, all done
tests/xfs/237 +2 -1
index 91f56d6c1..eb5dc5d2f 100755
@@ -68,7 +68,8 @@ $XFS_IO_PROG -f -c "pwrite -S 0x63 $bufsize 1" $testdir/file2 >> $seqres.full
 $XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $TEST_DIR/moo >> $seqres.full
 _scratch_sync
 _dmerror_load_error_table
-$AIO_TEST -a $alignment -f DIRECT -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full
+$AIO_TEST -a $alignment -f DIRECT -b $bufsize $TEST_DIR/moo $testdir/file2 |& \
+		_filter_flakey_EIO "write missed bytes expect 8388608 got 0"
 _dmerror_load_working_table
 _dmerror_unmount
 _dmerror_mount
tests/xfs/240 +3 -2
index e95cf3f5d..8916828a8 100755
@@ -64,8 +64,9 @@ $XFS_IO_PROG -f -c "pwrite -S 0x63 $bufsize 1" $testdir/file2 >> $seqres.full
 $XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $TEST_DIR/moo >> $seqres.full
 _scratch_sync
 _dmerror_load_error_table
-$AIO_TEST -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full
-$XFS_IO_PROG -c "fdatasync" $testdir/file2
+$AIO_TEST -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full 2>&1
+$XFS_IO_PROG -c "fdatasync" $testdir/file2 |& \
+		_filter_flakey_EIO "fdatasync: Input/output error"
 _dmerror_load_working_table
 _dmerror_unmount
 _dmerror_mount
Signature · 1 line
2.45.2
#45 lore raw

PATCH 29/40 generic/085: general cleanup for reliability and debugging

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

This test was quite unreliable during development of the parallel check runner. It redirects all errors to /dev/null, so there was no way to debug it when it failed.

Use common mount/unmount helpers, redirect errors to $seqres.full, make sure the cleanup code is always run at test exit and only attempt to kill processes if they are still running during cleanup.

---
 tests/generic/085 | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)
tests/generic/085 +12 -9
index 8fbcdce2e..7671a36ab 100755
@@ -16,18 +16,20 @@ _begin_fstest auto freeze mount
 # Override the default cleanup function.
 _cleanup()
 {
+	cleanup_dmdev
 	cd /
 	rm -f $tmp.*
-	cleanup_dmdev
 }
 
 cleanup_dmdev()
 {
+	if [ -n "$pid" ]; then
+		kill -9 $pid 2>> $seqres.full
+		wait $pid 2>> $seqres.full
+	fi
 	# in case it's still suspended and/or mounted
-	$DMSETUP_PROG resume $lvdev >/dev/null 2>&1
-	[ -n "$pid" ] && kill -9 $pid 2>/dev/null
-	wait $pid
-	_unmount $lvdev >/dev/null 2>&1
+	$DMSETUP_PROG resume $lvdev >> $seqres.full 2>&1
+	_unmount -q $SCRATCH_MNT
 	_dmsetup_remove $node
 }
 
@@ -64,18 +66,19 @@ setup_dmdev
 # important, as long as they're racing with each other. So just throw away the
 # outputs and ignore the results.
 for ((i=0; i<100; i++)); do
-	$DMSETUP_PROG suspend $lvdev >/dev/null 2>&1
-	$DMSETUP_PROG resume $lvdev >/dev/null 2>&1
+	$DMSETUP_PROG suspend $lvdev >> $seqres.full 2>&1
+	$DMSETUP_PROG resume $lvdev >> $seqres.full 2>&1
 done &
 pid=$!
 for ((i=0; i<100; i++)); do
-	_mount $lvdev $SCRATCH_MNT >/dev/null 2>&1
-	_unmount $lvdev >/dev/null 2>&1
+	_mount $lvdev $SCRATCH_MNT >> $seqres.full 2>&1
+	_unmount $lvdev >> $seqres.full 2>&1
 done &
 pid="$pid $!"
 
 wait $pid
 unset pid
 
+cleanup_dmdev
 status=0
 exit
Signature · 1 line
2.45.2
#46 lore raw

PATCH 30/40 fstests: don't use directory stacks

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Using bash directory stacking (pushd, popd, etc) seems to be somewhat unreliable. I've been seeing occasional random failures from both pushd and popd commands that cause the test to fail, and there does not appear to be any reason for the failures occurring.

Rather than wasting time chasing ghosts, just get rid of the directory stacking altogether.

---
 tests/generic/099 |  8 +++-----
 tests/generic/109 |  5 +----
 tests/generic/135 | 24 +++++++++++-------------
 tests/generic/707 |  7 ++++---
 4 files changed, 19 insertions(+), 25 deletions(-)
tests/generic/099 +3 -5
index c7d5932b6..980fd38f3 100755
@@ -253,15 +253,13 @@ echo ""
 echo "=== Recursive change ACL ==="
 rm -fr root
 mkdir root
-pushd root >/dev/null
 # create an arbitrary little tree
-for i in 1 2 3 4 5 6 7 8 9 0
-do
+( cd root ; for i in 1 2 3 4 5 6 7 8 9 0; do
 	mkdir -p a/$i
 	mkdir -p b/c$i/$i
 	touch a/$i/mumble
-done
-popd >/dev/null
+done )
+
 chown -R 12345:54321 root
 echo "Change #1..."
 _runas -u 12345 -g 54321 -- chacl -r u::rwx,g::-w-,o::--x root
tests/generic/109 +1 -4
index 2b0b438cf..894e07167 100755
@@ -54,10 +54,7 @@ _scratch_mount >> $seqres.full 2>&1
 # Test different directory sizes to test various directory formats
 for f in 1 2 3 4 5 8 12 18 27 40 60 90 135 202 303 454 681 1020 1530 2295; do
 	mkdir $SCRATCH_MNT/dir$f
-	pushd $SCRATCH_MNT/dir$f >/dev/null
-	filldir $f
-	renamedir
-	popd >/dev/null
+	( cd $SCRATCH_MNT/dir$f ; filldir $f ; renamedir )
 done
 
 status=0
tests/generic/135 +11 -13
index 30c52af72..cb9407fdf 100755
@@ -21,30 +21,28 @@ _scratch_mount
 cd $SCRATCH_MNT
 
 # create file with async I/O
-$XFS_IO_PROG -f -c 'pwrite -b 4k -S 0x12 0 4k' async_file > /dev/null
+$XFS_IO_PROG -f -c 'pwrite -b 4k -S 0x12 0 4k' $SCRATCH_MNT/async_file > /dev/null
 
 # create file with sync I/O
-$XFS_IO_PROG -f -s -c 'pwrite -b 4k -S 0x34 0 4k' sync_file > /dev/null
+$XFS_IO_PROG -f -s -c 'pwrite -b 4k -S 0x34 0 4k' $SCRATCH_MNT/sync_file > /dev/null
 
 # create file with direct I/O
-$XFS_IO_PROG -f -d -c 'pwrite -b 4k -S 0x56 0 4k' direct_file > /dev/null
+$XFS_IO_PROG -f -d -c 'pwrite -b 4k -S 0x56 0 4k' $SCRATCH_MNT/direct_file > /dev/null
 
 # create file, truncate and then dirty again
-$XFS_IO_PROG -f -c 'pwrite -b 4k -S 0x78 0 4k' trunc_file > /dev/null
-$XFS_IO_PROG -f -c 'truncate 2k' trunc_file > /dev/null
-$XFS_IO_PROG -c 'pwrite 1k 0 1k' trunc_file > /dev/null
+$XFS_IO_PROG -f -c 'pwrite -b 4k -S 0x78 0 4k' $SCRATCH_MNT/trunc_file > /dev/null
+$XFS_IO_PROG -f -c 'truncate 2k' $SCRATCH_MNT/trunc_file > /dev/null
+$XFS_IO_PROG -c 'pwrite 1k 0 1k' $SCRATCH_MNT/trunc_file > /dev/null
 
-pushd / > /dev/null
 _scratch_cycle_mount
-popd > /dev/null
 
 # check file size and contents
-od -Ad -x async_file
-od -Ad -x sync_file
-od -Ad -x direct_file
-od -Ad -x trunc_file
+od -Ad -x $SCRATCH_MNT/async_file
+od -Ad -x $SCRATCH_MNT/sync_file
+od -Ad -x $SCRATCH_MNT/direct_file
+od -Ad -x $SCRATCH_MNT/trunc_file
 
-rm -f async_file sync_file direct_file trunc_file
+rm -f $SCRATCH_MNT/*
 
 status=0
 exit 
tests/generic/707 +4 -3
index fd02eacf9..3d8fac4bd 100755
@@ -42,7 +42,7 @@ create_files()
 	# We use slightly longer file name to make directory grow faster and
 	# hopefully convert between various types
 	for (( i = 0; i < $files; i++ )); do
-		touch somewhatlongerfilename$i
+		echo -n > somewhatlongerfilename$i
 	done
 }
 
@@ -50,12 +50,13 @@ for (( i = 0; i <= $moves; i++ )); do
 	mkdir $SCRATCH_MNT/dir$i
 done
 
+start_dir=$PWD
 for (( l = 0; l < $loops; l++ )); do
 	mkdir $SCRATCH_MNT/dir0/dir
-	pushd $SCRATCH_MNT/dir0/dir &>/dev/null
+	cd $SCRATCH_MNT/dir0/dir
 	create_files &
 	BGPID=$!
-	popd &>/dev/null
+	cd $start_dir
 	for (( i = 0; i < $moves; i++ )); do
 		mv $SCRATCH_MNT/dir$i/dir $SCRATCH_MNT/dir$((i+1))/dir
 	done
Signature · 1 line
2.45.2
#47 lore raw

Re: [PATCH 30/40] fstests: don't use directory stacks

Quoted 62 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Using bash directory stacking (pushd, popd, etc) seems to be somewhat unreliable. I've been seeing occasional random failures from both pushd and popd commands that cause the test to fail, and there does not appear to be any reason for the failures occurring.

Rather than wasting time chasing ghosts, just get rid of the directory stacking altogether.

---
 tests/generic/099 |  8 +++-----
 tests/generic/109 |  5 +----
 tests/generic/135 | 24 +++++++++++-------------
 tests/generic/707 |  7 ++++---
 4 files changed, 19 insertions(+), 25 deletions(-)
tests/generic/099 +3 -5
index c7d5932b6..980fd38f3 100755
@@ -253,15 +253,13 @@ echo ""
 echo "=== Recursive change ACL ==="
 rm -fr root
 mkdir root
-pushd root >/dev/null
 # create an arbitrary little tree
-for i in 1 2 3 4 5 6 7 8 9 0
-do
+( cd root ; for i in 1 2 3 4 5 6 7 8 9 0; do
 	mkdir -p a/$i
 	mkdir -p b/c$i/$i
 	touch a/$i/mumble
-done
-popd >/dev/null
+done )
+
 chown -R 12345:54321 root
 echo "Change #1..."
 _runas -u 12345 -g 54321 -- chacl -r u::rwx,g::-w-,o::--x root
tests/generic/109 +1 -4
index 2b0b438cf..894e07167 100755
@@ -54,10 +54,7 @@ _scratch_mount >> $seqres.full 2>&1
 # Test different directory sizes to test various directory formats
 for f in 1 2 3 4 5 8 12 18 27 40 60 90 135 202 303 454 681 1020 1530 2295; do
 	mkdir $SCRATCH_MNT/dir$f
-	pushd $SCRATCH_MNT/dir$f >/dev/null
-	filldir $f
-	renamedir
-	popd >/dev/null
+	( cd $SCRATCH_MNT/dir$f ; filldir $f ; renamedir )
 done
 
 status=0
tests/generic/135 +0 -0
index 30c52af72..cb9407fdf 100755

tests/generic/135 _scratch_mount · line 21

Quoted 1 line
tests/generic/135 +0 -0
@@ -21,30 +21,28 @@ _scratch_mount
 cd $SCRATCH_MNT

^^^^^^^^^^^^^^^

Below "_scratch_cycle_mount" fails, due to above line. I'll remove it.

Thanks,
Zorro

Quoted 73 lines
tests/generic/135 +11 -13
 # create file with async I/O
-$XFS_IO_PROG -f -c 'pwrite -b 4k -S 0x12 0 4k' async_file > /dev/null
+$XFS_IO_PROG -f -c 'pwrite -b 4k -S 0x12 0 4k' $SCRATCH_MNT/async_file > /dev/null
 
 # create file with sync I/O
-$XFS_IO_PROG -f -s -c 'pwrite -b 4k -S 0x34 0 4k' sync_file > /dev/null
+$XFS_IO_PROG -f -s -c 'pwrite -b 4k -S 0x34 0 4k' $SCRATCH_MNT/sync_file > /dev/null
 
 # create file with direct I/O
-$XFS_IO_PROG -f -d -c 'pwrite -b 4k -S 0x56 0 4k' direct_file > /dev/null
+$XFS_IO_PROG -f -d -c 'pwrite -b 4k -S 0x56 0 4k' $SCRATCH_MNT/direct_file > /dev/null
 
 # create file, truncate and then dirty again
-$XFS_IO_PROG -f -c 'pwrite -b 4k -S 0x78 0 4k' trunc_file > /dev/null
-$XFS_IO_PROG -f -c 'truncate 2k' trunc_file > /dev/null
-$XFS_IO_PROG -c 'pwrite 1k 0 1k' trunc_file > /dev/null
+$XFS_IO_PROG -f -c 'pwrite -b 4k -S 0x78 0 4k' $SCRATCH_MNT/trunc_file > /dev/null
+$XFS_IO_PROG -f -c 'truncate 2k' $SCRATCH_MNT/trunc_file > /dev/null
+$XFS_IO_PROG -c 'pwrite 1k 0 1k' $SCRATCH_MNT/trunc_file > /dev/null
 
-pushd / > /dev/null
 _scratch_cycle_mount
-popd > /dev/null
 
 # check file size and contents
-od -Ad -x async_file
-od -Ad -x sync_file
-od -Ad -x direct_file
-od -Ad -x trunc_file
+od -Ad -x $SCRATCH_MNT/async_file
+od -Ad -x $SCRATCH_MNT/sync_file
+od -Ad -x $SCRATCH_MNT/direct_file
+od -Ad -x $SCRATCH_MNT/trunc_file
 
-rm -f async_file sync_file direct_file trunc_file
+rm -f $SCRATCH_MNT/*
 
 status=0
 exit 
tests/generic/707 +4 -3
index fd02eacf9..3d8fac4bd 100755
@@ -42,7 +42,7 @@ create_files()
 	# We use slightly longer file name to make directory grow faster and
 	# hopefully convert between various types
 	for (( i = 0; i < $files; i++ )); do
-		touch somewhatlongerfilename$i
+		echo -n > somewhatlongerfilename$i
 	done
 }
 
@@ -50,12 +50,13 @@ for (( i = 0; i <= $moves; i++ )); do
 	mkdir $SCRATCH_MNT/dir$i
 done
 
+start_dir=$PWD
 for (( l = 0; l < $loops; l++ )); do
 	mkdir $SCRATCH_MNT/dir0/dir
-	pushd $SCRATCH_MNT/dir0/dir &>/dev/null
+	cd $SCRATCH_MNT/dir0/dir
 	create_files &
 	BGPID=$!
-	popd &>/dev/null
+	cd $start_dir
 	for (( i = 0; i < $moves; i++ )); do
 		mv $SCRATCH_MNT/dir$i/dir $SCRATCH_MNT/dir$((i+1))/dir
 	done
Signature · 1 line
2.45.2
#48 lore raw

Re: [PATCH 30/40] fstests: don't use directory stacks

Quoted 66 lines · Zorro Lang
Quoted 62 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Using bash directory stacking (pushd, popd, etc) seems to be somewhat unreliable. I've been seeing occasional random failures from both pushd and popd commands that cause the test to fail, and there does not appear to be any reason for the failures occurring.

Rather than wasting time chasing ghosts, just get rid of the directory stacking altogether.

---
 tests/generic/099 |  8 +++-----
 tests/generic/109 |  5 +----
 tests/generic/135 | 24 +++++++++++-------------
 tests/generic/707 |  7 ++++---
 4 files changed, 19 insertions(+), 25 deletions(-)
tests/generic/099 +3 -5
index c7d5932b6..980fd38f3 100755
@@ -253,15 +253,13 @@ echo ""
 echo "=== Recursive change ACL ==="
 rm -fr root
 mkdir root
-pushd root >/dev/null
 # create an arbitrary little tree
-for i in 1 2 3 4 5 6 7 8 9 0
-do
+( cd root ; for i in 1 2 3 4 5 6 7 8 9 0; do
 	mkdir -p a/$i
 	mkdir -p b/c$i/$i
 	touch a/$i/mumble
-done
-popd >/dev/null
+done )
+
 chown -R 12345:54321 root
 echo "Change #1..."
 _runas -u 12345 -g 54321 -- chacl -r u::rwx,g::-w-,o::--x root
tests/generic/109 +1 -4
index 2b0b438cf..894e07167 100755
@@ -54,10 +54,7 @@ _scratch_mount >> $seqres.full 2>&1
 # Test different directory sizes to test various directory formats
 for f in 1 2 3 4 5 8 12 18 27 40 60 90 135 202 303 454 681 1020 1530 2295; do
 	mkdir $SCRATCH_MNT/dir$f
-	pushd $SCRATCH_MNT/dir$f >/dev/null
-	filldir $f
-	renamedir
-	popd >/dev/null
+	( cd $SCRATCH_MNT/dir$f ; filldir $f ; renamedir )
 done
 
 status=0
tests/generic/135 +0 -0
index 30c52af72..cb9407fdf 100755

tests/generic/135 _scratch_mount · line 21

Quoted 1 line
tests/generic/135 +0 -0
@@ -21,30 +21,28 @@ _scratch_mount
 cd $SCRATCH_MNT

^^^^^^^^^^^^^^^

Quoted 1 line

Below "_scratch_cycle_mount" fails, due to above line. I'll remove it.

I've updated my local tree with this fix and the others that you've noticed. Thanks!

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#49 lore raw

PATCH 31/40 fstests: clean up a couple of dm-flakey tests

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Just little things I've found that should be cleaned up.

---
 tests/generic/066 | 3 +++
 tests/generic/321 | 4 +++-
 tests/xfs/051     | 7 +++++++
 3 files changed, 13 insertions(+), 1 deletion(-)
tests/generic/066 +3 -0
index 37d04e069..9e4047a11 100755
@@ -24,6 +24,7 @@ _begin_fstest auto quick attr metadata log
 _cleanup()
 {
 	_cleanup_flakey
+	cd /
 	rm -f $tmp.*
 }
 
@@ -88,5 +89,7 @@ _flakey_drop_and_remount
 echo "xattr names and values after second fsync log replay:"
 _getfattr --absolute-names --dump $SCRATCH_MNT/foobar | _filter_scratch
 
+_unmount_flakey
+
 status=0
 exit
tests/generic/321 +3 -1
index 73a892334..df8f13597 100755
@@ -13,6 +13,8 @@ _begin_fstest auto quick metadata log
 _cleanup()
 {
 	_cleanup_flakey
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -57,7 +59,7 @@ _rename_test()
 	$XFS_IO_PROG -c "fsync" $SCRATCH_MNT/bar
 	$XFS_IO_PROG -c "fsync" $SCRATCH_MNT/bar/foo
 
-	_flakey_drop_and_remount\
+	_flakey_drop_and_remount
 
 	_ls_l $SCRATCH_MNT | tail -n +2 | awk '{ print $1, $9 }'
 	_ls_l $SCRATCH_MNT/bar | tail -n +2 | awk '{ print $1, $9 }'
tests/xfs/051 +7 -0
index fe3d75cab..ddc28ac97 100755
@@ -11,6 +11,13 @@
 . ./common/preamble
 _begin_fstest shutdown auto log metadata
 
+_cleanup()
+{
+	_cleanup_flakey
+	cd /
+	rm -f $tmp.*
+}
+
 # Import common functions.
 . ./common/dmflakey
 
Signature · 1 line
2.45.2
#50 lore raw

PATCH 32/40 fstests: clean up termination of various tests

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Accumulated minor fixes to improve reliablity of the termination of various tests when interrupted.

---
 tests/generic/084 | 12 +++++++++---
 tests/generic/464 | 10 +++++-----
 tests/generic/650 |  8 ++++----
 tests/xfs/013     |  1 +
 tests/xfs/558     |  1 +
 5 files changed, 20 insertions(+), 12 deletions(-)
tests/generic/084 +9 -3
index 137ba23cf..fe758f118 100755
@@ -12,6 +12,15 @@
 . ./common/preamble
 _begin_fstest auto metadata quick
 
+_cleanup()
+{
+	[ -n "$link_pids" ] && kill $link_pids
+	[ -n "$open_pid" ] && kill $open_pid
+	wait
+	cd /
+	rm -f $tmp.*
+}
+
 # Import common functions.
 . ./common/filter
 
@@ -57,9 +66,6 @@ done &
 sleep 5
 kill $! >/dev/null 2>&1
 
-kill $open_pid $link_pids >/dev/null 2>&1
-wait $open_pid $link_pids
-
 # all done, no oops/hang expected, _check_filesystems checks SCRATCH_DEV after test
 status=0
 exit
tests/generic/464 +5 -5
index f114208f9..ebf475094 100755
@@ -66,22 +66,22 @@ _scratch_mount
 # for each operation and runs for $LOOP_TIME seconds, and check filesystem
 # consistency after each iteration
 for i in `seq 1 $LOOP_CNT`; do
-	rm -f $stop
+	touch $stop
 	for j in `seq 1 $PROC_CNT`; do
-		while [ ! -e $stop ]; do
+		while [ -e $stop ]; do
 			do_write
 		done &
 
-		while [ ! -e $stop ]; do
+		while [ -e $stop ]; do
 			do_append
 		done &
 
-		while [ ! -e $stop ]; do
+		while [ -e $stop ]; do
 			do_writeback
 		done &
 	done
 	sleep $LOOP_TIME
-	touch $stop
+	rm -f $stop
 	wait
 
 	_scratch_unmount
tests/generic/650 +4 -4
index 36a23e48d..60f86fdf5 100755
@@ -52,8 +52,6 @@ nr_hotplug_cpus="${#hotplug_cpus[@]}"
 test "$nr_hotplug_cpus" -gt 0 || _notrun "CPU hotplugging not supported"
 
 stress_dir="$TEST_DIR/$seq"
-rm -r -f "$stress_dir"
-mkdir -p "$stress_dir"
 
 echo "Silence is golden."
 
@@ -79,12 +77,14 @@ fsstress_args+=(--duration="$((SOAK_DURATION / 10))")
 nr_ops=$((2500 * TIME_FACTOR))
 fsstress_args+=(-n $nr_ops)
 for ((i = 0; i < 10; i++)); do
-	_run_fsstress_bg -w "${fsstress_args[@]}"
-	_wait_for_fsstress
+	rm -rf "$stress_dir"
+	mkdir -p "$stress_dir"
+	_run_fsstress "${fsstress_args[@]}"
 	_test_cycle_mount
 done
 
 rm -f $sentinel_file
+wait
 
 # success, all done
 status=0
tests/xfs/013 +1 -0
index 5a92ef084..fd011445d 100755
@@ -120,6 +120,7 @@ _kill_fsstress
 # clean out the competing fsstress allocations, then everything else
 rm -rf $SCRATCH_MNT/fsstress
 rm -rf $SCRATCH_MNT/dir*
+wait
 
 _scratch_unmount
 
tests/xfs/558 +1 -0
index ba137ae8d..c594d8ccd 100755
@@ -193,6 +193,7 @@ wait_for_errortag &
 $XFS_IO_PROG -c "pwrite -S 0x56 $((2 * blksz)) $((2 * blksz))" \
 	-c 'fsync' $SCRATCH_MNT/file >> $seqres.full
 rm -f $sentryfile
+wait
 
 cat "$tracefile" >> $seqres.full
 grep -q iomap_invalid "$tracefile"
Signature · 1 line
2.45.2
#51 lore raw

PATCH 33/40 vfstests: some tests require the testdir to be shared

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

This ensures that these tests will run successfully when the parallel check infrastructure makes all the scratch and test mounts private.

---
 tests/generic/589 | 2 ++
 tests/generic/632 | 2 ++
 2 files changed, 4 insertions(+)
tests/generic/589 +2 -0
index b5860070b..0384083bb 100755
@@ -164,7 +164,9 @@ move_test()
 	move_run unbindable private
 }
 
+_mount --make-shared $TEST_DIR
 move_test
+_mount --make-private $TEST_DIR
 
 # success, all done
 status=0
tests/generic/632 +2 -0
index c7a1f1fad..c4d3555bc 100755
@@ -22,8 +22,10 @@ _begin_fstest auto quick mount
 _require_test
 _require_test_program "detached_mounts_propagation"
 
+_mount --make-shared $TEST_DIR
 $here/src/detached_mounts_propagation $TEST_DIR >> $seqres.full
 status=$?
+_mount --make-private $TEST_DIR
 
 echo silence is golden
 exit
Signature · 1 line
2.45.2
#52 lore raw

PATCH 34/40 xfs/629: single extent files should be within tolerance

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

The test passes if we have between 2 and 40 extents (despite what the comment says!), with the target being 20. There is absolutely no reason for considering a single extent file a failure - that indicates the filesystem completely defeated the fragmentation behaviour the test was trying to cause. Hence expand the range of "test pass" tolerance to 1-41 extents.

---
 tests/xfs/629 | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tests/xfs/629 +2 -2
index 34b3ca0d6..a2f345571 100755
@@ -59,8 +59,8 @@ _scratch_sync
 
 for ((n=0; n<nfiles; n++)); do
 	count=$(_count_extents $workfile.$n)
-	# Acceptible extent count range is 1-40
-	_within_tolerance "file.$n extent count" $count 21 19 -v
+	# Acceptible extent count range is 1-41
+	_within_tolerance "file.$n extent count" $count 21 20 -v
 done
 
 status=0
Signature · 1 line
2.45.2
#53 lore raw

PATCH 35/40 xfs/076: fix broken mkfs filtering

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

The test does not do what it says on the packet - the mkfs output is not actually passed to the mkfs filter, so it doesn't know what inode size mkfs actually used. Hence CHUNK_SIZE ends up being calculated as 0, and that means it enters an endless loop because offset never decreases.

Fix it by adding the missing line continuation.

---
 tests/xfs/076 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/xfs/076 +1 -1
index e315a067c..2ea6459b0 100755
@@ -68,7 +68,7 @@ _require_xfs_sparse_inodes
 # bitmap consuming all the free space in our small data device.
 unset SCRATCH_RTDEV
 
-_scratch_mkfs "-d size=96m -m crc=1 -i sparse" | tee -a $seqres.full |
+_scratch_mkfs "-d size=96m -m crc=1 -i sparse" | tee -a $seqres.full | \
 	_filter_mkfs > /dev/null 2> $tmp.mkfs
 . $tmp.mkfs	# for isize
 
Signature · 1 line
2.45.2
#54 lore raw

PATCH 36/40 fstests: capture some failures to seqres.full

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Whilst trying to debug test failures, I found a few places where errors needed to be redirected to $seqres.full rather than /dev/null. This is a collection of all the conversions that haven't been captured by some other bug fix patch.

Note that calling _check_filesystems() after removing the require_test/scratch files means it is a no-op, so I removed that call at the same time as capturing unmount failures after the test has failed.

---
 check             | 5 ++---
 tests/generic/049 | 2 +-
 tests/generic/603 | 6 +++---
 tests/xfs/495     | 2 +-
 4 files changed, 7 insertions(+), 8 deletions(-)
check +2 -3
index 9222cd7e4..8131f4e2e 100755
@@ -1004,14 +1004,13 @@ function run_section()
 
 		if [ $sts -ne 0 ]; then
 			_dump_err_cont "[failed, exit status $sts]"
-			_test_unmount 2> /dev/null
-			_scratch_unmount 2> /dev/null
+			_test_unmount 2>> $seqres.full
+			_scratch_unmount 2>> $seqres.full
 			rm -f ${RESULT_DIR}/require_test*
 			rm -f ${RESULT_DIR}/require_scratch*
 			# Even though we failed, there may be something interesting in
 			# dmesg which can help debugging.
 			_check_dmesg
-			(_adjust_oom_score 250; _check_filesystems)
 			tc_status="fail"
 		else
 			# The test apparently passed, so check for corruption
tests/generic/049 +1 -1
index 94b5afe96..fcfbd48ec 100755
@@ -56,7 +56,7 @@ i=1;
 while [ $i -lt 1000 ]
 do
 	file=$SCRATCH_MNT/$i
-	$XFS_IO_PROG -f -c "pwrite -b 32k -S 0xff 0 32k" $file > /dev/null
+	$XFS_IO_PROG -f -c "pwrite -b 32k -S 0xff 0 32k" $file >> $seqres.full
 	if [ $? -ne 0 ]
 	then
 		echo error creating/writing file $file
tests/generic/603 +3 -3
index a84695831..2a75cf9e0 100755
@@ -77,7 +77,7 @@ test_grace()
 		_filter_xfs_io_error | tee -a $seqres.full
 	repquota -v -$type $SCRATCH_MNT | grep -v "^root" >>$seqres.full 2>&1
 	# Reset grace time here, make below grace time test more accurate
-	setquota -$type $qa_user -T $bgrace $igrace $SCRATCH_MNT 2>/dev/null
+	setquota -$type $qa_user -T $bgrace $igrace $SCRATCH_MNT >>$seqres.full 2>&1
 	# Now sleep enough grace time and check that softlimit got enforced
 	sleep $((bgrace + 1))
 	echo "Try to write 1 one more block after grace..."
@@ -89,13 +89,13 @@ test_grace()
 	# And now the softlimit test for inodes
 	# First reset space limits so that we don't have problems with
 	# space reservations on XFS
-	setquota -$type $qa_user 0 0 3 100 $SCRATCH_MNT
+	setquota -$type $qa_user 0 0 3 100 $SCRATCH_MNT 
 	echo "Create 2 more files, over the inode softlimit..."
 	su $qa_user -c "touch $dir/file3 $dir/file4" 2>&1 >>$seqres.full | \
 		_filter_scratch | tee -a $seqres.full
 	repquota -v -$type $SCRATCH_MNT  | grep -v "^root" >>$seqres.full 2>&1
 	# Reset grace time here, make below grace time test more accurate
-	setquota -$type $qa_user -T $bgrace $igrace $SCRATCH_MNT 2>/dev/null
+	setquota -$type $qa_user -T $bgrace $igrace $SCRATCH_MNT >>$seqres.full 2>&1
 	# Wait and check grace time enforcement
 	sleep $((igrace+1))
 	echo "Try to create one more inode after grace..."
tests/xfs/495 +1 -1
index b42e7e5b3..ce724f381 100755
@@ -24,7 +24,7 @@ _require_populate_commands
 _require_xfs_db_command "fuzz"
 
 echo "Format and populate"
-_scratch_populate_cached nofill > $seqres.full 2>&1
+_scratch_populate_cached nofill >> $seqres.full 2>&1
 
 filter_nbrepair() {
 	grep rebuilding | sed -e 's/directory inode [0-9]*/directory inode XXX/g'
Signature · 1 line
2.45.2
#55 lore raw

PATCH 37/40 fstests: always use fail-at-unmount semantics for XFS

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Rather than require every test that tests unmount in failure conditions have to set up fail-at-unmount semantics for the underlying filesystem, use these semantics for all test and scratch device mounts.

This currently only affects XFS filesystems, and helps prevent unexpected unmount hangs in EIO tests because metadata writes are configured to try forever by default.

---
 common/rc | 2 ++
 1 file changed, 2 insertions(+)
common/rc +2 -0
index 3b4b9a59f..7f4a71bd3 100644
@@ -410,6 +410,7 @@ _try_scratch_mount()
 	mount_ret=$?
 	[ $mount_ret -ne 0 ] && return $mount_ret
 	_idmapped_mount $SCRATCH_DEV $SCRATCH_MNT
+	_prepare_for_eio_shutdown $SCRATCH_DEV
 }
 
 # mount scratch device with given options and _fail if mount fails
@@ -609,6 +610,7 @@ _test_mount()
     mount_ret=$?
     [ $mount_ret -ne 0 ] && return $mount_ret
     _idmapped_mount $TEST_DEV $TEST_DIR
+    _prepare_for_eio_shutdown $TEST_DEV
 }
 
 _test_unmount()
Signature · 1 line
2.45.2
#56 lore raw

PATCH 38/40 generic/062: don't leave debug files in $here on failure

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Move them to $seqres.<debug-file> so they don't pollute the source tree that fstests is running from.

---
 tests/generic/062 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/062 +1 -1
index f0904992d..896590403 100755
@@ -196,7 +196,7 @@ echo "*** compare before and after backups"
 diff $tmp.backup1 $tmp.backup2
 if [ $? -ne 0 ]; then
 	echo "urk, failed - creating $seq.backup1 and $seq.backup2"
-	cp $tmp.backup1 $seq.backup1 && cp $tmp.backup2 $seq.backup2
+	cp $tmp.backup1 $seqres.backup1 && cp $tmp.backup2 $seqres.backup2
 	status=1
 	exit
 fi
Signature · 1 line
2.45.2
#57 lore raw

PATCH 39/40 fstests: quota grace periods unreliable under load

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Starting the quota grace period doesn't necessary happen predictably when the system is under heavy load. This results in random test failures where grace period timeouts are expected.

---
 tests/generic/601 | 7 ++++---
 tests/xfs/050     | 5 +++++
 2 files changed, 9 insertions(+), 3 deletions(-)
tests/generic/601 +4 -3
index 9860505ba..78b6a4aaa 100755
@@ -52,9 +52,10 @@ $XFS_QUOTA_PROG -x -c "timer -u -i 100 $qa_user" $SCRATCH_MNT
 # raw ("since epoch") grace expiry
 get=`repquota -up $SCRATCH_MNT | grep  "^$qa_user" | awk '{print $NF}'`
 
-# Either the new expiry must match; or be one second after the set time, to
-# deal with the seconds counter incrementing.
-if [ "$get" != "$set" ] && [ "$get" -ne "$((set + 1))" ]; then
+# Either the new expiry must match or be soon after the set time as there
+# may be load related delays on setting the new timer. A 10s "grace" window
+# should be enough for everyone.
+if [ "$get" != "$set" ] && [ "$get" -gt "$((set + 10))" ]; then
 	echo "set grace to $set but got grace $get"
 fi
 
tests/xfs/050 +5 -0
index 7baaaeaa3..78303bf78 100755
@@ -115,6 +115,11 @@ _exercise()
 	echo
 	echo "*** push past the soft block limit" | tee -a $seqres.full
 	_file_as_id $SCRATCH_MNT/softie $id $type $bsize 300
+
+	# sync so that the data is actually allocated as soft limit timers
+	# are not started until block allocation is done. This means the test
+	# will work even if remount sliently fails due to busy mounts
+	_scratch_sync
 	_qmount
 	$XFS_QUOTA_PROG -D $tmp.projects -P $tmp.projid -x \
 		-c "repquota -birnN -$type" $SCRATCH_DEV |
Signature · 1 line
2.45.2
#58 lore raw

PATCH 40/40 fstests: check-parallel

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Runs tests in parallel runner threads. Each runner thread has it's own set of tests to run, and runs a separate instance of check to run those tests.

check-parallel sets up loop devices, mount points, results directories, etc for each instance and divides the tests up between the runner threads.

It currently hard codes the XFS and generic test lists, and then gives each check invocation an explicit list of tests to run. It also passes through exclusions so that test exclude filtering is still done by check.

This is far from ideal, but I didn't want to have to embark on a major refactoring of check to be able to run stuff in parallel. It was quite the challenge just to get all the tests and test infrastructure up to the point where they can run reliably in parallel.

Hence I've left the actual factoring of test selection and setup out of the patchset for the moment. The plan is to factor both the test setup and the test list runner loop out of check and share them between check and check-parallel, hence not requiring check-parallel to run check directly. That is future work, however.

With the current test runner setup, it is not uncommon to see >5000% cpu usage, 150-200kiops and 4-5GB/s of disk bandwidth being used when running 64 runners. This is a serious stress load as it is constantly mounting and unmounting dozens of filesystems, creating and destroying devices, dropping caches, running sync, running CPU hot plug, running page cache migration, etc.

The massive amount of IO that load generates causes qemu hosts to abort (i.e. crash) because they run out of vm map segments. Hence bumping up the max_map_count on the host like so:

echo 1048576 > /proc/sys/vm/max_map_count

is necessary.

There is no significant memory pressure to speak of from running the tests like this. I've seen a maximum of about 50GB of RAM used when running tests like this, so running on a 64p/64GB VM the additional concurrency doesn't really stress memory capacity like it does CPU and IO.

All the runners are executed in private mount namespaces. This is to prevent ephemeral mount namespace clones from taking a reference to every mounted filesystem in the machine and so causing random "device busy after unmount" failures in the tests that are running concurrently with the mount namespace setup and teardown.

A typical pstree -N mnt looks like:

$ pstree -N mnt
[4026531841]
bash
bash───pstree
[0]
sudo───sudo───check-parallel─┬─check-parallel───nsexec───check───311─┬─cut
                             │                                       └─md5sum
                             ├─check-parallel───nsexec───check───750─┬─750───sleep
                             │                                       └─750.fsstress───4*[750.fsstress───{750.fsstress}]
                             ├─check-parallel───nsexec───check───013───013───sed
                             ├─check-parallel───nsexec───check───251───cp
                             ├─check-parallel───nsexec───check───467───open_by_handle
                             ├─check-parallel───nsexec───check───650─┬─650───sleep
                             │                                       └─650.fsstress─┬─61*[650.fsstress───{650.fsstress}]
                             │                                                      └─2*[650.fsstress]
                             ├─check-parallel───nsexec───check───707
                             ├─check-parallel───nsexec───check───705
                             ├─check-parallel───nsexec───check───416
                             ├─check-parallel───nsexec───check───477───2*[open_by_handle]
                             ├─check-parallel───nsexec───check───140───140
                             ├─check-parallel───nsexec───check───562
                             ├─check-parallel───nsexec───check───415───xfs_io───{xfs_io}
                             ├─check-parallel───nsexec───check───291
                             ├─check-parallel───nsexec───check───017
                             ├─check-parallel───nsexec───check───016
                             ├─check-parallel───nsexec───check───168───2*[168───168]
                             ├─check-parallel───nsexec───check───672───2*[672───672]
                             ├─check-parallel───nsexec───check───170─┬─170───170───170
                             │                                       └─170───170
                             ├─check-parallel───nsexec───check───531───122*[t_open_tmpfiles]
                             ├─check-parallel───nsexec───check───387
                             ├─check-parallel───nsexec───check───748
                             ├─check-parallel───nsexec───check───388─┬─388.fsstress───4*[388.fsstress───{388.fsstress}]
                             │                                       └─sleep
                             ├─check-parallel───nsexec───check───328───328
                             ├─check-parallel───nsexec───check───352
                             ├─check-parallel───nsexec───check───042
                             ├─check-parallel───nsexec───check───426───open_by_handle
                             ├─check-parallel───nsexec───check───756───2*[open_by_handle]
                             ├─check-parallel───nsexec───check───227
                             ├─check-parallel───nsexec───check───208───aio-dio-invalid───2*[aio-dio-invalid]
                             ├─check-parallel───nsexec───check───746───cp
                             ├─check-parallel───nsexec───check───187───187
                             ├─check-parallel───nsexec───check───027───8*[027]
                             ├─check-parallel───nsexec───check───045───xfs_io───{xfs_io}
                             ├─check-parallel───nsexec───check───044
                             ├─check-parallel───nsexec───check───204
                             ├─check-parallel───nsexec───check───186───186
                             ├─check-parallel───nsexec───check───449
                             ├─check-parallel───nsexec───check───231───su───fsx
                             ├─check-parallel───nsexec───check───509
                             ├─check-parallel───nsexec───check───127───5*[127───fsx]
                             ├─check-parallel───nsexec───check───047
                             ├─check-parallel───nsexec───check───043
                             ├─check-parallel───nsexec───check───475───pkill
                             ├─check-parallel───nsexec───check───299─┬─fio─┬─4*[fio]
                             │                                       │     ├─2*[fio───4*[{fio}]]
                             │                                       │     └─{fio}
                             │                                       └─pgrep
                             ├─check-parallel───nsexec───check───551───aio-dio-write-v
                             ├─check-parallel───nsexec───check───323───aio-last-ref-he───100*[{aio-last-ref-he}]
                             ├─check-parallel───nsexec───check───648───sleep
                             ├─check-parallel───nsexec───check───046
                             ├─check-parallel───nsexec───check───753─┬─753.fsstress───4*[753.fsstress]
                             │                                       └─pkill
                             ├─check-parallel───nsexec───check───507───507
                             ├─check-parallel───nsexec───check───629─┬─3*[629───xfs_io───{xfs_io}]
                             │                                       └─5*[629]
                             ├─check-parallel───nsexec───check───073───umount
                             ├─check-parallel───nsexec───check───615───615
                             ├─check-parallel───nsexec───check───176───punch-alternati
                             ├─check-parallel───nsexec───check───294
                             ├─check-parallel───nsexec───check───236───236
                             ├─check-parallel───nsexec───check───165─┬─165─┬─165─┬─cut
                             │                                       │     │     └─xfs_io───{xfs_io}
                             │                                       │     └─165───grep
                             │                                       └─165
                             ├─check-parallel───nsexec───check───259───sync
                             ├─check-parallel───nsexec───check───442───442.fsstress───4*[442.fsstress───{442.fsstress}]
                             ├─check-parallel───nsexec───check───558───255*[558]
                             ├─check-parallel───nsexec───check───358───358───358
                             ├─check-parallel───nsexec───check───169───169
                             └─check-parallel───nsexec───check───297─┬─297.fsstress─┬─284*[297.fsstress───{297.fsstress}]
                                                                     │              └─716*[297.fsstress]
                                                                     └─sleep

A typical test run looks like:

$ time sudo ./check-parallel /mnt/xfs -s xfs -x dump
Runner 63 Failures:  xfs/170
Runner 36 Failures:  xfs/050
Runner 30 Failures:  xfs/273
Runner 29 Failures:  generic/135
Runner 25 Failures:  generic/603
Tests run: 1140
Failure count: 5

Ten slowest tests - runtime in seconds:
xfs/013 454
generic/707 414
generic/017 398
generic/387 395
generic/748 390
xfs/140 351
generic/562 351
generic/705 347
generic/251 344
xfs/016 343

Cleanup on Aisle 5?

total 0
crw-------. 1 root root 10, 236 Nov 27 09:27 control
lrwxrwxrwx. 1 root root 7 Nov 27 09:27 fast -> ../dm-0
/dev/mapper/fast 1.4T 192G 1.2T 14% /mnt/xfs

real    9m29.056s
user    0m0.005s
sys     0m0.022s

$

Yeah, that runtime is real - under 10 minutes for a full XFS auto group test run. When running this normally (i.e. via check) on this machine, it usually takes just under 4 hours to run the same set of tests. i.e. I can run ./check-parallel roughly 25x times on this machine in the same time it takes to run ./check.

---
 check          |   7 +-
 check-parallel | 205 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 208 insertions(+), 4 deletions(-)
 create mode 100755 check-parallel
check +3 -4
index 8131f4e2e..607d2456e 100755
@@ -33,7 +33,7 @@ exclude_tests=()
 _err_msg=""
 
 # start the initialisation work now
-iam=check
+iam=check.$$
 
 # mkfs.xfs uses the presence of both of these variables to enable formerly
 # supported tiny filesystem configurations that fstests use for fuzz testing
@@ -460,7 +460,7 @@ fi
 
 _wrapup()
 {
-	seq="check"
+	seq="check.$$"
 	check="$RESULT_BASE/check"
 	$interrupt && sect_stop=`_wallclock`
 
@@ -552,7 +552,6 @@ _wrapup()
 
 	sum_bad=`expr $sum_bad + ${#bad[*]}`
 	_wipe_counters
-	rm -f /tmp/*.rawout /tmp/*.out /tmp/*.err /tmp/*.time
 	if ! $OPTIONS_HAVE_SECTIONS; then
 		rm -f $tmp.*
 	fi
@@ -808,7 +807,7 @@ function run_section()
 
 	init_rc
 
-	seq="check"
+	seq="check.$$"
 	check="$RESULT_BASE/check"
 
 	# don't leave old full output behind on a clean run
check-parallel new file +205 -0

205 lines not on this page · raw message

Signature · 1 line
2.45.2
#59 lore raw
In reply to Dave Chinner
Quoted 7 lines · Dave Chinner

Hi folks,

This patchset introduces the ability to run fstests concurrently instead of serially as the current check script does. A git branch containing this patchset can be pulled from here:

https://git.kernel.org/pub/scm/linux/kernel/git/dgc/xfstests-dev.git check-parallel

Hi Dave,

I've merged your "check-parallel" branch, and rebase on fstests' patches-in-queue branch (which is nearly the next release). I just pushed a new branch "for-dave-check-parallel" which fixed all conflicts. It'll be "next next" release, feel free to update base on that. I'll test that branch too :)

Thanks,
Zorro

Quoted 774 lines

The motivation for this is the ever growing runtime of fstests as more tests are added, along with the extremely low resource usage of individual tests. This means that a typical machine running fstests is under-utilised and a single auto group test set execution takes hours.

Yes, I know that I could scale out testing by running lots of little VMs at once, and I already do that. The problem is that I don't get back a complete auto test run result for hours. Want to check that a one-line fix has not caused any regressions? That's, at minimum, an overnight wait for the test farm to crunch through a dozen configs.

On my 64p/128GB RAM VM, 'check -g auto -s xfs -x dump' typically takes around 230-240 minutes to run. With this patchset applied, it runs the same set of tests in 8-10 minutes. I can run ~25 complete auto group test sets with check-parallel in the same time it takes check to run one.

IOWs, I can have the most common config complete a full regression test run as fast as I can turn around a new kernel with a new change to test. I have CPU, memory and IO to burn in my test machines, but what I lack is instant feedback for the change I just made. check-parallel is fast enough that it gives me pretty much instant feedback....

Most of this patchset is preparing infrastructure for concurrent test execution and fixing bugs in tests that I've found whilst getting concurrent execution working reliably. The infrastructure changes center around:

- getting rid of killall - there's nothing quite like one test killing the processes of 15 other tests at the same time...
- cleaning up background process instantiation and reaping, which is a lot more important when an interrupt needs to kill hundreds of processes instead of just a couple.
- isolating error reporting (e.g. dmesg filtering) so that one test failure doesn't trigger lots of other false failure detections
- sanitising the use of loopback devices
- avoiding the use of fixed device names - multiple tests need to use dm-flakey, dm-error, etc devices at the same time, so each test needs a unique device name
- marking tests that are unreliable when outside perturbations occur. These include tests that expect to find delalloc extents, write specific offset patterns in memory then sync them to create specific layouts, etc. e.g. If another test runs sync(1), then those write patterns no longer produce the expected output.
- taming tests that weren't designed for high CPU count machines
- replacing udevadm settle calls because udev is -never- idle when there are tens to hundreds of block devices and filesystems being rapidly set up and torn down.
- converting sync(1) and sync(2) to syncfs(2) to avoid ihaving hundreds of concurrent superblock list traversals lock-stepping with multiple mount/unmounts every second.

There are lots of little other things, but those are the main test and test infrastructure changes. Some of these are big - the fsstress execution rework touches 105 files, but it now means that every single fsstress execution in fstests is controlled by 4 helper functions:

_run_fsstress() - run fsstress synchronously
_run_fsstress_bg - run fsstress in background
_wait_for_fsstress - wait for background fsstress
_kill_fsstress - kill+wait for background fsstress

The test infrastructure also automatically handles cleanup of fsstress processes when the test is interrupted, so tests using fsstress don't need a custom _cleanup() function just to call _kill_fsstress(). This is something that should have been done a long time ago, but now it is critical for being able to manage multiple independent concurrent fsstress invocations sanely.

There are some tests that just can't be run reliably in a concurrent environment - if there is outside interference in, say, page cache flushing then the tests fail. These tests have been added to the "unreliable_in_parallel" test group with a comment explaining why they are unreliable. The check-parallel script automatically excludes this test group.

The only remaining set of tests that are somewhat flakey are the tests that exercise quotas. Quota tests randomly fail for various reasons. Sometimes they don't detect EDQUOT conditions. Sometimes repquota emits weird "device not found" errors. Sometimes grace periods don't start, sometimes they don't time out, or time out and then don't trigger EDQUOT. I don't know why these weird things are happening yet, and until I properly integrate test group selection into check-parallel I can't really isolate the quota tests to focus on them alone.

There are several patches that speed up test runtime. There were several tests that were taking 12-15 minutes to run each, and these made up >95% of the entire check-parallel runtime. In general, these tests have been made to run with more concurrency to speed them up. the result is that the longest individual test runtime has dropped to around 7 minutes, and the elapsed runtime for check-parallel has dropped to 8-10 minutes.

Hence there are 39 patches that are doing prep work on tests and infrastructure to make tests run reliabily in concurrent test contexts. The last patch is the check-parallel runner script that runs the tests concurrently.

The check-parallel script is still very rudimentary. I hard coded the tests it runs (generic+xfs auto tests) and the concurrency so that I could run explicit sets of tests with './check --exact-order <list>'. It is pointed at a mounted directory, and it creates all the infrastructure it needs to run the tests within that directory.

This enabled me to break up the tests across a set of identical runner process contexts. Each runner does:

- create runner directory
- create test and scratch image files
- create loopback devices for test and scratch devices
- sets up results directories
- execute check in it's own private mount namespace so it can't see any of the mounts that other runners are using.
- tears down loopback devices
- reports test failures.

If you run with the same directory over and over again, then it reuses the same runner infrastructure and test and scratch image files. The results directories are not overwritten as they are date-stamped, hence using the same mount point automatically creates a result archive for later data mining.

A typical target directory (/mnt/xfs) looks like:

/mnt/xfs/runner-0/...
/mnt/xfs/runner-1/...
....
/mnt/xfs/runner-63/...

And each runner directory:

log
results-2024-11-19-11:22:25/...
results-2024-11-19-12:36:28/...
.....
results-2024-11-27-13:32:42/...
scratch/
scratch.img
test/
test.img

The log file is the check output for that runner and should look familiar:

SECTION       -- xfs
FSTYP         -- xfs (debug)
PLATFORM      -- Linux/x86_64 test1 6.12.0-dgc+ #297 SMP PREEMPT_DYNAMIC Wed Nov 27 08:13:06 AEDT 2024
MKFS_OPTIONS  -- -f -m rmapbt=1 /dev/loop10

MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/loop10 /mnt/xfs/runner-10/scratch

generic/387        309s
generic/551        49s
generic/289        2s
xfs/274        3s
generic/677        3s
generic/143        5s
generic/304        1s
generic/390        4s
generic/427        4s
xfs/103        1s
generic/252        3s
xfs/045        1s
generic/374        2s
generic/002        1s
generic/534        1s
generic/039        1s

generic/595 [not run] No encryption support for xfs
xfs/122 [not run] Could not compile test program (see end of /mnt/xfs/runner-10/results-2024-11-27-13:32:42/xfs/xfs/122.full)
xfs/556 [not run] xfs_scrub not found
Ran: generic/387 generic/551 generic/289 xfs/274 generic/677 generic/143 generic/304 generic/390 generic/427 xfs/103 generic/252 xfs/045 generic/374 generic/002 generic/534 generic/039 generic/595 xfs/122 xfs/556 Not run: generic/595 xfs/122 xfs/556
Passed all 19 tests

SECTION -- xfs
=========================
Ran: generic/387 generic/551 generic/289 xfs/274 generic/677 generic/143 generic/304 generic/390 generic/427 xfs/103 generic/252 xfs/045 generic/374 generic/002 generic/534 generic/039 generic/595 xfs/122 xfs/556 Not run: generic/595 xfs/122 xfs/556
Passed all 19 tests

Doing something with all the log files from a run can be done with "/mnt/xfs/*/log". e.g. grep, vi, etc.

The results directory contains the same check.{full,log,time} and test results directories as per a normal check invocation, so there's little difference in checking/analysing results with a parallel execution run.

Because the tests run in private mount namespaces, it's easy to see what check-parallel is running at any point in time using 'pstree -N mnt'. Here's a run that is hung on an unmount not completing:

$ pstree -N mnt
[4026531841]
bash
bash───pstree
[0]
sudo───sudo───check-parallel─┬─check-parallel───nsexec───check───311───fsync-tester
                             ├─check-parallel───nsexec───check───467───open_by_handle
                             ├─check-parallel───nsexec───check───338
                             ├─check-parallel───nsexec───check───421
                             ├─check-parallel───nsexec───check───441
                             ├─check-parallel───nsexec───check───232
                             ├─check-parallel───nsexec───check───477───open_by_handle
                             ├─check-parallel───nsexec───check───420
                             ├─check-parallel───nsexec───check───426───open_by_handle
                             ├─check-parallel───nsexec───check───756───open_by_handle
                             ├─check-parallel───nsexec───check───231
                             ├─check-parallel───nsexec───check───475───475.fsstress───475.fsstress───{475.fsstress}
                             ├─check-parallel───nsexec───check───388───388.fsstress───388.fsstress───{388.fsstress}
                             ├─check-parallel───nsexec───check───259───sync
                             ├─check-parallel───nsexec───check───622───sync
                             ├─check-parallel───nsexec───check───318───sync
                             ├─check-parallel───nsexec───check───753───umount
                             ├─check-parallel───nsexec───check───086
                             ├─check-parallel───nsexec───check───648───648.fsstress───648.fsstress───{648.fsstress}
                             ├─check-parallel───nsexec───check───391
                             ├─check-parallel───nsexec───check───315───sync
                             └─check-parallel───nsexec───check───183───bulkstat_unlink

All the other processes are in sync or dropping caches and stuck waiting for the superblock s_umount lock that the unmount holds. Finding where it is stuck:

$ pgrep [u]mount
1081885
$ sudo cat /proc/1081885/stack
[<0>] xfs_ail_push_all_sync+0x9c/0xf0
[<0>] xfs_unmount_flush_inodes+0x41/0x70
[<0>] xfs_unmountfs+0x59/0x190
[<0>] xfs_fs_put_super+0x3b/0x90
[<0>] generic_shutdown_super+0x77/0x160
[<0>] kill_block_super+0x1b/0x40
[<0>] xfs_kill_sb+0x12/0x30
[<0>] deactivate_locked_super+0x38/0x100
[<0>] deactivate_super+0x41/0x50
[<0>] cleanup_mnt+0x9f/0x160
[<0>] __cleanup_mnt+0x12/0x20
[<0>] task_work_run+0x89/0xb0
[<0>] resume_user_mode_work+0x4f/0x60
[<0>] syscall_exit_to_user_mode+0x76/0xb0
[<0>] do_syscall_64+0x74/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
$

Yup, known bug - a shutdown vs XFS_ISTALE inode cluster freeing issue that leaves pinned, stale inodes in the AIL.

The point I'm making here is that running tests concurrently doesn't change anything material in how you'd go about discovering and diagnosing failures. The only difference is in what that initial failure might look like. e.g. Failures that result in dmesg output will cause a huge number of tests to all fail with "check dmesg for failure output" reports. Hence there is a bit of sifting to find which test triggered the dmesg output, but failure detection still works just fine.

The check-parallel script is really only at proof-of-concept stage. It is sufficient to run tests in parallel, but that's about it. most of the work so far has gone into making the generic and XFS tests run reliably in parallel. I have not run any other tests, and haven't done full conversions of other test directories or C code in src/. e.g. the sync -> syncfs was only done for the common dir and the generic and xfs tests dirs

I do not plan to do these conversions for ext4/btrfs/overlay/etc any time soon as these conversions are mostly about improving execution time, not test correctness. Hence they are not a priority for me - the priority is further developing the concurrent execution environment. i.e. check-parallel.

To that end, I need to factor all the test selection and exclusion code out of check, and do the same with the actual test list runner loop. That way I can reuse all the existing code from within the check-parallel context rather than having to call check to do all of that work itself. I would like to get check-parallel to the point where it is mostly just s/check/check-parallel/ on the command line to move from serial to concurrent test execution.

I also want to try to integrate the config section stuff into check-parallel. This is more a case of defining what devices the config needs to create (i.e. as loop devices) rather than what devices it should be using. I think I can do this just by defining a different set of environment variables (e.g. NEED_SCRATCHDEV, NEED_LOGDEV, etc) and triggering the loop device creation from these variables.

In a way, the fact that check-parallel bootstraps it's own runtime environment almost makes it entirely zero-config. As it stands right now, you should be able to pull this patchset, create your base test directory (make sure you have at least 100GB of free disk space), run './check-parallel <test_dir> -x dump' and it Should Just Work.

At this point, the test infrastructure problems are largely solved. My focus is now on further development of the check-parallel script and integrating it tightly into the existing check infrastructure rather than open-coding test lists and configuration information. This will probably take a bit of time, so I'd like to get the bug fixes, improvements and infrastructure changes underway so I'm not left carrying a huge patchset for months....

----------------------------------------------------------------
Dave Chinner (40):
xfs/448: get rid of assert-on-failure
fstests: cleanup fsstress process management
fuzzy: don't use killall
fstests: per-test dmflakey instances
fstests: per-test dmerror instances
fstests: per-test dmhuge instances
fstests: per-test dmthin instances
fstests: per-test dmdust instances
fstests: per-test dmdelay instances
fstests: fix DM device creation/removal vs udev races
fstests: use syncfs rather than sync
fstests: clean up mount and unmount operations
fstests: clean up loop device instantiation
fstests: xfs/227 is really slow
fstests: mark tests that are unreliable when run in parallel
fstests: use udevadm wait in preference to settle xfs/442: rescale load so it's not exponential xfs/176: fix broken setup code
xfs/177: remove unused slab object count location checks
fstests: remove uses of killall where possible generic/127: reduce runtime
quota: system project quota files need to be shared
dmesg: reduce noise from other tests
fstests: stop using /tmp directly
fstests: scale some tests for high CPU count sanity generic/310: cleanup killing background processes
filter: handle mount errors from CONFIG_BLK_DEV_WRITE_MOUNTED=y
filters: add a filter that accepts EIO instead of other errors generic/085: general cleanup for reliability and debugging
fstests: don't use directory stacks
fstests: clean up a couple of dm-flakey tests
fstests: clean up termination of various tests
vfstests: some tests require the testdir to be shared xfs/629: single extent files should be within tolerance xfs/076: fix broken mkfs filtering
fstests: capture some failures to seqres.full
fstests: always use fail-at-unmount semantics for XFS generic/062: don't leave debug files in $here on failure
fstests: quota grace periods unreliable under load
fstests: check-parallel

 check                                         |   12 -
 check-parallel                                |  205 ++++++++++++++++++
 common/btrfs                                  |    4
 common/config                                 |   36 ++-
 common/dmdelay                                |   24 +-
 common/dmdust                                 |   14 -
 common/dmerror                                |   74 +++---
 common/dmflakey                               |   60 ++---
 common/dmhugedisk                             |   21 +
 common/dmlogwrites                            |    4
 common/dmthin                                 |   12 -
 common/encrypt                                |    2
 common/filter                                 |   17 +
 common/fuzzy                                  |   37 +--
 common/log                                    |    2
 common/metadump                               |   32 +-
 common/overlay                                |   10
 common/populate                               |    8
 common/preamble                               |    1
 common/quota                                  |   37 ---
 common/rc                                     |  166 +++++++++++---
 common/repair                                 |    2
 common/report                                 |    2
 common/verity                                 |    2
 common/xfs                                    |    2
 doc/group-names.txt                           |    1
 doc/requirement-checking.txt                  |    6
 ltp/fsstress.c                                |   28 ++
 src/aio-dio-regress/aio-last-ref-held-by-io.c |    5
 src/dmerror                                   |    6
 tests/btrfs/004                               |   11
 tests/btrfs/007                               |    3
 tests/btrfs/012                               |    4
 tests/btrfs/028                               |    6
 tests/btrfs/049                               |    4
 tests/btrfs/057                               |    4
 tests/btrfs/060                               |   14 -
 tests/btrfs/061                               |   13 -
 tests/btrfs/062                               |   13 -
 tests/btrfs/063                               |   13 -
 tests/btrfs/064                               |   13 -
 tests/btrfs/065                               |   14 -
 tests/btrfs/066                               |   14 -
 tests/btrfs/067                               |   14 -
 tests/btrfs/068                               |   14 -
 tests/btrfs/069                               |   13 -
 tests/btrfs/070                               |   13 -
 tests/btrfs/071                               |   13 -
 tests/btrfs/072                               |   14 -
 tests/btrfs/073                               |   13 -
 tests/btrfs/074                               |   13 -
 tests/btrfs/078                               |   12 -
 tests/btrfs/100                               |    4
 tests/btrfs/101                               |    4
 tests/btrfs/136                               |    6
 tests/btrfs/160                               |    3
 tests/btrfs/192                               |   12 -
 tests/btrfs/195                               |    2
 tests/btrfs/212                               |   16 -
 tests/btrfs/232                               |    4
 tests/btrfs/252                               |    5
 tests/btrfs/261                               |    2
 tests/btrfs/284                               |    4
 tests/btrfs/286                               |    2
 tests/btrfs/291                               |    5
 tests/btrfs/320                               |    6
 tests/btrfs/332                               |    4
 tests/ext4/004                                |    4
 tests/ext4/057                                |   14 -
 tests/ext4/058                                |    3
 tests/ext4/307                                |    4
 tests/generic/013                             |   22 -
 tests/generic/015                             |    2
 tests/generic/019                             |   13 -
 tests/generic/029                             |    2
 tests/generic/030                             |    2
 tests/generic/032                             |    2
 tests/generic/034                             |    2
 tests/generic/039                             |    2
 tests/generic/040                             |    2
 tests/generic/041                             |    2
 tests/generic/042                             |    4
 tests/generic/048                             |    2
 tests/generic/049                             |    4
 tests/generic/050                             |    3
 tests/generic/051                             |   24 --
 tests/generic/054                             |    2
 tests/generic/055                             |    6
 tests/generic/057                             |    2
 tests/generic/059                             |    2
 tests/generic/062                             |    2
 tests/generic/065                             |    2
 tests/generic/066                             |    5
 tests/generic/067                             |   17 -
 tests/generic/068                             |    7
 tests/generic/070                             |   12 -
 tests/generic/073                             |    2
 tests/generic/076                             |   10
 tests/generic/076.out                         |    1
 tests/generic/081                             |    8
 tests/generic/083                             |   13 -
 tests/generic/083.out                         |    1
 tests/generic/084                             |   12 -
 tests/generic/085                             |   21 +
 tests/generic/090                             |    4
 tests/generic/092                             |    2
 tests/generic/098                             |    4
 tests/generic/099                             |    8
 tests/generic/101                             |    2
 tests/generic/104                             |    2
 tests/generic/106                             |    2
 tests/generic/107                             |    2
 tests/generic/108                             |    9
 tests/generic/109                             |    5
 tests/generic/117                             |    4
 tests/generic/127                             |   67 +++--
 tests/generic/127.out                         |    6
 tests/generic/135                             |   24 --
 tests/generic/150                             |    2
 tests/generic/151                             |    2
 tests/generic/152                             |    2
 tests/generic/157                             |    3
 tests/generic/158                             |    3
 tests/generic/159                             |    2
 tests/generic/160                             |    2
 tests/generic/162                             |    4
 tests/generic/163                             |    4
 tests/generic/164                             |    4
 tests/generic/165                             |    4
 tests/generic/166                             |    6
 tests/generic/167                             |    4
 tests/generic/168                             |    7
 tests/generic/170                             |    7
 tests/generic/171                             |    6
 tests/generic/172                             |    6
 tests/generic/173                             |    6
 tests/generic/174                             |    6
 tests/generic/204                             |    8
 tests/generic/232                             |    5
 tests/generic/232.out                         |    1
 tests/generic/247                             |    2
 tests/generic/250                             |    2
 tests/generic/251                             |    5
 tests/generic/252                             |    5
 tests/generic/265                             |    2
 tests/generic/266                             |    2
 tests/generic/267                             |    2
 tests/generic/268                             |    2
 tests/generic/269                             |    8
 tests/generic/270                             |   10
 tests/generic/271                             |    2
 tests/generic/272                             |    2
 tests/generic/273                             |    2
 tests/generic/274                             |    6
 tests/generic/275                             |    6
 tests/generic/276                             |    2
 tests/generic/278                             |    2
 tests/generic/279                             |    2
 tests/generic/281                             |    2
 tests/generic/282                             |    2
 tests/generic/283                             |    2
 tests/generic/306                             |    2
 tests/generic/310                             |   24 +-
 tests/generic/315                             |    2
 tests/generic/317                             |    2
 tests/generic/318                             |    2
 tests/generic/321                             |    4
 tests/generic/323                             |    7
 tests/generic/325                             |    2
 tests/generic/328                             |    4
 tests/generic/329                             |    5
 tests/generic/330                             |    2
 tests/generic/331                             |    7
 tests/generic/332                             |    4
 tests/generic/333                             |    6
 tests/generic/334                             |    6
 tests/generic/335                             |    2
 tests/generic/336                             |    9
 tests/generic/341                             |    2
 tests/generic/342                             |    2
 tests/generic/343                             |    2
 tests/generic/347                             |    2
 tests/generic/348                             |    2
 tests/generic/353                             |    2
 tests/generic/361                             |    8
 tests/generic/373                             |    4
 tests/generic/374                             |    4
 tests/generic/376                             |    2
 tests/generic/382                             |    2
 tests/generic/387                             |    2
 tests/generic/388                             |   24 --
 tests/generic/390                             |   11
 tests/generic/391                             |    2
 tests/generic/395                             |    2
 tests/generic/409                             |   11
 tests/generic/410                             |   11
 tests/generic/411                             |   11
 tests/generic/416                             |    2
 tests/generic/422                             |    4
 tests/generic/425                             |    2
 tests/generic/441                             |    2
 tests/generic/459                             |    8
 tests/generic/461                             |   17 -
 tests/generic/464                             |   10
 tests/generic/474                             |    4
 tests/generic/475                             |   17 -
 tests/generic/476                             |   15 -
 tests/generic/479                             |    2
 tests/generic/480                             |    2
 tests/generic/482                             |   12 -
 tests/generic/483                             |    2
 tests/generic/484                             |    3
 tests/generic/489                             |    2
 tests/generic/502                             |    2
 tests/generic/505                             |    2
 tests/generic/506                             |    2
 tests/generic/507                             |    2
 tests/generic/508                             |    2
 tests/generic/510                             |    2
 tests/generic/520                             |    2
 tests/generic/526                             |    2
 tests/generic/527                             |    2
 tests/generic/530                             |    2
 tests/generic/531                             |    8
 tests/generic/535                             |    2
 tests/generic/546                             |    2
 tests/generic/547                             |    5
 tests/generic/556                             |    2
 tests/generic/560                             |    7
 tests/generic/561                             |   25 +-
 tests/generic/563                             |   24 +-
 tests/generic/564                             |   12 -
 tests/generic/579                             |    8
 tests/generic/585                             |    4
 tests/generic/589                             |   11
 tests/generic/590                             |    9
 tests/generic/599                             |    2
 tests/generic/601                             |    7
 tests/generic/603                             |   14 -
 tests/generic/604                             |    2
 tests/generic/610                             |    2
 tests/generic/620                             |    1
 tests/generic/628                             |    4
 tests/generic/629                             |    4
 tests/generic/631                             |    2
 tests/generic/632                             |    2
 tests/generic/640                             |    2
 tests/generic/642                             |   10
 tests/generic/648                             |   25 --
 tests/generic/650                             |   19 -
 tests/generic/670                             |    2
 tests/generic/671                             |    2
 tests/generic/672                             |    2
 tests/generic/673                             |    2
 tests/generic/674                             |    2
 tests/generic/675                             |    2
 tests/generic/677                             |    2
 tests/generic/683                             |    2
 tests/generic/684                             |    2
 tests/generic/685                             |    2
 tests/generic/686                             |    2
 tests/generic/687                             |    2
 tests/generic/688                             |    2
 tests/generic/690                             |    2
 tests/generic/691                             |    6
 tests/generic/694                             |    2
 tests/generic/695                             |    2
 tests/generic/698                             |    4
 tests/generic/699                             |    8
 tests/generic/703                             |    2
 tests/generic/704                             |    2
 tests/generic/707                             |    7
 tests/generic/716                             |    2
 tests/generic/717                             |    2
 tests/generic/718                             |    2
 tests/generic/719                             |    2
 tests/generic/721                             |    2
 tests/generic/722                             |   15 -
 tests/generic/725                             |    2
 tests/generic/726                             |    2
 tests/generic/727                             |    2
 tests/generic/730                             |    2
 tests/generic/731                             |    2
 tests/generic/732                             |    4
 tests/generic/735                             |    2
 tests/generic/738                             |    2
 tests/generic/741                             |    2
 tests/generic/743                             |    4
 tests/generic/744                             |   10
 tests/generic/745                             |    4
 tests/generic/746                             |   18 -
 tests/generic/747                             |    4
 tests/generic/749                             |    4
 tests/generic/750                             |   10
 tests/generic/751                             |    1
 tests/generic/753                             |   17 -
 tests/overlay/019                             |   48 ++--
 tests/overlay/021                             |    8
 tests/overlay/058                             |   12 -
 tests/xfs/006                                 |    7
 tests/xfs/011                                 |   11
 tests/xfs/013                                 |   22 -
 tests/xfs/014                                 |   11
 tests/xfs/016                                 |    4
 tests/xfs/017                                 |   13 -
 tests/xfs/017.out                             |    1
 tests/xfs/032                                 |    2
 tests/xfs/049                                 |   39 ++-
 tests/xfs/050                                 |    5
 tests/xfs/051                                 |   10
 tests/xfs/052                                 |    2
 tests/xfs/057                                 |    9
 tests/xfs/070                                 |    9
 tests/xfs/073                                 |   36 +--
 tests/xfs/074                                 |   23 +-
 tests/xfs/076                                 |    8
 tests/xfs/077                                 |    2
 tests/xfs/078                                 |   20 -
 tests/xfs/079                                 |   17 -
 tests/xfs/104                                 |   14 -
 tests/xfs/110                                 |    2
 tests/xfs/118                                 |    4
 tests/xfs/119                                 |    2
 tests/xfs/128                                 |    2
 tests/xfs/133                                 |    2
 tests/xfs/134                                 |    2
 tests/xfs/137                                 |    2
 tests/xfs/141                                 |   13 -
 tests/xfs/148                                 |   29 +-
 tests/xfs/149                                 |    8
 tests/xfs/154                                 |    1
 tests/xfs/158                                 |    4
 tests/xfs/161                                 |    2
 tests/xfs/167                                 |   17 -
 tests/xfs/168                                 |    8
 tests/xfs/176                                 |   10
 tests/xfs/177                                 |   15 -
 tests/xfs/186                                 |    4
 tests/xfs/195                                 |    2
 tests/xfs/201                                 |    4
 tests/xfs/212                                 |    2
 tests/xfs/216                                 |   23 +-
 tests/xfs/217                                 |   24 +-
 tests/xfs/227                                 |   59 +++--
 tests/xfs/231                                 |    4
 tests/xfs/232                                 |   10
 tests/xfs/234                                 |    2
 tests/xfs/236                                 |    2
 tests/xfs/237                                 |   13 -
 tests/xfs/239                                 |    2
 tests/xfs/240                                 |    7
 tests/xfs/241                                 |    2
 tests/xfs/243                                 |   11
 tests/xfs/246                                 |    2
 tests/xfs/250                                 |   19 +
 tests/xfs/259                                 |   13 -
 tests/xfs/264                                 |    4
 tests/xfs/265                                 |    2
 tests/xfs/270                                 |    2
 tests/xfs/272                                 |    2
 tests/xfs/274                                 |    2
 tests/xfs/276                                 |    2
 tests/xfs/289                                 |    4
 tests/xfs/291                                 |    4
 tests/xfs/297                                 |   12 -
 tests/xfs/300                                 |    8
 tests/xfs/305                                 |    8
 tests/xfs/309                                 |    2
 tests/xfs/312                                 |    2
 tests/xfs/313                                 |    2
 tests/xfs/314                                 |    2
 tests/xfs/315                                 |    4
 tests/xfs/316                                 |    2
 tests/xfs/317                                 |    2
 tests/xfs/318                                 |    4
 tests/xfs/319                                 |    2
 tests/xfs/320                                 |    2
 tests/xfs/321                                 |    2
 tests/xfs/322                                 |    2
 tests/xfs/323                                 |    2
 tests/xfs/324                                 |    2
 tests/xfs/325                                 |    4
 tests/xfs/326                                 |    4
 tests/xfs/327                                 |    2
 tests/xfs/420                                 |    2
 tests/xfs/421                                 |    2
 tests/xfs/423                                 |    2
 tests/xfs/438                                 |    2
 tests/xfs/440                                 |    8
 tests/xfs/442                                 |   15 -
 tests/xfs/448                                 |    6
 tests/xfs/495                                 |    2
 tests/xfs/501                                 |    2
 tests/xfs/502                                 |    2
 tests/xfs/507                                 |    2
 tests/xfs/511                                 |    2
 tests/xfs/513                                 |   48 +---
 tests/xfs/519                                 |    2
 tests/xfs/520                                 |    2
 tests/xfs/521                                 |    8
 tests/xfs/527                                 |    5
 tests/xfs/528                                 |   10
 tests/xfs/530                                 |   11
 tests/xfs/538                                 |    4
 tests/xfs/541                                 |    2
 tests/xfs/544                                 |    2
 tests/xfs/553                                 |    4
 tests/xfs/558                                 |    7
 tests/xfs/601                                 |    2
 tests/xfs/606                                 |   14 -
 tests/xfs/607                                 |    6
 tests/xfs/609                                 |   20 -
 tests/xfs/610                                 |   20 -
 tests/xfs/613                                 |   44 +--
 tests/xfs/613.out                             |    1
 tests/xfs/617                                 |    2
 tests/xfs/629                                 |    6
 tests/xfs/630                                 |    2
 tests/xfs/631                                 |    9
 tests/xfs/790                                 |    2
 tests/xfs/791                                 |    2
 tests/xfs/792                                 |    2
 tests/xfs/802                                 |    7
 423 files changed, 1827 insertions(+), 1629 deletions(-)
#60 lore raw
Quoted 9 lines · Zorro Lang
Quoted 7 lines · Dave Chinner

Hi folks,

This patchset introduces the ability to run fstests concurrently instead of serially as the current check script does. A git branch containing this patchset can be pulled from here:

https://git.kernel.org/pub/scm/linux/kernel/git/dgc/xfstests-dev.git check-parallel

Quoted 7 lines

Hi Dave,

I've merged your "check-parallel" branch, and rebase on fstests' patches-in-queue branch (which is nearly the next release). I just pushed a new branch "for-dave-check-parallel" which fixed all conflicts. It'll be "next next" release, feel free to update base on that. I'll test that branch too :)

I ran this through my test infrastructure at zorro's request. I saw a bunch of loop dev errors trickle out:

129.out.bad renamed from 129.out +3 -0
@@ -2,3 +2,6 @@
 Create the original file blocks
 Reflink every other block
 Create metadump file, restore it and check restored fs
+losetup: /dev/loop0: detach failed: No such device or address
+Cannot destroy loop device /dev/loop0
+(see /var/tmp/fstests/xfs/129.full for details)

and I noticed the runtimes for running serially went way up. Not sure if that was because my dev tree has a bunch of metadir fixes in it or not; will run that again over the weekend with upstream tot to see if it that brings the total runtime back down.

--D

Quoted 779 lines

Thanks,
Zorro

Quoted 774 lines

The motivation for this is the ever growing runtime of fstests as more tests are added, along with the extremely low resource usage of individual tests. This means that a typical machine running fstests is under-utilised and a single auto group test set execution takes hours.

Yes, I know that I could scale out testing by running lots of little VMs at once, and I already do that. The problem is that I don't get back a complete auto test run result for hours. Want to check that a one-line fix has not caused any regressions? That's, at minimum, an overnight wait for the test farm to crunch through a dozen configs.

On my 64p/128GB RAM VM, 'check -g auto -s xfs -x dump' typically takes around 230-240 minutes to run. With this patchset applied, it runs the same set of tests in 8-10 minutes. I can run ~25 complete auto group test sets with check-parallel in the same time it takes check to run one.

IOWs, I can have the most common config complete a full regression test run as fast as I can turn around a new kernel with a new change to test. I have CPU, memory and IO to burn in my test machines, but what I lack is instant feedback for the change I just made. check-parallel is fast enough that it gives me pretty much instant feedback....

Most of this patchset is preparing infrastructure for concurrent test execution and fixing bugs in tests that I've found whilst getting concurrent execution working reliably. The infrastructure changes center around:

- getting rid of killall - there's nothing quite like one test killing the processes of 15 other tests at the same time...
- cleaning up background process instantiation and reaping, which is a lot more important when an interrupt needs to kill hundreds of processes instead of just a couple.
- isolating error reporting (e.g. dmesg filtering) so that one test failure doesn't trigger lots of other false failure detections
- sanitising the use of loopback devices
- avoiding the use of fixed device names - multiple tests need to use dm-flakey, dm-error, etc devices at the same time, so each test needs a unique device name
- marking tests that are unreliable when outside perturbations occur. These include tests that expect to find delalloc extents, write specific offset patterns in memory then sync them to create specific layouts, etc. e.g. If another test runs sync(1), then those write patterns no longer produce the expected output.
- taming tests that weren't designed for high CPU count machines
- replacing udevadm settle calls because udev is -never- idle when there are tens to hundreds of block devices and filesystems being rapidly set up and torn down.
- converting sync(1) and sync(2) to syncfs(2) to avoid ihaving hundreds of concurrent superblock list traversals lock-stepping with multiple mount/unmounts every second.

There are lots of little other things, but those are the main test and test infrastructure changes. Some of these are big - the fsstress execution rework touches 105 files, but it now means that every single fsstress execution in fstests is controlled by 4 helper functions:

_run_fsstress() - run fsstress synchronously
_run_fsstress_bg - run fsstress in background
_wait_for_fsstress - wait for background fsstress
_kill_fsstress - kill+wait for background fsstress

The test infrastructure also automatically handles cleanup of fsstress processes when the test is interrupted, so tests using fsstress don't need a custom _cleanup() function just to call _kill_fsstress(). This is something that should have been done a long time ago, but now it is critical for being able to manage multiple independent concurrent fsstress invocations sanely.

There are some tests that just can't be run reliably in a concurrent environment - if there is outside interference in, say, page cache flushing then the tests fail. These tests have been added to the "unreliable_in_parallel" test group with a comment explaining why they are unreliable. The check-parallel script automatically excludes this test group.

The only remaining set of tests that are somewhat flakey are the tests that exercise quotas. Quota tests randomly fail for various reasons. Sometimes they don't detect EDQUOT conditions. Sometimes repquota emits weird "device not found" errors. Sometimes grace periods don't start, sometimes they don't time out, or time out and then don't trigger EDQUOT. I don't know why these weird things are happening yet, and until I properly integrate test group selection into check-parallel I can't really isolate the quota tests to focus on them alone.

There are several patches that speed up test runtime. There were several tests that were taking 12-15 minutes to run each, and these made up >95% of the entire check-parallel runtime. In general, these tests have been made to run with more concurrency to speed them up. the result is that the longest individual test runtime has dropped to around 7 minutes, and the elapsed runtime for check-parallel has dropped to 8-10 minutes.

Hence there are 39 patches that are doing prep work on tests and infrastructure to make tests run reliabily in concurrent test contexts. The last patch is the check-parallel runner script that runs the tests concurrently.

The check-parallel script is still very rudimentary. I hard coded the tests it runs (generic+xfs auto tests) and the concurrency so that I could run explicit sets of tests with './check --exact-order <list>'. It is pointed at a mounted directory, and it creates all the infrastructure it needs to run the tests within that directory.

This enabled me to break up the tests across a set of identical runner process contexts. Each runner does:

- create runner directory
- create test and scratch image files
- create loopback devices for test and scratch devices
- sets up results directories
- execute check in it's own private mount namespace so it can't see any of the mounts that other runners are using.
- tears down loopback devices
- reports test failures.

If you run with the same directory over and over again, then it reuses the same runner infrastructure and test and scratch image files. The results directories are not overwritten as they are date-stamped, hence using the same mount point automatically creates a result archive for later data mining.

A typical target directory (/mnt/xfs) looks like:

/mnt/xfs/runner-0/...
/mnt/xfs/runner-1/...
....
/mnt/xfs/runner-63/...

And each runner directory:

log
results-2024-11-19-11:22:25/...
results-2024-11-19-12:36:28/...
.....
results-2024-11-27-13:32:42/...
scratch/
scratch.img
test/
test.img

The log file is the check output for that runner and should look familiar:

SECTION       -- xfs
FSTYP         -- xfs (debug)
PLATFORM      -- Linux/x86_64 test1 6.12.0-dgc+ #297 SMP PREEMPT_DYNAMIC Wed Nov 27 08:13:06 AEDT 2024
MKFS_OPTIONS  -- -f -m rmapbt=1 /dev/loop10

MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/loop10 /mnt/xfs/runner-10/scratch

generic/387        309s
generic/551        49s
generic/289        2s
xfs/274        3s
generic/677        3s
generic/143        5s
generic/304        1s
generic/390        4s
generic/427        4s
xfs/103        1s
generic/252        3s
xfs/045        1s
generic/374        2s
generic/002        1s
generic/534        1s
generic/039        1s

generic/595 [not run] No encryption support for xfs
xfs/122 [not run] Could not compile test program (see end of /mnt/xfs/runner-10/results-2024-11-27-13:32:42/xfs/xfs/122.full)
xfs/556 [not run] xfs_scrub not found
Ran: generic/387 generic/551 generic/289 xfs/274 generic/677 generic/143 generic/304 generic/390 generic/427 xfs/103 generic/252 xfs/045 generic/374 generic/002 generic/534 generic/039 generic/595 xfs/122 xfs/556 Not run: generic/595 xfs/122 xfs/556
Passed all 19 tests

SECTION -- xfs
=========================
Ran: generic/387 generic/551 generic/289 xfs/274 generic/677 generic/143 generic/304 generic/390 generic/427 xfs/103 generic/252 xfs/045 generic/374 generic/002 generic/534 generic/039 generic/595 xfs/122 xfs/556 Not run: generic/595 xfs/122 xfs/556
Passed all 19 tests

Doing something with all the log files from a run can be done with "/mnt/xfs/*/log". e.g. grep, vi, etc.

The results directory contains the same check.{full,log,time} and test results directories as per a normal check invocation, so there's little difference in checking/analysing results with a parallel execution run.

Because the tests run in private mount namespaces, it's easy to see what check-parallel is running at any point in time using 'pstree -N mnt'. Here's a run that is hung on an unmount not completing:

$ pstree -N mnt
[4026531841]
bash
bash───pstree
[0]
sudo───sudo───check-parallel─┬─check-parallel───nsexec───check───311───fsync-tester
                             ├─check-parallel───nsexec───check───467───open_by_handle
                             ├─check-parallel───nsexec───check───338
                             ├─check-parallel───nsexec───check───421
                             ├─check-parallel───nsexec───check───441
                             ├─check-parallel───nsexec───check───232
                             ├─check-parallel───nsexec───check───477───open_by_handle
                             ├─check-parallel───nsexec───check───420
                             ├─check-parallel───nsexec───check───426───open_by_handle
                             ├─check-parallel───nsexec───check───756───open_by_handle
                             ├─check-parallel───nsexec───check───231
                             ├─check-parallel───nsexec───check───475───475.fsstress───475.fsstress───{475.fsstress}
                             ├─check-parallel───nsexec───check───388───388.fsstress───388.fsstress───{388.fsstress}
                             ├─check-parallel───nsexec───check───259───sync
                             ├─check-parallel───nsexec───check───622───sync
                             ├─check-parallel───nsexec───check───318───sync
                             ├─check-parallel───nsexec───check───753───umount
                             ├─check-parallel───nsexec───check───086
                             ├─check-parallel───nsexec───check───648───648.fsstress───648.fsstress───{648.fsstress}
                             ├─check-parallel───nsexec───check───391
                             ├─check-parallel───nsexec───check───315───sync
                             └─check-parallel───nsexec───check───183───bulkstat_unlink

All the other processes are in sync or dropping caches and stuck waiting for the superblock s_umount lock that the unmount holds. Finding where it is stuck:

$ pgrep [u]mount
1081885
$ sudo cat /proc/1081885/stack
[<0>] xfs_ail_push_all_sync+0x9c/0xf0
[<0>] xfs_unmount_flush_inodes+0x41/0x70
[<0>] xfs_unmountfs+0x59/0x190
[<0>] xfs_fs_put_super+0x3b/0x90
[<0>] generic_shutdown_super+0x77/0x160
[<0>] kill_block_super+0x1b/0x40
[<0>] xfs_kill_sb+0x12/0x30
[<0>] deactivate_locked_super+0x38/0x100
[<0>] deactivate_super+0x41/0x50
[<0>] cleanup_mnt+0x9f/0x160
[<0>] __cleanup_mnt+0x12/0x20
[<0>] task_work_run+0x89/0xb0
[<0>] resume_user_mode_work+0x4f/0x60
[<0>] syscall_exit_to_user_mode+0x76/0xb0
[<0>] do_syscall_64+0x74/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
$

Yup, known bug - a shutdown vs XFS_ISTALE inode cluster freeing issue that leaves pinned, stale inodes in the AIL.

The point I'm making here is that running tests concurrently doesn't change anything material in how you'd go about discovering and diagnosing failures. The only difference is in what that initial failure might look like. e.g. Failures that result in dmesg output will cause a huge number of tests to all fail with "check dmesg for failure output" reports. Hence there is a bit of sifting to find which test triggered the dmesg output, but failure detection still works just fine.

The check-parallel script is really only at proof-of-concept stage. It is sufficient to run tests in parallel, but that's about it. most of the work so far has gone into making the generic and XFS tests run reliably in parallel. I have not run any other tests, and haven't done full conversions of other test directories or C code in src/. e.g. the sync -> syncfs was only done for the common dir and the generic and xfs tests dirs

I do not plan to do these conversions for ext4/btrfs/overlay/etc any time soon as these conversions are mostly about improving execution time, not test correctness. Hence they are not a priority for me - the priority is further developing the concurrent execution environment. i.e. check-parallel.

To that end, I need to factor all the test selection and exclusion code out of check, and do the same with the actual test list runner loop. That way I can reuse all the existing code from within the check-parallel context rather than having to call check to do all of that work itself. I would like to get check-parallel to the point where it is mostly just s/check/check-parallel/ on the command line to move from serial to concurrent test execution.

I also want to try to integrate the config section stuff into check-parallel. This is more a case of defining what devices the config needs to create (i.e. as loop devices) rather than what devices it should be using. I think I can do this just by defining a different set of environment variables (e.g. NEED_SCRATCHDEV, NEED_LOGDEV, etc) and triggering the loop device creation from these variables.

In a way, the fact that check-parallel bootstraps it's own runtime environment almost makes it entirely zero-config. As it stands right now, you should be able to pull this patchset, create your base test directory (make sure you have at least 100GB of free disk space), run './check-parallel <test_dir> -x dump' and it Should Just Work.

At this point, the test infrastructure problems are largely solved. My focus is now on further development of the check-parallel script and integrating it tightly into the existing check infrastructure rather than open-coding test lists and configuration information. This will probably take a bit of time, so I'd like to get the bug fixes, improvements and infrastructure changes underway so I'm not left carrying a huge patchset for months....

----------------------------------------------------------------
Dave Chinner (40):
xfs/448: get rid of assert-on-failure
fstests: cleanup fsstress process management
fuzzy: don't use killall
fstests: per-test dmflakey instances
fstests: per-test dmerror instances
fstests: per-test dmhuge instances
fstests: per-test dmthin instances
fstests: per-test dmdust instances
fstests: per-test dmdelay instances
fstests: fix DM device creation/removal vs udev races
fstests: use syncfs rather than sync
fstests: clean up mount and unmount operations
fstests: clean up loop device instantiation
fstests: xfs/227 is really slow
fstests: mark tests that are unreliable when run in parallel
fstests: use udevadm wait in preference to settle xfs/442: rescale load so it's not exponential xfs/176: fix broken setup code
xfs/177: remove unused slab object count location checks
fstests: remove uses of killall where possible generic/127: reduce runtime
quota: system project quota files need to be shared
dmesg: reduce noise from other tests
fstests: stop using /tmp directly
fstests: scale some tests for high CPU count sanity generic/310: cleanup killing background processes
filter: handle mount errors from CONFIG_BLK_DEV_WRITE_MOUNTED=y
filters: add a filter that accepts EIO instead of other errors generic/085: general cleanup for reliability and debugging
fstests: don't use directory stacks
fstests: clean up a couple of dm-flakey tests
fstests: clean up termination of various tests
vfstests: some tests require the testdir to be shared xfs/629: single extent files should be within tolerance xfs/076: fix broken mkfs filtering
fstests: capture some failures to seqres.full
fstests: always use fail-at-unmount semantics for XFS generic/062: don't leave debug files in $here on failure
fstests: quota grace periods unreliable under load
fstests: check-parallel

 check                                         |   12 -
 check-parallel                                |  205 ++++++++++++++++++
 common/btrfs                                  |    4
 common/config                                 |   36 ++-
 common/dmdelay                                |   24 +-
 common/dmdust                                 |   14 -
 common/dmerror                                |   74 +++---
 common/dmflakey                               |   60 ++---
 common/dmhugedisk                             |   21 +
 common/dmlogwrites                            |    4
 common/dmthin                                 |   12 -
 common/encrypt                                |    2
 common/filter                                 |   17 +
 common/fuzzy                                  |   37 +--
 common/log                                    |    2
 common/metadump                               |   32 +-
 common/overlay                                |   10
 common/populate                               |    8
 common/preamble                               |    1
 common/quota                                  |   37 ---
 common/rc                                     |  166 +++++++++++---
 common/repair                                 |    2
 common/report                                 |    2
 common/verity                                 |    2
 common/xfs                                    |    2
 doc/group-names.txt                           |    1
 doc/requirement-checking.txt                  |    6
 ltp/fsstress.c                                |   28 ++
 src/aio-dio-regress/aio-last-ref-held-by-io.c |    5
 src/dmerror                                   |    6
 tests/btrfs/004                               |   11
 tests/btrfs/007                               |    3
 tests/btrfs/012                               |    4
 tests/btrfs/028                               |    6
 tests/btrfs/049                               |    4
 tests/btrfs/057                               |    4
 tests/btrfs/060                               |   14 -
 tests/btrfs/061                               |   13 -
 tests/btrfs/062                               |   13 -
 tests/btrfs/063                               |   13 -
 tests/btrfs/064                               |   13 -
 tests/btrfs/065                               |   14 -
 tests/btrfs/066                               |   14 -
 tests/btrfs/067                               |   14 -
 tests/btrfs/068                               |   14 -
 tests/btrfs/069                               |   13 -
 tests/btrfs/070                               |   13 -
 tests/btrfs/071                               |   13 -
 tests/btrfs/072                               |   14 -
 tests/btrfs/073                               |   13 -
 tests/btrfs/074                               |   13 -
 tests/btrfs/078                               |   12 -
 tests/btrfs/100                               |    4
 tests/btrfs/101                               |    4
 tests/btrfs/136                               |    6
 tests/btrfs/160                               |    3
 tests/btrfs/192                               |   12 -
 tests/btrfs/195                               |    2
 tests/btrfs/212                               |   16 -
 tests/btrfs/232                               |    4
 tests/btrfs/252                               |    5
 tests/btrfs/261                               |    2
 tests/btrfs/284                               |    4
 tests/btrfs/286                               |    2
 tests/btrfs/291                               |    5
 tests/btrfs/320                               |    6
 tests/btrfs/332                               |    4
 tests/ext4/004                                |    4
 tests/ext4/057                                |   14 -
 tests/ext4/058                                |    3
 tests/ext4/307                                |    4
 tests/generic/013                             |   22 -
 tests/generic/015                             |    2
 tests/generic/019                             |   13 -
 tests/generic/029                             |    2
 tests/generic/030                             |    2
 tests/generic/032                             |    2
 tests/generic/034                             |    2
 tests/generic/039                             |    2
 tests/generic/040                             |    2
 tests/generic/041                             |    2
 tests/generic/042                             |    4
 tests/generic/048                             |    2
 tests/generic/049                             |    4
 tests/generic/050                             |    3
 tests/generic/051                             |   24 --
 tests/generic/054                             |    2
 tests/generic/055                             |    6
 tests/generic/057                             |    2
 tests/generic/059                             |    2
 tests/generic/062                             |    2
 tests/generic/065                             |    2
 tests/generic/066                             |    5
 tests/generic/067                             |   17 -
 tests/generic/068                             |    7
 tests/generic/070                             |   12 -
 tests/generic/073                             |    2
 tests/generic/076                             |   10
 tests/generic/076.out                         |    1
 tests/generic/081                             |    8
 tests/generic/083                             |   13 -
 tests/generic/083.out                         |    1
 tests/generic/084                             |   12 -
 tests/generic/085                             |   21 +
 tests/generic/090                             |    4
 tests/generic/092                             |    2
 tests/generic/098                             |    4
 tests/generic/099                             |    8
 tests/generic/101                             |    2
 tests/generic/104                             |    2
 tests/generic/106                             |    2
 tests/generic/107                             |    2
 tests/generic/108                             |    9
 tests/generic/109                             |    5
 tests/generic/117                             |    4
 tests/generic/127                             |   67 +++--
 tests/generic/127.out                         |    6
 tests/generic/135                             |   24 --
 tests/generic/150                             |    2
 tests/generic/151                             |    2
 tests/generic/152                             |    2
 tests/generic/157                             |    3
 tests/generic/158                             |    3
 tests/generic/159                             |    2
 tests/generic/160                             |    2
 tests/generic/162                             |    4
 tests/generic/163                             |    4
 tests/generic/164                             |    4
 tests/generic/165                             |    4
 tests/generic/166                             |    6
 tests/generic/167                             |    4
 tests/generic/168                             |    7
 tests/generic/170                             |    7
 tests/generic/171                             |    6
 tests/generic/172                             |    6
 tests/generic/173                             |    6
 tests/generic/174                             |    6
 tests/generic/204                             |    8
 tests/generic/232                             |    5
 tests/generic/232.out                         |    1
 tests/generic/247                             |    2
 tests/generic/250                             |    2
 tests/generic/251                             |    5
 tests/generic/252                             |    5
 tests/generic/265                             |    2
 tests/generic/266                             |    2
 tests/generic/267                             |    2
 tests/generic/268                             |    2
 tests/generic/269                             |    8
 tests/generic/270                             |   10
 tests/generic/271                             |    2
 tests/generic/272                             |    2
 tests/generic/273                             |    2
 tests/generic/274                             |    6
 tests/generic/275                             |    6
 tests/generic/276                             |    2
 tests/generic/278                             |    2
 tests/generic/279                             |    2
 tests/generic/281                             |    2
 tests/generic/282                             |    2
 tests/generic/283                             |    2
 tests/generic/306                             |    2
 tests/generic/310                             |   24 +-
 tests/generic/315                             |    2
 tests/generic/317                             |    2
 tests/generic/318                             |    2
 tests/generic/321                             |    4
 tests/generic/323                             |    7
 tests/generic/325                             |    2
 tests/generic/328                             |    4
 tests/generic/329                             |    5
 tests/generic/330                             |    2
 tests/generic/331                             |    7
 tests/generic/332                             |    4
 tests/generic/333                             |    6
 tests/generic/334                             |    6
 tests/generic/335                             |    2
 tests/generic/336                             |    9
 tests/generic/341                             |    2
 tests/generic/342                             |    2
 tests/generic/343                             |    2
 tests/generic/347                             |    2
 tests/generic/348                             |    2
 tests/generic/353                             |    2
 tests/generic/361                             |    8
 tests/generic/373                             |    4
 tests/generic/374                             |    4
 tests/generic/376                             |    2
 tests/generic/382                             |    2
 tests/generic/387                             |    2
 tests/generic/388                             |   24 --
 tests/generic/390                             |   11
 tests/generic/391                             |    2
 tests/generic/395                             |    2
 tests/generic/409                             |   11
 tests/generic/410                             |   11
 tests/generic/411                             |   11
 tests/generic/416                             |    2
 tests/generic/422                             |    4
 tests/generic/425                             |    2
 tests/generic/441                             |    2
 tests/generic/459                             |    8
 tests/generic/461                             |   17 -
 tests/generic/464                             |   10
 tests/generic/474                             |    4
 tests/generic/475                             |   17 -
 tests/generic/476                             |   15 -
 tests/generic/479                             |    2
 tests/generic/480                             |    2
 tests/generic/482                             |   12 -
 tests/generic/483                             |    2
 tests/generic/484                             |    3
 tests/generic/489                             |    2
 tests/generic/502                             |    2
 tests/generic/505                             |    2
 tests/generic/506                             |    2
 tests/generic/507                             |    2
 tests/generic/508                             |    2
 tests/generic/510                             |    2
 tests/generic/520                             |    2
 tests/generic/526                             |    2
 tests/generic/527                             |    2
 tests/generic/530                             |    2
 tests/generic/531                             |    8
 tests/generic/535                             |    2
 tests/generic/546                             |    2
 tests/generic/547                             |    5
 tests/generic/556                             |    2
 tests/generic/560                             |    7
 tests/generic/561                             |   25 +-
 tests/generic/563                             |   24 +-
 tests/generic/564                             |   12 -
 tests/generic/579                             |    8
 tests/generic/585                             |    4
 tests/generic/589                             |   11
 tests/generic/590                             |    9
 tests/generic/599                             |    2
 tests/generic/601                             |    7
 tests/generic/603                             |   14 -
 tests/generic/604                             |    2
 tests/generic/610                             |    2
 tests/generic/620                             |    1
 tests/generic/628                             |    4
 tests/generic/629                             |    4
 tests/generic/631                             |    2
 tests/generic/632                             |    2
 tests/generic/640                             |    2
 tests/generic/642                             |   10
 tests/generic/648                             |   25 --
 tests/generic/650                             |   19 -
 tests/generic/670                             |    2
 tests/generic/671                             |    2
 tests/generic/672                             |    2
 tests/generic/673                             |    2
 tests/generic/674                             |    2
 tests/generic/675                             |    2
 tests/generic/677                             |    2
 tests/generic/683                             |    2
 tests/generic/684                             |    2
 tests/generic/685                             |    2
 tests/generic/686                             |    2
 tests/generic/687                             |    2
 tests/generic/688                             |    2
 tests/generic/690                             |    2
 tests/generic/691                             |    6
 tests/generic/694                             |    2
 tests/generic/695                             |    2
 tests/generic/698                             |    4
 tests/generic/699                             |    8
 tests/generic/703                             |    2
 tests/generic/704                             |    2
 tests/generic/707                             |    7
 tests/generic/716                             |    2
 tests/generic/717                             |    2
 tests/generic/718                             |    2
 tests/generic/719                             |    2
 tests/generic/721                             |    2
 tests/generic/722                             |   15 -
 tests/generic/725                             |    2
 tests/generic/726                             |    2
 tests/generic/727                             |    2
 tests/generic/730                             |    2
 tests/generic/731                             |    2
 tests/generic/732                             |    4
 tests/generic/735                             |    2
 tests/generic/738                             |    2
 tests/generic/741                             |    2
 tests/generic/743                             |    4
 tests/generic/744                             |   10
 tests/generic/745                             |    4
 tests/generic/746                             |   18 -
 tests/generic/747                             |    4
 tests/generic/749                             |    4
 tests/generic/750                             |   10
 tests/generic/751                             |    1
 tests/generic/753                             |   17 -
 tests/overlay/019                             |   48 ++--
 tests/overlay/021                             |    8
 tests/overlay/058                             |   12 -
 tests/xfs/006                                 |    7
 tests/xfs/011                                 |   11
 tests/xfs/013                                 |   22 -
 tests/xfs/014                                 |   11
 tests/xfs/016                                 |    4
 tests/xfs/017                                 |   13 -
 tests/xfs/017.out                             |    1
 tests/xfs/032                                 |    2
 tests/xfs/049                                 |   39 ++-
 tests/xfs/050                                 |    5
 tests/xfs/051                                 |   10
 tests/xfs/052                                 |    2
 tests/xfs/057                                 |    9
 tests/xfs/070                                 |    9
 tests/xfs/073                                 |   36 +--
 tests/xfs/074                                 |   23 +-
 tests/xfs/076                                 |    8
 tests/xfs/077                                 |    2
 tests/xfs/078                                 |   20 -
 tests/xfs/079                                 |   17 -
 tests/xfs/104                                 |   14 -
 tests/xfs/110                                 |    2
 tests/xfs/118                                 |    4
 tests/xfs/119                                 |    2
 tests/xfs/128                                 |    2
 tests/xfs/133                                 |    2
 tests/xfs/134                                 |    2
 tests/xfs/137                                 |    2
 tests/xfs/141                                 |   13 -
 tests/xfs/148                                 |   29 +-
 tests/xfs/149                                 |    8
 tests/xfs/154                                 |    1
 tests/xfs/158                                 |    4
 tests/xfs/161                                 |    2
 tests/xfs/167                                 |   17 -
 tests/xfs/168                                 |    8
 tests/xfs/176                                 |   10
 tests/xfs/177                                 |   15 -
 tests/xfs/186                                 |    4
 tests/xfs/195                                 |    2
 tests/xfs/201                                 |    4
 tests/xfs/212                                 |    2
 tests/xfs/216                                 |   23 +-
 tests/xfs/217                                 |   24 +-
 tests/xfs/227                                 |   59 +++--
 tests/xfs/231                                 |    4
 tests/xfs/232                                 |   10
 tests/xfs/234                                 |    2
 tests/xfs/236                                 |    2
 tests/xfs/237                                 |   13 -
 tests/xfs/239                                 |    2
 tests/xfs/240                                 |    7
 tests/xfs/241                                 |    2
 tests/xfs/243                                 |   11
 tests/xfs/246                                 |    2
 tests/xfs/250                                 |   19 +
 tests/xfs/259                                 |   13 -
 tests/xfs/264                                 |    4
 tests/xfs/265                                 |    2
 tests/xfs/270                                 |    2
 tests/xfs/272                                 |    2
 tests/xfs/274                                 |    2
 tests/xfs/276                                 |    2
 tests/xfs/289                                 |    4
 tests/xfs/291                                 |    4
 tests/xfs/297                                 |   12 -
 tests/xfs/300                                 |    8
 tests/xfs/305                                 |    8
 tests/xfs/309                                 |    2
 tests/xfs/312                                 |    2
 tests/xfs/313                                 |    2
 tests/xfs/314                                 |    2
 tests/xfs/315                                 |    4
 tests/xfs/316                                 |    2
 tests/xfs/317                                 |    2
 tests/xfs/318                                 |    4
 tests/xfs/319                                 |    2
 tests/xfs/320                                 |    2
 tests/xfs/321                                 |    2
 tests/xfs/322                                 |    2
 tests/xfs/323                                 |    2
 tests/xfs/324                                 |    2
 tests/xfs/325                                 |    4
 tests/xfs/326                                 |    4
 tests/xfs/327                                 |    2
 tests/xfs/420                                 |    2
 tests/xfs/421                                 |    2
 tests/xfs/423                                 |    2
 tests/xfs/438                                 |    2
 tests/xfs/440                                 |    8
 tests/xfs/442                                 |   15 -
 tests/xfs/448                                 |    6
 tests/xfs/495                                 |    2
 tests/xfs/501                                 |    2
 tests/xfs/502                                 |    2
 tests/xfs/507                                 |    2
 tests/xfs/511                                 |    2
 tests/xfs/513                                 |   48 +---
 tests/xfs/519                                 |    2
 tests/xfs/520                                 |    2
 tests/xfs/521                                 |    8
 tests/xfs/527                                 |    5
 tests/xfs/528                                 |   10
 tests/xfs/530                                 |   11
 tests/xfs/538                                 |    4
 tests/xfs/541                                 |    2
 tests/xfs/544                                 |    2
 tests/xfs/553                                 |    4
 tests/xfs/558                                 |    7
 tests/xfs/601                                 |    2
 tests/xfs/606                                 |   14 -
 tests/xfs/607                                 |    6
 tests/xfs/609                                 |   20 -
 tests/xfs/610                                 |   20 -
 tests/xfs/613                                 |   44 +--
 tests/xfs/613.out                             |    1
 tests/xfs/617                                 |    2
 tests/xfs/629                                 |    6
 tests/xfs/630                                 |    2
 tests/xfs/631                                 |    9
 tests/xfs/790                                 |    2
 tests/xfs/791                                 |    2
 tests/xfs/792                                 |    2
 tests/xfs/802                                 |    7
 423 files changed, 1827 insertions(+), 1629 deletions(-)
#61 lore raw
Quoted 31 lines · Darrick J. Wong
Quoted 9 lines · Zorro Lang
Quoted 7 lines · Dave Chinner

Hi folks,

This patchset introduces the ability to run fstests concurrently instead of serially as the current check script does. A git branch containing this patchset can be pulled from here:

https://git.kernel.org/pub/scm/linux/kernel/git/dgc/xfstests-dev.git check-parallel

Quoted 7 lines

Hi Dave,

I've merged your "check-parallel" branch, and rebase on fstests' patches-in-queue branch (which is nearly the next release). I just pushed a new branch "for-dave-check-parallel" which fixed all conflicts. It'll be "next next" release, feel free to update base on that. I'll test that branch too :)

I ran this through my test infrastructure at zorro's request. I saw a bunch of loop dev errors trickle out:

129.out.bad renamed from 129.out +3 -0
@@ -2,3 +2,6 @@
 Create the original file blocks
 Reflink every other block
 Create metadump file, restore it and check restored fs
+losetup: /dev/loop0: detach failed: No such device or address
+Cannot destroy loop device /dev/loop0
+(see /var/tmp/fstests/xfs/129.full for details)
Quoted 4 lines

and I noticed the runtimes for running serially went way up. Not sure if that was because my dev tree has a bunch of metadir fixes in it or not; will run that again over the weekend with upstream tot to see if it that brings the total runtime back down.

Thanks Darrick! The "[PATCH] fstests: clean up loop device instantiation" does below change [1], it looks different with your original code. You trys to check [ -n "$XFS_METADUMP_IMG" ] before loop detaching, and you use a while loop to do the loop detaching. Does this change break your original test?

Thanks,
Zorro

[1]

common/metadump +3 -11
@@ -24,17 +24,9 @@ _xfs_cleanup_verify_metadump()
 
        test -n "$XFS_METADUMP_FILE" && rm -f "$XFS_METADUMP_FILE"
 
-       if [ -n "$XFS_METADUMP_IMG" ]; then
-               losetup -n -a -O BACK-FILE,NAME | grep "^$XFS_METADUMP_IMG" | while read backing ldev; do
-                       losetup -d "$ldev"
-               done
-
-               # Don't call rm directly with a globbed argument here to avoid
-               # issues issues with variable expansions.
-               for img in "$XFS_METADUMP_IMG"*; do
-                       test -e "$img" && rm -f "$img"
-               done
-       fi
+       [ -n "$md_data_loop_dev" ] && _destroy_loop_device $md_data_loop_dev
+       [ -n "$md_log_loop_dev" ] && _destroy_loop_device $md_log_loop_dev
+       rm -f $data_img $log_img
 
Quoted 783 lines

--D

Quoted 779 lines

Thanks,
Zorro

Quoted 774 lines

The motivation for this is the ever growing runtime of fstests as more tests are added, along with the extremely low resource usage of individual tests. This means that a typical machine running fstests is under-utilised and a single auto group test set execution takes hours.

Yes, I know that I could scale out testing by running lots of little VMs at once, and I already do that. The problem is that I don't get back a complete auto test run result for hours. Want to check that a one-line fix has not caused any regressions? That's, at minimum, an overnight wait for the test farm to crunch through a dozen configs.

On my 64p/128GB RAM VM, 'check -g auto -s xfs -x dump' typically takes around 230-240 minutes to run. With this patchset applied, it runs the same set of tests in 8-10 minutes. I can run ~25 complete auto group test sets with check-parallel in the same time it takes check to run one.

IOWs, I can have the most common config complete a full regression test run as fast as I can turn around a new kernel with a new change to test. I have CPU, memory and IO to burn in my test machines, but what I lack is instant feedback for the change I just made. check-parallel is fast enough that it gives me pretty much instant feedback....

Most of this patchset is preparing infrastructure for concurrent test execution and fixing bugs in tests that I've found whilst getting concurrent execution working reliably. The infrastructure changes center around:

- getting rid of killall - there's nothing quite like one test killing the processes of 15 other tests at the same time...
- cleaning up background process instantiation and reaping, which is a lot more important when an interrupt needs to kill hundreds of processes instead of just a couple.
- isolating error reporting (e.g. dmesg filtering) so that one test failure doesn't trigger lots of other false failure detections
- sanitising the use of loopback devices
- avoiding the use of fixed device names - multiple tests need to use dm-flakey, dm-error, etc devices at the same time, so each test needs a unique device name
- marking tests that are unreliable when outside perturbations occur. These include tests that expect to find delalloc extents, write specific offset patterns in memory then sync them to create specific layouts, etc. e.g. If another test runs sync(1), then those write patterns no longer produce the expected output.
- taming tests that weren't designed for high CPU count machines
- replacing udevadm settle calls because udev is -never- idle when there are tens to hundreds of block devices and filesystems being rapidly set up and torn down.
- converting sync(1) and sync(2) to syncfs(2) to avoid ihaving hundreds of concurrent superblock list traversals lock-stepping with multiple mount/unmounts every second.

There are lots of little other things, but those are the main test and test infrastructure changes. Some of these are big - the fsstress execution rework touches 105 files, but it now means that every single fsstress execution in fstests is controlled by 4 helper functions:

_run_fsstress() - run fsstress synchronously
_run_fsstress_bg - run fsstress in background
_wait_for_fsstress - wait for background fsstress
_kill_fsstress - kill+wait for background fsstress

The test infrastructure also automatically handles cleanup of fsstress processes when the test is interrupted, so tests using fsstress don't need a custom _cleanup() function just to call _kill_fsstress(). This is something that should have been done a long time ago, but now it is critical for being able to manage multiple independent concurrent fsstress invocations sanely.

There are some tests that just can't be run reliably in a concurrent environment - if there is outside interference in, say, page cache flushing then the tests fail. These tests have been added to the "unreliable_in_parallel" test group with a comment explaining why they are unreliable. The check-parallel script automatically excludes this test group.

The only remaining set of tests that are somewhat flakey are the tests that exercise quotas. Quota tests randomly fail for various reasons. Sometimes they don't detect EDQUOT conditions. Sometimes repquota emits weird "device not found" errors. Sometimes grace periods don't start, sometimes they don't time out, or time out and then don't trigger EDQUOT. I don't know why these weird things are happening yet, and until I properly integrate test group selection into check-parallel I can't really isolate the quota tests to focus on them alone.

There are several patches that speed up test runtime. There were several tests that were taking 12-15 minutes to run each, and these made up >95% of the entire check-parallel runtime. In general, these tests have been made to run with more concurrency to speed them up. the result is that the longest individual test runtime has dropped to around 7 minutes, and the elapsed runtime for check-parallel has dropped to 8-10 minutes.

Hence there are 39 patches that are doing prep work on tests and infrastructure to make tests run reliabily in concurrent test contexts. The last patch is the check-parallel runner script that runs the tests concurrently.

The check-parallel script is still very rudimentary. I hard coded the tests it runs (generic+xfs auto tests) and the concurrency so that I could run explicit sets of tests with './check --exact-order <list>'. It is pointed at a mounted directory, and it creates all the infrastructure it needs to run the tests within that directory.

This enabled me to break up the tests across a set of identical runner process contexts. Each runner does:

- create runner directory
- create test and scratch image files
- create loopback devices for test and scratch devices
- sets up results directories
- execute check in it's own private mount namespace so it can't see any of the mounts that other runners are using.
- tears down loopback devices
- reports test failures.

If you run with the same directory over and over again, then it reuses the same runner infrastructure and test and scratch image files. The results directories are not overwritten as they are date-stamped, hence using the same mount point automatically creates a result archive for later data mining.

A typical target directory (/mnt/xfs) looks like:

/mnt/xfs/runner-0/...
/mnt/xfs/runner-1/...
....
/mnt/xfs/runner-63/...

And each runner directory:

log
results-2024-11-19-11:22:25/...
results-2024-11-19-12:36:28/...
.....
results-2024-11-27-13:32:42/...
scratch/
scratch.img
test/
test.img

The log file is the check output for that runner and should look familiar:

SECTION       -- xfs
FSTYP         -- xfs (debug)
PLATFORM      -- Linux/x86_64 test1 6.12.0-dgc+ #297 SMP PREEMPT_DYNAMIC Wed Nov 27 08:13:06 AEDT 2024
MKFS_OPTIONS  -- -f -m rmapbt=1 /dev/loop10

MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/loop10 /mnt/xfs/runner-10/scratch

generic/387        309s
generic/551        49s
generic/289        2s
xfs/274        3s
generic/677        3s
generic/143        5s
generic/304        1s
generic/390        4s
generic/427        4s
xfs/103        1s
generic/252        3s
xfs/045        1s
generic/374        2s
generic/002        1s
generic/534        1s
generic/039        1s

generic/595 [not run] No encryption support for xfs
xfs/122 [not run] Could not compile test program (see end of /mnt/xfs/runner-10/results-2024-11-27-13:32:42/xfs/xfs/122.full)
xfs/556 [not run] xfs_scrub not found
Ran: generic/387 generic/551 generic/289 xfs/274 generic/677 generic/143 generic/304 generic/390 generic/427 xfs/103 generic/252 xfs/045 generic/374 generic/002 generic/534 generic/039 generic/595 xfs/122 xfs/556 Not run: generic/595 xfs/122 xfs/556
Passed all 19 tests

SECTION -- xfs
=========================
Ran: generic/387 generic/551 generic/289 xfs/274 generic/677 generic/143 generic/304 generic/390 generic/427 xfs/103 generic/252 xfs/045 generic/374 generic/002 generic/534 generic/039 generic/595 xfs/122 xfs/556 Not run: generic/595 xfs/122 xfs/556
Passed all 19 tests

Doing something with all the log files from a run can be done with "/mnt/xfs/*/log". e.g. grep, vi, etc.

The results directory contains the same check.{full,log,time} and test results directories as per a normal check invocation, so there's little difference in checking/analysing results with a parallel execution run.

Because the tests run in private mount namespaces, it's easy to see what check-parallel is running at any point in time using 'pstree -N mnt'. Here's a run that is hung on an unmount not completing:

$ pstree -N mnt
[4026531841]
bash
bash───pstree
[0]
sudo───sudo───check-parallel─┬─check-parallel───nsexec───check───311───fsync-tester
                             ├─check-parallel───nsexec───check───467───open_by_handle
                             ├─check-parallel───nsexec───check───338
                             ├─check-parallel───nsexec───check───421
                             ├─check-parallel───nsexec───check───441
                             ├─check-parallel───nsexec───check───232
                             ├─check-parallel───nsexec───check───477───open_by_handle
                             ├─check-parallel───nsexec───check───420
                             ├─check-parallel───nsexec───check───426───open_by_handle
                             ├─check-parallel───nsexec───check───756───open_by_handle
                             ├─check-parallel───nsexec───check───231
                             ├─check-parallel───nsexec───check───475───475.fsstress───475.fsstress───{475.fsstress}
                             ├─check-parallel───nsexec───check───388───388.fsstress───388.fsstress───{388.fsstress}
                             ├─check-parallel───nsexec───check───259───sync
                             ├─check-parallel───nsexec───check───622───sync
                             ├─check-parallel───nsexec───check───318───sync
                             ├─check-parallel───nsexec───check───753───umount
                             ├─check-parallel───nsexec───check───086
                             ├─check-parallel───nsexec───check───648───648.fsstress───648.fsstress───{648.fsstress}
                             ├─check-parallel───nsexec───check───391
                             ├─check-parallel───nsexec───check───315───sync
                             └─check-parallel───nsexec───check───183───bulkstat_unlink

All the other processes are in sync or dropping caches and stuck waiting for the superblock s_umount lock that the unmount holds. Finding where it is stuck:

$ pgrep [u]mount
1081885
$ sudo cat /proc/1081885/stack
[<0>] xfs_ail_push_all_sync+0x9c/0xf0
[<0>] xfs_unmount_flush_inodes+0x41/0x70
[<0>] xfs_unmountfs+0x59/0x190
[<0>] xfs_fs_put_super+0x3b/0x90
[<0>] generic_shutdown_super+0x77/0x160
[<0>] kill_block_super+0x1b/0x40
[<0>] xfs_kill_sb+0x12/0x30
[<0>] deactivate_locked_super+0x38/0x100
[<0>] deactivate_super+0x41/0x50
[<0>] cleanup_mnt+0x9f/0x160
[<0>] __cleanup_mnt+0x12/0x20
[<0>] task_work_run+0x89/0xb0
[<0>] resume_user_mode_work+0x4f/0x60
[<0>] syscall_exit_to_user_mode+0x76/0xb0
[<0>] do_syscall_64+0x74/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
$

Yup, known bug - a shutdown vs XFS_ISTALE inode cluster freeing issue that leaves pinned, stale inodes in the AIL.

The point I'm making here is that running tests concurrently doesn't change anything material in how you'd go about discovering and diagnosing failures. The only difference is in what that initial failure might look like. e.g. Failures that result in dmesg output will cause a huge number of tests to all fail with "check dmesg for failure output" reports. Hence there is a bit of sifting to find which test triggered the dmesg output, but failure detection still works just fine.

The check-parallel script is really only at proof-of-concept stage. It is sufficient to run tests in parallel, but that's about it. most of the work so far has gone into making the generic and XFS tests run reliably in parallel. I have not run any other tests, and haven't done full conversions of other test directories or C code in src/. e.g. the sync -> syncfs was only done for the common dir and the generic and xfs tests dirs

I do not plan to do these conversions for ext4/btrfs/overlay/etc any time soon as these conversions are mostly about improving execution time, not test correctness. Hence they are not a priority for me - the priority is further developing the concurrent execution environment. i.e. check-parallel.

To that end, I need to factor all the test selection and exclusion code out of check, and do the same with the actual test list runner loop. That way I can reuse all the existing code from within the check-parallel context rather than having to call check to do all of that work itself. I would like to get check-parallel to the point where it is mostly just s/check/check-parallel/ on the command line to move from serial to concurrent test execution.

I also want to try to integrate the config section stuff into check-parallel. This is more a case of defining what devices the config needs to create (i.e. as loop devices) rather than what devices it should be using. I think I can do this just by defining a different set of environment variables (e.g. NEED_SCRATCHDEV, NEED_LOGDEV, etc) and triggering the loop device creation from these variables.

In a way, the fact that check-parallel bootstraps it's own runtime environment almost makes it entirely zero-config. As it stands right now, you should be able to pull this patchset, create your base test directory (make sure you have at least 100GB of free disk space), run './check-parallel <test_dir> -x dump' and it Should Just Work.

At this point, the test infrastructure problems are largely solved. My focus is now on further development of the check-parallel script and integrating it tightly into the existing check infrastructure rather than open-coding test lists and configuration information. This will probably take a bit of time, so I'd like to get the bug fixes, improvements and infrastructure changes underway so I'm not left carrying a huge patchset for months....

----------------------------------------------------------------
Dave Chinner (40):
xfs/448: get rid of assert-on-failure
fstests: cleanup fsstress process management
fuzzy: don't use killall
fstests: per-test dmflakey instances
fstests: per-test dmerror instances
fstests: per-test dmhuge instances
fstests: per-test dmthin instances
fstests: per-test dmdust instances
fstests: per-test dmdelay instances
fstests: fix DM device creation/removal vs udev races
fstests: use syncfs rather than sync
fstests: clean up mount and unmount operations
fstests: clean up loop device instantiation
fstests: xfs/227 is really slow
fstests: mark tests that are unreliable when run in parallel
fstests: use udevadm wait in preference to settle xfs/442: rescale load so it's not exponential xfs/176: fix broken setup code
xfs/177: remove unused slab object count location checks
fstests: remove uses of killall where possible generic/127: reduce runtime
quota: system project quota files need to be shared
dmesg: reduce noise from other tests
fstests: stop using /tmp directly
fstests: scale some tests for high CPU count sanity generic/310: cleanup killing background processes
filter: handle mount errors from CONFIG_BLK_DEV_WRITE_MOUNTED=y
filters: add a filter that accepts EIO instead of other errors generic/085: general cleanup for reliability and debugging
fstests: don't use directory stacks
fstests: clean up a couple of dm-flakey tests
fstests: clean up termination of various tests
vfstests: some tests require the testdir to be shared xfs/629: single extent files should be within tolerance xfs/076: fix broken mkfs filtering
fstests: capture some failures to seqres.full
fstests: always use fail-at-unmount semantics for XFS generic/062: don't leave debug files in $here on failure
fstests: quota grace periods unreliable under load
fstests: check-parallel

 check                                         |   12 -
 check-parallel                                |  205 ++++++++++++++++++
 common/btrfs                                  |    4
 common/config                                 |   36 ++-
 common/dmdelay                                |   24 +-
 common/dmdust                                 |   14 -
 common/dmerror                                |   74 +++---
 common/dmflakey                               |   60 ++---
 common/dmhugedisk                             |   21 +
 common/dmlogwrites                            |    4
 common/dmthin                                 |   12 -
 common/encrypt                                |    2
 common/filter                                 |   17 +
 common/fuzzy                                  |   37 +--
 common/log                                    |    2
 common/metadump                               |   32 +-
 common/overlay                                |   10
 common/populate                               |    8
 common/preamble                               |    1
 common/quota                                  |   37 ---
 common/rc                                     |  166 +++++++++++---
 common/repair                                 |    2
 common/report                                 |    2
 common/verity                                 |    2
 common/xfs                                    |    2
 doc/group-names.txt                           |    1
 doc/requirement-checking.txt                  |    6
 ltp/fsstress.c                                |   28 ++
 src/aio-dio-regress/aio-last-ref-held-by-io.c |    5
 src/dmerror                                   |    6
 tests/btrfs/004                               |   11
 tests/btrfs/007                               |    3
 tests/btrfs/012                               |    4
 tests/btrfs/028                               |    6
 tests/btrfs/049                               |    4
 tests/btrfs/057                               |    4
 tests/btrfs/060                               |   14 -
 tests/btrfs/061                               |   13 -
 tests/btrfs/062                               |   13 -
 tests/btrfs/063                               |   13 -
 tests/btrfs/064                               |   13 -
 tests/btrfs/065                               |   14 -
 tests/btrfs/066                               |   14 -
 tests/btrfs/067                               |   14 -
 tests/btrfs/068                               |   14 -
 tests/btrfs/069                               |   13 -
 tests/btrfs/070                               |   13 -
 tests/btrfs/071                               |   13 -
 tests/btrfs/072                               |   14 -
 tests/btrfs/073                               |   13 -
 tests/btrfs/074                               |   13 -
 tests/btrfs/078                               |   12 -
 tests/btrfs/100                               |    4
 tests/btrfs/101                               |    4
 tests/btrfs/136                               |    6
 tests/btrfs/160                               |    3
 tests/btrfs/192                               |   12 -
 tests/btrfs/195                               |    2
 tests/btrfs/212                               |   16 -
 tests/btrfs/232                               |    4
 tests/btrfs/252                               |    5
 tests/btrfs/261                               |    2
 tests/btrfs/284                               |    4
 tests/btrfs/286                               |    2
 tests/btrfs/291                               |    5
 tests/btrfs/320                               |    6
 tests/btrfs/332                               |    4
 tests/ext4/004                                |    4
 tests/ext4/057                                |   14 -
 tests/ext4/058                                |    3
 tests/ext4/307                                |    4
 tests/generic/013                             |   22 -
 tests/generic/015                             |    2
 tests/generic/019                             |   13 -
 tests/generic/029                             |    2
 tests/generic/030                             |    2
 tests/generic/032                             |    2
 tests/generic/034                             |    2
 tests/generic/039                             |    2
 tests/generic/040                             |    2
 tests/generic/041                             |    2
 tests/generic/042                             |    4
 tests/generic/048                             |    2
 tests/generic/049                             |    4
 tests/generic/050                             |    3
 tests/generic/051                             |   24 --
 tests/generic/054                             |    2
 tests/generic/055                             |    6
 tests/generic/057                             |    2
 tests/generic/059                             |    2
 tests/generic/062                             |    2
 tests/generic/065                             |    2
 tests/generic/066                             |    5
 tests/generic/067                             |   17 -
 tests/generic/068                             |    7
 tests/generic/070                             |   12 -
 tests/generic/073                             |    2
 tests/generic/076                             |   10
 tests/generic/076.out                         |    1
 tests/generic/081                             |    8
 tests/generic/083                             |   13 -
 tests/generic/083.out                         |    1
 tests/generic/084                             |   12 -
 tests/generic/085                             |   21 +
 tests/generic/090                             |    4
 tests/generic/092                             |    2
 tests/generic/098                             |    4
 tests/generic/099                             |    8
 tests/generic/101                             |    2
 tests/generic/104                             |    2
 tests/generic/106                             |    2
 tests/generic/107                             |    2
 tests/generic/108                             |    9
 tests/generic/109                             |    5
 tests/generic/117                             |    4
 tests/generic/127                             |   67 +++--
 tests/generic/127.out                         |    6
 tests/generic/135                             |   24 --
 tests/generic/150                             |    2
 tests/generic/151                             |    2
 tests/generic/152                             |    2
 tests/generic/157                             |    3
 tests/generic/158                             |    3
 tests/generic/159                             |    2
 tests/generic/160                             |    2
 tests/generic/162                             |    4
 tests/generic/163                             |    4
 tests/generic/164                             |    4
 tests/generic/165                             |    4
 tests/generic/166                             |    6
 tests/generic/167                             |    4
 tests/generic/168                             |    7
 tests/generic/170                             |    7
 tests/generic/171                             |    6
 tests/generic/172                             |    6
 tests/generic/173                             |    6
 tests/generic/174                             |    6
 tests/generic/204                             |    8
 tests/generic/232                             |    5
 tests/generic/232.out                         |    1
 tests/generic/247                             |    2
 tests/generic/250                             |    2
 tests/generic/251                             |    5
 tests/generic/252                             |    5
 tests/generic/265                             |    2
 tests/generic/266                             |    2
 tests/generic/267                             |    2
 tests/generic/268                             |    2
 tests/generic/269                             |    8
 tests/generic/270                             |   10
 tests/generic/271                             |    2
 tests/generic/272                             |    2
 tests/generic/273                             |    2
 tests/generic/274                             |    6
 tests/generic/275                             |    6
 tests/generic/276                             |    2
 tests/generic/278                             |    2
 tests/generic/279                             |    2
 tests/generic/281                             |    2
 tests/generic/282                             |    2
 tests/generic/283                             |    2
 tests/generic/306                             |    2
 tests/generic/310                             |   24 +-
 tests/generic/315                             |    2
 tests/generic/317                             |    2
 tests/generic/318                             |    2
 tests/generic/321                             |    4
 tests/generic/323                             |    7
 tests/generic/325                             |    2
 tests/generic/328                             |    4
 tests/generic/329                             |    5
 tests/generic/330                             |    2
 tests/generic/331                             |    7
 tests/generic/332                             |    4
 tests/generic/333                             |    6
 tests/generic/334                             |    6
 tests/generic/335                             |    2
 tests/generic/336                             |    9
 tests/generic/341                             |    2
 tests/generic/342                             |    2
 tests/generic/343                             |    2
 tests/generic/347                             |    2
 tests/generic/348                             |    2
 tests/generic/353                             |    2
 tests/generic/361                             |    8
 tests/generic/373                             |    4
 tests/generic/374                             |    4
 tests/generic/376                             |    2
 tests/generic/382                             |    2
 tests/generic/387                             |    2
 tests/generic/388                             |   24 --
 tests/generic/390                             |   11
 tests/generic/391                             |    2
 tests/generic/395                             |    2
 tests/generic/409                             |   11
 tests/generic/410                             |   11
 tests/generic/411                             |   11
 tests/generic/416                             |    2
 tests/generic/422                             |    4
 tests/generic/425                             |    2
 tests/generic/441                             |    2
 tests/generic/459                             |    8
 tests/generic/461                             |   17 -
 tests/generic/464                             |   10
 tests/generic/474                             |    4
 tests/generic/475                             |   17 -
 tests/generic/476                             |   15 -
 tests/generic/479                             |    2
 tests/generic/480                             |    2
 tests/generic/482                             |   12 -
 tests/generic/483                             |    2
 tests/generic/484                             |    3
 tests/generic/489                             |    2
 tests/generic/502                             |    2
 tests/generic/505                             |    2
 tests/generic/506                             |    2
 tests/generic/507                             |    2
 tests/generic/508                             |    2
 tests/generic/510                             |    2
 tests/generic/520                             |    2
 tests/generic/526                             |    2
 tests/generic/527                             |    2
 tests/generic/530                             |    2
 tests/generic/531                             |    8
 tests/generic/535                             |    2
 tests/generic/546                             |    2
 tests/generic/547                             |    5
 tests/generic/556                             |    2
 tests/generic/560                             |    7
 tests/generic/561                             |   25 +-
 tests/generic/563                             |   24 +-
 tests/generic/564                             |   12 -
 tests/generic/579                             |    8
 tests/generic/585                             |    4
 tests/generic/589                             |   11
 tests/generic/590                             |    9
 tests/generic/599                             |    2
 tests/generic/601                             |    7
 tests/generic/603                             |   14 -
 tests/generic/604                             |    2
 tests/generic/610                             |    2
 tests/generic/620                             |    1
 tests/generic/628                             |    4
 tests/generic/629                             |    4
 tests/generic/631                             |    2
 tests/generic/632                             |    2
 tests/generic/640                             |    2
 tests/generic/642                             |   10
 tests/generic/648                             |   25 --
 tests/generic/650                             |   19 -
 tests/generic/670                             |    2
 tests/generic/671                             |    2
 tests/generic/672                             |    2
 tests/generic/673                             |    2
 tests/generic/674                             |    2
 tests/generic/675                             |    2
 tests/generic/677                             |    2
 tests/generic/683                             |    2
 tests/generic/684                             |    2
 tests/generic/685                             |    2
 tests/generic/686                             |    2
 tests/generic/687                             |    2
 tests/generic/688                             |    2
 tests/generic/690                             |    2
 tests/generic/691                             |    6
 tests/generic/694                             |    2
 tests/generic/695                             |    2
 tests/generic/698                             |    4
 tests/generic/699                             |    8
 tests/generic/703                             |    2
 tests/generic/704                             |    2
 tests/generic/707                             |    7
 tests/generic/716                             |    2
 tests/generic/717                             |    2
 tests/generic/718                             |    2
 tests/generic/719                             |    2
 tests/generic/721                             |    2
 tests/generic/722                             |   15 -
 tests/generic/725                             |    2
 tests/generic/726                             |    2
 tests/generic/727                             |    2
 tests/generic/730                             |    2
 tests/generic/731                             |    2
 tests/generic/732                             |    4
 tests/generic/735                             |    2
 tests/generic/738                             |    2
 tests/generic/741                             |    2
 tests/generic/743                             |    4
 tests/generic/744                             |   10
 tests/generic/745                             |    4
 tests/generic/746                             |   18 -
 tests/generic/747                             |    4
 tests/generic/749                             |    4
 tests/generic/750                             |   10
 tests/generic/751                             |    1
 tests/generic/753                             |   17 -
 tests/overlay/019                             |   48 ++--
 tests/overlay/021                             |    8
 tests/overlay/058                             |   12 -
 tests/xfs/006                                 |    7
 tests/xfs/011                                 |   11
 tests/xfs/013                                 |   22 -
 tests/xfs/014                                 |   11
 tests/xfs/016                                 |    4
 tests/xfs/017                                 |   13 -
 tests/xfs/017.out                             |    1
 tests/xfs/032                                 |    2
 tests/xfs/049                                 |   39 ++-
 tests/xfs/050                                 |    5
 tests/xfs/051                                 |   10
 tests/xfs/052                                 |    2
 tests/xfs/057                                 |    9
 tests/xfs/070                                 |    9
 tests/xfs/073                                 |   36 +--
 tests/xfs/074                                 |   23 +-
 tests/xfs/076                                 |    8
 tests/xfs/077                                 |    2
 tests/xfs/078                                 |   20 -
 tests/xfs/079                                 |   17 -
 tests/xfs/104                                 |   14 -
 tests/xfs/110                                 |    2
 tests/xfs/118                                 |    4
 tests/xfs/119                                 |    2
 tests/xfs/128                                 |    2
 tests/xfs/133                                 |    2
 tests/xfs/134                                 |    2
 tests/xfs/137                                 |    2
 tests/xfs/141                                 |   13 -
 tests/xfs/148                                 |   29 +-
 tests/xfs/149                                 |    8
 tests/xfs/154                                 |    1
 tests/xfs/158                                 |    4
 tests/xfs/161                                 |    2
 tests/xfs/167                                 |   17 -
 tests/xfs/168                                 |    8
 tests/xfs/176                                 |   10
 tests/xfs/177                                 |   15 -
 tests/xfs/186                                 |    4
 tests/xfs/195                                 |    2
 tests/xfs/201                                 |    4
 tests/xfs/212                                 |    2
 tests/xfs/216                                 |   23 +-
 tests/xfs/217                                 |   24 +-
 tests/xfs/227                                 |   59 +++--
 tests/xfs/231                                 |    4
 tests/xfs/232                                 |   10
 tests/xfs/234                                 |    2
 tests/xfs/236                                 |    2
 tests/xfs/237                                 |   13 -
 tests/xfs/239                                 |    2
 tests/xfs/240                                 |    7
 tests/xfs/241                                 |    2
 tests/xfs/243                                 |   11
 tests/xfs/246                                 |    2
 tests/xfs/250                                 |   19 +
 tests/xfs/259                                 |   13 -
 tests/xfs/264                                 |    4
 tests/xfs/265                                 |    2
 tests/xfs/270                                 |    2
 tests/xfs/272                                 |    2
 tests/xfs/274                                 |    2
 tests/xfs/276                                 |    2
 tests/xfs/289                                 |    4
 tests/xfs/291                                 |    4
 tests/xfs/297                                 |   12 -
 tests/xfs/300                                 |    8
 tests/xfs/305                                 |    8
 tests/xfs/309                                 |    2
 tests/xfs/312                                 |    2
 tests/xfs/313                                 |    2
 tests/xfs/314                                 |    2
 tests/xfs/315                                 |    4
 tests/xfs/316                                 |    2
 tests/xfs/317                                 |    2
 tests/xfs/318                                 |    4
 tests/xfs/319                                 |    2
 tests/xfs/320                                 |    2
 tests/xfs/321                                 |    2
 tests/xfs/322                                 |    2
 tests/xfs/323                                 |    2
 tests/xfs/324                                 |    2
 tests/xfs/325                                 |    4
 tests/xfs/326                                 |    4
 tests/xfs/327                                 |    2
 tests/xfs/420                                 |    2
 tests/xfs/421                                 |    2
 tests/xfs/423                                 |    2
 tests/xfs/438                                 |    2
 tests/xfs/440                                 |    8
 tests/xfs/442                                 |   15 -
 tests/xfs/448                                 |    6
 tests/xfs/495                                 |    2
 tests/xfs/501                                 |    2
 tests/xfs/502                                 |    2
 tests/xfs/507                                 |    2
 tests/xfs/511                                 |    2
 tests/xfs/513                                 |   48 +---
 tests/xfs/519                                 |    2
 tests/xfs/520                                 |    2
 tests/xfs/521                                 |    8
 tests/xfs/527                                 |    5
 tests/xfs/528                                 |   10
 tests/xfs/530                                 |   11
 tests/xfs/538                                 |    4
 tests/xfs/541                                 |    2
 tests/xfs/544                                 |    2
 tests/xfs/553                                 |    4
 tests/xfs/558                                 |    7
 tests/xfs/601                                 |    2
 tests/xfs/606                                 |   14 -
 tests/xfs/607                                 |    6
 tests/xfs/609                                 |   20 -
 tests/xfs/610                                 |   20 -
 tests/xfs/613                                 |   44 +--
 tests/xfs/613.out                             |    1
 tests/xfs/617                                 |    2
 tests/xfs/629                                 |    6
 tests/xfs/630                                 |    2
 tests/xfs/631                                 |    9
 tests/xfs/790                                 |    2
 tests/xfs/791                                 |    2
 tests/xfs/792                                 |    2
 tests/xfs/802                                 |    7
 423 files changed, 1827 insertions(+), 1629 deletions(-)
#62 lore raw
In reply to Darrick J. Wong
Quoted 24 lines · Darrick J. Wong
Quoted 9 lines · Zorro Lang
Quoted 7 lines · Dave Chinner

Hi folks,

This patchset introduces the ability to run fstests concurrently instead of serially as the current check script does. A git branch containing this patchset can be pulled from here:

https://git.kernel.org/pub/scm/linux/kernel/git/dgc/xfstests-dev.git check-parallel

Quoted 7 lines

Hi Dave,

I've merged your "check-parallel" branch, and rebase on fstests' patches-in-queue branch (which is nearly the next release). I just pushed a new branch "for-dave-check-parallel" which fixed all conflicts. It'll be "next next" release, feel free to update base on that. I'll test that branch too :)

I ran this through my test infrastructure at zorro's request. I saw a bunch of loop dev errors trickle out:

129.out.bad line 7

Quoted 6 lines
129.out.bad renamed from 129.out +3 -0
@@ -2,3 +2,6 @@
 Create the original file blocks
 Reflink every other block
 Create metadump file, restore it and check restored fs
+losetup: /dev/loop0: detach failed: No such device or address
+Cannot destroy loop device /dev/loop0
+(see /var/tmp/fstests/xfs/129.full for details)

Almost certainly I missed the conversion of names in _xfs_verify_metadump_v1() from "data_loop" to "md_data_loop_dev" and such. common/metadump is liley missing "unset md_data_loop_dev" after destroying the loop devices, too.

Not sure why that isn't triggering on my setup, trivial to fix. I'll sort it out and fold it back into the original loopdev cleanup patch in the set.

Quoted 1 line

and I noticed the runtimes for running serially went way up.

Not seeing that here; I don't think any of the changes I've made should affect the runtime of a normal check test pass; the tests should take the same time to run or run faster after this patchset, even serially...

Quoted 4 lines

Not sure
if that was because my dev tree has a bunch of metadir fixes in it or not; will run that again over the weekend with upstream tot to see if it that brings the total runtime back down.

OK.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#63 lore raw
Quoted 41 lines · Dave Chinner
Quoted 24 lines · Darrick J. Wong
Quoted 9 lines · Zorro Lang
Quoted 7 lines · Dave Chinner

Hi folks,

This patchset introduces the ability to run fstests concurrently instead of serially as the current check script does. A git branch containing this patchset can be pulled from here:

https://git.kernel.org/pub/scm/linux/kernel/git/dgc/xfstests-dev.git check-parallel

Quoted 7 lines

Hi Dave,

I've merged your "check-parallel" branch, and rebase on fstests' patches-in-queue branch (which is nearly the next release). I just pushed a new branch "for-dave-check-parallel" which fixed all conflicts. It'll be "next next" release, feel free to update base on that. I'll test that branch too :)

I ran this through my test infrastructure at zorro's request. I saw a bunch of loop dev errors trickle out:

129.out.bad line 7

Quoted 6 lines
129.out.bad renamed from 129.out +3 -0
@@ -2,3 +2,6 @@
 Create the original file blocks
 Reflink every other block
 Create metadump file, restore it and check restored fs
+losetup: /dev/loop0: detach failed: No such device or address
+Cannot destroy loop device /dev/loop0
+(see /var/tmp/fstests/xfs/129.full for details)

Almost certainly I missed the conversion of names in _xfs_verify_metadump_v1() from "data_loop" to "md_data_loop_dev" and such. common/metadump is liley missing "unset md_data_loop_dev" after destroying the loop devices, too.

Not sure why that isn't triggering on my setup, trivial to fix. I'll sort it out and fold it back into the original loopdev cleanup patch in the set.

Quoted 6 lines
Quoted 1 line

and I noticed the runtimes for running serially went way up.

Not seeing that here; I don't think any of the changes I've made should affect the runtime of a normal check test pass; the tests should take the same time to run or run faster after this patchset, even serially...

Hi Dave,

I replied several emails for this issue:

https://lore.kernel.org/fstests/20241207195101.hfg3m4pgghoo7ebv@dell-per750-06-vm-08.rhts.eng.pek2.redhat.com/T/#mb1da5dddd053dcd5ed8ec15c45ce8e3fa55c2d38

I've tried to fix this and all other small issues on "for-dave-check-parallel" branch:

  # git clone -b for-dave-check-parallel git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git

Could you take a look at it? Currently my test passed on this branch. As my original plan, I'd like to merge for-dave-check-parallel branch onto for-next and push in this week.

So if no one has other critical issue to report, I'll push it as plan. Then we can fix later issues, and other's patches can move on.

Thanks,
Zorro

Quoted 13 lines
Quoted 5 lines

Not sure
if that was because my dev tree has a bunch of metadir fixes in it or not; will run that again over the weekend with upstream tot to see if it that brings the total runtime back down.

OK.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#64 lore raw
Quoted 55 lines · Zorro Lang
Quoted 41 lines · Dave Chinner
Quoted 24 lines · Darrick J. Wong
Quoted 9 lines · Zorro Lang
Quoted 7 lines · Dave Chinner

Hi folks,

This patchset introduces the ability to run fstests concurrently instead of serially as the current check script does. A git branch containing this patchset can be pulled from here:

https://git.kernel.org/pub/scm/linux/kernel/git/dgc/xfstests-dev.git check-parallel

Quoted 7 lines

Hi Dave,

I've merged your "check-parallel" branch, and rebase on fstests' patches-in-queue branch (which is nearly the next release). I just pushed a new branch "for-dave-check-parallel" which fixed all conflicts. It'll be "next next" release, feel free to update base on that. I'll test that branch too :)

I ran this through my test infrastructure at zorro's request. I saw a bunch of loop dev errors trickle out:

129.out.bad line 7

Quoted 6 lines
129.out.bad renamed from 129.out +3 -0
@@ -2,3 +2,6 @@
 Create the original file blocks
 Reflink every other block
 Create metadump file, restore it and check restored fs
+losetup: /dev/loop0: detach failed: No such device or address
+Cannot destroy loop device /dev/loop0
+(see /var/tmp/fstests/xfs/129.full for details)

Almost certainly I missed the conversion of names in _xfs_verify_metadump_v1() from "data_loop" to "md_data_loop_dev" and such. common/metadump is liley missing "unset md_data_loop_dev" after destroying the loop devices, too.

Not sure why that isn't triggering on my setup, trivial to fix. I'll sort it out and fold it back into the original loopdev cleanup patch in the set.

Quoted 6 lines
Quoted 1 line

and I noticed the runtimes for running serially went way up.

Not seeing that here; I don't think any of the changes I've made should affect the runtime of a normal check test pass; the tests should take the same time to run or run faster after this patchset, even serially...

Hi Dave,

I replied several emails for this issue:

https://lore.kernel.org/fstests/20241207195101.hfg3m4pgghoo7ebv@dell-per750-06-vm-08.rhts.eng.pek2.redhat.com/T/#mb1da5dddd053dcd5ed8ec15c45ce8e3fa55c2d38

Quoted 8 lines

I've tried to fix this and all other small issues on "for-dave-check-parallel" branch:

  # git clone -b for-dave-check-parallel git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git

Could you take a look at it? Currently my test passed on this branch. As my original plan, I'd like to merge for-dave-check-parallel branch onto for-next and push in this week.

If it fixes the reported failure, then I'm fine with that.

Unfortunately, the machine I've been developing this code on has now been dead for a week and a half (main board failure), and the vendor doesn't seem to care about the "NBD on-site" warranty SLA...

I have no idea when it'll be fixed, so if what you've done works for everyone else right now, I'll clean up the remaining things I noticed when I've got the machine back up and running and can test this stuff again...

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]