Skip to contentSkip to conversation

xfstests: fixes for CRCs and small devices

PATCH In review Dave Chinner 2 participants 13 messages Feb 24, 2014 – Feb 25, 2014

lore
Share & tools
lore
#0 lore raw

cover xfstests: fixes for CRCs and small devices

Hi folks,

The following set of patches fix two sets of issues. The first is that small, fast filesystems like on a ramdisk trigger some interesting failures due to the rate of modification being so high on such devices. Hence we can do things like run a small filesystem out of space in a few seconds, whereas it woul dtake a few minutes on a SSD or spinning disk. Hence some tests require some small tweaks to avoid failures in such situations.

The second set of patches continue the work of fixing test failures on CRC enabled filesystems due to issues with the tests themselves. There are stll more test fixes needed, but I'm now down to single digit failure rates on CRC enabled filesystems with these patches.

Cheers,

Dave.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw

PATCH 1/6 xfs/033: add golden output for CRC enabled filesystems

From: Dave Chinner <[email protected]>

CRC enabled filesystems emit different errors on corruption. Specifically, inode corruption is picked up much earlier due to verifier failures (e.g. incorrect inode identifier) and so xfs_repair throws errors sufficiently different that filtering cannot hide the differences. Hence simply add a new golden output file and link it appropriately once we know what type of filesystem we are testing.

---
 common/filter               |   3 +
 tests/xfs/033               |  10 ++-
 tests/xfs/033.crc.out.linux | 197 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 207 insertions(+), 3 deletions(-)
 create mode 100644 tests/xfs/033.crc.out.linux
common/filter +3 -0
index 04d12c4..a7d2aa9 100644
@@ -152,6 +152,9 @@ _filter_mkfs()
     if (/^\s+=\s+sectsz=(\d+)\s+attr=(\d+)/) {
         print STDERR "sectsz=$1\nattr=$2\n";
     }
+    if (/^\s+=\s+crc=(\d)/) {
+        print STDERR "_fs_has_crcs=$1\n";
+    }
     if (/^data\s+=\s+bsize=(\d+)\s+blocks=(\d+), imaxpct=(\d+)/) {
 	print STDERR "dbsize=$1\ndblocks=$2\nimaxpct=$3\n";
 	print STDOUT "data     = bsize=XXX blocks=XXX, imaxpct=PCT\n";
tests/xfs/033 +7 -3
index bf906e0..576d437 100755
@@ -45,9 +45,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 . ./common/filter
 . ./common/repair
 
-# link correct .out file
-_link_out_file $seq.out $seqfull.out
-
 # nuke the root, rt bitmap, and rt summary inodes
 # 
 _check_root_inos()
@@ -86,6 +83,13 @@ _scratch_mkfs_xfs | _filter_mkfs 2>$tmp.mkfs
 [ $isize -lt 512 ] && \
 	_scratch_mkfs_xfs -isize=512 | _filter_mkfs >/dev/null 2>&1
 
+# link correct .out file
+if [ $_fs_has_crcs -eq 1 ]; then
+	_link_out_file $seq.crc.out $seqfull.out
+else
+	_link_out_file $seq.out $seqfull.out
+fi
+
 `xfs_db -r -c sb -c p $SCRATCH_DEV | grep 'ino = ' | \
 	sed -e 's/ //g' -e 's/^/export /'`
 
tests/xfs/033.crc.out.linux new file +197 -0

197 lines not on this page · raw message

Signature · 6 lines
1.8.4.rc3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw

PATCH 2/6 xfs/189: noattr2 invalid for CRC enabled filesystems

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Version 5 filesystems always have attr2 format enabled, and it cannot be turned off via the noattr2 mount option. As such, attempts to mount with noattr2 will be rejected and this causes cascading failures within the test.

Hence detect if we've created a CRC enabled filesystem, and if this is the case _notrun the test.

---
 common/attr   | 10 ++++++++++
 tests/xfs/189 |  2 ++
 2 files changed, 12 insertions(+)
common/attr +10 -0
index 1f3e57f..83bf9d1 100644
@@ -196,6 +196,16 @@ _require_attr_v1()
 		|| _notrun "attr v1 not supported on $SCRATCH_DEV"
 }
 
+# check if we support the noattr2 mount option
+_require_noattr2()
+{
+	_scratch_mkfs_xfs > /dev/null 2>&1 \
+		|| _fail "_scratch_mkfs_xfs failed on $SCRATCH_DEV"
+	_mount -o noattr2 $SCRATCH_DEV > /dev/null 2>&1 \
+		|| _notrun "noattr2 mount option not supported on $SCRATCH_DEV"
+	_scratch_unmount
+}
+
 # getfattr -R returns info in readdir order which varies from fs to fs.
 # This sorts the output by filename
 _sort_getfattr_output()
tests/xfs/189 +2 -0
index 27bfb63..a56bbde 100755
@@ -230,6 +230,7 @@ _putback_scratch_fstab()
 # get standard environment, filters and checks
 . ./common/rc
 . ./common/filter
+. ./common/attr
 
 # real QA test starts here
 _supported_fs xfs
@@ -237,6 +238,7 @@ _supported_os Linux
 
 _need_to_be_root
 _require_scratch
+_require_noattr2
 
 unset SCRATCH_RTDEV
 unset SCRATCH_LOGDEV
Signature · 6 lines
1.8.4.rc3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#5 lore raw

PATCH 3/6 generic/003: ensure time changes between stat calls

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

On a fast device like a ramdisk, kernel time may not have changed between a stat of a file and some operation on it immediately afterwards. Hence there is no guarantee that an operation actually changes the timestamps of a file immediately after it is stat'd.

Hence, ensure that the times will change by sleeping for a second between the initial stat that reads the timestamps and the operations that is supposed to modify them. This way we ensure that the timestamp will change if the filesystem is correctly implemented.

While there, fix the indenting to be 8 space tabs and correct the header which is missing the bash shell declaration and the test number identifier.

---
 tests/generic/003 | 110 ++++++++++++++++++++++++++++++------------------------
 1 file changed, 61 insertions(+), 49 deletions(-)
tests/generic/003 +61 -49
index 3a11ab3..5c2d383 100755
@@ -1,3 +1,6 @@
+#! /bin/bash
+# FS QA Test No. generic/003
+#
 # Tests the noatime, relatime, strictatime and nodiratime mount options.
 # There is an extra check for Btrfs to ensure that the access time is
 # never updated on read-only subvolumes. (Regression test for bug fixed
@@ -49,41 +52,40 @@ _require_relatime
 rm -f $seqres.full
 
 _stat() {
-    stat --printf="%x;%y;%z" $1
+	stat --printf="%x;%y;%z" $1
 }
 
 _compare_stat_times() {
-    updated=$1      # 3 chars indicating if access, modify and
-                    # change times should be updated (Y) or not (N)
-    IFS=';' read -a first_stat <<< "$2"   # Convert first stat to array
-    IFS=';' read -a second_stat <<< "$3"  # Convert second stat to array
-    test_step=$4    # Will be printed to output stream in case of an
-                    # error, to make debugging easier
-    types=( access modify change )
-
-    for i in 0 1 2; do
-        if [ "${first_stat[$i]}" == "${second_stat[$i]}" ]; then
-            if [ "${updated:$i:1}" == "Y" ]; then
-                echo -n "ERROR: ${types[$i]} time has not been updated "
-                echo $test_step
-            fi
-        else
-            if [ "${updated:$i:1}" == "N" ]; then
-                echo -n "ERROR: ${types[$i]} time has changed "
-                echo $test_step
-            fi
-        fi
-    done
+	updated=$1	# 3 chars indicating if access, modify and
+			# change times should be updated (Y) or not (N)
+	IFS=';' read -a first_stat <<< "$2"   # Convert first stat to array
+	IFS=';' read -a second_stat <<< "$3"  # Convert second stat to array
+	test_step=$4	# Will be printed to output stream in case of an
+			# error, to make debugging easier
+	types=( access modify change )
+
+	for i in 0 1 2; do
+		if [ "${first_stat[$i]}" == "${second_stat[$i]}" ]; then
+			if [ "${updated:$i:1}" == "N" ]; then
+				continue;
+			fi
+			echo -n "ERROR: ${types[$i]} time has not been updated "
+			echo $test_step
+		elif [ "${updated:$i:1}" == "N" ]; then
+			echo -n "ERROR: ${types[$i]} time has changed "
+			echo $test_step
+		fi
+	done
 }
 
 _scratch_mkfs >> $seqres.full 2>&1 || _fail "mkfs failed"
 _scratch_mount "-o relatime"
 
 if [ "$FSTYP" = "btrfs" ]; then
-    TPATH=$SCRATCH_MNT/sub1
-    $BTRFS_UTIL_PROG subvolume create $TPATH > $seqres.full
+	TPATH=$SCRATCH_MNT/sub1
+	$BTRFS_UTIL_PROG subvolume create $TPATH > $seqres.full
 else
-    TPATH=$SCRATCH_MNT
+	TPATH=$SCRATCH_MNT
 fi
 
 mkdir $TPATH/dir1
@@ -91,62 +93,69 @@ echo "aaa" > $TPATH/dir1/file1
 file1_stat_before_first_access=`_stat $TPATH/dir1/file1`
 
 # Accessing file1 the first time
+sleep 1
 cat $TPATH/dir1/file1 > /dev/null
 file1_stat_after_first_access=`_stat $TPATH/dir1/file1`
 _compare_stat_times YNN "$file1_stat_before_first_access" \
-    "$file1_stat_after_first_access" "after accessing file1 first time"
+	"$file1_stat_after_first_access" "after accessing file1 first time"
 
 # Accessing file1 a second time
+sleep 1
 cat $TPATH/dir1/file1 > /dev/null
 file1_stat_after_second_access=`_stat $TPATH/dir1/file1`
 _compare_stat_times NNN "$file1_stat_after_first_access" \
-    "$file1_stat_after_second_access" "after accessing file1 second time"
+	"$file1_stat_after_second_access" "after accessing file1 second time"
 
 # Remounting with nodiratime option
 _scratch_unmount
 _scratch_mount "-o nodiratime"
 file1_stat_after_remount=`_stat $TPATH/dir1/file1`
 _compare_stat_times NNN "$file1_stat_after_second_access" \
-    "$file1_stat_after_remount" "for file1 after remount"
+	"$file1_stat_after_remount" "for file1 after remount"
 
 # Creating dir2 and file2, checking directory stats
 mkdir $TPATH/dir2
 dir2_stat_before_file_creation=`_stat $TPATH/dir2`
+sleep 1
 echo "bbb" > $TPATH/dir2/file2
 dir2_stat_after_file_creation=`_stat $TPATH/dir2`
 _compare_stat_times NYY "$dir2_stat_before_file_creation" \
-    "$dir2_stat_after_file_creation" "for dir2 after file creation"
+	"$dir2_stat_after_file_creation" "for dir2 after file creation"
 
 # Accessing file2
 file2_stat_before_first_access=`_stat $TPATH/dir2/file2`
+sleep 1
 cat $TPATH/dir2/file2 > /dev/null
 file2_stat_after_first_access=`_stat $TPATH/dir2/file2`
 _compare_stat_times YNN "$file2_stat_before_first_access" \
-    "$file2_stat_after_first_access" "after accessing file2"
+	"$file2_stat_after_first_access" "after accessing file2"
 dir2_stat_after_file_access=`_stat $TPATH/dir2`
 _compare_stat_times NNN "$dir2_stat_after_file_creation" \
-    "$dir2_stat_after_file_access" "for dir2 after file access"
+	"$dir2_stat_after_file_access" "for dir2 after file access"
 
 # Remounting with noatime option, creating a file and accessing it
 _scratch_unmount
 _scratch_mount "-o noatime"
 echo "ccc" > $TPATH/dir2/file3
 file3_stat_before_first_access=`_stat $TPATH/dir2/file3`
+sleep 1
 cat $TPATH/dir2/file3 > /dev/null
 file3_stat_after_first_access=`_stat $TPATH/dir2/file3`
 _compare_stat_times NNN "$file3_stat_before_first_access" \
-    "$file3_stat_after_first_access" "after accessing file3 first time"
+	"$file3_stat_after_first_access" "after accessing file3 first time"
 
 # Checking that the modify and change times are still updated
 file1_stat_before_modify=`_stat $TPATH/dir1/file1`
+sleep 1
 echo "xyz" > $TPATH/dir1/file1
 file1_stat_after_modify=`_stat $TPATH/dir1/file1`
 _compare_stat_times NYY "$file1_stat_before_modify" \
-    "$file1_stat_after_modify" "after modifying file1"
+	"$file1_stat_after_modify" "after modifying file1"
+sleep 1
 mv $TPATH/dir1/file1 $TPATH/dir1/file1_renamed
 file1_stat_after_change=`_stat $TPATH/dir1/file1_renamed`
 _compare_stat_times NNY "$file1_stat_after_modify" \
-    "$file1_stat_after_change" "after changing file1"
+	"$file1_stat_after_change" "after changing file1"
 
 # Remounting with strictatime option and
 # accessing a previously created file twice
@@ -155,30 +164,33 @@ _scratch_mount "-o strictatime"
 cat $TPATH/dir2/file3 > /dev/null
 file3_stat_after_second_access=`_stat $TPATH/dir2/file3`
 _compare_stat_times YNN "$file3_stat_after_first_access" \
-    "$file3_stat_after_second_access" "after accessing file3 second time"
+	"$file3_stat_after_second_access" "after accessing file3 second time"
+sleep 1
 cat $TPATH/dir2/file3 > /dev/null
 file3_stat_after_third_access=`_stat $TPATH/dir2/file3`
 _compare_stat_times YNN "$file3_stat_after_second_access" \
-    "$file3_stat_after_third_access" "after accessing file3 third time"
+	"$file3_stat_after_third_access" "after accessing file3 third time"
 
 # Btrfs only: Creating readonly snapshot. Access time should never
 # be updated, even when the strictatime mount option is active
 if [ "$FSTYP" = "btrfs" ]; then
-    SPATH=$SCRATCH_MNT/snap1
-    btrfs subvol snapshot -r $TPATH $SPATH >> $seqres.full
-    dir2_stat_readonly_before_access=`_stat $SPATH/dir2`
-    ls $SPATH/dir2 >> $seqres.full
-    cat $SPATH/dir2/file3 >> $seqres.full
-    dir2_stat_readonly_after_access=`_stat $SPATH/dir2`
-    _compare_stat_times NNN "$dir2_stat_readonly_before_access" \
-        "$dir2_stat_readonly_after_access" "for dir in readonly subvol"
-    file3_stat_readonly_after_access=`_stat $SPATH/dir2/file3`
-    _compare_stat_times NNN "$file3_stat_after_third_access" \
-        "$file3_stat_readonly_after_access" "for file in readonly subvol"
+	SPATH=$SCRATCH_MNT/snap1
+	btrfs subvol snapshot -r $TPATH $SPATH >> $seqres.full
+	dir2_stat_readonly_before_access=`_stat $SPATH/dir2`
+	sleep 1
+	ls $SPATH/dir2 >> $seqres.full
+	cat $SPATH/dir2/file3 >> $seqres.full
+	dir2_stat_readonly_after_access=`_stat $SPATH/dir2`
+	_compare_stat_times NNN "$dir2_stat_readonly_before_access" \
+		"$dir2_stat_readonly_after_access" "for dir in readonly subvol"
+	file3_stat_readonly_after_access=`_stat $SPATH/dir2/file3`
+	_compare_stat_times NNN "$file3_stat_after_third_access" \
+		"$file3_stat_readonly_after_access" "for file in readonly subvol"
 fi
 
 # Remounting read-only. Access time should never be updated, despite the
 # strictatime mount option.
+sleep 1
 dir2_stat_before_ro_mount=`_stat $TPATH/dir2`
 file3_stat_before_ro_mount=`_stat $TPATH/dir2/file3`
 _scratch_unmount
@@ -187,10 +199,10 @@ ls $TPATH/dir2 > /dev/null
 cat $TPATH/dir2/file3 > /dev/null
 dir2_stat_after_ro_mount=`_stat $TPATH/dir2`
 _compare_stat_times NNN "$dir2_stat_before_ro_mount" \
-    "$dir2_stat_after_ro_mount" "for dir in read-only filesystem"
+	"$dir2_stat_after_ro_mount" "for dir in read-only filesystem"
 file3_stat_after_ro_mount=`_stat $TPATH/dir2/file3`
 _compare_stat_times NNN "$file3_stat_before_ro_mount" \
-    "$file3_stat_after_ro_mount" "for file in read-only filesystem"
+	"$file3_stat_after_ro_mount" "for file in read-only filesystem"
 
 # success, all done
 _scratch_unmount
Signature · 6 lines
1.8.4.rc3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#7 lore raw

PATCH 4/6 shared/051: CRC enabled filesystems handle more than 25 ACLs

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

shared/051 fails on CRC enabled filesystems because it doesn't get an erro on setting the 26th ACL on a file. CRC enabled filesystems support more than 25, so there shoul dbe no error.

Allow the test to pass on both non-CRC and CRC enabled filesystems by detecting the type of filesystem configuration and using that information to fake the error message so that the golden output match does not fail. We also need to filter the 26th ACL from the output otherwise that will cause failures, too.

---
 tests/shared/051 | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
tests/shared/051 +16 -1
index e957f86..1d4edfb 100755
@@ -319,6 +319,14 @@ chacl a::rwx file1
 set +x
 
 #-------------------------------------------------------
+#
+# On CRC enabled filesystems, we can support many more than 25 ACLs, and
+# so we have to detect this case and "fake" the error if CRCs are enabled.
+if [ $FSTYP = "xfs" ]; then
+	xfs_info $TEST_DIR | _filter_mkfs >/dev/null 2> $tmp.info
+	cat $tmp.info >> $seqres.full
+	. $tmp.info
+fi
 
 echo ""
 echo "=== Test out large ACLs  ==="
@@ -343,7 +351,14 @@ getfacl --numeric largeaclfile | _filter_aces
 
 echo "1 above xfs acl max"
 chacl $acl3 largeaclfile
-getfacl --numeric largeaclfile | _filter_aces
+if [ $_fs_has_crcs = "1" ]; then
+	# fake the error message for CRC enabled filesystems. It should succeed,
+	# so if there was any error from the above command we are going to fail
+	# regardless of whether we emit this or not.
+	echo "chacl: cannot set access acl on \"largeaclfile\": Invalid argument"
+fi
+# and filter the extra ACL we might have added.
+getfacl --numeric largeaclfile | _filter_aces | sed -e "/^user:22:rwx/d"
 
 echo "use 16 aces"
 chacl $acl4 largeaclfile
Signature · 6 lines
1.8.4.rc3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#8 lore raw

Re: [PATCH 4/6] shared/051: CRC enabled filesystems handle more than 25 ACLs

Quoted 11 lines · Dave Chinner

From: Dave Chinner <[email protected]>

shared/051 fails on CRC enabled filesystems because it doesn't get an erro on setting the 26th ACL on a file. CRC enabled filesystems support more than 25, so there shoul dbe no error.

Allow the test to pass on both non-CRC and CRC enabled filesystems by detecting the type of filesystem configuration and using that information to fake the error message so that the golden output match does not fail. We also need to filter the 26th ACL from the output otherwise that will cause failures, too.

I don't really like this - we'd better of defining a function saying how many ACEs a filesystem supports and use that. Also I think the test for these large ACLs should be a separate test from the other tests currently in 051 as it's been causing a fair amount of issues in the past with various filesystems.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#9 lore raw

PATCH 5/6 xfs/217: prevent enospc failures on small test devices

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

I'm running xfstests against a ramdisk, so I'm limited in size of the test and scratch devices. While there are large enough to hold a filesystem image with a 2GB log, the way the log changes position in an image file as the size of the filesystem increases means that the aggregated disk space of xfs/217 is more than enough to run a 4GB TEST_DEV out of space and hence fail the test.

To avoid this problem, punch out the image file between every mkfs iteration so that it only consumes the space needed by each individual mkfs tests, not an aggregation of them all.

---
 tests/xfs/217 | 4 ++++
 1 file changed, 4 insertions(+)
tests/xfs/217 +4 -0
index 6c439c0..ab55a30 100755
@@ -63,6 +63,10 @@ _do_mkfs()
 		mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
 		umount -d $LOOP_MNT > /dev/null 2>&1
+
+		# punch out the previous blocks so that we keep the amount of
+		# disk space the test requires down to a minimum.
+		$XFS_IO_PROG -f -c "unresvsp 0 16383g" $LOOP_DEV
 	done
 }
 # make large holey file
Signature · 6 lines
1.8.4.rc3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#11 lore raw

PATCH 6/6 xfs/167: need at least 10GB of scratch space to run

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

When running on a ramdisk, the fsstress background workload consumes a GB of disk space every 5 seconds. This leads to the test failing with ENOSPC because the test file cannot be created due otthe background load cosuming it all. Hence don't run this test unless the scratch device is large enough not to hit ENOSPC conditions.

---
 tests/xfs/167 | 6 ++++++
 1 file changed, 6 insertions(+)
tests/xfs/167 +6 -0
index ab0156f..875bd31 100755
@@ -62,6 +62,12 @@ _require_scratch
 _scratch_mkfs_xfs >/dev/null 2>&1
 _scratch_mount
 
+# fast devices can consume disk space at a rate of 1GB every 5s via the
+# background workload. With 50 test loops, at 1 second per loop, that means we
+# need at least 10GB of disk space to ensure this test will not fail with ENOSPC
+# errors.
+_require_fs_space $SCRATCH_MNT 10485760
+
 TEST_FILE=$SCRATCH_MNT/test_file
 TEST_PROG=$here/src/unwritten_sync
 LOOPS=50
Signature · 6 lines
1.8.4.rc3

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs