xfstests: fixes for CRCs and small devices
cover xfstests: fixes for CRCs and small devices
Hi folks,
The following set of patches fix two sets of issues. The first is that small, fast filesystems like on a ramdisk trigger some interesting failures due to the rate of modification being so high on such devices. Hence we can do things like run a small filesystem out of space in a few seconds, whereas it woul dtake a few minutes on a SSD or spinning disk. Hence some tests require some small tweaks to avoid failures in such situations.
The second set of patches continue the work of fixing test failures on CRC enabled filesystems due to issues with the tests themselves. There are stll more test fixes needed, but I'm now down to single digit failure rates on CRC enabled filesystems with these patches.
Cheers,
Dave.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 1/6 xfs/033: add golden output for CRC enabled filesystems
From: Dave Chinner <[email protected]>
CRC enabled filesystems emit different errors on corruption. Specifically, inode corruption is picked up much earlier due to verifier failures (e.g. incorrect inode identifier) and so xfs_repair throws errors sufficiently different that filtering cannot hide the differences. Hence simply add a new golden output file and link it appropriately once we know what type of filesystem we are testing.
--- common/filter | 3 + tests/xfs/033 | 10 ++- tests/xfs/033.crc.out.linux | 197 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 207 insertions(+), 3 deletions(-) create mode 100644 tests/xfs/033.crc.out.linux
common/filter +3 -0
@@ -152,6 +152,9 @@ _filter_mkfs() if (/^\s+=\s+sectsz=(\d+)\s+attr=(\d+)/) { print STDERR "sectsz=$1\nattr=$2\n"; } + if (/^\s+=\s+crc=(\d)/) { + print STDERR "_fs_has_crcs=$1\n"; + } if (/^data\s+=\s+bsize=(\d+)\s+blocks=(\d+), imaxpct=(\d+)/) { print STDERR "dbsize=$1\ndblocks=$2\nimaxpct=$3\n"; print STDOUT "data = bsize=XXX blocks=XXX, imaxpct=PCT\n";
tests/xfs/033 +7 -3
@@ -45,9 +45,6 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 . ./common/filter . ./common/repair -# link correct .out file -_link_out_file $seq.out $seqfull.out - # nuke the root, rt bitmap, and rt summary inodes # _check_root_inos()
@@ -86,6 +83,13 @@ _scratch_mkfs_xfs | _filter_mkfs 2>$tmp.mkfs [ $isize -lt 512 ] && \ _scratch_mkfs_xfs -isize=512 | _filter_mkfs >/dev/null 2>&1 +# link correct .out file +if [ $_fs_has_crcs -eq 1 ]; then + _link_out_file $seq.crc.out $seqfull.out +else + _link_out_file $seq.out $seqfull.out +fi + `xfs_db -r -c sb -c p $SCRATCH_DEV | grep 'ino = ' | \ sed -e 's/ //g' -e 's/^/export /'`
tests/xfs/033.crc.out.linux new file +197 -0
197 lines not on this page · raw message
Signature · 6 lines
1.8.4.rc3 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 1/6] xfs/033: add golden output for CRC enabled filesystems
Looks good,
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 2/6 xfs/189: noattr2 invalid for CRC enabled filesystems
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Version 5 filesystems always have attr2 format enabled, and it cannot be turned off via the noattr2 mount option. As such, attempts to mount with noattr2 will be rejected and this causes cascading failures within the test.
Hence detect if we've created a CRC enabled filesystem, and if this is the case _notrun the test.
--- common/attr | 10 ++++++++++ tests/xfs/189 | 2 ++ 2 files changed, 12 insertions(+)
common/attr +10 -0
@@ -196,6 +196,16 @@ _require_attr_v1() || _notrun "attr v1 not supported on $SCRATCH_DEV" } +# check if we support the noattr2 mount option +_require_noattr2() +{ + _scratch_mkfs_xfs > /dev/null 2>&1 \ + || _fail "_scratch_mkfs_xfs failed on $SCRATCH_DEV" + _mount -o noattr2 $SCRATCH_DEV > /dev/null 2>&1 \ + || _notrun "noattr2 mount option not supported on $SCRATCH_DEV" + _scratch_unmount +} + # getfattr -R returns info in readdir order which varies from fs to fs. # This sorts the output by filename _sort_getfattr_output()
tests/xfs/189 +2 -0
@@ -230,6 +230,7 @@ _putback_scratch_fstab() # get standard environment, filters and checks . ./common/rc . ./common/filter +. ./common/attr # real QA test starts here _supported_fs xfs
@@ -237,6 +238,7 @@ _supported_os Linux _need_to_be_root _require_scratch +_require_noattr2 unset SCRATCH_RTDEV unset SCRATCH_LOGDEV
Signature · 6 lines
1.8.4.rc3 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 2/6] xfs/189: noattr2 invalid for CRC enabled filesystems
Looks good,
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 3/6 generic/003: ensure time changes between stat calls
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
On a fast device like a ramdisk, kernel time may not have changed between a stat of a file and some operation on it immediately afterwards. Hence there is no guarantee that an operation actually changes the timestamps of a file immediately after it is stat'd.
Hence, ensure that the times will change by sleeping for a second between the initial stat that reads the timestamps and the operations that is supposed to modify them. This way we ensure that the timestamp will change if the filesystem is correctly implemented.
While there, fix the indenting to be 8 space tabs and correct the header which is missing the bash shell declaration and the test number identifier.
--- tests/generic/003 | 110 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 61 insertions(+), 49 deletions(-)
tests/generic/003 +61 -49
@@ -1,3 +1,6 @@ +#! /bin/bash +# FS QA Test No. generic/003 +# # Tests the noatime, relatime, strictatime and nodiratime mount options. # There is an extra check for Btrfs to ensure that the access time is # never updated on read-only subvolumes. (Regression test for bug fixed
@@ -49,41 +52,40 @@ _require_relatime rm -f $seqres.full _stat() { - stat --printf="%x;%y;%z" $1 + stat --printf="%x;%y;%z" $1 } _compare_stat_times() { - updated=$1 # 3 chars indicating if access, modify and - # change times should be updated (Y) or not (N) - IFS=';' read -a first_stat <<< "$2" # Convert first stat to array - IFS=';' read -a second_stat <<< "$3" # Convert second stat to array - test_step=$4 # Will be printed to output stream in case of an - # error, to make debugging easier - types=( access modify change ) - - for i in 0 1 2; do - if [ "${first_stat[$i]}" == "${second_stat[$i]}" ]; then - if [ "${updated:$i:1}" == "Y" ]; then - echo -n "ERROR: ${types[$i]} time has not been updated " - echo $test_step - fi - else - if [ "${updated:$i:1}" == "N" ]; then - echo -n "ERROR: ${types[$i]} time has changed " - echo $test_step - fi - fi - done + updated=$1 # 3 chars indicating if access, modify and + # change times should be updated (Y) or not (N) + IFS=';' read -a first_stat <<< "$2" # Convert first stat to array + IFS=';' read -a second_stat <<< "$3" # Convert second stat to array + test_step=$4 # Will be printed to output stream in case of an + # error, to make debugging easier + types=( access modify change ) + + for i in 0 1 2; do + if [ "${first_stat[$i]}" == "${second_stat[$i]}" ]; then + if [ "${updated:$i:1}" == "N" ]; then + continue; + fi + echo -n "ERROR: ${types[$i]} time has not been updated " + echo $test_step + elif [ "${updated:$i:1}" == "N" ]; then + echo -n "ERROR: ${types[$i]} time has changed " + echo $test_step + fi + done } _scratch_mkfs >> $seqres.full 2>&1 || _fail "mkfs failed" _scratch_mount "-o relatime" if [ "$FSTYP" = "btrfs" ]; then - TPATH=$SCRATCH_MNT/sub1 - $BTRFS_UTIL_PROG subvolume create $TPATH > $seqres.full + TPATH=$SCRATCH_MNT/sub1 + $BTRFS_UTIL_PROG subvolume create $TPATH > $seqres.full else - TPATH=$SCRATCH_MNT + TPATH=$SCRATCH_MNT fi mkdir $TPATH/dir1
@@ -91,62 +93,69 @@ echo "aaa" > $TPATH/dir1/file1 file1_stat_before_first_access=`_stat $TPATH/dir1/file1` # Accessing file1 the first time +sleep 1 cat $TPATH/dir1/file1 > /dev/null file1_stat_after_first_access=`_stat $TPATH/dir1/file1` _compare_stat_times YNN "$file1_stat_before_first_access" \ - "$file1_stat_after_first_access" "after accessing file1 first time" + "$file1_stat_after_first_access" "after accessing file1 first time" # Accessing file1 a second time +sleep 1 cat $TPATH/dir1/file1 > /dev/null file1_stat_after_second_access=`_stat $TPATH/dir1/file1` _compare_stat_times NNN "$file1_stat_after_first_access" \ - "$file1_stat_after_second_access" "after accessing file1 second time" + "$file1_stat_after_second_access" "after accessing file1 second time" # Remounting with nodiratime option _scratch_unmount _scratch_mount "-o nodiratime" file1_stat_after_remount=`_stat $TPATH/dir1/file1` _compare_stat_times NNN "$file1_stat_after_second_access" \ - "$file1_stat_after_remount" "for file1 after remount" + "$file1_stat_after_remount" "for file1 after remount" # Creating dir2 and file2, checking directory stats mkdir $TPATH/dir2 dir2_stat_before_file_creation=`_stat $TPATH/dir2` +sleep 1 echo "bbb" > $TPATH/dir2/file2 dir2_stat_after_file_creation=`_stat $TPATH/dir2` _compare_stat_times NYY "$dir2_stat_before_file_creation" \ - "$dir2_stat_after_file_creation" "for dir2 after file creation" + "$dir2_stat_after_file_creation" "for dir2 after file creation" # Accessing file2 file2_stat_before_first_access=`_stat $TPATH/dir2/file2` +sleep 1 cat $TPATH/dir2/file2 > /dev/null file2_stat_after_first_access=`_stat $TPATH/dir2/file2` _compare_stat_times YNN "$file2_stat_before_first_access" \ - "$file2_stat_after_first_access" "after accessing file2" + "$file2_stat_after_first_access" "after accessing file2" dir2_stat_after_file_access=`_stat $TPATH/dir2` _compare_stat_times NNN "$dir2_stat_after_file_creation" \ - "$dir2_stat_after_file_access" "for dir2 after file access" + "$dir2_stat_after_file_access" "for dir2 after file access" # Remounting with noatime option, creating a file and accessing it _scratch_unmount _scratch_mount "-o noatime" echo "ccc" > $TPATH/dir2/file3 file3_stat_before_first_access=`_stat $TPATH/dir2/file3` +sleep 1 cat $TPATH/dir2/file3 > /dev/null file3_stat_after_first_access=`_stat $TPATH/dir2/file3` _compare_stat_times NNN "$file3_stat_before_first_access" \ - "$file3_stat_after_first_access" "after accessing file3 first time" + "$file3_stat_after_first_access" "after accessing file3 first time" # Checking that the modify and change times are still updated file1_stat_before_modify=`_stat $TPATH/dir1/file1` +sleep 1 echo "xyz" > $TPATH/dir1/file1 file1_stat_after_modify=`_stat $TPATH/dir1/file1` _compare_stat_times NYY "$file1_stat_before_modify" \ - "$file1_stat_after_modify" "after modifying file1" + "$file1_stat_after_modify" "after modifying file1" +sleep 1 mv $TPATH/dir1/file1 $TPATH/dir1/file1_renamed file1_stat_after_change=`_stat $TPATH/dir1/file1_renamed` _compare_stat_times NNY "$file1_stat_after_modify" \ - "$file1_stat_after_change" "after changing file1" + "$file1_stat_after_change" "after changing file1" # Remounting with strictatime option and # accessing a previously created file twice
@@ -155,30 +164,33 @@ _scratch_mount "-o strictatime" cat $TPATH/dir2/file3 > /dev/null file3_stat_after_second_access=`_stat $TPATH/dir2/file3` _compare_stat_times YNN "$file3_stat_after_first_access" \ - "$file3_stat_after_second_access" "after accessing file3 second time" + "$file3_stat_after_second_access" "after accessing file3 second time" +sleep 1 cat $TPATH/dir2/file3 > /dev/null file3_stat_after_third_access=`_stat $TPATH/dir2/file3` _compare_stat_times YNN "$file3_stat_after_second_access" \ - "$file3_stat_after_third_access" "after accessing file3 third time" + "$file3_stat_after_third_access" "after accessing file3 third time" # Btrfs only: Creating readonly snapshot. Access time should never # be updated, even when the strictatime mount option is active if [ "$FSTYP" = "btrfs" ]; then - SPATH=$SCRATCH_MNT/snap1 - btrfs subvol snapshot -r $TPATH $SPATH >> $seqres.full - dir2_stat_readonly_before_access=`_stat $SPATH/dir2` - ls $SPATH/dir2 >> $seqres.full - cat $SPATH/dir2/file3 >> $seqres.full - dir2_stat_readonly_after_access=`_stat $SPATH/dir2` - _compare_stat_times NNN "$dir2_stat_readonly_before_access" \ - "$dir2_stat_readonly_after_access" "for dir in readonly subvol" - file3_stat_readonly_after_access=`_stat $SPATH/dir2/file3` - _compare_stat_times NNN "$file3_stat_after_third_access" \ - "$file3_stat_readonly_after_access" "for file in readonly subvol" + SPATH=$SCRATCH_MNT/snap1 + btrfs subvol snapshot -r $TPATH $SPATH >> $seqres.full + dir2_stat_readonly_before_access=`_stat $SPATH/dir2` + sleep 1 + ls $SPATH/dir2 >> $seqres.full + cat $SPATH/dir2/file3 >> $seqres.full + dir2_stat_readonly_after_access=`_stat $SPATH/dir2` + _compare_stat_times NNN "$dir2_stat_readonly_before_access" \ + "$dir2_stat_readonly_after_access" "for dir in readonly subvol" + file3_stat_readonly_after_access=`_stat $SPATH/dir2/file3` + _compare_stat_times NNN "$file3_stat_after_third_access" \ + "$file3_stat_readonly_after_access" "for file in readonly subvol" fi # Remounting read-only. Access time should never be updated, despite the # strictatime mount option. +sleep 1 dir2_stat_before_ro_mount=`_stat $TPATH/dir2` file3_stat_before_ro_mount=`_stat $TPATH/dir2/file3` _scratch_unmount
@@ -187,10 +199,10 @@ ls $TPATH/dir2 > /dev/null cat $TPATH/dir2/file3 > /dev/null dir2_stat_after_ro_mount=`_stat $TPATH/dir2` _compare_stat_times NNN "$dir2_stat_before_ro_mount" \ - "$dir2_stat_after_ro_mount" "for dir in read-only filesystem" + "$dir2_stat_after_ro_mount" "for dir in read-only filesystem" file3_stat_after_ro_mount=`_stat $TPATH/dir2/file3` _compare_stat_times NNN "$file3_stat_before_ro_mount" \ - "$file3_stat_after_ro_mount" "for file in read-only filesystem" + "$file3_stat_after_ro_mount" "for file in read-only filesystem" # success, all done _scratch_unmount
Signature · 6 lines
1.8.4.rc3 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 3/6] generic/003: ensure time changes between stat calls
Looks good,
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 4/6 shared/051: CRC enabled filesystems handle more than 25 ACLs
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
shared/051 fails on CRC enabled filesystems because it doesn't get an erro on setting the 26th ACL on a file. CRC enabled filesystems support more than 25, so there shoul dbe no error.
Allow the test to pass on both non-CRC and CRC enabled filesystems by detecting the type of filesystem configuration and using that information to fake the error message so that the golden output match does not fail. We also need to filter the 26th ACL from the output otherwise that will cause failures, too.
--- tests/shared/051 | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
tests/shared/051 +16 -1
@@ -319,6 +319,14 @@ chacl a::rwx file1 set +x #------------------------------------------------------- +# +# On CRC enabled filesystems, we can support many more than 25 ACLs, and +# so we have to detect this case and "fake" the error if CRCs are enabled. +if [ $FSTYP = "xfs" ]; then + xfs_info $TEST_DIR | _filter_mkfs >/dev/null 2> $tmp.info + cat $tmp.info >> $seqres.full + . $tmp.info +fi echo "" echo "=== Test out large ACLs ==="
@@ -343,7 +351,14 @@ getfacl --numeric largeaclfile | _filter_aces echo "1 above xfs acl max" chacl $acl3 largeaclfile -getfacl --numeric largeaclfile | _filter_aces +if [ $_fs_has_crcs = "1" ]; then + # fake the error message for CRC enabled filesystems. It should succeed, + # so if there was any error from the above command we are going to fail + # regardless of whether we emit this or not. + echo "chacl: cannot set access acl on \"largeaclfile\": Invalid argument" +fi +# and filter the extra ACL we might have added. +getfacl --numeric largeaclfile | _filter_aces | sed -e "/^user:22:rwx/d" echo "use 16 aces" chacl $acl4 largeaclfile
Signature · 6 lines
1.8.4.rc3 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 4/6] shared/051: CRC enabled filesystems handle more than 25 ACLs
Quoted 11 lines · Dave Chinner
From: Dave Chinner <[email protected]>
shared/051 fails on CRC enabled filesystems because it doesn't get an erro on setting the 26th ACL on a file. CRC enabled filesystems support more than 25, so there shoul dbe no error.
Allow the test to pass on both non-CRC and CRC enabled filesystems by detecting the type of filesystem configuration and using that information to fake the error message so that the golden output match does not fail. We also need to filter the 26th ACL from the output otherwise that will cause failures, too.
I don't really like this - we'd better of defining a function saying how many ACEs a filesystem supports and use that. Also I think the test for these large ACLs should be a separate test from the other tests currently in 051 as it's been causing a fair amount of issues in the past with various filesystems.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 5/6 xfs/217: prevent enospc failures on small test devices
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
I'm running xfstests against a ramdisk, so I'm limited in size of the test and scratch devices. While there are large enough to hold a filesystem image with a 2GB log, the way the log changes position in an image file as the size of the filesystem increases means that the aggregated disk space of xfs/217 is more than enough to run a 4GB TEST_DEV out of space and hence fail the test.
To avoid this problem, punch out the image file between every mkfs iteration so that it only consumes the space needed by each individual mkfs tests, not an aggregation of them all.
--- tests/xfs/217 | 4 ++++ 1 file changed, 4 insertions(+)
tests/xfs/217 +4 -0
@@ -63,6 +63,10 @@ _do_mkfs() mount -o loop -t xfs $LOOP_DEV $LOOP_MNT echo "test write" > $LOOP_MNT/test umount -d $LOOP_MNT > /dev/null 2>&1 + + # punch out the previous blocks so that we keep the amount of + # disk space the test requires down to a minimum. + $XFS_IO_PROG -f -c "unresvsp 0 16383g" $LOOP_DEV done } # make large holey file
Signature · 6 lines
1.8.4.rc3 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 5/6] xfs/217: prevent enospc failures on small test devices
Looks good,
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 6/6 xfs/167: need at least 10GB of scratch space to run
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
When running on a ramdisk, the fsstress background workload consumes a GB of disk space every 5 seconds. This leads to the test failing with ENOSPC because the test file cannot be created due otthe background load cosuming it all. Hence don't run this test unless the scratch device is large enough not to hit ENOSPC conditions.
--- tests/xfs/167 | 6 ++++++ 1 file changed, 6 insertions(+)
tests/xfs/167 +6 -0
@@ -62,6 +62,12 @@ _require_scratch _scratch_mkfs_xfs >/dev/null 2>&1 _scratch_mount +# fast devices can consume disk space at a rate of 1GB every 5s via the +# background workload. With 50 test loops, at 1 second per loop, that means we +# need at least 10GB of disk space to ensure this test will not fail with ENOSPC +# errors. +_require_fs_space $SCRATCH_MNT 10485760 + TEST_FILE=$SCRATCH_MNT/test_file TEST_PROG=$here/src/unwritten_sync LOOPS=50
Signature · 6 lines
1.8.4.rc3 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 6/6] xfs/167: need at least 10GB of scratch space to run
Looks good,
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs