Skip to contentSkip to conversation

xfstests: various fixes

PATCH In review Dave Chinner 2 participants 13 messages May 1, 2013 – May 3, 2013

lore
Share & tools
lore
#0 lore raw

cover xfstests: various fixes

Hi folks,

These are various fixes for xfstests.

The first patch is an infrastructure regression fix, resulting from the patchset that split up the tests into subdirectories. It also moves all the check.* output files to the result directory rather than leaving them in the root directory of xfstests.

The second is a bunch of changes to generic/310 that should have been done in the review cycle. It now works reliably for me.

The last 3 patches fix bugs in tests that result in files being left in the root directory of xfstests.

Cheers,

Dave.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw

PATCH 1/5 xfstests: fix last test runtime output

From: Dave Chinner <[email protected]>

Prior to the test directory split-up, xfstests used to output the time it previously took to run a specific test in it's output. This was broken during the split up, as test identifiers changed and the result output changed.

To fix this, the search for previous test results needs to look at the sequence number rather than the absolute sequence for the test. The new output looks the same as the old functionality:

generic/001 4s ... 5s
generic/002 1s ... 0s
generic/005 1s ... 1s
generic/006 1s ... 1s
generic/007 2s ... 1s

Where the first column is the time of the previous test run, and the second column is the time that this run took.

Further, the check files used to generate this information are not being output properly in the result directory, and so various log files are not getting written to the correct location or file names. For example, the calls to _check_test_fs would output failures to ".full", while other messages would be output to check.full, but none would output to the corect location of RESULT_BASE/check.full.

Fix all this mess up so that all the check files for a specific run end up in RESULT_BASE, and ensure the timing code checks the right file and dumps output.

---
 check |   48 +++++++++++++++++++++++++++---------------------
 1 file changed, 27 insertions(+), 21 deletions(-)
check +27 -21
index 966fbe5..a79747e 100755
@@ -293,14 +293,17 @@ fi
 
 _wrapup()
 {
+    seq="check"
+    check="$RESULT_BASE/check"
+
     if $showme
     then
 	:
     elif $needwrap
     then
-	if [ -f check.time -a -f $tmp.time ]
+	if [ -f $check.time -a -f $tmp.time ]
 	then
-	    cat check.time $tmp.time \
+	    cat $check.time $tmp.time \
 	    | $AWK_PROG '
 	{ t[$1] = $2 }
 END	{ if (NR > 0) {
@@ -308,13 +311,13 @@ END	{ if (NR > 0) {
 	  }
 	}' \
 	    | sort -n >$tmp.out
-	    mv $tmp.out check.time
+	    mv $tmp.out $check.time
 	fi
 
-	echo "" >>check.log
-	date >>check.log
-	echo $list | fmt | sed -e 's/^/    /' -e "s;$SRC_DIR/;;g" >>check.log
-	$interrupt && echo "Interrupted!" >>check.log
+	echo "" >>$check.log
+	date >>$check.log
+	echo $list | fmt | sed -e 's/^/    /' -e "s;$SRC_DIR/;;g" >>$check.log
+	$interrupt && echo "Interrupted!" >>$check.log
         
         if [ ! -z "$n_try" -a $n_try != 0 ]
 	then
@@ -324,18 +327,18 @@ END	{ if (NR > 0) {
 	if [ ! -z "$notrun" ]
 	then
 	    echo "Not run:$notrun"
-	    echo "Not run:$notrun" >>check.log
+	    echo "Not run:$notrun" >>$check.log
 	fi
 
         if [ ! -z "$n_bad" -a $n_bad != 0 ]
 	then
 	    echo "Failures:$bad"
 	    echo "Failed $n_bad of $n_try tests"
-	    echo "Failures:$bad" | fmt >>check.log
-	    echo "Failed $n_bad of $n_try tests" >>check.log
+	    echo "Failures:$bad" | fmt >>$check.log
+	    echo "Failed $n_bad of $n_try tests" >>$check.log
 	else
 	    echo "Passed all $n_try tests"
-	    echo "Passed all $n_try tests" >>check.log
+	    echo "Passed all $n_try tests" >>$check.log
 	fi
 	needwrap=false
     fi
@@ -346,10 +349,19 @@ END	{ if (NR > 0) {
 
 trap "_wrapup; exit \$status" 0 1 2 3 15
 
+mkdir -p $RESULT_BASE
+if [ ! -d $RESULT_BASE ]; then
+	echo "failed to create results directory $RESULTS_BASE"
+	exit 1;
+fi
+
+seq="check"
+check="$RESULT_BASE/check"
+
 # don't leave old full output behind on a clean run
-rm -f check.full
+rm -f $check.full
 
-[ -f check.time ] || touch check.time
+[ -f $check.time ] || touch $check.time
 
 # print out our test configuration
 echo "FSTYP         -- `_full_fstyp_details`"
@@ -385,13 +397,7 @@ if [ ! -z "$SCRATCH_DEV" ]; then
   fi
 fi
 
-mkdir -p $RESULT_BASE
-if [ ! -d $RESULT_BASE ]; then
-	echo "failed to create results directory $RESULTS_BASE"
-	exit 1;
-fi
-
-seq="check"
+seqres="$check"
 _check_test_fs
 
 for seq in $list
@@ -432,7 +438,7 @@ do
 	fi
 
 	# slashes now in names, sed barfs on them so use grep
-	lasttime=`grep -w ^$seq check.time | awk '// {print $2}'`
+	lasttime=`grep -w ^$seqnum $check.time | awk '// {print $2}'`
 	if [ "X$lasttime" != X ]; then
 		echo -n " ${lasttime}s ..."
 	else
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw

PATCH 2/5 xfstests: 310 fails with existing directory error

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Test 310 fails with:

mkdir: cannot create directory `/mnt/test/tmp': File exists

$TEST_DIR is persistent, so test directories need to be created with "mkdir -p" so they don't fail if the directory already exists.

Many other things need fixing, too.
- Tests should define directories they use on $TEST_DIR by their sequence number, not generic names.

- Use a variable for the directory the test runs in ($SEQ_DIR, in this case) to avoid having to manually code it everywhere.

- New binaries need to be added to .gitignore.

- Return status for shell functions is 0 for success, non-zero for failure.

- Setting status=0 if there is no failure in the first test means that even if the second test fails, the test will still pass. Change the test to use "_fatal" when a kernel bug is detected, and only set status=0 when the entire test has finished.

- reduce the default runtime by to roughly a minute and scale it with the stress load factor variables. In most cases, this test is never going to hit problems (as they've already been fixed) so running it for ~4 minutes is mostly a waste of time...

---
 .gitignore        |    2 ++
 tests/generic/310 |   35 +++++++++++++++++------------------
 2 files changed, 19 insertions(+), 18 deletions(-)
.gitignore +2 -0
index 7a10feb..0bd48c3 100644
@@ -67,6 +67,8 @@
 /src/preallo_rw_pattern_writer
 /src/pwrite_mmap_blocked
 /src/randholes
+/src/t_readdir_1
+/src/t_readdir_2
 /src/rename
 /src/resvtest
 /src/runas
tests/generic/310 +17 -18
index b5316cd..26d2d4a 100755
@@ -75,12 +75,11 @@ check_kernel_bug()
 	new_warning=`dmesg | grep -c "^WARNING"`
 	new_lockdep=`dmesg | grep -c "possible recursive locking detected"`
 
-
 	# no kernel bug is detected
 	if [ $new_bug -eq $nr_bug -a $new_error -eq $nr_error -a \
 	     $new_null -eq $nr_null -a $new_warning -eq $nr_warning -a \
 	     $new_lockdep -eq $nr_lockdep ]; then
-		return 1
+		return 0
 	fi
 
 	nr_bug=$new_bug
@@ -88,37 +87,36 @@ check_kernel_bug()
 	nr_null=$new_null
 	nr_warning=$new_warning
 	nr_lockdep=$new_lockdep
+	return 1
 }
-mkdir $TEST_DIR/tmp
+
+RUN_TIME=$((30 * $TIME_FACTOR))
+
+SEQ_DIR=$TEST_DIR/$seq
+mkdir -p $SEQ_DIR
 for n in {1..4096}; do
-	touch $TEST_DIR/tmp/$n
+	touch $SEQ_DIR/$n
 done
 
 _test_read()
 {
-	src/t_readdir_1 $TEST_DIR/tmp &
-	sleep 100
+	src/t_readdir_1 $SEQ_DIR &
+	sleep $RUN_TIME
 	killall src/t_readdir_1
 	check_kernel_bug
-	if [ $? -eq 1 ]; then
-		status=0
-	else
-		echo "error: kernel bug was found, you can see the
-			dmesg for more infomation."
+	if [ $? -ne 0 ]; then
+		_fatal "kernel bug detected, check dmesg for more infomation."
 	fi
 }
 
 _test_lseek()
 {
-	src/t_readdir_2 $TEST_DIR/tmp &
-	sleep 100
+	src/t_readdir_2 $SEQ_DIR &
+	sleep $RUN_TIME
 	killall src/t_readdir_2
 	check_kernel_bug
-	if [ $? -eq 1 ]; then
-		status=0
-	else
-		echo "error: kernel bug was found, you can see the
-			dmesg for more infomation."
+	if [ $? -ne 0 ]; then
+		_fatal "kernel bug detected, check dmesg for more infomation."
 	fi
 }
 
@@ -127,4 +125,5 @@ _test_lseek
 
 # success, all done
 echo "*** done"
+status=0
 exit
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#5 lore raw

PATCH 3/5 xfstests: filter EA paths used by dump

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Test 063 fails because the diff output now has entire paths to the files in the results directory in it rather than just the file name. Add the results directory to the directory filter used by the dump tests to remove the path from the diff output.

---
 common/dump |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
common/dump +2 -1
index 73d0304..7fe50f6 100644
@@ -927,6 +927,7 @@ _dir_filter()
     -e "s#$restore_sdir#RESTORE_SUBDIR#g" \
     -e "s#$$#PID#g" \
     -e "/Only in SCRATCH_MNT: .use_space/d" \
+    -e "s#$RESULT_DIR/##g" \
 
 }
 
@@ -1358,7 +1359,7 @@ _diff_compare_eas()
     | sed -e "s#$restore_sdir\/##" \
     | tee $seqres.ea2 \
     | _dir_filter
-    diff -s $seqres.ea1 $seqres.ea2
+    diff -s $seqres.ea1 $seqres.ea2 | _dir_filter
 }
 
 
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#7 lore raw

PATCH 4/5 xfstests: fix incorrect redirect in generic/232

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

generic/232 attempts to direct output to tee, but instead of using a pipe it uses an append operator. Hence it leaves a file named "tee" in the root directory of the xfstests execution path. Just direct the output to the $seqres.full file rather than trying to tee it into the test output as well.

---
 tests/generic/232 |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/232 +1 -1
index 2402c45..8ada8aa 100755
@@ -55,7 +55,7 @@ _fsstress()
 	count=2000
 	args=`_scale_fsstress_args -d $out -n $count -p 7`
 
-	echo "fsstress $args" >> tee -a $seqres.full
+	echo "fsstress $args" >> $seqres.full
 	if ! $FSSTRESS_PROG $args | tee -a $seqres.full | _filter_num
 	then
 		echo "    fsstress $args returned $?"
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#9 lore raw

PATCH 5/5 xfstests: fix broken redirects in generic/131

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

generic/131 attempts to kill processes that may no longer exist when the test finishes and has a broken redirect for the error messages and they end up in a file nemaed "1" in the xfstests root instead of /dev/null.

Not only that, the attempts to redirect stderr to stdout in the middle of the test use incorrect redirect syntax, so they create an empty file named "1" in the xfstests root...

IOWs, all the redirects in the test are broken. Fix them and clean up the failure case to use the exit trap to trigger the cleanup function....

---
 tests/generic/131 |   18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)
tests/generic/131 +8 -10
index 95eb612..6c3131c 100755
@@ -33,8 +33,8 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 
 _cleanup()
 {
-    kill $locktest_pid2 2&>1 /dev/null
-    kill $locktest_pid1 2&>1 /dev/null
+    kill $locktest_pid2 > /dev/null 2>&1
+    kill $locktest_pid1 > /dev/null 2>&1
     _cleanup_testdir
 }
 
@@ -60,23 +60,21 @@ elif [ $$ -gt 32000 ]; then
 fi
 
 # Start the server
-src/locktest -p $PORT $TESTFILE > $testdir/server.out 2>&1 &
+src/locktest -p $PORT $TESTFILE 2>&1 > $testdir/server.out &
 locktest_pid1=$!
 
 sleep 1
 
 # Start the client
-src/locktest -p $PORT -h localhost $TESTFILE > $testdir/client.out 2>&1
+src/locktest -p $PORT -h localhost $TESTFILE 2>&1 > $testdir/client.out
 locktest_pid2=$!
 result=$?
 if [ $result -eq 0 ]; then
-    echo success!
+	echo success!
+	status=0
 else
-    echo "Client reported failure ($result)"
-    cat $testdir/*.out
-    _cleanup
-    exit $status
+	echo "Client reported failure ($result)"
+	cat $testdir/*.out
 fi
 
-status=0
 exit
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#11 lore raw
In reply to Dave Chinner
Quoted 9 lines · Dave Chinner

Hi folks,

These are various fixes for xfstests.

The first patch is an infrastructure regression fix, resulting from the patchset that split up the tests into subdirectories. It also moves all the check.* output files to the result directory rather than leaving them in the root directory of xfstests.

Quoted 5 lines

The second is a bunch of changes to generic/310 that should have been done in the review cycle. It now works reliably for me.

The last 3 patches fix bugs in tests that result in files being left in the root directory of xfstests.

FWIW, with the V2 xfsprogs kernel sync tarball I've posted and these fixes to xfstests, I'm down to only 3 failures in xfstests for 4k block size filesystem tests:

Failures: generic/233 shared/298 xfs/296

generic/233 appears to be a small quota accounting mismatch, probably related to speculative preallocation. Not a major issue.

shared/298 is failing due to the ENOSPC problem Eric posted patches to fix. Test bug.

xfs/296 is failing due to an unresolved xfsdump issue w.r.t restoring security attributes. Unresolved.

So, I'm much happier now about the state of xfsprogs and xfstests than I was this morning. I'll be happier still when I fix the fsx failures on 512 byte block size filesystems, because that will bring my 512 byte block size filesystem tests to close to the same level. That's for tomorrow, though.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#12 lore raw

Thanks Dave, this patchset has been committed.

--Rich

commit 87f5e8e2dbf02b3c4d804a797ed5a7a643f4b64e

Author: Dave Chinner <[email protected]>
Date:   Wed May 1 08:31:59 2013 +0000

xfstests: fix last test runtime output

commit 97540f14f71e6785838532a982c443571380cfcc

Author: Dave Chinner <[email protected]>
Date:   Wed May 1 08:32:00 2013 +0000

xfstests: 310 fails with existing directory error

commit 3ac75605de1fbbaae9505799b46c44fbe053f04e

Author: Dave Chinner <[email protected]>
Date:   Wed May 1 08:32:01 2013 +0000

xfstests: filter EA paths used by dump

commit e6c6abd6dcacafa778d610451a023ec93c1b352e

Author: Dave Chinner <[email protected]>
Date:   Wed May 1 08:32:02 2013 +0000

xfstests: fix incorrect redirect in generic/232

commit 627427bb1bb4007c661a178bbeb4ea0bd0167c6f

Author: Dave Chinner <[email protected]>
Date:   Wed May 1 08:32:03 2013 +0000

xfstests: fix broken redirects in generic/131

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs