Skip to contentSkip to conversation

check: annotate good and expunged tests in results

PATCH In review [email protected] 3 participants 11 messages Apr 12, 2018 – Feb 14, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 check: annotate good and expunged tests in results

From: Jeff Mahoney <[email protected]>

Currently, we only create results files when a test has failed or was supposed to run but some dependency wasn't met causing it not to run. Short of saving the summary at the end of the run, there's no way to tell which tests passed or which tests weren't run due to being excluded.

This patch moves successful test results to $seqres.out.good to annotate good results. It also adds tests excluded by group to the $tmp.xlist file and adds annotations for every test excluded. When a test is expunged during execution, an expunged message will be issued and a $seqres.expunged file will be created, both containing the reason for the test being expunged. Reasons can be "command line", "file $filename", or "group <group> [group...]".

This makes the output more noisy in the expunged case and makes startup take slightly longer, but ends up with results that can be more easily parsed by automated tools.

---
 check | 57 +++++++++++++++++++++++++++++++--------------------------
 1 file changed, 31 insertions(+), 26 deletions(-)
check +31 -26
index 546683c5..9a1f4e7e 100755
@@ -173,28 +173,18 @@ get_all_tests()
 	done
 }
 
-# takes the list of tests to run in $tmp.list, and removes the tests passed to
-# the function from that list.
+# takes the list of tests to run in $tmp.list and adds them to the excluded
+# test list, annotated with the group that excluded each one
 trim_test_list()
 {
+	group=$1
+	shift
 	test_list="$*"
 
-	rm -f $tmp.grep
-	numsed=0
-	for t in $test_list
-	do
-	    if [ $numsed -gt 100 ]; then
-		grep -v -f $tmp.grep <$tmp.list >$tmp.tmp
-		mv $tmp.tmp $tmp.list
-		numsed=0
-		rm -f $tmp.grep
-	    fi
-	    echo "^$t\$" >>$tmp.grep
-	    numsed=`expr $numsed + 1`
+	for test in $test_list; do
+		test=${test##tests/}
+		echo "$test group $group" >> $tmp.xlist
 	done
-	grep -v -f $tmp.grep <$tmp.list >$tmp.tmp
-	mv $tmp.tmp $tmp.list
-	rm -f $tmp.grep
 }
 
 
@@ -246,7 +236,7 @@ _prepare_test_list()
 			exit 1
 		fi
 
-		trim_test_list $list
+		trim_test_list $xgroup $list
 	done
 
 	# sort the list of tests into numeric order
@@ -285,13 +275,15 @@ while [ $# -gt 0 ]; do
 		for d in $SRC_GROUPS $FSTYP; do
 			[ -f $SRC_DIR/$d/$xfile ] || continue
 			for f in `sed "s/#.*$//" $SRC_DIR/$d/$xfile`; do
-				echo $d/$f >> $tmp.xlist
+				echo "$d/$f command line" >> $tmp.xlist
 			done
 		done
 		;;
 	-E)	xfile=$2; shift ;
 		if [ -f $xfile ]; then
-			sed "s/#.*$//" "$xfile" >> $tmp.xlist
+			sed -e "s/#.*$//" \
+			    -e "s;$; file $xfile;" "$xfile" \
+			    >> $tmp.xlist
 	        fi
 		;;
 	-s)	RUN_SECTION="$RUN_SECTION $2"; shift ;;
@@ -491,11 +483,17 @@ _check_filesystems()
 _expunge_test()
 {
 	local TEST_ID="$1"
+	local OUTPUT="$2"
 	if [ -s $tmp.xlist ]; then
-		if grep -q $TEST_ID $tmp.xlist; then
-			echo "       [expunged]"
+		grep $TEST_ID $tmp.xlist > $tmp._expunge_test
+		if [ $? -eq 0 ]; then
+			sed -e "s;$TEST_ID ;;" $tmp._expunge_test | \
+				tr '\n' ' ' |  sed -e 's; group;;g'
+			echo
+			rm -f $tmp._expunge_test
 			return 1
 		fi
+		rm -f $tmp._expunge_test
 	fi
 	return 0
 }
@@ -670,8 +668,10 @@ for section in $HOST_OPTIONS_SECTIONS; do
 	    echo -n "$seqnum"
 
 	    if $showme; then
-		_expunge_test $seqnum
+		_expunge_test $seqnum > $tmp.xreason
 		if [ $? -eq 1 ]; then
+			echo -n "        [expunged] "
+			cat $tmp.xreason
 			continue
 		fi
 		echo
@@ -689,11 +689,15 @@ for section in $HOST_OPTIONS_SECTIONS; do
 	    else
 		# really going to try and run this one
 		#
-		rm -f $seqres.out.bad
+		rm -f $seqres.out.bad $seqres.out.good $seqres.expunged
+		rm -f $seqres.notrun
 
 		# check if we really should run it
-		_expunge_test $seqnum
+		_expunge_test $seqnum > $tmp.xreason
 		if [ $? -eq 1 ]; then
+			mv $tmp.xreason $seqres.expunged
+			echo -n "        [expunged] "
+			cat $seqres.expunged
 			continue
 		fi
 
@@ -704,7 +708,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
 		else
 			echo -n "	"	# prettier output with timestamps.
 		fi
-		rm -f core $seqres.notrun
+		rm -f core
 
 		start=`_wallclock`
 		$timestamp && echo -n "	["`date "+%T"`"]"
@@ -770,6 +774,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
 			    else
 				echo "$seqnum `expr $stop - $start`" >>$tmp.time
 				echo -n " `expr $stop - $start`s"
+				mv $tmp.out $seqres.out.good
 			    fi
 			    echo ""
 			else
Signature · 1 line
2.15.1
#1 lore raw
Quoted 1 line · On Thu, Apr 12, 2018 at 05:38:38PM -0400, [email protected] wrote:

From: Jeff Mahoney <[email protected]>

[Sorry for the late review..]

Quoted 9 lines

Currently, we only create results files when a test has failed or was supposed to run but some dependency wasn't met causing it not to run. Short of saving the summary at the end of the run, there's no way to tell which tests passed or which tests weren't run due to being excluded.

This patch moves successful test results to $seqres.out.good to annotate good results. It also adds tests excluded by group to the $tmp.xlist file and adds annotations for every test excluded. When a test is expunged

I can see that annotating tests excluded by group is helpful, but I don't think we need $seqres.out.good, passed tests are already shown by check, and $seqres.out.good is same as the golden image, it looks redundant.

Quoted 19 lines

during execution, an expunged message will be issued and a $seqres.expunged file will be created, both containing the reason for the test being expunged. Reasons can be "command line", "file $filename", or "group <group> [group...]".

This makes the output more noisy in the expunged case and makes startup take slightly longer, but ends up with results that can be more easily parsed by automated tools.

---
 check | 57 +++++++++++++++++++++++++++++++--------------------------
 1 file changed, 31 insertions(+), 26 deletions(-)
check +0 -0
index 546683c5..9a1f4e7e 100755

check get_all_tests() · line 177

Quoted 7 lines
check +2 -2
@@ -173,28 +173,18 @@ get_all_tests()
 	done
 }
 
-# takes the list of tests to run in $tmp.list, and removes the tests passed to
-# the function from that list.
+# takes the list of tests to run in $tmp.list and adds them to the excluded
+# test list, annotated with the group that excluded each one

The comments don't seem quite right, the new behavior doesn't take tests from $tmp.list, it just edits the $tmp.xlist file for later use.

check get_all_tests() · line 180

Quoted 3 lines
check +1 -0
 trim_test_list()
 {
+	group=$1

Please declare local variable as 'local' when possible.

Quoted 39 lines
check +5 -17
+	shift
 	test_list="$*"
 
-	rm -f $tmp.grep
-	numsed=0
-	for t in $test_list
-	do
-	    if [ $numsed -gt 100 ]; then
-		grep -v -f $tmp.grep <$tmp.list >$tmp.tmp
-		mv $tmp.tmp $tmp.list
-		numsed=0
-		rm -f $tmp.grep
-	    fi
-	    echo "^$t\$" >>$tmp.grep
-	    numsed=`expr $numsed + 1`
+	for test in $test_list; do
+		test=${test##tests/}
+		echo "$test group $group" >> $tmp.xlist
 	done
-	grep -v -f $tmp.grep <$tmp.list >$tmp.tmp
-	mv $tmp.tmp $tmp.list
-	rm -f $tmp.grep
 }
 
 
@@ -246,7 +236,7 @@ _prepare_test_list()
 			exit 1
 		fi
 
-		trim_test_list $list
+		trim_test_list $xgroup $list
 	done
 
 	# sort the list of tests into numeric order
@@ -285,13 +275,15 @@ while [ $# -gt 0 ]; do
 		for d in $SRC_GROUPS $FSTYP; do
 			[ -f $SRC_DIR/$d/$xfile ] || continue
 			for f in `sed "s/#.*$//" $SRC_DIR/$d/$xfile`; do
-				echo $d/$f >> $tmp.xlist

check while [ $# -gt 0 ]; do · line 286

Quoted 10 lines
check +4 -1
+				echo "$d/$f command line" >> $tmp.xlist
 			done
 		done
 		;;
 	-E)	xfile=$2; shift ;
 		if [ -f $xfile ]; then
-			sed "s/#.*$//" "$xfile" >> $tmp.xlist
+			sed -e "s/#.*$//" \
+			    -e "s;$; file $xfile;" "$xfile" \
+			    >> $tmp.xlist

When using config section, e.g. ./check -s <sec_name>, and FSTYP is different in new section, check will do _prepare_test_list again, so tests selected by "-x <xgroup>" will be listed twice in $tmp.xlist file, as a result, I see duplicated group names in the annotate message, e.g.

generic/458 [expunged] group metadata metadata

I think we should move above "-X/-E" handlings to _prepare_test_list, only assign variable "xfile" here (use a new variable for -E option, perhaps). So that we could simply remove $tmp.xlist before calling _prepare_test_list again, and it could exclude all specified tests only once. Maybe as patch 1/2 in a patchset.

check _check_filesystems() · line 486

Quoted 8 lines
check +1 -0
 	        fi
 		;;
 	-s)	RUN_SECTION="$RUN_SECTION $2"; shift ;;
@@ -491,11 +483,17 @@ _check_filesystems()
 _expunge_test()
 {
 	local TEST_ID="$1"
+	local OUTPUT="$2"

OUTPUT is not used.

Quoted 55 lines
check +17 -6
 	if [ -s $tmp.xlist ]; then
-		if grep -q $TEST_ID $tmp.xlist; then
-			echo "       [expunged]"
+		grep $TEST_ID $tmp.xlist > $tmp._expunge_test
+		if [ $? -eq 0 ]; then
+			sed -e "s;$TEST_ID ;;" $tmp._expunge_test | \
+				tr '\n' ' ' |  sed -e 's; group;;g'
+			echo
+			rm -f $tmp._expunge_test
 			return 1
 		fi
+		rm -f $tmp._expunge_test
 	fi
 	return 0
 }
@@ -670,8 +668,10 @@ for section in $HOST_OPTIONS_SECTIONS; do
 	    echo -n "$seqnum"
 
 	    if $showme; then
-		_expunge_test $seqnum
+		_expunge_test $seqnum > $tmp.xreason
 		if [ $? -eq 1 ]; then
+			echo -n "        [expunged] "
+			cat $tmp.xreason
 			continue
 		fi
 		echo
@@ -689,11 +689,15 @@ for section in $HOST_OPTIONS_SECTIONS; do
 	    else
 		# really going to try and run this one
 		#
-		rm -f $seqres.out.bad
+		rm -f $seqres.out.bad $seqres.out.good $seqres.expunged
+		rm -f $seqres.notrun
 
 		# check if we really should run it
-		_expunge_test $seqnum
+		_expunge_test $seqnum > $tmp.xreason
 		if [ $? -eq 1 ]; then
+			mv $tmp.xreason $seqres.expunged
+			echo -n "        [expunged] "
+			cat $seqres.expunged
 			continue
 		fi
 
@@ -704,7 +708,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
 		else
 			echo -n "	"	# prettier output with timestamps.
 		fi
-		rm -f core $seqres.notrun
+		rm -f core
 
 		start=`_wallclock`
 		$timestamp && echo -n "	["`date "+%T"`"]"

check for section in $HOST_OPTIONS_SECTIONS; do · line 777

Quoted 4 lines
check +1 -0
@@ -770,6 +774,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
 			    else
 				echo "$seqnum `expr $stop - $start`" >>$tmp.time
 				echo -n " `expr $stop - $start`s"
+				mv $tmp.out $seqres.out.good

As said above, I don't think it's that useful.

Thanks,
Eryu

Quoted 10 lines
check +0 -0
 			    fi
 			    echo ""
 			else
Signature · 6 lines
2.15.1

--
To unsubscribe from this list: send the line "unsubscribe fstests" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#2 lore raw
Quoted 4 lines · Eryu Guan
Quoted 1 line · On Thu, Apr 12, 2018 at 05:38:38PM -0400, [email protected] wrote:

From: Jeff Mahoney <[email protected]>

[Sorry for the late review..]

Thanks for the review just the same.

Quoted 5 lines
Quoted 5 lines

Currently, we only create results files when a test has failed or was supposed to run but some dependency wasn't met causing it not to run. Short of saving the summary at the end of the run, there's no way to tell which tests passed or which tests weren't run due to being excluded.

Quoted 8 lines
Quoted 3 lines

This patch moves successful test results to $seqres.out.good to annotate good results. It also adds tests excluded by group to the $tmp.xlist file and adds annotations for every test excluded. When a test is expunged

I can see that annotating tests excluded by group is helpful, but I don't think we need $seqres.out.good, passed tests are already shown by check, and $seqres.out.good is same as the golden image, it looks redundant.

The contents are redundant, but the purpose of the file existing isn't. My goal is to be able to point a script at the results directory for a test run and autogenerate a table of results and/or compare them to known results. Right now, there's no way to tell whether a test was run at all, only that it failed or was skipped due to some missing dependency. The expunge changes handle the explicitly skipped cases, but without the .good files, there's nothing indicating a test actually ran or when it last did. The contents don't have to be the output. If you can think of something more useful there (or if we just create an empty file), that's fine too.

Quoted 7 lines

check get_all_tests() · line 177

Quoted 4 lines
check +2 -2
-# takes the list of tests to run in $tmp.list, and removes the tests passed to
-# the function from that list.
+# takes the list of tests to run in $tmp.list and adds them to the excluded
+# test list, annotated with the group that excluded each one

The comments don't seem quite right, the new behavior doesn't take tests from $tmp.list, it just edits the $tmp.xlist file for later use.

Right. I missed updating the comment after changing the behavior.

Quoted 5 lines

check get_all_tests() · line 180

Quoted 3 lines

trim_test_list()
{

check +1 -0
+	group=$1

Please declare local variable as 'local' when possible.

Ok.

Quoted 5 lines

When using config section, e.g. ./check -s <sec_name>, and FSTYP is different in new section, check will do _prepare_test_list again, so tests selected by "-x <xgroup>" will be listed twice in $tmp.xlist file, as a result, I see duplicated group names in the annotate message, e.g.

Quoted 7 lines

generic/458 [expunged] group metadata metadata

I think we should move above "-X/-E" handlings to _prepare_test_list, only assign variable "xfile" here (use a new variable for -E option, perhaps). So that we could simply remove $tmp.xlist before calling _prepare_test_list again, and it could exclude all specified tests only once. Maybe as patch 1/2 in a patchset.

Ok, that's easy enough to do.

Quoted 10 lines

check _check_filesystems() · line 486

Quoted 8 lines
check +1 -0
 	        fi
 		;;
 	-s)	RUN_SECTION="$RUN_SECTION $2"; shift ;;
@@ -491,11 +483,17 @@ _check_filesystems()
 _expunge_test()
 {
 	local TEST_ID="$1"
+	local OUTPUT="$2"

OUTPUT is not used.

Ack.

Quoted 7 lines

check for section in $HOST_OPTIONS_SECTIONS; do · line 777

Quoted 5 lines
check +1 -0
@@ -770,6 +774,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
 			    else
 				echo "$seqnum `expr $stop - $start`" >>$tmp.time
 				echo -n " `expr $stop - $start`s"
+				mv $tmp.out $seqres.out.good

As said above, I don't think it's that useful.

It doesn't need to be the contents of the output, but I'd like it to be *something* and moving a file that already exists is as easy as creating a new one.

-Jeff

Signature · 2 lines
Jeff Mahoney
SUSE Labs
#3 lore raw
Quoted 22 lines · Jeff Mahoney
Quoted 4 lines · Eryu Guan
Quoted 1 line · On Thu, Apr 12, 2018 at 05:38:38PM -0400, [email protected] wrote:

From: Jeff Mahoney <[email protected]>

[Sorry for the late review..]

Thanks for the review just the same.

Quoted 5 lines
Quoted 5 lines

Currently, we only create results files when a test has failed or was supposed to run but some dependency wasn't met causing it not to run. Short of saving the summary at the end of the run, there's no way to tell which tests passed or which tests weren't run due to being excluded.

Quoted 8 lines
Quoted 3 lines

This patch moves successful test results to $seqres.out.good to annotate good results. It also adds tests excluded by group to the $tmp.xlist file and adds annotations for every test excluded. When a test is expunged

I can see that annotating tests excluded by group is helpful, but I don't think we need $seqres.out.good, passed tests are already shown by check, and $seqres.out.good is same as the golden image, it looks redundant.

Quoted 4 lines

The contents are redundant, but the purpose of the file existing isn't. My goal is to be able to point a script at the results directory for a test run and autogenerate a table of results and/or compare them to known results. Right now, there's no way to tell whether a test was run

But the test run & pass info is already available from the check output and the test result summary at the end of check. Is that sufficient for you? Also, we already have mechanism to generate a test report in xunit format, i.e. ./check -R xunit -g auto, which includes results for passed & failed & notrun tests.

My concern is that saving .good files consume more and more space over time, and they don't provide any additional information (even with empty files that don't require much space).

Thanks,
Eryu

Quoted 69 lines

at all, only that it failed or was skipped due to some missing dependency. The expunge changes handle the explicitly skipped cases, but without the .good files, there's nothing indicating a test actually ran or when it last did. The contents don't have to be the output. If you can think of something more useful there (or if we just create an empty file), that's fine too.

Quoted 7 lines

check get_all_tests() · line 177

Quoted 4 lines
check +2 -2
-# takes the list of tests to run in $tmp.list, and removes the tests passed to
-# the function from that list.
+# takes the list of tests to run in $tmp.list and adds them to the excluded
+# test list, annotated with the group that excluded each one

The comments don't seem quite right, the new behavior doesn't take tests from $tmp.list, it just edits the $tmp.xlist file for later use.

Right. I missed updating the comment after changing the behavior.

Quoted 5 lines

check get_all_tests() · line 180

Quoted 3 lines

trim_test_list()
{

check +1 -0
+	group=$1

Please declare local variable as 'local' when possible.

Ok.

Quoted 5 lines

When using config section, e.g. ./check -s <sec_name>, and FSTYP is different in new section, check will do _prepare_test_list again, so tests selected by "-x <xgroup>" will be listed twice in $tmp.xlist file, as a result, I see duplicated group names in the annotate message, e.g.

Quoted 7 lines

generic/458 [expunged] group metadata metadata

I think we should move above "-X/-E" handlings to _prepare_test_list, only assign variable "xfile" here (use a new variable for -E option, perhaps). So that we could simply remove $tmp.xlist before calling _prepare_test_list again, and it could exclude all specified tests only once. Maybe as patch 1/2 in a patchset.

Ok, that's easy enough to do.

Quoted 10 lines

check _check_filesystems() · line 486

Quoted 8 lines
check +1 -0
 	        fi
 		;;
 	-s)	RUN_SECTION="$RUN_SECTION $2"; shift ;;
@@ -491,11 +483,17 @@ _check_filesystems()
 _expunge_test()
 {
 	local TEST_ID="$1"
+	local OUTPUT="$2"

OUTPUT is not used.

Ack.

Quoted 7 lines

check for section in $HOST_OPTIONS_SECTIONS; do · line 777

Quoted 5 lines
check +1 -0
@@ -770,6 +774,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
 			    else
 				echo "$seqnum `expr $stop - $start`" >>$tmp.time
 				echo -n " `expr $stop - $start`s"
+				mv $tmp.out $seqres.out.good

As said above, I don't think it's that useful.

It doesn't need to be the contents of the output, but I'd like it to be *something* and moving a file that already exists is as easy as creating a new one.

-Jeff

Signature · 2 lines
Jeff Mahoney
SUSE Labs
#4 lore raw
Quoted 6 lines · Eryu Guan

But the test run & pass info is already available from the check output and the test result summary at the end of check. Is that sufficient for you? Also, we already have mechanism to generate a test report in xunit format, i.e. ./check -R xunit -g auto, which includes results for passed & failed & notrun tests.

Do we have a way to parse the results *after* a run? For instance, if you forgot -R xunit ?

Luis

#5 lore raw
Quoted 10 lines · Luis R. Rodriguez
Quoted 6 lines · Eryu Guan

But the test run & pass info is already available from the check output and the test result summary at the end of check. Is that sufficient for you? Also, we already have mechanism to generate a test report in xunit format, i.e. ./check -R xunit -g auto, which includes results for passed & failed & notrun tests.

Do we have a way to parse the results *after* a run? For instance, if you forgot -R xunit ?

There's a tools/compare-failures script that takes the outputs of check as inputs and compares the results. But, TBH, I never run it after reviewing it.. Perhaps it could be enhanced somehow.

Thanks,
Eryu

#6 lore raw
Quoted 9 lines · Eryu Guan
Quoted 8 lines · Luis R. Rodriguez
Quoted 6 lines · Eryu Guan

But the test run & pass info is already available from the check output and the test result summary at the end of check. Is that sufficient for you? Also, we already have mechanism to generate a test report in xunit format, i.e. ./check -R xunit -g auto, which includes results for passed & failed & notrun tests.

Quoted 6 lines
Quoted 2 lines

Do we have a way to parse the results *after* a run? For instance, if you forgot -R xunit ?

There's a tools/compare-failures script that takes the outputs of check as inputs and compares the results. But, TBH, I never run it after reviewing it.. Perhaps it could be enhanced somehow.

Yeah, that takes check output. Without capturing it at runtime, it can't compare anything.

The XML report may do most of what I want provided we can enable it by default and write it someplace safe rather than clean it up automatically when the test run is interrupted.

I already have test code that extends it to output expunged tests and to add an explicit <pass/> element. It saves the timestamps already, so that's a plus.

-Jeff

Signature · 2 lines
Jeff Mahoney
SUSE Labs
#7 lore raw
Quoted 20 lines · Jeff Mahoney
Quoted 9 lines · Eryu Guan
Quoted 8 lines · Luis R. Rodriguez
Quoted 6 lines · Eryu Guan

But the test run & pass info is already available from the check output and the test result summary at the end of check. Is that sufficient for you? Also, we already have mechanism to generate a test report in xunit format, i.e. ./check -R xunit -g auto, which includes results for passed & failed & notrun tests.

Quoted 6 lines
Quoted 2 lines

Do we have a way to parse the results *after* a run? For instance, if you forgot -R xunit ?

There's a tools/compare-failures script that takes the outputs of check as inputs and compares the results. But, TBH, I never run it after reviewing it.. Perhaps it could be enhanced somehow.

Yeah, that takes check output. Without capturing it at runtime, it can't compare anything.

Quoted 7 lines

The XML report may do most of what I want provided we can enable it by default and write it someplace safe rather than clean it up automatically when the test run is interrupted.

I already have test code that extends it to output expunged tests and to add an explicit <pass/> element. It saves the timestamps already, so that's a plus.

And shortly after writing this, I had a test run hang. Since it hung, even writing it on exit wouldn't have worked. There's not enough information leftover to generate it. I think I'd still like the results directory to contain the information required to generate it as a post-mortem.

-Jeff

Signature · 2 lines
Jeff Mahoney
SUSE Labs
#8 lore raw
Quoted 29 lines · Jeff Mahoney
Quoted 20 lines · Jeff Mahoney
Quoted 9 lines · Eryu Guan
Quoted 8 lines · Luis R. Rodriguez
Quoted 6 lines · Eryu Guan

But the test run & pass info is already available from the check output and the test result summary at the end of check. Is that sufficient for you? Also, we already have mechanism to generate a test report in xunit format, i.e. ./check -R xunit -g auto, which includes results for passed & failed & notrun tests.

Quoted 6 lines
Quoted 2 lines

Do we have a way to parse the results *after* a run? For instance, if you forgot -R xunit ?

There's a tools/compare-failures script that takes the outputs of check as inputs and compares the results. But, TBH, I never run it after reviewing it.. Perhaps it could be enhanced somehow.

Yeah, that takes check output. Without capturing it at runtime, it can't compare anything.

Quoted 7 lines

The XML report may do most of what I want provided we can enable it by default and write it someplace safe rather than clean it up automatically when the test run is interrupted.

I already have test code that extends it to output expunged tests and to add an explicit <pass/> element. It saves the timestamps already, so that's a plus.

Quoted 5 lines

And shortly after writing this, I had a test run hang. Since it hung, even writing it on exit wouldn't have worked. There's not enough information leftover to generate it. I think I'd still like the results directory to contain the information required to generate it as a post-mortem.

After thinking on it a bit more, since you object to writing a bunch of files by default, we could accomplish the same goal by adding a "files" report type that does this without dropping files for everyone. I'm working that up now.

-Jeff

Signature · 2 lines
Jeff Mahoney
SUSE Labs
#9 lore raw
Quoted 37 lines · Jeff Mahoney
Quoted 29 lines · Jeff Mahoney
Quoted 20 lines · Jeff Mahoney
Quoted 9 lines · Eryu Guan
Quoted 8 lines · Luis R. Rodriguez
Quoted 6 lines · Eryu Guan

But the test run & pass info is already available from the check output and the test result summary at the end of check. Is that sufficient for you? Also, we already have mechanism to generate a test report in xunit format, i.e. ./check -R xunit -g auto, which includes results for passed & failed & notrun tests.

Quoted 6 lines
Quoted 2 lines

Do we have a way to parse the results *after* a run? For instance, if you forgot -R xunit ?

There's a tools/compare-failures script that takes the outputs of check as inputs and compares the results. But, TBH, I never run it after reviewing it.. Perhaps it could be enhanced somehow.

Yeah, that takes check output. Without capturing it at runtime, it can't compare anything.

Quoted 7 lines

The XML report may do most of what I want provided we can enable it by default and write it someplace safe rather than clean it up automatically when the test run is interrupted.

I already have test code that extends it to output expunged tests and to add an explicit <pass/> element. It saves the timestamps already, so that's a plus.

Quoted 5 lines

And shortly after writing this, I had a test run hang. Since it hung, even writing it on exit wouldn't have worked. There's not enough information leftover to generate it. I think I'd still like the results directory to contain the information required to generate it as a post-mortem.

Quoted 4 lines

After thinking on it a bit more, since you object to writing a bunch of files by default, we could accomplish the same goal by adding a "files" report type that does this without dropping files for everyone. I'm working that up now.

How's that going BTW? :)

Luis

#10 lore raw
Quoted 38 lines · Luis Chamberlain
Quoted 37 lines · Jeff Mahoney
Quoted 34 lines · Jeff Mahoney
Quoted 20 lines · Jeff Mahoney
Quoted 9 lines · Eryu Guan
Quoted 8 lines · Luis R. Rodriguez
Quoted 6 lines · Eryu Guan

But the test run & pass info is already available from the check output and the test result summary at the end of check. Is that sufficient for you? Also, we already have mechanism to generate a test report in xunit format, i.e. ./check -R xunit -g auto, which includes results for passed & failed & notrun tests.

Quoted 6 lines
Quoted 2 lines

Do we have a way to parse the results *after* a run? For instance, if you forgot -R xunit ?

There's a tools/compare-failures script that takes the outputs of check as inputs and compares the results. But, TBH, I never run it after reviewing it.. Perhaps it could be enhanced somehow.

Yeah, that takes check output. Without capturing it at runtime, it can't compare anything.

Quoted 7 lines

The XML report may do most of what I want provided we can enable it by default and write it someplace safe rather than clean it up automatically when the test run is interrupted.

I already have test code that extends it to output expunged tests and to add an explicit <pass/> element. It saves the timestamps already, so that's a plus.

And shortly after writing this, I had a test run hang. Since it hung, even writing it on exit wouldn't have worked. There's not enough information leftover to generate it. I think I'd still like the results directory to contain the information required to generate it as a post-mortem.

Quoted 6 lines
Quoted 4 lines

After thinking on it a bit more, since you object to writing a bunch of files by default, we could accomplish the same goal by adding a "files" report type that does this without dropping files for everyone. I'm working that up now.

How's that going BTW? :)

I've had it working for a while. It's kind of messy at the moment and drops files in the same output dir as the test results. I'm reworking it a bit to use a separate files directory under the report/section directory so it's easier for scripts to work with.

-Jeff

Signature · 2 lines
Jeff Mahoney
SUSE Labs