check: annotate good and expunged tests in results
PATCH 1/1 check: annotate good and expunged tests in results
From: Jeff Mahoney <[email protected]>
Currently, we only create results files when a test has failed or was supposed to run but some dependency wasn't met causing it not to run. Short of saving the summary at the end of the run, there's no way to tell which tests passed or which tests weren't run due to being excluded.
This patch moves successful test results to $seqres.out.good to annotate good results. It also adds tests excluded by group to the $tmp.xlist file and adds annotations for every test excluded. When a test is expunged during execution, an expunged message will be issued and a $seqres.expunged file will be created, both containing the reason for the test being expunged. Reasons can be "command line", "file $filename", or "group <group> [group...]".
This makes the output more noisy in the expunged case and makes startup take slightly longer, but ends up with results that can be more easily parsed by automated tools.
--- check | 57 +++++++++++++++++++++++++++++++-------------------------- 1 file changed, 31 insertions(+), 26 deletions(-)
check +31 -26
@@ -173,28 +173,18 @@ get_all_tests() done } -# takes the list of tests to run in $tmp.list, and removes the tests passed to -# the function from that list. +# takes the list of tests to run in $tmp.list and adds them to the excluded +# test list, annotated with the group that excluded each one trim_test_list() { + group=$1 + shift test_list="$*" - rm -f $tmp.grep - numsed=0 - for t in $test_list - do - if [ $numsed -gt 100 ]; then - grep -v -f $tmp.grep <$tmp.list >$tmp.tmp - mv $tmp.tmp $tmp.list - numsed=0 - rm -f $tmp.grep - fi - echo "^$t\$" >>$tmp.grep - numsed=`expr $numsed + 1` + for test in $test_list; do + test=${test##tests/} + echo "$test group $group" >> $tmp.xlist done - grep -v -f $tmp.grep <$tmp.list >$tmp.tmp - mv $tmp.tmp $tmp.list - rm -f $tmp.grep }
@@ -246,7 +236,7 @@ _prepare_test_list() exit 1 fi - trim_test_list $list + trim_test_list $xgroup $list done # sort the list of tests into numeric order
@@ -285,13 +275,15 @@ while [ $# -gt 0 ]; do for d in $SRC_GROUPS $FSTYP; do [ -f $SRC_DIR/$d/$xfile ] || continue for f in `sed "s/#.*$//" $SRC_DIR/$d/$xfile`; do - echo $d/$f >> $tmp.xlist + echo "$d/$f command line" >> $tmp.xlist done done ;; -E) xfile=$2; shift ; if [ -f $xfile ]; then - sed "s/#.*$//" "$xfile" >> $tmp.xlist + sed -e "s/#.*$//" \ + -e "s;$; file $xfile;" "$xfile" \ + >> $tmp.xlist fi ;; -s) RUN_SECTION="$RUN_SECTION $2"; shift ;;
@@ -491,11 +483,17 @@ _check_filesystems() _expunge_test() { local TEST_ID="$1" + local OUTPUT="$2" if [ -s $tmp.xlist ]; then - if grep -q $TEST_ID $tmp.xlist; then - echo " [expunged]" + grep $TEST_ID $tmp.xlist > $tmp._expunge_test + if [ $? -eq 0 ]; then + sed -e "s;$TEST_ID ;;" $tmp._expunge_test | \ + tr '\n' ' ' | sed -e 's; group;;g' + echo + rm -f $tmp._expunge_test return 1 fi + rm -f $tmp._expunge_test fi return 0 }
@@ -670,8 +668,10 @@ for section in $HOST_OPTIONS_SECTIONS; do echo -n "$seqnum" if $showme; then - _expunge_test $seqnum + _expunge_test $seqnum > $tmp.xreason if [ $? -eq 1 ]; then + echo -n " [expunged] " + cat $tmp.xreason continue fi echo
@@ -689,11 +689,15 @@ for section in $HOST_OPTIONS_SECTIONS; do else # really going to try and run this one # - rm -f $seqres.out.bad + rm -f $seqres.out.bad $seqres.out.good $seqres.expunged + rm -f $seqres.notrun # check if we really should run it - _expunge_test $seqnum + _expunge_test $seqnum > $tmp.xreason if [ $? -eq 1 ]; then + mv $tmp.xreason $seqres.expunged + echo -n " [expunged] " + cat $seqres.expunged continue fi
@@ -704,7 +708,7 @@ for section in $HOST_OPTIONS_SECTIONS; do else echo -n " " # prettier output with timestamps. fi - rm -f core $seqres.notrun + rm -f core start=`_wallclock` $timestamp && echo -n " ["`date "+%T"`"]"
@@ -770,6 +774,7 @@ for section in $HOST_OPTIONS_SECTIONS; do else echo "$seqnum `expr $stop - $start`" >>$tmp.time echo -n " `expr $stop - $start`s" + mv $tmp.out $seqres.out.good fi echo "" else
Signature · 1 line
2.15.1
Quoted 1 line · On Thu, Apr 12, 2018 at 05:38:38PM -0400, [email protected] wrote:
From: Jeff Mahoney <[email protected]>
[Sorry for the late review..]
Quoted 9 lines
Currently, we only create results files when a test has failed or was supposed to run but some dependency wasn't met causing it not to run. Short of saving the summary at the end of the run, there's no way to tell which tests passed or which tests weren't run due to being excluded.
This patch moves successful test results to $seqres.out.good to annotate good results. It also adds tests excluded by group to the $tmp.xlist file and adds annotations for every test excluded. When a test is expunged
I can see that annotating tests excluded by group is helpful, but I don't think we need $seqres.out.good, passed tests are already shown by check, and $seqres.out.good is same as the golden image, it looks redundant.
Quoted 19 lines
during execution, an expunged message will be issued and a $seqres.expunged file will be created, both containing the reason for the test being expunged. Reasons can be "command line", "file $filename", or "group <group> [group...]".
This makes the output more noisy in the expunged case and makes startup take slightly longer, but ends up with results that can be more easily parsed by automated tools.
Signed-off-by Jeff Mahoney <[email protected]>--- check | 57 +++++++++++++++++++++++++++++++-------------------------- 1 file changed, 31 insertions(+), 26 deletions(-)check +0
-0
check get_all_tests() · line 177
Quoted 7 lines
check +2
-2@@ -173,28 +173,18 @@ get_all_tests() done } -# takes the list of tests to run in $tmp.list, and removes the tests passed to -# the function from that list. +# takes the list of tests to run in $tmp.list and adds them to the excluded +# test list, annotated with the group that excluded each one
The comments don't seem quite right, the new behavior doesn't take tests from $tmp.list, it just edits the $tmp.xlist file for later use.
check get_all_tests() · line 180
Quoted 3 lines
check +1
-0trim_test_list() { + group=$1
Please declare local variable as 'local' when possible.
Quoted 39 lines
check +5
-17+ shift test_list="$*" - rm -f $tmp.grep - numsed=0 - for t in $test_list - do - if [ $numsed -gt 100 ]; then - grep -v -f $tmp.grep <$tmp.list >$tmp.tmp - mv $tmp.tmp $tmp.list - numsed=0 - rm -f $tmp.grep - fi - echo "^$t\$" >>$tmp.grep - numsed=`expr $numsed + 1` + for test in $test_list; do + test=${test##tests/} + echo "$test group $group" >> $tmp.xlist done - grep -v -f $tmp.grep <$tmp.list >$tmp.tmp - mv $tmp.tmp $tmp.list - rm -f $tmp.grep }@@ -246,7 +236,7 @@ _prepare_test_list() exit 1 fi - trim_test_list $list + trim_test_list $xgroup $list done # sort the list of tests into numeric order@@ -285,13 +275,15 @@ while [ $# -gt 0 ]; do for d in $SRC_GROUPS $FSTYP; do [ -f $SRC_DIR/$d/$xfile ] || continue for f in `sed "s/#.*$//" $SRC_DIR/$d/$xfile`; do - echo $d/$f >> $tmp.xlist
check while [ $# -gt 0 ]; do · line 286
Quoted 10 lines
check +4
-1+ echo "$d/$f command line" >> $tmp.xlist done done ;; -E) xfile=$2; shift ; if [ -f $xfile ]; then - sed "s/#.*$//" "$xfile" >> $tmp.xlist + sed -e "s/#.*$//" \ + -e "s;$; file $xfile;" "$xfile" \ + >> $tmp.xlist
When using config section, e.g. ./check -s <sec_name>, and FSTYP is different in new section, check will do _prepare_test_list again, so tests selected by "-x <xgroup>" will be listed twice in $tmp.xlist file, as a result, I see duplicated group names in the annotate message, e.g.
generic/458 [expunged] group metadata metadata
I think we should move above "-X/-E" handlings to _prepare_test_list, only assign variable "xfile" here (use a new variable for -E option, perhaps). So that we could simply remove $tmp.xlist before calling _prepare_test_list again, and it could exclude all specified tests only once. Maybe as patch 1/2 in a patchset.
check _check_filesystems() · line 486
Quoted 8 lines
check +1
-0fi ;; -s) RUN_SECTION="$RUN_SECTION $2"; shift ;;@@ -491,11 +483,17 @@ _check_filesystems() _expunge_test() { local TEST_ID="$1" + local OUTPUT="$2"
OUTPUT is not used.
Quoted 55 lines
check +17
-6if [ -s $tmp.xlist ]; then - if grep -q $TEST_ID $tmp.xlist; then - echo " [expunged]" + grep $TEST_ID $tmp.xlist > $tmp._expunge_test + if [ $? -eq 0 ]; then + sed -e "s;$TEST_ID ;;" $tmp._expunge_test | \ + tr '\n' ' ' | sed -e 's; group;;g' + echo + rm -f $tmp._expunge_test return 1 fi + rm -f $tmp._expunge_test fi return 0 }@@ -670,8 +668,10 @@ for section in $HOST_OPTIONS_SECTIONS; do echo -n "$seqnum" if $showme; then - _expunge_test $seqnum + _expunge_test $seqnum > $tmp.xreason if [ $? -eq 1 ]; then + echo -n " [expunged] " + cat $tmp.xreason continue fi echo@@ -689,11 +689,15 @@ for section in $HOST_OPTIONS_SECTIONS; do else # really going to try and run this one # - rm -f $seqres.out.bad + rm -f $seqres.out.bad $seqres.out.good $seqres.expunged + rm -f $seqres.notrun # check if we really should run it - _expunge_test $seqnum + _expunge_test $seqnum > $tmp.xreason if [ $? -eq 1 ]; then + mv $tmp.xreason $seqres.expunged + echo -n " [expunged] " + cat $seqres.expunged continue fi@@ -704,7 +708,7 @@ for section in $HOST_OPTIONS_SECTIONS; do else echo -n " " # prettier output with timestamps. fi - rm -f core $seqres.notrun + rm -f core start=`_wallclock` $timestamp && echo -n " ["`date "+%T"`"]"
check for section in $HOST_OPTIONS_SECTIONS; do · line 777
Quoted 4 lines
check +1
-0@@ -770,6 +774,7 @@ for section in $HOST_OPTIONS_SECTIONS; do else echo "$seqnum `expr $stop - $start`" >>$tmp.time echo -n " `expr $stop - $start`s" + mv $tmp.out $seqres.out.good
As said above, I don't think it's that useful.
Thanks,
Eryu
Quoted 10 lines
check +0
-0fi echo "" elseSignature · 6 lines
2.15.1 -- To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · Eryu Guan
Quoted 1 line · On Thu, Apr 12, 2018 at 05:38:38PM -0400, [email protected] wrote:
From: Jeff Mahoney <[email protected]>
[Sorry for the late review..]
Thanks for the review just the same.
Quoted 5 lines
Quoted 5 lines
Currently, we only create results files when a test has failed or was supposed to run but some dependency wasn't met causing it not to run. Short of saving the summary at the end of the run, there's no way to tell which tests passed or which tests weren't run due to being excluded.
Quoted 8 lines
Quoted 3 lines
This patch moves successful test results to $seqres.out.good to annotate good results. It also adds tests excluded by group to the $tmp.xlist file and adds annotations for every test excluded. When a test is expunged
I can see that annotating tests excluded by group is helpful, but I don't think we need $seqres.out.good, passed tests are already shown by check, and $seqres.out.good is same as the golden image, it looks redundant.
The contents are redundant, but the purpose of the file existing isn't. My goal is to be able to point a script at the results directory for a test run and autogenerate a table of results and/or compare them to known results. Right now, there's no way to tell whether a test was run at all, only that it failed or was skipped due to some missing dependency. The expunge changes handle the explicitly skipped cases, but without the .good files, there's nothing indicating a test actually ran or when it last did. The contents don't have to be the output. If you can think of something more useful there (or if we just create an empty file), that's fine too.
Quoted 7 lines
check get_all_tests() · line 177
Quoted 4 lines
check +2
-2-# takes the list of tests to run in $tmp.list, and removes the tests passed to -# the function from that list. +# takes the list of tests to run in $tmp.list and adds them to the excluded +# test list, annotated with the group that excluded each oneThe comments don't seem quite right, the new behavior doesn't take tests from $tmp.list, it just edits the $tmp.xlist file for later use.
Right. I missed updating the comment after changing the behavior.
Quoted 5 lines
check get_all_tests() · line 180
Quoted 3 lines
trim_test_list()
{check +1
-0+ group=$1Please declare local variable as 'local' when possible.
Ok.
Quoted 5 lines
When using config section, e.g. ./check -s <
sec_name>, and FSTYP is different in new section, check will do_prepare_test_listagain, so tests selected by "-x <xgroup>" will be listed twice in $tmp.xlist file, as a result, I see duplicated group names in the annotate message, e.g.
Quoted 7 lines
generic/458 [expunged] group metadata metadata
I think we should move above "-X/-E" handlings to
_prepare_test_list, only assign variable "xfile" here (use a new variable for -E option, perhaps). So that we could simply remove $tmp.xlist before calling_prepare_test_listagain, and it could exclude all specified tests only once. Maybe as patch 1/2 in a patchset.
Ok, that's easy enough to do.
Quoted 10 lines
check _check_filesystems() · line 486
Quoted 8 lines
check +1
-0fi ;; -s) RUN_SECTION="$RUN_SECTION $2"; shift ;;@@ -491,11 +483,17 @@ _check_filesystems() _expunge_test() { local TEST_ID="$1" + local OUTPUT="$2"OUTPUT is not used.
Ack.
Quoted 7 lines
check for section in $HOST_OPTIONS_SECTIONS; do · line 777
Quoted 5 lines
check +1
-0@@ -770,6 +774,7 @@ for section in $HOST_OPTIONS_SECTIONS; do else echo "$seqnum `expr $stop - $start`" >>$tmp.time echo -n " `expr $stop - $start`s" + mv $tmp.out $seqres.out.goodAs said above, I don't think it's that useful.
It doesn't need to be the contents of the output, but I'd like it to be *something* and moving a file that already exists is as easy as creating a new one.
-Jeff
Signature · 2 lines
Jeff Mahoney SUSE Labs
Quoted 22 lines · Jeff Mahoney
Quoted 4 lines · Eryu Guan
Quoted 1 line · On Thu, Apr 12, 2018 at 05:38:38PM -0400, [email protected] wrote:
From: Jeff Mahoney <[email protected]>
[Sorry for the late review..]
Thanks for the review just the same.
Quoted 5 lines
Quoted 5 lines
Currently, we only create results files when a test has failed or was supposed to run but some dependency wasn't met causing it not to run. Short of saving the summary at the end of the run, there's no way to tell which tests passed or which tests weren't run due to being excluded.
Quoted 8 lines
Quoted 3 lines
This patch moves successful test results to $seqres.out.good to annotate good results. It also adds tests excluded by group to the $tmp.xlist file and adds annotations for every test excluded. When a test is expunged
I can see that annotating tests excluded by group is helpful, but I don't think we need $seqres.out.good, passed tests are already shown by check, and $seqres.out.good is same as the golden image, it looks redundant.
Quoted 4 lines
The contents are redundant, but the purpose of the file existing isn't. My goal is to be able to point a script at the results directory for a test run and autogenerate a table of results and/or compare them to known results. Right now, there's no way to tell whether a test was run
But the test run & pass info is already available from the check output and the test result summary at the end of check. Is that sufficient for you? Also, we already have mechanism to generate a test report in xunit format, i.e. ./check -R xunit -g auto, which includes results for passed & failed & notrun tests.
My concern is that saving .good files consume more and more space over time, and they don't provide any additional information (even with empty files that don't require much space).
Thanks,
Eryu
Quoted 69 lines
at all, only that it failed or was skipped due to some missing dependency. The expunge changes handle the explicitly skipped cases, but without the .good files, there's nothing indicating a test actually ran or when it last did. The contents don't have to be the output. If you can think of something more useful there (or if we just create an empty file), that's fine too.
Quoted 7 lines
check get_all_tests() · line 177
Quoted 4 lines
check +2
-2-# takes the list of tests to run in $tmp.list, and removes the tests passed to -# the function from that list. +# takes the list of tests to run in $tmp.list and adds them to the excluded +# test list, annotated with the group that excluded each oneThe comments don't seem quite right, the new behavior doesn't take tests from $tmp.list, it just edits the $tmp.xlist file for later use.
Right. I missed updating the comment after changing the behavior.
Quoted 5 lines
check get_all_tests() · line 180
Quoted 3 lines
trim_test_list()
{check +1
-0+ group=$1Please declare local variable as 'local' when possible.
Ok.
Quoted 5 lines
When using config section, e.g. ./check -s <
sec_name>, and FSTYP is different in new section, check will do_prepare_test_listagain, so tests selected by "-x <xgroup>" will be listed twice in $tmp.xlist file, as a result, I see duplicated group names in the annotate message, e.g.Quoted 7 lines
generic/458 [expunged] group metadata metadata
I think we should move above "-X/-E" handlings to
_prepare_test_list, only assign variable "xfile" here (use a new variable for -E option, perhaps). So that we could simply remove $tmp.xlist before calling_prepare_test_listagain, and it could exclude all specified tests only once. Maybe as patch 1/2 in a patchset.Ok, that's easy enough to do.
Quoted 10 lines
check _check_filesystems() · line 486
Quoted 8 lines
check +1
-0fi ;; -s) RUN_SECTION="$RUN_SECTION $2"; shift ;;@@ -491,11 +483,17 @@ _check_filesystems() _expunge_test() { local TEST_ID="$1" + local OUTPUT="$2"OUTPUT is not used.
Ack.
Quoted 7 lines
check for section in $HOST_OPTIONS_SECTIONS; do · line 777
Quoted 5 lines
check +1
-0@@ -770,6 +774,7 @@ for section in $HOST_OPTIONS_SECTIONS; do else echo "$seqnum `expr $stop - $start`" >>$tmp.time echo -n " `expr $stop - $start`s" + mv $tmp.out $seqres.out.goodAs said above, I don't think it's that useful.
It doesn't need to be the contents of the output, but I'd like it to be *something* and moving a file that already exists is as easy as creating a new one.
-Jeff
Signature · 2 lines
Jeff Mahoney SUSE Labs
Quoted 6 lines · Eryu Guan
But the test run & pass info is already available from the check output and the test result summary at the end of check. Is that sufficient for you? Also, we already have mechanism to generate a test report in xunit format, i.e. ./check -R xunit -g auto, which includes results for passed & failed & notrun tests.
Do we have a way to parse the results *after* a run? For instance, if you forgot -R xunit ?
Luis
Quoted 10 lines · Luis R. Rodriguez
Quoted 6 lines · Eryu Guan
But the test run & pass info is already available from the check output and the test result summary at the end of check. Is that sufficient for you? Also, we already have mechanism to generate a test report in xunit format, i.e. ./check -R xunit -g auto, which includes results for passed & failed & notrun tests.
Do we have a way to parse the results *after* a run? For instance, if you forgot -R xunit ?
There's a tools/compare-failures script that takes the outputs of check as inputs and compares the results. But, TBH, I never run it after reviewing it.. Perhaps it could be enhanced somehow.
Thanks,
Eryu
Quoted 9 lines · Eryu Guan
Quoted 8 lines · Luis R. Rodriguez
Quoted 6 lines · Eryu Guan
But the test run & pass info is already available from the check output and the test result summary at the end of check. Is that sufficient for you? Also, we already have mechanism to generate a test report in xunit format, i.e. ./check -R xunit -g auto, which includes results for passed & failed & notrun tests.
Quoted 6 lines
Quoted 2 lines
Do we have a way to parse the results *after* a run? For instance, if you forgot -R xunit ?
There's a
tools/compare-failuresscript that takes the outputs of check as inputs and compares the results. But, TBH, I never run it after reviewing it.. Perhaps it could be enhanced somehow.
Yeah, that takes check output. Without capturing it at runtime, it can't compare anything.
The XML report may do most of what I want provided we can enable it by default and write it someplace safe rather than clean it up automatically when the test run is interrupted.
I already have test code that extends it to output expunged tests and to add an explicit <pass/> element. It saves the timestamps already, so that's a plus.
-Jeff
Signature · 2 lines
Jeff Mahoney SUSE Labs
Quoted 20 lines · Jeff Mahoney
Quoted 9 lines · Eryu Guan
Quoted 8 lines · Luis R. Rodriguez
Quoted 6 lines · Eryu Guan
But the test run & pass info is already available from the check output and the test result summary at the end of check. Is that sufficient for you? Also, we already have mechanism to generate a test report in xunit format, i.e. ./check -R xunit -g auto, which includes results for passed & failed & notrun tests.
Quoted 6 lines
Quoted 2 lines
Do we have a way to parse the results *after* a run? For instance, if you forgot -R xunit ?
There's a
tools/compare-failuresscript that takes the outputs of check as inputs and compares the results. But, TBH, I never run it after reviewing it.. Perhaps it could be enhanced somehow.Yeah, that takes check output. Without capturing it at runtime, it can't compare anything.
Quoted 7 lines
The XML report may do most of what I want provided we can enable it by default and write it someplace safe rather than clean it up automatically when the test run is interrupted.
I already have test code that extends it to output expunged tests and to add an explicit <pass/> element. It saves the timestamps already, so that's a plus.
And shortly after writing this, I had a test run hang. Since it hung, even writing it on exit wouldn't have worked. There's not enough information leftover to generate it. I think I'd still like the results directory to contain the information required to generate it as a post-mortem.
-Jeff
Signature · 2 lines
Jeff Mahoney SUSE Labs
Quoted 29 lines · Jeff Mahoney
Quoted 20 lines · Jeff Mahoney
Quoted 9 lines · Eryu Guan
Quoted 8 lines · Luis R. Rodriguez
Quoted 6 lines · Eryu Guan
But the test run & pass info is already available from the check output and the test result summary at the end of check. Is that sufficient for you? Also, we already have mechanism to generate a test report in xunit format, i.e. ./check -R xunit -g auto, which includes results for passed & failed & notrun tests.
Quoted 6 lines
Quoted 2 lines
Do we have a way to parse the results *after* a run? For instance, if you forgot -R xunit ?
There's a
tools/compare-failuresscript that takes the outputs of check as inputs and compares the results. But, TBH, I never run it after reviewing it.. Perhaps it could be enhanced somehow.Yeah, that takes check output. Without capturing it at runtime, it can't compare anything.
Quoted 7 lines
The XML report may do most of what I want provided we can enable it by default and write it someplace safe rather than clean it up automatically when the test run is interrupted.
I already have test code that extends it to output expunged tests and to add an explicit <pass/> element. It saves the timestamps already, so that's a plus.
Quoted 5 lines
And shortly after writing this, I had a test run hang. Since it hung, even writing it on exit wouldn't have worked. There's not enough information leftover to generate it. I think I'd still like the results directory to contain the information required to generate it as a post-mortem.
After thinking on it a bit more, since you object to writing a bunch of files by default, we could accomplish the same goal by adding a "files" report type that does this without dropping files for everyone. I'm working that up now.
-Jeff
Signature · 2 lines
Jeff Mahoney SUSE Labs
Quoted 37 lines · Jeff Mahoney
Quoted 29 lines · Jeff Mahoney
Quoted 20 lines · Jeff Mahoney
Quoted 9 lines · Eryu Guan
Quoted 8 lines · Luis R. Rodriguez
Quoted 6 lines · Eryu Guan
But the test run & pass info is already available from the check output and the test result summary at the end of check. Is that sufficient for you? Also, we already have mechanism to generate a test report in xunit format, i.e. ./check -R xunit -g auto, which includes results for passed & failed & notrun tests.
Quoted 6 lines
Quoted 2 lines
Do we have a way to parse the results *after* a run? For instance, if you forgot -R xunit ?
There's a
tools/compare-failuresscript that takes the outputs of check as inputs and compares the results. But, TBH, I never run it after reviewing it.. Perhaps it could be enhanced somehow.Yeah, that takes check output. Without capturing it at runtime, it can't compare anything.
Quoted 7 lines
The XML report may do most of what I want provided we can enable it by default and write it someplace safe rather than clean it up automatically when the test run is interrupted.
I already have test code that extends it to output expunged tests and to add an explicit <pass/> element. It saves the timestamps already, so that's a plus.
Quoted 5 lines
And shortly after writing this, I had a test run hang. Since it hung, even writing it on exit wouldn't have worked. There's not enough information leftover to generate it. I think I'd still like the results directory to contain the information required to generate it as a post-mortem.
Quoted 4 lines
After thinking on it a bit more, since you object to writing a bunch of files by default, we could accomplish the same goal by adding a "files" report type that does this without dropping files for everyone. I'm working that up now.
How's that going BTW? :)
Luis
Quoted 38 lines · Luis Chamberlain
Quoted 37 lines · Jeff Mahoney
Quoted 34 lines · Jeff Mahoney
Quoted 20 lines · Jeff Mahoney
Quoted 9 lines · Eryu Guan
Quoted 8 lines · Luis R. Rodriguez
Quoted 6 lines · Eryu Guan
But the test run & pass info is already available from the check output and the test result summary at the end of check. Is that sufficient for you? Also, we already have mechanism to generate a test report in xunit format, i.e. ./check -R xunit -g auto, which includes results for passed & failed & notrun tests.
Quoted 6 lines
Quoted 2 lines
Do we have a way to parse the results *after* a run? For instance, if you forgot -R xunit ?
There's a
tools/compare-failuresscript that takes the outputs of check as inputs and compares the results. But, TBH, I never run it after reviewing it.. Perhaps it could be enhanced somehow.Yeah, that takes check output. Without capturing it at runtime, it can't compare anything.
Quoted 7 lines
The XML report may do most of what I want provided we can enable it by default and write it someplace safe rather than clean it up automatically when the test run is interrupted.
I already have test code that extends it to output expunged tests and to add an explicit <pass/> element. It saves the timestamps already, so that's a plus.
And shortly after writing this, I had a test run hang. Since it hung, even writing it on exit wouldn't have worked. There's not enough information leftover to generate it. I think I'd still like the results directory to contain the information required to generate it as a post-mortem.
Quoted 6 lines
Quoted 4 lines
After thinking on it a bit more, since you object to writing a bunch of files by default, we could accomplish the same goal by adding a "files" report type that does this without dropping files for everyone. I'm working that up now.
How's that going BTW? :)
I've had it working for a while. It's kind of messy at the moment and drops files in the same output dir as the test results. I'm reworking it a bit to use a separate files directory under the report/section directory so it's easier for scripts to work with.
-Jeff
Signature · 2 lines
Jeff Mahoney SUSE Labs