check: add option to rerun failed tests
v4 cover check: add option to rerun failed tests
This patchset adds support to loop on failed tests, as proposed by Ted Ts'o in https://lwn.net/Articles/897061/:
add a mode that will immediately rerun a failed test 25 or 100 times to establish a failure percentage.
Changes since (v3), following Darrick's review:
- extended full/out.bad/etc. retention to cover more paths and remove stale .rerun# files
Changes since (v2), following Darrick's review:
- dropped RFC flag
- rebased atop v2022.07.03
- simplified test iterator
+ results stashed at the end of each iteration, rather than start of next / loop-exit
- dropped aggregate loop stats message from xunit report
- squashed full/dmesg/out.bad file retention patch with rerun patch
Changes since (v1):
- rebased atop upstream v2022.06.26
- added a few extra cleanup commits
- append details for every rerun to xunit output
+ provide aggregate stats via <testcase status=X>
- extend _stash_test_status to call report hook, as well as save failure artifacts with a .rerun# suffix
Diffstat:
check | 137 +++++++++++++++++++++++++++++++++++++------------- common/report | 90 ++++++++++++++++----------------- 2 files changed, 144 insertions(+), 83 deletions(-)
PATCH v4 1/5 report: use array for REPORT_ENV_LIST
There's no need for multiple assignments.
--- common/report | 26 +++++--------------------- 1 file changed, 5 insertions(+), 21 deletions(-)
common/report +5 -21
@@ -4,26 +4,10 @@ # List of xfstests's enviroment variables to include reports ## TODO automate list population inside common/conf -REPORT_ENV_LIST="$REPORT_ENV_LIST SECTION" -REPORT_ENV_LIST="$REPORT_ENV_LIST FSTYP" -REPORT_ENV_LIST="$REPORT_ENV_LIST PLATFORM" -REPORT_ENV_LIST="$REPORT_ENV_LIST MKFS_OPTIONS" -REPORT_ENV_LIST="$REPORT_ENV_LIST MOUNT_OPTIONS" - -REPORT_ENV_LIST="$REPORT_ENV_LIST HOST_OPTIONS" -REPORT_ENV_LIST="$REPORT_ENV_LIST CHECK_OPTIONS" -REPORT_ENV_LIST="$REPORT_ENV_LIST XFS_MKFS_OPTIONS" -REPORT_ENV_LIST="$REPORT_ENV_LIST TIME_FACTOR" -REPORT_ENV_LIST="$REPORT_ENV_LIST LOAD_FACTOR" - -REPORT_ENV_LIST="$REPORT_ENV_LIST TEST_DIR" -REPORT_ENV_LIST="$REPORT_ENV_LIST TEST_DEV" -REPORT_ENV_LIST="$REPORT_ENV_LIST SCRATCH_DEV" -REPORT_ENV_LIST="$REPORT_ENV_LIST SCRATCH_MNT" - -REPORT_ENV_LIST="$REPORT_ENV_LIST OVL_UPPER" -REPORT_ENV_LIST="$REPORT_ENV_LIST OVL_LOWER" -REPORT_ENV_LIST="$REPORT_ENV_LIST OVL_WORK" +REPORT_ENV_LIST=("SECTION" "FSTYP" "PLATFORM" "MKFS_OPTIONS" "MOUNT_OPTIONS" \ + "HOST_OPTIONS" "CHECK_OPTIONS" "XFS_MKFS_OPTIONS" \ + "TIME_FACTOR" "LOAD_FACTOR" "TEST_DIR" "TEST_DEV" \ + "SCRATCH_DEV" "SCRATCH_MNT" "OVL_UPPER" "OVL_LOWER" "OVL_WORK") encode_xml() {
@@ -70,7 +54,7 @@ _xunit_make_section_report() # Properties echo -e "\t<properties>" >> $REPORT_DIR/result.xml - for p in $REPORT_ENV_LIST;do + for p in "${REPORT_ENV_LIST[@]}"; do _xunit_add_property "$p" done echo -e "\t</properties>" >> $REPORT_DIR/result.xml
Signature · 1 line
2.35.3
PATCH v4 2/5 report: pass through most details as function parameters
In reply to David DisseldorpReport generation currently involves reaching into a whole bunch of globals for things like section name and start/end times. Pass these through as explicit function parameters to avoid unintentional breakage.
One minor fix included is the default xunit error message, which used $sequm instead of $seqnum.
--- check | 14 +++++++---- common/report | 64 +++++++++++++++++++++++++++++---------------------- 2 files changed, 47 insertions(+), 31 deletions(-)
check +10 -4
@@ -432,7 +432,9 @@ _wrapup() if $showme && $needwrap; then if $do_report; then # $showme = all selected tests are notrun (no tries) - _make_section_report "${#notrun[*]}" "0" "${#notrun[*]}" + _make_section_report "$section" "${#notrun[*]}" "0" \ + "${#notrun[*]}" \ + "$((sect_stop - sect_start))" fi needwrap=false elif $needwrap; then
@@ -493,7 +495,9 @@ _wrapup() fi echo "" >>$tmp.summary if $do_report; then - _make_section_report "${#try[*]}" "${#bad[*]}" "${#notrun[*]}" + _make_section_report "$section" "${#try[*]}" \ + "${#bad[*]}" "${#notrun[*]}" \ + "$((sect_stop - sect_start))" fi needwrap=false fi
@@ -736,7 +740,8 @@ function run_section() bad+=("$seqnum") fi if $do_report && [[ ! $tc_status =~ ^(init|expunge)$ ]]; then - _make_testcase_report "$prev_seq" "$tc_status" + _make_testcase_report "$section" "$seqnum" \ + "$tc_status" "$((stop - start))" fi prev_seq="$seq"
@@ -940,7 +945,8 @@ function run_section() bad+=("$seqnum") fi if $do_report && [[ ! $tc_status =~ ^(init|expunge)$ ]]; then - _make_testcase_report "$prev_seq" "$tc_status" + _make_testcase_report "$section" "$seqnum" "$tc_status" \ + "$((stop - start))" fi sect_stop=`_wallclock`
common/report +37 -27
120 lines not on this page · raw message
Signature · 1 line
2.35.3
PATCH v4 3/5 check: make a few variables local
In reply to David DisseldorpThe variables aren't used outside of function scope. Also convert one timestamp output to use the helper.
--- check | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
check +5 -5
@@ -179,10 +179,10 @@ get_all_tests() # the function from that list. trim_test_list() { - test_list="$*" + local test_list="$*" rm -f $tmp.grep - numsed=0 + local numsed=0 for t in $test_list do if [ $numsed -gt 100 ]; then
@@ -207,7 +207,7 @@ _wallclock() _timestamp() { - now=`date "+%T"` + local now=`date "+%T"` echo -n " [$now]" }
@@ -603,7 +603,7 @@ fi function run_section() { - local section=$1 + local section=$1 skip OLD_FSTYP=$FSTYP OLD_TEST_FS_MOUNT_OPTS=$TEST_FS_MOUNT_OPTS
@@ -820,7 +820,7 @@ function run_section() rm -f core $seqres.notrun start=`_wallclock` - $timestamp && echo -n " ["`date "+%T"`"]" + $timestamp && _timestamp [ ! -x $seq ] && chmod u+x $seq # ensure we can run it $LOGGER_PROG "run xfstest $seqnum" if [ -w /dev/kmsg ]; then
Signature · 1 line
2.35.3
PATCH v4 4/5 check: append bad / notrun arrays in helper function
In reply to David DisseldorpCurrently the @try, @bad and @notrun arrays are appended with seqnum at different points in the main run_section() loop:
- @try: shortly prior to test script execution
- @notrun: on list (check -n), or after .notrun flagged test completion
- @bad: at the start of subsequent test loop and loop exit
For future loop-test-following-failure functionality it makes sense to combine some of these steps. This change moves both @notrun and @bad appends into a helper function which is called at the end of each loop iteration.
--- check | 68 ++++++++++++++++++++++++++++++++++------------------------- 1 file changed, 39 insertions(+), 29 deletions(-)
check +39 -29
@@ -553,6 +553,32 @@ _expunge_test() return 0 } +# Retain in @bad / @notrun the result of the just-run @test_seq. @try array +# entries are added prior to execution. +_stash_test_status() { + local test_seq="$1" + local test_status="$2" + + if $do_report && [[ $test_status != "expunge" ]]; then + _make_testcase_report "$section" "$test_seq" \ + "$test_status" "$((stop - start))" + fi + + case "$test_status" in + fail) + bad+=("$test_seq") + ;; + list|notrun) + notrun+=("$test_seq") + ;; + pass|expunge) + ;; + *) + echo "Unexpected test $test_seq status: $test_status" + ;; + esac +} + # Can we run systemd scopes? HAVE_SYSTEMD_SCOPES= systemctl reset-failed "fstests-check" &>/dev/null
@@ -732,19 +758,8 @@ function run_section() seqres="$check" _check_test_fs - local tc_status="init" - prev_seq="" + local tc_status for seq in $list ; do - # Run report for previous test! - if [ "$tc_status" == "fail" ]; then - bad+=("$seqnum") - fi - if $do_report && [[ ! $tc_status =~ ^(init|expunge)$ ]]; then - _make_testcase_report "$section" "$seqnum" \ - "$tc_status" "$((stop - start))" - fi - - prev_seq="$seq" if [ ! -f $seq ]; then # Try to get full name in case the user supplied only # seq id and the test has a name. A bit of hassle to
@@ -784,20 +799,21 @@ function run_section() if $showme; then _expunge_test $seqnum if [ $? -eq 1 ]; then - tc_status="expunge" - continue + tc_status="expunge" + else + echo + start=0 + stop=0 + tc_status="list" fi - echo - start=0 - stop=0 - tc_status="list" - notrun+=("$seqnum") + _stash_test_status "$seqnum" "$tc_status" continue fi tc_status="pass" if [ ! -f $seq ]; then echo " - no such test?" + _stash_test_status "$seqnum" "$tc_status" continue fi
@@ -808,6 +824,7 @@ function run_section() _expunge_test $seqnum if [ $? -eq 1 ]; then tc_status="expunge" + _stash_test_status "$seqnum" "$tc_status" continue fi
@@ -857,8 +874,8 @@ function run_section() $timestamp && echo " [not run]" && \ echo -n " $seqnum -- " cat $seqres.notrun - notrun+=("$seqnum") tc_status="notrun" + _stash_test_status "$seqnum" "$tc_status" # Unmount the scratch fs so that we can wipe the scratch # dev state prior to the next test run.
@@ -903,6 +920,7 @@ function run_section() if [ ! -f $seq.out ]; then _dump_err "no qualified output" tc_status="fail" + _stash_test_status "$seqnum" "$tc_status" continue; fi
@@ -938,17 +956,9 @@ function run_section() rm -f $seqres.hints fi fi + _stash_test_status "$seqnum" "$tc_status" done - # make sure we record the status of the last test we ran. - if [ "$tc_status" == "fail" ]; then - bad+=("$seqnum") - fi - if $do_report && [[ ! $tc_status =~ ^(init|expunge)$ ]]; then - _make_testcase_report "$section" "$seqnum" "$tc_status" \ - "$((stop - start))" - fi - sect_stop=`_wallclock` interrupt=false _wrapup
Signature · 1 line
2.35.3
PATCH v4 5/5 check: add -L <n> parameter to rerun failed tests
In reply to David DisseldorpIf check is run with -L <n>, then a failed test will be rerun <n> times before proceeding to the next test. Following completion of the rerun loop, aggregate pass/fail statistics are printed.
Rerun tests will be tracked as a single failure in overall pass/fail metrics (via @try and @bad), with .out.bad, .dmesg, .core, .hints, .notrun and .full saved using a .rerun# suffix.
--- check | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 54 insertions(+), 3 deletions(-)
check +54 -3
@@ -26,6 +26,7 @@ do_report=false DUMP_OUTPUT=false iterations=1 istop=false +loop_on_fail=0 # This is a global variable used to pass test failure text to reporting gunk _err_msg=""
@@ -78,6 +79,7 @@ check options --large-fs optimise scratch device for large filesystems -s section run only specified section from config file -S section exclude the specified section from the config file + -L <n> loop tests <n> times following a failure, measuring aggregate pass/fail metrics testlist options -g group[,group...] include tests from these groups
@@ -336,6 +338,9 @@ while [ $# -gt 0 ]; do ;; --large-fs) export LARGE_SCRATCH_DEV=yes ;; --extra-space=*) export SCRATCH_DEV_EMPTY_SPACE=${r#*=} ;; + -L) [[ $2 =~ ^[0-9]+$ ]] || usage + loop_on_fail=$2; shift + ;; -*) usage ;; *) # not an argument, we've got tests now.
@@ -553,6 +558,19 @@ _expunge_test() return 0 } +# retain files which would be overwritten in subsequent reruns of the same test +_stash_fail_loop_files() { + local seq_prefix="${REPORT_DIR}/${1}" + local cp_suffix="$2" + + for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core" ".hints"; do + rm -f "${seq_prefix}${i}${cp_suffix}" + if [ -f "${seq_prefix}${i}" ]; then + cp "${seq_prefix}${i}" "${seq_prefix}${i}${cp_suffix}" + fi + done +} + # Retain in @bad / @notrun the result of the just-run @test_seq. @try array # entries are added prior to execution. _stash_test_status() {
@@ -564,8 +582,35 @@ _stash_test_status() { "$test_status" "$((stop - start))" fi + if ((${#loop_status[*]} > 0)); then + # continuing or completing rerun-on-failure loop + _stash_fail_loop_files "$test_seq" ".rerun${#loop_status[*]}" + loop_status+=("$test_status") + if ((${#loop_status[*]} > loop_on_fail)); then + printf "%s aggregate results across %d runs: " \ + "$test_seq" "${#loop_status[*]}" + awk "BEGIN { + n=split(\"${loop_status[*]}\", arr);"' + for (i = 1; i <= n; i++) + stats[arr[i]]++; + for (x in stats) + printf("%s=%d (%.1f%%)", + (i-- > n ? x : ", " x), + stats[x], 100 * stats[x] / n); + }' + echo + loop_status=() + fi + return # only stash @bad result for initial failure in loop + fi + case "$test_status" in fail) + if ((loop_on_fail > 0)); then + # initial failure, start rerun-on-failure loop + _stash_fail_loop_files "$test_seq" ".rerun0" + loop_status+=("$test_status") + fi bad+=("$test_seq") ;; list|notrun)
@@ -758,8 +803,12 @@ function run_section() seqres="$check" _check_test_fs - local tc_status - for seq in $list ; do + loop_status=() # track rerun-on-failure state + local tc_status ix + local -a _list=( $list ) + for ((ix = 0; ix < ${#_list[*]}; !${#loop_status[*]} && ix++)); do + seq="${_list[$ix]}" + if [ ! -f $seq ]; then # Try to get full name in case the user supplied only # seq id and the test has a name. A bit of hassle to
@@ -829,7 +878,9 @@ function run_section() fi # record that we really tried to run this test. - try+=("$seqnum") + if ((!${#loop_status[*]})); then + try+=("$seqnum") + fi awk 'BEGIN {lasttime=" "} \ $1 == "'$seqnum'" {lasttime=" " $2 "s ... "; exit} \
Signature · 1 line
2.35.3
Re: [PATCH v4 5/5] check: add -L <n> parameter to rerun failed tests
Quoted 11 lines · David Disseldorp
If check is run with -L <n>, then a failed test will be rerun <n> times before proceeding to the next test. Following completion of the rerun loop, aggregate pass/fail statistics are printed.
Rerun tests will be tracked as a single failure in overall pass/fail metrics (via @try and @bad), with .out.bad, .dmesg, .core, .hints, .notrun and .full saved using a .rerun# suffix.
Suggested-by Theodore Ts'o <[email protected]>Signed-off-by David Disseldorp <[email protected]>
Looks good!
--D
Quoted 119 lines
--- check | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 54 insertions(+), 3 deletions(-)check +54
-3@@ -26,6 +26,7 @@ do_report=false DUMP_OUTPUT=false iterations=1 istop=false +loop_on_fail=0 # This is a global variable used to pass test failure text to reporting gunk _err_msg=""@@ -78,6 +79,7 @@ check options --large-fs optimise scratch device for large filesystems -s section run only specified section from config file -S section exclude the specified section from the config file + -L <n> loop tests <n> times following a failure, measuring aggregate pass/fail metrics testlist options -g group[,group...] include tests from these groups@@ -336,6 +338,9 @@ while [ $# -gt 0 ]; do ;; --large-fs) export LARGE_SCRATCH_DEV=yes ;; --extra-space=*) export SCRATCH_DEV_EMPTY_SPACE=${r#*=} ;; + -L) [[ $2 =~ ^[0-9]+$ ]] || usage + loop_on_fail=$2; shift + ;; -*) usage ;; *) # not an argument, we've got tests now.@@ -553,6 +558,19 @@ _expunge_test() return 0 } +# retain files which would be overwritten in subsequent reruns of the same test +_stash_fail_loop_files() { + local seq_prefix="${REPORT_DIR}/${1}" + local cp_suffix="$2" + + for i in ".full" ".dmesg" ".out.bad" ".notrun" ".core" ".hints"; do + rm -f "${seq_prefix}${i}${cp_suffix}" + if [ -f "${seq_prefix}${i}" ]; then + cp "${seq_prefix}${i}" "${seq_prefix}${i}${cp_suffix}" + fi + done +} + # Retain in @bad / @notrun the result of the just-run @test_seq. @try array # entries are added prior to execution. _stash_test_status() {@@ -564,8 +582,35 @@ _stash_test_status() { "$test_status" "$((stop - start))" fi + if ((${#loop_status[*]} > 0)); then + # continuing or completing rerun-on-failure loop + _stash_fail_loop_files "$test_seq" ".rerun${#loop_status[*]}" + loop_status+=("$test_status") + if ((${#loop_status[*]} > loop_on_fail)); then + printf "%s aggregate results across %d runs: " \ + "$test_seq" "${#loop_status[*]}" + awk "BEGIN { + n=split(\"${loop_status[*]}\", arr);"' + for (i = 1; i <= n; i++) + stats[arr[i]]++; + for (x in stats) + printf("%s=%d (%.1f%%)", + (i-- > n ? x : ", " x), + stats[x], 100 * stats[x] / n); + }' + echo + loop_status=() + fi + return # only stash @bad result for initial failure in loop + fi + case "$test_status" in fail) + if ((loop_on_fail > 0)); then + # initial failure, start rerun-on-failure loop + _stash_fail_loop_files "$test_seq" ".rerun0" + loop_status+=("$test_status") + fi bad+=("$test_seq") ;; list|notrun)@@ -758,8 +803,12 @@ function run_section() seqres="$check" _check_test_fs - local tc_status - for seq in $list ; do + loop_status=() # track rerun-on-failure state + local tc_status ix + local -a _list=( $list ) + for ((ix = 0; ix < ${#_list[*]}; !${#loop_status[*]} && ix++)); do + seq="${_list[$ix]}" + if [ ! -f $seq ]; then # Try to get full name in case the user supplied only # seq id and the test has a name. A bit of hassle to@@ -829,7 +878,9 @@ function run_section() fi # record that we really tried to run this test. - try+=("$seqnum") + if ((!${#loop_status[*]})); then + try+=("$seqnum") + fi awk 'BEGIN {lasttime=" "} \ $1 == "'$seqnum'" {lasttime=" " $2 "s ... "; exit} \Signature · 1 line
2.35.3
Quoted 4 lines · David Disseldorp
This patchset adds support to loop on failed tests, as proposed by Ted Ts'o in https://lwn.net/Articles/897061/:
add a mode that will immediately rerun a failed test 25 or 100 times to establish a failure percentage.
This patchset is good to me. if no more objections, I think it's time to merge this feature. I'll give it a testing, then try to push it this weekend.
Thanks,
Zorro
Quoted 28 lines
Changes since (v3), following Darrick's review:
- extended full/out.bad/etc. retention to cover more paths and remove stale .rerun# filesChanges since (v2), following Darrick's review:
- dropped RFC flag
- rebased atop v2022.07.03
- simplified test iterator
+ results stashed at the end of each iteration, rather than start of next / loop-exit
- dropped aggregate loop stats message from xunit report
- squashed full/dmesg/out.bad file retention patch with rerun patchChanges since (v1):
- rebased atop upstream v2022.06.26
- added a few extra cleanup commits
- append details for every rerun to xunit output
+ provide aggregate stats via <testcase status=X>
- extend_stash_test_statusto call report hook, as well as save failure artifacts with a .rerun# suffixDiffstat:
check | 137 +++++++++++++++++++++++++++++++++++++------------- common/report | 90 ++++++++++++++++----------------- 2 files changed, 144 insertions(+), 83 deletions(-)