Skip to contentSkip to conversation

runtime: indicate failure on crash/timeout/abort in TAP

PATCH Superseded Nico Boehr 3 participants 4 messages Mar 10, 2022 – Mar 22, 2022 KVM

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Nico Boehr

author
#0 lore raw

PATCH 1/1 runtime: indicate failure on crash/timeout/abort in TAP

When we have crashes, timeouts or aborts, there is currently no indication for this in the TAP output. When all reports() up to this point succeeded, this might result in a TAP file looking completely fine even though things went terribly wrong.

For example, when I set the timeout for the diag288 test on s390x to 1 second, it fails because it takes quite long, which is properly indicated in the normal output:

$ ./run_tests.sh diag288
FAIL diag288 (timeout; duration=1s)

But, when I enable TAP output, I get this:

$ ./run_tests.sh -t diag288
TAP version 13
ok 1 - diag288: diag288: privileged: Program interrupt: expected(2) == received(2)
ok 2 - diag288: diag288: specification: uneven: Program interrupt: expected(6) == received(6)
ok 3 - diag288: diag288: specification: unsupported action: Program interrupt: expected(6) == received(6)
ok 4 - diag288: diag288: specification: unsupported function: Program interrupt: expected(6) == received(6)
ok 5 - diag288: diag288: specification: no init: Program interrupt: expected(6) == received(6)
ok 6 - diag288: diag288: specification: min timer: Program interrupt: expected(6) == received(6)
1..6

Which looks like a completely fine TAP file, but actually we ran into a timeout and didn't even run all tests.

With this patch, we get an additional line at the end which properly shows something went wrong:

not ok 7 - diag288: timeout; duration=1s

---
 scripts/runtime.bash | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
scripts/runtime.bash +11 -1
index 6d5fced94246..b41b3d444e27 100644
@@ -163,9 +163,19 @@ function run()
         print_result "SKIP" $testname "$summary"
     elif [ $ret -eq 124 ]; then
         print_result "FAIL" $testname "" "timeout; duration=$timeout"
+        if [[ $tap_output != "no" ]]; then
+            echo "not ok TEST_NUMBER - ${testname}: timeout; duration=$timeout" >&3
+        fi
     elif [ $ret -gt 127 ]; then
-        print_result "FAIL" $testname "" "terminated on SIG$(kill -l $(($ret - 128)))"
+        signame="SIG"$(kill -l $(($ret - 128)))
+        print_result "FAIL" $testname "" "terminated on $signame"
+        if [[ $tap_output != "no" ]]; then
+            echo "not ok TEST_NUMBER - ${testname}: terminated on $signame" >&3
+        fi
     else
+        if [ $ret -eq 127 ] && [[ $tap_output != "no" ]]; then
+            echo "not ok TEST_NUMBER - ${testname}: aborted" >&3
+        fi
         print_result "FAIL" $testname "$summary"
     fi
 
Signature · 1 line
2.31.1
#1 lore raw
Quoted 25 lines · Nico Boehr

When we have crashes, timeouts or aborts, there is currently no indication for this in the TAP output.
When all reports() up to this point succeeded, this
might result in a TAP file looking completely fine even though things went terribly wrong.

For example, when I set the timeout for the diag288 test on s390x to 1 second, it fails because it takes quite long, which is properly indicated in the normal output:

$ ./run_tests.sh diag288
FAIL diag288 (timeout; duration=1s)

But, when I enable TAP output, I get this:

$ ./run_tests.sh -t diag288
TAP version 13
ok 1 - diag288: diag288: privileged: Program interrupt: expected(2) == received(2)
ok 2 - diag288: diag288: specification: uneven: Program interrupt: expected(6) == received(6)
ok 3 - diag288: diag288: specification: unsupported action: Program interrupt: expected(6) == received(6)
ok 4 - diag288: diag288: specification: unsupported function: Program interrupt: expected(6) == received(6)
ok 5 - diag288: diag288: specification: no init: Program interrupt: expected(6) == received(6)
ok 6 - diag288: diag288: specification: min timer: Program interrupt: expected(6) == received(6)
1..6
Quoted 7 lines

Which looks like a completely fine TAP file, but actually we ran into a timeout and didn't even run all tests.

With this patch, we get an additional line at the end which properly shows something went wrong:

not ok 7 - diag288: timeout; duration=1s

This results from the fact that the TAP13 test result is generated by the function RUNTIME_log_stdout and not by print_result (see commit 6e1d3752d7ca ("tap13: list testcases individually")). In RUNTIME_log_stdout we don’t have access to the QEMU command exit code. So what you’re doing here is to workaround that fact… I’m not sure how to fix this properly without refactoring the whole code :/ Maybe Paolo knows a better fix.

Some small nits below… (I don’t make any comments regarding quoting since it was already suboptimal in the code).

Quoted 16 lines
---
 scripts/runtime.bash | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
scripts/runtime.bash +2 -0
index 6d5fced94246..b41b3d444e27 100644
@@ -163,9 +163,19 @@ function run()
         print_result "SKIP" $testname "$summary"
     elif [ $ret -eq 124 ]; then
         print_result "FAIL" $testname "" "timeout; duration=$timeout"
+        if [[ $tap_output != "no" ]]; then
+            echo "not ok TEST_NUMBER - ${testname}: timeout; duration=$timeout" >&3

scripts/runtime.bash function run() · line 176

Quoted 10 lines
scripts/runtime.bash +7 -1
+        fi
     elif [ $ret -gt 127 ]; then
-        print_result "FAIL" $testname "" "terminated on SIG$(kill -l $(($ret - 128)))"
+        signame="SIG"$(kill -l $(($ret - 128)))
+        print_result "FAIL" $testname "" "terminated on $signame"
+        if [[ $tap_output != "no" ]]; then
+            echo "not ok TEST_NUMBER - ${testname}: terminated on $signame" >&3
+        fi
     else
+        if [ $ret -eq 127 ] && [[ $tap_output != "no" ]]; then

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This is a new case, no? If so please add a separate patch creating another elif branch.

Quoted 7 lines
scripts/runtime.bash +2 -0
+            echo "not ok TEST_NUMBER - ${testname}: aborted" >&3
+        fi
         print_result "FAIL" $testname "$summary"
     fi
 
Signature · 1 line
2.31.1

Otherwise, this patch fixes the problem you’ve mentioned - although it leads to even more fragmented code. But I can't think of a better (easy) fix right now either.

author
#2 lore raw
Quoted 1 line · Marc Hartmayer

Nico Boehr <[email protected]> writes:

[...]

Quoted 8 lines
Quoted 8 lines

Which looks like a completely fine TAP file, but actually we ran into a timeout
and didn't even run all tests.

With this patch, we get an additional line at the end which properly shows
something went wrong:

Quoted 8 lines
Quoted 1 line

not ok 7 - diag288: timeout; duration=1s

This results from the fact that the TAP13 test result is generated by the function RUNTIME_log_stdout and not by print_result (see commit
6e1d3752d7ca ("tap13: list testcases individually")). In RUNTIME_log_stdout we don’t have access to the QEMU command exit code.

Basically yes. If we had that we could do all the TAP special handling there.

Quoted 1 line

[...]

Quoted 19 lines
Quoted 19 lines
scripts/runtime.bash +0 -0
index 6d5fced94246..b41b3d444e27 100644
@@ -163,9 +163,19 @@ function run()

         print_result "SKIP" $testname "$summary"
     elif [ $ret -eq 124 ]; then
         print_result "FAIL" $testname "" "timeout; duration=$timeout"
+        if [[ $tap_output != "no" ]]; then
+            echo "not ok TEST_NUMBER - ${testname}: timeout; duration=$timeout" >&3
+        fi
     elif [ $ret -gt 127 ]; then
-        print_result "FAIL" $testname "" "terminated on SIG$(kill -l $(($ret - 128)))"
+        signame="SIG"$(kill -l $(($ret - 128)))
+        print_result "FAIL" $testname "" "terminated on $signame" +        if [[ $tap_output != "no" ]]; then

Quoted 8 lines
Quoted 5 lines

+            echo "not ok TEST_NUMBER - ${testname}: terminated on $signame" >&3
+        fi
     else
+        if [ $ret -eq 127 ] && [[ $tap_output != "no" ]]; then

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This is a new case, no? If so please add a separate patch creating another elif branch.

Probably depends on what you mean by 'new'. The else branch handles the test aborting (for example, exception in the guest) _and_ the case of at least one report failing.

In the latter case, I wanted no additional line in the TAP because we can already see the failed report there.

Making the if an elif makes sense, will do that.

I don't get what you would want to see in a separate patch, can you please make a pseudocode example?

Reviewed-by on v1 1/1
#3 lore raw
In reply to Nico Boehr
Quoted 42 lines · Nico Boehr

When we have crashes, timeouts or aborts, there is currently no indication for this in the TAP output. When all reports() up to this point succeeded, this might result in a TAP file looking completely fine even though things went terribly wrong.

For example, when I set the timeout for the diag288 test on s390x to 1 second, it fails because it takes quite long, which is properly indicated in the normal output:

$ ./run_tests.sh diag288
FAIL diag288 (timeout; duration=1s)

But, when I enable TAP output, I get this:

$ ./run_tests.sh -t diag288
TAP version 13
ok 1 - diag288: diag288: privileged: Program interrupt: expected(2) == received(2)
ok 2 - diag288: diag288: specification: uneven: Program interrupt: expected(6) == received(6)
ok 3 - diag288: diag288: specification: unsupported action: Program interrupt: expected(6) == received(6)
ok 4 - diag288: diag288: specification: unsupported function: Program interrupt: expected(6) == received(6)
ok 5 - diag288: diag288: specification: no init: Program interrupt: expected(6) == received(6)
ok 6 - diag288: diag288: specification: min timer: Program interrupt: expected(6) == received(6)
1..6

Which looks like a completely fine TAP file, but actually we ran into a timeout and didn't even run all tests.

With this patch, we get an additional line at the end which properly shows something went wrong:

not ok 7 - diag288: timeout; duration=1s

---
  scripts/runtime.bash | 12 +++++++++++-
  1 file changed, 11 insertions(+), 1 deletion(-)
scripts/runtime.bash +0 -0
index 6d5fced94246..b41b3d444e27 100644

scripts/runtime.bash function run() · line 166

Quoted 4 lines
scripts/runtime.bash +1 -0
@@ -163,9 +163,19 @@ function run()
          print_result "SKIP" $testname "$summary"
      elif [ $ret -eq 124 ]; then
          print_result "FAIL" $testname "" "timeout; duration=$timeout"
+        if [[ $tap_output != "no" ]]; then

I'd like to avoid "[[" in new code, and the double negation (!= "no) also looks a little bit ugly ... could you please replace this line with:

	if [ "$tap_output" = "yes" ]; then

?

scripts/runtime.bash function run() · line 172

Quoted 7 lines
scripts/runtime.bash +5 -1
+            echo "not ok TEST_NUMBER - ${testname}: timeout; duration=$timeout" >&3
+        fi
      elif [ $ret -gt 127 ]; then
-        print_result "FAIL" $testname "" "terminated on SIG$(kill -l $(($ret - 128)))"
+        signame="SIG"$(kill -l $(($ret - 128)))
+        print_result "FAIL" $testname "" "terminated on $signame"
+        if [[ $tap_output != "no" ]]; then

dito

scripts/runtime.bash function run() · line 176

Quoted 4 lines
scripts/runtime.bash +3 -0
+            echo "not ok TEST_NUMBER - ${testname}: terminated on $signame" >&3
+        fi
      else
+        if [ $ret -eq 127 ] && [[ $tap_output != "no" ]]; then

dito (especially since this mixes [ and [[ in one line)

scripts/runtime.bash function run() · line 180

Quoted 4 lines
scripts/runtime.bash +2 -0
+            echo "not ok TEST_NUMBER - ${testname}: aborted" >&3
+        fi
          print_result "FAIL" $testname "$summary"
      fi

As Marc already mentioned, it's indeed a little bit sad that we now have parts of the TAP handling in run_tests.sh and some parts in scripts/runtime.bash, but I also can't think of a much nicer solution right now ... so with this cosmetics fixed: