Skip to contentSkip to conversation

runtime: indicate failure on crash/timeout/abort in TAP

PATCH In review Nico Boehr 2 participants 3 messages Mar 22, 2022 – Mar 23, 2022 KVM

lore
Share & tools
lore

Nico Boehr

author
#0 lore raw

v3 cover runtime: indicate failure on crash/timeout/abort in TAP

Sorry for the noise, I sent my v2 a little too early and forgot to include some more feedback.

Changelog v3->v2:
---
* I forgot to include Marc's suggestion to make the else if ... an elif, now included.
* Thomas actually suggested to quote $tap_output, which I missed, now quoted.

Changelog v2->v1:
---
* Change [[ to [
* Remove double negation

Nico Boehr (1):
runtime: indicate failure on crash/timeout/abort in TAP

 scripts/runtime.bash | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
Signature · 1 line
2.31.1
author
#1 lore raw

PATCH v3 1/1 runtime: indicate failure on crash/timeout/abort in TAP

When we have crashes, timeouts or aborts, there is currently no indication for this in the TAP output. When all reports() up to this point succeeded, this might result in a TAP file looking completely fine even though things went terribly wrong.

For example, when I set the timeout for the diag288 test on s390x to 1 second, it fails because it takes quite long, which is properly indicated in the normal output:

$ ./run_tests.sh diag288
FAIL diag288 (timeout; duration=1s)

But, when I enable TAP output, I get this:

$ ./run_tests.sh -t diag288
TAP version 13
ok 1 - diag288: diag288: privileged: Program interrupt: expected(2) == received(2)
ok 2 - diag288: diag288: specification: uneven: Program interrupt: expected(6) == received(6)
ok 3 - diag288: diag288: specification: unsupported action: Program interrupt: expected(6) == received(6)
ok 4 - diag288: diag288: specification: unsupported function: Program interrupt: expected(6) == received(6)
ok 5 - diag288: diag288: specification: no init: Program interrupt: expected(6) == received(6)
ok 6 - diag288: diag288: specification: min timer: Program interrupt: expected(6) == received(6)
1..6

Which looks like a completely fine TAP file, but actually we ran into a timeout and didn't even run all tests.

With this patch, we get an additional line at the end which properly shows something went wrong:

not ok 7 - diag288: timeout; duration=1s

---
 scripts/runtime.bash | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
scripts/runtime.bash +10 -1
index 6d5fced94246..86405604522d 100644
@@ -163,8 +163,17 @@ function run()
         print_result "SKIP" $testname "$summary"
     elif [ $ret -eq 124 ]; then
         print_result "FAIL" $testname "" "timeout; duration=$timeout"
+        if [ "$tap_output" = "yes" ]; then
+            echo "not ok TEST_NUMBER - ${testname}: timeout; duration=$timeout" >&3
+        fi
     elif [ $ret -gt 127 ]; then
-        print_result "FAIL" $testname "" "terminated on SIG$(kill -l $(($ret - 128)))"
+        signame="SIG"$(kill -l $(($ret - 128)))
+        print_result "FAIL" $testname "" "terminated on $signame"
+        if [ "$tap_output" = "yes" ]; then
+            echo "not ok TEST_NUMBER - ${testname}: terminated on $signame" >&3
+        fi
+    elif [ $ret -eq 127 ] && [ "$tap_output" = "yes" ]; then
+        echo "not ok TEST_NUMBER - ${testname}: aborted" >&3
     else
         print_result "FAIL" $testname "$summary"
     fi
Signature · 1 line
2.31.1
#2 lore raw
Quoted 51 lines · Nico Boehr

When we have crashes, timeouts or aborts, there is currently no indication for this in the TAP output. When all reports() up to this point succeeded, this might result in a TAP file looking completely fine even though things went terribly wrong.

For example, when I set the timeout for the diag288 test on s390x to 1 second, it fails because it takes quite long, which is properly indicated in the normal output:

$ ./run_tests.sh diag288
FAIL diag288 (timeout; duration=1s)

But, when I enable TAP output, I get this:

$ ./run_tests.sh -t diag288
TAP version 13
ok 1 - diag288: diag288: privileged: Program interrupt: expected(2) == received(2)
ok 2 - diag288: diag288: specification: uneven: Program interrupt: expected(6) == received(6)
ok 3 - diag288: diag288: specification: unsupported action: Program interrupt: expected(6) == received(6)
ok 4 - diag288: diag288: specification: unsupported function: Program interrupt: expected(6) == received(6)
ok 5 - diag288: diag288: specification: no init: Program interrupt: expected(6) == received(6)
ok 6 - diag288: diag288: specification: min timer: Program interrupt: expected(6) == received(6)
1..6

Which looks like a completely fine TAP file, but actually we ran into a timeout and didn't even run all tests.

With this patch, we get an additional line at the end which properly shows something went wrong:

not ok 7 - diag288: timeout; duration=1s

---
  scripts/runtime.bash | 11 ++++++++++-
  1 file changed, 10 insertions(+), 1 deletion(-)
scripts/runtime.bash +3 -1
index 6d5fced94246..86405604522d 100644
@@ -163,8 +163,17 @@ function run()
          print_result "SKIP" $testname "$summary"
      elif [ $ret -eq 124 ]; then
          print_result "FAIL" $testname "" "timeout; duration=$timeout"
+        if [ "$tap_output" = "yes" ]; then
+            echo "not ok TEST_NUMBER - ${testname}: timeout; duration=$timeout" >&3
+        fi
      elif [ $ret -gt 127 ]; then
-        print_result "FAIL" $testname "" "terminated on SIG$(kill -l $(($ret - 128)))"

scripts/runtime.bash function run() · line 179

Quoted 10 lines
scripts/runtime.bash +7 -0
+        signame="SIG"$(kill -l $(($ret - 128)))
+        print_result "FAIL" $testname "" "terminated on $signame"
+        if [ "$tap_output" = "yes" ]; then
+            echo "not ok TEST_NUMBER - ${testname}: terminated on $signame" >&3
+        fi
+    elif [ $ret -eq 127 ] && [ "$tap_output" = "yes" ]; then
+        echo "not ok TEST_NUMBER - ${testname}: aborted" >&3
      else
          print_result "FAIL" $testname "$summary"
      fi

Thanks, I've pushed it now to the repository!

Thomas