Detecting crashes in TAP output
cover Detecting crashes in TAP output
Hi,
I recently had a test on s390x which caused an exception in the guest. As expected, run_tests.sh reported it as failing:
FAIL sometest (1 tests)
However, when I turn on the TAP output everything looks like it is fine:
TAP version 13
ok 1 - sometest: sometest: some report
1..1
There is no way to see from the TAP output the test actually had an unexpected exception. Our internal scripts rely on the TAP output and will thus believe everything is fine, even though it really isn't.
In the logfile, one can at least see the exception backtrace, but if something exits silently, there is no indication something went wrong there either.
TAP provides the test plan (the "1..1") as a solution to this problem. It gives the expected number of test lines if all tests would run. The harness can count test lines in TAP output and compare this to the number of tests in the plan and report an error if it doesn't match.
This won't work with kvm-unit-tests, since there really isn't any way to know how many report()s and thus test lines you will have without actually running the tests.
That's why I came up with an alternative approach in the patch below. It adds an additional test line to the TAP output and the logfiles which states the overall result of a test.
With the attached patch, the TAP output of the test above will now look like this:
TAP version 13
ok 1 - sometest: sometest: some report
not ok 2 - sometest: (1 tests)
1..2
This shows something went wrong.
Your feedback or alternative solutions to this problem are welcome.
Nico Boehr (1):
scripts/runtime: add test result to log and TAP output
scripts/runtime.bash | 2 ++ 1 file changed, 2 insertions(+)
Signature · 1 line
2.31.1
RFC PATCH 1/1 scripts/runtime: add test result to log and TAP output
Currently, when a test exits prematurely for whatever reason (crash, guest exception, ...) the test is correctly reported as FAIL by run_tests.sh - even when all report()s up to that point passed.
The failure is reported by print_result which checks (among others) the exit code of the arch-specific run script.
But, as soon as one enables the TAP output in run_tests.sh, there is no way to see that any more, as the print_result is discarded and only the run script's output is converted to TAP.
External test runners relying on TAP output will thus believe everything is fine even though we got a crash.
The same also applies to the logfiles.
As a simple fix, have print_result also print to RUNTIME_log_stderr. For each test, we will then get an additional test line in the TAP which reports the test's result:
not ok 36 - css: (35 tests, 2 unexpected failures)
The log files will also contain the result:
FAIL: (35 tests, 2 unexpected failures)
This makes it easy to see whether we had a premature exit in the test. The disadvantage being the number of test lines in the TAP will no longer match the number of report()s in a test.
--- scripts/runtime.bash | 2 ++ 1 file changed, 2 insertions(+)
scripts/runtime.bash +2 -0
@@ -60,8 +60,10 @@ function print_result() if [ -z "$reason" ]; then echo "`$status` $testname $summary" + RUNTIME_log_stderr "$testname" <<< "$status: $summary" else echo "`$status` $testname ($reason)" + RUNTIME_log_stderr "$testname" <<< "$status: $reason" fi }
Signature · 1 line
2.31.1