lib: Do not report failures when test passes
Subsystems (2)
PATCH 1/1 lib: Do not report failures when test passes
From: Nadav Amit <[email protected]>
Currently, if a test is expected to fail, but surprisingly it passes, the test is considered as "failing". This means that running on old KVM-unit-tests on new KVM versions can falsely report failures.
Fix it and simplify the logic.
--- lib/report.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
lib/report.c +6 -4
@@ -96,10 +96,12 @@ static void va_report(const char *msg_fmt,puts ("\n" );if (skip) skipped++; -else if (xfail && !pass) - xfailures++; -else if (xfail || !pass) - failures++; +else if (!pass) { +if (xfail) + xfailures++; +else + failures++; + }spin_unlock (&lock); }
Signature · 1 line
2.17.1
Quoted 4 lines · On Tue, Apr 16, 2019 at 10:18:11PM -0700, [email protected] wrote:
From: Nadav Amit <[email protected]>
Currently, if a test is expected to fail, but surprisingly it passes, the test is considered as "failing".
It's not "failing", it's failing. If a test is expected to pass then it shouldn't be getting reported with report_xfail().
If you know of an xfail test that is passing, and believe it should be passing now, then please send a patch changing the report_xfail() to report().
Quoted 2 lines
This means that running on old
KVM-unit-tests on new KVM versions can falsely report failures.
Why would one want to run old kvm-unit-tests on new kvm? We primarily care about latest kvm-unit-tests running on latest kvm, but there are cases where we want to run latest kvm-unit-tests on old kvm, like when bisecting something with a new test. In some cases that can be risky, though, if a test case can trigger host crashes. For those cases we have the errata framework.
Thanks,
drew
Quoted 32 lines
Fix it and simplify the logic.
Signed-off-by Nadav Amit <[email protected]>--- lib/report.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)lib/report.c +6
-4@@ -96,10 +96,12 @@ static void va_report(const char *msg_fmt,puts ("\n" );if (skip) skipped++; -else if (xfail && !pass) - xfailures++; -else if (xfail || !pass) - failures++; +else if (!pass) { +if (xfail) + xfailures++; +else + failures++; + }spin_unlock (&lock); }Signature · 1 line
2.17.1
Quoted 10 lines · Andrew Jones
Quoted 4 lines · On Tue, Apr 16, 2019 at 10:18:11PM -0700, [email protected] wrote:
From: Nadav Amit <[email protected]>
Currently, if a test is expected to fail, but surprisingly it passes, the test is considered as "failing".
It's not "failing", it's failing. If a test is expected to pass then it shouldn't be getting reported with
report_xfail().
I find this terminology confusing. For instance, there are some tests which are probabilistic (e.g., test_sti_nmi) - let’s assume you expect one to fail and it passes, would you say that you encountered a failure?
Quoted 1 line
Why would one want to run old kvm-unit-tests on new kvm?
I can think of a couple of reasons, but I am not going to argue too much.
Quoted 8 lines · Nadav Amit
Quoted 8 lines · Andrew Jones
Quoted 4 lines · On Tue, Apr 16, 2019 at 10:18:11PM -0700, [email protected] wrote:
From: Nadav Amit <[email protected]>
Currently, if a test is expected to fail, but surprisingly it passes, the test is considered as "failing".
Quoted 6 lines
Quoted 2 lines
It's not "failing", it's failing. If a test is expected to pass then it shouldn't be getting reported with
report_xfail().I find this terminology confusing. For instance, there are some tests which are probabilistic (e.g.,
test_sti_nmi) - let’s assume you expect one to fail and it passes, would you say that you encountered a failure?
When testing something probabilistic you should take enough samples that you can check for the expected frequency of the event.
xfail means the test is expected to fail. If it doesn't fail then the software under test changed since the writing of the test. While it's possible that something got fixed (turning an xfail to a pass), it's also possible that the xfail started passing because something got even more broken then before. The big, fat FAIL alerts testers to it.
Quoted 5 lines
Quoted 2 lines
Why would one want to run old kvm-unit-tests on new kvm?
I can think of a couple of reasons, but I am not going to argue too much.
I'm not arguing either. I'm curious. I can't really see why one would want to test with old kvm-unit-tests unless bisecting a problem with new kvm-unit-tests. And, if new kvm-unit-tests brings in some undesired dependency, then we should discuss removing it, rather than avoid the new versions. (That's just a hypothetical if, because I can't think of any new dependencies we've added.)
Thanks,
drew
Paolo Bonzini
maintainerQuoted 6 lines · Nadav Amit
Quoted 2 lines
It's not "failing", it's failing. If a test is expected to pass then it shouldn't be getting reported with
report_xfail().I find this terminology confusing. For instance, there are some tests which are probabilistic (e.g.,
test_sti_nmi) - let’s assume you expect one to fail and it passes, would you say that you encountered a failure?
Yes. :) Probabilistic tests should be changed so that the probability of an incorrect result is very, very small. XPASS are for known bugs or known virtualization holes, not for probabilistic tests. Basically, all they do is spare you from having to invert the result of the test, so that you can write
// i actually isn't zero
report_xfail("i should be zero", true, i == 0);instead of
report_xfail("i should be zero, but isn't", i != 0);XPASS tests are a pleasant kind of failure, but still a surprise that should be inspected.
There are several testsuite harnesses that fail on XPASS (dejagnu and meson, for example), and others that succeed on XPASS (for example "prove", the original TAP client). In other cases again it's customizable, for example PyTest ignores both XFAIL and XPASS results, but it does have an "xfail_strict" option where XPASS will cause the test suite to fail.
Paolo
Quoted 25 lines · Paolo Bonzini
Quoted 5 lines · Nadav Amit
Quoted 2 lines
It's not "failing", it's failing. If a test is expected to pass then it shouldn't be getting reported with
report_xfail().I find this terminology confusing. For instance, there are some tests which are probabilistic (e.g.,
test_sti_nmi) - let’s assume you expect one to fail and it passes, would you say that you encountered a failure?Yes. :) Probabilistic tests should be changed so that the probability of an incorrect result is very, very small. XPASS are for known bugs or known virtualization holes, not for probabilistic tests. Basically, all they do is spare you from having to invert the result of the test, so that you can write
// i actually isn't zero report_xfail("i should be zero", true, i == 0);instead of
report_xfail("i should be zero, but isn't", i != 0);XPASS tests are a pleasant kind of failure, but still a surprise that should be inspected.
Quoted 6 lines
There are several testsuite harnesses that fail on XPASS (dejagnu and meson, for example), and others that succeed on XPASS (for example "prove", the original TAP client). In other cases again it's customizable, for example PyTest ignores both XFAIL and XPASS results, but it does have an "
xfail_strict" option where XPASS will cause the test suite to fail.
Thanks you both (you and Andrew) for the detailed explanations. I actually did not know XPASS/XFAIL are a common terminology.