tests/igt: dmesg noise is a kernel failure
PATCH 1/1 tests/igt: dmesg noise is a kernel failure
At least when testing the kernel. In normal programs pretty much all the dmesg noise would simply be replaced by debug asserts, but in the kernel we try rely hard to not fall over minor inconsistencies.
Still for CI purposes there's not really a difference, hence don't treat it as such.
Motivated since once again I've seen a statistics where this was split up, and then a reduction of "failures" (but in reality just trading them in for more "warnings") praised as success.
--- tests/igt.py | 5 +++++ 1 file changed, 5 insertions(+)
tests/igt.py +5 -0
@@ -53,6 +53,7 @@ def check_environment(): """ debugfs_path = "/sys/kernel/debug/dri" + if os.getuid() != 0: raise exceptions.PiglitInternalError( "Test Environment check: not root!")
@@ -123,6 +124,10 @@ class IGTTest(Test): else: self.result.result = 'fail' + # all dmesg noise is considered a test failure when testing the kernel + if self.result.dmesg + self.result.result = 'fail' + def list_tests(listname): """Parse igt test list and return them as a list."""
Signature · 6 lines
2.9.3 _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 10 lines · Daniel Vetter
At least when testing the kernel. In normal programs pretty much all the dmesg noise would simply be replaced by debug asserts, but in the kernel we try rely hard to not fall over minor inconsistencies.
Still for CI purposes there's not really a difference, hence don't treat it as such.
Motivated since once again I've seen a statistics where this was split up, and then a reduction of "failures" (but in reality just trading them in for more "warnings") praised as success.
Hear, hear! dfail == fail, and dwarn == warn.
Calling a failure, a dfail just softens that it failed. -Chris
Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
A message that is not in the archive (sent off-list or not yet received): 20161006091629.GI22676-aII6DKEyn0pWYbfKqPwjAkR8Iwp7RQ6xAL8bYrjMMd8@public.gmane.org
Quoted 8 lines · Chris Wilson
Quoted 7 lines · Daniel Vetter
At least when testing the kernel. In normal programs pretty much all the dmesg noise would simply be replaced by debug asserts, but in the kernel we try rely hard to not fall over minor inconsistencies.
Still for CI purposes there's not really a difference, hence don't treat it as such.
Quoted 7 lines
Quoted 3 lines
Motivated since once again I've seen a statistics where this was split up, and then a reduction of "failures" (but in reality just trading them in for more "warnings") praised as success.
Hear, hear! dfail == fail, and dwarn == warn.
Calling a failure, a dfail just softens that it failed.
Maybe a bit more motiviation why I think all dmesg noise should be tracked as a full-on failure. In userspace there's tons of consistency checks, and those are done using assert(). Debug builds have them, production ones dont, and if you hit it your app dies, and it's counted as a failure under CI.
In the kernel it's not great if we just die, so we try very hard to limp along in all these cases instead of dying. But it's still a consistency check firing, and imo should be treated as such. "warn" would then just be for for igt self-checks in the test library/infrastructure itself.
The other reasons is CI: "warn" essentially means "something might be wrong, a human needs to check". That's not all that useful for automating stuff, hence why I think we really shouldn't have that status, at least as a usual one. If we go with relabelling, then a consequence is also that we might need to remove some more questionable in-kernel checks.
Re Ville's concern: I don't mind whether we call it dmesg-fail or fail in the end. If it helps folks to scan through results I'm fine with changing it to dmesg-fail. But dmesg-warn imo needs to go (for igt). I think dmesg-warn is the right choice for everything else, where some kernel noise might not always mean that there's an issue with your userspace. So again needs a human to check what's going on. -Daniel
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Piglit mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/piglit
Quoted 31 lines · Daniel Vetter
At least when testing the kernel. In normal programs pretty much all the dmesg noise would simply be replaced by debug asserts, but in the kernel we try rely hard to not fall over minor inconsistencies.
Still for CI purposes there's not really a difference, hence don't treat it as such.
Motivated since once again I've seen a statistics where this was split up, and then a reduction of "failures" (but in reality just trading them in for more "warnings") praised as success.
Cc Petri Latvala <[email protected]>Signed-off-by Daniel Vetter <[email protected]>--- tests/igt.py | 5 +++++ 1 file changed, 5 insertions(+)tests/igt.py +1
-0@@ -53,6 +53,7 @@ def check_environment(): """ debugfs_path = "/sys/kernel/debug/dri" + if os.getuid() != 0: raise exceptions.PiglitInternalError( "Test Environment check: not root!")
tests/igt.py class IGTTest(Test): · line 129
Quoted 6 lines
tests/igt.py +3
-0@@ -123,6 +124,10 @@ class IGTTest(Test): else: self.result.result = 'fail' + # all dmesg noise is considered a test failure when testing the kernel + if self.result.dmesg + self.result.result = 'fail'
Hmm. So now all errors will look the same? I think I'm usually more interested in the dmesg warns, so this would perhaps make it harder for me to spot them. Not sure I like that.
Quoted 11 lines
tests/igt.py +1
-0+ def list_tests(listname): """Parse igt test list and return them as a list."""Signature · 6 lines
2.9.3 _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Signature · 6 lines
Ville Syrjälä Intel OTC _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
A message that is not in the archive (sent off-list or not yet received): 20161006090717.23527-1-daniel.vetter-/[email protected]
Dylan Baker
Acked-byQuoting Daniel Vetter (2016-10-06 02:07:17)
Quoted 48 lines
At least when testing the kernel. In normal programs pretty much all the dmesg noise would simply be replaced by debug asserts, but in the kernel we try rely hard to not fall over minor inconsistencies.
Still for CI purposes there's not really a difference, hence don't treat it as such.
Motivated since once again I've seen a statistics where this was split up, and then a reduction of "failures" (but in reality just trading them in for more "warnings") praised as success.
Cc Petri Latvala <[email protected]>Signed-off-by Daniel Vetter <[email protected]>--- tests/igt.py | 5 +++++ 1 file changed, 5 insertions(+)tests/igt.py +5
-0@@ -53,6 +53,7 @@ def check_environment(): """ debugfs_path = "/sys/kernel/debug/dri" + if os.getuid() != 0: raise exceptions.PiglitInternalError( "Test Environment check: not root!")@@ -123,6 +124,10 @@ class IGTTest(Test): else: self.result.result = 'fail' + # all dmesg noise is considered a test failure when testing the kernel + if self.result.dmesg + self.result.result = 'fail' + def list_tests(listname): """Parse igt test list and return them as a list."""Signature · 6 lines
2.9.3 _______________________________________________ Piglit mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/piglit
makes sense to me,
_______________________________________________
Piglit mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/piglit