Skip to contentSkip to conversation

tests/igt: dmesg noise is a kernel failure

PATCH In review Daniel Vetter 4 participants 4 messages Oct 7, 2016

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tests/igt: dmesg noise is a kernel failure

At least when testing the kernel. In normal programs pretty much all the dmesg noise would simply be replaced by debug asserts, but in the kernel we try rely hard to not fall over minor inconsistencies.

Still for CI purposes there's not really a difference, hence don't treat it as such.

Motivated since once again I've seen a statistics where this was split up, and then a reduction of "failures" (but in reality just trading them in for more "warnings") praised as success.

v2: Clamp to "dmesg-fail" to keep dmesg noise easily identifiable (Ville).

---
 tests/igt.py | 4 ++++
 1 file changed, 4 insertions(+)
tests/igt.py +4 -0
index 7ebb03646b50..21e55e115654 100644
@@ -123,6 +123,10 @@ class IGTTest(Test):
         else:
             self.result.result = 'fail'
 
+        # all dmesg noise is considered a test failure when testing the kernel
+        if self.result.dmesg
+            self.result.result = 'dmesg-fail'
+
 
 def list_tests(listname):
     """Parse igt test list and return them as a list."""
Signature · 6 lines
2.9.3

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#1 lore raw
Quoted 29 lines · Daniel Vetter

At least when testing the kernel. In normal programs pretty much all the dmesg noise would simply be replaced by debug asserts, but in the kernel we try rely hard to not fall over minor inconsistencies.

Still for CI purposes there's not really a difference, hence don't treat it as such.

Motivated since once again I've seen a statistics where this was split up, and then a reduction of "failures" (but in reality just trading them in for more "warnings") praised as success.

v2: Clamp to "dmesg-fail" to keep dmesg noise easily identifiable (Ville).

---
 tests/igt.py | 4 ++++
 1 file changed, 4 insertions(+)
tests/igt.py +0 -0
index 7ebb03646b50..21e55e115654 100644

tests/igt.py class IGTTest(Test): · line 128

Quoted 6 lines
tests/igt.py +3 -0
@@ -123,6 +123,10 @@ class IGTTest(Test):
         else:
             self.result.result = 'fail'
 
+        # all dmesg noise is considered a test failure when testing the kernel
+        if self.result.dmesg
+            self.result.result = 'dmesg-fail'

This is changing a fail to dmesg-fail. I hate that. -Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#2 lore raw

Double checking the full picture of the impact:
- Current - test result value list for igt (=default piglit): pass, warn, dmesg-warn, fail, dmesg-fail, timeout, crash, incomplete
- Intent with proposal below - test result value list for igt (kernel testing): pass, warn, fail, dmesg-fail, timeout, crash, incomplete Right or wrong?
Or will fail vanish from this list too like Chris said? Or will there be scenario where one can have fail without dmesg?

BR, Jari

-----Original Message-----
From: Chris Wilson [mailto:[email protected]]
Sent: Friday, October 7, 2016 10:12 AM
To: Daniel Vetter <[email protected]>
Cc: Intel Graphics Development <[email protected]>; piglit discussion list <[email protected]>; Ville Syrjälä <[email protected]>; Tahvanainen, Jari <[email protected]>; Latvala, Petri <[email protected]>; Vetter, Daniel <[email protected]>
Subject: Re: [PATCH] tests/igt: dmesg noise is a kernel failure

Quoted 29 lines · Daniel Vetter

At least when testing the kernel. In normal programs pretty much all the dmesg noise would simply be replaced by debug asserts, but in the kernel we try rely hard to not fall over minor inconsistencies.

Still for CI purposes there's not really a difference, hence don't treat it as such.

Motivated since once again I've seen a statistics where this was split up, and then a reduction of "failures" (but in reality just trading them in for more "warnings") praised as success.

v2: Clamp to "dmesg-fail" to keep dmesg noise easily identifiable (Ville).

---
 tests/igt.py | 4 ++++
 1 file changed, 4 insertions(+)
tests/igt.py index renamed from tests/igt.py +0 -0

7ebb03646b50..21e55e115654 100644

tests/igt.py class IGTTest(Test): · line 128

Quoted 6 lines
tests/igt.py +3 -0
@@ -123,6 +123,10 @@ class IGTTest(Test):
         else:
             self.result.result = 'fail'
 
+        # all dmesg noise is considered a test failure when testing the kernel
+        if self.result.dmesg
+            self.result.result = 'dmesg-fail'

This is changing a fail to dmesg-fail. I hate that. -Chris

--
Chris Wilson, Intel Open Source Technology Centre
---------------------------------------------------------------------
Intel Finland Oy
Registered Address: PL 281, 00181 Helsinki
Business Identity Code: 0357606 - 4
Domiciled in Helsinki

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#3 lore raw
In reply to Chris Wilson
Quoted 33 lines · Chris Wilson
Quoted 32 lines · Daniel Vetter

At least when testing the kernel. In normal programs pretty much all the dmesg noise would simply be replaced by debug asserts, but in the kernel we try rely hard to not fall over minor inconsistencies.

Still for CI purposes there's not really a difference, hence don't treat it as such.

Motivated since once again I've seen a statistics where this was split up, and then a reduction of "failures" (but in reality just trading them in for more "warnings") praised as success.

v2: Clamp to "dmesg-fail" to keep dmesg noise easily identifiable (Ville).

---
 tests/igt.py | 4 ++++
 1 file changed, 4 insertions(+)
tests/igt.py +0 -0
index 7ebb03646b50..21e55e115654 100644
@@ -123,6 +123,10 @@ class IGTTest(Test):
         else:
             self.result.result = 'fail'
 
Quoted 5 lines

tests/igt.py class IGTTest(Test): · line 128

Quoted 3 lines
tests/igt.py +3 -0
+        # all dmesg noise is considered a test failure when testing the kernel
+        if self.result.dmesg
+            self.result.result = 'dmesg-fail'

This is changing a fail to dmesg-fail. I hate that.

I don't know if there's a way to win here. We're trying to reduce 4 states to 3 states for whatever reason, so we're going to lose information :(

Doing it this way you'll have to go through every dmesg-fail as well as the fails if you want to see all test failures. Doing it the other way means going through all the fails to find the dmesg warns.

Signature · 6 lines
Ville Syrjälä
Intel OTC
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx