Skip to contentSkip to conversation

runner: Also consider TAINT_MACHINE_CHECK as abortable taint

PATCH In review Petri Latvala 4 participants 9 messages Jun 5, 2019 – Jun 6, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 runner: Also consider TAINT_MACHINE_CHECK as abortable taint

---
 runner/executor.c | 1 +
 1 file changed, 1 insertion(+)
runner/executor.c +1 -0
index 7e5fbe8f..69ef21e5 100644
@@ -142,6 +142,7 @@ static const struct {
 	unsigned long bit;
 	const char *explanation;
 } abort_taints[] = {
+  {(1 << 4), "TAINT_MACHINE_CHECK: Processor reported a Machine Check Exception."},
   {(1 << 5), "TAINT_BAD_PAGE: Bad page reference or an unexpected page flags."},
   {(1 << 7), "TAINT_DIE: Kernel has died - BUG/OOPS."},
   {(1 << 9), "TAINT_WARN: WARN_ON has happened."},
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#1 lore raw
Quoted 2 lines · Petri Latvala

I've seen lots of machines where these happen as normal side-effect of thermal throtlling. For some value of "normal".

Do we really want to reboot on these? It could be like the network thing I recently disabled, and then everyone started screaming because our machines where constantly rebooting due to network cards/drivers temporarily having a bad time (but usually recovering). -Daniel

Quoted 23 lines
---
 runner/executor.c | 1 +
 1 file changed, 1 insertion(+)
runner/executor.c +1 -0
index 7e5fbe8f..69ef21e5 100644
@@ -142,6 +142,7 @@ static const struct {
 	unsigned long bit;
 	const char *explanation;
 } abort_taints[] = {
+  {(1 << 4), "TAINT_MACHINE_CHECK: Processor reported a Machine Check Exception."},
   {(1 << 5), "TAINT_BAD_PAGE: Bad page reference or an unexpected page flags."},
   {(1 << 7), "TAINT_DIE: Kernel has died - BUG/OOPS."},
   {(1 << 9), "TAINT_WARN: WARN_ON has happened."},
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#2 lore raw
Quoted 7 lines · Daniel Vetter
Quoted 2 lines · Petri Latvala

I've seen lots of machines where these happen as normal side-effect of thermal throtlling. For some value of "normal".

Thermals could be ok and filtered if we can be somewhat certain that the state didn't suffer. Can we?

Quoted 4 lines

Do we really want to reboot on these? It could be like the network thing I recently disabled, and then everyone started screaming because our machines where constantly rebooting due to network cards/drivers temporarily having a bad time (but usually recovering).

Reboot? This won't get fixed by reboot as the hw is busted.

The question is that is it fruitful to weed out software failures with a broken hardware.

-Mika

Quoted 20 lines

-Daniel

Quoted 19 lines
---
 runner/executor.c | 1 +
 1 file changed, 1 insertion(+)
runner/executor.c +1 -0
index 7e5fbe8f..69ef21e5 100644
@@ -142,6 +142,7 @@ static const struct {
 	unsigned long bit;
 	const char *explanation;
 } abort_taints[] = {
+  {(1 << 4), "TAINT_MACHINE_CHECK: Processor reported a Machine Check Exception."},
   {(1 << 5), "TAINT_BAD_PAGE: Bad page reference or an unexpected page flags."},
   {(1 << 7), "TAINT_DIE: Kernel has died - BUG/OOPS."},
   {(1 << 9), "TAINT_WARN: WARN_ON has happened."},
Signature · 1 line
2.19.1
Quoted 13 lines
Quoted 4 lines

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#3 lore raw
Quoted 18 lines · Mika Kuoppala
Quoted 7 lines · Daniel Vetter
Quoted 2 lines · Petri Latvala

I've seen lots of machines where these happen as normal side-effect of thermal throtlling. For some value of "normal".

Thermals could be ok and filtered if we can be somewhat certain that the state didn't suffer. Can we?

Quoted 4 lines

Do we really want to reboot on these? It could be like the network thing I recently disabled, and then everyone started screaming because our machines where constantly rebooting due to network cards/drivers temporarily having a bad time (but usually recovering).

Quoted 4 lines

Reboot? This won't get fixed by reboot as the hw is busted.

The question is that is it fruitful to weed out software failures with a broken hardware.

Yeah, I forgot to mention this aspect.

We don't want to just reboot on this: We want to

1) never file a cibuglog filter on TAINT_MACHINE_CHECK. We want that to always be critical.
2) if we get that abort, replace the hw.

Whether that's possible depends on when/if we get that particular taint happening...

Signature · 5 lines
Petri Latvala
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#4 lore raw
In reply to Daniel Vetter
Quoted 11 lines · Daniel Vetter
Quoted 2 lines · Petri Latvala

I've seen lots of machines where these happen as normal side-effect of thermal throtlling. For some value of "normal".

Do we really want to reboot on these? It could be like the network thing I recently disabled, and then everyone started screaming because our machines where constantly rebooting due to network cards/drivers temporarily having a bad time (but usually recovering).

I've seen some MCE log messages on dmesgs, quite often on one of the BXTs for example. How often those MCE triggers caused taint is another question.

Reading the mce code, it seems to be thermal _failure_ that causes a taint. And all of these add_taint() calls also use LOCKDEP_NOW_UNRELIABLE so we're already deep under the bus if we get that taint.

Signature · 5 lines
Petri Latvala
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#5 lore raw
Quoted 14 lines · Petri Latvala
Quoted 11 lines · Daniel Vetter
Quoted 2 lines · Petri Latvala

I've seen lots of machines where these happen as normal side-effect of thermal throtlling. For some value of "normal".

Do we really want to reboot on these? It could be like the network thing I recently disabled, and then everyone started screaming because our machines where constantly rebooting due to network cards/drivers temporarily having a bad time (but usually recovering).

Quoted 8 lines

I've seen some MCE log messages on dmesgs, quite often on one of the BXTs for example. How often those MCE triggers caused taint is another question.

Reading the mce code, it seems to be thermal _failure_ that causes a taint. And all of these add_taint() calls also use LOCKDEP_NOW_UNRELIABLE so we're already deep under the bus if we get that taint.

Hm ok if lockdep is gone then we reboot anyway. I guess ack from me then.

Since the goal is to throw these machines out, shouldn't we have a special error for these? Instead of just aborted I mean.
-Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
Reviewed-by on v1 1/1
#6 lore raw
In reply to Petri Latvala
Quoted 2 lines · Petri Latvala

If we get one, the machine should be removed from the pool.

Quoted 18 lines
---
 runner/executor.c | 1 +
 1 file changed, 1 insertion(+)
runner/executor.c +1 -0
index 7e5fbe8f..69ef21e5 100644
@@ -142,6 +142,7 @@ static const struct {
 	unsigned long bit;
 	const char *explanation;
 } abort_taints[] = {
+  {(1 << 4), "TAINT_MACHINE_CHECK: Processor reported a Machine Check Exception."},
   {(1 << 5), "TAINT_BAD_PAGE: Bad page reference or an unexpected page flags."},
   {(1 << 7), "TAINT_DIE: Kernel has died - BUG/OOPS."},
   {(1 << 9), "TAINT_WARN: WARN_ON has happened."},
Signature · 1 line
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#7 lore raw

[igt-dev] ✓ Fi.CI.BAT: success for runner: Also consider TAINT_MACHINE_CHECK as abortable taint

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: runner: Also consider TAINT_MACHINE_CHECK as abortable taint
URL : https://patchwork.freedesktop.org/series/61651/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_6195 -> IGTPW_3116
====================================================

Summary
-------

**SUCCESS**

No regressions found.

External URL: https://patchwork.freedesktop.org/api/1.0/series/61651/revisions/1/mbox/

Known issues
------------

Here are the changes found in IGTPW_3116 that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@i915_module_load@reload:
- fi-icl-u3: [PASS][1] -> [DMESG-WARN][2] ([fdo#106107])
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/fi-icl-u3/igt@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/fi-icl-u3/igt@[email protected]

* igt@kms_cursor_legacy@basic-flip-after-cursor-legacy:
- fi-icl-u3: [PASS][3] -> [DMESG-WARN][4] ([fdo#107724]) +2 similar issues
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/fi-icl-u3/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/fi-icl-u3/igt@[email protected]

#### Possible fixes ####

* igt@kms_frontbuffer_tracking@basic:
- fi-hsw-peppy: [DMESG-WARN][5] ([fdo#102614]) -> [PASS][6]
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/fi-hsw-peppy/igt@[email protected]
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/fi-hsw-peppy/igt@[email protected]

* igt@prime_vgem@basic-fence-flip:
- fi-icl-u3: [DMESG-WARN][7] ([fdo#107724]) -> [PASS][8] +1 similar issue
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/fi-icl-u3/igt@[email protected]
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/fi-icl-u3/igt@[email protected]

[fdo#102614]: https://bugs.freedesktop.org/show_bug.cgi?id=102614
[fdo#106107]: https://bugs.freedesktop.org/show_bug.cgi?id=106107
[fdo#107724]: https://bugs.freedesktop.org/show_bug.cgi?id=107724

Participating hosts (52 -> 46)
------------------------------

Additional (1): fi-apl-guc
Missing (7): fi-ilk-m540 fi-hsw-4200u fi-byt-squawks fi-bsw-cyan fi-kbl-7560u fi-byt-clapper fi-bdw-samus

Build changes
-------------

* IGT: IGT_5039 -> IGTPW_3116

CI_DRM_6195: 06b71939f2477c76f9eecb1dd5e99dcb25cb8371 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_3116: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/
IGT_5039: 2d4f470bba1cb51ed116fb80b170f717c6294714 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#8 lore raw

[igt-dev] ✓ Fi.CI.IGT: success for runner: Also consider TAINT_MACHINE_CHECK as abortable taint

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: runner: Also consider TAINT_MACHINE_CHECK as abortable taint
URL : https://patchwork.freedesktop.org/series/61651/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_6195_full -> IGTPW_3116_full
====================================================

Summary
-------

**SUCCESS**

No regressions found.

External URL: https://patchwork.freedesktop.org/api/1.0/series/61651/revisions/1/mbox/

Possible new issues
-------------------

Here are the unknown changes that may have been introduced in IGTPW_3116_full:

### IGT changes ###

#### Suppressed ####

The following results come from untrusted machines, tests, or statuses. They do not affect the overall result.

* {igt@gem_ctx_shared@exec-shared-gtt-render}:
- shard-apl: NOTRUN -> [FAIL][1]
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-apl6/igt@[email protected]

* {igt@gem_exec_balancer@smoke}:
- shard-iclb: [PASS][2] -> [SKIP][3]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-iclb1/igt@[email protected]
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-iclb3/igt@[email protected]

Known issues
------------

Here are the changes found in IGTPW_3116_full that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@gem_eio@reset-stress:
- shard-snb: [PASS][4] -> [FAIL][5] ([fdo#109661])
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-snb1/igt@[email protected]
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-snb6/igt@[email protected]

* igt@gem_mmap_gtt@forked-medium-copy-odd:
- shard-iclb: [PASS][6] -> [INCOMPLETE][7] ([fdo#107713])
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-iclb8/igt@[email protected]
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-iclb8/igt@[email protected]

* igt@gem_softpin@noreloc-s3:
- shard-apl: [PASS][8] -> [DMESG-WARN][9] ([fdo#108566]) +3 similar issues
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-apl8/igt@[email protected]
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-apl5/igt@[email protected]

* igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-blt:
- shard-hsw: [PASS][10] -> [SKIP][11] ([fdo#109271]) +7 similar issues
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-hsw2/igt@[email protected]
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-hsw1/igt@[email protected]

* igt@kms_frontbuffer_tracking@fbc-badstride:
- shard-iclb: [PASS][12] -> [FAIL][13] ([fdo#103167]) +4 similar issues
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-iclb8/igt@[email protected]
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-iclb7/igt@[email protected]

* igt@kms_setmode@basic:
- shard-kbl: [PASS][14] -> [FAIL][15] ([fdo#99912])
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-kbl3/igt@[email protected]
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-kbl1/igt@[email protected]

* igt@kms_sysfs_edid_timing:
- shard-hsw: [PASS][16] -> [FAIL][17] ([fdo#100047])
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-hsw4/igt@kms_sysfs_edid_timing.html
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-hsw1/igt@kms_sysfs_edid_timing.html

#### Possible fixes ####

* {igt@gem_exec_balancer@bonded-imm}:
- shard-iclb: [FAIL][18] -> [PASS][19]
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-iclb4/igt@[email protected]
[19]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-iclb3/igt@[email protected]

* igt@gem_tiled_swapping@non-threaded:
- shard-apl: [DMESG-WARN][20] ([fdo#108686]) -> [PASS][21]
[20]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-apl2/igt@[email protected]
[21]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-apl5/igt@[email protected]

* igt@gem_workarounds@suspend-resume-context:
- shard-apl: [DMESG-WARN][22] ([fdo#108566]) -> [PASS][23] +4 similar issues
[22]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-apl6/igt@[email protected]
[23]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-apl8/igt@[email protected]

* igt@kms_flip@flip-vs-suspend:
- shard-kbl: [INCOMPLETE][24] ([fdo#103665]) -> [PASS][25]
[24]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-kbl6/igt@[email protected]
[25]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-kbl1/igt@[email protected]

* igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-render:
- shard-iclb: [FAIL][26] ([fdo#103167]) -> [PASS][27] +1 similar issue
[26]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-iclb7/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-render.html
[27]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-iclb3/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-indfb-draw-render.html

* igt@kms_frontbuffer_tracking@fbc-1p-rte:
- shard-iclb: [FAIL][28] ([fdo#103167] / [fdo#110378]) -> [PASS][29]
[28]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-iclb7/igt@[email protected]
[29]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-iclb5/igt@[email protected]

* igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-pwrite:
- shard-hsw: [SKIP][30] ([fdo#109271]) -> [PASS][31] +16 similar issues
[30]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-hsw1/igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-pwrite.html
[31]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-hsw2/igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-cur-indfb-draw-pwrite.html

* igt@kms_psr2_su@page_flip:
- shard-iclb: [SKIP][32] ([fdo#109642]) -> [PASS][33]
[32]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-iclb1/igt@kms_psr2_su@page_flip.html
[33]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-iclb2/igt@kms_psr2_su@page_flip.html

* igt@kms_psr@psr2_cursor_plane_move:
- shard-iclb: [SKIP][34] ([fdo#109441]) -> [PASS][35]
[34]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-iclb1/igt@kms_psr@psr2_cursor_plane_move.html
[35]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-iclb2/igt@kms_psr@psr2_cursor_plane_move.html

* igt@kms_setmode@basic:
- shard-apl: [FAIL][36] ([fdo#99912]) -> [PASS][37]
[36]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6195/shard-apl8/igt@[email protected]
[37]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/shard-apl4/igt@[email protected]

{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).

[fdo#100047]: https://bugs.freedesktop.org/show_bug.cgi?id=100047
[fdo#103167]: https://bugs.freedesktop.org/show_bug.cgi?id=103167
[fdo#103665]: https://bugs.freedesktop.org/show_bug.cgi?id=103665
[fdo#107713]: https://bugs.freedesktop.org/show_bug.cgi?id=107713
[fdo#108566]: https://bugs.freedesktop.org/show_bug.cgi?id=108566
[fdo#108686]: https://bugs.freedesktop.org/show_bug.cgi?id=108686
[fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
[fdo#109441]: https://bugs.freedesktop.org/show_bug.cgi?id=109441
[fdo#109642]: https://bugs.freedesktop.org/show_bug.cgi?id=109642
[fdo#109661]: https://bugs.freedesktop.org/show_bug.cgi?id=109661
[fdo#110378]: https://bugs.freedesktop.org/show_bug.cgi?id=110378
[fdo#99912]: https://bugs.freedesktop.org/show_bug.cgi?id=99912

Participating hosts (10 -> 6)
------------------------------

Missing (4): pig-skl-6260u shard-skl pig-hsw-4770r pig-glk-j5005

Build changes
-------------

* IGT: IGT_5039 -> IGTPW_3116
* Piglit: piglit_4509 -> None

CI_DRM_6195: 06b71939f2477c76f9eecb1dd5e99dcb25cb8371 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_3116: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/
IGT_5039: 2d4f470bba1cb51ed116fb80b170f717c6294714 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
piglit_4509: fdc5a4ca11124ab8413c7988896eec4c97336694 @ git://anongit.freedesktop.org/piglit

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3116/
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev