Skip to contentSkip to conversation

drm/i915: Save hangcheck score across resets

PATCH In review Mika Kuoppala 3 participants 3 messages Oct 6, 2016

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/i915: Save hangcheck score across resets

Hangcheck score has been zeroed on engine init, which happens after reset recovery. This has worked well as we always reset all engines on hang, and also discarded all work submitted to engines.

With commit 821ed7df6e2a ("drm/i915: Update reset path to fix incomplete requests") driver gained capability to only discard the request or requests that were directly involved with the hang, and those who were deemed innocent, were replayed intact.

Our hangcheck works by periodically sampling the engine state and then doing checks in multiple stages to see if engine is making progress. The engine capabilities differ. With render engine, we have a more ways to measure the progress and thus more checks and stages. With other engines, we only sample the seqno and head movement.

Now consider that blitter engine is waiting on render and render engine has a batch which has stuck. Due to simpler checks, the blitter engine hangcheck score accumulates faster and reaches reset threshold quicker. We also blame the blitter for the hang as it had the highest score when recovery started.

Blaming the wrong engine, we don't find the actual guilty request and most critically, won't make any progress after the reset. That will lead to second hang, with the same pattern, ad infinitum.

Previously the false blaming of engine was not critical as score was only used as a trigger for full reset and debug aid in error states. But now, the score is essential of finding the culprit request.

To fix this, keep the hangcheck scores across resets. We already have a decay mechanism in place if progress is being made. This ensures that even if we blame the wrong engine once, we don't do it twice or consistently, and the real culprit request will be cleared, real progress will be made and this untangles rest of the engines and lead to successful recovery.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=98104

---
 drivers/gpu/drm/i915/intel_engine_cs.c | 1 -
 1 file changed, 1 deletion(-)
drivers/gpu/drm/i915/intel_engine_cs.c +0 -1
index d00ec805f93d..4bb869eb11bc 100644
@@ -209,7 +209,6 @@ void intel_engine_init_seqno(struct intel_engine_cs *engine, u32 seqno)
 
 void intel_engine_init_hangcheck(struct intel_engine_cs *engine)
 {
-	memset(&engine->hangcheck, 0, sizeof(engine->hangcheck));
 	clear_bit(engine->id, &engine->i915->gpu_error.missed_irq_rings);
 	if (intel_engine_has_waiter(engine))
 		i915_queue_hangcheck(engine->i915);
Signature · 6 lines
2.7.4

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#1 lore raw
Quoted 16 lines · Mika Kuoppala

Hangcheck score has been zeroed on engine init, which happens after reset recovery. This has worked well as we always reset all engines on hang, and also discarded all work submitted to engines.

With commit 821ed7df6e2a ("drm/i915: Update reset path to fix incomplete requests") driver gained capability to only discard the request or requests that were directly involved with the hang, and those who were deemed innocent, were replayed intact.

Our hangcheck works by periodically sampling the engine state and then doing checks in multiple stages to see if engine is making progress. The engine capabilities differ. With render engine, we have a more ways to measure the progress and thus more checks and stages. With other engines, we only sample the seqno and head movement.

Quoted 5 lines

Now consider that blitter engine is waiting on render and render engine has a batch which has stuck. Due to simpler checks, the blitter engine hangcheck score accumulates faster and reaches reset threshold quicker. We also blame the blitter for the hang as it had the highest score when recovery started.

This is the bug. It shouldn't accumulate any score in this case as the engine is not active.

This patch is not the right approach for the issue as described here. Because as soon as the blitter engine is active again, there is a very real danger of it being declared guilty and reset.

The patch has merit, but not for this issue...
-Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#2 lore raw

✗ Fi.CI.BAT: failure for drm/i915: Save hangcheck score across resets

In reply to Mika Kuoppala
== Series Details ==

== Series Details ==

Series: drm/i915: Save hangcheck score across resets
URL : https://patchwork.freedesktop.org/series/13367/
State : failure

== Summary ==

Series 13367v1 drm/i915: Save hangcheck score across resets
https://patchwork.freedesktop.org/api/1.0/series/13367/revisions/1/mbox/

Test gem_close_race:

        Subgroup basic-threads:
                pass       -> FAIL       (fi-snb-2600)
                pass       -> FAIL       (fi-bsw-n3050)
                pass       -> FAIL       (fi-byt-j1900)
                pass       -> FAIL       (fi-ivb-3770)
                pass       -> FAIL       (fi-hsw-4770)
                pass       -> FAIL       (fi-hsw-4770r)
                pass       -> FAIL       (fi-byt-n2820)
                pass       -> FAIL       (fi-bxt-t5700)
                pass       -> FAIL       (fi-skl-6260u)
                pass       -> FAIL       (fi-ivb-3520m)
                pass       -> FAIL       (fi-bdw-5557u)
                pass       -> FAIL       (fi-snb-2520m)
                pass       -> FAIL       (fi-skl-6700hq)
Test kms_pipe_crc_basic:
        Subgroup suspend-read-crc-pipe-a:
                dmesg-warn -> PASS       (fi-byt-j1900)
        Subgroup suspend-read-crc-pipe-b:
                dmesg-warn -> PASS       (fi-byt-j1900)
fi-bdw-5557u     total:244  pass:228  dwarn:0   dfail:0   fail:1   skip:15 
fi-bsw-n3050     total:244  pass:201  dwarn:0   dfail:0   fail:1   skip:42 
fi-bxt-t5700     total:244  pass:213  dwarn:0   dfail:0   fail:1   skip:30 
fi-byt-j1900     total:244  pass:211  dwarn:0   dfail:0   fail:2   skip:31 
fi-byt-n2820     total:244  pass:207  dwarn:0   dfail:0   fail:2   skip:35 
fi-hsw-4770      total:244  pass:221  dwarn:0   dfail:0   fail:1   skip:22 
fi-hsw-4770r     total:244  pass:221  dwarn:0   dfail:0   fail:1   skip:22 
fi-ilk-650       total:244  pass:182  dwarn:0   dfail:0   fail:2   skip:60 
fi-ivb-3520m     total:244  pass:218  dwarn:0   dfail:0   fail:1   skip:25 
fi-ivb-3770      total:244  pass:206  dwarn:0   dfail:0   fail:1   skip:37 
fi-kbl-7200u     total:244  pass:220  dwarn:0   dfail:0   fail:0   skip:24 
fi-skl-6260u     total:244  pass:229  dwarn:0   dfail:0   fail:1   skip:14 
fi-skl-6700hq    total:244  pass:221  dwarn:0   dfail:0   fail:1   skip:22 
fi-skl-6700k     total:244  pass:219  dwarn:1   dfail:0   fail:0   skip:24 
fi-skl-6770hq    total:244  pass:228  dwarn:1   dfail:0   fail:1   skip:14 
fi-snb-2520m     total:244  pass:207  dwarn:0   dfail:0   fail:1   skip:36 
fi-snb-2600      total:244  pass:206  dwarn:0   dfail:0   fail:1   skip:37 

Results at /archive/results/CI_IGT_test/Patchwork_2635/

2dff18acaa95a26b882a5f9910d7ded514f18415 drm-intel-nightly: 2016y-10m-05d-13h-58m-08s UTC integration manifest b60a823 drm/i915: Save hangcheck score across resets

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx