Skip to contentSkip to conversation

Per Engine hang detection and recovery

PATCH In review Siluvery, Arun 2 participants 4 messages Nov 11, 2013

lore
Share & tools
lore
#0 lore raw

cover Per Engine hang detection and recovery

From: "Siluvery, Arun" <[email protected]>

This patchset contains changes for Timeout detection and recovery (TDR) which provides per-engine hang detection and recovery.
The current driver performs full gpu reset in case of a hang, TDR attempts to only reset the engine that is hung and it falls back to full reset if it fails.

Full GPU reset can leave the system in a state where the display updates intermittently and possibly lock-up depending on the work load at the time of hang. TDR can help recover the system in those case thus increasing the stability.

The changes are split in multiple patches.
1. Ring utility functions to save/restore context, reset ring etc
2. TDR hang detection logic and error recovery function
3. Debugfs changes to export TDR statistics.

I have tested these changes on drm-intel-nightly with simple test which inserts a bad batch buffer on the specific to trigger a hang. TDR logic then detects this and recovers from it by skipping the bad batch.

Please review and give your comments.

regards
Arun

Siluvery, Arun (3):
drm/1915: Add ring functions to save/restore context for per-ring
reset
drm/i915: Per-engine Timeout detection and recovery on HSW drm/i915: Export TDR hang count to debugfs

 drivers/gpu/drm/i915/i915_debugfs.c     |  68 +++-
 drivers/gpu/drm/i915/i915_dma.c         |  16 +-
 drivers/gpu/drm/i915/i915_drv.c         | 195 +++++++++-
 drivers/gpu/drm/i915/i915_drv.h         |  92 ++++-
 drivers/gpu/drm/i915/i915_gem.c         |  77 +++-
 drivers/gpu/drm/i915/i915_gpu_error.c   |  25 +-
 drivers/gpu/drm/i915/i915_irq.c         | 556 ++++++++++++++++-------------
 drivers/gpu/drm/i915/i915_reg.h         |   7 +
 drivers/gpu/drm/i915/intel_display.c    |  25 +-
 drivers/gpu/drm/i915/intel_ringbuffer.c | 607 +++++++++++++++++++++++++++++++-
 drivers/gpu/drm/i915/intel_ringbuffer.h |  51 +++
 drivers/gpu/drm/i915/intel_uncore.c     |  31 +-
 include/drm/drmP.h                      |   7 +
 13 files changed, 1467 insertions(+), 290 deletions(-)
Signature · 1 line
1.8.4
#1 lore raw
Quoted 10 lines · Arun

From: "Siluvery, Arun" <[email protected]>

This patchset contains changes for Timeout detection and recovery (TDR) which provides per-engine hang detection and recovery.
The current driver performs full gpu reset in case of a hang, TDR attempts to only reset the engine that is hung and it falls back to full reset if it fails.

Full GPU reset can leave the system in a state where the display updates intermittently and possibly lock-up depending on the work load at the time of hang. TDR can help recover the system in those case thus increasing the stability.

Are these hw lockups you've seen with full gpu reset or just kernel deadlocks? If it's the latter we've recently (re-)fixed a bunch of those, and if there are new ones we definitely want to fix them and add testcases to igt. So if you could share some of these hangs and their analysis/testcases that's be very interesting.

That's of course on top of any other reset improvements.

Quoted 8 lines

The changes are split in multiple patches.
1. Ring utility functions to save/restore context, reset ring etc
2. TDR hang detection logic and error recovery function
3. Debugfs changes to export TDR statistics.

I have tested these changes on drm-intel-nightly with simple test which inserts a bad batch buffer on the specific to trigger a hang. TDR logic then detects this and recovers from it by skipping the bad batch.

I want this testcase (as a patch to igt).

Quoted 1 line

Please review and give your comments.

I'll try to have a look later this week, atm still busy with bdw upstreaming. One more meta-comment though: Something with your git setup seems to be broken, the patches don't have in-reply-to headers pointing at this cover letter and hence the threading is a bit broken.

Cheers, Daniel

Quoted 33 lines

regards
Arun

Siluvery, Arun (3):
drm/1915: Add ring functions to save/restore context for per-ring
reset
drm/i915: Per-engine Timeout detection and recovery on HSW drm/i915: Export TDR hang count to debugfs

 drivers/gpu/drm/i915/i915_debugfs.c     |  68 +++-
 drivers/gpu/drm/i915/i915_dma.c         |  16 +-
 drivers/gpu/drm/i915/i915_drv.c         | 195 +++++++++-
 drivers/gpu/drm/i915/i915_drv.h         |  92 ++++-
 drivers/gpu/drm/i915/i915_gem.c         |  77 +++-
 drivers/gpu/drm/i915/i915_gpu_error.c   |  25 +-
 drivers/gpu/drm/i915/i915_irq.c         | 556 ++++++++++++++++-------------
 drivers/gpu/drm/i915/i915_reg.h         |   7 +
 drivers/gpu/drm/i915/intel_display.c    |  25 +-
 drivers/gpu/drm/i915/intel_ringbuffer.c | 607 +++++++++++++++++++++++++++++++-
 drivers/gpu/drm/i915/intel_ringbuffer.h |  51 +++
 drivers/gpu/drm/i915/intel_uncore.c     |  31 +-
 include/drm/drmP.h                      |   7 +
 13 files changed, 1467 insertions(+), 290 deletions(-)
Signature · 7 lines
1.8.4


_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#2 lore raw
Quoted 12 lines · Daniel Vetter
Quoted 10 lines · Arun

From: "Siluvery, Arun" <[email protected]>

This patchset contains changes for Timeout detection and recovery (TDR) which provides per-engine hang detection and recovery.
The current driver performs full gpu reset in case of a hang, TDR attempts to only reset the engine that is hung and it falls back to full reset if it fails.

Full GPU reset can leave the system in a state where the display updates intermittently and possibly lock-up depending on the work load at the time of hang. TDR can help recover the system in those case thus increasing the stability.

Quoted 7 lines

Are these hw lockups you've seen with full gpu reset or just kernel deadlocks? If it's the latter we've recently (re-)fixed a bunch of those, and if there are new ones we definitely want to fix them and add testcases to igt. So if you could share some of these hangs and their analysis/testcases that's be very interesting.

That's of course on top of any other reset improvements.

I think these are kernel lockups, unfortunately when this happens there is no response from the kernel, sending break is also not helping. I will try to get more details on this.

Quoted 11 lines
Quoted 9 lines

The changes are split in multiple patches.
1. Ring utility functions to save/restore context, reset ring etc
2. TDR hang detection logic and error recovery function
3. Debugfs changes to export TDR statistics.

I have tested these changes on drm-intel-nightly with simple test which inserts a bad batch buffer on the specific to trigger a hang. TDR logic then detects this and recovers from it by skipping the bad batch.

I want this testcase (as a patch to igt).

ok, I will send it to the mailing list.

Quoted 7 lines
Quoted 2 lines

Please review and give your comments.

I'll try to have a look later this week, atm still busy with bdw upstreaming. One more meta-comment though: Something with your git setup seems to be broken, the patches don't have in-reply-to headers pointing at this cover letter and hence the threading is a bit broken.

ok thanks.
yes my mistake I missed an option while generating the patches. Do you suggest resending all patches again?

Quoted 36 lines

Cheers, Daniel

Quoted 33 lines

regards
Arun

Siluvery, Arun (3):
drm/1915: Add ring functions to save/restore context for per-ring
reset
drm/i915: Per-engine Timeout detection and recovery on HSW drm/i915: Export TDR hang count to debugfs

 drivers/gpu/drm/i915/i915_debugfs.c     |  68 +++-
 drivers/gpu/drm/i915/i915_dma.c         |  16 +-
 drivers/gpu/drm/i915/i915_drv.c         | 195 +++++++++-
 drivers/gpu/drm/i915/i915_drv.h         |  92 ++++-
 drivers/gpu/drm/i915/i915_gem.c         |  77 +++-
 drivers/gpu/drm/i915/i915_gpu_error.c   |  25 +-
 drivers/gpu/drm/i915/i915_irq.c         | 556 ++++++++++++++++-------------
 drivers/gpu/drm/i915/i915_reg.h         |   7 +
 drivers/gpu/drm/i915/intel_display.c    |  25 +-
 drivers/gpu/drm/i915/intel_ringbuffer.c | 607 +++++++++++++++++++++++++++++++-
 drivers/gpu/drm/i915/intel_ringbuffer.h |  51 +++
 drivers/gpu/drm/i915/intel_uncore.c     |  31 +-
 include/drm/drmP.h                      |   7 +
 13 files changed, 1467 insertions(+), 290 deletions(-)
Signature · 7 lines
1.8.4


_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#3 lore raw
Quoted 19 lines · Arun
Quoted 18 lines · Daniel Vetter
Quoted 10 lines · Arun

From: "Siluvery, Arun" <[email protected]>

This patchset contains changes for Timeout detection and recovery (TDR) which provides per-engine hang detection and recovery.
The current driver performs full gpu reset in case of a hang, TDR attempts to only reset the engine that is hung and it falls back to full reset if it fails.

Full GPU reset can leave the system in a state where the display updates intermittently and possibly lock-up depending on the work load at the time of hang. TDR can help recover the system in those case thus increasing the stability.

Are these hw lockups you've seen with full gpu reset or just kernel deadlocks? If it's the latter we've recently (re-)fixed a bunch of those, and if there are new ones we definitely want to fix them and add testcases to igt. So if you could share some of these hangs and their analysis/testcases that's be very interesting.

Quoted 5 lines
Quoted 1 line

That's of course on top of any other reset improvements.

I think these are kernel lockups, unfortunately when this happens there is no response from the kernel, sending break is also not helping. I will try to get more details on this.

Enabling lockdep and making sure you have the stuck task warning enabled occasionally helps to get something out of the system. Is the entire box dead or just everything related to gfx?

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

The changes are split in multiple patches.
1. Ring utility functions to save/restore context, reset ring etc
2. TDR hang detection logic and error recovery function
3. Debugfs changes to export TDR statistics.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

I have tested these changes on drm-intel-nightly with simple test which inserts a bad batch buffer on the specific to trigger a hang. TDR logic then detects this and recovers from it by skipping the bad batch.

I want this testcase (as a patch to igt).

ok, I will send it to the mailing list.

Thanks.

Quoted 10 lines
Quoted 6 lines
Quoted 1 line

Please review and give your comments.

I'll try to have a look later this week, atm still busy with bdw upstreaming. One more meta-comment though: Something with your git setup seems to be broken, the patches don't have in-reply-to headers pointing at this cover letter and hence the threading is a bit broken.

ok thanks.
yes my mistake I missed an option while generating the patches. Do you suggest resending all patches again?

No need, was just a quick reminder for next time around. Without threading patch groups are harder to find, especially when a bigger discussion ensued.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch