Per Engine hang detection and recovery
cover Per Engine hang detection and recovery
From: "Siluvery, Arun" <[email protected]>
This patchset contains changes for Timeout detection and recovery (TDR) which provides per-engine hang detection and recovery.
The current driver performs full gpu reset in case of a hang, TDR attempts to only reset the engine that is hung and it falls back to full reset if it fails.
Full GPU reset can leave the system in a state where the display updates intermittently and possibly lock-up depending on the work load at the time of hang. TDR can help recover the system in those case thus increasing the stability.
The changes are split in multiple patches.
1. Ring utility functions to save/restore context, reset ring etc
2. TDR hang detection logic and error recovery function
3. Debugfs changes to export TDR statistics.
I have tested these changes on drm-intel-nightly with simple test which inserts a bad batch buffer on the specific to trigger a hang. TDR logic then detects this and recovers from it by skipping the bad batch.
Please review and give your comments.
regards
Arun
Siluvery, Arun (3):
drm/1915: Add ring functions to save/restore context for per-ring
reset
drm/i915: Per-engine Timeout detection and recovery on HSW drm/i915: Export TDR hang count to debugfs
drivers/gpu/drm/i915/i915_debugfs.c | 68 +++- drivers/gpu/drm/i915/i915_dma.c | 16 +- drivers/gpu/drm/i915/i915_drv.c | 195 +++++++++- drivers/gpu/drm/i915/i915_drv.h | 92 ++++- drivers/gpu/drm/i915/i915_gem.c | 77 +++- drivers/gpu/drm/i915/i915_gpu_error.c | 25 +- drivers/gpu/drm/i915/i915_irq.c | 556 ++++++++++++++++------------- drivers/gpu/drm/i915/i915_reg.h | 7 + drivers/gpu/drm/i915/intel_display.c | 25 +- drivers/gpu/drm/i915/intel_ringbuffer.c | 607 +++++++++++++++++++++++++++++++- drivers/gpu/drm/i915/intel_ringbuffer.h | 51 +++ drivers/gpu/drm/i915/intel_uncore.c | 31 +- include/drm/drmP.h | 7 + 13 files changed, 1467 insertions(+), 290 deletions(-)
Signature · 1 line
1.8.4
Quoted 10 lines · Arun
From: "Siluvery, Arun" <[email protected]>
This patchset contains changes for Timeout detection and recovery (TDR) which provides per-engine hang detection and recovery.
The current driver performs full gpu reset in case of a hang, TDR attempts to only reset the engine that is hung and it falls back to full reset if it fails.Full GPU reset can leave the system in a state where the display updates intermittently and possibly lock-up depending on the work load at the time of hang. TDR can help recover the system in those case thus increasing the stability.
Are these hw lockups you've seen with full gpu reset or just kernel deadlocks? If it's the latter we've recently (re-)fixed a bunch of those, and if there are new ones we definitely want to fix them and add testcases to igt. So if you could share some of these hangs and their analysis/testcases that's be very interesting.
That's of course on top of any other reset improvements.
Quoted 8 lines
The changes are split in multiple patches.
1. Ring utility functions to save/restore context, reset ring etc
2. TDR hang detection logic and error recovery function
3. Debugfs changes to export TDR statistics.I have tested these changes on drm-intel-nightly with simple test which inserts a bad batch buffer on the specific to trigger a hang. TDR logic then detects this and recovers from it by skipping the bad batch.
I want this testcase (as a patch to igt).
Quoted 1 line
Please review and give your comments.
I'll try to have a look later this week, atm still busy with bdw upstreaming. One more meta-comment though: Something with your git setup seems to be broken, the patches don't have in-reply-to headers pointing at this cover letter and hence the threading is a bit broken.
Cheers, Daniel
Quoted 33 lines
regards
ArunSiluvery, Arun (3):
drm/1915: Add ring functions to save/restore context for per-ring
reset
drm/i915: Per-engine Timeout detection and recovery on HSW drm/i915: Export TDR hang count to debugfsdrivers/gpu/drm/i915/i915_debugfs.c | 68 +++- drivers/gpu/drm/i915/i915_dma.c | 16 +- drivers/gpu/drm/i915/i915_drv.c | 195 +++++++++- drivers/gpu/drm/i915/i915_drv.h | 92 ++++- drivers/gpu/drm/i915/i915_gem.c | 77 +++- drivers/gpu/drm/i915/i915_gpu_error.c | 25 +- drivers/gpu/drm/i915/i915_irq.c | 556 ++++++++++++++++------------- drivers/gpu/drm/i915/i915_reg.h | 7 + drivers/gpu/drm/i915/intel_display.c | 25 +- drivers/gpu/drm/i915/intel_ringbuffer.c | 607 +++++++++++++++++++++++++++++++- drivers/gpu/drm/i915/intel_ringbuffer.h | 51 +++ drivers/gpu/drm/i915/intel_uncore.c | 31 +- include/drm/drmP.h | 7 + 13 files changed, 1467 insertions(+), 290 deletions(-)Signature · 7 lines
1.8.4 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
Quoted 12 lines · Daniel Vetter
Quoted 10 lines · Arun
From: "Siluvery, Arun" <[email protected]>
This patchset contains changes for Timeout detection and recovery (TDR) which provides per-engine hang detection and recovery.
The current driver performs full gpu reset in case of a hang, TDR attempts to only reset the engine that is hung and it falls back to full reset if it fails.Full GPU reset can leave the system in a state where the display updates intermittently and possibly lock-up depending on the work load at the time of hang. TDR can help recover the system in those case thus increasing the stability.
Quoted 7 lines
Are these hw lockups you've seen with full gpu reset or just kernel deadlocks? If it's the latter we've recently (re-)fixed a bunch of those, and if there are new ones we definitely want to fix them and add testcases to igt. So if you could share some of these hangs and their analysis/testcases that's be very interesting.
That's of course on top of any other reset improvements.
I think these are kernel lockups, unfortunately when this happens there is no response from the kernel, sending break is also not helping. I will try to get more details on this.
Quoted 11 lines
Quoted 9 lines
The changes are split in multiple patches.
1. Ring utility functions to save/restore context, reset ring etc
2. TDR hang detection logic and error recovery function
3. Debugfs changes to export TDR statistics.I have tested these changes on drm-intel-nightly with simple test which inserts a bad batch buffer on the specific to trigger a hang. TDR logic then detects this and recovers from it by skipping the bad batch.
I want this testcase (as a patch to igt).
ok, I will send it to the mailing list.
Quoted 7 lines
Quoted 2 lines
Please review and give your comments.
I'll try to have a look later this week, atm still busy with bdw upstreaming. One more meta-comment though: Something with your git setup seems to be broken, the patches don't have in-reply-to headers pointing at this cover letter and hence the threading is a bit broken.
ok thanks.
yes my mistake I missed an option while generating the patches. Do you suggest resending all patches again?
Quoted 36 lines
Cheers, Daniel
Quoted 33 lines
regards
ArunSiluvery, Arun (3):
drm/1915: Add ring functions to save/restore context for per-ring
reset
drm/i915: Per-engine Timeout detection and recovery on HSW drm/i915: Export TDR hang count to debugfsdrivers/gpu/drm/i915/i915_debugfs.c | 68 +++- drivers/gpu/drm/i915/i915_dma.c | 16 +- drivers/gpu/drm/i915/i915_drv.c | 195 +++++++++- drivers/gpu/drm/i915/i915_drv.h | 92 ++++- drivers/gpu/drm/i915/i915_gem.c | 77 +++- drivers/gpu/drm/i915/i915_gpu_error.c | 25 +- drivers/gpu/drm/i915/i915_irq.c | 556 ++++++++++++++++------------- drivers/gpu/drm/i915/i915_reg.h | 7 + drivers/gpu/drm/i915/intel_display.c | 25 +- drivers/gpu/drm/i915/intel_ringbuffer.c | 607 +++++++++++++++++++++++++++++++- drivers/gpu/drm/i915/intel_ringbuffer.h | 51 +++ drivers/gpu/drm/i915/intel_uncore.c | 31 +- include/drm/drmP.h | 7 + 13 files changed, 1467 insertions(+), 290 deletions(-)Signature · 7 lines
1.8.4 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 19 lines · Arun
Quoted 18 lines · Daniel Vetter
Quoted 10 lines · Arun
From: "Siluvery, Arun" <[email protected]>
This patchset contains changes for Timeout detection and recovery (TDR) which provides per-engine hang detection and recovery.
The current driver performs full gpu reset in case of a hang, TDR attempts to only reset the engine that is hung and it falls back to full reset if it fails.Full GPU reset can leave the system in a state where the display updates intermittently and possibly lock-up depending on the work load at the time of hang. TDR can help recover the system in those case thus increasing the stability.
Are these hw lockups you've seen with full gpu reset or just kernel deadlocks? If it's the latter we've recently (re-)fixed a bunch of those, and if there are new ones we definitely want to fix them and add testcases to igt. So if you could share some of these hangs and their analysis/testcases that's be very interesting.
Quoted 5 lines
Quoted 1 line
That's of course on top of any other reset improvements.
I think these are kernel lockups, unfortunately when this happens there is no response from the kernel, sending break is also not helping. I will try to get more details on this.
Enabling lockdep and making sure you have the stuck task warning enabled occasionally helps to get something out of the system. Is the entire box dead or just everything related to gfx?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
The changes are split in multiple patches.
1. Ring utility functions to save/restore context, reset ring etc
2. TDR hang detection logic and error recovery function
3. Debugfs changes to export TDR statistics.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I have tested these changes on drm-intel-nightly with simple test which inserts a bad batch buffer on the specific to trigger a hang. TDR logic then detects this and recovers from it by skipping the bad batch.
I want this testcase (as a patch to igt).
ok, I will send it to the mailing list.
Thanks.
Quoted 10 lines
Quoted 6 lines
Quoted 1 line
Please review and give your comments.
I'll try to have a look later this week, atm still busy with bdw upstreaming. One more meta-comment though: Something with your git setup seems to be broken, the patches don't have in-reply-to headers pointing at this cover letter and hence the threading is a bit broken.
ok thanks.
yes my mistake I missed an option while generating the patches. Do you suggest resending all patches again?
No need, was just a quick reminder for next time around. Without threading patch groups are harder to find, especially when a bigger discussion ensued.
-Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch