How to assign blame when multiple rings are hung
Hi,
I am working with a patchset [1] which, originally, aimed to fix how we find out the guilty batches with ppgtt.
But during the review it became clear that I don't have a clear idea how the behaviour should be when multiple rings encounter a problematic batch at the same time.
The following i-g-t patch will add test which asserts that both contexts get blame of having (problematic) batch active during hang.
The patch set [1] will fail with this test case as it will blame only the first context that injected the hang. We would need to change the test to for it to pass:
- assert_reset_status(fd[1], 0, RS_BATCH_ACTIVE);
+ assert_reset_status(fd[1], 0, RS_BATCH_PENDING);
I lean towards that both contexts get their batch_active count increased. As other rings might gain contexts and we could already reset individual rings instead of whole GPU.
But we need to take a pick so thats why the RFC.
Thoughts?
--
[1]: https://github.com/mkuoppal/linux/commits/one_guilty
Mika Kuoppala (1):
tests/gem_reset_stats: add subtest hang-render-and-<ring>
tests/gem_reset_stats.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
Signature · 1 line
1.7.9.5
RFC 1/1 tests/gem_reset_stats: add subtest hang-render-and-<ring>
If multiple rings are hung at the same time by two different contexts, both should get get blamed for hanging the GPU.
--- tests/gem_reset_stats.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
tests/gem_reset_stats.c +34 -0
@@ -1017,6 +1017,37 @@ static void test_params(void)close (fd); } +static void test_hang_render_and (void ) +{ +int fd[2 ]; +int h[2 ]; +int ctx; +const struct target_ring *ring = current_ring; + + fd[0 ] =drm_open_any (); + fd[1 ] =drm_open_any (); + + ctx =context_create (fd[0 ]); +igt_assert (ctx >=0 ); + + h[0 ] =inject_hang_ring (fd[0 ], ctx,I915_EXEC_RENDER ); + h[1 ] =inject_hang_ring (fd[1 ],0 , ring->exec); + +gem_sync (fd[1 ], h[1 ]); + +assert_reset_status (fd[0 ], ctx,RS_BATCH_ACTIVE ); +assert_reset_status (fd[1 ],0 ,RS_BATCH_ACTIVE ); + +gem_close (fd[1 ], h[1 ]); +gem_close (fd[0 ], h[0 ]); + +context_destroy (fd[0 ], ctx); + +close (fd[1 ]); +close (fd[0 ]); +} + +#define RING_HAS_CONTEXTS current_ring->contexts(current_ring)#define RUN_CTX_TEST (...) do { igt_skip_on(RING_HAS_CONTEXTS == false); __VA_ARGS__; } while (0)
@@ -1093,5 +1124,8 @@ igt_maintest_close_pending_fork (true);test_close_pending_fork (false); } + + igt_subtest_f("hang-render-and-%s", name) +test_hang_render_and (); } }
Signature · 1 line
1.7.9.5
Quoted 19 lines · Mika Kuoppala
Hi,
I am working with a patchset [1] which, originally, aimed to fix how we find out the guilty batches with ppgtt.
But during the review it became clear that I don't have a clear idea how the behaviour should be when multiple rings encounter a problematic batch at the same time.
The following i-g-t patch will add test which asserts that both contexts get blame of having (problematic) batch active during hang.
The patch set [1] will fail with this test case as it will blame only the first context that injected the hang. We would need to change the test to for it to pass:
(quoted diff) +1
-1- assert_reset_status(fd[1], 0, RS_BATCH_ACTIVE); + assert_reset_status(fd[1], 0, RS_BATCH_PENDING);
Quoted 6 lines
I lean towards that both contexts get their
batch_activecount increased. As other rings might gain contexts and we could already reset individual rings instead of whole GPU.But we need to take a pick so thats why the RFC.
Thoughts?
Assuming idealised code, both get blamed today. Which gets blamed first is decided at random (whichever accumulates hangscore quickest), that triggers either a full GPU reset and replay of unaffected batches, or a ring reset (in which we should not touch the other context on the other rings). Then once the GPU is running again, it will hang on the other ring and we will detect it and start the blame game all over again. We do have a fairness issue whereby a sequence of bad batches on one ring may prevent us detecting a hang on the other - but if we have replay working, then we carry over the hangscore as well and so the blame should be fairly appropriated.
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Quoted 27 lines · Chris Wilson
Quoted 19 lines · Mika Kuoppala
Hi,
I am working with a patchset [1] which, originally, aimed to fix how we find out the guilty batches with ppgtt.
But during the review it became clear that I don't have a clear idea how the behaviour should be when multiple rings encounter a problematic batch at the same time.
The following i-g-t patch will add test which asserts that both contexts get blame of having (problematic) batch active during hang.
The patch set [1] will fail with this test case as it will blame only the first context that injected the hang. We would need to change the test to for it to pass:
(quoted diff) +1
-1- assert_reset_status(fd[1], 0, RS_BATCH_ACTIVE); + assert_reset_status(fd[1], 0, RS_BATCH_PENDING);Quoted 6 lines
I lean towards that both contexts get their
batch_activecount increased. As other rings might gain contexts and we could already reset individual rings instead of whole GPU.But we need to take a pick so thats why the RFC.
Thoughts?
Quoted 10 lines
Assuming idealised code, both get blamed today. Which gets blamed first is decided at random (whichever accumulates hangscore quickest), that triggers either a full GPU reset and replay of unaffected batches, or a ring reset (in which we should not touch the other context on the other rings). Then once the GPU is running again, it will hang on the other ring and we will detect it and start the blame game all over again. We do have a fairness issue whereby a sequence of bad batches on one ring may prevent us detecting a hang on the other - but if we have replay working, then we carry over the hangscore as well and so the blame should be fairly appropriated.
I agree, in the end when two batches hang the gpu on different rings, both should get blamed. And as opposed to two independent batches being submitted to the same ring where both hang (but the 2nd one could be victimized by the first) I can't come up with a reasonable scenario where we can't do this.
-Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch