Skip to contentSkip to conversation

How to assign blame when multiple rings are hung

RFC New Mika Kuoppala 3 participants 4 messages Jan 28, 2014

lore
Share & tools
lore
#0 lore raw

Hi,

I am working with a patchset [1] which, originally, aimed to fix how we find out the guilty batches with ppgtt.

But during the review it became clear that I don't have a clear idea how the behaviour should be when multiple rings encounter a problematic batch at the same time.

The following i-g-t patch will add test which asserts that both contexts get blame of having (problematic) batch active during hang.

The patch set [1] will fail with this test case as it will blame only the first context that injected the hang. We would need to change the test to for it to pass:
- assert_reset_status(fd[1], 0, RS_BATCH_ACTIVE);
+ assert_reset_status(fd[1], 0, RS_BATCH_PENDING);

I lean towards that both contexts get their batch_active count increased. As other rings might gain contexts and we could already reset individual rings instead of whole GPU.

But we need to take a pick so thats why the RFC.
Thoughts?

--
[1]: https://github.com/mkuoppal/linux/commits/one_guilty

Mika Kuoppala (1):
tests/gem_reset_stats: add subtest hang-render-and-<ring>

 tests/gem_reset_stats.c |   34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
Signature · 1 line
1.7.9.5
#1 lore raw

RFC 1/1 tests/gem_reset_stats: add subtest hang-render-and-<ring>

If multiple rings are hung at the same time by two different contexts, both should get get blamed for hanging the GPU.

---
 tests/gem_reset_stats.c |   34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
tests/gem_reset_stats.c +34 -0
index 331d954..c1bf72e 100644
@@ -1017,6 +1017,37 @@ static void test_params(void)
 	close(fd);
 }
 
+static void test_hang_render_and(void)
+{
+	int fd[2];
+	int h[2];
+	int ctx;
+	const struct target_ring *ring = current_ring;
+
+	fd[0] = drm_open_any();
+	fd[1] = drm_open_any();
+
+	ctx = context_create(fd[0]);
+	igt_assert(ctx >= 0);
+
+	h[0] = inject_hang_ring(fd[0], ctx, I915_EXEC_RENDER);
+	h[1] = inject_hang_ring(fd[1], 0, ring->exec);
+
+	gem_sync(fd[1], h[1]);
+
+	assert_reset_status(fd[0], ctx, RS_BATCH_ACTIVE);
+	assert_reset_status(fd[1], 0, RS_BATCH_ACTIVE);
+
+	gem_close(fd[1], h[1]);
+	gem_close(fd[0], h[0]);
+
+	context_destroy(fd[0], ctx);
+
+	close(fd[1]);
+	close(fd[0]);
+}
+
+
 #define RING_HAS_CONTEXTS current_ring->contexts(current_ring)
 #define RUN_CTX_TEST(...) do { igt_skip_on(RING_HAS_CONTEXTS == false); __VA_ARGS__; } while (0)
 
@@ -1093,5 +1124,8 @@ igt_main
 			test_close_pending_fork(true);
 			test_close_pending_fork(false);
 		}
+
+		igt_subtest_f("hang-render-and-%s", name)
+			test_hang_render_and();
 	}
 }
Signature · 1 line
1.7.9.5
#2 lore raw
In reply to Mika Kuoppala
Quoted 19 lines · Mika Kuoppala

Hi,

I am working with a patchset [1] which, originally, aimed to fix how we find out the guilty batches with ppgtt.

But during the review it became clear that I don't have a clear idea how the behaviour should be when multiple rings encounter a problematic batch at the same time.

The following i-g-t patch will add test which asserts that both contexts get blame of having (problematic) batch active during hang.

The patch set [1] will fail with this test case as it will blame only the first context that injected the hang. We would need to change the test to for it to pass:

(quoted diff) +1 -1
-       assert_reset_status(fd[1], 0, RS_BATCH_ACTIVE);
+       assert_reset_status(fd[1], 0, RS_BATCH_PENDING);
Quoted 6 lines

I lean towards that both contexts get their batch_active count increased. As other rings might gain contexts and we could already reset individual rings instead of whole GPU.

But we need to take a pick so thats why the RFC.
Thoughts?

Assuming idealised code, both get blamed today. Which gets blamed first is decided at random (whichever accumulates hangscore quickest), that triggers either a full GPU reset and replay of unaffected batches, or a ring reset (in which we should not touch the other context on the other rings). Then once the GPU is running again, it will hang on the other ring and we will detect it and start the blame game all over again. We do have a fairness issue whereby a sequence of bad batches on one ring may prevent us detecting a hang on the other - but if we have replay working, then we carry over the hangscore as well and so the blame should be fairly appropriated.
-Chris

Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
#3 lore raw
Quoted 27 lines · Chris Wilson
Quoted 19 lines · Mika Kuoppala

Hi,

I am working with a patchset [1] which, originally, aimed to fix how we find out the guilty batches with ppgtt.

But during the review it became clear that I don't have a clear idea how the behaviour should be when multiple rings encounter a problematic batch at the same time.

The following i-g-t patch will add test which asserts that both contexts get blame of having (problematic) batch active during hang.

The patch set [1] will fail with this test case as it will blame only the first context that injected the hang. We would need to change the test to for it to pass:

(quoted diff) +1 -1
-       assert_reset_status(fd[1], 0, RS_BATCH_ACTIVE);
+       assert_reset_status(fd[1], 0, RS_BATCH_PENDING);
Quoted 6 lines

I lean towards that both contexts get their batch_active count increased. As other rings might gain contexts and we could already reset individual rings instead of whole GPU.

But we need to take a pick so thats why the RFC.
Thoughts?

Quoted 10 lines

Assuming idealised code, both get blamed today. Which gets blamed first is decided at random (whichever accumulates hangscore quickest), that triggers either a full GPU reset and replay of unaffected batches, or a ring reset (in which we should not touch the other context on the other rings). Then once the GPU is running again, it will hang on the other ring and we will detect it and start the blame game all over again. We do have a fairness issue whereby a sequence of bad batches on one ring may prevent us detecting a hang on the other - but if we have replay working, then we carry over the hangscore as well and so the blame should be fairly appropriated.

I agree, in the end when two batches hang the gpu on different rings, both should get blamed. And as opposed to two independent batches being submitted to the same ring where both hang (but the 2nd one could be victimized by the first) I can't come up with a reasonable scenario where we can't do this.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch