Skip to contentSkip to conversation

drm/amdgpu: fix deadlock of reservation between cs and gpu reset v2

PATCH In review Chunming Zhou 3 participants 4 messages Apr 28, 2017 3 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/amdgpu: fix deadlock of reservation between cs and gpu reset v2

the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2. gpu_reset routine will also need pd reservation to restore page table from their shadow.
3. cs is waiting for gpu_reset complete, but gpu reset is waiting for cs releases reservation.

v2: handle amdgpu_cs_submit error path.

Change-Id I0f66d04b2bef3433035109623c8a5c5992c84202
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +4 -0
index 26168df..699f5fe 100644
@@ -1074,6 +1074,7 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p,
 	cs->out.handle = amdgpu_ctx_add_fence(p->ctx, ring, p->fence);
 	job->uf_sequence = cs->out.handle;
 	amdgpu_job_free_resources(job);
+	amdgpu_cs_parser_fini(p, 0, true);
 
 	trace_amdgpu_cs_ioctl(job);
 	amd_sched_entity_push_job(&job->base);
@@ -1129,7 +1130,10 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
 		goto out;
 
 	r = amdgpu_cs_submit(&parser, cs);
+	if (r)
+		goto out;
 
+	return 0;
 out:
 	amdgpu_cs_parser_fini(&parser, r, reserved_buffers);
 	return r;
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1493347534-8201-1-git-send-email-David1.Zhou-5C7GfCeVMHo@public.gmane.org

Reviewed-by Junwei Zhang
#2 lore raw

Nice catch!

Regards,
Jerry (Junwei Zhang)

Linux Base Graphics
SRDC Software Development
_____________________________________

Quoted 57 lines

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Chunming Zhou
Sent: Friday, April 28, 2017 10:46
To: [email protected]
Cc: Zhou, David(ChunMing)
Subject: [PATCH] drm/amdgpu: fix deadlock of reservation between cs and gpu reset v2

the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2. gpu_reset routine will also need pd reservation to restore page table from their shadow.
3. cs is waiting for gpu_reset complete, but gpu reset is waiting for cs releases reservation.

v2: handle amdgpu_cs_submit error path.

Change-Id I0f66d04b2bef3433035109623c8a5c5992c84202
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -0
index 26168df..699f5fe 100644
@@ -1074,6 +1074,7 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser
 *p,
 	cs->out.handle = amdgpu_ctx_add_fence(p->ctx, ring, p->fence);
 	job->uf_sequence = cs->out.handle;
 	amdgpu_job_free_resources(job);
+	amdgpu_cs_parser_fini(p, 0, true);
 
 	trace_amdgpu_cs_ioctl(job);
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +3 -0
 	amd_sched_entity_push_job(&job->base);
@@ -1129,7 +1130,10 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void
 *data, struct drm_file *filp)
 		goto out;
 
 	r = amdgpu_cs_submit(&parser, cs);
+	if (r)
+		goto out;
 
+	return 0;
 out:
 	amdgpu_cs_parser_fini(&parser, r, reserved_buffers);

return r;
--
1.9.1

Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): DM5PR12MB181832DC8E8A41338E5748F4FF130-2J9CzHegvk+QhrfEZJlvtAdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org

#4 lore raw

Indeed, but after a bit of thinking I've found another problem with that patch.

When two threads are pushing jobs into the same scheduler context we don't guarantee correct execution order any more!

Before that patch it was handled by the exclusiveness we had because of reserving the VM page tables, but now nothing prevents us from calling amd_sched_entity_push_job() in nondeterministic order.

In other words we need an additional lock in amdgpu_ctx_ring or something like that.

Regards,
Christian.

Am 28.04.2017 um 04:51 schrieb Zhang, Jerry:

Quoted 68 lines

Nice catch!

Regards,
Jerry (Junwei Zhang)

Linux Base Graphics
SRDC Software Development
_____________________________________

Quoted 57 lines

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Chunming Zhou
Sent: Friday, April 28, 2017 10:46
To: [email protected]
Cc: Zhou, David(ChunMing)
Subject: [PATCH] drm/amdgpu: fix deadlock of reservation between cs and gpu reset v2

the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2. gpu_reset routine will also need pd reservation to restore page table from their shadow.
3. cs is waiting for gpu_reset complete, but gpu reset is waiting for cs releases reservation.

v2: handle amdgpu_cs_submit error path.

Change-Id I0f66d04b2bef3433035109623c8a5c5992c84202
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 4 ++++
  1 file changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -0
index 26168df..699f5fe 100644
@@ -1074,6 +1074,7 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser
 *p,
  	cs->out.handle = amdgpu_ctx_add_fence(p->ctx, ring, p->fence);
  	job->uf_sequence = cs->out.handle;
  	amdgpu_job_free_resources(job);
+	amdgpu_cs_parser_fini(p, 0, true);
 
  	trace_amdgpu_cs_ioctl(job);
  	amd_sched_entity_push_job(&job->base);
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +3 -0
@@ -1129,7 +1130,10 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void
 *data, struct drm_file *filp)
  		goto out;
 
  	r = amdgpu_cs_submit(&parser, cs);
+	if (r)
+		goto out;
 
+	return 0;
  out:
  	amdgpu_cs_parser_fini(&parser, r, reserved_buffers);

return r;
--
1.9.1

Quoted 8 lines
Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): a8fe5428-756b-9fb8-a7ba-52ea9058f90c-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

author
#6 lore raw

Agree, but libdrm doesn't allow concurrent submissions from same context, like protection 'pthread_mutex_lock(&context->sequence_mutex);' in amdgpu_cs_submit_one.

Regards,
David Zhou

Quoted 4 lines · Christian König

Indeed, but after a bit of thinking I've found another problem with that patch.

When two threads are pushing jobs into the same scheduler context we don't guarantee correct execution order any more!

Quoted 87 lines

Before that patch it was handled by the exclusiveness we had because of reserving the VM page tables, but now nothing prevents us from calling amd_sched_entity_push_job() in nondeterministic order.

In other words we need an additional lock in amdgpu_ctx_ring or something like that.

Regards,
Christian.

Am 28.04.2017 um 04:51 schrieb Zhang, Jerry:

Quoted 77 lines

Nice catch!

Regards,
Jerry (Junwei Zhang)

Linux Base Graphics
SRDC Software Development
_____________________________________

Quoted 54 lines

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Chunming Zhou
Sent: Friday, April 28, 2017 10:46
To: [email protected]
Cc: Zhou, David(ChunMing)
Subject: [PATCH] drm/amdgpu: fix deadlock of reservation between cs and gpu reset v2

the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2. gpu_reset routine will also need pd reservation to restore page table from their shadow.
3. cs is waiting for gpu_reset complete, but gpu reset is waiting for cs releases reservation.

v2: handle amdgpu_cs_submit error path.

Change-Id I0f66d04b2bef3433035109623c8a5c5992c84202
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 4 ++++
  1 file changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0 -0
index 26168df..699f5fe 100644
@@ -1074,6 +1074,7 @@ static int amdgpu_cs_submit(struct 

amdgpu_cs_parser
*p,

      cs->out.handle = amdgpu_ctx_add_fence(p->ctx, ring, p->fence);
      job->uf_sequence = cs->out.handle;
      amdgpu_job_free_resources(job);
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +4 -0
+    amdgpu_cs_parser_fini(p, 0, true);
 
      trace_amdgpu_cs_ioctl(job);
      amd_sched_entity_push_job(&job->base);
@@ -1129,7 +1130,10 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void
 *data, struct drm_file *filp)
          goto out;
 
      r = amdgpu_cs_submit(&parser, cs);
+    if (r)
+        goto out;
 
+    return 0;
  out:
      amdgpu_cs_parser_fini(&parser, r, reserved_buffers);
Quoted 8 lines
      return r;
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx