Skip to contentSkip to conversation

drm/amdgpu: fix deadlock of reservation between cs and gpu reset

PATCH In review Chunming Zhou 4 participants 7 messages Apr 27, 2017 – Apr 28, 2017 5 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/amdgpu: fix deadlock of reservation between cs and gpu reset

the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2. gpu_reset routine will also need pd reservation to restore page table from their shadow.
3. cs is waiting for gpu_reset complete, but gpu reset is waiting for cs releases reservation.

Change-Id I0f66d04b2bef3433035109623c8a5c5992c84202
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +2 -0
index 9edb1a4..a6722a7 100644
@@ -1076,6 +1076,7 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p,
 	amdgpu_job_free_resources(job);
 
 	trace_amdgpu_cs_ioctl(job);
+	amdgpu_cs_parser_fini(p, 0, true);
 	amd_sched_entity_push_job(&job->base);
 
 	return 0;
@@ -1130,6 +1131,7 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
 
 	r = amdgpu_cs_submit(&parser, cs);
 
+	return r;
 out:
 	amdgpu_cs_parser_fini(&parser, r, reserved_buffers);
 	return r;
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1493281554-9898-1-git-send-email-David1.Zhou-5C7GfCeVMHo@public.gmane.org

#2 lore raw
Quoted 16 lines · Chunming Zhou

the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2. gpu_reset routine will also need pd reservation to restore page table from their shadow.
3. cs is waiting for gpu_reset complete, but gpu reset is waiting for cs releases reservation.

Change-Id I0f66d04b2bef3433035109623c8a5c5992c84202
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0 -0
index 9edb1a4..a6722a7 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c static int amdgpu_cs_submit(struct amdgpu_cs_parser *p, · line 1079

Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -0
@@ -1076,6 +1076,7 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p,
  	amdgpu_job_free_resources(job);
 
  	trace_amdgpu_cs_ioctl(job);
+	amdgpu_cs_parser_fini(p, 0, true);

Please confirm:
It will not free/release more things that may be accessed in job working process, as cs_parse_fini free so many things related to parser.

Or we can just release the pd reservation here?

Jerry

drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp) · line 1137

Quoted 12 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -0
  	amd_sched_entity_push_job(&job->base);
 
  	return 0;
@@ -1130,6 +1131,7 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
 
  	r = amdgpu_cs_submit(&parser, cs);
 
+	return r;
  out:
  	amdgpu_cs_parser_fini(&parser, r, reserved_buffers);
  	return r;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw

Am 27.04.2017 um 11:08 schrieb Zhang, Jerry (Junwei):

Quoted 20 lines · Chunming Zhou
Quoted 19 lines

the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2. gpu_reset routine will also need pd reservation to restore page table from their shadow.
3. cs is waiting for gpu_reset complete, but gpu reset is waiting for cs releases reservation.

Change-Id I0f66d04b2bef3433035109623c8a5c5992c84202
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0 -0
index 9edb1a4..a6722a7 100644
@@ -1076,6 +1076,7 @@ static int amdgpu_cs_submit(struct 
 amdgpu_cs_parser *p,
      amdgpu_job_free_resources(job);
 
Quoted 5 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c static int amdgpu_cs_submit(struct amdgpu_cs_parser *p, · line 1079

Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -0
      trace_amdgpu_cs_ioctl(job);
+    amdgpu_cs_parser_fini(p, 0, true);

Please confirm:
It will not free/release more things that may be accessed in job working process, as cs_parse_fini free so many things related to parser.

Yeah, that indeed won't work. amdgpu_cs_parser_fini() does:

        if (parser->job)
                amdgpu_job_free(parser->job);

And amdgpu_cs_submit() sets the fence, so it must be called before amdgpu_cs_parser_fini().

But apart from that this is a rather nifty idea. What was the problem with the initial patch?

Quoted 2 lines

Or we can just release the pd reservation here?

We could run into problem with the reservation ticked with that. So I would want to avoid that.

Christian.

Quoted 11 lines

Jerry

Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0 -0
 amd_sched_entity_push_job(&job->base);
 
      return 0;
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0 -0
@@ -1130,6 +1131,7 @@ int amdgpu_cs_ioctl(struct drm_device *dev, 
 void *data, struct drm_file *filp)
 
      r = amdgpu_cs_submit(&parser, cs);
 
Quoted 9 lines
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -0
+    return r;
  out:
      amdgpu_cs_parser_fini(&parser, r, reserved_buffers);
      return r;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 62336e5c-571a-196f-6c07-be25c928cdd3-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

author
#6 lore raw
Quoted 21 lines · Christian König

Am 27.04.2017 um 11:08 schrieb Zhang, Jerry (Junwei):

Quoted 20 lines · Chunming Zhou
Quoted 19 lines

the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2. gpu_reset routine will also need pd reservation to restore page table from their shadow.
3. cs is waiting for gpu_reset complete, but gpu reset is waiting for cs releases reservation.

Change-Id I0f66d04b2bef3433035109623c8a5c5992c84202
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0 -0
index 9edb1a4..a6722a7 100644
@@ -1076,6 +1076,7 @@ static int amdgpu_cs_submit(struct 
 amdgpu_cs_parser *p,
      amdgpu_job_free_resources(job);
 
Quoted 8 lines
Quoted 5 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c static int amdgpu_cs_submit(struct amdgpu_cs_parser *p, · line 1079

Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -0
      trace_amdgpu_cs_ioctl(job);
+    amdgpu_cs_parser_fini(p, 0, true);

Please confirm:
It will not free/release more things that may be accessed in job working process, as cs_parse_fini free so many things related to parser.

Yeah, that indeed won't work. amdgpu_cs_parser_fini() does:

No, with my patch, parser->job already is NULL here, same as previous sequence, just put push_job action after amdgpu_cs_parser_fini()

Quoted 4 lines
        if (parser->job)
                amdgpu_job_free(parser->job);

And amdgpu_cs_submit() sets the fence, so it must be called before amdgpu_cs_parser_fini().

yes, fence is already generated, so amdgpu_cs_parser_fini before pushing job is completely safe here.

Quoted 2 lines

But apart from that this is a rather nifty idea. What was the problem with the initial patch?

Just as comments in patch, which is found by Monk.

Regards,
David Zhou

Quoted 30 lines
Quoted 3 lines

Or we can just release the pd reservation here?

We could run into problem with the reservation ticked with that. So I would want to avoid that.

Christian.

Quoted 20 lines

Jerry

Quoted 11 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0 -0
 amd_sched_entity_push_job(&job->base);
 
      return 0;
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -0
@@ -1130,6 +1131,7 @@ int amdgpu_cs_ioctl(struct drm_device *dev, 
 void *data, struct drm_file *filp)
 
      r = amdgpu_cs_submit(&parser, cs);
 
+    return r;
  out:
      amdgpu_cs_parser_fini(&parser, r, reserved_buffers);
Quoted 2 lines
      return r;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): [email protected]

#8 lore raw

Am 27.04.2017 um 12:23 schrieb zhoucm1:

Quoted 30 lines · Christian König
Quoted 27 lines

Am 27.04.2017 um 11:08 schrieb Zhang, Jerry (Junwei):

Quoted 25 lines · Chunming Zhou
Quoted 16 lines

the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2. gpu_reset routine will also need pd reservation to restore page table from their shadow.
3. cs is waiting for gpu_reset complete, but gpu reset is waiting for cs releases reservation.

Change-Id I0f66d04b2bef3433035109623c8a5c5992c84202
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0 -0
index 9edb1a4..a6722a7 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c static int amdgpu_cs_submit(struct · line 1080

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -0
@@ -1076,6 +1076,7 @@ static int amdgpu_cs_submit(struct 
 amdgpu_cs_parser *p,
      amdgpu_job_free_resources(job);
 
      trace_amdgpu_cs_ioctl(job);
+    amdgpu_cs_parser_fini(p, 0, true);

Please confirm:
It will not free/release more things that may be accessed in job working process, as cs_parse_fini free so many things related to parser.

Quoted 3 lines
Quoted 2 lines

Yeah, that indeed won't work. amdgpu_cs_parser_fini() does:

No, with my patch, parser->job already is NULL here, same as previous sequence, just put push_job action after amdgpu_cs_parser_fini()

Ah! You add that into amdgpu_cs_submit! That was the point I was missing.

Review

Quoted 3 lines
  	trace_amdgpu_cs_ioctl(job);
(quoted diff) +1 -0
+	amdgpu_cs_parser_fini(p, 0, true);
  	amd_sched_entity_push_job(&job->base);

Can we keep the trace_amdgpu_cs_ioctl(); directly before the amd_sched_entity_push_job()?

With that changed, the patch is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 42 lines
Quoted 4 lines
        if (parser->job)
                amdgpu_job_free(parser->job);

And amdgpu_cs_submit() sets the fence, so it must be called before amdgpu_cs_parser_fini().

yes, fence is already generated, so amdgpu_cs_parser_fini before pushing job is completely safe here.

Quoted 2 lines

But apart from that this is a rather nifty idea. What was the problem with the initial patch?

Just as comments in patch, which is found by Monk.

Regards,
David Zhou

Quoted 30 lines
Quoted 3 lines

Or we can just release the pd reservation here?

We could run into problem with the reservation ticked with that. So I would want to avoid that.

Christian.

Quoted 20 lines

Jerry

Quoted 11 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0 -0
 amd_sched_entity_push_job(&job->base);
 
      return 0;
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -0
@@ -1130,6 +1131,7 @@ int amdgpu_cs_ioctl(struct drm_device *dev, 
 void *data, struct drm_file *filp)
 
      r = amdgpu_cs_submit(&parser, cs);
 
+    return r;
  out:
      amdgpu_cs_parser_fini(&parser, r, reserved_buffers);
Quoted 2 lines
      return r;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): b6e45c8f-40a1-4185-89fb-1f4f7719c0fb-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

Reviewed-by Junwei Zhang
#10 lore raw
Quoted 35 lines · Christian König

Am 27.04.2017 um 12:23 schrieb zhoucm1:

Quoted 34 lines · Christian König
Quoted 31 lines

Am 27.04.2017 um 11:08 schrieb Zhang, Jerry (Junwei):

Quoted 29 lines · Chunming Zhou
Quoted 20 lines

the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2. gpu_reset routine will also need pd reservation to restore page table from their shadow.
3. cs is waiting for gpu_reset complete, but gpu reset is waiting for cs releases reservation.

Change-Id I0f66d04b2bef3433035109623c8a5c5992c84202
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0 -0
index 9edb1a4..a6722a7 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c static int amdgpu_cs_submit(struct amdgpu_cs_parser *p, · line 1079

Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -0
@@ -1076,6 +1076,7 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p,
      amdgpu_job_free_resources(job);
 
      trace_amdgpu_cs_ioctl(job);
+    amdgpu_cs_parser_fini(p, 0, true);

Please confirm:
It will not free/release more things that may be accessed in job working process, as cs_parse_fini free so many things related to parser.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Yeah, that indeed won't work. amdgpu_cs_parser_fini() does:

No, with my patch, parser->job already is NULL here, same as previous sequence, just put push_job action after amdgpu_cs_parser_fini()

Ah! You add that into amdgpu_cs_submit! That was the point I was missing.

Thanks your all input.
See it too.

Feel free to add my RB, if need.

Jerry

Quoted 4 lines
Quoted 4 lines
      trace_amdgpu_cs_ioctl(job);
+    amdgpu_cs_parser_fini(p, 0, true);
      amd_sched_entity_push_job(&job->base);
Quoted 61 lines

Can we keep the trace_amdgpu_cs_ioctl(); directly before the amd_sched_entity_push_job()?

With that changed, the patch is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 50 lines
Quoted 5 lines
        if (parser->job)
                amdgpu_job_free(parser->job);

And amdgpu_cs_submit() sets the fence, so it must be called before amdgpu_cs_parser_fini().

yes, fence is already generated, so amdgpu_cs_parser_fini before pushing job is completely safe here.

Quoted 3 lines

But apart from that this is a rather nifty idea. What was the problem with the initial patch?

Just as comments in patch, which is found by Monk.

Regards,
David Zhou

Quoted 31 lines
Quoted 3 lines

Or we can just release the pd reservation here?

We could run into problem with the reservation ticked with that. So I would want to avoid that.

Christian.

Quoted 20 lines

Jerry

Quoted 11 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -0
 amd_sched_entity_push_job(&job->base);
 
      return 0;
@@ -1130,6 +1131,7 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void
 *data, struct drm_file *filp)
 
      r = amdgpu_cs_submit(&parser, cs);
 
+    return r;
  out:
      amdgpu_cs_parser_fini(&parser, r, reserved_buffers);
Quoted 2 lines
      return r;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Reviewed-by Monk Liu
#11 lore raw
In reply to a message not in the archive

A really good improvement

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of zhoucm1
Sent: Thursday, April 27, 2017 6:24 PM
To: Christian König <[email protected]>; Zhang, Jerry <[email protected]>; [email protected]
Cc: Liu, Monk <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: fix deadlock of reservation between cs and gpu reset

Quoted 25 lines · Christian König

Am 27.04.2017 um 11:08 schrieb Zhang, Jerry (Junwei):

Quoted 24 lines · Chunming Zhou
Quoted 23 lines

the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2. gpu_reset routine will also need pd reservation to restore page table from their shadow.
3. cs is waiting for gpu_reset complete, but gpu reset is waiting for cs releases reservation.

Change-Id I0f66d04b2bef3433035109623c8a5c5992c84202
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0 -0
index 9edb1a4..a6722a7 100644
@@ -1076,6 +1076,7 @@ static int amdgpu_cs_submit(struct 
 amdgpu_cs_parser *p,
      amdgpu_job_free_resources(job);
 
Quoted 9 lines
Quoted 6 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c static int amdgpu_cs_submit(struct · line 1080

Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -0
      trace_amdgpu_cs_ioctl(job);
+    amdgpu_cs_parser_fini(p, 0, true);

Please confirm:
It will not free/release more things that may be accessed in job working process, as cs_parse_fini free so many things related to parser.

Yeah, that indeed won't work. amdgpu_cs_parser_fini() does:

No, with my patch, parser->job already is NULL here, same as previous sequence, just put push_job action after amdgpu_cs_parser_fini()

Quoted 5 lines
        if (parser->job)
                amdgpu_job_free(parser->job);

And amdgpu_cs_submit() sets the fence, so it must be called before amdgpu_cs_parser_fini().

yes, fence is already generated, so amdgpu_cs_parser_fini before pushing job is completely safe here.

Quoted 3 lines

But apart from that this is a rather nifty idea. What was the problem with the initial patch?

Just as comments in patch, which is found by Monk.

Regards,
David Zhou

Quoted 31 lines
Quoted 3 lines

Or we can just release the pd reservation here?

We could run into problem with the reservation ticked with that. So I would want to avoid that.

Christian.

Quoted 20 lines

Jerry

Quoted 11 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0 -0
 amd_sched_entity_push_job(&job->base);
 
      return 0;
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -0
@@ -1130,6 +1131,7 @@ int amdgpu_cs_ioctl(struct drm_device *dev, 
 void *data, struct drm_file *filp)
 
      r = amdgpu_cs_submit(&parser, cs);
 
+    return r;
  out:
      amdgpu_cs_parser_fini(&parser, r, reserved_buffers);
Quoted 2 lines
      return r;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx