drm/amdgpu: fix deadlock of reservation between cs and gpu reset
Subsystems (2)
PATCH 1/1 drm/amdgpu: fix deadlock of reservation between cs and gpu reset
the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2. gpu_reset routine will also need pd reservation to restore page table from their shadow.
3. cs is waiting for gpu_reset complete, but gpu reset is waiting for cs releases reservation.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +2 -0
@@ -1076,6 +1076,7 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p,amdgpu_job_free_resources (job);trace_amdgpu_cs_ioctl (job); +amdgpu_cs_parser_fini (p,0 , true);amd_sched_entity_push_job (&job->base);return 0 ;
@@ -1130,6 +1131,7 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp) r =amdgpu_cs_submit (&parser, cs); +return r; out:amdgpu_cs_parser_fini (&parser, r, reserved_buffers);return r;
Signature · 6 lines
1.9.1 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 1493281554-9898-1-git-send-email-David1.Zhou-5C7GfCeVMHo@public.gmane.org
Quoted 16 lines · Chunming Zhou
the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2.gpu_resetroutine will also need pd reservation to restore page table from their shadow.
3. cs is waiting forgpu_resetcomplete, but gpu reset is waiting for cs releases reservation.Change-Id I0f66d04b2bef3433035109623c8a5c5992c84202Signed-off-by Chunming Zhou <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 ++ 1 file changed, 2 insertions(+)drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c static int amdgpu_cs_submit(struct amdgpu_cs_parser *p, · line 1079
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0@@ -1076,6 +1076,7 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p,amdgpu_job_free_resources (job);trace_amdgpu_cs_ioctl (job); +amdgpu_cs_parser_fini (p,0 , true);
Please confirm:
It will not free/release more things that may be accessed in job working process, as cs_parse_fini free so many things related to parser.
Or we can just release the pd reservation here?
Jerry
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp) · line 1137
Quoted 12 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0amd_sched_entity_push_job (&job->base);return 0 ;@@ -1130,6 +1131,7 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp) r =amdgpu_cs_submit (&parser, cs); +return r; out:amdgpu_cs_parser_fini (&parser, r, reserved_buffers);return r;
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): [email protected]
Am 27.04.2017 um 11:08 schrieb Zhang, Jerry (Junwei):
Quoted 20 lines · Chunming Zhou
Quoted 19 lines
the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2.gpu_resetroutine will also need pd reservation to restore page table from their shadow.
3. cs is waiting forgpu_resetcomplete, but gpu reset is waiting for cs releases reservation.Change-Id I0f66d04b2bef3433035109623c8a5c5992c84202Signed-off-by Chunming Zhou <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 ++ 1 file changed, 2 insertions(+)drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -1076,6 +1076,7 @@ static int amdgpu_cs_submit(structamdgpu_cs_parser *p,amdgpu_job_free_resources (job );
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c static int amdgpu_cs_submit(struct amdgpu_cs_parser *p, · line 1079
Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0trace_amdgpu_cs_ioctl (job); +amdgpu_cs_parser_fini (p,0 , true);Please confirm:
It will not free/release more things that may be accessed in job working process, ascs_parse_finifree so many things related to parser.
Yeah, that indeed won't work. amdgpu_cs_parser_fini() does:
if (parser->job)
amdgpu_job_free(parser->job);And amdgpu_cs_submit() sets the fence, so it must be called before amdgpu_cs_parser_fini().
But apart from that this is a rather nifty idea. What was the problem with the initial patch?
Quoted 2 lines
Or we can just release the pd reservation here?
We could run into problem with the reservation ticked with that. So I would want to avoid that.
Christian.
Quoted 11 lines
Jerry
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0amd_sched_entity_push_job (&job->base);return 0;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -1130,6 +1131,7 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp) r =amdgpu_cs_submit (&parser, cs);
Quoted 9 lines
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0+return r; out:amdgpu_cs_parser_fini (&parser, r, reserved_buffers);return r;_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 62336e5c-571a-196f-6c07-be25c928cdd3-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org
Quoted 21 lines · Christian König
Am 27.04.2017 um 11:08 schrieb Zhang, Jerry (Junwei):
Quoted 20 lines · Chunming Zhou
Quoted 19 lines
the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2.gpu_resetroutine will also need pd reservation to restore page table from their shadow.
3. cs is waiting forgpu_resetcomplete, but gpu reset is waiting for cs releases reservation.Change-Id I0f66d04b2bef3433035109623c8a5c5992c84202Signed-off-by Chunming Zhou <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 ++ 1 file changed, 2 insertions(+)drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -1076,6 +1076,7 @@ static int amdgpu_cs_submit(structamdgpu_cs_parser *p,amdgpu_job_free_resources (job );
Quoted 8 lines
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c static int amdgpu_cs_submit(struct amdgpu_cs_parser *p, · line 1079
Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0trace_amdgpu_cs_ioctl (job); +amdgpu_cs_parser_fini (p,0 , true);Please confirm:
It will not free/release more things that may be accessed in job working process, ascs_parse_finifree so many things related to parser.Yeah, that indeed won't work.
amdgpu_cs_parser_fini()does:
No, with my patch, parser->job already is NULL here, same as previous sequence, just put push_job action after amdgpu_cs_parser_fini()
Quoted 4 lines
if (parser->job) amdgpu_job_free(parser->job);And
amdgpu_cs_submit()sets the fence, so it must be called beforeamdgpu_cs_parser_fini().
yes, fence is already generated, so amdgpu_cs_parser_fini before pushing job is completely safe here.
Quoted 2 lines
But apart from that this is a rather nifty idea. What was the problem with the initial patch?
Just as comments in patch, which is found by Monk.
Regards,
David Zhou
Quoted 30 lines
Quoted 3 lines
Or we can just release the pd reservation here?
We could run into problem with the reservation ticked with that. So I would want to avoid that.
Christian.
Quoted 20 lines
Jerry
Quoted 11 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0amd_sched_entity_push_job (&job->base);return 0;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0@@ -1130,6 +1131,7 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp) r =amdgpu_cs_submit (&parser, cs); +return r; out:amdgpu_cs_parser_fini (&parser, r, reserved_buffers);Quoted 2 lines
return r;_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): [email protected]
Am 27.04.2017 um 12:23 schrieb zhoucm1:
Quoted 30 lines · Christian König
Quoted 27 lines
Am 27.04.2017 um 11:08 schrieb Zhang, Jerry (Junwei):
Quoted 25 lines · Chunming Zhou
Quoted 16 lines
the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2.gpu_resetroutine will also need pd reservation to restore page table from their shadow.
3. cs is waiting forgpu_resetcomplete, but gpu reset is waiting for cs releases reservation.Change-Id I0f66d04b2bef3433035109623c8a5c5992c84202Signed-off-by Chunming Zhou <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 ++ 1 file changed, 2 insertions(+)drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c static int amdgpu_cs_submit(struct · line 1080
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0@@ -1076,6 +1076,7 @@ static int amdgpu_cs_submit(structamdgpu_cs_parser *p,amdgpu_job_free_resources (job );trace_amdgpu_cs_ioctl (job); +amdgpu_cs_parser_fini (p,0 , true);Please confirm:
It will not free/release more things that may be accessed in job working process, ascs_parse_finifree so many things related to parser.
Quoted 3 lines
Quoted 2 lines
Yeah, that indeed won't work.
amdgpu_cs_parser_fini()does:No, with my patch,
parser->jobalready is NULL here, same as previous sequence, just putpush_jobaction afteramdgpu_cs_parser_fini()
Ah! You add that into amdgpu_cs_submit! That was the point I was missing.
Review
Quoted 3 lines
trace_amdgpu_cs_ioctl(job);(quoted diff) +1
-0+ amdgpu_cs_parser_fini(p, 0, true); amd_sched_entity_push_job(&job->base);
Can we keep the trace_amdgpu_cs_ioctl(); directly before the amd_sched_entity_push_job()?
With that changed, the patch is Reviewed-by: Christian König <[email protected]>.
Regards,
Christian.
Quoted 42 lines
Quoted 4 lines
if (parser->job) amdgpu_job_free(parser->job);And
amdgpu_cs_submit()sets the fence, so it must be called beforeamdgpu_cs_parser_fini().yes, fence is already generated, so
amdgpu_cs_parser_finibefore pushing job is completely safe here.Quoted 2 lines
But apart from that this is a rather nifty idea. What was the problem with the initial patch?
Just as comments in patch, which is found by Monk.
Regards,
David ZhouQuoted 30 lines
Quoted 3 lines
Or we can just release the pd reservation here?
We could run into problem with the reservation ticked with that. So I would want to avoid that.
Christian.
Quoted 20 lines
Jerry
Quoted 11 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0amd_sched_entity_push_job (&job->base);return 0;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0@@ -1130,6 +1131,7 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp) r =amdgpu_cs_submit (&parser, cs); +return r; out:amdgpu_cs_parser_fini (&parser, r, reserved_buffers);Quoted 2 lines
return r;_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Quoted 4 lines
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): b6e45c8f-40a1-4185-89fb-1f4f7719c0fb-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org
Zhang, Jerry (Junwei)
Reviewed-by Junwei ZhangQuoted 35 lines · Christian König
Am 27.04.2017 um 12:23 schrieb zhoucm1:
Quoted 34 lines · Christian König
Quoted 31 lines
Am 27.04.2017 um 11:08 schrieb Zhang, Jerry (Junwei):
Quoted 29 lines · Chunming Zhou
Quoted 20 lines
the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2.gpu_resetroutine will also need pd reservation to restore page table from their shadow.
3. cs is waiting forgpu_resetcomplete, but gpu reset is waiting for cs releases reservation.Change-Id I0f66d04b2bef3433035109623c8a5c5992c84202Signed-off-by Chunming Zhou <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 ++ 1 file changed, 2 insertions(+)drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c static int amdgpu_cs_submit(struct amdgpu_cs_parser *p, · line 1079
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0@@ -1076,6 +1076,7 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p,amdgpu_job_free_resources (job);trace_amdgpu_cs_ioctl (job); +amdgpu_cs_parser_fini (p,0 , true);Please confirm:
It will not free/release more things that may be accessed in job working process, ascs_parse_finifree so many things related to parser.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Yeah, that indeed won't work.
amdgpu_cs_parser_fini()does:No, with my patch,
parser->jobalready is NULL here, same as previous sequence, just putpush_jobaction afteramdgpu_cs_parser_fini()Ah! You add that into
amdgpu_cs_submit! That was the point I was missing.
Thanks your all input.
See it too.
Feel free to add my RB, if need.
Jerry
Quoted 4 lines
Quoted 4 lines
trace_amdgpu_cs_ioctl(job); + amdgpu_cs_parser_fini(p, 0, true); amd_sched_entity_push_job(&job->base);
Quoted 61 lines
Can we keep the
trace_amdgpu_cs_ioctl(); directly before theamd_sched_entity_push_job()?With that changed, the patch is Reviewed-by: Christian König <[email protected]>.
Regards,
Christian.Quoted 50 lines
Quoted 5 lines
if (parser->job) amdgpu_job_free(parser->job);And
amdgpu_cs_submit()sets the fence, so it must be called beforeamdgpu_cs_parser_fini().yes, fence is already generated, so
amdgpu_cs_parser_finibefore pushing job is completely safe here.Quoted 3 lines
But apart from that this is a rather nifty idea. What was the problem with the initial patch?
Just as comments in patch, which is found by Monk.
Regards,
David ZhouQuoted 31 lines
Quoted 3 lines
Or we can just release the pd reservation here?
We could run into problem with the reservation ticked with that. So I would want to avoid that.
Christian.
Quoted 20 lines
Jerry
Quoted 11 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0amd_sched_entity_push_job (&job->base);return 0 ;@@ -1130,6 +1131,7 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp) r =amdgpu_cs_submit (&parser, cs); +return r; out:amdgpu_cs_parser_fini (&parser, r, reserved_buffers);Quoted 2 lines
return r;_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Liu, Monk
Reviewed-by Monk LiuA really good improvement
-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of zhoucm1
Sent: Thursday, April 27, 2017 6:24 PM
To: Christian König <[email protected]>; Zhang, Jerry <[email protected]>; [email protected]
Cc: Liu, Monk <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: fix deadlock of reservation between cs and gpu reset
Quoted 25 lines · Christian König
Am 27.04.2017 um 11:08 schrieb Zhang, Jerry (Junwei):
Quoted 24 lines · Chunming Zhou
Quoted 23 lines
the case could happen when gpu reset:
1. when gpu reset, cs can be continue until sw queue is full, then push job will wait with holding pd reservation.
2.gpu_resetroutine will also need pd reservation to restore page table from their shadow.
3. cs is waiting forgpu_resetcomplete, but gpu reset is waiting for cs releases reservation.Change-Id I0f66d04b2bef3433035109623c8a5c5992c84202Signed-off-by Chunming Zhou <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 ++ 1 file changed, 2 insertions(+)drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0@@ -1076,6 +1076,7 @@ static int amdgpu_cs_submit(structamdgpu_cs_parser *p,amdgpu_job_free_resources (job );
Quoted 9 lines
Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c static int amdgpu_cs_submit(struct · line 1080
Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0trace_amdgpu_cs_ioctl (job); +amdgpu_cs_parser_fini (p,0 , true);Please confirm:
It will not free/release more things that may be accessed in job working process, ascs_parse_finifree so many things related to parser.Yeah, that indeed won't work.
amdgpu_cs_parser_fini()does:
No, with my patch, parser->job already is NULL here, same as previous sequence, just put push_job action after amdgpu_cs_parser_fini()
Quoted 5 lines
if (parser->job) amdgpu_job_free(parser->job);And
amdgpu_cs_submit()sets the fence, so it must be called beforeamdgpu_cs_parser_fini().
yes, fence is already generated, so amdgpu_cs_parser_fini before pushing job is completely safe here.
Quoted 3 lines
But apart from that this is a rather nifty idea. What was the problem with the initial patch?
Just as comments in patch, which is found by Monk.
Regards,
David Zhou
Quoted 31 lines
Quoted 3 lines
Or we can just release the pd reservation here?
We could run into problem with the reservation ticked with that. So I would want to avoid that.
Christian.
Quoted 20 lines
Jerry
Quoted 11 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0
-0amd_sched_entity_push_job (&job->base);return 0;drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1
-0@@ -1130,6 +1131,7 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp) r =amdgpu_cs_submit (&parser, cs); +return r; out:amdgpu_cs_parser_fini (&parser, r, reserved_buffers);Quoted 2 lines
return r;_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx