drm/amdgpu: dequeue mes scheduler during fini
Subsystems (2)
PATCH v3 1/1 drm/amdgpu: dequeue mes scheduler during fini
Update: Remove redundant comments as Christian suggests.
[Why]
If mes is not dequeued during fini, mes will be in an uncleaned state during reload, then mes couldn't receive some commands which leads to reload failure.
[How]
Perform MES dequeue via MMIO after all the unmap jobs are done by mes and before kiq fini.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h | 3 ++ drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c | 3 ++ drivers/gpu/drm/amd/amdgpu/mes_v11_0.c | 41 +++++++++++++++++++++++-- 3 files changed, 44 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +3 -0
@@ -130,6 +130,9 @@ struct amdgpu_mes {int (*kiq_hw_init)(struct amdgpu_device *adev);int (*kiq_hw_fini)(struct amdgpu_device *adev); +/* dequeue sched pipe via kiq */ +void (*kiq_dequeue_sched)(struct amdgpu_device *adev); +/* ip specific functions */ const struct amdgpu_mes_funcs *funcs; };
drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c +3 -0
@@ -4406,6 +4406,9 @@ static int gfx_v11_0_hw_fini(void *handle)if (amdgpu_gfx_disable_kcq (adev))DRM_ERROR ("KCQ disable failed\n" ); +if (adev->mes.ring.sched.ready && adev->mes.kiq_dequeue_sched) + adev->mes.kiq_dequeue_sched (adev); +amdgpu_mes_kiq_hw_fini (adev); }
drivers/gpu/drm/amd/amdgpu/mes_v11_0.c +38 -3
65 lines not on this page · raw message
Signature · 1 line
2.25.1
Am 14.10.22 um 05:34 schrieb YuBiao Wang:
Quoted 7 lines
Update: Remove redundant comments as Christian suggests.
[Why]
If mes is not dequeued during fini, mes will be in an uncleaned state during reload, then mes couldn't receive some commands which leads to reload failure.
Quoted 5 lines
[How]
Perform MES dequeue via MMIO after all the unmap jobs are done by mes and before kiq fini.Signed-off-by YuBiao Wang <[email protected]>
Quoted 107 lines
--- drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h | 3 ++ drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c | 3 ++ drivers/gpu/drm/amd/amdgpu/mes_v11_0.c | 41 +++++++++++++++++++++++-- 3 files changed, 44 insertions(+), 3 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +3
-0@@ -130,6 +130,9 @@ struct amdgpu_mes {int (*kiq_hw_init)(struct amdgpu_device *adev);int (*kiq_hw_fini)(struct amdgpu_device *adev); +/* dequeue sched pipe via kiq */ +void (*kiq_dequeue_sched)(struct amdgpu_device *adev); +/* ip specific functions */ const struct amdgpu_mes_funcs *funcs; };drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c +3
-0@@ -4406,6 +4406,9 @@ static int gfx_v11_0_hw_fini(void *handle)if (amdgpu_gfx_disable_kcq (adev))DRM_ERROR ("KCQ disable failed\n" ); +if (adev->mes.ring.sched.ready && adev->mes.kiq_dequeue_sched) + adev->mes.kiq_dequeue_sched (adev); +amdgpu_mes_kiq_hw_fini (adev); }drivers/gpu/drm/amd/amdgpu/mes_v11_0.c +38
-3@@ -44,6 +44,7 @@ MODULE_FIRMWARE("amdgpu/gc_11_0_3_mes1.bin");static int mes_v11_0_hw_fini (void *handle);static int mes_v11_0_kiq_hw_init (struct amdgpu_device *adev);static int mes_v11_0_kiq_hw_fini (struct amdgpu_device *adev); +static void mes_v11_0_kiq_dequeue_sched (struct amdgpu_device *adev);#define MES_EOP_SIZE 2048@@ -1078,6 +1079,7 @@ static int mes_v11_0_sw_init(void *handle) adev->mes.funcs = &mes_v11_0_funcs; adev->mes.kiq_hw_init = &mes_v11_0_kiq_hw_init; adev->mes.kiq_hw_fini = &mes_v11_0_kiq_hw_fini; + adev->mes.kiq_dequeue_sched = &mes_v11_0_kiq_dequeue_sched; r =amdgpu_mes_init (adev);if (r)@@ -1151,6 +1153,42 @@ static int mes_v11_0_sw_fini(void *handle)return 0 ; } +static void mes_v11_0_kiq_dequeue_sched (struct amdgpu_device *adev) +{ +uint32_t data; +int i; + +mutex_lock (&adev->srbm_mutex); +soc21_grbm_select (adev,3 ,AMDGPU_MES_SCHED_PIPE ,0 ,0 ); + +/* disable the queue if it's active */ +if (RREG32_SOC15 (GC ,0 , regCP_HQD_ACTIVE) &1 ) { +WREG32_SOC15 (GC ,0 , regCP_HQD_DEQUEUE_REQUEST,1 ); +for (i =0 ; i < adev->usec_timeout; i++) { +if (!(RREG32_SOC15 (GC ,0 , regCP_HQD_ACTIVE) &1 )) +break ; +udelay (1 ); + } + } + data =RREG32_SOC15 (GC ,0 , regCP_HQD_PQ_DOORBELL_CONTROL); + data =REG_SET_FIELD (data,CP_HQD_PQ_DOORBELL_CONTROL , +DOORBELL_EN ,0 ); + data =REG_SET_FIELD (data,CP_HQD_PQ_DOORBELL_CONTROL , +DOORBELL_HIT ,1 ); +WREG32_SOC15 (GC ,0 , regCP_HQD_PQ_DOORBELL_CONTROL, data); + +WREG32_SOC15 (GC ,0 , regCP_HQD_PQ_DOORBELL_CONTROL,0 ); + +WREG32_SOC15 (GC ,0 , regCP_HQD_PQ_WPTR_LO,0 ); +WREG32_SOC15 (GC ,0 , regCP_HQD_PQ_WPTR_HI,0 ); +WREG32_SOC15 (GC ,0 , regCP_HQD_PQ_RPTR,0 ); + +soc21_grbm_select (adev,0 ,0 ,0 ,0 ); +mutex_unlock (&adev->srbm_mutex); + + adev->mes.ring.sched.ready = false; +} +static void mes_v11_0_kiq_setting (struct amdgpu_ring *ring) {uint32_t tmp;@@ -1257,9 +1295,6 @@ static int mes_v11_0_hw_init(void *handle)static int mes_v11_0_hw_fini (void *handle) { -struct amdgpu_device *adev = (struct amdgpu_device *)handle; - - adev->mes.ring.sched.ready = false;return 0 ; }
Wang, YuBiao
Acked-by Christian KönigHi Alex,
Could you help review this patch? Thanks.
Best Regards,
Yubiao Wang
-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Friday, October 14, 2022 3:08 PM
To: Wang, YuBiao <[email protected]>; [email protected]
Cc: Grodzovsky, Andrey <[email protected]>; Quan, Evan <[email protected]>; Chen, Horace <[email protected]>; Tuikov, Luben <[email protected]>; Deucher, Alexander <[email protected]>; Xiao, Jack <[email protected]>; Zhang, Hawking <[email protected]>; Liu, Monk <[email protected]>; Xu, Feifei <[email protected]>; Wang, Yang(Kevin) <[email protected]>
Subject: Re: [PATCH v3] drm/amdgpu: dequeue mes scheduler during fini
Am 14.10.22 um 05:34 schrieb YuBiao Wang:
Quoted 7 lines
Update: Remove redundant comments as Christian suggests.
[Why]
If mes is not dequeued during fini, mes will be in an uncleaned state during reload, then mes couldn't receive some commands which leads to reload failure.
Quoted 5 lines
[How]
Perform MES dequeue via MMIO after all the unmap jobs are done by mes and before kiq fini.Signed-off-by YuBiao Wang <[email protected]>
Quoted 109 lines
--- drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h | 3 ++ drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c | 3 ++ drivers/gpu/drm/amd/amdgpu/mes_v11_0.c | 41 +++++++++++++++++++++++-- 3 files changed, 44 insertions(+), 3 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +3
-0@@ -130,6 +130,9 @@ struct amdgpu_mes {int (*kiq_hw_init)(struct amdgpu_device *adev);int (*kiq_hw_fini)(struct amdgpu_device *adev); +/* dequeue sched pipe via kiq */ +void (*kiq_dequeue_sched)(struct amdgpu_device *adev); +/* ip specific functions */ const struct amdgpu_mes_funcs *funcs; };drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c +3
-0@@ -4406,6 +4406,9 @@ static int gfx_v11_0_hw_fini(void *handle)if (amdgpu_gfx_disable_kcq (adev))DRM_ERROR ("KCQ disable failed\n" ); +if (adev->mes.ring.sched.ready && adev->mes.kiq_dequeue_sched) + adev->mes.kiq_dequeue_sched (adev); +amdgpu_mes_kiq_hw_fini (adev); }drivers/gpu/drm/amd/amdgpu/mes_v11_0.c +37
-3@@ -44,6 +44,7 @@ MODULE_FIRMWARE("amdgpu/gc_11_0_3_mes1.bin");static int mes_v11_0_hw_fini (void *handle);static int mes_v11_0_kiq_hw_init (struct amdgpu_device *adev);static int mes_v11_0_kiq_hw_fini (struct amdgpu_device *adev); +static void mes_v11_0_kiq_dequeue_sched (struct amdgpu_device *adev);#define MES_EOP_SIZE 2048@@ -1078,6 +1079,7 @@ static int mes_v11_0_sw_init(void *handle) adev->mes.funcs = &mes_v11_0_funcs; adev->mes.kiq_hw_init = &mes_v11_0_kiq_hw_init; adev->mes.kiq_hw_fini = &mes_v11_0_kiq_hw_fini; + adev->mes.kiq_dequeue_sched = &mes_v11_0_kiq_dequeue_sched; r =amdgpu_mes_init (adev);if (r)@@ -1151,6 +1153,42 @@ static int mes_v11_0_sw_fini(void *handle)return 0 ; } +static void mes_v11_0_kiq_dequeue_sched (struct amdgpu_device *adev) { +uint32_t data; +int i; + +mutex_lock (&adev->srbm_mutex); +soc21_grbm_select (adev,3 ,AMDGPU_MES_SCHED_PIPE ,0 ,0 ); + +/* disable the queue if it's active */ +if (RREG32_SOC15 (GC ,0 , regCP_HQD_ACTIVE) &1 ) { +WREG32_SOC15 (GC ,0 , regCP_HQD_DEQUEUE_REQUEST,1 ); +for (i =0 ; i < adev->usec_timeout; i++) { +if (!(RREG32_SOC15 (GC ,0 , regCP_HQD_ACTIVE) &1 )) +break ; +udelay (1 ); + } + } + data =RREG32_SOC15 (GC ,0 , regCP_HQD_PQ_DOORBELL_CONTROL); + data =REG_SET_FIELD (data,CP_HQD_PQ_DOORBELL_CONTROL , +DOORBELL_EN ,0 ); + data =REG_SET_FIELD (data,CP_HQD_PQ_DOORBELL_CONTROL , +DOORBELL_HIT ,1 ); +WREG32_SOC15 (GC ,0 , regCP_HQD_PQ_DOORBELL_CONTROL, data); + +WREG32_SOC15 (GC ,0 , regCP_HQD_PQ_DOORBELL_CONTROL,0 ); + +WREG32_SOC15 (GC ,0 , regCP_HQD_PQ_WPTR_LO,0 ); +WREG32_SOC15 (GC ,0 , regCP_HQD_PQ_WPTR_HI,0 ); +WREG32_SOC15 (GC ,0 , regCP_HQD_PQ_RPTR,0 ); + +soc21_grbm_select (adev,0 ,0 ,0 ,0 ); +mutex_unlock (&adev->srbm_mutex); + + adev->mes.ring.sched.ready = false; +} +static void mes_v11_0_kiq_setting (struct amdgpu_ring *ring) {uint32_t tmp;@@ -1257,9 +1295,6 @@ static int mes_v11_0_hw_init(void *handle)static int mes_v11_0_hw_fini (void *handle) { -struct amdgpu_device *adev = (struct amdgpu_device *)handle; - - adev->mes.ring.sched.ready = false;return 0 ; }
[AMD Official Use Only - General]
Quoted 3 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ /* dequeue sched pipe via kiq */ + void (*kiq_dequeue_sched)(struct amdgpu_device *adev); +
It's unnecessary to expose a new callback function proto to perform dequeuing scheduler queue,
for kiq_fini or mes_hw_fini callback had got the propriate sequence to do this during fini procedure.
In addition, it seems that the issue is caused by that unclear CP_HQD_PQ_WPTR/CONTROL registers would trigger scheduler reading from unexpected address once CP_MES_CNTL is enabled:
a. for dequeue operation uses MMIO rather than kiq, can the dequeue operation move to mes_v11_hw_fini?
b. if something went wrong, it's better move the dequeue operation inside mes_v11_0_kiq_hw_fini before disabling mes controller.
Regards,
Jack
________________________________
From: YuBiao Wang <[email protected]>
Sent: Friday, 14 October 2022 11:34
To: [email protected] <[email protected]>
Cc: Grodzovsky, Andrey <[email protected]>; Quan, Evan <[email protected]>; Chen, Horace <[email protected]>; Tuikov, Luben <[email protected]>; Koenig, Christian <[email protected]>; Deucher, Alexander <[email protected]>; Xiao, Jack <[email protected]>; Zhang, Hawking <[email protected]>; Liu, Monk <[email protected]>; Xu, Feifei <[email protected]>; Wang, Yang(Kevin) <[email protected]>; Wang, YuBiao <[email protected]>
Subject: [PATCH v3] drm/amdgpu: dequeue mes scheduler during fini
Update: Remove redundant comments as Christian suggests.
[Why]
If mes is not dequeued during fini, mes will be in an uncleaned state during reload, then mes couldn't receive some commands which leads to reload failure.
[How]
Perform MES dequeue via MMIO after all the unmap jobs are done by mes and before kiq fini.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h | 3 ++ drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c | 3 ++ drivers/gpu/drm/amd/amdgpu/mes_v11_0.c | 41 +++++++++++++++++++++++-- 3 files changed, 44 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +3 -0
@@ -130,6 +130,9 @@ struct amdgpu_mes {int (*kiq_hw_init)(struct amdgpu_device *adev);int (*kiq_hw_fini)(struct amdgpu_device *adev); +/* dequeue sched pipe via kiq */ +void (*kiq_dequeue_sched)(struct amdgpu_device *adev); +/* ip specific functions */ const struct amdgpu_mes_funcs *funcs; };
drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c +3 -0
@@ -4406,6 +4406,9 @@ static int gfx_v11_0_hw_fini(void *handle)if (amdgpu_gfx_disable_kcq (adev))DRM_ERROR ("KCQ disable failed\n" ); +if (adev->mes.ring.sched.ready && adev->mes.kiq_dequeue_sched) + adev->mes.kiq_dequeue_sched (adev); +amdgpu_mes_kiq_hw_fini (adev); }
drivers/gpu/drm/amd/amdgpu/mes_v11_0.c +38 -3
65 lines not on this page · raw message
--
2.25.1