drm/sched: Fix deadlock in drm_sched_entity_kill_jobs_cb
Subsystems (4)
PATCH v3 1/1 drm/sched: Fix deadlock in drm_sched_entity_kill_jobs_cb
The Mesa issue referenced below pointed out a possible deadlock:
[ 1231.611031] Possible interrupt unsafe locking scenario:
[ 1231.611033] CPU0 CPU1
[ 1231.611034] ---- ----
[ 1231.611035] lock(&xa->xa_lock#17);
[ 1231.611038] local_irq_disable();
[ 1231.611039] lock(&fence->lock);
[ 1231.611041] lock(&xa->xa_lock#17);
[ 1231.611044] <Interrupt>
[ 1231.611045] lock(&fence->lock);
[ 1231.611047]
*** DEADLOCK ***In this example, CPU0 would be any function accessing job->dependencies through the xa_* functions that doesn't disable interrupts (eg:drm_sched_job_add_dependency, drm_sched_entity_kill_jobs_cb).
CPU1 is executing drm_sched_entity_kill_jobs_cb as a fence signalling callback so in an interrupt context. It will deadlock when trying to grab the xa_lock which is already held by CPU0.
Replacing all xa_* usage by their xa_*_irq counterparts would fix this issue, but Christian pointed out another issue: dma_fence_signal takes fence.lock and so does dma_fence_add_callback.
dma_fence_signal() // locks f1.lock
-> drm_sched_entity_kill_jobs_cb()
-> foreach dependencies
-> dma_fence_add_callback() // locks f2.lock
This will deadlock if f1 and f2 share the same spinlock.
To fix both issues, the code iterating on dependencies and re-arming them is moved out to drm_sched_entity_kill_jobs_work.
v2: reworded commit message (Philipp)
v3: added Fixes tag (Philipp)
--- drivers/gpu/drm/scheduler/sched_entity.c | 34 +++++++++++++----------- 1 file changed, 19 insertions(+), 15 deletions(-)
drivers/gpu/drm/scheduler/sched_entity.c +19 -15
@@ -173,26 +173,15 @@ int drm_sched_entity_error(struct drm_sched_entity *entity) }EXPORT_SYMBOL (drm_sched_entity_error); +static void drm_sched_entity_kill_jobs_cb (struct dma_fence *f, +struct dma_fence_cb *cb); +static void drm_sched_entity_kill_jobs_work (struct work_struct *wrk) {struct drm_sched_job *job =container_of (wrk,typeof (*job), work); - -drm_sched_fence_scheduled (job->s_fence,NULL ); -drm_sched_fence_finished (job->s_fence, -ESRCH ); -WARN_ON (job->s_fence->parent); - job->sched->ops->free_job (job); -} - -/* Signal the scheduler finished fence when the entity in question is killed. */ -static void drm_sched_entity_kill_jobs_cb (struct dma_fence *f, -struct dma_fence_cb *cb) -{ - struct drm_sched_job *job = container_of(cb, struct drm_sched_job, - finish_cb); +struct dma_fence *f;unsigned long index; -dma_fence_put (f); -/* Wait for all dependencies to avoid data corruptions */ xa_for_each (&job->dependencies, index, f) {struct drm_sched_fence *s_fence =to_drm_sched_fence (f);
@@ -220,6 +209,21 @@ static void drm_sched_entity_kill_jobs_cb(struct dma_fence *f,dma_fence_put (f); } +drm_sched_fence_scheduled (job->s_fence,NULL ); +drm_sched_fence_finished (job->s_fence, -ESRCH ); +WARN_ON (job->s_fence->parent); + job->sched->ops->free_job (job); +} + +/* Signal the scheduler finished fence when the entity in question is killed. */ +static void drm_sched_entity_kill_jobs_cb (struct dma_fence *f, +struct dma_fence_cb *cb) +{ + struct drm_sched_job *job = container_of(cb, struct drm_sched_job, + finish_cb); + +dma_fence_put (f); +INIT_WORK (&job->work, drm_sched_entity_kill_jobs_work);schedule_work (&job->work); }
Signature · 1 line
2.43.0
Quoted 32 lines · Pierre-Eric Pelloux-Prayer
The Mesa issue referenced below pointed out a possible deadlock:
[ 1231.611031] Possible interrupt unsafe locking scenario:[ 1231.611033] CPU0 CPU1 [ 1231.611034] ---- ---- [ 1231.611035] lock(&xa->xa_lock#17); [ 1231.611038] local_irq_disable(); [ 1231.611039] lock(&fence->lock); [ 1231.611041] lock(&xa->xa_lock#17); [ 1231.611044] <Interrupt> [ 1231.611045] lock(&fence->lock); [ 1231.611047]*** DEADLOCK ***
In this example, CPU0 would be any function accessing
job->dependenciesthrough the xa_* functions that doesn't disable interrupts (eg:drm_sched_job_add_dependency,drm_sched_entity_kill_jobs_cb).CPU1 is executing
drm_sched_entity_kill_jobs_cbas a fence signalling callback so in an interrupt context. It will deadlock when trying to grab thexa_lockwhich is already held by CPU0.Replacing all xa_* usage by their xa_*_irq counterparts would fix this issue, but Christian pointed out another issue:
dma_fence_signaltakes fence.lock and so doesdma_fence_add_callback.
dma_fence_signal()// locks f1.lock
->drm_sched_entity_kill_jobs_cb()
-> foreach dependencies
->dma_fence_add_callback()// locks f2.lock
Quoted 7 lines
This will deadlock if f1 and f2 share the same spinlock.
To fix both issues, the code iterating on dependencies and re-arming them is moved out to
drm_sched_entity_kill_jobs_work.v2: reworded commit message (Philipp)
v3: added Fixes tag (Philipp)
Thx for the update.
In the future please put the changelog below between a pair of '---'
---
v2: …
v3: …
---
Some things I have unfortunately overlooked below.
Quoted 2 lines
Fixes 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini")
We should +Cc stable. It's a deadlock after all.
Quoted 17 lines
Reported-by Mikhail Gavrilov <[email protected]>Suggested-by Christian König <[email protected]>Reviewed-by Christian König <[email protected]>Signed-off-by Pierre-Eric Pelloux-Prayer <[email protected]>---
drivers/gpu/drm/scheduler/sched_entity.c | 34 +++++++++++++-----------1 file changed, 19 insertions(+), 15 deletions(-)drivers/gpu/drm/scheduler/sched_entity.c +0
-0@@ -173,26 +173,15 @@ int drm_sched_entity_error(struct drm_sched_entity *entity) } EXPORT_SYMBOL(drm_sched_entity_error);
drivers/gpu/drm/scheduler/sched_entity.c int drm_sched_entity_error(struct drm_sched_entity *entity) · line 177
Quoted 2 lines
drivers/gpu/drm/scheduler/sched_entity.c +2
-0+static void drm_sched_entity_kill_jobs_cb (struct dma_fence *f, + struct dma_fence_cb *cb);
It's far better to move the function up instead. Can you do that?
Quoted 2 lines
+
[…]
drivers/gpu/drm/scheduler/sched_entity.c static void drm_sched_entity_kill_jobs_cb(struct dma_fence *f, · line 225
Quoted 8 lines
drivers/gpu/drm/scheduler/sched_entity.c +8
-0+/* Signal the scheduler finished fence when the entity in question is killed. */ +static void drm_sched_entity_kill_jobs_cb (struct dma_fence *f, + struct dma_fence_cb *cb) +{ + struct drm_sched_job *job = container_of(cb, struct drm_sched_job, + finish_cb); + +dma_fence_put (f);
It would be great if we knew what fence is being dropped here and why. I know you're just moving the pre-existing code, but if you should know, informing about that via comment would be great.
Optional.
Rest of the code looks good. No further objections.
P.
Le 04/11/2025 à 13:43, Philipp Stanner a écrit :
Quoted 41 lines · Pierre-Eric Pelloux-Prayer
Quoted 32 lines
The Mesa issue referenced below pointed out a possible deadlock:
[ 1231.611031] Possible interrupt unsafe locking scenario:[ 1231.611033] CPU0 CPU1 [ 1231.611034] ---- ---- [ 1231.611035] lock(&xa->xa_lock#17); [ 1231.611038] local_irq_disable(); [ 1231.611039] lock(&fence->lock); [ 1231.611041] lock(&xa->xa_lock#17); [ 1231.611044] <Interrupt> [ 1231.611045] lock(&fence->lock); [ 1231.611047] *** DEADLOCK ***In this example, CPU0 would be any function accessing
job->dependenciesthrough the xa_* functions that doesn't disable interrupts (eg:drm_sched_job_add_dependency,drm_sched_entity_kill_jobs_cb).CPU1 is executing
drm_sched_entity_kill_jobs_cbas a fence signalling callback so in an interrupt context. It will deadlock when trying to grab thexa_lockwhich is already held by CPU0.Replacing all xa_* usage by their xa_*_irq counterparts would fix this issue, but Christian pointed out another issue:
dma_fence_signaltakes fence.lock and so doesdma_fence_add_callback.
dma_fence_signal()// locks f1.lock
->drm_sched_entity_kill_jobs_cb()
-> foreach dependencies
->dma_fence_add_callback()// locks f2.lockQuoted 7 lines
This will deadlock if f1 and f2 share the same spinlock.
To fix both issues, the code iterating on dependencies and re-arming them is moved out to
drm_sched_entity_kill_jobs_work.v2: reworded commit message (Philipp)
v3: added Fixes tag (Philipp)
Quoted 7 lines
Thx for the update.
In the future please put the changelog below between a pair of '---'---
v2: …
v3: …
---
OK.
Quoted 7 lines
Some things I have unfortunately overlooked below.
Quoted 2 lines
Fixes 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini")We should +Cc stable. It's a deadlock after all.
OK.
Quoted 18 lines
Quoted 18 lines
Reported-by Mikhail Gavrilov <[email protected]>Suggested-by Christian König <[email protected]>Reviewed-by Christian König <[email protected]>Signed-off-by Pierre-Eric Pelloux-Prayer <[email protected]>--- drivers/gpu/drm/scheduler/sched_entity.c | 34 +++++++++++++----------- 1 file changed, 19 insertions(+), 15 deletions(-)drivers/gpu/drm/scheduler/sched_entity.c +0
-0@@ -173,26 +173,15 @@ int drm_sched_entity_error(struct drm_sched_entity *entity) } EXPORT_SYMBOL(drm_sched_entity_error);
Quoted 4 lines
drivers/gpu/drm/scheduler/sched_entity.c int drm_sched_entity_error(struct drm_sched_entity *entity) · line 177
Quoted 2 lines
drivers/gpu/drm/scheduler/sched_entity.c +2
-0+static void drm_sched_entity_kill_jobs_cb (struct dma_fence *f, + struct dma_fence_cb *cb);It's far better to move the function up instead. Can you do that?
Since drm_sched_entity_kill_jobs_cb uses drm_sched_entity_kill_jobs and vice versa, I'll have to forward declare one of the 2 functions anyway.
Quoted 6 lines
Quoted 3 lines
drivers/gpu/drm/scheduler/sched_entity.c +1
-0+[…]
Quoted 13 lines
drivers/gpu/drm/scheduler/sched_entity.c static void drm_sched_entity_kill_jobs_cb(struct dma_fence *f, · line 225
Quoted 8 lines
drivers/gpu/drm/scheduler/sched_entity.c +8
-0+/* Signal the scheduler finished fence when the entity in question is killed. */ +static void drm_sched_entity_kill_jobs_cb (struct dma_fence *f, + struct dma_fence_cb *cb) +{ + struct drm_sched_job *job = container_of(cb, struct drm_sched_job, + finish_cb); + +dma_fence_put (f);It would be great if we knew what fence is being dropped here and why. I know you're just moving the pre-existing code, but if you should know, informing about that via comment would be great.
As discussed offline, "f" is simply the fence being signaled, I'm not sure it warrants a comment.
Regards,
Pierre-Eric
Quoted 7 lines
Optional.
Rest of the code looks good. No further objections.
P.
Quoted 37 lines · Pierre-Eric Pelloux-Prayer
Le 04/11/2025 à 13:43, Philipp Stanner a écrit :
Quoted 7 lines
Some things I have unfortunately overlooked below.
Quoted 2 lines
Fixes 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini")We should +Cc stable. It's a deadlock after all.
OK.
Quoted 10 lines
Quoted 10 lines
Reported-by Mikhail Gavrilov <[email protected]>Suggested-by Christian König <[email protected]>Reviewed-by Christian König <[email protected]>Signed-off-by Pierre-Eric Pelloux-Prayer <[email protected]>---
drivers/gpu/drm/scheduler/sched_entity.c | 34 +++++++++++++-----------1 file changed, 19 insertions(+), 15 deletions(-)Quoted 12 lines
drivers/gpu/drm/scheduler/sched_entity.c int drm_sched_entity_error(struct drm_sched_entity *entity) · line 177
Quoted 10 lines
drivers/gpu/drm/scheduler/sched_entity.c +0
-0@@ -173,26 +173,15 @@ int drm_sched_entity_error(struct drm_sched_entity *entity) } EXPORT_SYMBOL(drm_sched_entity_error);drivers/gpu/drm/scheduler/sched_entity.c +2
-0+static void drm_sched_entity_kill_jobs_cb (struct dma_fence *f, + struct dma_fence_cb *cb);It's far better to move the function up instead. Can you do that?
Quoted 2 lines
Since
drm_sched_entity_kill_jobs_cbusesdrm_sched_entity_kill_jobsand vice versa, I'll have to forward declare one of the 2 functions anyway.
Ah, right.
OK then.
I can push this and +Cc stable in the commit message if you want.
P.
Le 04/11/2025 à 16:30, Philipp Stanner a écrit :
Quoted 38 lines · Pierre-Eric Pelloux-Prayer
Quoted 37 lines
Le 04/11/2025 à 13:43, Philipp Stanner a écrit :
Quoted 7 lines
Some things I have unfortunately overlooked below.
Quoted 2 lines
Fixes 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini")We should +Cc stable. It's a deadlock after all.
OK.
Quoted 22 lines
Quoted 15 lines
Reported-by Mikhail Gavrilov <[email protected]>Suggested-by Christian König <[email protected]>Reviewed-by Christian König <[email protected]>Signed-off-by Pierre-Eric Pelloux-Prayer <[email protected]>--- drivers/gpu/drm/scheduler/sched_entity.c | 34 +++++++++++++----------- 1 file changed, 19 insertions(+), 15 deletions(-)drivers/gpu/drm/scheduler/sched_entity.c +0
-0drivers/gpu/drm/scheduler/sched_entity.c int drm_sched_entity_error(struct drm_sched_entity *entity) · line 177
Quoted 5 lines
drivers/gpu/drm/scheduler/sched_entity.c +2
-0@@ -173,26 +173,15 @@ int drm_sched_entity_error(struct drm_sched_entity *entity) } EXPORT_SYMBOL(drm_sched_entity_error); +static void drm_sched_entity_kill_jobs_cb (struct dma_fence *f, + struct dma_fence_cb *cb);It's far better to move the function up instead. Can you do that?
Quoted 8 lines
Quoted 2 lines
Since
drm_sched_entity_kill_jobs_cbusesdrm_sched_entity_kill_jobsand vice versa, I'll have to forward declare one of the 2 functions anyway.Ah, right.
OK then.I can push this and +Cc stable in the commit message if you want.
Would be great, thanks!
Pierre-Eric
Philipp Stanner
AppliedQuoted 40 lines · Pierre-Eric Pelloux-Prayer
The Mesa issue referenced below pointed out a possible deadlock:
[ 1231.611031] Possible interrupt unsafe locking scenario:[ 1231.611033] CPU0 CPU1 [ 1231.611034] ---- ---- [ 1231.611035] lock(&xa->xa_lock#17); [ 1231.611038] local_irq_disable(); [ 1231.611039] lock(&fence->lock); [ 1231.611041] lock(&xa->xa_lock#17); [ 1231.611044] <Interrupt> [ 1231.611045] lock(&fence->lock); [ 1231.611047]*** DEADLOCK ***
In this example, CPU0 would be any function accessing
job->dependenciesthrough the xa_* functions that doesn't disable interrupts (eg:drm_sched_job_add_dependency,drm_sched_entity_kill_jobs_cb).CPU1 is executing
drm_sched_entity_kill_jobs_cbas a fence signalling callback so in an interrupt context. It will deadlock when trying to grab thexa_lockwhich is already held by CPU0.Replacing all xa_* usage by their xa_*_irq counterparts would fix this issue, but Christian pointed out another issue:
dma_fence_signaltakes fence.lock and so doesdma_fence_add_callback.
dma_fence_signal()// locks f1.lock
->drm_sched_entity_kill_jobs_cb()
-> foreach dependencies
->dma_fence_add_callback()// locks f2.lockThis will deadlock if f1 and f2 share the same spinlock.
To fix both issues, the code iterating on dependencies and re-arming them is moved out to
drm_sched_entity_kill_jobs_work.v2: reworded commit message (Philipp)
v3: added Fixes tag (Philipp)
Quoted 6 lines
Fixes 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini")Reported-by Mikhail Gavrilov <[email protected]>Suggested-by Christian König <[email protected]>Reviewed-by Christian König <[email protected]>Signed-off-by Pierre-Eric Pelloux-Prayer <[email protected]>
Pushed to drm-misc-fixes.
Thx, Pierre-Eric!
P.