Skip to contentSkip to conversation

drm/scheduler: Fix lockup in drm_sched_entity_kill()

PATCH Applied Dmitry Osipenko 3 participants 7 messages Nov 23, 2022 – Jan 2, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/scheduler: Fix lockup in drm_sched_entity_kill()

The drm_sched_entity_kill() is invoked twice by drm_sched_entity_destroy() while userspace process is exiting or being killed. First time it's invoked when sched entity is flushed and second time when entity is released. This causes a lockup within wait_for_completion(entity_idle) due to how completion API works.

Calling wait_for_completion() more times than complete() was invoked is a error condition that causes lockup because completion internally uses counter for complete/wait calls. The complete_all() must be used instead in such cases.

This patch fixes lockup of Panfrost driver that is reproducible by killing any application in a middle of 3d drawing operation.

Fixes 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini")
---
 drivers/gpu/drm/scheduler/sched_entity.c | 2 +-
 drivers/gpu/drm/scheduler/sched_main.c   | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
drivers/gpu/drm/scheduler/sched_entity.c +1 -1
index fe09e5be79bd..15d04a0ec623 100644
@@ -81,7 +81,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,
 	init_completion(&entity->entity_idle);
 
 	/* We start in an idle state. */
-	complete(&entity->entity_idle);
+	complete_all(&entity->entity_idle);
 
 	spin_lock_init(&entity->rq_lock);
 	spsc_queue_init(&entity->job_queue);
drivers/gpu/drm/scheduler/sched_main.c +2 -2
index 6ce04c2e90c0..857ec20be9e8 100644
@@ -1026,7 +1026,7 @@ static int drm_sched_main(void *param)
 		sched_job = drm_sched_entity_pop_job(entity);
 
 		if (!sched_job) {
-			complete(&entity->entity_idle);
+			complete_all(&entity->entity_idle);
 			continue;
 		}
 
@@ -1037,7 +1037,7 @@ static int drm_sched_main(void *param)
 
 		trace_drm_run_job(sched_job, entity);
 		fence = sched->ops->run_job(sched_job);
-		complete(&entity->entity_idle);
+		complete_all(&entity->entity_idle);
 		drm_sched_fence_scheduled(s_fence);
 
 		if (!IS_ERR_OR_NULL(fence)) {
Signature · 1 line
2.38.1
#1 lore raw

Am 23.11.22 um 01:13 schrieb Dmitry Osipenko:

Quoted 11 lines

The drm_sched_entity_kill() is invoked twice by drm_sched_entity_destroy() while userspace process is exiting or being killed. First time it's invoked when sched entity is flushed and second time when entity is released. This causes a lockup within wait_for_completion(entity_idle) due to how completion API works.

Calling wait_for_completion() more times than complete() was invoked is a error condition that causes lockup because completion internally uses counter for complete/wait calls. The complete_all() must be used instead in such cases.

Quoted 5 lines

This patch fixes lockup of Panfrost driver that is reproducible by killing any application in a middle of 3d drawing operation.

Fixes 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini")

Oh, good point. Reviewed-by: Christian König <[email protected]>

Quoted 40 lines
---
  drivers/gpu/drm/scheduler/sched_entity.c | 2 +-
  drivers/gpu/drm/scheduler/sched_main.c   | 4 ++--
  2 files changed, 3 insertions(+), 3 deletions(-)
drivers/gpu/drm/scheduler/sched_entity.c +1 -1
index fe09e5be79bd..15d04a0ec623 100644
@@ -81,7 +81,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,
  	init_completion(&entity->entity_idle);
 
  	/* We start in an idle state. */
-	complete(&entity->entity_idle);
+	complete_all(&entity->entity_idle);
 
  	spin_lock_init(&entity->rq_lock);
  	spsc_queue_init(&entity->job_queue);
drivers/gpu/drm/scheduler/sched_main.c +2 -2
index 6ce04c2e90c0..857ec20be9e8 100644
@@ -1026,7 +1026,7 @@ static int drm_sched_main(void *param)
  		sched_job = drm_sched_entity_pop_job(entity);
 
  		if (!sched_job) {
-			complete(&entity->entity_idle);
+			complete_all(&entity->entity_idle);
  			continue;
  		}
 
@@ -1037,7 +1037,7 @@ static int drm_sched_main(void *param)
 
  		trace_drm_run_job(sched_job, entity);
  		fence = sched->ops->run_job(sched_job);
-		complete(&entity->entity_idle);
+		complete_all(&entity->entity_idle);
  		drm_sched_fence_scheduled(s_fence);
 
  		if (!IS_ERR_OR_NULL(fence)) {
author Applied
#2 lore raw
In reply to Dmitry Osipenko
Quoted 48 lines · Dmitry Osipenko

The drm_sched_entity_kill() is invoked twice by drm_sched_entity_destroy() while userspace process is exiting or being killed. First time it's invoked when sched entity is flushed and second time when entity is released. This causes a lockup within wait_for_completion(entity_idle) due to how completion API works.

Calling wait_for_completion() more times than complete() was invoked is a error condition that causes lockup because completion internally uses counter for complete/wait calls. The complete_all() must be used instead in such cases.

This patch fixes lockup of Panfrost driver that is reproducible by killing any application in a middle of 3d drawing operation.

Fixes 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini")
---
 drivers/gpu/drm/scheduler/sched_entity.c | 2 +-
 drivers/gpu/drm/scheduler/sched_main.c   | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
drivers/gpu/drm/scheduler/sched_entity.c +1 -1
index fe09e5be79bd..15d04a0ec623 100644
@@ -81,7 +81,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,
 	init_completion(&entity->entity_idle);
 
 	/* We start in an idle state. */
-	complete(&entity->entity_idle);
+	complete_all(&entity->entity_idle);
 
 	spin_lock_init(&entity->rq_lock);
 	spsc_queue_init(&entity->job_queue);
drivers/gpu/drm/scheduler/sched_main.c +1 -1
index 6ce04c2e90c0..857ec20be9e8 100644
@@ -1026,7 +1026,7 @@ static int drm_sched_main(void *param)
 		sched_job = drm_sched_entity_pop_job(entity);
 
 		if (!sched_job) {
-			complete(&entity->entity_idle);
+			complete_all(&entity->entity_idle);
 			continue;
 		}
 

drivers/gpu/drm/scheduler/sched_main.c static int drm_sched_main(void *param) · line 1043

Quoted 8 lines
drivers/gpu/drm/scheduler/sched_main.c +1 -1
@@ -1037,7 +1037,7 @@ static int drm_sched_main(void *param)
 
 		trace_drm_run_job(sched_job, entity);
 		fence = sched->ops->run_job(sched_job);
-		complete(&entity->entity_idle);
+		complete_all(&entity->entity_idle);
 		drm_sched_fence_scheduled(s_fence);
 
 		if (!IS_ERR_OR_NULL(fence)) {

Applied to drm-misc-next

Signature · 2 lines
Best regards,
Dmitry
Tested-by on v1 1/1
#3 lore raw
In reply to Dmitry Osipenko

Hi Dmitry / Christian, thanks for the fix!

(And thanks Melissa for pointing that, saving me lots of time in research heh)

Is this fix planned to be released on 6.2-rc cycle? I've just tested it on Steam Deck, and it resolved a lockup observed (since v6.2-rc1) - exactly the same thing mentioned in the commit message.

FWIW:

Cheers,

Guilherme

#4 lore raw

Hi,

Quoted 11 lines · Guilherme G. Piccoli

Hi Dmitry / Christian, thanks for the fix!

(And thanks Melissa for pointing that, saving me lots of time in research heh)

Is this fix planned to be released on 6.2-rc cycle? I've just tested it on Steam Deck, and it resolved a lockup observed (since v6.2-rc1) - exactly the same thing mentioned in the commit message.

FWIW:

I'll push the patch to misc-fixes as soon as it will be rebased on 6.2-rc. Thanks!

Best regards,
Dmitry

#5 lore raw
Quoted 7 lines · Dmitry Osipenko

[...]
I'll push the patch to misc-fixes as soon as it will be rebased on 6.2-rc. Thanks!

Best regards,
Dmitry

Thank you Dmitry, much appreciated!
Cheers,

Guilherme

author Applied
#6 lore raw
In reply to Dmitry Osipenko
Quoted 48 lines · Dmitry Osipenko

The drm_sched_entity_kill() is invoked twice by drm_sched_entity_destroy() while userspace process is exiting or being killed. First time it's invoked when sched entity is flushed and second time when entity is released. This causes a lockup within wait_for_completion(entity_idle) due to how completion API works.

Calling wait_for_completion() more times than complete() was invoked is a error condition that causes lockup because completion internally uses counter for complete/wait calls. The complete_all() must be used instead in such cases.

This patch fixes lockup of Panfrost driver that is reproducible by killing any application in a middle of 3d drawing operation.

Fixes 2fdb8a8f07c2 ("drm/scheduler: rework entity flush, kill and fini")
---
 drivers/gpu/drm/scheduler/sched_entity.c | 2 +-
 drivers/gpu/drm/scheduler/sched_main.c   | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
drivers/gpu/drm/scheduler/sched_entity.c +1 -1
index fe09e5be79bd..15d04a0ec623 100644
@@ -81,7 +81,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,
 	init_completion(&entity->entity_idle);
 
 	/* We start in an idle state. */
-	complete(&entity->entity_idle);
+	complete_all(&entity->entity_idle);
 
 	spin_lock_init(&entity->rq_lock);
 	spsc_queue_init(&entity->job_queue);
drivers/gpu/drm/scheduler/sched_main.c +1 -1
index 6ce04c2e90c0..857ec20be9e8 100644
@@ -1026,7 +1026,7 @@ static int drm_sched_main(void *param)
 		sched_job = drm_sched_entity_pop_job(entity);
 
 		if (!sched_job) {
-			complete(&entity->entity_idle);
+			complete_all(&entity->entity_idle);
 			continue;
 		}
 

drivers/gpu/drm/scheduler/sched_main.c static int drm_sched_main(void *param) · line 1043

Quoted 8 lines
drivers/gpu/drm/scheduler/sched_main.c +1 -1
@@ -1037,7 +1037,7 @@ static int drm_sched_main(void *param)
 
 		trace_drm_run_job(sched_job, entity);
 		fence = sched->ops->run_job(sched_job);
-		complete(&entity->entity_idle);
+		complete_all(&entity->entity_idle);
 		drm_sched_fence_scheduled(s_fence);
 
 		if (!IS_ERR_OR_NULL(fence)) {

Applied to drm-misc-next-fixes

Signature · 2 lines
Best regards,
Dmitry