Skip to contentSkip to conversation

drm/sched: Protect entity->last_scheduled with spinlock

PATCH Superseded Philipp Stanner 2 participants 2 messages Jun 24

Subsystems (4)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Philipp Stanner

author maintainer
#0 lore raw

PATCH 1/1 drm/sched: Protect entity->last_scheduled with spinlock

The entity->last_scheduled field has always been set and read with special RCU functions in addition to memory barriers. There is no obvious reason for that, since the entity lock is available and taken at all places that evaluate the last_scheduled field. The only exception is drm_sched_entity_error(), which is not performance critical in any way.

Improve robustness, readability and maintainability by replacing RCU and barriers with the lock.

As a preparational step, while at it, also guard spsc_queue_pop() with the lock, since spsc_queue is deprecated and supposed to be replaced with a locked list.

---
Tested with drm_sched unit tests, which all ran fine.

---
 drivers/gpu/drm/scheduler/sched_entity.c | 49 +++++++++++-------------
 include/drm/gpu_scheduler.h              |  9 ++---
 2 files changed, 26 insertions(+), 32 deletions(-)
drivers/gpu/drm/scheduler/sched_entity.c +22 -27
index c51101ec70c1..95b2c48a604a 100644
@@ -135,7 +135,6 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,
 	entity->num_sched_list = num_sched_list;
 	entity->sched_list = num_sched_list > 1 ? sched_list : NULL;
 	entity->rq = &sched_list[0]->rq;
-	RCU_INIT_POINTER(entity->last_scheduled, NULL);
 	RB_CLEAR_NODE(&entity->rb_tree_node);
 	init_completion(&entity->entity_idle);
 
@@ -201,10 +200,10 @@ int drm_sched_entity_error(struct drm_sched_entity *entity)
 	struct dma_fence *fence;
 	int r;
 
-	rcu_read_lock();
-	fence = rcu_dereference(entity->last_scheduled);
+	spin_lock(&entity->lock);
+	fence = entity->last_scheduled;
 	r = fence ? fence->error : 0;
-	rcu_read_unlock();
+	spin_unlock(&entity->lock);
 
 	return r;
 }
@@ -288,8 +287,10 @@ void drm_sched_entity_kill(struct drm_sched_entity *entity)
 	wait_for_completion(&entity->entity_idle);
 
 	/* The entity is guaranteed to not be used by the scheduler */
-	prev = rcu_dereference_check(entity->last_scheduled, true);
+	spin_lock(&entity->lock);
+	prev = entity->last_scheduled;
 	dma_fence_get(prev);
+	spin_unlock(&entity->lock);
 	while ((job = drm_sched_entity_queue_pop(entity))) {
 		struct drm_sched_fence *s_fence = job->s_fence;
 
@@ -381,8 +382,8 @@ void drm_sched_entity_fini(struct drm_sched_entity *entity)
 		entity->dependency = NULL;
 	}
 
-	dma_fence_put(rcu_dereference_check(entity->last_scheduled, true));
-	RCU_INIT_POINTER(entity->last_scheduled, NULL);
+	dma_fence_put(entity->last_scheduled);
+	WRITE_ONCE(entity->last_scheduled, NULL);
 	drm_sched_entity_stats_put(entity->stats);
 }
 EXPORT_SYMBOL(drm_sched_entity_fini);
@@ -523,18 +524,18 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
 	if (entity->guilty && atomic_read(entity->guilty))
 		dma_fence_set_error(&sched_job->s_fence->finished, -ECANCELED);
 
-	dma_fence_put(rcu_dereference_check(entity->last_scheduled, true));
-	rcu_assign_pointer(entity->last_scheduled,
-			   dma_fence_get(&sched_job->s_fence->finished));
+	spin_lock(&entity->lock);
+	dma_fence_put(entity->last_scheduled);
+	entity->last_scheduled = dma_fence_get(&sched_job->s_fence->finished);
 
-	/*
-	 * If the queue is empty we allow drm_sched_entity_select_rq() to
-	 * locklessly access ->last_scheduled. This only works if we set the
-	 * pointer before we dequeue and if we a write barrier here.
+	/* A recent rework required taking the spinlock above. Since spsc_queue
+	 * is scheduled for removal as per the DRM-TODO-list, we access it here
+	 * locked already to prepare for that cleanup.
+	 *
+	 * TODO: Fully replace spsc_queue with a locked (h)list.
 	 */
-	smp_wmb();
-
 	spsc_queue_pop(&entity->job_queue);
+	spin_unlock(&entity->lock);
 
 	drm_sched_rq_pop_entity(entity);
 
@@ -561,21 +562,15 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity)
 	if (spsc_queue_count(&entity->job_queue))
 		return;
 
-	/*
-	 * Only when the queue is empty are we guaranteed that
-	 * drm_sched_run_job_work() cannot change entity->last_scheduled. To
-	 * enforce ordering we need a read barrier here. See
-	 * drm_sched_entity_pop_job() for the other side.
-	 */
-	smp_rmb();
-
-	fence = rcu_dereference_check(entity->last_scheduled, true);
+	spin_lock(&entity->lock);
+	fence = entity->last_scheduled;
 
 	/* stay on the same engine if the previous job hasn't finished */
-	if (fence && !dma_fence_is_signaled(fence))
+	if (fence && !dma_fence_is_signaled(fence)) {
+		spin_unlock(&entity->lock);
 		return;
+	}
 
-	spin_lock(&entity->lock);
 	sched = drm_sched_pick_best(entity->sched_list, entity->num_sched_list);
 	rq = sched ? &sched->rq : NULL;
 	if (rq != entity->rq) {
include/drm/gpu_scheduler.h +4 -5
index d61c19e78182..176ff1f936cd 100644
@@ -100,7 +100,8 @@ struct drm_sched_entity {
 	 * @lock:
 	 *
 	 * Lock protecting the run-queue (@rq) to which this entity belongs,
-	 * @priority and the list of schedulers (@sched_list, @num_sched_list).
+	 * @priority, @last_scheduled and the list of schedulers (@sched_list,
+	 * @num_sched_list).
 	 */
 	spinlock_t			lock;
 
@@ -202,11 +203,9 @@ struct drm_sched_entity {
 	/**
 	 * @last_scheduled:
 	 *
-	 * Points to the finished fence of the last scheduled job. Only written
-	 * by drm_sched_entity_pop_job(). Can be accessed locklessly from
-	 * drm_sched_job_arm() if the queue is empty.
+	 * Points to the finished fence of the last scheduled job.
 	 */
-	struct dma_fence __rcu		*last_scheduled;
+	struct dma_fence		*last_scheduled;
 
 	/**
 	 * @last_user: last group leader pushing a job into the entity.

base-commit: 60b5fa6edfef867322fce7c8306e5c4b46211be7

Signature · 1 line
2.54.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Calling dma_fence_put() while holding the entity->lock spinlock introduces a risk of AB-BA lock…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Calling dma_fence_put() while holding the entity->lock spinlock introduces a risk of AB-BA lock inversion and deadlock with driver locks. --

commit 21ecb5d3d71a1baa4ff6c29fb3ddc0deff8299e3
Author: Philipp Stanner <[email protected]>

drm/sched: Protect entity->last_scheduled with spinlock

This commit replaces the RCU protection and memory barriers used for entity->last_scheduled with the existing entity->lock spinlock. It also guards the spsc_queue_pop() call with the same lock as a preparatory step for future cleanups.

Quoted 4 lines
drivers/gpu/drm/scheduler/sched_entity.c +0 -0
index c51101ec70c16..95b2c48a604a6 100644

[ ... ]

Quoted 4 lines
drivers/gpu/drm/scheduler/sched_entity.c +0 -0
@@ -523,18 +524,18 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
 	if (entity->guilty && atomic_read(entity->guilty))
 		dma_fence_set_error(&sched_job->s_fence->finished, -ECANCELED);
 

drivers/gpu/drm/scheduler/sched_entity.c struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity) · line 528

Quoted 10 lines
drivers/gpu/drm/scheduler/sched_entity.c +2 -8
-	dma_fence_put(rcu_dereference_check(entity->last_scheduled, true));
-	rcu_assign_pointer(entity->last_scheduled,
-			   dma_fence_get(&sched_job->s_fence->finished));
-
-	/*
-	 * If the queue is empty we allow drm_sched_entity_select_rq() to
-	 * locklessly access ->last_scheduled. This only works if we set the
-	 * pointer before we dequeue and if we a write barrier here.
+	spin_lock(&entity->lock);
+	dma_fence_put(entity->last_scheduled);

[Severity: High]
Could calling dma_fence_put() while holding entity->lock introduce an AB-BA lock inversion risk here?

If dropping the final reference to the scheduled fence invokes its release callback, this can synchronously drop the reference to the underlying driver hardware fence (fence->parent). When the hardware fence's reference count drops to zero, the driver-specific release callback is invoked synchronously while entity->lock is held.

Drivers commonly acquire their own internal locks (e.g., submission queue locks) in their fence release callbacks. However, during job submission, drivers typically hold these same internal locks before calling drm_sched_entity_push_job(), which subsequently acquires entity->lock. This reverse lock ordering might create a circular lock dependency.

Would it be safer to grab a local reference to entity->last_scheduled, update the pointer, drop the spinlock, and then call dma_fence_put() on the local reference outside the critical section?

Quoted 15 lines
drivers/gpu/drm/scheduler/sched_entity.c +8 -2
+	entity->last_scheduled = dma_fence_get(&sched_job->s_fence->finished);
+
+	/* A recent rework required taking the spinlock above. Since spsc_queue
+	 * is scheduled for removal as per the DRM-TODO-list, we access it here
+	 * locked already to prepare for that cleanup.
+	 *
+	 * TODO: Fully replace spsc_queue with a locked (h)list.
 	 */
-	smp_wmb();
-
 	spsc_queue_pop(&entity->job_queue);
+	spin_unlock(&entity->lock);
 
 	drm_sched_rq_pop_entity(entity);
 
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1