Skip to contentSkip to conversation

drm/sched: Introduce more locking to entity

PATCH In review Philipp Stanner 5 participants 11 messages Sep 10 – Sep 17

Subsystems (3)
DRM drivers DRM misc DRM GPU scheduler
lore
Share & tools
lore

Philipp Stanner

author maintainer
#0 lore raw

v3 cover drm/sched: Introduce more locking to entity

Changes since v2:
- Fix ordering bug between spsc_queue_pop() and

    drm_sched_rq_pop_entity().

Changes since v1:
- Remove a bunch of patches; make this series only about locking entity->last_scheduled. The rest shall be done in separate patches and series. Consequently, also do not lock spsc_queue, yet.
- Move lock-cycle patch to first position. (Tvrtko)

Both Tvrtko [1] and I [2] have recently proposed some improvals for drm_sched.

While taking Tvrtko's feedback into account for my patch, I realized that both his and my patch can be fully replaced with a bigger and far more beautiful series.

If I am not mistaken, it turns out that the entire entity->entity_idle completion is also nothing but a workaround around the grave mistake of not using the greatest helper with parallel programming that exists in computer science: Locking.

This series adds locking to the last_scheduled field and all checks related to detect the idleness of the entity. As before, the job_scheduled event queue causes the periodic checks.

This way, we can get rid of memory barriers, RCU, a few lines of code, make things more readable, understandable...

Greetings,
Philipp

[1] https://lore.kernel.org/dri-devel/[email protected]/
[2] https://lore.kernel.org/dri-devel/[email protected]/

Philipp Stanner (3):
drm/sched: Lock drm_sched_rq_pop_entity() externally drm/sched: Lock spsc_queue_pop() in drm_sched_entity_pop_job() drm/sched: Protect entity->last_scheduled with spinlock

 drivers/gpu/drm/scheduler/sched_entity.c | 53 ++++++++++--------------
 drivers/gpu/drm/scheduler/sched_rq.c     |  4 +-
 include/drm/gpu_scheduler.h              | 10 ++---
 3 files changed, 28 insertions(+), 39 deletions(-)
Signature · 1 line
2.55.0
author maintainer
#1 lore raw

PATCH v3 1/3 drm/sched: Lock drm_sched_rq_pop_entity() externally

In order to protect entity->last_scheduled with a spinlock, adding locking to drm_sched_entity_pop_job() is necessary. This would lead to a slightly suboptimal lock-unlock-relock pattern with drm_sched_rq_pop_entity().

As a preparational step for adding the locking, lock drm_sched_rq_pop_entity() externally.

Signed-off-by Philipp Stanner <[email protected]>
---
 drivers/gpu/drm/scheduler/sched_entity.c | 2 ++
 drivers/gpu/drm/scheduler/sched_rq.c     | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/scheduler/sched_entity.c +2 -0
index bf97508a45b9..274d7a702298 100644
@@ -565,7 +565,9 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
 
 	spsc_queue_pop(&entity->job_queue);
 
+	spin_lock(&entity->lock);
 	drm_sched_rq_pop_entity(entity);
+	spin_unlock(&entity->lock);
 
 	/* Jobs and entities might have different lifecycles. Since we're
 	 * removing the job from the entities queue, set the jobs entity pointer
drivers/gpu/drm/scheduler/sched_rq.c +2 -2
index 0464d324d98d..696792a18708 100644
@@ -346,11 +346,12 @@ void drm_sched_rq_pop_entity(struct drm_sched_entity *entity)
 	struct drm_sched_job *next_job;
 	struct drm_sched_rq *rq;
 
+	lockdep_assert_held(&entity->lock);
+
 	/*
 	 * Update the entity's location in the min heap according to
 	 * the timestamp of the next job, if any.
 	 */
-	spin_lock(&entity->lock);
 	rq = entity->rq;
 	spin_lock(&rq->lock);
 	next_job = drm_sched_entity_queue_peek(entity);

@@ -376,7 +377,6 @@ void drm_sched_rq_pop_entity(struct drm_sched_entity *entity)
 		}
 	}
 	spin_unlock(&rq->lock);
-	spin_unlock(&entity->lock);
 }
 
 /**
Signature · 1 line
2.55.0
AI review
#2 lore raw

Re: [PATCH v3 1/3] drm/sched: Lock drm_sched_rq_pop_entity() externally

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Pre-existing NULL pointer dereference in drm_sched_job_arm() if all schedulers in the entity's list are not ready. --

commit 7804887bbadf221d2ccaad830fc83a57f0e74d93
Author: Philipp Stanner <[email protected]>

drm/sched: Lock drm_sched_rq_pop_entity() externally

As a preparational step for protecting entity->last_scheduled with a spinlock, this commit modifies drm_sched_entity_pop_job() to lock drm_sched_rq_pop_entity() externally.

drivers/gpu/drm/scheduler/sched_entity.c struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity) · line 570

Quoted 11 lines
drivers/gpu/drm/scheduler/sched_entity.c +2 -0
index bf97508a45b9a..274d7a7022987 100644
@@ -565,7 +565,9 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
 
 	spsc_queue_pop(&entity->job_queue);
 
+	spin_lock(&entity->lock);
 	drm_sched_rq_pop_entity(entity);
+	spin_unlock(&entity->lock);

[Severity: High]
This isn't a bug introduced by this patch, but this series leaves a pre-existing NULL pointer dereference unresolved in the entity runqueue selection path.

If all schedulers in the entity's list have sched->ready == false (e.g., during a GPU hang or reset, as documented in the drm_sched_job_init comments), drm_sched_pick_best() returns NULL. This causes drm_sched_entity_select_rq() to assign NULL to entity->rq:

drivers/gpu/drm/scheduler/sched_entity.c:drm_sched_entity_select_rq() {
    ...
    sched = drm_sched_pick_best(entity->sched_list, entity->num_sched_list);
    rq = sched ? sched->sched_rq[entity->rq_priority] : NULL;
    if (rq != entity->rq) {
        drm_sched_rq_remove_entity(entity->rq, entity);
        entity->rq = rq;
    }
    ...
}

Upon returning to drm_sched_job_arm(), the code unconditionally dereferences the NULL entity->rq:

drivers/gpu/drm/scheduler/sched_main.c:drm_sched_job_arm() {
    ...
    drm_sched_entity_select_rq(entity);
    sched = entity->rq->sched;
    ...
}

Can this lead to a kernel panic (NULL pointer dereference) if userspace continues to submit jobs via ioctl after a hardware fault?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author maintainer
#3 lore raw

PATCH v3 2/3 drm/sched: Lock spsc_queue_pop() in drm_sched_entity_pop_job()

In reply to Philipp Stanner

As a preparational step to lock the access of entity->last_scheduled, it is necessary to lock the call to spsc_queue_pop() in drm_sched_entity_pop_job(). The reason is that later the existing lock needs to be moved upward and the relative order between spsc_queue_pop() and drm_sched_rq_pop_entity() needs to be preserved.

Guard spsc_queue_pop() with the existing spinlock.

Signed-off-by Philipp Stanner <[email protected]>
---
 drivers/gpu/drm/scheduler/sched_entity.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/gpu/drm/scheduler/sched_entity.c +1 -2
index 274d7a702298..e168f445f2ab 100644
@@ -563,9 +563,8 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
 	 */
 	smp_wmb();
 
-	spsc_queue_pop(&entity->job_queue);
-
 	spin_lock(&entity->lock);
+	spsc_queue_pop(&entity->job_queue);
 	drm_sched_rq_pop_entity(entity);
 	spin_unlock(&entity->lock);
Signature · 1 line
2.55.0
author maintainer
#4 lore raw

PATCH v3 3/3 drm/sched: Protect entity->last_scheduled with spinlock

In reply to Philipp Stanner

The entity->last_scheduled field has always been set and read with special RCU functions in addition to memory barriers.

This was added in

commit 70102d77ff22 ("drm/scheduler: add drm_sched_entity_error and use rcu for last_scheduled")

however, no proper justification for that mechanism was provided. There seems to be no obvious reason, since the entity lock is available and taken at all places that evaluate the last_scheduled field. The only exception is drm_sched_entity_error(), which is not performance critical in any way.

Improve robustness, readability and maintainability by replacing RCU and barriers with the lock.

Signed-off-by Philipp Stanner <[email protected]>
---
 drivers/gpu/drm/scheduler/sched_entity.c | 50 ++++++++++--------------
 include/drm/gpu_scheduler.h              | 10 ++---
 2 files changed, 24 insertions(+), 36 deletions(-)
drivers/gpu/drm/scheduler/sched_entity.c +20 -30
index e168f445f2ab..fa4a9388e5c9 100644
@@ -136,7 +136,6 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,
 			      DRM_SCHED_PRIORITY_KERNEL : priority;
 	entity->num_sched_list = num_sched_list;
 	entity->sched_list = num_sched_list > 1 ? sched_list : NULL;
-	RCU_INIT_POINTER(entity->last_scheduled, NULL);
 	RB_CLEAR_NODE(&entity->rb_tree_node);
 
 	if (!sched_list[0]->sched_rq) {

@@ -233,10 +232,10 @@ int drm_sched_entity_error(struct drm_sched_entity *entity)
 	struct dma_fence *fence;
 	int r;
 
-	rcu_read_lock();
-	fence = rcu_dereference(entity->last_scheduled);
+	spin_lock(&entity->lock);
+	fence = entity->last_scheduled;
 	r = fence ? fence->error : 0;
-	rcu_read_unlock();
+	spin_unlock(&entity->lock);
 
 	return r;
 }

@@ -319,9 +318,10 @@ void drm_sched_entity_kill(struct drm_sched_entity *entity)
 	/* Make sure this entity is not used by the scheduler at the moment */
 	wait_for_completion(&entity->entity_idle);
 
-	/* The entity is guaranteed to not be used by the scheduler */
-	prev = rcu_dereference_check(entity->last_scheduled, true);
+	spin_lock(&entity->lock);
+	prev = entity->last_scheduled;
 	dma_fence_get(prev);
+	spin_unlock(&entity->lock);
 	while ((job = drm_sched_entity_queue_pop(entity))) {
 		struct drm_sched_fence *s_fence = job->s_fence;
 

@@ -413,8 +413,7 @@ void drm_sched_entity_fini(struct drm_sched_entity *entity)
 		entity->dependency = NULL;
 	}
 
-	dma_fence_put(rcu_dereference_check(entity->last_scheduled, true));
-	RCU_INIT_POINTER(entity->last_scheduled, NULL);
+	dma_fence_put(entity->last_scheduled);
 	drm_sched_entity_stats_put(entity->stats);
 }
 EXPORT_SYMBOL(drm_sched_entity_fini);

@@ -536,6 +535,10 @@ drm_sched_job_dependency(struct drm_sched_job *job,
 
 struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
 {
+	/* Helper to avoid dropping the reference while the entity lock is held,
+	 * just to have some more robustness.
+	 */
+	struct dma_fence *prev_last_scheduled;
 	struct drm_sched_job *sched_job;
 
 	sched_job = drm_sched_entity_queue_peek(entity);

@@ -552,22 +555,15 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
 	if (entity->guilty && atomic_read(entity->guilty))
 		dma_fence_set_error(&sched_job->s_fence->finished, -ECANCELED);
 
-	dma_fence_put(rcu_dereference_check(entity->last_scheduled, true));
-	rcu_assign_pointer(entity->last_scheduled,
-			   dma_fence_get(&sched_job->s_fence->finished));
-
-	/*
-	 * If the queue is empty we allow drm_sched_entity_select_rq() to
-	 * locklessly access ->last_scheduled. This only works if we set the
-	 * pointer before we dequeue and if we a write barrier here.
-	 */
-	smp_wmb();
-
 	spin_lock(&entity->lock);
+	prev_last_scheduled = entity->last_scheduled;
+	entity->last_scheduled = dma_fence_get(&sched_job->s_fence->finished);
 	spsc_queue_pop(&entity->job_queue);
 	drm_sched_rq_pop_entity(entity);
 	spin_unlock(&entity->lock);
 
+	dma_fence_put(prev_last_scheduled);
+
 	/* Jobs and entities might have different lifecycles. Since we're
 	 * removing the job from the entities queue, set the jobs entity pointer
 	 * to NULL to prevent any future access of the entity through this job.

@@ -591,21 +587,15 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity)
 	if (spsc_queue_count(&entity->job_queue))
 		return;
 
-	/*
-	 * Only when the queue is empty are we guaranteed that
-	 * drm_sched_run_job_work() cannot change entity->last_scheduled. To
-	 * enforce ordering we need a read barrier here. See
-	 * drm_sched_entity_pop_job() for the other side.
-	 */
-	smp_rmb();
-
-	fence = rcu_dereference_check(entity->last_scheduled, true);
+	spin_lock(&entity->lock);
+	fence = entity->last_scheduled;
 
 	/* stay on the same engine if the previous job hasn't finished */
-	if (fence && !dma_fence_is_signaled(fence))
+	if (fence && !dma_fence_is_signaled(fence)) {
+		spin_unlock(&entity->lock);
 		return;
+	}
 
-	spin_lock(&entity->lock);
 	sched = drm_sched_pick_best(entity->sched_list, entity->num_sched_list);
 	rq = sched ? sched->sched_rq[entity->rq_priority] : NULL;
 	if (rq != entity->rq) {
include/drm/gpu_scheduler.h +4 -6
index 7a64cc11de08..aeeea6efc623 100644
@@ -100,8 +100,8 @@ struct drm_sched_entity {
 	 * @lock:
 	 *
 	 * Lock protecting the run-queue (@rq) to which this entity belongs,
-	 * @priority, the list of schedulers (@sched_list, @num_sched_list) and
-	 * the @rr_ts field.
+	 * @priority, @last_scheduled and the list of schedulers (@sched_list,
+	 * @num_sched_list).
 	 */
 	spinlock_t			lock;
 

@@ -215,11 +215,9 @@ struct drm_sched_entity {
 	/**
 	 * @last_scheduled:
 	 *
-	 * Points to the finished fence of the last scheduled job. Only written
-	 * by drm_sched_entity_pop_job(). Can be accessed locklessly from
-	 * drm_sched_job_arm() if the queue is empty.
+	 * Points to the finished fence of the last scheduled job.
 	 */
-	struct dma_fence __rcu		*last_scheduled;
+	struct dma_fence		*last_scheduled;
 
 	/**
 	 * @last_user: last group leader pushing a job into the entity.
Signature · 1 line
2.55.0
#5 lore raw

Re: [PATCH v3 3/3] drm/sched: Protect entity->last_scheduled with spinlock

+ Christian's e-mail which he reads :)

Quoted 5 lines · Philipp Stanner

The entity->last_scheduled field has always been set and read with special RCU functions in addition to memory barriers.

This was added in

Quoted 7 lines

commit 70102d77ff22 ("drm/scheduler: add drm_sched_entity_error and use rcu for last_scheduled")

however, no proper justification for that mechanism was provided. There seems to be no obvious reason, since the entity lock is available and taken at all places that evaluate the last_scheduled field. The only exception is drm_sched_entity_error(), which is not performance critical in any way.

FWIW the entry paths that I found from the amdgpu side are from the job submission side where it is called a few times for each job via amdgpu_ctx_get_entity(), amdgpu_job_prepare_job() and amdgpu_vm_generation(). Last one being both in amdgpu_job_alloc() and amdgpu_job_run(). If I did not miss any, the change effectively adds four lock-unlock cycles to the job submission path. Plus one more for schedulers which use load balancing in some cases.

I don't know if avoiding that was the reason RCU was chosen, perhaps Christian remembers.

Also, to repeat what I said before, I benchmarked this a bit under extreme circumstances (4x non vsynced vkgears) and did not find a regression in fps.

Quoted 36 lines

Improve robustness, readability and maintainability by replacing RCU and barriers with the lock.

Signed-off-by Philipp Stanner <[email protected]>
---
  drivers/gpu/drm/scheduler/sched_entity.c | 50 ++++++++++--------------
  include/drm/gpu_scheduler.h              | 10 ++---
  2 files changed, 24 insertions(+), 36 deletions(-)
drivers/gpu/drm/scheduler/sched_entity.c +3 -4
index e168f445f2ab..fa4a9388e5c9 100644
@@ -136,7 +136,6 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,
  			      DRM_SCHED_PRIORITY_KERNEL : priority;
  	entity->num_sched_list = num_sched_list;
  	entity->sched_list = num_sched_list > 1 ? sched_list : NULL;
-	RCU_INIT_POINTER(entity->last_scheduled, NULL);
  	RB_CLEAR_NODE(&entity->rb_tree_node);
 
  	if (!sched_list[0]->sched_rq) {

@@ -233,10 +232,10 @@ int drm_sched_entity_error(struct drm_sched_entity *entity)
  	struct dma_fence *fence;
  	int r;
 
-	rcu_read_lock();
-	fence = rcu_dereference(entity->last_scheduled);
+	spin_lock(&entity->lock);
+	fence = entity->last_scheduled;
  	r = fence ? fence->error : 0;
-	rcu_read_unlock();
+	spin_unlock(&entity->lock);
 
  	return r;
  }

drivers/gpu/drm/scheduler/sched_entity.c void drm_sched_entity_kill(struct drm_sched_entity *entity) · line 323

Quoted 8 lines
drivers/gpu/drm/scheduler/sched_entity.c +2 -2
@@ -319,9 +318,10 @@ void drm_sched_entity_kill(struct drm_sched_entity *entity)
  	/* Make sure this entity is not used by the scheduler at the moment */
  	wait_for_completion(&entity->entity_idle);
 
-	/* The entity is guaranteed to not be used by the scheduler */
-	prev = rcu_dereference_check(entity->last_scheduled, true);
+	spin_lock(&entity->lock);
+	prev = entity->last_scheduled;
  	dma_fence_get(prev);

Here you can compact with the idiomatic:

prev = dma_fence_get(entity->last_scheduled);

Otherwise looks plausible to me. Lets see what sashiko will have to say this time round.

Regards,

Tvrtko

Quoted 108 lines
drivers/gpu/drm/scheduler/sched_entity.c +15 -24
+	spin_unlock(&entity->lock);
  	while ((job = drm_sched_entity_queue_pop(entity))) {
  		struct drm_sched_fence *s_fence = job->s_fence;
 

@@ -413,8 +413,7 @@ void drm_sched_entity_fini(struct drm_sched_entity *entity)
  		entity->dependency = NULL;
  	}
 
-	dma_fence_put(rcu_dereference_check(entity->last_scheduled, true));
-	RCU_INIT_POINTER(entity->last_scheduled, NULL);
+	dma_fence_put(entity->last_scheduled);
  	drm_sched_entity_stats_put(entity->stats);
  }
  EXPORT_SYMBOL(drm_sched_entity_fini);

@@ -536,6 +535,10 @@ drm_sched_job_dependency(struct drm_sched_job *job,
 
  struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
  {
+	/* Helper to avoid dropping the reference while the entity lock is held,
+	 * just to have some more robustness.
+	 */
+	struct dma_fence *prev_last_scheduled;
  	struct drm_sched_job *sched_job;
 
  	sched_job = drm_sched_entity_queue_peek(entity);

@@ -552,22 +555,15 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
  	if (entity->guilty && atomic_read(entity->guilty))
  		dma_fence_set_error(&sched_job->s_fence->finished, -ECANCELED);
 
-	dma_fence_put(rcu_dereference_check(entity->last_scheduled, true));
-	rcu_assign_pointer(entity->last_scheduled,
-			   dma_fence_get(&sched_job->s_fence->finished));
-
-	/*
-	 * If the queue is empty we allow drm_sched_entity_select_rq() to
-	 * locklessly access ->last_scheduled. This only works if we set the
-	 * pointer before we dequeue and if we a write barrier here.
-	 */
-	smp_wmb();
-
  	spin_lock(&entity->lock);
+	prev_last_scheduled = entity->last_scheduled;
+	entity->last_scheduled = dma_fence_get(&sched_job->s_fence->finished);
  	spsc_queue_pop(&entity->job_queue);
  	drm_sched_rq_pop_entity(entity);
  	spin_unlock(&entity->lock);
 
+	dma_fence_put(prev_last_scheduled);
+
  	/* Jobs and entities might have different lifecycles. Since we're
  	 * removing the job from the entities queue, set the jobs entity pointer
  	 * to NULL to prevent any future access of the entity through this job.

@@ -591,21 +587,15 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity)
  	if (spsc_queue_count(&entity->job_queue))
  		return;
 
-	/*
-	 * Only when the queue is empty are we guaranteed that
-	 * drm_sched_run_job_work() cannot change entity->last_scheduled. To
-	 * enforce ordering we need a read barrier here. See
-	 * drm_sched_entity_pop_job() for the other side.
-	 */
-	smp_rmb();
-
-	fence = rcu_dereference_check(entity->last_scheduled, true);
+	spin_lock(&entity->lock);
+	fence = entity->last_scheduled;
 
  	/* stay on the same engine if the previous job hasn't finished */
-	if (fence && !dma_fence_is_signaled(fence))
+	if (fence && !dma_fence_is_signaled(fence)) {
+		spin_unlock(&entity->lock);
  		return;
+	}
 
-	spin_lock(&entity->lock);
  	sched = drm_sched_pick_best(entity->sched_list, entity->num_sched_list);
  	rq = sched ? sched->sched_rq[entity->rq_priority] : NULL;
  	if (rq != entity->rq) {
include/drm/gpu_scheduler.h +4 -6
index 7a64cc11de08..aeeea6efc623 100644
@@ -100,8 +100,8 @@ struct drm_sched_entity {
  	 * @lock:
  	 *
  	 * Lock protecting the run-queue (@rq) to which this entity belongs,
-	 * @priority, the list of schedulers (@sched_list, @num_sched_list) and
-	 * the @rr_ts field.
+	 * @priority, @last_scheduled and the list of schedulers (@sched_list,
+	 * @num_sched_list).
  	 */
  	spinlock_t			lock;
 

@@ -215,11 +215,9 @@ struct drm_sched_entity {
  	/**
  	 * @last_scheduled:
  	 *
-	 * Points to the finished fence of the last scheduled job. Only written
-	 * by drm_sched_entity_pop_job(). Can be accessed locklessly from
-	 * drm_sched_job_arm() if the queue is empty.
+	 * Points to the finished fence of the last scheduled job.
  	 */
-	struct dma_fence __rcu		*last_scheduled;
+	struct dma_fence		*last_scheduled;
 
  	/**
  	 * @last_user: last group leader pushing a job into the entity.
#6 lore raw

Re: [PATCH v3 3/3] drm/sched: Protect entity->last_scheduled with spinlock

Quoted 3 lines · Tvrtko Ursulin

Also, to repeat what I said before, I benchmarked this a bit under extreme circumstances (4x non vsynced vkgears) and did not find a regression in fps.

Does that mean you give this an official Tested-by: … ?

Quoted 4 lines

Here you can compact with the idiomatic:

  prev = dma_fence_get(entity->last_scheduled);

Good idea, will do.

Thx,
P.

#7 lore raw

Re: [PATCH v3 3/3] drm/sched: Protect entity->last_scheduled with spinlock

Quoted 7 lines · Philipp Stanner
Quoted 3 lines · Tvrtko Ursulin

Also, to repeat what I said before, I benchmarked this a bit under extreme circumstances (4x non vsynced vkgears) and did not find a regression in fps.

Does that mean you give this an official Tested-by: … ?

Not sure if it was the same version I tested back then so best I re-test it quickly once you post the rebase. I was actually planning to review it once sashiko is happy and Christian at least acks it, which he now did.

Regards,

Tvrtko

Quoted 8 lines
Quoted 3 lines

Here you can compact with the idiomatic:

  prev = dma_fence_get(entity->last_scheduled);

Good idea, will do.

Thx,
P.

reviewer Acked-by on v3 3/3
#8 lore raw

Re: [PATCH v3 3/3] drm/sched: Protect entity->last_scheduled with spinlock

In reply to Philipp Stanner
Quoted 13 lines · Philipp Stanner

The entity->last_scheduled field has always been set and read with special RCU functions in addition to memory barriers.

This was added in

commit 70102d77ff22 ("drm/scheduler: add drm_sched_entity_error and use rcu for last_scheduled")

however, no proper justification for that mechanism was provided. There seems to be no obvious reason, since the entity lock is available and taken at all places that evaluate the last_scheduled field. The only exception is drm_sched_entity_error(), which is not performance critical in any way.

Quoted 4 lines

Improve robustness, readability and maintainability by replacing RCU and barriers with the lock.

Signed-off-by Philipp Stanner <[email protected]>
Acked-by Christian König <[email protected]>
Quoted 148 lines
---
 drivers/gpu/drm/scheduler/sched_entity.c | 50 ++++++++++--------------
 include/drm/gpu_scheduler.h              | 10 ++---
 2 files changed, 24 insertions(+), 36 deletions(-)
drivers/gpu/drm/scheduler/sched_entity.c +20 -30
index e168f445f2ab..fa4a9388e5c9 100644
@@ -136,7 +136,6 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,
 			      DRM_SCHED_PRIORITY_KERNEL : priority;
 	entity->num_sched_list = num_sched_list;
 	entity->sched_list = num_sched_list > 1 ? sched_list : NULL;
-	RCU_INIT_POINTER(entity->last_scheduled, NULL);
 	RB_CLEAR_NODE(&entity->rb_tree_node);
 
 	if (!sched_list[0]->sched_rq) {

@@ -233,10 +232,10 @@ int drm_sched_entity_error(struct drm_sched_entity *entity)
 	struct dma_fence *fence;
 	int r;
 
-	rcu_read_lock();
-	fence = rcu_dereference(entity->last_scheduled);
+	spin_lock(&entity->lock);
+	fence = entity->last_scheduled;
 	r = fence ? fence->error : 0;
-	rcu_read_unlock();
+	spin_unlock(&entity->lock);
 
 	return r;
 }

@@ -319,9 +318,10 @@ void drm_sched_entity_kill(struct drm_sched_entity *entity)
 	/* Make sure this entity is not used by the scheduler at the moment */
 	wait_for_completion(&entity->entity_idle);
 
-	/* The entity is guaranteed to not be used by the scheduler */
-	prev = rcu_dereference_check(entity->last_scheduled, true);
+	spin_lock(&entity->lock);
+	prev = entity->last_scheduled;
 	dma_fence_get(prev);
+	spin_unlock(&entity->lock);
 	while ((job = drm_sched_entity_queue_pop(entity))) {
 		struct drm_sched_fence *s_fence = job->s_fence;
 

@@ -413,8 +413,7 @@ void drm_sched_entity_fini(struct drm_sched_entity *entity)
 		entity->dependency = NULL;
 	}
 
-	dma_fence_put(rcu_dereference_check(entity->last_scheduled, true));
-	RCU_INIT_POINTER(entity->last_scheduled, NULL);
+	dma_fence_put(entity->last_scheduled);
 	drm_sched_entity_stats_put(entity->stats);
 }
 EXPORT_SYMBOL(drm_sched_entity_fini);

@@ -536,6 +535,10 @@ drm_sched_job_dependency(struct drm_sched_job *job,
 
 struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
 {
+	/* Helper to avoid dropping the reference while the entity lock is held,
+	 * just to have some more robustness.
+	 */
+	struct dma_fence *prev_last_scheduled;
 	struct drm_sched_job *sched_job;
 
 	sched_job = drm_sched_entity_queue_peek(entity);

@@ -552,22 +555,15 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
 	if (entity->guilty && atomic_read(entity->guilty))
 		dma_fence_set_error(&sched_job->s_fence->finished, -ECANCELED);
 
-	dma_fence_put(rcu_dereference_check(entity->last_scheduled, true));
-	rcu_assign_pointer(entity->last_scheduled,
-			   dma_fence_get(&sched_job->s_fence->finished));
-
-	/*
-	 * If the queue is empty we allow drm_sched_entity_select_rq() to
-	 * locklessly access ->last_scheduled. This only works if we set the
-	 * pointer before we dequeue and if we a write barrier here.
-	 */
-	smp_wmb();
-
 	spin_lock(&entity->lock);
+	prev_last_scheduled = entity->last_scheduled;
+	entity->last_scheduled = dma_fence_get(&sched_job->s_fence->finished);
 	spsc_queue_pop(&entity->job_queue);
 	drm_sched_rq_pop_entity(entity);
 	spin_unlock(&entity->lock);
 
+	dma_fence_put(prev_last_scheduled);
+
 	/* Jobs and entities might have different lifecycles. Since we're
 	 * removing the job from the entities queue, set the jobs entity pointer
 	 * to NULL to prevent any future access of the entity through this job.

@@ -591,21 +587,15 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity)
 	if (spsc_queue_count(&entity->job_queue))
 		return;
 
-	/*
-	 * Only when the queue is empty are we guaranteed that
-	 * drm_sched_run_job_work() cannot change entity->last_scheduled. To
-	 * enforce ordering we need a read barrier here. See
-	 * drm_sched_entity_pop_job() for the other side.
-	 */
-	smp_rmb();
-
-	fence = rcu_dereference_check(entity->last_scheduled, true);
+	spin_lock(&entity->lock);
+	fence = entity->last_scheduled;
 
 	/* stay on the same engine if the previous job hasn't finished */
-	if (fence && !dma_fence_is_signaled(fence))
+	if (fence && !dma_fence_is_signaled(fence)) {
+		spin_unlock(&entity->lock);
 		return;
+	}
 
-	spin_lock(&entity->lock);
 	sched = drm_sched_pick_best(entity->sched_list, entity->num_sched_list);
 	rq = sched ? sched->sched_rq[entity->rq_priority] : NULL;
 	if (rq != entity->rq) {
include/drm/gpu_scheduler.h +4 -6
index 7a64cc11de08..aeeea6efc623 100644
@@ -100,8 +100,8 @@ struct drm_sched_entity {
 	 * @lock:
 	 *
 	 * Lock protecting the run-queue (@rq) to which this entity belongs,
-	 * @priority, the list of schedulers (@sched_list, @num_sched_list) and
-	 * the @rr_ts field.
+	 * @priority, @last_scheduled and the list of schedulers (@sched_list,
+	 * @num_sched_list).
 	 */
 	spinlock_t			lock;
 

@@ -215,11 +215,9 @@ struct drm_sched_entity {
 	/**
 	 * @last_scheduled:
 	 *
-	 * Points to the finished fence of the last scheduled job. Only written
-	 * by drm_sched_entity_pop_job(). Can be accessed locklessly from
-	 * drm_sched_job_arm() if the queue is empty.
+	 * Points to the finished fence of the last scheduled job.
 	 */
-	struct dma_fence __rcu		*last_scheduled;
+	struct dma_fence		*last_scheduled;
 
 	/**
 	 * @last_user: last group leader pushing a job into the entity.
#9 lore raw
In reply to Philipp Stanner
Quoted 33 lines · Philipp Stanner

Changes since v2:
- Fix ordering bug between spsc_queue_pop() and

     drm_sched_rq_pop_entity().

Changes since v1:
- Remove a bunch of patches; make this series only about locking entity->last_scheduled. The rest shall be done in separate patches and series. Consequently, also do not lock spsc_queue, yet.
- Move lock-cycle patch to first position. (Tvrtko)

Both Tvrtko [1] and I [2] have recently proposed some improvals for drm_sched.

While taking Tvrtko's feedback into account for my patch, I realized that both his and my patch can be fully replaced with a bigger and far more beautiful series.

If I am not mistaken, it turns out that the entire entity->entity_idle completion is also nothing but a workaround around the grave mistake of not using the greatest helper with parallel programming that exists in computer science: Locking.

This series adds locking to the last_scheduled field and all checks related to detect the idleness of the entity. As before, the job_scheduled event queue causes the periodic checks.

This way, we can get rid of memory barriers, RCU, a few lines of code, make things more readable, understandable...

Greetings,
Philipp

Quoted 6 lines

[1] https://lore.kernel.org/dri-devel/[email protected]/
[2] https://lore.kernel.org/dri-devel/[email protected]/

Philipp Stanner (3):
drm/sched: Lock drm_sched_rq_pop_entity() externally drm/sched: Lock spsc_queue_pop() in drm_sched_entity_pop_job()

FWIW if you could review https://lore.kernel.org/dri-devel/[email protected]/ you could drop the first two patches from your series. Extra benefit is that patch fixes a bug and has been tested by the user.

Regards,

Tvrtko

Quoted 7 lines
   drm/sched: Protect entity->last_scheduled with spinlock
  drivers/gpu/drm/scheduler/sched_entity.c | 53 ++++++++++--------------
  drivers/gpu/drm/scheduler/sched_rq.c     |  4 +-
  include/drm/gpu_scheduler.h              | 10 ++---
  3 files changed, 28 insertions(+), 39 deletions(-)
#10 lore raw
Quoted 4 lines · Tvrtko Ursulin

FWIW if you could review
https://lore.kernel.org/dri-devel/[email protected]/
you could drop the first two patches from your series. Extra benefit is that patch fixes a bug and has been tested by the user.

See the other thread; I'm not super-hard opposed, but if we can agree on getting CFS to a non-experimental status by first improving spsc_queue, then it would be nicer from the diff / git log perspective to have these little patches as a preparational base-line for the spsc rework :)

Greetings
Philipp