Skip to contentSkip to conversation

drm/sched: Documentation and refcount improvements

PATCH Superseded Philipp Stanner 3 participants 8 messages Jan 21, 2025 – Jan 27, 2025

Subsystems (4)

v2 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Philipp Stanner

author maintainer
#0 lore raw

v2 cover drm/sched: Documentation and refcount improvements

Changes in v2:
- Document what run_job() is allowed to return. (Tvrtko)
- Delete confusing comment about putting the fence. (Danilo)
- Apply Danilo's RB to patch 1.
- Delete info about job recovery for entities in patch 3. (Danilo, me)
- Set the term "ring" as fix term for both HW rings and FW rings. A ring shall always be the thingy on the CPU ;) (Danilo)
- Many (all) other comments improvements in patch 3. (Danilo)

This is as series succeeding my previous patch [1].

I recognized that we are still referring to a non-existing function and a deprecated one in the callback docu. We should probably also point out the important distinction between hardware and firmware schedulers more cleanly.

Please give me feedback, especially on the RFC comments in patch3.

(This series still fires docu-build-warnings. I want to gather feedback on the opion questions first and will solve them in v2.)

Thank you,
Philipp

[1] https://lore.kernel.org/all/[email protected]/

Philipp Stanner (3):
drm/sched: Document run_job() refcount hazard drm/sched: Adjust outdated docu for run_job() drm/sched: Update timedout_job()'s documentation

 drivers/gpu/drm/scheduler/sched_main.c |   5 +-
 include/drm/gpu_scheduler.h            | 106 ++++++++++++++++---------
 2 files changed, 71 insertions(+), 40 deletions(-)
Signature · 1 line
2.47.1
author maintainer
#1 lore raw

PATCH v2 1/3 drm/sched: Document run_job() refcount hazard

From: Philipp Stanner <[email protected]>

drm_sched_backend_ops.run_job() returns a dma_fence for the scheduler. That fence is signalled by the driver once the hardware completed the associated job. The scheduler does not increment the reference count on that fence, but implicitly expects to inherit this fence from run_job().

This is relatively subtle and prone to misunderstandings.

This implies that, to keep a reference for itself, a driver needs to call dma_fence_get() in addition to dma_fence_init() in that callback.

It's further complicated by the fact that the scheduler even decrements the refcount in drm_sched_run_job_work() since it created a new reference in drm_sched_fence_scheduled(). It does, however, still use its pointer to the fence after calling dma_fence_put() - which is safe because of the aforementioned new reference, but actually still violates the refcounting rules.

Move the call to dma_fence_put() to the position behind the last usage of the fence.

Document the necessity to increment the reference count in drm_sched_backend_ops.run_job().

---
 drivers/gpu/drm/scheduler/sched_main.c |  5 ++---
 include/drm/gpu_scheduler.h            | 19 +++++++++++++++----
 2 files changed, 17 insertions(+), 7 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +2 -3
index 57da84908752..7e69ebc09513 100644
@@ -1218,15 +1218,14 @@ static void drm_sched_run_job_work(struct work_struct *w)
 	drm_sched_fence_scheduled(s_fence, fence);
 
 	if (!IS_ERR_OR_NULL(fence)) {
-		/* Drop for original kref_init of the fence */
-		dma_fence_put(fence);
-
 		r = dma_fence_add_callback(fence, &sched_job->cb,
 					   drm_sched_job_done_cb);
 		if (r == -ENOENT)
 			drm_sched_job_done(sched_job, fence->error);
 		else if (r)
 			DRM_DEV_ERROR(sched->dev, "fence add callback failed (%d)\n", r);
+
+		dma_fence_put(fence);
 	} else {
 		drm_sched_job_done(sched_job, IS_ERR(fence) ?
 				   PTR_ERR(fence) : 0);
include/drm/gpu_scheduler.h +15 -4
index 95e17504e46a..d5cd2a78f27c 100644
@@ -420,10 +420,21 @@ struct drm_sched_backend_ops {
 					 struct drm_sched_entity *s_entity);
 
 	/**
-         * @run_job: Called to execute the job once all of the dependencies
-         * have been resolved.  This may be called multiple times, if
-	 * timedout_job() has happened and drm_sched_job_recovery()
-	 * decides to try it again.
+	 * @run_job: Called to execute the job once all of the dependencies
+	 * have been resolved. This may be called multiple times, if
+	 * timedout_job() has happened and drm_sched_job_recovery() decides to
+	 * try it again.
+	 *
+	 * @sched_job: the job to run
+	 *
+	 * Returns: dma_fence the driver must signal once the hardware has
+	 *	completed the job ("hardware fence").
+	 *
+	 * Note that the scheduler expects to 'inherit' its own reference to
+	 * this fence from the callback. It does not invoke an extra
+	 * dma_fence_get() on it. Consequently, this callback must take a
+	 * reference for the scheduler, and additional ones for the driver's
+	 * respective needs.
 	 */
 	struct dma_fence *(*run_job)(struct drm_sched_job *sched_job);
 
Signature · 1 line
2.47.1
author maintainer
#2 lore raw

PATCH v2 2/3 drm/sched: Adjust outdated docu for run_job()

In reply to Philipp Stanner

The documentation for drm_sched_backend_ops.run_job() mentions a certain function called drm_sched_job_recovery(). This function does not exist. What's actually meant is drm_sched_resubmit_jobs(), which is by now also deprecated.

Remove the mention of the removed function.

Discourage the behavior of drm_sched_backend_ops.run_job() being called multiple times for the same job.

---
Folks, I need input for those "refcount" rules. I say that we either delete that section or someone (Christian?) should provide details about what those rules are, as Danilo requested.

P.

---
 include/drm/gpu_scheduler.h | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)
include/drm/gpu_scheduler.h +10 -5
index d5cd2a78f27c..cf40fdb55541 100644
@@ -421,14 +421,19 @@ struct drm_sched_backend_ops {
 
 	/**
 	 * @run_job: Called to execute the job once all of the dependencies
-	 * have been resolved. This may be called multiple times, if
-	 * timedout_job() has happened and drm_sched_job_recovery() decides to
-	 * try it again.
+	 * have been resolved.
+	 *
+	 * The deprecated drm_sched_resubmit_jobs() (called from
+	 * drm_sched_backend_ops.timedout_job()) can invoke this again with the
+	 * same parameters. Doing this is strongly discouraged because it
+	 * violates dma_fence rules.
 	 *
 	 * @sched_job: the job to run
 	 *
-	 * Returns: dma_fence the driver must signal once the hardware has
-	 *	completed the job ("hardware fence").
+	 * Returns:
+	 * On success: dma_fence the driver must signal once the hardware has
+	 * completed the job ("hardware fence").
+	 * On failure: NULL or an ERR_PTR.
 	 *
 	 * Note that the scheduler expects to 'inherit' its own reference to
 	 * this fence from the callback. It does not invoke an extra
Signature · 1 line
2.47.1
author maintainer
#3 lore raw

PATCH v2 3/3 drm/sched: Update timedout_job()'s documentation

In reply to Philipp Stanner

drm_sched_backend_ops.timedout_job()'s documentation is outdated. It mentions the deprecated function drm_sched_resubmit_job(). Furthermore, it does not point out the important distinction between hardware and firmware schedulers.

Since firmware schedulers tyipically only use one entity per scheduler, timeout handling is significantly more simple because the entity the faulted job came from can just be killed without affecting innocent processes.

Update the documentation with that distinction and other details.

Reformat the docstring to work to a unified style with the other handles.

---
 include/drm/gpu_scheduler.h | 82 ++++++++++++++++++++++---------------
 1 file changed, 49 insertions(+), 33 deletions(-)
include/drm/gpu_scheduler.h +49 -33
index cf40fdb55541..4806740b9023 100644
@@ -394,8 +394,14 @@ static inline bool drm_sched_invalidate_job(struct drm_sched_job *s_job,
 }
 
 enum drm_gpu_sched_stat {
-	DRM_GPU_SCHED_STAT_NONE, /* Reserve 0 */
+	/* Reserve 0 */
+	DRM_GPU_SCHED_STAT_NONE,
+
+	/* Operation succeeded */
 	DRM_GPU_SCHED_STAT_NOMINAL,
+
+	/* Failure because dev is no longer available, for example because
+	 * it was unplugged. */
 	DRM_GPU_SCHED_STAT_ENODEV,
 };
 
@@ -447,43 +453,53 @@ struct drm_sched_backend_ops {
 	 * @timedout_job: Called when a job has taken too long to execute,
 	 * to trigger GPU recovery.
 	 *
-	 * This method is called in a workqueue context.
+	 * @sched_job: The job that has timed out
 	 *
-	 * Drivers typically issue a reset to recover from GPU hangs, and this
-	 * procedure usually follows the following workflow:
+	 * Returns: A drm_gpu_sched_stat enum.
 	 *
-	 * 1. Stop the scheduler using drm_sched_stop(). This will park the
-	 *    scheduler thread and cancel the timeout work, guaranteeing that
-	 *    nothing is queued while we reset the hardware queue
-	 * 2. Try to gracefully stop non-faulty jobs (optional)
-	 * 3. Issue a GPU reset (driver-specific)
-	 * 4. Re-submit jobs using drm_sched_resubmit_jobs()
-	 * 5. Restart the scheduler using drm_sched_start(). At that point, new
-	 *    jobs can be queued, and the scheduler thread is unblocked
+	 * Drivers typically issue a reset to recover from GPU hangs.
+	 * This procedure looks very different depending on whether a firmware
+	 * or a hardware scheduler is being used.
+	 *
+	 * For a FIRMWARE SCHEDULER, each (pseudo-)ring has one scheduler, and
+	 * each scheduler has one entity. Hence, you typically follow those
+	 * steps:
+	 *
+	 * 1. Stop the scheduler using drm_sched_stop(). This will pause the
+	 *    scheduler workqueues and cancel the timeout work, guaranteeing
+	 *    that nothing is queued while we remove the ring.
+	 * 2. Remove the ring. In most (all?) cases the firmware will make sure
+	 *    that the corresponding parts of the hardware are resetted, and that
+	 *    other rings are not impacted.
+	 * 3. Kill the entity the faulted job stems from, and the associated
+	 *    scheduler.
+	 *
+	 *
+	 * For a HARDWARE SCHEDULER, each ring also has one scheduler, but each
+	 * scheduler is typically associated with many entities. This implies
+	 * that all entities associated with the affected scheduler cannot be
+	 * torn down, because this would effectively also kill innocent
+	 * userspace processes which did not submit faulty jobs (for example).
+	 *
+	 * Consequently, the procedure to recover with a hardware scheduler
+	 * should look like this:
+	 *
+	 * 1. Stop all schedulers impacted by the reset using drm_sched_stop().
+	 * 2. Figure out to which entity the faulted job belongs to.
+	 * 3. Kill that entity.
+	 * 4. Issue a GPU reset on all faulty rings (driver-specific).
+	 * 5. Re-submit jobs on all schedulers impacted by re-submitting them to
+	 *    the entities which are still alive.
+	 * 6. Restart all schedulers that were stopped in step #1 using
+	 *    drm_sched_start().
 	 *
 	 * Note that some GPUs have distinct hardware queues but need to reset
 	 * the GPU globally, which requires extra synchronization between the
-	 * timeout handler of the different &drm_gpu_scheduler. One way to
-	 * achieve this synchronization is to create an ordered workqueue
-	 * (using alloc_ordered_workqueue()) at the driver level, and pass this
-	 * queue to drm_sched_init(), to guarantee that timeout handlers are
-	 * executed sequentially. The above workflow needs to be slightly
-	 * adjusted in that case:
-	 *
-	 * 1. Stop all schedulers impacted by the reset using drm_sched_stop()
-	 * 2. Try to gracefully stop non-faulty jobs on all queues impacted by
-	 *    the reset (optional)
-	 * 3. Issue a GPU reset on all faulty queues (driver-specific)
-	 * 4. Re-submit jobs on all schedulers impacted by the reset using
-	 *    drm_sched_resubmit_jobs()
-	 * 5. Restart all schedulers that were stopped in step #1 using
-	 *    drm_sched_start()
-	 *
-	 * Return DRM_GPU_SCHED_STAT_NOMINAL, when all is normal,
-	 * and the underlying driver has started or completed recovery.
-	 *
-	 * Return DRM_GPU_SCHED_STAT_ENODEV, if the device is no longer
-	 * available, i.e. has been unplugged.
+	 * timeout handlers of different schedulers. One way to achieve this
+	 * synchronization is to create an ordered workqueue (using
+	 * alloc_ordered_workqueue()) at the driver level, and pass this queue
+	 * as drm_sched_init()'s @timeout_wq parameter. This will guarantee
+	 * that timeout handlers are executed sequentially.
 	 */
 	enum drm_gpu_sched_stat (*timedout_job)(struct drm_sched_job *sched_job);
 
Signature · 1 line
2.47.1
#7 lore raw
In reply to Philipp Stanner
Quoted 8 lines · Philipp Stanner

Changes in v2:
  - Document what run_job() is allowed to return. (Tvrtko)
  - Delete confusing comment about putting the fence. (Danilo)
  - Apply Danilo's RB to patch 1.
  - Delete info about job recovery for entities in patch 3. (Danilo, me)
  - Set the term "ring" as fix term for both HW rings and FW rings. A ring shall always be the thingy on the CPU ;) (Danilo)

s/CPU/GPU

obviously.

P.

Quoted 34 lines

  - Many (all) other comments improvements in patch 3. (Danilo)

This is as series succeeding my previous patch [1].

I recognized that we are still referring to a non-existing function and
a deprecated one in the callback docu. We should probably also point out
the important distinction between hardware and firmware schedulers more
cleanly.

Please give me feedback, especially on the RFC comments in patch3.

(This series still fires docu-build-warnings. I want to gather feedback
on the opion questions first and will solve them in v2.)

Thank you,
Philipp

[1]
https://lore.kernel.org/all/[email protected]/

Philipp Stanner (3):
  drm/sched: Document run_job() refcount hazard drm/sched: Adjust outdated docu for run_job() drm/sched: Update timedout_job()'s documentation

 drivers/gpu/drm/scheduler/sched_main.c |   5 +- include/drm/gpu_scheduler.h            | 106 ++++++++++++++++------- --

 2 files changed, 71 insertions(+), 40 deletions(-)
maintainer
#4 lore raw

Re: [PATCH v2 3/3] drm/sched: Update timedout_job()'s documentation

In reply to Philipp Stanner
Quoted 41 lines · Philipp Stanner

drm_sched_backend_ops.timedout_job()'s documentation is outdated. It mentions the deprecated function drm_sched_resubmit_job(). Furthermore, it does not point out the important distinction between hardware and firmware schedulers.

Since firmware schedulers tyipically only use one entity per scheduler, timeout handling is significantly more simple because the entity the faulted job came from can just be killed without affecting innocent processes.

Update the documentation with that distinction and other details.

Reformat the docstring to work to a unified style with the other handles.

---
 include/drm/gpu_scheduler.h | 82 ++++++++++++++++++++++---------------
 1 file changed, 49 insertions(+), 33 deletions(-)
include/drm/gpu_scheduler.h +7 -1
index cf40fdb55541..4806740b9023 100644
@@ -394,8 +394,14 @@ static inline bool drm_sched_invalidate_job(struct drm_sched_job *s_job,
 }
 
 enum drm_gpu_sched_stat {
-	DRM_GPU_SCHED_STAT_NONE, /* Reserve 0 */
+	/* Reserve 0 */
+	DRM_GPU_SCHED_STAT_NONE,
+
+	/* Operation succeeded */
 	DRM_GPU_SCHED_STAT_NOMINAL,
+
+	/* Failure because dev is no longer available, for example because
+	 * it was unplugged. */
 	DRM_GPU_SCHED_STAT_ENODEV,
 };
 

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 450

Quoted 4 lines
include/drm/gpu_scheduler.h +0 -1
@@ -447,43 +453,53 @@ struct drm_sched_backend_ops {
 	 * @timedout_job: Called when a job has taken too long to execute,
 	 * to trigger GPU recovery.
 	 *
-	 * This method is called in a workqueue context.

Why remove this line?

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 458

Quoted 5 lines
include/drm/gpu_scheduler.h +2 -2
+	 * @sched_job: The job that has timed out
 	 *
-	 * Drivers typically issue a reset to recover from GPU hangs, and this
-	 * procedure usually follows the following workflow:
+	 * Returns: A drm_gpu_sched_stat enum.

Maybe "The status of the scheduler, defined by &drm_gpu_sched_stat".

I think you forgot to add the corresponding parts in the documentation of drm_gpu_sched_stat.

Quoted 4 lines
include/drm/gpu_scheduler.h +0 -3
 	 *
-	 * 1. Stop the scheduler using drm_sched_stop(). This will park the
-	 *    scheduler thread and cancel the timeout work, guaranteeing that
-	 *    nothing is queued while we reset the hardware queue

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 464

Quoted 10 lines
include/drm/gpu_scheduler.h +5 -5
-	 * 2. Try to gracefully stop non-faulty jobs (optional)
-	 * 3. Issue a GPU reset (driver-specific)
-	 * 4. Re-submit jobs using drm_sched_resubmit_jobs()
-	 * 5. Restart the scheduler using drm_sched_start(). At that point, new
-	 *    jobs can be queued, and the scheduler thread is unblocked
+	 * Drivers typically issue a reset to recover from GPU hangs.
+	 * This procedure looks very different depending on whether a firmware
+	 * or a hardware scheduler is being used.
+	 *
+	 * For a FIRMWARE SCHEDULER, each (pseudo-)ring has one scheduler, and

Why pseudo? It's still a real ring buffer.

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 466

Quoted 2 lines
include/drm/gpu_scheduler.h +2 -0
+	 * each scheduler has one entity. Hence, you typically follow those
+	 * steps:

Maybe better "Hence, the steps taken typically look as follows:".

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 470

Quoted 4 lines
include/drm/gpu_scheduler.h +4 -0
+	 *
+	 * 1. Stop the scheduler using drm_sched_stop(). This will pause the
+	 *    scheduler workqueues and cancel the timeout work, guaranteeing
+	 *    that nothing is queued while we remove the ring.

"while the ring is removed"

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 471

Quoted 1 line
include/drm/gpu_scheduler.h +1 -0
+	 * 2. Remove the ring. In most (all?) cases the firmware will make sure

At least I don't know about other cases and I also don't think it'd make a lot of sense if it'd be different. But of course there's no rule preventing people to implement things weirdly.

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 474

Quoted 3 lines
include/drm/gpu_scheduler.h +3 -0
+	 *    that the corresponding parts of the hardware are resetted, and that
+	 *    other rings are not impacted.
+	 * 3. Kill the entity the faulted job stems from, and the associated

There can only be one entity in this case, so you can drop "the faulted job stems from".

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 479

Quoted 5 lines
include/drm/gpu_scheduler.h +5 -0
+	 *    scheduler.
+	 *
+	 *
+	 * For a HARDWARE SCHEDULER, each ring also has one scheduler, but each
+	 * scheduler is typically associated with many entities. This implies

What about "each scheduler can be scheduling one or more entities"?

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 480

Quoted 1 line
include/drm/gpu_scheduler.h +1 -0
+	 * that all entities associated with the affected scheduler cannot be

I think you want to say that not all entites can be torn down, rather than none of them can be torn down.

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 482

Quoted 2 lines
include/drm/gpu_scheduler.h +2 -0
+	 * torn down, because this would effectively also kill innocent
+	 * userspace processes which did not submit faulty jobs (for example).

This is phrased ambiguously, "kill userspace processs" typically means something different than you mean in this context.

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 489

Quoted 7 lines
include/drm/gpu_scheduler.h +7 -0
+	 *
+	 * Consequently, the procedure to recover with a hardware scheduler
+	 * should look like this:
+	 *
+	 * 1. Stop all schedulers impacted by the reset using drm_sched_stop().
+	 * 2. Figure out to which entity the faulted job belongs to.
+	 * 3. Kill that entity.

I'd combine the two steps: "2. Kill the entity the faulty job originates from".

Quoted 40 lines
include/drm/gpu_scheduler.h +10 -21
+	 * 4. Issue a GPU reset on all faulty rings (driver-specific).
+	 * 5. Re-submit jobs on all schedulers impacted by re-submitting them to
+	 *    the entities which are still alive.
+	 * 6. Restart all schedulers that were stopped in step #1 using
+	 *    drm_sched_start().
 	 *
 	 * Note that some GPUs have distinct hardware queues but need to reset
 	 * the GPU globally, which requires extra synchronization between the
-	 * timeout handler of the different &drm_gpu_scheduler. One way to
-	 * achieve this synchronization is to create an ordered workqueue
-	 * (using alloc_ordered_workqueue()) at the driver level, and pass this
-	 * queue to drm_sched_init(), to guarantee that timeout handlers are
-	 * executed sequentially. The above workflow needs to be slightly
-	 * adjusted in that case:
-	 *
-	 * 1. Stop all schedulers impacted by the reset using drm_sched_stop()
-	 * 2. Try to gracefully stop non-faulty jobs on all queues impacted by
-	 *    the reset (optional)
-	 * 3. Issue a GPU reset on all faulty queues (driver-specific)
-	 * 4. Re-submit jobs on all schedulers impacted by the reset using
-	 *    drm_sched_resubmit_jobs()
-	 * 5. Restart all schedulers that were stopped in step #1 using
-	 *    drm_sched_start()
-	 *
-	 * Return DRM_GPU_SCHED_STAT_NOMINAL, when all is normal,
-	 * and the underlying driver has started or completed recovery.
-	 *
-	 * Return DRM_GPU_SCHED_STAT_ENODEV, if the device is no longer
-	 * available, i.e. has been unplugged.
+	 * timeout handlers of different schedulers. One way to achieve this
+	 * synchronization is to create an ordered workqueue (using
+	 * alloc_ordered_workqueue()) at the driver level, and pass this queue
+	 * as drm_sched_init()'s @timeout_wq parameter. This will guarantee
+	 * that timeout handlers are executed sequentially.
 	 */
 	enum drm_gpu_sched_stat (*timedout_job)(struct drm_sched_job *sched_job);
 
Signature · 1 line
2.47.1
#5 lore raw

Re: [PATCH v2 3/3] drm/sched: Update timedout_job()'s documentation

Quoted 50 lines · Danilo Krummrich
Quoted 49 lines · Philipp Stanner

drm_sched_backend_ops.timedout_job()'s documentation is outdated. It
mentions the deprecated function drm_sched_resubmit_job(). Furthermore,
it does not point out the important distinction between hardware and
firmware schedulers.

Since firmware schedulers tyipically only use one entity per scheduler,
timeout handling is significantly more simple because the entity the
faulted job came from can just be killed without affecting innocent processes.

Update the documentation with that distinction and other details.

Reformat the docstring to work to a unified style with the other handles.

---
 include/drm/gpu_scheduler.h | 82 ++++++++++++++++++++++-----------
----

 1 file changed, 49 insertions(+), 33 deletions(-)
include/drm/gpu_scheduler.h +0 -0
index cf40fdb55541..4806740b9023 100644
@@ -394,8 +394,14 @@ static inline bool
 drm_sched_invalidate_job(struct drm_sched_job *s_job,
  }


 enum drm_gpu_sched_stat {

include/drm/gpu_scheduler.h +4 -1
-	DRM_GPU_SCHED_STAT_NONE, /* Reserve 0 */
+	/* Reserve 0 */
+	DRM_GPU_SCHED_STAT_NONE,
+
+	/* Operation succeeded */

  DRM_GPU_SCHED_STAT_NOMINAL,

include/drm/gpu_scheduler.h +2 -0
+
+	/* Failure because dev is no longer available, for example

because

include/drm/gpu_scheduler.h +1 -0
+	 * it was unplugged. */

  DRM_GPU_SCHED_STAT_ENODEV,
 };

Quoted 8 lines

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 451

Quoted 6 lines
include/drm/gpu_scheduler.h +0 -1
@@ -447,43 +453,53 @@ struct drm_sched_backend_ops {
  	 * @timedout_job: Called when a job has taken too long to
 execute,
  	 * to trigger GPU recovery.
  	 *
-	 * This method is called in a workqueue context.

Why remove this line?

I felt its surplus. All the functions here are callbacks that are invoked by "the scheduler". I thought that's all the driver really needs to know. Why should it care about the wq context?

Also, it's the only function for which the context is mentioned. If we keep it here, we should probably provide it everywhere else, too.

Quoted 10 lines

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 458

Quoted 7 lines
include/drm/gpu_scheduler.h +1 -1
+	 * @sched_job: The job that has timed out
  	 *
-	 * Drivers typically issue a reset to recover from GPU

hangs, and this

include/drm/gpu_scheduler.h +1 -1
-	 * procedure usually follows the following workflow:
+	 * Returns: A drm_gpu_sched_stat enum.

Maybe "The status of the scheduler, defined by &drm_gpu_sched_stat".

Quoted 3 lines

I think you forgot to add the corresponding parts in the documentation of
drm_gpu_sched_stat.

What do you mean, precisely? I added information to that enum. You mean that I should add that that enum is a return type for this callback here?

Quoted 40 lines
Quoted 21 lines

  *

include/drm/gpu_scheduler.h +0 -1
-	 * 1. Stop the scheduler using drm_sched_stop(). This will

park the

include/drm/gpu_scheduler.h +0 -1
-	 *    scheduler thread and cancel the timeout work,

guaranteeing that

include/drm/gpu_scheduler.h +0 -5
-	 *    nothing is queued while we reset the hardware queue
-	 * 2. Try to gracefully stop non-faulty jobs (optional)
-	 * 3. Issue a GPU reset (driver-specific)
-	 * 4. Re-submit jobs using drm_sched_resubmit_jobs()
-	 * 5. Restart the scheduler using drm_sched_start(). At

that point, new

include/drm/gpu_scheduler.h +0 -1
-	 *    jobs can be queued, and the scheduler thread is

unblocked

include/drm/gpu_scheduler.h +1 -0
+	 * Drivers typically issue a reset to recover from GPU

hangs.

include/drm/gpu_scheduler.h +1 -0
+	 * This procedure looks very different depending on

whether a firmware

include/drm/gpu_scheduler.h +3 -0
+	 * or a hardware scheduler is being used.
+	 *
+	 * For a FIRMWARE SCHEDULER, each (pseudo-)ring has one
Quoted 1 line

scheduler, and

Why pseudo? It's still a real ring buffer.

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 466

Quoted 3 lines
include/drm/gpu_scheduler.h +1 -0
+	 * each scheduler has one entity. Hence, you typically

follow those

include/drm/gpu_scheduler.h +1 -0
+	 * steps:

Maybe better "Hence, the steps taken typically look as follows:".

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 470

Quoted 6 lines
include/drm/gpu_scheduler.h +2 -0
+	 *
+	 * 1. Stop the scheduler using drm_sched_stop(). This will

pause the

include/drm/gpu_scheduler.h +1 -0
+	 *    scheduler workqueues and cancel the timeout work,

guaranteeing

include/drm/gpu_scheduler.h +1 -0
+	 *    that nothing is queued while we remove the ring.

"while the ring is removed"

Quoted 8 lines
Quoted 2 lines
include/drm/gpu_scheduler.h +1 -0
+	 * 2. Remove the ring. In most (all?) cases the firmware

will make sure

At least I don't know about other cases and I also don't think it'd make a lot
of sense if it'd be different. But of course there's no rule preventing people
to implement things weirdly.

Seems like we can then use an absolute phrase here and who really wants to do weird things won't be stopped by that anyways :]

Quoted 28 lines
Quoted 6 lines
include/drm/gpu_scheduler.h +1 -0
+	 *    that the corresponding parts of the hardware are

resetted, and that

include/drm/gpu_scheduler.h +2 -0
+	 *    other rings are not impacted.
+	 * 3. Kill the entity the faulted job stems from, and the

associated

There can only be one entity in this case, so you can drop "the faulted job
stems from".

Quoted 7 lines
include/drm/gpu_scheduler.h +4 -0
+	 *    scheduler.
+	 *
+	 *
+	 * For a HARDWARE SCHEDULER, each ring also has one

scheduler, but each

include/drm/gpu_scheduler.h +1 -0
+	 * scheduler is typically associated with many entities.

This implies

What about "each scheduler can be scheduling one or more entities"?

Quoted 2 lines
include/drm/gpu_scheduler.h +1 -0
+	 * that all entities associated with the affected

scheduler cannot be

I think you want to say that not all entites can be torn down, rather than none
of them can be torn down.

Quoted 8 lines
Quoted 4 lines
include/drm/gpu_scheduler.h +1 -0
+	 * torn down, because this would effectively also kill

innocent

include/drm/gpu_scheduler.h +1 -0
+	 * userspace processes which did not submit faulty jobs

(for example).

This is phrased ambiguously, "kill userspace processs" typically means something
different than you mean in this context.

then let's say "down, because this would also affect users that did not provide faulty jobs through their entities.", ack?

Danke,
P.

Quoted 77 lines

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 489

Quoted 11 lines
include/drm/gpu_scheduler.h +2 -0
+	 *
+	 * Consequently, the procedure to recover with a hardware

scheduler

include/drm/gpu_scheduler.h +3 -0
+	 * should look like this:
+	 *
+	 * 1. Stop all schedulers impacted by the reset using

drm_sched_stop().

include/drm/gpu_scheduler.h +1 -0
+	 * 2. Figure out to which entity the faulted job belongs

to.

include/drm/gpu_scheduler.h +1 -0
+	 * 3. Kill that entity.

I'd combine the two steps: "2. Kill the entity the faulty job originates from".

Quoted 62 lines
include/drm/gpu_scheduler.h +1 -0
+	 * 4. Issue a GPU reset on all faulty rings (driver-

specific).

include/drm/gpu_scheduler.h +1 -0
+	 * 5. Re-submit jobs on all schedulers impacted by re-

submitting them to

include/drm/gpu_scheduler.h +2 -0
+	 *    the entities which are still alive.
+	 * 6. Restart all schedulers that were stopped in step #1

using

include/drm/gpu_scheduler.h +1 -0
+	 *    drm_sched_start().

  *
  * Note that some GPUs have distinct hardware queues but need to reset
  * the GPU globally, which requires extra synchronization between the

include/drm/gpu_scheduler.h +0 -1
-	 * timeout handler of the different &drm_gpu_scheduler.

One way to

include/drm/gpu_scheduler.h +0 -1
-	 * achieve this synchronization is to create an ordered

workqueue

include/drm/gpu_scheduler.h +0 -1
-	 * (using alloc_ordered_workqueue()) at the driver level,

and pass this

include/drm/gpu_scheduler.h +0 -1
-	 * queue to drm_sched_init(), to guarantee that timeout

handlers are

include/drm/gpu_scheduler.h +0 -1
-	 * executed sequentially. The above workflow needs to be

slightly

include/drm/gpu_scheduler.h +0 -3
-	 * adjusted in that case:
-	 *
-	 * 1. Stop all schedulers impacted by the reset using

drm_sched_stop()

include/drm/gpu_scheduler.h +0 -1
-	 * 2. Try to gracefully stop non-faulty jobs on all queues

impacted by

include/drm/gpu_scheduler.h +0 -2
-	 *    the reset (optional)
-	 * 3. Issue a GPU reset on all faulty queues (driver-

specific)

include/drm/gpu_scheduler.h +0 -1
-	 * 4. Re-submit jobs on all schedulers impacted by the

reset using

include/drm/gpu_scheduler.h +0 -2
-	 *    drm_sched_resubmit_jobs()
-	 * 5. Restart all schedulers that were stopped in step #1

using

include/drm/gpu_scheduler.h +0 -4
-	 *    drm_sched_start()
-	 *
-	 * Return DRM_GPU_SCHED_STAT_NOMINAL, when all is normal,
-	 * and the underlying driver has started or completed

recovery.

include/drm/gpu_scheduler.h +0 -2
-	 *
-	 * Return DRM_GPU_SCHED_STAT_ENODEV, if the device is no

longer

include/drm/gpu_scheduler.h +1 -1
-	 * available, i.e. has been unplugged.
+	 * timeout handlers of different schedulers. One way to

achieve this

include/drm/gpu_scheduler.h +1 -0
+	 * synchronization is to create an ordered workqueue

(using

include/drm/gpu_scheduler.h +1 -0
+	 * alloc_ordered_workqueue()) at the driver level, and

pass this queue

include/drm/gpu_scheduler.h +1 -0
+	 * as drm_sched_init()'s @timeout_wq parameter. This will

guarantee

include/drm/gpu_scheduler.h +1 -0
+	 * that timeout handlers are executed sequentially.

  */
  enum drm_gpu_sched_stat (*timedout_job)(struct drm_sched_job *sched_job);

Signature · 1 line
2.47.1
maintainer
#6 lore raw

Re: [PATCH v2 3/3] drm/sched: Update timedout_job()'s documentation

Quoted 60 lines · Philipp Stanner
Quoted 50 lines · Danilo Krummrich
Quoted 49 lines · Philipp Stanner

drm_sched_backend_ops.timedout_job()'s documentation is outdated. It
mentions the deprecated function drm_sched_resubmit_job(). Furthermore,
it does not point out the important distinction between hardware and
firmware schedulers.

Since firmware schedulers tyipically only use one entity per scheduler,
timeout handling is significantly more simple because the entity the
faulted job came from can just be killed without affecting innocent processes.

Update the documentation with that distinction and other details.

Reformat the docstring to work to a unified style with the other handles.

---
 include/drm/gpu_scheduler.h | 82 ++++++++++++++++++++++-----------
----

 1 file changed, 49 insertions(+), 33 deletions(-)
include/drm/gpu_scheduler.h +0 -0
index cf40fdb55541..4806740b9023 100644
@@ -394,8 +394,14 @@ static inline bool
 drm_sched_invalidate_job(struct drm_sched_job *s_job,
  }


 enum drm_gpu_sched_stat {

include/drm/gpu_scheduler.h +4 -1
-	DRM_GPU_SCHED_STAT_NONE, /* Reserve 0 */
+	/* Reserve 0 */
+	DRM_GPU_SCHED_STAT_NONE,
+
+	/* Operation succeeded */

  DRM_GPU_SCHED_STAT_NOMINAL,

include/drm/gpu_scheduler.h +2 -0
+
+	/* Failure because dev is no longer available, for example

because

include/drm/gpu_scheduler.h +1 -0
+	 * it was unplugged. */

  DRM_GPU_SCHED_STAT_ENODEV,
 };

Quoted 8 lines

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 451

Quoted 6 lines
include/drm/gpu_scheduler.h +0 -1
@@ -447,43 +453,53 @@ struct drm_sched_backend_ops {
  	 * @timedout_job: Called when a job has taken too long to
 execute,
  	 * to trigger GPU recovery.
  	 *
-	 * This method is called in a workqueue context.

Why remove this line?

Quoted 3 lines

I felt its surplus. All the functions here are callbacks that are invoked by "the scheduler". I thought that's all the driver really needs to know. Why should it care about the wq context?

Yes, I think we should even be more clear and say which workqueue it's scheduled on. The fact that this runs in the context of workqueues is not transpararent to users. It's even that the exact workqueue to use is specified in drm_sched_init().

It's a good hint for drivers in terms of dma-fence handling.

Quoted 3 lines

Also, it's the only function for which the context is mentioned. If we keep it here, we should probably provide it everywhere else, too.

Sounds good.

Quoted 11 lines
Quoted 11 lines

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 458

Quoted 8 lines
include/drm/gpu_scheduler.h +1 -1
+	 * @sched_job: The job that has timed out
  	 *
-	 * Drivers typically issue a reset to recover from GPU

hangs, and this

include/drm/gpu_scheduler.h +1 -1
-	 * procedure usually follows the following workflow:
+	 * Returns: A drm_gpu_sched_stat enum.

Maybe "The status of the scheduler, defined by &drm_gpu_sched_stat".

Quoted 7 lines
Quoted 3 lines

I think you forgot to add the corresponding parts in the documentation of
drm_gpu_sched_stat.

What do you mean, precisely? I added information to that enum. You mean that I should add that that enum is a return type for this callback here?

You did add information to &drm_gpu_sched_stat, but no kernel doc comments you can actually refer to.

Quoted 90 lines
Quoted 41 lines
Quoted 22 lines

  *

include/drm/gpu_scheduler.h +0 -1
-	 * 1. Stop the scheduler using drm_sched_stop(). This will

park the

include/drm/gpu_scheduler.h +0 -1
-	 *    scheduler thread and cancel the timeout work,

guaranteeing that

include/drm/gpu_scheduler.h +0 -5
-	 *    nothing is queued while we reset the hardware queue
-	 * 2. Try to gracefully stop non-faulty jobs (optional)
-	 * 3. Issue a GPU reset (driver-specific)
-	 * 4. Re-submit jobs using drm_sched_resubmit_jobs()
-	 * 5. Restart the scheduler using drm_sched_start(). At

that point, new

include/drm/gpu_scheduler.h +0 -1
-	 *    jobs can be queued, and the scheduler thread is

unblocked

include/drm/gpu_scheduler.h +1 -0
+	 * Drivers typically issue a reset to recover from GPU

hangs.

include/drm/gpu_scheduler.h +1 -0
+	 * This procedure looks very different depending on

whether a firmware

include/drm/gpu_scheduler.h +3 -0
+	 * or a hardware scheduler is being used.
+	 *
+	 * For a FIRMWARE SCHEDULER, each (pseudo-)ring has one
Quoted 1 line

scheduler, and

Why pseudo? It's still a real ring buffer.

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 466

Quoted 3 lines
include/drm/gpu_scheduler.h +1 -0
+	 * each scheduler has one entity. Hence, you typically

follow those

include/drm/gpu_scheduler.h +1 -0
+	 * steps:

Maybe better "Hence, the steps taken typically look as follows:".

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 470

Quoted 6 lines
include/drm/gpu_scheduler.h +2 -0
+	 *
+	 * 1. Stop the scheduler using drm_sched_stop(). This will

pause the

include/drm/gpu_scheduler.h +1 -0
+	 *    scheduler workqueues and cancel the timeout work,

guaranteeing

include/drm/gpu_scheduler.h +1 -0
+	 *    that nothing is queued while we remove the ring.

"while the ring is removed"

Quoted 8 lines
Quoted 2 lines
include/drm/gpu_scheduler.h +1 -0
+	 * 2. Remove the ring. In most (all?) cases the firmware

will make sure

At least I don't know about other cases and I also don't think it'd make a lot
of sense if it'd be different. But of course there's no rule preventing people
to implement things weirdly.

Seems like we can then use an absolute phrase here and who really wants to do weird things won't be stopped by that anyways :]

Quoted 28 lines
Quoted 6 lines
include/drm/gpu_scheduler.h +1 -0
+	 *    that the corresponding parts of the hardware are

resetted, and that

include/drm/gpu_scheduler.h +2 -0
+	 *    other rings are not impacted.
+	 * 3. Kill the entity the faulted job stems from, and the

associated

There can only be one entity in this case, so you can drop "the faulted job
stems from".

Quoted 7 lines
include/drm/gpu_scheduler.h +4 -0
+	 *    scheduler.
+	 *
+	 *
+	 * For a HARDWARE SCHEDULER, each ring also has one

scheduler, but each

include/drm/gpu_scheduler.h +1 -0
+	 * scheduler is typically associated with many entities.

This implies

What about "each scheduler can be scheduling one or more entities"?

Quoted 2 lines
include/drm/gpu_scheduler.h +1 -0
+	 * that all entities associated with the affected

scheduler cannot be

I think you want to say that not all entites can be torn down, rather than none
of them can be torn down.

Quoted 8 lines
Quoted 4 lines
include/drm/gpu_scheduler.h +1 -0
+	 * torn down, because this would effectively also kill

innocent

include/drm/gpu_scheduler.h +1 -0
+	 * userspace processes which did not submit faulty jobs

(for example).

This is phrased ambiguously, "kill userspace processs" typically means something
different than you mean in this context.

Quoted 2 lines

then let's say "down, because this would also affect users that did not provide faulty jobs through their entities.", ack?

Sounds good.

Quoted 83 lines

Danke,
P.

Quoted 77 lines

include/drm/gpu_scheduler.h struct drm_sched_backend_ops { · line 489

Quoted 11 lines
include/drm/gpu_scheduler.h +2 -0
+	 *
+	 * Consequently, the procedure to recover with a hardware

scheduler

include/drm/gpu_scheduler.h +3 -0
+	 * should look like this:
+	 *
+	 * 1. Stop all schedulers impacted by the reset using

drm_sched_stop().

include/drm/gpu_scheduler.h +1 -0
+	 * 2. Figure out to which entity the faulted job belongs

to.

include/drm/gpu_scheduler.h +1 -0
+	 * 3. Kill that entity.

I'd combine the two steps: "2. Kill the entity the faulty job originates from".

Quoted 62 lines
include/drm/gpu_scheduler.h +1 -0
+	 * 4. Issue a GPU reset on all faulty rings (driver-

specific).

include/drm/gpu_scheduler.h +1 -0
+	 * 5. Re-submit jobs on all schedulers impacted by re-

submitting them to

include/drm/gpu_scheduler.h +2 -0
+	 *    the entities which are still alive.
+	 * 6. Restart all schedulers that were stopped in step #1

using

include/drm/gpu_scheduler.h +1 -0
+	 *    drm_sched_start().

  *
  * Note that some GPUs have distinct hardware queues but need to reset
  * the GPU globally, which requires extra synchronization between the

include/drm/gpu_scheduler.h +0 -1
-	 * timeout handler of the different &drm_gpu_scheduler.

One way to

include/drm/gpu_scheduler.h +0 -1
-	 * achieve this synchronization is to create an ordered

workqueue

include/drm/gpu_scheduler.h +0 -1
-	 * (using alloc_ordered_workqueue()) at the driver level,

and pass this

include/drm/gpu_scheduler.h +0 -1
-	 * queue to drm_sched_init(), to guarantee that timeout

handlers are

include/drm/gpu_scheduler.h +0 -1
-	 * executed sequentially. The above workflow needs to be

slightly

include/drm/gpu_scheduler.h +0 -3
-	 * adjusted in that case:
-	 *
-	 * 1. Stop all schedulers impacted by the reset using

drm_sched_stop()

include/drm/gpu_scheduler.h +0 -1
-	 * 2. Try to gracefully stop non-faulty jobs on all queues

impacted by

include/drm/gpu_scheduler.h +0 -2
-	 *    the reset (optional)
-	 * 3. Issue a GPU reset on all faulty queues (driver-

specific)

include/drm/gpu_scheduler.h +0 -1
-	 * 4. Re-submit jobs on all schedulers impacted by the

reset using

include/drm/gpu_scheduler.h +0 -2
-	 *    drm_sched_resubmit_jobs()
-	 * 5. Restart all schedulers that were stopped in step #1

using

include/drm/gpu_scheduler.h +0 -4
-	 *    drm_sched_start()
-	 *
-	 * Return DRM_GPU_SCHED_STAT_NOMINAL, when all is normal,
-	 * and the underlying driver has started or completed

recovery.

include/drm/gpu_scheduler.h +0 -2
-	 *
-	 * Return DRM_GPU_SCHED_STAT_ENODEV, if the device is no

longer

include/drm/gpu_scheduler.h +1 -1
-	 * available, i.e. has been unplugged.
+	 * timeout handlers of different schedulers. One way to

achieve this

include/drm/gpu_scheduler.h +1 -0
+	 * synchronization is to create an ordered workqueue

(using

include/drm/gpu_scheduler.h +1 -0
+	 * alloc_ordered_workqueue()) at the driver level, and

pass this queue

include/drm/gpu_scheduler.h +1 -0
+	 * as drm_sched_init()'s @timeout_wq parameter. This will

guarantee

include/drm/gpu_scheduler.h +1 -0
+	 * that timeout handlers are executed sequentially.

  */
  enum drm_gpu_sched_stat (*timedout_job)(struct drm_sched_job *sched_job);

Signature · 1 line
2.47.1