Skip to contentSkip to conversation

drm: Don't free jobs in wait_event_interruptible()

PATCH Superseded Steven Price 2 participants 6 messages Sep 26, 2019

Subsystems (3)

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 drm: Don't free jobs in wait_event_interruptible()

drm_sched_cleanup_jobs() attempts to free finished jobs, however because it is called as the condition of wait_event_interruptible() it must not sleep. Unfortuantly some free callbacks (notibly for Panfrost) do sleep.

Instead let's rename drm_sched_cleanup_jobs() to drm_sched_get_cleanup_job() and simply return a job for processing if there is one. The caller can then call the free_job() callback outside the wait_event_interruptible() where sleeping is possible before re-checking and returning to sleep if necessary.

---
Changes from v3:
* drm_sched_main() re-arms the timeout for the next job after calling free_job()

 drivers/gpu/drm/scheduler/sched_main.c | 45 +++++++++++++++-----------
 1 file changed, 26 insertions(+), 19 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +26 -19
index 9a0ee74d82dc..148468447ba9 100644
@@ -622,43 +622,41 @@ static void drm_sched_process_job(struct dma_fence *f, struct dma_fence_cb *cb)
 }
 
 /**
- * drm_sched_cleanup_jobs - destroy finished jobs
+ * drm_sched_get_cleanup_job - fetch the next finished job to be destroyed
  *
  * @sched: scheduler instance
  *
- * Remove all finished jobs from the mirror list and destroy them.
+ * Returns the next finished job from the mirror list (if there is one)
+ * ready for it to be destroyed.
  */
-static void drm_sched_cleanup_jobs(struct drm_gpu_scheduler *sched)
+static struct drm_sched_job *
+drm_sched_get_cleanup_job(struct drm_gpu_scheduler *sched)
 {
+	struct drm_sched_job *job = NULL;
 	unsigned long flags;
 
 	/* Don't destroy jobs while the timeout worker is running */
 	if (sched->timeout != MAX_SCHEDULE_TIMEOUT &&
 	    !cancel_delayed_work(&sched->work_tdr))
-		return;
-
+		return NULL;
 
-	while (!list_empty(&sched->ring_mirror_list)) {
-		struct drm_sched_job *job;
+	spin_lock_irqsave(&sched->job_list_lock, flags);
 
-		job = list_first_entry(&sched->ring_mirror_list,
+	job = list_first_entry_or_null(&sched->ring_mirror_list,
 				       struct drm_sched_job, node);
-		if (!dma_fence_is_signaled(&job->s_fence->finished))
-			break;
 
-		spin_lock_irqsave(&sched->job_list_lock, flags);
+	if (job && dma_fence_is_signaled(&job->s_fence->finished)) {
 		/* remove job from ring_mirror_list */
 		list_del_init(&job->node);
-		spin_unlock_irqrestore(&sched->job_list_lock, flags);
-
-		sched->ops->free_job(job);
+	} else {
+		job = NULL;
+		/* queue timeout for next job */
+		drm_sched_start_timeout(sched);
 	}
 
-	/* queue timeout for next job */
-	spin_lock_irqsave(&sched->job_list_lock, flags);
-	drm_sched_start_timeout(sched);
 	spin_unlock_irqrestore(&sched->job_list_lock, flags);
 
+	return job;
 }
 
 /**
@@ -698,12 +696,21 @@ static int drm_sched_main(void *param)
 		struct drm_sched_fence *s_fence;
 		struct drm_sched_job *sched_job;
 		struct dma_fence *fence;
+		struct drm_sched_job *cleanup_job = NULL;
 
 		wait_event_interruptible(sched->wake_up_worker,
-					 (drm_sched_cleanup_jobs(sched),
+					 (cleanup_job = drm_sched_get_cleanup_job(sched)) ||
 					 (!drm_sched_blocked(sched) &&
 					  (entity = drm_sched_select_entity(sched))) ||
-					 kthread_should_stop()));
+					 kthread_should_stop());
+
+		while (cleanup_job) {
+			sched->ops->free_job(cleanup_job);
+			/* queue timeout for next job */
+			drm_sched_start_timeout(sched);
+
+			cleanup_job = drm_sched_get_cleanup_job(sched);
+		}
 
 		if (!entity)
 			continue;
Signature · 1 line
2.20.1
#1 lore raw
Quoted 93 lines · Steven Price

drm_sched_cleanup_jobs() attempts to free finished jobs, however because it is called as the condition of wait_event_interruptible() it must not sleep. Unfortuantly some free callbacks (notibly for Panfrost) do sleep.

Instead let's rename drm_sched_cleanup_jobs() to drm_sched_get_cleanup_job() and simply return a job for processing if there is one. The caller can then call the free_job() callback outside the wait_event_interruptible() where sleeping is possible before re-checking and returning to sleep if necessary.

---
Changes from v3:
* drm_sched_main() re-arms the timeout for the next job after calling

    free_job()
  drivers/gpu/drm/scheduler/sched_main.c | 45 +++++++++++++++-----------
  1 file changed, 26 insertions(+), 19 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +17 -18
index 9a0ee74d82dc..148468447ba9 100644
@@ -622,43 +622,41 @@ static void drm_sched_process_job(struct dma_fence *f, struct dma_fence_cb *cb)
  }
 
  /**
- * drm_sched_cleanup_jobs - destroy finished jobs
+ * drm_sched_get_cleanup_job - fetch the next finished job to be destroyed
   *
   * @sched: scheduler instance
   *
- * Remove all finished jobs from the mirror list and destroy them.
+ * Returns the next finished job from the mirror list (if there is one)
+ * ready for it to be destroyed.
   */
-static void drm_sched_cleanup_jobs(struct drm_gpu_scheduler *sched)
+static struct drm_sched_job *
+drm_sched_get_cleanup_job(struct drm_gpu_scheduler *sched)
  {
+	struct drm_sched_job *job = NULL;
  	unsigned long flags;
 
  	/* Don't destroy jobs while the timeout worker is running */
  	if (sched->timeout != MAX_SCHEDULE_TIMEOUT &&
  	    !cancel_delayed_work(&sched->work_tdr))
-		return;
-
+		return NULL;
 
-	while (!list_empty(&sched->ring_mirror_list)) {
-		struct drm_sched_job *job;
+	spin_lock_irqsave(&sched->job_list_lock, flags);
 
-		job = list_first_entry(&sched->ring_mirror_list,
+	job = list_first_entry_or_null(&sched->ring_mirror_list,
  				       struct drm_sched_job, node);
-		if (!dma_fence_is_signaled(&job->s_fence->finished))
-			break;
 
-		spin_lock_irqsave(&sched->job_list_lock, flags);
+	if (job && dma_fence_is_signaled(&job->s_fence->finished)) {
  		/* remove job from ring_mirror_list */
  		list_del_init(&job->node);
-		spin_unlock_irqrestore(&sched->job_list_lock, flags);
-
-		sched->ops->free_job(job);
+	} else {
+		job = NULL;
+		/* queue timeout for next job */
+		drm_sched_start_timeout(sched);
  	}
 
-	/* queue timeout for next job */
-	spin_lock_irqsave(&sched->job_list_lock, flags);
-	drm_sched_start_timeout(sched);
  	spin_unlock_irqrestore(&sched->job_list_lock, flags);
 
+	return job;
  }
 
  /**
@@ -698,12 +696,21 @@ static int drm_sched_main(void *param)
  		struct drm_sched_fence *s_fence;
  		struct drm_sched_job *sched_job;
  		struct dma_fence *fence;
+		struct drm_sched_job *cleanup_job = NULL;
 
  		wait_event_interruptible(sched->wake_up_worker,
-					 (drm_sched_cleanup_jobs(sched),
+					 (cleanup_job = drm_sched_get_cleanup_job(sched)) ||
  					 (!drm_sched_blocked(sched) &&
  					  (entity = drm_sched_select_entity(sched))) ||

drivers/gpu/drm/scheduler/sched_main.c static int drm_sched_main(void *param) · line 713

Quoted 10 lines
drivers/gpu/drm/scheduler/sched_main.c +9 -1
-					 kthread_should_stop()));
+					 kthread_should_stop());
+
+		while (cleanup_job) {
+			sched->ops->free_job(cleanup_job);
+			/* queue timeout for next job */
+			drm_sched_start_timeout(sched);
+
+			cleanup_job = drm_sched_get_cleanup_job(sched);
+		}

Why drm_sched_start_timeout is called both here and inside drm_sched_get_cleanup_job ? And also why call it multiple times in the loop instead of only once after the loop is done  ?

Andrey

Quoted 3 lines
drivers/gpu/drm/scheduler/sched_main.c +0 -0
  		if (!entity)
  			continue;
#2 lore raw
Quoted 102 lines · Andrey
Quoted 100 lines · Steven Price

drm_sched_cleanup_jobs() attempts to free finished jobs, however because it is called as the condition of wait_event_interruptible() it must not sleep. Unfortuantly some free callbacks (notibly for Panfrost) do sleep.

Instead let's rename drm_sched_cleanup_jobs() to drm_sched_get_cleanup_job() and simply return a job for processing if there is one. The caller can then call the free_job() callback outside the wait_event_interruptible() where sleeping is possible before re-checking and returning to sleep if necessary.

---
Changes from v3:
* drm_sched_main() re-arms the timeout for the next job after calling

    free_job()
  drivers/gpu/drm/scheduler/sched_main.c | 45 +++++++++++++++-----------
  1 file changed, 26 insertions(+), 19 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +23 -19
index 9a0ee74d82dc..148468447ba9 100644
@@ -622,43 +622,41 @@ static void drm_sched_process_job(struct dma_fence *f, struct dma_fence_cb *cb)
  }
 
  /**
- * drm_sched_cleanup_jobs - destroy finished jobs
+ * drm_sched_get_cleanup_job - fetch the next finished job to be destroyed
   *
   * @sched: scheduler instance
   *
- * Remove all finished jobs from the mirror list and destroy them.
+ * Returns the next finished job from the mirror list (if there is one)
+ * ready for it to be destroyed.
   */
-static void drm_sched_cleanup_jobs(struct drm_gpu_scheduler *sched)
+static struct drm_sched_job *
+drm_sched_get_cleanup_job(struct drm_gpu_scheduler *sched)
  {
+	struct drm_sched_job *job = NULL;
  	unsigned long flags;
 
  	/* Don't destroy jobs while the timeout worker is running */
  	if (sched->timeout != MAX_SCHEDULE_TIMEOUT &&
  	    !cancel_delayed_work(&sched->work_tdr))
-		return;
-
+		return NULL;
 
-	while (!list_empty(&sched->ring_mirror_list)) {
-		struct drm_sched_job *job;
+	spin_lock_irqsave(&sched->job_list_lock, flags);
 
-		job = list_first_entry(&sched->ring_mirror_list,
+	job = list_first_entry_or_null(&sched->ring_mirror_list,
  				       struct drm_sched_job, node);
-		if (!dma_fence_is_signaled(&job->s_fence->finished))
-			break;
 
-		spin_lock_irqsave(&sched->job_list_lock, flags);
+	if (job && dma_fence_is_signaled(&job->s_fence->finished)) {
  		/* remove job from ring_mirror_list */
  		list_del_init(&job->node);
-		spin_unlock_irqrestore(&sched->job_list_lock, flags);
-
-		sched->ops->free_job(job);
+	} else {
+		job = NULL;
+		/* queue timeout for next job */
+		drm_sched_start_timeout(sched);
  	}
 
-	/* queue timeout for next job */
-	spin_lock_irqsave(&sched->job_list_lock, flags);
-	drm_sched_start_timeout(sched);
  	spin_unlock_irqrestore(&sched->job_list_lock, flags);
 
+	return job;
  }
 
  /**
@@ -698,12 +696,21 @@ static int drm_sched_main(void *param)
  		struct drm_sched_fence *s_fence;
  		struct drm_sched_job *sched_job;
  		struct dma_fence *fence;
+		struct drm_sched_job *cleanup_job = NULL;
 
  		wait_event_interruptible(sched->wake_up_worker,
-					 (drm_sched_cleanup_jobs(sched),
+					 (cleanup_job = drm_sched_get_cleanup_job(sched)) ||
  					 (!drm_sched_blocked(sched) &&
  					  (entity = drm_sched_select_entity(sched))) ||
-					 kthread_should_stop()));
+					 kthread_should_stop());
+
+		while (cleanup_job) {
+			sched->ops->free_job(cleanup_job);
+			/* queue timeout for next job */
+			drm_sched_start_timeout(sched);
Quoted 8 lines

drivers/gpu/drm/scheduler/sched_main.c static int drm_sched_main(void *param) · line 713

Quoted 3 lines
drivers/gpu/drm/scheduler/sched_main.c +3 -0
+
+			cleanup_job = drm_sched_get_cleanup_job(sched);
+		}

Why drm_sched_start_timeout is called both here and inside drm_sched_get_cleanup_job ? And also why call it multiple times in the loop instead of only once after the loop is done  ?

Christian pointed out to be that the first thing drm_sched_get_cleanup_job does is call cancel_delayed_work(), and if that returns false then it bails out with a NULL return. So to actually get another job (if one exists) the timeout has to be restarted.

It's also necessary to restart the timeout in the case where the return is NULL which is handled in the function itself.

TBH I'm not sure whether this while loop is worth it - it may be better to replace it with simply:

	if (cleanup_job) {
		sched->ops->free_job(cleanup_job);
		/* queue timeout for next job */
		drm_sched_start_timeout(sched);
	}

The outer loop would then handle the next call to
drm_sched_get_cleanup_job() as necessary.

Steve

#3 lore raw
Quoted 109 lines · Steven Price
Quoted 100 lines · Andrey
Quoted 99 lines · Steven Price

drm_sched_cleanup_jobs() attempts to free finished jobs, however because it is called as the condition of wait_event_interruptible() it must not sleep. Unfortuantly some free callbacks (notibly for Panfrost) do sleep.

Instead let's rename drm_sched_cleanup_jobs() to drm_sched_get_cleanup_job() and simply return a job for processing if there is one. The caller can then call the free_job() callback outside the wait_event_interruptible() where sleeping is possible before re-checking and returning to sleep if necessary.

---
Changes from v3:

   * drm_sched_main() re-arms the timeout for the next job after calling
     free_job()

   drivers/gpu/drm/scheduler/sched_main.c | 45 +++++++++++++++-----------
   1 file changed, 26 insertions(+), 19 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +22 -19
index 9a0ee74d82dc..148468447ba9 100644
@@ -622,43 +622,41 @@ static void drm_sched_process_job(struct dma_fence *f, struct dma_fence_cb *cb)
   }
 
   /**
- * drm_sched_cleanup_jobs - destroy finished jobs
+ * drm_sched_get_cleanup_job - fetch the next finished job to be destroyed
    *
    * @sched: scheduler instance
    *
- * Remove all finished jobs from the mirror list and destroy them.
+ * Returns the next finished job from the mirror list (if there is one)
+ * ready for it to be destroyed.
    */
-static void drm_sched_cleanup_jobs(struct drm_gpu_scheduler *sched)
+static struct drm_sched_job *
+drm_sched_get_cleanup_job(struct drm_gpu_scheduler *sched)
   {
+	struct drm_sched_job *job = NULL;
   	unsigned long flags;
 
   	/* Don't destroy jobs while the timeout worker is running */
   	if (sched->timeout != MAX_SCHEDULE_TIMEOUT &&
   	    !cancel_delayed_work(&sched->work_tdr))
-		return;
-
+		return NULL;
 
-	while (!list_empty(&sched->ring_mirror_list)) {
-		struct drm_sched_job *job;
+	spin_lock_irqsave(&sched->job_list_lock, flags);
 
-		job = list_first_entry(&sched->ring_mirror_list,
+	job = list_first_entry_or_null(&sched->ring_mirror_list,
   				       struct drm_sched_job, node);
-		if (!dma_fence_is_signaled(&job->s_fence->finished))
-			break;
 
-		spin_lock_irqsave(&sched->job_list_lock, flags);
+	if (job && dma_fence_is_signaled(&job->s_fence->finished)) {
   		/* remove job from ring_mirror_list */
   		list_del_init(&job->node);
-		spin_unlock_irqrestore(&sched->job_list_lock, flags);
-
-		sched->ops->free_job(job);
+	} else {
+		job = NULL;
+		/* queue timeout for next job */
+		drm_sched_start_timeout(sched);
   	}
 
-	/* queue timeout for next job */
-	spin_lock_irqsave(&sched->job_list_lock, flags);
-	drm_sched_start_timeout(sched);
   	spin_unlock_irqrestore(&sched->job_list_lock, flags);
 
+	return job;
   }
 
   /**
@@ -698,12 +696,21 @@ static int drm_sched_main(void *param)
   		struct drm_sched_fence *s_fence;
   		struct drm_sched_job *sched_job;
   		struct dma_fence *fence;
+		struct drm_sched_job *cleanup_job = NULL;
 
   		wait_event_interruptible(sched->wake_up_worker,
-					 (drm_sched_cleanup_jobs(sched),
+					 (cleanup_job = drm_sched_get_cleanup_job(sched)) ||
   					 (!drm_sched_blocked(sched) &&
   					  (entity = drm_sched_select_entity(sched))) ||
-					 kthread_should_stop()));
+					 kthread_should_stop());
+
+		while (cleanup_job) {
+			sched->ops->free_job(cleanup_job);
+			/* queue timeout for next job */
Quoted 8 lines

drivers/gpu/drm/scheduler/sched_main.c static int drm_sched_main(void *param) · line 713

Quoted 4 lines
drivers/gpu/drm/scheduler/sched_main.c +4 -0
+			drm_sched_start_timeout(sched);
+
+			cleanup_job = drm_sched_get_cleanup_job(sched);
+		}

Why drm_sched_start_timeout is called both here and inside drm_sched_get_cleanup_job ? And also why call it multiple times in the loop instead of only once after the loop is done  ?

Quoted 4 lines

Christian pointed out to be that the first thing drm_sched_get_cleanup_job does is call cancel_delayed_work(), and if that returns false then it bails out with a NULL return. So to actually get another job (if one exists) the timeout has to be restarted.

For this case where timeout work already in progress note that drm_sched_job_timedout restarts the timeout in it's end so it should be ok to restart the timeout unconditionally inside drm_sched_get_cleanup_job as it was done before.

Quoted 7 lines

It's also necessary to restart the timeout in the case where the return is NULL which is handled in the function itself.

TBH I'm not sure whether this while loop is worth it - it may be better to replace it with simply:

Quoted 8 lines
	if (cleanup_job) {
		sched->ops->free_job(cleanup_job);
		/* queue timeout for next job */
		drm_sched_start_timeout(sched);
	}

The outer loop would then handle the next call to
drm_sched_get_cleanup_job() as necessary.

What outer loop ?

Andrey

Quoted 2 lines

Steve

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#4 lore raw
Quoted 117 lines · Andrey
Quoted 109 lines · Steven Price
Quoted 100 lines · Andrey
Quoted 99 lines · Steven Price

drm_sched_cleanup_jobs() attempts to free finished jobs, however because it is called as the condition of wait_event_interruptible() it must not sleep. Unfortuantly some free callbacks (notibly for Panfrost) do sleep.

Instead let's rename drm_sched_cleanup_jobs() to drm_sched_get_cleanup_job() and simply return a job for processing if there is one. The caller can then call the free_job() callback outside the wait_event_interruptible() where sleeping is possible before re-checking and returning to sleep if necessary.

---
Changes from v3:

   * drm_sched_main() re-arms the timeout for the next job after calling
     free_job()

   drivers/gpu/drm/scheduler/sched_main.c | 45 +++++++++++++++-----------
   1 file changed, 26 insertions(+), 19 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +22 -19
index 9a0ee74d82dc..148468447ba9 100644
@@ -622,43 +622,41 @@ static void drm_sched_process_job(struct dma_fence *f, struct dma_fence_cb *cb)
   }
 
   /**
- * drm_sched_cleanup_jobs - destroy finished jobs
+ * drm_sched_get_cleanup_job - fetch the next finished job to be destroyed
    *
    * @sched: scheduler instance
    *
- * Remove all finished jobs from the mirror list and destroy them.
+ * Returns the next finished job from the mirror list (if there is one)
+ * ready for it to be destroyed.
    */
-static void drm_sched_cleanup_jobs(struct drm_gpu_scheduler *sched)
+static struct drm_sched_job *
+drm_sched_get_cleanup_job(struct drm_gpu_scheduler *sched)
   {
+	struct drm_sched_job *job = NULL;
   	unsigned long flags;
 
   	/* Don't destroy jobs while the timeout worker is running */
   	if (sched->timeout != MAX_SCHEDULE_TIMEOUT &&
   	    !cancel_delayed_work(&sched->work_tdr))
-		return;
-
+		return NULL;
 
-	while (!list_empty(&sched->ring_mirror_list)) {
-		struct drm_sched_job *job;
+	spin_lock_irqsave(&sched->job_list_lock, flags);
 
-		job = list_first_entry(&sched->ring_mirror_list,
+	job = list_first_entry_or_null(&sched->ring_mirror_list,
   				       struct drm_sched_job, node);
-		if (!dma_fence_is_signaled(&job->s_fence->finished))
-			break;
 
-		spin_lock_irqsave(&sched->job_list_lock, flags);
+	if (job && dma_fence_is_signaled(&job->s_fence->finished)) {
   		/* remove job from ring_mirror_list */
   		list_del_init(&job->node);
-		spin_unlock_irqrestore(&sched->job_list_lock, flags);
-
-		sched->ops->free_job(job);
+	} else {
+		job = NULL;
+		/* queue timeout for next job */
+		drm_sched_start_timeout(sched);
   	}
 
-	/* queue timeout for next job */
-	spin_lock_irqsave(&sched->job_list_lock, flags);
-	drm_sched_start_timeout(sched);
   	spin_unlock_irqrestore(&sched->job_list_lock, flags);
 
+	return job;
   }
 
   /**
@@ -698,12 +696,21 @@ static int drm_sched_main(void *param)
   		struct drm_sched_fence *s_fence;
   		struct drm_sched_job *sched_job;
   		struct dma_fence *fence;
+		struct drm_sched_job *cleanup_job = NULL;
 
   		wait_event_interruptible(sched->wake_up_worker,
-					 (drm_sched_cleanup_jobs(sched),
+					 (cleanup_job = drm_sched_get_cleanup_job(sched)) ||
   					 (!drm_sched_blocked(sched) &&
   					  (entity = drm_sched_select_entity(sched))) ||
-					 kthread_should_stop()));
+					 kthread_should_stop());
+
+		while (cleanup_job) {
+			sched->ops->free_job(cleanup_job);
+			/* queue timeout for next job */
Quoted 8 lines

drivers/gpu/drm/scheduler/sched_main.c static int drm_sched_main(void *param) · line 713

Quoted 4 lines
drivers/gpu/drm/scheduler/sched_main.c +4 -0
+			drm_sched_start_timeout(sched);
+
+			cleanup_job = drm_sched_get_cleanup_job(sched);
+		}

Why drm_sched_start_timeout is called both here and inside drm_sched_get_cleanup_job ? And also why call it multiple times in the loop instead of only once after the loop is done  ?

Quoted 4 lines

Christian pointed out to be that the first thing drm_sched_get_cleanup_job does is call cancel_delayed_work(), and if that returns false then it bails out with a NULL return. So to actually get another job (if one exists) the timeout has to be restarted.

Quoted 4 lines

For this case where timeout work already in progress note that drm_sched_job_timedout restarts the timeout in it's end so it should be ok to restart the timeout unconditionally inside drm_sched_get_cleanup_job as it was done before.

I may have misinterpreted Christian[1], but I interpreted the below as meaning that he'd prefer the caller (drm_sched_main()) to handle the drm_sched_start_timeout() in this case:

Quoted 5 lines · Christian
Quoted 2 lines

Alternatively the caller could manually re-arm the timeout after handling the job free.

I don't see anything that could go wrong immediately, but that is probably the cleaner approach.

[1]
https://lore.kernel.org/lkml/[email protected]/

I wasn't entirely sure if it was safe to leave the timeout running while the job free was going on which would be the case if drm_sched_get_cleanup_job() re-armed unconditionally.

Quoted 13 lines
Quoted 13 lines

It's also necessary to restart the timeout in the case where the return is NULL which is handled in the function itself.

TBH I'm not sure whether this while loop is worth it - it may be better to replace it with simply:

	if (cleanup_job) {
		sched->ops->free_job(cleanup_job);
		/* queue timeout for next job */
		drm_sched_start_timeout(sched);
	}
Quoted 5 lines
Quoted 2 lines

The outer loop would then handle the next call to
drm_sched_get_cleanup_job() as necessary.

What outer loop ?

There's a loop round the code in drm_sched_main():

	while (!kthread_should_stop()) {
		[...]
		wait_event_interruptible(...)

		while (cleanup_job) {...}

		if (!entity)
			continue;

		[...]
	}

So after handling the cleanup_job case, the outer loop will loop back round to the wait_event_interruptible() call. One other concern is whether it's possible for the thread to be signalled to stop before the clean up has been completed. The while loop I added ensures that all jobs are actually cleaned up before kthread_should_stop() is checked.

Steve
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#5 lore raw
Quoted 165 lines · Steven Price
Quoted 114 lines · Andrey
Quoted 108 lines · Steven Price
Quoted 99 lines · Andrey
Quoted 98 lines · Steven Price

drm_sched_cleanup_jobs() attempts to free finished jobs, however because it is called as the condition of wait_event_interruptible() it must not sleep. Unfortuantly some free callbacks (notibly for Panfrost) do sleep.

Instead let's rename drm_sched_cleanup_jobs() to drm_sched_get_cleanup_job() and simply return a job for processing if there is one. The caller can then call the free_job() callback outside the wait_event_interruptible() where sleeping is possible before re-checking and returning to sleep if necessary.

---
Changes from v3:

    * drm_sched_main() re-arms the timeout for the next job after calling
      free_job()

    drivers/gpu/drm/scheduler/sched_main.c | 45 +++++++++++++++-----------
    1 file changed, 26 insertions(+), 19 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +21 -19
index 9a0ee74d82dc..148468447ba9 100644
@@ -622,43 +622,41 @@ static void drm_sched_process_job(struct dma_fence *f, struct dma_fence_cb *cb)
    }
 
    /**
- * drm_sched_cleanup_jobs - destroy finished jobs
+ * drm_sched_get_cleanup_job - fetch the next finished job to be destroyed
     *
     * @sched: scheduler instance
     *
- * Remove all finished jobs from the mirror list and destroy them.
+ * Returns the next finished job from the mirror list (if there is one)
+ * ready for it to be destroyed.
     */
-static void drm_sched_cleanup_jobs(struct drm_gpu_scheduler *sched)
+static struct drm_sched_job *
+drm_sched_get_cleanup_job(struct drm_gpu_scheduler *sched)
    {
+	struct drm_sched_job *job = NULL;
    	unsigned long flags;
 
    	/* Don't destroy jobs while the timeout worker is running */
    	if (sched->timeout != MAX_SCHEDULE_TIMEOUT &&
    	    !cancel_delayed_work(&sched->work_tdr))
-		return;
-
+		return NULL;
 
-	while (!list_empty(&sched->ring_mirror_list)) {
-		struct drm_sched_job *job;
+	spin_lock_irqsave(&sched->job_list_lock, flags);
 
-		job = list_first_entry(&sched->ring_mirror_list,
+	job = list_first_entry_or_null(&sched->ring_mirror_list,
    				       struct drm_sched_job, node);
-		if (!dma_fence_is_signaled(&job->s_fence->finished))
-			break;
 
-		spin_lock_irqsave(&sched->job_list_lock, flags);
+	if (job && dma_fence_is_signaled(&job->s_fence->finished)) {
    		/* remove job from ring_mirror_list */
    		list_del_init(&job->node);
-		spin_unlock_irqrestore(&sched->job_list_lock, flags);
-
-		sched->ops->free_job(job);
+	} else {
+		job = NULL;
+		/* queue timeout for next job */
+		drm_sched_start_timeout(sched);
    	}
 
-	/* queue timeout for next job */
-	spin_lock_irqsave(&sched->job_list_lock, flags);
-	drm_sched_start_timeout(sched);
    	spin_unlock_irqrestore(&sched->job_list_lock, flags);
 
+	return job;
    }
 
    /**
@@ -698,12 +696,21 @@ static int drm_sched_main(void *param)
    		struct drm_sched_fence *s_fence;
    		struct drm_sched_job *sched_job;
    		struct dma_fence *fence;
+		struct drm_sched_job *cleanup_job = NULL;
 
    		wait_event_interruptible(sched->wake_up_worker,
-					 (drm_sched_cleanup_jobs(sched),
+					 (cleanup_job = drm_sched_get_cleanup_job(sched)) ||
    					 (!drm_sched_blocked(sched) &&
    					  (entity = drm_sched_select_entity(sched))) ||
-					 kthread_should_stop()));
+					 kthread_should_stop());
+
+		while (cleanup_job) {
+			sched->ops->free_job(cleanup_job);
Quoted 8 lines

drivers/gpu/drm/scheduler/sched_main.c static int drm_sched_main(void *param) · line 713

Quoted 5 lines
drivers/gpu/drm/scheduler/sched_main.c +5 -0
+			/* queue timeout for next job */
+			drm_sched_start_timeout(sched);
+
+			cleanup_job = drm_sched_get_cleanup_job(sched);
+		}

Why drm_sched_start_timeout is called both here and inside drm_sched_get_cleanup_job ? And also why call it multiple times in the loop instead of only once after the loop is done  ?

Quoted 4 lines

Christian pointed out to be that the first thing drm_sched_get_cleanup_job does is call cancel_delayed_work(), and if that returns false then it bails out with a NULL return. So to actually get another job (if one exists) the timeout has to be restarted.

Quoted 4 lines

For this case where timeout work already in progress note that drm_sched_job_timedout restarts the timeout in it's end so it should be ok to restart the timeout unconditionally inside drm_sched_get_cleanup_job as it was done before.

I may have misinterpreted Christian[1], but I interpreted the below as meaning that he'd prefer the caller (drm_sched_main()) to handle the drm_sched_start_timeout() in this case:

Quoted 4 lines · Christian
Quoted 2 lines

Alternatively the caller could manually re-arm the timeout after handling the job free.

I don't see anything that could go wrong immediately, but that is probably the cleaner approach.

[1]
https://lore.kernel.org/lkml/[email protected]/

I wasn't entirely sure if it was safe to leave the timeout running while the job free was going on which would be the case if drm_sched_get_cleanup_job() re-armed unconditionally.

Quoted 12 lines
Quoted 12 lines

It's also necessary to restart the timeout in the case where the return is NULL which is handled in the function itself.

TBH I'm not sure whether this while loop is worth it - it may be better to replace it with simply:

	if (cleanup_job) {
		sched->ops->free_job(cleanup_job);
		/* queue timeout for next job */
		drm_sched_start_timeout(sched);
	}
Quoted 4 lines
Quoted 2 lines

The outer loop would then handle the next call to
drm_sched_get_cleanup_job() as necessary.

What outer loop ?

There's a loop round the code in drm_sched_main():

	while (!kthread_should_stop()) {
		[...]
		wait_event_interruptible(...)

		while (cleanup_job) {...}

		if (!entity)
			continue;

		[...]
	}
Quoted 7 lines

So after handling the cleanup_job case, the outer loop will loop back round to the wait_event_interruptible() call. One other concern is whether it's possible for the thread to be signalled to stop before the clean up has been completed. The while loop I added ensures that all jobs are actually cleaned up before kthread_should_stop() is checked.

Steve

Yes, seems to me only the inner while loop will guarantee proper free of all signaled jobs in mirorr list if the thread is stopped.

Andrey

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel