Skip to contentSkip to conversation

drm/panfrost: Handle resetting on timeout better

PATCH In review Steven Price 2 participants 3 messages Oct 9, 2019

Subsystems (3)
lore
Share & tools
lore

Steven Price

author maintainer
#0 lore raw

PATCH v2 1/2 drm/panfrost: Handle resetting on timeout better

Panfrost uses multiple schedulers (one for each slot, so 2 in reality), and on a timeout has to stop all the schedulers to safely perform a reset. However more than one scheduler can trigger a timeout at the same time. This race condition results in jobs being freed while they are still in use.

When stopping other slots use cancel_delayed_work_sync() to ensure that any timeout started for that slot has completed. Also use mutex_trylock() to obtain reset_lock. This means that only one thread attempts the reset, the other threads will simply complete without doing anything (the first thread will wait for this in the call to cancel_delayed_work_sync()).

While we're here and since the function is already dependent on sched_job not being NULL, let's remove the unnecessary checks.

Fixes aa20236784ab ("drm/panfrost: Prevent concurrent resets")

---
v2:
* Added fixes and tested-by tags
* Moved cleanup of panfrost_core_dump() comment to separate patch

 drivers/gpu/drm/panfrost/panfrost_job.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)
drivers/gpu/drm/panfrost/panfrost_job.c +11 -5
index a58551668d9a..21f34d44aac2 100644
@@ -381,13 +381,19 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job)
 		job_read(pfdev, JS_TAIL_LO(js)),
 		sched_job);
 
-	mutex_lock(&pfdev->reset_lock);
+	if (!mutex_trylock(&pfdev->reset_lock))
+		return;
 
-	for (i = 0; i < NUM_JOB_SLOTS; i++)
-		drm_sched_stop(&pfdev->js->queue[i].sched, sched_job);
+	for (i = 0; i < NUM_JOB_SLOTS; i++) {
+		struct drm_gpu_scheduler *sched = &pfdev->js->queue[i].sched;
+
+		drm_sched_stop(sched, sched_job);
+		if (js != i)
+			/* Ensure any timeouts on other slots have finished */
+			cancel_delayed_work_sync(&sched->work_tdr);
+	}
 
-	if (sched_job)
-		drm_sched_increase_karma(sched_job);
+	drm_sched_increase_karma(sched_job);
 
 	spin_lock_irqsave(&pfdev->js->job_lock, flags);
 	for (i = 0; i < NUM_JOB_SLOTS; i++) {
Signature · 1 line
2.20.1
author maintainer
#1 lore raw

PATCH v2 2/2 drm/panfrost: Remove commented out call to panfrost_core_dump

panfrost_core_dump() has never existed in mainline, so remove it and add a TODO entry that core dump support is currently lacking.

---
v2:
* New patch

 drivers/gpu/drm/panfrost/TODO           | 2 ++
 drivers/gpu/drm/panfrost/panfrost_job.c | 2 --
 2 files changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/panfrost/TODO +2 -0
index 536a0d4f8d29..8c811a9e683b 100644
@@ -10,3 +10,5 @@
 
 - Compute job support. So called 'compute only' jobs need to be plumbed up to
   userspace.
+
+- Support core dump on job failure
drivers/gpu/drm/panfrost/panfrost_job.c +0 -2
index 21f34d44aac2..33bf25ba506e 100644
@@ -404,8 +404,6 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job)
 	}
 	spin_unlock_irqrestore(&pfdev->js->job_lock, flags);
 
-	/* panfrost_core_dump(pfdev); */
-
 	panfrost_devfreq_record_transition(pfdev, js);
 	panfrost_device_reset(pfdev);
 
Signature · 1 line
2.20.1
maintainer
#2 lore raw
In reply to Steven Price
Quoted 20 lines · Steven Price

Panfrost uses multiple schedulers (one for each slot, so 2 in reality), and on a timeout has to stop all the schedulers to safely perform a reset. However more than one scheduler can trigger a timeout at the same time. This race condition results in jobs being freed while they are still in use.

When stopping other slots use cancel_delayed_work_sync() to ensure that any timeout started for that slot has completed. Also use mutex_trylock() to obtain reset_lock. This means that only one thread attempts the reset, the other threads will simply complete without doing anything (the first thread will wait for this in the call to cancel_delayed_work_sync()).

While we're here and since the function is already dependent on sched_job not being NULL, let's remove the unnecessary checks.

Fixes aa20236784ab ("drm/panfrost: Prevent concurrent resets")
Quoted 7 lines

---
v2:
* Added fixes and tested-by tags
* Moved cleanup of panfrost_core_dump() comment to separate patch

 drivers/gpu/drm/panfrost/panfrost_job.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

Both patches applied.

Rob