Skip to contentSkip to conversation

drm/panfrost: Handle resetting on timeout better

PATCH Superseded Steven Price 3 participants 5 messages Oct 7, 2019 – Oct 9, 2019

Subsystems (4)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Steven Price

author maintainer
#0 lore raw

PATCH 1/1 drm/panfrost: Handle resetting on timeout better

Panfrost uses multiple schedulers (one for each slot, so 2 in reality), and on a timeout has to stop all the schedulers to safely perform a reset. However more than one scheduler can trigger a timeout at the same time. This race condition results in jobs being freed while they are still in use.

When stopping other slots use cancel_delayed_work_sync() to ensure that any timeout started for that slot has completed. Also use mutex_trylock() to obtain reset_lock. This means that only one thread attempts the reset, the other threads will simply complete without doing anything (the first thread will wait for this in the call to cancel_delayed_work_sync()).

While we're here and since the function is already dependent on sched_job not being NULL, let's remove the unnecessary checks, along with a commented out call to panfrost_core_dump() which has never existed in mainline.

---
This is a tidied up version of the patch orginally posted here:
http://lkml.kernel.org/r/26ae2a4d-8df1-e8db-3060-41638ed63e2a%40arm.com

 drivers/gpu/drm/panfrost/panfrost_job.c | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)
drivers/gpu/drm/panfrost/panfrost_job.c +11 -6
index a58551668d9a..dcc9a7603685 100644
@@ -381,13 +381,19 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job)
 		job_read(pfdev, JS_TAIL_LO(js)),
 		sched_job);
 
-	mutex_lock(&pfdev->reset_lock);
+	if (!mutex_trylock(&pfdev->reset_lock))
+		return;
 
-	for (i = 0; i < NUM_JOB_SLOTS; i++)
-		drm_sched_stop(&pfdev->js->queue[i].sched, sched_job);
+	for (i = 0; i < NUM_JOB_SLOTS; i++) {
+		struct drm_gpu_scheduler *sched = &pfdev->js->queue[i].sched;
+
+		drm_sched_stop(sched, sched_job);
+		if (js != i)
+			/* Ensure any timeouts on other slots have finished */
+			cancel_delayed_work_sync(&sched->work_tdr);
+	}
 
-	if (sched_job)
-		drm_sched_increase_karma(sched_job);
+	drm_sched_increase_karma(sched_job);
 
 	spin_lock_irqsave(&pfdev->js->job_lock, flags);
 	for (i = 0; i < NUM_JOB_SLOTS; i++) {
@@ -398,7 +404,6 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job)
 	}
 	spin_unlock_irqrestore(&pfdev->js->job_lock, flags);
 
-	/* panfrost_core_dump(pfdev); */
 
 	panfrost_devfreq_record_transition(pfdev, js);
 	panfrost_device_reset(pfdev);
Signature · 6 lines
2.20.1

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#1 lore raw

Hi Steven,

Quoted 13 lines · Steven Price

Panfrost uses multiple schedulers (one for each slot, so 2 in reality), and on a timeout has to stop all the schedulers to safely perform a reset. However more than one scheduler can trigger a timeout at the same time. This race condition results in jobs being freed while they are still in use.

When stopping other slots use cancel_delayed_work_sync() to ensure that any timeout started for that slot has completed. Also use mutex_trylock() to obtain reset_lock. This means that only one thread attempts the reset, the other threads will simply complete without doing anything (the first thread will wait for this in the call to cancel_delayed_work_sync()).

Quoted 5 lines

While we're here and since the function is already dependent on sched_job not being NULL, let's remove the unnecessary checks, along with a commented out call to panfrost_core_dump() which has never existed in mainline.

A Fixes: tags would be welcome here so it would be backported to v5.3

Quoted 24 lines

---
This is a tidied up version of the patch orginally posted here:
http://lkml.kernel.org/r/26ae2a4d-8df1-e8db-3060-41638ed63e2a%40arm.com

 drivers/gpu/drm/panfrost/panfrost_job.c | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)
drivers/gpu/drm/panfrost/panfrost_job.c +4 -3
index a58551668d9a..dcc9a7603685 100644
@@ -381,13 +381,19 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job)
 		job_read(pfdev, JS_TAIL_LO(js)),
 		sched_job);
 
-	mutex_lock(&pfdev->reset_lock);
+	if (!mutex_trylock(&pfdev->reset_lock))
+		return;
 
-	for (i = 0; i < NUM_JOB_SLOTS; i++)
-		drm_sched_stop(&pfdev->js->queue[i].sched, sched_job);
+	for (i = 0; i < NUM_JOB_SLOTS; i++) {
+		struct drm_gpu_scheduler *sched = &pfdev->js->queue[i].sched;

drivers/gpu/drm/panfrost/panfrost_job.c static void panfrost_job_timedout(struct drm_sched_job *sched_job) · line 396

Quoted 10 lines
drivers/gpu/drm/panfrost/panfrost_job.c +7 -2
+
+		drm_sched_stop(sched, sched_job);
+		if (js != i)
+			/* Ensure any timeouts on other slots have finished */
+			cancel_delayed_work_sync(&sched->work_tdr);
+	}
 
-	if (sched_job)
-		drm_sched_increase_karma(sched_job);
+	drm_sched_increase_karma(sched_job);

Indeed looks cleaner.

drivers/gpu/drm/panfrost/panfrost_job.c static void panfrost_job_timedout(struct drm_sched_job *sched_job) · line 401

Quoted 8 lines
drivers/gpu/drm/panfrost/panfrost_job.c +0 -1
 	spin_lock_irqsave(&pfdev->js->job_lock, flags);
 	for (i = 0; i < NUM_JOB_SLOTS; i++) {
@@ -398,7 +404,6 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job)
 	}
 	spin_unlock_irqrestore(&pfdev->js->job_lock, flags);
 
-	/* panfrost_core_dump(pfdev); */

This should be cleaned in another patch !

drivers/gpu/drm/panfrost/panfrost_job.c static void panfrost_job_timedout(struct drm_sched_job *sched_job)

Quoted 4 lines
drivers/gpu/drm/panfrost/panfrost_job.c +0 -0
 	panfrost_devfreq_record_transition(pfdev, js);
 	panfrost_device_reset(pfdev);
 

Thanks,
Testing it right now with the last change removed (doesn't apply on v5.3 with it), results in a few hours... or minutes !

Neil
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#2 lore raw
Quoted 68 lines · Neil Armstrong

Hi Steven,

Quoted 13 lines · Steven Price

Panfrost uses multiple schedulers (one for each slot, so 2 in reality), and on a timeout has to stop all the schedulers to safely perform a reset. However more than one scheduler can trigger a timeout at the same time. This race condition results in jobs being freed while they are still in use.

When stopping other slots use cancel_delayed_work_sync() to ensure that any timeout started for that slot has completed. Also use mutex_trylock() to obtain reset_lock. This means that only one thread attempts the reset, the other threads will simply complete without doing anything (the first thread will wait for this in the call to cancel_delayed_work_sync()).

Quoted 5 lines

While we're here and since the function is already dependent on sched_job not being NULL, let's remove the unnecessary checks, along with a commented out call to panfrost_core_dump() which has never existed in mainline.

A Fixes: tags would be welcome here so it would be backported to v5.3

Quoted 24 lines

---
This is a tidied up version of the patch orginally posted here:
http://lkml.kernel.org/r/26ae2a4d-8df1-e8db-3060-41638ed63e2a%40arm.com

  drivers/gpu/drm/panfrost/panfrost_job.c | 17 +++++++++++------
  1 file changed, 11 insertions(+), 6 deletions(-)
drivers/gpu/drm/panfrost/panfrost_job.c +4 -3
index a58551668d9a..dcc9a7603685 100644
@@ -381,13 +381,19 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job)
  		job_read(pfdev, JS_TAIL_LO(js)),
  		sched_job);
 
-	mutex_lock(&pfdev->reset_lock);
+	if (!mutex_trylock(&pfdev->reset_lock))
+		return;
 
-	for (i = 0; i < NUM_JOB_SLOTS; i++)
-		drm_sched_stop(&pfdev->js->queue[i].sched, sched_job);
+	for (i = 0; i < NUM_JOB_SLOTS; i++) {
+		struct drm_gpu_scheduler *sched = &pfdev->js->queue[i].sched;

drivers/gpu/drm/panfrost/panfrost_job.c static void panfrost_job_timedout(struct drm_sched_job *sched_job) · line 396

Quoted 10 lines
drivers/gpu/drm/panfrost/panfrost_job.c +7 -2
+
+		drm_sched_stop(sched, sched_job);
+		if (js != i)
+			/* Ensure any timeouts on other slots have finished */
+			cancel_delayed_work_sync(&sched->work_tdr);
+	}
 
-	if (sched_job)
-		drm_sched_increase_karma(sched_job);
+	drm_sched_increase_karma(sched_job);

Indeed looks cleaner.

Quoted 7 lines
drivers/gpu/drm/panfrost/panfrost_job.c +0 -0
  	spin_lock_irqsave(&pfdev->js->job_lock, flags);
  	for (i = 0; i < NUM_JOB_SLOTS; i++) {
@@ -398,7 +404,6 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job)
  	}
  	spin_unlock_irqrestore(&pfdev->js->job_lock, flags);
 
Quoted 3 lines

drivers/gpu/drm/panfrost/panfrost_job.c static void panfrost_job_timedout(struct drm_sched_job *sched_job) · line 401

Quoted 1 line
drivers/gpu/drm/panfrost/panfrost_job.c +0 -1
-	/* panfrost_core_dump(pfdev); */

This should be cleaned in another patch !

Seems to me that this should be some kind of TODO, see etnaviv_core_dump() for the kind of things we could be doing.

Maybe we can delete this line and mention this in the TODO file?

Cheers,

Tomeu

Quoted 12 lines

drivers/gpu/drm/panfrost/panfrost_job.c static void panfrost_job_timedout(struct drm_sched_job *sched_job)

Quoted 4 lines
drivers/gpu/drm/panfrost/panfrost_job.c +0 -0
  	panfrost_devfreq_record_transition(pfdev, js);
  	panfrost_device_reset(pfdev);
 

Thanks,
Testing it right now with the last change removed (doesn't apply on v5.3 with it), results in a few hours... or minutes !

Neil

author maintainer
#3 lore raw
Quoted 71 lines · Tomeu Vizoso
Quoted 60 lines · Neil Armstrong

Hi Steven,

Quoted 13 lines · Steven Price

Panfrost uses multiple schedulers (one for each slot, so 2 in reality), and on a timeout has to stop all the schedulers to safely perform a reset. However more than one scheduler can trigger a timeout at the same time. This race condition results in jobs being freed while they are still in use.

When stopping other slots use cancel_delayed_work_sync() to ensure that any timeout started for that slot has completed. Also use mutex_trylock() to obtain reset_lock. This means that only one thread attempts the reset, the other threads will simply complete without doing anything (the first thread will wait for this in the call to cancel_delayed_work_sync()).

Quoted 5 lines

While we're here and since the function is already dependent on sched_job not being NULL, let's remove the unnecessary checks, along with a commented out call to panfrost_core_dump() which has never existed in mainline.

A Fixes: tags would be welcome here so it would be backported to v5.3

Quoted 23 lines

---
This is a tidied up version of the patch orginally posted here:
http://lkml.kernel.org/r/26ae2a4d-8df1-e8db-3060-41638ed63e2a%40arm.com

  drivers/gpu/drm/panfrost/panfrost_job.c | 17 +++++++++++------

  1 file changed, 11 insertions(+), 6 deletions(-)
drivers/gpu/drm/panfrost/panfrost_job.c +3 -1
index a58551668d9a..dcc9a7603685 100644
@@ -381,13 +381,19 @@ static void panfrost_job_timedout(struct
 drm_sched_job *sched_job)
           job_read(pfdev, JS_TAIL_LO(js)),
           sched_job);
   -    mutex_lock(&pfdev->reset_lock);
+    if (!mutex_trylock(&pfdev->reset_lock))
+        return;
   -    for (i = 0; i < NUM_JOB_SLOTS; i++)
-        drm_sched_stop(&pfdev->js->queue[i].sched, sched_job);
+    for (i = 0; i < NUM_JOB_SLOTS; i++) {

drivers/gpu/drm/panfrost/panfrost_job.c static void panfrost_job_timedout(struct · line 397

Quoted 10 lines
drivers/gpu/drm/panfrost/panfrost_job.c +8 -1
+        struct drm_gpu_scheduler *sched = &pfdev->js->queue[i].sched;
+
+        drm_sched_stop(sched, sched_job);
+        if (js != i)
+            /* Ensure any timeouts on other slots have finished */
+            cancel_delayed_work_sync(&sched->work_tdr);
+    }
   -    if (sched_job)
-        drm_sched_increase_karma(sched_job);
+    drm_sched_increase_karma(sched_job);

Indeed looks cleaner.

Quoted 9 lines
Quoted 7 lines
drivers/gpu/drm/panfrost/panfrost_job.c +0 -0
         spin_lock_irqsave(&pfdev->js->job_lock, flags);
       for (i = 0; i < NUM_JOB_SLOTS; i++) {
@@ -398,7 +404,6 @@ static void panfrost_job_timedout(struct
 drm_sched_job *sched_job)
       }
       spin_unlock_irqrestore(&pfdev->js->job_lock, flags);

  -    /* panfrost_core_dump(pfdev); */

This should be cleaned in another patch !

Quoted 4 lines

Seems to me that this should be some kind of TODO, see etnaviv_core_dump() for the kind of things we could be doing.

Maybe we can delete this line and mention this in the TODO file?

Fair enough - I'll split this into a separate patch and add an entry to the TODO file. kbase has a mechanism to "dump on job fault" [1],[2] so we could do something similar.

Steve

[1]
https://gitlab.freedesktop.org/panfrost/mali_kbase/blob/master/driver/product/kernel/drivers/gpu/arm/midgard/backend/gpu/mali_kbase_debug_job_fault_backend.c

[2]
https://gitlab.freedesktop.org/panfrost/mali_kbase/blob/master/driver/product/kernel/drivers/gpu/arm/midgard/mali_kbase_debug_job_fault.c

Quoted 20 lines

Cheers,

Tomeu

Quoted 4 lines
Quoted 3 lines

        panfrost_devfreq_record_transition(pfdev, js); panfrost_device_reset(pfdev);

Quoted 8 lines

Thanks,
Testing it right now with the last change removed (doesn't apply on v5.3 with it),
results in a few hours... or minutes !

Neil

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Tested-by on v1 1/1
#4 lore raw
In reply to Steven Price
Quoted 57 lines · Steven Price

Panfrost uses multiple schedulers (one for each slot, so 2 in reality), and on a timeout has to stop all the schedulers to safely perform a reset. However more than one scheduler can trigger a timeout at the same time. This race condition results in jobs being freed while they are still in use.

When stopping other slots use cancel_delayed_work_sync() to ensure that any timeout started for that slot has completed. Also use mutex_trylock() to obtain reset_lock. This means that only one thread attempts the reset, the other threads will simply complete without doing anything (the first thread will wait for this in the call to cancel_delayed_work_sync()).

While we're here and since the function is already dependent on sched_job not being NULL, let's remove the unnecessary checks, along with a commented out call to panfrost_core_dump() which has never existed in mainline.

---
This is a tidied up version of the patch orginally posted here:
http://lkml.kernel.org/r/26ae2a4d-8df1-e8db-3060-41638ed63e2a%40arm.com

 drivers/gpu/drm/panfrost/panfrost_job.c | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)
drivers/gpu/drm/panfrost/panfrost_job.c +11 -5
index a58551668d9a..dcc9a7603685 100644
@@ -381,13 +381,19 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job)
 		job_read(pfdev, JS_TAIL_LO(js)),
 		sched_job);
 
-	mutex_lock(&pfdev->reset_lock);
+	if (!mutex_trylock(&pfdev->reset_lock))
+		return;
 
-	for (i = 0; i < NUM_JOB_SLOTS; i++)
-		drm_sched_stop(&pfdev->js->queue[i].sched, sched_job);
+	for (i = 0; i < NUM_JOB_SLOTS; i++) {
+		struct drm_gpu_scheduler *sched = &pfdev->js->queue[i].sched;
+
+		drm_sched_stop(sched, sched_job);
+		if (js != i)
+			/* Ensure any timeouts on other slots have finished */
+			cancel_delayed_work_sync(&sched->work_tdr);
+	}
 
-	if (sched_job)
-		drm_sched_increase_karma(sched_job);
+	drm_sched_increase_karma(sched_job);
 
 	spin_lock_irqsave(&pfdev->js->job_lock, flags);
 	for (i = 0; i < NUM_JOB_SLOTS; i++) {

drivers/gpu/drm/panfrost/panfrost_job.c static void panfrost_job_timedout(struct drm_sched_job *sched_job) · line 409

Quoted 7 lines
drivers/gpu/drm/panfrost/panfrost_job.c +0 -1
@@ -398,7 +404,6 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job)
 	}
 	spin_unlock_irqrestore(&pfdev->js->job_lock, flags);
 
-	/* panfrost_core_dump(pfdev); */
 
 	panfrost_devfreq_record_transition(pfdev, js);
 	panfrost_device_reset(pfdev);

It ran successfully 10 dEQP tests without crashing the Amlogic S912 with Mali T820: