drm/panfrost: Handle resetting on timeout better
Steven Price
maintainerPATCH 1/1 drm/panfrost: Handle resetting on timeout better
Panfrost uses multiple schedulers (one for each slot, so 2 in reality), and on a timeout has to stop all the schedulers to safely perform a reset. However more than one scheduler can trigger a timeout at the same time. This race condition results in jobs being freed while they are still in use.
When stopping other slots use cancel_delayed_work_sync() to ensure that any timeout started for that slot has completed. Also use mutex_trylock() to obtain reset_lock. This means that only one thread attempts the reset, the other threads will simply complete without doing anything (the first thread will wait for this in the call to cancel_delayed_work_sync()).
While we're here and since the function is already dependent on sched_job not being NULL, let's remove the unnecessary checks, along with a commented out call to panfrost_core_dump() which has never existed in mainline.
---
This is a tidied up version of the patch orginally posted here:
http://lkml.kernel.org/r/26ae2a4d-8df1-e8db-3060-41638ed63e2a%40arm.com
drivers/gpu/drm/panfrost/panfrost_job.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)
drivers/gpu/drm/panfrost/panfrost_job.c +11 -6
@@ -381,13 +381,19 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job)job_read (pfdev,JS_TAIL_LO (js)), sched_job); -mutex_lock (&pfdev->reset_lock); +if (!mutex_trylock (&pfdev->reset_lock)) +return ; -for (i =0 ; i <NUM_JOB_SLOTS ; i++) -drm_sched_stop (&pfdev->js->queue[i].sched, sched_job); +for (i =0 ; i <NUM_JOB_SLOTS ; i++) { +struct drm_gpu_scheduler *sched = &pfdev->js->queue[i].sched; + +drm_sched_stop (sched, sched_job); +if (js != i) +/* Ensure any timeouts on other slots have finished */ +cancel_delayed_work_sync (&sched->work_tdr); + } -if (sched_job) -drm_sched_increase_karma (sched_job); +drm_sched_increase_karma (sched_job);spin_lock_irqsave (&pfdev->js->job_lock, flags);for (i =0 ; i <NUM_JOB_SLOTS ; i++) {
@@ -398,7 +404,6 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job) }spin_unlock_irqrestore (&pfdev->js->job_lock, flags); -/* panfrost_core_dump(pfdev); */ panfrost_devfreq_record_transition (pfdev, js);panfrost_device_reset (pfdev);
Signature · 6 lines
2.20.1 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Hi Steven,
Quoted 13 lines · Steven Price
Panfrost uses multiple schedulers (one for each slot, so 2 in reality), and on a timeout has to stop all the schedulers to safely perform a reset. However more than one scheduler can trigger a timeout at the same time. This race condition results in jobs being freed while they are still in use.
When stopping other slots use
cancel_delayed_work_sync()to ensure that any timeout started for that slot has completed. Also usemutex_trylock()to obtainreset_lock. This means that only one thread attempts the reset, the other threads will simply complete without doing anything (the first thread will wait for this in the call tocancel_delayed_work_sync()).
Quoted 5 lines
While we're here and since the function is already dependent on
sched_jobnot being NULL, let's remove the unnecessary checks, along with a commented out call topanfrost_core_dump()which has never existed in mainline.
A Fixes: tags would be welcome here so it would be backported to v5.3
Quoted 24 lines
Signed-off-by Steven Price <[email protected]>---
This is a tidied up version of the patch orginally posted here:
http://lkml.kernel.org/r/26ae2a4d-8df1-e8db-3060-41638ed63e2a%40arm.comdrivers/gpu/drm/panfrost/panfrost_job.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)drivers/gpu/drm/panfrost/panfrost_job.c +4
-3@@ -381,13 +381,19 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job)job_read (pfdev,JS_TAIL_LO (js)), sched_job); -mutex_lock (&pfdev->reset_lock); +if (!mutex_trylock (&pfdev->reset_lock)) +return ; -for (i =0 ; i <NUM_JOB_SLOTS ; i++) -drm_sched_stop (&pfdev->js->queue[i].sched, sched_job); +for (i =0 ; i <NUM_JOB_SLOTS ; i++) { +struct drm_gpu_scheduler *sched = &pfdev->js->queue[i].sched;
drivers/gpu/drm/panfrost/panfrost_job.c static void panfrost_job_timedout(struct drm_sched_job *sched_job) · line 396
Quoted 10 lines
drivers/gpu/drm/panfrost/panfrost_job.c +7
-2+ +drm_sched_stop (sched, sched_job); +if (js != i) +/* Ensure any timeouts on other slots have finished */ +cancel_delayed_work_sync (&sched->work_tdr); + } -if (sched_job) -drm_sched_increase_karma (sched_job); +drm_sched_increase_karma (sched_job);
Indeed looks cleaner.
drivers/gpu/drm/panfrost/panfrost_job.c static void panfrost_job_timedout(struct drm_sched_job *sched_job) · line 401
Quoted 8 lines
drivers/gpu/drm/panfrost/panfrost_job.c +0
-1spin_lock_irqsave (&pfdev->js->job_lock, flags);for (i =0 ; i <NUM_JOB_SLOTS ; i++) {@@ -398,7 +404,6 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job) }spin_unlock_irqrestore (&pfdev->js->job_lock, flags); -/* panfrost_core_dump(pfdev); */
This should be cleaned in another patch !
drivers/gpu/drm/panfrost/panfrost_job.c static void panfrost_job_timedout(struct drm_sched_job *sched_job)
Quoted 4 lines
drivers/gpu/drm/panfrost/panfrost_job.c +0
-0panfrost_devfreq_record_transition (pfdev, js);panfrost_device_reset (pfdev);
Thanks,
Testing it right now with the last change removed (doesn't apply on v5.3 with it), results in a few hours... or minutes !
Neil
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 68 lines · Neil Armstrong
Hi Steven,
Quoted 13 lines · Steven Price
Panfrost uses multiple schedulers (one for each slot, so 2 in reality), and on a timeout has to stop all the schedulers to safely perform a reset. However more than one scheduler can trigger a timeout at the same time. This race condition results in jobs being freed while they are still in use.
When stopping other slots use
cancel_delayed_work_sync()to ensure that any timeout started for that slot has completed. Also usemutex_trylock()to obtainreset_lock. This means that only one thread attempts the reset, the other threads will simply complete without doing anything (the first thread will wait for this in the call tocancel_delayed_work_sync()).Quoted 5 lines
While we're here and since the function is already dependent on
sched_jobnot being NULL, let's remove the unnecessary checks, along with a commented out call topanfrost_core_dump()which has never existed in mainline.A Fixes: tags would be welcome here so it would be backported to v5.3
Quoted 24 lines
Signed-off-by Steven Price <[email protected]>---
This is a tidied up version of the patch orginally posted here:
http://lkml.kernel.org/r/26ae2a4d-8df1-e8db-3060-41638ed63e2a%40arm.comdrivers/gpu/drm/panfrost/panfrost_job.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)drivers/gpu/drm/panfrost/panfrost_job.c +4
-3@@ -381,13 +381,19 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job)job_read (pfdev,JS_TAIL_LO (js)), sched_job); -mutex_lock (&pfdev->reset_lock); +if (!mutex_trylock (&pfdev->reset_lock)) +return ; -for (i =0 ; i <NUM_JOB_SLOTS ; i++) -drm_sched_stop (&pfdev->js->queue[i].sched, sched_job); +for (i =0 ; i <NUM_JOB_SLOTS ; i++) { +struct drm_gpu_scheduler *sched = &pfdev->js->queue[i].sched;drivers/gpu/drm/panfrost/panfrost_job.c static void panfrost_job_timedout(struct drm_sched_job *sched_job) · line 396
Quoted 10 lines
drivers/gpu/drm/panfrost/panfrost_job.c +7
-2+ +drm_sched_stop (sched, sched_job); +if (js != i) +/* Ensure any timeouts on other slots have finished */ +cancel_delayed_work_sync (&sched->work_tdr); + } -if (sched_job) -drm_sched_increase_karma (sched_job); +drm_sched_increase_karma (sched_job);Indeed looks cleaner.
Quoted 7 lines
drivers/gpu/drm/panfrost/panfrost_job.c +0
-0spin_lock_irqsave (&pfdev->js->job_lock, flags);for (i =0 ; i <NUM_JOB_SLOTS ; i++) {@@ -398,7 +404,6 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job) }spin_unlock_irqrestore (&pfdev->js->job_lock, flags);
Quoted 3 lines
drivers/gpu/drm/panfrost/panfrost_job.c static void panfrost_job_timedout(struct drm_sched_job *sched_job) · line 401
Quoted 1 line
drivers/gpu/drm/panfrost/panfrost_job.c +0
-1-/* panfrost_core_dump(pfdev); */ This should be cleaned in another patch !
Seems to me that this should be some kind of TODO, see etnaviv_core_dump() for the kind of things we could be doing.
Maybe we can delete this line and mention this in the TODO file?
Cheers,
Tomeu
Quoted 12 lines
drivers/gpu/drm/panfrost/panfrost_job.c static void panfrost_job_timedout(struct drm_sched_job *sched_job)
Quoted 4 lines
drivers/gpu/drm/panfrost/panfrost_job.c +0
-0panfrost_devfreq_record_transition (pfdev, js);panfrost_device_reset (pfdev);Thanks,
Testing it right now with the last change removed (doesn't apply on v5.3 with it), results in a few hours... or minutes !Neil
Steven Price
maintainerQuoted 71 lines · Tomeu Vizoso
Quoted 60 lines · Neil Armstrong
Hi Steven,
Quoted 13 lines · Steven Price
Panfrost uses multiple schedulers (one for each slot, so 2 in reality), and on a timeout has to stop all the schedulers to safely perform a reset. However more than one scheduler can trigger a timeout at the same time. This race condition results in jobs being freed while they are still in use.
When stopping other slots use
cancel_delayed_work_sync()to ensure that any timeout started for that slot has completed. Also usemutex_trylock()to obtainreset_lock. This means that only one thread attempts the reset, the other threads will simply complete without doing anything (the first thread will wait for this in the call tocancel_delayed_work_sync()).Quoted 5 lines
While we're here and since the function is already dependent on
sched_jobnot being NULL, let's remove the unnecessary checks, along with a commented out call topanfrost_core_dump()which has never existed in mainline.A Fixes: tags would be welcome here so it would be backported to v5.3
Quoted 23 lines
Signed-off-by Steven Price <[email protected]>---
This is a tidied up version of the patch orginally posted here:
http://lkml.kernel.org/r/26ae2a4d-8df1-e8db-3060-41638ed63e2a%40arm.com
drivers/gpu/drm/panfrost/panfrost_job.c| 17 +++++++++++------1 file changed, 11 insertions(+), 6 deletions(-)drivers/gpu/drm/panfrost/panfrost_job.c +3
-1@@ -381,13 +381,19 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job) job_read(pfdev, JS_TAIL_LO(js)), sched_job); - mutex_lock(&pfdev->reset_lock); + if (!mutex_trylock(&pfdev->reset_lock)) + return; - for (i = 0; i < NUM_JOB_SLOTS; i++) - drm_sched_stop(&pfdev->js->queue[i].sched, sched_job); + for (i = 0; i < NUM_JOB_SLOTS; i++) {drivers/gpu/drm/panfrost/panfrost_job.c static void panfrost_job_timedout(struct · line 397
Quoted 10 lines
drivers/gpu/drm/panfrost/panfrost_job.c +8
-1+ struct drm_gpu_scheduler *sched = &pfdev->js->queue[i].sched; + + drm_sched_stop(sched, sched_job); + if (js != i) + /* Ensure any timeouts on other slots have finished */ + cancel_delayed_work_sync(&sched->work_tdr); + } - if (sched_job) - drm_sched_increase_karma(sched_job); + drm_sched_increase_karma(sched_job);Indeed looks cleaner.
Quoted 9 lines
Quoted 7 lines
drivers/gpu/drm/panfrost/panfrost_job.c +0
-0spin_lock_irqsave(&pfdev->js->job_lock, flags); for (i = 0; i < NUM_JOB_SLOTS; i++) {@@ -398,7 +404,6 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job) } spin_unlock_irqrestore(&pfdev->js->job_lock, flags);- /*
panfrost_core_dump(pfdev); */This should be cleaned in another patch !
Quoted 4 lines
Seems to me that this should be some kind of TODO, see
etnaviv_core_dump()for the kind of things we could be doing.Maybe we can delete this line and mention this in the TODO file?
Fair enough - I'll split this into a separate patch and add an entry to the TODO file. kbase has a mechanism to "dump on job fault" [1],[2] so we could do something similar.
Steve
Quoted 20 lines
Cheers,
Tomeu
Quoted 4 lines
Quoted 3 lines
panfrost_devfreq_record_transition(pfdev, js);panfrost_device_reset(pfdev);Quoted 8 lines
Thanks,
Testing it right now with the last change removed (doesn't apply on v5.3 with it),
results in a few hours... or minutes !Neil
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 57 lines · Steven Price
Panfrost uses multiple schedulers (one for each slot, so 2 in reality), and on a timeout has to stop all the schedulers to safely perform a reset. However more than one scheduler can trigger a timeout at the same time. This race condition results in jobs being freed while they are still in use.
When stopping other slots use
cancel_delayed_work_sync()to ensure that any timeout started for that slot has completed. Also usemutex_trylock()to obtainreset_lock. This means that only one thread attempts the reset, the other threads will simply complete without doing anything (the first thread will wait for this in the call tocancel_delayed_work_sync()).While we're here and since the function is already dependent on
sched_jobnot being NULL, let's remove the unnecessary checks, along with a commented out call topanfrost_core_dump()which has never existed in mainline.Signed-off-by Steven Price <[email protected]>---
This is a tidied up version of the patch orginally posted here:
http://lkml.kernel.org/r/26ae2a4d-8df1-e8db-3060-41638ed63e2a%40arm.comdrivers/gpu/drm/panfrost/panfrost_job.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)drivers/gpu/drm/panfrost/panfrost_job.c +11
-5@@ -381,13 +381,19 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job)job_read (pfdev,JS_TAIL_LO (js)), sched_job); -mutex_lock (&pfdev->reset_lock); +if (!mutex_trylock (&pfdev->reset_lock)) +return ; -for (i =0 ; i <NUM_JOB_SLOTS ; i++) -drm_sched_stop (&pfdev->js->queue[i].sched, sched_job); +for (i =0 ; i <NUM_JOB_SLOTS ; i++) { +struct drm_gpu_scheduler *sched = &pfdev->js->queue[i].sched; + +drm_sched_stop (sched, sched_job); +if (js != i) +/* Ensure any timeouts on other slots have finished */ +cancel_delayed_work_sync (&sched->work_tdr); + } -if (sched_job) -drm_sched_increase_karma (sched_job); +drm_sched_increase_karma (sched_job);spin_lock_irqsave (&pfdev->js->job_lock, flags);for (i =0 ; i <NUM_JOB_SLOTS ; i++) {
drivers/gpu/drm/panfrost/panfrost_job.c static void panfrost_job_timedout(struct drm_sched_job *sched_job) · line 409
Quoted 7 lines
drivers/gpu/drm/panfrost/panfrost_job.c +0
-1@@ -398,7 +404,6 @@ static void panfrost_job_timedout(struct drm_sched_job *sched_job) }spin_unlock_irqrestore (&pfdev->js->job_lock, flags); -/* panfrost_core_dump(pfdev); */ panfrost_devfreq_record_transition (pfdev, js);panfrost_device_reset (pfdev);
It ran successfully 10 dEQP tests without crashing the Amlogic S912 with Mali T820: