drm/sched: fix the bug of time out calculation
Subsystems (3)
PATCH 1/1 drm/sched: fix the bug of time out calculation
the original logic is wrong that the timeout will not be retriggerd after the previous job siganled, and that lead to the scenario that all jobs in the same scheduler shares the same timeout timer from the very begining job in this scheduler which is wrong.
we should modify the timer everytime a previous job signaled.
--- drivers/gpu/drm/scheduler/sched_main.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
drivers/gpu/drm/scheduler/sched_main.c +12 -0
@@ -235,6 +235,13 @@ static void drm_sched_start_timeout(struct drm_gpu_scheduler *sched)schedule_delayed_work (&sched->work_tdr, sched->timeout); } +static void drm_sched_restart_timeout (struct drm_gpu_scheduler *sched) +{ +if (sched->timeout !=MAX_SCHEDULE_TIMEOUT && + !list_empty (&sched->pending_list)) +mod_delayed_work (system_wq, &sched->work_tdr, sched->timeout); +} +/** * drm_sched_fault - immediately start timeout handler *
@@ -693,6 +700,11 @@ drm_sched_get_cleanup_job(struct drm_gpu_scheduler *sched)if (job &&dma_fence_is_signaled (&job->s_fence->finished)) {/* remove job from pending_list */ list_del_init (&job->list); + +/* once the job deleted from pending list we should restart +* the timeout calculation for the next job. +*/ +drm_sched_restart_timeout (sched);/* make the scheduled timestamp more accurate */ next =list_first_entry_or_null (&sched->pending_list,typeof (*next), list);
Signature · 1 line
2.7.4
Quoted 31 lines · Monk Liu
the original logic is wrong that the timeout will not be retriggerd after the previous job siganled, and that lead to the scenario that all jobs in the same scheduler shares the same timeout timer from the very begining job in this scheduler which is wrong.
we should modify the timer everytime a previous job signaled.
Signed-off-by Monk Liu <[email protected]>--- drivers/gpu/drm/scheduler/sched_main.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)drivers/gpu/drm/scheduler/sched_main.c +7
-0@@ -235,6 +235,13 @@ static void drm_sched_start_timeout(struct drm_gpu_scheduler *sched)schedule_delayed_work (&sched->work_tdr, sched->timeout); } +static void drm_sched_restart_timeout (struct drm_gpu_scheduler *sched) +{ +if (sched->timeout !=MAX_SCHEDULE_TIMEOUT && + !list_empty (&sched->pending_list)) +mod_delayed_work (system_wq, &sched->work_tdr, sched->timeout); +} +/** * drm_sched_fault - immediately start timeout handler *
drivers/gpu/drm/scheduler/sched_main.c drm_sched_get_cleanup_job(struct drm_gpu_scheduler *sched) · line 707
Quoted 8 lines
drivers/gpu/drm/scheduler/sched_main.c +5
-0@@ -693,6 +700,11 @@ drm_sched_get_cleanup_job(struct drm_gpu_scheduler *sched)if (job &&dma_fence_is_signaled (&job->s_fence->finished)) {/* remove job from pending_list */ list_del_init (&job->list); + +/* once the job deleted from pending list we should restart +* the timeout calculation for the next job. +*/ +drm_sched_restart_timeout (sched);
I think this should work, but 2 points -
1st you should probably remove this now https://elixir.bootlin.com/linux/v5.14-rc1/source/drivers/gpu/drm/scheduler/sched_main.c#L797
2nd - if you have two adjacent jobs started very closely you effectively letting the second job to be twice longer hang without TDR because you reset TDR timer for it when it's almost expired. If we could have TTL (time to live counter) for each job and then do mod_delayed_work to the TTL of the following job instead of just full timer reset then this would be more precise. But this is more of recommendation for improvement.
Andrey
Quoted 3 lines
drivers/gpu/drm/scheduler/sched_main.c +0
-0/* make the scheduled timestamp more accurate */ next =list_first_entry_or_null (&sched->pending_list,typeof (*next), list);
Quoted 14 lines · Andrey Grodzovsky
Quoted 12 lines · Monk Liu
the original logic is wrong that the timeout will not be retriggerd after the previous job siganled, and that lead to the scenario that all jobs in the same scheduler shares the same timeout timer from the very begining job in this scheduler which is wrong.
we should modify the timer everytime a previous job signaled.
Signed-off-by Monk Liu <[email protected]>---
drivers/gpu/drm/scheduler/sched_main.c| 12 ++++++++++++1 file changed, 12 insertions(+)
Quoted 13 lines
drivers/gpu/drm/scheduler/sched_main.c static void drm_sched_start_timeout(struct · line 242
Quoted 13 lines
drivers/gpu/drm/scheduler/sched_main.c +4
-0@@ -235,6 +235,13 @@ static void drm_sched_start_timeout(struct drm_gpu_scheduler *sched) schedule_delayed_work(&sched->work_tdr, sched->timeout); } +static void drm_sched_restart_timeout(struct drm_gpu_scheduler *sched) +{ + if (sched->timeout != MAX_SCHEDULE_TIMEOUT && + !list_empty(&sched->pending_list)) + mod_delayed_work(system_wq, &sched->work_tdr, sched->timeout);
3d point - if list empty you need to cancel the timer, let the new job coming after that restart it.
Andrey
Quoted 31 lines
Quoted 6 lines
drivers/gpu/drm/scheduler/sched_main.c +2
-0+} + /*** drm_sched_fault - immediately start timeout handler *
drivers/gpu/drm/scheduler/sched_main.c drm_sched_get_cleanup_job(struct · line 708
Quoted 9 lines
drivers/gpu/drm/scheduler/sched_main.c +5
-0@@ -693,6 +700,11 @@ drm_sched_get_cleanup_job(struct drm_gpu_scheduler *sched) if (job && dma_fence_is_signaled(&job->s_fence->finished)) { /* remove job from pending_list */ list_del_init(&job->list); + + /* once the job deleted from pending list we should restart + * the timeout calculation for the next job. + */ + drm_sched_restart_timeout(sched);I think this should work, but 2 points -
1st you should probably remove this now https://elixir.bootlin.com/linux/v5.14-rc1/source/drivers/gpu/drm/scheduler/sched_main.c#L797
2nd - if you have two adjacent jobs started very closely you effectively letting the second job to be twice longer hang without TDR because you reset TDR timer for it when it's almost expired. If we could have TTL (time to live counter) for each job and then do
mod_delayed_workto the TTL of the following job instead of just full timer reset then this would be more precise. But this is more of recommendation for improvement.Andrey
Quoted 3 lines
drivers/gpu/drm/scheduler/sched_main.c +0
-0/* make the scheduled timestamp more accurate */ next = list_first_entry_or_null(&sched->pending_list,
typeof(*next), list);
[AMD Official Use Only]
Quoted 1 line
Quoted 1 line
3d point - if list empty you need to cancel the timer, let the new job coming after that restart it.
Quoted 1 line
Quoted 1 line
2nd - if you have two adjacent jobs started very closely you effectively letting the second job to be twice longer hang without TDR because you reset TDR timer for it when it's almost expired. If we could have TTL (time to live counter) for each job and then do mod_delayed_work to the TTL of the following job instead of just full timer reset then this would be more precise. But this is more of recommendation for improvement.
Quoted 1 line
Quoted 1 line
1st you should probably remove this now
https://elixir.bootlin.com/linux/v5.14-rc1/source/drivers/gpu/drm/scheduler/sched_main.c#L797
I checked and thought all above points from you, end up with a V2 patch, please take a look again.
Thanks
------------------------------------------
Monk Liu | Cloud-GPU Core team
------------------------------------------
-----Original Message-----
From: Grodzovsky, Andrey <[email protected]>
Sent: Tuesday, August 24, 2021 11:02 PM
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH] drm/sched: fix the bug of time out calculation
Quoted 14 lines · Andrey Grodzovsky
Quoted 12 lines · Monk Liu
the original logic is wrong that the timeout will not be retriggerd after the previous job siganled, and that lead to the scenario that all jobs in the same scheduler shares the same timeout timer from the very begining job in this scheduler which is wrong.
we should modify the timer everytime a previous job signaled.
Signed-off-by Monk Liu <[email protected]>---
drivers/gpu/drm/scheduler/sched_main.c| 12 ++++++++++++1 file changed, 12 insertions(+)
Quoted 16 lines
Quoted 11 lines
drivers/gpu/drm/scheduler/sched_main.c +0
-0@@ -235,6 +235,13 @@ static void drm_sched_start_timeout(struct drm_gpu_scheduler *sched) schedule_delayed_work(&sched->work_tdr, sched->timeout); }+static void
drm_sched_restart_timeout(struct drm_gpu_scheduler *sched)drivers/gpu/drm/scheduler/sched_main.c static void drm_sched_start_timeout(struct drm_gpu_scheduler *sched)
Quoted 5 lines
drivers/gpu/drm/scheduler/sched_main.c +5
-0+{ + if (sched->timeout != MAX_SCHEDULE_TIMEOUT && + !list_empty(&sched->pending_list)) + mod_delayed_work(system_wq, &sched->work_tdr, +sched->timeout);
3d point - if list empty you need to cancel the timer, let the new job coming after that restart it.
Andrey
Quoted 37 lines
Quoted 6 lines
drivers/gpu/drm/scheduler/sched_main.c +2
-0+} +/**
*drm_sched_fault- immediately start timeout handler *drivers/gpu/drm/scheduler/sched_main.c drm_sched_get_cleanup_job(struct · line 708
Quoted 9 lines
drivers/gpu/drm/scheduler/sched_main.c +5
-0@@ -693,6 +700,11 @@ drm_sched_get_cleanup_job(struct drm_gpu_scheduler *sched) if (job && dma_fence_is_signaled(&job->s_fence->finished)) { /* remove job from pending_list */ list_del_init(&job->list); + + /* once the job deleted from pending list we should restart + * the timeout calculation for the next job. + */ + drm_sched_restart_timeout(sched);I think this should work, but 2 points -
1st you should probably remove this now
https://elixir.bootlin.com/linux/v5.14-rc1/source/drivers/gpu/drm/sche
duler/sched_main.c#L7972nd - if you have two adjacent jobs started very closely you effectively letting the second job to be twice longer hang without TDR because you reset TDR timer for it when it's almost expired. If we could have TTL (time to live counter) for each job and then do
mod_delayed_workto the TTL of the following job instead of just full timer reset then this would be more precise. But this is more of recommendation for improvement.Andrey
Quoted 3 lines
drivers/gpu/drm/scheduler/sched_main.c +0
-0/* make the scheduled timestamp more accurate */ next = list_first_entry_or_null(&sched->pending_list,
typeof(*next), list);