Skip to contentSkip to conversation

drm/sched: fix the bug of time out calculation

PATCH In review Monk Liu 2 participants 4 messages Aug 24, 2021 – Aug 25, 2021

Subsystems (3)
lore
Share & tools
lore

Monk Liu

author
#0 lore raw

PATCH 1/1 drm/sched: fix the bug of time out calculation

the original logic is wrong that the timeout will not be retriggerd after the previous job siganled, and that lead to the scenario that all jobs in the same scheduler shares the same timeout timer from the very begining job in this scheduler which is wrong.

we should modify the timer everytime a previous job signaled.

---
 drivers/gpu/drm/scheduler/sched_main.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
drivers/gpu/drm/scheduler/sched_main.c +12 -0
index a2a9536..fb27025 100644
@@ -235,6 +235,13 @@ static void drm_sched_start_timeout(struct drm_gpu_scheduler *sched)
 		schedule_delayed_work(&sched->work_tdr, sched->timeout);
 }
 
+static void drm_sched_restart_timeout(struct drm_gpu_scheduler *sched)
+{
+	if (sched->timeout != MAX_SCHEDULE_TIMEOUT &&
+	    !list_empty(&sched->pending_list))
+		mod_delayed_work(system_wq, &sched->work_tdr, sched->timeout);
+}
+
 /**
  * drm_sched_fault - immediately start timeout handler
  *
@@ -693,6 +700,11 @@ drm_sched_get_cleanup_job(struct drm_gpu_scheduler *sched)
 	if (job && dma_fence_is_signaled(&job->s_fence->finished)) {
 		/* remove job from pending_list */
 		list_del_init(&job->list);
+
+		/* once the job deleted from pending list we should restart
+		 * the timeout calculation for the next job.
+		 */
+		drm_sched_restart_timeout(sched);
 		/* make the scheduled timestamp more accurate */
 		next = list_first_entry_or_null(&sched->pending_list,
 						typeof(*next), list);
Signature · 1 line
2.7.4
#1 lore raw
Quoted 31 lines · Monk Liu

the original logic is wrong that the timeout will not be retriggerd after the previous job siganled, and that lead to the scenario that all jobs in the same scheduler shares the same timeout timer from the very begining job in this scheduler which is wrong.

we should modify the timer everytime a previous job signaled.

---
  drivers/gpu/drm/scheduler/sched_main.c | 12 ++++++++++++
  1 file changed, 12 insertions(+)
drivers/gpu/drm/scheduler/sched_main.c +7 -0
index a2a9536..fb27025 100644
@@ -235,6 +235,13 @@ static void drm_sched_start_timeout(struct drm_gpu_scheduler *sched)
  		schedule_delayed_work(&sched->work_tdr, sched->timeout);
  }
 
+static void drm_sched_restart_timeout(struct drm_gpu_scheduler *sched)
+{
+	if (sched->timeout != MAX_SCHEDULE_TIMEOUT &&
+	    !list_empty(&sched->pending_list))
+		mod_delayed_work(system_wq, &sched->work_tdr, sched->timeout);
+}
+
  /**
   * drm_sched_fault - immediately start timeout handler
   *

drivers/gpu/drm/scheduler/sched_main.c drm_sched_get_cleanup_job(struct drm_gpu_scheduler *sched) · line 707

Quoted 8 lines
drivers/gpu/drm/scheduler/sched_main.c +5 -0
@@ -693,6 +700,11 @@ drm_sched_get_cleanup_job(struct drm_gpu_scheduler *sched)
  	if (job && dma_fence_is_signaled(&job->s_fence->finished)) {
  		/* remove job from pending_list */
  		list_del_init(&job->list);
+
+		/* once the job deleted from pending list we should restart
+		 * the timeout calculation for the next job.
+		 */
+		drm_sched_restart_timeout(sched);

I think this should work, but 2 points -

1st you should probably remove this now https://elixir.bootlin.com/linux/v5.14-rc1/source/drivers/gpu/drm/scheduler/sched_main.c#L797

2nd - if you have two adjacent jobs started very closely you effectively letting the second job to be twice longer hang without TDR because you reset TDR timer for it when it's almost expired. If we could have TTL (time to live counter) for each job and then do mod_delayed_work to the TTL of the following job instead of just full timer reset then this would be more precise. But this is more of recommendation for improvement.

Andrey

Quoted 3 lines
drivers/gpu/drm/scheduler/sched_main.c +0 -0
  		/* make the scheduled timestamp more accurate */
  		next = list_first_entry_or_null(&sched->pending_list,
  						typeof(*next), list);
#2 lore raw
Quoted 14 lines · Andrey Grodzovsky
Quoted 12 lines · Monk Liu

the original logic is wrong that the timeout will not be retriggerd after the previous job siganled, and that lead to the scenario that all jobs in the same scheduler shares the same timeout timer from the very begining job in this scheduler which is wrong.

we should modify the timer everytime a previous job signaled.

---
  drivers/gpu/drm/scheduler/sched_main.c | 12 ++++++++++++

  1 file changed, 12 insertions(+)
Quoted 13 lines

drivers/gpu/drm/scheduler/sched_main.c static void drm_sched_start_timeout(struct · line 242

Quoted 13 lines
drivers/gpu/drm/scheduler/sched_main.c +4 -0
index a2a9536..fb27025 100644
@@ -235,6 +235,13 @@ static void drm_sched_start_timeout(struct 
 drm_gpu_scheduler *sched)
           schedule_delayed_work(&sched->work_tdr, sched->timeout);
   }
   +static void drm_sched_restart_timeout(struct drm_gpu_scheduler *sched)
+{
+    if (sched->timeout != MAX_SCHEDULE_TIMEOUT &&
+        !list_empty(&sched->pending_list))
+        mod_delayed_work(system_wq, &sched->work_tdr, sched->timeout);

3d point - if list empty you need to cancel the timer, let the new job coming after that restart it.

Andrey

Quoted 31 lines
Quoted 6 lines
drivers/gpu/drm/scheduler/sched_main.c +2 -0
+}
+
   /**
    * drm_sched_fault - immediately start timeout handler

   *

drivers/gpu/drm/scheduler/sched_main.c drm_sched_get_cleanup_job(struct · line 708

Quoted 9 lines
drivers/gpu/drm/scheduler/sched_main.c +5 -0
@@ -693,6 +700,11 @@ drm_sched_get_cleanup_job(struct 
 drm_gpu_scheduler *sched)
       if (job && dma_fence_is_signaled(&job->s_fence->finished)) {
           /* remove job from pending_list */
           list_del_init(&job->list);
+
+        /* once the job deleted from pending list we should restart
+         * the timeout calculation for the next job.
+         */
+        drm_sched_restart_timeout(sched);

I think this should work, but 2 points -

1st you should probably remove this now https://elixir.bootlin.com/linux/v5.14-rc1/source/drivers/gpu/drm/scheduler/sched_main.c#L797

2nd - if you have two adjacent jobs started very closely you effectively letting the second job to be twice longer hang without TDR because you reset TDR timer for it when it's almost expired. If we could have TTL (time to live counter) for each job and then do mod_delayed_work to the TTL of the following job instead of just full timer reset then this would be more precise. But this is more of recommendation for improvement.

Andrey

Quoted 3 lines
drivers/gpu/drm/scheduler/sched_main.c +0 -0
           /* make the scheduled timestamp more accurate */
           next = list_first_entry_or_null(&sched->pending_list,

                          typeof(*next), list);

author
#3 lore raw

[AMD Official Use Only]

Quoted 1 line
Quoted 1 line

3d point - if list empty you need to cancel the timer, let the new job coming after that restart it.

Quoted 1 line
Quoted 1 line

2nd - if you have two adjacent jobs started very closely you effectively letting the second job to be twice longer hang without TDR because you reset TDR timer for it when it's almost expired. If we could have TTL (time to live counter) for each job and then do mod_delayed_work to the TTL of the following job instead of just full timer reset then this would be more precise. But this is more of recommendation for improvement.

Quoted 1 line
Quoted 1 line

1st you should probably remove this now
https://elixir.bootlin.com/linux/v5.14-rc1/source/drivers/gpu/drm/scheduler/sched_main.c#L797

I checked and thought all above points from you, end up with a V2 patch, please take a look again.

Thanks

------------------------------------------
Monk Liu | Cloud-GPU Core team
------------------------------------------

-----Original Message-----
From: Grodzovsky, Andrey <[email protected]>
Sent: Tuesday, August 24, 2021 11:02 PM
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH] drm/sched: fix the bug of time out calculation

Quoted 14 lines · Andrey Grodzovsky
Quoted 12 lines · Monk Liu

the original logic is wrong that the timeout will not be retriggerd after the previous job siganled, and that lead to the scenario that all jobs in the same scheduler shares the same timeout timer from the very begining job in this scheduler which is wrong.

we should modify the timer everytime a previous job signaled.

---
  drivers/gpu/drm/scheduler/sched_main.c | 12 ++++++++++++

  1 file changed, 12 insertions(+)
Quoted 16 lines
Quoted 11 lines
drivers/gpu/drm/scheduler/sched_main.c +0 -0
index a2a9536..fb27025 100644
@@ -235,6 +235,13 @@ static void drm_sched_start_timeout(struct 
 drm_gpu_scheduler *sched)
           schedule_delayed_work(&sched->work_tdr, sched->timeout);
   }

  +static void drm_sched_restart_timeout(struct drm_gpu_scheduler *sched)

drivers/gpu/drm/scheduler/sched_main.c static void drm_sched_start_timeout(struct drm_gpu_scheduler *sched)

Quoted 5 lines
drivers/gpu/drm/scheduler/sched_main.c +5 -0
+{
+    if (sched->timeout != MAX_SCHEDULE_TIMEOUT &&
+        !list_empty(&sched->pending_list))
+        mod_delayed_work(system_wq, &sched->work_tdr, 
+sched->timeout);

3d point - if list empty you need to cancel the timer, let the new job coming after that restart it.

Andrey

Quoted 37 lines
Quoted 6 lines
drivers/gpu/drm/scheduler/sched_main.c +2 -0
+}
+

  /**
   * drm_sched_fault - immediately start timeout handler *

drivers/gpu/drm/scheduler/sched_main.c drm_sched_get_cleanup_job(struct · line 708

Quoted 9 lines
drivers/gpu/drm/scheduler/sched_main.c +5 -0
@@ -693,6 +700,11 @@ drm_sched_get_cleanup_job(struct 
 drm_gpu_scheduler *sched)
       if (job && dma_fence_is_signaled(&job->s_fence->finished)) {
           /* remove job from pending_list */
           list_del_init(&job->list);
+
+        /* once the job deleted from pending list we should restart
+         * the timeout calculation for the next job.
+         */
+        drm_sched_restart_timeout(sched);

I think this should work, but 2 points -

1st you should probably remove this now
https://elixir.bootlin.com/linux/v5.14-rc1/source/drivers/gpu/drm/sche
duler/sched_main.c#L797

2nd - if you have two adjacent jobs started very closely you effectively letting the second job to be twice longer hang without TDR because you reset TDR timer for it when it's almost expired. If we could have TTL (time to live counter) for each job and then do mod_delayed_work to the TTL of the following job instead of just full timer reset then this would be more precise. But this is more of recommendation for improvement.

Andrey

Quoted 3 lines
drivers/gpu/drm/scheduler/sched_main.c +0 -0
           /* make the scheduled timestamp more accurate */
           next = list_first_entry_or_null(&sched->pending_list,

                          typeof(*next), list);