Skip to contentSkip to conversation

drm/amdgpu: extend the default timeout for kernel compute queues

PATCH In review Feifei Xu 4 participants 5 messages Apr 20, 2023 – Apr 21, 2023

Subsystems (2)
lore
Share & tools
lore

Feifei Xu

author
#0 lore raw

PATCH 1/1 drm/amdgpu: extend the default timeout for kernel compute queues

Extend to 120s. The default timeout value should also extend if compute shader execution time extended. Otherwise some stress test will trigger compute ring timeout in software.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -2
index e536886f6d42..1f98b4b0a549 100644
@@ -3475,7 +3475,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev)
 
 	/*
 	 * By default timeout for non compute jobs is 10000
-	 * and 60000 for compute jobs.
+	 * and 120000 for compute jobs.
 	 * In SR-IOV or passthrough mode, timeout for compute
 	 * jobs are 60000 by default.
 	 */
@@ -3485,7 +3485,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev)
 		adev->compute_timeout = amdgpu_sriov_is_pp_one_vf(adev) ?
 					msecs_to_jiffies(60000) : msecs_to_jiffies(10000);
 	else
-		adev->compute_timeout =  msecs_to_jiffies(60000);
+		adev->compute_timeout =  msecs_to_jiffies(120000);
 
 	if (strnlen(input, AMDGPU_MAX_TIMEOUT_PARAM_LENGTH)) {
 		while ((timeout_setting = strsep(&input, ",")) &&
Signature · 1 line
2.34.1
Reviewed-by on v1 1/1
#1 lore raw

[AMD Official Use Only - General]

Regards,
Hawking
-----Original Message-----
From: Feifei Xu <[email protected]>
Sent: Thursday, April 20, 2023 17:20
To: [email protected]
Cc: Xu, Feifei <[email protected]>; Zhang, Hawking <[email protected]>
Subject: [PATCH] drm/amdgpu: extend the default timeout for kernel compute queues

Extend to 120s. The default timeout value should also extend if compute shader execution time extended. Otherwise some stress test will trigger compute ring timeout in software.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -2
index e536886f6d42..1f98b4b0a549 100644
@@ -3475,7 +3475,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev)
 
        /*
         * By default timeout for non compute jobs is 10000
-        * and 60000 for compute jobs.
+        * and 120000 for compute jobs.
         * In SR-IOV or passthrough mode, timeout for compute
         * jobs are 60000 by default.
         */
@@ -3485,7 +3485,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev)
                adev->compute_timeout = amdgpu_sriov_is_pp_one_vf(adev) ?
                                        msecs_to_jiffies(60000) : msecs_to_jiffies(10000);
        else
-               adev->compute_timeout =  msecs_to_jiffies(60000);
+               adev->compute_timeout =  msecs_to_jiffies(120000);
 
        if (strnlen(input, AMDGPU_MAX_TIMEOUT_PARAM_LENGTH)) {
                while ((timeout_setting = strsep(&input, ",")) &&

--
2.34.1

#2 lore raw
In reply to Feifei Xu
Quoted 4 lines · Feifei Xu

Extend to 120s. The default timeout value should also extend if compute shader execution time extended. Otherwise some stress test will trigger compute ring timeout in software.

I think that's probably too long. 2 minutes is a long time to have a hung system. I think we should rework the tests or use ROCm for long running test cases.

Alex

Quoted 31 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -3
index e536886f6d42..1f98b4b0a549 100644
@@ -3475,7 +3475,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev)
 
        /*
         * By default timeout for non compute jobs is 10000
-        * and 60000 for compute jobs.
+        * and 120000 for compute jobs.
         * In SR-IOV or passthrough mode, timeout for compute
         * jobs are 60000 by default.
         */
@@ -3485,7 +3485,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev)
                adev->compute_timeout = amdgpu_sriov_is_pp_one_vf(adev) ?
                                        msecs_to_jiffies(60000) : msecs_to_jiffies(10000);
        else
-               adev->compute_timeout =  msecs_to_jiffies(60000);
+               adev->compute_timeout =  msecs_to_jiffies(120000);
 
        if (strnlen(input, AMDGPU_MAX_TIMEOUT_PARAM_LENGTH)) {
                while ((timeout_setting = strsep(&input, ",")) &&
--

2.34.1

#3 lore raw

Am 20.04.23 um 14:56 schrieb Alex Deucher:

Quoted 7 lines · Feifei Xu
Quoted 3 lines

Extend to 120s. The default timeout value should also extend if compute shader execution time extended. Otherwise some stress test will trigger compute ring timeout in software.

I think that's probably too long. 2 minutes is a long time to have a hung system. I think we should rework the tests or use ROCm for long running test cases.

Yeah, agree. This came up multiple times now and even 60000 is way to much actually.

You need to keep in mind that this has dependencies and it essentially means that the system sometimes needs 60sec in case of a bug to become responsible again.

So clearly a NAK for this.

Christian.

Quoted 33 lines

Alex

Quoted 30 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -3
index e536886f6d42..1f98b4b0a549 100644
@@ -3475,7 +3475,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev)
 
         /*
          * By default timeout for non compute jobs is 10000
-        * and 60000 for compute jobs.
+        * and 120000 for compute jobs.
          * In SR-IOV or passthrough mode, timeout for compute
          * jobs are 60000 by default.
          */
@@ -3485,7 +3485,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev)
                 adev->compute_timeout = amdgpu_sriov_is_pp_one_vf(adev) ?
                                         msecs_to_jiffies(60000) : msecs_to_jiffies(10000);
         else
-               adev->compute_timeout =  msecs_to_jiffies(60000);
+               adev->compute_timeout =  msecs_to_jiffies(120000);
 
         if (strnlen(input, AMDGPU_MAX_TIMEOUT_PARAM_LENGTH)) {
                 while ((timeout_setting = strsep(&input, ",")) &&
--

2.34.1

author
#4 lore raw
In reply to Alex Deucher

[AMD Official Use Only - General]

For some Vulkan stress tests, it might be not possible to rewrite using ROCm.
After a twice think, it might be too risky if we put 120s, because of the softlockup timeout set to 120s.

To support some stress tests like the one which recently I saw on stressbench (Vulkan stress test), if we shorten the 120s to a reasonable range like 100s, it can also fix the software hang.

-----Original Message-----
From: Alex Deucher <[email protected]>
Sent: Thursday, April 20, 2023 8:57 PM
To: Xu, Feifei <[email protected]>
Cc: [email protected]; Zhang, Hawking <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: extend the default timeout for kernel compute queues

Quoted 4 lines · Feifei Xu

Extend to 120s. The default timeout value should also extend if compute shader execution time extended. Otherwise some stress test will trigger compute ring timeout in software.

I think that's probably too long. 2 minutes is a long time to have a hung system. I think we should rework the tests or use ROCm for long running test cases.

Alex

Quoted 33 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -1
index e536886f6d42..1f98b4b0a549 100644
@@ -3475,7 +3475,7 @@ static int 
 amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev)
 
        /*
         * By default timeout for non compute jobs is 10000
-        * and 60000 for compute jobs.
+        * and 120000 for compute jobs.
         * In SR-IOV or passthrough mode, timeout for compute
         * jobs are 60000 by default.
         */
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -2
@@ -3485,7 +3485,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev)
                adev->compute_timeout = amdgpu_sriov_is_pp_one_vf(adev) ?
                                        msecs_to_jiffies(60000) : msecs_to_jiffies(10000);
        else
-               adev->compute_timeout =  msecs_to_jiffies(60000);
+               adev->compute_timeout =  msecs_to_jiffies(120000);
 
        if (strnlen(input, AMDGPU_MAX_TIMEOUT_PARAM_LENGTH)) {
                while ((timeout_setting = strsep(&input, ",")) &&
--

2.34.1