drm/amdgpu: extend the default timeout for kernel compute queues
Subsystems (2)
PATCH 1/1 drm/amdgpu: extend the default timeout for kernel compute queues
Extend to 120s. The default timeout value should also extend if compute shader execution time extended. Otherwise some stress test will trigger compute ring timeout in software.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -2
@@ -3475,7 +3475,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev)/* * By default timeout for non compute jobs is 10000 -* and 60000 for compute jobs. +* and 120000 for compute jobs. * In SR-IOV or passthrough mode, timeout for compute * jobs are 60000 by default. */
@@ -3485,7 +3485,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev) adev->compute_timeout =amdgpu_sriov_is_pp_one_vf (adev) ?msecs_to_jiffies (60000 ) :msecs_to_jiffies (10000 ); else - adev->compute_timeout = msecs_to_jiffies(60000); + adev->compute_timeout = msecs_to_jiffies(120000);if (strnlen (input,AMDGPU_MAX_TIMEOUT_PARAM_LENGTH )) { while ((timeout_setting = strsep(&input, ",")) &&
Signature · 1 line
2.34.1
[AMD Official Use Only - General]
Regards,
Hawking
-----Original Message-----
From: Feifei Xu <[email protected]>
Sent: Thursday, April 20, 2023 17:20
To: [email protected]
Cc: Xu, Feifei <[email protected]>; Zhang, Hawking <[email protected]>
Subject: [PATCH] drm/amdgpu: extend the default timeout for kernel compute queues
Extend to 120s. The default timeout value should also extend if compute shader execution time extended. Otherwise some stress test will trigger compute ring timeout in software.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -2
@@ -3475,7 +3475,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev)/* * By default timeout for non compute jobs is 10000 -* and 60000 for compute jobs. +* and 120000 for compute jobs. * In SR-IOV or passthrough mode, timeout for compute * jobs are 60000 by default. */
@@ -3485,7 +3485,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev) adev->compute_timeout =amdgpu_sriov_is_pp_one_vf (adev) ?msecs_to_jiffies (60000 ) :msecs_to_jiffies (10000 ); else - adev->compute_timeout = msecs_to_jiffies(60000); + adev->compute_timeout = msecs_to_jiffies(120000);if (strnlen (input,AMDGPU_MAX_TIMEOUT_PARAM_LENGTH )) { while ((timeout_setting = strsep(&input, ",")) &&
--
2.34.1
Quoted 4 lines · Feifei Xu
Extend to 120s. The default timeout value should also extend if compute shader execution time extended. Otherwise some stress test will trigger compute ring timeout in software.
I think that's probably too long. 2 minutes is a long time to have a hung system. I think we should rework the tests or use ROCm for long running test cases.
Alex
Quoted 31 lines
Signed-off-by Feifei Xu <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2
-3@@ -3475,7 +3475,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev)/* * By default timeout for non compute jobs is 10000 -* and 60000 for compute jobs. +* and 120000 for compute jobs. * In SR-IOV or passthrough mode, timeout for compute * jobs are 60000 by default. */ @@ -3485,7 +3485,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev) adev->compute_timeout =amdgpu_sriov_is_pp_one_vf (adev) ?msecs_to_jiffies (60000 ) :msecs_to_jiffies (10000 ); else - adev->compute_timeout = msecs_to_jiffies(60000); + adev->compute_timeout = msecs_to_jiffies(120000);if (strnlen (input,AMDGPU_MAX_TIMEOUT_PARAM_LENGTH )) { while ((timeout_setting = strsep(&input, ",")) && --2.34.1
Am 20.04.23 um 14:56 schrieb Alex Deucher:
Quoted 7 lines · Feifei Xu
Quoted 3 lines
Extend to 120s. The default timeout value should also extend if compute shader execution time extended. Otherwise some stress test will trigger compute ring timeout in software.
I think that's probably too long. 2 minutes is a long time to have a hung system. I think we should rework the tests or use ROCm for long running test cases.
Yeah, agree. This came up multiple times now and even 60000 is way to much actually.
You need to keep in mind that this has dependencies and it essentially means that the system sometimes needs 60sec in case of a bug to become responsible again.
So clearly a NAK for this.
Christian.
Quoted 33 lines
Alex
Quoted 30 lines
Signed-off-by Feifei Xu <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2
-3@@ -3475,7 +3475,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev)/* * By default timeout for non compute jobs is 10000 -* and 60000 for compute jobs. +* and 120000 for compute jobs. * In SR-IOV or passthrough mode, timeout for compute * jobs are 60000 by default. */ @@ -3485,7 +3485,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev) adev->compute_timeout =amdgpu_sriov_is_pp_one_vf (adev) ?msecs_to_jiffies (60000 ) :msecs_to_jiffies (10000 ); else - adev->compute_timeout = msecs_to_jiffies(60000); + adev->compute_timeout = msecs_to_jiffies(120000);if (strnlen (input,AMDGPU_MAX_TIMEOUT_PARAM_LENGTH )) { while ((timeout_setting = strsep(&input, ",")) && --2.34.1
[AMD Official Use Only - General]
For some Vulkan stress tests, it might be not possible to rewrite using ROCm.
After a twice think, it might be too risky if we put 120s, because of the softlockup timeout set to 120s.
To support some stress tests like the one which recently I saw on stressbench (Vulkan stress test), if we shorten the 120s to a reasonable range like 100s, it can also fix the software hang.
-----Original Message-----
From: Alex Deucher <[email protected]>
Sent: Thursday, April 20, 2023 8:57 PM
To: Xu, Feifei <[email protected]>
Cc: [email protected]; Zhang, Hawking <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: extend the default timeout for kernel compute queues
Quoted 4 lines · Feifei Xu
Extend to 120s. The default timeout value should also extend if compute shader execution time extended. Otherwise some stress test will trigger compute ring timeout in software.
I think that's probably too long. 2 minutes is a long time to have a hung system. I think we should rework the tests or use ROCm for long running test cases.
Alex
Quoted 33 lines
Signed-off-by Feifei Xu <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1@@ -3475,7 +3475,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev)/* * By default timeout for non compute jobs is 10000 -* and 60000 for compute jobs. +* and 120000 for compute jobs. * In SR-IOV or passthrough mode, timeout for compute * jobs are 60000 by default. */drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-2@@ -3485,7 +3485,7 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev) adev->compute_timeout =amdgpu_sriov_is_pp_one_vf (adev) ?msecs_to_jiffies (60000 ) :msecs_to_jiffies (10000 ); else - adev->compute_timeout = msecs_to_jiffies(60000); + adev->compute_timeout = msecs_to_jiffies(120000);if (strnlen (input,AMDGPU_MAX_TIMEOUT_PARAM_LENGTH )) { while ((timeout_setting = strsep(&input, ",")) && --2.34.1