amd/amdkfd: sync all devices to wait all processes being evicted
Subsystems (3)
PATCH 1/2 amd/amdkfd: sync all devices to wait all processes being evicted
If there are more than one device doing reset in parallel, the first device will call kfd_suspend_all_processes() to evict all processes on all devices, this call takes time to finish. other device will start reset and recover without waiting. if the process has not been evicted before doing recover, it will be restored, then caused page fault.
--- drivers/gpu/drm/amd/amdkfd/kfd_device.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_device.c +1 -1
@@ -969,11 +969,11 @@ void kgd2kfd_suspend(struct kfd_dev *kfd, bool run_pm)if (!run_pm) {mutex_lock (&kfd_processes_mutex); count = ++kfd_locked; -mutex_unlock (&kfd_processes_mutex);/* For first KFD device suspend all the KFD processes */ if (count ==1 )kfd_suspend_all_processes (); +mutex_unlock (&kfd_processes_mutex); }for (i =0 ; i < kfd->num_nodes; i++) {
Signature · 1 line
2.25.1
PATCH 2/2 amd/amdgpu: improve VF recover time
1. change AMDGPU_VF2PF_UPDATE_MAX_RETRY_LIMIT from 30 to 5.
2. set fatel error detected flag.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h | 2 +- 3 files changed, 3 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -0
@@ -4982,6 +4982,7 @@ static int amdgpu_device_reset_sriov(struct amdgpu_device *adev, r =amdgpu_virt_reset_gpu (adev);if (r)return r; +amdgpu_ras_set_fed (adev, false);amdgpu_irq_gpu_reset_resume_helper (adev);/* some sw clean up VF needs to do before recover */
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c +1 -0
@@ -593,6 +593,7 @@ static void amdgpu_virt_update_vf2pf_work_item(struct work_struct *work) adev->virt.vf2pf_update_retry_cnt++;if ((adev->virt.vf2pf_update_retry_cnt >=AMDGPU_VF2PF_UPDATE_MAX_RETRY_LIMIT ) &&amdgpu_sriov_runtime (adev) && !amdgpu_in_reset (adev)) { +amdgpu_ras_set_fed (adev, true);if (amdgpu_reset_domain_schedule (adev->reset_domain, &adev->virt.flr_work))return ;
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h +1 -1
@@ -52,7 +52,7 @@/* tonga/fiji use this offset */ #define mmBIF_IOV_FUNC_IDENTIFIER 0x1503 -#define AMDGPU_VF2PF_UPDATE_MAX_RETRY_LIMIT 30 +#define AMDGPU_VF2PF_UPDATE_MAX_RETRY_LIMIT 5enum amdgpu_sriov_vf_mode {SRIOV_VF_MODE_BARE_METAL =0 ,
Signature · 1 line
2.25.1
Felix Kuehling
maintainerQuoted 9 lines · Zhigang Luo
If there are more than one device doing reset in parallel, the first device will call
kfd_suspend_all_processes()to evict all processes on all devices, this call takes time to finish. other device will start reset and recover without waiting. if the process has not been evicted before doing recover, it will be restored, then caused page fault.Signed-off-by Zhigang Luo<[email protected]>Change-Id Ib1eddb56b69ecd41fe703abd169944154f48b0cd
Please remove the Change-Id: before you push. Other than that, this patch is
Quoted 9 lines
--- drivers/gpu/drm/amd/amdkfd/kfd_device.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/drm/amd/amdkfd/kfd_device.c +0
-0
drivers/gpu/drm/amd/amdkfd/kfd_device.c void kgd2kfd_suspend(struct kfd_dev *kfd, bool run_pm) · line 975
Quoted 8 lines
drivers/gpu/drm/amd/amdkfd/kfd_device.c +0
-1@@ -969,11 +969,11 @@ void kgd2kfd_suspend(struct kfd_dev *kfd, bool run_pm)if (!run_pm) {mutex_lock (&kfd_processes_mutex); count = ++kfd_locked; -mutex_unlock (&kfd_processes_mutex);/* For first KFD device suspend all the KFD processes */ if (count ==1 )kfd_suspend_all_processes ();
This could be simplified now. The variable "count" was only needed for the broken attempt to do call suspend outside the lock. Now you can just do:
mutex_lock(&kfd_processes_mutex); if (++kfd_locked == 1) kfd_suspend_all_processes(); mutex_unlock(&kfd_processes_mutex);
To be consistent, we probably need to make a similar change in kgd2kfd_resume and run kfd_resume_all_processes under the lock as well. Otherwise there could be a race condition between suspend and resume.
Regards,
Felix
Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/kfd_device.c +1
-0+mutex_unlock (&kfd_processes_mutex); }for (i =0 ; i < kfd->num_nodes; i++) {
Quoted 21 lines · Zhigang Luo
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
If there are more than one device doing reset in parallel, the first device will call
kfd_suspend_all_processes()to evict all processes on all devices, this call takes time to finish. other device will start reset and recover without waiting. if the process has not been evicted before doing recover, it will be restored, then caused page fault.Signed-off-by Zhigang Luo <[email protected]>Change-Id Ib1eddb56b69ecd41fe703abd169944154f48b0cd--- drivers/gpu/drm/amd/amdkfd/kfd_device.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/drm/amd/amdkfd/kfd_device.c +0
-0
drivers/gpu/drm/amd/amdkfd/kfd_device.c void kgd2kfd_suspend(struct kfd_dev *kfd, bool run_pm) · line 977
Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_device.c +1
-1@@ -969,11 +969,11 @@ void kgd2kfd_suspend(struct kfd_dev *kfd, bool run_pm)if (!run_pm) {mutex_lock (&kfd_processes_mutex); count = ++kfd_locked; -mutex_unlock (&kfd_processes_mutex);/* For first KFD device suspend all the KFD processes */ if (count ==1 )kfd_suspend_all_processes (); +mutex_unlock (&kfd_processes_mutex); }
I do not understand why use kfd_lock here. You want evict all processes when first device got suspended. The kfd_lock indicates if all kfd driver functions got locked. It is not same meaning as device suspend. That is not your patch issue, but I think using different flag to record device suspend is better. ex, if kfd_lock got set for some other reasons, we would skip evicting processes here.
Regards
Xiaogang
Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/kfd_device.c +0
-1for (i =0 ; i < kfd->num_nodes; i++) { --2.25.1