drm/amdkfd: make sure VM is ready for updating operations
Subsystems (3)
PATCH 1/1 drm/amdkfd: make sure VM is ready for updating operations
When VM is in evicting state, amdgpu_vm_update_range would return -EBUSY. Then restore_process_worker runs into a dead loop.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +6 -0
@@ -2900,6 +2900,12 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence __rcu *amdgpu_sync_create (&sync_obj); + ret =process_validate_vms (process_info,NULL ); +if (ret) { +pr_debug ("Validating VMs failed, ret: %d\n" , ret); + goto validate_map_fail; + } +/* Validate BOs and map them to GPUVM (update VM page tables). */ list_for_each_entry (mem, &process_info->kfd_bo_list, validate_list) {
Signature · 1 line
2.25.1
Christian König
maintainerAm 07.04.24 um 06:52 schrieb Lang Yu:
Quoted 4 lines
When VM is in evicting state,
amdgpu_vm_update_rangewould return-EBUSY. Thenrestore_process_workerruns into a dead loop.Fixes 2fdba514ad5a ("drm/amdgpu: Auto-validate DMABuf imports in compute VMs")
Mhm, while it would be good to have this case handled as error it should never occur in practice since we should have validated the VM before validating the DMA-bufs.
@Felix isn't that something we have taken care of?
Regards,
Christian.
Quoted 23 lines
Signed-off-by Lang Yu <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 6 ++++++ 1 file changed, 6 insertions(+)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +6
-0@@ -2900,6 +2900,12 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence __rcu *amdgpu_sync_create (&sync_obj); + ret =process_validate_vms (process_info,NULL ); +if (ret) { +pr_debug ("Validating VMs failed, ret: %d\n" , ret); + goto validate_map_fail; + } +/* Validate BOs and map them to GPUVM (update VM page tables). */ list_for_each_entry (mem, &process_info->kfd_bo_list, validate_list) {
[AMD Official Use Only - General]
Quoted 11 lines
-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Monday, April 8, 2024 3:55 PM
To: Yu, Lang <[email protected]>; [email protected]
Cc: Kuehling, Felix <[email protected]>
Subject: Re: [PATCH] drm/amdkfd: make sure VM is ready for updating operationsAm 07.04.24 um 06:52 schrieb Lang Yu:
Quoted 1 line
When VM is in evicting state,
amdgpu_vm_update_rangewould return -EBUSY.
Quoted 8 lines
Quoted 4 lines
Then
restore_process_workerruns into a dead loop.Fixes 2fdba514ad5a ("drm/amdgpu: Auto-validate DMABuf imports incompute VMs")
Mhm, while it would be good to have this case handled as error it should never occur in practice since we should have validated the VM before validating the DMA-bufs.
When page table BOs were evicted but not validated before updating page tables, VM is still in evicting state, then the issue happened.
Regards,
Lang
Quoted 34 lines
@Felix isn't that something we have taken care of?
Regards,
Christian.Quoted 12 lines
Signed-off-by Lang Yu <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 6 ++++++ 1 file changed, 6 insertions(+)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2900,6 +2900,12 @@ intQuoted 1 line
amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct
dma_fencedrivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence __rcu *
Quoted 11 lines
__rcu*amdgpu_sync_create(&sync_obj);drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +6
-0+ ret =process_validate_vms (process_info,NULL ); +if (ret) { +pr_debug ("Validating VMs failed, ret: %d\n" , ret); + goto validate_map_fail; + } +/* Validate BOs and map them to GPUVM (update VM page tables). */
Quoted 2 lines
list_for_each_entry(mem, &process_info->kfd_bo_list, validate_list) {
Felix Kuehling
maintainerQuoted 10 lines · Christian König
Am 07.04.24 um 06:52 schrieb Lang Yu:
Quoted 5 lines
When VM is in evicting state,
amdgpu_vm_update_rangewould return-EBUSY.
Thenrestore_process_workerruns into a dead loop.Fixes 2fdba514ad5a ("drm/amdgpu: Auto-validate DMABuf imports in compute VMs")Mhm, while it would be good to have this case handled as error it should never occur in practice since we should have validated the VM before validating the DMA-bufs.
@Felix isn't that something we have taken care of?
The problem I saw when I implemented Auto-validate was, that migration of a BO invalidates its DMABuf attachments. So I need to validate the DMABuf attachments after validating the BOs they attach to. This auto-validation happens in amdgpu_vm_validate. So I needed to do the VM validation after the BO validation. The problem now seems to be that the BO validation happens in the same loop as the page table update. And the page table update fails if the VM is not valid.
I never saw this problem in my testing, probably because I never got my page tables evicted?
Anyway, I think the solution is to split the BO validation and page table update into two separate loops in amdgpu_amdkfd_restore_process_pos:
1. Validate BOs
2. Validate VM (and DMABuf attachments)
3. Update page tables for the BOs validated above
Regards,
Felix
Quoted 28 lines
Regards,
Christian.Quoted 22 lines
Signed-off-by Lang Yu <[email protected]>---
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c| 6 ++++++1 file changed, 6 insertions(+)drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0
-0@@ -2900,6 +2900,12 @@ intamdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct
dma_fence__rcu*amdgpu_sync_create(&sync_obj);
+ ret =process_validate_vms(process_info, NULL);
+ if (ret) {
+pr_debug("Validating VMs failed, ret: %d\n", ret);
+ gotovalidate_map_fail;
+ }
+
/* Validate BOs and map them to GPUVM (update VM page tables). */list_for_each_entry(mem, &process_info->kfd_bo_list, validate_list){
[Public]
Quoted 1 line
I never saw this problem in my testing, probably because I never got my page tables evicted?
I observed this problem on APUs with default 512MB VRAM when allocating memory aggressively from different APPs.
Will try to modify the patch per your suggestions. Thanks!
Regards,
Lang
From: Kuehling, Felix <[email protected]>
Sent: Wednesday, April 10, 2024 8:32 AM
To: Koenig, Christian <[email protected]>; Yu, Lang <[email protected]>; [email protected]
Subject: Re: [PATCH] drm/amdkfd: make sure VM is ready for updating operations
On 2024-04-08 3:55, Christian König wrote:
Am 07.04.24 um 06:52 schrieb Lang Yu:
When VM is in evicting state, amdgpu_vm_update_range would return -EBUSY. Then restore_process_worker runs into a dead loop.
Mhm, while it would be good to have this case handled as error it should never occur in practice since we should have validated the VM before validating the DMA-bufs.
@Felix isn't that something we have taken care of?
The problem I saw when I implemented Auto-validate was, that migration of a BO invalidates its DMABuf attachments. So I need to validate the DMABuf attachments after validating the BOs they attach to. This auto-validation happens in amdgpu_vm_validate. So I needed to do the VM validation after the BO validation. The problem now seems to be that the BO validation happens in the same loop as the page table update. And the page table update fails if the VM is not valid.
I never saw this problem in my testing, probably because I never got my page tables evicted?
Anyway, I think the solution is to split the BO validation and page table update into two separate loops in amdgpu_amdkfd_restore_process_pos:
1. Validate BOs 2. Validate VM (and DMABuf attachments) 3. Update page tables for the BOs validated above
Regards,
Felix
Regards,
Christian.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 6 ++++++ 1 file changed, 6 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +5 -0
@@ -2900,6 +2900,12 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence __rcu *amdgpu_sync_create (&sync_obj); + ret =process_validate_vms (process_info,NULL ); +if (ret) { +pr_debug ("Validating VMs failed, ret: %d\n" , ret); + goto validate_map_fail; + } +/* Validate BOs and map them to GPUVM (update VM page tables). */ list_for_each_entry (mem, &process_info->kfd_bo_list, validate_list) {