Skip to contentSkip to conversation

drm/amdkfd: make sure VM is ready for updating operations

PATCH Superseded Lang Yu 3 participants 5 messages Apr 7, 2024 – Apr 10, 2024

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Lang Yu

author
#0 lore raw

PATCH 1/1 drm/amdkfd: make sure VM is ready for updating operations

When VM is in evicting state, amdgpu_vm_update_range would return -EBUSY. Then restore_process_worker runs into a dead loop.

Fixes 2fdba514ad5a ("drm/amdgpu: Auto-validate DMABuf imports in compute VMs")
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +6 -0
index 0ae9fd844623..8c71fe07807a 100644
@@ -2900,6 +2900,12 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence __rcu *
 
 	amdgpu_sync_create(&sync_obj);
 
+	ret = process_validate_vms(process_info, NULL);
+	if (ret) {
+		pr_debug("Validating VMs failed, ret: %d\n", ret);
+		goto validate_map_fail;
+	}
+
 	/* Validate BOs and map them to GPUVM (update VM page tables). */
 	list_for_each_entry(mem, &process_info->kfd_bo_list,
 			    validate_list) {
Signature · 1 line
2.25.1
maintainer
#1 lore raw

Am 07.04.24 um 06:52 schrieb Lang Yu:

Quoted 4 lines

When VM is in evicting state, amdgpu_vm_update_range would return -EBUSY. Then restore_process_worker runs into a dead loop.

Fixes 2fdba514ad5a ("drm/amdgpu: Auto-validate DMABuf imports in compute VMs")

Mhm, while it would be good to have this case handled as error it should never occur in practice since we should have validated the VM before validating the DMA-bufs.

@Felix isn't that something we have taken care of?

Regards,
Christian.

Quoted 23 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 6 ++++++
  1 file changed, 6 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +6 -0
index 0ae9fd844623..8c71fe07807a 100644
@@ -2900,6 +2900,12 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence __rcu *
 
  	amdgpu_sync_create(&sync_obj);
 
+	ret = process_validate_vms(process_info, NULL);
+	if (ret) {
+		pr_debug("Validating VMs failed, ret: %d\n", ret);
+		goto validate_map_fail;
+	}
+
  	/* Validate BOs and map them to GPUVM (update VM page tables). */
  	list_for_each_entry(mem, &process_info->kfd_bo_list,
  			    validate_list) {
author
#2 lore raw

[AMD Official Use Only - General]

Quoted 11 lines

-----Original Message-----
From: Koenig, Christian <[email protected]>
Sent: Monday, April 8, 2024 3:55 PM
To: Yu, Lang <[email protected]>; [email protected]
Cc: Kuehling, Felix <[email protected]>
Subject: Re: [PATCH] drm/amdkfd: make sure VM is ready for updating operations

Am 07.04.24 um 06:52 schrieb Lang Yu:

Quoted 1 line

When VM is in evicting state, amdgpu_vm_update_range would return -

EBUSY.

Quoted 8 lines
Quoted 4 lines

Then restore_process_worker runs into a dead loop.

Fixes 2fdba514ad5a ("drm/amdgpu: Auto-validate DMABuf imports in

compute VMs")

Mhm, while it would be good to have this case handled as error it should never occur in practice since we should have validated the VM before validating the DMA-bufs.

When page table BOs were evicted but not validated before updating page tables, VM is still in evicting state, then the issue happened.

Regards,
Lang

Quoted 34 lines

@Felix isn't that something we have taken care of?

Regards,
Christian.

Quoted 12 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 6 ++++++
  1 file changed, 6 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0 -0
index 0ae9fd844623..8c71fe07807a 100644
@@ -2900,6 +2900,12 @@ int
Quoted 1 line

amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct

dma_fence

drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence __rcu *

Quoted 11 lines

__rcu *

     amdgpu_sync_create(&sync_obj);
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +6 -0
+    ret = process_validate_vms(process_info, NULL);
+    if (ret) {
+            pr_debug("Validating VMs failed, ret: %d\n", ret);
+            goto validate_map_fail;
+    }
+
     /* Validate BOs and map them to GPUVM (update VM page tables).

*/

Quoted 2 lines
     list_for_each_entry(mem, &process_info->kfd_bo_list,
                         validate_list) {
maintainer
#3 lore raw
In reply to Christian König
Quoted 10 lines · Christian König

Am 07.04.24 um 06:52 schrieb Lang Yu:

Quoted 5 lines

When VM is in evicting state, amdgpu_vm_update_range would return -EBUSY.
Then restore_process_worker runs into a dead loop.

Fixes 2fdba514ad5a ("drm/amdgpu: Auto-validate DMABuf imports in compute VMs")

Mhm, while it would be good to have this case handled as error it should never occur in practice since we should have validated the VM before validating the DMA-bufs.

@Felix isn't that something we have taken care of?

The problem I saw when I implemented Auto-validate was, that migration of a BO invalidates its DMABuf attachments. So I need to validate the DMABuf attachments after validating the BOs they attach to. This auto-validation happens in amdgpu_vm_validate. So I needed to do the VM validation after the BO validation. The problem now seems to be that the BO validation happens in the same loop as the page table update. And the page table update fails if the VM is not valid.

I never saw this problem in my testing, probably because I never got my page tables evicted?

Anyway, I think the solution is to split the BO validation and page table update into two separate loops in amdgpu_amdkfd_restore_process_pos:

1. Validate BOs
2. Validate VM (and DMABuf attachments)
3. Update page tables for the BOs validated above

Regards,
  Felix

Quoted 28 lines

Regards,
Christian.

Quoted 22 lines

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 6 ++++++

  1 file changed, 6 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +0 -0
index 0ae9fd844623..8c71fe07807a 100644
@@ -2900,6 +2900,12 @@ int 

amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence __rcu * amdgpu_sync_create(&sync_obj);
  +    ret = process_validate_vms(process_info, NULL);
+    if (ret) {
+        pr_debug("Validating VMs failed, ret: %d\n", ret);
+        goto validate_map_fail;
+    }
+
      /* Validate BOs and map them to GPUVM (update VM page tables). */ list_for_each_entry(mem, &process_info->kfd_bo_list, validate_list) {

author
#4 lore raw

[Public]

Quoted 1 line

I never saw this problem in my testing, probably because I never got my page tables evicted?

I observed this problem on APUs with default 512MB VRAM when allocating memory aggressively from different APPs.

Will try to modify the patch per your suggestions. Thanks!

Regards,
Lang

From: Kuehling, Felix <[email protected]>
Sent: Wednesday, April 10, 2024 8:32 AM
To: Koenig, Christian <[email protected]>; Yu, Lang <[email protected]>; [email protected]
Subject: Re: [PATCH] drm/amdkfd: make sure VM is ready for updating operations

On 2024-04-08 3:55, Christian König wrote:
Am 07.04.24 um 06:52 schrieb Lang Yu:

When VM is in evicting state, amdgpu_vm_update_range would return -EBUSY. Then restore_process_worker runs into a dead loop.

Fixes 2fdba514ad5a ("drm/amdgpu: Auto-validate DMABuf imports in compute VMs")

Mhm, while it would be good to have this case handled as error it should never occur in practice since we should have validated the VM before validating the DMA-bufs.

@Felix isn't that something we have taken care of?

The problem I saw when I implemented Auto-validate was, that migration of a BO invalidates its DMABuf attachments. So I need to validate the DMABuf attachments after validating the BOs they attach to. This auto-validation happens in amdgpu_vm_validate. So I needed to do the VM validation after the BO validation. The problem now seems to be that the BO validation happens in the same loop as the page table update. And the page table update fails if the VM is not valid.

I never saw this problem in my testing, probably because I never got my page tables evicted?

Anyway, I think the solution is to split the BO validation and page table update into two separate loops in amdgpu_amdkfd_restore_process_pos:

  1.  Validate BOs
  2.  Validate VM (and DMABuf attachments)
  3.  Update page tables for the BOs validated above

Regards,
Felix

Regards,
Christian.

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 6 ++++++
  1 file changed, 6 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +5 -0
index 0ae9fd844623..8c71fe07807a 100644
@@ -2900,6 +2900,12 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence __rcu *
        amdgpu_sync_create(&sync_obj);
  +    ret = process_validate_vms(process_info, NULL);
+    if (ret) {
+        pr_debug("Validating VMs failed, ret: %d\n", ret);
+        goto validate_map_fail;
+    }
+
      /* Validate BOs and map them to GPUVM (update VM page tables). */
      list_for_each_entry(mem, &process_info->kfd_bo_list,
                  validate_list) {