Skip to contentSkip to conversation

drm/amdgpu: Checkpoint and Restore VRAM BOs without VA

PATCH Superseded Ramesh Errabolu 2 participants 4 messages Jul 24, 2023 – Jul 25, 2023

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/amdgpu: Checkpoint and Restore VRAM BOs without VA

Extend checkpoint logic to allow inclusion of VRAM BOs that do not have a VA attached

---
 drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +4 -2
index 40ac093b5035..5cc00ff4b635 100644
@@ -1845,7 +1845,8 @@ static uint32_t get_process_num_bos(struct kfd_process *p)
 		idr_for_each_entry(&pdd->alloc_idr, mem, id) {
 			struct kgd_mem *kgd_mem = (struct kgd_mem *)mem;
 
-			if ((uint64_t)kgd_mem->va > pdd->gpuvm_base)
+			if (((uint64_t)kgd_mem->va > pdd->gpuvm_base) ||
+			    (kgd_mem->va == 0))
 				num_of_bos++;
 		}
 	}
@@ -1917,7 +1918,8 @@ static int criu_checkpoint_bos(struct kfd_process *p,
 			kgd_mem = (struct kgd_mem *)mem;
 			dumper_bo = kgd_mem->bo;
 
-			if ((uint64_t)kgd_mem->va <= pdd->gpuvm_base)
+			if (((uint64_t)kgd_mem->va <= pdd->gpuvm_base) &&
+				!(kgd_mem->va == 0))
 				continue;
 
 			bo_bucket = &bo_buckets[bo_index];
Signature · 1 line
2.25.1
maintainer
#1 lore raw
Quoted 13 lines · Ramesh Errabolu

Extend checkpoint logic to allow inclusion of VRAM BOs that do not have a VA attached

---
  drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +0 -0
index 40ac093b5035..5cc00ff4b635 100644

drivers/gpu/drm/amd/amdkfd/kfd_chardev.c static uint32_t get_process_num_bos(struct kfd_process *p) · line 1849

Quoted 6 lines
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +2 -1
@@ -1845,7 +1845,8 @@ static uint32_t get_process_num_bos(struct kfd_process *p)
  		idr_for_each_entry(&pdd->alloc_idr, mem, id) {
  			struct kgd_mem *kgd_mem = (struct kgd_mem *)mem;
 
-			if ((uint64_t)kgd_mem->va > pdd->gpuvm_base)
+			if (((uint64_t)kgd_mem->va > pdd->gpuvm_base) ||
+			    (kgd_mem->va == 0))

I'm trying to remember what this condition is there to protect against, because it almost looks like we could drop the entire condition. I think it's there to avoid checkpointing the TMA/TBA BOs allocated by KFD itself.

That said, you have some unnecessary parentheses in this expression. And just using !x to check for 0 is usually preferred in the kernel. This should work and is more readable IMO:

+ if ((uint64_t)kgd_mem->va > pdd->gpuvm_base || !kgd_mem->va)

drivers/gpu/drm/amd/amdkfd/kfd_chardev.c static int criu_checkpoint_bos(struct kfd_process *p, · line 1922

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +2 -1
  				num_of_bos++;
  		}
  	}
@@ -1917,7 +1918,8 @@ static int criu_checkpoint_bos(struct kfd_process *p,
  			kgd_mem = (struct kgd_mem *)mem;
  			dumper_bo = kgd_mem->bo;
 
-			if ((uint64_t)kgd_mem->va <= pdd->gpuvm_base)
+			if (((uint64_t)kgd_mem->va <= pdd->gpuvm_base) &&
+				!(kgd_mem->va == 0))

Similar to above:

+ if (kgd_mem->va && (uint64_t)kgd_mem->va <= pdd->gpuvm_base)

Regards,
  Felix

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +0 -0
  				continue;
 
  			bo_bucket = &bo_buckets[bo_index];
#2 lore raw

[AMD Official Use Only - General]

Responses inline.

-----Original Message-----
From: Kuehling, Felix <[email protected]>
Sent: Monday, July 24, 2023 2:51 PM
To: [email protected]; Errabolu, Ramesh <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Checkpoint and Restore VRAM BOs without VA

Quoted 14 lines · Ramesh Errabolu

Extend checkpoint logic to allow inclusion of VRAM BOs that do not have a VA attached

---
  drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +0 -0
index 40ac093b5035..5cc00ff4b635 100644

drivers/gpu/drm/amd/amdkfd/kfd_chardev.c static uint32_t get_process_num_bos(struct kfd_process *p) · line 1849

Quoted 6 lines
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +2 -1
@@ -1845,7 +1845,8 @@ static uint32_t get_process_num_bos(struct kfd_process *p)
              idr_for_each_entry(&pdd->alloc_idr, mem, id) {
                      struct kgd_mem *kgd_mem = (struct kgd_mem *)mem;
 
-                     if ((uint64_t)kgd_mem->va > pdd->gpuvm_base)
+                     if (((uint64_t)kgd_mem->va > pdd->gpuvm_base) ||
+                         (kgd_mem->va == 0))

I'm trying to remember what this condition is there to protect against, because it almost looks like we could drop the entire condition. I think it's there to avoid checkpointing the TMA/TBA BOs allocated by KFD itself.

Ramesh: I am unsure as to how we can detect TMA/TBA BOs if we don't want them checkpointed. Alternatively we can checkpoint and restore TMA/TBA BOs without loss of function. If this o.k. we can drop the check that determines BO qualification.

That said, you have some unnecessary parentheses in this expression. And just using !x to check for 0 is usually preferred in the kernel. This should work and is more readable IMO:

+ if ((uint64_t)kgd_mem->va > pdd->gpuvm_base || !kgd_mem->va)

drivers/gpu/drm/amd/amdkfd/kfd_chardev.c static int criu_checkpoint_bos(struct kfd_process *p, · line 1922

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +2 -1
                              num_of_bos++;
              }
      }
@@ -1917,7 +1918,8 @@ static int criu_checkpoint_bos(struct kfd_process *p,
                      kgd_mem = (struct kgd_mem *)mem;
                      dumper_bo = kgd_mem->bo;
 
-                     if ((uint64_t)kgd_mem->va <= pdd->gpuvm_base)
+                     if (((uint64_t)kgd_mem->va <= pdd->gpuvm_base) &&
+                             !(kgd_mem->va == 0))

Similar to above:

+ if (kgd_mem->va && (uint64_t)kgd_mem->va <= pdd->gpuvm_base)

Regards,
Felix

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +0 -0
                              continue;
 
                      bo_bucket = &bo_buckets[bo_index];
maintainer
#3 lore raw

Am 2023-07-25 um 16:04 schrieb Errabolu, Ramesh:

Quoted 29 lines

[AMD Official Use Only - General]

Responses inline.

-----Original Message-----
From: Kuehling, Felix <[email protected]>
Sent: Monday, July 24, 2023 2:51 PM
To: [email protected]; Errabolu, Ramesh <[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Checkpoint and Restore VRAM BOs without VA

Quoted 17 lines · Ramesh Errabolu

Extend checkpoint logic to allow inclusion of VRAM BOs that do not have a VA attached

---
   drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 6 ++++--
   1 file changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +0 -0
index 40ac093b5035..5cc00ff4b635 100644
@@ -1845,7 +1845,8 @@ static uint32_t get_process_num_bos(struct kfd_process *p)
               idr_for_each_entry(&pdd->alloc_idr, mem, id) {
                       struct kgd_mem *kgd_mem = (struct kgd_mem *)mem;
 
Quoted 6 lines

drivers/gpu/drm/amd/amdkfd/kfd_chardev.c static uint32_t get_process_num_bos(struct kfd_process *p) · line 1849

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +2 -1
-                     if ((uint64_t)kgd_mem->va > pdd->gpuvm_base)
+                     if (((uint64_t)kgd_mem->va > pdd->gpuvm_base) ||
+                         (kgd_mem->va == 0))

I'm trying to remember what this condition is there to protect against, because it almost looks like we could drop the entire condition. I think it's there to avoid checkpointing the TMA/TBA BOs allocated by KFD itself.

Ramesh: I am unsure as to how we can detect TMA/TBA BOs if we don't want them checkpointed. Alternatively we can checkpoint and restore TMA/TBA BOs without loss of function. If this o.k. we can drop the check that determines BO qualification.

It's OK. Currently they have a VA > 0 and < gpuvm_base. So this check will still work if you only allow BOs with VA == 0.

There is a patch in the works to move the TMA and TBA to the upper half of the virtual address space. Then we'll need to update this check to exclude anything that has bit 63 of the VA set.

Regards,
  Felix

Quoted 27 lines

That said, you have some unnecessary parentheses in this expression. And just using !x to check for 0 is usually preferred in the kernel. This should work and is more readable IMO:

+ if ((uint64_t)kgd_mem->va > pdd->gpuvm_base || !kgd_mem->va)

drivers/gpu/drm/amd/amdkfd/kfd_chardev.c static int criu_checkpoint_bos(struct kfd_process *p, · line 1922

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +2 -1
                               num_of_bos++;
               }
       }
@@ -1917,7 +1918,8 @@ static int criu_checkpoint_bos(struct kfd_process *p,
                       kgd_mem = (struct kgd_mem *)mem;
                       dumper_bo = kgd_mem->bo;
 
-                     if ((uint64_t)kgd_mem->va <= pdd->gpuvm_base)
+                     if (((uint64_t)kgd_mem->va <= pdd->gpuvm_base) &&
+                             !(kgd_mem->va == 0))

Similar to above:

+ if (kgd_mem->va && (uint64_t)kgd_mem->va <= pdd->gpuvm_base)

Regards,
Felix

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +0 -0
                               continue;
 
                       bo_bucket = &bo_buckets[bo_index];