Skip to contentSkip to conversation

drm/amdkfd: Optimize svm range map to GPU with XNACK on

PATCH In review Philip Yang 3 participants 4 messages Apr 24, 2023 – Apr 28, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/amdkfd: Optimize svm range map to GPU with XNACK on

With XNACK on if svm_range_set_attr set the range access or access_in_place attribute, we don't call svm_range_validate_and_map to update GPU mapping. This avoids prefaulting the range pages on system memory if the range is not prefetch to VRAM and not mapped to GPUs.

---
 drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +3 -1
index 129ef44f41e9..af7594b1b4c6 100644
@@ -734,7 +734,9 @@ svm_range_apply_attrs(struct kfd_process *p, struct svm_range *prange,
 		case KFD_IOCTL_SVM_ATTR_ACCESS:
 		case KFD_IOCTL_SVM_ATTR_ACCESS_IN_PLACE:
 		case KFD_IOCTL_SVM_ATTR_NO_ACCESS:
-			*update_mapping = true;
+			if (!p->xnack_enabled)
+				*update_mapping = true;
+
 			gpuidx = kfd_process_gpuidx_from_gpuid(p,
 							       attrs[i].value);
 			if (attrs[i].type == KFD_IOCTL_SVM_ATTR_NO_ACCESS) {
Signature · 1 line
2.35.1
maintainer
#1 lore raw
Quoted 15 lines · Philip Yang

With XNACK on if svm_range_set_attr set the range access or access_in_place attribute, we don't call svm_range_validate_and_map to update GPU mapping. This avoids prefaulting the range pages on system memory if the range is not prefetch to VRAM and not mapped to GPUs.

---
  drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
index 129ef44f41e9..af7594b1b4c6 100644

drivers/gpu/drm/amd/amdkfd/kfd_svm.c svm_range_apply_attrs(struct kfd_process *p, struct svm_range *prange, · line 739

Quoted 7 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +3 -1
@@ -734,7 +734,9 @@ svm_range_apply_attrs(struct kfd_process *p, struct svm_range *prange,
  		case KFD_IOCTL_SVM_ATTR_ACCESS:
  		case KFD_IOCTL_SVM_ATTR_ACCESS_IN_PLACE:
  		case KFD_IOCTL_SVM_ATTR_NO_ACCESS:
-			*update_mapping = true;
+			if (!p->xnack_enabled)
+				*update_mapping = true;
+

For NO_ACCESS we need to update the mapping to ensure that the PTEs are invalidated. For ACCESS or ACCESS_IN_PLACE we can leave it for a page fault.

Regards,
  Felix

Quoted 3 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
  			gpuidx = kfd_process_gpuidx_from_gpuid(p,
  							       attrs[i].value);
  			if (attrs[i].type == KFD_IOCTL_SVM_ATTR_NO_ACCESS) {
#2 lore raw
Quoted 12 lines · Felix Kuehling
Quoted 10 lines · Philip Yang

With XNACK on if svm_range_set_attr set the range access or access_in_place attribute, we don't call svm_range_validate_and_map to update GPU mapping. This avoids prefaulting the range pages on system memory if the range is not prefetch to VRAM and not mapped to GPUs.

---
  drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 4 +++-

  1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 15 lines
Quoted 6 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
index 129ef44f41e9..af7594b1b4c6 100644
@@ -734,7 +734,9 @@ svm_range_apply_attrs(struct kfd_process *p, 
 struct svm_range *prange,
Quoted 7 lines

          case KFD_IOCTL_SVM_ATTR_ACCESS:
          case KFD_IOCTL_SVM_ATTR_ACCESS_IN_PLACE:
          case KFD_IOCTL_SVM_ATTR_NO_ACCESS:
-            *update_mapping = true;
+            if (!p->xnack_enabled)
+                *update_mapping = true;
+

For NO_ACCESS we need to update the mapping to ensure that the PTEs are invalidated. For ACCESS or ACCESS_IN_PLACE we can leave it for a page fault.

For NO_ACCESS, there is todo comment "TODO: unmap ranges from GPU that lost access", this should be handled by another patch. This patch only address that for XNACK ON, we can setup GPU mapping after page fault, without prefaulting and mapping to GPU for ACCESS or ACCESS_IN_PLACE.

Regards,

Philip

Quoted 8 lines

Regards,
  Felix

Quoted 3 lines

              gpuidx = kfd_process_gpuidx_from_gpuid(p, attrs[i].value); if (attrs[i].type == KFD_IOCTL_SVM_ATTR_NO_ACCESS) {

maintainer Reviewed-by
#3 lore raw
Quoted 30 lines · Philip Yang
Quoted 12 lines · Felix Kuehling
Quoted 10 lines · Philip Yang

With XNACK on if svm_range_set_attr set the range access or access_in_place attribute, we don't call svm_range_validate_and_map to update GPU mapping. This avoids prefaulting the range pages on system memory if the range is not prefetch to VRAM and not mapped to GPUs.

---
  drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 4 +++-

  1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 15 lines
Quoted 6 lines
drivers/gpu/drm/amd/amdkfd/kfd_svm.c +0 -0
index 129ef44f41e9..af7594b1b4c6 100644
@@ -734,7 +734,9 @@ svm_range_apply_attrs(struct kfd_process *p, 
 struct svm_range *prange,
Quoted 7 lines

          case KFD_IOCTL_SVM_ATTR_ACCESS:
          case KFD_IOCTL_SVM_ATTR_ACCESS_IN_PLACE:
          case KFD_IOCTL_SVM_ATTR_NO_ACCESS:
-            *update_mapping = true;
+            if (!p->xnack_enabled)
+                *update_mapping = true;
+

For NO_ACCESS we need to update the mapping to ensure that the PTEs are invalidated. For ACCESS or ACCESS_IN_PLACE we can leave it for a page fault.

Quoted 1 line

For NO_ACCESS, there is todo comment "TODO: unmap ranges from GPU that lost access", this should be handled by another patch. This patch only address that for XNACK ON, we can setup GPU mapping after page fault, without prefaulting and mapping to GPU for ACCESS or ACCESS_IN_PLACE.

OK. Then you'll probably change the same code again in that follow-up patch. This patch is

Quoted 13 lines

Regards,

Philip

Quoted 8 lines

Regards,
  Felix

Quoted 3 lines

              gpuidx = kfd_process_gpuidx_from_gpuid(p, attrs[i].value); if (attrs[i].type == KFD_IOCTL_SVM_ATTR_NO_ACCESS) {