Skip to contentSkip to conversation

Introduce GPU SVM and Xe SVM implementation

PATCH Superseded Matthew Brost 6 participants 129 messages Oct 16, 2024 – Dec 17, 2024 Page 1 of 3

Subsystems (8)

v2 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Matthew Brost

author maintainer
#0 lore raw

v2 cover Introduce GPU SVM and Xe SVM implementation

Version 2 of GPU SVM has been promoted to the proper series from an RFC. Thanks to everyone (especially Sima and Thomas) for their numerous reviews on revision 1 and for helping to address many design issues.

This version has been tested with IGT [1] on PVC, BMG, and LNL.

Major changes in v2:
- Dropped mmap write abuse
- core MM locking and retry loops instead of driver locking to avoid races
- Removed physical to virtual references
- Embedded structure/ops for drm_gpusvm_devmem
- Fixed mremap and fork issues
- Added DRM pagemap
- Included RFC documentation in the kernel doc

Known issues in v2:
- gpusvm/pagemap is still located in the Xe directory; it can be moved to the drm directory in the next revision, but will remain in Xe for streamlined testing for now
- drm_gpusvm_range_evict looks up VMA; it may be possible to use hmm_range_fault here instead, with some minor tweaks to HMM

Matt

[1] https://patchwork.freedesktop.org/series/137545/#rev3

Matthew Brost (26):
drm/xe: Retry BO allocation
mm/migrate: Add migrate_device_prepopulated_range mm/migrate: Trylock device page in do_swap_page drm/gpusvm: Add support for GPU Shared Virtual Memory drm/xe/uapi: Add DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATON flag drm/xe: Add SVM init / close / fini to faulting VMs drm/xe: Add SVM range invalidation
drm/gpuvm: Add DRM_GPUVA_OP_USER
drm/xe: Add (re)bind to SVM page fault handler drm/xe: Add SVM garbage collector
drm/xe: Add unbind to SVM garbage collector
drm/xe: Do not allow system allocator VMA unbind if the GPU has
bindings
drm/xe: Enable system allocator uAPI
drm/xe: Add migrate layer functions for SVM support drm/xe: Add SVM device memory mirroring
drm/xe: Add drm_gpusvm_devmem to xe_bo
drm/xe: Add GPUSVM devic memory copy vfunc functions drm/xe: Add Xe SVM populate_devmem_pfn vfunc
drm/xe: Add Xe SVM devmem_release vfunc
drm/xe: Add BO flags required for SVM
drm/xe: Add SVM VRAM migration
drm/xe: Basic SVM BO eviction
drm/xe: Add SVM debug
drm/xe: Add modparam for SVM notifier size
drm/xe: Add always_migrate_to_vram modparam
drm/doc: gpusvm: Add GPU SVM documentation

Thomas Hellström (3):
drm/pagemap: Add DRM pagemap
drm/xe: Add dma_addr res cursor
drm/xe: Add drm_pagemap ops to SVM

 Documentation/gpu/rfc/gpusvm.rst     |   70 +
 Documentation/gpu/rfc/index.rst      |    4 +
 drivers/gpu/drm/xe/Makefile          |    4 +-
 drivers/gpu/drm/xe/drm_gpusvm.c      | 2074 ++++++++++++++++++++++++++
 drivers/gpu/drm/xe/drm_gpusvm.h      |  447 ++++++
 drivers/gpu/drm/xe/drm_pagemap.h     |  103 ++
 drivers/gpu/drm/xe/xe_bo.c           |   43 +-
 drivers/gpu/drm/xe/xe_bo.h           |    2 +
 drivers/gpu/drm/xe/xe_bo_types.h     |    5 +
 drivers/gpu/drm/xe/xe_device_types.h |   15 +
 drivers/gpu/drm/xe/xe_gt_pagefault.c |   17 +-
 drivers/gpu/drm/xe/xe_migrate.c      |  149 ++
 drivers/gpu/drm/xe/xe_migrate.h      |   10 +
 drivers/gpu/drm/xe/xe_module.c       |    7 +
 drivers/gpu/drm/xe/xe_module.h       |    2 +
 drivers/gpu/drm/xe/xe_pt.c           |  362 ++++-
 drivers/gpu/drm/xe/xe_pt.h           |    3 +
 drivers/gpu/drm/xe/xe_pt_types.h     |    2 +
 drivers/gpu/drm/xe/xe_res_cursor.h   |  116 +-
 drivers/gpu/drm/xe/xe_svm.c          |  840 +++++++++++
 drivers/gpu/drm/xe/xe_svm.h          |   60 +
 drivers/gpu/drm/xe/xe_tile.c         |    5 +
 drivers/gpu/drm/xe/xe_vm.c           |  286 +++-
 drivers/gpu/drm/xe/xe_vm.h           |   15 +-
 drivers/gpu/drm/xe/xe_vm_types.h     |   44 +
 include/drm/drm_gpuvm.h              |    5 +
 include/linux/migrate.h              |    1 +
 include/uapi/drm/xe_drm.h            |   19 +-
 mm/memory.c                          |   13 +-
 mm/migrate_device.c                  |  104 +-
 30 files changed, 4695 insertions(+), 132 deletions(-)
 create mode 100644 Documentation/gpu/rfc/gpusvm.rst
 create mode 100644 drivers/gpu/drm/xe/drm_gpusvm.c
 create mode 100644 drivers/gpu/drm/xe/drm_gpusvm.h
 create mode 100644 drivers/gpu/drm/xe/drm_pagemap.h
 create mode 100644 drivers/gpu/drm/xe/xe_svm.c
 create mode 100644 drivers/gpu/drm/xe/xe_svm.h
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

PATCH v2 1/29 drm/xe: Retry BO allocation

TTM doesn't support fair eviction via WW locking, this mitigated in by using retry loops in exec and preempt rebind worker. Extend this retry loop to BO allocation. Once TTM supports fair eviction this patch can be reverted.

---
 drivers/gpu/drm/xe/xe_bo.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_bo.c +7 -1
index 5b232f2951b1..a02d63e322ae 100644
@@ -2011,6 +2011,7 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 	struct xe_file *xef = to_xe_file(file);
 	struct drm_xe_gem_create *args = data;
 	struct xe_vm *vm = NULL;
+	ktime_t end = 0;
 	struct xe_bo *bo;
 	unsigned int bo_flags;
 	u32 handle;
@@ -2083,11 +2084,14 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 		vm = xe_vm_lookup(xef, args->vm_id);
 		if (XE_IOCTL_DBG(xe, !vm))
 			return -ENOENT;
+	}
+
+retry:
+	if (vm) {
 		err = xe_vm_lock(vm, true);
 		if (err)
 			goto out_vm;
 	}
-
 	bo = xe_bo_create_user(xe, NULL, vm, args->size, args->cpu_caching,
 			       bo_flags);
 
@@ -2096,6 +2100,8 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 
 	if (IS_ERR(bo)) {
 		err = PTR_ERR(bo);
+		if (xe_vm_validate_should_retry(NULL, err, &end))
+			goto retry;
 		goto out_vm;
 	}
 
Signature · 1 line
2.34.1
author maintainer
#2 lore raw

PATCH v2 2/29 mm/migrate: Add migrate_device_prepopulated_range

In reply to Matthew Brost

Add migrate_device_prepoluated_range which prepares an array of pre-populated device pages for migration.

v2:
- s/migrate_device_vma_range/migrate_device_prepopulated_range
- Drop extra mmu invalidation (Vetter)

---
 include/linux/migrate.h |  1 +
 mm/migrate_device.c     | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 36 insertions(+)
include/linux/migrate.h +1 -0
index 002e49b2ebd9..9146ed39a2a3 100644
@@ -229,6 +229,7 @@ void migrate_vma_pages(struct migrate_vma *migrate);
 void migrate_vma_finalize(struct migrate_vma *migrate);
 int migrate_device_range(unsigned long *src_pfns, unsigned long start,
 			unsigned long npages);
+int migrate_device_prepopulated_range(unsigned long *src_pfns, unsigned long npages);
 void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns,
 			unsigned long npages);
 void migrate_device_finalize(unsigned long *src_pfns,
mm/migrate_device.c +35 -0
index 9cf26592ac93..f163c2131022 100644
@@ -924,6 +924,41 @@ int migrate_device_range(unsigned long *src_pfns, unsigned long start,
 }
 EXPORT_SYMBOL(migrate_device_range);
 
+/**
+ * migrate_device_prepopulated_range() - migrate device private pfns to normal memory.
+ * @src_pfns: pre-popluated array of source device private pfns to migrate.
+ * @npages: number of pages to migrate.
+ *
+ * Similar to migrate_device_range() but supports non-contiguous pre-popluated
+ * array of device pages to migrate.
+ */
+int migrate_device_prepopulated_range(unsigned long *src_pfns, unsigned long npages)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; i++) {
+		struct page *page = pfn_to_page(src_pfns[i]);
+
+		if (!get_page_unless_zero(page)) {
+			src_pfns[i] = 0;
+			continue;
+		}
+
+		if (!trylock_page(page)) {
+			src_pfns[i] = 0;
+			put_page(page);
+			continue;
+		}
+
+		src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;
+	}
+
+	migrate_device_unmap(src_pfns, npages, NULL);
+
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
 /*
  * Migrate a device coherent folio back to normal memory. The caller should have
  * a reference on folio which will be copied to the new folio if migration is
Signature · 1 line
2.34.1
#3 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

Quoted 2 lines · Matthew Brost

Add migrate_device_prepoluated_range which prepares an array of pre-populated device pages for migration.

It would be nice if the commit message could also include an explanation of why the existing migrate_device_range() is inadequate for your needs.

Quoted 30 lines

v2:
- s/migrate_device_vma_range/migrate_device_prepopulated_range
- Drop extra mmu invalidation (Vetter)

---
 include/linux/migrate.h |  1 +
 mm/migrate_device.c     | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 36 insertions(+)
include/linux/migrate.h +1 -0
index 002e49b2ebd9..9146ed39a2a3 100644
@@ -229,6 +229,7 @@ void migrate_vma_pages(struct migrate_vma *migrate);
 void migrate_vma_finalize(struct migrate_vma *migrate);
 int migrate_device_range(unsigned long *src_pfns, unsigned long start,
 			unsigned long npages);
+int migrate_device_prepopulated_range(unsigned long *src_pfns, unsigned long npages);
 void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns,
 			unsigned long npages);
 void migrate_device_finalize(unsigned long *src_pfns,
mm/migrate_device.c +0 -0
index 9cf26592ac93..f163c2131022 100644
@@ -924,6 +924,41 @@ int migrate_device_range(unsigned long *src_pfns, unsigned long start,
 }
 EXPORT_SYMBOL(migrate_device_range);

mm/migrate_device.c int migrate_device_range(unsigned long *src_pfns, unsigned long start, · line 935

Quoted 10 lines
mm/migrate_device.c +9 -0
 
+/**
+ * migrate_device_prepopulated_range() - migrate device private pfns to normal memory.
+ * @src_pfns: pre-popluated array of source device private pfns to migrate.
+ * @npages: number of pages to migrate.
+ *
+ * Similar to migrate_device_range() but supports non-contiguous pre-popluated
+ * array of device pages to migrate.
+ */
+int migrate_device_prepopulated_range(unsigned long *src_pfns, unsigned long npages)

I don't love the name, I think because it is quite long and makes me think of something complicated like prefaulting. Perhaps migrate_device_pfns()?

Quoted 8 lines
mm/migrate_device.c +8 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; i++) {
+		struct page *page = pfn_to_page(src_pfns[i]);
+
+		if (!get_page_unless_zero(page)) {
+			src_pfns[i] = 0;

mm/migrate_device.c int migrate_device_range(unsigned long *src_pfns, unsigned long start, · line 953

Quoted 10 lines
mm/migrate_device.c +10 -0
+			continue;
+		}
+
+		if (!trylock_page(page)) {
+			src_pfns[i] = 0;
+			put_page(page);
+			continue;
+		}
+
+		src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like migrate_device_range(). But more importantly this should be split out as a function that both migrate_device_range() and this function can call given this bit is identical.

Quoted 11 lines
mm/migrate_device.c +8 -0
+	}
+
+	migrate_device_unmap(src_pfns, npages, NULL);
+
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
 /*
  * Migrate a device coherent folio back to normal memory. The caller should have
  * a reference on folio which will be copied to the new folio if migration is
author maintainer
#4 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

Quoted 9 lines · Alistair Popple
Quoted 2 lines · Matthew Brost

Add migrate_device_prepoluated_range which prepares an array of pre-populated device pages for migration.

It would be nice if the commit message could also include an explanation of why the existing migrate_device_range() is inadequate for your needs.

Yea, my bad. It should explain this is required for non-contiguous device pages. I suppose I could always iterate for contiguous regions with migrate_device_range too if you think that is better.

Quoted 31 lines
Quoted 31 lines

v2:
- s/migrate_device_vma_range/migrate_device_prepopulated_range
- Drop extra mmu invalidation (Vetter)

---
 include/linux/migrate.h |  1 +
 mm/migrate_device.c     | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 36 insertions(+)
include/linux/migrate.h +1 -0
index 002e49b2ebd9..9146ed39a2a3 100644
@@ -229,6 +229,7 @@ void migrate_vma_pages(struct migrate_vma *migrate);
 void migrate_vma_finalize(struct migrate_vma *migrate);
 int migrate_device_range(unsigned long *src_pfns, unsigned long start,
 			unsigned long npages);
+int migrate_device_prepopulated_range(unsigned long *src_pfns, unsigned long npages);
 void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns,
 			unsigned long npages);
 void migrate_device_finalize(unsigned long *src_pfns,
mm/migrate_device.c +0 -0
index 9cf26592ac93..f163c2131022 100644
@@ -924,6 +924,41 @@ int migrate_device_range(unsigned long *src_pfns, unsigned long start,
 }
 EXPORT_SYMBOL(migrate_device_range);
 
Quoted 14 lines

mm/migrate_device.c int migrate_device_range(unsigned long *src_pfns, unsigned long start, · line 935

Quoted 9 lines
mm/migrate_device.c +9 -0
+/**
+ * migrate_device_prepopulated_range() - migrate device private pfns to normal memory.
+ * @src_pfns: pre-popluated array of source device private pfns to migrate.
+ * @npages: number of pages to migrate.
+ *
+ * Similar to migrate_device_range() but supports non-contiguous pre-popluated
+ * array of device pages to migrate.
+ */
+int migrate_device_prepopulated_range(unsigned long *src_pfns, unsigned long npages)

I don't love the name, I think because it is quite long and makes me think of something complicated like prefaulting. Perhaps migrate_device_pfns()?

Sure.

Quoted 15 lines
Quoted 15 lines
mm/migrate_device.c +15 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; i++) {
+		struct page *page = pfn_to_page(src_pfns[i]);
+
+		if (!get_page_unless_zero(page)) {
+			src_pfns[i] = 0;
+			continue;
+		}
+
+		if (!trylock_page(page)) {
+			src_pfns[i] = 0;
+			put_page(page);
+			continue;
Quoted 9 lines

mm/migrate_device.c int migrate_device_range(unsigned long *src_pfns, unsigned long start, · line 953

Quoted 3 lines
mm/migrate_device.c +3 -0
+		}
+
+		src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like migrate_device_range(). But more importantly this should be split out as a function that both migrate_device_range() and this function can call given this bit is identical.

Missed the folio conversion and agree a helper shared between this function and migrate_device_range would be a good idea. Let add that.

Matt

Quoted 12 lines
Quoted 11 lines
mm/migrate_device.c +8 -0
+	}
+
+	migrate_device_unmap(src_pfns, npages, NULL);
+
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
 /*
  * Migrate a device coherent folio back to normal memory. The caller should have
  * a reference on folio which will be copied to the new folio if migration is
author maintainer
#5 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

Quoted 88 lines · Matthew Brost
Quoted 9 lines · Alistair Popple
Quoted 2 lines · Matthew Brost

Add migrate_device_prepoluated_range which prepares an array of pre-populated device pages for migration.

It would be nice if the commit message could also include an explanation of why the existing migrate_device_range() is inadequate for your needs.

Yea, my bad. It should explain this is required for non-contiguous device pages. I suppose I could always iterate for contiguous regions with migrate_device_range too if you think that is better.

Quoted 31 lines
Quoted 31 lines

v2:
- s/migrate_device_vma_range/migrate_device_prepopulated_range
- Drop extra mmu invalidation (Vetter)

---
 include/linux/migrate.h |  1 +
 mm/migrate_device.c     | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 36 insertions(+)
include/linux/migrate.h +1 -0
index 002e49b2ebd9..9146ed39a2a3 100644
@@ -229,6 +229,7 @@ void migrate_vma_pages(struct migrate_vma *migrate);
 void migrate_vma_finalize(struct migrate_vma *migrate);
 int migrate_device_range(unsigned long *src_pfns, unsigned long start,
 			unsigned long npages);
+int migrate_device_prepopulated_range(unsigned long *src_pfns, unsigned long npages);
 void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns,
 			unsigned long npages);
 void migrate_device_finalize(unsigned long *src_pfns,
mm/migrate_device.c +0 -0
index 9cf26592ac93..f163c2131022 100644
@@ -924,6 +924,41 @@ int migrate_device_range(unsigned long *src_pfns, unsigned long start,
 }
 EXPORT_SYMBOL(migrate_device_range);
 
Quoted 14 lines

mm/migrate_device.c int migrate_device_range(unsigned long *src_pfns, unsigned long start, · line 935

Quoted 9 lines
mm/migrate_device.c +9 -0
+/**
+ * migrate_device_prepopulated_range() - migrate device private pfns to normal memory.
+ * @src_pfns: pre-popluated array of source device private pfns to migrate.
+ * @npages: number of pages to migrate.
+ *
+ * Similar to migrate_device_range() but supports non-contiguous pre-popluated
+ * array of device pages to migrate.
+ */
+int migrate_device_prepopulated_range(unsigned long *src_pfns, unsigned long npages)

I don't love the name, I think because it is quite long and makes me think of something complicated like prefaulting. Perhaps migrate_device_pfns()?

Sure.

Quoted 15 lines
Quoted 15 lines
mm/migrate_device.c +15 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; i++) {
+		struct page *page = pfn_to_page(src_pfns[i]);
+
+		if (!get_page_unless_zero(page)) {
+			src_pfns[i] = 0;
+			continue;
+		}
+
+		if (!trylock_page(page)) {
+			src_pfns[i] = 0;
+			put_page(page);
+			continue;
Quoted 9 lines

mm/migrate_device.c int migrate_device_range(unsigned long *src_pfns, unsigned long start, · line 953

Quoted 3 lines
mm/migrate_device.c +3 -0
+		}
+
+		src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like migrate_device_range(). But more importantly this should be split out as a function that both migrate_device_range() and this function can call given this bit is identical.

Quoted 3 lines

Missed the folio conversion and agree a helper shared between this function and migrate_device_range would be a good idea. Let add that.

Alistair,

Ok, I think now I want to go slightly different direction here to give GPUSVM a bit more control over several eviction scenarios.

What if I exported the helper discussed above, e.g.,

905 unsigned long migrate_device_pfn_lock(unsigned long pfn) 906 {
907 struct folio *folio;
908
909 folio = folio_get_nontail_page(pfn_to_page(pfn));
910 if (!folio)
911 return 0;
912
913 if (!folio_trylock(folio)) {
914 folio_put(folio);
915 return 0;
916 }
917
918 return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
919 }
920 EXPORT_SYMBOL(migrate_device_pfn_lock);

And then also export migrate_device_unmap.

The usage here would be let a driver collect the device pages in virtual address range via hmm_range_fault, lock device pages under notifier lock ensuring device pages are valid, drop the notifier lock and call migrate_device_unmap. Sima has strongly suggested avoiding a CPUVMA lookup during eviction cases and this would let me fixup drm_gpusvm_range_evict in [1] to avoid this.

It would also make the function exported in this patch unnecessary too as non-contiguous pfns can be setup on driver side via migrate_device_pfn_lock and then migrate_device_unmap can be called. This also another eviction usage in GPUSVM, see drm_gpusvm_evict_to_ram in [1].

Do you see an issue exporting migrate_device_pfn_lock, migrate_device_unmap?

Matt

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2

Quoted 14 lines

Matt

Quoted 12 lines
Quoted 11 lines
mm/migrate_device.c +8 -0
+	}
+
+	migrate_device_unmap(src_pfns, npages, NULL);
+
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
 /*
  * Migrate a device coherent folio back to normal memory. The caller should have
  * a reference on folio which will be copied to the new folio if migration is
#6 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

Quoted 2 lines · Matthew Brost
Quoted 1 line · Matthew Brost

On Wed, Oct 16, 2024 at 03:04:06PM +1100, Alistair Popple wrote:

[...]

Quoted 53 lines
Quoted 25 lines
Quoted 15 lines
Quoted 15 lines
mm/migrate_device.c +15 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; i++) {
+		struct page *page = pfn_to_page(src_pfns[i]);
+
+		if (!get_page_unless_zero(page)) {
+			src_pfns[i] = 0;
+			continue;
+		}
+
+		if (!trylock_page(page)) {
+			src_pfns[i] = 0;
+			put_page(page);
+			continue;
Quoted 9 lines

mm/migrate_device.c int migrate_device_range(unsigned long *src_pfns, unsigned long start, · line 953

Quoted 3 lines
mm/migrate_device.c +3 -0
+		}
+
+		src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like migrate_device_range(). But more importantly this should be split out as a function that both migrate_device_range() and this function can call given this bit is identical.

Quoted 3 lines

Missed the folio conversion and agree a helper shared between this function and migrate_device_range would be a good idea. Let add that.

Alistair,

Ok, I think now I want to go slightly different direction here to give GPUSVM a bit more control over several eviction scenarios.

What if I exported the helper discussed above, e.g.,

905 unsigned long migrate_device_pfn_lock(unsigned long pfn) 906 {
907 struct folio *folio;
908
909 folio = folio_get_nontail_page(pfn_to_page(pfn));
910 if (!folio)
911 return 0;
912
913 if (!folio_trylock(folio)) {
914 folio_put(folio);
915 return 0;
916 }
917
918 return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
919 }
920 EXPORT_SYMBOL(migrate_device_pfn_lock);

Quoted 6 lines

And then also export migrate_device_unmap.

The usage here would be let a driver collect the device pages in virtual address range via hmm_range_fault, lock device pages under notifier lock ensuring device pages are valid, drop the notifier lock and call migrate_device_unmap.

I'm still working through this series but that seems a bit dubious, the locking here is pretty subtle and easy to get wrong so seeing some code would help me a lot in understanding what you're suggesting.

Quoted 3 lines

Sima has strongly suggested avoiding a CPUVMA lookup during eviction cases and this would let me fixup drm_gpusvm_range_evict in [1] to avoid this.

That sounds reasonable but for context do you have a link to the comments/discussion on this? I couldn't readily find it, but I may have just missed it.

Quoted 8 lines

It would also make the function exported in this patch unnecessary too as non-contiguous pfns can be setup on driver side via migrate_device_pfn_lock and then migrate_device_unmap can be called. This also another eviction usage in GPUSVM, see drm_gpusvm_evict_to_ram in [1].

Do you see an issue exporting migrate_device_pfn_lock, migrate_device_unmap?

If there is a good justification for it I can't see a problem with exporting it. That said I don't really understand why you would want/need to split those steps up but I'll wait to see the code.

- Alistair

Quoted 18 lines

Matt

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2

Quoted 14 lines

Matt

Quoted 12 lines
Quoted 11 lines
mm/migrate_device.c +8 -0
+	}
+
+	migrate_device_unmap(src_pfns, npages, NULL);
+
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
 /*
  * Migrate a device coherent folio back to normal memory. The caller should have
  * a reference on folio which will be copied to the new folio if migration is
author maintainer
#7 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

Quoted 63 lines · Alistair Popple
Quoted 2 lines · Matthew Brost
Quoted 1 line · Matthew Brost

On Wed, Oct 16, 2024 at 03:04:06PM +1100, Alistair Popple wrote:

[...]

Quoted 55 lines
Quoted 25 lines
Quoted 15 lines
Quoted 15 lines
mm/migrate_device.c +15 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; i++) {
+		struct page *page = pfn_to_page(src_pfns[i]);
+
+		if (!get_page_unless_zero(page)) {
+			src_pfns[i] = 0;
+			continue;
+		}
+
+		if (!trylock_page(page)) {
+			src_pfns[i] = 0;
+			put_page(page);
+			continue;
Quoted 9 lines

mm/migrate_device.c int migrate_device_range(unsigned long *src_pfns, unsigned long start, · line 953

Quoted 3 lines
mm/migrate_device.c +3 -0
+		}
+
+		src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like migrate_device_range(). But more importantly this should be split out as a function that both migrate_device_range() and this function can call given this bit is identical.

Quoted 3 lines

Missed the folio conversion and agree a helper shared between this function and migrate_device_range would be a good idea. Let add that.

Alistair,

Ok, I think now I want to go slightly different direction here to give GPUSVM a bit more control over several eviction scenarios.

What if I exported the helper discussed above, e.g.,

905 unsigned long migrate_device_pfn_lock(unsigned long pfn) 906 {
907 struct folio *folio;
908
909 folio = folio_get_nontail_page(pfn_to_page(pfn));
910 if (!folio)
911 return 0;
912
913 if (!folio_trylock(folio)) {
914 folio_put(folio);
915 return 0;
916 }
917
918 return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
919 }
920 EXPORT_SYMBOL(migrate_device_pfn_lock);

And then also export migrate_device_unmap.

Quoted 9 lines
Quoted 4 lines

The usage here would be let a driver collect the device pages in virtual address range via hmm_range_fault, lock device pages under notifier lock ensuring device pages are valid, drop the notifier lock and call migrate_device_unmap.

I'm still working through this series but that seems a bit dubious, the locking here is pretty subtle and easy to get wrong so seeing some code would help me a lot in understanding what you're suggesting.

For sure locking in tricky, my mistake on not working through this before sending out the next rev but it came to mind after sending + regarding some late feedback from Thomas about using hmm for eviction [2]. His suggestion of using hmm_range_fault to trigger migration doesn't work for coherent pages, while something like below does.

[2] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1125461

Here is a snippet I have locally which seems to work.

2024 retry:
2025 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2026 hmm_range.hmm_pfns = src;
2027
2028 while (true) {

2029                 mmap_read_lock(mm);
2030                 err = hmm_range_fault(&hmm_range);
2031                 mmap_read_unlock(mm);
2032                 if (err == -EBUSY) {

2033 if (time_after(jiffies, timeout))
2034 break;
2035
2036 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2037 continue;
2038 }
2039 break;
2040 }
2041 if (err)
2042 goto err_put;
2043
2044 drm_gpusvm_notifier_lock(gpusvm);
2045 if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {

2046                 drm_gpusvm_notifier_unlock(gpusvm);
2047                 memset(src, 0, sizeof(*src) * npages);
2048                 goto retry;

2049 }
2050 for (i = 0; i < npages; ++i) {
2051 struct page *page = hmm_pfn_to_page(src[i]);
2052
2053 if (page && (is_device_private_page(page) ||
2054 is_device_coherent_page(page)) && page->zone_device_data)
2055 src[i] = src[i] & ~HMM_PFN_FLAGS;
2056 else
2057 src[i] = 0;
2058 if (src[i])
2059 src[i] = migrate_device_pfn_lock(src[i]);
2060 }
2061 drm_gpusvm_notifier_unlock(gpusvm);
2062
2063 migrate_device_unmap(src, npages, NULL);
...
2101 migrate_device_pages(src, dst, npages);
2102 migrate_device_finalize(src, dst, npages);

Quoted 8 lines
Quoted 3 lines

Sima has strongly suggested avoiding a CPUVMA lookup during eviction cases and this would let me fixup drm_gpusvm_range_evict in [1] to avoid this.

That sounds reasonable but for context do you have a link to the comments/discussion on this? I couldn't readily find it, but I may have just missed it.

See in [4], search for '2. eviction' comment from sima.

[3] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1110726
[4] https://lore.kernel.org/all/BYAPR11MB3159A304925168D8B6B4671292692@BYAPR11MB3159.namprd11.prod.outlook.com/T/#m89cd6a37778ba5271d5381ebeb03e1f963856a78

Quoted 6 lines
Quoted 6 lines

It would also make the function exported in this patch unnecessary too as non-contiguous pfns can be setup on driver side via migrate_device_pfn_lock and then migrate_device_unmap can be called. This also another eviction usage in GPUSVM, see drm_gpusvm_evict_to_ram in [1].

Quoted 7 lines
Quoted 2 lines

Do you see an issue exporting migrate_device_pfn_lock, migrate_device_unmap?

If there is a good justification for it I can't see a problem with exporting it. That said I don't really understand why you would want/need to split those steps up but I'll wait to see the code.

It is so the device pages returned from hmm_range_fault, which are only guaranteed to be valid under the notifier lock + a seqno check, to be locked and ref taken for migration. migrate_device_unmap() can trigger a MMU invalidation which takes the notifier lock thus calling the function which combines migrate_device_pfn_lock + migrate_device_unmap deadlocks.

I think this flow makes sense and agree in general this likely better than looking at a CPUVMA.

Matt

Quoted 21 lines

- Alistair

Quoted 18 lines

Matt

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2

Quoted 14 lines

Matt

Quoted 12 lines
Quoted 11 lines
mm/migrate_device.c +8 -0
+	}
+
+	migrate_device_unmap(src_pfns, npages, NULL);
+
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
 /*
  * Migrate a device coherent folio back to normal memory. The caller should have
  * a reference on folio which will be copied to the new folio if migration is
#8 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

Quoted 109 lines · Matthew Brost
Quoted 63 lines · Alistair Popple
Quoted 2 lines · Matthew Brost
Quoted 1 line · Matthew Brost

On Wed, Oct 16, 2024 at 03:04:06PM +1100, Alistair Popple wrote:

[...]

Quoted 55 lines
Quoted 25 lines
Quoted 15 lines
Quoted 15 lines
mm/migrate_device.c +15 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; i++) {
+		struct page *page = pfn_to_page(src_pfns[i]);
+
+		if (!get_page_unless_zero(page)) {
+			src_pfns[i] = 0;
+			continue;
+		}
+
+		if (!trylock_page(page)) {
+			src_pfns[i] = 0;
+			put_page(page);
+			continue;
Quoted 9 lines

mm/migrate_device.c int migrate_device_range(unsigned long *src_pfns, unsigned long start, · line 953

Quoted 3 lines
mm/migrate_device.c +3 -0
+		}
+
+		src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like migrate_device_range(). But more importantly this should be split out as a function that both migrate_device_range() and this function can call given this bit is identical.

Quoted 3 lines

Missed the folio conversion and agree a helper shared between this function and migrate_device_range would be a good idea. Let add that.

Alistair,

Ok, I think now I want to go slightly different direction here to give GPUSVM a bit more control over several eviction scenarios.

What if I exported the helper discussed above, e.g.,

905 unsigned long migrate_device_pfn_lock(unsigned long pfn) 906 {
907 struct folio *folio;
908
909 folio = folio_get_nontail_page(pfn_to_page(pfn));
910 if (!folio)
911 return 0;
912
913 if (!folio_trylock(folio)) {
914 folio_put(folio);
915 return 0;
916 }
917
918 return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
919 }
920 EXPORT_SYMBOL(migrate_device_pfn_lock);

And then also export migrate_device_unmap.

Quoted 9 lines
Quoted 4 lines

The usage here would be let a driver collect the device pages in virtual address range via hmm_range_fault, lock device pages under notifier lock ensuring device pages are valid, drop the notifier lock and call migrate_device_unmap.

I'm still working through this series but that seems a bit dubious, the locking here is pretty subtle and easy to get wrong so seeing some code would help me a lot in understanding what you're suggesting.

For sure locking in tricky, my mistake on not working through this before sending out the next rev but it came to mind after sending + regarding some late feedback from Thomas about using hmm for eviction [2]. His suggestion of using hmm_range_fault to trigger migration doesn't work for coherent pages, while something like below does.

[2] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1125461

Here is a snippet I have locally which seems to work.

2024 retry:
2025 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2026 hmm_range.hmm_pfns = src;
2027
2028 while (true) {

2029                 mmap_read_lock(mm);
2030                 err = hmm_range_fault(&hmm_range);
2031                 mmap_read_unlock(mm);
2032                 if (err == -EBUSY) {

2033 if (time_after(jiffies, timeout))
2034 break;
2035
2036 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2037 continue;
2038 }
2039 break;
2040 }
2041 if (err)
2042 goto err_put;
2043
2044 drm_gpusvm_notifier_lock(gpusvm);
2045 if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {

2046                 drm_gpusvm_notifier_unlock(gpusvm);
2047                 memset(src, 0, sizeof(*src) * npages);
2048                 goto retry;
Quoted 13 lines

2049 }
2050 for (i = 0; i < npages; ++i) {
2051 struct page *page = hmm_pfn_to_page(src[i]);
2052
2053 if (page && (is_device_private_page(page) ||
2054 is_device_coherent_page(page)) && page->zone_device_data)
2055 src[i] = src[i] & ~HMM_PFN_FLAGS;
2056 else
2057 src[i] = 0;
2058 if (src[i])
2059 src[i] = migrate_device_pfn_lock(src[i]);
2060 }
2061 drm_gpusvm_notifier_unlock(gpusvm);

Practically for eviction isn't this much the same as calling migrate_vma_setup()? And also for eviction as Sima mentioned you probably shouldn't be looking at mm/vma structs.

Quoted 10 lines

2063 migrate_device_unmap(src, npages, NULL);
...
2101 migrate_device_pages(src, dst, npages);
2102 migrate_device_finalize(src, dst, npages);

Quoted 4 lines
Quoted 3 lines

Sima has strongly suggested avoiding a CPUVMA lookup during eviction cases and this would let me fixup drm_gpusvm_range_evict in [1] to avoid this.

Quoted 6 lines
Quoted 4 lines

That sounds reasonable but for context do you have a link to the comments/discussion on this? I couldn't readily find it, but I may have just missed it.

See in [4], search for '2. eviction' comment from sima.

Thanks for pointing that out. For reference here's Sima's comment:

Quoted 9 lines

2. eviction

Requirements much like migrate_to_ram, because otherwise we break the migration gurantee:

- Only looking at physical memory datastructures and locks, no looking at mm/vma structs or relying on those being locked. We rely entirely on reverse maps from try_to_migrate to find all the mappings on both cpu and gpu side (cpu only zone device swap or migration pte entries ofc).

I also very much agree with this. That's basically why I added migrate_device_range(), so that we can forcibly evict pages when the driver needs them freed (eg. driver unload, low memory, etc.). In general it is impossible to guarantee eviction og all pages using just hmm_range_fault().

Quoted 17 lines

[3] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1110726
[4] https://lore.kernel.org/all/BYAPR11MB3159A304925168D8B6B4671292692@BYAPR11MB3159.namprd11.prod.outlook.com/T/#m89cd6a37778ba5271d5381ebeb03e1f963856a78

Quoted 6 lines
Quoted 6 lines

It would also make the function exported in this patch unnecessary too as non-contiguous pfns can be setup on driver side via migrate_device_pfn_lock and then migrate_device_unmap can be called. This also another eviction usage in GPUSVM, see drm_gpusvm_evict_to_ram in [1].

Quoted 7 lines
Quoted 2 lines

Do you see an issue exporting migrate_device_pfn_lock, migrate_device_unmap?

If there is a good justification for it I can't see a problem with exporting it. That said I don't really understand why you would want/need to split those steps up but I'll wait to see the code.

Quoted 8 lines

It is so the device pages returned from hmm_range_fault, which are only guaranteed to be valid under the notifier lock + a seqno check, to be locked and ref taken for migration. migrate_device_unmap() can trigger a MMU invalidation which takes the notifier lock thus calling the function which combines migrate_device_pfn_lock + migrate_device_unmap deadlocks.

I think this flow makes sense and agree in general this likely better than looking at a CPUVMA.

I'm still a bit confused about what is better with this flow if you are still calling hmm_range_fault(). How is it better than just calling migrate_vma_setup()? Obviously it will fault the pages in, but it seems we're talking about eviction here so I don't understand why that would be relevant. And hmm_range_fault() still requires the VMA, although I need to look at the patches more closely, probably CPUVMA is a DRM specific concept?

Thanks.

- Alistair

Quoted 23 lines

Matt

Quoted 21 lines

- Alistair

Quoted 18 lines

Matt

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2

Quoted 14 lines

Matt

Quoted 12 lines
Quoted 11 lines
mm/migrate_device.c +8 -0
+	}
+
+	migrate_device_unmap(src_pfns, npages, NULL);
+
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
 /*
  * Migrate a device coherent folio back to normal memory. The caller should have
  * a reference on folio which will be copied to the new folio if migration is
author maintainer
#9 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

Quoted 126 lines · Alistair Popple
Quoted 109 lines · Matthew Brost
Quoted 63 lines · Alistair Popple
Quoted 2 lines · Matthew Brost
Quoted 1 line · Matthew Brost

On Wed, Oct 16, 2024 at 03:04:06PM +1100, Alistair Popple wrote:

[...]

Quoted 55 lines
Quoted 25 lines
Quoted 15 lines
Quoted 15 lines
(quoted diff) +15 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; i++) {
+		struct page *page = pfn_to_page(src_pfns[i]);
+
+		if (!get_page_unless_zero(page)) {
+			src_pfns[i] = 0;
+			continue;
+		}
+
+		if (!trylock_page(page)) {
+			src_pfns[i] = 0;
+			put_page(page);
+			continue;
Quoted 9 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+		}
+
+		src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like migrate_device_range(). But more importantly this should be split out as a function that both migrate_device_range() and this function can call given this bit is identical.

Quoted 3 lines

Missed the folio conversion and agree a helper shared between this function and migrate_device_range would be a good idea. Let add that.

Alistair,

Ok, I think now I want to go slightly different direction here to give GPUSVM a bit more control over several eviction scenarios.

What if I exported the helper discussed above, e.g.,

905 unsigned long migrate_device_pfn_lock(unsigned long pfn) 906 {
907 struct folio *folio;
908
909 folio = folio_get_nontail_page(pfn_to_page(pfn));
910 if (!folio)
911 return 0;
912
913 if (!folio_trylock(folio)) {
914 folio_put(folio);
915 return 0;
916 }
917
918 return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
919 }
920 EXPORT_SYMBOL(migrate_device_pfn_lock);

And then also export migrate_device_unmap.

Quoted 9 lines
Quoted 4 lines

The usage here would be let a driver collect the device pages in virtual address range via hmm_range_fault, lock device pages under notifier lock ensuring device pages are valid, drop the notifier lock and call migrate_device_unmap.

I'm still working through this series but that seems a bit dubious, the locking here is pretty subtle and easy to get wrong so seeing some code would help me a lot in understanding what you're suggesting.

For sure locking in tricky, my mistake on not working through this before sending out the next rev but it came to mind after sending + regarding some late feedback from Thomas about using hmm for eviction [2]. His suggestion of using hmm_range_fault to trigger migration doesn't work for coherent pages, while something like below does.

[2] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1125461

Here is a snippet I have locally which seems to work.

2024 retry:
2025 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2026 hmm_range.hmm_pfns = src;
2027
2028 while (true) {

2029                 mmap_read_lock(mm);
2030                 err = hmm_range_fault(&hmm_range);
2031                 mmap_read_unlock(mm);
2032                 if (err == -EBUSY) {

2033 if (time_after(jiffies, timeout))
2034 break;
2035
2036 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2037 continue;
2038 }
2039 break;
2040 }
2041 if (err)
2042 goto err_put;
2043
2044 drm_gpusvm_notifier_lock(gpusvm);
2045 if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {

2046                 drm_gpusvm_notifier_unlock(gpusvm);
2047                 memset(src, 0, sizeof(*src) * npages);
2048                 goto retry;
Quoted 13 lines

2049 }
2050 for (i = 0; i < npages; ++i) {
2051 struct page *page = hmm_pfn_to_page(src[i]);
2052
2053 if (page && (is_device_private_page(page) ||
2054 is_device_coherent_page(page)) && page->zone_device_data)
2055 src[i] = src[i] & ~HMM_PFN_FLAGS;
2056 else
2057 src[i] = 0;
2058 if (src[i])
2059 src[i] = migrate_device_pfn_lock(src[i]);
2060 }
2061 drm_gpusvm_notifier_unlock(gpusvm);

Quoted 4 lines

Practically for eviction isn't this much the same as calling migrate_vma_setup()? And also for eviction as Sima mentioned you probably shouldn't be looking at mm/vma structs.

hmm_range_fault is just collecting the pages, internally I suppose it does look at a VMA (struct vm_area_struct) but I think the point is drivers should not be looking at VMA here.

Quoted 29 lines
Quoted 10 lines

2063 migrate_device_unmap(src, npages, NULL);
...
2101 migrate_device_pages(src, dst, npages);
2102 migrate_device_finalize(src, dst, npages);

Quoted 4 lines
Quoted 3 lines

Sima has strongly suggested avoiding a CPUVMA lookup during eviction cases and this would let me fixup drm_gpusvm_range_evict in [1] to avoid this.

Quoted 6 lines
Quoted 4 lines

That sounds reasonable but for context do you have a link to the comments/discussion on this? I couldn't readily find it, but I may have just missed it.

See in [4], search for '2. eviction' comment from sima.

Thanks for pointing that out. For reference here's Sima's comment:

Quoted 9 lines

2. eviction

Requirements much like migrate_to_ram, because otherwise we break the migration gurantee:

- Only looking at physical memory datastructures and locks, no looking at mm/vma structs or relying on those being locked. We rely entirely on reverse maps from try_to_migrate to find all the mappings on both cpu and gpu side (cpu only zone device swap or migration pte entries ofc).

Quoted 6 lines

I also very much agree with this. That's basically why I added migrate_device_range(), so that we can forcibly evict pages when the driver needs them freed (eg. driver unload, low memory, etc.). In general it is impossible to guarantee eviction og all pages using just hmm_range_fault().

In this code path we don't have device pages available, hence the purposed collection via hmm_range_fault.

Quoted 23 lines
Quoted 23 lines

[3] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1110726
[4] https://lore.kernel.org/all/BYAPR11MB3159A304925168D8B6B4671292692@BYAPR11MB3159.namprd11.prod.outlook.com/T/#m89cd6a37778ba5271d5381ebeb03e1f963856a78

Quoted 6 lines
Quoted 6 lines

It would also make the function exported in this patch unnecessary too as non-contiguous pfns can be setup on driver side via migrate_device_pfn_lock and then migrate_device_unmap can be called. This also another eviction usage in GPUSVM, see drm_gpusvm_evict_to_ram in [1].

Quoted 7 lines
Quoted 2 lines

Do you see an issue exporting migrate_device_pfn_lock, migrate_device_unmap?

If there is a good justification for it I can't see a problem with exporting it. That said I don't really understand why you would want/need to split those steps up but I'll wait to see the code.

It is so the device pages returned from hmm_range_fault, which are only guaranteed to be valid under the notifier lock + a seqno check, to be locked and ref taken for migration. migrate_device_unmap() can trigger a MMU invalidation which takes the notifier lock thus calling the function which combines migrate_device_pfn_lock + migrate_device_unmap deadlocks.

Quoted 6 lines
Quoted 2 lines

I think this flow makes sense and agree in general this likely better than looking at a CPUVMA.

I'm still a bit confused about what is better with this flow if you are still calling hmm_range_fault(). How is it better than just calling migrate_vma_setup()? Obviously it will fault the pages in, but it seems

The code in rev2 calls migrate_vma_setup but the requires a struct vm_area_struct argument whereas hmm_range_fault does not.

Quoted 3 lines

we're talking about eviction here so I don't understand why that would be relevant. And hmm_range_fault() still requires the VMA, although I need to look at the patches more closely, probably CPUVMA is a DRM

'hmm_range_fault() still requires the VMA' internal yes, but again not as argument. This is about avoiding a driver side lookup of the VMA.

CPUVMA == struct vm_area_struct in this email.

Matt

Quoted 30 lines

specific concept?

Thanks.

- Alistair

Quoted 23 lines

Matt

Quoted 21 lines

- Alistair

Quoted 18 lines

Matt

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2

Quoted 14 lines

Matt

Quoted 12 lines
Quoted 11 lines
(quoted diff) +8 -0
+	}
+
+	migrate_device_unmap(src_pfns, npages, NULL);
+
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
 /*
  * Migrate a device coherent folio back to normal memory. The caller should have
  * a reference on folio which will be copied to the new folio if migration is
#10 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

Quoted 127 lines · Matthew Brost
Quoted 126 lines · Alistair Popple
Quoted 122 lines · Matthew Brost
Quoted 63 lines · Alistair Popple
Quoted 2 lines · Matthew Brost
Quoted 1 line · Matthew Brost

On Wed, Oct 16, 2024 at 03:04:06PM +1100, Alistair Popple wrote:

[...]

Quoted 55 lines
Quoted 25 lines
Quoted 15 lines
Quoted 15 lines
(quoted diff) +15 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; i++) {
+		struct page *page = pfn_to_page(src_pfns[i]);
+
+		if (!get_page_unless_zero(page)) {
+			src_pfns[i] = 0;
+			continue;
+		}
+
+		if (!trylock_page(page)) {
+			src_pfns[i] = 0;
+			put_page(page);
+			continue;
Quoted 9 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+		}
+
+		src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like migrate_device_range(). But more importantly this should be split out as a function that both migrate_device_range() and this function can call given this bit is identical.

Quoted 3 lines

Missed the folio conversion and agree a helper shared between this function and migrate_device_range would be a good idea. Let add that.

Alistair,

Ok, I think now I want to go slightly different direction here to give GPUSVM a bit more control over several eviction scenarios.

What if I exported the helper discussed above, e.g.,

905 unsigned long migrate_device_pfn_lock(unsigned long pfn) 906 {
907 struct folio *folio;
908
909 folio = folio_get_nontail_page(pfn_to_page(pfn));
910 if (!folio)
911 return 0;
912
913 if (!folio_trylock(folio)) {
914 folio_put(folio);
915 return 0;
916 }
917
918 return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
919 }
920 EXPORT_SYMBOL(migrate_device_pfn_lock);

And then also export migrate_device_unmap.

Quoted 9 lines
Quoted 4 lines

The usage here would be let a driver collect the device pages in virtual address range via hmm_range_fault, lock device pages under notifier lock ensuring device pages are valid, drop the notifier lock and call migrate_device_unmap.

I'm still working through this series but that seems a bit dubious, the locking here is pretty subtle and easy to get wrong so seeing some code would help me a lot in understanding what you're suggesting.

For sure locking in tricky, my mistake on not working through this before sending out the next rev but it came to mind after sending + regarding some late feedback from Thomas about using hmm for eviction [2]. His suggestion of using hmm_range_fault to trigger migration doesn't work for coherent pages, while something like below does.

[2] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1125461

Here is a snippet I have locally which seems to work.

2024 retry:
2025 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2026 hmm_range.hmm_pfns = src;
2027
2028 while (true) {

2029                 mmap_read_lock(mm);
2030                 err = hmm_range_fault(&hmm_range);
2031                 mmap_read_unlock(mm);
2032                 if (err == -EBUSY) {

2033 if (time_after(jiffies, timeout))
2034 break;
2035
2036 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2037 continue;
2038 }
2039 break;
2040 }
2041 if (err)
2042 goto err_put;
2043
2044 drm_gpusvm_notifier_lock(gpusvm);
2045 if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {

2046                 drm_gpusvm_notifier_unlock(gpusvm);
2047                 memset(src, 0, sizeof(*src) * npages);
2048                 goto retry;

2049 }
2050 for (i = 0; i < npages; ++i) {
2051 struct page *page = hmm_pfn_to_page(src[i]);
2052
2053 if (page && (is_device_private_page(page) ||
2054 is_device_coherent_page(page)) && page->zone_device_data)
2055 src[i] = src[i] & ~HMM_PFN_FLAGS;
2056 else
2057 src[i] = 0;
2058 if (src[i])
2059 src[i] = migrate_device_pfn_lock(src[i]);
2060 }
2061 drm_gpusvm_notifier_unlock(gpusvm);

Quoted 8 lines
Quoted 4 lines

Practically for eviction isn't this much the same as calling migrate_vma_setup()? And also for eviction as Sima mentioned you probably shouldn't be looking at mm/vma structs.

hmm_range_fault is just collecting the pages, internally I suppose it does look at a VMA (struct vm_area_struct) but I think the point is drivers should not be looking at VMA here.

migrate_vma_setup() is designed to be called by drivers and needs a vma, so in general I don't see a problem with drivers looking up vma's. The problem arises specifically for eviction and whether or not that happens in the driver or hmm_range_fault() is pretty irrelevant IMHO for the issues there (see below).

Quoted 36 lines
Quoted 29 lines
Quoted 10 lines

2063 migrate_device_unmap(src, npages, NULL);
...
2101 migrate_device_pages(src, dst, npages);
2102 migrate_device_finalize(src, dst, npages);

Quoted 4 lines
Quoted 3 lines

Sima has strongly suggested avoiding a CPUVMA lookup during eviction cases and this would let me fixup drm_gpusvm_range_evict in [1] to avoid this.

Quoted 6 lines
Quoted 4 lines

That sounds reasonable but for context do you have a link to the comments/discussion on this? I couldn't readily find it, but I may have just missed it.

See in [4], search for '2. eviction' comment from sima.

Thanks for pointing that out. For reference here's Sima's comment:

Quoted 9 lines

2. eviction

Requirements much like migrate_to_ram, because otherwise we break the migration gurantee:

- Only looking at physical memory datastructures and locks, no looking at mm/vma structs or relying on those being locked. We rely entirely on reverse maps from try_to_migrate to find all the mappings on both cpu and gpu side (cpu only zone device swap or migration pte entries ofc).

Quoted 6 lines

I also very much agree with this. That's basically why I added migrate_device_range(), so that we can forcibly evict pages when the driver needs them freed (eg. driver unload, low memory, etc.). In general it is impossible to guarantee eviction og all pages using just hmm_range_fault().

Quoted 2 lines

In this code path we don't have device pages available, hence the purposed collection via hmm_range_fault.

Why don't you have the pfns requiring eviction available? I need to read this series in more depth, but generally hmm_range_fault() can't gurantee you will find every device page.

Quoted 26 lines
Quoted 26 lines
Quoted 25 lines

[3] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1110726
[4] https://lore.kernel.org/all/BYAPR11MB3159A304925168D8B6B4671292692@BYAPR11MB3159.namprd11.prod.outlook.com/T/#m89cd6a37778ba5271d5381ebeb03e1f963856a78

Quoted 6 lines
Quoted 6 lines

It would also make the function exported in this patch unnecessary too as non-contiguous pfns can be setup on driver side via migrate_device_pfn_lock and then migrate_device_unmap can be called. This also another eviction usage in GPUSVM, see drm_gpusvm_evict_to_ram in [1].

Quoted 7 lines
Quoted 2 lines

Do you see an issue exporting migrate_device_pfn_lock, migrate_device_unmap?

If there is a good justification for it I can't see a problem with exporting it. That said I don't really understand why you would want/need to split those steps up but I'll wait to see the code.

It is so the device pages returned from hmm_range_fault, which are only guaranteed to be valid under the notifier lock + a seqno check, to be locked and ref taken for migration. migrate_device_unmap() can trigger a MMU invalidation which takes the notifier lock thus calling the function which combines migrate_device_pfn_lock + migrate_device_unmap deadlocks.

I think this flow makes sense and agree in general this likely better than looking at a CPUVMA.

Quoted 6 lines
Quoted 3 lines

I'm still a bit confused about what is better with this flow if you are still calling hmm_range_fault(). How is it better than just calling migrate_vma_setup()? Obviously it will fault the pages in, but it seems

The code in rev2 calls migrate_vma_setup but the requires a struct vm_area_struct argument whereas hmm_range_fault does not.

I'm not sure that's a good enough justfication because the problem isn't whether you're looking up vma's in driver code or mm code. The problem is you are looking up vma's at all and all that goes with that (mainly taking mmap lock, etc.)

And for eviction hmm_range_fault() won't even find all the pages because their virtual address may have changed - consider what happens in cases of mremap(), fork(), etc. So eviction really needs physical pages (pfn's), not virtual addresses.

Quoted 8 lines
Quoted 3 lines

we're talking about eviction here so I don't understand why that would be relevant. And hmm_range_fault() still requires the VMA, although I need to look at the patches more closely, probably CPUVMA is a DRM

'hmm_range_fault() still requires the VMA' internal yes, but again not as argument. This is about avoiding a driver side lookup of the VMA.

CPUVMA == struct vm_area_struct in this email.

Thanks for the clarification.

- Alistair

Quoted 32 lines

Matt

Quoted 30 lines

specific concept?

Thanks.

- Alistair

Quoted 23 lines

Matt

Quoted 21 lines

- Alistair

Quoted 18 lines

Matt

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2

Quoted 14 lines

Matt

Quoted 12 lines
Quoted 11 lines
(quoted diff) +8 -0
+	}
+
+	migrate_device_unmap(src_pfns, npages, NULL);
+
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
 /*
  * Migrate a device coherent folio back to normal memory. The caller should have
  * a reference on folio which will be copied to the new folio if migration is
author maintainer
#11 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

Quoted 139 lines · Alistair Popple
Quoted 127 lines · Matthew Brost
Quoted 126 lines · Alistair Popple
Quoted 122 lines · Matthew Brost
Quoted 63 lines · Alistair Popple
Quoted 2 lines · Matthew Brost
Quoted 1 line · Matthew Brost

On Wed, Oct 16, 2024 at 03:04:06PM +1100, Alistair Popple wrote:

[...]

Quoted 55 lines
Quoted 25 lines
Quoted 15 lines
Quoted 15 lines
(quoted diff) +15 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; i++) {
+		struct page *page = pfn_to_page(src_pfns[i]);
+
+		if (!get_page_unless_zero(page)) {
+			src_pfns[i] = 0;
+			continue;
+		}
+
+		if (!trylock_page(page)) {
+			src_pfns[i] = 0;
+			put_page(page);
+			continue;
Quoted 9 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+		}
+
+		src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like migrate_device_range(). But more importantly this should be split out as a function that both migrate_device_range() and this function can call given this bit is identical.

Quoted 3 lines

Missed the folio conversion and agree a helper shared between this function and migrate_device_range would be a good idea. Let add that.

Alistair,

Ok, I think now I want to go slightly different direction here to give GPUSVM a bit more control over several eviction scenarios.

What if I exported the helper discussed above, e.g.,

905 unsigned long migrate_device_pfn_lock(unsigned long pfn) 906 {
907 struct folio *folio;
908
909 folio = folio_get_nontail_page(pfn_to_page(pfn));
910 if (!folio)
911 return 0;
912
913 if (!folio_trylock(folio)) {
914 folio_put(folio);
915 return 0;
916 }
917
918 return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
919 }
920 EXPORT_SYMBOL(migrate_device_pfn_lock);

And then also export migrate_device_unmap.

Quoted 9 lines
Quoted 4 lines

The usage here would be let a driver collect the device pages in virtual address range via hmm_range_fault, lock device pages under notifier lock ensuring device pages are valid, drop the notifier lock and call migrate_device_unmap.

I'm still working through this series but that seems a bit dubious, the locking here is pretty subtle and easy to get wrong so seeing some code would help me a lot in understanding what you're suggesting.

For sure locking in tricky, my mistake on not working through this before sending out the next rev but it came to mind after sending + regarding some late feedback from Thomas about using hmm for eviction [2]. His suggestion of using hmm_range_fault to trigger migration doesn't work for coherent pages, while something like below does.

[2] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1125461

Here is a snippet I have locally which seems to work.

2024 retry:
2025 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2026 hmm_range.hmm_pfns = src;
2027
2028 while (true) {

2029                 mmap_read_lock(mm);
2030                 err = hmm_range_fault(&hmm_range);
2031                 mmap_read_unlock(mm);
2032                 if (err == -EBUSY) {

2033 if (time_after(jiffies, timeout))
2034 break;
2035
2036 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2037 continue;
2038 }
2039 break;
2040 }
2041 if (err)
2042 goto err_put;
2043
2044 drm_gpusvm_notifier_lock(gpusvm);
2045 if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {

2046                 drm_gpusvm_notifier_unlock(gpusvm);
2047                 memset(src, 0, sizeof(*src) * npages);
2048                 goto retry;

2049 }
2050 for (i = 0; i < npages; ++i) {
2051 struct page *page = hmm_pfn_to_page(src[i]);
2052
2053 if (page && (is_device_private_page(page) ||
2054 is_device_coherent_page(page)) && page->zone_device_data)
2055 src[i] = src[i] & ~HMM_PFN_FLAGS;
2056 else
2057 src[i] = 0;
2058 if (src[i])
2059 src[i] = migrate_device_pfn_lock(src[i]);
2060 }
2061 drm_gpusvm_notifier_unlock(gpusvm);

Quoted 8 lines
Quoted 4 lines

Practically for eviction isn't this much the same as calling migrate_vma_setup()? And also for eviction as Sima mentioned you probably shouldn't be looking at mm/vma structs.

hmm_range_fault is just collecting the pages, internally I suppose it does look at a VMA (struct vm_area_struct) but I think the point is drivers should not be looking at VMA here.

Quoted 6 lines

migrate_vma_setup() is designed to be called by drivers and needs a vma, so in general I don't see a problem with drivers looking up vma's. The problem arises specifically for eviction and whether or not that happens in the driver or hmm_range_fault() is pretty irrelevant IMHO for the issues there (see below).

Ok, if you think it ok for drivers to lookup the VMA then purposed exporting of migrate_device_pfn_lock & migrate_device_unmap is not needed, rather just the original function exported in the this patch.

More below too.

Quoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 10 lines

2063 migrate_device_unmap(src, npages, NULL);
...
2101 migrate_device_pages(src, dst, npages);
2102 migrate_device_finalize(src, dst, npages);

Quoted 4 lines
Quoted 3 lines

Sima has strongly suggested avoiding a CPUVMA lookup during eviction cases and this would let me fixup drm_gpusvm_range_evict in [1] to avoid this.

Quoted 6 lines
Quoted 4 lines

That sounds reasonable but for context do you have a link to the comments/discussion on this? I couldn't readily find it, but I may have just missed it.

See in [4], search for '2. eviction' comment from sima.

Thanks for pointing that out. For reference here's Sima's comment:

Quoted 9 lines

2. eviction

Requirements much like migrate_to_ram, because otherwise we break the migration gurantee:

- Only looking at physical memory datastructures and locks, no looking at mm/vma structs or relying on those being locked. We rely entirely on reverse maps from try_to_migrate to find all the mappings on both cpu and gpu side (cpu only zone device swap or migration pte entries ofc).

I also very much agree with this. That's basically why I added migrate_device_range(), so that we can forcibly evict pages when the driver needs them freed (eg. driver unload, low memory, etc.). In general it is impossible to guarantee eviction og all pages using just hmm_range_fault().

Quoted 7 lines
Quoted 2 lines

In this code path we don't have device pages available, hence the purposed collection via hmm_range_fault.

Why don't you have the pfns requiring eviction available? I need to read this series in more depth, but generally hmm_range_fault() can't gurantee you will find every device page.

There are two cases for eviction in my series:

1. TTM decides it needs to move memory. This calls drm_gpusvm_evict_to_ram. In this case the device pfns are available directly from drm_gpusvm_devmem so the migrate_device_* calls be used here albiet with the new function added in this patch as device pfns may be non-contiguous.

2. An inconsistent state for VA range occurs (mixed system and device pages, partial unmap of a range, etc...). Here we want to evict the range ram to make the state consistent. No device pages are available due to an intentional disconnect between a virtual range and physical drm_gpusvm_devmem, thus the device pages have to be looked up. This the function drm_gpusvm_range_evict. Based on what you tell me, it likely is fine the way originally coded in v2 (vma lookup + migrate_vma_*) vs using hmm_range_fault like I have suggested here.

Note #2 may be removed or unnecessary at some point if we decide to add support for ininconsistent state in GPU SVM and Xe. Keeping it simple for now though. See 'Ranges with mixed system and device pages' in [5].

[5] https://patchwork.freedesktop.org/patch/619819/?series=137870&rev=2

Quoted 38 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines

[3] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1110726
[4] https://lore.kernel.org/all/BYAPR11MB3159A304925168D8B6B4671292692@BYAPR11MB3159.namprd11.prod.outlook.com/T/#m89cd6a37778ba5271d5381ebeb03e1f963856a78

Quoted 6 lines
Quoted 6 lines

It would also make the function exported in this patch unnecessary too as non-contiguous pfns can be setup on driver side via migrate_device_pfn_lock and then migrate_device_unmap can be called. This also another eviction usage in GPUSVM, see drm_gpusvm_evict_to_ram in [1].

Quoted 7 lines
Quoted 2 lines

Do you see an issue exporting migrate_device_pfn_lock, migrate_device_unmap?

If there is a good justification for it I can't see a problem with exporting it. That said I don't really understand why you would want/need to split those steps up but I'll wait to see the code.

It is so the device pages returned from hmm_range_fault, which are only guaranteed to be valid under the notifier lock + a seqno check, to be locked and ref taken for migration. migrate_device_unmap() can trigger a MMU invalidation which takes the notifier lock thus calling the function which combines migrate_device_pfn_lock + migrate_device_unmap deadlocks.

I think this flow makes sense and agree in general this likely better than looking at a CPUVMA.

Quoted 6 lines
Quoted 3 lines

I'm still a bit confused about what is better with this flow if you are still calling hmm_range_fault(). How is it better than just calling migrate_vma_setup()? Obviously it will fault the pages in, but it seems

The code in rev2 calls migrate_vma_setup but the requires a struct vm_area_struct argument whereas hmm_range_fault does not.

I'm not sure that's a good enough justfication because the problem isn't whether you're looking up vma's in driver code or mm code. The problem is you are looking up vma's at all and all that goes with that (mainly taking mmap lock, etc.)

Quoted 5 lines

And for eviction hmm_range_fault() won't even find all the pages because their virtual address may have changed - consider what happens in cases of mremap(), fork(), etc. So eviction really needs physical pages (pfn's), not virtual addresses.

See above, #1 yes we use a physical pages. For #2 it is about making the state consistent within a virtual address range.

Matt

Quoted 46 lines
Quoted 8 lines
Quoted 3 lines

we're talking about eviction here so I don't understand why that would be relevant. And hmm_range_fault() still requires the VMA, although I need to look at the patches more closely, probably CPUVMA is a DRM

'hmm_range_fault() still requires the VMA' internal yes, but again not as argument. This is about avoiding a driver side lookup of the VMA.

CPUVMA == struct vm_area_struct in this email.

Thanks for the clarification.

- Alistair

Quoted 32 lines

Matt

Quoted 30 lines

specific concept?

Thanks.

- Alistair

Quoted 23 lines

Matt

Quoted 21 lines

- Alistair

Quoted 18 lines

Matt

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2

Quoted 14 lines

Matt

Quoted 12 lines
Quoted 11 lines
(quoted diff) +8 -0
+	}
+
+	migrate_device_unmap(src_pfns, npages, NULL);
+
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
 /*
  * Migrate a device coherent folio back to normal memory. The caller should have
  * a reference on folio which will be copied to the new folio if migration is
#12 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

Quoted 197 lines · Matthew Brost
Quoted 139 lines · Alistair Popple
Quoted 127 lines · Matthew Brost
Quoted 126 lines · Alistair Popple
Quoted 122 lines · Matthew Brost
Quoted 63 lines · Alistair Popple
Quoted 2 lines · Matthew Brost
Quoted 1 line · Matthew Brost

On Wed, Oct 16, 2024 at 03:04:06PM +1100, Alistair Popple wrote:

[...]

Quoted 55 lines
Quoted 25 lines
Quoted 15 lines
Quoted 15 lines
(quoted diff) +15 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; i++) {
+		struct page *page = pfn_to_page(src_pfns[i]);
+
+		if (!get_page_unless_zero(page)) {
+			src_pfns[i] = 0;
+			continue;
+		}
+
+		if (!trylock_page(page)) {
+			src_pfns[i] = 0;
+			put_page(page);
+			continue;
Quoted 9 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+		}
+
+		src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like migrate_device_range(). But more importantly this should be split out as a function that both migrate_device_range() and this function can call given this bit is identical.

Quoted 3 lines

Missed the folio conversion and agree a helper shared between this function and migrate_device_range would be a good idea. Let add that.

Alistair,

Ok, I think now I want to go slightly different direction here to give GPUSVM a bit more control over several eviction scenarios.

What if I exported the helper discussed above, e.g.,

905 unsigned long migrate_device_pfn_lock(unsigned long pfn) 906 {
907 struct folio *folio;
908
909 folio = folio_get_nontail_page(pfn_to_page(pfn));
910 if (!folio)
911 return 0;
912
913 if (!folio_trylock(folio)) {
914 folio_put(folio);
915 return 0;
916 }
917
918 return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
919 }
920 EXPORT_SYMBOL(migrate_device_pfn_lock);

And then also export migrate_device_unmap.

Quoted 9 lines
Quoted 4 lines

The usage here would be let a driver collect the device pages in virtual address range via hmm_range_fault, lock device pages under notifier lock ensuring device pages are valid, drop the notifier lock and call migrate_device_unmap.

I'm still working through this series but that seems a bit dubious, the locking here is pretty subtle and easy to get wrong so seeing some code would help me a lot in understanding what you're suggesting.

For sure locking in tricky, my mistake on not working through this before sending out the next rev but it came to mind after sending + regarding some late feedback from Thomas about using hmm for eviction [2]. His suggestion of using hmm_range_fault to trigger migration doesn't work for coherent pages, while something like below does.

[2] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1125461

Here is a snippet I have locally which seems to work.

2024 retry:
2025 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2026 hmm_range.hmm_pfns = src;
2027
2028 while (true) {

2029                 mmap_read_lock(mm);
2030                 err = hmm_range_fault(&hmm_range);
2031                 mmap_read_unlock(mm);
2032                 if (err == -EBUSY) {

2033 if (time_after(jiffies, timeout))
2034 break;
2035
2036 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2037 continue;
2038 }
2039 break;
2040 }
2041 if (err)
2042 goto err_put;
2043
2044 drm_gpusvm_notifier_lock(gpusvm);
2045 if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {

2046                 drm_gpusvm_notifier_unlock(gpusvm);
2047                 memset(src, 0, sizeof(*src) * npages);
2048                 goto retry;

2049 }
2050 for (i = 0; i < npages; ++i) {
2051 struct page *page = hmm_pfn_to_page(src[i]);
2052
2053 if (page && (is_device_private_page(page) ||
2054 is_device_coherent_page(page)) && page->zone_device_data)
2055 src[i] = src[i] & ~HMM_PFN_FLAGS;
2056 else
2057 src[i] = 0;
2058 if (src[i])
2059 src[i] = migrate_device_pfn_lock(src[i]);
2060 }
2061 drm_gpusvm_notifier_unlock(gpusvm);

Quoted 8 lines
Quoted 4 lines

Practically for eviction isn't this much the same as calling migrate_vma_setup()? And also for eviction as Sima mentioned you probably shouldn't be looking at mm/vma structs.

hmm_range_fault is just collecting the pages, internally I suppose it does look at a VMA (struct vm_area_struct) but I think the point is drivers should not be looking at VMA here.

Quoted 6 lines

migrate_vma_setup() is designed to be called by drivers and needs a vma, so in general I don't see a problem with drivers looking up vma's. The problem arises specifically for eviction and whether or not that happens in the driver or hmm_range_fault() is pretty irrelevant IMHO for the issues there (see below).

Ok, if you think it ok for drivers to lookup the VMA then purposed exporting of migrate_device_pfn_lock & migrate_device_unmap is not needed, rather just the original function exported in the this patch.

More below too.

Quoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 10 lines

2063 migrate_device_unmap(src, npages, NULL);
...
2101 migrate_device_pages(src, dst, npages);
2102 migrate_device_finalize(src, dst, npages);

Quoted 4 lines
Quoted 3 lines

Sima has strongly suggested avoiding a CPUVMA lookup during eviction cases and this would let me fixup drm_gpusvm_range_evict in [1] to avoid this.

Quoted 6 lines
Quoted 4 lines

That sounds reasonable but for context do you have a link to the comments/discussion on this? I couldn't readily find it, but I may have just missed it.

See in [4], search for '2. eviction' comment from sima.

Thanks for pointing that out. For reference here's Sima's comment:

Quoted 9 lines

2. eviction

Requirements much like migrate_to_ram, because otherwise we break the migration gurantee:

- Only looking at physical memory datastructures and locks, no looking at mm/vma structs or relying on those being locked. We rely entirely on reverse maps from try_to_migrate to find all the mappings on both cpu and gpu side (cpu only zone device swap or migration pte entries ofc).

I also very much agree with this. That's basically why I added migrate_device_range(), so that we can forcibly evict pages when the driver needs them freed (eg. driver unload, low memory, etc.). In general it is impossible to guarantee eviction og all pages using just hmm_range_fault().

Quoted 7 lines
Quoted 2 lines

In this code path we don't have device pages available, hence the purposed collection via hmm_range_fault.

Why don't you have the pfns requiring eviction available? I need to read this series in more depth, but generally hmm_range_fault() can't gurantee you will find every device page.

Quoted 7 lines

There are two cases for eviction in my series:

1. TTM decides it needs to move memory. This calls drm_gpusvm_evict_to_ram. In this case the device pfns are available directly from drm_gpusvm_devmem so the migrate_device_* calls be used here albiet with the new function added in this patch as device pfns may be non-contiguous.

That makes sense and is generally what I think of when I'm thinking of eviction. The new function makes sense too - migrate_device_range() was primarily introduced to allow a driver to evict all device-private pages from a GPU so didn't consider non-contiguous cases, etc.

Quoted 8 lines

2. An inconsistent state for VA range occurs (mixed system and device pages, partial unmap of a range, etc...). Here we want to evict the range ram to make the state consistent. No device pages are available due to an intentional disconnect between a virtual range and physical drm_gpusvm_devmem, thus the device pages have to be looked up. This the function drm_gpusvm_range_evict. Based on what you tell me, it likely is fine the way originally coded in v2 (vma lookup + migrate_vma_*) vs using hmm_range_fault like I have suggested here.

Thanks for the explanation. I think vma lookup + migrate_vma_setup() is fine for this usage and is exactly what you want - it was designed to either select all the system memory pages or device-private pages within a VA range and migrate them.

FWIW I have toyed with the idea of a combined hmm_range_fault()/migrate_vma_setup() front-end to the rest of the migrate_vma_*() process but haven't come up with something nice as yet. I don't think mixing the two in an open-coded fashion is a good idea though, I'd rather we come up with a new API that addresses the short-comings of migrate_vma_setup().

Quoted 44 lines

Note #2 may be removed or unnecessary at some point if we decide to add support for ininconsistent state in GPU SVM and Xe. Keeping it simple for now though. See 'Ranges with mixed system and device pages' in [5].

[5] https://patchwork.freedesktop.org/patch/619819/?series=137870&rev=2

Quoted 38 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines

[3] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1110726
[4] https://lore.kernel.org/all/BYAPR11MB3159A304925168D8B6B4671292692@BYAPR11MB3159.namprd11.prod.outlook.com/T/#m89cd6a37778ba5271d5381ebeb03e1f963856a78

Quoted 6 lines
Quoted 6 lines

It would also make the function exported in this patch unnecessary too as non-contiguous pfns can be setup on driver side via migrate_device_pfn_lock and then migrate_device_unmap can be called. This also another eviction usage in GPUSVM, see drm_gpusvm_evict_to_ram in [1].

Quoted 7 lines
Quoted 2 lines

Do you see an issue exporting migrate_device_pfn_lock, migrate_device_unmap?

If there is a good justification for it I can't see a problem with exporting it. That said I don't really understand why you would want/need to split those steps up but I'll wait to see the code.

It is so the device pages returned from hmm_range_fault, which are only guaranteed to be valid under the notifier lock + a seqno check, to be locked and ref taken for migration. migrate_device_unmap() can trigger a MMU invalidation which takes the notifier lock thus calling the function which combines migrate_device_pfn_lock + migrate_device_unmap deadlocks.

I think this flow makes sense and agree in general this likely better than looking at a CPUVMA.

Quoted 6 lines
Quoted 3 lines

I'm still a bit confused about what is better with this flow if you are still calling hmm_range_fault(). How is it better than just calling migrate_vma_setup()? Obviously it will fault the pages in, but it seems

The code in rev2 calls migrate_vma_setup but the requires a struct vm_area_struct argument whereas hmm_range_fault does not.

I'm not sure that's a good enough justfication because the problem isn't whether you're looking up vma's in driver code or mm code. The problem is you are looking up vma's at all and all that goes with that (mainly taking mmap lock, etc.)

Quoted 8 lines
Quoted 5 lines

And for eviction hmm_range_fault() won't even find all the pages because their virtual address may have changed - consider what happens in cases of mremap(), fork(), etc. So eviction really needs physical pages (pfn's), not virtual addresses.

See above, #1 yes we use a physical pages. For #2 it is about making the state consistent within a virtual address range.

Yep, makes sense now. For migration of physical pages you want migrate_device_*, virtual address ranges want migrate_vma_*

- Alistair

Quoted 48 lines

Matt

Quoted 46 lines
Quoted 8 lines
Quoted 3 lines

we're talking about eviction here so I don't understand why that would be relevant. And hmm_range_fault() still requires the VMA, although I need to look at the patches more closely, probably CPUVMA is a DRM

'hmm_range_fault() still requires the VMA' internal yes, but again not as argument. This is about avoiding a driver side lookup of the VMA.

CPUVMA == struct vm_area_struct in this email.

Thanks for the clarification.

- Alistair

Quoted 32 lines

Matt

Quoted 30 lines

specific concept?

Thanks.

- Alistair

Quoted 23 lines

Matt

Quoted 21 lines

- Alistair

Quoted 18 lines

Matt

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2

Quoted 14 lines

Matt

Quoted 12 lines
Quoted 11 lines
(quoted diff) +8 -0
+	}
+
+	migrate_device_unmap(src_pfns, npages, NULL);
+
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
 /*
  * Migrate a device coherent folio back to normal memory. The caller should have
  * a reference on folio which will be copied to the new folio if migration is
author maintainer
#13 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

Quoted 227 lines · Alistair Popple
Quoted 197 lines · Matthew Brost
Quoted 139 lines · Alistair Popple
Quoted 127 lines · Matthew Brost
Quoted 126 lines · Alistair Popple
Quoted 122 lines · Matthew Brost
Quoted 63 lines · Alistair Popple
Quoted 2 lines · Matthew Brost
Quoted 1 line · Matthew Brost

On Wed, Oct 16, 2024 at 03:04:06PM +1100, Alistair Popple wrote:

[...]

Quoted 55 lines
Quoted 25 lines
Quoted 15 lines
Quoted 15 lines
(quoted diff) +15 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; i++) {
+		struct page *page = pfn_to_page(src_pfns[i]);
+
+		if (!get_page_unless_zero(page)) {
+			src_pfns[i] = 0;
+			continue;
+		}
+
+		if (!trylock_page(page)) {
+			src_pfns[i] = 0;
+			put_page(page);
+			continue;
Quoted 9 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+		}
+
+		src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like migrate_device_range(). But more importantly this should be split out as a function that both migrate_device_range() and this function can call given this bit is identical.

Quoted 3 lines

Missed the folio conversion and agree a helper shared between this function and migrate_device_range would be a good idea. Let add that.

Alistair,

Ok, I think now I want to go slightly different direction here to give GPUSVM a bit more control over several eviction scenarios.

What if I exported the helper discussed above, e.g.,

905 unsigned long migrate_device_pfn_lock(unsigned long pfn) 906 {
907 struct folio *folio;
908
909 folio = folio_get_nontail_page(pfn_to_page(pfn));
910 if (!folio)
911 return 0;
912
913 if (!folio_trylock(folio)) {
914 folio_put(folio);
915 return 0;
916 }
917
918 return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
919 }
920 EXPORT_SYMBOL(migrate_device_pfn_lock);

And then also export migrate_device_unmap.

Quoted 9 lines
Quoted 4 lines

The usage here would be let a driver collect the device pages in virtual address range via hmm_range_fault, lock device pages under notifier lock ensuring device pages are valid, drop the notifier lock and call migrate_device_unmap.

I'm still working through this series but that seems a bit dubious, the locking here is pretty subtle and easy to get wrong so seeing some code would help me a lot in understanding what you're suggesting.

For sure locking in tricky, my mistake on not working through this before sending out the next rev but it came to mind after sending + regarding some late feedback from Thomas about using hmm for eviction [2]. His suggestion of using hmm_range_fault to trigger migration doesn't work for coherent pages, while something like below does.

[2] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1125461

Here is a snippet I have locally which seems to work.

2024 retry:
2025 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2026 hmm_range.hmm_pfns = src;
2027
2028 while (true) {

2029                 mmap_read_lock(mm);
2030                 err = hmm_range_fault(&hmm_range);
2031                 mmap_read_unlock(mm);
2032                 if (err == -EBUSY) {

2033 if (time_after(jiffies, timeout))
2034 break;
2035
2036 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2037 continue;
2038 }
2039 break;
2040 }
2041 if (err)
2042 goto err_put;
2043
2044 drm_gpusvm_notifier_lock(gpusvm);
2045 if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {

2046                 drm_gpusvm_notifier_unlock(gpusvm);
2047                 memset(src, 0, sizeof(*src) * npages);
2048                 goto retry;

2049 }
2050 for (i = 0; i < npages; ++i) {
2051 struct page *page = hmm_pfn_to_page(src[i]);
2052
2053 if (page && (is_device_private_page(page) ||
2054 is_device_coherent_page(page)) && page->zone_device_data)
2055 src[i] = src[i] & ~HMM_PFN_FLAGS;
2056 else
2057 src[i] = 0;
2058 if (src[i])
2059 src[i] = migrate_device_pfn_lock(src[i]);
2060 }
2061 drm_gpusvm_notifier_unlock(gpusvm);

Quoted 8 lines
Quoted 4 lines

Practically for eviction isn't this much the same as calling migrate_vma_setup()? And also for eviction as Sima mentioned you probably shouldn't be looking at mm/vma structs.

hmm_range_fault is just collecting the pages, internally I suppose it does look at a VMA (struct vm_area_struct) but I think the point is drivers should not be looking at VMA here.

Quoted 6 lines

migrate_vma_setup() is designed to be called by drivers and needs a vma, so in general I don't see a problem with drivers looking up vma's. The problem arises specifically for eviction and whether or not that happens in the driver or hmm_range_fault() is pretty irrelevant IMHO for the issues there (see below).

Ok, if you think it ok for drivers to lookup the VMA then purposed exporting of migrate_device_pfn_lock & migrate_device_unmap is not needed, rather just the original function exported in the this patch.

More below too.

Quoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 10 lines

2063 migrate_device_unmap(src, npages, NULL);
...
2101 migrate_device_pages(src, dst, npages);
2102 migrate_device_finalize(src, dst, npages);

Quoted 4 lines
Quoted 3 lines

Sima has strongly suggested avoiding a CPUVMA lookup during eviction cases and this would let me fixup drm_gpusvm_range_evict in [1] to avoid this.

Quoted 6 lines
Quoted 4 lines

That sounds reasonable but for context do you have a link to the comments/discussion on this? I couldn't readily find it, but I may have just missed it.

See in [4], search for '2. eviction' comment from sima.

Thanks for pointing that out. For reference here's Sima's comment:

Quoted 9 lines

2. eviction

Requirements much like migrate_to_ram, because otherwise we break the migration gurantee:

- Only looking at physical memory datastructures and locks, no looking at mm/vma structs or relying on those being locked. We rely entirely on reverse maps from try_to_migrate to find all the mappings on both cpu and gpu side (cpu only zone device swap or migration pte entries ofc).

I also very much agree with this. That's basically why I added migrate_device_range(), so that we can forcibly evict pages when the driver needs them freed (eg. driver unload, low memory, etc.). In general it is impossible to guarantee eviction og all pages using just hmm_range_fault().

Quoted 7 lines
Quoted 2 lines

In this code path we don't have device pages available, hence the purposed collection via hmm_range_fault.

Why don't you have the pfns requiring eviction available? I need to read this series in more depth, but generally hmm_range_fault() can't gurantee you will find every device page.

Quoted 7 lines

There are two cases for eviction in my series:

1. TTM decides it needs to move memory. This calls drm_gpusvm_evict_to_ram. In this case the device pfns are available directly from drm_gpusvm_devmem so the migrate_device_* calls be used here albiet with the new function added in this patch as device pfns may be non-contiguous.

That makes sense and is generally what I think of when I'm thinking of eviction. The new function makes sense too - migrate_device_range() was primarily introduced to allow a driver to evict all device-private pages from a GPU so didn't consider non-contiguous cases, etc.

Quoted 8 lines

2. An inconsistent state for VA range occurs (mixed system and device pages, partial unmap of a range, etc...). Here we want to evict the range ram to make the state consistent. No device pages are available due to an intentional disconnect between a virtual range and physical drm_gpusvm_devmem, thus the device pages have to be looked up. This the function drm_gpusvm_range_evict. Based on what you tell me, it likely is fine the way originally coded in v2 (vma lookup + migrate_vma_*) vs using hmm_range_fault like I have suggested here.

Thanks for the explanation. I think vma lookup + migrate_vma_setup() is fine for this usage and is exactly what you want - it was designed to either select all the system memory pages or device-private pages within a VA range and migrate them.

Quoted 7 lines

FWIW I have toyed with the idea of a combined hmm_range_fault()/migrate_vma_setup() front-end to the rest of the migrate_vma_*() process but haven't come up with something nice as yet. I don't think mixing the two in an open-coded fashion is a good idea though, I'd rather we come up with a new API that addresses the short-comings of migrate_vma_setup().

I think that would good. Here we actually need to lookup multiple VMAs and have a sequence of migrate_vma_* calls as it possible for VMAs to have changed after the driver range was created. It might be nice to hide the VMA lookup from the drivers with an API saying collect and migrate all pages of a type in a VA range much like hmm_range_fault. If the range spans multiple VMAs that would be hidden from the caller.

Matt

Quoted 107 lines
Quoted 44 lines

Note #2 may be removed or unnecessary at some point if we decide to add support for ininconsistent state in GPU SVM and Xe. Keeping it simple for now though. See 'Ranges with mixed system and device pages' in [5].

[5] https://patchwork.freedesktop.org/patch/619819/?series=137870&rev=2

Quoted 38 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines

[3] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1110726
[4] https://lore.kernel.org/all/BYAPR11MB3159A304925168D8B6B4671292692@BYAPR11MB3159.namprd11.prod.outlook.com/T/#m89cd6a37778ba5271d5381ebeb03e1f963856a78

Quoted 6 lines
Quoted 6 lines

It would also make the function exported in this patch unnecessary too as non-contiguous pfns can be setup on driver side via migrate_device_pfn_lock and then migrate_device_unmap can be called. This also another eviction usage in GPUSVM, see drm_gpusvm_evict_to_ram in [1].

Quoted 7 lines
Quoted 2 lines

Do you see an issue exporting migrate_device_pfn_lock, migrate_device_unmap?

If there is a good justification for it I can't see a problem with exporting it. That said I don't really understand why you would want/need to split those steps up but I'll wait to see the code.

It is so the device pages returned from hmm_range_fault, which are only guaranteed to be valid under the notifier lock + a seqno check, to be locked and ref taken for migration. migrate_device_unmap() can trigger a MMU invalidation which takes the notifier lock thus calling the function which combines migrate_device_pfn_lock + migrate_device_unmap deadlocks.

I think this flow makes sense and agree in general this likely better than looking at a CPUVMA.

Quoted 6 lines
Quoted 3 lines

I'm still a bit confused about what is better with this flow if you are still calling hmm_range_fault(). How is it better than just calling migrate_vma_setup()? Obviously it will fault the pages in, but it seems

The code in rev2 calls migrate_vma_setup but the requires a struct vm_area_struct argument whereas hmm_range_fault does not.

I'm not sure that's a good enough justfication because the problem isn't whether you're looking up vma's in driver code or mm code. The problem is you are looking up vma's at all and all that goes with that (mainly taking mmap lock, etc.)

Quoted 8 lines
Quoted 5 lines

And for eviction hmm_range_fault() won't even find all the pages because their virtual address may have changed - consider what happens in cases of mremap(), fork(), etc. So eviction really needs physical pages (pfn's), not virtual addresses.

See above, #1 yes we use a physical pages. For #2 it is about making the state consistent within a virtual address range.

Yep, makes sense now. For migration of physical pages you want migrate_device_*, virtual address ranges want migrate_vma_*

- Alistair

Quoted 48 lines

Matt

Quoted 46 lines
Quoted 8 lines
Quoted 3 lines

we're talking about eviction here so I don't understand why that would be relevant. And hmm_range_fault() still requires the VMA, although I need to look at the patches more closely, probably CPUVMA is a DRM

'hmm_range_fault() still requires the VMA' internal yes, but again not as argument. This is about avoiding a driver side lookup of the VMA.

CPUVMA == struct vm_area_struct in this email.

Thanks for the clarification.

- Alistair

Quoted 32 lines

Matt

Quoted 30 lines

specific concept?

Thanks.

- Alistair

Quoted 23 lines

Matt

Quoted 21 lines

- Alistair

Quoted 18 lines

Matt

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2

Quoted 14 lines

Matt

Quoted 12 lines
Quoted 11 lines
(quoted diff) +8 -0
+	}
+
+	migrate_device_unmap(src_pfns, npages, NULL);
+
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
 /*
  * Migrate a device coherent folio back to normal memory. The caller should have
  * a reference on folio which will be copied to the new folio if migration is
#14 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

Quoted 236 lines · Matthew Brost
Quoted 227 lines · Alistair Popple
Quoted 197 lines · Matthew Brost
Quoted 139 lines · Alistair Popple
Quoted 127 lines · Matthew Brost
Quoted 126 lines · Alistair Popple
Quoted 122 lines · Matthew Brost
Quoted 63 lines · Alistair Popple
Quoted 2 lines · Matthew Brost
Quoted 1 line · Matthew Brost

On Wed, Oct 16, 2024 at 03:04:06PM +1100, Alistair Popple wrote:

[...]

Quoted 55 lines
Quoted 25 lines
Quoted 15 lines
Quoted 15 lines
(quoted diff) +15 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; i++) {
+		struct page *page = pfn_to_page(src_pfns[i]);
+
+		if (!get_page_unless_zero(page)) {
+			src_pfns[i] = 0;
+			continue;
+		}
+
+		if (!trylock_page(page)) {
+			src_pfns[i] = 0;
+			put_page(page);
+			continue;
Quoted 9 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+		}
+
+		src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like migrate_device_range(). But more importantly this should be split out as a function that both migrate_device_range() and this function can call given this bit is identical.

Quoted 3 lines

Missed the folio conversion and agree a helper shared between this function and migrate_device_range would be a good idea. Let add that.

Alistair,

Ok, I think now I want to go slightly different direction here to give GPUSVM a bit more control over several eviction scenarios.

What if I exported the helper discussed above, e.g.,

905 unsigned long migrate_device_pfn_lock(unsigned long pfn) 906 {
907 struct folio *folio;
908
909 folio = folio_get_nontail_page(pfn_to_page(pfn));
910 if (!folio)
911 return 0;
912
913 if (!folio_trylock(folio)) {
914 folio_put(folio);
915 return 0;
916 }
917
918 return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
919 }
920 EXPORT_SYMBOL(migrate_device_pfn_lock);

And then also export migrate_device_unmap.

Quoted 9 lines
Quoted 4 lines

The usage here would be let a driver collect the device pages in virtual address range via hmm_range_fault, lock device pages under notifier lock ensuring device pages are valid, drop the notifier lock and call migrate_device_unmap.

I'm still working through this series but that seems a bit dubious, the locking here is pretty subtle and easy to get wrong so seeing some code would help me a lot in understanding what you're suggesting.

For sure locking in tricky, my mistake on not working through this before sending out the next rev but it came to mind after sending + regarding some late feedback from Thomas about using hmm for eviction [2]. His suggestion of using hmm_range_fault to trigger migration doesn't work for coherent pages, while something like below does.

[2] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1125461

Here is a snippet I have locally which seems to work.

2024 retry:
2025 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2026 hmm_range.hmm_pfns = src;
2027
2028 while (true) {

2029                 mmap_read_lock(mm);
2030                 err = hmm_range_fault(&hmm_range);
2031                 mmap_read_unlock(mm);
2032                 if (err == -EBUSY) {

2033 if (time_after(jiffies, timeout))
2034 break;
2035
2036 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2037 continue;
2038 }
2039 break;
2040 }
2041 if (err)
2042 goto err_put;
2043
2044 drm_gpusvm_notifier_lock(gpusvm);
2045 if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {

2046                 drm_gpusvm_notifier_unlock(gpusvm);
2047                 memset(src, 0, sizeof(*src) * npages);
2048                 goto retry;

2049 }
2050 for (i = 0; i < npages; ++i) {
2051 struct page *page = hmm_pfn_to_page(src[i]);
2052
2053 if (page && (is_device_private_page(page) ||
2054 is_device_coherent_page(page)) && page->zone_device_data)
2055 src[i] = src[i] & ~HMM_PFN_FLAGS;
2056 else
2057 src[i] = 0;
2058 if (src[i])
2059 src[i] = migrate_device_pfn_lock(src[i]);
2060 }
2061 drm_gpusvm_notifier_unlock(gpusvm);

Quoted 8 lines
Quoted 4 lines

Practically for eviction isn't this much the same as calling migrate_vma_setup()? And also for eviction as Sima mentioned you probably shouldn't be looking at mm/vma structs.

hmm_range_fault is just collecting the pages, internally I suppose it does look at a VMA (struct vm_area_struct) but I think the point is drivers should not be looking at VMA here.

Quoted 6 lines

migrate_vma_setup() is designed to be called by drivers and needs a vma, so in general I don't see a problem with drivers looking up vma's. The problem arises specifically for eviction and whether or not that happens in the driver or hmm_range_fault() is pretty irrelevant IMHO for the issues there (see below).

Ok, if you think it ok for drivers to lookup the VMA then purposed exporting of migrate_device_pfn_lock & migrate_device_unmap is not needed, rather just the original function exported in the this patch.

More below too.

Quoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 10 lines

2063 migrate_device_unmap(src, npages, NULL);
...
2101 migrate_device_pages(src, dst, npages);
2102 migrate_device_finalize(src, dst, npages);

Quoted 4 lines
Quoted 3 lines

Sima has strongly suggested avoiding a CPUVMA lookup during eviction cases and this would let me fixup drm_gpusvm_range_evict in [1] to avoid this.

Quoted 6 lines
Quoted 4 lines

That sounds reasonable but for context do you have a link to the comments/discussion on this? I couldn't readily find it, but I may have just missed it.

See in [4], search for '2. eviction' comment from sima.

Thanks for pointing that out. For reference here's Sima's comment:

Quoted 9 lines

2. eviction

Requirements much like migrate_to_ram, because otherwise we break the migration gurantee:

- Only looking at physical memory datastructures and locks, no looking at mm/vma structs or relying on those being locked. We rely entirely on reverse maps from try_to_migrate to find all the mappings on both cpu and gpu side (cpu only zone device swap or migration pte entries ofc).

I also very much agree with this. That's basically why I added migrate_device_range(), so that we can forcibly evict pages when the driver needs them freed (eg. driver unload, low memory, etc.). In general it is impossible to guarantee eviction og all pages using just hmm_range_fault().

Quoted 7 lines
Quoted 2 lines

In this code path we don't have device pages available, hence the purposed collection via hmm_range_fault.

Why don't you have the pfns requiring eviction available? I need to read this series in more depth, but generally hmm_range_fault() can't gurantee you will find every device page.

Quoted 7 lines

There are two cases for eviction in my series:

1. TTM decides it needs to move memory. This calls drm_gpusvm_evict_to_ram. In this case the device pfns are available directly from drm_gpusvm_devmem so the migrate_device_* calls be used here albiet with the new function added in this patch as device pfns may be non-contiguous.

That makes sense and is generally what I think of when I'm thinking of eviction. The new function makes sense too - migrate_device_range() was primarily introduced to allow a driver to evict all device-private pages from a GPU so didn't consider non-contiguous cases, etc.

Quoted 8 lines

2. An inconsistent state for VA range occurs (mixed system and device pages, partial unmap of a range, etc...). Here we want to evict the range ram to make the state consistent. No device pages are available due to an intentional disconnect between a virtual range and physical drm_gpusvm_devmem, thus the device pages have to be looked up. This the function drm_gpusvm_range_evict. Based on what you tell me, it likely is fine the way originally coded in v2 (vma lookup + migrate_vma_*) vs using hmm_range_fault like I have suggested here.

Thanks for the explanation. I think vma lookup + migrate_vma_setup() is fine for this usage and is exactly what you want - it was designed to either select all the system memory pages or device-private pages within a VA range and migrate them.

Quoted 7 lines

FWIW I have toyed with the idea of a combined hmm_range_fault()/migrate_vma_setup() front-end to the rest of the migrate_vma_*() process but haven't come up with something nice as yet. I don't think mixing the two in an open-coded fashion is a good idea though, I'd rather we come up with a new API that addresses the short-comings of migrate_vma_setup().

Quoted 6 lines

I think that would good. Here we actually need to lookup multiple VMAs and have a sequence of migrate_vma_* calls as it possible for VMAs to have changed after the driver range was created. It might be nice to hide the VMA lookup from the drivers with an API saying collect and migrate all pages of a type in a VA range much like hmm_range_fault. If the range spans multiple VMAs that would be hidden from the caller.

Ok. I wasn't really considering multiple VMAs. UVM and Nouveau don't really have a requirement to migrate across multiple VMAs but if that's neccessary I think an API that hides that specifically for working with migrate_vma_*() might make sense.

Quoted 5 lines

Matt

Quoted 3 lines
Quoted 3 lines

Note #2 may be removed or unnecessary at some point if we decide to add support for ininconsistent state in GPU SVM and Xe. Keeping it simple for now though. See 'Ranges with mixed system and device pages' in [5].

As someone not very familiar with some of the DRM layers can I ask why having virtual address ranges with a mix of system and device pages is hard to support? It seems to me that in practice it might be quite difficult to keep a VMA range as exclusively all in system memory or all in device memory.

Quoted 103 lines
Quoted 103 lines
Quoted 40 lines

[5] https://patchwork.freedesktop.org/patch/619819/?series=137870&rev=2

Quoted 38 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines

[3] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1110726
[4] https://lore.kernel.org/all/BYAPR11MB3159A304925168D8B6B4671292692@BYAPR11MB3159.namprd11.prod.outlook.com/T/#m89cd6a37778ba5271d5381ebeb03e1f963856a78

Quoted 6 lines
Quoted 6 lines

It would also make the function exported in this patch unnecessary too as non-contiguous pfns can be setup on driver side via migrate_device_pfn_lock and then migrate_device_unmap can be called. This also another eviction usage in GPUSVM, see drm_gpusvm_evict_to_ram in [1].

Quoted 7 lines
Quoted 2 lines

Do you see an issue exporting migrate_device_pfn_lock, migrate_device_unmap?

If there is a good justification for it I can't see a problem with exporting it. That said I don't really understand why you would want/need to split those steps up but I'll wait to see the code.

It is so the device pages returned from hmm_range_fault, which are only guaranteed to be valid under the notifier lock + a seqno check, to be locked and ref taken for migration. migrate_device_unmap() can trigger a MMU invalidation which takes the notifier lock thus calling the function which combines migrate_device_pfn_lock + migrate_device_unmap deadlocks.

I think this flow makes sense and agree in general this likely better than looking at a CPUVMA.

Quoted 6 lines
Quoted 3 lines

I'm still a bit confused about what is better with this flow if you are still calling hmm_range_fault(). How is it better than just calling migrate_vma_setup()? Obviously it will fault the pages in, but it seems

The code in rev2 calls migrate_vma_setup but the requires a struct vm_area_struct argument whereas hmm_range_fault does not.

I'm not sure that's a good enough justfication because the problem isn't whether you're looking up vma's in driver code or mm code. The problem is you are looking up vma's at all and all that goes with that (mainly taking mmap lock, etc.)

Quoted 8 lines
Quoted 5 lines

And for eviction hmm_range_fault() won't even find all the pages because their virtual address may have changed - consider what happens in cases of mremap(), fork(), etc. So eviction really needs physical pages (pfn's), not virtual addresses.

See above, #1 yes we use a physical pages. For #2 it is about making the state consistent within a virtual address range.

Yep, makes sense now. For migration of physical pages you want migrate_device_*, virtual address ranges want migrate_vma_*

- Alistair

Quoted 48 lines

Matt

Quoted 46 lines
Quoted 8 lines
Quoted 3 lines

we're talking about eviction here so I don't understand why that would be relevant. And hmm_range_fault() still requires the VMA, although I need to look at the patches more closely, probably CPUVMA is a DRM

'hmm_range_fault() still requires the VMA' internal yes, but again not as argument. This is about avoiding a driver side lookup of the VMA.

CPUVMA == struct vm_area_struct in this email.

Thanks for the clarification.

- Alistair

Quoted 32 lines

Matt

Quoted 30 lines

specific concept?

Thanks.

- Alistair

Quoted 23 lines

Matt

Quoted 21 lines

- Alistair

Quoted 18 lines

Matt

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2

Quoted 14 lines

Matt

Quoted 12 lines
Quoted 11 lines
(quoted diff) +8 -0
+	}
+
+	migrate_device_unmap(src_pfns, npages, NULL);
+
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
 /*
  * Migrate a device coherent folio back to normal memory. The caller should have
  * a reference on folio which will be copied to the new folio if migration is
#15 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

Quoted 220 lines · Alistair Popple
Quoted 212 lines · Matthew Brost
Quoted 204 lines · Alistair Popple
Quoted 177 lines · Matthew Brost
Quoted 124 lines · Alistair Popple
Quoted 122 lines · Matthew Brost
Quoted 121 lines · Alistair Popple
Quoted 102 lines · Matthew Brost
Quoted 58 lines · Alistair Popple
Quoted 2 lines · Matthew Brost
Quoted 1 line · Matthew Brost

On Wed, Oct 16, 2024 at 03:04:06PM +1100, Alistair Popple wrote:

[...]

Quoted 52 lines
Quoted 23 lines
Quoted 14 lines
Quoted 14 lines

+{
+ unsigned long i;
+
+ for (i = 0; i < npages; i++) {
+ struct page *page = pfn_to_page(src_pfns[i]);
+
+ if (!get_page_unless_zero(page)) {
+ src_pfns[i] = 0;
+ continue;
+ }
+
+ if (!trylock_page(page)) {
+ src_pfns[i] = 0;
+ put_page(page);

Quoted 9 lines
Quoted 4 lines

+ continue;
+ }
+
+ src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like migrate_device_range(). But more importantly this should be split out as a function that both migrate_device_range() and this function can call given this bit is identical.

Quoted 3 lines

Missed the folio conversion and agree a helper shared between this function and migrate_device_range would be a good idea. Let add that.

Alistair,

Ok, I think now I want to go slightly different direction here to give GPUSVM a bit more control over several eviction scenarios.

What if I exported the helper discussed above, e.g.,

905 unsigned long migrate_device_pfn_lock(unsigned long pfn) 906 {
907 struct folio *folio;
908
909 folio = folio_get_nontail_page(pfn_to_page(pfn));
910 if (!folio)
911 return 0;
912
913 if (!folio_trylock(folio)) {
914 folio_put(folio);
915 return 0;
916 }
917
918 return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
919 }
920 EXPORT_SYMBOL(migrate_device_pfn_lock);

And then also export migrate_device_unmap.

Quoted 8 lines
Quoted 4 lines

The usage here would be let a driver collect the device pages in virtual address range via hmm_range_fault, lock device pages under notifier lock ensuring device pages are valid, drop the notifier lock and call migrate_device_unmap.

I'm still working through this series but that seems a bit dubious, the locking here is pretty subtle and easy to get wrong so seeing some code would help me a lot in understanding what you're suggesting.

For sure locking in tricky, my mistake on not working through this before sending out the next rev but it came to mind after sending + regarding some late feedback from Thomas about using hmm for eviction [2]. His suggestion of using hmm_range_fault to trigger migration doesn't work for coherent pages, while something like below does.

[2] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1125461

Here is a snippet I have locally which seems to work.

2024 retry:
2025 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2026 hmm_range.hmm_pfns = src;
2027
2028 while (true) {

2029                 mmap_read_lock(mm);
2030                 err = hmm_range_fault(&hmm_range);
2031                 mmap_read_unlock(mm);
2032                 if (err == -EBUSY) {

2033 if (time_after(jiffies, timeout))
2034 break;
2035
2036 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2037 continue;
2038 }
2039 break;
2040 }
2041 if (err)
2042 goto err_put;
2043
2044 drm_gpusvm_notifier_lock(gpusvm);
2045 if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {

2046                 drm_gpusvm_notifier_unlock(gpusvm);
2047                 memset(src, 0, sizeof(*src) * npages);
2048                 goto retry;
Quoted 13 lines

2049 }
2050 for (i = 0; i < npages; ++i) {
2051 struct page *page = hmm_pfn_to_page(src[i]);
2052
2053 if (page && (is_device_private_page(page) ||
2054 is_device_coherent_page(page)) && page->zone_device_data)
2055 src[i] = src[i] & ~HMM_PFN_FLAGS;
2056 else
2057 src[i] = 0;
2058 if (src[i])
2059 src[i] = migrate_device_pfn_lock(src[i]);
2060 }
2061 drm_gpusvm_notifier_unlock(gpusvm);

Practically for eviction isn't this much the same as calling migrate_vma_setup()? And also for eviction as Sima mentioned you probably shouldn't be looking at mm/vma structs.

Quoted 9 lines
Quoted 3 lines

hmm_range_fault is just collecting the pages, internally I suppose it does look at a VMA (struct vm_area_struct) but I think the point is drivers should not be looking at VMA here.

migrate_vma_setup() is designed to be called by drivers and needs a vma, so in general I don't see a problem with drivers looking up vma's. The problem arises specifically for eviction and whether or not that happens in the driver or hmm_range_fault() is pretty irrelevant IMHO for the issues there (see below).

Ok, if you think it ok for drivers to lookup the VMA then purposed exporting of migrate_device_pfn_lock & migrate_device_unmap is not needed, rather just the original function exported in the this patch.

More below too.

Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
Quoted 6 lines

2063 migrate_device_unmap(src, npages, NULL);
...
2101 migrate_device_pages(src, dst, npages);
2102 migrate_device_finalize(src, dst, npages);

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

Sima has strongly suggested avoiding a CPUVMA lookup during eviction cases and this would let me fixup drm_gpusvm_range_evict in [1] to avoid this.

That sounds reasonable but for context do you have a link to the comments/discussion on this? I couldn't readily find it, but I may have just missed it.

See in [4], search for '2. eviction' comment from sima.

Thanks for pointing that out. For reference here's Sima's comment:

Quoted 9 lines

2. eviction

Requirements much like migrate_to_ram, because otherwise we break the migration gurantee:

- Only looking at physical memory datastructures and locks, no looking at mm/vma structs or relying on those being locked. We rely entirely on reverse maps from try_to_migrate to find all the mappings on both cpu and gpu side (cpu only zone device swap or migration pte entries ofc).

I also very much agree with this. That's basically why I added migrate_device_range(), so that we can forcibly evict pages when the driver needs them freed (eg. driver unload, low memory, etc.). In general it is impossible to guarantee eviction og all pages using just hmm_range_fault().

Quoted 6 lines
Quoted 2 lines

In this code path we don't have device pages available, hence the purposed collection via hmm_range_fault.

Why don't you have the pfns requiring eviction available? I need to read this series in more depth, but generally hmm_range_fault() can't gurantee you will find every device page.

Quoted 7 lines

There are two cases for eviction in my series:

1. TTM decides it needs to move memory. This calls drm_gpusvm_evict_to_ram. In this case the device pfns are available directly from drm_gpusvm_devmem so the migrate_device_* calls be used here albiet with the new function added in this patch as device pfns may be non-contiguous.

That makes sense and is generally what I think of when I'm thinking of eviction. The new function makes sense too - migrate_device_range() was primarily introduced to allow a driver to evict all device-private pages from a GPU so didn't consider non-contiguous cases, etc.

Quoted 8 lines

2. An inconsistent state for VA range occurs (mixed system and device pages, partial unmap of a range, etc...). Here we want to evict the range ram to make the state consistent. No device pages are available due to an intentional disconnect between a virtual range and physical drm_gpusvm_devmem, thus the device pages have to be looked up. This the function drm_gpusvm_range_evict. Based on what you tell me, it likely is fine the way originally coded in v2 (vma lookup + migrate_vma_*) vs using hmm_range_fault like I have suggested here.

Thanks for the explanation. I think vma lookup + migrate_vma_setup() is fine for this usage and is exactly what you want - it was designed to either select all the system memory pages or device-private pages within a VA range and migrate them.

Quoted 7 lines

FWIW I have toyed with the idea of a combined hmm_range_fault()/migrate_vma_setup() front-end to the rest of the migrate_vma_*() process but haven't come up with something nice as yet. I don't think mixing the two in an open-coded fashion is a good idea though, I'd rather we come up with a new API that addresses the short-comings of migrate_vma_setup().

Quoted 6 lines

I think that would good. Here we actually need to lookup multiple VMAs and have a sequence of migrate_vma_* calls as it possible for VMAs to have changed after the driver range was created. It might be nice to hide the VMA lookup from the drivers with an API saying collect and migrate all pages of a type in a VA range much like hmm_range_fault. If the range spans multiple VMAs that would be hidden from the caller.

Quoted 4 lines

Ok. I wasn't really considering multiple VMAs. UVM and Nouveau don't really have a requirement to migrate across multiple VMAs but if that's neccessary I think an API that hides that specifically for working with migrate_vma_*() might make sense.

Yes that's what I'm currently doing. You call it in a loop, the fault+migrate prepare part chunks the calls to vma boundaries and you do the migrations for each vma and loop until the whole range done.

Quoted 101 lines
Quoted 6 lines

Matt

Quoted 3 lines
Quoted 3 lines

Note #2 may be removed or unnecessary at some point if we decide to add support for ininconsistent state in GPU SVM and Xe. Keeping it simple for now though. See 'Ranges with mixed system and device pages' in [5].

As someone not very familiar with some of the DRM layers can I ask why having virtual address ranges with a mix of system and device pages is hard to support? It seems to me that in practice it might be quite difficult to keep a VMA range as exclusively all in system memory or all in device memory.

Quoted 89 lines
Quoted 89 lines
Quoted 35 lines

[5] https://patchwork.freedesktop.org/patch/619819/?series=137870&rev=2

Quoted 33 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

[3] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1110726
[4] https://lore.kernel.org/all/BYAPR11MB3159A304925168D8B6B4671292692@BYAPR11MB3159.namprd11.prod.outlook.com/T/#m89cd6a37778ba5271d5381ebeb03e1f963856a78

Quoted 6 lines
Quoted 6 lines

It would also make the function exported in this patch unnecessary too as non-contiguous pfns can be setup on driver side via migrate_device_pfn_lock and then migrate_device_unmap can be called. This also another eviction usage in GPUSVM, see drm_gpusvm_evict_to_ram in [1].

Quoted 6 lines
Quoted 2 lines

Do you see an issue exporting migrate_device_pfn_lock, migrate_device_unmap?

If there is a good justification for it I can't see a problem with exporting it. That said I don't really understand why you would want/need to split those steps up but I'll wait to see the code.

It is so the device pages returned from hmm_range_fault, which are only guaranteed to be valid under the notifier lock + a seqno check, to be locked and ref taken for migration. migrate_device_unmap() can trigger a MMU invalidation which takes the notifier lock thus calling the function which combines migrate_device_pfn_lock + migrate_device_unmap deadlocks.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I think this flow makes sense and agree in general this likely better than looking at a CPUVMA.

I'm still a bit confused about what is better with this flow if you are still calling hmm_range_fault(). How is it better than just calling migrate_vma_setup()? Obviously it will fault the pages in, but it seems

The code in rev2 calls migrate_vma_setup but the requires a struct vm_area_struct argument whereas hmm_range_fault does not.

I'm not sure that's a good enough justfication because the problem isn't whether you're looking up vma's in driver code or mm code. The problem is you are looking up vma's at all and all that goes with that (mainly taking mmap lock, etc.)

Quoted 7 lines
Quoted 5 lines

And for eviction hmm_range_fault() won't even find all the pages because their virtual address may have changed - consider what happens in cases of mremap(), fork(), etc. So eviction really needs physical pages (pfn's), not virtual addresses.

See above, #1 yes we use a physical pages. For #2 it is about making the state consistent within a virtual address range.

Yep, makes sense now. For migration of physical pages you want migrate_device_*, virtual address ranges want migrate_vma_*

- Alistair

Quoted 42 lines

Matt

Quoted 40 lines
Quoted 7 lines
Quoted 3 lines

we're talking about eviction here so I don't understand why that would be relevant. And hmm_range_fault() still requires the VMA, although I need to look at the patches more closely, probably CPUVMA is a DRM

'hmm_range_fault() still requires the VMA' internal yes, but again not as argument. This is about avoiding a driver side lookup of the VMA.

CPUVMA == struct vm_area_struct in this email.

Thanks for the clarification.

- Alistair

Quoted 29 lines

Matt

Quoted 27 lines

specific concept?

Thanks.

- Alistair

Quoted 21 lines

Matt

Quoted 19 lines

- Alistair

Quoted 17 lines

Matt

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2

Quoted 13 lines

Matt

Quoted 11 lines
Quoted 11 lines

+ }
+
+ migrate_device_unmap(src_pfns, npages, NULL);
+
+ return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
/*
* Migrate a device coherent folio back to normal memory. The caller should have
* a reference on folio which will be copied to the new folio if migration is

author maintainer
#16 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

In reply to Alistair Popple
Quoted 246 lines · Alistair Popple
Quoted 236 lines · Matthew Brost
Quoted 227 lines · Alistair Popple
Quoted 197 lines · Matthew Brost
Quoted 139 lines · Alistair Popple
Quoted 127 lines · Matthew Brost
Quoted 126 lines · Alistair Popple
Quoted 122 lines · Matthew Brost
Quoted 63 lines · Alistair Popple
Quoted 2 lines · Matthew Brost
Quoted 1 line · Matthew Brost

On Wed, Oct 16, 2024 at 03:04:06PM +1100, Alistair Popple wrote:

[...]

Quoted 55 lines
Quoted 25 lines
Quoted 15 lines
Quoted 15 lines

+{
+ unsigned long i;
+
+ for (i = 0; i < npages; i++) {
+ struct page *page = pfn_to_page(src_pfns[i]);
+
+ if (!get_page_unless_zero(page)) {
+ src_pfns[i] = 0;
+ continue;
+ }
+
+ if (!trylock_page(page)) {
+ src_pfns[i] = 0;
+ put_page(page);
+ continue;

Quoted 9 lines
Quoted 3 lines

+ }
+
+ src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like migrate_device_range(). But more importantly this should be split out as a function that both migrate_device_range() and this function can call given this bit is identical.

Quoted 3 lines

Missed the folio conversion and agree a helper shared between this function and migrate_device_range would be a good idea. Let add that.

Alistair,

Ok, I think now I want to go slightly different direction here to give GPUSVM a bit more control over several eviction scenarios.

What if I exported the helper discussed above, e.g.,

905 unsigned long migrate_device_pfn_lock(unsigned long pfn) 906 {
907 struct folio *folio;
908
909 folio = folio_get_nontail_page(pfn_to_page(pfn));
910 if (!folio)
911 return 0;
912
913 if (!folio_trylock(folio)) {
914 folio_put(folio);
915 return 0;
916 }
917
918 return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
919 }
920 EXPORT_SYMBOL(migrate_device_pfn_lock);

And then also export migrate_device_unmap.

Quoted 9 lines
Quoted 4 lines

The usage here would be let a driver collect the device pages in virtual address range via hmm_range_fault, lock device pages under notifier lock ensuring device pages are valid, drop the notifier lock and call migrate_device_unmap.

I'm still working through this series but that seems a bit dubious, the locking here is pretty subtle and easy to get wrong so seeing some code would help me a lot in understanding what you're suggesting.

For sure locking in tricky, my mistake on not working through this before sending out the next rev but it came to mind after sending + regarding some late feedback from Thomas about using hmm for eviction [2]. His suggestion of using hmm_range_fault to trigger migration doesn't work for coherent pages, while something like below does.

[2] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1125461

Here is a snippet I have locally which seems to work.

2024 retry:
2025 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2026 hmm_range.hmm_pfns = src;
2027
2028 while (true) {

2029                 mmap_read_lock(mm);
2030                 err = hmm_range_fault(&hmm_range);
2031                 mmap_read_unlock(mm);
2032                 if (err == -EBUSY) {

2033 if (time_after(jiffies, timeout))
2034 break;
2035
2036 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2037 continue;
2038 }
2039 break;
2040 }
2041 if (err)
2042 goto err_put;
2043
2044 drm_gpusvm_notifier_lock(gpusvm);
2045 if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {

2046                 drm_gpusvm_notifier_unlock(gpusvm);
2047                 memset(src, 0, sizeof(*src) * npages);
2048                 goto retry;

2049 }
2050 for (i = 0; i < npages; ++i) {
2051 struct page *page = hmm_pfn_to_page(src[i]);
2052
2053 if (page && (is_device_private_page(page) ||
2054 is_device_coherent_page(page)) && page->zone_device_data)
2055 src[i] = src[i] & ~HMM_PFN_FLAGS;
2056 else
2057 src[i] = 0;
2058 if (src[i])
2059 src[i] = migrate_device_pfn_lock(src[i]);
2060 }
2061 drm_gpusvm_notifier_unlock(gpusvm);

Quoted 8 lines
Quoted 4 lines

Practically for eviction isn't this much the same as calling migrate_vma_setup()? And also for eviction as Sima mentioned you probably shouldn't be looking at mm/vma structs.

hmm_range_fault is just collecting the pages, internally I suppose it does look at a VMA (struct vm_area_struct) but I think the point is drivers should not be looking at VMA here.

Quoted 6 lines

migrate_vma_setup() is designed to be called by drivers and needs a vma, so in general I don't see a problem with drivers looking up vma's. The problem arises specifically for eviction and whether or not that happens in the driver or hmm_range_fault() is pretty irrelevant IMHO for the issues there (see below).

Ok, if you think it ok for drivers to lookup the VMA then purposed exporting of migrate_device_pfn_lock & migrate_device_unmap is not needed, rather just the original function exported in the this patch.

More below too.

Quoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 10 lines

2063 migrate_device_unmap(src, npages, NULL);
...
2101 migrate_device_pages(src, dst, npages);
2102 migrate_device_finalize(src, dst, npages);

Quoted 4 lines
Quoted 3 lines

Sima has strongly suggested avoiding a CPUVMA lookup during eviction cases and this would let me fixup drm_gpusvm_range_evict in [1] to avoid this.

Quoted 6 lines
Quoted 4 lines

That sounds reasonable but for context do you have a link to the comments/discussion on this? I couldn't readily find it, but I may have just missed it.

See in [4], search for '2. eviction' comment from sima.

Thanks for pointing that out. For reference here's Sima's comment:

Quoted 9 lines

2. eviction

Requirements much like migrate_to_ram, because otherwise we break the migration gurantee:

- Only looking at physical memory datastructures and locks, no looking at mm/vma structs or relying on those being locked. We rely entirely on reverse maps from try_to_migrate to find all the mappings on both cpu and gpu side (cpu only zone device swap or migration pte entries ofc).

I also very much agree with this. That's basically why I added migrate_device_range(), so that we can forcibly evict pages when the driver needs them freed (eg. driver unload, low memory, etc.). In general it is impossible to guarantee eviction og all pages using just hmm_range_fault().

Quoted 7 lines
Quoted 2 lines

In this code path we don't have device pages available, hence the purposed collection via hmm_range_fault.

Why don't you have the pfns requiring eviction available? I need to read this series in more depth, but generally hmm_range_fault() can't gurantee you will find every device page.

Quoted 7 lines

There are two cases for eviction in my series:

1. TTM decides it needs to move memory. This calls drm_gpusvm_evict_to_ram. In this case the device pfns are available directly from drm_gpusvm_devmem so the migrate_device_* calls be used here albiet with the new function added in this patch as device pfns may be non-contiguous.

That makes sense and is generally what I think of when I'm thinking of eviction. The new function makes sense too - migrate_device_range() was primarily introduced to allow a driver to evict all device-private pages from a GPU so didn't consider non-contiguous cases, etc.

Quoted 8 lines

2. An inconsistent state for VA range occurs (mixed system and device pages, partial unmap of a range, etc...). Here we want to evict the range ram to make the state consistent. No device pages are available due to an intentional disconnect between a virtual range and physical drm_gpusvm_devmem, thus the device pages have to be looked up. This the function drm_gpusvm_range_evict. Based on what you tell me, it likely is fine the way originally coded in v2 (vma lookup + migrate_vma_*) vs using hmm_range_fault like I have suggested here.

Thanks for the explanation. I think vma lookup + migrate_vma_setup() is fine for this usage and is exactly what you want - it was designed to either select all the system memory pages or device-private pages within a VA range and migrate them.

Quoted 7 lines

FWIW I have toyed with the idea of a combined hmm_range_fault()/migrate_vma_setup() front-end to the rest of the migrate_vma_*() process but haven't come up with something nice as yet. I don't think mixing the two in an open-coded fashion is a good idea though, I'd rather we come up with a new API that addresses the short-comings of migrate_vma_setup().

Quoted 6 lines

I think that would good. Here we actually need to lookup multiple VMAs and have a sequence of migrate_vma_* calls as it possible for VMAs to have changed after the driver range was created. It might be nice to hide the VMA lookup from the drivers with an API saying collect and migrate all pages of a type in a VA range much like hmm_range_fault. If the range spans multiple VMAs that would be hidden from the caller.

Quoted 5 lines

Ok. I wasn't really considering multiple VMAs. UVM and Nouveau don't really have a requirement to migrate across multiple VMAs but if that's neccessary I think an API that hides that specifically for working with migrate_vma_*() might make sense.

We can run into multiple VMA scenarios if a user does something rude like this:

mmap 0x000000...0x1fffff -> fault migrates 2M to VRAM and creates an internal range to track
munmap 0x080000...0x17ffff -> now we have two VMAs instead of one and the range has a hole in it

In this scenario, which we believe to rare / unsual, we just evict remaining VRAM pages to SRAM, destroy range, and fixup on next GPU fault.

Quoted 6 lines
Quoted 5 lines

Matt

Quoted 3 lines
Quoted 3 lines

Note #2 may be removed or unnecessary at some point if we decide to add support for ininconsistent state in GPU SVM and Xe. Keeping it simple for now though. See 'Ranges with mixed system and device pages' in [5].

Quoted 6 lines

As someone not very familiar with some of the DRM layers can I ask why having virtual address ranges with a mix of system and device pages is hard to support? It seems to me that in practice it might be quite difficult to keep a VMA range as exclusively all in system memory or all in device memory.

A few things that make this difficult are:

- Our (Xe) bind code would need to be updated to support this
- TTM / DRM buddy allocator doesn't support freeing / reallocation of individual pages rather aligned chunks of initial allocation size (e.g., 2M would be common allocation size).
- Spliting ranges would add complications

All workable problems but since we are writing a new common implementation trying to keep it as simple as possible for initial merge of the design. Almost certainly at some point we will add support for mixed ranges to the common GPU SVM layer with a driver choosing if it wants mixed or non-mixed ranges via a flag to function calls.

wrt to being difficult keeping exclusively in system or vram, in addition to the above case the only other case I have found in which this occurs is CPU and GPU faults to same address range racing. This can cause hmm_range_fault to grab a set mixed pages. In this case again we do an eviction remaining page and restart the GPU fault.

I don't have real workloads yet but I do have a very aggressive test case that intentionally does things which could break the design in a highly parallel manner and the design as work. Is it ideal? Maybe not. But getting in a simple design which we can build upon is the current goal.

Matt

Quoted 104 lines
Quoted 103 lines
Quoted 103 lines
Quoted 40 lines

[5] https://patchwork.freedesktop.org/patch/619819/?series=137870&rev=2

Quoted 38 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines

[3] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1110726
[4] https://lore.kernel.org/all/BYAPR11MB3159A304925168D8B6B4671292692@BYAPR11MB3159.namprd11.prod.outlook.com/T/#m89cd6a37778ba5271d5381ebeb03e1f963856a78

Quoted 6 lines
Quoted 6 lines

It would also make the function exported in this patch unnecessary too as non-contiguous pfns can be setup on driver side via migrate_device_pfn_lock and then migrate_device_unmap can be called. This also another eviction usage in GPUSVM, see drm_gpusvm_evict_to_ram in [1].

Quoted 7 lines
Quoted 2 lines

Do you see an issue exporting migrate_device_pfn_lock, migrate_device_unmap?

If there is a good justification for it I can't see a problem with exporting it. That said I don't really understand why you would want/need to split those steps up but I'll wait to see the code.

It is so the device pages returned from hmm_range_fault, which are only guaranteed to be valid under the notifier lock + a seqno check, to be locked and ref taken for migration. migrate_device_unmap() can trigger a MMU invalidation which takes the notifier lock thus calling the function which combines migrate_device_pfn_lock + migrate_device_unmap deadlocks.

I think this flow makes sense and agree in general this likely better than looking at a CPUVMA.

Quoted 6 lines
Quoted 3 lines

I'm still a bit confused about what is better with this flow if you are still calling hmm_range_fault(). How is it better than just calling migrate_vma_setup()? Obviously it will fault the pages in, but it seems

The code in rev2 calls migrate_vma_setup but the requires a struct vm_area_struct argument whereas hmm_range_fault does not.

I'm not sure that's a good enough justfication because the problem isn't whether you're looking up vma's in driver code or mm code. The problem is you are looking up vma's at all and all that goes with that (mainly taking mmap lock, etc.)

Quoted 8 lines
Quoted 5 lines

And for eviction hmm_range_fault() won't even find all the pages because their virtual address may have changed - consider what happens in cases of mremap(), fork(), etc. So eviction really needs physical pages (pfn's), not virtual addresses.

See above, #1 yes we use a physical pages. For #2 it is about making the state consistent within a virtual address range.

Yep, makes sense now. For migration of physical pages you want migrate_device_*, virtual address ranges want migrate_vma_*

- Alistair

Quoted 48 lines

Matt

Quoted 46 lines
Quoted 8 lines
Quoted 3 lines

we're talking about eviction here so I don't understand why that would be relevant. And hmm_range_fault() still requires the VMA, although I need to look at the patches more closely, probably CPUVMA is a DRM

'hmm_range_fault() still requires the VMA' internal yes, but again not as argument. This is about avoiding a driver side lookup of the VMA.

CPUVMA == struct vm_area_struct in this email.

Thanks for the clarification.

- Alistair

Quoted 32 lines

Matt

Quoted 30 lines

specific concept?

Thanks.

- Alistair

Quoted 23 lines

Matt

Quoted 21 lines

- Alistair

Quoted 18 lines

Matt

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2

Quoted 14 lines

Matt

Quoted 12 lines
Quoted 11 lines

+ }
+
+ migrate_device_unmap(src_pfns, npages, NULL);
+
+ return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
/*
* Migrate a device coherent folio back to normal memory. The caller should have
* a reference on folio which will be copied to the new folio if migration is

author maintainer
#17 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

Quoted 256 lines · Matthew Brost
Quoted 246 lines · Alistair Popple
Quoted 236 lines · Matthew Brost
Quoted 227 lines · Alistair Popple
Quoted 197 lines · Matthew Brost
Quoted 139 lines · Alistair Popple
Quoted 127 lines · Matthew Brost
Quoted 126 lines · Alistair Popple
Quoted 122 lines · Matthew Brost
Quoted 63 lines · Alistair Popple
Quoted 2 lines · Matthew Brost
Quoted 1 line · Matthew Brost

On Wed, Oct 16, 2024 at 03:04:06PM +1100, Alistair Popple wrote:

[...]

Quoted 55 lines
Quoted 25 lines
Quoted 24 lines

+{
+ unsigned long i;
+
+ for (i = 0; i < npages; i++) {
+ struct page *page = pfn_to_page(src_pfns[i]);
+
+ if (!get_page_unless_zero(page)) {
+ src_pfns[i] = 0;
+ continue;
+ }
+
+ if (!trylock_page(page)) {
+ src_pfns[i] = 0;
+ put_page(page);
+ continue;
+ }
+
+ src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like
migrate_device_range(). But more importantly this should be split out as
a function that both migrate_device_range() and this function can call
given this bit is identical.

Quoted 3 lines

Missed the folio conversion and agree a helper shared between this function and migrate_device_range would be a good idea. Let add that.

Alistair,

Ok, I think now I want to go slightly different direction here to give GPUSVM a bit more control over several eviction scenarios.

What if I exported the helper discussed above, e.g.,

905 unsigned long migrate_device_pfn_lock(unsigned long pfn) 906 {
907 struct folio *folio;
908
909 folio = folio_get_nontail_page(pfn_to_page(pfn));
910 if (!folio)
911 return 0;
912
913 if (!folio_trylock(folio)) {
914 folio_put(folio);
915 return 0;
916 }
917
918 return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
919 }
920 EXPORT_SYMBOL(migrate_device_pfn_lock);

And then also export migrate_device_unmap.

Quoted 9 lines
Quoted 4 lines

The usage here would be let a driver collect the device pages in virtual address range via hmm_range_fault, lock device pages under notifier lock ensuring device pages are valid, drop the notifier lock and call migrate_device_unmap.

I'm still working through this series but that seems a bit dubious, the locking here is pretty subtle and easy to get wrong so seeing some code would help me a lot in understanding what you're suggesting.

For sure locking in tricky, my mistake on not working through this before sending out the next rev but it came to mind after sending + regarding some late feedback from Thomas about using hmm for eviction [2]. His suggestion of using hmm_range_fault to trigger migration doesn't work for coherent pages, while something like below does.

[2] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1125461

Here is a snippet I have locally which seems to work.

2024 retry:
2025 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2026 hmm_range.hmm_pfns = src;
2027
2028 while (true) {

2029                 mmap_read_lock(mm);
2030                 err = hmm_range_fault(&hmm_range);
2031                 mmap_read_unlock(mm);
2032                 if (err == -EBUSY) {

2033 if (time_after(jiffies, timeout))
2034 break;
2035
2036 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2037 continue;
2038 }
2039 break;
2040 }
2041 if (err)
2042 goto err_put;
2043
2044 drm_gpusvm_notifier_lock(gpusvm);
2045 if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {

2046                 drm_gpusvm_notifier_unlock(gpusvm);
2047                 memset(src, 0, sizeof(*src) * npages);
2048                 goto retry;

2049 }
2050 for (i = 0; i < npages; ++i) {
2051 struct page *page = hmm_pfn_to_page(src[i]);
2052
2053 if (page && (is_device_private_page(page) ||
2054 is_device_coherent_page(page)) && page->zone_device_data)
2055 src[i] = src[i] & ~HMM_PFN_FLAGS;
2056 else
2057 src[i] = 0;
2058 if (src[i])
2059 src[i] = migrate_device_pfn_lock(src[i]);
2060 }
2061 drm_gpusvm_notifier_unlock(gpusvm);

Quoted 8 lines
Quoted 4 lines

Practically for eviction isn't this much the same as calling migrate_vma_setup()? And also for eviction as Sima mentioned you probably shouldn't be looking at mm/vma structs.

hmm_range_fault is just collecting the pages, internally I suppose it does look at a VMA (struct vm_area_struct) but I think the point is drivers should not be looking at VMA here.

Quoted 6 lines

migrate_vma_setup() is designed to be called by drivers and needs a vma, so in general I don't see a problem with drivers looking up vma's. The problem arises specifically for eviction and whether or not that happens in the driver or hmm_range_fault() is pretty irrelevant IMHO for the issues there (see below).

Ok, if you think it ok for drivers to lookup the VMA then purposed exporting of migrate_device_pfn_lock & migrate_device_unmap is not needed, rather just the original function exported in the this patch.

More below too.

Quoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 10 lines

2063 migrate_device_unmap(src, npages, NULL);
...
2101 migrate_device_pages(src, dst, npages);
2102 migrate_device_finalize(src, dst, npages);

Quoted 4 lines
Quoted 3 lines

Sima has strongly suggested avoiding a CPUVMA lookup during eviction cases and this would let me fixup drm_gpusvm_range_evict in [1] to avoid this.

Quoted 6 lines
Quoted 4 lines

That sounds reasonable but for context do you have a link to the comments/discussion on this? I couldn't readily find it, but I may have just missed it.

See in [4], search for '2. eviction' comment from sima.

Thanks for pointing that out. For reference here's Sima's comment:

Quoted 9 lines

2. eviction

Requirements much like migrate_to_ram, because otherwise we break the migration gurantee:

- Only looking at physical memory datastructures and locks, no looking at mm/vma structs or relying on those being locked. We rely entirely on reverse maps from try_to_migrate to find all the mappings on both cpu and gpu side (cpu only zone device swap or migration pte entries ofc).

I also very much agree with this. That's basically why I added migrate_device_range(), so that we can forcibly evict pages when the driver needs them freed (eg. driver unload, low memory, etc.). In general it is impossible to guarantee eviction og all pages using just hmm_range_fault().

Quoted 7 lines
Quoted 2 lines

In this code path we don't have device pages available, hence the purposed collection via hmm_range_fault.

Why don't you have the pfns requiring eviction available? I need to read this series in more depth, but generally hmm_range_fault() can't gurantee you will find every device page.

Quoted 7 lines

There are two cases for eviction in my series:

1. TTM decides it needs to move memory. This calls drm_gpusvm_evict_to_ram. In this case the device pfns are available directly from drm_gpusvm_devmem so the migrate_device_* calls be used here albiet with the new function added in this patch as device pfns may be non-contiguous.

That makes sense and is generally what I think of when I'm thinking of eviction. The new function makes sense too - migrate_device_range() was primarily introduced to allow a driver to evict all device-private pages from a GPU so didn't consider non-contiguous cases, etc.

Quoted 8 lines

2. An inconsistent state for VA range occurs (mixed system and device pages, partial unmap of a range, etc...). Here we want to evict the range ram to make the state consistent. No device pages are available due to an intentional disconnect between a virtual range and physical drm_gpusvm_devmem, thus the device pages have to be looked up. This the function drm_gpusvm_range_evict. Based on what you tell me, it likely is fine the way originally coded in v2 (vma lookup + migrate_vma_*) vs using hmm_range_fault like I have suggested here.

Thanks for the explanation. I think vma lookup + migrate_vma_setup() is fine for this usage and is exactly what you want - it was designed to either select all the system memory pages or device-private pages within a VA range and migrate them.

Quoted 7 lines

FWIW I have toyed with the idea of a combined hmm_range_fault()/migrate_vma_setup() front-end to the rest of the migrate_vma_*() process but haven't come up with something nice as yet. I don't think mixing the two in an open-coded fashion is a good idea though, I'd rather we come up with a new API that addresses the short-comings of migrate_vma_setup().

Quoted 6 lines

I think that would good. Here we actually need to lookup multiple VMAs and have a sequence of migrate_vma_* calls as it possible for VMAs to have changed after the driver range was created. It might be nice to hide the VMA lookup from the drivers with an API saying collect and migrate all pages of a type in a VA range much like hmm_range_fault. If the range spans multiple VMAs that would be hidden from the caller.

Quoted 5 lines

Ok. I wasn't really considering multiple VMAs. UVM and Nouveau don't really have a requirement to migrate across multiple VMAs but if that's neccessary I think an API that hides that specifically for working with migrate_vma_*() might make sense.

We can run into multiple VMA scenarios if a user does something rude like this:

Quoted 19 lines

mmap 0x000000...0x1fffff -> fault migrates 2M to VRAM and creates an internal range to track
munmap 0x080000...0x17ffff -> now we have two VMAs instead of one and the range has a hole in it

In this scenario, which we believe to rare / unsual, we just evict remaining VRAM pages to SRAM, destroy range, and fixup on next GPU fault.

Quoted 12 lines
Quoted 5 lines

Matt

Quoted 3 lines
Quoted 3 lines

Note #2 may be removed or unnecessary at some point if we decide to add support for ininconsistent state in GPU SVM and Xe. Keeping it simple for now though. See 'Ranges with mixed system and device pages' in [5].

As someone not very familiar with some of the DRM layers can I ask why having virtual address ranges with a mix of system and device pages is hard to support? It seems to me that in practice it might be quite difficult to keep a VMA range as exclusively all in system memory or all in device memory.

Sorry double reply - missed 'VMA range as exclusively' comment.

I'm refering to DRM GPU SVM range being exclusive in system or device memory not a VMA. A DRM GPU SVM range is relatively small and controled by the driver, via a table, and VMA boundaries. In Xe, we current support 4k, 64k, or 2M DRM GPU SVM ranges. So say if a VMA is 1 GB, we can have individual 2M ranges in either system or device memory based on access patterns. What range allocation size, that is migration grainularity which all pages in the range being exclusively in system or device memory.

Matt

Quoted 133 lines

A few things that make this difficult are:

- Our (Xe) bind code would need to be updated to support this
- TTM / DRM buddy allocator doesn't support freeing / reallocation of individual pages rather aligned chunks of initial allocation size (e.g., 2M would be common allocation size).
- Spliting ranges would add complications

All workable problems but since we are writing a new common implementation trying to keep it as simple as possible for initial merge of the design. Almost certainly at some point we will add support for mixed ranges to the common GPU SVM layer with a driver choosing if it wants mixed or non-mixed ranges via a flag to function calls.

wrt to being difficult keeping exclusively in system or vram, in addition to the above case the only other case I have found in which this occurs is CPU and GPU faults to same address range racing. This can cause hmm_range_fault to grab a set mixed pages. In this case again we do an eviction remaining page and restart the GPU fault.

I don't have real workloads yet but I do have a very aggressive test case that intentionally does things which could break the design in a highly parallel manner and the design as work. Is it ideal? Maybe not. But getting in a simple design which we can build upon is the current goal.

Matt

Quoted 104 lines
Quoted 103 lines
Quoted 103 lines
Quoted 40 lines

[5] https://patchwork.freedesktop.org/patch/619819/?series=137870&rev=2

Quoted 38 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines

[3] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1110726
[4] https://lore.kernel.org/all/BYAPR11MB3159A304925168D8B6B4671292692@BYAPR11MB3159.namprd11.prod.outlook.com/T/#m89cd6a37778ba5271d5381ebeb03e1f963856a78

Quoted 6 lines
Quoted 6 lines

It would also make the function exported in this patch unnecessary too as non-contiguous pfns can be setup on driver side via migrate_device_pfn_lock and then migrate_device_unmap can be called. This also another eviction usage in GPUSVM, see drm_gpusvm_evict_to_ram in [1].

Quoted 7 lines
Quoted 2 lines

Do you see an issue exporting migrate_device_pfn_lock, migrate_device_unmap?

If there is a good justification for it I can't see a problem with exporting it. That said I don't really understand why you would want/need to split those steps up but I'll wait to see the code.

It is so the device pages returned from hmm_range_fault, which are only guaranteed to be valid under the notifier lock + a seqno check, to be locked and ref taken for migration. migrate_device_unmap() can trigger a MMU invalidation which takes the notifier lock thus calling the function which combines migrate_device_pfn_lock + migrate_device_unmap deadlocks.

I think this flow makes sense and agree in general this likely better than looking at a CPUVMA.

Quoted 6 lines
Quoted 3 lines

I'm still a bit confused about what is better with this flow if you are still calling hmm_range_fault(). How is it better than just calling migrate_vma_setup()? Obviously it will fault the pages in, but it seems

The code in rev2 calls migrate_vma_setup but the requires a struct vm_area_struct argument whereas hmm_range_fault does not.

I'm not sure that's a good enough justfication because the problem isn't whether you're looking up vma's in driver code or mm code. The problem is you are looking up vma's at all and all that goes with that (mainly taking mmap lock, etc.)

Quoted 8 lines
Quoted 5 lines

And for eviction hmm_range_fault() won't even find all the pages because their virtual address may have changed - consider what happens in cases of mremap(), fork(), etc. So eviction really needs physical pages (pfn's), not virtual addresses.

See above, #1 yes we use a physical pages. For #2 it is about making the state consistent within a virtual address range.

Yep, makes sense now. For migration of physical pages you want migrate_device_*, virtual address ranges want migrate_vma_*

- Alistair

Quoted 48 lines

Matt

Quoted 46 lines
Quoted 8 lines
Quoted 3 lines

we're talking about eviction here so I don't understand why that would be relevant. And hmm_range_fault() still requires the VMA, although I need to look at the patches more closely, probably CPUVMA is a DRM

'hmm_range_fault() still requires the VMA' internal yes, but again not as argument. This is about avoiding a driver side lookup of the VMA.

CPUVMA == struct vm_area_struct in this email.

Thanks for the clarification.

- Alistair

Quoted 32 lines

Matt

Quoted 30 lines

specific concept?

Thanks.

- Alistair

Quoted 23 lines

Matt

Quoted 21 lines

- Alistair

Quoted 18 lines

Matt

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2

Quoted 14 lines

Matt

Quoted 12 lines

+ }
+
+ migrate_device_unmap(src_pfns, npages, NULL);
+
+ return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
/*
* Migrate a device coherent folio back to normal memory. The caller should have
* a reference on folio which will be copied to the new folio if migration is

#18 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

In reply to Matthew Brost
Quoted 247 lines · Matthew Brost
Quoted 246 lines · Alistair Popple
Quoted 242 lines · Matthew Brost
Quoted 227 lines · Alistair Popple
Quoted 197 lines · Matthew Brost
Quoted 139 lines · Alistair Popple
Quoted 127 lines · Matthew Brost
Quoted 126 lines · Alistair Popple
Quoted 122 lines · Matthew Brost
Quoted 63 lines · Alistair Popple
Quoted 2 lines · Matthew Brost
Quoted 1 line · Matthew Brost

On Wed, Oct 16, 2024 at 03:04:06PM +1100, Alistair Popple wrote:

[...]

Quoted 55 lines
Quoted 25 lines
Quoted 24 lines

+{
+ unsigned long i;
+
+ for (i = 0; i < npages; i++) {
+ struct page *page = pfn_to_page(src_pfns[i]);
+
+ if (!get_page_unless_zero(page)) {
+ src_pfns[i] = 0;
+ continue;
+ }
+
+ if (!trylock_page(page)) {
+ src_pfns[i] = 0;
+ put_page(page);
+ continue;
+ }
+
+ src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like
migrate_device_range(). But more importantly this should be split out as
a function that both migrate_device_range() and this function can call
given this bit is identical.

Quoted 3 lines

Missed the folio conversion and agree a helper shared between this function and migrate_device_range would be a good idea. Let add that.

Alistair,

Ok, I think now I want to go slightly different direction here to give GPUSVM a bit more control over several eviction scenarios.

What if I exported the helper discussed above, e.g.,

905 unsigned long migrate_device_pfn_lock(unsigned long pfn) 906 {
907 struct folio *folio;
908
909 folio = folio_get_nontail_page(pfn_to_page(pfn));
910 if (!folio)
911 return 0;
912
913 if (!folio_trylock(folio)) {
914 folio_put(folio);
915 return 0;
916 }
917
918 return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
919 }
920 EXPORT_SYMBOL(migrate_device_pfn_lock);

And then also export migrate_device_unmap.

Quoted 9 lines
Quoted 4 lines

The usage here would be let a driver collect the device pages in virtual address range via hmm_range_fault, lock device pages under notifier lock ensuring device pages are valid, drop the notifier lock and call migrate_device_unmap.

I'm still working through this series but that seems a bit dubious, the locking here is pretty subtle and easy to get wrong so seeing some code would help me a lot in understanding what you're suggesting.

For sure locking in tricky, my mistake on not working through this before sending out the next rev but it came to mind after sending + regarding some late feedback from Thomas about using hmm for eviction [2]. His suggestion of using hmm_range_fault to trigger migration doesn't work for coherent pages, while something like below does.

[2] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1125461

Here is a snippet I have locally which seems to work.

2024 retry:
2025 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2026 hmm_range.hmm_pfns = src;
2027
2028 while (true) {

2029                 mmap_read_lock(mm);
2030                 err = hmm_range_fault(&hmm_range);
2031                 mmap_read_unlock(mm);
2032                 if (err == -EBUSY) {

2033 if (time_after(jiffies, timeout))
2034 break;
2035
2036 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2037 continue;
2038 }
2039 break;
2040 }
2041 if (err)
2042 goto err_put;
2043
2044 drm_gpusvm_notifier_lock(gpusvm);
2045 if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {

2046                 drm_gpusvm_notifier_unlock(gpusvm);
2047                 memset(src, 0, sizeof(*src) * npages);
2048                 goto retry;

2049 }
2050 for (i = 0; i < npages; ++i) {
2051 struct page *page = hmm_pfn_to_page(src[i]);
2052
2053 if (page && (is_device_private_page(page) ||
2054 is_device_coherent_page(page)) && page->zone_device_data)
2055 src[i] = src[i] & ~HMM_PFN_FLAGS;
2056 else
2057 src[i] = 0;
2058 if (src[i])
2059 src[i] = migrate_device_pfn_lock(src[i]);
2060 }
2061 drm_gpusvm_notifier_unlock(gpusvm);

Quoted 8 lines
Quoted 4 lines

Practically for eviction isn't this much the same as calling migrate_vma_setup()? And also for eviction as Sima mentioned you probably shouldn't be looking at mm/vma structs.

hmm_range_fault is just collecting the pages, internally I suppose it does look at a VMA (struct vm_area_struct) but I think the point is drivers should not be looking at VMA here.

Quoted 6 lines

migrate_vma_setup() is designed to be called by drivers and needs a vma, so in general I don't see a problem with drivers looking up vma's. The problem arises specifically for eviction and whether or not that happens in the driver or hmm_range_fault() is pretty irrelevant IMHO for the issues there (see below).

Ok, if you think it ok for drivers to lookup the VMA then purposed exporting of migrate_device_pfn_lock & migrate_device_unmap is not needed, rather just the original function exported in the this patch.

More below too.

Quoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 10 lines

2063 migrate_device_unmap(src, npages, NULL);
...
2101 migrate_device_pages(src, dst, npages);
2102 migrate_device_finalize(src, dst, npages);

Quoted 4 lines
Quoted 3 lines

Sima has strongly suggested avoiding a CPUVMA lookup during eviction cases and this would let me fixup drm_gpusvm_range_evict in [1] to avoid this.

Quoted 6 lines
Quoted 4 lines

That sounds reasonable but for context do you have a link to the comments/discussion on this? I couldn't readily find it, but I may have just missed it.

See in [4], search for '2. eviction' comment from sima.

Thanks for pointing that out. For reference here's Sima's comment:

Quoted 9 lines

2. eviction

Requirements much like migrate_to_ram, because otherwise we break the migration gurantee:

- Only looking at physical memory datastructures and locks, no looking at mm/vma structs or relying on those being locked. We rely entirely on reverse maps from try_to_migrate to find all the mappings on both cpu and gpu side (cpu only zone device swap or migration pte entries ofc).

I also very much agree with this. That's basically why I added migrate_device_range(), so that we can forcibly evict pages when the driver needs them freed (eg. driver unload, low memory, etc.). In general it is impossible to guarantee eviction og all pages using just hmm_range_fault().

Quoted 7 lines
Quoted 2 lines

In this code path we don't have device pages available, hence the purposed collection via hmm_range_fault.

Why don't you have the pfns requiring eviction available? I need to read this series in more depth, but generally hmm_range_fault() can't gurantee you will find every device page.

Quoted 7 lines

There are two cases for eviction in my series:

1. TTM decides it needs to move memory. This calls drm_gpusvm_evict_to_ram. In this case the device pfns are available directly from drm_gpusvm_devmem so the migrate_device_* calls be used here albiet with the new function added in this patch as device pfns may be non-contiguous.

That makes sense and is generally what I think of when I'm thinking of eviction. The new function makes sense too - migrate_device_range() was primarily introduced to allow a driver to evict all device-private pages from a GPU so didn't consider non-contiguous cases, etc.

Quoted 8 lines

2. An inconsistent state for VA range occurs (mixed system and device pages, partial unmap of a range, etc...). Here we want to evict the range ram to make the state consistent. No device pages are available due to an intentional disconnect between a virtual range and physical drm_gpusvm_devmem, thus the device pages have to be looked up. This the function drm_gpusvm_range_evict. Based on what you tell me, it likely is fine the way originally coded in v2 (vma lookup + migrate_vma_*) vs using hmm_range_fault like I have suggested here.

Thanks for the explanation. I think vma lookup + migrate_vma_setup() is fine for this usage and is exactly what you want - it was designed to either select all the system memory pages or device-private pages within a VA range and migrate them.

Quoted 7 lines

FWIW I have toyed with the idea of a combined hmm_range_fault()/migrate_vma_setup() front-end to the rest of the migrate_vma_*() process but haven't come up with something nice as yet. I don't think mixing the two in an open-coded fashion is a good idea though, I'd rather we come up with a new API that addresses the short-comings of migrate_vma_setup().

I think that would good. Here we actually need to lookup multiple VMAs and have a sequence of migrate_vma_* calls as it possible for VMAs to have changed after the driver range was created. It might be nice to hide the VMA lookup from the drivers with an API saying collect and migrate all pages of a type in a VA range much like hmm_range_fault. If the range spans multiple VMAs that would be hidden from the caller.

Quoted 8 lines
Quoted 5 lines

Ok. I wasn't really considering multiple VMAs. UVM and Nouveau don't really have a requirement to migrate across multiple VMAs but if that's neccessary I think an API that hides that specifically for working with migrate_vma_*() might make sense.

We can run into multiple VMA scenarios if a user does something rude like this:

fork and mremap were the other "rude" things we've had fun with. They basically mean you can get references to device pages which a driver can't track with virtual addresses.

Quoted 40 lines

mmap 0x000000...0x1fffff -> fault migrates 2M to VRAM and creates an internal range to track
munmap 0x080000...0x17ffff -> now we have two VMAs instead of one and the range has a hole in it

In this scenario, which we believe to rare / unsual, we just evict remaining VRAM pages to SRAM, destroy range, and fixup on next GPU fault.

Quoted 6 lines
Quoted 5 lines

Matt

Quoted 3 lines
Quoted 3 lines

Note #2 may be removed or unnecessary at some point if we decide to add support for ininconsistent state in GPU SVM and Xe. Keeping it simple for now though. See 'Ranges with mixed system and device pages' in [5].

Quoted 6 lines

As someone not very familiar with some of the DRM layers can I ask why having virtual address ranges with a mix of system and device pages is hard to support? It seems to me that in practice it might be quite difficult to keep a VMA range as exclusively all in system memory or all in device memory.

A few things that make this difficult are:

- Our (Xe) bind code would need to be updated to support this
- TTM / DRM buddy allocator doesn't support freeing / reallocation of individual pages rather aligned chunks of initial allocation size (e.g., 2M would be common allocation size).
- Spliting ranges would add complications

All workable problems but since we are writing a new common implementation trying to keep it as simple as possible for initial merge of the design. Almost certainly at some point we will add support for mixed ranges to the common GPU SVM layer with a driver choosing if it wants mixed or non-mixed ranges via a flag to function calls.

wrt to being difficult keeping exclusively in system or vram, in addition to the above case the only other case I have found in which this occurs is CPU and GPU faults to same address range racing. This can cause hmm_range_fault to grab a set mixed pages. In this case again we do an eviction remaining page and restart the GPU fault.

Quoted 5 lines

I don't have real workloads yet but I do have a very aggressive test case that intentionally does things which could break the design in a highly parallel manner and the design as work. Is it ideal? Maybe not. But getting in a simple design which we can build upon is the current goal.

Taking a simple approach first definitely sounds lie the right approach to me. I was just interested in the background because it wasn't something I'd run into (though we built on top of something quite different to the DRM layer). But I have often thought that the interfaces we have between core mm and GPU drivers is still a bit too low level at the moment and is calling out for a slightly higher level common implementation in the middle so am very interested to see where this all goes. Thanks.

- Alistair

Quoted 106 lines

Matt

Quoted 104 lines
Quoted 103 lines
Quoted 103 lines
Quoted 40 lines

[5] https://patchwork.freedesktop.org/patch/619819/?series=137870&rev=2

Quoted 38 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines

[3] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1110726
[4] https://lore.kernel.org/all/BYAPR11MB3159A304925168D8B6B4671292692@BYAPR11MB3159.namprd11.prod.outlook.com/T/#m89cd6a37778ba5271d5381ebeb03e1f963856a78

Quoted 6 lines
Quoted 6 lines

It would also make the function exported in this patch unnecessary too as non-contiguous pfns can be setup on driver side via migrate_device_pfn_lock and then migrate_device_unmap can be called. This also another eviction usage in GPUSVM, see drm_gpusvm_evict_to_ram in [1].

Quoted 7 lines
Quoted 2 lines

Do you see an issue exporting migrate_device_pfn_lock, migrate_device_unmap?

If there is a good justification for it I can't see a problem with exporting it. That said I don't really understand why you would want/need to split those steps up but I'll wait to see the code.

It is so the device pages returned from hmm_range_fault, which are only guaranteed to be valid under the notifier lock + a seqno check, to be locked and ref taken for migration. migrate_device_unmap() can trigger a MMU invalidation which takes the notifier lock thus calling the function which combines migrate_device_pfn_lock + migrate_device_unmap deadlocks.

I think this flow makes sense and agree in general this likely better than looking at a CPUVMA.

Quoted 6 lines
Quoted 3 lines

I'm still a bit confused about what is better with this flow if you are still calling hmm_range_fault(). How is it better than just calling migrate_vma_setup()? Obviously it will fault the pages in, but it seems

The code in rev2 calls migrate_vma_setup but the requires a struct vm_area_struct argument whereas hmm_range_fault does not.

I'm not sure that's a good enough justfication because the problem isn't whether you're looking up vma's in driver code or mm code. The problem is you are looking up vma's at all and all that goes with that (mainly taking mmap lock, etc.)

Quoted 8 lines
Quoted 5 lines

And for eviction hmm_range_fault() won't even find all the pages because their virtual address may have changed - consider what happens in cases of mremap(), fork(), etc. So eviction really needs physical pages (pfn's), not virtual addresses.

See above, #1 yes we use a physical pages. For #2 it is about making the state consistent within a virtual address range.

Yep, makes sense now. For migration of physical pages you want migrate_device_*, virtual address ranges want migrate_vma_*

- Alistair

Quoted 48 lines

Matt

Quoted 46 lines
Quoted 8 lines
Quoted 3 lines

we're talking about eviction here so I don't understand why that would be relevant. And hmm_range_fault() still requires the VMA, although I need to look at the patches more closely, probably CPUVMA is a DRM

'hmm_range_fault() still requires the VMA' internal yes, but again not as argument. This is about avoiding a driver side lookup of the VMA.

CPUVMA == struct vm_area_struct in this email.

Thanks for the clarification.

- Alistair

Quoted 32 lines

Matt

Quoted 30 lines

specific concept?

Thanks.

- Alistair

Quoted 23 lines

Matt

Quoted 21 lines

- Alistair

Quoted 18 lines

Matt

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2

Quoted 14 lines

Matt

Quoted 12 lines

+ }
+
+ migrate_device_unmap(src_pfns, npages, NULL);
+
+ return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
/*
* Migrate a device coherent folio back to normal memory. The caller should have
* a reference on folio which will be copied to the new folio if migration is

author maintainer
#19 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

Quoted 256 lines · Alistair Popple
Quoted 253 lines · Matthew Brost
Quoted 251 lines · Alistair Popple
Quoted 236 lines · Matthew Brost
Quoted 227 lines · Alistair Popple
Quoted 197 lines · Matthew Brost
Quoted 139 lines · Alistair Popple
Quoted 127 lines · Matthew Brost
Quoted 126 lines · Alistair Popple
Quoted 122 lines · Matthew Brost
Quoted 63 lines · Alistair Popple
Quoted 2 lines · Matthew Brost
Quoted 1 line · Matthew Brost

On Wed, Oct 16, 2024 at 03:04:06PM +1100, Alistair Popple wrote:

[...]

Quoted 55 lines
Quoted 28 lines

+{
+ unsigned long i;
+
+ for (i = 0; i < npages; i++) {
+ struct page *page = pfn_to_page(src_pfns[i]);
+
+ if (!get_page_unless_zero(page)) {
+ src_pfns[i] = 0;
+ continue;
+ }
+
+ if (!trylock_page(page)) {
+ src_pfns[i] = 0;
+ put_page(page);
+ continue;
+ }
+
+ src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like
migrate_device_range(). But more importantly this should be split out as
a function that both migrate_device_range() and this function can call
given this bit is identical.

Missed the folio conversion and agree a helper shared between this
function and migrate_device_range would be a good idea. Let add that.

Alistair,

Ok, I think now I want to go slightly different direction here to give GPUSVM a bit more control over several eviction scenarios.

What if I exported the helper discussed above, e.g.,

905 unsigned long migrate_device_pfn_lock(unsigned long pfn) 906 {
907 struct folio *folio;
908
909 folio = folio_get_nontail_page(pfn_to_page(pfn));
910 if (!folio)
911 return 0;
912
913 if (!folio_trylock(folio)) {
914 folio_put(folio);
915 return 0;
916 }
917
918 return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
919 }
920 EXPORT_SYMBOL(migrate_device_pfn_lock);

And then also export migrate_device_unmap.

Quoted 9 lines
Quoted 4 lines

The usage here would be let a driver collect the device pages in virtual address range via hmm_range_fault, lock device pages under notifier lock ensuring device pages are valid, drop the notifier lock and call migrate_device_unmap.

I'm still working through this series but that seems a bit dubious, the locking here is pretty subtle and easy to get wrong so seeing some code would help me a lot in understanding what you're suggesting.

For sure locking in tricky, my mistake on not working through this before sending out the next rev but it came to mind after sending + regarding some late feedback from Thomas about using hmm for eviction [2]. His suggestion of using hmm_range_fault to trigger migration doesn't work for coherent pages, while something like below does.

[2] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1125461

Here is a snippet I have locally which seems to work.

2024 retry:
2025 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2026 hmm_range.hmm_pfns = src;
2027
2028 while (true) {

2029                 mmap_read_lock(mm);
2030                 err = hmm_range_fault(&hmm_range);
2031                 mmap_read_unlock(mm);
2032                 if (err == -EBUSY) {

2033 if (time_after(jiffies, timeout))
2034 break;
2035
2036 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2037 continue;
2038 }
2039 break;
2040 }
2041 if (err)
2042 goto err_put;
2043
2044 drm_gpusvm_notifier_lock(gpusvm);
2045 if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {

2046                 drm_gpusvm_notifier_unlock(gpusvm);
2047                 memset(src, 0, sizeof(*src) * npages);
2048                 goto retry;

2049 }
2050 for (i = 0; i < npages; ++i) {
2051 struct page *page = hmm_pfn_to_page(src[i]);
2052
2053 if (page && (is_device_private_page(page) ||
2054 is_device_coherent_page(page)) && page->zone_device_data)
2055 src[i] = src[i] & ~HMM_PFN_FLAGS;
2056 else
2057 src[i] = 0;
2058 if (src[i])
2059 src[i] = migrate_device_pfn_lock(src[i]);
2060 }
2061 drm_gpusvm_notifier_unlock(gpusvm);

Quoted 8 lines
Quoted 4 lines

Practically for eviction isn't this much the same as calling migrate_vma_setup()? And also for eviction as Sima mentioned you probably shouldn't be looking at mm/vma structs.

hmm_range_fault is just collecting the pages, internally I suppose it does look at a VMA (struct vm_area_struct) but I think the point is drivers should not be looking at VMA here.

Quoted 6 lines

migrate_vma_setup() is designed to be called by drivers and needs a vma, so in general I don't see a problem with drivers looking up vma's. The problem arises specifically for eviction and whether or not that happens in the driver or hmm_range_fault() is pretty irrelevant IMHO for the issues there (see below).

Ok, if you think it ok for drivers to lookup the VMA then purposed exporting of migrate_device_pfn_lock & migrate_device_unmap is not needed, rather just the original function exported in the this patch.

More below too.

Quoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 10 lines

2063 migrate_device_unmap(src, npages, NULL);
...
2101 migrate_device_pages(src, dst, npages);
2102 migrate_device_finalize(src, dst, npages);

Quoted 4 lines
Quoted 3 lines

Sima has strongly suggested avoiding a CPUVMA lookup during eviction cases and this would let me fixup drm_gpusvm_range_evict in [1] to avoid this.

Quoted 6 lines
Quoted 4 lines

That sounds reasonable but for context do you have a link to the comments/discussion on this? I couldn't readily find it, but I may have just missed it.

See in [4], search for '2. eviction' comment from sima.

Thanks for pointing that out. For reference here's Sima's comment:

Quoted 9 lines

2. eviction

Requirements much like migrate_to_ram, because otherwise we break the migration gurantee:

- Only looking at physical memory datastructures and locks, no looking at mm/vma structs or relying on those being locked. We rely entirely on reverse maps from try_to_migrate to find all the mappings on both cpu and gpu side (cpu only zone device swap or migration pte entries ofc).

I also very much agree with this. That's basically why I added migrate_device_range(), so that we can forcibly evict pages when the driver needs them freed (eg. driver unload, low memory, etc.). In general it is impossible to guarantee eviction og all pages using just hmm_range_fault().

Quoted 7 lines
Quoted 2 lines

In this code path we don't have device pages available, hence the purposed collection via hmm_range_fault.

Why don't you have the pfns requiring eviction available? I need to read this series in more depth, but generally hmm_range_fault() can't gurantee you will find every device page.

Quoted 7 lines

There are two cases for eviction in my series:

1. TTM decides it needs to move memory. This calls drm_gpusvm_evict_to_ram. In this case the device pfns are available directly from drm_gpusvm_devmem so the migrate_device_* calls be used here albiet with the new function added in this patch as device pfns may be non-contiguous.

That makes sense and is generally what I think of when I'm thinking of eviction. The new function makes sense too - migrate_device_range() was primarily introduced to allow a driver to evict all device-private pages from a GPU so didn't consider non-contiguous cases, etc.

Quoted 8 lines

2. An inconsistent state for VA range occurs (mixed system and device pages, partial unmap of a range, etc...). Here we want to evict the range ram to make the state consistent. No device pages are available due to an intentional disconnect between a virtual range and physical drm_gpusvm_devmem, thus the device pages have to be looked up. This the function drm_gpusvm_range_evict. Based on what you tell me, it likely is fine the way originally coded in v2 (vma lookup + migrate_vma_*) vs using hmm_range_fault like I have suggested here.

Thanks for the explanation. I think vma lookup + migrate_vma_setup() is fine for this usage and is exactly what you want - it was designed to either select all the system memory pages or device-private pages within a VA range and migrate them.

Quoted 7 lines

FWIW I have toyed with the idea of a combined hmm_range_fault()/migrate_vma_setup() front-end to the rest of the migrate_vma_*() process but haven't come up with something nice as yet. I don't think mixing the two in an open-coded fashion is a good idea though, I'd rather we come up with a new API that addresses the short-comings of migrate_vma_setup().

Quoted 6 lines

I think that would good. Here we actually need to lookup multiple VMAs and have a sequence of migrate_vma_* calls as it possible for VMAs to have changed after the driver range was created. It might be nice to hide the VMA lookup from the drivers with an API saying collect and migrate all pages of a type in a VA range much like hmm_range_fault. If the range spans multiple VMAs that would be hidden from the caller.

Ok. I wasn't really considering multiple VMAs. UVM and Nouveau don't really have a requirement to migrate across multiple VMAs but if that's neccessary I think an API that hides that specifically for working with migrate_vma_*() might make sense.

Quoted 7 lines
Quoted 2 lines

We can run into multiple VMA scenarios if a user does something rude like this:

fork and mremap were the other "rude" things we've had fun with. They basically mean you can get references to device pages which a driver can't track with virtual addresses.

Yes, I've tested those two and are fun as well. But both are COW which hasn't turned out to be too difficult.

Quoted 46 lines
Quoted 40 lines

mmap 0x000000...0x1fffff -> fault migrates 2M to VRAM and creates an internal range to track
munmap 0x080000...0x17ffff -> now we have two VMAs instead of one and the range has a hole in it

In this scenario, which we believe to rare / unsual, we just evict remaining VRAM pages to SRAM, destroy range, and fixup on next GPU fault.

Quoted 6 lines
Quoted 5 lines

Matt

Quoted 3 lines
Quoted 3 lines

Note #2 may be removed or unnecessary at some point if we decide to add support for ininconsistent state in GPU SVM and Xe. Keeping it simple for now though. See 'Ranges with mixed system and device pages' in [5].

Quoted 6 lines

As someone not very familiar with some of the DRM layers can I ask why having virtual address ranges with a mix of system and device pages is hard to support? It seems to me that in practice it might be quite difficult to keep a VMA range as exclusively all in system memory or all in device memory.

A few things that make this difficult are:

- Our (Xe) bind code would need to be updated to support this
- TTM / DRM buddy allocator doesn't support freeing / reallocation of individual pages rather aligned chunks of initial allocation size (e.g., 2M would be common allocation size).
- Spliting ranges would add complications

All workable problems but since we are writing a new common implementation trying to keep it as simple as possible for initial merge of the design. Almost certainly at some point we will add support for mixed ranges to the common GPU SVM layer with a driver choosing if it wants mixed or non-mixed ranges via a flag to function calls.

wrt to being difficult keeping exclusively in system or vram, in addition to the above case the only other case I have found in which this occurs is CPU and GPU faults to same address range racing. This can cause hmm_range_fault to grab a set mixed pages. In this case again we do an eviction remaining page and restart the GPU fault.

Quoted 5 lines

I don't have real workloads yet but I do have a very aggressive test case that intentionally does things which could break the design in a highly parallel manner and the design as work. Is it ideal? Maybe not. But getting in a simple design which we can build upon is the current goal.

Quoted 8 lines

Taking a simple approach first definitely sounds lie the right approach to me. I was just interested in the background because it wasn't something I'd run into (though we built on top of something quite different to the DRM layer). But I have often thought that the interfaces we have between core mm and GPU drivers is still a bit too low level at the moment and is calling out for a slightly higher level common implementation in the middle so am very interested to see where this all goes. Thanks.

The idea is build something common which other DRM drivers can use or perhaps even pull out of the DRM layer eventually into a common device layer for SVM.

Matt

Quoted 110 lines

- Alistair

Quoted 106 lines

Matt

Quoted 104 lines
Quoted 103 lines
Quoted 103 lines
Quoted 40 lines

[5] https://patchwork.freedesktop.org/patch/619819/?series=137870&rev=2

Quoted 38 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines

[3] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1110726
[4] https://lore.kernel.org/all/BYAPR11MB3159A304925168D8B6B4671292692@BYAPR11MB3159.namprd11.prod.outlook.com/T/#m89cd6a37778ba5271d5381ebeb03e1f963856a78

Quoted 6 lines
Quoted 6 lines

It would also make the function exported in this patch unnecessary too as non-contiguous pfns can be setup on driver side via migrate_device_pfn_lock and then migrate_device_unmap can be called. This also another eviction usage in GPUSVM, see drm_gpusvm_evict_to_ram in [1].

Quoted 7 lines
Quoted 2 lines

Do you see an issue exporting migrate_device_pfn_lock, migrate_device_unmap?

If there is a good justification for it I can't see a problem with exporting it. That said I don't really understand why you would want/need to split those steps up but I'll wait to see the code.

It is so the device pages returned from hmm_range_fault, which are only guaranteed to be valid under the notifier lock + a seqno check, to be locked and ref taken for migration. migrate_device_unmap() can trigger a MMU invalidation which takes the notifier lock thus calling the function which combines migrate_device_pfn_lock + migrate_device_unmap deadlocks.

I think this flow makes sense and agree in general this likely better than looking at a CPUVMA.

Quoted 6 lines
Quoted 3 lines

I'm still a bit confused about what is better with this flow if you are still calling hmm_range_fault(). How is it better than just calling migrate_vma_setup()? Obviously it will fault the pages in, but it seems

The code in rev2 calls migrate_vma_setup but the requires a struct vm_area_struct argument whereas hmm_range_fault does not.

I'm not sure that's a good enough justfication because the problem isn't whether you're looking up vma's in driver code or mm code. The problem is you are looking up vma's at all and all that goes with that (mainly taking mmap lock, etc.)

Quoted 8 lines
Quoted 5 lines

And for eviction hmm_range_fault() won't even find all the pages because their virtual address may have changed - consider what happens in cases of mremap(), fork(), etc. So eviction really needs physical pages (pfn's), not virtual addresses.

See above, #1 yes we use a physical pages. For #2 it is about making the state consistent within a virtual address range.

Yep, makes sense now. For migration of physical pages you want migrate_device_*, virtual address ranges want migrate_vma_*

- Alistair

Quoted 48 lines

Matt

Quoted 46 lines
Quoted 8 lines
Quoted 3 lines

we're talking about eviction here so I don't understand why that would be relevant. And hmm_range_fault() still requires the VMA, although I need to look at the patches more closely, probably CPUVMA is a DRM

'hmm_range_fault() still requires the VMA' internal yes, but again not as argument. This is about avoiding a driver side lookup of the VMA.

CPUVMA == struct vm_area_struct in this email.

Thanks for the clarification.

- Alistair

Quoted 32 lines

Matt

Quoted 30 lines

specific concept?

Thanks.

- Alistair

Quoted 23 lines

Matt

Quoted 21 lines

- Alistair

Quoted 18 lines

Matt

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2

Quoted 14 lines

Matt

+ }
+
+ migrate_device_unmap(src_pfns, npages, NULL);
+
+ return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
/*
* Migrate a device coherent folio back to normal memory. The caller should have
* a reference on folio which will be copied to the new folio if migration is

#20 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

In reply to Alistair Popple

Hi,

Quoted 204 lines · Alistair Popple
Quoted 177 lines · Matthew Brost
Quoted 124 lines · Alistair Popple
Quoted 122 lines · Matthew Brost
Quoted 121 lines · Alistair Popple
Quoted 102 lines · Matthew Brost
Quoted 58 lines · Alistair Popple
Quoted 2 lines · Matthew Brost
Quoted 1 line · Matthew Brost

On Wed, Oct 16, 2024 at 03:04:06PM +1100, Alistair Popple wrote:

[...]

Quoted 52 lines
Quoted 23 lines
Quoted 14 lines
Quoted 14 lines
(quoted diff) +14 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; i++) {
+		struct page *page = pfn_to_page(src_pfns[i]);
+
+		if (!get_page_unless_zero(page)) {
+			src_pfns[i] = 0;
+			continue;
+		}
+
+		if (!trylock_page(page)) {
+			src_pfns[i] = 0;
+			put_page(page);
Quoted 9 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+			continue;
+		}
+
+		src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like migrate_device_range(). But more importantly this should be split out as a function that both migrate_device_range() and this function can call given this bit is identical.

Quoted 3 lines

Missed the folio conversion and agree a helper shared between this function and migrate_device_range would be a good idea. Let add that.

Alistair,

Ok, I think now I want to go slightly different direction here to give GPUSVM a bit more control over several eviction scenarios.

What if I exported the helper discussed above, e.g.,

905 unsigned long migrate_device_pfn_lock(unsigned long pfn) 906 {
907 struct folio *folio;
908
909 folio = folio_get_nontail_page(pfn_to_page(pfn));
910 if (!folio)
911 return 0;
912
913 if (!folio_trylock(folio)) {
914 folio_put(folio);
915 return 0;
916 }
917
918 return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
919 }
920 EXPORT_SYMBOL(migrate_device_pfn_lock);

And then also export migrate_device_unmap.

Quoted 8 lines
Quoted 4 lines

The usage here would be let a driver collect the device pages in virtual address range via hmm_range_fault, lock device pages under notifier lock ensuring device pages are valid, drop the notifier lock and call migrate_device_unmap.

I'm still working through this series but that seems a bit dubious, the locking here is pretty subtle and easy to get wrong so seeing some code would help me a lot in understanding what you're suggesting.

For sure locking in tricky, my mistake on not working through this before sending out the next rev but it came to mind after sending + regarding some late feedback from Thomas about using hmm for eviction [2]. His suggestion of using hmm_range_fault to trigger migration doesn't work for coherent pages, while something like below does.

[2] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1125461

Here is a snippet I have locally which seems to work.

2024 retry:
2025 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2026 hmm_range.hmm_pfns = src;
2027
2028 while (true) {

2029                 mmap_read_lock(mm);
2030                 err = hmm_range_fault(&hmm_range);
2031                 mmap_read_unlock(mm);
2032                 if (err == -EBUSY) {

2033 if (time_after(jiffies, timeout))
2034 break;
2035
2036 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2037 continue;
2038 }
2039 break;
2040 }
2041 if (err)
2042 goto err_put;
2043
2044 drm_gpusvm_notifier_lock(gpusvm);
2045 if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {

2046                 drm_gpusvm_notifier_unlock(gpusvm);
2047                 memset(src, 0, sizeof(*src) * npages);
2048                 goto retry;
Quoted 13 lines

2049 }
2050 for (i = 0; i < npages; ++i) {
2051 struct page *page = hmm_pfn_to_page(src[i]);
2052
2053 if (page && (is_device_private_page(page) ||
2054 is_device_coherent_page(page)) && page->zone_device_data)
2055 src[i] = src[i] & ~HMM_PFN_FLAGS;
2056 else
2057 src[i] = 0;
2058 if (src[i])
2059 src[i] = migrate_device_pfn_lock(src[i]);
2060 }
2061 drm_gpusvm_notifier_unlock(gpusvm);

Practically for eviction isn't this much the same as calling migrate_vma_setup()? And also for eviction as Sima mentioned you probably shouldn't be looking at mm/vma structs.

Quoted 9 lines
Quoted 3 lines

hmm_range_fault is just collecting the pages, internally I suppose it does look at a VMA (struct vm_area_struct) but I think the point is drivers should not be looking at VMA here.

migrate_vma_setup() is designed to be called by drivers and needs a vma, so in general I don't see a problem with drivers looking up vma's. The problem arises specifically for eviction and whether or not that happens in the driver or hmm_range_fault() is pretty irrelevant IMHO for the issues there (see below).

Ok, if you think it ok for drivers to lookup the VMA then purposed exporting of migrate_device_pfn_lock & migrate_device_unmap is not needed, rather just the original function exported in the this patch.

More below too.

Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
Quoted 6 lines

2063 migrate_device_unmap(src, npages, NULL);
...
2101 migrate_device_pages(src, dst, npages);
2102 migrate_device_finalize(src, dst, npages);

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

Sima has strongly suggested avoiding a CPUVMA lookup during eviction cases and this would let me fixup drm_gpusvm_range_evict in [1] to avoid this.

That sounds reasonable but for context do you have a link to the comments/discussion on this? I couldn't readily find it, but I may have just missed it.

See in [4], search for '2. eviction' comment from sima.

Thanks for pointing that out. For reference here's Sima's comment:

Quoted 9 lines

2. eviction

Requirements much like migrate_to_ram, because otherwise we break the migration gurantee:

- Only looking at physical memory datastructures and locks, no looking at mm/vma structs or relying on those being locked. We rely entirely on reverse maps from try_to_migrate to find all the mappings on both cpu and gpu side (cpu only zone device swap or migration pte entries ofc).

I also very much agree with this. That's basically why I added migrate_device_range(), so that we can forcibly evict pages when the driver needs them freed (eg. driver unload, low memory, etc.). In general it is impossible to guarantee eviction og all pages using just hmm_range_fault().

Quoted 6 lines
Quoted 2 lines

In this code path we don't have device pages available, hence the purposed collection via hmm_range_fault.

Why don't you have the pfns requiring eviction available? I need to read this series in more depth, but generally hmm_range_fault() can't gurantee you will find every device page.

Quoted 7 lines

There are two cases for eviction in my series:

1. TTM decides it needs to move memory. This calls drm_gpusvm_evict_to_ram. In this case the device pfns are available directly from drm_gpusvm_devmem so the migrate_device_* calls be used here albiet with the new function added in this patch as device pfns may be non-contiguous.

That makes sense and is generally what I think of when I'm thinking of eviction. The new function makes sense too - migrate_device_range() was primarily introduced to allow a driver to evict all device-private pages from a GPU so didn't consider non-contiguous cases, etc.

Quoted 8 lines

2. An inconsistent state for VA range occurs (mixed system and device pages, partial unmap of a range, etc...). Here we want to evict the range ram to make the state consistent. No device pages are available due to an intentional disconnect between a virtual range and physical drm_gpusvm_devmem, thus the device pages have to be looked up. This the function drm_gpusvm_range_evict. Based on what you tell me, it likely is fine the way originally coded in v2 (vma lookup + migrate_vma_*) vs using hmm_range_fault like I have suggested here.

Thanks for the explanation. I think vma lookup + migrate_vma_setup() is fine for this usage and is exactly what you want - it was designed to either select all the system memory pages or device-private pages within a VA range and migrate them.

Quoted 6 lines

FWIW I have toyed with the idea of a combined hmm_range_fault()/migrate_vma_setup() front-end to the rest of the migrate_vma_*() process but haven't come up with something nice as yet. I don't think mixing the two in an open-coded fashion is a good idea though, I'd rather we come up with a new API that addresses the short-comings of migrate_vma_setup().

This is what I have been implementing and have a WIP version now, will cleanup, test and send soon.

It does the migration entry installing while faulting pages, and you continue migrate with normal migrate_vma_() flow.

Quoted 82 lines
Quoted 39 lines

Note #2 may be removed or unnecessary at some point if we decide to add support for ininconsistent state in GPU SVM and Xe. Keeping it simple for now though. See 'Ranges with mixed system and device pages' in [5].

[5] https://patchwork.freedesktop.org/patch/619819/?series=137870&rev=2

Quoted 33 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

[3] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1110726
[4] https://lore.kernel.org/all/BYAPR11MB3159A304925168D8B6B4671292692@BYAPR11MB3159.namprd11.prod.outlook.com/T/#m89cd6a37778ba5271d5381ebeb03e1f963856a78

Quoted 6 lines
Quoted 6 lines

It would also make the function exported in this patch unnecessary too as non-contiguous pfns can be setup on driver side via migrate_device_pfn_lock and then migrate_device_unmap can be called. This also another eviction usage in GPUSVM, see drm_gpusvm_evict_to_ram in [1].

Quoted 6 lines
Quoted 2 lines

Do you see an issue exporting migrate_device_pfn_lock, migrate_device_unmap?

If there is a good justification for it I can't see a problem with exporting it. That said I don't really understand why you would want/need to split those steps up but I'll wait to see the code.

It is so the device pages returned from hmm_range_fault, which are only guaranteed to be valid under the notifier lock + a seqno check, to be locked and ref taken for migration. migrate_device_unmap() can trigger a MMU invalidation which takes the notifier lock thus calling the function which combines migrate_device_pfn_lock + migrate_device_unmap deadlocks.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I think this flow makes sense and agree in general this likely better than looking at a CPUVMA.

I'm still a bit confused about what is better with this flow if you are still calling hmm_range_fault(). How is it better than just calling migrate_vma_setup()? Obviously it will fault the pages in, but it seems

The code in rev2 calls migrate_vma_setup but the requires a struct vm_area_struct argument whereas hmm_range_fault does not.

I'm not sure that's a good enough justfication because the problem isn't whether you're looking up vma's in driver code or mm code. The problem is you are looking up vma's at all and all that goes with that (mainly taking mmap lock, etc.)

Quoted 7 lines
Quoted 5 lines

And for eviction hmm_range_fault() won't even find all the pages because their virtual address may have changed - consider what happens in cases of mremap(), fork(), etc. So eviction really needs physical pages (pfn's), not virtual addresses.

See above, #1 yes we use a physical pages. For #2 it is about making the state consistent within a virtual address range.

Yep, makes sense now. For migration of physical pages you want migrate_device_*, virtual address ranges want migrate_vma_*

- Alistair

Quoted 31 lines

Matt

Quoted 29 lines
Quoted 7 lines
Quoted 3 lines

we're talking about eviction here so I don't understand why that would be relevant. And hmm_range_fault() still requires the VMA, although I need to look at the patches more closely, probably CPUVMA is a DRM

'hmm_range_fault() still requires the VMA' internal yes, but again not as argument. This is about avoiding a driver side lookup of the VMA.

CPUVMA == struct vm_area_struct in this email.

Thanks for the clarification.

- Alistair

Quoted 18 lines

Matt

Quoted 16 lines

specific concept?

Thanks.

- Alistair

Quoted 10 lines

Matt

Quoted 8 lines

- Alistair

Quoted 6 lines

Matt

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2

Quoted 2 lines

Matt

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

Review

Quoted 11 lines
(quoted diff) +8 -0
+	}
+
+	migrate_device_unmap(src_pfns, npages, NULL);
+
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
 /*
  * Migrate a device coherent folio back to normal memory. The caller should have
  * a reference on folio which will be copied to the new folio if migration is

--Mika

#21 lore raw

Re: [PATCH v2 02/29] mm/migrate: Add migrate_device_prepopulated_range

Quoted 214 lines · Mika Penttilä

Hi,

Quoted 204 lines · Alistair Popple
Quoted 177 lines · Matthew Brost
Quoted 124 lines · Alistair Popple
Quoted 122 lines · Matthew Brost
Quoted 121 lines · Alistair Popple
Quoted 102 lines · Matthew Brost
Quoted 58 lines · Alistair Popple
Quoted 2 lines · Matthew Brost
Quoted 1 line · Matthew Brost

On Wed, Oct 16, 2024 at 03:04:06PM +1100, Alistair Popple wrote:

[...]

Quoted 52 lines
Quoted 23 lines
Quoted 14 lines
Quoted 14 lines
(quoted diff) +14 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; i++) {
+		struct page *page = pfn_to_page(src_pfns[i]);
+
+		if (!get_page_unless_zero(page)) {
+			src_pfns[i] = 0;
+			continue;
+		}
+
+		if (!trylock_page(page)) {
+			src_pfns[i] = 0;
+			put_page(page);
Quoted 9 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+			continue;
+		}
+
+		src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;

This needs to be converted to use a folio like migrate_device_range(). But more importantly this should be split out as a function that both migrate_device_range() and this function can call given this bit is identical.

Quoted 3 lines

Missed the folio conversion and agree a helper shared between this function and migrate_device_range would be a good idea. Let add that.

Alistair,

Ok, I think now I want to go slightly different direction here to give GPUSVM a bit more control over several eviction scenarios.

What if I exported the helper discussed above, e.g.,

905 unsigned long migrate_device_pfn_lock(unsigned long pfn) 906 {
907 struct folio *folio;
908
909 folio = folio_get_nontail_page(pfn_to_page(pfn));
910 if (!folio)
911 return 0;
912
913 if (!folio_trylock(folio)) {
914 folio_put(folio);
915 return 0;
916 }
917
918 return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
919 }
920 EXPORT_SYMBOL(migrate_device_pfn_lock);

And then also export migrate_device_unmap.

Quoted 8 lines
Quoted 4 lines

The usage here would be let a driver collect the device pages in virtual address range via hmm_range_fault, lock device pages under notifier lock ensuring device pages are valid, drop the notifier lock and call migrate_device_unmap.

I'm still working through this series but that seems a bit dubious, the locking here is pretty subtle and easy to get wrong so seeing some code would help me a lot in understanding what you're suggesting.

For sure locking in tricky, my mistake on not working through this before sending out the next rev but it came to mind after sending + regarding some late feedback from Thomas about using hmm for eviction [2]. His suggestion of using hmm_range_fault to trigger migration doesn't work for coherent pages, while something like below does.

[2] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1125461

Here is a snippet I have locally which seems to work.

2024 retry:
2025 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2026 hmm_range.hmm_pfns = src;
2027
2028 while (true) {

2029                 mmap_read_lock(mm);
2030                 err = hmm_range_fault(&hmm_range);
2031                 mmap_read_unlock(mm);
2032                 if (err == -EBUSY) {

2033 if (time_after(jiffies, timeout))
2034 break;
2035
2036 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
2037 continue;
2038 }
2039 break;
2040 }
2041 if (err)
2042 goto err_put;
2043
2044 drm_gpusvm_notifier_lock(gpusvm);
2045 if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {

2046                 drm_gpusvm_notifier_unlock(gpusvm);
2047                 memset(src, 0, sizeof(*src) * npages);
2048                 goto retry;
Quoted 13 lines

2049 }
2050 for (i = 0; i < npages; ++i) {
2051 struct page *page = hmm_pfn_to_page(src[i]);
2052
2053 if (page && (is_device_private_page(page) ||
2054 is_device_coherent_page(page)) && page->zone_device_data)
2055 src[i] = src[i] & ~HMM_PFN_FLAGS;
2056 else
2057 src[i] = 0;
2058 if (src[i])
2059 src[i] = migrate_device_pfn_lock(src[i]);
2060 }
2061 drm_gpusvm_notifier_unlock(gpusvm);

Practically for eviction isn't this much the same as calling migrate_vma_setup()? And also for eviction as Sima mentioned you probably shouldn't be looking at mm/vma structs.

Quoted 9 lines
Quoted 3 lines

hmm_range_fault is just collecting the pages, internally I suppose it does look at a VMA (struct vm_area_struct) but I think the point is drivers should not be looking at VMA here.

migrate_vma_setup() is designed to be called by drivers and needs a vma, so in general I don't see a problem with drivers looking up vma's. The problem arises specifically for eviction and whether or not that happens in the driver or hmm_range_fault() is pretty irrelevant IMHO for the issues there (see below).

Ok, if you think it ok for drivers to lookup the VMA then purposed exporting of migrate_device_pfn_lock & migrate_device_unmap is not needed, rather just the original function exported in the this patch.

More below too.

Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
Quoted 6 lines

2063 migrate_device_unmap(src, npages, NULL);
...
2101 migrate_device_pages(src, dst, npages);
2102 migrate_device_finalize(src, dst, npages);

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

Sima has strongly suggested avoiding a CPUVMA lookup during eviction cases and this would let me fixup drm_gpusvm_range_evict in [1] to avoid this.

That sounds reasonable but for context do you have a link to the comments/discussion on this? I couldn't readily find it, but I may have just missed it.

See in [4], search for '2. eviction' comment from sima.

Thanks for pointing that out. For reference here's Sima's comment:

Quoted 9 lines

2. eviction

Requirements much like migrate_to_ram, because otherwise we break the migration gurantee:

- Only looking at physical memory datastructures and locks, no looking at mm/vma structs or relying on those being locked. We rely entirely on reverse maps from try_to_migrate to find all the mappings on both cpu and gpu side (cpu only zone device swap or migration pte entries ofc).

I also very much agree with this. That's basically why I added migrate_device_range(), so that we can forcibly evict pages when the driver needs them freed (eg. driver unload, low memory, etc.). In general it is impossible to guarantee eviction og all pages using just hmm_range_fault().

Quoted 6 lines
Quoted 2 lines

In this code path we don't have device pages available, hence the purposed collection via hmm_range_fault.

Why don't you have the pfns requiring eviction available? I need to read this series in more depth, but generally hmm_range_fault() can't gurantee you will find every device page.

Quoted 7 lines

There are two cases for eviction in my series:

1. TTM decides it needs to move memory. This calls drm_gpusvm_evict_to_ram. In this case the device pfns are available directly from drm_gpusvm_devmem so the migrate_device_* calls be used here albiet with the new function added in this patch as device pfns may be non-contiguous.

That makes sense and is generally what I think of when I'm thinking of eviction. The new function makes sense too - migrate_device_range() was primarily introduced to allow a driver to evict all device-private pages from a GPU so didn't consider non-contiguous cases, etc.

Quoted 8 lines

2. An inconsistent state for VA range occurs (mixed system and device pages, partial unmap of a range, etc...). Here we want to evict the range ram to make the state consistent. No device pages are available due to an intentional disconnect between a virtual range and physical drm_gpusvm_devmem, thus the device pages have to be looked up. This the function drm_gpusvm_range_evict. Based on what you tell me, it likely is fine the way originally coded in v2 (vma lookup + migrate_vma_*) vs using hmm_range_fault like I have suggested here.

Thanks for the explanation. I think vma lookup + migrate_vma_setup() is fine for this usage and is exactly what you want - it was designed to either select all the system memory pages or device-private pages within a VA range and migrate them.

Quoted 6 lines

FWIW I have toyed with the idea of a combined hmm_range_fault()/migrate_vma_setup() front-end to the rest of the migrate_vma_*() process but haven't come up with something nice as yet. I don't think mixing the two in an open-coded fashion is a good idea though, I'd rather we come up with a new API that addresses the short-comings of migrate_vma_setup().

Quoted 5 lines

This is what I have been implementing and have a WIP version now, will cleanup, test and send soon.

It does the migration entry installing while faulting pages, and you continue migrate with normal migrate_vma_() flow.

Oh nice! Thanks for looking further into that idea, I'm looking forward to seeing the results. For background Mika and I had an offline discussion about this a little while back but I wasn't sure if it had gone anywhere.

Quoted 95 lines
Quoted 82 lines
Quoted 39 lines

Note #2 may be removed or unnecessary at some point if we decide to add support for ininconsistent state in GPU SVM and Xe. Keeping it simple for now though. See 'Ranges with mixed system and device pages' in [5].

[5] https://patchwork.freedesktop.org/patch/619819/?series=137870&rev=2

Quoted 33 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

[3] https://patchwork.freedesktop.org/patch/610957/?series=137870&rev=1#comment_1110726
[4] https://lore.kernel.org/all/BYAPR11MB3159A304925168D8B6B4671292692@BYAPR11MB3159.namprd11.prod.outlook.com/T/#m89cd6a37778ba5271d5381ebeb03e1f963856a78

Quoted 6 lines
Quoted 6 lines

It would also make the function exported in this patch unnecessary too as non-contiguous pfns can be setup on driver side via migrate_device_pfn_lock and then migrate_device_unmap can be called. This also another eviction usage in GPUSVM, see drm_gpusvm_evict_to_ram in [1].

Quoted 6 lines
Quoted 2 lines

Do you see an issue exporting migrate_device_pfn_lock, migrate_device_unmap?

If there is a good justification for it I can't see a problem with exporting it. That said I don't really understand why you would want/need to split those steps up but I'll wait to see the code.

It is so the device pages returned from hmm_range_fault, which are only guaranteed to be valid under the notifier lock + a seqno check, to be locked and ref taken for migration. migrate_device_unmap() can trigger a MMU invalidation which takes the notifier lock thus calling the function which combines migrate_device_pfn_lock + migrate_device_unmap deadlocks.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I think this flow makes sense and agree in general this likely better than looking at a CPUVMA.

I'm still a bit confused about what is better with this flow if you are still calling hmm_range_fault(). How is it better than just calling migrate_vma_setup()? Obviously it will fault the pages in, but it seems

The code in rev2 calls migrate_vma_setup but the requires a struct vm_area_struct argument whereas hmm_range_fault does not.

I'm not sure that's a good enough justfication because the problem isn't whether you're looking up vma's in driver code or mm code. The problem is you are looking up vma's at all and all that goes with that (mainly taking mmap lock, etc.)

Quoted 7 lines
Quoted 5 lines

And for eviction hmm_range_fault() won't even find all the pages because their virtual address may have changed - consider what happens in cases of mremap(), fork(), etc. So eviction really needs physical pages (pfn's), not virtual addresses.

See above, #1 yes we use a physical pages. For #2 it is about making the state consistent within a virtual address range.

Yep, makes sense now. For migration of physical pages you want migrate_device_*, virtual address ranges want migrate_vma_*

- Alistair

Quoted 31 lines

Matt

Quoted 29 lines
Quoted 7 lines
Quoted 3 lines

we're talking about eviction here so I don't understand why that would be relevant. And hmm_range_fault() still requires the VMA, although I need to look at the patches more closely, probably CPUVMA is a DRM

'hmm_range_fault() still requires the VMA' internal yes, but again not as argument. This is about avoiding a driver side lookup of the VMA.

CPUVMA == struct vm_area_struct in this email.

Thanks for the clarification.

- Alistair

Quoted 18 lines

Matt

Quoted 16 lines

specific concept?

Thanks.

- Alistair

Quoted 10 lines

Matt

Quoted 8 lines

- Alistair

Quoted 6 lines

Matt

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2

Quoted 2 lines

Matt

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

Review

Quoted 11 lines
(quoted diff) +8 -0
+	}
+
+	migrate_device_unmap(src_pfns, npages, NULL);
+
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_prepopulated_range);
+
 /*
  * Migrate a device coherent folio back to normal memory. The caller should have
  * a reference on folio which will be copied to the new folio if migration is

--Mika

author maintainer
#22 lore raw

PATCH v2 3/29 mm/migrate: Trylock device page in do_swap_page

In reply to Matthew Brost

Avoid multiple CPU page faults to the same device page racing by trying to lock the page in do_swap_page before taking an extra reference to the page. This prevents scenarios where multiple CPU page faults each take an extra reference to a device page, which could abort migration in folio_migrate_mapping. With the device page being locked in do_swap_page, the migrate_vma_* functions need to be updated to avoid locking the fault_page argument.

Prior to this change, a livelock scenario could occur in Xe's (Intel GPU DRM driver) SVM implementation if enough threads faulted the same device page.

Cc Philip Yang <[email protected]>
Cc Felix Kuehling <[email protected]>
Cc Christian König <[email protected]>
Cc Andrew Morton <[email protected]>
---
 mm/memory.c         | 13 ++++++---
 mm/migrate_device.c | 69 ++++++++++++++++++++++++++++++---------------
 2 files changed, 56 insertions(+), 26 deletions(-)
mm/memory.c +9 -4
index 2366578015ad..b72bde782611 100644
@@ -4252,10 +4252,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 			 * Get a page reference while we know the page can't be
 			 * freed.
 			 */
-			get_page(vmf->page);
-			pte_unmap_unlock(vmf->pte, vmf->ptl);
-			ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
-			put_page(vmf->page);
+			if (trylock_page(vmf->page)) {
+				get_page(vmf->page);
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+				ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
+				put_page(vmf->page);
+				unlock_page(vmf->page);
+			} else {
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+			}
 		} else if (is_hwpoison_entry(entry)) {
 			ret = VM_FAULT_HWPOISON;
 		} else if (is_pte_marker_entry(entry)) {
mm/migrate_device.c +47 -22

157 lines not on this page · raw message

Signature · 1 line
2.34.1
#23 lore raw

Re: [PATCH v2 03/29] mm/migrate: Trylock device page in do_swap_page

Quoted 31 lines · Matthew Brost

Avoid multiple CPU page faults to the same device page racing by trying to lock the page in do_swap_page before taking an extra reference to the page. This prevents scenarios where multiple CPU page faults each take an extra reference to a device page, which could abort migration in folio_migrate_mapping. With the device page being locked in do_swap_page, the migrate_vma_* functions need to be updated to avoid locking the fault_page argument.

Prior to this change, a livelock scenario could occur in Xe's (Intel GPU DRM driver) SVM implementation if enough threads faulted the same device page.

Cc Philip Yang <[email protected]>
Cc Felix Kuehling <[email protected]>
Cc Christian König <[email protected]>
Cc Andrew Morton <[email protected]>
---
 mm/memory.c         | 13 ++++++---
 mm/migrate_device.c | 69 ++++++++++++++++++++++++++++++---------------
 2 files changed, 56 insertions(+), 26 deletions(-)
mm/memory.c +0 -0
index 2366578015ad..b72bde782611 100644
@@ -4252,10 +4252,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 			 * Get a page reference while we know the page can't be
 			 * freed.
 			 */

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4260

Quoted 10 lines
mm/memory.c +6 -4
-			get_page(vmf->page);
-			pte_unmap_unlock(vmf->pte, vmf->ptl);
-			ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
-			put_page(vmf->page);
+			if (trylock_page(vmf->page)) {
+				get_page(vmf->page);
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+				ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
+				put_page(vmf->page);
+				unlock_page(vmf->page);

I don't think my previous review of this change has really been addressed. Why don't we just install the migration entry here? Seems like it would be a much simpler way of solving this.

Quoted 179 lines
mm/memory.c +3 -0
+			} else {
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+			}
 		} else if (is_hwpoison_entry(entry)) {
 			ret = VM_FAULT_HWPOISON;
 		} else if (is_pte_marker_entry(entry)) {
mm/migrate_device.c +47 -22
index f163c2131022..2477d39f57be 100644
@@ -60,6 +60,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 				   struct mm_walk *walk)
 {
 	struct migrate_vma *migrate = walk->private;
+	struct folio *fault_folio = migrate->fault_page ?
+		page_folio(migrate->fault_page) : NULL;
 	struct vm_area_struct *vma = walk->vma;
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long addr = start, unmapped = 0;
@@ -88,11 +90,13 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
 			folio_get(folio);
 			spin_unlock(ptl);
-			if (unlikely(!folio_trylock(folio)))
+			if (unlikely(fault_folio != folio &&
+				     !folio_trylock(folio)))
 				return migrate_vma_collect_skip(start, end,
 								walk);
 			ret = split_folio(folio);
-			folio_unlock(folio);
+			if (fault_folio != folio)
+				folio_unlock(folio);
 			folio_put(folio);
 			if (ret)
 				return migrate_vma_collect_skip(start, end,
@@ -192,7 +196,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 		 * optimisation to avoid walking the rmap later with
 		 * try_to_migrate().
 		 */
-		if (folio_trylock(folio)) {
+		if (fault_folio == folio || folio_trylock(folio)) {
 			bool anon_exclusive;
 			pte_t swp_pte;
 
@@ -204,7 +208,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
 				if (folio_try_share_anon_rmap_pte(folio, page)) {
 					set_pte_at(mm, addr, ptep, pte);
-					folio_unlock(folio);
+					if (fault_folio != folio)
+						folio_unlock(folio);
 					folio_put(folio);
 					mpfn = 0;
 					goto next;
@@ -363,6 +368,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 					  unsigned long npages,
 					  struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i, restore = 0;
 	bool allow_drain = true;
 	unsigned long unmapped = 0;
@@ -427,7 +434,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 		remove_migration_ptes(folio, folio, 0);
 
 		src_pfns[i] = 0;
-		folio_unlock(folio);
+		if (fault_folio != folio)
+			folio_unlock(folio);
 		folio_put(folio);
 		restore--;
 	}
@@ -536,6 +544,8 @@ int migrate_vma_setup(struct migrate_vma *args)
 		return -EINVAL;
 	if (args->fault_page && !is_device_private_page(args->fault_page))
 		return -EINVAL;
+	if (args->fault_page && !PageLocked(args->fault_page))
+		return -EINVAL;
 
 	memset(args->src, 0, sizeof(*args->src) * nr_pages);
 	args->cpages = 0;
@@ -799,19 +809,13 @@ void migrate_vma_pages(struct migrate_vma *migrate)
 }
 EXPORT_SYMBOL(migrate_vma_pages);
 
-/*
- * migrate_device_finalize() - complete page migration
- * @src_pfns: src_pfns returned from migrate_device_range()
- * @dst_pfns: array of pfns allocated by the driver to migrate memory to
- * @npages: number of pages in the range
- *
- * Completes migration of the page by removing special migration entries.
- * Drivers must ensure copying of page data is complete and visible to the CPU
- * before calling this.
- */
-void migrate_device_finalize(unsigned long *src_pfns,
-			unsigned long *dst_pfns, unsigned long npages)
+static void __migrate_device_finalize(unsigned long *src_pfns,
+				      unsigned long *dst_pfns,
+				      unsigned long npages,
+				      struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i;
 
 	for (i = 0; i < npages; i++) {
@@ -824,7 +828,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
 		if (!page) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);
 				folio_put(dst);
 			}
 			continue;
@@ -834,14 +839,16 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
 		if (!(src_pfns[i] & MIGRATE_PFN_MIGRATE) || !dst) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);
 				folio_put(dst);
 			}
 			dst = src;
 		}
 
 		remove_migration_ptes(src, dst, 0);
-		folio_unlock(src);
+		if (fault_folio != src)
+			folio_unlock(src);
 
 		if (folio_is_zone_device(src))
 			folio_put(src);
@@ -849,7 +856,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 			folio_putback_lru(src);
 
 		if (dst != src) {
-			folio_unlock(dst);
+			if (fault_folio != dst)
+				folio_unlock(dst);
 			if (folio_is_zone_device(dst))
 				folio_put(dst);
 			else
@@ -857,6 +865,22 @@ void migrate_device_finalize(unsigned long *src_pfns,
 		}
 	}
 }
+
+/*
+ * migrate_device_finalize() - complete page migration
+ * @src_pfns: src_pfns returned from migrate_device_range()
+ * @dst_pfns: array of pfns allocated by the driver to migrate memory to
+ * @npages: number of pages in the range
+ *
+ * Completes migration of the page by removing special migration entries.
+ * Drivers must ensure copying of page data is complete and visible to the CPU
+ * before calling this.
+ */
+void migrate_device_finalize(unsigned long *src_pfns,
+			unsigned long *dst_pfns, unsigned long npages)
+{
+	return __migrate_device_finalize(src_pfns, dst_pfns, npages, NULL);
+}
 EXPORT_SYMBOL(migrate_device_finalize);
 
 /**
@@ -872,7 +896,8 @@ EXPORT_SYMBOL(migrate_device_finalize);
  */
 void migrate_vma_finalize(struct migrate_vma *migrate)
 {
-	migrate_device_finalize(migrate->src, migrate->dst, migrate->npages);
+	__migrate_device_finalize(migrate->src, migrate->dst, migrate->npages,
+				  migrate->fault_page);
 }
 EXPORT_SYMBOL(migrate_vma_finalize);
author maintainer
#24 lore raw

Re: [PATCH v2 03/29] mm/migrate: Trylock device page in do_swap_page

Quoted 40 lines · Alistair Popple
Quoted 37 lines · Matthew Brost

Avoid multiple CPU page faults to the same device page racing by trying to lock the page in do_swap_page before taking an extra reference to the page. This prevents scenarios where multiple CPU page faults each take an extra reference to a device page, which could abort migration in folio_migrate_mapping. With the device page being locked in do_swap_page, the migrate_vma_* functions need to be updated to avoid locking the fault_page argument.

Prior to this change, a livelock scenario could occur in Xe's (Intel GPU DRM driver) SVM implementation if enough threads faulted the same device page.

Cc Philip Yang <[email protected]>
Cc Felix Kuehling <[email protected]>
Cc Christian König <[email protected]>
Cc Andrew Morton <[email protected]>
---
 mm/memory.c         | 13 ++++++---
 mm/migrate_device.c | 69 ++++++++++++++++++++++++++++++---------------
 2 files changed, 56 insertions(+), 26 deletions(-)
mm/memory.c +2 -4
index 2366578015ad..b72bde782611 100644
@@ -4252,10 +4252,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 			 * Get a page reference while we know the page can't be
 			 * freed.
 			 */
-			get_page(vmf->page);
-			pte_unmap_unlock(vmf->pte, vmf->ptl);
-			ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
-			put_page(vmf->page);
+			if (trylock_page(vmf->page)) {
+				get_page(vmf->page);
Quoted 9 lines

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4260

Quoted 4 lines
mm/memory.c +4 -0
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+				ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
+				put_page(vmf->page);
+				unlock_page(vmf->page);

I don't think my previous review of this change has really been addressed. Why don't we just install the migration entry here? Seems like it would be a much simpler way of solving this.

I should have mentioned this in the cover-letter, I haven't got around to trying that out yet. Included this existing version for correctness but I also think this is not strickly required to merge this series as our locking in migrate_to_ram only relies on the core MM locks so some thread would eventually win the race and make forward progress.

So I guess just ignore this patch and will send an updated version individually with installing a migration entry in do_swap_page. If for some reason that doesn't work, I'll respond here explaining why.

Matt

Quoted 180 lines
Quoted 179 lines
mm/memory.c +3 -0
+			} else {
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+			}
 		} else if (is_hwpoison_entry(entry)) {
 			ret = VM_FAULT_HWPOISON;
 		} else if (is_pte_marker_entry(entry)) {
mm/migrate_device.c +47 -22
index f163c2131022..2477d39f57be 100644
@@ -60,6 +60,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 				   struct mm_walk *walk)
 {
 	struct migrate_vma *migrate = walk->private;
+	struct folio *fault_folio = migrate->fault_page ?
+		page_folio(migrate->fault_page) : NULL;
 	struct vm_area_struct *vma = walk->vma;
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long addr = start, unmapped = 0;
@@ -88,11 +90,13 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
 			folio_get(folio);
 			spin_unlock(ptl);
-			if (unlikely(!folio_trylock(folio)))
+			if (unlikely(fault_folio != folio &&
+				     !folio_trylock(folio)))
 				return migrate_vma_collect_skip(start, end,
 								walk);
 			ret = split_folio(folio);
-			folio_unlock(folio);
+			if (fault_folio != folio)
+				folio_unlock(folio);
 			folio_put(folio);
 			if (ret)
 				return migrate_vma_collect_skip(start, end,
@@ -192,7 +196,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 		 * optimisation to avoid walking the rmap later with
 		 * try_to_migrate().
 		 */
-		if (folio_trylock(folio)) {
+		if (fault_folio == folio || folio_trylock(folio)) {
 			bool anon_exclusive;
 			pte_t swp_pte;
 
@@ -204,7 +208,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
 				if (folio_try_share_anon_rmap_pte(folio, page)) {
 					set_pte_at(mm, addr, ptep, pte);
-					folio_unlock(folio);
+					if (fault_folio != folio)
+						folio_unlock(folio);
 					folio_put(folio);
 					mpfn = 0;
 					goto next;
@@ -363,6 +368,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 					  unsigned long npages,
 					  struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i, restore = 0;
 	bool allow_drain = true;
 	unsigned long unmapped = 0;
@@ -427,7 +434,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 		remove_migration_ptes(folio, folio, 0);
 
 		src_pfns[i] = 0;
-		folio_unlock(folio);
+		if (fault_folio != folio)
+			folio_unlock(folio);
 		folio_put(folio);
 		restore--;
 	}
@@ -536,6 +544,8 @@ int migrate_vma_setup(struct migrate_vma *args)
 		return -EINVAL;
 	if (args->fault_page && !is_device_private_page(args->fault_page))
 		return -EINVAL;
+	if (args->fault_page && !PageLocked(args->fault_page))
+		return -EINVAL;
 
 	memset(args->src, 0, sizeof(*args->src) * nr_pages);
 	args->cpages = 0;
@@ -799,19 +809,13 @@ void migrate_vma_pages(struct migrate_vma *migrate)
 }
 EXPORT_SYMBOL(migrate_vma_pages);
 
-/*
- * migrate_device_finalize() - complete page migration
- * @src_pfns: src_pfns returned from migrate_device_range()
- * @dst_pfns: array of pfns allocated by the driver to migrate memory to
- * @npages: number of pages in the range
- *
- * Completes migration of the page by removing special migration entries.
- * Drivers must ensure copying of page data is complete and visible to the CPU
- * before calling this.
- */
-void migrate_device_finalize(unsigned long *src_pfns,
-			unsigned long *dst_pfns, unsigned long npages)
+static void __migrate_device_finalize(unsigned long *src_pfns,
+				      unsigned long *dst_pfns,
+				      unsigned long npages,
+				      struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i;
 
 	for (i = 0; i < npages; i++) {
@@ -824,7 +828,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
 		if (!page) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);
 				folio_put(dst);
 			}
 			continue;
@@ -834,14 +839,16 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
 		if (!(src_pfns[i] & MIGRATE_PFN_MIGRATE) || !dst) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);
 				folio_put(dst);
 			}
 			dst = src;
 		}
 
 		remove_migration_ptes(src, dst, 0);
-		folio_unlock(src);
+		if (fault_folio != src)
+			folio_unlock(src);
 
 		if (folio_is_zone_device(src))
 			folio_put(src);
@@ -849,7 +856,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 			folio_putback_lru(src);
 
 		if (dst != src) {
-			folio_unlock(dst);
+			if (fault_folio != dst)
+				folio_unlock(dst);
 			if (folio_is_zone_device(dst))
 				folio_put(dst);
 			else
@@ -857,6 +865,22 @@ void migrate_device_finalize(unsigned long *src_pfns,
 		}
 	}
 }
+
+/*
+ * migrate_device_finalize() - complete page migration
+ * @src_pfns: src_pfns returned from migrate_device_range()
+ * @dst_pfns: array of pfns allocated by the driver to migrate memory to
+ * @npages: number of pages in the range
+ *
+ * Completes migration of the page by removing special migration entries.
+ * Drivers must ensure copying of page data is complete and visible to the CPU
+ * before calling this.
+ */
+void migrate_device_finalize(unsigned long *src_pfns,
+			unsigned long *dst_pfns, unsigned long npages)
+{
+	return __migrate_device_finalize(src_pfns, dst_pfns, npages, NULL);
+}
 EXPORT_SYMBOL(migrate_device_finalize);
 
 /**
@@ -872,7 +896,8 @@ EXPORT_SYMBOL(migrate_device_finalize);
  */
 void migrate_vma_finalize(struct migrate_vma *migrate)
 {
-	migrate_device_finalize(migrate->src, migrate->dst, migrate->npages);
+	__migrate_device_finalize(migrate->src, migrate->dst, migrate->npages,
+				  migrate->fault_page);
 }
 EXPORT_SYMBOL(migrate_vma_finalize);
#25 lore raw

Re: [PATCH v2 03/29] mm/migrate: Trylock device page in do_swap_page

Quoted 57 lines · Matthew Brost
Quoted 40 lines · Alistair Popple
Quoted 37 lines · Matthew Brost

Avoid multiple CPU page faults to the same device page racing by trying to lock the page in do_swap_page before taking an extra reference to the page. This prevents scenarios where multiple CPU page faults each take an extra reference to a device page, which could abort migration in folio_migrate_mapping. With the device page being locked in do_swap_page, the migrate_vma_* functions need to be updated to avoid locking the fault_page argument.

Prior to this change, a livelock scenario could occur in Xe's (Intel GPU DRM driver) SVM implementation if enough threads faulted the same device page.

Cc Philip Yang <[email protected]>
Cc Felix Kuehling <[email protected]>
Cc Christian König <[email protected]>
Cc Andrew Morton <[email protected]>
---
 mm/memory.c         | 13 ++++++---
 mm/migrate_device.c | 69 ++++++++++++++++++++++++++++++---------------
 2 files changed, 56 insertions(+), 26 deletions(-)
mm/memory.c +2 -4
index 2366578015ad..b72bde782611 100644
@@ -4252,10 +4252,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 			 * Get a page reference while we know the page can't be
 			 * freed.
 			 */
-			get_page(vmf->page);
-			pte_unmap_unlock(vmf->pte, vmf->ptl);
-			ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
-			put_page(vmf->page);
+			if (trylock_page(vmf->page)) {
+				get_page(vmf->page);
Quoted 9 lines

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4260

Quoted 4 lines
mm/memory.c +4 -0
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+				ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
+				put_page(vmf->page);
+				unlock_page(vmf->page);

I don't think my previous review of this change has really been addressed. Why don't we just install the migration entry here? Seems like it would be a much simpler way of solving this.

I should have mentioned this in the cover-letter, I haven't got around to trying that out yet. Included this existing version for correctness but I also think this is not strickly required to merge this series as our locking in migrate_to_ram only relies on the core MM locks so some thread would eventually win the race and make forward progress.

Quoted 3 lines

So I guess just ignore this patch and will send an updated version individually with installing a migration entry in do_swap_page. If for some reason that doesn't work, I'll respond here explaining why.

That would be great. I have a fairly strong preference for doing that instead of adding more special cases for the fault page in the migration code. And if we can't do that it would be good to understand why. Thanks.

- Alistair

Quoted 182 lines

Matt

Quoted 180 lines
Quoted 179 lines
mm/memory.c +3 -0
+			} else {
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+			}
 		} else if (is_hwpoison_entry(entry)) {
 			ret = VM_FAULT_HWPOISON;
 		} else if (is_pte_marker_entry(entry)) {
mm/migrate_device.c +47 -22
index f163c2131022..2477d39f57be 100644
@@ -60,6 +60,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 				   struct mm_walk *walk)
 {
 	struct migrate_vma *migrate = walk->private;
+	struct folio *fault_folio = migrate->fault_page ?
+		page_folio(migrate->fault_page) : NULL;
 	struct vm_area_struct *vma = walk->vma;
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long addr = start, unmapped = 0;
@@ -88,11 +90,13 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
 			folio_get(folio);
 			spin_unlock(ptl);
-			if (unlikely(!folio_trylock(folio)))
+			if (unlikely(fault_folio != folio &&
+				     !folio_trylock(folio)))
 				return migrate_vma_collect_skip(start, end,
 								walk);
 			ret = split_folio(folio);
-			folio_unlock(folio);
+			if (fault_folio != folio)
+				folio_unlock(folio);
 			folio_put(folio);
 			if (ret)
 				return migrate_vma_collect_skip(start, end,
@@ -192,7 +196,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 		 * optimisation to avoid walking the rmap later with
 		 * try_to_migrate().
 		 */
-		if (folio_trylock(folio)) {
+		if (fault_folio == folio || folio_trylock(folio)) {
 			bool anon_exclusive;
 			pte_t swp_pte;
 
@@ -204,7 +208,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
 				if (folio_try_share_anon_rmap_pte(folio, page)) {
 					set_pte_at(mm, addr, ptep, pte);
-					folio_unlock(folio);
+					if (fault_folio != folio)
+						folio_unlock(folio);
 					folio_put(folio);
 					mpfn = 0;
 					goto next;
@@ -363,6 +368,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 					  unsigned long npages,
 					  struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i, restore = 0;
 	bool allow_drain = true;
 	unsigned long unmapped = 0;
@@ -427,7 +434,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 		remove_migration_ptes(folio, folio, 0);
 
 		src_pfns[i] = 0;
-		folio_unlock(folio);
+		if (fault_folio != folio)
+			folio_unlock(folio);
 		folio_put(folio);
 		restore--;
 	}
@@ -536,6 +544,8 @@ int migrate_vma_setup(struct migrate_vma *args)
 		return -EINVAL;
 	if (args->fault_page && !is_device_private_page(args->fault_page))
 		return -EINVAL;
+	if (args->fault_page && !PageLocked(args->fault_page))
+		return -EINVAL;
 
 	memset(args->src, 0, sizeof(*args->src) * nr_pages);
 	args->cpages = 0;
@@ -799,19 +809,13 @@ void migrate_vma_pages(struct migrate_vma *migrate)
 }
 EXPORT_SYMBOL(migrate_vma_pages);
 
-/*
- * migrate_device_finalize() - complete page migration
- * @src_pfns: src_pfns returned from migrate_device_range()
- * @dst_pfns: array of pfns allocated by the driver to migrate memory to
- * @npages: number of pages in the range
- *
- * Completes migration of the page by removing special migration entries.
- * Drivers must ensure copying of page data is complete and visible to the CPU
- * before calling this.
- */
-void migrate_device_finalize(unsigned long *src_pfns,
-			unsigned long *dst_pfns, unsigned long npages)
+static void __migrate_device_finalize(unsigned long *src_pfns,
+				      unsigned long *dst_pfns,
+				      unsigned long npages,
+				      struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i;
 
 	for (i = 0; i < npages; i++) {
@@ -824,7 +828,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
 		if (!page) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);
 				folio_put(dst);
 			}
 			continue;
@@ -834,14 +839,16 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
 		if (!(src_pfns[i] & MIGRATE_PFN_MIGRATE) || !dst) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);
 				folio_put(dst);
 			}
 			dst = src;
 		}
 
 		remove_migration_ptes(src, dst, 0);
-		folio_unlock(src);
+		if (fault_folio != src)
+			folio_unlock(src);
 
 		if (folio_is_zone_device(src))
 			folio_put(src);
@@ -849,7 +856,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 			folio_putback_lru(src);
 
 		if (dst != src) {
-			folio_unlock(dst);
+			if (fault_folio != dst)
+				folio_unlock(dst);
 			if (folio_is_zone_device(dst))
 				folio_put(dst);
 			else
@@ -857,6 +865,22 @@ void migrate_device_finalize(unsigned long *src_pfns,
 		}
 	}
 }
+
+/*
+ * migrate_device_finalize() - complete page migration
+ * @src_pfns: src_pfns returned from migrate_device_range()
+ * @dst_pfns: array of pfns allocated by the driver to migrate memory to
+ * @npages: number of pages in the range
+ *
+ * Completes migration of the page by removing special migration entries.
+ * Drivers must ensure copying of page data is complete and visible to the CPU
+ * before calling this.
+ */
+void migrate_device_finalize(unsigned long *src_pfns,
+			unsigned long *dst_pfns, unsigned long npages)
+{
+	return __migrate_device_finalize(src_pfns, dst_pfns, npages, NULL);
+}
 EXPORT_SYMBOL(migrate_device_finalize);
 
 /**
@@ -872,7 +896,8 @@ EXPORT_SYMBOL(migrate_device_finalize);
  */
 void migrate_vma_finalize(struct migrate_vma *migrate)
 {
-	migrate_device_finalize(migrate->src, migrate->dst, migrate->npages);
+	__migrate_device_finalize(migrate->src, migrate->dst, migrate->npages,
+				  migrate->fault_page);
 }
 EXPORT_SYMBOL(migrate_vma_finalize);
author maintainer
#26 lore raw

Re: [PATCH v2 03/29] mm/migrate: Trylock device page in do_swap_page

Quoted 60 lines · Alistair Popple
Quoted 57 lines · Matthew Brost
Quoted 40 lines · Alistair Popple
Quoted 37 lines · Matthew Brost

Avoid multiple CPU page faults to the same device page racing by trying to lock the page in do_swap_page before taking an extra reference to the page. This prevents scenarios where multiple CPU page faults each take an extra reference to a device page, which could abort migration in folio_migrate_mapping. With the device page being locked in do_swap_page, the migrate_vma_* functions need to be updated to avoid locking the fault_page argument.

Prior to this change, a livelock scenario could occur in Xe's (Intel GPU DRM driver) SVM implementation if enough threads faulted the same device page.

Cc Philip Yang <[email protected]>
Cc Felix Kuehling <[email protected]>
Cc Christian König <[email protected]>
Cc Andrew Morton <[email protected]>
---
 mm/memory.c         | 13 ++++++---
 mm/migrate_device.c | 69 ++++++++++++++++++++++++++++++---------------
 2 files changed, 56 insertions(+), 26 deletions(-)
mm/memory.c +2 -4
index 2366578015ad..b72bde782611 100644
@@ -4252,10 +4252,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 			 * Get a page reference while we know the page can't be
 			 * freed.
 			 */
-			get_page(vmf->page);
-			pte_unmap_unlock(vmf->pte, vmf->ptl);
-			ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
-			put_page(vmf->page);
+			if (trylock_page(vmf->page)) {
+				get_page(vmf->page);
Quoted 9 lines

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4260

Quoted 4 lines
mm/memory.c +4 -0
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+				ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
+				put_page(vmf->page);
+				unlock_page(vmf->page);

I don't think my previous review of this change has really been addressed. Why don't we just install the migration entry here? Seems like it would be a much simpler way of solving this.

I should have mentioned this in the cover-letter, I haven't got around to trying that out yet. Included this existing version for correctness but I also think this is not strickly required to merge this series as our locking in migrate_to_ram only relies on the core MM locks so some thread would eventually win the race and make forward progress.

Quoted 9 lines
Quoted 3 lines

So I guess just ignore this patch and will send an updated version individually with installing a migration entry in do_swap_page. If for some reason that doesn't work, I'll respond here explaining why.

That would be great. I have a fairly strong preference for doing that instead of adding more special cases for the fault page in the migration code. And if we can't do that it would be good to understand why. Thanks.

I've looked into this and actually prefer the approach in this patch.

Consider the scenario where we install a migration entry, but migrate_to_ram fails. How do we handle this?

We don't know where migrate_to_ram failed. Was migrate_device_finalize called, removing the migration PTE? Do we need to special-case failures in migrate_to_ram to prevent migrate_device_finalize from removing the faulting page's migration entry? Should we check for a migration entry after migrate_to_ram and remove it if it exists?

Now, if migrate_to_ram succeeds, it seems the migration entry should be removed in migrate_device_finalize since the new page is only available there. We could return the new page in migrate_to_ram, but that feels messy.

Additionally, the page lock needs to be held across migrate_to_ram, as this patch does, so we'll require some special handling in migrate_device_finalize to avoid unlocking the faulting page.

Finally, installing a migration entry is non-trivial, while taking a page reference under a lock is straightforward.

Given all this, I prefer to keep this patch as it is.

Matt

Quoted 185 lines

- Alistair

Quoted 182 lines

Matt

Quoted 180 lines
Quoted 179 lines
mm/memory.c +3 -0
+			} else {
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+			}
 		} else if (is_hwpoison_entry(entry)) {
 			ret = VM_FAULT_HWPOISON;
 		} else if (is_pte_marker_entry(entry)) {
mm/migrate_device.c +47 -22
index f163c2131022..2477d39f57be 100644
@@ -60,6 +60,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 				   struct mm_walk *walk)
 {
 	struct migrate_vma *migrate = walk->private;
+	struct folio *fault_folio = migrate->fault_page ?
+		page_folio(migrate->fault_page) : NULL;
 	struct vm_area_struct *vma = walk->vma;
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long addr = start, unmapped = 0;
@@ -88,11 +90,13 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
 			folio_get(folio);
 			spin_unlock(ptl);
-			if (unlikely(!folio_trylock(folio)))
+			if (unlikely(fault_folio != folio &&
+				     !folio_trylock(folio)))
 				return migrate_vma_collect_skip(start, end,
 								walk);
 			ret = split_folio(folio);
-			folio_unlock(folio);
+			if (fault_folio != folio)
+				folio_unlock(folio);
 			folio_put(folio);
 			if (ret)
 				return migrate_vma_collect_skip(start, end,
@@ -192,7 +196,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 		 * optimisation to avoid walking the rmap later with
 		 * try_to_migrate().
 		 */
-		if (folio_trylock(folio)) {
+		if (fault_folio == folio || folio_trylock(folio)) {
 			bool anon_exclusive;
 			pte_t swp_pte;
 
@@ -204,7 +208,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
 				if (folio_try_share_anon_rmap_pte(folio, page)) {
 					set_pte_at(mm, addr, ptep, pte);
-					folio_unlock(folio);
+					if (fault_folio != folio)
+						folio_unlock(folio);
 					folio_put(folio);
 					mpfn = 0;
 					goto next;
@@ -363,6 +368,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 					  unsigned long npages,
 					  struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i, restore = 0;
 	bool allow_drain = true;
 	unsigned long unmapped = 0;
@@ -427,7 +434,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 		remove_migration_ptes(folio, folio, 0);
 
 		src_pfns[i] = 0;
-		folio_unlock(folio);
+		if (fault_folio != folio)
+			folio_unlock(folio);
 		folio_put(folio);
 		restore--;
 	}
@@ -536,6 +544,8 @@ int migrate_vma_setup(struct migrate_vma *args)
 		return -EINVAL;
 	if (args->fault_page && !is_device_private_page(args->fault_page))
 		return -EINVAL;
+	if (args->fault_page && !PageLocked(args->fault_page))
+		return -EINVAL;
 
 	memset(args->src, 0, sizeof(*args->src) * nr_pages);
 	args->cpages = 0;
@@ -799,19 +809,13 @@ void migrate_vma_pages(struct migrate_vma *migrate)
 }
 EXPORT_SYMBOL(migrate_vma_pages);
 
-/*
- * migrate_device_finalize() - complete page migration
- * @src_pfns: src_pfns returned from migrate_device_range()
- * @dst_pfns: array of pfns allocated by the driver to migrate memory to
- * @npages: number of pages in the range
- *
- * Completes migration of the page by removing special migration entries.
- * Drivers must ensure copying of page data is complete and visible to the CPU
- * before calling this.
- */
-void migrate_device_finalize(unsigned long *src_pfns,
-			unsigned long *dst_pfns, unsigned long npages)
+static void __migrate_device_finalize(unsigned long *src_pfns,
+				      unsigned long *dst_pfns,
+				      unsigned long npages,
+				      struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i;
 
 	for (i = 0; i < npages; i++) {
@@ -824,7 +828,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
 		if (!page) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);
 				folio_put(dst);
 			}
 			continue;
@@ -834,14 +839,16 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
 		if (!(src_pfns[i] & MIGRATE_PFN_MIGRATE) || !dst) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);
 				folio_put(dst);
 			}
 			dst = src;
 		}
 
 		remove_migration_ptes(src, dst, 0);
-		folio_unlock(src);
+		if (fault_folio != src)
+			folio_unlock(src);
 
 		if (folio_is_zone_device(src))
 			folio_put(src);
@@ -849,7 +856,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 			folio_putback_lru(src);
 
 		if (dst != src) {
-			folio_unlock(dst);
+			if (fault_folio != dst)
+				folio_unlock(dst);
 			if (folio_is_zone_device(dst))
 				folio_put(dst);
 			else
@@ -857,6 +865,22 @@ void migrate_device_finalize(unsigned long *src_pfns,
 		}
 	}
 }
+
+/*
+ * migrate_device_finalize() - complete page migration
+ * @src_pfns: src_pfns returned from migrate_device_range()
+ * @dst_pfns: array of pfns allocated by the driver to migrate memory to
+ * @npages: number of pages in the range
+ *
+ * Completes migration of the page by removing special migration entries.
+ * Drivers must ensure copying of page data is complete and visible to the CPU
+ * before calling this.
+ */
+void migrate_device_finalize(unsigned long *src_pfns,
+			unsigned long *dst_pfns, unsigned long npages)
+{
+	return __migrate_device_finalize(src_pfns, dst_pfns, npages, NULL);
+}
 EXPORT_SYMBOL(migrate_device_finalize);
 
 /**
@@ -872,7 +896,8 @@ EXPORT_SYMBOL(migrate_device_finalize);
  */
 void migrate_vma_finalize(struct migrate_vma *migrate)
 {
-	migrate_device_finalize(migrate->src, migrate->dst, migrate->npages);
+	__migrate_device_finalize(migrate->src, migrate->dst, migrate->npages,
+				  migrate->fault_page);
 }
 EXPORT_SYMBOL(migrate_vma_finalize);
#27 lore raw

Re: [PATCH v2 03/29] mm/migrate: Trylock device page in do_swap_page

Quoted 65 lines · Matthew Brost
Quoted 64 lines · Alistair Popple
Quoted 60 lines · Matthew Brost
Quoted 40 lines · Alistair Popple
Quoted 37 lines · Matthew Brost

Avoid multiple CPU page faults to the same device page racing by trying to lock the page in do_swap_page before taking an extra reference to the page. This prevents scenarios where multiple CPU page faults each take an extra reference to a device page, which could abort migration in folio_migrate_mapping. With the device page being locked in do_swap_page, the migrate_vma_* functions need to be updated to avoid locking the fault_page argument.

Prior to this change, a livelock scenario could occur in Xe's (Intel GPU DRM driver) SVM implementation if enough threads faulted the same device page.

Cc Philip Yang <[email protected]>
Cc Felix Kuehling <[email protected]>
Cc Christian König <[email protected]>
Cc Andrew Morton <[email protected]>
---
 mm/memory.c         | 13 ++++++---
 mm/migrate_device.c | 69 ++++++++++++++++++++++++++++++---------------
 2 files changed, 56 insertions(+), 26 deletions(-)
mm/memory.c +2 -4
index 2366578015ad..b72bde782611 100644
@@ -4252,10 +4252,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 			 * Get a page reference while we know the page can't be
 			 * freed.
 			 */
-			get_page(vmf->page);
-			pte_unmap_unlock(vmf->pte, vmf->ptl);
-			ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
-			put_page(vmf->page);
+			if (trylock_page(vmf->page)) {
+				get_page(vmf->page);
Quoted 9 lines

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4260

Quoted 4 lines
mm/memory.c +4 -0
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+				ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
+				put_page(vmf->page);
+				unlock_page(vmf->page);

I don't think my previous review of this change has really been addressed. Why don't we just install the migration entry here? Seems like it would be a much simpler way of solving this.

I should have mentioned this in the cover-letter, I haven't got around to trying that out yet. Included this existing version for correctness but I also think this is not strickly required to merge this series as our locking in migrate_to_ram only relies on the core MM locks so some thread would eventually win the race and make forward progress.

So I guess just ignore this patch and will send an updated version individually with installing a migration entry in do_swap_page. If for some reason that doesn't work, I'll respond here explaining why.

Quoted 7 lines
Quoted 5 lines

That would be great. I have a fairly strong preference for doing that instead of adding more special cases for the fault page in the migration code. And if we can't do that it would be good to understand why. Thanks.

I've looked into this and actually prefer the approach in this patch.

Thanks for looking into this.

Quoted 8 lines

Consider the scenario where we install a migration entry, but migrate_to_ram fails. How do we handle this?

We don't know where migrate_to_ram failed. Was migrate_device_finalize called, removing the migration PTE? Do we need to special-case failures in migrate_to_ram to prevent migrate_device_finalize from removing the faulting page's migration entry? Should we check for a migration entry after migrate_to_ram and remove it if it exists?

The driver should always call migrate_device_finalize(). On failure it will remove the migration entry and remap the original device private page. That obviously doesn't handle the fault but the process is about to die anyway with a SIGBUS because migrate_to_ram() can't fail.

Quoted 4 lines

Now, if migrate_to_ram succeeds, it seems the migration entry should be removed in migrate_device_finalize since the new page is only available there. We could return the new page in migrate_to_ram, but that feels messy.

Agreed - I would expect migrate_device_finalize() to always be called and remove the migration entry.

Quoted 3 lines

Additionally, the page lock needs to be held across migrate_to_ram, as this patch does, so we'll require some special handling in migrate_device_finalize to avoid unlocking the faulting page.

Or just unlock it in migrate_device_finalize(). I agree locking it one place and unlocking it in another is a bit ugly though.

Quoted 2 lines

Finally, installing a migration entry is non-trivial, while taking a page reference under a lock is straightforward.

I didn't think it was that hard once you have the PTL - although there is a bit of account keeping the same as migrate_vma_collect_pmd() but that could be abstracted into a common function. The advantage is it saves a page table walk, but I suppose that isn't that relevant if you're migrating a group of pages.

Quoted 1 line

Given all this, I prefer to keep this patch as it is.

Ok, I will take a closer look at it. Thanks.

- Alistair

Quoted 187 lines

Matt

Quoted 185 lines

- Alistair

Quoted 182 lines

Matt

Quoted 180 lines
Quoted 179 lines
mm/memory.c +3 -0
+			} else {
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+			}
 		} else if (is_hwpoison_entry(entry)) {
 			ret = VM_FAULT_HWPOISON;
 		} else if (is_pte_marker_entry(entry)) {
mm/migrate_device.c +47 -22
index f163c2131022..2477d39f57be 100644
@@ -60,6 +60,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 				   struct mm_walk *walk)
 {
 	struct migrate_vma *migrate = walk->private;
+	struct folio *fault_folio = migrate->fault_page ?
+		page_folio(migrate->fault_page) : NULL;
 	struct vm_area_struct *vma = walk->vma;
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long addr = start, unmapped = 0;
@@ -88,11 +90,13 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
 			folio_get(folio);
 			spin_unlock(ptl);
-			if (unlikely(!folio_trylock(folio)))
+			if (unlikely(fault_folio != folio &&
+				     !folio_trylock(folio)))
 				return migrate_vma_collect_skip(start, end,
 								walk);
 			ret = split_folio(folio);
-			folio_unlock(folio);
+			if (fault_folio != folio)
+				folio_unlock(folio);
 			folio_put(folio);
 			if (ret)
 				return migrate_vma_collect_skip(start, end,
@@ -192,7 +196,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 		 * optimisation to avoid walking the rmap later with
 		 * try_to_migrate().
 		 */
-		if (folio_trylock(folio)) {
+		if (fault_folio == folio || folio_trylock(folio)) {
 			bool anon_exclusive;
 			pte_t swp_pte;
 
@@ -204,7 +208,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
 				if (folio_try_share_anon_rmap_pte(folio, page)) {
 					set_pte_at(mm, addr, ptep, pte);
-					folio_unlock(folio);
+					if (fault_folio != folio)
+						folio_unlock(folio);
 					folio_put(folio);
 					mpfn = 0;
 					goto next;
@@ -363,6 +368,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 					  unsigned long npages,
 					  struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i, restore = 0;
 	bool allow_drain = true;
 	unsigned long unmapped = 0;
@@ -427,7 +434,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 		remove_migration_ptes(folio, folio, 0);
 
 		src_pfns[i] = 0;
-		folio_unlock(folio);
+		if (fault_folio != folio)
+			folio_unlock(folio);
 		folio_put(folio);
 		restore--;
 	}
@@ -536,6 +544,8 @@ int migrate_vma_setup(struct migrate_vma *args)
 		return -EINVAL;
 	if (args->fault_page && !is_device_private_page(args->fault_page))
 		return -EINVAL;
+	if (args->fault_page && !PageLocked(args->fault_page))
+		return -EINVAL;
 
 	memset(args->src, 0, sizeof(*args->src) * nr_pages);
 	args->cpages = 0;
@@ -799,19 +809,13 @@ void migrate_vma_pages(struct migrate_vma *migrate)
 }
 EXPORT_SYMBOL(migrate_vma_pages);
 
-/*
- * migrate_device_finalize() - complete page migration
- * @src_pfns: src_pfns returned from migrate_device_range()
- * @dst_pfns: array of pfns allocated by the driver to migrate memory to
- * @npages: number of pages in the range
- *
- * Completes migration of the page by removing special migration entries.
- * Drivers must ensure copying of page data is complete and visible to the CPU
- * before calling this.
- */
-void migrate_device_finalize(unsigned long *src_pfns,
-			unsigned long *dst_pfns, unsigned long npages)
+static void __migrate_device_finalize(unsigned long *src_pfns,
+				      unsigned long *dst_pfns,
+				      unsigned long npages,
+				      struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i;
 
 	for (i = 0; i < npages; i++) {
@@ -824,7 +828,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
 		if (!page) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);
 				folio_put(dst);
 			}
 			continue;
@@ -834,14 +839,16 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
 		if (!(src_pfns[i] & MIGRATE_PFN_MIGRATE) || !dst) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);
 				folio_put(dst);
 			}
 			dst = src;
 		}
 
 		remove_migration_ptes(src, dst, 0);
-		folio_unlock(src);
+		if (fault_folio != src)
+			folio_unlock(src);
 
 		if (folio_is_zone_device(src))
 			folio_put(src);
@@ -849,7 +856,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 			folio_putback_lru(src);
 
 		if (dst != src) {
-			folio_unlock(dst);
+			if (fault_folio != dst)
+				folio_unlock(dst);
 			if (folio_is_zone_device(dst))
 				folio_put(dst);
 			else
@@ -857,6 +865,22 @@ void migrate_device_finalize(unsigned long *src_pfns,
 		}
 	}
 }
+
+/*
+ * migrate_device_finalize() - complete page migration
+ * @src_pfns: src_pfns returned from migrate_device_range()
+ * @dst_pfns: array of pfns allocated by the driver to migrate memory to
+ * @npages: number of pages in the range
+ *
+ * Completes migration of the page by removing special migration entries.
+ * Drivers must ensure copying of page data is complete and visible to the CPU
+ * before calling this.
+ */
+void migrate_device_finalize(unsigned long *src_pfns,
+			unsigned long *dst_pfns, unsigned long npages)
+{
+	return __migrate_device_finalize(src_pfns, dst_pfns, npages, NULL);
+}
 EXPORT_SYMBOL(migrate_device_finalize);
 
 /**
@@ -872,7 +896,8 @@ EXPORT_SYMBOL(migrate_device_finalize);
  */
 void migrate_vma_finalize(struct migrate_vma *migrate)
 {
-	migrate_device_finalize(migrate->src, migrate->dst, migrate->npages);
+	__migrate_device_finalize(migrate->src, migrate->dst, migrate->npages,
+				  migrate->fault_page);
 }
 EXPORT_SYMBOL(migrate_vma_finalize);
author maintainer
#28 lore raw

Re: [PATCH v2 03/29] mm/migrate: Trylock device page in do_swap_page

Quoted 87 lines · Alistair Popple
Quoted 65 lines · Matthew Brost
Quoted 64 lines · Alistair Popple
Quoted 60 lines · Matthew Brost
Quoted 40 lines · Alistair Popple
Quoted 37 lines · Matthew Brost

Avoid multiple CPU page faults to the same device page racing by trying to lock the page in do_swap_page before taking an extra reference to the page. This prevents scenarios where multiple CPU page faults each take an extra reference to a device page, which could abort migration in folio_migrate_mapping. With the device page being locked in do_swap_page, the migrate_vma_* functions need to be updated to avoid locking the fault_page argument.

Prior to this change, a livelock scenario could occur in Xe's (Intel GPU DRM driver) SVM implementation if enough threads faulted the same device page.

Cc Philip Yang <[email protected]>
Cc Felix Kuehling <[email protected]>
Cc Christian König <[email protected]>
Cc Andrew Morton <[email protected]>
---
 mm/memory.c         | 13 ++++++---
 mm/migrate_device.c | 69 ++++++++++++++++++++++++++++++---------------
 2 files changed, 56 insertions(+), 26 deletions(-)
mm/memory.c +2 -4
index 2366578015ad..b72bde782611 100644
@@ -4252,10 +4252,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 			 * Get a page reference while we know the page can't be
 			 * freed.
 			 */
-			get_page(vmf->page);
-			pte_unmap_unlock(vmf->pte, vmf->ptl);
-			ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
-			put_page(vmf->page);
+			if (trylock_page(vmf->page)) {
+				get_page(vmf->page);
Quoted 9 lines

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4260

Quoted 4 lines
mm/memory.c +4 -0
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+				ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
+				put_page(vmf->page);
+				unlock_page(vmf->page);

I don't think my previous review of this change has really been addressed. Why don't we just install the migration entry here? Seems like it would be a much simpler way of solving this.

I should have mentioned this in the cover-letter, I haven't got around to trying that out yet. Included this existing version for correctness but I also think this is not strickly required to merge this series as our locking in migrate_to_ram only relies on the core MM locks so some thread would eventually win the race and make forward progress.

So I guess just ignore this patch and will send an updated version individually with installing a migration entry in do_swap_page. If for some reason that doesn't work, I'll respond here explaining why.

Quoted 7 lines
Quoted 5 lines

That would be great. I have a fairly strong preference for doing that instead of adding more special cases for the fault page in the migration code. And if we can't do that it would be good to understand why. Thanks.

I've looked into this and actually prefer the approach in this patch.

Thanks for looking into this.

Quoted 8 lines

Consider the scenario where we install a migration entry, but migrate_to_ram fails. How do we handle this?

We don't know where migrate_to_ram failed. Was migrate_device_finalize called, removing the migration PTE? Do we need to special-case failures in migrate_to_ram to prevent migrate_device_finalize from removing the faulting page's migration entry? Should we check for a migration entry after migrate_to_ram and remove it if it exists?

Quoted 5 lines

The driver should always call migrate_device_finalize(). On failure it will remove the migration entry and remap the original device private page. That obviously doesn't handle the fault but the process is about to die anyway with a SIGBUS because migrate_to_ram() can't fail.

What if migrate_to_ram fails before calling migrate_vma_setup - e.g. a kmalloc of the arguments fails? Very ackward situation.

Quoted 8 lines
Quoted 4 lines

Now, if migrate_to_ram succeeds, it seems the migration entry should be removed in migrate_device_finalize since the new page is only available there. We could return the new page in migrate_to_ram, but that feels messy.

Agreed - I would expect migrate_device_finalize() to always be called and remove the migration entry.

Quoted 6 lines
Quoted 3 lines

Additionally, the page lock needs to be held across migrate_to_ram, as this patch does, so we'll require some special handling in migrate_device_finalize to avoid unlocking the faulting page.

Or just unlock it in migrate_device_finalize(). I agree locking it one place and unlocking it in another is a bit ugly though.

Agreed.

Quoted 10 lines
Quoted 3 lines

Finally, installing a migration entry is non-trivial, while taking a page reference under a lock is straightforward.

I didn't think it was that hard once you have the PTL - although there is a bit of account keeping the same as migrate_vma_collect_pmd() but that could be abstracted into a common function. The advantage is it saves a page table walk, but I suppose that isn't that relevant if you're migrating a group of pages.

A helper would definitely be required if we do this.

Quoted 4 lines
Quoted 1 line

Given all this, I prefer to keep this patch as it is.

Ok, I will take a closer look at it. Thanks.

+1. Let me know what you come up with. This patch doesn't strickly block my work but it would good to have something in to fix this problem soon.

Matt

Quoted 190 lines

- Alistair

Quoted 187 lines

Matt

Quoted 185 lines

- Alistair

Quoted 182 lines

Matt

Quoted 180 lines
Quoted 179 lines
mm/memory.c +3 -0
+			} else {
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+			}
 		} else if (is_hwpoison_entry(entry)) {
 			ret = VM_FAULT_HWPOISON;
 		} else if (is_pte_marker_entry(entry)) {
mm/migrate_device.c +47 -22
index f163c2131022..2477d39f57be 100644
@@ -60,6 +60,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 				   struct mm_walk *walk)
 {
 	struct migrate_vma *migrate = walk->private;
+	struct folio *fault_folio = migrate->fault_page ?
+		page_folio(migrate->fault_page) : NULL;
 	struct vm_area_struct *vma = walk->vma;
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long addr = start, unmapped = 0;
@@ -88,11 +90,13 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
 			folio_get(folio);
 			spin_unlock(ptl);
-			if (unlikely(!folio_trylock(folio)))
+			if (unlikely(fault_folio != folio &&
+				     !folio_trylock(folio)))
 				return migrate_vma_collect_skip(start, end,
 								walk);
 			ret = split_folio(folio);
-			folio_unlock(folio);
+			if (fault_folio != folio)
+				folio_unlock(folio);
 			folio_put(folio);
 			if (ret)
 				return migrate_vma_collect_skip(start, end,
@@ -192,7 +196,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 		 * optimisation to avoid walking the rmap later with
 		 * try_to_migrate().
 		 */
-		if (folio_trylock(folio)) {
+		if (fault_folio == folio || folio_trylock(folio)) {
 			bool anon_exclusive;
 			pte_t swp_pte;
 
@@ -204,7 +208,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
 				if (folio_try_share_anon_rmap_pte(folio, page)) {
 					set_pte_at(mm, addr, ptep, pte);
-					folio_unlock(folio);
+					if (fault_folio != folio)
+						folio_unlock(folio);
 					folio_put(folio);
 					mpfn = 0;
 					goto next;
@@ -363,6 +368,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 					  unsigned long npages,
 					  struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i, restore = 0;
 	bool allow_drain = true;
 	unsigned long unmapped = 0;
@@ -427,7 +434,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 		remove_migration_ptes(folio, folio, 0);
 
 		src_pfns[i] = 0;
-		folio_unlock(folio);
+		if (fault_folio != folio)
+			folio_unlock(folio);
 		folio_put(folio);
 		restore--;
 	}
@@ -536,6 +544,8 @@ int migrate_vma_setup(struct migrate_vma *args)
 		return -EINVAL;
 	if (args->fault_page && !is_device_private_page(args->fault_page))
 		return -EINVAL;
+	if (args->fault_page && !PageLocked(args->fault_page))
+		return -EINVAL;
 
 	memset(args->src, 0, sizeof(*args->src) * nr_pages);
 	args->cpages = 0;
@@ -799,19 +809,13 @@ void migrate_vma_pages(struct migrate_vma *migrate)
 }
 EXPORT_SYMBOL(migrate_vma_pages);
 
-/*
- * migrate_device_finalize() - complete page migration
- * @src_pfns: src_pfns returned from migrate_device_range()
- * @dst_pfns: array of pfns allocated by the driver to migrate memory to
- * @npages: number of pages in the range
- *
- * Completes migration of the page by removing special migration entries.
- * Drivers must ensure copying of page data is complete and visible to the CPU
- * before calling this.
- */
-void migrate_device_finalize(unsigned long *src_pfns,
-			unsigned long *dst_pfns, unsigned long npages)
+static void __migrate_device_finalize(unsigned long *src_pfns,
+				      unsigned long *dst_pfns,
+				      unsigned long npages,
+				      struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i;
 
 	for (i = 0; i < npages; i++) {
@@ -824,7 +828,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
 		if (!page) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);
 				folio_put(dst);
 			}
 			continue;
@@ -834,14 +839,16 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
 		if (!(src_pfns[i] & MIGRATE_PFN_MIGRATE) || !dst) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);
 				folio_put(dst);
 			}
 			dst = src;
 		}
 
 		remove_migration_ptes(src, dst, 0);
-		folio_unlock(src);
+		if (fault_folio != src)
+			folio_unlock(src);
 
 		if (folio_is_zone_device(src))
 			folio_put(src);
@@ -849,7 +856,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 			folio_putback_lru(src);
 
 		if (dst != src) {
-			folio_unlock(dst);
+			if (fault_folio != dst)
+				folio_unlock(dst);
 			if (folio_is_zone_device(dst))
 				folio_put(dst);
 			else
@@ -857,6 +865,22 @@ void migrate_device_finalize(unsigned long *src_pfns,
 		}
 	}
 }
+
+/*
+ * migrate_device_finalize() - complete page migration
+ * @src_pfns: src_pfns returned from migrate_device_range()
+ * @dst_pfns: array of pfns allocated by the driver to migrate memory to
+ * @npages: number of pages in the range
+ *
+ * Completes migration of the page by removing special migration entries.
+ * Drivers must ensure copying of page data is complete and visible to the CPU
+ * before calling this.
+ */
+void migrate_device_finalize(unsigned long *src_pfns,
+			unsigned long *dst_pfns, unsigned long npages)
+{
+	return __migrate_device_finalize(src_pfns, dst_pfns, npages, NULL);
+}
 EXPORT_SYMBOL(migrate_device_finalize);
 
 /**
@@ -872,7 +896,8 @@ EXPORT_SYMBOL(migrate_device_finalize);
  */
 void migrate_vma_finalize(struct migrate_vma *migrate)
 {
-	migrate_device_finalize(migrate->src, migrate->dst, migrate->npages);
+	__migrate_device_finalize(migrate->src, migrate->dst, migrate->npages,
+				  migrate->fault_page);
 }
 EXPORT_SYMBOL(migrate_vma_finalize);
#29 lore raw

Re: [PATCH v2 03/29] mm/migrate: Trylock device page in do_swap_page

In reply to Matthew Brost
Quoted 31 lines · Matthew Brost

Avoid multiple CPU page faults to the same device page racing by trying to lock the page in do_swap_page before taking an extra reference to the page. This prevents scenarios where multiple CPU page faults each take an extra reference to a device page, which could abort migration in folio_migrate_mapping. With the device page being locked in do_swap_page, the migrate_vma_* functions need to be updated to avoid locking the fault_page argument.

Prior to this change, a livelock scenario could occur in Xe's (Intel GPU DRM driver) SVM implementation if enough threads faulted the same device page.

Cc Philip Yang <[email protected]>
Cc Felix Kuehling <[email protected]>
Cc Christian König <[email protected]>
Cc Andrew Morton <[email protected]>
---
 mm/memory.c         | 13 ++++++---
 mm/migrate_device.c | 69 ++++++++++++++++++++++++++++++---------------
 2 files changed, 56 insertions(+), 26 deletions(-)
mm/memory.c +0 -0
index 2366578015ad..b72bde782611 100644
@@ -4252,10 +4252,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 			 * Get a page reference while we know the page can't be
 			 * freed.
 			 */

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4260

Quoted 10 lines
mm/memory.c +6 -4
-			get_page(vmf->page);
-			pte_unmap_unlock(vmf->pte, vmf->ptl);
-			ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
-			put_page(vmf->page);
+			if (trylock_page(vmf->page)) {
+				get_page(vmf->page);
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+				ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
+				put_page(vmf->page);
+				unlock_page(vmf->page);

Isn't the order wrong here? In the common case put_page() will have just dropped the last reference on the page and freed it so the unlock_page() needs to happen first.

Quoted 20 lines
mm/memory.c +3 -0
+			} else {
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+			}
 		} else if (is_hwpoison_entry(entry)) {
 			ret = VM_FAULT_HWPOISON;
 		} else if (is_pte_marker_entry(entry)) {
mm/migrate_device.c +2 -0
index f163c2131022..2477d39f57be 100644
@@ -60,6 +60,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 				   struct mm_walk *walk)
 {
 	struct migrate_vma *migrate = walk->private;
+	struct folio *fault_folio = migrate->fault_page ?
+		page_folio(migrate->fault_page) : NULL;
 	struct vm_area_struct *vma = walk->vma;
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long addr = start, unmapped = 0;

mm/migrate_device.c static int migrate_vma_collect_pmd(pmd_t *pmdp, · line 93

Quoted 5 lines
mm/migrate_device.c +1 -1
@@ -88,11 +90,13 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
 			folio_get(folio);
 			spin_unlock(ptl);
-			if (unlikely(!folio_trylock(folio)))
+			if (unlikely(fault_folio != folio &&

We don't currently support large ZONE_DEVICE pages so we should never get here. I think a WARN_ON_ONCE(fault_folio == folio) and bail would be better.

Quoted 84 lines
mm/migrate_device.c +18 -16
+				     !folio_trylock(folio)))
 				return migrate_vma_collect_skip(start, end,
 								walk);
 			ret = split_folio(folio);
-			folio_unlock(folio);
+			if (fault_folio != folio)
+				folio_unlock(folio);
 			folio_put(folio);
 			if (ret)
 				return migrate_vma_collect_skip(start, end,
@@ -192,7 +196,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 		 * optimisation to avoid walking the rmap later with
 		 * try_to_migrate().
 		 */
-		if (folio_trylock(folio)) {
+		if (fault_folio == folio || folio_trylock(folio)) {
 			bool anon_exclusive;
 			pte_t swp_pte;
 
@@ -204,7 +208,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
 				if (folio_try_share_anon_rmap_pte(folio, page)) {
 					set_pte_at(mm, addr, ptep, pte);
-					folio_unlock(folio);
+					if (fault_folio != folio)
+						folio_unlock(folio);
 					folio_put(folio);
 					mpfn = 0;
 					goto next;
@@ -363,6 +368,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 					  unsigned long npages,
 					  struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i, restore = 0;
 	bool allow_drain = true;
 	unsigned long unmapped = 0;
@@ -427,7 +434,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 		remove_migration_ptes(folio, folio, 0);
 
 		src_pfns[i] = 0;
-		folio_unlock(folio);
+		if (fault_folio != folio)
+			folio_unlock(folio);
 		folio_put(folio);
 		restore--;
 	}
@@ -536,6 +544,8 @@ int migrate_vma_setup(struct migrate_vma *args)
 		return -EINVAL;
 	if (args->fault_page && !is_device_private_page(args->fault_page))
 		return -EINVAL;
+	if (args->fault_page && !PageLocked(args->fault_page))
+		return -EINVAL;
 
 	memset(args->src, 0, sizeof(*args->src) * nr_pages);
 	args->cpages = 0;
@@ -799,19 +809,13 @@ void migrate_vma_pages(struct migrate_vma *migrate)
 }
 EXPORT_SYMBOL(migrate_vma_pages);
 
-/*
- * migrate_device_finalize() - complete page migration
- * @src_pfns: src_pfns returned from migrate_device_range()
- * @dst_pfns: array of pfns allocated by the driver to migrate memory to
- * @npages: number of pages in the range
- *
- * Completes migration of the page by removing special migration entries.
- * Drivers must ensure copying of page data is complete and visible to the CPU
- * before calling this.
- */
-void migrate_device_finalize(unsigned long *src_pfns,
-			unsigned long *dst_pfns, unsigned long npages)
+static void __migrate_device_finalize(unsigned long *src_pfns,
+				      unsigned long *dst_pfns,
+				      unsigned long npages,
+				      struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i;
 
 	for (i = 0; i < npages; i++) {

mm/migrate_device.c void migrate_device_finalize(unsigned long *src_pfns, · line 832

Quoted 6 lines
mm/migrate_device.c +2 -1
@@ -824,7 +828,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
 		if (!page) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);

How could the destination page be the faulting page? I think we can drop this check.

mm/migrate_device.c void migrate_device_finalize(unsigned long *src_pfns, · line 843

Quoted 10 lines
mm/migrate_device.c +2 -1
 				folio_put(dst);
 			}
 			continue;
@@ -834,14 +839,16 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
 		if (!(src_pfns[i] & MIGRATE_PFN_MIGRATE) || !dst) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);

Likewise.

Quoted 13 lines
mm/migrate_device.c +2 -1
 				folio_put(dst);
 			}
 			dst = src;
 		}
 
 		remove_migration_ptes(src, dst, 0);
-		folio_unlock(src);
+		if (fault_folio != src)
+			folio_unlock(src);
 
 		if (folio_is_zone_device(src))
 			folio_put(src);

mm/migrate_device.c void migrate_device_finalize(unsigned long *src_pfns, · line 860

Quoted 6 lines
mm/migrate_device.c +2 -1
@@ -849,7 +856,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 			folio_putback_lru(src);
 
 		if (dst != src) {
-			folio_unlock(dst);
+			if (fault_folio != dst)
+				folio_unlock(dst);

This one also seems unnecessary.

- Alistair

Quoted 35 lines
mm/migrate_device.c +18 -1
 			if (folio_is_zone_device(dst))
 				folio_put(dst);
 			else
@@ -857,6 +865,22 @@ void migrate_device_finalize(unsigned long *src_pfns,
 		}
 	}
 }
+
+/*
+ * migrate_device_finalize() - complete page migration
+ * @src_pfns: src_pfns returned from migrate_device_range()
+ * @dst_pfns: array of pfns allocated by the driver to migrate memory to
+ * @npages: number of pages in the range
+ *
+ * Completes migration of the page by removing special migration entries.
+ * Drivers must ensure copying of page data is complete and visible to the CPU
+ * before calling this.
+ */
+void migrate_device_finalize(unsigned long *src_pfns,
+			unsigned long *dst_pfns, unsigned long npages)
+{
+	return __migrate_device_finalize(src_pfns, dst_pfns, npages, NULL);
+}
 EXPORT_SYMBOL(migrate_device_finalize);
 
 /**
@@ -872,7 +896,8 @@ EXPORT_SYMBOL(migrate_device_finalize);
  */
 void migrate_vma_finalize(struct migrate_vma *migrate)
 {
-	migrate_device_finalize(migrate->src, migrate->dst, migrate->npages);
+	__migrate_device_finalize(migrate->src, migrate->dst, migrate->npages,
+				  migrate->fault_page);
 }
 EXPORT_SYMBOL(migrate_vma_finalize);
author maintainer
#30 lore raw

Re: [PATCH v2 03/29] mm/migrate: Trylock device page in do_swap_page

Quoted 40 lines · Alistair Popple
Quoted 37 lines · Matthew Brost

Avoid multiple CPU page faults to the same device page racing by trying to lock the page in do_swap_page before taking an extra reference to the page. This prevents scenarios where multiple CPU page faults each take an extra reference to a device page, which could abort migration in folio_migrate_mapping. With the device page being locked in do_swap_page, the migrate_vma_* functions need to be updated to avoid locking the fault_page argument.

Prior to this change, a livelock scenario could occur in Xe's (Intel GPU DRM driver) SVM implementation if enough threads faulted the same device page.

Cc Philip Yang <[email protected]>
Cc Felix Kuehling <[email protected]>
Cc Christian König <[email protected]>
Cc Andrew Morton <[email protected]>
---
 mm/memory.c         | 13 ++++++---
 mm/migrate_device.c | 69 ++++++++++++++++++++++++++++++---------------
 2 files changed, 56 insertions(+), 26 deletions(-)
mm/memory.c +2 -4
index 2366578015ad..b72bde782611 100644
@@ -4252,10 +4252,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 			 * Get a page reference while we know the page can't be
 			 * freed.
 			 */
-			get_page(vmf->page);
-			pte_unmap_unlock(vmf->pte, vmf->ptl);
-			ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
-			put_page(vmf->page);
+			if (trylock_page(vmf->page)) {
+				get_page(vmf->page);
Quoted 9 lines

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4260

Quoted 4 lines
mm/memory.c +4 -0
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+				ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
+				put_page(vmf->page);
+				unlock_page(vmf->page);

Isn't the order wrong here? In the common case put_page() will have just dropped the last reference on the page and freed it so the unlock_page() needs to happen first.

Yes, this appears wrong. I haven't seen this show up as a problem but certainly a put should be done after dropping the lock.

Quoted 21 lines
Quoted 21 lines
mm/memory.c +3 -0
+			} else {
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+			}
 		} else if (is_hwpoison_entry(entry)) {
 			ret = VM_FAULT_HWPOISON;
 		} else if (is_pte_marker_entry(entry)) {
mm/migrate_device.c +2 -0
index f163c2131022..2477d39f57be 100644
@@ -60,6 +60,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 				   struct mm_walk *walk)
 {
 	struct migrate_vma *migrate = walk->private;
+	struct folio *fault_folio = migrate->fault_page ?
+		page_folio(migrate->fault_page) : NULL;
 	struct vm_area_struct *vma = walk->vma;
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long addr = start, unmapped = 0;
@@ -88,11 +90,13 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
Quoted 9 lines

mm/migrate_device.c static int migrate_vma_collect_pmd(pmd_t *pmdp, · line 93

Quoted 4 lines
mm/migrate_device.c +1 -1
 			folio_get(folio);
 			spin_unlock(ptl);
-			if (unlikely(!folio_trylock(folio)))
+			if (unlikely(fault_folio != folio &&

We don't currently support large ZONE_DEVICE pages so we should never get here. I think a WARN_ON_ONCE(fault_folio == folio) and bail would be better.

Sure will fix.

Quoted 85 lines
Quoted 85 lines
mm/migrate_device.c +18 -16
+				     !folio_trylock(folio)))
 				return migrate_vma_collect_skip(start, end,
 								walk);
 			ret = split_folio(folio);
-			folio_unlock(folio);
+			if (fault_folio != folio)
+				folio_unlock(folio);
 			folio_put(folio);
 			if (ret)
 				return migrate_vma_collect_skip(start, end,
@@ -192,7 +196,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 		 * optimisation to avoid walking the rmap later with
 		 * try_to_migrate().
 		 */
-		if (folio_trylock(folio)) {
+		if (fault_folio == folio || folio_trylock(folio)) {
 			bool anon_exclusive;
 			pte_t swp_pte;
 
@@ -204,7 +208,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
 				if (folio_try_share_anon_rmap_pte(folio, page)) {
 					set_pte_at(mm, addr, ptep, pte);
-					folio_unlock(folio);
+					if (fault_folio != folio)
+						folio_unlock(folio);
 					folio_put(folio);
 					mpfn = 0;
 					goto next;
@@ -363,6 +368,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 					  unsigned long npages,
 					  struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i, restore = 0;
 	bool allow_drain = true;
 	unsigned long unmapped = 0;
@@ -427,7 +434,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 		remove_migration_ptes(folio, folio, 0);
 
 		src_pfns[i] = 0;
-		folio_unlock(folio);
+		if (fault_folio != folio)
+			folio_unlock(folio);
 		folio_put(folio);
 		restore--;
 	}
@@ -536,6 +544,8 @@ int migrate_vma_setup(struct migrate_vma *args)
 		return -EINVAL;
 	if (args->fault_page && !is_device_private_page(args->fault_page))
 		return -EINVAL;
+	if (args->fault_page && !PageLocked(args->fault_page))
+		return -EINVAL;
 
 	memset(args->src, 0, sizeof(*args->src) * nr_pages);
 	args->cpages = 0;
@@ -799,19 +809,13 @@ void migrate_vma_pages(struct migrate_vma *migrate)
 }
 EXPORT_SYMBOL(migrate_vma_pages);
 
-/*
- * migrate_device_finalize() - complete page migration
- * @src_pfns: src_pfns returned from migrate_device_range()
- * @dst_pfns: array of pfns allocated by the driver to migrate memory to
- * @npages: number of pages in the range
- *
- * Completes migration of the page by removing special migration entries.
- * Drivers must ensure copying of page data is complete and visible to the CPU
- * before calling this.
- */
-void migrate_device_finalize(unsigned long *src_pfns,
-			unsigned long *dst_pfns, unsigned long npages)
+static void __migrate_device_finalize(unsigned long *src_pfns,
+				      unsigned long *dst_pfns,
+				      unsigned long npages,
+				      struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i;
 
 	for (i = 0; i < npages; i++) {
@@ -824,7 +828,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
Quoted 9 lines

mm/migrate_device.c void migrate_device_finalize(unsigned long *src_pfns, · line 832

Quoted 5 lines
mm/migrate_device.c +2 -1
 		if (!page) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);

How could the destination page be the faulting page? I think we can drop this check.

Yea, will drop.

Quoted 5 lines
Quoted 5 lines
mm/migrate_device.c +0 -0
 				folio_put(dst);
 			}
 			continue;
@@ -834,14 +839,16 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
Quoted 8 lines

mm/migrate_device.c void migrate_device_finalize(unsigned long *src_pfns, · line 843

Quoted 5 lines
mm/migrate_device.c +2 -1
 		if (!(src_pfns[i] & MIGRATE_PFN_MIGRATE) || !dst) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);

Likewise.

Same here.

Quoted 15 lines
Quoted 15 lines
mm/migrate_device.c +2 -1
 				folio_put(dst);
 			}
 			dst = src;
 		}
 
 		remove_migration_ptes(src, dst, 0);
-		folio_unlock(src);
+		if (fault_folio != src)
+			folio_unlock(src);
 
 		if (folio_is_zone_device(src))
 			folio_put(src);
@@ -849,7 +856,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 			folio_putback_lru(src);
 
Quoted 7 lines

mm/migrate_device.c void migrate_device_finalize(unsigned long *src_pfns, · line 860

Quoted 4 lines
mm/migrate_device.c +2 -1
 		if (dst != src) {
-			folio_unlock(dst);
+			if (fault_folio != dst)
+				folio_unlock(dst);

This one also seems unnecessary.

Same here.

Thanks - Matt

Quoted 38 lines

- Alistair

Quoted 35 lines
mm/migrate_device.c +18 -1
 			if (folio_is_zone_device(dst))
 				folio_put(dst);
 			else
@@ -857,6 +865,22 @@ void migrate_device_finalize(unsigned long *src_pfns,
 		}
 	}
 }
+
+/*
+ * migrate_device_finalize() - complete page migration
+ * @src_pfns: src_pfns returned from migrate_device_range()
+ * @dst_pfns: array of pfns allocated by the driver to migrate memory to
+ * @npages: number of pages in the range
+ *
+ * Completes migration of the page by removing special migration entries.
+ * Drivers must ensure copying of page data is complete and visible to the CPU
+ * before calling this.
+ */
+void migrate_device_finalize(unsigned long *src_pfns,
+			unsigned long *dst_pfns, unsigned long npages)
+{
+	return __migrate_device_finalize(src_pfns, dst_pfns, npages, NULL);
+}
 EXPORT_SYMBOL(migrate_device_finalize);
 
 /**
@@ -872,7 +896,8 @@ EXPORT_SYMBOL(migrate_device_finalize);
  */
 void migrate_vma_finalize(struct migrate_vma *migrate)
 {
-	migrate_device_finalize(migrate->src, migrate->dst, migrate->npages);
+	__migrate_device_finalize(migrate->src, migrate->dst, migrate->npages,
+				  migrate->fault_page);
 }
 EXPORT_SYMBOL(migrate_vma_finalize);
author maintainer
#31 lore raw

Re: [PATCH v2 03/29] mm/migrate: Trylock device page in do_swap_page

Quoted 182 lines · Matthew Brost
Quoted 40 lines · Alistair Popple
Quoted 37 lines · Matthew Brost

Avoid multiple CPU page faults to the same device page racing by trying to lock the page in do_swap_page before taking an extra reference to the page. This prevents scenarios where multiple CPU page faults each take an extra reference to a device page, which could abort migration in folio_migrate_mapping. With the device page being locked in do_swap_page, the migrate_vma_* functions need to be updated to avoid locking the fault_page argument.

Prior to this change, a livelock scenario could occur in Xe's (Intel GPU DRM driver) SVM implementation if enough threads faulted the same device page.

Cc Philip Yang <[email protected]>
Cc Felix Kuehling <[email protected]>
Cc Christian König <[email protected]>
Cc Andrew Morton <[email protected]>
---
 mm/memory.c         | 13 ++++++---
 mm/migrate_device.c | 69 ++++++++++++++++++++++++++++++---------------
 2 files changed, 56 insertions(+), 26 deletions(-)
mm/memory.c +2 -4
index 2366578015ad..b72bde782611 100644
@@ -4252,10 +4252,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 			 * Get a page reference while we know the page can't be
 			 * freed.
 			 */
-			get_page(vmf->page);
-			pte_unmap_unlock(vmf->pte, vmf->ptl);
-			ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
-			put_page(vmf->page);
+			if (trylock_page(vmf->page)) {
+				get_page(vmf->page);
Quoted 9 lines

mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4260

Quoted 4 lines
mm/memory.c +4 -0
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+				ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
+				put_page(vmf->page);
+				unlock_page(vmf->page);

Isn't the order wrong here? In the common case put_page() will have just dropped the last reference on the page and freed it so the unlock_page() needs to happen first.

Yes, this appears wrong. I haven't seen this show up as a problem but certainly a put should be done after dropping the lock.

Quoted 21 lines
Quoted 21 lines
mm/memory.c +3 -0
+			} else {
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+			}
 		} else if (is_hwpoison_entry(entry)) {
 			ret = VM_FAULT_HWPOISON;
 		} else if (is_pte_marker_entry(entry)) {
mm/migrate_device.c +2 -0
index f163c2131022..2477d39f57be 100644
@@ -60,6 +60,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 				   struct mm_walk *walk)
 {
 	struct migrate_vma *migrate = walk->private;
+	struct folio *fault_folio = migrate->fault_page ?
+		page_folio(migrate->fault_page) : NULL;
 	struct vm_area_struct *vma = walk->vma;
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long addr = start, unmapped = 0;
@@ -88,11 +90,13 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
Quoted 9 lines

mm/migrate_device.c static int migrate_vma_collect_pmd(pmd_t *pmdp, · line 93

Quoted 4 lines
mm/migrate_device.c +1 -1
 			folio_get(folio);
 			spin_unlock(ptl);
-			if (unlikely(!folio_trylock(folio)))
+			if (unlikely(fault_folio != folio &&

We don't currently support large ZONE_DEVICE pages so we should never get here. I think a WARN_ON_ONCE(fault_folio == folio) and bail would be better.

Sure will fix.

Quoted 85 lines
Quoted 85 lines
mm/migrate_device.c +18 -16
+				     !folio_trylock(folio)))
 				return migrate_vma_collect_skip(start, end,
 								walk);
 			ret = split_folio(folio);
-			folio_unlock(folio);
+			if (fault_folio != folio)
+				folio_unlock(folio);
 			folio_put(folio);
 			if (ret)
 				return migrate_vma_collect_skip(start, end,
@@ -192,7 +196,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 		 * optimisation to avoid walking the rmap later with
 		 * try_to_migrate().
 		 */
-		if (folio_trylock(folio)) {
+		if (fault_folio == folio || folio_trylock(folio)) {
 			bool anon_exclusive;
 			pte_t swp_pte;
 
@@ -204,7 +208,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 
 				if (folio_try_share_anon_rmap_pte(folio, page)) {
 					set_pte_at(mm, addr, ptep, pte);
-					folio_unlock(folio);
+					if (fault_folio != folio)
+						folio_unlock(folio);
 					folio_put(folio);
 					mpfn = 0;
 					goto next;
@@ -363,6 +368,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 					  unsigned long npages,
 					  struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i, restore = 0;
 	bool allow_drain = true;
 	unsigned long unmapped = 0;
@@ -427,7 +434,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns,
 		remove_migration_ptes(folio, folio, 0);
 
 		src_pfns[i] = 0;
-		folio_unlock(folio);
+		if (fault_folio != folio)
+			folio_unlock(folio);
 		folio_put(folio);
 		restore--;
 	}
@@ -536,6 +544,8 @@ int migrate_vma_setup(struct migrate_vma *args)
 		return -EINVAL;
 	if (args->fault_page && !is_device_private_page(args->fault_page))
 		return -EINVAL;
+	if (args->fault_page && !PageLocked(args->fault_page))
+		return -EINVAL;
 
 	memset(args->src, 0, sizeof(*args->src) * nr_pages);
 	args->cpages = 0;
@@ -799,19 +809,13 @@ void migrate_vma_pages(struct migrate_vma *migrate)
 }
 EXPORT_SYMBOL(migrate_vma_pages);
 
-/*
- * migrate_device_finalize() - complete page migration
- * @src_pfns: src_pfns returned from migrate_device_range()
- * @dst_pfns: array of pfns allocated by the driver to migrate memory to
- * @npages: number of pages in the range
- *
- * Completes migration of the page by removing special migration entries.
- * Drivers must ensure copying of page data is complete and visible to the CPU
- * before calling this.
- */
-void migrate_device_finalize(unsigned long *src_pfns,
-			unsigned long *dst_pfns, unsigned long npages)
+static void __migrate_device_finalize(unsigned long *src_pfns,
+				      unsigned long *dst_pfns,
+				      unsigned long npages,
+				      struct page *fault_page)
 {
+	struct folio *fault_folio = fault_page ?
+		page_folio(fault_page) : NULL;
 	unsigned long i;
 
 	for (i = 0; i < npages; i++) {
@@ -824,7 +828,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
Quoted 9 lines

mm/migrate_device.c void migrate_device_finalize(unsigned long *src_pfns, · line 832

Quoted 5 lines
mm/migrate_device.c +2 -1
 		if (!page) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);

How could the destination page be the faulting page? I think we can drop this check.

Quoted 2 lines

Yea, will drop.

Actually landing on a WARN_ON_ONCE(fault_folio == dst) to catch potentail bugs. Will include that in my next rev. If you disagree let me know and can discuss ahead of next rev.

Matt

Quoted 81 lines
Quoted 5 lines
Quoted 5 lines
mm/migrate_device.c +0 -0
 				folio_put(dst);
 			}
 			continue;
@@ -834,14 +839,16 @@ void migrate_device_finalize(unsigned long *src_pfns,
 
Quoted 8 lines

mm/migrate_device.c void migrate_device_finalize(unsigned long *src_pfns, · line 843

Quoted 5 lines
mm/migrate_device.c +2 -1
 		if (!(src_pfns[i] & MIGRATE_PFN_MIGRATE) || !dst) {
 			if (dst) {
-				folio_unlock(dst);
+				if (fault_folio != dst)
+					folio_unlock(dst);

Likewise.

Same here.

Quoted 15 lines
Quoted 15 lines
mm/migrate_device.c +2 -1
 				folio_put(dst);
 			}
 			dst = src;
 		}
 
 		remove_migration_ptes(src, dst, 0);
-		folio_unlock(src);
+		if (fault_folio != src)
+			folio_unlock(src);
 
 		if (folio_is_zone_device(src))
 			folio_put(src);
@@ -849,7 +856,8 @@ void migrate_device_finalize(unsigned long *src_pfns,
 			folio_putback_lru(src);
 
Quoted 7 lines

mm/migrate_device.c void migrate_device_finalize(unsigned long *src_pfns, · line 860

Quoted 4 lines
mm/migrate_device.c +2 -1
 		if (dst != src) {
-			folio_unlock(dst);
+			if (fault_folio != dst)
+				folio_unlock(dst);

This one also seems unnecessary.

Same here.

Thanks - Matt

Quoted 38 lines

- Alistair

Quoted 35 lines
mm/migrate_device.c +18 -1
 			if (folio_is_zone_device(dst))
 				folio_put(dst);
 			else
@@ -857,6 +865,22 @@ void migrate_device_finalize(unsigned long *src_pfns,
 		}
 	}
 }
+
+/*
+ * migrate_device_finalize() - complete page migration
+ * @src_pfns: src_pfns returned from migrate_device_range()
+ * @dst_pfns: array of pfns allocated by the driver to migrate memory to
+ * @npages: number of pages in the range
+ *
+ * Completes migration of the page by removing special migration entries.
+ * Drivers must ensure copying of page data is complete and visible to the CPU
+ * before calling this.
+ */
+void migrate_device_finalize(unsigned long *src_pfns,
+			unsigned long *dst_pfns, unsigned long npages)
+{
+	return __migrate_device_finalize(src_pfns, dst_pfns, npages, NULL);
+}
 EXPORT_SYMBOL(migrate_device_finalize);
 
 /**
@@ -872,7 +896,8 @@ EXPORT_SYMBOL(migrate_device_finalize);
  */
 void migrate_vma_finalize(struct migrate_vma *migrate)
 {
-	migrate_device_finalize(migrate->src, migrate->dst, migrate->npages);
+	__migrate_device_finalize(migrate->src, migrate->dst, migrate->npages,
+				  migrate->fault_page);
 }
 EXPORT_SYMBOL(migrate_vma_finalize);
author maintainer
#32 lore raw

PATCH v2 4/29 drm/pagemap: Add DRM pagemap

In reply to Matthew Brost

From: Thomas Hellström <[email protected]>

Introduce drm_pagemap ops to map and unmap dma to VRAM resources. In the local memory case it's a matter of merely providing an offset into the device's physical address. For future p2p the map and unmap functions may encode as needed.

Similar to how dma-buf works, let the memory provider (drm_pagemap) provide the mapping functionality.

---
 drivers/gpu/drm/xe/drm_pagemap.h | 103 +++++++++++++++++++++++++++++++
 1 file changed, 103 insertions(+)
 create mode 100644 drivers/gpu/drm/xe/drm_pagemap.h
drivers/gpu/drm/xe/drm_pagemap.h new file +103 -0
new file mode 100644
index 000000000000..b6b387b81479
@@ -0,0 +1,103 @@
+/* SPDX-License-Identifier: MIT */
+#ifndef _DRM_PAGEMAP_H_
+#define _DRM_PAGEMAP_H_
+
+#include <linux/dma-direction.h>
+#include <linux/hmm.h>
+#include <linux/types.h>
+
+struct drm_pagemap;
+struct device;
+
+/**
+ * enum drm_interconnect_protocol - Used to identify an interconnect protocol.
+ */
+enum drm_interconnect_protocol {
+	DRM_INTERCONNECT_SYSTEM,    /* DMA map is system pages. */
+	DRM_INTERCONNECT_PCIE_P2P,  /* DMA map is PCIE P2P */
+	DRM_INTERCONNECT_DRIVER,    /* DMA map is driver defined */
+	/* A driver can add private values beyond DRM_INTERCONNECT_DRIVER */
+};
+
+/**
+ * struct drm_pagemap_dma_addr - DMA address representation.
+ * @addr: The dma address or driver-defined address for driver private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size is PAGE_SIZE << order).
+ * @dir: The DMA direction.
+ *
+ * Note: There is room for improvement here. We should be able to pack into
+ * 64 bits.
+ */
+struct drm_pagemap_dma_addr {
+	dma_addr_t addr;
+	u64 proto : 54;
+	u64 order : 8;
+	u64 dir : 2;
+};
+
+/**
+ * drm_pagemap_dma_addr_encode() - Encode a dma address with metadata
+ * @addr: The dma address or driver-defined address for driver private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size is PAGE_SIZE << order).
+ * @dir: The DMA direction.
+ *
+ * Return: A struct drm_pagemap_dma_addr encoding the above information.
+ */
+static inline struct drm_pagemap_dma_addr
+drm_pagemap_dma_addr_encode(dma_addr_t addr,
+			    enum drm_interconnect_protocol proto,
+			    unsigned int order,
+			    enum dma_data_direction dir)
+{
+	return (struct drm_pagemap_dma_addr) {
+		.addr = addr,
+		.proto = proto,
+		.order = order,
+		.dir = dir,
+	};
+}
+
+/**
+ * struct drm_pagemap_ops: Ops for a drm-pagemap.
+ */
+struct drm_pagemap_ops {
+	/**
+	 * @map_dma: Map for dma access or provide a virtual address suitable for
+	 * @dev.
+	 * @dpagemap: The struct drm_pagemap for the page.
+	 * @dev: The dma mapper.
+	 * @page: The page to map.
+	 * @dir: The transfer direction.
+	 * @protocol: The protocol to use.
+	 */
+	struct drm_pagemap_dma_addr (*map_dma)(struct drm_pagemap *dpagemap,
+					       struct device *dev,
+					       struct page *page,
+					       unsigned int order,
+					       enum dma_data_direction dir);
+
+	/**
+	 * @unmap_dma: Unmap a dma address previously obtained using  @map_dma.
+	 * @dev: The dma unmapper.
+	 * @addr: The dma address obtained when mapping.
+	 */
+	void (*unmap_dma)(struct drm_pagemap *dpagemap,
+			  struct device *dev,
+			  struct drm_pagemap_dma_addr addr);
+
+};
+
+/**
+ * struct drm_pagemap: Additional information for a struct dev_pagemap
+ * used for device p2p handshaking.
+ * @ops: The struct drm_pagemap_ops.
+ * @dev: The struct drevice owning the device-private memory.
+ */
+struct drm_pagemap {
+	const struct drm_pagemap_ops *ops;
+	struct device *dev;
+};
+
+#endif
Signature · 1 line
2.34.1
author maintainer
#33 lore raw

PATCH v2 5/29 drm/gpusvm: Add support for GPU Shared Virtual Memory

In reply to Matthew Brost

This patch introduces support for GPU Shared Virtual Memory (SVM) in the Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.

The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also provides mechanisms for allocating, deallocating, and migrating memory regions between system RAM and GPU VRAM.

This is largely inspired by GPUVM.

v2:
- Take order into account in check pages
- Clear range->pages in get pages error
- Drop setting dirty or accessed bit in get pages (Vetter)
- Remove mmap assert for cpu faults
- Drop mmap write lock abuse (Vetter, Christian)
- Decouple zdd from range (Vetter, Oak)
- Add drm_gpusvm_range_evict, make it work with coherent pages
- Export drm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
- mmget/put in drm_gpusvm_evict_to_sram
- Drop range->vram_alloation variable
- Don't return in drm_gpusvm_evict_to_sram until all pages detached
- Don't warn on mixing sram and device pages
- Update kernel doc
- Add coherent page support to get pages
- Use DMA_FROM_DEVICE rather than DMA_BIDIRECTIONAL
- Add struct drm_gpusvm_vram and ops (Thomas)
- Update the range's seqno if the range is valid (Thomas)
- Remove the is_unmapped check before hmm_range_fault (Thomas)
- Use drm_pagemap (Thomas)
- Drop kfree_mapping (Thomas)
- dma mapp pages under notifier lock (Thomas)
- Remove ctx.prefault
- Remove ctx.mmap_locked
- Add ctx.check_pages
- s/vram/devmem (Thomas)

---
 drivers/gpu/drm/xe/Makefile     |    3 +-
 drivers/gpu/drm/xe/drm_gpusvm.c | 2074 +++++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/drm_gpusvm.h |  447 +++++++
 3 files changed, 2523 insertions(+), 1 deletion(-)
 create mode 100644 drivers/gpu/drm/xe/drm_gpusvm.c
 create mode 100644 drivers/gpu/drm/xe/drm_gpusvm.h
drivers/gpu/drm/xe/Makefile +2 -1
index da80c29aa363..8d991d4a92a5 100644
@@ -25,7 +25,8 @@ $(obj)/generated/%_wa_oob.c $(obj)/generated/%_wa_oob.h: $(obj)/xe_gen_wa_oob \
 
 # core driver code
 
-xe-y += xe_bb.o \
+xe-y += drm_gpusvm.o \
+	xe_bb.o \
 	xe_bo.o \
 	xe_bo_evict.o \
 	xe_devcoredump.o \
drivers/gpu/drm/xe/drm_gpusvm.c new file +2074 -0

2,074 lines not on this page · raw message

drivers/gpu/drm/xe/drm_gpusvm.h new file +447 -0

447 lines not on this page · raw message

Signature · 1 line
2.34.1
#34 lore raw

Re: [PATCH v2 05/29] drm/gpusvm: Add support for GPU Shared Virtual Memory

Quoted 174 lines · Matthew Brost

This patch introduces support for GPU Shared Virtual Memory (SVM) in the
Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.

The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also
provides mechanisms for allocating, deallocating, and migrating memory
regions between system RAM and GPU VRAM.

This is largely inspired by GPUVM.

v2:
 - Take order into account in check pages
 - Clear range->pages in get pages error
 - Drop setting dirty or accessed bit in get pages (Vetter)
 - Remove mmap assert for cpu faults
 - Drop mmap write lock abuse (Vetter, Christian)
 - Decouple zdd from range (Vetter, Oak)
 - Add drm_gpusvm_range_evict, make it work with coherent pages
 - Export drm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
 - mmget/put in drm_gpusvm_evict_to_sram
 - Drop range->vram_alloation variable
 - Don't return in drm_gpusvm_evict_to_sram until all pages detached
 - Don't warn on mixing sram and device pages
 - Update kernel doc
 - Add coherent page support to get pages
 - Use DMA_FROM_DEVICE rather than DMA_BIDIRECTIONAL
 - Add struct drm_gpusvm_vram and ops (Thomas)
 - Update the range's seqno if the range is valid (Thomas)
 - Remove the is_unmapped check before hmm_range_fault (Thomas)
 - Use drm_pagemap (Thomas)
 - Drop kfree_mapping (Thomas)
 - dma mapp pages under notifier lock (Thomas)
 - Remove ctx.prefault
 - Remove ctx.mmap_locked
 - Add ctx.check_pages
 - s/vram/devmem (Thomas)

---
 drivers/gpu/drm/xe/Makefile     |    3 +-
 drivers/gpu/drm/xe/drm_gpusvm.c | 2074
+++++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/drm_gpusvm.h |  447 +++++++

 3 files changed, 2523 insertions(+), 1 deletion(-)

 create mode 100644 drivers/gpu/drm/xe/drm_gpusvm.c create mode 100644 drivers/gpu/drm/xe/drm_gpusvm.h

drivers/gpu/drm/xe/Makefile +0 -0
index da80c29aa363..8d991d4a92a5 100644
@@ -25,7 +25,8 @@ $(obj)/generated/%_wa_oob.c
 $(obj)/generated/%_wa_oob.h: $(obj)/xe_gen_wa_oob \

 # core driver code
 
drivers/gpu/drm/xe/Makefile +2 -1
-xe-y += xe_bb.o \
+xe-y += drm_gpusvm.o \
+	xe_bb.o \

  xe_bo.o \
  xe_bo_evict.o \
  xe_devcoredump.o \

drivers/gpu/drm/xe/drm_gpusvm.c new file +26 -0
new file mode 100644
index 000000000000..1ff104d2b42c
@@ -0,0 +1,2074 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2024 Intel Corporation
+ *
+ * Authors:
+ *     Matthew Brost <[email protected]>
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/interval_tree_generic.h>
+#include <linux/hmm.h>
+#include <linux/memremap.h>
+#include <linux/migrate.h>
+#include <linux/mm_types.h>
+#include <linux/pagemap.h>
+#include <linux/slab.h>
+
+#include <drm/drm_device.h>
+#include "drm/drm_print.h"
+#include "drm_gpusvm.h"
+#include "drm_pagemap.h"
+
+/**
+ * DOC: Overview
+ *
+ * GPU Shared Virtual Memory (GPU SVM) layer for the Direct

Rendering Manager (DRM)
+ *
+ * The GPU SVM layer is a component of the DRM framework designed to manage shared
+ * virtual memory between the CPU and GPU. It enables efficient data exchange and
+ * processing for GPU-accelerated applications by allowing memory sharing and
+ * synchronization between the CPU's and GPU's virtual address spaces.
+ *
+ * Key GPU SVM Components:
+ * - Notifiers: Notifiers: Used for tracking memory intervals and notifying the
+ * GPU of changes, notifiers are sized based on a GPU
SVM
+ * initialization parameter, with a recommendation of
512M or
+ * larger. They maintain a Red-BlacK tree and a list of
ranges that
+ * fall within the notifier interval. Notifiers are
tracked within
+ * a GPU SVM Red-BlacK tree and list and are
dynamically inserted
+ * or removed as ranges within the interval are created
or
+ * destroyed.
+ * - Ranges: Represent memory ranges mapped in a DRM device and managed
+ *      by GPU SVM. They are sized based on an array of chunk
sizes, which
+ *      is a GPU SVM initialization parameter, and the CPU
address space.
+ *      Upon GPU fault, the largest aligned chunk that fits
within the
+ *      faulting CPU address space is chosen for the range
size. Ranges are
+ *      expected to be dynamically allocated on GPU fault and
removed on an
+ *      MMU notifier UNMAP event. As mentioned above, ranges
are tracked in
+ *      a notifier's Red-Black tree.
+ * - Operations: Define the interface for driver-specific GPU SVM operations
+ *               such as range allocation, notifier allocation, and
+ *               invalidations.
+ * - Device Memory Allocations: Embedded structure containing enough information
+ *                              for GPU SVM to migrate to / from device memory.
+ * - Device Memory Operations: Define the interface for driver- specific device
+ *                             memory operations release memory, populate pfns,
+ *                             and copy to / from device memory.
+ *
+ * This layer provides interfaces for allocating, mapping, migrating, and
+ * releasing memory ranges between the CPU and GPU. It handles all core memory
+ * management interactions (DMA mapping, HMM, and migration) and provides
+ * driver-specific virtual functions (vfuncs). This infrastructure is sufficient
+ * to build the expected driver components for an SVM implementation as detailed
+ * below.

Quoted 8 lines

+ *
+ * Expected Driver Components:
+ * - GPU page fault handler: Used to create ranges and notifiers based on the
+ *      fault address, optionally migrate the
range to
+ *      device memory, and create GPU bindings.
+ * - Garbage collector: Used to destroy GPU bindings for ranges.

Perhaps "clean up GPU bindings for ranges" to differentiate from unmapping GPU bindings for ranges which needs to be done in the notifier callback?

Quoted 6 lines

Ranges are
+ * expected to be added to the garbage
collector upon
+ * MMU_NOTIFY_UNMAP event.
+ */
+

- Notifier callback, to unmap GPU bindings for ranges.

Quoted 5 lines

+/**
+ * DOC: Locking
+ *
+ * GPU SVM handles locking for core MM interactions, i.e., it locks/unlocks the

Quoted 8 lines

+ * mmap lock as needed.
+ *
+ * GPU SVM introduces a global notifier lock, which safeguards the notifier's
+ * range RB tree and list, as well as the range's DMA mappings and sequence
+ * number. GPU SVM manages all necessary locking and unlocking operations,

How difficult would it be to make this per-notifier?
One of the design comments we got from Jason was to prioritize avoiding core slowdowns and any fault processing might block an unrelated notifier during dma mapping and page-table commit.
I think this should at least be considered as a follow-up.

Quoted 3 lines

+ * except for the recheck of the range's sequence number
+ * (mmu_interval_read_retry) when the driver is committing GPU bindings. This

Perhaps add a discussion on valid pages rather than valid sequence number, since the sequence number might be bumped even if pages stay valid for the range, as the sequence number spans the whole notifier.

Quoted 9 lines

+ * lock corresponds to the 'driver->update' lock mentioned in the HMM
+ * documentation (TODO: Link). Future revisions may transition from a GPU SVM
+ * global lock to a per-notifier lock if finer-grained locking is deemed
+ * necessary.
+ *
+ * In addition to the locking mentioned above, the driver should

Quoted 8 lines

implement a
+ * lock to safeguard core GPU SVM function calls that modify state, such as
+ * drm_gpusvm_range_find_or_insert and drm_gpusvm_range_remove. Alternatively,
+ * these core functions can be called within a single kernel thread, for
+ * instance, using an ordered work queue.

I don't think not we should encourage the use of ordered workqueues to protect data / state? Locks should really be the preferred mechanism?

Quoted 5 lines

This lock is denoted as
+ * 'driver_svm_lock' in code examples. Finer grained driver side locking should
+ * also be possible for concurrent GPU fault processing within a single GPU SVM.

GPUVM (or rather the gem part of GPUVM that resides in drm_gem.h) allows for the driver to register a lock map which, if present, is used in the code to assert locks are correctly taken.

In the xe example, if we're using the vm lock to protect, then we could register that and thoroughly annotate the gpusvm code with lockdep asserts. That will probably help making the code a lot easier to maintain moving forward.

Quoted 13 lines

+ */
+GPUVM (or rGPUVM (or rather the gem part of GPUVM that resides in drm_gem.h) allows for the driver to register a lock map which, if present, is used in the code to assert locks are correctly taken.

In the xe example, if we're using the vm lock to protect, then we could register that and thoroughly annotate the gpusvm code with lockdep asserts. That will probably help making the code a lot easier to maintain moving forward.GPUVM (or rather the gem part of GPUVM that resides in drm_gem.h) allows for the driver to register a lock map which, if present, is used in the code to assert locks are correctly taken.

Quoted 6 lines

In the xe example, if we're using the vm lock to protect, then we could register that and thoroughly annotate the gpusvm code with lockdep asserts. That will probably help making the code a lot easier to maintain moving forward.ather the gem part of GPUVM that resides in drm_gem.h) allows for the driver to register a lock map which, if present, is used in the code to assert locks are correctly taken.

In the xe example, if we're using the vm lock to protect, then we could register that and thoroughly annotate the gpusvm code with lockdep asserts. That will probably help making the code a lot easier to maintain moving forward.

Quoted 2 lines

+/**
+ * DOC: Migrataion

s/Migrataion/Migration/

Quoted 209 lines

+ *
+ * The migration support is quite simple, allowing migration between RAM and
+ * device memory at the range granularity. For example, GPU SVM currently does not
+ * support mixing RAM and device memory pages within a range. This means that upon GPU
+ * fault, the entire range can be migrated to device memory, and upon CPU fault, the
+ * entire range is migrated to RAM. Mixed RAM and device memory storage within a range
+ * could be added in the future if required.
+ *
+ * The reasoning for only supporting range granularity is as
follows: it
+ * simplifies the implementation, and range sizes are driver-defined and should
+ * be relatively small.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Partial Unmapping of Ranges
+ *
+ * Partial unmapping of ranges (e.g., 1M out of 2M is unmapped by

CPU resulting
+ * in MMU_NOTIFY_UNMAP event) presents several challenges, with the main one
+ * being that a subset of the range still has CPU and GPU mappings. If the
+ * backing store for the range is in device memory, a subset of the backing store has
+ * references. One option would be to split the range and device memory backing store,
+ * but the implementation for this would be quite complicated. Given that
+ * partial unmappings are rare and driver-defined range sizes are relatively
+ * small, GPU SVM does not support splitting of ranges.
+ *
+ * With no support for range splitting, upon partial unmapping of a range, the
+ * driver is expected to invalidate and destroy the entire range. If the range
+ * has device memory as its backing, the driver is also expected to migrate any
+ * remaining pages back to RAM.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Examples
+ *
+ * This section provides two examples of how to build the expected

driver
+ * components: the GPU page fault handler and the garbage collector. A third
+ * example demonstrates a sample invalidation driver vfunc.
+ *
+ * The generic code provided does not include logic for complex migration
+ * policies, optimized invalidations, fined grained driver locking, or other
+ * potentially required driver locking (e.g., DMA-resv locks).
+ *
+ * 1) GPU page fault handler
+ *
+ * int driver_bind_range(struct drm_gpusvm *gpusvm, struct
drm_gpusvm_range *range)
+ * {
+ * int err = 0;
+ *
+ * driver_alloc_and_setup_memory_for_bind(gpusvm,
range);
+ *
+ * drm_gpusvm_notifier_lock(gpusvm);
+ * if (drm_gpusvm_range_pages_valid(range))
+ * driver_commit_bind(gpusvm, range);
+ * else
+ * err = -EAGAIN;
+ * drm_gpusvm_notifier_unlock(gpusvm);
+ *
+ * return err;
+ * }
+ *
+ * int driver_gpu_fault(struct drm_gpusvm *gpusvm, u64
fault_addr,
+ *      u64 gpuva_start, u64 gpuva_end)
+ * {
+ * struct drm_gpusvm_ctx ctx = {};
+ * int err;
+ *
+ * driver_svm_lock();
+ * retry:
+ * // Always process UNMAPs first so view of GPU SVM
ranges is current
+ * driver_garbage_collector(gpusvm);
+ *
+ * range = drm_gpusvm_range_find_or_insert(gpusvm,
fault_addr,
+ * gpuva_start,
gpuva_end,
+ *         &ctx);
+ * if (IS_ERR(range)) {
+ * err = PTR_ERR(range);
+ * goto unlock;
+ * }
+ *
+ * if (driver_migration_policy(range)) {
+ * devmem = driver_alloc_devmem();
+ * err = drm_gpusvm_migrate_to_devmem(gpusvm,
range,
+ *
devmem_allocation,
+ *    &ctx);
+ * if (err) // CPU mappings may have
changed
+ * goto retry;
+ * }
+ *
+ * err = drm_gpusvm_range_get_pages(gpusvm, range,
&ctx);
+ * if (err == -EOPNOTSUPP || err == -EFAULT || err == -
EPERM) { // CPU mappings changed
+ * if (err == -EOPNOTSUPP)
+ * drm_gpusvm_range_evict(gpusvm,
range);
+ * goto retry;
+ * } else if (err) {
+ * goto unlock;
+ * }
+ *
+ * err = driver_bind_range(gpusvm, range);
+ * if (err == -EAGAIN) // CPU mappings changed
+ * goto retry
+ *
+ * unlock:
+ * driver_svm_unlock();
+ * return err;
+ * }
+ *
+ * 2) Garbage Collector.
+ *
+ * void __driver_garbage_collector(struct drm_gpusvm *gpusvm,
+ * struct drm_gpusvm_range
*range)
+ * {
+ * assert_driver_svm_locked(gpusvm);
+ *
+ * // Partial unmap, migrate any remaining device
memory pages back to RAM
+ * if (range->flags.partial_unmap)
+ * drm_gpusvm_range_evict(gpusvm, range);
+ *
+ * driver_unbind_range(range);
+ * drm_gpusvm_range_remove(gpusvm, range);
+ * }
+ *
+ * void driver_garbage_collector(struct drm_gpusvm *gpusvm)
+ * {
+ * assert_driver_svm_locked(gpusvm);
+ *
+ * for_each_range_in_garbage_collector(gpusvm, range)
+ * __driver_garbage_collector(gpusvm, range);
+ * }
+ *
+ * 3) Invalidation driver vfunc.
+ *
+ * void driver_invalidation(struct drm_gpusvm *gpusvm,
+ * struct drm_gpusvm_notifier
*notifier,
+ * const struct mmu_notifier_range
*mmu_range)
+ * {
+ * struct drm_gpusvm_ctx ctx = { .in_notifier = true,
};
+ * struct drm_gpusvm_range *range = NULL;
+ *
+ * driver_invalidate_device_tlb(gpusvm, mmu_range-

Quoted 1 line

start, mmu_range->end);

+ *
+ * drm_gpusvm_for_each_range(range, notifier,
mmu_range->start,
+ *   mmu_range->end) {
+ * drm_gpusvm_range_unmap_pages(gpusvm, range,
&ctx);
+ *
+ * if (mmu_range->event != MMU_NOTIFY_UNMAP)
+ * continue;
+ *
+ * drm_gpusvm_range_set_unmapped(range,
mmu_range);
+ * driver_garbage_collector_add(gpusvm, range);
+ * }
+ * }
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+
+#define DRM_GPUSVM_RANGE_START(_range)	((_range)->va.start)
+#define DRM_GPUSVM_RANGE_END(_range)	((_range)->va.end - 1)
+INTERVAL_TREE_DEFINE(struct drm_gpusvm_range, rb.node, u64,

rb.__subtree_last,

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+		     DRM_GPUSVM_RANGE_START, DRM_GPUSVM_RANGE_END,
+		     static __maybe_unused, range);
+
+#define DRM_GPUSVM_NOTIFIER_START(_notifier)	((_notifier)-
Quoted 1 line

interval.start)

+#define DRM_GPUSVM_NOTIFIER_END(_notifier) ((_notifier)-

Quoted 1 line

interval.end - 1)

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+INTERVAL_TREE_DEFINE(struct drm_gpusvm_notifier, rb.node, u64,
+		     rb.__subtree_last, DRM_GPUSVM_NOTIFIER_START,
+		     DRM_GPUSVM_NOTIFIER_END, static __maybe_unused,
Quoted 1 line

notifier);

Did you look at removing these instantiations after the RFC comments, and instead embed a struct interval_tree_node?

Perhaps the notifier tree could use a maple tree?

Quoted 17 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+
+/**
+ * npages_in_range() - Calculate the number of pages in a given

range
+ * @start__: The start address of the range
+ * @end__: The end address of the range
+ *
+ * This macro calculates the number of pages in a given memory range,
+ * specified by the start and end addresses. It divides the difference
+ * between the end and start addresses by the page size (PAGE_SIZE) to
+ * determine the number of pages in the range.
+ *
+ * Return: The number of pages in the specified range.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c line 280

Quoted 2 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +2 -0
+#define npages_in_range(start__, end__)	\
+	(((end__) - (start__)) >> PAGE_SHIFT)

Could use a static function?

Quoted 20 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+
+/**
+ * struct drm_gpusvm_zdd - GPU SVM zone device data
+ *
+ * @refcount: Reference count for the zdd
+ * @destroy_work: Work structure for asynchronous zdd destruction
+ * @devmem_allocation: device memory allocation
+ * @device_private_page_owner: Device private pages owner
+ *
+ * This structure serves as a generic wrapper installed in
+ * page->zone_device_data. It provides infrastructure for looking up

a device
+ * memory allocation upon CPU page fault and asynchronously releasing device
+ * memory once the CPU has no page references. Asynchronous release is useful
+ * because CPU page references can be dropped in IRQ contexts, while releasing
+ * device memory likely requires sleeping locks.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c line 302

Quoted 6 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+struct drm_gpusvm_zdd {
+	struct kref refcount;
+	struct work_struct destroy_work;
+	struct drm_gpusvm_devmem *devmem_allocation;
+	void *device_private_page_owner;
+};

I think the zdd and migration helpers should be moved to drm_pagemap.c We should consider looking at that once patches for drm_pagemap functionality are posted.

TBC
/Thomas

author maintainer
#35 lore raw

Re: [PATCH v2 05/29] drm/gpusvm: Add support for GPU Shared Virtual Memory

Quoted 179 lines · Thomas Hellström
Quoted 178 lines · Matthew Brost

This patch introduces support for GPU Shared Virtual Memory (SVM) in the
Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.

The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also
provides mechanisms for allocating, deallocating, and migrating memory
regions between system RAM and GPU VRAM.

This is largely inspired by GPUVM.

v2:
 - Take order into account in check pages
 - Clear range->pages in get pages error
 - Drop setting dirty or accessed bit in get pages (Vetter)
 - Remove mmap assert for cpu faults
 - Drop mmap write lock abuse (Vetter, Christian)
 - Decouple zdd from range (Vetter, Oak)
 - Add drm_gpusvm_range_evict, make it work with coherent pages
 - Export drm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
 - mmget/put in drm_gpusvm_evict_to_sram
 - Drop range->vram_alloation variable
 - Don't return in drm_gpusvm_evict_to_sram until all pages detached
 - Don't warn on mixing sram and device pages
 - Update kernel doc
 - Add coherent page support to get pages
 - Use DMA_FROM_DEVICE rather than DMA_BIDIRECTIONAL
 - Add struct drm_gpusvm_vram and ops (Thomas)
 - Update the range's seqno if the range is valid (Thomas)
 - Remove the is_unmapped check before hmm_range_fault (Thomas)
 - Use drm_pagemap (Thomas)
 - Drop kfree_mapping (Thomas)
 - dma mapp pages under notifier lock (Thomas)
 - Remove ctx.prefault
 - Remove ctx.mmap_locked
 - Add ctx.check_pages
 - s/vram/devmem (Thomas)

---
 drivers/gpu/drm/xe/Makefile     |    3 +-
 drivers/gpu/drm/xe/drm_gpusvm.c | 2074
+++++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/drm_gpusvm.h |  447 +++++++

 3 files changed, 2523 insertions(+), 1 deletion(-)

 create mode 100644 drivers/gpu/drm/xe/drm_gpusvm.c create mode 100644 drivers/gpu/drm/xe/drm_gpusvm.h

drivers/gpu/drm/xe/Makefile +0 -0
index da80c29aa363..8d991d4a92a5 100644
@@ -25,7 +25,8 @@ $(obj)/generated/%_wa_oob.c
 $(obj)/generated/%_wa_oob.h: $(obj)/xe_gen_wa_oob \

 # core driver code
 
drivers/gpu/drm/xe/Makefile +2 -1
-xe-y += xe_bb.o \
+xe-y += drm_gpusvm.o \
+	xe_bb.o \

  xe_bo.o \
  xe_bo_evict.o \
  xe_devcoredump.o \

drivers/gpu/drm/xe/drm_gpusvm.c new file +26 -0
new file mode 100644
index 000000000000..1ff104d2b42c
@@ -0,0 +1,2074 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2024 Intel Corporation
+ *
+ * Authors:
+ *     Matthew Brost <[email protected]>
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/interval_tree_generic.h>
+#include <linux/hmm.h>
+#include <linux/memremap.h>
+#include <linux/migrate.h>
+#include <linux/mm_types.h>
+#include <linux/pagemap.h>
+#include <linux/slab.h>
+
+#include <drm/drm_device.h>
+#include "drm/drm_print.h"
+#include "drm_gpusvm.h"
+#include "drm_pagemap.h"
+
+/**
+ * DOC: Overview
+ *
+ * GPU Shared Virtual Memory (GPU SVM) layer for the Direct

Rendering Manager (DRM)
+ *
+ * The GPU SVM layer is a component of the DRM framework designed to manage shared
+ * virtual memory between the CPU and GPU. It enables efficient data exchange and
+ * processing for GPU-accelerated applications by allowing memory sharing and
+ * synchronization between the CPU's and GPU's virtual address spaces.
+ *
+ * Key GPU SVM Components:
+ * - Notifiers: Notifiers: Used for tracking memory intervals and notifying the
+ * GPU of changes, notifiers are sized based on a GPU
SVM
+ * initialization parameter, with a recommendation of
512M or
+ * larger. They maintain a Red-BlacK tree and a list of
ranges that
+ * fall within the notifier interval. Notifiers are
tracked within
+ * a GPU SVM Red-BlacK tree and list and are
dynamically inserted
+ * or removed as ranges within the interval are created
or
+ * destroyed.
+ * - Ranges: Represent memory ranges mapped in a DRM device and managed
+ *      by GPU SVM. They are sized based on an array of chunk
sizes, which
+ *      is a GPU SVM initialization parameter, and the CPU
address space.
+ *      Upon GPU fault, the largest aligned chunk that fits
within the
+ *      faulting CPU address space is chosen for the range
size. Ranges are
+ *      expected to be dynamically allocated on GPU fault and
removed on an
+ *      MMU notifier UNMAP event. As mentioned above, ranges
are tracked in
+ *      a notifier's Red-Black tree.
+ * - Operations: Define the interface for driver-specific GPU SVM operations
+ *               such as range allocation, notifier allocation, and
+ *               invalidations.
+ * - Device Memory Allocations: Embedded structure containing enough information
+ *                              for GPU SVM to migrate to / from device memory.
+ * - Device Memory Operations: Define the interface for driver- specific device
+ *                             memory operations release memory, populate pfns,
+ *                             and copy to / from device memory.
+ *
+ * This layer provides interfaces for allocating, mapping, migrating, and
+ * releasing memory ranges between the CPU and GPU. It handles all core memory
+ * management interactions (DMA mapping, HMM, and migration) and provides
+ * driver-specific virtual functions (vfuncs). This infrastructure is sufficient
+ * to build the expected driver components for an SVM implementation as detailed
+ * below.
+ *
+ * Expected Driver Components:
+ * - GPU page fault handler: Used to create ranges and notifiers based on the

Quoted 9 lines
Quoted 4 lines

+ *      fault address, optionally migrate the
range to
+ *      device memory, and create GPU bindings.
+ * - Garbage collector: Used to destroy GPU bindings for ranges.

Perhaps "clean up GPU bindings for ranges" to differentiate from unmapping GPU bindings for ranges which needs to be done in the notifier callback?

Let make this more clear.

Quoted 9 lines
Quoted 6 lines

Ranges are
+ * expected to be added to the garbage
collector upon
+ * MMU_NOTIFY_UNMAP event.
+ */
+

- Notifier callback, to unmap GPU bindings for ranges.

Will add something.

Quoted 14 lines
Quoted 5 lines

+/**
+ * DOC: Locking
+ *
+ * GPU SVM handles locking for core MM interactions, i.e., it locks/unlocks the

Quoted 8 lines

+ * mmap lock as needed.
+ *
+ * GPU SVM introduces a global notifier lock, which safeguards the notifier's
+ * range RB tree and list, as well as the range's DMA mappings and sequence
+ * number. GPU SVM manages all necessary locking and unlocking operations,

Quoted 6 lines

How difficult would it be to make this per-notifier?
One of the design comments we got from Jason was to prioritize avoiding core slowdowns and any fault processing might block an unrelated notifier during dma mapping and page-table commit.
I think this should at least be considered as a follow-up.

I don't think it is particular hard for SVM but if userptr is built on top of this it gets tricky. The reason being for userptr we can have an array of binds with multiple notifiers so taking multiple notifier locks will confuse lockdep or worse we could deadlock.

I'd say let keep it as is, see what the userptr rework looks like and then adjust based on that.

Quoted 8 lines
Quoted 3 lines

+ * except for the recheck of the range's sequence number
+ * (mmu_interval_read_retry) when the driver is committing GPU bindings. This

Perhaps add a discussion on valid pages rather than valid sequence number, since the sequence number might be bumped even if pages stay valid for the range, as the sequence number spans the whole notifier.

Yes. Good idea.

Quoted 18 lines
Quoted 9 lines

+ * lock corresponds to the 'driver->update' lock mentioned in the HMM
+ * documentation (TODO: Link). Future revisions may transition from a GPU SVM
+ * global lock to a per-notifier lock if finer-grained locking is deemed
+ * necessary.
+ *
+ * In addition to the locking mentioned above, the driver should

Quoted 8 lines

implement a
+ * lock to safeguard core GPU SVM function calls that modify state, such as
+ * drm_gpusvm_range_find_or_insert and drm_gpusvm_range_remove. Alternatively,
+ * these core functions can be called within a single kernel thread, for
+ * instance, using an ordered work queue.

Quoted 3 lines

I don't think not we should encourage the use of ordered workqueues to protect data / state? Locks should really be the preferred mechanism?

Let me drop this comment.

Quoted 10 lines
Quoted 5 lines

This lock is denoted as
+ * 'driver_svm_lock' in code examples. Finer grained driver side locking should
+ * also be possible for concurrent GPU fault processing within a single GPU SVM.

GPUVM (or rather the gem part of GPUVM that resides in drm_gem.h) allows for the driver to register a lock map which, if present, is used in the code to assert locks are correctly taken.

So something like drm_gem_gpuva_set_lock? Sure. I think really the only functions which need this assert are the range insert and removal functions though IIRC from my fined grained locking (concurrent GPU fault processing) prototype. I guess I could just add it everywhere in this version and adjust when we land finer grained locking.

Quoted 25 lines

In the xe example, if we're using the vm lock to protect, then we could register that and thoroughly annotate the gpusvm code with lockdep asserts. That will probably help making the code a lot easier to maintain moving forward.

Quoted 13 lines

+ */
+GPUVM (or rGPUVM (or rather the gem part of GPUVM that resides in drm_gem.h) allows for the driver to register a lock map which, if present, is used in the code to assert locks are correctly taken.

In the xe example, if we're using the vm lock to protect, then we could register that and thoroughly annotate the gpusvm code with lockdep asserts. That will probably help making the code a lot easier to maintain moving forward.GPUVM (or rather the gem part of GPUVM that resides in drm_gem.h) allows for the driver to register a lock map which, if present, is used in the code to assert locks are correctly taken.

Quoted 6 lines

In the xe example, if we're using the vm lock to protect, then we could register that and thoroughly annotate the gpusvm code with lockdep asserts. That will probably help making the code a lot easier to maintain moving forward.ather the gem part of GPUVM that resides in drm_gem.h) allows for the driver to register a lock map which, if present, is used in the code to assert locks are correctly taken.

Quoted 4 lines

In the xe example, if we're using the vm lock to protect, then we could register that and thoroughly annotate the gpusvm code with lockdep asserts. That will probably help making the code a lot easier to maintain moving forward.

See above, agree.

Quoted 5 lines
Quoted 2 lines

+/**
+ * DOC: Migrataion

s/Migrataion/Migration/

Opps. Will fix.

Quoted 205 lines
Quoted 205 lines

+ *
+ * The migration support is quite simple, allowing migration between RAM and
+ * device memory at the range granularity. For example, GPU SVM currently does not
+ * support mixing RAM and device memory pages within a range. This means that upon GPU
+ * fault, the entire range can be migrated to device memory, and upon CPU fault, the
+ * entire range is migrated to RAM. Mixed RAM and device memory storage within a range
+ * could be added in the future if required.
+ *
+ * The reasoning for only supporting range granularity is as
follows: it
+ * simplifies the implementation, and range sizes are driver-defined and should
+ * be relatively small.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Partial Unmapping of Ranges
+ *
+ * Partial unmapping of ranges (e.g., 1M out of 2M is unmapped by

CPU resulting
+ * in MMU_NOTIFY_UNMAP event) presents several challenges, with the main one
+ * being that a subset of the range still has CPU and GPU mappings. If the
+ * backing store for the range is in device memory, a subset of the backing store has
+ * references. One option would be to split the range and device memory backing store,
+ * but the implementation for this would be quite complicated. Given that
+ * partial unmappings are rare and driver-defined range sizes are relatively
+ * small, GPU SVM does not support splitting of ranges.
+ *
+ * With no support for range splitting, upon partial unmapping of a range, the
+ * driver is expected to invalidate and destroy the entire range. If the range
+ * has device memory as its backing, the driver is also expected to migrate any
+ * remaining pages back to RAM.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Examples
+ *
+ * This section provides two examples of how to build the expected

driver
+ * components: the GPU page fault handler and the garbage collector. A third
+ * example demonstrates a sample invalidation driver vfunc.
+ *
+ * The generic code provided does not include logic for complex migration
+ * policies, optimized invalidations, fined grained driver locking, or other
+ * potentially required driver locking (e.g., DMA-resv locks).
+ *
+ * 1) GPU page fault handler
+ *
+ * int driver_bind_range(struct drm_gpusvm *gpusvm, struct
drm_gpusvm_range *range)
+ * {
+ * int err = 0;
+ *
+ * driver_alloc_and_setup_memory_for_bind(gpusvm,
range);
+ *
+ * drm_gpusvm_notifier_lock(gpusvm);
+ * if (drm_gpusvm_range_pages_valid(range))
+ * driver_commit_bind(gpusvm, range);
+ * else
+ * err = -EAGAIN;
+ * drm_gpusvm_notifier_unlock(gpusvm);
+ *
+ * return err;
+ * }
+ *
+ * int driver_gpu_fault(struct drm_gpusvm *gpusvm, u64
fault_addr,
+ *      u64 gpuva_start, u64 gpuva_end)
+ * {
+ * struct drm_gpusvm_ctx ctx = {};
+ * int err;
+ *
+ * driver_svm_lock();
+ * retry:
+ * // Always process UNMAPs first so view of GPU SVM
ranges is current
+ * driver_garbage_collector(gpusvm);
+ *
+ * range = drm_gpusvm_range_find_or_insert(gpusvm,
fault_addr,
+ * gpuva_start,
gpuva_end,
+ *         &ctx);
+ * if (IS_ERR(range)) {
+ * err = PTR_ERR(range);
+ * goto unlock;
+ * }
+ *
+ * if (driver_migration_policy(range)) {
+ * devmem = driver_alloc_devmem();
+ * err = drm_gpusvm_migrate_to_devmem(gpusvm,
range,
+ *
devmem_allocation,
+ *    &ctx);
+ * if (err) // CPU mappings may have
changed
+ * goto retry;
+ * }
+ *
+ * err = drm_gpusvm_range_get_pages(gpusvm, range,
&ctx);
+ * if (err == -EOPNOTSUPP || err == -EFAULT || err == -
EPERM) { // CPU mappings changed
+ * if (err == -EOPNOTSUPP)
+ * drm_gpusvm_range_evict(gpusvm,
range);
+ * goto retry;
+ * } else if (err) {
+ * goto unlock;
+ * }
+ *
+ * err = driver_bind_range(gpusvm, range);
+ * if (err == -EAGAIN) // CPU mappings changed
+ * goto retry
+ *
+ * unlock:
+ * driver_svm_unlock();
+ * return err;
+ * }
+ *
+ * 2) Garbage Collector.
+ *
+ * void __driver_garbage_collector(struct drm_gpusvm *gpusvm,
+ * struct drm_gpusvm_range
*range)
+ * {
+ * assert_driver_svm_locked(gpusvm);
+ *
+ * // Partial unmap, migrate any remaining device
memory pages back to RAM
+ * if (range->flags.partial_unmap)
+ * drm_gpusvm_range_evict(gpusvm, range);
+ *
+ * driver_unbind_range(range);
+ * drm_gpusvm_range_remove(gpusvm, range);
+ * }
+ *
+ * void driver_garbage_collector(struct drm_gpusvm *gpusvm)
+ * {
+ * assert_driver_svm_locked(gpusvm);
+ *
+ * for_each_range_in_garbage_collector(gpusvm, range)
+ * __driver_garbage_collector(gpusvm, range);
+ * }
+ *
+ * 3) Invalidation driver vfunc.
+ *
+ * void driver_invalidation(struct drm_gpusvm *gpusvm,
+ * struct drm_gpusvm_notifier
*notifier,
+ * const struct mmu_notifier_range
*mmu_range)
+ * {
+ * struct drm_gpusvm_ctx ctx = { .in_notifier = true,
};
+ * struct drm_gpusvm_range *range = NULL;
+ *
+ * driver_invalidate_device_tlb(gpusvm, mmu_range-

Quoted 1 line

start, mmu_range->end);

+ *
+ * drm_gpusvm_for_each_range(range, notifier,
mmu_range->start,
+ *   mmu_range->end) {
+ * drm_gpusvm_range_unmap_pages(gpusvm, range,
&ctx);
+ *
+ * if (mmu_range->event != MMU_NOTIFY_UNMAP)
+ * continue;
+ *
+ * drm_gpusvm_range_set_unmapped(range,
mmu_range);
+ * driver_garbage_collector_add(gpusvm, range);
+ * }
+ * }
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+
+#define DRM_GPUSVM_RANGE_START(_range)	((_range)->va.start)
+#define DRM_GPUSVM_RANGE_END(_range)	((_range)->va.end - 1)
+INTERVAL_TREE_DEFINE(struct drm_gpusvm_range, rb.node, u64,

rb.__subtree_last,

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+		     DRM_GPUSVM_RANGE_START, DRM_GPUSVM_RANGE_END,
+		     static __maybe_unused, range);
+
+#define DRM_GPUSVM_NOTIFIER_START(_notifier)	((_notifier)-
Quoted 1 line

interval.start)

+#define DRM_GPUSVM_NOTIFIER_END(_notifier) ((_notifier)-

Quoted 9 lines
Quoted 5 lines
Quoted 1 line

interval.end - 1)

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+INTERVAL_TREE_DEFINE(struct drm_gpusvm_notifier, rb.node, u64,
+		     rb.__subtree_last, DRM_GPUSVM_NOTIFIER_START,
+		     DRM_GPUSVM_NOTIFIER_END, static __maybe_unused,

notifier);

Did you look at removing these instantiations after the RFC comments, and instead embed a struct interval_tree_node?

Both gpu vm an xe_range_fence define an interval tree like this so figured it was fine but also some of this was lazyness on my part.

Quoted 2 lines

Perhaps the notifier tree could use a maple tree?

Let me in earnest follow up on using a maple tree. My understanding is a maple tree is designed for exactly this (VA tracking) and the only reason GPU VM doesn't use it is because it has memory allocations which break the dma-fencing rules. I think there is a version on GPU VM out there with a maple tree so I don't have to think to hard about this.

Quoted 13 lines
Quoted 13 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+
+/**
+ * npages_in_range() - Calculate the number of pages in a given

range
+ * @start__: The start address of the range
+ * @end__: The end address of the range
+ *
+ * This macro calculates the number of pages in a given memory range,
+ * specified by the start and end addresses. It divides the difference
+ * between the end and start addresses by the page size (PAGE_SIZE) to

Quoted 9 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 280

Quoted 6 lines

+ * determine the number of pages in the range.
+ *
+ * Return: The number of pages in the specified range.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +2 -0
+#define npages_in_range(start__, end__)	\
+	(((end__) - (start__)) >> PAGE_SHIFT)

Could use a static function?

See my other reply [1]. Will changes macro -> functions where is makes sense.

[1] https://patchwork.freedesktop.org/patch/619809/?series=137870&rev=2#comment_1133611

Quoted 22 lines
Quoted 22 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+
+/**
+ * struct drm_gpusvm_zdd - GPU SVM zone device data
+ *
+ * @refcount: Reference count for the zdd
+ * @destroy_work: Work structure for asynchronous zdd destruction
+ * @devmem_allocation: device memory allocation
+ * @device_private_page_owner: Device private pages owner
+ *
+ * This structure serves as a generic wrapper installed in
+ * page->zone_device_data. It provides infrastructure for looking up

a device
+ * memory allocation upon CPU page fault and asynchronously releasing device
+ * memory once the CPU has no page references. Asynchronous release is useful
+ * because CPU page references can be dropped in IRQ contexts, while releasing
+ * device memory likely requires sleeping locks.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +2 -0
+struct drm_gpusvm_zdd {
+	struct kref refcount;
Quoted 9 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 302

Quoted 4 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+	struct work_struct destroy_work;
+	struct drm_gpusvm_devmem *devmem_allocation;
+	void *device_private_page_owner;
+};

I think the zdd and migration helpers should be moved to drm_pagemap.c We should consider looking at that once patches for drm_pagemap functionality are posted.

We have another thread discussing this. Let's continue this there.

Matt

Quoted 4 lines

TBC
/Thomas

#36 lore raw

Re: [PATCH v2 05/29] drm/gpusvm: Add support for GPU Shared Virtual Memory

In reply to Matthew Brost

On Tue, 2024-10-15 at 20:24 -0700, Matthew Brost wrote:

Continued review.

Quoted 29 lines

+/**
+ * struct drm_gpusvm_zdd - GPU SVM zone device data
+ *
+ * @refcount: Reference count for the zdd
+ * @destroy_work: Work structure for asynchronous zdd destruction
+ * @devmem_allocation: device memory allocation
+ * @device_private_page_owner: Device private pages owner
+ *
+ * This structure serves as a generic wrapper installed in
+ * page->zone_device_data. It provides infrastructure for looking up a device
+ * memory allocation upon CPU page fault and asynchronously releasing device
+ * memory once the CPU has no page references. Asynchronous release is useful
+ * because CPU page references can be dropped in IRQ contexts, while releasing
+ * device memory likely requires sleeping locks.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+struct drm_gpusvm_zdd {
+	struct kref refcount;
+	struct work_struct destroy_work;
+	struct drm_gpusvm_devmem *devmem_allocation;
+	void *device_private_page_owner;
+};
+
+/**
+ * drm_gpusvm_zdd_destroy_work_func - Work function for destroying a
Quoted 1 line

zdd

NIT: Even if the above kerneldoc format works, I keep trying to enforce using () after function names and function-like macros, like described
here: https://docs.kernel.org/doc-guide/kernel-doc.html Could we update? Also that doc calls for using "Return:" instead of "Returns:".

Quoted 89 lines

+ * @w: Pointer to the work_struct
+ *
+ * This function releases device memory, puts GPU SVM range, and frees zdd.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_zdd_destroy_work_func(struct work_struct *w)
+{
+	struct drm_gpusvm_zdd *zdd =
+		container_of(w, struct drm_gpusvm_zdd,

destroy_work);

(quoted diff) +1 -0
+	const struct drm_gpusvm_devmem_ops *ops = zdd-
Quoted 1 line

devmem_allocation ?

drivers/gpu/drm/xe/drm_gpusvm.c new file +12 -0
+		zdd->devmem_allocation->ops : NULL;
+
+	if (zdd->devmem_allocation && ops->devmem_release)
+		ops->devmem_release(zdd->devmem_allocation);
+	kfree(zdd);
+}
+
+/**
+ * drm_gpusvm_zdd_alloc - Allocate a zdd structure.
+ * @device_private_page_owner: Device private pages owner
+ *
+ * This function allocates and initializes a new zdd structure. It

sets up the
+ * reference count and initializes the destroy work.
+ *
+ * Returns:
+ * Pointer to the allocated zdd on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+static struct drm_gpusvm_zdd *
+drm_gpusvm_zdd_alloc(void *device_private_page_owner)
+{
+	struct drm_gpusvm_zdd *zdd;
+
+	zdd = kmalloc(sizeof(*zdd), GFP_KERNEL);
+	if (!zdd)
+		return NULL;
+
+	kref_init(&zdd->refcount);
+	INIT_WORK(&zdd->destroy_work,

drm_gpusvm_zdd_destroy_work_func);

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+	zdd->devmem_allocation = NULL;
+	zdd->device_private_page_owner = device_private_page_owner;
+
+	return zdd;
+}
+
+/**
+ * drm_gpusvm_zdd_get - Get a reference to a zdd structure.
+ * @zdd: Pointer to the zdd structure.
+ *
+ * This function increments the reference count of the provided zdd

structure.
+ *
+ * Returns: Pointer to the zdd structure.
+ */
+static struct drm_gpusvm_zdd *drm_gpusvm_zdd_get(struct drm_gpusvm_zdd *zdd)

drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+{
+	kref_get(&zdd->refcount);
+	return zdd;
+}
+
+/**
+ * drm_gpusvm_zdd_destroy - Destroy a zdd structure.
+ * @ref: Pointer to the reference count structure.
+ *
+ * This function queues the destroy_work of the zdd for asynchronous

destruction.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+static void drm_gpusvm_zdd_destroy(struct kref *ref)
+{
+	struct drm_gpusvm_zdd *zdd =
+		container_of(ref, struct drm_gpusvm_zdd, refcount);
+
+	if (zdd->devmem_allocation)
+		WRITE_ONCE(zdd->devmem_allocation->detached, true);
+	schedule_work(&zdd->destroy_work);
+}
+
+/**
+ * drm_gpusvm_zdd_put - Put a zdd reference.
+ * @zdd: Pointer to the zdd structure.
+ *
+ * This function decrements the reference count of the provided zdd

structure
+ * and schedules its destruction if the count drops to zero.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c line 389

Quoted 4 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_zdd_put(struct drm_gpusvm_zdd *zdd)
+{
+	kref_put(&zdd->refcount, drm_gpusvm_zdd_destroy);
+}

As mentioned earlier, I think the above drm_gpusvm_zdd functions should move to drm_pagemap.c. I don't think they are used in drm_gpusvm other than to, at get_pages time, ensure all device private pages are from the same pagemap?

Quoted 57 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+
+/**
+ * drm_gpusvm_range_find - Find GPU SVM range from GPU SVM notifier
+ * @notifier: Pointer to the GPU SVM notifier structure.
+ * @start: Start address of the range
+ * @end: End address of the range
+ *
+ * Return: A pointer to the drm_gpusvm_range if found or NULL
+ */
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, u64

start, u64 end)

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+{
+	return range_iter_first(&notifier->root, start, end - 1);
+}
+
+/**
+ * drm_gpusvm_for_each_range_safe - Safely iterate over GPU SVM

ranges in a notifier
+ * @range__: Iterator variable for the ranges
+ * @next__: Iterator variable for the ranges temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier while
+ * removing ranges from it.
+ */
+#define drm_gpusvm_for_each_range_safe(range__, next__, notifier__,
start__, end__) \

(quoted diff) +1 -0
+	for ((range__) = drm_gpusvm_range_find((notifier__),

(start__), (end__)), \

(quoted diff) +1 -0
+	     (next__) =

__drm_gpusvm_range_next(range__); \

(quoted diff) +1 -0
+	     (range__) && (range__->va.start <

(end__)); \

(quoted diff) +1 -0
+	     (range__) = (next__), (next__) =

__drm_gpusvm_range_next(range__))

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+
+/**
+ * __drm_gpusvm_notifier_next - get the next drm_gpusvm_notifier in

the list
+ * @notifier: a pointer to the current drm_gpusvm_notifier
+ *
+ * Return: A pointer to the next drm_gpusvm_notifier if available, or NULL if
+ *         the current notifier is the last one or if the input notifier is
+ *         NULL.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+static struct drm_gpusvm_notifier *
+__drm_gpusvm_notifier_next(struct drm_gpusvm_notifier *notifier)
+{
+	if (notifier && !list_is_last(&notifier->rb.entry,
+				      &notifier->gpusvm-
Quoted 1 line

notifier_list))

drivers/gpu/drm/xe/drm_gpusvm.c line 435

Quoted 1 line
drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+		return list_next_entry(notifier, rb.entry);

Why aren't we using notifier_iter_next() here? Then the linked list could be skipped.

Quoted 20 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+
+	return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_notifier - Iterate over GPU SVM notifiers in

a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm.
+ */
+#define drm_gpusvm_for_each_notifier(notifier__, gpusvm__, start__,
end__) \

(quoted diff) +1 -0
+	for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,

(start__), (end__) - 1); \

(quoted diff) +1 -0
+	     (notifier__) && (notifier__->interval.start <

(end__)); \

Review

Quoted 2 lines
(quoted diff) +2 -0
+	     (notifier__) = __drm_gpusvm_notifier_next(notifier__))
+

Looks like end__ is not honored except for the first iteration. Relates to the above question.

Quoted 22 lines

+/**
+ * drm_gpusvm_for_each_notifier_safe - Safely iterate over GPU SVM notifiers in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @next__: Iterator variable for the notifiers temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm while
+ * removing notifiers from it.
+ */
+#define drm_gpusvm_for_each_notifier_safe(notifier__, next__,
gpusvm__, start__, end__) \

(quoted diff) +1 -0
+	for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,

(start__), (end__) - 1), \

(quoted diff) +1 -0
+	     (next__) =

__drm_gpusvm_notifier_next(notifier__); \

(quoted diff) +1 -0
+	     (notifier__) && (notifier__->interval.start <

(end__)); \

(quoted diff) +1 -0
+	     (notifier__) = (next__), (next__) =
Quoted 1 line

__drm_gpusvm_notifier_next(notifier__))

Same here.

Quoted 129 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+
+/**
+ * drm_gpusvm_notifier_invalidate - Invalidate a GPU SVM notifier.
+ * @mni: Pointer to the mmu_interval_notifier structure.
+ * @mmu_range: Pointer to the mmu_notifier_range structure.
+ * @cur_seq: Current sequence number.
+ *
+ * This function serves as a generic MMU notifier for GPU SVM. It

sets the MMU
+ * notifier sequence number and calls the driver invalidate vfunc under
+ * gpusvm->notifier_lock.
+ *
+ * Returns:
+ * true if the operation succeeds, false otherwise.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+static bool
+drm_gpusvm_notifier_invalidate(struct mmu_interval_notifier *mni,
+			       const struct mmu_notifier_range

*mmu_range,

drivers/gpu/drm/xe/drm_gpusvm.c new file +19 -0
+			       unsigned long cur_seq)
+{
+	struct drm_gpusvm_notifier *notifier =
+		container_of(mni, typeof(*notifier), notifier);
+	struct drm_gpusvm *gpusvm = notifier->gpusvm;
+
+	if (!mmu_notifier_range_blockable(mmu_range))
+		return false;
+
+	down_write(&gpusvm->notifier_lock);
+	mmu_interval_set_seq(mni, cur_seq);
+	gpusvm->ops->invalidate(gpusvm, notifier, mmu_range);
+	up_write(&gpusvm->notifier_lock);
+
+	return true;
+}
+
+/**
+ * drm_gpusvm_notifier_ops - MMU interval notifier operations for

GPU SVM
+ */
+static const struct mmu_interval_notifier_ops
drm_gpusvm_notifier_ops = {

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+	.invalidate = drm_gpusvm_notifier_invalidate,
+};
+
+/**
+ * drm_gpusvm_init - Initialize the GPU SVM.
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @name: Name of the GPU SVM.
+ * @drm: Pointer to the DRM device structure.
+ * @mm: Pointer to the mm_struct for the address space.
+ * @device_private_page_owner: Device private pages owner.
+ * @mm_start: Start address of GPU SVM.
+ * @mm_range: Range of the GPU SVM.
+ * @notifier_size: Size of individual notifiers.
+ * @ops: Pointer to the operations structure for GPU SVM.
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range

allocation.
+ *               Entries should be powers of 2 in descending order with last
+ *               entry being SZ_4K.
+ * @num_chunks: Number of chunks.
+ *
+ * This function initializes the GPU SVM.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void

*device_private_page_owner,

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+		    u64 mm_start, u64 mm_range, u64 notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const u64 *chunk_sizes, int num_chunks)
+{
+	if (!ops->invalidate || !num_chunks)
+		return -EINVAL;
+
+	gpusvm->name = name;
+	gpusvm->drm = drm;
+	gpusvm->mm = mm;
+	gpusvm->device_private_page_owner =

device_private_page_owner;

drivers/gpu/drm/xe/drm_gpusvm.c new file +26 -0
+	gpusvm->mm_start = mm_start;
+	gpusvm->mm_range = mm_range;
+	gpusvm->notifier_size = notifier_size;
+	gpusvm->ops = ops;
+	gpusvm->chunk_sizes = chunk_sizes;
+	gpusvm->num_chunks = num_chunks;
+
+	mmgrab(mm);
+	gpusvm->root = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&gpusvm->notifier_list);
+
+	init_rwsem(&gpusvm->notifier_lock);
+
+	fs_reclaim_acquire(GFP_KERNEL);
+	might_lock(&gpusvm->notifier_lock);
+	fs_reclaim_release(GFP_KERNEL);
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_notifier_find - Find GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure
+ * @fault_addr__: Fault address
+ *
+ * This macro finds the GPU SVM notifier associated with the fault

address.
+ *
+ * Returns:
+ * Pointer to the GPU SVM notifier on success, NULL otherwise.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+#define drm_gpusvm_notifier_find(gpusvm__, fault_addr__)	\
+	notifier_iter_first(&(gpusvm__)->root, (fault_addr__),	\
+			    (fault_addr__ + 1))
+
+/**
+ * to_drm_gpusvm_notifier - retrieve the container struct for a

given rbtree node
+ * @node__: a pointer to the rbtree node embedded within a drm_gpusvm_notifier struct
+ *
+ * Return: A pointer to the containing drm_gpusvm_notifier structure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c line 588

Quoted 3 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+#define to_drm_gpusvm_notifier(__node)				\
+	container_of((__node), struct drm_gpusvm_notifier, rb.node)
+

There appears to be a number of function-like macros in the code, which look like they can be converted to functions. Linux prefers functions over macros when possible:

https://www.kernel.org/doc/html/v5.8/process/coding-style.html#macros-enums-and-rtl

Quoted 34 lines

+/**
+ * drm_gpusvm_notifier_insert - Insert GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function inserts the GPU SVM notifier into the GPU SVM RB tree and list.
+ */

(quoted diff) +2 -0
+static void drm_gpusvm_notifier_insert(struct drm_gpusvm *gpusvm,
+				       struct drm_gpusvm_notifier

*notifier)

drivers/gpu/drm/xe/drm_gpusvm.c new file +21 -0
+{
+	struct rb_node *node;
+	struct list_head *head;
+
+	notifier_insert(notifier, &gpusvm->root);
+
+	node = rb_prev(&notifier->rb.node);
+	if (node)
+		head = &(to_drm_gpusvm_notifier(node))->rb.entry;
+	else
+		head = &gpusvm->notifier_list;
+
+	list_add(&notifier->rb.entry, head);
+}
+
+/**
+ * drm_gpusvm_notifier_remove - Remove GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM tructure
+ * @notifier__: Pointer to the GPU SVM notifier structure
+ *
+ * This macro removes the GPU SVM notifier from the GPU SVM RB tree

and list.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c line 622

Quoted 3 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+#define drm_gpusvm_notifier_remove(gpusvm__, notifier__)	\
+	notifier_remove((notifier__), &(gpusvm__)->root);	\
+	list_del(&(notifier__)->rb.entry)

Unless this can be made a function, Pls use
do { } while (0)

Quoted 137 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+
+/**
+ * drm_gpusvm_fini - Finalize the GPU SVM.
+ * @gpusvm: Pointer to the GPU SVM structure.
+ *
+ * This function finalizes the GPU SVM by cleaning up any remaining

ranges and
+ * notifiers, and dropping a reference to struct MM.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm)
+{
+	struct drm_gpusvm_notifier *notifier, *next;
+
+	drm_gpusvm_for_each_notifier_safe(notifier, next, gpusvm, 0,

LONG_MAX) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+		struct drm_gpusvm_range *range, *__next;
+
+		/*
+		 * Remove notifier first to avoid racing with any

invalidation

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+		 */
+		mmu_interval_notifier_remove(&notifier->notifier);
+		notifier->flags.removed = true;
+
+		drm_gpusvm_for_each_range_safe(range, __next,

notifier, 0,

drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+					       LONG_MAX)
+			drm_gpusvm_range_remove(gpusvm, range);
+	}
+
+	mmdrop(gpusvm->mm);
+	WARN_ON(!RB_EMPTY_ROOT(&gpusvm->root.rb_root));
+}
+
+/**
+ * drm_gpusvm_notifier_alloc - Allocate GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ *
+ * This function allocates and initializes the GPU SVM notifier

structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM notifier on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+static struct drm_gpusvm_notifier *
+drm_gpusvm_notifier_alloc(struct drm_gpusvm *gpusvm, u64 fault_addr)
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	if (gpusvm->ops->notifier_alloc)
+		notifier = gpusvm->ops->notifier_alloc();
+	else
+		notifier = kzalloc(sizeof(*notifier), GFP_KERNEL);
+
+	if (!notifier)
+		return ERR_PTR(-ENOMEM);
+
+	notifier->gpusvm = gpusvm;
+	notifier->interval.start = ALIGN_DOWN(fault_addr, gpusvm-
Quoted 1 line

notifier_size);

(quoted diff) +1 -0
+	notifier->interval.end = ALIGN(fault_addr + 1, gpusvm-
Quoted 1 line

notifier_size);

drivers/gpu/drm/xe/drm_gpusvm.c new file +16 -0
+	INIT_LIST_HEAD(&notifier->rb.entry);
+	notifier->root = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&notifier->range_list);
+
+	return notifier;
+}
+
+/**
+ * drm_gpusvm_notifier_free - Free GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function frees the GPU SVM notifier structure.
+ */
+static void drm_gpusvm_notifier_free(struct drm_gpusvm *gpusvm,
+				     struct drm_gpusvm_notifier

*notifier)

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+{
+	WARN_ON(!RB_EMPTY_ROOT(&notifier->root.rb_root));
+
+	if (gpusvm->ops->notifier_free)
+		gpusvm->ops->notifier_free(notifier);
+	else
+		kfree(notifier);
+}
+
+/**
+ * to_drm_gpusvm_range - retrieve the container struct for a given

rbtree node
+ * @node__: a pointer to the rbtree node embedded within a drm_gpusvm_range struct
+ *
+ * Return: A pointer to the containing drm_gpusvm_range structure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+#define to_drm_gpusvm_range(node__)	\
+	container_of((node__), struct drm_gpusvm_range, rb.node)
+
+/**
+ * drm_gpusvm_range_insert - Insert GPU SVM range
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function inserts the GPU SVM range into the notifier RB tree

and list.
+ */
+static void drm_gpusvm_range_insert(struct drm_gpusvm_notifier *notifier,

drivers/gpu/drm/xe/drm_gpusvm.c new file +24 -0
+				    struct drm_gpusvm_range *range)
+{
+	struct rb_node *node;
+	struct list_head *head;
+
+	drm_gpusvm_notifier_lock(notifier->gpusvm);
+	range_insert(range, &notifier->root);
+
+	node = rb_prev(&range->rb.node);
+	if (node)
+		head = &(to_drm_gpusvm_range(node))->rb.entry;
+	else
+		head = &notifier->range_list;
+
+	list_add(&range->rb.entry, head);
+	drm_gpusvm_notifier_unlock(notifier->gpusvm);
+}
+
+/**
+ * __drm_gpusvm_range_remove - Remove GPU SVM range
+ * @notifier__: Pointer to the GPU SVM notifier structure
+ * @range__: Pointer to the GPU SVM range structure
+ *
+ * This macro removes the GPU SVM range from the notifier RB tree

and list.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c line 748

Quoted 3 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+#define __drm_gpusvm_range_remove(notifier__, range__)		\
+	range_remove((range__), &(notifier__)->root);		\
+	list_del(&(range__)->rb.entry)

Same thing as for the notifier rb tree. And do we need the linked list?

Quoted 201 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+
+/**
+ * drm_gpusvm_range_alloc - Allocate GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @fault_addr: Fault address
+ * @chunk_size: Chunk size
+ * @migrate_devmem: Flag indicating whether to migrate device memory
+ *
+ * This function allocates and initializes the GPU SVM range

structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM range on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+static struct drm_gpusvm_range *
+drm_gpusvm_range_alloc(struct drm_gpusvm *gpusvm,
+		       struct drm_gpusvm_notifier *notifier,
+		       u64 fault_addr, u64 chunk_size, bool

migrate_devmem)

drivers/gpu/drm/xe/drm_gpusvm.c new file +31 -0
+{
+	struct drm_gpusvm_range *range;
+
+	if (gpusvm->ops->range_alloc)
+		range = gpusvm->ops->range_alloc(gpusvm);
+	else
+		range = kzalloc(sizeof(*range), GFP_KERNEL);
+
+	if (!range)
+		return ERR_PTR(-ENOMEM);
+
+	kref_init(&range->refcount);
+	range->gpusvm = gpusvm;
+	range->notifier = notifier;
+	range->va.start = ALIGN_DOWN(fault_addr, chunk_size);
+	range->va.end = ALIGN(fault_addr + 1, chunk_size);
+	INIT_LIST_HEAD(&range->rb.entry);
+	range->notifier_seq = LONG_MAX;
+	range->flags.migrate_devmem = migrate_devmem ? 1 : 0;
+
+	return range;
+}
+
+/**
+ * drm_gpusvm_check_pages - Check pages
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @start: Start address
+ * @end: End address
+ *
+ * Check if pages between start and end have been faulted in on the

CPU. Use to
+ * prevent migration of pages without CPU backing store.
+ *
+ * Returns:
+ * True if pages have been faulted into CPU, False otherwise
+ */

(quoted diff) +2 -0
+static bool drm_gpusvm_check_pages(struct drm_gpusvm *gpusvm,
+				   struct drm_gpusvm_notifier

*notifier,

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+				   u64 start, u64 end)
+{
+	struct hmm_range hmm_range = {
+		.default_flags = 0,
+		.notifier = &notifier->notifier,
+		.start = start,
+		.end = end,
+		.dev_private_owner = gpusvm-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+	};
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+	unsigned long *pfns;
+	unsigned long npages = npages_in_range(start, end);
+	int err, i;
+
+	mmap_assert_locked(gpusvm->mm);
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL);
+	if (!pfns)
+		return false;
+
+	hmm_range.notifier_seq = mmu_interval_read_begin(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+	hmm_range.hmm_pfns = pfns;
+
+	while (true) {
+		err = hmm_range_fault(&hmm_range);
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq =

mmu_interval_read_begin(&notifier->notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +22 -0
+			continue;
+		}
+		break;
+	}
+	if (err)
+		goto err_free;
+
+	for (i = 0; i < npages;) {
+		if (!(pfns[i] & HMM_PFN_VALID)) {
+			err = -EFAULT;
+			goto err_free;
+		}
+		i += 0x1 << hmm_pfn_to_map_order(pfns[i]);
+	}
+
+err_free:
+	kvfree(pfns);
+	return err ? false : true;
+}
+
+/**
+ * drm_gpusvm_range_chunk_size - Determine chunk size for GPU SVM

range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @vas: Pointer to the virtual memory area structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @check_pages: Flag indicating whether to check pages
+ *
+ * This function determines the chunk size for the GPU SVM range based on the
+ * fault address, GPU SVM chunk sizes, existing GPU SVM ranges, and the virtual
+ * memory area boundaries.
+ *
+ * Returns:
+ * Chunk size on success, LONG_MAX on failure.
+ */

(quoted diff) +2 -0
+static u64 drm_gpusvm_range_chunk_size(struct drm_gpusvm *gpusvm,
+				       struct drm_gpusvm_notifier

*notifier,

(quoted diff) +2 -0
+				       struct vm_area_struct *vas,
+				       u64 fault_addr, u64

gpuva_start,

(quoted diff) +1 -0
+				       u64 gpuva_end, bool

check_pages)

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+{
+	u64 start, end;
+	int i = 0;
+
+retry:
+	for (; i < gpusvm->num_chunks; ++i) {
+		start = ALIGN_DOWN(fault_addr, gpusvm-
Quoted 1 line

chunk_sizes[i]);

drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+		end = ALIGN(fault_addr + 1, gpusvm->chunk_sizes[i]);
+
+		if (start >= vas->vm_start && end <= vas->vm_end &&
+		    start >= notifier->interval.start &&
+		    end <= notifier->interval.end &&
+		    start >= gpuva_start && end <= gpuva_end)
+			break;
+	}
+
+	if (i == gpusvm->num_chunks)
+		return LONG_MAX;
+
+	/*
+	 * If allocation more than page, ensure not to overlap with

existing

drivers/gpu/drm/xe/drm_gpusvm.c new file +13 -0
+	 * ranges.
+	 */
+	if (end - start != SZ_4K) {
+		struct drm_gpusvm_range *range;
+
+		range = drm_gpusvm_range_find(notifier, start, end);
+		if (range) {
+			++i;
+			goto retry;
+		}
+
+		/*
+		 * XXX: Only create range on pages CPU has faulted

in. Without

(quoted diff) +1 -0
+		 * this check, or prefault, on BMG

'xe_exec_system_allocator --r

(quoted diff) +1 -0
+		 * process-many-malloc' fails. In the failure case,

each process

(quoted diff) +1 -0
+		 * mallocs 16k but the CPU VMA is ~128k which

results in 64k SVM

(quoted diff) +1 -0
+		 * ranges. When migrating the SVM ranges, some

processes fail in

(quoted diff) +1 -0
+		 * drm_gpusvm_migrate_to_devmem with 'migrate.cpages

!= npages'

(quoted diff) +1 -0
+		 * and then upon drm_gpusvm_range_get_pages device

pages from

(quoted diff) +1 -0
+		 * other processes are collected + faulted in which

creates all

(quoted diff) +1 -0
+		 * sorts of problems. Unsure exactly how this

happening, also

(quoted diff) +1 -0
+		 * problem goes away if 'xe_exec_system_allocator --

r

(quoted diff) +1 -0
+		 * process-many-malloc' mallocs at least 64k at a

time.

Review

Quoted 1 line
(quoted diff) +1 -0
+		 */

Needs to be figured out. I think even in the system allocator case, if a user uses malloc() to allocate a GPU only buffer we'd need to support that?

Quoted 34 lines
(quoted diff) +2 -0
+		if (check_pages &&
+		    !drm_gpusvm_check_pages(gpusvm, notifier, start,

end)) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +17 -0
+			++i;
+			goto retry;
+		}
+	}
+
+	return end - start;
+}
+
+/**
+ * drm_gpusvm_range_find_or_insert - Find or insert GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @ctx: GPU SVM context
+ *
+ * This function finds or inserts a newly allocated a GPU SVM range

based on the
+ * fault address. Caller must hold a lock to protect range lookup and insertion.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range on success, ERR_PTR() on failure.
+ */

(quoted diff) +2 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, u64

fault_addr,

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+				u64 gpuva_start, u64 gpuva_end,
+				const struct drm_gpusvm_ctx *ctx)
+{
+	struct drm_gpusvm_notifier *notifier;

drivers/gpu/drm/xe/drm_gpusvm.c line 962

Quoted 10 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+	struct drm_gpusvm_range *range;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	bool notifier_alloc = false;
+	u64 chunk_size;
+	int err;
+	bool migrate_devmem;
+
+	if (fault_addr < gpusvm->mm_start ||
+	    fault_addr > gpusvm->mm_start + gpusvm->mm_range) {

return ERR_PTR(-EINVAL)?

Quoted 153 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +12 -0
+		err = -EINVAL;
+		goto err_out;
+	}
+
+	if (!mmget_not_zero(mm)) {
+		err = -EFAULT;
+		goto err_out;
+	}
+
+	notifier = drm_gpusvm_notifier_find(gpusvm, fault_addr);
+	if (!notifier) {
+		notifier = drm_gpusvm_notifier_alloc(gpusvm,

fault_addr);

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+		if (IS_ERR(notifier)) {
+			err = PTR_ERR(notifier);
+			goto err_mmunlock;
+		}
+		notifier_alloc = true;
+		err = mmu_interval_notifier_insert(&notifier-
Quoted 1 line

notifier,

(quoted diff) +1 -0
+						   mm, notifier-
Quoted 1 line

interval.start,

(quoted diff) +1 -0
+						   notifier-
Quoted 1 line

interval.end -

(quoted diff) +1 -0
+						   notifier-
Quoted 1 line

interval.start,

(quoted diff) +1 -0
+						  

&drm_gpusvm_notifier_ops);

drivers/gpu/drm/xe/drm_gpusvm.c new file +18 -0
+		if (err)
+			goto err_notifier;
+	}
+
+	mmap_read_lock(mm);
+
+	vas = vma_lookup(mm, fault_addr);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_notifier_remove;
+	}
+
+	if (!ctx->read_only && !(vas->vm_flags & VM_WRITE)) {
+		err = -EPERM;
+		goto err_notifier_remove;
+	}
+
+	range = drm_gpusvm_range_find(notifier, fault_addr,

fault_addr + 1);

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+	if (range)
+		goto out_mmunlock;
+	/*
+	 * XXX: Short-circuiting migration based on migrate_vma_*

current

(quoted diff) +1 -0
+	 * limitations. If/when migrate_vma_* add more support, this

logic will

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+	 * have to change.
+	 */
+	migrate_devmem = ctx->devmem_possible &&
+		vma_is_anonymous(vas) && !is_vm_hugetlb_page(vas);
+
+	chunk_size = drm_gpusvm_range_chunk_size(gpusvm, notifier,

vas,

(quoted diff) +1 -0
+						 fault_addr,

gpuva_start,

(quoted diff) +1 -0
+						 gpuva_end,

migrate_devmem &&

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+						 ctx->check_pages);
+	if (chunk_size == LONG_MAX) {
+		err = -EINVAL;
+		goto err_notifier_remove;
+	}
+
+	range = drm_gpusvm_range_alloc(gpusvm, notifier, fault_addr,

chunk_size,

drivers/gpu/drm/xe/drm_gpusvm.c new file +31 -0
+				       migrate_devmem);
+	if (IS_ERR(range)) {
+		err = PTR_ERR(range);
+		goto err_notifier_remove;
+	}
+
+	drm_gpusvm_range_insert(notifier, range);
+	if (notifier_alloc)
+		drm_gpusvm_notifier_insert(gpusvm, notifier);
+
+out_mmunlock:
+	mmap_read_unlock(mm);
+	mmput(mm);
+
+	return range;
+
+err_notifier_remove:
+	mmap_read_unlock(mm);
+	if (notifier_alloc)
+		mmu_interval_notifier_remove(&notifier->notifier);
+err_notifier:
+	if (notifier_alloc)
+		drm_gpusvm_notifier_free(gpusvm, notifier);
+err_mmunlock:
+	mmput(mm);
+err_out:
+	return ERR_PTR(err);
+}
+
+/**
+ * __drm_gpusvm_range_unmap_pages - Unmap pages associated with a

GPU SVM range (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @npages: Number of pages to unmap
+ *
+ * This function unmap pages associated with a GPU SVM range. Assumes and
+ * asserts correct locking is in place when called.
+ */
+static void __drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,

(quoted diff) +1 -0
+					   struct drm_gpusvm_range

*range,

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+					   unsigned long npages)
+{
+	unsigned long i, j;
+	struct drm_pagemap *dpagemap = range->dpagemap;
+	struct device *dev = gpusvm->drm->dev;
+
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	if (range->flags.has_dma_mapping) {
+		for (i = 0, j = 0; i < npages; j++) {
+			struct drm_pagemap_dma_addr *addr = &range-
Quoted 1 line

dma_addr[j];

(quoted diff) +2 -0
+
+			if (addr->proto == DRM_INTERCONNECT_SYSTEM)

{

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+				dma_unmap_page(dev,
+					       addr->addr,
+					       PAGE_SIZE << addr-
Quoted 1 line

order,

(quoted diff) +2 -0
+					       addr->dir);
+			} else if (dpagemap && dpagemap->ops-
Quoted 1 line

unmap_dma) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+				dpagemap->ops->unmap_dma(dpagemap,
+							 dev,
+							 *addr);
+			}
+			i += 1 << addr->order;
+		}
+		range->flags.has_devmem_pages = false;
+		range->flags.has_dma_mapping = false;
+		range->dpagemap = NULL;
+	}
+}
+
+/**
+ * drm_gpusvm_range_free_pages - Free pages associated with a GPU
Quoted 5 lines

SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function free pages associated with a GPU SVM range.

Frees the dma address array

Quoted 314 lines

+ */

(quoted diff) +2 -0
+static void drm_gpusvm_range_free_pages(struct drm_gpusvm *gpusvm,
+					struct drm_gpusvm_range

*range)

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+{
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	if (range->dma_addr) {
+		kvfree(range->dma_addr);
+		range->dma_addr = NULL;
+	}
+}
+
+/**
+ * drm_gpusvm_range_remove - Remove GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range to be removed
+ *
+ * This function removes the specified GPU SVM range and also

removes the parent
+ * GPU SVM notifier if no more ranges remain in the notifier. The caller must
+ * hold a lock to protect range and notifier removal.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range)
+{
+	unsigned long npages = npages_in_range(range->va.start,

range->va.end);

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+	struct drm_gpusvm_notifier *notifier;
+
+	notifier = drm_gpusvm_notifier_find(gpusvm, range-
Quoted 1 line

va.start);

drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+	if (WARN_ON_ONCE(!notifier))
+		return;
+
+	drm_gpusvm_notifier_lock(gpusvm);
+	__drm_gpusvm_range_unmap_pages(gpusvm, range, npages);
+	drm_gpusvm_range_free_pages(gpusvm, range);
+	__drm_gpusvm_range_remove(notifier, range);
+	drm_gpusvm_notifier_unlock(gpusvm);
+
+	drm_gpusvm_range_put(range);
+
+	if (RB_EMPTY_ROOT(&notifier->root.rb_root)) {
+		if (!notifier->flags.removed)
+			mmu_interval_notifier_remove(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+		drm_gpusvm_notifier_remove(gpusvm, notifier);
+		drm_gpusvm_notifier_free(gpusvm, notifier);
+	}
+}
+
+/**
+ * drm_gpusvm_range_get - Get a reference to GPU SVM range
+ * @range: Pointer to the GPU SVM range
+ *
+ * This function increments the reference count of the specified GPU

SVM range.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range)
+{
+	kref_get(&range->refcount);
+
+	return range;
+}
+
+/**
+ * drm_gpusvm_range_destroy - Destroy GPU SVM range
+ * @refcount: Pointer to the reference counter embedded in the GPU

SVM range
+ *
+ * This function destroys the specified GPU SVM range when its reference count
+ * reaches zero. If a custom range-free function is provided, it is invoked to
+ * free the range; otherwise, the range is deallocated using kfree().
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_range_destroy(struct kref *refcount)
+{
+	struct drm_gpusvm_range *range =
+		container_of(refcount, struct drm_gpusvm_range,

refcount);

drivers/gpu/drm/xe/drm_gpusvm.c new file +13 -0
+	struct drm_gpusvm *gpusvm = range->gpusvm;
+
+	if (gpusvm->ops->range_free)
+		gpusvm->ops->range_free(range);
+	else
+		kfree(range);
+}
+
+/**
+ * drm_gpusvm_range_put - Put a reference to GPU SVM range
+ * @range: Pointer to the GPU SVM range
+ *
+ * This function decrements the reference count of the specified GPU

SVM range
+ * and frees it when the count reaches zero.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range)
+{
+	kref_put(&range->refcount, drm_gpusvm_range_destroy);
+}
+
+/**
+ * drm_gpusvm_range_pages_valid - GPU SVM range pages valid
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid.

Expected be
+ * called holding gpusvm->notifier_lock and as the last step before commiting a
+ * GPU binding.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range)
+{
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	return range->flags.has_devmem_pages || range-
Quoted 1 line

flags.has_dma_mapping;

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+}
+
+/**
+ * drm_gpusvm_range_pages_valid_unlocked - GPU SVM range pages valid

unlocked
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid. Expected be
+ * called without holding gpusvm->notifier_lock.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+static bool
+drm_gpusvm_range_pages_valid_unlocked(struct drm_gpusvm *gpusvm,
+				      struct drm_gpusvm_range

*range)

drivers/gpu/drm/xe/drm_gpusvm.c new file +22 -0
+{
+	bool pages_valid;
+
+	if (!range->dma_addr)
+		return false;
+
+	drm_gpusvm_notifier_lock(gpusvm);
+	pages_valid = drm_gpusvm_range_pages_valid(gpusvm, range);
+	if (!pages_valid)
+		drm_gpusvm_range_free_pages(gpusvm, range);
+	drm_gpusvm_notifier_unlock(gpusvm);
+
+	return pages_valid;
+}
+
+/**
+ * drm_gpusvm_range_get_pages - Get pages for a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function gets pages for a GPU SVM range and ensures they are

mapped for
+ * DMA access.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	struct mmu_interval_notifier *notifier = &range->notifier-
Quoted 1 line

notifier;

(quoted diff) +2 -0
+	struct hmm_range hmm_range = {
+		.default_flags = HMM_PFN_REQ_FAULT | (ctx->read_only

? 0 :

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+			HMM_PFN_REQ_WRITE),
+		.notifier = notifier,
+		.start = range->va.start,
+		.end = range->va.end,
+		.dev_private_owner = gpusvm-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	struct drm_gpusvm_zdd *zdd;
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

(quoted diff) +2 -0
+	unsigned long i, j;
+	unsigned long npages = npages_in_range(range->va.start,

range->va.end);

drivers/gpu/drm/xe/drm_gpusvm.c new file +33 -0
+	unsigned long num_dma_mapped;
+	unsigned int order = 0;
+	unsigned long *pfns;
+	struct page **pages;
+	int err = 0;
+	struct dev_pagemap *pagemap;
+	struct drm_pagemap *dpagemap;
+
+retry:
+	hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
+	if (drm_gpusvm_range_pages_valid_unlocked(gpusvm, range))
+		goto set_seqno;
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL);
+	if (!pfns)
+		return -ENOMEM;
+
+	if (!mmget_not_zero(mm)) {
+		err = -EFAULT;
+		goto err_out;
+	}
+
+	hmm_range.hmm_pfns = pfns;
+	while (true) {
+		mmap_read_lock(mm);
+		err = hmm_range_fault(&hmm_range);
+		mmap_read_unlock(mm);
+
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq =

mmu_interval_read_begin(notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +17 -0
+			continue;
+		}
+		break;
+	}
+	mmput(mm);
+	if (err)
+		goto err_free;
+
+	pages = (struct page **)pfns;
+map_pages:
+	/*
+	 * Perform all dma mappings under the notifier lock to not
+	 * access freed pages. A notifier will either block on
+	 * the notifier lock or unmap dma.
+	 */
+	drm_gpusvm_notifier_lock(gpusvm);
+	if (mmu_interval_read_retry(notifier,

hmm_range.notifier_seq)) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+		drm_gpusvm_notifier_unlock(gpusvm);
+		goto retry;
+	}
+
+	if (!range->dma_addr) {
+		/* Unlock and restart mapping to allocate memory. */
+		drm_gpusvm_notifier_unlock(gpusvm);
+		range->dma_addr = kvmalloc_array(npages,

sizeof(*range->dma_addr),

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+						 GFP_KERNEL);
+		if (!range->dma_addr) {
+			err = -ENOMEM;
+			goto err_free;
+		}
+		goto map_pages;
+	}
+
+	zdd = NULL;
+	num_dma_mapped = 0;
+	for (i = 0, j = 0; i < npages; ++j) {
+		struct page *page = hmm_pfn_to_page(pfns[i]);
+
+		order = hmm_pfn_to_map_order(pfns[i]);
+		if (is_device_private_page(page) ||

is_device_coherent_page(page)) {

(quoted diff) +1 -0
+			if (zdd != page->zone_device_data && i > 0)

{

drivers/gpu/drm/xe/drm_gpusvm.c new file +12 -0
+				err = -EOPNOTSUPP;
+				goto err_unmap;
+			}
+			zdd = page->zone_device_data;
+			if (pagemap != page->pgmap) {
+				if (i > 0) {
+					err = -EOPNOTSUPP;
+					goto err_unmap;
+				}
+
+				pagemap = page->pgmap;
+				dpagemap = zdd->devmem_allocation-
Quoted 1 line

dpagemap;

(quoted diff) +1 -0
+				if (drm_WARN_ON(gpusvm->drm,

!dpagemap)) {

(quoted diff) +2 -0
+					/*
+					 * Raced. This is not

supposed to happen

(quoted diff) +1 -0
+					 * since hmm_range_fault()

should've migrated

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+					 * this page to system.
+					 */
+					err = -EAGAIN;
+					goto err_unmap;
+				}
+			}
+			range->dma_addr[j] =
+				dpagemap->ops->map_dma(dpagemap,

gpusvm->drm->dev,

drivers/gpu/drm/xe/drm_gpusvm.c new file +2 -0
+						       page, order,
+						      

DMA_BIDIRECTIONAL);

(quoted diff) +1 -0
+			if (dma_mapping_error(gpusvm->drm->dev,

range->dma_addr[j].addr)) {

drivers/gpu/drm/xe/drm_gpusvm.c line 1385

Quoted 10 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+				err = -EFAULT;
+				goto err_unmap;
+			}
+
+			pages[i] = page;
+		} else {
+			dma_addr_t addr;
+
+			if (is_zone_device_page(page) || zdd) {
+				err = -EOPNOTSUPP;

I suppose before merging we want to support mixed ranges since migration is best effort only, or what are the plans here?

Quoted 72 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+				goto err_unmap;
+			}
+
+			addr = dma_map_page(gpusvm->drm->dev,
+					    page, 0,
+					    PAGE_SIZE << order,
+					    DMA_BIDIRECTIONAL);
+			if (dma_mapping_error(gpusvm->drm->dev,

addr)) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+				err = -EFAULT;
+				goto err_unmap;
+			}
+
+			range->dma_addr[j] =

drm_pagemap_dma_addr_encode

(quoted diff) +1 -0
+				(addr, DRM_INTERCONNECT_SYSTEM,

order,

drivers/gpu/drm/xe/drm_gpusvm.c new file +21 -0
+				 DMA_BIDIRECTIONAL);
+		}
+		i += 1 << order;
+		num_dma_mapped = i;
+	}
+
+	range->flags.has_dma_mapping = true;
+	if (zdd) {
+		range->flags.has_devmem_pages = true;
+		range->dpagemap = dpagemap;
+	}
+
+	drm_gpusvm_notifier_unlock(gpusvm);
+	kvfree(pfns);
+set_seqno:
+	range->notifier_seq = hmm_range.notifier_seq;
+
+	return 0;
+
+err_unmap:
+	__drm_gpusvm_range_unmap_pages(gpusvm, range,

num_dma_mapped);

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+	drm_gpusvm_notifier_unlock(gpusvm);
+err_free:
+	kvfree(pfns);
+err_out:
+	if (err == -EAGAIN)
+		goto retry;
+	return err;
+}
+
+/**
+ * drm_gpusvm_range_unmap_pages - Unmap pages associated with a GPU

SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed that gpusvm->notifier_lock is held in write mode; if it
+ * is clear, it acquires gpusvm->notifier_lock in read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops-

Quoted 1 line

invalidate for IOMMU

+ * security model.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx)
+{
+	unsigned long npages = npages_in_range(range->va.start,

range->va.end);

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+

drivers/gpu/drm/xe/drm_gpusvm.c line 1457

Quoted 10 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+	if (ctx->in_notifier)
+		lockdep_assert_held_write(&gpusvm->notifier_lock);
+	else
+		drm_gpusvm_notifier_lock(gpusvm);
+
+	__drm_gpusvm_range_unmap_pages(gpusvm, range, npages);
+
+	if (!ctx->in_notifier)
+		drm_gpusvm_notifier_unlock(gpusvm);
+}

NIT: Separate functions for locked / unlocked makes life easier for static code analyzers.

Section below I think should belong to drm_pagemap.c

drivers/gpu/drm/xe/drm_gpusvm.c line 1465

Quoted 8 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+
+/**
+ * drm_gpusvm_migration_put_page - Put a migration page
+ * @page: Pointer to the page to put
+ *
+ * This function unlocks and puts a page.
+ */
+static void drm_gpusvm_migration_put_page(struct page *page)

_unlock_put_page()?

Quoted 86 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+{
+	unlock_page(page);
+	put_page(page);
+}
+
+/**
+ * drm_gpusvm_migration_put_pages - Put migration pages
+ * @npages: Number of pages
+ * @migrate_pfn: Array of migrate page frame numbers
+ *
+ * This function puts an array of pages.
+ */
+static void drm_gpusvm_migration_put_pages(unsigned long npages,
+					   unsigned long

*migrate_pfn)

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!migrate_pfn[i])
+			continue;
+
+		drm_gpusvm_migration_put_page(migrate_pfn_to_page(mi

grate_pfn[i]));

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+		migrate_pfn[i] = 0;
+	}
+}
+
+/**
+ * drm_gpusvm_get_devmem_page - Get a reference to a device memory

page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU SVM zone
+ * device data and initializes it for zone device usage.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+static void drm_gpusvm_get_devmem_page(struct page *page,
+				     struct drm_gpusvm_zdd *zdd)
+{
+	page->zone_device_data = drm_gpusvm_zdd_get(zdd);
+	zone_device_page_init(page);
+}
+
+/**
+ * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM

migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Return: 0 on success, -EFAULT if an error occurs during mapping.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+static int drm_gpusvm_migrate_map_pages(struct device *dev,
+					dma_addr_t *dma_addr,
+					long unsigned int

*migrate_pfn,

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+					unsigned long npages,
+					enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page =

migrate_pfn_to_page(migrate_pfn[i]);

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+
+		if (!page)
+			continue;
+
+		if (WARN_ON_ONCE(is_zone_device_page(page)))
+			return -EFAULT;
+
+		dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE,

dir);

drivers/gpu/drm/xe/drm_gpusvm.c line 1545

Quoted 6 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+		if (dma_mapping_error(dev, dma_addr[i]))
+			return -EFAULT;
+	}
+
+	return 0;
+}

TBC'd

/Thomas

author maintainer
#37 lore raw

Re: [PATCH v2 05/29] drm/gpusvm: Add support for GPU Shared Virtual Memory

Quoted 31 lines · Thomas Hellström

On Tue, 2024-10-15 at 20:24 -0700, Matthew Brost wrote:

Continued review.

Quoted 27 lines

+/**
+ * struct drm_gpusvm_zdd - GPU SVM zone device data
+ *
+ * @refcount: Reference count for the zdd
+ * @destroy_work: Work structure for asynchronous zdd destruction
+ * @devmem_allocation: device memory allocation
+ * @device_private_page_owner: Device private pages owner
+ *
+ * This structure serves as a generic wrapper installed in
+ * page->zone_device_data. It provides infrastructure for looking up a device
+ * memory allocation upon CPU page fault and asynchronously releasing device
+ * memory once the CPU has no page references. Asynchronous release is useful
+ * because CPU page references can be dropped in IRQ contexts, while releasing
+ * device memory likely requires sleeping locks.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+struct drm_gpusvm_zdd {
+	struct kref refcount;
+	struct work_struct destroy_work;
+	struct drm_gpusvm_devmem *devmem_allocation;
+	void *device_private_page_owner;
+};
+
Quoted 10 lines
Quoted 3 lines

+/**
+ * drm_gpusvm_zdd_destroy_work_func - Work function for destroying a zdd

NIT: Even if the above kerneldoc format works, I keep trying to enforce using () after function names and function-like macros, like described
here: https://docs.kernel.org/doc-guide/kernel-doc.html Could we update? Also that doc calls for using "Return:" instead of "Returns:".

Will fix up. Thanks for the ref.

Quoted 90 lines
Quoted 90 lines

+ * @w: Pointer to the work_struct
+ *
+ * This function releases device memory, puts GPU SVM range, and frees zdd.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_zdd_destroy_work_func(struct work_struct *w)
+{
+	struct drm_gpusvm_zdd *zdd =
+		container_of(w, struct drm_gpusvm_zdd,

destroy_work);

(quoted diff) +1 -0
+	const struct drm_gpusvm_devmem_ops *ops = zdd-
Quoted 1 line

devmem_allocation ?

drivers/gpu/drm/xe/drm_gpusvm.c new file +12 -0
+		zdd->devmem_allocation->ops : NULL;
+
+	if (zdd->devmem_allocation && ops->devmem_release)
+		ops->devmem_release(zdd->devmem_allocation);
+	kfree(zdd);
+}
+
+/**
+ * drm_gpusvm_zdd_alloc - Allocate a zdd structure.
+ * @device_private_page_owner: Device private pages owner
+ *
+ * This function allocates and initializes a new zdd structure. It

sets up the
+ * reference count and initializes the destroy work.
+ *
+ * Returns:
+ * Pointer to the allocated zdd on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+static struct drm_gpusvm_zdd *
+drm_gpusvm_zdd_alloc(void *device_private_page_owner)
+{
+	struct drm_gpusvm_zdd *zdd;
+
+	zdd = kmalloc(sizeof(*zdd), GFP_KERNEL);
+	if (!zdd)
+		return NULL;
+
+	kref_init(&zdd->refcount);
+	INIT_WORK(&zdd->destroy_work,

drm_gpusvm_zdd_destroy_work_func);

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+	zdd->devmem_allocation = NULL;
+	zdd->device_private_page_owner = device_private_page_owner;
+
+	return zdd;
+}
+
+/**
+ * drm_gpusvm_zdd_get - Get a reference to a zdd structure.
+ * @zdd: Pointer to the zdd structure.
+ *
+ * This function increments the reference count of the provided zdd

structure.
+ *
+ * Returns: Pointer to the zdd structure.
+ */
+static struct drm_gpusvm_zdd *drm_gpusvm_zdd_get(struct drm_gpusvm_zdd *zdd)

drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+{
+	kref_get(&zdd->refcount);
+	return zdd;
+}
+
+/**
+ * drm_gpusvm_zdd_destroy - Destroy a zdd structure.
+ * @ref: Pointer to the reference count structure.
+ *
+ * This function queues the destroy_work of the zdd for asynchronous

destruction.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+static void drm_gpusvm_zdd_destroy(struct kref *ref)
+{
+	struct drm_gpusvm_zdd *zdd =
+		container_of(ref, struct drm_gpusvm_zdd, refcount);
+
+	if (zdd->devmem_allocation)
+		WRITE_ONCE(zdd->devmem_allocation->detached, true);
+	schedule_work(&zdd->destroy_work);
+}
+
+/**
+ * drm_gpusvm_zdd_put - Put a zdd reference.
+ * @zdd: Pointer to the zdd structure.
+ *
+ * This function decrements the reference count of the provided zdd

structure
+ * and schedules its destruction if the count drops to zero.
+ */
+static void drm_gpusvm_zdd_put(struct drm_gpusvm_zdd *zdd)

Quoted 9 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 389

Quoted 3 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+{
+	kref_put(&zdd->refcount, drm_gpusvm_zdd_destroy);
+}

As mentioned earlier, I think the above drm_gpusvm_zdd functions should move to drm_pagemap.c. I don't think they are used in drm_gpusvm other than to, at get_pages time, ensure all device private pages are from the same pagemap?

The are used in __drm_gpusvm_migrate_to_ram to find devmem_allocation and associated ops.

Also in drm_gpusvm_migrate_to_ram to find the size and device_private_page_owner.

I think the placement here is correct for now but open to shuffling this around in the future if this makes sense.

Quoted 56 lines
Quoted 56 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+
+/**
+ * drm_gpusvm_range_find - Find GPU SVM range from GPU SVM notifier
+ * @notifier: Pointer to the GPU SVM notifier structure.
+ * @start: Start address of the range
+ * @end: End address of the range
+ *
+ * Return: A pointer to the drm_gpusvm_range if found or NULL
+ */
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, u64

start, u64 end)

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+{
+	return range_iter_first(&notifier->root, start, end - 1);
+}
+
+/**
+ * drm_gpusvm_for_each_range_safe - Safely iterate over GPU SVM

ranges in a notifier
+ * @range__: Iterator variable for the ranges
+ * @next__: Iterator variable for the ranges temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier while
+ * removing ranges from it.
+ */
+#define drm_gpusvm_for_each_range_safe(range__, next__, notifier__,
start__, end__) \

(quoted diff) +1 -0
+	for ((range__) = drm_gpusvm_range_find((notifier__),

(start__), (end__)), \

(quoted diff) +1 -0
+	     (next__) =

__drm_gpusvm_range_next(range__); \

(quoted diff) +1 -0
+	     (range__) && (range__->va.start <

(end__)); \

(quoted diff) +1 -0
+	     (range__) = (next__), (next__) =

__drm_gpusvm_range_next(range__))

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+
+/**
+ * __drm_gpusvm_notifier_next - get the next drm_gpusvm_notifier in

the list
+ * @notifier: a pointer to the current drm_gpusvm_notifier
+ *
+ * Return: A pointer to the next drm_gpusvm_notifier if available, or NULL if
+ *         the current notifier is the last one or if the input notifier is
+ *         NULL.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+static struct drm_gpusvm_notifier *
+__drm_gpusvm_notifier_next(struct drm_gpusvm_notifier *notifier)
+{
+	if (notifier && !list_is_last(&notifier->rb.entry,
+				      &notifier->gpusvm-
Quoted 6 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 435

Quoted 2 lines
Quoted 1 line

notifier_list))

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+		return list_next_entry(notifier, rb.entry);

Why aren't we using notifier_iter_next() here? Then the linked list could be skipped.

I shamlessly copied this from GPU VM. I think the list is useful for faster iteration and safe removal of items while walking.

Quoted 17 lines
Quoted 17 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+
+	return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_notifier - Iterate over GPU SVM notifiers in

a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm.
+ */
+#define drm_gpusvm_for_each_notifier(notifier__, gpusvm__, start__,
end__) \

(quoted diff) +1 -0
+	for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,
Quoted 9 lines

Review

Quoted 5 lines

(start__), (end__) - 1); \

(quoted diff) +1 -0
+	     (notifier__) && (notifier__->interval.start <

(end__)); \

(quoted diff) +2 -0
+	     (notifier__) = __drm_gpusvm_notifier_next(notifier__))
+

Looks like end__ is not honored except for the first iteration. Relates to the above question.

Again shameless copy from GPU VM... Missing what the problem is. The condition to break the loop is:

'(notifier__) && (notifier__->interval.start < (end__)'

Quoted 17 lines
Quoted 17 lines

+/**
+ * drm_gpusvm_for_each_notifier_safe - Safely iterate over GPU SVM notifiers in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @next__: Iterator variable for the notifiers temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm while
+ * removing notifiers from it.
+ */
+#define drm_gpusvm_for_each_notifier_safe(notifier__, next__,
gpusvm__, start__, end__) \

(quoted diff) +1 -0
+	for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,

(start__), (end__) - 1), \

Quoted 9 lines
Quoted 6 lines
(quoted diff) +1 -0
+	     (next__) =

__drm_gpusvm_notifier_next(notifier__); \

(quoted diff) +1 -0
+	     (notifier__) && (notifier__->interval.start <

(end__)); \

(quoted diff) +1 -0
+	     (notifier__) = (next__), (next__) =

__drm_gpusvm_notifier_next(notifier__))

Same here.

Alsp present:

(notifier__) && (notifier__->interval.start < (end__)

Quoted 137 lines
Quoted 129 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+
+/**
+ * drm_gpusvm_notifier_invalidate - Invalidate a GPU SVM notifier.
+ * @mni: Pointer to the mmu_interval_notifier structure.
+ * @mmu_range: Pointer to the mmu_notifier_range structure.
+ * @cur_seq: Current sequence number.
+ *
+ * This function serves as a generic MMU notifier for GPU SVM. It

sets the MMU
+ * notifier sequence number and calls the driver invalidate vfunc under
+ * gpusvm->notifier_lock.
+ *
+ * Returns:
+ * true if the operation succeeds, false otherwise.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+static bool
+drm_gpusvm_notifier_invalidate(struct mmu_interval_notifier *mni,
+			       const struct mmu_notifier_range

*mmu_range,

drivers/gpu/drm/xe/drm_gpusvm.c new file +19 -0
+			       unsigned long cur_seq)
+{
+	struct drm_gpusvm_notifier *notifier =
+		container_of(mni, typeof(*notifier), notifier);
+	struct drm_gpusvm *gpusvm = notifier->gpusvm;
+
+	if (!mmu_notifier_range_blockable(mmu_range))
+		return false;
+
+	down_write(&gpusvm->notifier_lock);
+	mmu_interval_set_seq(mni, cur_seq);
+	gpusvm->ops->invalidate(gpusvm, notifier, mmu_range);
+	up_write(&gpusvm->notifier_lock);
+
+	return true;
+}
+
+/**
+ * drm_gpusvm_notifier_ops - MMU interval notifier operations for

GPU SVM
+ */
+static const struct mmu_interval_notifier_ops
drm_gpusvm_notifier_ops = {

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+	.invalidate = drm_gpusvm_notifier_invalidate,
+};
+
+/**
+ * drm_gpusvm_init - Initialize the GPU SVM.
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @name: Name of the GPU SVM.
+ * @drm: Pointer to the DRM device structure.
+ * @mm: Pointer to the mm_struct for the address space.
+ * @device_private_page_owner: Device private pages owner.
+ * @mm_start: Start address of GPU SVM.
+ * @mm_range: Range of the GPU SVM.
+ * @notifier_size: Size of individual notifiers.
+ * @ops: Pointer to the operations structure for GPU SVM.
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range

allocation.
+ *               Entries should be powers of 2 in descending order with last
+ *               entry being SZ_4K.
+ * @num_chunks: Number of chunks.
+ *
+ * This function initializes the GPU SVM.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void

*device_private_page_owner,

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+		    u64 mm_start, u64 mm_range, u64 notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const u64 *chunk_sizes, int num_chunks)
+{
+	if (!ops->invalidate || !num_chunks)
+		return -EINVAL;
+
+	gpusvm->name = name;
+	gpusvm->drm = drm;
+	gpusvm->mm = mm;
+	gpusvm->device_private_page_owner =

device_private_page_owner;

drivers/gpu/drm/xe/drm_gpusvm.c new file +26 -0
+	gpusvm->mm_start = mm_start;
+	gpusvm->mm_range = mm_range;
+	gpusvm->notifier_size = notifier_size;
+	gpusvm->ops = ops;
+	gpusvm->chunk_sizes = chunk_sizes;
+	gpusvm->num_chunks = num_chunks;
+
+	mmgrab(mm);
+	gpusvm->root = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&gpusvm->notifier_list);
+
+	init_rwsem(&gpusvm->notifier_lock);
+
+	fs_reclaim_acquire(GFP_KERNEL);
+	might_lock(&gpusvm->notifier_lock);
+	fs_reclaim_release(GFP_KERNEL);
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_notifier_find - Find GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure
+ * @fault_addr__: Fault address
+ *
+ * This macro finds the GPU SVM notifier associated with the fault

address.
+ *
+ * Returns:
+ * Pointer to the GPU SVM notifier on success, NULL otherwise.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+#define drm_gpusvm_notifier_find(gpusvm__, fault_addr__)	\
+	notifier_iter_first(&(gpusvm__)->root, (fault_addr__),	\
+			    (fault_addr__ + 1))
+
+/**
+ * to_drm_gpusvm_notifier - retrieve the container struct for a

given rbtree node
+ * @node__: a pointer to the rbtree node embedded within a drm_gpusvm_notifier struct
+ *
+ * Return: A pointer to the containing drm_gpusvm_notifier structure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c line 588

Quoted 3 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+#define to_drm_gpusvm_notifier(__node)				\
+	container_of((__node), struct drm_gpusvm_notifier, rb.node)
+

There appears to be a number of function-like macros in the code, which look like they can be converted to functions. Linux prefers functions over macros when possible:

Quoted 2 lines

https://www.kernel.org/doc/html/v5.8/process/coding-style.html#macros-enums-and-rtl

Will convert all macros to functions if possible. Again thanks for ref.

Quoted 33 lines
Quoted 33 lines

+/**
+ * drm_gpusvm_notifier_insert - Insert GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function inserts the GPU SVM notifier into the GPU SVM RB tree and list.
+ */

(quoted diff) +2 -0
+static void drm_gpusvm_notifier_insert(struct drm_gpusvm *gpusvm,
+				       struct drm_gpusvm_notifier

*notifier)

drivers/gpu/drm/xe/drm_gpusvm.c new file +21 -0
+{
+	struct rb_node *node;
+	struct list_head *head;
+
+	notifier_insert(notifier, &gpusvm->root);
+
+	node = rb_prev(&notifier->rb.node);
+	if (node)
+		head = &(to_drm_gpusvm_notifier(node))->rb.entry;
+	else
+		head = &gpusvm->notifier_list;
+
+	list_add(&notifier->rb.entry, head);
+}
+
+/**
+ * drm_gpusvm_notifier_remove - Remove GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM tructure
+ * @notifier__: Pointer to the GPU SVM notifier structure
+ *
+ * This macro removes the GPU SVM notifier from the GPU SVM RB tree
Quoted 9 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 622

Quoted 5 lines

and list.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+#define drm_gpusvm_notifier_remove(gpusvm__, notifier__)	\
+	notifier_remove((notifier__), &(gpusvm__)->root);	\
+	list_del(&(notifier__)->rb.entry)

Unless this can be made a function, Pls use
do { } while (0)

I think it can be made a function or otherwise yea will use do { } while (0).

Quoted 135 lines
Quoted 135 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+
+/**
+ * drm_gpusvm_fini - Finalize the GPU SVM.
+ * @gpusvm: Pointer to the GPU SVM structure.
+ *
+ * This function finalizes the GPU SVM by cleaning up any remaining

ranges and
+ * notifiers, and dropping a reference to struct MM.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm)
+{
+	struct drm_gpusvm_notifier *notifier, *next;
+
+	drm_gpusvm_for_each_notifier_safe(notifier, next, gpusvm, 0,

LONG_MAX) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+		struct drm_gpusvm_range *range, *__next;
+
+		/*
+		 * Remove notifier first to avoid racing with any

invalidation

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+		 */
+		mmu_interval_notifier_remove(&notifier->notifier);
+		notifier->flags.removed = true;
+
+		drm_gpusvm_for_each_range_safe(range, __next,

notifier, 0,

drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+					       LONG_MAX)
+			drm_gpusvm_range_remove(gpusvm, range);
+	}
+
+	mmdrop(gpusvm->mm);
+	WARN_ON(!RB_EMPTY_ROOT(&gpusvm->root.rb_root));
+}
+
+/**
+ * drm_gpusvm_notifier_alloc - Allocate GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ *
+ * This function allocates and initializes the GPU SVM notifier

structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM notifier on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+static struct drm_gpusvm_notifier *
+drm_gpusvm_notifier_alloc(struct drm_gpusvm *gpusvm, u64 fault_addr)
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	if (gpusvm->ops->notifier_alloc)
+		notifier = gpusvm->ops->notifier_alloc();
+	else
+		notifier = kzalloc(sizeof(*notifier), GFP_KERNEL);
+
+	if (!notifier)
+		return ERR_PTR(-ENOMEM);
+
+	notifier->gpusvm = gpusvm;
+	notifier->interval.start = ALIGN_DOWN(fault_addr, gpusvm-
Quoted 1 line

notifier_size);

(quoted diff) +1 -0
+	notifier->interval.end = ALIGN(fault_addr + 1, gpusvm-
Quoted 1 line

notifier_size);

drivers/gpu/drm/xe/drm_gpusvm.c new file +16 -0
+	INIT_LIST_HEAD(&notifier->rb.entry);
+	notifier->root = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&notifier->range_list);
+
+	return notifier;
+}
+
+/**
+ * drm_gpusvm_notifier_free - Free GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function frees the GPU SVM notifier structure.
+ */
+static void drm_gpusvm_notifier_free(struct drm_gpusvm *gpusvm,
+				     struct drm_gpusvm_notifier

*notifier)

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+{
+	WARN_ON(!RB_EMPTY_ROOT(&notifier->root.rb_root));
+
+	if (gpusvm->ops->notifier_free)
+		gpusvm->ops->notifier_free(notifier);
+	else
+		kfree(notifier);
+}
+
+/**
+ * to_drm_gpusvm_range - retrieve the container struct for a given

rbtree node
+ * @node__: a pointer to the rbtree node embedded within a drm_gpusvm_range struct
+ *
+ * Return: A pointer to the containing drm_gpusvm_range structure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+#define to_drm_gpusvm_range(node__)	\
+	container_of((node__), struct drm_gpusvm_range, rb.node)
+
+/**
+ * drm_gpusvm_range_insert - Insert GPU SVM range
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function inserts the GPU SVM range into the notifier RB tree

and list.
+ */
+static void drm_gpusvm_range_insert(struct drm_gpusvm_notifier *notifier,

drivers/gpu/drm/xe/drm_gpusvm.c new file +23 -0
+				    struct drm_gpusvm_range *range)
+{
+	struct rb_node *node;
+	struct list_head *head;
+
+	drm_gpusvm_notifier_lock(notifier->gpusvm);
+	range_insert(range, &notifier->root);
+
+	node = rb_prev(&range->rb.node);
+	if (node)
+		head = &(to_drm_gpusvm_range(node))->rb.entry;
+	else
+		head = &notifier->range_list;
+
+	list_add(&range->rb.entry, head);
+	drm_gpusvm_notifier_unlock(notifier->gpusvm);
+}
+
+/**
+ * __drm_gpusvm_range_remove - Remove GPU SVM range
+ * @notifier__: Pointer to the GPU SVM notifier structure
+ * @range__: Pointer to the GPU SVM range structure
+ *
Quoted 9 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 748

Quoted 6 lines

+ * This macro removes the GPU SVM range from the notifier RB tree and list.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+#define __drm_gpusvm_range_remove(notifier__, range__)		\
+	range_remove((range__), &(notifier__)->root);		\
+	list_del(&(range__)->rb.entry)

Same thing as for the notifier rb tree. And do we need the linked list?

Same answer.

Quoted 199 lines
Quoted 199 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+
+/**
+ * drm_gpusvm_range_alloc - Allocate GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @fault_addr: Fault address
+ * @chunk_size: Chunk size
+ * @migrate_devmem: Flag indicating whether to migrate device memory
+ *
+ * This function allocates and initializes the GPU SVM range

structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM range on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+static struct drm_gpusvm_range *
+drm_gpusvm_range_alloc(struct drm_gpusvm *gpusvm,
+		       struct drm_gpusvm_notifier *notifier,
+		       u64 fault_addr, u64 chunk_size, bool

migrate_devmem)

drivers/gpu/drm/xe/drm_gpusvm.c new file +31 -0
+{
+	struct drm_gpusvm_range *range;
+
+	if (gpusvm->ops->range_alloc)
+		range = gpusvm->ops->range_alloc(gpusvm);
+	else
+		range = kzalloc(sizeof(*range), GFP_KERNEL);
+
+	if (!range)
+		return ERR_PTR(-ENOMEM);
+
+	kref_init(&range->refcount);
+	range->gpusvm = gpusvm;
+	range->notifier = notifier;
+	range->va.start = ALIGN_DOWN(fault_addr, chunk_size);
+	range->va.end = ALIGN(fault_addr + 1, chunk_size);
+	INIT_LIST_HEAD(&range->rb.entry);
+	range->notifier_seq = LONG_MAX;
+	range->flags.migrate_devmem = migrate_devmem ? 1 : 0;
+
+	return range;
+}
+
+/**
+ * drm_gpusvm_check_pages - Check pages
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @start: Start address
+ * @end: End address
+ *
+ * Check if pages between start and end have been faulted in on the

CPU. Use to
+ * prevent migration of pages without CPU backing store.
+ *
+ * Returns:
+ * True if pages have been faulted into CPU, False otherwise
+ */

(quoted diff) +2 -0
+static bool drm_gpusvm_check_pages(struct drm_gpusvm *gpusvm,
+				   struct drm_gpusvm_notifier

*notifier,

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+				   u64 start, u64 end)
+{
+	struct hmm_range hmm_range = {
+		.default_flags = 0,
+		.notifier = &notifier->notifier,
+		.start = start,
+		.end = end,
+		.dev_private_owner = gpusvm-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+	};
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+	unsigned long *pfns;
+	unsigned long npages = npages_in_range(start, end);
+	int err, i;
+
+	mmap_assert_locked(gpusvm->mm);
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL);
+	if (!pfns)
+		return false;
+
+	hmm_range.notifier_seq = mmu_interval_read_begin(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+	hmm_range.hmm_pfns = pfns;
+
+	while (true) {
+		err = hmm_range_fault(&hmm_range);
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq =

mmu_interval_read_begin(&notifier->notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +22 -0
+			continue;
+		}
+		break;
+	}
+	if (err)
+		goto err_free;
+
+	for (i = 0; i < npages;) {
+		if (!(pfns[i] & HMM_PFN_VALID)) {
+			err = -EFAULT;
+			goto err_free;
+		}
+		i += 0x1 << hmm_pfn_to_map_order(pfns[i]);
+	}
+
+err_free:
+	kvfree(pfns);
+	return err ? false : true;
+}
+
+/**
+ * drm_gpusvm_range_chunk_size - Determine chunk size for GPU SVM

range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @vas: Pointer to the virtual memory area structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @check_pages: Flag indicating whether to check pages
+ *
+ * This function determines the chunk size for the GPU SVM range based on the
+ * fault address, GPU SVM chunk sizes, existing GPU SVM ranges, and the virtual
+ * memory area boundaries.
+ *
+ * Returns:
+ * Chunk size on success, LONG_MAX on failure.
+ */

(quoted diff) +2 -0
+static u64 drm_gpusvm_range_chunk_size(struct drm_gpusvm *gpusvm,
+				       struct drm_gpusvm_notifier

*notifier,

(quoted diff) +2 -0
+				       struct vm_area_struct *vas,
+				       u64 fault_addr, u64

gpuva_start,

(quoted diff) +1 -0
+				       u64 gpuva_end, bool

check_pages)

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+{
+	u64 start, end;
+	int i = 0;
+
+retry:
+	for (; i < gpusvm->num_chunks; ++i) {
+		start = ALIGN_DOWN(fault_addr, gpusvm-
Quoted 1 line

chunk_sizes[i]);

drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+		end = ALIGN(fault_addr + 1, gpusvm->chunk_sizes[i]);
+
+		if (start >= vas->vm_start && end <= vas->vm_end &&
+		    start >= notifier->interval.start &&
+		    end <= notifier->interval.end &&
+		    start >= gpuva_start && end <= gpuva_end)
+			break;
+	}
+
+	if (i == gpusvm->num_chunks)
+		return LONG_MAX;
+
+	/*
+	 * If allocation more than page, ensure not to overlap with

existing

drivers/gpu/drm/xe/drm_gpusvm.c new file +13 -0
+	 * ranges.
+	 */
+	if (end - start != SZ_4K) {
+		struct drm_gpusvm_range *range;
+
+		range = drm_gpusvm_range_find(notifier, start, end);
+		if (range) {
+			++i;
+			goto retry;
+		}
+
+		/*
+		 * XXX: Only create range on pages CPU has faulted

in. Without

(quoted diff) +1 -0
+		 * this check, or prefault, on BMG

'xe_exec_system_allocator --r

(quoted diff) +1 -0
+		 * process-many-malloc' fails. In the failure case,

each process

(quoted diff) +1 -0
+		 * mallocs 16k but the CPU VMA is ~128k which

results in 64k SVM

(quoted diff) +1 -0
+		 * ranges. When migrating the SVM ranges, some

processes fail in

(quoted diff) +1 -0
+		 * drm_gpusvm_migrate_to_devmem with 'migrate.cpages

!= npages'

(quoted diff) +1 -0
+		 * and then upon drm_gpusvm_range_get_pages device

pages from

(quoted diff) +1 -0
+		 * other processes are collected + faulted in which

creates all

(quoted diff) +1 -0
+		 * sorts of problems. Unsure exactly how this

happening, also

(quoted diff) +1 -0
+		 * problem goes away if 'xe_exec_system_allocator --
Quoted 9 lines

Review

Quoted 4 lines

r

(quoted diff) +1 -0
+		 * process-many-malloc' mallocs at least 64k at a

time.

(quoted diff) +1 -0
+		 */

Needs to be figured out. I think even in the system allocator case, if a user uses malloc() to allocate a GPU only buffer we'd need to support that?

I'm not understanding this comment but I do agree what is going on here needs to be figured out.

This comment is actually a bit stale - I think the above test case will pass now if ctx.check_pages is false with a retry loop triggered in GPU fault handler because of mixed pages. However it does appear the test case still finds device pages in hmm_range_fault mapped into a different process which I think should be impossible. Wondering if there is hmm / mm core bug here my test case hits? Let me page this information back and dig in here to see if I can explain what is going on better. Will take sometime but should be able to focus on this during the week

Also I think leaving in the check_pages option is a good thing. A call then can choose between 2 things:

1. Only create GPU mappings for CPU pages faulted in (ctx.check_pages = true)
2. create GPU mappings for a VMA and fault in CPU pages (ctx.check_pages = false)

If we support 2, then I think xe_svm_copy needs to be updated to clear VRAM for pages which the CPU has not faulted in.

Quoted 39 lines
Quoted 39 lines
(quoted diff) +2 -0
+		if (check_pages &&
+		    !drm_gpusvm_check_pages(gpusvm, notifier, start,

end)) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +17 -0
+			++i;
+			goto retry;
+		}
+	}
+
+	return end - start;
+}
+
+/**
+ * drm_gpusvm_range_find_or_insert - Find or insert GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @ctx: GPU SVM context
+ *
+ * This function finds or inserts a newly allocated a GPU SVM range

based on the
+ * fault address. Caller must hold a lock to protect range lookup and insertion.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range on success, ERR_PTR() on failure.
+ */

(quoted diff) +2 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, u64

fault_addr,

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+				u64 gpuva_start, u64 gpuva_end,
+				const struct drm_gpusvm_ctx *ctx)
+{
+	struct drm_gpusvm_notifier *notifier;
+	struct drm_gpusvm_range *range;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	bool notifier_alloc = false;
Quoted 9 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 962

Quoted 6 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+	u64 chunk_size;
+	int err;
+	bool migrate_devmem;
+
+	if (fault_addr < gpusvm->mm_start ||
+	    fault_addr > gpusvm->mm_start + gpusvm->mm_range) {

return ERR_PTR(-EINVAL)?

Sure.

Quoted 7 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 969

Quoted 7 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+		err = -EINVAL;
+		goto err_out;
+	}
+
+	if (!mmget_not_zero(mm)) {
+		err = -EFAULT;
+		goto err_out;

And here too.

Quoted 145 lines
Quoted 145 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+	}
+
+	notifier = drm_gpusvm_notifier_find(gpusvm, fault_addr);
+	if (!notifier) {
+		notifier = drm_gpusvm_notifier_alloc(gpusvm,

fault_addr);

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+		if (IS_ERR(notifier)) {
+			err = PTR_ERR(notifier);
+			goto err_mmunlock;
+		}
+		notifier_alloc = true;
+		err = mmu_interval_notifier_insert(&notifier-
Quoted 1 line

notifier,

(quoted diff) +1 -0
+						   mm, notifier-
Quoted 1 line

interval.start,

(quoted diff) +1 -0
+						   notifier-
Quoted 1 line

interval.end -

(quoted diff) +1 -0
+						   notifier-
Quoted 1 line

interval.start,

(quoted diff) +1 -0
+						  

&drm_gpusvm_notifier_ops);

drivers/gpu/drm/xe/drm_gpusvm.c new file +18 -0
+		if (err)
+			goto err_notifier;
+	}
+
+	mmap_read_lock(mm);
+
+	vas = vma_lookup(mm, fault_addr);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_notifier_remove;
+	}
+
+	if (!ctx->read_only && !(vas->vm_flags & VM_WRITE)) {
+		err = -EPERM;
+		goto err_notifier_remove;
+	}
+
+	range = drm_gpusvm_range_find(notifier, fault_addr,

fault_addr + 1);

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+	if (range)
+		goto out_mmunlock;
+	/*
+	 * XXX: Short-circuiting migration based on migrate_vma_*

current

(quoted diff) +1 -0
+	 * limitations. If/when migrate_vma_* add more support, this

logic will

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+	 * have to change.
+	 */
+	migrate_devmem = ctx->devmem_possible &&
+		vma_is_anonymous(vas) && !is_vm_hugetlb_page(vas);
+
+	chunk_size = drm_gpusvm_range_chunk_size(gpusvm, notifier,

vas,

(quoted diff) +1 -0
+						 fault_addr,

gpuva_start,

(quoted diff) +1 -0
+						 gpuva_end,

migrate_devmem &&

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+						 ctx->check_pages);
+	if (chunk_size == LONG_MAX) {
+		err = -EINVAL;
+		goto err_notifier_remove;
+	}
+
+	range = drm_gpusvm_range_alloc(gpusvm, notifier, fault_addr,

chunk_size,

drivers/gpu/drm/xe/drm_gpusvm.c new file +31 -0
+				       migrate_devmem);
+	if (IS_ERR(range)) {
+		err = PTR_ERR(range);
+		goto err_notifier_remove;
+	}
+
+	drm_gpusvm_range_insert(notifier, range);
+	if (notifier_alloc)
+		drm_gpusvm_notifier_insert(gpusvm, notifier);
+
+out_mmunlock:
+	mmap_read_unlock(mm);
+	mmput(mm);
+
+	return range;
+
+err_notifier_remove:
+	mmap_read_unlock(mm);
+	if (notifier_alloc)
+		mmu_interval_notifier_remove(&notifier->notifier);
+err_notifier:
+	if (notifier_alloc)
+		drm_gpusvm_notifier_free(gpusvm, notifier);
+err_mmunlock:
+	mmput(mm);
+err_out:
+	return ERR_PTR(err);
+}
+
+/**
+ * __drm_gpusvm_range_unmap_pages - Unmap pages associated with a

GPU SVM range (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @npages: Number of pages to unmap
+ *
+ * This function unmap pages associated with a GPU SVM range. Assumes and
+ * asserts correct locking is in place when called.
+ */
+static void __drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,

(quoted diff) +1 -0
+					   struct drm_gpusvm_range

*range,

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+					   unsigned long npages)
+{
+	unsigned long i, j;
+	struct drm_pagemap *dpagemap = range->dpagemap;
+	struct device *dev = gpusvm->drm->dev;
+
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	if (range->flags.has_dma_mapping) {
+		for (i = 0, j = 0; i < npages; j++) {
+			struct drm_pagemap_dma_addr *addr = &range-
Quoted 1 line

dma_addr[j];

(quoted diff) +2 -0
+
+			if (addr->proto == DRM_INTERCONNECT_SYSTEM)

{

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+				dma_unmap_page(dev,
+					       addr->addr,
+					       PAGE_SIZE << addr-
Quoted 1 line

order,

(quoted diff) +2 -0
+					       addr->dir);
+			} else if (dpagemap && dpagemap->ops-
Quoted 1 line

unmap_dma) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +13 -0
+				dpagemap->ops->unmap_dma(dpagemap,
+							 dev,
+							 *addr);
+			}
+			i += 1 << addr->order;
+		}
+		range->flags.has_devmem_pages = false;
+		range->flags.has_dma_mapping = false;
+		range->dpagemap = NULL;
+	}
+}
+
+/**
Quoted 9 lines
Quoted 6 lines

+ * drm_gpusvm_range_free_pages - Free pages associated with a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function free pages associated with a GPU SVM range.

Frees the dma address array

Yes.

Quoted 320 lines
Quoted 320 lines

+ */

(quoted diff) +2 -0
+static void drm_gpusvm_range_free_pages(struct drm_gpusvm *gpusvm,
+					struct drm_gpusvm_range

*range)

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+{
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	if (range->dma_addr) {
+		kvfree(range->dma_addr);
+		range->dma_addr = NULL;
+	}
+}
+
+/**
+ * drm_gpusvm_range_remove - Remove GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range to be removed
+ *
+ * This function removes the specified GPU SVM range and also

removes the parent
+ * GPU SVM notifier if no more ranges remain in the notifier. The caller must
+ * hold a lock to protect range and notifier removal.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range)
+{
+	unsigned long npages = npages_in_range(range->va.start,

range->va.end);

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+	struct drm_gpusvm_notifier *notifier;
+
+	notifier = drm_gpusvm_notifier_find(gpusvm, range-
Quoted 1 line

va.start);

drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+	if (WARN_ON_ONCE(!notifier))
+		return;
+
+	drm_gpusvm_notifier_lock(gpusvm);
+	__drm_gpusvm_range_unmap_pages(gpusvm, range, npages);
+	drm_gpusvm_range_free_pages(gpusvm, range);
+	__drm_gpusvm_range_remove(notifier, range);
+	drm_gpusvm_notifier_unlock(gpusvm);
+
+	drm_gpusvm_range_put(range);
+
+	if (RB_EMPTY_ROOT(&notifier->root.rb_root)) {
+		if (!notifier->flags.removed)
+			mmu_interval_notifier_remove(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+		drm_gpusvm_notifier_remove(gpusvm, notifier);
+		drm_gpusvm_notifier_free(gpusvm, notifier);
+	}
+}
+
+/**
+ * drm_gpusvm_range_get - Get a reference to GPU SVM range
+ * @range: Pointer to the GPU SVM range
+ *
+ * This function increments the reference count of the specified GPU

SVM range.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range)
+{
+	kref_get(&range->refcount);
+
+	return range;
+}
+
+/**
+ * drm_gpusvm_range_destroy - Destroy GPU SVM range
+ * @refcount: Pointer to the reference counter embedded in the GPU

SVM range
+ *
+ * This function destroys the specified GPU SVM range when its reference count
+ * reaches zero. If a custom range-free function is provided, it is invoked to
+ * free the range; otherwise, the range is deallocated using kfree().
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_range_destroy(struct kref *refcount)
+{
+	struct drm_gpusvm_range *range =
+		container_of(refcount, struct drm_gpusvm_range,

refcount);

drivers/gpu/drm/xe/drm_gpusvm.c new file +13 -0
+	struct drm_gpusvm *gpusvm = range->gpusvm;
+
+	if (gpusvm->ops->range_free)
+		gpusvm->ops->range_free(range);
+	else
+		kfree(range);
+}
+
+/**
+ * drm_gpusvm_range_put - Put a reference to GPU SVM range
+ * @range: Pointer to the GPU SVM range
+ *
+ * This function decrements the reference count of the specified GPU

SVM range
+ * and frees it when the count reaches zero.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range)
+{
+	kref_put(&range->refcount, drm_gpusvm_range_destroy);
+}
+
+/**
+ * drm_gpusvm_range_pages_valid - GPU SVM range pages valid
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid.

Expected be
+ * called holding gpusvm->notifier_lock and as the last step before commiting a
+ * GPU binding.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range)
+{
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	return range->flags.has_devmem_pages || range-
Quoted 1 line

flags.has_dma_mapping;

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+}
+
+/**
+ * drm_gpusvm_range_pages_valid_unlocked - GPU SVM range pages valid

unlocked
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid. Expected be
+ * called without holding gpusvm->notifier_lock.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+static bool
+drm_gpusvm_range_pages_valid_unlocked(struct drm_gpusvm *gpusvm,
+				      struct drm_gpusvm_range

*range)

drivers/gpu/drm/xe/drm_gpusvm.c new file +22 -0
+{
+	bool pages_valid;
+
+	if (!range->dma_addr)
+		return false;
+
+	drm_gpusvm_notifier_lock(gpusvm);
+	pages_valid = drm_gpusvm_range_pages_valid(gpusvm, range);
+	if (!pages_valid)
+		drm_gpusvm_range_free_pages(gpusvm, range);
+	drm_gpusvm_notifier_unlock(gpusvm);
+
+	return pages_valid;
+}
+
+/**
+ * drm_gpusvm_range_get_pages - Get pages for a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function gets pages for a GPU SVM range and ensures they are

mapped for
+ * DMA access.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	struct mmu_interval_notifier *notifier = &range->notifier-
Quoted 1 line

notifier;

(quoted diff) +2 -0
+	struct hmm_range hmm_range = {
+		.default_flags = HMM_PFN_REQ_FAULT | (ctx->read_only

? 0 :

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+			HMM_PFN_REQ_WRITE),
+		.notifier = notifier,
+		.start = range->va.start,
+		.end = range->va.end,
+		.dev_private_owner = gpusvm-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	struct drm_gpusvm_zdd *zdd;
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

(quoted diff) +2 -0
+	unsigned long i, j;
+	unsigned long npages = npages_in_range(range->va.start,

range->va.end);

drivers/gpu/drm/xe/drm_gpusvm.c new file +33 -0
+	unsigned long num_dma_mapped;
+	unsigned int order = 0;
+	unsigned long *pfns;
+	struct page **pages;
+	int err = 0;
+	struct dev_pagemap *pagemap;
+	struct drm_pagemap *dpagemap;
+
+retry:
+	hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
+	if (drm_gpusvm_range_pages_valid_unlocked(gpusvm, range))
+		goto set_seqno;
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL);
+	if (!pfns)
+		return -ENOMEM;
+
+	if (!mmget_not_zero(mm)) {
+		err = -EFAULT;
+		goto err_out;
+	}
+
+	hmm_range.hmm_pfns = pfns;
+	while (true) {
+		mmap_read_lock(mm);
+		err = hmm_range_fault(&hmm_range);
+		mmap_read_unlock(mm);
+
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq =

mmu_interval_read_begin(notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +17 -0
+			continue;
+		}
+		break;
+	}
+	mmput(mm);
+	if (err)
+		goto err_free;
+
+	pages = (struct page **)pfns;
+map_pages:
+	/*
+	 * Perform all dma mappings under the notifier lock to not
+	 * access freed pages. A notifier will either block on
+	 * the notifier lock or unmap dma.
+	 */
+	drm_gpusvm_notifier_lock(gpusvm);
+	if (mmu_interval_read_retry(notifier,

hmm_range.notifier_seq)) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+		drm_gpusvm_notifier_unlock(gpusvm);
+		goto retry;
+	}
+
+	if (!range->dma_addr) {
+		/* Unlock and restart mapping to allocate memory. */
+		drm_gpusvm_notifier_unlock(gpusvm);
+		range->dma_addr = kvmalloc_array(npages,

sizeof(*range->dma_addr),

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+						 GFP_KERNEL);
+		if (!range->dma_addr) {
+			err = -ENOMEM;
+			goto err_free;
+		}
+		goto map_pages;
+	}
+
+	zdd = NULL;
+	num_dma_mapped = 0;
+	for (i = 0, j = 0; i < npages; ++j) {
+		struct page *page = hmm_pfn_to_page(pfns[i]);
+
+		order = hmm_pfn_to_map_order(pfns[i]);
+		if (is_device_private_page(page) ||

is_device_coherent_page(page)) {

(quoted diff) +1 -0
+			if (zdd != page->zone_device_data && i > 0)

{

drivers/gpu/drm/xe/drm_gpusvm.c new file +12 -0
+				err = -EOPNOTSUPP;
+				goto err_unmap;
+			}
+			zdd = page->zone_device_data;
+			if (pagemap != page->pgmap) {
+				if (i > 0) {
+					err = -EOPNOTSUPP;
+					goto err_unmap;
+				}
+
+				pagemap = page->pgmap;
+				dpagemap = zdd->devmem_allocation-
Quoted 1 line

dpagemap;

(quoted diff) +1 -0
+				if (drm_WARN_ON(gpusvm->drm,

!dpagemap)) {

(quoted diff) +2 -0
+					/*
+					 * Raced. This is not

supposed to happen

(quoted diff) +1 -0
+					 * since hmm_range_fault()

should've migrated

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+					 * this page to system.
+					 */
+					err = -EAGAIN;
+					goto err_unmap;
+				}
+			}
+			range->dma_addr[j] =
+				dpagemap->ops->map_dma(dpagemap,

gpusvm->drm->dev,

drivers/gpu/drm/xe/drm_gpusvm.c new file +2 -0
+						       page, order,
+						      

DMA_BIDIRECTIONAL);

(quoted diff) +1 -0
+			if (dma_mapping_error(gpusvm->drm->dev,

range->dma_addr[j].addr)) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+				err = -EFAULT;
+				goto err_unmap;
+			}
+
+			pages[i] = page;
Quoted 9 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 1385

Quoted 5 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+		} else {
+			dma_addr_t addr;
+
+			if (is_zone_device_page(page) || zdd) {
+				err = -EOPNOTSUPP;

I suppose before merging we want to support mixed ranges since migration is best effort only, or what are the plans here?

I'd say initial merge no mixed support given that adds complexity and the current code is very stable - i.e., get in a simple and stable baseline and then build complexity on top incrementally. I have a lot perf optimization I'd like to get in but omitting for now to stick to the aforementioned plan.

Longtern I think a drm_gpusvm_ctx argument will control if we want mixed mappings within a range.

Quoted 77 lines
Quoted 77 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+				goto err_unmap;
+			}
+
+			addr = dma_map_page(gpusvm->drm->dev,
+					    page, 0,
+					    PAGE_SIZE << order,
+					    DMA_BIDIRECTIONAL);
+			if (dma_mapping_error(gpusvm->drm->dev,

addr)) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+				err = -EFAULT;
+				goto err_unmap;
+			}
+
+			range->dma_addr[j] =

drm_pagemap_dma_addr_encode

(quoted diff) +1 -0
+				(addr, DRM_INTERCONNECT_SYSTEM,

order,

drivers/gpu/drm/xe/drm_gpusvm.c new file +21 -0
+				 DMA_BIDIRECTIONAL);
+		}
+		i += 1 << order;
+		num_dma_mapped = i;
+	}
+
+	range->flags.has_dma_mapping = true;
+	if (zdd) {
+		range->flags.has_devmem_pages = true;
+		range->dpagemap = dpagemap;
+	}
+
+	drm_gpusvm_notifier_unlock(gpusvm);
+	kvfree(pfns);
+set_seqno:
+	range->notifier_seq = hmm_range.notifier_seq;
+
+	return 0;
+
+err_unmap:
+	__drm_gpusvm_range_unmap_pages(gpusvm, range,

num_dma_mapped);

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+	drm_gpusvm_notifier_unlock(gpusvm);
+err_free:
+	kvfree(pfns);
+err_out:
+	if (err == -EAGAIN)
+		goto retry;
+	return err;
+}
+
+/**
+ * drm_gpusvm_range_unmap_pages - Unmap pages associated with a GPU

SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed that gpusvm->notifier_lock is held in write mode; if it
+ * is clear, it acquires gpusvm->notifier_lock in read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops-

Quoted 1 line

invalidate for IOMMU

+ * security model.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx)
+{
+	unsigned long npages = npages_in_range(range->va.start,

range->va.end);

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+
+	if (ctx->in_notifier)
+		lockdep_assert_held_write(&gpusvm->notifier_lock);
+	else
+		drm_gpusvm_notifier_lock(gpusvm);
+
Quoted 9 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 1457

Quoted 5 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+	__drm_gpusvm_range_unmap_pages(gpusvm, range, npages);
+
+	if (!ctx->in_notifier)
+		drm_gpusvm_notifier_unlock(gpusvm);
+}

NIT: Separate functions for locked / unlocked makes life easier for static code analyzers.

Willl do.

Quoted 3 lines

Section below I think should belong to drm_pagemap.c

Diagree. See my comments on zdd above. Also drm_gpusvm_migration_put_pages uses migration pfns which definitely should not be in drm_pagemap.c.

Quoted 97 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 1465

Quoted 8 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+
+/**
+ * drm_gpusvm_migration_put_page - Put a migration page
+ * @page: Pointer to the page to put
+ *
+ * This function unlocks and puts a page.
+ */
+static void drm_gpusvm_migration_put_page(struct page *page)

_unlock_put_page()?

Quoted 86 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+{
+	unlock_page(page);
+	put_page(page);
+}
+
+/**
+ * drm_gpusvm_migration_put_pages - Put migration pages
+ * @npages: Number of pages
+ * @migrate_pfn: Array of migrate page frame numbers
+ *
+ * This function puts an array of pages.
+ */
+static void drm_gpusvm_migration_put_pages(unsigned long npages,
+					   unsigned long

*migrate_pfn)

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!migrate_pfn[i])
+			continue;
+
+		drm_gpusvm_migration_put_page(migrate_pfn_to_page(mi

grate_pfn[i]));

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+		migrate_pfn[i] = 0;
+	}
+}
+
+/**
+ * drm_gpusvm_get_devmem_page - Get a reference to a device memory

page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU SVM zone
+ * device data and initializes it for zone device usage.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+static void drm_gpusvm_get_devmem_page(struct page *page,
+				     struct drm_gpusvm_zdd *zdd)
+{
+	page->zone_device_data = drm_gpusvm_zdd_get(zdd);
+	zone_device_page_init(page);
+}
+
+/**
+ * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM

migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Return: 0 on success, -EFAULT if an error occurs during mapping.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+static int drm_gpusvm_migrate_map_pages(struct device *dev,
+					dma_addr_t *dma_addr,
+					long unsigned int

*migrate_pfn,

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+					unsigned long npages,
+					enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page =

migrate_pfn_to_page(migrate_pfn[i]);

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+
+		if (!page)
+			continue;
+
+		if (WARN_ON_ONCE(is_zone_device_page(page)))
+			return -EFAULT;
+
+		dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE,

dir);

Quoted 9 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 1545

Quoted 6 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+		if (dma_mapping_error(dev, dma_addr[i]))
+			return -EFAULT;
+	}
+
+	return 0;
+}

TBC'd

Thanks for the comments!

Matt

Quoted 2 lines

/Thomas

#38 lore raw

Re: [PATCH v2 05/29] drm/gpusvm: Add support for GPU Shared Virtual Memory

Quoted 169 lines · Matthew Brost
Quoted 40 lines · Thomas Hellström

On Tue, 2024-10-15 at 20:24 -0700, Matthew Brost wrote:

Continued review.

Quoted 33 lines

+/**
+ * struct drm_gpusvm_zdd - GPU SVM zone device data
+ *
+ * @refcount: Reference count for the zdd
+ * @destroy_work: Work structure for asynchronous zdd destruction
+ * @devmem_allocation: device memory allocation
+ * @device_private_page_owner: Device private pages owner
+ *
+ * This structure serves as a generic wrapper installed in
+ * page->zone_device_data. It provides infrastructure for looking up
a device
+ * memory allocation upon CPU page fault and asynchronously releasing device
+ * memory once the CPU has no page references. Asynchronous release
is useful
+ * because CPU page references can be dropped in IRQ contexts, while
releasing
+ * device memory likely requires sleeping locks.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+struct drm_gpusvm_zdd {
+	struct kref refcount;
+	struct work_struct destroy_work;
+	struct drm_gpusvm_devmem *devmem_allocation;
+	void *device_private_page_owner;
+};
+
+/**
+ * drm_gpusvm_zdd_destroy_work_func - Work function for
Quoted 2 lines

destroying a
zdd

Quoted 9 lines

NIT: Even if the above kerneldoc format works, I keep trying to enforce
using () after function names and function-like macros, like described
here: https://docs.kernel.org/doc-guide/kernel-doc.html Could we update? Also that doc calls for using "Return:" instead of "Returns:".

Will fix up. Thanks for the ref.

Quoted 103 lines
Quoted 98 lines

+ * @w: Pointer to the work_struct
+ *
+ * This function releases device memory, puts GPU SVM range, and frees zdd.
+ */
+static void drm_gpusvm_zdd_destroy_work_func(struct work_struct *w)

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+{
+	struct drm_gpusvm_zdd *zdd =
+		container_of(w, struct drm_gpusvm_zdd,

destroy_work);

(quoted diff) +1 -0
+	const struct drm_gpusvm_devmem_ops *ops = zdd-
Quoted 1 line

devmem_allocation ?

drivers/gpu/drm/xe/drm_gpusvm.c new file +12 -0
+		zdd->devmem_allocation->ops : NULL;
+
+	if (zdd->devmem_allocation && ops->devmem_release)
+		ops->devmem_release(zdd->devmem_allocation);
+	kfree(zdd);
+}
+
+/**
+ * drm_gpusvm_zdd_alloc - Allocate a zdd structure.
+ * @device_private_page_owner: Device private pages owner
+ *
+ * This function allocates and initializes a new zdd structure.

It
sets up the
+ * reference count and initializes the destroy work.
+ *
+ * Returns:
+ * Pointer to the allocated zdd on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+static struct drm_gpusvm_zdd *
+drm_gpusvm_zdd_alloc(void *device_private_page_owner)
+{
+	struct drm_gpusvm_zdd *zdd;
+
+	zdd = kmalloc(sizeof(*zdd), GFP_KERNEL);
+	if (!zdd)
+		return NULL;
+
+	kref_init(&zdd->refcount);
+	INIT_WORK(&zdd->destroy_work,

drm_gpusvm_zdd_destroy_work_func);

(quoted diff) +2 -0
+	zdd->devmem_allocation = NULL;
+	zdd->device_private_page_owner =

device_private_page_owner;

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+
+	return zdd;
+}
+
+/**
+ * drm_gpusvm_zdd_get - Get a reference to a zdd structure.
+ * @zdd: Pointer to the zdd structure.
+ *
+ * This function increments the reference count of the provided

zdd
structure.
+ *
+ * Returns: Pointer to the zdd structure.
+ */
+static struct drm_gpusvm_zdd *drm_gpusvm_zdd_get(struct drm_gpusvm_zdd *zdd)

drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+{
+	kref_get(&zdd->refcount);
+	return zdd;
+}
+
+/**
+ * drm_gpusvm_zdd_destroy - Destroy a zdd structure.
+ * @ref: Pointer to the reference count structure.
+ *
+ * This function queues the destroy_work of the zdd for

asynchronous
destruction.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_zdd_destroy(struct kref *ref)
+{
+	struct drm_gpusvm_zdd *zdd =
+		container_of(ref, struct drm_gpusvm_zdd,

refcount);

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+
+	if (zdd->devmem_allocation)
+		WRITE_ONCE(zdd->devmem_allocation->detached,

true);

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+	schedule_work(&zdd->destroy_work);
+}
+
+/**
+ * drm_gpusvm_zdd_put - Put a zdd reference.
+ * @zdd: Pointer to the zdd structure.
+ *
+ * This function decrements the reference count of the provided

zdd
structure
+ * and schedules its destruction if the count drops to zero.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c line 389

Quoted 4 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_zdd_put(struct drm_gpusvm_zdd *zdd)
+{
+	kref_put(&zdd->refcount, drm_gpusvm_zdd_destroy);
+}
Quoted 8 lines

As mentioned earlier, I think the above drm_gpusvm_zdd functions should
move to drm_pagemap.c. I don't think they are used in drm_gpusvm other
than to, at get_pages time, ensure all device private pages are from
the same pagemap?

The are used in __drm_gpusvm_migrate_to_ram to find devmem_allocation and
associated ops.

Quoted 6 lines

Also in drm_gpusvm_migrate_to_ram to find the size and device_private_page_owner.

I think the placement here is correct for now but open to shuffling this around
in the future if this makes sense.

Yeah I was thinking with the split in the multi-device series, (which is yet to be posted, though, the drm_pagemap op (*populate_mm)() would in effect handle all migration to vram, and the dev_pagemap op would handle migration to ram, and the ranges would no longer keep track of the vram allocations.

This means that no low-level migration function would take drm_gpusvm as an argument (well drm_gpusvm_range_evict() would, but that would, as we discussed before probably want to evict *all* device private pages, so that it would likely need to be implemented with hmm_range_fault()?

So this was mostly trying to avoid future shuffling around, but agree it depends on dev_pagemap patches.

Quoted 70 lines
Quoted 63 lines
Quoted 63 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+
+/**
+ * drm_gpusvm_range_find - Find GPU SVM range from GPU SVM

notifier
+ * @notifier: Pointer to the GPU SVM notifier structure.
+ * @start: Start address of the range
+ * @end: End address of the range
+ *
+ * Return: A pointer to the drm_gpusvm_range if found or NULL
+ */

(quoted diff) +2 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, u64

start, u64 end)

(quoted diff) +2 -0
+{
+	return range_iter_first(&notifier->root, start, end -

1);

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+}
+
+/**
+ * drm_gpusvm_for_each_range_safe - Safely iterate over GPU SVM

ranges in a notifier
+ * @range__: Iterator variable for the ranges
+ * @next__: Iterator variable for the ranges temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier
while
+ * removing ranges from it.
+ */
+#define drm_gpusvm_for_each_range_safe(range__, next__, notifier__,
start__, end__) \

(quoted diff) +1 -0
+	for ((range__) = drm_gpusvm_range_find((notifier__),

(start__), (end__)), \

(quoted diff) +1 -0
+	     (next__) =

__drm_gpusvm_range_next(range__); \

(quoted diff) +1 -0
+	     (range__) && (range__->va.start <

(end__)); \

(quoted diff) +1 -0
+	     (range__) = (next__), (next__) =

__drm_gpusvm_range_next(range__))

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+
+/**
+ * __drm_gpusvm_notifier_next - get the next drm_gpusvm_notifier

in
the list
+ * @notifier: a pointer to the current drm_gpusvm_notifier
+ *
+ * Return: A pointer to the next drm_gpusvm_notifier if available,
or NULL if
+ *         the current notifier is the last one or if the input notifier is
+ *         NULL.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+static struct drm_gpusvm_notifier *
+__drm_gpusvm_notifier_next(struct drm_gpusvm_notifier *notifier)
+{
+	if (notifier && !list_is_last(&notifier->rb.entry,
+				      &notifier->gpusvm-
Quoted 6 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 435

Quoted 2 lines
Quoted 1 line

notifier_list))

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+		return list_next_entry(notifier, rb.entry);

Why aren't we using notifier_iter_next() here? Then the linked list could be skipped.

Quoted 3 lines

I shamlessly copied this from GPU VM. I think the list is useful for faster
iteration and safe removal of items while walking.

We havehttps://elixir.bootlin.com/linux/v6.12-rc6/source/include/linux/interval_tree_generic.h#L24

to relate to. Now GPUVM can't use the generic version since it needs u64 intervals. These trees need unsigned long only so it should be ok. And safe removal, isn't that possible to implement without the list? Then it's really only the linked list as a perf optimization I guess, but we have a lot of those pending...

Quoted 34 lines
Quoted 22 lines
Quoted 22 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+
+	return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_notifier - Iterate over GPU SVM notifiers

in
a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm.
+ */
+#define drm_gpusvm_for_each_notifier(notifier__, gpusvm__, start__,
end__) \

(quoted diff) +1 -0
+	for ((notifier__) = notifier_iter_first(&(gpusvm__)-
Quoted 1 line

root,

Quoted 11 lines
Quoted 6 lines

(start__), (end__) - 1); \

(quoted diff) +1 -0
+	     (notifier__) && (notifier__->interval.start <

(end__)); \

(quoted diff) +1 -0
+	     (notifier__) =

__drm_gpusvm_notifier_next(notifier__))
+

Looks like end__ is not honored except for the first iteration. Relates
to the above question.

Quoted 5 lines

Again shameless copy from GPU VM... Missing what the problem is. The condition
to break the loop is:

'(notifier__) && (notifier__->interval.start < (end__)'

Ah yes, you're right. I missed that.

Quoted 641 lines
Quoted 20 lines
Quoted 20 lines

+/**
+ * drm_gpusvm_for_each_notifier_safe - Safely iterate over GPU SVM
notifiers in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @next__: Iterator variable for the notifiers temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm
while
+ * removing notifiers from it.
+ */
+#define drm_gpusvm_for_each_notifier_safe(notifier__, next__,
gpusvm__, start__, end__) \

(quoted diff) +1 -0
+	for ((notifier__) = notifier_iter_first(&(gpusvm__)-
Quoted 1 line

root,

Quoted 10 lines
Quoted 7 lines

(start__), (end__) - 1), \

(quoted diff) +1 -0
+	     (next__) =

__drm_gpusvm_notifier_next(notifier__); \

(quoted diff) +1 -0
+	     (notifier__) && (notifier__->interval.start <

(end__)); \

(quoted diff) +1 -0
+	     (notifier__) = (next__), (next__) =

__drm_gpusvm_notifier_next(notifier__))

Same here.

Alsp present:

 (notifier__) && (notifier__->interval.start < (end__)

Quoted 152 lines
Quoted 143 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+
+/**
+ * drm_gpusvm_notifier_invalidate - Invalidate a GPU SVM

notifier.
+ * @mni: Pointer to the mmu_interval_notifier structure.
+ * @mmu_range: Pointer to the mmu_notifier_range structure.
+ * @cur_seq: Current sequence number.
+ *
+ * This function serves as a generic MMU notifier for GPU SVM. It
sets the MMU
+ * notifier sequence number and calls the driver invalidate vfunc
under
+ * gpusvm->notifier_lock.
+ *
+ * Returns:
+ * true if the operation succeeds, false otherwise.
+ */

(quoted diff) +2 -0
+static bool
+drm_gpusvm_notifier_invalidate(struct mmu_interval_notifier

*mni,

(quoted diff) +1 -0
+			       const struct mmu_notifier_range

*mmu_range,

drivers/gpu/drm/xe/drm_gpusvm.c new file +19 -0
+			       unsigned long cur_seq)
+{
+	struct drm_gpusvm_notifier *notifier =
+		container_of(mni, typeof(*notifier), notifier);
+	struct drm_gpusvm *gpusvm = notifier->gpusvm;
+
+	if (!mmu_notifier_range_blockable(mmu_range))
+		return false;
+
+	down_write(&gpusvm->notifier_lock);
+	mmu_interval_set_seq(mni, cur_seq);
+	gpusvm->ops->invalidate(gpusvm, notifier, mmu_range);
+	up_write(&gpusvm->notifier_lock);
+
+	return true;
+}
+
+/**
+ * drm_gpusvm_notifier_ops - MMU interval notifier operations

for
GPU SVM
+ */
+static const struct mmu_interval_notifier_ops
drm_gpusvm_notifier_ops = {

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+	.invalidate = drm_gpusvm_notifier_invalidate,
+};
+
+/**
+ * drm_gpusvm_init - Initialize the GPU SVM.
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @name: Name of the GPU SVM.
+ * @drm: Pointer to the DRM device structure.
+ * @mm: Pointer to the mm_struct for the address space.
+ * @device_private_page_owner: Device private pages owner.
+ * @mm_start: Start address of GPU SVM.
+ * @mm_range: Range of the GPU SVM.
+ * @notifier_size: Size of individual notifiers.
+ * @ops: Pointer to the operations structure for GPU SVM.
+ * @chunk_sizes: Pointer to the array of chunk sizes used in

range
allocation.
+ *               Entries should be powers of 2 in descending order
with last
+ *               entry being SZ_4K.
+ * @num_chunks: Number of chunks.
+ *
+ * This function initializes the GPU SVM.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void

*device_private_page_owner,

(quoted diff) +1 -0
+		    u64 mm_start, u64 mm_range, u64

notifier_size,

drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+		    const struct drm_gpusvm_ops *ops,
+		    const u64 *chunk_sizes, int num_chunks)
+{
+	if (!ops->invalidate || !num_chunks)
+		return -EINVAL;
+
+	gpusvm->name = name;
+	gpusvm->drm = drm;
+	gpusvm->mm = mm;
+	gpusvm->device_private_page_owner =

device_private_page_owner;

drivers/gpu/drm/xe/drm_gpusvm.c new file +26 -0
+	gpusvm->mm_start = mm_start;
+	gpusvm->mm_range = mm_range;
+	gpusvm->notifier_size = notifier_size;
+	gpusvm->ops = ops;
+	gpusvm->chunk_sizes = chunk_sizes;
+	gpusvm->num_chunks = num_chunks;
+
+	mmgrab(mm);
+	gpusvm->root = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&gpusvm->notifier_list);
+
+	init_rwsem(&gpusvm->notifier_lock);
+
+	fs_reclaim_acquire(GFP_KERNEL);
+	might_lock(&gpusvm->notifier_lock);
+	fs_reclaim_release(GFP_KERNEL);
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_notifier_find - Find GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure
+ * @fault_addr__: Fault address
+ *
+ * This macro finds the GPU SVM notifier associated with the

fault
address.
+ *
+ * Returns:
+ * Pointer to the GPU SVM notifier on success, NULL otherwise.
+ */
+#define drm_gpusvm_notifier_find(gpusvm__,
fault_addr__) \

(quoted diff) +1 -0
+	notifier_iter_first(&(gpusvm__)->root,

(fault_addr__), \

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+			    (fault_addr__ + 1))
+
+/**
+ * to_drm_gpusvm_notifier - retrieve the container struct for a

given rbtree node
+ * @node__: a pointer to the rbtree node embedded within a drm_gpusvm_notifier struct
+ *
+ * Return: A pointer to the containing drm_gpusvm_notifier structure.
+ */
+#define
to_drm_gpusvm_notifier(__node) \

(quoted diff) +1 -0
+	container_of((__node), struct drm_gpusvm_notifier,
Quoted 2 lines

rb.node)
+

There appears to be a number of function-like macros in the code, which
look like they can be converted to functions. Linux prefers functions
over macros when possible:

Quoted 2 lines

https://www.kernel.org/doc/html/v5.8/process/coding-style.html#macros-enums-and-rtl

Will convert all macros to functions if possible. Again thanks for ref.

Quoted 43 lines
Quoted 43 lines

+/**
+ * drm_gpusvm_notifier_insert - Insert GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function inserts the GPU SVM notifier into the GPU SVM RB
tree and list.
+ */
+static void drm_gpusvm_notifier_insert(struct drm_gpusvm *gpusvm,

(quoted diff) +1 -0
+				       struct

drm_gpusvm_notifier
*notifier)

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+{
+	struct rb_node *node;
+	struct list_head *head;
+
+	notifier_insert(notifier, &gpusvm->root);
+
+	node = rb_prev(&notifier->rb.node);
+	if (node)
+		head = &(to_drm_gpusvm_notifier(node))-
Quoted 1 line

rb.entry;

drivers/gpu/drm/xe/drm_gpusvm.c new file +12 -0
+	else
+		head = &gpusvm->notifier_list;
+
+	list_add(&notifier->rb.entry, head);
+}
+
+/**
+ * drm_gpusvm_notifier_remove - Remove GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM tructure
+ * @notifier__: Pointer to the GPU SVM notifier structure
+ *
+ * This macro removes the GPU SVM notifier from the GPU SVM RB

tree
and list.
+ */
+#define drm_gpusvm_notifier_remove(gpusvm__,
notifier__) \

(quoted diff) +1 -0
+	notifier_remove((notifier__), &(gpusvm__)-
Quoted 6 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 622

Quoted 2 lines
Quoted 1 line

root); \

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+	list_del(&(notifier__)->rb.entry)

Unless this can be made a function, Pls use
do { } while (0)

I think it can be made a function or otherwise yea will use do { } while (0).

Quoted 154 lines
Quoted 154 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+
+/**
+ * drm_gpusvm_fini - Finalize the GPU SVM.
+ * @gpusvm: Pointer to the GPU SVM structure.
+ *
+ * This function finalizes the GPU SVM by cleaning up any

remaining
ranges and
+ * notifiers, and dropping a reference to struct MM.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm)
+{
+	struct drm_gpusvm_notifier *notifier, *next;
+
+	drm_gpusvm_for_each_notifier_safe(notifier, next,

gpusvm, 0,
LONG_MAX) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+		struct drm_gpusvm_range *range, *__next;
+
+		/*
+		 * Remove notifier first to avoid racing with

any
invalidation

(quoted diff) +2 -0
+		 */
+		mmu_interval_notifier_remove(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+		notifier->flags.removed = true;
+
+		drm_gpusvm_for_each_range_safe(range, __next,

notifier, 0,

drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+					       LONG_MAX)
+			drm_gpusvm_range_remove(gpusvm, range);
+	}
+
+	mmdrop(gpusvm->mm);
+	WARN_ON(!RB_EMPTY_ROOT(&gpusvm->root.rb_root));
+}
+
+/**
+ * drm_gpusvm_notifier_alloc - Allocate GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ *
+ * This function allocates and initializes the GPU SVM notifier

structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM notifier on success, ERR_PTR()
on failure.
+ */

(quoted diff) +2 -0
+static struct drm_gpusvm_notifier *
+drm_gpusvm_notifier_alloc(struct drm_gpusvm *gpusvm, u64

fault_addr)

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	if (gpusvm->ops->notifier_alloc)
+		notifier = gpusvm->ops->notifier_alloc();
+	else
+		notifier = kzalloc(sizeof(*notifier),

GFP_KERNEL);

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+
+	if (!notifier)
+		return ERR_PTR(-ENOMEM);
+
+	notifier->gpusvm = gpusvm;
+	notifier->interval.start = ALIGN_DOWN(fault_addr,

gpusvm-

Quoted 1 line

notifier_size);

(quoted diff) +1 -0
+	notifier->interval.end = ALIGN(fault_addr + 1, gpusvm-
Quoted 1 line

notifier_size);

drivers/gpu/drm/xe/drm_gpusvm.c new file +16 -0
+	INIT_LIST_HEAD(&notifier->rb.entry);
+	notifier->root = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&notifier->range_list);
+
+	return notifier;
+}
+
+/**
+ * drm_gpusvm_notifier_free - Free GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function frees the GPU SVM notifier structure.
+ */
+static void drm_gpusvm_notifier_free(struct drm_gpusvm *gpusvm,
+				     struct drm_gpusvm_notifier

*notifier)

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+{
+	WARN_ON(!RB_EMPTY_ROOT(&notifier->root.rb_root));
+
+	if (gpusvm->ops->notifier_free)
+		gpusvm->ops->notifier_free(notifier);
+	else
+		kfree(notifier);
+}
+
+/**
+ * to_drm_gpusvm_range - retrieve the container struct for a

given
rbtree node
+ * @node__: a pointer to the rbtree node embedded within a drm_gpusvm_range struct
+ *
+ * Return: A pointer to the containing drm_gpusvm_range structure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+#define to_drm_gpusvm_range(node__)	\
+	container_of((node__), struct drm_gpusvm_range, rb.node)
+
+/**
+ * drm_gpusvm_range_insert - Insert GPU SVM range
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function inserts the GPU SVM range into the notifier RB

tree
and list.
+ */
+static void drm_gpusvm_range_insert(struct drm_gpusvm_notifier *notifier,

(quoted diff) +1 -0
+				    struct drm_gpusvm_range

*range)

drivers/gpu/drm/xe/drm_gpusvm.c new file +23 -0
+{
+	struct rb_node *node;
+	struct list_head *head;
+
+	drm_gpusvm_notifier_lock(notifier->gpusvm);
+	range_insert(range, &notifier->root);
+
+	node = rb_prev(&range->rb.node);
+	if (node)
+		head = &(to_drm_gpusvm_range(node))->rb.entry;
+	else
+		head = &notifier->range_list;
+
+	list_add(&range->rb.entry, head);
+	drm_gpusvm_notifier_unlock(notifier->gpusvm);
+}
+
+/**
+ * __drm_gpusvm_range_remove - Remove GPU SVM range
+ * @notifier__: Pointer to the GPU SVM notifier structure
+ * @range__: Pointer to the GPU SVM range structure
+ *
+ * This macro removes the GPU SVM range from the notifier RB

tree
and list.
+ */
+#define __drm_gpusvm_range_remove(notifier__,
range__) \

(quoted diff) +1 -0
+	range_remove((range__), &(notifier__)-
Quoted 6 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 748

Quoted 2 lines
Quoted 1 line

root); \

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+	list_del(&(range__)->rb.entry)

Same thing as for the notifier rb tree. And do we need the linked list?

Same answer.

Quoted 222 lines
Quoted 222 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+
+/**
+ * drm_gpusvm_range_alloc - Allocate GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @fault_addr: Fault address
+ * @chunk_size: Chunk size
+ * @migrate_devmem: Flag indicating whether to migrate device

memory
+ *
+ * This function allocates and initializes the GPU SVM range structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM range on success, ERR_PTR() on
failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+static struct drm_gpusvm_range *
+drm_gpusvm_range_alloc(struct drm_gpusvm *gpusvm,
+		       struct drm_gpusvm_notifier *notifier,
+		       u64 fault_addr, u64 chunk_size, bool

migrate_devmem)

drivers/gpu/drm/xe/drm_gpusvm.c new file +31 -0
+{
+	struct drm_gpusvm_range *range;
+
+	if (gpusvm->ops->range_alloc)
+		range = gpusvm->ops->range_alloc(gpusvm);
+	else
+		range = kzalloc(sizeof(*range), GFP_KERNEL);
+
+	if (!range)
+		return ERR_PTR(-ENOMEM);
+
+	kref_init(&range->refcount);
+	range->gpusvm = gpusvm;
+	range->notifier = notifier;
+	range->va.start = ALIGN_DOWN(fault_addr, chunk_size);
+	range->va.end = ALIGN(fault_addr + 1, chunk_size);
+	INIT_LIST_HEAD(&range->rb.entry);
+	range->notifier_seq = LONG_MAX;
+	range->flags.migrate_devmem = migrate_devmem ? 1 : 0;
+
+	return range;
+}
+
+/**
+ * drm_gpusvm_check_pages - Check pages
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @start: Start address
+ * @end: End address
+ *
+ * Check if pages between start and end have been faulted in on

the
CPU. Use to
+ * prevent migration of pages without CPU backing store.
+ *
+ * Returns:
+ * True if pages have been faulted into CPU, False otherwise
+ */

(quoted diff) +2 -0
+static bool drm_gpusvm_check_pages(struct drm_gpusvm *gpusvm,
+				   struct drm_gpusvm_notifier

*notifier,

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+				   u64 start, u64 end)
+{
+	struct hmm_range hmm_range = {
+		.default_flags = 0,
+		.notifier = &notifier->notifier,
+		.start = start,
+		.end = end,
+		.dev_private_owner = gpusvm-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+	};
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+	unsigned long *pfns;
+	unsigned long npages = npages_in_range(start, end);
+	int err, i;
+
+	mmap_assert_locked(gpusvm->mm);
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns),

GFP_KERNEL);

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+	if (!pfns)
+		return false;
+
+	hmm_range.notifier_seq =

mmu_interval_read_begin(&notifier-

Quoted 1 line

notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+	hmm_range.hmm_pfns = pfns;
+
+	while (true) {
+		err = hmm_range_fault(&hmm_range);
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq =

mmu_interval_read_begin(&notifier->notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +22 -0
+			continue;
+		}
+		break;
+	}
+	if (err)
+		goto err_free;
+
+	for (i = 0; i < npages;) {
+		if (!(pfns[i] & HMM_PFN_VALID)) {
+			err = -EFAULT;
+			goto err_free;
+		}
+		i += 0x1 << hmm_pfn_to_map_order(pfns[i]);
+	}
+
+err_free:
+	kvfree(pfns);
+	return err ? false : true;
+}
+
+/**
+ * drm_gpusvm_range_chunk_size - Determine chunk size for GPU

SVM
range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @vas: Pointer to the virtual memory area structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @check_pages: Flag indicating whether to check pages
+ *
+ * This function determines the chunk size for the GPU SVM range based on the
+ * fault address, GPU SVM chunk sizes, existing GPU SVM ranges, and
the virtual
+ * memory area boundaries.
+ *
+ * Returns:
+ * Chunk size on success, LONG_MAX on failure.
+ */
+static u64 drm_gpusvm_range_chunk_size(struct drm_gpusvm *gpusvm,

(quoted diff) +1 -0
+				       struct

drm_gpusvm_notifier
*notifier,

(quoted diff) +1 -0
+				       struct vm_area_struct

*vas,

(quoted diff) +1 -0
+				       u64 fault_addr, u64

gpuva_start,

(quoted diff) +1 -0
+				       u64 gpuva_end, bool

check_pages)

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+{
+	u64 start, end;
+	int i = 0;
+
+retry:
+	for (; i < gpusvm->num_chunks; ++i) {
+		start = ALIGN_DOWN(fault_addr, gpusvm-
Quoted 1 line

chunk_sizes[i]);

(quoted diff) +1 -0
+		end = ALIGN(fault_addr + 1, gpusvm-
Quoted 1 line

chunk_sizes[i]);

(quoted diff) +2 -0
+
+		if (start >= vas->vm_start && end <= vas->vm_end

&&

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+		    start >= notifier->interval.start &&
+		    end <= notifier->interval.end &&
+		    start >= gpuva_start && end <= gpuva_end)
+			break;
+	}
+
+	if (i == gpusvm->num_chunks)
+		return LONG_MAX;
+
+	/*
+	 * If allocation more than page, ensure not to overlap

with
existing

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+	 * ranges.
+	 */
+	if (end - start != SZ_4K) {
+		struct drm_gpusvm_range *range;
+
+		range = drm_gpusvm_range_find(notifier, start,

end);

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+		if (range) {
+			++i;
+			goto retry;
+		}
+
+		/*
+		 * XXX: Only create range on pages CPU has

faulted
in. Without

(quoted diff) +1 -0
+		 * this check, or prefault, on BMG

'xe_exec_system_allocator --r

(quoted diff) +1 -0
+		 * process-many-malloc' fails. In the failure

case,
each process

(quoted diff) +1 -0
+		 * mallocs 16k but the CPU VMA is ~128k which

results in 64k SVM

(quoted diff) +1 -0
+		 * ranges. When migrating the SVM ranges, some

processes fail in

(quoted diff) +1 -0
+		 * drm_gpusvm_migrate_to_devmem with

'migrate.cpages
!= npages'

(quoted diff) +1 -0
+		 * and then upon drm_gpusvm_range_get_pages

device
pages from

(quoted diff) +1 -0
+		 * other processes are collected + faulted in

which
creates all

(quoted diff) +1 -0
+		 * sorts of problems. Unsure exactly how this

happening, also

(quoted diff) +1 -0
+		 * problem goes away if

'xe_exec_system_allocator --
r

(quoted diff) +1 -0
+		 * process-many-malloc' mallocs at least 64k at

a

Quoted 9 lines

Review

Quoted 2 lines

time.

(quoted diff) +1 -0
+		 */

Needs to be figured out. I think even in the system allocator case, if
a user uses malloc() to allocate a GPU only buffer we'd need to support
that?

Quoted 3 lines

I'm not understanding this comment but I do agree what is going on here needs to
be figured out.

What I meant was let's say the user mallocs a big buffer to be used by the gpu only, and hence should ideally be in device memory, but it's never faulted by the CPU. I guess my question should be reformulated, what would happen then? Wouldn't it remain in system ram forever?

Quoted 629 lines

This comment is actually a bit stale - I think the above test case will pass now
if ctx.check_pages is false with a retry loop triggered in GPU fault handler
because of mixed pages. However it does appear the test case still finds device
pages in hmm_range_fault mapped into a different process which I think should be
impossible. Wondering if there is hmm / mm core bug here my test case hits? Let
me page this information back and dig in here to see if I can explain what is
going on better. Will take sometime but should be able to focus on this during
the week

Also I think leaving in the check_pages option is a good thing. A call then can
choose between 2 things:

1. Only create GPU mappings for CPU pages faulted in (ctx.check_pages = true)
2. create GPU mappings for a VMA and fault in CPU pages (ctx.check_pages =
false)

If we support 2, then I think xe_svm_copy needs to be updated to clear VRAM for
pages which the CPU has not faulted in.

Quoted 45 lines
Quoted 45 lines
(quoted diff) +2 -0
+		if (check_pages &&
+		    !drm_gpusvm_check_pages(gpusvm, notifier,

start,
end)) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+			++i;
+			goto retry;
+		}
+	}
+
+	return end - start;
+}
+
+/**
+ * drm_gpusvm_range_find_or_insert - Find or insert GPU SVM

range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @ctx: GPU SVM context
+ *
+ * This function finds or inserts a newly allocated a GPU SVM range
based on the
+ * fault address. Caller must hold a lock to protect range lookup
and insertion.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range on success, ERR_PTR() on failure.
+ */

(quoted diff) +2 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, u64

fault_addr,

(quoted diff) +2 -0
+				u64 gpuva_start, u64 gpuva_end,
+				const struct drm_gpusvm_ctx

*ctx)

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+{
+	struct drm_gpusvm_notifier *notifier;
+	struct drm_gpusvm_range *range;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	bool notifier_alloc = false;
Quoted 9 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 962

Quoted 6 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+	u64 chunk_size;
+	int err;
+	bool migrate_devmem;
+
+	if (fault_addr < gpusvm->mm_start ||
+	    fault_addr > gpusvm->mm_start + gpusvm->mm_range) {

return ERR_PTR(-EINVAL)?

Sure.

Quoted 7 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 969

Quoted 7 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+		err = -EINVAL;
+		goto err_out;
+	}
+
+	if (!mmget_not_zero(mm)) {
+		err = -EFAULT;
+		goto err_out;

And here too.

Quoted 159 lines
Quoted 159 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+	}
+
+	notifier = drm_gpusvm_notifier_find(gpusvm, fault_addr);
+	if (!notifier) {
+		notifier = drm_gpusvm_notifier_alloc(gpusvm,

fault_addr);

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+		if (IS_ERR(notifier)) {
+			err = PTR_ERR(notifier);
+			goto err_mmunlock;
+		}
+		notifier_alloc = true;
+		err = mmu_interval_notifier_insert(&notifier-
Quoted 1 line

notifier,

(quoted diff) +1 -0
+						   mm, notifier-
Quoted 1 line

interval.start,

(quoted diff) +1 -0
+						   notifier-
Quoted 1 line

interval.end -

(quoted diff) +1 -0
+						   notifier-
Quoted 1 line

interval.start,

(quoted diff) +1 -0
+						  

&drm_gpusvm_notifier_ops);

drivers/gpu/drm/xe/drm_gpusvm.c new file +18 -0
+		if (err)
+			goto err_notifier;
+	}
+
+	mmap_read_lock(mm);
+
+	vas = vma_lookup(mm, fault_addr);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_notifier_remove;
+	}
+
+	if (!ctx->read_only && !(vas->vm_flags & VM_WRITE)) {
+		err = -EPERM;
+		goto err_notifier_remove;
+	}
+
+	range = drm_gpusvm_range_find(notifier, fault_addr,

fault_addr + 1);

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+	if (range)
+		goto out_mmunlock;
+	/*
+	 * XXX: Short-circuiting migration based on

migrate_vma_*
current

(quoted diff) +1 -0
+	 * limitations. If/when migrate_vma_* add more support,

this
logic will

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+	 * have to change.
+	 */
+	migrate_devmem = ctx->devmem_possible &&
+		vma_is_anonymous(vas) &&

!is_vm_hugetlb_page(vas);

(quoted diff) +2 -0
+
+	chunk_size = drm_gpusvm_range_chunk_size(gpusvm,

notifier,
vas,

(quoted diff) +1 -0
+						 fault_addr,

gpuva_start,

(quoted diff) +1 -0
+						 gpuva_end,

migrate_devmem &&

(quoted diff) +1 -0
+						 ctx-
Quoted 1 line

check_pages);

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+	if (chunk_size == LONG_MAX) {
+		err = -EINVAL;
+		goto err_notifier_remove;
+	}
+
+	range = drm_gpusvm_range_alloc(gpusvm, notifier,

fault_addr,
chunk_size,

drivers/gpu/drm/xe/drm_gpusvm.c new file +20 -0
+				       migrate_devmem);
+	if (IS_ERR(range)) {
+		err = PTR_ERR(range);
+		goto err_notifier_remove;
+	}
+
+	drm_gpusvm_range_insert(notifier, range);
+	if (notifier_alloc)
+		drm_gpusvm_notifier_insert(gpusvm, notifier);
+
+out_mmunlock:
+	mmap_read_unlock(mm);
+	mmput(mm);
+
+	return range;
+
+err_notifier_remove:
+	mmap_read_unlock(mm);
+	if (notifier_alloc)
+		mmu_interval_notifier_remove(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+err_notifier:
+	if (notifier_alloc)
+		drm_gpusvm_notifier_free(gpusvm, notifier);
+err_mmunlock:
+	mmput(mm);
+err_out:
+	return ERR_PTR(err);
+}
+
+/**
+ * __drm_gpusvm_range_unmap_pages - Unmap pages associated with

a
GPU SVM range (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @npages: Number of pages to unmap
+ *
+ * This function unmap pages associated with a GPU SVM range. Assumes and
+ * asserts correct locking is in place when called.
+ */
+static void __drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,

(quoted diff) +1 -0
+					   struct

drm_gpusvm_range
*range,

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+					   unsigned long npages)
+{
+	unsigned long i, j;
+	struct drm_pagemap *dpagemap = range->dpagemap;
+	struct device *dev = gpusvm->drm->dev;
+
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	if (range->flags.has_dma_mapping) {
+		for (i = 0, j = 0; i < npages; j++) {
+			struct drm_pagemap_dma_addr *addr =

&range-

Quoted 1 line

dma_addr[j];

(quoted diff) +2 -0
+
+			if (addr->proto ==

DRM_INTERCONNECT_SYSTEM)
{

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+				dma_unmap_page(dev,
+					       addr->addr,
+					       PAGE_SIZE <<

addr-

Quoted 1 line

order,

(quoted diff) +2 -0
+					       addr->dir);
+			} else if (dpagemap && dpagemap->ops-
Quoted 1 line

unmap_dma) {

(quoted diff) +1 -0
+				dpagemap->ops-
Quoted 1 line

unmap_dma(dpagemap,

drivers/gpu/drm/xe/drm_gpusvm.c new file +13 -0
+							 dev,
+							 *addr);
+			}
+			i += 1 << addr->order;
+		}
+		range->flags.has_devmem_pages = false;
+		range->flags.has_dma_mapping = false;
+		range->dpagemap = NULL;
+	}
+}
+
+/**
+ * drm_gpusvm_range_free_pages - Free pages associated with a
Quoted 9 lines
Quoted 6 lines

GPU
SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function free pages associated with a GPU SVM range.

Frees the dma address array

Yes.

Quoted 342 lines
Quoted 342 lines

+ */
+static void drm_gpusvm_range_free_pages(struct drm_gpusvm *gpusvm,

(quoted diff) +1 -0
+					struct drm_gpusvm_range

*range)

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+{
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	if (range->dma_addr) {
+		kvfree(range->dma_addr);
+		range->dma_addr = NULL;
+	}
+}
+
+/**
+ * drm_gpusvm_range_remove - Remove GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range to be removed
+ *
+ * This function removes the specified GPU SVM range and also

removes the parent
+ * GPU SVM notifier if no more ranges remain in the notifier. The
caller must
+ * hold a lock to protect range and notifier removal.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range)
+{
+	unsigned long npages = npages_in_range(range->va.start,

range->va.end);

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+	struct drm_gpusvm_notifier *notifier;
+
+	notifier = drm_gpusvm_notifier_find(gpusvm, range-
Quoted 1 line

va.start);

drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+	if (WARN_ON_ONCE(!notifier))
+		return;
+
+	drm_gpusvm_notifier_lock(gpusvm);
+	__drm_gpusvm_range_unmap_pages(gpusvm, range, npages);
+	drm_gpusvm_range_free_pages(gpusvm, range);
+	__drm_gpusvm_range_remove(notifier, range);
+	drm_gpusvm_notifier_unlock(gpusvm);
+
+	drm_gpusvm_range_put(range);
+
+	if (RB_EMPTY_ROOT(&notifier->root.rb_root)) {
+		if (!notifier->flags.removed)
+			mmu_interval_notifier_remove(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+		drm_gpusvm_notifier_remove(gpusvm, notifier);
+		drm_gpusvm_notifier_free(gpusvm, notifier);
+	}
+}
+
+/**
+ * drm_gpusvm_range_get - Get a reference to GPU SVM range
+ * @range: Pointer to the GPU SVM range
+ *
+ * This function increments the reference count of the specified

GPU
SVM range.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range)
+{
+	kref_get(&range->refcount);
+
+	return range;
+}
+
+/**
+ * drm_gpusvm_range_destroy - Destroy GPU SVM range
+ * @refcount: Pointer to the reference counter embedded in the

GPU
SVM range
+ *
+ * This function destroys the specified GPU SVM range when its reference count
+ * reaches zero. If a custom range-free function is provided, it is
invoked to
+ * free the range; otherwise, the range is deallocated using kfree().
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_range_destroy(struct kref *refcount)
+{
+	struct drm_gpusvm_range *range =
+		container_of(refcount, struct drm_gpusvm_range,

refcount);

drivers/gpu/drm/xe/drm_gpusvm.c new file +13 -0
+	struct drm_gpusvm *gpusvm = range->gpusvm;
+
+	if (gpusvm->ops->range_free)
+		gpusvm->ops->range_free(range);
+	else
+		kfree(range);
+}
+
+/**
+ * drm_gpusvm_range_put - Put a reference to GPU SVM range
+ * @range: Pointer to the GPU SVM range
+ *
+ * This function decrements the reference count of the specified

GPU
SVM range
+ * and frees it when the count reaches zero.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range)
+{
+	kref_put(&range->refcount, drm_gpusvm_range_destroy);
+}
+
+/**
+ * drm_gpusvm_range_pages_valid - GPU SVM range pages valid
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid.

Expected be
+ * called holding gpusvm->notifier_lock and as the last step before
commiting a
+ * GPU binding.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */

(quoted diff) +2 -0
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range

*range)

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+{
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	return range->flags.has_devmem_pages || range-
Quoted 1 line

flags.has_dma_mapping;

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+}
+
+/**
+ * drm_gpusvm_range_pages_valid_unlocked - GPU SVM range pages

valid
unlocked
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid. Expected be
+ * called without holding gpusvm->notifier_lock.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+static bool
+drm_gpusvm_range_pages_valid_unlocked(struct drm_gpusvm *gpusvm,
+				      struct drm_gpusvm_range

*range)

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+{
+	bool pages_valid;
+
+	if (!range->dma_addr)
+		return false;
+
+	drm_gpusvm_notifier_lock(gpusvm);
+	pages_valid = drm_gpusvm_range_pages_valid(gpusvm,

range);

drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+	if (!pages_valid)
+		drm_gpusvm_range_free_pages(gpusvm, range);
+	drm_gpusvm_notifier_unlock(gpusvm);
+
+	return pages_valid;
+}
+
+/**
+ * drm_gpusvm_range_get_pages - Get pages for a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function gets pages for a GPU SVM range and ensures they

are
mapped for
+ * DMA access.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	struct mmu_interval_notifier *notifier = &range-
Quoted 2 lines

notifier-
notifier;

(quoted diff) +2 -0
+	struct hmm_range hmm_range = {
+		.default_flags = HMM_PFN_REQ_FAULT | (ctx-
Quoted 1 line

read_only

? 0 :

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+			HMM_PFN_REQ_WRITE),
+		.notifier = notifier,
+		.start = range->va.start,
+		.end = range->va.end,
+		.dev_private_owner = gpusvm-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	struct drm_gpusvm_zdd *zdd;
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

(quoted diff) +2 -0
+	unsigned long i, j;
+	unsigned long npages = npages_in_range(range->va.start,

range->va.end);

drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+	unsigned long num_dma_mapped;
+	unsigned int order = 0;
+	unsigned long *pfns;
+	struct page **pages;
+	int err = 0;
+	struct dev_pagemap *pagemap;
+	struct drm_pagemap *dpagemap;
+
+retry:
+	hmm_range.notifier_seq =

mmu_interval_read_begin(notifier);

(quoted diff) +1 -0
+	if (drm_gpusvm_range_pages_valid_unlocked(gpusvm,

range))

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+		goto set_seqno;
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns),

GFP_KERNEL);

drivers/gpu/drm/xe/drm_gpusvm.c new file +19 -0
+	if (!pfns)
+		return -ENOMEM;
+
+	if (!mmget_not_zero(mm)) {
+		err = -EFAULT;
+		goto err_out;
+	}
+
+	hmm_range.hmm_pfns = pfns;
+	while (true) {
+		mmap_read_lock(mm);
+		err = hmm_range_fault(&hmm_range);
+		mmap_read_unlock(mm);
+
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq =

mmu_interval_read_begin(notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +12 -0
+			continue;
+		}
+		break;
+	}
+	mmput(mm);
+	if (err)
+		goto err_free;
+
+	pages = (struct page **)pfns;
+map_pages:
+	/*
+	 * Perform all dma mappings under the notifier lock to

not

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+	 * access freed pages. A notifier will either block on
+	 * the notifier lock or unmap dma.
+	 */
+	drm_gpusvm_notifier_lock(gpusvm);
+	if (mmu_interval_read_retry(notifier,

hmm_range.notifier_seq)) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+		drm_gpusvm_notifier_unlock(gpusvm);
+		goto retry;
+	}
+
+	if (!range->dma_addr) {
+		/* Unlock and restart mapping to allocate

memory. */

(quoted diff) +2 -0
+		drm_gpusvm_notifier_unlock(gpusvm);
+		range->dma_addr = kvmalloc_array(npages,

sizeof(*range->dma_addr),

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+						 GFP_KERNEL);
+		if (!range->dma_addr) {
+			err = -ENOMEM;
+			goto err_free;
+		}
+		goto map_pages;
+	}
+
+	zdd = NULL;
+	num_dma_mapped = 0;
+	for (i = 0, j = 0; i < npages; ++j) {
+		struct page *page = hmm_pfn_to_page(pfns[i]);
+
+		order = hmm_pfn_to_map_order(pfns[i]);
+		if (is_device_private_page(page) ||

is_device_coherent_page(page)) {

(quoted diff) +1 -0
+			if (zdd != page->zone_device_data && i >

0)
{

drivers/gpu/drm/xe/drm_gpusvm.c new file +12 -0
+				err = -EOPNOTSUPP;
+				goto err_unmap;
+			}
+			zdd = page->zone_device_data;
+			if (pagemap != page->pgmap) {
+				if (i > 0) {
+					err = -EOPNOTSUPP;
+					goto err_unmap;
+				}
+
+				pagemap = page->pgmap;
+				dpagemap = zdd-
Quoted 2 lines

devmem_allocation-
dpagemap;

(quoted diff) +1 -0
+				if (drm_WARN_ON(gpusvm->drm,

!dpagemap)) {

(quoted diff) +2 -0
+					/*
+					 * Raced. This is not

supposed to happen

(quoted diff) +1 -0
+					 * since

hmm_range_fault()
should've migrated

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+					 * this page to system.
+					 */
+					err = -EAGAIN;
+					goto err_unmap;
+				}
+			}
+			range->dma_addr[j] =
+				dpagemap->ops->map_dma(dpagemap,

gpusvm->drm->dev,

(quoted diff) +1 -0
+						       page,

order,

(quoted diff) +1 -0
+						      

DMA_BIDIRECTIONAL);

(quoted diff) +1 -0
+			if (dma_mapping_error(gpusvm->drm->dev,

range->dma_addr[j].addr)) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+				err = -EFAULT;
+				goto err_unmap;
+			}
+
+			pages[i] = page;
Quoted 9 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 1385

Quoted 5 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+		} else {
+			dma_addr_t addr;
+
+			if (is_zone_device_page(page) || zdd) {
+				err = -EOPNOTSUPP;

I suppose before merging we want to support mixed ranges since migration is best effort only, or what are the plans here?

I'd say initial merge no mixed support given that adds complexity and the
current code is very stable - i.e., get in a simple and stable baseline and then
build complexity on top incrementally. I have a lot perf optimization I'd like
to get in but omitting for now to stick to the aforementioned plan.

Quoted 3 lines

Longtern I think a drm_gpusvm_ctx argument will control if we want mixed
mappings within a range.

OK.

Quoted 98 lines

Quoted 81 lines
Quoted 81 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+				goto err_unmap;
+			}
+
+			addr = dma_map_page(gpusvm->drm->dev,
+					    page, 0,
+					    PAGE_SIZE << order,
+					    DMA_BIDIRECTIONAL);
+			if (dma_mapping_error(gpusvm->drm->dev,

addr)) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+				err = -EFAULT;
+				goto err_unmap;
+			}
+
+			range->dma_addr[j] =

drm_pagemap_dma_addr_encode

(quoted diff) +1 -0
+				(addr, DRM_INTERCONNECT_SYSTEM,

order,

drivers/gpu/drm/xe/drm_gpusvm.c new file +21 -0
+				 DMA_BIDIRECTIONAL);
+		}
+		i += 1 << order;
+		num_dma_mapped = i;
+	}
+
+	range->flags.has_dma_mapping = true;
+	if (zdd) {
+		range->flags.has_devmem_pages = true;
+		range->dpagemap = dpagemap;
+	}
+
+	drm_gpusvm_notifier_unlock(gpusvm);
+	kvfree(pfns);
+set_seqno:
+	range->notifier_seq = hmm_range.notifier_seq;
+
+	return 0;
+
+err_unmap:
+	__drm_gpusvm_range_unmap_pages(gpusvm, range,

num_dma_mapped);

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+	drm_gpusvm_notifier_unlock(gpusvm);
+err_free:
+	kvfree(pfns);
+err_out:
+	if (err == -EAGAIN)
+		goto retry;
+	return err;
+}
+
+/**
+ * drm_gpusvm_range_unmap_pages - Unmap pages associated with a

GPU
SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If
@in_notifier
+ * is set, it is assumed that gpusvm->notifier_lock is held in write
mode; if it
+ * is clear, it acquires gpusvm->notifier_lock in read mode. Must be
called on
+ * each GPU SVM range attached to notifier in gpusvm->ops-

Quoted 1 line

invalidate for IOMMU

+ * security model.
+ */

(quoted diff) +2 -0
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range

*range,

(quoted diff) +1 -0
+				  const struct drm_gpusvm_ctx

*ctx)

(quoted diff) +2 -0
+{
+	unsigned long npages = npages_in_range(range->va.start,

range->va.end);

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+
+	if (ctx->in_notifier)
+		lockdep_assert_held_write(&gpusvm-
Quoted 1 line

notifier_lock);

Quoted 12 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 1457

Quoted 8 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+	else
+		drm_gpusvm_notifier_lock(gpusvm);
+
+	__drm_gpusvm_range_unmap_pages(gpusvm, range, npages);
+
+	if (!ctx->in_notifier)
+		drm_gpusvm_notifier_unlock(gpusvm);
+}

NIT: Separate functions for locked / unlocked makes life easier for static code analyzers.

Willl do.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

Section below I think should belong to drm_pagemap.c

Diagree. See my comments on zdd above. Also drm_gpusvm_migration_put_pages uses
migration pfns which definitely should not be in drm_pagemap.c.

Like mentioned above, with upcoming populate_mm() moving to being a drm_pagemap op, and none of the low level migration helpers taking drm_gpusvm as an argument, I think it makes sense, but if you rather want to look at shuffling things around when that is in place, that's ok. Agree, though that anything needing struct drm_gpusvm should *not* be in drm_pagemap.c

Thanks,
Thomas

Quoted 124 lines

Quoted 107 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 1465

Quoted 8 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+
+/**
+ * drm_gpusvm_migration_put_page - Put a migration page
+ * @page: Pointer to the page to put
+ *
+ * This function unlocks and puts a page.
+ */
+static void drm_gpusvm_migration_put_page(struct page *page)

_unlock_put_page()?

Quoted 96 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+{
+	unlock_page(page);
+	put_page(page);
+}
+
+/**
+ * drm_gpusvm_migration_put_pages - Put migration pages
+ * @npages: Number of pages
+ * @migrate_pfn: Array of migrate page frame numbers
+ *
+ * This function puts an array of pages.
+ */
+static void drm_gpusvm_migration_put_pages(unsigned long npages,
+					   unsigned long

*migrate_pfn)

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!migrate_pfn[i])
+			continue;
+
+		drm_gpusvm_migration_put_page(migrate_pfn_to_pag

e(mi
grate_pfn[i]));

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+		migrate_pfn[i] = 0;
+	}
+}
+
+/**
+ * drm_gpusvm_get_devmem_page - Get a reference to a device

memory
page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU
SVM zone
+ * device data and initializes it for zone device usage.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+static void drm_gpusvm_get_devmem_page(struct page *page,
+				     struct drm_gpusvm_zdd *zdd)
+{
+	page->zone_device_data = drm_gpusvm_zdd_get(zdd);
+	zone_device_page_init(page);
+}
+
+/**
+ * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU

SVM
migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped
pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn,
maps the
+ * corresponding page, and stores the DMA address in the provided
@dma_addr
+ * array.
+ *
+ * Return: 0 on success, -EFAULT if an error occurs during mapping.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+static int drm_gpusvm_migrate_map_pages(struct device *dev,
+					dma_addr_t *dma_addr,
+					long unsigned int

*migrate_pfn,

(quoted diff) +2 -0
+					unsigned long npages,
+					enum dma_data_direction

dir)

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page =

migrate_pfn_to_page(migrate_pfn[i]);

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+
+		if (!page)
+			continue;
+
+		if (WARN_ON_ONCE(is_zone_device_page(page)))
+			return -EFAULT;
+
+		dma_addr[i] = dma_map_page(dev, page, 0,

PAGE_SIZE,
dir);

Quoted 9 lines

drivers/gpu/drm/xe/drm_gpusvm.c line 1545

Quoted 6 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+		if (dma_mapping_error(dev, dma_addr[i]))
+			return -EFAULT;
+	}
+
+	return 0;
+}

TBC'd

Thanks for the comments!

Matt

Quoted 2 lines

/Thomas