Skip to contentSkip to conversation

Introduce GPU SVM and Xe SVM implementation

PATCH Superseded Matthew Brost 8 participants 103 messages Jan 29, 2025 – Feb 11, 2025 Page 1 of 3

Subsystems (9)

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Matthew Brost

author maintainer
#0 lore raw

v4 cover Introduce GPU SVM and Xe SVM implementation

Version 4 of GPU SVM. Thanks to everyone (especially Sima, Thomas, Alistair, Himal) for their numerous reviews on revision 1, 2, 3 and for helping to address many design issues.

This version has been tested with IGT [1] on PVC, BMG, and LNL. Also tested with level0 (UMD) PR [2].

Major changes in v2:
- Dropped mmap write abuse
- core MM locking and retry loops instead of driver locking to avoid races
- Removed physical to virtual references
- Embedded structure/ops for drm_gpusvm_devmem
- Fixed mremap and fork issues
- Added DRM pagemap
- Included RFC documentation in the kernel doc

Major changes in v3:
- Move GPU SVM and DRM pagemap to DRM level
- Mostly addresses Thomas's feedback, lots of small changes documented in each individual patch change log

Major changes in v4:
- Pull documentation patch in
- Fix Kconfig / VRAM migration issue
- Address feedback which came out of internal multi-GPU implementation

Known issues in v4:
- Check pages still exists, changed to threshold in this version which is better but still need to root cause cross process page finding on small user allocations.

Matt

[1] https://patchwork.freedesktop.org/series/137545/#rev3
[2] https://github.com/intel/compute-runtime/pull/782

Matthew Brost (29):
drm/xe: Retry BO allocation
mm/migrate: Add migrate_device_pfns
mm/migrate: Trylock device page in do_swap_page drm/gpusvm: Add support for GPU Shared Virtual Memory drm/xe: Select DRM_GPUSVM Kconfig
drm/xe/uapi: Add DRM_XE_VM_BIND_FLAG_CPU_ADDR_MIRROR flag drm/xe: Add SVM init / close / fini to faulting VMs drm/xe: Nuke VM's mapping upon close
drm/xe: Add SVM range invalidation and page fault handler drm/gpuvm: Add DRM_GPUVA_OP_DRIVER
drm/xe: Add (re)bind to SVM page fault handler
drm/xe: Add SVM garbage collector
drm/xe: Add unbind to SVM garbage collector
drm/xe: Do not allow CPU address mirror VMA unbind if the GPU has
bindings
drm/xe: Enable CPU address mirror uAPI
drm/xe/uapi: Add DRM_XE_QUERY_CONFIG_FLAG_HAS_CPU_ADDR_MIRROR drm/xe: Add migrate layer functions for SVM support drm/xe: Add SVM device memory mirroring
drm/xe: Add drm_gpusvm_devmem to xe_bo
drm/xe: Add GPUSVM device memory copy vfunc functions drm/xe: Add Xe SVM populate_devmem_pfn GPU SVM vfunc drm/xe: Add Xe SVM devmem_release GPU SVM vfunc drm/xe: Add BO flags required for SVM
drm/xe: Add SVM VRAM migration
drm/xe: Basic SVM BO eviction
drm/xe: Add SVM debug
drm/xe: Add modparam for SVM notifier size
drm/xe: Add always_migrate_to_vram modparam
drm/doc: gpusvm: Add GPU SVM documentation

Thomas Hellström (4):
drm/pagemap: Add DRM pagemap
drm/xe/bo: Introduce xe_bo_put_async
drm/xe: Add dma_addr res cursor
drm/xe: Add drm_pagemap ops to SVM

 Documentation/gpu/rfc/gpusvm.rst            |   84 +
 Documentation/gpu/rfc/index.rst             |    4 +
 drivers/gpu/drm/Kconfig                     |    9 +
 drivers/gpu/drm/Makefile                    |    1 +
 drivers/gpu/drm/drm_gpusvm.c                | 2240 +++++++++++++++++++
 drivers/gpu/drm/xe/Kconfig                  |   10 +
 drivers/gpu/drm/xe/Makefile                 |    1 +
 drivers/gpu/drm/xe/xe_bo.c                  |   63 +-
 drivers/gpu/drm/xe/xe_bo.h                  |   14 +
 drivers/gpu/drm/xe/xe_bo_types.h            |    4 +
 drivers/gpu/drm/xe/xe_device.c              |    3 +
 drivers/gpu/drm/xe/xe_device_types.h        |   22 +
 drivers/gpu/drm/xe/xe_gt_pagefault.c        |   17 +-
 drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c |   24 +
 drivers/gpu/drm/xe/xe_gt_tlb_invalidation.h |    2 +
 drivers/gpu/drm/xe/xe_migrate.c             |  175 ++
 drivers/gpu/drm/xe/xe_migrate.h             |   10 +
 drivers/gpu/drm/xe/xe_module.c              |    7 +
 drivers/gpu/drm/xe/xe_module.h              |    2 +
 drivers/gpu/drm/xe/xe_pt.c                  |  393 +++-
 drivers/gpu/drm/xe/xe_pt.h                  |    5 +
 drivers/gpu/drm/xe/xe_pt_types.h            |    2 +
 drivers/gpu/drm/xe/xe_query.c               |    5 +-
 drivers/gpu/drm/xe/xe_res_cursor.h          |  116 +-
 drivers/gpu/drm/xe/xe_svm.c                 |  946 ++++++++
 drivers/gpu/drm/xe/xe_svm.h                 |   84 +
 drivers/gpu/drm/xe/xe_tile.c                |    5 +
 drivers/gpu/drm/xe/xe_vm.c                  |  375 +++-
 drivers/gpu/drm/xe/xe_vm.h                  |   15 +-
 drivers/gpu/drm/xe/xe_vm_types.h            |   57 +
 include/drm/drm_gpusvm.h                    |  445 ++++
 include/drm/drm_gpuvm.h                     |    5 +
 include/drm/drm_pagemap.h                   |  105 +
 include/linux/migrate.h                     |    1 +
 include/uapi/drm/xe_drm.h                   |   22 +-
 mm/memory.c                                 |   13 +-
 mm/migrate_device.c                         |  116 +-
 37 files changed, 5245 insertions(+), 157 deletions(-)
 create mode 100644 Documentation/gpu/rfc/gpusvm.rst
 create mode 100644 drivers/gpu/drm/drm_gpusvm.c
 create mode 100644 drivers/gpu/drm/xe/xe_svm.c
 create mode 100644 drivers/gpu/drm/xe/xe_svm.h
 create mode 100644 include/drm/drm_gpusvm.h
 create mode 100644 include/drm/drm_pagemap.h
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

PATCH v4 1/33 drm/xe: Retry BO allocation

TTM doesn't support fair eviction via WW locking, this mitigated in by using retry loops in exec and preempt rebind worker. Extend this retry loop to BO allocation. Once TTM supports fair eviction this patch can be reverted.

v4:
- Keep line break (Stuart)

---
 drivers/gpu/drm/xe/xe_bo.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +7 -0
index c32201123d44..fb1629d9d566 100644
@@ -2162,6 +2162,7 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 	struct xe_file *xef = to_xe_file(file);
 	struct drm_xe_gem_create *args = data;
 	struct xe_vm *vm = NULL;
+	ktime_t end = 0;
 	struct xe_bo *bo;
 	unsigned int bo_flags;
 	u32 handle;
@@ -2234,6 +2235,10 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 		vm = xe_vm_lookup(xef, args->vm_id);
 		if (XE_IOCTL_DBG(xe, !vm))
 			return -ENOENT;
+	}
+
+retry:
+	if (vm) {
 		err = xe_vm_lock(vm, true);
 		if (err)
 			goto out_vm;
@@ -2247,6 +2252,8 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 
 	if (IS_ERR(bo)) {
 		err = PTR_ERR(bo);
+		if (xe_vm_validate_should_retry(NULL, err, &end))
+			goto retry;
 		goto out_vm;
 	}
 
Signature · 1 line
2.34.1
author maintainer
#2 lore raw

PATCH v4 2/33 mm/migrate: Add migrate_device_pfns

In reply to Matthew Brost

Add migrate_device_pfns which prepares an array of pre-populated device pages for migration. This is needed for eviction of known set of non-contiguous devices pages to cpu pages which is a common case for SVM in DRM drivers using TTM.

v2:
- s/migrate_device_vma_range/migrate_device_prepopulated_range
- Drop extra mmu invalidation (Vetter)
v3:
- s/migrate_device_prepopulated_range/migrate_device_pfns (Alistar)
- Use helper to lock device pages (Alistar)
- Update commit message with why this is required (Alistar)

---
 include/linux/migrate.h |  1 +
 mm/migrate_device.c     | 52 +++++++++++++++++++++++++++++------------
 2 files changed, 38 insertions(+), 15 deletions(-)
include/linux/migrate.h +1 -0
index 002e49b2ebd9..6254746648cc 100644
@@ -229,6 +229,7 @@ void migrate_vma_pages(struct migrate_vma *migrate);
 void migrate_vma_finalize(struct migrate_vma *migrate);
 int migrate_device_range(unsigned long *src_pfns, unsigned long start,
 			unsigned long npages);
+int migrate_device_pfns(unsigned long *src_pfns, unsigned long npages);
 void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns,
 			unsigned long npages);
 void migrate_device_finalize(unsigned long *src_pfns,
mm/migrate_device.c +37 -15

72 lines not on this page · raw message

Signature · 1 line
2.34.1
reviewer Reviewed-by on v4 2/33
#3 lore raw

Re: [PATCH v4 02/33] mm/migrate: Add migrate_device_pfns

Quoted 5 lines · Matthew Brost

Add migrate_device_pfns which prepares an array of pre-populated device pages for migration. This is needed for eviction of known set of non-contiguous devices pages to cpu pages which is a common case for SVM in DRM drivers using TTM.

Quoted 7 lines

v2:
- s/migrate_device_vma_range/migrate_device_prepopulated_range
- Drop extra mmu invalidation (Vetter)
v3:
- s/migrate_device_prepopulated_range/migrate_device_pfns (Alistar)
- Use helper to lock device pages (Alistar)
- Update commit message with why this is required (Alistar)

Thanks! Looks good to me now so:

Quoted 100 lines
---
 include/linux/migrate.h |  1 +
 mm/migrate_device.c     | 52 +++++++++++++++++++++++++++++------------
 2 files changed, 38 insertions(+), 15 deletions(-)
include/linux/migrate.h +1 -0
index 002e49b2ebd9..6254746648cc 100644
@@ -229,6 +229,7 @@ void migrate_vma_pages(struct migrate_vma *migrate);
 void migrate_vma_finalize(struct migrate_vma *migrate);
 int migrate_device_range(unsigned long *src_pfns, unsigned long start,
 			unsigned long npages);
+int migrate_device_pfns(unsigned long *src_pfns, unsigned long npages);
 void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns,
 			unsigned long npages);
 void migrate_device_finalize(unsigned long *src_pfns,
mm/migrate_device.c +37 -15
index 9cf26592ac93..19960743f927 100644
@@ -876,6 +876,22 @@ void migrate_vma_finalize(struct migrate_vma *migrate)
 }
 EXPORT_SYMBOL(migrate_vma_finalize);
 
+static unsigned long migrate_device_pfn_lock(unsigned long pfn)
+{
+	struct folio *folio;
+
+	folio = folio_get_nontail_page(pfn_to_page(pfn));
+	if (!folio)
+		return 0;
+
+	if (!folio_trylock(folio)) {
+		folio_put(folio);
+		return 0;
+	}
+
+	return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
+}
+
 /**
  * migrate_device_range() - migrate device private pfns to normal memory.
  * @src_pfns: array large enough to hold migrating source device private pfns.
@@ -900,29 +916,35 @@ int migrate_device_range(unsigned long *src_pfns, unsigned long start,
 {
 	unsigned long i, pfn;
 
-	for (pfn = start, i = 0; i < npages; pfn++, i++) {
-		struct folio *folio;
+	for (pfn = start, i = 0; i < npages; pfn++, i++)
+		src_pfns[i] = migrate_device_pfn_lock(pfn);
 
-		folio = folio_get_nontail_page(pfn_to_page(pfn));
-		if (!folio) {
-			src_pfns[i] = 0;
-			continue;
-		}
+	migrate_device_unmap(src_pfns, npages, NULL);
 
-		if (!folio_trylock(folio)) {
-			src_pfns[i] = 0;
-			folio_put(folio);
-			continue;
-		}
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_range);
 
-		src_pfns[i] = migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
-	}
+/**
+ * migrate_device_pfns() - migrate device private pfns to normal memory.
+ * @src_pfns: pre-popluated array of source device private pfns to migrate.
+ * @npages: number of pages to migrate.
+ *
+ * Similar to migrate_device_range() but supports non-contiguous pre-popluated
+ * array of device pages to migrate.
+ */
+int migrate_device_pfns(unsigned long *src_pfns, unsigned long npages)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; i++)
+		src_pfns[i] = migrate_device_pfn_lock(src_pfns[i]);
 
 	migrate_device_unmap(src_pfns, npages, NULL);
 
 	return 0;
 }
-EXPORT_SYMBOL(migrate_device_range);
+EXPORT_SYMBOL(migrate_device_pfns);
 
 /*
  * Migrate a device coherent folio back to normal memory. The caller should have
Signature · 1 line
2.34.1
Reviewed-by on v4 2/33
#4 lore raw

Re: [PATCH v4 02/33] mm/migrate: Add migrate_device_pfns

In reply to Matthew Brost
Quoted 100 lines · Matthew Brost

Add migrate_device_pfns which prepares an array of pre-populated device pages for migration. This is needed for eviction of known set of non-contiguous devices pages to cpu pages which is a common case for SVM in DRM drivers using TTM.

v2:
- s/migrate_device_vma_range/migrate_device_prepopulated_range
- Drop extra mmu invalidation (Vetter)
v3:
- s/migrate_device_prepopulated_range/migrate_device_pfns (Alistar)
- Use helper to lock device pages (Alistar)
- Update commit message with why this is required (Alistar)

---
  include/linux/migrate.h |  1 +
  mm/migrate_device.c     | 52 +++++++++++++++++++++++++++++------------
  2 files changed, 38 insertions(+), 15 deletions(-)
include/linux/migrate.h +1 -0
index 002e49b2ebd9..6254746648cc 100644
@@ -229,6 +229,7 @@ void migrate_vma_pages(struct migrate_vma *migrate);
  void migrate_vma_finalize(struct migrate_vma *migrate);
  int migrate_device_range(unsigned long *src_pfns, unsigned long start,
  			unsigned long npages);
+int migrate_device_pfns(unsigned long *src_pfns, unsigned long npages);
  void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns,
  			unsigned long npages);
  void migrate_device_finalize(unsigned long *src_pfns,
mm/migrate_device.c +36 -14
index 9cf26592ac93..19960743f927 100644
@@ -876,6 +876,22 @@ void migrate_vma_finalize(struct migrate_vma *migrate)
  }
  EXPORT_SYMBOL(migrate_vma_finalize);
 
+static unsigned long migrate_device_pfn_lock(unsigned long pfn)
+{
+	struct folio *folio;
+
+	folio = folio_get_nontail_page(pfn_to_page(pfn));
+	if (!folio)
+		return 0;
+
+	if (!folio_trylock(folio)) {
+		folio_put(folio);
+		return 0;
+	}
+
+	return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
+}
+
  /**
   * migrate_device_range() - migrate device private pfns to normal memory.
   * @src_pfns: array large enough to hold migrating source device private pfns.
@@ -900,29 +916,35 @@ int migrate_device_range(unsigned long *src_pfns, unsigned long start,
  {
  	unsigned long i, pfn;
 
-	for (pfn = start, i = 0; i < npages; pfn++, i++) {
-		struct folio *folio;
+	for (pfn = start, i = 0; i < npages; pfn++, i++)
+		src_pfns[i] = migrate_device_pfn_lock(pfn);
 
-		folio = folio_get_nontail_page(pfn_to_page(pfn));
-		if (!folio) {
-			src_pfns[i] = 0;
-			continue;
-		}
+	migrate_device_unmap(src_pfns, npages, NULL);
 
-		if (!folio_trylock(folio)) {
-			src_pfns[i] = 0;
-			folio_put(folio);
-			continue;
-		}
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_range);
 
-		src_pfns[i] = migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
-	}
+/**
+ * migrate_device_pfns() - migrate device private pfns to normal memory.
+ * @src_pfns: pre-popluated array of source device private pfns to migrate.
+ * @npages: number of pages to migrate.
+ *
+ * Similar to migrate_device_range() but supports non-contiguous pre-popluated
+ * array of device pages to migrate.
+ */
+int migrate_device_pfns(unsigned long *src_pfns, unsigned long npages)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; i++)
+		src_pfns[i] = migrate_device_pfn_lock(src_pfns[i]);

mm/migrate_device.c int migrate_device_range(unsigned long *src_pfns, unsigned long start, · line 950

Quoted 10 lines
mm/migrate_device.c +1 -1
 
  	migrate_device_unmap(src_pfns, npages, NULL);
 
  	return 0;
  }
-EXPORT_SYMBOL(migrate_device_range);
+EXPORT_SYMBOL(migrate_device_pfns);
 
  /*
   * Migrate a device coherent folio back to normal memory. The caller should have

Looks good to me and I have confirmed that a code flow has been added that calls this function from the actual driver (xe), which did not exist in the previous v3 patch series.
(This code flow called when ttm performs eviction when ttm needs)

There seems to be no test scenario for testing device memory pressure in the test cases mentioned in the cover letter. Do you plan to add this scenario as well? ( Please correct me if I misunderstood.)

author maintainer
#5 lore raw

Re: [PATCH v4 02/33] mm/migrate: Add migrate_device_pfns

Quoted 110 lines · Gwan-gyeong Mun
Quoted 92 lines · Matthew Brost

Add migrate_device_pfns which prepares an array of pre-populated device pages for migration. This is needed for eviction of known set of non-contiguous devices pages to cpu pages which is a common case for SVM in DRM drivers using TTM.

v2:
- s/migrate_device_vma_range/migrate_device_prepopulated_range
- Drop extra mmu invalidation (Vetter)
v3:
- s/migrate_device_prepopulated_range/migrate_device_pfns (Alistar)
- Use helper to lock device pages (Alistar)
- Update commit message with why this is required (Alistar)

---
  include/linux/migrate.h |  1 +
  mm/migrate_device.c     | 52 +++++++++++++++++++++++++++++------------
  2 files changed, 38 insertions(+), 15 deletions(-)
include/linux/migrate.h +1 -0
index 002e49b2ebd9..6254746648cc 100644
@@ -229,6 +229,7 @@ void migrate_vma_pages(struct migrate_vma *migrate);
  void migrate_vma_finalize(struct migrate_vma *migrate);
  int migrate_device_range(unsigned long *src_pfns, unsigned long start,
  			unsigned long npages);
+int migrate_device_pfns(unsigned long *src_pfns, unsigned long npages);
  void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns,
  			unsigned long npages);
  void migrate_device_finalize(unsigned long *src_pfns,
mm/migrate_device.c +33 -14
index 9cf26592ac93..19960743f927 100644
@@ -876,6 +876,22 @@ void migrate_vma_finalize(struct migrate_vma *migrate)
  }
  EXPORT_SYMBOL(migrate_vma_finalize);
+static unsigned long migrate_device_pfn_lock(unsigned long pfn)
+{
+	struct folio *folio;
+
+	folio = folio_get_nontail_page(pfn_to_page(pfn));
+	if (!folio)
+		return 0;
+
+	if (!folio_trylock(folio)) {
+		folio_put(folio);
+		return 0;
+	}
+
+	return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
+}
+
  /**
   * migrate_device_range() - migrate device private pfns to normal memory.
   * @src_pfns: array large enough to hold migrating source device private pfns.
@@ -900,29 +916,35 @@ int migrate_device_range(unsigned long *src_pfns, unsigned long start,
  {
  	unsigned long i, pfn;
-	for (pfn = start, i = 0; i < npages; pfn++, i++) {
-		struct folio *folio;
+	for (pfn = start, i = 0; i < npages; pfn++, i++)
+		src_pfns[i] = migrate_device_pfn_lock(pfn);
-		folio = folio_get_nontail_page(pfn_to_page(pfn));
-		if (!folio) {
-			src_pfns[i] = 0;
-			continue;
-		}
+	migrate_device_unmap(src_pfns, npages, NULL);
-		if (!folio_trylock(folio)) {
-			src_pfns[i] = 0;
-			folio_put(folio);
-			continue;
-		}
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_range);
-		src_pfns[i] = migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
-	}
+/**
+ * migrate_device_pfns() - migrate device private pfns to normal memory.
+ * @src_pfns: pre-popluated array of source device private pfns to migrate.
+ * @npages: number of pages to migrate.
+ *
+ * Similar to migrate_device_range() but supports non-contiguous pre-popluated
+ * array of device pages to migrate.
+ */
+int migrate_device_pfns(unsigned long *src_pfns, unsigned long npages)
+{
+	unsigned long i;

mm/migrate_device.c int migrate_device_range(unsigned long *src_pfns, unsigned long start, · line 943

Quoted 10 lines
mm/migrate_device.c +4 -1
+
+	for (i = 0; i < npages; i++)
+		src_pfns[i] = migrate_device_pfn_lock(src_pfns[i]);
  	migrate_device_unmap(src_pfns, npages, NULL);
  	return 0;
  }
-EXPORT_SYMBOL(migrate_device_range);
+EXPORT_SYMBOL(migrate_device_pfns);
  /*
   * Migrate a device coherent folio back to normal memory. The caller should have

Looks good to me and I have confirmed that a code flow has been added that calls this function from the actual driver (xe),
which did not exist in the previous v3 patch series.
(This code flow called when ttm performs eviction when ttm needs)

Quoted 5 lines

There seems to be no test scenario for testing device memory pressure in the test cases mentioned in the cover letter.
Do you plan to add this scenario as well? ( Please correct me if I misunderstood.)

The *evict sections in [1] to test device memory pressure.

Matt

[1] https://patchwork.freedesktop.org/series/137545/#rev3

Quoted 2 lines
author maintainer
#6 lore raw

PATCH v4 3/33 mm/migrate: Trylock device page in do_swap_page

In reply to Matthew Brost

Avoid multiple CPU page faults to the same device page racing by trying to lock the page in do_swap_page before taking an extra reference to the page. This prevents scenarios where multiple CPU page faults each take an extra reference to a device page, which could abort migration in folio_migrate_mapping. With the device page being locked in do_swap_page, the migrate_vma_* functions need to be updated to avoid locking the fault_page argument.

Prior to this change, a livelock scenario could occur in Xe's (Intel GPU DRM driver) SVM implementation if enough threads faulted the same device page.

v3:
- Put page after unlocking page (Alistair)
- Warn on spliting a TPH which is fault page (Alistair)
- Warn on dst page == fault page (Alistair)

Cc Alistair Popple <[email protected]>
Cc Philip Yang <[email protected]>
Cc Felix Kuehling <[email protected]>
Cc Christian König <[email protected]>
Cc Andrew Morton <[email protected]>
---
 mm/memory.c         | 13 ++++++---
 mm/migrate_device.c | 64 ++++++++++++++++++++++++++++++++-------------
 2 files changed, 55 insertions(+), 22 deletions(-)
mm/memory.c +9 -4
index 398c031be9ba..a4776e58b0e5 100644
@@ -4267,10 +4267,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 			 * Get a page reference while we know the page can't be
 			 * freed.
 			 */
-			get_page(vmf->page);
-			pte_unmap_unlock(vmf->pte, vmf->ptl);
-			ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
-			put_page(vmf->page);
+			if (trylock_page(vmf->page)) {
+				get_page(vmf->page);
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+				ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);
+				unlock_page(vmf->page);
+				put_page(vmf->page);
+			} else {
+				pte_unmap_unlock(vmf->pte, vmf->ptl);
+			}
 		} else if (is_hwpoison_entry(entry)) {
 			ret = VM_FAULT_HWPOISON;
 		} else if (is_pte_marker_entry(entry)) {
mm/migrate_device.c +46 -18

153 lines not on this page · raw message

Signature · 1 line
2.34.1
author maintainer
#7 lore raw

PATCH v4 4/33 drm/pagemap: Add DRM pagemap

In reply to Matthew Brost

From: Thomas Hellström <[email protected]>

Introduce drm_pagemap ops to map and unmap dma to VRAM resources. In the local memory case it's a matter of merely providing an offset into the device's physical address. For future p2p the map and unmap functions may encode as needed.

Similar to how dma-buf works, let the memory provider (drm_pagemap) provide the mapping functionality.

v3:
- Move to drm level include
v4:
- Fix kernel doc (G.G.)

---
 include/drm/drm_pagemap.h | 105 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 105 insertions(+)
 create mode 100644 include/drm/drm_pagemap.h
include/drm/drm_pagemap.h new file +105 -0
new file mode 100644
index 000000000000..2b610ccf7e30
@@ -0,0 +1,105 @@
+/* SPDX-License-Identifier: MIT */
+#ifndef _DRM_PAGEMAP_H_
+#define _DRM_PAGEMAP_H_
+
+#include <linux/dma-direction.h>
+#include <linux/hmm.h>
+#include <linux/types.h>
+
+struct drm_pagemap;
+struct device;
+
+/**
+ * enum drm_interconnect_protocol - Used to identify an interconnect protocol.
+ */
+enum drm_interconnect_protocol {
+	DRM_INTERCONNECT_SYSTEM,    /* DMA map is system pages. */
+	DRM_INTERCONNECT_PCIE_P2P,  /* DMA map is PCIE P2P */
+	DRM_INTERCONNECT_DRIVER,    /* DMA map is driver defined */
+	/* A driver can add private values beyond DRM_INTERCONNECT_DRIVER */
+};
+
+/**
+ * struct drm_pagemap_dma_addr - DMA address representation.
+ * @addr: The dma address or driver-defined address for driver private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size is PAGE_SIZE << order).
+ * @dir: The DMA direction.
+ *
+ * Note: There is room for improvement here. We should be able to pack into
+ * 64 bits.
+ */
+struct drm_pagemap_dma_addr {
+	dma_addr_t addr;
+	u64 proto : 54;
+	u64 order : 8;
+	u64 dir : 2;
+};
+
+/**
+ * drm_pagemap_dma_addr_encode() - Encode a dma address with metadata
+ * @addr: The dma address or driver-defined address for driver private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size is PAGE_SIZE << order).
+ * @dir: The DMA direction.
+ *
+ * Return: A struct drm_pagemap_dma_addr encoding the above information.
+ */
+static inline struct drm_pagemap_dma_addr
+drm_pagemap_dma_addr_encode(dma_addr_t addr,
+			    enum drm_interconnect_protocol proto,
+			    unsigned int order,
+			    enum dma_data_direction dir)
+{
+	return (struct drm_pagemap_dma_addr) {
+		.addr = addr,
+		.proto = proto,
+		.order = order,
+		.dir = dir,
+	};
+}
+
+/**
+ * struct drm_pagemap_ops: Ops for a drm-pagemap.
+ */
+struct drm_pagemap_ops {
+	/**
+	 * @map_dma: Map for dma access or provide a virtual address suitable for
+	 *
+	 * @dpagemap: The struct drm_pagemap for the page.
+	 * @dev: The dma mapper.
+	 * @page: The page to map.
+	 * @order: The page order of the dma mapping. (Size is PAGE_SIZE << order).
+	 * @dir: The transfer direction.
+	 */
+	struct drm_pagemap_dma_addr (*map_dma)(struct drm_pagemap *dpagemap,
+					       struct device *dev,
+					       struct page *page,
+					       unsigned int order,
+					       enum dma_data_direction dir);
+
+	/**
+	 * @unmap_dma: Unmap a dma address previously obtained using @map_dma.
+	 *
+	 * @dpagemap: The struct drm_pagemap for the mapping.
+	 * @dev: The dma unmapper.
+	 * @addr: The dma address obtained when mapping.
+	 */
+	void (*unmap_dma)(struct drm_pagemap *dpagemap,
+			  struct device *dev,
+			  struct drm_pagemap_dma_addr addr);
+
+};
+
+/**
+ * struct drm_pagemap: Additional information for a struct dev_pagemap
+ * used for device p2p handshaking.
+ * @ops: The struct drm_pagemap_ops.
+ * @dev: The struct drevice owning the device-private memory.
+ */
+struct drm_pagemap {
+	const struct drm_pagemap_ops *ops;
+	struct device *dev;
+};
+
+#endif
Signature · 1 line
2.34.1
#8 lore raw

Re: [PATCH v4 04/33] drm/pagemap: Add DRM pagemap

Quoted 10 lines · Matthew Brost

From: Thomas Hellström <[email protected]>

Introduce drm_pagemap ops to map and unmap dma to VRAM resources. In the
local memory case it's a matter of merely providing an offset into the
device's physical address. For future p2p the map and unmap functions may
encode as needed.

Quoted 3 lines

Similar to how dma-buf works, let the memory provider (drm_pagemap) provide
the mapping functionality.

It should be noted that the long term idea for dma mapping is to have that done by the client instead of by the memory provider, which Jason reminded me of in a discussion on dri-devel. The dma-mapping here is modeled after how it's done for dma-buf, where the exporter maps dma.

So following that, it might be that we should move these dma-mapping ops to the drm_gpusvm().

The situation I can think of, where this might be a problem is that if the device-private struct page to dma address mapping is not known to the client.

/Thomas

Quoted 140 lines

v3:
 - Move to drm level include
v4:
 - Fix kernel doc (G.G.)

---
 include/drm/drm_pagemap.h | 105
++++++++++++++++++++++++++++++++++++++

 1 file changed, 105 insertions(+)

 create mode 100644 include/drm/drm_pagemap.h

include/drm/drm_pagemap.h new file +13 -0
new file mode 100644
index 000000000000..2b610ccf7e30
@@ -0,0 +1,105 @@
+/* SPDX-License-Identifier: MIT */
+#ifndef _DRM_PAGEMAP_H_
+#define _DRM_PAGEMAP_H_
+
+#include <linux/dma-direction.h>
+#include <linux/hmm.h>
+#include <linux/types.h>
+
+struct drm_pagemap;
+struct device;
+
+/**
+ * enum drm_interconnect_protocol - Used to identify an interconnect

protocol.
+ */

include/drm/drm_pagemap.h new file +5 -0
+enum drm_interconnect_protocol {
+	DRM_INTERCONNECT_SYSTEM,    /* DMA map is system pages. */
+	DRM_INTERCONNECT_PCIE_P2P,  /* DMA map is PCIE P2P */
+	DRM_INTERCONNECT_DRIVER,    /* DMA map is driver defined */
+	/* A driver can add private values beyond

DRM_INTERCONNECT_DRIVER */

include/drm/drm_pagemap.h new file +5 -0
+};
+
+/**
+ * struct drm_pagemap_dma_addr - DMA address representation.
+ * @addr: The dma address or driver-defined address for driver

private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size is PAGE_SIZE << order).
+ * @dir: The DMA direction.
+ *
+ * Note: There is room for improvement here. We should be able to pack into
+ * 64 bits.
+ */

include/drm/drm_pagemap.h new file +9 -0
+struct drm_pagemap_dma_addr {
+	dma_addr_t addr;
+	u64 proto : 54;
+	u64 order : 8;
+	u64 dir : 2;
+};
+
+/**
+ * drm_pagemap_dma_addr_encode() - Encode a dma address with

metadata
+ * @addr: The dma address or driver-defined address for driver private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size is PAGE_SIZE << order).
+ * @dir: The DMA direction.
+ *
+ * Return: A struct drm_pagemap_dma_addr encoding the above information.
+ */

include/drm/drm_pagemap.h new file +20 -0
+static inline struct drm_pagemap_dma_addr
+drm_pagemap_dma_addr_encode(dma_addr_t addr,
+			    enum drm_interconnect_protocol proto,
+			    unsigned int order,
+			    enum dma_data_direction dir)
+{
+	return (struct drm_pagemap_dma_addr) {
+		.addr = addr,
+		.proto = proto,
+		.order = order,
+		.dir = dir,
+	};
+}
+
+/**
+ * struct drm_pagemap_ops: Ops for a drm-pagemap.
+ */
+struct drm_pagemap_ops {
+	/**
+	 * @map_dma: Map for dma access or provide a virtual address

suitable for

include/drm/drm_pagemap.h new file +5 -0
+	 *
+	 * @dpagemap: The struct drm_pagemap for the page.
+	 * @dev: The dma mapper.
+	 * @page: The page to map.
+	 * @order: The page order of the dma mapping. (Size is

PAGE_SIZE << order).

include/drm/drm_pagemap.h new file +3 -0
+	 * @dir: The transfer direction.
+	 */
+	struct drm_pagemap_dma_addr (*map_dma)(struct drm_pagemap

*dpagemap,

include/drm/drm_pagemap.h new file +4 -0
+					       struct device *dev,
+					       struct page *page,
+					       unsigned int order,
+					       enum

dma_data_direction dir);

include/drm/drm_pagemap.h new file +3 -0
+
+	/**
+	 * @unmap_dma: Unmap a dma address previously obtained using

@map_dma.

include/drm/drm_pagemap.h new file +13 -0
+	 *
+	 * @dpagemap: The struct drm_pagemap for the mapping.
+	 * @dev: The dma unmapper.
+	 * @addr: The dma address obtained when mapping.
+	 */
+	void (*unmap_dma)(struct drm_pagemap *dpagemap,
+			  struct device *dev,
+			  struct drm_pagemap_dma_addr addr);
+
+};
+
+/**
+ * struct drm_pagemap: Additional information for a struct

dev_pagemap
+ * used for device p2p handshaking.
+ * @ops: The struct drm_pagemap_ops.
+ * @dev: The struct drevice owning the device-private memory.
+ */

include/drm/drm_pagemap.h new file +6 -0
+struct drm_pagemap {
+	const struct drm_pagemap_ops *ops;
+	struct device *dev;
+};
+
+#endif
author maintainer
#9 lore raw

Re: [PATCH v4 04/33] drm/pagemap: Add DRM pagemap

Quoted 11 lines · Thomas Hellström
Quoted 10 lines · Matthew Brost

From: Thomas Hellström <[email protected]>

Introduce drm_pagemap ops to map and unmap dma to VRAM resources. In the
local memory case it's a matter of merely providing an offset into the
device's physical address. For future p2p the map and unmap functions may
encode as needed.

Quoted 4 lines
Quoted 3 lines

Similar to how dma-buf works, let the memory provider (drm_pagemap) provide
the mapping functionality.

Trying to parse all of this.

Quoted 2 lines

It should be noted that the long term idea for dma mapping is to have that done by the client instead of by the memory provider, which Jason

- Client here is the device mapping the memory.
- Memory provider is the device where the memory is located?

Did I get this correct?

Quoted 6 lines

reminded me of in a discussion on dri-devel. The dma-mapping here is modeled after how it's done for dma-buf, where the exporter maps dma.

So following that, it might be that we should move these dma-mapping ops to the drm_gpusvm().

So we move ops to the local client (gpusvm) rather than remote device, right?

Quoted 4 lines

The situation I can think of, where this might be a problem is that if the device-private struct page to dma address mapping is not known to the client.

I'm not following this but I agree if dma mapping at the client we need the remote device structure given how the dma mapping API works.

So to wrap it up - what, if anything, do you think we need to do to this individual patch as part of this series?

Matt

Quoted 147 lines

/Thomas

Quoted 140 lines

v3:
 - Move to drm level include
v4:
 - Fix kernel doc (G.G.)

---
 include/drm/drm_pagemap.h | 105
++++++++++++++++++++++++++++++++++++++

 1 file changed, 105 insertions(+)

 create mode 100644 include/drm/drm_pagemap.h

include/drm/drm_pagemap.h new file +13 -0
new file mode 100644
index 000000000000..2b610ccf7e30
@@ -0,0 +1,105 @@
+/* SPDX-License-Identifier: MIT */
+#ifndef _DRM_PAGEMAP_H_
+#define _DRM_PAGEMAP_H_
+
+#include <linux/dma-direction.h>
+#include <linux/hmm.h>
+#include <linux/types.h>
+
+struct drm_pagemap;
+struct device;
+
+/**
+ * enum drm_interconnect_protocol - Used to identify an interconnect

protocol.
+ */

include/drm/drm_pagemap.h new file +5 -0
+enum drm_interconnect_protocol {
+	DRM_INTERCONNECT_SYSTEM,    /* DMA map is system pages. */
+	DRM_INTERCONNECT_PCIE_P2P,  /* DMA map is PCIE P2P */
+	DRM_INTERCONNECT_DRIVER,    /* DMA map is driver defined */
+	/* A driver can add private values beyond

DRM_INTERCONNECT_DRIVER */

include/drm/drm_pagemap.h new file +5 -0
+};
+
+/**
+ * struct drm_pagemap_dma_addr - DMA address representation.
+ * @addr: The dma address or driver-defined address for driver

private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size is PAGE_SIZE << order).
+ * @dir: The DMA direction.
+ *
+ * Note: There is room for improvement here. We should be able to pack into
+ * 64 bits.
+ */

include/drm/drm_pagemap.h new file +9 -0
+struct drm_pagemap_dma_addr {
+	dma_addr_t addr;
+	u64 proto : 54;
+	u64 order : 8;
+	u64 dir : 2;
+};
+
+/**
+ * drm_pagemap_dma_addr_encode() - Encode a dma address with

metadata
+ * @addr: The dma address or driver-defined address for driver private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size is PAGE_SIZE << order).
+ * @dir: The DMA direction.
+ *
+ * Return: A struct drm_pagemap_dma_addr encoding the above information.
+ */

include/drm/drm_pagemap.h new file +20 -0
+static inline struct drm_pagemap_dma_addr
+drm_pagemap_dma_addr_encode(dma_addr_t addr,
+			    enum drm_interconnect_protocol proto,
+			    unsigned int order,
+			    enum dma_data_direction dir)
+{
+	return (struct drm_pagemap_dma_addr) {
+		.addr = addr,
+		.proto = proto,
+		.order = order,
+		.dir = dir,
+	};
+}
+
+/**
+ * struct drm_pagemap_ops: Ops for a drm-pagemap.
+ */
+struct drm_pagemap_ops {
+	/**
+	 * @map_dma: Map for dma access or provide a virtual address

suitable for

include/drm/drm_pagemap.h new file +5 -0
+	 *
+	 * @dpagemap: The struct drm_pagemap for the page.
+	 * @dev: The dma mapper.
+	 * @page: The page to map.
+	 * @order: The page order of the dma mapping. (Size is

PAGE_SIZE << order).

include/drm/drm_pagemap.h new file +3 -0
+	 * @dir: The transfer direction.
+	 */
+	struct drm_pagemap_dma_addr (*map_dma)(struct drm_pagemap

*dpagemap,

include/drm/drm_pagemap.h new file +4 -0
+					       struct device *dev,
+					       struct page *page,
+					       unsigned int order,
+					       enum

dma_data_direction dir);

include/drm/drm_pagemap.h new file +3 -0
+
+	/**
+	 * @unmap_dma: Unmap a dma address previously obtained using

@map_dma.

include/drm/drm_pagemap.h new file +13 -0
+	 *
+	 * @dpagemap: The struct drm_pagemap for the mapping.
+	 * @dev: The dma unmapper.
+	 * @addr: The dma address obtained when mapping.
+	 */
+	void (*unmap_dma)(struct drm_pagemap *dpagemap,
+			  struct device *dev,
+			  struct drm_pagemap_dma_addr addr);
+
+};
+
+/**
+ * struct drm_pagemap: Additional information for a struct

dev_pagemap
+ * used for device p2p handshaking.
+ * @ops: The struct drm_pagemap_ops.
+ * @dev: The struct drevice owning the device-private memory.
+ */

include/drm/drm_pagemap.h new file +6 -0
+struct drm_pagemap {
+	const struct drm_pagemap_ops *ops;
+	struct device *dev;
+};
+
+#endif
#10 lore raw

Re: [PATCH v4 04/33] drm/pagemap: Add DRM pagemap

Quoted 54 lines · Matthew Brost
Quoted 14 lines · Thomas Hellström
Quoted 13 lines · Matthew Brost

From: Thomas Hellström <[email protected]>

Introduce drm_pagemap ops to map and unmap dma to VRAM resources. In
the
local memory case it's a matter of merely providing an offset into
the
device's physical address. For future p2p the map and unmap functions
may
encode as needed.

Quoted 5 lines
Quoted 4 lines

Similar to how dma-buf works, let the memory provider (drm_pagemap)
provide
the mapping functionality.

Trying to parse all of this.

Quoted 4 lines

It should be noted that the long term idea for dma mapping is to have
that done by the client instead of by the memory provider, which Jason

- Client here is the device mapping the memory.
- Memory provider is the device where the memory is located?

Did I get this correct?

Quoted 9 lines

reminded me of in a discussion on dri-devel. The dma-mapping here is
modeled after how it's done for dma-buf, where the exporter maps dma.

So following that, it might be that we should move these dma- mapping
ops to the drm_gpusvm().

So we move ops to the local client (gpusvm) rather than remote device,
right?

Quoted 6 lines

The situation I can think of, where this might be a problem is that if
the device-private struct page to dma address mapping is not known to
the client.

Quoted 7 lines

I'm not following this but I agree if dma mapping at the client we need
the remote device structure given how the dma mapping API works.

So to wrap it up - what, if anything, do you think we need to do to this
individual patch as part of this series?

I've been thinking a bit more about this, and I think a change we can do is to rename these methods to something along device_map() and device_unmap(). The purpose would be to emphasize that the resulting addresses are typically not meaningful outside of the driver, and not to be confused with standard dma-mapping.

/Thomas

Quoted 165 lines

Matt

Quoted 162 lines

/Thomas

Quoted 155 lines

v3:
 - Move to drm level include
v4:
 - Fix kernel doc (G.G.)

Signed-off-by Thomas Hellström

<[email protected]>
---
 include/drm/drm_pagemap.h | 105
++++++++++++++++++++++++++++++++++++++

 1 file changed, 105 insertions(+)

 create mode 100644 include/drm/drm_pagemap.h

include/drm/drm_pagemap.h new file +13 -0
new file mode 100644
index 000000000000..2b610ccf7e30
@@ -0,0 +1,105 @@
+/* SPDX-License-Identifier: MIT */
+#ifndef _DRM_PAGEMAP_H_
+#define _DRM_PAGEMAP_H_
+
+#include <linux/dma-direction.h>
+#include <linux/hmm.h>
+#include <linux/types.h>
+
+struct drm_pagemap;
+struct device;
+
+/**
+ * enum drm_interconnect_protocol - Used to identify an

interconnect
protocol.
+ */

include/drm/drm_pagemap.h new file +2 -0
+enum drm_interconnect_protocol {
+	DRM_INTERCONNECT_SYSTEM,    /* DMA map is system pages.

*/

include/drm/drm_pagemap.h new file +2 -0
+	DRM_INTERCONNECT_PCIE_P2P,  /* DMA map is PCIE P2P */
+	DRM_INTERCONNECT_DRIVER,    /* DMA map is driver defined

*/

include/drm/drm_pagemap.h new file +1 -0
+	/* A driver can add private values beyond

DRM_INTERCONNECT_DRIVER */

include/drm/drm_pagemap.h new file +5 -0
+};
+
+/**
+ * struct drm_pagemap_dma_addr - DMA address representation.
+ * @addr: The dma address or driver-defined address for driver

private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size is PAGE_SIZE <<
order).
+ * @dir: The DMA direction.
+ *
+ * Note: There is room for improvement here. We should be able to
pack into
+ * 64 bits.
+ */

include/drm/drm_pagemap.h new file +9 -0
+struct drm_pagemap_dma_addr {
+	dma_addr_t addr;
+	u64 proto : 54;
+	u64 order : 8;
+	u64 dir : 2;
+};
+
+/**
+ * drm_pagemap_dma_addr_encode() - Encode a dma address with

metadata
+ * @addr: The dma address or driver-defined address for driver private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size is PAGE_SIZE <<
order).
+ * @dir: The DMA direction.
+ *
+ * Return: A struct drm_pagemap_dma_addr encoding the above information.
+ */

include/drm/drm_pagemap.h new file +3 -0
+static inline struct drm_pagemap_dma_addr
+drm_pagemap_dma_addr_encode(dma_addr_t addr,
+			    enum drm_interconnect_protocol

proto,

include/drm/drm_pagemap.h new file +17 -0
+			    unsigned int order,
+			    enum dma_data_direction dir)
+{
+	return (struct drm_pagemap_dma_addr) {
+		.addr = addr,
+		.proto = proto,
+		.order = order,
+		.dir = dir,
+	};
+}
+
+/**
+ * struct drm_pagemap_ops: Ops for a drm-pagemap.
+ */
+struct drm_pagemap_ops {
+	/**
+	 * @map_dma: Map for dma access or provide a virtual

address
suitable for

include/drm/drm_pagemap.h new file +5 -0
+	 *
+	 * @dpagemap: The struct drm_pagemap for the page.
+	 * @dev: The dma mapper.
+	 * @page: The page to map.
+	 * @order: The page order of the dma mapping. (Size is

PAGE_SIZE << order).

include/drm/drm_pagemap.h new file +3 -0
+	 * @dir: The transfer direction.
+	 */
+	struct drm_pagemap_dma_addr (*map_dma)(struct

drm_pagemap
*dpagemap,

include/drm/drm_pagemap.h new file +1 -0
+					       struct device

*dev,

include/drm/drm_pagemap.h new file +1 -0
+					       struct page

*page,

include/drm/drm_pagemap.h new file +1 -0
+					       unsigned int

order,

include/drm/drm_pagemap.h new file +1 -0
+					       enum

dma_data_direction dir);

include/drm/drm_pagemap.h new file +3 -0
+
+	/**
+	 * @unmap_dma: Unmap a dma address previously obtained

using
@map_dma.

include/drm/drm_pagemap.h new file +13 -0
+	 *
+	 * @dpagemap: The struct drm_pagemap for the mapping.
+	 * @dev: The dma unmapper.
+	 * @addr: The dma address obtained when mapping.
+	 */
+	void (*unmap_dma)(struct drm_pagemap *dpagemap,
+			  struct device *dev,
+			  struct drm_pagemap_dma_addr addr);
+
+};
+
+/**
+ * struct drm_pagemap: Additional information for a struct

dev_pagemap
+ * used for device p2p handshaking.
+ * @ops: The struct drm_pagemap_ops.
+ * @dev: The struct drevice owning the device-private memory.
+ */

include/drm/drm_pagemap.h new file +6 -0
+struct drm_pagemap {
+	const struct drm_pagemap_ops *ops;
+	struct device *dev;
+};
+
+#endif
author maintainer
#11 lore raw

Re: [PATCH v4 04/33] drm/pagemap: Add DRM pagemap

Quoted 63 lines · Thomas Hellström
Quoted 54 lines · Matthew Brost
Quoted 14 lines · Thomas Hellström
Quoted 13 lines · Matthew Brost

From: Thomas Hellström <[email protected]>

Introduce drm_pagemap ops to map and unmap dma to VRAM resources. In
the
local memory case it's a matter of merely providing an offset into
the
device's physical address. For future p2p the map and unmap functions
may
encode as needed.

Quoted 5 lines
Quoted 4 lines

Similar to how dma-buf works, let the memory provider (drm_pagemap)
provide
the mapping functionality.

Trying to parse all of this.

Quoted 4 lines

It should be noted that the long term idea for dma mapping is to have
that done by the client instead of by the memory provider, which Jason

- Client here is the device mapping the memory.
- Memory provider is the device where the memory is located?

Did I get this correct?

Quoted 9 lines

reminded me of in a discussion on dri-devel. The dma-mapping here is
modeled after how it's done for dma-buf, where the exporter maps dma.

So following that, it might be that we should move these dma- mapping
ops to the drm_gpusvm().

So we move ops to the local client (gpusvm) rather than remote device,
right?

Quoted 6 lines

The situation I can think of, where this might be a problem is that if
the device-private struct page to dma address mapping is not known to
the client.

Quoted 7 lines

I'm not following this but I agree if dma mapping at the client we need
the remote device structure given how the dma mapping API works.

So to wrap it up - what, if anything, do you think we need to do to this
individual patch as part of this series?

Quoted 6 lines

I've been thinking a bit more about this, and I think a change we can do is to rename these methods to something along device_map() and device_unmap(). The purpose would be to emphasize that the resulting addresses are typically not meaningful outside of the driver, and not to be confused with standard dma-mapping.

Sure. I can rename this.

Matt

Quoted 169 lines

/Thomas

Quoted 165 lines

Matt

Quoted 162 lines

/Thomas

Quoted 155 lines

v3:
 - Move to drm level include
v4:
 - Fix kernel doc (G.G.)

Signed-off-by Thomas Hellström

<[email protected]>
---
 include/drm/drm_pagemap.h | 105
++++++++++++++++++++++++++++++++++++++

 1 file changed, 105 insertions(+)

 create mode 100644 include/drm/drm_pagemap.h

include/drm/drm_pagemap.h new file +13 -0
new file mode 100644
index 000000000000..2b610ccf7e30
@@ -0,0 +1,105 @@
+/* SPDX-License-Identifier: MIT */
+#ifndef _DRM_PAGEMAP_H_
+#define _DRM_PAGEMAP_H_
+
+#include <linux/dma-direction.h>
+#include <linux/hmm.h>
+#include <linux/types.h>
+
+struct drm_pagemap;
+struct device;
+
+/**
+ * enum drm_interconnect_protocol - Used to identify an

interconnect
protocol.
+ */

include/drm/drm_pagemap.h new file +2 -0
+enum drm_interconnect_protocol {
+	DRM_INTERCONNECT_SYSTEM,    /* DMA map is system pages.

*/

include/drm/drm_pagemap.h new file +2 -0
+	DRM_INTERCONNECT_PCIE_P2P,  /* DMA map is PCIE P2P */
+	DRM_INTERCONNECT_DRIVER,    /* DMA map is driver defined

*/

include/drm/drm_pagemap.h new file +1 -0
+	/* A driver can add private values beyond

DRM_INTERCONNECT_DRIVER */

include/drm/drm_pagemap.h new file +5 -0
+};
+
+/**
+ * struct drm_pagemap_dma_addr - DMA address representation.
+ * @addr: The dma address or driver-defined address for driver

private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size is PAGE_SIZE <<
order).
+ * @dir: The DMA direction.
+ *
+ * Note: There is room for improvement here. We should be able to
pack into
+ * 64 bits.
+ */

include/drm/drm_pagemap.h new file +9 -0
+struct drm_pagemap_dma_addr {
+	dma_addr_t addr;
+	u64 proto : 54;
+	u64 order : 8;
+	u64 dir : 2;
+};
+
+/**
+ * drm_pagemap_dma_addr_encode() - Encode a dma address with

metadata
+ * @addr: The dma address or driver-defined address for driver private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size is PAGE_SIZE <<
order).
+ * @dir: The DMA direction.
+ *
+ * Return: A struct drm_pagemap_dma_addr encoding the above information.
+ */

include/drm/drm_pagemap.h new file +3 -0
+static inline struct drm_pagemap_dma_addr
+drm_pagemap_dma_addr_encode(dma_addr_t addr,
+			    enum drm_interconnect_protocol

proto,

include/drm/drm_pagemap.h new file +17 -0
+			    unsigned int order,
+			    enum dma_data_direction dir)
+{
+	return (struct drm_pagemap_dma_addr) {
+		.addr = addr,
+		.proto = proto,
+		.order = order,
+		.dir = dir,
+	};
+}
+
+/**
+ * struct drm_pagemap_ops: Ops for a drm-pagemap.
+ */
+struct drm_pagemap_ops {
+	/**
+	 * @map_dma: Map for dma access or provide a virtual

address
suitable for

include/drm/drm_pagemap.h new file +5 -0
+	 *
+	 * @dpagemap: The struct drm_pagemap for the page.
+	 * @dev: The dma mapper.
+	 * @page: The page to map.
+	 * @order: The page order of the dma mapping. (Size is

PAGE_SIZE << order).

include/drm/drm_pagemap.h new file +3 -0
+	 * @dir: The transfer direction.
+	 */
+	struct drm_pagemap_dma_addr (*map_dma)(struct

drm_pagemap
*dpagemap,

include/drm/drm_pagemap.h new file +1 -0
+					       struct device

*dev,

include/drm/drm_pagemap.h new file +1 -0
+					       struct page

*page,

include/drm/drm_pagemap.h new file +1 -0
+					       unsigned int

order,

include/drm/drm_pagemap.h new file +1 -0
+					       enum

dma_data_direction dir);

include/drm/drm_pagemap.h new file +3 -0
+
+	/**
+	 * @unmap_dma: Unmap a dma address previously obtained

using
@map_dma.

include/drm/drm_pagemap.h new file +13 -0
+	 *
+	 * @dpagemap: The struct drm_pagemap for the mapping.
+	 * @dev: The dma unmapper.
+	 * @addr: The dma address obtained when mapping.
+	 */
+	void (*unmap_dma)(struct drm_pagemap *dpagemap,
+			  struct device *dev,
+			  struct drm_pagemap_dma_addr addr);
+
+};
+
+/**
+ * struct drm_pagemap: Additional information for a struct

dev_pagemap
+ * used for device p2p handshaking.
+ * @ops: The struct drm_pagemap_ops.
+ * @dev: The struct drevice owning the device-private memory.
+ */

include/drm/drm_pagemap.h new file +6 -0
+struct drm_pagemap {
+	const struct drm_pagemap_ops *ops;
+	struct device *dev;
+};
+
+#endif
author maintainer
#12 lore raw

PATCH v4 5/33 drm/xe/bo: Introduce xe_bo_put_async

In reply to Matthew Brost

From: Thomas Hellström <[email protected]>

Introduce xe_bo_put_async to put a bo where the context is such that the bo destructor can't run due to lockdep problems or atomic context.

If the put is the final put, freeing will be done from a work item.

---
 drivers/gpu/drm/xe/xe_bo.c           | 25 +++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_bo.h           | 13 +++++++++++++
 drivers/gpu/drm/xe/xe_device.c       |  3 +++
 drivers/gpu/drm/xe/xe_device_types.h |  8 ++++++++
 4 files changed, 49 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +25 -0
index fb1629d9d566..e914a60b8afc 100644
@@ -2544,6 +2544,31 @@ void xe_bo_put_commit(struct llist_head *deferred)
 		drm_gem_object_free(&bo->ttm.base.refcount);
 }
 
+static void xe_bo_dev_work_func(struct work_struct *work)
+{
+	struct xe_bo_dev *bo_dev = container_of(work, typeof(*bo_dev), async_free);
+
+	xe_bo_put_commit(&bo_dev->async_list);
+}
+
+/**
+ * xe_bo_dev_init() - Initialize BO dev to manage async BO freeing
+ * @bo_dev: The BO dev structure
+ */
+void xe_bo_dev_init(struct xe_bo_dev *bo_dev)
+{
+	INIT_WORK(&bo_dev->async_free, xe_bo_dev_work_func);
+}
+
+/**
+ * xe_bo_dev_fini() - Finalize BO dev managing async BO freeing
+ * @bo_dev: The BO dev structure
+ */
+void xe_bo_dev_fini(struct xe_bo_dev *bo_dev)
+{
+	flush_work(&bo_dev->async_free);
+}
+
 void xe_bo_put(struct xe_bo *bo)
 {
 	struct xe_tile *tile;
drivers/gpu/drm/xe/xe_bo.h +13 -0
index 04995c5ced32..ce55a2bb13f6 100644
@@ -317,6 +317,19 @@ xe_bo_put_deferred(struct xe_bo *bo, struct llist_head *deferred)
 
 void xe_bo_put_commit(struct llist_head *deferred);
 
+static inline void
+xe_bo_put_async(struct xe_bo *bo)
+{
+	struct xe_bo_dev *bo_device = &xe_bo_device(bo)->bo_device;
+
+	if (xe_bo_put_deferred(bo, &bo_device->async_list))
+		schedule_work(&bo_device->async_free);
+}
+
+void xe_bo_dev_init(struct xe_bo_dev *bo_device);
+
+void xe_bo_dev_fini(struct xe_bo_dev *bo_device);
+
 struct sg_table *xe_bo_sg(struct xe_bo *bo);
 
 /*
drivers/gpu/drm/xe/xe_device.c +3 -0
index 8fedc72e9db4..5fac3d40cc8e 100644
@@ -387,6 +387,8 @@ static void xe_device_destroy(struct drm_device *dev, void *dummy)
 {
 	struct xe_device *xe = to_xe_device(dev);
 
+	xe_bo_dev_fini(&xe->bo_device);
+
 	if (xe->preempt_fence_wq)
 		destroy_workqueue(xe->preempt_fence_wq);
 
@@ -424,6 +426,7 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,
 	if (WARN_ON(err))
 		goto err;
 
+	xe_bo_dev_init(&xe->bo_device);
 	err = drmm_add_action_or_reset(&xe->drm, xe_device_destroy, NULL);
 	if (err)
 		goto err;
drivers/gpu/drm/xe/xe_device_types.h +8 -0
index 89f532b67bc4..71151532e28f 100644
@@ -519,6 +519,14 @@ struct xe_device {
 		int mode;
 	} wedged;
 
+	/** @bo_device: Struct to control async free of BOs */
+	struct xe_bo_dev {
+		/** @async_free: Free worker */
+		struct work_struct async_free;
+		/** @async_list: List of BOs to be freed */
+		struct llist_head async_list;
+	} bo_device;
+
 	/** @pmu: performance monitoring unit */
 	struct xe_pmu pmu;
 
Signature · 1 line
2.34.1
#13 lore raw

Re: [PATCH v4 05/33] drm/xe/bo: Introduce xe_bo_put_async

Quoted 64 lines · Matthew Brost

From: Thomas Hellström <[email protected]>

Introduce xe_bo_put_async to put a bo where the context is such that the bo destructor can't run due to lockdep problems or atomic context.

If the put is the final put, freeing will be done from a work item.

---
 drivers/gpu/drm/xe/xe_bo.c           | 25 +++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_bo.h           | 13 +++++++++++++ drivers/gpu/drm/xe/xe_device.c       |  3 +++ drivers/gpu/drm/xe/xe_device_types.h |  8 ++++++++

 4 files changed, 49 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +0 -0
index fb1629d9d566..e914a60b8afc 100644
@@ -2544,6 +2544,31 @@ void xe_bo_put_commit(struct llist_head
 *deferred)
  		drm_gem_object_free(&bo->ttm.base.refcount);
  }

drivers/gpu/drm/xe/xe_bo.c +3 -0
+static void xe_bo_dev_work_func(struct work_struct *work)
+{
+	struct xe_bo_dev *bo_dev = container_of(work,

typeof(*bo_dev), async_free);

drivers/gpu/drm/xe/xe_bo.c +22 -0
+
+	xe_bo_put_commit(&bo_dev->async_list);
+}
+
+/**
+ * xe_bo_dev_init() - Initialize BO dev to manage async BO freeing
+ * @bo_dev: The BO dev structure
+ */
+void xe_bo_dev_init(struct xe_bo_dev *bo_dev)
+{
+	INIT_WORK(&bo_dev->async_free, xe_bo_dev_work_func);
+}
+
+/**
+ * xe_bo_dev_fini() - Finalize BO dev managing async BO freeing
+ * @bo_dev: The BO dev structure
+ */
+void xe_bo_dev_fini(struct xe_bo_dev *bo_dev)
+{
+	flush_work(&bo_dev->async_free);
+}
+

 void xe_bo_put(struct xe_bo *bo)
 {
  struct xe_tile *tile;

drivers/gpu/drm/xe/xe_bo.h +0 -0
index 04995c5ced32..ce55a2bb13f6 100644
@@ -317,6 +317,19 @@ xe_bo_put_deferred(struct xe_bo *bo, struct
 llist_head *deferred)


 void xe_bo_put_commit(struct llist_head *deferred);

drivers/gpu/drm/xe/xe_bo.h xe_bo_put_deferred(struct xe_bo *bo, struct llist_head *deferred) · line 321

Quoted 2 lines
drivers/gpu/drm/xe/xe_bo.h +2 -0
+static inline void
+xe_bo_put_async(struct xe_bo *bo)

Needs kerneldoc. I will rebase my multi-device series on this one, Let me know if you'll add that or if I should do it when rebasing my multi- device series on this one.

Quoted 59 lines
drivers/gpu/drm/xe/xe_bo.h +11 -0
+{
+	struct xe_bo_dev *bo_device = &xe_bo_device(bo)->bo_device;
+
+	if (xe_bo_put_deferred(bo, &bo_device->async_list))
+		schedule_work(&bo_device->async_free);
+}
+
+void xe_bo_dev_init(struct xe_bo_dev *bo_device);
+
+void xe_bo_dev_fini(struct xe_bo_dev *bo_device);
+

 struct sg_table *xe_bo_sg(struct xe_bo *bo);

 /*

drivers/gpu/drm/xe/xe_device.c +0 -0
index 8fedc72e9db4..5fac3d40cc8e 100644
@@ -387,6 +387,8 @@ static void xe_device_destroy(struct drm_device
 *dev, void *dummy)
  {
  	struct xe_device *xe = to_xe_device(dev);

drivers/gpu/drm/xe/xe_device.c +2 -0
+	xe_bo_dev_fini(&xe->bo_device);
+

  if (xe->preempt_fence_wq) destroy_workqueue(xe->preempt_fence_wq);

drivers/gpu/drm/xe/xe_device.c +0 -0
@@ -424,6 +426,7 @@ struct xe_device *xe_device_create(struct pci_dev
 *pdev,
  	if (WARN_ON(err))
  		goto err;

drivers/gpu/drm/xe/xe_device.c +1 -0
+	xe_bo_dev_init(&xe->bo_device);

  err = drmm_add_action_or_reset(&xe->drm, xe_device_destroy, NULL);
  if (err)
  goto err;

drivers/gpu/drm/xe/xe_device_types.h +0 -0
index 89f532b67bc4..71151532e28f 100644
@@ -519,6 +519,14 @@ struct xe_device {
  		int mode;
  	} wedged;

drivers/gpu/drm/xe/xe_device_types.h +8 -0
+	/** @bo_device: Struct to control async free of BOs */
+	struct xe_bo_dev {
+		/** @async_free: Free worker */
+		struct work_struct async_free;
+		/** @async_list: List of BOs to be freed */
+		struct llist_head async_list;
+	} bo_device;
+

  /** @pmu: performance monitoring unit */
  struct xe_pmu pmu;

author maintainer
#14 lore raw

Re: [PATCH v4 05/33] drm/xe/bo: Introduce xe_bo_put_async

Quoted 63 lines · Thomas Hellström
Quoted 62 lines · Matthew Brost

From: Thomas Hellström <[email protected]>

Introduce xe_bo_put_async to put a bo where the context is such that the bo destructor can't run due to lockdep problems or atomic context.

If the put is the final put, freeing will be done from a work item.

---
 drivers/gpu/drm/xe/xe_bo.c           | 25 +++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_bo.h           | 13 +++++++++++++ drivers/gpu/drm/xe/xe_device.c       |  3 +++ drivers/gpu/drm/xe/xe_device_types.h |  8 ++++++++

 4 files changed, 49 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +0 -0
index fb1629d9d566..e914a60b8afc 100644
@@ -2544,6 +2544,31 @@ void xe_bo_put_commit(struct llist_head
 *deferred)
  		drm_gem_object_free(&bo->ttm.base.refcount);
  }

drivers/gpu/drm/xe/xe_bo.c +3 -0
+static void xe_bo_dev_work_func(struct work_struct *work)
+{
+	struct xe_bo_dev *bo_dev = container_of(work,

typeof(*bo_dev), async_free);

drivers/gpu/drm/xe/xe_bo.c +22 -0
+
+	xe_bo_put_commit(&bo_dev->async_list);
+}
+
+/**
+ * xe_bo_dev_init() - Initialize BO dev to manage async BO freeing
+ * @bo_dev: The BO dev structure
+ */
+void xe_bo_dev_init(struct xe_bo_dev *bo_dev)
+{
+	INIT_WORK(&bo_dev->async_free, xe_bo_dev_work_func);
+}
+
+/**
+ * xe_bo_dev_fini() - Finalize BO dev managing async BO freeing
+ * @bo_dev: The BO dev structure
+ */
+void xe_bo_dev_fini(struct xe_bo_dev *bo_dev)
+{
+	flush_work(&bo_dev->async_free);
+}
+

 void xe_bo_put(struct xe_bo *bo)
 {
  struct xe_tile *tile;

drivers/gpu/drm/xe/xe_bo.h +0 -0
index 04995c5ced32..ce55a2bb13f6 100644
@@ -317,6 +317,19 @@ xe_bo_put_deferred(struct xe_bo *bo, struct
 llist_head *deferred)

Quoted 9 lines

drivers/gpu/drm/xe/xe_bo.h xe_bo_put_deferred(struct xe_bo *bo, struct llist_head *deferred) · line 321

Quoted 4 lines

 void xe_bo_put_commit(struct llist_head *deferred);

drivers/gpu/drm/xe/xe_bo.h +2 -0
+static inline void
+xe_bo_put_async(struct xe_bo *bo)

Needs kerneldoc. I will rebase my multi-device series on this one, Let me know if you'll add that or if I should do it when rebasing my multi- device series on this one.

Yep. Added kernel for structures / exported functions but missed this inline. I should be able to write something here.

Matt

Quoted 60 lines
Quoted 59 lines
drivers/gpu/drm/xe/xe_bo.h +11 -0
+{
+	struct xe_bo_dev *bo_device = &xe_bo_device(bo)->bo_device;
+
+	if (xe_bo_put_deferred(bo, &bo_device->async_list))
+		schedule_work(&bo_device->async_free);
+}
+
+void xe_bo_dev_init(struct xe_bo_dev *bo_device);
+
+void xe_bo_dev_fini(struct xe_bo_dev *bo_device);
+

 struct sg_table *xe_bo_sg(struct xe_bo *bo);

 /*

drivers/gpu/drm/xe/xe_device.c +0 -0
index 8fedc72e9db4..5fac3d40cc8e 100644
@@ -387,6 +387,8 @@ static void xe_device_destroy(struct drm_device
 *dev, void *dummy)
  {
  	struct xe_device *xe = to_xe_device(dev);

drivers/gpu/drm/xe/xe_device.c +2 -0
+	xe_bo_dev_fini(&xe->bo_device);
+

  if (xe->preempt_fence_wq) destroy_workqueue(xe->preempt_fence_wq);

drivers/gpu/drm/xe/xe_device.c +0 -0
@@ -424,6 +426,7 @@ struct xe_device *xe_device_create(struct pci_dev
 *pdev,
  	if (WARN_ON(err))
  		goto err;

drivers/gpu/drm/xe/xe_device.c +1 -0
+	xe_bo_dev_init(&xe->bo_device);

  err = drmm_add_action_or_reset(&xe->drm, xe_device_destroy, NULL);
  if (err)
  goto err;

drivers/gpu/drm/xe/xe_device_types.h +0 -0
index 89f532b67bc4..71151532e28f 100644
@@ -519,6 +519,14 @@ struct xe_device {
  		int mode;
  	} wedged;

drivers/gpu/drm/xe/xe_device_types.h +8 -0
+	/** @bo_device: Struct to control async free of BOs */
+	struct xe_bo_dev {
+		/** @async_free: Free worker */
+		struct work_struct async_free;
+		/** @async_list: List of BOs to be freed */
+		struct llist_head async_list;
+	} bo_device;
+

  /** @pmu: performance monitoring unit */
  struct xe_pmu pmu;

author maintainer
#15 lore raw

PATCH v4 6/33 drm/gpusvm: Add support for GPU Shared Virtual Memory

In reply to Matthew Brost

This patch introduces support for GPU Shared Virtual Memory (SVM) in the Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.

The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also provides mechanisms for allocating, deallocating, and migrating memory regions between system RAM and GPU VRAM.

This is largely inspired by GPUVM.

v2:
- Take order into account in check pages
- Clear range->pages in get pages error
- Drop setting dirty or accessed bit in get pages (Vetter)
- Remove mmap assert for cpu faults
- Drop mmap write lock abuse (Vetter, Christian)
- Decouple zdd from range (Vetter, Oak)
- Add drm_gpusvm_range_evict, make it work with coherent pages
- Export drm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
- mmget/put in drm_gpusvm_evict_to_sram
- Drop range->vram_alloation variable
- Don't return in drm_gpusvm_evict_to_sram until all pages detached
- Don't warn on mixing sram and device pages
- Update kernel doc
- Add coherent page support to get pages
- Use DMA_FROM_DEVICE rather than DMA_BIDIRECTIONAL
- Add struct drm_gpusvm_vram and ops (Thomas)
- Update the range's seqno if the range is valid (Thomas)
- Remove the is_unmapped check before hmm_range_fault (Thomas)
- Use drm_pagemap (Thomas)
- Drop kfree_mapping (Thomas)
- dma mapp pages under notifier lock (Thomas)
- Remove ctx.prefault
- Remove ctx.mmap_locked
- Add ctx.check_pages
- s/vram/devmem (Thomas)
v3:
- Fix memory leak drm_gpusvm_range_get_pages
- Only migrate pages with same zdd on CPU fault
- Loop over al VMAs in drm_gpusvm_range_evict
- Make GPUSVM a drm level module
- GPL or MIT license
- Update main kernel doc (Thomas)
- Prefer foo() vs foo for functions in kernel doc (Thomas)
- Prefer functions over macros (Thomas)
- Use unsigned long vs u64 for addresses (Thomas)
- Use standard interval_tree (Thomas)
- s/drm_gpusvm_migration_put_page/drm_gpusvm_migration_unlock_put_page (Thomas)
- Drop err_out label in drm_gpusvm_range_find_or_insert (Thomas)
- Fix kernel doc in drm_gpusvm_range_free_pages (Thomas)
- Newlines between functions defs in header file (Thomas)
- Drop shall language in driver vfunc kernel doc (Thomas)
- Move some static inlines from head to C file (Thomas)
- Don't allocate pages under page lock in drm_gpusvm_migrate_populate_ram_pfn (Thomas)
- Change check_pages to a threshold
v4:
- Fix NULL ptr deref in drm_gpusvm_migrate_populate_ram_pfn (Thomas, Himal)
- Fix check pages threshold
- Check for range being unmapped under notifier lock in get pages (Testing)
- Fix characters per line
- Drop WRITE_ONCE for zdd->devmem_allocation assignment (Thomas)
- Use completion for devmem_allocation->detached (Thomas)
- Make GPU SVM depend on ZONE_DEVICE (CI)
- Use hmm_range_fault for eviction (Thomas)
- Drop zdd worker (Thomas)

---
 drivers/gpu/drm/Kconfig      |    9 +
 drivers/gpu/drm/Makefile     |    1 +
 drivers/gpu/drm/drm_gpusvm.c | 2240 ++++++++++++++++++++++++++++++++++
 include/drm/drm_gpusvm.h     |  445 +++++++
 4 files changed, 2695 insertions(+)
 create mode 100644 drivers/gpu/drm/drm_gpusvm.c
 create mode 100644 include/drm/drm_gpusvm.h
drivers/gpu/drm/Kconfig +9 -0
index fbef3f471bd0..f03862e379fb 100644
@@ -278,6 +278,15 @@ config DRM_GPUVM
 	  GPU-VM representation providing helpers to manage a GPUs virtual
 	  address space
 
+config DRM_GPUSVM
+	tristate
+	depends on DRM
+	depends on DEVICE_MIGRATION
+	depends on ZONE_DEVICE
+	help
+	  GPU-SVM representation providing helpers to manage a GPUs shared
+	  virtual memory
+
 config DRM_BUDDY
 	tristate
 	depends on DRM
drivers/gpu/drm/Makefile +1 -0
index 85af94bb907d..ca03df8d2729 100644
@@ -104,6 +104,7 @@ obj-$(CONFIG_DRM_PANEL_BACKLIGHT_QUIRKS) += drm_panel_backlight_quirks.o
 #
 obj-$(CONFIG_DRM_EXEC) += drm_exec.o
 obj-$(CONFIG_DRM_GPUVM) += drm_gpuvm.o
+obj-$(CONFIG_DRM_GPUSVM) += drm_gpusvm.o
 
 obj-$(CONFIG_DRM_BUDDY) += drm_buddy.o
 
drivers/gpu/drm/drm_gpusvm.c new file +2240 -0

2,240 lines not on this page · raw message

include/drm/drm_gpusvm.h new file +445 -0

445 lines not on this page · raw message

Signature · 1 line
2.34.1
#16 lore raw

Re: [PATCH v4 06/33] drm/gpusvm: Add support for GPU Shared Virtual Memory

Quoted 77 lines · Matthew Brost

This patch introduces support for GPU Shared Virtual Memory (SVM) in the
Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.

The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also
provides mechanisms for allocating, deallocating, and migrating memory
regions between system RAM and GPU VRAM.

This is largely inspired by GPUVM.

v2:
 - Take order into account in check pages
 - Clear range->pages in get pages error
 - Drop setting dirty or accessed bit in get pages (Vetter)
 - Remove mmap assert for cpu faults
 - Drop mmap write lock abuse (Vetter, Christian)
 - Decouple zdd from range (Vetter, Oak)
 - Add drm_gpusvm_range_evict, make it work with coherent pages
 - Export drm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
 - mmget/put in drm_gpusvm_evict_to_sram
 - Drop range->vram_alloation variable
 - Don't return in drm_gpusvm_evict_to_sram until all pages detached
 - Don't warn on mixing sram and device pages
 - Update kernel doc
 - Add coherent page support to get pages
 - Use DMA_FROM_DEVICE rather than DMA_BIDIRECTIONAL
 - Add struct drm_gpusvm_vram and ops (Thomas)
 - Update the range's seqno if the range is valid (Thomas)
 - Remove the is_unmapped check before hmm_range_fault (Thomas)
 - Use drm_pagemap (Thomas)
 - Drop kfree_mapping (Thomas)
 - dma mapp pages under notifier lock (Thomas)
 - Remove ctx.prefault
 - Remove ctx.mmap_locked
 - Add ctx.check_pages
 - s/vram/devmem (Thomas)
v3:
 - Fix memory leak drm_gpusvm_range_get_pages
 - Only migrate pages with same zdd on CPU fault
 - Loop over al VMAs in drm_gpusvm_range_evict
 - Make GPUSVM a drm level module
 - GPL or MIT license
 - Update main kernel doc (Thomas)
 - Prefer foo() vs foo for functions in kernel doc (Thomas)
 - Prefer functions over macros (Thomas)
 - Use unsigned long vs u64 for addresses (Thomas)
 - Use standard interval_tree (Thomas)
 - s/drm_gpusvm_migration_put_page/drm_gpusvm_migration_unlock_put_page (Thomas)
 - Drop err_out label in drm_gpusvm_range_find_or_insert (Thomas)
 - Fix kernel doc in drm_gpusvm_range_free_pages (Thomas)
 - Newlines between functions defs in header file (Thomas)
 - Drop shall language in driver vfunc kernel doc (Thomas)
 - Move some static inlines from head to C file (Thomas)
 - Don't allocate pages under page lock in drm_gpusvm_migrate_populate_ram_pfn (Thomas)
 - Change check_pages to a threshold
v4:
 - Fix NULL ptr deref in drm_gpusvm_migrate_populate_ram_pfn (Thomas, Himal)
 - Fix check pages threshold
 - Check for range being unmapped under notifier lock in get pages (Testing)
 - Fix characters per line
 - Drop WRITE_ONCE for zdd->devmem_allocation assignment (Thomas)
 - Use completion for devmem_allocation->detached (Thomas)
 - Make GPU SVM depend on ZONE_DEVICE (CI)
 - Use hmm_range_fault for eviction (Thomas)
 - Drop zdd worker (Thomas)

Quoted 6 lines

Let's consider my minor contributions to this version as review comments for now, so no SOB needed.

Quoted 1,699 lines

---
 drivers/gpu/drm/Kconfig      |    9 +
 drivers/gpu/drm/Makefile     |    1 +
 drivers/gpu/drm/drm_gpusvm.c | 2240
++++++++++++++++++++++++++++++++++
 include/drm/drm_gpusvm.h     |  445 +++++++

 4 files changed, 2695 insertions(+)

 create mode 100644 drivers/gpu/drm/drm_gpusvm.c
 create mode 100644 include/drm/drm_gpusvm.h

drivers/gpu/drm/Kconfig +0 -0
index fbef3f471bd0..f03862e379fb 100644
@@ -278,6 +278,15 @@ config DRM_GPUVM

    GPU-VM representation providing helpers to manage a GPUs virtual
    address space

drivers/gpu/drm/Kconfig +7 -0
+config DRM_GPUSVM
+	tristate
+	depends on DRM
+	depends on DEVICE_MIGRATION
+	depends on ZONE_DEVICE
+	help
+	  GPU-SVM representation providing helpers to manage a GPUs

shared

drivers/gpu/drm/Kconfig +2 -0
+	  virtual memory
+

 config DRM_BUDDY
  tristate
  depends on DRM

drivers/gpu/drm/Makefile +0 -0
index 85af94bb907d..ca03df8d2729 100644
@@ -104,6 +104,7 @@ obj-$(CONFIG_DRM_PANEL_BACKLIGHT_QUIRKS) +=

drm_panel_backlight_quirks.o
 #
 obj-$(CONFIG_DRM_EXEC) += drm_exec.o
 obj-$(CONFIG_DRM_GPUVM) += drm_gpuvm.o
+obj-$(CONFIG_DRM_GPUSVM) += drm_gpusvm.o

 obj-$(CONFIG_DRM_BUDDY) += drm_buddy.o

drivers/gpu/drm/drm_gpusvm.c new file +25 -0
new file mode 100644
index 000000000000..1c63da4d3cc2
@@ -0,0 +1,2240 @@
+// SPDX-License-Identifier: GPL-2.0-only OR MIT
+/*
+ * Copyright © 2024 Intel Corporation
+ *
+ * Authors:
+ *     Matthew Brost <[email protected]>
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/hmm.h>
+#include <linux/memremap.h>
+#include <linux/migrate.h>
+#include <linux/mm_types.h>
+#include <linux/pagemap.h>
+#include <linux/slab.h>
+
+#include <drm/drm_device.h>
+#include <drm/drm_gpusvm.h>
+#include <drm/drm_pagemap.h>
+#include <drm/drm_print.h>
+
+/**
+ * DOC: Overview
+ *
+ * GPU Shared Virtual Memory (GPU SVM) layer for the Direct

Rendering Manager (DRM)
+ *
+ * The GPU SVM layer is a component of the DRM framework designed to manage shared
+ * virtual memory between the CPU and GPU. It enables efficient data exchange and
+ * processing for GPU-accelerated applications by allowing memory sharing and
+ * synchronization between the CPU's and GPU's virtual address spaces.
+ *
+ * Key GPU SVM Components:
+ * - Notifiers: Notifiers: Used for tracking memory intervals and notifying the
+ * GPU of changes, notifiers are sized based on a GPU
SVM
+ * initialization parameter, with a recommendation of
512M or
+ * larger. They maintain a Red-BlacK tree and a list of
ranges that
+ * fall within the notifier interval. Notifiers are
tracked within
+ * a GPU SVM Red-BlacK tree and list and are
dynamically inserted
+ * or removed as ranges within the interval are created
or
+ * destroyed.
+ * - Ranges: Represent memory ranges mapped in a DRM device and managed
+ *      by GPU SVM. They are sized based on an array of chunk
sizes, which
+ *      is a GPU SVM initialization parameter, and the CPU
address space.
+ *      Upon GPU fault, the largest aligned chunk that fits
within the
+ *      faulting CPU address space is chosen for the range
size. Ranges are
+ *      expected to be dynamically allocated on GPU fault and
removed on an
+ *      MMU notifier UNMAP event. As mentioned above, ranges
are tracked in
+ *      a notifier's Red-Black tree.
+ * - Operations: Define the interface for driver-specific GPU SVM operations
+ *               such as range allocation, notifier allocation, and
+ *               invalidations.
+ * - Device Memory Allocations: Embedded structure containing enough information
+ *                              for GPU SVM to migrate to / from device memory.
+ * - Device Memory Operations: Define the interface for driver- specific device
+ *                             memory operations release memory, populate pfns,
+ *                             and copy to / from device memory.
+ *
+ * This layer provides interfaces for allocating, mapping, migrating, and
+ * releasing memory ranges between the CPU and GPU. It handles all core memory
+ * management interactions (DMA mapping, HMM, and migration) and provides
+ * driver-specific virtual functions (vfuncs). This infrastructure is sufficient
+ * to build the expected driver components for an SVM implementation as detailed
+ * below.
+ *
+ * Expected Driver Components:
+ * - GPU page fault handler: Used to create ranges and notifiers based on the
+ *      fault address, optionally migrate the
range to
+ *      device memory, and create GPU bindings.
+ * - Garbage collector: Used to unmap and destroy GPU bindings for ranges.
+ * Ranges are expected to be added to the
garbage collector
+ * upon a MMU_NOTIFY_UNMAP event in notifier
callback.
+ * - Notifier callback: Used to invalidate and DMA unmap GPU bindings for
+ * ranges.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Locking
+ *
+ * GPU SVM handles locking for core MM interactions, i.e., it

locks/unlocks the
+ * mmap lock as needed.
+ *
+ * GPU SVM introduces a global notifier lock, which safeguards the notifier's
+ * range RB tree and list, as well as the range's DMA mappings and sequence
+ * number. GPU SVM manages all necessary locking and unlocking operations,
+ * except for the recheck range's pages being valid
+ * (drm_gpusvm_range_pages_valid) when the driver is committing GPU bindings. This
+ * lock corresponds to the 'driver->update' lock mentioned in the HMM
+ * documentation (TODO: Link). Future revisions may transition from a GPU SVM
+ * global lock to a per-notifier lock if finer-grained locking is deemed
+ * necessary.
+ *
+ * In addition to the locking mentioned above, the driver should implement a
+ * lock to safeguard core GPU SVM function calls that modify state, such as
+ * drm_gpusvm_range_find_or_insert and drm_gpusvm_range_remove. This lock is
+ * denoted as 'driver_svm_lock' in code examples. Finer grained driver side
+ * locking should also be possible for concurrent GPU fault processing within a
+ * single GPU SVM. The 'driver_svm_lock' can be via drm_gpusvm_driver_set_lock
+ * to add annotations to GPU SVM.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Migration
+ *
+ * The migration support is quite simple, allowing migration between

RAM and
+ * device memory at the range granularity. For example, GPU SVM currently does not
+ * support mixing RAM and device memory pages within a range. This means that upon GPU
+ * fault, the entire range can be migrated to device memory, and upon CPU fault, the
+ * entire range is migrated to RAM. Mixed RAM and device memory storage within a range
+ * could be added in the future if required.
+ *
+ * The reasoning for only supporting range granularity is as
follows: it
+ * simplifies the implementation, and range sizes are driver-defined and should
+ * be relatively small.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Partial Unmapping of Ranges
+ *
+ * Partial unmapping of ranges (e.g., 1M out of 2M is unmapped by

CPU resulting
+ * in MMU_NOTIFY_UNMAP event) presents several challenges, with the main one
+ * being that a subset of the range still has CPU and GPU mappings. If the
+ * backing store for the range is in device memory, a subset of the backing store has
+ * references. One option would be to split the range and device memory backing store,
+ * but the implementation for this would be quite complicated. Given that
+ * partial unmappings are rare and driver-defined range sizes are relatively
+ * small, GPU SVM does not support splitting of ranges.
+ *
+ * With no support for range splitting, upon partial unmapping of a range, the
+ * driver is expected to invalidate and destroy the entire range. If the range
+ * has device memory as its backing, the driver is also expected to migrate any
+ * remaining pages back to RAM.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Examples
+ *
+ * This section provides three examples of how to build the expected

driver
+ * components: the GPU page fault handler, the garbage collector, and the
+ * notifier callback.
+ *
+ * The generic code provided does not include logic for complex migration
+ * policies, optimized invalidations, fined grained driver locking, or other
+ * potentially required driver locking (e.g., DMA-resv locks).
+ *
+ * 1) GPU page fault handler
+ *
+ * int driver_bind_range(struct drm_gpusvm *gpusvm, struct
drm_gpusvm_range *range)
+ * {
+ * int err = 0;
+ *
+ * driver_alloc_and_setup_memory_for_bind(gpusvm,
range);
+ *
+ * drm_gpusvm_notifier_lock(gpusvm);
+ * if (drm_gpusvm_range_pages_valid(range))
+ * driver_commit_bind(gpusvm, range);
+ * else
+ * err = -EAGAIN;
+ * drm_gpusvm_notifier_unlock(gpusvm);
+ *
+ * return err;
+ * }
+ *
+ * int driver_gpu_fault(struct drm_gpusvm *gpusvm, unsigned
long fault_addr,
+ *      unsigned long gpuva_start, unsigned
long gpuva_end)
+ * {
+ * struct drm_gpusvm_ctx ctx = {};
+ * int err;
+ *
+ * driver_svm_lock();
+ * retry:
+ * // Always process UNMAPs first so view of GPU SVM
ranges is current
+ * driver_garbage_collector(gpusvm);
+ *
+ * range = drm_gpusvm_range_find_or_insert(gpusvm,
fault_addr,
+ * gpuva_start,
gpuva_end,
+ *         &ctx);
+ * if (IS_ERR(range)) {
+ * err = PTR_ERR(range);
+ * goto unlock;
+ * }
+ *
+ * if (driver_migration_policy(range)) {
+ * devmem = driver_alloc_devmem();
+ * err = drm_gpusvm_migrate_to_devmem(gpusvm,
range,
+ *
devmem_allocation,
+ *    &ctx);
+ * if (err) // CPU mappings may have
changed
+ * goto retry;
+ * }
+ *
+ * err = drm_gpusvm_range_get_pages(gpusvm, range,
&ctx);
+ * if (err == -EOPNOTSUPP || err == -EFAULT || err == -
EPERM) { // CPU mappings changed
+ * if (err == -EOPNOTSUPP)
+ * drm_gpusvm_range_evict(gpusvm,
range);
+ * goto retry;
+ * } else if (err) {
+ * goto unlock;
+ * }
+ *
+ * err = driver_bind_range(gpusvm, range);
+ * if (err == -EAGAIN) // CPU mappings changed
+ * goto retry
+ *
+ * unlock:
+ * driver_svm_unlock();
+ * return err;
+ * }
+ *
+ * 2) Garbage Collector.
+ *
+ * void __driver_garbage_collector(struct drm_gpusvm *gpusvm,
+ * struct drm_gpusvm_range
*range)
+ * {
+ * assert_driver_svm_locked(gpusvm);
+ *
+ * // Partial unmap, migrate any remaining device
memory pages back to RAM
+ * if (range->flags.partial_unmap)
+ * drm_gpusvm_range_evict(gpusvm, range);
+ *
+ * driver_unbind_range(range);
+ * drm_gpusvm_range_remove(gpusvm, range);
+ * }
+ *
+ * void driver_garbage_collector(struct drm_gpusvm *gpusvm)
+ * {
+ * assert_driver_svm_locked(gpusvm);
+ *
+ * for_each_range_in_garbage_collector(gpusvm, range)
+ * __driver_garbage_collector(gpusvm, range);
+ * }
+ *
+ * 3) Notifier callback.
+ *
+ * void driver_invalidation(struct drm_gpusvm *gpusvm,
+ * struct drm_gpusvm_notifier
*notifier,
+ * const struct mmu_notifier_range
*mmu_range)
+ * {
+ * struct drm_gpusvm_ctx ctx = { .in_notifier = true,
};
+ * struct drm_gpusvm_range *range = NULL;
+ *
+ * driver_invalidate_device_pages(gpusvm, mmu_range-

Quoted 1 line

start, mmu_range->end);

+ *
+ * drm_gpusvm_for_each_range(range, notifier,
mmu_range->start,
+ *   mmu_range->end) {
+ * drm_gpusvm_range_unmap_pages(gpusvm, range,
&ctx);
+ *
+ * if (mmu_range->event != MMU_NOTIFY_UNMAP)
+ * continue;
+ *
+ * drm_gpusvm_range_set_unmapped(range,
mmu_range);
+ * driver_garbage_collector_add(gpusvm, range);
+ * }
+ * }
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+
+/**
+ * npages_in_range() - Calculate the number of pages in a given

range
+ * @start: The start address of the range
+ * @end: The end address of the range
+ *
+ * This macro calculates the number of pages in a given memory range,
+ * specified by the start and end addresses. It divides the difference
+ * between the end and start addresses by the page size (PAGE_SIZE) to
+ * determine the number of pages in the range.
+ *
+ * Returns: The number of pages in the specified range.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+static unsigned long
+npages_in_range(unsigned long start, unsigned long end)
+{
+	return (end - start) >> PAGE_SHIFT;
+}
+
+/**
+ * struct drm_gpusvm_zdd - GPU SVM zone device data
+ *
+ * @refcount: Reference count for the zdd
+ * @devmem_allocation: device memory allocation
+ * @device_private_page_owner: Device private pages owner
+ *
+ * This structure serves as a generic wrapper installed in
+ * page->zone_device_data. It provides infrastructure for looking up

a device
+ * memory allocation upon CPU page fault and asynchronously releasing device
+ * memory once the CPU has no page references. Asynchronous release is useful
+ * because CPU page references can be dropped in IRQ contexts, while releasing
+ * device memory likely requires sleeping locks.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+struct drm_gpusvm_zdd {
+	struct kref refcount;
+	struct drm_gpusvm_devmem *devmem_allocation;
+	void *device_private_page_owner;
+};
+
+/**
+ * drm_gpusvm_zdd_alloc() - Allocate a zdd structure.
+ * @device_private_page_owner: Device private pages owner
+ *
+ * This function allocates and initializes a new zdd structure. It

sets up the
+ * reference count and initializes the destroy work.
+ *
+ * Returns:
+ * Pointer to the allocated zdd on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +21 -0
+static struct drm_gpusvm_zdd *
+drm_gpusvm_zdd_alloc(void *device_private_page_owner)
+{
+	struct drm_gpusvm_zdd *zdd;
+
+	zdd = kmalloc(sizeof(*zdd), GFP_KERNEL);
+	if (!zdd)
+		return NULL;
+
+	kref_init(&zdd->refcount);
+	zdd->devmem_allocation = NULL;
+	zdd->device_private_page_owner = device_private_page_owner;
+
+	return zdd;
+}
+
+/**
+ * drm_gpusvm_zdd_get() - Get a reference to a zdd structure.
+ * @zdd: Pointer to the zdd structure.
+ *
+ * This function increments the reference count of the provided zdd

structure.
+ *
+ * Returns: Pointer to the zdd structure.
+ */
+static struct drm_gpusvm_zdd *drm_gpusvm_zdd_get(struct drm_gpusvm_zdd *zdd)

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+{
+	kref_get(&zdd->refcount);
+	return zdd;
+}
+
+/**
+ * drm_gpusvm_zdd_destroy() - Destroy a zdd structure.
+ * @ref: Pointer to the reference count structure.
+ *
+ * This function queues the destroy_work of the zdd for asynchronous

destruction.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +19 -0
+static void drm_gpusvm_zdd_destroy(struct kref *ref)
+{
+	struct drm_gpusvm_zdd *zdd =
+		container_of(ref, struct drm_gpusvm_zdd, refcount);
+	struct drm_gpusvm_devmem *devmem = zdd->devmem_allocation;
+
+	if (devmem) {
+		complete_all(&devmem->detached);
+		if (devmem->ops->devmem_release)
+			devmem->ops->devmem_release(devmem);
+	}
+	kfree(zdd);
+}
+
+/**
+ * drm_gpusvm_zdd_put() - Put a zdd reference.
+ * @zdd: Pointer to the zdd structure.
+ *
+ * This function decrements the reference count of the provided zdd

structure
+ * and schedules its destruction if the count drops to zero.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+static void drm_gpusvm_zdd_put(struct drm_gpusvm_zdd *zdd)
+{
+	kref_put(&zdd->refcount, drm_gpusvm_zdd_destroy);
+}
+
+/**
+ * drm_gpusvm_range_find() - Find GPU SVM range from GPU SVM

notifier
+ * @notifier: Pointer to the GPU SVM notifier structure.
+ * @start: Start address of the range
+ * @end: End address of the range
+ *
+ * Returns: A pointer to the drm_gpusvm_range if found or NULL
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, unsigned

long start,

drivers/gpu/drm/drm_gpusvm.c new file +8 -1
+		      unsigned long end)
+{
+	struct interval_tree_node *itree;
+
+	itree = interval_tree_iter_first(&notifier->root, start, end
- 1);
+
+	if (itree)
+		return container_of(itree, struct drm_gpusvm_range,

itree);

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+	else
+		return NULL;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_find);
+
+/**
+ * drm_gpusvm_for_each_range_safe() - Safely iterate over GPU SVM

ranges in a notifier
+ * @range__: Iterator variable for the ranges
+ * @next__: Iterator variable for the ranges temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier while
+ * removing ranges from it.
+ */
+#define drm_gpusvm_for_each_range_safe(range__, next__, notifier__,
start__, end__) \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	for ((range__) = drm_gpusvm_range_find((notifier__),

(start__), (end__)), \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (next__) =

__drm_gpusvm_range_next(range__); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (range__) && (range__->itree.start <

(end__)); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (range__) = (next__), (next__) =

__drm_gpusvm_range_next(range__))

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+
+/**
+ * __drm_gpusvm_notifier_next() - get the next drm_gpusvm_notifier

in the list
+ * @notifier: a pointer to the current drm_gpusvm_notifier
+ *
+ * Returns: A pointer to the next drm_gpusvm_notifier if available, or NULL if
+ *         the current notifier is the last one or if the input notifier is
+ *         NULL.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+static struct drm_gpusvm_notifier *
+__drm_gpusvm_notifier_next(struct drm_gpusvm_notifier *notifier)
+{
+	if (notifier && !list_is_last(&notifier->entry,
+				      &notifier->gpusvm-
Quoted 1 line

notifier_list))

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+		return list_next_entry(notifier, entry);
+
+	return NULL;
+}
+
+static struct drm_gpusvm_notifier *
+notifier_iter_first(struct rb_root_cached *root, unsigned long

start,

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+		    unsigned long last)
+{
+	struct interval_tree_node *itree;
+
+	itree = interval_tree_iter_first(root, start, last);
+
+	if (itree)
+		return container_of(itree, struct

drm_gpusvm_notifier, itree);

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+	else
+		return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_notifier() - Iterate over GPU SVM notifiers

in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm.
+ */
+#define drm_gpusvm_for_each_notifier(notifier__, gpusvm__, start__,
end__) \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,

(start__), (end__) - 1); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (notifier__) && (notifier__->itree.start <

(end__)); \

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+	     (notifier__) = __drm_gpusvm_notifier_next(notifier__))
+
+/**
+ * drm_gpusvm_for_each_notifier_safe() - Safely iterate over GPU SVM

notifiers in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @next__: Iterator variable for the notifiers temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm while
+ * removing notifiers from it.
+ */
+#define drm_gpusvm_for_each_notifier_safe(notifier__, next__,
gpusvm__, start__, end__) \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,

(start__), (end__) - 1), \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (next__) =

__drm_gpusvm_notifier_next(notifier__); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (notifier__) && (notifier__->itree.start <

(end__)); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (notifier__) = (next__), (next__) =

__drm_gpusvm_notifier_next(notifier__))

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+
+/**
+ * drm_gpusvm_notifier_invalidate() - Invalidate a GPU SVM notifier.
+ * @mni: Pointer to the mmu_interval_notifier structure.
+ * @mmu_range: Pointer to the mmu_notifier_range structure.
+ * @cur_seq: Current sequence number.
+ *
+ * This function serves as a generic MMU notifier for GPU SVM. It

sets the MMU
+ * notifier sequence number and calls the driver invalidate vfunc under
+ * gpusvm->notifier_lock.
+ *
+ * Returns:
+ * true if the operation succeeds, false otherwise.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+static bool
+drm_gpusvm_notifier_invalidate(struct mmu_interval_notifier *mni,
+			       const struct mmu_notifier_range

*mmu_range,

drivers/gpu/drm/drm_gpusvm.c new file +19 -0
+			       unsigned long cur_seq)
+{
+	struct drm_gpusvm_notifier *notifier =
+		container_of(mni, typeof(*notifier), notifier);
+	struct drm_gpusvm *gpusvm = notifier->gpusvm;
+
+	if (!mmu_notifier_range_blockable(mmu_range))
+		return false;
+
+	down_write(&gpusvm->notifier_lock);
+	mmu_interval_set_seq(mni, cur_seq);
+	gpusvm->ops->invalidate(gpusvm, notifier, mmu_range);
+	up_write(&gpusvm->notifier_lock);
+
+	return true;
+}
+
+/**
+ * drm_gpusvm_notifier_ops - MMU interval notifier operations for

GPU SVM
+ */
+static const struct mmu_interval_notifier_ops
drm_gpusvm_notifier_ops = {

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+	.invalidate = drm_gpusvm_notifier_invalidate,
+};
+
+/**
+ * drm_gpusvm_init() - Initialize the GPU SVM.
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @name: Name of the GPU SVM.
+ * @drm: Pointer to the DRM device structure.
+ * @mm: Pointer to the mm_struct for the address space.
+ * @device_private_page_owner: Device private pages owner.
+ * @mm_start: Start address of GPU SVM.
+ * @mm_range: Range of the GPU SVM.
+ * @notifier_size: Size of individual notifiers.
+ * @ops: Pointer to the operations structure for GPU SVM.
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range

allocation.
+ *               Entries should be powers of 2 in descending order with last
+ *               entry being SZ_4K.
+ * @num_chunks: Number of chunks.
+ *
+ * This function initializes the GPU SVM.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void

*device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+		    unsigned long mm_start, unsigned long mm_range,
+		    unsigned long notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const unsigned long *chunk_sizes, int

num_chunks)

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+{
+	if (!ops->invalidate || !num_chunks)
+		return -EINVAL;
+
+	gpusvm->name = name;
+	gpusvm->drm = drm;
+	gpusvm->mm = mm;
+	gpusvm->device_private_page_owner =

device_private_page_owner;

drivers/gpu/drm/drm_gpusvm.c new file +31 -0
+	gpusvm->mm_start = mm_start;
+	gpusvm->mm_range = mm_range;
+	gpusvm->notifier_size = notifier_size;
+	gpusvm->ops = ops;
+	gpusvm->chunk_sizes = chunk_sizes;
+	gpusvm->num_chunks = num_chunks;
+
+	mmgrab(mm);
+	gpusvm->root = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&gpusvm->notifier_list);
+
+	init_rwsem(&gpusvm->notifier_lock);
+
+	fs_reclaim_acquire(GFP_KERNEL);
+	might_lock(&gpusvm->notifier_lock);
+	fs_reclaim_release(GFP_KERNEL);
+
+#ifdef CONFIG_LOCKDEP
+	gpusvm->lock_dep_map = NULL;
+#endif
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_init);
+
+/**
+ * drm_gpusvm_notifier_find() - Find GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ *
+ * This function finds the GPU SVM notifier associated with the

fault address.
+ *
+ * Returns:
+ * Pointer to the GPU SVM notifier on success, NULL otherwise.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+static struct drm_gpusvm_notifier *
+drm_gpusvm_notifier_find(struct drm_gpusvm *gpusvm,
+			 unsigned long fault_addr)
+{
+	return notifier_iter_first(&gpusvm->root, fault_addr,

fault_addr + 1);

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+}
+
+/**
+ * to_drm_gpusvm_notifier() - retrieve the container struct for a

given rbtree node
+ * @node: a pointer to the rbtree node embedded within a drm_gpusvm_notifier struct
+ *
+ * Returns: A pointer to the containing drm_gpusvm_notifier structure.
+ */
+static struct drm_gpusvm_notifier *to_drm_gpusvm_notifier(struct rb_node *node)

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+{
+	return container_of(node, struct drm_gpusvm_notifier,

itree.rb);

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+}
+
+/**
+ * drm_gpusvm_notifier_insert() - Insert GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function inserts the GPU SVM notifier into the GPU SVM RB

tree and list.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static void drm_gpusvm_notifier_insert(struct drm_gpusvm *gpusvm,
+				       struct drm_gpusvm_notifier

*notifier)

drivers/gpu/drm/drm_gpusvm.c new file +21 -0
+{
+	struct rb_node *node;
+	struct list_head *head;
+
+	interval_tree_insert(&notifier->itree, &gpusvm->root);
+
+	node = rb_prev(&notifier->itree.rb);
+	if (node)
+		head = &(to_drm_gpusvm_notifier(node))->entry;
+	else
+		head = &gpusvm->notifier_list;
+
+	list_add(&notifier->entry, head);
+}
+
+/**
+ * drm_gpusvm_notifier_remove() - Remove GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM tructure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function removes the GPU SVM notifier from the GPU SVM RB

tree and list.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static void drm_gpusvm_notifier_remove(struct drm_gpusvm *gpusvm,
+				       struct drm_gpusvm_notifier

*notifier)

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+{
+	interval_tree_remove(&notifier->itree, &gpusvm->root);
+	list_del(&notifier->entry);
+}
+
+/**
+ * drm_gpusvm_fini() - Finalize the GPU SVM.
+ * @gpusvm: Pointer to the GPU SVM structure.
+ *
+ * This function finalizes the GPU SVM by cleaning up any remaining

ranges and
+ * notifiers, and dropping a reference to struct MM.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm)
+{
+	struct drm_gpusvm_notifier *notifier, *next;
+
+	drm_gpusvm_for_each_notifier_safe(notifier, next, gpusvm, 0,

LONG_MAX) {

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+		struct drm_gpusvm_range *range, *__next;
+
+		/*
+		 * Remove notifier first to avoid racing with any

invalidation

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+		 */
+		mmu_interval_notifier_remove(&notifier->notifier);
+		notifier->flags.removed = true;
+
+		drm_gpusvm_for_each_range_safe(range, __next,

notifier, 0,

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+					       LONG_MAX)
+			drm_gpusvm_range_remove(gpusvm, range);
+	}
+
+	mmdrop(gpusvm->mm);
+	WARN_ON(!RB_EMPTY_ROOT(&gpusvm->root.rb_root));
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_fini);
+
+/**
+ * drm_gpusvm_notifier_alloc() - Allocate GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ *
+ * This function allocates and initializes the GPU SVM notifier

structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM notifier on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static struct drm_gpusvm_notifier *
+drm_gpusvm_notifier_alloc(struct drm_gpusvm *gpusvm, unsigned long

fault_addr)

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	if (gpusvm->ops->notifier_alloc)
+		notifier = gpusvm->ops->notifier_alloc();
+	else
+		notifier = kzalloc(sizeof(*notifier), GFP_KERNEL);
+
+	if (!notifier)
+		return ERR_PTR(-ENOMEM);
+
+	notifier->gpusvm = gpusvm;
+	notifier->itree.start = ALIGN_DOWN(fault_addr, gpusvm-
Quoted 1 line

notifier_size);

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	notifier->itree.last = ALIGN(fault_addr + 1, gpusvm-
Quoted 1 line

notifier_size) - 1;

drivers/gpu/drm/drm_gpusvm.c new file +16 -0
+	INIT_LIST_HEAD(&notifier->entry);
+	notifier->root = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&notifier->range_list);
+
+	return notifier;
+}
+
+/**
+ * drm_gpusvm_notifier_free() - Free GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function frees the GPU SVM notifier structure.
+ */
+static void drm_gpusvm_notifier_free(struct drm_gpusvm *gpusvm,
+				     struct drm_gpusvm_notifier

*notifier)

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+{
+	WARN_ON(!RB_EMPTY_ROOT(&notifier->root.rb_root));
+
+	if (gpusvm->ops->notifier_free)
+		gpusvm->ops->notifier_free(notifier);
+	else
+		kfree(notifier);
+}
+
+/**
+ * to_drm_gpusvm_range() - retrieve the container struct for a given

rbtree node
+ * @node: a pointer to the rbtree node embedded within a drm_gpusvm_range struct
+ *
+ * Returns: A pointer to the containing drm_gpusvm_range structure.
+ */
+static struct drm_gpusvm_range *to_drm_gpusvm_range(struct rb_node *node)

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+{
+	return container_of(node, struct drm_gpusvm_range,

itree.rb);

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+}
+
+/**
+ * drm_gpusvm_range_insert() - Insert GPU SVM range
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function inserts the GPU SVM range into the notifier RB tree

and list.
+ */
+static void drm_gpusvm_range_insert(struct drm_gpusvm_notifier *notifier,

drivers/gpu/drm/drm_gpusvm.c new file +24 -0
+				    struct drm_gpusvm_range *range)
+{
+	struct rb_node *node;
+	struct list_head *head;
+
+	drm_gpusvm_notifier_lock(notifier->gpusvm);
+	interval_tree_insert(&range->itree, &notifier->root);
+
+	node = rb_prev(&range->itree.rb);
+	if (node)
+		head = &(to_drm_gpusvm_range(node))->entry;
+	else
+		head = &notifier->range_list;
+
+	list_add(&range->entry, head);
+	drm_gpusvm_notifier_unlock(notifier->gpusvm);
+}
+
+/**
+ * __drm_gpusvm_range_remove() - Remove GPU SVM range
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This macro removes the GPU SVM range from the notifier RB tree

and list.
+ */
+static void __drm_gpusvm_range_remove(struct drm_gpusvm_notifier *notifier,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				      struct drm_gpusvm_range

*range)

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+{
+	interval_tree_remove(&range->itree, &notifier->root);
+	list_del(&range->entry);
+}
+
+/**
+ * drm_gpusvm_range_alloc() - Allocate GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @fault_addr: Fault address
+ * @chunk_size: Chunk size
+ * @migrate_devmem: Flag indicating whether to migrate device memory
+ *
+ * This function allocates and initializes the GPU SVM range

structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM range on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+static struct drm_gpusvm_range *
+drm_gpusvm_range_alloc(struct drm_gpusvm *gpusvm,
+		       struct drm_gpusvm_notifier *notifier,
+		       unsigned long fault_addr, unsigned long

chunk_size,

drivers/gpu/drm/drm_gpusvm.c new file +32 -0
+		       bool migrate_devmem)
+{
+	struct drm_gpusvm_range *range;
+
+	if (gpusvm->ops->range_alloc)
+		range = gpusvm->ops->range_alloc(gpusvm);
+	else
+		range = kzalloc(sizeof(*range), GFP_KERNEL);
+
+	if (!range)
+		return ERR_PTR(-ENOMEM);
+
+	kref_init(&range->refcount);
+	range->gpusvm = gpusvm;
+	range->notifier = notifier;
+	range->itree.start = ALIGN_DOWN(fault_addr, chunk_size);
+	range->itree.last = ALIGN(fault_addr + 1, chunk_size) - 1;
+	INIT_LIST_HEAD(&range->entry);
+	range->notifier_seq = LONG_MAX;
+	range->flags.migrate_devmem = migrate_devmem ? 1 : 0;
+
+	return range;
+}
+
+/**
+ * drm_gpusvm_check_pages() - Check pages
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @start: Start address
+ * @end: End address
+ *
+ * Check if pages between start and end have been faulted in on the

CPU. Use to
+ * prevent migration of pages without CPU backing store.
+ *
+ * Returns:
+ * True if pages have been faulted into CPU, False otherwise
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static bool drm_gpusvm_check_pages(struct drm_gpusvm *gpusvm,
+				   struct drm_gpusvm_notifier

*notifier,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				   unsigned long start, unsigned

long end)

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+{
+	struct hmm_range hmm_range = {
+		.default_flags = 0,
+		.notifier = &notifier->notifier,
+		.start = start,
+		.end = end,
+		.dev_private_owner = gpusvm-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+	};
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+	unsigned long *pfns;
+	unsigned long npages = npages_in_range(start, end);
+	int err, i;
+
+	mmap_assert_locked(gpusvm->mm);
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL);
+	if (!pfns)
+		return false;
+
+	hmm_range.notifier_seq = mmu_interval_read_begin(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+	hmm_range.hmm_pfns = pfns;
+
+	while (true) {
+		err = hmm_range_fault(&hmm_range);
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq =
+				mmu_interval_read_begin(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/drm_gpusvm.c new file +22 -0
+			continue;
+		}
+		break;
+	}
+	if (err)
+		goto err_free;
+
+	for (i = 0; i < npages;) {
+		if (!(pfns[i] & HMM_PFN_VALID)) {
+			err = -EFAULT;
+			goto err_free;
+		}
+		i += 0x1 << hmm_pfn_to_map_order(pfns[i]);
+	}
+
+err_free:
+	kvfree(pfns);
+	return err ? false : true;
+}
+
+/**
+ * drm_gpusvm_range_chunk_size() - Determine chunk size for GPU SVM

range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @vas: Pointer to the virtual memory area structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @check_pages_threshold: Check CPU pages for present threshold
+ *
+ * This function determines the chunk size for the GPU SVM range based on the
+ * fault address, GPU SVM chunk sizes, existing GPU SVM ranges, and the virtual
+ * memory area boundaries.
+ *
+ * Returns:
+ * Chunk size on success, LONG_MAX on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+static unsigned long
+drm_gpusvm_range_chunk_size(struct drm_gpusvm *gpusvm,
+			    struct drm_gpusvm_notifier *notifier,
+			    struct vm_area_struct *vas,
+			    unsigned long fault_addr,
+			    unsigned long gpuva_start,
+			    unsigned long gpuva_end,
+			    unsigned long check_pages_threshold)
+{
+	unsigned long start, end;
+	int i = 0;
+
+retry:
+	for (; i < gpusvm->num_chunks; ++i) {
+		start = ALIGN_DOWN(fault_addr, gpusvm-
Quoted 1 line

chunk_sizes[i]);

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+		end = ALIGN(fault_addr + 1, gpusvm->chunk_sizes[i]);
+
+		if (start >= vas->vm_start && end <= vas->vm_end &&
+		    start >= notifier->itree.start &&
+		    end <= notifier->itree.last + 1 &&
+		    start >= gpuva_start && end <= gpuva_end)
+			break;
+	}
+
+	if (i == gpusvm->num_chunks)
+		return LONG_MAX;
+
+	/*
+	 * If allocation more than page, ensure not to overlap with

existing

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+	 * ranges.
+	 */
+	if (end - start != SZ_4K) {
+		struct drm_gpusvm_range *range;
+
+		range = drm_gpusvm_range_find(notifier, start, end);
+		if (range) {
+			++i;
+			goto retry;
+		}
+
+		/*
+		 * XXX: Only create range on pages CPU has faulted

in. Without

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * this check, or prefault, on BMG

'xe_exec_system_allocator --r

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * process-many-malloc' fails. In the failure case,

each process

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * mallocs 16k but the CPU VMA is ~128k which

results in 64k SVM

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * ranges. When migrating the SVM ranges, some

processes fail in

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * drm_gpusvm_migrate_to_devmem with 'migrate.cpages

!= npages'

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * and then upon drm_gpusvm_range_get_pages device

pages from

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * other processes are collected + faulted in which

creates all

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * sorts of problems. Unsure exactly how this

happening, also

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * problem goes away if 'xe_exec_system_allocator --

r

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * process-many-malloc' mallocs at least 64k at a

time.

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+		 */
+		if (end - start <= check_pages_threshold &&
+		    !drm_gpusvm_check_pages(gpusvm, notifier, start,

end)) {

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+			++i;
+			goto retry;
+		}
+	}
+
+	return end - start;
+}
+
+/**
+ * drm_gpusvm_range_find_or_insert() - Find or insert GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @ctx: GPU SVM context
+ *
+ * This function finds or inserts a newly allocated a GPU SVM range

based on the
+ * fault address. Caller must hold a lock to protect range lookup and insertion.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +28 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm,
+				unsigned long fault_addr,
+				unsigned long gpuva_start,
+				unsigned long gpuva_end,
+				const struct drm_gpusvm_ctx *ctx)
+{
+	struct drm_gpusvm_notifier *notifier;
+	struct drm_gpusvm_range *range;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	bool notifier_alloc = false;
+	unsigned long chunk_size;
+	int err;
+	bool migrate_devmem;
+
+	drm_gpusvm_driver_lock_held(gpusvm);
+
+	if (fault_addr < gpusvm->mm_start ||
+	    fault_addr > gpusvm->mm_start + gpusvm->mm_range)
+		return ERR_PTR(-EINVAL);
+
+	if (!mmget_not_zero(mm))
+		return ERR_PTR(-EFAULT);
+
+	notifier = drm_gpusvm_notifier_find(gpusvm, fault_addr);
+	if (!notifier) {
+		notifier = drm_gpusvm_notifier_alloc(gpusvm,

fault_addr);

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+		if (IS_ERR(notifier)) {
+			err = PTR_ERR(notifier);
+			goto err_mmunlock;
+		}
+		notifier_alloc = true;
+		err = mmu_interval_notifier_insert(&notifier-
Quoted 1 line

notifier,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						   mm, notifier-
Quoted 1 line

itree.start,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						   notifier-
Quoted 1 line

itree.last -

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						   notifier-
Quoted 1 line

itree.start + 1,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						  

&drm_gpusvm_notifier_ops);

drivers/gpu/drm/drm_gpusvm.c new file +18 -0
+		if (err)
+			goto err_notifier;
+	}
+
+	mmap_read_lock(mm);
+
+	vas = vma_lookup(mm, fault_addr);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_notifier_remove;
+	}
+
+	if (!ctx->read_only && !(vas->vm_flags & VM_WRITE)) {
+		err = -EPERM;
+		goto err_notifier_remove;
+	}
+
+	range = drm_gpusvm_range_find(notifier, fault_addr,

fault_addr + 1);

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+	if (range)
+		goto out_mmunlock;
+	/*
+	 * XXX: Short-circuiting migration based on migrate_vma_*

current

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	 * limitations. If/when migrate_vma_* add more support, this

logic will

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+	 * have to change.
+	 */
+	migrate_devmem = ctx->devmem_possible &&
+		vma_is_anonymous(vas) && !is_vm_hugetlb_page(vas);
+
+	chunk_size = drm_gpusvm_range_chunk_size(gpusvm, notifier,

vas,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						 fault_addr,

gpuva_start,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+						 gpuva_end,
+						 ctx-
Quoted 1 line

check_pages_threshold);

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+	if (chunk_size == LONG_MAX) {
+		err = -EINVAL;
+		goto err_notifier_remove;
+	}
+
+	range = drm_gpusvm_range_alloc(gpusvm, notifier, fault_addr,

chunk_size,

drivers/gpu/drm/drm_gpusvm.c new file +31 -0
+				       migrate_devmem);
+	if (IS_ERR(range)) {
+		err = PTR_ERR(range);
+		goto err_notifier_remove;
+	}
+
+	drm_gpusvm_range_insert(notifier, range);
+	if (notifier_alloc)
+		drm_gpusvm_notifier_insert(gpusvm, notifier);
+
+out_mmunlock:
+	mmap_read_unlock(mm);
+	mmput(mm);
+
+	return range;
+
+err_notifier_remove:
+	mmap_read_unlock(mm);
+	if (notifier_alloc)
+		mmu_interval_notifier_remove(&notifier->notifier);
+err_notifier:
+	if (notifier_alloc)
+		drm_gpusvm_notifier_free(gpusvm, notifier);
+err_mmunlock:
+	mmput(mm);
+	return ERR_PTR(err);
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_find_or_insert);
+
+/**
+ * __drm_gpusvm_range_unmap_pages() - Unmap pages associated with a

GPU SVM range (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @npages: Number of pages to unmap
+ *
+ * This function unmap pages associated with a GPU SVM range. Assumes and
+ * asserts correct locking is in place when called.
+ */
+static void __drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					   struct drm_gpusvm_range

*range,

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+					   unsigned long npages)
+{
+	unsigned long i, j;
+	struct drm_pagemap *dpagemap = range->dpagemap;
+	struct device *dev = gpusvm->drm->dev;
+
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	if (range->flags.has_dma_mapping) {
+		for (i = 0, j = 0; i < npages; j++) {
+			struct drm_pagemap_dma_addr *addr = &range-
Quoted 1 line

dma_addr[j];

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+			if (addr->proto == DRM_INTERCONNECT_SYSTEM)
+				dma_unmap_page(dev,
+					       addr->addr,
+					       PAGE_SIZE << addr-
Quoted 1 line

order,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					       addr->dir);
+			else if (dpagemap && dpagemap->ops-
Quoted 1 line

unmap_dma)

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+				dpagemap->ops->unmap_dma(dpagemap,
+							 dev,
+							 *addr);
+			i += 1 << addr->order;
+		}
+		range->flags.has_devmem_pages = false;
+		range->flags.has_dma_mapping = false;
+		range->dpagemap = NULL;
+	}
+}
+
+/**
+ * drm_gpusvm_range_free_pages() - Free pages associated with a GPU

SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function frees the dma address array associated with a GPU SVM range.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static void drm_gpusvm_range_free_pages(struct drm_gpusvm *gpusvm,
+					struct drm_gpusvm_range

*range)

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+{
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	if (range->dma_addr) {
+		kvfree(range->dma_addr);
+		range->dma_addr = NULL;
+	}
+}
+
+/**
+ * drm_gpusvm_range_remove() - Remove GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range to be removed
+ *
+ * This function removes the specified GPU SVM range and also

removes the parent
+ * GPU SVM notifier if no more ranges remain in the notifier. The caller must
+ * hold a lock to protect range and notifier removal.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range)
+{
+	unsigned long npages = npages_in_range(range->itree.start,
+					       range->itree.last +

1);

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+	struct drm_gpusvm_notifier *notifier;
+
+	drm_gpusvm_driver_lock_held(gpusvm);
+
+	notifier = drm_gpusvm_notifier_find(gpusvm, range-
Quoted 1 line

itree.start);

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+	if (WARN_ON_ONCE(!notifier))
+		return;
+
+	drm_gpusvm_notifier_lock(gpusvm);
+	__drm_gpusvm_range_unmap_pages(gpusvm, range, npages);
+	drm_gpusvm_range_free_pages(gpusvm, range);
+	__drm_gpusvm_range_remove(notifier, range);
+	drm_gpusvm_notifier_unlock(gpusvm);
+
+	drm_gpusvm_range_put(range);
+
+	if (RB_EMPTY_ROOT(&notifier->root.rb_root)) {
+		if (!notifier->flags.removed)
+			mmu_interval_notifier_remove(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+		drm_gpusvm_notifier_remove(gpusvm, notifier);
+		drm_gpusvm_notifier_free(gpusvm, notifier);
+	}
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_remove);
+
+/**
+ * drm_gpusvm_range_get() - Get a reference to GPU SVM range
+ * @range: Pointer to the GPU SVM range
+ *
+ * This function increments the reference count of the specified GPU

SVM range.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range)
+{
+	kref_get(&range->refcount);
+
+	return range;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_get);
+
+/**
+ * drm_gpusvm_range_destroy() - Destroy GPU SVM range
+ * @refcount: Pointer to the reference counter embedded in the GPU

SVM range
+ *
+ * This function destroys the specified GPU SVM range when its reference count
+ * reaches zero. If a custom range-free function is provided, it is invoked to
+ * free the range; otherwise, the range is deallocated using kfree().
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_range_destroy(struct kref *refcount)
+{
+	struct drm_gpusvm_range *range =
+		container_of(refcount, struct drm_gpusvm_range,

refcount);

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+	struct drm_gpusvm *gpusvm = range->gpusvm;
+
+	if (gpusvm->ops->range_free)
+		gpusvm->ops->range_free(range);
+	else
+		kfree(range);
+}
+
+/**
+ * drm_gpusvm_range_put() - Put a reference to GPU SVM range
+ * @range: Pointer to the GPU SVM range
+ *
+ * This function decrements the reference count of the specified GPU

SVM range
+ * and frees it when the count reaches zero.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range)
+{
+	kref_put(&range->refcount, drm_gpusvm_range_destroy);
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_put);
+
+/**
+ * drm_gpusvm_range_pages_valid() - GPU SVM range pages valid
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid.

Expected be
+ * called holding gpusvm->notifier_lock and as the last step before committing a
+ * GPU binding. This is akin to a notifier seqno check in the HMM documentation
+ * but due to wider notifiers (i.e., notifiers which span multiple ranges) this
+ * function is required for finer grained checking (i.e., per range) if pages
+ * are valid.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range)
+{
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	return range->flags.has_devmem_pages || range-
Quoted 1 line

flags.has_dma_mapping;

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_pages_valid);
+
+/**
+ * drm_gpusvm_range_pages_valid_unlocked() - GPU SVM range pages

valid unlocked
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid. Expected be
+ * called without holding gpusvm->notifier_lock.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+static bool
+drm_gpusvm_range_pages_valid_unlocked(struct drm_gpusvm *gpusvm,
+				      struct drm_gpusvm_range

*range)

drivers/gpu/drm/drm_gpusvm.c new file +22 -0
+{
+	bool pages_valid;
+
+	if (!range->dma_addr)
+		return false;
+
+	drm_gpusvm_notifier_lock(gpusvm);
+	pages_valid = drm_gpusvm_range_pages_valid(gpusvm, range);
+	if (!pages_valid)
+		drm_gpusvm_range_free_pages(gpusvm, range);
+	drm_gpusvm_notifier_unlock(gpusvm);
+
+	return pages_valid;
+}
+
+/**
+ * drm_gpusvm_range_get_pages() - Get pages for a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function gets pages for a GPU SVM range and ensures they are

mapped for
+ * DMA access.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	struct mmu_interval_notifier *notifier = &range->notifier-
Quoted 1 line

notifier;

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+	struct hmm_range hmm_range = {
+		.default_flags = HMM_PFN_REQ_FAULT | (ctx->read_only

? 0 :

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+			HMM_PFN_REQ_WRITE),
+		.notifier = notifier,
+		.start = range->itree.start,
+		.end = range->itree.last + 1,
+		.dev_private_owner = gpusvm-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	struct drm_gpusvm_zdd *zdd;
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+	unsigned long i, j;
+	unsigned long npages = npages_in_range(range->itree.start,
+					       range->itree.last +

1);

drivers/gpu/drm/drm_gpusvm.c new file +58 -0
+	unsigned long num_dma_mapped;
+	unsigned int order = 0;
+	unsigned long *pfns;
+	struct page **pages;
+	int err = 0;
+	struct dev_pagemap *pagemap;
+	struct drm_pagemap *dpagemap;
+
+retry:
+	hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
+	if (drm_gpusvm_range_pages_valid_unlocked(gpusvm, range))
+		goto set_seqno;
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL);
+	if (!pfns)
+		return -ENOMEM;
+
+	if (!mmget_not_zero(mm)) {
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	hmm_range.hmm_pfns = pfns;
+	while (true) {
+		mmap_read_lock(mm);
+		err = hmm_range_fault(&hmm_range);
+		mmap_read_unlock(mm);
+
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq =
+				mmu_interval_read_begin(notifier);
+			continue;
+		}
+		break;
+	}
+	mmput(mm);
+	if (err)
+		goto err_free;
+
+	pages = (struct page **)pfns;
+map_pages:
+	/*
+	 * Perform all dma mappings under the notifier lock to not
+	 * access freed pages. A notifier will either block on
+	 * the notifier lock or unmap dma.
+	 */
+	drm_gpusvm_notifier_lock(gpusvm);
+
+	if (range->flags.unmapped) {
+		drm_gpusvm_notifier_unlock(gpusvm);
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	if (mmu_interval_read_retry(notifier,

hmm_range.notifier_seq)) {

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+		drm_gpusvm_notifier_unlock(gpusvm);
+		kvfree(pfns);
+		goto retry;
+	}
+
+	if (!range->dma_addr) {
+		/* Unlock and restart mapping to allocate memory. */
+		drm_gpusvm_notifier_unlock(gpusvm);
+		range->dma_addr = kvmalloc_array(npages,
+						 sizeof(*range-
Quoted 1 line

dma_addr),

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+						 GFP_KERNEL);
+		if (!range->dma_addr) {
+			err = -ENOMEM;
+			goto err_free;
+		}
+		goto map_pages;
+	}
+
+	zdd = NULL;
+	num_dma_mapped = 0;
+	for (i = 0, j = 0; i < npages; ++j) {
+		struct page *page = hmm_pfn_to_page(pfns[i]);
+
+		order = hmm_pfn_to_map_order(pfns[i]);
+		if (is_device_private_page(page) ||
+		    is_device_coherent_page(page)) {
+			if (zdd != page->zone_device_data && i > 0)

{

drivers/gpu/drm/drm_gpusvm.c line 1395

Quoted 3 lines
drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+				err = -EOPNOTSUPP;
+				goto err_unmap;
+			}

I wonder, in a step towards make the get_pages() more permissive, Could one drop the above check that all device pages are using the same allocation and just verify that they are from the same pagemap, like below?

Quoted 597 lines
drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+			zdd = page->zone_device_data;
+			if (pagemap != page->pgmap) {
+				if (i > 0) {
+					err = -EOPNOTSUPP;
+					goto err_unmap;
+				}
+
+				pagemap = page->pgmap;
+				dpagemap = zdd->devmem_allocation-
Quoted 1 line

dpagemap;

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				if (drm_WARN_ON(gpusvm->drm,

!dpagemap)) {

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					/*
+					 * Raced. This is not

supposed to happen

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					 * since hmm_range_fault()

should've migrated

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+					 * this page to system.
+					 */
+					err = -EAGAIN;
+					goto err_unmap;
+				}
+			}
+			range->dma_addr[j] =
+				dpagemap->ops->map_dma(dpagemap,
+						       gpusvm->drm-
Quoted 1 line

dev,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+						       page, order,
+						      

DMA_BIDIRECTIONAL);

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+			if (dma_mapping_error(gpusvm->drm->dev,
+					      range-
Quoted 1 line

dma_addr[j].addr)) {

drivers/gpu/drm/drm_gpusvm.c new file +18 -0
+				err = -EFAULT;
+				goto err_unmap;
+			}
+
+			pages[i] = page;
+		} else {
+			dma_addr_t addr;
+
+			if (is_zone_device_page(page) || zdd) {
+				err = -EOPNOTSUPP;
+				goto err_unmap;
+			}
+
+			addr = dma_map_page(gpusvm->drm->dev,
+					    page, 0,
+					    PAGE_SIZE << order,
+					    DMA_BIDIRECTIONAL);
+			if (dma_mapping_error(gpusvm->drm->dev,

addr)) {

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+				err = -EFAULT;
+				goto err_unmap;
+			}
+
+			range->dma_addr[j] =

drm_pagemap_dma_addr_encode

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				(addr, DRM_INTERCONNECT_SYSTEM,

order,

drivers/gpu/drm/drm_gpusvm.c new file +21 -0
+				 DMA_BIDIRECTIONAL);
+		}
+		i += 1 << order;
+		num_dma_mapped = i;
+	}
+
+	range->flags.has_dma_mapping = true;
+	if (zdd) {
+		range->flags.has_devmem_pages = true;
+		range->dpagemap = dpagemap;
+	}
+
+	drm_gpusvm_notifier_unlock(gpusvm);
+	kvfree(pfns);
+set_seqno:
+	range->notifier_seq = hmm_range.notifier_seq;
+
+	return 0;
+
+err_unmap:
+	__drm_gpusvm_range_unmap_pages(gpusvm, range,

num_dma_mapped);

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+	drm_gpusvm_notifier_unlock(gpusvm);
+err_free:
+	kvfree(pfns);
+	if (err == -EAGAIN)
+		goto retry;
+	return err;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_get_pages);
+
+/**
+ * drm_gpusvm_range_unmap_pages() - Unmap pages associated with a

GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed that gpusvm->notifier_lock is held in write mode; if it
+ * is clear, it acquires gpusvm->notifier_lock in read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops-

Quoted 1 line

invalidate for IOMMU

+ * security model.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx)
+{
+	unsigned long npages = npages_in_range(range->itree.start,
+					       range->itree.last +

1);

drivers/gpu/drm/drm_gpusvm.c new file +33 -0
+
+	if (ctx->in_notifier)
+		lockdep_assert_held_write(&gpusvm->notifier_lock);
+	else
+		drm_gpusvm_notifier_lock(gpusvm);
+
+	__drm_gpusvm_range_unmap_pages(gpusvm, range, npages);
+
+	if (!ctx->in_notifier)
+		drm_gpusvm_notifier_unlock(gpusvm);
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_unmap_pages);
+
+/**
+ * drm_gpusvm_migration_unlock_put_page() - Put a migration page
+ * @page: Pointer to the page to put
+ *
+ * This function unlocks and puts a page.
+ */
+static void drm_gpusvm_migration_unlock_put_page(struct page *page)
+{
+	unlock_page(page);
+	put_page(page);
+}
+
+/**
+ * drm_gpusvm_migration_unlock_put_pages() - Put migration pages
+ * @npages: Number of pages
+ * @migrate_pfn: Array of migrate page frame numbers
+ *
+ * This function unlocks and puts an array of pages.
+ */
+static void drm_gpusvm_migration_unlock_put_pages(unsigned long

npages,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						  unsigned long

*migrate_pfn)

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page;
+
+		if (!migrate_pfn[i])
+			continue;
+
+		page = migrate_pfn_to_page(migrate_pfn[i]);
+		drm_gpusvm_migration_unlock_put_page(page);
+		migrate_pfn[i] = 0;
+	}
+}
+
+/**
+ * drm_gpusvm_get_devmem_page() - Get a reference to a device memory

page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU SVM zone
+ * device data and initializes it for zone device usage.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+static void drm_gpusvm_get_devmem_page(struct page *page,
+				     struct drm_gpusvm_zdd *zdd)
+{
+	page->zone_device_data = drm_gpusvm_zdd_get(zdd);
+	zone_device_page_init(page);
+}
+
+/**
+ * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM

migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Returns: 0 on success, -EFAULT if an error occurs during mapping.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+static int drm_gpusvm_migrate_map_pages(struct device *dev,
+					dma_addr_t *dma_addr,
+					unsigned long *migrate_pfn,
+					unsigned long npages,
+					enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page =

migrate_pfn_to_page(migrate_pfn[i]);

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+
+		if (!page)
+			continue;
+
+		if (WARN_ON_ONCE(is_zone_device_page(page)))
+			return -EFAULT;
+
+		dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE,

dir);

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+		if (dma_mapping_error(dev, dma_addr[i]))
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped

for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_migrate_unmap_pages(struct device *dev,
+					   dma_addr_t *dma_addr,
+					   unsigned long npages,
+					   enum dma_data_direction

dir)

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!dma_addr[i] || dma_mapping_error(dev,

dma_addr[i]))

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+			continue;
+
+		dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir);
+	}
+}
+
+/**
+ * drm_gpusvm_migrate_to_devmem() - Migrate GPU SVM range to device

memory
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @devmem_allocation: Pointer to the device memory allocation. The caller
+ *                     should hold a reference to the device memory allocation,
+ *                     which should be dropped via ops-

Quoted 1 line

devmem_release or upon

+ *                     the failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to device memory. It performs the
+ * necessary setup and invokes the driver-specific operations for migration to
+ * device memory. Upon successful return, @devmem_allocation can safely reference @range
+ * until ops->devmem_release is called which only upon successful return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+int drm_gpusvm_migrate_to_devmem(struct drm_gpusvm *gpusvm,
+				 struct drm_gpusvm_range *range,
+				 struct drm_gpusvm_devmem

*devmem_allocation,

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+				 const struct drm_gpusvm_ctx *ctx)
+{
+	const struct drm_gpusvm_devmem_ops *ops = devmem_allocation-
Quoted 1 line

ops;

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	unsigned long start = range->itree.start, end = range-
Quoted 1 line

itree.last + 1;

drivers/gpu/drm/drm_gpusvm.c new file +45 -0
+	struct migrate_vma migrate = {
+		.start		= start,
+		.end		= end,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_SYSTEM,
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	unsigned long i, npages = npages_in_range(start, end);
+	struct vm_area_struct *vas;
+	struct drm_gpusvm_zdd *zdd = NULL;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int err;
+
+	if (!range->flags.migrate_devmem)
+		return -EINVAL;
+
+	if (!ops->populate_devmem_pfn || !ops->copy_to_devmem ||
+	    !ops->copy_to_ram)
+		return -EOPNOTSUPP;
+
+	if (!mmget_not_zero(mm)) {
+		err = -EFAULT;
+		goto err_out;
+	}
+	mmap_read_lock(mm);
+
+	vas = vma_lookup(mm, start);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end > vas->vm_end || start < vas->vm_start) {
+		err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	if (!vma_is_anonymous(vas)) {
+		err = -EBUSY;
+		goto err_mmunlock;
+	}
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +

sizeof(*dma_addr) +

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_mmunlock;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr))

* npages;

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+
+	zdd = drm_gpusvm_zdd_alloc(gpusvm-
Quoted 1 line

device_private_page_owner);

drivers/gpu/drm/drm_gpusvm.c new file +24 -0
+	if (!zdd) {
+		err = -ENOMEM;
+		goto err_free;
+	}
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	if (!migrate.cpages) {
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	if (migrate.cpages != npages) {
+		err = -EBUSY;
+		goto err_finalize;
+	}
+
+	err = ops->populate_devmem_pfn(devmem_allocation, npages,

migrate.dst);

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(devmem_allocation->dev,

dma_addr,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					   migrate.src, npages,

DMA_TO_DEVICE);

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = pfn_to_page(migrate.dst[i]);
+
+		pages[i] = page;
+		migrate.dst[i] = migrate_pfn(migrate.dst[i]);
+		drm_gpusvm_get_devmem_page(page, zdd);
+	}
+
+	err = ops->copy_to_devmem(pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+	/* Upon success bind devmem allocation to range and zdd */
+	zdd->devmem_allocation = devmem_allocation;	/* Owns ref

*/

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_unlock_put_pages(npages,

migrate.dst);

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(devmem_allocation->dev,

dma_addr, npages,

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+				       DMA_TO_DEVICE);
+err_free:
+	if (zdd)
+		drm_gpusvm_zdd_put(zdd);
+	kvfree(buf);
+err_mmunlock:
+	mmap_read_unlock(mm);
+	mmput(mm);
+err_out:
+	return err;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_migrate_to_devmem);
+
+/**
+ * drm_gpusvm_migrate_populate_ram_pfn() - Populate RAM PFNs for a

VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @fault_page: Fault page
+ * @npages: Number of pages to populate
+ * @mpages: Number of pages to migrate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the RAM migrate page frame numbers (PFNs) for the
+ * specified VM area structure. It allocates and locks pages in the VM area for
+ * RAM usage. If vas is non-NULL use alloc_page_vma for allocation, if NULL use
+ * alloc_page for allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_ram_pfn(struct vm_area_struct *vas,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					       struct page

*fault_page,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					       unsigned long npages,
+					       unsigned long

*mpages,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					       unsigned long

*src_mpfn,

drivers/gpu/drm/drm_gpusvm.c new file +23 -0
+					       unsigned long *mpfn,
+					       unsigned long addr)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i, addr += PAGE_SIZE) {
+		struct page *page, *src_page;
+
+		if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE))
+			continue;
+
+		src_page = migrate_pfn_to_page(src_mpfn[i]);
+		if (!src_page)
+			continue;
+
+		if (fault_page) {
+			if (src_page->zone_device_data !=
+			    fault_page->zone_device_data)
+				continue;
+		}
+
+		if (vas)
+			page = alloc_page_vma(GFP_HIGHUSER, vas,

addr);

drivers/gpu/drm/drm_gpusvm.c new file +39 -0
+		else
+			page = alloc_page(GFP_HIGHUSER);
+
+		if (!page)
+			goto free_pages;
+
+		mpfn[i] = migrate_pfn(page_to_pfn(page));
+	}
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = migrate_pfn_to_page(mpfn[i]);
+
+		if (!page)
+			continue;
+
+		WARN_ON_ONCE(!trylock_page(page));
+		++*mpages;
+	}
+
+	return 0;
+
+free_pages:
+	for (i = 0; i < npages; ++i) {
+		struct page *page = migrate_pfn_to_page(mpfn[i]);
+
+		if (!page)
+			continue;
+
+		put_page(page);
+		mpfn[i] = 0;
+	}
+	return -ENOMEM;
+}
+
+/**
+ * drm_gpusvm_evict_to_ram() - Evict GPU SVM range to RAM
+ * @devmem_allocation: Pointer to the device memory allocation
+ *
+ * Similar to __drm_gpusvm_migrate_to_ram but does not require mmap

lock and
+ * migration done via migrate_device_* functions.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_evict_to_ram(struct drm_gpusvm_devmem *devmem_allocation)

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+{
+	const struct drm_gpusvm_devmem_ops *ops = devmem_allocation-
Quoted 1 line

ops;

drivers/gpu/drm/drm_gpusvm.c new file +25 -0
+	unsigned long npages, mpages = 0;
+	struct page **pages;
+	unsigned long *src, *dst;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+	unsigned int retry_count = 2;
+
+	npages = devmem_allocation->size >> PAGE_SHIFT;
+
+retry:
+	if (!mmget_not_zero(devmem_allocation->mm))
+		return -EFAULT;
+
+	buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr)
+
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	src = buf;
+	dst = buf + (sizeof(*src) * npages);
+	dma_addr = buf + (2 * sizeof(*src) * npages);
+	pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) *

npages;

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+
+	err = ops->populate_devmem_pfn(devmem_allocation, npages,

src);

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+	if (err)
+		goto err_free;
+
+	err = migrate_device_pfns(src, npages);
+	if (err)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_ram_pfn(NULL, NULL,

npages, &mpages,

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+						  src, dst, 0);
+	if (err || !mpages)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(devmem_allocation->dev,

dma_addr,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					   dst, npages,

DMA_FROM_DEVICE);

drivers/gpu/drm/drm_gpusvm.c new file +16 -0
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(src[i]);
+
+	err = ops->copy_to_ram(pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_unlock_put_pages(npages, dst);
+	migrate_device_pages(src, dst, npages);
+	migrate_device_finalize(src, dst, npages);
+	drm_gpusvm_migrate_unmap_pages(devmem_allocation->dev,

dma_addr, npages,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+				       DMA_FROM_DEVICE);
+err_free:

drivers/gpu/drm/drm_gpusvm.c line 1929

Quoted 10 lines
drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+	kvfree(buf);
+err_out:
+	mmput_async(devmem_allocation->mm);
+
+	if (completion_done(&devmem_allocation->detached))
+		return 0;
+
+	if (!err || retry_count--) {
+		cond_resched();
+		goto retry;

I think we might end up in an infinite loop here if someone unknown holds a device page reference. Then err will be zero and we'll continously trying to evict that page until the holder drops its refcount. The caller would probably want to move on trying to evict something else. So what about retrying once in that case and then return -EBUSY?

Quoted 875 lines
drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+	}
+
+	return err;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_evict_to_ram);
+
+/**
+ * __drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM

(internal)
+ * @vas: Pointer to the VM area structure
+ * @device_private_page_owner: Device private pages owner
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @fault_addr: Fault address
+ * @size: Size of migration
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to RAM. It sets up the migration, populates + dma maps RAM PFNs, and
+ * invokes the driver-specific operations for migration to RAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static int __drm_gpusvm_migrate_to_ram(struct vm_area_struct *vas,
+				       void

*device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+				       struct page *page,
+				       unsigned long fault_addr,
+				       unsigned long size)
+{
+	struct migrate_vma migrate = {
+		.vma		= vas,
+		.pgmap_owner	= device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_DEVICE_PRIVATE

|

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+			MIGRATE_VMA_SELECT_DEVICE_COHERENT,
+		.fault_page	= page,
+	};
+	struct drm_gpusvm_zdd *zdd;
+	const struct drm_gpusvm_devmem_ops *ops;
+	struct device *dev;
+	unsigned long npages, mpages = 0;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	unsigned long start, end;
+	void *buf;
+	int i, err = 0;
+
+	start = ALIGN_DOWN(fault_addr, size);
+	end = ALIGN(fault_addr + 1, size);
+
+	/* Corner where VMA area struct has been partially unmapped

*/

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+	if (start < vas->vm_start)
+		start = vas->vm_start;
+	if (end > vas->vm_end)
+		end = vas->vm_end;
+
+	migrate.start = start;
+	migrate.end = end;
+	npages = npages_in_range(start, end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +

sizeof(*dma_addr) +

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr))

* npages;

drivers/gpu/drm/drm_gpusvm.c new file +30 -0
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/* Raced with another CPU fault, nothing to do */
+	if (!migrate.cpages)
+		goto err_free;
+
+	if (!page) {
+		for (i = 0; i < npages; ++i) {
+			if (!(migrate.src[i] & MIGRATE_PFN_MIGRATE))
+				continue;
+
+			page = migrate_pfn_to_page(migrate.src[i]);
+			break;
+		}
+
+		if (!page)
+			goto err_finalize;
+	}
+	zdd = page->zone_device_data;
+	ops = zdd->devmem_allocation->ops;
+	dev = zdd->devmem_allocation->dev;
+
+	err = drm_gpusvm_migrate_populate_ram_pfn(vas, page, npages,

&mpages,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						  migrate.src,

migrate.dst,

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+						  start);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(dev, dma_addr,

migrate.dst, npages,

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+					   DMA_FROM_DEVICE);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(migrate.src[i]);
+
+	err = ops->copy_to_ram(pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_unlock_put_pages(npages,

migrate.dst);

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(dev, dma_addr, npages,
+				       DMA_FROM_DEVICE);
+err_free:
+	kvfree(buf);
+err_out:
+
+	return err;
+}
+
+/**
+ * drm_gpusvm_range_evict - Evict GPU SVM range
+ * @pagemap: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range to be removed
+ *
+ * This function evicts the specified GPU SVM range. This function

will not
+ * evict coherent pages.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+int drm_gpusvm_range_evict(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_range *range)
+{
+	struct mmu_interval_notifier *notifier = &range->notifier-
Quoted 1 line

notifier;

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+	struct hmm_range hmm_range = {
+		.default_flags = HMM_PFN_REQ_FAULT,
+		.notifier = notifier,
+		.start = range->itree.start,
+		.end = range->itree.last + 1,
+		.dev_private_owner = NULL,
+	};
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+	unsigned long *pfns;
+	unsigned long npages = npages_in_range(range->itree.start,
+					       range->itree.last +

1);

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+	int err = 0;
+	struct mm_struct *mm = gpusvm->mm;
+
+	if (!mmget_not_zero(mm))
+		return -EFAULT;
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL);
+	if (!pfns)
+		return -ENOMEM;
+
+	hmm_range.hmm_pfns = pfns;
+	while (!time_after(jiffies, timeout)) {
+		hmm_range.notifier_seq =

mmu_interval_read_begin(notifier);

drivers/gpu/drm/drm_gpusvm.c new file +21 -0
+		if (time_after(jiffies, timeout)) {
+			err = -ETIME;
+			break;
+		}
+
+		mmap_read_lock(mm);
+		err = hmm_range_fault(&hmm_range);
+		mmap_read_unlock(mm);
+		if (err != -EBUSY)
+			break;
+	}
+
+	kvfree(pfns);
+	mmput(mm);
+
+	return err;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_evict);
+
+/**
+ * drm_gpusvm_page_free() - Put GPU SVM zone device data associated

with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device data
+ * associated with a page when it is being released.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+static void drm_gpusvm_page_free(struct page *page)
+{
+	drm_gpusvm_zdd_put(page->zone_device_data);
+}
+
+/**
+ * drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM (page

fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ * VM_FAULT_SIGBUS on failure, 0 on success.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+static vm_fault_t drm_gpusvm_migrate_to_ram(struct vm_fault *vmf)
+{
+	struct drm_gpusvm_zdd *zdd = vmf->page->zone_device_data;
+	int err;
+
+	err = __drm_gpusvm_migrate_to_ram(vmf->vma,
+					  zdd-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					  vmf->page, vmf->address,
+					  zdd->devmem_allocation-
Quoted 1 line

size);

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+
+	return err ? VM_FAULT_SIGBUS : 0;
+}
+
+/**
+ * drm_gpusvm_pagemap_ops() - Device page map operations for GPU SVM
+ */
+static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = {
+	.page_free = drm_gpusvm_page_free,
+	.migrate_to_ram = drm_gpusvm_migrate_to_ram,
+};
+
+/**
+ * drm_gpusvm_pagemap_ops_get() - Retrieve GPU SVM device page map

operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void)
+{
+	return &drm_gpusvm_pagemap_ops;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_pagemap_ops_get);
+
+/**
+ * drm_gpusvm_has_mapping() - Check if GPU SVM has mapping for the

given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, unsigned long start,

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+			    unsigned long end)
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	drm_gpusvm_for_each_notifier(notifier, gpusvm, start, end) {
+		struct drm_gpusvm_range *range = NULL;
+
+		drm_gpusvm_for_each_range(range, notifier, start,

end)

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+			return true;
+	}
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_has_mapping);
+
+/**
+ * drm_gpusvm_range_set_unmapped() - Mark a GPU SVM range as

unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets the partial_unmap flag
+ * if the range partially falls within the provided MMU notifier range.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+void drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,
+				   const struct mmu_notifier_range

*mmu_range)

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+{
+	lockdep_assert_held_write(&range->gpusvm->notifier_lock);
+
+	range->flags.unmapped = true;
+	if (range->itree.start < mmu_range->start ||
+	    range->itree.last + 1 > mmu_range->end)
+		range->flags.partial_unmap = true;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_set_unmapped);
+
+/**
+ * drm_gpusvm_devmem_init() - Initialize a GPU SVM device memory

allocation
+ *
+ * @dev: Pointer to the device structure which device memory allocation belongs to
+ * @mm: Pointer to the mm_struct for the address space
+ * @ops: Pointer to the operations structure for GPU SVM device memory
+ * @dpagemap: The struct drm_pagemap we're allocating from.
+ * @size: Size of device memory allocation
+ */
+void drm_gpusvm_devmem_init(struct drm_gpusvm_devmem *devmem_allocation,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+			    struct device *dev, struct mm_struct

*mm,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+			    const struct drm_gpusvm_devmem_ops *ops,
+			    struct drm_pagemap *dpagemap, size_t

size)

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+{
+	init_completion(&devmem_allocation->detached);
+	devmem_allocation->dev = dev;
+	devmem_allocation->mm = mm;
+	devmem_allocation->ops = ops;
+	devmem_allocation->dpagemap = dpagemap;
+	devmem_allocation->size = size;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_devmem_init);
+
+MODULE_DESCRIPTION("DRM GPUSVM");
+MODULE_LICENSE("GPL");
include/drm/drm_gpusvm.h new file +24 -0
new file mode 100644
index 000000000000..ea31db0be841
@@ -0,0 +1,445 @@
+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef __DRM_GPUSVM_H__
+#define __DRM_GPUSVM_H__
+
+#include <linux/kref.h>
+#include <linux/interval_tree.h>
+#include <linux/mmu_notifier.h>
+
+struct dev_pagemap_ops;
+struct drm_device;
+struct drm_gpusvm;
+struct drm_gpusvm_notifier;
+struct drm_gpusvm_ops;
+struct drm_gpusvm_range;
+struct drm_gpusvm_devmem;
+struct drm_pagemap;
+struct drm_pagemap_dma_addr;
+
+/**
+ * struct drm_gpusvm_devmem_ops - Operations structure for GPU SVM

device memory
+ *
+ * This structure defines the operations for GPU Shared Virtual Memory (SVM)
+ * device memory. These operations are provided by the GPU driver to manage device memory
+ * allocations and perform operations such as migration between device memory and system
+ * RAM.
+ */

include/drm/drm_gpusvm.h new file +3 -0
+struct drm_gpusvm_devmem_ops {
+	/**
+	 * @devmem_release: Release device memory allocation

(optional)

include/drm/drm_gpusvm.h new file +3 -0
+	 * @devmem_allocation: device memory allocation
+	 *
+	 * Release device memory allocation and drop a reference to

device

include/drm/drm_gpusvm.h new file +3 -0
+	 * memory allocation.
+	 */
+	void (*devmem_release)(struct drm_gpusvm_devmem

*devmem_allocation);

include/drm/drm_gpusvm.h new file +3 -0
+
+	/**
+	 * @populate_devmem_pfn: Populate device memory PFN

(required for migration)

include/drm/drm_gpusvm.h new file +10 -0
+	 * @devmem_allocation: device memory allocation
+	 * @npages: Number of pages to populate
+	 * @pfn: Array of page frame numbers to populate
+	 *
+	 * Populate device memory page frame numbers (PFN).
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*populate_devmem_pfn)(struct drm_gpusvm_devmem

*devmem_allocation,

include/drm/drm_gpusvm.h new file +1 -0
+				   unsigned long npages, unsigned

long *pfn);

include/drm/drm_gpusvm.h new file +3 -0
+
+	/**
+	 * @copy_to_devmem: Copy to device memory (required for

migration)

include/drm/drm_gpusvm.h new file +1 -0
+	 * @pages: Pointer to array of device memory pages

(destination)

include/drm/drm_gpusvm.h new file +16 -0
+	 * @dma_addr: Pointer to array of DMA addresses (source)
+	 * @npages: Number of pages to copy
+	 *
+	 * Copy pages to device memory.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_devmem)(struct page **pages,
+			      dma_addr_t *dma_addr,
+			      unsigned long npages);
+
+	/**
+	 * @copy_to_ram: Copy to system RAM (required for migration)
+	 * @pages: Pointer to array of device memory pages (source)
+	 * @dma_addr: Pointer to array of DMA addresses

(destination)

include/drm/drm_gpusvm.h new file +14 -0
+	 * @npages: Number of pages to copy
+	 *
+	 * Copy pages to system RAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_ram)(struct page **pages,
+			   dma_addr_t *dma_addr,
+			   unsigned long npages);
+};
+
+/**
+ * struct drm_gpusvm_devmem - Structure representing a GPU SVM

device memory allocation
+ *
+ * @dev: Pointer to the device structure which device memory allocation belongs to
+ * @mm: Pointer to the mm_struct for the address space
+ * @detached: device memory allocations is detached from device pages
+ * @ops: Pointer to the operations structure for GPU SVM device memory
+ * @dpagemap: The struct drm_pagemap of the pages this allocation belongs to.
+ * @size: Size of device memory allocation
+ */

include/drm/drm_gpusvm.h new file +13 -0
+struct drm_gpusvm_devmem {
+	struct device *dev;
+	struct mm_struct *mm;
+	struct completion detached;
+	const struct drm_gpusvm_devmem_ops *ops;
+	struct drm_pagemap *dpagemap;
+	size_t size;
+};
+
+/**
+ * struct drm_gpusvm_ops - Operations structure for GPU SVM
+ *
+ * This structure defines the operations for GPU Shared Virtual

Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * notifiers.
+ */

include/drm/drm_gpusvm.h new file +8 -0
+struct drm_gpusvm_ops {
+	/**
+	 * @notifier_alloc: Allocate a GPU SVM notifier (optional)
+	 *
+	 * Allocate a GPU SVM notifier.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM notifier on success,

NULL on failure.

include/drm/drm_gpusvm.h new file +19 -0
+	 */
+	struct drm_gpusvm_notifier *(*notifier_alloc)(void);
+
+	/**
+	 * @notifier_free: Free a GPU SVM notifier (optional)
+	 * @notifier: Pointer to the GPU SVM notifier to be freed
+	 *
+	 * Free a GPU SVM notifier.
+	 */
+	void (*notifier_free)(struct drm_gpusvm_notifier *notifier);
+
+	/**
+	 * @range_alloc: Allocate a GPU SVM range (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 *
+	 * Allocate a GPU SVM range.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM range on success, NULL

on failure.

include/drm/drm_gpusvm.h new file +2 -0
+	 */
+	struct drm_gpusvm_range *(*range_alloc)(struct drm_gpusvm

*gpusvm);

include/drm/drm_gpusvm.h new file +16 -0
+
+	/**
+	 * @range_free: Free a GPU SVM range (optional)
+	 * @range: Pointer to the GPU SVM range to be freed
+	 *
+	 * Free a GPU SVM range.
+	 */
+	void (*range_free)(struct drm_gpusvm_range *range);
+
+	/**
+	 * @invalidate: Invalidate GPU SVM notifier (required)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @notifier: Pointer to the GPU SVM notifier
+	 * @mmu_range: Pointer to the mmu_notifier_range structure
+	 *
+	 * Invalidate the GPU page tables. It can safely walk the

notifier range

include/drm/drm_gpusvm.h new file +1 -0
+	 * RB tree/list in this function. Called while holding the

notifier lock.

include/drm/drm_gpusvm.h new file +4 -0
+	 */
+	void (*invalidate)(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_notifier *notifier,
+			   const struct mmu_notifier_range

*mmu_range);

include/drm/drm_gpusvm.h new file +4 -0
+};
+
+/**
+ * struct drm_gpusvm_notifier - Structure representing a GPU SVM

notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @itree: Interval tree node for the notifier (inserted in GPU SVM)
+ * @entry: List entry to fast interval tree traversal
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ *              interval tree. This is useful to keep iterating ranges while
+ *              doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier has been
+ *                 removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */

include/drm/drm_gpusvm.h new file +19 -0
+struct drm_gpusvm_notifier {
+	struct drm_gpusvm *gpusvm;
+	struct mmu_interval_notifier notifier;
+	struct interval_tree_node itree;
+	struct list_head entry;
+	struct rb_root_cached root;
+	struct list_head range_list;
+	struct {
+		u32 removed : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm_range - Structure representing a GPU SVM range
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier
+ * @refcount: Reference count for the range
+ * @itree: Interval tree node for the range (inserted in GPU SVM

notifier)
+ * @entry: List entry to fast interval tree traversal
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @dma_addr: DMA address array
+ * @dpagemap: The struct drm_pagemap of the device pages we're dma- mapping.
+ *            Note this is assuming only one drm_pagemap per range is allowed.
+ * @flags.migrate_devmem: Flag indicating whether the range can be migrated to device memory
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_devmem_pages: Flag indicating if the range has devmem pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA mapping
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */

include/drm/drm_gpusvm.h new file +13 -0
+struct drm_gpusvm_range {
+	struct drm_gpusvm *gpusvm;
+	struct drm_gpusvm_notifier *notifier;
+	struct kref refcount;
+	struct interval_tree_node itree;
+	struct list_head entry;
+	unsigned long notifier_seq;
+	struct drm_pagemap_dma_addr *dma_addr;
+	struct drm_pagemap *dpagemap;
+	struct {
+		/* All flags below must be set upon creation */
+		u16 migrate_devmem : 1;
+		/* All flags below must be set / cleared under

notifier lock */

include/drm/drm_gpusvm.h new file +19 -0
+		u16 unmapped : 1;
+		u16 partial_unmap : 1;
+		u16 has_devmem_pages : 1;
+		u16 has_dma_mapping : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm - GPU SVM structure
+ *
+ * @name: Name of the GPU SVM
+ * @drm: Pointer to the DRM device structure
+ * @mm: Pointer to the mm_struct for the address space
+ * @device_private_page_owner: Device private pages owner
+ * @mm_start: Start address of GPU SVM
+ * @mm_range: Range of the GPU SVM
+ * @notifier_size: Size of individual notifiers
+ * @ops: Pointer to the operations structure for GPU SVM
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range

allocation.
+ *               Entries should be powers of 2 in descending order.
+ * @num_chunks: Number of chunks
+ * @notifier_lock: Read-write semaphore for protecting notifier operations
+ * @root: Cached root node of the Red-Black tree containing GPU SVM notifiers
+ * @notifier_list: list head containing of notifiers in the same order they
+ *                 appear in interval tree. This is useful to keep iterating
+ *                 notifiers while doing modifications to RB tree.
+ *
+ * This structure represents a GPU SVM (Shared Virtual Memory) used for tracking
+ * memory ranges mapped in a DRM (Direct Rendering Manager) device.
+ *
+ * No reference counting is provided, as this is expected to be embedded in the
+ * driver VM structure along with the struct drm_gpuvm, which handles reference
+ * counting.
+ */

include/drm/drm_gpusvm.h new file +17 -0
+struct drm_gpusvm {
+	const char *name;
+	struct drm_device *drm;
+	struct mm_struct *mm;
+	void *device_private_page_owner;
+	unsigned long mm_start;
+	unsigned long mm_range;
+	unsigned long notifier_size;
+	const struct drm_gpusvm_ops *ops;
+	const unsigned long *chunk_sizes;
+	int num_chunks;
+	struct rw_semaphore notifier_lock;
+	struct rb_root_cached root;
+	struct list_head notifier_list;
+#ifdef CONFIG_LOCKDEP
+	/**
+	 * @lock_dep_map: Annotates drm_gpusvm_range_find_or_insert

and

include/drm/drm_gpusvm.h new file +10 -0
+	 * drm_gpusvm_range_remove with a driver provided lock.
+	 */
+	struct lockdep_map *lock_dep_map;
+#endif
+};
+
+/**
+ * struct drm_gpusvm_ctx - DRM GPU SVM context
+ *
+ * @check_pages_threshold: Check CPU pages for present if chunk is

less than or
+ *                         equal to threshold. If not present, reduce chunk
+ *                         size.
+ * @in_notifier: entering from a MMU notifier
+ * @read_only: operating on read-only memory
+ * @devmem_possible: possible to use device memory
+ *
+ * Context that is DRM GPUSVM is operating in (i.e. user arguments).
+ */

include/drm/drm_gpusvm.h new file +10 -0
+struct drm_gpusvm_ctx {
+	unsigned long check_pages_threshold;
+	unsigned int in_notifier :1;
+	unsigned int read_only :1;
+	unsigned int devmem_possible :1;
+};
+
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void

*device_private_page_owner,

include/drm/drm_gpusvm.h new file +4 -0
+		    unsigned long mm_start, unsigned long mm_range,
+		    unsigned long notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const unsigned long *chunk_sizes, int

num_chunks);

include/drm/drm_gpusvm.h new file +37 -0
+
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm);
+
+void drm_gpusvm_free(struct drm_gpusvm *gpusvm);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm,
+				unsigned long fault_addr,
+				unsigned long gpuva_start,
+				unsigned long gpuva_end,
+				const struct drm_gpusvm_ctx *ctx);
+
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range);
+
+int drm_gpusvm_range_evict(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_range *range);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range);
+
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range);
+
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range);
+
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx);
+
+int drm_gpusvm_migrate_to_devmem(struct drm_gpusvm *gpusvm,
+				 struct drm_gpusvm_range *range,
+				 struct drm_gpusvm_devmem

*devmem_allocation,

include/drm/drm_gpusvm.h new file +3 -0
+				 const struct drm_gpusvm_ctx *ctx);
+
+int drm_gpusvm_evict_to_ram(struct drm_gpusvm_devmem

*devmem_allocation);

include/drm/drm_gpusvm.h new file +4 -0
+
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void);
+
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, unsigned long

start,

include/drm/drm_gpusvm.h new file +4 -0
+			    unsigned long end);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, unsigned

long start,

include/drm/drm_gpusvm.h new file +4 -0
+		      unsigned long end);
+
+void drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,
+				   const struct mmu_notifier_range

*mmu_range);

include/drm/drm_gpusvm.h new file +2 -0
+
+void drm_gpusvm_devmem_init(struct drm_gpusvm_devmem

*devmem_allocation,

include/drm/drm_gpusvm.h new file +1 -0
+			    struct device *dev, struct mm_struct

*mm,

include/drm/drm_gpusvm.h new file +2 -0
+			    const struct drm_gpusvm_devmem_ops *ops,
+			    struct drm_pagemap *dpagemap, size_t

size);

include/drm/drm_gpusvm.h new file +4 -0
+
+#ifdef CONFIG_LOCKDEP
+/**
+ * drm_gpusvm_driver_set_lock() - Set the lock protecting accesses

to GPU SVM
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @lock: the lock used to protect the gpuva list. The locking primitive
+ * must contain a dep_map field.
+ *
+ * Call this to annotate drm_gpusvm_range_find_or_insert and
+ * drm_gpusvm_range_remove.
+ */

include/drm/drm_gpusvm.h new file +4 -0
+#define drm_gpusvm_driver_set_lock(gpusvm, lock) \
+	do { \
+		if (!WARN((gpusvm)->lock_dep_map, \
+			  "GPUSVM range lock should be set only

once."))\

include/drm/drm_gpusvm.h new file +1 -0
+			(gpusvm)->lock_dep_map = &(lock)-
Quoted 1 line

dep_map; \

include/drm/drm_gpusvm.h new file +31 -0
+	} while (0)
+#define drm_gpusvm_driver_lock_held(gpusvm) \
+	do { \
+		if ((gpusvm)->lock_dep_map)	\
+			lock_is_held((gpusvm)->lock_dep_map);	\
+	} while (0)
+#else
+#define drm_gpusvm_driver_set_lock(gpusvm, lock) do {} while (0)
+#define drm_gpusvm_driver_lock_held(gpusvm) do {} while (0)
+#endif
+
+/**
+ * drm_gpusvm_notifier_lock() - Lock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, take lock
+ */
+#define drm_gpusvm_notifier_lock(gpusvm__)	\
+	down_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * drm_gpusvm_notifier_unlock() - Unlock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, drop lock
+ */
+#define drm_gpusvm_notifier_unlock(gpusvm__)	\
+	up_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * __drm_gpusvm_range_next() - Get the next GPU SVM range in the

list
+ * @range: a pointer to the current GPU SVM range
+ *
+ * Return: A pointer to the next drm_gpusvm_range if available, or NULL if the
+ *         current range is the last one or if the input range is NULL.
+ */

include/drm/drm_gpusvm.h new file +12 -0
+static inline struct drm_gpusvm_range *
+__drm_gpusvm_range_next(struct drm_gpusvm_range *range)
+{
+	if (range && !list_is_last(&range->entry,
+				   &range->notifier->range_list))
+		return list_next_entry(range, entry);
+
+	return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_range() - Iterate over GPU SVM ranges in a

notifier
+ * @range__: Iterator variable for the ranges. If set, it indicates the start of
+ *      the iterator. If NULL, call drm_gpusvm_range_find() to
get the range.
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier. It is safe
+ * to use while holding the driver SVM lock or the notifier lock.
+ */
+#define drm_gpusvm_for_each_range(range__, notifier__, start__,
end__) \

include/drm/drm_gpusvm.h new file +1 -0
+	for ((range__) = (range__)

?: \

include/drm/drm_gpusvm.h new file +1 -0
+	     drm_gpusvm_range_find((notifier__), (start__),

(end__)); \

include/drm/drm_gpusvm.h new file +1 -0
+	     (range__) && (range__->itree.start <

(end__)); \

include/drm/drm_gpusvm.h new file +3 -0
+	     (range__) = __drm_gpusvm_range_next(range__))
+
+#endif /* __DRM_GPUSVM_H__ */
#17 lore raw

Re: [PATCH v4 06/33] drm/gpusvm: Add support for GPU Shared Virtual Memory

In reply to Matthew Brost
Quoted 75 lines · Matthew Brost

This patch introduces support for GPU Shared Virtual Memory (SVM) in the Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.

The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also provides mechanisms for allocating, deallocating, and migrating memory regions between system RAM and GPU VRAM.

This is largely inspired by GPUVM.

v2:
- Take order into account in check pages
- Clear range->pages in get pages error
- Drop setting dirty or accessed bit in get pages (Vetter)
- Remove mmap assert for cpu faults
- Drop mmap write lock abuse (Vetter, Christian)
- Decouple zdd from range (Vetter, Oak)
- Add drm_gpusvm_range_evict, make it work with coherent pages
- Export drm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
- mmget/put in drm_gpusvm_evict_to_sram
- Drop range->vram_alloation variable
- Don't return in drm_gpusvm_evict_to_sram until all pages detached
- Don't warn on mixing sram and device pages
- Update kernel doc
- Add coherent page support to get pages
- Use DMA_FROM_DEVICE rather than DMA_BIDIRECTIONAL
- Add struct drm_gpusvm_vram and ops (Thomas)
- Update the range's seqno if the range is valid (Thomas)
- Remove the is_unmapped check before hmm_range_fault (Thomas)
- Use drm_pagemap (Thomas)
- Drop kfree_mapping (Thomas)
- dma mapp pages under notifier lock (Thomas)
- Remove ctx.prefault
- Remove ctx.mmap_locked
- Add ctx.check_pages
- s/vram/devmem (Thomas)
v3:
- Fix memory leak drm_gpusvm_range_get_pages
- Only migrate pages with same zdd on CPU fault
- Loop over al VMAs in drm_gpusvm_range_evict
- Make GPUSVM a drm level module
- GPL or MIT license
- Update main kernel doc (Thomas)
- Prefer foo() vs foo for functions in kernel doc (Thomas)
- Prefer functions over macros (Thomas)
- Use unsigned long vs u64 for addresses (Thomas)
- Use standard interval_tree (Thomas)
- s/drm_gpusvm_migration_put_page/drm_gpusvm_migration_unlock_put_page (Thomas)
- Drop err_out label in drm_gpusvm_range_find_or_insert (Thomas)
- Fix kernel doc in drm_gpusvm_range_free_pages (Thomas)
- Newlines between functions defs in header file (Thomas)
- Drop shall language in driver vfunc kernel doc (Thomas)
- Move some static inlines from head to C file (Thomas)
- Don't allocate pages under page lock in drm_gpusvm_migrate_populate_ram_pfn (Thomas)
- Change check_pages to a threshold
v4:
- Fix NULL ptr deref in drm_gpusvm_migrate_populate_ram_pfn (Thomas, Himal)
- Fix check pages threshold
- Check for range being unmapped under notifier lock in get pages (Testing)
- Fix characters per line
- Drop WRITE_ONCE for zdd->devmem_allocation assignment (Thomas)
- Use completion for devmem_allocation->detached (Thomas)
- Make GPU SVM depend on ZONE_DEVICE (CI)
- Use hmm_range_fault for eviction (Thomas)
- Drop zdd worker (Thomas)

---

<snip>

Quoted 103 lines
drivers/gpu/drm/drm_gpusvm.c new file +103 -0
+/**
+ * __drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM (internal)
+ * @vas: Pointer to the VM area structure
+ * @device_private_page_owner: Device private pages owner
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @fault_addr: Fault address
+ * @size: Size of migration
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to RAM. It sets up the migration, populates + dma maps RAM PFNs, and
+ * invokes the driver-specific operations for migration to RAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int __drm_gpusvm_migrate_to_ram(struct vm_area_struct *vas,
+				       void *device_private_page_owner,
+				       struct page *page,
+				       unsigned long fault_addr,
+				       unsigned long size)
+{
+	struct migrate_vma migrate = {
+		.vma		= vas,
+		.pgmap_owner	= device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_DEVICE_PRIVATE |
+			MIGRATE_VMA_SELECT_DEVICE_COHERENT,
+		.fault_page	= page,
+	};
+	struct drm_gpusvm_zdd *zdd;
+	const struct drm_gpusvm_devmem_ops *ops;
+	struct device *dev;
+	unsigned long npages, mpages = 0;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	unsigned long start, end;
+	void *buf;
+	int i, err = 0;
+
+	start = ALIGN_DOWN(fault_addr, size);
+	end = ALIGN(fault_addr + 1, size);
+
+	/* Corner where VMA area struct has been partially unmapped */
+	if (start < vas->vm_start)
+		start = vas->vm_start;
+	if (end > vas->vm_end)
+		end = vas->vm_end;
+
+	migrate.start = start;
+	migrate.end = end;
+	npages = npages_in_range(start, end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/* Raced with another CPU fault, nothing to do */
+	if (!migrate.cpages)
+		goto err_free;
+
+	if (!page) {
+		for (i = 0; i < npages; ++i) {
+			if (!(migrate.src[i] & MIGRATE_PFN_MIGRATE))
+				continue;
+
+			page = migrate_pfn_to_page(migrate.src[i]);
+			break;
+		}
+
+		if (!page)
+			goto err_finalize;
+	}
+	zdd = page->zone_device_data;
+	ops = zdd->devmem_allocation->ops;
+	dev = zdd->devmem_allocation->dev;
+
+	err = drm_gpusvm_migrate_populate_ram_pfn(vas, page, npages, &mpages,
+						  migrate.src, migrate.dst,
+						  start);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(dev, dma_addr, migrate.dst, npages,
+					   DMA_FROM_DEVICE);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(migrate.src[i]);
+
+	err = ops->copy_to_ram(pages, dma_addr, npages);

drivers/gpu/drm/drm_gpusvm.c line 2048

Quoted 10 lines
drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_unlock_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(dev, dma_addr, npages,
+				       DMA_FROM_DEVICE);

clang for me is throwing:

drivers/gpu/drm/drm_gpusvm.c:2017:7: error: variable 'dev' is used uninitialized whenever 'if' condition is true [-Werror,-Wsometimes-uninitialized]
2017 | if (!page)
| ^~~~~
drivers/gpu/drm/drm_gpusvm.c:2047:33: note: uninitialized use occurs here
2047 | drm_gpusvm_migrate_unmap_pages(dev, dma_addr, npages,
| ^~~
drivers/gpu/drm/drm_gpusvm.c:2017:3: note: remove the 'if' if its condition is always false

 2017 |                 if (!page)
      |                 ^~~~~~~~~~
 2018 |                         goto err_finalize;
      |                         ~~~~~~~~~~~~~~~~~
drivers/gpu/drm/drm_gpusvm.c:1966:20: note: initialize the variable 'dev' to silence this warning
 1966 |         struct device *dev;
      |                           ^
      |                            = NULL

1 error generated.

#18 lore raw

Re: [PATCH v4 06/33] drm/gpusvm: Add support for GPU Shared Virtual Memory

Quoted 201 lines · Matthew Auld
Quoted 75 lines · Matthew Brost

This patch introduces support for GPU Shared Virtual Memory (SVM) in the Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.

The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also provides mechanisms for allocating, deallocating, and migrating memory regions between system RAM and GPU VRAM.

This is largely inspired by GPUVM.

v2:
  - Take order into account in check pages
  - Clear range->pages in get pages error
  - Drop setting dirty or accessed bit in get pages (Vetter)
  - Remove mmap assert for cpu faults
  - Drop mmap write lock abuse (Vetter, Christian)
  - Decouple zdd from range (Vetter, Oak)
  - Add drm_gpusvm_range_evict, make it work with coherent pages
  - Export drm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
  - mmget/put in drm_gpusvm_evict_to_sram
  - Drop range->vram_alloation variable
  - Don't return in drm_gpusvm_evict_to_sram until all pages detached
  - Don't warn on mixing sram and device pages
  - Update kernel doc
  - Add coherent page support to get pages
  - Use DMA_FROM_DEVICE rather than DMA_BIDIRECTIONAL
  - Add struct drm_gpusvm_vram and ops (Thomas)
  - Update the range's seqno if the range is valid (Thomas)
  - Remove the is_unmapped check before hmm_range_fault (Thomas)
  - Use drm_pagemap (Thomas)
  - Drop kfree_mapping (Thomas)
  - dma mapp pages under notifier lock (Thomas)
  - Remove ctx.prefault
  - Remove ctx.mmap_locked
  - Add ctx.check_pages
  - s/vram/devmem (Thomas)
v3:
  - Fix memory leak drm_gpusvm_range_get_pages
  - Only migrate pages with same zdd on CPU fault
  - Loop over al VMAs in drm_gpusvm_range_evict
  - Make GPUSVM a drm level module
  - GPL or MIT license
  - Update main kernel doc (Thomas)
  - Prefer foo() vs foo for functions in kernel doc (Thomas)
  - Prefer functions over macros (Thomas)
  - Use unsigned long vs u64 for addresses (Thomas)
  - Use standard interval_tree (Thomas)
  - s/drm_gpusvm_migration_put_page/ drm_gpusvm_migration_unlock_put_page (Thomas)
  - Drop err_out label in drm_gpusvm_range_find_or_insert (Thomas)
  - Fix kernel doc in drm_gpusvm_range_free_pages (Thomas)
  - Newlines between functions defs in header file (Thomas)
  - Drop shall language in driver vfunc kernel doc (Thomas)
  - Move some static inlines from head to C file (Thomas)
  - Don't allocate pages under page lock in drm_gpusvm_migrate_populate_ram_pfn (Thomas)
  - Change check_pages to a threshold
v4:
  - Fix NULL ptr deref in drm_gpusvm_migrate_populate_ram_pfn (Thomas, Himal)
  - Fix check pages threshold
  - Check for range being unmapped under notifier lock in get pages (Testing)
  - Fix characters per line
  - Drop WRITE_ONCE for zdd->devmem_allocation assignment (Thomas)
  - Use completion for devmem_allocation->detached (Thomas)
  - Make GPU SVM depend on ZONE_DEVICE (CI)
  - Use hmm_range_fault for eviction (Thomas)
  - Drop zdd worker (Thomas)

---

<snip>

Quoted 109 lines

+/**
+ * __drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM (internal)
+ * @vas: Pointer to the VM area structure
+ * @device_private_page_owner: Device private pages owner
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @fault_addr: Fault address
+ * @size: Size of migration
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to RAM. It sets up the migration, populates + dma maps RAM PFNs, and
+ * invokes the driver-specific operations for migration to RAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +37 -0
+static int __drm_gpusvm_migrate_to_ram(struct vm_area_struct *vas,
+                       void *device_private_page_owner,
+                       struct page *page,
+                       unsigned long fault_addr,
+                       unsigned long size)
+{
+    struct migrate_vma migrate = {
+        .vma        = vas,
+        .pgmap_owner    = device_private_page_owner,
+        .flags        = MIGRATE_VMA_SELECT_DEVICE_PRIVATE |
+            MIGRATE_VMA_SELECT_DEVICE_COHERENT,
+        .fault_page    = page,
+    };
+    struct drm_gpusvm_zdd *zdd;
+    const struct drm_gpusvm_devmem_ops *ops;
+    struct device *dev;
+    unsigned long npages, mpages = 0;
+    struct page **pages;
+    dma_addr_t *dma_addr;
+    unsigned long start, end;
+    void *buf;
+    int i, err = 0;
+
+    start = ALIGN_DOWN(fault_addr, size);
+    end = ALIGN(fault_addr + 1, size);
+
+    /* Corner where VMA area struct has been partially unmapped */
+    if (start < vas->vm_start)
+        start = vas->vm_start;
+    if (end > vas->vm_end)
+        end = vas->vm_end;
+
+    migrate.start = start;
+    migrate.end = end;
+    npages = npages_in_range(start, end);
+
+    buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + 

sizeof(*dma_addr) +
+               sizeof(*pages), GFP_KERNEL);
+    if (!buf) {
+        err = -ENOMEM;
+        goto err_out;
+    }
+    dma_addr = buf + (2 * sizeof(*migrate.src) * npages); +    pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;

drivers/gpu/drm/drm_gpusvm.c new file +30 -0
+
+    migrate.vma = vas;
+    migrate.src = buf;
+    migrate.dst = migrate.src + npages;
+
+    err = migrate_vma_setup(&migrate);
+    if (err)
+        goto err_free;
+
+    /* Raced with another CPU fault, nothing to do */
+    if (!migrate.cpages)
+        goto err_free;
+
+    if (!page) {
+        for (i = 0; i < npages; ++i) {
+            if (!(migrate.src[i] & MIGRATE_PFN_MIGRATE))
+                continue;
+
+            page = migrate_pfn_to_page(migrate.src[i]);
+            break;
+        }
+
+        if (!page)
+            goto err_finalize;
+    }
+    zdd = page->zone_device_data;
+    ops = zdd->devmem_allocation->ops;
+    dev = zdd->devmem_allocation->dev;
+
+    err = drm_gpusvm_migrate_populate_ram_pfn(vas, page, npages, 

&mpages,
+                          migrate.src, migrate.dst,
+                          start);
+    if (err)
+        goto err_finalize;
+
+    err = drm_gpusvm_migrate_map_pages(dev, dma_addr, migrate.dst, npages,
+                       DMA_FROM_DEVICE);
+    if (err)
+        goto err_finalize;

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+    for (i = 0; i < npages; ++i)
+        pages[i] = migrate_pfn_to_page(migrate.src[i]);
+
+    err = ops->copy_to_ram(pages, dma_addr, npages);

drivers/gpu/drm/drm_gpusvm.c line 2048

Quoted 10 lines
drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+    if (err)
+        goto err_finalize;
+
+err_finalize:
+    if (err)
+        drm_gpusvm_migration_unlock_put_pages(npages, migrate.dst);
+    migrate_vma_pages(&migrate);
+    migrate_vma_finalize(&migrate);
+    drm_gpusvm_migrate_unmap_pages(dev, dma_addr, npages,
+                       DMA_FROM_DEVICE);

clang for me is throwing:

Quoted 16 lines

drivers/gpu/drm/drm_gpusvm.c:2017:7: error: variable 'dev' is used uninitialized whenever 'if' condition is true [-Werror,-Wsometimes- uninitialized]
 2017 |                 if (!page)
      |                     ^~~~~
drivers/gpu/drm/drm_gpusvm.c:2047:33: note: uninitialized use occurs here
 2047 |         drm_gpusvm_migrate_unmap_pages(dev, dma_addr, npages,
      |                                        ^~~
drivers/gpu/drm/drm_gpusvm.c:2017:3: note: remove the 'if' if its condition is always false
 2017 |                 if (!page)
      |                 ^~~~~~~~~~
 2018 |                         goto err_finalize;
      |                         ~~~~~~~~~~~~~~~~~
drivers/gpu/drm/drm_gpusvm.c:1966:20: note: initialize the variable 'dev' to silence this warning
 1966 |         struct device *dev;
      |                           ^
      |                            = NULL
1 error generated.

I also reported this issue in the v3 patch, but it doesn't seem to have been fixed in v4 yet.

https://lore.kernel.org/dri-devel/[email protected]/

Br,

G.G.

author maintainer
#19 lore raw

Re: [PATCH v4 06/33] drm/gpusvm: Add support for GPU Shared Virtual Memory

Quoted 229 lines · Gwan-gyeong Mun
Quoted 217 lines · Matthew Auld
Quoted 79 lines · Matthew Brost

This patch introduces support for GPU Shared Virtual Memory (SVM) in the Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.

The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also provides mechanisms for allocating, deallocating, and migrating memory regions between system RAM and GPU VRAM.

This is largely inspired by GPUVM.

v2:
  - Take order into account in check pages
  - Clear range->pages in get pages error
  - Drop setting dirty or accessed bit in get pages (Vetter)
  - Remove mmap assert for cpu faults
  - Drop mmap write lock abuse (Vetter, Christian)
  - Decouple zdd from range (Vetter, Oak)
  - Add drm_gpusvm_range_evict, make it work with coherent pages
  - Export drm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
  - mmget/put in drm_gpusvm_evict_to_sram
  - Drop range->vram_alloation variable
  - Don't return in drm_gpusvm_evict_to_sram until all pages detached
  - Don't warn on mixing sram and device pages
  - Update kernel doc
  - Add coherent page support to get pages
  - Use DMA_FROM_DEVICE rather than DMA_BIDIRECTIONAL
  - Add struct drm_gpusvm_vram and ops (Thomas)
  - Update the range's seqno if the range is valid (Thomas)
  - Remove the is_unmapped check before hmm_range_fault (Thomas)
  - Use drm_pagemap (Thomas)
  - Drop kfree_mapping (Thomas)
  - dma mapp pages under notifier lock (Thomas)
  - Remove ctx.prefault
  - Remove ctx.mmap_locked
  - Add ctx.check_pages
  - s/vram/devmem (Thomas)
v3:
  - Fix memory leak drm_gpusvm_range_get_pages
  - Only migrate pages with same zdd on CPU fault
  - Loop over al VMAs in drm_gpusvm_range_evict
  - Make GPUSVM a drm level module
  - GPL or MIT license
  - Update main kernel doc (Thomas)
  - Prefer foo() vs foo for functions in kernel doc (Thomas)
  - Prefer functions over macros (Thomas)
  - Use unsigned long vs u64 for addresses (Thomas)
  - Use standard interval_tree (Thomas)
  - s/drm_gpusvm_migration_put_page/ drm_gpusvm_migration_unlock_put_page (Thomas)
  - Drop err_out label in drm_gpusvm_range_find_or_insert (Thomas)
  - Fix kernel doc in drm_gpusvm_range_free_pages (Thomas)
  - Newlines between functions defs in header file (Thomas)
  - Drop shall language in driver vfunc kernel doc (Thomas)
  - Move some static inlines from head to C file (Thomas)
  - Don't allocate pages under page lock in drm_gpusvm_migrate_populate_ram_pfn (Thomas)
  - Change check_pages to a threshold
v4:
  - Fix NULL ptr deref in drm_gpusvm_migrate_populate_ram_pfn (Thomas, Himal)
  - Fix check pages threshold
  - Check for range being unmapped under notifier lock in get pages (Testing)
  - Fix characters per line
  - Drop WRITE_ONCE for zdd->devmem_allocation assignment (Thomas)
  - Use completion for devmem_allocation->detached (Thomas)
  - Make GPU SVM depend on ZONE_DEVICE (CI)
  - Use hmm_range_fault for eviction (Thomas)
  - Drop zdd worker (Thomas)

---

<snip>

Quoted 109 lines

+/**
+ * __drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM (internal)
+ * @vas: Pointer to the VM area structure
+ * @device_private_page_owner: Device private pages owner
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @fault_addr: Fault address
+ * @size: Size of migration
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to RAM. It sets up the migration, populates + dma maps RAM PFNs, and
+ * invokes the driver-specific operations for migration to RAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +37 -0
+static int __drm_gpusvm_migrate_to_ram(struct vm_area_struct *vas,
+                       void *device_private_page_owner,
+                       struct page *page,
+                       unsigned long fault_addr,
+                       unsigned long size)
+{
+    struct migrate_vma migrate = {
+        .vma        = vas,
+        .pgmap_owner    = device_private_page_owner,
+        .flags        = MIGRATE_VMA_SELECT_DEVICE_PRIVATE |
+            MIGRATE_VMA_SELECT_DEVICE_COHERENT,
+        .fault_page    = page,
+    };
+    struct drm_gpusvm_zdd *zdd;
+    const struct drm_gpusvm_devmem_ops *ops;
+    struct device *dev;
+    unsigned long npages, mpages = 0;
+    struct page **pages;
+    dma_addr_t *dma_addr;
+    unsigned long start, end;
+    void *buf;
+    int i, err = 0;
+
+    start = ALIGN_DOWN(fault_addr, size);
+    end = ALIGN(fault_addr + 1, size);
+
+    /* Corner where VMA area struct has been partially unmapped */
+    if (start < vas->vm_start)
+        start = vas->vm_start;
+    if (end > vas->vm_end)
+        end = vas->vm_end;
+
+    migrate.start = start;
+    migrate.end = end;
+    npages = npages_in_range(start, end);
+
+    buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +

sizeof(*dma_addr) +
+               sizeof(*pages), GFP_KERNEL);
+    if (!buf) {
+        err = -ENOMEM;
+        goto err_out;
+    }
+    dma_addr = buf + (2 * sizeof(*migrate.src) * npages); +    pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;

drivers/gpu/drm/drm_gpusvm.c new file +30 -0
+
+    migrate.vma = vas;
+    migrate.src = buf;
+    migrate.dst = migrate.src + npages;
+
+    err = migrate_vma_setup(&migrate);
+    if (err)
+        goto err_free;
+
+    /* Raced with another CPU fault, nothing to do */
+    if (!migrate.cpages)
+        goto err_free;
+
+    if (!page) {
+        for (i = 0; i < npages; ++i) {
+            if (!(migrate.src[i] & MIGRATE_PFN_MIGRATE))
+                continue;
+
+            page = migrate_pfn_to_page(migrate.src[i]);
+            break;
+        }
+
+        if (!page)
+            goto err_finalize;
+    }
+    zdd = page->zone_device_data;
+    ops = zdd->devmem_allocation->ops;
+    dev = zdd->devmem_allocation->dev;
+
+    err = drm_gpusvm_migrate_populate_ram_pfn(vas, page, npages,

&mpages,
+                          migrate.src, migrate.dst,
+                          start);
+    if (err)
+        goto err_finalize;
+
+    err = drm_gpusvm_migrate_map_pages(dev, dma_addr, migrate.dst, npages,
+                       DMA_FROM_DEVICE);
+    if (err)
+        goto err_finalize;

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+    for (i = 0; i < npages; ++i)
+        pages[i] = migrate_pfn_to_page(migrate.src[i]);
+
+    err = ops->copy_to_ram(pages, dma_addr, npages);

drivers/gpu/drm/drm_gpusvm.c line 2048

Quoted 10 lines
drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+    if (err)
+        goto err_finalize;
+
+err_finalize:
+    if (err)
+        drm_gpusvm_migration_unlock_put_pages(npages, migrate.dst);
+    migrate_vma_pages(&migrate);
+    migrate_vma_finalize(&migrate);
+    drm_gpusvm_migrate_unmap_pages(dev, dma_addr, npages,
+                       DMA_FROM_DEVICE);

clang for me is throwing:

drivers/gpu/drm/drm_gpusvm.c:2017:7: error: variable 'dev' is used uninitialized whenever 'if' condition is true [-Werror,-Wsometimes- uninitialized]
 2017 |                 if (!page)
      |                     ^~~~~
drivers/gpu/drm/drm_gpusvm.c:2047:33: note: uninitialized use occurs here
 2047 |         drm_gpusvm_migrate_unmap_pages(dev, dma_addr, npages, |                                        ^~~
drivers/gpu/drm/drm_gpusvm.c:2017:3: note: remove the 'if' if its condition is always false
 2017 |                 if (!page)
      |                 ^~~~~~~~~~

Quoted 8 lines

 2018 |                         goto err_finalize;
      |                         ~~~~~~~~~~~~~~~~~
drivers/gpu/drm/drm_gpusvm.c:1966:20: note: initialize the variable 'dev' to silence this warning
 1966 |         struct device *dev;
      |                           ^
      |                            = NULL
1 error generated.

Quoted 5 lines

I also reported this issue in the v3 patch, but it doesn't seem to have been fixed in v4 yet.

https://lore.kernel.org/dri-devel/[email protected]/

Yea. The uninitialization is in fact harmless as drm_gpusvm_migrate_unmap_pages is a NOP in this path which I stated. I was unware a tool was complaining about this. I think get this fixed then.

Thanks,
Matt

Quoted 4 lines

Br,

G.G.

#20 lore raw

Re: [PATCH v4 06/33] drm/gpusvm: Add support for GPU Shared Virtual Memory

In reply to Matthew Brost
Quoted 3,192 lines · Matthew Brost

This patch introduces support for GPU Shared Virtual Memory (SVM) in the
Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.

The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also
provides mechanisms for allocating, deallocating, and migrating memory
regions between system RAM and GPU VRAM.

This is largely inspired by GPUVM.

v2:
 - Take order into account in check pages
 - Clear range->pages in get pages error
 - Drop setting dirty or accessed bit in get pages (Vetter)
 - Remove mmap assert for cpu faults
 - Drop mmap write lock abuse (Vetter, Christian)
 - Decouple zdd from range (Vetter, Oak)
 - Add drm_gpusvm_range_evict, make it work with coherent pages
 - Export drm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
 - mmget/put in drm_gpusvm_evict_to_sram
 - Drop range->vram_alloation variable
 - Don't return in drm_gpusvm_evict_to_sram until all pages detached
 - Don't warn on mixing sram and device pages
 - Update kernel doc
 - Add coherent page support to get pages
 - Use DMA_FROM_DEVICE rather than DMA_BIDIRECTIONAL
 - Add struct drm_gpusvm_vram and ops (Thomas)
 - Update the range's seqno if the range is valid (Thomas)
 - Remove the is_unmapped check before hmm_range_fault (Thomas)
 - Use drm_pagemap (Thomas)
 - Drop kfree_mapping (Thomas)
 - dma mapp pages under notifier lock (Thomas)
 - Remove ctx.prefault
 - Remove ctx.mmap_locked
 - Add ctx.check_pages
 - s/vram/devmem (Thomas)
v3:
 - Fix memory leak drm_gpusvm_range_get_pages
 - Only migrate pages with same zdd on CPU fault
 - Loop over al VMAs in drm_gpusvm_range_evict
 - Make GPUSVM a drm level module
 - GPL or MIT license
 - Update main kernel doc (Thomas)
 - Prefer foo() vs foo for functions in kernel doc (Thomas)
 - Prefer functions over macros (Thomas)
 - Use unsigned long vs u64 for addresses (Thomas)
 - Use standard interval_tree (Thomas)
 - s/drm_gpusvm_migration_put_page/drm_gpusvm_migration_unlock_put_page (Thomas)
 - Drop err_out label in drm_gpusvm_range_find_or_insert (Thomas)
 - Fix kernel doc in drm_gpusvm_range_free_pages (Thomas)
 - Newlines between functions defs in header file (Thomas)
 - Drop shall language in driver vfunc kernel doc (Thomas)
 - Move some static inlines from head to C file (Thomas)
 - Don't allocate pages under page lock in drm_gpusvm_migrate_populate_ram_pfn (Thomas)
 - Change check_pages to a threshold
v4:
 - Fix NULL ptr deref in drm_gpusvm_migrate_populate_ram_pfn (Thomas, Himal)
 - Fix check pages threshold
 - Check for range being unmapped under notifier lock in get pages (Testing)
 - Fix characters per line
 - Drop WRITE_ONCE for zdd->devmem_allocation assignment (Thomas)
 - Use completion for devmem_allocation->detached (Thomas)
 - Make GPU SVM depend on ZONE_DEVICE (CI)
 - Use hmm_range_fault for eviction (Thomas)
 - Drop zdd worker (Thomas)

---
 drivers/gpu/drm/Kconfig      |    9 +
 drivers/gpu/drm/Makefile     |    1 +
 drivers/gpu/drm/drm_gpusvm.c | 2240
++++++++++++++++++++++++++++++++++
 include/drm/drm_gpusvm.h     |  445 +++++++

 4 files changed, 2695 insertions(+)

 create mode 100644 drivers/gpu/drm/drm_gpusvm.c
 create mode 100644 include/drm/drm_gpusvm.h

drivers/gpu/drm/Kconfig +0 -0
index fbef3f471bd0..f03862e379fb 100644
@@ -278,6 +278,15 @@ config DRM_GPUVM

    GPU-VM representation providing helpers to manage a GPUs virtual
    address space

drivers/gpu/drm/Kconfig +7 -0
+config DRM_GPUSVM
+	tristate
+	depends on DRM
+	depends on DEVICE_MIGRATION
+	depends on ZONE_DEVICE
+	help
+	  GPU-SVM representation providing helpers to manage a GPUs

shared

drivers/gpu/drm/Kconfig +2 -0
+	  virtual memory
+

 config DRM_BUDDY
  tristate
  depends on DRM

drivers/gpu/drm/Makefile +0 -0
index 85af94bb907d..ca03df8d2729 100644
@@ -104,6 +104,7 @@ obj-$(CONFIG_DRM_PANEL_BACKLIGHT_QUIRKS) +=

drm_panel_backlight_quirks.o
 #
 obj-$(CONFIG_DRM_EXEC) += drm_exec.o
 obj-$(CONFIG_DRM_GPUVM) += drm_gpuvm.o
+obj-$(CONFIG_DRM_GPUSVM) += drm_gpusvm.o

 obj-$(CONFIG_DRM_BUDDY) += drm_buddy.o

drivers/gpu/drm/drm_gpusvm.c new file +25 -0
new file mode 100644
index 000000000000..1c63da4d3cc2
@@ -0,0 +1,2240 @@
+// SPDX-License-Identifier: GPL-2.0-only OR MIT
+/*
+ * Copyright © 2024 Intel Corporation
+ *
+ * Authors:
+ *     Matthew Brost <[email protected]>
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/hmm.h>
+#include <linux/memremap.h>
+#include <linux/migrate.h>
+#include <linux/mm_types.h>
+#include <linux/pagemap.h>
+#include <linux/slab.h>
+
+#include <drm/drm_device.h>
+#include <drm/drm_gpusvm.h>
+#include <drm/drm_pagemap.h>
+#include <drm/drm_print.h>
+
+/**
+ * DOC: Overview
+ *
+ * GPU Shared Virtual Memory (GPU SVM) layer for the Direct

Rendering Manager (DRM)
+ *
+ * The GPU SVM layer is a component of the DRM framework designed to manage shared
+ * virtual memory between the CPU and GPU. It enables efficient data exchange and
+ * processing for GPU-accelerated applications by allowing memory sharing and
+ * synchronization between the CPU's and GPU's virtual address spaces.
+ *
+ * Key GPU SVM Components:
+ * - Notifiers: Notifiers: Used for tracking memory intervals and notifying the
+ * GPU of changes, notifiers are sized based on a GPU
SVM
+ * initialization parameter, with a recommendation of
512M or
+ * larger. They maintain a Red-BlacK tree and a list of
ranges that
+ * fall within the notifier interval. Notifiers are
tracked within
+ * a GPU SVM Red-BlacK tree and list and are
dynamically inserted
+ * or removed as ranges within the interval are created
or
+ * destroyed.
+ * - Ranges: Represent memory ranges mapped in a DRM device and managed
+ *      by GPU SVM. They are sized based on an array of chunk
sizes, which
+ *      is a GPU SVM initialization parameter, and the CPU
address space.
+ *      Upon GPU fault, the largest aligned chunk that fits
within the
+ *      faulting CPU address space is chosen for the range
size. Ranges are
+ *      expected to be dynamically allocated on GPU fault and
removed on an
+ *      MMU notifier UNMAP event. As mentioned above, ranges
are tracked in
+ *      a notifier's Red-Black tree.
+ * - Operations: Define the interface for driver-specific GPU SVM operations
+ *               such as range allocation, notifier allocation, and
+ *               invalidations.
+ * - Device Memory Allocations: Embedded structure containing enough information
+ *                              for GPU SVM to migrate to / from device memory.
+ * - Device Memory Operations: Define the interface for driver- specific device
+ *                             memory operations release memory, populate pfns,
+ *                             and copy to / from device memory.
+ *
+ * This layer provides interfaces for allocating, mapping, migrating, and
+ * releasing memory ranges between the CPU and GPU. It handles all core memory
+ * management interactions (DMA mapping, HMM, and migration) and provides
+ * driver-specific virtual functions (vfuncs). This infrastructure is sufficient
+ * to build the expected driver components for an SVM implementation as detailed
+ * below.
+ *
+ * Expected Driver Components:
+ * - GPU page fault handler: Used to create ranges and notifiers based on the
+ *      fault address, optionally migrate the
range to
+ *      device memory, and create GPU bindings.
+ * - Garbage collector: Used to unmap and destroy GPU bindings for ranges.
+ * Ranges are expected to be added to the
garbage collector
+ * upon a MMU_NOTIFY_UNMAP event in notifier
callback.
+ * - Notifier callback: Used to invalidate and DMA unmap GPU bindings for
+ * ranges.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Locking
+ *
+ * GPU SVM handles locking for core MM interactions, i.e., it

locks/unlocks the
+ * mmap lock as needed.
+ *
+ * GPU SVM introduces a global notifier lock, which safeguards the notifier's
+ * range RB tree and list, as well as the range's DMA mappings and sequence
+ * number. GPU SVM manages all necessary locking and unlocking operations,
+ * except for the recheck range's pages being valid
+ * (drm_gpusvm_range_pages_valid) when the driver is committing GPU bindings. This
+ * lock corresponds to the 'driver->update' lock mentioned in the HMM
+ * documentation (TODO: Link). Future revisions may transition from a GPU SVM
+ * global lock to a per-notifier lock if finer-grained locking is deemed
+ * necessary.
+ *
+ * In addition to the locking mentioned above, the driver should implement a
+ * lock to safeguard core GPU SVM function calls that modify state, such as
+ * drm_gpusvm_range_find_or_insert and drm_gpusvm_range_remove. This lock is
+ * denoted as 'driver_svm_lock' in code examples. Finer grained driver side
+ * locking should also be possible for concurrent GPU fault processing within a
+ * single GPU SVM. The 'driver_svm_lock' can be via drm_gpusvm_driver_set_lock
+ * to add annotations to GPU SVM.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Migration
+ *
+ * The migration support is quite simple, allowing migration between

RAM and
+ * device memory at the range granularity. For example, GPU SVM currently does not
+ * support mixing RAM and device memory pages within a range. This means that upon GPU
+ * fault, the entire range can be migrated to device memory, and upon CPU fault, the
+ * entire range is migrated to RAM. Mixed RAM and device memory storage within a range
+ * could be added in the future if required.
+ *
+ * The reasoning for only supporting range granularity is as
follows: it
+ * simplifies the implementation, and range sizes are driver-defined and should
+ * be relatively small.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Partial Unmapping of Ranges
+ *
+ * Partial unmapping of ranges (e.g., 1M out of 2M is unmapped by

CPU resulting
+ * in MMU_NOTIFY_UNMAP event) presents several challenges, with the main one
+ * being that a subset of the range still has CPU and GPU mappings. If the
+ * backing store for the range is in device memory, a subset of the backing store has
+ * references. One option would be to split the range and device memory backing store,
+ * but the implementation for this would be quite complicated. Given that
+ * partial unmappings are rare and driver-defined range sizes are relatively
+ * small, GPU SVM does not support splitting of ranges.
+ *
+ * With no support for range splitting, upon partial unmapping of a range, the
+ * driver is expected to invalidate and destroy the entire range. If the range
+ * has device memory as its backing, the driver is also expected to migrate any
+ * remaining pages back to RAM.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Examples
+ *
+ * This section provides three examples of how to build the expected

driver
+ * components: the GPU page fault handler, the garbage collector, and the
+ * notifier callback.
+ *
+ * The generic code provided does not include logic for complex migration
+ * policies, optimized invalidations, fined grained driver locking, or other
+ * potentially required driver locking (e.g., DMA-resv locks).
+ *
+ * 1) GPU page fault handler
+ *
+ * int driver_bind_range(struct drm_gpusvm *gpusvm, struct
drm_gpusvm_range *range)
+ * {
+ * int err = 0;
+ *
+ * driver_alloc_and_setup_memory_for_bind(gpusvm,
range);
+ *
+ * drm_gpusvm_notifier_lock(gpusvm);
+ * if (drm_gpusvm_range_pages_valid(range))
+ * driver_commit_bind(gpusvm, range);
+ * else
+ * err = -EAGAIN;
+ * drm_gpusvm_notifier_unlock(gpusvm);
+ *
+ * return err;
+ * }
+ *
+ * int driver_gpu_fault(struct drm_gpusvm *gpusvm, unsigned
long fault_addr,
+ *      unsigned long gpuva_start, unsigned
long gpuva_end)
+ * {
+ * struct drm_gpusvm_ctx ctx = {};
+ * int err;
+ *
+ * driver_svm_lock();
+ * retry:
+ * // Always process UNMAPs first so view of GPU SVM
ranges is current
+ * driver_garbage_collector(gpusvm);
+ *
+ * range = drm_gpusvm_range_find_or_insert(gpusvm,
fault_addr,
+ * gpuva_start,
gpuva_end,
+ *         &ctx);
+ * if (IS_ERR(range)) {
+ * err = PTR_ERR(range);
+ * goto unlock;
+ * }
+ *
+ * if (driver_migration_policy(range)) {
+ * devmem = driver_alloc_devmem();
+ * err = drm_gpusvm_migrate_to_devmem(gpusvm,
range,
+ *
devmem_allocation,
+ *    &ctx);
+ * if (err) // CPU mappings may have
changed
+ * goto retry;
+ * }
+ *
+ * err = drm_gpusvm_range_get_pages(gpusvm, range,
&ctx);
+ * if (err == -EOPNOTSUPP || err == -EFAULT || err == -
EPERM) { // CPU mappings changed
+ * if (err == -EOPNOTSUPP)
+ * drm_gpusvm_range_evict(gpusvm,
range);
+ * goto retry;
+ * } else if (err) {
+ * goto unlock;
+ * }
+ *
+ * err = driver_bind_range(gpusvm, range);
+ * if (err == -EAGAIN) // CPU mappings changed
+ * goto retry
+ *
+ * unlock:
+ * driver_svm_unlock();
+ * return err;
+ * }
+ *
+ * 2) Garbage Collector.
+ *
+ * void __driver_garbage_collector(struct drm_gpusvm *gpusvm,
+ * struct drm_gpusvm_range
*range)
+ * {
+ * assert_driver_svm_locked(gpusvm);
+ *
+ * // Partial unmap, migrate any remaining device
memory pages back to RAM
+ * if (range->flags.partial_unmap)
+ * drm_gpusvm_range_evict(gpusvm, range);
+ *
+ * driver_unbind_range(range);
+ * drm_gpusvm_range_remove(gpusvm, range);
+ * }
+ *
+ * void driver_garbage_collector(struct drm_gpusvm *gpusvm)
+ * {
+ * assert_driver_svm_locked(gpusvm);
+ *
+ * for_each_range_in_garbage_collector(gpusvm, range)
+ * __driver_garbage_collector(gpusvm, range);
+ * }
+ *
+ * 3) Notifier callback.
+ *
+ * void driver_invalidation(struct drm_gpusvm *gpusvm,
+ * struct drm_gpusvm_notifier
*notifier,
+ * const struct mmu_notifier_range
*mmu_range)
+ * {
+ * struct drm_gpusvm_ctx ctx = { .in_notifier = true,
};
+ * struct drm_gpusvm_range *range = NULL;
+ *
+ * driver_invalidate_device_pages(gpusvm, mmu_range-

Quoted 1 line

start, mmu_range->end);

+ *
+ * drm_gpusvm_for_each_range(range, notifier,
mmu_range->start,
+ *   mmu_range->end) {
+ * drm_gpusvm_range_unmap_pages(gpusvm, range,
&ctx);
+ *
+ * if (mmu_range->event != MMU_NOTIFY_UNMAP)
+ * continue;
+ *
+ * drm_gpusvm_range_set_unmapped(range,
mmu_range);
+ * driver_garbage_collector_add(gpusvm, range);
+ * }
+ * }
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+
+/**
+ * npages_in_range() - Calculate the number of pages in a given

range
+ * @start: The start address of the range
+ * @end: The end address of the range
+ *
+ * This macro calculates the number of pages in a given memory range,
+ * specified by the start and end addresses. It divides the difference
+ * between the end and start addresses by the page size (PAGE_SIZE) to
+ * determine the number of pages in the range.
+ *
+ * Returns: The number of pages in the specified range.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+static unsigned long
+npages_in_range(unsigned long start, unsigned long end)
+{
+	return (end - start) >> PAGE_SHIFT;
+}
+
+/**
+ * struct drm_gpusvm_zdd - GPU SVM zone device data
+ *
+ * @refcount: Reference count for the zdd
+ * @devmem_allocation: device memory allocation
+ * @device_private_page_owner: Device private pages owner
+ *
+ * This structure serves as a generic wrapper installed in
+ * page->zone_device_data. It provides infrastructure for looking up

a device
+ * memory allocation upon CPU page fault and asynchronously releasing device
+ * memory once the CPU has no page references. Asynchronous release is useful
+ * because CPU page references can be dropped in IRQ contexts, while releasing
+ * device memory likely requires sleeping locks.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+struct drm_gpusvm_zdd {
+	struct kref refcount;
+	struct drm_gpusvm_devmem *devmem_allocation;
+	void *device_private_page_owner;
+};
+
+/**
+ * drm_gpusvm_zdd_alloc() - Allocate a zdd structure.
+ * @device_private_page_owner: Device private pages owner
+ *
+ * This function allocates and initializes a new zdd structure. It

sets up the
+ * reference count and initializes the destroy work.
+ *
+ * Returns:
+ * Pointer to the allocated zdd on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +21 -0
+static struct drm_gpusvm_zdd *
+drm_gpusvm_zdd_alloc(void *device_private_page_owner)
+{
+	struct drm_gpusvm_zdd *zdd;
+
+	zdd = kmalloc(sizeof(*zdd), GFP_KERNEL);
+	if (!zdd)
+		return NULL;
+
+	kref_init(&zdd->refcount);
+	zdd->devmem_allocation = NULL;
+	zdd->device_private_page_owner = device_private_page_owner;
+
+	return zdd;
+}
+
+/**
+ * drm_gpusvm_zdd_get() - Get a reference to a zdd structure.
+ * @zdd: Pointer to the zdd structure.
+ *
+ * This function increments the reference count of the provided zdd

structure.
+ *
+ * Returns: Pointer to the zdd structure.
+ */
+static struct drm_gpusvm_zdd *drm_gpusvm_zdd_get(struct drm_gpusvm_zdd *zdd)

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+{
+	kref_get(&zdd->refcount);
+	return zdd;
+}
+
+/**
+ * drm_gpusvm_zdd_destroy() - Destroy a zdd structure.
+ * @ref: Pointer to the reference count structure.
+ *
+ * This function queues the destroy_work of the zdd for asynchronous

destruction.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +19 -0
+static void drm_gpusvm_zdd_destroy(struct kref *ref)
+{
+	struct drm_gpusvm_zdd *zdd =
+		container_of(ref, struct drm_gpusvm_zdd, refcount);
+	struct drm_gpusvm_devmem *devmem = zdd->devmem_allocation;
+
+	if (devmem) {
+		complete_all(&devmem->detached);
+		if (devmem->ops->devmem_release)
+			devmem->ops->devmem_release(devmem);
+	}
+	kfree(zdd);
+}
+
+/**
+ * drm_gpusvm_zdd_put() - Put a zdd reference.
+ * @zdd: Pointer to the zdd structure.
+ *
+ * This function decrements the reference count of the provided zdd

structure
+ * and schedules its destruction if the count drops to zero.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+static void drm_gpusvm_zdd_put(struct drm_gpusvm_zdd *zdd)
+{
+	kref_put(&zdd->refcount, drm_gpusvm_zdd_destroy);
+}
+
+/**
+ * drm_gpusvm_range_find() - Find GPU SVM range from GPU SVM

notifier
+ * @notifier: Pointer to the GPU SVM notifier structure.
+ * @start: Start address of the range
+ * @end: End address of the range
+ *
+ * Returns: A pointer to the drm_gpusvm_range if found or NULL
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, unsigned

long start,

drivers/gpu/drm/drm_gpusvm.c new file +8 -1
+		      unsigned long end)
+{
+	struct interval_tree_node *itree;
+
+	itree = interval_tree_iter_first(&notifier->root, start, end
- 1);
+
+	if (itree)
+		return container_of(itree, struct drm_gpusvm_range,

itree);

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+	else
+		return NULL;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_find);
+
+/**
+ * drm_gpusvm_for_each_range_safe() - Safely iterate over GPU SVM

ranges in a notifier
+ * @range__: Iterator variable for the ranges
+ * @next__: Iterator variable for the ranges temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier while
+ * removing ranges from it.
+ */
+#define drm_gpusvm_for_each_range_safe(range__, next__, notifier__,
start__, end__) \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	for ((range__) = drm_gpusvm_range_find((notifier__),

(start__), (end__)), \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (next__) =

__drm_gpusvm_range_next(range__); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (range__) && (range__->itree.start <

(end__)); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (range__) = (next__), (next__) =

__drm_gpusvm_range_next(range__))

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+
+/**
+ * __drm_gpusvm_notifier_next() - get the next drm_gpusvm_notifier

in the list
+ * @notifier: a pointer to the current drm_gpusvm_notifier
+ *
+ * Returns: A pointer to the next drm_gpusvm_notifier if available, or NULL if
+ *         the current notifier is the last one or if the input notifier is
+ *         NULL.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+static struct drm_gpusvm_notifier *
+__drm_gpusvm_notifier_next(struct drm_gpusvm_notifier *notifier)
+{
+	if (notifier && !list_is_last(&notifier->entry,
+				      &notifier->gpusvm-
Quoted 1 line

notifier_list))

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+		return list_next_entry(notifier, entry);
+
+	return NULL;
+}
+
+static struct drm_gpusvm_notifier *
+notifier_iter_first(struct rb_root_cached *root, unsigned long

start,

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+		    unsigned long last)
+{
+	struct interval_tree_node *itree;
+
+	itree = interval_tree_iter_first(root, start, last);
+
+	if (itree)
+		return container_of(itree, struct

drm_gpusvm_notifier, itree);

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+	else
+		return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_notifier() - Iterate over GPU SVM notifiers

in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm.
+ */
+#define drm_gpusvm_for_each_notifier(notifier__, gpusvm__, start__,
end__) \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,

(start__), (end__) - 1); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (notifier__) && (notifier__->itree.start <

(end__)); \

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+	     (notifier__) = __drm_gpusvm_notifier_next(notifier__))
+
+/**
+ * drm_gpusvm_for_each_notifier_safe() - Safely iterate over GPU SVM

notifiers in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @next__: Iterator variable for the notifiers temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm while
+ * removing notifiers from it.
+ */
+#define drm_gpusvm_for_each_notifier_safe(notifier__, next__,
gpusvm__, start__, end__) \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,

(start__), (end__) - 1), \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (next__) =

__drm_gpusvm_notifier_next(notifier__); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (notifier__) && (notifier__->itree.start <

(end__)); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (notifier__) = (next__), (next__) =

__drm_gpusvm_notifier_next(notifier__))

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+
+/**
+ * drm_gpusvm_notifier_invalidate() - Invalidate a GPU SVM notifier.
+ * @mni: Pointer to the mmu_interval_notifier structure.
+ * @mmu_range: Pointer to the mmu_notifier_range structure.
+ * @cur_seq: Current sequence number.
+ *
+ * This function serves as a generic MMU notifier for GPU SVM. It

sets the MMU
+ * notifier sequence number and calls the driver invalidate vfunc under
+ * gpusvm->notifier_lock.
+ *
+ * Returns:
+ * true if the operation succeeds, false otherwise.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+static bool
+drm_gpusvm_notifier_invalidate(struct mmu_interval_notifier *mni,
+			       const struct mmu_notifier_range

*mmu_range,

drivers/gpu/drm/drm_gpusvm.c new file +19 -0
+			       unsigned long cur_seq)
+{
+	struct drm_gpusvm_notifier *notifier =
+		container_of(mni, typeof(*notifier), notifier);
+	struct drm_gpusvm *gpusvm = notifier->gpusvm;
+
+	if (!mmu_notifier_range_blockable(mmu_range))
+		return false;
+
+	down_write(&gpusvm->notifier_lock);
+	mmu_interval_set_seq(mni, cur_seq);
+	gpusvm->ops->invalidate(gpusvm, notifier, mmu_range);
+	up_write(&gpusvm->notifier_lock);
+
+	return true;
+}
+
+/**
+ * drm_gpusvm_notifier_ops - MMU interval notifier operations for

GPU SVM
+ */
+static const struct mmu_interval_notifier_ops
drm_gpusvm_notifier_ops = {

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+	.invalidate = drm_gpusvm_notifier_invalidate,
+};
+
+/**
+ * drm_gpusvm_init() - Initialize the GPU SVM.
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @name: Name of the GPU SVM.
+ * @drm: Pointer to the DRM device structure.
+ * @mm: Pointer to the mm_struct for the address space.
+ * @device_private_page_owner: Device private pages owner.
+ * @mm_start: Start address of GPU SVM.
+ * @mm_range: Range of the GPU SVM.
+ * @notifier_size: Size of individual notifiers.
+ * @ops: Pointer to the operations structure for GPU SVM.
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range

allocation.
+ *               Entries should be powers of 2 in descending order with last
+ *               entry being SZ_4K.
+ * @num_chunks: Number of chunks.
+ *
+ * This function initializes the GPU SVM.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void

*device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+		    unsigned long mm_start, unsigned long mm_range,
+		    unsigned long notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const unsigned long *chunk_sizes, int

num_chunks)

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+{
+	if (!ops->invalidate || !num_chunks)
+		return -EINVAL;
+
+	gpusvm->name = name;
+	gpusvm->drm = drm;
+	gpusvm->mm = mm;
+	gpusvm->device_private_page_owner =

device_private_page_owner;

drivers/gpu/drm/drm_gpusvm.c new file +31 -0
+	gpusvm->mm_start = mm_start;
+	gpusvm->mm_range = mm_range;
+	gpusvm->notifier_size = notifier_size;
+	gpusvm->ops = ops;
+	gpusvm->chunk_sizes = chunk_sizes;
+	gpusvm->num_chunks = num_chunks;
+
+	mmgrab(mm);
+	gpusvm->root = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&gpusvm->notifier_list);
+
+	init_rwsem(&gpusvm->notifier_lock);
+
+	fs_reclaim_acquire(GFP_KERNEL);
+	might_lock(&gpusvm->notifier_lock);
+	fs_reclaim_release(GFP_KERNEL);
+
+#ifdef CONFIG_LOCKDEP
+	gpusvm->lock_dep_map = NULL;
+#endif
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_init);
+
+/**
+ * drm_gpusvm_notifier_find() - Find GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ *
+ * This function finds the GPU SVM notifier associated with the

fault address.
+ *
+ * Returns:
+ * Pointer to the GPU SVM notifier on success, NULL otherwise.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+static struct drm_gpusvm_notifier *
+drm_gpusvm_notifier_find(struct drm_gpusvm *gpusvm,
+			 unsigned long fault_addr)
+{
+	return notifier_iter_first(&gpusvm->root, fault_addr,

fault_addr + 1);

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+}
+
+/**
+ * to_drm_gpusvm_notifier() - retrieve the container struct for a

given rbtree node
+ * @node: a pointer to the rbtree node embedded within a drm_gpusvm_notifier struct
+ *
+ * Returns: A pointer to the containing drm_gpusvm_notifier structure.
+ */
+static struct drm_gpusvm_notifier *to_drm_gpusvm_notifier(struct rb_node *node)

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+{
+	return container_of(node, struct drm_gpusvm_notifier,

itree.rb);

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+}
+
+/**
+ * drm_gpusvm_notifier_insert() - Insert GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function inserts the GPU SVM notifier into the GPU SVM RB

tree and list.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static void drm_gpusvm_notifier_insert(struct drm_gpusvm *gpusvm,
+				       struct drm_gpusvm_notifier

*notifier)

drivers/gpu/drm/drm_gpusvm.c new file +21 -0
+{
+	struct rb_node *node;
+	struct list_head *head;
+
+	interval_tree_insert(&notifier->itree, &gpusvm->root);
+
+	node = rb_prev(&notifier->itree.rb);
+	if (node)
+		head = &(to_drm_gpusvm_notifier(node))->entry;
+	else
+		head = &gpusvm->notifier_list;
+
+	list_add(&notifier->entry, head);
+}
+
+/**
+ * drm_gpusvm_notifier_remove() - Remove GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM tructure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function removes the GPU SVM notifier from the GPU SVM RB

tree and list.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static void drm_gpusvm_notifier_remove(struct drm_gpusvm *gpusvm,
+				       struct drm_gpusvm_notifier

*notifier)

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+{
+	interval_tree_remove(&notifier->itree, &gpusvm->root);
+	list_del(&notifier->entry);
+}
+
+/**
+ * drm_gpusvm_fini() - Finalize the GPU SVM.
+ * @gpusvm: Pointer to the GPU SVM structure.
+ *
+ * This function finalizes the GPU SVM by cleaning up any remaining

ranges and
+ * notifiers, and dropping a reference to struct MM.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm)
+{
+	struct drm_gpusvm_notifier *notifier, *next;
+
+	drm_gpusvm_for_each_notifier_safe(notifier, next, gpusvm, 0,

LONG_MAX) {

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+		struct drm_gpusvm_range *range, *__next;
+
+		/*
+		 * Remove notifier first to avoid racing with any

invalidation

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+		 */
+		mmu_interval_notifier_remove(&notifier->notifier);
+		notifier->flags.removed = true;
+
+		drm_gpusvm_for_each_range_safe(range, __next,

notifier, 0,

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+					       LONG_MAX)
+			drm_gpusvm_range_remove(gpusvm, range);
+	}
+
+	mmdrop(gpusvm->mm);
+	WARN_ON(!RB_EMPTY_ROOT(&gpusvm->root.rb_root));
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_fini);
+
+/**
+ * drm_gpusvm_notifier_alloc() - Allocate GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ *
+ * This function allocates and initializes the GPU SVM notifier

structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM notifier on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static struct drm_gpusvm_notifier *
+drm_gpusvm_notifier_alloc(struct drm_gpusvm *gpusvm, unsigned long

fault_addr)

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	if (gpusvm->ops->notifier_alloc)
+		notifier = gpusvm->ops->notifier_alloc();
+	else
+		notifier = kzalloc(sizeof(*notifier), GFP_KERNEL);
+
+	if (!notifier)
+		return ERR_PTR(-ENOMEM);
+
+	notifier->gpusvm = gpusvm;
+	notifier->itree.start = ALIGN_DOWN(fault_addr, gpusvm-
Quoted 1 line

notifier_size);

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	notifier->itree.last = ALIGN(fault_addr + 1, gpusvm-
Quoted 1 line

notifier_size) - 1;

drivers/gpu/drm/drm_gpusvm.c new file +16 -0
+	INIT_LIST_HEAD(&notifier->entry);
+	notifier->root = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&notifier->range_list);
+
+	return notifier;
+}
+
+/**
+ * drm_gpusvm_notifier_free() - Free GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function frees the GPU SVM notifier structure.
+ */
+static void drm_gpusvm_notifier_free(struct drm_gpusvm *gpusvm,
+				     struct drm_gpusvm_notifier

*notifier)

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+{
+	WARN_ON(!RB_EMPTY_ROOT(&notifier->root.rb_root));
+
+	if (gpusvm->ops->notifier_free)
+		gpusvm->ops->notifier_free(notifier);
+	else
+		kfree(notifier);
+}
+
+/**
+ * to_drm_gpusvm_range() - retrieve the container struct for a given

rbtree node
+ * @node: a pointer to the rbtree node embedded within a drm_gpusvm_range struct
+ *
+ * Returns: A pointer to the containing drm_gpusvm_range structure.
+ */
+static struct drm_gpusvm_range *to_drm_gpusvm_range(struct rb_node *node)

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+{
+	return container_of(node, struct drm_gpusvm_range,

itree.rb);

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+}
+
+/**
+ * drm_gpusvm_range_insert() - Insert GPU SVM range
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function inserts the GPU SVM range into the notifier RB tree

and list.
+ */
+static void drm_gpusvm_range_insert(struct drm_gpusvm_notifier *notifier,

drivers/gpu/drm/drm_gpusvm.c new file +24 -0
+				    struct drm_gpusvm_range *range)
+{
+	struct rb_node *node;
+	struct list_head *head;
+
+	drm_gpusvm_notifier_lock(notifier->gpusvm);
+	interval_tree_insert(&range->itree, &notifier->root);
+
+	node = rb_prev(&range->itree.rb);
+	if (node)
+		head = &(to_drm_gpusvm_range(node))->entry;
+	else
+		head = &notifier->range_list;
+
+	list_add(&range->entry, head);
+	drm_gpusvm_notifier_unlock(notifier->gpusvm);
+}
+
+/**
+ * __drm_gpusvm_range_remove() - Remove GPU SVM range
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This macro removes the GPU SVM range from the notifier RB tree

and list.
+ */
+static void __drm_gpusvm_range_remove(struct drm_gpusvm_notifier *notifier,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				      struct drm_gpusvm_range

*range)

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+{
+	interval_tree_remove(&range->itree, &notifier->root);
+	list_del(&range->entry);
+}
+
+/**
+ * drm_gpusvm_range_alloc() - Allocate GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @fault_addr: Fault address
+ * @chunk_size: Chunk size
+ * @migrate_devmem: Flag indicating whether to migrate device memory
+ *
+ * This function allocates and initializes the GPU SVM range

structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM range on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+static struct drm_gpusvm_range *
+drm_gpusvm_range_alloc(struct drm_gpusvm *gpusvm,
+		       struct drm_gpusvm_notifier *notifier,
+		       unsigned long fault_addr, unsigned long

chunk_size,

drivers/gpu/drm/drm_gpusvm.c new file +32 -0
+		       bool migrate_devmem)
+{
+	struct drm_gpusvm_range *range;
+
+	if (gpusvm->ops->range_alloc)
+		range = gpusvm->ops->range_alloc(gpusvm);
+	else
+		range = kzalloc(sizeof(*range), GFP_KERNEL);
+
+	if (!range)
+		return ERR_PTR(-ENOMEM);
+
+	kref_init(&range->refcount);
+	range->gpusvm = gpusvm;
+	range->notifier = notifier;
+	range->itree.start = ALIGN_DOWN(fault_addr, chunk_size);
+	range->itree.last = ALIGN(fault_addr + 1, chunk_size) - 1;
+	INIT_LIST_HEAD(&range->entry);
+	range->notifier_seq = LONG_MAX;
+	range->flags.migrate_devmem = migrate_devmem ? 1 : 0;
+
+	return range;
+}
+
+/**
+ * drm_gpusvm_check_pages() - Check pages
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @start: Start address
+ * @end: End address
+ *
+ * Check if pages between start and end have been faulted in on the

CPU. Use to
+ * prevent migration of pages without CPU backing store.
+ *
+ * Returns:
+ * True if pages have been faulted into CPU, False otherwise
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static bool drm_gpusvm_check_pages(struct drm_gpusvm *gpusvm,
+				   struct drm_gpusvm_notifier

*notifier,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				   unsigned long start, unsigned

long end)

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+{
+	struct hmm_range hmm_range = {
+		.default_flags = 0,
+		.notifier = &notifier->notifier,
+		.start = start,
+		.end = end,
+		.dev_private_owner = gpusvm-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+	};
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+	unsigned long *pfns;
+	unsigned long npages = npages_in_range(start, end);
+	int err, i;
+
+	mmap_assert_locked(gpusvm->mm);
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL);
+	if (!pfns)
+		return false;
+
+	hmm_range.notifier_seq = mmu_interval_read_begin(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+	hmm_range.hmm_pfns = pfns;
+
+	while (true) {
+		err = hmm_range_fault(&hmm_range);
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq =
+				mmu_interval_read_begin(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/drm_gpusvm.c new file +22 -0
+			continue;
+		}
+		break;
+	}
+	if (err)
+		goto err_free;
+
+	for (i = 0; i < npages;) {
+		if (!(pfns[i] & HMM_PFN_VALID)) {
+			err = -EFAULT;
+			goto err_free;
+		}
+		i += 0x1 << hmm_pfn_to_map_order(pfns[i]);
+	}
+
+err_free:
+	kvfree(pfns);
+	return err ? false : true;
+}
+
+/**
+ * drm_gpusvm_range_chunk_size() - Determine chunk size for GPU SVM

range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @vas: Pointer to the virtual memory area structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @check_pages_threshold: Check CPU pages for present threshold
+ *
+ * This function determines the chunk size for the GPU SVM range based on the
+ * fault address, GPU SVM chunk sizes, existing GPU SVM ranges, and the virtual
+ * memory area boundaries.
+ *
+ * Returns:
+ * Chunk size on success, LONG_MAX on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+static unsigned long
+drm_gpusvm_range_chunk_size(struct drm_gpusvm *gpusvm,
+			    struct drm_gpusvm_notifier *notifier,
+			    struct vm_area_struct *vas,
+			    unsigned long fault_addr,
+			    unsigned long gpuva_start,
+			    unsigned long gpuva_end,
+			    unsigned long check_pages_threshold)
+{
+	unsigned long start, end;
+	int i = 0;
+
+retry:
+	for (; i < gpusvm->num_chunks; ++i) {
+		start = ALIGN_DOWN(fault_addr, gpusvm-
Quoted 1 line

chunk_sizes[i]);

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+		end = ALIGN(fault_addr + 1, gpusvm->chunk_sizes[i]);
+
+		if (start >= vas->vm_start && end <= vas->vm_end &&
+		    start >= notifier->itree.start &&
+		    end <= notifier->itree.last + 1 &&
+		    start >= gpuva_start && end <= gpuva_end)
+			break;
+	}
+
+	if (i == gpusvm->num_chunks)
+		return LONG_MAX;
+
+	/*
+	 * If allocation more than page, ensure not to overlap with

existing

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+	 * ranges.
+	 */
+	if (end - start != SZ_4K) {
+		struct drm_gpusvm_range *range;
+
+		range = drm_gpusvm_range_find(notifier, start, end);
+		if (range) {
+			++i;
+			goto retry;
+		}
+
+		/*
+		 * XXX: Only create range on pages CPU has faulted

in. Without

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * this check, or prefault, on BMG

'xe_exec_system_allocator --r

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * process-many-malloc' fails. In the failure case,

each process

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * mallocs 16k but the CPU VMA is ~128k which

results in 64k SVM

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * ranges. When migrating the SVM ranges, some

processes fail in

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * drm_gpusvm_migrate_to_devmem with 'migrate.cpages

!= npages'

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * and then upon drm_gpusvm_range_get_pages device

pages from

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * other processes are collected + faulted in which

creates all

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * sorts of problems. Unsure exactly how this

happening, also

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * problem goes away if 'xe_exec_system_allocator --

r

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * process-many-malloc' mallocs at least 64k at a

time.

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+		 */
+		if (end - start <= check_pages_threshold &&
+		    !drm_gpusvm_check_pages(gpusvm, notifier, start,

end)) {

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+			++i;
+			goto retry;
+		}
+	}
+
+	return end - start;
+}
+
+/**
+ * drm_gpusvm_range_find_or_insert() - Find or insert GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @ctx: GPU SVM context
+ *
+ * This function finds or inserts a newly allocated a GPU SVM range

based on the
+ * fault address. Caller must hold a lock to protect range lookup and insertion.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +28 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm,
+				unsigned long fault_addr,
+				unsigned long gpuva_start,
+				unsigned long gpuva_end,
+				const struct drm_gpusvm_ctx *ctx)
+{
+	struct drm_gpusvm_notifier *notifier;
+	struct drm_gpusvm_range *range;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	bool notifier_alloc = false;
+	unsigned long chunk_size;
+	int err;
+	bool migrate_devmem;
+
+	drm_gpusvm_driver_lock_held(gpusvm);
+
+	if (fault_addr < gpusvm->mm_start ||
+	    fault_addr > gpusvm->mm_start + gpusvm->mm_range)
+		return ERR_PTR(-EINVAL);
+
+	if (!mmget_not_zero(mm))
+		return ERR_PTR(-EFAULT);
+
+	notifier = drm_gpusvm_notifier_find(gpusvm, fault_addr);
+	if (!notifier) {
+		notifier = drm_gpusvm_notifier_alloc(gpusvm,

fault_addr);

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+		if (IS_ERR(notifier)) {
+			err = PTR_ERR(notifier);
+			goto err_mmunlock;
+		}
+		notifier_alloc = true;
+		err = mmu_interval_notifier_insert(&notifier-
Quoted 1 line

notifier,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						   mm, notifier-
Quoted 1 line

itree.start,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						   notifier-
Quoted 1 line

itree.last -

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						   notifier-
Quoted 1 line

itree.start + 1,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						  

&drm_gpusvm_notifier_ops);

drivers/gpu/drm/drm_gpusvm.c new file +18 -0
+		if (err)
+			goto err_notifier;
+	}
+
+	mmap_read_lock(mm);
+
+	vas = vma_lookup(mm, fault_addr);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_notifier_remove;
+	}
+
+	if (!ctx->read_only && !(vas->vm_flags & VM_WRITE)) {
+		err = -EPERM;
+		goto err_notifier_remove;
+	}
+
+	range = drm_gpusvm_range_find(notifier, fault_addr,

fault_addr + 1);

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+	if (range)
+		goto out_mmunlock;
+	/*
+	 * XXX: Short-circuiting migration based on migrate_vma_*

current

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	 * limitations. If/when migrate_vma_* add more support, this

logic will

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+	 * have to change.
+	 */
+	migrate_devmem = ctx->devmem_possible &&
+		vma_is_anonymous(vas) && !is_vm_hugetlb_page(vas);
+
+	chunk_size = drm_gpusvm_range_chunk_size(gpusvm, notifier,

vas,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						 fault_addr,

gpuva_start,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+						 gpuva_end,
+						 ctx-
Quoted 1 line

check_pages_threshold);

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+	if (chunk_size == LONG_MAX) {
+		err = -EINVAL;
+		goto err_notifier_remove;
+	}
+
+	range = drm_gpusvm_range_alloc(gpusvm, notifier, fault_addr,

chunk_size,

drivers/gpu/drm/drm_gpusvm.c new file +31 -0
+				       migrate_devmem);
+	if (IS_ERR(range)) {
+		err = PTR_ERR(range);
+		goto err_notifier_remove;
+	}
+
+	drm_gpusvm_range_insert(notifier, range);
+	if (notifier_alloc)
+		drm_gpusvm_notifier_insert(gpusvm, notifier);
+
+out_mmunlock:
+	mmap_read_unlock(mm);
+	mmput(mm);
+
+	return range;
+
+err_notifier_remove:
+	mmap_read_unlock(mm);
+	if (notifier_alloc)
+		mmu_interval_notifier_remove(&notifier->notifier);
+err_notifier:
+	if (notifier_alloc)
+		drm_gpusvm_notifier_free(gpusvm, notifier);
+err_mmunlock:
+	mmput(mm);
+	return ERR_PTR(err);
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_find_or_insert);
+
+/**
+ * __drm_gpusvm_range_unmap_pages() - Unmap pages associated with a

GPU SVM range (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @npages: Number of pages to unmap
+ *
+ * This function unmap pages associated with a GPU SVM range. Assumes and
+ * asserts correct locking is in place when called.
+ */
+static void __drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					   struct drm_gpusvm_range

*range,

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+					   unsigned long npages)
+{
+	unsigned long i, j;
+	struct drm_pagemap *dpagemap = range->dpagemap;
+	struct device *dev = gpusvm->drm->dev;
+
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	if (range->flags.has_dma_mapping) {
+		for (i = 0, j = 0; i < npages; j++) {
+			struct drm_pagemap_dma_addr *addr = &range-
Quoted 1 line

dma_addr[j];

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+			if (addr->proto == DRM_INTERCONNECT_SYSTEM)
+				dma_unmap_page(dev,
+					       addr->addr,
+					       PAGE_SIZE << addr-
Quoted 1 line

order,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					       addr->dir);
+			else if (dpagemap && dpagemap->ops-
Quoted 1 line

unmap_dma)

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+				dpagemap->ops->unmap_dma(dpagemap,
+							 dev,
+							 *addr);
+			i += 1 << addr->order;
+		}
+		range->flags.has_devmem_pages = false;
+		range->flags.has_dma_mapping = false;
+		range->dpagemap = NULL;
+	}
+}
+
+/**
+ * drm_gpusvm_range_free_pages() - Free pages associated with a GPU

SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function frees the dma address array associated with a GPU SVM range.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static void drm_gpusvm_range_free_pages(struct drm_gpusvm *gpusvm,
+					struct drm_gpusvm_range

*range)

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+{
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	if (range->dma_addr) {
+		kvfree(range->dma_addr);
+		range->dma_addr = NULL;
+	}
+}
+
+/**
+ * drm_gpusvm_range_remove() - Remove GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range to be removed
+ *
+ * This function removes the specified GPU SVM range and also

removes the parent
+ * GPU SVM notifier if no more ranges remain in the notifier. The caller must
+ * hold a lock to protect range and notifier removal.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range)
+{
+	unsigned long npages = npages_in_range(range->itree.start,
+					       range->itree.last +

1);

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+	struct drm_gpusvm_notifier *notifier;
+
+	drm_gpusvm_driver_lock_held(gpusvm);
+
+	notifier = drm_gpusvm_notifier_find(gpusvm, range-
Quoted 1 line

itree.start);

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+	if (WARN_ON_ONCE(!notifier))
+		return;
+
+	drm_gpusvm_notifier_lock(gpusvm);
+	__drm_gpusvm_range_unmap_pages(gpusvm, range, npages);
+	drm_gpusvm_range_free_pages(gpusvm, range);
+	__drm_gpusvm_range_remove(notifier, range);
+	drm_gpusvm_notifier_unlock(gpusvm);
+
+	drm_gpusvm_range_put(range);
+
+	if (RB_EMPTY_ROOT(&notifier->root.rb_root)) {
+		if (!notifier->flags.removed)
+			mmu_interval_notifier_remove(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+		drm_gpusvm_notifier_remove(gpusvm, notifier);
+		drm_gpusvm_notifier_free(gpusvm, notifier);
+	}
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_remove);
+
+/**
+ * drm_gpusvm_range_get() - Get a reference to GPU SVM range
+ * @range: Pointer to the GPU SVM range
+ *
+ * This function increments the reference count of the specified GPU

SVM range.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range)
+{
+	kref_get(&range->refcount);
+
+	return range;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_get);
+
+/**
+ * drm_gpusvm_range_destroy() - Destroy GPU SVM range
+ * @refcount: Pointer to the reference counter embedded in the GPU

SVM range
+ *
+ * This function destroys the specified GPU SVM range when its reference count
+ * reaches zero. If a custom range-free function is provided, it is invoked to
+ * free the range; otherwise, the range is deallocated using kfree().
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_range_destroy(struct kref *refcount)
+{
+	struct drm_gpusvm_range *range =
+		container_of(refcount, struct drm_gpusvm_range,

refcount);

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+	struct drm_gpusvm *gpusvm = range->gpusvm;
+
+	if (gpusvm->ops->range_free)
+		gpusvm->ops->range_free(range);
+	else
+		kfree(range);
+}
+
+/**
+ * drm_gpusvm_range_put() - Put a reference to GPU SVM range
+ * @range: Pointer to the GPU SVM range
+ *
+ * This function decrements the reference count of the specified GPU

SVM range
+ * and frees it when the count reaches zero.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range)
+{
+	kref_put(&range->refcount, drm_gpusvm_range_destroy);
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_put);
+
+/**
+ * drm_gpusvm_range_pages_valid() - GPU SVM range pages valid
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid.

Expected be
+ * called holding gpusvm->notifier_lock and as the last step before committing a
+ * GPU binding. This is akin to a notifier seqno check in the HMM documentation
+ * but due to wider notifiers (i.e., notifiers which span multiple ranges) this
+ * function is required for finer grained checking (i.e., per range) if pages
+ * are valid.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range)
+{
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	return range->flags.has_devmem_pages || range-
Quoted 1 line

flags.has_dma_mapping;

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_pages_valid);
+
+/**
+ * drm_gpusvm_range_pages_valid_unlocked() - GPU SVM range pages

valid unlocked
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid. Expected be
+ * called without holding gpusvm->notifier_lock.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+static bool
+drm_gpusvm_range_pages_valid_unlocked(struct drm_gpusvm *gpusvm,
+				      struct drm_gpusvm_range

*range)

drivers/gpu/drm/drm_gpusvm.c new file +22 -0
+{
+	bool pages_valid;
+
+	if (!range->dma_addr)
+		return false;
+
+	drm_gpusvm_notifier_lock(gpusvm);
+	pages_valid = drm_gpusvm_range_pages_valid(gpusvm, range);
+	if (!pages_valid)
+		drm_gpusvm_range_free_pages(gpusvm, range);
+	drm_gpusvm_notifier_unlock(gpusvm);
+
+	return pages_valid;
+}
+
+/**
+ * drm_gpusvm_range_get_pages() - Get pages for a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function gets pages for a GPU SVM range and ensures they are

mapped for
+ * DMA access.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	struct mmu_interval_notifier *notifier = &range->notifier-
Quoted 1 line

notifier;

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+	struct hmm_range hmm_range = {
+		.default_flags = HMM_PFN_REQ_FAULT | (ctx->read_only

? 0 :

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+			HMM_PFN_REQ_WRITE),
+		.notifier = notifier,
+		.start = range->itree.start,
+		.end = range->itree.last + 1,
+		.dev_private_owner = gpusvm-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	struct drm_gpusvm_zdd *zdd;
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+	unsigned long i, j;
+	unsigned long npages = npages_in_range(range->itree.start,
+					       range->itree.last +

1);

drivers/gpu/drm/drm_gpusvm.c new file +58 -0
+	unsigned long num_dma_mapped;
+	unsigned int order = 0;
+	unsigned long *pfns;
+	struct page **pages;
+	int err = 0;
+	struct dev_pagemap *pagemap;
+	struct drm_pagemap *dpagemap;
+
+retry:
+	hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
+	if (drm_gpusvm_range_pages_valid_unlocked(gpusvm, range))
+		goto set_seqno;
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL);
+	if (!pfns)
+		return -ENOMEM;
+
+	if (!mmget_not_zero(mm)) {
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	hmm_range.hmm_pfns = pfns;
+	while (true) {
+		mmap_read_lock(mm);
+		err = hmm_range_fault(&hmm_range);
+		mmap_read_unlock(mm);
+
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq =
+				mmu_interval_read_begin(notifier);
+			continue;
+		}
+		break;
+	}
+	mmput(mm);
+	if (err)
+		goto err_free;
+
+	pages = (struct page **)pfns;
+map_pages:
+	/*
+	 * Perform all dma mappings under the notifier lock to not
+	 * access freed pages. A notifier will either block on
+	 * the notifier lock or unmap dma.
+	 */
+	drm_gpusvm_notifier_lock(gpusvm);
+
+	if (range->flags.unmapped) {
+		drm_gpusvm_notifier_unlock(gpusvm);
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	if (mmu_interval_read_retry(notifier,

hmm_range.notifier_seq)) {

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+		drm_gpusvm_notifier_unlock(gpusvm);
+		kvfree(pfns);
+		goto retry;
+	}
+
+	if (!range->dma_addr) {
+		/* Unlock and restart mapping to allocate memory. */
+		drm_gpusvm_notifier_unlock(gpusvm);
+		range->dma_addr = kvmalloc_array(npages,
+						 sizeof(*range-
Quoted 1 line

dma_addr),

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+						 GFP_KERNEL);
+		if (!range->dma_addr) {
+			err = -ENOMEM;
+			goto err_free;
+		}
+		goto map_pages;
+	}
+
+	zdd = NULL;
+	num_dma_mapped = 0;
+	for (i = 0, j = 0; i < npages; ++j) {
+		struct page *page = hmm_pfn_to_page(pfns[i]);
+
+		order = hmm_pfn_to_map_order(pfns[i]);
+		if (is_device_private_page(page) ||
+		    is_device_coherent_page(page)) {
+			if (zdd != page->zone_device_data && i > 0)

{

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+				err = -EOPNOTSUPP;
+				goto err_unmap;
+			}
+			zdd = page->zone_device_data;
+			if (pagemap != page->pgmap) {
+				if (i > 0) {
+					err = -EOPNOTSUPP;
+					goto err_unmap;
+				}
+
+				pagemap = page->pgmap;
+				dpagemap = zdd->devmem_allocation-
Quoted 1 line

dpagemap;

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				if (drm_WARN_ON(gpusvm->drm,

!dpagemap)) {

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					/*
+					 * Raced. This is not

supposed to happen

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					 * since hmm_range_fault()

should've migrated

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+					 * this page to system.
+					 */
+					err = -EAGAIN;
+					goto err_unmap;
+				}
+			}
+			range->dma_addr[j] =
+				dpagemap->ops->map_dma(dpagemap,
+						       gpusvm->drm-
Quoted 1 line

dev,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+						       page, order,
+						      

DMA_BIDIRECTIONAL);

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+			if (dma_mapping_error(gpusvm->drm->dev,
+					      range-
Quoted 1 line

dma_addr[j].addr)) {

drivers/gpu/drm/drm_gpusvm.c new file +18 -0
+				err = -EFAULT;
+				goto err_unmap;
+			}
+
+			pages[i] = page;
+		} else {
+			dma_addr_t addr;
+
+			if (is_zone_device_page(page) || zdd) {
+				err = -EOPNOTSUPP;
+				goto err_unmap;
+			}
+
+			addr = dma_map_page(gpusvm->drm->dev,
+					    page, 0,
+					    PAGE_SIZE << order,
+					    DMA_BIDIRECTIONAL);
+			if (dma_mapping_error(gpusvm->drm->dev,

addr)) {

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+				err = -EFAULT;
+				goto err_unmap;
+			}
+
+			range->dma_addr[j] =

drm_pagemap_dma_addr_encode

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				(addr, DRM_INTERCONNECT_SYSTEM,

order,

drivers/gpu/drm/drm_gpusvm.c new file +21 -0
+				 DMA_BIDIRECTIONAL);
+		}
+		i += 1 << order;
+		num_dma_mapped = i;
+	}
+
+	range->flags.has_dma_mapping = true;
+	if (zdd) {
+		range->flags.has_devmem_pages = true;
+		range->dpagemap = dpagemap;
+	}
+
+	drm_gpusvm_notifier_unlock(gpusvm);
+	kvfree(pfns);
+set_seqno:
+	range->notifier_seq = hmm_range.notifier_seq;
+
+	return 0;
+
+err_unmap:
+	__drm_gpusvm_range_unmap_pages(gpusvm, range,

num_dma_mapped);

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+	drm_gpusvm_notifier_unlock(gpusvm);
+err_free:
+	kvfree(pfns);
+	if (err == -EAGAIN)
+		goto retry;
+	return err;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_get_pages);
+
+/**
+ * drm_gpusvm_range_unmap_pages() - Unmap pages associated with a

GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed that gpusvm->notifier_lock is held in write mode; if it
+ * is clear, it acquires gpusvm->notifier_lock in read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops-

Quoted 1 line

invalidate for IOMMU

+ * security model.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx)
+{
+	unsigned long npages = npages_in_range(range->itree.start,
+					       range->itree.last +

1);

drivers/gpu/drm/drm_gpusvm.c new file +33 -0
+
+	if (ctx->in_notifier)
+		lockdep_assert_held_write(&gpusvm->notifier_lock);
+	else
+		drm_gpusvm_notifier_lock(gpusvm);
+
+	__drm_gpusvm_range_unmap_pages(gpusvm, range, npages);
+
+	if (!ctx->in_notifier)
+		drm_gpusvm_notifier_unlock(gpusvm);
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_unmap_pages);
+
+/**
+ * drm_gpusvm_migration_unlock_put_page() - Put a migration page
+ * @page: Pointer to the page to put
+ *
+ * This function unlocks and puts a page.
+ */
+static void drm_gpusvm_migration_unlock_put_page(struct page *page)
+{
+	unlock_page(page);
+	put_page(page);
+}
+
+/**
+ * drm_gpusvm_migration_unlock_put_pages() - Put migration pages
+ * @npages: Number of pages
+ * @migrate_pfn: Array of migrate page frame numbers
+ *
+ * This function unlocks and puts an array of pages.
+ */
+static void drm_gpusvm_migration_unlock_put_pages(unsigned long

npages,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						  unsigned long

*migrate_pfn)

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page;
+
+		if (!migrate_pfn[i])
+			continue;
+
+		page = migrate_pfn_to_page(migrate_pfn[i]);
+		drm_gpusvm_migration_unlock_put_page(page);
+		migrate_pfn[i] = 0;
+	}
+}
+
+/**
+ * drm_gpusvm_get_devmem_page() - Get a reference to a device memory

page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU SVM zone
+ * device data and initializes it for zone device usage.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+static void drm_gpusvm_get_devmem_page(struct page *page,
+				     struct drm_gpusvm_zdd *zdd)
+{
+	page->zone_device_data = drm_gpusvm_zdd_get(zdd);
+	zone_device_page_init(page);
+}
+
+/**
+ * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM

migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Returns: 0 on success, -EFAULT if an error occurs during mapping.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+static int drm_gpusvm_migrate_map_pages(struct device *dev,
+					dma_addr_t *dma_addr,
+					unsigned long *migrate_pfn,
+					unsigned long npages,
+					enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page =

migrate_pfn_to_page(migrate_pfn[i]);

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+
+		if (!page)
+			continue;
+
+		if (WARN_ON_ONCE(is_zone_device_page(page)))
+			return -EFAULT;
+
+		dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE,

dir);

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+		if (dma_mapping_error(dev, dma_addr[i]))
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped

for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_migrate_unmap_pages(struct device *dev,
+					   dma_addr_t *dma_addr,
+					   unsigned long npages,
+					   enum dma_data_direction

dir)

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!dma_addr[i] || dma_mapping_error(dev,

dma_addr[i]))

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+			continue;
+
+		dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir);
+	}
+}
+
+/**
+ * drm_gpusvm_migrate_to_devmem() - Migrate GPU SVM range to device

memory
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @devmem_allocation: Pointer to the device memory allocation. The caller
+ *                     should hold a reference to the device memory allocation,
+ *                     which should be dropped via ops-

Quoted 1 line

devmem_release or upon

+ *                     the failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to device memory. It performs the
+ * necessary setup and invokes the driver-specific operations for migration to
+ * device memory. Upon successful return, @devmem_allocation can safely reference @range
+ * until ops->devmem_release is called which only upon successful return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+int drm_gpusvm_migrate_to_devmem(struct drm_gpusvm *gpusvm,
+				 struct drm_gpusvm_range *range,
+				 struct drm_gpusvm_devmem

*devmem_allocation,

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+				 const struct drm_gpusvm_ctx *ctx)
+{
+	const struct drm_gpusvm_devmem_ops *ops = devmem_allocation-
Quoted 1 line

ops;

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	unsigned long start = range->itree.start, end = range-
Quoted 1 line

itree.last + 1;

drivers/gpu/drm/drm_gpusvm.c new file +45 -0
+	struct migrate_vma migrate = {
+		.start		= start,
+		.end		= end,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_SYSTEM,
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	unsigned long i, npages = npages_in_range(start, end);
+	struct vm_area_struct *vas;
+	struct drm_gpusvm_zdd *zdd = NULL;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int err;
+
+	if (!range->flags.migrate_devmem)
+		return -EINVAL;
+
+	if (!ops->populate_devmem_pfn || !ops->copy_to_devmem ||
+	    !ops->copy_to_ram)
+		return -EOPNOTSUPP;
+
+	if (!mmget_not_zero(mm)) {
+		err = -EFAULT;
+		goto err_out;
+	}
+	mmap_read_lock(mm);
+
+	vas = vma_lookup(mm, start);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end > vas->vm_end || start < vas->vm_start) {
+		err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	if (!vma_is_anonymous(vas)) {
+		err = -EBUSY;
+		goto err_mmunlock;
+	}
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +

sizeof(*dma_addr) +

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_mmunlock;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr))

* npages;

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+
+	zdd = drm_gpusvm_zdd_alloc(gpusvm-
Quoted 1 line

device_private_page_owner);

drivers/gpu/drm/drm_gpusvm.c new file +24 -0
+	if (!zdd) {
+		err = -ENOMEM;
+		goto err_free;
+	}
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	if (!migrate.cpages) {
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	if (migrate.cpages != npages) {
+		err = -EBUSY;
+		goto err_finalize;
+	}
+
+	err = ops->populate_devmem_pfn(devmem_allocation, npages,

migrate.dst);

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(devmem_allocation->dev,

dma_addr,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					   migrate.src, npages,

DMA_TO_DEVICE);

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = pfn_to_page(migrate.dst[i]);
+
+		pages[i] = page;
+		migrate.dst[i] = migrate_pfn(migrate.dst[i]);
+		drm_gpusvm_get_devmem_page(page, zdd);
+	}
+
+	err = ops->copy_to_devmem(pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+	/* Upon success bind devmem allocation to range and zdd */
+	zdd->devmem_allocation = devmem_allocation;	/* Owns ref

*/

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_unlock_put_pages(npages,

migrate.dst);

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(devmem_allocation->dev,

dma_addr, npages,

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+				       DMA_TO_DEVICE);
+err_free:
+	if (zdd)
+		drm_gpusvm_zdd_put(zdd);
+	kvfree(buf);
+err_mmunlock:
+	mmap_read_unlock(mm);
+	mmput(mm);
+err_out:
+	return err;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_migrate_to_devmem);
+
+/**
+ * drm_gpusvm_migrate_populate_ram_pfn() - Populate RAM PFNs for a

VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @fault_page: Fault page
+ * @npages: Number of pages to populate
+ * @mpages: Number of pages to migrate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the RAM migrate page frame numbers (PFNs) for the
+ * specified VM area structure. It allocates and locks pages in the VM area for
+ * RAM usage. If vas is non-NULL use alloc_page_vma for allocation, if NULL use
+ * alloc_page for allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_ram_pfn(struct vm_area_struct *vas,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					       struct page

*fault_page,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					       unsigned long npages,
+					       unsigned long

*mpages,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					       unsigned long

*src_mpfn,

drivers/gpu/drm/drm_gpusvm.c new file +23 -0
+					       unsigned long *mpfn,
+					       unsigned long addr)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i, addr += PAGE_SIZE) {
+		struct page *page, *src_page;
+
+		if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE))
+			continue;
+
+		src_page = migrate_pfn_to_page(src_mpfn[i]);
+		if (!src_page)
+			continue;
+
+		if (fault_page) {
+			if (src_page->zone_device_data !=
+			    fault_page->zone_device_data)
+				continue;
+		}
+
+		if (vas)
+			page = alloc_page_vma(GFP_HIGHUSER, vas,

addr);

drivers/gpu/drm/drm_gpusvm.c new file +39 -0
+		else
+			page = alloc_page(GFP_HIGHUSER);
+
+		if (!page)
+			goto free_pages;
+
+		mpfn[i] = migrate_pfn(page_to_pfn(page));
+	}
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = migrate_pfn_to_page(mpfn[i]);
+
+		if (!page)
+			continue;
+
+		WARN_ON_ONCE(!trylock_page(page));
+		++*mpages;
+	}
+
+	return 0;
+
+free_pages:
+	for (i = 0; i < npages; ++i) {
+		struct page *page = migrate_pfn_to_page(mpfn[i]);
+
+		if (!page)
+			continue;
+
+		put_page(page);
+		mpfn[i] = 0;
+	}
+	return -ENOMEM;
+}
+
+/**
+ * drm_gpusvm_evict_to_ram() - Evict GPU SVM range to RAM
+ * @devmem_allocation: Pointer to the device memory allocation
+ *
+ * Similar to __drm_gpusvm_migrate_to_ram but does not require mmap

lock and
+ * migration done via migrate_device_* functions.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_evict_to_ram(struct drm_gpusvm_devmem *devmem_allocation)

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+{
+	const struct drm_gpusvm_devmem_ops *ops = devmem_allocation-
Quoted 1 line

ops;

drivers/gpu/drm/drm_gpusvm.c new file +25 -0
+	unsigned long npages, mpages = 0;
+	struct page **pages;
+	unsigned long *src, *dst;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+	unsigned int retry_count = 2;
+
+	npages = devmem_allocation->size >> PAGE_SHIFT;
+
+retry:
+	if (!mmget_not_zero(devmem_allocation->mm))
+		return -EFAULT;
+
+	buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr)
+
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	src = buf;
+	dst = buf + (sizeof(*src) * npages);
+	dma_addr = buf + (2 * sizeof(*src) * npages);
+	pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) *

npages;

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+
+	err = ops->populate_devmem_pfn(devmem_allocation, npages,

src);

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+	if (err)
+		goto err_free;
+
+	err = migrate_device_pfns(src, npages);
+	if (err)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_ram_pfn(NULL, NULL,

npages, &mpages,

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+						  src, dst, 0);
+	if (err || !mpages)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(devmem_allocation->dev,

dma_addr,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					   dst, npages,

DMA_FROM_DEVICE);

drivers/gpu/drm/drm_gpusvm.c new file +16 -0
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(src[i]);
+
+	err = ops->copy_to_ram(pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_unlock_put_pages(npages, dst);
+	migrate_device_pages(src, dst, npages);
+	migrate_device_finalize(src, dst, npages);
+	drm_gpusvm_migrate_unmap_pages(devmem_allocation->dev,

dma_addr, npages,

drivers/gpu/drm/drm_gpusvm.c new file +20 -0
+				       DMA_FROM_DEVICE);
+err_free:
+	kvfree(buf);
+err_out:
+	mmput_async(devmem_allocation->mm);
+
+	if (completion_done(&devmem_allocation->detached))
+		return 0;
+
+	if (!err || retry_count--) {
+		cond_resched();
+		goto retry;
+	}
+
+	return err;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_evict_to_ram);
+
+/**
+ * __drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM

(internal)
+ * @vas: Pointer to the VM area structure
+ * @device_private_page_owner: Device private pages owner
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @fault_addr: Fault address
+ * @size: Size of migration
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to RAM. It sets up the migration, populates + dma maps RAM PFNs, and
+ * invokes the driver-specific operations for migration to RAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static int __drm_gpusvm_migrate_to_ram(struct vm_area_struct *vas,
+				       void

*device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+				       struct page *page,
+				       unsigned long fault_addr,
+				       unsigned long size)
+{
+	struct migrate_vma migrate = {
+		.vma		= vas,
+		.pgmap_owner	= device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_DEVICE_PRIVATE

|

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+			MIGRATE_VMA_SELECT_DEVICE_COHERENT,
+		.fault_page	= page,
+	};
+	struct drm_gpusvm_zdd *zdd;
+	const struct drm_gpusvm_devmem_ops *ops;
+	struct device *dev;
+	unsigned long npages, mpages = 0;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	unsigned long start, end;
+	void *buf;
+	int i, err = 0;
+
+	start = ALIGN_DOWN(fault_addr, size);
+	end = ALIGN(fault_addr + 1, size);
+
+	/* Corner where VMA area struct has been partially unmapped

*/

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+	if (start < vas->vm_start)
+		start = vas->vm_start;
+	if (end > vas->vm_end)
+		end = vas->vm_end;
+
+	migrate.start = start;
+	migrate.end = end;
+	npages = npages_in_range(start, end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +

sizeof(*dma_addr) +

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr))

* npages;

drivers/gpu/drm/drm_gpusvm.c new file +30 -0
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/* Raced with another CPU fault, nothing to do */
+	if (!migrate.cpages)
+		goto err_free;
+
+	if (!page) {
+		for (i = 0; i < npages; ++i) {
+			if (!(migrate.src[i] & MIGRATE_PFN_MIGRATE))
+				continue;
+
+			page = migrate_pfn_to_page(migrate.src[i]);
+			break;
+		}
+
+		if (!page)
+			goto err_finalize;
+	}
+	zdd = page->zone_device_data;
+	ops = zdd->devmem_allocation->ops;
+	dev = zdd->devmem_allocation->dev;
+
+	err = drm_gpusvm_migrate_populate_ram_pfn(vas, page, npages,

&mpages,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						  migrate.src,

migrate.dst,

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+						  start);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(dev, dma_addr,

migrate.dst, npages,

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+					   DMA_FROM_DEVICE);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(migrate.src[i]);
+
+	err = ops->copy_to_ram(pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_unlock_put_pages(npages,

migrate.dst);

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(dev, dma_addr, npages,
+				       DMA_FROM_DEVICE);
+err_free:
+	kvfree(buf);
+err_out:
+
+	return err;
+}
+
+/**
+ * drm_gpusvm_range_evict - Evict GPU SVM range
+ * @pagemap: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range to be removed
+ *
+ * This function evicts the specified GPU SVM range. This function

will not
+ * evict coherent pages.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+int drm_gpusvm_range_evict(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_range *range)
+{
+	struct mmu_interval_notifier *notifier = &range->notifier-
Quoted 1 line

notifier;

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+	struct hmm_range hmm_range = {
+		.default_flags = HMM_PFN_REQ_FAULT,
+		.notifier = notifier,
+		.start = range->itree.start,
+		.end = range->itree.last + 1,
+		.dev_private_owner = NULL,
+	};
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+	unsigned long *pfns;
+	unsigned long npages = npages_in_range(range->itree.start,
+					       range->itree.last +

1);

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+	int err = 0;
+	struct mm_struct *mm = gpusvm->mm;
+
+	if (!mmget_not_zero(mm))
+		return -EFAULT;
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL);
+	if (!pfns)
+		return -ENOMEM;
+
+	hmm_range.hmm_pfns = pfns;
+	while (!time_after(jiffies, timeout)) {
+		hmm_range.notifier_seq =

mmu_interval_read_begin(notifier);

drivers/gpu/drm/drm_gpusvm.c new file +21 -0
+		if (time_after(jiffies, timeout)) {
+			err = -ETIME;
+			break;
+		}
+
+		mmap_read_lock(mm);
+		err = hmm_range_fault(&hmm_range);
+		mmap_read_unlock(mm);
+		if (err != -EBUSY)
+			break;
+	}
+
+	kvfree(pfns);
+	mmput(mm);
+
+	return err;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_evict);
+
+/**
+ * drm_gpusvm_page_free() - Put GPU SVM zone device data associated

with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device data
+ * associated with a page when it is being released.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+static void drm_gpusvm_page_free(struct page *page)
+{
+	drm_gpusvm_zdd_put(page->zone_device_data);
+}
+
+/**
+ * drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM (page

fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ * VM_FAULT_SIGBUS on failure, 0 on success.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+static vm_fault_t drm_gpusvm_migrate_to_ram(struct vm_fault *vmf)
+{
+	struct drm_gpusvm_zdd *zdd = vmf->page->zone_device_data;
+	int err;
+
+	err = __drm_gpusvm_migrate_to_ram(vmf->vma,
+					  zdd-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					  vmf->page, vmf->address,
+					  zdd->devmem_allocation-
Quoted 1 line

size);

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+
+	return err ? VM_FAULT_SIGBUS : 0;
+}
+
+/**
+ * drm_gpusvm_pagemap_ops() - Device page map operations for GPU SVM
+ */
+static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = {
+	.page_free = drm_gpusvm_page_free,
+	.migrate_to_ram = drm_gpusvm_migrate_to_ram,
+};
+
+/**
+ * drm_gpusvm_pagemap_ops_get() - Retrieve GPU SVM device page map

operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void)
+{
+	return &drm_gpusvm_pagemap_ops;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_pagemap_ops_get);
+
+/**
+ * drm_gpusvm_has_mapping() - Check if GPU SVM has mapping for the

given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, unsigned long start,

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+			    unsigned long end)
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	drm_gpusvm_for_each_notifier(notifier, gpusvm, start, end) {
+		struct drm_gpusvm_range *range = NULL;
+
+		drm_gpusvm_for_each_range(range, notifier, start,

end)

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+			return true;
+	}
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_has_mapping);
+
+/**
+ * drm_gpusvm_range_set_unmapped() - Mark a GPU SVM range as

unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets the partial_unmap flag
+ * if the range partially falls within the provided MMU notifier range.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+void drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,
+				   const struct mmu_notifier_range

*mmu_range)

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+{
+	lockdep_assert_held_write(&range->gpusvm->notifier_lock);
+
+	range->flags.unmapped = true;
+	if (range->itree.start < mmu_range->start ||
+	    range->itree.last + 1 > mmu_range->end)
+		range->flags.partial_unmap = true;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_set_unmapped);
+
+/**
+ * drm_gpusvm_devmem_init() - Initialize a GPU SVM device memory

allocation
+ *
+ * @dev: Pointer to the device structure which device memory allocation belongs to
+ * @mm: Pointer to the mm_struct for the address space
+ * @ops: Pointer to the operations structure for GPU SVM device memory
+ * @dpagemap: The struct drm_pagemap we're allocating from.
+ * @size: Size of device memory allocation
+ */
+void drm_gpusvm_devmem_init(struct drm_gpusvm_devmem *devmem_allocation,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+			    struct device *dev, struct mm_struct

*mm,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+			    const struct drm_gpusvm_devmem_ops *ops,
+			    struct drm_pagemap *dpagemap, size_t

size)

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+{
+	init_completion(&devmem_allocation->detached);
+	devmem_allocation->dev = dev;
+	devmem_allocation->mm = mm;
+	devmem_allocation->ops = ops;
+	devmem_allocation->dpagemap = dpagemap;
+	devmem_allocation->size = size;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_devmem_init);
+
+MODULE_DESCRIPTION("DRM GPUSVM");
+MODULE_LICENSE("GPL");
include/drm/drm_gpusvm.h new file +24 -0
new file mode 100644
index 000000000000..ea31db0be841
@@ -0,0 +1,445 @@
+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef __DRM_GPUSVM_H__
+#define __DRM_GPUSVM_H__
+
+#include <linux/kref.h>
+#include <linux/interval_tree.h>
+#include <linux/mmu_notifier.h>
+
+struct dev_pagemap_ops;
+struct drm_device;
+struct drm_gpusvm;
+struct drm_gpusvm_notifier;
+struct drm_gpusvm_ops;
+struct drm_gpusvm_range;
+struct drm_gpusvm_devmem;
+struct drm_pagemap;
+struct drm_pagemap_dma_addr;
+
+/**
+ * struct drm_gpusvm_devmem_ops - Operations structure for GPU SVM

device memory
+ *
+ * This structure defines the operations for GPU Shared Virtual Memory (SVM)
+ * device memory. These operations are provided by the GPU driver to manage device memory
+ * allocations and perform operations such as migration between device memory and system
+ * RAM.
+ */

include/drm/drm_gpusvm.h new file +3 -0
+struct drm_gpusvm_devmem_ops {
+	/**
+	 * @devmem_release: Release device memory allocation

(optional)

include/drm/drm_gpusvm.h new file +3 -0
+	 * @devmem_allocation: device memory allocation
+	 *
+	 * Release device memory allocation and drop a reference to

device

include/drm/drm_gpusvm.h new file +3 -0
+	 * memory allocation.
+	 */
+	void (*devmem_release)(struct drm_gpusvm_devmem

*devmem_allocation);

include/drm/drm_gpusvm.h new file +3 -0
+
+	/**
+	 * @populate_devmem_pfn: Populate device memory PFN

(required for migration)

include/drm/drm_gpusvm.h new file +10 -0
+	 * @devmem_allocation: device memory allocation
+	 * @npages: Number of pages to populate
+	 * @pfn: Array of page frame numbers to populate
+	 *
+	 * Populate device memory page frame numbers (PFN).
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*populate_devmem_pfn)(struct drm_gpusvm_devmem

*devmem_allocation,

include/drm/drm_gpusvm.h new file +1 -0
+				   unsigned long npages, unsigned

long *pfn);

include/drm/drm_gpusvm.h new file +3 -0
+
+	/**
+	 * @copy_to_devmem: Copy to device memory (required for

migration)

include/drm/drm_gpusvm.h new file +1 -0
+	 * @pages: Pointer to array of device memory pages

(destination)

include/drm/drm_gpusvm.h new file +16 -0
+	 * @dma_addr: Pointer to array of DMA addresses (source)
+	 * @npages: Number of pages to copy
+	 *
+	 * Copy pages to device memory.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_devmem)(struct page **pages,
+			      dma_addr_t *dma_addr,
+			      unsigned long npages);
+
+	/**
+	 * @copy_to_ram: Copy to system RAM (required for migration)
+	 * @pages: Pointer to array of device memory pages (source)
+	 * @dma_addr: Pointer to array of DMA addresses

(destination)

include/drm/drm_gpusvm.h new file +14 -0
+	 * @npages: Number of pages to copy
+	 *
+	 * Copy pages to system RAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_ram)(struct page **pages,
+			   dma_addr_t *dma_addr,
+			   unsigned long npages);
+};
+
+/**
+ * struct drm_gpusvm_devmem - Structure representing a GPU SVM

device memory allocation
+ *
+ * @dev: Pointer to the device structure which device memory allocation belongs to
+ * @mm: Pointer to the mm_struct for the address space
+ * @detached: device memory allocations is detached from device pages
+ * @ops: Pointer to the operations structure for GPU SVM device memory
+ * @dpagemap: The struct drm_pagemap of the pages this allocation belongs to.
+ * @size: Size of device memory allocation
+ */

include/drm/drm_gpusvm.h new file +13 -0
+struct drm_gpusvm_devmem {
+	struct device *dev;
+	struct mm_struct *mm;
+	struct completion detached;
+	const struct drm_gpusvm_devmem_ops *ops;
+	struct drm_pagemap *dpagemap;
+	size_t size;
+};
+
+/**
+ * struct drm_gpusvm_ops - Operations structure for GPU SVM
+ *
+ * This structure defines the operations for GPU Shared Virtual

Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * notifiers.
+ */

include/drm/drm_gpusvm.h new file +8 -0
+struct drm_gpusvm_ops {
+	/**
+	 * @notifier_alloc: Allocate a GPU SVM notifier (optional)
+	 *
+	 * Allocate a GPU SVM notifier.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM notifier on success,

NULL on failure.

include/drm/drm_gpusvm.h new file +19 -0
+	 */
+	struct drm_gpusvm_notifier *(*notifier_alloc)(void);
+
+	/**
+	 * @notifier_free: Free a GPU SVM notifier (optional)
+	 * @notifier: Pointer to the GPU SVM notifier to be freed
+	 *
+	 * Free a GPU SVM notifier.
+	 */
+	void (*notifier_free)(struct drm_gpusvm_notifier *notifier);
+
+	/**
+	 * @range_alloc: Allocate a GPU SVM range (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 *
+	 * Allocate a GPU SVM range.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM range on success, NULL

on failure.

include/drm/drm_gpusvm.h new file +2 -0
+	 */
+	struct drm_gpusvm_range *(*range_alloc)(struct drm_gpusvm

*gpusvm);

include/drm/drm_gpusvm.h new file +16 -0
+
+	/**
+	 * @range_free: Free a GPU SVM range (optional)
+	 * @range: Pointer to the GPU SVM range to be freed
+	 *
+	 * Free a GPU SVM range.
+	 */
+	void (*range_free)(struct drm_gpusvm_range *range);
+
+	/**
+	 * @invalidate: Invalidate GPU SVM notifier (required)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @notifier: Pointer to the GPU SVM notifier
+	 * @mmu_range: Pointer to the mmu_notifier_range structure
+	 *
+	 * Invalidate the GPU page tables. It can safely walk the

notifier range

include/drm/drm_gpusvm.h new file +1 -0
+	 * RB tree/list in this function. Called while holding the

notifier lock.

include/drm/drm_gpusvm.h new file +4 -0
+	 */
+	void (*invalidate)(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_notifier *notifier,
+			   const struct mmu_notifier_range

*mmu_range);

include/drm/drm_gpusvm.h new file +4 -0
+};
+
+/**
+ * struct drm_gpusvm_notifier - Structure representing a GPU SVM

notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @itree: Interval tree node for the notifier (inserted in GPU SVM)
+ * @entry: List entry to fast interval tree traversal
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ *              interval tree. This is useful to keep iterating ranges while
+ *              doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier has been
+ *                 removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */

include/drm/drm_gpusvm.h new file +19 -0
+struct drm_gpusvm_notifier {
+	struct drm_gpusvm *gpusvm;
+	struct mmu_interval_notifier notifier;
+	struct interval_tree_node itree;
+	struct list_head entry;
+	struct rb_root_cached root;
+	struct list_head range_list;
+	struct {
+		u32 removed : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm_range - Structure representing a GPU SVM range
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier
+ * @refcount: Reference count for the range
+ * @itree: Interval tree node for the range (inserted in GPU SVM

notifier)
+ * @entry: List entry to fast interval tree traversal
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @dma_addr: DMA address array
+ * @dpagemap: The struct drm_pagemap of the device pages we're dma- mapping.
+ *            Note this is assuming only one drm_pagemap per range is allowed.
+ * @flags.migrate_devmem: Flag indicating whether the range can be migrated to device memory
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_devmem_pages: Flag indicating if the range has devmem pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA mapping
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */

include/drm/drm_gpusvm.h new file +13 -0
+struct drm_gpusvm_range {
+	struct drm_gpusvm *gpusvm;
+	struct drm_gpusvm_notifier *notifier;
+	struct kref refcount;
+	struct interval_tree_node itree;
+	struct list_head entry;
+	unsigned long notifier_seq;
+	struct drm_pagemap_dma_addr *dma_addr;
+	struct drm_pagemap *dpagemap;
+	struct {
+		/* All flags below must be set upon creation */
+		u16 migrate_devmem : 1;
+		/* All flags below must be set / cleared under

notifier lock */

include/drm/drm_gpusvm.h new file +19 -0
+		u16 unmapped : 1;
+		u16 partial_unmap : 1;
+		u16 has_devmem_pages : 1;
+		u16 has_dma_mapping : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm - GPU SVM structure
+ *
+ * @name: Name of the GPU SVM
+ * @drm: Pointer to the DRM device structure
+ * @mm: Pointer to the mm_struct for the address space
+ * @device_private_page_owner: Device private pages owner
+ * @mm_start: Start address of GPU SVM
+ * @mm_range: Range of the GPU SVM
+ * @notifier_size: Size of individual notifiers
+ * @ops: Pointer to the operations structure for GPU SVM
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range

allocation.
+ *               Entries should be powers of 2 in descending order.
+ * @num_chunks: Number of chunks
+ * @notifier_lock: Read-write semaphore for protecting notifier operations
+ * @root: Cached root node of the Red-Black tree containing GPU SVM notifiers
+ * @notifier_list: list head containing of notifiers in the same order they
+ *                 appear in interval tree. This is useful to keep iterating
+ *                 notifiers while doing modifications to RB tree.
+ *
+ * This structure represents a GPU SVM (Shared Virtual Memory) used for tracking
+ * memory ranges mapped in a DRM (Direct Rendering Manager) device.
+ *
+ * No reference counting is provided, as this is expected to be embedded in the
+ * driver VM structure along with the struct drm_gpuvm, which handles reference
+ * counting.
+ */

include/drm/drm_gpusvm.h new file +17 -0
+struct drm_gpusvm {
+	const char *name;
+	struct drm_device *drm;
+	struct mm_struct *mm;
+	void *device_private_page_owner;
+	unsigned long mm_start;
+	unsigned long mm_range;
+	unsigned long notifier_size;
+	const struct drm_gpusvm_ops *ops;
+	const unsigned long *chunk_sizes;
+	int num_chunks;
+	struct rw_semaphore notifier_lock;
+	struct rb_root_cached root;
+	struct list_head notifier_list;
+#ifdef CONFIG_LOCKDEP
+	/**
+	 * @lock_dep_map: Annotates drm_gpusvm_range_find_or_insert

and

include/drm/drm_gpusvm.h new file +10 -0
+	 * drm_gpusvm_range_remove with a driver provided lock.
+	 */
+	struct lockdep_map *lock_dep_map;
+#endif
+};
+
+/**
+ * struct drm_gpusvm_ctx - DRM GPU SVM context
+ *
+ * @check_pages_threshold: Check CPU pages for present if chunk is

less than or
+ *                         equal to threshold. If not present, reduce chunk
+ *                         size.
+ * @in_notifier: entering from a MMU notifier
+ * @read_only: operating on read-only memory
+ * @devmem_possible: possible to use device memory
+ *
+ * Context that is DRM GPUSVM is operating in (i.e. user arguments).
+ */

include/drm/drm_gpusvm.h new file +10 -0
+struct drm_gpusvm_ctx {
+	unsigned long check_pages_threshold;
+	unsigned int in_notifier :1;
+	unsigned int read_only :1;
+	unsigned int devmem_possible :1;
+};
+
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void

*device_private_page_owner,

include/drm/drm_gpusvm.h new file +4 -0
+		    unsigned long mm_start, unsigned long mm_range,
+		    unsigned long notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const unsigned long *chunk_sizes, int

num_chunks);

include/drm/drm_gpusvm.h new file +37 -0
+
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm);
+
+void drm_gpusvm_free(struct drm_gpusvm *gpusvm);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm,
+				unsigned long fault_addr,
+				unsigned long gpuva_start,
+				unsigned long gpuva_end,
+				const struct drm_gpusvm_ctx *ctx);
+
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range);
+
+int drm_gpusvm_range_evict(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_range *range);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range);
+
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range);
+
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range);
+
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx);
+
+int drm_gpusvm_migrate_to_devmem(struct drm_gpusvm *gpusvm,
+				 struct drm_gpusvm_range *range,
+				 struct drm_gpusvm_devmem

*devmem_allocation,

include/drm/drm_gpusvm.h new file +3 -0
+				 const struct drm_gpusvm_ctx *ctx);
+
+int drm_gpusvm_evict_to_ram(struct drm_gpusvm_devmem

*devmem_allocation);

include/drm/drm_gpusvm.h new file +4 -0
+
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void);
+
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, unsigned long

start,

include/drm/drm_gpusvm.h new file +4 -0
+			    unsigned long end);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, unsigned

long start,

include/drm/drm_gpusvm.h new file +4 -0
+		      unsigned long end);
+
+void drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,
+				   const struct mmu_notifier_range

*mmu_range);

include/drm/drm_gpusvm.h new file +2 -0
+
+void drm_gpusvm_devmem_init(struct drm_gpusvm_devmem

*devmem_allocation,

include/drm/drm_gpusvm.h new file +1 -0
+			    struct device *dev, struct mm_struct

*mm,

include/drm/drm_gpusvm.h new file +2 -0
+			    const struct drm_gpusvm_devmem_ops *ops,
+			    struct drm_pagemap *dpagemap, size_t

size);

include/drm/drm_gpusvm.h new file +4 -0
+
+#ifdef CONFIG_LOCKDEP
+/**
+ * drm_gpusvm_driver_set_lock() - Set the lock protecting accesses

to GPU SVM
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @lock: the lock used to protect the gpuva list. The locking primitive
+ * must contain a dep_map field.
+ *
+ * Call this to annotate drm_gpusvm_range_find_or_insert and
+ * drm_gpusvm_range_remove.
+ */

include/drm/drm_gpusvm.h new file +4 -0
+#define drm_gpusvm_driver_set_lock(gpusvm, lock) \
+	do { \
+		if (!WARN((gpusvm)->lock_dep_map, \
+			  "GPUSVM range lock should be set only

once."))\

include/drm/drm_gpusvm.h new file +1 -0
+			(gpusvm)->lock_dep_map = &(lock)-
Quoted 1 line

dep_map; \

include/drm/drm_gpusvm.h line 387

Quoted 6 lines
include/drm/drm_gpusvm.h new file +6 -0
+	} while (0)
+#define drm_gpusvm_driver_lock_held(gpusvm) \
+	do { \
+		if ((gpusvm)->lock_dep_map)	\
+			lock_is_held((gpusvm)->lock_dep_map);	\
+	} while (0)

Could we use static functions for those above

Also I don't think the drm_gpusvm_driver_lock_held() does what it's intended to do? There's an assert missing.

Quoted 66 lines
include/drm/drm_gpusvm.h new file +25 -0
+#else
+#define drm_gpusvm_driver_set_lock(gpusvm, lock) do {} while (0)
+#define drm_gpusvm_driver_lock_held(gpusvm) do {} while (0)
+#endif
+
+/**
+ * drm_gpusvm_notifier_lock() - Lock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, take lock
+ */
+#define drm_gpusvm_notifier_lock(gpusvm__)	\
+	down_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * drm_gpusvm_notifier_unlock() - Unlock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, drop lock
+ */
+#define drm_gpusvm_notifier_unlock(gpusvm__)	\
+	up_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * __drm_gpusvm_range_next() - Get the next GPU SVM range in the

list
+ * @range: a pointer to the current GPU SVM range
+ *
+ * Return: A pointer to the next drm_gpusvm_range if available, or NULL if the
+ *         current range is the last one or if the input range is NULL.
+ */

include/drm/drm_gpusvm.h new file +12 -0
+static inline struct drm_gpusvm_range *
+__drm_gpusvm_range_next(struct drm_gpusvm_range *range)
+{
+	if (range && !list_is_last(&range->entry,
+				   &range->notifier->range_list))
+		return list_next_entry(range, entry);
+
+	return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_range() - Iterate over GPU SVM ranges in a

notifier
+ * @range__: Iterator variable for the ranges. If set, it indicates the start of
+ *      the iterator. If NULL, call drm_gpusvm_range_find() to
get the range.
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier. It is safe
+ * to use while holding the driver SVM lock or the notifier lock.
+ */
+#define drm_gpusvm_for_each_range(range__, notifier__, start__,
end__) \

include/drm/drm_gpusvm.h new file +1 -0
+	for ((range__) = (range__)

?: \

include/drm/drm_gpusvm.h new file +1 -0
+	     drm_gpusvm_range_find((notifier__), (start__),

(end__)); \

include/drm/drm_gpusvm.h new file +1 -0
+	     (range__) && (range__->itree.start <

(end__)); \

include/drm/drm_gpusvm.h line 445

Quoted 3 lines
include/drm/drm_gpusvm.h new file +3 -0
+	     (range__) = __drm_gpusvm_range_next(range__))
+
+#endif /* __DRM_GPUSVM_H__ */

Otherwise LGTM.

/Thomas

author maintainer
#21 lore raw

Re: [PATCH v4 06/33] drm/gpusvm: Add support for GPU Shared Virtual Memory

Quoted 3,193 lines · Thomas Hellström
Quoted 3,192 lines · Matthew Brost

This patch introduces support for GPU Shared Virtual Memory (SVM) in the
Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.

The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also
provides mechanisms for allocating, deallocating, and migrating memory
regions between system RAM and GPU VRAM.

This is largely inspired by GPUVM.

v2:
 - Take order into account in check pages
 - Clear range->pages in get pages error
 - Drop setting dirty or accessed bit in get pages (Vetter)
 - Remove mmap assert for cpu faults
 - Drop mmap write lock abuse (Vetter, Christian)
 - Decouple zdd from range (Vetter, Oak)
 - Add drm_gpusvm_range_evict, make it work with coherent pages
 - Export drm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
 - mmget/put in drm_gpusvm_evict_to_sram
 - Drop range->vram_alloation variable
 - Don't return in drm_gpusvm_evict_to_sram until all pages detached
 - Don't warn on mixing sram and device pages
 - Update kernel doc
 - Add coherent page support to get pages
 - Use DMA_FROM_DEVICE rather than DMA_BIDIRECTIONAL
 - Add struct drm_gpusvm_vram and ops (Thomas)
 - Update the range's seqno if the range is valid (Thomas)
 - Remove the is_unmapped check before hmm_range_fault (Thomas)
 - Use drm_pagemap (Thomas)
 - Drop kfree_mapping (Thomas)
 - dma mapp pages under notifier lock (Thomas)
 - Remove ctx.prefault
 - Remove ctx.mmap_locked
 - Add ctx.check_pages
 - s/vram/devmem (Thomas)
v3:
 - Fix memory leak drm_gpusvm_range_get_pages
 - Only migrate pages with same zdd on CPU fault
 - Loop over al VMAs in drm_gpusvm_range_evict
 - Make GPUSVM a drm level module
 - GPL or MIT license
 - Update main kernel doc (Thomas)
 - Prefer foo() vs foo for functions in kernel doc (Thomas)
 - Prefer functions over macros (Thomas)
 - Use unsigned long vs u64 for addresses (Thomas)
 - Use standard interval_tree (Thomas)
 - s/drm_gpusvm_migration_put_page/drm_gpusvm_migration_unlock_put_page (Thomas)
 - Drop err_out label in drm_gpusvm_range_find_or_insert (Thomas)
 - Fix kernel doc in drm_gpusvm_range_free_pages (Thomas)
 - Newlines between functions defs in header file (Thomas)
 - Drop shall language in driver vfunc kernel doc (Thomas)
 - Move some static inlines from head to C file (Thomas)
 - Don't allocate pages under page lock in drm_gpusvm_migrate_populate_ram_pfn (Thomas)
 - Change check_pages to a threshold
v4:
 - Fix NULL ptr deref in drm_gpusvm_migrate_populate_ram_pfn (Thomas, Himal)
 - Fix check pages threshold
 - Check for range being unmapped under notifier lock in get pages (Testing)
 - Fix characters per line
 - Drop WRITE_ONCE for zdd->devmem_allocation assignment (Thomas)
 - Use completion for devmem_allocation->detached (Thomas)
 - Make GPU SVM depend on ZONE_DEVICE (CI)
 - Use hmm_range_fault for eviction (Thomas)
 - Drop zdd worker (Thomas)

---
 drivers/gpu/drm/Kconfig      |    9 +
 drivers/gpu/drm/Makefile     |    1 +
 drivers/gpu/drm/drm_gpusvm.c | 2240
++++++++++++++++++++++++++++++++++
 include/drm/drm_gpusvm.h     |  445 +++++++

 4 files changed, 2695 insertions(+)

 create mode 100644 drivers/gpu/drm/drm_gpusvm.c
 create mode 100644 include/drm/drm_gpusvm.h

drivers/gpu/drm/Kconfig +0 -0
index fbef3f471bd0..f03862e379fb 100644
@@ -278,6 +278,15 @@ config DRM_GPUVM

    GPU-VM representation providing helpers to manage a GPUs virtual
    address space

drivers/gpu/drm/Kconfig +7 -0
+config DRM_GPUSVM
+	tristate
+	depends on DRM
+	depends on DEVICE_MIGRATION
+	depends on ZONE_DEVICE
+	help
+	  GPU-SVM representation providing helpers to manage a GPUs

shared

drivers/gpu/drm/Kconfig +2 -0
+	  virtual memory
+

 config DRM_BUDDY
  tristate
  depends on DRM

drivers/gpu/drm/Makefile +0 -0
index 85af94bb907d..ca03df8d2729 100644
@@ -104,6 +104,7 @@ obj-$(CONFIG_DRM_PANEL_BACKLIGHT_QUIRKS) +=

drm_panel_backlight_quirks.o
 #
 obj-$(CONFIG_DRM_EXEC) += drm_exec.o
 obj-$(CONFIG_DRM_GPUVM) += drm_gpuvm.o
+obj-$(CONFIG_DRM_GPUSVM) += drm_gpusvm.o

 obj-$(CONFIG_DRM_BUDDY) += drm_buddy.o

drivers/gpu/drm/drm_gpusvm.c new file +25 -0
new file mode 100644
index 000000000000..1c63da4d3cc2
@@ -0,0 +1,2240 @@
+// SPDX-License-Identifier: GPL-2.0-only OR MIT
+/*
+ * Copyright © 2024 Intel Corporation
+ *
+ * Authors:
+ *     Matthew Brost <[email protected]>
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/hmm.h>
+#include <linux/memremap.h>
+#include <linux/migrate.h>
+#include <linux/mm_types.h>
+#include <linux/pagemap.h>
+#include <linux/slab.h>
+
+#include <drm/drm_device.h>
+#include <drm/drm_gpusvm.h>
+#include <drm/drm_pagemap.h>
+#include <drm/drm_print.h>
+
+/**
+ * DOC: Overview
+ *
+ * GPU Shared Virtual Memory (GPU SVM) layer for the Direct

Rendering Manager (DRM)
+ *
+ * The GPU SVM layer is a component of the DRM framework designed to manage shared
+ * virtual memory between the CPU and GPU. It enables efficient data exchange and
+ * processing for GPU-accelerated applications by allowing memory sharing and
+ * synchronization between the CPU's and GPU's virtual address spaces.
+ *
+ * Key GPU SVM Components:
+ * - Notifiers: Notifiers: Used for tracking memory intervals and notifying the
+ * GPU of changes, notifiers are sized based on a GPU
SVM
+ * initialization parameter, with a recommendation of
512M or
+ * larger. They maintain a Red-BlacK tree and a list of
ranges that
+ * fall within the notifier interval. Notifiers are
tracked within
+ * a GPU SVM Red-BlacK tree and list and are
dynamically inserted
+ * or removed as ranges within the interval are created
or
+ * destroyed.
+ * - Ranges: Represent memory ranges mapped in a DRM device and managed
+ *      by GPU SVM. They are sized based on an array of chunk
sizes, which
+ *      is a GPU SVM initialization parameter, and the CPU
address space.
+ *      Upon GPU fault, the largest aligned chunk that fits
within the
+ *      faulting CPU address space is chosen for the range
size. Ranges are
+ *      expected to be dynamically allocated on GPU fault and
removed on an
+ *      MMU notifier UNMAP event. As mentioned above, ranges
are tracked in
+ *      a notifier's Red-Black tree.
+ * - Operations: Define the interface for driver-specific GPU SVM operations
+ *               such as range allocation, notifier allocation, and
+ *               invalidations.
+ * - Device Memory Allocations: Embedded structure containing enough information
+ *                              for GPU SVM to migrate to / from device memory.
+ * - Device Memory Operations: Define the interface for driver- specific device
+ *                             memory operations release memory, populate pfns,
+ *                             and copy to / from device memory.
+ *
+ * This layer provides interfaces for allocating, mapping, migrating, and
+ * releasing memory ranges between the CPU and GPU. It handles all core memory
+ * management interactions (DMA mapping, HMM, and migration) and provides
+ * driver-specific virtual functions (vfuncs). This infrastructure is sufficient
+ * to build the expected driver components for an SVM implementation as detailed
+ * below.
+ *
+ * Expected Driver Components:
+ * - GPU page fault handler: Used to create ranges and notifiers based on the
+ *      fault address, optionally migrate the
range to
+ *      device memory, and create GPU bindings.
+ * - Garbage collector: Used to unmap and destroy GPU bindings for ranges.
+ * Ranges are expected to be added to the
garbage collector
+ * upon a MMU_NOTIFY_UNMAP event in notifier
callback.
+ * - Notifier callback: Used to invalidate and DMA unmap GPU bindings for
+ * ranges.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Locking
+ *
+ * GPU SVM handles locking for core MM interactions, i.e., it

locks/unlocks the
+ * mmap lock as needed.
+ *
+ * GPU SVM introduces a global notifier lock, which safeguards the notifier's
+ * range RB tree and list, as well as the range's DMA mappings and sequence
+ * number. GPU SVM manages all necessary locking and unlocking operations,
+ * except for the recheck range's pages being valid
+ * (drm_gpusvm_range_pages_valid) when the driver is committing GPU bindings. This
+ * lock corresponds to the 'driver->update' lock mentioned in the HMM
+ * documentation (TODO: Link). Future revisions may transition from a GPU SVM
+ * global lock to a per-notifier lock if finer-grained locking is deemed
+ * necessary.
+ *
+ * In addition to the locking mentioned above, the driver should implement a
+ * lock to safeguard core GPU SVM function calls that modify state, such as
+ * drm_gpusvm_range_find_or_insert and drm_gpusvm_range_remove. This lock is
+ * denoted as 'driver_svm_lock' in code examples. Finer grained driver side
+ * locking should also be possible for concurrent GPU fault processing within a
+ * single GPU SVM. The 'driver_svm_lock' can be via drm_gpusvm_driver_set_lock
+ * to add annotations to GPU SVM.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Migration
+ *
+ * The migration support is quite simple, allowing migration between

RAM and
+ * device memory at the range granularity. For example, GPU SVM currently does not
+ * support mixing RAM and device memory pages within a range. This means that upon GPU
+ * fault, the entire range can be migrated to device memory, and upon CPU fault, the
+ * entire range is migrated to RAM. Mixed RAM and device memory storage within a range
+ * could be added in the future if required.
+ *
+ * The reasoning for only supporting range granularity is as
follows: it
+ * simplifies the implementation, and range sizes are driver-defined and should
+ * be relatively small.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Partial Unmapping of Ranges
+ *
+ * Partial unmapping of ranges (e.g., 1M out of 2M is unmapped by

CPU resulting
+ * in MMU_NOTIFY_UNMAP event) presents several challenges, with the main one
+ * being that a subset of the range still has CPU and GPU mappings. If the
+ * backing store for the range is in device memory, a subset of the backing store has
+ * references. One option would be to split the range and device memory backing store,
+ * but the implementation for this would be quite complicated. Given that
+ * partial unmappings are rare and driver-defined range sizes are relatively
+ * small, GPU SVM does not support splitting of ranges.
+ *
+ * With no support for range splitting, upon partial unmapping of a range, the
+ * driver is expected to invalidate and destroy the entire range. If the range
+ * has device memory as its backing, the driver is also expected to migrate any
+ * remaining pages back to RAM.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Examples
+ *
+ * This section provides three examples of how to build the expected

driver
+ * components: the GPU page fault handler, the garbage collector, and the
+ * notifier callback.
+ *
+ * The generic code provided does not include logic for complex migration
+ * policies, optimized invalidations, fined grained driver locking, or other
+ * potentially required driver locking (e.g., DMA-resv locks).
+ *
+ * 1) GPU page fault handler
+ *
+ * int driver_bind_range(struct drm_gpusvm *gpusvm, struct
drm_gpusvm_range *range)
+ * {
+ * int err = 0;
+ *
+ * driver_alloc_and_setup_memory_for_bind(gpusvm,
range);
+ *
+ * drm_gpusvm_notifier_lock(gpusvm);
+ * if (drm_gpusvm_range_pages_valid(range))
+ * driver_commit_bind(gpusvm, range);
+ * else
+ * err = -EAGAIN;
+ * drm_gpusvm_notifier_unlock(gpusvm);
+ *
+ * return err;
+ * }
+ *
+ * int driver_gpu_fault(struct drm_gpusvm *gpusvm, unsigned
long fault_addr,
+ *      unsigned long gpuva_start, unsigned
long gpuva_end)
+ * {
+ * struct drm_gpusvm_ctx ctx = {};
+ * int err;
+ *
+ * driver_svm_lock();
+ * retry:
+ * // Always process UNMAPs first so view of GPU SVM
ranges is current
+ * driver_garbage_collector(gpusvm);
+ *
+ * range = drm_gpusvm_range_find_or_insert(gpusvm,
fault_addr,
+ * gpuva_start,
gpuva_end,
+ *         &ctx);
+ * if (IS_ERR(range)) {
+ * err = PTR_ERR(range);
+ * goto unlock;
+ * }
+ *
+ * if (driver_migration_policy(range)) {
+ * devmem = driver_alloc_devmem();
+ * err = drm_gpusvm_migrate_to_devmem(gpusvm,
range,
+ *
devmem_allocation,
+ *    &ctx);
+ * if (err) // CPU mappings may have
changed
+ * goto retry;
+ * }
+ *
+ * err = drm_gpusvm_range_get_pages(gpusvm, range,
&ctx);
+ * if (err == -EOPNOTSUPP || err == -EFAULT || err == -
EPERM) { // CPU mappings changed
+ * if (err == -EOPNOTSUPP)
+ * drm_gpusvm_range_evict(gpusvm,
range);
+ * goto retry;
+ * } else if (err) {
+ * goto unlock;
+ * }
+ *
+ * err = driver_bind_range(gpusvm, range);
+ * if (err == -EAGAIN) // CPU mappings changed
+ * goto retry
+ *
+ * unlock:
+ * driver_svm_unlock();
+ * return err;
+ * }
+ *
+ * 2) Garbage Collector.
+ *
+ * void __driver_garbage_collector(struct drm_gpusvm *gpusvm,
+ * struct drm_gpusvm_range
*range)
+ * {
+ * assert_driver_svm_locked(gpusvm);
+ *
+ * // Partial unmap, migrate any remaining device
memory pages back to RAM
+ * if (range->flags.partial_unmap)
+ * drm_gpusvm_range_evict(gpusvm, range);
+ *
+ * driver_unbind_range(range);
+ * drm_gpusvm_range_remove(gpusvm, range);
+ * }
+ *
+ * void driver_garbage_collector(struct drm_gpusvm *gpusvm)
+ * {
+ * assert_driver_svm_locked(gpusvm);
+ *
+ * for_each_range_in_garbage_collector(gpusvm, range)
+ * __driver_garbage_collector(gpusvm, range);
+ * }
+ *
+ * 3) Notifier callback.
+ *
+ * void driver_invalidation(struct drm_gpusvm *gpusvm,
+ * struct drm_gpusvm_notifier
*notifier,
+ * const struct mmu_notifier_range
*mmu_range)
+ * {
+ * struct drm_gpusvm_ctx ctx = { .in_notifier = true,
};
+ * struct drm_gpusvm_range *range = NULL;
+ *
+ * driver_invalidate_device_pages(gpusvm, mmu_range-

Quoted 1 line

start, mmu_range->end);

+ *
+ * drm_gpusvm_for_each_range(range, notifier,
mmu_range->start,
+ *   mmu_range->end) {
+ * drm_gpusvm_range_unmap_pages(gpusvm, range,
&ctx);
+ *
+ * if (mmu_range->event != MMU_NOTIFY_UNMAP)
+ * continue;
+ *
+ * drm_gpusvm_range_set_unmapped(range,
mmu_range);
+ * driver_garbage_collector_add(gpusvm, range);
+ * }
+ * }
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+
+/**
+ * npages_in_range() - Calculate the number of pages in a given

range
+ * @start: The start address of the range
+ * @end: The end address of the range
+ *
+ * This macro calculates the number of pages in a given memory range,
+ * specified by the start and end addresses. It divides the difference
+ * between the end and start addresses by the page size (PAGE_SIZE) to
+ * determine the number of pages in the range.
+ *
+ * Returns: The number of pages in the specified range.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+static unsigned long
+npages_in_range(unsigned long start, unsigned long end)
+{
+	return (end - start) >> PAGE_SHIFT;
+}
+
+/**
+ * struct drm_gpusvm_zdd - GPU SVM zone device data
+ *
+ * @refcount: Reference count for the zdd
+ * @devmem_allocation: device memory allocation
+ * @device_private_page_owner: Device private pages owner
+ *
+ * This structure serves as a generic wrapper installed in
+ * page->zone_device_data. It provides infrastructure for looking up

a device
+ * memory allocation upon CPU page fault and asynchronously releasing device
+ * memory once the CPU has no page references. Asynchronous release is useful
+ * because CPU page references can be dropped in IRQ contexts, while releasing
+ * device memory likely requires sleeping locks.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+struct drm_gpusvm_zdd {
+	struct kref refcount;
+	struct drm_gpusvm_devmem *devmem_allocation;
+	void *device_private_page_owner;
+};
+
+/**
+ * drm_gpusvm_zdd_alloc() - Allocate a zdd structure.
+ * @device_private_page_owner: Device private pages owner
+ *
+ * This function allocates and initializes a new zdd structure. It

sets up the
+ * reference count and initializes the destroy work.
+ *
+ * Returns:
+ * Pointer to the allocated zdd on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +21 -0
+static struct drm_gpusvm_zdd *
+drm_gpusvm_zdd_alloc(void *device_private_page_owner)
+{
+	struct drm_gpusvm_zdd *zdd;
+
+	zdd = kmalloc(sizeof(*zdd), GFP_KERNEL);
+	if (!zdd)
+		return NULL;
+
+	kref_init(&zdd->refcount);
+	zdd->devmem_allocation = NULL;
+	zdd->device_private_page_owner = device_private_page_owner;
+
+	return zdd;
+}
+
+/**
+ * drm_gpusvm_zdd_get() - Get a reference to a zdd structure.
+ * @zdd: Pointer to the zdd structure.
+ *
+ * This function increments the reference count of the provided zdd

structure.
+ *
+ * Returns: Pointer to the zdd structure.
+ */
+static struct drm_gpusvm_zdd *drm_gpusvm_zdd_get(struct drm_gpusvm_zdd *zdd)

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+{
+	kref_get(&zdd->refcount);
+	return zdd;
+}
+
+/**
+ * drm_gpusvm_zdd_destroy() - Destroy a zdd structure.
+ * @ref: Pointer to the reference count structure.
+ *
+ * This function queues the destroy_work of the zdd for asynchronous

destruction.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +19 -0
+static void drm_gpusvm_zdd_destroy(struct kref *ref)
+{
+	struct drm_gpusvm_zdd *zdd =
+		container_of(ref, struct drm_gpusvm_zdd, refcount);
+	struct drm_gpusvm_devmem *devmem = zdd->devmem_allocation;
+
+	if (devmem) {
+		complete_all(&devmem->detached);
+		if (devmem->ops->devmem_release)
+			devmem->ops->devmem_release(devmem);
+	}
+	kfree(zdd);
+}
+
+/**
+ * drm_gpusvm_zdd_put() - Put a zdd reference.
+ * @zdd: Pointer to the zdd structure.
+ *
+ * This function decrements the reference count of the provided zdd

structure
+ * and schedules its destruction if the count drops to zero.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+static void drm_gpusvm_zdd_put(struct drm_gpusvm_zdd *zdd)
+{
+	kref_put(&zdd->refcount, drm_gpusvm_zdd_destroy);
+}
+
+/**
+ * drm_gpusvm_range_find() - Find GPU SVM range from GPU SVM

notifier
+ * @notifier: Pointer to the GPU SVM notifier structure.
+ * @start: Start address of the range
+ * @end: End address of the range
+ *
+ * Returns: A pointer to the drm_gpusvm_range if found or NULL
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, unsigned

long start,

drivers/gpu/drm/drm_gpusvm.c new file +8 -1
+		      unsigned long end)
+{
+	struct interval_tree_node *itree;
+
+	itree = interval_tree_iter_first(&notifier->root, start, end
- 1);
+
+	if (itree)
+		return container_of(itree, struct drm_gpusvm_range,

itree);

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+	else
+		return NULL;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_find);
+
+/**
+ * drm_gpusvm_for_each_range_safe() - Safely iterate over GPU SVM

ranges in a notifier
+ * @range__: Iterator variable for the ranges
+ * @next__: Iterator variable for the ranges temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier while
+ * removing ranges from it.
+ */
+#define drm_gpusvm_for_each_range_safe(range__, next__, notifier__,
start__, end__) \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	for ((range__) = drm_gpusvm_range_find((notifier__),

(start__), (end__)), \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (next__) =

__drm_gpusvm_range_next(range__); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (range__) && (range__->itree.start <

(end__)); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (range__) = (next__), (next__) =

__drm_gpusvm_range_next(range__))

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+
+/**
+ * __drm_gpusvm_notifier_next() - get the next drm_gpusvm_notifier

in the list
+ * @notifier: a pointer to the current drm_gpusvm_notifier
+ *
+ * Returns: A pointer to the next drm_gpusvm_notifier if available, or NULL if
+ *         the current notifier is the last one or if the input notifier is
+ *         NULL.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+static struct drm_gpusvm_notifier *
+__drm_gpusvm_notifier_next(struct drm_gpusvm_notifier *notifier)
+{
+	if (notifier && !list_is_last(&notifier->entry,
+				      &notifier->gpusvm-
Quoted 1 line

notifier_list))

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+		return list_next_entry(notifier, entry);
+
+	return NULL;
+}
+
+static struct drm_gpusvm_notifier *
+notifier_iter_first(struct rb_root_cached *root, unsigned long

start,

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+		    unsigned long last)
+{
+	struct interval_tree_node *itree;
+
+	itree = interval_tree_iter_first(root, start, last);
+
+	if (itree)
+		return container_of(itree, struct

drm_gpusvm_notifier, itree);

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+	else
+		return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_notifier() - Iterate over GPU SVM notifiers

in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm.
+ */
+#define drm_gpusvm_for_each_notifier(notifier__, gpusvm__, start__,
end__) \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,

(start__), (end__) - 1); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (notifier__) && (notifier__->itree.start <

(end__)); \

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+	     (notifier__) = __drm_gpusvm_notifier_next(notifier__))
+
+/**
+ * drm_gpusvm_for_each_notifier_safe() - Safely iterate over GPU SVM

notifiers in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @next__: Iterator variable for the notifiers temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm while
+ * removing notifiers from it.
+ */
+#define drm_gpusvm_for_each_notifier_safe(notifier__, next__,
gpusvm__, start__, end__) \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,

(start__), (end__) - 1), \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (next__) =

__drm_gpusvm_notifier_next(notifier__); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (notifier__) && (notifier__->itree.start <

(end__)); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (notifier__) = (next__), (next__) =

__drm_gpusvm_notifier_next(notifier__))

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+
+/**
+ * drm_gpusvm_notifier_invalidate() - Invalidate a GPU SVM notifier.
+ * @mni: Pointer to the mmu_interval_notifier structure.
+ * @mmu_range: Pointer to the mmu_notifier_range structure.
+ * @cur_seq: Current sequence number.
+ *
+ * This function serves as a generic MMU notifier for GPU SVM. It

sets the MMU
+ * notifier sequence number and calls the driver invalidate vfunc under
+ * gpusvm->notifier_lock.
+ *
+ * Returns:
+ * true if the operation succeeds, false otherwise.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+static bool
+drm_gpusvm_notifier_invalidate(struct mmu_interval_notifier *mni,
+			       const struct mmu_notifier_range

*mmu_range,

drivers/gpu/drm/drm_gpusvm.c new file +19 -0
+			       unsigned long cur_seq)
+{
+	struct drm_gpusvm_notifier *notifier =
+		container_of(mni, typeof(*notifier), notifier);
+	struct drm_gpusvm *gpusvm = notifier->gpusvm;
+
+	if (!mmu_notifier_range_blockable(mmu_range))
+		return false;
+
+	down_write(&gpusvm->notifier_lock);
+	mmu_interval_set_seq(mni, cur_seq);
+	gpusvm->ops->invalidate(gpusvm, notifier, mmu_range);
+	up_write(&gpusvm->notifier_lock);
+
+	return true;
+}
+
+/**
+ * drm_gpusvm_notifier_ops - MMU interval notifier operations for

GPU SVM
+ */
+static const struct mmu_interval_notifier_ops
drm_gpusvm_notifier_ops = {

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+	.invalidate = drm_gpusvm_notifier_invalidate,
+};
+
+/**
+ * drm_gpusvm_init() - Initialize the GPU SVM.
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @name: Name of the GPU SVM.
+ * @drm: Pointer to the DRM device structure.
+ * @mm: Pointer to the mm_struct for the address space.
+ * @device_private_page_owner: Device private pages owner.
+ * @mm_start: Start address of GPU SVM.
+ * @mm_range: Range of the GPU SVM.
+ * @notifier_size: Size of individual notifiers.
+ * @ops: Pointer to the operations structure for GPU SVM.
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range

allocation.
+ *               Entries should be powers of 2 in descending order with last
+ *               entry being SZ_4K.
+ * @num_chunks: Number of chunks.
+ *
+ * This function initializes the GPU SVM.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void

*device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+		    unsigned long mm_start, unsigned long mm_range,
+		    unsigned long notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const unsigned long *chunk_sizes, int

num_chunks)

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+{
+	if (!ops->invalidate || !num_chunks)
+		return -EINVAL;
+
+	gpusvm->name = name;
+	gpusvm->drm = drm;
+	gpusvm->mm = mm;
+	gpusvm->device_private_page_owner =

device_private_page_owner;

drivers/gpu/drm/drm_gpusvm.c new file +31 -0
+	gpusvm->mm_start = mm_start;
+	gpusvm->mm_range = mm_range;
+	gpusvm->notifier_size = notifier_size;
+	gpusvm->ops = ops;
+	gpusvm->chunk_sizes = chunk_sizes;
+	gpusvm->num_chunks = num_chunks;
+
+	mmgrab(mm);
+	gpusvm->root = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&gpusvm->notifier_list);
+
+	init_rwsem(&gpusvm->notifier_lock);
+
+	fs_reclaim_acquire(GFP_KERNEL);
+	might_lock(&gpusvm->notifier_lock);
+	fs_reclaim_release(GFP_KERNEL);
+
+#ifdef CONFIG_LOCKDEP
+	gpusvm->lock_dep_map = NULL;
+#endif
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_init);
+
+/**
+ * drm_gpusvm_notifier_find() - Find GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ *
+ * This function finds the GPU SVM notifier associated with the

fault address.
+ *
+ * Returns:
+ * Pointer to the GPU SVM notifier on success, NULL otherwise.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+static struct drm_gpusvm_notifier *
+drm_gpusvm_notifier_find(struct drm_gpusvm *gpusvm,
+			 unsigned long fault_addr)
+{
+	return notifier_iter_first(&gpusvm->root, fault_addr,

fault_addr + 1);

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+}
+
+/**
+ * to_drm_gpusvm_notifier() - retrieve the container struct for a

given rbtree node
+ * @node: a pointer to the rbtree node embedded within a drm_gpusvm_notifier struct
+ *
+ * Returns: A pointer to the containing drm_gpusvm_notifier structure.
+ */
+static struct drm_gpusvm_notifier *to_drm_gpusvm_notifier(struct rb_node *node)

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+{
+	return container_of(node, struct drm_gpusvm_notifier,

itree.rb);

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+}
+
+/**
+ * drm_gpusvm_notifier_insert() - Insert GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function inserts the GPU SVM notifier into the GPU SVM RB

tree and list.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static void drm_gpusvm_notifier_insert(struct drm_gpusvm *gpusvm,
+				       struct drm_gpusvm_notifier

*notifier)

drivers/gpu/drm/drm_gpusvm.c new file +21 -0
+{
+	struct rb_node *node;
+	struct list_head *head;
+
+	interval_tree_insert(&notifier->itree, &gpusvm->root);
+
+	node = rb_prev(&notifier->itree.rb);
+	if (node)
+		head = &(to_drm_gpusvm_notifier(node))->entry;
+	else
+		head = &gpusvm->notifier_list;
+
+	list_add(&notifier->entry, head);
+}
+
+/**
+ * drm_gpusvm_notifier_remove() - Remove GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM tructure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function removes the GPU SVM notifier from the GPU SVM RB

tree and list.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static void drm_gpusvm_notifier_remove(struct drm_gpusvm *gpusvm,
+				       struct drm_gpusvm_notifier

*notifier)

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+{
+	interval_tree_remove(&notifier->itree, &gpusvm->root);
+	list_del(&notifier->entry);
+}
+
+/**
+ * drm_gpusvm_fini() - Finalize the GPU SVM.
+ * @gpusvm: Pointer to the GPU SVM structure.
+ *
+ * This function finalizes the GPU SVM by cleaning up any remaining

ranges and
+ * notifiers, and dropping a reference to struct MM.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm)
+{
+	struct drm_gpusvm_notifier *notifier, *next;
+
+	drm_gpusvm_for_each_notifier_safe(notifier, next, gpusvm, 0,

LONG_MAX) {

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+		struct drm_gpusvm_range *range, *__next;
+
+		/*
+		 * Remove notifier first to avoid racing with any

invalidation

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+		 */
+		mmu_interval_notifier_remove(&notifier->notifier);
+		notifier->flags.removed = true;
+
+		drm_gpusvm_for_each_range_safe(range, __next,

notifier, 0,

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+					       LONG_MAX)
+			drm_gpusvm_range_remove(gpusvm, range);
+	}
+
+	mmdrop(gpusvm->mm);
+	WARN_ON(!RB_EMPTY_ROOT(&gpusvm->root.rb_root));
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_fini);
+
+/**
+ * drm_gpusvm_notifier_alloc() - Allocate GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ *
+ * This function allocates and initializes the GPU SVM notifier

structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM notifier on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static struct drm_gpusvm_notifier *
+drm_gpusvm_notifier_alloc(struct drm_gpusvm *gpusvm, unsigned long

fault_addr)

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	if (gpusvm->ops->notifier_alloc)
+		notifier = gpusvm->ops->notifier_alloc();
+	else
+		notifier = kzalloc(sizeof(*notifier), GFP_KERNEL);
+
+	if (!notifier)
+		return ERR_PTR(-ENOMEM);
+
+	notifier->gpusvm = gpusvm;
+	notifier->itree.start = ALIGN_DOWN(fault_addr, gpusvm-
Quoted 1 line

notifier_size);

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	notifier->itree.last = ALIGN(fault_addr + 1, gpusvm-
Quoted 1 line

notifier_size) - 1;

drivers/gpu/drm/drm_gpusvm.c new file +16 -0
+	INIT_LIST_HEAD(&notifier->entry);
+	notifier->root = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&notifier->range_list);
+
+	return notifier;
+}
+
+/**
+ * drm_gpusvm_notifier_free() - Free GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function frees the GPU SVM notifier structure.
+ */
+static void drm_gpusvm_notifier_free(struct drm_gpusvm *gpusvm,
+				     struct drm_gpusvm_notifier

*notifier)

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+{
+	WARN_ON(!RB_EMPTY_ROOT(&notifier->root.rb_root));
+
+	if (gpusvm->ops->notifier_free)
+		gpusvm->ops->notifier_free(notifier);
+	else
+		kfree(notifier);
+}
+
+/**
+ * to_drm_gpusvm_range() - retrieve the container struct for a given

rbtree node
+ * @node: a pointer to the rbtree node embedded within a drm_gpusvm_range struct
+ *
+ * Returns: A pointer to the containing drm_gpusvm_range structure.
+ */
+static struct drm_gpusvm_range *to_drm_gpusvm_range(struct rb_node *node)

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+{
+	return container_of(node, struct drm_gpusvm_range,

itree.rb);

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+}
+
+/**
+ * drm_gpusvm_range_insert() - Insert GPU SVM range
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function inserts the GPU SVM range into the notifier RB tree

and list.
+ */
+static void drm_gpusvm_range_insert(struct drm_gpusvm_notifier *notifier,

drivers/gpu/drm/drm_gpusvm.c new file +24 -0
+				    struct drm_gpusvm_range *range)
+{
+	struct rb_node *node;
+	struct list_head *head;
+
+	drm_gpusvm_notifier_lock(notifier->gpusvm);
+	interval_tree_insert(&range->itree, &notifier->root);
+
+	node = rb_prev(&range->itree.rb);
+	if (node)
+		head = &(to_drm_gpusvm_range(node))->entry;
+	else
+		head = &notifier->range_list;
+
+	list_add(&range->entry, head);
+	drm_gpusvm_notifier_unlock(notifier->gpusvm);
+}
+
+/**
+ * __drm_gpusvm_range_remove() - Remove GPU SVM range
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This macro removes the GPU SVM range from the notifier RB tree

and list.
+ */
+static void __drm_gpusvm_range_remove(struct drm_gpusvm_notifier *notifier,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				      struct drm_gpusvm_range

*range)

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+{
+	interval_tree_remove(&range->itree, &notifier->root);
+	list_del(&range->entry);
+}
+
+/**
+ * drm_gpusvm_range_alloc() - Allocate GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @fault_addr: Fault address
+ * @chunk_size: Chunk size
+ * @migrate_devmem: Flag indicating whether to migrate device memory
+ *
+ * This function allocates and initializes the GPU SVM range

structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM range on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+static struct drm_gpusvm_range *
+drm_gpusvm_range_alloc(struct drm_gpusvm *gpusvm,
+		       struct drm_gpusvm_notifier *notifier,
+		       unsigned long fault_addr, unsigned long

chunk_size,

drivers/gpu/drm/drm_gpusvm.c new file +32 -0
+		       bool migrate_devmem)
+{
+	struct drm_gpusvm_range *range;
+
+	if (gpusvm->ops->range_alloc)
+		range = gpusvm->ops->range_alloc(gpusvm);
+	else
+		range = kzalloc(sizeof(*range), GFP_KERNEL);
+
+	if (!range)
+		return ERR_PTR(-ENOMEM);
+
+	kref_init(&range->refcount);
+	range->gpusvm = gpusvm;
+	range->notifier = notifier;
+	range->itree.start = ALIGN_DOWN(fault_addr, chunk_size);
+	range->itree.last = ALIGN(fault_addr + 1, chunk_size) - 1;
+	INIT_LIST_HEAD(&range->entry);
+	range->notifier_seq = LONG_MAX;
+	range->flags.migrate_devmem = migrate_devmem ? 1 : 0;
+
+	return range;
+}
+
+/**
+ * drm_gpusvm_check_pages() - Check pages
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @start: Start address
+ * @end: End address
+ *
+ * Check if pages between start and end have been faulted in on the

CPU. Use to
+ * prevent migration of pages without CPU backing store.
+ *
+ * Returns:
+ * True if pages have been faulted into CPU, False otherwise
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static bool drm_gpusvm_check_pages(struct drm_gpusvm *gpusvm,
+				   struct drm_gpusvm_notifier

*notifier,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				   unsigned long start, unsigned

long end)

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+{
+	struct hmm_range hmm_range = {
+		.default_flags = 0,
+		.notifier = &notifier->notifier,
+		.start = start,
+		.end = end,
+		.dev_private_owner = gpusvm-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+	};
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+	unsigned long *pfns;
+	unsigned long npages = npages_in_range(start, end);
+	int err, i;
+
+	mmap_assert_locked(gpusvm->mm);
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL);
+	if (!pfns)
+		return false;
+
+	hmm_range.notifier_seq = mmu_interval_read_begin(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+	hmm_range.hmm_pfns = pfns;
+
+	while (true) {
+		err = hmm_range_fault(&hmm_range);
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq =
+				mmu_interval_read_begin(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/drm_gpusvm.c new file +22 -0
+			continue;
+		}
+		break;
+	}
+	if (err)
+		goto err_free;
+
+	for (i = 0; i < npages;) {
+		if (!(pfns[i] & HMM_PFN_VALID)) {
+			err = -EFAULT;
+			goto err_free;
+		}
+		i += 0x1 << hmm_pfn_to_map_order(pfns[i]);
+	}
+
+err_free:
+	kvfree(pfns);
+	return err ? false : true;
+}
+
+/**
+ * drm_gpusvm_range_chunk_size() - Determine chunk size for GPU SVM

range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @vas: Pointer to the virtual memory area structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @check_pages_threshold: Check CPU pages for present threshold
+ *
+ * This function determines the chunk size for the GPU SVM range based on the
+ * fault address, GPU SVM chunk sizes, existing GPU SVM ranges, and the virtual
+ * memory area boundaries.
+ *
+ * Returns:
+ * Chunk size on success, LONG_MAX on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+static unsigned long
+drm_gpusvm_range_chunk_size(struct drm_gpusvm *gpusvm,
+			    struct drm_gpusvm_notifier *notifier,
+			    struct vm_area_struct *vas,
+			    unsigned long fault_addr,
+			    unsigned long gpuva_start,
+			    unsigned long gpuva_end,
+			    unsigned long check_pages_threshold)
+{
+	unsigned long start, end;
+	int i = 0;
+
+retry:
+	for (; i < gpusvm->num_chunks; ++i) {
+		start = ALIGN_DOWN(fault_addr, gpusvm-
Quoted 1 line

chunk_sizes[i]);

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+		end = ALIGN(fault_addr + 1, gpusvm->chunk_sizes[i]);
+
+		if (start >= vas->vm_start && end <= vas->vm_end &&
+		    start >= notifier->itree.start &&
+		    end <= notifier->itree.last + 1 &&
+		    start >= gpuva_start && end <= gpuva_end)
+			break;
+	}
+
+	if (i == gpusvm->num_chunks)
+		return LONG_MAX;
+
+	/*
+	 * If allocation more than page, ensure not to overlap with

existing

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+	 * ranges.
+	 */
+	if (end - start != SZ_4K) {
+		struct drm_gpusvm_range *range;
+
+		range = drm_gpusvm_range_find(notifier, start, end);
+		if (range) {
+			++i;
+			goto retry;
+		}
+
+		/*
+		 * XXX: Only create range on pages CPU has faulted

in. Without

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * this check, or prefault, on BMG

'xe_exec_system_allocator --r

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * process-many-malloc' fails. In the failure case,

each process

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * mallocs 16k but the CPU VMA is ~128k which

results in 64k SVM

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * ranges. When migrating the SVM ranges, some

processes fail in

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * drm_gpusvm_migrate_to_devmem with 'migrate.cpages

!= npages'

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * and then upon drm_gpusvm_range_get_pages device

pages from

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * other processes are collected + faulted in which

creates all

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * sorts of problems. Unsure exactly how this

happening, also

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * problem goes away if 'xe_exec_system_allocator --

r

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * process-many-malloc' mallocs at least 64k at a

time.

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+		 */
+		if (end - start <= check_pages_threshold &&
+		    !drm_gpusvm_check_pages(gpusvm, notifier, start,

end)) {

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+			++i;
+			goto retry;
+		}
+	}
+
+	return end - start;
+}
+
+/**
+ * drm_gpusvm_range_find_or_insert() - Find or insert GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @ctx: GPU SVM context
+ *
+ * This function finds or inserts a newly allocated a GPU SVM range

based on the
+ * fault address. Caller must hold a lock to protect range lookup and insertion.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +28 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm,
+				unsigned long fault_addr,
+				unsigned long gpuva_start,
+				unsigned long gpuva_end,
+				const struct drm_gpusvm_ctx *ctx)
+{
+	struct drm_gpusvm_notifier *notifier;
+	struct drm_gpusvm_range *range;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	bool notifier_alloc = false;
+	unsigned long chunk_size;
+	int err;
+	bool migrate_devmem;
+
+	drm_gpusvm_driver_lock_held(gpusvm);
+
+	if (fault_addr < gpusvm->mm_start ||
+	    fault_addr > gpusvm->mm_start + gpusvm->mm_range)
+		return ERR_PTR(-EINVAL);
+
+	if (!mmget_not_zero(mm))
+		return ERR_PTR(-EFAULT);
+
+	notifier = drm_gpusvm_notifier_find(gpusvm, fault_addr);
+	if (!notifier) {
+		notifier = drm_gpusvm_notifier_alloc(gpusvm,

fault_addr);

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+		if (IS_ERR(notifier)) {
+			err = PTR_ERR(notifier);
+			goto err_mmunlock;
+		}
+		notifier_alloc = true;
+		err = mmu_interval_notifier_insert(&notifier-
Quoted 1 line

notifier,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						   mm, notifier-
Quoted 1 line

itree.start,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						   notifier-
Quoted 1 line

itree.last -

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						   notifier-
Quoted 1 line

itree.start + 1,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						  

&drm_gpusvm_notifier_ops);

drivers/gpu/drm/drm_gpusvm.c new file +18 -0
+		if (err)
+			goto err_notifier;
+	}
+
+	mmap_read_lock(mm);
+
+	vas = vma_lookup(mm, fault_addr);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_notifier_remove;
+	}
+
+	if (!ctx->read_only && !(vas->vm_flags & VM_WRITE)) {
+		err = -EPERM;
+		goto err_notifier_remove;
+	}
+
+	range = drm_gpusvm_range_find(notifier, fault_addr,

fault_addr + 1);

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+	if (range)
+		goto out_mmunlock;
+	/*
+	 * XXX: Short-circuiting migration based on migrate_vma_*

current

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	 * limitations. If/when migrate_vma_* add more support, this

logic will

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+	 * have to change.
+	 */
+	migrate_devmem = ctx->devmem_possible &&
+		vma_is_anonymous(vas) && !is_vm_hugetlb_page(vas);
+
+	chunk_size = drm_gpusvm_range_chunk_size(gpusvm, notifier,

vas,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						 fault_addr,

gpuva_start,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+						 gpuva_end,
+						 ctx-
Quoted 1 line

check_pages_threshold);

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+	if (chunk_size == LONG_MAX) {
+		err = -EINVAL;
+		goto err_notifier_remove;
+	}
+
+	range = drm_gpusvm_range_alloc(gpusvm, notifier, fault_addr,

chunk_size,

drivers/gpu/drm/drm_gpusvm.c new file +31 -0
+				       migrate_devmem);
+	if (IS_ERR(range)) {
+		err = PTR_ERR(range);
+		goto err_notifier_remove;
+	}
+
+	drm_gpusvm_range_insert(notifier, range);
+	if (notifier_alloc)
+		drm_gpusvm_notifier_insert(gpusvm, notifier);
+
+out_mmunlock:
+	mmap_read_unlock(mm);
+	mmput(mm);
+
+	return range;
+
+err_notifier_remove:
+	mmap_read_unlock(mm);
+	if (notifier_alloc)
+		mmu_interval_notifier_remove(&notifier->notifier);
+err_notifier:
+	if (notifier_alloc)
+		drm_gpusvm_notifier_free(gpusvm, notifier);
+err_mmunlock:
+	mmput(mm);
+	return ERR_PTR(err);
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_find_or_insert);
+
+/**
+ * __drm_gpusvm_range_unmap_pages() - Unmap pages associated with a

GPU SVM range (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @npages: Number of pages to unmap
+ *
+ * This function unmap pages associated with a GPU SVM range. Assumes and
+ * asserts correct locking is in place when called.
+ */
+static void __drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					   struct drm_gpusvm_range

*range,

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+					   unsigned long npages)
+{
+	unsigned long i, j;
+	struct drm_pagemap *dpagemap = range->dpagemap;
+	struct device *dev = gpusvm->drm->dev;
+
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	if (range->flags.has_dma_mapping) {
+		for (i = 0, j = 0; i < npages; j++) {
+			struct drm_pagemap_dma_addr *addr = &range-
Quoted 1 line

dma_addr[j];

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+			if (addr->proto == DRM_INTERCONNECT_SYSTEM)
+				dma_unmap_page(dev,
+					       addr->addr,
+					       PAGE_SIZE << addr-
Quoted 1 line

order,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					       addr->dir);
+			else if (dpagemap && dpagemap->ops-
Quoted 1 line

unmap_dma)

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+				dpagemap->ops->unmap_dma(dpagemap,
+							 dev,
+							 *addr);
+			i += 1 << addr->order;
+		}
+		range->flags.has_devmem_pages = false;
+		range->flags.has_dma_mapping = false;
+		range->dpagemap = NULL;
+	}
+}
+
+/**
+ * drm_gpusvm_range_free_pages() - Free pages associated with a GPU

SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function frees the dma address array associated with a GPU SVM range.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static void drm_gpusvm_range_free_pages(struct drm_gpusvm *gpusvm,
+					struct drm_gpusvm_range

*range)

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+{
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	if (range->dma_addr) {
+		kvfree(range->dma_addr);
+		range->dma_addr = NULL;
+	}
+}
+
+/**
+ * drm_gpusvm_range_remove() - Remove GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range to be removed
+ *
+ * This function removes the specified GPU SVM range and also

removes the parent
+ * GPU SVM notifier if no more ranges remain in the notifier. The caller must
+ * hold a lock to protect range and notifier removal.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range)
+{
+	unsigned long npages = npages_in_range(range->itree.start,
+					       range->itree.last +

1);

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+	struct drm_gpusvm_notifier *notifier;
+
+	drm_gpusvm_driver_lock_held(gpusvm);
+
+	notifier = drm_gpusvm_notifier_find(gpusvm, range-
Quoted 1 line

itree.start);

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+	if (WARN_ON_ONCE(!notifier))
+		return;
+
+	drm_gpusvm_notifier_lock(gpusvm);
+	__drm_gpusvm_range_unmap_pages(gpusvm, range, npages);
+	drm_gpusvm_range_free_pages(gpusvm, range);
+	__drm_gpusvm_range_remove(notifier, range);
+	drm_gpusvm_notifier_unlock(gpusvm);
+
+	drm_gpusvm_range_put(range);
+
+	if (RB_EMPTY_ROOT(&notifier->root.rb_root)) {
+		if (!notifier->flags.removed)
+			mmu_interval_notifier_remove(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+		drm_gpusvm_notifier_remove(gpusvm, notifier);
+		drm_gpusvm_notifier_free(gpusvm, notifier);
+	}
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_remove);
+
+/**
+ * drm_gpusvm_range_get() - Get a reference to GPU SVM range
+ * @range: Pointer to the GPU SVM range
+ *
+ * This function increments the reference count of the specified GPU

SVM range.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range)
+{
+	kref_get(&range->refcount);
+
+	return range;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_get);
+
+/**
+ * drm_gpusvm_range_destroy() - Destroy GPU SVM range
+ * @refcount: Pointer to the reference counter embedded in the GPU

SVM range
+ *
+ * This function destroys the specified GPU SVM range when its reference count
+ * reaches zero. If a custom range-free function is provided, it is invoked to
+ * free the range; otherwise, the range is deallocated using kfree().
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_range_destroy(struct kref *refcount)
+{
+	struct drm_gpusvm_range *range =
+		container_of(refcount, struct drm_gpusvm_range,

refcount);

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+	struct drm_gpusvm *gpusvm = range->gpusvm;
+
+	if (gpusvm->ops->range_free)
+		gpusvm->ops->range_free(range);
+	else
+		kfree(range);
+}
+
+/**
+ * drm_gpusvm_range_put() - Put a reference to GPU SVM range
+ * @range: Pointer to the GPU SVM range
+ *
+ * This function decrements the reference count of the specified GPU

SVM range
+ * and frees it when the count reaches zero.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range)
+{
+	kref_put(&range->refcount, drm_gpusvm_range_destroy);
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_put);
+
+/**
+ * drm_gpusvm_range_pages_valid() - GPU SVM range pages valid
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid.

Expected be
+ * called holding gpusvm->notifier_lock and as the last step before committing a
+ * GPU binding. This is akin to a notifier seqno check in the HMM documentation
+ * but due to wider notifiers (i.e., notifiers which span multiple ranges) this
+ * function is required for finer grained checking (i.e., per range) if pages
+ * are valid.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range)
+{
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	return range->flags.has_devmem_pages || range-
Quoted 1 line

flags.has_dma_mapping;

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_pages_valid);
+
+/**
+ * drm_gpusvm_range_pages_valid_unlocked() - GPU SVM range pages

valid unlocked
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid. Expected be
+ * called without holding gpusvm->notifier_lock.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+static bool
+drm_gpusvm_range_pages_valid_unlocked(struct drm_gpusvm *gpusvm,
+				      struct drm_gpusvm_range

*range)

drivers/gpu/drm/drm_gpusvm.c new file +22 -0
+{
+	bool pages_valid;
+
+	if (!range->dma_addr)
+		return false;
+
+	drm_gpusvm_notifier_lock(gpusvm);
+	pages_valid = drm_gpusvm_range_pages_valid(gpusvm, range);
+	if (!pages_valid)
+		drm_gpusvm_range_free_pages(gpusvm, range);
+	drm_gpusvm_notifier_unlock(gpusvm);
+
+	return pages_valid;
+}
+
+/**
+ * drm_gpusvm_range_get_pages() - Get pages for a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function gets pages for a GPU SVM range and ensures they are

mapped for
+ * DMA access.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	struct mmu_interval_notifier *notifier = &range->notifier-
Quoted 1 line

notifier;

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+	struct hmm_range hmm_range = {
+		.default_flags = HMM_PFN_REQ_FAULT | (ctx->read_only

? 0 :

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+			HMM_PFN_REQ_WRITE),
+		.notifier = notifier,
+		.start = range->itree.start,
+		.end = range->itree.last + 1,
+		.dev_private_owner = gpusvm-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	struct drm_gpusvm_zdd *zdd;
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+	unsigned long i, j;
+	unsigned long npages = npages_in_range(range->itree.start,
+					       range->itree.last +

1);

drivers/gpu/drm/drm_gpusvm.c new file +58 -0
+	unsigned long num_dma_mapped;
+	unsigned int order = 0;
+	unsigned long *pfns;
+	struct page **pages;
+	int err = 0;
+	struct dev_pagemap *pagemap;
+	struct drm_pagemap *dpagemap;
+
+retry:
+	hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
+	if (drm_gpusvm_range_pages_valid_unlocked(gpusvm, range))
+		goto set_seqno;
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL);
+	if (!pfns)
+		return -ENOMEM;
+
+	if (!mmget_not_zero(mm)) {
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	hmm_range.hmm_pfns = pfns;
+	while (true) {
+		mmap_read_lock(mm);
+		err = hmm_range_fault(&hmm_range);
+		mmap_read_unlock(mm);
+
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq =
+				mmu_interval_read_begin(notifier);
+			continue;
+		}
+		break;
+	}
+	mmput(mm);
+	if (err)
+		goto err_free;
+
+	pages = (struct page **)pfns;
+map_pages:
+	/*
+	 * Perform all dma mappings under the notifier lock to not
+	 * access freed pages. A notifier will either block on
+	 * the notifier lock or unmap dma.
+	 */
+	drm_gpusvm_notifier_lock(gpusvm);
+
+	if (range->flags.unmapped) {
+		drm_gpusvm_notifier_unlock(gpusvm);
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	if (mmu_interval_read_retry(notifier,

hmm_range.notifier_seq)) {

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+		drm_gpusvm_notifier_unlock(gpusvm);
+		kvfree(pfns);
+		goto retry;
+	}
+
+	if (!range->dma_addr) {
+		/* Unlock and restart mapping to allocate memory. */
+		drm_gpusvm_notifier_unlock(gpusvm);
+		range->dma_addr = kvmalloc_array(npages,
+						 sizeof(*range-
Quoted 1 line

dma_addr),

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+						 GFP_KERNEL);
+		if (!range->dma_addr) {
+			err = -ENOMEM;
+			goto err_free;
+		}
+		goto map_pages;
+	}
+
+	zdd = NULL;
+	num_dma_mapped = 0;
+	for (i = 0, j = 0; i < npages; ++j) {
+		struct page *page = hmm_pfn_to_page(pfns[i]);
+
+		order = hmm_pfn_to_map_order(pfns[i]);
+		if (is_device_private_page(page) ||
+		    is_device_coherent_page(page)) {
+			if (zdd != page->zone_device_data && i > 0)

{

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+				err = -EOPNOTSUPP;
+				goto err_unmap;
+			}
+			zdd = page->zone_device_data;
+			if (pagemap != page->pgmap) {
+				if (i > 0) {
+					err = -EOPNOTSUPP;
+					goto err_unmap;
+				}
+
+				pagemap = page->pgmap;
+				dpagemap = zdd->devmem_allocation-
Quoted 1 line

dpagemap;

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				if (drm_WARN_ON(gpusvm->drm,

!dpagemap)) {

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					/*
+					 * Raced. This is not

supposed to happen

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					 * since hmm_range_fault()

should've migrated

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+					 * this page to system.
+					 */
+					err = -EAGAIN;
+					goto err_unmap;
+				}
+			}
+			range->dma_addr[j] =
+				dpagemap->ops->map_dma(dpagemap,
+						       gpusvm->drm-
Quoted 1 line

dev,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+						       page, order,
+						      

DMA_BIDIRECTIONAL);

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+			if (dma_mapping_error(gpusvm->drm->dev,
+					      range-
Quoted 1 line

dma_addr[j].addr)) {

drivers/gpu/drm/drm_gpusvm.c new file +18 -0
+				err = -EFAULT;
+				goto err_unmap;
+			}
+
+			pages[i] = page;
+		} else {
+			dma_addr_t addr;
+
+			if (is_zone_device_page(page) || zdd) {
+				err = -EOPNOTSUPP;
+				goto err_unmap;
+			}
+
+			addr = dma_map_page(gpusvm->drm->dev,
+					    page, 0,
+					    PAGE_SIZE << order,
+					    DMA_BIDIRECTIONAL);
+			if (dma_mapping_error(gpusvm->drm->dev,

addr)) {

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+				err = -EFAULT;
+				goto err_unmap;
+			}
+
+			range->dma_addr[j] =

drm_pagemap_dma_addr_encode

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				(addr, DRM_INTERCONNECT_SYSTEM,

order,

drivers/gpu/drm/drm_gpusvm.c new file +21 -0
+				 DMA_BIDIRECTIONAL);
+		}
+		i += 1 << order;
+		num_dma_mapped = i;
+	}
+
+	range->flags.has_dma_mapping = true;
+	if (zdd) {
+		range->flags.has_devmem_pages = true;
+		range->dpagemap = dpagemap;
+	}
+
+	drm_gpusvm_notifier_unlock(gpusvm);
+	kvfree(pfns);
+set_seqno:
+	range->notifier_seq = hmm_range.notifier_seq;
+
+	return 0;
+
+err_unmap:
+	__drm_gpusvm_range_unmap_pages(gpusvm, range,

num_dma_mapped);

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+	drm_gpusvm_notifier_unlock(gpusvm);
+err_free:
+	kvfree(pfns);
+	if (err == -EAGAIN)
+		goto retry;
+	return err;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_get_pages);
+
+/**
+ * drm_gpusvm_range_unmap_pages() - Unmap pages associated with a

GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed that gpusvm->notifier_lock is held in write mode; if it
+ * is clear, it acquires gpusvm->notifier_lock in read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops-

Quoted 1 line

invalidate for IOMMU

+ * security model.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx)
+{
+	unsigned long npages = npages_in_range(range->itree.start,
+					       range->itree.last +

1);

drivers/gpu/drm/drm_gpusvm.c new file +33 -0
+
+	if (ctx->in_notifier)
+		lockdep_assert_held_write(&gpusvm->notifier_lock);
+	else
+		drm_gpusvm_notifier_lock(gpusvm);
+
+	__drm_gpusvm_range_unmap_pages(gpusvm, range, npages);
+
+	if (!ctx->in_notifier)
+		drm_gpusvm_notifier_unlock(gpusvm);
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_unmap_pages);
+
+/**
+ * drm_gpusvm_migration_unlock_put_page() - Put a migration page
+ * @page: Pointer to the page to put
+ *
+ * This function unlocks and puts a page.
+ */
+static void drm_gpusvm_migration_unlock_put_page(struct page *page)
+{
+	unlock_page(page);
+	put_page(page);
+}
+
+/**
+ * drm_gpusvm_migration_unlock_put_pages() - Put migration pages
+ * @npages: Number of pages
+ * @migrate_pfn: Array of migrate page frame numbers
+ *
+ * This function unlocks and puts an array of pages.
+ */
+static void drm_gpusvm_migration_unlock_put_pages(unsigned long

npages,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						  unsigned long

*migrate_pfn)

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page;
+
+		if (!migrate_pfn[i])
+			continue;
+
+		page = migrate_pfn_to_page(migrate_pfn[i]);
+		drm_gpusvm_migration_unlock_put_page(page);
+		migrate_pfn[i] = 0;
+	}
+}
+
+/**
+ * drm_gpusvm_get_devmem_page() - Get a reference to a device memory

page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU SVM zone
+ * device data and initializes it for zone device usage.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+static void drm_gpusvm_get_devmem_page(struct page *page,
+				     struct drm_gpusvm_zdd *zdd)
+{
+	page->zone_device_data = drm_gpusvm_zdd_get(zdd);
+	zone_device_page_init(page);
+}
+
+/**
+ * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM

migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Returns: 0 on success, -EFAULT if an error occurs during mapping.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+static int drm_gpusvm_migrate_map_pages(struct device *dev,
+					dma_addr_t *dma_addr,
+					unsigned long *migrate_pfn,
+					unsigned long npages,
+					enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page =

migrate_pfn_to_page(migrate_pfn[i]);

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+
+		if (!page)
+			continue;
+
+		if (WARN_ON_ONCE(is_zone_device_page(page)))
+			return -EFAULT;
+
+		dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE,

dir);

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+		if (dma_mapping_error(dev, dma_addr[i]))
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped

for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_migrate_unmap_pages(struct device *dev,
+					   dma_addr_t *dma_addr,
+					   unsigned long npages,
+					   enum dma_data_direction

dir)

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!dma_addr[i] || dma_mapping_error(dev,

dma_addr[i]))

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+			continue;
+
+		dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir);
+	}
+}
+
+/**
+ * drm_gpusvm_migrate_to_devmem() - Migrate GPU SVM range to device

memory
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @devmem_allocation: Pointer to the device memory allocation. The caller
+ *                     should hold a reference to the device memory allocation,
+ *                     which should be dropped via ops-

Quoted 1 line

devmem_release or upon

+ *                     the failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to device memory. It performs the
+ * necessary setup and invokes the driver-specific operations for migration to
+ * device memory. Upon successful return, @devmem_allocation can safely reference @range
+ * until ops->devmem_release is called which only upon successful return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+int drm_gpusvm_migrate_to_devmem(struct drm_gpusvm *gpusvm,
+				 struct drm_gpusvm_range *range,
+				 struct drm_gpusvm_devmem

*devmem_allocation,

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+				 const struct drm_gpusvm_ctx *ctx)
+{
+	const struct drm_gpusvm_devmem_ops *ops = devmem_allocation-
Quoted 1 line

ops;

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	unsigned long start = range->itree.start, end = range-
Quoted 1 line

itree.last + 1;

drivers/gpu/drm/drm_gpusvm.c new file +45 -0
+	struct migrate_vma migrate = {
+		.start		= start,
+		.end		= end,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_SYSTEM,
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	unsigned long i, npages = npages_in_range(start, end);
+	struct vm_area_struct *vas;
+	struct drm_gpusvm_zdd *zdd = NULL;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int err;
+
+	if (!range->flags.migrate_devmem)
+		return -EINVAL;
+
+	if (!ops->populate_devmem_pfn || !ops->copy_to_devmem ||
+	    !ops->copy_to_ram)
+		return -EOPNOTSUPP;
+
+	if (!mmget_not_zero(mm)) {
+		err = -EFAULT;
+		goto err_out;
+	}
+	mmap_read_lock(mm);
+
+	vas = vma_lookup(mm, start);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end > vas->vm_end || start < vas->vm_start) {
+		err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	if (!vma_is_anonymous(vas)) {
+		err = -EBUSY;
+		goto err_mmunlock;
+	}
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +

sizeof(*dma_addr) +

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_mmunlock;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr))

* npages;

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+
+	zdd = drm_gpusvm_zdd_alloc(gpusvm-
Quoted 1 line

device_private_page_owner);

drivers/gpu/drm/drm_gpusvm.c new file +24 -0
+	if (!zdd) {
+		err = -ENOMEM;
+		goto err_free;
+	}
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	if (!migrate.cpages) {
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	if (migrate.cpages != npages) {
+		err = -EBUSY;
+		goto err_finalize;
+	}
+
+	err = ops->populate_devmem_pfn(devmem_allocation, npages,

migrate.dst);

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(devmem_allocation->dev,

dma_addr,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					   migrate.src, npages,

DMA_TO_DEVICE);

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = pfn_to_page(migrate.dst[i]);
+
+		pages[i] = page;
+		migrate.dst[i] = migrate_pfn(migrate.dst[i]);
+		drm_gpusvm_get_devmem_page(page, zdd);
+	}
+
+	err = ops->copy_to_devmem(pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+	/* Upon success bind devmem allocation to range and zdd */
+	zdd->devmem_allocation = devmem_allocation;	/* Owns ref

*/

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_unlock_put_pages(npages,

migrate.dst);

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(devmem_allocation->dev,

dma_addr, npages,

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+				       DMA_TO_DEVICE);
+err_free:
+	if (zdd)
+		drm_gpusvm_zdd_put(zdd);
+	kvfree(buf);
+err_mmunlock:
+	mmap_read_unlock(mm);
+	mmput(mm);
+err_out:
+	return err;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_migrate_to_devmem);
+
+/**
+ * drm_gpusvm_migrate_populate_ram_pfn() - Populate RAM PFNs for a

VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @fault_page: Fault page
+ * @npages: Number of pages to populate
+ * @mpages: Number of pages to migrate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the RAM migrate page frame numbers (PFNs) for the
+ * specified VM area structure. It allocates and locks pages in the VM area for
+ * RAM usage. If vas is non-NULL use alloc_page_vma for allocation, if NULL use
+ * alloc_page for allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_ram_pfn(struct vm_area_struct *vas,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					       struct page

*fault_page,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					       unsigned long npages,
+					       unsigned long

*mpages,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					       unsigned long

*src_mpfn,

drivers/gpu/drm/drm_gpusvm.c new file +23 -0
+					       unsigned long *mpfn,
+					       unsigned long addr)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i, addr += PAGE_SIZE) {
+		struct page *page, *src_page;
+
+		if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE))
+			continue;
+
+		src_page = migrate_pfn_to_page(src_mpfn[i]);
+		if (!src_page)
+			continue;
+
+		if (fault_page) {
+			if (src_page->zone_device_data !=
+			    fault_page->zone_device_data)
+				continue;
+		}
+
+		if (vas)
+			page = alloc_page_vma(GFP_HIGHUSER, vas,

addr);

drivers/gpu/drm/drm_gpusvm.c new file +39 -0
+		else
+			page = alloc_page(GFP_HIGHUSER);
+
+		if (!page)
+			goto free_pages;
+
+		mpfn[i] = migrate_pfn(page_to_pfn(page));
+	}
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = migrate_pfn_to_page(mpfn[i]);
+
+		if (!page)
+			continue;
+
+		WARN_ON_ONCE(!trylock_page(page));
+		++*mpages;
+	}
+
+	return 0;
+
+free_pages:
+	for (i = 0; i < npages; ++i) {
+		struct page *page = migrate_pfn_to_page(mpfn[i]);
+
+		if (!page)
+			continue;
+
+		put_page(page);
+		mpfn[i] = 0;
+	}
+	return -ENOMEM;
+}
+
+/**
+ * drm_gpusvm_evict_to_ram() - Evict GPU SVM range to RAM
+ * @devmem_allocation: Pointer to the device memory allocation
+ *
+ * Similar to __drm_gpusvm_migrate_to_ram but does not require mmap

lock and
+ * migration done via migrate_device_* functions.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_evict_to_ram(struct drm_gpusvm_devmem *devmem_allocation)

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+{
+	const struct drm_gpusvm_devmem_ops *ops = devmem_allocation-
Quoted 1 line

ops;

drivers/gpu/drm/drm_gpusvm.c new file +25 -0
+	unsigned long npages, mpages = 0;
+	struct page **pages;
+	unsigned long *src, *dst;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+	unsigned int retry_count = 2;
+
+	npages = devmem_allocation->size >> PAGE_SHIFT;
+
+retry:
+	if (!mmget_not_zero(devmem_allocation->mm))
+		return -EFAULT;
+
+	buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr)
+
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	src = buf;
+	dst = buf + (sizeof(*src) * npages);
+	dma_addr = buf + (2 * sizeof(*src) * npages);
+	pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) *

npages;

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+
+	err = ops->populate_devmem_pfn(devmem_allocation, npages,

src);

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+	if (err)
+		goto err_free;
+
+	err = migrate_device_pfns(src, npages);
+	if (err)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_ram_pfn(NULL, NULL,

npages, &mpages,

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+						  src, dst, 0);
+	if (err || !mpages)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(devmem_allocation->dev,

dma_addr,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					   dst, npages,

DMA_FROM_DEVICE);

drivers/gpu/drm/drm_gpusvm.c new file +16 -0
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(src[i]);
+
+	err = ops->copy_to_ram(pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_unlock_put_pages(npages, dst);
+	migrate_device_pages(src, dst, npages);
+	migrate_device_finalize(src, dst, npages);
+	drm_gpusvm_migrate_unmap_pages(devmem_allocation->dev,

dma_addr, npages,

drivers/gpu/drm/drm_gpusvm.c new file +20 -0
+				       DMA_FROM_DEVICE);
+err_free:
+	kvfree(buf);
+err_out:
+	mmput_async(devmem_allocation->mm);
+
+	if (completion_done(&devmem_allocation->detached))
+		return 0;
+
+	if (!err || retry_count--) {
+		cond_resched();
+		goto retry;
+	}
+
+	return err;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_evict_to_ram);
+
+/**
+ * __drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM

(internal)
+ * @vas: Pointer to the VM area structure
+ * @device_private_page_owner: Device private pages owner
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @fault_addr: Fault address
+ * @size: Size of migration
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to RAM. It sets up the migration, populates + dma maps RAM PFNs, and
+ * invokes the driver-specific operations for migration to RAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static int __drm_gpusvm_migrate_to_ram(struct vm_area_struct *vas,
+				       void

*device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+				       struct page *page,
+				       unsigned long fault_addr,
+				       unsigned long size)
+{
+	struct migrate_vma migrate = {
+		.vma		= vas,
+		.pgmap_owner	= device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_DEVICE_PRIVATE

|

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+			MIGRATE_VMA_SELECT_DEVICE_COHERENT,
+		.fault_page	= page,
+	};
+	struct drm_gpusvm_zdd *zdd;
+	const struct drm_gpusvm_devmem_ops *ops;
+	struct device *dev;
+	unsigned long npages, mpages = 0;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	unsigned long start, end;
+	void *buf;
+	int i, err = 0;
+
+	start = ALIGN_DOWN(fault_addr, size);
+	end = ALIGN(fault_addr + 1, size);
+
+	/* Corner where VMA area struct has been partially unmapped

*/

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+	if (start < vas->vm_start)
+		start = vas->vm_start;
+	if (end > vas->vm_end)
+		end = vas->vm_end;
+
+	migrate.start = start;
+	migrate.end = end;
+	npages = npages_in_range(start, end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +

sizeof(*dma_addr) +

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr))

* npages;

drivers/gpu/drm/drm_gpusvm.c new file +30 -0
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/* Raced with another CPU fault, nothing to do */
+	if (!migrate.cpages)
+		goto err_free;
+
+	if (!page) {
+		for (i = 0; i < npages; ++i) {
+			if (!(migrate.src[i] & MIGRATE_PFN_MIGRATE))
+				continue;
+
+			page = migrate_pfn_to_page(migrate.src[i]);
+			break;
+		}
+
+		if (!page)
+			goto err_finalize;
+	}
+	zdd = page->zone_device_data;
+	ops = zdd->devmem_allocation->ops;
+	dev = zdd->devmem_allocation->dev;
+
+	err = drm_gpusvm_migrate_populate_ram_pfn(vas, page, npages,

&mpages,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						  migrate.src,

migrate.dst,

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+						  start);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(dev, dma_addr,

migrate.dst, npages,

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+					   DMA_FROM_DEVICE);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(migrate.src[i]);
+
+	err = ops->copy_to_ram(pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_unlock_put_pages(npages,

migrate.dst);

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(dev, dma_addr, npages,
+				       DMA_FROM_DEVICE);
+err_free:
+	kvfree(buf);
+err_out:
+
+	return err;
+}
+
+/**
+ * drm_gpusvm_range_evict - Evict GPU SVM range
+ * @pagemap: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range to be removed
+ *
+ * This function evicts the specified GPU SVM range. This function

will not
+ * evict coherent pages.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+int drm_gpusvm_range_evict(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_range *range)
+{
+	struct mmu_interval_notifier *notifier = &range->notifier-
Quoted 1 line

notifier;

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+	struct hmm_range hmm_range = {
+		.default_flags = HMM_PFN_REQ_FAULT,
+		.notifier = notifier,
+		.start = range->itree.start,
+		.end = range->itree.last + 1,
+		.dev_private_owner = NULL,
+	};
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+	unsigned long *pfns;
+	unsigned long npages = npages_in_range(range->itree.start,
+					       range->itree.last +

1);

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+	int err = 0;
+	struct mm_struct *mm = gpusvm->mm;
+
+	if (!mmget_not_zero(mm))
+		return -EFAULT;
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL);
+	if (!pfns)
+		return -ENOMEM;
+
+	hmm_range.hmm_pfns = pfns;
+	while (!time_after(jiffies, timeout)) {
+		hmm_range.notifier_seq =

mmu_interval_read_begin(notifier);

drivers/gpu/drm/drm_gpusvm.c new file +21 -0
+		if (time_after(jiffies, timeout)) {
+			err = -ETIME;
+			break;
+		}
+
+		mmap_read_lock(mm);
+		err = hmm_range_fault(&hmm_range);
+		mmap_read_unlock(mm);
+		if (err != -EBUSY)
+			break;
+	}
+
+	kvfree(pfns);
+	mmput(mm);
+
+	return err;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_evict);
+
+/**
+ * drm_gpusvm_page_free() - Put GPU SVM zone device data associated

with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device data
+ * associated with a page when it is being released.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+static void drm_gpusvm_page_free(struct page *page)
+{
+	drm_gpusvm_zdd_put(page->zone_device_data);
+}
+
+/**
+ * drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM (page

fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ * VM_FAULT_SIGBUS on failure, 0 on success.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+static vm_fault_t drm_gpusvm_migrate_to_ram(struct vm_fault *vmf)
+{
+	struct drm_gpusvm_zdd *zdd = vmf->page->zone_device_data;
+	int err;
+
+	err = __drm_gpusvm_migrate_to_ram(vmf->vma,
+					  zdd-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					  vmf->page, vmf->address,
+					  zdd->devmem_allocation-
Quoted 1 line

size);

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+
+	return err ? VM_FAULT_SIGBUS : 0;
+}
+
+/**
+ * drm_gpusvm_pagemap_ops() - Device page map operations for GPU SVM
+ */
+static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = {
+	.page_free = drm_gpusvm_page_free,
+	.migrate_to_ram = drm_gpusvm_migrate_to_ram,
+};
+
+/**
+ * drm_gpusvm_pagemap_ops_get() - Retrieve GPU SVM device page map

operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void)
+{
+	return &drm_gpusvm_pagemap_ops;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_pagemap_ops_get);
+
+/**
+ * drm_gpusvm_has_mapping() - Check if GPU SVM has mapping for the

given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, unsigned long start,

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+			    unsigned long end)
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	drm_gpusvm_for_each_notifier(notifier, gpusvm, start, end) {
+		struct drm_gpusvm_range *range = NULL;
+
+		drm_gpusvm_for_each_range(range, notifier, start,

end)

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+			return true;
+	}
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_has_mapping);
+
+/**
+ * drm_gpusvm_range_set_unmapped() - Mark a GPU SVM range as

unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets the partial_unmap flag
+ * if the range partially falls within the provided MMU notifier range.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+void drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,
+				   const struct mmu_notifier_range

*mmu_range)

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+{
+	lockdep_assert_held_write(&range->gpusvm->notifier_lock);
+
+	range->flags.unmapped = true;
+	if (range->itree.start < mmu_range->start ||
+	    range->itree.last + 1 > mmu_range->end)
+		range->flags.partial_unmap = true;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_set_unmapped);
+
+/**
+ * drm_gpusvm_devmem_init() - Initialize a GPU SVM device memory

allocation
+ *
+ * @dev: Pointer to the device structure which device memory allocation belongs to
+ * @mm: Pointer to the mm_struct for the address space
+ * @ops: Pointer to the operations structure for GPU SVM device memory
+ * @dpagemap: The struct drm_pagemap we're allocating from.
+ * @size: Size of device memory allocation
+ */
+void drm_gpusvm_devmem_init(struct drm_gpusvm_devmem *devmem_allocation,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+			    struct device *dev, struct mm_struct

*mm,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+			    const struct drm_gpusvm_devmem_ops *ops,
+			    struct drm_pagemap *dpagemap, size_t

size)

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+{
+	init_completion(&devmem_allocation->detached);
+	devmem_allocation->dev = dev;
+	devmem_allocation->mm = mm;
+	devmem_allocation->ops = ops;
+	devmem_allocation->dpagemap = dpagemap;
+	devmem_allocation->size = size;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_devmem_init);
+
+MODULE_DESCRIPTION("DRM GPUSVM");
+MODULE_LICENSE("GPL");
include/drm/drm_gpusvm.h new file +24 -0
new file mode 100644
index 000000000000..ea31db0be841
@@ -0,0 +1,445 @@
+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef __DRM_GPUSVM_H__
+#define __DRM_GPUSVM_H__
+
+#include <linux/kref.h>
+#include <linux/interval_tree.h>
+#include <linux/mmu_notifier.h>
+
+struct dev_pagemap_ops;
+struct drm_device;
+struct drm_gpusvm;
+struct drm_gpusvm_notifier;
+struct drm_gpusvm_ops;
+struct drm_gpusvm_range;
+struct drm_gpusvm_devmem;
+struct drm_pagemap;
+struct drm_pagemap_dma_addr;
+
+/**
+ * struct drm_gpusvm_devmem_ops - Operations structure for GPU SVM

device memory
+ *
+ * This structure defines the operations for GPU Shared Virtual Memory (SVM)
+ * device memory. These operations are provided by the GPU driver to manage device memory
+ * allocations and perform operations such as migration between device memory and system
+ * RAM.
+ */

include/drm/drm_gpusvm.h new file +3 -0
+struct drm_gpusvm_devmem_ops {
+	/**
+	 * @devmem_release: Release device memory allocation

(optional)

include/drm/drm_gpusvm.h new file +3 -0
+	 * @devmem_allocation: device memory allocation
+	 *
+	 * Release device memory allocation and drop a reference to

device

include/drm/drm_gpusvm.h new file +3 -0
+	 * memory allocation.
+	 */
+	void (*devmem_release)(struct drm_gpusvm_devmem

*devmem_allocation);

include/drm/drm_gpusvm.h new file +3 -0
+
+	/**
+	 * @populate_devmem_pfn: Populate device memory PFN

(required for migration)

include/drm/drm_gpusvm.h new file +10 -0
+	 * @devmem_allocation: device memory allocation
+	 * @npages: Number of pages to populate
+	 * @pfn: Array of page frame numbers to populate
+	 *
+	 * Populate device memory page frame numbers (PFN).
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*populate_devmem_pfn)(struct drm_gpusvm_devmem

*devmem_allocation,

include/drm/drm_gpusvm.h new file +1 -0
+				   unsigned long npages, unsigned

long *pfn);

include/drm/drm_gpusvm.h new file +3 -0
+
+	/**
+	 * @copy_to_devmem: Copy to device memory (required for

migration)

include/drm/drm_gpusvm.h new file +1 -0
+	 * @pages: Pointer to array of device memory pages

(destination)

include/drm/drm_gpusvm.h new file +16 -0
+	 * @dma_addr: Pointer to array of DMA addresses (source)
+	 * @npages: Number of pages to copy
+	 *
+	 * Copy pages to device memory.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_devmem)(struct page **pages,
+			      dma_addr_t *dma_addr,
+			      unsigned long npages);
+
+	/**
+	 * @copy_to_ram: Copy to system RAM (required for migration)
+	 * @pages: Pointer to array of device memory pages (source)
+	 * @dma_addr: Pointer to array of DMA addresses

(destination)

include/drm/drm_gpusvm.h new file +14 -0
+	 * @npages: Number of pages to copy
+	 *
+	 * Copy pages to system RAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_ram)(struct page **pages,
+			   dma_addr_t *dma_addr,
+			   unsigned long npages);
+};
+
+/**
+ * struct drm_gpusvm_devmem - Structure representing a GPU SVM

device memory allocation
+ *
+ * @dev: Pointer to the device structure which device memory allocation belongs to
+ * @mm: Pointer to the mm_struct for the address space
+ * @detached: device memory allocations is detached from device pages
+ * @ops: Pointer to the operations structure for GPU SVM device memory
+ * @dpagemap: The struct drm_pagemap of the pages this allocation belongs to.
+ * @size: Size of device memory allocation
+ */

include/drm/drm_gpusvm.h new file +13 -0
+struct drm_gpusvm_devmem {
+	struct device *dev;
+	struct mm_struct *mm;
+	struct completion detached;
+	const struct drm_gpusvm_devmem_ops *ops;
+	struct drm_pagemap *dpagemap;
+	size_t size;
+};
+
+/**
+ * struct drm_gpusvm_ops - Operations structure for GPU SVM
+ *
+ * This structure defines the operations for GPU Shared Virtual

Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * notifiers.
+ */

include/drm/drm_gpusvm.h new file +8 -0
+struct drm_gpusvm_ops {
+	/**
+	 * @notifier_alloc: Allocate a GPU SVM notifier (optional)
+	 *
+	 * Allocate a GPU SVM notifier.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM notifier on success,

NULL on failure.

include/drm/drm_gpusvm.h new file +19 -0
+	 */
+	struct drm_gpusvm_notifier *(*notifier_alloc)(void);
+
+	/**
+	 * @notifier_free: Free a GPU SVM notifier (optional)
+	 * @notifier: Pointer to the GPU SVM notifier to be freed
+	 *
+	 * Free a GPU SVM notifier.
+	 */
+	void (*notifier_free)(struct drm_gpusvm_notifier *notifier);
+
+	/**
+	 * @range_alloc: Allocate a GPU SVM range (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 *
+	 * Allocate a GPU SVM range.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM range on success, NULL

on failure.

include/drm/drm_gpusvm.h new file +2 -0
+	 */
+	struct drm_gpusvm_range *(*range_alloc)(struct drm_gpusvm

*gpusvm);

include/drm/drm_gpusvm.h new file +16 -0
+
+	/**
+	 * @range_free: Free a GPU SVM range (optional)
+	 * @range: Pointer to the GPU SVM range to be freed
+	 *
+	 * Free a GPU SVM range.
+	 */
+	void (*range_free)(struct drm_gpusvm_range *range);
+
+	/**
+	 * @invalidate: Invalidate GPU SVM notifier (required)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @notifier: Pointer to the GPU SVM notifier
+	 * @mmu_range: Pointer to the mmu_notifier_range structure
+	 *
+	 * Invalidate the GPU page tables. It can safely walk the

notifier range

include/drm/drm_gpusvm.h new file +1 -0
+	 * RB tree/list in this function. Called while holding the

notifier lock.

include/drm/drm_gpusvm.h new file +4 -0
+	 */
+	void (*invalidate)(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_notifier *notifier,
+			   const struct mmu_notifier_range

*mmu_range);

include/drm/drm_gpusvm.h new file +4 -0
+};
+
+/**
+ * struct drm_gpusvm_notifier - Structure representing a GPU SVM

notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @itree: Interval tree node for the notifier (inserted in GPU SVM)
+ * @entry: List entry to fast interval tree traversal
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ *              interval tree. This is useful to keep iterating ranges while
+ *              doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier has been
+ *                 removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */

include/drm/drm_gpusvm.h new file +19 -0
+struct drm_gpusvm_notifier {
+	struct drm_gpusvm *gpusvm;
+	struct mmu_interval_notifier notifier;
+	struct interval_tree_node itree;
+	struct list_head entry;
+	struct rb_root_cached root;
+	struct list_head range_list;
+	struct {
+		u32 removed : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm_range - Structure representing a GPU SVM range
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier
+ * @refcount: Reference count for the range
+ * @itree: Interval tree node for the range (inserted in GPU SVM

notifier)
+ * @entry: List entry to fast interval tree traversal
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @dma_addr: DMA address array
+ * @dpagemap: The struct drm_pagemap of the device pages we're dma- mapping.
+ *            Note this is assuming only one drm_pagemap per range is allowed.
+ * @flags.migrate_devmem: Flag indicating whether the range can be migrated to device memory
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_devmem_pages: Flag indicating if the range has devmem pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA mapping
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */

include/drm/drm_gpusvm.h new file +13 -0
+struct drm_gpusvm_range {
+	struct drm_gpusvm *gpusvm;
+	struct drm_gpusvm_notifier *notifier;
+	struct kref refcount;
+	struct interval_tree_node itree;
+	struct list_head entry;
+	unsigned long notifier_seq;
+	struct drm_pagemap_dma_addr *dma_addr;
+	struct drm_pagemap *dpagemap;
+	struct {
+		/* All flags below must be set upon creation */
+		u16 migrate_devmem : 1;
+		/* All flags below must be set / cleared under

notifier lock */

include/drm/drm_gpusvm.h new file +19 -0
+		u16 unmapped : 1;
+		u16 partial_unmap : 1;
+		u16 has_devmem_pages : 1;
+		u16 has_dma_mapping : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm - GPU SVM structure
+ *
+ * @name: Name of the GPU SVM
+ * @drm: Pointer to the DRM device structure
+ * @mm: Pointer to the mm_struct for the address space
+ * @device_private_page_owner: Device private pages owner
+ * @mm_start: Start address of GPU SVM
+ * @mm_range: Range of the GPU SVM
+ * @notifier_size: Size of individual notifiers
+ * @ops: Pointer to the operations structure for GPU SVM
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range

allocation.
+ *               Entries should be powers of 2 in descending order.
+ * @num_chunks: Number of chunks
+ * @notifier_lock: Read-write semaphore for protecting notifier operations
+ * @root: Cached root node of the Red-Black tree containing GPU SVM notifiers
+ * @notifier_list: list head containing of notifiers in the same order they
+ *                 appear in interval tree. This is useful to keep iterating
+ *                 notifiers while doing modifications to RB tree.
+ *
+ * This structure represents a GPU SVM (Shared Virtual Memory) used for tracking
+ * memory ranges mapped in a DRM (Direct Rendering Manager) device.
+ *
+ * No reference counting is provided, as this is expected to be embedded in the
+ * driver VM structure along with the struct drm_gpuvm, which handles reference
+ * counting.
+ */

include/drm/drm_gpusvm.h new file +17 -0
+struct drm_gpusvm {
+	const char *name;
+	struct drm_device *drm;
+	struct mm_struct *mm;
+	void *device_private_page_owner;
+	unsigned long mm_start;
+	unsigned long mm_range;
+	unsigned long notifier_size;
+	const struct drm_gpusvm_ops *ops;
+	const unsigned long *chunk_sizes;
+	int num_chunks;
+	struct rw_semaphore notifier_lock;
+	struct rb_root_cached root;
+	struct list_head notifier_list;
+#ifdef CONFIG_LOCKDEP
+	/**
+	 * @lock_dep_map: Annotates drm_gpusvm_range_find_or_insert

and

include/drm/drm_gpusvm.h new file +10 -0
+	 * drm_gpusvm_range_remove with a driver provided lock.
+	 */
+	struct lockdep_map *lock_dep_map;
+#endif
+};
+
+/**
+ * struct drm_gpusvm_ctx - DRM GPU SVM context
+ *
+ * @check_pages_threshold: Check CPU pages for present if chunk is

less than or
+ *                         equal to threshold. If not present, reduce chunk
+ *                         size.
+ * @in_notifier: entering from a MMU notifier
+ * @read_only: operating on read-only memory
+ * @devmem_possible: possible to use device memory
+ *
+ * Context that is DRM GPUSVM is operating in (i.e. user arguments).
+ */

include/drm/drm_gpusvm.h new file +10 -0
+struct drm_gpusvm_ctx {
+	unsigned long check_pages_threshold;
+	unsigned int in_notifier :1;
+	unsigned int read_only :1;
+	unsigned int devmem_possible :1;
+};
+
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void

*device_private_page_owner,

include/drm/drm_gpusvm.h new file +4 -0
+		    unsigned long mm_start, unsigned long mm_range,
+		    unsigned long notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const unsigned long *chunk_sizes, int

num_chunks);

include/drm/drm_gpusvm.h new file +37 -0
+
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm);
+
+void drm_gpusvm_free(struct drm_gpusvm *gpusvm);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm,
+				unsigned long fault_addr,
+				unsigned long gpuva_start,
+				unsigned long gpuva_end,
+				const struct drm_gpusvm_ctx *ctx);
+
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range);
+
+int drm_gpusvm_range_evict(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_range *range);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range);
+
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range);
+
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range);
+
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx);
+
+int drm_gpusvm_migrate_to_devmem(struct drm_gpusvm *gpusvm,
+				 struct drm_gpusvm_range *range,
+				 struct drm_gpusvm_devmem

*devmem_allocation,

include/drm/drm_gpusvm.h new file +3 -0
+				 const struct drm_gpusvm_ctx *ctx);
+
+int drm_gpusvm_evict_to_ram(struct drm_gpusvm_devmem

*devmem_allocation);

include/drm/drm_gpusvm.h new file +4 -0
+
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void);
+
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, unsigned long

start,

include/drm/drm_gpusvm.h new file +4 -0
+			    unsigned long end);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, unsigned

long start,

include/drm/drm_gpusvm.h new file +4 -0
+		      unsigned long end);
+
+void drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,
+				   const struct mmu_notifier_range

*mmu_range);

include/drm/drm_gpusvm.h new file +2 -0
+
+void drm_gpusvm_devmem_init(struct drm_gpusvm_devmem

*devmem_allocation,

include/drm/drm_gpusvm.h new file +1 -0
+			    struct device *dev, struct mm_struct

*mm,

include/drm/drm_gpusvm.h new file +2 -0
+			    const struct drm_gpusvm_devmem_ops *ops,
+			    struct drm_pagemap *dpagemap, size_t

size);

include/drm/drm_gpusvm.h new file +4 -0
+
+#ifdef CONFIG_LOCKDEP
+/**
+ * drm_gpusvm_driver_set_lock() - Set the lock protecting accesses

to GPU SVM
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @lock: the lock used to protect the gpuva list. The locking primitive
+ * must contain a dep_map field.
+ *
+ * Call this to annotate drm_gpusvm_range_find_or_insert and
+ * drm_gpusvm_range_remove.
+ */

include/drm/drm_gpusvm.h new file +4 -0
+#define drm_gpusvm_driver_set_lock(gpusvm, lock) \
+	do { \
+		if (!WARN((gpusvm)->lock_dep_map, \
+			  "GPUSVM range lock should be set only

once."))\

include/drm/drm_gpusvm.h new file +1 -0
+			(gpusvm)->lock_dep_map = &(lock)-
Quoted 1 line

dep_map; \

Quoted 9 lines

include/drm/drm_gpusvm.h line 387

Quoted 6 lines
include/drm/drm_gpusvm.h new file +6 -0
+	} while (0)
+#define drm_gpusvm_driver_lock_held(gpusvm) \
+	do { \
+		if ((gpusvm)->lock_dep_map)	\
+			lock_is_held((gpusvm)->lock_dep_map);	\
+	} while (0)

Could we use static functions for those above

Static should work. Will change.

Quoted 3 lines

Also I don't think the drm_gpusvm_driver_lock_held() does what it's intended to do? There's an assert missing.

'lock_is_held' is an assert, right? I based this code existing code drm_gem_gpuva_assert_lock_held which uses 'lock_is_held'.

This probably should be changed to 'lock_is_held_type(..., 0)' to indicate lock is held in exclusive mode though.

Quoted 64 lines
Quoted 64 lines
include/drm/drm_gpusvm.h new file +25 -0
+#else
+#define drm_gpusvm_driver_set_lock(gpusvm, lock) do {} while (0)
+#define drm_gpusvm_driver_lock_held(gpusvm) do {} while (0)
+#endif
+
+/**
+ * drm_gpusvm_notifier_lock() - Lock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, take lock
+ */
+#define drm_gpusvm_notifier_lock(gpusvm__)	\
+	down_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * drm_gpusvm_notifier_unlock() - Unlock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, drop lock
+ */
+#define drm_gpusvm_notifier_unlock(gpusvm__)	\
+	up_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * __drm_gpusvm_range_next() - Get the next GPU SVM range in the

list
+ * @range: a pointer to the current GPU SVM range
+ *
+ * Return: A pointer to the next drm_gpusvm_range if available, or NULL if the
+ *         current range is the last one or if the input range is NULL.
+ */

include/drm/drm_gpusvm.h new file +12 -0
+static inline struct drm_gpusvm_range *
+__drm_gpusvm_range_next(struct drm_gpusvm_range *range)
+{
+	if (range && !list_is_last(&range->entry,
+				   &range->notifier->range_list))
+		return list_next_entry(range, entry);
+
+	return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_range() - Iterate over GPU SVM ranges in a

notifier
+ * @range__: Iterator variable for the ranges. If set, it indicates the start of
+ *      the iterator. If NULL, call drm_gpusvm_range_find() to
get the range.
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier. It is safe
+ * to use while holding the driver SVM lock or the notifier lock.
+ */
+#define drm_gpusvm_for_each_range(range__, notifier__, start__,
end__) \

include/drm/drm_gpusvm.h new file +1 -0
+	for ((range__) = (range__)

?: \

include/drm/drm_gpusvm.h new file +1 -0
+	     drm_gpusvm_range_find((notifier__), (start__),
Quoted 9 lines

include/drm/drm_gpusvm.h line 445

Quoted 6 lines

(end__)); \

include/drm/drm_gpusvm.h new file +1 -0
+	     (range__) && (range__->itree.start <

(end__)); \

include/drm/drm_gpusvm.h new file +3 -0
+	     (range__) = __drm_gpusvm_range_next(range__))
+
+#endif /* __DRM_GPUSVM_H__ */

Otherwise LGTM.

Thanks,
Matt

Quoted 4 lines

/Thomas

#22 lore raw

Re: [PATCH v4 06/33] drm/gpusvm: Add support for GPU Shared Virtual Memory

Quoted 3,535 lines · Matthew Brost
Quoted 3,528 lines · Thomas Hellström
Quoted 3,527 lines · Matthew Brost

This patch introduces support for GPU Shared Virtual Memory (SVM) in
the
Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and
flexibility in GPU computing tasks.

The patch adds the necessary infrastructure for SVM, including data
structures and functions for managing SVM ranges and notifiers. It
also
provides mechanisms for allocating, deallocating, and migrating memory
regions between system RAM and GPU VRAM.

This is largely inspired by GPUVM.

v2:
 - Take order into account in check pages
 - Clear range->pages in get pages error
 - Drop setting dirty or accessed bit in get pages (Vetter)
 - Remove mmap assert for cpu faults
 - Drop mmap write lock abuse (Vetter, Christian)
 - Decouple zdd from range (Vetter, Oak)
 - Add drm_gpusvm_range_evict, make it work with coherent pages
 - Export drm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
 - mmget/put in drm_gpusvm_evict_to_sram
 - Drop range->vram_alloation variable
 - Don't return in drm_gpusvm_evict_to_sram until all pages detached
 - Don't warn on mixing sram and device pages
 - Update kernel doc
 - Add coherent page support to get pages
 - Use DMA_FROM_DEVICE rather than DMA_BIDIRECTIONAL
 - Add struct drm_gpusvm_vram and ops (Thomas)
 - Update the range's seqno if the range is valid (Thomas)
 - Remove the is_unmapped check before hmm_range_fault (Thomas)
 - Use drm_pagemap (Thomas)
 - Drop kfree_mapping (Thomas)
 - dma mapp pages under notifier lock (Thomas)
 - Remove ctx.prefault
 - Remove ctx.mmap_locked
 - Add ctx.check_pages
 - s/vram/devmem (Thomas)
v3:
 - Fix memory leak drm_gpusvm_range_get_pages
 - Only migrate pages with same zdd on CPU fault
 - Loop over al VMAs in drm_gpusvm_range_evict
 - Make GPUSVM a drm level module
 - GPL or MIT license
 - Update main kernel doc (Thomas)
 - Prefer foo() vs foo for functions in kernel doc (Thomas)
 - Prefer functions over macros (Thomas)
 - Use unsigned long vs u64 for addresses (Thomas)
 - Use standard interval_tree (Thomas)
 - s/drm_gpusvm_migration_put_page/drm_gpusvm_migration_unlock_put_p age
(Thomas)
 - Drop err_out label in drm_gpusvm_range_find_or_insert (Thomas)
 - Fix kernel doc in drm_gpusvm_range_free_pages (Thomas)
 - Newlines between functions defs in header file (Thomas)
 - Drop shall language in driver vfunc kernel doc (Thomas)
 - Move some static inlines from head to C file (Thomas)
 - Don't allocate pages under page lock in drm_gpusvm_migrate_populate_ram_pfn (Thomas)
 - Change check_pages to a threshold
v4:
 - Fix NULL ptr deref in drm_gpusvm_migrate_populate_ram_pfn (Thomas,
Himal)
 - Fix check pages threshold
 - Check for range being unmapped under notifier lock in get pages
(Testing)
 - Fix characters per line
 - Drop WRITE_ONCE for zdd->devmem_allocation assignment (Thomas)
 - Use completion for devmem_allocation->detached (Thomas)
 - Make GPU SVM depend on ZONE_DEVICE (CI)
 - Use hmm_range_fault for eviction (Thomas)
 - Drop zdd worker (Thomas)

Cc Simona Vetter <[email protected]>
Cc Dave Airlie <[email protected]>
Cc Christian König <[email protected]>
Signed-off-by Thomas Hellström

<[email protected]>
---
 drivers/gpu/drm/Kconfig      |    9 +
 drivers/gpu/drm/Makefile     |    1 +
 drivers/gpu/drm/drm_gpusvm.c | 2240
++++++++++++++++++++++++++++++++++
 include/drm/drm_gpusvm.h     |  445 +++++++

 4 files changed, 2695 insertions(+)

 create mode 100644 drivers/gpu/drm/drm_gpusvm.c
 create mode 100644 include/drm/drm_gpusvm.h

drivers/gpu/drm/Kconfig +0 -0
index fbef3f471bd0..f03862e379fb 100644
@@ -278,6 +278,15 @@ config DRM_GPUVM

    GPU-VM representation providing helpers to manage a GPUs
virtual
    address space

drivers/gpu/drm/Kconfig +7 -0
+config DRM_GPUSVM
+	tristate
+	depends on DRM
+	depends on DEVICE_MIGRATION
+	depends on ZONE_DEVICE
+	help
+	  GPU-SVM representation providing helpers to manage a

GPUs
shared

drivers/gpu/drm/Kconfig +2 -0
+	  virtual memory
+

 config DRM_BUDDY
  tristate
  depends on DRM

drivers/gpu/drm/Makefile +0 -0
index 85af94bb907d..ca03df8d2729 100644
@@ -104,6 +104,7 @@ obj-$(CONFIG_DRM_PANEL_BACKLIGHT_QUIRKS) +=

drm_panel_backlight_quirks.o
 #
 obj-$(CONFIG_DRM_EXEC) += drm_exec.o
 obj-$(CONFIG_DRM_GPUVM) += drm_gpuvm.o
+obj-$(CONFIG_DRM_GPUSVM) += drm_gpusvm.o

 obj-$(CONFIG_DRM_BUDDY) += drm_buddy.o

drivers/gpu/drm/drm_gpusvm.c new file +25 -0
new file mode 100644
index 000000000000..1c63da4d3cc2
@@ -0,0 +1,2240 @@
+// SPDX-License-Identifier: GPL-2.0-only OR MIT
+/*
+ * Copyright © 2024 Intel Corporation
+ *
+ * Authors:
+ *     Matthew Brost <[email protected]>
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/hmm.h>
+#include <linux/memremap.h>
+#include <linux/migrate.h>
+#include <linux/mm_types.h>
+#include <linux/pagemap.h>
+#include <linux/slab.h>
+
+#include <drm/drm_device.h>
+#include <drm/drm_gpusvm.h>
+#include <drm/drm_pagemap.h>
+#include <drm/drm_print.h>
+
+/**
+ * DOC: Overview
+ *
+ * GPU Shared Virtual Memory (GPU SVM) layer for the Direct

Rendering Manager (DRM)
+ *
+ * The GPU SVM layer is a component of the DRM framework designed to
manage shared
+ * virtual memory between the CPU and GPU. It enables efficient data
exchange and
+ * processing for GPU-accelerated applications by allowing memory
sharing and
+ * synchronization between the CPU's and GPU's virtual address spaces.
+ *
+ * Key GPU SVM Components:
+ * - Notifiers: Notifiers: Used for tracking memory intervals and
notifying the
+ * GPU of changes, notifiers are sized based on a
GPU
SVM
+ * initialization parameter, with a recommendation
of
512M or
+ * larger. They maintain a Red-BlacK tree and a
list of
ranges that
+ * fall within the notifier interval. Notifiers are
tracked within
+ * a GPU SVM Red-BlacK tree and list and are
dynamically inserted
+ * or removed as ranges within the interval are
created
or
+ * destroyed.
+ * - Ranges: Represent memory ranges mapped in a DRM device and managed
+ *      by GPU SVM. They are sized based on an array of
chunk
sizes, which
+ *      is a GPU SVM initialization parameter, and the CPU
address space.
+ *      Upon GPU fault, the largest aligned chunk that fits
within the
+ *      faulting CPU address space is chosen for the range
size. Ranges are
+ *      expected to be dynamically allocated on GPU fault
and
removed on an
+ *      MMU notifier UNMAP event. As mentioned above,
ranges
are tracked in
+ *      a notifier's Red-Black tree.
+ * - Operations: Define the interface for driver-specific GPU SVM
operations
+ *               such as range allocation, notifier allocation, and
+ *               invalidations.
+ * - Device Memory Allocations: Embedded structure containing enough
information
+ *                              for GPU SVM to migrate to / from device memory.
+ * - Device Memory Operations: Define the interface for driver- specific device
+ *                             memory operations release memory, populate pfns,
+ *                             and copy to / from device memory.
+ *
+ * This layer provides interfaces for allocating, mapping, migrating, and
+ * releasing memory ranges between the CPU and GPU. It handles all
core memory
+ * management interactions (DMA mapping, HMM, and migration) and provides
+ * driver-specific virtual functions (vfuncs). This infrastructure
is sufficient
+ * to build the expected driver components for an SVM implementation
as detailed
+ * below.
+ *
+ * Expected Driver Components:
+ * - GPU page fault handler: Used to create ranges and notifiers based on the
+ *      fault address, optionally migrate
the
range to
+ *      device memory, and create GPU
bindings.
+ * - Garbage collector: Used to unmap and destroy GPU bindings for
ranges.
+ * Ranges are expected to be added to the
garbage collector
+ * upon a MMU_NOTIFY_UNMAP event in
notifier
callback.
+ * - Notifier callback: Used to invalidate and DMA unmap GPU bindings for
+ * ranges.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Locking
+ *
+ * GPU SVM handles locking for core MM interactions, i.e., it

locks/unlocks the
+ * mmap lock as needed.
+ *
+ * GPU SVM introduces a global notifier lock, which safeguards the
notifier's
+ * range RB tree and list, as well as the range's DMA mappings and
sequence
+ * number. GPU SVM manages all necessary locking and unlocking operations,
+ * except for the recheck range's pages being valid
+ * (drm_gpusvm_range_pages_valid) when the driver is committing GPU
bindings. This
+ * lock corresponds to the 'driver->update' lock mentioned in the
HMM
+ * documentation (TODO: Link). Future revisions may transition from
a GPU SVM
+ * global lock to a per-notifier lock if finer-grained locking is
deemed
+ * necessary.
+ *
+ * In addition to the locking mentioned above, the driver should implement a
+ * lock to safeguard core GPU SVM function calls that modify state,
such as
+ * drm_gpusvm_range_find_or_insert and drm_gpusvm_range_remove. This
lock is
+ * denoted as 'driver_svm_lock' in code examples. Finer grained driver side
+ * locking should also be possible for concurrent GPU fault processing within a
+ * single GPU SVM. The 'driver_svm_lock' can be via drm_gpusvm_driver_set_lock
+ * to add annotations to GPU SVM.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Migration
+ *
+ * The migration support is quite simple, allowing migration

between
RAM and
+ * device memory at the range granularity. For example, GPU SVM currently does not
+ * support mixing RAM and device memory pages within a range. This
means that upon GPU
+ * fault, the entire range can be migrated to device memory, and upon CPU fault, the
+ * entire range is migrated to RAM. Mixed RAM and device memory storage within a range
+ * could be added in the future if required.
+ *
+ * The reasoning for only supporting range granularity is as
follows: it
+ * simplifies the implementation, and range sizes are driver- defined
and should
+ * be relatively small.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Partial Unmapping of Ranges
+ *
+ * Partial unmapping of ranges (e.g., 1M out of 2M is unmapped

by
CPU resulting
+ * in MMU_NOTIFY_UNMAP event) presents several challenges, with the
main one
+ * being that a subset of the range still has CPU and GPU mappings.
If the
+ * backing store for the range is in device memory, a subset of the
backing store has
+ * references. One option would be to split the range and device memory backing store,
+ * but the implementation for this would be quite complicated. Given
that
+ * partial unmappings are rare and driver-defined range sizes are
relatively
+ * small, GPU SVM does not support splitting of ranges.
+ *
+ * With no support for range splitting, upon partial unmapping of a
range, the
+ * driver is expected to invalidate and destroy the entire range. If
the range
+ * has device memory as its backing, the driver is also expected to
migrate any
+ * remaining pages back to RAM.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Examples
+ *
+ * This section provides three examples of how to build the

expected
driver
+ * components: the GPU page fault handler, the garbage collector,
and the
+ * notifier callback.
+ *
+ * The generic code provided does not include logic for complex migration
+ * policies, optimized invalidations, fined grained driver locking,
or other
+ * potentially required driver locking (e.g., DMA-resv locks).
+ *
+ * 1) GPU page fault handler
+ *
+ * int driver_bind_range(struct drm_gpusvm *gpusvm, struct
drm_gpusvm_range *range)
+ * {
+ * int err = 0;
+ *
+ * driver_alloc_and_setup_memory_for_bind(gpusvm,
range);
+ *
+ * drm_gpusvm_notifier_lock(gpusvm);
+ * if (drm_gpusvm_range_pages_valid(range))
+ * driver_commit_bind(gpusvm, range);
+ * else
+ * err = -EAGAIN;
+ * drm_gpusvm_notifier_unlock(gpusvm);
+ *
+ * return err;
+ * }
+ *
+ * int driver_gpu_fault(struct drm_gpusvm *gpusvm, unsigned
long fault_addr,
+ *      unsigned long gpuva_start, unsigned
long gpuva_end)
+ * {
+ * struct drm_gpusvm_ctx ctx = {};
+ * int err;
+ *
+ * driver_svm_lock();
+ * retry:
+ * // Always process UNMAPs first so view of GPU
SVM
ranges is current
+ * driver_garbage_collector(gpusvm);
+ *
+ * range = drm_gpusvm_range_find_or_insert(gpusvm,
fault_addr,
+
* gpuva_start,
gpuva_end,
+ *         &ctx);
+ * if (IS_ERR(range)) {
+ * err = PTR_ERR(range);
+ * goto unlock;
+ * }
+ *
+ * if (driver_migration_policy(range)) {
+ * devmem = driver_alloc_devmem();
+ * err =
drm_gpusvm_migrate_to_devmem(gpusvm,
range,
+ *
devmem_allocation,
+ *
&ctx);
+ * if (err) // CPU mappings may have
changed
+ * goto retry;
+ * }
+ *
+ * err = drm_gpusvm_range_get_pages(gpusvm, range,
&ctx);
+ * if (err == -EOPNOTSUPP || err == -EFAULT || err
== -
EPERM) { // CPU mappings changed
+ * if (err == -EOPNOTSUPP)
+ * drm_gpusvm_range_evict(gpusvm,
range);
+ * goto retry;
+ * } else if (err) {
+ * goto unlock;
+ * }
+ *
+ * err = driver_bind_range(gpusvm, range);
+ * if (err == -EAGAIN) // CPU mappings changed
+ * goto retry
+ *
+ * unlock:
+ * driver_svm_unlock();
+ * return err;
+ * }
+ *
+ * 2) Garbage Collector.
+ *
+ * void __driver_garbage_collector(struct drm_gpusvm
*gpusvm,
+ * struct drm_gpusvm_range
*range)
+ * {
+ * assert_driver_svm_locked(gpusvm);
+ *
+ * // Partial unmap, migrate any remaining device
memory pages back to RAM
+ * if (range->flags.partial_unmap)
+ * drm_gpusvm_range_evict(gpusvm, range);
+ *
+ * driver_unbind_range(range);
+ * drm_gpusvm_range_remove(gpusvm, range);
+ * }
+ *
+ * void driver_garbage_collector(struct drm_gpusvm *gpusvm)
+ * {
+ * assert_driver_svm_locked(gpusvm);
+ *
+ * for_each_range_in_garbage_collector(gpusvm,
range)
+ * __driver_garbage_collector(gpusvm,
range);
+ * }
+ *
+ * 3) Notifier callback.
+ *
+ * void driver_invalidation(struct drm_gpusvm *gpusvm,
+ * struct drm_gpusvm_notifier
*notifier,
+ * const struct mmu_notifier_range
*mmu_range)
+ * {
+ * struct drm_gpusvm_ctx ctx = { .in_notifier =
true,
};
+ * struct drm_gpusvm_range *range = NULL;
+ *
+ * driver_invalidate_device_pages(gpusvm,
mmu_range-

Quoted 1 line

start, mmu_range->end);

+ *
+ * drm_gpusvm_for_each_range(range, notifier,
mmu_range->start,
+ *   mmu_range->end) {
+ * drm_gpusvm_range_unmap_pages(gpusvm,
range,
&ctx);
+ *
+ * if (mmu_range->event !=
MMU_NOTIFY_UNMAP)
+ * continue;
+ *
+ * drm_gpusvm_range_set_unmapped(range,
mmu_range);
+ * driver_garbage_collector_add(gpusvm,
range);
+ * }
+ * }
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+
+/**
+ * npages_in_range() - Calculate the number of pages in a given

range
+ * @start: The start address of the range
+ * @end: The end address of the range
+ *
+ * This macro calculates the number of pages in a given memory range,
+ * specified by the start and end addresses. It divides the difference
+ * between the end and start addresses by the page size (PAGE_SIZE)
to
+ * determine the number of pages in the range.
+ *
+ * Returns: The number of pages in the specified range.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+static unsigned long
+npages_in_range(unsigned long start, unsigned long end)
+{
+	return (end - start) >> PAGE_SHIFT;
+}
+
+/**
+ * struct drm_gpusvm_zdd - GPU SVM zone device data
+ *
+ * @refcount: Reference count for the zdd
+ * @devmem_allocation: device memory allocation
+ * @device_private_page_owner: Device private pages owner
+ *
+ * This structure serves as a generic wrapper installed in
+ * page->zone_device_data. It provides infrastructure for

looking up
a device
+ * memory allocation upon CPU page fault and asynchronously releasing device
+ * memory once the CPU has no page references. Asynchronous release
is useful
+ * because CPU page references can be dropped in IRQ contexts, while
releasing
+ * device memory likely requires sleeping locks.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+struct drm_gpusvm_zdd {
+	struct kref refcount;
+	struct drm_gpusvm_devmem *devmem_allocation;
+	void *device_private_page_owner;
+};
+
+/**
+ * drm_gpusvm_zdd_alloc() - Allocate a zdd structure.
+ * @device_private_page_owner: Device private pages owner
+ *
+ * This function allocates and initializes a new zdd structure.

It
sets up the
+ * reference count and initializes the destroy work.
+ *
+ * Returns:
+ * Pointer to the allocated zdd on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+static struct drm_gpusvm_zdd *
+drm_gpusvm_zdd_alloc(void *device_private_page_owner)
+{
+	struct drm_gpusvm_zdd *zdd;
+
+	zdd = kmalloc(sizeof(*zdd), GFP_KERNEL);
+	if (!zdd)
+		return NULL;
+
+	kref_init(&zdd->refcount);
+	zdd->devmem_allocation = NULL;
+	zdd->device_private_page_owner =

device_private_page_owner;

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+
+	return zdd;
+}
+
+/**
+ * drm_gpusvm_zdd_get() - Get a reference to a zdd structure.
+ * @zdd: Pointer to the zdd structure.
+ *
+ * This function increments the reference count of the provided

zdd
structure.
+ *
+ * Returns: Pointer to the zdd structure.
+ */
+static struct drm_gpusvm_zdd *drm_gpusvm_zdd_get(struct drm_gpusvm_zdd *zdd)

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+{
+	kref_get(&zdd->refcount);
+	return zdd;
+}
+
+/**
+ * drm_gpusvm_zdd_destroy() - Destroy a zdd structure.
+ * @ref: Pointer to the reference count structure.
+ *
+ * This function queues the destroy_work of the zdd for

asynchronous
destruction.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_zdd_destroy(struct kref *ref)
+{
+	struct drm_gpusvm_zdd *zdd =
+		container_of(ref, struct drm_gpusvm_zdd,

refcount);

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	struct drm_gpusvm_devmem *devmem = zdd-
Quoted 1 line

devmem_allocation;

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+
+	if (devmem) {
+		complete_all(&devmem->detached);
+		if (devmem->ops->devmem_release)
+			devmem->ops->devmem_release(devmem);
+	}
+	kfree(zdd);
+}
+
+/**
+ * drm_gpusvm_zdd_put() - Put a zdd reference.
+ * @zdd: Pointer to the zdd structure.
+ *
+ * This function decrements the reference count of the provided

zdd
structure
+ * and schedules its destruction if the count drops to zero.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+static void drm_gpusvm_zdd_put(struct drm_gpusvm_zdd *zdd)
+{
+	kref_put(&zdd->refcount, drm_gpusvm_zdd_destroy);
+}
+
+/**
+ * drm_gpusvm_range_find() - Find GPU SVM range from GPU SVM

notifier
+ * @notifier: Pointer to the GPU SVM notifier structure.
+ * @start: Start address of the range
+ * @end: End address of the range
+ *
+ * Returns: A pointer to the drm_gpusvm_range if found or NULL
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier,

unsigned
long start,

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+		      unsigned long end)
+{
+	struct interval_tree_node *itree;
+
+	itree = interval_tree_iter_first(&notifier->root, start,

end
- 1);

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+
+	if (itree)
+		return container_of(itree, struct

drm_gpusvm_range,
itree);

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+	else
+		return NULL;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_find);
+
+/**
+ * drm_gpusvm_for_each_range_safe() - Safely iterate over GPU

SVM
ranges in a notifier
+ * @range__: Iterator variable for the ranges
+ * @next__: Iterator variable for the ranges temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier
while
+ * removing ranges from it.
+ */
+#define drm_gpusvm_for_each_range_safe(range__, next__, notifier__,
start__, end__) \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	for ((range__) = drm_gpusvm_range_find((notifier__),

(start__), (end__)), \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (next__) =

__drm_gpusvm_range_next(range__); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (range__) && (range__->itree.start <

(end__)); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (range__) = (next__), (next__) =

__drm_gpusvm_range_next(range__))

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+
+/**
+ * __drm_gpusvm_notifier_next() - get the next

drm_gpusvm_notifier
in the list
+ * @notifier: a pointer to the current drm_gpusvm_notifier
+ *
+ * Returns: A pointer to the next drm_gpusvm_notifier if available,
or NULL if
+ *         the current notifier is the last one or if the input notifier is
+ *         NULL.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+static struct drm_gpusvm_notifier *
+__drm_gpusvm_notifier_next(struct drm_gpusvm_notifier *notifier)
+{
+	if (notifier && !list_is_last(&notifier->entry,
+				      &notifier->gpusvm-
Quoted 1 line

notifier_list))

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+		return list_next_entry(notifier, entry);
+
+	return NULL;
+}
+
+static struct drm_gpusvm_notifier *
+notifier_iter_first(struct rb_root_cached *root, unsigned long

start,

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+		    unsigned long last)
+{
+	struct interval_tree_node *itree;
+
+	itree = interval_tree_iter_first(root, start, last);
+
+	if (itree)
+		return container_of(itree, struct

drm_gpusvm_notifier, itree);

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+	else
+		return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_notifier() - Iterate over GPU SVM

notifiers
in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm.
+ */
+#define drm_gpusvm_for_each_notifier(notifier__, gpusvm__, start__,
end__) \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	for ((notifier__) = notifier_iter_first(&(gpusvm__)-
Quoted 1 line

root,

(start__), (end__) - 1); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (notifier__) && (notifier__->itree.start <

(end__)); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (notifier__) =

__drm_gpusvm_notifier_next(notifier__))

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+
+/**
+ * drm_gpusvm_for_each_notifier_safe() - Safely iterate over GPU

SVM
notifiers in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @next__: Iterator variable for the notifiers temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm
while
+ * removing notifiers from it.
+ */
+#define drm_gpusvm_for_each_notifier_safe(notifier__, next__,
gpusvm__, start__, end__) \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	for ((notifier__) = notifier_iter_first(&(gpusvm__)-
Quoted 1 line

root,

(start__), (end__) - 1), \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (next__) =

__drm_gpusvm_notifier_next(notifier__); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (notifier__) && (notifier__->itree.start <

(end__)); \

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	     (notifier__) = (next__), (next__) =

__drm_gpusvm_notifier_next(notifier__))

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+
+/**
+ * drm_gpusvm_notifier_invalidate() - Invalidate a GPU SVM

notifier.
+ * @mni: Pointer to the mmu_interval_notifier structure.
+ * @mmu_range: Pointer to the mmu_notifier_range structure.
+ * @cur_seq: Current sequence number.
+ *
+ * This function serves as a generic MMU notifier for GPU SVM. It
sets the MMU
+ * notifier sequence number and calls the driver invalidate vfunc
under
+ * gpusvm->notifier_lock.
+ *
+ * Returns:
+ * true if the operation succeeds, false otherwise.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static bool
+drm_gpusvm_notifier_invalidate(struct mmu_interval_notifier

*mni,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+			       const struct mmu_notifier_range

*mmu_range,

drivers/gpu/drm/drm_gpusvm.c new file +19 -0
+			       unsigned long cur_seq)
+{
+	struct drm_gpusvm_notifier *notifier =
+		container_of(mni, typeof(*notifier), notifier);
+	struct drm_gpusvm *gpusvm = notifier->gpusvm;
+
+	if (!mmu_notifier_range_blockable(mmu_range))
+		return false;
+
+	down_write(&gpusvm->notifier_lock);
+	mmu_interval_set_seq(mni, cur_seq);
+	gpusvm->ops->invalidate(gpusvm, notifier, mmu_range);
+	up_write(&gpusvm->notifier_lock);
+
+	return true;
+}
+
+/**
+ * drm_gpusvm_notifier_ops - MMU interval notifier operations

for
GPU SVM
+ */
+static const struct mmu_interval_notifier_ops
drm_gpusvm_notifier_ops = {

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+	.invalidate = drm_gpusvm_notifier_invalidate,
+};
+
+/**
+ * drm_gpusvm_init() - Initialize the GPU SVM.
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @name: Name of the GPU SVM.
+ * @drm: Pointer to the DRM device structure.
+ * @mm: Pointer to the mm_struct for the address space.
+ * @device_private_page_owner: Device private pages owner.
+ * @mm_start: Start address of GPU SVM.
+ * @mm_range: Range of the GPU SVM.
+ * @notifier_size: Size of individual notifiers.
+ * @ops: Pointer to the operations structure for GPU SVM.
+ * @chunk_sizes: Pointer to the array of chunk sizes used in

range
allocation.
+ *               Entries should be powers of 2 in descending order
with last
+ *               entry being SZ_4K.
+ * @num_chunks: Number of chunks.
+ *
+ * This function initializes the GPU SVM.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void

*device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		    unsigned long mm_start, unsigned long

mm_range,

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+		    unsigned long notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const unsigned long *chunk_sizes, int

num_chunks)

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+{
+	if (!ops->invalidate || !num_chunks)
+		return -EINVAL;
+
+	gpusvm->name = name;
+	gpusvm->drm = drm;
+	gpusvm->mm = mm;
+	gpusvm->device_private_page_owner =

device_private_page_owner;

drivers/gpu/drm/drm_gpusvm.c new file +31 -0
+	gpusvm->mm_start = mm_start;
+	gpusvm->mm_range = mm_range;
+	gpusvm->notifier_size = notifier_size;
+	gpusvm->ops = ops;
+	gpusvm->chunk_sizes = chunk_sizes;
+	gpusvm->num_chunks = num_chunks;
+
+	mmgrab(mm);
+	gpusvm->root = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&gpusvm->notifier_list);
+
+	init_rwsem(&gpusvm->notifier_lock);
+
+	fs_reclaim_acquire(GFP_KERNEL);
+	might_lock(&gpusvm->notifier_lock);
+	fs_reclaim_release(GFP_KERNEL);
+
+#ifdef CONFIG_LOCKDEP
+	gpusvm->lock_dep_map = NULL;
+#endif
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_init);
+
+/**
+ * drm_gpusvm_notifier_find() - Find GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ *
+ * This function finds the GPU SVM notifier associated with the

fault address.
+ *
+ * Returns:
+ * Pointer to the GPU SVM notifier on success, NULL otherwise.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+static struct drm_gpusvm_notifier *
+drm_gpusvm_notifier_find(struct drm_gpusvm *gpusvm,
+			 unsigned long fault_addr)
+{
+	return notifier_iter_first(&gpusvm->root, fault_addr,

fault_addr + 1);

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+}
+
+/**
+ * to_drm_gpusvm_notifier() - retrieve the container struct for

a
given rbtree node
+ * @node: a pointer to the rbtree node embedded within a drm_gpusvm_notifier struct
+ *
+ * Returns: A pointer to the containing drm_gpusvm_notifier structure.
+ */
+static struct drm_gpusvm_notifier *to_drm_gpusvm_notifier(struct rb_node *node)

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+{
+	return container_of(node, struct drm_gpusvm_notifier,

itree.rb);

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+}
+
+/**
+ * drm_gpusvm_notifier_insert() - Insert GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function inserts the GPU SVM notifier into the GPU SVM

RB
tree and list.
+ */
+static void drm_gpusvm_notifier_insert(struct drm_gpusvm *gpusvm,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				       struct

drm_gpusvm_notifier
*notifier)

drivers/gpu/drm/drm_gpusvm.c new file +21 -0
+{
+	struct rb_node *node;
+	struct list_head *head;
+
+	interval_tree_insert(&notifier->itree, &gpusvm->root);
+
+	node = rb_prev(&notifier->itree.rb);
+	if (node)
+		head = &(to_drm_gpusvm_notifier(node))->entry;
+	else
+		head = &gpusvm->notifier_list;
+
+	list_add(&notifier->entry, head);
+}
+
+/**
+ * drm_gpusvm_notifier_remove() - Remove GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM tructure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function removes the GPU SVM notifier from the GPU SVM

RB
tree and list.
+ */
+static void drm_gpusvm_notifier_remove(struct drm_gpusvm *gpusvm,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				       struct

drm_gpusvm_notifier
*notifier)

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+{
+	interval_tree_remove(&notifier->itree, &gpusvm->root);
+	list_del(&notifier->entry);
+}
+
+/**
+ * drm_gpusvm_fini() - Finalize the GPU SVM.
+ * @gpusvm: Pointer to the GPU SVM structure.
+ *
+ * This function finalizes the GPU SVM by cleaning up any

remaining
ranges and
+ * notifiers, and dropping a reference to struct MM.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm)
+{
+	struct drm_gpusvm_notifier *notifier, *next;
+
+	drm_gpusvm_for_each_notifier_safe(notifier, next,

gpusvm, 0,
LONG_MAX) {

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+		struct drm_gpusvm_range *range, *__next;
+
+		/*
+		 * Remove notifier first to avoid racing with

any
invalidation

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+		 */
+		mmu_interval_notifier_remove(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+		notifier->flags.removed = true;
+
+		drm_gpusvm_for_each_range_safe(range, __next,

notifier, 0,

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+					       LONG_MAX)
+			drm_gpusvm_range_remove(gpusvm, range);
+	}
+
+	mmdrop(gpusvm->mm);
+	WARN_ON(!RB_EMPTY_ROOT(&gpusvm->root.rb_root));
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_fini);
+
+/**
+ * drm_gpusvm_notifier_alloc() - Allocate GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ *
+ * This function allocates and initializes the GPU SVM notifier

structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM notifier on success, ERR_PTR()
on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static struct drm_gpusvm_notifier *
+drm_gpusvm_notifier_alloc(struct drm_gpusvm *gpusvm, unsigned

long
fault_addr)

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	if (gpusvm->ops->notifier_alloc)
+		notifier = gpusvm->ops->notifier_alloc();
+	else
+		notifier = kzalloc(sizeof(*notifier),

GFP_KERNEL);

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+
+	if (!notifier)
+		return ERR_PTR(-ENOMEM);
+
+	notifier->gpusvm = gpusvm;
+	notifier->itree.start = ALIGN_DOWN(fault_addr, gpusvm-
Quoted 1 line

notifier_size);

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	notifier->itree.last = ALIGN(fault_addr + 1, gpusvm-
Quoted 1 line

notifier_size) - 1;

drivers/gpu/drm/drm_gpusvm.c new file +16 -0
+	INIT_LIST_HEAD(&notifier->entry);
+	notifier->root = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&notifier->range_list);
+
+	return notifier;
+}
+
+/**
+ * drm_gpusvm_notifier_free() - Free GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function frees the GPU SVM notifier structure.
+ */
+static void drm_gpusvm_notifier_free(struct drm_gpusvm *gpusvm,
+				     struct drm_gpusvm_notifier

*notifier)

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+{
+	WARN_ON(!RB_EMPTY_ROOT(&notifier->root.rb_root));
+
+	if (gpusvm->ops->notifier_free)
+		gpusvm->ops->notifier_free(notifier);
+	else
+		kfree(notifier);
+}
+
+/**
+ * to_drm_gpusvm_range() - retrieve the container struct for a

given
rbtree node
+ * @node: a pointer to the rbtree node embedded within a drm_gpusvm_range struct
+ *
+ * Returns: A pointer to the containing drm_gpusvm_range structure.
+ */
+static struct drm_gpusvm_range *to_drm_gpusvm_range(struct rb_node
*node)

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+{
+	return container_of(node, struct drm_gpusvm_range,

itree.rb);

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+}
+
+/**
+ * drm_gpusvm_range_insert() - Insert GPU SVM range
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function inserts the GPU SVM range into the notifier RB

tree
and list.
+ */
+static void drm_gpusvm_range_insert(struct drm_gpusvm_notifier *notifier,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				    struct drm_gpusvm_range

*range)

drivers/gpu/drm/drm_gpusvm.c new file +23 -0
+{
+	struct rb_node *node;
+	struct list_head *head;
+
+	drm_gpusvm_notifier_lock(notifier->gpusvm);
+	interval_tree_insert(&range->itree, &notifier->root);
+
+	node = rb_prev(&range->itree.rb);
+	if (node)
+		head = &(to_drm_gpusvm_range(node))->entry;
+	else
+		head = &notifier->range_list;
+
+	list_add(&range->entry, head);
+	drm_gpusvm_notifier_unlock(notifier->gpusvm);
+}
+
+/**
+ * __drm_gpusvm_range_remove() - Remove GPU SVM range
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This macro removes the GPU SVM range from the notifier RB

tree
and list.
+ */
+static void __drm_gpusvm_range_remove(struct drm_gpusvm_notifier *notifier,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				      struct drm_gpusvm_range

*range)

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+{
+	interval_tree_remove(&range->itree, &notifier->root);
+	list_del(&range->entry);
+}
+
+/**
+ * drm_gpusvm_range_alloc() - Allocate GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @fault_addr: Fault address
+ * @chunk_size: Chunk size
+ * @migrate_devmem: Flag indicating whether to migrate device

memory
+ *
+ * This function allocates and initializes the GPU SVM range structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM range on success, ERR_PTR() on
failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+static struct drm_gpusvm_range *
+drm_gpusvm_range_alloc(struct drm_gpusvm *gpusvm,
+		       struct drm_gpusvm_notifier *notifier,
+		       unsigned long fault_addr, unsigned long

chunk_size,

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+		       bool migrate_devmem)
+{
+	struct drm_gpusvm_range *range;
+
+	if (gpusvm->ops->range_alloc)
+		range = gpusvm->ops->range_alloc(gpusvm);
+	else
+		range = kzalloc(sizeof(*range), GFP_KERNEL);
+
+	if (!range)
+		return ERR_PTR(-ENOMEM);
+
+	kref_init(&range->refcount);
+	range->gpusvm = gpusvm;
+	range->notifier = notifier;
+	range->itree.start = ALIGN_DOWN(fault_addr, chunk_size);
+	range->itree.last = ALIGN(fault_addr + 1, chunk_size) -

1;

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+	INIT_LIST_HEAD(&range->entry);
+	range->notifier_seq = LONG_MAX;
+	range->flags.migrate_devmem = migrate_devmem ? 1 : 0;
+
+	return range;
+}
+
+/**
+ * drm_gpusvm_check_pages() - Check pages
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @start: Start address
+ * @end: End address
+ *
+ * Check if pages between start and end have been faulted in on

the
CPU. Use to
+ * prevent migration of pages without CPU backing store.
+ *
+ * Returns:
+ * True if pages have been faulted into CPU, False otherwise
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+static bool drm_gpusvm_check_pages(struct drm_gpusvm *gpusvm,
+				   struct drm_gpusvm_notifier

*notifier,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				   unsigned long start, unsigned

long end)

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+{
+	struct hmm_range hmm_range = {
+		.default_flags = 0,
+		.notifier = &notifier->notifier,
+		.start = start,
+		.end = end,
+		.dev_private_owner = gpusvm-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+	};
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+	unsigned long *pfns;
+	unsigned long npages = npages_in_range(start, end);
+	int err, i;
+
+	mmap_assert_locked(gpusvm->mm);
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns),

GFP_KERNEL);

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+	if (!pfns)
+		return false;
+
+	hmm_range.notifier_seq =

mmu_interval_read_begin(&notifier-

Quoted 1 line

notifier);

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+	hmm_range.hmm_pfns = pfns;
+
+	while (true) {
+		err = hmm_range_fault(&hmm_range);
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq =
+				mmu_interval_read_begin(&notifie

r-

Quoted 1 line

notifier);

drivers/gpu/drm/drm_gpusvm.c new file +22 -0
+			continue;
+		}
+		break;
+	}
+	if (err)
+		goto err_free;
+
+	for (i = 0; i < npages;) {
+		if (!(pfns[i] & HMM_PFN_VALID)) {
+			err = -EFAULT;
+			goto err_free;
+		}
+		i += 0x1 << hmm_pfn_to_map_order(pfns[i]);
+	}
+
+err_free:
+	kvfree(pfns);
+	return err ? false : true;
+}
+
+/**
+ * drm_gpusvm_range_chunk_size() - Determine chunk size for GPU

SVM
range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @vas: Pointer to the virtual memory area structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @check_pages_threshold: Check CPU pages for present threshold
+ *
+ * This function determines the chunk size for the GPU SVM range based on the
+ * fault address, GPU SVM chunk sizes, existing GPU SVM ranges, and
the virtual
+ * memory area boundaries.
+ *
+ * Returns:
+ * Chunk size on success, LONG_MAX on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+static unsigned long
+drm_gpusvm_range_chunk_size(struct drm_gpusvm *gpusvm,
+			    struct drm_gpusvm_notifier

*notifier,

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+			    struct vm_area_struct *vas,
+			    unsigned long fault_addr,
+			    unsigned long gpuva_start,
+			    unsigned long gpuva_end,
+			    unsigned long check_pages_threshold)
+{
+	unsigned long start, end;
+	int i = 0;
+
+retry:
+	for (; i < gpusvm->num_chunks; ++i) {
+		start = ALIGN_DOWN(fault_addr, gpusvm-
Quoted 1 line

chunk_sizes[i]);

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		end = ALIGN(fault_addr + 1, gpusvm-
Quoted 1 line

chunk_sizes[i]);

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+
+		if (start >= vas->vm_start && end <= vas->vm_end

&&

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+		    start >= notifier->itree.start &&
+		    end <= notifier->itree.last + 1 &&
+		    start >= gpuva_start && end <= gpuva_end)
+			break;
+	}
+
+	if (i == gpusvm->num_chunks)
+		return LONG_MAX;
+
+	/*
+	 * If allocation more than page, ensure not to overlap

with
existing

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+	 * ranges.
+	 */
+	if (end - start != SZ_4K) {
+		struct drm_gpusvm_range *range;
+
+		range = drm_gpusvm_range_find(notifier, start,

end);

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+		if (range) {
+			++i;
+			goto retry;
+		}
+
+		/*
+		 * XXX: Only create range on pages CPU has

faulted
in. Without

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * this check, or prefault, on BMG

'xe_exec_system_allocator --r

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * process-many-malloc' fails. In the failure

case,
each process

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * mallocs 16k but the CPU VMA is ~128k which

results in 64k SVM

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * ranges. When migrating the SVM ranges, some

processes fail in

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * drm_gpusvm_migrate_to_devmem with

'migrate.cpages
!= npages'

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * and then upon drm_gpusvm_range_get_pages

device
pages from

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * other processes are collected + faulted in

which
creates all

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * sorts of problems. Unsure exactly how this

happening, also

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * problem goes away if

'xe_exec_system_allocator --
r

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+		 * process-many-malloc' mallocs at least 64k at

a
time.

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+		 */
+		if (end - start <= check_pages_threshold &&
+		    !drm_gpusvm_check_pages(gpusvm, notifier,

start,
end)) {

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+			++i;
+			goto retry;
+		}
+	}
+
+	return end - start;
+}
+
+/**
+ * drm_gpusvm_range_find_or_insert() - Find or insert GPU SVM

range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @ctx: GPU SVM context
+ *
+ * This function finds or inserts a newly allocated a GPU SVM range
based on the
+ * fault address. Caller must hold a lock to protect range lookup
and insertion.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm,
+				unsigned long fault_addr,
+				unsigned long gpuva_start,
+				unsigned long gpuva_end,
+				const struct drm_gpusvm_ctx

*ctx)

drivers/gpu/drm/drm_gpusvm.c new file +22 -0
+{
+	struct drm_gpusvm_notifier *notifier;
+	struct drm_gpusvm_range *range;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	bool notifier_alloc = false;
+	unsigned long chunk_size;
+	int err;
+	bool migrate_devmem;
+
+	drm_gpusvm_driver_lock_held(gpusvm);
+
+	if (fault_addr < gpusvm->mm_start ||
+	    fault_addr > gpusvm->mm_start + gpusvm->mm_range)
+		return ERR_PTR(-EINVAL);
+
+	if (!mmget_not_zero(mm))
+		return ERR_PTR(-EFAULT);
+
+	notifier = drm_gpusvm_notifier_find(gpusvm, fault_addr);
+	if (!notifier) {
+		notifier = drm_gpusvm_notifier_alloc(gpusvm,

fault_addr);

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+		if (IS_ERR(notifier)) {
+			err = PTR_ERR(notifier);
+			goto err_mmunlock;
+		}
+		notifier_alloc = true;
+		err = mmu_interval_notifier_insert(&notifier-
Quoted 1 line

notifier,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						   mm, notifier-
Quoted 1 line

itree.start,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						   notifier-
Quoted 1 line

itree.last -

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						   notifier-
Quoted 1 line

itree.start + 1,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						  

&drm_gpusvm_notifier_ops);

drivers/gpu/drm/drm_gpusvm.c new file +18 -0
+		if (err)
+			goto err_notifier;
+	}
+
+	mmap_read_lock(mm);
+
+	vas = vma_lookup(mm, fault_addr);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_notifier_remove;
+	}
+
+	if (!ctx->read_only && !(vas->vm_flags & VM_WRITE)) {
+		err = -EPERM;
+		goto err_notifier_remove;
+	}
+
+	range = drm_gpusvm_range_find(notifier, fault_addr,

fault_addr + 1);

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+	if (range)
+		goto out_mmunlock;
+	/*
+	 * XXX: Short-circuiting migration based on

migrate_vma_*
current

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	 * limitations. If/when migrate_vma_* add more support,

this
logic will

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+	 * have to change.
+	 */
+	migrate_devmem = ctx->devmem_possible &&
+		vma_is_anonymous(vas) &&

!is_vm_hugetlb_page(vas);

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+
+	chunk_size = drm_gpusvm_range_chunk_size(gpusvm,

notifier,
vas,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						 fault_addr,

gpuva_start,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+						 gpuva_end,
+						 ctx-
Quoted 1 line

check_pages_threshold);

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+	if (chunk_size == LONG_MAX) {
+		err = -EINVAL;
+		goto err_notifier_remove;
+	}
+
+	range = drm_gpusvm_range_alloc(gpusvm, notifier,

fault_addr,
chunk_size,

drivers/gpu/drm/drm_gpusvm.c new file +20 -0
+				       migrate_devmem);
+	if (IS_ERR(range)) {
+		err = PTR_ERR(range);
+		goto err_notifier_remove;
+	}
+
+	drm_gpusvm_range_insert(notifier, range);
+	if (notifier_alloc)
+		drm_gpusvm_notifier_insert(gpusvm, notifier);
+
+out_mmunlock:
+	mmap_read_unlock(mm);
+	mmput(mm);
+
+	return range;
+
+err_notifier_remove:
+	mmap_read_unlock(mm);
+	if (notifier_alloc)
+		mmu_interval_notifier_remove(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+err_notifier:
+	if (notifier_alloc)
+		drm_gpusvm_notifier_free(gpusvm, notifier);
+err_mmunlock:
+	mmput(mm);
+	return ERR_PTR(err);
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_find_or_insert);
+
+/**
+ * __drm_gpusvm_range_unmap_pages() - Unmap pages associated

with a
GPU SVM range (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @npages: Number of pages to unmap
+ *
+ * This function unmap pages associated with a GPU SVM range. Assumes and
+ * asserts correct locking is in place when called.
+ */
+static void __drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					   struct

drm_gpusvm_range
*range,

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+					   unsigned long npages)
+{
+	unsigned long i, j;
+	struct drm_pagemap *dpagemap = range->dpagemap;
+	struct device *dev = gpusvm->drm->dev;
+
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	if (range->flags.has_dma_mapping) {
+		for (i = 0, j = 0; i < npages; j++) {
+			struct drm_pagemap_dma_addr *addr =

&range-

Quoted 1 line

dma_addr[j];

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+
+			if (addr->proto ==

DRM_INTERCONNECT_SYSTEM)

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+				dma_unmap_page(dev,
+					       addr->addr,
+					       PAGE_SIZE <<

addr-

Quoted 1 line

order,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					       addr->dir);
+			else if (dpagemap && dpagemap->ops-
Quoted 1 line

unmap_dma)

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				dpagemap->ops-
Quoted 1 line

unmap_dma(dpagemap,

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+							 dev,
+							 *addr);
+			i += 1 << addr->order;
+		}
+		range->flags.has_devmem_pages = false;
+		range->flags.has_dma_mapping = false;
+		range->dpagemap = NULL;
+	}
+}
+
+/**
+ * drm_gpusvm_range_free_pages() - Free pages associated with a

GPU
SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function frees the dma address array associated with a GPU
SVM range.
+ */
+static void drm_gpusvm_range_free_pages(struct drm_gpusvm *gpusvm,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					struct drm_gpusvm_range

*range)

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+{
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	if (range->dma_addr) {
+		kvfree(range->dma_addr);
+		range->dma_addr = NULL;
+	}
+}
+
+/**
+ * drm_gpusvm_range_remove() - Remove GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range to be removed
+ *
+ * This function removes the specified GPU SVM range and also

removes the parent
+ * GPU SVM notifier if no more ranges remain in the notifier. The
caller must
+ * hold a lock to protect range and notifier removal.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range)
+{
+	unsigned long npages = npages_in_range(range-
Quoted 1 line

itree.start,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					       range->itree.last
+

1);

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+	struct drm_gpusvm_notifier *notifier;
+
+	drm_gpusvm_driver_lock_held(gpusvm);
+
+	notifier = drm_gpusvm_notifier_find(gpusvm, range-
Quoted 1 line

itree.start);

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+	if (WARN_ON_ONCE(!notifier))
+		return;
+
+	drm_gpusvm_notifier_lock(gpusvm);
+	__drm_gpusvm_range_unmap_pages(gpusvm, range, npages);
+	drm_gpusvm_range_free_pages(gpusvm, range);
+	__drm_gpusvm_range_remove(notifier, range);
+	drm_gpusvm_notifier_unlock(gpusvm);
+
+	drm_gpusvm_range_put(range);
+
+	if (RB_EMPTY_ROOT(&notifier->root.rb_root)) {
+		if (!notifier->flags.removed)
+			mmu_interval_notifier_remove(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+		drm_gpusvm_notifier_remove(gpusvm, notifier);
+		drm_gpusvm_notifier_free(gpusvm, notifier);
+	}
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_remove);
+
+/**
+ * drm_gpusvm_range_get() - Get a reference to GPU SVM range
+ * @range: Pointer to the GPU SVM range
+ *
+ * This function increments the reference count of the specified

GPU
SVM range.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range)
+{
+	kref_get(&range->refcount);
+
+	return range;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_get);
+
+/**
+ * drm_gpusvm_range_destroy() - Destroy GPU SVM range
+ * @refcount: Pointer to the reference counter embedded in the

GPU
SVM range
+ *
+ * This function destroys the specified GPU SVM range when its reference count
+ * reaches zero. If a custom range-free function is provided, it is
invoked to
+ * free the range; otherwise, the range is deallocated using kfree().
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_range_destroy(struct kref *refcount)
+{
+	struct drm_gpusvm_range *range =
+		container_of(refcount, struct drm_gpusvm_range,

refcount);

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+	struct drm_gpusvm *gpusvm = range->gpusvm;
+
+	if (gpusvm->ops->range_free)
+		gpusvm->ops->range_free(range);
+	else
+		kfree(range);
+}
+
+/**
+ * drm_gpusvm_range_put() - Put a reference to GPU SVM range
+ * @range: Pointer to the GPU SVM range
+ *
+ * This function decrements the reference count of the specified

GPU
SVM range
+ * and frees it when the count reaches zero.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range)
+{
+	kref_put(&range->refcount, drm_gpusvm_range_destroy);
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_put);
+
+/**
+ * drm_gpusvm_range_pages_valid() - GPU SVM range pages valid
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid.

Expected be
+ * called holding gpusvm->notifier_lock and as the last step before
committing a
+ * GPU binding. This is akin to a notifier seqno check in the HMM
documentation
+ * but due to wider notifiers (i.e., notifiers which span multiple
ranges) this
+ * function is required for finer grained checking (i.e., per range)
if pages
+ * are valid.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range

*range)

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+{
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	return range->flags.has_devmem_pages || range-
Quoted 1 line

flags.has_dma_mapping;

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_pages_valid);
+
+/**
+ * drm_gpusvm_range_pages_valid_unlocked() - GPU SVM range pages

valid unlocked
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid. Expected be
+ * called without holding gpusvm->notifier_lock.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+static bool
+drm_gpusvm_range_pages_valid_unlocked(struct drm_gpusvm *gpusvm,
+				      struct drm_gpusvm_range

*range)

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+{
+	bool pages_valid;
+
+	if (!range->dma_addr)
+		return false;
+
+	drm_gpusvm_notifier_lock(gpusvm);
+	pages_valid = drm_gpusvm_range_pages_valid(gpusvm,

range);

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+	if (!pages_valid)
+		drm_gpusvm_range_free_pages(gpusvm, range);
+	drm_gpusvm_notifier_unlock(gpusvm);
+
+	return pages_valid;
+}
+
+/**
+ * drm_gpusvm_range_get_pages() - Get pages for a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function gets pages for a GPU SVM range and ensures they

are
mapped for
+ * DMA access.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	struct mmu_interval_notifier *notifier = &range-
Quoted 2 lines

notifier-
notifier;

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+	struct hmm_range hmm_range = {
+		.default_flags = HMM_PFN_REQ_FAULT | (ctx-
Quoted 1 line

read_only

? 0 :

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+			HMM_PFN_REQ_WRITE),
+		.notifier = notifier,
+		.start = range->itree.start,
+		.end = range->itree.last + 1,
+		.dev_private_owner = gpusvm-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	struct drm_gpusvm_zdd *zdd;
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+	unsigned long i, j;
+	unsigned long npages = npages_in_range(range-
Quoted 1 line

itree.start,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					       range->itree.last
+

1);

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+	unsigned long num_dma_mapped;
+	unsigned int order = 0;
+	unsigned long *pfns;
+	struct page **pages;
+	int err = 0;
+	struct dev_pagemap *pagemap;
+	struct drm_pagemap *dpagemap;
+
+retry:
+	hmm_range.notifier_seq =

mmu_interval_read_begin(notifier);

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	if (drm_gpusvm_range_pages_valid_unlocked(gpusvm,

range))

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+		goto set_seqno;
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns),

GFP_KERNEL);

drivers/gpu/drm/drm_gpusvm.c new file +20 -0
+	if (!pfns)
+		return -ENOMEM;
+
+	if (!mmget_not_zero(mm)) {
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	hmm_range.hmm_pfns = pfns;
+	while (true) {
+		mmap_read_lock(mm);
+		err = hmm_range_fault(&hmm_range);
+		mmap_read_unlock(mm);
+
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq =
+				mmu_interval_read_begin(notifier

);

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+			continue;
+		}
+		break;
+	}
+	mmput(mm);
+	if (err)
+		goto err_free;
+
+	pages = (struct page **)pfns;
+map_pages:
+	/*
+	 * Perform all dma mappings under the notifier lock to

not

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+	 * access freed pages. A notifier will either block on
+	 * the notifier lock or unmap dma.
+	 */
+	drm_gpusvm_notifier_lock(gpusvm);
+
+	if (range->flags.unmapped) {
+		drm_gpusvm_notifier_unlock(gpusvm);
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	if (mmu_interval_read_retry(notifier,

hmm_range.notifier_seq)) {

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+		drm_gpusvm_notifier_unlock(gpusvm);
+		kvfree(pfns);
+		goto retry;
+	}
+
+	if (!range->dma_addr) {
+		/* Unlock and restart mapping to allocate

memory. */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+		drm_gpusvm_notifier_unlock(gpusvm);
+		range->dma_addr = kvmalloc_array(npages,
+						 sizeof(*range-
Quoted 1 line

dma_addr),

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+						 GFP_KERNEL);
+		if (!range->dma_addr) {
+			err = -ENOMEM;
+			goto err_free;
+		}
+		goto map_pages;
+	}
+
+	zdd = NULL;
+	num_dma_mapped = 0;
+	for (i = 0, j = 0; i < npages; ++j) {
+		struct page *page = hmm_pfn_to_page(pfns[i]);
+
+		order = hmm_pfn_to_map_order(pfns[i]);
+		if (is_device_private_page(page) ||
+		    is_device_coherent_page(page)) {
+			if (zdd != page->zone_device_data && i >

0)
{

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+				err = -EOPNOTSUPP;
+				goto err_unmap;
+			}
+			zdd = page->zone_device_data;
+			if (pagemap != page->pgmap) {
+				if (i > 0) {
+					err = -EOPNOTSUPP;
+					goto err_unmap;
+				}
+
+				pagemap = page->pgmap;
+				dpagemap = zdd-
Quoted 2 lines

devmem_allocation-
dpagemap;

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				if (drm_WARN_ON(gpusvm->drm,

!dpagemap)) {

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					/*
+					 * Raced. This is not

supposed to happen

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					 * since

hmm_range_fault()
should've migrated

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+					 * this page to system.
+					 */
+					err = -EAGAIN;
+					goto err_unmap;
+				}
+			}
+			range->dma_addr[j] =
+				dpagemap->ops->map_dma(dpagemap,
+						       gpusvm-
Quoted 2 lines

drm-
dev,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						       page,

order,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						      

DMA_BIDIRECTIONAL);

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+			if (dma_mapping_error(gpusvm->drm->dev,
+					      range-
Quoted 1 line

dma_addr[j].addr)) {

drivers/gpu/drm/drm_gpusvm.c new file +18 -0
+				err = -EFAULT;
+				goto err_unmap;
+			}
+
+			pages[i] = page;
+		} else {
+			dma_addr_t addr;
+
+			if (is_zone_device_page(page) || zdd) {
+				err = -EOPNOTSUPP;
+				goto err_unmap;
+			}
+
+			addr = dma_map_page(gpusvm->drm->dev,
+					    page, 0,
+					    PAGE_SIZE << order,
+					    DMA_BIDIRECTIONAL);
+			if (dma_mapping_error(gpusvm->drm->dev,

addr)) {

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+				err = -EFAULT;
+				goto err_unmap;
+			}
+
+			range->dma_addr[j] =

drm_pagemap_dma_addr_encode

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				(addr, DRM_INTERCONNECT_SYSTEM,

order,

drivers/gpu/drm/drm_gpusvm.c new file +21 -0
+				 DMA_BIDIRECTIONAL);
+		}
+		i += 1 << order;
+		num_dma_mapped = i;
+	}
+
+	range->flags.has_dma_mapping = true;
+	if (zdd) {
+		range->flags.has_devmem_pages = true;
+		range->dpagemap = dpagemap;
+	}
+
+	drm_gpusvm_notifier_unlock(gpusvm);
+	kvfree(pfns);
+set_seqno:
+	range->notifier_seq = hmm_range.notifier_seq;
+
+	return 0;
+
+err_unmap:
+	__drm_gpusvm_range_unmap_pages(gpusvm, range,

num_dma_mapped);

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+	drm_gpusvm_notifier_unlock(gpusvm);
+err_free:
+	kvfree(pfns);
+	if (err == -EAGAIN)
+		goto retry;
+	return err;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_get_pages);
+
+/**
+ * drm_gpusvm_range_unmap_pages() - Unmap pages associated with

a
GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If
@in_notifier
+ * is set, it is assumed that gpusvm->notifier_lock is held in write
mode; if it
+ * is clear, it acquires gpusvm->notifier_lock in read mode. Must be
called on
+ * each GPU SVM range attached to notifier in gpusvm->ops-

Quoted 1 line

invalidate for IOMMU

+ * security model.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range

*range,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				  const struct drm_gpusvm_ctx

*ctx)

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+{
+	unsigned long npages = npages_in_range(range-
Quoted 1 line

itree.start,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					       range->itree.last
+

1);

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+
+	if (ctx->in_notifier)
+		lockdep_assert_held_write(&gpusvm-
Quoted 1 line

notifier_lock);

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+	else
+		drm_gpusvm_notifier_lock(gpusvm);
+
+	__drm_gpusvm_range_unmap_pages(gpusvm, range, npages);
+
+	if (!ctx->in_notifier)
+		drm_gpusvm_notifier_unlock(gpusvm);
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_unmap_pages);
+
+/**
+ * drm_gpusvm_migration_unlock_put_page() - Put a migration page
+ * @page: Pointer to the page to put
+ *
+ * This function unlocks and puts a page.
+ */
+static void drm_gpusvm_migration_unlock_put_page(struct page

*page)

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+{
+	unlock_page(page);
+	put_page(page);
+}
+
+/**
+ * drm_gpusvm_migration_unlock_put_pages() - Put migration pages
+ * @npages: Number of pages
+ * @migrate_pfn: Array of migrate page frame numbers
+ *
+ * This function unlocks and puts an array of pages.
+ */
+static void drm_gpusvm_migration_unlock_put_pages(unsigned long

npages,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						  unsigned long

*migrate_pfn)

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page;
+
+		if (!migrate_pfn[i])
+			continue;
+
+		page = migrate_pfn_to_page(migrate_pfn[i]);
+		drm_gpusvm_migration_unlock_put_page(page);
+		migrate_pfn[i] = 0;
+	}
+}
+
+/**
+ * drm_gpusvm_get_devmem_page() - Get a reference to a device

memory
page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU
SVM zone
+ * device data and initializes it for zone device usage.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+static void drm_gpusvm_get_devmem_page(struct page *page,
+				     struct drm_gpusvm_zdd *zdd)
+{
+	page->zone_device_data = drm_gpusvm_zdd_get(zdd);
+	zone_device_page_init(page);
+}
+
+/**
+ * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU

SVM
migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped
pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn,
maps the
+ * corresponding page, and stores the DMA address in the provided
@dma_addr
+ * array.
+ *
+ * Returns: 0 on success, -EFAULT if an error occurs during mapping.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+static int drm_gpusvm_migrate_map_pages(struct device *dev,
+					dma_addr_t *dma_addr,
+					unsigned long

*migrate_pfn,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					unsigned long npages,
+					enum dma_data_direction

dir)

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page =

migrate_pfn_to_page(migrate_pfn[i]);

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+
+		if (!page)
+			continue;
+
+		if (WARN_ON_ONCE(is_zone_device_page(page)))
+			return -EFAULT;
+
+		dma_addr[i] = dma_map_page(dev, page, 0,

PAGE_SIZE,
dir);

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+		if (dma_mapping_error(dev, dma_addr[i]))
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_migrate_unmap_pages() - Unmap pages previously

mapped
for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU
Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_migrate_unmap_pages(struct device *dev,
+					   dma_addr_t *dma_addr,
+					   unsigned long npages,
+					   enum

dma_data_direction
dir)

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!dma_addr[i] || dma_mapping_error(dev,

dma_addr[i]))

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+			continue;
+
+		dma_unmap_page(dev, dma_addr[i], PAGE_SIZE,

dir);

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+	}
+}
+
+/**
+ * drm_gpusvm_migrate_to_devmem() - Migrate GPU SVM range to

device
memory
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @devmem_allocation: Pointer to the device memory allocation. The
caller
+ *                     should hold a reference to the device memory
allocation,
+ *                     which should be dropped via ops-

Quoted 1 line

devmem_release or upon

+ *                     the failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to device memory. It performs the
+ * necessary setup and invokes the driver-specific operations for
migration to
+ * device memory. Upon successful return, @devmem_allocation can safely reference @range
+ * until ops->devmem_release is called which only upon successful
return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+int drm_gpusvm_migrate_to_devmem(struct drm_gpusvm *gpusvm,
+				 struct drm_gpusvm_range *range,
+				 struct drm_gpusvm_devmem

*devmem_allocation,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				 const struct drm_gpusvm_ctx

*ctx)

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+{
+	const struct drm_gpusvm_devmem_ops *ops =

devmem_allocation-

Quoted 1 line

ops;

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	unsigned long start = range->itree.start, end = range-
Quoted 1 line

itree.last + 1;

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+	struct migrate_vma migrate = {
+		.start		= start,
+		.end		= end,
+		.pgmap_owner	= gpusvm-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +41 -0
+		.flags		= MIGRATE_VMA_SELECT_SYSTEM,
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	unsigned long i, npages = npages_in_range(start, end);
+	struct vm_area_struct *vas;
+	struct drm_gpusvm_zdd *zdd = NULL;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int err;
+
+	if (!range->flags.migrate_devmem)
+		return -EINVAL;
+
+	if (!ops->populate_devmem_pfn || !ops->copy_to_devmem ||
+	    !ops->copy_to_ram)
+		return -EOPNOTSUPP;
+
+	if (!mmget_not_zero(mm)) {
+		err = -EFAULT;
+		goto err_out;
+	}
+	mmap_read_lock(mm);
+
+	vas = vma_lookup(mm, start);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end > vas->vm_end || start < vas->vm_start) {
+		err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	if (!vma_is_anonymous(vas)) {
+		err = -EBUSY;
+		goto err_mmunlock;
+	}
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +

sizeof(*dma_addr) +

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_mmunlock;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) +

sizeof(*dma_addr))
* npages;

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+
+	zdd = drm_gpusvm_zdd_alloc(gpusvm-
Quoted 1 line

device_private_page_owner);

drivers/gpu/drm/drm_gpusvm.c new file +24 -0
+	if (!zdd) {
+		err = -ENOMEM;
+		goto err_free;
+	}
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	if (!migrate.cpages) {
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	if (migrate.cpages != npages) {
+		err = -EBUSY;
+		goto err_finalize;
+	}
+
+	err = ops->populate_devmem_pfn(devmem_allocation,

npages,
migrate.dst);

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(devmem_allocation-
Quoted 1 line

dev,

dma_addr,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					   migrate.src, npages,

DMA_TO_DEVICE);

drivers/gpu/drm/drm_gpusvm.c new file +16 -0
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = pfn_to_page(migrate.dst[i]);
+
+		pages[i] = page;
+		migrate.dst[i] = migrate_pfn(migrate.dst[i]);
+		drm_gpusvm_get_devmem_page(page, zdd);
+	}
+
+	err = ops->copy_to_devmem(pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+	/* Upon success bind devmem allocation to range and zdd

*/

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+	zdd->devmem_allocation = devmem_allocation;	/* Owns

ref
*/

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_unlock_put_pages(npages,

migrate.dst);

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(devmem_allocation->dev,

dma_addr, npages,

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+				       DMA_TO_DEVICE);
+err_free:
+	if (zdd)
+		drm_gpusvm_zdd_put(zdd);
+	kvfree(buf);
+err_mmunlock:
+	mmap_read_unlock(mm);
+	mmput(mm);
+err_out:
+	return err;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_migrate_to_devmem);
+
+/**
+ * drm_gpusvm_migrate_populate_ram_pfn() - Populate RAM PFNs for

a
VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @fault_page: Fault page
+ * @npages: Number of pages to populate
+ * @mpages: Number of pages to migrate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the RAM migrate page frame numbers (PFNs)
for the
+ * specified VM area structure. It allocates and locks pages in the
VM area for
+ * RAM usage. If vas is non-NULL use alloc_page_vma for allocation,
if NULL use
+ * alloc_page for allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_ram_pfn(struct vm_area_struct
*vas,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					       struct page

*fault_page,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					       unsigned long

npages,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					       unsigned long

*mpages,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					       unsigned long

*src_mpfn,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					       unsigned long

*mpfn,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					       unsigned long

addr)

drivers/gpu/drm/drm_gpusvm.c new file +21 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i, addr += PAGE_SIZE) {
+		struct page *page, *src_page;
+
+		if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE))
+			continue;
+
+		src_page = migrate_pfn_to_page(src_mpfn[i]);
+		if (!src_page)
+			continue;
+
+		if (fault_page) {
+			if (src_page->zone_device_data !=
+			    fault_page->zone_device_data)
+				continue;
+		}
+
+		if (vas)
+			page = alloc_page_vma(GFP_HIGHUSER, vas,

addr);

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+		else
+			page = alloc_page(GFP_HIGHUSER);
+
+		if (!page)
+			goto free_pages;
+
+		mpfn[i] = migrate_pfn(page_to_pfn(page));
+	}
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page =

migrate_pfn_to_page(mpfn[i]);

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+
+		if (!page)
+			continue;
+
+		WARN_ON_ONCE(!trylock_page(page));
+		++*mpages;
+	}
+
+	return 0;
+
+free_pages:
+	for (i = 0; i < npages; ++i) {
+		struct page *page =

migrate_pfn_to_page(mpfn[i]);

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+
+		if (!page)
+			continue;
+
+		put_page(page);
+		mpfn[i] = 0;
+	}
+	return -ENOMEM;
+}
+
+/**
+ * drm_gpusvm_evict_to_ram() - Evict GPU SVM range to RAM
+ * @devmem_allocation: Pointer to the device memory allocation
+ *
+ * Similar to __drm_gpusvm_migrate_to_ram but does not require

mmap
lock and
+ * migration done via migrate_device_* functions.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_evict_to_ram(struct drm_gpusvm_devmem *devmem_allocation)

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+{
+	const struct drm_gpusvm_devmem_ops *ops =

devmem_allocation-

Quoted 1 line

ops;

drivers/gpu/drm/drm_gpusvm.c new file +15 -0
+	unsigned long npages, mpages = 0;
+	struct page **pages;
+	unsigned long *src, *dst;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+	unsigned int retry_count = 2;
+
+	npages = devmem_allocation->size >> PAGE_SHIFT;
+
+retry:
+	if (!mmget_not_zero(devmem_allocation->mm))
+		return -EFAULT;
+
+	buf = kvcalloc(npages, 2 * sizeof(*src) +

sizeof(*dma_addr)

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	src = buf;
+	dst = buf + (sizeof(*src) * npages);
+	dma_addr = buf + (2 * sizeof(*src) * npages);
+	pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) *

npages;

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+
+	err = ops->populate_devmem_pfn(devmem_allocation,

npages,
src);

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+	if (err)
+		goto err_free;
+
+	err = migrate_device_pfns(src, npages);
+	if (err)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_ram_pfn(NULL, NULL,

npages, &mpages,

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+						  src, dst, 0);
+	if (err || !mpages)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(devmem_allocation-
Quoted 1 line

dev,

dma_addr,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					   dst, npages,

DMA_FROM_DEVICE);

drivers/gpu/drm/drm_gpusvm.c new file +13 -0
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(src[i]);
+
+	err = ops->copy_to_ram(pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_unlock_put_pages(npages,

dst);

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+	migrate_device_pages(src, dst, npages);
+	migrate_device_finalize(src, dst, npages);
+	drm_gpusvm_migrate_unmap_pages(devmem_allocation->dev,

dma_addr, npages,

drivers/gpu/drm/drm_gpusvm.c new file +20 -0
+				       DMA_FROM_DEVICE);
+err_free:
+	kvfree(buf);
+err_out:
+	mmput_async(devmem_allocation->mm);
+
+	if (completion_done(&devmem_allocation->detached))
+		return 0;
+
+	if (!err || retry_count--) {
+		cond_resched();
+		goto retry;
+	}
+
+	return err;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_evict_to_ram);
+
+/**
+ * __drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM

(internal)
+ * @vas: Pointer to the VM area structure
+ * @device_private_page_owner: Device private pages owner
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @fault_addr: Fault address
+ * @size: Size of migration
+ *
+ * This internal function performs the migration of the specified
GPU SVM range
+ * to RAM. It sets up the migration, populates + dma maps RAM PFNs,
and
+ * invokes the driver-specific operations for migration to RAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int __drm_gpusvm_migrate_to_ram(struct vm_area_struct *vas,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				       void

*device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+				       struct page *page,
+				       unsigned long fault_addr,
+				       unsigned long size)
+{
+	struct migrate_vma migrate = {
+		.vma		= vas,
+		.pgmap_owner	= device_private_page_owner,
+		.flags		=

MIGRATE_VMA_SELECT_DEVICE_PRIVATE

Quoted 1 line
drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+			MIGRATE_VMA_SELECT_DEVICE_COHERENT,
+		.fault_page	= page,
+	};
+	struct drm_gpusvm_zdd *zdd;
+	const struct drm_gpusvm_devmem_ops *ops;
+	struct device *dev;
+	unsigned long npages, mpages = 0;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	unsigned long start, end;
+	void *buf;
+	int i, err = 0;
+
+	start = ALIGN_DOWN(fault_addr, size);
+	end = ALIGN(fault_addr + 1, size);
+
+	/* Corner where VMA area struct has been partially

unmapped
*/

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+	if (start < vas->vm_start)
+		start = vas->vm_start;
+	if (end > vas->vm_end)
+		end = vas->vm_end;
+
+	migrate.start = start;
+	migrate.end = end;
+	npages = npages_in_range(start, end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +

sizeof(*dma_addr) +

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) +

sizeof(*dma_addr))
* npages;

drivers/gpu/drm/drm_gpusvm.c new file +16 -0
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/* Raced with another CPU fault, nothing to do */
+	if (!migrate.cpages)
+		goto err_free;
+
+	if (!page) {
+		for (i = 0; i < npages; ++i) {
+			if (!(migrate.src[i] &

MIGRATE_PFN_MIGRATE))

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+				continue;
+
+			page =

migrate_pfn_to_page(migrate.src[i]);

drivers/gpu/drm/drm_gpusvm.c new file +11 -0
+			break;
+		}
+
+		if (!page)
+			goto err_finalize;
+	}
+	zdd = page->zone_device_data;
+	ops = zdd->devmem_allocation->ops;
+	dev = zdd->devmem_allocation->dev;
+
+	err = drm_gpusvm_migrate_populate_ram_pfn(vas, page,

npages,
&mpages,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+						  migrate.src,

migrate.dst,

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+						  start);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(dev, dma_addr,

migrate.dst, npages,

drivers/gpu/drm/drm_gpusvm.c new file +14 -0
+					   DMA_FROM_DEVICE);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(migrate.src[i]);
+
+	err = ops->copy_to_ram(pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_unlock_put_pages(npages,

migrate.dst);

drivers/gpu/drm/drm_gpusvm.c new file +17 -0
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(dev, dma_addr, npages,
+				       DMA_FROM_DEVICE);
+err_free:
+	kvfree(buf);
+err_out:
+
+	return err;
+}
+
+/**
+ * drm_gpusvm_range_evict - Evict GPU SVM range
+ * @pagemap: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range to be removed
+ *
+ * This function evicts the specified GPU SVM range. This

function
will not
+ * evict coherent pages.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+int drm_gpusvm_range_evict(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_range *range)
+{
+	struct mmu_interval_notifier *notifier = &range-
Quoted 2 lines

notifier-
notifier;

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+	struct hmm_range hmm_range = {
+		.default_flags = HMM_PFN_REQ_FAULT,
+		.notifier = notifier,
+		.start = range->itree.start,
+		.end = range->itree.last + 1,
+		.dev_private_owner = NULL,
+	};
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+	unsigned long *pfns;
+	unsigned long npages = npages_in_range(range-
Quoted 1 line

itree.start,

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+					       range->itree.last
+

1);

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+	int err = 0;
+	struct mm_struct *mm = gpusvm->mm;
+
+	if (!mmget_not_zero(mm))
+		return -EFAULT;
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns),

GFP_KERNEL);

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+	if (!pfns)
+		return -ENOMEM;
+
+	hmm_range.hmm_pfns = pfns;
+	while (!time_after(jiffies, timeout)) {
+		hmm_range.notifier_seq =

mmu_interval_read_begin(notifier);

drivers/gpu/drm/drm_gpusvm.c new file +21 -0
+		if (time_after(jiffies, timeout)) {
+			err = -ETIME;
+			break;
+		}
+
+		mmap_read_lock(mm);
+		err = hmm_range_fault(&hmm_range);
+		mmap_read_unlock(mm);
+		if (err != -EBUSY)
+			break;
+	}
+
+	kvfree(pfns);
+	mmput(mm);
+
+	return err;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_evict);
+
+/**
+ * drm_gpusvm_page_free() - Put GPU SVM zone device data

associated
with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device
data
+ * associated with a page when it is being released.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+static void drm_gpusvm_page_free(struct page *page)
+{
+	drm_gpusvm_zdd_put(page->zone_device_data);
+}
+
+/**
+ * drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM

(page
fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM
range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page
and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ * VM_FAULT_SIGBUS on failure, 0 on success.
+ */
+static vm_fault_t drm_gpusvm_migrate_to_ram(struct vm_fault *vmf)

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+{
+	struct drm_gpusvm_zdd *zdd = vmf->page-
Quoted 1 line

zone_device_data;

drivers/gpu/drm/drm_gpusvm.c new file +4 -0
+	int err;
+
+	err = __drm_gpusvm_migrate_to_ram(vmf->vma,
+					  zdd-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					  vmf->page, vmf-
Quoted 1 line

address,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+					  zdd-
Quoted 2 lines

devmem_allocation-
size);

drivers/gpu/drm/drm_gpusvm.c new file +6 -0
+
+	return err ? VM_FAULT_SIGBUS : 0;
+}
+
+/**
+ * drm_gpusvm_pagemap_ops() - Device page map operations for GPU

SVM
+ */

drivers/gpu/drm/drm_gpusvm.c new file +7 -0
+static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = {
+	.page_free = drm_gpusvm_page_free,
+	.migrate_to_ram = drm_gpusvm_migrate_to_ram,
+};
+
+/**
+ * drm_gpusvm_pagemap_ops_get() - Retrieve GPU SVM device page

map
operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */

drivers/gpu/drm/drm_gpusvm.c new file +8 -0
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void)
+{
+	return &drm_gpusvm_pagemap_ops;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_pagemap_ops_get);
+
+/**
+ * drm_gpusvm_has_mapping() - Check if GPU SVM has mapping for

the
given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, unsigned long
start,

drivers/gpu/drm/drm_gpusvm.c new file +5 -0
+			    unsigned long end)
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	drm_gpusvm_for_each_notifier(notifier, gpusvm, start,

end) {

drivers/gpu/drm/drm_gpusvm.c new file +3 -0
+		struct drm_gpusvm_range *range = NULL;
+
+		drm_gpusvm_for_each_range(range, notifier,

start,
end)

drivers/gpu/drm/drm_gpusvm.c new file +9 -0
+			return true;
+	}
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_has_mapping);
+
+/**
+ * drm_gpusvm_range_set_unmapped() - Mark a GPU SVM range as

unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets the partial_unmap flag
+ * if the range partially falls within the provided MMU notifier range.
+ */
+void drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+				   const struct

mmu_notifier_range
*mmu_range)

drivers/gpu/drm/drm_gpusvm.c new file +2 -0
+{
+	lockdep_assert_held_write(&range->gpusvm-
Quoted 1 line

notifier_lock);

drivers/gpu/drm/drm_gpusvm.c new file +10 -0
+
+	range->flags.unmapped = true;
+	if (range->itree.start < mmu_range->start ||
+	    range->itree.last + 1 > mmu_range->end)
+		range->flags.partial_unmap = true;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_range_set_unmapped);
+
+/**
+ * drm_gpusvm_devmem_init() - Initialize a GPU SVM device memory

allocation
+ *
+ * @dev: Pointer to the device structure which device memory allocation belongs to
+ * @mm: Pointer to the mm_struct for the address space
+ * @ops: Pointer to the operations structure for GPU SVM device memory
+ * @dpagemap: The struct drm_pagemap we're allocating from.
+ * @size: Size of device memory allocation
+ */
+void drm_gpusvm_devmem_init(struct drm_gpusvm_devmem *devmem_allocation,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+			    struct device *dev, struct mm_struct

*mm,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+			    const struct drm_gpusvm_devmem_ops

*ops,

drivers/gpu/drm/drm_gpusvm.c new file +1 -0
+			    struct drm_pagemap *dpagemap, size_t

size)

drivers/gpu/drm/drm_gpusvm.c new file +12 -0
+{
+	init_completion(&devmem_allocation->detached);
+	devmem_allocation->dev = dev;
+	devmem_allocation->mm = mm;
+	devmem_allocation->ops = ops;
+	devmem_allocation->dpagemap = dpagemap;
+	devmem_allocation->size = size;
+}
+EXPORT_SYMBOL_GPL(drm_gpusvm_devmem_init);
+
+MODULE_DESCRIPTION("DRM GPUSVM");
+MODULE_LICENSE("GPL");
include/drm/drm_gpusvm.h new file +24 -0
new file mode 100644
index 000000000000..ea31db0be841
@@ -0,0 +1,445 @@
+/* SPDX-License-Identifier: GPL-2.0-only OR MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef __DRM_GPUSVM_H__
+#define __DRM_GPUSVM_H__
+
+#include <linux/kref.h>
+#include <linux/interval_tree.h>
+#include <linux/mmu_notifier.h>
+
+struct dev_pagemap_ops;
+struct drm_device;
+struct drm_gpusvm;
+struct drm_gpusvm_notifier;
+struct drm_gpusvm_ops;
+struct drm_gpusvm_range;
+struct drm_gpusvm_devmem;
+struct drm_pagemap;
+struct drm_pagemap_dma_addr;
+
+/**
+ * struct drm_gpusvm_devmem_ops - Operations structure for GPU

SVM
device memory
+ *
+ * This structure defines the operations for GPU Shared Virtual Memory (SVM)
+ * device memory. These operations are provided by the GPU driver to
manage device memory
+ * allocations and perform operations such as migration between device memory and system
+ * RAM.
+ */

include/drm/drm_gpusvm.h new file +3 -0
+struct drm_gpusvm_devmem_ops {
+	/**
+	 * @devmem_release: Release device memory allocation

(optional)

include/drm/drm_gpusvm.h new file +3 -0
+	 * @devmem_allocation: device memory allocation
+	 *
+	 * Release device memory allocation and drop a reference

to
device

include/drm/drm_gpusvm.h new file +3 -0
+	 * memory allocation.
+	 */
+	void (*devmem_release)(struct drm_gpusvm_devmem

*devmem_allocation);

include/drm/drm_gpusvm.h new file +3 -0
+
+	/**
+	 * @populate_devmem_pfn: Populate device memory PFN

(required for migration)

include/drm/drm_gpusvm.h new file +10 -0
+	 * @devmem_allocation: device memory allocation
+	 * @npages: Number of pages to populate
+	 * @pfn: Array of page frame numbers to populate
+	 *
+	 * Populate device memory page frame numbers (PFN).
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*populate_devmem_pfn)(struct drm_gpusvm_devmem

*devmem_allocation,

include/drm/drm_gpusvm.h new file +1 -0
+				   unsigned long npages,

unsigned
long *pfn);

include/drm/drm_gpusvm.h new file +3 -0
+
+	/**
+	 * @copy_to_devmem: Copy to device memory (required for

migration)

include/drm/drm_gpusvm.h new file +1 -0
+	 * @pages: Pointer to array of device memory pages

(destination)

include/drm/drm_gpusvm.h new file +14 -0
+	 * @dma_addr: Pointer to array of DMA addresses (source)
+	 * @npages: Number of pages to copy
+	 *
+	 * Copy pages to device memory.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_devmem)(struct page **pages,
+			      dma_addr_t *dma_addr,
+			      unsigned long npages);
+
+	/**
+	 * @copy_to_ram: Copy to system RAM (required for

migration)

include/drm/drm_gpusvm.h new file +1 -0
+	 * @pages: Pointer to array of device memory pages

(source)

include/drm/drm_gpusvm.h new file +1 -0
+	 * @dma_addr: Pointer to array of DMA addresses

(destination)

include/drm/drm_gpusvm.h new file +14 -0
+	 * @npages: Number of pages to copy
+	 *
+	 * Copy pages to system RAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_ram)(struct page **pages,
+			   dma_addr_t *dma_addr,
+			   unsigned long npages);
+};
+
+/**
+ * struct drm_gpusvm_devmem - Structure representing a GPU SVM

device memory allocation
+ *
+ * @dev: Pointer to the device structure which device memory allocation belongs to
+ * @mm: Pointer to the mm_struct for the address space
+ * @detached: device memory allocations is detached from device pages
+ * @ops: Pointer to the operations structure for GPU SVM device memory
+ * @dpagemap: The struct drm_pagemap of the pages this allocation
belongs to.
+ * @size: Size of device memory allocation
+ */

include/drm/drm_gpusvm.h new file +13 -0
+struct drm_gpusvm_devmem {
+	struct device *dev;
+	struct mm_struct *mm;
+	struct completion detached;
+	const struct drm_gpusvm_devmem_ops *ops;
+	struct drm_pagemap *dpagemap;
+	size_t size;
+};
+
+/**
+ * struct drm_gpusvm_ops - Operations structure for GPU SVM
+ *
+ * This structure defines the operations for GPU Shared Virtual

Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * notifiers.
+ */

include/drm/drm_gpusvm.h new file +3 -0
+struct drm_gpusvm_ops {
+	/**
+	 * @notifier_alloc: Allocate a GPU SVM notifier

(optional)

include/drm/drm_gpusvm.h new file +5 -0
+	 *
+	 * Allocate a GPU SVM notifier.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM notifier on success,

NULL on failure.

include/drm/drm_gpusvm.h new file +6 -0
+	 */
+	struct drm_gpusvm_notifier *(*notifier_alloc)(void);
+
+	/**
+	 * @notifier_free: Free a GPU SVM notifier (optional)
+	 * @notifier: Pointer to the GPU SVM notifier to be

freed

include/drm/drm_gpusvm.h new file +4 -0
+	 *
+	 * Free a GPU SVM notifier.
+	 */
+	void (*notifier_free)(struct drm_gpusvm_notifier

*notifier);

include/drm/drm_gpusvm.h new file +9 -0
+
+	/**
+	 * @range_alloc: Allocate a GPU SVM range (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 *
+	 * Allocate a GPU SVM range.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM range on success,

NULL
on failure.

include/drm/drm_gpusvm.h new file +2 -0
+	 */
+	struct drm_gpusvm_range *(*range_alloc)(struct

drm_gpusvm
*gpusvm);

include/drm/drm_gpusvm.h new file +14 -0
+
+	/**
+	 * @range_free: Free a GPU SVM range (optional)
+	 * @range: Pointer to the GPU SVM range to be freed
+	 *
+	 * Free a GPU SVM range.
+	 */
+	void (*range_free)(struct drm_gpusvm_range *range);
+
+	/**
+	 * @invalidate: Invalidate GPU SVM notifier (required)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @notifier: Pointer to the GPU SVM notifier
+	 * @mmu_range: Pointer to the mmu_notifier_range

structure

include/drm/drm_gpusvm.h new file +2 -0
+	 *
+	 * Invalidate the GPU page tables. It can safely walk

the
notifier range

include/drm/drm_gpusvm.h new file +1 -0
+	 * RB tree/list in this function. Called while holding

the
notifier lock.

include/drm/drm_gpusvm.h new file +4 -0
+	 */
+	void (*invalidate)(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_notifier *notifier,
+			   const struct mmu_notifier_range

*mmu_range);

include/drm/drm_gpusvm.h new file +4 -0
+};
+
+/**
+ * struct drm_gpusvm_notifier - Structure representing a GPU SVM

notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @itree: Interval tree node for the notifier (inserted in GPU SVM)
+ * @entry: List entry to fast interval tree traversal
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ *              interval tree. This is useful to keep iterating ranges while
+ *              doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier
has been
+ *                 removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */

include/drm/drm_gpusvm.h new file +14 -0
+struct drm_gpusvm_notifier {
+	struct drm_gpusvm *gpusvm;
+	struct mmu_interval_notifier notifier;
+	struct interval_tree_node itree;
+	struct list_head entry;
+	struct rb_root_cached root;
+	struct list_head range_list;
+	struct {
+		u32 removed : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm_range - Structure representing a GPU SVM

range
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier
+ * @refcount: Reference count for the range
+ * @itree: Interval tree node for the range (inserted in GPU SVM notifier)
+ * @entry: List entry to fast interval tree traversal
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @dma_addr: DMA address array
+ * @dpagemap: The struct drm_pagemap of the device pages we're dma-
mapping.
+ *            Note this is assuming only one drm_pagemap per range
is allowed.
+ * @flags.migrate_devmem: Flag indicating whether the range can be
migrated to device memory
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_devmem_pages: Flag indicating if the range has devmem
pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA
mapping
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */

include/drm/drm_gpusvm.h new file +13 -0
+struct drm_gpusvm_range {
+	struct drm_gpusvm *gpusvm;
+	struct drm_gpusvm_notifier *notifier;
+	struct kref refcount;
+	struct interval_tree_node itree;
+	struct list_head entry;
+	unsigned long notifier_seq;
+	struct drm_pagemap_dma_addr *dma_addr;
+	struct drm_pagemap *dpagemap;
+	struct {
+		/* All flags below must be set upon creation */
+		u16 migrate_devmem : 1;
+		/* All flags below must be set / cleared under

notifier lock */

include/drm/drm_gpusvm.h new file +19 -0
+		u16 unmapped : 1;
+		u16 partial_unmap : 1;
+		u16 has_devmem_pages : 1;
+		u16 has_dma_mapping : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm - GPU SVM structure
+ *
+ * @name: Name of the GPU SVM
+ * @drm: Pointer to the DRM device structure
+ * @mm: Pointer to the mm_struct for the address space
+ * @device_private_page_owner: Device private pages owner
+ * @mm_start: Start address of GPU SVM
+ * @mm_range: Range of the GPU SVM
+ * @notifier_size: Size of individual notifiers
+ * @ops: Pointer to the operations structure for GPU SVM
+ * @chunk_sizes: Pointer to the array of chunk sizes used in

30 lines not shown · raw message

include/drm/drm_gpusvm.h new file +17 -0
+struct drm_gpusvm {
+	const char *name;
+	struct drm_device *drm;
+	struct mm_struct *mm;
+	void *device_private_page_owner;
+	unsigned long mm_start;
+	unsigned long mm_range;
+	unsigned long notifier_size;
+	const struct drm_gpusvm_ops *ops;
+	const unsigned long *chunk_sizes;
+	int num_chunks;
+	struct rw_semaphore notifier_lock;
+	struct rb_root_cached root;
+	struct list_head notifier_list;
+#ifdef CONFIG_LOCKDEP
+	/**
[… more lines in the raw message]

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

2 lines not shown · raw message

include/drm/drm_gpusvm.h new file +10 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

12 lines not shown · raw message

include/drm/drm_gpusvm.h new file +10 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +1 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +3 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +11 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +14 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +4 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +3 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +1 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +4 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +1 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +2 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +4 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

2 lines not shown · raw message

include/drm/drm_gpusvm.h new file +4 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

2 lines not shown · raw message

include/drm/drm_gpusvm.h new file +3 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +1 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

2 lines not shown · raw message

include/drm/drm_gpusvm.h new file +2 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +1 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +1 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +1 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +4 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

10 lines not shown · raw message

include/drm/drm_gpusvm.h new file +4 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +1 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

1 line not shown · raw message

include/drm/drm_gpusvm.h new file +5 -0

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Quoted 5 lines

3 lines not shown · raw message

Quoted 8 lines

6 lines not shown · raw message

3 lines not shown · raw message

Parts of this message are not shown here. View the raw message.