Skip to contentSkip to conversation

Introduce GPU SVM and Xe SVM implementation

RFC PATCH Superseded Matthew Brost 8 participants 100 messages Aug 28, 2024 – Oct 16, 2024 Page 1 of 6

Subsystems (8)

v1 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Matthew Brost

author maintainer
#0 lore raw

cover Introduce GPU SVM and Xe SVM implementation

Continuation of SVM work by Oak Zeng [1][2] based on community feedback. Introduces GPU SVM layer and new Xe uAPI. Supports GPU page faults for system allocations (e.g., malloc), runtime allocations (e.g., binding a BO), migration to and from VRAM, and unified eviction (BO and SVM VRAM allocations can evict each other). Fully tested; more on this below.

The patch breakdown is as follows:

1. Preparation patches already on the list [3].
- Patches 1-3.
- Please refrain from reviewing these here.
2. New migrate layer functionality
- Patch 4.
- Required for eviction to avoid locking inversion between dma-resv and mmap lock.
3. GPU SVM.
- Patch 5.
- This is what needs community review.
- Inspired by GPUVM.
- Kernel doc should explain design principles.
- There is certainly room for optimization of the implementation and improvements with existing core MM interaction. Pulling in pending DMA mapping work [4] and additional core MM support for SVM is also likely desired. However, this serves as a good starting point for any SVM discussions and could be used as a stepping stone to future core MM work.
3. Basic SVM support in Xe (i.e., SRAM backing only).
- Patches 6-15.
- The uAPI in the patch could benefit from community input.
4. SVM VRAM migration support in Xe.
- Patches 16-23.
- Using TMM BOs for SVM VRAM allocations could use community input. Patch 23 has a detailed explaination of this design choice in the commit message.
5. SVM eviction support in Xe.
- Patch 24.
- Should work with exhaustive eviction [5] when it merges.
6. Xe SVM debug / tuning.
- Patch 25-28.

Kernel documentation and commit messages are relatively light, aside from GPU SVM and uAPI patches as this is an RFC.

Testing has been conducted quite thoroughly with new IGT [6]. Various system allocation types (malloc, mmap, mmap flags, huge pages, different sizes, different alignments), mixing runtime allocations, unmapping corners, invalid faults, and eviction have been tested. Testing scales from single thread to multiple threads and multiple processes. Tests pass on LNL, BMG, PVC 1 tile, and PVC 2 tile.

1. Multiple GPU support.
- This is likely to follow or occur in parallel to this work.
2. Userptr unification with GPU SVM.
- This is essentially designed in my head (likely involving a few new GPU SVM layer functions) but would require some fairly invasive changes to Xe KMD to test out. Therefore, I would like GPU SVM to be reviewed first before proceeding with these changes.
3. Madvise and prefetch IOCTLs
- This is likely to follow or occur in parallel to this work.

Given the size of the series, I have pushed a GitLab branch for reference [7].

Matt

[1] https://patchwork.freedesktop.org/series/128910/
[2] https://patchwork.freedesktop.org/series/132229/
[3] https://patchwork.freedesktop.org/series/137805/
[4] https://lore.kernel.org/linux-rdma/[email protected]/
[5] https://patchwork.freedesktop.org/series/133643/
[6] https://patchwork.freedesktop.org/patch/610942/?series=137545&rev=2
[7] https://gitlab.freedesktop.org/mbrost/xe-kernel-driver-svm-post-8-27-24/-/tree/post?ref_type=heads

Matthew Brost (28):
dma-buf: Split out dma fence array create into alloc and arm functions drm/xe: Invalidate media_gt TLBs in PT code
drm/xe: Retry BO allocation
mm/migrate: Add migrate_device_vma_range
drm/gpusvm: Add support for GPU Shared Virtual Memory drm/xe/uapi: Add DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATON flag drm/xe: Add SVM init / fini to faulting VMs
drm/xe: Add dma_addr res cursor
drm/xe: Add SVM range invalidation
drm/gpuvm: Add DRM_GPUVA_OP_USER
drm/xe: Add (re)bind to SVM page fault handler
drm/xe: Add SVM garbage collector
drm/xe: Add unbind to SVM garbage collector
drm/xe: Do not allow system allocator VMA unbind if the GPU has
bindings
drm/xe: Enable system allocator uAPI
drm/xe: Add migrate layer functions for SVM support drm/xe: Add SVM device memory mirroring
drm/xe: Add GPUSVM copy SRAM / VRAM vfunc functions drm/xe: Update PT layer to understand ranges in VRAM drm/xe: Add Xe SVM populate_vram_pfn vfunc
drm/xe: Add Xe SVM vram_release vfunc
drm/xe: Add BO flags required for SVM
drm/xe: Add SVM VRAM migration
drm/xe: Basic SVM BO eviction
drm/xe: Add SVM debug
drm/xe: Add modparam for SVM notifier size
drm/xe: Add modparam for SVM prefault
drm/gpusvm: Ensure all pages migrated upon eviction

 drivers/dma-buf/dma-fence-array.c    |   78 +-
 drivers/gpu/drm/xe/Makefile          |    4 +-
 drivers/gpu/drm/xe/drm_gpusvm.c      | 2213 ++++++++++++++++++++++++++
 drivers/gpu/drm/xe/drm_gpusvm.h      |  415 +++++
 drivers/gpu/drm/xe/xe_bo.c           |   54 +-
 drivers/gpu/drm/xe/xe_bo.h           |    2 +
 drivers/gpu/drm/xe/xe_bo_types.h     |    3 +
 drivers/gpu/drm/xe/xe_device_types.h |    8 +
 drivers/gpu/drm/xe/xe_gt_pagefault.c |   17 +-
 drivers/gpu/drm/xe/xe_migrate.c      |  150 ++
 drivers/gpu/drm/xe/xe_migrate.h      |   10 +
 drivers/gpu/drm/xe/xe_module.c       |    7 +
 drivers/gpu/drm/xe/xe_module.h       |    2 +
 drivers/gpu/drm/xe/xe_pt.c           |  456 +++++-
 drivers/gpu/drm/xe/xe_pt.h           |    3 +
 drivers/gpu/drm/xe/xe_pt_types.h     |    2 +
 drivers/gpu/drm/xe/xe_res_cursor.h   |   50 +-
 drivers/gpu/drm/xe/xe_svm.c          |  775 +++++++++
 drivers/gpu/drm/xe/xe_svm.h          |   70 +
 drivers/gpu/drm/xe/xe_tile.c         |    5 +
 drivers/gpu/drm/xe/xe_vm.c           |  286 +++-
 drivers/gpu/drm/xe/xe_vm.h           |   15 +-
 drivers/gpu/drm/xe/xe_vm_types.h     |   44 +
 include/drm/drm_gpuvm.h              |    5 +
 include/linux/dma-fence-array.h      |    6 +
 include/linux/migrate.h              |    3 +
 include/uapi/drm/xe_drm.h            |   19 +-
 mm/migrate_device.c                  |   53 +
 28 files changed, 4615 insertions(+), 140 deletions(-)
 create mode 100644 drivers/gpu/drm/xe/drm_gpusvm.c
 create mode 100644 drivers/gpu/drm/xe/drm_gpusvm.h
 create mode 100644 drivers/gpu/drm/xe/xe_svm.c
 create mode 100644 drivers/gpu/drm/xe/xe_svm.h
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

RFC PATCH 1/28 dma-buf: Split out dma fence array create into alloc and arm functions

Useful to preallocate dma fence array and then arm in path of reclaim or a dma fence.

v2:
- s/arm/init (Christian)
- Drop !array warn (Christian)

Cc Sumit Semwal <[email protected]>
Cc Christian König <[email protected]>
---
 drivers/dma-buf/dma-fence-array.c | 78 ++++++++++++++++++++++---------
 include/linux/dma-fence-array.h   |  6 +++
 2 files changed, 63 insertions(+), 21 deletions(-)
drivers/dma-buf/dma-fence-array.c +57 -21
index c74ac197d5fe..0659e6b29b3c 100644
@@ -144,37 +144,38 @@ const struct dma_fence_ops dma_fence_array_ops = {
 EXPORT_SYMBOL(dma_fence_array_ops);
 
 /**
- * dma_fence_array_create - Create a custom fence array
+ * dma_fence_array_alloc - Allocate a custom fence array
+ * @num_fences:		[in]	number of fences to add in the array
+ *
+ * Return dma fence array on success, NULL on failure
+ */
+struct dma_fence_array *dma_fence_array_alloc(int num_fences)
+{
+	struct dma_fence_array *array;
+
+	return kzalloc(struct_size(array, callbacks, num_fences), GFP_KERNEL);
+}
+EXPORT_SYMBOL(dma_fence_array_alloc);
+
+/**
+ * dma_fence_array_init - Arm a custom fence array
+ * @array:		[in]	dma fence array to arm
  * @num_fences:		[in]	number of fences to add in the array
  * @fences:		[in]	array containing the fences
  * @context:		[in]	fence context to use
  * @seqno:		[in]	sequence number to use
  * @signal_on_any:	[in]	signal on any fence in the array
  *
- * Allocate a dma_fence_array object and initialize the base fence with
- * dma_fence_init().
- * In case of error it returns NULL.
- *
- * The caller should allocate the fences array with num_fences size
- * and fill it with the fences it wants to add to the object. Ownership of this
- * array is taken and dma_fence_put() is used on each fence on release.
- *
- * If @signal_on_any is true the fence array signals if any fence in the array
- * signals, otherwise it signals when all fences in the array signal.
+ * Implementation of @dma_fence_array_create without allocation. Useful to arm a
+ * preallocated dma fence fence in the path of reclaim or dma fence signaling.
  */
-struct dma_fence_array *dma_fence_array_create(int num_fences,
-					       struct dma_fence **fences,
-					       u64 context, unsigned seqno,
-					       bool signal_on_any)
+void dma_fence_array_init(struct dma_fence_array *array,
+			  int num_fences, struct dma_fence **fences,
+			  u64 context, unsigned seqno,
+			  bool signal_on_any)
 {
-	struct dma_fence_array *array;
-
 	WARN_ON(!num_fences || !fences);
 
-	array = kzalloc(struct_size(array, callbacks, num_fences), GFP_KERNEL);
-	if (!array)
-		return NULL;
-
 	array->num_fences = num_fences;
 
 	spin_lock_init(&array->lock);
@@ -200,6 +201,41 @@ struct dma_fence_array *dma_fence_array_create(int num_fences,
 	 */
 	while (num_fences--)
 		WARN_ON(dma_fence_is_container(fences[num_fences]));
+}
+EXPORT_SYMBOL(dma_fence_array_init);
+
+/**
+ * dma_fence_array_create - Create a custom fence array
+ * @num_fences:		[in]	number of fences to add in the array
+ * @fences:		[in]	array containing the fences
+ * @context:		[in]	fence context to use
+ * @seqno:		[in]	sequence number to use
+ * @signal_on_any:	[in]	signal on any fence in the array
+ *
+ * Allocate a dma_fence_array object and initialize the base fence with
+ * dma_fence_init().
+ * In case of error it returns NULL.
+ *
+ * The caller should allocate the fences array with num_fences size
+ * and fill it with the fences it wants to add to the object. Ownership of this
+ * array is taken and dma_fence_put() is used on each fence on release.
+ *
+ * If @signal_on_any is true the fence array signals if any fence in the array
+ * signals, otherwise it signals when all fences in the array signal.
+ */
+struct dma_fence_array *dma_fence_array_create(int num_fences,
+					       struct dma_fence **fences,
+					       u64 context, unsigned seqno,
+					       bool signal_on_any)
+{
+	struct dma_fence_array *array;
+
+	array = dma_fence_array_alloc(num_fences);
+	if (!array)
+		return NULL;
+
+	dma_fence_array_init(array, num_fences, fences,
+			     context, seqno, signal_on_any);
 
 	return array;
 }
include/linux/dma-fence-array.h +6 -0
index 29c5650c1038..079b3dec0a16 100644
@@ -79,6 +79,12 @@ to_dma_fence_array(struct dma_fence *fence)
 	for (index = 0, fence = dma_fence_array_first(head); fence;	\
 	     ++(index), fence = dma_fence_array_next(head, index))
 
+struct dma_fence_array *dma_fence_array_alloc(int num_fences);
+void dma_fence_array_init(struct dma_fence_array *array,
+			  int num_fences, struct dma_fence **fences,
+			  u64 context, unsigned seqno,
+			  bool signal_on_any);
+
 struct dma_fence_array *dma_fence_array_create(int num_fences,
 					       struct dma_fence **fences,
 					       u64 context, unsigned seqno,
Signature · 1 line
2.34.1
author maintainer
#2 lore raw

RFC PATCH 2/28 drm/xe: Invalidate media_gt TLBs in PT code

In reply to Matthew Brost

Testing on LNL has shown media GT's TLBs need to be invalidated via the GuC, update PT code appropriately.

v2:
- Do dma_fence_get before first call of invalidation_fence_init (Himal)
- No need to check for valid chain fence (Himal)
v3:
- Use dma-fence-array

Fixes 3330361543fc ("drm/xe/lnl: Add LNL platform definition")
---
 drivers/gpu/drm/xe/xe_pt.c | 117 ++++++++++++++++++++++++++++++-------
 1 file changed, 96 insertions(+), 21 deletions(-)
drivers/gpu/drm/xe/xe_pt.c +96 -21
index 579ed31b46db..d6353e8969f0 100644
@@ -3,6 +3,8 @@
  * Copyright © 2022 Intel Corporation
  */
 
+#include <linux/dma-fence-array.h>
+
 #include "xe_pt.h"
 
 #include "regs/xe_gtt_defs.h"
@@ -1627,9 +1629,11 @@ xe_pt_update_ops_rfence_interval(struct xe_vm_pgtable_update_ops *pt_update_ops,
 
 static int vma_reserve_fences(struct xe_device *xe, struct xe_vma *vma)
 {
+	int shift = xe_device_get_root_tile(xe)->media_gt ? 1 : 0;
+
 	if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm)
 		return dma_resv_reserve_fences(xe_vma_bo(vma)->ttm.base.resv,
-					       xe->info.tile_count);
+					       xe->info.tile_count << shift);
 
 	return 0;
 }
@@ -1816,6 +1820,7 @@ int xe_pt_update_ops_prepare(struct xe_tile *tile, struct xe_vma_ops *vops)
 	struct xe_vm_pgtable_update_ops *pt_update_ops =
 		&vops->pt_update_ops[tile->id];
 	struct xe_vma_op *op;
+	int shift = tile->media_gt ? 1 : 0;
 	int err;
 
 	lockdep_assert_held(&vops->vm->lock);
@@ -1824,7 +1829,7 @@ int xe_pt_update_ops_prepare(struct xe_tile *tile, struct xe_vma_ops *vops)
 	xe_pt_update_ops_init(pt_update_ops);
 
 	err = dma_resv_reserve_fences(xe_vm_resv(vops->vm),
-				      tile_to_xe(tile)->info.tile_count);
+				      tile_to_xe(tile)->info.tile_count << shift);
 	if (err)
 		return err;
 
@@ -1849,13 +1854,20 @@ int xe_pt_update_ops_prepare(struct xe_tile *tile, struct xe_vma_ops *vops)
 
 static void bind_op_commit(struct xe_vm *vm, struct xe_tile *tile,
 			   struct xe_vm_pgtable_update_ops *pt_update_ops,
-			   struct xe_vma *vma, struct dma_fence *fence)
+			   struct xe_vma *vma, struct dma_fence *fence,
+			   struct dma_fence *fence2)
 {
-	if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm)
+	if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm) {
 		dma_resv_add_fence(xe_vma_bo(vma)->ttm.base.resv, fence,
 				   pt_update_ops->wait_vm_bookkeep ?
 				   DMA_RESV_USAGE_KERNEL :
 				   DMA_RESV_USAGE_BOOKKEEP);
+		if (fence2)
+			dma_resv_add_fence(xe_vma_bo(vma)->ttm.base.resv, fence2,
+					   pt_update_ops->wait_vm_bookkeep ?
+					   DMA_RESV_USAGE_KERNEL :
+					   DMA_RESV_USAGE_BOOKKEEP);
+	}
 	vma->tile_present |= BIT(tile->id);
 	vma->tile_staged &= ~BIT(tile->id);
 	if (xe_vma_is_userptr(vma)) {
@@ -1875,13 +1887,20 @@ static void bind_op_commit(struct xe_vm *vm, struct xe_tile *tile,
 
 static void unbind_op_commit(struct xe_vm *vm, struct xe_tile *tile,
 			     struct xe_vm_pgtable_update_ops *pt_update_ops,
-			     struct xe_vma *vma, struct dma_fence *fence)
+			     struct xe_vma *vma, struct dma_fence *fence,
+			     struct dma_fence *fence2)
 {
-	if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm)
+	if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm) {
 		dma_resv_add_fence(xe_vma_bo(vma)->ttm.base.resv, fence,
 				   pt_update_ops->wait_vm_bookkeep ?
 				   DMA_RESV_USAGE_KERNEL :
 				   DMA_RESV_USAGE_BOOKKEEP);
+		if (fence2)
+			dma_resv_add_fence(xe_vma_bo(vma)->ttm.base.resv, fence2,
+					   pt_update_ops->wait_vm_bookkeep ?
+					   DMA_RESV_USAGE_KERNEL :
+					   DMA_RESV_USAGE_BOOKKEEP);
+	}
 	vma->tile_present &= ~BIT(tile->id);
 	if (!vma->tile_present) {
 		list_del_init(&vma->combined_links.rebind);
@@ -1898,7 +1917,8 @@ static void unbind_op_commit(struct xe_vm *vm, struct xe_tile *tile,
 static void op_commit(struct xe_vm *vm,
 		      struct xe_tile *tile,
 		      struct xe_vm_pgtable_update_ops *pt_update_ops,
-		      struct xe_vma_op *op, struct dma_fence *fence)
+		      struct xe_vma_op *op, struct dma_fence *fence,
+		      struct dma_fence *fence2)
 {
 	xe_vm_assert_held(vm);
 
@@ -1907,26 +1927,28 @@ static void op_commit(struct xe_vm *vm,
 		if (!op->map.immediate && xe_vm_in_fault_mode(vm))
 			break;
 
-		bind_op_commit(vm, tile, pt_update_ops, op->map.vma, fence);
+		bind_op_commit(vm, tile, pt_update_ops, op->map.vma, fence,
+			       fence2);
 		break;
 	case DRM_GPUVA_OP_REMAP:
 		unbind_op_commit(vm, tile, pt_update_ops,
-				 gpuva_to_vma(op->base.remap.unmap->va), fence);
+				 gpuva_to_vma(op->base.remap.unmap->va), fence,
+				 fence2);
 
 		if (op->remap.prev)
 			bind_op_commit(vm, tile, pt_update_ops, op->remap.prev,
-				       fence);
+				       fence, fence2);
 		if (op->remap.next)
 			bind_op_commit(vm, tile, pt_update_ops, op->remap.next,
-				       fence);
+				       fence, fence2);
 		break;
 	case DRM_GPUVA_OP_UNMAP:
 		unbind_op_commit(vm, tile, pt_update_ops,
-				 gpuva_to_vma(op->base.unmap.va), fence);
+				 gpuva_to_vma(op->base.unmap.va), fence, fence2);
 		break;
 	case DRM_GPUVA_OP_PREFETCH:
 		bind_op_commit(vm, tile, pt_update_ops,
-			       gpuva_to_vma(op->base.prefetch.va), fence);
+			       gpuva_to_vma(op->base.prefetch.va), fence, fence2);
 		break;
 	default:
 		drm_warn(&vm->xe->drm, "NOT POSSIBLE");
@@ -1963,7 +1985,9 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)
 	struct xe_vm_pgtable_update_ops *pt_update_ops =
 		&vops->pt_update_ops[tile->id];
 	struct dma_fence *fence;
-	struct invalidation_fence *ifence = NULL;
+	struct invalidation_fence *ifence = NULL, *mfence = NULL;
+	struct dma_fence **fences = NULL;
+	struct dma_fence_array *cf = NULL;
 	struct xe_range_fence *rfence;
 	struct xe_vma_op *op;
 	int err = 0, i;
@@ -1996,6 +2020,23 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)
 			err = -ENOMEM;
 			goto kill_vm_tile1;
 		}
+		if (tile->media_gt) {
+			mfence = kzalloc(sizeof(*ifence), GFP_KERNEL);
+			if (!mfence) {
+				err = -ENOMEM;
+				goto free_ifence;
+			}
+			fences = kmalloc_array(2, sizeof(*fences), GFP_KERNEL);
+			if (!fences) {
+				err = -ENOMEM;
+				goto free_ifence;
+			}
+			cf = dma_fence_array_alloc(2);
+			if (!cf) {
+				err = -ENOMEM;
+				goto free_ifence;
+			}
+		}
 	}
 
 	rfence = kzalloc(sizeof(*rfence), GFP_KERNEL);
@@ -2027,19 +2068,50 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)
 
 	/* tlb invalidation must be done before signaling rebind */
 	if (ifence) {
+		if (mfence)
+			dma_fence_get(fence);
 		invalidation_fence_init(tile->primary_gt, ifence, fence,
 					pt_update_ops->start,
 					pt_update_ops->last, vm->usm.asid);
-		fence = &ifence->base.base;
+		if (mfence) {
+			invalidation_fence_init(tile->media_gt, mfence, fence,
+						pt_update_ops->start,
+						pt_update_ops->last, vm->usm.asid);
+			fences[0] = &ifence->base.base;
+			fences[1] = &mfence->base.base;
+			dma_fence_array_init(cf, 2, fences,
+					     vm->composite_fence_ctx,
+					     vm->composite_fence_seqno++,
+					     false);
+			fence = &cf->base;
+		} else {
+			fence = &ifence->base.base;
+		}
 	}
 
-	dma_resv_add_fence(xe_vm_resv(vm), fence,
-			   pt_update_ops->wait_vm_bookkeep ?
-			   DMA_RESV_USAGE_KERNEL :
-			   DMA_RESV_USAGE_BOOKKEEP);
+	if (!mfence) {
+		dma_resv_add_fence(xe_vm_resv(vm), fence,
+				   pt_update_ops->wait_vm_bookkeep ?
+				   DMA_RESV_USAGE_KERNEL :
+				   DMA_RESV_USAGE_BOOKKEEP);
 
-	list_for_each_entry(op, &vops->list, link)
-		op_commit(vops->vm, tile, pt_update_ops, op, fence);
+		list_for_each_entry(op, &vops->list, link)
+			op_commit(vops->vm, tile, pt_update_ops, op, fence, NULL);
+	} else {
+		dma_resv_add_fence(xe_vm_resv(vm), &ifence->base.base,
+				   pt_update_ops->wait_vm_bookkeep ?
+				   DMA_RESV_USAGE_KERNEL :
+				   DMA_RESV_USAGE_BOOKKEEP);
+
+		dma_resv_add_fence(xe_vm_resv(vm), &mfence->base.base,
+				   pt_update_ops->wait_vm_bookkeep ?
+				   DMA_RESV_USAGE_KERNEL :
+				   DMA_RESV_USAGE_BOOKKEEP);
+
+		list_for_each_entry(op, &vops->list, link)
+			op_commit(vops->vm, tile, pt_update_ops, op,
+				  &ifence->base.base, &mfence->base.base);
+	}
 
 	if (pt_update_ops->needs_userptr_lock)
 		up_read(&vm->userptr.notifier_lock);
@@ -2049,6 +2121,9 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)
 free_rfence:
 	kfree(rfence);
 free_ifence:
+	kfree(cf);
+	kfree(fences);
+	kfree(mfence);
 	kfree(ifence);
 kill_vm_tile1:
 	if (err != -EAGAIN && tile->id)
Signature · 1 line
2.34.1
author maintainer
#3 lore raw

RFC PATCH 3/28 drm/xe: Retry BO allocation

In reply to Matthew Brost

TTM doesn't support fair eviction via WW locking, this mitigated in by using retry loops in exec and preempt rebind worker. Extend this retry loop to BO allocation. Once TTM supports fair eviction this patch can be reverted.

---
 drivers/gpu/drm/xe/xe_bo.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_bo.c +7 -1
index cbe7bf098970..b6c6a4a3b4d4 100644
@@ -1977,6 +1977,7 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 	struct xe_file *xef = to_xe_file(file);
 	struct drm_xe_gem_create *args = data;
 	struct xe_vm *vm = NULL;
+	ktime_t end = 0;
 	struct xe_bo *bo;
 	unsigned int bo_flags;
 	u32 handle;
@@ -2042,11 +2043,14 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 		vm = xe_vm_lookup(xef, args->vm_id);
 		if (XE_IOCTL_DBG(xe, !vm))
 			return -ENOENT;
+	}
+
+retry:
+	if (vm) {
 		err = xe_vm_lock(vm, true);
 		if (err)
 			goto out_vm;
 	}
-
 	bo = xe_bo_create_user(xe, NULL, vm, args->size, args->cpu_caching,
 			       bo_flags);
 
@@ -2055,6 +2059,8 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 
 	if (IS_ERR(bo)) {
 		err = PTR_ERR(bo);
+		if (xe_vm_validate_should_retry(NULL, err, &end))
+			goto retry;
 		goto out_vm;
 	}
 
Signature · 1 line
2.34.1
author maintainer
#4 lore raw

RFC PATCH 4/28 mm/migrate: Add migrate_device_vma_range

In reply to Matthew Brost

Add migrate_device_vma_range which prepares an array of pre-populated device pages for migration and issues a MMU invalidation.

---
 include/linux/migrate.h |  3 +++
 mm/migrate_device.c     | 53 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)
include/linux/migrate.h +3 -0
index 644be30b69c8..e8cce05bf9c2 100644
@@ -226,6 +226,9 @@ void migrate_vma_pages(struct migrate_vma *migrate);
 void migrate_vma_finalize(struct migrate_vma *migrate);
 int migrate_device_range(unsigned long *src_pfns, unsigned long start,
 			unsigned long npages);
+int migrate_device_vma_range(struct mm_struct *mm, void *pgmap_owner,
+			     unsigned long *src_pfns, unsigned long npages,
+			     unsigned long start);
 void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns,
 			unsigned long npages);
 void migrate_device_finalize(unsigned long *src_pfns,
mm/migrate_device.c +53 -0

59 lines not on this page · raw message

Signature · 1 line
2.34.1
#5 lore raw

Re: [RFC PATCH 04/28] mm/migrate: Add migrate_device_vma_range

Quoted 50 lines · Matthew Brost

Add migrate_device_vma_range which prepares an array of pre-populated device pages for migration and issues a MMU invalidation.

---
 include/linux/migrate.h |  3 +++
 mm/migrate_device.c     | 53 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)
include/linux/migrate.h +3 -0
index 644be30b69c8..e8cce05bf9c2 100644
@@ -226,6 +226,9 @@ void migrate_vma_pages(struct migrate_vma *migrate);
 void migrate_vma_finalize(struct migrate_vma *migrate);
 int migrate_device_range(unsigned long *src_pfns, unsigned long start,
 			unsigned long npages);
+int migrate_device_vma_range(struct mm_struct *mm, void *pgmap_owner,
+			     unsigned long *src_pfns, unsigned long npages,
+			     unsigned long start);
 void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns,
 			unsigned long npages);
 void migrate_device_finalize(unsigned long *src_pfns,
mm/migrate_device.c +18 -0
index 6d66dc1c6ffa..e25f12a132e8 100644
@@ -920,6 +920,59 @@ int migrate_device_range(unsigned long *src_pfns, unsigned long start,
 }
 EXPORT_SYMBOL(migrate_device_range);
 
+/**
+ * migrate_device_vma_range() - migrate device private pfns to normal memory and
+ * trigger MMU invalidation.
+ * @mm: struct mm of device pages.
+ * @src_pfns: pre-popluated array of source device private pfns to migrate.
+ * @pgmap_owner: page group map owner of device pages.
+ * @npages: number of pages to migrate.
+ * @start: VMA start of device pages.
+ *
+ * Similar to migrate_device_range() but supports non-contiguous pre-popluated
+ * array of device pages to migrate. Also triggers MMU invalidation. Useful in
+ * device memory eviction paths where lock is held protecting the device pages
+ * but where the mmap lock cannot be taken to due to a locking inversion (e.g.
+ * DRM drivers). Since the mmap lock is not required to be held, the MMU
+ * invalidation can race with with VMA start being repurposed, worst case this
+ * would result in an unecessary invalidation.
+ */
+int migrate_device_vma_range(struct mm_struct *mm, void *pgmap_owner,

mm/migrate_device.c int migrate_device_range(unsigned long *src_pfns, unsigned long start, · line 950

Quoted 10 lines
mm/migrate_device.c +10 -0
+			     unsigned long *src_pfns, unsigned long npages,
+			     unsigned long start)
+{
+	struct mmu_notifier_range range;
+	unsigned long i;
+
+	mmu_notifier_range_init_owner(&range, MMU_NOTIFY_MIGRATE, 0,
+				      mm, start, start + npages * PAGE_SIZE,
+				      pgmap_owner);
+	mmu_notifier_invalidate_range_start(&range);

This isn't needed, try_to_migrate called from migrate_device_unmap already has a notifier, if there's actually any ptes to clear. If you need this one you've missed a pte clear notification somewhere, or there's some other bad bug somewhere.
-Sima

Quoted 31 lines
mm/migrate_device.c +25 -0
+
+	for (i = 0; i < npages; i++) {
+		struct page *page = pfn_to_page(src_pfns[i]);
+
+		if (!get_page_unless_zero(page)) {
+			src_pfns[i] = 0;
+			continue;
+		}
+
+		if (!trylock_page(page)) {
+			src_pfns[i] = 0;
+			put_page(page);
+			continue;
+		}
+
+		src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;
+	}
+
+	migrate_device_unmap(src_pfns, npages, NULL);
+	mmu_notifier_invalidate_range_end(&range);
+
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_vma_range);
+
 /*
  * Migrate a device coherent page back to normal memory. The caller should have
  * a reference on page which will be copied to the new page if migration is
Signature · 1 line
2.34.1
Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#6 lore raw

Re: [RFC PATCH 04/28] mm/migrate: Add migrate_device_vma_range

Quoted 58 lines · Daniel Vetter
Quoted 57 lines · Matthew Brost

Add migrate_device_vma_range which prepares an array of pre-populated device pages for migration and issues a MMU invalidation.

---
 include/linux/migrate.h |  3 +++
 mm/migrate_device.c     | 53 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)
include/linux/migrate.h +3 -0
index 644be30b69c8..e8cce05bf9c2 100644
@@ -226,6 +226,9 @@ void migrate_vma_pages(struct migrate_vma *migrate);
 void migrate_vma_finalize(struct migrate_vma *migrate);
 int migrate_device_range(unsigned long *src_pfns, unsigned long start,
 			unsigned long npages);
+int migrate_device_vma_range(struct mm_struct *mm, void *pgmap_owner,
+			     unsigned long *src_pfns, unsigned long npages,
+			     unsigned long start);
 void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns,
 			unsigned long npages);
 void migrate_device_finalize(unsigned long *src_pfns,
mm/migrate_device.c +25 -0
index 6d66dc1c6ffa..e25f12a132e8 100644
@@ -920,6 +920,59 @@ int migrate_device_range(unsigned long *src_pfns, unsigned long start,
 }
 EXPORT_SYMBOL(migrate_device_range);
 
+/**
+ * migrate_device_vma_range() - migrate device private pfns to normal memory and
+ * trigger MMU invalidation.
+ * @mm: struct mm of device pages.
+ * @src_pfns: pre-popluated array of source device private pfns to migrate.
+ * @pgmap_owner: page group map owner of device pages.
+ * @npages: number of pages to migrate.
+ * @start: VMA start of device pages.
+ *
+ * Similar to migrate_device_range() but supports non-contiguous pre-popluated
+ * array of device pages to migrate. Also triggers MMU invalidation. Useful in
+ * device memory eviction paths where lock is held protecting the device pages
+ * but where the mmap lock cannot be taken to due to a locking inversion (e.g.
+ * DRM drivers). Since the mmap lock is not required to be held, the MMU
+ * invalidation can race with with VMA start being repurposed, worst case this
+ * would result in an unecessary invalidation.
+ */
+int migrate_device_vma_range(struct mm_struct *mm, void *pgmap_owner,
+			     unsigned long *src_pfns, unsigned long npages,
+			     unsigned long start)
+{
+	struct mmu_notifier_range range;
+	unsigned long i;
+
+	mmu_notifier_range_init_owner(&range, MMU_NOTIFY_MIGRATE, 0,
Quoted 8 lines

mm/migrate_device.c int migrate_device_range(unsigned long *src_pfns, unsigned long start, · line 950

Quoted 3 lines
mm/migrate_device.c +3 -0
+				      mm, start, start + npages * PAGE_SIZE,
+				      pgmap_owner);
+	mmu_notifier_invalidate_range_start(&range);

This isn't needed, try_to_migrate called from migrate_device_unmap already has a notifier, if there's actually any ptes to clear. If you need this one you've missed a pte clear notification somewhere, or there's some other bad bug somewhere.

Thanks for the tip, let me pull this out and confirm that we get a notifier from try_to_migrate when this function is called. Agree if we do get a notifier, this is not needed.

Matt

Quoted 38 lines

-Sima

Quoted 31 lines
mm/migrate_device.c +25 -0
+
+	for (i = 0; i < npages; i++) {
+		struct page *page = pfn_to_page(src_pfns[i]);
+
+		if (!get_page_unless_zero(page)) {
+			src_pfns[i] = 0;
+			continue;
+		}
+
+		if (!trylock_page(page)) {
+			src_pfns[i] = 0;
+			put_page(page);
+			continue;
+		}
+
+		src_pfns[i] = migrate_pfn(src_pfns[i]) | MIGRATE_PFN_MIGRATE;
+	}
+
+	migrate_device_unmap(src_pfns, npages, NULL);
+	mmu_notifier_invalidate_range_end(&range);
+
+	return 0;
+}
+EXPORT_SYMBOL(migrate_device_vma_range);
+
 /*
  * Migrate a device coherent page back to normal memory. The caller should have
  * a reference on page which will be copied to the new page if migration is
Signature · 1 line
2.34.1
Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#7 lore raw

RFC PATCH 5/28 drm/gpusvm: Add support for GPU Shared Virtual Memory

In reply to Matthew Brost

This patch introduces support for GPU Shared Virtual Memory (SVM) in the Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.

The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also provides mechanisms for allocating, deallocating, and migrating memory regions between system RAM and GPU VRAM.

This mid-layer is largely inspired by GPUVM.

Cc Dave Airlie <[email protected]>
Cc Thomas Hellström <[email protected]>
Cc Christian König <[email protected]>
---
 drivers/gpu/drm/xe/Makefile     |    3 +-
 drivers/gpu/drm/xe/drm_gpusvm.c | 2174 +++++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/drm_gpusvm.h |  415 ++++++
 3 files changed, 2591 insertions(+), 1 deletion(-)
 create mode 100644 drivers/gpu/drm/xe/drm_gpusvm.c
 create mode 100644 drivers/gpu/drm/xe/drm_gpusvm.h
drivers/gpu/drm/xe/Makefile +2 -1
index b9670ae09a9e..b8fc2ee58f1a 100644
@@ -25,7 +25,8 @@ $(obj)/generated/%_wa_oob.c $(obj)/generated/%_wa_oob.h: $(obj)/xe_gen_wa_oob \
 
 # core driver code
 
-xe-y += xe_bb.o \
+xe-y += drm_gpusvm.o \
+	xe_bb.o \
 	xe_bo.o \
 	xe_bo_evict.o \
 	xe_devcoredump.o \
drivers/gpu/drm/xe/drm_gpusvm.c new file +2174 -0

2,174 lines not on this page · raw message

drivers/gpu/drm/xe/drm_gpusvm.h new file +415 -0

415 lines not on this page · raw message

Signature · 1 line
2.34.1
#8 lore raw

Re: [RFC PATCH 05/28] drm/gpusvm: Add support for GPU Shared Virtual Memory

drivers/gpu/drm/xe/drm_gpusvm.c line 1353

Quoted 11 lines · Matthew Brost
drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+		if (!ctx->mmap_locked) {
+			/*
+			 * XXX: HMM locking document indicates only a read-lock
+			 * is required but there apears to be a window between
+			 * the MMU_NOTIFY_MIGRATE event triggered in a CPU fault
+			 * via migrate_vma_setup and the pages actually moving
+			 * in migrate_vma_finalize in which this code can grab
+			 * garbage pages. Grabbing the write-lock if the range
+			 * is attached to vram appears to protect against this
+			 * race.
+			 */

This one is really scary, since it means the entire migrate pte trickery is essentially completely busted. Grabbing the mmap write lock just means you block out pretty much everything interesting from concurrently happening.

My gut feeling says we need to figure out what's happening here, because this looks a bit too fundamental to me.
-Sima

Quoted 1,245 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +821 -0
+			if (vram_pages)
+				mmap_write_lock(mm);
+			else
+				mmap_read_lock(mm);
+		}
+		err = hmm_range_fault(&hmm_range);
+		if (!ctx->mmap_locked) {
+			if (vram_pages)
+				mmap_write_unlock(mm);
+			else
+				mmap_read_unlock(mm);
+		}
+
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
+			continue;
+		}
+		break;
+	}
+	if (!ctx->mmap_locked)
+		mmput(mm);
+	if (err)
+		goto err_free;
+
+	pages = (struct page **)pfns;
+
+	if (ctx->prefault) {
+		range->pages = pages;
+		goto set_seqno;
+	}
+
+map_pages:
+	if (is_device_private_page(hmm_pfn_to_page(pfns[0]))) {
+		WARN_ON_ONCE(!range->vram_allocation);
+
+		for (i = 0; i < npages; ++i) {
+			pages[i] = hmm_pfn_to_page(pfns[i]);
+
+			if (WARN_ON_ONCE(!is_device_private_page(pages[i]))) {
+				err = -EOPNOTSUPP;
+				goto err_free;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->flags.has_vram_pages = true;
+		range->pages = pages;
+		if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm, range);
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	} else {
+		dma_addr_t *dma_addr = (dma_addr_t *)pfns;
+
+		for_each_dma_page(i, j, npages, order) {
+			if (WARN_ON_ONCE(i && order !=
+					 hmm_pfn_to_map_order(pfns[i]))) {
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+			order = hmm_pfn_to_map_order(pfns[i]);
+
+			pages[j] = hmm_pfn_to_page(pfns[i]);
+			if (WARN_ON_ONCE(is_zone_device_page(pages[j]))) {
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+
+			set_page_dirty_lock(pages[j]);
+			mark_page_accessed(pages[j]);
+
+			dma_addr[j] = dma_map_page(gpusvm->drm->dev,
+						   pages[j], 0,
+						   PAGE_SIZE << order,
+						   DMA_BIDIRECTIONAL);
+			if (dma_mapping_error(gpusvm->drm->dev, dma_addr[j])) {
+				err = -EFAULT;
+				npages = i;
+				goto err_unmap;
+			}
+		}
+
+		/* Huge pages, reduce memory footprint */
+		if (order) {
+			dma_addr = kmalloc_array(j, sizeof(*dma_addr),
+						 GFP_KERNEL);
+			if (dma_addr) {
+				for (i = 0; i < j; ++i)
+					dma_addr[i] = (dma_addr_t)pfns[i];
+				kvfree(pfns);
+				kfree_mapping = true;
+			} else {
+				dma_addr = (dma_addr_t *)pfns;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->order = order;
+		range->flags.kfree_mapping = kfree_mapping;
+		range->flags.has_dma_mapping = true;
+		range->dma_addr = dma_addr;
+		range->vram_allocation = NULL;
+		if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm, range);
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	}
+
+	if (err == -EAGAIN)
+		goto retry;
+set_seqno:
+	range->notifier_seq = hmm_range.notifier_seq;
+
+	return 0;
+
+err_unmap:
+	for_each_dma_page(i, j, npages, order)
+		dma_unmap_page(gpusvm->drm->dev,
+			       (dma_addr_t)pfns[j],
+			       PAGE_SIZE << order, DMA_BIDIRECTIONAL);
+err_free:
+	if (alloc_pfns)
+		kvfree(pfns);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_range_unmap_pages - Unmap pages associated with a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed that gpusvm->notifier_lock is held in write mode; if it
+ * is clear, it acquires gpusvm->notifier_lock in read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops->invalidate for IOMMU
+ * security model.
+ */
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx)
+{
+	if (ctx->in_notifier)
+		lockdep_assert_held_write(&gpusvm->notifier_lock);
+	else
+		drm_gpusvm_notifier_lock(gpusvm);
+
+	__drm_gpusvm_range_unmap_pages(gpusvm, range);
+
+	if (!ctx->in_notifier)
+		drm_gpusvm_notifier_unlock(gpusvm);
+}
+
+/**
+ * drm_gpusvm_migration_put_page - Put a migration page
+ * @page: Pointer to the page to put
+ *
+ * This function unlocks and puts a page.
+ */
+static void drm_gpusvm_migration_put_page(struct page *page)
+{
+	unlock_page(page);
+	put_page(page);
+}
+
+/**
+ * drm_gpusvm_migration_put_pages - Put migration pages
+ * @npages: Number of pages
+ * @migrate_pfn: Array of migrate page frame numbers
+ *
+ * This function puts an array of pages.
+ */
+static void drm_gpusvm_migration_put_pages(unsigned long npages,
+					   unsigned long *migrate_pfn)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!migrate_pfn[i])
+			continue;
+
+		drm_gpusvm_migration_put_page(migrate_pfn_to_page(migrate_pfn[i]));
+		migrate_pfn[i] = 0;
+	}
+}
+
+/**
+ * drm_gpusvm_get_vram_page - Get a reference to a VRAM page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU SVM zone
+ * device data and initializes it for zone device usage.
+ */
+static void drm_gpusvm_get_vram_page(struct page *page,
+				     struct drm_gpusvm_zdd *zdd)
+{
+	page->zone_device_data = drm_gpusvm_zdd_get(zdd);
+	zone_device_page_init(page);
+}
+
+/**
+ * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Return: 0 on success, -EFAULT if an error occurs during mapping.
+ */
+static int drm_gpusvm_migrate_map_pages(struct device *dev,
+					dma_addr_t *dma_addr,
+					long unsigned int *migrate_pfn,
+					unsigned long npages,
+					enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = migrate_pfn_to_page(migrate_pfn[i]);
+
+		if (!page)
+			continue;
+
+		if (WARN_ON_ONCE(is_zone_device_page(page)))
+			return -EFAULT;
+
+		dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE, dir);
+		if (dma_mapping_error(dev, dma_addr[i]))
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */
+static void drm_gpusvm_migrate_unmap_pages(struct device *dev,
+					   dma_addr_t *dma_addr,
+					   unsigned long npages,
+					   enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!dma_addr[i] || dma_mapping_error(dev, dma_addr[i]))
+			continue;
+
+		dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir);
+	}
+}
+
+/**
+ * drm_gpusvm_migrate_to_vram - Migrate GPU SVM range to VRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *                   failure of this function.
+ * @vram_allocation: Driver-private pointer to the VRAM allocation. The caller
+ *                   should hold a reference to the VRAM allocation, which
+ *                   should be dropped via ops->vram_allocation or upon the
+ *                   failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to VRAM. It performs the
+ * necessary setup and invokes the driver-specific operations for migration to
+ * VRAM. Upon successful return, @vram_allocation can safely reference @range
+ * until ops->vram_release is called which only upon successful return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct migrate_vma migrate = {
+		.start		= start,
+		.end		= end,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_SYSTEM,
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	unsigned long i, npages = npages_in_range(start, end);
+	struct vm_area_struct *vas;
+	struct drm_gpusvm_zdd *zdd = NULL;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int err;
+
+	if (!range->flags.migrate_vram)
+		return -EINVAL;
+
+	if (!gpusvm->ops->populate_vram_pfn || !gpusvm->ops->copy_to_vram ||
+	    !gpusvm->ops->copy_to_sram)
+		return -EOPNOTSUPP;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		mmap_write_lock(mm);
+	}
+
+	mmap_assert_locked(mm);
+
+	vas = vma_lookup(mm, start);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end > vas->vm_end || start < vas->vm_start) {
+		err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	if (!vma_is_anonymous(vas)) {
+		err = -EBUSY;
+		goto err_mmunlock;
+	}
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_mmunlock;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+	zdd = drm_gpusvm_zdd_alloc(range);
+	if (!zdd) {
+		err = -ENOMEM;
+		goto err_free;
+	}
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/*
+	 * FIXME: Below cases, !migrate.cpages and migrate.cpages != npages, not
+	 * always an error. Need to revisit possible cases and how to handle. We
+	 * could prefault on migrate.cpages != npages via hmm_range_fault.
+	 */
+
+	if (!migrate.cpages) {
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	if (migrate.cpages != npages) {
+		err = -EBUSY;
+		goto err_finalize;
+	}
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm, vram_allocation, npages,
+					     migrate.dst);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   migrate.src, npages, DMA_TO_DEVICE);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = pfn_to_page(migrate.dst[i]);
+
+		pages[i] = page;
+		migrate.dst[i] = migrate_pfn(migrate.dst[i]);
+		drm_gpusvm_get_vram_page(page, zdd);
+	}
+
+	err = gpusvm->ops->copy_to_vram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+	/* Upon success bind vram allocation to range and zdd */
+	range->vram_allocation = vram_allocation;
+	WRITE_ONCE(zdd->vram_allocation, vram_allocation);	/* Owns ref */
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_TO_DEVICE);
+err_free:
+	if (zdd)
+		drm_gpusvm_zdd_put(zdd);
+	kvfree(buf);
+err_mmunlock:
+	if (!ctx->mmap_locked) {
+		mmap_write_unlock(mm);
+		mmput(mm);
+	}
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_populate_sram_pfn - Populate SRAM PFNs for a VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @npages: Number of pages to populate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the SRAM migrate page frame numbers (PFNs) for the
+ * specified VM area structure. It allocates and locks pages in the VM area for
+ * SRAM usage. If vas is non-NULL use alloc_page_vma for allocation, if NULL use
+ * alloc_page for allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_sram_pfn(struct vm_area_struct *vas,
+						unsigned long npages,
+						unsigned long *src_mpfn,
+						unsigned long *mpfn, u64 addr)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i, addr += PAGE_SIZE) {
+		struct page *page;
+
+		if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE))
+			continue;
+
+		if (vas)
+			page = alloc_page_vma(GFP_HIGHUSER, vas, addr);
+		else
+			page = alloc_page(GFP_HIGHUSER);
+
+		if (!page)
+			return -ENOMEM;
+
+		lock_page(page);
+		mpfn[i] = migrate_pfn(page_to_pfn(page));
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_evict_to_sram - Evict GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * Similar to __drm_gpusvm_migrate_to_sram but does not require mmap lock and
+ * migration done via migrate_device_* functions. Fallback path as it is
+ * preferred to issue migrations with mmap lock.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_evict_to_sram(struct drm_gpusvm *gpusvm,
+				    struct drm_gpusvm_range *range)
+{
+	unsigned long npages;
+	struct page **pages;
+	unsigned long *src, *dst;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	npages = npages_in_range(range->va.start, range->va.end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	src = buf;
+	dst = buf + (sizeof(*src) * npages);
+	dma_addr = buf + (2 * sizeof(*src) * npages);
+	pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) * npages;
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm, range->vram_allocation,
+					     npages, src);
+	if (err)
+		goto err_free;
+
+	err = migrate_device_vma_range(gpusvm->mm,
+				       gpusvm->device_private_page_owner, src,
+				       npages, range->va.start);
+	if (err)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(NULL, npages, src, dst, 0);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   dst, npages, DMA_BIDIRECTIONAL);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, dst);
+	migrate_device_pages(src, dst, npages);
+	migrate_device_finalize(src, dst, npages);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+
+	return err;
+}
+
+/**
+ * __drm_gpusvm_migrate_to_sram - Migrate GPU SVM range to SRAM (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @vas: Pointer to the VM area structure
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @start: Start address of the migration range
+ * @end: End address of the migration range
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to SRAM. It sets up the migration, populates + dma maps SRAM PFNs, and
+ * invokes the driver-specific operations for migration to SRAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int __drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+					struct vm_area_struct *vas,
+					struct page *page,
+					u64 start, u64 end)
+{
+	struct migrate_vma migrate = {
+		.vma		= vas,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_DEVICE_PRIVATE,
+		.fault_page	= page,
+	};
+	unsigned long npages;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	mmap_assert_locked(gpusvm->mm);
+
+	/* Corner where VMA area struct has been partially unmapped */
+	if (start < vas->vm_start)
+		start = vas->vm_start;
+	if (end > vas->vm_end)
+		end = vas->vm_end;
+
+	migrate.start = start;
+	migrate.end = end;
+	npages = npages_in_range(start, end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/* Raced with another CPU fault, nothing to do */
+	if (!migrate.cpages)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(vas, npages,
+						   migrate.src, migrate.dst,
+						   start);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   migrate.dst, npages,
+					   DMA_BIDIRECTIONAL);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(migrate.src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+	mmap_assert_locked(gpusvm->mm);
+
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_to_sram - Migrate (evict) GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function initiates the migration of the specified GPU SVM range to
+ * SRAM. It performs necessary checks and invokes the internal migration
+ * function for actual migration.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	int err;
+	bool retry = false;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		if (ctx->trylock_mmap) {
+			if (!mmap_read_trylock(mm))  {
+				err = drm_gpusvm_evict_to_sram(gpusvm, range);
+				goto err_mmput;
+			}
+		} else {
+			mmap_read_lock(mm);
+		}
+	}
+
+	mmap_assert_locked(mm);
+
+	/*
+	 * Loop required to find all VMA area structs for the corner case when
+	 * VRAM backing has been partially unmapped from MM's address space.
+	 */
+again:
+	vas = find_vma(mm, start);
+	if (!vas) {
+		if (!retry)
+			err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end <= vas->vm_start || start >= vas->vm_end) {
+		if (!retry)
+			err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	err = __drm_gpusvm_migrate_to_sram(gpusvm, vas, NULL, start, end);
+	if (err)
+		goto err_mmunlock;
+
+	if (vas->vm_end < end) {
+		retry = true;
+		start = vas->vm_end;
+		goto again;
+	}
+
+	if (!ctx->mmap_locked) {
+		mmap_read_unlock(mm);
+		/*
+		 * Using mmput_async as this function can be called while
+		 * holding a dma-resv lock, and a final put can grab the mmap
+		 * lock, causing a lock inversion.
+		 */
+		mmput_async(mm);
+	}
+
+	return 0;
+
+err_mmunlock:
+	if (!ctx->mmap_locked)
+		mmap_read_unlock(mm);
+err_mmput:
+	if (!ctx->mmap_locked)
+		mmput_async(mm);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_page_free - Put GPU SVM zone device data associated with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device data
+ * associated with a page when it is being released.
+ */
+static void drm_gpusvm_page_free(struct page *page)
+{
+	drm_gpusvm_zdd_put(page->zone_device_data);
+}
+
+/**
+ * drm_gpusvm_migrate_to_ram - Migrate GPU SVM range to RAM (page fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ * VM_FAULT_SIGBUS on failure, 0 on success.
+ */
+static vm_fault_t drm_gpusvm_migrate_to_ram(struct vm_fault *vmf)
+{
+	struct drm_gpusvm_zdd *zdd = vmf->page->zone_device_data;
+	int err;
+
+	err = __drm_gpusvm_migrate_to_sram(zdd->range->gpusvm,
+					   vmf->vma, vmf->page,
+					   zdd->range->va.start,
+					   zdd->range->va.end);
+
+	return err ? VM_FAULT_SIGBUS : 0;
+}
+
+/**
+ * drm_gpusvm_pagemap_ops - Device page map operations for GPU SVM
+ */
+static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = {
+	.page_free = drm_gpusvm_page_free,
+	.migrate_to_ram = drm_gpusvm_migrate_to_ram,
+};
+
+/**
+ * drm_gpusvm_pagemap_ops_get - Retrieve GPU SVM device page map operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void)
+{
+	return &drm_gpusvm_pagemap_ops;
+}
+
+/**
+ * drm_gpusvm_has_mapping - Check if GPU SVM has mapping for the given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end)
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	drm_gpusvm_for_each_notifier(notifier, gpusvm, start, end) {
+		struct drm_gpusvm_range *range = NULL;
+
+		drm_gpusvm_for_each_range(range, notifier, start, end)
+			return true;
+	}
+
+	return false;
+}
drivers/gpu/drm/xe/drm_gpusvm.h new file +415 -0
new file mode 100644
index 000000000000..0ea70f8534a8
@@ -0,0 +1,415 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef __DRM_GPUSVM_H__
+#define __DRM_GPUSVM_H__
+
+#include <linux/kref.h>
+#include <linux/mmu_notifier.h>
+#include <linux/workqueue.h>
+
+struct dev_pagemap_ops;
+struct drm_device;
+struct drm_gpusvm;
+struct drm_gpusvm_notifier;
+struct drm_gpusvm_ops;
+struct drm_gpusvm_range;
+
+/**
+ * struct drm_gpusvm_ops - Operations structure for GPU SVM
+ *
+ * This structure defines the operations for GPU Shared Virtual Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * perform operations such as migration between VRAM and system RAM.
+ */
+struct drm_gpusvm_ops {
+	/**
+	 * @notifier_alloc: Allocate a GPU SVM notifier (optional)
+	 *
+	 * This function shall allocate a GPU SVM notifier.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM notifier on success, NULL on failure.
+	 */
+	struct drm_gpusvm_notifier *(*notifier_alloc)(void);
+
+	/**
+	 * @notifier_free: Free a GPU SVM notifier (optional)
+	 * @notifier: Pointer to the GPU SVM notifier to be freed
+	 *
+	 * This function shall free a GPU SVM notifier.
+	 */
+	void (*notifier_free)(struct drm_gpusvm_notifier *notifier);
+
+	/**
+	 * @range_alloc: Allocate a GPU SVM range (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 *
+	 * This function shall allocate a GPU SVM range.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM range on success, NULL on failure.
+	 */
+	struct drm_gpusvm_range *(*range_alloc)(struct drm_gpusvm *gpusvm);
+
+	/**
+	 * @range_free: Free a GPU SVM range (optional)
+	 * @range: Pointer to the GPU SVM range to be freed
+	 *
+	 * This function shall free a GPU SVM range.
+	 */
+	void (*range_free)(struct drm_gpusvm_range *range);
+
+	/**
+	 * @vram_release: Release VRAM allocation (optional)
+	 * @vram_allocation: Driver-private pointer to the VRAM allocation
+	 *
+	 * This function shall release VRAM allocation and expects to drop a
+	 * reference to VRAM allocation.
+	 */
+	void (*vram_release)(void *vram_allocation);
+
+	/**
+	 * @populate_vram_pfn: Populate VRAM PFN (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @vram_allocation: Driver-private pointer to the VRAM allocation
+	 * @npages: Number of pages to populate
+	 * @pfn: Array of page frame numbers to populate
+	 *
+	 * This function shall populate VRAM page frame numbers (PFN).
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*populate_vram_pfn)(struct drm_gpusvm *gpusvm,
+				 void *vram_allocation,
+				 unsigned long npages,
+				 unsigned long *pfn);
+
+	/**
+	 * @copy_to_vram: Copy to VRAM (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (destination)
+	 * @dma_addr: Pointer to array of DMA addresses (source)
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to VRAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_vram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @copy_to_sram: Copy to system RAM (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (source)
+	 * @dma_addr: Pointer to array of DMA addresses (destination)
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to system RAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_sram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @invalidate: Invalidate GPU SVM notifier (required)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @notifier: Pointer to the GPU SVM notifier
+	 * @mmu_range: Pointer to the mmu_notifier_range structure
+	 *
+	 * This function shall invalidate the GPU page tables. It can safely
+	 * walk the notifier range RB tree/list in this function. Called while
+	 * holding the notifier lock.
+	 */
+	void (*invalidate)(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_notifier *notifier,
+			   const struct mmu_notifier_range *mmu_range);
+};
+
+/**
+ * struct drm_gpusvm_notifier - Structure representing a GPU SVM notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @interval: Interval for the notifier
+ * @rb: Red-black tree node for the parent GPU SVM structure notifier tree
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ *              interval tree. This is useful to keep iterating ranges while
+ *              doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier has been
+ *                 removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */
+struct drm_gpusvm_notifier {
+	struct drm_gpusvm *gpusvm;
+	struct mmu_interval_notifier notifier;
+	struct {
+		u64 start;
+		u64 end;
+	} interval;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct rb_root_cached root;
+	struct list_head range_list;
+	struct {
+		u32 removed : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm_range - Structure representing a GPU SVM range
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier
+ * @refcount: Reference count for the range
+ * @rb: Red-black tree node for the parent GPU SVM notifier structure range tree
+ * @va: Virtual address range
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @pages: Pointer to the array of pages (if backing store is in VRAM)
+ * @dma_addr: DMA address array (if backing store is SRAM and DMA mapped)
+ * @vram_allocation: Driver-private pointer to the VRAM allocation
+ * @order: Order of dma mapping. i.e. PAGE_SIZE << order is mapping size
+ * @flags.migrate_vram: Flag indicating whether the range can be migrated to VRAM
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_vram_pages: Flag indicating if the range has vram pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA mapping
+ * @flags.kfree_mapping: Flag indicating @dma_addr is a compact allocation based
+ *                       on @order which releases via kfree
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */
+struct drm_gpusvm_range {
+	struct drm_gpusvm *gpusvm;
+	struct drm_gpusvm_notifier *notifier;
+	struct kref refcount;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct {
+		u64 start;
+		u64 end;
+	} va;
+	unsigned long notifier_seq;
+	union {
+		struct page **pages;
+		dma_addr_t *dma_addr;
+	};
+	void *vram_allocation;
+	u16 order;
+	struct {
+		/* All flags below must be set upon creation */
+		u16 migrate_vram : 1;
+		/* All flags below must be set / cleared under notifier lock */
+		u16 unmapped : 1;
+		u16 partial_unmap : 1;
+		u16 has_vram_pages : 1;
+		u16 has_dma_mapping : 1;
+		u16 kfree_mapping : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm - GPU SVM structure
+ *
+ * @name: Name of the GPU SVM
+ * @drm: Pointer to the DRM device structure
+ * @mm: Pointer to the mm_struct for the address space
+ * @device_private_page_owner: Device private pages owner
+ * @mm_start: Start address of GPU SVM
+ * @mm_range: Range of the GPU SVM
+ * @notifier_size: Size of individual notifiers
+ * @ops: Pointer to the operations structure for GPU SVM
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range allocation.
+ *               Entries should be powers of 2 in descending order.
+ * @num_chunks: Number of chunks
+ * @notifier_lock: Read-write semaphore for protecting notifier operations
+ * @zdd_wq: Workqueue for deferred work on zdd destruction
+ * @root: Cached root node of the Red-Black tree containing GPU SVM notifiers
+ * @notifier_list: list head containing of notifiers in the same order they
+ *                 appear in interval tree. This is useful to keep iterating
+ *                 notifiers while doing modifications to RB tree.
+ *
+ * This structure represents a GPU SVM (Shared Virtual Memory) used for tracking
+ * memory ranges mapped in a DRM (Direct Rendering Manager) device.
+ *
+ * No reference counting is provided, as this is expected to be embedded in the
+ * driver VM structure along with the struct drm_gpuvm, which handles reference
+ * counting.
+ */
+struct drm_gpusvm {
+	const char *name;
+	struct drm_device *drm;
+	struct mm_struct *mm;
+	void *device_private_page_owner;
+	u64 mm_start;
+	u64 mm_range;
+	u64 notifier_size;
+	const struct drm_gpusvm_ops *ops;
+	const u64 *chunk_sizes;
+	int num_chunks;
+	struct rw_semaphore notifier_lock;
+	struct workqueue_struct *zdd_wq;
+	struct rb_root_cached root;
+	struct list_head notifier_list;
+};
+
+/**
+ * struct drm_gpusvm_ctx - DRM GPU SVM context
+ *
+ * @mmap_locked: mmap lock is locked
+ * @trylock_mmap: trylock mmap lock, used to avoid locking inversions
+ *                (e.g.dma-revs -> mmap lock)
+ * @in_notifier: entering from a MMU notifier
+ * @read_only: operating on read-only memory
+ * @vram_possible: possible to use VRAM
+ * @prefault: prefault pages
+ *
+ * Context that is DRM GPUSVM is operating in (i.e. user arguments).
+ */
+struct drm_gpusvm_ctx {
+	u32 mmap_locked :1;
+	u32 trylock_mmap :1;
+	u32 in_notifier :1;
+	u32 read_only :1;
+	u32 vram_possible :1;
+	u32 prefault :1;
+};
+
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void *device_private_page_owner,
+		    u64 mm_start, u64 mm_range, u64 notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const u64 *chunk_sizes, int num_chunks);
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm);
+void drm_gpusvm_free(struct drm_gpusvm *gpusvm);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, u64 fault_addr,
+				u64 gpuva_start, u64 gpuva_end,
+				const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range);
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range);
+
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range);
+
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx);
+
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx);
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void);
+
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, u64 start, u64 end);
+
+/**
+ * drm_gpusvm_notifier_lock - Lock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, take lock
+ */
+#define drm_gpusvm_notifier_lock(gpusvm__)	\
+	down_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * drm_gpusvm_notifier_unlock - Unlock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, drop lock
+ */
+#define drm_gpusvm_notifier_unlock(gpusvm__)	\
+	up_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * __drm_gpusvm_range_next - Get the next GPU SVM range in the list
+ * @range: a pointer to the current GPU SVM range
+ *
+ * Return: A pointer to the next drm_gpusvm_range if available, or NULL if the
+ *         current range is the last one or if the input range is NULL.
+ */
+static inline struct drm_gpusvm_range *
+__drm_gpusvm_range_next(struct drm_gpusvm_range *range)
+{
+	if (range && !list_is_last(&range->rb.entry,
+				   &range->notifier->range_list))
+		return list_next_entry(range, rb.entry);
+
+	return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_range - Iterate over GPU SVM ranges in a notifier
+ * @range__: Iterator variable for the ranges. If set, it indicates the start of
+ *	     the iterator. If NULL, call drm_gpusvm_range_find() to get the range.
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier. It is safe
+ * to use while holding the driver SVM lock or the notifier lock.
+ */
+#define drm_gpusvm_for_each_range(range__, notifier__, start__, end__)	\
+	for ((range__) = (range__) ?:					\
+	     drm_gpusvm_range_find((notifier__), (start__), (end__));	\
+	     (range__) && (range__->va.start < (end__));		\
+	     (range__) = __drm_gpusvm_range_next(range__))
+
+/**
+ * drm_gpusvm_range_set_unmapped - Mark a GPU SVM range as unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets the partial_unmap flag
+ * if the range partially falls within the provided MMU notifier range.
+ */
+static inline void
+drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,
+			      const struct mmu_notifier_range *mmu_range)
+{
+	lockdep_assert_held_write(&range->gpusvm->notifier_lock);
+
+	range->flags.unmapped = true;
+	if (range->va.start < mmu_range->start ||
+	    range->va.end > mmu_range->end)
+		range->flags.partial_unmap = true;
+}
+
+#endif /* __DRM_GPUSVM_H__ */
Signature · 1 line
2.34.1
Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
maintainer
#9 lore raw

Re: [RFC PATCH 05/28] drm/gpusvm: Add support for GPU Shared Virtual Memory

Am 28.08.24 um 16:31 schrieb Daniel Vetter:

Quoted 17 lines · Matthew Brost

drivers/gpu/drm/xe/drm_gpusvm.c line 1353

Quoted 11 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+		if (!ctx->mmap_locked) {
+			/*
+			 * XXX: HMM locking document indicates only a read-lock
+			 * is required but there apears to be a window between
+			 * the MMU_NOTIFY_MIGRATE event triggered in a CPU fault
+			 * via migrate_vma_setup and the pages actually moving
+			 * in migrate_vma_finalize in which this code can grab
+			 * garbage pages. Grabbing the write-lock if the range
+			 * is attached to vram appears to protect against this
+			 * race.
+			 */

This one is really scary, since it means the entire migrate pte trickery is essentially completely busted. Grabbing the mmap write lock just means you block out pretty much everything interesting from concurrently happening.

Quoted 2 lines

My gut feeling says we need to figure out what's happening here, because this looks a bit too fundamental to me.

I think I have at least a high level understanding what's going on here, Felix and especially Philip should know more of the details.

In general grabbing the mm_lock to protect PTEs from changing is completely nonsense. The mm_lock is to protect the VMAs and *not* the PTEs!

Even with the write side of the mm_lock taken it is perfectly possible that PTE change. It's just less likely.

We run into multiple issues before we figured out this important distinction as well.

Christian.

Quoted 1,248 lines

-Sima

Quoted 1,245 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +821 -0
+			if (vram_pages)
+				mmap_write_lock(mm);
+			else
+				mmap_read_lock(mm);
+		}
+		err = hmm_range_fault(&hmm_range);
+		if (!ctx->mmap_locked) {
+			if (vram_pages)
+				mmap_write_unlock(mm);
+			else
+				mmap_read_unlock(mm);
+		}
+
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
+			continue;
+		}
+		break;
+	}
+	if (!ctx->mmap_locked)
+		mmput(mm);
+	if (err)
+		goto err_free;
+
+	pages = (struct page **)pfns;
+
+	if (ctx->prefault) {
+		range->pages = pages;
+		goto set_seqno;
+	}
+
+map_pages:
+	if (is_device_private_page(hmm_pfn_to_page(pfns[0]))) {
+		WARN_ON_ONCE(!range->vram_allocation);
+
+		for (i = 0; i < npages; ++i) {
+			pages[i] = hmm_pfn_to_page(pfns[i]);
+
+			if (WARN_ON_ONCE(!is_device_private_page(pages[i]))) {
+				err = -EOPNOTSUPP;
+				goto err_free;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->flags.has_vram_pages = true;
+		range->pages = pages;
+		if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm, range);
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	} else {
+		dma_addr_t *dma_addr = (dma_addr_t *)pfns;
+
+		for_each_dma_page(i, j, npages, order) {
+			if (WARN_ON_ONCE(i && order !=
+					 hmm_pfn_to_map_order(pfns[i]))) {
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+			order = hmm_pfn_to_map_order(pfns[i]);
+
+			pages[j] = hmm_pfn_to_page(pfns[i]);
+			if (WARN_ON_ONCE(is_zone_device_page(pages[j]))) {
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+
+			set_page_dirty_lock(pages[j]);
+			mark_page_accessed(pages[j]);
+
+			dma_addr[j] = dma_map_page(gpusvm->drm->dev,
+						   pages[j], 0,
+						   PAGE_SIZE << order,
+						   DMA_BIDIRECTIONAL);
+			if (dma_mapping_error(gpusvm->drm->dev, dma_addr[j])) {
+				err = -EFAULT;
+				npages = i;
+				goto err_unmap;
+			}
+		}
+
+		/* Huge pages, reduce memory footprint */
+		if (order) {
+			dma_addr = kmalloc_array(j, sizeof(*dma_addr),
+						 GFP_KERNEL);
+			if (dma_addr) {
+				for (i = 0; i < j; ++i)
+					dma_addr[i] = (dma_addr_t)pfns[i];
+				kvfree(pfns);
+				kfree_mapping = true;
+			} else {
+				dma_addr = (dma_addr_t *)pfns;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->order = order;
+		range->flags.kfree_mapping = kfree_mapping;
+		range->flags.has_dma_mapping = true;
+		range->dma_addr = dma_addr;
+		range->vram_allocation = NULL;
+		if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm, range);
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	}
+
+	if (err == -EAGAIN)
+		goto retry;
+set_seqno:
+	range->notifier_seq = hmm_range.notifier_seq;
+
+	return 0;
+
+err_unmap:
+	for_each_dma_page(i, j, npages, order)
+		dma_unmap_page(gpusvm->drm->dev,
+			       (dma_addr_t)pfns[j],
+			       PAGE_SIZE << order, DMA_BIDIRECTIONAL);
+err_free:
+	if (alloc_pfns)
+		kvfree(pfns);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_range_unmap_pages - Unmap pages associated with a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed that gpusvm->notifier_lock is held in write mode; if it
+ * is clear, it acquires gpusvm->notifier_lock in read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops->invalidate for IOMMU
+ * security model.
+ */
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx)
+{
+	if (ctx->in_notifier)
+		lockdep_assert_held_write(&gpusvm->notifier_lock);
+	else
+		drm_gpusvm_notifier_lock(gpusvm);
+
+	__drm_gpusvm_range_unmap_pages(gpusvm, range);
+
+	if (!ctx->in_notifier)
+		drm_gpusvm_notifier_unlock(gpusvm);
+}
+
+/**
+ * drm_gpusvm_migration_put_page - Put a migration page
+ * @page: Pointer to the page to put
+ *
+ * This function unlocks and puts a page.
+ */
+static void drm_gpusvm_migration_put_page(struct page *page)
+{
+	unlock_page(page);
+	put_page(page);
+}
+
+/**
+ * drm_gpusvm_migration_put_pages - Put migration pages
+ * @npages: Number of pages
+ * @migrate_pfn: Array of migrate page frame numbers
+ *
+ * This function puts an array of pages.
+ */
+static void drm_gpusvm_migration_put_pages(unsigned long npages,
+					   unsigned long *migrate_pfn)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!migrate_pfn[i])
+			continue;
+
+		drm_gpusvm_migration_put_page(migrate_pfn_to_page(migrate_pfn[i]));
+		migrate_pfn[i] = 0;
+	}
+}
+
+/**
+ * drm_gpusvm_get_vram_page - Get a reference to a VRAM page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU SVM zone
+ * device data and initializes it for zone device usage.
+ */
+static void drm_gpusvm_get_vram_page(struct page *page,
+				     struct drm_gpusvm_zdd *zdd)
+{
+	page->zone_device_data = drm_gpusvm_zdd_get(zdd);
+	zone_device_page_init(page);
+}
+
+/**
+ * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Return: 0 on success, -EFAULT if an error occurs during mapping.
+ */
+static int drm_gpusvm_migrate_map_pages(struct device *dev,
+					dma_addr_t *dma_addr,
+					long unsigned int *migrate_pfn,
+					unsigned long npages,
+					enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = migrate_pfn_to_page(migrate_pfn[i]);
+
+		if (!page)
+			continue;
+
+		if (WARN_ON_ONCE(is_zone_device_page(page)))
+			return -EFAULT;
+
+		dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE, dir);
+		if (dma_mapping_error(dev, dma_addr[i]))
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */
+static void drm_gpusvm_migrate_unmap_pages(struct device *dev,
+					   dma_addr_t *dma_addr,
+					   unsigned long npages,
+					   enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!dma_addr[i] || dma_mapping_error(dev, dma_addr[i]))
+			continue;
+
+		dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir);
+	}
+}
+
+/**
+ * drm_gpusvm_migrate_to_vram - Migrate GPU SVM range to VRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *                   failure of this function.
+ * @vram_allocation: Driver-private pointer to the VRAM allocation. The caller
+ *                   should hold a reference to the VRAM allocation, which
+ *                   should be dropped via ops->vram_allocation or upon the
+ *                   failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to VRAM. It performs the
+ * necessary setup and invokes the driver-specific operations for migration to
+ * VRAM. Upon successful return, @vram_allocation can safely reference @range
+ * until ops->vram_release is called which only upon successful return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct migrate_vma migrate = {
+		.start		= start,
+		.end		= end,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_SYSTEM,
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	unsigned long i, npages = npages_in_range(start, end);
+	struct vm_area_struct *vas;
+	struct drm_gpusvm_zdd *zdd = NULL;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int err;
+
+	if (!range->flags.migrate_vram)
+		return -EINVAL;
+
+	if (!gpusvm->ops->populate_vram_pfn || !gpusvm->ops->copy_to_vram ||
+	    !gpusvm->ops->copy_to_sram)
+		return -EOPNOTSUPP;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		mmap_write_lock(mm);
+	}
+
+	mmap_assert_locked(mm);
+
+	vas = vma_lookup(mm, start);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end > vas->vm_end || start < vas->vm_start) {
+		err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	if (!vma_is_anonymous(vas)) {
+		err = -EBUSY;
+		goto err_mmunlock;
+	}
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_mmunlock;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+	zdd = drm_gpusvm_zdd_alloc(range);
+	if (!zdd) {
+		err = -ENOMEM;
+		goto err_free;
+	}
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/*
+	 * FIXME: Below cases, !migrate.cpages and migrate.cpages != npages, not
+	 * always an error. Need to revisit possible cases and how to handle. We
+	 * could prefault on migrate.cpages != npages via hmm_range_fault.
+	 */
+
+	if (!migrate.cpages) {
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	if (migrate.cpages != npages) {
+		err = -EBUSY;
+		goto err_finalize;
+	}
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm, vram_allocation, npages,
+					     migrate.dst);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   migrate.src, npages, DMA_TO_DEVICE);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = pfn_to_page(migrate.dst[i]);
+
+		pages[i] = page;
+		migrate.dst[i] = migrate_pfn(migrate.dst[i]);
+		drm_gpusvm_get_vram_page(page, zdd);
+	}
+
+	err = gpusvm->ops->copy_to_vram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+	/* Upon success bind vram allocation to range and zdd */
+	range->vram_allocation = vram_allocation;
+	WRITE_ONCE(zdd->vram_allocation, vram_allocation);	/* Owns ref */
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_TO_DEVICE);
+err_free:
+	if (zdd)
+		drm_gpusvm_zdd_put(zdd);
+	kvfree(buf);
+err_mmunlock:
+	if (!ctx->mmap_locked) {
+		mmap_write_unlock(mm);
+		mmput(mm);
+	}
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_populate_sram_pfn - Populate SRAM PFNs for a VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @npages: Number of pages to populate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the SRAM migrate page frame numbers (PFNs) for the
+ * specified VM area structure. It allocates and locks pages in the VM area for
+ * SRAM usage. If vas is non-NULL use alloc_page_vma for allocation, if NULL use
+ * alloc_page for allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_sram_pfn(struct vm_area_struct *vas,
+						unsigned long npages,
+						unsigned long *src_mpfn,
+						unsigned long *mpfn, u64 addr)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i, addr += PAGE_SIZE) {
+		struct page *page;
+
+		if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE))
+			continue;
+
+		if (vas)
+			page = alloc_page_vma(GFP_HIGHUSER, vas, addr);
+		else
+			page = alloc_page(GFP_HIGHUSER);
+
+		if (!page)
+			return -ENOMEM;
+
+		lock_page(page);
+		mpfn[i] = migrate_pfn(page_to_pfn(page));
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_evict_to_sram - Evict GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * Similar to __drm_gpusvm_migrate_to_sram but does not require mmap lock and
+ * migration done via migrate_device_* functions. Fallback path as it is
+ * preferred to issue migrations with mmap lock.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_evict_to_sram(struct drm_gpusvm *gpusvm,
+				    struct drm_gpusvm_range *range)
+{
+	unsigned long npages;
+	struct page **pages;
+	unsigned long *src, *dst;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	npages = npages_in_range(range->va.start, range->va.end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	src = buf;
+	dst = buf + (sizeof(*src) * npages);
+	dma_addr = buf + (2 * sizeof(*src) * npages);
+	pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) * npages;
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm, range->vram_allocation,
+					     npages, src);
+	if (err)
+		goto err_free;
+
+	err = migrate_device_vma_range(gpusvm->mm,
+				       gpusvm->device_private_page_owner, src,
+				       npages, range->va.start);
+	if (err)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(NULL, npages, src, dst, 0);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   dst, npages, DMA_BIDIRECTIONAL);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, dst);
+	migrate_device_pages(src, dst, npages);
+	migrate_device_finalize(src, dst, npages);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+
+	return err;
+}
+
+/**
+ * __drm_gpusvm_migrate_to_sram - Migrate GPU SVM range to SRAM (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @vas: Pointer to the VM area structure
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @start: Start address of the migration range
+ * @end: End address of the migration range
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to SRAM. It sets up the migration, populates + dma maps SRAM PFNs, and
+ * invokes the driver-specific operations for migration to SRAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int __drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+					struct vm_area_struct *vas,
+					struct page *page,
+					u64 start, u64 end)
+{
+	struct migrate_vma migrate = {
+		.vma		= vas,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_DEVICE_PRIVATE,
+		.fault_page	= page,
+	};
+	unsigned long npages;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	mmap_assert_locked(gpusvm->mm);
+
+	/* Corner where VMA area struct has been partially unmapped */
+	if (start < vas->vm_start)
+		start = vas->vm_start;
+	if (end > vas->vm_end)
+		end = vas->vm_end;
+
+	migrate.start = start;
+	migrate.end = end;
+	npages = npages_in_range(start, end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/* Raced with another CPU fault, nothing to do */
+	if (!migrate.cpages)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(vas, npages,
+						   migrate.src, migrate.dst,
+						   start);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   migrate.dst, npages,
+					   DMA_BIDIRECTIONAL);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(migrate.src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+	mmap_assert_locked(gpusvm->mm);
+
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_to_sram - Migrate (evict) GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function initiates the migration of the specified GPU SVM range to
+ * SRAM. It performs necessary checks and invokes the internal migration
+ * function for actual migration.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	int err;
+	bool retry = false;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		if (ctx->trylock_mmap) {
+			if (!mmap_read_trylock(mm))  {
+				err = drm_gpusvm_evict_to_sram(gpusvm, range);
+				goto err_mmput;
+			}
+		} else {
+			mmap_read_lock(mm);
+		}
+	}
+
+	mmap_assert_locked(mm);
+
+	/*
+	 * Loop required to find all VMA area structs for the corner case when
+	 * VRAM backing has been partially unmapped from MM's address space.
+	 */
+again:
+	vas = find_vma(mm, start);
+	if (!vas) {
+		if (!retry)
+			err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end <= vas->vm_start || start >= vas->vm_end) {
+		if (!retry)
+			err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	err = __drm_gpusvm_migrate_to_sram(gpusvm, vas, NULL, start, end);
+	if (err)
+		goto err_mmunlock;
+
+	if (vas->vm_end < end) {
+		retry = true;
+		start = vas->vm_end;
+		goto again;
+	}
+
+	if (!ctx->mmap_locked) {
+		mmap_read_unlock(mm);
+		/*
+		 * Using mmput_async as this function can be called while
+		 * holding a dma-resv lock, and a final put can grab the mmap
+		 * lock, causing a lock inversion.
+		 */
+		mmput_async(mm);
+	}
+
+	return 0;
+
+err_mmunlock:
+	if (!ctx->mmap_locked)
+		mmap_read_unlock(mm);
+err_mmput:
+	if (!ctx->mmap_locked)
+		mmput_async(mm);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_page_free - Put GPU SVM zone device data associated with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device data
+ * associated with a page when it is being released.
+ */
+static void drm_gpusvm_page_free(struct page *page)
+{
+	drm_gpusvm_zdd_put(page->zone_device_data);
+}
+
+/**
+ * drm_gpusvm_migrate_to_ram - Migrate GPU SVM range to RAM (page fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ * VM_FAULT_SIGBUS on failure, 0 on success.
+ */
+static vm_fault_t drm_gpusvm_migrate_to_ram(struct vm_fault *vmf)
+{
+	struct drm_gpusvm_zdd *zdd = vmf->page->zone_device_data;
+	int err;
+
+	err = __drm_gpusvm_migrate_to_sram(zdd->range->gpusvm,
+					   vmf->vma, vmf->page,
+					   zdd->range->va.start,
+					   zdd->range->va.end);
+
+	return err ? VM_FAULT_SIGBUS : 0;
+}
+
+/**
+ * drm_gpusvm_pagemap_ops - Device page map operations for GPU SVM
+ */
+static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = {
+	.page_free = drm_gpusvm_page_free,
+	.migrate_to_ram = drm_gpusvm_migrate_to_ram,
+};
+
+/**
+ * drm_gpusvm_pagemap_ops_get - Retrieve GPU SVM device page map operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void)
+{
+	return &drm_gpusvm_pagemap_ops;
+}
+
+/**
+ * drm_gpusvm_has_mapping - Check if GPU SVM has mapping for the given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end)
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	drm_gpusvm_for_each_notifier(notifier, gpusvm, start, end) {
+		struct drm_gpusvm_range *range = NULL;
+
+		drm_gpusvm_for_each_range(range, notifier, start, end)
+			return true;
+	}
+
+	return false;
+}
drivers/gpu/drm/xe/drm_gpusvm.h new file +415 -0
new file mode 100644
index 000000000000..0ea70f8534a8
@@ -0,0 +1,415 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef __DRM_GPUSVM_H__
+#define __DRM_GPUSVM_H__
+
+#include <linux/kref.h>
+#include <linux/mmu_notifier.h>
+#include <linux/workqueue.h>
+
+struct dev_pagemap_ops;
+struct drm_device;
+struct drm_gpusvm;
+struct drm_gpusvm_notifier;
+struct drm_gpusvm_ops;
+struct drm_gpusvm_range;
+
+/**
+ * struct drm_gpusvm_ops - Operations structure for GPU SVM
+ *
+ * This structure defines the operations for GPU Shared Virtual Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * perform operations such as migration between VRAM and system RAM.
+ */
+struct drm_gpusvm_ops {
+	/**
+	 * @notifier_alloc: Allocate a GPU SVM notifier (optional)
+	 *
+	 * This function shall allocate a GPU SVM notifier.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM notifier on success, NULL on failure.
+	 */
+	struct drm_gpusvm_notifier *(*notifier_alloc)(void);
+
+	/**
+	 * @notifier_free: Free a GPU SVM notifier (optional)
+	 * @notifier: Pointer to the GPU SVM notifier to be freed
+	 *
+	 * This function shall free a GPU SVM notifier.
+	 */
+	void (*notifier_free)(struct drm_gpusvm_notifier *notifier);
+
+	/**
+	 * @range_alloc: Allocate a GPU SVM range (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 *
+	 * This function shall allocate a GPU SVM range.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM range on success, NULL on failure.
+	 */
+	struct drm_gpusvm_range *(*range_alloc)(struct drm_gpusvm *gpusvm);
+
+	/**
+	 * @range_free: Free a GPU SVM range (optional)
+	 * @range: Pointer to the GPU SVM range to be freed
+	 *
+	 * This function shall free a GPU SVM range.
+	 */
+	void (*range_free)(struct drm_gpusvm_range *range);
+
+	/**
+	 * @vram_release: Release VRAM allocation (optional)
+	 * @vram_allocation: Driver-private pointer to the VRAM allocation
+	 *
+	 * This function shall release VRAM allocation and expects to drop a
+	 * reference to VRAM allocation.
+	 */
+	void (*vram_release)(void *vram_allocation);
+
+	/**
+	 * @populate_vram_pfn: Populate VRAM PFN (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @vram_allocation: Driver-private pointer to the VRAM allocation
+	 * @npages: Number of pages to populate
+	 * @pfn: Array of page frame numbers to populate
+	 *
+	 * This function shall populate VRAM page frame numbers (PFN).
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*populate_vram_pfn)(struct drm_gpusvm *gpusvm,
+				 void *vram_allocation,
+				 unsigned long npages,
+				 unsigned long *pfn);
+
+	/**
+	 * @copy_to_vram: Copy to VRAM (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (destination)
+	 * @dma_addr: Pointer to array of DMA addresses (source)
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to VRAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_vram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @copy_to_sram: Copy to system RAM (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (source)
+	 * @dma_addr: Pointer to array of DMA addresses (destination)
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to system RAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_sram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @invalidate: Invalidate GPU SVM notifier (required)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @notifier: Pointer to the GPU SVM notifier
+	 * @mmu_range: Pointer to the mmu_notifier_range structure
+	 *
+	 * This function shall invalidate the GPU page tables. It can safely
+	 * walk the notifier range RB tree/list in this function. Called while
+	 * holding the notifier lock.
+	 */
+	void (*invalidate)(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_notifier *notifier,
+			   const struct mmu_notifier_range *mmu_range);
+};
+
+/**
+ * struct drm_gpusvm_notifier - Structure representing a GPU SVM notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @interval: Interval for the notifier
+ * @rb: Red-black tree node for the parent GPU SVM structure notifier tree
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ *              interval tree. This is useful to keep iterating ranges while
+ *              doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier has been
+ *                 removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */
+struct drm_gpusvm_notifier {
+	struct drm_gpusvm *gpusvm;
+	struct mmu_interval_notifier notifier;
+	struct {
+		u64 start;
+		u64 end;
+	} interval;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct rb_root_cached root;
+	struct list_head range_list;
+	struct {
+		u32 removed : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm_range - Structure representing a GPU SVM range
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier
+ * @refcount: Reference count for the range
+ * @rb: Red-black tree node for the parent GPU SVM notifier structure range tree
+ * @va: Virtual address range
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @pages: Pointer to the array of pages (if backing store is in VRAM)
+ * @dma_addr: DMA address array (if backing store is SRAM and DMA mapped)
+ * @vram_allocation: Driver-private pointer to the VRAM allocation
+ * @order: Order of dma mapping. i.e. PAGE_SIZE << order is mapping size
+ * @flags.migrate_vram: Flag indicating whether the range can be migrated to VRAM
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_vram_pages: Flag indicating if the range has vram pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA mapping
+ * @flags.kfree_mapping: Flag indicating @dma_addr is a compact allocation based
+ *                       on @order which releases via kfree
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */
+struct drm_gpusvm_range {
+	struct drm_gpusvm *gpusvm;
+	struct drm_gpusvm_notifier *notifier;
+	struct kref refcount;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct {
+		u64 start;
+		u64 end;
+	} va;
+	unsigned long notifier_seq;
+	union {
+		struct page **pages;
+		dma_addr_t *dma_addr;
+	};
+	void *vram_allocation;
+	u16 order;
+	struct {
+		/* All flags below must be set upon creation */
+		u16 migrate_vram : 1;
+		/* All flags below must be set / cleared under notifier lock */
+		u16 unmapped : 1;
+		u16 partial_unmap : 1;
+		u16 has_vram_pages : 1;
+		u16 has_dma_mapping : 1;
+		u16 kfree_mapping : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm - GPU SVM structure
+ *
+ * @name: Name of the GPU SVM
+ * @drm: Pointer to the DRM device structure
+ * @mm: Pointer to the mm_struct for the address space
+ * @device_private_page_owner: Device private pages owner
+ * @mm_start: Start address of GPU SVM
+ * @mm_range: Range of the GPU SVM
+ * @notifier_size: Size of individual notifiers
+ * @ops: Pointer to the operations structure for GPU SVM
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range allocation.
+ *               Entries should be powers of 2 in descending order.
+ * @num_chunks: Number of chunks
+ * @notifier_lock: Read-write semaphore for protecting notifier operations
+ * @zdd_wq: Workqueue for deferred work on zdd destruction
+ * @root: Cached root node of the Red-Black tree containing GPU SVM notifiers
+ * @notifier_list: list head containing of notifiers in the same order they
+ *                 appear in interval tree. This is useful to keep iterating
+ *                 notifiers while doing modifications to RB tree.
+ *
+ * This structure represents a GPU SVM (Shared Virtual Memory) used for tracking
+ * memory ranges mapped in a DRM (Direct Rendering Manager) device.
+ *
+ * No reference counting is provided, as this is expected to be embedded in the
+ * driver VM structure along with the struct drm_gpuvm, which handles reference
+ * counting.
+ */
+struct drm_gpusvm {
+	const char *name;
+	struct drm_device *drm;
+	struct mm_struct *mm;
+	void *device_private_page_owner;
+	u64 mm_start;
+	u64 mm_range;
+	u64 notifier_size;
+	const struct drm_gpusvm_ops *ops;
+	const u64 *chunk_sizes;
+	int num_chunks;
+	struct rw_semaphore notifier_lock;
+	struct workqueue_struct *zdd_wq;
+	struct rb_root_cached root;
+	struct list_head notifier_list;
+};
+
+/**
+ * struct drm_gpusvm_ctx - DRM GPU SVM context
+ *
+ * @mmap_locked: mmap lock is locked
+ * @trylock_mmap: trylock mmap lock, used to avoid locking inversions
+ *                (e.g.dma-revs -> mmap lock)
+ * @in_notifier: entering from a MMU notifier
+ * @read_only: operating on read-only memory
+ * @vram_possible: possible to use VRAM
+ * @prefault: prefault pages
+ *
+ * Context that is DRM GPUSVM is operating in (i.e. user arguments).
+ */
+struct drm_gpusvm_ctx {
+	u32 mmap_locked :1;
+	u32 trylock_mmap :1;
+	u32 in_notifier :1;
+	u32 read_only :1;
+	u32 vram_possible :1;
+	u32 prefault :1;
+};
+
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void *device_private_page_owner,
+		    u64 mm_start, u64 mm_range, u64 notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const u64 *chunk_sizes, int num_chunks);
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm);
+void drm_gpusvm_free(struct drm_gpusvm *gpusvm);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, u64 fault_addr,
+				u64 gpuva_start, u64 gpuva_end,
+				const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range);
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range);
+
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range);
+
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx);
+
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx);
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void);
+
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, u64 start, u64 end);
+
+/**
+ * drm_gpusvm_notifier_lock - Lock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, take lock
+ */
+#define drm_gpusvm_notifier_lock(gpusvm__)	\
+	down_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * drm_gpusvm_notifier_unlock - Unlock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, drop lock
+ */
+#define drm_gpusvm_notifier_unlock(gpusvm__)	\
+	up_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * __drm_gpusvm_range_next - Get the next GPU SVM range in the list
+ * @range: a pointer to the current GPU SVM range
+ *
+ * Return: A pointer to the next drm_gpusvm_range if available, or NULL if the
+ *         current range is the last one or if the input range is NULL.
+ */
+static inline struct drm_gpusvm_range *
+__drm_gpusvm_range_next(struct drm_gpusvm_range *range)
+{
+	if (range && !list_is_last(&range->rb.entry,
+				   &range->notifier->range_list))
+		return list_next_entry(range, rb.entry);
+
+	return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_range - Iterate over GPU SVM ranges in a notifier
+ * @range__: Iterator variable for the ranges. If set, it indicates the start of
+ *	     the iterator. If NULL, call drm_gpusvm_range_find() to get the range.
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier. It is safe
+ * to use while holding the driver SVM lock or the notifier lock.
+ */
+#define drm_gpusvm_for_each_range(range__, notifier__, start__, end__)	\
+	for ((range__) = (range__) ?:					\
+	     drm_gpusvm_range_find((notifier__), (start__), (end__));	\
+	     (range__) && (range__->va.start < (end__));		\
+	     (range__) = __drm_gpusvm_range_next(range__))
+
+/**
+ * drm_gpusvm_range_set_unmapped - Mark a GPU SVM range as unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets the partial_unmap flag
+ * if the range partially falls within the provided MMU notifier range.
+ */
+static inline void
+drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,
+			      const struct mmu_notifier_range *mmu_range)
+{
+	lockdep_assert_held_write(&range->gpusvm->notifier_lock);
+
+	range->flags.unmapped = true;
+	if (range->va.start < mmu_range->start ||
+	    range->va.end > mmu_range->end)
+		range->flags.partial_unmap = true;
+}
+
+#endif /* __DRM_GPUSVM_H__ */
Signature · 1 line
2.34.1
author maintainer
#10 lore raw

Re: [RFC PATCH 05/28] drm/gpusvm: Add support for GPU Shared Virtual Memory

Quoted 13 lines · Christian König

Am 28.08.24 um 16:31 schrieb Daniel Vetter:

Quoted 12 lines · Matthew Brost

drivers/gpu/drm/xe/drm_gpusvm.c line 1353

Quoted 11 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+		if (!ctx->mmap_locked) {
+			/*
+			 * XXX: HMM locking document indicates only a read-lock
+			 * is required but there apears to be a window between
+			 * the MMU_NOTIFY_MIGRATE event triggered in a CPU fault
+			 * via migrate_vma_setup and the pages actually moving
+			 * in migrate_vma_finalize in which this code can grab
+			 * garbage pages. Grabbing the write-lock if the range
+			 * is attached to vram appears to protect against this
+			 * race.
+			 */

Thanks the comments, replying to both of you inline.

Quoted 7 lines
Quoted 7 lines

This one is really scary, since it means the entire migrate pte trickery is essentially completely busted. Grabbing the mmap write lock just means you block out pretty much everything interesting from concurrently happening.

My gut feeling says we need to figure out what's happening here, because this looks a bit too fundamental to me.

I agree. I haven’t looked into this issue for a couple of months but really need to understand what is going on.

I should have mentioned this in the cover letter: the goal of this series was to produce something for review that is stable and supports UMDs/user applications. It was not intended to be presented as a final solution. This issue certainly falls into the category of "needs to be understood and requires a proper fix."

One open question I have is whether the test case that triggers this issue is even defined behavior. The test creates concurrent access between the GPU and CPU to the same memory address, resulting in GPU and CPU faults racing against each other. It’s possible that this is undefined behavior, so data corruption might be acceptable—i.e., the kernel can’t crash, but incorrect results might be permissible.

e.g. This is the only defined usage model:

alloc_memory();
start_compute_kernel();
sync_on_compute_kernel_completion();
read_memory();

Hopefully, in the next week or so, I'll be heavily engaging with the UMD teams. Development can then start, and applications will be running soon after. This will allow us to address issues like this, collect data on memory usage, and verify some of the assumptions I've made, such as optimizing for 2M+ allocations.

Quoted 4 lines

I think I have at least a high level understanding what's going on here, Felix and especially Philip should know more of the details.

I meant to reach out to AMD for issues like this. So, Felix ([email protected]) and Philip ([email protected]) would be good contacts?

Quoted 3 lines

In general grabbing the mm_lock to protect PTEs from changing is completely nonsense. The mm_lock is to protect the VMAs and *not* the PTEs!

Thanks for the hint. I believe that in the AMD implementation, I noticed some additional locks for migration, which might be how you mitigated this issue.

I must say it is a bit unfortunate that the HMM locking documentation doesn’t mention this. I believe the documentation needs additional information, which I can add once we finalize the solution.

Matt

Quoted 1,257 lines

Even with the write side of the mm_lock taken it is perfectly possible that PTE change. It's just less likely.

We run into multiple issues before we figured out this important distinction as well.

Christian.

Quoted 1,248 lines

-Sima

Quoted 1,245 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +821 -0
+			if (vram_pages)
+				mmap_write_lock(mm);
+			else
+				mmap_read_lock(mm);
+		}
+		err = hmm_range_fault(&hmm_range);
+		if (!ctx->mmap_locked) {
+			if (vram_pages)
+				mmap_write_unlock(mm);
+			else
+				mmap_read_unlock(mm);
+		}
+
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
+			continue;
+		}
+		break;
+	}
+	if (!ctx->mmap_locked)
+		mmput(mm);
+	if (err)
+		goto err_free;
+
+	pages = (struct page **)pfns;
+
+	if (ctx->prefault) {
+		range->pages = pages;
+		goto set_seqno;
+	}
+
+map_pages:
+	if (is_device_private_page(hmm_pfn_to_page(pfns[0]))) {
+		WARN_ON_ONCE(!range->vram_allocation);
+
+		for (i = 0; i < npages; ++i) {
+			pages[i] = hmm_pfn_to_page(pfns[i]);
+
+			if (WARN_ON_ONCE(!is_device_private_page(pages[i]))) {
+				err = -EOPNOTSUPP;
+				goto err_free;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->flags.has_vram_pages = true;
+		range->pages = pages;
+		if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm, range);
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	} else {
+		dma_addr_t *dma_addr = (dma_addr_t *)pfns;
+
+		for_each_dma_page(i, j, npages, order) {
+			if (WARN_ON_ONCE(i && order !=
+					 hmm_pfn_to_map_order(pfns[i]))) {
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+			order = hmm_pfn_to_map_order(pfns[i]);
+
+			pages[j] = hmm_pfn_to_page(pfns[i]);
+			if (WARN_ON_ONCE(is_zone_device_page(pages[j]))) {
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+
+			set_page_dirty_lock(pages[j]);
+			mark_page_accessed(pages[j]);
+
+			dma_addr[j] = dma_map_page(gpusvm->drm->dev,
+						   pages[j], 0,
+						   PAGE_SIZE << order,
+						   DMA_BIDIRECTIONAL);
+			if (dma_mapping_error(gpusvm->drm->dev, dma_addr[j])) {
+				err = -EFAULT;
+				npages = i;
+				goto err_unmap;
+			}
+		}
+
+		/* Huge pages, reduce memory footprint */
+		if (order) {
+			dma_addr = kmalloc_array(j, sizeof(*dma_addr),
+						 GFP_KERNEL);
+			if (dma_addr) {
+				for (i = 0; i < j; ++i)
+					dma_addr[i] = (dma_addr_t)pfns[i];
+				kvfree(pfns);
+				kfree_mapping = true;
+			} else {
+				dma_addr = (dma_addr_t *)pfns;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->order = order;
+		range->flags.kfree_mapping = kfree_mapping;
+		range->flags.has_dma_mapping = true;
+		range->dma_addr = dma_addr;
+		range->vram_allocation = NULL;
+		if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm, range);
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	}
+
+	if (err == -EAGAIN)
+		goto retry;
+set_seqno:
+	range->notifier_seq = hmm_range.notifier_seq;
+
+	return 0;
+
+err_unmap:
+	for_each_dma_page(i, j, npages, order)
+		dma_unmap_page(gpusvm->drm->dev,
+			       (dma_addr_t)pfns[j],
+			       PAGE_SIZE << order, DMA_BIDIRECTIONAL);
+err_free:
+	if (alloc_pfns)
+		kvfree(pfns);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_range_unmap_pages - Unmap pages associated with a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed that gpusvm->notifier_lock is held in write mode; if it
+ * is clear, it acquires gpusvm->notifier_lock in read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops->invalidate for IOMMU
+ * security model.
+ */
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx)
+{
+	if (ctx->in_notifier)
+		lockdep_assert_held_write(&gpusvm->notifier_lock);
+	else
+		drm_gpusvm_notifier_lock(gpusvm);
+
+	__drm_gpusvm_range_unmap_pages(gpusvm, range);
+
+	if (!ctx->in_notifier)
+		drm_gpusvm_notifier_unlock(gpusvm);
+}
+
+/**
+ * drm_gpusvm_migration_put_page - Put a migration page
+ * @page: Pointer to the page to put
+ *
+ * This function unlocks and puts a page.
+ */
+static void drm_gpusvm_migration_put_page(struct page *page)
+{
+	unlock_page(page);
+	put_page(page);
+}
+
+/**
+ * drm_gpusvm_migration_put_pages - Put migration pages
+ * @npages: Number of pages
+ * @migrate_pfn: Array of migrate page frame numbers
+ *
+ * This function puts an array of pages.
+ */
+static void drm_gpusvm_migration_put_pages(unsigned long npages,
+					   unsigned long *migrate_pfn)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!migrate_pfn[i])
+			continue;
+
+		drm_gpusvm_migration_put_page(migrate_pfn_to_page(migrate_pfn[i]));
+		migrate_pfn[i] = 0;
+	}
+}
+
+/**
+ * drm_gpusvm_get_vram_page - Get a reference to a VRAM page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU SVM zone
+ * device data and initializes it for zone device usage.
+ */
+static void drm_gpusvm_get_vram_page(struct page *page,
+				     struct drm_gpusvm_zdd *zdd)
+{
+	page->zone_device_data = drm_gpusvm_zdd_get(zdd);
+	zone_device_page_init(page);
+}
+
+/**
+ * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Return: 0 on success, -EFAULT if an error occurs during mapping.
+ */
+static int drm_gpusvm_migrate_map_pages(struct device *dev,
+					dma_addr_t *dma_addr,
+					long unsigned int *migrate_pfn,
+					unsigned long npages,
+					enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = migrate_pfn_to_page(migrate_pfn[i]);
+
+		if (!page)
+			continue;
+
+		if (WARN_ON_ONCE(is_zone_device_page(page)))
+			return -EFAULT;
+
+		dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE, dir);
+		if (dma_mapping_error(dev, dma_addr[i]))
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */
+static void drm_gpusvm_migrate_unmap_pages(struct device *dev,
+					   dma_addr_t *dma_addr,
+					   unsigned long npages,
+					   enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!dma_addr[i] || dma_mapping_error(dev, dma_addr[i]))
+			continue;
+
+		dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir);
+	}
+}
+
+/**
+ * drm_gpusvm_migrate_to_vram - Migrate GPU SVM range to VRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *                   failure of this function.
+ * @vram_allocation: Driver-private pointer to the VRAM allocation. The caller
+ *                   should hold a reference to the VRAM allocation, which
+ *                   should be dropped via ops->vram_allocation or upon the
+ *                   failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to VRAM. It performs the
+ * necessary setup and invokes the driver-specific operations for migration to
+ * VRAM. Upon successful return, @vram_allocation can safely reference @range
+ * until ops->vram_release is called which only upon successful return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct migrate_vma migrate = {
+		.start		= start,
+		.end		= end,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_SYSTEM,
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	unsigned long i, npages = npages_in_range(start, end);
+	struct vm_area_struct *vas;
+	struct drm_gpusvm_zdd *zdd = NULL;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int err;
+
+	if (!range->flags.migrate_vram)
+		return -EINVAL;
+
+	if (!gpusvm->ops->populate_vram_pfn || !gpusvm->ops->copy_to_vram ||
+	    !gpusvm->ops->copy_to_sram)
+		return -EOPNOTSUPP;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		mmap_write_lock(mm);
+	}
+
+	mmap_assert_locked(mm);
+
+	vas = vma_lookup(mm, start);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end > vas->vm_end || start < vas->vm_start) {
+		err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	if (!vma_is_anonymous(vas)) {
+		err = -EBUSY;
+		goto err_mmunlock;
+	}
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_mmunlock;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+	zdd = drm_gpusvm_zdd_alloc(range);
+	if (!zdd) {
+		err = -ENOMEM;
+		goto err_free;
+	}
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/*
+	 * FIXME: Below cases, !migrate.cpages and migrate.cpages != npages, not
+	 * always an error. Need to revisit possible cases and how to handle. We
+	 * could prefault on migrate.cpages != npages via hmm_range_fault.
+	 */
+
+	if (!migrate.cpages) {
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	if (migrate.cpages != npages) {
+		err = -EBUSY;
+		goto err_finalize;
+	}
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm, vram_allocation, npages,
+					     migrate.dst);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   migrate.src, npages, DMA_TO_DEVICE);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = pfn_to_page(migrate.dst[i]);
+
+		pages[i] = page;
+		migrate.dst[i] = migrate_pfn(migrate.dst[i]);
+		drm_gpusvm_get_vram_page(page, zdd);
+	}
+
+	err = gpusvm->ops->copy_to_vram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+	/* Upon success bind vram allocation to range and zdd */
+	range->vram_allocation = vram_allocation;
+	WRITE_ONCE(zdd->vram_allocation, vram_allocation);	/* Owns ref */
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_TO_DEVICE);
+err_free:
+	if (zdd)
+		drm_gpusvm_zdd_put(zdd);
+	kvfree(buf);
+err_mmunlock:
+	if (!ctx->mmap_locked) {
+		mmap_write_unlock(mm);
+		mmput(mm);
+	}
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_populate_sram_pfn - Populate SRAM PFNs for a VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @npages: Number of pages to populate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the SRAM migrate page frame numbers (PFNs) for the
+ * specified VM area structure. It allocates and locks pages in the VM area for
+ * SRAM usage. If vas is non-NULL use alloc_page_vma for allocation, if NULL use
+ * alloc_page for allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_sram_pfn(struct vm_area_struct *vas,
+						unsigned long npages,
+						unsigned long *src_mpfn,
+						unsigned long *mpfn, u64 addr)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i, addr += PAGE_SIZE) {
+		struct page *page;
+
+		if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE))
+			continue;
+
+		if (vas)
+			page = alloc_page_vma(GFP_HIGHUSER, vas, addr);
+		else
+			page = alloc_page(GFP_HIGHUSER);
+
+		if (!page)
+			return -ENOMEM;
+
+		lock_page(page);
+		mpfn[i] = migrate_pfn(page_to_pfn(page));
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_evict_to_sram - Evict GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * Similar to __drm_gpusvm_migrate_to_sram but does not require mmap lock and
+ * migration done via migrate_device_* functions. Fallback path as it is
+ * preferred to issue migrations with mmap lock.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_evict_to_sram(struct drm_gpusvm *gpusvm,
+				    struct drm_gpusvm_range *range)
+{
+	unsigned long npages;
+	struct page **pages;
+	unsigned long *src, *dst;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	npages = npages_in_range(range->va.start, range->va.end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	src = buf;
+	dst = buf + (sizeof(*src) * npages);
+	dma_addr = buf + (2 * sizeof(*src) * npages);
+	pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) * npages;
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm, range->vram_allocation,
+					     npages, src);
+	if (err)
+		goto err_free;
+
+	err = migrate_device_vma_range(gpusvm->mm,
+				       gpusvm->device_private_page_owner, src,
+				       npages, range->va.start);
+	if (err)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(NULL, npages, src, dst, 0);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   dst, npages, DMA_BIDIRECTIONAL);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, dst);
+	migrate_device_pages(src, dst, npages);
+	migrate_device_finalize(src, dst, npages);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+
+	return err;
+}
+
+/**
+ * __drm_gpusvm_migrate_to_sram - Migrate GPU SVM range to SRAM (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @vas: Pointer to the VM area structure
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @start: Start address of the migration range
+ * @end: End address of the migration range
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to SRAM. It sets up the migration, populates + dma maps SRAM PFNs, and
+ * invokes the driver-specific operations for migration to SRAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int __drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+					struct vm_area_struct *vas,
+					struct page *page,
+					u64 start, u64 end)
+{
+	struct migrate_vma migrate = {
+		.vma		= vas,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_DEVICE_PRIVATE,
+		.fault_page	= page,
+	};
+	unsigned long npages;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	mmap_assert_locked(gpusvm->mm);
+
+	/* Corner where VMA area struct has been partially unmapped */
+	if (start < vas->vm_start)
+		start = vas->vm_start;
+	if (end > vas->vm_end)
+		end = vas->vm_end;
+
+	migrate.start = start;
+	migrate.end = end;
+	npages = npages_in_range(start, end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/* Raced with another CPU fault, nothing to do */
+	if (!migrate.cpages)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(vas, npages,
+						   migrate.src, migrate.dst,
+						   start);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   migrate.dst, npages,
+					   DMA_BIDIRECTIONAL);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(migrate.src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+	mmap_assert_locked(gpusvm->mm);
+
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_to_sram - Migrate (evict) GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function initiates the migration of the specified GPU SVM range to
+ * SRAM. It performs necessary checks and invokes the internal migration
+ * function for actual migration.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	int err;
+	bool retry = false;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		if (ctx->trylock_mmap) {
+			if (!mmap_read_trylock(mm))  {
+				err = drm_gpusvm_evict_to_sram(gpusvm, range);
+				goto err_mmput;
+			}
+		} else {
+			mmap_read_lock(mm);
+		}
+	}
+
+	mmap_assert_locked(mm);
+
+	/*
+	 * Loop required to find all VMA area structs for the corner case when
+	 * VRAM backing has been partially unmapped from MM's address space.
+	 */
+again:
+	vas = find_vma(mm, start);
+	if (!vas) {
+		if (!retry)
+			err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end <= vas->vm_start || start >= vas->vm_end) {
+		if (!retry)
+			err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	err = __drm_gpusvm_migrate_to_sram(gpusvm, vas, NULL, start, end);
+	if (err)
+		goto err_mmunlock;
+
+	if (vas->vm_end < end) {
+		retry = true;
+		start = vas->vm_end;
+		goto again;
+	}
+
+	if (!ctx->mmap_locked) {
+		mmap_read_unlock(mm);
+		/*
+		 * Using mmput_async as this function can be called while
+		 * holding a dma-resv lock, and a final put can grab the mmap
+		 * lock, causing a lock inversion.
+		 */
+		mmput_async(mm);
+	}
+
+	return 0;
+
+err_mmunlock:
+	if (!ctx->mmap_locked)
+		mmap_read_unlock(mm);
+err_mmput:
+	if (!ctx->mmap_locked)
+		mmput_async(mm);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_page_free - Put GPU SVM zone device data associated with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device data
+ * associated with a page when it is being released.
+ */
+static void drm_gpusvm_page_free(struct page *page)
+{
+	drm_gpusvm_zdd_put(page->zone_device_data);
+}
+
+/**
+ * drm_gpusvm_migrate_to_ram - Migrate GPU SVM range to RAM (page fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ * VM_FAULT_SIGBUS on failure, 0 on success.
+ */
+static vm_fault_t drm_gpusvm_migrate_to_ram(struct vm_fault *vmf)
+{
+	struct drm_gpusvm_zdd *zdd = vmf->page->zone_device_data;
+	int err;
+
+	err = __drm_gpusvm_migrate_to_sram(zdd->range->gpusvm,
+					   vmf->vma, vmf->page,
+					   zdd->range->va.start,
+					   zdd->range->va.end);
+
+	return err ? VM_FAULT_SIGBUS : 0;
+}
+
+/**
+ * drm_gpusvm_pagemap_ops - Device page map operations for GPU SVM
+ */
+static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = {
+	.page_free = drm_gpusvm_page_free,
+	.migrate_to_ram = drm_gpusvm_migrate_to_ram,
+};
+
+/**
+ * drm_gpusvm_pagemap_ops_get - Retrieve GPU SVM device page map operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void)
+{
+	return &drm_gpusvm_pagemap_ops;
+}
+
+/**
+ * drm_gpusvm_has_mapping - Check if GPU SVM has mapping for the given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end)
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	drm_gpusvm_for_each_notifier(notifier, gpusvm, start, end) {
+		struct drm_gpusvm_range *range = NULL;
+
+		drm_gpusvm_for_each_range(range, notifier, start, end)
+			return true;
+	}
+
+	return false;
+}
drivers/gpu/drm/xe/drm_gpusvm.h new file +415 -0
new file mode 100644
index 000000000000..0ea70f8534a8
@@ -0,0 +1,415 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef __DRM_GPUSVM_H__
+#define __DRM_GPUSVM_H__
+
+#include <linux/kref.h>
+#include <linux/mmu_notifier.h>
+#include <linux/workqueue.h>
+
+struct dev_pagemap_ops;
+struct drm_device;
+struct drm_gpusvm;
+struct drm_gpusvm_notifier;
+struct drm_gpusvm_ops;
+struct drm_gpusvm_range;
+
+/**
+ * struct drm_gpusvm_ops - Operations structure for GPU SVM
+ *
+ * This structure defines the operations for GPU Shared Virtual Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * perform operations such as migration between VRAM and system RAM.
+ */
+struct drm_gpusvm_ops {
+	/**
+	 * @notifier_alloc: Allocate a GPU SVM notifier (optional)
+	 *
+	 * This function shall allocate a GPU SVM notifier.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM notifier on success, NULL on failure.
+	 */
+	struct drm_gpusvm_notifier *(*notifier_alloc)(void);
+
+	/**
+	 * @notifier_free: Free a GPU SVM notifier (optional)
+	 * @notifier: Pointer to the GPU SVM notifier to be freed
+	 *
+	 * This function shall free a GPU SVM notifier.
+	 */
+	void (*notifier_free)(struct drm_gpusvm_notifier *notifier);
+
+	/**
+	 * @range_alloc: Allocate a GPU SVM range (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 *
+	 * This function shall allocate a GPU SVM range.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM range on success, NULL on failure.
+	 */
+	struct drm_gpusvm_range *(*range_alloc)(struct drm_gpusvm *gpusvm);
+
+	/**
+	 * @range_free: Free a GPU SVM range (optional)
+	 * @range: Pointer to the GPU SVM range to be freed
+	 *
+	 * This function shall free a GPU SVM range.
+	 */
+	void (*range_free)(struct drm_gpusvm_range *range);
+
+	/**
+	 * @vram_release: Release VRAM allocation (optional)
+	 * @vram_allocation: Driver-private pointer to the VRAM allocation
+	 *
+	 * This function shall release VRAM allocation and expects to drop a
+	 * reference to VRAM allocation.
+	 */
+	void (*vram_release)(void *vram_allocation);
+
+	/**
+	 * @populate_vram_pfn: Populate VRAM PFN (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @vram_allocation: Driver-private pointer to the VRAM allocation
+	 * @npages: Number of pages to populate
+	 * @pfn: Array of page frame numbers to populate
+	 *
+	 * This function shall populate VRAM page frame numbers (PFN).
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*populate_vram_pfn)(struct drm_gpusvm *gpusvm,
+				 void *vram_allocation,
+				 unsigned long npages,
+				 unsigned long *pfn);
+
+	/**
+	 * @copy_to_vram: Copy to VRAM (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (destination)
+	 * @dma_addr: Pointer to array of DMA addresses (source)
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to VRAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_vram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @copy_to_sram: Copy to system RAM (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (source)
+	 * @dma_addr: Pointer to array of DMA addresses (destination)
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to system RAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_sram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @invalidate: Invalidate GPU SVM notifier (required)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @notifier: Pointer to the GPU SVM notifier
+	 * @mmu_range: Pointer to the mmu_notifier_range structure
+	 *
+	 * This function shall invalidate the GPU page tables. It can safely
+	 * walk the notifier range RB tree/list in this function. Called while
+	 * holding the notifier lock.
+	 */
+	void (*invalidate)(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_notifier *notifier,
+			   const struct mmu_notifier_range *mmu_range);
+};
+
+/**
+ * struct drm_gpusvm_notifier - Structure representing a GPU SVM notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @interval: Interval for the notifier
+ * @rb: Red-black tree node for the parent GPU SVM structure notifier tree
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ *              interval tree. This is useful to keep iterating ranges while
+ *              doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier has been
+ *                 removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */
+struct drm_gpusvm_notifier {
+	struct drm_gpusvm *gpusvm;
+	struct mmu_interval_notifier notifier;
+	struct {
+		u64 start;
+		u64 end;
+	} interval;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct rb_root_cached root;
+	struct list_head range_list;
+	struct {
+		u32 removed : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm_range - Structure representing a GPU SVM range
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier
+ * @refcount: Reference count for the range
+ * @rb: Red-black tree node for the parent GPU SVM notifier structure range tree
+ * @va: Virtual address range
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @pages: Pointer to the array of pages (if backing store is in VRAM)
+ * @dma_addr: DMA address array (if backing store is SRAM and DMA mapped)
+ * @vram_allocation: Driver-private pointer to the VRAM allocation
+ * @order: Order of dma mapping. i.e. PAGE_SIZE << order is mapping size
+ * @flags.migrate_vram: Flag indicating whether the range can be migrated to VRAM
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_vram_pages: Flag indicating if the range has vram pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA mapping
+ * @flags.kfree_mapping: Flag indicating @dma_addr is a compact allocation based
+ *                       on @order which releases via kfree
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */
+struct drm_gpusvm_range {
+	struct drm_gpusvm *gpusvm;
+	struct drm_gpusvm_notifier *notifier;
+	struct kref refcount;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct {
+		u64 start;
+		u64 end;
+	} va;
+	unsigned long notifier_seq;
+	union {
+		struct page **pages;
+		dma_addr_t *dma_addr;
+	};
+	void *vram_allocation;
+	u16 order;
+	struct {
+		/* All flags below must be set upon creation */
+		u16 migrate_vram : 1;
+		/* All flags below must be set / cleared under notifier lock */
+		u16 unmapped : 1;
+		u16 partial_unmap : 1;
+		u16 has_vram_pages : 1;
+		u16 has_dma_mapping : 1;
+		u16 kfree_mapping : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm - GPU SVM structure
+ *
+ * @name: Name of the GPU SVM
+ * @drm: Pointer to the DRM device structure
+ * @mm: Pointer to the mm_struct for the address space
+ * @device_private_page_owner: Device private pages owner
+ * @mm_start: Start address of GPU SVM
+ * @mm_range: Range of the GPU SVM
+ * @notifier_size: Size of individual notifiers
+ * @ops: Pointer to the operations structure for GPU SVM
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range allocation.
+ *               Entries should be powers of 2 in descending order.
+ * @num_chunks: Number of chunks
+ * @notifier_lock: Read-write semaphore for protecting notifier operations
+ * @zdd_wq: Workqueue for deferred work on zdd destruction
+ * @root: Cached root node of the Red-Black tree containing GPU SVM notifiers
+ * @notifier_list: list head containing of notifiers in the same order they
+ *                 appear in interval tree. This is useful to keep iterating
+ *                 notifiers while doing modifications to RB tree.
+ *
+ * This structure represents a GPU SVM (Shared Virtual Memory) used for tracking
+ * memory ranges mapped in a DRM (Direct Rendering Manager) device.
+ *
+ * No reference counting is provided, as this is expected to be embedded in the
+ * driver VM structure along with the struct drm_gpuvm, which handles reference
+ * counting.
+ */
+struct drm_gpusvm {
+	const char *name;
+	struct drm_device *drm;
+	struct mm_struct *mm;
+	void *device_private_page_owner;
+	u64 mm_start;
+	u64 mm_range;
+	u64 notifier_size;
+	const struct drm_gpusvm_ops *ops;
+	const u64 *chunk_sizes;
+	int num_chunks;
+	struct rw_semaphore notifier_lock;
+	struct workqueue_struct *zdd_wq;
+	struct rb_root_cached root;
+	struct list_head notifier_list;
+};
+
+/**
+ * struct drm_gpusvm_ctx - DRM GPU SVM context
+ *
+ * @mmap_locked: mmap lock is locked
+ * @trylock_mmap: trylock mmap lock, used to avoid locking inversions
+ *                (e.g.dma-revs -> mmap lock)
+ * @in_notifier: entering from a MMU notifier
+ * @read_only: operating on read-only memory
+ * @vram_possible: possible to use VRAM
+ * @prefault: prefault pages
+ *
+ * Context that is DRM GPUSVM is operating in (i.e. user arguments).
+ */
+struct drm_gpusvm_ctx {
+	u32 mmap_locked :1;
+	u32 trylock_mmap :1;
+	u32 in_notifier :1;
+	u32 read_only :1;
+	u32 vram_possible :1;
+	u32 prefault :1;
+};
+
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void *device_private_page_owner,
+		    u64 mm_start, u64 mm_range, u64 notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const u64 *chunk_sizes, int num_chunks);
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm);
+void drm_gpusvm_free(struct drm_gpusvm *gpusvm);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, u64 fault_addr,
+				u64 gpuva_start, u64 gpuva_end,
+				const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range);
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range);
+
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range);
+
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx);
+
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx);
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void);
+
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, u64 start, u64 end);
+
+/**
+ * drm_gpusvm_notifier_lock - Lock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, take lock
+ */
+#define drm_gpusvm_notifier_lock(gpusvm__)	\
+	down_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * drm_gpusvm_notifier_unlock - Unlock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, drop lock
+ */
+#define drm_gpusvm_notifier_unlock(gpusvm__)	\
+	up_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * __drm_gpusvm_range_next - Get the next GPU SVM range in the list
+ * @range: a pointer to the current GPU SVM range
+ *
+ * Return: A pointer to the next drm_gpusvm_range if available, or NULL if the
+ *         current range is the last one or if the input range is NULL.
+ */
+static inline struct drm_gpusvm_range *
+__drm_gpusvm_range_next(struct drm_gpusvm_range *range)
+{
+	if (range && !list_is_last(&range->rb.entry,
+				   &range->notifier->range_list))
+		return list_next_entry(range, rb.entry);
+
+	return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_range - Iterate over GPU SVM ranges in a notifier
+ * @range__: Iterator variable for the ranges. If set, it indicates the start of
+ *	     the iterator. If NULL, call drm_gpusvm_range_find() to get the range.
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier. It is safe
+ * to use while holding the driver SVM lock or the notifier lock.
+ */
+#define drm_gpusvm_for_each_range(range__, notifier__, start__, end__)	\
+	for ((range__) = (range__) ?:					\
+	     drm_gpusvm_range_find((notifier__), (start__), (end__));	\
+	     (range__) && (range__->va.start < (end__));		\
+	     (range__) = __drm_gpusvm_range_next(range__))
+
+/**
+ * drm_gpusvm_range_set_unmapped - Mark a GPU SVM range as unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets the partial_unmap flag
+ * if the range partially falls within the provided MMU notifier range.
+ */
+static inline void
+drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,
+			      const struct mmu_notifier_range *mmu_range)
+{
+	lockdep_assert_held_write(&range->gpusvm->notifier_lock);
+
+	range->flags.unmapped = true;
+	if (range->va.start < mmu_range->start ||
+	    range->va.end > mmu_range->end)
+		range->flags.partial_unmap = true;
+}
+
+#endif /* __DRM_GPUSVM_H__ */
Signature · 1 line
2.34.1
#11 lore raw

Re: [RFC PATCH 05/28] drm/gpusvm: Add support for GPU Shared Virtual Memory

Quoted 56 lines · Matthew Brost
Quoted 13 lines · Christian König

Am 28.08.24 um 16:31 schrieb Daniel Vetter:

Quoted 12 lines · Matthew Brost
Quoted 11 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+         if (!ctx->mmap_locked) {
+                 /*
+                  * XXX: HMM locking document indicates only a read-lock
+                  * is required but there apears to be a window between
+                  * the MMU_NOTIFY_MIGRATE event triggered in a CPU fault
+                  * via migrate_vma_setup and the pages actually moving
+                  * in migrate_vma_finalize in which this code can grab
+                  * garbage pages. Grabbing the write-lock if the range
+                  * is attached to vram appears to protect against this
+                  * race.
+                  */

Thanks the comments, replying to both of you inline.

Quoted 7 lines
Quoted 7 lines

This one is really scary, since it means the entire migrate pte trickery is essentially completely busted. Grabbing the mmap write lock just means you block out pretty much everything interesting from concurrently happening.

My gut feeling says we need to figure out what's happening here, because this looks a bit too fundamental to me.

I agree. I haven’t looked into this issue for a couple of months but really need to understand what is going on.

I should have mentioned this in the cover letter: the goal of this series was to produce something for review that is stable and supports UMDs/user applications. It was not intended to be presented as a final solution. This issue certainly falls into the category of "needs to be understood and requires a proper fix."

One open question I have is whether the test case that triggers this issue is even defined behavior. The test creates concurrent access between the GPU and CPU to the same memory address, resulting in GPU and CPU faults racing against each other. It’s possible that this is undefined behavior, so data corruption might be acceptable—i.e., the kernel can’t crash, but incorrect results might be permissible.

e.g. This is the only defined usage model:

alloc_memory();
start_compute_kernel();
sync_on_compute_kernel_completion();
read_memory();

Hopefully, in the next week or so, I'll be heavily engaging with the UMD teams. Development can then start, and applications will be running soon after. This will allow us to address issues like this, collect data on memory usage, and verify some of the assumptions I've made, such as optimizing for 2M+ allocations.

Quoted 1 line
Quoted 7 lines
Quoted 3 lines

I think I have at least a high level understanding what's going on here, Felix and especially Philip should know more of the details.

I meant to reach out to AMD for issues like this. So, Felix ([email protected]) and Philip ([email protected]) would be good contacts?

Yes.

Alex

Quoted 1,272 lines
Quoted 4 lines

In general grabbing the mm_lock to protect PTEs from changing is completely nonsense. The mm_lock is to protect the VMAs and *not* the PTEs!

Thanks for the hint. I believe that in the AMD implementation, I noticed some additional locks for migration, which might be how you mitigated this issue.

I must say it is a bit unfortunate that the HMM locking documentation doesn’t mention this. I believe the documentation needs additional information, which I can add once we finalize the solution.

Matt

Quoted 1,257 lines

Even with the write side of the mm_lock taken it is perfectly possible that PTE change. It's just less likely.

We run into multiple issues before we figured out this important distinction as well.

Christian.

Quoted 1,248 lines

-Sima

Quoted 1,245 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +821 -0
+                 if (vram_pages)
+                         mmap_write_lock(mm);
+                 else
+                         mmap_read_lock(mm);
+         }
+         err = hmm_range_fault(&hmm_range);
+         if (!ctx->mmap_locked) {
+                 if (vram_pages)
+                         mmap_write_unlock(mm);
+                 else
+                         mmap_read_unlock(mm);
+         }
+
+         if (err == -EBUSY) {
+                 if (time_after(jiffies, timeout))
+                         break;
+
+                 hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
+                 continue;
+         }
+         break;
+ }
+ if (!ctx->mmap_locked)
+         mmput(mm);
+ if (err)
+         goto err_free;
+
+ pages = (struct page **)pfns;
+
+ if (ctx->prefault) {
+         range->pages = pages;
+         goto set_seqno;
+ }
+
+map_pages:
+ if (is_device_private_page(hmm_pfn_to_page(pfns[0]))) {
+         WARN_ON_ONCE(!range->vram_allocation);
+
+         for (i = 0; i < npages; ++i) {
+                 pages[i] = hmm_pfn_to_page(pfns[i]);
+
+                 if (WARN_ON_ONCE(!is_device_private_page(pages[i]))) {
+                         err = -EOPNOTSUPP;
+                         goto err_free;
+                 }
+         }
+
+         /* Do not race with notifier unmapping pages */
+         drm_gpusvm_notifier_lock(gpusvm);
+         range->flags.has_vram_pages = true;
+         range->pages = pages;
+         if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+                 err = -EAGAIN;
+                 __drm_gpusvm_range_unmap_pages(gpusvm, range);
+         }
+         drm_gpusvm_notifier_unlock(gpusvm);
+ } else {
+         dma_addr_t *dma_addr = (dma_addr_t *)pfns;
+
+         for_each_dma_page(i, j, npages, order) {
+                 if (WARN_ON_ONCE(i && order !=
+                                  hmm_pfn_to_map_order(pfns[i]))) {
+                         err = -EOPNOTSUPP;
+                         npages = i;
+                         goto err_unmap;
+                 }
+                 order = hmm_pfn_to_map_order(pfns[i]);
+
+                 pages[j] = hmm_pfn_to_page(pfns[i]);
+                 if (WARN_ON_ONCE(is_zone_device_page(pages[j]))) {
+                         err = -EOPNOTSUPP;
+                         npages = i;
+                         goto err_unmap;
+                 }
+
+                 set_page_dirty_lock(pages[j]);
+                 mark_page_accessed(pages[j]);
+
+                 dma_addr[j] = dma_map_page(gpusvm->drm->dev,
+                                            pages[j], 0,
+                                            PAGE_SIZE << order,
+                                            DMA_BIDIRECTIONAL);
+                 if (dma_mapping_error(gpusvm->drm->dev, dma_addr[j])) {
+                         err = -EFAULT;
+                         npages = i;
+                         goto err_unmap;
+                 }
+         }
+
+         /* Huge pages, reduce memory footprint */
+         if (order) {
+                 dma_addr = kmalloc_array(j, sizeof(*dma_addr),
+                                          GFP_KERNEL);
+                 if (dma_addr) {
+                         for (i = 0; i < j; ++i)
+                                 dma_addr[i] = (dma_addr_t)pfns[i];
+                         kvfree(pfns);
+                         kfree_mapping = true;
+                 } else {
+                         dma_addr = (dma_addr_t *)pfns;
+                 }
+         }
+
+         /* Do not race with notifier unmapping pages */
+         drm_gpusvm_notifier_lock(gpusvm);
+         range->order = order;
+         range->flags.kfree_mapping = kfree_mapping;
+         range->flags.has_dma_mapping = true;
+         range->dma_addr = dma_addr;
+         range->vram_allocation = NULL;
+         if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+                 err = -EAGAIN;
+                 __drm_gpusvm_range_unmap_pages(gpusvm, range);
+         }
+         drm_gpusvm_notifier_unlock(gpusvm);
+ }
+
+ if (err == -EAGAIN)
+         goto retry;
+set_seqno:
+ range->notifier_seq = hmm_range.notifier_seq;
+
+ return 0;
+
+err_unmap:
+ for_each_dma_page(i, j, npages, order)
+         dma_unmap_page(gpusvm->drm->dev,
+                        (dma_addr_t)pfns[j],
+                        PAGE_SIZE << order, DMA_BIDIRECTIONAL);
+err_free:
+ if (alloc_pfns)
+         kvfree(pfns);
+err_out:
+ return err;
+}
+
+/**
+ * drm_gpusvm_range_unmap_pages - Unmap pages associated with a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed that gpusvm->notifier_lock is held in write mode; if it
+ * is clear, it acquires gpusvm->notifier_lock in read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops->invalidate for IOMMU
+ * security model.
+ */
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+                           struct drm_gpusvm_range *range,
+                           const struct drm_gpusvm_ctx *ctx)
+{
+ if (ctx->in_notifier)
+         lockdep_assert_held_write(&gpusvm->notifier_lock);
+ else
+         drm_gpusvm_notifier_lock(gpusvm);
+
+ __drm_gpusvm_range_unmap_pages(gpusvm, range);
+
+ if (!ctx->in_notifier)
+         drm_gpusvm_notifier_unlock(gpusvm);
+}
+
+/**
+ * drm_gpusvm_migration_put_page - Put a migration page
+ * @page: Pointer to the page to put
+ *
+ * This function unlocks and puts a page.
+ */
+static void drm_gpusvm_migration_put_page(struct page *page)
+{
+ unlock_page(page);
+ put_page(page);
+}
+
+/**
+ * drm_gpusvm_migration_put_pages - Put migration pages
+ * @npages: Number of pages
+ * @migrate_pfn: Array of migrate page frame numbers
+ *
+ * This function puts an array of pages.
+ */
+static void drm_gpusvm_migration_put_pages(unsigned long npages,
+                                    unsigned long *migrate_pfn)
+{
+ unsigned long i;
+
+ for (i = 0; i < npages; ++i) {
+         if (!migrate_pfn[i])
+                 continue;
+
+         drm_gpusvm_migration_put_page(migrate_pfn_to_page(migrate_pfn[i]));
+         migrate_pfn[i] = 0;
+ }
+}
+
+/**
+ * drm_gpusvm_get_vram_page - Get a reference to a VRAM page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU SVM zone
+ * device data and initializes it for zone device usage.
+ */
+static void drm_gpusvm_get_vram_page(struct page *page,
+                              struct drm_gpusvm_zdd *zdd)
+{
+ page->zone_device_data = drm_gpusvm_zdd_get(zdd);
+ zone_device_page_init(page);
+}
+
+/**
+ * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Return: 0 on success, -EFAULT if an error occurs during mapping.
+ */
+static int drm_gpusvm_migrate_map_pages(struct device *dev,
+                                 dma_addr_t *dma_addr,
+                                 long unsigned int *migrate_pfn,
+                                 unsigned long npages,
+                                 enum dma_data_direction dir)
+{
+ unsigned long i;
+
+ for (i = 0; i < npages; ++i) {
+         struct page *page = migrate_pfn_to_page(migrate_pfn[i]);
+
+         if (!page)
+                 continue;
+
+         if (WARN_ON_ONCE(is_zone_device_page(page)))
+                 return -EFAULT;
+
+         dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE, dir);
+         if (dma_mapping_error(dev, dma_addr[i]))
+                 return -EFAULT;
+ }
+
+ return 0;
+}
+
+/**
+ * drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */
+static void drm_gpusvm_migrate_unmap_pages(struct device *dev,
+                                    dma_addr_t *dma_addr,
+                                    unsigned long npages,
+                                    enum dma_data_direction dir)
+{
+ unsigned long i;
+
+ for (i = 0; i < npages; ++i) {
+         if (!dma_addr[i] || dma_mapping_error(dev, dma_addr[i]))
+                 continue;
+
+         dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir);
+ }
+}
+
+/**
+ * drm_gpusvm_migrate_to_vram - Migrate GPU SVM range to VRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *                   failure of this function.
+ * @vram_allocation: Driver-private pointer to the VRAM allocation. The caller
+ *                   should hold a reference to the VRAM allocation, which
+ *                   should be dropped via ops->vram_allocation or upon the
+ *                   failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to VRAM. It performs the
+ * necessary setup and invokes the driver-specific operations for migration to
+ * VRAM. Upon successful return, @vram_allocation can safely reference @range
+ * until ops->vram_release is called which only upon successful return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+                        struct drm_gpusvm_range *range,
+                        void *vram_allocation,
+                        const struct drm_gpusvm_ctx *ctx)
+{
+ u64 start = range->va.start, end = range->va.end;
+ struct migrate_vma migrate = {
+         .start          = start,
+         .end            = end,
+         .pgmap_owner    = gpusvm->device_private_page_owner,
+         .flags          = MIGRATE_VMA_SELECT_SYSTEM,
+ };
+ struct mm_struct *mm = gpusvm->mm;
+ unsigned long i, npages = npages_in_range(start, end);
+ struct vm_area_struct *vas;
+ struct drm_gpusvm_zdd *zdd = NULL;
+ struct page **pages;
+ dma_addr_t *dma_addr;
+ void *buf;
+ int err;
+
+ if (!range->flags.migrate_vram)
+         return -EINVAL;
+
+ if (!gpusvm->ops->populate_vram_pfn || !gpusvm->ops->copy_to_vram ||
+     !gpusvm->ops->copy_to_sram)
+         return -EOPNOTSUPP;
+
+ if (!ctx->mmap_locked) {
+         if (!mmget_not_zero(mm)) {
+                 err = -EFAULT;
+                 goto err_out;
+         }
+         mmap_write_lock(mm);
+ }
+
+ mmap_assert_locked(mm);
+
+ vas = vma_lookup(mm, start);
+ if (!vas) {
+         err = -ENOENT;
+         goto err_mmunlock;
+ }
+
+ if (end > vas->vm_end || start < vas->vm_start) {
+         err = -EINVAL;
+         goto err_mmunlock;
+ }
+
+ if (!vma_is_anonymous(vas)) {
+         err = -EBUSY;
+         goto err_mmunlock;
+ }
+
+ buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+                sizeof(*pages), GFP_KERNEL);
+ if (!buf) {
+         err = -ENOMEM;
+         goto err_mmunlock;
+ }
+ dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+ pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+ zdd = drm_gpusvm_zdd_alloc(range);
+ if (!zdd) {
+         err = -ENOMEM;
+         goto err_free;
+ }
+
+ migrate.vma = vas;
+ migrate.src = buf;
+ migrate.dst = migrate.src + npages;
+
+ err = migrate_vma_setup(&migrate);
+ if (err)
+         goto err_free;
+
+ /*
+  * FIXME: Below cases, !migrate.cpages and migrate.cpages != npages, not
+  * always an error. Need to revisit possible cases and how to handle. We
+  * could prefault on migrate.cpages != npages via hmm_range_fault.
+  */
+
+ if (!migrate.cpages) {
+         err = -EFAULT;
+         goto err_free;
+ }
+
+ if (migrate.cpages != npages) {
+         err = -EBUSY;
+         goto err_finalize;
+ }
+
+ err = gpusvm->ops->populate_vram_pfn(gpusvm, vram_allocation, npages,
+                                      migrate.dst);
+ if (err)
+         goto err_finalize;
+
+ err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+                                    migrate.src, npages, DMA_TO_DEVICE);
+ if (err)
+         goto err_finalize;
+
+ for (i = 0; i < npages; ++i) {
+         struct page *page = pfn_to_page(migrate.dst[i]);
+
+         pages[i] = page;
+         migrate.dst[i] = migrate_pfn(migrate.dst[i]);
+         drm_gpusvm_get_vram_page(page, zdd);
+ }
+
+ err = gpusvm->ops->copy_to_vram(gpusvm, pages, dma_addr, npages);
+ if (err)
+         goto err_finalize;
+
+ /* Upon success bind vram allocation to range and zdd */
+ range->vram_allocation = vram_allocation;
+ WRITE_ONCE(zdd->vram_allocation, vram_allocation);      /* Owns ref */
+
+err_finalize:
+ if (err)
+         drm_gpusvm_migration_put_pages(npages, migrate.dst);
+ migrate_vma_pages(&migrate);
+ migrate_vma_finalize(&migrate);
+ drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+                                DMA_TO_DEVICE);
+err_free:
+ if (zdd)
+         drm_gpusvm_zdd_put(zdd);
+ kvfree(buf);
+err_mmunlock:
+ if (!ctx->mmap_locked) {
+         mmap_write_unlock(mm);
+         mmput(mm);
+ }
+err_out:
+ return err;
+}
+
+/**
+ * drm_gpusvm_migrate_populate_sram_pfn - Populate SRAM PFNs for a VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @npages: Number of pages to populate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the SRAM migrate page frame numbers (PFNs) for the
+ * specified VM area structure. It allocates and locks pages in the VM area for
+ * SRAM usage. If vas is non-NULL use alloc_page_vma for allocation, if NULL use
+ * alloc_page for allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_sram_pfn(struct vm_area_struct *vas,
+                                         unsigned long npages,
+                                         unsigned long *src_mpfn,
+                                         unsigned long *mpfn, u64 addr)
+{
+ unsigned long i;
+
+ for (i = 0; i < npages; ++i, addr += PAGE_SIZE) {
+         struct page *page;
+
+         if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE))
+                 continue;
+
+         if (vas)
+                 page = alloc_page_vma(GFP_HIGHUSER, vas, addr);
+         else
+                 page = alloc_page(GFP_HIGHUSER);
+
+         if (!page)
+                 return -ENOMEM;
+
+         lock_page(page);
+         mpfn[i] = migrate_pfn(page_to_pfn(page));
+ }
+
+ return 0;
+}
+
+/**
+ * drm_gpusvm_evict_to_sram - Evict GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * Similar to __drm_gpusvm_migrate_to_sram but does not require mmap lock and
+ * migration done via migrate_device_* functions. Fallback path as it is
+ * preferred to issue migrations with mmap lock.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_evict_to_sram(struct drm_gpusvm *gpusvm,
+                             struct drm_gpusvm_range *range)
+{
+ unsigned long npages;
+ struct page **pages;
+ unsigned long *src, *dst;
+ dma_addr_t *dma_addr;
+ void *buf;
+ int i, err = 0;
+
+ npages = npages_in_range(range->va.start, range->va.end);
+
+ buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr) +
+                sizeof(*pages), GFP_KERNEL);
+ if (!buf) {
+         err = -ENOMEM;
+         goto err_out;
+ }
+ src = buf;
+ dst = buf + (sizeof(*src) * npages);
+ dma_addr = buf + (2 * sizeof(*src) * npages);
+ pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) * npages;
+
+ err = gpusvm->ops->populate_vram_pfn(gpusvm, range->vram_allocation,
+                                      npages, src);
+ if (err)
+         goto err_free;
+
+ err = migrate_device_vma_range(gpusvm->mm,
+                                gpusvm->device_private_page_owner, src,
+                                npages, range->va.start);
+ if (err)
+         goto err_free;
+
+ err = drm_gpusvm_migrate_populate_sram_pfn(NULL, npages, src, dst, 0);
+ if (err)
+         goto err_finalize;
+
+ err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+                                    dst, npages, DMA_BIDIRECTIONAL);
+ if (err)
+         goto err_finalize;
+
+ for (i = 0; i < npages; ++i)
+         pages[i] = migrate_pfn_to_page(src[i]);
+
+ err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+ if (err)
+         goto err_finalize;
+
+err_finalize:
+ if (err)
+         drm_gpusvm_migration_put_pages(npages, dst);
+ migrate_device_pages(src, dst, npages);
+ migrate_device_finalize(src, dst, npages);
+ drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+                                DMA_BIDIRECTIONAL);
+err_free:
+ kvfree(buf);
+err_out:
+
+ return err;
+}
+
+/**
+ * __drm_gpusvm_migrate_to_sram - Migrate GPU SVM range to SRAM (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @vas: Pointer to the VM area structure
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @start: Start address of the migration range
+ * @end: End address of the migration range
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to SRAM. It sets up the migration, populates + dma maps SRAM PFNs, and
+ * invokes the driver-specific operations for migration to SRAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int __drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+                                 struct vm_area_struct *vas,
+                                 struct page *page,
+                                 u64 start, u64 end)
+{
+ struct migrate_vma migrate = {
+         .vma            = vas,
+         .pgmap_owner    = gpusvm->device_private_page_owner,
+         .flags          = MIGRATE_VMA_SELECT_DEVICE_PRIVATE,
+         .fault_page     = page,
+ };
+ unsigned long npages;
+ struct page **pages;
+ dma_addr_t *dma_addr;
+ void *buf;
+ int i, err = 0;
+
+ mmap_assert_locked(gpusvm->mm);
+
+ /* Corner where VMA area struct has been partially unmapped */
+ if (start < vas->vm_start)
+         start = vas->vm_start;
+ if (end > vas->vm_end)
+         end = vas->vm_end;
+
+ migrate.start = start;
+ migrate.end = end;
+ npages = npages_in_range(start, end);
+
+ buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+                sizeof(*pages), GFP_KERNEL);
+ if (!buf) {
+         err = -ENOMEM;
+         goto err_out;
+ }
+ dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+ pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+ migrate.vma = vas;
+ migrate.src = buf;
+ migrate.dst = migrate.src + npages;
+
+ err = migrate_vma_setup(&migrate);
+ if (err)
+         goto err_free;
+
+ /* Raced with another CPU fault, nothing to do */
+ if (!migrate.cpages)
+         goto err_free;
+
+ err = drm_gpusvm_migrate_populate_sram_pfn(vas, npages,
+                                            migrate.src, migrate.dst,
+                                            start);
+ if (err)
+         goto err_finalize;
+
+ err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+                                    migrate.dst, npages,
+                                    DMA_BIDIRECTIONAL);
+ if (err)
+         goto err_finalize;
+
+ for (i = 0; i < npages; ++i)
+         pages[i] = migrate_pfn_to_page(migrate.src[i]);
+
+ err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+ if (err)
+         goto err_finalize;
+
+err_finalize:
+ if (err)
+         drm_gpusvm_migration_put_pages(npages, migrate.dst);
+ migrate_vma_pages(&migrate);
+ migrate_vma_finalize(&migrate);
+ drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+                                DMA_BIDIRECTIONAL);
+err_free:
+ kvfree(buf);
+err_out:
+ mmap_assert_locked(gpusvm->mm);
+
+ return err;
+}
+
+/**
+ * drm_gpusvm_migrate_to_sram - Migrate (evict) GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function initiates the migration of the specified GPU SVM range to
+ * SRAM. It performs necessary checks and invokes the internal migration
+ * function for actual migration.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+                        struct drm_gpusvm_range *range,
+                        const struct drm_gpusvm_ctx *ctx)
+{
+ u64 start = range->va.start, end = range->va.end;
+ struct mm_struct *mm = gpusvm->mm;
+ struct vm_area_struct *vas;
+ int err;
+ bool retry = false;
+
+ if (!ctx->mmap_locked) {
+         if (!mmget_not_zero(mm)) {
+                 err = -EFAULT;
+                 goto err_out;
+         }
+         if (ctx->trylock_mmap) {
+                 if (!mmap_read_trylock(mm))  {
+                         err = drm_gpusvm_evict_to_sram(gpusvm, range);
+                         goto err_mmput;
+                 }
+         } else {
+                 mmap_read_lock(mm);
+         }
+ }
+
+ mmap_assert_locked(mm);
+
+ /*
+  * Loop required to find all VMA area structs for the corner case when
+  * VRAM backing has been partially unmapped from MM's address space.
+  */
+again:
+ vas = find_vma(mm, start);
+ if (!vas) {
+         if (!retry)
+                 err = -ENOENT;
+         goto err_mmunlock;
+ }
+
+ if (end <= vas->vm_start || start >= vas->vm_end) {
+         if (!retry)
+                 err = -EINVAL;
+         goto err_mmunlock;
+ }
+
+ err = __drm_gpusvm_migrate_to_sram(gpusvm, vas, NULL, start, end);
+ if (err)
+         goto err_mmunlock;
+
+ if (vas->vm_end < end) {
+         retry = true;
+         start = vas->vm_end;
+         goto again;
+ }
+
+ if (!ctx->mmap_locked) {
+         mmap_read_unlock(mm);
+         /*
+          * Using mmput_async as this function can be called while
+          * holding a dma-resv lock, and a final put can grab the mmap
+          * lock, causing a lock inversion.
+          */
+         mmput_async(mm);
+ }
+
+ return 0;
+
+err_mmunlock:
+ if (!ctx->mmap_locked)
+         mmap_read_unlock(mm);
+err_mmput:
+ if (!ctx->mmap_locked)
+         mmput_async(mm);
+err_out:
+ return err;
+}
+
+/**
+ * drm_gpusvm_page_free - Put GPU SVM zone device data associated with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device data
+ * associated with a page when it is being released.
+ */
+static void drm_gpusvm_page_free(struct page *page)
+{
+ drm_gpusvm_zdd_put(page->zone_device_data);
+}
+
+/**
+ * drm_gpusvm_migrate_to_ram - Migrate GPU SVM range to RAM (page fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ * VM_FAULT_SIGBUS on failure, 0 on success.
+ */
+static vm_fault_t drm_gpusvm_migrate_to_ram(struct vm_fault *vmf)
+{
+ struct drm_gpusvm_zdd *zdd = vmf->page->zone_device_data;
+ int err;
+
+ err = __drm_gpusvm_migrate_to_sram(zdd->range->gpusvm,
+                                    vmf->vma, vmf->page,
+                                    zdd->range->va.start,
+                                    zdd->range->va.end);
+
+ return err ? VM_FAULT_SIGBUS : 0;
+}
+
+/**
+ * drm_gpusvm_pagemap_ops - Device page map operations for GPU SVM
+ */
+static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = {
+ .page_free = drm_gpusvm_page_free,
+ .migrate_to_ram = drm_gpusvm_migrate_to_ram,
+};
+
+/**
+ * drm_gpusvm_pagemap_ops_get - Retrieve GPU SVM device page map operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void)
+{
+ return &drm_gpusvm_pagemap_ops;
+}
+
+/**
+ * drm_gpusvm_has_mapping - Check if GPU SVM has mapping for the given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end)
+{
+ struct drm_gpusvm_notifier *notifier;
+
+ drm_gpusvm_for_each_notifier(notifier, gpusvm, start, end) {
+         struct drm_gpusvm_range *range = NULL;
+
+         drm_gpusvm_for_each_range(range, notifier, start, end)
+                 return true;
+ }
+
+ return false;
+}
drivers/gpu/drm/xe/drm_gpusvm.h new file +415 -1
new file mode 100644
index 000000000000..0ea70f8534a8
@@ -0,0 +1,415 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef __DRM_GPUSVM_H__
+#define __DRM_GPUSVM_H__
+
+#include <linux/kref.h>
+#include <linux/mmu_notifier.h>
+#include <linux/workqueue.h>
+
+struct dev_pagemap_ops;
+struct drm_device;
+struct drm_gpusvm;
+struct drm_gpusvm_notifier;
+struct drm_gpusvm_ops;
+struct drm_gpusvm_range;
+
+/**
+ * struct drm_gpusvm_ops - Operations structure for GPU SVM
+ *
+ * This structure defines the operations for GPU Shared Virtual Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * perform operations such as migration between VRAM and system RAM.
+ */
+struct drm_gpusvm_ops {
+ /**
+  * @notifier_alloc: Allocate a GPU SVM notifier (optional)
+  *
+  * This function shall allocate a GPU SVM notifier.
+  *
+  * Returns:
+  * Pointer to the allocated GPU SVM notifier on success, NULL on failure.
+  */
+ struct drm_gpusvm_notifier *(*notifier_alloc)(void);
+
+ /**
+  * @notifier_free: Free a GPU SVM notifier (optional)
+  * @notifier: Pointer to the GPU SVM notifier to be freed
+  *
+  * This function shall free a GPU SVM notifier.
+  */
+ void (*notifier_free)(struct drm_gpusvm_notifier *notifier);
+
+ /**
+  * @range_alloc: Allocate a GPU SVM range (optional)
+  * @gpusvm: Pointer to the GPU SVM
+  *
+  * This function shall allocate a GPU SVM range.
+  *
+  * Returns:
+  * Pointer to the allocated GPU SVM range on success, NULL on failure.
+  */
+ struct drm_gpusvm_range *(*range_alloc)(struct drm_gpusvm *gpusvm);
+
+ /**
+  * @range_free: Free a GPU SVM range (optional)
+  * @range: Pointer to the GPU SVM range to be freed
+  *
+  * This function shall free a GPU SVM range.
+  */
+ void (*range_free)(struct drm_gpusvm_range *range);
+
+ /**
+  * @vram_release: Release VRAM allocation (optional)
+  * @vram_allocation: Driver-private pointer to the VRAM allocation
+  *
+  * This function shall release VRAM allocation and expects to drop a
+  * reference to VRAM allocation.
+  */
+ void (*vram_release)(void *vram_allocation);
+
+ /**
+  * @populate_vram_pfn: Populate VRAM PFN (required for migration)
+  * @gpusvm: Pointer to the GPU SVM
+  * @vram_allocation: Driver-private pointer to the VRAM allocation
+  * @npages: Number of pages to populate
+  * @pfn: Array of page frame numbers to populate
+  *
+  * This function shall populate VRAM page frame numbers (PFN).
+  *
+  * Returns:
+  * 0 on success, a negative error code on failure.
+  */
+ int (*populate_vram_pfn)(struct drm_gpusvm *gpusvm,
+                          void *vram_allocation,
+                          unsigned long npages,
+                          unsigned long *pfn);
+
+ /**
+  * @copy_to_vram: Copy to VRAM (required for migration)
+  * @gpusvm: Pointer to the GPU SVM
+  * @pages: Pointer to array of VRAM pages (destination)
+  * @dma_addr: Pointer to array of DMA addresses (source)
+  * @npages: Number of pages to copy
+  *
+  * This function shall copy pages to VRAM.
+  *
+  * Returns:
+  * 0 on success, a negative error code on failure.
+  */
+ int (*copy_to_vram)(struct drm_gpusvm *gpusvm,
+                     struct page **pages,
+                     dma_addr_t *dma_addr,
+                     unsigned long npages);
+
+ /**
+  * @copy_to_sram: Copy to system RAM (required for migration)
+  * @gpusvm: Pointer to the GPU SVM
+  * @pages: Pointer to array of VRAM pages (source)
+  * @dma_addr: Pointer to array of DMA addresses (destination)
+  * @npages: Number of pages to copy
+  *
+  * This function shall copy pages to system RAM.
+  *
+  * Returns:
+  * 0 on success, a negative error code on failure.
+  */
+ int (*copy_to_sram)(struct drm_gpusvm *gpusvm,
+                     struct page **pages,
+                     dma_addr_t *dma_addr,
+                     unsigned long npages);
+
+ /**
+  * @invalidate: Invalidate GPU SVM notifier (required)
+  * @gpusvm: Pointer to the GPU SVM
+  * @notifier: Pointer to the GPU SVM notifier
+  * @mmu_range: Pointer to the mmu_notifier_range structure
+  *
+  * This function shall invalidate the GPU page tables. It can safely
+  * walk the notifier range RB tree/list in this function. Called while
+  * holding the notifier lock.
+  */
+ void (*invalidate)(struct drm_gpusvm *gpusvm,
+                    struct drm_gpusvm_notifier *notifier,
+                    const struct mmu_notifier_range *mmu_range);
+};
+
+/**
+ * struct drm_gpusvm_notifier - Structure representing a GPU SVM notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @interval: Interval for the notifier
+ * @rb: Red-black tree node for the parent GPU SVM structure notifier tree
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ *              interval tree. This is useful to keep iterating ranges while
+ *              doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier has been
+ *                 removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */
+struct drm_gpusvm_notifier {
+ struct drm_gpusvm *gpusvm;
+ struct mmu_interval_notifier notifier;
+ struct {
+         u64 start;
+         u64 end;
+ } interval;
+ struct {
+         struct rb_node node;
+         struct list_head entry;
+         u64 __subtree_last;
+ } rb;
+ struct rb_root_cached root;
+ struct list_head range_list;
+ struct {
+         u32 removed : 1;
+ } flags;
+};
+
+/**
+ * struct drm_gpusvm_range - Structure representing a GPU SVM range
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier
+ * @refcount: Reference count for the range
+ * @rb: Red-black tree node for the parent GPU SVM notifier structure range tree
+ * @va: Virtual address range
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @pages: Pointer to the array of pages (if backing store is in VRAM)
+ * @dma_addr: DMA address array (if backing store is SRAM and DMA mapped)
+ * @vram_allocation: Driver-private pointer to the VRAM allocation
+ * @order: Order of dma mapping. i.e. PAGE_SIZE << order is mapping size
+ * @flags.migrate_vram: Flag indicating whether the range can be migrated to VRAM
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_vram_pages: Flag indicating if the range has vram pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA mapping
+ * @flags.kfree_mapping: Flag indicating @dma_addr is a compact allocation based
+ *                       on @order which releases via kfree
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */
+struct drm_gpusvm_range {
+ struct drm_gpusvm *gpusvm;
+ struct drm_gpusvm_notifier *notifier;
+ struct kref refcount;
+ struct {
+         struct rb_node node;
+         struct list_head entry;
+         u64 __subtree_last;
+ } rb;
+ struct {
+         u64 start;
+         u64 end;
+ } va;
+ unsigned long notifier_seq;
+ union {
+         struct page **pages;
+         dma_addr_t *dma_addr;
+ };
+ void *vram_allocation;
+ u16 order;
+ struct {
+         /* All flags below must be set upon creation */
+         u16 migrate_vram : 1;
+         /* All flags below must be set / cleared under notifier lock */
+         u16 unmapped : 1;
+         u16 partial_unmap : 1;
+         u16 has_vram_pages : 1;
+         u16 has_dma_mapping : 1;
+         u16 kfree_mapping : 1;
+ } flags;
+};
+
+/**
+ * struct drm_gpusvm - GPU SVM structure
+ *
+ * @name: Name of the GPU SVM
+ * @drm: Pointer to the DRM device structure
+ * @mm: Pointer to the mm_struct for the address space
+ * @device_private_page_owner: Device private pages owner
+ * @mm_start: Start address of GPU SVM
+ * @mm_range: Range of the GPU SVM
+ * @notifier_size: Size of individual notifiers
+ * @ops: Pointer to the operations structure for GPU SVM
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range allocation.
+ *               Entries should be powers of 2 in descending order.
+ * @num_chunks: Number of chunks
+ * @notifier_lock: Read-write semaphore for protecting notifier operations
+ * @zdd_wq: Workqueue for deferred work on zdd destruction
+ * @root: Cached root node of the Red-Black tree containing GPU SVM notifiers
+ * @notifier_list: list head containing of notifiers in the same order they
+ *                 appear in interval tree. This is useful to keep iterating
+ *                 notifiers while doing modifications to RB tree.
+ *
+ * This structure represents a GPU SVM (Shared Virtual Memory) used for tracking
+ * memory ranges mapped in a DRM (Direct Rendering Manager) device.
+ *
+ * No reference counting is provided, as this is expected to be embedded in the
+ * driver VM structure along with the struct drm_gpuvm, which handles reference
+ * counting.
+ */
+struct drm_gpusvm {
+ const char *name;
+ struct drm_device *drm;
+ struct mm_struct *mm;
+ void *device_private_page_owner;
+ u64 mm_start;
+ u64 mm_range;
+ u64 notifier_size;
+ const struct drm_gpusvm_ops *ops;
+ const u64 *chunk_sizes;
+ int num_chunks;
+ struct rw_semaphore notifier_lock;
+ struct workqueue_struct *zdd_wq;
+ struct rb_root_cached root;
+ struct list_head notifier_list;
+};
+
+/**
+ * struct drm_gpusvm_ctx - DRM GPU SVM context
+ *
+ * @mmap_locked: mmap lock is locked
+ * @trylock_mmap: trylock mmap lock, used to avoid locking inversions
+ *                (e.g.dma-revs -> mmap lock)
+ * @in_notifier: entering from a MMU notifier
+ * @read_only: operating on read-only memory
+ * @vram_possible: possible to use VRAM
+ * @prefault: prefault pages
+ *
+ * Context that is DRM GPUSVM is operating in (i.e. user arguments).
+ */
+struct drm_gpusvm_ctx {
+ u32 mmap_locked :1;
+ u32 trylock_mmap :1;
+ u32 in_notifier :1;
+ u32 read_only :1;
+ u32 vram_possible :1;
+ u32 prefault :1;
+};
+
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+             const char *name, struct drm_device *drm,
+             struct mm_struct *mm, void *device_private_page_owner,
+             u64 mm_start, u64 mm_range, u64 notifier_size,
+             const struct drm_gpusvm_ops *ops,
+             const u64 *chunk_sizes, int num_chunks);
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm);
+void drm_gpusvm_free(struct drm_gpusvm *gpusvm);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, u64 fault_addr,
+                         u64 gpuva_start, u64 gpuva_end,
+                         const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+                      struct drm_gpusvm_range *range);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range);
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range);
+
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+                           struct drm_gpusvm_range *range);
+
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+                        struct drm_gpusvm_range *range,
+                        const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+                           struct drm_gpusvm_range *range,
+                           const struct drm_gpusvm_ctx *ctx);
+
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+                        struct drm_gpusvm_range *range,
+                        void *vram_allocation,
+                        const struct drm_gpusvm_ctx *ctx);
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+                        struct drm_gpusvm_range *range,
+                        const struct drm_gpusvm_ctx *ctx);
+
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void);
+
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, u64 start, u64 end);
+
+/**
+ * drm_gpusvm_notifier_lock - Lock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, take lock
+ */
+#define drm_gpusvm_notifier_lock(gpusvm__)       \
+ down_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * drm_gpusvm_notifier_unlock - Unlock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, drop lock
+ */
+#define drm_gpusvm_notifier_unlock(gpusvm__)     \
+ up_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * __drm_gpusvm_range_next - Get the next GPU SVM range in the list
+ * @range: a pointer to the current GPU SVM range
+ *
+ * Return: A pointer to the next drm_gpusvm_range if available, or NULL if the
+ *         current range is the last one or if the input range is NULL.
+ */
+static inline struct drm_gpusvm_range *
+__drm_gpusvm_range_next(struct drm_gpusvm_range *range)
+{
+ if (range && !list_is_last(&range->rb.entry,
+                            &range->notifier->range_list))
+         return list_next_entry(range, rb.entry);
+
+ return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_range - Iterate over GPU SVM ranges in a notifier
+ * @range__: Iterator variable for the ranges. If set, it indicates the start of
+ *            the iterator. If NULL, call drm_gpusvm_range_find() to get the range.
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier. It is safe
+ * to use while holding the driver SVM lock or the notifier lock.
+ */
+#define drm_gpusvm_for_each_range(range__, notifier__, start__, end__)   \
+ for ((range__) = (range__) ?:                                   \
+      drm_gpusvm_range_find((notifier__), (start__), (end__));   \
+      (range__) && (range__->va.start < (end__));                \
+      (range__) = __drm_gpusvm_range_next(range__))
+
+/**
+ * drm_gpusvm_range_set_unmapped - Mark a GPU SVM range as unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets the partial_unmap flag
+ * if the range partially falls within the provided MMU notifier range.
+ */
+static inline void
+drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,
+                       const struct mmu_notifier_range *mmu_range)
+{
+ lockdep_assert_held_write(&range->gpusvm->notifier_lock);
+
+ range->flags.unmapped = true;
+ if (range->va.start < mmu_range->start ||
+     range->va.end > mmu_range->end)
+         range->flags.partial_unmap = true;
+}
+
+#endif /* __DRM_GPUSVM_H__ */
--

2.34.1

#12 lore raw

Re: [RFC PATCH 05/28] drm/gpusvm: Add support for GPU Shared Virtual Memory

In reply to Matthew Brost
Quoted 34 lines · Matthew Brost
Quoted 13 lines · Christian König

Am 28.08.24 um 16:31 schrieb Daniel Vetter:

Quoted 12 lines · Matthew Brost

drivers/gpu/drm/xe/drm_gpusvm.c line 1353

Quoted 11 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+		if (!ctx->mmap_locked) {
+			/*
+			 * XXX: HMM locking document indicates only a read-lock
+			 * is required but there apears to be a window between
+			 * the MMU_NOTIFY_MIGRATE event triggered in a CPU fault
+			 * via migrate_vma_setup and the pages actually moving
+			 * in migrate_vma_finalize in which this code can grab
+			 * garbage pages. Grabbing the write-lock if the range
+			 * is attached to vram appears to protect against this
+			 * race.
+			 */

Thanks the comments, replying to both of you inline.

Quoted 7 lines
Quoted 7 lines

This one is really scary, since it means the entire migrate pte trickery is essentially completely busted. Grabbing the mmap write lock just means you block out pretty much everything interesting from concurrently happening.

My gut feeling says we need to figure out what's happening here, because this looks a bit too fundamental to me.

I agree. I haven’t looked into this issue for a couple of months but really need to understand what is going on.

I should have mentioned this in the cover letter: the goal of this series was to produce something for review that is stable and supports UMDs/user applications. It was not intended to be presented as a final solution. This issue certainly falls into the category of "needs to be understood and requires a proper fix."

Quoted 6 lines

One open question I have is whether the test case that triggers this issue is even defined behavior. The test creates concurrent access between the GPU and CPU to the same memory address, resulting in GPU and CPU faults racing against each other. It’s possible that this is undefined behavior, so data corruption might be acceptable—i.e., the kernel can’t crash, but incorrect results might be permissible.

Yes this is supposed to be defined, at least from an hmm pov. And core mm/ is ridiculous in how many races it allows, especially around concurrent fault handling.

It is ofc really slow if every fault results in a migration, but that's a matter of the application setting stupid memory migration hints for the gpu.

Quoted 22 lines

e.g. This is the only defined usage model:

alloc_memory();
start_compute_kernel();
sync_on_compute_kernel_completion();
read_memory();

Hopefully, in the next week or so, I'll be heavily engaging with the UMD teams. Development can then start, and applications will be running soon after. This will allow us to address issues like this, collect data on memory usage, and verify some of the assumptions I've made, such as optimizing for 2M+ allocations.

Quoted 4 lines

I think I have at least a high level understanding what's going on here, Felix and especially Philip should know more of the details.

I meant to reach out to AMD for issues like this. So, Felix ([email protected]) and Philip ([email protected]) would be good contacts?

Quoted 7 lines
Quoted 3 lines

In general grabbing the mm_lock to protect PTEs from changing is completely nonsense. The mm_lock is to protect the VMAs and *not* the PTEs!

Thanks for the hint. I believe that in the AMD implementation, I noticed some additional locks for migration, which might be how you mitigated this issue.

Yeah, so in general hold mmap_reading is indeed pure magic thinking for preventing pte changes, like Christian points out. It doesn't stop invalidates, and with the per vma locking it also doesn't stop new valid ptes from being inserted at least for anon memory.

Except migration pte entries that point at vram pages are special, and are _only_ resolved while holding mmap_read. Which means holding mmap_write for the case of looking up our own vram pages with hmm_range_fault actually prevents issues. And so this duct-tape of holding mmap_write very much looks like a working hack to plug any races against concurrently ongoing migrations to system memory due to cpu faults.

An even more fun corner case is multiple concurrent cpu faults on the same vram page. fork gets you that, or maybe a bit more reasonable mremap with MREMAP_DONTUNMAP | MREMAP_MAYMOVE. I think just hammer the same va with multiple threads along isn't enough, it's better to have a private va for each thread pointing at the same anon memory page, so that you can get more parallel faults due to finely grained pte locking.

Would be a good testcase to add, if you don't have it yet.

Quoted 3 lines

I must say it is a bit unfortunate that the HMM locking documentation doesn’t mention this. I believe the documentation needs additional information, which I can add once we finalize the solution.

Yeah, at least from my very cursory lock you don't have enough locking. I've written an in-depth reply to patch 23 with the high-level summary of my thoughts.

Cheers, Sima

Quoted 1,260 lines

Matt

Quoted 1,257 lines

Even with the write side of the mm_lock taken it is perfectly possible that PTE change. It's just less likely.

We run into multiple issues before we figured out this important distinction as well.

Christian.

Quoted 1,248 lines

-Sima

Quoted 1,245 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +821 -0
+			if (vram_pages)
+				mmap_write_lock(mm);
+			else
+				mmap_read_lock(mm);
+		}
+		err = hmm_range_fault(&hmm_range);
+		if (!ctx->mmap_locked) {
+			if (vram_pages)
+				mmap_write_unlock(mm);
+			else
+				mmap_read_unlock(mm);
+		}
+
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
+			continue;
+		}
+		break;
+	}
+	if (!ctx->mmap_locked)
+		mmput(mm);
+	if (err)
+		goto err_free;
+
+	pages = (struct page **)pfns;
+
+	if (ctx->prefault) {
+		range->pages = pages;
+		goto set_seqno;
+	}
+
+map_pages:
+	if (is_device_private_page(hmm_pfn_to_page(pfns[0]))) {
+		WARN_ON_ONCE(!range->vram_allocation);
+
+		for (i = 0; i < npages; ++i) {
+			pages[i] = hmm_pfn_to_page(pfns[i]);
+
+			if (WARN_ON_ONCE(!is_device_private_page(pages[i]))) {
+				err = -EOPNOTSUPP;
+				goto err_free;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->flags.has_vram_pages = true;
+		range->pages = pages;
+		if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm, range);
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	} else {
+		dma_addr_t *dma_addr = (dma_addr_t *)pfns;
+
+		for_each_dma_page(i, j, npages, order) {
+			if (WARN_ON_ONCE(i && order !=
+					 hmm_pfn_to_map_order(pfns[i]))) {
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+			order = hmm_pfn_to_map_order(pfns[i]);
+
+			pages[j] = hmm_pfn_to_page(pfns[i]);
+			if (WARN_ON_ONCE(is_zone_device_page(pages[j]))) {
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+
+			set_page_dirty_lock(pages[j]);
+			mark_page_accessed(pages[j]);
+
+			dma_addr[j] = dma_map_page(gpusvm->drm->dev,
+						   pages[j], 0,
+						   PAGE_SIZE << order,
+						   DMA_BIDIRECTIONAL);
+			if (dma_mapping_error(gpusvm->drm->dev, dma_addr[j])) {
+				err = -EFAULT;
+				npages = i;
+				goto err_unmap;
+			}
+		}
+
+		/* Huge pages, reduce memory footprint */
+		if (order) {
+			dma_addr = kmalloc_array(j, sizeof(*dma_addr),
+						 GFP_KERNEL);
+			if (dma_addr) {
+				for (i = 0; i < j; ++i)
+					dma_addr[i] = (dma_addr_t)pfns[i];
+				kvfree(pfns);
+				kfree_mapping = true;
+			} else {
+				dma_addr = (dma_addr_t *)pfns;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->order = order;
+		range->flags.kfree_mapping = kfree_mapping;
+		range->flags.has_dma_mapping = true;
+		range->dma_addr = dma_addr;
+		range->vram_allocation = NULL;
+		if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm, range);
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	}
+
+	if (err == -EAGAIN)
+		goto retry;
+set_seqno:
+	range->notifier_seq = hmm_range.notifier_seq;
+
+	return 0;
+
+err_unmap:
+	for_each_dma_page(i, j, npages, order)
+		dma_unmap_page(gpusvm->drm->dev,
+			       (dma_addr_t)pfns[j],
+			       PAGE_SIZE << order, DMA_BIDIRECTIONAL);
+err_free:
+	if (alloc_pfns)
+		kvfree(pfns);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_range_unmap_pages - Unmap pages associated with a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed that gpusvm->notifier_lock is held in write mode; if it
+ * is clear, it acquires gpusvm->notifier_lock in read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops->invalidate for IOMMU
+ * security model.
+ */
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx)
+{
+	if (ctx->in_notifier)
+		lockdep_assert_held_write(&gpusvm->notifier_lock);
+	else
+		drm_gpusvm_notifier_lock(gpusvm);
+
+	__drm_gpusvm_range_unmap_pages(gpusvm, range);
+
+	if (!ctx->in_notifier)
+		drm_gpusvm_notifier_unlock(gpusvm);
+}
+
+/**
+ * drm_gpusvm_migration_put_page - Put a migration page
+ * @page: Pointer to the page to put
+ *
+ * This function unlocks and puts a page.
+ */
+static void drm_gpusvm_migration_put_page(struct page *page)
+{
+	unlock_page(page);
+	put_page(page);
+}
+
+/**
+ * drm_gpusvm_migration_put_pages - Put migration pages
+ * @npages: Number of pages
+ * @migrate_pfn: Array of migrate page frame numbers
+ *
+ * This function puts an array of pages.
+ */
+static void drm_gpusvm_migration_put_pages(unsigned long npages,
+					   unsigned long *migrate_pfn)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!migrate_pfn[i])
+			continue;
+
+		drm_gpusvm_migration_put_page(migrate_pfn_to_page(migrate_pfn[i]));
+		migrate_pfn[i] = 0;
+	}
+}
+
+/**
+ * drm_gpusvm_get_vram_page - Get a reference to a VRAM page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU SVM zone
+ * device data and initializes it for zone device usage.
+ */
+static void drm_gpusvm_get_vram_page(struct page *page,
+				     struct drm_gpusvm_zdd *zdd)
+{
+	page->zone_device_data = drm_gpusvm_zdd_get(zdd);
+	zone_device_page_init(page);
+}
+
+/**
+ * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Return: 0 on success, -EFAULT if an error occurs during mapping.
+ */
+static int drm_gpusvm_migrate_map_pages(struct device *dev,
+					dma_addr_t *dma_addr,
+					long unsigned int *migrate_pfn,
+					unsigned long npages,
+					enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = migrate_pfn_to_page(migrate_pfn[i]);
+
+		if (!page)
+			continue;
+
+		if (WARN_ON_ONCE(is_zone_device_page(page)))
+			return -EFAULT;
+
+		dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE, dir);
+		if (dma_mapping_error(dev, dma_addr[i]))
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */
+static void drm_gpusvm_migrate_unmap_pages(struct device *dev,
+					   dma_addr_t *dma_addr,
+					   unsigned long npages,
+					   enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!dma_addr[i] || dma_mapping_error(dev, dma_addr[i]))
+			continue;
+
+		dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir);
+	}
+}
+
+/**
+ * drm_gpusvm_migrate_to_vram - Migrate GPU SVM range to VRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *                   failure of this function.
+ * @vram_allocation: Driver-private pointer to the VRAM allocation. The caller
+ *                   should hold a reference to the VRAM allocation, which
+ *                   should be dropped via ops->vram_allocation or upon the
+ *                   failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to VRAM. It performs the
+ * necessary setup and invokes the driver-specific operations for migration to
+ * VRAM. Upon successful return, @vram_allocation can safely reference @range
+ * until ops->vram_release is called which only upon successful return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct migrate_vma migrate = {
+		.start		= start,
+		.end		= end,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_SYSTEM,
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	unsigned long i, npages = npages_in_range(start, end);
+	struct vm_area_struct *vas;
+	struct drm_gpusvm_zdd *zdd = NULL;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int err;
+
+	if (!range->flags.migrate_vram)
+		return -EINVAL;
+
+	if (!gpusvm->ops->populate_vram_pfn || !gpusvm->ops->copy_to_vram ||
+	    !gpusvm->ops->copy_to_sram)
+		return -EOPNOTSUPP;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		mmap_write_lock(mm);
+	}
+
+	mmap_assert_locked(mm);
+
+	vas = vma_lookup(mm, start);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end > vas->vm_end || start < vas->vm_start) {
+		err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	if (!vma_is_anonymous(vas)) {
+		err = -EBUSY;
+		goto err_mmunlock;
+	}
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_mmunlock;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+	zdd = drm_gpusvm_zdd_alloc(range);
+	if (!zdd) {
+		err = -ENOMEM;
+		goto err_free;
+	}
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/*
+	 * FIXME: Below cases, !migrate.cpages and migrate.cpages != npages, not
+	 * always an error. Need to revisit possible cases and how to handle. We
+	 * could prefault on migrate.cpages != npages via hmm_range_fault.
+	 */
+
+	if (!migrate.cpages) {
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	if (migrate.cpages != npages) {
+		err = -EBUSY;
+		goto err_finalize;
+	}
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm, vram_allocation, npages,
+					     migrate.dst);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   migrate.src, npages, DMA_TO_DEVICE);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = pfn_to_page(migrate.dst[i]);
+
+		pages[i] = page;
+		migrate.dst[i] = migrate_pfn(migrate.dst[i]);
+		drm_gpusvm_get_vram_page(page, zdd);
+	}
+
+	err = gpusvm->ops->copy_to_vram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+	/* Upon success bind vram allocation to range and zdd */
+	range->vram_allocation = vram_allocation;
+	WRITE_ONCE(zdd->vram_allocation, vram_allocation);	/* Owns ref */
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_TO_DEVICE);
+err_free:
+	if (zdd)
+		drm_gpusvm_zdd_put(zdd);
+	kvfree(buf);
+err_mmunlock:
+	if (!ctx->mmap_locked) {
+		mmap_write_unlock(mm);
+		mmput(mm);
+	}
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_populate_sram_pfn - Populate SRAM PFNs for a VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @npages: Number of pages to populate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the SRAM migrate page frame numbers (PFNs) for the
+ * specified VM area structure. It allocates and locks pages in the VM area for
+ * SRAM usage. If vas is non-NULL use alloc_page_vma for allocation, if NULL use
+ * alloc_page for allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_sram_pfn(struct vm_area_struct *vas,
+						unsigned long npages,
+						unsigned long *src_mpfn,
+						unsigned long *mpfn, u64 addr)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i, addr += PAGE_SIZE) {
+		struct page *page;
+
+		if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE))
+			continue;
+
+		if (vas)
+			page = alloc_page_vma(GFP_HIGHUSER, vas, addr);
+		else
+			page = alloc_page(GFP_HIGHUSER);
+
+		if (!page)
+			return -ENOMEM;
+
+		lock_page(page);
+		mpfn[i] = migrate_pfn(page_to_pfn(page));
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_evict_to_sram - Evict GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * Similar to __drm_gpusvm_migrate_to_sram but does not require mmap lock and
+ * migration done via migrate_device_* functions. Fallback path as it is
+ * preferred to issue migrations with mmap lock.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_evict_to_sram(struct drm_gpusvm *gpusvm,
+				    struct drm_gpusvm_range *range)
+{
+	unsigned long npages;
+	struct page **pages;
+	unsigned long *src, *dst;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	npages = npages_in_range(range->va.start, range->va.end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	src = buf;
+	dst = buf + (sizeof(*src) * npages);
+	dma_addr = buf + (2 * sizeof(*src) * npages);
+	pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) * npages;
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm, range->vram_allocation,
+					     npages, src);
+	if (err)
+		goto err_free;
+
+	err = migrate_device_vma_range(gpusvm->mm,
+				       gpusvm->device_private_page_owner, src,
+				       npages, range->va.start);
+	if (err)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(NULL, npages, src, dst, 0);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   dst, npages, DMA_BIDIRECTIONAL);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, dst);
+	migrate_device_pages(src, dst, npages);
+	migrate_device_finalize(src, dst, npages);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+
+	return err;
+}
+
+/**
+ * __drm_gpusvm_migrate_to_sram - Migrate GPU SVM range to SRAM (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @vas: Pointer to the VM area structure
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @start: Start address of the migration range
+ * @end: End address of the migration range
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to SRAM. It sets up the migration, populates + dma maps SRAM PFNs, and
+ * invokes the driver-specific operations for migration to SRAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int __drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+					struct vm_area_struct *vas,
+					struct page *page,
+					u64 start, u64 end)
+{
+	struct migrate_vma migrate = {
+		.vma		= vas,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_DEVICE_PRIVATE,
+		.fault_page	= page,
+	};
+	unsigned long npages;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	mmap_assert_locked(gpusvm->mm);
+
+	/* Corner where VMA area struct has been partially unmapped */
+	if (start < vas->vm_start)
+		start = vas->vm_start;
+	if (end > vas->vm_end)
+		end = vas->vm_end;
+
+	migrate.start = start;
+	migrate.end = end;
+	npages = npages_in_range(start, end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/* Raced with another CPU fault, nothing to do */
+	if (!migrate.cpages)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(vas, npages,
+						   migrate.src, migrate.dst,
+						   start);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   migrate.dst, npages,
+					   DMA_BIDIRECTIONAL);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(migrate.src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+	mmap_assert_locked(gpusvm->mm);
+
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_to_sram - Migrate (evict) GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function initiates the migration of the specified GPU SVM range to
+ * SRAM. It performs necessary checks and invokes the internal migration
+ * function for actual migration.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	int err;
+	bool retry = false;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		if (ctx->trylock_mmap) {
+			if (!mmap_read_trylock(mm))  {
+				err = drm_gpusvm_evict_to_sram(gpusvm, range);
+				goto err_mmput;
+			}
+		} else {
+			mmap_read_lock(mm);
+		}
+	}
+
+	mmap_assert_locked(mm);
+
+	/*
+	 * Loop required to find all VMA area structs for the corner case when
+	 * VRAM backing has been partially unmapped from MM's address space.
+	 */
+again:
+	vas = find_vma(mm, start);
+	if (!vas) {
+		if (!retry)
+			err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end <= vas->vm_start || start >= vas->vm_end) {
+		if (!retry)
+			err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	err = __drm_gpusvm_migrate_to_sram(gpusvm, vas, NULL, start, end);
+	if (err)
+		goto err_mmunlock;
+
+	if (vas->vm_end < end) {
+		retry = true;
+		start = vas->vm_end;
+		goto again;
+	}
+
+	if (!ctx->mmap_locked) {
+		mmap_read_unlock(mm);
+		/*
+		 * Using mmput_async as this function can be called while
+		 * holding a dma-resv lock, and a final put can grab the mmap
+		 * lock, causing a lock inversion.
+		 */
+		mmput_async(mm);
+	}
+
+	return 0;
+
+err_mmunlock:
+	if (!ctx->mmap_locked)
+		mmap_read_unlock(mm);
+err_mmput:
+	if (!ctx->mmap_locked)
+		mmput_async(mm);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_page_free - Put GPU SVM zone device data associated with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device data
+ * associated with a page when it is being released.
+ */
+static void drm_gpusvm_page_free(struct page *page)
+{
+	drm_gpusvm_zdd_put(page->zone_device_data);
+}
+
+/**
+ * drm_gpusvm_migrate_to_ram - Migrate GPU SVM range to RAM (page fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ * VM_FAULT_SIGBUS on failure, 0 on success.
+ */
+static vm_fault_t drm_gpusvm_migrate_to_ram(struct vm_fault *vmf)
+{
+	struct drm_gpusvm_zdd *zdd = vmf->page->zone_device_data;
+	int err;
+
+	err = __drm_gpusvm_migrate_to_sram(zdd->range->gpusvm,
+					   vmf->vma, vmf->page,
+					   zdd->range->va.start,
+					   zdd->range->va.end);
+
+	return err ? VM_FAULT_SIGBUS : 0;
+}
+
+/**
+ * drm_gpusvm_pagemap_ops - Device page map operations for GPU SVM
+ */
+static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = {
+	.page_free = drm_gpusvm_page_free,
+	.migrate_to_ram = drm_gpusvm_migrate_to_ram,
+};
+
+/**
+ * drm_gpusvm_pagemap_ops_get - Retrieve GPU SVM device page map operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void)
+{
+	return &drm_gpusvm_pagemap_ops;
+}
+
+/**
+ * drm_gpusvm_has_mapping - Check if GPU SVM has mapping for the given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end)
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	drm_gpusvm_for_each_notifier(notifier, gpusvm, start, end) {
+		struct drm_gpusvm_range *range = NULL;
+
+		drm_gpusvm_for_each_range(range, notifier, start, end)
+			return true;
+	}
+
+	return false;
+}
drivers/gpu/drm/xe/drm_gpusvm.h new file +415 -0
new file mode 100644
index 000000000000..0ea70f8534a8
@@ -0,0 +1,415 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef __DRM_GPUSVM_H__
+#define __DRM_GPUSVM_H__
+
+#include <linux/kref.h>
+#include <linux/mmu_notifier.h>
+#include <linux/workqueue.h>
+
+struct dev_pagemap_ops;
+struct drm_device;
+struct drm_gpusvm;
+struct drm_gpusvm_notifier;
+struct drm_gpusvm_ops;
+struct drm_gpusvm_range;
+
+/**
+ * struct drm_gpusvm_ops - Operations structure for GPU SVM
+ *
+ * This structure defines the operations for GPU Shared Virtual Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * perform operations such as migration between VRAM and system RAM.
+ */
+struct drm_gpusvm_ops {
+	/**
+	 * @notifier_alloc: Allocate a GPU SVM notifier (optional)
+	 *
+	 * This function shall allocate a GPU SVM notifier.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM notifier on success, NULL on failure.
+	 */
+	struct drm_gpusvm_notifier *(*notifier_alloc)(void);
+
+	/**
+	 * @notifier_free: Free a GPU SVM notifier (optional)
+	 * @notifier: Pointer to the GPU SVM notifier to be freed
+	 *
+	 * This function shall free a GPU SVM notifier.
+	 */
+	void (*notifier_free)(struct drm_gpusvm_notifier *notifier);
+
+	/**
+	 * @range_alloc: Allocate a GPU SVM range (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 *
+	 * This function shall allocate a GPU SVM range.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM range on success, NULL on failure.
+	 */
+	struct drm_gpusvm_range *(*range_alloc)(struct drm_gpusvm *gpusvm);
+
+	/**
+	 * @range_free: Free a GPU SVM range (optional)
+	 * @range: Pointer to the GPU SVM range to be freed
+	 *
+	 * This function shall free a GPU SVM range.
+	 */
+	void (*range_free)(struct drm_gpusvm_range *range);
+
+	/**
+	 * @vram_release: Release VRAM allocation (optional)
+	 * @vram_allocation: Driver-private pointer to the VRAM allocation
+	 *
+	 * This function shall release VRAM allocation and expects to drop a
+	 * reference to VRAM allocation.
+	 */
+	void (*vram_release)(void *vram_allocation);
+
+	/**
+	 * @populate_vram_pfn: Populate VRAM PFN (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @vram_allocation: Driver-private pointer to the VRAM allocation
+	 * @npages: Number of pages to populate
+	 * @pfn: Array of page frame numbers to populate
+	 *
+	 * This function shall populate VRAM page frame numbers (PFN).
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*populate_vram_pfn)(struct drm_gpusvm *gpusvm,
+				 void *vram_allocation,
+				 unsigned long npages,
+				 unsigned long *pfn);
+
+	/**
+	 * @copy_to_vram: Copy to VRAM (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (destination)
+	 * @dma_addr: Pointer to array of DMA addresses (source)
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to VRAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_vram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @copy_to_sram: Copy to system RAM (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (source)
+	 * @dma_addr: Pointer to array of DMA addresses (destination)
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to system RAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_sram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @invalidate: Invalidate GPU SVM notifier (required)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @notifier: Pointer to the GPU SVM notifier
+	 * @mmu_range: Pointer to the mmu_notifier_range structure
+	 *
+	 * This function shall invalidate the GPU page tables. It can safely
+	 * walk the notifier range RB tree/list in this function. Called while
+	 * holding the notifier lock.
+	 */
+	void (*invalidate)(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_notifier *notifier,
+			   const struct mmu_notifier_range *mmu_range);
+};
+
+/**
+ * struct drm_gpusvm_notifier - Structure representing a GPU SVM notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @interval: Interval for the notifier
+ * @rb: Red-black tree node for the parent GPU SVM structure notifier tree
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ *              interval tree. This is useful to keep iterating ranges while
+ *              doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier has been
+ *                 removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */
+struct drm_gpusvm_notifier {
+	struct drm_gpusvm *gpusvm;
+	struct mmu_interval_notifier notifier;
+	struct {
+		u64 start;
+		u64 end;
+	} interval;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct rb_root_cached root;
+	struct list_head range_list;
+	struct {
+		u32 removed : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm_range - Structure representing a GPU SVM range
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier
+ * @refcount: Reference count for the range
+ * @rb: Red-black tree node for the parent GPU SVM notifier structure range tree
+ * @va: Virtual address range
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @pages: Pointer to the array of pages (if backing store is in VRAM)
+ * @dma_addr: DMA address array (if backing store is SRAM and DMA mapped)
+ * @vram_allocation: Driver-private pointer to the VRAM allocation
+ * @order: Order of dma mapping. i.e. PAGE_SIZE << order is mapping size
+ * @flags.migrate_vram: Flag indicating whether the range can be migrated to VRAM
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_vram_pages: Flag indicating if the range has vram pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA mapping
+ * @flags.kfree_mapping: Flag indicating @dma_addr is a compact allocation based
+ *                       on @order which releases via kfree
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */
+struct drm_gpusvm_range {
+	struct drm_gpusvm *gpusvm;
+	struct drm_gpusvm_notifier *notifier;
+	struct kref refcount;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct {
+		u64 start;
+		u64 end;
+	} va;
+	unsigned long notifier_seq;
+	union {
+		struct page **pages;
+		dma_addr_t *dma_addr;
+	};
+	void *vram_allocation;
+	u16 order;
+	struct {
+		/* All flags below must be set upon creation */
+		u16 migrate_vram : 1;
+		/* All flags below must be set / cleared under notifier lock */
+		u16 unmapped : 1;
+		u16 partial_unmap : 1;
+		u16 has_vram_pages : 1;
+		u16 has_dma_mapping : 1;
+		u16 kfree_mapping : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm - GPU SVM structure
+ *
+ * @name: Name of the GPU SVM
+ * @drm: Pointer to the DRM device structure
+ * @mm: Pointer to the mm_struct for the address space
+ * @device_private_page_owner: Device private pages owner
+ * @mm_start: Start address of GPU SVM
+ * @mm_range: Range of the GPU SVM
+ * @notifier_size: Size of individual notifiers
+ * @ops: Pointer to the operations structure for GPU SVM
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range allocation.
+ *               Entries should be powers of 2 in descending order.
+ * @num_chunks: Number of chunks
+ * @notifier_lock: Read-write semaphore for protecting notifier operations
+ * @zdd_wq: Workqueue for deferred work on zdd destruction
+ * @root: Cached root node of the Red-Black tree containing GPU SVM notifiers
+ * @notifier_list: list head containing of notifiers in the same order they
+ *                 appear in interval tree. This is useful to keep iterating
+ *                 notifiers while doing modifications to RB tree.
+ *
+ * This structure represents a GPU SVM (Shared Virtual Memory) used for tracking
+ * memory ranges mapped in a DRM (Direct Rendering Manager) device.
+ *
+ * No reference counting is provided, as this is expected to be embedded in the
+ * driver VM structure along with the struct drm_gpuvm, which handles reference
+ * counting.
+ */
+struct drm_gpusvm {
+	const char *name;
+	struct drm_device *drm;
+	struct mm_struct *mm;
+	void *device_private_page_owner;
+	u64 mm_start;
+	u64 mm_range;
+	u64 notifier_size;
+	const struct drm_gpusvm_ops *ops;
+	const u64 *chunk_sizes;
+	int num_chunks;
+	struct rw_semaphore notifier_lock;
+	struct workqueue_struct *zdd_wq;
+	struct rb_root_cached root;
+	struct list_head notifier_list;
+};
+
+/**
+ * struct drm_gpusvm_ctx - DRM GPU SVM context
+ *
+ * @mmap_locked: mmap lock is locked
+ * @trylock_mmap: trylock mmap lock, used to avoid locking inversions
+ *                (e.g.dma-revs -> mmap lock)
+ * @in_notifier: entering from a MMU notifier
+ * @read_only: operating on read-only memory
+ * @vram_possible: possible to use VRAM
+ * @prefault: prefault pages
+ *
+ * Context that is DRM GPUSVM is operating in (i.e. user arguments).
+ */
+struct drm_gpusvm_ctx {
+	u32 mmap_locked :1;
+	u32 trylock_mmap :1;
+	u32 in_notifier :1;
+	u32 read_only :1;
+	u32 vram_possible :1;
+	u32 prefault :1;
+};
+
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void *device_private_page_owner,
+		    u64 mm_start, u64 mm_range, u64 notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const u64 *chunk_sizes, int num_chunks);
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm);
+void drm_gpusvm_free(struct drm_gpusvm *gpusvm);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, u64 fault_addr,
+				u64 gpuva_start, u64 gpuva_end,
+				const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range);
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range);
+
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range);
+
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx);
+
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx);
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void);
+
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, u64 start, u64 end);
+
+/**
+ * drm_gpusvm_notifier_lock - Lock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, take lock
+ */
+#define drm_gpusvm_notifier_lock(gpusvm__)	\
+	down_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * drm_gpusvm_notifier_unlock - Unlock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, drop lock
+ */
+#define drm_gpusvm_notifier_unlock(gpusvm__)	\
+	up_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * __drm_gpusvm_range_next - Get the next GPU SVM range in the list
+ * @range: a pointer to the current GPU SVM range
+ *
+ * Return: A pointer to the next drm_gpusvm_range if available, or NULL if the
+ *         current range is the last one or if the input range is NULL.
+ */
+static inline struct drm_gpusvm_range *
+__drm_gpusvm_range_next(struct drm_gpusvm_range *range)
+{
+	if (range && !list_is_last(&range->rb.entry,
+				   &range->notifier->range_list))
+		return list_next_entry(range, rb.entry);
+
+	return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_range - Iterate over GPU SVM ranges in a notifier
+ * @range__: Iterator variable for the ranges. If set, it indicates the start of
+ *	     the iterator. If NULL, call drm_gpusvm_range_find() to get the range.
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier. It is safe
+ * to use while holding the driver SVM lock or the notifier lock.
+ */
+#define drm_gpusvm_for_each_range(range__, notifier__, start__, end__)	\
+	for ((range__) = (range__) ?:					\
+	     drm_gpusvm_range_find((notifier__), (start__), (end__));	\
+	     (range__) && (range__->va.start < (end__));		\
+	     (range__) = __drm_gpusvm_range_next(range__))
+
+/**
+ * drm_gpusvm_range_set_unmapped - Mark a GPU SVM range as unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets the partial_unmap flag
+ * if the range partially falls within the provided MMU notifier range.
+ */
+static inline void
+drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,
+			      const struct mmu_notifier_range *mmu_range)
+{
+	lockdep_assert_held_write(&range->gpusvm->notifier_lock);
+
+	range->flags.unmapped = true;
+	if (range->va.start < mmu_range->start ||
+	    range->va.end > mmu_range->end)
+		range->flags.partial_unmap = true;
+}
+
+#endif /* __DRM_GPUSVM_H__ */
Signature · 1 line
2.34.1
Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#13 lore raw

Re: [RFC PATCH 05/28] drm/gpusvm: Add support for GPU Shared Virtual Memory

Quoted 76 lines · Daniel Vetter
Quoted 34 lines · Matthew Brost
Quoted 13 lines · Christian König

Am 28.08.24 um 16:31 schrieb Daniel Vetter:

Quoted 12 lines · Matthew Brost

drivers/gpu/drm/xe/drm_gpusvm.c line 1353

Quoted 11 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+		if (!ctx->mmap_locked) {
+			/*
+			 * XXX: HMM locking document indicates only a read-lock
+			 * is required but there apears to be a window between
+			 * the MMU_NOTIFY_MIGRATE event triggered in a CPU fault
+			 * via migrate_vma_setup and the pages actually moving
+			 * in migrate_vma_finalize in which this code can grab
+			 * garbage pages. Grabbing the write-lock if the range
+			 * is attached to vram appears to protect against this
+			 * race.
+			 */

Thanks the comments, replying to both of you inline.

Quoted 7 lines
Quoted 7 lines

This one is really scary, since it means the entire migrate pte trickery is essentially completely busted. Grabbing the mmap write lock just means you block out pretty much everything interesting from concurrently happening.

My gut feeling says we need to figure out what's happening here, because this looks a bit too fundamental to me.

I agree. I haven’t looked into this issue for a couple of months but really need to understand what is going on.

I should have mentioned this in the cover letter: the goal of this series was to produce something for review that is stable and supports UMDs/user applications. It was not intended to be presented as a final solution. This issue certainly falls into the category of "needs to be understood and requires a proper fix."

Quoted 6 lines

One open question I have is whether the test case that triggers this issue is even defined behavior. The test creates concurrent access between the GPU and CPU to the same memory address, resulting in GPU and CPU faults racing against each other. It’s possible that this is undefined behavior, so data corruption might be acceptable—i.e., the kernel can’t crash, but incorrect results might be permissible.

Yes this is supposed to be defined, at least from an hmm pov. And core mm/ is ridiculous in how many races it allows, especially around concurrent fault handling.

It is ofc really slow if every fault results in a migration, but that's a matter of the application setting stupid memory migration hints for the gpu.

Quoted 26 lines

e.g. This is the only defined usage model:

alloc_memory();
start_compute_kernel();
sync_on_compute_kernel_completion();
read_memory();

Hopefully, in the next week or so, I'll be heavily engaging with the UMD teams. Development can then start, and applications will be running soon after. This will allow us to address issues like this, collect data on memory usage, and verify some of the assumptions I've made, such as optimizing for 2M+ allocations.

Quoted 4 lines

I think I have at least a high level understanding what's going on here, Felix and especially Philip should know more of the details.

I meant to reach out to AMD for issues like this. So, Felix ([email protected]) and Philip ([email protected]) would be good contacts?

Quoted 3 lines

In general grabbing the mm_lock to protect PTEs from changing is completely nonsense. The mm_lock is to protect the VMAs and *not* the PTEs!

Quoted 7 lines
Quoted 3 lines

Thanks for the hint. I believe that in the AMD implementation, I noticed some additional locks for migration, which might be how you mitigated this issue.

Yeah, so in general hold mmap_reading is indeed pure magic thinking for preventing pte changes, like Christian points out. It doesn't stop invalidates, and with the per vma locking it also doesn't stop new valid

Invalidations happening to parallel to migrations, get pages, or bindings should be fine. The notifier lock usage should make all of this safe.

Quoted 9 lines

ptes from being inserted at least for anon memory.

Except migration pte entries that point at vram pages are special, and are _only_ resolved while holding mmap_read. Which means holding mmap_write for the case of looking up our own vram pages with hmm_range_fault actually prevents issues. And so this duct-tape of holding mmap_write very much looks like a working hack to plug any races against concurrently ongoing migrations to system memory due to cpu faults.

Agree holding mmap_write is a hack. Looking at AMD 'To serialize concurrent migrations or validations of the same range, the prange->migrate_mutex must be held.', seemly I could drop mmap write lock abuse and use something like this here. The would like be an inner lock of the mmap lock.

Does this seem like a reasonable thing to explore?

Quoted 2 lines

An even more fun corner case is multiple concurrent cpu faults on the same vram page. fork gets you that, or maybe a bit more reasonable mremap with

My understanding is memory shared between processes cannot migrated due to current limitations migrate layer.

e.g. mmap called with MAP_SHARED is not eligible for migration.

Unsure what the behavior is fork() is called on a process with memory in VRAM and the child tries to access it. Maybe fork() is different than MAP_SHARED where as parent / child processes can share memory in VRAM?

Also really I'm unsure what would happen if user space calls fork() and has an Xe VM open and tries to use it too. Before commenting more on this, I need play around with test cases like this educate myself.

My current test doesn't use mremap, agree that would be to good add. Again before commenting more here, let add more test cases to educate myself.

Quoted 2 lines

MREMAP_DONTUNMAP | MREMAP_MAYMOVE. I think just hammer the same va with multiple threads along isn't enough, it's better to have a private va for

I do have test cases where multiple CPU faults from threads hammer the same memory. Found some bugs in my initial code but as far as I can tell multiple CPU faults in parallel occur in my testing and do work.

Quoted 1 line

each thread pointing at the same anon memory page, so that you can get

You are losing me here - 'private va for each thread pointing at the same anon memory page'. This is a fork() case where the parent allocates memory and then all children try to read in parallel?

Quoted 4 lines

more parallel faults due to finely grained pte locking.

Would be a good testcase to add, if you don't have it yet.

See above, agree these are good test cases which I haven't considered and will expand my suite to include these. Thanks for the tip - IMO testing is as important or even more important than the KMD design and need to ensure I have all possible uses covered.

Quoted 8 lines
Quoted 3 lines

I must say it is a bit unfortunate that the HMM locking documentation doesn’t mention this. I believe the documentation needs additional information, which I can add once we finalize the solution.

Yeah, at least from my very cursory lock you don't have enough locking. I've written an in-depth reply to patch 23 with the high-level summary of my thoughts.

Will look and reply there.

Matt

Quoted 1,267 lines

Cheers, Sima

Quoted 1,260 lines

Matt

Quoted 1,257 lines

Even with the write side of the mm_lock taken it is perfectly possible that PTE change. It's just less likely.

We run into multiple issues before we figured out this important distinction as well.

Christian.

Quoted 1,248 lines

-Sima

Quoted 1,245 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +821 -0
+			if (vram_pages)
+				mmap_write_lock(mm);
+			else
+				mmap_read_lock(mm);
+		}
+		err = hmm_range_fault(&hmm_range);
+		if (!ctx->mmap_locked) {
+			if (vram_pages)
+				mmap_write_unlock(mm);
+			else
+				mmap_read_unlock(mm);
+		}
+
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
+			continue;
+		}
+		break;
+	}
+	if (!ctx->mmap_locked)
+		mmput(mm);
+	if (err)
+		goto err_free;
+
+	pages = (struct page **)pfns;
+
+	if (ctx->prefault) {
+		range->pages = pages;
+		goto set_seqno;
+	}
+
+map_pages:
+	if (is_device_private_page(hmm_pfn_to_page(pfns[0]))) {
+		WARN_ON_ONCE(!range->vram_allocation);
+
+		for (i = 0; i < npages; ++i) {
+			pages[i] = hmm_pfn_to_page(pfns[i]);
+
+			if (WARN_ON_ONCE(!is_device_private_page(pages[i]))) {
+				err = -EOPNOTSUPP;
+				goto err_free;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->flags.has_vram_pages = true;
+		range->pages = pages;
+		if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm, range);
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	} else {
+		dma_addr_t *dma_addr = (dma_addr_t *)pfns;
+
+		for_each_dma_page(i, j, npages, order) {
+			if (WARN_ON_ONCE(i && order !=
+					 hmm_pfn_to_map_order(pfns[i]))) {
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+			order = hmm_pfn_to_map_order(pfns[i]);
+
+			pages[j] = hmm_pfn_to_page(pfns[i]);
+			if (WARN_ON_ONCE(is_zone_device_page(pages[j]))) {
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+
+			set_page_dirty_lock(pages[j]);
+			mark_page_accessed(pages[j]);
+
+			dma_addr[j] = dma_map_page(gpusvm->drm->dev,
+						   pages[j], 0,
+						   PAGE_SIZE << order,
+						   DMA_BIDIRECTIONAL);
+			if (dma_mapping_error(gpusvm->drm->dev, dma_addr[j])) {
+				err = -EFAULT;
+				npages = i;
+				goto err_unmap;
+			}
+		}
+
+		/* Huge pages, reduce memory footprint */
+		if (order) {
+			dma_addr = kmalloc_array(j, sizeof(*dma_addr),
+						 GFP_KERNEL);
+			if (dma_addr) {
+				for (i = 0; i < j; ++i)
+					dma_addr[i] = (dma_addr_t)pfns[i];
+				kvfree(pfns);
+				kfree_mapping = true;
+			} else {
+				dma_addr = (dma_addr_t *)pfns;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->order = order;
+		range->flags.kfree_mapping = kfree_mapping;
+		range->flags.has_dma_mapping = true;
+		range->dma_addr = dma_addr;
+		range->vram_allocation = NULL;
+		if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm, range);
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	}
+
+	if (err == -EAGAIN)
+		goto retry;
+set_seqno:
+	range->notifier_seq = hmm_range.notifier_seq;
+
+	return 0;
+
+err_unmap:
+	for_each_dma_page(i, j, npages, order)
+		dma_unmap_page(gpusvm->drm->dev,
+			       (dma_addr_t)pfns[j],
+			       PAGE_SIZE << order, DMA_BIDIRECTIONAL);
+err_free:
+	if (alloc_pfns)
+		kvfree(pfns);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_range_unmap_pages - Unmap pages associated with a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed that gpusvm->notifier_lock is held in write mode; if it
+ * is clear, it acquires gpusvm->notifier_lock in read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops->invalidate for IOMMU
+ * security model.
+ */
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx)
+{
+	if (ctx->in_notifier)
+		lockdep_assert_held_write(&gpusvm->notifier_lock);
+	else
+		drm_gpusvm_notifier_lock(gpusvm);
+
+	__drm_gpusvm_range_unmap_pages(gpusvm, range);
+
+	if (!ctx->in_notifier)
+		drm_gpusvm_notifier_unlock(gpusvm);
+}
+
+/**
+ * drm_gpusvm_migration_put_page - Put a migration page
+ * @page: Pointer to the page to put
+ *
+ * This function unlocks and puts a page.
+ */
+static void drm_gpusvm_migration_put_page(struct page *page)
+{
+	unlock_page(page);
+	put_page(page);
+}
+
+/**
+ * drm_gpusvm_migration_put_pages - Put migration pages
+ * @npages: Number of pages
+ * @migrate_pfn: Array of migrate page frame numbers
+ *
+ * This function puts an array of pages.
+ */
+static void drm_gpusvm_migration_put_pages(unsigned long npages,
+					   unsigned long *migrate_pfn)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!migrate_pfn[i])
+			continue;
+
+		drm_gpusvm_migration_put_page(migrate_pfn_to_page(migrate_pfn[i]));
+		migrate_pfn[i] = 0;
+	}
+}
+
+/**
+ * drm_gpusvm_get_vram_page - Get a reference to a VRAM page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU SVM zone
+ * device data and initializes it for zone device usage.
+ */
+static void drm_gpusvm_get_vram_page(struct page *page,
+				     struct drm_gpusvm_zdd *zdd)
+{
+	page->zone_device_data = drm_gpusvm_zdd_get(zdd);
+	zone_device_page_init(page);
+}
+
+/**
+ * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Return: 0 on success, -EFAULT if an error occurs during mapping.
+ */
+static int drm_gpusvm_migrate_map_pages(struct device *dev,
+					dma_addr_t *dma_addr,
+					long unsigned int *migrate_pfn,
+					unsigned long npages,
+					enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = migrate_pfn_to_page(migrate_pfn[i]);
+
+		if (!page)
+			continue;
+
+		if (WARN_ON_ONCE(is_zone_device_page(page)))
+			return -EFAULT;
+
+		dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE, dir);
+		if (dma_mapping_error(dev, dma_addr[i]))
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */
+static void drm_gpusvm_migrate_unmap_pages(struct device *dev,
+					   dma_addr_t *dma_addr,
+					   unsigned long npages,
+					   enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!dma_addr[i] || dma_mapping_error(dev, dma_addr[i]))
+			continue;
+
+		dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir);
+	}
+}
+
+/**
+ * drm_gpusvm_migrate_to_vram - Migrate GPU SVM range to VRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *                   failure of this function.
+ * @vram_allocation: Driver-private pointer to the VRAM allocation. The caller
+ *                   should hold a reference to the VRAM allocation, which
+ *                   should be dropped via ops->vram_allocation or upon the
+ *                   failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to VRAM. It performs the
+ * necessary setup and invokes the driver-specific operations for migration to
+ * VRAM. Upon successful return, @vram_allocation can safely reference @range
+ * until ops->vram_release is called which only upon successful return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct migrate_vma migrate = {
+		.start		= start,
+		.end		= end,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_SYSTEM,
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	unsigned long i, npages = npages_in_range(start, end);
+	struct vm_area_struct *vas;
+	struct drm_gpusvm_zdd *zdd = NULL;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int err;
+
+	if (!range->flags.migrate_vram)
+		return -EINVAL;
+
+	if (!gpusvm->ops->populate_vram_pfn || !gpusvm->ops->copy_to_vram ||
+	    !gpusvm->ops->copy_to_sram)
+		return -EOPNOTSUPP;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		mmap_write_lock(mm);
+	}
+
+	mmap_assert_locked(mm);
+
+	vas = vma_lookup(mm, start);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end > vas->vm_end || start < vas->vm_start) {
+		err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	if (!vma_is_anonymous(vas)) {
+		err = -EBUSY;
+		goto err_mmunlock;
+	}
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_mmunlock;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+	zdd = drm_gpusvm_zdd_alloc(range);
+	if (!zdd) {
+		err = -ENOMEM;
+		goto err_free;
+	}
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/*
+	 * FIXME: Below cases, !migrate.cpages and migrate.cpages != npages, not
+	 * always an error. Need to revisit possible cases and how to handle. We
+	 * could prefault on migrate.cpages != npages via hmm_range_fault.
+	 */
+
+	if (!migrate.cpages) {
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	if (migrate.cpages != npages) {
+		err = -EBUSY;
+		goto err_finalize;
+	}
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm, vram_allocation, npages,
+					     migrate.dst);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   migrate.src, npages, DMA_TO_DEVICE);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = pfn_to_page(migrate.dst[i]);
+
+		pages[i] = page;
+		migrate.dst[i] = migrate_pfn(migrate.dst[i]);
+		drm_gpusvm_get_vram_page(page, zdd);
+	}
+
+	err = gpusvm->ops->copy_to_vram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+	/* Upon success bind vram allocation to range and zdd */
+	range->vram_allocation = vram_allocation;
+	WRITE_ONCE(zdd->vram_allocation, vram_allocation);	/* Owns ref */
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_TO_DEVICE);
+err_free:
+	if (zdd)
+		drm_gpusvm_zdd_put(zdd);
+	kvfree(buf);
+err_mmunlock:
+	if (!ctx->mmap_locked) {
+		mmap_write_unlock(mm);
+		mmput(mm);
+	}
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_populate_sram_pfn - Populate SRAM PFNs for a VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @npages: Number of pages to populate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the SRAM migrate page frame numbers (PFNs) for the
+ * specified VM area structure. It allocates and locks pages in the VM area for
+ * SRAM usage. If vas is non-NULL use alloc_page_vma for allocation, if NULL use
+ * alloc_page for allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_sram_pfn(struct vm_area_struct *vas,
+						unsigned long npages,
+						unsigned long *src_mpfn,
+						unsigned long *mpfn, u64 addr)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i, addr += PAGE_SIZE) {
+		struct page *page;
+
+		if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE))
+			continue;
+
+		if (vas)
+			page = alloc_page_vma(GFP_HIGHUSER, vas, addr);
+		else
+			page = alloc_page(GFP_HIGHUSER);
+
+		if (!page)
+			return -ENOMEM;
+
+		lock_page(page);
+		mpfn[i] = migrate_pfn(page_to_pfn(page));
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_evict_to_sram - Evict GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * Similar to __drm_gpusvm_migrate_to_sram but does not require mmap lock and
+ * migration done via migrate_device_* functions. Fallback path as it is
+ * preferred to issue migrations with mmap lock.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_evict_to_sram(struct drm_gpusvm *gpusvm,
+				    struct drm_gpusvm_range *range)
+{
+	unsigned long npages;
+	struct page **pages;
+	unsigned long *src, *dst;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	npages = npages_in_range(range->va.start, range->va.end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	src = buf;
+	dst = buf + (sizeof(*src) * npages);
+	dma_addr = buf + (2 * sizeof(*src) * npages);
+	pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) * npages;
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm, range->vram_allocation,
+					     npages, src);
+	if (err)
+		goto err_free;
+
+	err = migrate_device_vma_range(gpusvm->mm,
+				       gpusvm->device_private_page_owner, src,
+				       npages, range->va.start);
+	if (err)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(NULL, npages, src, dst, 0);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   dst, npages, DMA_BIDIRECTIONAL);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, dst);
+	migrate_device_pages(src, dst, npages);
+	migrate_device_finalize(src, dst, npages);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+
+	return err;
+}
+
+/**
+ * __drm_gpusvm_migrate_to_sram - Migrate GPU SVM range to SRAM (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @vas: Pointer to the VM area structure
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @start: Start address of the migration range
+ * @end: End address of the migration range
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to SRAM. It sets up the migration, populates + dma maps SRAM PFNs, and
+ * invokes the driver-specific operations for migration to SRAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int __drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+					struct vm_area_struct *vas,
+					struct page *page,
+					u64 start, u64 end)
+{
+	struct migrate_vma migrate = {
+		.vma		= vas,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_DEVICE_PRIVATE,
+		.fault_page	= page,
+	};
+	unsigned long npages;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	mmap_assert_locked(gpusvm->mm);
+
+	/* Corner where VMA area struct has been partially unmapped */
+	if (start < vas->vm_start)
+		start = vas->vm_start;
+	if (end > vas->vm_end)
+		end = vas->vm_end;
+
+	migrate.start = start;
+	migrate.end = end;
+	npages = npages_in_range(start, end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/* Raced with another CPU fault, nothing to do */
+	if (!migrate.cpages)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(vas, npages,
+						   migrate.src, migrate.dst,
+						   start);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   migrate.dst, npages,
+					   DMA_BIDIRECTIONAL);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(migrate.src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+	mmap_assert_locked(gpusvm->mm);
+
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_to_sram - Migrate (evict) GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function initiates the migration of the specified GPU SVM range to
+ * SRAM. It performs necessary checks and invokes the internal migration
+ * function for actual migration.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	int err;
+	bool retry = false;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		if (ctx->trylock_mmap) {
+			if (!mmap_read_trylock(mm))  {
+				err = drm_gpusvm_evict_to_sram(gpusvm, range);
+				goto err_mmput;
+			}
+		} else {
+			mmap_read_lock(mm);
+		}
+	}
+
+	mmap_assert_locked(mm);
+
+	/*
+	 * Loop required to find all VMA area structs for the corner case when
+	 * VRAM backing has been partially unmapped from MM's address space.
+	 */
+again:
+	vas = find_vma(mm, start);
+	if (!vas) {
+		if (!retry)
+			err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end <= vas->vm_start || start >= vas->vm_end) {
+		if (!retry)
+			err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	err = __drm_gpusvm_migrate_to_sram(gpusvm, vas, NULL, start, end);
+	if (err)
+		goto err_mmunlock;
+
+	if (vas->vm_end < end) {
+		retry = true;
+		start = vas->vm_end;
+		goto again;
+	}
+
+	if (!ctx->mmap_locked) {
+		mmap_read_unlock(mm);
+		/*
+		 * Using mmput_async as this function can be called while
+		 * holding a dma-resv lock, and a final put can grab the mmap
+		 * lock, causing a lock inversion.
+		 */
+		mmput_async(mm);
+	}
+
+	return 0;
+
+err_mmunlock:
+	if (!ctx->mmap_locked)
+		mmap_read_unlock(mm);
+err_mmput:
+	if (!ctx->mmap_locked)
+		mmput_async(mm);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_page_free - Put GPU SVM zone device data associated with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device data
+ * associated with a page when it is being released.
+ */
+static void drm_gpusvm_page_free(struct page *page)
+{
+	drm_gpusvm_zdd_put(page->zone_device_data);
+}
+
+/**
+ * drm_gpusvm_migrate_to_ram - Migrate GPU SVM range to RAM (page fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ * VM_FAULT_SIGBUS on failure, 0 on success.
+ */
+static vm_fault_t drm_gpusvm_migrate_to_ram(struct vm_fault *vmf)
+{
+	struct drm_gpusvm_zdd *zdd = vmf->page->zone_device_data;
+	int err;
+
+	err = __drm_gpusvm_migrate_to_sram(zdd->range->gpusvm,
+					   vmf->vma, vmf->page,
+					   zdd->range->va.start,
+					   zdd->range->va.end);
+
+	return err ? VM_FAULT_SIGBUS : 0;
+}
+
+/**
+ * drm_gpusvm_pagemap_ops - Device page map operations for GPU SVM
+ */
+static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = {
+	.page_free = drm_gpusvm_page_free,
+	.migrate_to_ram = drm_gpusvm_migrate_to_ram,
+};
+
+/**
+ * drm_gpusvm_pagemap_ops_get - Retrieve GPU SVM device page map operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void)
+{
+	return &drm_gpusvm_pagemap_ops;
+}
+
+/**
+ * drm_gpusvm_has_mapping - Check if GPU SVM has mapping for the given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end)
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	drm_gpusvm_for_each_notifier(notifier, gpusvm, start, end) {
+		struct drm_gpusvm_range *range = NULL;
+
+		drm_gpusvm_for_each_range(range, notifier, start, end)
+			return true;
+	}
+
+	return false;
+}
drivers/gpu/drm/xe/drm_gpusvm.h new file +415 -0
new file mode 100644
index 000000000000..0ea70f8534a8
@@ -0,0 +1,415 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef __DRM_GPUSVM_H__
+#define __DRM_GPUSVM_H__
+
+#include <linux/kref.h>
+#include <linux/mmu_notifier.h>
+#include <linux/workqueue.h>
+
+struct dev_pagemap_ops;
+struct drm_device;
+struct drm_gpusvm;
+struct drm_gpusvm_notifier;
+struct drm_gpusvm_ops;
+struct drm_gpusvm_range;
+
+/**
+ * struct drm_gpusvm_ops - Operations structure for GPU SVM
+ *
+ * This structure defines the operations for GPU Shared Virtual Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * perform operations such as migration between VRAM and system RAM.
+ */
+struct drm_gpusvm_ops {
+	/**
+	 * @notifier_alloc: Allocate a GPU SVM notifier (optional)
+	 *
+	 * This function shall allocate a GPU SVM notifier.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM notifier on success, NULL on failure.
+	 */
+	struct drm_gpusvm_notifier *(*notifier_alloc)(void);
+
+	/**
+	 * @notifier_free: Free a GPU SVM notifier (optional)
+	 * @notifier: Pointer to the GPU SVM notifier to be freed
+	 *
+	 * This function shall free a GPU SVM notifier.
+	 */
+	void (*notifier_free)(struct drm_gpusvm_notifier *notifier);
+
+	/**
+	 * @range_alloc: Allocate a GPU SVM range (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 *
+	 * This function shall allocate a GPU SVM range.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM range on success, NULL on failure.
+	 */
+	struct drm_gpusvm_range *(*range_alloc)(struct drm_gpusvm *gpusvm);
+
+	/**
+	 * @range_free: Free a GPU SVM range (optional)
+	 * @range: Pointer to the GPU SVM range to be freed
+	 *
+	 * This function shall free a GPU SVM range.
+	 */
+	void (*range_free)(struct drm_gpusvm_range *range);
+
+	/**
+	 * @vram_release: Release VRAM allocation (optional)
+	 * @vram_allocation: Driver-private pointer to the VRAM allocation
+	 *
+	 * This function shall release VRAM allocation and expects to drop a
+	 * reference to VRAM allocation.
+	 */
+	void (*vram_release)(void *vram_allocation);
+
+	/**
+	 * @populate_vram_pfn: Populate VRAM PFN (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @vram_allocation: Driver-private pointer to the VRAM allocation
+	 * @npages: Number of pages to populate
+	 * @pfn: Array of page frame numbers to populate
+	 *
+	 * This function shall populate VRAM page frame numbers (PFN).
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*populate_vram_pfn)(struct drm_gpusvm *gpusvm,
+				 void *vram_allocation,
+				 unsigned long npages,
+				 unsigned long *pfn);
+
+	/**
+	 * @copy_to_vram: Copy to VRAM (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (destination)
+	 * @dma_addr: Pointer to array of DMA addresses (source)
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to VRAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_vram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @copy_to_sram: Copy to system RAM (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (source)
+	 * @dma_addr: Pointer to array of DMA addresses (destination)
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to system RAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_sram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @invalidate: Invalidate GPU SVM notifier (required)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @notifier: Pointer to the GPU SVM notifier
+	 * @mmu_range: Pointer to the mmu_notifier_range structure
+	 *
+	 * This function shall invalidate the GPU page tables. It can safely
+	 * walk the notifier range RB tree/list in this function. Called while
+	 * holding the notifier lock.
+	 */
+	void (*invalidate)(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_notifier *notifier,
+			   const struct mmu_notifier_range *mmu_range);
+};
+
+/**
+ * struct drm_gpusvm_notifier - Structure representing a GPU SVM notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @interval: Interval for the notifier
+ * @rb: Red-black tree node for the parent GPU SVM structure notifier tree
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ *              interval tree. This is useful to keep iterating ranges while
+ *              doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier has been
+ *                 removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */
+struct drm_gpusvm_notifier {
+	struct drm_gpusvm *gpusvm;
+	struct mmu_interval_notifier notifier;
+	struct {
+		u64 start;
+		u64 end;
+	} interval;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct rb_root_cached root;
+	struct list_head range_list;
+	struct {
+		u32 removed : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm_range - Structure representing a GPU SVM range
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier
+ * @refcount: Reference count for the range
+ * @rb: Red-black tree node for the parent GPU SVM notifier structure range tree
+ * @va: Virtual address range
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @pages: Pointer to the array of pages (if backing store is in VRAM)
+ * @dma_addr: DMA address array (if backing store is SRAM and DMA mapped)
+ * @vram_allocation: Driver-private pointer to the VRAM allocation
+ * @order: Order of dma mapping. i.e. PAGE_SIZE << order is mapping size
+ * @flags.migrate_vram: Flag indicating whether the range can be migrated to VRAM
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_vram_pages: Flag indicating if the range has vram pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA mapping
+ * @flags.kfree_mapping: Flag indicating @dma_addr is a compact allocation based
+ *                       on @order which releases via kfree
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */
+struct drm_gpusvm_range {
+	struct drm_gpusvm *gpusvm;
+	struct drm_gpusvm_notifier *notifier;
+	struct kref refcount;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct {
+		u64 start;
+		u64 end;
+	} va;
+	unsigned long notifier_seq;
+	union {
+		struct page **pages;
+		dma_addr_t *dma_addr;
+	};
+	void *vram_allocation;
+	u16 order;
+	struct {
+		/* All flags below must be set upon creation */
+		u16 migrate_vram : 1;
+		/* All flags below must be set / cleared under notifier lock */
+		u16 unmapped : 1;
+		u16 partial_unmap : 1;
+		u16 has_vram_pages : 1;
+		u16 has_dma_mapping : 1;
+		u16 kfree_mapping : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm - GPU SVM structure
+ *
+ * @name: Name of the GPU SVM
+ * @drm: Pointer to the DRM device structure
+ * @mm: Pointer to the mm_struct for the address space
+ * @device_private_page_owner: Device private pages owner
+ * @mm_start: Start address of GPU SVM
+ * @mm_range: Range of the GPU SVM
+ * @notifier_size: Size of individual notifiers
+ * @ops: Pointer to the operations structure for GPU SVM
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range allocation.
+ *               Entries should be powers of 2 in descending order.
+ * @num_chunks: Number of chunks
+ * @notifier_lock: Read-write semaphore for protecting notifier operations
+ * @zdd_wq: Workqueue for deferred work on zdd destruction
+ * @root: Cached root node of the Red-Black tree containing GPU SVM notifiers
+ * @notifier_list: list head containing of notifiers in the same order they
+ *                 appear in interval tree. This is useful to keep iterating
+ *                 notifiers while doing modifications to RB tree.
+ *
+ * This structure represents a GPU SVM (Shared Virtual Memory) used for tracking
+ * memory ranges mapped in a DRM (Direct Rendering Manager) device.
+ *
+ * No reference counting is provided, as this is expected to be embedded in the
+ * driver VM structure along with the struct drm_gpuvm, which handles reference
+ * counting.
+ */
+struct drm_gpusvm {
+	const char *name;
+	struct drm_device *drm;
+	struct mm_struct *mm;
+	void *device_private_page_owner;
+	u64 mm_start;
+	u64 mm_range;
+	u64 notifier_size;
+	const struct drm_gpusvm_ops *ops;
+	const u64 *chunk_sizes;
+	int num_chunks;
+	struct rw_semaphore notifier_lock;
+	struct workqueue_struct *zdd_wq;
+	struct rb_root_cached root;
+	struct list_head notifier_list;
+};
+
+/**
+ * struct drm_gpusvm_ctx - DRM GPU SVM context
+ *
+ * @mmap_locked: mmap lock is locked
+ * @trylock_mmap: trylock mmap lock, used to avoid locking inversions
+ *                (e.g.dma-revs -> mmap lock)
+ * @in_notifier: entering from a MMU notifier
+ * @read_only: operating on read-only memory
+ * @vram_possible: possible to use VRAM
+ * @prefault: prefault pages
+ *
+ * Context that is DRM GPUSVM is operating in (i.e. user arguments).
+ */
+struct drm_gpusvm_ctx {
+	u32 mmap_locked :1;
+	u32 trylock_mmap :1;
+	u32 in_notifier :1;
+	u32 read_only :1;
+	u32 vram_possible :1;
+	u32 prefault :1;
+};
+
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void *device_private_page_owner,
+		    u64 mm_start, u64 mm_range, u64 notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const u64 *chunk_sizes, int num_chunks);
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm);
+void drm_gpusvm_free(struct drm_gpusvm *gpusvm);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, u64 fault_addr,
+				u64 gpuva_start, u64 gpuva_end,
+				const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range);
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range);
+
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range);
+
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx);
+
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx);
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void);
+
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, u64 start, u64 end);
+
+/**
+ * drm_gpusvm_notifier_lock - Lock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, take lock
+ */
+#define drm_gpusvm_notifier_lock(gpusvm__)	\
+	down_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * drm_gpusvm_notifier_unlock - Unlock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, drop lock
+ */
+#define drm_gpusvm_notifier_unlock(gpusvm__)	\
+	up_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * __drm_gpusvm_range_next - Get the next GPU SVM range in the list
+ * @range: a pointer to the current GPU SVM range
+ *
+ * Return: A pointer to the next drm_gpusvm_range if available, or NULL if the
+ *         current range is the last one or if the input range is NULL.
+ */
+static inline struct drm_gpusvm_range *
+__drm_gpusvm_range_next(struct drm_gpusvm_range *range)
+{
+	if (range && !list_is_last(&range->rb.entry,
+				   &range->notifier->range_list))
+		return list_next_entry(range, rb.entry);
+
+	return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_range - Iterate over GPU SVM ranges in a notifier
+ * @range__: Iterator variable for the ranges. If set, it indicates the start of
+ *	     the iterator. If NULL, call drm_gpusvm_range_find() to get the range.
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier. It is safe
+ * to use while holding the driver SVM lock or the notifier lock.
+ */
+#define drm_gpusvm_for_each_range(range__, notifier__, start__, end__)	\
+	for ((range__) = (range__) ?:					\
+	     drm_gpusvm_range_find((notifier__), (start__), (end__));	\
+	     (range__) && (range__->va.start < (end__));		\
+	     (range__) = __drm_gpusvm_range_next(range__))
+
+/**
+ * drm_gpusvm_range_set_unmapped - Mark a GPU SVM range as unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets the partial_unmap flag
+ * if the range partially falls within the provided MMU notifier range.
+ */
+static inline void
+drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,
+			      const struct mmu_notifier_range *mmu_range)
+{
+	lockdep_assert_held_write(&range->gpusvm->notifier_lock);
+
+	range->flags.unmapped = true;
+	if (range->va.start < mmu_range->start ||
+	    range->va.end > mmu_range->end)
+		range->flags.partial_unmap = true;
+}
+
+#endif /* __DRM_GPUSVM_H__ */
Signature · 1 line
2.34.1
Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#14 lore raw

Re: [RFC PATCH 05/28] drm/gpusvm: Add support for GPU Shared Virtual Memory

Quoted 99 lines · Matthew Brost
Quoted 76 lines · Daniel Vetter
Quoted 34 lines · Matthew Brost
Quoted 13 lines · Christian König

Am 28.08.24 um 16:31 schrieb Daniel Vetter:

Quoted 12 lines · Matthew Brost

drivers/gpu/drm/xe/drm_gpusvm.c line 1353

Quoted 11 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+		if (!ctx->mmap_locked) {
+			/*
+			 * XXX: HMM locking document indicates only a read-lock
+			 * is required but there apears to be a window between
+			 * the MMU_NOTIFY_MIGRATE event triggered in a CPU fault
+			 * via migrate_vma_setup and the pages actually moving
+			 * in migrate_vma_finalize in which this code can grab
+			 * garbage pages. Grabbing the write-lock if the range
+			 * is attached to vram appears to protect against this
+			 * race.
+			 */

Thanks the comments, replying to both of you inline.

Quoted 7 lines
Quoted 7 lines

This one is really scary, since it means the entire migrate pte trickery is essentially completely busted. Grabbing the mmap write lock just means you block out pretty much everything interesting from concurrently happening.

My gut feeling says we need to figure out what's happening here, because this looks a bit too fundamental to me.

I agree. I haven’t looked into this issue for a couple of months but really need to understand what is going on.

I should have mentioned this in the cover letter: the goal of this series was to produce something for review that is stable and supports UMDs/user applications. It was not intended to be presented as a final solution. This issue certainly falls into the category of "needs to be understood and requires a proper fix."

Quoted 6 lines

One open question I have is whether the test case that triggers this issue is even defined behavior. The test creates concurrent access between the GPU and CPU to the same memory address, resulting in GPU and CPU faults racing against each other. It’s possible that this is undefined behavior, so data corruption might be acceptable—i.e., the kernel can’t crash, but incorrect results might be permissible.

Yes this is supposed to be defined, at least from an hmm pov. And core mm/ is ridiculous in how many races it allows, especially around concurrent fault handling.

It is ofc really slow if every fault results in a migration, but that's a matter of the application setting stupid memory migration hints for the gpu.

Quoted 26 lines

e.g. This is the only defined usage model:

alloc_memory();
start_compute_kernel();
sync_on_compute_kernel_completion();
read_memory();

Hopefully, in the next week or so, I'll be heavily engaging with the UMD teams. Development can then start, and applications will be running soon after. This will allow us to address issues like this, collect data on memory usage, and verify some of the assumptions I've made, such as optimizing for 2M+ allocations.

Quoted 4 lines

I think I have at least a high level understanding what's going on here, Felix and especially Philip should know more of the details.

I meant to reach out to AMD for issues like this. So, Felix ([email protected]) and Philip ([email protected]) would be good contacts?

Quoted 3 lines

In general grabbing the mm_lock to protect PTEs from changing is completely nonsense. The mm_lock is to protect the VMAs and *not* the PTEs!

Quoted 7 lines
Quoted 3 lines

Thanks for the hint. I believe that in the AMD implementation, I noticed some additional locks for migration, which might be how you mitigated this issue.

Yeah, so in general hold mmap_reading is indeed pure magic thinking for preventing pte changes, like Christian points out. It doesn't stop invalidates, and with the per vma locking it also doesn't stop new valid

Invalidations happening to parallel to migrations, get pages, or bindings should be fine. The notifier lock usage should make all of this safe.

Quoted 9 lines

ptes from being inserted at least for anon memory.

Except migration pte entries that point at vram pages are special, and are _only_ resolved while holding mmap_read. Which means holding mmap_write for the case of looking up our own vram pages with hmm_range_fault actually prevents issues. And so this duct-tape of holding mmap_write very much looks like a working hack to plug any races against concurrently ongoing migrations to system memory due to cpu faults.

Quoted 7 lines

Agree holding mmap_write is a hack. Looking at AMD 'To serialize concurrent migrations or validations of the same range, the prange->migrate_mutex must be held.', seemly I could drop mmap write lock abuse and use something like this here. The would like be an inner lock of the mmap lock.

Does this seem like a reasonable thing to explore?

Meta-comment: Since I've learned as I typed replies there's a bit a mess in my suggestions/questions ...

See the other replies, I think prage->migrate_mutex doesn't work because we need a lock on the physical storage, not on the virtual range. Because thanks to mremap (if I understand that thing right) and fork does do not need to line up, or be unique.

The other thing I only slowly realized is that the code in migrate.c forces full page/folio lock semantics on use already, like a migration done by core mm code between sram and sram. So I /think/ that if we follow the rules correctly, the page/folio lock should be enough to sufficiently serialize migrations.

But then there's amd's migration_mutex, and your migration vs fault troubles, and I'm honestly not really understanding those races ..

So my current thinking is that page/folio lock should be enough, or we have bugs or wrong understanding of how this should work.

Quoted 7 lines
Quoted 2 lines

An even more fun corner case is multiple concurrent cpu faults on the same vram page. fork gets you that, or maybe a bit more reasonable mremap with

My understanding is memory shared between processes cannot migrated due to current limitations migrate layer.

e.g. mmap called with MAP_SHARED is not eligible for migration.

Hm, where is that limitation? All the code I've seen suggests it all should work, and the only memory not eligible for hmm/zone_device migration is pagecache. But any kind of anon memory, whether private (i.e. cow on fork) or shared should work?

If that's not the case then yeah a _lot_ of what I said is just plain wrong I think ...

Quoted 11 lines

Unsure what the behavior is fork() is called on a process with memory in VRAM and the child tries to access it. Maybe fork() is different than MAP_SHARED where as parent / child processes can share memory in VRAM?

Also really I'm unsure what would happen if user space calls fork() and has an Xe VM open and tries to use it too. Before commenting more on this, I need play around with test cases like this educate myself.

My current test doesn't use mremap, agree that would be to good add. Again before commenting more here, let add more test cases to educate myself.

Yeah I definitely need more learning too.

Quoted 8 lines
Quoted 3 lines

MREMAP_DONTUNMAP | MREMAP_MAYMOVE. I think just hammer the same va with multiple threads along isn't enough, it's better to have a private va for

I do have test cases where multiple CPU faults from threads hammer the same memory. Found some bugs in my initial code but as far as I can tell multiple CPU faults in parallel occur in my testing and do work.

Quoted 5 lines
Quoted 1 line

each thread pointing at the same anon memory page, so that you can get

You are losing me here - 'private va for each thread pointing at the same anon memory page'. This is a fork() case where the parent allocates memory and then all children try to read in parallel?

shared anon memory should be a thing, at least to my knowledge. Which might be wrong ...

Quoted 14 lines
Quoted 4 lines

more parallel faults due to finely grained pte locking.

Would be a good testcase to add, if you don't have it yet.

See above, agree these are good test cases which I haven't considered and will expand my suite to include these. Thanks for the tip - IMO testing is as important or even more important than the KMD design and need to ensure I have all possible uses covered.

Quoted 4 lines
Quoted 3 lines

I must say it is a bit unfortunate that the HMM locking documentation doesn’t mention this. I believe the documentation needs additional information, which I can add once we finalize the solution.

Quoted 6 lines
Quoted 4 lines

Yeah, at least from my very cursory lock you don't have enough locking. I've written an in-depth reply to patch 23 with the high-level summary of my thoughts.

Will look and reply there.

Cheers, Sima

Quoted 1,270 lines

Matt

Quoted 1,267 lines

Cheers, Sima

Quoted 1,260 lines

Matt

Quoted 1,257 lines

Even with the write side of the mm_lock taken it is perfectly possible that PTE change. It's just less likely.

We run into multiple issues before we figured out this important distinction as well.

Christian.

Quoted 1,248 lines

-Sima

Quoted 1,245 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +821 -0
+			if (vram_pages)
+				mmap_write_lock(mm);
+			else
+				mmap_read_lock(mm);
+		}
+		err = hmm_range_fault(&hmm_range);
+		if (!ctx->mmap_locked) {
+			if (vram_pages)
+				mmap_write_unlock(mm);
+			else
+				mmap_read_unlock(mm);
+		}
+
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
+			continue;
+		}
+		break;
+	}
+	if (!ctx->mmap_locked)
+		mmput(mm);
+	if (err)
+		goto err_free;
+
+	pages = (struct page **)pfns;
+
+	if (ctx->prefault) {
+		range->pages = pages;
+		goto set_seqno;
+	}
+
+map_pages:
+	if (is_device_private_page(hmm_pfn_to_page(pfns[0]))) {
+		WARN_ON_ONCE(!range->vram_allocation);
+
+		for (i = 0; i < npages; ++i) {
+			pages[i] = hmm_pfn_to_page(pfns[i]);
+
+			if (WARN_ON_ONCE(!is_device_private_page(pages[i]))) {
+				err = -EOPNOTSUPP;
+				goto err_free;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->flags.has_vram_pages = true;
+		range->pages = pages;
+		if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm, range);
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	} else {
+		dma_addr_t *dma_addr = (dma_addr_t *)pfns;
+
+		for_each_dma_page(i, j, npages, order) {
+			if (WARN_ON_ONCE(i && order !=
+					 hmm_pfn_to_map_order(pfns[i]))) {
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+			order = hmm_pfn_to_map_order(pfns[i]);
+
+			pages[j] = hmm_pfn_to_page(pfns[i]);
+			if (WARN_ON_ONCE(is_zone_device_page(pages[j]))) {
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+
+			set_page_dirty_lock(pages[j]);
+			mark_page_accessed(pages[j]);
+
+			dma_addr[j] = dma_map_page(gpusvm->drm->dev,
+						   pages[j], 0,
+						   PAGE_SIZE << order,
+						   DMA_BIDIRECTIONAL);
+			if (dma_mapping_error(gpusvm->drm->dev, dma_addr[j])) {
+				err = -EFAULT;
+				npages = i;
+				goto err_unmap;
+			}
+		}
+
+		/* Huge pages, reduce memory footprint */
+		if (order) {
+			dma_addr = kmalloc_array(j, sizeof(*dma_addr),
+						 GFP_KERNEL);
+			if (dma_addr) {
+				for (i = 0; i < j; ++i)
+					dma_addr[i] = (dma_addr_t)pfns[i];
+				kvfree(pfns);
+				kfree_mapping = true;
+			} else {
+				dma_addr = (dma_addr_t *)pfns;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->order = order;
+		range->flags.kfree_mapping = kfree_mapping;
+		range->flags.has_dma_mapping = true;
+		range->dma_addr = dma_addr;
+		range->vram_allocation = NULL;
+		if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm, range);
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	}
+
+	if (err == -EAGAIN)
+		goto retry;
+set_seqno:
+	range->notifier_seq = hmm_range.notifier_seq;
+
+	return 0;
+
+err_unmap:
+	for_each_dma_page(i, j, npages, order)
+		dma_unmap_page(gpusvm->drm->dev,
+			       (dma_addr_t)pfns[j],
+			       PAGE_SIZE << order, DMA_BIDIRECTIONAL);
+err_free:
+	if (alloc_pfns)
+		kvfree(pfns);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_range_unmap_pages - Unmap pages associated with a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed that gpusvm->notifier_lock is held in write mode; if it
+ * is clear, it acquires gpusvm->notifier_lock in read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops->invalidate for IOMMU
+ * security model.
+ */
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx)
+{
+	if (ctx->in_notifier)
+		lockdep_assert_held_write(&gpusvm->notifier_lock);
+	else
+		drm_gpusvm_notifier_lock(gpusvm);
+
+	__drm_gpusvm_range_unmap_pages(gpusvm, range);
+
+	if (!ctx->in_notifier)
+		drm_gpusvm_notifier_unlock(gpusvm);
+}
+
+/**
+ * drm_gpusvm_migration_put_page - Put a migration page
+ * @page: Pointer to the page to put
+ *
+ * This function unlocks and puts a page.
+ */
+static void drm_gpusvm_migration_put_page(struct page *page)
+{
+	unlock_page(page);
+	put_page(page);
+}
+
+/**
+ * drm_gpusvm_migration_put_pages - Put migration pages
+ * @npages: Number of pages
+ * @migrate_pfn: Array of migrate page frame numbers
+ *
+ * This function puts an array of pages.
+ */
+static void drm_gpusvm_migration_put_pages(unsigned long npages,
+					   unsigned long *migrate_pfn)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!migrate_pfn[i])
+			continue;
+
+		drm_gpusvm_migration_put_page(migrate_pfn_to_page(migrate_pfn[i]));
+		migrate_pfn[i] = 0;
+	}
+}
+
+/**
+ * drm_gpusvm_get_vram_page - Get a reference to a VRAM page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU SVM zone
+ * device data and initializes it for zone device usage.
+ */
+static void drm_gpusvm_get_vram_page(struct page *page,
+				     struct drm_gpusvm_zdd *zdd)
+{
+	page->zone_device_data = drm_gpusvm_zdd_get(zdd);
+	zone_device_page_init(page);
+}
+
+/**
+ * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Return: 0 on success, -EFAULT if an error occurs during mapping.
+ */
+static int drm_gpusvm_migrate_map_pages(struct device *dev,
+					dma_addr_t *dma_addr,
+					long unsigned int *migrate_pfn,
+					unsigned long npages,
+					enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = migrate_pfn_to_page(migrate_pfn[i]);
+
+		if (!page)
+			continue;
+
+		if (WARN_ON_ONCE(is_zone_device_page(page)))
+			return -EFAULT;
+
+		dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE, dir);
+		if (dma_mapping_error(dev, dma_addr[i]))
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */
+static void drm_gpusvm_migrate_unmap_pages(struct device *dev,
+					   dma_addr_t *dma_addr,
+					   unsigned long npages,
+					   enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!dma_addr[i] || dma_mapping_error(dev, dma_addr[i]))
+			continue;
+
+		dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir);
+	}
+}
+
+/**
+ * drm_gpusvm_migrate_to_vram - Migrate GPU SVM range to VRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *                   failure of this function.
+ * @vram_allocation: Driver-private pointer to the VRAM allocation. The caller
+ *                   should hold a reference to the VRAM allocation, which
+ *                   should be dropped via ops->vram_allocation or upon the
+ *                   failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to VRAM. It performs the
+ * necessary setup and invokes the driver-specific operations for migration to
+ * VRAM. Upon successful return, @vram_allocation can safely reference @range
+ * until ops->vram_release is called which only upon successful return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct migrate_vma migrate = {
+		.start		= start,
+		.end		= end,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_SYSTEM,
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	unsigned long i, npages = npages_in_range(start, end);
+	struct vm_area_struct *vas;
+	struct drm_gpusvm_zdd *zdd = NULL;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int err;
+
+	if (!range->flags.migrate_vram)
+		return -EINVAL;
+
+	if (!gpusvm->ops->populate_vram_pfn || !gpusvm->ops->copy_to_vram ||
+	    !gpusvm->ops->copy_to_sram)
+		return -EOPNOTSUPP;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		mmap_write_lock(mm);
+	}
+
+	mmap_assert_locked(mm);
+
+	vas = vma_lookup(mm, start);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end > vas->vm_end || start < vas->vm_start) {
+		err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	if (!vma_is_anonymous(vas)) {
+		err = -EBUSY;
+		goto err_mmunlock;
+	}
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_mmunlock;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+	zdd = drm_gpusvm_zdd_alloc(range);
+	if (!zdd) {
+		err = -ENOMEM;
+		goto err_free;
+	}
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/*
+	 * FIXME: Below cases, !migrate.cpages and migrate.cpages != npages, not
+	 * always an error. Need to revisit possible cases and how to handle. We
+	 * could prefault on migrate.cpages != npages via hmm_range_fault.
+	 */
+
+	if (!migrate.cpages) {
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	if (migrate.cpages != npages) {
+		err = -EBUSY;
+		goto err_finalize;
+	}
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm, vram_allocation, npages,
+					     migrate.dst);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   migrate.src, npages, DMA_TO_DEVICE);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = pfn_to_page(migrate.dst[i]);
+
+		pages[i] = page;
+		migrate.dst[i] = migrate_pfn(migrate.dst[i]);
+		drm_gpusvm_get_vram_page(page, zdd);
+	}
+
+	err = gpusvm->ops->copy_to_vram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+	/* Upon success bind vram allocation to range and zdd */
+	range->vram_allocation = vram_allocation;
+	WRITE_ONCE(zdd->vram_allocation, vram_allocation);	/* Owns ref */
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_TO_DEVICE);
+err_free:
+	if (zdd)
+		drm_gpusvm_zdd_put(zdd);
+	kvfree(buf);
+err_mmunlock:
+	if (!ctx->mmap_locked) {
+		mmap_write_unlock(mm);
+		mmput(mm);
+	}
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_populate_sram_pfn - Populate SRAM PFNs for a VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @npages: Number of pages to populate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the SRAM migrate page frame numbers (PFNs) for the
+ * specified VM area structure. It allocates and locks pages in the VM area for
+ * SRAM usage. If vas is non-NULL use alloc_page_vma for allocation, if NULL use
+ * alloc_page for allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_sram_pfn(struct vm_area_struct *vas,
+						unsigned long npages,
+						unsigned long *src_mpfn,
+						unsigned long *mpfn, u64 addr)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i, addr += PAGE_SIZE) {
+		struct page *page;
+
+		if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE))
+			continue;
+
+		if (vas)
+			page = alloc_page_vma(GFP_HIGHUSER, vas, addr);
+		else
+			page = alloc_page(GFP_HIGHUSER);
+
+		if (!page)
+			return -ENOMEM;
+
+		lock_page(page);
+		mpfn[i] = migrate_pfn(page_to_pfn(page));
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_evict_to_sram - Evict GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * Similar to __drm_gpusvm_migrate_to_sram but does not require mmap lock and
+ * migration done via migrate_device_* functions. Fallback path as it is
+ * preferred to issue migrations with mmap lock.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_evict_to_sram(struct drm_gpusvm *gpusvm,
+				    struct drm_gpusvm_range *range)
+{
+	unsigned long npages;
+	struct page **pages;
+	unsigned long *src, *dst;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	npages = npages_in_range(range->va.start, range->va.end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	src = buf;
+	dst = buf + (sizeof(*src) * npages);
+	dma_addr = buf + (2 * sizeof(*src) * npages);
+	pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) * npages;
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm, range->vram_allocation,
+					     npages, src);
+	if (err)
+		goto err_free;
+
+	err = migrate_device_vma_range(gpusvm->mm,
+				       gpusvm->device_private_page_owner, src,
+				       npages, range->va.start);
+	if (err)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(NULL, npages, src, dst, 0);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   dst, npages, DMA_BIDIRECTIONAL);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, dst);
+	migrate_device_pages(src, dst, npages);
+	migrate_device_finalize(src, dst, npages);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+
+	return err;
+}
+
+/**
+ * __drm_gpusvm_migrate_to_sram - Migrate GPU SVM range to SRAM (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @vas: Pointer to the VM area structure
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @start: Start address of the migration range
+ * @end: End address of the migration range
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to SRAM. It sets up the migration, populates + dma maps SRAM PFNs, and
+ * invokes the driver-specific operations for migration to SRAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int __drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+					struct vm_area_struct *vas,
+					struct page *page,
+					u64 start, u64 end)
+{
+	struct migrate_vma migrate = {
+		.vma		= vas,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_DEVICE_PRIVATE,
+		.fault_page	= page,
+	};
+	unsigned long npages;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	mmap_assert_locked(gpusvm->mm);
+
+	/* Corner where VMA area struct has been partially unmapped */
+	if (start < vas->vm_start)
+		start = vas->vm_start;
+	if (end > vas->vm_end)
+		end = vas->vm_end;
+
+	migrate.start = start;
+	migrate.end = end;
+	npages = npages_in_range(start, end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/* Raced with another CPU fault, nothing to do */
+	if (!migrate.cpages)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(vas, npages,
+						   migrate.src, migrate.dst,
+						   start);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   migrate.dst, npages,
+					   DMA_BIDIRECTIONAL);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(migrate.src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+	mmap_assert_locked(gpusvm->mm);
+
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_to_sram - Migrate (evict) GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function initiates the migration of the specified GPU SVM range to
+ * SRAM. It performs necessary checks and invokes the internal migration
+ * function for actual migration.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	int err;
+	bool retry = false;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		if (ctx->trylock_mmap) {
+			if (!mmap_read_trylock(mm))  {
+				err = drm_gpusvm_evict_to_sram(gpusvm, range);
+				goto err_mmput;
+			}
+		} else {
+			mmap_read_lock(mm);
+		}
+	}
+
+	mmap_assert_locked(mm);
+
+	/*
+	 * Loop required to find all VMA area structs for the corner case when
+	 * VRAM backing has been partially unmapped from MM's address space.
+	 */
+again:
+	vas = find_vma(mm, start);
+	if (!vas) {
+		if (!retry)
+			err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end <= vas->vm_start || start >= vas->vm_end) {
+		if (!retry)
+			err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	err = __drm_gpusvm_migrate_to_sram(gpusvm, vas, NULL, start, end);
+	if (err)
+		goto err_mmunlock;
+
+	if (vas->vm_end < end) {
+		retry = true;
+		start = vas->vm_end;
+		goto again;
+	}
+
+	if (!ctx->mmap_locked) {
+		mmap_read_unlock(mm);
+		/*
+		 * Using mmput_async as this function can be called while
+		 * holding a dma-resv lock, and a final put can grab the mmap
+		 * lock, causing a lock inversion.
+		 */
+		mmput_async(mm);
+	}
+
+	return 0;
+
+err_mmunlock:
+	if (!ctx->mmap_locked)
+		mmap_read_unlock(mm);
+err_mmput:
+	if (!ctx->mmap_locked)
+		mmput_async(mm);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_page_free - Put GPU SVM zone device data associated with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device data
+ * associated with a page when it is being released.
+ */
+static void drm_gpusvm_page_free(struct page *page)
+{
+	drm_gpusvm_zdd_put(page->zone_device_data);
+}
+
+/**
+ * drm_gpusvm_migrate_to_ram - Migrate GPU SVM range to RAM (page fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ * VM_FAULT_SIGBUS on failure, 0 on success.
+ */
+static vm_fault_t drm_gpusvm_migrate_to_ram(struct vm_fault *vmf)
+{
+	struct drm_gpusvm_zdd *zdd = vmf->page->zone_device_data;
+	int err;
+
+	err = __drm_gpusvm_migrate_to_sram(zdd->range->gpusvm,
+					   vmf->vma, vmf->page,
+					   zdd->range->va.start,
+					   zdd->range->va.end);
+
+	return err ? VM_FAULT_SIGBUS : 0;
+}
+
+/**
+ * drm_gpusvm_pagemap_ops - Device page map operations for GPU SVM
+ */
+static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = {
+	.page_free = drm_gpusvm_page_free,
+	.migrate_to_ram = drm_gpusvm_migrate_to_ram,
+};
+
+/**
+ * drm_gpusvm_pagemap_ops_get - Retrieve GPU SVM device page map operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void)
+{
+	return &drm_gpusvm_pagemap_ops;
+}
+
+/**
+ * drm_gpusvm_has_mapping - Check if GPU SVM has mapping for the given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end)
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	drm_gpusvm_for_each_notifier(notifier, gpusvm, start, end) {
+		struct drm_gpusvm_range *range = NULL;
+
+		drm_gpusvm_for_each_range(range, notifier, start, end)
+			return true;
+	}
+
+	return false;
+}
drivers/gpu/drm/xe/drm_gpusvm.h new file +415 -0
new file mode 100644
index 000000000000..0ea70f8534a8
@@ -0,0 +1,415 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef __DRM_GPUSVM_H__
+#define __DRM_GPUSVM_H__
+
+#include <linux/kref.h>
+#include <linux/mmu_notifier.h>
+#include <linux/workqueue.h>
+
+struct dev_pagemap_ops;
+struct drm_device;
+struct drm_gpusvm;
+struct drm_gpusvm_notifier;
+struct drm_gpusvm_ops;
+struct drm_gpusvm_range;
+
+/**
+ * struct drm_gpusvm_ops - Operations structure for GPU SVM
+ *
+ * This structure defines the operations for GPU Shared Virtual Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * perform operations such as migration between VRAM and system RAM.
+ */
+struct drm_gpusvm_ops {
+	/**
+	 * @notifier_alloc: Allocate a GPU SVM notifier (optional)
+	 *
+	 * This function shall allocate a GPU SVM notifier.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM notifier on success, NULL on failure.
+	 */
+	struct drm_gpusvm_notifier *(*notifier_alloc)(void);
+
+	/**
+	 * @notifier_free: Free a GPU SVM notifier (optional)
+	 * @notifier: Pointer to the GPU SVM notifier to be freed
+	 *
+	 * This function shall free a GPU SVM notifier.
+	 */
+	void (*notifier_free)(struct drm_gpusvm_notifier *notifier);
+
+	/**
+	 * @range_alloc: Allocate a GPU SVM range (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 *
+	 * This function shall allocate a GPU SVM range.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM range on success, NULL on failure.
+	 */
+	struct drm_gpusvm_range *(*range_alloc)(struct drm_gpusvm *gpusvm);
+
+	/**
+	 * @range_free: Free a GPU SVM range (optional)
+	 * @range: Pointer to the GPU SVM range to be freed
+	 *
+	 * This function shall free a GPU SVM range.
+	 */
+	void (*range_free)(struct drm_gpusvm_range *range);
+
+	/**
+	 * @vram_release: Release VRAM allocation (optional)
+	 * @vram_allocation: Driver-private pointer to the VRAM allocation
+	 *
+	 * This function shall release VRAM allocation and expects to drop a
+	 * reference to VRAM allocation.
+	 */
+	void (*vram_release)(void *vram_allocation);
+
+	/**
+	 * @populate_vram_pfn: Populate VRAM PFN (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @vram_allocation: Driver-private pointer to the VRAM allocation
+	 * @npages: Number of pages to populate
+	 * @pfn: Array of page frame numbers to populate
+	 *
+	 * This function shall populate VRAM page frame numbers (PFN).
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*populate_vram_pfn)(struct drm_gpusvm *gpusvm,
+				 void *vram_allocation,
+				 unsigned long npages,
+				 unsigned long *pfn);
+
+	/**
+	 * @copy_to_vram: Copy to VRAM (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (destination)
+	 * @dma_addr: Pointer to array of DMA addresses (source)
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to VRAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_vram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @copy_to_sram: Copy to system RAM (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (source)
+	 * @dma_addr: Pointer to array of DMA addresses (destination)
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to system RAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_sram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @invalidate: Invalidate GPU SVM notifier (required)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @notifier: Pointer to the GPU SVM notifier
+	 * @mmu_range: Pointer to the mmu_notifier_range structure
+	 *
+	 * This function shall invalidate the GPU page tables. It can safely
+	 * walk the notifier range RB tree/list in this function. Called while
+	 * holding the notifier lock.
+	 */
+	void (*invalidate)(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_notifier *notifier,
+			   const struct mmu_notifier_range *mmu_range);
+};
+
+/**
+ * struct drm_gpusvm_notifier - Structure representing a GPU SVM notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @interval: Interval for the notifier
+ * @rb: Red-black tree node for the parent GPU SVM structure notifier tree
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ *              interval tree. This is useful to keep iterating ranges while
+ *              doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier has been
+ *                 removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */
+struct drm_gpusvm_notifier {
+	struct drm_gpusvm *gpusvm;
+	struct mmu_interval_notifier notifier;
+	struct {
+		u64 start;
+		u64 end;
+	} interval;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct rb_root_cached root;
+	struct list_head range_list;
+	struct {
+		u32 removed : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm_range - Structure representing a GPU SVM range
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier
+ * @refcount: Reference count for the range
+ * @rb: Red-black tree node for the parent GPU SVM notifier structure range tree
+ * @va: Virtual address range
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @pages: Pointer to the array of pages (if backing store is in VRAM)
+ * @dma_addr: DMA address array (if backing store is SRAM and DMA mapped)
+ * @vram_allocation: Driver-private pointer to the VRAM allocation
+ * @order: Order of dma mapping. i.e. PAGE_SIZE << order is mapping size
+ * @flags.migrate_vram: Flag indicating whether the range can be migrated to VRAM
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_vram_pages: Flag indicating if the range has vram pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA mapping
+ * @flags.kfree_mapping: Flag indicating @dma_addr is a compact allocation based
+ *                       on @order which releases via kfree
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */
+struct drm_gpusvm_range {
+	struct drm_gpusvm *gpusvm;
+	struct drm_gpusvm_notifier *notifier;
+	struct kref refcount;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct {
+		u64 start;
+		u64 end;
+	} va;
+	unsigned long notifier_seq;
+	union {
+		struct page **pages;
+		dma_addr_t *dma_addr;
+	};
+	void *vram_allocation;
+	u16 order;
+	struct {
+		/* All flags below must be set upon creation */
+		u16 migrate_vram : 1;
+		/* All flags below must be set / cleared under notifier lock */
+		u16 unmapped : 1;
+		u16 partial_unmap : 1;
+		u16 has_vram_pages : 1;
+		u16 has_dma_mapping : 1;
+		u16 kfree_mapping : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm - GPU SVM structure
+ *
+ * @name: Name of the GPU SVM
+ * @drm: Pointer to the DRM device structure
+ * @mm: Pointer to the mm_struct for the address space
+ * @device_private_page_owner: Device private pages owner
+ * @mm_start: Start address of GPU SVM
+ * @mm_range: Range of the GPU SVM
+ * @notifier_size: Size of individual notifiers
+ * @ops: Pointer to the operations structure for GPU SVM
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range allocation.
+ *               Entries should be powers of 2 in descending order.
+ * @num_chunks: Number of chunks
+ * @notifier_lock: Read-write semaphore for protecting notifier operations
+ * @zdd_wq: Workqueue for deferred work on zdd destruction
+ * @root: Cached root node of the Red-Black tree containing GPU SVM notifiers
+ * @notifier_list: list head containing of notifiers in the same order they
+ *                 appear in interval tree. This is useful to keep iterating
+ *                 notifiers while doing modifications to RB tree.
+ *
+ * This structure represents a GPU SVM (Shared Virtual Memory) used for tracking
+ * memory ranges mapped in a DRM (Direct Rendering Manager) device.
+ *
+ * No reference counting is provided, as this is expected to be embedded in the
+ * driver VM structure along with the struct drm_gpuvm, which handles reference
+ * counting.
+ */
+struct drm_gpusvm {
+	const char *name;
+	struct drm_device *drm;
+	struct mm_struct *mm;
+	void *device_private_page_owner;
+	u64 mm_start;
+	u64 mm_range;
+	u64 notifier_size;
+	const struct drm_gpusvm_ops *ops;
+	const u64 *chunk_sizes;
+	int num_chunks;
+	struct rw_semaphore notifier_lock;
+	struct workqueue_struct *zdd_wq;
+	struct rb_root_cached root;
+	struct list_head notifier_list;
+};
+
+/**
+ * struct drm_gpusvm_ctx - DRM GPU SVM context
+ *
+ * @mmap_locked: mmap lock is locked
+ * @trylock_mmap: trylock mmap lock, used to avoid locking inversions
+ *                (e.g.dma-revs -> mmap lock)
+ * @in_notifier: entering from a MMU notifier
+ * @read_only: operating on read-only memory
+ * @vram_possible: possible to use VRAM
+ * @prefault: prefault pages
+ *
+ * Context that is DRM GPUSVM is operating in (i.e. user arguments).
+ */
+struct drm_gpusvm_ctx {
+	u32 mmap_locked :1;
+	u32 trylock_mmap :1;
+	u32 in_notifier :1;
+	u32 read_only :1;
+	u32 vram_possible :1;
+	u32 prefault :1;
+};
+
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void *device_private_page_owner,
+		    u64 mm_start, u64 mm_range, u64 notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const u64 *chunk_sizes, int num_chunks);
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm);
+void drm_gpusvm_free(struct drm_gpusvm *gpusvm);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, u64 fault_addr,
+				u64 gpuva_start, u64 gpuva_end,
+				const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range);
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range);
+
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range);
+
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx);
+
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx);
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void);
+
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, u64 start, u64 end);
+
+/**
+ * drm_gpusvm_notifier_lock - Lock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, take lock
+ */
+#define drm_gpusvm_notifier_lock(gpusvm__)	\
+	down_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * drm_gpusvm_notifier_unlock - Unlock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, drop lock
+ */
+#define drm_gpusvm_notifier_unlock(gpusvm__)	\
+	up_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * __drm_gpusvm_range_next - Get the next GPU SVM range in the list
+ * @range: a pointer to the current GPU SVM range
+ *
+ * Return: A pointer to the next drm_gpusvm_range if available, or NULL if the
+ *         current range is the last one or if the input range is NULL.
+ */
+static inline struct drm_gpusvm_range *
+__drm_gpusvm_range_next(struct drm_gpusvm_range *range)
+{
+	if (range && !list_is_last(&range->rb.entry,
+				   &range->notifier->range_list))
+		return list_next_entry(range, rb.entry);
+
+	return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_range - Iterate over GPU SVM ranges in a notifier
+ * @range__: Iterator variable for the ranges. If set, it indicates the start of
+ *	     the iterator. If NULL, call drm_gpusvm_range_find() to get the range.
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier. It is safe
+ * to use while holding the driver SVM lock or the notifier lock.
+ */
+#define drm_gpusvm_for_each_range(range__, notifier__, start__, end__)	\
+	for ((range__) = (range__) ?:					\
+	     drm_gpusvm_range_find((notifier__), (start__), (end__));	\
+	     (range__) && (range__->va.start < (end__));		\
+	     (range__) = __drm_gpusvm_range_next(range__))
+
+/**
+ * drm_gpusvm_range_set_unmapped - Mark a GPU SVM range as unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets the partial_unmap flag
+ * if the range partially falls within the provided MMU notifier range.
+ */
+static inline void
+drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,
+			      const struct mmu_notifier_range *mmu_range)
+{
+	lockdep_assert_held_write(&range->gpusvm->notifier_lock);
+
+	range->flags.unmapped = true;
+	if (range->va.start < mmu_range->start ||
+	    range->va.end > mmu_range->end)
+		range->flags.partial_unmap = true;
+}
+
+#endif /* __DRM_GPUSVM_H__ */
Signature · 1 line
2.34.1
Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
Signature · 3 lines
Simona Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#15 lore raw

Re: [RFC PATCH 05/28] drm/gpusvm: Add support for GPU Shared Virtual Memory

In reply to Daniel Vetter
Quoted 18 lines · Daniel Vetter

drivers/gpu/drm/xe/drm_gpusvm.c line 1353

Quoted 11 lines · Matthew Brost
drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+		if (!ctx->mmap_locked) {
+			/*
+			 * XXX: HMM locking document indicates only a read-lock
+			 * is required but there apears to be a window between
+			 * the MMU_NOTIFY_MIGRATE event triggered in a CPU fault
+			 * via migrate_vma_setup and the pages actually moving
+			 * in migrate_vma_finalize in which this code can grab
+			 * garbage pages. Grabbing the write-lock if the range
+			 * is attached to vram appears to protect against this
+			 * race.
+			 */

This one is really scary, since it means the entire migrate pte trickery is essentially completely busted. Grabbing the mmap write lock just means you block out pretty much everything interesting from concurrently happening.

Quoted 4 lines

My gut feeling says we need to figure out what's happening here, because this looks a bit too fundamental to me.
-Sima

Sima,

I’ve already replied to this.

We’ve discussed the mmap write hack extensively, so I’m not quite sure where to put this. The reply chain is quickly becoming a mess. However, I’ve looked into this and collected some data points based on your feedback.

I’ve pushed a branch [1] with the updated code.

The first new commit [2] removes the mmap write lock hack and addresses an issue related to VRAM migrations, which couldn’t collect all VRAM pages without this hack.

With this commit [2], xe_exec_system_allocator --r twice*race* fails quite regularly, perhaps 25% of the time. This test is a single-thread/process test that races CPU and GPU faults with migration.

It fails with the following dmesg:

[   68.473007] WARNING: CPU: 12 PID: 1643 at drivers/gpu/drm/xe/drm_gpusvm.c:1407 drm_gpusvm_range_get_pages+0xbda/0x1480 [xe]
...
[   68.473836] xe 0000:03:00.0: [drm:pf_queue_work_func [xe]] Fault response: Unsuccessful -95
[   68.474024] xe 0000:03:00.0: [drm:xe_guc_exec_queue_memory_cat_error_handler [xe]] GT1: Engine memory cat error: engine_class=vecs, logical_mask: 0x2, guc_id=0
[   68.474163] xe 0000:03:00.0: [drm] exec queue reset detected
[   68.474696] xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vecs, logical_mask: 0x2, guc_id=0

This means hmm_range_fault collects a mix of SRAM and VRAM pages, which my design aims to avoid. Perhaps allowing a mix of SRAM and VRAM pages in my design might work, but I highly doubt it based on AMD's range->migration_mutex and my inspection of the migration layer. Allowing mixed mappings would introduce significant complexity, so I’d prefer to avoid this if possible. Additionally, allowing mixed mappings would eliminate the use of huge GPU pages when race like this occurs.

I also implemented a retry loop to see if the system stabilizes with either only SRAM or VRAM pages. Unfortunately, it results in a continuous loop of drm_gpusvm_range_get_pages / hmm_range_fault until the test case kills the MM due to a timeout.

Next, I added a lock similar to AMD's range->migration_lock, but using an rwsem [3]. The semantics are to allow read access for CPU access and write access for GPU access, thus enabling parallel CPU page faults for the same range which matching existing core semantics. This provides finer granularity compared to using the mmap write lock; it only disallows CPU and GPU servicing in parallel for a given range, rather than the entire MM. It also aligns with AMD’s approach. I haven’t checked Nvidia’s approach wrt this locking but can do so if you think it would be helpful.

Matt

[1] https://gitlab.freedesktop.org/mbrost/xe-kernel-driver-svm-post-8-27-24/-/commits/mmap_write_lock
[2] https://gitlab.freedesktop.org/mbrost/xe-kernel-driver-svm-post-8-27-24/-/commit/6cf67d98c719ffbb4ac6124a7cb81d797a5bad9f
[3] https://gitlab.freedesktop.org/mbrost/xe-kernel-driver-svm-post-8-27-24/-/commit/2b62075d193265b2c1634ecfd0497dffd2e18c13

Quoted 1,251 lines
Quoted 1,246 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +821 -0
+			if (vram_pages)
+				mmap_write_lock(mm);
+			else
+				mmap_read_lock(mm);
+		}
+		err = hmm_range_fault(&hmm_range);
+		if (!ctx->mmap_locked) {
+			if (vram_pages)
+				mmap_write_unlock(mm);
+			else
+				mmap_read_unlock(mm);
+		}
+
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
+			continue;
+		}
+		break;
+	}
+	if (!ctx->mmap_locked)
+		mmput(mm);
+	if (err)
+		goto err_free;
+
+	pages = (struct page **)pfns;
+
+	if (ctx->prefault) {
+		range->pages = pages;
+		goto set_seqno;
+	}
+
+map_pages:
+	if (is_device_private_page(hmm_pfn_to_page(pfns[0]))) {
+		WARN_ON_ONCE(!range->vram_allocation);
+
+		for (i = 0; i < npages; ++i) {
+			pages[i] = hmm_pfn_to_page(pfns[i]);
+
+			if (WARN_ON_ONCE(!is_device_private_page(pages[i]))) {
+				err = -EOPNOTSUPP;
+				goto err_free;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->flags.has_vram_pages = true;
+		range->pages = pages;
+		if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm, range);
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	} else {
+		dma_addr_t *dma_addr = (dma_addr_t *)pfns;
+
+		for_each_dma_page(i, j, npages, order) {
+			if (WARN_ON_ONCE(i && order !=
+					 hmm_pfn_to_map_order(pfns[i]))) {
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+			order = hmm_pfn_to_map_order(pfns[i]);
+
+			pages[j] = hmm_pfn_to_page(pfns[i]);
+			if (WARN_ON_ONCE(is_zone_device_page(pages[j]))) {
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+
+			set_page_dirty_lock(pages[j]);
+			mark_page_accessed(pages[j]);
+
+			dma_addr[j] = dma_map_page(gpusvm->drm->dev,
+						   pages[j], 0,
+						   PAGE_SIZE << order,
+						   DMA_BIDIRECTIONAL);
+			if (dma_mapping_error(gpusvm->drm->dev, dma_addr[j])) {
+				err = -EFAULT;
+				npages = i;
+				goto err_unmap;
+			}
+		}
+
+		/* Huge pages, reduce memory footprint */
+		if (order) {
+			dma_addr = kmalloc_array(j, sizeof(*dma_addr),
+						 GFP_KERNEL);
+			if (dma_addr) {
+				for (i = 0; i < j; ++i)
+					dma_addr[i] = (dma_addr_t)pfns[i];
+				kvfree(pfns);
+				kfree_mapping = true;
+			} else {
+				dma_addr = (dma_addr_t *)pfns;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->order = order;
+		range->flags.kfree_mapping = kfree_mapping;
+		range->flags.has_dma_mapping = true;
+		range->dma_addr = dma_addr;
+		range->vram_allocation = NULL;
+		if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm, range);
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	}
+
+	if (err == -EAGAIN)
+		goto retry;
+set_seqno:
+	range->notifier_seq = hmm_range.notifier_seq;
+
+	return 0;
+
+err_unmap:
+	for_each_dma_page(i, j, npages, order)
+		dma_unmap_page(gpusvm->drm->dev,
+			       (dma_addr_t)pfns[j],
+			       PAGE_SIZE << order, DMA_BIDIRECTIONAL);
+err_free:
+	if (alloc_pfns)
+		kvfree(pfns);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_range_unmap_pages - Unmap pages associated with a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed that gpusvm->notifier_lock is held in write mode; if it
+ * is clear, it acquires gpusvm->notifier_lock in read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops->invalidate for IOMMU
+ * security model.
+ */
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx)
+{
+	if (ctx->in_notifier)
+		lockdep_assert_held_write(&gpusvm->notifier_lock);
+	else
+		drm_gpusvm_notifier_lock(gpusvm);
+
+	__drm_gpusvm_range_unmap_pages(gpusvm, range);
+
+	if (!ctx->in_notifier)
+		drm_gpusvm_notifier_unlock(gpusvm);
+}
+
+/**
+ * drm_gpusvm_migration_put_page - Put a migration page
+ * @page: Pointer to the page to put
+ *
+ * This function unlocks and puts a page.
+ */
+static void drm_gpusvm_migration_put_page(struct page *page)
+{
+	unlock_page(page);
+	put_page(page);
+}
+
+/**
+ * drm_gpusvm_migration_put_pages - Put migration pages
+ * @npages: Number of pages
+ * @migrate_pfn: Array of migrate page frame numbers
+ *
+ * This function puts an array of pages.
+ */
+static void drm_gpusvm_migration_put_pages(unsigned long npages,
+					   unsigned long *migrate_pfn)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!migrate_pfn[i])
+			continue;
+
+		drm_gpusvm_migration_put_page(migrate_pfn_to_page(migrate_pfn[i]));
+		migrate_pfn[i] = 0;
+	}
+}
+
+/**
+ * drm_gpusvm_get_vram_page - Get a reference to a VRAM page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU SVM zone
+ * device data and initializes it for zone device usage.
+ */
+static void drm_gpusvm_get_vram_page(struct page *page,
+				     struct drm_gpusvm_zdd *zdd)
+{
+	page->zone_device_data = drm_gpusvm_zdd_get(zdd);
+	zone_device_page_init(page);
+}
+
+/**
+ * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Return: 0 on success, -EFAULT if an error occurs during mapping.
+ */
+static int drm_gpusvm_migrate_map_pages(struct device *dev,
+					dma_addr_t *dma_addr,
+					long unsigned int *migrate_pfn,
+					unsigned long npages,
+					enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = migrate_pfn_to_page(migrate_pfn[i]);
+
+		if (!page)
+			continue;
+
+		if (WARN_ON_ONCE(is_zone_device_page(page)))
+			return -EFAULT;
+
+		dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE, dir);
+		if (dma_mapping_error(dev, dma_addr[i]))
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */
+static void drm_gpusvm_migrate_unmap_pages(struct device *dev,
+					   dma_addr_t *dma_addr,
+					   unsigned long npages,
+					   enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!dma_addr[i] || dma_mapping_error(dev, dma_addr[i]))
+			continue;
+
+		dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir);
+	}
+}
+
+/**
+ * drm_gpusvm_migrate_to_vram - Migrate GPU SVM range to VRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *                   failure of this function.
+ * @vram_allocation: Driver-private pointer to the VRAM allocation. The caller
+ *                   should hold a reference to the VRAM allocation, which
+ *                   should be dropped via ops->vram_allocation or upon the
+ *                   failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to VRAM. It performs the
+ * necessary setup and invokes the driver-specific operations for migration to
+ * VRAM. Upon successful return, @vram_allocation can safely reference @range
+ * until ops->vram_release is called which only upon successful return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct migrate_vma migrate = {
+		.start		= start,
+		.end		= end,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_SYSTEM,
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	unsigned long i, npages = npages_in_range(start, end);
+	struct vm_area_struct *vas;
+	struct drm_gpusvm_zdd *zdd = NULL;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int err;
+
+	if (!range->flags.migrate_vram)
+		return -EINVAL;
+
+	if (!gpusvm->ops->populate_vram_pfn || !gpusvm->ops->copy_to_vram ||
+	    !gpusvm->ops->copy_to_sram)
+		return -EOPNOTSUPP;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		mmap_write_lock(mm);
+	}
+
+	mmap_assert_locked(mm);
+
+	vas = vma_lookup(mm, start);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end > vas->vm_end || start < vas->vm_start) {
+		err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	if (!vma_is_anonymous(vas)) {
+		err = -EBUSY;
+		goto err_mmunlock;
+	}
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_mmunlock;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+	zdd = drm_gpusvm_zdd_alloc(range);
+	if (!zdd) {
+		err = -ENOMEM;
+		goto err_free;
+	}
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/*
+	 * FIXME: Below cases, !migrate.cpages and migrate.cpages != npages, not
+	 * always an error. Need to revisit possible cases and how to handle. We
+	 * could prefault on migrate.cpages != npages via hmm_range_fault.
+	 */
+
+	if (!migrate.cpages) {
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	if (migrate.cpages != npages) {
+		err = -EBUSY;
+		goto err_finalize;
+	}
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm, vram_allocation, npages,
+					     migrate.dst);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   migrate.src, npages, DMA_TO_DEVICE);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = pfn_to_page(migrate.dst[i]);
+
+		pages[i] = page;
+		migrate.dst[i] = migrate_pfn(migrate.dst[i]);
+		drm_gpusvm_get_vram_page(page, zdd);
+	}
+
+	err = gpusvm->ops->copy_to_vram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+	/* Upon success bind vram allocation to range and zdd */
+	range->vram_allocation = vram_allocation;
+	WRITE_ONCE(zdd->vram_allocation, vram_allocation);	/* Owns ref */
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_TO_DEVICE);
+err_free:
+	if (zdd)
+		drm_gpusvm_zdd_put(zdd);
+	kvfree(buf);
+err_mmunlock:
+	if (!ctx->mmap_locked) {
+		mmap_write_unlock(mm);
+		mmput(mm);
+	}
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_populate_sram_pfn - Populate SRAM PFNs for a VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @npages: Number of pages to populate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the SRAM migrate page frame numbers (PFNs) for the
+ * specified VM area structure. It allocates and locks pages in the VM area for
+ * SRAM usage. If vas is non-NULL use alloc_page_vma for allocation, if NULL use
+ * alloc_page for allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_sram_pfn(struct vm_area_struct *vas,
+						unsigned long npages,
+						unsigned long *src_mpfn,
+						unsigned long *mpfn, u64 addr)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i, addr += PAGE_SIZE) {
+		struct page *page;
+
+		if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE))
+			continue;
+
+		if (vas)
+			page = alloc_page_vma(GFP_HIGHUSER, vas, addr);
+		else
+			page = alloc_page(GFP_HIGHUSER);
+
+		if (!page)
+			return -ENOMEM;
+
+		lock_page(page);
+		mpfn[i] = migrate_pfn(page_to_pfn(page));
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_evict_to_sram - Evict GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * Similar to __drm_gpusvm_migrate_to_sram but does not require mmap lock and
+ * migration done via migrate_device_* functions. Fallback path as it is
+ * preferred to issue migrations with mmap lock.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_evict_to_sram(struct drm_gpusvm *gpusvm,
+				    struct drm_gpusvm_range *range)
+{
+	unsigned long npages;
+	struct page **pages;
+	unsigned long *src, *dst;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	npages = npages_in_range(range->va.start, range->va.end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	src = buf;
+	dst = buf + (sizeof(*src) * npages);
+	dma_addr = buf + (2 * sizeof(*src) * npages);
+	pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) * npages;
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm, range->vram_allocation,
+					     npages, src);
+	if (err)
+		goto err_free;
+
+	err = migrate_device_vma_range(gpusvm->mm,
+				       gpusvm->device_private_page_owner, src,
+				       npages, range->va.start);
+	if (err)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(NULL, npages, src, dst, 0);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   dst, npages, DMA_BIDIRECTIONAL);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, dst);
+	migrate_device_pages(src, dst, npages);
+	migrate_device_finalize(src, dst, npages);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+
+	return err;
+}
+
+/**
+ * __drm_gpusvm_migrate_to_sram - Migrate GPU SVM range to SRAM (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @vas: Pointer to the VM area structure
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @start: Start address of the migration range
+ * @end: End address of the migration range
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to SRAM. It sets up the migration, populates + dma maps SRAM PFNs, and
+ * invokes the driver-specific operations for migration to SRAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int __drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+					struct vm_area_struct *vas,
+					struct page *page,
+					u64 start, u64 end)
+{
+	struct migrate_vma migrate = {
+		.vma		= vas,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_DEVICE_PRIVATE,
+		.fault_page	= page,
+	};
+	unsigned long npages;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	mmap_assert_locked(gpusvm->mm);
+
+	/* Corner where VMA area struct has been partially unmapped */
+	if (start < vas->vm_start)
+		start = vas->vm_start;
+	if (end > vas->vm_end)
+		end = vas->vm_end;
+
+	migrate.start = start;
+	migrate.end = end;
+	npages = npages_in_range(start, end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/* Raced with another CPU fault, nothing to do */
+	if (!migrate.cpages)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(vas, npages,
+						   migrate.src, migrate.dst,
+						   start);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   migrate.dst, npages,
+					   DMA_BIDIRECTIONAL);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(migrate.src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+	mmap_assert_locked(gpusvm->mm);
+
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_to_sram - Migrate (evict) GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function initiates the migration of the specified GPU SVM range to
+ * SRAM. It performs necessary checks and invokes the internal migration
+ * function for actual migration.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	int err;
+	bool retry = false;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		if (ctx->trylock_mmap) {
+			if (!mmap_read_trylock(mm))  {
+				err = drm_gpusvm_evict_to_sram(gpusvm, range);
+				goto err_mmput;
+			}
+		} else {
+			mmap_read_lock(mm);
+		}
+	}
+
+	mmap_assert_locked(mm);
+
+	/*
+	 * Loop required to find all VMA area structs for the corner case when
+	 * VRAM backing has been partially unmapped from MM's address space.
+	 */
+again:
+	vas = find_vma(mm, start);
+	if (!vas) {
+		if (!retry)
+			err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end <= vas->vm_start || start >= vas->vm_end) {
+		if (!retry)
+			err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	err = __drm_gpusvm_migrate_to_sram(gpusvm, vas, NULL, start, end);
+	if (err)
+		goto err_mmunlock;
+
+	if (vas->vm_end < end) {
+		retry = true;
+		start = vas->vm_end;
+		goto again;
+	}
+
+	if (!ctx->mmap_locked) {
+		mmap_read_unlock(mm);
+		/*
+		 * Using mmput_async as this function can be called while
+		 * holding a dma-resv lock, and a final put can grab the mmap
+		 * lock, causing a lock inversion.
+		 */
+		mmput_async(mm);
+	}
+
+	return 0;
+
+err_mmunlock:
+	if (!ctx->mmap_locked)
+		mmap_read_unlock(mm);
+err_mmput:
+	if (!ctx->mmap_locked)
+		mmput_async(mm);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_page_free - Put GPU SVM zone device data associated with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device data
+ * associated with a page when it is being released.
+ */
+static void drm_gpusvm_page_free(struct page *page)
+{
+	drm_gpusvm_zdd_put(page->zone_device_data);
+}
+
+/**
+ * drm_gpusvm_migrate_to_ram - Migrate GPU SVM range to RAM (page fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ * VM_FAULT_SIGBUS on failure, 0 on success.
+ */
+static vm_fault_t drm_gpusvm_migrate_to_ram(struct vm_fault *vmf)
+{
+	struct drm_gpusvm_zdd *zdd = vmf->page->zone_device_data;
+	int err;
+
+	err = __drm_gpusvm_migrate_to_sram(zdd->range->gpusvm,
+					   vmf->vma, vmf->page,
+					   zdd->range->va.start,
+					   zdd->range->va.end);
+
+	return err ? VM_FAULT_SIGBUS : 0;
+}
+
+/**
+ * drm_gpusvm_pagemap_ops - Device page map operations for GPU SVM
+ */
+static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = {
+	.page_free = drm_gpusvm_page_free,
+	.migrate_to_ram = drm_gpusvm_migrate_to_ram,
+};
+
+/**
+ * drm_gpusvm_pagemap_ops_get - Retrieve GPU SVM device page map operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void)
+{
+	return &drm_gpusvm_pagemap_ops;
+}
+
+/**
+ * drm_gpusvm_has_mapping - Check if GPU SVM has mapping for the given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end)
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	drm_gpusvm_for_each_notifier(notifier, gpusvm, start, end) {
+		struct drm_gpusvm_range *range = NULL;
+
+		drm_gpusvm_for_each_range(range, notifier, start, end)
+			return true;
+	}
+
+	return false;
+}
drivers/gpu/drm/xe/drm_gpusvm.h new file +415 -0
new file mode 100644
index 000000000000..0ea70f8534a8
@@ -0,0 +1,415 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef __DRM_GPUSVM_H__
+#define __DRM_GPUSVM_H__
+
+#include <linux/kref.h>
+#include <linux/mmu_notifier.h>
+#include <linux/workqueue.h>
+
+struct dev_pagemap_ops;
+struct drm_device;
+struct drm_gpusvm;
+struct drm_gpusvm_notifier;
+struct drm_gpusvm_ops;
+struct drm_gpusvm_range;
+
+/**
+ * struct drm_gpusvm_ops - Operations structure for GPU SVM
+ *
+ * This structure defines the operations for GPU Shared Virtual Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * perform operations such as migration between VRAM and system RAM.
+ */
+struct drm_gpusvm_ops {
+	/**
+	 * @notifier_alloc: Allocate a GPU SVM notifier (optional)
+	 *
+	 * This function shall allocate a GPU SVM notifier.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM notifier on success, NULL on failure.
+	 */
+	struct drm_gpusvm_notifier *(*notifier_alloc)(void);
+
+	/**
+	 * @notifier_free: Free a GPU SVM notifier (optional)
+	 * @notifier: Pointer to the GPU SVM notifier to be freed
+	 *
+	 * This function shall free a GPU SVM notifier.
+	 */
+	void (*notifier_free)(struct drm_gpusvm_notifier *notifier);
+
+	/**
+	 * @range_alloc: Allocate a GPU SVM range (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 *
+	 * This function shall allocate a GPU SVM range.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM range on success, NULL on failure.
+	 */
+	struct drm_gpusvm_range *(*range_alloc)(struct drm_gpusvm *gpusvm);
+
+	/**
+	 * @range_free: Free a GPU SVM range (optional)
+	 * @range: Pointer to the GPU SVM range to be freed
+	 *
+	 * This function shall free a GPU SVM range.
+	 */
+	void (*range_free)(struct drm_gpusvm_range *range);
+
+	/**
+	 * @vram_release: Release VRAM allocation (optional)
+	 * @vram_allocation: Driver-private pointer to the VRAM allocation
+	 *
+	 * This function shall release VRAM allocation and expects to drop a
+	 * reference to VRAM allocation.
+	 */
+	void (*vram_release)(void *vram_allocation);
+
+	/**
+	 * @populate_vram_pfn: Populate VRAM PFN (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @vram_allocation: Driver-private pointer to the VRAM allocation
+	 * @npages: Number of pages to populate
+	 * @pfn: Array of page frame numbers to populate
+	 *
+	 * This function shall populate VRAM page frame numbers (PFN).
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*populate_vram_pfn)(struct drm_gpusvm *gpusvm,
+				 void *vram_allocation,
+				 unsigned long npages,
+				 unsigned long *pfn);
+
+	/**
+	 * @copy_to_vram: Copy to VRAM (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (destination)
+	 * @dma_addr: Pointer to array of DMA addresses (source)
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to VRAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_vram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @copy_to_sram: Copy to system RAM (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (source)
+	 * @dma_addr: Pointer to array of DMA addresses (destination)
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to system RAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_sram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @invalidate: Invalidate GPU SVM notifier (required)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @notifier: Pointer to the GPU SVM notifier
+	 * @mmu_range: Pointer to the mmu_notifier_range structure
+	 *
+	 * This function shall invalidate the GPU page tables. It can safely
+	 * walk the notifier range RB tree/list in this function. Called while
+	 * holding the notifier lock.
+	 */
+	void (*invalidate)(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_notifier *notifier,
+			   const struct mmu_notifier_range *mmu_range);
+};
+
+/**
+ * struct drm_gpusvm_notifier - Structure representing a GPU SVM notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @interval: Interval for the notifier
+ * @rb: Red-black tree node for the parent GPU SVM structure notifier tree
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ *              interval tree. This is useful to keep iterating ranges while
+ *              doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier has been
+ *                 removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */
+struct drm_gpusvm_notifier {
+	struct drm_gpusvm *gpusvm;
+	struct mmu_interval_notifier notifier;
+	struct {
+		u64 start;
+		u64 end;
+	} interval;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct rb_root_cached root;
+	struct list_head range_list;
+	struct {
+		u32 removed : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm_range - Structure representing a GPU SVM range
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier
+ * @refcount: Reference count for the range
+ * @rb: Red-black tree node for the parent GPU SVM notifier structure range tree
+ * @va: Virtual address range
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @pages: Pointer to the array of pages (if backing store is in VRAM)
+ * @dma_addr: DMA address array (if backing store is SRAM and DMA mapped)
+ * @vram_allocation: Driver-private pointer to the VRAM allocation
+ * @order: Order of dma mapping. i.e. PAGE_SIZE << order is mapping size
+ * @flags.migrate_vram: Flag indicating whether the range can be migrated to VRAM
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_vram_pages: Flag indicating if the range has vram pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA mapping
+ * @flags.kfree_mapping: Flag indicating @dma_addr is a compact allocation based
+ *                       on @order which releases via kfree
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */
+struct drm_gpusvm_range {
+	struct drm_gpusvm *gpusvm;
+	struct drm_gpusvm_notifier *notifier;
+	struct kref refcount;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct {
+		u64 start;
+		u64 end;
+	} va;
+	unsigned long notifier_seq;
+	union {
+		struct page **pages;
+		dma_addr_t *dma_addr;
+	};
+	void *vram_allocation;
+	u16 order;
+	struct {
+		/* All flags below must be set upon creation */
+		u16 migrate_vram : 1;
+		/* All flags below must be set / cleared under notifier lock */
+		u16 unmapped : 1;
+		u16 partial_unmap : 1;
+		u16 has_vram_pages : 1;
+		u16 has_dma_mapping : 1;
+		u16 kfree_mapping : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm - GPU SVM structure
+ *
+ * @name: Name of the GPU SVM
+ * @drm: Pointer to the DRM device structure
+ * @mm: Pointer to the mm_struct for the address space
+ * @device_private_page_owner: Device private pages owner
+ * @mm_start: Start address of GPU SVM
+ * @mm_range: Range of the GPU SVM
+ * @notifier_size: Size of individual notifiers
+ * @ops: Pointer to the operations structure for GPU SVM
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range allocation.
+ *               Entries should be powers of 2 in descending order.
+ * @num_chunks: Number of chunks
+ * @notifier_lock: Read-write semaphore for protecting notifier operations
+ * @zdd_wq: Workqueue for deferred work on zdd destruction
+ * @root: Cached root node of the Red-Black tree containing GPU SVM notifiers
+ * @notifier_list: list head containing of notifiers in the same order they
+ *                 appear in interval tree. This is useful to keep iterating
+ *                 notifiers while doing modifications to RB tree.
+ *
+ * This structure represents a GPU SVM (Shared Virtual Memory) used for tracking
+ * memory ranges mapped in a DRM (Direct Rendering Manager) device.
+ *
+ * No reference counting is provided, as this is expected to be embedded in the
+ * driver VM structure along with the struct drm_gpuvm, which handles reference
+ * counting.
+ */
+struct drm_gpusvm {
+	const char *name;
+	struct drm_device *drm;
+	struct mm_struct *mm;
+	void *device_private_page_owner;
+	u64 mm_start;
+	u64 mm_range;
+	u64 notifier_size;
+	const struct drm_gpusvm_ops *ops;
+	const u64 *chunk_sizes;
+	int num_chunks;
+	struct rw_semaphore notifier_lock;
+	struct workqueue_struct *zdd_wq;
+	struct rb_root_cached root;
+	struct list_head notifier_list;
+};
+
+/**
+ * struct drm_gpusvm_ctx - DRM GPU SVM context
+ *
+ * @mmap_locked: mmap lock is locked
+ * @trylock_mmap: trylock mmap lock, used to avoid locking inversions
+ *                (e.g.dma-revs -> mmap lock)
+ * @in_notifier: entering from a MMU notifier
+ * @read_only: operating on read-only memory
+ * @vram_possible: possible to use VRAM
+ * @prefault: prefault pages
+ *
+ * Context that is DRM GPUSVM is operating in (i.e. user arguments).
+ */
+struct drm_gpusvm_ctx {
+	u32 mmap_locked :1;
+	u32 trylock_mmap :1;
+	u32 in_notifier :1;
+	u32 read_only :1;
+	u32 vram_possible :1;
+	u32 prefault :1;
+};
+
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void *device_private_page_owner,
+		    u64 mm_start, u64 mm_range, u64 notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const u64 *chunk_sizes, int num_chunks);
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm);
+void drm_gpusvm_free(struct drm_gpusvm *gpusvm);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, u64 fault_addr,
+				u64 gpuva_start, u64 gpuva_end,
+				const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range);
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range);
+
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range);
+
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx);
+
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx);
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void);
+
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, u64 start, u64 end);
+
+/**
+ * drm_gpusvm_notifier_lock - Lock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, take lock
+ */
+#define drm_gpusvm_notifier_lock(gpusvm__)	\
+	down_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * drm_gpusvm_notifier_unlock - Unlock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, drop lock
+ */
+#define drm_gpusvm_notifier_unlock(gpusvm__)	\
+	up_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * __drm_gpusvm_range_next - Get the next GPU SVM range in the list
+ * @range: a pointer to the current GPU SVM range
+ *
+ * Return: A pointer to the next drm_gpusvm_range if available, or NULL if the
+ *         current range is the last one or if the input range is NULL.
+ */
+static inline struct drm_gpusvm_range *
+__drm_gpusvm_range_next(struct drm_gpusvm_range *range)
+{
+	if (range && !list_is_last(&range->rb.entry,
+				   &range->notifier->range_list))
+		return list_next_entry(range, rb.entry);
+
+	return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_range - Iterate over GPU SVM ranges in a notifier
+ * @range__: Iterator variable for the ranges. If set, it indicates the start of
+ *	     the iterator. If NULL, call drm_gpusvm_range_find() to get the range.
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier. It is safe
+ * to use while holding the driver SVM lock or the notifier lock.
+ */
+#define drm_gpusvm_for_each_range(range__, notifier__, start__, end__)	\
+	for ((range__) = (range__) ?:					\
+	     drm_gpusvm_range_find((notifier__), (start__), (end__));	\
+	     (range__) && (range__->va.start < (end__));		\
+	     (range__) = __drm_gpusvm_range_next(range__))
+
+/**
+ * drm_gpusvm_range_set_unmapped - Mark a GPU SVM range as unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets the partial_unmap flag
+ * if the range partially falls within the provided MMU notifier range.
+ */
+static inline void
+drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,
+			      const struct mmu_notifier_range *mmu_range)
+{
+	lockdep_assert_held_write(&range->gpusvm->notifier_lock);
+
+	range->flags.unmapped = true;
+	if (range->va.start < mmu_range->start ||
+	    range->va.end > mmu_range->end)
+		range->flags.partial_unmap = true;
+}
+
+#endif /* __DRM_GPUSVM_H__ */
Signature · 1 line
2.34.1
Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#16 lore raw

Re: [RFC PATCH 05/28] drm/gpusvm: Add support for GPU Shared Virtual Memory

Quoted 52 lines · Matthew Brost
Quoted 18 lines · Daniel Vetter

drivers/gpu/drm/xe/drm_gpusvm.c line 1353

Quoted 11 lines · Matthew Brost
drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+		if (!ctx->mmap_locked) {
+			/*
+			 * XXX: HMM locking document indicates only a read-lock
+			 * is required but there apears to be a window between
+			 * the MMU_NOTIFY_MIGRATE event triggered in a CPU fault
+			 * via migrate_vma_setup and the pages actually moving
+			 * in migrate_vma_finalize in which this code can grab
+			 * garbage pages. Grabbing the write-lock if the range
+			 * is attached to vram appears to protect against this
+			 * race.
+			 */

This one is really scary, since it means the entire migrate pte trickery is essentially completely busted. Grabbing the mmap write lock just means you block out pretty much everything interesting from concurrently happening.

Quoted 4 lines

My gut feeling says we need to figure out what's happening here, because this looks a bit too fundamental to me.
-Sima

Sima,

I’ve already replied to this.

We’ve discussed the mmap write hack extensively, so I’m not quite sure where to put this. The reply chain is quickly becoming a mess. However, I’ve looked into this and collected some data points based on your feedback.

I’ve pushed a branch [1] with the updated code.

The first new commit [2] removes the mmap write lock hack and addresses an issue related to VRAM migrations, which couldn’t collect all VRAM pages without this hack.

With this commit [2], xe_exec_system_allocator --r twice*race* fails quite regularly, perhaps 25% of the time. This test is a single-thread/process test that races CPU and GPU faults with migration.

It fails with the following dmesg:

[   68.473007] WARNING: CPU: 12 PID: 1643 at drivers/gpu/drm/xe/drm_gpusvm.c:1407 drm_gpusvm_range_get_pages+0xbda/0x1480 [xe]
...
[   68.473836] xe 0000:03:00.0: [drm:pf_queue_work_func [xe]] Fault response: Unsuccessful -95
[   68.474024] xe 0000:03:00.0: [drm:xe_guc_exec_queue_memory_cat_error_handler [xe]] GT1: Engine memory cat error: engine_class=vecs, logical_mask: 0x2, guc_id=0
[   68.474163] xe 0000:03:00.0: [drm] exec queue reset detected
[   68.474696] xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vecs, logical_mask: 0x2, guc_id=0
Quoted 7 lines

This means hmm_range_fault collects a mix of SRAM and VRAM pages, which my design aims to avoid. Perhaps allowing a mix of SRAM and VRAM pages in my design might work, but I highly doubt it based on AMD's range->migration_mutex and my inspection of the migration layer. Allowing mixed mappings would introduce significant complexity, so I’d prefer to avoid this if possible. Additionally, allowing mixed mappings would eliminate the use of huge GPU pages when race like this occurs.

Ah, if the issue is just that you get a mix of sram and vram pages from hmm_range_fault, then I think that answers all my questions. From the discussion we had and your comment it sounded like you're getting complete nonsense, or missing an invalidation, or something else equally scary.

Thanks a lot for these details, I'm a lot less worried now here.

Quoted 4 lines

I also implemented a retry loop to see if the system stabilizes with either only SRAM or VRAM pages. Unfortunately, it results in a continuous loop of drm_gpusvm_range_get_pages / hmm_range_fault until the test case kills the MM due to a timeout.

Yeah, the core mm makes no guarantees about forward progress for groups of pages/folios. So if we go with core mm locking rules, then you have to deal with individual pages/folios and anything bigger is just a performance optimasation that must fall back to a per-page/folio approach.

Quoted 9 lines

Next, I added a lock similar to AMD's range->migration_lock, but using an rwsem [3]. The semantics are to allow read access for CPU access and write access for GPU access, thus enabling parallel CPU page faults for the same range which matching existing core semantics. This provides finer granularity compared to using the mmap write lock; it only disallows CPU and GPU servicing in parallel for a given range, rather than the entire MM. It also aligns with AMD’s approach. I haven’t checked Nvidia’s approach wrt this locking but can do so if you think it would be helpful.

Yeah minus the entire question whether va base locking is ok, something like amd's migration_mutex should also close the race you're having here.

Cheers, Sima

Quoted 1,258 lines

Matt

[1] https://gitlab.freedesktop.org/mbrost/xe-kernel-driver-svm-post-8-27-24/-/commits/mmap_write_lock
[2] https://gitlab.freedesktop.org/mbrost/xe-kernel-driver-svm-post-8-27-24/-/commit/6cf67d98c719ffbb4ac6124a7cb81d797a5bad9f
[3] https://gitlab.freedesktop.org/mbrost/xe-kernel-driver-svm-post-8-27-24/-/commit/2b62075d193265b2c1634ecfd0497dffd2e18c13

Quoted 1,251 lines
Quoted 1,246 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +821 -0
+			if (vram_pages)
+				mmap_write_lock(mm);
+			else
+				mmap_read_lock(mm);
+		}
+		err = hmm_range_fault(&hmm_range);
+		if (!ctx->mmap_locked) {
+			if (vram_pages)
+				mmap_write_unlock(mm);
+			else
+				mmap_read_unlock(mm);
+		}
+
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
+			continue;
+		}
+		break;
+	}
+	if (!ctx->mmap_locked)
+		mmput(mm);
+	if (err)
+		goto err_free;
+
+	pages = (struct page **)pfns;
+
+	if (ctx->prefault) {
+		range->pages = pages;
+		goto set_seqno;
+	}
+
+map_pages:
+	if (is_device_private_page(hmm_pfn_to_page(pfns[0]))) {
+		WARN_ON_ONCE(!range->vram_allocation);
+
+		for (i = 0; i < npages; ++i) {
+			pages[i] = hmm_pfn_to_page(pfns[i]);
+
+			if (WARN_ON_ONCE(!is_device_private_page(pages[i]))) {
+				err = -EOPNOTSUPP;
+				goto err_free;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->flags.has_vram_pages = true;
+		range->pages = pages;
+		if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm, range);
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	} else {
+		dma_addr_t *dma_addr = (dma_addr_t *)pfns;
+
+		for_each_dma_page(i, j, npages, order) {
+			if (WARN_ON_ONCE(i && order !=
+					 hmm_pfn_to_map_order(pfns[i]))) {
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+			order = hmm_pfn_to_map_order(pfns[i]);
+
+			pages[j] = hmm_pfn_to_page(pfns[i]);
+			if (WARN_ON_ONCE(is_zone_device_page(pages[j]))) {
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+
+			set_page_dirty_lock(pages[j]);
+			mark_page_accessed(pages[j]);
+
+			dma_addr[j] = dma_map_page(gpusvm->drm->dev,
+						   pages[j], 0,
+						   PAGE_SIZE << order,
+						   DMA_BIDIRECTIONAL);
+			if (dma_mapping_error(gpusvm->drm->dev, dma_addr[j])) {
+				err = -EFAULT;
+				npages = i;
+				goto err_unmap;
+			}
+		}
+
+		/* Huge pages, reduce memory footprint */
+		if (order) {
+			dma_addr = kmalloc_array(j, sizeof(*dma_addr),
+						 GFP_KERNEL);
+			if (dma_addr) {
+				for (i = 0; i < j; ++i)
+					dma_addr[i] = (dma_addr_t)pfns[i];
+				kvfree(pfns);
+				kfree_mapping = true;
+			} else {
+				dma_addr = (dma_addr_t *)pfns;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->order = order;
+		range->flags.kfree_mapping = kfree_mapping;
+		range->flags.has_dma_mapping = true;
+		range->dma_addr = dma_addr;
+		range->vram_allocation = NULL;
+		if (mmu_interval_read_retry(notifier, hmm_range.notifier_seq)) {
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm, range);
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	}
+
+	if (err == -EAGAIN)
+		goto retry;
+set_seqno:
+	range->notifier_seq = hmm_range.notifier_seq;
+
+	return 0;
+
+err_unmap:
+	for_each_dma_page(i, j, npages, order)
+		dma_unmap_page(gpusvm->drm->dev,
+			       (dma_addr_t)pfns[j],
+			       PAGE_SIZE << order, DMA_BIDIRECTIONAL);
+err_free:
+	if (alloc_pfns)
+		kvfree(pfns);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_range_unmap_pages - Unmap pages associated with a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed that gpusvm->notifier_lock is held in write mode; if it
+ * is clear, it acquires gpusvm->notifier_lock in read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops->invalidate for IOMMU
+ * security model.
+ */
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx)
+{
+	if (ctx->in_notifier)
+		lockdep_assert_held_write(&gpusvm->notifier_lock);
+	else
+		drm_gpusvm_notifier_lock(gpusvm);
+
+	__drm_gpusvm_range_unmap_pages(gpusvm, range);
+
+	if (!ctx->in_notifier)
+		drm_gpusvm_notifier_unlock(gpusvm);
+}
+
+/**
+ * drm_gpusvm_migration_put_page - Put a migration page
+ * @page: Pointer to the page to put
+ *
+ * This function unlocks and puts a page.
+ */
+static void drm_gpusvm_migration_put_page(struct page *page)
+{
+	unlock_page(page);
+	put_page(page);
+}
+
+/**
+ * drm_gpusvm_migration_put_pages - Put migration pages
+ * @npages: Number of pages
+ * @migrate_pfn: Array of migrate page frame numbers
+ *
+ * This function puts an array of pages.
+ */
+static void drm_gpusvm_migration_put_pages(unsigned long npages,
+					   unsigned long *migrate_pfn)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!migrate_pfn[i])
+			continue;
+
+		drm_gpusvm_migration_put_page(migrate_pfn_to_page(migrate_pfn[i]));
+		migrate_pfn[i] = 0;
+	}
+}
+
+/**
+ * drm_gpusvm_get_vram_page - Get a reference to a VRAM page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU SVM zone
+ * device data and initializes it for zone device usage.
+ */
+static void drm_gpusvm_get_vram_page(struct page *page,
+				     struct drm_gpusvm_zdd *zdd)
+{
+	page->zone_device_data = drm_gpusvm_zdd_get(zdd);
+	zone_device_page_init(page);
+}
+
+/**
+ * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Return: 0 on success, -EFAULT if an error occurs during mapping.
+ */
+static int drm_gpusvm_migrate_map_pages(struct device *dev,
+					dma_addr_t *dma_addr,
+					long unsigned int *migrate_pfn,
+					unsigned long npages,
+					enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = migrate_pfn_to_page(migrate_pfn[i]);
+
+		if (!page)
+			continue;
+
+		if (WARN_ON_ONCE(is_zone_device_page(page)))
+			return -EFAULT;
+
+		dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE, dir);
+		if (dma_mapping_error(dev, dma_addr[i]))
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */
+static void drm_gpusvm_migrate_unmap_pages(struct device *dev,
+					   dma_addr_t *dma_addr,
+					   unsigned long npages,
+					   enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!dma_addr[i] || dma_mapping_error(dev, dma_addr[i]))
+			continue;
+
+		dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir);
+	}
+}
+
+/**
+ * drm_gpusvm_migrate_to_vram - Migrate GPU SVM range to VRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *                   failure of this function.
+ * @vram_allocation: Driver-private pointer to the VRAM allocation. The caller
+ *                   should hold a reference to the VRAM allocation, which
+ *                   should be dropped via ops->vram_allocation or upon the
+ *                   failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to VRAM. It performs the
+ * necessary setup and invokes the driver-specific operations for migration to
+ * VRAM. Upon successful return, @vram_allocation can safely reference @range
+ * until ops->vram_release is called which only upon successful return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct migrate_vma migrate = {
+		.start		= start,
+		.end		= end,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_SYSTEM,
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	unsigned long i, npages = npages_in_range(start, end);
+	struct vm_area_struct *vas;
+	struct drm_gpusvm_zdd *zdd = NULL;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int err;
+
+	if (!range->flags.migrate_vram)
+		return -EINVAL;
+
+	if (!gpusvm->ops->populate_vram_pfn || !gpusvm->ops->copy_to_vram ||
+	    !gpusvm->ops->copy_to_sram)
+		return -EOPNOTSUPP;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		mmap_write_lock(mm);
+	}
+
+	mmap_assert_locked(mm);
+
+	vas = vma_lookup(mm, start);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end > vas->vm_end || start < vas->vm_start) {
+		err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	if (!vma_is_anonymous(vas)) {
+		err = -EBUSY;
+		goto err_mmunlock;
+	}
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_mmunlock;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+	zdd = drm_gpusvm_zdd_alloc(range);
+	if (!zdd) {
+		err = -ENOMEM;
+		goto err_free;
+	}
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/*
+	 * FIXME: Below cases, !migrate.cpages and migrate.cpages != npages, not
+	 * always an error. Need to revisit possible cases and how to handle. We
+	 * could prefault on migrate.cpages != npages via hmm_range_fault.
+	 */
+
+	if (!migrate.cpages) {
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	if (migrate.cpages != npages) {
+		err = -EBUSY;
+		goto err_finalize;
+	}
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm, vram_allocation, npages,
+					     migrate.dst);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   migrate.src, npages, DMA_TO_DEVICE);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = pfn_to_page(migrate.dst[i]);
+
+		pages[i] = page;
+		migrate.dst[i] = migrate_pfn(migrate.dst[i]);
+		drm_gpusvm_get_vram_page(page, zdd);
+	}
+
+	err = gpusvm->ops->copy_to_vram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+	/* Upon success bind vram allocation to range and zdd */
+	range->vram_allocation = vram_allocation;
+	WRITE_ONCE(zdd->vram_allocation, vram_allocation);	/* Owns ref */
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_TO_DEVICE);
+err_free:
+	if (zdd)
+		drm_gpusvm_zdd_put(zdd);
+	kvfree(buf);
+err_mmunlock:
+	if (!ctx->mmap_locked) {
+		mmap_write_unlock(mm);
+		mmput(mm);
+	}
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_populate_sram_pfn - Populate SRAM PFNs for a VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @npages: Number of pages to populate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the SRAM migrate page frame numbers (PFNs) for the
+ * specified VM area structure. It allocates and locks pages in the VM area for
+ * SRAM usage. If vas is non-NULL use alloc_page_vma for allocation, if NULL use
+ * alloc_page for allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_sram_pfn(struct vm_area_struct *vas,
+						unsigned long npages,
+						unsigned long *src_mpfn,
+						unsigned long *mpfn, u64 addr)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i, addr += PAGE_SIZE) {
+		struct page *page;
+
+		if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE))
+			continue;
+
+		if (vas)
+			page = alloc_page_vma(GFP_HIGHUSER, vas, addr);
+		else
+			page = alloc_page(GFP_HIGHUSER);
+
+		if (!page)
+			return -ENOMEM;
+
+		lock_page(page);
+		mpfn[i] = migrate_pfn(page_to_pfn(page));
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_evict_to_sram - Evict GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * Similar to __drm_gpusvm_migrate_to_sram but does not require mmap lock and
+ * migration done via migrate_device_* functions. Fallback path as it is
+ * preferred to issue migrations with mmap lock.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_evict_to_sram(struct drm_gpusvm *gpusvm,
+				    struct drm_gpusvm_range *range)
+{
+	unsigned long npages;
+	struct page **pages;
+	unsigned long *src, *dst;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	npages = npages_in_range(range->va.start, range->va.end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	src = buf;
+	dst = buf + (sizeof(*src) * npages);
+	dma_addr = buf + (2 * sizeof(*src) * npages);
+	pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) * npages;
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm, range->vram_allocation,
+					     npages, src);
+	if (err)
+		goto err_free;
+
+	err = migrate_device_vma_range(gpusvm->mm,
+				       gpusvm->device_private_page_owner, src,
+				       npages, range->va.start);
+	if (err)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(NULL, npages, src, dst, 0);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   dst, npages, DMA_BIDIRECTIONAL);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, dst);
+	migrate_device_pages(src, dst, npages);
+	migrate_device_finalize(src, dst, npages);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+
+	return err;
+}
+
+/**
+ * __drm_gpusvm_migrate_to_sram - Migrate GPU SVM range to SRAM (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @vas: Pointer to the VM area structure
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @start: Start address of the migration range
+ * @end: End address of the migration range
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to SRAM. It sets up the migration, populates + dma maps SRAM PFNs, and
+ * invokes the driver-specific operations for migration to SRAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int __drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+					struct vm_area_struct *vas,
+					struct page *page,
+					u64 start, u64 end)
+{
+	struct migrate_vma migrate = {
+		.vma		= vas,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_DEVICE_PRIVATE,
+		.fault_page	= page,
+	};
+	unsigned long npages;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	mmap_assert_locked(gpusvm->mm);
+
+	/* Corner where VMA area struct has been partially unmapped */
+	if (start < vas->vm_start)
+		start = vas->vm_start;
+	if (end > vas->vm_end)
+		end = vas->vm_end;
+
+	migrate.start = start;
+	migrate.end = end;
+	npages = npages_in_range(start, end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) + sizeof(*dma_addr) +
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr)) * npages;
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/* Raced with another CPU fault, nothing to do */
+	if (!migrate.cpages)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(vas, npages,
+						   migrate.src, migrate.dst,
+						   start);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev, dma_addr,
+					   migrate.dst, npages,
+					   DMA_BIDIRECTIONAL);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(migrate.src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr, npages);
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr, npages,
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+	mmap_assert_locked(gpusvm->mm);
+
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_to_sram - Migrate (evict) GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function initiates the migration of the specified GPU SVM range to
+ * SRAM. It performs necessary checks and invokes the internal migration
+ * function for actual migration.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	int err;
+	bool retry = false;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		if (ctx->trylock_mmap) {
+			if (!mmap_read_trylock(mm))  {
+				err = drm_gpusvm_evict_to_sram(gpusvm, range);
+				goto err_mmput;
+			}
+		} else {
+			mmap_read_lock(mm);
+		}
+	}
+
+	mmap_assert_locked(mm);
+
+	/*
+	 * Loop required to find all VMA area structs for the corner case when
+	 * VRAM backing has been partially unmapped from MM's address space.
+	 */
+again:
+	vas = find_vma(mm, start);
+	if (!vas) {
+		if (!retry)
+			err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end <= vas->vm_start || start >= vas->vm_end) {
+		if (!retry)
+			err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	err = __drm_gpusvm_migrate_to_sram(gpusvm, vas, NULL, start, end);
+	if (err)
+		goto err_mmunlock;
+
+	if (vas->vm_end < end) {
+		retry = true;
+		start = vas->vm_end;
+		goto again;
+	}
+
+	if (!ctx->mmap_locked) {
+		mmap_read_unlock(mm);
+		/*
+		 * Using mmput_async as this function can be called while
+		 * holding a dma-resv lock, and a final put can grab the mmap
+		 * lock, causing a lock inversion.
+		 */
+		mmput_async(mm);
+	}
+
+	return 0;
+
+err_mmunlock:
+	if (!ctx->mmap_locked)
+		mmap_read_unlock(mm);
+err_mmput:
+	if (!ctx->mmap_locked)
+		mmput_async(mm);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_page_free - Put GPU SVM zone device data associated with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device data
+ * associated with a page when it is being released.
+ */
+static void drm_gpusvm_page_free(struct page *page)
+{
+	drm_gpusvm_zdd_put(page->zone_device_data);
+}
+
+/**
+ * drm_gpusvm_migrate_to_ram - Migrate GPU SVM range to RAM (page fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ * VM_FAULT_SIGBUS on failure, 0 on success.
+ */
+static vm_fault_t drm_gpusvm_migrate_to_ram(struct vm_fault *vmf)
+{
+	struct drm_gpusvm_zdd *zdd = vmf->page->zone_device_data;
+	int err;
+
+	err = __drm_gpusvm_migrate_to_sram(zdd->range->gpusvm,
+					   vmf->vma, vmf->page,
+					   zdd->range->va.start,
+					   zdd->range->va.end);
+
+	return err ? VM_FAULT_SIGBUS : 0;
+}
+
+/**
+ * drm_gpusvm_pagemap_ops - Device page map operations for GPU SVM
+ */
+static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = {
+	.page_free = drm_gpusvm_page_free,
+	.migrate_to_ram = drm_gpusvm_migrate_to_ram,
+};
+
+/**
+ * drm_gpusvm_pagemap_ops_get - Retrieve GPU SVM device page map operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void)
+{
+	return &drm_gpusvm_pagemap_ops;
+}
+
+/**
+ * drm_gpusvm_has_mapping - Check if GPU SVM has mapping for the given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end)
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	drm_gpusvm_for_each_notifier(notifier, gpusvm, start, end) {
+		struct drm_gpusvm_range *range = NULL;
+
+		drm_gpusvm_for_each_range(range, notifier, start, end)
+			return true;
+	}
+
+	return false;
+}
drivers/gpu/drm/xe/drm_gpusvm.h new file +415 -0
new file mode 100644
index 000000000000..0ea70f8534a8
@@ -0,0 +1,415 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef __DRM_GPUSVM_H__
+#define __DRM_GPUSVM_H__
+
+#include <linux/kref.h>
+#include <linux/mmu_notifier.h>
+#include <linux/workqueue.h>
+
+struct dev_pagemap_ops;
+struct drm_device;
+struct drm_gpusvm;
+struct drm_gpusvm_notifier;
+struct drm_gpusvm_ops;
+struct drm_gpusvm_range;
+
+/**
+ * struct drm_gpusvm_ops - Operations structure for GPU SVM
+ *
+ * This structure defines the operations for GPU Shared Virtual Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * perform operations such as migration between VRAM and system RAM.
+ */
+struct drm_gpusvm_ops {
+	/**
+	 * @notifier_alloc: Allocate a GPU SVM notifier (optional)
+	 *
+	 * This function shall allocate a GPU SVM notifier.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM notifier on success, NULL on failure.
+	 */
+	struct drm_gpusvm_notifier *(*notifier_alloc)(void);
+
+	/**
+	 * @notifier_free: Free a GPU SVM notifier (optional)
+	 * @notifier: Pointer to the GPU SVM notifier to be freed
+	 *
+	 * This function shall free a GPU SVM notifier.
+	 */
+	void (*notifier_free)(struct drm_gpusvm_notifier *notifier);
+
+	/**
+	 * @range_alloc: Allocate a GPU SVM range (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 *
+	 * This function shall allocate a GPU SVM range.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM range on success, NULL on failure.
+	 */
+	struct drm_gpusvm_range *(*range_alloc)(struct drm_gpusvm *gpusvm);
+
+	/**
+	 * @range_free: Free a GPU SVM range (optional)
+	 * @range: Pointer to the GPU SVM range to be freed
+	 *
+	 * This function shall free a GPU SVM range.
+	 */
+	void (*range_free)(struct drm_gpusvm_range *range);
+
+	/**
+	 * @vram_release: Release VRAM allocation (optional)
+	 * @vram_allocation: Driver-private pointer to the VRAM allocation
+	 *
+	 * This function shall release VRAM allocation and expects to drop a
+	 * reference to VRAM allocation.
+	 */
+	void (*vram_release)(void *vram_allocation);
+
+	/**
+	 * @populate_vram_pfn: Populate VRAM PFN (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @vram_allocation: Driver-private pointer to the VRAM allocation
+	 * @npages: Number of pages to populate
+	 * @pfn: Array of page frame numbers to populate
+	 *
+	 * This function shall populate VRAM page frame numbers (PFN).
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*populate_vram_pfn)(struct drm_gpusvm *gpusvm,
+				 void *vram_allocation,
+				 unsigned long npages,
+				 unsigned long *pfn);
+
+	/**
+	 * @copy_to_vram: Copy to VRAM (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (destination)
+	 * @dma_addr: Pointer to array of DMA addresses (source)
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to VRAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_vram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @copy_to_sram: Copy to system RAM (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (source)
+	 * @dma_addr: Pointer to array of DMA addresses (destination)
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to system RAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_sram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @invalidate: Invalidate GPU SVM notifier (required)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @notifier: Pointer to the GPU SVM notifier
+	 * @mmu_range: Pointer to the mmu_notifier_range structure
+	 *
+	 * This function shall invalidate the GPU page tables. It can safely
+	 * walk the notifier range RB tree/list in this function. Called while
+	 * holding the notifier lock.
+	 */
+	void (*invalidate)(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_notifier *notifier,
+			   const struct mmu_notifier_range *mmu_range);
+};
+
+/**
+ * struct drm_gpusvm_notifier - Structure representing a GPU SVM notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @interval: Interval for the notifier
+ * @rb: Red-black tree node for the parent GPU SVM structure notifier tree
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ *              interval tree. This is useful to keep iterating ranges while
+ *              doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier has been
+ *                 removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */
+struct drm_gpusvm_notifier {
+	struct drm_gpusvm *gpusvm;
+	struct mmu_interval_notifier notifier;
+	struct {
+		u64 start;
+		u64 end;
+	} interval;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct rb_root_cached root;
+	struct list_head range_list;
+	struct {
+		u32 removed : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm_range - Structure representing a GPU SVM range
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier
+ * @refcount: Reference count for the range
+ * @rb: Red-black tree node for the parent GPU SVM notifier structure range tree
+ * @va: Virtual address range
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @pages: Pointer to the array of pages (if backing store is in VRAM)
+ * @dma_addr: DMA address array (if backing store is SRAM and DMA mapped)
+ * @vram_allocation: Driver-private pointer to the VRAM allocation
+ * @order: Order of dma mapping. i.e. PAGE_SIZE << order is mapping size
+ * @flags.migrate_vram: Flag indicating whether the range can be migrated to VRAM
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_vram_pages: Flag indicating if the range has vram pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA mapping
+ * @flags.kfree_mapping: Flag indicating @dma_addr is a compact allocation based
+ *                       on @order which releases via kfree
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */
+struct drm_gpusvm_range {
+	struct drm_gpusvm *gpusvm;
+	struct drm_gpusvm_notifier *notifier;
+	struct kref refcount;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct {
+		u64 start;
+		u64 end;
+	} va;
+	unsigned long notifier_seq;
+	union {
+		struct page **pages;
+		dma_addr_t *dma_addr;
+	};
+	void *vram_allocation;
+	u16 order;
+	struct {
+		/* All flags below must be set upon creation */
+		u16 migrate_vram : 1;
+		/* All flags below must be set / cleared under notifier lock */
+		u16 unmapped : 1;
+		u16 partial_unmap : 1;
+		u16 has_vram_pages : 1;
+		u16 has_dma_mapping : 1;
+		u16 kfree_mapping : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm - GPU SVM structure
+ *
+ * @name: Name of the GPU SVM
+ * @drm: Pointer to the DRM device structure
+ * @mm: Pointer to the mm_struct for the address space
+ * @device_private_page_owner: Device private pages owner
+ * @mm_start: Start address of GPU SVM
+ * @mm_range: Range of the GPU SVM
+ * @notifier_size: Size of individual notifiers
+ * @ops: Pointer to the operations structure for GPU SVM
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range allocation.
+ *               Entries should be powers of 2 in descending order.
+ * @num_chunks: Number of chunks
+ * @notifier_lock: Read-write semaphore for protecting notifier operations
+ * @zdd_wq: Workqueue for deferred work on zdd destruction
+ * @root: Cached root node of the Red-Black tree containing GPU SVM notifiers
+ * @notifier_list: list head containing of notifiers in the same order they
+ *                 appear in interval tree. This is useful to keep iterating
+ *                 notifiers while doing modifications to RB tree.
+ *
+ * This structure represents a GPU SVM (Shared Virtual Memory) used for tracking
+ * memory ranges mapped in a DRM (Direct Rendering Manager) device.
+ *
+ * No reference counting is provided, as this is expected to be embedded in the
+ * driver VM structure along with the struct drm_gpuvm, which handles reference
+ * counting.
+ */
+struct drm_gpusvm {
+	const char *name;
+	struct drm_device *drm;
+	struct mm_struct *mm;
+	void *device_private_page_owner;
+	u64 mm_start;
+	u64 mm_range;
+	u64 notifier_size;
+	const struct drm_gpusvm_ops *ops;
+	const u64 *chunk_sizes;
+	int num_chunks;
+	struct rw_semaphore notifier_lock;
+	struct workqueue_struct *zdd_wq;
+	struct rb_root_cached root;
+	struct list_head notifier_list;
+};
+
+/**
+ * struct drm_gpusvm_ctx - DRM GPU SVM context
+ *
+ * @mmap_locked: mmap lock is locked
+ * @trylock_mmap: trylock mmap lock, used to avoid locking inversions
+ *                (e.g.dma-revs -> mmap lock)
+ * @in_notifier: entering from a MMU notifier
+ * @read_only: operating on read-only memory
+ * @vram_possible: possible to use VRAM
+ * @prefault: prefault pages
+ *
+ * Context that is DRM GPUSVM is operating in (i.e. user arguments).
+ */
+struct drm_gpusvm_ctx {
+	u32 mmap_locked :1;
+	u32 trylock_mmap :1;
+	u32 in_notifier :1;
+	u32 read_only :1;
+	u32 vram_possible :1;
+	u32 prefault :1;
+};
+
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void *device_private_page_owner,
+		    u64 mm_start, u64 mm_range, u64 notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const u64 *chunk_sizes, int num_chunks);
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm);
+void drm_gpusvm_free(struct drm_gpusvm *gpusvm);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, u64 fault_addr,
+				u64 gpuva_start, u64 gpuva_end,
+				const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range);
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range);
+
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range);
+
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx);
+
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx);
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void);
+
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, u64 start, u64 end);
+
+/**
+ * drm_gpusvm_notifier_lock - Lock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, take lock
+ */
+#define drm_gpusvm_notifier_lock(gpusvm__)	\
+	down_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * drm_gpusvm_notifier_unlock - Unlock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, drop lock
+ */
+#define drm_gpusvm_notifier_unlock(gpusvm__)	\
+	up_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * __drm_gpusvm_range_next - Get the next GPU SVM range in the list
+ * @range: a pointer to the current GPU SVM range
+ *
+ * Return: A pointer to the next drm_gpusvm_range if available, or NULL if the
+ *         current range is the last one or if the input range is NULL.
+ */
+static inline struct drm_gpusvm_range *
+__drm_gpusvm_range_next(struct drm_gpusvm_range *range)
+{
+	if (range && !list_is_last(&range->rb.entry,
+				   &range->notifier->range_list))
+		return list_next_entry(range, rb.entry);
+
+	return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_range - Iterate over GPU SVM ranges in a notifier
+ * @range__: Iterator variable for the ranges. If set, it indicates the start of
+ *	     the iterator. If NULL, call drm_gpusvm_range_find() to get the range.
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier. It is safe
+ * to use while holding the driver SVM lock or the notifier lock.
+ */
+#define drm_gpusvm_for_each_range(range__, notifier__, start__, end__)	\
+	for ((range__) = (range__) ?:					\
+	     drm_gpusvm_range_find((notifier__), (start__), (end__));	\
+	     (range__) && (range__->va.start < (end__));		\
+	     (range__) = __drm_gpusvm_range_next(range__))
+
+/**
+ * drm_gpusvm_range_set_unmapped - Mark a GPU SVM range as unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets the partial_unmap flag
+ * if the range partially falls within the provided MMU notifier range.
+ */
+static inline void
+drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,
+			      const struct mmu_notifier_range *mmu_range)
+{
+	lockdep_assert_held_write(&range->gpusvm->notifier_lock);
+
+	range->flags.unmapped = true;
+	if (range->va.start < mmu_range->start ||
+	    range->va.end > mmu_range->end)
+		range->flags.partial_unmap = true;
+}
+
+#endif /* __DRM_GPUSVM_H__ */
Signature · 1 line
2.34.1
Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
Signature · 3 lines
Simona Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#17 lore raw

Re: [RFC PATCH 05/28] drm/gpusvm: Add support for GPU Shared Virtual Memory

In reply to Matthew Brost
Quoted 23 lines · Matthew Brost
drivers/gpu/drm/xe/drm_gpusvm.c new file +23 -0
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	int err;
+	bool retry = false;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		if (ctx->trylock_mmap) {
+			if (!mmap_read_trylock(mm))  {
+				err = drm_gpusvm_evict_to_sram(gpusvm, range);
+				goto err_mmput;
+			}
+		} else {
+			mmap_read_lock(mm);
+		}

drivers/gpu/drm/xe/drm_gpusvm.c line 2053

Quoted 10 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+	}
+
+	mmap_assert_locked(mm);
+
+	/*
+	 * Loop required to find all VMA area structs for the corner case when
+	 * VRAM backing has been partially unmapped from MM's address space.
+	 */
+again:
+	vas = find_vma(mm, start);

So a hiliarous case that amdkfd gets a bit better but still not entirely is that the original vma might entirely gone. Even when you can still get at the mm of that process. This happens with cow (or shared too I think) mappings in forked child processes, or also if you play fun mremap games.

I think that outside of the ->migrate_to_ram callback migration/eviction to sram cannot assume there's any reasonable vma around and has to unconditionally go with the drm_gpusvm_evict_to_sram path.

Also in the migrate_to_ram case the vma is essentially nothing else that informational about which ranges we might need if we prefault a bit (in case the child changed the vma compared to the original one). So it's good to as parameter for migrate_vma_setup, but absolutely nothing else.

amdkfd almost gets this right by being entirely based on their svm_range structures, except they still have the lingering check that the orignal mm is still alive. Of course you cannot ever use that memory on the gpu anymore, but the child process could get very pissed if their memory is suddenly gone. Also the eviction code has the same issue as yours and limits itself to vma that still exist in the original mm, leaving anything that's orphaned in children or remaps stuck in vram. At least that's my understanding, I might very well be wrong.

So probably want a bunch of these testcases too to make sure that all works, and we're not stuck with memory allocations in vram that we can't move out.
-Sima

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#18 lore raw

Re: [RFC PATCH 05/28] drm/gpusvm: Add support for GPU Shared Virtual Memory

Quoted 40 lines · Daniel Vetter
Quoted 23 lines · Matthew Brost
drivers/gpu/drm/xe/drm_gpusvm.c new file +23 -0
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	int err;
+	bool retry = false;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		if (ctx->trylock_mmap) {
+			if (!mmap_read_trylock(mm))  {
+				err = drm_gpusvm_evict_to_sram(gpusvm, range);
+				goto err_mmput;
+			}
+		} else {
+			mmap_read_lock(mm);
+		}

drivers/gpu/drm/xe/drm_gpusvm.c line 2053

Quoted 10 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+	}
+
+	mmap_assert_locked(mm);
+
+	/*
+	 * Loop required to find all VMA area structs for the corner case when
+	 * VRAM backing has been partially unmapped from MM's address space.
+	 */
+again:
+	vas = find_vma(mm, start);

So a hiliarous case that amdkfd gets a bit better but still not entirely is that the original vma might entirely gone. Even when you can still get at the mm of that process. This happens with cow (or shared too I think) mappings in forked child processes, or also if you play fun mremap games.

Quoted 4 lines

I think that outside of the ->migrate_to_ram callback migration/eviction to sram cannot assume there's any reasonable vma around and has to unconditionally go with the drm_gpusvm_evict_to_sram path.

See my response here [1]. Let me drop the whole trylock thing and convert to an 'evict' flag which calls drm_gpusvm_evict_to_sram in places where Xe needs to evict VRAM. Or maybe just export that function and call it directly. That way the only place the VMA is looked up for SRAM -> VRAM is upon CPU page fault.

[1] https://patchwork.freedesktop.org/patch/610955/?series=137870&rev=1#comment_1111164

Quoted 14 lines

Also in the migrate_to_ram case the vma is essentially nothing else that informational about which ranges we might need if we prefault a bit (in case the child changed the vma compared to the original one). So it's good to as parameter for migrate_vma_setup, but absolutely nothing else.

amdkfd almost gets this right by being entirely based on their svm_range structures, except they still have the lingering check that the orignal mm is still alive. Of course you cannot ever use that memory on the gpu anymore, but the child process could get very pissed if their memory is suddenly gone. Also the eviction code has the same issue as yours and limits itself to vma that still exist in the original mm, leaving anything that's orphaned in children or remaps stuck in vram. At least that's my understanding, I might very well be wrong.

Quoted 3 lines

So probably want a bunch of these testcases too to make sure that all works, and we're not stuck with memory allocations in vram that we can't move out.

When writing some additional test cases, let me add hooks in my IGTs to be able to verify we are not orphaning VRAM too.

Matt

Quoted 5 lines

-Sima

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#19 lore raw

Re: [RFC PATCH 05/28] drm/gpusvm: Add support for GPU Shared Virtual Memory

Quoted 46 lines · Matthew Brost
Quoted 40 lines · Daniel Vetter
Quoted 23 lines · Matthew Brost
drivers/gpu/drm/xe/drm_gpusvm.c new file +23 -0
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	int err;
+	bool retry = false;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		if (ctx->trylock_mmap) {
+			if (!mmap_read_trylock(mm))  {
+				err = drm_gpusvm_evict_to_sram(gpusvm, range);
+				goto err_mmput;
+			}
+		} else {
+			mmap_read_lock(mm);
+		}

drivers/gpu/drm/xe/drm_gpusvm.c line 2053

Quoted 10 lines
drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+	}
+
+	mmap_assert_locked(mm);
+
+	/*
+	 * Loop required to find all VMA area structs for the corner case when
+	 * VRAM backing has been partially unmapped from MM's address space.
+	 */
+again:
+	vas = find_vma(mm, start);

So a hiliarous case that amdkfd gets a bit better but still not entirely is that the original vma might entirely gone. Even when you can still get at the mm of that process. This happens with cow (or shared too I think) mappings in forked child processes, or also if you play fun mremap games.

Quoted 4 lines

I think that outside of the ->migrate_to_ram callback migration/eviction to sram cannot assume there's any reasonable vma around and has to unconditionally go with the drm_gpusvm_evict_to_sram path.

Quoted 5 lines

See my response here [1]. Let me drop the whole trylock thing and convert to an 'evict' flag which calls drm_gpusvm_evict_to_sram in places where Xe needs to evict VRAM. Or maybe just export that function and call it directly. That way the only place the VMA is looked up for SRAM -> VRAM is upon CPU page fault.

Yeah I think a dedicated path for migrate_to_ram hook that goes directly into your evict_to_sram path is the design-clean approach here imo.

Quoted 16 lines

[1] https://patchwork.freedesktop.org/patch/610955/?series=137870&rev=1#comment_1111164

Quoted 14 lines

Also in the migrate_to_ram case the vma is essentially nothing else that informational about which ranges we might need if we prefault a bit (in case the child changed the vma compared to the original one). So it's good to as parameter for migrate_vma_setup, but absolutely nothing else.

amdkfd almost gets this right by being entirely based on their svm_range structures, except they still have the lingering check that the orignal mm is still alive. Of course you cannot ever use that memory on the gpu anymore, but the child process could get very pissed if their memory is suddenly gone. Also the eviction code has the same issue as yours and limits itself to vma that still exist in the original mm, leaving anything that's orphaned in children or remaps stuck in vram. At least that's my understanding, I might very well be wrong.

Quoted 6 lines
Quoted 3 lines

So probably want a bunch of these testcases too to make sure that all works, and we're not stuck with memory allocations in vram that we can't move out.

When writing some additional test cases, let me add hooks in my IGTs to be able to verify we are not orphaning VRAM too.

So maybe apply caution, I'm honestly not sure whether core mm makes any guarantees about not orphaning stuff, at least for a little bit.

Over the w/e my brain tossed me the "so are we sure we can tear down our zone_device data, the page array specifically" brain teaser. And I think the answer is that we have to wait until all page references disappear, which might take a long time. Core mm makes no guarantee about elevated page references disappearing in a timely manner, at least as far as I know. Which is also why migration is a best effort thing only.

Cheers, Sima

Signature · 3 lines
Simona Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#20 lore raw

Re: [RFC PATCH 05/28] drm/gpusvm: Add support for GPU Shared Virtual Memory

In reply to Matthew Brost

Hi, Matt.

Some initial design comments / questions:

Quoted 95 lines · Matthew Brost

This patch introduces support for GPU Shared Virtual Memory (SVM) in the
Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.

The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also
provides mechanisms for allocating, deallocating, and migrating memory
regions between system RAM and GPU VRAM.

This mid-layer is largely inspired by GPUVM.

Cc Dave Airlie <[email protected]>
Cc Thomas Hellström <[email protected]>
Cc Christian König <[email protected]>

---
 drivers/gpu/drm/xe/Makefile     |    3 +-
 drivers/gpu/drm/xe/drm_gpusvm.c | 2174
+++++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/drm_gpusvm.h |  415 ++++++

 3 files changed, 2591 insertions(+), 1 deletion(-)

 create mode 100644 drivers/gpu/drm/xe/drm_gpusvm.c create mode 100644 drivers/gpu/drm/xe/drm_gpusvm.h

drivers/gpu/drm/xe/Makefile +0 -0
index b9670ae09a9e..b8fc2ee58f1a 100644
@@ -25,7 +25,8 @@ $(obj)/generated/%_wa_oob.c
 $(obj)/generated/%_wa_oob.h: $(obj)/xe_gen_wa_oob \

 # core driver code
 
drivers/gpu/drm/xe/Makefile +2 -1
-xe-y += xe_bb.o \
+xe-y += drm_gpusvm.o \
+	xe_bb.o \

  xe_bo.o \
  xe_bo_evict.o \
  xe_devcoredump.o \

drivers/gpu/drm/xe/drm_gpusvm.c new file +24 -0
new file mode 100644
index 000000000000..fc1e44e6ae72
@@ -0,0 +1,2174 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2024 Intel Corporation
+ *
+ * Authors:
+ *     Matthew Brost <[email protected]>
+ */
+
+#include <linux/dma-mapping.h>
+#include <linux/interval_tree_generic.h>
+#include <linux/hmm.h>
+#include <linux/memremap.h>
+#include <linux/migrate.h>
+#include <linux/mm_types.h>
+#include <linux/pagemap.h>
+#include <linux/slab.h>
+
+#include <drm/drm_device.h>
+#include "drm_gpusvm.h"
+
+/**
+ * DOC: Overview
+ *
+ * GPU Shared Virtual Memory (GPU SVM) layer for the Direct

Rendering Manager (DRM)
+ *
+ * The GPU SVM layer is a component of the DRM framework designed to manage shared
+ * virtual memory between the CPU and GPU. It enables efficient data exchange and
+ * processing for GPU-accelerated applications by allowing memory sharing and
+ * synchronization between the CPU's and GPU's virtual address spaces.
+ *
+ * Key GPU SVM Components:
+ * - Notifiers: Notifiers: Used for tracking memory intervals and notifying the
+ * GPU of changes, notifiers are sized based on a GPU
SVM
+ * initialization parameter, with a recommendation of
512M or
+ * larger. They maintain a Red-BlacK tree and a list of

Quoted 8 lines

ranges that
+ * fall within the notifier interval. Notifiers are
tracked within
+ * a GPU SVM Red-BlacK tree and list and are
dynamically inserted
+ * or removed as ranges within the interval are created
or
+ * destroyed.

What is the benefit of this extra layer compared to direct insertion of ranges using mmu_interval_notifier_insert?

IIRC the argument made previously about having wide notifiers was that the rb tree lookups inside the core were costly and if there were only a few, then the rb tree lookups within a notifier range could be replaced with the page-table radix-tree-like lookup, so each lookup complexity would be O(log(n_notifiers) + page_table_depth).

But now we have first an rb-tree lookup in the core and then an rb-tree lookup within each notifier yeilding O(log(n_ranges))

I can see a small benefit in that inserting directly into the core rb- tree will block pending ongoing invalidations, but at a cost of an extra multiplexing layer.

Quoted 7 lines

+ * - Ranges: Represent memory ranges mapped in a DRM device and managed
+ *      by GPU SVM. They are sized based on an array of chunk
sizes, which
+ *      is a GPU SVM initialization parameter, and the CPU
address space.
+ *      Upon GPU fault, the largest aligned chunk that fits

Quoted 8 lines

within the
+ *      faulting CPU address space is chosen for the range
size. Ranges are
+ *      expected to be dynamically allocated on GPU fault and
removed on an
+ *      MMU notifier UNMAP event. As mentioned above, ranges
are tracked in
+ *      a notifier's Red-Black tree.

How do ranges and chunks map to

a) Prefaulting granularity
b) Migration granularity?

Quoted 113 lines

+ * - Operations: Define the interface for driver-specific SVM operations such as
+ * allocation, page collection, migration,
invalidations, and VRAM
+ * release.
+ *
+ * This layer provides interfaces for allocating, mapping, migrating, and
+ * releasing memory ranges between the CPU and GPU. It handles all core memory
+ * management interactions (DMA mapping, HMM, and migration) and provides
+ * driver-specific virtual functions (vfuncs). This infrastructure is sufficient
+ * to build the expected driver components for an SVM implementation as detailed
+ * below.
+ *
+ * Expected Driver Components:
+ * - GPU page fault handler: Used to create ranges and notifiers based on the
+ *      fault address, optionally migrate the
range to
+ *      VRAM, and create GPU bindings.
+ * - Garbage collector: Used to destroy GPU bindings for ranges. Ranges are
+ * expected to be added to the garbage
collector upon
+ * MMU_NOTIFY_UNMAP event.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Locking
+ *
+ * GPU SVM handles locking for core MM interactions, i.e., it

locks/unlocks the
+ * mmap lock as needed. Alternatively, if the driver prefers to handle the mmap
+ * lock itself, a 'locked' argument is provided to the functions that require
+ * the mmap lock. This option may be useful for drivers that need to call into
+ * GPU SVM while also holding a dma-resv lock, thus preventing locking
+ * inversions between the mmap and dma-resv locks.
+ *
+ * GPU SVM introduces a global notifier lock, which safeguards the notifier's
+ * range RB tree and list, as well as the range's DMA mappings and sequence
+ * number. GPU SVM manages all necessary locking and unlocking operations,
+ * except for the recheck of the range's sequence number
+ * (mmu_interval_read_retry) when the driver is committing GPU bindings. This
+ * lock corresponds to the 'driver->update' lock mentioned in the HMM
+ * documentation (TODO: Link). Future revisions may transition from a GPU SVM
+ * global lock to a per-notifier lock if finer-grained locking is deemed
+ * necessary.
+ *
+ * In addition to the locking mentioned above, the driver should implement a
+ * lock to safeguard core GPU SVM function calls that modify state, such as
+ * drm_gpusvm_range_find_or_insert and drm_gpusvm_range_remove. Alternatively,
+ * these core functions can be called within a single kernel thread, for
+ * instance, using an ordered work queue. This lock is denoted as
+ * 'driver_svm_lock' in code examples.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Migrataion
+ *
+ * The migration support is quite simple, allowing migration between

SRAM and
+ * VRAM at the range granularity. For example, GPU SVM currently does not
+ * support mixing SRAM and VRAM pages within a range. This means that upon GPU
+ * fault, the entire range can be migrated to VRAM, and upon CPU fault, the
+ * entire range is migrated to SRAM.
+ *
+ * The reasoning for only supporting range granularity is as
follows: it
+ * simplifies the implementation, and range sizes are driver-defined and should
+ * be relatively small.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Partial Unmapping of Ranges
+ *
+ * Partial unmapping of ranges (e.g., 1M out of 2M is unmapped by

CPU resulting
+ * in MMU_NOTIFY_UNMAP event) presents several challenges, with the main one
+ * being that a subset of the range still has CPU and GPU mappings. If the
+ * backing store for the range is in VRAM, a subset of the backing store has
+ * references. One option would be to split the range and VRAM backing store,
+ * but the implementation for this would be quite complicated. Given that
+ * partial unmappings are rare and driver-defined range sizes are relatively
+ * small, GPU SVM does not support splitting of ranges.

Quoted 8 lines

+ *
+ * With no support for range splitting, upon partial unmapping of a range, the
+ * driver is expected to invalidate and destroy the entire range. If the range
+ * has VRAM as its backing, the driver is also expected to migrate any remaining
+ * pages back to SRAM.

So what happens if we get a one-page invalidation, say protection change event, or NUMA accounting event, in the middle of a range? Can we unmap just that single gpu pte covering that range, that is, how do the ranges map to invalidation granularity? Does this differ between igfx an dgfx?

Thanks,
Thomas

Quoted 2,818 lines

+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+
+/**
+ * DOC: Examples
+ *
+ * This section provides two examples of how to build the expected

driver
+ * components: the GPU page fault handler and the garbage collector. A third
+ * example demonstrates a sample invalidation driver vfunc.
+ *
+ * The generic code provided does not include logic for complex migration
+ * policies, optimized invalidations, or other potentially required driver
+ * locking (e.g., DMA-resv locks).
+ *
+ * 1) GPU page fault handler
+ *
+ * int driver_bind_range(struct drm_gpusvm *gpusvm, struct
drm_gpusvm_range *range)
+ * {
+ * int err = 0;
+ *
+ * driver_alloc_and_setup_memory_for_bind(gpusvm,
range);
+ *
+ * drm_gpusvm_notifier_lock(gpusvm);
+ * if (drm_gpusvm_range_pages_valid(range))
+ * driver_commit_bind(gpusvm, range);
+ * else
+ * err = -EAGAIN;
+ * drm_gpusvm_notifier_unlock(gpusvm);
+ *
+ * return err;
+ * }
+ *
+ * int driver_gpu_fault(struct drm_gpusvm *gpusvm, u64
fault_addr,
+ *      u64 gpuva_start, u64 gpuva_end)
+ * {
+ * struct drm_gpusvm_ctx ctx = {};
+ * int err;
+ *
+ * driver_svm_lock();
+ * retry:
+ * // Always process UNMAPs first so view of GPU SVM
ranges is current
+ * driver_garbage_collector(gpusvm);
+ *
+ * range = drm_gpusvm_range_find_or_insert(gpusvm,
fault_addr,
+ * gpuva_start,
gpuva_end,
+ *         &ctx);
+ * if (IS_ERR(range)) {
+ * err = PTR_ERR(range);
+ * goto unlock;
+ * }
+ *
+ * if (driver_migration_policy(range)) {
+ * bo = driver_alloc_bo();
+ * err = drm_gpusvm_migrate_to_vram(gpusvm,
range, bo, &ctx);
+ * if (err) // CPU mappings may have
changed
+ * goto retry;
+ * }
+ *
+ * err = drm_gpusvm_range_get_pages(gpusvm, range,
&ctx);
+ * if (err == -EFAULT || err == -EPERM) // CPU
mappings changed
+ * goto retry;
+ * else if (err)
+ * goto unlock;
+ *
+ * err = driver_bind_range(gpusvm, range);
+ * if (err == -EAGAIN) // CPU mappings changed
+ * goto retry
+ *
+ * unlock:
+ * driver_svm_unlock();
+ * return err;
+ * }
+ *
+ * 2) Garbage Collector.
+ *
+ * void __driver_garbage_collector(struct drm_gpusvm *gpusvm,
+ * struct drm_gpusvm_range
*range)
+ * {
+ * struct drm_gpusvm_ctx ctx = {};
+ *
+ * assert_driver_svm_locked(gpusvm);
+ *
+ * // Partial unmap, migrate any remaining VRAM pages
back to SRAM
+ * if (range->flags.partial_unmap)
+ * drm_gpusvm_migrate_to_sram(gpusvm, range,
&ctx);
+ *
+ * driver_unbind_range(range);
+ * drm_gpusvm_range_remove(gpusvm, range);
+ * }
+ *
+ * void driver_garbage_collector(struct drm_gpusvm *gpusvm)
+ * {
+ * assert_driver_svm_locked(gpusvm);
+ *
+ * for_each_range_in_garbage_collector(gpusvm, range)
+ * __driver_garbage_collector(gpusvm, range);
+ * }
+ *
+ * 3) Invalidation driver vfunc.
+ *
+ * void driver_invalidation(struct drm_gpusvm *gpusvm,
+ * struct drm_gpusvm_notifier
*notifier,
+ * const struct mmu_notifier_range
*mmu_range)
+ * {
+ * struct drm_gpusvm_ctx ctx = { .in_notifier = true,
};
+ * struct drm_gpusvm_range *range = NULL;
+ *
+ * driver_invalidate_device_tlb(gpusvm, mmu_range-

Quoted 1 line

start, mmu_range->end);

+ *
+ * drm_gpusvm_for_each_range(range, notifier,
mmu_range->start,
+ *   mmu_range->end) {
+ * drm_gpusvm_range_unmap_pages(gpusvm, range,
&ctx);
+ *
+ * if (mmu_range->event != MMU_NOTIFY_UNMAP)
+ * continue;
+ *
+ * drm_gpusvm_range_set_unmapped(range,
mmu_range);
+ * driver_garbage_collector_add(gpusvm, range);
+ * }
+ * }
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+
+#define DRM_GPUSVM_RANGE_START(_range)	((_range)->va.start)
+#define DRM_GPUSVM_RANGE_END(_range)	((_range)->va.end - 1)
+INTERVAL_TREE_DEFINE(struct drm_gpusvm_range, rb.node, u64,

rb.__subtree_last,

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+		     DRM_GPUSVM_RANGE_START, DRM_GPUSVM_RANGE_END,
+		     static __maybe_unused, range);
+
+#define DRM_GPUSVM_NOTIFIER_START(_notifier)	((_notifier)-
Quoted 1 line

interval.start)

+#define DRM_GPUSVM_NOTIFIER_END(_notifier) ((_notifier)-

Quoted 1 line

interval.end - 1)

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+INTERVAL_TREE_DEFINE(struct drm_gpusvm_notifier, rb.node, u64,
+		     rb.__subtree_last, DRM_GPUSVM_NOTIFIER_START,
+		     DRM_GPUSVM_NOTIFIER_END, static __maybe_unused,

notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+
+/**
+ * npages_in_range() - Calculate the number of pages in a given

range
+ * @start__: The start address of the range
+ * @end__: The end address of the range
+ *
+ * This macro calculates the number of pages in a given memory range,
+ * specified by the start and end addresses. It divides the difference
+ * between the end and start addresses by the page size (PAGE_SIZE) to
+ * determine the number of pages in the range.
+ *
+ * Return: The number of pages in the specified range.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +13 -0
+#define npages_in_range(start__, end__)	\
+	(((end__) - (start__)) >> PAGE_SHIFT)
+
+/**
+ * struct drm_gpusvm_zdd - GPU SVM zone device data
+ *
+ * @refcount: Reference count for the zdd
+ * @destroy_work: Work structure for asynchronous zdd destruction
+ * @range: Pointer to the GPU SVM range
+ * @vram_allocation: Driver-private pointer to the VRAM allocation
+ *
+ * This structure serves as a generic wrapper installed in
+ * page->zone_device_data. It provides infrastructure for looking up

a range
+ * upon CPU page fault and asynchronously releasing VRAM once the CPU has no
+ * page references. Asynchronous release is useful because CPU page references
+ * can be dropped in IRQ contexts, while releasing VRAM likely requires sleeping
+ * locks.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+struct drm_gpusvm_zdd {
+	struct kref refcount;
+	struct work_struct destroy_work;
+	struct drm_gpusvm_range *range;
+	void *vram_allocation;
+};
+
+/**
+ * drm_gpusvm_zdd_destroy_work_func - Work function for destroying a

zdd
+ * @w: Pointer to the work_struct
+ *
+ * This function releases VRAM, puts GPU SVM range, and frees zdd.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_zdd_destroy_work_func(struct work_struct *w)
+{
+	struct drm_gpusvm_zdd *zdd =
+		container_of(w, struct drm_gpusvm_zdd,

destroy_work);

drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+	struct drm_gpusvm_range *range = zdd->range;
+	struct drm_gpusvm *gpusvm = range->gpusvm;
+
+	if (gpusvm->ops->vram_release && zdd->vram_allocation)
+		gpusvm->ops->vram_release(zdd->vram_allocation);
+	drm_gpusvm_range_put(range);
+	kfree(zdd);
+}
+
+/**
+ * drm_gpusvm_zdd_alloc - Allocate a zdd structure.
+ * @range: Pointer to the GPU SVM range.
+ *
+ * This function allocates and initializes a new zdd structure. It

sets up the
+ * reference count, initializes the destroy work, and links the provided GPU SVM
+ * range.
+ *
+ * Returns:
+ * Pointer to the allocated zdd on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+static struct drm_gpusvm_zdd *
+drm_gpusvm_zdd_alloc(struct drm_gpusvm_range *range)
+{
+	struct drm_gpusvm_zdd *zdd;
+
+	zdd = kmalloc(sizeof(*zdd), GFP_KERNEL);
+	if (!zdd)
+		return NULL;
+
+	kref_init(&zdd->refcount);
+	INIT_WORK(&zdd->destroy_work,

drm_gpusvm_zdd_destroy_work_func);

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+	zdd->range = drm_gpusvm_range_get(range);
+	zdd->vram_allocation = NULL;
+
+	return zdd;
+}
+
+/**
+ * drm_gpusvm_zdd_get - Get a reference to a zdd structure.
+ * @zdd: Pointer to the zdd structure.
+ *
+ * This function increments the reference count of the provided zdd

structure.
+ *
+ * Returns: Pointer to the zdd structure.
+ */
+static struct drm_gpusvm_zdd *drm_gpusvm_zdd_get(struct drm_gpusvm_zdd *zdd)

drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+{
+	kref_get(&zdd->refcount);
+	return zdd;
+}
+
+/**
+ * drm_gpusvm_zdd_destroy - Destroy a zdd structure.
+ * @ref: Pointer to the reference count structure.
+ *
+ * This function queues the destroy_work of the zdd for asynchronous

destruction.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+static void drm_gpusvm_zdd_destroy(struct kref *ref)
+{
+	struct drm_gpusvm_zdd *zdd =
+		container_of(ref, struct drm_gpusvm_zdd, refcount);
+	struct drm_gpusvm *gpusvm = zdd->range->gpusvm;
+
+	queue_work(gpusvm->zdd_wq, &zdd->destroy_work);
+}
+
+/**
+ * drm_gpusvm_zdd_put - Put a zdd reference.
+ * @zdd: Pointer to the zdd structure.
+ *
+ * This function decrements the reference count of the provided zdd

structure
+ * and schedules its destruction if the count drops to zero.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+static void drm_gpusvm_zdd_put(struct drm_gpusvm_zdd *zdd)
+{
+	kref_put(&zdd->refcount, drm_gpusvm_zdd_destroy);
+}
+
+/**
+ * drm_gpusvm_range_find - Find GPU SVM range from GPU SVM notifier
+ * @notifier: Pointer to the GPU SVM notifier structure.
+ * @start: Start address of the range
+ * @end: End address of the range
+ *
+ * Return: A pointer to the drm_gpusvm_range if found or NULL
+ */
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, u64

start, u64 end)

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+{
+	return range_iter_first(&notifier->root, start, end - 1);
+}
+
+/**
+ * drm_gpusvm_for_each_range_safe - Safely iterate over GPU SVM

ranges in a notifier
+ * @range__: Iterator variable for the ranges
+ * @next__: Iterator variable for the ranges temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier while
+ * removing ranges from it.
+ */
+#define drm_gpusvm_for_each_range_safe(range__, next__, notifier__,
start__, end__) \

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+	for ((range__) = drm_gpusvm_range_find((notifier__),

(start__), (end__)), \

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+	     (next__) =

__drm_gpusvm_range_next(range__); \

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+	     (range__) && (range__->va.start <

(end__)); \

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+	     (range__) = (next__), (next__) =

__drm_gpusvm_range_next(range__))

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+
+/**
+ * __drm_gpusvm_notifier_next - get the next drm_gpusvm_notifier in

the list
+ * @notifier: a pointer to the current drm_gpusvm_notifier
+ *
+ * Return: A pointer to the next drm_gpusvm_notifier if available, or NULL if
+ *         the current notifier is the last one or if the input notifier is
+ *         NULL.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+static struct drm_gpusvm_notifier *
+__drm_gpusvm_notifier_next(struct drm_gpusvm_notifier *notifier)
+{
+	if (notifier && !list_is_last(&notifier->rb.entry,
+				      &notifier->gpusvm-
Quoted 1 line

notifier_list))

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+		return list_next_entry(notifier, rb.entry);
+
+	return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_notifier - Iterate over GPU SVM notifiers in

a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm.
+ */
+#define drm_gpusvm_for_each_notifier(notifier__, gpusvm__, start__,
end__) \

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+	for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,

(start__), (end__) - 1); \

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+	     (notifier__) && (notifier__->interval.start <

(end__)); \

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+	     (notifier__) = __drm_gpusvm_notifier_next(notifier__))
+
+/**
+ * drm_gpusvm_for_each_notifier_safe - Safely iterate over GPU SVM

notifiers in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @next__: Iterator variable for the notifiers temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm while
+ * removing notifiers from it.
+ */
+#define drm_gpusvm_for_each_notifier_safe(notifier__, next__,
gpusvm__, start__, end__) \

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+	for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,

(start__), (end__) - 1), \

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+	     (next__) =

__drm_gpusvm_notifier_next(notifier__); \

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+	     (notifier__) && (notifier__->interval.start <

(end__)); \

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+	     (notifier__) = (next__), (next__) =

__drm_gpusvm_notifier_next(notifier__))

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+
+/**
+ * drm_gpusvm_notifier_invalidate - Invalidate a GPU SVM notifier.
+ * @mni: Pointer to the mmu_interval_notifier structure.
+ * @mmu_range: Pointer to the mmu_notifier_range structure.
+ * @cur_seq: Current sequence number.
+ *
+ * This function serves as a generic MMU notifier for GPU SVM. It

sets the MMU
+ * notifier sequence number and calls the driver invalidate vfunc under
+ * gpusvm->notifier_lock.
+ *
+ * Returns:
+ * true if the operation succeeds, false otherwise.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+static bool
+drm_gpusvm_notifier_invalidate(struct mmu_interval_notifier *mni,
+			       const struct mmu_notifier_range

*mmu_range,

drivers/gpu/drm/xe/drm_gpusvm.c new file +19 -0
+			       unsigned long cur_seq)
+{
+	struct drm_gpusvm_notifier *notifier =
+		container_of(mni, typeof(*notifier), notifier);
+	struct drm_gpusvm *gpusvm = notifier->gpusvm;
+
+	if (!mmu_notifier_range_blockable(mmu_range))
+		return false;
+
+	down_write(&gpusvm->notifier_lock);
+	mmu_interval_set_seq(mni, cur_seq);
+	gpusvm->ops->invalidate(gpusvm, notifier, mmu_range);
+	up_write(&gpusvm->notifier_lock);
+
+	return true;
+}
+
+/**
+ * drm_gpusvm_notifier_ops - MMU interval notifier operations for

GPU SVM
+ */
+static const struct mmu_interval_notifier_ops
drm_gpusvm_notifier_ops = {

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+	.invalidate = drm_gpusvm_notifier_invalidate,
+};
+
+/**
+ * drm_gpusvm_init - Initialize the GPU SVM.
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @name: Name of the GPU SVM.
+ * @drm: Pointer to the DRM device structure.
+ * @mm: Pointer to the mm_struct for the address space.
+ * @device_private_page_owner: Device private pages owner.
+ * @mm_start: Start address of GPU SVM.
+ * @mm_range: Range of the GPU SVM.
+ * @notifier_size: Size of individual notifiers.
+ * @ops: Pointer to the operations structure for GPU SVM.
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range

allocation.
+ *               Entries should be powers of 2 in descending order with last
+ *               entry being SZ_4K.
+ * @num_chunks: Number of chunks.
+ *
+ * This function initializes the GPU SVM.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void

*device_private_page_owner,

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+		    u64 mm_start, u64 mm_range, u64 notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const u64 *chunk_sizes, int num_chunks)
+{
+	if (!ops->invalidate || !num_chunks)
+		return -EINVAL;
+
+	gpusvm->name = name;
+	gpusvm->drm = drm;
+	gpusvm->mm = mm;
+	gpusvm->device_private_page_owner =

device_private_page_owner;

drivers/gpu/drm/xe/drm_gpusvm.c new file +27 -0
+	gpusvm->mm_start = mm_start;
+	gpusvm->mm_range = mm_range;
+	gpusvm->notifier_size = notifier_size;
+	gpusvm->ops = ops;
+	gpusvm->chunk_sizes = chunk_sizes;
+	gpusvm->num_chunks = num_chunks;
+	gpusvm->zdd_wq = system_wq;
+
+	mmgrab(mm);
+	gpusvm->root = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&gpusvm->notifier_list);
+
+	init_rwsem(&gpusvm->notifier_lock);
+
+	fs_reclaim_acquire(GFP_KERNEL);
+	might_lock(&gpusvm->notifier_lock);
+	fs_reclaim_release(GFP_KERNEL);
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_notifier_find - Find GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure
+ * @fault_addr__: Fault address
+ *
+ * This macro finds the GPU SVM notifier associated with the fault

address.
+ *
+ * Returns:
+ * Pointer to the GPU SVM notifier on success, NULL otherwise.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+#define drm_gpusvm_notifier_find(gpusvm__, fault_addr__)	\
+	notifier_iter_first(&(gpusvm__)->root, (fault_addr__),	\
+			    (fault_addr__ + 1))
+
+/**
+ * to_drm_gpusvm_notifier - retrieve the container struct for a

given rbtree node
+ * @node__: a pointer to the rbtree node embedded within a drm_gpusvm_notifier struct
+ *
+ * Return: A pointer to the containing drm_gpusvm_notifier structure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+#define to_drm_gpusvm_notifier(__node)				\
+	container_of((__node), struct drm_gpusvm_notifier, rb.node)
+
+/**
+ * drm_gpusvm_notifier_insert - Insert GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function inserts the GPU SVM notifier into the GPU SVM RB

tree and list.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +2 -0
+static void drm_gpusvm_notifier_insert(struct drm_gpusvm *gpusvm,
+				       struct drm_gpusvm_notifier

*notifier)

drivers/gpu/drm/xe/drm_gpusvm.c new file +21 -0
+{
+	struct rb_node *node;
+	struct list_head *head;
+
+	notifier_insert(notifier, &gpusvm->root);
+
+	node = rb_prev(&notifier->rb.node);
+	if (node)
+		head = &(to_drm_gpusvm_notifier(node))->rb.entry;
+	else
+		head = &gpusvm->notifier_list;
+
+	list_add(&notifier->rb.entry, head);
+}
+
+/**
+ * drm_gpusvm_notifier_remove - Remove GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM tructure
+ * @notifier__: Pointer to the GPU SVM notifier structure
+ *
+ * This macro removes the GPU SVM notifier from the GPU SVM RB tree

and list.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+#define drm_gpusvm_notifier_remove(gpusvm__, notifier__)	\
+	notifier_remove((notifier__), &(gpusvm__)->root);	\
+	list_del(&(notifier__)->rb.entry)
+
+/**
+ * drm_gpusvm_fini - Finalize the GPU SVM.
+ * @gpusvm: Pointer to the GPU SVM structure.
+ *
+ * This function finalizes the GPU SVM by cleaning up any remaining

ranges and
+ * notifiers, and dropping a reference to struct MM.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm)
+{
+	struct drm_gpusvm_notifier *notifier, *next;
+
+	drm_gpusvm_for_each_notifier_safe(notifier, next, gpusvm, 0,

LONG_MAX) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+		struct drm_gpusvm_range *range, *__next;
+
+		/*
+		 * Remove notifier first to avoid racing with any

invalidation

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+		 */
+		mmu_interval_notifier_remove(&notifier->notifier);
+		notifier->flags.removed = true;
+
+		drm_gpusvm_for_each_range_safe(range, __next,

notifier, 0,

drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+					       LONG_MAX)
+			drm_gpusvm_range_remove(gpusvm, range);
+	}
+
+	mmdrop(gpusvm->mm);
+	WARN_ON(!RB_EMPTY_ROOT(&gpusvm->root.rb_root));
+}
+
+/**
+ * drm_gpusvm_notifier_alloc - Allocate GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ *
+ * This function allocates and initializes the GPU SVM notifier

structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM notifier on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+static struct drm_gpusvm_notifier *
+drm_gpusvm_notifier_alloc(struct drm_gpusvm *gpusvm, u64 fault_addr)
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	if (gpusvm->ops->notifier_alloc)
+		notifier = gpusvm->ops->notifier_alloc();
+	else
+		notifier = kzalloc(sizeof(*notifier), GFP_KERNEL);
+
+	if (!notifier)
+		return ERR_PTR(-ENOMEM);
+
+	notifier->gpusvm = gpusvm;
+	notifier->interval.start = ALIGN_DOWN(fault_addr, gpusvm-
Quoted 1 line

notifier_size);

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+	notifier->interval.end = ALIGN(fault_addr + 1, gpusvm-
Quoted 1 line

notifier_size);

drivers/gpu/drm/xe/drm_gpusvm.c new file +16 -0
+	INIT_LIST_HEAD(&notifier->rb.entry);
+	notifier->root = RB_ROOT_CACHED;
+	INIT_LIST_HEAD(&notifier->range_list);
+
+	return notifier;
+}
+
+/**
+ * drm_gpusvm_notifier_free - Free GPU SVM notifier
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ *
+ * This function frees the GPU SVM notifier structure.
+ */
+static void drm_gpusvm_notifier_free(struct drm_gpusvm *gpusvm,
+				     struct drm_gpusvm_notifier

*notifier)

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+{
+	WARN_ON(!RB_EMPTY_ROOT(&notifier->root.rb_root));
+
+	if (gpusvm->ops->notifier_free)
+		gpusvm->ops->notifier_free(notifier);
+	else
+		kfree(notifier);
+}
+
+/**
+ * to_drm_gpusvm_range - retrieve the container struct for a given

rbtree node
+ * @node__: a pointer to the rbtree node embedded within a drm_gpusvm_range struct
+ *
+ * Return: A pointer to the containing drm_gpusvm_range structure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+#define to_drm_gpusvm_range(node__)	\
+	container_of((node__), struct drm_gpusvm_range, rb.node)
+
+/**
+ * drm_gpusvm_range_insert - Insert GPU SVM range
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function inserts the GPU SVM range into the notifier RB tree

and list.
+ */
+static void drm_gpusvm_range_insert(struct drm_gpusvm_notifier *notifier,

drivers/gpu/drm/xe/drm_gpusvm.c new file +24 -0
+				    struct drm_gpusvm_range *range)
+{
+	struct rb_node *node;
+	struct list_head *head;
+
+	drm_gpusvm_notifier_lock(notifier->gpusvm);
+	range_insert(range, &notifier->root);
+
+	node = rb_prev(&range->rb.node);
+	if (node)
+		head = &(to_drm_gpusvm_range(node))->rb.entry;
+	else
+		head = &notifier->range_list;
+
+	list_add(&range->rb.entry, head);
+	drm_gpusvm_notifier_unlock(notifier->gpusvm);
+}
+
+/**
+ * __drm_gpusvm_range_remove - Remove GPU SVM range
+ * @notifier__: Pointer to the GPU SVM notifier structure
+ * @range__: Pointer to the GPU SVM range structure
+ *
+ * This macro removes the GPU SVM range from the notifier RB tree

and list.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +13 -0
+#define __drm_gpusvm_range_remove(notifier__, range__)		\
+	range_remove((range__), &(notifier__)->root);		\
+	list_del(&(range__)->rb.entry)
+
+/**
+ * drm_gpusvm_range_alloc - Allocate GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @fault_addr: Fault address
+ * @chunk_size: Chunk size
+ * @migrate_vram: Flag indicating whether to migrate VRAM
+ *
+ * This function allocates and initializes the GPU SVM range

structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM range on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+static struct drm_gpusvm_range *
+drm_gpusvm_range_alloc(struct drm_gpusvm *gpusvm,
+		       struct drm_gpusvm_notifier *notifier,
+		       u64 fault_addr, u64 chunk_size, bool

migrate_vram)

drivers/gpu/drm/xe/drm_gpusvm.c new file +31 -0
+{
+	struct drm_gpusvm_range *range;
+
+	if (gpusvm->ops->range_alloc)
+		range = gpusvm->ops->range_alloc(gpusvm);
+	else
+		range = kzalloc(sizeof(*range), GFP_KERNEL);
+
+	if (!range)
+		return ERR_PTR(-ENOMEM);
+
+	kref_init(&range->refcount);
+	range->gpusvm = gpusvm;
+	range->notifier = notifier;
+	range->va.start = ALIGN_DOWN(fault_addr, chunk_size);
+	range->va.end = ALIGN(fault_addr + 1, chunk_size);
+	INIT_LIST_HEAD(&range->rb.entry);
+	range->notifier_seq = LONG_MAX;
+	range->flags.migrate_vram = migrate_vram ? 1 : 0;
+
+	return range;
+}
+
+/**
+ * drm_gpusvm_check_pages - Check pages
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @start: Start address
+ * @end: End address
+ *
+ * Check if pages between start and end have been faulted in on the

CPU. Use to
+ * prevent migration of pages without CPU backing store.
+ *
+ * Returns:
+ * True if pages have been faulted into CPU, False otherwise
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +2 -0
+static bool drm_gpusvm_check_pages(struct drm_gpusvm *gpusvm,
+				   struct drm_gpusvm_notifier

*notifier,

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+				   u64 start, u64 end)
+{
+	struct hmm_range hmm_range = {
+		.default_flags = 0,
+		.notifier = &notifier->notifier,
+		.start = start,
+		.end = end,
+		.dev_private_owner = gpusvm-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+	};
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+	unsigned long *pfns;
+	unsigned long npages = npages_in_range(start, end);
+	int err, i;
+
+	mmap_assert_locked(gpusvm->mm);
+
+	pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL);
+	if (!pfns)
+		return false;
+
+	hmm_range.notifier_seq = mmu_interval_read_begin(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+	hmm_range.hmm_pfns = pfns;
+
+	while (true) {
+		err = hmm_range_fault(&hmm_range);
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq =

mmu_interval_read_begin(&notifier->notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +21 -0
+			continue;
+		}
+		break;
+	}
+	if (err)
+		goto err_free;
+
+	for (i = 0; i < npages; ++i) {
+		if (!(pfns[i] & HMM_PFN_VALID)) {
+			err = -EFAULT;
+			goto err_free;
+		}
+	}
+
+err_free:
+	kvfree(pfns);
+	return err ? false : true;
+}
+
+/**
+ * drm_gpusvm_range_chunk_size - Determine chunk size for GPU SVM

range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @vas: Pointer to the virtual memory area structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @check_pages: Flag indicating whether to check pages
+ *
+ * This function determines the chunk size for the GPU SVM range based on the
+ * fault address, GPU SVM chunk sizes, existing GPU SVM ranges, and the virtual
+ * memory area boundaries.
+ *
+ * Returns:
+ * Chunk size on success, LONG_MAX on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +2 -0
+static u64 drm_gpusvm_range_chunk_size(struct drm_gpusvm *gpusvm,
+				       struct drm_gpusvm_notifier

*notifier,

drivers/gpu/drm/xe/drm_gpusvm.c new file +2 -0
+				       struct vm_area_struct *vas,
+				       u64 fault_addr, u64

gpuva_start,

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+				       u64 gpuva_end, bool

check_pages)

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+{
+	u64 start, end;
+	int i = 0;
+
+retry:
+	for (; i < gpusvm->num_chunks; ++i) {
+		start = ALIGN_DOWN(fault_addr, gpusvm-
Quoted 1 line

chunk_sizes[i]);

drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+		end = ALIGN(fault_addr + 1, gpusvm->chunk_sizes[i]);
+
+		if (start >= vas->vm_start && end <= vas->vm_end &&
+		    start >= notifier->interval.start &&
+		    end <= notifier->interval.end &&
+		    start >= gpuva_start && end <= gpuva_end)
+			break;
+	}
+
+	if (i == gpusvm->num_chunks)
+		return LONG_MAX;
+
+	/*
+	 * If allocation more than page, ensure not to overlap with

existing

drivers/gpu/drm/xe/drm_gpusvm.c new file +13 -0
+	 * ranges.
+	 */
+	if (end - start != SZ_4K) {
+		struct drm_gpusvm_range *range;
+
+		range = drm_gpusvm_range_find(notifier, start, end);
+		if (range) {
+			++i;
+			goto retry;
+		}
+
+		/*
+		 * XXX: Only create range on pages CPU has faulted

in. Without

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+		 * this check, or prefault, on BMG

'xe_exec_system_allocator --r

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+		 * process-many-malloc' fails. In the failure case,

each process

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+		 * mallocs 16k but the CPU VMA is ~128k which

results in 64k SVM

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+		 * ranges. When migrating the SVM ranges, some

processes fail in

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+		 * drm_gpusvm_migrate_to_vram with 'migrate.cpages

!= npages'

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+		 * and then upon drm_gpusvm_range_get_pages device

pages from

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+		 * other processes are collected + faulted in which

creates all

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+		 * sorts of problems. Unsure exactly how this

happening, also

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+		 * problem goes away if 'xe_exec_system_allocator --

r

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+		 * process-many-malloc' mallocs at least 64k at a

time.

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+		 */
+		if (check_pages &&
+		    !drm_gpusvm_check_pages(gpusvm, notifier, start,

end)) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +17 -0
+			++i;
+			goto retry;
+		}
+	}
+
+	return end - start;
+}
+
+/**
+ * drm_gpusvm_range_find_or_insert - Find or insert GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @ctx: GPU SVM context
+ *
+ * This function finds or inserts a newly allocated a GPU SVM range

based on the
+ * fault address. Caller must hold a lock to protect range lookup and insertion.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range on success, ERR_PTR() on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +2 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, u64

fault_addr,

drivers/gpu/drm/xe/drm_gpusvm.c new file +31 -0
+				u64 gpuva_start, u64 gpuva_end,
+				const struct drm_gpusvm_ctx *ctx)
+{
+	struct drm_gpusvm_notifier *notifier;
+	struct drm_gpusvm_range *range;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	bool notifier_alloc = false;
+	u64 chunk_size;
+	int err;
+	bool migrate_vram;
+
+	if (fault_addr < gpusvm->mm_start ||
+	    fault_addr > gpusvm->mm_start + gpusvm->mm_range) {
+		err = -EINVAL;
+		goto err_out;
+	}
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		mmap_write_lock(mm);
+	}
+
+	mmap_assert_write_locked(mm);
+
+	notifier = drm_gpusvm_notifier_find(gpusvm, fault_addr);
+	if (!notifier) {
+		notifier = drm_gpusvm_notifier_alloc(gpusvm,

fault_addr);

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+		if (IS_ERR(notifier)) {
+			err = PTR_ERR(notifier);
+			goto err_mmunlock;
+		}
+		notifier_alloc = true;
+		err = mmu_interval_notifier_insert_locked(&notifier-
Quoted 1 line

notifier,

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+							  mm,

notifier->interval.start,

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+							  notifier-
Quoted 1 line

interval.end -

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+							  notifier-
Quoted 1 line

interval.start,

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+							 

&drm_gpusvm_notifier_ops);

drivers/gpu/drm/xe/drm_gpusvm.c new file +16 -0
+		if (err)
+			goto err_notifier;
+	}
+
+	vas = vma_lookup(mm, fault_addr);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_notifier_remove;
+	}
+
+	if (!ctx->read_only && !(vas->vm_flags & VM_WRITE)) {
+		err = -EPERM;
+		goto err_notifier_remove;
+	}
+
+	range = drm_gpusvm_range_find(notifier, fault_addr,

fault_addr + 1);

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+	if (range)
+		goto out_mmunlock;
+	/*
+	 * XXX: Short-circuiting migration based on migrate_vma_*

current

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+	 * limitations. If/when migrate_vma_* add more support, this

logic will

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+	 * have to change.
+	 */
+	migrate_vram = ctx->vram_possible &&
+		vma_is_anonymous(vas) && !is_vm_hugetlb_page(vas);
+
+	chunk_size = drm_gpusvm_range_chunk_size(gpusvm, notifier,

vas,

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+						 fault_addr,

gpuva_start,

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+						 gpuva_end,

migrate_vram &&

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+						 !ctx->prefault);
+	if (chunk_size == LONG_MAX) {
+		err = -EINVAL;
+		goto err_notifier_remove;
+	}
+
+	range = drm_gpusvm_range_alloc(gpusvm, notifier, fault_addr,

chunk_size,

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+				       migrate_vram);
+	if (IS_ERR(range)) {
+		err = PTR_ERR(range);
+		goto err_notifier_remove;
+	}
+
+	drm_gpusvm_range_insert(notifier, range);
+	if (notifier_alloc)
+		drm_gpusvm_notifier_insert(gpusvm, notifier);
+
+	if (ctx->prefault) {
+		struct drm_gpusvm_ctx __ctx = *ctx;
+
+		__ctx.mmap_locked = true;
+		err = drm_gpusvm_range_get_pages(gpusvm, range,

&__ctx);

drivers/gpu/drm/xe/drm_gpusvm.c new file +37 -0
+		if (err)
+			goto err_range_remove;
+	}
+
+out_mmunlock:
+	if (!ctx->mmap_locked) {
+		mmap_write_unlock(mm);
+		mmput(mm);
+	}
+
+	return range;
+
+err_range_remove:
+	__drm_gpusvm_range_remove(notifier, range);
+err_notifier_remove:
+	if (notifier_alloc)
+		mmu_interval_notifier_remove(&notifier->notifier);
+err_notifier:
+	if (notifier_alloc)
+		drm_gpusvm_notifier_free(gpusvm, notifier);
+err_mmunlock:
+	if (!ctx->mmap_locked) {
+		mmap_write_unlock(mm);
+		mmput(mm);
+	}
+err_out:
+	return ERR_PTR(err);
+}
+
+/**
+ * for_each_dma_page - iterate over pages in a DMA regio`n
+ * @i__: the current page index in the iteration
+ * @j__: the current page index, log order, in the iteration
+ * @npages__: the total number of pages in the DMA region
+ * @order__: the order of the pages in the DMA region
+ *
+ * This macro iterates over each page in a DMA region. The DMA

region
+ * is assumed to be composed of 2^@order__ pages, and the macro will
+ * step through the region one block of 2^@order__ pages at a time.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+#define for_each_dma_page(i__, j__, npages__, order__)	\
+	for ((i__) = 0, (j__) = 0; (i__) < (npages__);	\
+	     (j__)++, (i__) += 0x1 << (order__))
+
+/**
+ * __drm_gpusvm_range_unmap_pages - Unmap pages associated with a

GPU SVM range (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function unmap pages associated with a GPU SVM range. Assumes and
+ * asserts correct locking is in place when called.
+ */
+static void __drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+					   struct drm_gpusvm_range

*range)

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+{
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	if (range->pages) {
+		unsigned long i, j, npages = npages_in_range(range-
Quoted 1 line

va.start,

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+							     range-
Quoted 1 line

va.end);

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+
+		if (range->flags.has_dma_mapping) {
+			for_each_dma_page(i, j, npages, range-
Quoted 1 line

order)

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+				dma_unmap_page(gpusvm->drm->dev,
+					       range->dma_addr[j],
+					       PAGE_SIZE << range-
Quoted 1 line

order,

drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+					       DMA_BIDIRECTIONAL);
+		}
+
+		range->flags.has_vram_pages = false;
+		range->flags.has_dma_mapping = false;
+	}
+}
+
+/**
+ * drm_gpusvm_range_free_pages - Free pages associated with a GPU

SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function free pages associated with a GPU SVM range.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +2 -0
+static void drm_gpusvm_range_free_pages(struct drm_gpusvm *gpusvm,
+					struct drm_gpusvm_range

*range)

drivers/gpu/drm/xe/drm_gpusvm.c new file +21 -0
+{
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	if (range->pages) {
+		if (range->flags.kfree_mapping) {
+			kfree(range->dma_addr);
+			range->flags.kfree_mapping = false;
+			range->pages = NULL;
+		} else {
+			kvfree(range->pages);
+			range->pages = NULL;
+		}
+	}
+}
+
+/**
+ * drm_gpusvm_range_remove - Remove GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range to be removed
+ *
+ * This function removes the specified GPU SVM range and also

removes the parent
+ * GPU SVM notifier if no more ranges remain in the notifier. The caller must
+ * hold a lock to protect range and notifier removal.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range)
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	notifier = drm_gpusvm_notifier_find(gpusvm, range-
Quoted 1 line

va.start);

drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+	if (WARN_ON_ONCE(!notifier))
+		return;
+
+	drm_gpusvm_notifier_lock(gpusvm);
+	__drm_gpusvm_range_unmap_pages(gpusvm, range);
+	drm_gpusvm_range_free_pages(gpusvm, range);
+	__drm_gpusvm_range_remove(notifier, range);
+	drm_gpusvm_notifier_unlock(gpusvm);
+
+	drm_gpusvm_range_put(range);
+
+	if (RB_EMPTY_ROOT(&notifier->root.rb_root)) {
+		if (!notifier->flags.removed)
+			mmu_interval_notifier_remove(&notifier-
Quoted 1 line

notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+		drm_gpusvm_notifier_remove(gpusvm, notifier);
+		drm_gpusvm_notifier_free(gpusvm, notifier);
+	}
+}
+
+/**
+ * drm_gpusvm_range_get - Get a reference to GPU SVM range
+ * @range: Pointer to the GPU SVM range
+ *
+ * This function increments the reference count of the specified GPU

SVM range.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range)
+{
+	kref_get(&range->refcount);
+
+	return range;
+}
+
+/**
+ * drm_gpusvm_range_destroy - Destroy GPU SVM range
+ * @refcount: Pointer to the reference counter embedded in the GPU

SVM range
+ *
+ * This function destroys the specified GPU SVM range when its reference count
+ * reaches zero. If a custom range-free function is provided, it is invoked to
+ * free the range; otherwise, the range is deallocated using kfree().
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_range_destroy(struct kref *refcount)
+{
+	struct drm_gpusvm_range *range =
+		container_of(refcount, struct drm_gpusvm_range,

refcount);

drivers/gpu/drm/xe/drm_gpusvm.c new file +13 -0
+	struct drm_gpusvm *gpusvm = range->gpusvm;
+
+	if (gpusvm->ops->range_free)
+		gpusvm->ops->range_free(range);
+	else
+		kfree(range);
+}
+
+/**
+ * drm_gpusvm_range_put - Put a reference to GPU SVM range
+ * @range: Pointer to the GPU SVM range
+ *
+ * This function decrements the reference count of the specified GPU

SVM range
+ * and frees it when the count reaches zero.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range)
+{
+	kref_put(&range->refcount, drm_gpusvm_range_destroy);
+}
+
+/**
+ * drm_gpusvm_range_pages_valid - GPU SVM range pages valid
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid.

Expected be
+ * called holding gpusvm->notifier_lock and as the last step before commiting a
+ * GPU binding.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range)
+{
+	lockdep_assert_held(&gpusvm->notifier_lock);
+
+	return range->flags.has_vram_pages || range-
Quoted 1 line

flags.has_dma_mapping;

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+}
+
+/**
+ * drm_gpusvm_range_pages_valid_unlocked - GPU SVM range pages valid

unlocked
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid. Expected be
+ * called without holding gpusvm->notifier_lock.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+static bool
+drm_gpusvm_range_pages_valid_unlocked(struct drm_gpusvm *gpusvm,
+				      struct drm_gpusvm_range

*range)

drivers/gpu/drm/xe/drm_gpusvm.c new file +25 -0
+{
+	bool pages_valid;
+
+	if (!range->pages)
+		return false;
+
+	drm_gpusvm_notifier_lock(gpusvm);
+	pages_valid = drm_gpusvm_range_pages_valid(gpusvm, range);
+	if (!pages_valid && range->flags.kfree_mapping) {
+		kfree(range->dma_addr);
+		range->flags.kfree_mapping = false;
+		range->pages = NULL;
+	}
+	drm_gpusvm_notifier_unlock(gpusvm);
+
+	return pages_valid;
+}
+
+/**
+ * drm_gpusvm_range_get_pages - Get pages for a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function gets pages for a GPU SVM range and ensures they are

mapped for
+ * DMA access.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	struct mmu_interval_notifier *notifier = &range->notifier-
Quoted 1 line

notifier;

drivers/gpu/drm/xe/drm_gpusvm.c new file +2 -0
+	struct hmm_range hmm_range = {
+		.default_flags = HMM_PFN_REQ_FAULT | (ctx->read_only

? 0 :

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+			HMM_PFN_REQ_WRITE),
+		.notifier = notifier,
+		.start = range->va.start,
+		.end = range->va.end,
+		.dev_private_owner = gpusvm-
Quoted 1 line

device_private_page_owner,

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	unsigned long timeout =
+		jiffies +

msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);

drivers/gpu/drm/xe/drm_gpusvm.c new file +2 -0
+	unsigned long i, j;
+	unsigned long npages = npages_in_range(range->va.start,

range->va.end);

drivers/gpu/drm/xe/drm_gpusvm.c new file +14 -0
+	unsigned int order = 0;
+	unsigned long *pfns;
+	struct page **pages;
+	int err = 0;
+	bool vram_pages = !!range->flags.migrate_vram;
+	bool alloc_pfns = false, kfree_mapping;
+
+retry:
+	kfree_mapping = false;
+	hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
+	if (drm_gpusvm_range_pages_valid_unlocked(gpusvm, range))
+		return 0;
+
+	if (range->notifier_seq == hmm_range.notifier_seq && range-
Quoted 1 line

pages) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+		if (ctx->prefault)
+			return 0;
+
+		pfns = (unsigned long *)range->pages;
+		pages = range->pages;
+		goto map_pages;
+	}
+
+	if (!range->pages) {
+		pfns = kvmalloc_array(npages, sizeof(*pfns),

GFP_KERNEL);

drivers/gpu/drm/xe/drm_gpusvm.c new file +25 -0
+		if (!pfns)
+			return -ENOMEM;
+		alloc_pfns = true;
+	} else {
+		pfns = (unsigned long *)range->pages;
+	}
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+	}
+
+	hmm_range.hmm_pfns = pfns;
+	while (true) {
+		/* Must be checked after mmu_interval_read_begin */
+		if (range->flags.unmapped) {
+			err = -EFAULT;
+			break;
+		}
+
+		if (!ctx->mmap_locked) {
+			/*
+			 * XXX: HMM locking document indicates only

a read-lock

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+			 * is required but there apears to be a

window between

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+			 * the MMU_NOTIFY_MIGRATE event triggered in

a CPU fault

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+			 * via migrate_vma_setup and the pages

actually moving

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+			 * in migrate_vma_finalize in which this

code can grab

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+			 * garbage pages. Grabbing the write-lock if

the range

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+			 * is attached to vram appears to protect

against this

drivers/gpu/drm/xe/drm_gpusvm.c new file +20 -0
+			 * race.
+			 */
+			if (vram_pages)
+				mmap_write_lock(mm);
+			else
+				mmap_read_lock(mm);
+		}
+		err = hmm_range_fault(&hmm_range);
+		if (!ctx->mmap_locked) {
+			if (vram_pages)
+				mmap_write_unlock(mm);
+			else
+				mmap_read_unlock(mm);
+		}
+
+		if (err == -EBUSY) {
+			if (time_after(jiffies, timeout))
+				break;
+
+			hmm_range.notifier_seq =

mmu_interval_read_begin(notifier);

drivers/gpu/drm/xe/drm_gpusvm.c new file +24 -0
+			continue;
+		}
+		break;
+	}
+	if (!ctx->mmap_locked)
+		mmput(mm);
+	if (err)
+		goto err_free;
+
+	pages = (struct page **)pfns;
+
+	if (ctx->prefault) {
+		range->pages = pages;
+		goto set_seqno;
+	}
+
+map_pages:
+	if (is_device_private_page(hmm_pfn_to_page(pfns[0]))) {
+		WARN_ON_ONCE(!range->vram_allocation);
+
+		for (i = 0; i < npages; ++i) {
+			pages[i] = hmm_pfn_to_page(pfns[i]);
+
+			if

(WARN_ON_ONCE(!is_device_private_page(pages[i]))) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+				err = -EOPNOTSUPP;
+				goto err_free;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->flags.has_vram_pages = true;
+		range->pages = pages;
+		if (mmu_interval_read_retry(notifier,

hmm_range.notifier_seq)) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +2 -0
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm,

range);

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	} else {
+		dma_addr_t *dma_addr = (dma_addr_t *)pfns;
+
+		for_each_dma_page(i, j, npages, order) {
+			if (WARN_ON_ONCE(i && order !=
+					

hmm_pfn_to_map_order(pfns[i]))) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+			order = hmm_pfn_to_map_order(pfns[i]);
+
+			pages[j] = hmm_pfn_to_page(pfns[i]);
+			if

(WARN_ON_ONCE(is_zone_device_page(pages[j]))) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+				err = -EOPNOTSUPP;
+				npages = i;
+				goto err_unmap;
+			}
+
+			set_page_dirty_lock(pages[j]);
+			mark_page_accessed(pages[j]);
+
+			dma_addr[j] = dma_map_page(gpusvm->drm->dev,
+						   pages[j], 0,
+						   PAGE_SIZE <<

order,

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+						  

DMA_BIDIRECTIONAL);

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+			if (dma_mapping_error(gpusvm->drm->dev,

dma_addr[j])) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+				err = -EFAULT;
+				npages = i;
+				goto err_unmap;
+			}
+		}
+
+		/* Huge pages, reduce memory footprint */
+		if (order) {
+			dma_addr = kmalloc_array(j,

sizeof(*dma_addr),

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+						 GFP_KERNEL);
+			if (dma_addr) {
+				for (i = 0; i < j; ++i)
+					dma_addr[i] =

(dma_addr_t)pfns[i];

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+				kvfree(pfns);
+				kfree_mapping = true;
+			} else {
+				dma_addr = (dma_addr_t *)pfns;
+			}
+		}
+
+		/* Do not race with notifier unmapping pages */
+		drm_gpusvm_notifier_lock(gpusvm);
+		range->order = order;
+		range->flags.kfree_mapping = kfree_mapping;
+		range->flags.has_dma_mapping = true;
+		range->dma_addr = dma_addr;
+		range->vram_allocation = NULL;
+		if (mmu_interval_read_retry(notifier,

hmm_range.notifier_seq)) {

drivers/gpu/drm/xe/drm_gpusvm.c new file +2 -0
+			err = -EAGAIN;
+			__drm_gpusvm_range_unmap_pages(gpusvm,

range);

drivers/gpu/drm/xe/drm_gpusvm.c new file +16 -0
+		}
+		drm_gpusvm_notifier_unlock(gpusvm);
+	}
+
+	if (err == -EAGAIN)
+		goto retry;
+set_seqno:
+	range->notifier_seq = hmm_range.notifier_seq;
+
+	return 0;
+
+err_unmap:
+	for_each_dma_page(i, j, npages, order)
+		dma_unmap_page(gpusvm->drm->dev,
+			       (dma_addr_t)pfns[j],
+			       PAGE_SIZE << order,

DMA_BIDIRECTIONAL);

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+err_free:
+	if (alloc_pfns)
+		kvfree(pfns);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_range_unmap_pages - Unmap pages associated with a GPU

SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed that gpusvm->notifier_lock is held in write mode; if it
+ * is clear, it acquires gpusvm->notifier_lock in read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops-

Quoted 1 line

invalidate for IOMMU

+ * security model.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +36 -0
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx)
+{
+	if (ctx->in_notifier)
+		lockdep_assert_held_write(&gpusvm->notifier_lock);
+	else
+		drm_gpusvm_notifier_lock(gpusvm);
+
+	__drm_gpusvm_range_unmap_pages(gpusvm, range);
+
+	if (!ctx->in_notifier)
+		drm_gpusvm_notifier_unlock(gpusvm);
+}
+
+/**
+ * drm_gpusvm_migration_put_page - Put a migration page
+ * @page: Pointer to the page to put
+ *
+ * This function unlocks and puts a page.
+ */
+static void drm_gpusvm_migration_put_page(struct page *page)
+{
+	unlock_page(page);
+	put_page(page);
+}
+
+/**
+ * drm_gpusvm_migration_put_pages - Put migration pages
+ * @npages: Number of pages
+ * @migrate_pfn: Array of migrate page frame numbers
+ *
+ * This function puts an array of pages.
+ */
+static void drm_gpusvm_migration_put_pages(unsigned long npages,
+					   unsigned long

*migrate_pfn)

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!migrate_pfn[i])
+			continue;
+
+		drm_gpusvm_migration_put_page(migrate_pfn_to_page(mi

grate_pfn[i]));

drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+		migrate_pfn[i] = 0;
+	}
+}
+
+/**
+ * drm_gpusvm_get_vram_page - Get a reference to a VRAM page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU

SVM zone
+ * device data and initializes it for zone device usage.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+static void drm_gpusvm_get_vram_page(struct page *page,
+				     struct drm_gpusvm_zdd *zdd)
+{
+	page->zone_device_data = drm_gpusvm_zdd_get(zdd);
+	zone_device_page_init(page);
+}
+
+/**
+ * drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM

migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Return: 0 on success, -EFAULT if an error occurs during mapping.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +3 -0
+static int drm_gpusvm_migrate_map_pages(struct device *dev,
+					dma_addr_t *dma_addr,
+					long unsigned int

*migrate_pfn,

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+					unsigned long npages,
+					enum dma_data_direction dir)
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page =

migrate_pfn_to_page(migrate_pfn[i]);

drivers/gpu/drm/xe/drm_gpusvm.c new file +8 -0
+
+		if (!page)
+			continue;
+
+		if (WARN_ON_ONCE(is_zone_device_page(page)))
+			return -EFAULT;
+
+		dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE,

dir);

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+		if (dma_mapping_error(dev, dma_addr[i]))
+			return -EFAULT;
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped

for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g., DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+static void drm_gpusvm_migrate_unmap_pages(struct device *dev,
+					   dma_addr_t *dma_addr,
+					   unsigned long npages,
+					   enum dma_data_direction

dir)

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i) {
+		if (!dma_addr[i] || dma_mapping_error(dev,

dma_addr[i]))

drivers/gpu/drm/xe/drm_gpusvm.c new file +12 -0
+			continue;
+
+		dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir);
+	}
+}
+
+/**
+ * drm_gpusvm_migrate_to_vram - Migrate GPU SVM range to VRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *                   failure of this function.
+ * @vram_allocation: Driver-private pointer to the VRAM allocation.

The caller
+ *                   should hold a reference to the VRAM allocation, which
+ *                   should be dropped via ops->vram_allocation or upon the
+ *                   failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to VRAM. It performs the
+ * necessary setup and invokes the driver-specific operations for migration to
+ * VRAM. Upon successful return, @vram_allocation can safely reference @range
+ * until ops->vram_release is called which only upon successful return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +25 -0
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct migrate_vma migrate = {
+		.start		= start,
+		.end		= end,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_SYSTEM,
+	};
+	struct mm_struct *mm = gpusvm->mm;
+	unsigned long i, npages = npages_in_range(start, end);
+	struct vm_area_struct *vas;
+	struct drm_gpusvm_zdd *zdd = NULL;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int err;
+
+	if (!range->flags.migrate_vram)
+		return -EINVAL;
+
+	if (!gpusvm->ops->populate_vram_pfn || !gpusvm->ops-
Quoted 1 line

copy_to_vram ||

drivers/gpu/drm/xe/drm_gpusvm.c new file +30 -0
+	    !gpusvm->ops->copy_to_sram)
+		return -EOPNOTSUPP;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		mmap_write_lock(mm);
+	}
+
+	mmap_assert_locked(mm);
+
+	vas = vma_lookup(mm, start);
+	if (!vas) {
+		err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end > vas->vm_end || start < vas->vm_start) {
+		err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	if (!vma_is_anonymous(vas)) {
+		err = -EBUSY;
+		goto err_mmunlock;
+	}
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +

sizeof(*dma_addr) +

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_mmunlock;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr))

* npages;

drivers/gpu/drm/xe/drm_gpusvm.c new file +17 -0
+
+	zdd = drm_gpusvm_zdd_alloc(range);
+	if (!zdd) {
+		err = -ENOMEM;
+		goto err_free;
+	}
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/*
+	 * FIXME: Below cases, !migrate.cpages and migrate.cpages !=

npages, not

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+	 * always an error. Need to revisit possible cases and how

to handle. We

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+	 * could prefault on migrate.cpages != npages via

hmm_range_fault.

drivers/gpu/drm/xe/drm_gpusvm.c new file +13 -0
+	 */
+
+	if (!migrate.cpages) {
+		err = -EFAULT;
+		goto err_free;
+	}
+
+	if (migrate.cpages != npages) {
+		err = -EBUSY;
+		goto err_finalize;
+	}
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm,

vram_allocation, npages,

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+					     migrate.dst);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev,

dma_addr,

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+					   migrate.src, npages,

DMA_TO_DEVICE);

drivers/gpu/drm/xe/drm_gpusvm.c new file +12 -0
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i) {
+		struct page *page = pfn_to_page(migrate.dst[i]);
+
+		pages[i] = page;
+		migrate.dst[i] = migrate_pfn(migrate.dst[i]);
+		drm_gpusvm_get_vram_page(page, zdd);
+	}
+
+	err = gpusvm->ops->copy_to_vram(gpusvm, pages, dma_addr,

npages);

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+	if (err)
+		goto err_finalize;
+
+	/* Upon success bind vram allocation to range and zdd */
+	range->vram_allocation = vram_allocation;
+	WRITE_ONCE(zdd->vram_allocation, vram_allocation);	/*

Owns ref */

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr,

npages,

drivers/gpu/drm/xe/drm_gpusvm.c new file +16 -0
+				       DMA_TO_DEVICE);
+err_free:
+	if (zdd)
+		drm_gpusvm_zdd_put(zdd);
+	kvfree(buf);
+err_mmunlock:
+	if (!ctx->mmap_locked) {
+		mmap_write_unlock(mm);
+		mmput(mm);
+	}
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_populate_sram_pfn - Populate SRAM PFNs for a

VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @npages: Number of pages to populate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the SRAM migrate page frame numbers (PFNs) for the
+ * specified VM area structure. It allocates and locks pages in the VM area for
+ * SRAM usage. If vas is non-NULL use alloc_page_vma for allocation, if NULL use
+ * alloc_page for allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_sram_pfn(struct vm_area_struct *vas,

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+						unsigned long

npages,

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+						unsigned long

*src_mpfn,

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+						unsigned long *mpfn,

u64 addr)

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+{
+	unsigned long i;
+
+	for (i = 0; i < npages; ++i, addr += PAGE_SIZE) {
+		struct page *page;
+
+		if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE))
+			continue;
+
+		if (vas)
+			page = alloc_page_vma(GFP_HIGHUSER, vas,

addr);

drivers/gpu/drm/xe/drm_gpusvm.c new file +19 -0
+		else
+			page = alloc_page(GFP_HIGHUSER);
+
+		if (!page)
+			return -ENOMEM;
+
+		lock_page(page);
+		mpfn[i] = migrate_pfn(page_to_pfn(page));
+	}
+
+	return 0;
+}
+
+/**
+ * drm_gpusvm_evict_to_sram - Evict GPU SVM range to SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * Similar to __drm_gpusvm_migrate_to_sram but does not require mmap

lock and
+ * migration done via migrate_device_* functions. Fallback path as it is
+ * preferred to issue migrations with mmap lock.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +23 -0
+static int drm_gpusvm_evict_to_sram(struct drm_gpusvm *gpusvm,
+				    struct drm_gpusvm_range *range)
+{
+	unsigned long npages;
+	struct page **pages;
+	unsigned long *src, *dst;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	npages = npages_in_range(range->va.start, range->va.end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr)
+
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	src = buf;
+	dst = buf + (sizeof(*src) * npages);
+	dma_addr = buf + (2 * sizeof(*src) * npages);
+	pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) *

npages;

drivers/gpu/drm/xe/drm_gpusvm.c new file +2 -0
+
+	err = gpusvm->ops->populate_vram_pfn(gpusvm, range-
Quoted 1 line

vram_allocation,

drivers/gpu/drm/xe/drm_gpusvm.c new file +6 -0
+					     npages, src);
+	if (err)
+		goto err_free;
+
+	err = migrate_device_vma_range(gpusvm->mm,
+				       gpusvm-
Quoted 1 line

device_private_page_owner, src,

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+				       npages, range->va.start);
+	if (err)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(NULL, npages,

src, dst, 0);

drivers/gpu/drm/xe/drm_gpusvm.c new file +4 -0
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev,

dma_addr,

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+					   dst, npages,

DMA_BIDIRECTIONAL);

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr,

npages);

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, dst);
+	migrate_device_pages(src, dst, npages);
+	migrate_device_finalize(src, dst, npages);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr,

npages,

drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+
+	return err;
+}
+
+/**
+ * __drm_gpusvm_migrate_to_sram - Migrate GPU SVM range to SRAM

(internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @vas: Pointer to the VM area structure
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @start: Start address of the migration range
+ * @end: End address of the migration range
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to SRAM. It sets up the migration, populates + dma maps SRAM PFNs, and
+ * invokes the driver-specific operations for migration to SRAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +20 -0
+static int __drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+					struct vm_area_struct *vas,
+					struct page *page,
+					u64 start, u64 end)
+{
+	struct migrate_vma migrate = {
+		.vma		= vas,
+		.pgmap_owner	= gpusvm->device_private_page_owner,
+		.flags		= MIGRATE_VMA_SELECT_DEVICE_PRIVATE,
+		.fault_page	= page,
+	};
+	unsigned long npages;
+	struct page **pages;
+	dma_addr_t *dma_addr;
+	void *buf;
+	int i, err = 0;
+
+	mmap_assert_locked(gpusvm->mm);
+
+	/* Corner where VMA area struct has been partially unmapped

*/

drivers/gpu/drm/xe/drm_gpusvm.c new file +10 -0
+	if (start < vas->vm_start)
+		start = vas->vm_start;
+	if (end > vas->vm_end)
+		end = vas->vm_end;
+
+	migrate.start = start;
+	migrate.end = end;
+	npages = npages_in_range(start, end);
+
+	buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +

sizeof(*dma_addr) +

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+		       sizeof(*pages), GFP_KERNEL);
+	if (!buf) {
+		err = -ENOMEM;
+		goto err_out;
+	}
+	dma_addr = buf + (2 * sizeof(*migrate.src) * npages);
+	pages = buf + (2 * sizeof(*migrate.src) + sizeof(*dma_addr))

* npages;

drivers/gpu/drm/xe/drm_gpusvm.c new file +15 -0
+
+	migrate.vma = vas;
+	migrate.src = buf;
+	migrate.dst = migrate.src + npages;
+
+	err = migrate_vma_setup(&migrate);
+	if (err)
+		goto err_free;
+
+	/* Raced with another CPU fault, nothing to do */
+	if (!migrate.cpages)
+		goto err_free;
+
+	err = drm_gpusvm_migrate_populate_sram_pfn(vas, npages,
+						   migrate.src,

migrate.dst,

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+						   start);
+	if (err)
+		goto err_finalize;
+
+	err = drm_gpusvm_migrate_map_pages(gpusvm->drm->dev,

dma_addr,

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+					   migrate.dst, npages,
+					   DMA_BIDIRECTIONAL);
+	if (err)
+		goto err_finalize;
+
+	for (i = 0; i < npages; ++i)
+		pages[i] = migrate_pfn_to_page(migrate.src[i]);
+
+	err = gpusvm->ops->copy_to_sram(gpusvm, pages, dma_addr,

npages);

drivers/gpu/drm/xe/drm_gpusvm.c new file +9 -0
+	if (err)
+		goto err_finalize;
+
+err_finalize:
+	if (err)
+		drm_gpusvm_migration_put_pages(npages, migrate.dst);
+	migrate_vma_pages(&migrate);
+	migrate_vma_finalize(&migrate);
+	drm_gpusvm_migrate_unmap_pages(gpusvm->drm->dev, dma_addr,

npages,

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+				       DMA_BIDIRECTIONAL);
+err_free:
+	kvfree(buf);
+err_out:
+	mmap_assert_locked(gpusvm->mm);
+
+	return err;
+}
+
+/**
+ * drm_gpusvm_migrate_to_sram - Migrate (evict) GPU SVM range to

SRAM
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function initiates the migration of the specified GPU SVM range to
+ * SRAM. It performs necessary checks and invokes the internal migration
+ * function for actual migration.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +18 -0
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx)
+{
+	u64 start = range->va.start, end = range->va.end;
+	struct mm_struct *mm = gpusvm->mm;
+	struct vm_area_struct *vas;
+	int err;
+	bool retry = false;
+
+	if (!ctx->mmap_locked) {
+		if (!mmget_not_zero(mm)) {
+			err = -EFAULT;
+			goto err_out;
+		}
+		if (ctx->trylock_mmap) {
+			if (!mmap_read_trylock(mm))  {
+				err =

drm_gpusvm_evict_to_sram(gpusvm, range);

drivers/gpu/drm/xe/drm_gpusvm.c new file +11 -0
+				goto err_mmput;
+			}
+		} else {
+			mmap_read_lock(mm);
+		}
+	}
+
+	mmap_assert_locked(mm);
+
+	/*
+	 * Loop required to find all VMA area structs for the corner

case when

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+	 * VRAM backing has been partially unmapped from MM's

address space.

drivers/gpu/drm/xe/drm_gpusvm.c new file +16 -0
+	 */
+again:
+	vas = find_vma(mm, start);
+	if (!vas) {
+		if (!retry)
+			err = -ENOENT;
+		goto err_mmunlock;
+	}
+
+	if (end <= vas->vm_start || start >= vas->vm_end) {
+		if (!retry)
+			err = -EINVAL;
+		goto err_mmunlock;
+	}
+
+	err = __drm_gpusvm_migrate_to_sram(gpusvm, vas, NULL, start,

end);

drivers/gpu/drm/xe/drm_gpusvm.c new file +13 -0
+	if (err)
+		goto err_mmunlock;
+
+	if (vas->vm_end < end) {
+		retry = true;
+		start = vas->vm_end;
+		goto again;
+	}
+
+	if (!ctx->mmap_locked) {
+		mmap_read_unlock(mm);
+		/*
+		 * Using mmput_async as this function can be called

while

drivers/gpu/drm/xe/drm_gpusvm.c new file +1 -0
+		 * holding a dma-resv lock, and a final put can grab

the mmap

drivers/gpu/drm/xe/drm_gpusvm.c new file +19 -0
+		 * lock, causing a lock inversion.
+		 */
+		mmput_async(mm);
+	}
+
+	return 0;
+
+err_mmunlock:
+	if (!ctx->mmap_locked)
+		mmap_read_unlock(mm);
+err_mmput:
+	if (!ctx->mmap_locked)
+		mmput_async(mm);
+err_out:
+	return err;
+}
+
+/**
+ * drm_gpusvm_page_free - Put GPU SVM zone device data associated

with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device data
+ * associated with a page when it is being released.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+static void drm_gpusvm_page_free(struct page *page)
+{
+	drm_gpusvm_zdd_put(page->zone_device_data);
+}
+
+/**
+ * drm_gpusvm_migrate_to_ram - Migrate GPU SVM range to RAM (page

fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ * VM_FAULT_SIGBUS on failure, 0 on success.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +23 -0
+static vm_fault_t drm_gpusvm_migrate_to_ram(struct vm_fault *vmf)
+{
+	struct drm_gpusvm_zdd *zdd = vmf->page->zone_device_data;
+	int err;
+
+	err = __drm_gpusvm_migrate_to_sram(zdd->range->gpusvm,
+					   vmf->vma, vmf->page,
+					   zdd->range->va.start,
+					   zdd->range->va.end);
+
+	return err ? VM_FAULT_SIGBUS : 0;
+}
+
+/**
+ * drm_gpusvm_pagemap_ops - Device page map operations for GPU SVM
+ */
+static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = {
+	.page_free = drm_gpusvm_page_free,
+	.migrate_to_ram = drm_gpusvm_migrate_to_ram,
+};
+
+/**
+ * drm_gpusvm_pagemap_ops_get - Retrieve GPU SVM device page map

operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void)
+{
+	return &drm_gpusvm_pagemap_ops;
+}
+
+/**
+ * drm_gpusvm_has_mapping - Check if GPU SVM has mapping for the

given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start, u64 end)

drivers/gpu/drm/xe/drm_gpusvm.c new file +7 -0
+{
+	struct drm_gpusvm_notifier *notifier;
+
+	drm_gpusvm_for_each_notifier(notifier, gpusvm, start, end) {
+		struct drm_gpusvm_range *range = NULL;
+
+		drm_gpusvm_for_each_range(range, notifier, start,

end)

drivers/gpu/drm/xe/drm_gpusvm.c new file +5 -0
+			return true;
+	}
+
+	return false;
+}
drivers/gpu/drm/xe/drm_gpusvm.h new file +23 -0
new file mode 100644
index 000000000000..0ea70f8534a8
@@ -0,0 +1,415 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2024 Intel Corporation
+ */
+
+#ifndef __DRM_GPUSVM_H__
+#define __DRM_GPUSVM_H__
+
+#include <linux/kref.h>
+#include <linux/mmu_notifier.h>
+#include <linux/workqueue.h>
+
+struct dev_pagemap_ops;
+struct drm_device;
+struct drm_gpusvm;
+struct drm_gpusvm_notifier;
+struct drm_gpusvm_ops;
+struct drm_gpusvm_range;
+
+/**
+ * struct drm_gpusvm_ops - Operations structure for GPU SVM
+ *
+ * This structure defines the operations for GPU Shared Virtual

Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * perform operations such as migration between VRAM and system RAM.
+ */

drivers/gpu/drm/xe/drm_gpusvm.h new file +8 -0
+struct drm_gpusvm_ops {
+	/**
+	 * @notifier_alloc: Allocate a GPU SVM notifier (optional)
+	 *
+	 * This function shall allocate a GPU SVM notifier.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM notifier on success,

NULL on failure.

drivers/gpu/drm/xe/drm_gpusvm.h new file +19 -0
+	 */
+	struct drm_gpusvm_notifier *(*notifier_alloc)(void);
+
+	/**
+	 * @notifier_free: Free a GPU SVM notifier (optional)
+	 * @notifier: Pointer to the GPU SVM notifier to be freed
+	 *
+	 * This function shall free a GPU SVM notifier.
+	 */
+	void (*notifier_free)(struct drm_gpusvm_notifier *notifier);
+
+	/**
+	 * @range_alloc: Allocate a GPU SVM range (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 *
+	 * This function shall allocate a GPU SVM range.
+	 *
+	 * Returns:
+	 * Pointer to the allocated GPU SVM range on success, NULL

on failure.

drivers/gpu/drm/xe/drm_gpusvm.h new file +2 -0
+	 */
+	struct drm_gpusvm_range *(*range_alloc)(struct drm_gpusvm

*gpusvm);

drivers/gpu/drm/xe/drm_gpusvm.h new file +12 -0
+
+	/**
+	 * @range_free: Free a GPU SVM range (optional)
+	 * @range: Pointer to the GPU SVM range to be freed
+	 *
+	 * This function shall free a GPU SVM range.
+	 */
+	void (*range_free)(struct drm_gpusvm_range *range);
+
+	/**
+	 * @vram_release: Release VRAM allocation (optional)
+	 * @vram_allocation: Driver-private pointer to the VRAM

allocation

drivers/gpu/drm/xe/drm_gpusvm.h new file +2 -0
+	 *
+	 * This function shall release VRAM allocation and expects

to drop a

drivers/gpu/drm/xe/drm_gpusvm.h new file +6 -0
+	 * reference to VRAM allocation.
+	 */
+	void (*vram_release)(void *vram_allocation);
+
+	/**
+	 * @populate_vram_pfn: Populate VRAM PFN (required for

migration)

drivers/gpu/drm/xe/drm_gpusvm.h new file +2 -0
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @vram_allocation: Driver-private pointer to the VRAM

allocation

drivers/gpu/drm/xe/drm_gpusvm.h new file +4 -0
+	 * @npages: Number of pages to populate
+	 * @pfn: Array of page frame numbers to populate
+	 *
+	 * This function shall populate VRAM page frame numbers

(PFN).

drivers/gpu/drm/xe/drm_gpusvm.h new file +28 -0
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*populate_vram_pfn)(struct drm_gpusvm *gpusvm,
+				 void *vram_allocation,
+				 unsigned long npages,
+				 unsigned long *pfn);
+
+	/**
+	 * @copy_to_vram: Copy to VRAM (required for migration)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (destination)
+	 * @dma_addr: Pointer to array of DMA addresses (source)
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to VRAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_vram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @copy_to_sram: Copy to system RAM (required for

migration)

drivers/gpu/drm/xe/drm_gpusvm.h new file +3 -0
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @pages: Pointer to array of VRAM pages (source)
+	 * @dma_addr: Pointer to array of DMA addresses

(destination)

drivers/gpu/drm/xe/drm_gpusvm.h new file +19 -0
+	 * @npages: Number of pages to copy
+	 *
+	 * This function shall copy pages to system RAM.
+	 *
+	 * Returns:
+	 * 0 on success, a negative error code on failure.
+	 */
+	int (*copy_to_sram)(struct drm_gpusvm *gpusvm,
+			    struct page **pages,
+			    dma_addr_t *dma_addr,
+			    unsigned long npages);
+
+	/**
+	 * @invalidate: Invalidate GPU SVM notifier (required)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @notifier: Pointer to the GPU SVM notifier
+	 * @mmu_range: Pointer to the mmu_notifier_range structure
+	 *
+	 * This function shall invalidate the GPU page tables. It

can safely

drivers/gpu/drm/xe/drm_gpusvm.h new file +1 -0
+	 * walk the notifier range RB tree/list in this function.

Called while

drivers/gpu/drm/xe/drm_gpusvm.h new file +5 -0
+	 * holding the notifier lock.
+	 */
+	void (*invalidate)(struct drm_gpusvm *gpusvm,
+			   struct drm_gpusvm_notifier *notifier,
+			   const struct mmu_notifier_range

*mmu_range);

drivers/gpu/drm/xe/drm_gpusvm.h new file +4 -0
+};
+
+/**
+ * struct drm_gpusvm_notifier - Structure representing a GPU SVM

notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @interval: Interval for the notifier
+ * @rb: Red-black tree node for the parent GPU SVM structure notifier tree
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ *              interval tree. This is useful to keep iterating ranges while
+ *              doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier has been
+ *                 removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */

drivers/gpu/drm/xe/drm_gpusvm.h new file +26 -0
+struct drm_gpusvm_notifier {
+	struct drm_gpusvm *gpusvm;
+	struct mmu_interval_notifier notifier;
+	struct {
+		u64 start;
+		u64 end;
+	} interval;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct rb_root_cached root;
+	struct list_head range_list;
+	struct {
+		u32 removed : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm_range - Structure representing a GPU SVM range
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier
+ * @refcount: Reference count for the range
+ * @rb: Red-black tree node for the parent GPU SVM notifier

structure range tree
+ * @va: Virtual address range
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @pages: Pointer to the array of pages (if backing store is in VRAM)
+ * @dma_addr: DMA address array (if backing store is SRAM and DMA mapped)
+ * @vram_allocation: Driver-private pointer to the VRAM allocation
+ * @order: Order of dma mapping. i.e. PAGE_SIZE << order is mapping size
+ * @flags.migrate_vram: Flag indicating whether the range can be migrated to VRAM
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_vram_pages: Flag indicating if the range has vram pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA mapping
+ * @flags.kfree_mapping: Flag indicating @dma_addr is a compact allocation based
+ *                       on @order which releases via kfree
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */

drivers/gpu/drm/xe/drm_gpusvm.h new file +24 -0
+struct drm_gpusvm_range {
+	struct drm_gpusvm *gpusvm;
+	struct drm_gpusvm_notifier *notifier;
+	struct kref refcount;
+	struct {
+		struct rb_node node;
+		struct list_head entry;
+		u64 __subtree_last;
+	} rb;
+	struct {
+		u64 start;
+		u64 end;
+	} va;
+	unsigned long notifier_seq;
+	union {
+		struct page **pages;
+		dma_addr_t *dma_addr;
+	};
+	void *vram_allocation;
+	u16 order;
+	struct {
+		/* All flags below must be set upon creation */
+		u16 migrate_vram : 1;
+		/* All flags below must be set / cleared under

notifier lock */

drivers/gpu/drm/xe/drm_gpusvm.h new file +20 -0
+		u16 unmapped : 1;
+		u16 partial_unmap : 1;
+		u16 has_vram_pages : 1;
+		u16 has_dma_mapping : 1;
+		u16 kfree_mapping : 1;
+	} flags;
+};
+
+/**
+ * struct drm_gpusvm - GPU SVM structure
+ *
+ * @name: Name of the GPU SVM
+ * @drm: Pointer to the DRM device structure
+ * @mm: Pointer to the mm_struct for the address space
+ * @device_private_page_owner: Device private pages owner
+ * @mm_start: Start address of GPU SVM
+ * @mm_range: Range of the GPU SVM
+ * @notifier_size: Size of individual notifiers
+ * @ops: Pointer to the operations structure for GPU SVM
+ * @chunk_sizes: Pointer to the array of chunk sizes used in range

allocation.
+ *               Entries should be powers of 2 in descending order.
+ * @num_chunks: Number of chunks
+ * @notifier_lock: Read-write semaphore for protecting notifier operations
+ * @zdd_wq: Workqueue for deferred work on zdd destruction
+ * @root: Cached root node of the Red-Black tree containing GPU SVM notifiers
+ * @notifier_list: list head containing of notifiers in the same order they
+ *                 appear in interval tree. This is useful to keep iterating
+ *                 notifiers while doing modifications to RB tree.
+ *
+ * This structure represents a GPU SVM (Shared Virtual Memory) used for tracking
+ * memory ranges mapped in a DRM (Direct Rendering Manager) device.
+ *
+ * No reference counting is provided, as this is expected to be embedded in the
+ * driver VM structure along with the struct drm_gpuvm, which handles reference
+ * counting.
+ */

drivers/gpu/drm/xe/drm_gpusvm.h new file +22 -0
+struct drm_gpusvm {
+	const char *name;
+	struct drm_device *drm;
+	struct mm_struct *mm;
+	void *device_private_page_owner;
+	u64 mm_start;
+	u64 mm_range;
+	u64 notifier_size;
+	const struct drm_gpusvm_ops *ops;
+	const u64 *chunk_sizes;
+	int num_chunks;
+	struct rw_semaphore notifier_lock;
+	struct workqueue_struct *zdd_wq;
+	struct rb_root_cached root;
+	struct list_head notifier_list;
+};
+
+/**
+ * struct drm_gpusvm_ctx - DRM GPU SVM context
+ *
+ * @mmap_locked: mmap lock is locked
+ * @trylock_mmap: trylock mmap lock, used to avoid locking

inversions
+ *                (e.g.dma-revs -> mmap lock)
+ * @in_notifier: entering from a MMU notifier
+ * @read_only: operating on read-only memory
+ * @vram_possible: possible to use VRAM
+ * @prefault: prefault pages
+ *
+ * Context that is DRM GPUSVM is operating in (i.e. user arguments).
+ */

drivers/gpu/drm/xe/drm_gpusvm.h new file +12 -0
+struct drm_gpusvm_ctx {
+	u32 mmap_locked :1;
+	u32 trylock_mmap :1;
+	u32 in_notifier :1;
+	u32 read_only :1;
+	u32 vram_possible :1;
+	u32 prefault :1;
+};
+
+int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
+		    const char *name, struct drm_device *drm,
+		    struct mm_struct *mm, void

*device_private_page_owner,

drivers/gpu/drm/xe/drm_gpusvm.h new file +8 -0
+		    u64 mm_start, u64 mm_range, u64 notifier_size,
+		    const struct drm_gpusvm_ops *ops,
+		    const u64 *chunk_sizes, int num_chunks);
+void drm_gpusvm_fini(struct drm_gpusvm *gpusvm);
+void drm_gpusvm_free(struct drm_gpusvm *gpusvm);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find_or_insert(struct drm_gpusvm *gpusvm, u64

fault_addr,

drivers/gpu/drm/xe/drm_gpusvm.h new file +30 -0
+				u64 gpuva_start, u64 gpuva_end,
+				const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_remove(struct drm_gpusvm *gpusvm,
+			     struct drm_gpusvm_range *range);
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_get(struct drm_gpusvm_range *range);
+void drm_gpusvm_range_put(struct drm_gpusvm_range *range);
+
+bool drm_gpusvm_range_pages_valid(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range);
+
+int drm_gpusvm_range_get_pages(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+void drm_gpusvm_range_unmap_pages(struct drm_gpusvm *gpusvm,
+				  struct drm_gpusvm_range *range,
+				  const struct drm_gpusvm_ctx *ctx);
+
+int drm_gpusvm_migrate_to_vram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       void *vram_allocation,
+			       const struct drm_gpusvm_ctx *ctx);
+int drm_gpusvm_migrate_to_sram(struct drm_gpusvm *gpusvm,
+			       struct drm_gpusvm_range *range,
+			       const struct drm_gpusvm_ctx *ctx);
+
+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get(void);
+
+bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, u64 start,

u64 end);

drivers/gpu/drm/xe/drm_gpusvm.h new file +3 -0
+
+struct drm_gpusvm_range *
+drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, u64

start, u64 end);

drivers/gpu/drm/xe/drm_gpusvm.h new file +24 -0
+
+/**
+ * drm_gpusvm_notifier_lock - Lock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, take lock
+ */
+#define drm_gpusvm_notifier_lock(gpusvm__)	\
+	down_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * drm_gpusvm_notifier_unlock - Unlock GPU SVM notifier
+ * @gpusvm__: Pointer to the GPU SVM structure.
+ *
+ * Abstract client usage GPU SVM notifier lock, drop lock
+ */
+#define drm_gpusvm_notifier_unlock(gpusvm__)	\
+	up_read(&(gpusvm__)->notifier_lock)
+
+/**
+ * __drm_gpusvm_range_next - Get the next GPU SVM range in the list
+ * @range: a pointer to the current GPU SVM range
+ *
+ * Return: A pointer to the next drm_gpusvm_range if available, or

NULL if the
+ *         current range is the last one or if the input range is NULL.
+ */

drivers/gpu/drm/xe/drm_gpusvm.h new file +12 -0
+static inline struct drm_gpusvm_range *
+__drm_gpusvm_range_next(struct drm_gpusvm_range *range)
+{
+	if (range && !list_is_last(&range->rb.entry,
+				   &range->notifier->range_list))
+		return list_next_entry(range, rb.entry);
+
+	return NULL;
+}
+
+/**
+ * drm_gpusvm_for_each_range - Iterate over GPU SVM ranges in a

notifier
+ * @range__: Iterator variable for the ranges. If set, it indicates the start of
+ *      the iterator. If NULL, call drm_gpusvm_range_find() to
get the range.
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier. It is safe
+ * to use while holding the driver SVM lock or the notifier lock.
+ */
+#define drm_gpusvm_for_each_range(range__, notifier__, start__,
end__) \

drivers/gpu/drm/xe/drm_gpusvm.h new file +1 -0
+	for ((range__) = (range__)

?: \

drivers/gpu/drm/xe/drm_gpusvm.h new file +1 -0
+	     drm_gpusvm_range_find((notifier__), (start__),

(end__)); \

drivers/gpu/drm/xe/drm_gpusvm.h new file +1 -0
+	     (range__) && (range__->va.start <

(end__)); \

drivers/gpu/drm/xe/drm_gpusvm.h new file +8 -0
+	     (range__) = __drm_gpusvm_range_next(range__))
+
+/**
+ * drm_gpusvm_range_set_unmapped - Mark a GPU SVM range as unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets the

partial_unmap flag
+ * if the range partially falls within the provided MMU notifier range.
+ */

drivers/gpu/drm/xe/drm_gpusvm.h new file +3 -0
+static inline void
+drm_gpusvm_range_set_unmapped(struct drm_gpusvm_range *range,
+			      const struct mmu_notifier_range

*mmu_range)

drivers/gpu/drm/xe/drm_gpusvm.h new file +10 -0
+{
+	lockdep_assert_held_write(&range->gpusvm->notifier_lock);
+
+	range->flags.unmapped = true;
+	if (range->va.start < mmu_range->start ||
+	    range->va.end > mmu_range->end)
+		range->flags.partial_unmap = true;
+}
+
+#endif /* __DRM_GPUSVM_H__ */