Basic system allocator support in xe driver
Subsystems (4)
v2 cover Basic system allocator support in xe driver
This is the v2 of basic system allocator support in xe kmd driver.
v1 is here: https://lore.kernel.org/dri-devel/[email protected]/
Significant design changes were made since v1, based on drm community review feedback:
1) Introduce vm_bind uAPI for system allocator. With this uAPI, user can optionally bind CPU virtual address range A..B to GPU virtual address range C..D. Right now we force A..B == C..D since we don't have a valid use case where A..B != C..D. But the interface is built so we can extend easily in the future if valid use case come out. See patch 3 to 8 for this work.
2) Unify system allocator and user ptr code. Now system allocator and userptr share the same codes for gpu page table programming, mmu interval notifier and vma invalidation, page fault handling and lock design. The codes are more unified.
This work is built on top of Matt Brost's huge vm_bind refactor series. The first patch is a squash of Matt's 30 patch series for reference purpose.
This work is still at its early stage. It is sent out so we can get some early eyes on it. We are open to any comments and suggestions.
The work that are planned in our bucket are:
*Virtual address range based memory attributes and hints: We plan to expose uAPI for user to set memory attributes such as preferred location or migration granularity etc to a virtual address range. This is important to tune SVM performance.
*GPU vram eviction: One key design choice of this series is, SVM layer allocate GPU memory directly from drm buddy allocator, instead of from xe vram manager. There is no BO (buffer object) concept in this implementation. The key benefit of this approach is we can migrate memory at page granularity easily. This also means SVM bypasses TTM's memory eviction logic. But we want the SVM memory and BO driver memory can mutually evicted each other. We have some prove of concept work to rework TTM resource manager for this purpose, see
https://lore.kernel.org/dri-devel/[email protected]/
We will continue work on that series then implement SVM's eviction function based on the concept of shared drm LRU list b/t SVM and TTM/BO driver.
* Try 1 vma with N PT_state for system allocator and userptr. One gigantic vma to hold address space initial default constant state and N PT_state to hold mutable page table state. Also try to register only one mmu interval notifier for the whole address range.
* Multiple GPU device support
Matthew Brost (7):
drm/xe: Refactor vm_bind
drm/xe: Invalidate userptr VMA on page pin fault
drm/xe: Drop unused arguments from vm_bind_ioctl_ops_parse drm/xe: Fix op->tile_mask for fault mode
drm/xe/uapi: Add DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATOR flag drm/xe: Create userptr if page fault occurs on system_allocator VMA drm/xe: Add faulted userptr VMA garbage collector
Oak Zeng (24):
drm/xe/svm: Add SVM document
drm/xe: Introduce helper to populate userptr
drm/xe: Introduce a helper to free sg table
drm/xe: Use hmm_range_fault to populate user pages drm/xe/svm: Remap and provide memmap backing for GPU vram drm/xe/svm: Introduce DRM_XE_SVM kernel config
drm/xe: Introduce helper to get tile from memory region drm/xe: Introduce a helper to get dpa from pfn drm/xe/svm: Get xe memory region from page
drm/xe: Get xe_vma from xe_userptr
drm/xe/svm: Build userptr sg table for device pages drm/xe/svm: Determine a vma is backed by device memory drm/xe: add xe lock document
drm/xe/svm: Introduce svm migration function drm/xe/svm: implement functions to allocate and free device memory drm/xe/svm: Trace buddy block allocation and free drm/xe/svm: Create and destroy xe svm
drm/xe/svm: Add vm to xe_svm process
drm/xe: Make function lookup_vma public
drm/xe/svm: Handle CPU page fault
drm/xe/svm: Introduce helper to migrate vma to vram drm/xe/svm: trace svm migration
drm/xe/svm: Add a helper to determine a vma is fault userptr drm/xe/svm: Migration from sram to vram for system allocator
Documentation/gpu/xe/index.rst | 2 + Documentation/gpu/xe/xe_lock.rst | 8 + Documentation/gpu/xe/xe_svm.rst | 8 + drivers/gpu/drm/xe/Kconfig | 22 + drivers/gpu/drm/xe/Makefile | 6 + drivers/gpu/drm/xe/tests/xe_migrate.c | 86 - drivers/gpu/drm/xe/xe_bo.c | 7 +- drivers/gpu/drm/xe/xe_bo.h | 4 +- drivers/gpu/drm/xe/xe_device.c | 35 + drivers/gpu/drm/xe/xe_device.h | 10 + drivers/gpu/drm/xe/xe_device_types.h | 24 + drivers/gpu/drm/xe/xe_exec.c | 41 +- drivers/gpu/drm/xe/xe_exec_queue.c | 120 +- drivers/gpu/drm/xe/xe_exec_queue_types.h | 20 +- drivers/gpu/drm/xe/xe_gt_pagefault.c | 52 +- drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c | 59 +- drivers/gpu/drm/xe/xe_gt_tlb_invalidation.h | 3 + drivers/gpu/drm/xe/xe_guc_submit.c | 22 +- drivers/gpu/drm/xe/xe_hmm.c | 329 ++++ drivers/gpu/drm/xe/xe_hmm.h | 18 + drivers/gpu/drm/xe/xe_lock_doc.h | 113 ++ drivers/gpu/drm/xe/xe_migrate.c | 602 ++++--- drivers/gpu/drm/xe/xe_migrate.h | 53 +- drivers/gpu/drm/xe/xe_mmio.c | 6 + drivers/gpu/drm/xe/xe_pci.c | 1 + drivers/gpu/drm/xe/xe_pt.c | 1301 +++++++++----- drivers/gpu/drm/xe/xe_pt.h | 15 +- drivers/gpu/drm/xe/xe_pt_exec_queue.c | 180 ++ drivers/gpu/drm/xe/xe_pt_exec_queue.h | 14 + drivers/gpu/drm/xe/xe_pt_types.h | 53 + drivers/gpu/drm/xe/xe_sched_job.c | 68 +- drivers/gpu/drm/xe/xe_sched_job_types.h | 31 +- drivers/gpu/drm/xe/xe_svm.c | 122 ++ drivers/gpu/drm/xe/xe_svm.h | 88 + drivers/gpu/drm/xe/xe_svm_devmem.c | 231 +++ drivers/gpu/drm/xe/xe_svm_doc.h | 121 ++ drivers/gpu/drm/xe/xe_svm_migrate.c | 340 ++++ drivers/gpu/drm/xe/xe_sync.c | 15 + drivers/gpu/drm/xe/xe_sync.h | 1 + drivers/gpu/drm/xe/xe_tile.c | 7 + drivers/gpu/drm/xe/xe_trace.h | 69 +- drivers/gpu/drm/xe/xe_uc_fw.c | 1 + drivers/gpu/drm/xe/xe_vm.c | 1768 ++++++++++--------- drivers/gpu/drm/xe/xe_vm.h | 40 +- drivers/gpu/drm/xe/xe_vm_types.h | 229 ++- include/drm/xe_pciids.h | 16 + include/uapi/drm/xe_drm.h | 15 +- 47 files changed, 4432 insertions(+), 1944 deletions(-) create mode 100644 Documentation/gpu/xe/xe_lock.rst create mode 100644 Documentation/gpu/xe/xe_svm.rst create mode 100644 drivers/gpu/drm/xe/xe_hmm.c create mode 100644 drivers/gpu/drm/xe/xe_hmm.h create mode 100644 drivers/gpu/drm/xe/xe_lock_doc.h create mode 100644 drivers/gpu/drm/xe/xe_pt_exec_queue.c create mode 100644 drivers/gpu/drm/xe/xe_pt_exec_queue.h create mode 100644 drivers/gpu/drm/xe/xe_svm.c create mode 100644 drivers/gpu/drm/xe/xe_svm.h create mode 100644 drivers/gpu/drm/xe/xe_svm_devmem.c create mode 100644 drivers/gpu/drm/xe/xe_svm_doc.h create mode 100644 drivers/gpu/drm/xe/xe_svm_migrate.c
Signature · 1 line
2.26.3
PATCH v2 1/31 drm/xe: Refactor vm_bind
From: Matthew Brost <[email protected]>
drm/xe: Lock all gpuva ops during VM bind IOCTL
Lock all gpuva ops and validate all BOs in a single step durin the VM bind IOCTL. This help with the transition to making all gpuva ops in a VM bind IOCTL a single atomic job.
drm/xe: Add ops_execute function which returns a fence
Add ops_execute function which returns a fence. This will be helpful to initiate all binds (VM bind IOCTL, rebinds in exec IOCTL, rebinds in preempt rebind worker, and rebinds in pagefaults) via a gpuva ops list. Returning a fence is needed in various paths.
drm/xe: Move migrate to prefetch to op_lock function
Migrates need to be done under drm exec to make lockdep happy, move the migrate done for prefetches under the op_lock function.
drm/xe: Add struct xe_vma_ops abstraction
Having a structure which encapsulates a list of VMA operations will help enable 1 job for the entire list.
drm/xe: Update xe_vm_rebind to use dummy VMA operations
All bind interfaces are transitioning to use VMA ops, update xe_vm_rebind to use VMA ops.
drm/xe: Simplify VM bind IOCTL error handling and cleanup
Clean up everything in VM bind IOCTL in 1 path for both errors and non-errors. Also move VM bind IOCTL cleanup from ops (also used by non-IOCTL binds) to the VM bind IOCTL.
drm/xe: Update pagefaults to use dummy VMA operations
All bind interfaces are transitioning to use VMA ops, update pagefaults to use VMA ops.
drm/xe: s/xe_tile_migrate_engine/xe_tile_migrate_exec_queue
xe_engine is now xe_exec_queue, adjust this function's name to reflect.
drm/xe: Add some members to xe_vma_ops
This will help with moving to single jobs for many bind operations.
drm/xe: Add vm_bind_ioctl_ops_install_fences helper
Simplify VM bind code by signaling out-fences / destroying VMAs in a single location. Will help with transition single job for many bind ops.
drm/xe: Move setting last fence to vm_bind_ioctl_ops_install_fences
This moves setting of the last fence to a single location.
drm/xe: Move ufence check to op_lock
Rather than checking for an unsignaled ufence ay unbind time, check for this during the op_lock function. This will help with the transition to job 1 per VM bind IOCTL.
drm/xe: Move ufence add to vm_bind_ioctl_ops_install_fences
Rather than adding a ufence to a VMA in the bind function, add the ufence to all VMAs in the IOCTL that require binds in vm_bind_ioctl_ops_install_fences. This will help with the transition to job 1 per VM bind IOCTL.
drm/xe: Add xe_gt_tlb_invalidation_range and convert PT layer to use this
xe_gt_tlb_invalidation_range accepts a start and end address rather than a VMA. This will enable multiple VMAs to be invalidated in a single invalidation. Update the PT layer to use this new function.
drm/xe: Add xe_vm_pgtable_update_op to xe_vma_ops
Will help with the converstion to 1 job per VM bind IOCTL. Allocation only implemented in this patch.
drm/xe: Use ordered WQ for TLB invalidation fences
TLB invalidation fences need to be ordered within an exec queue and if an unordered WQ is used TLB invalidation fences could be reordered. Use an ordered WQ to fix this.
drm/xe: Delete PT update selftest
IGTs (e.g. xe_vm) can provide the exact same coverage as the PT update selftest. The PT update selftest is dependent on internal functions which can change thus maintaining this test is costly and provide no extra coverage. Delete this test.
drm/xe: Convert multiple bind ops into single job
This aligns with the uAPI of an array of binds or single bind that results in multiple GPUVA ops to be considered a single atomic operations.
drm/xe: Remove old functions defs in xe_pt.h
__xe_pt_bind_vma and __xe_pt_unbind_vma are unused, remove these.
drm/xe: Update PT layer with better error handling
Update PT layer so if a memory allocation for a PTE fails the error can be propagated to the user without requiring to be killed.
drm/xe: Update xe_vm_rebind to return int
Now that rebinds are installed in the kernel dma-resv slot the fence returned from xe_vm_rebind is unused aside from error checking. Update to xe_vm_rebind to return int.
drm/xe: Move vma rebinding to the drm_exec locking loop
Rebinding might allocate page-table bos, causing evictions. To support blocking locking during these evictions, perform the rebinding in the drm_exec locking loop.
drm/xe: Update VM trace events
The trace events have changed moving to a single job per VM bind IOCTL, update the trace events align with old behavior as much as possible.
drm/xe: Update clear / populate arguments
This will help implement CPU binds in run_job() as 'struct xe_migrate_pt_update' is not available at the time of run_job().
drm/xe: Add __xe_migrate_update_pgtables_cpu helper
This will help implement CPU binds as the submision backend can call this helper when a bind jobs dependencies are resolved.
drm/xe: CPU binds for jobs
No reason to use the GPU for binds. In run_job use the CPU to do binds once the bind job dependencies are resolved.
drm/xe: Don't use migrate exec queue for page fault binds
Now that the CPU is always used for binds even in jobs, CPU bind jobs can pass GPU jobs in the same exec queue resulting dma-fences signaling out-of-order. Use a dedicated exec queue for binds issued from page faults to avoid ordering issues.
drm/xe: Add VM bind IOCTL error injection
Add VM bind IOCTL error injection which steals MSB of the bind flags field which if set injects errors at various points in the VM bind IOCTL. Intended to validate error paths. Enabled by CONFIG_DRM_XE_DEBUG.
drm/xe/guc: Assert time'd out jobs are not from a VM exec queue
With CPU binds jobs cannot timeout, assert this is not happening.
drm/xe: Add PT exec queues
Add PT exec queues which are used to implement VM bind / unbind operations. PT exec queues use a different DRM scheduler backend (compared GuC / execlist submission backends) which use the CPU to update page tables once all dependecies for a job are resolved.
drm/xe: Add PVC support
Add PVC pcie IDs and GuC firmware.
--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/tests/xe_migrate.c | 86 -- drivers/gpu/drm/xe/xe_bo.c | 7 +- drivers/gpu/drm/xe/xe_bo.h | 4 +- drivers/gpu/drm/xe/xe_device.c | 35 + drivers/gpu/drm/xe/xe_device.h | 2 + drivers/gpu/drm/xe/xe_device_types.h | 16 + drivers/gpu/drm/xe/xe_exec.c | 41 +- drivers/gpu/drm/xe/xe_exec_queue.c | 120 +- drivers/gpu/drm/xe/xe_exec_queue_types.h | 20 +- drivers/gpu/drm/xe/xe_gt_pagefault.c | 10 +- drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c | 59 +- drivers/gpu/drm/xe/xe_gt_tlb_invalidation.h | 3 + drivers/gpu/drm/xe/xe_guc_submit.c | 22 +- drivers/gpu/drm/xe/xe_migrate.c | 385 ++---- drivers/gpu/drm/xe/xe_migrate.h | 46 +- drivers/gpu/drm/xe/xe_pci.c | 1 + drivers/gpu/drm/xe/xe_pt.c | 1242 ++++++++++++-------
17 lines not shown · raw message
drivers/gpu/drm/xe/Makefile +1 -0
@@ -118,6 +118,7 @@ xe-y += xe_bb.o \ xe_pm.o \ xe_preempt_fence.o \ xe_pt.o \ + xe_pt_exec_queue.o \ xe_pt_walk.o \ xe_query.o \ xe_range_fence.o \
drivers/gpu/drm/xe/tests/xe_migrate.c +0 -86
104 lines not on this page · raw message
drivers/gpu/drm/xe/xe_bo.c +3 -4
@@ -2265,16 +2265,16 @@ void __xe_bo_release_dummy(struct kref *kref)/** * xe_bo_put_commit() - Put bos whose put was deferred by xe_bo_put_deferred(). +* @xe: Xe device * @deferred: The lockless list used for the call to xe_bo_put_deferred(). * * Puts all bos whose put was deferred by xe_bo_put_deferred(). * The @deferred list can be either an onstack local list or a global * shared list used by a workqueue. */ -void xe_bo_put_commit (struct llist_head *deferred) +void xe_bo_put_commit (struct xe_device *xe,struct llist_head *deferred) {struct llist_node *freed; -struct xe_bo *bo, *next;if (!deferred)return ;
@@ -2283,8 +2283,7 @@ void xe_bo_put_commit(struct llist_head *deferred)if (!freed)return ; -llist_for_each_entry_safe (bo, next, freed, freed) -drm_gem_object_free (&bo->ttm.base.refcount); +xe_device_put_deferred (xe, freed); }/**
drivers/gpu/drm/xe/xe_bo.h +2 -2
@@ -10,7 +10,6 @@#include "xe_bo_types.h" #include "xe_macros.h" -#include "xe_vm_types.h" #include "xe_vm.h" /**
@@ -309,10 +308,11 @@ xe_bo_put_deferred(struct xe_bo *bo, struct llist_head *deferred)if (!kref_put (&bo->ttm.base.refcount, __xe_bo_release_dummy))return false; +xe_vm_get (bo->vm);return llist_add (&bo->freed, deferred); } -void xe_bo_put_commit (struct llist_head *deferred); +void xe_bo_put_commit (struct xe_device *xe,struct llist_head *deferred);struct sg_table *xe_bo_sg (struct xe_bo *bo);
drivers/gpu/drm/xe/xe_device.c +35 -0
@@ -226,6 +226,9 @@ static void xe_device_destroy(struct drm_device *dev, void *dummy) {struct xe_device *xe =to_xe_device (dev); +flush_work (&xe->mem.deferred_work); +xe_assert (xe, !llist_del_all (&xe->mem.deferred)); +if (xe->ordered_wq)destroy_workqueue (xe->ordered_wq);
@@ -235,6 +238,35 @@ static void xe_device_destroy(struct drm_device *dev, void *dummy)ttm_device_fini (&xe->ttm); } +void xe_device_put_deferred (struct xe_device *xe,struct llist_node *deferred) +{ +struct xe_bo *bo, *next; + +llist_for_each_entry_safe (bo, next, deferred, freed) { +init_llist_node (&bo->freed); +llist_add (&bo->freed, &xe->mem.deferred); + } +queue_work (system_wq, &xe->mem.deferred_work); +} + +static void deferred_work (struct work_struct *w) +{ + struct xe_device *xe = container_of(w, struct xe_device, + mem.deferred_work); +struct llist_node *freed =llist_del_all (&xe->mem.deferred); +struct xe_bo *bo, *next; + +if (!freed) +return ; + +llist_for_each_entry_safe (bo, next, freed, freed) { +struct xe_vm *vm = bo->vm; + +drm_gem_object_free (&bo->ttm.base.refcount); +xe_vm_put (vm); + } +} +struct xe_device *xe_device_create (struct pci_dev *pdev,const struct pci_device_id *ent) {
@@ -299,6 +331,9 @@ struct xe_device *xe_device_create(struct pci_dev *pdev, goto err; } +init_llist_head (&xe->mem.deferred); +INIT_WORK (&xe->mem.deferred_work, deferred_work); + err =xe_display_create (xe);if (WARN_ON (err)) goto err;
drivers/gpu/drm/xe/xe_device.h +2 -0
@@ -176,4 +176,6 @@ void xe_device_snapshot_print(struct xe_device *xe, struct drm_printer *p);u64 xe_device_canonicalize_addr (struct xe_device *xe,u64 address);u64 xe_device_uncanonicalize_addr (struct xe_device *xe,u64 address); +void xe_device_put_deferred (struct xe_device *xe,struct llist_node *deferred); +#endif
drivers/gpu/drm/xe/xe_device_types.h +16 -0
@@ -22,6 +22,10 @@#include "xe_sriov_types.h" #include "xe_step_types.h" +#if IS_ENABLED (CONFIG_DRM_XE_DEBUG ) +#define TEST_VM_OPS_ERROR +#endif +#if IS_ENABLED (CONFIG_DRM_XE_DISPLAY )#include "soc/intel_pch.h" #include "intel_display_core.h"
@@ -315,6 +319,10 @@ struct xe_device {struct xe_mem_region vram;/** @mem.sys_mgr: system TTM manager */ struct ttm_resource_manager sys_mgr; +/** @mem.deferred: deferred list to destroy PT entries */ +struct llist_head deferred; +/** @mem.deferred_work: worker to destroy PT entries */ +struct work_struct deferred_work; } mem;/** @sriov: device level virtualization data */
@@ -455,6 +463,14 @@ struct xe_device {/** @needs_flr_on_fini: requests function-reset on fini */ bool needs_flr_on_fini; +#ifdef TEST_VM_OPS_ERROR +/** +* @vm_inject_error_position: inject errors at different places in VM +* bind IOCTL based on this value +*/ +u8 vm_inject_error_position; +#endif +/* private: */ #if IS_ENABLED (CONFIG_DRM_XE_DISPLAY )
drivers/gpu/drm/xe/xe_exec.c +6 -35
43 lines not on this page · raw message
drivers/gpu/drm/xe/xe_exec_queue.c +26 -94
120 lines not on this page · raw message
drivers/gpu/drm/xe/xe_exec_queue_types.h +8 -12
Lines of this file are not shown. View the raw message.
drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c +44 -15
59 lines not on this page · raw message
drivers/gpu/drm/xe/xe_gt_tlb_invalidation.h +3 -0
Lines of this file are not shown. View the raw message.
drivers/gpu/drm/xe/xe_migrate.c +111 -274
385 lines not on this page · raw message
drivers/gpu/drm/xe/xe_migrate.h +30 -16
46 lines not on this page · raw message
drivers/gpu/drm/xe/xe_pt.c +821 -421
1,242 lines not on this page · raw message
drivers/gpu/drm/xe/xe_pt_exec_queue.c new file +180 -0
180 lines not on this page · raw message
drivers/gpu/drm/xe/xe_pt_exec_queue.h new file +14 -0
Lines of this file are not shown. View the raw message.
drivers/gpu/drm/xe/xe_pt_types.h +53 -0
53 lines not on this page · raw message
drivers/gpu/drm/xe/xe_sched_job.c +43 -25
68 lines not on this page · raw message
drivers/gpu/drm/xe/xe_sched_job_types.h +29 -2
Lines of this file are not shown. View the raw message.
drivers/gpu/drm/xe/xe_vm.c +523 -601
1,124 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm_types.h +111 -87
198 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
Parts of this message are not shown here. View the raw message.
PATCH v2 2/31 drm/xe/svm: Add SVM document
In reply to Oak ZengAdd shared virtual memory document.
--- Documentation/gpu/xe/index.rst | 1 + Documentation/gpu/xe/xe_svm.rst | 8 +++ drivers/gpu/drm/xe/xe_svm_doc.h | 121 ++++++++++++++++++++++++++++++++ 3 files changed, 130 insertions(+) create mode 100644 Documentation/gpu/xe/xe_svm.rst create mode 100644 drivers/gpu/drm/xe/xe_svm_doc.h
Documentation/gpu/xe/index.rst +1 -0
@@ -23,3 +23,4 @@ DG2, etc is provided to prototype the driver. xe_firmware xe_tile xe_debugging + xe_svm
Documentation/gpu/xe/xe_svm.rst new file +8 -0
@@ -0,0 +1,8 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +============= +Shared virtual memory +============= + +.. kernel-doc:: drivers/gpu/drm/xe/xe_svm_doc.h + :doc: Shared virtual memory
drivers/gpu/drm/xe/xe_svm_doc.h new file +121 -0
121 lines not on this page · raw message
Signature · 1 line
2.26.3
PATCH v2 3/31 drm/xe: Invalidate userptr VMA on page pin fault
In reply to Oak ZengFrom: Matthew Brost <[email protected]>
Rather than return an error to the user or ban the VM when userptr VMA page pin fails with -EFAULT, invalidate VMA mappings. This supports the UMD use case of freeing userptr while still having bindings.
--- drivers/gpu/drm/xe/xe_gt_pagefault.c | 4 ++-- drivers/gpu/drm/xe/xe_trace.h | 2 +- drivers/gpu/drm/xe/xe_vm.c | 20 +++++++++++++------- drivers/gpu/drm/xe/xe_vm_types.h | 7 ++----- 4 files changed, 18 insertions(+), 15 deletions(-)
drivers/gpu/drm/xe/xe_gt_pagefault.c +2 -2
@@ -68,7 +68,7 @@ static bool access_is_atomic(enum access_type access_type)static bool vma_is_valid (struct xe_tile *tile,struct xe_vma *vma) {return BIT (tile->id) & vma->tile_present && - !(BIT (tile->id) & vma->usm.tile_invalidated); + !(BIT (tile->id) & vma->tile_invalidated); }static bool vma_matches (struct xe_vma *vma,u64 page_addr)
@@ -230,7 +230,7 @@ static int handle_pagefault(struct xe_gt *gt, struct pagefault *pf)if (xe_vma_is_userptr (vma)) ret =xe_vma_userptr_check_repin (to_userptr_vma (vma)); - vma->usm.tile_invalidated &= ~BIT (tile->id); + vma->tile_invalidated &= ~BIT (tile->id); unlock_dma_resv:drm_exec_fini (&exec);
drivers/gpu/drm/xe/xe_trace.h +1 -1
@@ -464,7 +464,7 @@ DEFINE_EVENT(xe_vma, xe_vma_userptr_invalidate,TP_ARGS (vma) ); -DEFINE_EVENT (xe_vma, xe_vma_usm_invalidate, +DEFINE_EVENT (xe_vma, xe_vma_invalidate, TP_PROTO(struct xe_vma *vma),TP_ARGS (vma) );
drivers/gpu/drm/xe/xe_vm.c +13 -7
@@ -723,11 +723,18 @@ int xe_vm_userptr_pin(struct xe_vm *vm)list_for_each_entry_safe (uvma, next, &vm->userptr.repin_list, userptr.repin_link) { err =xe_vma_userptr_pin_pages (uvma); -if (err <0 ) -return err; -list_del_init (&uvma->userptr.repin_link); -list_move_tail (&uvma->vma.combined_links.rebind, &vm->rebind_list); +if (err == -EFAULT ) { + err =xe_vm_invalidate_vma (&uvma->vma); +if (err) +return err; + }else { +if (err <0 ) +return err; + +list_move_tail (&uvma->vma.combined_links.rebind, + &vm->rebind_list); + } }return 0 ;
@@ -3136,9 +3143,8 @@ int xe_vm_invalidate_vma(struct xe_vma *vma)u8 id;int ret; -xe_assert (xe,xe_vm_in_fault_mode (xe_vma_vm (vma)));xe_assert (xe, !xe_vma_is_null (vma)); -trace_xe_vma_usm_invalidate (vma); +trace_xe_vma_invalidate (vma);/* Check that we don't race with page-table updates */ if (IS_ENABLED (CONFIG_PROVE_LOCKING )) {
@@ -3176,7 +3182,7 @@ int xe_vm_invalidate_vma(struct xe_vma *vma) } } - vma->usm.tile_invalidated = vma->tile_mask; + vma->tile_invalidated = vma->tile_mask;return 0 ; }
drivers/gpu/drm/xe/xe_vm_types.h +2 -5
@@ -96,11 +96,8 @@ struct xe_vma {struct work_struct destroy_work; }; -/** @usm: unified shared memory state */ -struct { -/** @tile_invalidated: VMA has been invalidated */ -u8 tile_invalidated; - } usm; +/** @tile_invalidated: VMA has been invalidated */ +u8 tile_invalidated;/** @tile_mask: Tile mask of where to create binding for this VMA */ u8 tile_mask;
Signature · 1 line
2.26.3
PATCH v2 4/31 drm/xe: Drop unused arguments from vm_bind_ioctl_ops_parse
In reply to Oak ZengFrom: Matthew Brost <[email protected]>
Drop exec queue and last arguments from vm_bind_ioctl_ops_parse as these are unused.
--- drivers/gpu/drm/xe/xe_vm.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
drivers/gpu/drm/xe/xe_vm.c +4 -5
@@ -2142,10 +2142,9 @@ static int xe_vma_op_commit(struct xe_vm *vm, struct xe_vma_op *op)return err; } -static int vm_bind_ioctl_ops_parse (struct xe_vm *vm,struct xe_exec_queue *q, -struct drm_gpuva_ops *ops, +static int vm_bind_ioctl_ops_parse (struct xe_vm *vm,struct drm_gpuva_ops *ops,struct xe_sync_entry *syncs,u32 num_syncs, -struct xe_vma_ops *vops,bool last) +struct xe_vma_ops *vops) {struct xe_device *xe = vm->xe;struct drm_gpuva_op *__op;
@@ -3037,8 +3036,8 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) goto unwind_ops; } - err =vm_bind_ioctl_ops_parse (vm, q, ops[i], syncs, num_syncs, - &vops, i == args->num_binds -1 ); + err =vm_bind_ioctl_ops_parse (vm, ops[i], syncs, num_syncs, + &vops);if (err) goto unwind_ops;
Signature · 1 line
2.26.3
PATCH v2 5/31 drm/xe: Fix op->tile_mask for fault mode
In reply to Oak ZengFrom: Matthew Brost <[email protected]>
op->tile_mask might be a subset of all tiles if in fault mode. Fix unmaps by setting op->tile_mask the unmapped VMA's tile_present field.
FIXME: This should be squashed into an eariler patch
--- drivers/gpu/drm/xe/xe_vm.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/gpu/drm/xe/xe_vm.c +8 -0
@@ -2190,6 +2190,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct drm_gpuva_ops *ops,struct xe_vma *old =gpuva_to_vma (op->base.remap.unmap->va); + op->tile_mask = old->tile_present; op->remap.start =xe_vma_start (old); op->remap.range =xe_vma_size (old);
@@ -2273,6 +2274,13 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct drm_gpuva_ops *ops,break ; }case DRM_GPUVA_OP_UNMAP : + { +struct xe_vma *vma =gpuva_to_vma (op->base.unmap.va); + + op->tile_mask = vma->tile_present; +xe_vma_ops_incr_pt_update_ops (vops, op->tile_mask); +break ; + }case DRM_GPUVA_OP_PREFETCH :/* FIXME: Need to skip some prefetch ops */ xe_vma_ops_incr_pt_update_ops (vops, op->tile_mask);
Signature · 1 line
2.26.3
PATCH v2 6/31 drm/xe/uapi: Add DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATOR flag
In reply to Oak ZengFrom: Matthew Brost <[email protected]>
Add DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATOR flag which is used to create unpopulated (no memory backing or GPU page tables) VMAs. These VMAs are referred to as system allocator VMAs. The idea is on page fault the memory back and GPU page tables will be populated.
FIXME: Only supporting 1 to 1 mapping between user address space and GPU address space
v1: enforce DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATOR fo fault mode VMs
--- drivers/gpu/drm/xe/xe_pt.c | 73 +++++++++++++---- drivers/gpu/drm/xe/xe_vm.c | 132 +++++++++++++++++++------------ drivers/gpu/drm/xe/xe_vm.h | 8 +- drivers/gpu/drm/xe/xe_vm_types.h | 3 + include/uapi/drm/xe_drm.h | 15 +++- 5 files changed, 161 insertions(+), 70 deletions(-)
drivers/gpu/drm/xe/xe_pt.c +59 -14
@@ -1030,6 +1030,11 @@ static int op_add_deps(struct xe_vm *vm, struct xe_vma_op *op, {int err =0 ; +/* +* No need to check for is_system_allocator here as vma_add_deps is a +* NOP if VMA is_system_allocator +*/ +switch (op->base.op) {case DRM_GPUVA_OP_MAP :if (!op->map.immediate &&xe_vm_in_fault_mode (vm))
@@ -1602,6 +1607,7 @@ static int bind_op_prepare(struct xe_vm *vm, struct xe_tile *tile,struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[current_op];int err; +xe_tile_assert (tile, !xe_vma_is_system_allocator (vma));xe_bo_assert_held (xe_vma_bo (vma)); vm_dbg(&xe_vma_vm(vma)->xe->drm,
@@ -1659,6 +1665,7 @@ static int unbind_op_prepare(struct xe_tile *tile,u32 current_op = pt_update_ops->current_op;struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[current_op]; +xe_tile_assert (tile, !xe_vma_is_system_allocator (vma));xe_bo_assert_held (xe_vma_bo (vma)); vm_dbg(&xe_vma_vm(vma)->xe->drm,
@@ -1694,15 +1701,21 @@ static int op_prepare(struct xe_vm *vm,switch (op->base.op) {case DRM_GPUVA_OP_MAP : -if (!op->map.immediate &&xe_vm_in_fault_mode (vm)) +if ((!op->map.immediate &&xe_vm_in_fault_mode (vm)) || + op->map.is_system_allocator)break ; err =bind_op_prepare (vm, tile, pt_update_ops, op->map.vma); pt_update_ops->wait_vm_kernel = true;break ;case DRM_GPUVA_OP_REMAP : - err =unbind_op_prepare (tile, pt_update_ops, -gpuva_to_vma (op->base.remap.unmap->va)); + { +struct xe_vma *old =gpuva_to_vma (op->base.remap.unmap->va); + +if (xe_vma_is_system_allocator (old)) +break ; + + err =unbind_op_prepare (tile, pt_update_ops, old);if (!err && op->remap.prev) { err = bind_op_prepare(vm, tile, pt_update_ops,
@@ -1715,15 +1728,28 @@ static int op_prepare(struct xe_vm *vm, pt_update_ops->wait_vm_bookkeep = true; }break ; + }case DRM_GPUVA_OP_UNMAP : - err =unbind_op_prepare (tile, pt_update_ops, -gpuva_to_vma (op->base.unmap.va)); + { +struct xe_vma *vma =gpuva_to_vma (op->base.unmap.va); + +if (xe_vma_is_system_allocator (vma)) +break ; + + err =unbind_op_prepare (tile, pt_update_ops, vma);break ; + }case DRM_GPUVA_OP_PREFETCH : - err =bind_op_prepare (vm, tile, pt_update_ops, -gpuva_to_vma (op->base.prefetch.va)); + { +struct xe_vma *vma =gpuva_to_vma (op->base.prefetch.va); + +if (xe_vma_is_system_allocator (vma)) +break ; + + err =bind_op_prepare (vm, tile, pt_update_ops, vma); pt_update_ops->wait_vm_kernel = true;break ; + }default :drm_warn (&vm->xe->drm,"NOT POSSIBLE" ); }
@@ -1785,6 +1811,8 @@ static void bind_op_commit(struct xe_vm *vm, struct xe_tile *tile,struct xe_vm_pgtable_update_ops *pt_update_ops, struct xe_vma *vma, struct dma_fence *fence) { + xe_tile_assert(tile, !xe_vma_is_system_allocator(vma)); + if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm) dma_resv_add_fence(xe_vma_bo(vma)->ttm.base.resv, fence, pt_update_ops->wait_vm_bookkeep ?
@@ -1810,6 +1838,8 @@ static void unbind_op_commit(struct xe_vm *vm, struct xe_tile *tile,struct xe_vm_pgtable_update_ops *pt_update_ops, struct xe_vma *vma, struct dma_fence *fence) { + xe_tile_assert(tile, !xe_vma_is_system_allocator(vma)); + if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm) dma_resv_add_fence(xe_vma_bo(vma)->ttm.base.resv, fence, pt_update_ops->wait_vm_bookkeep ?
@@ -1837,14 +1867,20 @@ static void op_commit(struct xe_vm *vm,switch (op->base.op) {case DRM_GPUVA_OP_MAP : -if (!op->map.immediate &&xe_vm_in_fault_mode (vm)) +if ((!op->map.immediate &&xe_vm_in_fault_mode (vm)) || + op->map.is_system_allocator)break ;bind_op_commit (vm, tile, pt_update_ops, op->map.vma, fence);break ;case DRM_GPUVA_OP_REMAP : -unbind_op_commit (vm, tile, pt_update_ops, -gpuva_to_vma (op->base.remap.unmap->va), fence); + { +struct xe_vma *old =gpuva_to_vma (op->base.remap.unmap->va); + +if (xe_vma_is_system_allocator (old)) +break ; + +unbind_op_commit (vm, tile, pt_update_ops, old, fence);if (op->remap.prev) bind_op_commit(vm, tile, pt_update_ops, op->remap.prev,
@@ -1853,14 +1889,23 @@ static void op_commit(struct xe_vm *vm,bind_op_commit (vm, tile, pt_update_ops, op->remap.next, fence);break ; + }case DRM_GPUVA_OP_UNMAP : -unbind_op_commit (vm, tile, pt_update_ops, -gpuva_to_vma (op->base.unmap.va), fence); + { +struct xe_vma *vma =gpuva_to_vma (op->base.unmap.va); + +if (!xe_vma_is_system_allocator (vma)) +unbind_op_commit (vm, tile, pt_update_ops, vma, fence);break ; + }case DRM_GPUVA_OP_PREFETCH : -bind_op_commit (vm, tile, pt_update_ops, -gpuva_to_vma (op->base.prefetch.va), fence); + { +struct xe_vma *vma =gpuva_to_vma (op->base.prefetch.va); + +if (!xe_vma_is_system_allocator (vma)) +bind_op_commit (vm, tile, pt_update_ops, vma, fence);break ; + }default :drm_warn (&vm->xe->drm,"NOT POSSIBLE" ); }
drivers/gpu/drm/xe/xe_vm.c +80 -52
314 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm.h +7 -1
@@ -150,6 +150,11 @@ static inline bool xe_vma_is_null(struct xe_vma *vma)return vma->gpuva.flags &DRM_GPUVA_SPARSE ; } +static inline bool xe_vma_is_system_allocator (struct xe_vma *vma) +{ +return vma->gpuva.flags &XE_VMA_SYSTEM_ALLOCATOR ; +} +static inline bool xe_vma_has_no_bo (struct xe_vma *vma) {return !xe_vma_bo (vma);
@@ -157,7 +162,8 @@ static inline bool xe_vma_has_no_bo(struct xe_vma *vma)static inline bool xe_vma_is_userptr (struct xe_vma *vma) { -return xe_vma_has_no_bo (vma) && !xe_vma_is_null (vma); +return xe_vma_has_no_bo (vma) && !xe_vma_is_null (vma) && + !xe_vma_is_system_allocator (vma); }/**
drivers/gpu/drm/xe/xe_vm_types.h +3 -0
@@ -45,6 +45,7 @@ struct xe_vm_pgtable_update_op;#define XE_VMA_PTE_64K (DRM_GPUVA_USERBITS << 8)#define XE_VMA_PTE_COMPACT (DRM_GPUVA_USERBITS << 9)#define XE_VMA_DUMPABLE (DRM_GPUVA_USERBITS << 10) +#define XE_VMA_SYSTEM_ALLOCATOR (DRM_GPUVA_USERBITS << 11)/** struct xe_userptr - User pointer */ struct xe_userptr {
@@ -141,6 +142,8 @@ struct xe_vma_op_map {bool immediate;/** @is_null: is NULL binding */ bool is_null; +/** @is_system_allocator: is system allocator binding */ +bool is_system_allocator;/** @dumpable: whether BO is dumped on GPU hang */ bool dumpable;/** @pat_index: The pat index to use for this operation. */
include/uapi/drm/xe_drm.h +12 -3
@@ -869,6 +869,12 @@ struct drm_xe_vm_destroy {* will only be valid for DRM_XE_VM_BIND_OP_MAP operations, the BO * handle MBZ, and the BO offset MBZ. This flag is intended to * implement VK sparse bindings. +* - %DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATOR - When the system allocator flag is +* set, no mappings are created rather the range is reserved for system +* allocations which will be populated on GPU page faults. Only valid on VMs +* with DRM_XE_VM_CREATE_FLAG_FAULT_MODE set. The system allocator flag are +* only valid for DRM_XE_VM_BIND_OP_MAP operations, the BO handle MBZ, and +* the BO offset MBZ. */ struct drm_xe_vm_bind_op {/** @extensions: Pointer to the first extension struct, if any */
@@ -921,7 +927,9 @@ struct drm_xe_vm_bind_op {* on the @pat_index. For such mappings there is no actual memory being * mapped (the address in the PTE is invalid), so the various PAT memory * attributes likely do not apply. Simply leaving as zero is one -* option (still a valid pat_index). +* option (still a valid pat_index). Same applies to +* DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATOR bindings as for such mapping +* there is no actual memory being mapped. */ __u16 pat_index;
@@ -955,8 +963,9 @@ struct drm_xe_vm_bind_op {/** @op: Bind operation to perform */ __u32 op; -#define DRM_XE_VM_BIND_FLAG_NULL (1 << 2) -#define DRM_XE_VM_BIND_FLAG_DUMPABLE (1 << 3) +#define DRM_XE_VM_BIND_FLAG_NULL (1 << 2) +#define DRM_XE_VM_BIND_FLAG_DUMPABLE (1 << 3) +#define DRM_XE_VM_BIND_FLAG_SYSTEM_ALLOCATOR (1 << 4)/** @flags: Bind flags */ __u32 flags;
Signature · 1 line
2.26.3
PATCH v2 7/31 drm/xe: Create userptr if page fault occurs on system_allocator VMA
In reply to Oak ZengFrom: Matthew Brost <[email protected]>
If a page fault occurs on system_allocator VMA, create a userptr VMA to replaced fault region and map to GPU.
v1: Pass userptr to the req_offset of sm_map_ops_create function. This
fix malloc'd memory failure (Oak)
--- drivers/gpu/drm/xe/xe_gt_pagefault.c | 13 +++ drivers/gpu/drm/xe/xe_vm.c | 115 +++++++++++++++++++++++++-- drivers/gpu/drm/xe/xe_vm.h | 2 + drivers/gpu/drm/xe/xe_vm_types.h | 3 + 4 files changed, 128 insertions(+), 5 deletions(-)
drivers/gpu/drm/xe/xe_gt_pagefault.c +13 -0
@@ -166,6 +166,19 @@ static int handle_pagefault(struct xe_gt *gt, struct pagefault *pf) goto unlock_vm; } +/* +* Create userptr VMA if fault occurs in a range reserved for system +* allocator. +*/ +if (xe_vma_is_system_allocator (vma)) { + vma =xe_vm_fault_userptr (vm, pf->page_addr); +if (IS_ERR (vma)) { +xe_vm_kill (vm, true); + ret =PTR_ERR (vma); + goto unlock_vm; + } + } +if (!xe_vma_is_userptr (vma) || !xe_vma_userptr_check_repin (to_userptr_vma (vma))) {downgrade_write (&vm->lock);
drivers/gpu/drm/xe/xe_vm.c +110 -5
136 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm.h +2 -0
@@ -244,6 +244,8 @@ int xe_vma_userptr_pin_pages(struct xe_userptr_vma *uvma);int xe_vma_userptr_check_repin (struct xe_userptr_vma *uvma); +struct xe_vma *xe_vm_fault_userptr (struct xe_vm *vm,u64 fault_addr); +bool xe_vm_validate_should_retry (struct drm_exec *exec,int err,ktime_t *end);int xe_analyze_vm (struct drm_printer *p,struct xe_vm *vm,int gt_id);
drivers/gpu/drm/xe/xe_vm_types.h +3 -0
@@ -233,6 +233,9 @@ struct xe_vm {struct xe_device *xe; +/** @mm: user MM of VM */ +struct mm_struct *mm; +/* exec queue used for (un)binding vma's */ struct xe_exec_queue *q;
Signature · 1 line
2.26.3
PATCH v2 8/31 drm/xe: Add faulted userptr VMA garbage collector
In reply to Oak ZengFrom: Matthew Brost <[email protected]>
When a faulted userptr VMA (allocated by page handler) is invalidated add to list which a garbage collector will unmap from GPU, destroy faulted userptr VMA, and replace with system_allocator VMA.
v1: Run gargabe collector only on MMU_NOTIFY_UNMAP event. For other
events, we just invalidate GPU page table but keep the vma because the userptr is still exist. On next GPU access, we will revalidate and rebind this userptr to GPU(Oak)
--- drivers/gpu/drm/xe/xe_gt_pagefault.c | 6 ++ drivers/gpu/drm/xe/xe_vm.c | 151 +++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_vm.h | 1 + drivers/gpu/drm/xe/xe_vm_types.h | 12 +++ 4 files changed, 170 insertions(+)
drivers/gpu/drm/xe/xe_gt_pagefault.c +6 -0
@@ -154,12 +154,18 @@ static int handle_pagefault(struct xe_gt *gt, struct pagefault *pf)return -EINVAL ; retry_userptr: +xe_vm_userptr_garbage_collector (vm); +/* * TODO: Avoid exclusive lock if VM doesn't have userptrs, or * start out read-locked? */ down_write (&vm->lock); write_locked = true; +if (xe_vm_is_closed_or_banned (vm)) { + ret = -ENOENT ; + goto unlock_vm; + } vma =lookup_vma (vm, pf->page_addr);if (!vma) { ret = -EINVAL ;
drivers/gpu/drm/xe/xe_vm.c +151 -0
186 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm.h +1 -0
@@ -276,6 +276,7 @@ void xe_vma_ops_free(struct xe_vma_ops *vops);struct dma_fence *xe_vm_ops_execute (struct xe_vm *vm,struct xe_vma_ops *vops);void xe_vm_kill (struct xe_vm *vm,bool unlocked); +void xe_vm_userptr_garbage_collector (struct xe_vm *vm);#if IS_ENABLED (CONFIG_DRM_XE_DEBUG_VM )#define vm_dbg drm_dbg
drivers/gpu/drm/xe/xe_vm_types.h +12 -0
@@ -46,6 +46,7 @@ struct xe_vm_pgtable_update_op;#define XE_VMA_PTE_COMPACT (DRM_GPUVA_USERBITS << 9)#define XE_VMA_DUMPABLE (DRM_GPUVA_USERBITS << 10)#define XE_VMA_SYSTEM_ALLOCATOR (DRM_GPUVA_USERBITS << 11) +#define XE_VMA_FAULT_USERPTR (DRM_GPUVA_USERBITS << 12)/** struct xe_userptr - User pointer */ struct xe_userptr {
@@ -326,6 +327,17 @@ struct xe_vm {* write mode. */ struct list_head invalidated; +/** +* @userptr.fault_invalidated: List of invalidated userptrs, +* craeted by page fault, which will be destroy by the garbage +* collector. Protected from access with the @invalidated_lock. +*/ +struct list_head fault_invalidated; +/** +* @userptr.garbage_collector: worker to implement destroying of +* userptrs on @userptr.fault_invalidated list. +*/ +struct work_struct garbage_collector; } userptr;/** @preempt: preempt state */
Signature · 1 line
2.26.3
PATCH v2 9/31 drm/xe: Introduce helper to populate userptr
In reply to Oak ZengIntroduce a helper function xe_userptr_populate_range to populate a a userptr range. This functions calls hmm_range_fault to read CPU page tables and populate all pfns/pages of this virtual address range.
If the populated page is system memory page, dma-mapping is performed to get a dma-address which can be used later for GPU to access pages.
If the populated page is device private page, we calculate the dpa ( device physical address) of the page. This will be handled in future patches.
The dma-address or dpa is then saved in userptr's sg table. This is prepare work to replace the get_user_pages_fast code in userptr code path.
v1: Address review comments:
separate a npage_in_range function (Matt)
reparameterize function xe_userptr_populate_range function (Matt) move mmu_interval_read_begin() call into while loop (Thomas) s/mark_range_accessed/xe_mark_range_accessed (Thomas) use set_page_dirty_lock (vs set_page_dirty) (Thomas) move a few checking in xe_vma_userptr_pin_pages to hmm.c (Matt)
v2: Remove device private page support. Only support system
pages for now. use dma-map-sg rather than dma-map-page (Matt/Thomas)
--- drivers/gpu/drm/xe/Kconfig | 1 + drivers/gpu/drm/xe/Makefile | 2 + drivers/gpu/drm/xe/xe_hmm.c | 224 ++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_hmm.h | 17 +++ 4 files changed, 244 insertions(+) create mode 100644 drivers/gpu/drm/xe/xe_hmm.c create mode 100644 drivers/gpu/drm/xe/xe_hmm.h
drivers/gpu/drm/xe/Kconfig +1 -0
@@ -41,6 +41,7 @@ config DRM_XE select MMU_NOTIFIER select WANT_DEV_COREDUMP select AUXILIARY_BUS + select HMM_MIRROR help Experimental driver for Intel Xe series GPUs
drivers/gpu/drm/xe/Makefile +2 -0
@@ -146,6 +146,8 @@ xe-y += xe_bb.o \ xe_wa.o \ xe_wopcm.o +xe-$(CONFIG_HMM_MIRROR) += xe_hmm.o + # graphics hardware monitoring (HWMON) support xe-$(CONFIG_HWMON) += xe_hwmon.o
drivers/gpu/drm/xe/xe_hmm.c new file +224 -0
224 lines not on this page · raw message
drivers/gpu/drm/xe/xe_hmm.h new file +17 -0
@@ -0,0 +1,17 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2024 Intel Corporation +*/ + +#include <linux/types.h> + +struct xe_userptr_vma ; + +#if IS_ENABLED (CONFIG_HMM_MIRROR ) +int xe_userptr_populate_range (struct xe_userptr_vma *uvma); +#else +static inline int xe_userptr_populate_range (struct xe_userptr_vma *uvma) +{ +return -ENODEV ; +} +#endif
Signature · 1 line
2.26.3
PATCH v2 10/31 drm/xe: Introduce a helper to free sg table
In reply to Oak ZengIntroduce xe_userptr_free_sg helper to dma-unmap all addresses in userptr's sg table and free sg table.
Suggested by: Matthew Brost <[email protected]>
--- drivers/gpu/drm/xe/xe_hmm.c | 30 ++++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_hmm.h | 1 + 2 files changed, 31 insertions(+)
drivers/gpu/drm/xe/xe_hmm.c +30 -0
@@ -3,6 +3,7 @@* Copyright © 2024 Intel Corporation */ +#include <linux/scatterlist.h> #include <linux/mmu_notifier.h> #include <linux/dma-mapping.h> #include <linux/memremap.h>
@@ -107,6 +108,32 @@ static int xe_build_sg(struct xe_device *xe, struct hmm_range *range,return ret; } +/** +* xe_userptr_free_sg() - Free the scatter gather table of userptr +* +* @uvma: the userptr vma which hold the scatter gather table +* +* With function xe_userptr_populate_range, we allocate storage of +* the userptr sg table. This is a helper function to free this +* sg table, and dma unmap the address in the table. +*/ +void xe_userptr_free_sg (struct xe_userptr_vma *uvma) +{ +struct xe_userptr *userptr = &uvma->userptr; +struct xe_vma *vma = &uvma->vma; +bool write = !xe_vma_read_only (vma); +struct xe_vm *vm =xe_vma_vm (vma); +struct xe_device *xe = vm->xe; +struct device *dev = xe->drm.dev; + +xe_assert (xe, userptr->sg); +dma_unmap_sgtable (dev, userptr->sg, + write ?DMA_BIDIRECTIONAL :DMA_TO_DEVICE ,0 ); + +sg_free_table (userptr->sg); + userptr->sg =NULL ; +} +/** * xe_userptr_populate_range() - Populate physical pages of a virtual * address range
@@ -156,6 +183,9 @@ int xe_userptr_populate_range(struct xe_userptr_vma *uvma)if (notifier_seq == userptr->notifier_seq)return 0 ; +if (userptr->sg) +xe_userptr_free_sg (uvma); + npages =xe_npages_in_range (start, end); pfns =kvmalloc_array (npages,sizeof (*pfns),GFP_KERNEL );if (unlikely (!pfns))
drivers/gpu/drm/xe/xe_hmm.h +1 -0
@@ -15,3 +15,4 @@ static inline int xe_userptr_populate_range(struct xe_userptr_vma *uvma) return -ENODEV; }#endif +void xe_userptr_free_sg (struct xe_userptr_vma *uvma);
Signature · 1 line
2.26.3
PATCH v2 11/31 drm/xe: Use hmm_range_fault to populate user pages
In reply to Oak ZengThis is an effort to unify hmmptr (aka system allocator) and userptr code. hmm_range_fault is used to populate a virtual address range for both hmmptr and userptr, instead of hmmptr using hmm_range_fault and userptr using get_user_pages_fast.
This also aligns with AMD gpu driver's behavior. In long term, we plan to put some common helpers in this area to drm layer so it can be re-used by different vendors.
--- drivers/gpu/drm/xe/xe_vm.c | 122 ++++--------------------------------- 1 file changed, 12 insertions(+), 110 deletions(-)
drivers/gpu/drm/xe/xe_vm.c +12 -110
@@ -39,6 +39,7 @@#include "xe_sync.h" #include "xe_trace.h" #include "xe_wa.h" +#include "xe_hmm.h" static struct drm_gem_object *xe_vm_obj (struct xe_vm *vm) {
@@ -66,113 +67,21 @@ int xe_vma_userptr_check_repin(struct xe_userptr_vma *uvma)int xe_vma_userptr_pin_pages (struct xe_userptr_vma *uvma) { -struct xe_userptr *userptr = &uvma->userptr;struct xe_vma *vma = &uvma->vma;struct xe_vm *vm =xe_vma_vm (vma);struct xe_device *xe = vm->xe; -const unsigned long num_pages =xe_vma_size (vma) >>PAGE_SHIFT ; -struct page **pages; -bool in_kthread = !current->mm; -unsigned long notifier_seq; -int pinned, ret, i; -bool read_only =xe_vma_read_only (vma); +struct xe_userptr *userptr; +int ret;lockdep_assert_held (&vm->lock);xe_assert (xe,xe_vma_is_userptr (vma)); -retry: -if (vma->gpuva.flags &XE_VMA_DESTROYED ) -return 0 ; - - notifier_seq =mmu_interval_read_begin (&userptr->notifier); -if (notifier_seq == userptr->notifier_seq) -return 0 ; - - pages =kvmalloc_array (num_pages,sizeof (*pages),GFP_KERNEL ); -if (!pages) -return -ENOMEM ; - -if (userptr->sg) { -dma_unmap_sgtable (xe->drm.dev, - userptr->sg, - read_only ?DMA_TO_DEVICE : -DMA_BIDIRECTIONAL ,0 ); -sg_free_table (userptr->sg); - userptr->sg =NULL ; - } - - pinned = ret =0 ; -if (in_kthread) { -if (!mmget_not_zero (userptr->notifier.mm)) { - ret = -EFAULT ; - goto mm_closed; - } -kthread_use_mm (userptr->notifier.mm); - } - -while (pinned < num_pages) { - ret =get_user_pages_fast (xe_vma_userptr (vma) + - pinned *PAGE_SIZE , - num_pages - pinned, - read_only ?0 :FOLL_WRITE , - &pages[pinned]); -if (ret <0 ) -break ; - - pinned += ret; - ret =0 ; - } -if (in_kthread) { -kthread_unuse_mm (userptr->notifier.mm); -mmput (userptr->notifier.mm); - } -mm_closed: -if (ret) - goto out; - - ret =sg_alloc_table_from_pages_segment (&userptr->sgt, pages, - pinned,0 , - (u64 )pinned <<PAGE_SHIFT , -xe_sg_segment_size (xe->drm.dev), -GFP_KERNEL ); -if (ret) { - userptr->sg =NULL ; - goto out; - } - userptr->sg = &userptr->sgt; - - ret =dma_map_sgtable (xe->drm.dev, userptr->sg, - read_only ?DMA_TO_DEVICE : -DMA_BIDIRECTIONAL , -DMA_ATTR_SKIP_CPU_SYNC | -DMA_ATTR_NO_KERNEL_MAPPING ); -if (ret) { -sg_free_table (userptr->sg); - userptr->sg =NULL ; - goto out; - } - -for (i =0 ; i < pinned; ++i) { -if (!read_only) { -lock_page (pages[i]); -set_page_dirty (pages[i]); -unlock_page (pages[i]); - } + userptr = &uvma->userptr; +mmap_read_lock (userptr->notifier.mm); + ret =xe_userptr_populate_range (uvma); +mmap_read_unlock (userptr->notifier.mm); -mark_page_accessed (pages[i]); - } - -out: -release_pages (pages, pinned); -kvfree (pages); - -if (!(ret <0 )) { - userptr->notifier_seq = notifier_seq; -if (xe_vma_userptr_check_repin (uvma) == -EAGAIN ) - goto retry; - } - -return ret <0 ? ret :0 ; +return ret; }static bool preempt_fences_waiting (struct xe_vm *vm)
@@ -1016,8 +925,6 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm,static void xe_vma_destroy_late (struct xe_vma *vma) {struct xe_vm *vm =xe_vma_vm (vma); -struct xe_device *xe = vm->xe; -bool read_only =xe_vma_read_only (vma);if (vma->ufence) {xe_sync_ufence_put (vma->ufence);
@@ -1025,16 +932,11 @@ static void xe_vma_destroy_late(struct xe_vma *vma) }if (xe_vma_is_userptr (vma)) { -struct xe_userptr *userptr = &to_userptr_vma (vma)->userptr; +struct xe_userptr_vma *uvma =to_userptr_vma (vma); +struct xe_userptr *userptr = &uvma->userptr; -if (userptr->sg) { -dma_unmap_sgtable (xe->drm.dev, - userptr->sg, - read_only ?DMA_TO_DEVICE : -DMA_BIDIRECTIONAL ,0 ); -sg_free_table (userptr->sg); - userptr->sg =NULL ; - } +if (userptr->sg) +xe_userptr_free_sg (uvma);/* * Since userptr pages are not pinned, we can't remove
Signature · 1 line
2.26.3
PATCH v2 12/31 drm/xe/svm: Remap and provide memmap backing for GPU vram
In reply to Oak ZengMemory remap GPU vram using devm_memremap_pages, so each GPU vram page is backed by a struct page.
Those struct pages are created to allow hmm migrate buffer b/t GPU vram and CPU system memory using existing Linux migration mechanism (i.e., migrating b/t CPU system memory and hard disk).
This is prepare work to enable svm (shared virtual memory) through Linux kernel hmm framework. The memory remap's page map type is set to MEMORY_DEVICE_PRIVATE for now. This means even though each GPU vram page get a struct page and can be mapped in CPU page table, but such pages are treated as GPU's private resource, so CPU can't access them. If CPU access such page, a page fault is triggered and page will be migrate to system memory.
For GPU device which supports coherent memory protocol b/t CPU and GPU (such as CXL and CAPI protocol), we can remap device memory as MEMORY_DEVICE_COHERENT. This is TBD.
v1:
Changes per code review feedback from Matt:
change .o order in Makefile
fix indentation
change code order in mmio_fini
remove unnecessary header file
uniform xe_svm_devm_add/_remove parameter use tile (vs dev) as pagemap.owner during memremap only remap vram for platform that support usm
Changes per review feedback from Brian:
s/xe_svm_devm_add/xe_devm_add s/xe_svm_devm_remove/xe_devm_remove
move calling of xe_devm_add to xe_tile.c
--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_device_types.h | 8 +++ drivers/gpu/drm/xe/xe_mmio.c | 6 ++ drivers/gpu/drm/xe/xe_svm.h | 15 +++++ drivers/gpu/drm/xe/xe_svm_devmem.c | 89 ++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_tile.c | 4 ++ 6 files changed, 123 insertions(+) create mode 100644 drivers/gpu/drm/xe/xe_svm.h create mode 100644 drivers/gpu/drm/xe/xe_svm_devmem.c
drivers/gpu/drm/xe/Makefile +1 -0
@@ -129,6 +129,7 @@ xe-y += xe_bb.o \ xe_sa.o \ xe_sched_job.o \ xe_step.o \ + xe_svm_devmem.o \ xe_sync.o \ xe_tile.o \ xe_tile_sysfs.o \
drivers/gpu/drm/xe/xe_device_types.h +8 -0
@@ -103,6 +103,14 @@ struct xe_mem_region {resource_size_t actual_physical_size;/** @mapping: pointer to VRAM mappable space */ void __iomem *mapping; +/** @pagemap: Used to remap device memory as ZONE_DEVICE */ +struct dev_pagemap pagemap; +/** +* @hpa_base: base host physical address +* +* This is generated when remap device memory as ZONE_DEVICE +*/ +resource_size_t hpa_base; };/**
drivers/gpu/drm/xe/xe_mmio.c +6 -0
@@ -22,6 +22,7 @@#include "xe_module.h" #include "xe_sriov.h" #include "xe_tile.h" +#include "xe_svm.h" #define XEHP_MTCFG_ADDR XE_REG(0x101800)#define TILE_COUNT REG_GENMASK(15, 8)
@@ -354,6 +355,11 @@ void xe_mmio_probe_tiles(struct xe_device *xe)static void mmio_fini (struct drm_device *drm,void *arg) {struct xe_device *xe = arg; +struct xe_tile *tile; +u8 id; + +for_each_tile (tile, xe, id) +xe_devm_remove (tile, &tile->mem.vram);pci_iounmap (to_pci_dev (xe->drm.dev), xe->mmio.regs);if (xe->mem.vram.mapping)
drivers/gpu/drm/xe/xe_svm.h new file +15 -0
@@ -0,0 +1,15 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2023 Intel Corporation +*/ + +#ifndef __XE_SVM_H +#define __XE_SVM_H + +struct xe_tile ; +struct xe_mem_region ; + +int xe_devm_add (struct xe_tile *tile,struct xe_mem_region *mr); +void xe_devm_remove (struct xe_tile *tile,struct xe_mem_region *mr); + +#endif
drivers/gpu/drm/xe/xe_svm_devmem.c new file +89 -0
89 lines not on this page · raw message
drivers/gpu/drm/xe/xe_tile.c +4 -0
@@ -14,6 +14,7 @@#include "xe_tile_sysfs.h" #include "xe_ttm_vram_mgr.h" #include "xe_wa.h" +#include "xe_svm.h" /** * DOC: Multi-tile Design
@@ -158,6 +159,7 @@ static int tile_ttm_mgr_init(struct xe_tile *tile)*/ int xe_tile_init_noalloc (struct xe_tile *tile) { +struct xe_device *xe =tile_to_xe (tile);int err;xe_device_mem_access_get (tile_to_xe (tile));
@@ -175,6 +177,8 @@ int xe_tile_init_noalloc(struct xe_tile *tile)xe_tile_sysfs_init (tile); +if (xe->info.has_usm) +xe_devm_add (tile, &tile->mem.vram); err_mem_access:xe_device_mem_access_put (tile_to_xe (tile));return err;
Signature · 1 line
2.26.3
Matthew Brost
maintainerRe: [v2 12/31] drm/xe/svm: Remap and provide memmap backing for GPU vram
On Tue, Apr 09, 2024 at 04:17:23PM -0400, Oak Zeng wrote:
Oak - Doing a very high level review due to early stage of the code.
Quoted 55 lines
Memory remap GPU vram using
devm_memremap_pages, so each GPU vram page is backed by a struct page.Those struct pages are created to allow hmm migrate buffer b/t GPU vram and CPU system memory using existing Linux migration mechanism (i.e., migrating b/t CPU system memory and hard disk).
This is prepare work to enable svm (shared virtual memory) through Linux kernel hmm framework. The memory remap's page map type is set to
MEMORY_DEVICE_PRIVATEfor now. This means even though each GPU vram page get a struct page and can be mapped in CPU page table, but such pages are treated as GPU's private resource, so CPU can't access them. If CPU access such page, a page fault is triggered and page will be migrate to system memory.For GPU device which supports coherent memory protocol b/t CPU and GPU (such as CXL and CAPI protocol), we can remap device memory as
MEMORY_DEVICE_COHERENT. This is TBD.v1:
Changes per code review feedback from Matt:
change .o order inMakefile
fix indentation
change code order inmmio_fini
remove unnecessary header file
uniform xe_svm_devm_add/_remove parameter use tile (vs dev) as pagemap.owner during memremap only remap vram for platform that support usm
Changes per review feedback from Brian:
s/xe_svm_devm_add/xe_devm_add s/xe_svm_devm_remove/xe_devm_remove
move calling ofxe_devm_addtoxe_tile.cSigned-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana Vishwanathapura <[email protected]>Signed-off-by Niranjana Vishwanathapura <[email protected]>Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_device_types.h | 8 +++ drivers/gpu/drm/xe/xe_mmio.c | 6 ++ drivers/gpu/drm/xe/xe_svm.h | 15 +++++ drivers/gpu/drm/xe/xe_svm_devmem.c | 89 ++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_tile.c | 4 ++ 6 files changed, 123 insertions(+) create mode 100644 drivers/gpu/drm/xe/xe_svm.h create mode 100644 drivers/gpu/drm/xe/xe_svm_devmem.cdrivers/gpu/drm/xe/Makefile +0
-0
drivers/gpu/drm/xe/Makefile xe-y += xe_bb.o \ · line 132
Quoted 4 lines
drivers/gpu/drm/xe/Makefile +1
-0@@ -129,6 +129,7 @@ xe-y += xe_bb.o \ xe_sa.o \ xe_sched_job.o \ xe_step.o \ + xe_svm_devmem.o \
This goes for the series - IMO let's have all the svm code in one file unless we have a really good reason not to. For the series I see 4:
mbrost@lstrano-desk:xe$ ls -la *.c | grep svm -rw-rw-r-- 1 mbrost mbrost 8975 Apr 10 11:17 xe_svm.c -rw-rw-r-- 1 mbrost mbrost 6774 Apr 10 11:17 xe_svm_devmem.c -rw-rw-r-- 1 mbrost mbrost 10636 Apr 10 11:17 xe_svm_migrate.c -rw-rw-r-- 1 mbrost mbrost 5940 Apr 10 11:17 xe_svm_range.c
Personally I'd like the name xe_devmem.c (or xe_devm.c) open to xe_svm.c I guess too.
Whatever name we land on let's also try to make sure all exported functions (in *.h file) start with the same prefix as the file too.
Quoted 120 lines
drivers/gpu/drm/xe/Makefile +0
-0xe_sync.o \ xe_tile.o \ xe_tile_sysfs.o \drivers/gpu/drm/xe/xe_device_types.h +8
-0@@ -103,6 +103,14 @@ struct xe_mem_region {resource_size_t actual_physical_size;/** @mapping: pointer to VRAM mappable space */ void __iomem *mapping; +/** @pagemap: Used to remap device memory as ZONE_DEVICE */ +struct dev_pagemap pagemap; +/** +* @hpa_base: base host physical address +* +* This is generated when remap device memory as ZONE_DEVICE +*/ +resource_size_t hpa_base; };/** drivers/gpu/drm/xe/xe_mmio.c +6
-0@@ -22,6 +22,7 @@#include "xe_module.h" #include "xe_sriov.h" #include "xe_tile.h" +#include "xe_svm.h" #define XEHP_MTCFG_ADDR XE_REG(0x101800)#define TILE_COUNT REG_GENMASK(15, 8)@@ -354,6 +355,11 @@ void xe_mmio_probe_tiles(struct xe_device *xe)static void mmio_fini (struct drm_device *drm,void *arg) {struct xe_device *xe = arg; +struct xe_tile *tile; +u8 id; + +for_each_tile (tile, xe, id) +xe_devm_remove (tile, &tile->mem.vram);pci_iounmap (to_pci_dev (xe->drm.dev), xe->mmio.regs);if (xe->mem.vram.mapping)drivers/gpu/drm/xe/xe_svm.h new file +15
-0@@ -0,0 +1,15 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2023 Intel Corporation +*/ + +#ifndef __XE_SVM_H +#define __XE_SVM_H + +struct xe_tile ; +struct xe_mem_region ; + +int xe_devm_add (struct xe_tile *tile,struct xe_mem_region *mr); +void xe_devm_remove (struct xe_tile *tile,struct xe_mem_region *mr); + +#endif drivers/gpu/drm/xe/xe_svm_devmem.c new file +47
-0@@ -0,0 +1,89 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2023 Intel Corporation +*/ + +#include <linux/mm_types.h> +#include <linux/sched/mm.h> + +#include "xe_device_types.h" +#include "xe_svm.h" + + +static vm_fault_t xe_devm_migrate_to_ram (struct vm_fault *vmf) +{ +return 0 ; +} + +static void xe_devm_page_free (struct page *page) +{ +} + +static const struct dev_pagemap_ops xe_devm_pagemap_ops = { + .page_free = xe_devm_page_free, + .migrate_to_ram = xe_devm_migrate_to_ram, +}; + +/** +* xe_devm_add: Remap and provide memmap backing for device memory +* @tile: tile that the memory region blongs to +* @mr: memory region to remap +* +* This remap device memory to host physical address space and create +* struct page to back device memory +* +* Return: 0 on success standard error code otherwise +*/ +int xe_devm_add (struct xe_tile *tile,struct xe_mem_region *mr) +{ +struct xe_device *xe =tile_to_xe (tile); +struct device *dev = &to_pci_dev (xe->drm.dev)->dev; +struct resource *res; +void *addr; +int ret; + + res =devm_request_free_mem_region (dev, &iomem_resource, + mr->usable_size); +if (IS_ERR (res)) {
drivers/gpu/drm/xe/xe_svm_devmem.c line 57
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c new file +10
-0+ ret =PTR_ERR (res); +return ret; + } + + mr->pagemap.type =MEMORY_DEVICE_PRIVATE ; + mr->pagemap.range.start = res->start; + mr->pagemap.range.end = res->end; + mr->pagemap.nr_range =1 ; + mr->pagemap.ops = &xe_devm_pagemap_ops; + mr->pagemap.owner = xe;
Nit: I know I suggested this another series too - helper to go from xe -> owner which can be used in various places we set this.
Quoted 15 lines
drivers/gpu/drm/xe/xe_svm_devmem.c new file +15
-0+ addr =devm_memremap_pages (dev, &mr->pagemap); +if (IS_ERR (addr)) { +devm_release_mem_region (dev, res->start,resource_size (res)); + ret =PTR_ERR (addr); +drm_err (&xe->drm,"Failed to remap tile %d memory, errno %d\n" , + tile->id, ret); +return ret; + } + mr->hpa_base = res->start; + +drm_info (&xe->drm,"Added tile %d memory [%llx-%llx] to devm, remapped to %pr\n" , + tile->id, mr->io_start, mr->io_start + mr->usable_size, res); +return 0 ; +} +
drivers/gpu/drm/xe/xe_svm_devmem.c line 82
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c new file +10
-0+/** +* xe_devm_remove: Unmap device memory and free resources +* @tile: xe tile +* @mr: memory region to remove +*/ +void xe_devm_remove (struct xe_tile *tile,struct xe_mem_region *mr) +{ +struct device *dev = &to_pci_dev (tile->xe->drm.dev)->dev; + +/*FIXME: Does below cause a kernel hange during moduel remove?*/
Goes for the series, try resolve issues rather than having FIXMEs.
Matt
Quoted 39 lines
drivers/gpu/drm/xe/xe_svm_devmem.c new file +7
-0+if (mr->hpa_base) { +devm_memunmap_pages (dev, &mr->pagemap); +devm_release_mem_region (dev, mr->pagemap.range.start, + mr->pagemap.range.end - mr->pagemap.range.start +1 ); + } +} +drivers/gpu/drm/xe/xe_tile.c +4
-0@@ -14,6 +14,7 @@#include "xe_tile_sysfs.h" #include "xe_ttm_vram_mgr.h" #include "xe_wa.h" +#include "xe_svm.h" /** * DOC: Multi-tile Design @@ -158,6 +159,7 @@ static int tile_ttm_mgr_init(struct xe_tile *tile)*/ int xe_tile_init_noalloc (struct xe_tile *tile) { +struct xe_device *xe =tile_to_xe (tile);int err;xe_device_mem_access_get (tile_to_xe (tile));@@ -175,6 +177,8 @@ int xe_tile_init_noalloc(struct xe_tile *tile)xe_tile_sysfs_init (tile); +if (xe->info.has_usm) +xe_devm_add (tile, &tile->mem.vram); err_mem_access:xe_device_mem_access_put (tile_to_xe (tile));return err;Signature · 1 line
2.26.3
Matthew Brost
maintainerRe: [v2 12/31] drm/xe/svm: Remap and provide memmap backing for GPU vram
In reply to Oak ZengQuoted 200 lines · Oak Zeng
Memory remap GPU vram using
devm_memremap_pages, so each GPU vram page is backed by a struct page.Those struct pages are created to allow hmm migrate buffer b/t GPU vram and CPU system memory using existing Linux migration mechanism (i.e., migrating b/t CPU system memory and hard disk).
This is prepare work to enable svm (shared virtual memory) through Linux kernel hmm framework. The memory remap's page map type is set to
MEMORY_DEVICE_PRIVATEfor now. This means even though each GPU vram page get a struct page and can be mapped in CPU page table, but such pages are treated as GPU's private resource, so CPU can't access them. If CPU access such page, a page fault is triggered and page will be migrate to system memory.For GPU device which supports coherent memory protocol b/t CPU and GPU (such as CXL and CAPI protocol), we can remap device memory as
MEMORY_DEVICE_COHERENT. This is TBD.v1:
Changes per code review feedback from Matt:
change .o order inMakefile
fix indentation
change code order inmmio_fini
remove unnecessary header file
uniform xe_svm_devm_add/_remove parameter use tile (vs dev) as pagemap.owner during memremap only remap vram for platform that support usm
Changes per review feedback from Brian:
s/xe_svm_devm_add/xe_devm_add s/xe_svm_devm_remove/xe_devm_remove
move calling ofxe_devm_addtoxe_tile.cSigned-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana Vishwanathapura <[email protected]>Signed-off-by Niranjana Vishwanathapura <[email protected]>Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_device_types.h | 8 +++ drivers/gpu/drm/xe/xe_mmio.c | 6 ++ drivers/gpu/drm/xe/xe_svm.h | 15 +++++ drivers/gpu/drm/xe/xe_svm_devmem.c | 89 ++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_tile.c | 4 ++ 6 files changed, 123 insertions(+) create mode 100644 drivers/gpu/drm/xe/xe_svm.h create mode 100644 drivers/gpu/drm/xe/xe_svm_devmem.cdrivers/gpu/drm/xe/Makefile +1
-0@@ -129,6 +129,7 @@ xe-y += xe_bb.o \ xe_sa.o \ xe_sched_job.o \ xe_step.o \ + xe_svm_devmem.o \ xe_sync.o \ xe_tile.o \ xe_tile_sysfs.o \drivers/gpu/drm/xe/xe_device_types.h +8
-0@@ -103,6 +103,14 @@ struct xe_mem_region {resource_size_t actual_physical_size;/** @mapping: pointer to VRAM mappable space */ void __iomem *mapping; +/** @pagemap: Used to remap device memory as ZONE_DEVICE */ +struct dev_pagemap pagemap; +/** +* @hpa_base: base host physical address +* +* This is generated when remap device memory as ZONE_DEVICE +*/ +resource_size_t hpa_base; };/** drivers/gpu/drm/xe/xe_mmio.c +6
-0@@ -22,6 +22,7 @@#include "xe_module.h" #include "xe_sriov.h" #include "xe_tile.h" +#include "xe_svm.h" #define XEHP_MTCFG_ADDR XE_REG(0x101800)#define TILE_COUNT REG_GENMASK(15, 8)@@ -354,6 +355,11 @@ void xe_mmio_probe_tiles(struct xe_device *xe)static void mmio_fini (struct drm_device *drm,void *arg) {struct xe_device *xe = arg; +struct xe_tile *tile; +u8 id; + +for_each_tile (tile, xe, id) +xe_devm_remove (tile, &tile->mem.vram);pci_iounmap (to_pci_dev (xe->drm.dev), xe->mmio.regs);if (xe->mem.vram.mapping)drivers/gpu/drm/xe/xe_svm.h new file +15
-0@@ -0,0 +1,15 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2023 Intel Corporation +*/ + +#ifndef __XE_SVM_H +#define __XE_SVM_H + +struct xe_tile ; +struct xe_mem_region ; + +int xe_devm_add (struct xe_tile *tile,struct xe_mem_region *mr); +void xe_devm_remove (struct xe_tile *tile,struct xe_mem_region *mr); + +#endif drivers/gpu/drm/xe/xe_svm_devmem.c new file +68
-0@@ -0,0 +1,89 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2023 Intel Corporation +*/ + +#include <linux/mm_types.h> +#include <linux/sched/mm.h> + +#include "xe_device_types.h" +#include "xe_svm.h" + + +static vm_fault_t xe_devm_migrate_to_ram (struct vm_fault *vmf) +{ +return 0 ; +} + +static void xe_devm_page_free (struct page *page) +{ +} + +static const struct dev_pagemap_ops xe_devm_pagemap_ops = { + .page_free = xe_devm_page_free, + .migrate_to_ram = xe_devm_migrate_to_ram, +}; + +/** +* xe_devm_add: Remap and provide memmap backing for device memory +* @tile: tile that the memory region blongs to +* @mr: memory region to remap +* +* This remap device memory to host physical address space and create +* struct page to back device memory +* +* Return: 0 on success standard error code otherwise +*/ +int xe_devm_add (struct xe_tile *tile,struct xe_mem_region *mr) +{ +struct xe_device *xe =tile_to_xe (tile); +struct device *dev = &to_pci_dev (xe->drm.dev)->dev; +struct resource *res; +void *addr; +int ret; + + res =devm_request_free_mem_region (dev, &iomem_resource, + mr->usable_size); +if (IS_ERR (res)) { + ret =PTR_ERR (res); +return ret; + } + + mr->pagemap.type =MEMORY_DEVICE_PRIVATE ; + mr->pagemap.range.start = res->start; + mr->pagemap.range.end = res->end; + mr->pagemap.nr_range =1 ; + mr->pagemap.ops = &xe_devm_pagemap_ops; + mr->pagemap.owner = xe; + addr =devm_memremap_pages (dev, &mr->pagemap); +if (IS_ERR (addr)) { +devm_release_mem_region (dev, res->start,resource_size (res)); + ret =PTR_ERR (addr); +drm_err (&xe->drm,"Failed to remap tile %d memory, errno %d\n" , + tile->id, ret); +return ret; + } + mr->hpa_base = res->start; + + drm_info(&xe->drm, "Added tile %d memory [%llx-%llx] to devm, remapped to %pr\n",
drivers/gpu/drm/xe/xe_svm_devmem.c line 78
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c new file +10
-0+ tile->id, mr->io_start, mr->io_start + mr->usable_size, res); +return 0 ; +} + +/** +* xe_devm_remove: Unmap device memory and free resources +* @tile: xe tile +* @mr: memory region to remove +*/ +void xe_devm_remove (struct xe_tile *tile,struct xe_mem_region *mr)
Also I don't think function is not needed...
devm_memremap_pages registers devm_memremap_pages_release via evm_add_action_or_reset...
And if it was we'd want to register a devm_fini function rather than exporting a function and call it from the mmio layer.
Matt
Quoted 43 lines
drivers/gpu/drm/xe/xe_svm_devmem.c new file +11
-0+{ +struct device *dev = &to_pci_dev (tile->xe->drm.dev)->dev; + +/*FIXME: Does below cause a kernel hange during moduel remove?*/ +if (mr->hpa_base) { +devm_memunmap_pages (dev, &mr->pagemap); +devm_release_mem_region (dev, mr->pagemap.range.start, + mr->pagemap.range.end - mr->pagemap.range.start +1 ); + } +} +drivers/gpu/drm/xe/xe_tile.c +4
-0@@ -14,6 +14,7 @@#include "xe_tile_sysfs.h" #include "xe_ttm_vram_mgr.h" #include "xe_wa.h" +#include "xe_svm.h" /** * DOC: Multi-tile Design @@ -158,6 +159,7 @@ static int tile_ttm_mgr_init(struct xe_tile *tile)*/ int xe_tile_init_noalloc (struct xe_tile *tile) { +struct xe_device *xe =tile_to_xe (tile);int err;xe_device_mem_access_get (tile_to_xe (tile));@@ -175,6 +177,8 @@ int xe_tile_init_noalloc(struct xe_tile *tile)xe_tile_sysfs_init (tile); +if (xe->info.has_usm) +xe_devm_add (tile, &tile->mem.vram); err_mem_access:xe_device_mem_access_put (tile_to_xe (tile));return err;Signature · 1 line
2.26.3
PATCH v2 13/31 drm/xe/svm: Introduce DRM_XE_SVM kernel config
In reply to Oak ZengIntroduce a DRM_XE_SVM kernel config entry for xe svm feature. xe svm feature allows share virtual address space between CPU and GPU program.
v1: Improve commit message (Thomas)
Avoid using #if directive (Thomas)
--- drivers/gpu/drm/xe/Kconfig | 21 +++++++++++++++++++++ drivers/gpu/drm/xe/xe_tile.c | 7 +++++-- 2 files changed, 26 insertions(+), 2 deletions(-)
drivers/gpu/drm/xe/Kconfig +21 -0
@@ -84,6 +84,27 @@ config DRM_XE_FORCE_PROBE 4571. Use "!*" to block the probe of the driver for all known devices. +config DRM_XE_SVM + bool "Enable Shared Virtual Memory support in xe" + depends on DRM_XE + depends on ARCH_ENABLE_MEMORY_HOTPLUG + depends on ARCH_ENABLE_MEMORY_HOTREMOVE + depends on MEMORY_HOTPLUG + depends on MEMORY_HOTREMOVE + depends on ARCH_HAS_PTE_DEVMAP + depends on SPARSEMEM_VMEMMAP + depends on ZONE_DEVICE + depends on DEVICE_PRIVATE + depends on MMU + select HMM_MIRROR + select MMU_NOTIFIER + default y + help + Choose this option if you want Shared Virtual Memory (SVM) + support in xe. With SVM, virtual address space is shared + between CPU and GPU. This means any virtual address such + as malloc or mmap returns, variables on stack, or global + memory pointers, can be used for GPU transparently. menu "drm/Xe Debugging" depends on DRM_XE
drivers/gpu/drm/xe/xe_tile.c +5 -2
@@ -159,9 +159,12 @@ static int tile_ttm_mgr_init(struct xe_tile *tile)*/ int xe_tile_init_noalloc (struct xe_tile *tile) { -struct xe_device *xe =tile_to_xe (tile); +struct xe_device __maybe_unused *xe;int err; +if (IS_ENABLED (CONFIG_DRM_XE_SVM )) + xe =tile_to_xe (tile); +xe_device_mem_access_get (tile_to_xe (tile)); err =tile_ttm_mgr_init (tile);
@@ -177,7 +180,7 @@ int xe_tile_init_noalloc(struct xe_tile *tile)xe_tile_sysfs_init (tile); -if (xe->info.has_usm) +if (IS_ENABLED (CONFIG_DRM_XE_SVM ) && xe->info.has_usm)xe_devm_add (tile, &tile->mem.vram); err_mem_access:xe_device_mem_access_put (tile_to_xe (tile));
Signature · 1 line
2.26.3
Matthew Brost
maintainerRe: [v2 13/31] drm/xe/svm: Introduce DRM_XE_SVM kernel config
Quoted 1 line · Oak Zeng
Introduce a
DRM_XE_SVMkernel config entry for
Maybe consider another name for this? I could see use cases for non-SVM where we still want private pages mapped (e.g. VRAM userptrs on non-faulting devices). Don't really have suggestion but worth considering.
Quoted 54 lines
xe svm feature. xe svm feature allows share virtual address space between CPU and GPU program.
v1: Improve commit message (Thomas)
Avoid using #if directive (Thomas)Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana Vishwanathapura <[email protected]>Signed-off-by Niranjana Vishwanathapura <[email protected]>Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/Kconfig | 21 +++++++++++++++++++++ drivers/gpu/drm/xe/xe_tile.c | 7 +++++-- 2 files changed, 26 insertions(+), 2 deletions(-)drivers/gpu/drm/xe/Kconfig +21
-0@@ -84,6 +84,27 @@ config DRM_XE_FORCE_PROBE 4571. Use "!*" to block the probe of the driver for all known devices. +config DRM_XE_SVM + bool "Enable Shared Virtual Memory support in xe" + depends on DRM_XE + depends on ARCH_ENABLE_MEMORY_HOTPLUG + depends on ARCH_ENABLE_MEMORY_HOTREMOVE + depends on MEMORY_HOTPLUG + depends on MEMORY_HOTREMOVE + depends on ARCH_HAS_PTE_DEVMAP + depends on SPARSEMEM_VMEMMAP + depends on ZONE_DEVICE + depends on DEVICE_PRIVATE + depends on MMU + select HMM_MIRROR + select MMU_NOTIFIER + default y + help + Choose this option if you want Shared Virtual Memory (SVM) + support in xe. With SVM, virtual address space is shared + between CPU and GPU. This means any virtual address such + as malloc or mmap returns, variables on stack, or global + memory pointers, can be used for GPU transparently. menu "drm/Xe Debugging" depends on DRM_XEdrivers/gpu/drm/xe/xe_tile.c +0
-0
drivers/gpu/drm/xe/xe_tile.c static int tile_ttm_mgr_init(struct xe_tile *tile) · line 162
Quoted 5 lines
drivers/gpu/drm/xe/xe_tile.c +1
-1@@ -159,9 +159,12 @@ static int tile_ttm_mgr_init(struct xe_tile *tile)*/ int xe_tile_init_noalloc (struct xe_tile *tile) { -struct xe_device *xe =tile_to_xe (tile); +struct xe_device __maybe_unused *xe;
Just assign this here blindly? The __maybe_unused should suppress the warning CONFIG_DRM_XE_SVM is false and should just compile out if it is.
Matt
Quoted 20 lines
drivers/gpu/drm/xe/xe_tile.c +4
-1int err; +if (IS_ENABLED (CONFIG_DRM_XE_SVM )) + xe =tile_to_xe (tile); +xe_device_mem_access_get (tile_to_xe (tile)); err =tile_ttm_mgr_init (tile);@@ -177,7 +180,7 @@ int xe_tile_init_noalloc(struct xe_tile *tile)xe_tile_sysfs_init (tile); -if (xe->info.has_usm) +if (IS_ENABLED (CONFIG_DRM_XE_SVM ) && xe->info.has_usm)xe_devm_add (tile, &tile->mem.vram); err_mem_access:xe_device_mem_access_put (tile_to_xe (tile));Signature · 1 line
2.26.3
RE: [v2 13/31] drm/xe/svm: Introduce DRM_XE_SVM kernel config
Hi Matt,
I replied a few to those review comments. Then I was interrupted by the v3 respin of this series.
I now come back to revisit those comments, to make sure those comments are either get address in v3, or I reply if I have more comments.
Quoted 10 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 5:13 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 13/31] drm/xe/svm: IntroduceDRM_XE_SVMkernel config
Quoted 7 lines
Quoted 1 line · Oak Zeng
Introduce a
DRM_XE_SVMkernel config entry forMaybe consider another name for this? I could see use cases for non-SVM where we still want private pages mapped (e.g. VRAM userptrs on non-faulting devices). Don't really have suggestion but worth considering.
It is first time I heard the concept of VRAM userptr. Do we plan to support it?
On a non-faulting device, you would have to allocate vram for userptr at vm_bind time, right?
Quoted 50 lines
Quoted 9 lines
xe svm feature. xe svm feature allows share virtual address space between CPU and GPU program.
v1: Improve commit message (Thomas)
Avoid using #if directive (Thomas)Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana VishwanathapuraQuoted 1 line
Signed-off-by Niranjana VishwanathapuraQuoted 38 lines
Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/Kconfig | 21 +++++++++++++++++++++ drivers/gpu/drm/xe/xe_tile.c | 7 +++++-- 2 files changed, 26 insertions(+), 2 deletions(-)drivers/gpu/drm/xe/Kconfig +21
-0@@ -84,6 +84,27 @@ config DRM_XE_FORCE_PROBE 4571. Use "!*" to block the probe of the driver for all known devices. +config DRM_XE_SVM + bool "Enable Shared Virtual Memory support in xe" + depends on DRM_XE + depends on ARCH_ENABLE_MEMORY_HOTPLUG + depends on ARCH_ENABLE_MEMORY_HOTREMOVE + depends on MEMORY_HOTPLUG + depends on MEMORY_HOTREMOVE + depends on ARCH_HAS_PTE_DEVMAP + depends on SPARSEMEM_VMEMMAP + depends on ZONE_DEVICE + depends on DEVICE_PRIVATE + depends on MMU + select HMM_MIRROR + select MMU_NOTIFIER + default y + help + Choose this option if you want Shared Virtual Memory (SVM) + support in xe. With SVM, virtual address space is shared + between CPU and GPU. This means any virtual address such + as malloc or mmap returns, variables on stack, or global + memory pointers, can be used for GPU transparently.
Quoted 15 lines
drivers/gpu/drm/xe/xe_tile.c static int tile_ttm_mgr_init(struct xe_tile *tile) · line 162
Quoted 12 lines
drivers/gpu/drm/xe/Kconfig +0
-0menu "drm/Xe Debugging" depends on DRM_XEdrivers/gpu/drm/xe/xe_tile.c +1
-1@@ -159,9 +159,12 @@ static int tile_ttm_mgr_init(struct xe_tile *tile)*/ int xe_tile_init_noalloc (struct xe_tile *tile) { -struct xe_device *xe =tile_to_xe (tile); +struct xe_device __maybe_unused *xe;Just assign this here blindly? The
__maybe_unusedshould suppress the warningCONFIG_DRM_XE_SVMis false and should just compile out if it is.
Did you mean:
struct xe_device __maybe_unused *xe = tile_to_xe(tile);
we have an extra xe assignment when DRM_XE_SVM is not enabled.
Also if we assign blindly, we wouldn't need __maybe_unused, as xe is always assigned (so used)
Oak
Quoted 23 lines
Matt
Quoted 20 lines
drivers/gpu/drm/xe/xe_tile.c +4
-2int err; +if (IS_ENABLED (CONFIG_DRM_XE_SVM )) + xe =tile_to_xe (tile); +xe_device_mem_access_get (tile_to_xe (tile)); err =tile_ttm_mgr_init (tile);@@ -177,7 +180,7 @@ int xe_tile_init_noalloc(struct xe_tile *tile)xe_tile_sysfs_init (tile); -if (xe->info.has_usm) +if (IS_ENABLED (CONFIG_DRM_XE_SVM ) && xe->info.has_usm)xe_devm_add (tile, &tile->mem.vram); err_mem_access:xe_device_mem_access_put (tile_to_xe (tile)); --2.26.3
PATCH v2 14/31 drm/xe: Introduce helper to get tile from memory region
In reply to Oak ZengIntroduce a simple helper to retrieve tile from memory region
v1: move the function to xe_device.h (Matt)
improve commit message, add kerneldoc (Thomas)
--- drivers/gpu/drm/xe/xe_device.h | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/gpu/drm/xe/xe_device.h +8 -0
@@ -178,4 +178,12 @@ u64 xe_device_uncanonicalize_addr(struct xe_device *xe, u64 address);void xe_device_put_deferred (struct xe_device *xe,struct llist_node *deferred); +/** +* xe_mem_region_to_tile() - retrieve tile from memory region +* @mr: the memory region we retrieve tile from +*/ +static inline struct xe_tile *xe_mem_region_to_tile (struct xe_mem_region *mr) +{ + return container_of(mr, struct xe_tile, mem.vram); +}#endif
Signature · 1 line
2.26.3
Matthew Brost
maintainerRe: [v2 14/31] drm/xe: Introduce helper to get tile from memory region
Quoted 6 lines · Oak Zeng
Introduce a simple helper to retrieve tile from memory region
v1: move the function to
xe_device.h(Matt)
improve commit message, add kerneldoc (Thomas)Signed-off-by Oak Zeng <[email protected]>
This LGTM but can it be moved to xe_tile.h? That might be a better place but I know xe_device.h and xe_tile.h are intertwined a bit.
Matt
Quoted 24 lines
--- drivers/gpu/drm/xe/xe_device.h | 8 ++++++++ 1 file changed, 8 insertions(+)drivers/gpu/drm/xe/xe_device.h +8
-0@@ -178,4 +178,12 @@ u64 xe_device_uncanonicalize_addr(struct xe_device *xe, u64 address);void xe_device_put_deferred (struct xe_device *xe,struct llist_node *deferred); +/** +* xe_mem_region_to_tile() - retrieve tile from memory region +* @mr: the memory region we retrieve tile from +*/ +static inline struct xe_tile *xe_mem_region_to_tile (struct xe_mem_region *mr) +{ + return container_of(mr, struct xe_tile, mem.vram); +}#endif Signature · 1 line
2.26.3
PATCH v2 15/31 drm/xe: Introduce a helper to get dpa from pfn
In reply to Oak ZengSince we now create struct page backing for each vram page, each vram page now also has a pfn, just like system memory. This allow us to calcuate device physical address from pfn.
v1: move the function to xe_svm.h (Matt)
s/vram_pfn_to_dpa/xe_mem_region_pfn_to_dpa (Matt) add kernel document for the helper (Thomas)
--- drivers/gpu/drm/xe/xe_svm.h | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_svm.h +25 -2
@@ -6,8 +6,31 @@#ifndef __XE_SVM_H#define __XE_SVM_H -struct xe_tile ; -struct xe_mem_region ; +#include "xe_device_types.h" +#include "xe_device.h" +#include "xe_assert.h" + +/** +* xe_mem_region_pfn_to_dpa() - Calculate page's dpa from pfn +* +* @mr: The memory region that page resides in +* @pfn: page frame number of the page +* +* Returns: the device physical address of the page +*/ +static inline u64 xe_mem_region_pfn_to_dpa (struct xe_mem_region *mr,u64 pfn) +{ +u64 dpa; +struct xe_tile *tile =xe_mem_region_to_tile (mr); +struct xe_device *xe =tile_to_xe (tile); +u64 offset; + +xe_assert (xe, (pfn <<PAGE_SHIFT ) >= mr->hpa_base); + offset = (pfn <<PAGE_SHIFT ) - mr->hpa_base; + dpa = mr->dpa_base + offset; + +return dpa; +}int xe_devm_add (struct xe_tile *tile,struct xe_mem_region *mr);void xe_devm_remove (struct xe_tile *tile,struct xe_mem_region *mr);
Signature · 1 line
2.26.3
Matthew Brost
maintainerRe: [v2 15/31] drm/xe: Introduce a helper to get dpa from pfn
Quoted 18 lines · Oak Zeng
Since we now create struct page backing for each vram page, each vram page now also has a pfn, just like system memory. This allow us to calcuate device physical address from pfn.
v1: move the function to
xe_svm.h(Matt)
s/vram_pfn_to_dpa/xe_mem_region_pfn_to_dpa (Matt) add kernel document for the helper (Thomas)Signed-off-by Oak Zeng <[email protected]>--- drivers/gpu/drm/xe/xe_svm.h | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-)drivers/gpu/drm/xe/xe_svm.h +0
-0
drivers/gpu/drm/xe/xe_svm.h line 11
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm.h +3
-2@@ -6,8 +6,31 @@#ifndef __XE_SVM_H#define __XE_SVM_H -struct xe_tile ; -struct xe_mem_region ; +#include "xe_device_types.h" +#include "xe_device.h" +#include "xe_assert.h"
Hmm, including all these headers is a frowned upon and indicates to me this likely the wrong location. The new header hopefully is clean with only forward delc and function defs. I know Xe headers are not great at this but lets not make this worse than it is.
Maybe should be xe_device.h? Also if we move the entire implementation 1 *.c file it is possible this function be private to that C file too.
drivers/gpu/drm/xe/xe_svm.h line 21
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm.h +10
-0+ +/** +* xe_mem_region_pfn_to_dpa() - Calculate page's dpa from pfn +* +* @mr: The memory region that page resides in +* @pfn: page frame number of the page +* +* Returns: the device physical address of the page +*/ +static inline u64 xe_mem_region_pfn_to_dpa (struct xe_mem_region *mr,u64 pfn)
I'd change to xe_mem_region_page_to_dpa with a struct page argument rather than pfn. The pfn then be derived from the page.
I think this better as we will 3 types pfn all with different values / shifts.
- hmm pfn
- migrate pfn
- linux core pfn
If a migrate pfn or hmm pfn were passed in as argument we'd get the wrong dpa. I think passing in a page is safer and less bug prone. In my example if we had a migrate pfn or hmm pfn, we'd use the appropriate helper to get the struct page.
This also aligns with how a similar AMD helper (svm_migrate_addr, [1]) is implemented.
Matt
[1] https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c#L234
Quoted 18 lines
drivers/gpu/drm/xe/xe_svm.h +12
-0+{ +u64 dpa; +struct xe_tile *tile =xe_mem_region_to_tile (mr); +struct xe_device *xe =tile_to_xe (tile); +u64 offset; + +xe_assert (xe, (pfn <<PAGE_SHIFT ) >= mr->hpa_base); + offset = (pfn <<PAGE_SHIFT ) - mr->hpa_base; + dpa = mr->dpa_base + offset; + +return dpa; +}int xe_devm_add (struct xe_tile *tile,struct xe_mem_region *mr);void xe_devm_remove (struct xe_tile *tile,struct xe_mem_region *mr);Signature · 1 line
2.26.3
PATCH v2 16/31 drm/xe/svm: Get xe memory region from page
In reply to Oak ZengFor gpu vram page, we now have a struct page backing of it. struct page's pgmap points to xe_memory_region's pagemap. This allow us to retrieve xe_memory_region from struct page.
v1: move the function to xe_svm.h
--- drivers/gpu/drm/xe/xe_svm.h | 11 +++++++++++ 1 file changed, 11 insertions(+)
drivers/gpu/drm/xe/xe_svm.h +11 -0
@@ -6,6 +6,7 @@#ifndef __XE_SVM_H#define __XE_SVM_H +#include <linux/mm_types.h> #include "xe_device_types.h" #include "xe_device.h" #include "xe_assert.h"
@@ -35,4 +36,14 @@ static inline u64 xe_mem_region_pfn_to_dpa(struct xe_mem_region *mr, u64 pfn)int xe_devm_add (struct xe_tile *tile,struct xe_mem_region *mr);void xe_devm_remove (struct xe_tile *tile,struct xe_mem_region *mr); +/** +* xe_page_to_mem_region() - Get a page's memory region +* +* @page: a struct page pointer pointing to a page in vram memory region +*/ +static inline struct xe_mem_region *xe_page_to_mem_region (struct page *page) +{ + return container_of(page->pgmap, struct xe_mem_region, pagemap); +} +#endif
Signature · 1 line
2.26.3
Matthew Brost
maintainerRe: [v2 16/31] drm/xe/svm: Get xe memory region from page
drivers/gpu/drm/xe/xe_svm.h
Quoted 16 lines · Oak Zeng
For gpu vram page, we now have a struct page backing of it. struct page's pgmap points to xe_memory_region's pagemap. This allow us to retrieve
xe_memory_regionfrom struct page.v1: move the function to
xe_svm.hSigned-off-by Oak Zeng <[email protected]>--- drivers/gpu/drm/xe/xe_svm.h | 11 +++++++++++ 1 file changed, 11 insertions(+)drivers/gpu/drm/xe/xe_svm.h +0
-0
See my comments about location in previous patch and also if this can be a private function if implementation is in 1 *.c file.
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm.h +1
-0@@ -6,6 +6,7 @@#ifndef __XE_SVM_H#define __XE_SVM_H +#include <linux/mm_types.h> #include "xe_device_types.h" #include "xe_device.h" #include "xe_assert.h" @@ -35,4 +36,14 @@ static inline u64 xe_mem_region_pfn_to_dpa(struct xe_mem_region *mr, u64 pfn)int xe_devm_add (struct xe_tile *tile,struct xe_mem_region *mr);void xe_devm_remove (struct xe_tile *tile,struct xe_mem_region *mr);
drivers/gpu/drm/xe/xe_svm.h static inline u64 xe_mem_region_pfn_to_dpa(struct xe_mem_region *mr, u64 pfn) · line 47
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm.h +9
-0+/** +* xe_page_to_mem_region() - Get a page's memory region +* +* @page: a struct page pointer pointing to a page in vram memory region +*/ +static inline struct xe_mem_region *xe_page_to_mem_region (struct page *page) +{ + return container_of(page->pgmap, struct xe_mem_region, pagemap); +}
If the previous patch is xe_mem_region_page_to_dpa and want very robust code we could add an assert to that function.
xe_assert(xe, mr == xe_page_to_mem_region(page));
Matt
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm.h +1
-0+#endif Signature · 1 line
2.26.3
PATCH v2 17/31 drm/xe: Get xe_vma from xe_userptr
In reply to Oak ZengIntroduce a helper to get xe_vma from xe_userptr.
--- drivers/gpu/drm/xe/xe_vm.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
drivers/gpu/drm/xe/xe_vm.h +14 -0
@@ -178,6 +178,20 @@ static inline struct xe_userptr_vma *to_userptr_vma(struct xe_vma *vma) return container_of(vma, struct xe_userptr_vma, vma); } +/** +* xe_userptr_to_vma() - Return xe_vma from a xe_userptr pointer +* +* @userptr: The userptr struct pointer +*/ + +static inline struct xe_vma *xe_userptr_to_vma (struct xe_userptr *userptr) +{ +struct xe_userptr_vma *uvma; + + uvma = container_of(userptr, struct xe_userptr_vma, userptr); +return &uvma->vma; +} +u64 xe_vm_pdp4_descriptor (struct xe_vm *vm,struct xe_tile *tile); int xe_vm_create_ioctl(struct drm_device *dev, void *data,
Signature · 1 line
2.26.3
Matthew Brost
maintainerRe: [v2 17/31] drm/xe: Get xe_vma from xe_userptr
Quoted 12 lines · Oak Zeng
Introduce a helper to get
xe_vmafromxe_userptr.Signed-off-by Oak Zeng <[email protected]>--- drivers/gpu/drm/xe/xe_vm.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+)drivers/gpu/drm/xe/xe_vm.h +0
-0
drivers/gpu/drm/xe/xe_vm.h static inline struct xe_userptr_vma *to_userptr_vma(struct xe_vma *vma) · line 186
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.h +6
-0@@ -178,6 +178,20 @@ static inline struct xe_userptr_vma *to_userptr_vma(struct xe_vma *vma) return container_of(vma, struct xe_userptr_vma, vma); } +/** +* xe_userptr_to_vma() - Return xe_vma from a xe_userptr pointer +* +* @userptr: The userptr struct pointer +*/ +
Extra newline. Otherwise LGTM.
Matt
Quoted 14 lines
drivers/gpu/drm/xe/xe_vm.h +8
-0+static inline struct xe_vma *xe_userptr_to_vma (struct xe_userptr *userptr) +{ +struct xe_userptr_vma *uvma; + + uvma = container_of(userptr, struct xe_userptr_vma, userptr); +return &uvma->vma; +} +u64 xe_vm_pdp4_descriptor (struct xe_vm *vm,struct xe_tile *tile); int xe_vm_create_ioctl(struct drm_device *dev, void *data,Signature · 1 line
2.26.3
PATCH v2 18/31 drm/xe/svm: Build userptr sg table for device pages
In reply to Oak ZengPreviously function xe_build_sg only support userptr with system memory pages. Now this function is extended to support userptr with device pages backing as well.
For device pages, there is no need of dma-mapping. Instead, we calculated the device page's dpa (device physical address) and use dpa to fill sg table.
As of now, we assume each userptr is only backed either by all system memory pages or all by device pages. There is no support of mixture backing of device and system memory pages.
--- drivers/gpu/drm/xe/xe_hmm.c | 121 +++++++++++++++++++++++++------ drivers/gpu/drm/xe/xe_vm_types.h | 2 + 2 files changed, 100 insertions(+), 23 deletions(-)
drivers/gpu/drm/xe/xe_hmm.c +98 -23
@@ -11,6 +11,7 @@#include <linux/hmm.h> #include <linux/mm.h> #include "xe_hmm.h" +#include "xe_svm.h" #include "xe_vm.h" #include "xe_bo.h"
@@ -43,15 +44,90 @@ static void xe_mark_range_accessed(struct hmm_range *range, bool write) } } +/** +* xe_build_sg_device_pages() - build sg table for userptr when the backing store +* is device pages +* +* @st: sg table to build +* @hmm_pfns: pfn array of the userptr +* @pages: struct page arrary of this userptr +* @npages: how many pages in this userptr +*/ +static int xe_build_sg_device_pages (struct sg_table *st,unsigned long *hmm_pfns, +struct page **pages,uint64_t npages) +{ +struct scatterlist *sg; +int i; + + sg =NULL ; + st->nents =0 ; +if (unlikely (sg_alloc_table (st, npages,GFP_KERNEL ))) +return -ENOMEM ; + +for (i =0 ; i < npages; i++) { +unsigned long addr; +struct xe_mem_region *mr; + + mr =xe_page_to_mem_region (pages[i]); + addr =xe_mem_region_pfn_to_dpa (mr, hmm_pfns[i]); +if (sg && (addr == (sg_dma_address (sg) + sg->length))) { + sg->length +=PAGE_SIZE ; +sg_dma_len (sg) +=PAGE_SIZE ; +continue ; + } + + sg = sg ?sg_next (sg) : st->sgl; +sg_dma_address (sg) = addr; +sg_dma_len (sg) =PAGE_SIZE ; + sg->length =PAGE_SIZE ; + st->nents++; + } + +sg_mark_end (sg); +return 0 ; +} + +/** +* xe_validate_hmm_pfns() - validate all pages in a userptr belong to one memory +* region, and populate the pages array. +* +* @userptr: The userptr to validate +* @hmm_pfns: an array holding hmm pfns +* @npages: number of pages of this userptr +* @pages: output parameter to hold the populated pages from pfn. +*/ +static void xe_validate_hmm_pfns (struct xe_userptr *userptr,unsigned long *hmm_pfns, +uint64_t npages,struct page **pages) +{ +int i; +struct xe_vma *vma =xe_userptr_to_vma (userptr); +struct xe_vm *vm =xe_vma_vm (vma); + + pages[0 ] =hmm_pfn_to_page (hmm_pfns[0 ]); + userptr->is_device_pages =is_device_private_page (pages[0 ]); +for (i =1 ; i < npages; i++) { + pages[i] =hmm_pfn_to_page (hmm_pfns[i]); +/** +* We currently assume no mixture of device pages and system memory +* pages in one userptr. If it turns out this is not true, we will +* either split userptr into device pages based and system memory +* based, or support a mixture backing store in one userptr. +*/ +xe_assert (vm->xe, + userptr->is_device_pages ==is_device_private_page (pages[i])); + } +} + +/** * xe_build_sg() - build a scatter gather table for all the physical pages/pfn * in a hmm_range. dma-map pages if necessary. dma-address is save in sg table * and will be used to program GPU page table later. * * @xe: the xe device who will access the dma-address in sg table +* @userptr: the userptr that we build the sg table for * @range: the hmm range that we build the sg table from. range->hmm_pfns[] * has the pfn numbers of pages that back up this hmm address range. -* @st: pointer to the sg table. * @write: whether we write to this range. This decides dma map direction * for system pages. If write we map it bi-diretional; otherwise * DMA_TO_DEVICE
@@ -64,11 +140,6 @@ static void xe_mark_range_accessed(struct hmm_range *range, bool write)* access memory. So if the memory is system memory, we need to * do a dma-mapping so it can be accessed by GPU/DMA. * -* FIXME: This function currently only support pages in system -* memory. If the memory is GPU local memory (of the GPU who -* is going to access memory), we need gpu dpa (device physical -* address), and there is no need of dma-mapping. This is TBD. -* * FIXME: dma-mapping for peer gpu device to access remote gpu's * memory. Add this when you support p2p *
@@ -77,12 +148,13 @@ static void xe_mark_range_accessed(struct hmm_range *range, bool write)* * Returns 0 if successful; -ENOMEM if fails to allocate memory */ -static int xe_build_sg (struct xe_device *xe,struct hmm_range *range, -struct sg_table *st,bool write) +static int xe_build_sg (struct xe_device *xe,struct xe_userptr *userptr, +struct hmm_range *range,bool write) { +struct sg_table *st = &userptr->sgt;struct device *dev = xe->drm.dev;struct page **pages; -u64 i, npages; +u64 npages;int ret; npages =xe_npages_in_range (range->start, range->end);
@@ -90,19 +162,22 @@ static int xe_build_sg(struct xe_device *xe, struct hmm_range *range,if (!pages)return -ENOMEM ; -for (i =0 ; i < npages; i++) { - pages[i] =hmm_pfn_to_page (range->hmm_pfns[i]); -xe_assert (xe, !is_device_private_page (pages[i])); - } - - ret =sg_alloc_table_from_pages_segment (st, pages, npages,0 , - npages <<PAGE_SHIFT ,xe_sg_segment_size (dev),GFP_KERNEL ); -if (ret) - goto free_pages; +xe_validate_hmm_pfns (userptr, range->hmm_pfns, npages, pages); +if (!userptr->is_device_pages) { + ret =sg_alloc_table_from_pages_segment (st, pages, npages,0 , + npages <<PAGE_SHIFT ,xe_sg_segment_size (dev),GFP_KERNEL ); +if (ret) + goto free_pages; - ret =dma_map_sgtable (dev, st, write ?DMA_BIDIRECTIONAL :DMA_TO_DEVICE , -DMA_ATTR_SKIP_CPU_SYNC |DMA_ATTR_NO_KERNEL_MAPPING ); + ret =dma_map_sgtable (dev, st, write ?DMA_BIDIRECTIONAL :DMA_TO_DEVICE , +DMA_ATTR_SKIP_CPU_SYNC |DMA_ATTR_NO_KERNEL_MAPPING ); + }else { + ret =xe_build_sg_device_pages (st, range->hmm_pfns, pages, npages); +if (ret) + goto free_pages; + } + userptr->sg = st; free_pages:kvfree (pages);return ret;
@@ -127,7 +202,8 @@ void xe_userptr_free_sg(struct xe_userptr_vma *uvma)struct device *dev = xe->drm.dev;xe_assert (xe, userptr->sg); -dma_unmap_sgtable (dev, userptr->sg, +if (!userptr->is_device_pages) +dma_unmap_sgtable (dev, userptr->sg, write ?DMA_BIDIRECTIONAL :DMA_TO_DEVICE ,0 );sg_free_table (userptr->sg);
@@ -239,12 +315,11 @@ int xe_userptr_populate_range(struct xe_userptr_vma *uvma)if (ret) goto free_pfns; - ret =xe_build_sg (vm->xe, &hmm_range, &userptr->sgt, write); + ret =xe_build_sg (vm->xe, userptr, &hmm_range, write);if (ret) goto free_pfns;xe_mark_range_accessed (&hmm_range, write); - userptr->sg = &userptr->sgt; userptr->notifier_seq = hmm_range.notifier_seq; free_pfns:
drivers/gpu/drm/xe/xe_vm_types.h +2 -0
@@ -64,6 +64,8 @@ struct xe_userptr {struct sg_table *sg;/** @notifier_seq: notifier sequence number */ unsigned long notifier_seq; +/** @is_device_pages: the backing store is in device memory*/ +bool is_device_pages;/** * @initial_bind: user pointer has been bound at least once. * write: vm->userptr.notifier_lock in read mode and vm->resv held.
Signature · 1 line
2.26.3
Matthew Brost
maintainerRe: [v2 18/31] drm/xe/svm: Build userptr sg table for device pages
Quoted 4 lines · Oak Zeng
Previously function
xe_build_sgonly support userptr with system memory pages. Now this function is extended to support userptr with device pages backing as well.
Quoted 8 lines
For device pages, there is no need of dma-mapping. Instead, we calculated the device page's dpa (device physical address) and use dpa to fill sg table.
As of now, we assume each userptr is only backed either by all system memory pages or all by device pages. There is no support of mixture backing of device and system memory pages.
I'm not sure if this required as per Jason's suggestion or rather insistence to not use a sg list for a collection of dpas [1].
For single device we should just be able to the buddy blocks as the cursor which I suggest in [1]. Maybe this doesn't work in a multi-device case but it certainly should work for a single device. Since we are working a single device first, let get it working without abusing a SG list.
Matt
[1] https://patchwork.freedesktop.org/patch/574894/?series=128910&rev=1
Quoted 213 lines
Signed-off-by Oak Zeng <[email protected]>--- drivers/gpu/drm/xe/xe_hmm.c | 121 +++++++++++++++++++++++++------ drivers/gpu/drm/xe/xe_vm_types.h | 2 + 2 files changed, 100 insertions(+), 23 deletions(-)drivers/gpu/drm/xe/xe_hmm.c +98
-23@@ -11,6 +11,7 @@#include <linux/hmm.h> #include <linux/mm.h> #include "xe_hmm.h" +#include "xe_svm.h" #include "xe_vm.h" #include "xe_bo.h" @@ -43,15 +44,90 @@ static void xe_mark_range_accessed(struct hmm_range *range, bool write) } } +/** +* xe_build_sg_device_pages() - build sg table for userptr when the backing store +* is device pages +* +* @st: sg table to build +* @hmm_pfns: pfn array of the userptr +* @pages: struct page arrary of this userptr +* @npages: how many pages in this userptr +*/ +static int xe_build_sg_device_pages (struct sg_table *st,unsigned long *hmm_pfns, +struct page **pages,uint64_t npages) +{ +struct scatterlist *sg; +int i; + + sg =NULL ; + st->nents =0 ; +if (unlikely (sg_alloc_table (st, npages,GFP_KERNEL ))) +return -ENOMEM ; + +for (i =0 ; i < npages; i++) { +unsigned long addr; +struct xe_mem_region *mr; + + mr =xe_page_to_mem_region (pages[i]); + addr =xe_mem_region_pfn_to_dpa (mr, hmm_pfns[i]); +if (sg && (addr == (sg_dma_address (sg) + sg->length))) { + sg->length +=PAGE_SIZE ; +sg_dma_len (sg) +=PAGE_SIZE ; +continue ; + } + + sg = sg ?sg_next (sg) : st->sgl; +sg_dma_address (sg) = addr; +sg_dma_len (sg) =PAGE_SIZE ; + sg->length =PAGE_SIZE ; + st->nents++; + } + +sg_mark_end (sg); +return 0 ; +} + +/** +* xe_validate_hmm_pfns() - validate all pages in a userptr belong to one memory +* region, and populate the pages array. +* +* @userptr: The userptr to validate +* @hmm_pfns: an array holding hmm pfns +* @npages: number of pages of this userptr +* @pages: output parameter to hold the populated pages from pfn. +*/ +static void xe_validate_hmm_pfns (struct xe_userptr *userptr,unsigned long *hmm_pfns, +uint64_t npages,struct page **pages) +{ +int i; +struct xe_vma *vma =xe_userptr_to_vma (userptr); +struct xe_vm *vm =xe_vma_vm (vma); + + pages[0 ] =hmm_pfn_to_page (hmm_pfns[0 ]); + userptr->is_device_pages =is_device_private_page (pages[0 ]); +for (i =1 ; i < npages; i++) { + pages[i] =hmm_pfn_to_page (hmm_pfns[i]); +/** +* We currently assume no mixture of device pages and system memory +* pages in one userptr. If it turns out this is not true, we will +* either split userptr into device pages based and system memory +* based, or support a mixture backing store in one userptr. +*/ +xe_assert (vm->xe, + userptr->is_device_pages ==is_device_private_page (pages[i])); + } +} + +/** * xe_build_sg() - build a scatter gather table for all the physical pages/pfn * in a hmm_range. dma-map pages if necessary. dma-address is save in sg table * and will be used to program GPU page table later. * * @xe: the xe device who will access the dma-address in sg table +* @userptr: the userptr that we build the sg table for * @range: the hmm range that we build the sg table from. range->hmm_pfns[] * has the pfn numbers of pages that back up this hmm address range. -* @st: pointer to the sg table. * @write: whether we write to this range. This decides dma map direction * for system pages. If write we map it bi-diretional; otherwise * DMA_TO_DEVICE @@ -64,11 +140,6 @@ static void xe_mark_range_accessed(struct hmm_range *range, bool write)* access memory. So if the memory is system memory, we need to * do a dma-mapping so it can be accessed by GPU/DMA. * -* FIXME: This function currently only support pages in system -* memory. If the memory is GPU local memory (of the GPU who -* is going to access memory), we need gpu dpa (device physical -* address), and there is no need of dma-mapping. This is TBD. -* * FIXME: dma-mapping for peer gpu device to access remote gpu's * memory. Add this when you support p2p * @@ -77,12 +148,13 @@ static void xe_mark_range_accessed(struct hmm_range *range, bool write)* * Returns 0 if successful; -ENOMEM if fails to allocate memory */ -static int xe_build_sg (struct xe_device *xe,struct hmm_range *range, -struct sg_table *st,bool write) +static int xe_build_sg (struct xe_device *xe,struct xe_userptr *userptr, +struct hmm_range *range,bool write) { +struct sg_table *st = &userptr->sgt;struct device *dev = xe->drm.dev;struct page **pages; -u64 i, npages; +u64 npages;int ret; npages =xe_npages_in_range (range->start, range->end);@@ -90,19 +162,22 @@ static int xe_build_sg(struct xe_device *xe, struct hmm_range *range,if (!pages)return -ENOMEM ; -for (i =0 ; i < npages; i++) { - pages[i] =hmm_pfn_to_page (range->hmm_pfns[i]); -xe_assert (xe, !is_device_private_page (pages[i])); - } - - ret =sg_alloc_table_from_pages_segment (st, pages, npages,0 , - npages <<PAGE_SHIFT ,xe_sg_segment_size (dev),GFP_KERNEL ); -if (ret) - goto free_pages; +xe_validate_hmm_pfns (userptr, range->hmm_pfns, npages, pages); +if (!userptr->is_device_pages) { + ret =sg_alloc_table_from_pages_segment (st, pages, npages,0 , + npages <<PAGE_SHIFT ,xe_sg_segment_size (dev),GFP_KERNEL ); +if (ret) + goto free_pages; - ret =dma_map_sgtable (dev, st, write ?DMA_BIDIRECTIONAL :DMA_TO_DEVICE , -DMA_ATTR_SKIP_CPU_SYNC |DMA_ATTR_NO_KERNEL_MAPPING ); + ret =dma_map_sgtable (dev, st, write ?DMA_BIDIRECTIONAL :DMA_TO_DEVICE , +DMA_ATTR_SKIP_CPU_SYNC |DMA_ATTR_NO_KERNEL_MAPPING ); + }else { + ret =xe_build_sg_device_pages (st, range->hmm_pfns, pages, npages); +if (ret) + goto free_pages; + } + userptr->sg = st; free_pages:kvfree (pages);return ret;@@ -127,7 +202,8 @@ void xe_userptr_free_sg(struct xe_userptr_vma *uvma)struct device *dev = xe->drm.dev;xe_assert (xe, userptr->sg); -dma_unmap_sgtable (dev, userptr->sg, +if (!userptr->is_device_pages) +dma_unmap_sgtable (dev, userptr->sg, write ?DMA_BIDIRECTIONAL :DMA_TO_DEVICE ,0 );sg_free_table (userptr->sg);@@ -239,12 +315,11 @@ int xe_userptr_populate_range(struct xe_userptr_vma *uvma)if (ret) goto free_pfns; - ret =xe_build_sg (vm->xe, &hmm_range, &userptr->sgt, write); + ret =xe_build_sg (vm->xe, userptr, &hmm_range, write);if (ret) goto free_pfns;xe_mark_range_accessed (&hmm_range, write); - userptr->sg = &userptr->sgt; userptr->notifier_seq = hmm_range.notifier_seq; free_pfns:drivers/gpu/drm/xe/xe_vm_types.h +2
-0@@ -64,6 +64,8 @@ struct xe_userptr {struct sg_table *sg;/** @notifier_seq: notifier sequence number */ unsigned long notifier_seq; +/** @is_device_pages: the backing store is in device memory*/ +bool is_device_pages;/** * @initial_bind: user pointer has been bound at least once. * write: vm->userptr.notifier_lock in read mode and vm->resv held. Signature · 1 line
2.26.3
PATCH v2 19/31 drm/xe/svm: Determine a vma is backed by device memory
In reply to Oak ZengWith system allocator, a userptr can now be back by device memory also. Introduce a helper function xe_vma_is_devmem to determine whether a vma is backed by device memory.
--- drivers/gpu/drm/xe/xe_pt.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_pt.c +12 -2
@@ -577,6 +577,17 @@ static const struct xe_pt_walk_ops xe_pt_stage_bind_ops = { .pt_entry = xe_pt_stage_bind_entry, }; +static bool xe_vma_is_devmem (struct xe_vma *vma) +{ +if (xe_vma_is_userptr (vma)) { +struct xe_userptr_vma *uvma =to_userptr_vma (vma); +return uvma->userptr.is_device_pages; + }else { +struct xe_bo *bo =xe_vma_bo (vma); +return bo && (xe_bo_is_vram (bo) ||xe_bo_is_stolen_devmem (bo)); + } +} +/** * xe_pt_stage_bind() - Build a disconnected page-table tree for a given address * range.
@@ -601,8 +612,7 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, {struct xe_device *xe =tile_to_xe (tile);struct xe_bo *bo =xe_vma_bo (vma); -bool is_devmem = !xe_vma_is_userptr (vma) && bo && - (xe_bo_is_vram (bo) ||xe_bo_is_stolen_devmem (bo)); +bool is_devmem =xe_vma_is_devmem (vma);struct xe_res_cursor curs;struct xe_pt_stage_bind_walk xe_walk = { .base = {
Signature · 1 line
2.26.3
Matthew Brost
maintainerRe: [v2 19/31] drm/xe/svm: Determine a vma is backed by device memory
Quoted 14 lines · Oak Zeng
With system allocator, a userptr can now be back by device memory also. Introduce a helper function
xe_vma_is_devmemto determine whether a vma is backed by device memory.Signed-off-by Oak Zeng <[email protected]>--- drivers/gpu/drm/xe/xe_pt.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)drivers/gpu/drm/xe/xe_pt.c +0
-0
drivers/gpu/drm/xe/xe_pt.c static const struct xe_pt_walk_ops xe_pt_stage_bind_ops = { · line 580
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +1
-0@@ -577,6 +577,17 @@ static const struct xe_pt_walk_ops xe_pt_stage_bind_ops = { .pt_entry = xe_pt_stage_bind_entry, }; +static bool xe_vma_is_devmem (struct xe_vma *vma)
At some point we probably want to scrub the driver as we intermix devmem, vram, and lmem nomenclature. I think in case we mean the same thing too. Anwyays that is a little out of scope here.
drivers/gpu/drm/xe/xe_pt.c static const struct xe_pt_walk_ops xe_pt_stage_bind_ops = { · line 584
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +4
-0+{ +if (xe_vma_is_userptr (vma)) { +struct xe_userptr_vma *uvma =to_userptr_vma (vma); +return uvma->userptr.is_device_pages;
Helper itself LGTM. Maybe promote to xe_vm.c/xe_vm.h?
Also consider other options rather than userptr.is_device_pages flag here (e.g. look for buddy blocks, check a gpuvm flags, etc...). Can live with a flag but we can do without it, great.
Matt
Quoted 22 lines
drivers/gpu/drm/xe/xe_pt.c +7
-2+ }else { +struct xe_bo *bo =xe_vma_bo (vma); +return bo && (xe_bo_is_vram (bo) ||xe_bo_is_stolen_devmem (bo)); + } +} +/** * xe_pt_stage_bind() - Build a disconnected page-table tree for a given address * range. @@ -601,8 +612,7 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, {struct xe_device *xe =tile_to_xe (tile);struct xe_bo *bo =xe_vma_bo (vma); -bool is_devmem = !xe_vma_is_userptr (vma) && bo && - (xe_bo_is_vram (bo) ||xe_bo_is_stolen_devmem (bo)); +bool is_devmem =xe_vma_is_devmem (vma);struct xe_res_cursor curs;struct xe_pt_stage_bind_walk xe_walk = { .base = {Signature · 1 line
2.26.3
RE: [v2 19/31] drm/xe/svm: Determine a vma is backed by device memory
Quoted 29 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 5:57 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 19/31] drm/xe/svm: Determine a vma is backed by device memorydrivers/gpu/drm/xe/xe_pt.c static const struct xe_pt_walk_ops
Quoted 14 lines · Oak Zeng
With system allocator, a userptr can now be back by device memory also. Introduce a helper function
xe_vma_is_devmemto determine whether a vma is backed by device memory.Signed-off-by Oak Zeng <[email protected]>--- drivers/gpu/drm/xe/xe_pt.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -577,6 +577,17 @@ static const struct xe_pt_walk_ops
xe_pt_stage_bind_ops= {
Quoted 8 lines
drivers/gpu/drm/xe/xe_pt.c static const struct xe_pt_walk_ops xe_pt_stage_bind_ops = { · line 580
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +1
-0.pt_entry = xe_pt_stage_bind_entry, }; +static bool xe_vma_is_devmem (struct xe_vma *vma)At some point we probably want to scrub the driver as we intermix devmem, vram, and lmem nomenclature. I think in case we mean the same thing too. Anwyays that is a little out of scope here.
Agree we are mix using those terms.
Devmem is still good to me in this case. I will keep it for now.
Quoted 11 lines
drivers/gpu/drm/xe/xe_pt.c static const struct xe_pt_walk_ops xe_pt_stage_bind_ops = { · line 584
Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c +4
-0+{ +if (xe_vma_is_userptr (vma)) { +struct xe_userptr_vma *uvma =to_userptr_vma (vma); +return uvma->userptr.is_device_pages;Helper itself LGTM. Maybe promote to
xe_vm.c/xe_vm.h?Also consider other options rather than userptr.is_device_pages flag here (e.g. look for buddy blocks, check a gpuvm flags, etc...). Can live with a flag but we can do without it, great.
For now this helper is only used in this file. I will keep it local static in xe_pt.c for now. We can consider a move if it is used in other file in the future.
In v3, I introduced range based page table update. Instead of update page table for the whole vma, we can now update only a sub-range of a vma.
For system allocator, the backing store of a vma can now be a mixture placement of system memory and device memory. I will update this helper to reflect such change.
Oak
Quoted 28 lines
Matt
drivers/gpu/drm/xe/xe_pt.c static const struct xe_pt_walk_ops xe_pt_stage_bind_ops = {
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c +3
-0+ }else { +struct xe_bo *bo =xe_vma_bo (vma); + return bo && (xe_bo_is_vram(bo) ||
xe_bo_is_stolen_devmem(bo));drivers/gpu/drm/xe/xe_pt.c static const struct xe_pt_walk_ops xe_pt_stage_bind_ops = {
Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c +3
-0+ } +} +/** * xe_pt_stage_bind() - Build a disconnected page-table tree for a given address
drivers/gpu/drm/xe/xe_pt.c xe_pt_stage_bind(struct xe_tile *tile, struct
Quoted 2 lines
drivers/gpu/drm/xe/xe_pt.c +0
-0* range. @@ -601,8 +612,7 @@ xe_pt_stage_bind(struct xe_tile *tile, struct
xe_vma*vma,Quoted 12 lines
drivers/gpu/drm/xe/xe_pt.c +1
-3{struct xe_device *xe =tile_to_xe (tile);struct xe_bo *bo =xe_vma_bo (vma); -bool is_devmem = !xe_vma_is_userptr (vma) && bo && - (xe_bo_is_vram (bo) ||xe_bo_is_stolen_devmem (bo)); +bool is_devmem =xe_vma_is_devmem (vma);struct xe_res_cursor curs;struct xe_pt_stage_bind_walk xe_walk = { .base = { --2.26.3
PATCH v2 20/31 drm/xe: add xe lock document
In reply to Oak ZengThis is not intended a complete documentation of xe locks. It only documents some key locks used in xe driver and gives an example to illustrate the lock usage.
This is just a start. We should eventually refine this document.
--- Documentation/gpu/xe/index.rst | 1 + Documentation/gpu/xe/xe_lock.rst | 8 +++ drivers/gpu/drm/xe/xe_lock_doc.h | 113 +++++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_vm_types.h | 2 +- 4 files changed, 123 insertions(+), 1 deletion(-) create mode 100644 Documentation/gpu/xe/xe_lock.rst create mode 100644 drivers/gpu/drm/xe/xe_lock_doc.h
Documentation/gpu/xe/index.rst +1 -0
@@ -24,3 +24,4 @@ DG2, etc is provided to prototype the driver. xe_tile xe_debugging xe_svm + xe_lock
Documentation/gpu/xe/xe_lock.rst new file +8 -0
@@ -0,0 +1,8 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +============== +xe lock design +============== + +.. kernel-doc:: drivers/gpu/drm/xe/xe_lock_doc.h + :doc: xe lock design
drivers/gpu/drm/xe/xe_lock_doc.h new file +113 -0
113 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm_types.h +1 -1
@@ -271,7 +271,7 @@ struct xe_vm {/** * @lock: outer most lock, protects objects of anything attached to this -* VM +* VM. See more details in xe_lock_doc.h */ struct rw_semaphore lock;/**
Signature · 1 line
2.26.3
PATCH v2 21/31 drm/xe/svm: Introduce svm migration function
In reply to Oak ZengIntroduce xe_migrate_pa function for data migration. This function is similar to xe_migrate_copy function but has different parameters. Instead of BO and ttm resource parameters, it has source and destination buffer's physical address as parameter. This function is intended to be used by svm sub-system which doesn't have BO and TTM concept.
--- drivers/gpu/drm/xe/xe_migrate.c | 217 ++++++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_migrate.h | 7 ++ 2 files changed, 224 insertions(+)
drivers/gpu/drm/xe/xe_migrate.c +217 -0
@@ -462,6 +462,37 @@ static bool xe_migrate_allow_identity(u64 size, const struct xe_res_cursor *cur)return cur->size >= size; } +/** +* pte_update_cmd_size() - calculate the batch buffer command size +* to update a flat page table. +* +* @size: The virtual address range size of the page table to update +* +* The page table to update is supposed to be a flat 1 level page +* table with all entries pointing to 4k pages. +* +* Return the number of dwords of the update command +*/ +static u32 pte_update_cmd_size (u64 size) +{ +u32 dword; +u64 entries =DIV_ROUND_UP (size,XE_PAGE_SIZE ); + +XE_WARN_ON (size >MAX_PREEMPTDISABLE_TRANSFER ); +/* +* MI_STORE_DATA_IMM command is used to update page table. Each +* instruction can update maximumly 0x1ff pte entries. To update +* n (n <= 0x1ff) pte entries, we need: +* 1 dword for the MI_STORE_DATA_IMM command header (opcode etc) +* 2 dword for the page table's physical location +* 2*n dword for value of pte to fill (each pte entry is 2 dwords) +*/ + dword = (1 +2 ) *DIV_ROUND_UP (entries,0x1ff ); + dword += entries *2 ; + +return dword; +} + static u32 pte_update_size(struct xe_migrate *m, bool is_vram, struct ttm_resource *res,
@@ -562,6 +593,48 @@ static void emit_pte(struct xe_migrate *m, } } +/** +* build_pt_update_batch_sram() - build batch buffer commands to update +* migration vm page table for system memory +* +* @m: The migration context +* @bb: The batch buffer which hold the page table update commands +* @pt_offset: The offset of page table to update, in byte +* @pa: device physical address you want the page table to point to +* @size: size of the virtual address space you want the page table to cover +*/ +static void build_pt_update_batch_sram (struct xe_migrate *m, +struct xe_bb *bb,u32 pt_offset, +u64 pa,u32 size) +{ +u16 pat_index =tile_to_xe (m->tile)->pat.idx[XE_CACHE_WB ]; +u32 ptes; + + ptes =DIV_ROUND_UP (size,XE_PAGE_SIZE ); +while (ptes) { +u32 chunk =min (0x1ffU , ptes); + + bb->cs[bb->len++] =MI_STORE_DATA_IMM |MI_SDI_NUM_QW (chunk); + bb->cs[bb->len++] = pt_offset; + bb->cs[bb->len++] =0 ; + + pt_offset += chunk *8 ; + ptes -= chunk; + +while (chunk--) { +u64 addr; + + addr = pa &PAGE_MASK ; + addr = m->q->vm->pt_ops->pte_encode_addr (m->tile->xe, + addr, pat_index, +0 , false,0 ); + bb->cs[bb->len++] =lower_32_bits (addr); + bb->cs[bb->len++] =upper_32_bits (addr); + pa +=XE_PAGE_SIZE ; + } + } +} +#define EMIT_COPY_CCS_DW 5 static void emit_copy_ccs(struct xe_gt *gt, struct xe_bb *bb, u64 dst_ofs, bool dst_is_indirect,
@@ -879,6 +952,150 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,return fence; } +/** +* xe_migrate_pa() - Migrate buffers with src and dst physical address +* +* @m: The migration context +* @src_pa: physical address of source, from GPU's point of view. This is a +* device physical address (dpa) when source is in vram. When source is in +* system memory, this is a dma mapped host physical address +* @src_is_vram: True if source buffer is in vram. +* @dst_pa: physical address of destination, from GPU's point of view. This is a +* device physical address (dpa) when source is in vram. When source is in +* system memory, this is a dma mapped host physical address +* @dst_is_vram: True if destination buffer is in vram. +* @size: The size of data to copy. +* +* Copy @size bytes of data from @src_pa to @dst_pa. The functionality +* and behavior of this function is similar to xe_migrate_copy function, but +* the interface is different. This function is a helper function supposed to +* be used by SVM subsytem. Since in SVM subsystem there is no buffer object +* and ttm, there is no src/dst bo as function input. Instead, we directly use +* src/dst's physical address as function input. +* +* Since the back store of any user malloc'ed or mmap'ed memory can be placed in +* system memory, it can not be compressed. Thus this function doesn't need +* to consider copy CCS (compression control surface) data as xe_migrate_copy did. +* +* This function assumes the source buffer and destination buffer are all physically +* contiguous. +* +* We use gpu blitter to copy data. Source and destination are first mapped to +* migration vm which is a flat one level (L0) page table, then blitter is used to +* perform the copy. +* +* Return: Pointer to a dma_fence representing the last copy batch, or +* an error pointer on failure. If there is a failure, any copy operation +* started by the function call has been synced. +*/ +struct dma_fence *xe_migrate_pa (struct xe_migrate *m, +u64 src_pa, +bool src_is_vram, +u64 dst_pa, +bool dst_is_vram, +u64 size) +{ +#define NUM_PT_PER_BLIT (MAX_PREEMPTDISABLE_TRANSFER / SZ_2M) +struct xe_gt *gt = m->tile->primary_gt; +struct xe_device *xe =gt_to_xe (gt); +struct dma_fence *fence =NULL ; +u64 src_L0_ofs, dst_L0_ofs; +u64 round_update_size; +/* A slot is a 4K page of page table, covers 2M virtual address*/ +u32 pt_slot; +int err; + +while (size) { +u32 batch_size =2 ;/* arb_clear() + MI_BATCH_BUFFER_END */ +struct xe_sched_job *job; +struct xe_bb *bb; +u32 update_idx; + +/* Maximumly copy MAX_PREEMPTDISABLE_TRANSFER bytes. Why?*/ + round_update_size =min_t (u64, size,MAX_PREEMPTDISABLE_TRANSFER ); + +/* src pte update*/ +if (!src_is_vram) + batch_size +=pte_update_cmd_size (round_update_size); +/* dst pte update*/ +if (!dst_is_vram) + batch_size +=pte_update_cmd_size (round_update_size); + +/* Copy command size*/ + batch_size +=EMIT_COPY_DW ; + + bb =xe_bb_new (gt, batch_size, true); +if (IS_ERR (bb)) { + err =PTR_ERR (bb); + goto err_sync; + } + +if (!src_is_vram) { + pt_slot =0 ; +build_pt_update_batch_sram (m, bb, pt_slot *XE_PAGE_SIZE , + src_pa, round_update_size); + src_L0_ofs =xe_migrate_vm_addr (pt_slot,0 ); + } +else + src_L0_ofs =xe_migrate_vram_ofs (xe, src_pa); + +if (!dst_is_vram) { + pt_slot =NUM_PT_PER_BLIT ; +build_pt_update_batch_sram (m, bb, pt_slot *XE_PAGE_SIZE , + dst_pa, round_update_size); + dst_L0_ofs =xe_migrate_vm_addr (pt_slot,0 ); + } +else + dst_L0_ofs =xe_migrate_vram_ofs (xe, dst_pa); + + + bb->cs[bb->len++] =MI_BATCH_BUFFER_END ; + update_idx = bb->len; + +emit_copy (gt, bb, src_L0_ofs, dst_L0_ofs, round_update_size, +XE_PAGE_SIZE ); + +mutex_lock (&m->job_mutex); + job =xe_bb_create_migration_job (m->q, bb, +xe_migrate_batch_base (m, true), + update_idx); +if (IS_ERR (job)) { + err =PTR_ERR (job); + goto err; + } + +xe_sched_job_add_migrate_flush (job,0 ); +xe_sched_job_arm (job); +dma_fence_put (fence); + fence =dma_fence_get (&job->drm.s_fence->finished); +xe_sched_job_push (job); +dma_fence_put (m->fence); + m->fence =dma_fence_get (fence); + +mutex_unlock (&m->job_mutex); + +xe_bb_free (bb, fence); + size -= round_update_size; + src_pa += round_update_size; + dst_pa += round_update_size; +continue ; + +err: +mutex_unlock (&m->job_mutex); +xe_bb_free (bb,NULL ); + +err_sync: +/* Sync partial copy if any. FIXME: under job_mutex? */ +if (fence) { +dma_fence_wait (fence, false); +dma_fence_put (fence); + } + +return ERR_PTR (err); + } + +return fence; +}static void emit_clear_link_copy (struct xe_gt *gt,struct xe_bb *bb,u64 src_ofs,u32 size,u32 pitch) {
drivers/gpu/drm/xe/xe_migrate.h +7 -0
@@ -101,6 +101,13 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,struct ttm_resource *dst, bool copy_only_ccs); +struct dma_fence *xe_migrate_pa (struct xe_migrate *m, +u64 src_pa, +bool src_is_vram, +u64 dst_pa, +bool dst_is_vram, +u64 size); +struct dma_fence *xe_migrate_clear (struct xe_migrate *m,struct xe_bo *bo,struct ttm_resource *dst);
Signature · 1 line
2.26.3
Matthew Brost
maintainerRe: [v2 21/31] drm/xe/svm: Introduce svm migration function
Quoted 145 lines · Oak Zeng
Introduce
xe_migrate_pafunction for data migration. This function is similar toxe_migrate_copyfunction but has different parameters. Instead of BO and ttm resource parameters, it has source and destination buffer's physical address as parameter. This function is intended to be used by svm sub-system which doesn't have BO and TTM concept.Signed-off-by Oak Zeng <[email protected]>Cc Niranjana Vishwanathapura <[email protected]>Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/xe_migrate.c | 217 ++++++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_migrate.h | 7 ++ 2 files changed, 224 insertions(+)drivers/gpu/drm/xe/xe_migrate.c +105
-0@@ -462,6 +462,37 @@ static bool xe_migrate_allow_identity(u64 size, const struct xe_res_cursor *cur)return cur->size >= size; } +/** +* pte_update_cmd_size() - calculate the batch buffer command size +* to update a flat page table. +* +* @size: The virtual address range size of the page table to update +* +* The page table to update is supposed to be a flat 1 level page +* table with all entries pointing to 4k pages. +* +* Return the number of dwords of the update command +*/ +static u32 pte_update_cmd_size (u64 size) +{ +u32 dword; +u64 entries =DIV_ROUND_UP (size,XE_PAGE_SIZE ); + +XE_WARN_ON (size >MAX_PREEMPTDISABLE_TRANSFER ); +/* +* MI_STORE_DATA_IMM command is used to update page table. Each +* instruction can update maximumly 0x1ff pte entries. To update +* n (n <= 0x1ff) pte entries, we need: +* 1 dword for the MI_STORE_DATA_IMM command header (opcode etc) +* 2 dword for the page table's physical location +* 2*n dword for value of pte to fill (each pte entry is 2 dwords) +*/ + dword = (1 +2 ) *DIV_ROUND_UP (entries,0x1ff ); + dword += entries *2 ; + +return dword; +} + static u32 pte_update_size(struct xe_migrate *m, bool is_vram, struct ttm_resource *res,@@ -562,6 +593,48 @@ static void emit_pte(struct xe_migrate *m, } } +/** +* build_pt_update_batch_sram() - build batch buffer commands to update +* migration vm page table for system memory +* +* @m: The migration context +* @bb: The batch buffer which hold the page table update commands +* @pt_offset: The offset of page table to update, in byte +* @pa: device physical address you want the page table to point to +* @size: size of the virtual address space you want the page table to cover +*/ +static void build_pt_update_batch_sram (struct xe_migrate *m, +struct xe_bb *bb,u32 pt_offset, +u64 pa,u32 size) +{ +u16 pat_index =tile_to_xe (m->tile)->pat.idx[XE_CACHE_WB ]; +u32 ptes; + + ptes =DIV_ROUND_UP (size,XE_PAGE_SIZE ); +while (ptes) { +u32 chunk =min (0x1ffU , ptes); + + bb->cs[bb->len++] =MI_STORE_DATA_IMM |MI_SDI_NUM_QW (chunk); + bb->cs[bb->len++] = pt_offset; + bb->cs[bb->len++] =0 ; + + pt_offset += chunk *8 ; + ptes -= chunk; + +while (chunk--) { +u64 addr; + + addr = pa &PAGE_MASK ; + addr = m->q->vm->pt_ops->pte_encode_addr (m->tile->xe, + addr, pat_index, +0 , false,0 ); + bb->cs[bb->len++] =lower_32_bits (addr); + bb->cs[bb->len++] =upper_32_bits (addr); + pa +=XE_PAGE_SIZE ; + } + } +} +#define EMIT_COPY_CCS_DW 5 static void emit_copy_ccs(struct xe_gt *gt, struct xe_bb *bb, u64 dst_ofs, bool dst_is_indirect,@@ -879,6 +952,150 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,return fence; } +/** +* xe_migrate_pa() - Migrate buffers with src and dst physical address +* +* @m: The migration context +* @src_pa: physical address of source, from GPU's point of view. This is a +* device physical address (dpa) when source is in vram. When source is in +* system memory, this is a dma mapped host physical address +* @src_is_vram: True if source buffer is in vram. +* @dst_pa: physical address of destination, from GPU's point of view. This is a +* device physical address (dpa) when source is in vram. When source is in +* system memory, this is a dma mapped host physical address +* @dst_is_vram: True if destination buffer is in vram. +* @size: The size of data to copy. +* +* Copy @size bytes of data from @src_pa to @dst_pa. The functionality +* and behavior of this function is similar to xe_migrate_copy function, but +* the interface is different. This function is a helper function supposed to +* be used by SVM subsytem. Since in SVM subsystem there is no buffer object +* and ttm, there is no src/dst bo as function input. Instead, we directly use +* src/dst's physical address as function input. +* +* Since the back store of any user malloc'ed or mmap'ed memory can be placed in +* system memory, it can not be compressed. Thus this function doesn't need +* to consider copy CCS (compression control surface) data as xe_migrate_copy did. +* +* This function assumes the source buffer and destination buffer are all physically +* contiguous. +* +* We use gpu blitter to copy data. Source and destination are first mapped to +* migration vm which is a flat one level (L0) page table, then blitter is used to +* perform the copy. +*
drivers/gpu/drm/xe/xe_migrate.c struct dma_fence *xe_migrate_copy(struct xe_migrate *m, · line 996
Quoted 10 lines
drivers/gpu/drm/xe/xe_migrate.c +10
-0+* Return: Pointer to a dma_fence representing the last copy batch, or +* an error pointer on failure. If there is a failure, any copy operation +* started by the function call has been synced. +*/ +struct dma_fence *xe_migrate_pa (struct xe_migrate *m, +u64 src_pa, +bool src_is_vram, +u64 dst_pa, +bool dst_is_vram, +u64 size)
This assume both addresses are contiguous if size > 4k.
I don't think needs to be the case when one of the addresses a sram (dma_addr) as we dynamically map in sram pages into PT entries. i.e only VRAM addresses need to be contiguous.
I'd suggest this function take from array of dma_addr and 1 vram address to maximize copy efficiency. Also add a direction variable too (i.e. is vram the source or destination).
drivers/gpu/drm/xe/xe_migrate.c struct dma_fence *xe_migrate_copy(struct xe_migrate *m, · line 1008
Quoted 12 lines
drivers/gpu/drm/xe/xe_migrate.c +12
-0+{ +#define NUM_PT_PER_BLIT (MAX_PREEMPTDISABLE_TRANSFER / SZ_2M) +struct xe_gt *gt = m->tile->primary_gt; +struct xe_device *xe =gt_to_xe (gt); +struct dma_fence *fence =NULL ; +u64 src_L0_ofs, dst_L0_ofs; +u64 round_update_size; +/* A slot is a 4K page of page table, covers 2M virtual address*/ +u32 pt_slot; +int err; + +while (size) {
We might not need this loop either if we make the caller enforce the chunking (i.e. cap size as 2 MB or whatever MAX_PREEMPTDISABLE_TRANSFER is).
Quoted 98 lines
drivers/gpu/drm/xe/xe_migrate.c +90
-0+u32 batch_size =2 ;/* arb_clear() + MI_BATCH_BUFFER_END */ +struct xe_sched_job *job; +struct xe_bb *bb; +u32 update_idx; + +/* Maximumly copy MAX_PREEMPTDISABLE_TRANSFER bytes. Why?*/ + round_update_size =min_t (u64, size,MAX_PREEMPTDISABLE_TRANSFER ); + +/* src pte update*/ +if (!src_is_vram) + batch_size +=pte_update_cmd_size (round_update_size); +/* dst pte update*/ +if (!dst_is_vram) + batch_size +=pte_update_cmd_size (round_update_size); + +/* Copy command size*/ + batch_size +=EMIT_COPY_DW ; + + bb =xe_bb_new (gt, batch_size, true); +if (IS_ERR (bb)) { + err =PTR_ERR (bb); + goto err_sync; + } + +if (!src_is_vram) { + pt_slot =0 ; +build_pt_update_batch_sram (m, bb, pt_slot *XE_PAGE_SIZE , + src_pa, round_update_size); + src_L0_ofs =xe_migrate_vm_addr (pt_slot,0 ); + } +else + src_L0_ofs =xe_migrate_vram_ofs (xe, src_pa); + +if (!dst_is_vram) { + pt_slot =NUM_PT_PER_BLIT ; +build_pt_update_batch_sram (m, bb, pt_slot *XE_PAGE_SIZE , + dst_pa, round_update_size); + dst_L0_ofs =xe_migrate_vm_addr (pt_slot,0 ); + } +else + dst_L0_ofs =xe_migrate_vram_ofs (xe, dst_pa); + + + bb->cs[bb->len++] =MI_BATCH_BUFFER_END ; + update_idx = bb->len; + +emit_copy (gt, bb, src_L0_ofs, dst_L0_ofs, round_update_size, +XE_PAGE_SIZE ); + +mutex_lock (&m->job_mutex); + job =xe_bb_create_migration_job (m->q, bb, +xe_migrate_batch_base (m, true), + update_idx); +if (IS_ERR (job)) { + err =PTR_ERR (job); + goto err; + } + +xe_sched_job_add_migrate_flush (job,0 ); +xe_sched_job_arm (job); +dma_fence_put (fence); + fence =dma_fence_get (&job->drm.s_fence->finished); +xe_sched_job_push (job); +dma_fence_put (m->fence); + m->fence =dma_fence_get (fence); + +mutex_unlock (&m->job_mutex); + +xe_bb_free (bb, fence); + size -= round_update_size; + src_pa += round_update_size; + dst_pa += round_update_size; +continue ; + +err: +mutex_unlock (&m->job_mutex); +xe_bb_free (bb,NULL ); + +err_sync: +/* Sync partial copy if any. FIXME: under job_mutex? */ +if (fence) { +dma_fence_wait (fence, false); +dma_fence_put (fence); + } + +return ERR_PTR (err); + } + +return fence; +}static void emit_clear_link_copy (struct xe_gt *gt,struct xe_bb *bb,u64 src_ofs,u32 size,u32 pitch) {drivers/gpu/drm/xe/xe_migrate.h +0
-0
drivers/gpu/drm/xe/xe_migrate.h struct dma_fence *xe_migrate_copy(struct xe_migrate *m, · line 110
Quoted 10 lines
drivers/gpu/drm/xe/xe_migrate.h +7
-0@@ -101,6 +101,13 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,struct ttm_resource *dst, bool copy_only_ccs); +struct dma_fence *xe_migrate_pa (struct xe_migrate *m, +u64 src_pa, +bool src_is_vram, +u64 dst_pa, +bool dst_is_vram, +u64 size); +
An option we be export xe_migrate_from_vram / xe_migrate_to_vram and then internally call the function I suggest above with the correct direction agrument too.
Matt
Quoted 6 lines
drivers/gpu/drm/xe/xe_migrate.h +0
-0struct dma_fence *xe_migrate_clear (struct xe_migrate *m,struct xe_bo *bo,struct ttm_resource *dst);Signature · 1 line
2.26.3
PATCH v2 22/31 drm/xe/svm: implement functions to allocate and free device memory
In reply to Oak ZengFunction xe_devm_alloc_pages allocate pages from drm buddy and perform house keeping work for all the pages allocated, such as get a page refcount, keep a bitmap of all pages to denote whether a page is in use, put pages to a drm lru list for eviction purpose.
Function xe_devm_free_blocks return list of memory blocks to drm buddy allocator.
Function xe_devm_free_page is a call back function from hmm layer. It is called whenever a page's refcount reaches to 1. This function clears the bit of this page in the bitmap. If all the bits in the bitmap is cleared, it means all the pages have been freed, we return all the pages in this memory block back to drm buddy.
--- drivers/gpu/drm/xe/xe_svm.h | 7 ++ drivers/gpu/drm/xe/xe_svm_devmem.c | 147 ++++++++++++++++++++++++++++- 2 files changed, 152 insertions(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_svm.h +7 -0
@@ -46,4 +46,11 @@ static inline struct xe_mem_region *xe_page_to_mem_region(struct page *page) return container_of(page->pgmap, struct xe_mem_region, pagemap); } +int xe_devm_alloc_pages (struct xe_tile *tile, +unsigned long npages, +struct list_head *blocks, +unsigned long *pfn); + +void xe_devm_free_blocks (struct list_head *blocks); +void xe_devm_page_free (struct page *page);#endif
drivers/gpu/drm/xe/xe_svm_devmem.c +145 -2
163 lines not on this page · raw message
Signature · 1 line
2.26.3
Matthew Brost
maintainerRe: [v2 22/31] drm/xe/svm: implement functions to allocate and free device memory
Quoted 20 lines · Oak Zeng
Function
xe_devm_alloc_pagesallocate pages from drm buddy and perform house keeping work for all the pages allocated, such as get a page refcount, keep a bitmap of all pages to denote whether a page is in use, put pages to a drm lru list for eviction purpose.Function
xe_devm_free_blocksreturn list of memory blocks to drm buddy allocator.Function
xe_devm_free_pageis a call back function from hmm layer. It is called whenever a page's refcount reaches to 1. This function clears the bit of this page in the bitmap. If all the bits in the bitmap is cleared, it means all the pages have been freed, we return all the pages in this memory block back to drm buddy.Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana Vishwanathapura <[email protected]>Signed-off-by Niranjana Vishwanathapura <[email protected]>Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>
Quoted 3 lines
--- drivers/gpu/drm/xe/xe_svm.h | 7 ++ drivers/gpu/drm/xe/xe_svm_devmem.c | 147 ++++++++++++++++++++++++++++-
See comments about file in previous patches, they apply here too.
Quoted 47 lines
2 files changed, 152 insertions(+), 2 deletions(-)drivers/gpu/drm/xe/xe_svm.h +7
-0@@ -46,4 +46,11 @@ static inline struct xe_mem_region *xe_page_to_mem_region(struct page *page) return container_of(page->pgmap, struct xe_mem_region, pagemap); } +int xe_devm_alloc_pages (struct xe_tile *tile, +unsigned long npages, +struct list_head *blocks, +unsigned long *pfn); + +void xe_devm_free_blocks (struct list_head *blocks); +void xe_devm_page_free (struct page *page);#endif drivers/gpu/drm/xe/xe_svm_devmem.c +17
-1@@ -5,18 +5,161 @@#include <linux/mm_types.h> #include <linux/sched/mm.h> - +#include <linux/gfp.h> +#include <linux/migrate.h> +#include <linux/dma-mapping.h> +#include <linux/dma-fence.h> +#include <linux/bitops.h> +#include <linux/bitmap.h> +#include <drm/drm_buddy.h> #include "xe_device_types.h" #include "xe_svm.h" +#include "xe_migrate.h" +#include "xe_ttm_vram_mgr_types.h" +#include "xe_assert.h" +/** +* struct xe_svm_block_meta - svm uses this data structure to manage each +* block allocated from drm buddy. This will be set to the drm_buddy_block's +* private field. +* +* @lru: used to link this block to drm's lru lists. This will be replace +* with struct drm_lru_entity later.
drivers/gpu/drm/xe/xe_svm_devmem.c line 37
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+* @tile: tile from which we allocated this block +* @bitmap: A bitmap of each page in this block. 1 means this page is used, +* 0 means this page is idle. When all bits of this block are 0, it is time +* to return this block to drm buddy subsystem. +*/ +struct xe_svm_block_meta { +struct list_head lru; +struct xe_tile *tile; +unsigned long bitmap[]; +};
This looks not needed to me but admittedly haven't looked at the LRU stuff.
I am thinking roughly...
- I think we drop all this special tracking (kill xe_svm_block_meta)
- Have functions to allocate / free the buddy blocks, store buddy blocks in userptr
- Blocks are allocated before migration to VRAM
- Blocks can be freed on either CPU fault after migration or on VMA destroy (probably depends on madvive hints for VMA where we free blocks)
- Blocks allocated / freed at ia chunk (xe_vma in this code) granularity (conceptually the same if we switch to 1 to N ratio between xe_vma & pt_state)
- block->private == memory region so we can get pfn from block
- When we need migrate_pfns we loop over buddy blocks populating migrate.dst
Also I noticed the drm_buddy_* calls in this file are not protected by a lock, we will need that. Currently it is tile->mem.vram_mgr->lock in the VRAM mgr code, we either need to reach into there or move this lock to common place so the VRAM manager and block allocations for SVM don't race with each other.
Matt
Quoted 132 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +118
-1static vm_fault_t xe_devm_migrate_to_ram (struct vm_fault *vmf) {return 0 ; } -static void xe_devm_page_free (struct page *page) +static u64 block_offset_to_pfn (struct xe_mem_region *mr,u64 offset) +{ +/** DRM buddy's block offset is 0-based*/ + offset += mr->hpa_base; + +return PHYS_PFN (offset); +} + +/** FIXME: we locked page by calling zone_device_page_init +* in xe_devm_alloc_pages. Should we unlock pages here? +*/ +static void free_block (struct drm_buddy_block *block) +{ +struct xe_svm_block_meta *meta = + (struct xe_svm_block_meta *)block->private; +struct xe_tile *tile = meta->tile; +struct drm_buddy *mm = &tile->mem.vram_mgr->mm; + +kfree (block->private); +drm_buddy_free_block (mm, block); +} + +void xe_devm_page_free (struct page *page) +{ +struct drm_buddy_block *block = + (struct drm_buddy_block *)page->zone_device_data; +struct xe_svm_block_meta *meta = + (struct xe_svm_block_meta *)block->private; +struct xe_tile *tile = meta->tile; +struct xe_mem_region *mr = &tile->mem.vram; +struct drm_buddy *mm = &tile->mem.vram_mgr->mm; +u64 size =drm_buddy_block_size (mm, block); +u64 pages_per_block = size >>PAGE_SHIFT ; +u64 block_pfn_first = +block_offset_to_pfn (mr,drm_buddy_block_offset (block)); +u64 page_pfn =page_to_pfn (page); +u64 i = page_pfn - block_pfn_first; + +xe_assert (tile->xe, i < pages_per_block); +clear_bit (i, meta->bitmap); +if (bitmap_empty (meta->bitmap, pages_per_block)) +free_block (block); +} + +/** +* xe_devm_alloc_pages() - allocate device pages from buddy allocator +* +* @xe_tile: which tile to allocate device memory from +* @npages: how many pages to allocate +* @blocks: used to return the allocated blocks +* @pfn: used to return the pfn of all allocated pages. Must be big enough +* to hold at @npages entries. +* +* This function allocate blocks of memory from drm buddy allocator, and +* performs initialization work: set struct page::zone_device_data to point +* to the memory block; set/initialize drm_buddy_block::private field; +* lock_page for each page allocated; add memory block to lru managers lru +* list - this is TBD. +* +* return: 0 on success +* error code otherwise +*/ +int xe_devm_alloc_pages (struct xe_tile *tile, +unsigned long npages, +struct list_head *blocks, +unsigned long *pfn) +{ +struct drm_buddy *mm = &tile->mem.vram_mgr->mm; +struct drm_buddy_block *block, *tmp; +u64 size = npages <<PAGE_SHIFT ; +int ret =0 , i, j =0 ; + + ret =drm_buddy_alloc_blocks (mm,0 , mm->size, size,PAGE_SIZE , + blocks,DRM_BUDDY_TOPDOWN_ALLOCATION ); + +if (unlikely (ret)) +return ret; + +list_for_each_entry_safe (block, tmp, blocks, link) { +struct xe_mem_region *mr = &tile->mem.vram; +u64 block_pfn_first, pages_per_block; +struct xe_svm_block_meta *meta; +u32 meta_size; + + size =drm_buddy_block_size (mm, block); + pages_per_block = size >>PAGE_SHIFT ; + meta_size =BITS_TO_BYTES (pages_per_block) + +sizeof (struct xe_svm_block_meta ); + meta =kzalloc (meta_size,GFP_KERNEL ); +bitmap_fill (meta->bitmap, pages_per_block); + meta->tile = tile; + block->private = meta; + block_pfn_first = +block_offset_to_pfn (mr,drm_buddy_block_offset (block)); +for (i =0 ; i < pages_per_block; i++) { +struct page *page; + + pfn[j++] = block_pfn_first + i; + page =pfn_to_page (block_pfn_first + i); +/**Lock page per hmm requirement, see hmm.rst.*/ +zone_device_page_init (page); + page->zone_device_data = block; + } + } + +return ret; +} + +/** +* xe_devm_free_blocks() - free all memory blocks +* +* @blocks: memory blocks list head +*/ +void xe_devm_free_blocks (struct list_head *blocks) { +struct drm_buddy_block *block, *tmp; + +list_for_each_entry_safe (block, tmp, blocks, link) +free_block (block); }static const struct dev_pagemap_ops xe_devm_pagemap_ops = {Signature · 1 line
2.26.3
RE: [v2 22/31] drm/xe/svm: implement functions to allocate and free device memory
Hi Matt,
Quoted 106 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 6:24 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 22/31] drm/xe/svm: implement functions to allocate and free device memoryQuoted 8 lines · Oak Zeng
Function
xe_devm_alloc_pagesallocate pages from drm buddy and perform house keeping work for all the pages allocated, such as get a page refcount, keep a bitmap of all pages to denote whether a page is in use, put pages to a drm lru list for eviction purpose.Function
xe_devm_free_blocksreturn list of memory blocks to drm buddy allocator.Quoted 8 lines
Function
xe_devm_free_pageis a call back function from hmm layer. It is called whenever a page's refcount reaches to 1. This function clears the bit of this page in the bitmap. If all the bits in the bitmap is cleared, it means all the pages have been freed, we return all the pages in this memory block back to drm buddy.Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana VishwanathapuraQuoted 1 line
Signed-off-by Niranjana VishwanathapuraQuoted 6 lines
Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/xe_svm.h | 7 ++ drivers/gpu/drm/xe/xe_svm_devmem.c | 147++++++++++++++++++++++++++++-
See comments about file in previous patches, they apply here too.
drivers/gpu/drm/xe/xe_svm.h static inline struct xe_mem_region
Quoted 7 lines
2 files changed, 152 insertions(+), 2 deletions(-)drivers/gpu/drm/xe/xe_svm.h +0
-0@@ -46,4 +46,11 @@ static inline struct xe_mem_region*xe_page_to_mem_region(struct page *page)
Review
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm.h +7
-0return container_of(page->pgmap, struct xe_mem_region, pagemap); } +int xe_devm_alloc_pages (struct xe_tile *tile, +unsigned long npages, +struct list_head *blocks, +unsigned long *pfn); + +void xe_devm_free_blocks (struct list_head *blocks); +void xe_devm_page_free (struct page *page);#endif (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_devmem.cQuoted 14 lines
index
31af56e8285a..5ba0cd9a70b0100644drivers/gpu/drm/xe/xe_svm_devmem.c +6
-1@@ -5,18 +5,161 @@#include <linux/mm_types.h> #include <linux/sched/mm.h> - +#include <linux/gfp.h> +#include <linux/migrate.h> +#include <linux/dma-mapping.h> +#include <linux/dma-fence.h> +#include <linux/bitops.h> +#include <linux/bitmap.h> drivers/gpu/drm/xe/xe_svm_devmem.c line 23
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +7
-0+#include <drm/drm_buddy.h> #include "xe_device_types.h" #include "xe_svm.h" +#include "xe_migrate.h" +#include "xe_ttm_vram_mgr_types.h" +#include "xe_assert.h" +/** +* struct xe_svm_block_meta - svm uses this data structure to manage each +* block allocated from drm buddy. This will be set to the drm_buddy_block's
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +4
-0+* private field. +* +* @lru: used to link this block to drm's lru lists. This will be replace +* with struct drm_lru_entity later. drivers/gpu/drm/xe/xe_svm_devmem.c line 37
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+* @tile: tile from which we allocated this block +* @bitmap: A bitmap of each page in this block. 1 means this page is used, +* 0 means this page is idle. When all bits of this block are 0, it is time +* to return this block to drm buddy subsystem. +*/ +struct xe_svm_block_meta { +struct list_head lru; +struct xe_tile *tile; +unsigned long bitmap[]; +};This looks not needed to me but admittedly haven't looked at the LRU stuff.
I am thinking roughly...
Quoted 13 lines
- I think we drop all this special tracking (kill
xe_svm_block_meta)
- Have functions to allocate / free the buddy blocks, store buddy blocks in userptr
- Blocks are allocated before migration to VRAM
- Blocks can be freed on either CPU fault after migration or on VMA destroy (probably depends on madvive hints for VMA where we free blocks)
- Blocks allocated / freed at ia chunk (xe_vmain this code) granularity (conceptually the same if we switch to 1 to N ratio betweenxe_vma&pt_state)
-block->private== memory region so we can get pfn from block
- When we needmigrate_pfnswe loop over buddy blocks populating migrate.dst
I thought into your scheme. The free of device memory is not completely controlled by driver. Core mm can call back to driver to free a device memory page. The xe_devm_page_free in this series is a callback function registered to core mm. this is why in above data structure I have to have a bitmap. This bitmap is used to mark which page is freed, when all pages in a buddy block are freed, it is time to free the whole buddy block.
In your scheme, we allocate/free at xe_vma granularity. So I imagine you would have a list of buddy blocks in usrptr, and free all blocks in the list when every pages in all blocks are freed. While my scheme free memory at the buddy block granularity - I think it natural because the buddy free interface is also block based.
You would eventually need to introduce a lru link to link each buddy block to a lru list when vram eviction come into picture.
So I just explained why above xe_svm_block_meta was introduced, such as why the bitmap and lru field is necessary to me. if you drop this data structure, they will have to show up in another way.
Quoted 7 lines
Also I noticed the drm_buddy_* calls in this file are not protected by a lock, we will need that. Currently it is
tile->mem.vram_mgr->lockin the VRAM mgr code, we either need to reach into there or move this lock to common place so the VRAM manager and block allocations for SVM don't race with each other.
Yes, the lock has to be added. Thanks for pointing this out. Maybe move the tile->mem.vram_mgr->lock to the xe_tile level so it can be shared b/t BO-driver and system allocator?
Oak
Quoted 139 lines
Matt
Quoted 34 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +26
-1static vm_fault_t xe_devm_migrate_to_ram (struct vm_fault *vmf) {return 0 ; } -static void xe_devm_page_free(struct page *page) +static u64 block_offset_to_pfn (struct xe_mem_region *mr,u64 offset) +{ +/** DRM buddy's block offset is 0-based*/ + offset += mr->hpa_base; + +return PHYS_PFN (offset); +} + +/** FIXME: we locked page by calling zone_device_page_init +* in xe_devm_alloc_pages. Should we unlock pages here? +*/ +static void free_block (struct drm_buddy_block *block) +{ +struct xe_svm_block_meta *meta = + (struct xe_svm_block_meta *)block->private; +struct xe_tile *tile = meta->tile; +struct drm_buddy *mm = &tile->mem.vram_mgr->mm; + +kfree (block->private); +drm_buddy_free_block (mm, block); +} + +void xe_devm_page_free (struct page *page) +{ +struct drm_buddy_block *block = + (struct drm_buddy_block *)page- zone_device_data;drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +9
-0+struct xe_svm_block_meta *meta = + (struct xe_svm_block_meta *)block- private; +struct xe_tile *tile = meta->tile; +struct xe_mem_region *mr = &tile->mem.vram; +struct drm_buddy *mm = &tile->mem.vram_mgr->mm; +u64 size =drm_buddy_block_size (mm, block); +u64 pages_per_block = size >>PAGE_SHIFT ; +u64 block_pfn_first = + block_offset_to_pfn(mr,
drm_buddy_block_offset(block));Quoted 29 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +29
-0+u64 page_pfn =page_to_pfn (page); +u64 i = page_pfn - block_pfn_first; + +xe_assert (tile->xe, i < pages_per_block); +clear_bit (i, meta->bitmap); +if (bitmap_empty (meta->bitmap, pages_per_block)) +free_block (block); +} + +/** +* xe_devm_alloc_pages() - allocate device pages from buddy allocator +* +* @xe_tile: which tile to allocate device memory from +* @npages: how many pages to allocate +* @blocks: used to return the allocated blocks +* @pfn: used to return the pfn of all allocated pages. Must be big enough +* to hold at @npages entries. +* +* This function allocate blocks of memory from drm buddy allocator, and +* performs initialization work: set struct page::zone_device_data to point +* to the memory block; set/initialize drm_buddy_block::private field; +* lock_page for each page allocated; add memory block to lru managers lru +* list - this is TBD. +* +* return: 0 on success +* error code otherwise +*/ +int xe_devm_alloc_pages(struct xe_tile *tile, + unsigned long npages,drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+struct list_head *blocks, + unsigned long *pfn) +{ + struct drm_buddy *mm = &tile->mem.vram_mgr->mm; +struct drm_buddy_block *block, *tmp; +u64 size = npages <<PAGE_SHIFT ; +int ret =0 , i, j =0 ; + + ret = drm_buddy_alloc_blocks(mm, 0, mm->size, size, PAGE_SIZE, + blocks,
DRM_BUDDY_TOPDOWN_ALLOCATION);Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+ +if (unlikely (ret)) +return ret; + +list_for_each_entry_safe (block, tmp, blocks, link) { +struct xe_mem_region *mr = &tile->mem.vram; +u64 block_pfn_first, pages_per_block; +struct xe_svm_block_meta *meta; +u32 meta_size; +drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+ size =drm_buddy_block_size (mm, block); + pages_per_block = size >>PAGE_SHIFT ; + meta_size =BITS_TO_BYTES (pages_per_block) + +sizeof (struct xe_svm_block_meta ); + meta =kzalloc (meta_size,GFP_KERNEL ); +bitmap_fill (meta->bitmap, pages_per_block); + meta->tile = tile; + block->private = meta; + block_pfn_first = + block_offset_to_pfn(mr,
drm_buddy_block_offset(block));Quoted 31 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +24
-0+for (i =0 ; i < pages_per_block; i++) { +struct page *page; + + pfn[j++] = block_pfn_first + i; + page =pfn_to_page (block_pfn_first + i); +/**Lock page per hmm requirement, see hmm.rst.*/ +zone_device_page_init (page); + page->zone_device_data = block; + } + } + +return ret; +} + +/** +* xe_devm_free_blocks() - free all memory blocks +* +* @blocks: memory blocks list head +*/ +void xe_devm_free_blocks (struct list_head *blocks) { +struct drm_buddy_block *block, *tmp; + +list_for_each_entry_safe (block, tmp, blocks, link) +free_block (block); }static const struct
dev_pagemap_opsxe_devm_pagemap_ops= {
--
2.26.3
Matthew Brost
maintainerRe: [v2 22/31] drm/xe/svm: implement functions to allocate and free device memory
Quoted 122 lines · Oak
Hi Matt,
Quoted 106 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 6:24 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 22/31] drm/xe/svm: implement functions to allocate and free device memoryQuoted 8 lines · Oak Zeng
Function
xe_devm_alloc_pagesallocate pages from drm buddy and perform house keeping work for all the pages allocated, such as get a page refcount, keep a bitmap of all pages to denote whether a page is in use, put pages to a drm lru list for eviction purpose.Function
xe_devm_free_blocksreturn list of memory blocks to drm buddy allocator.Quoted 8 lines
Function
xe_devm_free_pageis a call back function from hmm layer. It is called whenever a page's refcount reaches to 1. This function clears the bit of this page in the bitmap. If all the bits in the bitmap is cleared, it means all the pages have been freed, we return all the pages in this memory block back to drm buddy.Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana VishwanathapuraQuoted 1 line
Signed-off-by Niranjana VishwanathapuraQuoted 6 lines
Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/xe_svm.h | 7 ++ drivers/gpu/drm/xe/xe_svm_devmem.c | 147++++++++++++++++++++++++++++-
See comments about file in previous patches, they apply here too.
drivers/gpu/drm/xe/xe_svm.h static inline struct xe_mem_region
Quoted 7 lines
2 files changed, 152 insertions(+), 2 deletions(-)drivers/gpu/drm/xe/xe_svm.h +0
-0@@ -46,4 +46,11 @@ static inline struct xe_mem_region*xe_page_to_mem_region(struct page *page)
Review
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm.h +7
-0return container_of(page->pgmap, struct xe_mem_region, pagemap); } +int xe_devm_alloc_pages (struct xe_tile *tile, +unsigned long npages, +struct list_head *blocks, +unsigned long *pfn); + +void xe_devm_free_blocks (struct list_head *blocks); +void xe_devm_page_free (struct page *page);#endif (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_devmem.cQuoted 14 lines
index
31af56e8285a..5ba0cd9a70b0100644drivers/gpu/drm/xe/xe_svm_devmem.c +6
-1@@ -5,18 +5,161 @@#include <linux/mm_types.h> #include <linux/sched/mm.h> - +#include <linux/gfp.h> +#include <linux/migrate.h> +#include <linux/dma-mapping.h> +#include <linux/dma-fence.h> +#include <linux/bitops.h> +#include <linux/bitmap.h> drivers/gpu/drm/xe/xe_svm_devmem.c line 23
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +7
-0+#include <drm/drm_buddy.h> #include "xe_device_types.h" #include "xe_svm.h" +#include "xe_migrate.h" +#include "xe_ttm_vram_mgr_types.h" +#include "xe_assert.h" +/** +* struct xe_svm_block_meta - svm uses this data structure to manage each +* block allocated from drm buddy. This will be set to the drm_buddy_block's
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +4
-0+* private field. +* +* @lru: used to link this block to drm's lru lists. This will be replace +* with struct drm_lru_entity later. drivers/gpu/drm/xe/xe_svm_devmem.c line 37
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+* @tile: tile from which we allocated this block +* @bitmap: A bitmap of each page in this block. 1 means this page is used, +* 0 means this page is idle. When all bits of this block are 0, it is time +* to return this block to drm buddy subsystem. +*/ +struct xe_svm_block_meta { +struct list_head lru; +struct xe_tile *tile; +unsigned long bitmap[]; +};This looks not needed to me but admittedly haven't looked at the LRU stuff.
I am thinking roughly...
Quoted 13 lines
- I think we drop all this special tracking (kill
xe_svm_block_meta)
- Have functions to allocate / free the buddy blocks, store buddy blocks in userptr
- Blocks are allocated before migration to VRAM
- Blocks can be freed on either CPU fault after migration or on VMA destroy (probably depends on madvive hints for VMA where we free blocks)
- Blocks allocated / freed at ia chunk (xe_vmain this code) granularity (conceptually the same if we switch to 1 to N ratio betweenxe_vma&pt_state)
-block->private== memory region so we can get pfn from block
- When we needmigrate_pfnswe loop over buddy blocks populating migrate.dst
Quoted 2 lines
I thought into your scheme. The free of device memory is not completely controlled by driver. Core mm can call back to driver to free a device memory page. The
xe_devm_page_freein this series is a callback function registered to core mm. this is why in above data structure I have to have a bitmap. This bitmap is used to mark which page is freed, when all pages in a buddy block are freed, it is time to free the whole buddy block.
Certainly in this scenario we'd also get a mmu notifier with MMU_NOTIFY_UNMAP when pages are being free'd too, right? The notifier puts the VMA in the garbage collector and the blocks are free'd when VMA is destroyed.
The garbage collector only supports complete unmaps at the moment but if we need to support partial unmaps (likely do as a users can munmap partial buffers) we can with garbage collector transfering ownership of blocks from one VMA to another as needed.
It is possible I don't fully understand the ref couting scheme for pages either and we will need to implement the dev_pagemap_ops.free_pagses (seems likely now that I am typing) rather than the notifier scheme described above...
If we need to do this, then roughly...
- page->zone_device_data is still the Xe chunk (xe_vma currently)
- Ref count device pages in Xe chunk (or perhaps individual block?, need to think about this more but certainly bitmap is overkill)
- free_pages decrements ref count
- When ref count goes to zero, free blocks
Again this seems to align with Nouveu and AMD (haven't check Nvidia's open source driver) and aligns with the design in Xe of everything being a chunk grainularity (e.g. no partial migrations within a chunk).
I guess we will a need to a test to do partial unmaps to figure out all of these details...
Quoted 6 lines
In your scheme, we allocate/free at
xe_vmagranularity. So I imagine you would have a list of buddy blocks in usrptr, and free all blocks in the list when every pages in all blocks are freed. While my scheme free memory at the buddy block granularity - I think it natural because the buddy free interface is also block based.You would eventually need to introduce a lru link to link each buddy block to a lru list when vram eviction come into picture.
So I just explained why above
xe_svm_block_metawas introduced, such as why the bitmap and lru field is necessary to me. if you drop this data structure, they will have to show up in another way.
LRU will likely be at chunk grainularity too (i.e. xe_vma not at block level).
Also in most cases xe_vma == 1 block if the buddy allocator is doing its job so no reason to optimize for block level here.
Quoted 10 lines
Quoted 7 lines
Also I noticed the drm_buddy_* calls in this file are not protected by a lock, we will need that. Currently it is
tile->mem.vram_mgr->lockin the VRAM mgr code, we either need to reach into there or move this lock to common place so the VRAM manager and block allocations for SVM don't race with each other.Yes, the lock has to be added. Thanks for pointing this out. Maybe move the
tile->mem.vram_mgr->lockto thexe_tilelevel so it can be shared b/t BO-driver and system allocator?
Yea, tile->mem.vram_lock might be a better location.
Matt
Quoted 141 lines
Oak
Quoted 139 lines
Matt
Quoted 34 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +26
-1static vm_fault_t xe_devm_migrate_to_ram (struct vm_fault *vmf) {return 0 ; } -static void xe_devm_page_free(struct page *page) +static u64 block_offset_to_pfn (struct xe_mem_region *mr,u64 offset) +{ +/** DRM buddy's block offset is 0-based*/ + offset += mr->hpa_base; + +return PHYS_PFN (offset); +} + +/** FIXME: we locked page by calling zone_device_page_init +* in xe_devm_alloc_pages. Should we unlock pages here? +*/ +static void free_block (struct drm_buddy_block *block) +{ +struct xe_svm_block_meta *meta = + (struct xe_svm_block_meta *)block->private; +struct xe_tile *tile = meta->tile; +struct drm_buddy *mm = &tile->mem.vram_mgr->mm; + +kfree (block->private); +drm_buddy_free_block (mm, block); +} + +void xe_devm_page_free (struct page *page) +{ +struct drm_buddy_block *block = + (struct drm_buddy_block *)page- zone_device_data;drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +9
-0+struct xe_svm_block_meta *meta = + (struct xe_svm_block_meta *)block- private; +struct xe_tile *tile = meta->tile; +struct xe_mem_region *mr = &tile->mem.vram; +struct drm_buddy *mm = &tile->mem.vram_mgr->mm; +u64 size =drm_buddy_block_size (mm, block); +u64 pages_per_block = size >>PAGE_SHIFT ; +u64 block_pfn_first = + block_offset_to_pfn(mr,
drm_buddy_block_offset(block));Quoted 29 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +29
-0+u64 page_pfn =page_to_pfn (page); +u64 i = page_pfn - block_pfn_first; + +xe_assert (tile->xe, i < pages_per_block); +clear_bit (i, meta->bitmap); +if (bitmap_empty (meta->bitmap, pages_per_block)) +free_block (block); +} + +/** +* xe_devm_alloc_pages() - allocate device pages from buddy allocator +* +* @xe_tile: which tile to allocate device memory from +* @npages: how many pages to allocate +* @blocks: used to return the allocated blocks +* @pfn: used to return the pfn of all allocated pages. Must be big enough +* to hold at @npages entries. +* +* This function allocate blocks of memory from drm buddy allocator, and +* performs initialization work: set struct page::zone_device_data to point +* to the memory block; set/initialize drm_buddy_block::private field; +* lock_page for each page allocated; add memory block to lru managers lru +* list - this is TBD. +* +* return: 0 on success +* error code otherwise +*/ +int xe_devm_alloc_pages(struct xe_tile *tile, + unsigned long npages,drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+struct list_head *blocks, + unsigned long *pfn) +{ + struct drm_buddy *mm = &tile->mem.vram_mgr->mm; +struct drm_buddy_block *block, *tmp; +u64 size = npages <<PAGE_SHIFT ; +int ret =0 , i, j =0 ; + + ret = drm_buddy_alloc_blocks(mm, 0, mm->size, size, PAGE_SIZE, + blocks,
DRM_BUDDY_TOPDOWN_ALLOCATION);Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+ +if (unlikely (ret)) +return ret; + +list_for_each_entry_safe (block, tmp, blocks, link) { +struct xe_mem_region *mr = &tile->mem.vram; +u64 block_pfn_first, pages_per_block; +struct xe_svm_block_meta *meta; +u32 meta_size; +drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+ size =drm_buddy_block_size (mm, block); + pages_per_block = size >>PAGE_SHIFT ; + meta_size =BITS_TO_BYTES (pages_per_block) + +sizeof (struct xe_svm_block_meta ); + meta =kzalloc (meta_size,GFP_KERNEL ); +bitmap_fill (meta->bitmap, pages_per_block); + meta->tile = tile; + block->private = meta; + block_pfn_first = + block_offset_to_pfn(mr,
drm_buddy_block_offset(block));Quoted 31 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +24
-0+for (i =0 ; i < pages_per_block; i++) { +struct page *page; + + pfn[j++] = block_pfn_first + i; + page =pfn_to_page (block_pfn_first + i); +/**Lock page per hmm requirement, see hmm.rst.*/ +zone_device_page_init (page); + page->zone_device_data = block; + } + } + +return ret; +} + +/** +* xe_devm_free_blocks() - free all memory blocks +* +* @blocks: memory blocks list head +*/ +void xe_devm_free_blocks (struct list_head *blocks) { +struct drm_buddy_block *block, *tmp; + +list_for_each_entry_safe (block, tmp, blocks, link) +free_block (block); }static const struct
dev_pagemap_opsxe_devm_pagemap_ops= {
--
2.26.3
RE: [v2 22/31] drm/xe/svm: implement functions to allocate and free device memory
In reply to Matthew BrostHi Matt,
Quoted 91 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 6:24 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 22/31] drm/xe/svm: implement functions to allocate and free device memoryQuoted 1 line · Oak Zeng
Function
xe_devm_alloc_pagesallocate pages from drm buddy andperform
Quoted 7 lines
house keeping work for all the pages allocated, such as get a page refcount, keep a bitmap of all pages to denote whether a page is in use, put pages to a drm lru list for eviction purpose.
Function
xe_devm_free_blocksreturn list of memory blocks to drm buddy allocator.Quoted 8 lines
Function
xe_devm_free_pageis a call back function from hmm layer. It is called whenever a page's refcount reaches to 1. This function clears the bit of this page in the bitmap. If all the bits in the bitmap is cleared, it means all the pages have been freed, we return all the pages in this memory block back to drm buddy.Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana VishwanathapuraQuoted 1 line
Signed-off-by Niranjana VishwanathapuraQuoted 6 lines
Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/xe_svm.h | 7 ++ drivers/gpu/drm/xe/xe_svm_devmem.c | 147++++++++++++++++++++++++++++-
See comments about file in previous patches, they apply here too.
Review
Quoted 3 lines
2 files changed, 152 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm.hdrivers/gpu/drm/xe/xe_svm.h static inline struct xe_mem_region
Quoted 4 lines
index
624c1581f8ba..92a3ee90d5a7100644drivers/gpu/drm/xe/xe_svm.h +0
-0@@ -46,4 +46,11 @@ static inline struct xe_mem_region*xe_page_to_mem_region(struct page *page)
drivers/gpu/drm/xe/xe_svm.h static inline struct xe_mem_region · line 46
Quoted 1 line
drivers/gpu/drm/xe/xe_svm.h +0
-0return container_of(page->pgmap, struct xe_mem_region,pagemap);
Review
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm.h +7
-0} +int xe_devm_alloc_pages (struct xe_tile *tile, +unsigned long npages, +struct list_head *blocks, +unsigned long *pfn); + +void xe_devm_free_blocks (struct list_head *blocks); +void xe_devm_page_free (struct page *page);#endif (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_devmem.cQuoted 13 lines
index
31af56e8285a..5ba0cd9a70b0100644drivers/gpu/drm/xe/xe_svm_devmem.c +5
-1@@ -5,18 +5,161 @@#include <linux/mm_types.h> #include <linux/sched/mm.h> - +#include <linux/gfp.h> +#include <linux/migrate.h> +#include <linux/dma-mapping.h> +#include <linux/dma-fence.h> +#include <linux/bitops.h> drivers/gpu/drm/xe/xe_svm_devmem.c line 22
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +7
-0+#include <linux/bitmap.h> +#include <drm/drm_buddy.h> #include "xe_device_types.h" #include "xe_svm.h" +#include "xe_migrate.h" +#include "xe_ttm_vram_mgr_types.h" +#include "xe_assert.h" +/** +* struct xe_svm_block_meta - svm uses this data structure to manage each
drivers/gpu/drm/xe/xe_svm_devmem.c line 23
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_devmem.c +1
-0+* block allocated from drm buddy. This will be set to the drm_buddy_block's
Quoted 16 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +4
-0+* private field. +* +* @lru: used to link this block to drm's lru lists. This will be replace +* with struct drm_lru_entity later. drivers/gpu/drm/xe/xe_svm_devmem.c line 37
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+* @tile: tile from which we allocated this block +* @bitmap: A bitmap of each page in this block. 1 means this page is used, +* 0 means this page is idle. When all bits of this block are 0, it is time +* to return this block to drm buddy subsystem. +*/ +struct xe_svm_block_meta { +struct list_head lru; +struct xe_tile *tile; +unsigned long bitmap[]; +};This looks not needed to me but admittedly haven't looked at the LRU stuff.
I am moving to page granularity memory eviction, so we can either use the lru in the struct page itself, or I will have to introduce some other data structure which have a lru.
Yes, I removed block_meta in v3.
Quoted 4 lines
I am thinking roughly...
- I think we drop all this special tracking (kill
xe_svm_block_meta)
Agreed.
Quoted 2 lines
- Have functions to allocate / free the buddy blocks, store buddy blocks in userptr
Why we need to store buddy blocks in userptr? If you do this, you would need another block_list in userptr.
I currently use the struct page's zone_device_data to point to buddy_block. It seems work for me.
Quoted 1 line
- Blocks are allocated before migration to VRAM
Agreed
Quoted 3 lines
- Blocks can be freed on either CPU fault after migration or on VMA destroy (probably depends on madvive hints for VMA where we free blocks)
My current understanding is, once device pages is allocated and handed over to core mm/hmm, driver doesn't need to worry about the life cycle of device pages, i.e., core mm/hmm will take care by calling back to page_free vfunc.
Quoted 3 lines
- Blocks allocated / freed at ia chunk (
xe_vmain this code) granularity (conceptually the same if we switch to 1 to N ratio betweenxe_vma&pt_state)
As said, I am moving to page granularity, moving away from xe_vma granularity, to address concerns from the drm community discussion.
Quoted 1 line
-
block->private== memory region so we can get pfn from block
In v3 code, block->private is not used. Will use it if needed.
Each struct page has a pgmap pointer which points to xe_mem_region's pgmap memory. We can use this information to get a page/fpn's memory region.
Quoted 8 lines
- When we need
migrate_pfnswe loop over buddy blocks populating migrate.dstAlso I noticed the drm_buddy_* calls in this file are not protected by a lock, we will need that. Currently it is
tile->mem.vram_mgr->lockin the VRAM mgr code, we either need to reach into there or move this lock to common place so the VRAM manager and block allocations for SVM don't race with each other.
Ok, will add this lock. Lets keep the lock in vram_mgr->lock for now for simplicity
Oak
Quoted 141 lines
Matt
Quoted 34 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +26
-1static vm_fault_t xe_devm_migrate_to_ram (struct vm_fault *vmf) {return 0 ; } -static void xe_devm_page_free(struct page *page) +static u64 block_offset_to_pfn (struct xe_mem_region *mr,u64 offset) +{ +/** DRM buddy's block offset is 0-based*/ + offset += mr->hpa_base; + +return PHYS_PFN (offset); +} + +/** FIXME: we locked page by calling zone_device_page_init +* in xe_devm_alloc_pages. Should we unlock pages here? +*/ +static void free_block (struct drm_buddy_block *block) +{ +struct xe_svm_block_meta *meta = + (struct xe_svm_block_meta *)block->private; +struct xe_tile *tile = meta->tile; +struct drm_buddy *mm = &tile->mem.vram_mgr->mm; + +kfree (block->private); +drm_buddy_free_block (mm, block); +} + +void xe_devm_page_free (struct page *page) +{ +struct drm_buddy_block *block = + (struct drm_buddy_block *)page- zone_device_data;drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +9
-0+struct xe_svm_block_meta *meta = + (struct xe_svm_block_meta *)block- private; +struct xe_tile *tile = meta->tile; +struct xe_mem_region *mr = &tile->mem.vram; +struct drm_buddy *mm = &tile->mem.vram_mgr->mm; +u64 size =drm_buddy_block_size (mm, block); +u64 pages_per_block = size >>PAGE_SHIFT ; +u64 block_pfn_first = + block_offset_to_pfn(mr,
drm_buddy_block_offset(block));Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +12
-0+u64 page_pfn =page_to_pfn (page); +u64 i = page_pfn - block_pfn_first; + +xe_assert (tile->xe, i < pages_per_block); +clear_bit (i, meta->bitmap); +if (bitmap_empty (meta->bitmap, pages_per_block)) +free_block (block); +} + +/** +* xe_devm_alloc_pages() - allocate device pages from buddy allocator +* drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+* @xe_tile: which tile to allocate device memory from +* @npages: how many pages to allocate +* @blocks: used to return the allocated blocks +* @pfn: used to return the pfn of all allocated pages. Must be big enough +* to hold at @npages entries. +* +* This function allocate blocks of memory from drm buddy allocator, and +* performs initialization work: set struct page::zone_device_data to point +* to the memory block; set/initialize drm_buddy_block::private field; +* lock_page for each page allocated; add memory block to lru managers lru
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +7
-0+* list - this is TBD. +* +* return: 0 on success +* error code otherwise +*/ +int xe_devm_alloc_pages(struct xe_tile *tile, + unsigned long npages,drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+struct list_head *blocks, + unsigned long *pfn) +{ + struct drm_buddy *mm = &tile->mem.vram_mgr->mm; +struct drm_buddy_block *block, *tmp; +u64 size = npages <<PAGE_SHIFT ; +int ret =0 , i, j =0 ; + + ret = drm_buddy_alloc_blocks(mm, 0, mm->size, size, PAGE_SIZE, + blocks,
DRM_BUDDY_TOPDOWN_ALLOCATION);Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+ +if (unlikely (ret)) +return ret; + +list_for_each_entry_safe (block, tmp, blocks, link) { +struct xe_mem_region *mr = &tile->mem.vram; +u64 block_pfn_first, pages_per_block; +struct xe_svm_block_meta *meta; +u32 meta_size; +drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+ size =drm_buddy_block_size (mm, block); + pages_per_block = size >>PAGE_SHIFT ; + meta_size =BITS_TO_BYTES (pages_per_block) + +sizeof (struct xe_svm_block_meta ); + meta =kzalloc (meta_size,GFP_KERNEL ); +bitmap_fill (meta->bitmap, pages_per_block); + meta->tile = tile; + block->private = meta; + block_pfn_first = + block_offset_to_pfn(mr,
drm_buddy_block_offset(block));Quoted 31 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +24
-0+for (i =0 ; i < pages_per_block; i++) { +struct page *page; + + pfn[j++] = block_pfn_first + i; + page =pfn_to_page (block_pfn_first + i); +/**Lock page per hmm requirement, see hmm.rst.*/ +zone_device_page_init (page); + page->zone_device_data = block; + } + } + +return ret; +} + +/** +* xe_devm_free_blocks() - free all memory blocks +* +* @blocks: memory blocks list head +*/ +void xe_devm_free_blocks (struct list_head *blocks) { +struct drm_buddy_block *block, *tmp; + +list_for_each_entry_safe (block, tmp, blocks, link) +free_block (block); }static const struct
dev_pagemap_opsxe_devm_pagemap_ops= {
--
2.26.3
Matthew Brost
maintainerRe: [v2 22/31] drm/xe/svm: implement functions to allocate and free device memory
Quoted 110 lines · Oak
Hi Matt,
Quoted 91 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 6:24 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 22/31] drm/xe/svm: implement functions to allocate and free device memoryQuoted 1 line · Oak Zeng
Function
xe_devm_alloc_pagesallocate pages from drm buddy andperform
Quoted 7 lines
house keeping work for all the pages allocated, such as get a page refcount, keep a bitmap of all pages to denote whether a page is in use, put pages to a drm lru list for eviction purpose.
Function
xe_devm_free_blocksreturn list of memory blocks to drm buddy allocator.Quoted 8 lines
Function
xe_devm_free_pageis a call back function from hmm layer. It is called whenever a page's refcount reaches to 1. This function clears the bit of this page in the bitmap. If all the bits in the bitmap is cleared, it means all the pages have been freed, we return all the pages in this memory block back to drm buddy.Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana VishwanathapuraQuoted 1 line
Signed-off-by Niranjana VishwanathapuraQuoted 6 lines
Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/xe_svm.h | 7 ++ drivers/gpu/drm/xe/xe_svm_devmem.c | 147++++++++++++++++++++++++++++-
See comments about file in previous patches, they apply here too.
Review
Quoted 3 lines
2 files changed, 152 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm.hdrivers/gpu/drm/xe/xe_svm.h static inline struct xe_mem_region
Quoted 4 lines
index
624c1581f8ba..92a3ee90d5a7100644drivers/gpu/drm/xe/xe_svm.h +0
-0@@ -46,4 +46,11 @@ static inline struct xe_mem_region*xe_page_to_mem_region(struct page *page)
drivers/gpu/drm/xe/xe_svm.h static inline struct xe_mem_region · line 46
Quoted 1 line
drivers/gpu/drm/xe/xe_svm.h +0
-0return container_of(page->pgmap, struct xe_mem_region,pagemap);
Review
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm.h +7
-0} +int xe_devm_alloc_pages (struct xe_tile *tile, +unsigned long npages, +struct list_head *blocks, +unsigned long *pfn); + +void xe_devm_free_blocks (struct list_head *blocks); +void xe_devm_page_free (struct page *page);#endif (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_devmem.cQuoted 13 lines
index
31af56e8285a..5ba0cd9a70b0100644drivers/gpu/drm/xe/xe_svm_devmem.c +5
-1@@ -5,18 +5,161 @@#include <linux/mm_types.h> #include <linux/sched/mm.h> - +#include <linux/gfp.h> +#include <linux/migrate.h> +#include <linux/dma-mapping.h> +#include <linux/dma-fence.h> +#include <linux/bitops.h> drivers/gpu/drm/xe/xe_svm_devmem.c line 22
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +7
-0+#include <linux/bitmap.h> +#include <drm/drm_buddy.h> #include "xe_device_types.h" #include "xe_svm.h" +#include "xe_migrate.h" +#include "xe_ttm_vram_mgr_types.h" +#include "xe_assert.h" +/** +* struct xe_svm_block_meta - svm uses this data structure to manage each
drivers/gpu/drm/xe/xe_svm_devmem.c line 23
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_devmem.c +1
-0+* block allocated from drm buddy. This will be set to the drm_buddy_block's
Quoted 16 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +4
-0+* private field. +* +* @lru: used to link this block to drm's lru lists. This will be replace +* with struct drm_lru_entity later. drivers/gpu/drm/xe/xe_svm_devmem.c line 37
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+* @tile: tile from which we allocated this block +* @bitmap: A bitmap of each page in this block. 1 means this page is used, +* 0 means this page is idle. When all bits of this block are 0, it is time +* to return this block to drm buddy subsystem. +*/ +struct xe_svm_block_meta { +struct list_head lru; +struct xe_tile *tile; +unsigned long bitmap[]; +};This looks not needed to me but admittedly haven't looked at the LRU stuff.
Quoted 2 lines
I am moving to page granularity memory eviction, so we can either use the lru in the struct page itself, or I will have to introduce some other data structure which have a lru.
You almost certainly cannot use struct page, pretty sure that will not be well recieved putting a subsystem memory management feature into the core memory management.
I'm going to say this again, I disagree with this design decession as I do not think using a BO / or migration + eviction at allocation grainularity should be dismissed. Using a BO offers eviction more or less for free and possibly dma-buf reuse for multi-GPU. Please study my PoC [1]. It has SVM full featured and largely working.
If you have a different design, great. But I'd expect the next post to have feature parity, thorough testing, and well thought out design choices with explainations of said design choices beyond someone in the community said something so I am doing it this way (i.e. deep thought and understanding of how all the pieces fit together and why this design was chosen).
[1] https://gitlab.freedesktop.org/mbrost/xe-kernel-driver-svm-post/-/tree/post?ref_type=heads
Quoted 2 lines
Yes, I removed
block_metain v3.
I don't like speculation, said this many times. Let's see what rev3 looks like and will review then.
Quoted 4 lines
Quoted 4 lines
I am thinking roughly...
- I think we drop all this special tracking (kill
xe_svm_block_meta)
Quoted 9 lines
Agreed.
Quoted 2 lines
- Have functions to allocate / free the buddy blocks, store buddy blocks in userptr
Why we need to store buddy blocks in userptr? If you do this, you would need another
block_listin userptr.I currently use the struct page's
zone_device_datato point tobuddy_block. It seems work for me.
It doesn't. See my working PoC [1] zdd structure, almost certainly something like that will be needed for a variety of reasons. Also if you allocate a buddy block then it really isn't page grainularity nor does offer an advantage over BO wrt page grainularity. I stated this multiple time in lengthly explainations and not heard a reasonable argument against this.
Quoted 10 lines
Quoted 1 line
- Blocks are allocated before migration to VRAM
Agreed
Quoted 3 lines
- Blocks can be freed on either CPU fault after migration or on VMA destroy (probably depends on madvive hints for VMA where we free blocks)
My current understanding is, once device pages is allocated and handed over to core mm/hmm, driver doesn't need to worry about the life cycle of device pages, i.e., core mm/hmm will take care by calling back to
page_freevfunc.
Yes, we definitely need to the page_free vfunc callback implemented. This was a misunderstanding on my part, certainly you can make it work without this but it is not correct.
Quoted 6 lines
Quoted 3 lines
- Blocks allocated / freed at ia chunk (
xe_vmain this code) granularity (conceptually the same if we switch to 1 to N ratio betweenxe_vma&pt_state)As said, I am moving to page granularity, moving away from
xe_vmagranularity, to address concerns from the drm community discussion.
That's not what the community is saying or it is not at least how I interrupt this. The community is saying on a fault, allocate at a page granularity of the CPU mapping. e.g. If the CPU mapping is 4k, allocate 4k. If the CPU mapping is 2M, allocate 2M. There quite a bit of core MM work that would need to be done for this to properly work though, namly the migration layer splits TPH into smaller pages. I think that should fixed (e.g. teach core MM to understand device private pages that are larger than 4k, likewise coherent pages). Conceptually I think this is the right approach but would take quite a bit of work. But with that, if we layer the code properly only the DRM layer needs to be aware of this. e.g. We could build SVM without this like I do in my PoC and then switch over to this model once we complete the core MM work is done mostly only modifying the DRM layer. This would eliminate the partial unmapping / invalidation scenarios which my PoC implenents and provides documentation for.
Below general comment:
If we post something that we view as good and working, community feedback can be pushed back on with a justifiable design. Blindly doing things without understanding (e.g. community said do it this way) of the larger issues is not a great way to work. Nor is posting untested code with a partial implementation.
Quoted 6 lines
Quoted 1 line
-
block->private== memory region so we can get pfn from blockIn v3 code,
block->privateis not used. Will use it if needed.Each struct page has a pgmap pointer which points to xe_mem_region's pgmap memory. We can use this information to get a page/fpn's memory region.
That probably works but there may be scenario where you need to from block to mr in which the device pages are not readily available. You always can go from block -> pages -> mr but it may be advantageous to short circuit that with block -> mr, especially in hot (performance critial) paths.
Matt
Quoted 155 lines
Quoted 8 lines
- When we need
migrate_pfnswe loop over buddy blocks populating migrate.dstAlso I noticed the drm_buddy_* calls in this file are not protected by a lock, we will need that. Currently it is
tile->mem.vram_mgr->lockin the VRAM mgr code, we either need to reach into there or move this lock to common place so the VRAM manager and block allocations for SVM don't race with each other.Ok, will add this lock. Lets keep the lock in
vram_mgr->lockfor now for simplicityOak
Quoted 141 lines
Matt
Quoted 34 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +26
-1static vm_fault_t xe_devm_migrate_to_ram (struct vm_fault *vmf) {return 0 ; } -static void xe_devm_page_free(struct page *page) +static u64 block_offset_to_pfn (struct xe_mem_region *mr,u64 offset) +{ +/** DRM buddy's block offset is 0-based*/ + offset += mr->hpa_base; + +return PHYS_PFN (offset); +} + +/** FIXME: we locked page by calling zone_device_page_init +* in xe_devm_alloc_pages. Should we unlock pages here? +*/ +static void free_block (struct drm_buddy_block *block) +{ +struct xe_svm_block_meta *meta = + (struct xe_svm_block_meta *)block->private; +struct xe_tile *tile = meta->tile; +struct drm_buddy *mm = &tile->mem.vram_mgr->mm; + +kfree (block->private); +drm_buddy_free_block (mm, block); +} + +void xe_devm_page_free (struct page *page) +{ +struct drm_buddy_block *block = + (struct drm_buddy_block *)page- zone_device_data;drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +9
-0+struct xe_svm_block_meta *meta = + (struct xe_svm_block_meta *)block- private; +struct xe_tile *tile = meta->tile; +struct xe_mem_region *mr = &tile->mem.vram; +struct drm_buddy *mm = &tile->mem.vram_mgr->mm; +u64 size =drm_buddy_block_size (mm, block); +u64 pages_per_block = size >>PAGE_SHIFT ; +u64 block_pfn_first = + block_offset_to_pfn(mr,
drm_buddy_block_offset(block));Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +12
-0+u64 page_pfn =page_to_pfn (page); +u64 i = page_pfn - block_pfn_first; + +xe_assert (tile->xe, i < pages_per_block); +clear_bit (i, meta->bitmap); +if (bitmap_empty (meta->bitmap, pages_per_block)) +free_block (block); +} + +/** +* xe_devm_alloc_pages() - allocate device pages from buddy allocator +* drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+* @xe_tile: which tile to allocate device memory from +* @npages: how many pages to allocate +* @blocks: used to return the allocated blocks +* @pfn: used to return the pfn of all allocated pages. Must be big enough +* to hold at @npages entries. +* +* This function allocate blocks of memory from drm buddy allocator, and +* performs initialization work: set struct page::zone_device_data to point +* to the memory block; set/initialize drm_buddy_block::private field; +* lock_page for each page allocated; add memory block to lru managers lru
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +7
-0+* list - this is TBD. +* +* return: 0 on success +* error code otherwise +*/ +int xe_devm_alloc_pages(struct xe_tile *tile, + unsigned long npages,drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+struct list_head *blocks, + unsigned long *pfn) +{ + struct drm_buddy *mm = &tile->mem.vram_mgr->mm; +struct drm_buddy_block *block, *tmp; +u64 size = npages <<PAGE_SHIFT ; +int ret =0 , i, j =0 ; + + ret = drm_buddy_alloc_blocks(mm, 0, mm->size, size, PAGE_SIZE, + blocks,
DRM_BUDDY_TOPDOWN_ALLOCATION);Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+ +if (unlikely (ret)) +return ret; + +list_for_each_entry_safe (block, tmp, blocks, link) { +struct xe_mem_region *mr = &tile->mem.vram; +u64 block_pfn_first, pages_per_block; +struct xe_svm_block_meta *meta; +u32 meta_size; +drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+ size =drm_buddy_block_size (mm, block); + pages_per_block = size >>PAGE_SHIFT ; + meta_size =BITS_TO_BYTES (pages_per_block) + +sizeof (struct xe_svm_block_meta ); + meta =kzalloc (meta_size,GFP_KERNEL ); +bitmap_fill (meta->bitmap, pages_per_block); + meta->tile = tile; + block->private = meta; + block_pfn_first = + block_offset_to_pfn(mr,
drm_buddy_block_offset(block));Quoted 31 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +24
-0+for (i =0 ; i < pages_per_block; i++) { +struct page *page; + + pfn[j++] = block_pfn_first + i; + page =pfn_to_page (block_pfn_first + i); +/**Lock page per hmm requirement, see hmm.rst.*/ +zone_device_page_init (page); + page->zone_device_data = block; + } + } + +return ret; +} + +/** +* xe_devm_free_blocks() - free all memory blocks +* +* @blocks: memory blocks list head +*/ +void xe_devm_free_blocks (struct list_head *blocks) { +struct drm_buddy_block *block, *tmp; + +list_for_each_entry_safe (block, tmp, blocks, link) +free_block (block); }static const struct
dev_pagemap_opsxe_devm_pagemap_ops= {
--
2.26.3
RE: [v2 22/31] drm/xe/svm: implement functions to allocate and free device memory
Hi Matt,
Quoted 126 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, June 5, 2024 7:37 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 22/31] drm/xe/svm: implement functions to allocate and free device memoryQuoted 9 lines · Oak
Hi Matt,
Quoted 7 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 6:24 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected];Welty,
Quoted 4 lines
Quoted 4 lines
Brian <[email protected]>
Subject: Re: [v2 22/31] drm/xe/svm: implement functions to allocate and free device memoryQuoted 8 lines
Quoted 8 lines · Oak Zeng
Quoted 1 line
Function
xe_devm_alloc_pagesallocate pages from drm buddy andperform
Quoted 5 lines
house keeping work for all the pages allocated, such as get a page refcount, keep a bitmap of all pages to denote whether a page is in use, put pages to a drm lru list for eviction purpose.
Function
xe_devm_free_blocksreturn list of memory blocks to drmbuddy
Quoted 70 lines
Quoted 70 lines
Quoted 10 lines
allocator.
Function
xe_devm_free_pageis a call back function from hmm layer. It is called whenever a page's refcount reaches to 1. This function clears the bit of this page in the bitmap. If all the bits in the bitmap is cleared, it means all the pages have been freed, we return all the pages in this memory block back to drm buddy.Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana VishwanathapuraQuoted 1 line
Signed-off-by Niranjana VishwanathapuraQuoted 6 lines
Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/xe_svm.h | 7 ++ drivers/gpu/drm/xe/xe_svm_devmem.c | 147++++++++++++++++++++++++++++-
See comments about file in previous patches, they apply here too.
Review
Quoted 3 lines
2 files changed, 152 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm.hdrivers/gpu/drm/xe/xe_svm.h static inline struct xe_mem_region
Quoted 4 lines
index
624c1581f8ba..92a3ee90d5a7100644drivers/gpu/drm/xe/xe_svm.h +0
-0@@ -46,4 +46,11 @@ static inline struct xe_mem_region*xe_page_to_mem_region(struct page *page)
drivers/gpu/drm/xe/xe_svm.h static inline struct xe_mem_region · line 46
Quoted 1 line
drivers/gpu/drm/xe/xe_svm.h +0
-0return container_of(page->pgmap, struct xe_mem_region,pagemap);
Review
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm.h +7
-0} +int xe_devm_alloc_pages (struct xe_tile *tile, +unsigned long npages, +struct list_head *blocks, +unsigned long *pfn); + +void xe_devm_free_blocks (struct list_head *blocks); +void xe_devm_page_free (struct page *page);#endif (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_devmem.cQuoted 13 lines
index
31af56e8285a..5ba0cd9a70b0100644drivers/gpu/drm/xe/xe_svm_devmem.c +5
-1@@ -5,18 +5,161 @@#include <linux/mm_types.h> #include <linux/sched/mm.h> - +#include <linux/gfp.h> +#include <linux/migrate.h> +#include <linux/dma-mapping.h> +#include <linux/dma-fence.h> +#include <linux/bitops.h> drivers/gpu/drm/xe/xe_svm_devmem.c line 22
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +7
-0+#include <linux/bitmap.h> +#include <drm/drm_buddy.h> #include "xe_device_types.h" #include "xe_svm.h" +#include "xe_migrate.h" +#include "xe_ttm_vram_mgr_types.h" +#include "xe_assert.h" +/** +* struct xe_svm_block_meta - svm uses this data structure to manage each
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_devmem.c line 23
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_devmem.c +1
-0+* block allocated from drm buddy. This will be set to the drm_buddy_block's
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +6
-0+* private field. +* +* @lru: used to link this block to drm's lru lists. This will be replace +* with struct drm_lru_entity later. +* @tile: tile from which we allocated this block +* @bitmap: A bitmap of each page in this block. 1 means this page is used,
Quoted 10 lines
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c line 37
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +8
-0+* 0 means this page is idle. When all bits of this block are 0, it is time +* to return this block to drm buddy subsystem. +*/ +struct xe_svm_block_meta { +struct list_head lru; +struct xe_tile *tile; +unsigned long bitmap[]; +};This looks not needed to me but admittedly haven't looked at the LRU
stuff.
Quoted 1 line
Quoted 8 lines
Quoted 1 line
I am moving to page granularity memory eviction, so we can either use the
lru in the struct page itself, or I will have to introduce some other data structure which have a lru.
Quoted 1 line
You almost certainly cannot use struct page, pretty sure that will not be well recieved putting a subsystem memory management feature into the core memory management.
The reason I was think of struct page's lru because, the lru field is not used right now for device pages...the lru field is used for system pages only right now. So I don't see a conflict there. It is just a member not used by core mm and we can make use of it if we want.
But it is very possible that we don't use struct page's lru. In this series https://patchwork.freedesktop.org/patch/565501/?series=125879&rev=1, I have some drm_lru_entity concept. I need to go back to that series to work out the page granularity vram eviction part.
Quoted 6 lines
I'm going to say this again, I disagree with this design decession as I do not think using a BO / or migration + eviction at allocation grainularity should be dismissed. Using a BO offers eviction more or less for free and possibly dma-buf reuse for multi-GPU. Please study my PoC [1]. It has SVM full featured and largely working.
It is completely fine that people disagree with a design. Removing BO from system allocator was aligned with DRM community long time ago. The main problem of BO is the memory eviction and migration is not page granularity.
I know in v2 I am not truly page granularity either. But I am moving to true page granularity in v3, the memory allocation, free, migration and eviction are all at page granularity. It is very similar to the scheme Linux core mm has. There are much more work at buddy allocator and eviction lru, compared to the BO approach.
Please also note, when said page granularity, it could be one page, but can also multiple pages.
Quoted 7 lines
If you have a different design, great. But I'd expect the next post to have feature parity, thorough testing, and well thought out design choices with explainations of said design choices beyond someone in the community said something so I am doing it this way (i.e. deep thought and understanding of how all the pieces fit together and why this design was chosen).
I understand feature parity and testing are important. The previous posts (v1 and v2) were mainly to get some feedback on the design. Since we are at the 3rd respin of this series, it is probably a good idea to do more test before post.
Quoted 26 lines
[1] https://gitlab.freedesktop.org/mbrost/xe-kernel-driver-svm-post/-
/tree/post?ref_type=headsQuoted 2 lines
Yes, I removed
block_metain v3.I don't like speculation, said this many times. Let's see what rev3 looks like and will review then.
Quoted 10 lines
Quoted 4 lines
I am thinking roughly...
- I think we drop all this special tracking (kill
xe_svm_block_meta)Agreed.
Quoted 2 lines
- Have functions to allocate / free the buddy blocks, store buddy blocks in userptr
Why we need to store buddy blocks in userptr? If you do this, you would
need another
block_listin userptr.Quoted 2 lines
I currently use the struct page's
zone_device_datato point tobuddy_block.It seems work for me.
Quoted 1 line
Quoted 6 lines
It doesn't. See my working PoC [1] zdd structure, almost certainly something like that will be needed for a variety of reasons. Also if you allocate a buddy block then it really isn't page grainularity nor does offer an advantage over BO wrt page grainularity. I stated this multiple time in lengthly explainations and not heard a reasonable argument against this.
It is true that the drm buddy interface is not page granularity. But this should be mainly looked as a drawback of the buddy interface. If you look at the core mm buddy allocator, the interface is page centric. We are working on the drm buddy allocator to improve in this perspective.
Quoted 23 lines
Quoted 10 lines
Quoted 2 lines
- Blocks are allocated before migration to VRAM
Agreed
Quoted 3 lines
- Blocks can be freed on either CPU fault after migration or on VMA destroy (probably depends on madvive hints for VMA where we free blocks)
My current understanding is, once device pages is allocated and handed
over to core mm/hmm, driver doesn't need to worry about the life cycle of device pages, i.e., core mm/hmm will take care by calling back to
page_freevfunc.Quoted 1 line
Yes, we definitely need to the
page_freevfunc callback implemented. This was a misunderstanding on my part, certainly you can make it work without this but it is not correct.Quoted 4 lines
Quoted 3 lines
- Blocks allocated / freed at ia chunk (
xe_vmain this code) granularity (conceptually the same if we switch to 1 to N ratio betweenxe_vma&pt_state)
Quoted 8 lines
Quoted 1 line
As said, I am moving to page granularity, moving away from
xe_vmagranularity, to address concerns from the drm community discussion.
Quoted 1 line
That's not what the community is saying or it is not at least how I interrupt this. The community is saying on a fault, allocate at a page granularity of the CPU mapping. e.g. If the CPU mapping is 4k, allocate 4k. If the CPU mapping is 2M, allocate 2M.
I guess by CPU mapping you meant CPU vma.
I think we can create a mmu notifier to match the range of the cpu vma.
Regarding how much gpu memory be allocated and migrated, my current thinking is, this can be decided by a memory hint attribute. i.e., if CPU vma is 16M, but memory attribute say 2M, we can allocate/migrate 2M. if memory attributes says 32M, we will migrate 16M.
So I don't understand why you want to allocate the same size as CPU vma...
There quite a bit of core MM
Quoted 11 lines
work that would need to be done for this to properly work though, namly the migration layer splits TPH into smaller pages. I think that should fixed (e.g. teach core MM to understand device private pages that are larger than 4k, likewise coherent pages). Conceptually I think this is the right approach but would take quite a bit of work. But with that, if we layer the code properly only the DRM layer needs to be aware of this. e.g. We could build SVM without this like I do in my PoC and then switch over to this model once we complete the core MM work is done mostly only modifying the DRM layer. This would eliminate the partial unmapping / invalidation scenarios which my PoC implenents and provides documentation for.
I don't quite follow here. Afaik, THP is only enabled for linux core mm, managing system memory. I am not aware of it in xekmd. By THP do you mean THP in xekmd? Or huge page (compound page, 2M or 1G etc) to back up cpu vma in linux core mm?
Quoted 17 lines
Below general comment:
If we post something that we view as good and working, community feedback can be pushed back on with a justifiable design. Blindly doing things without understanding (e.g. community said do it this way) of the larger issues is not a great way to work. Nor is posting untested code with a partial implementation.Quoted 5 lines
Quoted 1 line
-
block->private== memory region so we can get pfn from blockIn v3 code,
block->privateis not used. Will use it if needed.Each struct page has a pgmap pointer which points to xe_mem_region's
pgmap memory. We can use this information to get a page/fpn's memory region.
Quoted 1 line
Quoted 5 lines
That probably works but there may be scenario where you need to from block to mr in which the device pages are not readily available. You always can go from block -> pages -> mr but it may be advantageous to short circuit that with block -> mr, especially in hot (performance critial) paths.
In my v3, I don't have a need for block to mr. but it is a good idea to use block private for this purpose.
Oak
Quoted 164 lines
Matt
Quoted 10 lines
Quoted 8 lines
- When we need
migrate_pfnswe loop over buddy blocks populating migrate.dstAlso I noticed the drm_buddy_* calls in this file are not protected by a lock, we will need that. Currently it is
tile->mem.vram_mgr->lockin the VRAM mgr code, we either need to reach into there or move this lock to common place so the VRAM manager and block allocations for SVM don't race with each other.Ok, will add this lock. Lets keep the lock in
vram_mgr->lockfor now forsimplicity
Quoted 52 lines
Oak
Quoted 48 lines
Matt
Quoted 34 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +26
-1static vm_fault_t xe_devm_migrate_to_ram (struct vm_fault *vmf) {return 0 ; } -static void xe_devm_page_free(struct page *page) +static u64 block_offset_to_pfn (struct xe_mem_region *mr,u64 offset) +{ +/** DRM buddy's block offset is 0-based*/ + offset += mr->hpa_base; + +return PHYS_PFN (offset); +} + +/** FIXME: we locked page by calling zone_device_page_init +* in xe_devm_alloc_pages. Should we unlock pages here? +*/ +static void free_block (struct drm_buddy_block *block) +{ +struct xe_svm_block_meta *meta = + (struct xe_svm_block_meta *)block->private; +struct xe_tile *tile = meta->tile; +struct drm_buddy *mm = &tile->mem.vram_mgr->mm; + +kfree (block->private); +drm_buddy_free_block (mm, block); +} + +void xe_devm_page_free (struct page *page) +{ +struct drm_buddy_block *block = + (struct drm_buddy_block *)page- zone_device_data;drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +9
-0+struct xe_svm_block_meta *meta = + (struct xe_svm_block_meta *)block- private; +struct xe_tile *tile = meta->tile; +struct xe_mem_region *mr = &tile->mem.vram; +struct drm_buddy *mm = &tile->mem.vram_mgr->mm; +u64 size =drm_buddy_block_size (mm, block); +u64 pages_per_block = size >>PAGE_SHIFT ; +u64 block_pfn_first = + block_offset_to_pfn(mr,
drm_buddy_block_offset(block));Quoted 11 lines
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +11
-0+u64 page_pfn =page_to_pfn (page); +u64 i = page_pfn - block_pfn_first; + +xe_assert (tile->xe, i < pages_per_block); +clear_bit (i, meta->bitmap); +if (bitmap_empty (meta->bitmap, pages_per_block)) +free_block (block); +} + +/** +* xe_devm_alloc_pages() - allocate device pages from buddy allocator
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +5
-0+* +* @xe_tile: which tile to allocate device memory from +* @npages: how many pages to allocate +* @blocks: used to return the allocated blocks +* @pfn: used to return the pfn of all allocated pages. Must be big enough
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +3
-0+* to hold at @npages entries. +* +* This function allocate blocks of memory from drm buddy allocator, and
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_devmem.c line 99
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_devmem.c +1
-0+* performs initialization work: set struct page::zone_device_data to point
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_devmem.c line 101
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +2
-0+* to the memory block; set/initialize drm_buddy_block::private field; +* lock_page for each page allocated; add memory block to lru managers
Quoted 71 lines
Quoted 71 lines
lru
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +7
-0+* list - this is TBD. +* +* return: 0 on success +* error code otherwise +*/ +int xe_devm_alloc_pages(struct xe_tile *tile, + unsigned long npages,drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+struct list_head *blocks, + unsigned long *pfn) +{ + struct drm_buddy *mm = &tile->mem.vram_mgr->mm; +struct drm_buddy_block *block, *tmp; +u64 size = npages <<PAGE_SHIFT ; +int ret =0 , i, j =0 ; + + ret = drm_buddy_alloc_blocks(mm, 0, mm->size, size, PAGE_SIZE, + blocks,
DRM_BUDDY_TOPDOWN_ALLOCATION);Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+ +if (unlikely (ret)) +return ret; + +list_for_each_entry_safe (block, tmp, blocks, link) { +struct xe_mem_region *mr = &tile->mem.vram; +u64 block_pfn_first, pages_per_block; +struct xe_svm_block_meta *meta; +u32 meta_size; +drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+ size =drm_buddy_block_size (mm, block); + pages_per_block = size >>PAGE_SHIFT ; + meta_size =BITS_TO_BYTES (pages_per_block) + +sizeof (struct xe_svm_block_meta ); + meta =kzalloc (meta_size,GFP_KERNEL ); +bitmap_fill (meta->bitmap, pages_per_block); + meta->tile = tile; + block->private = meta; + block_pfn_first = + block_offset_to_pfn(mr,
drm_buddy_block_offset(block));Quoted 31 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +24
-0+for (i =0 ; i < pages_per_block; i++) { +struct page *page; + + pfn[j++] = block_pfn_first + i; + page =pfn_to_page (block_pfn_first + i); +/**Lock page per hmm requirement, see hmm.rst.*/ +zone_device_page_init (page); + page->zone_device_data = block; + } + } + +return ret; +} + +/** +* xe_devm_free_blocks() - free all memory blocks +* +* @blocks: memory blocks list head +*/ +void xe_devm_free_blocks (struct list_head *blocks) { +struct drm_buddy_block *block, *tmp; + +list_for_each_entry_safe (block, tmp, blocks, link) +free_block (block); }static const struct
dev_pagemap_opsxe_devm_pagemap_ops= {
--
2.26.3
Matthew Brost
maintainerRe: [v2 22/31] drm/xe/svm: implement functions to allocate and free device memory
Quoted 133 lines · Oak
Hi Matt,
Quoted 131 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, June 5, 2024 7:37 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 22/31] drm/xe/svm: implement functions to allocate and free device memoryQuoted 9 lines · Oak
Hi Matt,
Quoted 7 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 6:24 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected];Welty,
Quoted 4 lines
Quoted 4 lines
Brian <[email protected]>
Subject: Re: [v2 22/31] drm/xe/svm: implement functions to allocate and free device memoryQuoted 8 lines
Quoted 8 lines · Oak Zeng
Quoted 1 line
Function
xe_devm_alloc_pagesallocate pages from drm buddy andperform
Quoted 5 lines
house keeping work for all the pages allocated, such as get a page refcount, keep a bitmap of all pages to denote whether a page is in use, put pages to a drm lru list for eviction purpose.
Function
xe_devm_free_blocksreturn list of memory blocks to drmbuddy
Quoted 70 lines
Quoted 70 lines
Quoted 10 lines
allocator.
Function
xe_devm_free_pageis a call back function from hmm layer. It is called whenever a page's refcount reaches to 1. This function clears the bit of this page in the bitmap. If all the bits in the bitmap is cleared, it means all the pages have been freed, we return all the pages in this memory block back to drm buddy.Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana VishwanathapuraQuoted 1 line
Signed-off-by Niranjana VishwanathapuraQuoted 6 lines
Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/xe_svm.h | 7 ++ drivers/gpu/drm/xe/xe_svm_devmem.c | 147++++++++++++++++++++++++++++-
See comments about file in previous patches, they apply here too.
Review
Quoted 3 lines
2 files changed, 152 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm.hdrivers/gpu/drm/xe/xe_svm.h static inline struct xe_mem_region
Quoted 4 lines
index
624c1581f8ba..92a3ee90d5a7100644drivers/gpu/drm/xe/xe_svm.h +0
-0@@ -46,4 +46,11 @@ static inline struct xe_mem_region*xe_page_to_mem_region(struct page *page)
drivers/gpu/drm/xe/xe_svm.h static inline struct xe_mem_region · line 46
Quoted 1 line
drivers/gpu/drm/xe/xe_svm.h +0
-0return container_of(page->pgmap, struct xe_mem_region,pagemap);
Review
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm.h +7
-0} +int xe_devm_alloc_pages (struct xe_tile *tile, +unsigned long npages, +struct list_head *blocks, +unsigned long *pfn); + +void xe_devm_free_blocks (struct list_head *blocks); +void xe_devm_page_free (struct page *page);#endif (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_devmem.cQuoted 13 lines
index
31af56e8285a..5ba0cd9a70b0100644drivers/gpu/drm/xe/xe_svm_devmem.c +5
-1@@ -5,18 +5,161 @@#include <linux/mm_types.h> #include <linux/sched/mm.h> - +#include <linux/gfp.h> +#include <linux/migrate.h> +#include <linux/dma-mapping.h> +#include <linux/dma-fence.h> +#include <linux/bitops.h> drivers/gpu/drm/xe/xe_svm_devmem.c line 22
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +7
-0+#include <linux/bitmap.h> +#include <drm/drm_buddy.h> #include "xe_device_types.h" #include "xe_svm.h" +#include "xe_migrate.h" +#include "xe_ttm_vram_mgr_types.h" +#include "xe_assert.h" +/** +* struct xe_svm_block_meta - svm uses this data structure to manage each
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_devmem.c line 23
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_devmem.c +1
-0+* block allocated from drm buddy. This will be set to the drm_buddy_block's
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +6
-0+* private field. +* +* @lru: used to link this block to drm's lru lists. This will be replace +* with struct drm_lru_entity later. +* @tile: tile from which we allocated this block +* @bitmap: A bitmap of each page in this block. 1 means this page is used,
Quoted 10 lines
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c line 37
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +8
-0+* 0 means this page is idle. When all bits of this block are 0, it is time +* to return this block to drm buddy subsystem. +*/ +struct xe_svm_block_meta { +struct list_head lru; +struct xe_tile *tile; +unsigned long bitmap[]; +};This looks not needed to me but admittedly haven't looked at the LRU
stuff.
Quoted 2 lines
I am moving to page granularity memory eviction, so we can either use the
lru in the struct page itself, or I will have to introduce some other data structure which have a lru.
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
You almost certainly cannot use struct page, pretty sure that will not be well recieved putting a subsystem memory management feature into the core memory management.
The reason I was think of struct page's lru because, the lru field is not used right now for device pages...the lru field is used for system pages only right now. So I don't see a conflict there. It is just a member not used by core mm and we can make use of it if we want.
If the core implementation changes, then are you broken. If it's not an explicitly provided private field to the upper layers, then no, I don't think you should just use it. At least, that's my understanding.
Quoted 2 lines
But it is very possible that we don't use struct page's lru. In this series https://patchwork.freedesktop.org/patch/565501/?series=125879&rev=1, I have some
drm_lru_entityconcept. I need to go back to that series to work out the page granularity vram eviction part.
I've seen that series and it raises some pretty serious concerns which I won't get into here.
Quoted 10 lines
Quoted 7 lines
I'm going to say this again, I disagree with this design decession as I do not think using a BO / or migration + eviction at allocation grainularity should be dismissed. Using a BO offers eviction more or less for free and possibly dma-buf reuse for multi-GPU. Please study my PoC [1]. It has SVM full featured and largely working.
It is completely fine that people disagree with a design. Removing BO from system allocator was aligned with DRM community long time ago. The main problem of BO is the memory eviction and migration is not page granularity.
Of course, it's okay to disagree. This is about how you're responding. I'm not buying, or even hearing an argument about how using a buddy allocation solves the page granularity problem. Simply stating something as a fact without providing examples of how it solves the problem means nothing. This is a continued pattern of behavior which I interrupt as feedback being ignored.
Furthermore, I don't believe the VRAM backing actually has anything to do with page migration. I see this as a core MM and DRM layer issue rather than a decision made at the driver level regarding the VRAM backing store. I'll elaborate on this below.
My concern here is that a design decision is being made without valid reasoning. If I hear valid reasoning, of course, I'm open to other ideas.
I wasn't involved the discussions a long time ago, I am involved now and have an opinion on a Xe level design choice.
Quoted 42 lines
I know in v2 I am not truly page granularity either. But I am moving to true page granularity in v3, the memory allocation, free, migration and eviction are all at page granularity. It is very similar to the scheme Linux core mm has. There are much more work at buddy allocator and eviction lru, compared to the BO approach.
Please also note, when said page granularity, it could be one page, but can also multiple pages.
Quoted 7 lines
If you have a different design, great. But I'd expect the next post to have feature parity, thorough testing, and well thought out design choices with explainations of said design choices beyond someone in the community said something so I am doing it this way (i.e. deep thought and understanding of how all the pieces fit together and why this design was chosen).
I understand feature parity and testing are important. The previous posts (v1 and v2) were mainly to get some feedback on the design. Since we are at the 3rd respin of this series, it is probably a good idea to do more test before post.
Quoted 26 lines
[1] https://gitlab.freedesktop.org/mbrost/xe-kernel-driver-svm-post/-
/tree/post?ref_type=headsQuoted 2 lines
Yes, I removed
block_metain v3.I don't like speculation, said this many times. Let's see what rev3 looks like and will review then.
Quoted 10 lines
Quoted 4 lines
I am thinking roughly...
- I think we drop all this special tracking (kill
xe_svm_block_meta)Agreed.
Quoted 2 lines
- Have functions to allocate / free the buddy blocks, store buddy blocks in userptr
Why we need to store buddy blocks in userptr? If you do this, you would
need another
block_listin userptr.Quoted 2 lines
I currently use the struct page's
zone_device_datato point tobuddy_block.It seems work for me.
Quoted 1 line
Quoted 9 lines
Quoted 6 lines
It doesn't. See my working PoC [1] zdd structure, almost certainly something like that will be needed for a variety of reasons. Also if you allocate a buddy block then it really isn't page grainularity nor does offer an advantage over BO wrt page grainularity. I stated this multiple time in lengthly explainations and not heard a reasonable argument against this.
It is true that the drm buddy interface is not page granularity. But this should be mainly looked as a drawback of the buddy interface. If you look at the core mm buddy allocator, the interface is page centric. We are working on the drm buddy allocator to improve in this perspective.
Let's put this another way: installing buddy_block to a zone_device_data doesn't work from a layering perspective. A DRM SVM layer cannot enforce a driver to use a buddy allocation for the VRAM backing — the VRAM backing needs to be opaque, with the driver making a choice of the backing store (BO, buddy allocation, or potentially something else).
Again, this is why in my PoC the VRAM allocation is a void * and a driver vfunc to signal release of the memory. I'm not saying I have this 100% correct; I probably don't. But conceptually, the VRAM store needs to be transparent to the DRM SVM layer. Any design that requires a specific type of backing store at the DRM level is not correct.
To give an example of this, neither a GEM BO nor a TTM BO is aware of the backing store. This is why TTM has memory managers which do the actual allocation/freeing of memory. Granted, it provides a TT manager for driver use for system memory, but that is just a reference implementation too — a driver can override it (at least that is my understanding). For VRAM allocations, the driver owns this. See xe_ttm_vram_mgr.c, which happens to be built on top of the buddy allocator — this is not a requirement.
Quoted 27 lines
Quoted 27 lines
Quoted 10 lines
Quoted 2 lines
- Blocks are allocated before migration to VRAM
Agreed
Quoted 3 lines
- Blocks can be freed on either CPU fault after migration or on VMA destroy (probably depends on madvive hints for VMA where we free blocks)
My current understanding is, once device pages is allocated and handed
over to core mm/hmm, driver doesn't need to worry about the life cycle of device pages, i.e., core mm/hmm will take care by calling back to
page_freevfunc.Quoted 1 line
Yes, we definitely need to the
page_freevfunc callback implemented. This was a misunderstanding on my part, certainly you can make it work without this but it is not correct.Quoted 5 lines
Quoted 3 lines
- Blocks allocated / freed at ia chunk (
xe_vmain this code) granularity (conceptually the same if we switch to 1 to N ratio betweenxe_vma&pt_state)As said, I am moving to page granularity, moving away from
xe_vmagranularity, to address concerns from the drm community discussion.
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
That's not what the community is saying or it is not at least how I interrupt this. The community is saying on a fault, allocate at a page granularity of the CPU mapping. e.g. If the CPU mapping is 4k, allocate 4k. If the CPU mapping is 2M, allocate 2M.
I guess by CPU mapping you meant CPU vma.
No. This would be when hmm_range_fault is called, the order size returned in the pfn. This is page-level granularity.
Quoted 2 lines
I think we can create a mmu notifier to match the range of the cpu vma.
That has nothing to do with page granularity. We should be able to create arbitrary sized notifiers which span many pages / cpu vma / gpu mappins. Your design likely should support that per feedback from Jason which I also happen to agree with.
Quoted 2 lines
Regarding how much gpu memory be allocated and migrated, my current thinking is, this can be decided by a memory hint attribute. i.e., if CPU vma is 16M, but memory attribute say 2M, we can allocate/migrate 2M. if memory attributes says 32M, we will migrate 16M.
By definition, this is not page granularity. If you want to migrate more CPU pages upon GPU fault (which we might), that is fine, but it needs to be done in a way that ensures allocations match CPU page sizes. So, upon CPU fault or MMU notifier event, we are operating on the CPU page size. Hopefully, a user is smart enough to use THP allocations in a performance-critical app. This is my view of page granularity. Maybe I am way off here, but I really don't think so.
I'm not arguing that page granularity is a must; in fact, I didn't implement it in my PoC. I'm saying that if we did, this is how it would look.
Quoted 2 lines
So I don't understand why you want to allocate the same size as CPU vma...
I'm not saying that. See above.
Quoted 14 lines
There quite a bit of core MM
Quoted 11 lines
work that would need to be done for this to properly work though, namly the migration layer splits TPH into smaller pages. I think that should fixed (e.g. teach core MM to understand device private pages that are larger than 4k, likewise coherent pages). Conceptually I think this is the right approach but would take quite a bit of work. But with that, if we layer the code properly only the DRM layer needs to be aware of this. e.g. We could build SVM without this like I do in my PoC and then switch over to this model once we complete the core MM work is done mostly only modifying the DRM layer. This would eliminate the partial unmapping / invalidation scenarios which my PoC implenents and provides documentation for.
Quoted 2 lines
I don't quite follow here. Afaik, THP is only enabled for linux core mm, managing system memory. I am not aware of it in xekmd. By THP do you mean THP in xekmd? Or huge page (compound page, 2M or 1G etc) to back up cpu vma in linux core mm?
Page granularity would be matching the CPU page size in migrations and not spliting THP upon migration.
With all of the above considered, the VRAM backing actually has nothing to do with page granularity. That is, the allocation size of the backing store is the key; the implementation of the backing store is immaterial here.
I concede that using a BO for every 4k allocation will be quite wasteful, but a buddy allocation wouldn't be all that much better either. Really, it is up to the user to ensure 2M (THP) allocations are used if supporting page granularity. If we can't trust the user to do so, then don't support page granularity and create arbitrary sizes based on the CPU VMA, as I do in my PoC.
Matt
Quoted 192 lines
Quoted 18 lines
Below general comment:
If we post something that we view as good and working, community feedback can be pushed back on with a justifiable design. Blindly doing things without understanding (e.g. community said do it this way) of the larger issues is not a great way to work. Nor is posting untested code with a partial implementation.Quoted 5 lines
Quoted 1 line
-
block->private== memory region so we can get pfn from blockIn v3 code,
block->privateis not used. Will use it if needed.Each struct page has a pgmap pointer which points to xe_mem_region's
pgmap memory. We can use this information to get a page/fpn's memory region.
Quoted 1 line
Quoted 5 lines
That probably works but there may be scenario where you need to from block to mr in which the device pages are not readily available. You always can go from block -> pages -> mr but it may be advantageous to short circuit that with block -> mr, especially in hot (performance critial) paths.
In my v3, I don't have a need for block to mr. but it is a good idea to use block private for this purpose.
Oak
Quoted 164 lines
Matt
Quoted 10 lines
Quoted 8 lines
- When we need
migrate_pfnswe loop over buddy blocks populating migrate.dstAlso I noticed the drm_buddy_* calls in this file are not protected by a lock, we will need that. Currently it is
tile->mem.vram_mgr->lockin the VRAM mgr code, we either need to reach into there or move this lock to common place so the VRAM manager and block allocations for SVM don't race with each other.Ok, will add this lock. Lets keep the lock in
vram_mgr->lockfor now forsimplicity
Quoted 52 lines
Oak
Quoted 48 lines
Matt
Quoted 34 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +26
-1static vm_fault_t xe_devm_migrate_to_ram (struct vm_fault *vmf) {return 0 ; } -static void xe_devm_page_free(struct page *page) +static u64 block_offset_to_pfn (struct xe_mem_region *mr,u64 offset) +{ +/** DRM buddy's block offset is 0-based*/ + offset += mr->hpa_base; + +return PHYS_PFN (offset); +} + +/** FIXME: we locked page by calling zone_device_page_init +* in xe_devm_alloc_pages. Should we unlock pages here? +*/ +static void free_block (struct drm_buddy_block *block) +{ +struct xe_svm_block_meta *meta = + (struct xe_svm_block_meta *)block->private; +struct xe_tile *tile = meta->tile; +struct drm_buddy *mm = &tile->mem.vram_mgr->mm; + +kfree (block->private); +drm_buddy_free_block (mm, block); +} + +void xe_devm_page_free (struct page *page) +{ +struct drm_buddy_block *block = + (struct drm_buddy_block *)page- zone_device_data;drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +9
-0+struct xe_svm_block_meta *meta = + (struct xe_svm_block_meta *)block- private; +struct xe_tile *tile = meta->tile; +struct xe_mem_region *mr = &tile->mem.vram; +struct drm_buddy *mm = &tile->mem.vram_mgr->mm; +u64 size =drm_buddy_block_size (mm, block); +u64 pages_per_block = size >>PAGE_SHIFT ; +u64 block_pfn_first = + block_offset_to_pfn(mr,
drm_buddy_block_offset(block));Quoted 11 lines
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +11
-0+u64 page_pfn =page_to_pfn (page); +u64 i = page_pfn - block_pfn_first; + +xe_assert (tile->xe, i < pages_per_block); +clear_bit (i, meta->bitmap); +if (bitmap_empty (meta->bitmap, pages_per_block)) +free_block (block); +} + +/** +* xe_devm_alloc_pages() - allocate device pages from buddy allocator
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +5
-0+* +* @xe_tile: which tile to allocate device memory from +* @npages: how many pages to allocate +* @blocks: used to return the allocated blocks +* @pfn: used to return the pfn of all allocated pages. Must be big enough
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +3
-0+* to hold at @npages entries. +* +* This function allocate blocks of memory from drm buddy allocator, and
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_devmem.c line 99
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_devmem.c +1
-0+* performs initialization work: set struct page::zone_device_data to point
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_devmem.c line 101
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +2
-0+* to the memory block; set/initialize drm_buddy_block::private field; +* lock_page for each page allocated; add memory block to lru managers
Quoted 71 lines
Quoted 71 lines
lru
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +7
-0+* list - this is TBD. +* +* return: 0 on success +* error code otherwise +*/ +int xe_devm_alloc_pages(struct xe_tile *tile, + unsigned long npages,drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+struct list_head *blocks, + unsigned long *pfn) +{ + struct drm_buddy *mm = &tile->mem.vram_mgr->mm; +struct drm_buddy_block *block, *tmp; +u64 size = npages <<PAGE_SHIFT ; +int ret =0 , i, j =0 ; + + ret = drm_buddy_alloc_blocks(mm, 0, mm->size, size, PAGE_SIZE, + blocks,
DRM_BUDDY_TOPDOWN_ALLOCATION);Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+ +if (unlikely (ret)) +return ret; + +list_for_each_entry_safe (block, tmp, blocks, link) { +struct xe_mem_region *mr = &tile->mem.vram; +u64 block_pfn_first, pages_per_block; +struct xe_svm_block_meta *meta; +u32 meta_size; +drivers/gpu/drm/xe/xe_svm_devmem.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+ size =drm_buddy_block_size (mm, block); + pages_per_block = size >>PAGE_SHIFT ; + meta_size =BITS_TO_BYTES (pages_per_block) + +sizeof (struct xe_svm_block_meta ); + meta =kzalloc (meta_size,GFP_KERNEL ); +bitmap_fill (meta->bitmap, pages_per_block); + meta->tile = tile; + block->private = meta; + block_pfn_first = + block_offset_to_pfn(mr,
drm_buddy_block_offset(block));Quoted 31 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +24
-0+for (i =0 ; i < pages_per_block; i++) { +struct page *page; + + pfn[j++] = block_pfn_first + i; + page =pfn_to_page (block_pfn_first + i); +/**Lock page per hmm requirement, see hmm.rst.*/ +zone_device_page_init (page); + page->zone_device_data = block; + } + } + +return ret; +} + +/** +* xe_devm_free_blocks() - free all memory blocks +* +* @blocks: memory blocks list head +*/ +void xe_devm_free_blocks (struct list_head *blocks) { +struct drm_buddy_block *block, *tmp; + +list_for_each_entry_safe (block, tmp, blocks, link) +free_block (block); }static const struct
dev_pagemap_opsxe_devm_pagemap_ops= {
--
2.26.3
Matthew Brost
maintainerRe: [v2 22/31] drm/xe/svm: implement functions to allocate and free device memory
In reply to Oak ZengQuoted 151 lines · Oak Zeng
Function
xe_devm_alloc_pagesallocate pages from drm buddy and perform house keeping work for all the pages allocated, such as get a page refcount, keep a bitmap of all pages to denote whether a page is in use, put pages to a drm lru list for eviction purpose.Function
xe_devm_free_blocksreturn list of memory blocks to drm buddy allocator.Function
xe_devm_free_pageis a call back function from hmm layer. It is called whenever a page's refcount reaches to 1. This function clears the bit of this page in the bitmap. If all the bits in the bitmap is cleared, it means all the pages have been freed, we return all the pages in this memory block back to drm buddy.Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana Vishwanathapura <[email protected]>Signed-off-by Niranjana Vishwanathapura <[email protected]>Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/xe_svm.h | 7 ++ drivers/gpu/drm/xe/xe_svm_devmem.c | 147 ++++++++++++++++++++++++++++- 2 files changed, 152 insertions(+), 2 deletions(-)drivers/gpu/drm/xe/xe_svm.h +7
-0@@ -46,4 +46,11 @@ static inline struct xe_mem_region *xe_page_to_mem_region(struct page *page) return container_of(page->pgmap, struct xe_mem_region, pagemap); } +int xe_devm_alloc_pages (struct xe_tile *tile, +unsigned long npages, +struct list_head *blocks, +unsigned long *pfn); + +void xe_devm_free_blocks (struct list_head *blocks); +void xe_devm_page_free (struct page *page);#endif drivers/gpu/drm/xe/xe_svm_devmem.c +91
-2@@ -5,18 +5,161 @@#include <linux/mm_types.h> #include <linux/sched/mm.h> - +#include <linux/gfp.h> +#include <linux/migrate.h> +#include <linux/dma-mapping.h> +#include <linux/dma-fence.h> +#include <linux/bitops.h> +#include <linux/bitmap.h> +#include <drm/drm_buddy.h> #include "xe_device_types.h" #include "xe_svm.h" +#include "xe_migrate.h" +#include "xe_ttm_vram_mgr_types.h" +#include "xe_assert.h" +/** +* struct xe_svm_block_meta - svm uses this data structure to manage each +* block allocated from drm buddy. This will be set to the drm_buddy_block's +* private field. +* +* @lru: used to link this block to drm's lru lists. This will be replace +* with struct drm_lru_entity later. +* @tile: tile from which we allocated this block +* @bitmap: A bitmap of each page in this block. 1 means this page is used, +* 0 means this page is idle. When all bits of this block are 0, it is time +* to return this block to drm buddy subsystem. +*/ +struct xe_svm_block_meta { +struct list_head lru; +struct xe_tile *tile; +unsigned long bitmap[]; +};static vm_fault_t xe_devm_migrate_to_ram (struct vm_fault *vmf) {return 0 ; } -static void xe_devm_page_free (struct page *page) +static u64 block_offset_to_pfn (struct xe_mem_region *mr,u64 offset) +{ +/** DRM buddy's block offset is 0-based*/ + offset += mr->hpa_base; + +return PHYS_PFN (offset); +} + +/** FIXME: we locked page by calling zone_device_page_init +* in xe_devm_alloc_pages. Should we unlock pages here? +*/ +static void free_block (struct drm_buddy_block *block) +{ +struct xe_svm_block_meta *meta = + (struct xe_svm_block_meta *)block->private; +struct xe_tile *tile = meta->tile; +struct drm_buddy *mm = &tile->mem.vram_mgr->mm; + +kfree (block->private); +drm_buddy_free_block (mm, block); +} + +void xe_devm_page_free (struct page *page) +{ +struct drm_buddy_block *block = + (struct drm_buddy_block *)page->zone_device_data; +struct xe_svm_block_meta *meta = + (struct xe_svm_block_meta *)block->private; +struct xe_tile *tile = meta->tile; +struct xe_mem_region *mr = &tile->mem.vram; +struct drm_buddy *mm = &tile->mem.vram_mgr->mm; +u64 size =drm_buddy_block_size (mm, block); +u64 pages_per_block = size >>PAGE_SHIFT ; +u64 block_pfn_first = +block_offset_to_pfn (mr,drm_buddy_block_offset (block)); +u64 page_pfn =page_to_pfn (page); +u64 i = page_pfn - block_pfn_first; + +xe_assert (tile->xe, i < pages_per_block); +clear_bit (i, meta->bitmap); +if (bitmap_empty (meta->bitmap, pages_per_block)) +free_block (block); +} + +/** +* xe_devm_alloc_pages() - allocate device pages from buddy allocator +* +* @xe_tile: which tile to allocate device memory from +* @npages: how many pages to allocate +* @blocks: used to return the allocated blocks +* @pfn: used to return the pfn of all allocated pages. Must be big enough +* to hold at @npages entries. +* +* This function allocate blocks of memory from drm buddy allocator, and +* performs initialization work: set struct page::zone_device_data to point +* to the memory block; set/initialize drm_buddy_block::private field; +* lock_page for each page allocated; add memory block to lru managers lru +* list - this is TBD. +* +* return: 0 on success +* error code otherwise +*/ +int xe_devm_alloc_pages(struct xe_tile *tile, + unsigned long npages,
drivers/gpu/drm/xe/xe_svm_devmem.c line 117
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +10
-0+struct list_head *blocks, + unsigned long *pfn) +{ + struct drm_buddy *mm = &tile->mem.vram_mgr->mm; +struct drm_buddy_block *block, *tmp; +u64 size = npages <<PAGE_SHIFT ; +int ret =0 , i, j =0 ; + + ret =drm_buddy_alloc_blocks (mm,0 , mm->size, size,PAGE_SIZE , + blocks,DRM_BUDDY_TOPDOWN_ALLOCATION );
Realized this while discussing ref counting off the list, the buddy allocation size can be either PAGE_SIZE or SZ_64K depending on platform too. We store this in VM via XE_VM_FLAG_64K flag.
Matt
Quoted 51 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +44
-0+ +if (unlikely (ret)) +return ret; + +list_for_each_entry_safe (block, tmp, blocks, link) { +struct xe_mem_region *mr = &tile->mem.vram; +u64 block_pfn_first, pages_per_block; +struct xe_svm_block_meta *meta; +u32 meta_size; + + size =drm_buddy_block_size (mm, block); + pages_per_block = size >>PAGE_SHIFT ; + meta_size =BITS_TO_BYTES (pages_per_block) + +sizeof (struct xe_svm_block_meta ); + meta =kzalloc (meta_size,GFP_KERNEL ); +bitmap_fill (meta->bitmap, pages_per_block); + meta->tile = tile; + block->private = meta; + block_pfn_first = +block_offset_to_pfn (mr,drm_buddy_block_offset (block)); +for (i =0 ; i < pages_per_block; i++) { +struct page *page; + + pfn[j++] = block_pfn_first + i; + page =pfn_to_page (block_pfn_first + i); +/**Lock page per hmm requirement, see hmm.rst.*/ +zone_device_page_init (page); + page->zone_device_data = block; + } + } + +return ret; +} + +/** +* xe_devm_free_blocks() - free all memory blocks +* +* @blocks: memory blocks list head +*/ +void xe_devm_free_blocks (struct list_head *blocks) { +struct drm_buddy_block *block, *tmp; + +list_for_each_entry_safe (block, tmp, blocks, link) +free_block (block); }static const struct dev_pagemap_ops xe_devm_pagemap_ops = {Signature · 1 line
2.26.3
PATCH v2 23/31 drm/xe/svm: Trace buddy block allocation and free
In reply to Oak Zengtrace_xe_buddy_block_alloc and trace_xe_buddy_block_free are added to trace buddy allocation and free.
--- drivers/gpu/drm/xe/xe_svm_devmem.c | 6 ++++- drivers/gpu/drm/xe/xe_trace.h | 35 ++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_svm_devmem.c +5 -1
@@ -17,6 +17,7 @@#include "xe_migrate.h" #include "xe_ttm_vram_mgr_types.h" #include "xe_assert.h" +#include "xe_trace.h" /** * struct xe_svm_block_meta - svm uses this data structure to manage each
@@ -81,8 +82,10 @@ void xe_devm_page_free(struct page *page)xe_assert (tile->xe, i < pages_per_block);clear_bit (i, meta->bitmap); -if (bitmap_empty (meta->bitmap, pages_per_block)) +if (bitmap_empty (meta->bitmap, pages_per_block)) {free_block (block); +trace_xe_buddy_block_free (block, size, block_pfn_first); + } }/**
@@ -135,6 +138,7 @@ int xe_devm_alloc_pages(struct xe_tile *tile, block->private = meta; block_pfn_first =block_offset_to_pfn (mr,drm_buddy_block_offset (block)); +trace_xe_buddy_block_alloc (block, size, block_pfn_first);for (i =0 ; i < pages_per_block; i++) {struct page *page;
drivers/gpu/drm/xe/xe_trace.h +35 -0
@@ -21,6 +21,7 @@#include "xe_guc_exec_queue_types.h" #include "xe_sched_job.h" #include "xe_vm.h" +#include <drm/drm_buddy.h> DECLARE_EVENT_CLASS(xe_gt_tlb_invalidation_fence, TP_PROTO(struct xe_gt_tlb_invalidation_fence *fence),
@@ -622,6 +623,40 @@ DEFINE_EVENT_PRINT(xe_guc_ctb, xe_guc_ctb_g2h, ); +DECLARE_EVENT_CLASS (xe_buddy_block, + TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn), +TP_ARGS (block, size, pfn), + +TP_STRUCT__entry ( + __field(u64, block) + __field(u64, header) + __field(u64, size) +__field (u64, pfn) + ), + +TP_fast_assign ( + __entry->block = (u64)block; + __entry->header = block->header; + __entry->size = size; + __entry->pfn = pfn; + ), + +TP_printk ("xe svm: allocated block %llx, block header %llx, size %llx, pfn %llx\n" , + __entry->block, __entry->header, __entry->size, __entry->pfn) +); + + +DEFINE_EVENT (xe_buddy_block, xe_buddy_block_alloc, + TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn), +TP_ARGS (block, size, pfn) +); + + +DEFINE_EVENT (xe_buddy_block, xe_buddy_block_free, + TP_PROTO(struct drm_buddy_block *block, u64 size, u64 pfn), +TP_ARGS (block, size, pfn) +); +#endif /* This part must be outside protection */
Signature · 1 line
2.26.3
PATCH v2 24/31 drm/xe/svm: Create and destroy xe svm
In reply to Oak ZengIntroduce a data structure xe_svm to represent a shared virtual address space b/t CPU program and GPU program. Each process can only have maximumly one xe_svm instance. One xe_svm can have multiple gpu vm.
Introduce helper functions to create and destroy xe_svm instance. Once xe_svm instance is created, it is added to a global hash table keyed by mm_struct. Later on we can retrieve xe_svm using mm_struct.
--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_svm.c | 77 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_svm.h | 23 +++++++++++ 3 files changed, 101 insertions(+) create mode 100644 drivers/gpu/drm/xe/xe_svm.c
drivers/gpu/drm/xe/Makefile +1 -0
@@ -129,6 +129,7 @@ xe-y += xe_bb.o \ xe_sa.o \ xe_sched_job.o \ xe_step.o \ + xe_svm.o \ xe_svm_devmem.o \ xe_sync.o \ xe_tile.o \
drivers/gpu/drm/xe/xe_svm.c new file +77 -0
77 lines not on this page · raw message
drivers/gpu/drm/xe/xe_svm.h +23 -0
@@ -11,6 +11,29 @@#include "xe_device.h" #include "xe_assert.h" + +/** +* struct xe_svm - data structure to represent a shared +* virtual address space from device side. xe_svm and +* mm_struct has a 1:1 relationship. +*/ +struct xe_svm { +/** @mm: The mm_struct corresponding to this xe_svm */ +struct mm_struct *mm; +/** +* @mutex: A lock protects below vm_list +*/ +struct mutex mutex; +/** @hnode: used to add this svm to a global xe_svm_hash table*/ +struct hlist_node hnode; +/** @vm_list: a list gpu vm in this svm space */ +struct list_head vm_list; +}; + +extern struct xe_svm *xe_create_svm (void ); +void xe_destroy_svm (struct xe_svm *svm); +extern struct xe_svm *xe_lookup_svm_by_mm (struct mm_struct *mm); +/** * xe_mem_region_pfn_to_dpa() - Calculate page's dpa from pfn *
Signature · 1 line
2.26.3
Matthew Brost
maintainerRe: [v2 24/31] drm/xe/svm: Create and destroy xe svm
Quoted 9 lines · Oak Zeng
Introduce a data structure
xe_svmto represent a shared virtual address space b/t CPU program and GPU program. Each process can only have maximumly onexe_svminstance. Onexe_svmcan have multiple gpu vm.Introduce helper functions to create and destroy
xe_svminstance. Oncexe_svminstance is created, it is added to a global hash table keyed bymm_struct. Later on we can retrievexe_svmusingmm_struct.
I don't think this needed at all, will explain a bit later in the series but quite sure this can be droppped entirely.
Matt
Quoted 140 lines
Signed-off-by Oak Zeng <[email protected]>--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_svm.c | 77 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_svm.h | 23 +++++++++++ 3 files changed, 101 insertions(+) create mode 100644 drivers/gpu/drm/xe/xe_svm.cdrivers/gpu/drm/xe/Makefile +1
-0@@ -129,6 +129,7 @@ xe-y += xe_bb.o \ xe_sa.o \ xe_sched_job.o \ xe_step.o \ + xe_svm.o \ xe_svm_devmem.o \ xe_sync.o \ xe_tile.o \drivers/gpu/drm/xe/xe_svm.c new file +77
-0@@ -0,0 +1,77 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2023 Intel Corporation +*/ + +#include <linux/mutex.h> +#include <linux/mm_types.h> +#include <linux/kernel.h> +#include <linux/hashtable.h> +#include "xe_svm.h" + +#define XE_MAX_SVM_PROCESS 5/* Maximumly support 32 SVM process*/ +DEFINE_HASHTABLE (xe_svm_table,XE_MAX_SVM_PROCESS ); + +/** +* xe_create_svm() - create a svm instance +* +* one xe_svm struct represent a shared address space +* between cpu and gpu program. So one xe_svm is associated +* to one mm_struct. +* +* If xe_svm for this process already exists, just return +* it; otherwise create one. +* +* Return the created xe svm struct pointer +*/ +struct xe_svm *xe_create_svm (void ) +{ +struct mm_struct *mm = current->mm; +struct xe_svm *svm; + + svm =xe_lookup_svm_by_mm (mm); +if (svm) +return svm; + + svm =kzalloc (sizeof (struct xe_svm ),GFP_KERNEL ); + svm->mm = mm; +mutex_init (&svm->mutex); +INIT_LIST_HEAD (&svm->vm_list); +/** Add svm to global xe_svm_table hash table +* use mm as key so later we can retrieve svm using mm +*/ +hash_add_rcu (xe_svm_table, &svm->hnode, (uintptr_t )mm); +return svm; +} + +/** +* xe_destroy_svm() - destroy a svm process +* +* @svm: the xe_svm to destroy +*/ +void xe_destroy_svm (struct xe_svm *svm) +{ +BUG_ON (list_empty (&svm->vm_list)); +hash_del_rcu (&svm->hnode); +mutex_destroy (&svm->mutex); +kfree (svm); +} + + +/** +* xe_lookup_svm_by_mm() - retrieve xe_svm from mm struct +* +* @mm: the mm struct of the svm to retrieve +* +* Return the xe_svm struct pointer, or NULL if fail +*/ +struct xe_svm *xe_lookup_svm_by_mm (struct mm_struct *mm) +{ +struct xe_svm *svm; + +hash_for_each_possible_rcu (xe_svm_table, svm, hnode, (uintptr_t )mm) +if (svm->mm == mm) +return svm; + +return NULL ; +}drivers/gpu/drm/xe/xe_svm.h +23
-0@@ -11,6 +11,29 @@#include "xe_device.h" #include "xe_assert.h" + +/** +* struct xe_svm - data structure to represent a shared +* virtual address space from device side. xe_svm and +* mm_struct has a 1:1 relationship. +*/ +struct xe_svm { +/** @mm: The mm_struct corresponding to this xe_svm */ +struct mm_struct *mm; +/** +* @mutex: A lock protects below vm_list +*/ +struct mutex mutex; +/** @hnode: used to add this svm to a global xe_svm_hash table*/ +struct hlist_node hnode; +/** @vm_list: a list gpu vm in this svm space */ +struct list_head vm_list; +}; + +extern struct xe_svm *xe_create_svm (void ); +void xe_destroy_svm (struct xe_svm *svm); +extern struct xe_svm *xe_lookup_svm_by_mm (struct mm_struct *mm); +/** * xe_mem_region_pfn_to_dpa() - Calculate page's dpa from pfn * Signature · 1 line
2.26.3
PATCH v2 25/31 drm/xe/svm: Add vm to xe_svm process
In reply to Oak ZengOne shared virtual address space (xe_svm) works across CPU and multiple GPUs under one CPU process. Each xe_svm process can have multiple gpu vm, for example, one gpu vm for one gpu card. Add gpu vm to the current xe_svm process during xe_vm creation to note this gpu vm participate the shared virtual address space with the current CPU process, also remove xe_vm from xe_svm on xe_vm destroy.
FIXME: right now we blindly add all xe_vm to svm. Should we introduce uAPI to allow user decide which xe_vm participate svm?
--- drivers/gpu/drm/xe/xe_svm.c | 45 ++++++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_svm.h | 3 +++ drivers/gpu/drm/xe/xe_vm.c | 5 ++++ drivers/gpu/drm/xe/xe_vm_types.h | 2 ++ 4 files changed, 55 insertions(+)
drivers/gpu/drm/xe/xe_svm.c +45 -0
@@ -8,6 +8,7 @@#include <linux/kernel.h> #include <linux/hashtable.h> #include "xe_svm.h" +#include "xe_vm_types.h" #define XE_MAX_SVM_PROCESS 5/* Maximumly support 32 SVM process*/ DEFINE_HASHTABLE (xe_svm_table,XE_MAX_SVM_PROCESS );
@@ -75,3 +76,47 @@ struct xe_svm *xe_lookup_svm_by_mm(struct mm_struct *mm)return NULL ; } + +/** +* xe_svm_add_vm() - add a gpu vm to the current svm process +* +* @vm: The gpu vm to add to the current svm process. +* +* One shared virtual address space (xe_svm) works across CPU +* and multiple GPUs. So each xe_svm process can have N gpu +* vm, for example, one gpu vm for on gpu card. This function +* add a gpu vm to the current xe_svm process. +*/ +void xe_svm_add_vm (struct xe_vm *vm) +{ +struct xe_svm *svm; + + svm =xe_lookup_svm_by_mm (current->mm); +if (!svm) + svm =xe_create_svm (); + +mutex_lock (&svm->mutex); +list_add (&vm->svm_link, &svm->vm_list); +mutex_unlock (&svm->mutex); +} + +/** +* xe_svm_remove_vm() - remove a gpu vm from svm process +* +* @vm: The gpu vm to remove from svm process. +*/ +void xe_svm_remove_vm (struct xe_vm *vm) +{ +struct xe_svm *svm; + + svm =xe_lookup_svm_by_mm (current->mm); +if (!svm) +return ; + +mutex_lock (&svm->mutex); +list_del (&vm->svm_link); +mutex_unlock (&svm->mutex); + +if (list_empty (&svm->vm_list)) +xe_destroy_svm (svm); +}
drivers/gpu/drm/xe/xe_svm.h +3 -0
@@ -11,6 +11,7 @@#include "xe_device.h" #include "xe_assert.h" +struct xe_vm ;/** * struct xe_svm - data structure to represent a shared
@@ -33,6 +34,8 @@ struct xe_svm {extern struct xe_svm *xe_create_svm (void );void xe_destroy_svm (struct xe_svm *svm);extern struct xe_svm *xe_lookup_svm_by_mm (struct mm_struct *mm); +void xe_svm_add_vm (struct xe_vm *vm); +void xe_svm_remove_vm (struct xe_vm *vm);/** * xe_mem_region_pfn_to_dpa() - Calculate page's dpa from pfn
drivers/gpu/drm/xe/xe_vm.c +5 -0
@@ -40,6 +40,7 @@#include "xe_trace.h" #include "xe_wa.h" #include "xe_hmm.h" +#include "xe_svm.h" static struct drm_gem_object *xe_vm_obj (struct xe_vm *vm) {
@@ -1347,6 +1348,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags)INIT_LIST_HEAD (&vm->userptr.repin_list);INIT_LIST_HEAD (&vm->userptr.invalidated);INIT_LIST_HEAD (&vm->userptr.fault_invalidated); +INIT_LIST_HEAD (&vm->svm_link);init_rwsem (&vm->userptr.notifier_lock);spin_lock_init (&vm->userptr.invalidated_lock);INIT_WORK (&vm->userptr.garbage_collector, vm_userptr_garbage_collector);
@@ -1445,6 +1447,8 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags) xe->usm.num_vm_in_non_fault_mode++;mutex_unlock (&xe->usm.lock); +/** FIXME: Should we add vm to svm conditionally? Per uAPI?*/ +xe_svm_add_vm (vm);trace_xe_vm_create (vm);return vm;
@@ -1562,6 +1566,7 @@ void xe_vm_close_and_put(struct xe_vm *vm)for_each_tile (tile, xe, id)xe_range_fence_tree_fini (&vm->rftree[id]); +xe_svm_remove_vm (vm);xe_vm_put (vm); }
drivers/gpu/drm/xe/xe_vm_types.h +2 -0
@@ -394,6 +394,8 @@ struct xe_vm {bool batch_invalidate_tlb;/** @xef: XE file handle for tracking this VM's drm client */ struct xe_file *xef; +/** @svm_link: used to link this vm to xe_svm's vm_list*/ + struct list_head svm_link; };#endif
Signature · 1 line
2.26.3
PATCH v2 26/31 drm/xe: Make function lookup_vma public
In reply to Oak ZengPublic this function as it will be used by later patches. Also rename it to xe_vm_lookup_vma
--- drivers/gpu/drm/xe/xe_gt_pagefault.c | 10 ++++++++-- drivers/gpu/drm/xe/xe_vm.h | 1 + 2 files changed, 9 insertions(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_gt_pagefault.c +8 -2
@@ -80,7 +80,13 @@ static bool vma_matches(struct xe_vma *vma, u64 page_addr)return true; } -static struct xe_vma *lookup_vma (struct xe_vm *vm,u64 page_addr) +/** +* xe_vm_lookup_vma() - look up a vma from address +* +* @vm: the xe_vm that the vma resides in +* @page_address: address to look up +*/ +struct xe_vma *xe_vm_lookup_vma (struct xe_vm *vm,u64 page_addr) {struct xe_vma *vma =NULL ;
@@ -166,7 +172,7 @@ static int handle_pagefault(struct xe_gt *gt, struct pagefault *pf) ret = -ENOENT ; goto unlock_vm; } - vma =lookup_vma (vm, pf->page_addr); + vma =xe_vm_lookup_vma (vm, pf->page_addr);if (!vma) { ret = -EINVAL ; goto unlock_vm;
drivers/gpu/drm/xe/xe_vm.h +1 -0
@@ -306,3 +306,4 @@ struct xe_vm_snapshot *xe_vm_snapshot_capture(struct xe_vm *vm);void xe_vm_snapshot_capture_delayed (struct xe_vm_snapshot *snap);void xe_vm_snapshot_print (struct xe_vm_snapshot *snap,struct drm_printer *p);void xe_vm_snapshot_free (struct xe_vm_snapshot *snap); +struct xe_vma *xe_vm_lookup_vma (struct xe_vm *vm,u64 page_addr);
Signature · 1 line
2.26.3
Matthew Brost
maintainerRe: [v2 26/31] drm/xe: Make function lookup_vma public
Quoted 3 lines · Oak Zeng
Public this function as it will be used by later patches. Also rename it to
xe_vm_lookup_vma
Like the previous patch, pretty sure this can be dropped too. Again will fully explain later.
Matt
Quoted 46 lines
Signed-off-by Oak Zeng <[email protected]>--- drivers/gpu/drm/xe/xe_gt_pagefault.c | 10 ++++++++-- drivers/gpu/drm/xe/xe_vm.h | 1 + 2 files changed, 9 insertions(+), 2 deletions(-)drivers/gpu/drm/xe/xe_gt_pagefault.c +8
-2@@ -80,7 +80,13 @@ static bool vma_matches(struct xe_vma *vma, u64 page_addr)return true; } -static struct xe_vma *lookup_vma (struct xe_vm *vm,u64 page_addr) +/** +* xe_vm_lookup_vma() - look up a vma from address +* +* @vm: the xe_vm that the vma resides in +* @page_address: address to look up +*/ +struct xe_vma *xe_vm_lookup_vma (struct xe_vm *vm,u64 page_addr) {struct xe_vma *vma =NULL ;@@ -166,7 +172,7 @@ static int handle_pagefault(struct xe_gt *gt, struct pagefault *pf) ret = -ENOENT ; goto unlock_vm; } - vma =lookup_vma (vm, pf->page_addr); + vma =xe_vm_lookup_vma (vm, pf->page_addr);if (!vma) { ret = -EINVAL ; goto unlock_vm;drivers/gpu/drm/xe/xe_vm.h +1
-0@@ -306,3 +306,4 @@ struct xe_vm_snapshot *xe_vm_snapshot_capture(struct xe_vm *vm);void xe_vm_snapshot_capture_delayed (struct xe_vm_snapshot *snap);void xe_vm_snapshot_print (struct xe_vm_snapshot *snap,struct drm_printer *p);void xe_vm_snapshot_free (struct xe_vm_snapshot *snap); +struct xe_vma *xe_vm_lookup_vma (struct xe_vm *vm,u64 page_addr);Signature · 1 line
2.26.3
PATCH v2 27/31 drm/xe/svm: Handle CPU page fault
In reply to Oak ZengUnder the picture of svm, CPU and GPU program share one same virtual address space. The backing store of this virtual address space can be either in system memory or device memory. Since GPU device memory is remaped as DEVICE_PRIVATE, CPU can't access it. Any CPU access to device memory causes a page fault. Implement a page fault handler to migrate memory back to system memory and map it to CPU page table so the CPU program can proceed.
Also unbind this page from GPU side, and free the original GPU device page
--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_svm.h | 8 +- drivers/gpu/drm/xe/xe_svm_devmem.c | 7 +- drivers/gpu/drm/xe/xe_svm_migrate.c | 222 ++++++++++++++++++++++++++++ 4 files changed, 230 insertions(+), 8 deletions(-) create mode 100644 drivers/gpu/drm/xe/xe_svm_migrate.c
drivers/gpu/drm/xe/Makefile +1 -0
@@ -131,6 +131,7 @@ xe-y += xe_bb.o \ xe_step.o \ xe_svm.o \ xe_svm_devmem.o \ + xe_svm_migrate.o \ xe_sync.o \ xe_tile.o \ xe_tile_sysfs.o \
drivers/gpu/drm/xe/xe_svm.h +6 -2
@@ -7,11 +7,11 @@#define __XE_SVM_H#include <linux/mm_types.h> +#include <linux/mm.h> #include "xe_device_types.h" #include "xe_device.h" #include "xe_assert.h" - -struct xe_vm ; +#include "xe_vm_types.h" /** * struct xe_svm - data structure to represent a shared
@@ -31,6 +31,9 @@ struct xe_svm {struct list_head vm_list; }; +#define xe_svm_for_each_vm (svm, vm) \ + list_for_each_entry(vm, &svm->vm_list, svm_link) +extern struct xe_svm *xe_create_svm (void );void xe_destroy_svm (struct xe_svm *svm);extern struct xe_svm *xe_lookup_svm_by_mm (struct mm_struct *mm);
@@ -79,4 +82,5 @@ int xe_devm_alloc_pages(struct xe_tile *tile,void xe_devm_free_blocks (struct list_head *blocks);void xe_devm_page_free (struct page *page); +vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf);#endif
drivers/gpu/drm/xe/xe_svm_devmem.c +1 -6
@@ -37,11 +37,6 @@ struct xe_svm_block_meta {unsigned long bitmap[]; }; -static vm_fault_t xe_devm_migrate_to_ram (struct vm_fault *vmf) -{ -return 0 ; -} -static u64 block_offset_to_pfn (struct xe_mem_region *mr,u64 offset) {/** DRM buddy's block offset is 0-based*/
@@ -168,7 +163,7 @@ void xe_devm_free_blocks(struct list_head *blocks)static const struct dev_pagemap_ops xe_devm_pagemap_ops = { .page_free = xe_devm_page_free, - .migrate_to_ram = xe_devm_migrate_to_ram, + .migrate_to_ram = xe_svm_migrate_to_sram, };/**
drivers/gpu/drm/xe/xe_svm_migrate.c new file +222 -0
222 lines not on this page · raw message
Signature · 1 line
2.26.3
Matthew Brost
maintainerRe: [v2 27/31] drm/xe/svm: Handle CPU page fault
Quoted 30 lines · Oak Zeng
Under the picture of svm, CPU and GPU program share one same virtual address space. The backing store of this virtual address space can be either in system memory or device memory. Since GPU device memory is remaped as
DEVICE_PRIVATE, CPU can't access it. Any CPU access to device memory causes a page fault. Implement a page fault handler to migrate memory back to system memory and map it to CPU page table so the CPU program can proceed.Also unbind this page from GPU side, and free the original GPU device page
Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana Vishwanathapura <[email protected]>Signed-off-by Niranjana Vishwanathapura <[email protected]>Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_svm.h | 8 +- drivers/gpu/drm/xe/xe_svm_devmem.c | 7 +- drivers/gpu/drm/xe/xe_svm_migrate.c | 222 ++++++++++++++++++++++++++++ 4 files changed, 230 insertions(+), 8 deletions(-) create mode 100644 drivers/gpu/drm/xe/xe_svm_migrate.cdrivers/gpu/drm/xe/Makefile +0
-0
drivers/gpu/drm/xe/Makefile xe-y += xe_bb.o \ · line 134
Quoted 4 lines
drivers/gpu/drm/xe/Makefile +1
-0@@ -131,6 +131,7 @@ xe-y += xe_bb.o \ xe_step.o \ xe_svm.o \ xe_svm_devmem.o \ + xe_svm_migrate.o \
See comments about file org, same thing applies here. Let's put all of the svm implementation in a single file.
Quoted 22 lines
drivers/gpu/drm/xe/Makefile +0
-0xe_sync.o \ xe_tile.o \ xe_tile_sysfs.o \drivers/gpu/drm/xe/xe_svm.h +2
-2@@ -7,11 +7,11 @@#define __XE_SVM_H#include <linux/mm_types.h> +#include <linux/mm.h> #include "xe_device_types.h" #include "xe_device.h" #include "xe_assert.h" - -struct xe_vm ; +#include "xe_vm_types.h" /** * struct xe_svm - data structure to represent a shared
drivers/gpu/drm/xe/xe_svm.h struct xe_svm { · line 36
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm.h +3
-0@@ -31,6 +31,9 @@ struct xe_svm {struct list_head vm_list; }; +#define xe_svm_for_each_vm (svm, vm) \ + list_for_each_entry(vm, &svm->vm_list, svm_link) +
Don't think this is need, see below.
Quoted 26 lines
drivers/gpu/drm/xe/xe_svm.h +1
-0extern struct xe_svm *xe_create_svm (void );void xe_destroy_svm (struct xe_svm *svm);extern struct xe_svm *xe_lookup_svm_by_mm (struct mm_struct *mm);@@ -79,4 +82,5 @@ int xe_devm_alloc_pages(struct xe_tile *tile,void xe_devm_free_blocks (struct list_head *blocks);void xe_devm_page_free (struct page *page); +vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf);#endif drivers/gpu/drm/xe/xe_svm_devmem.c +0
-5@@ -37,11 +37,6 @@ struct xe_svm_block_meta {unsigned long bitmap[]; }; -static vm_fault_t xe_devm_migrate_to_ram (struct vm_fault *vmf) -{ -return 0 ; -} -static u64 block_offset_to_pfn (struct xe_mem_region *mr,u64 offset) {/** DRM buddy's block offset is 0-based*/
drivers/gpu/drm/xe/xe_svm_devmem.c void xe_devm_free_blocks(struct list_head *blocks) · line 166
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +1
-1@@ -168,7 +163,7 @@ void xe_devm_free_blocks(struct list_head *blocks)static const struct dev_pagemap_ops xe_devm_pagemap_ops = { .page_free = xe_devm_page_free, - .migrate_to_ram = xe_devm_migrate_to_ram, + .migrate_to_ram = xe_svm_migrate_to_sram,
Again single file so this will be static function, no reason to export this.
Quoted 49 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +0
-0};/** drivers/gpu/drm/xe/xe_svm_migrate.c new file +40
-0@@ -0,0 +1,222 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2023 Intel Corporation +*/ + +#include <linux/gfp.h> +#include <linux/migrate.h> +#include <linux/dma-mapping.h> +#include <linux/dma-fence.h> +#include <linux/bitops.h> +#include <linux/bitmap.h> +#include <linux/kernel.h> +#include <linux/slab.h> +#include <drm/drm_buddy.h> +#include "xe_device_types.h" +#include "xe_device.h" +#include "xe_trace.h" +#include "xe_migrate.h" +#include "xe_ttm_vram_mgr_types.h" +#include "xe_assert.h" +#include "xe_pt.h" +#include "xe_svm.h" +#include "xe_vm.h" + + +/** +* alloc_host_page() - allocate one host page for the fault vma +* +* @dev: (GPU) device that will access the allocated page +* @vma: the fault vma that we need allocate page for +* @addr: the fault address. The allocated page is for this address +* @dma_addr: used to output the dma address of the allocated page. +* This dma address will be used for gpu to access this page. GPU +* access host page through a dma mapped address. +* @pfn: used to output the pfn of the allocated page. +* +* This function allocate one host page for the specified vma. It +* also does some prepare work for GPU to access this page, such +* as map this page to iommu (by calling dma_map_page). +*
drivers/gpu/drm/xe/xe_svm_migrate.c line 50
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+* When this function returns, the page is locked. +* +* Return struct page pointer when success +* NULL otherwise +*/ +static struct page *alloc_host_page (struct device *dev, +struct vm_area_struct *vma, +unsigned long addr, +dma_addr_t *dma_addr, +unsigned long *pfn)
Weird alignment, also I don't think we are want to allocate a page at time...
Beyond that, can't say I'm a fan of 2 arguments being return and populated here either (dma_addr_t *dma_addr, unsigned long *pfn). I haven't seen a lot that style function in Linux.
Probably makes more sense to have a function which allocates pages, locks them, and populates the pfn array (migrate_pfn) rather than doing this a page at a time.
drivers/gpu/drm/xe/xe_svm_migrate.c line 60
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+{ +struct page *page; + + page =alloc_page_vma (GFP_HIGHUSER , vma, addr); +if (unlikely (!page)) +return NULL ; + +/**Lock page per hmm requirement, see hmm.rst*/ +lock_page (page); + *dma_addr =dma_map_page (dev, page,0 ,PAGE_SIZE ,DMA_FROM_DEVICE );
The device is writing to these pages so I think DMA_BIDIRECTIONAL is needed, right? As mentioned above I think this should be broken out into a different step too.
Quoted 24 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +24
-0+if (unlikely (dma_mapping_error (dev, *dma_addr))) { +unlock_page (page); +__free_page (page); +return NULL ; + } + + *pfn =migrate_pfn (page_to_pfn (page)); +return page; +} + +static void free_host_page (struct page *page) +{ +unlock_page (page); +put_page (page); +} + +/** +* migrate_page_vram_to_ram() - migrate one page from vram to ram +* +* @vma: The vma that the page is mapped to +* @addr: The virtual address that the page is mapped to +* @src_pfn: src page's page frame number +* @dst_pfn: used to return dstination page (in system ram)'s pfn +*
drivers/gpu/drm/xe/xe_svm_migrate.c line 94
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+* Allocate one page in system ram and copy memory from device memory +* to system ram. +* +* Return: 0 if this page is already in sram (no need to migrate) +* 1: successfully migrated this page from vram to sram. +* error code otherwise +*/ +static int migrate_page_vram_to_ram (struct vm_area_struct *vma,unsigned long addr, +unsigned long src_pfn,unsigned long *dst_pfn) +{
We definitely don't want to copy 1 page at time. I touch on this in [1]. Basically this going to perform poorly unless we use larger copies, the migrate code supports non-contigous sram addresses, and vram addresses will likely be contigous due to the buddy allocator.
[1] https://patchwork.freedesktop.org/patch/588548/?series=132229&rev=1
drivers/gpu/drm/xe/xe_svm_migrate.c line 106
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +12
-0+struct xe_mem_region *mr; +struct xe_tile *tile; +struct xe_device *xe; +struct device *dev; +dma_addr_t dma_addr =0 ; +struct dma_fence *fence; +struct page *host_page; +struct page *src_page; +u64 src_dpa; + + src_page =migrate_pfn_to_page (src_pfn); +if (unlikely (!src_page || !(src_pfn &MIGRATE_PFN_MIGRATE )))
I'm going to say this is a bug if !src_page || !is_zone_device_page(src_page) || !(src_pfn & MIGRATE_PFN_MIGRATE) and we return -EFAULT (or another error code if that makes more sense). We are migrating from the device where we know we have backing store from the original fault.
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +11
-0+return 0 ; + + mr =xe_page_to_mem_region (src_page); + tile =xe_mem_region_to_tile (mr); + xe =tile_to_xe (tile); + dev = xe->drm.dev; + + src_dpa =xe_mem_region_pfn_to_dpa (mr, src_pfn); + host_page =alloc_host_page (dev, vma, addr, &dma_addr, dst_pfn); +if (!host_page) +return -ENOMEM ;
drivers/gpu/drm/xe/xe_svm_migrate.c line 127
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+ + fence =xe_migrate_pa (tile->migrate, src_dpa, true, + dma_addr, false,PAGE_SIZE ); +if (IS_ERR (fence)) { +dma_unmap_page (dev, dma_addr,PAGE_SIZE ,DMA_FROM_DEVICE ); +free_host_page (host_page); +return PTR_ERR (fence); + } + +dma_fence_wait (fence, false);
Even if we did want to migrate a page at a time, we only need to wait on the last fence due to the ordered nature of exec queues.
drivers/gpu/drm/xe/xe_svm_migrate.c line 129
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+dma_fence_put (fence); +dma_unmap_page (dev, dma_addr,PAGE_SIZE ,DMA_FROM_DEVICE );
With above, will likely unmap all dma pages in a single function once the last fence is signaled.
Quoted 25 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +25
-0+return 1 ; +} + +/** +* xe_svm_migrate_to_sram() - Migrate memory back to sram on CPU page fault +* +* @vmf: cpu vm fault structure, contains fault information such as vma etc. +* +* Note, this is in CPU's vm fault handler, caller holds the mmap read lock. +* +* This function migrate one gpu vma which contains the fault address to sram. +* We try to maintain a 1:1 mapping b/t the CPU vma and gpu vma (i.e., create one +* gpu vma for one cpu vma initially and try not to split it). So this scheme end +* up migrate at the vma granularity. This might not be the best performant scheme +* +* This can be tunned with a migration granularity for performance, for example, +* migration 2M for each CPU page fault, or let user specify how much to migrate. +* This is more complex due to vma splitting. +* +* This function should also update GPU page table, so the fault virtual address +* points to the same sram location from GPU side. This is TBD. +* +* Return: +* 0 on success +* VM_FAULT_SIGBUS: failed to migrate page to system memory, application
drivers/gpu/drm/xe/xe_svm_migrate.c line 164
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+* will be signaled a SIGBUG +*/ +vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf) +{ +struct xe_mem_region *mr =xe_page_to_mem_region (vmf->page); +struct xe_tile *tile =xe_mem_region_to_tile (mr); +struct xe_device *xe =tile_to_xe (tile); +struct vm_area_struct *vma = vmf->vma; +struct mm_struct *mm = vma->vm_mm; +struct xe_svm *svm =xe_lookup_svm_by_mm (mm);
I don't think this is needed... More below.
drivers/gpu/drm/xe/xe_svm_migrate.c line 176
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +12
-0+unsigned long addr = vma->vm_start; +u64 npages =vma_pages (vma); +struct xe_vma *xe_vma; +vm_fault_t ret =0 ; +struct xe_vm *vm; +void *buf; +int i; + +struct migrate_vma migrate_vma = { + .vma = vmf->vma, + .start = vma->vm_start, + .end = vma->vm_end,
So I know in my PoC I had the fault user pointer (xe_vma) == struct vm_area_struct of the GPU fault. That is definitely wrong. We likely want to allocate sub-range of vm_area_struct for the xe_vma, we can call this a chunk size. Logical chunks sizes would be aligned 2MB, 64k, and finally 4k in sizes trying the largest first... The chunk sizes are trivial as we likely can just have table with values, the key here is the vm_area_struct vm_start / vm_end are not what we want to use here rather xe_vma_start(vma) and xe_vma_end(vma). I think we get the xe_vma from the faulting page vmf->page->zone_device_data field unless you have another use that field...
I also comment on my patch with my suggestion implement chunk sizes too.
drivers/gpu/drm/xe/xe_svm_migrate.c line 177
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+ .pgmap_owner = xe,
Again helper for this.
drivers/gpu/drm/xe/xe_svm_migrate.c line 190
Quoted 13 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +13
-0+ .flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE, + .fault_page = vmf->page, + }; + + buf =kvcalloc (npages,2 *sizeof (*migrate_vma.src),GFP_KERNEL ); + migrate_vma.src = buf; + migrate_vma.dst = buf + npages; +if (migrate_vma_setup (&migrate_vma) <0 ) { + ret =VM_FAULT_SIGBUS ; + goto free_buf; + } + +if (!migrate_vma.cpages)
This is an error, need to set a return value.
drivers/gpu/drm/xe/xe_svm_migrate.c line 192
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+ goto free_buf; +
We probably should check migrate.cpages != npages too as I also think this is an error.
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +7
-0+for (i =0 ; i < npages; i++) { + ret =migrate_page_vram_to_ram (vma, addr, migrate_vma.src[i], + migrate_vma.dst + i); +if (ret <0 ) { + ret =VM_FAULT_SIGBUS ; +break ; + }
drivers/gpu/drm/xe/xe_svm_migrate.c line 209
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+ +/** Migration has been successful, free source page */ +if (ret ==1 ) { +struct page *src_page =migrate_pfn_to_page (migrate_vma.src[i]); + +xe_devm_page_free (src_page); + } + + addr +=PAGE_SIZE ; + }
I touch on this above, this should be reworked to roughly:
- alloc pages and populate migrate_vma.dst
- dma map sram pages
- migrate a chunk of contigous vram addresses at a time
- wait on last dma fence from migrate
- unmap dma pages
- unlock and free all pages
drivers/gpu/drm/xe/xe_svm_migrate.c line 216
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +7
-0+ +xe_svm_for_each_vm (svm, vm) { +xe_assert (xe, vm->mm == mm); + xe_vma =xe_vm_lookup_vma (vm, vmf->address); +if (xe_vma) +xe_vm_invalidate_vma (xe_vma); + }
I've touched on why this isn't needed... I think one of these migrate_vma_* functions will trigger all MMU notifiers registered for the range. The notifier owns the invalidate then.
Beyond this, maybe I'm confused about a few things and how this fits all together. Doesn't every user process have its own unique mm, fd, and vm (e.g. own address space)? If a user want a shared address space then use threads with a single mm, fd, and vm.
So even if we had to resolve the xe_vma's here and do an invalidate here very confused what this is doing. This is this the case with multiple devices and each VM points to a different device? Again so that case I don't think a xe_svm structure would be needed, on GPU fault we should be to detect from the faulting page zone_device_data and pgmap owner if the fault already has a physical backing on another GPU and resolve how to map it into GPU with a fault... Jason suggests this in the following thread [2] and I think I agree with him.
drivers/gpu/drm/xe/xe_svm_migrate.c line 217
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+migrate_vma_pages (&migrate_vma);
This logic is going to change but ...
On an error I think we only want to call migrate_vma_finalize to revert pages back to the original state (i.e. migrate_vma_pages commits the page changes which we don't want to do on an error).
drivers/gpu/drm/xe/xe_svm_migrate.c line 221
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +4
-0+migrate_vma_finalize (&migrate_vma); +free_buf: +kvfree (buf); +return 0 ;
I don't think 0 should blindly be return here, if there is an error return VM_FAULT_SIGBUS. We likely want a high level error message too.
Matt
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+}Signature · 1 line
2.26.3
RE: [v2 27/31] drm/xe/svm: Handle CPU page fault
Quoted 369 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 10:07 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 27/31] drm/xe/svm: Handle CPU page faultQuoted 8 lines · Oak Zeng
Under the picture of svm, CPU and GPU program share one same virtual address space. The backing store of this virtual address space can be either in system memory or device memory. Since GPU device memory is remaped as
DEVICE_PRIVATE, CPU can't access it. Any CPU access to device memory causes a page fault. Implement a page fault handler to migrate memory back to system memory and map it to CPU page table so the CPU program can proceed.Quoted 5 lines
Also unbind this page from GPU side, and free the original GPU device page
Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana VishwanathapuraQuoted 1 line
Signed-off-by Niranjana VishwanathapuraQuoted 8 lines
Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_svm.h | 8 +- drivers/gpu/drm/xe/xe_svm_devmem.c | 7 +- drivers/gpu/drm/xe/xe_svm_migrate.c | 222++++++++++++++++++++++++++++
drivers/gpu/drm/xe/Makefile xe-y += xe_bb.o \ · line 134
Quoted 12 lines
4 files changed, 230 insertions(+), 8 deletions(-) create mode 100644 drivers/gpu/drm/xe/xe_svm_migrate.cdrivers/gpu/drm/xe/Makefile +1
-0@@ -131,6 +131,7 @@ xe-y += xe_bb.o \ xe_step.o \ xe_svm.o \ xe_svm_devmem.o \ + xe_svm_migrate.o \See comments about file org, same thing applies here. Let's put all of the svm implementation in a single file.
Quoted 22 lines
drivers/gpu/drm/xe/Makefile +0
-0xe_sync.o \ xe_tile.o \ xe_tile_sysfs.o \drivers/gpu/drm/xe/xe_svm.h +2
-2@@ -7,11 +7,11 @@#define __XE_SVM_H#include <linux/mm_types.h> +#include <linux/mm.h> #include "xe_device_types.h" #include "xe_device.h" #include "xe_assert.h" - -struct xe_vm ; +#include "xe_vm_types.h" /** * struct xe_svm - data structure to represent a shared drivers/gpu/drm/xe/xe_svm.h struct xe_svm { · line 34
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm.h +1
-0@@ -31,6 +31,9 @@ struct xe_svm {struct list_head vm_list; }; +#define xe_svm_for_each_vm (svm, vm)\drivers/gpu/drm/xe/xe_svm.h struct xe_svm { · line 36
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm.h +2
-0+list_for_each_entry (vm, &svm->vm_list, svm_link) +Don't think this is need, see below.
Review
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm.h +1
-0extern struct xe_svm *xe_create_svm (void );void xe_destroy_svm (struct xe_svm *svm);extern struct xe_svm *xe_lookup_svm_by_mm (struct mm_struct *mm);@@ -79,4 +82,5 @@ int xe_devm_alloc_pages(struct xe_tile *tile,void xe_devm_free_blocks (struct list_head *blocks);void xe_devm_page_free (struct page *page); +vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf);#endif (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_devmem.cQuoted 16 lines
index
088ac209ad80..32ada458f1dd100644drivers/gpu/drm/xe/xe_svm_devmem.c +0
-5@@ -37,11 +37,6 @@ struct xe_svm_block_meta {unsigned long bitmap[]; }; -static vm_fault_t xe_devm_migrate_to_ram (struct vm_fault *vmf) -{ -return 0 ; -} -static u64 block_offset_to_pfn (struct xe_mem_region *mr,u64 offset) {/** DRM buddy's block offset is 0-based*/ drivers/gpu/drm/xe/xe_svm_devmem.c void xe_devm_free_blocks(struct list_head *blocks) · line 166
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +1
-1@@ -168,7 +163,7 @@ void xe_devm_free_blocks(struct list_head *blocks)static const struct dev_pagemap_ops xe_devm_pagemap_ops = { .page_free = xe_devm_page_free, - .migrate_to_ram = xe_devm_migrate_to_ram, + .migrate_to_ram = xe_svm_migrate_to_sram,Again single file so this will be static function, no reason to export this.
Review
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +0
-0};/** (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_migrate.cQuoted 42 lines
new file mode 100644
index 000000000000..0db831af098edrivers/gpu/drm/xe/xe_svm_migrate.c new file +37
-0@@ -0,0 +1,222 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2023 Intel Corporation +*/ + +#include <linux/gfp.h> +#include <linux/migrate.h> +#include <linux/dma-mapping.h> +#include <linux/dma-fence.h> +#include <linux/bitops.h> +#include <linux/bitmap.h> +#include <linux/kernel.h> +#include <linux/slab.h> +#include <drm/drm_buddy.h> +#include "xe_device_types.h" +#include "xe_device.h" +#include "xe_trace.h" +#include "xe_migrate.h" +#include "xe_ttm_vram_mgr_types.h" +#include "xe_assert.h" +#include "xe_pt.h" +#include "xe_svm.h" +#include "xe_vm.h" + + +/** +* alloc_host_page() - allocate one host page for the fault vma +* +* @dev: (GPU) device that will access the allocated page +* @vma: the fault vma that we need allocate page for +* @addr: the fault address. The allocated page is for this address +* @dma_addr: used to output the dma address of the allocated page. +* This dma address will be used for gpu to access this page. GPU +* access host page through a dma mapped address. +* @pfn: used to output the pfn of the allocated page. +* +* This function allocate one host page for the specified vma. It drivers/gpu/drm/xe/xe_svm_migrate.c line 47
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+* also does some prepare work for GPU to access this page, such +* as map this page to iommu (by calling dma_map_page). +* +* When this function returns, the page is locked. +* +* Return struct page pointer when success +* NULL otherwise +*/ +static struct page *alloc_host_page (struct device *dev, + struct vm_area_struct*vma,
drivers/gpu/drm/xe/xe_svm_migrate.c line 49
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+unsigned long addr, + dma_addr_t
*dma_addr,drivers/gpu/drm/xe/xe_svm_migrate.c line 50
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+ unsigned long *pfn)Weird alignment, also I don't think we are want to allocate a page at time...
Beyond that, can't say I'm a fan of 2 arguments being return and populated here either (
dma_addr_t*dma_addr, unsigned long *pfn). I haven't seen a lot that style function in Linux.Probably makes more sense to have a function which allocates pages, locks them, and populates the pfn array (
migrate_pfn) rather than doing this a page at a time.drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+{ +struct page *page; + + page =alloc_page_vma (GFP_HIGHUSER , vma, addr); +if (unlikely (!page)) +return NULL ; + +/**Lock page per hmm requirement, see hmm.rst*/ +lock_page (page); + *dma_addr = dma_map_page(dev, page, 0, PAGE_SIZE,
DMA_FROM_DEVICE);The device is writing to these pages so I think
DMA_BIDIRECTIONALis needed, right? As mentioned above I think this should be broken out into a different step too.Quoted 22 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +22
-0+if (unlikely (dma_mapping_error (dev, *dma_addr))) { +unlock_page (page); +__free_page (page); +return NULL ; + } + + *pfn =migrate_pfn (page_to_pfn (page)); +return page; +} + +static void free_host_page (struct page *page) +{ +unlock_page (page); +put_page (page); +} + +/** +* migrate_page_vram_to_ram() - migrate one page from vram to ram +* +* @vma: The vma that the page is mapped to +* @addr: The virtual address that the page is mapped to +* @src_pfn: src page's page frame number drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+* @dst_pfn: used to return dstination page (in system ram)'s pfn +* +* Allocate one page in system ram and copy memory from device memory +* to system ram. +* +* Return: 0 if this page is already in sram (no need to migrate) +* 1: successfully migrated this page from vram to sram. +* error code otherwise +*/ +static int migrate_page_vram_to_ram(struct vm_area_struct *vma,unsigned long addr,
drivers/gpu/drm/xe/xe_svm_migrate.c line 93
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+ unsigned long src_pfn,unsigned long
*dst_pfn)drivers/gpu/drm/xe/xe_svm_migrate.c line 94
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+{We definitely don't want to copy 1 page at time. I touch on this in [1]. Basically this going to perform poorly unless we use larger copies, the migrate code supports non-contigous sram addresses, and vram addresses will likely be contigous due to the buddy allocator.
[1] https://patchwork.freedesktop.org/patch/588548/?series=132229&rev=1
drivers/gpu/drm/xe/xe_svm_migrate.c line 106
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +12
-0+struct xe_mem_region *mr; +struct xe_tile *tile; +struct xe_device *xe; +struct device *dev; +dma_addr_t dma_addr =0 ; +struct dma_fence *fence; +struct page *host_page; +struct page *src_page; +u64 src_dpa; + + src_page =migrate_pfn_to_page (src_pfn); +if (unlikely (!src_page || !(src_pfn &MIGRATE_PFN_MIGRATE )))I'm going to say this is a bug if !src_page || !is_zone_device_page(src_page) || !(src_pfn &
MIGRATE_PFN_MIGRATE) and we return-EFAULT(or another error code if that makes more sense). We are migrating from the device where we know we have backing store from the original fault.Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +6
-0+return 0 ; + + mr =xe_page_to_mem_region (src_page); + tile =xe_mem_region_to_tile (mr); + xe =tile_to_xe (tile); + dev = xe->drm.dev;drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+ + src_dpa =xe_mem_region_pfn_to_dpa (mr, src_pfn); + host_page =alloc_host_page (dev, vma, addr, &dma_addr, dst_pfn); +if (!host_page) +return -ENOMEM ; + + fence =xe_migrate_pa (tile->migrate, src_dpa, true, + dma_addr, false,PAGE_SIZE ); +if (IS_ERR (fence)) { + dma_unmap_page(dev, dma_addr, PAGE_SIZE,
DMA_FROM_DEVICE);drivers/gpu/drm/xe/xe_svm_migrate.c line 127
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +5
-0+free_host_page (host_page); +return PTR_ERR (fence); + } + +dma_fence_wait (fence, false);Even if we did want to migrate a page at a time, we only need to wait on the last fence due to the ordered nature of exec queues.
drivers/gpu/drm/xe/xe_svm_migrate.c line 129
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+dma_fence_put (fence); +dma_unmap_page (dev, dma_addr,PAGE_SIZE ,DMA_FROM_DEVICE );With above, will likely unmap all dma pages in a single function once the last fence is signaled.
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +5
-0+return 1 ; +} + +/** +* xe_svm_migrate_to_sram() - Migrate memory back to sram on CPU page fault
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +6
-0+* +* @vmf: cpu vm fault structure, contains fault information such as vma etc. +* +* Note, this is in CPU's vm fault handler, caller holds the mmap read lock. +* +* This function migrate one gpu vma which contains the fault address to sram.
drivers/gpu/drm/xe/xe_svm_migrate.c line 141
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* We try to maintain a 1:1 mapping b/t the CPU vma and gpu vma (i.e., create one
drivers/gpu/drm/xe/xe_svm_migrate.c line 142
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* gpu vma for one cpu vma initially and try not to split it). So this scheme end
drivers/gpu/drm/xe/xe_svm_migrate.c line 143
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* up migrate at the vma granularity. This might not be the best performant scheme
drivers/gpu/drm/xe/xe_svm_migrate.c line 145
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+* +* This can be tunned with a migration granularity for performance, for example,
drivers/gpu/drm/xe/xe_svm_migrate.c line 146
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* migration 2M for each CPU page fault, or let user specify how much to migrate.
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +3
-0+* This is more complex due to vma splitting. +* +* This function should also update GPU page table, so the fault virtual address
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +5
-0+* points to the same sram location from GPU side. This is TBD. +* +* Return: +* 0 on success +* VM_FAULT_SIGBUS: failed to migrate page to system memory, application
drivers/gpu/drm/xe/xe_svm_migrate.c line 164
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+* will be signaled a SIGBUG +*/ +vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf) +{ +struct xe_mem_region *mr =xe_page_to_mem_region (vmf->page); +struct xe_tile *tile =xe_mem_region_to_tile (mr); +struct xe_device *xe =tile_to_xe (tile); +struct vm_area_struct *vma = vmf->vma; +struct mm_struct *mm = vma->vm_mm; +struct xe_svm *svm =xe_lookup_svm_by_mm (mm);I don't think this is needed... More below.
drivers/gpu/drm/xe/xe_svm_migrate.c line 176
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +12
-0+unsigned long addr = vma->vm_start; +u64 npages =vma_pages (vma); +struct xe_vma *xe_vma; +vm_fault_t ret =0 ; +struct xe_vm *vm; +void *buf; +int i; + +struct migrate_vma migrate_vma = { + .vma = vmf->vma, + .start = vma->vm_start, + .end = vma->vm_end,So I know in my PoC I had the fault user pointer (
xe_vma) == structvm_area_structof the GPU fault. That is definitely wrong. We likely
Quoted 8 lines
want to allocate sub-range of
vm_area_structfor thexe_vma, we can call this a chunk size. Logical chunks sizes would be aligned 2MB, 64k, and finally 4k in sizes trying the largest first... The chunk sizes are trivial as we likely can just have table with values, the key here is thevm_area_structvm_start/vm_endare not what we want to use here ratherxe_vma_start(vma)andxe_vma_end(vma). I think we get thexe_vmafrom the faulting pagevmf->page->zone_device_datafield unless you have another use that field...
You are right. Such work is planned in the memory attributes part that Himal is working on. We have a migration_granularity attribute which allow user to set the chunk size.
Quoted 58 lines
I also comment on my patch with my suggestion implement chunk sizes too.
drivers/gpu/drm/xe/xe_svm_migrate.c line 177
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+ .pgmap_owner = xe,Again helper for this.
drivers/gpu/drm/xe/xe_svm_migrate.c line 190
Quoted 13 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +13
-0+ .flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE, + .fault_page = vmf->page, + }; + + buf =kvcalloc (npages,2 *sizeof (*migrate_vma.src),GFP_KERNEL ); + migrate_vma.src = buf; + migrate_vma.dst = buf + npages; +if (migrate_vma_setup (&migrate_vma) <0 ) { + ret =VM_FAULT_SIGBUS ; + goto free_buf; + } + +if (!migrate_vma.cpages)This is an error, need to set a return value.
drivers/gpu/drm/xe/xe_svm_migrate.c line 192
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+ goto free_buf; +We probably should check migrate.cpages != npages too as I also think this is an error.
drivers/gpu/drm/xe/xe_svm_migrate.c line 194
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+for (i =0 ; i < npages; i++) { + ret = migrate_page_vram_to_ram(vma, addr,migrate_vma.src[i],
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 9 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +9
-0+ migrate_vma.dst + i); +if (ret <0 ) { + ret =VM_FAULT_SIGBUS ; +break ; + } + +/** Migration has been successful, free source page */ +if (ret ==1 ) { + struct page *src_page =
migrate_pfn_to_page(migrate_vma.src[i]);drivers/gpu/drm/xe/xe_svm_migrate.c line 209
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +6
-0+ +xe_devm_page_free (src_page); + } + + addr +=PAGE_SIZE ; + }I touch on this above, this should be reworked to roughly:
- alloc pages and populate migrate_vma.dst
- dma map sram pages
- migrate a chunk of contigous vram addresses at a time
- wait on last dma fence from migrate
- unmap dma pages
- unlock and free all pages
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm_migrate.c line 214
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +7
-0+ +xe_svm_for_each_vm (svm, vm) { +xe_assert (xe, vm->mm == mm); + xe_vma =xe_vm_lookup_vma (vm, vmf->address); +if (xe_vma)Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
I've touched on why this isn't needed... I think one of these migrate_vma_* functions will trigger all MMU notifiers registered for the range. The notifier owns the invalidate then.
Very good point. Yes after read migrate_vma_setup function, I agree this function will call mmu notifiers with MMU_NOTIFY_MIGRATE event. Today we invalidate vma with this event. So yes, above codes are not needed.
I do identified one potential improvement: when mmu notifier is called back with MMU_NOTIFY_MIGRATE event, if the migrate_vma_setup is called from the gpu page fault path, we can ignore the gpu vma invalidation as we will update gpu page table later after migration anyway. I think an page table invalidation is not needed in such case. But this should be just a minor improvement.
Quoted 5 lines
Beyond this, maybe I'm confused about a few things and how this fits all together. Doesn't every user process have its own unique mm, fd, and vm (e.g. own address space)? If a user want a shared address space then use threads with a single mm, fd, and vm.
Yes, this is also my understanding. Each user process has its own mm struct and device fds.
In a shared address space case, such as there are multiple pthread created through pthread_create in one process, all those pthreads should have different kernel task_struct, but all those task_struct (say we get it from "current" macro) should share one same mm struct, which means they all lives inside one cpu address space.
Now with this work, we are now basically extending this shared cpu address space to gpu program. So both cpu program and gpu program can share one address space.
Since we allow user to create multiple gpu vm for one device (lets focus on one device for now), so each shared address space can have multiple gpu vm... each gpuvm should be able to register its own mmu notifier to core mm, even if those notifier has the same address range. But I will have to test this out. If all this works, above codes are not needed. If different gpuvm can't register mmu notifier for same address range, then we would need a fix....
Quoted 4 lines
So even if we had to resolve the xe_vma's here and do an invalidate here very confused what this is doing. This is this the case with multiple devices and each VM points to a different device?
Right now I only focus on single device. See above. This is to solve one gpu device but multiple gpu vm case. But as said above, for now I don't think this is needed. I need to test more on the mmu notifier behavior: whether it allow us to insert two notifiers for the same range for one mm....
Oak
Again so that case I
Quoted 31 lines
don't think a
xe_svmstructure would be needed, on GPU fault we should be to detect from the faulting pagezone_device_dataand pgmap owner if the fault already has a physical backing on another GPU and resolve how to map it into GPU with a fault... Jason suggests this in the following thread [2] and I think I agree with him.[2] https://lore.kernel.org/all/5495090e-dee1-4c8e-91bc-
[email protected]/T/drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
This logic is going to change but ...
On an error I think we only want to call
migrate_vma_finalizeto revert pages back to the original state (i.e.migrate_vma_pagescommits the page changes which we don't want to do on an error).11 lines not shown · raw message
Parts of this message are not shown here. View the raw message.
Matthew Brost
maintainerRe: [v2 27/31] drm/xe/svm: Handle CPU page fault
Quoted 367 lines · Oak
Quoted 367 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 10:07 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 27/31] drm/xe/svm: Handle CPU page faultQuoted 8 lines · Oak Zeng
Under the picture of svm, CPU and GPU program share one same virtual address space. The backing store of this virtual address space can be either in system memory or device memory. Since GPU device memory is remaped as
DEVICE_PRIVATE, CPU can't access it. Any CPU access to device memory causes a page fault. Implement a page fault handler to migrate memory back to system memory and map it to CPU page table so the CPU program can proceed.Quoted 5 lines
Also unbind this page from GPU side, and free the original GPU device page
Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana VishwanathapuraQuoted 1 line
Signed-off-by Niranjana VishwanathapuraQuoted 8 lines
Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_svm.h | 8 +- drivers/gpu/drm/xe/xe_svm_devmem.c | 7 +- drivers/gpu/drm/xe/xe_svm_migrate.c | 222++++++++++++++++++++++++++++
drivers/gpu/drm/xe/Makefile xe-y += xe_bb.o \ · line 134
Quoted 12 lines
4 files changed, 230 insertions(+), 8 deletions(-) create mode 100644 drivers/gpu/drm/xe/xe_svm_migrate.cdrivers/gpu/drm/xe/Makefile +1
-0@@ -131,6 +131,7 @@ xe-y += xe_bb.o \ xe_step.o \ xe_svm.o \ xe_svm_devmem.o \ + xe_svm_migrate.o \See comments about file org, same thing applies here. Let's put all of the svm implementation in a single file.
Quoted 22 lines
drivers/gpu/drm/xe/Makefile +0
-0xe_sync.o \ xe_tile.o \ xe_tile_sysfs.o \drivers/gpu/drm/xe/xe_svm.h +2
-2@@ -7,11 +7,11 @@#define __XE_SVM_H#include <linux/mm_types.h> +#include <linux/mm.h> #include "xe_device_types.h" #include "xe_device.h" #include "xe_assert.h" - -struct xe_vm ; +#include "xe_vm_types.h" /** * struct xe_svm - data structure to represent a shared drivers/gpu/drm/xe/xe_svm.h struct xe_svm { · line 34
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm.h +1
-0@@ -31,6 +31,9 @@ struct xe_svm {struct list_head vm_list; }; +#define xe_svm_for_each_vm (svm, vm)\drivers/gpu/drm/xe/xe_svm.h struct xe_svm { · line 36
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm.h +2
-0+list_for_each_entry (vm, &svm->vm_list, svm_link) +Don't think this is need, see below.
Review
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm.h +1
-0extern struct xe_svm *xe_create_svm (void );void xe_destroy_svm (struct xe_svm *svm);extern struct xe_svm *xe_lookup_svm_by_mm (struct mm_struct *mm);@@ -79,4 +82,5 @@ int xe_devm_alloc_pages(struct xe_tile *tile,void xe_devm_free_blocks (struct list_head *blocks);void xe_devm_page_free (struct page *page); +vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf);#endif (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_devmem.cQuoted 16 lines
index
088ac209ad80..32ada458f1dd100644drivers/gpu/drm/xe/xe_svm_devmem.c +0
-5@@ -37,11 +37,6 @@ struct xe_svm_block_meta {unsigned long bitmap[]; }; -static vm_fault_t xe_devm_migrate_to_ram (struct vm_fault *vmf) -{ -return 0 ; -} -static u64 block_offset_to_pfn (struct xe_mem_region *mr,u64 offset) {/** DRM buddy's block offset is 0-based*/ drivers/gpu/drm/xe/xe_svm_devmem.c void xe_devm_free_blocks(struct list_head *blocks) · line 166
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +1
-1@@ -168,7 +163,7 @@ void xe_devm_free_blocks(struct list_head *blocks)static const struct dev_pagemap_ops xe_devm_pagemap_ops = { .page_free = xe_devm_page_free, - .migrate_to_ram = xe_devm_migrate_to_ram, + .migrate_to_ram = xe_svm_migrate_to_sram,Again single file so this will be static function, no reason to export this.
Review
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +0
-0};/** (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_migrate.cQuoted 42 lines
new file mode 100644
index 000000000000..0db831af098edrivers/gpu/drm/xe/xe_svm_migrate.c new file +37
-0@@ -0,0 +1,222 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2023 Intel Corporation +*/ + +#include <linux/gfp.h> +#include <linux/migrate.h> +#include <linux/dma-mapping.h> +#include <linux/dma-fence.h> +#include <linux/bitops.h> +#include <linux/bitmap.h> +#include <linux/kernel.h> +#include <linux/slab.h> +#include <drm/drm_buddy.h> +#include "xe_device_types.h" +#include "xe_device.h" +#include "xe_trace.h" +#include "xe_migrate.h" +#include "xe_ttm_vram_mgr_types.h" +#include "xe_assert.h" +#include "xe_pt.h" +#include "xe_svm.h" +#include "xe_vm.h" + + +/** +* alloc_host_page() - allocate one host page for the fault vma +* +* @dev: (GPU) device that will access the allocated page +* @vma: the fault vma that we need allocate page for +* @addr: the fault address. The allocated page is for this address +* @dma_addr: used to output the dma address of the allocated page. +* This dma address will be used for gpu to access this page. GPU +* access host page through a dma mapped address. +* @pfn: used to output the pfn of the allocated page. +* +* This function allocate one host page for the specified vma. It drivers/gpu/drm/xe/xe_svm_migrate.c line 47
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+* also does some prepare work for GPU to access this page, such +* as map this page to iommu (by calling dma_map_page). +* +* When this function returns, the page is locked. +* +* Return struct page pointer when success +* NULL otherwise +*/ +static struct page *alloc_host_page (struct device *dev, + struct vm_area_struct*vma,
drivers/gpu/drm/xe/xe_svm_migrate.c line 49
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+unsigned long addr, + dma_addr_t
*dma_addr,drivers/gpu/drm/xe/xe_svm_migrate.c line 50
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+ unsigned long *pfn)Weird alignment, also I don't think we are want to allocate a page at time...
Beyond that, can't say I'm a fan of 2 arguments being return and populated here either (
dma_addr_t*dma_addr, unsigned long *pfn). I haven't seen a lot that style function in Linux.Probably makes more sense to have a function which allocates pages, locks them, and populates the pfn array (
migrate_pfn) rather than doing this a page at a time.drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+{ +struct page *page; + + page =alloc_page_vma (GFP_HIGHUSER , vma, addr); +if (unlikely (!page)) +return NULL ; + +/**Lock page per hmm requirement, see hmm.rst*/ +lock_page (page); + *dma_addr = dma_map_page(dev, page, 0, PAGE_SIZE,
DMA_FROM_DEVICE);The device is writing to these pages so I think
DMA_BIDIRECTIONALis needed, right? As mentioned above I think this should be broken out into a different step too.Quoted 22 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +22
-0+if (unlikely (dma_mapping_error (dev, *dma_addr))) { +unlock_page (page); +__free_page (page); +return NULL ; + } + + *pfn =migrate_pfn (page_to_pfn (page)); +return page; +} + +static void free_host_page (struct page *page) +{ +unlock_page (page); +put_page (page); +} + +/** +* migrate_page_vram_to_ram() - migrate one page from vram to ram +* +* @vma: The vma that the page is mapped to +* @addr: The virtual address that the page is mapped to +* @src_pfn: src page's page frame number drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+* @dst_pfn: used to return dstination page (in system ram)'s pfn +* +* Allocate one page in system ram and copy memory from device memory +* to system ram. +* +* Return: 0 if this page is already in sram (no need to migrate) +* 1: successfully migrated this page from vram to sram. +* error code otherwise +*/ +static int migrate_page_vram_to_ram(struct vm_area_struct *vma,unsigned long addr,
drivers/gpu/drm/xe/xe_svm_migrate.c line 93
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+ unsigned long src_pfn,unsigned long
*dst_pfn)drivers/gpu/drm/xe/xe_svm_migrate.c line 94
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+{We definitely don't want to copy 1 page at time. I touch on this in [1]. Basically this going to perform poorly unless we use larger copies, the migrate code supports non-contigous sram addresses, and vram addresses will likely be contigous due to the buddy allocator.
[1] https://patchwork.freedesktop.org/patch/588548/?series=132229&rev=1
drivers/gpu/drm/xe/xe_svm_migrate.c line 106
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +12
-0+struct xe_mem_region *mr; +struct xe_tile *tile; +struct xe_device *xe; +struct device *dev; +dma_addr_t dma_addr =0 ; +struct dma_fence *fence; +struct page *host_page; +struct page *src_page; +u64 src_dpa; + + src_page =migrate_pfn_to_page (src_pfn); +if (unlikely (!src_page || !(src_pfn &MIGRATE_PFN_MIGRATE )))I'm going to say this is a bug if !src_page || !is_zone_device_page(src_page) || !(src_pfn &
MIGRATE_PFN_MIGRATE) and we return-EFAULT(or another error code if that makes more sense). We are migrating from the device where we know we have backing store from the original fault.Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +6
-0+return 0 ; + + mr =xe_page_to_mem_region (src_page); + tile =xe_mem_region_to_tile (mr); + xe =tile_to_xe (tile); + dev = xe->drm.dev;drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+ + src_dpa =xe_mem_region_pfn_to_dpa (mr, src_pfn); + host_page =alloc_host_page (dev, vma, addr, &dma_addr, dst_pfn); +if (!host_page) +return -ENOMEM ; + + fence =xe_migrate_pa (tile->migrate, src_dpa, true, + dma_addr, false,PAGE_SIZE ); +if (IS_ERR (fence)) { + dma_unmap_page(dev, dma_addr, PAGE_SIZE,
DMA_FROM_DEVICE);drivers/gpu/drm/xe/xe_svm_migrate.c line 127
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +5
-0+free_host_page (host_page); +return PTR_ERR (fence); + } + +dma_fence_wait (fence, false);Even if we did want to migrate a page at a time, we only need to wait on the last fence due to the ordered nature of exec queues.
drivers/gpu/drm/xe/xe_svm_migrate.c line 129
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+dma_fence_put (fence); +dma_unmap_page (dev, dma_addr,PAGE_SIZE ,DMA_FROM_DEVICE );With above, will likely unmap all dma pages in a single function once the last fence is signaled.
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +5
-0+return 1 ; +} + +/** +* xe_svm_migrate_to_sram() - Migrate memory back to sram on CPU page fault
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +6
-0+* +* @vmf: cpu vm fault structure, contains fault information such as vma etc. +* +* Note, this is in CPU's vm fault handler, caller holds the mmap read lock. +* +* This function migrate one gpu vma which contains the fault address to sram.
drivers/gpu/drm/xe/xe_svm_migrate.c line 141
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* We try to maintain a 1:1 mapping b/t the CPU vma and gpu vma (i.e., create one
drivers/gpu/drm/xe/xe_svm_migrate.c line 142
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* gpu vma for one cpu vma initially and try not to split it). So this scheme end
drivers/gpu/drm/xe/xe_svm_migrate.c line 143
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* up migrate at the vma granularity. This might not be the best performant scheme
drivers/gpu/drm/xe/xe_svm_migrate.c line 145
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+* +* This can be tunned with a migration granularity for performance, for example,
drivers/gpu/drm/xe/xe_svm_migrate.c line 146
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* migration 2M for each CPU page fault, or let user specify how much to migrate.
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +3
-0+* This is more complex due to vma splitting. +* +* This function should also update GPU page table, so the fault virtual address
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +5
-0+* points to the same sram location from GPU side. This is TBD. +* +* Return: +* 0 on success +* VM_FAULT_SIGBUS: failed to migrate page to system memory, application
drivers/gpu/drm/xe/xe_svm_migrate.c line 164
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+* will be signaled a SIGBUG +*/ +vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf) +{ +struct xe_mem_region *mr =xe_page_to_mem_region (vmf->page); +struct xe_tile *tile =xe_mem_region_to_tile (mr); +struct xe_device *xe =tile_to_xe (tile); +struct vm_area_struct *vma = vmf->vma; +struct mm_struct *mm = vma->vm_mm; +struct xe_svm *svm =xe_lookup_svm_by_mm (mm);I don't think this is needed... More below.
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +12
-0+unsigned long addr = vma->vm_start; +u64 npages =vma_pages (vma); +struct xe_vma *xe_vma; +vm_fault_t ret =0 ; +struct xe_vm *vm; +void *buf; +int i; + +struct migrate_vma migrate_vma = { + .vma = vmf->vma, + .start = vma->vm_start, + .end = vma->vm_end,
Quoted 8 lines
Quoted 8 lines
So I know in my PoC I had the fault user pointer (
xe_vma) == structvm_area_structof the GPU fault. That is definitely wrong. We likely want to allocate sub-range ofvm_area_structfor thexe_vma, we can call this a chunk size. Logical chunks sizes would be aligned 2MB, 64k, and finally 4k in sizes trying the largest first... The chunk sizes are trivial as we likely can just have table with values, the key here is thevm_area_structvm_start/vm_endare not what we want to use here ratherxe_vma_start(vma)andxe_vma_end(vma). I think we get thexe_vma
After I typed this, realized I made a mistake here...
s/xe_vma_start/xe_vma_userptr/
s/xe_vma_end/xe_vma_userptr + xe_vma_size/
But you get the idea - the zone_device_data points Xe specific chunk data (currently xe_vma, could be xe_pt_state our something later if we switch to 1:N).
Check AMD's + Nvidia's drivers and they both use this field in a similar way.
Quoted 5 lines
Quoted 2 lines
from the faulting page
vmf->page->zone_device_datafield unless you have another use that field...You are right. Such work is planned in the memory attributes part that Himal is working on. We have a
migration_granularityattribute which allow user to set the chunk size.
That makes sense. The chunk size is always just hint though, right?
Quoted 70 lines
Quoted 58 lines
I also comment on my patch with my suggestion implement chunk sizes too.
drivers/gpu/drm/xe/xe_svm_migrate.c line 177
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+ .pgmap_owner = xe,Again helper for this.
drivers/gpu/drm/xe/xe_svm_migrate.c line 190
Quoted 13 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +13
-0+ .flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE, + .fault_page = vmf->page, + }; + + buf =kvcalloc (npages,2 *sizeof (*migrate_vma.src),GFP_KERNEL ); + migrate_vma.src = buf; + migrate_vma.dst = buf + npages; +if (migrate_vma_setup (&migrate_vma) <0 ) { + ret =VM_FAULT_SIGBUS ; + goto free_buf; + } + +if (!migrate_vma.cpages)This is an error, need to set a return value.
drivers/gpu/drm/xe/xe_svm_migrate.c line 192
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+ goto free_buf; +We probably should check migrate.cpages != npages too as I also think this is an error.
drivers/gpu/drm/xe/xe_svm_migrate.c line 194
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+for (i =0 ; i < npages; i++) { + ret = migrate_page_vram_to_ram(vma, addr,migrate_vma.src[i],
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 9 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +9
-0+ migrate_vma.dst + i); +if (ret <0 ) { + ret =VM_FAULT_SIGBUS ; +break ; + } + +/** Migration has been successful, free source page */ +if (ret ==1 ) { + struct page *src_page =
migrate_pfn_to_page(migrate_vma.src[i]);drivers/gpu/drm/xe/xe_svm_migrate.c line 204
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +6
-0+Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
I touch on this above, this should be reworked to roughly:
- alloc pages and populate migrate_vma.dst
- dma map sram pages
- migrate a chunk of contigous vram addresses at a time
- wait on last dma fence from migrate
- unmap dma pages
- unlock and free all pagesQuoted 11 lines
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +7
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
I've touched on why this isn't needed... I think one of these migrate_vma_* functions will trigger all MMU notifiers registered for the range. The notifier owns the invalidate then.
Quoted 4 lines
Very good point. Yes after read
migrate_vma_setupfunction, I agree this function will call mmu notifiers withMMU_NOTIFY_MIGRATEevent. Today we invalidate vma with this event. So yes, above codes are not needed.I do identified one potential improvement: when mmu notifier is called back with
MMU_NOTIFY_MIGRATEevent, if themigrate_vma_setupis called from the gpu page fault path, we can ignore the gpu vma invalidation as we will update gpu page table later after migration anyway. I think an page table invalidation is not needed in such case. But this should be just a minor improvement.
We skip invalidations if the initial_bind flag is clear which should cover at the initial GPU fault. There is certainly room for improvement / optimizations in the MMU notifier though, it is kinda messy right now too. IMO work like this can be done once the basic design is working + tests in place to verify changes / optimizations.
Quoted 7 lines
Quoted 6 lines
Beyond this, maybe I'm confused about a few things and how this fits all together. Doesn't every user process have its own unique mm, fd, and vm (e.g. own address space)? If a user want a shared address space then use threads with a single mm, fd, and vm.
Quoted 8 lines
Yes, this is also my understanding. Each user process has its own mm struct and device fds.
In a shared address space case, such as there are multiple pthread created through
pthread_createin one process, all those pthreads should have different kerneltask_struct, but all thosetask_struct(say we get it from "current" macro) should share one same mm struct, which means they all lives inside one cpu address space.Now with this work, we are now basically extending this shared cpu address space to gpu program. So both cpu program and gpu program can share one address space.
Since we allow user to create multiple gpu vm for one device (lets focus on one device for now), so each shared address space can have multiple gpu vm... each gpuvm should be able to register its own mmu notifier to core mm, even if those notifier has the same address range. But I will have to test this out. If all this works, above codes are not needed. If different gpuvm can't register mmu notifier for same address range, then we would need a fix....
The mmu notifier code is implemented with the interval tree which supports overlapping rangers (i.e. we can have multiple VM's register notifiers with the sam address range in a single MM).
Quoted 8 lines
Quoted 5 lines
So even if we had to resolve the xe_vma's here and do an invalidate here very confused what this is doing. This is this the case with multiple devices and each VM points to a different device?
Right now I only focus on single device. See above. This is to solve one gpu device but multiple gpu vm case. But as said above, for now I don't think this is needed. I need to test more on the mmu notifier behavior: whether it allow us to insert two notifiers for the same range for one mm....
Agree that our focus should be on single device now. If that design it well thought out I don't think extending this to multiple devices will be a huge change either.
Matt
Quoted 34 lines
Oak
Again so that case I
Quoted 31 lines
don't think a
xe_svmstructure would be needed, on GPU fault we should be to detect from the faulting pagezone_device_dataand pgmap owner if the fault already has a physical backing on another GPU and resolve how to map it into GPU with a fault... Jason suggests this in the following thread [2] and I think I agree with him.18 lines not shown · raw message
Parts of this message are not shown here. View the raw message.
RE: [v2 27/31] drm/xe/svm: Handle CPU page fault
Quoted 415 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Friday, April 12, 2024 2:11 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 27/31] drm/xe/svm: Handle CPU page faultQuoted 104 lines · Oak
Quoted 104 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 10:07 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 27/31] drm/xe/svm: Handle CPU page faultQuoted 8 lines · Oak Zeng
Under the picture of svm, CPU and GPU program share one same virtual address space. The backing store of this virtual address space can be either in system memory or device memory. Since GPU device memory is remaped as
DEVICE_PRIVATE, CPU can't access it. Any CPU access to device memory causes a page fault. Implement a page fault handler to migrate memory back to system memory and map it to CPU page table so the CPU program can proceed.Quoted 5 lines
Also unbind this page from GPU side, and free the original GPU device page
Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana VishwanathapuraQuoted 1 line
Signed-off-by Niranjana VishwanathapuraQuoted 8 lines
Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_svm.h | 8 +- drivers/gpu/drm/xe/xe_svm_devmem.c | 7 +- drivers/gpu/drm/xe/xe_svm_migrate.c | 222++++++++++++++++++++++++++++
drivers/gpu/drm/xe/Makefile xe-y += xe_bb.o \ · line 134
Quoted 12 lines
4 files changed, 230 insertions(+), 8 deletions(-) create mode 100644 drivers/gpu/drm/xe/xe_svm_migrate.cdrivers/gpu/drm/xe/Makefile +1
-0@@ -131,6 +131,7 @@ xe-y += xe_bb.o \ xe_step.o \ xe_svm.o \ xe_svm_devmem.o \ + xe_svm_migrate.o \See comments about file org, same thing applies here. Let's put all of the svm implementation in a single file.
Quoted 22 lines
drivers/gpu/drm/xe/Makefile +0
-0xe_sync.o \ xe_tile.o \ xe_tile_sysfs.o \drivers/gpu/drm/xe/xe_svm.h +2
-2@@ -7,11 +7,11 @@#define __XE_SVM_H#include <linux/mm_types.h> +#include <linux/mm.h> #include "xe_device_types.h" #include "xe_device.h" #include "xe_assert.h" - -struct xe_vm ; +#include "xe_vm_types.h" /** * struct xe_svm - data structure to represent a shared drivers/gpu/drm/xe/xe_svm.h struct xe_svm { · line 34
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm.h +1
-0@@ -31,6 +31,9 @@ struct xe_svm {struct list_head vm_list; }; +#define xe_svm_for_each_vm (svm, vm)\drivers/gpu/drm/xe/xe_svm.h struct xe_svm { · line 36
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm.h +2
-0+list_for_each_entry (vm, &svm->vm_list, svm_link) +Don't think this is need, see below.
Review
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm.h +1
-0extern struct xe_svm *xe_create_svm (void );void xe_destroy_svm (struct xe_svm *svm);extern struct xe_svm *xe_lookup_svm_by_mm (struct mm_struct *mm);@@ -79,4 +82,5 @@ int xe_devm_alloc_pages(struct xe_tile *tile,void xe_devm_free_blocks (struct list_head *blocks);void xe_devm_page_free (struct page *page); +vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf);#endif (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_devmem.cQuoted 7 lines
index
088ac209ad80..32ada458f1dd100644drivers/gpu/drm/xe/xe_svm_devmem.c +0
-0@@ -37,11 +37,6 @@ struct xe_svm_block_meta {unsigned long bitmap[]; };Quoted 9 lines
Quoted 9 lines
drivers/gpu/drm/xe/xe_svm_devmem.c void xe_devm_free_blocks(struct list_head
Quoted 9 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +0
-5-static vm_fault_t xe_devm_migrate_to_ram (struct vm_fault *vmf) -{ -return 0 ; -} -static u64 block_offset_to_pfn (struct xe_mem_region *mr,u64 offset) {/** DRM buddy's block offset is 0-based*/ @@ -168,7 +163,7 @@ void xe_devm_free_blocks(struct list_head*blocks)
Quoted 116 lines
Quoted 116 lines
drivers/gpu/drm/xe/xe_svm_devmem.c void xe_devm_free_blocks(struct list_head *blocks) · line 166
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +1
-1static const struct dev_pagemap_ops xe_devm_pagemap_ops = { .page_free = xe_devm_page_free, - .migrate_to_ram = xe_devm_migrate_to_ram, + .migrate_to_ram = xe_svm_migrate_to_sram,Again single file so this will be static function, no reason to export this.
Review
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +0
-0};/** (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_migrate.cQuoted 42 lines
new file mode 100644
index 000000000000..0db831af098edrivers/gpu/drm/xe/xe_svm_migrate.c new file +37
-0@@ -0,0 +1,222 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2023 Intel Corporation +*/ + +#include <linux/gfp.h> +#include <linux/migrate.h> +#include <linux/dma-mapping.h> +#include <linux/dma-fence.h> +#include <linux/bitops.h> +#include <linux/bitmap.h> +#include <linux/kernel.h> +#include <linux/slab.h> +#include <drm/drm_buddy.h> +#include "xe_device_types.h" +#include "xe_device.h" +#include "xe_trace.h" +#include "xe_migrate.h" +#include "xe_ttm_vram_mgr_types.h" +#include "xe_assert.h" +#include "xe_pt.h" +#include "xe_svm.h" +#include "xe_vm.h" + + +/** +* alloc_host_page() - allocate one host page for the fault vma +* +* @dev: (GPU) device that will access the allocated page +* @vma: the fault vma that we need allocate page for +* @addr: the fault address. The allocated page is for this address +* @dma_addr: used to output the dma address of the allocated page. +* This dma address will be used for gpu to access this page. GPU +* access host page through a dma mapped address. +* @pfn: used to output the pfn of the allocated page. +* +* This function allocate one host page for the specified vma. It drivers/gpu/drm/xe/xe_svm_migrate.c line 47
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+* also does some prepare work for GPU to access this page, such +* as map this page to iommu (by calling dma_map_page). +* +* When this function returns, the page is locked. +* +* Return struct page pointer when success +* NULL otherwise +*/ +static struct page *alloc_host_page (struct device *dev, + struct vm_area_struct*vma,
drivers/gpu/drm/xe/xe_svm_migrate.c line 49
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+unsigned long addr, + dma_addr_t
*dma_addr,drivers/gpu/drm/xe/xe_svm_migrate.c line 50
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+ unsigned long *pfn)Weird alignment, also I don't think we are want to allocate a page at time...
Beyond that, can't say I'm a fan of 2 arguments being return and populated here either (
dma_addr_t*dma_addr, unsigned long *pfn). I haven't seen a lot that style function in Linux.Probably makes more sense to have a function which allocates pages, locks them, and populates the pfn array (
migrate_pfn) rather than doing this a page at a time.drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+{ +struct page *page; + + page =alloc_page_vma (GFP_HIGHUSER , vma, addr); +if (unlikely (!page)) +return NULL ; + +/**Lock page per hmm requirement, see hmm.rst*/ +lock_page (page); + *dma_addr = dma_map_page(dev, page, 0, PAGE_SIZE,
DMA_FROM_DEVICE);The device is writing to these pages so I think
DMA_BIDIRECTIONALis needed, right? As mentioned above I think this should be broken out into a different step too.Quoted 17 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +17
-0+if (unlikely (dma_mapping_error (dev, *dma_addr))) { +unlock_page (page); +__free_page (page); +return NULL ; + } + + *pfn =migrate_pfn (page_to_pfn (page)); +return page; +} + +static void free_host_page (struct page *page) +{ +unlock_page (page); +put_page (page); +} + +/** Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +8
-0+* migrate_page_vram_to_ram() - migrate one page from vram to ram +* +* @vma: The vma that the page is mapped to +* @addr: The virtual address that the page is mapped to +* @src_pfn: src page's page frame number +* @dst_pfn: used to return dstination page (in system ram)'s pfn +* +* Allocate one page in system ram and copy memory from device memory
Quoted 10 lines
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +7
-0+* to system ram. +* +* Return: 0 if this page is already in sram (no need to migrate) +* 1: successfully migrated this page from vram to sram. +* error code otherwise +*/ +static int migrate_page_vram_to_ram(struct vm_area_struct *vma,unsigned long addr,
drivers/gpu/drm/xe/xe_svm_migrate.c line 93
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+ unsigned long src_pfn,unsigned long
*dst_pfn)Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_migrate.c line 94
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+{We definitely don't want to copy 1 page at time. I touch on this in [1]. Basically this going to perform poorly unless we use larger copies, the migrate code supports non-contigous sram addresses, and vram addresses will likely be contigous due to the buddy allocator.
[1]
https://patchwork.freedesktop.org/patch/588548/?series=132229&rev=1
Quoted 16 lines
Quoted 16 lines
drivers/gpu/drm/xe/xe_svm_migrate.c line 106
Quoted 13 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +12
-0+struct xe_mem_region *mr; +struct xe_tile *tile; +struct xe_device *xe; +struct device *dev; +dma_addr_t dma_addr =0 ; +struct dma_fence *fence; +struct page *host_page; +struct page *src_page; +u64 src_dpa; + + src_page =migrate_pfn_to_page (src_pfn); +if (unlikely (!src_page || !(src_pfn &MIGRATE_PFN_MIGRATE )))I'm going to say this is a bug if !src_page || !is_zone_device_page(src_page) || !(src_pfn &
MIGRATE_PFN_MIGRATE)and
Quoted 36 lines
Quoted 36 lines
we return
-EFAULT(or another error code if that makes more sense). We are migrating from the device where we know we have backing store from the original fault.Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +6
-0+return 0 ; + + mr =xe_page_to_mem_region (src_page); + tile =xe_mem_region_to_tile (mr); + xe =tile_to_xe (tile); + dev = xe->drm.dev;drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+ + src_dpa =xe_mem_region_pfn_to_dpa (mr, src_pfn); + host_page =alloc_host_page (dev, vma, addr, &dma_addr, dst_pfn); +if (!host_page) +return -ENOMEM ; + + fence =xe_migrate_pa (tile->migrate, src_dpa, true, + dma_addr, false,PAGE_SIZE ); +if (IS_ERR (fence)) { + dma_unmap_page(dev, dma_addr, PAGE_SIZE,
DMA_FROM_DEVICE);drivers/gpu/drm/xe/xe_svm_migrate.c line 127
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +5
-0+free_host_page (host_page); +return PTR_ERR (fence); + } + +dma_fence_wait (fence, false);Even if we did want to migrate a page at a time, we only need to wait on the last fence due to the ordered nature of exec queues.
drivers/gpu/drm/xe/xe_svm_migrate.c line 129
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+dma_fence_put (fence); +dma_unmap_page (dev, dma_addr,PAGE_SIZE ,DMA_FROM_DEVICE );With above, will likely unmap all dma pages in a single function once the last fence is signaled.
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +5
-0+return 1 ; +} + +/** +* xe_svm_migrate_to_sram() - Migrate memory back to sram on CPU page
Quoted 3 lines
Quoted 3 lines
fault
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+* +* @vmf: cpu vm fault structure, contains fault information such as vma etc.
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c line 138
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+* +* Note, this is in CPU's vm fault handler, caller holds the mmap read lock.
Quoted 6 lines
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c line 140
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+* +* This function migrate one gpu vma which contains the fault address to sram.
drivers/gpu/drm/xe/xe_svm_migrate.c line 141
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* We try to maintain a 1:1 mapping b/t the CPU vma and gpu vma (i.e., create one
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* gpu vma for one cpu vma initially and try not to split it). So this scheme
Quoted 2 lines
Quoted 2 lines
end
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* up migrate at the vma granularity. This might not be the best performant
Quoted 5 lines
Quoted 5 lines
scheme
drivers/gpu/drm/xe/xe_svm_migrate.c line 145
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+* +* This can be tunned with a migration granularity for performance, for example,
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* migration 2M for each CPU page fault, or let user specify how much to
Quoted 37 lines
Quoted 37 lines
migrate.
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +3
-0+* This is more complex due to vma splitting. +* +* This function should also update GPU page table, so the fault virtual address
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +5
-0+* points to the same sram location from GPU side. This is TBD. +* +* Return: +* 0 on success +* VM_FAULT_SIGBUS: failed to migrate page to system memory, application
drivers/gpu/drm/xe/xe_svm_migrate.c line 164
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+* will be signaled a SIGBUG +*/ +vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf) +{ +struct xe_mem_region *mr =xe_page_to_mem_region (vmf->page); +struct xe_tile *tile =xe_mem_region_to_tile (mr); +struct xe_device *xe =tile_to_xe (tile); +struct vm_area_struct *vma = vmf->vma; +struct mm_struct *mm = vma->vm_mm; +struct xe_svm *svm =xe_lookup_svm_by_mm (mm);I don't think this is needed... More below.
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +12
-0+unsigned long addr = vma->vm_start; +u64 npages =vma_pages (vma); +struct xe_vma *xe_vma; +vm_fault_t ret =0 ; +struct xe_vm *vm; +void *buf; +int i; + +struct migrate_vma migrate_vma = { + .vma = vmf->vma, + .start = vma->vm_start,Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Quoted 8 lines
Quoted 8 lines
So I know in my PoC I had the fault user pointer (
xe_vma) == structvm_area_structof the GPU fault. That is definitely wrong. We likely want to allocate sub-range ofvm_area_structfor thexe_vma, we can call this a chunk size. Logical chunks sizes would be aligned 2MB, 64k, and finally 4k in sizes trying the largest first... The chunk sizes are trivial as we likely can just have table with values, the key here is thevm_area_structvm_start/vm_endare not what we want to use here ratherxe_vma_start(vma)andxe_vma_end(vma). I think we get the
xe_vmaAfter I typed this, realized I made a mistake here...
s/xe_vma_start/xe_vma_userptr/
s/xe_vma_end/xe_vma_userptr + xe_vma_size/But you get the idea - the
zone_device_datapoints Xe specific chunk data (currentlyxe_vma, could bexe_pt_stateour something later if we switch to 1:N).Check AMD's + Nvidia's drivers and they both use this field in a similar way.
Quoted 3 lines
Quoted 2 lines
from the faulting page
vmf->page->zone_device_datafield unless you have another use that field...
Quoted 6 lines
Quoted 1 line
You are right. Such work is planned in the memory attributes part that Himal
is working on. We have a
migration_granularityattribute which allow user to set the chunk size.Quoted 1 line
That makes sense. The chunk size is always just hint though, right?
I believe we should have a default chunk size, such as 2M, and user can overwrite it through hints.
Quoted 83 lines
Quoted 71 lines
Quoted 59 lines
I also comment on my patch with my suggestion implement chunk sizes too.
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Again helper for this.
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 13 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +13
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
This is an error, need to set a return value.
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
We probably should check migrate.cpages != npages too as I also think this is an error.
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
migrate_vma.src[i],
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 9 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +9
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
migrate_pfn_to_page(migrate_vma.src[i]);drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +6
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
I touch on this above, this should be reworked to roughly:
- alloc pages and populate migrate_vma.dst
- dma map sram pages
- migrate a chunk of contigous vram addresses at a time
- wait on last dma fence from migrate
- unmap dma pages
- unlock and free all pagesQuoted 11 lines
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +7
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
I've touched on why this isn't needed... I think one of these migrate_vma_* functions will trigger all MMU notifiers registered for the range. The notifier owns the invalidate then.
Quoted 1 line
Very good point. Yes after read
migrate_vma_setupfunction, I agree thisfunction will call mmu notifiers with
MMU_NOTIFY_MIGRATEevent. Today we invalidate vma with this event. So yes, above codes are not needed.Quoted 2 lines
I do identified one potential improvement: when mmu notifier is called back
with
MMU_NOTIFY_MIGRATEevent, if themigrate_vma_setupis called from the gpu page fault path, we can ignore the gpu vma invalidation as we will update gpu page table later after migration anyway. I think an page table invalidation is not needed in such case. But this should be just a minor improvement.Quoted 1 line
Quoted 6 lines
We skip invalidations if the
initial_bindflag is clear which should cover at the initial GPU fault. There is certainly room for improvement / optimizations in the MMU notifier though, it is kinda messy right now too. IMO work like this can be done once the basic design is working + tests in place to verify changes / optimizations.
agreed
Quoted 28 lines
Quoted 8 lines
Quoted 6 lines
Beyond this, maybe I'm confused about a few things and how this fits all together. Doesn't every user process have its own unique mm, fd, and vm (e.g. own address space)? If a user want a shared address space then use threads with a single mm, fd, and vm.
Yes, this is also my understanding. Each user process has its own mm struct
and device fds.
Quoted 2 lines
In a shared address space case, such as there are multiple pthread created
through
pthread_createin one process, all those pthreads should have different kerneltask_struct, but all thosetask_struct(say we get it from "current" macro) should share one same mm struct, which means they all lives inside one cpu address space.Quoted 2 lines
Now with this work, we are now basically extending this shared cpu address
space to gpu program. So both cpu program and gpu program can share one address space.
Quoted 2 lines
Since we allow user to create multiple gpu vm for one device (lets focus on
6 lines not shown · raw message
Quoted 3 lines
3 lines not shown · raw message
55 lines not shown · raw message
Parts of this message are not shown here. View the raw message.
RE: [v2 27/31] drm/xe/svm: Handle CPU page fault
In reply to Matthew BrostHi Matt,
Quoted 414 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Friday, April 12, 2024 2:11 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 27/31] drm/xe/svm: Handle CPU page faultQuoted 9 lines · Oak
Quoted 9 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 10:07 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected];Welty,
Quoted 28 lines
Quoted 28 lines
Brian <[email protected]>
Subject: Re: [v2 27/31] drm/xe/svm: Handle CPU page faultQuoted 8 lines · Oak Zeng
Under the picture of svm, CPU and GPU program share one same virtual address space. The backing store of this virtual address space can be either in system memory or device memory. Since GPU device memory is remaped as
DEVICE_PRIVATE, CPU can't access it. Any CPU access to device memory causes a page fault. Implement a page fault handler to migrate memory back to system memory and map it to CPU page table so the CPU program can proceed.Quoted 5 lines
Also unbind this page from GPU side, and free the original GPU device page
Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana VishwanathapuraQuoted 1 line
Signed-off-by Niranjana VishwanathapuraQuoted 8 lines
Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_svm.h | 8 +- drivers/gpu/drm/xe/xe_svm_devmem.c | 7 +- drivers/gpu/drm/xe/xe_svm_migrate.c | 222Quoted 5 lines
Quoted 5 lines
++++++++++++++++++++++++++++
Review
Quoted 4 lines
4 files changed, 230 insertions(+), 8 deletions(-) create mode 100644 drivers/gpu/drm/xe/xe_svm_migrate.c(quoted diff) +0
-0b/drivers/gpu/drm/xe/Makefile
Quoted 12 lines
Quoted 12 lines
drivers/gpu/drm/xe/Makefile xe-y += xe_bb.o \ · line 134
Quoted 8 lines
index
f89d77b6d654..65289acdd563100644drivers/gpu/drm/xe/Makefile +1
-0@@ -131,6 +131,7 @@ xe-y += xe_bb.o \ xe_step.o \ xe_svm.o \ xe_svm_devmem.o \ + xe_svm_migrate.o \See comments about file org, same thing applies here. Let's put all of the svm implementation in a single file.
Quoted 4 lines
Quoted 4 lines
Review
Quoted 4 lines
drivers/gpu/drm/xe/Makefile +0
-0xe_sync.o \ xe_tile.o \ xe_tile_sysfs.o \(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm.hQuoted 23 lines
Quoted 23 lines
Quoted 18 lines
index
f601dffe3fc1..c9e4239c44b4100644drivers/gpu/drm/xe/xe_svm.h +2
-2@@ -7,11 +7,11 @@#define __XE_SVM_H#include <linux/mm_types.h> +#include <linux/mm.h> #include "xe_device_types.h" #include "xe_device.h" #include "xe_assert.h" - -struct xe_vm ; +#include "xe_vm_types.h" /** * struct xe_svm - data structure to represent a shared drivers/gpu/drm/xe/xe_svm.h struct xe_svm { · line 34
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm.h +1
-0@@ -31,6 +31,9 @@ struct xe_svm {struct list_head vm_list; }; +#define xe_svm_for_each_vm (svm, vm)\Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm.h struct xe_svm { · line 36
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm.h +2
-0+list_for_each_entry (vm, &svm->vm_list, svm_link) +Don't think this is need, see below.
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm.h +0
-0extern struct xe_svm *xe_create_svm (void );void xe_destroy_svm (struct xe_svm *svm); extern struct xe_svm *xe_lookup_svm_by_mm(struct mm_struct*mm);
Quoted 15 lines
Quoted 15 lines
Review
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm.h +1
-0@@ -79,4 +82,5 @@ int xe_devm_alloc_pages(struct xe_tile *tile,void xe_devm_free_blocks (struct list_head *blocks);void xe_devm_page_free (struct page *page); +vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf);#endif (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_devmem.cQuoted 7 lines
index
088ac209ad80..32ada458f1dd100644drivers/gpu/drm/xe/xe_svm_devmem.c +0
-0@@ -37,11 +37,6 @@ struct xe_svm_block_meta {unsigned long bitmap[]; };Quoted 9 lines
Quoted 9 lines
drivers/gpu/drm/xe/xe_svm_devmem.c void xe_devm_free_blocks(struct list_head
Quoted 9 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +0
-5-static vm_fault_t xe_devm_migrate_to_ram (struct vm_fault *vmf) -{ -return 0 ; -} -static u64 block_offset_to_pfn (struct xe_mem_region *mr,u64 offset) {/** DRM buddy's block offset is 0-based*/ @@ -168,7 +163,7 @@ void xe_devm_free_blocks(struct list_head*blocks)
Quoted 58 lines
Quoted 58 lines
drivers/gpu/drm/xe/xe_svm_devmem.c void xe_devm_free_blocks(struct list_head *blocks) · line 166
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +1
-1static const struct dev_pagemap_ops xe_devm_pagemap_ops = { .page_free = xe_devm_page_free, - .migrate_to_ram = xe_devm_migrate_to_ram, + .migrate_to_ram = xe_svm_migrate_to_sram,Again single file so this will be static function, no reason to export this.
Review
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +0
-0};/** (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_migrate.cQuoted 44 lines
new file mode 100644
index 000000000000..0db831af098edrivers/gpu/drm/xe/xe_svm_migrate.c new file +39
-0@@ -0,0 +1,222 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2023 Intel Corporation +*/ + +#include <linux/gfp.h> +#include <linux/migrate.h> +#include <linux/dma-mapping.h> +#include <linux/dma-fence.h> +#include <linux/bitops.h> +#include <linux/bitmap.h> +#include <linux/kernel.h> +#include <linux/slab.h> +#include <drm/drm_buddy.h> +#include "xe_device_types.h" +#include "xe_device.h" +#include "xe_trace.h" +#include "xe_migrate.h" +#include "xe_ttm_vram_mgr_types.h" +#include "xe_assert.h" +#include "xe_pt.h" +#include "xe_svm.h" +#include "xe_vm.h" + + +/** +* alloc_host_page() - allocate one host page for the fault vma +* +* @dev: (GPU) device that will access the allocated page +* @vma: the fault vma that we need allocate page for +* @addr: the fault address. The allocated page is for this address +* @dma_addr: used to output the dma address of the allocated page. +* This dma address will be used for gpu to access this page. GPU +* access host page through a dma mapped address. +* @pfn: used to output the pfn of the allocated page. +* +* This function allocate one host page for the specified vma. It +* also does some prepare work for GPU to access this page, such +* as map this page to iommu (by calling dma_map_page). Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +8
-0+* +* When this function returns, the page is locked. +* +* Return struct page pointer when success +* NULL otherwise +*/ +static struct page *alloc_host_page(struct device *dev, + struct
vm_area_structQuoted 50 lines
Quoted 50 lines
*vma,
drivers/gpu/drm/xe/xe_svm_migrate.c line 49
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+unsigned long addr, + dma_addr_t
*dma_addr,drivers/gpu/drm/xe/xe_svm_migrate.c line 50
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+ unsigned long *pfn)Weird alignment, also I don't think we are want to allocate a page at time...
Beyond that, can't say I'm a fan of 2 arguments being return and populated here either (
dma_addr_t*dma_addr, unsigned long *pfn). I haven't seen a lot that style function in Linux.Probably makes more sense to have a function which allocates pages, locks them, and populates the pfn array (
migrate_pfn) rather than doing this a page at a time.drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+{ +struct page *page; + + page =alloc_page_vma (GFP_HIGHUSER , vma, addr); +if (unlikely (!page)) +return NULL ; + +/**Lock page per hmm requirement, see hmm.rst*/ +lock_page (page); + *dma_addr = dma_map_page(dev, page, 0, PAGE_SIZE,
DMA_FROM_DEVICE);The device is writing to these pages so I think
DMA_BIDIRECTIONALis needed, right? As mentioned above I think this should be broken out into a different step too.Quoted 17 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +17
-0+if (unlikely (dma_mapping_error (dev, *dma_addr))) { +unlock_page (page); +__free_page (page); +return NULL ; + } + + *pfn =migrate_pfn (page_to_pfn (page)); +return page; +} + +static void free_host_page (struct page *page) +{ +unlock_page (page); +put_page (page); +} + +/** Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +8
-0+* migrate_page_vram_to_ram() - migrate one page from vram to ram +* +* @vma: The vma that the page is mapped to +* @addr: The virtual address that the page is mapped to +* @src_pfn: src page's page frame number +* @dst_pfn: used to return dstination page (in system ram)'s pfn +* +* Allocate one page in system ram and copy memory from device memory
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +7
-0+* to system ram. +* +* Return: 0 if this page is already in sram (no need to migrate) +* 1: successfully migrated this page from vram to sram. +* error code otherwise +*/ +static int migrate_page_vram_to_ram(struct vm_area_struct *vma,Quoted 8 lines
Quoted 8 lines
unsigned long addr,
drivers/gpu/drm/xe/xe_svm_migrate.c line 93
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+ unsigned long src_pfn,unsigned long
*dst_pfn)drivers/gpu/drm/xe/xe_svm_migrate.c line 94
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+{We definitely don't want to copy 1 page at time. I touch on this in [1]. Basically this going to perform poorly unless we use larger copies, the migrate code supports non-contigous sram addresses, and vram
addresses
Quoted 3 lines
Quoted 3 lines
will likely be contigous due to the buddy allocator.
[1]
https://patchwork.freedesktop.org/patch/588548/?series=132229&rev=1
Quoted 16 lines
Quoted 16 lines
drivers/gpu/drm/xe/xe_svm_migrate.c line 106
Quoted 13 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +12
-0+struct xe_mem_region *mr; +struct xe_tile *tile; +struct xe_device *xe; +struct device *dev; +dma_addr_t dma_addr =0 ; +struct dma_fence *fence; +struct page *host_page; +struct page *src_page; +u64 src_dpa; + + src_page =migrate_pfn_to_page (src_pfn); +if (unlikely (!src_page || !(src_pfn &MIGRATE_PFN_MIGRATE )))I'm going to say this is a bug if !src_page ||
!is_zone_device_page(src_page) || !(src_pfn &
MIGRATE_PFN_MIGRATE) andQuoted 2 lines
Quoted 2 lines
we return
-EFAULT(or another error code if that makes more sense). We are migrating from the device where we know we have backing storefrom
Quoted 28 lines
Quoted 28 lines
the original fault.
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +6
-0+return 0 ; + + mr =xe_page_to_mem_region (src_page); + tile =xe_mem_region_to_tile (mr); + xe =tile_to_xe (tile); + dev = xe->drm.dev;drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+ + src_dpa =xe_mem_region_pfn_to_dpa (mr, src_pfn); + host_page =alloc_host_page (dev, vma, addr, &dma_addr, dst_pfn); +if (!host_page) +return -ENOMEM ; + + fence =xe_migrate_pa (tile->migrate, src_dpa, true, + dma_addr, false,PAGE_SIZE ); +if (IS_ERR (fence)) { + dma_unmap_page(dev, dma_addr, PAGE_SIZE,
DMA_FROM_DEVICE);drivers/gpu/drm/xe/xe_svm_migrate.c line 127
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +5
-0+free_host_page (host_page); +return PTR_ERR (fence); + } + +dma_fence_wait (fence, false);Even if we did want to migrate a page at a time, we only need to wait on the last fence due to the ordered nature of exec queues.
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c line 129
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+dma_fence_put (fence); + dma_unmap_page(dev, dma_addr, PAGE_SIZE,
DMA_FROM_DEVICE);Quoted 9 lines
Quoted 9 lines
With above, will likely unmap all dma pages in a single function once the last fence is signaled.
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +5
-0+return 1 ; +} + +/** +* xe_svm_migrate_to_sram() - Migrate memory back to sram on CPU page
Quoted 3 lines
Quoted 3 lines
fault
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+* +* @vmf: cpu vm fault structure, contains fault information such as vma etc.
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c line 138
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+* +* Note, this is in CPU's vm fault handler, caller holds the mmap read lock.
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c line 140
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+* +* This function migrate one gpu vma which contains the fault address to
Quoted 4 lines
Quoted 4 lines
sram.
drivers/gpu/drm/xe/xe_svm_migrate.c line 141
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* We try to maintain a 1:1 mapping b/t the CPU vma and gpu vma (i.e., create one
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* gpu vma for one cpu vma initially and try not to split it). So this scheme
Quoted 2 lines
Quoted 2 lines
end
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* up migrate at the vma granularity. This might not be the best performant
Quoted 3 lines
Quoted 3 lines
scheme
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+* +* This can be tunned with a migration granularity for performance, for
Quoted 2 lines
Quoted 2 lines
example,
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* migration 2M for each CPU page fault, or let user specify how much to
Quoted 36 lines
Quoted 36 lines
migrate.
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +3
-0+* This is more complex due to vma splitting. +* +* This function should also update GPU page table, so the fault virtual address
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +5
-0+* points to the same sram location from GPU side. This is TBD. +* +* Return: +* 0 on success +* VM_FAULT_SIGBUS: failed to migrate page to system memory, application
drivers/gpu/drm/xe/xe_svm_migrate.c line 158
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+* will be signaled a SIGBUG +*/ +vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf) +{Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
I don't think this is needed... More below.
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +12
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Quoted 8 lines
Quoted 8 lines
So I know in my PoC I had the fault user pointer (
xe_vma) == structvm_area_structof the GPU fault. That is definitely wrong. We likely want to allocate sub-range ofvm_area_structfor thexe_vma, we can call this a chunk size. Logical chunks sizes would be aligned 2MB, 64k, and finally 4k in sizes trying the largest first... The chunk sizes are trivial as we likely can just have table with values, the key here is thevm_area_structvm_start/vm_endare not what we want to usehere
Quoted 1 line
Quoted 1 line
rather
xe_vma_start(vma)andxe_vma_end(vma). I think we get the
xe_vmaAfter I typed this, realized I made a mistake here...
s/xe_vma_start/xe_vma_userptr/
s/xe_vma_end/xe_vma_userptr + xe_vma_size/
Quoted 6 lines
But you get the idea - the
zone_device_datapoints Xe specific chunk data (currentlyxe_vma, could bexe_pt_stateour something later if we switch to 1:N).Check AMD's + Nvidia's drivers and they both use this field in a similar way.
I got your idea. But as explained in a reply earlier, I am not convinced it is a good idea of using zdd to point to xe-vma.
Quoted 11 lines
Quoted 2 lines
Quoted 2 lines
from the faulting page
vmf->page->zone_device_datafield unless youhave
Quoted 3 lines
Quoted 1 line
another use that field...
You are right. Such work is planned in the memory attributes part that
Himal is working on. We have a
migration_granularityattribute which allow user to set the chunk size.Quoted 1 line
That makes sense. The chunk size is always just hint though, right?
We will have a default chunk size of 2M, which can be overwritten by hint
Quoted 84 lines
Quoted 3 lines
Quoted 3 lines
I also comment on my patch with my suggestion implement chunk sizes
too.
Quoted 68 lines
Quoted 56 lines
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Again helper for this.
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 13 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +13
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
This is an error, need to set a return value.
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
We probably should check migrate.cpages != npages too as I also think this is an error.
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
migrate_vma.src[i],
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 9 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +9
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
migrate_pfn_to_page(migrate_vma.src[i]);drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +6
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
I touch on this above, this should be reworked to roughly:
- alloc pages and populate migrate_vma.dst
- dma map sram pages
- migrate a chunk of contigous vram addresses at a time
- wait on last dma fence from migrate
- unmap dma pages
- unlock and free all pagesQuoted 11 lines
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +7
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
I've touched on why this isn't needed... I think one of these migrate_vma_* functions will trigger all MMU notifiers registered for the range. The notifier owns the invalidate then.
Quoted 1 line
Very good point. Yes after read
migrate_vma_setupfunction, I agree thisfunction will call mmu notifiers with
MMU_NOTIFY_MIGRATEevent. Today we invalidate vma with this event. So yes, above codes are not needed.Quoted 2 lines
I do identified one potential improvement: when mmu notifier is called
back with
MMU_NOTIFY_MIGRATEevent, if themigrate_vma_setupis called from the gpu page fault path, we can ignore the gpu vma invalidation as we will update gpu page table later after migration anyway. I think an page table invalidation is not needed in such case. But this should be just a minor improvement.Quoted 1 line
Quoted 5 lines
We skip invalidations if the
initial_bindflag is clear which should cover at the initial GPU fault. There is certainly room for improvement / optimizations in the MMU notifier though, it is kinda messy right now too. IMO work like this can be done once the basic design is working + tests in place to verify changes / optimizations.
Agreed.
Quoted 30 lines
Quoted 5 lines
Quoted 5 lines
Beyond this, maybe I'm confused about a few things and how this fits all together. Doesn't every user process have its own unique mm, fd, and
24 lines not shown · raw message
Quoted 3 lines
3 lines not shown · raw message
57 lines not shown · raw message
Parts of this message are not shown here. View the raw message.
RE: [v2 27/31] drm/xe/svm: Handle CPU page fault
In reply to Matthew BrostQuoted 41 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 10:07 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 27/31] drm/xe/svm: Handle CPU page faultQuoted 8 lines · Oak Zeng
Under the picture of svm, CPU and GPU program share one same virtual address space. The backing store of this virtual address space can be either in system memory or device memory. Since GPU device memory is remaped as
DEVICE_PRIVATE, CPU can't access it. Any CPU access to device memory causes a page fault. Implement a page fault handler to migrate memory back to system memory and map it to CPU page table so the CPU program can proceed.Quoted 5 lines
Also unbind this page from GPU side, and free the original GPU device page
Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana VishwanathapuraQuoted 1 line
Signed-off-by Niranjana VishwanathapuraQuoted 8 lines
Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_svm.h | 8 +- drivers/gpu/drm/xe/xe_svm_devmem.c | 7 +- drivers/gpu/drm/xe/xe_svm_migrate.c | 222++++++++++++++++++++++++++++
Quoted 3 lines
4 files changed, 230 insertions(+), 8 deletions(-) create mode 100644 drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 12 lines
drivers/gpu/drm/xe/Makefile xe-y += xe_bb.o \ · line 134
Quoted 9 lines
drivers/gpu/drm/xe/Makefile +1
-0@@ -131,6 +131,7 @@ xe-y += xe_bb.o \ xe_step.o \ xe_svm.o \ xe_svm_devmem.o \ + xe_svm_migrate.o \See comments about file org, same thing applies here. Let's put all of the svm implementation in a single file.
Did this in v3.
Quoted 62 lines
Review
Quoted 5 lines
drivers/gpu/drm/xe/Makefile +0
-0xe_sync.o \ xe_tile.o \ xe_tile_sysfs.o \(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm.hQuoted 18 lines
index
f601dffe3fc1..c9e4239c44b4100644drivers/gpu/drm/xe/xe_svm.h +2
-2@@ -7,11 +7,11 @@#define __XE_SVM_H#include <linux/mm_types.h> +#include <linux/mm.h> #include "xe_device_types.h" #include "xe_device.h" #include "xe_assert.h" - -struct xe_vm ; +#include "xe_vm_types.h" /** * struct xe_svm - data structure to represent a shared drivers/gpu/drm/xe/xe_svm.h struct xe_svm { · line 34
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm.h +1
-0@@ -31,6 +31,9 @@ struct xe_svm {struct list_head vm_list; }; +#define xe_svm_for_each_vm (svm, vm)\drivers/gpu/drm/xe/xe_svm.h struct xe_svm { · line 36
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm.h +2
-0+list_for_each_entry (vm, &svm->vm_list, svm_link) +Don't think this is need, see below.
Review
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm.h +1
-0extern struct xe_svm *xe_create_svm (void );void xe_destroy_svm (struct xe_svm *svm);extern struct xe_svm *xe_lookup_svm_by_mm (struct mm_struct *mm);@@ -79,4 +82,5 @@ int xe_devm_alloc_pages(struct xe_tile *tile,void xe_devm_free_blocks (struct list_head *blocks);void xe_devm_page_free (struct page *page); +vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf);#endif (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_devmem.cQuoted 17 lines
index
088ac209ad80..32ada458f1dd100644drivers/gpu/drm/xe/xe_svm_devmem.c +0
-5@@ -37,11 +37,6 @@ struct xe_svm_block_meta {unsigned long bitmap[]; }; -static vm_fault_t xe_devm_migrate_to_ram (struct vm_fault *vmf) -{ -return 0 ; -} -static u64 block_offset_to_pfn (struct xe_mem_region *mr,u64 offset) {/** DRM buddy's block offset is 0-based*/ @@ -168,7 +163,7 @@ void xe_devm_free_blocks(struct list_head *blocks)
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_devmem.c void xe_devm_free_blocks(struct list_head *blocks) · line 166
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +1
-1static const struct dev_pagemap_ops xe_devm_pagemap_ops = { .page_free = xe_devm_page_free, - .migrate_to_ram = xe_devm_migrate_to_ram, + .migrate_to_ram = xe_svm_migrate_to_sram,Again single file so this will be static function, no reason to export this.
Agreed.
Quoted 67 lines
Review
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_devmem.c +0
-0};/** (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_migrate.cQuoted 42 lines
new file mode 100644
index 000000000000..0db831af098edrivers/gpu/drm/xe/xe_svm_migrate.c new file +37
-0@@ -0,0 +1,222 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2023 Intel Corporation +*/ + +#include <linux/gfp.h> +#include <linux/migrate.h> +#include <linux/dma-mapping.h> +#include <linux/dma-fence.h> +#include <linux/bitops.h> +#include <linux/bitmap.h> +#include <linux/kernel.h> +#include <linux/slab.h> +#include <drm/drm_buddy.h> +#include "xe_device_types.h" +#include "xe_device.h" +#include "xe_trace.h" +#include "xe_migrate.h" +#include "xe_ttm_vram_mgr_types.h" +#include "xe_assert.h" +#include "xe_pt.h" +#include "xe_svm.h" +#include "xe_vm.h" + + +/** +* alloc_host_page() - allocate one host page for the fault vma +* +* @dev: (GPU) device that will access the allocated page +* @vma: the fault vma that we need allocate page for +* @addr: the fault address. The allocated page is for this address +* @dma_addr: used to output the dma address of the allocated page. +* This dma address will be used for gpu to access this page. GPU +* access host page through a dma mapped address. +* @pfn: used to output the pfn of the allocated page. +* +* This function allocate one host page for the specified vma. It drivers/gpu/drm/xe/xe_svm_migrate.c line 47
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+* also does some prepare work for GPU to access this page, such +* as map this page to iommu (by calling dma_map_page). +* +* When this function returns, the page is locked. +* +* Return struct page pointer when success +* NULL otherwise +*/ +static struct page *alloc_host_page(struct device *dev, + struct
vm_area_struct*vma,drivers/gpu/drm/xe/xe_svm_migrate.c line 49
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+unsigned long addr, + dma_addr_t
*dma_addr,drivers/gpu/drm/xe/xe_svm_migrate.c line 50
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+ unsigned long *pfn)Weird alignment, also I don't think we are want to allocate a page at time...
Quoted 7 lines
Beyond that, can't say I'm a fan of 2 arguments being return and populated here either (
dma_addr_t*dma_addr, unsigned long *pfn). I haven't seen a lot that style function in Linux.Probably makes more sense to have a function which allocates pages, locks them, and populates the pfn array (
migrate_pfn) rather than doing this a page at a time.
Agreed. In v3, I adopted Nvidia's new dma-mapping API which also requires a 2 steps way: 1) allocate pages 2) dma-mapping
Quoted 13 lines
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+{ +struct page *page; + + page =alloc_page_vma (GFP_HIGHUSER , vma, addr); +if (unlikely (!page)) +return NULL ; + +/**Lock page per hmm requirement, see hmm.rst*/ +lock_page (page); + *dma_addr = dma_map_page(dev, page, 0, PAGE_SIZE,
DMA_FROM_DEVICE);
Quoted 2 lines
The device is writing to these pages so I think
DMA_BIDIRECTIONALis needed, right?
We are copying from device memory to system memory here. Yes device writes to those pages, but the direction parameter is about the copy direction, not about read or write. I think FROM_DEVICE is still correct.
As mentioned above I think this should be broken out into
Quoted 1 line
a different step too.
Agreed.
Quoted 35 lines
Quoted 16 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +15
-0+if (unlikely (dma_mapping_error (dev, *dma_addr))) { +unlock_page (page); +__free_page (page); +return NULL ; + } + + *pfn =migrate_pfn (page_to_pfn (page)); +return page; +} + +static void free_host_page (struct page *page) +{ +unlock_page (page); +put_page (page); +}drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+ +/** +* migrate_page_vram_to_ram() - migrate one page from vram to ram +* +* @vma: The vma that the page is mapped to +* @addr: The virtual address that the page is mapped to +* @src_pfn: src page's page frame number +* @dst_pfn: used to return dstination page (in system ram)'s pfn +* +* Allocate one page in system ram and copy memory from device memory
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +7
-0+* to system ram. +* +* Return: 0 if this page is already in sram (no need to migrate) +* 1: successfully migrated this page from vram to sram. +* error code otherwise +*/ +static int migrate_page_vram_to_ram(struct vm_area_struct *vma,unsigned long addr,
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_migrate.c line 93
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+ unsigned long src_pfn,unsigned long
*dst_pfn)drivers/gpu/drm/xe/xe_svm_migrate.c line 94
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+{We definitely don't want to copy 1 page at time. I touch on this in [1]. Basically this going to perform poorly unless we use larger copies, the migrate code supports non-contigous sram addresses, and vram addresses will likely be contigous due to the buddy allocator.
Totally agreed we need to use one blitter command to copy many pages.. I was aware of this when I wrote this v2. And this is addressed in v3.
But also be aware, we might end up migrate multiple pages from system memory to device memory on each gpu page fault. But on CPU page fault, we only want to migrate 1 page to cover the fault address. I know in this v2, I migrated whole CPU VMA. I realized this is not correct after read more core mm and hmm code. The main reason is, the core mm only program one pte entry (covers 4k), even if we migrate multiple pages. See logic in function handle_pte_fault and do_swap_page. So I switched to one page scheme on v3. At least this is my current understanding.
Regarding vram contiguous, I think if we don't pass DRM_BUDDY_CONTIGUOUS_ALLOCATION, we still have chance run into non-contiguous vram. Even if the original allocation is contiguous, a portion of the contiguous vram can be migrated to system memory (and freed from device side), then migrate back to vram. We end up with non-contiguous vram in such case.
I am thinking not using identity mapping for vram ppgtt mapping for this case, just like what we did for system memory ppgtt mapping. This way we don't care whether vram is contiguous or not, one shot blitter command can do the work. Otherwise, we will still have to split the copy task per contiguous/non-contiguous vram. This is marked as FIXME in v3. What do you think?
Quoted 14 lines
[1] https://patchwork.freedesktop.org/patch/588548/?series=132229&rev=1
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +11
-0+struct xe_mem_region *mr; +struct xe_tile *tile; +struct xe_device *xe; +struct device *dev; +dma_addr_t dma_addr =0 ; +struct dma_fence *fence; +struct page *host_page; +struct page *src_page; +u64 src_dpa; + + src_page =migrate_pfn_to_page (src_pfn);
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_migrate.c line 106
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+if (unlikely (!src_page || !(src_pfn &MIGRATE_PFN_MIGRATE )))I'm going to say this is a bug if !src_page || !is_zone_device_page(src_page) || !(src_pfn &
MIGRATE_PFN_MIGRATE) and
we return-EFAULT(or another error code if that makes more sense). We are migrating from the device where we know we have backing store from the original fault.
Agreed under the context of this v2.
In v3, I am moving to a more page centric scheme. Think of this scenario: a virtual address range, let's say 2M, is migrated gpu device memory. Now CPU accessed one page in the middle of this 2M range, this single 4k page will be migrate back to system memory. HMM also triggers a invalidation callback to driver to invalidate this 4k range. Now gpu access this 4k range. Let's say our GPU migration granularity is 2M (the concept of chunk in your series) and migration policy says we need to migrate back to device memory, the migrate_vma_setup will be set up to cover a 2M range but hmm will tell us there is only one 4k page in system memory need to be migrated while all the rest (2M - 4K) need no migration. In this scenario MIGRATE_PFN_MIGRATE is not set for the 2M-4k. of course this scenario doesn't apply to above vram_to_sram migration.
Quoted 18 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +6
-0+return 0 ; + + mr =xe_page_to_mem_region (src_page); + tile =xe_mem_region_to_tile (mr); + xe =tile_to_xe (tile); + dev = xe->drm.dev;drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+ + src_dpa =xe_mem_region_pfn_to_dpa (mr, src_pfn); + host_page =alloc_host_page (dev, vma, addr, &dma_addr, dst_pfn); +if (!host_page) +return -ENOMEM ; + + fence =xe_migrate_pa (tile->migrate, src_dpa, true, + dma_addr, false,PAGE_SIZE ); +if (IS_ERR (fence)) { + dma_unmap_page(dev, dma_addr, PAGE_SIZE,
DMA_FROM_DEVICE);
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_migrate.c line 127
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +5
-0+free_host_page (host_page); +return PTR_ERR (fence); + } + +dma_fence_wait (fence, false);Even if we did want to migrate a page at a time, we only need to wait on the last fence due to the ordered nature of exec queues.
Sure. As said we migrate all pages in one shot in v3. Above comment doesn't apply anymore.
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c line 129
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+dma_fence_put (fence); + dma_unmap_page(dev, dma_addr, PAGE_SIZE,
DMA_FROM_DEVICE);With above, will likely unmap all dma pages in a single function once the last fence is signaled.
Yes, this was handled properly in v3.
Quoted 36 lines
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +5
-0+return 1 ; +} + +/** +* xe_svm_migrate_to_sram() - Migrate memory back to sram on CPU page fault
drivers/gpu/drm/xe/xe_svm_migrate.c line 136
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+* +* @vmf: cpu vm fault structure, contains fault information such as vma etc.
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +4
-0+* +* Note, this is in CPU's vm fault handler, caller holds the mmap read lock. +* +* This function migrate one gpu vma which contains the fault address to sram.
drivers/gpu/drm/xe/xe_svm_migrate.c line 141
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* We try to maintain a 1:1 mapping b/t the CPU vma and gpu vma (i.e., create one
drivers/gpu/drm/xe/xe_svm_migrate.c line 142
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* gpu vma for one cpu vma initially and try not to split it). So this scheme end
drivers/gpu/drm/xe/xe_svm_migrate.c line 143
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* up migrate at the vma granularity. This might not be the best performant scheme
drivers/gpu/drm/xe/xe_svm_migrate.c line 145
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0+* +* This can be tunned with a migration granularity for performance, for example,
drivers/gpu/drm/xe/xe_svm_migrate.c line 146
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+* migration 2M for each CPU page fault, or let user specify how much to migrate.
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +3
-0+* This is more complex due to vma splitting. +* +* This function should also update GPU page table, so the fault virtual address
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +5
-0+* points to the same sram location from GPU side. This is TBD. +* +* Return: +* 0 on success +* VM_FAULT_SIGBUS: failed to migrate page to system memory, application
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c line 164
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +10
-0+* will be signaled a SIGBUG +*/ +vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf) +{ +struct xe_mem_region *mr =xe_page_to_mem_region (vmf->page); +struct xe_tile *tile =xe_mem_region_to_tile (mr); +struct xe_device *xe =tile_to_xe (tile); +struct vm_area_struct *vma = vmf->vma; +struct mm_struct *mm = vma->vm_mm; +struct xe_svm *svm =xe_lookup_svm_by_mm (mm);I don't think this is needed... More below.
Yes. I removed the xe_svm concept in v3.
In my v3, we have a mm_struct pointer in gpuvm. If a gpuvm participate svm, the mm struct pointer will be set. This is from your PoC and I think this is the simplest implementation.
Depending on how people view it, we might still invite back the xe_svm or drm_gpusvm (as in your new series) though. It not a huge deal. Let's see how things go.
Quoted 16 lines
drivers/gpu/drm/xe/xe_svm_migrate.c line 176
Quoted 13 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +12
-0+unsigned long addr = vma->vm_start; +u64 npages =vma_pages (vma); +struct xe_vma *xe_vma; +vm_fault_t ret =0 ; +struct xe_vm *vm; +void *buf; +int i; + +struct migrate_vma migrate_vma = { + .vma = vmf->vma, + .start = vma->vm_start, + .end = vma->vm_end,So I know in my PoC I had the fault user pointer (
xe_vma) == structvm_area_structof the GPU fault. That is definitely wrong.
Can you explain why this is definitely wrong? I still think creating xe_vma/fault userptr to cover the whole struct vm_area_struct range is not a bad idea. Our mmu notififer is xe-vma based, so basically for each vm_area_struct/xe-vma we have a mmu interval notifier.
The migration and gpu page table update can be any sub-range of the xe-vma, as long as the boundary of the sub-range is page aligned.
We likely
Quoted 6 lines
want to allocate sub-range of
vm_area_structfor thexe_vma, we can call this a chunk size. Logical chunks sizes would be aligned 2MB, 64k, and finally 4k in sizes trying the largest first... The chunk sizes are trivial as we likely can just have table with values, the key here is thevm_area_structvm_start/vm_endare not what we want to use here ratherxe_vma_start(vma)andxe_vma_end(vma).
This is cpu page fault handler. I think it is just fine to use vma_area_struct.vm_start/end. In my opinion, referring to some gpu structure xe-vma-start/end in cpu page fault handler is a little strange.
As said, I have moved to one-page based scheme in cpu page fault handling. But even if we migrate whole vma, I don't see a problem of above codes. Can you explain why you want to use xe-vma here? My guess is, in your picture, you want one xe-vma be either backed by system memory, or gpu device memory but not a mixture. I am moving away from this scheme and adopting a page centric design where xe-vma can be backed by a mixture of sram and vram.
I think we get the
Quoted 3 lines
xe_vma
from the faulting pagevmf->page->zone_device_datafield unless you have another use that field...
I know this scheme works for you. But it is a design choice of how to use zone-device-data. In my v3, this is still used to hold buddy-block. I am not saying we can't do what you did, just we can do it differently.
Struct page is all about physical memory management. Couple it with virtual memory structure such as xe-vma is not a good sign to me.
Quoted 2 lines
I also comment on my patch with my suggestion implement chunk sizes too.
I did look at your patch and basically you are creating svm-rang at chunk size...
In my v3, I am staying with the same concept of v2: create xe-vma per cpu vma, but changed the migration/gpu pt mapping to be chunk size based, roughly.
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_migrate.c line 177
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+ .pgmap_owner = xe,Again helper for this.
Addressed in v3
Quoted 16 lines
Quoted 13 lines
drivers/gpu/drm/xe/xe_svm_migrate.c line 178
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c new file +1
-0+ .flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE,This is an error, need to set a return value.
Agreed
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
We probably should check migrate.cpages != npages too as I also think this is an error.
In the scheme of "no mixture placement of vma", yes. As said, I am moving away from this scheme. Let's see how well it moves.
Quoted 23 lines
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +2
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
migrate_vma.src[i],
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 9 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +9
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
migrate_pfn_to_page(migrate_vma.src[i]);drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +6
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
I touch on this above, this should be reworked to roughly:
Quoted 6 lines
- alloc pages and populate migrate_vma.dst
- dma map sram pages
- migrate a chunk of contigous vram addresses at a time
- wait on last dma fence from migrate
- unmap dma pages
- unlock and free all pages
In v3, I roughly have above process. But a lot of things have changed, let's review it once it is ready.
Quoted 27 lines
drivers/gpu/drm/xe/xe_svm_migrate.c
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_migrate.c new file +7
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
15 lines not shown · raw message
Quoted 2 lines
2 lines not shown · raw message
28 lines not shown · raw message
Parts of this message are not shown here. View the raw message.
PATCH v2 28/31 drm/xe/svm: Introduce helper to migrate vma to vram
In reply to Oak ZengIntroduce a helper function xe_svm_migrate_vma_to_vram.
Since the source pages of the svm range can be physically not contiguous, and the destination vram pages can also be not contiguous, there is no easy way to migrate multiple pages per blitter command. We do page by page migration for now.
Migration is best effort. Even if we fail to migrate some pages, we will try to migrate the rest pages.
FIXME: Use one blitter command to copy when both src and dst are physically contiguous
FIXME: when a vma is partially migrated, split vma as we assume no mixture vma placement.
--- drivers/gpu/drm/xe/xe_svm.h | 2 + drivers/gpu/drm/xe/xe_svm_migrate.c | 115 ++++++++++++++++++++++++++++ 2 files changed, 117 insertions(+)
drivers/gpu/drm/xe/xe_svm.h +2 -0
@@ -83,4 +83,6 @@ int xe_devm_alloc_pages(struct xe_tile *tile,void xe_devm_free_blocks (struct list_head *blocks);void xe_devm_page_free (struct page *page);vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf); +int xe_svm_migrate_vma_to_vram (struct xe_vm *vm,struct xe_vma *vma, +struct xe_tile *tile);#endif
drivers/gpu/drm/xe/xe_svm_migrate.c +115 -0
118 lines not on this page · raw message
Signature · 1 line
2.26.3
Matthew Brost
maintainerRe: [v2 28/31] drm/xe/svm: Introduce helper to migrate vma to vram
Quoted 7 lines · Oak Zeng
Introduce a helper function
xe_svm_migrate_vma_to_vram.Since the source pages of the svm range can be physically not contiguous, and the destination vram pages can also be not contiguous, there is no easy way to migrate multiple pages per blitter command. We do page by page migration for now.
Quoted 6 lines
Migration is best effort. Even if we fail to migrate some pages, we will try to migrate the rest pages.
FIXME: Use one blitter command to copy when both src and dst are physically contiguous
Yep, touch in this throughout the series. Only vram needs to be contiguous though as we dynamically create PT mappings for sram pages in the migrate code. Getting this in a must and should be done immediately IMO as this a very, very basic perform thing we know needs to be done. We will likely have to tune this code quite a bit for performance so getting known things done would be helpful.
Quoted 3 lines
FIXME: when a vma is partially migrated, split vma as we assume no mixture vma placement.
Agree we do not want support partial migrations. We likely want to return -EAGAIN for something and fault back to a smaller xe_vma chunk size which I discussed in [1] and add comment on in [2].
Migration should be best case too if we fail migrate we can always leave backing store in sram too.
I do have question though, when do we get partial migrations? A user having called mlock on some of the pages? I just want to make sure I fully understand that case.
[1] https://patchwork.freedesktop.org/patch/588526/?series=132229&rev=1
[2] https://patchwork.freedesktop.org/patch/588528/?series=132229&rev=1
Quoted 9 lines
Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana Vishwanathapura <[email protected]>Signed-off-by Niranjana Vishwanathapura <[email protected]>Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/xe_svm.h | 2 + drivers/gpu/drm/xe/xe_svm_migrate.c | 115 ++++++++++++++++++++++++++++
Same comment on file structure throughout the series apply here too.
Quoted 36 lines
2 files changed, 117 insertions(+)drivers/gpu/drm/xe/xe_svm.h +2
-0@@ -83,4 +83,6 @@ int xe_devm_alloc_pages(struct xe_tile *tile,void xe_devm_free_blocks (struct list_head *blocks);void xe_devm_page_free (struct page *page);vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf); +int xe_svm_migrate_vma_to_vram (struct xe_vm *vm,struct xe_vma *vma, +struct xe_tile *tile);#endif drivers/gpu/drm/xe/xe_svm_migrate.c +15
-0@@ -220,3 +220,118 @@ vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)kvfree (buf);return 0 ; } + +/** +* xe_svm_migrate_vma_to_vram() - migrate backing store of a vma to vram +* Must be called with mmap_read_lock held. +* @vm: the vm that the vma belongs to +* @vma: the vma to migrate. +* @tile: the destination tile which holds the new backing store of the range +* +* Returns: negative errno on faiure, 0 on success +*/ +int xe_svm_migrate_vma_to_vram (struct xe_vm *vm, +struct xe_vma *vma, +struct xe_tile *tile) +{ +struct mm_struct *mm = vm->mm;
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 247
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +10
-0+unsigned long start =xe_vma_start (vma); +unsigned long end =xe_vma_end (vma); +unsigned long npages = (end - start) >>PAGE_SHIFT ; +struct xe_mem_region *mr = &tile->mem.vram; +struct vm_area_struct *vas; + +struct migrate_vma migrate = { + .start = start, + .end = end, + .pgmap_owner = tile->xe,
Again helper to assign owner.
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 259
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +12
-0+ .flags = MIGRATE_VMA_SELECT_SYSTEM, + }; +struct device *dev = tile->xe->drm.dev; +dma_addr_t *src_dma_addr; +struct dma_fence *fence; +struct page *src_page; +LIST_HEAD (blocks); +int ret =0 , i; +u64 dst_dpa; +void *buf; + +mmap_assert_locked (mm);
This mmap_assert_locked is ambiguous, we should make it clear if this read or write locked. Doesn't it have to be write to do the migrate pages?
A larger question about the locking... The CPU fault handler holds the mmap lock in write mode, right?
I'm asking as basically I think at least initially we want to hold the mmap lock in a way that the GPU handler and CPU handler do not race. i.e. From fault userptr create in GPU fault handler to issuing the bind we prevent the CPU fault handler from running.
I'm having issues figuring out how to prevent races between initial binds of userptrs and userptr invalidates on faulting VMs. This race is seen any xe_exec_fault_mode for example... So preventing races between CPU / GPU fault handler with the mmap probably is a good idea initially. Likely can make the locking finer grained once this is all working and I figure out how to handle this race better.
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 261
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+ + vas =find_vma_intersection (mm, start, start +4 );
find_vma should work fine here.
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 273
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +12
-0+if (!vas) +return -ENOENT ; + + migrate.vma = vas; + buf =kvcalloc (npages,2 *sizeof (*migrate.src) +sizeof (*src_dma_addr), +GFP_KERNEL ); +if (!buf) +return -ENOMEM ; + migrate.src = buf; + migrate.dst = migrate.src + npages; + src_dma_addr = (dma_addr_t *) (migrate.dst + npages); + ret =xe_devm_alloc_pages (tile, npages, &blocks, migrate.dst);
Again as I discussed in [3] I think this should be broken out into a different step with the blocks allocated before this, and here just populate migrate.dst from the existing blocks.
[3] https://patchwork.freedesktop.org/patch/588523/?series=132229&rev=1
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +7
-0+if (ret) + goto kfree_buf; + + ret =migrate_vma_setup (&migrate); +if (ret) { +drm_err (&tile->xe->drm,"vma setup returned %d for range [%lx - %lx]\n" , + ret, start, end);
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 290
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +10
-0+ goto free_dst_pages; + } + +/**FIXME: partial migration of a range print a warning for now. +* If this message is printed, we need to split xe_vma as we +* don't support a mixture placement of one vma +*/ +if (migrate.cpages != npages) +drm_warn (&tile->xe->drm,"Partial migration for range [%lx - %lx], range is %ld pages, migrate only %ld pages\n" , + start, end, npages, migrate.cpages);
As discussed above, we shouldn't support this. We should fall back to smaller xe_vma chunk size until we find one that works or simply leave the pages in sram and map those pages to GPU.
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 295
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +5
-0+ +/**Migrate page by page for now. +* Both source pages and destination pages can physically not contiguous, +* there is no good way to migrate multiple pages per blitter command. +*/
Touched on this a bunch throughout the series, lets do better than a page a time migration.
Algorithm should be very similar to what I discussed here [4] but with a few key differences.
- I think the sram pages can be unpopulated (page == NULL) if the user has not yet touched the page
- Also I think the MIGRATE_PFN_MIGRATE bit being clear is valid
In these cases this indicate we have to issue a copy for the pages we are accumulating with contigous vram addresses.
[4] https://patchwork.freedesktop.org/patch/588526/?series=132229&rev=1
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 298
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0+for (i =0 ; i < npages; i++) { + src_page =migrate_pfn_to_page (migrate.src[i]); +if (unlikely (!src_page || !(migrate.src[i] &MIGRATE_PFN_MIGRATE )))
Discussed this in the CPU fault patch, once we call migrate_vma_setup, on subsequent errors we need to call migrate_vma_finalize to revert the pages to the original state. At least I think if I am reading the doc after this correctly.
Here on error we just free the pages...
Matt
Quoted 42 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +39
-0+ goto free_dst_page; + +xe_assert (tile->xe, !is_zone_device_page (src_page)); + src_dma_addr[i] =dma_map_page (dev, src_page,0 ,PAGE_SIZE ,DMA_TO_DEVICE ); +if (unlikely (dma_mapping_error (dev, src_dma_addr[i]))) { +drm_warn (&tile->xe->drm,"dma map error for host pfn %lx\n" , migrate.src[i]); + goto free_dst_page; + } + dst_dpa =xe_mem_region_pfn_to_dpa (mr, migrate.dst[i]); + fence =xe_migrate_pa (tile->migrate, src_dma_addr[i], false, + dst_dpa, true,PAGE_SIZE ); +if (IS_ERR (fence)) { +drm_warn (&tile->xe->drm,"migrate host page (pfn: %lx) to vram failed\n" , + migrate.src[i]); +/**Migration is best effort. Even we failed here, we continue*/ + goto free_dst_page; + } +/**FIXME: Use the first migration's out fence as the second migration's input fence, +* and so on. Only wait the out fence of last migration? +*/ +dma_fence_wait (fence, false); +dma_fence_put (fence); +free_dst_page: +xe_devm_page_free (pfn_to_page (migrate.dst[i])); + } + +for (i =0 ; i < npages; i++) +if (!(dma_mapping_error (dev, src_dma_addr[i]))) +dma_unmap_page (dev, src_dma_addr[i],PAGE_SIZE ,DMA_TO_DEVICE ); + +migrate_vma_pages (&migrate); +migrate_vma_finalize (&migrate); +free_dst_pages: +if (ret) +xe_devm_free_blocks (&blocks); +kfree_buf: +kfree (buf); +return ret; +}Signature · 1 line
2.26.3
RE: [v2 28/31] drm/xe/svm: Introduce helper to migrate vma to vram
Quoted 42 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 10:49 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 28/31] drm/xe/svm: Introduce helper to migrate vma to vramQuoted 7 lines · Oak Zeng
Introduce a helper function
xe_svm_migrate_vma_to_vram.Since the source pages of the svm range can be physically not contiguous, and the destination vram pages can also be not contiguous, there is no easy way to migrate multiple pages per blitter command. We do page by page migration for now.
Quoted 6 lines
Migration is best effort. Even if we fail to migrate some pages, we will try to migrate the rest pages.
FIXME: Use one blitter command to copy when both src and dst are physically contiguous
Yep, touch in this throughout the series. Only vram needs to be contiguous though as we dynamically create PT mappings for sram pages in the migrate code. Getting this in a must and should be done immediately IMO as this a very, very basic perform thing we know needs to be done. We will likely have to tune this code quite a bit for performance so getting known things done would be helpful.
Quoted 3 lines
FIXME: when a vma is partially migrated, split vma as we assume no mixture vma placement.
Agree we do not want support partial migrations. We likely want to return
-EAGAINfor something and fault back to a smallerxe_vmachunk size which I discussed in [1] and add comment on in [2].
Quoted 6 lines
Migration should be best case too if we fail migrate we can always leave backing store in sram too.
I do have question though, when do we get partial migrations? A user having called mlock on some of the pages? I just want to make sure I fully understand that case.
Yah, mlock could be one case...
I also looked the hmm code. There are few other cases where MIGRATE_PFN_MIGRATE is not set (so we skip migration after), such as pte is NULL and vma is file-backed (not anonymous); entry is swapped out to hard disk etc. see more details in function migrate_vma_collect_pmd.
Quoted 71 lines
[1] https://patchwork.freedesktop.org/patch/588526/?series=132229&rev=1
[2] https://patchwork.freedesktop.org/patch/588528/?series=132229&rev=1Quoted 2 lines
Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana VishwanathapuraQuoted 1 line
Signed-off-by Niranjana VishwanathapuraQuoted 6 lines
Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/xe_svm.h | 2 + drivers/gpu/drm/xe/xe_svm_migrate.c | 115++++++++++++++++++++++++++++
Same comment on file structure throughout the series apply here too.
Review
Quoted 14 lines
2 files changed, 117 insertions(+)drivers/gpu/drm/xe/xe_svm.h +2
-0@@ -83,4 +83,6 @@ int xe_devm_alloc_pages(struct xe_tile *tile,void xe_devm_free_blocks (struct list_head *blocks);void xe_devm_page_free (struct page *page);vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf); +int xe_svm_migrate_vma_to_vram (struct xe_vm *vm,struct xe_vma *vma, +struct xe_tile *tile);#endif (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_migrate.cdrivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct
Quoted 4 lines
index
0db831af098e..ab8dd1f58aa4100644drivers/gpu/drm/xe/xe_svm_migrate.c +0
-0@@ -220,3 +220,118 @@ vm_fault_t xe_svm_migrate_to_sram(struct
vm_fault*vmf)drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0kvfree (buf);return 0 ; } + +/** +* xe_svm_migrate_vma_to_vram() - migrate backing store of a vma to vram
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +12
-0+* Must be called with mmap_read_lock held. +* @vm: the vm that the vma belongs to +* @vma: the vma to migrate. +* @tile: the destination tile which holds the new backing store of the range +* +* Returns: negative errno on faiure, 0 on success +*/ +int xe_svm_migrate_vma_to_vram (struct xe_vm *vm, +struct xe_vma *vma, +struct xe_tile *tile) +{ +struct mm_struct *mm = vm->mm;drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 247
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +10
-0+unsigned long start =xe_vma_start (vma); +unsigned long end =xe_vma_end (vma); +unsigned long npages = (end - start) >>PAGE_SHIFT ; +struct xe_mem_region *mr = &tile->mem.vram; +struct vm_area_struct *vas; + +struct migrate_vma migrate = { + .start = start, + .end = end, + .pgmap_owner = tile->xe,Again helper to assign owner.
Quoted 16 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 259
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +12
-0+ .flags = MIGRATE_VMA_SELECT_SYSTEM, + }; +struct device *dev = tile->xe->drm.dev; +dma_addr_t *src_dma_addr; +struct dma_fence *fence; +struct page *src_page; +LIST_HEAD (blocks); +int ret =0 , i; +u64 dst_dpa; +void *buf; + +mmap_assert_locked (mm);This
mmap_assert_lockedis ambiguous, we should make it clear if this read or write locked. Doesn't it have to be write to do the migrate pages?
I followed hmm document (Documents/mm/hmm.rst), see section "Migration to and from device memory". It explicitly write a read_lock in this document.
I believe a read_lock is enough for the migrate_vma_setup/migrate_vma_finalize().
As I understand it, the mm.mmap_lock protect the process's address space. When we modify process's address space such as mmap/munmap, we need to hold a write mode lock; if we only read process's address space, such as in the migrate_vma_setup/finalize, or in the cpu page fault handler case, we only need a read mode lock.
I also cross checked amd driver. They also use a read lock.. see function svm_range_restore_pages in kfd_svm.c....
Quoted 3 lines
A larger question about the locking... The CPU fault handler holds the mmap lock in write mode, right?
No. since cpu fault handler doesn't modify process address space, instead it only fill up cpu page table for some valid address range, a read lock is enough.
Quoted 5 lines
I'm asking as basically I think at least initially we want to hold the mmap lock in a way that the GPU handler and CPU handler do not race. i.e. From fault userptr create in GPU fault handler to issuing the bind we prevent the CPU fault handler from running.
Yes we hold mmap_read_lock in both cpu and gpu fault handler to avoid that race.
In user mmap/munmap (such as kernel function vm_munmap), we hold a mmap_write_lock which prevent it racing with cpu and gpu fault handler.
Quoted 7 lines
I'm having issues figuring out how to prevent races between initial binds of userptrs and userptr invalidates on faulting VMs. This race is seen any
xe_exec_fault_modefor example... So preventing races between CPU / GPU fault handler with the mmap probably is a good idea initially. Likely can make the locking finer grained once this is all working and I figure out how to handle this race better.
I don't quite follow here.
Initial bind of user ptr... if you meant the bind in gpu page fault handler, then the racing with invalidation is roughly like below:
Invalidation is called with mmap_write_lock
In userptr_pin_page, we hold a mmap_read_lock, so we know during pin page, invalidation is excluded.
After pin, before programming gpu page table, we check whether there is invalidation happen *after last pin but before programming page table*, if that happened, we retry
Oak
Quoted 133 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 261
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+ + vas =find_vma_intersection (mm, start, start +4 );
find_vmashould work fine here.drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 273
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +12
-0+if (!vas) +return -ENOENT ; + + migrate.vma = vas; + buf =kvcalloc (npages,2 *sizeof (*migrate.src) +sizeof (*src_dma_addr), +GFP_KERNEL ); +if (!buf) +return -ENOMEM ; + migrate.src = buf; + migrate.dst = migrate.src + npages; + src_dma_addr = (dma_addr_t *) (migrate.dst + npages); + ret =xe_devm_alloc_pages (tile, npages, &blocks, migrate.dst);Again as I discussed in [3] I think this should be broken out into a different step with the blocks allocated before this, and here just populate migrate.dst from the existing blocks.
[3] https://patchwork.freedesktop.org/patch/588523/?series=132229&rev=1
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +6
-0+if (ret) + goto kfree_buf; + + ret =migrate_vma_setup (&migrate); +if (ret) { + drm_err(&tile->xe->drm, "vma setup returned %d for range[%lx - %lx]\n",
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +10
-0+ ret, start, end); + goto free_dst_pages; + } + +/**FIXME: partial migration of a range print a warning for now. +* If this message is printed, we need to split xe_vma as we +* don't support a mixture placement of one vma +*/ +if (migrate.cpages != npages) + drm_warn(&tile->xe->drm, "Partial migration for range [%lx -%lx], range is %ld pages, migrate only %ld pages\n",
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 290
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c +1
-0+ start, end, npages, migrate.cpages);As discussed above, we shouldn't support this. We should fall back to smaller
xe_vmachunk size until we find one that works or simply leave the pages in sram and map those pages to GPU.drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0+ +/**Migrate page by page for now. +* Both source pages and destination pages can physically not contiguous,
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 295
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+* there is no good way to migrate multiple pages per blitter command. +*/ Touched on this a bunch throughout the series, lets do better than a page a time migration.
Algorithm should be very similar to what I discussed here [4] but with a few key differences.
- I think the sram pages can be unpopulated (page == NULL) if the user has not yet touched the page
- Also I think theMIGRATE_PFN_MIGRATEbit being clear is validIn these cases this indicate we have to issue a copy for the pages we are accumulating with contigous vram addresses.
[4] https://patchwork.freedesktop.org/patch/588526/?series=132229&rev=1
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0+for (i =0 ; i < npages; i++) { + src_page =migrate_pfn_to_page (migrate.src[i]); + if (unlikely(!src_page || !(migrate.src[i] &
MIGRATE_PFN_MIGRATE)))Discussed this in the CPU fault patch, once we call
migrate_vma_setup, on subsequent errors we need to callmigrate_vma_finalizeto revert the pages to the original state. At least I think if I am reading the doc after this correctly.Here on error we just free the pages...
Matt
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +4
-0+ goto free_dst_page; + +xe_assert (tile->xe, !is_zone_device_page (src_page)); + src_dma_addr[i] = dma_map_page(dev, src_page, 0,
PAGE_SIZE,DMA_TO_DEVICE);drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 304
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+if (unlikely (dma_mapping_error (dev, src_dma_addr[i]))) { + drm_warn(&tile->xe->drm, "dma map error for hostpfn %lx\n", migrate.src[i]);
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +7
-0+ goto free_dst_page; + } + dst_dpa =xe_mem_region_pfn_to_dpa (mr, migrate.dst[i]); + fence =xe_migrate_pa (tile->migrate, src_dma_addr[i], false, + dst_dpa, true,PAGE_SIZE ); +if (IS_ERR (fence)) { + drm_warn(&tile->xe->drm, "migrate host page(pfn: %lx) to vram failed\n",
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 313
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+ migrate.src[i]); +/**Migration is best effort. Even we failed here, we continue*/
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0+ goto free_dst_page; + } +/**FIXME: Use the first migration's out fence as the second migration's input fence,
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +11
-0+* and so on. Only wait the out fence of last migration? +*/ +dma_fence_wait (fence, false); +dma_fence_put (fence); +free_dst_page: +xe_devm_page_free (pfn_to_page (migrate.dst[i])); + } + +for (i =0 ; i < npages; i++) +if (!(dma_mapping_error (dev, src_dma_addr[i]))) + dma_unmap_page(dev, src_dma_addr[i], PAGE_SIZE,
DMA_TO_DEVICE);Quoted 13 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +10
-1+ +migrate_vma_pages (&migrate); +migrate_vma_finalize (&migrate); +free_dst_pages: +if (ret) +xe_devm_free_blocks (&blocks); +kfree_buf: +kfree (buf); +return ret; +} --2.26.3
Matthew Brost
maintainerRe: [v2 28/31] drm/xe/svm: Introduce helper to migrate vma to vram
Quoted 142 lines · Oak
Quoted 42 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 10:49 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 28/31] drm/xe/svm: Introduce helper to migrate vma to vramQuoted 7 lines · Oak Zeng
Introduce a helper function
xe_svm_migrate_vma_to_vram.Since the source pages of the svm range can be physically not contiguous, and the destination vram pages can also be not contiguous, there is no easy way to migrate multiple pages per blitter command. We do page by page migration for now.
Quoted 6 lines
Migration is best effort. Even if we fail to migrate some pages, we will try to migrate the rest pages.
FIXME: Use one blitter command to copy when both src and dst are physically contiguous
Yep, touch in this throughout the series. Only vram needs to be contiguous though as we dynamically create PT mappings for sram pages in the migrate code. Getting this in a must and should be done immediately IMO as this a very, very basic perform thing we know needs to be done. We will likely have to tune this code quite a bit for performance so getting known things done would be helpful.
Quoted 3 lines
FIXME: when a vma is partially migrated, split vma as we assume no mixture vma placement.
Agree we do not want support partial migrations. We likely want to return
-EAGAINfor something and fault back to a smallerxe_vmachunk size which I discussed in [1] and add comment on in [2].Quoted 6 lines
Migration should be best case too if we fail migrate we can always leave backing store in sram too.
I do have question though, when do we get partial migrations? A user having called mlock on some of the pages? I just want to make sure I fully understand that case.
Yah, mlock could be one case...
I also looked the hmm code. There are few other cases where
MIGRATE_PFN_MIGRATEis not set (so we skip migration after), such as pte is NULL and vma is file-backed (not anonymous); entry is swapped out to hard disk etc. see more details in functionmigrate_vma_collect_pmd.Quoted 71 lines
[1] https://patchwork.freedesktop.org/patch/588526/?series=132229&rev=1
[2] https://patchwork.freedesktop.org/patch/588528/?series=132229&rev=1Quoted 2 lines
Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana VishwanathapuraQuoted 1 line
Signed-off-by Niranjana VishwanathapuraQuoted 6 lines
Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/xe_svm.h | 2 + drivers/gpu/drm/xe/xe_svm_migrate.c | 115++++++++++++++++++++++++++++
Same comment on file structure throughout the series apply here too.
Review
Quoted 14 lines
2 files changed, 117 insertions(+)drivers/gpu/drm/xe/xe_svm.h +2
-0@@ -83,4 +83,6 @@ int xe_devm_alloc_pages(struct xe_tile *tile,void xe_devm_free_blocks (struct list_head *blocks);void xe_devm_page_free (struct page *page);vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf); +int xe_svm_migrate_vma_to_vram (struct xe_vm *vm,struct xe_vma *vma, +struct xe_tile *tile);#endif (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_migrate.cdrivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct
Quoted 4 lines
index
0db831af098e..ab8dd1f58aa4100644drivers/gpu/drm/xe/xe_svm_migrate.c +0
-0@@ -220,3 +220,118 @@ vm_fault_t xe_svm_migrate_to_sram(struct
vm_fault*vmf)drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0kvfree (buf);return 0 ; } + +/** +* xe_svm_migrate_vma_to_vram() - migrate backing store of a vma to vram
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +12
-0+* Must be called with mmap_read_lock held. +* @vm: the vm that the vma belongs to +* @vma: the vma to migrate. +* @tile: the destination tile which holds the new backing store of the range +* +* Returns: negative errno on faiure, 0 on success +*/ +int xe_svm_migrate_vma_to_vram (struct xe_vm *vm, +struct xe_vma *vma, +struct xe_tile *tile) +{ +struct mm_struct *mm = vm->mm;drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 247
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +10
-0+unsigned long start =xe_vma_start (vma); +unsigned long end =xe_vma_end (vma); +unsigned long npages = (end - start) >>PAGE_SHIFT ; +struct xe_mem_region *mr = &tile->mem.vram; +struct vm_area_struct *vas; + +struct migrate_vma migrate = { + .start = start, + .end = end, + .pgmap_owner = tile->xe,Again helper to assign owner.
Quoted 16 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 259
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +12
-0+ .flags = MIGRATE_VMA_SELECT_SYSTEM, + }; +struct device *dev = tile->xe->drm.dev; +dma_addr_t *src_dma_addr; +struct dma_fence *fence; +struct page *src_page; +LIST_HEAD (blocks); +int ret =0 , i; +u64 dst_dpa; +void *buf; + +mmap_assert_locked (mm);This
mmap_assert_lockedis ambiguous, we should make it clear if this read or write locked. Doesn't it have to be write to do the migrate pages?
Quoted 8 lines
I followed hmm document (
Documents/mm/hmm.rst), see section "Migration to and from device memory". It explicitly write aread_lockin this document.I believe a
read_lockis enough for the migrate_vma_setup/migrate_vma_finalize().As I understand it, the mm.mmap_lock protect the process's address space. When we modify process's address space such as mmap/munmap, we need to hold a write mode lock; if we only read process's address space, such as in the migrate_vma_setup/finalize, or in the cpu page fault handler case, we only need a read mode lock.
I also cross checked amd driver. They also use a read lock.. see function
svm_range_restore_pagesinkfd_svm.c....
Yea, I see that too. Trying to figure out the locking, IMO the locking document might actually be wrong, or the very least the locking design is very ill-conceived. We can discuss internally a bit before I publically share my grievances.
Quoted 7 lines
Quoted 4 lines
A larger question about the locking... The CPU fault handler holds the mmap lock in write mode, right?
No. since cpu fault handler doesn't modify process address space, instead it only fill up cpu page table for some valid address range, a read lock is enough.
Ah, yes after digging around a bit I see this.
Quoted 8 lines
Quoted 5 lines
I'm asking as basically I think at least initially we want to hold the mmap lock in a way that the GPU handler and CPU handler do not race. i.e. From fault userptr create in GPU fault handler to issuing the bind we prevent the CPU fault handler from running.
Yes we hold
mmap_read_lockin both cpu and gpu fault handler to avoid that race.
That's not how rw lock work. 2 threads can both hold the read lock in parallel (shared read access), only 1 thread hold the write lock (exclusive write access, no one can hold read lock either). Thus my concern about the cpu and gpu fault handler running in parallel and the larger locking design questions. Again we can talk through this in detail internally a bit.
Quoted 12 lines
In user mmap/munmap (such as kernel function
vm_munmap), we hold ammap_write_lockwhich prevent it racing with cpu and gpu fault handler.Quoted 7 lines
I'm having issues figuring out how to prevent races between initial binds of userptrs and userptr invalidates on faulting VMs. This race is seen any
xe_exec_fault_modefor example... So preventing races between CPU / GPU fault handler with the mmap probably is a good idea initially. Likely can make the locking finer grained once this is all working and I figure out how to handle this race better.
Quoted 4 lines
I don't quite follow here.
Initial bind of user ptr... if you meant the bind in gpu page fault handler, then the racing with invalidation is roughly like below:
Invalidation is called withmmap_write_lock
Is it? If the notifer does the invalidation via migrate_vma_setup in the CPU fault handler we established about that only the mmap_read_lock is held.
Quoted 1 line
In
userptr_pin_page, we hold ammap_read_lock, so we know during pin page, invalidation is excluded.
Nope, see above invalidation can happen when userptr_pin_page is executing because of the read lock. The seqno check (described below) is what prevents programming of bad page tables.
Quoted 2 lines
After pin, before programming gpu page table, we check whether there is invalidation happen *after last pin but before programming page table*, if that happened, we retry
Yes, that is how it works on tip but I am refactoring it in [1]. I was trying to avoid the retry loop by turning PDE/PTE writes into clears if an invalidation happened but not sure if that works without a larger refactor due to nature PDEs being shared. I may need the retry loop but that also gets tricky with array of binds... A few options here and will land a on solution once [2] is merged.
Regardless my point here is still valid - it may not be the worst idea when getting this code initially up and running just to grab mmap_write_lock in GPU fault handler as a BLK (big kernel lock) to prevent all races. Once the code is stable and stress testing in place, switch to finer grained locking as define in HMM document (or newly defined if we fine locking design is insufficient).
Matt
[1] https://patchwork.freedesktop.org/series/125608/
[2] https://patchwork.freedesktop.org/series/132246/
Quoted 137 lines
Oak
Quoted 133 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 261
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+ + vas =find_vma_intersection (mm, start, start +4 );
find_vmashould work fine here.drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 273
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +12
-0+if (!vas) +return -ENOENT ; + + migrate.vma = vas; + buf =kvcalloc (npages,2 *sizeof (*migrate.src) +sizeof (*src_dma_addr), +GFP_KERNEL ); +if (!buf) +return -ENOMEM ; + migrate.src = buf; + migrate.dst = migrate.src + npages; + src_dma_addr = (dma_addr_t *) (migrate.dst + npages); + ret =xe_devm_alloc_pages (tile, npages, &blocks, migrate.dst);Again as I discussed in [3] I think this should be broken out into a different step with the blocks allocated before this, and here just populate migrate.dst from the existing blocks.
[3] https://patchwork.freedesktop.org/patch/588523/?series=132229&rev=1
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +6
-0+if (ret) + goto kfree_buf; + + ret =migrate_vma_setup (&migrate); +if (ret) { + drm_err(&tile->xe->drm, "vma setup returned %d for range[%lx - %lx]\n",
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +10
-0+ ret, start, end); + goto free_dst_pages; + } + +/**FIXME: partial migration of a range print a warning for now. +* If this message is printed, we need to split xe_vma as we +* don't support a mixture placement of one vma +*/ +if (migrate.cpages != npages) + drm_warn(&tile->xe->drm, "Partial migration for range [%lx -%lx], range is %ld pages, migrate only %ld pages\n",
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 290
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c +1
-0+ start, end, npages, migrate.cpages);As discussed above, we shouldn't support this. We should fall back to smaller
xe_vmachunk size until we find one that works or simply leave the pages in sram and map those pages to GPU.drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0+ +/**Migrate page by page for now. +* Both source pages and destination pages can physically not contiguous,
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 295
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+* there is no good way to migrate multiple pages per blitter command. +*/ Touched on this a bunch throughout the series, lets do better than a page a time migration.
Algorithm should be very similar to what I discussed here [4] but with a few key differences.
- I think the sram pages can be unpopulated (page == NULL) if the user has not yet touched the page
- Also I think theMIGRATE_PFN_MIGRATEbit being clear is validIn these cases this indicate we have to issue a copy for the pages we are accumulating with contigous vram addresses.
[4] https://patchwork.freedesktop.org/patch/588526/?series=132229&rev=1
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0+for (i =0 ; i < npages; i++) { + src_page =migrate_pfn_to_page (migrate.src[i]); + if (unlikely(!src_page || !(migrate.src[i] &
MIGRATE_PFN_MIGRATE)))Discussed this in the CPU fault patch, once we call
migrate_vma_setup, on subsequent errors we need to callmigrate_vma_finalizeto revert the pages to the original state. At least I think if I am reading the doc after this correctly.Here on error we just free the pages...
Matt
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +4
-0+ goto free_dst_page; + +xe_assert (tile->xe, !is_zone_device_page (src_page)); + src_dma_addr[i] = dma_map_page(dev, src_page, 0,
PAGE_SIZE,DMA_TO_DEVICE);drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 304
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+if (unlikely (dma_mapping_error (dev, src_dma_addr[i]))) { + drm_warn(&tile->xe->drm, "dma map error for hostpfn %lx\n", migrate.src[i]);
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +7
-0+ goto free_dst_page; + } + dst_dpa =xe_mem_region_pfn_to_dpa (mr, migrate.dst[i]); + fence =xe_migrate_pa (tile->migrate, src_dma_addr[i], false, + dst_dpa, true,PAGE_SIZE ); +if (IS_ERR (fence)) { + drm_warn(&tile->xe->drm, "migrate host page(pfn: %lx) to vram failed\n",
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 313
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+ migrate.src[i]); +/**Migration is best effort. Even we failed here, we continue*/
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0+ goto free_dst_page; + } +/**FIXME: Use the first migration's out fence as the second migration's input fence,
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +11
-0+* and so on. Only wait the out fence of last migration? +*/ +dma_fence_wait (fence, false); +dma_fence_put (fence); +free_dst_page: +xe_devm_page_free (pfn_to_page (migrate.dst[i])); + } + +for (i =0 ; i < npages; i++) +if (!(dma_mapping_error (dev, src_dma_addr[i]))) + dma_unmap_page(dev, src_dma_addr[i], PAGE_SIZE,
DMA_TO_DEVICE);Quoted 13 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +10
-1+ +migrate_vma_pages (&migrate); +migrate_vma_finalize (&migrate); +free_dst_pages: +if (ret) +xe_devm_free_blocks (&blocks); +kfree_buf: +kfree (buf); +return ret; +} --2.26.3
RE: [v2 28/31] drm/xe/svm: Introduce helper to migrate vma to vram
Hi Matt,
Quoted 37 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Monday, April 15, 2024 3:40 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 28/31] drm/xe/svm: Introduce helper to migrate vma to vramQuoted 9 lines · Oak
Quoted 9 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 10:49 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected];Welty,
Quoted 2 lines
Quoted 2 lines
Brian <[email protected]>
Subject: Re: [v2 28/31] drm/xe/svm: Introduce helper to migrate vma tovram
Quoted 12 lines
Quoted 12 lines · Oak Zeng
Quoted 10 lines
Introduce a helper function
xe_svm_migrate_vma_to_vram.Since the source pages of the svm range can be physically not contiguous, and the destination vram pages can also be not contiguous, there is no easy way to migrate multiple pages per blitter command. We do page by page migration for now.
Migration is best effort. Even if we fail to migrate some pages, we will try to migrate the rest pages.
Quoted 19 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
FIXME: Use one blitter command to copy when both src and dst are physically contiguous
Yep, touch in this throughout the series. Only vram needs to be contiguous though as we dynamically create PT mappings for sram pages
in
Quoted 12 lines
Quoted 12 lines
the migrate code. Getting this in a must and should be done immediately IMO as this a very, very basic perform thing we know needs to be done. We will likely have to tune this code quite a bit for performance so getting known things done would be helpful.
Quoted 3 lines
FIXME: when a vma is partially migrated, split vma as we assume no mixture vma placement.
Agree we do not want support partial migrations. We likely want to return
-EAGAINfor something and fault back to a smallerxe_vmachunk size which I discussed in [1] and add comment on in [2].
I thought into this when I worked on v3. I end up with keep the xe-vma not splitting, but migrate partially of a vma. A vma can have mixed backing store now... To support this, I had to make the page table update and invalidation range based (vs whole xe-vma based) I am testing it. Not very sure how well this goes.
One thing I realized that might now work well is the range based invalidation. Say we a pde entry covering a huge 2M or 1G page. If we only zap 4k in this rang, we will need to split pde entry and replace it with many pde/pte entries. Even though the zap_ptes/page table walk functions are range based, I doubt it works for such case. I will have to test it out. I think even if it doesn't work, it worth to make it work as an improvement area. For example, core mm obviously support such function, see zap_pte_range.
Quoted 153 lines
Quoted 11 lines
Quoted 7 lines
Migration should be best case too if we fail migrate we can always leave backing store in sram too.
I do have question though, when do we get partial migrations? A user having called mlock on some of the pages? I just want to make sure I fully understand that case.
Yah, mlock could be one case...
I also looked the hmm code. There are few other cases where
MIGRATE_PFN_MIGRATEis not set (so we skip migration after), such as pte is NULL and vma is file-backed (not anonymous); entry is swapped out to hard disk etc. see more details in functionmigrate_vma_collect_pmd.Quoted 4 lines
Quoted 4 lines
[1]
https://patchwork.freedesktop.org/patch/588526/?series=132229&rev=1
Quoted 1 line
Quoted 1 line
[2]
https://patchwork.freedesktop.org/patch/588528/?series=132229&rev=1
Quoted 12 lines
Quoted 12 lines
Quoted 3 lines
Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana VishwanathapuraQuoted 1 line
Signed-off-by Niranjana VishwanathapuraQuoted 6 lines
Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/xe_svm.h | 2 + drivers/gpu/drm/xe/xe_svm_migrate.c | 115Quoted 7 lines
Quoted 7 lines
++++++++++++++++++++++++++++
Same comment on file structure throughout the series apply here too.
Review
Quoted 3 lines
2 files changed, 117 insertions(+)(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm.hQuoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm.h int xe_devm_alloc_pages(struct xe_tile *tile, · line 86
Quoted 8 lines
index
c9e4239c44b4..18ce2e3757c5100644drivers/gpu/drm/xe/xe_svm.h +1
-0@@ -83,4 +83,6 @@ int xe_devm_alloc_pages(struct xe_tile *tile,void xe_devm_free_blocks (struct list_head *blocks);void xe_devm_page_free (struct page *page);vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf); +int xe_svm_migrate_vma_to_vram(struct xe_vm *vm, struct xe_vma*vma,
Quoted 9 lines
Quoted 9 lines
Review
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm.h +1
-0+ struct xe_tile *tile);#endif (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_migrate.cdrivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct
Quoted 4 lines
index
0db831af098e..ab8dd1f58aa4100644drivers/gpu/drm/xe/xe_svm_migrate.c +0
-0@@ -220,3 +220,118 @@ vm_fault_t xe_svm_migrate_to_sram(struct
vm_fault*vmf)Quoted 6 lines
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0kvfree (buf);return 0 ; } + +/** +* xe_svm_migrate_vma_to_vram() - migrate backing store of a vma to
Quoted 5 lines
Quoted 5 lines
vram
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +4
-0+* Must be called with mmap_read_lock held. +* @vm: the vm that the vma belongs to +* @vma: the vma to migrate. +* @tile: the destination tile which holds the new backing store of the range
Quoted 38 lines
Quoted 21 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +8
-0+* +* Returns: negative errno on faiure, 0 on success +*/ +int xe_svm_migrate_vma_to_vram (struct xe_vm *vm, +struct xe_vma *vma, +struct xe_tile *tile) +{ +struct mm_struct *mm = vm->mm;drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 247
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +10
-0+unsigned long start =xe_vma_start (vma); +unsigned long end =xe_vma_end (vma); +unsigned long npages = (end - start) >>PAGE_SHIFT ; +struct xe_mem_region *mr = &tile->mem.vram; +struct vm_area_struct *vas; + +struct migrate_vma migrate = { + .start = start, + .end = end, + .pgmap_owner = tile->xe,Again helper to assign owner.
Quoted 16 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 259
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +12
-0+ .flags = MIGRATE_VMA_SELECT_SYSTEM, + }; +struct device *dev = tile->xe->drm.dev; +dma_addr_t *src_dma_addr; +struct dma_fence *fence; +struct page *src_page; +LIST_HEAD (blocks); +int ret =0 , i; +u64 dst_dpa; +void *buf; + +mmap_assert_locked (mm);This
mmap_assert_lockedis ambiguous, we should make it clear if this read or write locked. Doesn't it have to be write to do the migrate pages?Quoted 1 line
I followed hmm document (
Documents/mm/hmm.rst), see section"Migration to and from device memory". It explicitly write a
read_lockin this document.Quoted 2 lines
I believe a
read_lockis enough for themigrate_vma_setup/migrate_vma_finalize().
Quoted 2 lines
As I understand it, the mm.mmap_lock protect the process's address space.
When we modify process's address space such as mmap/munmap, we need to hold a write mode lock; if we only read process's address space, such as in the migrate_vma_setup/finalize, or in the cpu page fault handler case, we only need a read mode lock.
Quoted 2 lines
I also cross checked amd driver. They also use a read lock.. see function
svm_range_restore_pagesinkfd_svm.c....Quoted 1 line
Yea, I see that too. Trying to figure out the locking, IMO the locking document might actually be wrong, or the very least the locking design is very ill-conceived. We can discuss internally a bit before I publically share my grievances.
Quoted 6 lines
Quoted 4 lines
A larger question about the locking... The CPU fault handler holds the mmap lock in write mode, right?
No. since cpu fault handler doesn't modify process address space, instead it
only fill up cpu page table for some valid address range, a read lock is enough.
Quoted 1 line
Ah, yes after digging around a bit I see this.
Quoted 7 lines
Quoted 5 lines
I'm asking as basically I think at least initially we want to hold the mmap lock in a way that the GPU handler and CPU handler do not race. i.e. From fault userptr create in GPU fault handler to issuing the bind we prevent the CPU fault handler from running.
Yes we hold
mmap_read_lockin both cpu and gpu fault handler to avoidthat race.
Quoted 1 line
Quoted 7 lines
That's not how rw lock work. 2 threads can both hold the read lock in parallel (shared read access), only 1 thread hold the write lock (exclusive write access, no one can hold read lock either). Thus my concern about the cpu and gpu fault handler running in parallel and the larger locking design questions. Again we can talk through this in detail internally a bit.
It is a great question of the gpu/cpu handler race. Obviously my above thinking is wrong. Here is my new understanding:
think about this scenario: address backing store is in vram - cpu access - cpu fault- migrate_vma_setup - invalidate gpu pt - gpu access same address - gpu fault - migrate_vma_setup triggered by gpu fault - this read back pfn of vram backing store because cpu fault is not finished/migrate_vma_finalized is not called yet, but since gpu side would only migrate memory in sram, so migrate_vma_setup from gpu side wouldn't return any valid pfn to migrate and gpu page fault handler would return successfully and gpu hw execution is resumed but immediately trigger another gpu page fault, until cpu side migration is done.
Similar when gpu fault happens during the other stage of cpu fault handler. Same thing when CPU fault handler interrupt a gpu fault handler.
This seems work to me but we need to test it out. We will add a test case for this race condition. @Bommu, Krishnaiah fyi.
Quoted 16 lines
Quoted 1 line
In user mmap/munmap (such as kernel function
vm_munmap), we hold a
mmap_write_lockwhich prevent it racing with cpu and gpu fault handler.Quoted 6 lines
Quoted 6 lines
I'm having issues figuring out how to prevent races between initial binds of userptrs and userptr invalidates on faulting VMs. This race is seen any
xe_exec_fault_modefor example... So preventing racesbetween
Quoted 7 lines
Quoted 3 lines
CPU / GPU fault handler with the mmap probably is a good idea initially. Likely can make the locking finer grained once this is all working and I figure out how to handle this race better.
I don't quite follow here.
Quoted 7 lines
Quoted 1 line
Initial bind of user ptr... if you meant the bind in gpu page fault handler,
then the racing with invalidation is roughly like below:
Quoted 1 line
Invalidation is called with
mmap_write_lockIs it? If the notifer does the invalidation via
migrate_vma_setupin the CPU fault handler we established about that only themmap_read_lockis held.
Yes the cpu fault triggered migrate-vma-setup/invalidation only hold read_lock.
It was the munmap triggered invalidation in my mind. For this case, we hold mmap_write_lock during invalidation, so it is mutual exclusive with gpu triggered migration.
But a cpu fault triggered invalidation can proceed during gpu fault triggered migration. As explained above, it cause extra gpu page table invalidation which is just fine because the consequence is just extra gpu page fault (and later migration). So functionally I don't see a problem of this scheme.
The main reason we hold a read lock in gpu fault handler is, gpu fault handler doesn't modify process address space, rather it only traverse the cpu page table.
Quoted 7 lines
Quoted 2 lines
In
userptr_pin_page, we hold ammap_read_lock, so we know during pinpage, invalidation is excluded.
Nope, see above invalidation can happen when
userptr_pin_pageis executing because of the read lock. The seqno check (described below) is what prevents programming of bad page tables.
You are right.
Quoted 6 lines
Quoted 2 lines
After pin, before programming gpu page table, we check whether there is
invalidation happen *after last pin but before programming page table*, if that happened, we retry
Quoted 1 line
Quoted 6 lines
Yes, that is how it works on tip but I am refactoring it in [1]. I was trying to avoid the retry loop by turning PDE/PTE writes into clears if an invalidation happened but not sure if that works without a larger refactor due to nature PDEs being shared. I may need the retry loop but that also gets tricky with array of binds... A few options here and will land a on solution once [2] is merged.
I discussed this retry loop (I refer to the retry loop in xe-pt-userptr-pre-commit function before your change)with Thomas some time ago. I agree the retry loop scheme work
But to me other scheme without loop also work. As long as we have a retry check before programming gpu page table, it is fine to me.
If you have further fix in this area, I will keep watching and pick it.
Quoted 7 lines
Regardless my point here is still valid - it may not be the worst idea when getting this code initially up and running just to grab
mmap_write_lockin GPU fault handler as a BLK (big kernel lock) to prevent all races. Once the code is stable and stress testing in place, switch to finer grained locking as define in HMM document (or newly defined if we fine locking design is insufficient).
As explained, I will keep the read-lock for now. And we will test it. If we do run into race condition, I will try write-lock and come back.
Thanks a lot for the great questions Matt! It is very helpful!
Oak
Quoted 147 lines
Matt
[1] https://patchwork.freedesktop.org/series/125608/
[2] https://patchwork.freedesktop.org/series/132246/Quoted 10 lines
Oak
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 261
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+ + vas =find_vma_intersection (mm, start, start +4 );
find_vmashould work fine here.Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +5
-0+if (!vas) +return -ENOENT ; + + migrate.vma = vas; + buf = kvcalloc(npages, 2* sizeof(*migrate.src) +
sizeof(*src_dma_addr),Quoted 12 lines
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 273
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +7
-0+ GFP_KERNEL); +if (!buf) +return -ENOMEM ; + migrate.src = buf; + migrate.dst = migrate.src + npages; + src_dma_addr = (dma_addr_t *) (migrate.dst + npages); + ret =xe_devm_alloc_pages (tile, npages, &blocks, migrate.dst);Again as I discussed in [3] I think this should be broken out into a different step with the blocks allocated before this, and here just populate migrate.dst from the existing blocks.
Quoted 1 line
Quoted 1 line
[3]
https://patchwork.freedesktop.org/patch/588523/?series=132229&rev=1
Quoted 25 lines
Quoted 25 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +6
-0+if (ret) + goto kfree_buf; + + ret =migrate_vma_setup (&migrate); +if (ret) { + drm_err(&tile->xe->drm, "vma setup returned %d for range[%lx - %lx]\n",
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +10
-0+ ret, start, end); + goto free_dst_pages; + } + +/**FIXME: partial migration of a range print a warning for now. +* If this message is printed, we need to split xe_vma as we +* don't support a mixture placement of one vma +*/ +if (migrate.cpages != npages) + drm_warn(&tile->xe->drm, "Partial migration for range [%lx -%lx], range is %ld pages, migrate only %ld pages\n",
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 290
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c +1
-0+ start, end, npages, migrate.cpages);As discussed above, we shouldn't support this. We should fall back to smaller
xe_vmachunk size until we find one that works or simply leave the pages in sram and map those pages to GPU.Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0+ +/**Migrate page by page for now. +* Both source pages and destination pages can physically not contiguous,
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 294
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c +1
-0+* there is no good way to migrate multiple pages per blitter command.
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 295
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c +1
-0+*/ Touched on this a bunch throughout the series, lets do better than a page a time migration.
Algorithm should be very similar to what I discussed here [4] but with a few key differences.
Quoted 8 lines
Quoted 8 lines
- I think the sram pages can be unpopulated (page == NULL) if the user has not yet touched the page
- Also I think theMIGRATE_PFN_MIGRATEbit being clear is validIn these cases this indicate we have to issue a copy for the pages we are accumulating with contigous vram addresses.
[4]
https://patchwork.freedesktop.org/patch/588526/?series=132229&rev=1
Quoted 63 lines
Quoted 63 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0+for (i =0 ; i < npages; i++) { + src_page =migrate_pfn_to_page (migrate.src[i]); + if (unlikely(!src_page || !(migrate.src[i] &
MIGRATE_PFN_MIGRATE)))Discussed this in the CPU fault patch, once we call
migrate_vma_setup, on subsequent errors we need to callmigrate_vma_finalizeto revert the pages to the original state. At least I think if I am reading the doc after this correctly.Here on error we just free the pages...
Matt
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +4
-0+ goto free_dst_page; + +xe_assert (tile->xe, !is_zone_device_page (src_page)); + src_dma_addr[i] = dma_map_page(dev, src_page, 0,
PAGE_SIZE,DMA_TO_DEVICE);drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 304
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+if (unlikely (dma_mapping_error (dev, src_dma_addr[i]))) { + drm_warn(&tile->xe->drm, "dma map error for hostpfn %lx\n", migrate.src[i]);
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +7
-0+ goto free_dst_page; + } + dst_dpa =xe_mem_region_pfn_to_dpa (mr, migrate.dst[i]); + fence =xe_migrate_pa (tile->migrate, src_dma_addr[i], false, + dst_dpa, true,PAGE_SIZE ); +if (IS_ERR (fence)) { + drm_warn(&tile->xe->drm, "migrate host page(pfn: %lx) to vram failed\n",
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 313
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+ migrate.src[i]); +/**Migration is best effort. Even we failed here, we continue*/
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0+ goto free_dst_page; + } +/**FIXME: Use the first migration's out fence as the second migration's input fence,
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +11
-0+* and so on. Only wait the out fence of last migration? +*/ +dma_fence_wait (fence, false); +dma_fence_put (fence); +free_dst_page: +xe_devm_page_free (pfn_to_page (migrate.dst[i])); + } + +for (i =0 ; i < npages; i++) +if (!(dma_mapping_error (dev, src_dma_addr[i]))) + dma_unmap_page(dev, src_dma_addr[i], PAGE_SIZE,
DMA_TO_DEVICE);Quoted 13 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +10
-1+ +migrate_vma_pages (&migrate); +migrate_vma_finalize (&migrate); +free_dst_pages: +if (ret) +xe_devm_free_blocks (&blocks); +kfree_buf: +kfree (buf); +return ret; +} --2.26.3
Matthew Brost
maintainerRe: [v2 28/31] drm/xe/svm: Introduce helper to migrate vma to vram
Quoted 55 lines · Oak
Hi Matt,
Quoted 53 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Monday, April 15, 2024 3:40 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 28/31] drm/xe/svm: Introduce helper to migrate vma to vramQuoted 9 lines · Oak
Quoted 9 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 10:49 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected];Welty,
Quoted 2 lines
Quoted 2 lines
Brian <[email protected]>
Subject: Re: [v2 28/31] drm/xe/svm: Introduce helper to migrate vma tovram
Quoted 12 lines
Quoted 12 lines · Oak Zeng
Quoted 10 lines
Introduce a helper function
xe_svm_migrate_vma_to_vram.Since the source pages of the svm range can be physically not contiguous, and the destination vram pages can also be not contiguous, there is no easy way to migrate multiple pages per blitter command. We do page by page migration for now.
Migration is best effort. Even if we fail to migrate some pages, we will try to migrate the rest pages.
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
FIXME: Use one blitter command to copy when both src and dst are physically contiguous
Yep, touch in this throughout the series. Only vram needs to be contiguous though as we dynamically create PT mappings for sram pages
in
Quoted 9 lines
Quoted 9 lines
the migrate code. Getting this in a must and should be done immediately IMO as this a very, very basic perform thing we know needs to be done. We will likely have to tune this code quite a bit for performance so getting known things done would be helpful.
Quoted 3 lines
FIXME: when a vma is partially migrated, split vma as we assume no mixture vma placement.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Agree we do not want support partial migrations. We likely want to return
-EAGAINfor something and fault back to a smallerxe_vmachunk size which I discussed in [1] and add comment on in [2].I thought into this when I worked on v3. I end up with keep the xe-vma not splitting, but migrate partially of a vma. A vma can have mixed backing store now... To support this, I had to make the page table update and invalidation range based (vs whole xe-vma based) I am testing it. Not very sure how well this goes.
A few things, I strongly oppose mixed storage within a single GPU mapping. I view this as an unnecessary complication.
A GPU mapping is not a VMA - it is a subset of a VMA. It has been agreed upon we have a 1:N relationship in SVM. With '1' being a GPU VMA which reflects user IOCTLs (e.g. initial large bind, madvise calls which can split the VMA). The 'N' being mappings dynamically created on upon GPU page fault. This a xe_svm_range (subclass of drm_svm_range) in my PoC. My expectation is that any post has something similar to this agreed upon relationship implemented.
Here my view is the 'N' never is mixed storage.
See my comments about 'page grainularity' migration in 'implement functions to allocate and free device memory'. If you are doing page grainularity migration, then 'N' is exactly 1 CPU page (could be THP) thus you never have a case where partial migrations or mixed storage is needed. Any spliting / mixed storage by definition is not page grainularity.
If we don't fully support page grainularity, I think the grainularity for any operation is allocation size of 'N'. e.g. Upon CPU page, CPU unmap, evict, 'N' is fully moved, invalidated, GPU unmapped. This simplifies the code quite a bit. See 'Migrataion', 'Partial Unmapping of Ranges', and 'Examples' in drm_gpusvm.c. Of course open to others idea here but anything that is more complicated that what I'm suggesting, expect push back.
Quoted 2 lines
One thing I realized that might now work well is the range based invalidation. Say we a pde entry covering a huge 2M or 1G page. If we only zap 4k in this rang, we will need to split pde entry and replace it with many pde/pte entries. Even though the zap_ptes/page table walk functions are range based, I doubt it works for such case. I will have to test it out. I think even if it doesn't work, it worth to make it work as an improvement area. For example, core mm obviously support such function, see
zap_pte_range.
If we have GPU page tables larger than the CPU pages, then we are not doing 'page grainularity'.
This is a large source of confusion for as you seem to have conflicting ideas in reply to reply. Do plan on supporting 'page grainularity' or not? This is unclear to me.
If we don't support 'page grainularity', then see my above for how I expect an impedance mismatch between the CPU page size and 'N'. That applies here as well.
Quoted 164 lines
Quoted 155 lines
Quoted 6 lines
Quoted 6 lines
Migration should be best case too if we fail migrate we can always leave backing store in sram too.
Quoted 7 lines
Quoted 3 lines
I do have question though, when do we get partial migrations? A user having called mlock on some of the pages? I just want to make sure I fully understand that case.
Yah, mlock could be one case...
I also looked the hmm code. There are few other cases where
MIGRATE_PFN_MIGRATEis not set (so we skip migration after), such as pte is NULL and vma is file-backed (not anonymous); entry is swapped out to hard disk etc. see more details in functionmigrate_vma_collect_pmd.Quoted 4 lines
Quoted 4 lines
[1]
https://patchwork.freedesktop.org/patch/588526/?series=132229&rev=1
Quoted 1 line
Quoted 1 line
[2]
https://patchwork.freedesktop.org/patch/588528/?series=132229&rev=1
Quoted 12 lines
Quoted 12 lines
Quoted 3 lines
Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana VishwanathapuraQuoted 1 line
Signed-off-by Niranjana VishwanathapuraQuoted 6 lines
Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/xe_svm.h | 2 + drivers/gpu/drm/xe/xe_svm_migrate.c | 115Quoted 7 lines
Quoted 7 lines
++++++++++++++++++++++++++++
Same comment on file structure throughout the series apply here too.
Review
Quoted 3 lines
2 files changed, 117 insertions(+)(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm.hQuoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm.h int xe_devm_alloc_pages(struct xe_tile *tile, · line 86
Quoted 8 lines
index
c9e4239c44b4..18ce2e3757c5100644drivers/gpu/drm/xe/xe_svm.h +1
-0@@ -83,4 +83,6 @@ int xe_devm_alloc_pages(struct xe_tile *tile,void xe_devm_free_blocks (struct list_head *blocks);void xe_devm_page_free (struct page *page);vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf); +int xe_svm_migrate_vma_to_vram(struct xe_vm *vm, struct xe_vma*vma,
Quoted 9 lines
Quoted 9 lines
Review
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm.h +1
-0+ struct xe_tile *tile);#endif (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_migrate.cdrivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct
Quoted 4 lines
index
0db831af098e..ab8dd1f58aa4100644drivers/gpu/drm/xe/xe_svm_migrate.c +0
-0@@ -220,3 +220,118 @@ vm_fault_t xe_svm_migrate_to_sram(struct
vm_fault*vmf)Quoted 6 lines
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0kvfree (buf);return 0 ; } + +/** +* xe_svm_migrate_vma_to_vram() - migrate backing store of a vma to
Quoted 5 lines
Quoted 5 lines
vram
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +4
-0+* Must be called with mmap_read_lock held. +* @vm: the vm that the vma belongs to +* @vma: the vma to migrate. +* @tile: the destination tile which holds the new backing store of the range
Quoted 38 lines
Quoted 21 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +8
-0+* +* Returns: negative errno on faiure, 0 on success +*/ +int xe_svm_migrate_vma_to_vram (struct xe_vm *vm, +struct xe_vma *vma, +struct xe_tile *tile) +{ +struct mm_struct *mm = vm->mm;drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 247
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +10
-0+unsigned long start =xe_vma_start (vma); +unsigned long end =xe_vma_end (vma); +unsigned long npages = (end - start) >>PAGE_SHIFT ; +struct xe_mem_region *mr = &tile->mem.vram; +struct vm_area_struct *vas; + +struct migrate_vma migrate = { + .start = start, + .end = end, + .pgmap_owner = tile->xe,Again helper to assign owner.
Quoted 16 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 259
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +12
-0+ .flags = MIGRATE_VMA_SELECT_SYSTEM, + }; +struct device *dev = tile->xe->drm.dev; +dma_addr_t *src_dma_addr; +struct dma_fence *fence; +struct page *src_page; +LIST_HEAD (blocks); +int ret =0 , i; +u64 dst_dpa; +void *buf; + +mmap_assert_locked (mm);This
mmap_assert_lockedis ambiguous, we should make it clear if this read or write locked. Doesn't it have to be write to do the migrate pages?Quoted 1 line
I followed hmm document (
Documents/mm/hmm.rst), see section"Migration to and from device memory". It explicitly write a
read_lockin this document.Quoted 2 lines
I believe a
read_lockis enough for themigrate_vma_setup/migrate_vma_finalize().
Quoted 2 lines
As I understand it, the mm.mmap_lock protect the process's address space.
When we modify process's address space such as mmap/munmap, we need to hold a write mode lock; if we only read process's address space, such as in the migrate_vma_setup/finalize, or in the cpu page fault handler case, we only need a read mode lock.
Quoted 2 lines
I also cross checked amd driver. They also use a read lock.. see function
svm_range_restore_pagesinkfd_svm.c....Quoted 1 line
Yea, I see that too. Trying to figure out the locking, IMO the locking document might actually be wrong, or the very least the locking design is very ill-conceived. We can discuss internally a bit before I publically share my grievances.
Quoted 6 lines
Quoted 4 lines
A larger question about the locking... The CPU fault handler holds the mmap lock in write mode, right?
No. since cpu fault handler doesn't modify process address space, instead it
only fill up cpu page table for some valid address range, a read lock is enough.
Quoted 1 line
Ah, yes after digging around a bit I see this.
Quoted 7 lines
Quoted 5 lines
I'm asking as basically I think at least initially we want to hold the mmap lock in a way that the GPU handler and CPU handler do not race. i.e. From fault userptr create in GPU fault handler to issuing the bind we prevent the CPU fault handler from running.
Yes we hold
mmap_read_lockin both cpu and gpu fault handler to avoidthat race.
Quoted 1 line
Quoted 7 lines
That's not how rw lock work. 2 threads can both hold the read lock in parallel (shared read access), only 1 thread hold the write lock (exclusive write access, no one can hold read lock either). Thus my concern about the cpu and gpu fault handler running in parallel and the larger locking design questions. Again we can talk through this in detail internally a bit.
Quoted 4 lines
It is a great question of the gpu/cpu handler race. Obviously my above thinking is wrong. Here is my new understanding:
think about this scenario: address backing store is in vram - cpu access - cpu fault-
migrate_vma_setup- invalidate gpu pt - gpu access same address - gpu fault -migrate_vma_setuptriggered by gpu fault - this read back pfn of vram backing store because cpu fault is not finished/migrate_vma_finalized is not called yet, but since gpu side would only migrate memory in sram, somigrate_vma_setupfrom gpu side wouldn't return any valid pfn to migrate and gpu page fault handler would return successfully and gpu hw execution is resumed but immediately trigger another gpu page fault, until cpu side migration is done.
This is hard to follow, especially given your comments drop wrap at 80 lines or so. Please use an email client which wraps. I don't think you reasoning is right.
I have worked this out and landed on HMM locking documentation is wrong or at least doesn't work unless a notifier is installed upon every GPU fault. Bascially the GPU handler takes mmap lock in write mode to work around this (a notifier installation takes mmap in write mode which is why I reasoned HMM lock doc suggestion works in some cases). Longterm will need some depth though about to fix this. It might be an extra MMU invalidation event upon migrate_vma_finalize or somethig... Again see my PoC, it has comments in the code explaining this.
Quoted 5 lines
Similar when gpu fault happens during the other stage of cpu fault handler. Same thing when CPU fault handler interrupt a gpu fault handler.
This seems work to me but we need to test it out. We will add a test case for this race condition. @Bommu, Krishnaiah fyi.
I have a test for this, see race sections in [1].
[1] https://patchwork.freedesktop.org/series/133846/
Quoted 24 lines
Quoted 16 lines
Quoted 1 line
In user mmap/munmap (such as kernel function
vm_munmap), we hold a
mmap_write_lockwhich prevent it racing with cpu and gpu fault handler.Quoted 6 lines
Quoted 6 lines
I'm having issues figuring out how to prevent races between initial binds of userptrs and userptr invalidates on faulting VMs. This race is seen any
xe_exec_fault_modefor example... So preventing racesbetween
Quoted 7 lines
Quoted 3 lines
CPU / GPU fault handler with the mmap probably is a good idea initially. Likely can make the locking finer grained once this is all working and I figure out how to handle this race better.
I don't quite follow here.
Quoted 7 lines
Quoted 1 line
Initial bind of user ptr... if you meant the bind in gpu page fault handler,
then the racing with invalidation is roughly like below:
Quoted 1 line
Invalidation is called with
mmap_write_lockIs it? If the notifer does the invalidation via
migrate_vma_setupin the CPU fault handler we established about that only themmap_read_lockis held.
Quoted 8 lines
Yes the cpu fault triggered migrate-vma-setup/invalidation only hold
read_lock.It was the munmap triggered invalidation in my mind. For this case, we hold
mmap_write_lockduring invalidation, so it is mutual exclusive with gpu triggered migration.But a cpu fault triggered invalidation can proceed during gpu fault triggered migration. As explained above, it cause extra gpu page table invalidation which is just fine because the consequence is just extra gpu page fault (and later migration). So functionally I don't see a problem of this scheme.
The main reason we hold a read lock in gpu fault handler is, gpu fault handler doesn't modify process address space, rather it only traverse the cpu page table.
Again not following, but quite sure your understand is not correct. See above.
Quoted 24 lines
Quoted 7 lines
Quoted 2 lines
In
userptr_pin_page, we hold ammap_read_lock, so we know during pinpage, invalidation is excluded.
Nope, see above invalidation can happen when
userptr_pin_pageis executing because of the read lock. The seqno check (described below) is what prevents programming of bad page tables.You are right.
Quoted 6 lines
Quoted 2 lines
After pin, before programming gpu page table, we check whether there is
invalidation happen *after last pin but before programming page table*, if that happened, we retry
Quoted 1 line
Quoted 6 lines
Yes, that is how it works on tip but I am refactoring it in [1]. I was trying to avoid the retry loop by turning PDE/PTE writes into clears if an invalidation happened but not sure if that works without a larger refactor due to nature PDEs being shared. I may need the retry loop but that also gets tricky with array of binds... A few options here and will land a on solution once [2] is merged.
Quoted 6 lines
I discussed this retry loop (I refer to the retry loop in xe-pt-userptr-pre-commit function before your change)with Thomas some time ago. I agree the retry loop scheme work
But to me other scheme without loop also work. As long as we have a retry check before programming gpu page table, it is fine to me.
If you have further fix in this area, I will keep watching and pick it.
I have this properly fixed now in [2]. Faulting VMs will return with -EAGAIN if the seqno check fails in a bind. If will be up to the page fault handler to retry. In the case of user binds, this so race I currently am just kicking the -EAGAIN to userpace to retry. I could change the later if people disagre and retry in the kernel too.
[2] https://patchwork.freedesktop.org/series/133034/.
Quoted 10 lines
Quoted 7 lines
Regardless my point here is still valid - it may not be the worst idea when getting this code initially up and running just to grab
mmap_write_lockin GPU fault handler as a BLK (big kernel lock) to prevent all races. Once the code is stable and stress testing in place, switch to finer grained locking as define in HMM document (or newly defined if we fine locking design is insufficient).As explained, I will keep the read-lock for now. And we will test it. If we do run into race condition, I will try write-lock and come back.
See above, you will hit a race.
I strongly suggest you checkout my working PoC and IGT and play around with that code. If you change from write lock to read lock in GPU handler around VRAM migration and hmm range fault, the 'race' sections fail.
Matt
Quoted 152 lines
Thanks a lot for the great questions Matt! It is very helpful!
Oak
Quoted 147 lines
Matt
[1] https://patchwork.freedesktop.org/series/125608/
[2] https://patchwork.freedesktop.org/series/132246/Quoted 10 lines
Oak
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 261
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+ + vas =find_vma_intersection (mm, start, start +4 );
find_vmashould work fine here.Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +5
-0+if (!vas) +return -ENOENT ; + + migrate.vma = vas; + buf = kvcalloc(npages, 2* sizeof(*migrate.src) +
sizeof(*src_dma_addr),Quoted 12 lines
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 273
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +7
-0+ GFP_KERNEL); +if (!buf) +return -ENOMEM ; + migrate.src = buf; + migrate.dst = migrate.src + npages; + src_dma_addr = (dma_addr_t *) (migrate.dst + npages); + ret =xe_devm_alloc_pages (tile, npages, &blocks, migrate.dst);Again as I discussed in [3] I think this should be broken out into a different step with the blocks allocated before this, and here just populate migrate.dst from the existing blocks.
Quoted 1 line
Quoted 1 line
[3]
https://patchwork.freedesktop.org/patch/588523/?series=132229&rev=1
Quoted 25 lines
Quoted 25 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +6
-0+if (ret) + goto kfree_buf; + + ret =migrate_vma_setup (&migrate); +if (ret) { + drm_err(&tile->xe->drm, "vma setup returned %d for range[%lx - %lx]\n",
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +10
-0+ ret, start, end); + goto free_dst_pages; + } + +/**FIXME: partial migration of a range print a warning for now. +* If this message is printed, we need to split xe_vma as we +* don't support a mixture placement of one vma +*/ +if (migrate.cpages != npages) + drm_warn(&tile->xe->drm, "Partial migration for range [%lx -%lx], range is %ld pages, migrate only %ld pages\n",
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 290
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c +1
-0+ start, end, npages, migrate.cpages);As discussed above, we shouldn't support this. We should fall back to smaller
xe_vmachunk size until we find one that works or simply leave the pages in sram and map those pages to GPU.Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0+ +/**Migrate page by page for now. +* Both source pages and destination pages can physically not contiguous,
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 294
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c +1
-0+* there is no good way to migrate multiple pages per blitter command.
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 295
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c +1
-0+*/ Touched on this a bunch throughout the series, lets do better than a page a time migration.
Algorithm should be very similar to what I discussed here [4] but with a few key differences.
Quoted 8 lines
Quoted 8 lines
- I think the sram pages can be unpopulated (page == NULL) if the user has not yet touched the page
- Also I think theMIGRATE_PFN_MIGRATEbit being clear is validIn these cases this indicate we have to issue a copy for the pages we are accumulating with contigous vram addresses.
[4]
https://patchwork.freedesktop.org/patch/588526/?series=132229&rev=1
Quoted 63 lines
Quoted 63 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0+for (i =0 ; i < npages; i++) { + src_page =migrate_pfn_to_page (migrate.src[i]); + if (unlikely(!src_page || !(migrate.src[i] &
MIGRATE_PFN_MIGRATE)))Discussed this in the CPU fault patch, once we call
migrate_vma_setup, on subsequent errors we need to callmigrate_vma_finalizeto revert the pages to the original state. At least I think if I am reading the doc after this correctly.Here on error we just free the pages...
Matt
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +4
-0+ goto free_dst_page; + +xe_assert (tile->xe, !is_zone_device_page (src_page)); + src_dma_addr[i] = dma_map_page(dev, src_page, 0,
PAGE_SIZE,DMA_TO_DEVICE);drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 304
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+if (unlikely (dma_mapping_error (dev, src_dma_addr[i]))) { + drm_warn(&tile->xe->drm, "dma map error for hostpfn %lx\n", migrate.src[i]);
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +7
-0+ goto free_dst_page; + } + dst_dpa =xe_mem_region_pfn_to_dpa (mr, migrate.dst[i]); + fence =xe_migrate_pa (tile->migrate, src_dma_addr[i], false, + dst_dpa, true,PAGE_SIZE ); +if (IS_ERR (fence)) { + drm_warn(&tile->xe->drm, "migrate host page(pfn: %lx) to vram failed\n",
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 313
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+ migrate.src[i]); +/**Migration is best effort. Even we failed here, we continue*/
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0+ goto free_dst_page; + } +/**FIXME: Use the first migration's out fence as the second migration's input fence,
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +11
-0+* and so on. Only wait the out fence of last migration? +*/ +dma_fence_wait (fence, false); +dma_fence_put (fence); +free_dst_page: +xe_devm_page_free (pfn_to_page (migrate.dst[i])); + } + +for (i =0 ; i < npages; i++) +if (!(dma_mapping_error (dev, src_dma_addr[i]))) + dma_unmap_page(dev, src_dma_addr[i], PAGE_SIZE,
DMA_TO_DEVICE);Quoted 13 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +10
-1+ +migrate_vma_pages (&migrate); +migrate_vma_finalize (&migrate); +free_dst_pages: +if (ret) +xe_devm_free_blocks (&blocks); +kfree_buf: +kfree (buf); +return ret; +} --2.26.3
Matthew Brost
maintainerRe: [v2 28/31] drm/xe/svm: Introduce helper to migrate vma to vram
Quoted 381 lines · Matthew Brost
Quoted 55 lines · Oak
Hi Matt,
Quoted 53 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Monday, April 15, 2024 3:40 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected]; Welty, Brian <[email protected]>
Subject: Re: [v2 28/31] drm/xe/svm: Introduce helper to migrate vma to vramQuoted 9 lines · Oak
Quoted 9 lines
-----Original Message-----
From: Brost, Matthew <[email protected]>
Sent: Wednesday, April 10, 2024 10:49 PM
To: Zeng, Oak <[email protected]>
Cc: [email protected]; Ghimiray, Himal Prasad <[email protected]>; Bommu, Krishnaiah <[email protected]>; [email protected];Welty,
Quoted 2 lines
Quoted 2 lines
Brian <[email protected]>
Subject: Re: [v2 28/31] drm/xe/svm: Introduce helper to migrate vma tovram
Quoted 12 lines
Quoted 12 lines · Oak Zeng
Quoted 10 lines
Introduce a helper function
xe_svm_migrate_vma_to_vram.Since the source pages of the svm range can be physically not contiguous, and the destination vram pages can also be not contiguous, there is no easy way to migrate multiple pages per blitter command. We do page by page migration for now.
Migration is best effort. Even if we fail to migrate some pages, we will try to migrate the rest pages.
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
FIXME: Use one blitter command to copy when both src and dst are physically contiguous
Yep, touch in this throughout the series. Only vram needs to be contiguous though as we dynamically create PT mappings for sram pages
in
Quoted 9 lines
Quoted 9 lines
the migrate code. Getting this in a must and should be done immediately IMO as this a very, very basic perform thing we know needs to be done. We will likely have to tune this code quite a bit for performance so getting known things done would be helpful.
Quoted 3 lines
FIXME: when a vma is partially migrated, split vma as we assume no mixture vma placement.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Agree we do not want support partial migrations. We likely want to return
-EAGAINfor something and fault back to a smallerxe_vmachunk size which I discussed in [1] and add comment on in [2].I thought into this when I worked on v3. I end up with keep the xe-vma not splitting, but migrate partially of a vma. A vma can have mixed backing store now... To support this, I had to make the page table update and invalidation range based (vs whole xe-vma based) I am testing it. Not very sure how well this goes.
A few things, I strongly oppose mixed storage within a single GPU mapping. I view this as an unnecessary complication.
A GPU mapping is not a VMA - it is a subset of a VMA. It has been agreed upon we have a 1:N relationship in SVM. With '1' being a GPU VMA which reflects user IOCTLs (e.g. initial large bind, madvise calls which can split the VMA). The 'N' being mappings dynamically created on upon GPU page fault. This a
xe_svm_range(subclass ofdrm_svm_range) in my PoC. My expectation is that any post has something similar to this agreed upon relationship implemented.Here my view is the 'N' never is mixed storage.
See my comments about 'page grainularity' migration in 'implement functions to allocate and free device memory'. If you are doing page grainularity migration, then 'N' is exactly 1 CPU page (could be THP) thus you never have a case where partial migrations or mixed storage is needed. Any spliting / mixed storage by definition is not page grainularity.
If we don't fully support page grainularity, I think the grainularity for any operation is allocation size of 'N'. e.g. Upon CPU page, CPU unmap, evict, 'N' is fully moved, invalidated, GPU unmapped. This simplifies the code quite a bit. See 'Migrataion', 'Partial Unmapping of Ranges', and 'Examples' in
drm_gpusvm.c. Of course open to others idea here but anything that is more complicated that what I'm suggesting, expect push back.Quoted 2 lines
One thing I realized that might now work well is the range based invalidation. Say we a pde entry covering a huge 2M or 1G page. If we only zap 4k in this rang, we will need to split pde entry and replace it with many pde/pte entries. Even though the zap_ptes/page table walk functions are range based, I doubt it works for such case. I will have to test it out. I think even if it doesn't work, it worth to make it work as an improvement area. For example, core mm obviously support such function, see
zap_pte_range.If we have GPU page tables larger than the CPU pages, then we are not doing 'page grainularity'.
This is a large source of confusion for as you seem to have conflicting ideas in reply to reply. Do plan on supporting 'page grainularity' or not? This is unclear to me.
If we don't support 'page grainularity', then see my above for how I expect an impedance mismatch between the CPU page size and 'N'. That applies here as well.
Quoted 164 lines
Quoted 155 lines
Quoted 6 lines
Quoted 6 lines
Migration should be best case too if we fail migrate we can always leave backing store in sram too.
Quoted 7 lines
Quoted 3 lines
I do have question though, when do we get partial migrations? A user having called mlock on some of the pages? I just want to make sure I fully understand that case.
Yah, mlock could be one case...
I also looked the hmm code. There are few other cases where
MIGRATE_PFN_MIGRATEis not set (so we skip migration after), such as pte is NULL and vma is file-backed (not anonymous); entry is swapped out to hard disk etc. see more details in functionmigrate_vma_collect_pmd.Quoted 4 lines
Quoted 4 lines
[1]
https://patchwork.freedesktop.org/patch/588526/?series=132229&rev=1
Quoted 1 line
Quoted 1 line
[2]
https://patchwork.freedesktop.org/patch/588528/?series=132229&rev=1
Quoted 12 lines
Quoted 12 lines
Quoted 3 lines
Signed-off-by Oak Zeng <[email protected]>Co-developed-by Niranjana VishwanathapuraQuoted 1 line
Signed-off-by Niranjana VishwanathapuraQuoted 6 lines
Cc Matthew Brost <[email protected]>Cc Thomas Hellström <[email protected]>Cc Brian Welty <[email protected]>--- drivers/gpu/drm/xe/xe_svm.h | 2 + drivers/gpu/drm/xe/xe_svm_migrate.c | 115Quoted 7 lines
Quoted 7 lines
++++++++++++++++++++++++++++
Same comment on file structure throughout the series apply here too.
Review
Quoted 3 lines
2 files changed, 117 insertions(+)(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm.hQuoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm.h int xe_devm_alloc_pages(struct xe_tile *tile, · line 86
Quoted 8 lines
index
c9e4239c44b4..18ce2e3757c5100644drivers/gpu/drm/xe/xe_svm.h +1
-0@@ -83,4 +83,6 @@ int xe_devm_alloc_pages(struct xe_tile *tile,void xe_devm_free_blocks (struct list_head *blocks);void xe_devm_page_free (struct page *page);vm_fault_t xe_svm_migrate_to_sram (struct vm_fault *vmf); +int xe_svm_migrate_vma_to_vram(struct xe_vm *vm, struct xe_vma*vma,
Quoted 9 lines
Quoted 9 lines
Review
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm.h +1
-0+ struct xe_tile *tile);#endif (quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_svm_migrate.cdrivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct
Quoted 4 lines
index
0db831af098e..ab8dd1f58aa4100644drivers/gpu/drm/xe/xe_svm_migrate.c +0
-0@@ -220,3 +220,118 @@ vm_fault_t xe_svm_migrate_to_sram(struct
vm_fault*vmf)Quoted 6 lines
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0kvfree (buf);return 0 ; } + +/** +* xe_svm_migrate_vma_to_vram() - migrate backing store of a vma to
Quoted 5 lines
Quoted 5 lines
vram
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +4
-0+* Must be called with mmap_read_lock held. +* @vm: the vm that the vma belongs to +* @vma: the vma to migrate. +* @tile: the destination tile which holds the new backing store of the range
Quoted 38 lines
Quoted 21 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +8
-0+* +* Returns: negative errno on faiure, 0 on success +*/ +int xe_svm_migrate_vma_to_vram (struct xe_vm *vm, +struct xe_vma *vma, +struct xe_tile *tile) +{ +struct mm_struct *mm = vm->mm;drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 247
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +10
-0+unsigned long start =xe_vma_start (vma); +unsigned long end =xe_vma_end (vma); +unsigned long npages = (end - start) >>PAGE_SHIFT ; +struct xe_mem_region *mr = &tile->mem.vram; +struct vm_area_struct *vas; + +struct migrate_vma migrate = { + .start = start, + .end = end, + .pgmap_owner = tile->xe,Again helper to assign owner.
Quoted 16 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 259
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +12
-0+ .flags = MIGRATE_VMA_SELECT_SYSTEM, + }; +struct device *dev = tile->xe->drm.dev; +dma_addr_t *src_dma_addr; +struct dma_fence *fence; +struct page *src_page; +LIST_HEAD (blocks); +int ret =0 , i; +u64 dst_dpa; +void *buf; + +mmap_assert_locked (mm);This
mmap_assert_lockedis ambiguous, we should make it clear if this read or write locked. Doesn't it have to be write to do the migrate pages?Quoted 1 line
I followed hmm document (
Documents/mm/hmm.rst), see section"Migration to and from device memory". It explicitly write a
read_lockin this document.Quoted 2 lines
I believe a
read_lockis enough for themigrate_vma_setup/migrate_vma_finalize().
Quoted 2 lines
As I understand it, the mm.mmap_lock protect the process's address space.
When we modify process's address space such as mmap/munmap, we need to hold a write mode lock; if we only read process's address space, such as in the migrate_vma_setup/finalize, or in the cpu page fault handler case, we only need a read mode lock.
Quoted 2 lines
I also cross checked amd driver. They also use a read lock.. see function
svm_range_restore_pagesinkfd_svm.c....Quoted 1 line
Yea, I see that too. Trying to figure out the locking, IMO the locking document might actually be wrong, or the very least the locking design is very ill-conceived. We can discuss internally a bit before I publically share my grievances.
Quoted 6 lines
Quoted 4 lines
A larger question about the locking... The CPU fault handler holds the mmap lock in write mode, right?
No. since cpu fault handler doesn't modify process address space, instead it
only fill up cpu page table for some valid address range, a read lock is enough.
Quoted 1 line
Ah, yes after digging around a bit I see this.
Quoted 7 lines
Quoted 5 lines
I'm asking as basically I think at least initially we want to hold the mmap lock in a way that the GPU handler and CPU handler do not race. i.e. From fault userptr create in GPU fault handler to issuing the bind we prevent the CPU fault handler from running.
Yes we hold
mmap_read_lockin both cpu and gpu fault handler to avoidthat race.
Quoted 1 line
Quoted 7 lines
That's not how rw lock work. 2 threads can both hold the read lock in parallel (shared read access), only 1 thread hold the write lock (exclusive write access, no one can hold read lock either). Thus my concern about the cpu and gpu fault handler running in parallel and the larger locking design questions. Again we can talk through this in detail internally a bit.
Quoted 4 lines
It is a great question of the gpu/cpu handler race. Obviously my above thinking is wrong. Here is my new understanding:
think about this scenario: address backing store is in vram - cpu access - cpu fault-
migrate_vma_setup- invalidate gpu pt - gpu access same address - gpu fault -migrate_vma_setuptriggered by gpu fault - this read back pfn of vram backing store because cpu fault is not finished/migrate_vma_finalized is not called yet, but since gpu side would only migrate memory in sram, somigrate_vma_setupfrom gpu side wouldn't return any valid pfn to migrate and gpu page fault handler would return successfully and gpu hw execution is resumed but immediately trigger another gpu page fault, until cpu side migration is done.This is hard to follow, especially given your comments drop wrap at 80 lines or so. Please use an email client which wraps. I don't think you reasoning is right.
I have worked this out and landed on HMM locking documentation is wrong or at least doesn't work unless a notifier is installed upon every GPU fault. Bascially the GPU handler takes mmap lock in write mode to work around this (a notifier installation takes mmap in write mode which is why I reasoned HMM lock doc suggestion works in some cases). Longterm will need some depth though about to fix this. It might be an extra MMU invalidation event upon
migrate_vma_finalizeor somethig... Again see my PoC, it has comments in the code explaining this.Quoted 5 lines
Similar when gpu fault happens during the other stage of cpu fault handler. Same thing when CPU fault handler interrupt a gpu fault handler.
This seems work to me but we need to test it out. We will add a test case for this race condition. @Bommu, Krishnaiah fyi.
I have a test for this, see race sections in [1].
[1] https://patchwork.freedesktop.org/series/133846/
Quoted 24 lines
Quoted 16 lines
Quoted 1 line
In user mmap/munmap (such as kernel function
vm_munmap), we hold a
mmap_write_lockwhich prevent it racing with cpu and gpu fault handler.Quoted 6 lines
Quoted 6 lines
I'm having issues figuring out how to prevent races between initial binds of userptrs and userptr invalidates on faulting VMs. This race is seen any
xe_exec_fault_modefor example... So preventing racesbetween
Quoted 7 lines
Quoted 3 lines
CPU / GPU fault handler with the mmap probably is a good idea initially. Likely can make the locking finer grained once this is all working and I figure out how to handle this race better.
I don't quite follow here.
Quoted 7 lines
Quoted 1 line
Initial bind of user ptr... if you meant the bind in gpu page fault handler,
then the racing with invalidation is roughly like below:
Quoted 1 line
Invalidation is called with
mmap_write_lockIs it? If the notifer does the invalidation via
migrate_vma_setupin the CPU fault handler we established about that only themmap_read_lockis held.Quoted 8 lines
Yes the cpu fault triggered migrate-vma-setup/invalidation only hold
read_lock.It was the munmap triggered invalidation in my mind. For this case, we hold
mmap_write_lockduring invalidation, so it is mutual exclusive with gpu triggered migration.But a cpu fault triggered invalidation can proceed during gpu fault triggered migration. As explained above, it cause extra gpu page table invalidation which is just fine because the consequence is just extra gpu page fault (and later migration). So functionally I don't see a problem of this scheme.
The main reason we hold a read lock in gpu fault handler is, gpu fault handler doesn't modify process address space, rather it only traverse the cpu page table.
Again not following, but quite sure your understand is not correct. See above.
Quoted 24 lines
Quoted 7 lines
Quoted 2 lines
In
userptr_pin_page, we hold ammap_read_lock, so we know during pinpage, invalidation is excluded.
Nope, see above invalidation can happen when
userptr_pin_pageis executing because of the read lock. The seqno check (described below) is what prevents programming of bad page tables.You are right.
Quoted 6 lines
Quoted 2 lines
After pin, before programming gpu page table, we check whether there is
invalidation happen *after last pin but before programming page table*, if that happened, we retry
Quoted 1 line
Quoted 6 lines
Yes, that is how it works on tip but I am refactoring it in [1]. I was trying to avoid the retry loop by turning PDE/PTE writes into clears if an invalidation happened but not sure if that works without a larger refactor due to nature PDEs being shared. I may need the retry loop but that also gets tricky with array of binds... A few options here and will land a on solution once [2] is merged.
Quoted 6 lines
I discussed this retry loop (I refer to the retry loop in xe-pt-userptr-pre-commit function before your change)with Thomas some time ago. I agree the retry loop scheme work
But to me other scheme without loop also work. As long as we have a retry check before programming gpu page table, it is fine to me.
If you have further fix in this area, I will keep watching and pick it.
I have this properly fixed now in [2]. Faulting VMs will return with
-EAGAINif the seqno check fails in a bind. If will be up to the page fault handler to retry. In the case of user binds, this so race I currently am just kicking the-EAGAINto userpace to retry. I could change the later if people disagre and retry in the kernel too.[2] https://patchwork.freedesktop.org/series/133034/.
Quoted 8 lines
Quoted 7 lines
Regardless my point here is still valid - it may not be the worst idea when getting this code initially up and running just to grab
mmap_write_lockin GPU fault handler as a BLK (big kernel lock) to prevent all races. Once the code is stable and stress testing in place, switch to finer grained locking as define in HMM document (or newly defined if we fine locking design is insufficient).
Quoted 5 lines
Quoted 2 lines
As explained, I will keep the read-lock for now. And we will test it. If we do run into race condition, I will try write-lock and come back.
See above, you will hit a race.
Hmm, I'm thinking now and some of the races might actually be avoidable 'page grainularity' migration... You'd have to play around with this but my initial thought is probably at least helps, if not out rights fixes the races.
Matt
Quoted 159 lines
I strongly suggest you checkout my working PoC and IGT and play around with that code. If you change from write lock to read lock in GPU handler around VRAM migration and hmm range fault, the 'race' sections fail.
Matt
Quoted 152 lines
Thanks a lot for the great questions Matt! It is very helpful!
Oak
Quoted 147 lines
Matt
[1] https://patchwork.freedesktop.org/series/125608/
[2] https://patchwork.freedesktop.org/series/132246/Quoted 10 lines
Oak
Quoted 6 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 261
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+ + vas =find_vma_intersection (mm, start, start +4 );
find_vmashould work fine here.Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +5
-0+if (!vas) +return -ENOENT ; + + migrate.vma = vas; + buf = kvcalloc(npages, 2* sizeof(*migrate.src) +
sizeof(*src_dma_addr),Quoted 12 lines
Quoted 12 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 273
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +7
-0+ GFP_KERNEL); +if (!buf) +return -ENOMEM ; + migrate.src = buf; + migrate.dst = migrate.src + npages; + src_dma_addr = (dma_addr_t *) (migrate.dst + npages); + ret =xe_devm_alloc_pages (tile, npages, &blocks, migrate.dst);Again as I discussed in [3] I think this should be broken out into a different step with the blocks allocated before this, and here just populate migrate.dst from the existing blocks.
Quoted 1 line
Quoted 1 line
[3]
https://patchwork.freedesktop.org/patch/588523/?series=132229&rev=1
Quoted 25 lines
Quoted 25 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +6
-0+if (ret) + goto kfree_buf; + + ret =migrate_vma_setup (&migrate); +if (ret) { + drm_err(&tile->xe->drm, "vma setup returned %d for range[%lx - %lx]\n",
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 10 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +10
-0+ ret, start, end); + goto free_dst_pages; + } + +/**FIXME: partial migration of a range print a warning for now. +* If this message is printed, we need to split xe_vma as we +* don't support a mixture placement of one vma +*/ +if (migrate.cpages != npages) + drm_warn(&tile->xe->drm, "Partial migration for range [%lx -%lx], range is %ld pages, migrate only %ld pages\n",
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf) · line 290
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c +1
-0+ start, end, npages, migrate.cpages);As discussed above, we shouldn't support this. We should fall back to smaller
xe_vmachunk size until we find one that works or simply leave the pages in sram and map those pages to GPU.Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0+ +/**Migrate page by page for now. +* Both source pages and destination pages can physically not contiguous,
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 294
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c +1
-0+* there is no good way to migrate multiple pages per blitter command.
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 295
Quoted 1 line
drivers/gpu/drm/xe/xe_svm_migrate.c +1
-0+*/ Touched on this a bunch throughout the series, lets do better than a page a time migration.
Algorithm should be very similar to what I discussed here [4] but with a few key differences.
Quoted 8 lines
Quoted 8 lines
- I think the sram pages can be unpopulated (page == NULL) if the user has not yet touched the page
- Also I think theMIGRATE_PFN_MIGRATEbit being clear is validIn these cases this indicate we have to issue a copy for the pages we are accumulating with contigous vram addresses.
[4]
https://patchwork.freedesktop.org/patch/588526/?series=132229&rev=1
Quoted 63 lines
Quoted 63 lines
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0+for (i =0 ; i < npages; i++) { + src_page =migrate_pfn_to_page (migrate.src[i]); + if (unlikely(!src_page || !(migrate.src[i] &
MIGRATE_PFN_MIGRATE)))Discussed this in the CPU fault patch, once we call
migrate_vma_setup, on subsequent errors we need to callmigrate_vma_finalizeto revert the pages to the original state. At least I think if I am reading the doc after this correctly.Here on error we just free the pages...
Matt
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 4 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +4
-0+ goto free_dst_page; + +xe_assert (tile->xe, !is_zone_device_page (src_page)); + src_dma_addr[i] = dma_map_page(dev, src_page, 0,
PAGE_SIZE,DMA_TO_DEVICE);drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 304
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+if (unlikely (dma_mapping_error (dev, src_dma_addr[i]))) { + drm_warn(&tile->xe->drm, "dma map error for hostpfn %lx\n", migrate.src[i]);
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 7 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +7
-0+ goto free_dst_page; + } + dst_dpa =xe_mem_region_pfn_to_dpa (mr, migrate.dst[i]); + fence =xe_migrate_pa (tile->migrate, src_dma_addr[i], false, + dst_dpa, true,PAGE_SIZE ); +if (IS_ERR (fence)) { + drm_warn(&tile->xe->drm, "migrate host page(pfn: %lx) to vram failed\n",
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct · line 313
Quoted 2 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +2
-0+ migrate.src[i]); +/**Migration is best effort. Even we failed here, we continue*/
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 3 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +3
-0+ goto free_dst_page; + } +/**FIXME: Use the first migration's out fence as the second migration's input fence,
drivers/gpu/drm/xe/xe_svm_migrate.c vm_fault_t xe_svm_migrate_to_sram(struct vm_fault *vmf)
Quoted 11 lines
drivers/gpu/drm/xe/xe_svm_migrate.c +11
-0+* and so on. Only wait the out fence of last migration? +*/ +dma_fence_wait (fence, false); +dma_fence_put (fence); +free_dst_page: +xe_devm_page_free (pfn_to_page (migrate.dst[i])); + } + +for (i =0 ; i < npages; i++) +if (!(dma_mapping_error (dev, src_dma_addr[i]))) + dma_unmap_page(dev, src_dma_addr[i], PAGE_SIZE,
DMA_TO_DEVICE);13 lines not shown · raw message
Part of this message is not shown here. View the raw message.