Introduce GPU SVM and Xe SVM implementation
Subsystems (9)
Matthew Brost
maintainerv4 cover Introduce GPU SVM and Xe SVM implementation
Version 4 of GPU SVM. Thanks to everyone (especially Sima, Thomas, Alistair, Himal) for their numerous reviews on revision 1, 2, 3 and for helping to address many design issues.
This version has been tested with IGT [1] on PVC, BMG, and LNL. Also tested with level0 (UMD) PR [2].
Major changes in v2:
- Dropped mmap write abuse
- core MM locking and retry loops instead of driver locking to avoid races
- Removed physical to virtual references
- Embedded structure/ops for drm_gpusvm_devmem
- Fixed mremap and fork issues
- Added DRM pagemap
- Included RFC documentation in the kernel doc
Major changes in v3:
- Move GPU SVM and DRM pagemap to DRM level
- Mostly addresses Thomas's feedback, lots of small changes documented in each individual patch change log
Major changes in v4:
- Pull documentation patch in
- Fix Kconfig / VRAM migration issue
- Address feedback which came out of internal multi-GPU implementation
Known issues in v4:
- Check pages still exists, changed to threshold in this version which is better but still need to root cause cross process page finding on small user allocations.
Matt
[1] https://patchwork.freedesktop.org/series/137545/#rev3
[2] https://github.com/intel/compute-runtime/pull/782
Matthew Brost (29):
drm/xe: Retry BO allocation
mm/migrate: Add migrate_device_pfns
mm/migrate: Trylock device page in do_swap_page drm/gpusvm: Add support for GPU Shared Virtual Memory drm/xe: Select DRM_GPUSVM Kconfig
drm/xe/uapi: Add DRM_XE_VM_BIND_FLAG_CPU_ADDR_MIRROR flag drm/xe: Add SVM init / close / fini to faulting VMs drm/xe: Nuke VM's mapping upon close
drm/xe: Add SVM range invalidation and page fault handler drm/gpuvm: Add DRM_GPUVA_OP_DRIVER
drm/xe: Add (re)bind to SVM page fault handler
drm/xe: Add SVM garbage collector
drm/xe: Add unbind to SVM garbage collector
drm/xe: Do not allow CPU address mirror VMA unbind if the GPU has
bindings
drm/xe: Enable CPU address mirror uAPI
drm/xe/uapi: Add DRM_XE_QUERY_CONFIG_FLAG_HAS_CPU_ADDR_MIRROR drm/xe: Add migrate layer functions for SVM support drm/xe: Add SVM device memory mirroring
drm/xe: Add drm_gpusvm_devmem to xe_bo
drm/xe: Add GPUSVM device memory copy vfunc functions drm/xe: Add Xe SVM populate_devmem_pfn GPU SVM vfunc drm/xe: Add Xe SVM devmem_release GPU SVM vfunc drm/xe: Add BO flags required for SVM
drm/xe: Add SVM VRAM migration
drm/xe: Basic SVM BO eviction
drm/xe: Add SVM debug
drm/xe: Add modparam for SVM notifier size
drm/xe: Add always_migrate_to_vram modparam
drm/doc: gpusvm: Add GPU SVM documentation
Thomas Hellström (4):
drm/pagemap: Add DRM pagemap
drm/xe/bo: Introduce xe_bo_put_async
drm/xe: Add dma_addr res cursor
drm/xe: Add drm_pagemap ops to SVM
Documentation/gpu/rfc/gpusvm.rst | 84 + Documentation/gpu/rfc/index.rst | 4 + drivers/gpu/drm/Kconfig | 9 + drivers/gpu/drm/Makefile | 1 + drivers/gpu/drm/drm_gpusvm.c | 2240 +++++++++++++++++++ drivers/gpu/drm/xe/Kconfig | 10 + drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_bo.c | 63 +- drivers/gpu/drm/xe/xe_bo.h | 14 + drivers/gpu/drm/xe/xe_bo_types.h | 4 + drivers/gpu/drm/xe/xe_device.c | 3 + drivers/gpu/drm/xe/xe_device_types.h | 22 + drivers/gpu/drm/xe/xe_gt_pagefault.c | 17 +- drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c | 24 + drivers/gpu/drm/xe/xe_gt_tlb_invalidation.h | 2 + drivers/gpu/drm/xe/xe_migrate.c | 175 ++ drivers/gpu/drm/xe/xe_migrate.h | 10 + drivers/gpu/drm/xe/xe_module.c | 7 + drivers/gpu/drm/xe/xe_module.h | 2 + drivers/gpu/drm/xe/xe_pt.c | 393 +++- drivers/gpu/drm/xe/xe_pt.h | 5 + drivers/gpu/drm/xe/xe_pt_types.h | 2 + drivers/gpu/drm/xe/xe_query.c | 5 +- drivers/gpu/drm/xe/xe_res_cursor.h | 116 +- drivers/gpu/drm/xe/xe_svm.c | 946 ++++++++ drivers/gpu/drm/xe/xe_svm.h | 84 + drivers/gpu/drm/xe/xe_tile.c | 5 + drivers/gpu/drm/xe/xe_vm.c | 375 +++- drivers/gpu/drm/xe/xe_vm.h | 15 +- drivers/gpu/drm/xe/xe_vm_types.h | 57 + include/drm/drm_gpusvm.h | 445 ++++ include/drm/drm_gpuvm.h | 5 + include/drm/drm_pagemap.h | 105 + include/linux/migrate.h | 1 + include/uapi/drm/xe_drm.h | 22 +- mm/memory.c | 13 +- mm/migrate_device.c | 116 +- 37 files changed, 5245 insertions(+), 157 deletions(-) create mode 100644 Documentation/gpu/rfc/gpusvm.rst create mode 100644 drivers/gpu/drm/drm_gpusvm.c create mode 100644 drivers/gpu/drm/xe/xe_svm.c create mode 100644 drivers/gpu/drm/xe/xe_svm.h create mode 100644 include/drm/drm_gpusvm.h create mode 100644 include/drm/drm_pagemap.h
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v4 1/33 drm/xe: Retry BO allocation
TTM doesn't support fair eviction via WW locking, this mitigated in by using retry loops in exec and preempt rebind worker. Extend this retry loop to BO allocation. Once TTM supports fair eviction this patch can be reverted.
v4:
- Keep line break (Stuart)
--- drivers/gpu/drm/xe/xe_bo.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +7 -0
@@ -2162,6 +2162,7 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,struct xe_file *xef =to_xe_file (file);struct drm_xe_gem_create *args = data;struct xe_vm *vm =NULL ; +ktime_t end =0 ;struct xe_bo *bo;unsigned int bo_flags;u32 handle;
@@ -2234,6 +2235,10 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data, vm =xe_vm_lookup (xef, args->vm_id);if (XE_IOCTL_DBG (xe, !vm))return -ENOENT ; + } + +retry: +if (vm) { err =xe_vm_lock (vm, true);if (err) goto out_vm;
@@ -2247,6 +2252,8 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,if (IS_ERR (bo)) { err =PTR_ERR (bo); +if (xe_vm_validate_should_retry (NULL , err, &end)) + goto retry; goto out_vm; }
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v4 2/33 mm/migrate: Add migrate_device_pfns
In reply to Matthew BrostAdd migrate_device_pfns which prepares an array of pre-populated device pages for migration. This is needed for eviction of known set of non-contiguous devices pages to cpu pages which is a common case for SVM in DRM drivers using TTM.
v2:
- s/migrate_device_vma_range/migrate_device_prepopulated_range
- Drop extra mmu invalidation (Vetter)
v3:
- s/migrate_device_prepopulated_range/migrate_device_pfns (Alistar)
- Use helper to lock device pages (Alistar)
- Update commit message with why this is required (Alistar)
--- include/linux/migrate.h | 1 + mm/migrate_device.c | 52 +++++++++++++++++++++++++++++------------ 2 files changed, 38 insertions(+), 15 deletions(-)
include/linux/migrate.h +1 -0
@@ -229,6 +229,7 @@ void migrate_vma_pages(struct migrate_vma *migrate);void migrate_vma_finalize (struct migrate_vma *migrate);int migrate_device_range (unsigned long *src_pfns,unsigned long start,unsigned long npages); +int migrate_device_pfns (unsigned long *src_pfns,unsigned long npages);void migrate_device_pages (unsigned long *src_pfns,unsigned long *dst_pfns,unsigned long npages); void migrate_device_finalize(unsigned long *src_pfns,
mm/migrate_device.c +37 -15
72 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v4 02/33] mm/migrate: Add migrate_device_pfns
Quoted 5 lines · Matthew Brost
Add
migrate_device_pfnswhich prepares an array of pre-populated device pages for migration. This is needed for eviction of known set of non-contiguous devices pages to cpu pages which is a common case for SVM in DRM drivers using TTM.
Quoted 7 lines
v2:
- s/migrate_device_vma_range/migrate_device_prepopulated_range
- Drop extra mmu invalidation (Vetter)
v3:
- s/migrate_device_prepopulated_range/migrate_device_pfns (Alistar)
- Use helper to lock device pages (Alistar)
- Update commit message with why this is required (Alistar)
Thanks! Looks good to me now so:
Quoted 100 lines
Cc Andrew Morton <[email protected]>Signed-off-by Matthew Brost <[email protected]>--- include/linux/migrate.h | 1 + mm/migrate_device.c | 52 +++++++++++++++++++++++++++++------------ 2 files changed, 38 insertions(+), 15 deletions(-)include/linux/migrate.h +1
-0@@ -229,6 +229,7 @@ void migrate_vma_pages(struct migrate_vma *migrate);void migrate_vma_finalize (struct migrate_vma *migrate);int migrate_device_range (unsigned long *src_pfns,unsigned long start,unsigned long npages); +int migrate_device_pfns (unsigned long *src_pfns,unsigned long npages);void migrate_device_pages (unsigned long *src_pfns,unsigned long *dst_pfns,unsigned long npages); void migrate_device_finalize(unsigned long *src_pfns,mm/migrate_device.c +37
-15@@ -876,6 +876,22 @@ void migrate_vma_finalize(struct migrate_vma *migrate) }EXPORT_SYMBOL (migrate_vma_finalize); +static unsigned long migrate_device_pfn_lock (unsigned long pfn) +{ +struct folio *folio; + + folio =folio_get_nontail_page (pfn_to_page (pfn)); +if (!folio) +return 0 ; + +if (!folio_trylock (folio)) { +folio_put (folio); +return 0 ; + } + +return migrate_pfn (pfn) |MIGRATE_PFN_MIGRATE ; +} +/** * migrate_device_range() - migrate device private pfns to normal memory. * @src_pfns: array large enough to hold migrating source device private pfns. @@ -900,29 +916,35 @@ int migrate_device_range(unsigned long *src_pfns, unsigned long start, {unsigned long i, pfn; -for (pfn = start, i =0 ; i < npages; pfn++, i++) { -struct folio *folio; +for (pfn = start, i =0 ; i < npages; pfn++, i++) + src_pfns[i] =migrate_device_pfn_lock (pfn); - folio =folio_get_nontail_page (pfn_to_page (pfn)); -if (!folio) { - src_pfns[i] =0 ; -continue ; - } +migrate_device_unmap (src_pfns, npages,NULL ); -if (!folio_trylock (folio)) { - src_pfns[i] =0 ; -folio_put (folio); -continue ; - } +return 0 ; +} +EXPORT_SYMBOL (migrate_device_range); - src_pfns[i] =migrate_pfn (pfn) |MIGRATE_PFN_MIGRATE ; - } +/** +* migrate_device_pfns() - migrate device private pfns to normal memory. +* @src_pfns: pre-popluated array of source device private pfns to migrate. +* @npages: number of pages to migrate. +* +* Similar to migrate_device_range() but supports non-contiguous pre-popluated +* array of device pages to migrate. +*/ +int migrate_device_pfns (unsigned long *src_pfns,unsigned long npages) +{ +unsigned long i; + +for (i =0 ; i < npages; i++) + src_pfns[i] =migrate_device_pfn_lock (src_pfns[i]);migrate_device_unmap (src_pfns, npages,NULL );return 0 ; } -EXPORT_SYMBOL (migrate_device_range); +EXPORT_SYMBOL (migrate_device_pfns);/* * Migrate a device coherent folio back to normal memory. The caller should have Signature · 1 line
2.34.1
Re: [PATCH v4 02/33] mm/migrate: Add migrate_device_pfns
In reply to Matthew BrostQuoted 100 lines · Matthew Brost
Add
migrate_device_pfnswhich prepares an array of pre-populated device pages for migration. This is needed for eviction of known set of non-contiguous devices pages to cpu pages which is a common case for SVM in DRM drivers using TTM.v2:
- s/migrate_device_vma_range/migrate_device_prepopulated_range
- Drop extra mmu invalidation (Vetter)
v3:
- s/migrate_device_prepopulated_range/migrate_device_pfns (Alistar)
- Use helper to lock device pages (Alistar)
- Update commit message with why this is required (Alistar)Cc Andrew Morton <[email protected]>Signed-off-by Matthew Brost <[email protected]>--- include/linux/migrate.h | 1 + mm/migrate_device.c | 52 +++++++++++++++++++++++++++++------------ 2 files changed, 38 insertions(+), 15 deletions(-)include/linux/migrate.h +1
-0@@ -229,6 +229,7 @@ void migrate_vma_pages(struct migrate_vma *migrate);void migrate_vma_finalize (struct migrate_vma *migrate);int migrate_device_range (unsigned long *src_pfns,unsigned long start,unsigned long npages); +int migrate_device_pfns (unsigned long *src_pfns,unsigned long npages);void migrate_device_pages (unsigned long *src_pfns,unsigned long *dst_pfns,unsigned long npages); void migrate_device_finalize(unsigned long *src_pfns,mm/migrate_device.c +36
-14@@ -876,6 +876,22 @@ void migrate_vma_finalize(struct migrate_vma *migrate) }EXPORT_SYMBOL (migrate_vma_finalize); +static unsigned long migrate_device_pfn_lock (unsigned long pfn) +{ +struct folio *folio; + + folio =folio_get_nontail_page (pfn_to_page (pfn)); +if (!folio) +return 0 ; + +if (!folio_trylock (folio)) { +folio_put (folio); +return 0 ; + } + +return migrate_pfn (pfn) |MIGRATE_PFN_MIGRATE ; +} +/** * migrate_device_range() - migrate device private pfns to normal memory. * @src_pfns: array large enough to hold migrating source device private pfns. @@ -900,29 +916,35 @@ int migrate_device_range(unsigned long *src_pfns, unsigned long start, {unsigned long i, pfn; -for (pfn = start, i =0 ; i < npages; pfn++, i++) { -struct folio *folio; +for (pfn = start, i =0 ; i < npages; pfn++, i++) + src_pfns[i] =migrate_device_pfn_lock (pfn); - folio =folio_get_nontail_page (pfn_to_page (pfn)); -if (!folio) { - src_pfns[i] =0 ; -continue ; - } +migrate_device_unmap (src_pfns, npages,NULL ); -if (!folio_trylock (folio)) { - src_pfns[i] =0 ; -folio_put (folio); -continue ; - } +return 0 ; +} +EXPORT_SYMBOL (migrate_device_range); - src_pfns[i] =migrate_pfn (pfn) |MIGRATE_PFN_MIGRATE ; - } +/** +* migrate_device_pfns() - migrate device private pfns to normal memory. +* @src_pfns: pre-popluated array of source device private pfns to migrate. +* @npages: number of pages to migrate. +* +* Similar to migrate_device_range() but supports non-contiguous pre-popluated +* array of device pages to migrate. +*/ +int migrate_device_pfns (unsigned long *src_pfns,unsigned long npages) +{ +unsigned long i; + +for (i =0 ; i < npages; i++) + src_pfns[i] =migrate_device_pfn_lock (src_pfns[i]);
mm/migrate_device.c int migrate_device_range(unsigned long *src_pfns, unsigned long start, · line 950
Quoted 10 lines
mm/migrate_device.c +1
-1migrate_device_unmap (src_pfns, npages,NULL );return 0 ; } -EXPORT_SYMBOL (migrate_device_range); +EXPORT_SYMBOL (migrate_device_pfns);/* * Migrate a device coherent folio back to normal memory. The caller should have
Looks good to me and I have confirmed that a code flow has been added that calls this function from the actual driver (xe), which did not exist in the previous v3 patch series.
(This code flow called when ttm performs eviction when ttm needs)
There seems to be no test scenario for testing device memory pressure in the test cases mentioned in the cover letter. Do you plan to add this scenario as well? ( Please correct me if I misunderstood.)
Matthew Brost
maintainerRe: [PATCH v4 02/33] mm/migrate: Add migrate_device_pfns
Quoted 110 lines · Gwan-gyeong Mun
Quoted 92 lines · Matthew Brost
Add
migrate_device_pfnswhich prepares an array of pre-populated device pages for migration. This is needed for eviction of known set of non-contiguous devices pages to cpu pages which is a common case for SVM in DRM drivers using TTM.v2:
- s/migrate_device_vma_range/migrate_device_prepopulated_range
- Drop extra mmu invalidation (Vetter)
v3:
- s/migrate_device_prepopulated_range/migrate_device_pfns (Alistar)
- Use helper to lock device pages (Alistar)
- Update commit message with why this is required (Alistar)Cc Andrew Morton <[email protected]>Signed-off-by Matthew Brost <[email protected]>--- include/linux/migrate.h | 1 + mm/migrate_device.c | 52 +++++++++++++++++++++++++++++------------ 2 files changed, 38 insertions(+), 15 deletions(-)include/linux/migrate.h +1
-0@@ -229,6 +229,7 @@ void migrate_vma_pages(struct migrate_vma *migrate);void migrate_vma_finalize (struct migrate_vma *migrate);int migrate_device_range (unsigned long *src_pfns,unsigned long start,unsigned long npages); +int migrate_device_pfns (unsigned long *src_pfns,unsigned long npages);void migrate_device_pages (unsigned long *src_pfns,unsigned long *dst_pfns,unsigned long npages); void migrate_device_finalize(unsigned long *src_pfns,mm/migrate_device.c +33
-14@@ -876,6 +876,22 @@ void migrate_vma_finalize(struct migrate_vma *migrate) }EXPORT_SYMBOL (migrate_vma_finalize); +static unsigned long migrate_device_pfn_lock (unsigned long pfn) +{ +struct folio *folio; + + folio =folio_get_nontail_page (pfn_to_page (pfn)); +if (!folio) +return 0 ; + +if (!folio_trylock (folio)) { +folio_put (folio); +return 0 ; + } + +return migrate_pfn (pfn) |MIGRATE_PFN_MIGRATE ; +} +/** * migrate_device_range() - migrate device private pfns to normal memory. * @src_pfns: array large enough to hold migrating source device private pfns. @@ -900,29 +916,35 @@ int migrate_device_range(unsigned long *src_pfns, unsigned long start, {unsigned long i, pfn; -for (pfn = start, i =0 ; i < npages; pfn++, i++) { -struct folio *folio; +for (pfn = start, i =0 ; i < npages; pfn++, i++) + src_pfns[i] =migrate_device_pfn_lock (pfn); - folio =folio_get_nontail_page (pfn_to_page (pfn)); -if (!folio) { - src_pfns[i] =0 ; -continue ; - } +migrate_device_unmap (src_pfns, npages,NULL ); -if (!folio_trylock (folio)) { - src_pfns[i] =0 ; -folio_put (folio); -continue ; - } +return 0 ; +} +EXPORT_SYMBOL (migrate_device_range); - src_pfns[i] =migrate_pfn (pfn) |MIGRATE_PFN_MIGRATE ; - } +/** +* migrate_device_pfns() - migrate device private pfns to normal memory. +* @src_pfns: pre-popluated array of source device private pfns to migrate. +* @npages: number of pages to migrate. +* +* Similar to migrate_device_range() but supports non-contiguous pre-popluated +* array of device pages to migrate. +*/ +int migrate_device_pfns (unsigned long *src_pfns,unsigned long npages) +{ +unsigned long i;mm/migrate_device.c int migrate_device_range(unsigned long *src_pfns, unsigned long start, · line 943
Quoted 10 lines
mm/migrate_device.c +4
-1+ +for (i =0 ; i < npages; i++) + src_pfns[i] =migrate_device_pfn_lock (src_pfns[i]);migrate_device_unmap (src_pfns, npages,NULL );return 0 ; } -EXPORT_SYMBOL (migrate_device_range); +EXPORT_SYMBOL (migrate_device_pfns);/* * Migrate a device coherent folio back to normal memory. The caller should have Looks good to me and I have confirmed that a code flow has been added that calls this function from the actual driver (xe),
which did not exist in the previous v3 patch series.
(This code flow called when ttm performs eviction when ttm needs)
Quoted 5 lines
There seems to be no test scenario for testing device memory pressure in the test cases mentioned in the cover letter.
Do you plan to add this scenario as well? ( Please correct me if I misunderstood.)
The *evict sections in [1] to test device memory pressure.
Matt
[1] https://patchwork.freedesktop.org/series/137545/#rev3
Quoted 2 lines
Reviewed-by Gwan-gyeong Mun <[email protected]>
Matthew Brost
maintainerPATCH v4 3/33 mm/migrate: Trylock device page in do_swap_page
In reply to Matthew BrostAvoid multiple CPU page faults to the same device page racing by trying to lock the page in do_swap_page before taking an extra reference to the page. This prevents scenarios where multiple CPU page faults each take an extra reference to a device page, which could abort migration in folio_migrate_mapping. With the device page being locked in do_swap_page, the migrate_vma_* functions need to be updated to avoid locking the fault_page argument.
Prior to this change, a livelock scenario could occur in Xe's (Intel GPU DRM driver) SVM implementation if enough threads faulted the same device page.
v3:
- Put page after unlocking page (Alistair)
- Warn on spliting a TPH which is fault page (Alistair)
- Warn on dst page == fault page (Alistair)
--- mm/memory.c | 13 ++++++--- mm/migrate_device.c | 64 ++++++++++++++++++++++++++++++++------------- 2 files changed, 55 insertions(+), 22 deletions(-)
mm/memory.c +9 -4
@@ -4267,10 +4267,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* Get a page reference while we know the page can't be * freed. */ -get_page (vmf->page); -pte_unmap_unlock (vmf->pte, vmf->ptl); - ret = vmf->page->pgmap->ops->migrate_to_ram (vmf); -put_page (vmf->page); +if (trylock_page (vmf->page)) { +get_page (vmf->page); +pte_unmap_unlock (vmf->pte, vmf->ptl); + ret = vmf->page->pgmap->ops->migrate_to_ram (vmf); +unlock_page (vmf->page); +put_page (vmf->page); + }else { +pte_unmap_unlock (vmf->pte, vmf->ptl); + } }else if (is_hwpoison_entry (entry)) { ret =VM_FAULT_HWPOISON ; }else if (is_pte_marker_entry (entry)) {
mm/migrate_device.c +46 -18
153 lines not on this page · raw message
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v4 4/33 drm/pagemap: Add DRM pagemap
In reply to Matthew BrostFrom: Thomas Hellström <[email protected]>
Introduce drm_pagemap ops to map and unmap dma to VRAM resources. In the local memory case it's a matter of merely providing an offset into the device's physical address. For future p2p the map and unmap functions may encode as needed.
Similar to how dma-buf works, let the memory provider (drm_pagemap) provide the mapping functionality.
v3:
- Move to drm level include
v4:
- Fix kernel doc (G.G.)
--- include/drm/drm_pagemap.h | 105 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 include/drm/drm_pagemap.h
include/drm/drm_pagemap.h new file +105 -0
@@ -0,0 +1,105 @@ +/* SPDX-License-Identifier: MIT */ +#ifndef _DRM_PAGEMAP_H_ +#define _DRM_PAGEMAP_H_ + +#include <linux/dma-direction.h> +#include <linux/hmm.h> +#include <linux/types.h> + +struct drm_pagemap ; +struct device ; + +/** +* enum drm_interconnect_protocol - Used to identify an interconnect protocol. +*/ +enum drm_interconnect_protocol { +DRM_INTERCONNECT_SYSTEM ,/* DMA map is system pages. */ +DRM_INTERCONNECT_PCIE_P2P ,/* DMA map is PCIE P2P */ +DRM_INTERCONNECT_DRIVER ,/* DMA map is driver defined */ +/* A driver can add private values beyond DRM_INTERCONNECT_DRIVER */ +}; + +/** +* struct drm_pagemap_dma_addr - DMA address representation. +* @addr: The dma address or driver-defined address for driver private interconnects. +* @proto: The interconnect protocol. +* @order: The page order of the dma mapping. (Size is PAGE_SIZE << order). +* @dir: The DMA direction. +* +* Note: There is room for improvement here. We should be able to pack into +* 64 bits. +*/ +struct drm_pagemap_dma_addr { +dma_addr_t addr; +u64 proto :54 ; +u64 order :8 ; +u64 dir :2 ; +}; + +/** +* drm_pagemap_dma_addr_encode() - Encode a dma address with metadata +* @addr: The dma address or driver-defined address for driver private interconnects. +* @proto: The interconnect protocol. +* @order: The page order of the dma mapping. (Size is PAGE_SIZE << order). +* @dir: The DMA direction. +* +* Return: A struct drm_pagemap_dma_addr encoding the above information. +*/ +static inline struct drm_pagemap_dma_addr +drm_pagemap_dma_addr_encode (dma_addr_t addr, +enum drm_interconnect_protocol proto, +unsigned int order, +enum dma_data_direction dir) +{ +return (struct drm_pagemap_dma_addr ) { + .addr = addr, + .proto = proto, + .order = order, + .dir = dir, + }; +} + +/** +* struct drm_pagemap_ops: Ops for a drm-pagemap. +*/ +struct drm_pagemap_ops { +/** +* @map_dma: Map for dma access or provide a virtual address suitable for +* +* @dpagemap: The struct drm_pagemap for the page. +* @dev: The dma mapper. +* @page: The page to map. +* @order: The page order of the dma mapping. (Size is PAGE_SIZE << order). +* @dir: The transfer direction. +*/ +struct drm_pagemap_dma_addr (*map_dma)(struct drm_pagemap *dpagemap, +struct device *dev, +struct page *page, +unsigned int order, +enum dma_data_direction dir); + +/** +* @unmap_dma: Unmap a dma address previously obtained using @map_dma. +* +* @dpagemap: The struct drm_pagemap for the mapping. +* @dev: The dma unmapper. +* @addr: The dma address obtained when mapping. +*/ +void (*unmap_dma)(struct drm_pagemap *dpagemap, +struct device *dev, +struct drm_pagemap_dma_addr addr); + +}; + +/** +* struct drm_pagemap: Additional information for a struct dev_pagemap +* used for device p2p handshaking. +* @ops: The struct drm_pagemap_ops. +* @dev: The struct drevice owning the device-private memory. +*/ +struct drm_pagemap { +const struct drm_pagemap_ops *ops; +struct device *dev; +}; + +#endif
Signature · 1 line
2.34.1
Thomas Hellström
maintainerRe: [PATCH v4 04/33] drm/pagemap: Add DRM pagemap
Quoted 10 lines · Matthew Brost
From: Thomas Hellström <[email protected]>
Introduce
drm_pagemapops to map and unmap dma to VRAM resources. In the
local memory case it's a matter of merely providing an offset into the
device's physical address. For future p2p the map and unmap functions may
encode as needed.
Quoted 3 lines
Similar to how dma-buf works, let the memory provider (
drm_pagemap) provide
the mapping functionality.
It should be noted that the long term idea for dma mapping is to have that done by the client instead of by the memory provider, which Jason reminded me of in a discussion on dri-devel. The dma-mapping here is modeled after how it's done for dma-buf, where the exporter maps dma.
So following that, it might be that we should move these dma-mapping ops to the drm_gpusvm().
The situation I can think of, where this might be a problem is that if the device-private struct page to dma address mapping is not known to the client.
/Thomas
Quoted 140 lines
v3:
- Move to drm level include
v4:
- Fix kernel doc (G.G.)Signed-off-by Matthew Brost <[email protected]>Signed-off-by Thomas Hellström <[email protected]>---
include/drm/drm_pagemap.h | 105
++++++++++++++++++++++++++++++++++++++1 file changed, 105 insertions(+)create mode 100644
include/drm/drm_pagemap.hinclude/drm/drm_pagemap.h new file +13
-0@@ -0,0 +1,105 @@ +/* SPDX-License-Identifier: MIT */ +#ifndef _DRM_PAGEMAP_H_ +#define _DRM_PAGEMAP_H_ + +#include <linux/dma-direction.h> +#include <linux/hmm.h> +#include <linux/types.h> + +struct drm_pagemap ; +struct device ; + +/** +* enum drm_interconnect_protocol - Used to identify an interconnect protocol.
+ */include/drm/drm_pagemap.h new file +5
-0+enum drm_interconnect_protocol { + DRM_INTERCONNECT_SYSTEM, /* DMA map is system pages. */ + DRM_INTERCONNECT_PCIE_P2P, /* DMA map is PCIE P2P */ + DRM_INTERCONNECT_DRIVER, /* DMA map is driver defined */ + /* A driver can add private values beyond
DRM_INTERCONNECT_DRIVER*/include/drm/drm_pagemap.h new file +5
-0+}; + +/** +* struct drm_pagemap_dma_addr - DMA address representation. +* @addr: The dma address or driver-defined address for driver private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size isPAGE_SIZE<< order).
+ * @dir: The DMA direction.
+ *
+ * Note: There is room for improvement here. We should be able to pack into
+ * 64 bits.
+ */include/drm/drm_pagemap.h new file +9
-0+struct drm_pagemap_dma_addr { +dma_addr_t addr; +u64 proto :54 ; +u64 order :8 ; +u64 dir :2 ; +}; + +/** +* drm_pagemap_dma_addr_encode() - Encode a dma address with metadata
+ * @addr: The dma address or driver-defined address for driver private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size isPAGE_SIZE<< order).
+ * @dir: The DMA direction.
+ *
+ * Return: A structdrm_pagemap_dma_addrencoding the above information.
+ */include/drm/drm_pagemap.h new file +20
-0+static inline struct drm_pagemap_dma_addr +drm_pagemap_dma_addr_encode (dma_addr_t addr, + enum drm_interconnect_protocol proto, + unsigned int order, + enum dma_data_direction dir) +{ +return (struct drm_pagemap_dma_addr ) { + .addr = addr, + .proto = proto, + .order = order, + .dir = dir, + }; +} + +/** +* struct drm_pagemap_ops: Ops for a drm-pagemap. +*/ +struct drm_pagemap_ops { +/** +* @map_dma: Map for dma access or provide a virtual address suitable for
include/drm/drm_pagemap.h new file +5
-0+* +* @dpagemap: The struct drm_pagemap for the page. +* @dev: The dma mapper. +* @page: The page to map. +* @order: The page order of the dma mapping. (Size is
PAGE_SIZE<< order).include/drm/drm_pagemap.h new file +3
-0+* @dir: The transfer direction. +*/ + struct drm_pagemap_dma_addr (*map_dma)(struct drm_pagemap*dpagemap,
include/drm/drm_pagemap.h new file +4
-0+ struct device *dev, + struct page *page, + unsigned int order, + enum
dma_data_directiondir);include/drm/drm_pagemap.h new file +3
-0+ +/** +* @unmap_dma: Unmap a dma address previously obtained using @map_dma.
include/drm/drm_pagemap.h new file +13
-0+* +* @dpagemap: The struct drm_pagemap for the mapping. +* @dev: The dma unmapper. +* @addr: The dma address obtained when mapping. +*/ +void (*unmap_dma)(struct drm_pagemap *dpagemap, + struct device *dev, + struct drm_pagemap_dma_addr addr); + +}; + +/** +* struct drm_pagemap: Additional information for a struct
dev_pagemap
+ * used for device p2p handshaking.
+ * @ops: The structdrm_pagemap_ops.
+ * @dev: The struct drevice owning the device-private memory.
+ */include/drm/drm_pagemap.h new file +6
-0+struct drm_pagemap { +const struct drm_pagemap_ops *ops; +struct device *dev; +}; + +#endif
Matthew Brost
maintainerRe: [PATCH v4 04/33] drm/pagemap: Add DRM pagemap
Quoted 11 lines · Thomas Hellström
Quoted 10 lines · Matthew Brost
From: Thomas Hellström <[email protected]>
Introduce
drm_pagemapops to map and unmap dma to VRAM resources. In the
local memory case it's a matter of merely providing an offset into the
device's physical address. For future p2p the map and unmap functions may
encode as needed.
Quoted 4 lines
Quoted 3 lines
Similar to how dma-buf works, let the memory provider (
drm_pagemap) provide
the mapping functionality.
Trying to parse all of this.
Quoted 2 lines
It should be noted that the long term idea for dma mapping is to have that done by the client instead of by the memory provider, which Jason
- Client here is the device mapping the memory.
- Memory provider is the device where the memory is located?
Did I get this correct?
Quoted 6 lines
reminded me of in a discussion on dri-devel. The dma-mapping here is modeled after how it's done for dma-buf, where the exporter maps dma.
So following that, it might be that we should move these dma-mapping ops to the
drm_gpusvm().
So we move ops to the local client (gpusvm) rather than remote device, right?
Quoted 4 lines
The situation I can think of, where this might be a problem is that if the device-private struct page to dma address mapping is not known to the client.
I'm not following this but I agree if dma mapping at the client we need the remote device structure given how the dma mapping API works.
So to wrap it up - what, if anything, do you think we need to do to this individual patch as part of this series?
Matt
Quoted 147 lines
/Thomas
Quoted 140 lines
v3:
- Move to drm level include
v4:
- Fix kernel doc (G.G.)Signed-off-by Matthew Brost <[email protected]>Signed-off-by Thomas Hellström <[email protected]>---
include/drm/drm_pagemap.h | 105
++++++++++++++++++++++++++++++++++++++1 file changed, 105 insertions(+)create mode 100644
include/drm/drm_pagemap.hinclude/drm/drm_pagemap.h new file +13
-0@@ -0,0 +1,105 @@ +/* SPDX-License-Identifier: MIT */ +#ifndef _DRM_PAGEMAP_H_ +#define _DRM_PAGEMAP_H_ + +#include <linux/dma-direction.h> +#include <linux/hmm.h> +#include <linux/types.h> + +struct drm_pagemap ; +struct device ; + +/** +* enum drm_interconnect_protocol - Used to identify an interconnect protocol.
+ */include/drm/drm_pagemap.h new file +5
-0+enum drm_interconnect_protocol { + DRM_INTERCONNECT_SYSTEM, /* DMA map is system pages. */ + DRM_INTERCONNECT_PCIE_P2P, /* DMA map is PCIE P2P */ + DRM_INTERCONNECT_DRIVER, /* DMA map is driver defined */ + /* A driver can add private values beyond
DRM_INTERCONNECT_DRIVER*/include/drm/drm_pagemap.h new file +5
-0+}; + +/** +* struct drm_pagemap_dma_addr - DMA address representation. +* @addr: The dma address or driver-defined address for driver private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size isPAGE_SIZE<< order).
+ * @dir: The DMA direction.
+ *
+ * Note: There is room for improvement here. We should be able to pack into
+ * 64 bits.
+ */include/drm/drm_pagemap.h new file +9
-0+struct drm_pagemap_dma_addr { +dma_addr_t addr; +u64 proto :54 ; +u64 order :8 ; +u64 dir :2 ; +}; + +/** +* drm_pagemap_dma_addr_encode() - Encode a dma address with metadata
+ * @addr: The dma address or driver-defined address for driver private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size isPAGE_SIZE<< order).
+ * @dir: The DMA direction.
+ *
+ * Return: A structdrm_pagemap_dma_addrencoding the above information.
+ */include/drm/drm_pagemap.h new file +20
-0+static inline struct drm_pagemap_dma_addr +drm_pagemap_dma_addr_encode (dma_addr_t addr, + enum drm_interconnect_protocol proto, + unsigned int order, + enum dma_data_direction dir) +{ +return (struct drm_pagemap_dma_addr ) { + .addr = addr, + .proto = proto, + .order = order, + .dir = dir, + }; +} + +/** +* struct drm_pagemap_ops: Ops for a drm-pagemap. +*/ +struct drm_pagemap_ops { +/** +* @map_dma: Map for dma access or provide a virtual address suitable for
include/drm/drm_pagemap.h new file +5
-0+* +* @dpagemap: The struct drm_pagemap for the page. +* @dev: The dma mapper. +* @page: The page to map. +* @order: The page order of the dma mapping. (Size is
PAGE_SIZE<< order).include/drm/drm_pagemap.h new file +3
-0+* @dir: The transfer direction. +*/ + struct drm_pagemap_dma_addr (*map_dma)(struct drm_pagemap*dpagemap,
include/drm/drm_pagemap.h new file +4
-0+ struct device *dev, + struct page *page, + unsigned int order, + enum
dma_data_directiondir);include/drm/drm_pagemap.h new file +3
-0+ +/** +* @unmap_dma: Unmap a dma address previously obtained using @map_dma.
include/drm/drm_pagemap.h new file +13
-0+* +* @dpagemap: The struct drm_pagemap for the mapping. +* @dev: The dma unmapper. +* @addr: The dma address obtained when mapping. +*/ +void (*unmap_dma)(struct drm_pagemap *dpagemap, + struct device *dev, + struct drm_pagemap_dma_addr addr); + +}; + +/** +* struct drm_pagemap: Additional information for a struct
dev_pagemap
+ * used for device p2p handshaking.
+ * @ops: The structdrm_pagemap_ops.
+ * @dev: The struct drevice owning the device-private memory.
+ */include/drm/drm_pagemap.h new file +6
-0+struct drm_pagemap { +const struct drm_pagemap_ops *ops; +struct device *dev; +}; + +#endif
Thomas Hellström
maintainerRe: [PATCH v4 04/33] drm/pagemap: Add DRM pagemap
Quoted 54 lines · Matthew Brost
Quoted 14 lines · Thomas Hellström
Quoted 13 lines · Matthew Brost
From: Thomas Hellström <[email protected]>
Introduce
drm_pagemapops to map and unmap dma to VRAM resources. In
the
local memory case it's a matter of merely providing an offset into
the
device's physical address. For future p2p the map and unmap functions
may
encode as needed.Quoted 5 lines
Quoted 4 lines
Similar to how dma-buf works, let the memory provider (
drm_pagemap)
provide
the mapping functionality.Trying to parse all of this.
Quoted 4 lines
It should be noted that the long term idea for dma mapping is to have
that done by the client instead of by the memory provider, which Jason- Client here is the device mapping the memory.
- Memory provider is the device where the memory is located?Did I get this correct?
Quoted 9 lines
reminded me of in a discussion on dri-devel. The dma-mapping here is
modeled after how it's done for dma-buf, where the exporter maps dma.So following that, it might be that we should move these dma- mapping
ops to thedrm_gpusvm().So we move ops to the local client (gpusvm) rather than remote device,
right?Quoted 6 lines
The situation I can think of, where this might be a problem is that if
the device-private struct page to dma address mapping is not known to
the client.
Quoted 7 lines
I'm not following this but I agree if dma mapping at the client we need
the remote device structure given how the dma mapping API works.So to wrap it up - what, if anything, do you think we need to do to this
individual patch as part of this series?
I've been thinking a bit more about this, and I think a change we can do is to rename these methods to something along device_map() and device_unmap(). The purpose would be to emphasize that the resulting addresses are typically not meaningful outside of the driver, and not to be confused with standard dma-mapping.
/Thomas
Quoted 165 lines
Matt
Quoted 162 lines
/Thomas
Quoted 155 lines
v3:
- Move to drm level include
v4:
- Fix kernel doc (G.G.)Signed-off-by Matthew Brost <[email protected]>Signed-off-by Thomas Hellström<[email protected]>
---
include/drm/drm_pagemap.h | 105
++++++++++++++++++++++++++++++++++++++1 file changed, 105 insertions(+)create mode 100644
include/drm/drm_pagemap.hinclude/drm/drm_pagemap.h new file +13
-0@@ -0,0 +1,105 @@ +/* SPDX-License-Identifier: MIT */ +#ifndef _DRM_PAGEMAP_H_ +#define _DRM_PAGEMAP_H_ + +#include <linux/dma-direction.h> +#include <linux/hmm.h> +#include <linux/types.h> + +struct drm_pagemap ; +struct device ; + +/** +* enum drm_interconnect_protocol - Used to identify an interconnect
protocol.
+ */include/drm/drm_pagemap.h new file +2
-0+enum drm_interconnect_protocol { + DRM_INTERCONNECT_SYSTEM, /* DMA map is system pages.*/
include/drm/drm_pagemap.h new file +2
-0+ DRM_INTERCONNECT_PCIE_P2P, /* DMA map is PCIE P2P */ + DRM_INTERCONNECT_DRIVER, /* DMA map is driver defined*/
include/drm/drm_pagemap.h new file +1
-0+/* A driver can add private values beyond
DRM_INTERCONNECT_DRIVER*/include/drm/drm_pagemap.h new file +5
-0+}; + +/** +* struct drm_pagemap_dma_addr - DMA address representation. +* @addr: The dma address or driver-defined address for driver private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size isPAGE_SIZE<<
order).
+ * @dir: The DMA direction.
+ *
+ * Note: There is room for improvement here. We should be able to
pack into
+ * 64 bits.
+ */include/drm/drm_pagemap.h new file +9
-0+struct drm_pagemap_dma_addr { +dma_addr_t addr; +u64 proto :54 ; +u64 order :8 ; +u64 dir :2 ; +}; + +/** +* drm_pagemap_dma_addr_encode() - Encode a dma address with metadata
+ * @addr: The dma address or driver-defined address for driver private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size isPAGE_SIZE<<
order).
+ * @dir: The DMA direction.
+ *
+ * Return: A structdrm_pagemap_dma_addrencoding the above information.
+ */include/drm/drm_pagemap.h new file +3
-0+static inline struct drm_pagemap_dma_addr +drm_pagemap_dma_addr_encode (dma_addr_t addr, + enum drm_interconnect_protocolproto,
include/drm/drm_pagemap.h new file +17
-0+ unsigned int order, + enum dma_data_direction dir) +{ +return (struct drm_pagemap_dma_addr ) { + .addr = addr, + .proto = proto, + .order = order, + .dir = dir, + }; +} + +/** +* struct drm_pagemap_ops: Ops for a drm-pagemap. +*/ +struct drm_pagemap_ops { +/** +* @map_dma: Map for dma access or provide a virtual address
suitable forinclude/drm/drm_pagemap.h new file +5
-0+* +* @dpagemap: The struct drm_pagemap for the page. +* @dev: The dma mapper. +* @page: The page to map. +* @order: The page order of the dma mapping. (Size is
PAGE_SIZE<< order).include/drm/drm_pagemap.h new file +3
-0+* @dir: The transfer direction. +*/ + struct drm_pagemap_dma_addr (*map_dma)(struct
drm_pagemap
*dpagemap,include/drm/drm_pagemap.h new file +1
-0+ struct device*dev,
include/drm/drm_pagemap.h new file +1
-0+ struct page*page,
include/drm/drm_pagemap.h new file +1
-0+ unsigned intorder,
include/drm/drm_pagemap.h new file +1
-0+ enum
dma_data_directiondir);include/drm/drm_pagemap.h new file +3
-0+ +/** +* @unmap_dma: Unmap a dma address previously obtained using
@map_dma.include/drm/drm_pagemap.h new file +13
-0+* +* @dpagemap: The struct drm_pagemap for the mapping. +* @dev: The dma unmapper. +* @addr: The dma address obtained when mapping. +*/ +void (*unmap_dma)(struct drm_pagemap *dpagemap, + struct device *dev, + struct drm_pagemap_dma_addr addr); + +}; + +/** +* struct drm_pagemap: Additional information for a struct
dev_pagemap
+ * used for device p2p handshaking.
+ * @ops: The structdrm_pagemap_ops.
+ * @dev: The struct drevice owning the device-private memory.
+ */include/drm/drm_pagemap.h new file +6
-0+struct drm_pagemap { +const struct drm_pagemap_ops *ops; +struct device *dev; +}; + +#endif
Matthew Brost
maintainerRe: [PATCH v4 04/33] drm/pagemap: Add DRM pagemap
Quoted 63 lines · Thomas Hellström
Quoted 54 lines · Matthew Brost
Quoted 14 lines · Thomas Hellström
Quoted 13 lines · Matthew Brost
From: Thomas Hellström <[email protected]>
Introduce
drm_pagemapops to map and unmap dma to VRAM resources. In
the
local memory case it's a matter of merely providing an offset into
the
device's physical address. For future p2p the map and unmap functions
may
encode as needed.Quoted 5 lines
Quoted 4 lines
Similar to how dma-buf works, let the memory provider (
drm_pagemap)
provide
the mapping functionality.Trying to parse all of this.
Quoted 4 lines
It should be noted that the long term idea for dma mapping is to have
that done by the client instead of by the memory provider, which Jason- Client here is the device mapping the memory.
- Memory provider is the device where the memory is located?Did I get this correct?
Quoted 9 lines
reminded me of in a discussion on dri-devel. The dma-mapping here is
modeled after how it's done for dma-buf, where the exporter maps dma.So following that, it might be that we should move these dma- mapping
ops to thedrm_gpusvm().So we move ops to the local client (gpusvm) rather than remote device,
right?Quoted 6 lines
The situation I can think of, where this might be a problem is that if
the device-private struct page to dma address mapping is not known to
the client.Quoted 7 lines
I'm not following this but I agree if dma mapping at the client we need
the remote device structure given how the dma mapping API works.So to wrap it up - what, if anything, do you think we need to do to this
individual patch as part of this series?
Quoted 6 lines
I've been thinking a bit more about this, and I think a change we can do is to rename these methods to something along
device_map()anddevice_unmap(). The purpose would be to emphasize that the resulting addresses are typically not meaningful outside of the driver, and not to be confused with standard dma-mapping.
Sure. I can rename this.
Matt
Quoted 169 lines
/Thomas
Quoted 165 lines
Matt
Quoted 162 lines
/Thomas
Quoted 155 lines
v3:
- Move to drm level include
v4:
- Fix kernel doc (G.G.)Signed-off-by Matthew Brost <[email protected]>Signed-off-by Thomas Hellström<[email protected]>
---
include/drm/drm_pagemap.h | 105
++++++++++++++++++++++++++++++++++++++1 file changed, 105 insertions(+)create mode 100644
include/drm/drm_pagemap.hinclude/drm/drm_pagemap.h new file +13
-0@@ -0,0 +1,105 @@ +/* SPDX-License-Identifier: MIT */ +#ifndef _DRM_PAGEMAP_H_ +#define _DRM_PAGEMAP_H_ + +#include <linux/dma-direction.h> +#include <linux/hmm.h> +#include <linux/types.h> + +struct drm_pagemap ; +struct device ; + +/** +* enum drm_interconnect_protocol - Used to identify an interconnect
protocol.
+ */include/drm/drm_pagemap.h new file +2
-0+enum drm_interconnect_protocol { + DRM_INTERCONNECT_SYSTEM, /* DMA map is system pages.*/
include/drm/drm_pagemap.h new file +2
-0+ DRM_INTERCONNECT_PCIE_P2P, /* DMA map is PCIE P2P */ + DRM_INTERCONNECT_DRIVER, /* DMA map is driver defined*/
include/drm/drm_pagemap.h new file +1
-0+/* A driver can add private values beyond
DRM_INTERCONNECT_DRIVER*/include/drm/drm_pagemap.h new file +5
-0+}; + +/** +* struct drm_pagemap_dma_addr - DMA address representation. +* @addr: The dma address or driver-defined address for driver private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size isPAGE_SIZE<<
order).
+ * @dir: The DMA direction.
+ *
+ * Note: There is room for improvement here. We should be able to
pack into
+ * 64 bits.
+ */include/drm/drm_pagemap.h new file +9
-0+struct drm_pagemap_dma_addr { +dma_addr_t addr; +u64 proto :54 ; +u64 order :8 ; +u64 dir :2 ; +}; + +/** +* drm_pagemap_dma_addr_encode() - Encode a dma address with metadata
+ * @addr: The dma address or driver-defined address for driver private interconnects.
+ * @proto: The interconnect protocol.
+ * @order: The page order of the dma mapping. (Size isPAGE_SIZE<<
order).
+ * @dir: The DMA direction.
+ *
+ * Return: A structdrm_pagemap_dma_addrencoding the above information.
+ */include/drm/drm_pagemap.h new file +3
-0+static inline struct drm_pagemap_dma_addr +drm_pagemap_dma_addr_encode (dma_addr_t addr, + enum drm_interconnect_protocolproto,
include/drm/drm_pagemap.h new file +17
-0+ unsigned int order, + enum dma_data_direction dir) +{ +return (struct drm_pagemap_dma_addr ) { + .addr = addr, + .proto = proto, + .order = order, + .dir = dir, + }; +} + +/** +* struct drm_pagemap_ops: Ops for a drm-pagemap. +*/ +struct drm_pagemap_ops { +/** +* @map_dma: Map for dma access or provide a virtual address
suitable forinclude/drm/drm_pagemap.h new file +5
-0+* +* @dpagemap: The struct drm_pagemap for the page. +* @dev: The dma mapper. +* @page: The page to map. +* @order: The page order of the dma mapping. (Size is
PAGE_SIZE<< order).include/drm/drm_pagemap.h new file +3
-0+* @dir: The transfer direction. +*/ + struct drm_pagemap_dma_addr (*map_dma)(struct
drm_pagemap
*dpagemap,include/drm/drm_pagemap.h new file +1
-0+ struct device*dev,
include/drm/drm_pagemap.h new file +1
-0+ struct page*page,
include/drm/drm_pagemap.h new file +1
-0+ unsigned intorder,
include/drm/drm_pagemap.h new file +1
-0+ enum
dma_data_directiondir);include/drm/drm_pagemap.h new file +3
-0+ +/** +* @unmap_dma: Unmap a dma address previously obtained using
@map_dma.include/drm/drm_pagemap.h new file +13
-0+* +* @dpagemap: The struct drm_pagemap for the mapping. +* @dev: The dma unmapper. +* @addr: The dma address obtained when mapping. +*/ +void (*unmap_dma)(struct drm_pagemap *dpagemap, + struct device *dev, + struct drm_pagemap_dma_addr addr); + +}; + +/** +* struct drm_pagemap: Additional information for a struct
dev_pagemap
+ * used for device p2p handshaking.
+ * @ops: The structdrm_pagemap_ops.
+ * @dev: The struct drevice owning the device-private memory.
+ */include/drm/drm_pagemap.h new file +6
-0+struct drm_pagemap { +const struct drm_pagemap_ops *ops; +struct device *dev; +}; + +#endif
Matthew Brost
maintainerPATCH v4 5/33 drm/xe/bo: Introduce xe_bo_put_async
In reply to Matthew BrostFrom: Thomas Hellström <[email protected]>
Introduce xe_bo_put_async to put a bo where the context is such that the bo destructor can't run due to lockdep problems or atomic context.
If the put is the final put, freeing will be done from a work item.
--- drivers/gpu/drm/xe/xe_bo.c | 25 +++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_bo.h | 13 +++++++++++++ drivers/gpu/drm/xe/xe_device.c | 3 +++ drivers/gpu/drm/xe/xe_device_types.h | 8 ++++++++ 4 files changed, 49 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +25 -0
@@ -2544,6 +2544,31 @@ void xe_bo_put_commit(struct llist_head *deferred)drm_gem_object_free (&bo->ttm.base.refcount); } +static void xe_bo_dev_work_func (struct work_struct *work) +{ +struct xe_bo_dev *bo_dev =container_of (work,typeof (*bo_dev), async_free); + +xe_bo_put_commit (&bo_dev->async_list); +} + +/** +* xe_bo_dev_init() - Initialize BO dev to manage async BO freeing +* @bo_dev: The BO dev structure +*/ +void xe_bo_dev_init (struct xe_bo_dev *bo_dev) +{ +INIT_WORK (&bo_dev->async_free, xe_bo_dev_work_func); +} + +/** +* xe_bo_dev_fini() - Finalize BO dev managing async BO freeing +* @bo_dev: The BO dev structure +*/ +void xe_bo_dev_fini (struct xe_bo_dev *bo_dev) +{ +flush_work (&bo_dev->async_free); +} +void xe_bo_put (struct xe_bo *bo) {struct xe_tile *tile;
drivers/gpu/drm/xe/xe_bo.h +13 -0
@@ -317,6 +317,19 @@ xe_bo_put_deferred(struct xe_bo *bo, struct llist_head *deferred)void xe_bo_put_commit (struct llist_head *deferred); +static inline void +xe_bo_put_async (struct xe_bo *bo) +{ +struct xe_bo_dev *bo_device = &xe_bo_device (bo)->bo_device; + +if (xe_bo_put_deferred (bo, &bo_device->async_list)) +schedule_work (&bo_device->async_free); +} + +void xe_bo_dev_init (struct xe_bo_dev *bo_device); + +void xe_bo_dev_fini (struct xe_bo_dev *bo_device); +struct sg_table *xe_bo_sg (struct xe_bo *bo);/*
drivers/gpu/drm/xe/xe_device.c +3 -0
@@ -387,6 +387,8 @@ static void xe_device_destroy(struct drm_device *dev, void *dummy) {struct xe_device *xe =to_xe_device (dev); +xe_bo_dev_fini (&xe->bo_device); +if (xe->preempt_fence_wq)destroy_workqueue (xe->preempt_fence_wq);
@@ -424,6 +426,7 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,if (WARN_ON (err)) goto err; +xe_bo_dev_init (&xe->bo_device); err =drmm_add_action_or_reset (&xe->drm, xe_device_destroy,NULL );if (err) goto err;
drivers/gpu/drm/xe/xe_device_types.h +8 -0
@@ -519,6 +519,14 @@ struct xe_device {int mode; } wedged; +/** @bo_device: Struct to control async free of BOs */ +struct xe_bo_dev { +/** @async_free: Free worker */ +struct work_struct async_free; +/** @async_list: List of BOs to be freed */ +struct llist_head async_list; + } bo_device; +/** @pmu: performance monitoring unit */ struct xe_pmu pmu;
Signature · 1 line
2.34.1
Thomas Hellström
maintainerRe: [PATCH v4 05/33] drm/xe/bo: Introduce xe_bo_put_async
Quoted 64 lines · Matthew Brost
From: Thomas Hellström <[email protected]>
Introduce
xe_bo_put_asyncto put a bo where the context is such that the bo destructor can't run due to lockdep problems or atomic context.If the put is the final put, freeing will be done from a work item.
Signed-off-by Matthew Brost <[email protected]>Signed-off-by Thomas Hellström <[email protected]>---
drivers/gpu/drm/xe/xe_bo.c | 25 +++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_bo.h | 13 +++++++++++++ drivers/gpu/drm/xe/xe_device.c | 3 +++drivers/gpu/drm/xe/xe_device_types.h| 8 ++++++++4 files changed, 49 insertions(+)drivers/gpu/drm/xe/xe_bo.c +0
-0@@ -2544,6 +2544,31 @@ void xe_bo_put_commit(struct llist_head *deferred) drm_gem_object_free(&bo->ttm.base.refcount); }drivers/gpu/drm/xe/xe_bo.c +3
-0+static void xe_bo_dev_work_func (struct work_struct *work) +{ + struct xe_bo_dev *bo_dev = container_of(work,
typeof(*bo_dev),async_free);drivers/gpu/drm/xe/xe_bo.c +22
-0+ +xe_bo_put_commit (&bo_dev->async_list); +} + +/** +* xe_bo_dev_init() - Initialize BO dev to manage async BO freeing +* @bo_dev: The BO dev structure +*/ +void xe_bo_dev_init (struct xe_bo_dev *bo_dev) +{ +INIT_WORK (&bo_dev->async_free, xe_bo_dev_work_func); +} + +/** +* xe_bo_dev_fini() - Finalize BO dev managing async BO freeing +* @bo_dev: The BO dev structure +*/ +void xe_bo_dev_fini (struct xe_bo_dev *bo_dev) +{ +flush_work (&bo_dev->async_free); +} +void
xe_bo_put(struct xe_bo *bo)
{
structxe_tile*tile;drivers/gpu/drm/xe/xe_bo.h +0
-0@@ -317,6 +317,19 @@ xe_bo_put_deferred(struct xe_bo *bo, struct llist_head *deferred)
voidxe_bo_put_commit(struct llist_head *deferred);
drivers/gpu/drm/xe/xe_bo.h xe_bo_put_deferred(struct xe_bo *bo, struct llist_head *deferred) · line 321
Quoted 2 lines
drivers/gpu/drm/xe/xe_bo.h +2
-0+static inline void +xe_bo_put_async (struct xe_bo *bo)
Needs kerneldoc. I will rebase my multi-device series on this one, Let me know if you'll add that or if I should do it when rebasing my multi- device series on this one.
Quoted 59 lines
drivers/gpu/drm/xe/xe_bo.h +11
-0+{ +struct xe_bo_dev *bo_device = &xe_bo_device (bo)->bo_device; + +if (xe_bo_put_deferred (bo, &bo_device->async_list)) +schedule_work (&bo_device->async_free); +} + +void xe_bo_dev_init (struct xe_bo_dev *bo_device); + +void xe_bo_dev_fini (struct xe_bo_dev *bo_device); +struct
sg_table*xe_bo_sg(structxe_bo*bo);
/*drivers/gpu/drm/xe/xe_device.c +0
-0@@ -387,6 +387,8 @@ static void xe_device_destroy(struct drm_device *dev, void *dummy) { struct xe_device *xe = to_xe_device(dev);drivers/gpu/drm/xe/xe_device.c +2
-0+xe_bo_dev_fini (&xe->bo_device); +if (
xe->preempt_fence_wq)destroy_workqueue(xe->preempt_fence_wq);drivers/gpu/drm/xe/xe_device.c +0
-0@@ -424,6 +426,7 @@ struct xe_device *xe_device_create(struct pci_dev *pdev, if (WARN_ON(err)) goto err;drivers/gpu/drm/xe/xe_device.c +1
-0+xe_bo_dev_init (&xe->bo_device);err =
drmm_add_action_or_reset(&xe->drm, xe_device_destroy, NULL);
if (err)
goto err;drivers/gpu/drm/xe/xe_device_types.h +0
-0@@ -519,6 +519,14 @@ struct xe_device { int mode; } wedged;drivers/gpu/drm/xe/xe_device_types.h +8
-0+/** @bo_device: Struct to control async free of BOs */ +struct xe_bo_dev { +/** @async_free: Free worker */ +struct work_struct async_free; +/** @async_list: List of BOs to be freed */ +struct llist_head async_list; + } bo_device; +/** @pmu: performance monitoring unit */
structxe_pmupmu;
Matthew Brost
maintainerRe: [PATCH v4 05/33] drm/xe/bo: Introduce xe_bo_put_async
Quoted 63 lines · Thomas Hellström
Quoted 62 lines · Matthew Brost
From: Thomas Hellström <[email protected]>
Introduce
xe_bo_put_asyncto put a bo where the context is such that the bo destructor can't run due to lockdep problems or atomic context.If the put is the final put, freeing will be done from a work item.
Signed-off-by Matthew Brost <[email protected]>Signed-off-by Thomas Hellström <[email protected]>---
drivers/gpu/drm/xe/xe_bo.c | 25 +++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_bo.h | 13 +++++++++++++ drivers/gpu/drm/xe/xe_device.c | 3 +++drivers/gpu/drm/xe/xe_device_types.h| 8 ++++++++4 files changed, 49 insertions(+)drivers/gpu/drm/xe/xe_bo.c +0
-0@@ -2544,6 +2544,31 @@ void xe_bo_put_commit(struct llist_head *deferred) drm_gem_object_free(&bo->ttm.base.refcount); }drivers/gpu/drm/xe/xe_bo.c +3
-0+static void xe_bo_dev_work_func (struct work_struct *work) +{ + struct xe_bo_dev *bo_dev = container_of(work,
typeof(*bo_dev),async_free);drivers/gpu/drm/xe/xe_bo.c +22
-0+ +xe_bo_put_commit (&bo_dev->async_list); +} + +/** +* xe_bo_dev_init() - Initialize BO dev to manage async BO freeing +* @bo_dev: The BO dev structure +*/ +void xe_bo_dev_init (struct xe_bo_dev *bo_dev) +{ +INIT_WORK (&bo_dev->async_free, xe_bo_dev_work_func); +} + +/** +* xe_bo_dev_fini() - Finalize BO dev managing async BO freeing +* @bo_dev: The BO dev structure +*/ +void xe_bo_dev_fini (struct xe_bo_dev *bo_dev) +{ +flush_work (&bo_dev->async_free); +} +void
xe_bo_put(struct xe_bo *bo)
{
structxe_tile*tile;drivers/gpu/drm/xe/xe_bo.h +0
-0@@ -317,6 +317,19 @@ xe_bo_put_deferred(struct xe_bo *bo, struct llist_head *deferred)
Quoted 9 lines
drivers/gpu/drm/xe/xe_bo.h xe_bo_put_deferred(struct xe_bo *bo, struct llist_head *deferred) · line 321
Quoted 4 lines
void
xe_bo_put_commit(struct llist_head *deferred);drivers/gpu/drm/xe/xe_bo.h +2
-0+static inline void +xe_bo_put_async (struct xe_bo *bo)Needs kerneldoc. I will rebase my multi-device series on this one, Let me know if you'll add that or if I should do it when rebasing my multi- device series on this one.
Yep. Added kernel for structures / exported functions but missed this inline. I should be able to write something here.
Matt
Quoted 60 lines
Quoted 59 lines
drivers/gpu/drm/xe/xe_bo.h +11
-0+{ +struct xe_bo_dev *bo_device = &xe_bo_device (bo)->bo_device; + +if (xe_bo_put_deferred (bo, &bo_device->async_list)) +schedule_work (&bo_device->async_free); +} + +void xe_bo_dev_init (struct xe_bo_dev *bo_device); + +void xe_bo_dev_fini (struct xe_bo_dev *bo_device); +struct
sg_table*xe_bo_sg(structxe_bo*bo);
/*drivers/gpu/drm/xe/xe_device.c +0
-0@@ -387,6 +387,8 @@ static void xe_device_destroy(struct drm_device *dev, void *dummy) { struct xe_device *xe = to_xe_device(dev);drivers/gpu/drm/xe/xe_device.c +2
-0+xe_bo_dev_fini (&xe->bo_device); +if (
xe->preempt_fence_wq)destroy_workqueue(xe->preempt_fence_wq);drivers/gpu/drm/xe/xe_device.c +0
-0@@ -424,6 +426,7 @@ struct xe_device *xe_device_create(struct pci_dev *pdev, if (WARN_ON(err)) goto err;drivers/gpu/drm/xe/xe_device.c +1
-0+xe_bo_dev_init (&xe->bo_device);err =
drmm_add_action_or_reset(&xe->drm, xe_device_destroy, NULL);
if (err)
goto err;drivers/gpu/drm/xe/xe_device_types.h +0
-0@@ -519,6 +519,14 @@ struct xe_device { int mode; } wedged;drivers/gpu/drm/xe/xe_device_types.h +8
-0+/** @bo_device: Struct to control async free of BOs */ +struct xe_bo_dev { +/** @async_free: Free worker */ +struct work_struct async_free; +/** @async_list: List of BOs to be freed */ +struct llist_head async_list; + } bo_device; +/** @pmu: performance monitoring unit */
structxe_pmupmu;
Matthew Brost
maintainerPATCH v4 6/33 drm/gpusvm: Add support for GPU Shared Virtual Memory
In reply to Matthew BrostThis patch introduces support for GPU Shared Virtual Memory (SVM) in the Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.
The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also provides mechanisms for allocating, deallocating, and migrating memory regions between system RAM and GPU VRAM.
This is largely inspired by GPUVM.
v2:
- Take order into account in check pages
- Clear range->pages in get pages error
- Drop setting dirty or accessed bit in get pages (Vetter)
- Remove mmap assert for cpu faults
- Drop mmap write lock abuse (Vetter, Christian)
- Decouple zdd from range (Vetter, Oak)
- Add drm_gpusvm_range_evict, make it work with coherent pages
- Export drm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
- mmget/put in drm_gpusvm_evict_to_sram
- Drop range->vram_alloation variable
- Don't return in drm_gpusvm_evict_to_sram until all pages detached
- Don't warn on mixing sram and device pages
- Update kernel doc
- Add coherent page support to get pages
- Use DMA_FROM_DEVICE rather than DMA_BIDIRECTIONAL
- Add struct drm_gpusvm_vram and ops (Thomas)
- Update the range's seqno if the range is valid (Thomas)
- Remove the is_unmapped check before hmm_range_fault (Thomas)
- Use drm_pagemap (Thomas)
- Drop kfree_mapping (Thomas)
- dma mapp pages under notifier lock (Thomas)
- Remove ctx.prefault
- Remove ctx.mmap_locked
- Add ctx.check_pages
- s/vram/devmem (Thomas)
v3:
- Fix memory leak drm_gpusvm_range_get_pages
- Only migrate pages with same zdd on CPU fault
- Loop over al VMAs in drm_gpusvm_range_evict
- Make GPUSVM a drm level module
- GPL or MIT license
- Update main kernel doc (Thomas)
- Prefer foo() vs foo for functions in kernel doc (Thomas)
- Prefer functions over macros (Thomas)
- Use unsigned long vs u64 for addresses (Thomas)
- Use standard interval_tree (Thomas)
- s/drm_gpusvm_migration_put_page/drm_gpusvm_migration_unlock_put_page (Thomas)
- Drop err_out label in drm_gpusvm_range_find_or_insert (Thomas)
- Fix kernel doc in drm_gpusvm_range_free_pages (Thomas)
- Newlines between functions defs in header file (Thomas)
- Drop shall language in driver vfunc kernel doc (Thomas)
- Move some static inlines from head to C file (Thomas)
- Don't allocate pages under page lock in drm_gpusvm_migrate_populate_ram_pfn (Thomas)
- Change check_pages to a threshold
v4:
- Fix NULL ptr deref in drm_gpusvm_migrate_populate_ram_pfn (Thomas, Himal)
- Fix check pages threshold
- Check for range being unmapped under notifier lock in get pages (Testing)
- Fix characters per line
- Drop WRITE_ONCE for zdd->devmem_allocation assignment (Thomas)
- Use completion for devmem_allocation->detached (Thomas)
- Make GPU SVM depend on ZONE_DEVICE (CI)
- Use hmm_range_fault for eviction (Thomas)
- Drop zdd worker (Thomas)
--- drivers/gpu/drm/Kconfig | 9 + drivers/gpu/drm/Makefile | 1 + drivers/gpu/drm/drm_gpusvm.c | 2240 ++++++++++++++++++++++++++++++++++ include/drm/drm_gpusvm.h | 445 +++++++ 4 files changed, 2695 insertions(+) create mode 100644 drivers/gpu/drm/drm_gpusvm.c create mode 100644 include/drm/drm_gpusvm.h
drivers/gpu/drm/Kconfig +9 -0
@@ -278,6 +278,15 @@ config DRM_GPUVM GPU-VM representation providing helpers to manage a GPUs virtual address space +config DRM_GPUSVM + tristate + depends on DRM + depends on DEVICE_MIGRATION + depends on ZONE_DEVICE + help + GPU-SVM representation providing helpers to manage a GPUs shared + virtual memory + config DRM_BUDDY tristate depends on DRM
drivers/gpu/drm/Makefile +1 -0
@@ -104,6 +104,7 @@ obj-$(CONFIG_DRM_PANEL_BACKLIGHT_QUIRKS) += drm_panel_backlight_quirks.o # obj-$(CONFIG_DRM_EXEC) += drm_exec.o obj-$(CONFIG_DRM_GPUVM) += drm_gpuvm.o +obj-$(CONFIG_DRM_GPUSVM) += drm_gpusvm.o obj-$(CONFIG_DRM_BUDDY) += drm_buddy.o
drivers/gpu/drm/drm_gpusvm.c new file +2240 -0
2,240 lines not on this page · raw message
include/drm/drm_gpusvm.h new file +445 -0
445 lines not on this page · raw message
Signature · 1 line
2.34.1
Thomas Hellström
maintainerRe: [PATCH v4 06/33] drm/gpusvm: Add support for GPU Shared Virtual Memory
Quoted 77 lines · Matthew Brost
This patch introduces support for GPU Shared Virtual Memory (SVM) in the
Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also
provides mechanisms for allocating, deallocating, and migrating memory
regions between system RAM and GPU VRAM.This is largely inspired by GPUVM.
v2:
- Take order into account in check pages
- Clearrange->pagesin get pages error
- Drop setting dirty or accessed bit in get pages (Vetter)
- Remove mmap assert for cpu faults
- Drop mmap write lock abuse (Vetter, Christian)
- Decouple zdd from range (Vetter, Oak)
- Adddrm_gpusvm_range_evict, make it work with coherent pages
- Exportdrm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
- mmget/put indrm_gpusvm_evict_to_sram
- Droprange->vram_alloationvariable
- Don't return indrm_gpusvm_evict_to_sramuntil all pages detached
- Don't warn on mixing sram and device pages
- Update kernel doc
- Add coherent page support to get pages
- UseDMA_FROM_DEVICErather thanDMA_BIDIRECTIONAL
- Add structdrm_gpusvm_vramand ops (Thomas)
- Update the range's seqno if the range is valid (Thomas)
- Remove theis_unmappedcheck beforehmm_range_fault(Thomas)
- Usedrm_pagemap(Thomas)
- Dropkfree_mapping(Thomas)
- dma mapp pages under notifier lock (Thomas)
- Remove ctx.prefault
- Remove ctx.mmap_locked
- Add ctx.check_pages
- s/vram/devmem (Thomas)
v3:
- Fix memory leakdrm_gpusvm_range_get_pages
- Only migrate pages with same zdd on CPU fault
- Loop over al VMAs indrm_gpusvm_range_evict
- Make GPUSVM a drm level module
- GPL or MIT license
- Update main kernel doc (Thomas)
- Preferfoo()vs foo for functions in kernel doc (Thomas)
- Prefer functions over macros (Thomas)
- Use unsigned long vs u64 for addresses (Thomas)
- Use standardinterval_tree(Thomas)
- s/drm_gpusvm_migration_put_page/drm_gpusvm_migration_unlock_put_page (Thomas)
- Droperr_outlabel indrm_gpusvm_range_find_or_insert(Thomas)
- Fix kernel doc indrm_gpusvm_range_free_pages(Thomas)
- Newlines between functions defs in header file (Thomas)
- Drop shall language in driver vfunc kernel doc (Thomas)
- Move some static inlines from head to C file (Thomas)
- Don't allocate pages under page lock indrm_gpusvm_migrate_populate_ram_pfn(Thomas)
- Changecheck_pagesto a threshold
v4:
- Fix NULL ptr deref indrm_gpusvm_migrate_populate_ram_pfn(Thomas, Himal)
- Fix check pages threshold
- Check for range being unmapped under notifier lock in get pages (Testing)
- Fix characters per line
- DropWRITE_ONCEforzdd->devmem_allocationassignment (Thomas)
- Use completion fordevmem_allocation->detached(Thomas)
- Make GPU SVM depend onZONE_DEVICE(CI)
- Usehmm_range_faultfor eviction (Thomas)
- Drop zdd worker (Thomas)
Quoted 6 lines
Cc Simona Vetter <[email protected]>Cc Dave Airlie <[email protected]>Cc Christian König <[email protected]>Cc <[email protected]>Signed-off-by Matthew Brost <[email protected]>Signed-off-by Thomas Hellström <[email protected]>
Let's consider my minor contributions to this version as review comments for now, so no SOB needed.
Quoted 1,699 lines
---
drivers/gpu/drm/Kconfig | 9 +
drivers/gpu/drm/Makefile | 1 +
drivers/gpu/drm/drm_gpusvm.c | 2240
++++++++++++++++++++++++++++++++++
include/drm/drm_gpusvm.h | 445 +++++++4 files changed, 2695 insertions(+)create mode 100644
drivers/gpu/drm/drm_gpusvm.c
create mode 100644include/drm/drm_gpusvm.hdrivers/gpu/drm/Kconfig +0
-0@@ -278,6 +278,15 @@ config DRM_GPUVMGPU-VM representation providing helpers to manage a GPUs virtual
address spacedrivers/gpu/drm/Kconfig +7
-0+config DRM_GPUSVM + tristate + depends on DRM + depends on DEVICE_MIGRATION + depends on ZONE_DEVICE + help + GPU-SVM representation providing helpers to manage a GPUsshared
drivers/gpu/drm/Kconfig +2
-0+ virtual memory +config
DRM_BUDDY
tristate
depends on DRMdrivers/gpu/drm/Makefile +0
-0@@ -104,6 +104,7 @@ obj-$(CONFIG_DRM_PANEL_BACKLIGHT_QUIRKS) +=drm_panel_backlight_quirks.o
#
obj-$(CONFIG_DRM_EXEC) += drm_exec.o
obj-$(CONFIG_DRM_GPUVM) += drm_gpuvm.o
+obj-$(CONFIG_DRM_GPUSVM) += drm_gpusvm.o
obj-$(CONFIG_DRM_BUDDY) += drm_buddy.odrivers/gpu/drm/drm_gpusvm.c new file +25
-0@@ -0,0 +1,2240 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* +* Copyright © 2024 Intel Corporation +* +* Authors: +* Matthew Brost <[email protected]> +*/ + +#include <linux/dma-mapping.h> +#include <linux/hmm.h> +#include <linux/memremap.h> +#include <linux/migrate.h> +#include <linux/mm_types.h> +#include <linux/pagemap.h> +#include <linux/slab.h> + +#include <drm/drm_device.h> +#include <drm/drm_gpusvm.h> +#include <drm/drm_pagemap.h> +#include <drm/drm_print.h> + +/** +* DOC: Overview +* +* GPU Shared Virtual Memory (GPU SVM) layer for the Direct Rendering Manager (DRM)
+ *
+ * The GPU SVM layer is a component of the DRM framework designed to manage shared
+ * virtual memory between the CPU and GPU. It enables efficient data exchange and
+ * processing for GPU-accelerated applications by allowing memory sharing and
+ * synchronization between the CPU's and GPU's virtual address spaces.
+ *
+ * Key GPU SVM Components:
+ * - Notifiers: Notifiers: Used for tracking memory intervals and notifying the
+ * GPU of changes, notifiers are sized based on a GPU
SVM
+ * initialization parameter, with a recommendation of
512M or
+ * larger. They maintain a Red-BlacK tree and a list of
ranges that
+ * fall within the notifier interval. Notifiers are
tracked within
+ * a GPU SVM Red-BlacK tree and list and are
dynamically inserted
+ * or removed as ranges within the interval are created
or
+ * destroyed.
+ * - Ranges: Represent memory ranges mapped in a DRM device and managed
+ * by GPU SVM. They are sized based on an array of chunk
sizes, which
+ * is a GPU SVM initialization parameter, and the CPU
address space.
+ * Upon GPU fault, the largest aligned chunk that fits
within the
+ * faulting CPU address space is chosen for the range
size. Ranges are
+ * expected to be dynamically allocated on GPU fault and
removed on an
+ * MMU notifier UNMAP event. As mentioned above, ranges
are tracked in
+ * a notifier's Red-Black tree.
+ * - Operations: Define the interface for driver-specific GPU SVM operations
+ * such as range allocation, notifier allocation, and
+ * invalidations.
+ * - Device Memory Allocations: Embedded structure containing enough information
+ * for GPU SVM to migrate to / from device memory.
+ * - Device Memory Operations: Define the interface for driver- specific device
+ * memory operations release memory, populate pfns,
+ * and copy to / from device memory.
+ *
+ * This layer provides interfaces for allocating, mapping, migrating, and
+ * releasing memory ranges between the CPU and GPU. It handles all core memory
+ * management interactions (DMA mapping, HMM, and migration) and provides
+ * driver-specific virtual functions (vfuncs). This infrastructure is sufficient
+ * to build the expected driver components for an SVM implementation as detailed
+ * below.
+ *
+ * Expected Driver Components:
+ * - GPU page fault handler: Used to create ranges and notifiers based on the
+ * fault address, optionally migrate the
range to
+ * device memory, and create GPU bindings.
+ * - Garbage collector: Used to unmap and destroy GPU bindings for ranges.
+ * Ranges are expected to be added to the
garbage collector
+ * upon aMMU_NOTIFY_UNMAPevent in notifier
callback.
+ * - Notifier callback: Used to invalidate and DMA unmap GPU bindings for
+ * ranges.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +/** +* DOC: Locking +* +* GPU SVM handles locking for core MM interactions, i.e., it locks/unlocks the
+ * mmap lock as needed.
+ *
+ * GPU SVM introduces a global notifier lock, which safeguards the notifier's
+ * range RB tree and list, as well as the range's DMA mappings and sequence
+ * number. GPU SVM manages all necessary locking and unlocking operations,
+ * except for the recheck range's pages being valid
+ * (drm_gpusvm_range_pages_valid) when the driver is committing GPU bindings. This
+ * lock corresponds to the 'driver->update' lock mentioned in the HMM
+ * documentation (TODO: Link). Future revisions may transition from a GPU SVM
+ * global lock to a per-notifier lock if finer-grained locking is deemed
+ * necessary.
+ *
+ * In addition to the locking mentioned above, the driver should implement a
+ * lock to safeguard core GPU SVM function calls that modify state, such as
+ *drm_gpusvm_range_find_or_insertanddrm_gpusvm_range_remove. This lock is
+ * denoted as 'driver_svm_lock' in code examples. Finer grained driver side
+ * locking should also be possible for concurrent GPU fault processing within a
+ * single GPU SVM. The 'driver_svm_lock' can be viadrm_gpusvm_driver_set_lock
+ * to add annotations to GPU SVM.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +/** +* DOC: Migration +* +* The migration support is quite simple, allowing migration between RAM and
+ * device memory at the range granularity. For example, GPU SVM currently does not
+ * support mixing RAM and device memory pages within a range. This means that upon GPU
+ * fault, the entire range can be migrated to device memory, and upon CPU fault, the
+ * entire range is migrated to RAM. Mixed RAM and device memory storage within a range
+ * could be added in the future if required.
+ *
+ * The reasoning for only supporting range granularity is as
follows: it
+ * simplifies the implementation, and range sizes are driver-defined and should
+ * be relatively small.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +/** +* DOC: Partial Unmapping of Ranges +* +* Partial unmapping of ranges (e.g., 1M out of 2M is unmapped by CPU resulting
+ * inMMU_NOTIFY_UNMAPevent) presents several challenges, with the main one
+ * being that a subset of the range still has CPU and GPU mappings. If the
+ * backing store for the range is in device memory, a subset of the backing store has
+ * references. One option would be to split the range and device memory backing store,
+ * but the implementation for this would be quite complicated. Given that
+ * partial unmappings are rare and driver-defined range sizes are relatively
+ * small, GPU SVM does not support splitting of ranges.
+ *
+ * With no support for range splitting, upon partial unmapping of a range, the
+ * driver is expected to invalidate and destroy the entire range. If the range
+ * has device memory as its backing, the driver is also expected to migrate any
+ * remaining pages back to RAM.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +/** +* DOC: Examples +* +* This section provides three examples of how to build the expected driver
+ * components: the GPU page fault handler, the garbage collector, and the
+ * notifier callback.
+ *
+ * The generic code provided does not include logic for complex migration
+ * policies, optimized invalidations, fined grained driver locking, or other
+ * potentially required driver locking (e.g., DMA-resv locks).
+ *
+ * 1) GPU page fault handler
+ *
+ * int driver_bind_range(structdrm_gpusvm*gpusvm, structdrm_gpusvm_range*range)
+ * {
+ * int err = 0;
+ *
+ * driver_alloc_and_setup_memory_for_bind(gpusvm,
range);
+ *
+ *drm_gpusvm_notifier_lock(gpusvm);
+ * if (drm_gpusvm_range_pages_valid(range))
+ *driver_commit_bind(gpusvm, range);
+ * else
+ * err =-EAGAIN;
+ *drm_gpusvm_notifier_unlock(gpusvm);
+ *
+ * return err;
+ * }
+ *
+ * int driver_gpu_fault(structdrm_gpusvm*gpusvm, unsigned
longfault_addr,
+ * unsigned longgpuva_start, unsigned
longgpuva_end)
+ * {
+ * structdrm_gpusvm_ctxctx = {};
+ * int err;
+ *
+ *driver_svm_lock();
+ * retry:
+ * // Always process UNMAPs first so view of GPU SVM
ranges is current
+ *driver_garbage_collector(gpusvm);
+ *
+ * range = drm_gpusvm_range_find_or_insert(gpusvm,fault_addr,
+ *gpuva_start,gpuva_end,
+ * &ctx);
+ * if (IS_ERR(range)) {
+ * err =PTR_ERR(range);
+ * goto unlock;
+ * }
+ *
+ * if (driver_migration_policy(range)) {
+ * devmem =driver_alloc_devmem();
+ * err = drm_gpusvm_migrate_to_devmem(gpusvm,
range,
+ *devmem_allocation,
+ * &ctx);
+ * if (err) // CPU mappings may have
changed
+ * goto retry;
+ * }
+ *
+ * err = drm_gpusvm_range_get_pages(gpusvm, range,
&ctx);
+ * if (err ==-EOPNOTSUPP|| err ==-EFAULT|| err == -
EPERM) { // CPU mappings changed
+ * if (err ==-EOPNOTSUPP)
+ * drm_gpusvm_range_evict(gpusvm,
range);
+ * goto retry;
+ * } else if (err) {
+ * goto unlock;
+ * }
+ *
+ * err =driver_bind_range(gpusvm, range);
+ * if (err ==-EAGAIN) // CPU mappings changed
+ * goto retry
+ *
+ * unlock:
+ *driver_svm_unlock();
+ * return err;
+ * }
+ *
+ * 2) Garbage Collector.
+ *
+ * void __driver_garbage_collector(structdrm_gpusvm*gpusvm,
+ * structdrm_gpusvm_range
*range)
+ * {
+ *assert_driver_svm_locked(gpusvm);
+ *
+ * // Partial unmap, migrate any remaining device
memory pages back to RAM
+ * if (range->flags.partial_unmap)
+ *drm_gpusvm_range_evict(gpusvm, range);
+ *
+ *driver_unbind_range(range);
+ *drm_gpusvm_range_remove(gpusvm, range);
+ * }
+ *
+ * voiddriver_garbage_collector(struct drm_gpusvm *gpusvm)
+ * {
+ *assert_driver_svm_locked(gpusvm);
+ *
+ *for_each_range_in_garbage_collector(gpusvm, range)
+ *__driver_garbage_collector(gpusvm, range);
+ * }
+ *
+ * 3) Notifier callback.
+ *
+ * void driver_invalidation(structdrm_gpusvm*gpusvm,
+ * structdrm_gpusvm_notifier
*notifier,
+ * const structmmu_notifier_range*mmu_range)
+ * {
+ * structdrm_gpusvm_ctxctx = { .in_notifier = true,
};
+ * structdrm_gpusvm_range*range = NULL;
+ *
+ * driver_invalidate_device_pages(gpusvm, mmu_range-Quoted 1 line
start,
mmu_range->end);+ *
+ * drm_gpusvm_for_each_range(range, notifier,mmu_range->start,
+ *mmu_range->end) {
+ * drm_gpusvm_range_unmap_pages(gpusvm, range,
&ctx);
+ *
+ * if (mmu_range->event!=MMU_NOTIFY_UNMAP)
+ * continue;
+ *
+ * drm_gpusvm_range_set_unmapped(range,mmu_range);
+ *driver_garbage_collector_add(gpusvm, range);
+ * }
+ * }
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ +/** +* npages_in_range() - Calculate the number of pages in a given range
+ * @start: The start address of the range
+ * @end: The end address of the range
+ *
+ * This macro calculates the number of pages in a given memory range,
+ * specified by the start and end addresses. It divides the difference
+ * between the end and start addresses by the page size (PAGE_SIZE) to
+ * determine the number of pages in the range.
+ *
+ * Returns: The number of pages in the specified range.
+ */drivers/gpu/drm/drm_gpusvm.c new file +15
-0+static unsigned long +npages_in_range (unsigned long start,unsigned long end) +{ +return (end - start) >>PAGE_SHIFT ; +} + +/** +* struct drm_gpusvm_zdd - GPU SVM zone device data +* +* @refcount: Reference count for the zdd +* @devmem_allocation: device memory allocation +* @device_private_page_owner: Device private pages owner +* +* This structure serves as a generic wrapper installed in +* page->zone_device_data. It provides infrastructure for looking up a device
+ * memory allocation upon CPU page fault and asynchronously releasing device
+ * memory once the CPU has no page references. Asynchronous release is useful
+ * because CPU page references can be dropped in IRQ contexts, while releasing
+ * device memory likely requires sleeping locks.
+ */drivers/gpu/drm/drm_gpusvm.c new file +11
-0+struct drm_gpusvm_zdd { +struct kref refcount; +struct drm_gpusvm_devmem *devmem_allocation; +void *device_private_page_owner; +}; + +/** +* drm_gpusvm_zdd_alloc() - Allocate a zdd structure. +* @device_private_page_owner: Device private pages owner +* +* This function allocates and initializes a new zdd structure. It sets up the
+ * reference count and initializes the destroy work.
+ *
+ * Returns:
+ * Pointer to the allocated zdd on success,ERR_PTR()on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +21
-0+static struct drm_gpusvm_zdd * +drm_gpusvm_zdd_alloc (void *device_private_page_owner) +{ +struct drm_gpusvm_zdd *zdd; + + zdd =kmalloc (sizeof (*zdd),GFP_KERNEL ); +if (!zdd) +return NULL ; + +kref_init (&zdd->refcount); + zdd->devmem_allocation =NULL ; + zdd->device_private_page_owner = device_private_page_owner; + +return zdd; +} + +/** +* drm_gpusvm_zdd_get() - Get a reference to a zdd structure. +* @zdd: Pointer to the zdd structure. +* +* This function increments the reference count of the provided zdd structure.
+ *
+ * Returns: Pointer to the zdd structure.
+ */
+static structdrm_gpusvm_zdd*drm_gpusvm_zdd_get(structdrm_gpusvm_zdd*zdd)drivers/gpu/drm/drm_gpusvm.c new file +10
-0+{ +kref_get (&zdd->refcount); +return zdd; +} + +/** +* drm_gpusvm_zdd_destroy() - Destroy a zdd structure. +* @ref: Pointer to the reference count structure. +* +* This function queues the destroy_work of the zdd for asynchronous destruction.
+ */drivers/gpu/drm/drm_gpusvm.c new file +19
-0+static void drm_gpusvm_zdd_destroy (struct kref *ref) +{ +struct drm_gpusvm_zdd *zdd = + container_of(ref, struct drm_gpusvm_zdd, refcount); +struct drm_gpusvm_devmem *devmem = zdd->devmem_allocation; + +if (devmem) { +complete_all (&devmem->detached); +if (devmem->ops->devmem_release) + devmem->ops->devmem_release (devmem); + } +kfree (zdd); +} + +/** +* drm_gpusvm_zdd_put() - Put a zdd reference. +* @zdd: Pointer to the zdd structure. +* +* This function decrements the reference count of the provided zdd structure
+ * and schedules its destruction if the count drops to zero.
+ */drivers/gpu/drm/drm_gpusvm.c new file +7
-0+static void drm_gpusvm_zdd_put (struct drm_gpusvm_zdd *zdd) +{ +kref_put (&zdd->refcount, drm_gpusvm_zdd_destroy); +} + +/** +* drm_gpusvm_range_find() - Find GPU SVM range from GPU SVM notifier
+ * @notifier: Pointer to the GPU SVM notifier structure.
+ * @start: Start address of the range
+ * @end: End address of the range
+ *
+ * Returns: A pointer to thedrm_gpusvm_rangeif found or NULL
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+struct drm_gpusvm_range * +drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, unsignedlong start,
drivers/gpu/drm/drm_gpusvm.c new file +8
-1+ unsigned long end) +{ +struct interval_tree_node *itree; + + itree = interval_tree_iter_first(¬ifier->root, start, end - 1); + + if (itree) + return container_of(itree, struct drm_gpusvm_range,itree);
drivers/gpu/drm/drm_gpusvm.c new file +7
-0+else +return NULL ; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_find); + +/** +* drm_gpusvm_for_each_range_safe() - Safely iterate over GPU SVM ranges in a notifier
+ * @range__: Iterator variable for the ranges
+ * @next__: Iterator variable for the ranges temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier while
+ * removing ranges from it.
+ */
+#define drm_gpusvm_for_each_range_safe(range__,next__,notifier__,start__,end__) \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ for ((range__) = drm_gpusvm_range_find((notifier__),(
start__), (end__)), \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (next__) =
__drm_gpusvm_range_next(range__); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (range__) && (range__->itree.start <(
end__)); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (range__) = (next__), (next__) =
__drm_gpusvm_range_next(range__))drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ +/** +* __drm_gpusvm_notifier_next() - get the next drm_gpusvm_notifier in the list
+ * @notifier: a pointer to the currentdrm_gpusvm_notifier
+ *
+ * Returns: A pointer to the nextdrm_gpusvm_notifierif available, or NULL if
+ * the current notifier is the last one or if the input notifier is
+ * NULL.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+static struct drm_gpusvm_notifier * +__drm_gpusvm_notifier_next(struct drm_gpusvm_notifier *notifier) +{ + if (notifier && !list_is_last(¬ifier->entry, + ¬ifier->gpusvm-Quoted 1 line
notifier_list))drivers/gpu/drm/drm_gpusvm.c new file +7
-0+return list_next_entry (notifier, entry); + +return NULL ; +} + +static struct drm_gpusvm_notifier * +notifier_iter_first(struct rb_root_cached *root, unsigned longstart,
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ unsigned long last) +{ +struct interval_tree_node *itree; + + itree =interval_tree_iter_first (root, start, last); + +if (itree) + return container_of(itree, struct
drm_gpusvm_notifier, itree);drivers/gpu/drm/drm_gpusvm.c new file +6
-0+else +return NULL ; +} + +/** +* drm_gpusvm_for_each_notifier() - Iterate over GPU SVM notifiers in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm.
+ */
+#define drm_gpusvm_for_each_notifier(notifier__,gpusvm__,start__,end__) \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,(
start__), (end__) - 1); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (notifier__) && (notifier__->itree.start <(
end__)); \drivers/gpu/drm/drm_gpusvm.c new file +4
-0+ (notifier__) = __drm_gpusvm_notifier_next(notifier__)) + +/** +* drm_gpusvm_for_each_notifier_safe() - Safely iterate over GPU SVM notifiers in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @next__: Iterator variable for the notifiers temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm while
+ * removing notifiers from it.
+ */
+#define drm_gpusvm_for_each_notifier_safe(notifier__,next__,gpusvm__,start__,end__) \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,(
start__), (end__) - 1), \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (next__) =
__drm_gpusvm_notifier_next(notifier__); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (notifier__) && (notifier__->itree.start <(
end__)); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (notifier__) = (next__), (next__) =
__drm_gpusvm_notifier_next(notifier__))drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ +/** +* drm_gpusvm_notifier_invalidate() - Invalidate a GPU SVM notifier. +* @mni: Pointer to the mmu_interval_notifier structure. +* @mmu_range: Pointer to the mmu_notifier_range structure. +* @cur_seq: Current sequence number. +* +* This function serves as a generic MMU notifier for GPU SVM. It sets the MMU
+ * notifier sequence number and calls the driver invalidate vfunc under
+ *gpusvm->notifier_lock.
+ *
+ * Returns:
+ * true if the operation succeeds, false otherwise.
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+static bool +drm_gpusvm_notifier_invalidate (struct mmu_interval_notifier *mni, + const struct mmu_notifier_range
*mmu_range,drivers/gpu/drm/drm_gpusvm.c new file +19
-0+ unsigned long cur_seq) +{ +struct drm_gpusvm_notifier *notifier = +container_of (mni,typeof (*notifier), notifier); +struct drm_gpusvm *gpusvm = notifier->gpusvm; + +if (!mmu_notifier_range_blockable (mmu_range)) +return false; + +down_write (&gpusvm->notifier_lock); +mmu_interval_set_seq (mni, cur_seq); + gpusvm->ops->invalidate (gpusvm, notifier, mmu_range); +up_write (&gpusvm->notifier_lock); + +return true; +} + +/** +* drm_gpusvm_notifier_ops - MMU interval notifier operations for GPU SVM
+ */
+static const structmmu_interval_notifier_opsdrm_gpusvm_notifier_ops= {drivers/gpu/drm/drm_gpusvm.c new file +15
-0+ .invalidate = drm_gpusvm_notifier_invalidate, +}; + +/** +* drm_gpusvm_init() - Initialize the GPU SVM. +* @gpusvm: Pointer to the GPU SVM structure. +* @name: Name of the GPU SVM. +* @drm: Pointer to the DRM device structure. +* @mm: Pointer to the mm_struct for the address space. +* @device_private_page_owner: Device private pages owner. +* @mm_start: Start address of GPU SVM. +* @mm_range: Range of the GPU SVM. +* @notifier_size: Size of individual notifiers. +* @ops: Pointer to the operations structure for GPU SVM. +* @chunk_sizes: Pointer to the array of chunk sizes used in range allocation.
+ * Entries should be powers of 2 in descending order with last
+ * entry beingSZ_4K.
+ * @num_chunks: Number of chunks.
+ *
+ * This function initializes the GPU SVM.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+int drm_gpusvm_init (struct drm_gpusvm *gpusvm, + const char *name, struct drm_device *drm, + struct mm_struct *mm, void
*device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +4
-0+ unsigned long mm_start, unsigned long mm_range, + unsigned long notifier_size, + const struct drm_gpusvm_ops *ops, + const unsigned long *chunk_sizes, int
num_chunks)drivers/gpu/drm/drm_gpusvm.c new file +8
-0+{ +if (!ops->invalidate || !num_chunks) +return -EINVAL ; + + gpusvm->name = name; + gpusvm->drm = drm; + gpusvm->mm = mm; + gpusvm->device_private_page_owner =
device_private_page_owner;drivers/gpu/drm/drm_gpusvm.c new file +31
-0+ gpusvm->mm_start = mm_start; + gpusvm->mm_range = mm_range; + gpusvm->notifier_size = notifier_size; + gpusvm->ops = ops; + gpusvm->chunk_sizes = chunk_sizes; + gpusvm->num_chunks = num_chunks; + +mmgrab (mm); + gpusvm->root =RB_ROOT_CACHED ; +INIT_LIST_HEAD (&gpusvm->notifier_list); + +init_rwsem (&gpusvm->notifier_lock); + +fs_reclaim_acquire (GFP_KERNEL ); +might_lock (&gpusvm->notifier_lock); +fs_reclaim_release (GFP_KERNEL ); + +#ifdef CONFIG_LOCKDEP + gpusvm->lock_dep_map =NULL ; +#endif + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_init); + +/** +* drm_gpusvm_notifier_find() - Find GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM structure +* @fault_addr: Fault address +* +* This function finds the GPU SVM notifier associated with the fault address.
+ *
+ * Returns:
+ * Pointer to the GPU SVM notifier on success, NULL otherwise.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+static struct drm_gpusvm_notifier * +drm_gpusvm_notifier_find (struct drm_gpusvm *gpusvm, +unsigned long fault_addr) +{ + return notifier_iter_first(&gpusvm->root, fault_addr,
fault_addr+ 1);drivers/gpu/drm/drm_gpusvm.c new file +4
-0+} + +/** +* to_drm_gpusvm_notifier() - retrieve the container struct for a given rbtree node
+ * @node: a pointer to the rbtree node embedded within adrm_gpusvm_notifierstruct
+ *
+ * Returns: A pointer to the containingdrm_gpusvm_notifierstructure.
+ */
+static structdrm_gpusvm_notifier*to_drm_gpusvm_notifier(structrb_node*node)drivers/gpu/drm/drm_gpusvm.c new file +2
-0+{ + return container_of(node, struct drm_gpusvm_notifier,itree.rb);
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+} + +/** +* drm_gpusvm_notifier_insert() - Insert GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier structure +* +* This function inserts the GPU SVM notifier into the GPU SVM RB tree and list.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static void drm_gpusvm_notifier_insert (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier*notifier)
drivers/gpu/drm/drm_gpusvm.c new file +21
-0+{ +struct rb_node *node; +struct list_head *head; + +interval_tree_insert (¬ifier->itree, &gpusvm->root); + + node =rb_prev (¬ifier->itree.rb); +if (node) + head = &(to_drm_gpusvm_notifier (node))->entry; +else + head = &gpusvm->notifier_list; + +list_add (¬ifier->entry, head); +} + +/** +* drm_gpusvm_notifier_remove() - Remove GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM tructure +* @notifier: Pointer to the GPU SVM notifier structure +* +* This function removes the GPU SVM notifier from the GPU SVM RB tree and list.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static void drm_gpusvm_notifier_remove (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier*notifier)
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+{ +interval_tree_remove (¬ifier->itree, &gpusvm->root); +list_del (¬ifier->entry); +} + +/** +* drm_gpusvm_fini() - Finalize the GPU SVM. +* @gpusvm: Pointer to the GPU SVM structure. +* +* This function finalizes the GPU SVM by cleaning up any remaining ranges and
+ * notifiers, and dropping a reference to struct MM.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+void drm_gpusvm_fini (struct drm_gpusvm *gpusvm) +{ +struct drm_gpusvm_notifier *notifier, *next; + + drm_gpusvm_for_each_notifier_safe(notifier, next, gpusvm, 0,
LONG_MAX) {drivers/gpu/drm/drm_gpusvm.c new file +4
-0+struct drm_gpusvm_range *range, *__next; + +/* +* Remove notifier first to avoid racing with any invalidation
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+*/ +mmu_interval_notifier_remove (¬ifier->notifier); + notifier->flags.removed = true; + + drm_gpusvm_for_each_range_safe(range, __next,notifier, 0,
drivers/gpu/drm/drm_gpusvm.c new file +15
-0+ LONG_MAX) +drm_gpusvm_range_remove (gpusvm ,range ); + } + +mmdrop (gpusvm->mm); +WARN_ON (!RB_EMPTY_ROOT (&gpusvm->root.rb_root)); +} +EXPORT_SYMBOL_GPL (drm_gpusvm_fini); + +/** +* drm_gpusvm_notifier_alloc() - Allocate GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM structure +* @fault_addr: Fault address +* +* This function allocates and initializes the GPU SVM notifier structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM notifier on success,ERR_PTR()on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static struct drm_gpusvm_notifier * +drm_gpusvm_notifier_alloc(struct drm_gpusvm *gpusvm, unsigned long
fault_addr)drivers/gpu/drm/drm_gpusvm.c new file +13
-0+{ +struct drm_gpusvm_notifier *notifier; + +if (gpusvm->ops->notifier_alloc) + notifier = gpusvm->ops->notifier_alloc (); +else + notifier =kzalloc (sizeof (*notifier),GFP_KERNEL ); + +if (!notifier) +return ERR_PTR (-ENOMEM ); + + notifier->gpusvm = gpusvm; + notifier->itree.start = ALIGN_DOWN(fault_addr, gpusvm-Quoted 1 line
notifier_size);drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ notifier->itree.last = ALIGN(fault_addr + 1, gpusvm-Quoted 1 line
notifier_size) - 1;drivers/gpu/drm/drm_gpusvm.c new file +16
-0+INIT_LIST_HEAD (¬ifier->entry); + notifier->root =RB_ROOT_CACHED ; +INIT_LIST_HEAD (¬ifier->range_list); + +return notifier; +} + +/** +* drm_gpusvm_notifier_free() - Free GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier structure +* +* This function frees the GPU SVM notifier structure. +*/ +static void drm_gpusvm_notifier_free (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier*notifier)
drivers/gpu/drm/drm_gpusvm.c new file +11
-0+{ +WARN_ON (!RB_EMPTY_ROOT (¬ifier->root.rb_root)); + +if (gpusvm->ops->notifier_free) + gpusvm->ops->notifier_free (notifier); +else +kfree (notifier); +} + +/** +* to_drm_gpusvm_range() - retrieve the container struct for a given rbtree node
+ * @node: a pointer to the rbtree node embedded within adrm_gpusvm_rangestruct
+ *
+ * Returns: A pointer to the containingdrm_gpusvm_rangestructure.
+ */
+static structdrm_gpusvm_range*to_drm_gpusvm_range(structrb_node*node)drivers/gpu/drm/drm_gpusvm.c new file +2
-0+{ + return container_of(node, struct drm_gpusvm_range,itree.rb);
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+} + +/** +* drm_gpusvm_range_insert() - Insert GPU SVM range +* @notifier: Pointer to the GPU SVM notifier structure +* @range: Pointer to the GPU SVM range structure +* +* This function inserts the GPU SVM range into the notifier RB tree and list.
+ */
+static void drm_gpusvm_range_insert(structdrm_gpusvm_notifier*notifier,drivers/gpu/drm/drm_gpusvm.c new file +24
-0+ struct drm_gpusvm_range *range) +{ +struct rb_node *node; +struct list_head *head; + +drm_gpusvm_notifier_lock (notifier->gpusvm); +interval_tree_insert (&range->itree, ¬ifier->root); + + node =rb_prev (&range->itree.rb); +if (node) + head = &(to_drm_gpusvm_range (node))->entry; +else + head = ¬ifier->range_list; + +list_add (&range->entry, head); +drm_gpusvm_notifier_unlock (notifier->gpusvm); +} + +/** +* __drm_gpusvm_range_remove() - Remove GPU SVM range +* @notifier: Pointer to the GPU SVM notifier structure +* @range: Pointer to the GPU SVM range structure +* +* This macro removes the GPU SVM range from the notifier RB tree and list.
+ */
+static void __drm_gpusvm_range_remove(structdrm_gpusvm_notifier*notifier,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct drm_gpusvm_range*range)
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+{ +interval_tree_remove (&range->itree, ¬ifier->root); +list_del (&range->entry); +} + +/** +* drm_gpusvm_range_alloc() - Allocate GPU SVM range +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier structure +* @fault_addr: Fault address +* @chunk_size: Chunk size +* @migrate_devmem: Flag indicating whether to migrate device memory +* +* This function allocates and initializes the GPU SVM range structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM range on success,ERR_PTR()on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+static struct drm_gpusvm_range * +drm_gpusvm_range_alloc (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + unsigned long fault_addr, unsigned long
chunk_size,drivers/gpu/drm/drm_gpusvm.c new file +32
-0+ bool migrate_devmem) +{ +struct drm_gpusvm_range *range; + +if (gpusvm->ops->range_alloc) + range = gpusvm->ops->range_alloc (gpusvm); +else + range =kzalloc (sizeof (*range),GFP_KERNEL ); + +if (!range) +return ERR_PTR (-ENOMEM ); + +kref_init (&range->refcount); + range->gpusvm = gpusvm; + range->notifier = notifier; + range->itree.start =ALIGN_DOWN (fault_addr, chunk_size); + range->itree.last =ALIGN (fault_addr +1 , chunk_size) -1 ; +INIT_LIST_HEAD (&range->entry); + range->notifier_seq =LONG_MAX ; + range->flags.migrate_devmem = migrate_devmem ?1 :0 ; + +return range; +} + +/** +* drm_gpusvm_check_pages() - Check pages +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier structure +* @start: Start address +* @end: End address +* +* Check if pages between start and end have been faulted in on the CPU. Use to
+ * prevent migration of pages without CPU backing store.
+ *
+ * Returns:
+ * True if pages have been faulted into CPU, False otherwise
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static bool drm_gpusvm_check_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier*notifier,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long start, unsignedlong end)
drivers/gpu/drm/drm_gpusvm.c new file +7
-0+{ + struct hmm_range hmm_range = { + .default_flags = 0, + .notifier = ¬ifier->notifier, + .start = start, + .end = end, + .dev_private_owner = gpusvm-Quoted 1 line
device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ }; +unsigned long timeout = + jiffies +
msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);drivers/gpu/drm/drm_gpusvm.c new file +11
-0+unsigned long *pfns; +unsigned long npages =npages_in_range (start, end); +int err, i; + +mmap_assert_locked (gpusvm->mm); + + pfns =kvmalloc_array (npages,sizeof (*pfns),GFP_KERNEL ); +if (!pfns) +return false; + + hmm_range.notifier_seq = mmu_interval_read_begin(¬ifier-Quoted 1 line
notifier);
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+ hmm_range.hmm_pfns = pfns; + +while (true) { + err =hmm_range_fault (&hmm_range); +if (err == -EBUSY ) { +if (time_after (jiffies, timeout)) +break ; + + hmm_range.notifier_seq = + mmu_interval_read_begin(¬ifier-Quoted 1 line
notifier);
drivers/gpu/drm/drm_gpusvm.c new file +22
-0+continue ; + } +break ; + } +if (err) + goto err_free; + +for (i =0 ; i < npages;) { +if (!(pfns[i] &HMM_PFN_VALID )) { + err = -EFAULT ; + goto err_free; + } + i +=0x1 <<hmm_pfn_to_map_order (pfns[i]); + } + +err_free: +kvfree (pfns); +return err ? false : true; +} + +/** +* drm_gpusvm_range_chunk_size() - Determine chunk size for GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @vas: Pointer to the virtual memory area structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @check_pages_threshold: Check CPU pages for present threshold
+ *
+ * This function determines the chunk size for the GPU SVM range based on the
+ * fault address, GPU SVM chunk sizes, existing GPU SVM ranges, and the virtual
+ * memory area boundaries.
+ *
+ * Returns:
+ * Chunk size on success,LONG_MAXon failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +15
-0+static unsigned long +drm_gpusvm_range_chunk_size (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + struct vm_area_struct *vas, + unsigned long fault_addr, + unsigned long gpuva_start, + unsigned long gpuva_end, + unsigned long check_pages_threshold) +{ +unsigned long start, end; +int i =0 ; + +retry: +for (; i < gpusvm->num_chunks; ++i) { + start = ALIGN_DOWN(fault_addr, gpusvm-Quoted 1 line
chunk_sizes[i]);
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+ end =ALIGN (fault_addr +1 , gpusvm->chunk_sizes[i]); + +if (start >= vas->vm_start && end <= vas->vm_end && + start >= notifier->itree.start && + end <= notifier->itree.last + 1 && + start >= gpuva_start && end <= gpuva_end) +break ; + } + +if (i == gpusvm->num_chunks) +return LONG_MAX ; + +/* +* If allocation more than page, ensure not to overlap with existing
drivers/gpu/drm/drm_gpusvm.c new file +13
-0+* ranges. +*/ +if (end - start !=SZ_4K ) { +struct drm_gpusvm_range *range; + + range =drm_gpusvm_range_find (notifier, start, end); +if (range) { + ++i; + goto retry; + } + +/* +* XXX: Only create range on pages CPU has faulted in. Without
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* this check, or prefault, on BMG '
xe_exec_system_allocator--rdrivers/gpu/drm/drm_gpusvm.c new file +1
-0+* process-many-malloc' fails. In the failure case, each process
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* mallocs 16k but the CPU VMA is ~128k which results in 64k SVM
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* ranges. When migrating the SVM ranges, some processes fail in
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* drm_gpusvm_migrate_to_devmem with 'migrate.cpages != npages'
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* and then upon drm_gpusvm_range_get_pages device pages from
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* other processes are collected + faulted in which creates all
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* sorts of problems. Unsure exactly how this happening, also
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* problem goes away if 'xe_exec_system_allocator -- r
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* process-many-malloc' mallocs at least 64k at a time.
drivers/gpu/drm/drm_gpusvm.c new file +3
-0+*/ +if (end - start <= check_pages_threshold && + !drm_gpusvm_check_pages(gpusvm, notifier, start,end)) {
drivers/gpu/drm/drm_gpusvm.c new file +17
-0+ ++i; + goto retry; + } + } + +return end - start; +} + +/** +* drm_gpusvm_range_find_or_insert() - Find or insert GPU SVM range +* @gpusvm: Pointer to the GPU SVM structure +* @fault_addr: Fault address +* @gpuva_start: Start address of GPUVA which mirrors CPU +* @gpuva_end: End address of GPUVA which mirrors CPU +* @ctx: GPU SVM context +* +* This function finds or inserts a newly allocated a GPU SVM range based on the
+ * fault address. Caller must hold a lock to protect range lookup and insertion.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range on success,ERR_PTR()on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +28
-0+struct drm_gpusvm_range * +drm_gpusvm_range_find_or_insert (struct drm_gpusvm *gpusvm, +unsigned long fault_addr, +unsigned long gpuva_start, +unsigned long gpuva_end, +const struct drm_gpusvm_ctx *ctx) +{ +struct drm_gpusvm_notifier *notifier; +struct drm_gpusvm_range *range; +struct mm_struct *mm = gpusvm->mm; +struct vm_area_struct *vas; +bool notifier_alloc = false; +unsigned long chunk_size; +int err; +bool migrate_devmem; + +drm_gpusvm_driver_lock_held (gpusvm); + +if (fault_addr < gpusvm->mm_start || + fault_addr > gpusvm->mm_start + gpusvm->mm_range) +return ERR_PTR (-EINVAL ); + +if (!mmget_not_zero (mm)) +return ERR_PTR (-EFAULT ); + + notifier =drm_gpusvm_notifier_find (gpusvm, fault_addr); +if (!notifier) { + notifier = drm_gpusvm_notifier_alloc(gpusvm,
fault_addr);drivers/gpu/drm/drm_gpusvm.c new file +6
-0+if (IS_ERR (notifier)) { + err =PTR_ERR (notifier); + goto err_mmunlock; + } + notifier_alloc = true; + err = mmu_interval_notifier_insert(¬ifier-Quoted 1 line
notifier,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ mm, notifier-Quoted 1 line
itree.start,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ notifier-Quoted 1 line
itree.last -
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ notifier-Quoted 1 line
itree.start + 1,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+
&drm_gpusvm_notifier_ops);drivers/gpu/drm/drm_gpusvm.c new file +18
-0+if (err) + goto err_notifier; + } + +mmap_read_lock (mm); + + vas =vma_lookup (mm, fault_addr); +if (!vas) { + err = -ENOENT ; + goto err_notifier_remove; + } + +if (!ctx->read_only && !(vas->vm_flags &VM_WRITE )) { + err = -EPERM ; + goto err_notifier_remove; + } + + range = drm_gpusvm_range_find(notifier, fault_addr,
fault_addr+ 1);drivers/gpu/drm/drm_gpusvm.c new file +4
-0+if (range) + goto out_mmunlock; +/* +* XXX: Short-circuiting migration based on migrate_vma_* current
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* limitations. If/when migrate_vma_* add more support, this logic will
drivers/gpu/drm/drm_gpusvm.c new file +6
-0+* have to change. +*/ + migrate_devmem = ctx->devmem_possible && +vma_is_anonymous (vas) && !is_vm_hugetlb_page (vas); + + chunk_size = drm_gpusvm_range_chunk_size(gpusvm, notifier,vas,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ fault_addr,
gpuva_start,drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ gpuva_end, + ctx-Quoted 1 line
check_pages_threshold);drivers/gpu/drm/drm_gpusvm.c new file +6
-0+if (chunk_size ==LONG_MAX ) { + err = -EINVAL ; + goto err_notifier_remove; + } + + range = drm_gpusvm_range_alloc(gpusvm, notifier, fault_addr,
chunk_size,drivers/gpu/drm/drm_gpusvm.c new file +31
-0+ migrate_devmem); +if (IS_ERR (range)) { + err =PTR_ERR (range); + goto err_notifier_remove; + } + +drm_gpusvm_range_insert (notifier, range); +if (notifier_alloc) +drm_gpusvm_notifier_insert (gpusvm, notifier); + +out_mmunlock: +mmap_read_unlock (mm); +mmput (mm); + +return range; + +err_notifier_remove: +mmap_read_unlock (mm); +if (notifier_alloc) +mmu_interval_notifier_remove (¬ifier->notifier); +err_notifier: +if (notifier_alloc) +drm_gpusvm_notifier_free (gpusvm, notifier); +err_mmunlock: +mmput (mm); +return ERR_PTR (err); +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_find_or_insert); + +/** +* __drm_gpusvm_range_unmap_pages() - Unmap pages associated with a GPU SVM range (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @npages: Number of pages to unmap
+ *
+ * This function unmap pages associated with a GPU SVM range. Assumes and
+ * asserts correct locking is in place when called.
+ */
+static void __drm_gpusvm_range_unmap_pages(structdrm_gpusvm*gpusvm,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct drm_gpusvm_range*range,
drivers/gpu/drm/drm_gpusvm.c new file +11
-0+ unsigned long npages) +{ +unsigned long i, j; +struct drm_pagemap *dpagemap = range->dpagemap; +struct device *dev = gpusvm->drm->dev; + +lockdep_assert_held (&gpusvm->notifier_lock); + +if (range->flags.has_dma_mapping) { +for (i =0 , j =0 ; i < npages; j++) { + struct drm_pagemap_dma_addr *addr = &range-Quoted 1 line
dma_addr[j];
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +if (addr->proto ==DRM_INTERCONNECT_SYSTEM ) +dma_unmap_page (dev, + addr->addr, + PAGE_SIZE << addr-Quoted 1 line
order,
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ addr->dir); + else if (dpagemap && dpagemap->ops-Quoted 1 line
unmap_dma)drivers/gpu/drm/drm_gpusvm.c new file +13
-0+ dpagemap->ops->unmap_dma (dpagemap, + dev, + *addr); + i +=1 << addr->order; + } + range->flags.has_devmem_pages = false; + range->flags.has_dma_mapping = false; + range->dpagemap =NULL ; + } +} + +/** +* drm_gpusvm_range_free_pages() - Free pages associated with a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function frees the dma address array associated with a GPU SVM range.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static void drm_gpusvm_range_free_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range*range)
drivers/gpu/drm/drm_gpusvm.c new file +15
-0+{ +lockdep_assert_held (&gpusvm->notifier_lock); + +if (range->dma_addr) { +kvfree (range->dma_addr); + range->dma_addr =NULL ; + } +} + +/** +* drm_gpusvm_range_remove() - Remove GPU SVM range +* @gpusvm: Pointer to the GPU SVM structure +* @range: Pointer to the GPU SVM range to be removed +* +* This function removes the specified GPU SVM range and also removes the parent
+ * GPU SVM notifier if no more ranges remain in the notifier. The caller must
+ * hold a lock to protect range and notifier removal.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+void drm_gpusvm_range_remove (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range) +{ + unsigned long npages = npages_in_range(range->itree.start, + range->itree.last +1);
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+struct drm_gpusvm_notifier *notifier; + +drm_gpusvm_driver_lock_held (gpusvm); + + notifier = drm_gpusvm_notifier_find(gpusvm, range-Quoted 1 line
itree.start);
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+if (WARN_ON_ONCE (!notifier)) +return ; + +drm_gpusvm_notifier_lock (gpusvm); +__drm_gpusvm_range_unmap_pages (gpusvm, range, npages); +drm_gpusvm_range_free_pages (gpusvm, range); +__drm_gpusvm_range_remove (notifier, range); +drm_gpusvm_notifier_unlock (gpusvm); + +drm_gpusvm_range_put (range); + +if (RB_EMPTY_ROOT (¬ifier->root.rb_root)) { +if (!notifier->flags.removed) + mmu_interval_notifier_remove(¬ifier-Quoted 1 line
notifier);
drivers/gpu/drm/drm_gpusvm.c new file +11
-0+drm_gpusvm_notifier_remove (gpusvm, notifier); +drm_gpusvm_notifier_free (gpusvm, notifier); + } +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_remove); + +/** +* drm_gpusvm_range_get() - Get a reference to GPU SVM range +* @range: Pointer to the GPU SVM range +* +* This function increments the reference count of the specified GPU SVM range.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range.
+ */drivers/gpu/drm/drm_gpusvm.c new file +12
-0+struct drm_gpusvm_range * +drm_gpusvm_range_get (struct drm_gpusvm_range *range) +{ +kref_get (&range->refcount); + +return range; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_get); + +/** +* drm_gpusvm_range_destroy() - Destroy GPU SVM range +* @refcount: Pointer to the reference counter embedded in the GPU SVM range
+ *
+ * This function destroys the specified GPU SVM range when its reference count
+ * reaches zero. If a custom range-free function is provided, it is invoked to
+ * free the range; otherwise, the range is deallocated usingkfree().
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+static void drm_gpusvm_range_destroy (struct kref *refcount) +{ +struct drm_gpusvm_range *range = + container_of(refcount, struct drm_gpusvm_range,refcount);
drivers/gpu/drm/drm_gpusvm.c new file +13
-0+struct drm_gpusvm *gpusvm = range->gpusvm; + +if (gpusvm->ops->range_free) + gpusvm->ops->range_free (range); +else +kfree (range); +} + +/** +* drm_gpusvm_range_put() - Put a reference to GPU SVM range +* @range: Pointer to the GPU SVM range +* +* This function decrements the reference count of the specified GPU SVM range
+ * and frees it when the count reaches zero.
+ */drivers/gpu/drm/drm_gpusvm.c new file +12
-0+void drm_gpusvm_range_put (struct drm_gpusvm_range *range) +{ +kref_put (&range->refcount, drm_gpusvm_range_destroy); +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_put); + +/** +* drm_gpusvm_range_pages_valid() - GPU SVM range pages valid +* @gpusvm: Pointer to the GPU SVM structure +* @range: Pointer to the GPU SVM range structure +* +* This function determines if a GPU SVM range pages are valid. Expected be
+ * called holdinggpusvm->notifier_lockand as the last step before committing a
+ * GPU binding. This is akin to a notifier seqno check in the HMM documentation
+ * but due to wider notifiers (i.e., notifiers which span multiple ranges) this
+ * function is required for finer grained checking (i.e., per range) if pages
+ * are valid.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */drivers/gpu/drm/drm_gpusvm.c new file +6
-0+bool drm_gpusvm_range_pages_valid (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range) +{ +lockdep_assert_held (&gpusvm->notifier_lock); + + return range->flags.has_devmem_pages || range-Quoted 1 line
flags.has_dma_mapping;
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_pages_valid); + +/** +* drm_gpusvm_range_pages_valid_unlocked() - GPU SVM range pages valid unlocked
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid. Expected be
+ * called without holdinggpusvm->notifier_lock.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+static bool +drm_gpusvm_range_pages_valid_unlocked (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range*range)
drivers/gpu/drm/drm_gpusvm.c new file +22
-0+{ +bool pages_valid; + +if (!range->dma_addr) +return false; + +drm_gpusvm_notifier_lock (gpusvm); + pages_valid =drm_gpusvm_range_pages_valid (gpusvm, range); +if (!pages_valid) +drm_gpusvm_range_free_pages (gpusvm, range); +drm_gpusvm_notifier_unlock (gpusvm); + +return pages_valid; +} + +/** +* drm_gpusvm_range_get_pages() - Get pages for a GPU SVM range +* @gpusvm: Pointer to the GPU SVM structure +* @range: Pointer to the GPU SVM range structure +* @ctx: GPU SVM context +* +* This function gets pages for a GPU SVM range and ensures they are mapped for
+ * DMA access.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+int drm_gpusvm_range_get_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx) +{ + struct mmu_interval_notifier *notifier = &range->notifier-Quoted 1 line
notifier;
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+struct hmm_range hmm_range = { + .default_flags = HMM_PFN_REQ_FAULT | (ctx->read_only? 0 :
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ HMM_PFN_REQ_WRITE), + .notifier = notifier, + .start = range->itree.start, + .end = range->itree.last + 1, + .dev_private_owner = gpusvm-Quoted 1 line
device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ }; +struct mm_struct *mm = gpusvm->mm; +struct drm_gpusvm_zdd *zdd; +unsigned long timeout = + jiffies +
msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);drivers/gpu/drm/drm_gpusvm.c new file +3
-0+unsigned long i, j; + unsigned long npages = npages_in_range(range->itree.start, + range->itree.last +1);
drivers/gpu/drm/drm_gpusvm.c new file +58
-0+unsigned long num_dma_mapped; +unsigned int order =0 ; +unsigned long *pfns; +struct page **pages; +int err =0 ; +struct dev_pagemap *pagemap; +struct drm_pagemap *dpagemap; + +retry: + hmm_range.notifier_seq =mmu_interval_read_begin (notifier); +if (drm_gpusvm_range_pages_valid_unlocked (gpusvm, range)) + goto set_seqno; + + pfns =kvmalloc_array (npages,sizeof (*pfns),GFP_KERNEL ); +if (!pfns) +return -ENOMEM ; + +if (!mmget_not_zero (mm)) { + err = -EFAULT ; + goto err_free; + } + + hmm_range.hmm_pfns = pfns; +while (true) { +mmap_read_lock (mm); + err =hmm_range_fault (&hmm_range); +mmap_read_unlock (mm); + +if (err == -EBUSY ) { +if (time_after (jiffies, timeout)) +break ; + + hmm_range.notifier_seq = +mmu_interval_read_begin (notifier); +continue ; + } +break ; + } +mmput (mm); +if (err) + goto err_free; + + pages = (struct page **)pfns; +map_pages: +/* +* Perform all dma mappings under the notifier lock to not +* access freed pages. A notifier will either block on +* the notifier lock or unmap dma. +*/ +drm_gpusvm_notifier_lock (gpusvm); + +if (range->flags.unmapped) { +drm_gpusvm_notifier_unlock (gpusvm); + err = -EFAULT ; + goto err_free; + } + + if (mmu_interval_read_retry(notifier,hmm_range.notifier_seq)) {
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+drm_gpusvm_notifier_unlock (gpusvm); +kvfree (pfns); + goto retry; + } + +if (!range->dma_addr) { +/* Unlock and restart mapping to allocate memory. */ +drm_gpusvm_notifier_unlock (gpusvm); + range->dma_addr = kvmalloc_array(npages, + sizeof(*range-Quoted 1 line
dma_addr),drivers/gpu/drm/drm_gpusvm.c new file +17
-0+ GFP_KERNEL); +if (!range->dma_addr) { + err = -ENOMEM ; + goto err_free; + } + goto map_pages; + } + + zdd =NULL ; + num_dma_mapped =0 ; +for (i =0 , j =0 ; i < npages; ++j) { +struct page *page =hmm_pfn_to_page (pfns[i]); + + order =hmm_pfn_to_map_order (pfns[i]); +if (is_device_private_page (page) || + is_device_coherent_page(page)) { +if (zdd != page->zone_device_data && i >0 ){
drivers/gpu/drm/drm_gpusvm.c line 1395
Quoted 3 lines
drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ err = -EOPNOTSUPP ; + goto err_unmap; + }
I wonder, in a step towards make the get_pages() more permissive, Could one drop the above check that all device pages are using the same allocation and just verify that they are from the same pagemap, like below?
Quoted 597 lines
drivers/gpu/drm/drm_gpusvm.c new file +9
-0+ zdd = page->zone_device_data; +if (pagemap != page->pgmap) { +if (i >0 ) { + err = -EOPNOTSUPP ; + goto err_unmap; + } + + pagemap = page->pgmap; + dpagemap = zdd->devmem_allocation-Quoted 1 line
dpagemap;
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ if (drm_WARN_ON(gpusvm->drm,!dpagemap)) {
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+/* +* Raced. This is not supposed to happen
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* since hmm_range_fault() should've migrated
drivers/gpu/drm/drm_gpusvm.c new file +9
-0+* this page to system. +*/ + err = -EAGAIN ; + goto err_unmap; + } + } + range->dma_addr[j] = + dpagemap->ops->map_dma(dpagemap, + gpusvm->drm-Quoted 1 line
dev,
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ page, order, +
DMA_BIDIRECTIONAL);drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ if (dma_mapping_error(gpusvm->drm->dev, + range-Quoted 1 line
dma_addr[j].addr)) {
drivers/gpu/drm/drm_gpusvm.c new file +18
-0+ err = -EFAULT ; + goto err_unmap; + } + + pages[i] = page; + }else { +dma_addr_t addr; + +if (is_zone_device_page (page) || zdd) { + err = -EOPNOTSUPP ; + goto err_unmap; + } + + addr =dma_map_page (gpusvm->drm->dev, + page, 0, + PAGE_SIZE << order, + DMA_BIDIRECTIONAL); + if (dma_mapping_error(gpusvm->drm->dev,addr)) {
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ err = -EFAULT ; + goto err_unmap; + } + + range->dma_addr[j] =
drm_pagemap_dma_addr_encodedrivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (addr, DRM_INTERCONNECT_SYSTEM,order,
drivers/gpu/drm/drm_gpusvm.c new file +21
-0+ DMA_BIDIRECTIONAL); + } + i +=1 << order; + num_dma_mapped = i; + } + + range->flags.has_dma_mapping = true; +if (zdd) { + range->flags.has_devmem_pages = true; + range->dpagemap = dpagemap; + } + +drm_gpusvm_notifier_unlock (gpusvm); +kvfree (pfns); +set_seqno: + range->notifier_seq = hmm_range.notifier_seq; + +return 0 ; + +err_unmap: + __drm_gpusvm_range_unmap_pages(gpusvm, range,
num_dma_mapped);drivers/gpu/drm/drm_gpusvm.c new file +11
-0+drm_gpusvm_notifier_unlock (gpusvm); +err_free: +kvfree (pfns); +if (err == -EAGAIN ) + goto retry; +return err; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_get_pages); + +/** +* drm_gpusvm_range_unmap_pages() - Unmap pages associated with a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed thatgpusvm->notifier_lockis held in write mode; if it
+ * is clear, it acquiresgpusvm->notifier_lockin read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops-Quoted 1 line
invalidate for IOMMU
+ * security model.
+ */drivers/gpu/drm/drm_gpusvm.c new file +6
-0+void drm_gpusvm_range_unmap_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx) +{ + unsigned long npages = npages_in_range(range->itree.start, + range->itree.last +1);
drivers/gpu/drm/drm_gpusvm.c new file +33
-0+ +if (ctx->in_notifier) +lockdep_assert_held_write (&gpusvm->notifier_lock); +else +drm_gpusvm_notifier_lock (gpusvm); + +__drm_gpusvm_range_unmap_pages (gpusvm, range, npages); + +if (!ctx->in_notifier) +drm_gpusvm_notifier_unlock (gpusvm); +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_unmap_pages); + +/** +* drm_gpusvm_migration_unlock_put_page() - Put a migration page +* @page: Pointer to the page to put +* +* This function unlocks and puts a page. +*/ +static void drm_gpusvm_migration_unlock_put_page (struct page *page) +{ +unlock_page (page); +put_page (page); +} + +/** +* drm_gpusvm_migration_unlock_put_pages() - Put migration pages +* @npages: Number of pages +* @migrate_pfn: Array of migrate page frame numbers +* +* This function unlocks and puts an array of pages. +*/ +static void drm_gpusvm_migration_unlock_put_pages(unsigned longnpages,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long
*migrate_pfn)drivers/gpu/drm/drm_gpusvm.c new file +17
-0+{ +unsigned long i; + +for (i =0 ; i < npages; ++i) { +struct page *page; + +if (!migrate_pfn[i]) +continue ; + + page =migrate_pfn_to_page (migrate_pfn[i]); +drm_gpusvm_migration_unlock_put_page (page); + migrate_pfn[i] =0 ; + } +} + +/** +* drm_gpusvm_get_devmem_page() - Get a reference to a device memory page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU SVM zone
+ * device data and initializes it for zone device usage.
+ */drivers/gpu/drm/drm_gpusvm.c new file +9
-0+static void drm_gpusvm_get_devmem_page (struct page *page, + struct drm_gpusvm_zdd *zdd) +{ + page->zone_device_data =drm_gpusvm_zdd_get (zdd); +zone_device_page_init (page); +} + +/** +* drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g.,DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Returns: 0 on success,-EFAULTif an error occurs during mapping.
+ */drivers/gpu/drm/drm_gpusvm.c new file +10
-0+static int drm_gpusvm_migrate_map_pages (struct device *dev, +dma_addr_t *dma_addr, +unsigned long *migrate_pfn, +unsigned long npages, +enum dma_data_direction dir) +{ +unsigned long i; + +for (i =0 ; i < npages; ++i) { + struct page *page =
migrate_pfn_to_page(migrate_pfn[i]);drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ +if (!page) +continue ; + +if (WARN_ON_ONCE (is_zone_device_page (page))) +return -EFAULT ; + + dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE,dir);
drivers/gpu/drm/drm_gpusvm.c new file +9
-0+if (dma_mapping_error (dev, dma_addr[i])) +return -EFAULT ; + } + +return 0 ; +} + +/** +* drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g.,DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+static void drm_gpusvm_migrate_unmap_pages (struct device *dev, + dma_addr_t *dma_addr, + unsigned long npages, + enum dma_data_directiondir)
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+{ + unsigned long i; + + for (i = 0; i < npages; ++i) { + if (!dma_addr[i] || dma_mapping_error(dev,dma_addr[i]))
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+continue ; + +dma_unmap_page (dev, dma_addr[i],PAGE_SIZE , dir); + } +} + +/** +* drm_gpusvm_migrate_to_devmem() - Migrate GPU SVM range to device memory
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @devmem_allocation: Pointer to the device memory allocation. The caller
+ * should hold a reference to the device memory allocation,
+ * which should be dropped via ops-Quoted 1 line
devmem_releaseor upon+ * the failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to device memory. It performs the
+ * necessary setup and invokes the driver-specific operations for migration to
+ * device memory. Upon successful return, @devmem_allocation can safely reference @range
+ * untilops->devmem_releaseis called which only upon successful return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+int drm_gpusvm_migrate_to_devmem (struct drm_gpusvm *gpusvm, +struct drm_gpusvm_range *range, + struct drm_gpusvm_devmem
*devmem_allocation,drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ const struct drm_gpusvm_ctx *ctx) +{ + const struct drm_gpusvm_devmem_ops *ops = devmem_allocation-Quoted 1 line
ops;
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long start = range->itree.start, end = range-Quoted 1 line
itree.last + 1;
drivers/gpu/drm/drm_gpusvm.c new file +45
-0+struct migrate_vma migrate = { + .start = start, + .end = end, + .pgmap_owner = gpusvm->device_private_page_owner, + .flags =MIGRATE_VMA_SELECT_SYSTEM , + }; +struct mm_struct *mm = gpusvm->mm; +unsigned long i, npages =npages_in_range (start, end); +struct vm_area_struct *vas; +struct drm_gpusvm_zdd *zdd =NULL ; +struct page **pages; +dma_addr_t *dma_addr; +void *buf; +int err; + +if (!range->flags.migrate_devmem) +return -EINVAL ; + +if (!ops->populate_devmem_pfn || !ops->copy_to_devmem || + !ops->copy_to_ram) +return -EOPNOTSUPP ; + +if (!mmget_not_zero (mm)) { + err = -EFAULT ; + goto err_out; + } +mmap_read_lock (mm); + + vas =vma_lookup (mm, start); +if (!vas) { + err = -ENOENT ; + goto err_mmunlock; + } + +if (end > vas->vm_end || start < vas->vm_start) { + err = -EINVAL ; + goto err_mmunlock; + } + +if (!vma_is_anonymous (vas)) { + err = -EBUSY ; + goto err_mmunlock; + } + + buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +
sizeof(*dma_addr)+drivers/gpu/drm/drm_gpusvm.c new file +7
-0+ sizeof(*pages), GFP_KERNEL); +if (!buf) { + err = -ENOMEM ; + goto err_mmunlock; + } + dma_addr = buf + (2 *sizeof (*migrate.src) * npages); + pages = buf + (2 *sizeof (*migrate.src) +sizeof (*dma_addr))* npages;
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ + zdd = drm_gpusvm_zdd_alloc(gpusvm-Quoted 1 line
device_private_page_owner);drivers/gpu/drm/drm_gpusvm.c new file +24
-0+if (!zdd) { + err = -ENOMEM ; + goto err_free; + } + + migrate.vma = vas; + migrate.src = buf; + migrate.dst = migrate.src + npages; + + err =migrate_vma_setup (&migrate); +if (err) + goto err_free; + +if (!migrate.cpages) { + err = -EFAULT ; + goto err_free; + } + +if (migrate.cpages != npages) { + err = -EBUSY ; + goto err_finalize; + } + + err = ops->populate_devmem_pfn(devmem_allocation, npages,migrate.dst);
drivers/gpu/drm/drm_gpusvm.c new file +4
-0+if (err) + goto err_finalize; + + err = drm_gpusvm_migrate_map_pages(devmem_allocation->dev,
dma_addr,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ migrate.src, npages,
DMA_TO_DEVICE);drivers/gpu/drm/drm_gpusvm.c new file +17
-0+if (err) + goto err_finalize; + +for (i =0 ; i < npages; ++i) { +struct page *page =pfn_to_page (migrate.dst[i]); + + pages[i] = page; + migrate.dst[i] =migrate_pfn (migrate.dst[i]); +drm_gpusvm_get_devmem_page (page, zdd); + } + + err = ops->copy_to_devmem (pages, dma_addr, npages); +if (err) + goto err_finalize; + +/* Upon success bind devmem allocation to range and zdd */ + zdd->devmem_allocation = devmem_allocation;/* Owns ref */
drivers/gpu/drm/drm_gpusvm.c new file +4
-0+ +err_finalize: +if (err) + drm_gpusvm_migration_unlock_put_pages(npages,migrate.dst);
drivers/gpu/drm/drm_gpusvm.c new file +3
-0+migrate_vma_pages (&migrate); +migrate_vma_finalize (&migrate); + drm_gpusvm_migrate_unmap_pages(devmem_allocation->dev,
dma_addr, npages,drivers/gpu/drm/drm_gpusvm.c new file +15
-0+ DMA_TO_DEVICE); +err_free: +if (zdd) +drm_gpusvm_zdd_put (zdd); +kvfree (buf); +err_mmunlock: +mmap_read_unlock (mm); +mmput (mm); +err_out: +return err; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_migrate_to_devmem); + +/** +* drm_gpusvm_migrate_populate_ram_pfn() - Populate RAM PFNs for a VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @fault_page: Fault page
+ * @npages: Number of pages to populate
+ * @mpages: Number of pages to migrate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the RAM migrate page frame numbers (PFNs) for the
+ * specified VM area structure. It allocates and locks pages in the VM area for
+ * RAM usage. If vas is non-NULL usealloc_page_vmafor allocation, if NULL use
+ *alloc_pagefor allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_ram_pfn(structvm_area_struct*vas,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct page
*fault_page,drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ unsigned long npages, + unsigned long*mpages,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long
*src_mpfn,drivers/gpu/drm/drm_gpusvm.c new file +23
-0+ unsigned long *mpfn, + unsigned long addr) +{ +unsigned long i; + +for (i =0 ; i < npages; ++i, addr +=PAGE_SIZE ) { +struct page *page, *src_page; + +if (!(src_mpfn[i] &MIGRATE_PFN_MIGRATE )) +continue ; + + src_page =migrate_pfn_to_page (src_mpfn[i]); +if (!src_page) +continue ; + +if (fault_page) { +if (src_page->zone_device_data != + fault_page->zone_device_data) +continue ; + } + +if (vas) + page = alloc_page_vma(GFP_HIGHUSER, vas,addr);
drivers/gpu/drm/drm_gpusvm.c new file +39
-0+else + page =alloc_page (GFP_HIGHUSER ); + +if (!page) + goto free_pages; + + mpfn[i] =migrate_pfn (page_to_pfn (page)); + } + +for (i =0 ; i < npages; ++i) { +struct page *page =migrate_pfn_to_page (mpfn[i]); + +if (!page) +continue ; + +WARN_ON_ONCE (!trylock_page (page)); + ++*mpages; + } + +return 0 ; + +free_pages: +for (i =0 ; i < npages; ++i) { +struct page *page =migrate_pfn_to_page (mpfn[i]); + +if (!page) +continue ; + +put_page (page); + mpfn[i] =0 ; + } +return -ENOMEM ; +} + +/** +* drm_gpusvm_evict_to_ram() - Evict GPU SVM range to RAM +* @devmem_allocation: Pointer to the device memory allocation +* +* Similar to __drm_gpusvm_migrate_to_ram but does not require mmap lock and
+ * migration done via migrate_device_* functions.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+intdrm_gpusvm_evict_to_ram(struct drm_gpusvm_devmem *devmem_allocation)drivers/gpu/drm/drm_gpusvm.c new file +2
-0+{ + const struct drm_gpusvm_devmem_ops *ops = devmem_allocation-Quoted 1 line
ops;
drivers/gpu/drm/drm_gpusvm.c new file +25
-0+unsigned long npages, mpages =0 ; +struct page **pages; +unsigned long *src, *dst; +dma_addr_t *dma_addr; +void *buf; +int i, err =0 ; +unsigned int retry_count =2 ; + + npages = devmem_allocation->size >>PAGE_SHIFT ; + +retry: +if (!mmget_not_zero (devmem_allocation->mm)) +return -EFAULT ; + + buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr) + + sizeof(*pages), GFP_KERNEL); +if (!buf) { + err = -ENOMEM ; + goto err_out; + } + src = buf; + dst = buf + (sizeof (*src) * npages); + dma_addr = buf + (2 *sizeof (*src) * npages); + pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) *npages;
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ + err = ops->populate_devmem_pfn(devmem_allocation, npages,src);
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+if (err) + goto err_free; + + err =migrate_device_pfns (src, npages); +if (err) + goto err_free; + + err = drm_gpusvm_migrate_populate_ram_pfn(NULL, NULL,npages, &mpages,
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ src, dst, 0); +if (err || !mpages) + goto err_finalize; + + err = drm_gpusvm_migrate_map_pages(devmem_allocation->dev,
dma_addr,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ dst, npages,
DMA_FROM_DEVICE);drivers/gpu/drm/drm_gpusvm.c new file +16
-0+if (err) + goto err_finalize; + +for (i =0 ; i < npages; ++i) + pages[i] =migrate_pfn_to_page (src[i]); + + err = ops->copy_to_ram (pages, dma_addr, npages); +if (err) + goto err_finalize; + +err_finalize: +if (err) +drm_gpusvm_migration_unlock_put_pages (npages, dst); +migrate_device_pages (src, dst, npages); +migrate_device_finalize (src, dst, npages); + drm_gpusvm_migrate_unmap_pages(devmem_allocation->dev,
dma_addr, npages,drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ DMA_FROM_DEVICE); +err_free:
drivers/gpu/drm/drm_gpusvm.c line 1929
Quoted 10 lines
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+kvfree (buf); +err_out: +mmput_async (devmem_allocation->mm); + +if (completion_done (&devmem_allocation->detached)) +return 0 ; + +if (!err || retry_count--) { +cond_resched (); + goto retry;
I think we might end up in an infinite loop here if someone unknown holds a device page reference. Then err will be zero and we'll continously trying to evict that page until the holder drops its refcount. The caller would probably want to move on trying to evict something else. So what about retrying once in that case and then return -EBUSY?
Quoted 875 lines
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ } + +return err; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_evict_to_ram); + +/** +* __drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM (internal)
+ * @vas: Pointer to the VM area structure
+ * @device_private_page_owner: Device private pages owner
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @fault_addr: Fault address
+ * @size: Size of migration
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to RAM. It sets up the migration, populates + dma maps RAM PFNs, and
+ * invokes the driver-specific operations for migration to RAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static int __drm_gpusvm_migrate_to_ram (struct vm_area_struct *vas, + void
*device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ struct page *page, + unsigned long fault_addr, + unsigned long size) +{ +struct migrate_vma migrate = { + .vma = vas, + .pgmap_owner = device_private_page_owner, + .flags =MIGRATE_VMA_SELECT_DEVICE_PRIVATE |
drivers/gpu/drm/drm_gpusvm.c new file +17
-0+ MIGRATE_VMA_SELECT_DEVICE_COHERENT, + .fault_page = page, + }; +struct drm_gpusvm_zdd *zdd; +const struct drm_gpusvm_devmem_ops *ops; +struct device *dev; +unsigned long npages, mpages =0 ; +struct page **pages; +dma_addr_t *dma_addr; +unsigned long start, end; +void *buf; +int i, err =0 ; + + start =ALIGN_DOWN (fault_addr, size); + end =ALIGN (fault_addr +1 , size); + +/* Corner where VMA area struct has been partially unmapped */
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+if (start < vas->vm_start) + start = vas->vm_start; +if (end > vas->vm_end) + end = vas->vm_end; + + migrate.start = start; + migrate.end = end; + npages =npages_in_range (start, end); + + buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +
sizeof(*dma_addr)+drivers/gpu/drm/drm_gpusvm.c new file +7
-0+ sizeof(*pages), GFP_KERNEL); +if (!buf) { + err = -ENOMEM ; + goto err_out; + } + dma_addr = buf + (2 *sizeof (*migrate.src) * npages); + pages = buf + (2 *sizeof (*migrate.src) +sizeof (*dma_addr))* npages;
drivers/gpu/drm/drm_gpusvm.c new file +30
-0+ + migrate.vma = vas; + migrate.src = buf; + migrate.dst = migrate.src + npages; + + err =migrate_vma_setup (&migrate); +if (err) + goto err_free; + +/* Raced with another CPU fault, nothing to do */ +if (!migrate.cpages) + goto err_free; + +if (!page) { +for (i =0 ; i < npages; ++i) { +if (!(migrate.src[i] &MIGRATE_PFN_MIGRATE )) +continue ; + + page =migrate_pfn_to_page (migrate.src[i]); +break ; + } + +if (!page) + goto err_finalize; + } + zdd = page->zone_device_data; + ops = zdd->devmem_allocation->ops; + dev = zdd->devmem_allocation->dev; + + err = drm_gpusvm_migrate_populate_ram_pfn(vas, page, npages,&mpages,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ migrate.src,migrate.dst,
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ start); +if (err) + goto err_finalize; + + err = drm_gpusvm_migrate_map_pages(dev, dma_addr,migrate.dst, npages,
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+ DMA_FROM_DEVICE); +if (err) + goto err_finalize; + +for (i =0 ; i < npages; ++i) + pages[i] =migrate_pfn_to_page (migrate.src[i]); + + err = ops->copy_to_ram (pages, dma_addr, npages); +if (err) + goto err_finalize; + +err_finalize: +if (err) + drm_gpusvm_migration_unlock_put_pages(npages,migrate.dst);
drivers/gpu/drm/drm_gpusvm.c new file +17
-0+migrate_vma_pages (&migrate); +migrate_vma_finalize (&migrate); +drm_gpusvm_migrate_unmap_pages (dev, dma_addr, npages, + DMA_FROM_DEVICE); +err_free: +kvfree (buf); +err_out: + +return err; +} + +/** +* drm_gpusvm_range_evict - Evict GPU SVM range +* @pagemap: Pointer to the GPU SVM structure +* @range: Pointer to the GPU SVM range to be removed +* +* This function evicts the specified GPU SVM range. This function will not
+ * evict coherent pages.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+int drm_gpusvm_range_evict (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range) +{ + struct mmu_interval_notifier *notifier = &range->notifier-Quoted 1 line
notifier;
drivers/gpu/drm/drm_gpusvm.c new file +9
-0+struct hmm_range hmm_range = { + .default_flags =HMM_PFN_REQ_FAULT , + .notifier = notifier, + .start = range->itree.start, + .end = range->itree.last +1 , + .dev_private_owner =NULL , + }; +unsigned long timeout = + jiffies +
msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);drivers/gpu/drm/drm_gpusvm.c new file +3
-0+unsigned long *pfns; + unsigned long npages = npages_in_range(range->itree.start, + range->itree.last +1);
drivers/gpu/drm/drm_gpusvm.c new file +13
-0+int err =0 ; +struct mm_struct *mm = gpusvm->mm; + +if (!mmget_not_zero (mm)) +return -EFAULT ; + + pfns =kvmalloc_array (npages,sizeof (*pfns),GFP_KERNEL ); +if (!pfns) +return -ENOMEM ; + + hmm_range.hmm_pfns = pfns; +while (!time_after (jiffies, timeout)) { + hmm_range.notifier_seq =
mmu_interval_read_begin(notifier);drivers/gpu/drm/drm_gpusvm.c new file +21
-0+if (time_after (jiffies, timeout)) { + err = -ETIME ; +break ; + } + +mmap_read_lock (mm); + err =hmm_range_fault (&hmm_range); +mmap_read_unlock (mm); +if (err != -EBUSY ) +break ; + } + +kvfree (pfns); +mmput (mm); + +return err; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_evict); + +/** +* drm_gpusvm_page_free() - Put GPU SVM zone device data associated with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device data
+ * associated with a page when it is being released.
+ */drivers/gpu/drm/drm_gpusvm.c new file +7
-0+static void drm_gpusvm_page_free (struct page *page) +{ +drm_gpusvm_zdd_put (page->zone_device_data); +} + +/** +* drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM (page fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ *VM_FAULT_SIGBUSon failure, 0 on success.
+ */drivers/gpu/drm/drm_gpusvm.c new file +7
-0+static vm_fault_t drm_gpusvm_migrate_to_ram (struct vm_fault *vmf) +{ +struct drm_gpusvm_zdd *zdd = vmf->page->zone_device_data; +int err; + + err = __drm_gpusvm_migrate_to_ram(vmf->vma, + zdd-Quoted 1 line
device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ vmf->page, vmf->address, + zdd->devmem_allocation-Quoted 1 line
size);
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+ +return err ?VM_FAULT_SIGBUS :0 ; +} + +/** +* drm_gpusvm_pagemap_ops() - Device page map operations for GPU SVM +*/ +static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = { + .page_free = drm_gpusvm_page_free, + .migrate_to_ram = drm_gpusvm_migrate_to_ram, +}; + +/** +* drm_gpusvm_pagemap_ops_get() - Retrieve GPU SVM device page map operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +8
-0+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get (void ) +{ +return &drm_gpusvm_pagemap_ops; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_pagemap_ops_get); + +/** +* drm_gpusvm_has_mapping() - Check if GPU SVM has mapping for the given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(structdrm_gpusvm*gpusvm, unsigned long start,drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ unsigned long end) +{ +struct drm_gpusvm_notifier *notifier; + +drm_gpusvm_for_each_notifier (notifier, gpusvm, start, end) { +struct drm_gpusvm_range *range =NULL ; + + drm_gpusvm_for_each_range(range, notifier, start,end)
drivers/gpu/drm/drm_gpusvm.c new file +9
-0+return true; + } + +return false; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_has_mapping); + +/** +* drm_gpusvm_range_set_unmapped() - Mark a GPU SVM range as unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets thepartial_unmapflag
+ * if the range partially falls within the provided MMU notifier range.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+void drm_gpusvm_range_set_unmapped (struct drm_gpusvm_range *range, + const struct mmu_notifier_range
*mmu_range)drivers/gpu/drm/drm_gpusvm.c new file +12
-0+{ +lockdep_assert_held_write (&range->gpusvm->notifier_lock); + + range->flags.unmapped = true; +if (range->itree.start < mmu_range->start || + range->itree.last + 1 > mmu_range->end) + range->flags.partial_unmap = true; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_set_unmapped); + +/** +* drm_gpusvm_devmem_init() - Initialize a GPU SVM device memory allocation
+ *
+ * @dev: Pointer to the device structure which device memory allocation belongs to
+ * @mm: Pointer to themm_structfor the address space
+ * @ops: Pointer to the operations structure for GPU SVM device memory
+ * @dpagemap: The structdrm_pagemapwe're allocating from.
+ * @size: Size of device memory allocation
+ */
+void drm_gpusvm_devmem_init(structdrm_gpusvm_devmem*devmem_allocation,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct device *dev, struct mm_struct*mm,
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ const struct drm_gpusvm_devmem_ops *ops, + struct drm_pagemap *dpagemap, size_tsize)
drivers/gpu/drm/drm_gpusvm.c new file +12
-0+{ +init_completion (&devmem_allocation->detached); + devmem_allocation->dev = dev; + devmem_allocation->mm = mm; + devmem_allocation->ops = ops; + devmem_allocation->dpagemap = dpagemap; + devmem_allocation->size = size; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_devmem_init); + +MODULE_DESCRIPTION ("DRM GPUSVM" ); +MODULE_LICENSE ("GPL" );include/drm/drm_gpusvm.h new file +24
-0@@ -0,0 +1,445 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +/* +* Copyright © 2024 Intel Corporation +*/ + +#ifndef __DRM_GPUSVM_H__ +#define __DRM_GPUSVM_H__ + +#include <linux/kref.h> +#include <linux/interval_tree.h> +#include <linux/mmu_notifier.h> + +struct dev_pagemap_ops ; +struct drm_device ; +struct drm_gpusvm ; +struct drm_gpusvm_notifier ; +struct drm_gpusvm_ops ; +struct drm_gpusvm_range ; +struct drm_gpusvm_devmem ; +struct drm_pagemap ; +struct drm_pagemap_dma_addr ; + +/** +* struct drm_gpusvm_devmem_ops - Operations structure for GPU SVM device memory
+ *
+ * This structure defines the operations for GPU Shared Virtual Memory (SVM)
+ * device memory. These operations are provided by the GPU driver to manage device memory
+ * allocations and perform operations such as migration between device memory and system
+ * RAM.
+ */include/drm/drm_gpusvm.h new file +3
-0+struct drm_gpusvm_devmem_ops { +/** +* @devmem_release: Release device memory allocation (optional)
include/drm/drm_gpusvm.h new file +3
-0+* @devmem_allocation: device memory allocation +* +* Release device memory allocation and drop a reference to device
include/drm/drm_gpusvm.h new file +3
-0+* memory allocation. +*/ + void (*devmem_release)(struct drm_gpusvm_devmem
*devmem_allocation);include/drm/drm_gpusvm.h new file +3
-0+ +/** +* @populate_devmem_pfn: Populate device memory PFN (required for migration)
include/drm/drm_gpusvm.h new file +10
-0+* @devmem_allocation: device memory allocation +* @npages: Number of pages to populate +* @pfn: Array of page frame numbers to populate +* +* Populate device memory page frame numbers (PFN). +* +* Returns: +* 0 on success, a negative error code on failure. +*/ + int (*populate_devmem_pfn)(struct drm_gpusvm_devmem
*devmem_allocation,include/drm/drm_gpusvm.h new file +1
-0+ unsigned long npages, unsignedlong *pfn);
include/drm/drm_gpusvm.h new file +3
-0+ +/** +* @copy_to_devmem: Copy to device memory (required for migration)
include/drm/drm_gpusvm.h new file +1
-0+* @pages: Pointer to array of device memory pages (destination)
include/drm/drm_gpusvm.h new file +16
-0+* @dma_addr: Pointer to array of DMA addresses (source) +* @npages: Number of pages to copy +* +* Copy pages to device memory. +* +* Returns: +* 0 on success, a negative error code on failure. +*/ +int (*copy_to_devmem)(struct page **pages, + dma_addr_t *dma_addr, + unsigned long npages); + +/** +* @copy_to_ram: Copy to system RAM (required for migration) +* @pages: Pointer to array of device memory pages (source) +* @dma_addr: Pointer to array of DMA addresses (destination)
include/drm/drm_gpusvm.h new file +14
-0+* @npages: Number of pages to copy +* +* Copy pages to system RAM. +* +* Returns: +* 0 on success, a negative error code on failure. +*/ +int (*copy_to_ram)(struct page **pages, + dma_addr_t *dma_addr, + unsigned long npages); +}; + +/** +* struct drm_gpusvm_devmem - Structure representing a GPU SVM device memory allocation
+ *
+ * @dev: Pointer to the device structure which device memory allocation belongs to
+ * @mm: Pointer to themm_structfor the address space
+ * @detached: device memory allocations is detached from device pages
+ * @ops: Pointer to the operations structure for GPU SVM device memory
+ * @dpagemap: The structdrm_pagemapof the pages this allocation belongs to.
+ * @size: Size of device memory allocation
+ */include/drm/drm_gpusvm.h new file +13
-0+struct drm_gpusvm_devmem { +struct device *dev; +struct mm_struct *mm; +struct completion detached; +const struct drm_gpusvm_devmem_ops *ops; +struct drm_pagemap *dpagemap; +size_t size; +}; + +/** +* struct drm_gpusvm_ops - Operations structure for GPU SVM +* +* This structure defines the operations for GPU Shared Virtual Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * notifiers.
+ */include/drm/drm_gpusvm.h new file +8
-0+struct drm_gpusvm_ops { +/** +* @notifier_alloc: Allocate a GPU SVM notifier (optional) +* +* Allocate a GPU SVM notifier. +* +* Returns: +* Pointer to the allocated GPU SVM notifier on success, NULL on failure.
include/drm/drm_gpusvm.h new file +19
-0+*/ +struct drm_gpusvm_notifier *(*notifier_alloc)(void ); + +/** +* @notifier_free: Free a GPU SVM notifier (optional) +* @notifier: Pointer to the GPU SVM notifier to be freed +* +* Free a GPU SVM notifier. +*/ +void (*notifier_free)(struct drm_gpusvm_notifier *notifier); + +/** +* @range_alloc: Allocate a GPU SVM range (optional) +* @gpusvm: Pointer to the GPU SVM +* +* Allocate a GPU SVM range. +* +* Returns: +* Pointer to the allocated GPU SVM range on success, NULL on failure.
include/drm/drm_gpusvm.h new file +2
-0+*/ + struct drm_gpusvm_range *(*range_alloc)(struct drm_gpusvm*gpusvm);
include/drm/drm_gpusvm.h new file +16
-0+ +/** +* @range_free: Free a GPU SVM range (optional) +* @range: Pointer to the GPU SVM range to be freed +* +* Free a GPU SVM range. +*/ +void (*range_free)(struct drm_gpusvm_range *range); + +/** +* @invalidate: Invalidate GPU SVM notifier (required) +* @gpusvm: Pointer to the GPU SVM +* @notifier: Pointer to the GPU SVM notifier +* @mmu_range: Pointer to the mmu_notifier_range structure +* +* Invalidate the GPU page tables. It can safely walk the notifier range
include/drm/drm_gpusvm.h new file +1
-0+* RB tree/list in this function. Called while holding the notifier lock.
include/drm/drm_gpusvm.h new file +4
-0+*/ +void (*invalidate)(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + const struct mmu_notifier_range
*mmu_range);include/drm/drm_gpusvm.h new file +4
-0+}; + +/** +* struct drm_gpusvm_notifier - Structure representing a GPU SVM notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @itree: Interval tree node for the notifier (inserted in GPU SVM)
+ * @entry: List entry to fast interval tree traversal
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ * interval tree. This is useful to keep iterating ranges while
+ * doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier has been
+ * removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */include/drm/drm_gpusvm.h new file +19
-0+struct drm_gpusvm_notifier { +struct drm_gpusvm *gpusvm; +struct mmu_interval_notifier notifier; +struct interval_tree_node itree; +struct list_head entry; +struct rb_root_cached root; +struct list_head range_list; +struct { +u32 removed :1 ; + } flags; +}; + +/** +* struct drm_gpusvm_range - Structure representing a GPU SVM range +* +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier +* @refcount: Reference count for the range +* @itree: Interval tree node for the range (inserted in GPU SVM notifier)
+ * @entry: List entry to fast interval tree traversal
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @dma_addr: DMA address array
+ * @dpagemap: The structdrm_pagemapof the device pages we're dma- mapping.
+ * Note this is assuming only onedrm_pagemapper range is allowed.
+ * @flags.migrate_devmem: Flag indicating whether the range can be migrated to device memory
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_devmem_pages: Flag indicating if the range has devmem pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA mapping
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */include/drm/drm_gpusvm.h new file +13
-0+struct drm_gpusvm_range { +struct drm_gpusvm *gpusvm; +struct drm_gpusvm_notifier *notifier; +struct kref refcount; +struct interval_tree_node itree; +struct list_head entry; +unsigned long notifier_seq; +struct drm_pagemap_dma_addr *dma_addr; +struct drm_pagemap *dpagemap; +struct { +/* All flags below must be set upon creation */ +u16 migrate_devmem :1 ; +/* All flags below must be set / cleared under notifier lock */
include/drm/drm_gpusvm.h new file +19
-0+ u16 unmapped : 1; + u16 partial_unmap : 1; + u16 has_devmem_pages : 1; + u16 has_dma_mapping : 1; + } flags; +}; + +/** +* struct drm_gpusvm - GPU SVM structure +* +* @name: Name of the GPU SVM +* @drm: Pointer to the DRM device structure +* @mm: Pointer to the mm_struct for the address space +* @device_private_page_owner: Device private pages owner +* @mm_start: Start address of GPU SVM +* @mm_range: Range of the GPU SVM +* @notifier_size: Size of individual notifiers +* @ops: Pointer to the operations structure for GPU SVM +* @chunk_sizes: Pointer to the array of chunk sizes used in range allocation.
+ * Entries should be powers of 2 in descending order.
+ * @num_chunks: Number of chunks
+ * @notifier_lock: Read-write semaphore for protecting notifier operations
+ * @root: Cached root node of the Red-Black tree containing GPU SVM notifiers
+ * @notifier_list: list head containing of notifiers in the same order they
+ * appear in interval tree. This is useful to keep iterating
+ * notifiers while doing modifications to RB tree.
+ *
+ * This structure represents a GPU SVM (Shared Virtual Memory) used for tracking
+ * memory ranges mapped in a DRM (Direct Rendering Manager) device.
+ *
+ * No reference counting is provided, as this is expected to be embedded in the
+ * driver VM structure along with the structdrm_gpuvm, which handles reference
+ * counting.
+ */include/drm/drm_gpusvm.h new file +17
-0+struct drm_gpusvm { +const char *name; +struct drm_device *drm; +struct mm_struct *mm; +void *device_private_page_owner; +unsigned long mm_start; +unsigned long mm_range; +unsigned long notifier_size; +const struct drm_gpusvm_ops *ops; +const unsigned long *chunk_sizes; +int num_chunks; +struct rw_semaphore notifier_lock; +struct rb_root_cached root; +struct list_head notifier_list; +#ifdef CONFIG_LOCKDEP +/** +* @lock_dep_map: Annotates drm_gpusvm_range_find_or_insert and
include/drm/drm_gpusvm.h new file +10
-0+* drm_gpusvm_range_remove with a driver provided lock. +*/ +struct lockdep_map *lock_dep_map; +#endif +}; + +/** +* struct drm_gpusvm_ctx - DRM GPU SVM context +* +* @check_pages_threshold: Check CPU pages for present if chunk is less than or
+ * equal to threshold. If not present, reduce chunk
+ * size.
+ * @in_notifier: entering from a MMU notifier
+ * @read_only: operating on read-only memory
+ * @devmem_possible: possible to use device memory
+ *
+ * Context that is DRM GPUSVM is operating in (i.e. user arguments).
+ */include/drm/drm_gpusvm.h new file +10
-0+struct drm_gpusvm_ctx { +unsigned long check_pages_threshold; +unsigned int in_notifier :1 ; +unsigned int read_only :1 ; +unsigned int devmem_possible :1 ; +}; + +int drm_gpusvm_init (struct drm_gpusvm *gpusvm, + const char *name, struct drm_device *drm, + struct mm_struct *mm, void
*device_private_page_owner,include/drm/drm_gpusvm.h new file +4
-0+ unsigned long mm_start, unsigned long mm_range, + unsigned long notifier_size, + const struct drm_gpusvm_ops *ops, + const unsigned long *chunk_sizes, int
num_chunks);include/drm/drm_gpusvm.h new file +37
-0+ +void drm_gpusvm_fini (struct drm_gpusvm *gpusvm); + +void drm_gpusvm_free (struct drm_gpusvm *gpusvm); + +struct drm_gpusvm_range * +drm_gpusvm_range_find_or_insert (struct drm_gpusvm *gpusvm, +unsigned long fault_addr, +unsigned long gpuva_start, +unsigned long gpuva_end, +const struct drm_gpusvm_ctx *ctx); + +void drm_gpusvm_range_remove (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range); + +int drm_gpusvm_range_evict (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range); + +struct drm_gpusvm_range * +drm_gpusvm_range_get (struct drm_gpusvm_range *range); + +void drm_gpusvm_range_put (struct drm_gpusvm_range *range); + +bool drm_gpusvm_range_pages_valid (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range); + +int drm_gpusvm_range_get_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx); + +void drm_gpusvm_range_unmap_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx); + +int drm_gpusvm_migrate_to_devmem (struct drm_gpusvm *gpusvm, +struct drm_gpusvm_range *range, + struct drm_gpusvm_devmem
*devmem_allocation,include/drm/drm_gpusvm.h new file +3
-0+ const struct drm_gpusvm_ctx *ctx); + +int drm_gpusvm_evict_to_ram(struct drm_gpusvm_devmem
*devmem_allocation);include/drm/drm_gpusvm.h new file +4
-0+ +const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get (void ); + +bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, unsigned longstart,
include/drm/drm_gpusvm.h new file +4
-0+ unsigned long end); + +struct drm_gpusvm_range * +drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, unsignedlong start,
include/drm/drm_gpusvm.h new file +4
-0+ unsigned long end); + +void drm_gpusvm_range_set_unmapped (struct drm_gpusvm_range *range, + const struct mmu_notifier_range
*mmu_range);include/drm/drm_gpusvm.h new file +2
-0+ +void drm_gpusvm_devmem_init(struct drm_gpusvm_devmem
*devmem_allocation,include/drm/drm_gpusvm.h new file +1
-0+ struct device *dev, struct mm_struct*mm,
include/drm/drm_gpusvm.h new file +2
-0+ const struct drm_gpusvm_devmem_ops *ops, + struct drm_pagemap *dpagemap, size_tsize);
include/drm/drm_gpusvm.h new file +4
-0+ +#ifdef CONFIG_LOCKDEP +/** +* drm_gpusvm_driver_set_lock() - Set the lock protecting accesses to GPU SVM
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @lock: the lock used to protect the gpuva list. The locking primitive
+ * must contain adep_mapfield.
+ *
+ * Call this to annotatedrm_gpusvm_range_find_or_insertand
+ *drm_gpusvm_range_remove.
+ */include/drm/drm_gpusvm.h new file +4
-0+#define drm_gpusvm_driver_set_lock (gpusvm, lock) \ + do { \ + if (!WARN((gpusvm)->lock_dep_map, \ + "GPUSVM range lock should be set onlyonce."))\
include/drm/drm_gpusvm.h new file +1
-0+ (gpusvm)->lock_dep_map = &(lock)-Quoted 1 line
dep_map; \include/drm/drm_gpusvm.h new file +31
-0+ } while (0) +#define drm_gpusvm_driver_lock_held (gpusvm) \ + do { \ + if ((gpusvm)->lock_dep_map) \ + lock_is_held((gpusvm)->lock_dep_map); \ + } while (0) +#else +#define drm_gpusvm_driver_set_lock (gpusvm, lock) do {} while (0) +#define drm_gpusvm_driver_lock_held (gpusvm) do {} while (0) +#endif + +/** +* drm_gpusvm_notifier_lock() - Lock GPU SVM notifier +* @gpusvm__: Pointer to the GPU SVM structure. +* +* Abstract client usage GPU SVM notifier lock, take lock +*/ +#define drm_gpusvm_notifier_lock (gpusvm__) \ + down_read(&(gpusvm__)->notifier_lock) + +/** +* drm_gpusvm_notifier_unlock() - Unlock GPU SVM notifier +* @gpusvm__: Pointer to the GPU SVM structure. +* +* Abstract client usage GPU SVM notifier lock, drop lock +*/ +#define drm_gpusvm_notifier_unlock (gpusvm__) \ + up_read(&(gpusvm__)->notifier_lock) + +/** +* __drm_gpusvm_range_next() - Get the next GPU SVM range in the list
+ * @range: a pointer to the current GPU SVM range
+ *
+ * Return: A pointer to the nextdrm_gpusvm_rangeif available, or NULL if the
+ * current range is the last one or if the input range is NULL.
+ */include/drm/drm_gpusvm.h new file +12
-0+static inline struct drm_gpusvm_range * +__drm_gpusvm_range_next (struct drm_gpusvm_range *range) +{ +if (range && !list_is_last (&range->entry, + &range->notifier->range_list)) +return list_next_entry (range, entry); + +return NULL ; +} + +/** +* drm_gpusvm_for_each_range() - Iterate over GPU SVM ranges in a notifier
+ * @range__: Iterator variable for the ranges. If set, it indicates the start of
+ * the iterator. If NULL, calldrm_gpusvm_range_find()to
get the range.
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier. It is safe
+ * to use while holding the driver SVM lock or the notifier lock.
+ */
+#define drm_gpusvm_for_each_range(range__,notifier__,start__,end__) \include/drm/drm_gpusvm.h new file +1
-0+ for ((range__) = (range__)?: \
include/drm/drm_gpusvm.h new file +1
-0+ drm_gpusvm_range_find((notifier__), (start__),(
end__)); \include/drm/drm_gpusvm.h new file +1
-0+ (range__) && (range__->itree.start <(
end__)); \include/drm/drm_gpusvm.h new file +3
-0+ (range__) = __drm_gpusvm_range_next(range__)) + +#endif /* __DRM_GPUSVM_H__ */
Re: [PATCH v4 06/33] drm/gpusvm: Add support for GPU Shared Virtual Memory
In reply to Matthew BrostQuoted 75 lines · Matthew Brost
This patch introduces support for GPU Shared Virtual Memory (SVM) in the Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.
The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also provides mechanisms for allocating, deallocating, and migrating memory regions between system RAM and GPU VRAM.
This is largely inspired by GPUVM.
v2:
- Take order into account in check pages
- Clearrange->pagesin get pages error
- Drop setting dirty or accessed bit in get pages (Vetter)
- Remove mmap assert for cpu faults
- Drop mmap write lock abuse (Vetter, Christian)
- Decouple zdd from range (Vetter, Oak)
- Adddrm_gpusvm_range_evict, make it work with coherent pages
- Exportdrm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
- mmget/put indrm_gpusvm_evict_to_sram
- Droprange->vram_alloationvariable
- Don't return indrm_gpusvm_evict_to_sramuntil all pages detached
- Don't warn on mixing sram and device pages
- Update kernel doc
- Add coherent page support to get pages
- UseDMA_FROM_DEVICErather thanDMA_BIDIRECTIONAL
- Add structdrm_gpusvm_vramand ops (Thomas)
- Update the range's seqno if the range is valid (Thomas)
- Remove theis_unmappedcheck beforehmm_range_fault(Thomas)
- Usedrm_pagemap(Thomas)
- Dropkfree_mapping(Thomas)
- dma mapp pages under notifier lock (Thomas)
- Remove ctx.prefault
- Remove ctx.mmap_locked
- Add ctx.check_pages
- s/vram/devmem (Thomas)
v3:
- Fix memory leakdrm_gpusvm_range_get_pages
- Only migrate pages with same zdd on CPU fault
- Loop over al VMAs indrm_gpusvm_range_evict
- Make GPUSVM a drm level module
- GPL or MIT license
- Update main kernel doc (Thomas)
- Preferfoo()vs foo for functions in kernel doc (Thomas)
- Prefer functions over macros (Thomas)
- Use unsigned long vs u64 for addresses (Thomas)
- Use standardinterval_tree(Thomas)
- s/drm_gpusvm_migration_put_page/drm_gpusvm_migration_unlock_put_page (Thomas)
- Droperr_outlabel indrm_gpusvm_range_find_or_insert(Thomas)
- Fix kernel doc indrm_gpusvm_range_free_pages(Thomas)
- Newlines between functions defs in header file (Thomas)
- Drop shall language in driver vfunc kernel doc (Thomas)
- Move some static inlines from head to C file (Thomas)
- Don't allocate pages under page lock indrm_gpusvm_migrate_populate_ram_pfn(Thomas)
- Changecheck_pagesto a threshold
v4:
- Fix NULL ptr deref indrm_gpusvm_migrate_populate_ram_pfn(Thomas, Himal)
- Fix check pages threshold
- Check for range being unmapped under notifier lock in get pages (Testing)
- Fix characters per line
- DropWRITE_ONCEforzdd->devmem_allocationassignment (Thomas)
- Use completion fordevmem_allocation->detached(Thomas)
- Make GPU SVM depend onZONE_DEVICE(CI)
- Usehmm_range_faultfor eviction (Thomas)
- Drop zdd worker (Thomas)Cc Simona Vetter <[email protected]>Cc Dave Airlie <[email protected]>Cc Christian König <[email protected]>Cc <[email protected]>Signed-off-by Matthew Brost <[email protected]>Signed-off-by Thomas Hellström <[email protected]>---
<snip>
Quoted 103 lines
drivers/gpu/drm/drm_gpusvm.c new file +103
-0+/** +* __drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM (internal) +* @vas: Pointer to the VM area structure +* @device_private_page_owner: Device private pages owner +* @page: Pointer to the page for fault handling (can be NULL) +* @fault_addr: Fault address +* @size: Size of migration +* +* This internal function performs the migration of the specified GPU SVM range +* to RAM. It sets up the migration, populates + dma maps RAM PFNs, and +* invokes the driver-specific operations for migration to RAM. +* +* Returns: +* 0 on success, negative error code on failure. +*/ +static int __drm_gpusvm_migrate_to_ram (struct vm_area_struct *vas, +void *device_private_page_owner, +struct page *page, +unsigned long fault_addr, +unsigned long size) +{ +struct migrate_vma migrate = { + .vma = vas, + .pgmap_owner = device_private_page_owner, + .flags =MIGRATE_VMA_SELECT_DEVICE_PRIVATE | +MIGRATE_VMA_SELECT_DEVICE_COHERENT , + .fault_page = page, + }; +struct drm_gpusvm_zdd *zdd; +const struct drm_gpusvm_devmem_ops *ops; +struct device *dev; +unsigned long npages, mpages =0 ; +struct page **pages; +dma_addr_t *dma_addr; +unsigned long start, end; +void *buf; +int i, err =0 ; + + start =ALIGN_DOWN (fault_addr, size); + end =ALIGN (fault_addr +1 , size); + +/* Corner where VMA area struct has been partially unmapped */ +if (start < vas->vm_start) + start = vas->vm_start; +if (end > vas->vm_end) + end = vas->vm_end; + + migrate.start = start; + migrate.end = end; + npages =npages_in_range (start, end); + + buf =kvcalloc (npages,2 *sizeof (*migrate.src) +sizeof (*dma_addr) + +sizeof (*pages),GFP_KERNEL ); +if (!buf) { + err = -ENOMEM ; + goto err_out; + } + dma_addr = buf + (2 *sizeof (*migrate.src) * npages); + pages = buf + (2 *sizeof (*migrate.src) +sizeof (*dma_addr)) * npages; + + migrate.vma = vas; + migrate.src = buf; + migrate.dst = migrate.src + npages; + + err =migrate_vma_setup (&migrate); +if (err) + goto err_free; + +/* Raced with another CPU fault, nothing to do */ +if (!migrate.cpages) + goto err_free; + +if (!page) { +for (i =0 ; i < npages; ++i) { +if (!(migrate.src[i] &MIGRATE_PFN_MIGRATE )) +continue ; + + page =migrate_pfn_to_page (migrate.src[i]); +break ; + } + +if (!page) + goto err_finalize; + } + zdd = page->zone_device_data; + ops = zdd->devmem_allocation->ops; + dev = zdd->devmem_allocation->dev; + + err =drm_gpusvm_migrate_populate_ram_pfn (vas, page, npages, &mpages, + migrate.src, migrate.dst, + start); +if (err) + goto err_finalize; + + err =drm_gpusvm_migrate_map_pages (dev, dma_addr, migrate.dst, npages, +DMA_FROM_DEVICE ); +if (err) + goto err_finalize; + +for (i =0 ; i < npages; ++i) + pages[i] =migrate_pfn_to_page (migrate.src[i]); + + err = ops->copy_to_ram (pages, dma_addr, npages);
drivers/gpu/drm/drm_gpusvm.c line 2048
Quoted 10 lines
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+if (err) + goto err_finalize; + +err_finalize: +if (err) +drm_gpusvm_migration_unlock_put_pages (npages, migrate.dst); +migrate_vma_pages (&migrate); +migrate_vma_finalize (&migrate); +drm_gpusvm_migrate_unmap_pages (dev, dma_addr, npages, +DMA_FROM_DEVICE );
clang for me is throwing:
drivers/gpu/drm/drm_gpusvm.c:2017:7: error: variable 'dev' is used uninitialized whenever 'if' condition is true [-Werror,-Wsometimes-uninitialized]
2017 | if (!page)
| ^~~~~drivers/gpu/drm/drm_gpusvm.c:2047:33: note: uninitialized use occurs here
2047 | drm_gpusvm_migrate_unmap_pages(dev, dma_addr, npages,
| ^~~drivers/gpu/drm/drm_gpusvm.c:2017:3: note: remove the 'if' if its condition is always false
2017 | if (!page)
| ^~~~~~~~~~
2018 | goto err_finalize;
| ~~~~~~~~~~~~~~~~~
drivers/gpu/drm/drm_gpusvm.c:1966:20: note: initialize the variable 'dev' to silence this warning
1966 | struct device *dev;
| ^
| = NULL1 error generated.
Re: [PATCH v4 06/33] drm/gpusvm: Add support for GPU Shared Virtual Memory
Quoted 201 lines · Matthew Auld
Quoted 75 lines · Matthew Brost
This patch introduces support for GPU Shared Virtual Memory (SVM) in the Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.
The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also provides mechanisms for allocating, deallocating, and migrating memory regions between system RAM and GPU VRAM.
This is largely inspired by GPUVM.
v2:
- Take order into account in check pages
- Clearrange->pagesin get pages error
- Drop setting dirty or accessed bit in get pages (Vetter)
- Remove mmap assert for cpu faults
- Drop mmap write lock abuse (Vetter, Christian)
- Decouple zdd from range (Vetter, Oak)
- Adddrm_gpusvm_range_evict, make it work with coherent pages
- Exportdrm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
- mmget/put indrm_gpusvm_evict_to_sram
- Droprange->vram_alloationvariable
- Don't return indrm_gpusvm_evict_to_sramuntil all pages detached
- Don't warn on mixing sram and device pages
- Update kernel doc
- Add coherent page support to get pages
- UseDMA_FROM_DEVICErather thanDMA_BIDIRECTIONAL
- Add structdrm_gpusvm_vramand ops (Thomas)
- Update the range's seqno if the range is valid (Thomas)
- Remove theis_unmappedcheck beforehmm_range_fault(Thomas)
- Usedrm_pagemap(Thomas)
- Dropkfree_mapping(Thomas)
- dma mapp pages under notifier lock (Thomas)
- Remove ctx.prefault
- Remove ctx.mmap_locked
- Add ctx.check_pages
- s/vram/devmem (Thomas)
v3:
- Fix memory leakdrm_gpusvm_range_get_pages
- Only migrate pages with same zdd on CPU fault
- Loop over al VMAs indrm_gpusvm_range_evict
- Make GPUSVM a drm level module
- GPL or MIT license
- Update main kernel doc (Thomas)
- Preferfoo()vs foo for functions in kernel doc (Thomas)
- Prefer functions over macros (Thomas)
- Use unsigned long vs u64 for addresses (Thomas)
- Use standardinterval_tree(Thomas)
- s/drm_gpusvm_migration_put_page/drm_gpusvm_migration_unlock_put_page(Thomas)
- Droperr_outlabel indrm_gpusvm_range_find_or_insert(Thomas)
- Fix kernel doc indrm_gpusvm_range_free_pages(Thomas)
- Newlines between functions defs in header file (Thomas)
- Drop shall language in driver vfunc kernel doc (Thomas)
- Move some static inlines from head to C file (Thomas)
- Don't allocate pages under page lock indrm_gpusvm_migrate_populate_ram_pfn(Thomas)
- Changecheck_pagesto a threshold
v4:
- Fix NULL ptr deref indrm_gpusvm_migrate_populate_ram_pfn(Thomas, Himal)
- Fix check pages threshold
- Check for range being unmapped under notifier lock in get pages (Testing)
- Fix characters per line
- DropWRITE_ONCEforzdd->devmem_allocationassignment (Thomas)
- Use completion fordevmem_allocation->detached(Thomas)
- Make GPU SVM depend onZONE_DEVICE(CI)
- Usehmm_range_faultfor eviction (Thomas)
- Drop zdd worker (Thomas)Cc Simona Vetter <[email protected]>Cc Dave Airlie <[email protected]>Cc Christian König <[email protected]>Cc <[email protected]>Signed-off-by Matthew Brost <[email protected]>Signed-off-by Thomas Hellström <[email protected]>---
<snip>
Quoted 109 lines
+/**
+ *__drm_gpusvm_migrate_to_ram()- Migrate GPU SVM range to RAM (internal)
+ * @vas: Pointer to the VM area structure
+ * @device_private_page_owner: Device private pages owner
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @fault_addr: Fault address
+ * @size: Size of migration
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to RAM. It sets up the migration, populates + dma maps RAM PFNs, and
+ * invokes the driver-specific operations for migration to RAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +37
-0+static int __drm_gpusvm_migrate_to_ram (struct vm_area_struct *vas, + void *device_private_page_owner, + struct page *page, + unsigned long fault_addr, + unsigned long size) +{ + struct migrate_vma migrate = { + .vma = vas, + .pgmap_owner = device_private_page_owner, + .flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE | + MIGRATE_VMA_SELECT_DEVICE_COHERENT, + .fault_page = page, + }; + struct drm_gpusvm_zdd *zdd; + const struct drm_gpusvm_devmem_ops *ops; + struct device *dev; + unsigned long npages, mpages = 0; + struct page **pages; + dma_addr_t *dma_addr; + unsigned long start, end; + void *buf; + int i, err = 0; + + start = ALIGN_DOWN(fault_addr, size); + end = ALIGN(fault_addr + 1, size); + + /* Corner where VMA area struct has been partially unmapped */ + if (start < vas->vm_start) + start = vas->vm_start; + if (end > vas->vm_end) + end = vas->vm_end; + + migrate.start = start; + migrate.end = end; + npages = npages_in_range(start, end); + + buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +
sizeof(*dma_addr)+
+sizeof(*pages),GFP_KERNEL);
+ if (!buf) {
+ err =-ENOMEM;
+ gotoerr_out;
+ }
+dma_addr= buf + (2 *sizeof(*migrate.src)* npages); + pages = buf + (2 *sizeof(*migrate.src)+sizeof(*dma_addr)) * npages;drivers/gpu/drm/drm_gpusvm.c new file +30
-0+ + migrate.vma = vas; + migrate.src = buf; + migrate.dst = migrate.src + npages; + + err = migrate_vma_setup(&migrate); + if (err) + goto err_free; + + /* Raced with another CPU fault, nothing to do */ + if (!migrate.cpages) + goto err_free; + + if (!page) { + for (i = 0; i < npages; ++i) { + if (!(migrate.src[i] & MIGRATE_PFN_MIGRATE)) + continue; + + page = migrate_pfn_to_page(migrate.src[i]); + break; + } + + if (!page) + goto err_finalize; + } + zdd = page->zone_device_data; + ops = zdd->devmem_allocation->ops; + dev = zdd->devmem_allocation->dev; + + err = drm_gpusvm_migrate_populate_ram_pfn(vas, page, npages,&mpages,
+ migrate.src, migrate.dst,
+ start);
+ if (err)
+ gotoerr_finalize;
+
+ err = drm_gpusvm_migrate_map_pages(dev,dma_addr, migrate.dst, npages,
+DMA_FROM_DEVICE);
+ if (err)
+ gotoerr_finalize;drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ + for (i = 0; i < npages; ++i) + pages[i] = migrate_pfn_to_page(migrate.src[i]); + + err = ops->copy_to_ram(pages, dma_addr, npages);drivers/gpu/drm/drm_gpusvm.c line 2048
Quoted 10 lines
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+ if (err) + goto err_finalize; + +err_finalize: + if (err) + drm_gpusvm_migration_unlock_put_pages(npages, migrate.dst); + migrate_vma_pages(&migrate); + migrate_vma_finalize(&migrate); + drm_gpusvm_migrate_unmap_pages(dev, dma_addr, npages, + DMA_FROM_DEVICE);clang for me is throwing:
Quoted 16 lines
drivers/gpu/drm/drm_gpusvm.c:2017:7: error: variable 'dev' is used uninitialized whenever 'if' condition is true [-Werror,-Wsometimes- uninitialized]
2017 | if (!page)
| ^~~~~drivers/gpu/drm/drm_gpusvm.c:2047:33: note: uninitialized use occurs here
2047 | drm_gpusvm_migrate_unmap_pages(dev,dma_addr, npages,
| ^~~drivers/gpu/drm/drm_gpusvm.c:2017:3: note: remove the 'if' if its condition is always false
2017 | if (!page)
| ^~~~~~~~~~
2018 | gotoerr_finalize;
| ~~~~~~~~~~~~~~~~~drivers/gpu/drm/drm_gpusvm.c:1966:20: note: initialize the variable 'dev' to silence this warning
1966 | struct device *dev;
| ^
| = NULL
1 error generated.
I also reported this issue in the v3 patch, but it doesn't seem to have been fixed in v4 yet.
https://lore.kernel.org/dri-devel/[email protected]/
Br,
G.G.
Matthew Brost
maintainerRe: [PATCH v4 06/33] drm/gpusvm: Add support for GPU Shared Virtual Memory
Quoted 229 lines · Gwan-gyeong Mun
Quoted 217 lines · Matthew Auld
Quoted 79 lines · Matthew Brost
This patch introduces support for GPU Shared Virtual Memory (SVM) in the Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.
The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also provides mechanisms for allocating, deallocating, and migrating memory regions between system RAM and GPU VRAM.
This is largely inspired by GPUVM.
v2:
- Take order into account in check pages
- Clearrange->pagesin get pages error
- Drop setting dirty or accessed bit in get pages (Vetter)
- Remove mmap assert for cpu faults
- Drop mmap write lock abuse (Vetter, Christian)
- Decouple zdd from range (Vetter, Oak)
- Adddrm_gpusvm_range_evict, make it work with coherent pages
- Exportdrm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
- mmget/put indrm_gpusvm_evict_to_sram
- Droprange->vram_alloationvariable
- Don't return indrm_gpusvm_evict_to_sramuntil all pages detached
- Don't warn on mixing sram and device pages
- Update kernel doc
- Add coherent page support to get pages
- UseDMA_FROM_DEVICErather thanDMA_BIDIRECTIONAL
- Add structdrm_gpusvm_vramand ops (Thomas)
- Update the range's seqno if the range is valid (Thomas)
- Remove theis_unmappedcheck beforehmm_range_fault(Thomas)
- Usedrm_pagemap(Thomas)
- Dropkfree_mapping(Thomas)
- dma mapp pages under notifier lock (Thomas)
- Remove ctx.prefault
- Remove ctx.mmap_locked
- Add ctx.check_pages
- s/vram/devmem (Thomas)
v3:
- Fix memory leakdrm_gpusvm_range_get_pages
- Only migrate pages with same zdd on CPU fault
- Loop over al VMAs indrm_gpusvm_range_evict
- Make GPUSVM a drm level module
- GPL or MIT license
- Update main kernel doc (Thomas)
- Preferfoo()vs foo for functions in kernel doc (Thomas)
- Prefer functions over macros (Thomas)
- Use unsigned long vs u64 for addresses (Thomas)
- Use standardinterval_tree(Thomas)
- s/drm_gpusvm_migration_put_page/drm_gpusvm_migration_unlock_put_page(Thomas)
- Droperr_outlabel indrm_gpusvm_range_find_or_insert(Thomas)
- Fix kernel doc indrm_gpusvm_range_free_pages(Thomas)
- Newlines between functions defs in header file (Thomas)
- Drop shall language in driver vfunc kernel doc (Thomas)
- Move some static inlines from head to C file (Thomas)
- Don't allocate pages under page lock indrm_gpusvm_migrate_populate_ram_pfn(Thomas)
- Changecheck_pagesto a threshold
v4:
- Fix NULL ptr deref indrm_gpusvm_migrate_populate_ram_pfn(Thomas, Himal)
- Fix check pages threshold
- Check for range being unmapped under notifier lock in get pages (Testing)
- Fix characters per line
- DropWRITE_ONCEforzdd->devmem_allocationassignment (Thomas)
- Use completion fordevmem_allocation->detached(Thomas)
- Make GPU SVM depend onZONE_DEVICE(CI)
- Usehmm_range_faultfor eviction (Thomas)
- Drop zdd worker (Thomas)Cc Simona Vetter <[email protected]>Cc Dave Airlie <[email protected]>Cc Christian König <[email protected]>Cc <[email protected]>Signed-off-by Matthew Brost <[email protected]>Signed-off-by Thomas Hellström <[email protected]>---
<snip>
Quoted 109 lines
+/**
+ *__drm_gpusvm_migrate_to_ram()- Migrate GPU SVM range to RAM (internal)
+ * @vas: Pointer to the VM area structure
+ * @device_private_page_owner: Device private pages owner
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @fault_addr: Fault address
+ * @size: Size of migration
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to RAM. It sets up the migration, populates + dma maps RAM PFNs, and
+ * invokes the driver-specific operations for migration to RAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +37
-0+static int __drm_gpusvm_migrate_to_ram (struct vm_area_struct *vas, + void *device_private_page_owner, + struct page *page, + unsigned long fault_addr, + unsigned long size) +{ + struct migrate_vma migrate = { + .vma = vas, + .pgmap_owner = device_private_page_owner, + .flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE | + MIGRATE_VMA_SELECT_DEVICE_COHERENT, + .fault_page = page, + }; + struct drm_gpusvm_zdd *zdd; + const struct drm_gpusvm_devmem_ops *ops; + struct device *dev; + unsigned long npages, mpages = 0; + struct page **pages; + dma_addr_t *dma_addr; + unsigned long start, end; + void *buf; + int i, err = 0; + + start = ALIGN_DOWN(fault_addr, size); + end = ALIGN(fault_addr + 1, size); + + /* Corner where VMA area struct has been partially unmapped */ + if (start < vas->vm_start) + start = vas->vm_start; + if (end > vas->vm_end) + end = vas->vm_end; + + migrate.start = start; + migrate.end = end; + npages = npages_in_range(start, end); + + buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +
sizeof(*dma_addr)+
+sizeof(*pages),GFP_KERNEL);
+ if (!buf) {
+ err =-ENOMEM;
+ gotoerr_out;
+ }
+dma_addr= buf + (2 *sizeof(*migrate.src)* npages); + pages = buf + (2 *sizeof(*migrate.src)+sizeof(*dma_addr)) * npages;drivers/gpu/drm/drm_gpusvm.c new file +30
-0+ + migrate.vma = vas; + migrate.src = buf; + migrate.dst = migrate.src + npages; + + err = migrate_vma_setup(&migrate); + if (err) + goto err_free; + + /* Raced with another CPU fault, nothing to do */ + if (!migrate.cpages) + goto err_free; + + if (!page) { + for (i = 0; i < npages; ++i) { + if (!(migrate.src[i] & MIGRATE_PFN_MIGRATE)) + continue; + + page = migrate_pfn_to_page(migrate.src[i]); + break; + } + + if (!page) + goto err_finalize; + } + zdd = page->zone_device_data; + ops = zdd->devmem_allocation->ops; + dev = zdd->devmem_allocation->dev; + + err = drm_gpusvm_migrate_populate_ram_pfn(vas, page, npages,&mpages,
+ migrate.src, migrate.dst,
+ start);
+ if (err)
+ gotoerr_finalize;
+
+ err = drm_gpusvm_migrate_map_pages(dev,dma_addr, migrate.dst, npages,
+DMA_FROM_DEVICE);
+ if (err)
+ gotoerr_finalize;drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ + for (i = 0; i < npages; ++i) + pages[i] = migrate_pfn_to_page(migrate.src[i]); + + err = ops->copy_to_ram(pages, dma_addr, npages);drivers/gpu/drm/drm_gpusvm.c line 2048
Quoted 10 lines
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+ if (err) + goto err_finalize; + +err_finalize: + if (err) + drm_gpusvm_migration_unlock_put_pages(npages, migrate.dst); + migrate_vma_pages(&migrate); + migrate_vma_finalize(&migrate); + drm_gpusvm_migrate_unmap_pages(dev, dma_addr, npages, + DMA_FROM_DEVICE);clang for me is throwing:
drivers/gpu/drm/drm_gpusvm.c:2017:7: error: variable 'dev' is used uninitialized whenever 'if' condition is true [-Werror,-Wsometimes- uninitialized]
2017 | if (!page)
| ^~~~~drivers/gpu/drm/drm_gpusvm.c:2047:33: note: uninitialized use occurs here
2047 | drm_gpusvm_migrate_unmap_pages(dev,dma_addr, npages, | ^~~drivers/gpu/drm/drm_gpusvm.c:2017:3: note: remove the 'if' if its condition is always false
2017 | if (!page)
| ^~~~~~~~~~Quoted 8 lines
2018 | goto
err_finalize;
| ~~~~~~~~~~~~~~~~~drivers/gpu/drm/drm_gpusvm.c:1966:20: note: initialize the variable 'dev' to silence this warning
1966 | struct device *dev;
| ^
| = NULL
1 error generated.
Quoted 5 lines
I also reported this issue in the v3 patch, but it doesn't seem to have been fixed in v4 yet.
Yea. The uninitialization is in fact harmless as drm_gpusvm_migrate_unmap_pages is a NOP in this path which I stated. I was unware a tool was complaining about this. I think get this fixed then.
Thanks,
Matt
Quoted 4 lines
Br,
G.G.
Thomas Hellström
maintainerRe: [PATCH v4 06/33] drm/gpusvm: Add support for GPU Shared Virtual Memory
In reply to Matthew BrostQuoted 3,192 lines · Matthew Brost
This patch introduces support for GPU Shared Virtual Memory (SVM) in the
Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also
provides mechanisms for allocating, deallocating, and migrating memory
regions between system RAM and GPU VRAM.This is largely inspired by GPUVM.
v2:
- Take order into account in check pages
- Clearrange->pagesin get pages error
- Drop setting dirty or accessed bit in get pages (Vetter)
- Remove mmap assert for cpu faults
- Drop mmap write lock abuse (Vetter, Christian)
- Decouple zdd from range (Vetter, Oak)
- Adddrm_gpusvm_range_evict, make it work with coherent pages
- Exportdrm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
- mmget/put indrm_gpusvm_evict_to_sram
- Droprange->vram_alloationvariable
- Don't return indrm_gpusvm_evict_to_sramuntil all pages detached
- Don't warn on mixing sram and device pages
- Update kernel doc
- Add coherent page support to get pages
- UseDMA_FROM_DEVICErather thanDMA_BIDIRECTIONAL
- Add structdrm_gpusvm_vramand ops (Thomas)
- Update the range's seqno if the range is valid (Thomas)
- Remove theis_unmappedcheck beforehmm_range_fault(Thomas)
- Usedrm_pagemap(Thomas)
- Dropkfree_mapping(Thomas)
- dma mapp pages under notifier lock (Thomas)
- Remove ctx.prefault
- Remove ctx.mmap_locked
- Add ctx.check_pages
- s/vram/devmem (Thomas)
v3:
- Fix memory leakdrm_gpusvm_range_get_pages
- Only migrate pages with same zdd on CPU fault
- Loop over al VMAs indrm_gpusvm_range_evict
- Make GPUSVM a drm level module
- GPL or MIT license
- Update main kernel doc (Thomas)
- Preferfoo()vs foo for functions in kernel doc (Thomas)
- Prefer functions over macros (Thomas)
- Use unsigned long vs u64 for addresses (Thomas)
- Use standardinterval_tree(Thomas)
- s/drm_gpusvm_migration_put_page/drm_gpusvm_migration_unlock_put_page (Thomas)
- Droperr_outlabel indrm_gpusvm_range_find_or_insert(Thomas)
- Fix kernel doc indrm_gpusvm_range_free_pages(Thomas)
- Newlines between functions defs in header file (Thomas)
- Drop shall language in driver vfunc kernel doc (Thomas)
- Move some static inlines from head to C file (Thomas)
- Don't allocate pages under page lock indrm_gpusvm_migrate_populate_ram_pfn(Thomas)
- Changecheck_pagesto a threshold
v4:
- Fix NULL ptr deref indrm_gpusvm_migrate_populate_ram_pfn(Thomas, Himal)
- Fix check pages threshold
- Check for range being unmapped under notifier lock in get pages (Testing)
- Fix characters per line
- DropWRITE_ONCEforzdd->devmem_allocationassignment (Thomas)
- Use completion fordevmem_allocation->detached(Thomas)
- Make GPU SVM depend onZONE_DEVICE(CI)
- Usehmm_range_faultfor eviction (Thomas)
- Drop zdd worker (Thomas)Cc Simona Vetter <[email protected]>Cc Dave Airlie <[email protected]>Cc Christian König <[email protected]>Cc <[email protected]>Signed-off-by Matthew Brost <[email protected]>Signed-off-by Thomas Hellström <[email protected]>---
drivers/gpu/drm/Kconfig | 9 +
drivers/gpu/drm/Makefile | 1 +
drivers/gpu/drm/drm_gpusvm.c | 2240
++++++++++++++++++++++++++++++++++
include/drm/drm_gpusvm.h | 445 +++++++4 files changed, 2695 insertions(+)create mode 100644
drivers/gpu/drm/drm_gpusvm.c
create mode 100644include/drm/drm_gpusvm.hdrivers/gpu/drm/Kconfig +0
-0@@ -278,6 +278,15 @@ config DRM_GPUVMGPU-VM representation providing helpers to manage a GPUs virtual
address spacedrivers/gpu/drm/Kconfig +7
-0+config DRM_GPUSVM + tristate + depends on DRM + depends on DEVICE_MIGRATION + depends on ZONE_DEVICE + help + GPU-SVM representation providing helpers to manage a GPUsshared
drivers/gpu/drm/Kconfig +2
-0+ virtual memory +config
DRM_BUDDY
tristate
depends on DRMdrivers/gpu/drm/Makefile +0
-0@@ -104,6 +104,7 @@ obj-$(CONFIG_DRM_PANEL_BACKLIGHT_QUIRKS) +=drm_panel_backlight_quirks.o
#
obj-$(CONFIG_DRM_EXEC) += drm_exec.o
obj-$(CONFIG_DRM_GPUVM) += drm_gpuvm.o
+obj-$(CONFIG_DRM_GPUSVM) += drm_gpusvm.o
obj-$(CONFIG_DRM_BUDDY) += drm_buddy.odrivers/gpu/drm/drm_gpusvm.c new file +25
-0@@ -0,0 +1,2240 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* +* Copyright © 2024 Intel Corporation +* +* Authors: +* Matthew Brost <[email protected]> +*/ + +#include <linux/dma-mapping.h> +#include <linux/hmm.h> +#include <linux/memremap.h> +#include <linux/migrate.h> +#include <linux/mm_types.h> +#include <linux/pagemap.h> +#include <linux/slab.h> + +#include <drm/drm_device.h> +#include <drm/drm_gpusvm.h> +#include <drm/drm_pagemap.h> +#include <drm/drm_print.h> + +/** +* DOC: Overview +* +* GPU Shared Virtual Memory (GPU SVM) layer for the Direct Rendering Manager (DRM)
+ *
+ * The GPU SVM layer is a component of the DRM framework designed to manage shared
+ * virtual memory between the CPU and GPU. It enables efficient data exchange and
+ * processing for GPU-accelerated applications by allowing memory sharing and
+ * synchronization between the CPU's and GPU's virtual address spaces.
+ *
+ * Key GPU SVM Components:
+ * - Notifiers: Notifiers: Used for tracking memory intervals and notifying the
+ * GPU of changes, notifiers are sized based on a GPU
SVM
+ * initialization parameter, with a recommendation of
512M or
+ * larger. They maintain a Red-BlacK tree and a list of
ranges that
+ * fall within the notifier interval. Notifiers are
tracked within
+ * a GPU SVM Red-BlacK tree and list and are
dynamically inserted
+ * or removed as ranges within the interval are created
or
+ * destroyed.
+ * - Ranges: Represent memory ranges mapped in a DRM device and managed
+ * by GPU SVM. They are sized based on an array of chunk
sizes, which
+ * is a GPU SVM initialization parameter, and the CPU
address space.
+ * Upon GPU fault, the largest aligned chunk that fits
within the
+ * faulting CPU address space is chosen for the range
size. Ranges are
+ * expected to be dynamically allocated on GPU fault and
removed on an
+ * MMU notifier UNMAP event. As mentioned above, ranges
are tracked in
+ * a notifier's Red-Black tree.
+ * - Operations: Define the interface for driver-specific GPU SVM operations
+ * such as range allocation, notifier allocation, and
+ * invalidations.
+ * - Device Memory Allocations: Embedded structure containing enough information
+ * for GPU SVM to migrate to / from device memory.
+ * - Device Memory Operations: Define the interface for driver- specific device
+ * memory operations release memory, populate pfns,
+ * and copy to / from device memory.
+ *
+ * This layer provides interfaces for allocating, mapping, migrating, and
+ * releasing memory ranges between the CPU and GPU. It handles all core memory
+ * management interactions (DMA mapping, HMM, and migration) and provides
+ * driver-specific virtual functions (vfuncs). This infrastructure is sufficient
+ * to build the expected driver components for an SVM implementation as detailed
+ * below.
+ *
+ * Expected Driver Components:
+ * - GPU page fault handler: Used to create ranges and notifiers based on the
+ * fault address, optionally migrate the
range to
+ * device memory, and create GPU bindings.
+ * - Garbage collector: Used to unmap and destroy GPU bindings for ranges.
+ * Ranges are expected to be added to the
garbage collector
+ * upon aMMU_NOTIFY_UNMAPevent in notifier
callback.
+ * - Notifier callback: Used to invalidate and DMA unmap GPU bindings for
+ * ranges.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +/** +* DOC: Locking +* +* GPU SVM handles locking for core MM interactions, i.e., it locks/unlocks the
+ * mmap lock as needed.
+ *
+ * GPU SVM introduces a global notifier lock, which safeguards the notifier's
+ * range RB tree and list, as well as the range's DMA mappings and sequence
+ * number. GPU SVM manages all necessary locking and unlocking operations,
+ * except for the recheck range's pages being valid
+ * (drm_gpusvm_range_pages_valid) when the driver is committing GPU bindings. This
+ * lock corresponds to the 'driver->update' lock mentioned in the HMM
+ * documentation (TODO: Link). Future revisions may transition from a GPU SVM
+ * global lock to a per-notifier lock if finer-grained locking is deemed
+ * necessary.
+ *
+ * In addition to the locking mentioned above, the driver should implement a
+ * lock to safeguard core GPU SVM function calls that modify state, such as
+ *drm_gpusvm_range_find_or_insertanddrm_gpusvm_range_remove. This lock is
+ * denoted as 'driver_svm_lock' in code examples. Finer grained driver side
+ * locking should also be possible for concurrent GPU fault processing within a
+ * single GPU SVM. The 'driver_svm_lock' can be viadrm_gpusvm_driver_set_lock
+ * to add annotations to GPU SVM.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +/** +* DOC: Migration +* +* The migration support is quite simple, allowing migration between RAM and
+ * device memory at the range granularity. For example, GPU SVM currently does not
+ * support mixing RAM and device memory pages within a range. This means that upon GPU
+ * fault, the entire range can be migrated to device memory, and upon CPU fault, the
+ * entire range is migrated to RAM. Mixed RAM and device memory storage within a range
+ * could be added in the future if required.
+ *
+ * The reasoning for only supporting range granularity is as
follows: it
+ * simplifies the implementation, and range sizes are driver-defined and should
+ * be relatively small.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +/** +* DOC: Partial Unmapping of Ranges +* +* Partial unmapping of ranges (e.g., 1M out of 2M is unmapped by CPU resulting
+ * inMMU_NOTIFY_UNMAPevent) presents several challenges, with the main one
+ * being that a subset of the range still has CPU and GPU mappings. If the
+ * backing store for the range is in device memory, a subset of the backing store has
+ * references. One option would be to split the range and device memory backing store,
+ * but the implementation for this would be quite complicated. Given that
+ * partial unmappings are rare and driver-defined range sizes are relatively
+ * small, GPU SVM does not support splitting of ranges.
+ *
+ * With no support for range splitting, upon partial unmapping of a range, the
+ * driver is expected to invalidate and destroy the entire range. If the range
+ * has device memory as its backing, the driver is also expected to migrate any
+ * remaining pages back to RAM.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +/** +* DOC: Examples +* +* This section provides three examples of how to build the expected driver
+ * components: the GPU page fault handler, the garbage collector, and the
+ * notifier callback.
+ *
+ * The generic code provided does not include logic for complex migration
+ * policies, optimized invalidations, fined grained driver locking, or other
+ * potentially required driver locking (e.g., DMA-resv locks).
+ *
+ * 1) GPU page fault handler
+ *
+ * int driver_bind_range(structdrm_gpusvm*gpusvm, structdrm_gpusvm_range*range)
+ * {
+ * int err = 0;
+ *
+ * driver_alloc_and_setup_memory_for_bind(gpusvm,
range);
+ *
+ *drm_gpusvm_notifier_lock(gpusvm);
+ * if (drm_gpusvm_range_pages_valid(range))
+ *driver_commit_bind(gpusvm, range);
+ * else
+ * err =-EAGAIN;
+ *drm_gpusvm_notifier_unlock(gpusvm);
+ *
+ * return err;
+ * }
+ *
+ * int driver_gpu_fault(structdrm_gpusvm*gpusvm, unsigned
longfault_addr,
+ * unsigned longgpuva_start, unsigned
longgpuva_end)
+ * {
+ * structdrm_gpusvm_ctxctx = {};
+ * int err;
+ *
+ *driver_svm_lock();
+ * retry:
+ * // Always process UNMAPs first so view of GPU SVM
ranges is current
+ *driver_garbage_collector(gpusvm);
+ *
+ * range = drm_gpusvm_range_find_or_insert(gpusvm,fault_addr,
+ *gpuva_start,gpuva_end,
+ * &ctx);
+ * if (IS_ERR(range)) {
+ * err =PTR_ERR(range);
+ * goto unlock;
+ * }
+ *
+ * if (driver_migration_policy(range)) {
+ * devmem =driver_alloc_devmem();
+ * err = drm_gpusvm_migrate_to_devmem(gpusvm,
range,
+ *devmem_allocation,
+ * &ctx);
+ * if (err) // CPU mappings may have
changed
+ * goto retry;
+ * }
+ *
+ * err = drm_gpusvm_range_get_pages(gpusvm, range,
&ctx);
+ * if (err ==-EOPNOTSUPP|| err ==-EFAULT|| err == -
EPERM) { // CPU mappings changed
+ * if (err ==-EOPNOTSUPP)
+ * drm_gpusvm_range_evict(gpusvm,
range);
+ * goto retry;
+ * } else if (err) {
+ * goto unlock;
+ * }
+ *
+ * err =driver_bind_range(gpusvm, range);
+ * if (err ==-EAGAIN) // CPU mappings changed
+ * goto retry
+ *
+ * unlock:
+ *driver_svm_unlock();
+ * return err;
+ * }
+ *
+ * 2) Garbage Collector.
+ *
+ * void __driver_garbage_collector(structdrm_gpusvm*gpusvm,
+ * structdrm_gpusvm_range
*range)
+ * {
+ *assert_driver_svm_locked(gpusvm);
+ *
+ * // Partial unmap, migrate any remaining device
memory pages back to RAM
+ * if (range->flags.partial_unmap)
+ *drm_gpusvm_range_evict(gpusvm, range);
+ *
+ *driver_unbind_range(range);
+ *drm_gpusvm_range_remove(gpusvm, range);
+ * }
+ *
+ * voiddriver_garbage_collector(struct drm_gpusvm *gpusvm)
+ * {
+ *assert_driver_svm_locked(gpusvm);
+ *
+ *for_each_range_in_garbage_collector(gpusvm, range)
+ *__driver_garbage_collector(gpusvm, range);
+ * }
+ *
+ * 3) Notifier callback.
+ *
+ * void driver_invalidation(structdrm_gpusvm*gpusvm,
+ * structdrm_gpusvm_notifier
*notifier,
+ * const structmmu_notifier_range*mmu_range)
+ * {
+ * structdrm_gpusvm_ctxctx = { .in_notifier = true,
};
+ * structdrm_gpusvm_range*range = NULL;
+ *
+ * driver_invalidate_device_pages(gpusvm, mmu_range-Quoted 1 line
start,
mmu_range->end);+ *
+ * drm_gpusvm_for_each_range(range, notifier,mmu_range->start,
+ *mmu_range->end) {
+ * drm_gpusvm_range_unmap_pages(gpusvm, range,
&ctx);
+ *
+ * if (mmu_range->event!=MMU_NOTIFY_UNMAP)
+ * continue;
+ *
+ * drm_gpusvm_range_set_unmapped(range,mmu_range);
+ *driver_garbage_collector_add(gpusvm, range);
+ * }
+ * }
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ +/** +* npages_in_range() - Calculate the number of pages in a given range
+ * @start: The start address of the range
+ * @end: The end address of the range
+ *
+ * This macro calculates the number of pages in a given memory range,
+ * specified by the start and end addresses. It divides the difference
+ * between the end and start addresses by the page size (PAGE_SIZE) to
+ * determine the number of pages in the range.
+ *
+ * Returns: The number of pages in the specified range.
+ */drivers/gpu/drm/drm_gpusvm.c new file +15
-0+static unsigned long +npages_in_range (unsigned long start,unsigned long end) +{ +return (end - start) >>PAGE_SHIFT ; +} + +/** +* struct drm_gpusvm_zdd - GPU SVM zone device data +* +* @refcount: Reference count for the zdd +* @devmem_allocation: device memory allocation +* @device_private_page_owner: Device private pages owner +* +* This structure serves as a generic wrapper installed in +* page->zone_device_data. It provides infrastructure for looking up a device
+ * memory allocation upon CPU page fault and asynchronously releasing device
+ * memory once the CPU has no page references. Asynchronous release is useful
+ * because CPU page references can be dropped in IRQ contexts, while releasing
+ * device memory likely requires sleeping locks.
+ */drivers/gpu/drm/drm_gpusvm.c new file +11
-0+struct drm_gpusvm_zdd { +struct kref refcount; +struct drm_gpusvm_devmem *devmem_allocation; +void *device_private_page_owner; +}; + +/** +* drm_gpusvm_zdd_alloc() - Allocate a zdd structure. +* @device_private_page_owner: Device private pages owner +* +* This function allocates and initializes a new zdd structure. It sets up the
+ * reference count and initializes the destroy work.
+ *
+ * Returns:
+ * Pointer to the allocated zdd on success,ERR_PTR()on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +21
-0+static struct drm_gpusvm_zdd * +drm_gpusvm_zdd_alloc (void *device_private_page_owner) +{ +struct drm_gpusvm_zdd *zdd; + + zdd =kmalloc (sizeof (*zdd),GFP_KERNEL ); +if (!zdd) +return NULL ; + +kref_init (&zdd->refcount); + zdd->devmem_allocation =NULL ; + zdd->device_private_page_owner = device_private_page_owner; + +return zdd; +} + +/** +* drm_gpusvm_zdd_get() - Get a reference to a zdd structure. +* @zdd: Pointer to the zdd structure. +* +* This function increments the reference count of the provided zdd structure.
+ *
+ * Returns: Pointer to the zdd structure.
+ */
+static structdrm_gpusvm_zdd*drm_gpusvm_zdd_get(structdrm_gpusvm_zdd*zdd)drivers/gpu/drm/drm_gpusvm.c new file +10
-0+{ +kref_get (&zdd->refcount); +return zdd; +} + +/** +* drm_gpusvm_zdd_destroy() - Destroy a zdd structure. +* @ref: Pointer to the reference count structure. +* +* This function queues the destroy_work of the zdd for asynchronous destruction.
+ */drivers/gpu/drm/drm_gpusvm.c new file +19
-0+static void drm_gpusvm_zdd_destroy (struct kref *ref) +{ +struct drm_gpusvm_zdd *zdd = + container_of(ref, struct drm_gpusvm_zdd, refcount); +struct drm_gpusvm_devmem *devmem = zdd->devmem_allocation; + +if (devmem) { +complete_all (&devmem->detached); +if (devmem->ops->devmem_release) + devmem->ops->devmem_release (devmem); + } +kfree (zdd); +} + +/** +* drm_gpusvm_zdd_put() - Put a zdd reference. +* @zdd: Pointer to the zdd structure. +* +* This function decrements the reference count of the provided zdd structure
+ * and schedules its destruction if the count drops to zero.
+ */drivers/gpu/drm/drm_gpusvm.c new file +7
-0+static void drm_gpusvm_zdd_put (struct drm_gpusvm_zdd *zdd) +{ +kref_put (&zdd->refcount, drm_gpusvm_zdd_destroy); +} + +/** +* drm_gpusvm_range_find() - Find GPU SVM range from GPU SVM notifier
+ * @notifier: Pointer to the GPU SVM notifier structure.
+ * @start: Start address of the range
+ * @end: End address of the range
+ *
+ * Returns: A pointer to thedrm_gpusvm_rangeif found or NULL
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+struct drm_gpusvm_range * +drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, unsignedlong start,
drivers/gpu/drm/drm_gpusvm.c new file +8
-1+ unsigned long end) +{ +struct interval_tree_node *itree; + + itree = interval_tree_iter_first(¬ifier->root, start, end - 1); + + if (itree) + return container_of(itree, struct drm_gpusvm_range,itree);
drivers/gpu/drm/drm_gpusvm.c new file +7
-0+else +return NULL ; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_find); + +/** +* drm_gpusvm_for_each_range_safe() - Safely iterate over GPU SVM ranges in a notifier
+ * @range__: Iterator variable for the ranges
+ * @next__: Iterator variable for the ranges temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier while
+ * removing ranges from it.
+ */
+#define drm_gpusvm_for_each_range_safe(range__,next__,notifier__,start__,end__) \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ for ((range__) = drm_gpusvm_range_find((notifier__),(
start__), (end__)), \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (next__) =
__drm_gpusvm_range_next(range__); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (range__) && (range__->itree.start <(
end__)); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (range__) = (next__), (next__) =
__drm_gpusvm_range_next(range__))drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ +/** +* __drm_gpusvm_notifier_next() - get the next drm_gpusvm_notifier in the list
+ * @notifier: a pointer to the currentdrm_gpusvm_notifier
+ *
+ * Returns: A pointer to the nextdrm_gpusvm_notifierif available, or NULL if
+ * the current notifier is the last one or if the input notifier is
+ * NULL.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+static struct drm_gpusvm_notifier * +__drm_gpusvm_notifier_next(struct drm_gpusvm_notifier *notifier) +{ + if (notifier && !list_is_last(¬ifier->entry, + ¬ifier->gpusvm-Quoted 1 line
notifier_list))drivers/gpu/drm/drm_gpusvm.c new file +7
-0+return list_next_entry (notifier, entry); + +return NULL ; +} + +static struct drm_gpusvm_notifier * +notifier_iter_first(struct rb_root_cached *root, unsigned longstart,
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ unsigned long last) +{ +struct interval_tree_node *itree; + + itree =interval_tree_iter_first (root, start, last); + +if (itree) + return container_of(itree, struct
drm_gpusvm_notifier, itree);drivers/gpu/drm/drm_gpusvm.c new file +6
-0+else +return NULL ; +} + +/** +* drm_gpusvm_for_each_notifier() - Iterate over GPU SVM notifiers in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm.
+ */
+#define drm_gpusvm_for_each_notifier(notifier__,gpusvm__,start__,end__) \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,(
start__), (end__) - 1); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (notifier__) && (notifier__->itree.start <(
end__)); \drivers/gpu/drm/drm_gpusvm.c new file +4
-0+ (notifier__) = __drm_gpusvm_notifier_next(notifier__)) + +/** +* drm_gpusvm_for_each_notifier_safe() - Safely iterate over GPU SVM notifiers in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @next__: Iterator variable for the notifiers temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm while
+ * removing notifiers from it.
+ */
+#define drm_gpusvm_for_each_notifier_safe(notifier__,next__,gpusvm__,start__,end__) \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,(
start__), (end__) - 1), \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (next__) =
__drm_gpusvm_notifier_next(notifier__); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (notifier__) && (notifier__->itree.start <(
end__)); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (notifier__) = (next__), (next__) =
__drm_gpusvm_notifier_next(notifier__))drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ +/** +* drm_gpusvm_notifier_invalidate() - Invalidate a GPU SVM notifier. +* @mni: Pointer to the mmu_interval_notifier structure. +* @mmu_range: Pointer to the mmu_notifier_range structure. +* @cur_seq: Current sequence number. +* +* This function serves as a generic MMU notifier for GPU SVM. It sets the MMU
+ * notifier sequence number and calls the driver invalidate vfunc under
+ *gpusvm->notifier_lock.
+ *
+ * Returns:
+ * true if the operation succeeds, false otherwise.
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+static bool +drm_gpusvm_notifier_invalidate (struct mmu_interval_notifier *mni, + const struct mmu_notifier_range
*mmu_range,drivers/gpu/drm/drm_gpusvm.c new file +19
-0+ unsigned long cur_seq) +{ +struct drm_gpusvm_notifier *notifier = +container_of (mni,typeof (*notifier), notifier); +struct drm_gpusvm *gpusvm = notifier->gpusvm; + +if (!mmu_notifier_range_blockable (mmu_range)) +return false; + +down_write (&gpusvm->notifier_lock); +mmu_interval_set_seq (mni, cur_seq); + gpusvm->ops->invalidate (gpusvm, notifier, mmu_range); +up_write (&gpusvm->notifier_lock); + +return true; +} + +/** +* drm_gpusvm_notifier_ops - MMU interval notifier operations for GPU SVM
+ */
+static const structmmu_interval_notifier_opsdrm_gpusvm_notifier_ops= {drivers/gpu/drm/drm_gpusvm.c new file +15
-0+ .invalidate = drm_gpusvm_notifier_invalidate, +}; + +/** +* drm_gpusvm_init() - Initialize the GPU SVM. +* @gpusvm: Pointer to the GPU SVM structure. +* @name: Name of the GPU SVM. +* @drm: Pointer to the DRM device structure. +* @mm: Pointer to the mm_struct for the address space. +* @device_private_page_owner: Device private pages owner. +* @mm_start: Start address of GPU SVM. +* @mm_range: Range of the GPU SVM. +* @notifier_size: Size of individual notifiers. +* @ops: Pointer to the operations structure for GPU SVM. +* @chunk_sizes: Pointer to the array of chunk sizes used in range allocation.
+ * Entries should be powers of 2 in descending order with last
+ * entry beingSZ_4K.
+ * @num_chunks: Number of chunks.
+ *
+ * This function initializes the GPU SVM.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+int drm_gpusvm_init (struct drm_gpusvm *gpusvm, + const char *name, struct drm_device *drm, + struct mm_struct *mm, void
*device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +4
-0+ unsigned long mm_start, unsigned long mm_range, + unsigned long notifier_size, + const struct drm_gpusvm_ops *ops, + const unsigned long *chunk_sizes, int
num_chunks)drivers/gpu/drm/drm_gpusvm.c new file +8
-0+{ +if (!ops->invalidate || !num_chunks) +return -EINVAL ; + + gpusvm->name = name; + gpusvm->drm = drm; + gpusvm->mm = mm; + gpusvm->device_private_page_owner =
device_private_page_owner;drivers/gpu/drm/drm_gpusvm.c new file +31
-0+ gpusvm->mm_start = mm_start; + gpusvm->mm_range = mm_range; + gpusvm->notifier_size = notifier_size; + gpusvm->ops = ops; + gpusvm->chunk_sizes = chunk_sizes; + gpusvm->num_chunks = num_chunks; + +mmgrab (mm); + gpusvm->root =RB_ROOT_CACHED ; +INIT_LIST_HEAD (&gpusvm->notifier_list); + +init_rwsem (&gpusvm->notifier_lock); + +fs_reclaim_acquire (GFP_KERNEL ); +might_lock (&gpusvm->notifier_lock); +fs_reclaim_release (GFP_KERNEL ); + +#ifdef CONFIG_LOCKDEP + gpusvm->lock_dep_map =NULL ; +#endif + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_init); + +/** +* drm_gpusvm_notifier_find() - Find GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM structure +* @fault_addr: Fault address +* +* This function finds the GPU SVM notifier associated with the fault address.
+ *
+ * Returns:
+ * Pointer to the GPU SVM notifier on success, NULL otherwise.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+static struct drm_gpusvm_notifier * +drm_gpusvm_notifier_find (struct drm_gpusvm *gpusvm, +unsigned long fault_addr) +{ + return notifier_iter_first(&gpusvm->root, fault_addr,
fault_addr+ 1);drivers/gpu/drm/drm_gpusvm.c new file +4
-0+} + +/** +* to_drm_gpusvm_notifier() - retrieve the container struct for a given rbtree node
+ * @node: a pointer to the rbtree node embedded within adrm_gpusvm_notifierstruct
+ *
+ * Returns: A pointer to the containingdrm_gpusvm_notifierstructure.
+ */
+static structdrm_gpusvm_notifier*to_drm_gpusvm_notifier(structrb_node*node)drivers/gpu/drm/drm_gpusvm.c new file +2
-0+{ + return container_of(node, struct drm_gpusvm_notifier,itree.rb);
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+} + +/** +* drm_gpusvm_notifier_insert() - Insert GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier structure +* +* This function inserts the GPU SVM notifier into the GPU SVM RB tree and list.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static void drm_gpusvm_notifier_insert (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier*notifier)
drivers/gpu/drm/drm_gpusvm.c new file +21
-0+{ +struct rb_node *node; +struct list_head *head; + +interval_tree_insert (¬ifier->itree, &gpusvm->root); + + node =rb_prev (¬ifier->itree.rb); +if (node) + head = &(to_drm_gpusvm_notifier (node))->entry; +else + head = &gpusvm->notifier_list; + +list_add (¬ifier->entry, head); +} + +/** +* drm_gpusvm_notifier_remove() - Remove GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM tructure +* @notifier: Pointer to the GPU SVM notifier structure +* +* This function removes the GPU SVM notifier from the GPU SVM RB tree and list.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static void drm_gpusvm_notifier_remove (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier*notifier)
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+{ +interval_tree_remove (¬ifier->itree, &gpusvm->root); +list_del (¬ifier->entry); +} + +/** +* drm_gpusvm_fini() - Finalize the GPU SVM. +* @gpusvm: Pointer to the GPU SVM structure. +* +* This function finalizes the GPU SVM by cleaning up any remaining ranges and
+ * notifiers, and dropping a reference to struct MM.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+void drm_gpusvm_fini (struct drm_gpusvm *gpusvm) +{ +struct drm_gpusvm_notifier *notifier, *next; + + drm_gpusvm_for_each_notifier_safe(notifier, next, gpusvm, 0,
LONG_MAX) {drivers/gpu/drm/drm_gpusvm.c new file +4
-0+struct drm_gpusvm_range *range, *__next; + +/* +* Remove notifier first to avoid racing with any invalidation
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+*/ +mmu_interval_notifier_remove (¬ifier->notifier); + notifier->flags.removed = true; + + drm_gpusvm_for_each_range_safe(range, __next,notifier, 0,
drivers/gpu/drm/drm_gpusvm.c new file +15
-0+ LONG_MAX) +drm_gpusvm_range_remove (gpusvm ,range ); + } + +mmdrop (gpusvm->mm); +WARN_ON (!RB_EMPTY_ROOT (&gpusvm->root.rb_root)); +} +EXPORT_SYMBOL_GPL (drm_gpusvm_fini); + +/** +* drm_gpusvm_notifier_alloc() - Allocate GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM structure +* @fault_addr: Fault address +* +* This function allocates and initializes the GPU SVM notifier structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM notifier on success,ERR_PTR()on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static struct drm_gpusvm_notifier * +drm_gpusvm_notifier_alloc(struct drm_gpusvm *gpusvm, unsigned long
fault_addr)drivers/gpu/drm/drm_gpusvm.c new file +13
-0+{ +struct drm_gpusvm_notifier *notifier; + +if (gpusvm->ops->notifier_alloc) + notifier = gpusvm->ops->notifier_alloc (); +else + notifier =kzalloc (sizeof (*notifier),GFP_KERNEL ); + +if (!notifier) +return ERR_PTR (-ENOMEM ); + + notifier->gpusvm = gpusvm; + notifier->itree.start = ALIGN_DOWN(fault_addr, gpusvm-Quoted 1 line
notifier_size);drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ notifier->itree.last = ALIGN(fault_addr + 1, gpusvm-Quoted 1 line
notifier_size) - 1;drivers/gpu/drm/drm_gpusvm.c new file +16
-0+INIT_LIST_HEAD (¬ifier->entry); + notifier->root =RB_ROOT_CACHED ; +INIT_LIST_HEAD (¬ifier->range_list); + +return notifier; +} + +/** +* drm_gpusvm_notifier_free() - Free GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier structure +* +* This function frees the GPU SVM notifier structure. +*/ +static void drm_gpusvm_notifier_free (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier*notifier)
drivers/gpu/drm/drm_gpusvm.c new file +11
-0+{ +WARN_ON (!RB_EMPTY_ROOT (¬ifier->root.rb_root)); + +if (gpusvm->ops->notifier_free) + gpusvm->ops->notifier_free (notifier); +else +kfree (notifier); +} + +/** +* to_drm_gpusvm_range() - retrieve the container struct for a given rbtree node
+ * @node: a pointer to the rbtree node embedded within adrm_gpusvm_rangestruct
+ *
+ * Returns: A pointer to the containingdrm_gpusvm_rangestructure.
+ */
+static structdrm_gpusvm_range*to_drm_gpusvm_range(structrb_node*node)drivers/gpu/drm/drm_gpusvm.c new file +2
-0+{ + return container_of(node, struct drm_gpusvm_range,itree.rb);
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+} + +/** +* drm_gpusvm_range_insert() - Insert GPU SVM range +* @notifier: Pointer to the GPU SVM notifier structure +* @range: Pointer to the GPU SVM range structure +* +* This function inserts the GPU SVM range into the notifier RB tree and list.
+ */
+static void drm_gpusvm_range_insert(structdrm_gpusvm_notifier*notifier,drivers/gpu/drm/drm_gpusvm.c new file +24
-0+ struct drm_gpusvm_range *range) +{ +struct rb_node *node; +struct list_head *head; + +drm_gpusvm_notifier_lock (notifier->gpusvm); +interval_tree_insert (&range->itree, ¬ifier->root); + + node =rb_prev (&range->itree.rb); +if (node) + head = &(to_drm_gpusvm_range (node))->entry; +else + head = ¬ifier->range_list; + +list_add (&range->entry, head); +drm_gpusvm_notifier_unlock (notifier->gpusvm); +} + +/** +* __drm_gpusvm_range_remove() - Remove GPU SVM range +* @notifier: Pointer to the GPU SVM notifier structure +* @range: Pointer to the GPU SVM range structure +* +* This macro removes the GPU SVM range from the notifier RB tree and list.
+ */
+static void __drm_gpusvm_range_remove(structdrm_gpusvm_notifier*notifier,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct drm_gpusvm_range*range)
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+{ +interval_tree_remove (&range->itree, ¬ifier->root); +list_del (&range->entry); +} + +/** +* drm_gpusvm_range_alloc() - Allocate GPU SVM range +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier structure +* @fault_addr: Fault address +* @chunk_size: Chunk size +* @migrate_devmem: Flag indicating whether to migrate device memory +* +* This function allocates and initializes the GPU SVM range structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM range on success,ERR_PTR()on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+static struct drm_gpusvm_range * +drm_gpusvm_range_alloc (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + unsigned long fault_addr, unsigned long
chunk_size,drivers/gpu/drm/drm_gpusvm.c new file +32
-0+ bool migrate_devmem) +{ +struct drm_gpusvm_range *range; + +if (gpusvm->ops->range_alloc) + range = gpusvm->ops->range_alloc (gpusvm); +else + range =kzalloc (sizeof (*range),GFP_KERNEL ); + +if (!range) +return ERR_PTR (-ENOMEM ); + +kref_init (&range->refcount); + range->gpusvm = gpusvm; + range->notifier = notifier; + range->itree.start =ALIGN_DOWN (fault_addr, chunk_size); + range->itree.last =ALIGN (fault_addr +1 , chunk_size) -1 ; +INIT_LIST_HEAD (&range->entry); + range->notifier_seq =LONG_MAX ; + range->flags.migrate_devmem = migrate_devmem ?1 :0 ; + +return range; +} + +/** +* drm_gpusvm_check_pages() - Check pages +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier structure +* @start: Start address +* @end: End address +* +* Check if pages between start and end have been faulted in on the CPU. Use to
+ * prevent migration of pages without CPU backing store.
+ *
+ * Returns:
+ * True if pages have been faulted into CPU, False otherwise
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static bool drm_gpusvm_check_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier*notifier,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long start, unsignedlong end)
drivers/gpu/drm/drm_gpusvm.c new file +7
-0+{ + struct hmm_range hmm_range = { + .default_flags = 0, + .notifier = ¬ifier->notifier, + .start = start, + .end = end, + .dev_private_owner = gpusvm-Quoted 1 line
device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ }; +unsigned long timeout = + jiffies +
msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);drivers/gpu/drm/drm_gpusvm.c new file +11
-0+unsigned long *pfns; +unsigned long npages =npages_in_range (start, end); +int err, i; + +mmap_assert_locked (gpusvm->mm); + + pfns =kvmalloc_array (npages,sizeof (*pfns),GFP_KERNEL ); +if (!pfns) +return false; + + hmm_range.notifier_seq = mmu_interval_read_begin(¬ifier-Quoted 1 line
notifier);
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+ hmm_range.hmm_pfns = pfns; + +while (true) { + err =hmm_range_fault (&hmm_range); +if (err == -EBUSY ) { +if (time_after (jiffies, timeout)) +break ; + + hmm_range.notifier_seq = + mmu_interval_read_begin(¬ifier-Quoted 1 line
notifier);
drivers/gpu/drm/drm_gpusvm.c new file +22
-0+continue ; + } +break ; + } +if (err) + goto err_free; + +for (i =0 ; i < npages;) { +if (!(pfns[i] &HMM_PFN_VALID )) { + err = -EFAULT ; + goto err_free; + } + i +=0x1 <<hmm_pfn_to_map_order (pfns[i]); + } + +err_free: +kvfree (pfns); +return err ? false : true; +} + +/** +* drm_gpusvm_range_chunk_size() - Determine chunk size for GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @vas: Pointer to the virtual memory area structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @check_pages_threshold: Check CPU pages for present threshold
+ *
+ * This function determines the chunk size for the GPU SVM range based on the
+ * fault address, GPU SVM chunk sizes, existing GPU SVM ranges, and the virtual
+ * memory area boundaries.
+ *
+ * Returns:
+ * Chunk size on success,LONG_MAXon failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +15
-0+static unsigned long +drm_gpusvm_range_chunk_size (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + struct vm_area_struct *vas, + unsigned long fault_addr, + unsigned long gpuva_start, + unsigned long gpuva_end, + unsigned long check_pages_threshold) +{ +unsigned long start, end; +int i =0 ; + +retry: +for (; i < gpusvm->num_chunks; ++i) { + start = ALIGN_DOWN(fault_addr, gpusvm-Quoted 1 line
chunk_sizes[i]);
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+ end =ALIGN (fault_addr +1 , gpusvm->chunk_sizes[i]); + +if (start >= vas->vm_start && end <= vas->vm_end && + start >= notifier->itree.start && + end <= notifier->itree.last + 1 && + start >= gpuva_start && end <= gpuva_end) +break ; + } + +if (i == gpusvm->num_chunks) +return LONG_MAX ; + +/* +* If allocation more than page, ensure not to overlap with existing
drivers/gpu/drm/drm_gpusvm.c new file +13
-0+* ranges. +*/ +if (end - start !=SZ_4K ) { +struct drm_gpusvm_range *range; + + range =drm_gpusvm_range_find (notifier, start, end); +if (range) { + ++i; + goto retry; + } + +/* +* XXX: Only create range on pages CPU has faulted in. Without
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* this check, or prefault, on BMG '
xe_exec_system_allocator--rdrivers/gpu/drm/drm_gpusvm.c new file +1
-0+* process-many-malloc' fails. In the failure case, each process
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* mallocs 16k but the CPU VMA is ~128k which results in 64k SVM
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* ranges. When migrating the SVM ranges, some processes fail in
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* drm_gpusvm_migrate_to_devmem with 'migrate.cpages != npages'
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* and then upon drm_gpusvm_range_get_pages device pages from
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* other processes are collected + faulted in which creates all
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* sorts of problems. Unsure exactly how this happening, also
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* problem goes away if 'xe_exec_system_allocator -- r
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* process-many-malloc' mallocs at least 64k at a time.
drivers/gpu/drm/drm_gpusvm.c new file +3
-0+*/ +if (end - start <= check_pages_threshold && + !drm_gpusvm_check_pages(gpusvm, notifier, start,end)) {
drivers/gpu/drm/drm_gpusvm.c new file +17
-0+ ++i; + goto retry; + } + } + +return end - start; +} + +/** +* drm_gpusvm_range_find_or_insert() - Find or insert GPU SVM range +* @gpusvm: Pointer to the GPU SVM structure +* @fault_addr: Fault address +* @gpuva_start: Start address of GPUVA which mirrors CPU +* @gpuva_end: End address of GPUVA which mirrors CPU +* @ctx: GPU SVM context +* +* This function finds or inserts a newly allocated a GPU SVM range based on the
+ * fault address. Caller must hold a lock to protect range lookup and insertion.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range on success,ERR_PTR()on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +28
-0+struct drm_gpusvm_range * +drm_gpusvm_range_find_or_insert (struct drm_gpusvm *gpusvm, +unsigned long fault_addr, +unsigned long gpuva_start, +unsigned long gpuva_end, +const struct drm_gpusvm_ctx *ctx) +{ +struct drm_gpusvm_notifier *notifier; +struct drm_gpusvm_range *range; +struct mm_struct *mm = gpusvm->mm; +struct vm_area_struct *vas; +bool notifier_alloc = false; +unsigned long chunk_size; +int err; +bool migrate_devmem; + +drm_gpusvm_driver_lock_held (gpusvm); + +if (fault_addr < gpusvm->mm_start || + fault_addr > gpusvm->mm_start + gpusvm->mm_range) +return ERR_PTR (-EINVAL ); + +if (!mmget_not_zero (mm)) +return ERR_PTR (-EFAULT ); + + notifier =drm_gpusvm_notifier_find (gpusvm, fault_addr); +if (!notifier) { + notifier = drm_gpusvm_notifier_alloc(gpusvm,
fault_addr);drivers/gpu/drm/drm_gpusvm.c new file +6
-0+if (IS_ERR (notifier)) { + err =PTR_ERR (notifier); + goto err_mmunlock; + } + notifier_alloc = true; + err = mmu_interval_notifier_insert(¬ifier-Quoted 1 line
notifier,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ mm, notifier-Quoted 1 line
itree.start,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ notifier-Quoted 1 line
itree.last -
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ notifier-Quoted 1 line
itree.start + 1,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+
&drm_gpusvm_notifier_ops);drivers/gpu/drm/drm_gpusvm.c new file +18
-0+if (err) + goto err_notifier; + } + +mmap_read_lock (mm); + + vas =vma_lookup (mm, fault_addr); +if (!vas) { + err = -ENOENT ; + goto err_notifier_remove; + } + +if (!ctx->read_only && !(vas->vm_flags &VM_WRITE )) { + err = -EPERM ; + goto err_notifier_remove; + } + + range = drm_gpusvm_range_find(notifier, fault_addr,
fault_addr+ 1);drivers/gpu/drm/drm_gpusvm.c new file +4
-0+if (range) + goto out_mmunlock; +/* +* XXX: Short-circuiting migration based on migrate_vma_* current
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* limitations. If/when migrate_vma_* add more support, this logic will
drivers/gpu/drm/drm_gpusvm.c new file +6
-0+* have to change. +*/ + migrate_devmem = ctx->devmem_possible && +vma_is_anonymous (vas) && !is_vm_hugetlb_page (vas); + + chunk_size = drm_gpusvm_range_chunk_size(gpusvm, notifier,vas,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ fault_addr,
gpuva_start,drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ gpuva_end, + ctx-Quoted 1 line
check_pages_threshold);drivers/gpu/drm/drm_gpusvm.c new file +6
-0+if (chunk_size ==LONG_MAX ) { + err = -EINVAL ; + goto err_notifier_remove; + } + + range = drm_gpusvm_range_alloc(gpusvm, notifier, fault_addr,
chunk_size,drivers/gpu/drm/drm_gpusvm.c new file +31
-0+ migrate_devmem); +if (IS_ERR (range)) { + err =PTR_ERR (range); + goto err_notifier_remove; + } + +drm_gpusvm_range_insert (notifier, range); +if (notifier_alloc) +drm_gpusvm_notifier_insert (gpusvm, notifier); + +out_mmunlock: +mmap_read_unlock (mm); +mmput (mm); + +return range; + +err_notifier_remove: +mmap_read_unlock (mm); +if (notifier_alloc) +mmu_interval_notifier_remove (¬ifier->notifier); +err_notifier: +if (notifier_alloc) +drm_gpusvm_notifier_free (gpusvm, notifier); +err_mmunlock: +mmput (mm); +return ERR_PTR (err); +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_find_or_insert); + +/** +* __drm_gpusvm_range_unmap_pages() - Unmap pages associated with a GPU SVM range (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @npages: Number of pages to unmap
+ *
+ * This function unmap pages associated with a GPU SVM range. Assumes and
+ * asserts correct locking is in place when called.
+ */
+static void __drm_gpusvm_range_unmap_pages(structdrm_gpusvm*gpusvm,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct drm_gpusvm_range*range,
drivers/gpu/drm/drm_gpusvm.c new file +11
-0+ unsigned long npages) +{ +unsigned long i, j; +struct drm_pagemap *dpagemap = range->dpagemap; +struct device *dev = gpusvm->drm->dev; + +lockdep_assert_held (&gpusvm->notifier_lock); + +if (range->flags.has_dma_mapping) { +for (i =0 , j =0 ; i < npages; j++) { + struct drm_pagemap_dma_addr *addr = &range-Quoted 1 line
dma_addr[j];
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +if (addr->proto ==DRM_INTERCONNECT_SYSTEM ) +dma_unmap_page (dev, + addr->addr, + PAGE_SIZE << addr-Quoted 1 line
order,
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ addr->dir); + else if (dpagemap && dpagemap->ops-Quoted 1 line
unmap_dma)drivers/gpu/drm/drm_gpusvm.c new file +13
-0+ dpagemap->ops->unmap_dma (dpagemap, + dev, + *addr); + i +=1 << addr->order; + } + range->flags.has_devmem_pages = false; + range->flags.has_dma_mapping = false; + range->dpagemap =NULL ; + } +} + +/** +* drm_gpusvm_range_free_pages() - Free pages associated with a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function frees the dma address array associated with a GPU SVM range.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static void drm_gpusvm_range_free_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range*range)
drivers/gpu/drm/drm_gpusvm.c new file +15
-0+{ +lockdep_assert_held (&gpusvm->notifier_lock); + +if (range->dma_addr) { +kvfree (range->dma_addr); + range->dma_addr =NULL ; + } +} + +/** +* drm_gpusvm_range_remove() - Remove GPU SVM range +* @gpusvm: Pointer to the GPU SVM structure +* @range: Pointer to the GPU SVM range to be removed +* +* This function removes the specified GPU SVM range and also removes the parent
+ * GPU SVM notifier if no more ranges remain in the notifier. The caller must
+ * hold a lock to protect range and notifier removal.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+void drm_gpusvm_range_remove (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range) +{ + unsigned long npages = npages_in_range(range->itree.start, + range->itree.last +1);
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+struct drm_gpusvm_notifier *notifier; + +drm_gpusvm_driver_lock_held (gpusvm); + + notifier = drm_gpusvm_notifier_find(gpusvm, range-Quoted 1 line
itree.start);
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+if (WARN_ON_ONCE (!notifier)) +return ; + +drm_gpusvm_notifier_lock (gpusvm); +__drm_gpusvm_range_unmap_pages (gpusvm, range, npages); +drm_gpusvm_range_free_pages (gpusvm, range); +__drm_gpusvm_range_remove (notifier, range); +drm_gpusvm_notifier_unlock (gpusvm); + +drm_gpusvm_range_put (range); + +if (RB_EMPTY_ROOT (¬ifier->root.rb_root)) { +if (!notifier->flags.removed) + mmu_interval_notifier_remove(¬ifier-Quoted 1 line
notifier);
drivers/gpu/drm/drm_gpusvm.c new file +11
-0+drm_gpusvm_notifier_remove (gpusvm, notifier); +drm_gpusvm_notifier_free (gpusvm, notifier); + } +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_remove); + +/** +* drm_gpusvm_range_get() - Get a reference to GPU SVM range +* @range: Pointer to the GPU SVM range +* +* This function increments the reference count of the specified GPU SVM range.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range.
+ */drivers/gpu/drm/drm_gpusvm.c new file +12
-0+struct drm_gpusvm_range * +drm_gpusvm_range_get (struct drm_gpusvm_range *range) +{ +kref_get (&range->refcount); + +return range; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_get); + +/** +* drm_gpusvm_range_destroy() - Destroy GPU SVM range +* @refcount: Pointer to the reference counter embedded in the GPU SVM range
+ *
+ * This function destroys the specified GPU SVM range when its reference count
+ * reaches zero. If a custom range-free function is provided, it is invoked to
+ * free the range; otherwise, the range is deallocated usingkfree().
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+static void drm_gpusvm_range_destroy (struct kref *refcount) +{ +struct drm_gpusvm_range *range = + container_of(refcount, struct drm_gpusvm_range,refcount);
drivers/gpu/drm/drm_gpusvm.c new file +13
-0+struct drm_gpusvm *gpusvm = range->gpusvm; + +if (gpusvm->ops->range_free) + gpusvm->ops->range_free (range); +else +kfree (range); +} + +/** +* drm_gpusvm_range_put() - Put a reference to GPU SVM range +* @range: Pointer to the GPU SVM range +* +* This function decrements the reference count of the specified GPU SVM range
+ * and frees it when the count reaches zero.
+ */drivers/gpu/drm/drm_gpusvm.c new file +12
-0+void drm_gpusvm_range_put (struct drm_gpusvm_range *range) +{ +kref_put (&range->refcount, drm_gpusvm_range_destroy); +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_put); + +/** +* drm_gpusvm_range_pages_valid() - GPU SVM range pages valid +* @gpusvm: Pointer to the GPU SVM structure +* @range: Pointer to the GPU SVM range structure +* +* This function determines if a GPU SVM range pages are valid. Expected be
+ * called holdinggpusvm->notifier_lockand as the last step before committing a
+ * GPU binding. This is akin to a notifier seqno check in the HMM documentation
+ * but due to wider notifiers (i.e., notifiers which span multiple ranges) this
+ * function is required for finer grained checking (i.e., per range) if pages
+ * are valid.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */drivers/gpu/drm/drm_gpusvm.c new file +6
-0+bool drm_gpusvm_range_pages_valid (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range) +{ +lockdep_assert_held (&gpusvm->notifier_lock); + + return range->flags.has_devmem_pages || range-Quoted 1 line
flags.has_dma_mapping;
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_pages_valid); + +/** +* drm_gpusvm_range_pages_valid_unlocked() - GPU SVM range pages valid unlocked
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid. Expected be
+ * called without holdinggpusvm->notifier_lock.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+static bool +drm_gpusvm_range_pages_valid_unlocked (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range*range)
drivers/gpu/drm/drm_gpusvm.c new file +22
-0+{ +bool pages_valid; + +if (!range->dma_addr) +return false; + +drm_gpusvm_notifier_lock (gpusvm); + pages_valid =drm_gpusvm_range_pages_valid (gpusvm, range); +if (!pages_valid) +drm_gpusvm_range_free_pages (gpusvm, range); +drm_gpusvm_notifier_unlock (gpusvm); + +return pages_valid; +} + +/** +* drm_gpusvm_range_get_pages() - Get pages for a GPU SVM range +* @gpusvm: Pointer to the GPU SVM structure +* @range: Pointer to the GPU SVM range structure +* @ctx: GPU SVM context +* +* This function gets pages for a GPU SVM range and ensures they are mapped for
+ * DMA access.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+int drm_gpusvm_range_get_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx) +{ + struct mmu_interval_notifier *notifier = &range->notifier-Quoted 1 line
notifier;
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+struct hmm_range hmm_range = { + .default_flags = HMM_PFN_REQ_FAULT | (ctx->read_only? 0 :
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ HMM_PFN_REQ_WRITE), + .notifier = notifier, + .start = range->itree.start, + .end = range->itree.last + 1, + .dev_private_owner = gpusvm-Quoted 1 line
device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ }; +struct mm_struct *mm = gpusvm->mm; +struct drm_gpusvm_zdd *zdd; +unsigned long timeout = + jiffies +
msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);drivers/gpu/drm/drm_gpusvm.c new file +3
-0+unsigned long i, j; + unsigned long npages = npages_in_range(range->itree.start, + range->itree.last +1);
drivers/gpu/drm/drm_gpusvm.c new file +58
-0+unsigned long num_dma_mapped; +unsigned int order =0 ; +unsigned long *pfns; +struct page **pages; +int err =0 ; +struct dev_pagemap *pagemap; +struct drm_pagemap *dpagemap; + +retry: + hmm_range.notifier_seq =mmu_interval_read_begin (notifier); +if (drm_gpusvm_range_pages_valid_unlocked (gpusvm, range)) + goto set_seqno; + + pfns =kvmalloc_array (npages,sizeof (*pfns),GFP_KERNEL ); +if (!pfns) +return -ENOMEM ; + +if (!mmget_not_zero (mm)) { + err = -EFAULT ; + goto err_free; + } + + hmm_range.hmm_pfns = pfns; +while (true) { +mmap_read_lock (mm); + err =hmm_range_fault (&hmm_range); +mmap_read_unlock (mm); + +if (err == -EBUSY ) { +if (time_after (jiffies, timeout)) +break ; + + hmm_range.notifier_seq = +mmu_interval_read_begin (notifier); +continue ; + } +break ; + } +mmput (mm); +if (err) + goto err_free; + + pages = (struct page **)pfns; +map_pages: +/* +* Perform all dma mappings under the notifier lock to not +* access freed pages. A notifier will either block on +* the notifier lock or unmap dma. +*/ +drm_gpusvm_notifier_lock (gpusvm); + +if (range->flags.unmapped) { +drm_gpusvm_notifier_unlock (gpusvm); + err = -EFAULT ; + goto err_free; + } + + if (mmu_interval_read_retry(notifier,hmm_range.notifier_seq)) {
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+drm_gpusvm_notifier_unlock (gpusvm); +kvfree (pfns); + goto retry; + } + +if (!range->dma_addr) { +/* Unlock and restart mapping to allocate memory. */ +drm_gpusvm_notifier_unlock (gpusvm); + range->dma_addr = kvmalloc_array(npages, + sizeof(*range-Quoted 1 line
dma_addr),drivers/gpu/drm/drm_gpusvm.c new file +17
-0+ GFP_KERNEL); +if (!range->dma_addr) { + err = -ENOMEM ; + goto err_free; + } + goto map_pages; + } + + zdd =NULL ; + num_dma_mapped =0 ; +for (i =0 , j =0 ; i < npages; ++j) { +struct page *page =hmm_pfn_to_page (pfns[i]); + + order =hmm_pfn_to_map_order (pfns[i]); +if (is_device_private_page (page) || + is_device_coherent_page(page)) { +if (zdd != page->zone_device_data && i >0 ){
drivers/gpu/drm/drm_gpusvm.c new file +12
-0+ err = -EOPNOTSUPP ; + goto err_unmap; + } + zdd = page->zone_device_data; +if (pagemap != page->pgmap) { +if (i >0 ) { + err = -EOPNOTSUPP ; + goto err_unmap; + } + + pagemap = page->pgmap; + dpagemap = zdd->devmem_allocation-Quoted 1 line
dpagemap;
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ if (drm_WARN_ON(gpusvm->drm,!dpagemap)) {
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+/* +* Raced. This is not supposed to happen
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* since hmm_range_fault() should've migrated
drivers/gpu/drm/drm_gpusvm.c new file +9
-0+* this page to system. +*/ + err = -EAGAIN ; + goto err_unmap; + } + } + range->dma_addr[j] = + dpagemap->ops->map_dma(dpagemap, + gpusvm->drm-Quoted 1 line
dev,
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ page, order, +
DMA_BIDIRECTIONAL);drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ if (dma_mapping_error(gpusvm->drm->dev, + range-Quoted 1 line
dma_addr[j].addr)) {
drivers/gpu/drm/drm_gpusvm.c new file +18
-0+ err = -EFAULT ; + goto err_unmap; + } + + pages[i] = page; + }else { +dma_addr_t addr; + +if (is_zone_device_page (page) || zdd) { + err = -EOPNOTSUPP ; + goto err_unmap; + } + + addr =dma_map_page (gpusvm->drm->dev, + page, 0, + PAGE_SIZE << order, + DMA_BIDIRECTIONAL); + if (dma_mapping_error(gpusvm->drm->dev,addr)) {
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ err = -EFAULT ; + goto err_unmap; + } + + range->dma_addr[j] =
drm_pagemap_dma_addr_encodedrivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (addr, DRM_INTERCONNECT_SYSTEM,order,
drivers/gpu/drm/drm_gpusvm.c new file +21
-0+ DMA_BIDIRECTIONAL); + } + i +=1 << order; + num_dma_mapped = i; + } + + range->flags.has_dma_mapping = true; +if (zdd) { + range->flags.has_devmem_pages = true; + range->dpagemap = dpagemap; + } + +drm_gpusvm_notifier_unlock (gpusvm); +kvfree (pfns); +set_seqno: + range->notifier_seq = hmm_range.notifier_seq; + +return 0 ; + +err_unmap: + __drm_gpusvm_range_unmap_pages(gpusvm, range,
num_dma_mapped);drivers/gpu/drm/drm_gpusvm.c new file +11
-0+drm_gpusvm_notifier_unlock (gpusvm); +err_free: +kvfree (pfns); +if (err == -EAGAIN ) + goto retry; +return err; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_get_pages); + +/** +* drm_gpusvm_range_unmap_pages() - Unmap pages associated with a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed thatgpusvm->notifier_lockis held in write mode; if it
+ * is clear, it acquiresgpusvm->notifier_lockin read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops-Quoted 1 line
invalidate for IOMMU
+ * security model.
+ */drivers/gpu/drm/drm_gpusvm.c new file +6
-0+void drm_gpusvm_range_unmap_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx) +{ + unsigned long npages = npages_in_range(range->itree.start, + range->itree.last +1);
drivers/gpu/drm/drm_gpusvm.c new file +33
-0+ +if (ctx->in_notifier) +lockdep_assert_held_write (&gpusvm->notifier_lock); +else +drm_gpusvm_notifier_lock (gpusvm); + +__drm_gpusvm_range_unmap_pages (gpusvm, range, npages); + +if (!ctx->in_notifier) +drm_gpusvm_notifier_unlock (gpusvm); +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_unmap_pages); + +/** +* drm_gpusvm_migration_unlock_put_page() - Put a migration page +* @page: Pointer to the page to put +* +* This function unlocks and puts a page. +*/ +static void drm_gpusvm_migration_unlock_put_page (struct page *page) +{ +unlock_page (page); +put_page (page); +} + +/** +* drm_gpusvm_migration_unlock_put_pages() - Put migration pages +* @npages: Number of pages +* @migrate_pfn: Array of migrate page frame numbers +* +* This function unlocks and puts an array of pages. +*/ +static void drm_gpusvm_migration_unlock_put_pages(unsigned longnpages,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long
*migrate_pfn)drivers/gpu/drm/drm_gpusvm.c new file +17
-0+{ +unsigned long i; + +for (i =0 ; i < npages; ++i) { +struct page *page; + +if (!migrate_pfn[i]) +continue ; + + page =migrate_pfn_to_page (migrate_pfn[i]); +drm_gpusvm_migration_unlock_put_page (page); + migrate_pfn[i] =0 ; + } +} + +/** +* drm_gpusvm_get_devmem_page() - Get a reference to a device memory page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU SVM zone
+ * device data and initializes it for zone device usage.
+ */drivers/gpu/drm/drm_gpusvm.c new file +9
-0+static void drm_gpusvm_get_devmem_page (struct page *page, + struct drm_gpusvm_zdd *zdd) +{ + page->zone_device_data =drm_gpusvm_zdd_get (zdd); +zone_device_page_init (page); +} + +/** +* drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g.,DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Returns: 0 on success,-EFAULTif an error occurs during mapping.
+ */drivers/gpu/drm/drm_gpusvm.c new file +10
-0+static int drm_gpusvm_migrate_map_pages (struct device *dev, +dma_addr_t *dma_addr, +unsigned long *migrate_pfn, +unsigned long npages, +enum dma_data_direction dir) +{ +unsigned long i; + +for (i =0 ; i < npages; ++i) { + struct page *page =
migrate_pfn_to_page(migrate_pfn[i]);drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ +if (!page) +continue ; + +if (WARN_ON_ONCE (is_zone_device_page (page))) +return -EFAULT ; + + dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE,dir);
drivers/gpu/drm/drm_gpusvm.c new file +9
-0+if (dma_mapping_error (dev, dma_addr[i])) +return -EFAULT ; + } + +return 0 ; +} + +/** +* drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g.,DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+static void drm_gpusvm_migrate_unmap_pages (struct device *dev, + dma_addr_t *dma_addr, + unsigned long npages, + enum dma_data_directiondir)
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+{ + unsigned long i; + + for (i = 0; i < npages; ++i) { + if (!dma_addr[i] || dma_mapping_error(dev,dma_addr[i]))
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+continue ; + +dma_unmap_page (dev, dma_addr[i],PAGE_SIZE , dir); + } +} + +/** +* drm_gpusvm_migrate_to_devmem() - Migrate GPU SVM range to device memory
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @devmem_allocation: Pointer to the device memory allocation. The caller
+ * should hold a reference to the device memory allocation,
+ * which should be dropped via ops-Quoted 1 line
devmem_releaseor upon+ * the failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to device memory. It performs the
+ * necessary setup and invokes the driver-specific operations for migration to
+ * device memory. Upon successful return, @devmem_allocation can safely reference @range
+ * untilops->devmem_releaseis called which only upon successful return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+int drm_gpusvm_migrate_to_devmem (struct drm_gpusvm *gpusvm, +struct drm_gpusvm_range *range, + struct drm_gpusvm_devmem
*devmem_allocation,drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ const struct drm_gpusvm_ctx *ctx) +{ + const struct drm_gpusvm_devmem_ops *ops = devmem_allocation-Quoted 1 line
ops;
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long start = range->itree.start, end = range-Quoted 1 line
itree.last + 1;
drivers/gpu/drm/drm_gpusvm.c new file +45
-0+struct migrate_vma migrate = { + .start = start, + .end = end, + .pgmap_owner = gpusvm->device_private_page_owner, + .flags =MIGRATE_VMA_SELECT_SYSTEM , + }; +struct mm_struct *mm = gpusvm->mm; +unsigned long i, npages =npages_in_range (start, end); +struct vm_area_struct *vas; +struct drm_gpusvm_zdd *zdd =NULL ; +struct page **pages; +dma_addr_t *dma_addr; +void *buf; +int err; + +if (!range->flags.migrate_devmem) +return -EINVAL ; + +if (!ops->populate_devmem_pfn || !ops->copy_to_devmem || + !ops->copy_to_ram) +return -EOPNOTSUPP ; + +if (!mmget_not_zero (mm)) { + err = -EFAULT ; + goto err_out; + } +mmap_read_lock (mm); + + vas =vma_lookup (mm, start); +if (!vas) { + err = -ENOENT ; + goto err_mmunlock; + } + +if (end > vas->vm_end || start < vas->vm_start) { + err = -EINVAL ; + goto err_mmunlock; + } + +if (!vma_is_anonymous (vas)) { + err = -EBUSY ; + goto err_mmunlock; + } + + buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +
sizeof(*dma_addr)+drivers/gpu/drm/drm_gpusvm.c new file +7
-0+ sizeof(*pages), GFP_KERNEL); +if (!buf) { + err = -ENOMEM ; + goto err_mmunlock; + } + dma_addr = buf + (2 *sizeof (*migrate.src) * npages); + pages = buf + (2 *sizeof (*migrate.src) +sizeof (*dma_addr))* npages;
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ + zdd = drm_gpusvm_zdd_alloc(gpusvm-Quoted 1 line
device_private_page_owner);drivers/gpu/drm/drm_gpusvm.c new file +24
-0+if (!zdd) { + err = -ENOMEM ; + goto err_free; + } + + migrate.vma = vas; + migrate.src = buf; + migrate.dst = migrate.src + npages; + + err =migrate_vma_setup (&migrate); +if (err) + goto err_free; + +if (!migrate.cpages) { + err = -EFAULT ; + goto err_free; + } + +if (migrate.cpages != npages) { + err = -EBUSY ; + goto err_finalize; + } + + err = ops->populate_devmem_pfn(devmem_allocation, npages,migrate.dst);
drivers/gpu/drm/drm_gpusvm.c new file +4
-0+if (err) + goto err_finalize; + + err = drm_gpusvm_migrate_map_pages(devmem_allocation->dev,
dma_addr,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ migrate.src, npages,
DMA_TO_DEVICE);drivers/gpu/drm/drm_gpusvm.c new file +17
-0+if (err) + goto err_finalize; + +for (i =0 ; i < npages; ++i) { +struct page *page =pfn_to_page (migrate.dst[i]); + + pages[i] = page; + migrate.dst[i] =migrate_pfn (migrate.dst[i]); +drm_gpusvm_get_devmem_page (page, zdd); + } + + err = ops->copy_to_devmem (pages, dma_addr, npages); +if (err) + goto err_finalize; + +/* Upon success bind devmem allocation to range and zdd */ + zdd->devmem_allocation = devmem_allocation;/* Owns ref */
drivers/gpu/drm/drm_gpusvm.c new file +4
-0+ +err_finalize: +if (err) + drm_gpusvm_migration_unlock_put_pages(npages,migrate.dst);
drivers/gpu/drm/drm_gpusvm.c new file +3
-0+migrate_vma_pages (&migrate); +migrate_vma_finalize (&migrate); + drm_gpusvm_migrate_unmap_pages(devmem_allocation->dev,
dma_addr, npages,drivers/gpu/drm/drm_gpusvm.c new file +15
-0+ DMA_TO_DEVICE); +err_free: +if (zdd) +drm_gpusvm_zdd_put (zdd); +kvfree (buf); +err_mmunlock: +mmap_read_unlock (mm); +mmput (mm); +err_out: +return err; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_migrate_to_devmem); + +/** +* drm_gpusvm_migrate_populate_ram_pfn() - Populate RAM PFNs for a VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @fault_page: Fault page
+ * @npages: Number of pages to populate
+ * @mpages: Number of pages to migrate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the RAM migrate page frame numbers (PFNs) for the
+ * specified VM area structure. It allocates and locks pages in the VM area for
+ * RAM usage. If vas is non-NULL usealloc_page_vmafor allocation, if NULL use
+ *alloc_pagefor allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_ram_pfn(structvm_area_struct*vas,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct page
*fault_page,drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ unsigned long npages, + unsigned long*mpages,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long
*src_mpfn,drivers/gpu/drm/drm_gpusvm.c new file +23
-0+ unsigned long *mpfn, + unsigned long addr) +{ +unsigned long i; + +for (i =0 ; i < npages; ++i, addr +=PAGE_SIZE ) { +struct page *page, *src_page; + +if (!(src_mpfn[i] &MIGRATE_PFN_MIGRATE )) +continue ; + + src_page =migrate_pfn_to_page (src_mpfn[i]); +if (!src_page) +continue ; + +if (fault_page) { +if (src_page->zone_device_data != + fault_page->zone_device_data) +continue ; + } + +if (vas) + page = alloc_page_vma(GFP_HIGHUSER, vas,addr);
drivers/gpu/drm/drm_gpusvm.c new file +39
-0+else + page =alloc_page (GFP_HIGHUSER ); + +if (!page) + goto free_pages; + + mpfn[i] =migrate_pfn (page_to_pfn (page)); + } + +for (i =0 ; i < npages; ++i) { +struct page *page =migrate_pfn_to_page (mpfn[i]); + +if (!page) +continue ; + +WARN_ON_ONCE (!trylock_page (page)); + ++*mpages; + } + +return 0 ; + +free_pages: +for (i =0 ; i < npages; ++i) { +struct page *page =migrate_pfn_to_page (mpfn[i]); + +if (!page) +continue ; + +put_page (page); + mpfn[i] =0 ; + } +return -ENOMEM ; +} + +/** +* drm_gpusvm_evict_to_ram() - Evict GPU SVM range to RAM +* @devmem_allocation: Pointer to the device memory allocation +* +* Similar to __drm_gpusvm_migrate_to_ram but does not require mmap lock and
+ * migration done via migrate_device_* functions.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+intdrm_gpusvm_evict_to_ram(struct drm_gpusvm_devmem *devmem_allocation)drivers/gpu/drm/drm_gpusvm.c new file +2
-0+{ + const struct drm_gpusvm_devmem_ops *ops = devmem_allocation-Quoted 1 line
ops;
drivers/gpu/drm/drm_gpusvm.c new file +25
-0+unsigned long npages, mpages =0 ; +struct page **pages; +unsigned long *src, *dst; +dma_addr_t *dma_addr; +void *buf; +int i, err =0 ; +unsigned int retry_count =2 ; + + npages = devmem_allocation->size >>PAGE_SHIFT ; + +retry: +if (!mmget_not_zero (devmem_allocation->mm)) +return -EFAULT ; + + buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr) + + sizeof(*pages), GFP_KERNEL); +if (!buf) { + err = -ENOMEM ; + goto err_out; + } + src = buf; + dst = buf + (sizeof (*src) * npages); + dma_addr = buf + (2 *sizeof (*src) * npages); + pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) *npages;
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ + err = ops->populate_devmem_pfn(devmem_allocation, npages,src);
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+if (err) + goto err_free; + + err =migrate_device_pfns (src, npages); +if (err) + goto err_free; + + err = drm_gpusvm_migrate_populate_ram_pfn(NULL, NULL,npages, &mpages,
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ src, dst, 0); +if (err || !mpages) + goto err_finalize; + + err = drm_gpusvm_migrate_map_pages(devmem_allocation->dev,
dma_addr,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ dst, npages,
DMA_FROM_DEVICE);drivers/gpu/drm/drm_gpusvm.c new file +16
-0+if (err) + goto err_finalize; + +for (i =0 ; i < npages; ++i) + pages[i] =migrate_pfn_to_page (src[i]); + + err = ops->copy_to_ram (pages, dma_addr, npages); +if (err) + goto err_finalize; + +err_finalize: +if (err) +drm_gpusvm_migration_unlock_put_pages (npages, dst); +migrate_device_pages (src, dst, npages); +migrate_device_finalize (src, dst, npages); + drm_gpusvm_migrate_unmap_pages(devmem_allocation->dev,
dma_addr, npages,drivers/gpu/drm/drm_gpusvm.c new file +20
-0+ DMA_FROM_DEVICE); +err_free: +kvfree (buf); +err_out: +mmput_async (devmem_allocation->mm); + +if (completion_done (&devmem_allocation->detached)) +return 0 ; + +if (!err || retry_count--) { +cond_resched (); + goto retry; + } + +return err; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_evict_to_ram); + +/** +* __drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM (internal)
+ * @vas: Pointer to the VM area structure
+ * @device_private_page_owner: Device private pages owner
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @fault_addr: Fault address
+ * @size: Size of migration
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to RAM. It sets up the migration, populates + dma maps RAM PFNs, and
+ * invokes the driver-specific operations for migration to RAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static int __drm_gpusvm_migrate_to_ram (struct vm_area_struct *vas, + void
*device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ struct page *page, + unsigned long fault_addr, + unsigned long size) +{ +struct migrate_vma migrate = { + .vma = vas, + .pgmap_owner = device_private_page_owner, + .flags =MIGRATE_VMA_SELECT_DEVICE_PRIVATE |
drivers/gpu/drm/drm_gpusvm.c new file +17
-0+ MIGRATE_VMA_SELECT_DEVICE_COHERENT, + .fault_page = page, + }; +struct drm_gpusvm_zdd *zdd; +const struct drm_gpusvm_devmem_ops *ops; +struct device *dev; +unsigned long npages, mpages =0 ; +struct page **pages; +dma_addr_t *dma_addr; +unsigned long start, end; +void *buf; +int i, err =0 ; + + start =ALIGN_DOWN (fault_addr, size); + end =ALIGN (fault_addr +1 , size); + +/* Corner where VMA area struct has been partially unmapped */
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+if (start < vas->vm_start) + start = vas->vm_start; +if (end > vas->vm_end) + end = vas->vm_end; + + migrate.start = start; + migrate.end = end; + npages =npages_in_range (start, end); + + buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +
sizeof(*dma_addr)+drivers/gpu/drm/drm_gpusvm.c new file +7
-0+ sizeof(*pages), GFP_KERNEL); +if (!buf) { + err = -ENOMEM ; + goto err_out; + } + dma_addr = buf + (2 *sizeof (*migrate.src) * npages); + pages = buf + (2 *sizeof (*migrate.src) +sizeof (*dma_addr))* npages;
drivers/gpu/drm/drm_gpusvm.c new file +30
-0+ + migrate.vma = vas; + migrate.src = buf; + migrate.dst = migrate.src + npages; + + err =migrate_vma_setup (&migrate); +if (err) + goto err_free; + +/* Raced with another CPU fault, nothing to do */ +if (!migrate.cpages) + goto err_free; + +if (!page) { +for (i =0 ; i < npages; ++i) { +if (!(migrate.src[i] &MIGRATE_PFN_MIGRATE )) +continue ; + + page =migrate_pfn_to_page (migrate.src[i]); +break ; + } + +if (!page) + goto err_finalize; + } + zdd = page->zone_device_data; + ops = zdd->devmem_allocation->ops; + dev = zdd->devmem_allocation->dev; + + err = drm_gpusvm_migrate_populate_ram_pfn(vas, page, npages,&mpages,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ migrate.src,migrate.dst,
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ start); +if (err) + goto err_finalize; + + err = drm_gpusvm_migrate_map_pages(dev, dma_addr,migrate.dst, npages,
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+ DMA_FROM_DEVICE); +if (err) + goto err_finalize; + +for (i =0 ; i < npages; ++i) + pages[i] =migrate_pfn_to_page (migrate.src[i]); + + err = ops->copy_to_ram (pages, dma_addr, npages); +if (err) + goto err_finalize; + +err_finalize: +if (err) + drm_gpusvm_migration_unlock_put_pages(npages,migrate.dst);
drivers/gpu/drm/drm_gpusvm.c new file +17
-0+migrate_vma_pages (&migrate); +migrate_vma_finalize (&migrate); +drm_gpusvm_migrate_unmap_pages (dev, dma_addr, npages, + DMA_FROM_DEVICE); +err_free: +kvfree (buf); +err_out: + +return err; +} + +/** +* drm_gpusvm_range_evict - Evict GPU SVM range +* @pagemap: Pointer to the GPU SVM structure +* @range: Pointer to the GPU SVM range to be removed +* +* This function evicts the specified GPU SVM range. This function will not
+ * evict coherent pages.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+int drm_gpusvm_range_evict (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range) +{ + struct mmu_interval_notifier *notifier = &range->notifier-Quoted 1 line
notifier;
drivers/gpu/drm/drm_gpusvm.c new file +9
-0+struct hmm_range hmm_range = { + .default_flags =HMM_PFN_REQ_FAULT , + .notifier = notifier, + .start = range->itree.start, + .end = range->itree.last +1 , + .dev_private_owner =NULL , + }; +unsigned long timeout = + jiffies +
msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);drivers/gpu/drm/drm_gpusvm.c new file +3
-0+unsigned long *pfns; + unsigned long npages = npages_in_range(range->itree.start, + range->itree.last +1);
drivers/gpu/drm/drm_gpusvm.c new file +13
-0+int err =0 ; +struct mm_struct *mm = gpusvm->mm; + +if (!mmget_not_zero (mm)) +return -EFAULT ; + + pfns =kvmalloc_array (npages,sizeof (*pfns),GFP_KERNEL ); +if (!pfns) +return -ENOMEM ; + + hmm_range.hmm_pfns = pfns; +while (!time_after (jiffies, timeout)) { + hmm_range.notifier_seq =
mmu_interval_read_begin(notifier);drivers/gpu/drm/drm_gpusvm.c new file +21
-0+if (time_after (jiffies, timeout)) { + err = -ETIME ; +break ; + } + +mmap_read_lock (mm); + err =hmm_range_fault (&hmm_range); +mmap_read_unlock (mm); +if (err != -EBUSY ) +break ; + } + +kvfree (pfns); +mmput (mm); + +return err; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_evict); + +/** +* drm_gpusvm_page_free() - Put GPU SVM zone device data associated with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device data
+ * associated with a page when it is being released.
+ */drivers/gpu/drm/drm_gpusvm.c new file +7
-0+static void drm_gpusvm_page_free (struct page *page) +{ +drm_gpusvm_zdd_put (page->zone_device_data); +} + +/** +* drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM (page fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ *VM_FAULT_SIGBUSon failure, 0 on success.
+ */drivers/gpu/drm/drm_gpusvm.c new file +7
-0+static vm_fault_t drm_gpusvm_migrate_to_ram (struct vm_fault *vmf) +{ +struct drm_gpusvm_zdd *zdd = vmf->page->zone_device_data; +int err; + + err = __drm_gpusvm_migrate_to_ram(vmf->vma, + zdd-Quoted 1 line
device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ vmf->page, vmf->address, + zdd->devmem_allocation-Quoted 1 line
size);
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+ +return err ?VM_FAULT_SIGBUS :0 ; +} + +/** +* drm_gpusvm_pagemap_ops() - Device page map operations for GPU SVM +*/ +static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = { + .page_free = drm_gpusvm_page_free, + .migrate_to_ram = drm_gpusvm_migrate_to_ram, +}; + +/** +* drm_gpusvm_pagemap_ops_get() - Retrieve GPU SVM device page map operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +8
-0+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get (void ) +{ +return &drm_gpusvm_pagemap_ops; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_pagemap_ops_get); + +/** +* drm_gpusvm_has_mapping() - Check if GPU SVM has mapping for the given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(structdrm_gpusvm*gpusvm, unsigned long start,drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ unsigned long end) +{ +struct drm_gpusvm_notifier *notifier; + +drm_gpusvm_for_each_notifier (notifier, gpusvm, start, end) { +struct drm_gpusvm_range *range =NULL ; + + drm_gpusvm_for_each_range(range, notifier, start,end)
drivers/gpu/drm/drm_gpusvm.c new file +9
-0+return true; + } + +return false; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_has_mapping); + +/** +* drm_gpusvm_range_set_unmapped() - Mark a GPU SVM range as unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets thepartial_unmapflag
+ * if the range partially falls within the provided MMU notifier range.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+void drm_gpusvm_range_set_unmapped (struct drm_gpusvm_range *range, + const struct mmu_notifier_range
*mmu_range)drivers/gpu/drm/drm_gpusvm.c new file +12
-0+{ +lockdep_assert_held_write (&range->gpusvm->notifier_lock); + + range->flags.unmapped = true; +if (range->itree.start < mmu_range->start || + range->itree.last + 1 > mmu_range->end) + range->flags.partial_unmap = true; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_set_unmapped); + +/** +* drm_gpusvm_devmem_init() - Initialize a GPU SVM device memory allocation
+ *
+ * @dev: Pointer to the device structure which device memory allocation belongs to
+ * @mm: Pointer to themm_structfor the address space
+ * @ops: Pointer to the operations structure for GPU SVM device memory
+ * @dpagemap: The structdrm_pagemapwe're allocating from.
+ * @size: Size of device memory allocation
+ */
+void drm_gpusvm_devmem_init(structdrm_gpusvm_devmem*devmem_allocation,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct device *dev, struct mm_struct*mm,
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ const struct drm_gpusvm_devmem_ops *ops, + struct drm_pagemap *dpagemap, size_tsize)
drivers/gpu/drm/drm_gpusvm.c new file +12
-0+{ +init_completion (&devmem_allocation->detached); + devmem_allocation->dev = dev; + devmem_allocation->mm = mm; + devmem_allocation->ops = ops; + devmem_allocation->dpagemap = dpagemap; + devmem_allocation->size = size; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_devmem_init); + +MODULE_DESCRIPTION ("DRM GPUSVM" ); +MODULE_LICENSE ("GPL" );include/drm/drm_gpusvm.h new file +24
-0@@ -0,0 +1,445 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +/* +* Copyright © 2024 Intel Corporation +*/ + +#ifndef __DRM_GPUSVM_H__ +#define __DRM_GPUSVM_H__ + +#include <linux/kref.h> +#include <linux/interval_tree.h> +#include <linux/mmu_notifier.h> + +struct dev_pagemap_ops ; +struct drm_device ; +struct drm_gpusvm ; +struct drm_gpusvm_notifier ; +struct drm_gpusvm_ops ; +struct drm_gpusvm_range ; +struct drm_gpusvm_devmem ; +struct drm_pagemap ; +struct drm_pagemap_dma_addr ; + +/** +* struct drm_gpusvm_devmem_ops - Operations structure for GPU SVM device memory
+ *
+ * This structure defines the operations for GPU Shared Virtual Memory (SVM)
+ * device memory. These operations are provided by the GPU driver to manage device memory
+ * allocations and perform operations such as migration between device memory and system
+ * RAM.
+ */include/drm/drm_gpusvm.h new file +3
-0+struct drm_gpusvm_devmem_ops { +/** +* @devmem_release: Release device memory allocation (optional)
include/drm/drm_gpusvm.h new file +3
-0+* @devmem_allocation: device memory allocation +* +* Release device memory allocation and drop a reference to device
include/drm/drm_gpusvm.h new file +3
-0+* memory allocation. +*/ + void (*devmem_release)(struct drm_gpusvm_devmem
*devmem_allocation);include/drm/drm_gpusvm.h new file +3
-0+ +/** +* @populate_devmem_pfn: Populate device memory PFN (required for migration)
include/drm/drm_gpusvm.h new file +10
-0+* @devmem_allocation: device memory allocation +* @npages: Number of pages to populate +* @pfn: Array of page frame numbers to populate +* +* Populate device memory page frame numbers (PFN). +* +* Returns: +* 0 on success, a negative error code on failure. +*/ + int (*populate_devmem_pfn)(struct drm_gpusvm_devmem
*devmem_allocation,include/drm/drm_gpusvm.h new file +1
-0+ unsigned long npages, unsignedlong *pfn);
include/drm/drm_gpusvm.h new file +3
-0+ +/** +* @copy_to_devmem: Copy to device memory (required for migration)
include/drm/drm_gpusvm.h new file +1
-0+* @pages: Pointer to array of device memory pages (destination)
include/drm/drm_gpusvm.h new file +16
-0+* @dma_addr: Pointer to array of DMA addresses (source) +* @npages: Number of pages to copy +* +* Copy pages to device memory. +* +* Returns: +* 0 on success, a negative error code on failure. +*/ +int (*copy_to_devmem)(struct page **pages, + dma_addr_t *dma_addr, + unsigned long npages); + +/** +* @copy_to_ram: Copy to system RAM (required for migration) +* @pages: Pointer to array of device memory pages (source) +* @dma_addr: Pointer to array of DMA addresses (destination)
include/drm/drm_gpusvm.h new file +14
-0+* @npages: Number of pages to copy +* +* Copy pages to system RAM. +* +* Returns: +* 0 on success, a negative error code on failure. +*/ +int (*copy_to_ram)(struct page **pages, + dma_addr_t *dma_addr, + unsigned long npages); +}; + +/** +* struct drm_gpusvm_devmem - Structure representing a GPU SVM device memory allocation
+ *
+ * @dev: Pointer to the device structure which device memory allocation belongs to
+ * @mm: Pointer to themm_structfor the address space
+ * @detached: device memory allocations is detached from device pages
+ * @ops: Pointer to the operations structure for GPU SVM device memory
+ * @dpagemap: The structdrm_pagemapof the pages this allocation belongs to.
+ * @size: Size of device memory allocation
+ */include/drm/drm_gpusvm.h new file +13
-0+struct drm_gpusvm_devmem { +struct device *dev; +struct mm_struct *mm; +struct completion detached; +const struct drm_gpusvm_devmem_ops *ops; +struct drm_pagemap *dpagemap; +size_t size; +}; + +/** +* struct drm_gpusvm_ops - Operations structure for GPU SVM +* +* This structure defines the operations for GPU Shared Virtual Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * notifiers.
+ */include/drm/drm_gpusvm.h new file +8
-0+struct drm_gpusvm_ops { +/** +* @notifier_alloc: Allocate a GPU SVM notifier (optional) +* +* Allocate a GPU SVM notifier. +* +* Returns: +* Pointer to the allocated GPU SVM notifier on success, NULL on failure.
include/drm/drm_gpusvm.h new file +19
-0+*/ +struct drm_gpusvm_notifier *(*notifier_alloc)(void ); + +/** +* @notifier_free: Free a GPU SVM notifier (optional) +* @notifier: Pointer to the GPU SVM notifier to be freed +* +* Free a GPU SVM notifier. +*/ +void (*notifier_free)(struct drm_gpusvm_notifier *notifier); + +/** +* @range_alloc: Allocate a GPU SVM range (optional) +* @gpusvm: Pointer to the GPU SVM +* +* Allocate a GPU SVM range. +* +* Returns: +* Pointer to the allocated GPU SVM range on success, NULL on failure.
include/drm/drm_gpusvm.h new file +2
-0+*/ + struct drm_gpusvm_range *(*range_alloc)(struct drm_gpusvm*gpusvm);
include/drm/drm_gpusvm.h new file +16
-0+ +/** +* @range_free: Free a GPU SVM range (optional) +* @range: Pointer to the GPU SVM range to be freed +* +* Free a GPU SVM range. +*/ +void (*range_free)(struct drm_gpusvm_range *range); + +/** +* @invalidate: Invalidate GPU SVM notifier (required) +* @gpusvm: Pointer to the GPU SVM +* @notifier: Pointer to the GPU SVM notifier +* @mmu_range: Pointer to the mmu_notifier_range structure +* +* Invalidate the GPU page tables. It can safely walk the notifier range
include/drm/drm_gpusvm.h new file +1
-0+* RB tree/list in this function. Called while holding the notifier lock.
include/drm/drm_gpusvm.h new file +4
-0+*/ +void (*invalidate)(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + const struct mmu_notifier_range
*mmu_range);include/drm/drm_gpusvm.h new file +4
-0+}; + +/** +* struct drm_gpusvm_notifier - Structure representing a GPU SVM notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @itree: Interval tree node for the notifier (inserted in GPU SVM)
+ * @entry: List entry to fast interval tree traversal
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ * interval tree. This is useful to keep iterating ranges while
+ * doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier has been
+ * removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */include/drm/drm_gpusvm.h new file +19
-0+struct drm_gpusvm_notifier { +struct drm_gpusvm *gpusvm; +struct mmu_interval_notifier notifier; +struct interval_tree_node itree; +struct list_head entry; +struct rb_root_cached root; +struct list_head range_list; +struct { +u32 removed :1 ; + } flags; +}; + +/** +* struct drm_gpusvm_range - Structure representing a GPU SVM range +* +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier +* @refcount: Reference count for the range +* @itree: Interval tree node for the range (inserted in GPU SVM notifier)
+ * @entry: List entry to fast interval tree traversal
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @dma_addr: DMA address array
+ * @dpagemap: The structdrm_pagemapof the device pages we're dma- mapping.
+ * Note this is assuming only onedrm_pagemapper range is allowed.
+ * @flags.migrate_devmem: Flag indicating whether the range can be migrated to device memory
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_devmem_pages: Flag indicating if the range has devmem pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA mapping
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */include/drm/drm_gpusvm.h new file +13
-0+struct drm_gpusvm_range { +struct drm_gpusvm *gpusvm; +struct drm_gpusvm_notifier *notifier; +struct kref refcount; +struct interval_tree_node itree; +struct list_head entry; +unsigned long notifier_seq; +struct drm_pagemap_dma_addr *dma_addr; +struct drm_pagemap *dpagemap; +struct { +/* All flags below must be set upon creation */ +u16 migrate_devmem :1 ; +/* All flags below must be set / cleared under notifier lock */
include/drm/drm_gpusvm.h new file +19
-0+ u16 unmapped : 1; + u16 partial_unmap : 1; + u16 has_devmem_pages : 1; + u16 has_dma_mapping : 1; + } flags; +}; + +/** +* struct drm_gpusvm - GPU SVM structure +* +* @name: Name of the GPU SVM +* @drm: Pointer to the DRM device structure +* @mm: Pointer to the mm_struct for the address space +* @device_private_page_owner: Device private pages owner +* @mm_start: Start address of GPU SVM +* @mm_range: Range of the GPU SVM +* @notifier_size: Size of individual notifiers +* @ops: Pointer to the operations structure for GPU SVM +* @chunk_sizes: Pointer to the array of chunk sizes used in range allocation.
+ * Entries should be powers of 2 in descending order.
+ * @num_chunks: Number of chunks
+ * @notifier_lock: Read-write semaphore for protecting notifier operations
+ * @root: Cached root node of the Red-Black tree containing GPU SVM notifiers
+ * @notifier_list: list head containing of notifiers in the same order they
+ * appear in interval tree. This is useful to keep iterating
+ * notifiers while doing modifications to RB tree.
+ *
+ * This structure represents a GPU SVM (Shared Virtual Memory) used for tracking
+ * memory ranges mapped in a DRM (Direct Rendering Manager) device.
+ *
+ * No reference counting is provided, as this is expected to be embedded in the
+ * driver VM structure along with the structdrm_gpuvm, which handles reference
+ * counting.
+ */include/drm/drm_gpusvm.h new file +17
-0+struct drm_gpusvm { +const char *name; +struct drm_device *drm; +struct mm_struct *mm; +void *device_private_page_owner; +unsigned long mm_start; +unsigned long mm_range; +unsigned long notifier_size; +const struct drm_gpusvm_ops *ops; +const unsigned long *chunk_sizes; +int num_chunks; +struct rw_semaphore notifier_lock; +struct rb_root_cached root; +struct list_head notifier_list; +#ifdef CONFIG_LOCKDEP +/** +* @lock_dep_map: Annotates drm_gpusvm_range_find_or_insert and
include/drm/drm_gpusvm.h new file +10
-0+* drm_gpusvm_range_remove with a driver provided lock. +*/ +struct lockdep_map *lock_dep_map; +#endif +}; + +/** +* struct drm_gpusvm_ctx - DRM GPU SVM context +* +* @check_pages_threshold: Check CPU pages for present if chunk is less than or
+ * equal to threshold. If not present, reduce chunk
+ * size.
+ * @in_notifier: entering from a MMU notifier
+ * @read_only: operating on read-only memory
+ * @devmem_possible: possible to use device memory
+ *
+ * Context that is DRM GPUSVM is operating in (i.e. user arguments).
+ */include/drm/drm_gpusvm.h new file +10
-0+struct drm_gpusvm_ctx { +unsigned long check_pages_threshold; +unsigned int in_notifier :1 ; +unsigned int read_only :1 ; +unsigned int devmem_possible :1 ; +}; + +int drm_gpusvm_init (struct drm_gpusvm *gpusvm, + const char *name, struct drm_device *drm, + struct mm_struct *mm, void
*device_private_page_owner,include/drm/drm_gpusvm.h new file +4
-0+ unsigned long mm_start, unsigned long mm_range, + unsigned long notifier_size, + const struct drm_gpusvm_ops *ops, + const unsigned long *chunk_sizes, int
num_chunks);include/drm/drm_gpusvm.h new file +37
-0+ +void drm_gpusvm_fini (struct drm_gpusvm *gpusvm); + +void drm_gpusvm_free (struct drm_gpusvm *gpusvm); + +struct drm_gpusvm_range * +drm_gpusvm_range_find_or_insert (struct drm_gpusvm *gpusvm, +unsigned long fault_addr, +unsigned long gpuva_start, +unsigned long gpuva_end, +const struct drm_gpusvm_ctx *ctx); + +void drm_gpusvm_range_remove (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range); + +int drm_gpusvm_range_evict (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range); + +struct drm_gpusvm_range * +drm_gpusvm_range_get (struct drm_gpusvm_range *range); + +void drm_gpusvm_range_put (struct drm_gpusvm_range *range); + +bool drm_gpusvm_range_pages_valid (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range); + +int drm_gpusvm_range_get_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx); + +void drm_gpusvm_range_unmap_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx); + +int drm_gpusvm_migrate_to_devmem (struct drm_gpusvm *gpusvm, +struct drm_gpusvm_range *range, + struct drm_gpusvm_devmem
*devmem_allocation,include/drm/drm_gpusvm.h new file +3
-0+ const struct drm_gpusvm_ctx *ctx); + +int drm_gpusvm_evict_to_ram(struct drm_gpusvm_devmem
*devmem_allocation);include/drm/drm_gpusvm.h new file +4
-0+ +const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get (void ); + +bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, unsigned longstart,
include/drm/drm_gpusvm.h new file +4
-0+ unsigned long end); + +struct drm_gpusvm_range * +drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, unsignedlong start,
include/drm/drm_gpusvm.h new file +4
-0+ unsigned long end); + +void drm_gpusvm_range_set_unmapped (struct drm_gpusvm_range *range, + const struct mmu_notifier_range
*mmu_range);include/drm/drm_gpusvm.h new file +2
-0+ +void drm_gpusvm_devmem_init(struct drm_gpusvm_devmem
*devmem_allocation,include/drm/drm_gpusvm.h new file +1
-0+ struct device *dev, struct mm_struct*mm,
include/drm/drm_gpusvm.h new file +2
-0+ const struct drm_gpusvm_devmem_ops *ops, + struct drm_pagemap *dpagemap, size_tsize);
include/drm/drm_gpusvm.h new file +4
-0+ +#ifdef CONFIG_LOCKDEP +/** +* drm_gpusvm_driver_set_lock() - Set the lock protecting accesses to GPU SVM
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @lock: the lock used to protect the gpuva list. The locking primitive
+ * must contain adep_mapfield.
+ *
+ * Call this to annotatedrm_gpusvm_range_find_or_insertand
+ *drm_gpusvm_range_remove.
+ */include/drm/drm_gpusvm.h new file +4
-0+#define drm_gpusvm_driver_set_lock (gpusvm, lock) \ + do { \ + if (!WARN((gpusvm)->lock_dep_map, \ + "GPUSVM range lock should be set onlyonce."))\
include/drm/drm_gpusvm.h new file +1
-0+ (gpusvm)->lock_dep_map = &(lock)-Quoted 1 line
dep_map; \
include/drm/drm_gpusvm.h line 387
Quoted 6 lines
include/drm/drm_gpusvm.h new file +6
-0+ } while (0) +#define drm_gpusvm_driver_lock_held (gpusvm) \ + do { \ + if ((gpusvm)->lock_dep_map) \ + lock_is_held((gpusvm)->lock_dep_map); \ + } while (0)
Could we use static functions for those above
Also I don't think the drm_gpusvm_driver_lock_held() does what it's intended to do? There's an assert missing.
Quoted 66 lines
include/drm/drm_gpusvm.h new file +25
-0+#else +#define drm_gpusvm_driver_set_lock (gpusvm, lock) do {} while (0) +#define drm_gpusvm_driver_lock_held (gpusvm) do {} while (0) +#endif + +/** +* drm_gpusvm_notifier_lock() - Lock GPU SVM notifier +* @gpusvm__: Pointer to the GPU SVM structure. +* +* Abstract client usage GPU SVM notifier lock, take lock +*/ +#define drm_gpusvm_notifier_lock (gpusvm__) \ + down_read(&(gpusvm__)->notifier_lock) + +/** +* drm_gpusvm_notifier_unlock() - Unlock GPU SVM notifier +* @gpusvm__: Pointer to the GPU SVM structure. +* +* Abstract client usage GPU SVM notifier lock, drop lock +*/ +#define drm_gpusvm_notifier_unlock (gpusvm__) \ + up_read(&(gpusvm__)->notifier_lock) + +/** +* __drm_gpusvm_range_next() - Get the next GPU SVM range in the list
+ * @range: a pointer to the current GPU SVM range
+ *
+ * Return: A pointer to the nextdrm_gpusvm_rangeif available, or NULL if the
+ * current range is the last one or if the input range is NULL.
+ */include/drm/drm_gpusvm.h new file +12
-0+static inline struct drm_gpusvm_range * +__drm_gpusvm_range_next (struct drm_gpusvm_range *range) +{ +if (range && !list_is_last (&range->entry, + &range->notifier->range_list)) +return list_next_entry (range, entry); + +return NULL ; +} + +/** +* drm_gpusvm_for_each_range() - Iterate over GPU SVM ranges in a notifier
+ * @range__: Iterator variable for the ranges. If set, it indicates the start of
+ * the iterator. If NULL, calldrm_gpusvm_range_find()to
get the range.
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier. It is safe
+ * to use while holding the driver SVM lock or the notifier lock.
+ */
+#define drm_gpusvm_for_each_range(range__,notifier__,start__,end__) \include/drm/drm_gpusvm.h new file +1
-0+ for ((range__) = (range__)?: \
include/drm/drm_gpusvm.h new file +1
-0+ drm_gpusvm_range_find((notifier__), (start__),(
end__)); \include/drm/drm_gpusvm.h new file +1
-0+ (range__) && (range__->itree.start <(
end__)); \
include/drm/drm_gpusvm.h line 445
Quoted 3 lines
include/drm/drm_gpusvm.h new file +3
-0+ (range__) = __drm_gpusvm_range_next(range__)) + +#endif /* __DRM_GPUSVM_H__ */
Otherwise LGTM.
/Thomas
Matthew Brost
maintainerRe: [PATCH v4 06/33] drm/gpusvm: Add support for GPU Shared Virtual Memory
Quoted 3,193 lines · Thomas Hellström
Quoted 3,192 lines · Matthew Brost
This patch introduces support for GPU Shared Virtual Memory (SVM) in the
Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and flexibility in GPU computing tasks.The patch adds the necessary infrastructure for SVM, including data structures and functions for managing SVM ranges and notifiers. It also
provides mechanisms for allocating, deallocating, and migrating memory
regions between system RAM and GPU VRAM.This is largely inspired by GPUVM.
v2:
- Take order into account in check pages
- Clearrange->pagesin get pages error
- Drop setting dirty or accessed bit in get pages (Vetter)
- Remove mmap assert for cpu faults
- Drop mmap write lock abuse (Vetter, Christian)
- Decouple zdd from range (Vetter, Oak)
- Adddrm_gpusvm_range_evict, make it work with coherent pages
- Exportdrm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
- mmget/put indrm_gpusvm_evict_to_sram
- Droprange->vram_alloationvariable
- Don't return indrm_gpusvm_evict_to_sramuntil all pages detached
- Don't warn on mixing sram and device pages
- Update kernel doc
- Add coherent page support to get pages
- UseDMA_FROM_DEVICErather thanDMA_BIDIRECTIONAL
- Add structdrm_gpusvm_vramand ops (Thomas)
- Update the range's seqno if the range is valid (Thomas)
- Remove theis_unmappedcheck beforehmm_range_fault(Thomas)
- Usedrm_pagemap(Thomas)
- Dropkfree_mapping(Thomas)
- dma mapp pages under notifier lock (Thomas)
- Remove ctx.prefault
- Remove ctx.mmap_locked
- Add ctx.check_pages
- s/vram/devmem (Thomas)
v3:
- Fix memory leakdrm_gpusvm_range_get_pages
- Only migrate pages with same zdd on CPU fault
- Loop over al VMAs indrm_gpusvm_range_evict
- Make GPUSVM a drm level module
- GPL or MIT license
- Update main kernel doc (Thomas)
- Preferfoo()vs foo for functions in kernel doc (Thomas)
- Prefer functions over macros (Thomas)
- Use unsigned long vs u64 for addresses (Thomas)
- Use standardinterval_tree(Thomas)
- s/drm_gpusvm_migration_put_page/drm_gpusvm_migration_unlock_put_page (Thomas)
- Droperr_outlabel indrm_gpusvm_range_find_or_insert(Thomas)
- Fix kernel doc indrm_gpusvm_range_free_pages(Thomas)
- Newlines between functions defs in header file (Thomas)
- Drop shall language in driver vfunc kernel doc (Thomas)
- Move some static inlines from head to C file (Thomas)
- Don't allocate pages under page lock indrm_gpusvm_migrate_populate_ram_pfn(Thomas)
- Changecheck_pagesto a threshold
v4:
- Fix NULL ptr deref indrm_gpusvm_migrate_populate_ram_pfn(Thomas, Himal)
- Fix check pages threshold
- Check for range being unmapped under notifier lock in get pages (Testing)
- Fix characters per line
- DropWRITE_ONCEforzdd->devmem_allocationassignment (Thomas)
- Use completion fordevmem_allocation->detached(Thomas)
- Make GPU SVM depend onZONE_DEVICE(CI)
- Usehmm_range_faultfor eviction (Thomas)
- Drop zdd worker (Thomas)Cc Simona Vetter <[email protected]>Cc Dave Airlie <[email protected]>Cc Christian König <[email protected]>Cc <[email protected]>Signed-off-by Matthew Brost <[email protected]>Signed-off-by Thomas Hellström <[email protected]>---
drivers/gpu/drm/Kconfig | 9 +
drivers/gpu/drm/Makefile | 1 +
drivers/gpu/drm/drm_gpusvm.c | 2240
++++++++++++++++++++++++++++++++++
include/drm/drm_gpusvm.h | 445 +++++++4 files changed, 2695 insertions(+)create mode 100644
drivers/gpu/drm/drm_gpusvm.c
create mode 100644include/drm/drm_gpusvm.hdrivers/gpu/drm/Kconfig +0
-0@@ -278,6 +278,15 @@ config DRM_GPUVMGPU-VM representation providing helpers to manage a GPUs virtual
address spacedrivers/gpu/drm/Kconfig +7
-0+config DRM_GPUSVM + tristate + depends on DRM + depends on DEVICE_MIGRATION + depends on ZONE_DEVICE + help + GPU-SVM representation providing helpers to manage a GPUsshared
drivers/gpu/drm/Kconfig +2
-0+ virtual memory +config
DRM_BUDDY
tristate
depends on DRMdrivers/gpu/drm/Makefile +0
-0@@ -104,6 +104,7 @@ obj-$(CONFIG_DRM_PANEL_BACKLIGHT_QUIRKS) +=drm_panel_backlight_quirks.o
#
obj-$(CONFIG_DRM_EXEC) += drm_exec.o
obj-$(CONFIG_DRM_GPUVM) += drm_gpuvm.o
+obj-$(CONFIG_DRM_GPUSVM) += drm_gpusvm.o
obj-$(CONFIG_DRM_BUDDY) += drm_buddy.odrivers/gpu/drm/drm_gpusvm.c new file +25
-0@@ -0,0 +1,2240 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* +* Copyright © 2024 Intel Corporation +* +* Authors: +* Matthew Brost <[email protected]> +*/ + +#include <linux/dma-mapping.h> +#include <linux/hmm.h> +#include <linux/memremap.h> +#include <linux/migrate.h> +#include <linux/mm_types.h> +#include <linux/pagemap.h> +#include <linux/slab.h> + +#include <drm/drm_device.h> +#include <drm/drm_gpusvm.h> +#include <drm/drm_pagemap.h> +#include <drm/drm_print.h> + +/** +* DOC: Overview +* +* GPU Shared Virtual Memory (GPU SVM) layer for the Direct Rendering Manager (DRM)
+ *
+ * The GPU SVM layer is a component of the DRM framework designed to manage shared
+ * virtual memory between the CPU and GPU. It enables efficient data exchange and
+ * processing for GPU-accelerated applications by allowing memory sharing and
+ * synchronization between the CPU's and GPU's virtual address spaces.
+ *
+ * Key GPU SVM Components:
+ * - Notifiers: Notifiers: Used for tracking memory intervals and notifying the
+ * GPU of changes, notifiers are sized based on a GPU
SVM
+ * initialization parameter, with a recommendation of
512M or
+ * larger. They maintain a Red-BlacK tree and a list of
ranges that
+ * fall within the notifier interval. Notifiers are
tracked within
+ * a GPU SVM Red-BlacK tree and list and are
dynamically inserted
+ * or removed as ranges within the interval are created
or
+ * destroyed.
+ * - Ranges: Represent memory ranges mapped in a DRM device and managed
+ * by GPU SVM. They are sized based on an array of chunk
sizes, which
+ * is a GPU SVM initialization parameter, and the CPU
address space.
+ * Upon GPU fault, the largest aligned chunk that fits
within the
+ * faulting CPU address space is chosen for the range
size. Ranges are
+ * expected to be dynamically allocated on GPU fault and
removed on an
+ * MMU notifier UNMAP event. As mentioned above, ranges
are tracked in
+ * a notifier's Red-Black tree.
+ * - Operations: Define the interface for driver-specific GPU SVM operations
+ * such as range allocation, notifier allocation, and
+ * invalidations.
+ * - Device Memory Allocations: Embedded structure containing enough information
+ * for GPU SVM to migrate to / from device memory.
+ * - Device Memory Operations: Define the interface for driver- specific device
+ * memory operations release memory, populate pfns,
+ * and copy to / from device memory.
+ *
+ * This layer provides interfaces for allocating, mapping, migrating, and
+ * releasing memory ranges between the CPU and GPU. It handles all core memory
+ * management interactions (DMA mapping, HMM, and migration) and provides
+ * driver-specific virtual functions (vfuncs). This infrastructure is sufficient
+ * to build the expected driver components for an SVM implementation as detailed
+ * below.
+ *
+ * Expected Driver Components:
+ * - GPU page fault handler: Used to create ranges and notifiers based on the
+ * fault address, optionally migrate the
range to
+ * device memory, and create GPU bindings.
+ * - Garbage collector: Used to unmap and destroy GPU bindings for ranges.
+ * Ranges are expected to be added to the
garbage collector
+ * upon aMMU_NOTIFY_UNMAPevent in notifier
callback.
+ * - Notifier callback: Used to invalidate and DMA unmap GPU bindings for
+ * ranges.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +/** +* DOC: Locking +* +* GPU SVM handles locking for core MM interactions, i.e., it locks/unlocks the
+ * mmap lock as needed.
+ *
+ * GPU SVM introduces a global notifier lock, which safeguards the notifier's
+ * range RB tree and list, as well as the range's DMA mappings and sequence
+ * number. GPU SVM manages all necessary locking and unlocking operations,
+ * except for the recheck range's pages being valid
+ * (drm_gpusvm_range_pages_valid) when the driver is committing GPU bindings. This
+ * lock corresponds to the 'driver->update' lock mentioned in the HMM
+ * documentation (TODO: Link). Future revisions may transition from a GPU SVM
+ * global lock to a per-notifier lock if finer-grained locking is deemed
+ * necessary.
+ *
+ * In addition to the locking mentioned above, the driver should implement a
+ * lock to safeguard core GPU SVM function calls that modify state, such as
+ *drm_gpusvm_range_find_or_insertanddrm_gpusvm_range_remove. This lock is
+ * denoted as 'driver_svm_lock' in code examples. Finer grained driver side
+ * locking should also be possible for concurrent GPU fault processing within a
+ * single GPU SVM. The 'driver_svm_lock' can be viadrm_gpusvm_driver_set_lock
+ * to add annotations to GPU SVM.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +/** +* DOC: Migration +* +* The migration support is quite simple, allowing migration between RAM and
+ * device memory at the range granularity. For example, GPU SVM currently does not
+ * support mixing RAM and device memory pages within a range. This means that upon GPU
+ * fault, the entire range can be migrated to device memory, and upon CPU fault, the
+ * entire range is migrated to RAM. Mixed RAM and device memory storage within a range
+ * could be added in the future if required.
+ *
+ * The reasoning for only supporting range granularity is as
follows: it
+ * simplifies the implementation, and range sizes are driver-defined and should
+ * be relatively small.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +/** +* DOC: Partial Unmapping of Ranges +* +* Partial unmapping of ranges (e.g., 1M out of 2M is unmapped by CPU resulting
+ * inMMU_NOTIFY_UNMAPevent) presents several challenges, with the main one
+ * being that a subset of the range still has CPU and GPU mappings. If the
+ * backing store for the range is in device memory, a subset of the backing store has
+ * references. One option would be to split the range and device memory backing store,
+ * but the implementation for this would be quite complicated. Given that
+ * partial unmappings are rare and driver-defined range sizes are relatively
+ * small, GPU SVM does not support splitting of ranges.
+ *
+ * With no support for range splitting, upon partial unmapping of a range, the
+ * driver is expected to invalidate and destroy the entire range. If the range
+ * has device memory as its backing, the driver is also expected to migrate any
+ * remaining pages back to RAM.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +/** +* DOC: Examples +* +* This section provides three examples of how to build the expected driver
+ * components: the GPU page fault handler, the garbage collector, and the
+ * notifier callback.
+ *
+ * The generic code provided does not include logic for complex migration
+ * policies, optimized invalidations, fined grained driver locking, or other
+ * potentially required driver locking (e.g., DMA-resv locks).
+ *
+ * 1) GPU page fault handler
+ *
+ * int driver_bind_range(structdrm_gpusvm*gpusvm, structdrm_gpusvm_range*range)
+ * {
+ * int err = 0;
+ *
+ * driver_alloc_and_setup_memory_for_bind(gpusvm,
range);
+ *
+ *drm_gpusvm_notifier_lock(gpusvm);
+ * if (drm_gpusvm_range_pages_valid(range))
+ *driver_commit_bind(gpusvm, range);
+ * else
+ * err =-EAGAIN;
+ *drm_gpusvm_notifier_unlock(gpusvm);
+ *
+ * return err;
+ * }
+ *
+ * int driver_gpu_fault(structdrm_gpusvm*gpusvm, unsigned
longfault_addr,
+ * unsigned longgpuva_start, unsigned
longgpuva_end)
+ * {
+ * structdrm_gpusvm_ctxctx = {};
+ * int err;
+ *
+ *driver_svm_lock();
+ * retry:
+ * // Always process UNMAPs first so view of GPU SVM
ranges is current
+ *driver_garbage_collector(gpusvm);
+ *
+ * range = drm_gpusvm_range_find_or_insert(gpusvm,fault_addr,
+ *gpuva_start,gpuva_end,
+ * &ctx);
+ * if (IS_ERR(range)) {
+ * err =PTR_ERR(range);
+ * goto unlock;
+ * }
+ *
+ * if (driver_migration_policy(range)) {
+ * devmem =driver_alloc_devmem();
+ * err = drm_gpusvm_migrate_to_devmem(gpusvm,
range,
+ *devmem_allocation,
+ * &ctx);
+ * if (err) // CPU mappings may have
changed
+ * goto retry;
+ * }
+ *
+ * err = drm_gpusvm_range_get_pages(gpusvm, range,
&ctx);
+ * if (err ==-EOPNOTSUPP|| err ==-EFAULT|| err == -
EPERM) { // CPU mappings changed
+ * if (err ==-EOPNOTSUPP)
+ * drm_gpusvm_range_evict(gpusvm,
range);
+ * goto retry;
+ * } else if (err) {
+ * goto unlock;
+ * }
+ *
+ * err =driver_bind_range(gpusvm, range);
+ * if (err ==-EAGAIN) // CPU mappings changed
+ * goto retry
+ *
+ * unlock:
+ *driver_svm_unlock();
+ * return err;
+ * }
+ *
+ * 2) Garbage Collector.
+ *
+ * void __driver_garbage_collector(structdrm_gpusvm*gpusvm,
+ * structdrm_gpusvm_range
*range)
+ * {
+ *assert_driver_svm_locked(gpusvm);
+ *
+ * // Partial unmap, migrate any remaining device
memory pages back to RAM
+ * if (range->flags.partial_unmap)
+ *drm_gpusvm_range_evict(gpusvm, range);
+ *
+ *driver_unbind_range(range);
+ *drm_gpusvm_range_remove(gpusvm, range);
+ * }
+ *
+ * voiddriver_garbage_collector(struct drm_gpusvm *gpusvm)
+ * {
+ *assert_driver_svm_locked(gpusvm);
+ *
+ *for_each_range_in_garbage_collector(gpusvm, range)
+ *__driver_garbage_collector(gpusvm, range);
+ * }
+ *
+ * 3) Notifier callback.
+ *
+ * void driver_invalidation(structdrm_gpusvm*gpusvm,
+ * structdrm_gpusvm_notifier
*notifier,
+ * const structmmu_notifier_range*mmu_range)
+ * {
+ * structdrm_gpusvm_ctxctx = { .in_notifier = true,
};
+ * structdrm_gpusvm_range*range = NULL;
+ *
+ * driver_invalidate_device_pages(gpusvm, mmu_range-Quoted 1 line
start,
mmu_range->end);+ *
+ * drm_gpusvm_for_each_range(range, notifier,mmu_range->start,
+ *mmu_range->end) {
+ * drm_gpusvm_range_unmap_pages(gpusvm, range,
&ctx);
+ *
+ * if (mmu_range->event!=MMU_NOTIFY_UNMAP)
+ * continue;
+ *
+ * drm_gpusvm_range_set_unmapped(range,mmu_range);
+ *driver_garbage_collector_add(gpusvm, range);
+ * }
+ * }
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ +/** +* npages_in_range() - Calculate the number of pages in a given range
+ * @start: The start address of the range
+ * @end: The end address of the range
+ *
+ * This macro calculates the number of pages in a given memory range,
+ * specified by the start and end addresses. It divides the difference
+ * between the end and start addresses by the page size (PAGE_SIZE) to
+ * determine the number of pages in the range.
+ *
+ * Returns: The number of pages in the specified range.
+ */drivers/gpu/drm/drm_gpusvm.c new file +15
-0+static unsigned long +npages_in_range (unsigned long start,unsigned long end) +{ +return (end - start) >>PAGE_SHIFT ; +} + +/** +* struct drm_gpusvm_zdd - GPU SVM zone device data +* +* @refcount: Reference count for the zdd +* @devmem_allocation: device memory allocation +* @device_private_page_owner: Device private pages owner +* +* This structure serves as a generic wrapper installed in +* page->zone_device_data. It provides infrastructure for looking up a device
+ * memory allocation upon CPU page fault and asynchronously releasing device
+ * memory once the CPU has no page references. Asynchronous release is useful
+ * because CPU page references can be dropped in IRQ contexts, while releasing
+ * device memory likely requires sleeping locks.
+ */drivers/gpu/drm/drm_gpusvm.c new file +11
-0+struct drm_gpusvm_zdd { +struct kref refcount; +struct drm_gpusvm_devmem *devmem_allocation; +void *device_private_page_owner; +}; + +/** +* drm_gpusvm_zdd_alloc() - Allocate a zdd structure. +* @device_private_page_owner: Device private pages owner +* +* This function allocates and initializes a new zdd structure. It sets up the
+ * reference count and initializes the destroy work.
+ *
+ * Returns:
+ * Pointer to the allocated zdd on success,ERR_PTR()on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +21
-0+static struct drm_gpusvm_zdd * +drm_gpusvm_zdd_alloc (void *device_private_page_owner) +{ +struct drm_gpusvm_zdd *zdd; + + zdd =kmalloc (sizeof (*zdd),GFP_KERNEL ); +if (!zdd) +return NULL ; + +kref_init (&zdd->refcount); + zdd->devmem_allocation =NULL ; + zdd->device_private_page_owner = device_private_page_owner; + +return zdd; +} + +/** +* drm_gpusvm_zdd_get() - Get a reference to a zdd structure. +* @zdd: Pointer to the zdd structure. +* +* This function increments the reference count of the provided zdd structure.
+ *
+ * Returns: Pointer to the zdd structure.
+ */
+static structdrm_gpusvm_zdd*drm_gpusvm_zdd_get(structdrm_gpusvm_zdd*zdd)drivers/gpu/drm/drm_gpusvm.c new file +10
-0+{ +kref_get (&zdd->refcount); +return zdd; +} + +/** +* drm_gpusvm_zdd_destroy() - Destroy a zdd structure. +* @ref: Pointer to the reference count structure. +* +* This function queues the destroy_work of the zdd for asynchronous destruction.
+ */drivers/gpu/drm/drm_gpusvm.c new file +19
-0+static void drm_gpusvm_zdd_destroy (struct kref *ref) +{ +struct drm_gpusvm_zdd *zdd = + container_of(ref, struct drm_gpusvm_zdd, refcount); +struct drm_gpusvm_devmem *devmem = zdd->devmem_allocation; + +if (devmem) { +complete_all (&devmem->detached); +if (devmem->ops->devmem_release) + devmem->ops->devmem_release (devmem); + } +kfree (zdd); +} + +/** +* drm_gpusvm_zdd_put() - Put a zdd reference. +* @zdd: Pointer to the zdd structure. +* +* This function decrements the reference count of the provided zdd structure
+ * and schedules its destruction if the count drops to zero.
+ */drivers/gpu/drm/drm_gpusvm.c new file +7
-0+static void drm_gpusvm_zdd_put (struct drm_gpusvm_zdd *zdd) +{ +kref_put (&zdd->refcount, drm_gpusvm_zdd_destroy); +} + +/** +* drm_gpusvm_range_find() - Find GPU SVM range from GPU SVM notifier
+ * @notifier: Pointer to the GPU SVM notifier structure.
+ * @start: Start address of the range
+ * @end: End address of the range
+ *
+ * Returns: A pointer to thedrm_gpusvm_rangeif found or NULL
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+struct drm_gpusvm_range * +drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, unsignedlong start,
drivers/gpu/drm/drm_gpusvm.c new file +8
-1+ unsigned long end) +{ +struct interval_tree_node *itree; + + itree = interval_tree_iter_first(¬ifier->root, start, end - 1); + + if (itree) + return container_of(itree, struct drm_gpusvm_range,itree);
drivers/gpu/drm/drm_gpusvm.c new file +7
-0+else +return NULL ; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_find); + +/** +* drm_gpusvm_for_each_range_safe() - Safely iterate over GPU SVM ranges in a notifier
+ * @range__: Iterator variable for the ranges
+ * @next__: Iterator variable for the ranges temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier while
+ * removing ranges from it.
+ */
+#define drm_gpusvm_for_each_range_safe(range__,next__,notifier__,start__,end__) \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ for ((range__) = drm_gpusvm_range_find((notifier__),(
start__), (end__)), \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (next__) =
__drm_gpusvm_range_next(range__); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (range__) && (range__->itree.start <(
end__)); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (range__) = (next__), (next__) =
__drm_gpusvm_range_next(range__))drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ +/** +* __drm_gpusvm_notifier_next() - get the next drm_gpusvm_notifier in the list
+ * @notifier: a pointer to the currentdrm_gpusvm_notifier
+ *
+ * Returns: A pointer to the nextdrm_gpusvm_notifierif available, or NULL if
+ * the current notifier is the last one or if the input notifier is
+ * NULL.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+static struct drm_gpusvm_notifier * +__drm_gpusvm_notifier_next(struct drm_gpusvm_notifier *notifier) +{ + if (notifier && !list_is_last(¬ifier->entry, + ¬ifier->gpusvm-Quoted 1 line
notifier_list))drivers/gpu/drm/drm_gpusvm.c new file +7
-0+return list_next_entry (notifier, entry); + +return NULL ; +} + +static struct drm_gpusvm_notifier * +notifier_iter_first(struct rb_root_cached *root, unsigned longstart,
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ unsigned long last) +{ +struct interval_tree_node *itree; + + itree =interval_tree_iter_first (root, start, last); + +if (itree) + return container_of(itree, struct
drm_gpusvm_notifier, itree);drivers/gpu/drm/drm_gpusvm.c new file +6
-0+else +return NULL ; +} + +/** +* drm_gpusvm_for_each_notifier() - Iterate over GPU SVM notifiers in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm.
+ */
+#define drm_gpusvm_for_each_notifier(notifier__,gpusvm__,start__,end__) \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,(
start__), (end__) - 1); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (notifier__) && (notifier__->itree.start <(
end__)); \drivers/gpu/drm/drm_gpusvm.c new file +4
-0+ (notifier__) = __drm_gpusvm_notifier_next(notifier__)) + +/** +* drm_gpusvm_for_each_notifier_safe() - Safely iterate over GPU SVM notifiers in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @next__: Iterator variable for the notifiers temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm while
+ * removing notifiers from it.
+ */
+#define drm_gpusvm_for_each_notifier_safe(notifier__,next__,gpusvm__,start__,end__) \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ for ((notifier__) = notifier_iter_first(&(gpusvm__)->root,(
start__), (end__) - 1), \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (next__) =
__drm_gpusvm_notifier_next(notifier__); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (notifier__) && (notifier__->itree.start <(
end__)); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (notifier__) = (next__), (next__) =
__drm_gpusvm_notifier_next(notifier__))drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ +/** +* drm_gpusvm_notifier_invalidate() - Invalidate a GPU SVM notifier. +* @mni: Pointer to the mmu_interval_notifier structure. +* @mmu_range: Pointer to the mmu_notifier_range structure. +* @cur_seq: Current sequence number. +* +* This function serves as a generic MMU notifier for GPU SVM. It sets the MMU
+ * notifier sequence number and calls the driver invalidate vfunc under
+ *gpusvm->notifier_lock.
+ *
+ * Returns:
+ * true if the operation succeeds, false otherwise.
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+static bool +drm_gpusvm_notifier_invalidate (struct mmu_interval_notifier *mni, + const struct mmu_notifier_range
*mmu_range,drivers/gpu/drm/drm_gpusvm.c new file +19
-0+ unsigned long cur_seq) +{ +struct drm_gpusvm_notifier *notifier = +container_of (mni,typeof (*notifier), notifier); +struct drm_gpusvm *gpusvm = notifier->gpusvm; + +if (!mmu_notifier_range_blockable (mmu_range)) +return false; + +down_write (&gpusvm->notifier_lock); +mmu_interval_set_seq (mni, cur_seq); + gpusvm->ops->invalidate (gpusvm, notifier, mmu_range); +up_write (&gpusvm->notifier_lock); + +return true; +} + +/** +* drm_gpusvm_notifier_ops - MMU interval notifier operations for GPU SVM
+ */
+static const structmmu_interval_notifier_opsdrm_gpusvm_notifier_ops= {drivers/gpu/drm/drm_gpusvm.c new file +15
-0+ .invalidate = drm_gpusvm_notifier_invalidate, +}; + +/** +* drm_gpusvm_init() - Initialize the GPU SVM. +* @gpusvm: Pointer to the GPU SVM structure. +* @name: Name of the GPU SVM. +* @drm: Pointer to the DRM device structure. +* @mm: Pointer to the mm_struct for the address space. +* @device_private_page_owner: Device private pages owner. +* @mm_start: Start address of GPU SVM. +* @mm_range: Range of the GPU SVM. +* @notifier_size: Size of individual notifiers. +* @ops: Pointer to the operations structure for GPU SVM. +* @chunk_sizes: Pointer to the array of chunk sizes used in range allocation.
+ * Entries should be powers of 2 in descending order with last
+ * entry beingSZ_4K.
+ * @num_chunks: Number of chunks.
+ *
+ * This function initializes the GPU SVM.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+int drm_gpusvm_init (struct drm_gpusvm *gpusvm, + const char *name, struct drm_device *drm, + struct mm_struct *mm, void
*device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +4
-0+ unsigned long mm_start, unsigned long mm_range, + unsigned long notifier_size, + const struct drm_gpusvm_ops *ops, + const unsigned long *chunk_sizes, int
num_chunks)drivers/gpu/drm/drm_gpusvm.c new file +8
-0+{ +if (!ops->invalidate || !num_chunks) +return -EINVAL ; + + gpusvm->name = name; + gpusvm->drm = drm; + gpusvm->mm = mm; + gpusvm->device_private_page_owner =
device_private_page_owner;drivers/gpu/drm/drm_gpusvm.c new file +31
-0+ gpusvm->mm_start = mm_start; + gpusvm->mm_range = mm_range; + gpusvm->notifier_size = notifier_size; + gpusvm->ops = ops; + gpusvm->chunk_sizes = chunk_sizes; + gpusvm->num_chunks = num_chunks; + +mmgrab (mm); + gpusvm->root =RB_ROOT_CACHED ; +INIT_LIST_HEAD (&gpusvm->notifier_list); + +init_rwsem (&gpusvm->notifier_lock); + +fs_reclaim_acquire (GFP_KERNEL ); +might_lock (&gpusvm->notifier_lock); +fs_reclaim_release (GFP_KERNEL ); + +#ifdef CONFIG_LOCKDEP + gpusvm->lock_dep_map =NULL ; +#endif + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_init); + +/** +* drm_gpusvm_notifier_find() - Find GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM structure +* @fault_addr: Fault address +* +* This function finds the GPU SVM notifier associated with the fault address.
+ *
+ * Returns:
+ * Pointer to the GPU SVM notifier on success, NULL otherwise.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+static struct drm_gpusvm_notifier * +drm_gpusvm_notifier_find (struct drm_gpusvm *gpusvm, +unsigned long fault_addr) +{ + return notifier_iter_first(&gpusvm->root, fault_addr,
fault_addr+ 1);drivers/gpu/drm/drm_gpusvm.c new file +4
-0+} + +/** +* to_drm_gpusvm_notifier() - retrieve the container struct for a given rbtree node
+ * @node: a pointer to the rbtree node embedded within adrm_gpusvm_notifierstruct
+ *
+ * Returns: A pointer to the containingdrm_gpusvm_notifierstructure.
+ */
+static structdrm_gpusvm_notifier*to_drm_gpusvm_notifier(structrb_node*node)drivers/gpu/drm/drm_gpusvm.c new file +2
-0+{ + return container_of(node, struct drm_gpusvm_notifier,itree.rb);
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+} + +/** +* drm_gpusvm_notifier_insert() - Insert GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier structure +* +* This function inserts the GPU SVM notifier into the GPU SVM RB tree and list.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static void drm_gpusvm_notifier_insert (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier*notifier)
drivers/gpu/drm/drm_gpusvm.c new file +21
-0+{ +struct rb_node *node; +struct list_head *head; + +interval_tree_insert (¬ifier->itree, &gpusvm->root); + + node =rb_prev (¬ifier->itree.rb); +if (node) + head = &(to_drm_gpusvm_notifier (node))->entry; +else + head = &gpusvm->notifier_list; + +list_add (¬ifier->entry, head); +} + +/** +* drm_gpusvm_notifier_remove() - Remove GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM tructure +* @notifier: Pointer to the GPU SVM notifier structure +* +* This function removes the GPU SVM notifier from the GPU SVM RB tree and list.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static void drm_gpusvm_notifier_remove (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier*notifier)
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+{ +interval_tree_remove (¬ifier->itree, &gpusvm->root); +list_del (¬ifier->entry); +} + +/** +* drm_gpusvm_fini() - Finalize the GPU SVM. +* @gpusvm: Pointer to the GPU SVM structure. +* +* This function finalizes the GPU SVM by cleaning up any remaining ranges and
+ * notifiers, and dropping a reference to struct MM.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+void drm_gpusvm_fini (struct drm_gpusvm *gpusvm) +{ +struct drm_gpusvm_notifier *notifier, *next; + + drm_gpusvm_for_each_notifier_safe(notifier, next, gpusvm, 0,
LONG_MAX) {drivers/gpu/drm/drm_gpusvm.c new file +4
-0+struct drm_gpusvm_range *range, *__next; + +/* +* Remove notifier first to avoid racing with any invalidation
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+*/ +mmu_interval_notifier_remove (¬ifier->notifier); + notifier->flags.removed = true; + + drm_gpusvm_for_each_range_safe(range, __next,notifier, 0,
drivers/gpu/drm/drm_gpusvm.c new file +15
-0+ LONG_MAX) +drm_gpusvm_range_remove (gpusvm ,range ); + } + +mmdrop (gpusvm->mm); +WARN_ON (!RB_EMPTY_ROOT (&gpusvm->root.rb_root)); +} +EXPORT_SYMBOL_GPL (drm_gpusvm_fini); + +/** +* drm_gpusvm_notifier_alloc() - Allocate GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM structure +* @fault_addr: Fault address +* +* This function allocates and initializes the GPU SVM notifier structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM notifier on success,ERR_PTR()on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static struct drm_gpusvm_notifier * +drm_gpusvm_notifier_alloc(struct drm_gpusvm *gpusvm, unsigned long
fault_addr)drivers/gpu/drm/drm_gpusvm.c new file +13
-0+{ +struct drm_gpusvm_notifier *notifier; + +if (gpusvm->ops->notifier_alloc) + notifier = gpusvm->ops->notifier_alloc (); +else + notifier =kzalloc (sizeof (*notifier),GFP_KERNEL ); + +if (!notifier) +return ERR_PTR (-ENOMEM ); + + notifier->gpusvm = gpusvm; + notifier->itree.start = ALIGN_DOWN(fault_addr, gpusvm-Quoted 1 line
notifier_size);drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ notifier->itree.last = ALIGN(fault_addr + 1, gpusvm-Quoted 1 line
notifier_size) - 1;drivers/gpu/drm/drm_gpusvm.c new file +16
-0+INIT_LIST_HEAD (¬ifier->entry); + notifier->root =RB_ROOT_CACHED ; +INIT_LIST_HEAD (¬ifier->range_list); + +return notifier; +} + +/** +* drm_gpusvm_notifier_free() - Free GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier structure +* +* This function frees the GPU SVM notifier structure. +*/ +static void drm_gpusvm_notifier_free (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier*notifier)
drivers/gpu/drm/drm_gpusvm.c new file +11
-0+{ +WARN_ON (!RB_EMPTY_ROOT (¬ifier->root.rb_root)); + +if (gpusvm->ops->notifier_free) + gpusvm->ops->notifier_free (notifier); +else +kfree (notifier); +} + +/** +* to_drm_gpusvm_range() - retrieve the container struct for a given rbtree node
+ * @node: a pointer to the rbtree node embedded within adrm_gpusvm_rangestruct
+ *
+ * Returns: A pointer to the containingdrm_gpusvm_rangestructure.
+ */
+static structdrm_gpusvm_range*to_drm_gpusvm_range(structrb_node*node)drivers/gpu/drm/drm_gpusvm.c new file +2
-0+{ + return container_of(node, struct drm_gpusvm_range,itree.rb);
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+} + +/** +* drm_gpusvm_range_insert() - Insert GPU SVM range +* @notifier: Pointer to the GPU SVM notifier structure +* @range: Pointer to the GPU SVM range structure +* +* This function inserts the GPU SVM range into the notifier RB tree and list.
+ */
+static void drm_gpusvm_range_insert(structdrm_gpusvm_notifier*notifier,drivers/gpu/drm/drm_gpusvm.c new file +24
-0+ struct drm_gpusvm_range *range) +{ +struct rb_node *node; +struct list_head *head; + +drm_gpusvm_notifier_lock (notifier->gpusvm); +interval_tree_insert (&range->itree, ¬ifier->root); + + node =rb_prev (&range->itree.rb); +if (node) + head = &(to_drm_gpusvm_range (node))->entry; +else + head = ¬ifier->range_list; + +list_add (&range->entry, head); +drm_gpusvm_notifier_unlock (notifier->gpusvm); +} + +/** +* __drm_gpusvm_range_remove() - Remove GPU SVM range +* @notifier: Pointer to the GPU SVM notifier structure +* @range: Pointer to the GPU SVM range structure +* +* This macro removes the GPU SVM range from the notifier RB tree and list.
+ */
+static void __drm_gpusvm_range_remove(structdrm_gpusvm_notifier*notifier,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct drm_gpusvm_range*range)
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+{ +interval_tree_remove (&range->itree, ¬ifier->root); +list_del (&range->entry); +} + +/** +* drm_gpusvm_range_alloc() - Allocate GPU SVM range +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier structure +* @fault_addr: Fault address +* @chunk_size: Chunk size +* @migrate_devmem: Flag indicating whether to migrate device memory +* +* This function allocates and initializes the GPU SVM range structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM range on success,ERR_PTR()on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+static struct drm_gpusvm_range * +drm_gpusvm_range_alloc (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + unsigned long fault_addr, unsigned long
chunk_size,drivers/gpu/drm/drm_gpusvm.c new file +32
-0+ bool migrate_devmem) +{ +struct drm_gpusvm_range *range; + +if (gpusvm->ops->range_alloc) + range = gpusvm->ops->range_alloc (gpusvm); +else + range =kzalloc (sizeof (*range),GFP_KERNEL ); + +if (!range) +return ERR_PTR (-ENOMEM ); + +kref_init (&range->refcount); + range->gpusvm = gpusvm; + range->notifier = notifier; + range->itree.start =ALIGN_DOWN (fault_addr, chunk_size); + range->itree.last =ALIGN (fault_addr +1 , chunk_size) -1 ; +INIT_LIST_HEAD (&range->entry); + range->notifier_seq =LONG_MAX ; + range->flags.migrate_devmem = migrate_devmem ?1 :0 ; + +return range; +} + +/** +* drm_gpusvm_check_pages() - Check pages +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier structure +* @start: Start address +* @end: End address +* +* Check if pages between start and end have been faulted in on the CPU. Use to
+ * prevent migration of pages without CPU backing store.
+ *
+ * Returns:
+ * True if pages have been faulted into CPU, False otherwise
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static bool drm_gpusvm_check_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier*notifier,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long start, unsignedlong end)
drivers/gpu/drm/drm_gpusvm.c new file +7
-0+{ + struct hmm_range hmm_range = { + .default_flags = 0, + .notifier = ¬ifier->notifier, + .start = start, + .end = end, + .dev_private_owner = gpusvm-Quoted 1 line
device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ }; +unsigned long timeout = + jiffies +
msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);drivers/gpu/drm/drm_gpusvm.c new file +11
-0+unsigned long *pfns; +unsigned long npages =npages_in_range (start, end); +int err, i; + +mmap_assert_locked (gpusvm->mm); + + pfns =kvmalloc_array (npages,sizeof (*pfns),GFP_KERNEL ); +if (!pfns) +return false; + + hmm_range.notifier_seq = mmu_interval_read_begin(¬ifier-Quoted 1 line
notifier);
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+ hmm_range.hmm_pfns = pfns; + +while (true) { + err =hmm_range_fault (&hmm_range); +if (err == -EBUSY ) { +if (time_after (jiffies, timeout)) +break ; + + hmm_range.notifier_seq = + mmu_interval_read_begin(¬ifier-Quoted 1 line
notifier);
drivers/gpu/drm/drm_gpusvm.c new file +22
-0+continue ; + } +break ; + } +if (err) + goto err_free; + +for (i =0 ; i < npages;) { +if (!(pfns[i] &HMM_PFN_VALID )) { + err = -EFAULT ; + goto err_free; + } + i +=0x1 <<hmm_pfn_to_map_order (pfns[i]); + } + +err_free: +kvfree (pfns); +return err ? false : true; +} + +/** +* drm_gpusvm_range_chunk_size() - Determine chunk size for GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @vas: Pointer to the virtual memory area structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @check_pages_threshold: Check CPU pages for present threshold
+ *
+ * This function determines the chunk size for the GPU SVM range based on the
+ * fault address, GPU SVM chunk sizes, existing GPU SVM ranges, and the virtual
+ * memory area boundaries.
+ *
+ * Returns:
+ * Chunk size on success,LONG_MAXon failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +15
-0+static unsigned long +drm_gpusvm_range_chunk_size (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + struct vm_area_struct *vas, + unsigned long fault_addr, + unsigned long gpuva_start, + unsigned long gpuva_end, + unsigned long check_pages_threshold) +{ +unsigned long start, end; +int i =0 ; + +retry: +for (; i < gpusvm->num_chunks; ++i) { + start = ALIGN_DOWN(fault_addr, gpusvm-Quoted 1 line
chunk_sizes[i]);
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+ end =ALIGN (fault_addr +1 , gpusvm->chunk_sizes[i]); + +if (start >= vas->vm_start && end <= vas->vm_end && + start >= notifier->itree.start && + end <= notifier->itree.last + 1 && + start >= gpuva_start && end <= gpuva_end) +break ; + } + +if (i == gpusvm->num_chunks) +return LONG_MAX ; + +/* +* If allocation more than page, ensure not to overlap with existing
drivers/gpu/drm/drm_gpusvm.c new file +13
-0+* ranges. +*/ +if (end - start !=SZ_4K ) { +struct drm_gpusvm_range *range; + + range =drm_gpusvm_range_find (notifier, start, end); +if (range) { + ++i; + goto retry; + } + +/* +* XXX: Only create range on pages CPU has faulted in. Without
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* this check, or prefault, on BMG '
xe_exec_system_allocator--rdrivers/gpu/drm/drm_gpusvm.c new file +1
-0+* process-many-malloc' fails. In the failure case, each process
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* mallocs 16k but the CPU VMA is ~128k which results in 64k SVM
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* ranges. When migrating the SVM ranges, some processes fail in
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* drm_gpusvm_migrate_to_devmem with 'migrate.cpages != npages'
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* and then upon drm_gpusvm_range_get_pages device pages from
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* other processes are collected + faulted in which creates all
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* sorts of problems. Unsure exactly how this happening, also
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* problem goes away if 'xe_exec_system_allocator -- r
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* process-many-malloc' mallocs at least 64k at a time.
drivers/gpu/drm/drm_gpusvm.c new file +3
-0+*/ +if (end - start <= check_pages_threshold && + !drm_gpusvm_check_pages(gpusvm, notifier, start,end)) {
drivers/gpu/drm/drm_gpusvm.c new file +17
-0+ ++i; + goto retry; + } + } + +return end - start; +} + +/** +* drm_gpusvm_range_find_or_insert() - Find or insert GPU SVM range +* @gpusvm: Pointer to the GPU SVM structure +* @fault_addr: Fault address +* @gpuva_start: Start address of GPUVA which mirrors CPU +* @gpuva_end: End address of GPUVA which mirrors CPU +* @ctx: GPU SVM context +* +* This function finds or inserts a newly allocated a GPU SVM range based on the
+ * fault address. Caller must hold a lock to protect range lookup and insertion.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range on success,ERR_PTR()on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +28
-0+struct drm_gpusvm_range * +drm_gpusvm_range_find_or_insert (struct drm_gpusvm *gpusvm, +unsigned long fault_addr, +unsigned long gpuva_start, +unsigned long gpuva_end, +const struct drm_gpusvm_ctx *ctx) +{ +struct drm_gpusvm_notifier *notifier; +struct drm_gpusvm_range *range; +struct mm_struct *mm = gpusvm->mm; +struct vm_area_struct *vas; +bool notifier_alloc = false; +unsigned long chunk_size; +int err; +bool migrate_devmem; + +drm_gpusvm_driver_lock_held (gpusvm); + +if (fault_addr < gpusvm->mm_start || + fault_addr > gpusvm->mm_start + gpusvm->mm_range) +return ERR_PTR (-EINVAL ); + +if (!mmget_not_zero (mm)) +return ERR_PTR (-EFAULT ); + + notifier =drm_gpusvm_notifier_find (gpusvm, fault_addr); +if (!notifier) { + notifier = drm_gpusvm_notifier_alloc(gpusvm,
fault_addr);drivers/gpu/drm/drm_gpusvm.c new file +6
-0+if (IS_ERR (notifier)) { + err =PTR_ERR (notifier); + goto err_mmunlock; + } + notifier_alloc = true; + err = mmu_interval_notifier_insert(¬ifier-Quoted 1 line
notifier,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ mm, notifier-Quoted 1 line
itree.start,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ notifier-Quoted 1 line
itree.last -
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ notifier-Quoted 1 line
itree.start + 1,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+
&drm_gpusvm_notifier_ops);drivers/gpu/drm/drm_gpusvm.c new file +18
-0+if (err) + goto err_notifier; + } + +mmap_read_lock (mm); + + vas =vma_lookup (mm, fault_addr); +if (!vas) { + err = -ENOENT ; + goto err_notifier_remove; + } + +if (!ctx->read_only && !(vas->vm_flags &VM_WRITE )) { + err = -EPERM ; + goto err_notifier_remove; + } + + range = drm_gpusvm_range_find(notifier, fault_addr,
fault_addr+ 1);drivers/gpu/drm/drm_gpusvm.c new file +4
-0+if (range) + goto out_mmunlock; +/* +* XXX: Short-circuiting migration based on migrate_vma_* current
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* limitations. If/when migrate_vma_* add more support, this logic will
drivers/gpu/drm/drm_gpusvm.c new file +6
-0+* have to change. +*/ + migrate_devmem = ctx->devmem_possible && +vma_is_anonymous (vas) && !is_vm_hugetlb_page (vas); + + chunk_size = drm_gpusvm_range_chunk_size(gpusvm, notifier,vas,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ fault_addr,
gpuva_start,drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ gpuva_end, + ctx-Quoted 1 line
check_pages_threshold);drivers/gpu/drm/drm_gpusvm.c new file +6
-0+if (chunk_size ==LONG_MAX ) { + err = -EINVAL ; + goto err_notifier_remove; + } + + range = drm_gpusvm_range_alloc(gpusvm, notifier, fault_addr,
chunk_size,drivers/gpu/drm/drm_gpusvm.c new file +31
-0+ migrate_devmem); +if (IS_ERR (range)) { + err =PTR_ERR (range); + goto err_notifier_remove; + } + +drm_gpusvm_range_insert (notifier, range); +if (notifier_alloc) +drm_gpusvm_notifier_insert (gpusvm, notifier); + +out_mmunlock: +mmap_read_unlock (mm); +mmput (mm); + +return range; + +err_notifier_remove: +mmap_read_unlock (mm); +if (notifier_alloc) +mmu_interval_notifier_remove (¬ifier->notifier); +err_notifier: +if (notifier_alloc) +drm_gpusvm_notifier_free (gpusvm, notifier); +err_mmunlock: +mmput (mm); +return ERR_PTR (err); +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_find_or_insert); + +/** +* __drm_gpusvm_range_unmap_pages() - Unmap pages associated with a GPU SVM range (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @npages: Number of pages to unmap
+ *
+ * This function unmap pages associated with a GPU SVM range. Assumes and
+ * asserts correct locking is in place when called.
+ */
+static void __drm_gpusvm_range_unmap_pages(structdrm_gpusvm*gpusvm,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct drm_gpusvm_range*range,
drivers/gpu/drm/drm_gpusvm.c new file +11
-0+ unsigned long npages) +{ +unsigned long i, j; +struct drm_pagemap *dpagemap = range->dpagemap; +struct device *dev = gpusvm->drm->dev; + +lockdep_assert_held (&gpusvm->notifier_lock); + +if (range->flags.has_dma_mapping) { +for (i =0 , j =0 ; i < npages; j++) { + struct drm_pagemap_dma_addr *addr = &range-Quoted 1 line
dma_addr[j];
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +if (addr->proto ==DRM_INTERCONNECT_SYSTEM ) +dma_unmap_page (dev, + addr->addr, + PAGE_SIZE << addr-Quoted 1 line
order,
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ addr->dir); + else if (dpagemap && dpagemap->ops-Quoted 1 line
unmap_dma)drivers/gpu/drm/drm_gpusvm.c new file +13
-0+ dpagemap->ops->unmap_dma (dpagemap, + dev, + *addr); + i +=1 << addr->order; + } + range->flags.has_devmem_pages = false; + range->flags.has_dma_mapping = false; + range->dpagemap =NULL ; + } +} + +/** +* drm_gpusvm_range_free_pages() - Free pages associated with a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function frees the dma address array associated with a GPU SVM range.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static void drm_gpusvm_range_free_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range*range)
drivers/gpu/drm/drm_gpusvm.c new file +15
-0+{ +lockdep_assert_held (&gpusvm->notifier_lock); + +if (range->dma_addr) { +kvfree (range->dma_addr); + range->dma_addr =NULL ; + } +} + +/** +* drm_gpusvm_range_remove() - Remove GPU SVM range +* @gpusvm: Pointer to the GPU SVM structure +* @range: Pointer to the GPU SVM range to be removed +* +* This function removes the specified GPU SVM range and also removes the parent
+ * GPU SVM notifier if no more ranges remain in the notifier. The caller must
+ * hold a lock to protect range and notifier removal.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+void drm_gpusvm_range_remove (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range) +{ + unsigned long npages = npages_in_range(range->itree.start, + range->itree.last +1);
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+struct drm_gpusvm_notifier *notifier; + +drm_gpusvm_driver_lock_held (gpusvm); + + notifier = drm_gpusvm_notifier_find(gpusvm, range-Quoted 1 line
itree.start);
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+if (WARN_ON_ONCE (!notifier)) +return ; + +drm_gpusvm_notifier_lock (gpusvm); +__drm_gpusvm_range_unmap_pages (gpusvm, range, npages); +drm_gpusvm_range_free_pages (gpusvm, range); +__drm_gpusvm_range_remove (notifier, range); +drm_gpusvm_notifier_unlock (gpusvm); + +drm_gpusvm_range_put (range); + +if (RB_EMPTY_ROOT (¬ifier->root.rb_root)) { +if (!notifier->flags.removed) + mmu_interval_notifier_remove(¬ifier-Quoted 1 line
notifier);
drivers/gpu/drm/drm_gpusvm.c new file +11
-0+drm_gpusvm_notifier_remove (gpusvm, notifier); +drm_gpusvm_notifier_free (gpusvm, notifier); + } +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_remove); + +/** +* drm_gpusvm_range_get() - Get a reference to GPU SVM range +* @range: Pointer to the GPU SVM range +* +* This function increments the reference count of the specified GPU SVM range.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range.
+ */drivers/gpu/drm/drm_gpusvm.c new file +12
-0+struct drm_gpusvm_range * +drm_gpusvm_range_get (struct drm_gpusvm_range *range) +{ +kref_get (&range->refcount); + +return range; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_get); + +/** +* drm_gpusvm_range_destroy() - Destroy GPU SVM range +* @refcount: Pointer to the reference counter embedded in the GPU SVM range
+ *
+ * This function destroys the specified GPU SVM range when its reference count
+ * reaches zero. If a custom range-free function is provided, it is invoked to
+ * free the range; otherwise, the range is deallocated usingkfree().
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+static void drm_gpusvm_range_destroy (struct kref *refcount) +{ +struct drm_gpusvm_range *range = + container_of(refcount, struct drm_gpusvm_range,refcount);
drivers/gpu/drm/drm_gpusvm.c new file +13
-0+struct drm_gpusvm *gpusvm = range->gpusvm; + +if (gpusvm->ops->range_free) + gpusvm->ops->range_free (range); +else +kfree (range); +} + +/** +* drm_gpusvm_range_put() - Put a reference to GPU SVM range +* @range: Pointer to the GPU SVM range +* +* This function decrements the reference count of the specified GPU SVM range
+ * and frees it when the count reaches zero.
+ */drivers/gpu/drm/drm_gpusvm.c new file +12
-0+void drm_gpusvm_range_put (struct drm_gpusvm_range *range) +{ +kref_put (&range->refcount, drm_gpusvm_range_destroy); +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_put); + +/** +* drm_gpusvm_range_pages_valid() - GPU SVM range pages valid +* @gpusvm: Pointer to the GPU SVM structure +* @range: Pointer to the GPU SVM range structure +* +* This function determines if a GPU SVM range pages are valid. Expected be
+ * called holdinggpusvm->notifier_lockand as the last step before committing a
+ * GPU binding. This is akin to a notifier seqno check in the HMM documentation
+ * but due to wider notifiers (i.e., notifiers which span multiple ranges) this
+ * function is required for finer grained checking (i.e., per range) if pages
+ * are valid.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */drivers/gpu/drm/drm_gpusvm.c new file +6
-0+bool drm_gpusvm_range_pages_valid (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range) +{ +lockdep_assert_held (&gpusvm->notifier_lock); + + return range->flags.has_devmem_pages || range-Quoted 1 line
flags.has_dma_mapping;
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_pages_valid); + +/** +* drm_gpusvm_range_pages_valid_unlocked() - GPU SVM range pages valid unlocked
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid. Expected be
+ * called without holdinggpusvm->notifier_lock.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+static bool +drm_gpusvm_range_pages_valid_unlocked (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range*range)
drivers/gpu/drm/drm_gpusvm.c new file +22
-0+{ +bool pages_valid; + +if (!range->dma_addr) +return false; + +drm_gpusvm_notifier_lock (gpusvm); + pages_valid =drm_gpusvm_range_pages_valid (gpusvm, range); +if (!pages_valid) +drm_gpusvm_range_free_pages (gpusvm, range); +drm_gpusvm_notifier_unlock (gpusvm); + +return pages_valid; +} + +/** +* drm_gpusvm_range_get_pages() - Get pages for a GPU SVM range +* @gpusvm: Pointer to the GPU SVM structure +* @range: Pointer to the GPU SVM range structure +* @ctx: GPU SVM context +* +* This function gets pages for a GPU SVM range and ensures they are mapped for
+ * DMA access.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+int drm_gpusvm_range_get_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx) +{ + struct mmu_interval_notifier *notifier = &range->notifier-Quoted 1 line
notifier;
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+struct hmm_range hmm_range = { + .default_flags = HMM_PFN_REQ_FAULT | (ctx->read_only? 0 :
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ HMM_PFN_REQ_WRITE), + .notifier = notifier, + .start = range->itree.start, + .end = range->itree.last + 1, + .dev_private_owner = gpusvm-Quoted 1 line
device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ }; +struct mm_struct *mm = gpusvm->mm; +struct drm_gpusvm_zdd *zdd; +unsigned long timeout = + jiffies +
msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);drivers/gpu/drm/drm_gpusvm.c new file +3
-0+unsigned long i, j; + unsigned long npages = npages_in_range(range->itree.start, + range->itree.last +1);
drivers/gpu/drm/drm_gpusvm.c new file +58
-0+unsigned long num_dma_mapped; +unsigned int order =0 ; +unsigned long *pfns; +struct page **pages; +int err =0 ; +struct dev_pagemap *pagemap; +struct drm_pagemap *dpagemap; + +retry: + hmm_range.notifier_seq =mmu_interval_read_begin (notifier); +if (drm_gpusvm_range_pages_valid_unlocked (gpusvm, range)) + goto set_seqno; + + pfns =kvmalloc_array (npages,sizeof (*pfns),GFP_KERNEL ); +if (!pfns) +return -ENOMEM ; + +if (!mmget_not_zero (mm)) { + err = -EFAULT ; + goto err_free; + } + + hmm_range.hmm_pfns = pfns; +while (true) { +mmap_read_lock (mm); + err =hmm_range_fault (&hmm_range); +mmap_read_unlock (mm); + +if (err == -EBUSY ) { +if (time_after (jiffies, timeout)) +break ; + + hmm_range.notifier_seq = +mmu_interval_read_begin (notifier); +continue ; + } +break ; + } +mmput (mm); +if (err) + goto err_free; + + pages = (struct page **)pfns; +map_pages: +/* +* Perform all dma mappings under the notifier lock to not +* access freed pages. A notifier will either block on +* the notifier lock or unmap dma. +*/ +drm_gpusvm_notifier_lock (gpusvm); + +if (range->flags.unmapped) { +drm_gpusvm_notifier_unlock (gpusvm); + err = -EFAULT ; + goto err_free; + } + + if (mmu_interval_read_retry(notifier,hmm_range.notifier_seq)) {
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+drm_gpusvm_notifier_unlock (gpusvm); +kvfree (pfns); + goto retry; + } + +if (!range->dma_addr) { +/* Unlock and restart mapping to allocate memory. */ +drm_gpusvm_notifier_unlock (gpusvm); + range->dma_addr = kvmalloc_array(npages, + sizeof(*range-Quoted 1 line
dma_addr),drivers/gpu/drm/drm_gpusvm.c new file +17
-0+ GFP_KERNEL); +if (!range->dma_addr) { + err = -ENOMEM ; + goto err_free; + } + goto map_pages; + } + + zdd =NULL ; + num_dma_mapped =0 ; +for (i =0 , j =0 ; i < npages; ++j) { +struct page *page =hmm_pfn_to_page (pfns[i]); + + order =hmm_pfn_to_map_order (pfns[i]); +if (is_device_private_page (page) || + is_device_coherent_page(page)) { +if (zdd != page->zone_device_data && i >0 ){
drivers/gpu/drm/drm_gpusvm.c new file +12
-0+ err = -EOPNOTSUPP ; + goto err_unmap; + } + zdd = page->zone_device_data; +if (pagemap != page->pgmap) { +if (i >0 ) { + err = -EOPNOTSUPP ; + goto err_unmap; + } + + pagemap = page->pgmap; + dpagemap = zdd->devmem_allocation-Quoted 1 line
dpagemap;
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ if (drm_WARN_ON(gpusvm->drm,!dpagemap)) {
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+/* +* Raced. This is not supposed to happen
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* since hmm_range_fault() should've migrated
drivers/gpu/drm/drm_gpusvm.c new file +9
-0+* this page to system. +*/ + err = -EAGAIN ; + goto err_unmap; + } + } + range->dma_addr[j] = + dpagemap->ops->map_dma(dpagemap, + gpusvm->drm-Quoted 1 line
dev,
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ page, order, +
DMA_BIDIRECTIONAL);drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ if (dma_mapping_error(gpusvm->drm->dev, + range-Quoted 1 line
dma_addr[j].addr)) {
drivers/gpu/drm/drm_gpusvm.c new file +18
-0+ err = -EFAULT ; + goto err_unmap; + } + + pages[i] = page; + }else { +dma_addr_t addr; + +if (is_zone_device_page (page) || zdd) { + err = -EOPNOTSUPP ; + goto err_unmap; + } + + addr =dma_map_page (gpusvm->drm->dev, + page, 0, + PAGE_SIZE << order, + DMA_BIDIRECTIONAL); + if (dma_mapping_error(gpusvm->drm->dev,addr)) {
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ err = -EFAULT ; + goto err_unmap; + } + + range->dma_addr[j] =
drm_pagemap_dma_addr_encodedrivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (addr, DRM_INTERCONNECT_SYSTEM,order,
drivers/gpu/drm/drm_gpusvm.c new file +21
-0+ DMA_BIDIRECTIONAL); + } + i +=1 << order; + num_dma_mapped = i; + } + + range->flags.has_dma_mapping = true; +if (zdd) { + range->flags.has_devmem_pages = true; + range->dpagemap = dpagemap; + } + +drm_gpusvm_notifier_unlock (gpusvm); +kvfree (pfns); +set_seqno: + range->notifier_seq = hmm_range.notifier_seq; + +return 0 ; + +err_unmap: + __drm_gpusvm_range_unmap_pages(gpusvm, range,
num_dma_mapped);drivers/gpu/drm/drm_gpusvm.c new file +11
-0+drm_gpusvm_notifier_unlock (gpusvm); +err_free: +kvfree (pfns); +if (err == -EAGAIN ) + goto retry; +return err; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_get_pages); + +/** +* drm_gpusvm_range_unmap_pages() - Unmap pages associated with a GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If @in_notifier
+ * is set, it is assumed thatgpusvm->notifier_lockis held in write mode; if it
+ * is clear, it acquiresgpusvm->notifier_lockin read mode. Must be called on
+ * each GPU SVM range attached to notifier in gpusvm->ops-Quoted 1 line
invalidate for IOMMU
+ * security model.
+ */drivers/gpu/drm/drm_gpusvm.c new file +6
-0+void drm_gpusvm_range_unmap_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx) +{ + unsigned long npages = npages_in_range(range->itree.start, + range->itree.last +1);
drivers/gpu/drm/drm_gpusvm.c new file +33
-0+ +if (ctx->in_notifier) +lockdep_assert_held_write (&gpusvm->notifier_lock); +else +drm_gpusvm_notifier_lock (gpusvm); + +__drm_gpusvm_range_unmap_pages (gpusvm, range, npages); + +if (!ctx->in_notifier) +drm_gpusvm_notifier_unlock (gpusvm); +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_unmap_pages); + +/** +* drm_gpusvm_migration_unlock_put_page() - Put a migration page +* @page: Pointer to the page to put +* +* This function unlocks and puts a page. +*/ +static void drm_gpusvm_migration_unlock_put_page (struct page *page) +{ +unlock_page (page); +put_page (page); +} + +/** +* drm_gpusvm_migration_unlock_put_pages() - Put migration pages +* @npages: Number of pages +* @migrate_pfn: Array of migrate page frame numbers +* +* This function unlocks and puts an array of pages. +*/ +static void drm_gpusvm_migration_unlock_put_pages(unsigned longnpages,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long
*migrate_pfn)drivers/gpu/drm/drm_gpusvm.c new file +17
-0+{ +unsigned long i; + +for (i =0 ; i < npages; ++i) { +struct page *page; + +if (!migrate_pfn[i]) +continue ; + + page =migrate_pfn_to_page (migrate_pfn[i]); +drm_gpusvm_migration_unlock_put_page (page); + migrate_pfn[i] =0 ; + } +} + +/** +* drm_gpusvm_get_devmem_page() - Get a reference to a device memory page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU SVM zone
+ * device data and initializes it for zone device usage.
+ */drivers/gpu/drm/drm_gpusvm.c new file +9
-0+static void drm_gpusvm_get_devmem_page (struct page *page, + struct drm_gpusvm_zdd *zdd) +{ + page->zone_device_data =drm_gpusvm_zdd_get (zdd); +zone_device_page_init (page); +} + +/** +* drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g.,DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn, maps the
+ * corresponding page, and stores the DMA address in the provided @dma_addr
+ * array.
+ *
+ * Returns: 0 on success,-EFAULTif an error occurs during mapping.
+ */drivers/gpu/drm/drm_gpusvm.c new file +10
-0+static int drm_gpusvm_migrate_map_pages (struct device *dev, +dma_addr_t *dma_addr, +unsigned long *migrate_pfn, +unsigned long npages, +enum dma_data_direction dir) +{ +unsigned long i; + +for (i =0 ; i < npages; ++i) { + struct page *page =
migrate_pfn_to_page(migrate_pfn[i]);drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ +if (!page) +continue ; + +if (WARN_ON_ONCE (is_zone_device_page (page))) +return -EFAULT ; + + dma_addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE,dir);
drivers/gpu/drm/drm_gpusvm.c new file +9
-0+if (dma_mapping_error (dev, dma_addr[i])) +return -EFAULT ; + } + +return 0 ; +} + +/** +* drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g.,DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+static void drm_gpusvm_migrate_unmap_pages (struct device *dev, + dma_addr_t *dma_addr, + unsigned long npages, + enum dma_data_directiondir)
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+{ + unsigned long i; + + for (i = 0; i < npages; ++i) { + if (!dma_addr[i] || dma_mapping_error(dev,dma_addr[i]))
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+continue ; + +dma_unmap_page (dev, dma_addr[i],PAGE_SIZE , dir); + } +} + +/** +* drm_gpusvm_migrate_to_devmem() - Migrate GPU SVM range to device memory
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @devmem_allocation: Pointer to the device memory allocation. The caller
+ * should hold a reference to the device memory allocation,
+ * which should be dropped via ops-Quoted 1 line
devmem_releaseor upon+ * the failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to device memory. It performs the
+ * necessary setup and invokes the driver-specific operations for migration to
+ * device memory. Upon successful return, @devmem_allocation can safely reference @range
+ * untilops->devmem_releaseis called which only upon successful return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+int drm_gpusvm_migrate_to_devmem (struct drm_gpusvm *gpusvm, +struct drm_gpusvm_range *range, + struct drm_gpusvm_devmem
*devmem_allocation,drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ const struct drm_gpusvm_ctx *ctx) +{ + const struct drm_gpusvm_devmem_ops *ops = devmem_allocation-Quoted 1 line
ops;
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long start = range->itree.start, end = range-Quoted 1 line
itree.last + 1;
drivers/gpu/drm/drm_gpusvm.c new file +45
-0+struct migrate_vma migrate = { + .start = start, + .end = end, + .pgmap_owner = gpusvm->device_private_page_owner, + .flags =MIGRATE_VMA_SELECT_SYSTEM , + }; +struct mm_struct *mm = gpusvm->mm; +unsigned long i, npages =npages_in_range (start, end); +struct vm_area_struct *vas; +struct drm_gpusvm_zdd *zdd =NULL ; +struct page **pages; +dma_addr_t *dma_addr; +void *buf; +int err; + +if (!range->flags.migrate_devmem) +return -EINVAL ; + +if (!ops->populate_devmem_pfn || !ops->copy_to_devmem || + !ops->copy_to_ram) +return -EOPNOTSUPP ; + +if (!mmget_not_zero (mm)) { + err = -EFAULT ; + goto err_out; + } +mmap_read_lock (mm); + + vas =vma_lookup (mm, start); +if (!vas) { + err = -ENOENT ; + goto err_mmunlock; + } + +if (end > vas->vm_end || start < vas->vm_start) { + err = -EINVAL ; + goto err_mmunlock; + } + +if (!vma_is_anonymous (vas)) { + err = -EBUSY ; + goto err_mmunlock; + } + + buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +
sizeof(*dma_addr)+drivers/gpu/drm/drm_gpusvm.c new file +7
-0+ sizeof(*pages), GFP_KERNEL); +if (!buf) { + err = -ENOMEM ; + goto err_mmunlock; + } + dma_addr = buf + (2 *sizeof (*migrate.src) * npages); + pages = buf + (2 *sizeof (*migrate.src) +sizeof (*dma_addr))* npages;
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ + zdd = drm_gpusvm_zdd_alloc(gpusvm-Quoted 1 line
device_private_page_owner);drivers/gpu/drm/drm_gpusvm.c new file +24
-0+if (!zdd) { + err = -ENOMEM ; + goto err_free; + } + + migrate.vma = vas; + migrate.src = buf; + migrate.dst = migrate.src + npages; + + err =migrate_vma_setup (&migrate); +if (err) + goto err_free; + +if (!migrate.cpages) { + err = -EFAULT ; + goto err_free; + } + +if (migrate.cpages != npages) { + err = -EBUSY ; + goto err_finalize; + } + + err = ops->populate_devmem_pfn(devmem_allocation, npages,migrate.dst);
drivers/gpu/drm/drm_gpusvm.c new file +4
-0+if (err) + goto err_finalize; + + err = drm_gpusvm_migrate_map_pages(devmem_allocation->dev,
dma_addr,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ migrate.src, npages,
DMA_TO_DEVICE);drivers/gpu/drm/drm_gpusvm.c new file +17
-0+if (err) + goto err_finalize; + +for (i =0 ; i < npages; ++i) { +struct page *page =pfn_to_page (migrate.dst[i]); + + pages[i] = page; + migrate.dst[i] =migrate_pfn (migrate.dst[i]); +drm_gpusvm_get_devmem_page (page, zdd); + } + + err = ops->copy_to_devmem (pages, dma_addr, npages); +if (err) + goto err_finalize; + +/* Upon success bind devmem allocation to range and zdd */ + zdd->devmem_allocation = devmem_allocation;/* Owns ref */
drivers/gpu/drm/drm_gpusvm.c new file +4
-0+ +err_finalize: +if (err) + drm_gpusvm_migration_unlock_put_pages(npages,migrate.dst);
drivers/gpu/drm/drm_gpusvm.c new file +3
-0+migrate_vma_pages (&migrate); +migrate_vma_finalize (&migrate); + drm_gpusvm_migrate_unmap_pages(devmem_allocation->dev,
dma_addr, npages,drivers/gpu/drm/drm_gpusvm.c new file +15
-0+ DMA_TO_DEVICE); +err_free: +if (zdd) +drm_gpusvm_zdd_put (zdd); +kvfree (buf); +err_mmunlock: +mmap_read_unlock (mm); +mmput (mm); +err_out: +return err; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_migrate_to_devmem); + +/** +* drm_gpusvm_migrate_populate_ram_pfn() - Populate RAM PFNs for a VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @fault_page: Fault page
+ * @npages: Number of pages to populate
+ * @mpages: Number of pages to migrate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the RAM migrate page frame numbers (PFNs) for the
+ * specified VM area structure. It allocates and locks pages in the VM area for
+ * RAM usage. If vas is non-NULL usealloc_page_vmafor allocation, if NULL use
+ *alloc_pagefor allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_ram_pfn(structvm_area_struct*vas,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct page
*fault_page,drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ unsigned long npages, + unsigned long*mpages,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long
*src_mpfn,drivers/gpu/drm/drm_gpusvm.c new file +23
-0+ unsigned long *mpfn, + unsigned long addr) +{ +unsigned long i; + +for (i =0 ; i < npages; ++i, addr +=PAGE_SIZE ) { +struct page *page, *src_page; + +if (!(src_mpfn[i] &MIGRATE_PFN_MIGRATE )) +continue ; + + src_page =migrate_pfn_to_page (src_mpfn[i]); +if (!src_page) +continue ; + +if (fault_page) { +if (src_page->zone_device_data != + fault_page->zone_device_data) +continue ; + } + +if (vas) + page = alloc_page_vma(GFP_HIGHUSER, vas,addr);
drivers/gpu/drm/drm_gpusvm.c new file +39
-0+else + page =alloc_page (GFP_HIGHUSER ); + +if (!page) + goto free_pages; + + mpfn[i] =migrate_pfn (page_to_pfn (page)); + } + +for (i =0 ; i < npages; ++i) { +struct page *page =migrate_pfn_to_page (mpfn[i]); + +if (!page) +continue ; + +WARN_ON_ONCE (!trylock_page (page)); + ++*mpages; + } + +return 0 ; + +free_pages: +for (i =0 ; i < npages; ++i) { +struct page *page =migrate_pfn_to_page (mpfn[i]); + +if (!page) +continue ; + +put_page (page); + mpfn[i] =0 ; + } +return -ENOMEM ; +} + +/** +* drm_gpusvm_evict_to_ram() - Evict GPU SVM range to RAM +* @devmem_allocation: Pointer to the device memory allocation +* +* Similar to __drm_gpusvm_migrate_to_ram but does not require mmap lock and
+ * migration done via migrate_device_* functions.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+intdrm_gpusvm_evict_to_ram(struct drm_gpusvm_devmem *devmem_allocation)drivers/gpu/drm/drm_gpusvm.c new file +2
-0+{ + const struct drm_gpusvm_devmem_ops *ops = devmem_allocation-Quoted 1 line
ops;
drivers/gpu/drm/drm_gpusvm.c new file +25
-0+unsigned long npages, mpages =0 ; +struct page **pages; +unsigned long *src, *dst; +dma_addr_t *dma_addr; +void *buf; +int i, err =0 ; +unsigned int retry_count =2 ; + + npages = devmem_allocation->size >>PAGE_SHIFT ; + +retry: +if (!mmget_not_zero (devmem_allocation->mm)) +return -EFAULT ; + + buf = kvcalloc(npages, 2 * sizeof(*src) + sizeof(*dma_addr) + + sizeof(*pages), GFP_KERNEL); +if (!buf) { + err = -ENOMEM ; + goto err_out; + } + src = buf; + dst = buf + (sizeof (*src) * npages); + dma_addr = buf + (2 *sizeof (*src) * npages); + pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) *npages;
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ + err = ops->populate_devmem_pfn(devmem_allocation, npages,src);
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+if (err) + goto err_free; + + err =migrate_device_pfns (src, npages); +if (err) + goto err_free; + + err = drm_gpusvm_migrate_populate_ram_pfn(NULL, NULL,npages, &mpages,
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ src, dst, 0); +if (err || !mpages) + goto err_finalize; + + err = drm_gpusvm_migrate_map_pages(devmem_allocation->dev,
dma_addr,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ dst, npages,
DMA_FROM_DEVICE);drivers/gpu/drm/drm_gpusvm.c new file +16
-0+if (err) + goto err_finalize; + +for (i =0 ; i < npages; ++i) + pages[i] =migrate_pfn_to_page (src[i]); + + err = ops->copy_to_ram (pages, dma_addr, npages); +if (err) + goto err_finalize; + +err_finalize: +if (err) +drm_gpusvm_migration_unlock_put_pages (npages, dst); +migrate_device_pages (src, dst, npages); +migrate_device_finalize (src, dst, npages); + drm_gpusvm_migrate_unmap_pages(devmem_allocation->dev,
dma_addr, npages,drivers/gpu/drm/drm_gpusvm.c new file +20
-0+ DMA_FROM_DEVICE); +err_free: +kvfree (buf); +err_out: +mmput_async (devmem_allocation->mm); + +if (completion_done (&devmem_allocation->detached)) +return 0 ; + +if (!err || retry_count--) { +cond_resched (); + goto retry; + } + +return err; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_evict_to_ram); + +/** +* __drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM (internal)
+ * @vas: Pointer to the VM area structure
+ * @device_private_page_owner: Device private pages owner
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @fault_addr: Fault address
+ * @size: Size of migration
+ *
+ * This internal function performs the migration of the specified GPU SVM range
+ * to RAM. It sets up the migration, populates + dma maps RAM PFNs, and
+ * invokes the driver-specific operations for migration to RAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static int __drm_gpusvm_migrate_to_ram (struct vm_area_struct *vas, + void
*device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ struct page *page, + unsigned long fault_addr, + unsigned long size) +{ +struct migrate_vma migrate = { + .vma = vas, + .pgmap_owner = device_private_page_owner, + .flags =MIGRATE_VMA_SELECT_DEVICE_PRIVATE |
drivers/gpu/drm/drm_gpusvm.c new file +17
-0+ MIGRATE_VMA_SELECT_DEVICE_COHERENT, + .fault_page = page, + }; +struct drm_gpusvm_zdd *zdd; +const struct drm_gpusvm_devmem_ops *ops; +struct device *dev; +unsigned long npages, mpages =0 ; +struct page **pages; +dma_addr_t *dma_addr; +unsigned long start, end; +void *buf; +int i, err =0 ; + + start =ALIGN_DOWN (fault_addr, size); + end =ALIGN (fault_addr +1 , size); + +/* Corner where VMA area struct has been partially unmapped */
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+if (start < vas->vm_start) + start = vas->vm_start; +if (end > vas->vm_end) + end = vas->vm_end; + + migrate.start = start; + migrate.end = end; + npages =npages_in_range (start, end); + + buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +
sizeof(*dma_addr)+drivers/gpu/drm/drm_gpusvm.c new file +7
-0+ sizeof(*pages), GFP_KERNEL); +if (!buf) { + err = -ENOMEM ; + goto err_out; + } + dma_addr = buf + (2 *sizeof (*migrate.src) * npages); + pages = buf + (2 *sizeof (*migrate.src) +sizeof (*dma_addr))* npages;
drivers/gpu/drm/drm_gpusvm.c new file +30
-0+ + migrate.vma = vas; + migrate.src = buf; + migrate.dst = migrate.src + npages; + + err =migrate_vma_setup (&migrate); +if (err) + goto err_free; + +/* Raced with another CPU fault, nothing to do */ +if (!migrate.cpages) + goto err_free; + +if (!page) { +for (i =0 ; i < npages; ++i) { +if (!(migrate.src[i] &MIGRATE_PFN_MIGRATE )) +continue ; + + page =migrate_pfn_to_page (migrate.src[i]); +break ; + } + +if (!page) + goto err_finalize; + } + zdd = page->zone_device_data; + ops = zdd->devmem_allocation->ops; + dev = zdd->devmem_allocation->dev; + + err = drm_gpusvm_migrate_populate_ram_pfn(vas, page, npages,&mpages,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ migrate.src,migrate.dst,
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ start); +if (err) + goto err_finalize; + + err = drm_gpusvm_migrate_map_pages(dev, dma_addr,migrate.dst, npages,
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+ DMA_FROM_DEVICE); +if (err) + goto err_finalize; + +for (i =0 ; i < npages; ++i) + pages[i] =migrate_pfn_to_page (migrate.src[i]); + + err = ops->copy_to_ram (pages, dma_addr, npages); +if (err) + goto err_finalize; + +err_finalize: +if (err) + drm_gpusvm_migration_unlock_put_pages(npages,migrate.dst);
drivers/gpu/drm/drm_gpusvm.c new file +17
-0+migrate_vma_pages (&migrate); +migrate_vma_finalize (&migrate); +drm_gpusvm_migrate_unmap_pages (dev, dma_addr, npages, + DMA_FROM_DEVICE); +err_free: +kvfree (buf); +err_out: + +return err; +} + +/** +* drm_gpusvm_range_evict - Evict GPU SVM range +* @pagemap: Pointer to the GPU SVM structure +* @range: Pointer to the GPU SVM range to be removed +* +* This function evicts the specified GPU SVM range. This function will not
+ * evict coherent pages.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+int drm_gpusvm_range_evict (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range) +{ + struct mmu_interval_notifier *notifier = &range->notifier-Quoted 1 line
notifier;
drivers/gpu/drm/drm_gpusvm.c new file +9
-0+struct hmm_range hmm_range = { + .default_flags =HMM_PFN_REQ_FAULT , + .notifier = notifier, + .start = range->itree.start, + .end = range->itree.last +1 , + .dev_private_owner =NULL , + }; +unsigned long timeout = + jiffies +
msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);drivers/gpu/drm/drm_gpusvm.c new file +3
-0+unsigned long *pfns; + unsigned long npages = npages_in_range(range->itree.start, + range->itree.last +1);
drivers/gpu/drm/drm_gpusvm.c new file +13
-0+int err =0 ; +struct mm_struct *mm = gpusvm->mm; + +if (!mmget_not_zero (mm)) +return -EFAULT ; + + pfns =kvmalloc_array (npages,sizeof (*pfns),GFP_KERNEL ); +if (!pfns) +return -ENOMEM ; + + hmm_range.hmm_pfns = pfns; +while (!time_after (jiffies, timeout)) { + hmm_range.notifier_seq =
mmu_interval_read_begin(notifier);drivers/gpu/drm/drm_gpusvm.c new file +21
-0+if (time_after (jiffies, timeout)) { + err = -ETIME ; +break ; + } + +mmap_read_lock (mm); + err =hmm_range_fault (&hmm_range); +mmap_read_unlock (mm); +if (err != -EBUSY ) +break ; + } + +kvfree (pfns); +mmput (mm); + +return err; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_evict); + +/** +* drm_gpusvm_page_free() - Put GPU SVM zone device data associated with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device data
+ * associated with a page when it is being released.
+ */drivers/gpu/drm/drm_gpusvm.c new file +7
-0+static void drm_gpusvm_page_free (struct page *page) +{ +drm_gpusvm_zdd_put (page->zone_device_data); +} + +/** +* drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM (page fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ *VM_FAULT_SIGBUSon failure, 0 on success.
+ */drivers/gpu/drm/drm_gpusvm.c new file +7
-0+static vm_fault_t drm_gpusvm_migrate_to_ram (struct vm_fault *vmf) +{ +struct drm_gpusvm_zdd *zdd = vmf->page->zone_device_data; +int err; + + err = __drm_gpusvm_migrate_to_ram(vmf->vma, + zdd-Quoted 1 line
device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ vmf->page, vmf->address, + zdd->devmem_allocation-Quoted 1 line
size);
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+ +return err ?VM_FAULT_SIGBUS :0 ; +} + +/** +* drm_gpusvm_pagemap_ops() - Device page map operations for GPU SVM +*/ +static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = { + .page_free = drm_gpusvm_page_free, + .migrate_to_ram = drm_gpusvm_migrate_to_ram, +}; + +/** +* drm_gpusvm_pagemap_ops_get() - Retrieve GPU SVM device page map operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +8
-0+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get (void ) +{ +return &drm_gpusvm_pagemap_ops; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_pagemap_ops_get); + +/** +* drm_gpusvm_has_mapping() - Check if GPU SVM has mapping for the given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(structdrm_gpusvm*gpusvm, unsigned long start,drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ unsigned long end) +{ +struct drm_gpusvm_notifier *notifier; + +drm_gpusvm_for_each_notifier (notifier, gpusvm, start, end) { +struct drm_gpusvm_range *range =NULL ; + + drm_gpusvm_for_each_range(range, notifier, start,end)
drivers/gpu/drm/drm_gpusvm.c new file +9
-0+return true; + } + +return false; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_has_mapping); + +/** +* drm_gpusvm_range_set_unmapped() - Mark a GPU SVM range as unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets thepartial_unmapflag
+ * if the range partially falls within the provided MMU notifier range.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+void drm_gpusvm_range_set_unmapped (struct drm_gpusvm_range *range, + const struct mmu_notifier_range
*mmu_range)drivers/gpu/drm/drm_gpusvm.c new file +12
-0+{ +lockdep_assert_held_write (&range->gpusvm->notifier_lock); + + range->flags.unmapped = true; +if (range->itree.start < mmu_range->start || + range->itree.last + 1 > mmu_range->end) + range->flags.partial_unmap = true; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_set_unmapped); + +/** +* drm_gpusvm_devmem_init() - Initialize a GPU SVM device memory allocation
+ *
+ * @dev: Pointer to the device structure which device memory allocation belongs to
+ * @mm: Pointer to themm_structfor the address space
+ * @ops: Pointer to the operations structure for GPU SVM device memory
+ * @dpagemap: The structdrm_pagemapwe're allocating from.
+ * @size: Size of device memory allocation
+ */
+void drm_gpusvm_devmem_init(structdrm_gpusvm_devmem*devmem_allocation,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct device *dev, struct mm_struct*mm,
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ const struct drm_gpusvm_devmem_ops *ops, + struct drm_pagemap *dpagemap, size_tsize)
drivers/gpu/drm/drm_gpusvm.c new file +12
-0+{ +init_completion (&devmem_allocation->detached); + devmem_allocation->dev = dev; + devmem_allocation->mm = mm; + devmem_allocation->ops = ops; + devmem_allocation->dpagemap = dpagemap; + devmem_allocation->size = size; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_devmem_init); + +MODULE_DESCRIPTION ("DRM GPUSVM" ); +MODULE_LICENSE ("GPL" );include/drm/drm_gpusvm.h new file +24
-0@@ -0,0 +1,445 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +/* +* Copyright © 2024 Intel Corporation +*/ + +#ifndef __DRM_GPUSVM_H__ +#define __DRM_GPUSVM_H__ + +#include <linux/kref.h> +#include <linux/interval_tree.h> +#include <linux/mmu_notifier.h> + +struct dev_pagemap_ops ; +struct drm_device ; +struct drm_gpusvm ; +struct drm_gpusvm_notifier ; +struct drm_gpusvm_ops ; +struct drm_gpusvm_range ; +struct drm_gpusvm_devmem ; +struct drm_pagemap ; +struct drm_pagemap_dma_addr ; + +/** +* struct drm_gpusvm_devmem_ops - Operations structure for GPU SVM device memory
+ *
+ * This structure defines the operations for GPU Shared Virtual Memory (SVM)
+ * device memory. These operations are provided by the GPU driver to manage device memory
+ * allocations and perform operations such as migration between device memory and system
+ * RAM.
+ */include/drm/drm_gpusvm.h new file +3
-0+struct drm_gpusvm_devmem_ops { +/** +* @devmem_release: Release device memory allocation (optional)
include/drm/drm_gpusvm.h new file +3
-0+* @devmem_allocation: device memory allocation +* +* Release device memory allocation and drop a reference to device
include/drm/drm_gpusvm.h new file +3
-0+* memory allocation. +*/ + void (*devmem_release)(struct drm_gpusvm_devmem
*devmem_allocation);include/drm/drm_gpusvm.h new file +3
-0+ +/** +* @populate_devmem_pfn: Populate device memory PFN (required for migration)
include/drm/drm_gpusvm.h new file +10
-0+* @devmem_allocation: device memory allocation +* @npages: Number of pages to populate +* @pfn: Array of page frame numbers to populate +* +* Populate device memory page frame numbers (PFN). +* +* Returns: +* 0 on success, a negative error code on failure. +*/ + int (*populate_devmem_pfn)(struct drm_gpusvm_devmem
*devmem_allocation,include/drm/drm_gpusvm.h new file +1
-0+ unsigned long npages, unsignedlong *pfn);
include/drm/drm_gpusvm.h new file +3
-0+ +/** +* @copy_to_devmem: Copy to device memory (required for migration)
include/drm/drm_gpusvm.h new file +1
-0+* @pages: Pointer to array of device memory pages (destination)
include/drm/drm_gpusvm.h new file +16
-0+* @dma_addr: Pointer to array of DMA addresses (source) +* @npages: Number of pages to copy +* +* Copy pages to device memory. +* +* Returns: +* 0 on success, a negative error code on failure. +*/ +int (*copy_to_devmem)(struct page **pages, + dma_addr_t *dma_addr, + unsigned long npages); + +/** +* @copy_to_ram: Copy to system RAM (required for migration) +* @pages: Pointer to array of device memory pages (source) +* @dma_addr: Pointer to array of DMA addresses (destination)
include/drm/drm_gpusvm.h new file +14
-0+* @npages: Number of pages to copy +* +* Copy pages to system RAM. +* +* Returns: +* 0 on success, a negative error code on failure. +*/ +int (*copy_to_ram)(struct page **pages, + dma_addr_t *dma_addr, + unsigned long npages); +}; + +/** +* struct drm_gpusvm_devmem - Structure representing a GPU SVM device memory allocation
+ *
+ * @dev: Pointer to the device structure which device memory allocation belongs to
+ * @mm: Pointer to themm_structfor the address space
+ * @detached: device memory allocations is detached from device pages
+ * @ops: Pointer to the operations structure for GPU SVM device memory
+ * @dpagemap: The structdrm_pagemapof the pages this allocation belongs to.
+ * @size: Size of device memory allocation
+ */include/drm/drm_gpusvm.h new file +13
-0+struct drm_gpusvm_devmem { +struct device *dev; +struct mm_struct *mm; +struct completion detached; +const struct drm_gpusvm_devmem_ops *ops; +struct drm_pagemap *dpagemap; +size_t size; +}; + +/** +* struct drm_gpusvm_ops - Operations structure for GPU SVM +* +* This structure defines the operations for GPU Shared Virtual Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * notifiers.
+ */include/drm/drm_gpusvm.h new file +8
-0+struct drm_gpusvm_ops { +/** +* @notifier_alloc: Allocate a GPU SVM notifier (optional) +* +* Allocate a GPU SVM notifier. +* +* Returns: +* Pointer to the allocated GPU SVM notifier on success, NULL on failure.
include/drm/drm_gpusvm.h new file +19
-0+*/ +struct drm_gpusvm_notifier *(*notifier_alloc)(void ); + +/** +* @notifier_free: Free a GPU SVM notifier (optional) +* @notifier: Pointer to the GPU SVM notifier to be freed +* +* Free a GPU SVM notifier. +*/ +void (*notifier_free)(struct drm_gpusvm_notifier *notifier); + +/** +* @range_alloc: Allocate a GPU SVM range (optional) +* @gpusvm: Pointer to the GPU SVM +* +* Allocate a GPU SVM range. +* +* Returns: +* Pointer to the allocated GPU SVM range on success, NULL on failure.
include/drm/drm_gpusvm.h new file +2
-0+*/ + struct drm_gpusvm_range *(*range_alloc)(struct drm_gpusvm*gpusvm);
include/drm/drm_gpusvm.h new file +16
-0+ +/** +* @range_free: Free a GPU SVM range (optional) +* @range: Pointer to the GPU SVM range to be freed +* +* Free a GPU SVM range. +*/ +void (*range_free)(struct drm_gpusvm_range *range); + +/** +* @invalidate: Invalidate GPU SVM notifier (required) +* @gpusvm: Pointer to the GPU SVM +* @notifier: Pointer to the GPU SVM notifier +* @mmu_range: Pointer to the mmu_notifier_range structure +* +* Invalidate the GPU page tables. It can safely walk the notifier range
include/drm/drm_gpusvm.h new file +1
-0+* RB tree/list in this function. Called while holding the notifier lock.
include/drm/drm_gpusvm.h new file +4
-0+*/ +void (*invalidate)(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + const struct mmu_notifier_range
*mmu_range);include/drm/drm_gpusvm.h new file +4
-0+}; + +/** +* struct drm_gpusvm_notifier - Structure representing a GPU SVM notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @itree: Interval tree node for the notifier (inserted in GPU SVM)
+ * @entry: List entry to fast interval tree traversal
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ * interval tree. This is useful to keep iterating ranges while
+ * doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier has been
+ * removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */include/drm/drm_gpusvm.h new file +19
-0+struct drm_gpusvm_notifier { +struct drm_gpusvm *gpusvm; +struct mmu_interval_notifier notifier; +struct interval_tree_node itree; +struct list_head entry; +struct rb_root_cached root; +struct list_head range_list; +struct { +u32 removed :1 ; + } flags; +}; + +/** +* struct drm_gpusvm_range - Structure representing a GPU SVM range +* +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier +* @refcount: Reference count for the range +* @itree: Interval tree node for the range (inserted in GPU SVM notifier)
+ * @entry: List entry to fast interval tree traversal
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @dma_addr: DMA address array
+ * @dpagemap: The structdrm_pagemapof the device pages we're dma- mapping.
+ * Note this is assuming only onedrm_pagemapper range is allowed.
+ * @flags.migrate_devmem: Flag indicating whether the range can be migrated to device memory
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_devmem_pages: Flag indicating if the range has devmem pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA mapping
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */include/drm/drm_gpusvm.h new file +13
-0+struct drm_gpusvm_range { +struct drm_gpusvm *gpusvm; +struct drm_gpusvm_notifier *notifier; +struct kref refcount; +struct interval_tree_node itree; +struct list_head entry; +unsigned long notifier_seq; +struct drm_pagemap_dma_addr *dma_addr; +struct drm_pagemap *dpagemap; +struct { +/* All flags below must be set upon creation */ +u16 migrate_devmem :1 ; +/* All flags below must be set / cleared under notifier lock */
include/drm/drm_gpusvm.h new file +19
-0+ u16 unmapped : 1; + u16 partial_unmap : 1; + u16 has_devmem_pages : 1; + u16 has_dma_mapping : 1; + } flags; +}; + +/** +* struct drm_gpusvm - GPU SVM structure +* +* @name: Name of the GPU SVM +* @drm: Pointer to the DRM device structure +* @mm: Pointer to the mm_struct for the address space +* @device_private_page_owner: Device private pages owner +* @mm_start: Start address of GPU SVM +* @mm_range: Range of the GPU SVM +* @notifier_size: Size of individual notifiers +* @ops: Pointer to the operations structure for GPU SVM +* @chunk_sizes: Pointer to the array of chunk sizes used in range allocation.
+ * Entries should be powers of 2 in descending order.
+ * @num_chunks: Number of chunks
+ * @notifier_lock: Read-write semaphore for protecting notifier operations
+ * @root: Cached root node of the Red-Black tree containing GPU SVM notifiers
+ * @notifier_list: list head containing of notifiers in the same order they
+ * appear in interval tree. This is useful to keep iterating
+ * notifiers while doing modifications to RB tree.
+ *
+ * This structure represents a GPU SVM (Shared Virtual Memory) used for tracking
+ * memory ranges mapped in a DRM (Direct Rendering Manager) device.
+ *
+ * No reference counting is provided, as this is expected to be embedded in the
+ * driver VM structure along with the structdrm_gpuvm, which handles reference
+ * counting.
+ */include/drm/drm_gpusvm.h new file +17
-0+struct drm_gpusvm { +const char *name; +struct drm_device *drm; +struct mm_struct *mm; +void *device_private_page_owner; +unsigned long mm_start; +unsigned long mm_range; +unsigned long notifier_size; +const struct drm_gpusvm_ops *ops; +const unsigned long *chunk_sizes; +int num_chunks; +struct rw_semaphore notifier_lock; +struct rb_root_cached root; +struct list_head notifier_list; +#ifdef CONFIG_LOCKDEP +/** +* @lock_dep_map: Annotates drm_gpusvm_range_find_or_insert and
include/drm/drm_gpusvm.h new file +10
-0+* drm_gpusvm_range_remove with a driver provided lock. +*/ +struct lockdep_map *lock_dep_map; +#endif +}; + +/** +* struct drm_gpusvm_ctx - DRM GPU SVM context +* +* @check_pages_threshold: Check CPU pages for present if chunk is less than or
+ * equal to threshold. If not present, reduce chunk
+ * size.
+ * @in_notifier: entering from a MMU notifier
+ * @read_only: operating on read-only memory
+ * @devmem_possible: possible to use device memory
+ *
+ * Context that is DRM GPUSVM is operating in (i.e. user arguments).
+ */include/drm/drm_gpusvm.h new file +10
-0+struct drm_gpusvm_ctx { +unsigned long check_pages_threshold; +unsigned int in_notifier :1 ; +unsigned int read_only :1 ; +unsigned int devmem_possible :1 ; +}; + +int drm_gpusvm_init (struct drm_gpusvm *gpusvm, + const char *name, struct drm_device *drm, + struct mm_struct *mm, void
*device_private_page_owner,include/drm/drm_gpusvm.h new file +4
-0+ unsigned long mm_start, unsigned long mm_range, + unsigned long notifier_size, + const struct drm_gpusvm_ops *ops, + const unsigned long *chunk_sizes, int
num_chunks);include/drm/drm_gpusvm.h new file +37
-0+ +void drm_gpusvm_fini (struct drm_gpusvm *gpusvm); + +void drm_gpusvm_free (struct drm_gpusvm *gpusvm); + +struct drm_gpusvm_range * +drm_gpusvm_range_find_or_insert (struct drm_gpusvm *gpusvm, +unsigned long fault_addr, +unsigned long gpuva_start, +unsigned long gpuva_end, +const struct drm_gpusvm_ctx *ctx); + +void drm_gpusvm_range_remove (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range); + +int drm_gpusvm_range_evict (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range); + +struct drm_gpusvm_range * +drm_gpusvm_range_get (struct drm_gpusvm_range *range); + +void drm_gpusvm_range_put (struct drm_gpusvm_range *range); + +bool drm_gpusvm_range_pages_valid (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range); + +int drm_gpusvm_range_get_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx); + +void drm_gpusvm_range_unmap_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx); + +int drm_gpusvm_migrate_to_devmem (struct drm_gpusvm *gpusvm, +struct drm_gpusvm_range *range, + struct drm_gpusvm_devmem
*devmem_allocation,include/drm/drm_gpusvm.h new file +3
-0+ const struct drm_gpusvm_ctx *ctx); + +int drm_gpusvm_evict_to_ram(struct drm_gpusvm_devmem
*devmem_allocation);include/drm/drm_gpusvm.h new file +4
-0+ +const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get (void ); + +bool drm_gpusvm_has_mapping(struct drm_gpusvm *gpusvm, unsigned longstart,
include/drm/drm_gpusvm.h new file +4
-0+ unsigned long end); + +struct drm_gpusvm_range * +drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier, unsignedlong start,
include/drm/drm_gpusvm.h new file +4
-0+ unsigned long end); + +void drm_gpusvm_range_set_unmapped (struct drm_gpusvm_range *range, + const struct mmu_notifier_range
*mmu_range);include/drm/drm_gpusvm.h new file +2
-0+ +void drm_gpusvm_devmem_init(struct drm_gpusvm_devmem
*devmem_allocation,include/drm/drm_gpusvm.h new file +1
-0+ struct device *dev, struct mm_struct*mm,
include/drm/drm_gpusvm.h new file +2
-0+ const struct drm_gpusvm_devmem_ops *ops, + struct drm_pagemap *dpagemap, size_tsize);
include/drm/drm_gpusvm.h new file +4
-0+ +#ifdef CONFIG_LOCKDEP +/** +* drm_gpusvm_driver_set_lock() - Set the lock protecting accesses to GPU SVM
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @lock: the lock used to protect the gpuva list. The locking primitive
+ * must contain adep_mapfield.
+ *
+ * Call this to annotatedrm_gpusvm_range_find_or_insertand
+ *drm_gpusvm_range_remove.
+ */include/drm/drm_gpusvm.h new file +4
-0+#define drm_gpusvm_driver_set_lock (gpusvm, lock) \ + do { \ + if (!WARN((gpusvm)->lock_dep_map, \ + "GPUSVM range lock should be set onlyonce."))\
include/drm/drm_gpusvm.h new file +1
-0+ (gpusvm)->lock_dep_map = &(lock)-Quoted 1 line
dep_map; \
Quoted 9 lines
include/drm/drm_gpusvm.h line 387
Quoted 6 lines
include/drm/drm_gpusvm.h new file +6
-0+ } while (0) +#define drm_gpusvm_driver_lock_held (gpusvm) \ + do { \ + if ((gpusvm)->lock_dep_map) \ + lock_is_held((gpusvm)->lock_dep_map); \ + } while (0)Could we use static functions for those above
Static should work. Will change.
Quoted 3 lines
Also I don't think the
drm_gpusvm_driver_lock_held()does what it's intended to do? There's an assert missing.
'lock_is_held' is an assert, right? I based this code existing code drm_gem_gpuva_assert_lock_held which uses 'lock_is_held'.
This probably should be changed to 'lock_is_held_type(..., 0)' to indicate lock is held in exclusive mode though.
Quoted 64 lines
Quoted 64 lines
include/drm/drm_gpusvm.h new file +25
-0+#else +#define drm_gpusvm_driver_set_lock (gpusvm, lock) do {} while (0) +#define drm_gpusvm_driver_lock_held (gpusvm) do {} while (0) +#endif + +/** +* drm_gpusvm_notifier_lock() - Lock GPU SVM notifier +* @gpusvm__: Pointer to the GPU SVM structure. +* +* Abstract client usage GPU SVM notifier lock, take lock +*/ +#define drm_gpusvm_notifier_lock (gpusvm__) \ + down_read(&(gpusvm__)->notifier_lock) + +/** +* drm_gpusvm_notifier_unlock() - Unlock GPU SVM notifier +* @gpusvm__: Pointer to the GPU SVM structure. +* +* Abstract client usage GPU SVM notifier lock, drop lock +*/ +#define drm_gpusvm_notifier_unlock (gpusvm__) \ + up_read(&(gpusvm__)->notifier_lock) + +/** +* __drm_gpusvm_range_next() - Get the next GPU SVM range in the list
+ * @range: a pointer to the current GPU SVM range
+ *
+ * Return: A pointer to the nextdrm_gpusvm_rangeif available, or NULL if the
+ * current range is the last one or if the input range is NULL.
+ */include/drm/drm_gpusvm.h new file +12
-0+static inline struct drm_gpusvm_range * +__drm_gpusvm_range_next (struct drm_gpusvm_range *range) +{ +if (range && !list_is_last (&range->entry, + &range->notifier->range_list)) +return list_next_entry (range, entry); + +return NULL ; +} + +/** +* drm_gpusvm_for_each_range() - Iterate over GPU SVM ranges in a notifier
+ * @range__: Iterator variable for the ranges. If set, it indicates the start of
+ * the iterator. If NULL, calldrm_gpusvm_range_find()to
get the range.
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier. It is safe
+ * to use while holding the driver SVM lock or the notifier lock.
+ */
+#define drm_gpusvm_for_each_range(range__,notifier__,start__,end__) \include/drm/drm_gpusvm.h new file +1
-0+ for ((range__) = (range__)?: \
include/drm/drm_gpusvm.h new file +1
-0+ drm_gpusvm_range_find((notifier__), (start__),
Quoted 9 lines
include/drm/drm_gpusvm.h line 445
Quoted 6 lines
(
end__)); \include/drm/drm_gpusvm.h new file +1
-0+ (range__) && (range__->itree.start <(
end__)); \include/drm/drm_gpusvm.h new file +3
-0+ (range__) = __drm_gpusvm_range_next(range__)) + +#endif /* __DRM_GPUSVM_H__ */ Otherwise LGTM.
Thanks,
Matt
Quoted 4 lines
/Thomas
Thomas Hellström
maintainerRe: [PATCH v4 06/33] drm/gpusvm: Add support for GPU Shared Virtual Memory
Quoted 3,535 lines · Matthew Brost
Quoted 3,528 lines · Thomas Hellström
Quoted 3,527 lines · Matthew Brost
This patch introduces support for GPU Shared Virtual Memory (SVM) in
the
Direct Rendering Manager (DRM) subsystem. SVM allows for seamless sharing of memory between the CPU and GPU, enhancing performance and
flexibility in GPU computing tasks.The patch adds the necessary infrastructure for SVM, including data
structures and functions for managing SVM ranges and notifiers. It
also
provides mechanisms for allocating, deallocating, and migrating memory
regions between system RAM and GPU VRAM.This is largely inspired by GPUVM.
v2:
- Take order into account in check pages
- Clearrange->pagesin get pages error
- Drop setting dirty or accessed bit in get pages (Vetter)
- Remove mmap assert for cpu faults
- Drop mmap write lock abuse (Vetter, Christian)
- Decouple zdd from range (Vetter, Oak)
- Adddrm_gpusvm_range_evict, make it work with coherent pages
- Exportdrm_gpusvm_evict_to_sram, only use in BO evict path (Vetter)
- mmget/put indrm_gpusvm_evict_to_sram
- Droprange->vram_alloationvariable
- Don't return indrm_gpusvm_evict_to_sramuntil all pages detached
- Don't warn on mixing sram and device pages
- Update kernel doc
- Add coherent page support to get pages
- UseDMA_FROM_DEVICErather thanDMA_BIDIRECTIONAL
- Add structdrm_gpusvm_vramand ops (Thomas)
- Update the range's seqno if the range is valid (Thomas)
- Remove theis_unmappedcheck beforehmm_range_fault(Thomas)
- Usedrm_pagemap(Thomas)
- Dropkfree_mapping(Thomas)
- dma mapp pages under notifier lock (Thomas)
- Remove ctx.prefault
- Remove ctx.mmap_locked
- Add ctx.check_pages
- s/vram/devmem (Thomas)
v3:
- Fix memory leakdrm_gpusvm_range_get_pages
- Only migrate pages with same zdd on CPU fault
- Loop over al VMAs indrm_gpusvm_range_evict
- Make GPUSVM a drm level module
- GPL or MIT license
- Update main kernel doc (Thomas)
- Preferfoo()vs foo for functions in kernel doc (Thomas)
- Prefer functions over macros (Thomas)
- Use unsigned long vs u64 for addresses (Thomas)
- Use standardinterval_tree(Thomas)
- s/drm_gpusvm_migration_put_page/drm_gpusvm_migration_unlock_put_p age
(Thomas)
- Droperr_outlabel indrm_gpusvm_range_find_or_insert(Thomas)
- Fix kernel doc indrm_gpusvm_range_free_pages(Thomas)
- Newlines between functions defs in header file (Thomas)
- Drop shall language in driver vfunc kernel doc (Thomas)
- Move some static inlines from head to C file (Thomas)
- Don't allocate pages under page lock indrm_gpusvm_migrate_populate_ram_pfn(Thomas)
- Changecheck_pagesto a threshold
v4:
- Fix NULL ptr deref indrm_gpusvm_migrate_populate_ram_pfn(Thomas,
Himal)
- Fix check pages threshold
- Check for range being unmapped under notifier lock in get pages
(Testing)
- Fix characters per line
- DropWRITE_ONCEforzdd->devmem_allocationassignment (Thomas)
- Use completion fordevmem_allocation->detached(Thomas)
- Make GPU SVM depend onZONE_DEVICE(CI)
- Usehmm_range_faultfor eviction (Thomas)
- Drop zdd worker (Thomas)Cc Simona Vetter <[email protected]>Cc Dave Airlie <[email protected]>Cc Christian König <[email protected]>Cc <[email protected]>Signed-off-by Matthew Brost <[email protected]>Signed-off-by Thomas Hellström<[email protected]>
---
drivers/gpu/drm/Kconfig | 9 +
drivers/gpu/drm/Makefile | 1 +
drivers/gpu/drm/drm_gpusvm.c | 2240
++++++++++++++++++++++++++++++++++
include/drm/drm_gpusvm.h | 445 +++++++4 files changed, 2695 insertions(+)create mode 100644
drivers/gpu/drm/drm_gpusvm.c
create mode 100644include/drm/drm_gpusvm.hdrivers/gpu/drm/Kconfig +0
-0@@ -278,6 +278,15 @@ config DRM_GPUVMGPU-VM representation providing helpers to manage a GPUs
virtual
address spacedrivers/gpu/drm/Kconfig +7
-0+config DRM_GPUSVM + tristate + depends on DRM + depends on DEVICE_MIGRATION + depends on ZONE_DEVICE + help + GPU-SVM representation providing helpers to manage aGPUs
shareddrivers/gpu/drm/Kconfig +2
-0+ virtual memory +config
DRM_BUDDY
tristate
depends on DRMdrivers/gpu/drm/Makefile +0
-0@@ -104,6 +104,7 @@ obj-$(CONFIG_DRM_PANEL_BACKLIGHT_QUIRKS) +=drm_panel_backlight_quirks.o
#
obj-$(CONFIG_DRM_EXEC) += drm_exec.o
obj-$(CONFIG_DRM_GPUVM) += drm_gpuvm.o
+obj-$(CONFIG_DRM_GPUSVM) += drm_gpusvm.o
obj-$(CONFIG_DRM_BUDDY) += drm_buddy.odrivers/gpu/drm/drm_gpusvm.c new file +25
-0@@ -0,0 +1,2240 @@ +// SPDX-License-Identifier: GPL-2.0-only OR MIT +/* +* Copyright © 2024 Intel Corporation +* +* Authors: +* Matthew Brost <[email protected]> +*/ + +#include <linux/dma-mapping.h> +#include <linux/hmm.h> +#include <linux/memremap.h> +#include <linux/migrate.h> +#include <linux/mm_types.h> +#include <linux/pagemap.h> +#include <linux/slab.h> + +#include <drm/drm_device.h> +#include <drm/drm_gpusvm.h> +#include <drm/drm_pagemap.h> +#include <drm/drm_print.h> + +/** +* DOC: Overview +* +* GPU Shared Virtual Memory (GPU SVM) layer for the Direct Rendering Manager (DRM)
+ *
+ * The GPU SVM layer is a component of the DRM framework designed to
manage shared
+ * virtual memory between the CPU and GPU. It enables efficient data
exchange and
+ * processing for GPU-accelerated applications by allowing memory
sharing and
+ * synchronization between the CPU's and GPU's virtual address spaces.
+ *
+ * Key GPU SVM Components:
+ * - Notifiers: Notifiers: Used for tracking memory intervals and
notifying the
+ * GPU of changes, notifiers are sized based on a
GPU
SVM
+ * initialization parameter, with a recommendation
of
512M or
+ * larger. They maintain a Red-BlacK tree and a
list of
ranges that
+ * fall within the notifier interval. Notifiers are
tracked within
+ * a GPU SVM Red-BlacK tree and list and are
dynamically inserted
+ * or removed as ranges within the interval are
created
or
+ * destroyed.
+ * - Ranges: Represent memory ranges mapped in a DRM device and managed
+ * by GPU SVM. They are sized based on an array of
chunk
sizes, which
+ * is a GPU SVM initialization parameter, and the CPU
address space.
+ * Upon GPU fault, the largest aligned chunk that fits
within the
+ * faulting CPU address space is chosen for the range
size. Ranges are
+ * expected to be dynamically allocated on GPU fault
and
removed on an
+ * MMU notifier UNMAP event. As mentioned above,
ranges
are tracked in
+ * a notifier's Red-Black tree.
+ * - Operations: Define the interface for driver-specific GPU SVM
operations
+ * such as range allocation, notifier allocation, and
+ * invalidations.
+ * - Device Memory Allocations: Embedded structure containing enough
information
+ * for GPU SVM to migrate to / from device memory.
+ * - Device Memory Operations: Define the interface for driver- specific device
+ * memory operations release memory, populate pfns,
+ * and copy to / from device memory.
+ *
+ * This layer provides interfaces for allocating, mapping, migrating, and
+ * releasing memory ranges between the CPU and GPU. It handles all
core memory
+ * management interactions (DMA mapping, HMM, and migration) and provides
+ * driver-specific virtual functions (vfuncs). This infrastructure
is sufficient
+ * to build the expected driver components for an SVM implementation
as detailed
+ * below.
+ *
+ * Expected Driver Components:
+ * - GPU page fault handler: Used to create ranges and notifiers based on the
+ * fault address, optionally migrate
the
range to
+ * device memory, and create GPU
bindings.
+ * - Garbage collector: Used to unmap and destroy GPU bindings for
ranges.
+ * Ranges are expected to be added to the
garbage collector
+ * upon aMMU_NOTIFY_UNMAPevent in
notifier
callback.
+ * - Notifier callback: Used to invalidate and DMA unmap GPU bindings for
+ * ranges.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +/** +* DOC: Locking +* +* GPU SVM handles locking for core MM interactions, i.e., it locks/unlocks the
+ * mmap lock as needed.
+ *
+ * GPU SVM introduces a global notifier lock, which safeguards the
notifier's
+ * range RB tree and list, as well as the range's DMA mappings and
sequence
+ * number. GPU SVM manages all necessary locking and unlocking operations,
+ * except for the recheck range's pages being valid
+ * (drm_gpusvm_range_pages_valid) when the driver is committing GPU
bindings. This
+ * lock corresponds to the 'driver->update' lock mentioned in the
HMM
+ * documentation (TODO: Link). Future revisions may transition from
a GPU SVM
+ * global lock to a per-notifier lock if finer-grained locking is
deemed
+ * necessary.
+ *
+ * In addition to the locking mentioned above, the driver should implement a
+ * lock to safeguard core GPU SVM function calls that modify state,
such as
+ *drm_gpusvm_range_find_or_insertanddrm_gpusvm_range_remove. This
lock is
+ * denoted as 'driver_svm_lock' in code examples. Finer grained driver side
+ * locking should also be possible for concurrent GPU fault processing within a
+ * single GPU SVM. The 'driver_svm_lock' can be viadrm_gpusvm_driver_set_lock
+ * to add annotations to GPU SVM.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +/** +* DOC: Migration +* +* The migration support is quite simple, allowing migration between
RAM and
+ * device memory at the range granularity. For example, GPU SVM currently does not
+ * support mixing RAM and device memory pages within a range. This
means that upon GPU
+ * fault, the entire range can be migrated to device memory, and upon CPU fault, the
+ * entire range is migrated to RAM. Mixed RAM and device memory storage within a range
+ * could be added in the future if required.
+ *
+ * The reasoning for only supporting range granularity is as
follows: it
+ * simplifies the implementation, and range sizes are driver- defined
and should
+ * be relatively small.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +/** +* DOC: Partial Unmapping of Ranges +* +* Partial unmapping of ranges (e.g., 1M out of 2M is unmapped by
CPU resulting
+ * inMMU_NOTIFY_UNMAPevent) presents several challenges, with the
main one
+ * being that a subset of the range still has CPU and GPU mappings.
If the
+ * backing store for the range is in device memory, a subset of the
backing store has
+ * references. One option would be to split the range and device memory backing store,
+ * but the implementation for this would be quite complicated. Given
that
+ * partial unmappings are rare and driver-defined range sizes are
relatively
+ * small, GPU SVM does not support splitting of ranges.
+ *
+ * With no support for range splitting, upon partial unmapping of a
range, the
+ * driver is expected to invalidate and destroy the entire range. If
the range
+ * has device memory as its backing, the driver is also expected to
migrate any
+ * remaining pages back to RAM.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ +/** +* DOC: Examples +* +* This section provides three examples of how to build the expected
driver
+ * components: the GPU page fault handler, the garbage collector,
and the
+ * notifier callback.
+ *
+ * The generic code provided does not include logic for complex migration
+ * policies, optimized invalidations, fined grained driver locking,
or other
+ * potentially required driver locking (e.g., DMA-resv locks).
+ *
+ * 1) GPU page fault handler
+ *
+ * int driver_bind_range(structdrm_gpusvm*gpusvm, structdrm_gpusvm_range*range)
+ * {
+ * int err = 0;
+ *
+ * driver_alloc_and_setup_memory_for_bind(gpusvm,
range);
+ *
+ *drm_gpusvm_notifier_lock(gpusvm);
+ * if (drm_gpusvm_range_pages_valid(range))
+ *driver_commit_bind(gpusvm, range);
+ * else
+ * err =-EAGAIN;
+ *drm_gpusvm_notifier_unlock(gpusvm);
+ *
+ * return err;
+ * }
+ *
+ * int driver_gpu_fault(structdrm_gpusvm*gpusvm, unsigned
longfault_addr,
+ * unsigned longgpuva_start, unsigned
longgpuva_end)
+ * {
+ * structdrm_gpusvm_ctxctx = {};
+ * int err;
+ *
+ *driver_svm_lock();
+ * retry:
+ * // Always process UNMAPs first so view of GPU
SVM
ranges is current
+ *driver_garbage_collector(gpusvm);
+ *
+ * range = drm_gpusvm_range_find_or_insert(gpusvm,fault_addr,
+
*gpuva_start,gpuva_end,
+ * &ctx);
+ * if (IS_ERR(range)) {
+ * err =PTR_ERR(range);
+ * goto unlock;
+ * }
+ *
+ * if (driver_migration_policy(range)) {
+ * devmem =driver_alloc_devmem();
+ * err =
drm_gpusvm_migrate_to_devmem(gpusvm,
range,
+ *devmem_allocation,
+ *
&ctx);
+ * if (err) // CPU mappings may have
changed
+ * goto retry;
+ * }
+ *
+ * err = drm_gpusvm_range_get_pages(gpusvm, range,
&ctx);
+ * if (err ==-EOPNOTSUPP|| err ==-EFAULT|| err
== -
EPERM) { // CPU mappings changed
+ * if (err ==-EOPNOTSUPP)
+ * drm_gpusvm_range_evict(gpusvm,
range);
+ * goto retry;
+ * } else if (err) {
+ * goto unlock;
+ * }
+ *
+ * err =driver_bind_range(gpusvm, range);
+ * if (err ==-EAGAIN) // CPU mappings changed
+ * goto retry
+ *
+ * unlock:
+ *driver_svm_unlock();
+ * return err;
+ * }
+ *
+ * 2) Garbage Collector.
+ *
+ * void __driver_garbage_collector(structdrm_gpusvm
*gpusvm,
+ * structdrm_gpusvm_range
*range)
+ * {
+ *assert_driver_svm_locked(gpusvm);
+ *
+ * // Partial unmap, migrate any remaining device
memory pages back to RAM
+ * if (range->flags.partial_unmap)
+ *drm_gpusvm_range_evict(gpusvm, range);
+ *
+ *driver_unbind_range(range);
+ *drm_gpusvm_range_remove(gpusvm, range);
+ * }
+ *
+ * voiddriver_garbage_collector(struct drm_gpusvm *gpusvm)
+ * {
+ *assert_driver_svm_locked(gpusvm);
+ *
+ * for_each_range_in_garbage_collector(gpusvm,
range)
+ * __driver_garbage_collector(gpusvm,
range);
+ * }
+ *
+ * 3) Notifier callback.
+ *
+ * void driver_invalidation(structdrm_gpusvm*gpusvm,
+ * structdrm_gpusvm_notifier
*notifier,
+ * const structmmu_notifier_range*mmu_range)
+ * {
+ * structdrm_gpusvm_ctxctx = { .in_notifier =
true,
};
+ * structdrm_gpusvm_range*range = NULL;
+ *
+ * driver_invalidate_device_pages(gpusvm,
mmu_range-Quoted 1 line
start,
mmu_range->end);+ *
+ * drm_gpusvm_for_each_range(range, notifier,mmu_range->start,
+ *mmu_range->end) {
+ * drm_gpusvm_range_unmap_pages(gpusvm,
range,
&ctx);
+ *
+ * if (mmu_range->event!=MMU_NOTIFY_UNMAP)
+ * continue;
+ *
+ * drm_gpusvm_range_set_unmapped(range,mmu_range);
+ * driver_garbage_collector_add(gpusvm,
range);
+ * }
+ * }
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ +/** +* npages_in_range() - Calculate the number of pages in a given range
+ * @start: The start address of the range
+ * @end: The end address of the range
+ *
+ * This macro calculates the number of pages in a given memory range,
+ * specified by the start and end addresses. It divides the difference
+ * between the end and start addresses by the page size (PAGE_SIZE)
to
+ * determine the number of pages in the range.
+ *
+ * Returns: The number of pages in the specified range.
+ */drivers/gpu/drm/drm_gpusvm.c new file +15
-0+static unsigned long +npages_in_range (unsigned long start,unsigned long end) +{ +return (end - start) >>PAGE_SHIFT ; +} + +/** +* struct drm_gpusvm_zdd - GPU SVM zone device data +* +* @refcount: Reference count for the zdd +* @devmem_allocation: device memory allocation +* @device_private_page_owner: Device private pages owner +* +* This structure serves as a generic wrapper installed in +* page->zone_device_data. It provides infrastructure for looking up
a device
+ * memory allocation upon CPU page fault and asynchronously releasing device
+ * memory once the CPU has no page references. Asynchronous release
is useful
+ * because CPU page references can be dropped in IRQ contexts, while
releasing
+ * device memory likely requires sleeping locks.
+ */drivers/gpu/drm/drm_gpusvm.c new file +11
-0+struct drm_gpusvm_zdd { +struct kref refcount; +struct drm_gpusvm_devmem *devmem_allocation; +void *device_private_page_owner; +}; + +/** +* drm_gpusvm_zdd_alloc() - Allocate a zdd structure. +* @device_private_page_owner: Device private pages owner +* +* This function allocates and initializes a new zdd structure. It
sets up the
+ * reference count and initializes the destroy work.
+ *
+ * Returns:
+ * Pointer to the allocated zdd on success,ERR_PTR()on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +12
-0+static struct drm_gpusvm_zdd * +drm_gpusvm_zdd_alloc (void *device_private_page_owner) +{ +struct drm_gpusvm_zdd *zdd; + + zdd =kmalloc (sizeof (*zdd),GFP_KERNEL ); +if (!zdd) +return NULL ; + +kref_init (&zdd->refcount); + zdd->devmem_allocation =NULL ; + zdd->device_private_page_owner =
device_private_page_owner;drivers/gpu/drm/drm_gpusvm.c new file +9
-0+ +return zdd; +} + +/** +* drm_gpusvm_zdd_get() - Get a reference to a zdd structure. +* @zdd: Pointer to the zdd structure. +* +* This function increments the reference count of the provided zdd
structure.
+ *
+ * Returns: Pointer to the zdd structure.
+ */
+static structdrm_gpusvm_zdd*drm_gpusvm_zdd_get(structdrm_gpusvm_zdd*zdd)drivers/gpu/drm/drm_gpusvm.c new file +10
-0+{ +kref_get (&zdd->refcount); +return zdd; +} + +/** +* drm_gpusvm_zdd_destroy() - Destroy a zdd structure. +* @ref: Pointer to the reference count structure. +* +* This function queues the destroy_work of the zdd for asynchronous
destruction.
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+static void drm_gpusvm_zdd_destroy (struct kref *ref) +{ +struct drm_gpusvm_zdd *zdd = + container_of(ref, struct drm_gpusvm_zdd,refcount);
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct drm_gpusvm_devmem *devmem = zdd-Quoted 1 line
devmem_allocation;drivers/gpu/drm/drm_gpusvm.c new file +14
-0+ +if (devmem) { +complete_all (&devmem->detached); +if (devmem->ops->devmem_release) + devmem->ops->devmem_release (devmem); + } +kfree (zdd); +} + +/** +* drm_gpusvm_zdd_put() - Put a zdd reference. +* @zdd: Pointer to the zdd structure. +* +* This function decrements the reference count of the provided zdd
structure
+ * and schedules its destruction if the count drops to zero.
+ */drivers/gpu/drm/drm_gpusvm.c new file +7
-0+static void drm_gpusvm_zdd_put (struct drm_gpusvm_zdd *zdd) +{ +kref_put (&zdd->refcount, drm_gpusvm_zdd_destroy); +} + +/** +* drm_gpusvm_range_find() - Find GPU SVM range from GPU SVM notifier
+ * @notifier: Pointer to the GPU SVM notifier structure.
+ * @start: Start address of the range
+ * @end: End address of the range
+ *
+ * Returns: A pointer to thedrm_gpusvm_rangeif found or NULL
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+struct drm_gpusvm_range * +drm_gpusvm_range_find(struct drm_gpusvm_notifier *notifier,unsigned
long start,drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ unsigned long end) +{ +struct interval_tree_node *itree; + + itree = interval_tree_iter_first(¬ifier->root, start,end
- 1);drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ +if (itree) + return container_of(itree, struct
drm_gpusvm_range,
itree);drivers/gpu/drm/drm_gpusvm.c new file +7
-0+else +return NULL ; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_find); + +/** +* drm_gpusvm_for_each_range_safe() - Safely iterate over GPU SVM
ranges in a notifier
+ * @range__: Iterator variable for the ranges
+ * @next__: Iterator variable for the ranges temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the range
+ * @end__: End address of the range
+ *
+ * This macro is used to iterate over GPU SVM ranges in a notifier
while
+ * removing ranges from it.
+ */
+#define drm_gpusvm_for_each_range_safe(range__,next__,notifier__,start__,end__) \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ for ((range__) = drm_gpusvm_range_find((notifier__),(
start__), (end__)), \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (next__) =
__drm_gpusvm_range_next(range__); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (range__) && (range__->itree.start <(
end__)); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (range__) = (next__), (next__) =
__drm_gpusvm_range_next(range__))drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ +/** +* __drm_gpusvm_notifier_next() - get the next
drm_gpusvm_notifier
in the list
+ * @notifier: a pointer to the currentdrm_gpusvm_notifier
+ *
+ * Returns: A pointer to the nextdrm_gpusvm_notifierif available,
or NULL if
+ * the current notifier is the last one or if the input notifier is
+ * NULL.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+static struct drm_gpusvm_notifier * +__drm_gpusvm_notifier_next(struct drm_gpusvm_notifier *notifier) +{ + if (notifier && !list_is_last(¬ifier->entry, + ¬ifier->gpusvm-Quoted 1 line
notifier_list))drivers/gpu/drm/drm_gpusvm.c new file +7
-0+return list_next_entry (notifier, entry); + +return NULL ; +} + +static struct drm_gpusvm_notifier * +notifier_iter_first(struct rb_root_cached *root, unsigned longstart,
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ unsigned long last) +{ +struct interval_tree_node *itree; + + itree =interval_tree_iter_first (root, start, last); + +if (itree) + return container_of(itree, struct
drm_gpusvm_notifier, itree);drivers/gpu/drm/drm_gpusvm.c new file +6
-0+else +return NULL ; +} + +/** +* drm_gpusvm_for_each_notifier() - Iterate over GPU SVM notifiers
in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm.
+ */
+#define drm_gpusvm_for_each_notifier(notifier__,gpusvm__,start__,end__) \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ for ((notifier__) = notifier_iter_first(&(gpusvm__)-Quoted 1 line
root,
(
start__), (end__) - 1); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (notifier__) && (notifier__->itree.start <(
end__)); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (notifier__) =
__drm_gpusvm_notifier_next(notifier__))drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ +/** +* drm_gpusvm_for_each_notifier_safe() - Safely iterate over GPU SVM
notifiers in a gpusvm
+ * @notifier__: Iterator variable for the notifiers
+ * @next__: Iterator variable for the notifiers temporay storage
+ * @notifier__: Pointer to the GPU SVM notifier
+ * @start__: Start address of the notifier
+ * @end__: End address of the notifier
+ *
+ * This macro is used to iterate over GPU SVM notifiers in a gpusvm
while
+ * removing notifiers from it.
+ */
+#define drm_gpusvm_for_each_notifier_safe(notifier__,next__,gpusvm__,start__,end__) \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ for ((notifier__) = notifier_iter_first(&(gpusvm__)-Quoted 1 line
root,
(
start__), (end__) - 1), \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (next__) =
__drm_gpusvm_notifier_next(notifier__); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (notifier__) && (notifier__->itree.start <(
end__)); \drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (notifier__) = (next__), (next__) =
__drm_gpusvm_notifier_next(notifier__))drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ +/** +* drm_gpusvm_notifier_invalidate() - Invalidate a GPU SVM notifier.
+ * @mni: Pointer to themmu_interval_notifierstructure.
+ * @mmu_range: Pointer to themmu_notifier_rangestructure.
+ * @cur_seq: Current sequence number.
+ *
+ * This function serves as a generic MMU notifier for GPU SVM. It
sets the MMU
+ * notifier sequence number and calls the driver invalidate vfunc
under
+ *gpusvm->notifier_lock.
+ *
+ * Returns:
+ * true if the operation succeeds, false otherwise.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static bool +drm_gpusvm_notifier_invalidate(struct mmu_interval_notifier*mni,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ const struct mmu_notifier_range
*mmu_range,drivers/gpu/drm/drm_gpusvm.c new file +19
-0+ unsigned long cur_seq) +{ +struct drm_gpusvm_notifier *notifier = +container_of (mni,typeof (*notifier), notifier); +struct drm_gpusvm *gpusvm = notifier->gpusvm; + +if (!mmu_notifier_range_blockable (mmu_range)) +return false; + +down_write (&gpusvm->notifier_lock); +mmu_interval_set_seq (mni, cur_seq); + gpusvm->ops->invalidate (gpusvm, notifier, mmu_range); +up_write (&gpusvm->notifier_lock); + +return true; +} + +/** +* drm_gpusvm_notifier_ops - MMU interval notifier operations for
GPU SVM
+ */
+static const structmmu_interval_notifier_opsdrm_gpusvm_notifier_ops= {drivers/gpu/drm/drm_gpusvm.c new file +15
-0+ .invalidate = drm_gpusvm_notifier_invalidate, +}; + +/** +* drm_gpusvm_init() - Initialize the GPU SVM. +* @gpusvm: Pointer to the GPU SVM structure. +* @name: Name of the GPU SVM. +* @drm: Pointer to the DRM device structure. +* @mm: Pointer to the mm_struct for the address space. +* @device_private_page_owner: Device private pages owner. +* @mm_start: Start address of GPU SVM. +* @mm_range: Range of the GPU SVM. +* @notifier_size: Size of individual notifiers. +* @ops: Pointer to the operations structure for GPU SVM. +* @chunk_sizes: Pointer to the array of chunk sizes used in range
allocation.
+ * Entries should be powers of 2 in descending order
with last
+ * entry beingSZ_4K.
+ * @num_chunks: Number of chunks.
+ *
+ * This function initializes the GPU SVM.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+int drm_gpusvm_init (struct drm_gpusvm *gpusvm, + const char *name, struct drm_device *drm, + struct mm_struct *mm, void
*device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long mm_start, unsigned long
mm_range,drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ unsigned long notifier_size, + const struct drm_gpusvm_ops *ops, + const unsigned long *chunk_sizes, int
num_chunks)drivers/gpu/drm/drm_gpusvm.c new file +8
-0+{ +if (!ops->invalidate || !num_chunks) +return -EINVAL ; + + gpusvm->name = name; + gpusvm->drm = drm; + gpusvm->mm = mm; + gpusvm->device_private_page_owner =
device_private_page_owner;drivers/gpu/drm/drm_gpusvm.c new file +31
-0+ gpusvm->mm_start = mm_start; + gpusvm->mm_range = mm_range; + gpusvm->notifier_size = notifier_size; + gpusvm->ops = ops; + gpusvm->chunk_sizes = chunk_sizes; + gpusvm->num_chunks = num_chunks; + +mmgrab (mm); + gpusvm->root =RB_ROOT_CACHED ; +INIT_LIST_HEAD (&gpusvm->notifier_list); + +init_rwsem (&gpusvm->notifier_lock); + +fs_reclaim_acquire (GFP_KERNEL ); +might_lock (&gpusvm->notifier_lock); +fs_reclaim_release (GFP_KERNEL ); + +#ifdef CONFIG_LOCKDEP + gpusvm->lock_dep_map =NULL ; +#endif + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_init); + +/** +* drm_gpusvm_notifier_find() - Find GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM structure +* @fault_addr: Fault address +* +* This function finds the GPU SVM notifier associated with the fault address.
+ *
+ * Returns:
+ * Pointer to the GPU SVM notifier on success, NULL otherwise.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+static struct drm_gpusvm_notifier * +drm_gpusvm_notifier_find (struct drm_gpusvm *gpusvm, +unsigned long fault_addr) +{ + return notifier_iter_first(&gpusvm->root, fault_addr,
fault_addr+ 1);drivers/gpu/drm/drm_gpusvm.c new file +4
-0+} + +/** +* to_drm_gpusvm_notifier() - retrieve the container struct for a
given rbtree node
+ * @node: a pointer to the rbtree node embedded within adrm_gpusvm_notifierstruct
+ *
+ * Returns: A pointer to the containingdrm_gpusvm_notifierstructure.
+ */
+static structdrm_gpusvm_notifier*to_drm_gpusvm_notifier(structrb_node*node)drivers/gpu/drm/drm_gpusvm.c new file +2
-0+{ + return container_of(node, struct drm_gpusvm_notifier,itree.rb);
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+} + +/** +* drm_gpusvm_notifier_insert() - Insert GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier structure +* +* This function inserts the GPU SVM notifier into the GPU SVM RB
tree and list.
+ */
+static void drm_gpusvm_notifier_insert(structdrm_gpusvm*gpusvm,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct
drm_gpusvm_notifier
*notifier)drivers/gpu/drm/drm_gpusvm.c new file +21
-0+{ +struct rb_node *node; +struct list_head *head; + +interval_tree_insert (¬ifier->itree, &gpusvm->root); + + node =rb_prev (¬ifier->itree.rb); +if (node) + head = &(to_drm_gpusvm_notifier (node))->entry; +else + head = &gpusvm->notifier_list; + +list_add (¬ifier->entry, head); +} + +/** +* drm_gpusvm_notifier_remove() - Remove GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM tructure +* @notifier: Pointer to the GPU SVM notifier structure +* +* This function removes the GPU SVM notifier from the GPU SVM RB
tree and list.
+ */
+static void drm_gpusvm_notifier_remove(structdrm_gpusvm*gpusvm,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct
drm_gpusvm_notifier
*notifier)drivers/gpu/drm/drm_gpusvm.c new file +10
-0+{ +interval_tree_remove (¬ifier->itree, &gpusvm->root); +list_del (¬ifier->entry); +} + +/** +* drm_gpusvm_fini() - Finalize the GPU SVM. +* @gpusvm: Pointer to the GPU SVM structure. +* +* This function finalizes the GPU SVM by cleaning up any remaining
ranges and
+ * notifiers, and dropping a reference to struct MM.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+void drm_gpusvm_fini (struct drm_gpusvm *gpusvm) +{ +struct drm_gpusvm_notifier *notifier, *next; + + drm_gpusvm_for_each_notifier_safe(notifier, next,gpusvm, 0,
LONG_MAX) {drivers/gpu/drm/drm_gpusvm.c new file +4
-0+struct drm_gpusvm_range *range, *__next; + +/* +* Remove notifier first to avoid racing with any
invalidationdrivers/gpu/drm/drm_gpusvm.c new file +2
-0+*/ + mmu_interval_notifier_remove(¬ifier-Quoted 1 line
notifier);
drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ notifier->flags.removed = true; + + drm_gpusvm_for_each_range_safe(range, __next,notifier, 0,
drivers/gpu/drm/drm_gpusvm.c new file +15
-0+ LONG_MAX) +drm_gpusvm_range_remove (gpusvm ,range ); + } + +mmdrop (gpusvm->mm); +WARN_ON (!RB_EMPTY_ROOT (&gpusvm->root.rb_root)); +} +EXPORT_SYMBOL_GPL (drm_gpusvm_fini); + +/** +* drm_gpusvm_notifier_alloc() - Allocate GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM structure +* @fault_addr: Fault address +* +* This function allocates and initializes the GPU SVM notifier structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM notifier on success,ERR_PTR()
on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static struct drm_gpusvm_notifier * +drm_gpusvm_notifier_alloc(struct drm_gpusvm *gpusvm, unsignedlong
fault_addr)drivers/gpu/drm/drm_gpusvm.c new file +7
-0+{ +struct drm_gpusvm_notifier *notifier; + +if (gpusvm->ops->notifier_alloc) + notifier = gpusvm->ops->notifier_alloc (); +else + notifier = kzalloc(sizeof(*notifier),
GFP_KERNEL);drivers/gpu/drm/drm_gpusvm.c new file +6
-0+ +if (!notifier) +return ERR_PTR (-ENOMEM ); + + notifier->gpusvm = gpusvm; + notifier->itree.start = ALIGN_DOWN(fault_addr, gpusvm-Quoted 1 line
notifier_size);drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ notifier->itree.last = ALIGN(fault_addr + 1, gpusvm-Quoted 1 line
notifier_size) - 1;drivers/gpu/drm/drm_gpusvm.c new file +16
-0+INIT_LIST_HEAD (¬ifier->entry); + notifier->root =RB_ROOT_CACHED ; +INIT_LIST_HEAD (¬ifier->range_list); + +return notifier; +} + +/** +* drm_gpusvm_notifier_free() - Free GPU SVM notifier +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier structure +* +* This function frees the GPU SVM notifier structure. +*/ +static void drm_gpusvm_notifier_free (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier*notifier)
drivers/gpu/drm/drm_gpusvm.c new file +11
-0+{ +WARN_ON (!RB_EMPTY_ROOT (¬ifier->root.rb_root)); + +if (gpusvm->ops->notifier_free) + gpusvm->ops->notifier_free (notifier); +else +kfree (notifier); +} + +/** +* to_drm_gpusvm_range() - retrieve the container struct for a given
rbtree node
+ * @node: a pointer to the rbtree node embedded within adrm_gpusvm_rangestruct
+ *
+ * Returns: A pointer to the containingdrm_gpusvm_rangestructure.
+ */
+static structdrm_gpusvm_range*to_drm_gpusvm_range(structrb_node
*node)drivers/gpu/drm/drm_gpusvm.c new file +2
-0+{ + return container_of(node, struct drm_gpusvm_range,itree.rb);
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+} + +/** +* drm_gpusvm_range_insert() - Insert GPU SVM range +* @notifier: Pointer to the GPU SVM notifier structure +* @range: Pointer to the GPU SVM range structure +* +* This function inserts the GPU SVM range into the notifier RB tree
and list.
+ */
+static void drm_gpusvm_range_insert(structdrm_gpusvm_notifier*notifier,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct drm_gpusvm_range*range)
drivers/gpu/drm/drm_gpusvm.c new file +23
-0+{ +struct rb_node *node; +struct list_head *head; + +drm_gpusvm_notifier_lock (notifier->gpusvm); +interval_tree_insert (&range->itree, ¬ifier->root); + + node =rb_prev (&range->itree.rb); +if (node) + head = &(to_drm_gpusvm_range (node))->entry; +else + head = ¬ifier->range_list; + +list_add (&range->entry, head); +drm_gpusvm_notifier_unlock (notifier->gpusvm); +} + +/** +* __drm_gpusvm_range_remove() - Remove GPU SVM range +* @notifier: Pointer to the GPU SVM notifier structure +* @range: Pointer to the GPU SVM range structure +* +* This macro removes the GPU SVM range from the notifier RB tree
and list.
+ */
+static void __drm_gpusvm_range_remove(structdrm_gpusvm_notifier*notifier,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct drm_gpusvm_range*range)
drivers/gpu/drm/drm_gpusvm.c new file +12
-0+{ +interval_tree_remove (&range->itree, ¬ifier->root); +list_del (&range->entry); +} + +/** +* drm_gpusvm_range_alloc() - Allocate GPU SVM range +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier structure +* @fault_addr: Fault address +* @chunk_size: Chunk size +* @migrate_devmem: Flag indicating whether to migrate device memory
+ *
+ * This function allocates and initializes the GPU SVM range structure.
+ *
+ * Returns:
+ * Pointer to the allocated GPU SVM range on success,ERR_PTR()on
failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+static struct drm_gpusvm_range * +drm_gpusvm_range_alloc (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + unsigned long fault_addr, unsigned long
chunk_size,drivers/gpu/drm/drm_gpusvm.c new file +17
-0+ bool migrate_devmem) +{ +struct drm_gpusvm_range *range; + +if (gpusvm->ops->range_alloc) + range = gpusvm->ops->range_alloc (gpusvm); +else + range =kzalloc (sizeof (*range),GFP_KERNEL ); + +if (!range) +return ERR_PTR (-ENOMEM ); + +kref_init (&range->refcount); + range->gpusvm = gpusvm; + range->notifier = notifier; + range->itree.start =ALIGN_DOWN (fault_addr, chunk_size); + range->itree.last = ALIGN(fault_addr + 1, chunk_size) -1;
drivers/gpu/drm/drm_gpusvm.c new file +15
-0+INIT_LIST_HEAD (&range->entry); + range->notifier_seq =LONG_MAX ; + range->flags.migrate_devmem = migrate_devmem ?1 :0 ; + +return range; +} + +/** +* drm_gpusvm_check_pages() - Check pages +* @gpusvm: Pointer to the GPU SVM structure +* @notifier: Pointer to the GPU SVM notifier structure +* @start: Start address +* @end: End address +* +* Check if pages between start and end have been faulted in on the
CPU. Use to
+ * prevent migration of pages without CPU backing store.
+ *
+ * Returns:
+ * True if pages have been faulted into CPU, False otherwise
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+static bool drm_gpusvm_check_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier*notifier,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long start, unsignedlong end)
drivers/gpu/drm/drm_gpusvm.c new file +7
-0+{ + struct hmm_range hmm_range = { + .default_flags = 0, + .notifier = ¬ifier->notifier, + .start = start, + .end = end, + .dev_private_owner = gpusvm-Quoted 1 line
device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ }; +unsigned long timeout = + jiffies +
msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);drivers/gpu/drm/drm_gpusvm.c new file +7
-0+unsigned long *pfns; +unsigned long npages =npages_in_range (start, end); +int err, i; + +mmap_assert_locked (gpusvm->mm); + + pfns = kvmalloc_array(npages, sizeof(*pfns),
GFP_KERNEL);drivers/gpu/drm/drm_gpusvm.c new file +4
-0+if (!pfns) +return false; + + hmm_range.notifier_seq =mmu_interval_read_begin(¬ifier-
Quoted 1 line
notifier);
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+ hmm_range.hmm_pfns = pfns; + +while (true) { + err =hmm_range_fault (&hmm_range); +if (err == -EBUSY ) { +if (time_after (jiffies, timeout)) +break ; + + hmm_range.notifier_seq = + mmu_interval_read_begin(¬ifier-
Quoted 1 line
notifier);
drivers/gpu/drm/drm_gpusvm.c new file +22
-0+continue ; + } +break ; + } +if (err) + goto err_free; + +for (i =0 ; i < npages;) { +if (!(pfns[i] &HMM_PFN_VALID )) { + err = -EFAULT ; + goto err_free; + } + i +=0x1 <<hmm_pfn_to_map_order (pfns[i]); + } + +err_free: +kvfree (pfns); +return err ? false : true; +} + +/** +* drm_gpusvm_range_chunk_size() - Determine chunk size for GPU SVM
range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier structure
+ * @vas: Pointer to the virtual memory area structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @check_pages_threshold: Check CPU pages for present threshold
+ *
+ * This function determines the chunk size for the GPU SVM range based on the
+ * fault address, GPU SVM chunk sizes, existing GPU SVM ranges, and
the virtual
+ * memory area boundaries.
+ *
+ * Returns:
+ * Chunk size on success,LONG_MAXon failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+static unsigned long +drm_gpusvm_range_chunk_size (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier*notifier,
drivers/gpu/drm/drm_gpusvm.c new file +12
-0+ struct vm_area_struct *vas, + unsigned long fault_addr, + unsigned long gpuva_start, + unsigned long gpuva_end, + unsigned long check_pages_threshold) +{ +unsigned long start, end; +int i =0 ; + +retry: +for (; i < gpusvm->num_chunks; ++i) { + start = ALIGN_DOWN(fault_addr, gpusvm-Quoted 1 line
chunk_sizes[i]);
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ end = ALIGN(fault_addr + 1, gpusvm-Quoted 1 line
chunk_sizes[i]);
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ + if (start >= vas->vm_start && end <= vas->vm_end&&
drivers/gpu/drm/drm_gpusvm.c new file +11
-0+ start >= notifier->itree.start && + end <= notifier->itree.last + 1 && + start >= gpuva_start && end <= gpuva_end) +break ; + } + +if (i == gpusvm->num_chunks) +return LONG_MAX ; + +/* +* If allocation more than page, ensure not to overlap with
existingdrivers/gpu/drm/drm_gpusvm.c new file +6
-0+* ranges. +*/ +if (end - start !=SZ_4K ) { +struct drm_gpusvm_range *range; + + range = drm_gpusvm_range_find(notifier, start,end);
drivers/gpu/drm/drm_gpusvm.c new file +7
-0+if (range) { + ++i; + goto retry; + } + +/* +* XXX: Only create range on pages CPU has faulted
in. Withoutdrivers/gpu/drm/drm_gpusvm.c new file +1
-0+* this check, or prefault, on BMG '
xe_exec_system_allocator--rdrivers/gpu/drm/drm_gpusvm.c new file +1
-0+* process-many-malloc' fails. In the failure case,
each processdrivers/gpu/drm/drm_gpusvm.c new file +1
-0+* mallocs 16k but the CPU VMA is ~128k which results in 64k SVM
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* ranges. When migrating the SVM ranges, some processes fail in
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* drm_gpusvm_migrate_to_devmem with 'migrate.cpages
!= npages'drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* and then upon drm_gpusvm_range_get_pages device
pages fromdrivers/gpu/drm/drm_gpusvm.c new file +1
-0+* other processes are collected + faulted in which
creates alldrivers/gpu/drm/drm_gpusvm.c new file +1
-0+* sorts of problems. Unsure exactly how this happening, also
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* problem goes away if '
xe_exec_system_allocator--
rdrivers/gpu/drm/drm_gpusvm.c new file +1
-0+* process-many-malloc' mallocs at least 64k at a
time.drivers/gpu/drm/drm_gpusvm.c new file +3
-0+*/ +if (end - start <= check_pages_threshold && + !drm_gpusvm_check_pages(gpusvm, notifier,start,
end)) {drivers/gpu/drm/drm_gpusvm.c new file +10
-0+ ++i; + goto retry; + } + } + +return end - start; +} + +/** +* drm_gpusvm_range_find_or_insert() - Find or insert GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @fault_addr: Fault address
+ * @gpuva_start: Start address of GPUVA which mirrors CPU
+ * @gpuva_end: End address of GPUVA which mirrors CPU
+ * @ctx: GPU SVM context
+ *
+ * This function finds or inserts a newly allocated a GPU SVM range
based on the
+ * fault address. Caller must hold a lock to protect range lookup
and insertion.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range on success,ERR_PTR()on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +6
-0+struct drm_gpusvm_range * +drm_gpusvm_range_find_or_insert (struct drm_gpusvm *gpusvm, +unsigned long fault_addr, +unsigned long gpuva_start, +unsigned long gpuva_end, + const struct drm_gpusvm_ctx*ctx)
drivers/gpu/drm/drm_gpusvm.c new file +22
-0+{ +struct drm_gpusvm_notifier *notifier; +struct drm_gpusvm_range *range; +struct mm_struct *mm = gpusvm->mm; +struct vm_area_struct *vas; +bool notifier_alloc = false; +unsigned long chunk_size; +int err; +bool migrate_devmem; + +drm_gpusvm_driver_lock_held (gpusvm); + +if (fault_addr < gpusvm->mm_start || + fault_addr > gpusvm->mm_start + gpusvm->mm_range) +return ERR_PTR (-EINVAL ); + +if (!mmget_not_zero (mm)) +return ERR_PTR (-EFAULT ); + + notifier =drm_gpusvm_notifier_find (gpusvm, fault_addr); +if (!notifier) { + notifier = drm_gpusvm_notifier_alloc(gpusvm,
fault_addr);drivers/gpu/drm/drm_gpusvm.c new file +6
-0+if (IS_ERR (notifier)) { + err =PTR_ERR (notifier); + goto err_mmunlock; + } + notifier_alloc = true; + err = mmu_interval_notifier_insert(¬ifier-Quoted 1 line
notifier,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ mm, notifier-Quoted 1 line
itree.start,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ notifier-Quoted 1 line
itree.last -
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ notifier-Quoted 1 line
itree.start + 1,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+
&drm_gpusvm_notifier_ops);drivers/gpu/drm/drm_gpusvm.c new file +18
-0+if (err) + goto err_notifier; + } + +mmap_read_lock (mm); + + vas =vma_lookup (mm, fault_addr); +if (!vas) { + err = -ENOENT ; + goto err_notifier_remove; + } + +if (!ctx->read_only && !(vas->vm_flags &VM_WRITE )) { + err = -EPERM ; + goto err_notifier_remove; + } + + range = drm_gpusvm_range_find(notifier, fault_addr,
fault_addr+ 1);drivers/gpu/drm/drm_gpusvm.c new file +4
-0+if (range) + goto out_mmunlock; +/* +* XXX: Short-circuiting migration based on migrate_vma_*
currentdrivers/gpu/drm/drm_gpusvm.c new file +1
-0+* limitations. If/when migrate_vma_* add more support, this
logic willdrivers/gpu/drm/drm_gpusvm.c new file +4
-0+* have to change. +*/ + migrate_devmem = ctx->devmem_possible && + vma_is_anonymous(vas) &&!is_vm_hugetlb_page(vas);
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ + chunk_size = drm_gpusvm_range_chunk_size(gpusvm,notifier,
vas,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ fault_addr,
gpuva_start,drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ gpuva_end, + ctx-Quoted 1 line
check_pages_threshold);drivers/gpu/drm/drm_gpusvm.c new file +6
-0+if (chunk_size ==LONG_MAX ) { + err = -EINVAL ; + goto err_notifier_remove; + } + + range = drm_gpusvm_range_alloc(gpusvm, notifier,
fault_addr,chunk_size,drivers/gpu/drm/drm_gpusvm.c new file +20
-0+ migrate_devmem); +if (IS_ERR (range)) { + err =PTR_ERR (range); + goto err_notifier_remove; + } + +drm_gpusvm_range_insert (notifier, range); +if (notifier_alloc) +drm_gpusvm_notifier_insert (gpusvm, notifier); + +out_mmunlock: +mmap_read_unlock (mm); +mmput (mm); + +return range; + +err_notifier_remove: +mmap_read_unlock (mm); +if (notifier_alloc) + mmu_interval_notifier_remove(¬ifier-Quoted 1 line
notifier);
drivers/gpu/drm/drm_gpusvm.c new file +11
-0+err_notifier: +if (notifier_alloc) +drm_gpusvm_notifier_free (gpusvm, notifier); +err_mmunlock: +mmput (mm); +return ERR_PTR (err); +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_find_or_insert); + +/** +* __drm_gpusvm_range_unmap_pages() - Unmap pages associated with a
GPU SVM range (internal)
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @npages: Number of pages to unmap
+ *
+ * This function unmap pages associated with a GPU SVM range. Assumes and
+ * asserts correct locking is in place when called.
+ */
+static void __drm_gpusvm_range_unmap_pages(structdrm_gpusvm*gpusvm,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct
drm_gpusvm_range
*range,drivers/gpu/drm/drm_gpusvm.c new file +11
-0+ unsigned long npages) +{ +unsigned long i, j; +struct drm_pagemap *dpagemap = range->dpagemap; +struct device *dev = gpusvm->drm->dev; + +lockdep_assert_held (&gpusvm->notifier_lock); + +if (range->flags.has_dma_mapping) { +for (i =0 , j =0 ; i < npages; j++) { + struct drm_pagemap_dma_addr *addr =&range-
Quoted 1 line
dma_addr[j];
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ + if (addr->proto ==
DRM_INTERCONNECT_SYSTEM)drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ dma_unmap_page(dev, + addr->addr, + PAGE_SIZE <<addr-
Quoted 1 line
order,
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ addr->dir); + else if (dpagemap && dpagemap->ops-Quoted 1 line
unmap_dma)drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ dpagemap->ops-Quoted 1 line
unmap_dma(dpagemap,
drivers/gpu/drm/drm_gpusvm.c new file +12
-0+ dev, + *addr); + i +=1 << addr->order; + } + range->flags.has_devmem_pages = false; + range->flags.has_dma_mapping = false; + range->dpagemap =NULL ; + } +} + +/** +* drm_gpusvm_range_free_pages() - Free pages associated with a GPU
SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function frees the dma address array associated with a GPU
SVM range.
+ */
+static void drm_gpusvm_range_free_pages(structdrm_gpusvm*gpusvm,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+struct drm_gpusvm_range *range)
drivers/gpu/drm/drm_gpusvm.c new file +15
-0+{ +lockdep_assert_held (&gpusvm->notifier_lock); + +if (range->dma_addr) { +kvfree (range->dma_addr); + range->dma_addr =NULL ; + } +} + +/** +* drm_gpusvm_range_remove() - Remove GPU SVM range +* @gpusvm: Pointer to the GPU SVM structure +* @range: Pointer to the GPU SVM range to be removed +* +* This function removes the specified GPU SVM range and also removes the parent
+ * GPU SVM notifier if no more ranges remain in the notifier. The
caller must
+ * hold a lock to protect range and notifier removal.
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+void drm_gpusvm_range_remove (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range) +{ + unsigned long npages = npages_in_range(range-Quoted 1 line
itree.start,
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ range->itree.last +1);
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+struct drm_gpusvm_notifier *notifier; + +drm_gpusvm_driver_lock_held (gpusvm); + + notifier = drm_gpusvm_notifier_find(gpusvm, range-Quoted 1 line
itree.start);
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+if (WARN_ON_ONCE (!notifier)) +return ; + +drm_gpusvm_notifier_lock (gpusvm); +__drm_gpusvm_range_unmap_pages (gpusvm, range, npages); +drm_gpusvm_range_free_pages (gpusvm, range); +__drm_gpusvm_range_remove (notifier, range); +drm_gpusvm_notifier_unlock (gpusvm); + +drm_gpusvm_range_put (range); + +if (RB_EMPTY_ROOT (¬ifier->root.rb_root)) { +if (!notifier->flags.removed) + mmu_interval_notifier_remove(¬ifier-Quoted 1 line
notifier);
drivers/gpu/drm/drm_gpusvm.c new file +11
-0+drm_gpusvm_notifier_remove (gpusvm, notifier); +drm_gpusvm_notifier_free (gpusvm, notifier); + } +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_remove); + +/** +* drm_gpusvm_range_get() - Get a reference to GPU SVM range +* @range: Pointer to the GPU SVM range +* +* This function increments the reference count of the specified GPU
SVM range.
+ *
+ * Returns:
+ * Pointer to the GPU SVM range.
+ */drivers/gpu/drm/drm_gpusvm.c new file +12
-0+struct drm_gpusvm_range * +drm_gpusvm_range_get (struct drm_gpusvm_range *range) +{ +kref_get (&range->refcount); + +return range; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_get); + +/** +* drm_gpusvm_range_destroy() - Destroy GPU SVM range +* @refcount: Pointer to the reference counter embedded in the GPU
SVM range
+ *
+ * This function destroys the specified GPU SVM range when its reference count
+ * reaches zero. If a custom range-free function is provided, it is
invoked to
+ * free the range; otherwise, the range is deallocated usingkfree().
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+static void drm_gpusvm_range_destroy (struct kref *refcount) +{ +struct drm_gpusvm_range *range = + container_of(refcount, struct drm_gpusvm_range,refcount);
drivers/gpu/drm/drm_gpusvm.c new file +13
-0+struct drm_gpusvm *gpusvm = range->gpusvm; + +if (gpusvm->ops->range_free) + gpusvm->ops->range_free (range); +else +kfree (range); +} + +/** +* drm_gpusvm_range_put() - Put a reference to GPU SVM range +* @range: Pointer to the GPU SVM range +* +* This function decrements the reference count of the specified GPU
SVM range
+ * and frees it when the count reaches zero.
+ */drivers/gpu/drm/drm_gpusvm.c new file +12
-0+void drm_gpusvm_range_put (struct drm_gpusvm_range *range) +{ +kref_put (&range->refcount, drm_gpusvm_range_destroy); +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_put); + +/** +* drm_gpusvm_range_pages_valid() - GPU SVM range pages valid +* @gpusvm: Pointer to the GPU SVM structure +* @range: Pointer to the GPU SVM range structure +* +* This function determines if a GPU SVM range pages are valid. Expected be
+ * called holdinggpusvm->notifier_lockand as the last step before
committing a
+ * GPU binding. This is akin to a notifier seqno check in the HMM
documentation
+ * but due to wider notifiers (i.e., notifiers which span multiple
ranges) this
+ * function is required for finer grained checking (i.e., per range)
if pages
+ * are valid.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+bool drm_gpusvm_range_pages_valid (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range*range)
drivers/gpu/drm/drm_gpusvm.c new file +4
-0+{ +lockdep_assert_held (&gpusvm->notifier_lock); + + return range->flags.has_devmem_pages || range-Quoted 1 line
flags.has_dma_mapping;
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_pages_valid); + +/** +* drm_gpusvm_range_pages_valid_unlocked() - GPU SVM range pages valid unlocked
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ *
+ * This function determines if a GPU SVM range pages are valid. Expected be
+ * called without holdinggpusvm->notifier_lock.
+ *
+ * Returns:
+ * True if GPU SVM range has valid pages, False otherwise
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+static bool +drm_gpusvm_range_pages_valid_unlocked (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range*range)
drivers/gpu/drm/drm_gpusvm.c new file +8
-0+{ +bool pages_valid; + +if (!range->dma_addr) +return false; + +drm_gpusvm_notifier_lock (gpusvm); + pages_valid = drm_gpusvm_range_pages_valid(gpusvm,range);
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+if (!pages_valid) +drm_gpusvm_range_free_pages (gpusvm, range); +drm_gpusvm_notifier_unlock (gpusvm); + +return pages_valid; +} + +/** +* drm_gpusvm_range_get_pages() - Get pages for a GPU SVM range +* @gpusvm: Pointer to the GPU SVM structure +* @range: Pointer to the GPU SVM range structure +* @ctx: GPU SVM context +* +* This function gets pages for a GPU SVM range and ensures they are
mapped for
+ * DMA access.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +5
-0+int drm_gpusvm_range_get_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range, + const struct drm_gpusvm_ctx *ctx) +{ + struct mmu_interval_notifier *notifier = &range-Quoted 2 lines
notifier-
notifier;drivers/gpu/drm/drm_gpusvm.c new file +2
-0+struct hmm_range hmm_range = { + .default_flags = HMM_PFN_REQ_FAULT | (ctx-Quoted 1 line
read_only? 0 :
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ HMM_PFN_REQ_WRITE), + .notifier = notifier, + .start = range->itree.start, + .end = range->itree.last + 1, + .dev_private_owner = gpusvm-Quoted 1 line
device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ }; +struct mm_struct *mm = gpusvm->mm; +struct drm_gpusvm_zdd *zdd; +unsigned long timeout = + jiffies +
msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);drivers/gpu/drm/drm_gpusvm.c new file +2
-0+unsigned long i, j; + unsigned long npages = npages_in_range(range-Quoted 1 line
itree.start,
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ range->itree.last +1);
drivers/gpu/drm/drm_gpusvm.c new file +10
-0+unsigned long num_dma_mapped; +unsigned int order =0 ; +unsigned long *pfns; +struct page **pages; +int err =0 ; +struct dev_pagemap *pagemap; +struct drm_pagemap *dpagemap; + +retry: + hmm_range.notifier_seq =
mmu_interval_read_begin(notifier);drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ if (drm_gpusvm_range_pages_valid_unlocked(gpusvm,range))
drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ goto set_seqno; + + pfns = kvmalloc_array(npages, sizeof(*pfns),
GFP_KERNEL);drivers/gpu/drm/drm_gpusvm.c new file +20
-0+if (!pfns) +return -ENOMEM ; + +if (!mmget_not_zero (mm)) { + err = -EFAULT ; + goto err_free; + } + + hmm_range.hmm_pfns = pfns; +while (true) { +mmap_read_lock (mm); + err =hmm_range_fault (&hmm_range); +mmap_read_unlock (mm); + +if (err == -EBUSY ) { +if (time_after (jiffies, timeout)) +break ; + + hmm_range.notifier_seq = + mmu_interval_read_begin(notifier);
drivers/gpu/drm/drm_gpusvm.c new file +12
-0+continue ; + } +break ; + } +mmput (mm); +if (err) + goto err_free; + + pages = (struct page **)pfns; +map_pages: +/* +* Perform all dma mappings under the notifier lock to not
drivers/gpu/drm/drm_gpusvm.c new file +12
-0+* access freed pages. A notifier will either block on +* the notifier lock or unmap dma. +*/ +drm_gpusvm_notifier_lock (gpusvm); + +if (range->flags.unmapped) { +drm_gpusvm_notifier_unlock (gpusvm); + err = -EFAULT ; + goto err_free; + } + + if (mmu_interval_read_retry(notifier,hmm_range.notifier_seq)) {
drivers/gpu/drm/drm_gpusvm.c new file +7
-0+drm_gpusvm_notifier_unlock (gpusvm); +kvfree (pfns); + goto retry; + } + +if (!range->dma_addr) { +/* Unlock and restart mapping to allocate memory. */
drivers/gpu/drm/drm_gpusvm.c new file +3
-0+drm_gpusvm_notifier_unlock (gpusvm); + range->dma_addr = kvmalloc_array(npages, + sizeof(*range-Quoted 1 line
dma_addr),drivers/gpu/drm/drm_gpusvm.c new file +17
-0+ GFP_KERNEL); +if (!range->dma_addr) { + err = -ENOMEM ; + goto err_free; + } + goto map_pages; + } + + zdd =NULL ; + num_dma_mapped =0 ; + for (i = 0, j = 0; i < npages; ++j) { + struct page *page = hmm_pfn_to_page(pfns[i]); + + order = hmm_pfn_to_map_order(pfns[i]); + if (is_device_private_page(page) || + is_device_coherent_page(page)) { + if (zdd != page->zone_device_data && i >0)
{drivers/gpu/drm/drm_gpusvm.c new file +12
-0+ err = -EOPNOTSUPP ; + goto err_unmap; + } + zdd = page->zone_device_data; +if (pagemap != page->pgmap) { +if (i >0 ) { + err = -EOPNOTSUPP ; + goto err_unmap; + } + + pagemap = page->pgmap; + dpagemap = zdd-Quoted 2 lines
devmem_allocation-
dpagemap;drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ if (drm_WARN_ON(gpusvm->drm,!dpagemap)) {
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+/* +* Raced. This is not supposed to happen
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+* since
hmm_range_fault()
should've migrateddrivers/gpu/drm/drm_gpusvm.c new file +9
-0+* this page to system. +*/ + err = -EAGAIN ; + goto err_unmap; + } + } + range->dma_addr[j] = + dpagemap->ops->map_dma(dpagemap, + gpusvm-Quoted 2 lines
drm-
dev,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ page,order,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+
DMA_BIDIRECTIONAL);drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ if (dma_mapping_error(gpusvm->drm->dev, + range-Quoted 1 line
dma_addr[j].addr)) {
drivers/gpu/drm/drm_gpusvm.c new file +18
-0+ err = -EFAULT ; + goto err_unmap; + } + + pages[i] = page; + }else { +dma_addr_t addr; + +if (is_zone_device_page (page) || zdd) { + err = -EOPNOTSUPP ; + goto err_unmap; + } + + addr =dma_map_page (gpusvm->drm->dev, + page, 0, + PAGE_SIZE << order, + DMA_BIDIRECTIONAL); + if (dma_mapping_error(gpusvm->drm->dev,addr)) {
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ err = -EFAULT ; + goto err_unmap; + } + + range->dma_addr[j] =
drm_pagemap_dma_addr_encodedrivers/gpu/drm/drm_gpusvm.c new file +1
-0+ (addr, DRM_INTERCONNECT_SYSTEM,order,
drivers/gpu/drm/drm_gpusvm.c new file +21
-0+ DMA_BIDIRECTIONAL); + } + i +=1 << order; + num_dma_mapped = i; + } + + range->flags.has_dma_mapping = true; +if (zdd) { + range->flags.has_devmem_pages = true; + range->dpagemap = dpagemap; + } + +drm_gpusvm_notifier_unlock (gpusvm); +kvfree (pfns); +set_seqno: + range->notifier_seq = hmm_range.notifier_seq; + +return 0 ; + +err_unmap: + __drm_gpusvm_range_unmap_pages(gpusvm, range,
num_dma_mapped);drivers/gpu/drm/drm_gpusvm.c new file +11
-0+drm_gpusvm_notifier_unlock (gpusvm); +err_free: +kvfree (pfns); +if (err == -EAGAIN ) + goto retry; +return err; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_get_pages); + +/** +* drm_gpusvm_range_unmap_pages() - Unmap pages associated with a
GPU SVM range
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @ctx: GPU SVM context
+ *
+ * This function unmaps pages associated with a GPU SVM range. If
@in_notifier
+ * is set, it is assumed thatgpusvm->notifier_lockis held in write
mode; if it
+ * is clear, it acquiresgpusvm->notifier_lockin read mode. Must be
called on
+ * each GPU SVM range attached to notifier in gpusvm->ops-Quoted 1 line
invalidate for IOMMU
+ * security model.
+ */drivers/gpu/drm/drm_gpusvm.c new file +2
-0+void drm_gpusvm_range_unmap_pages (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range*range,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ const struct drm_gpusvm_ctx*ctx)
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+{ + unsigned long npages = npages_in_range(range-Quoted 1 line
itree.start,
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ range->itree.last +1);
drivers/gpu/drm/drm_gpusvm.c new file +3
-0+ +if (ctx->in_notifier) + lockdep_assert_held_write(&gpusvm-Quoted 1 line
notifier_lock);drivers/gpu/drm/drm_gpusvm.c new file +17
-0+else +drm_gpusvm_notifier_lock (gpusvm); + +__drm_gpusvm_range_unmap_pages (gpusvm, range, npages); + +if (!ctx->in_notifier) +drm_gpusvm_notifier_unlock (gpusvm); +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_unmap_pages); + +/** +* drm_gpusvm_migration_unlock_put_page() - Put a migration page +* @page: Pointer to the page to put +* +* This function unlocks and puts a page. +*/ +static void drm_gpusvm_migration_unlock_put_page(struct page*page)
drivers/gpu/drm/drm_gpusvm.c new file +13
-0+{ +unlock_page (page); +put_page (page); +} + +/** +* drm_gpusvm_migration_unlock_put_pages() - Put migration pages +* @npages: Number of pages +* @migrate_pfn: Array of migrate page frame numbers +* +* This function unlocks and puts an array of pages. +*/ +static void drm_gpusvm_migration_unlock_put_pages(unsigned longnpages,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long
*migrate_pfn)drivers/gpu/drm/drm_gpusvm.c new file +17
-0+{ +unsigned long i; + +for (i =0 ; i < npages; ++i) { +struct page *page; + +if (!migrate_pfn[i]) +continue ; + + page =migrate_pfn_to_page (migrate_pfn[i]); +drm_gpusvm_migration_unlock_put_page (page); + migrate_pfn[i] =0 ; + } +} + +/** +* drm_gpusvm_get_devmem_page() - Get a reference to a device memory
page
+ * @page: Pointer to the page
+ * @zdd: Pointer to the GPU SVM zone device data
+ *
+ * This function associates the given page with the specified GPU
SVM zone
+ * device data and initializes it for zone device usage.
+ */drivers/gpu/drm/drm_gpusvm.c new file +9
-0+static void drm_gpusvm_get_devmem_page (struct page *page, + struct drm_gpusvm_zdd *zdd) +{ + page->zone_device_data =drm_gpusvm_zdd_get (zdd); +zone_device_page_init (page); +} + +/** +* drm_gpusvm_migrate_map_pages() - Map migration pages for GPU SVM
migration
+ * @dev: The device for which the pages are being mapped
+ * @dma_addr: Array to store DMA addresses corresponding to mapped
pages
+ * @migrate_pfn: Array of migrate page frame numbers to map
+ * @npages: Number of pages to map
+ * @dir: Direction of data transfer (e.g.,DMA_BIDIRECTIONAL)
+ *
+ * This function maps pages of memory for migration usage in GPU SVM. It
+ * iterates over each page frame number provided in @migrate_pfn,
maps the
+ * corresponding page, and stores the DMA address in the provided
@dma_addr
+ * array.
+ *
+ * Returns: 0 on success,-EFAULTif an error occurs during mapping.
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+static int drm_gpusvm_migrate_map_pages (struct device *dev, +dma_addr_t *dma_addr, + unsigned long
*migrate_pfn,drivers/gpu/drm/drm_gpusvm.c new file +2
-0+unsigned long npages, + enum dma_data_directiondir)
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+{ +unsigned long i; + +for (i =0 ; i < npages; ++i) { + struct page *page =
migrate_pfn_to_page(migrate_pfn[i]);drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ +if (!page) +continue ; + +if (WARN_ON_ONCE (is_zone_device_page (page))) +return -EFAULT ; + + dma_addr[i] = dma_map_page(dev, page, 0,
PAGE_SIZE,
dir);drivers/gpu/drm/drm_gpusvm.c new file +9
-0+if (dma_mapping_error (dev, dma_addr[i])) +return -EFAULT ; + } + +return 0 ; +} + +/** +* drm_gpusvm_migrate_unmap_pages() - Unmap pages previously mapped
for GPU SVM migration
+ * @dev: The device for which the pages were mapped
+ * @dma_addr: Array of DMA addresses corresponding to mapped pages
+ * @npages: Number of pages to unmap
+ * @dir: Direction of data transfer (e.g.,DMA_BIDIRECTIONAL)
+ *
+ * This function unmaps previously mapped pages of memory for GPU
Shared Virtual
+ * Memory (SVM). It iterates over each DMA address provided in @dma_addr, checks
+ * if it's valid and not already unmapped, and unmaps the corresponding page.
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+static void drm_gpusvm_migrate_unmap_pages(struct device *dev, + dma_addr_t *dma_addr, + unsigned long npages, + enum
dma_data_direction
dir)drivers/gpu/drm/drm_gpusvm.c new file +5
-0+{ + unsigned long i; + + for (i = 0; i < npages; ++i) { + if (!dma_addr[i] || dma_mapping_error(dev,dma_addr[i]))
drivers/gpu/drm/drm_gpusvm.c new file +3
-0+continue ; + + dma_unmap_page(dev, dma_addr[i], PAGE_SIZE,dir);
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ } +} + +/** +* drm_gpusvm_migrate_to_devmem() - Migrate GPU SVM range to device
memory
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @range: Pointer to the GPU SVM range structure
+ * @devmem_allocation: Pointer to the device memory allocation. The
caller
+ * should hold a reference to the device memory
allocation,
+ * which should be dropped via ops-Quoted 1 line
devmem_releaseor upon+ * the failure of this function.
+ * @ctx: GPU SVM context
+ *
+ * This function migrates the specified GPU SVM range to device memory. It performs the
+ * necessary setup and invokes the driver-specific operations for
migration to
+ * device memory. Upon successful return, @devmem_allocation can safely reference @range
+ * untilops->devmem_releaseis called which only upon successful
return.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +3
-0+int drm_gpusvm_migrate_to_devmem (struct drm_gpusvm *gpusvm, +struct drm_gpusvm_range *range, + struct drm_gpusvm_devmem
*devmem_allocation,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ const struct drm_gpusvm_ctx*ctx)
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+{ + const struct drm_gpusvm_devmem_ops *ops =devmem_allocation-
Quoted 1 line
ops;
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long start = range->itree.start, end = range-Quoted 1 line
itree.last + 1;
drivers/gpu/drm/drm_gpusvm.c new file +4
-0+struct migrate_vma migrate = { + .start = start, + .end = end, + .pgmap_owner = gpusvm-Quoted 1 line
device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +41
-0+ .flags = MIGRATE_VMA_SELECT_SYSTEM, + }; +struct mm_struct *mm = gpusvm->mm; +unsigned long i, npages =npages_in_range (start, end); +struct vm_area_struct *vas; +struct drm_gpusvm_zdd *zdd =NULL ; +struct page **pages; +dma_addr_t *dma_addr; +void *buf; +int err; + +if (!range->flags.migrate_devmem) +return -EINVAL ; + +if (!ops->populate_devmem_pfn || !ops->copy_to_devmem || + !ops->copy_to_ram) +return -EOPNOTSUPP ; + +if (!mmget_not_zero (mm)) { + err = -EFAULT ; + goto err_out; + } +mmap_read_lock (mm); + + vas =vma_lookup (mm, start); +if (!vas) { + err = -ENOENT ; + goto err_mmunlock; + } + +if (end > vas->vm_end || start < vas->vm_start) { + err = -EINVAL ; + goto err_mmunlock; + } + +if (!vma_is_anonymous (vas)) { + err = -EBUSY ; + goto err_mmunlock; + } + + buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +
sizeof(*dma_addr)+drivers/gpu/drm/drm_gpusvm.c new file +7
-0+ sizeof(*pages), GFP_KERNEL); +if (!buf) { + err = -ENOMEM ; + goto err_mmunlock; + } + dma_addr = buf + (2 *sizeof (*migrate.src) * npages); + pages = buf + (2 * sizeof(*migrate.src) +
sizeof(*dma_addr))
* npages;drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ + zdd = drm_gpusvm_zdd_alloc(gpusvm-Quoted 1 line
device_private_page_owner);drivers/gpu/drm/drm_gpusvm.c new file +24
-0+if (!zdd) { + err = -ENOMEM ; + goto err_free; + } + + migrate.vma = vas; + migrate.src = buf; + migrate.dst = migrate.src + npages; + + err =migrate_vma_setup (&migrate); +if (err) + goto err_free; + +if (!migrate.cpages) { + err = -EFAULT ; + goto err_free; + } + +if (migrate.cpages != npages) { + err = -EBUSY ; + goto err_finalize; + } + + err = ops->populate_devmem_pfn(devmem_allocation,npages,
migrate.dst);drivers/gpu/drm/drm_gpusvm.c new file +4
-0+if (err) + goto err_finalize; + + err = drm_gpusvm_migrate_map_pages(devmem_allocation-Quoted 1 line
dev,
dma_addr,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ migrate.src, npages,
DMA_TO_DEVICE);drivers/gpu/drm/drm_gpusvm.c new file +16
-0+if (err) + goto err_finalize; + +for (i =0 ; i < npages; ++i) { +struct page *page =pfn_to_page (migrate.dst[i]); + + pages[i] = page; + migrate.dst[i] =migrate_pfn (migrate.dst[i]); +drm_gpusvm_get_devmem_page (page, zdd); + } + + err = ops->copy_to_devmem (pages, dma_addr, npages); +if (err) + goto err_finalize; + +/* Upon success bind devmem allocation to range and zdd */
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ zdd->devmem_allocation = devmem_allocation;/* Owns ref
*/drivers/gpu/drm/drm_gpusvm.c new file +4
-0+ +err_finalize: +if (err) + drm_gpusvm_migration_unlock_put_pages(npages,migrate.dst);
drivers/gpu/drm/drm_gpusvm.c new file +3
-0+migrate_vma_pages (&migrate); +migrate_vma_finalize (&migrate); + drm_gpusvm_migrate_unmap_pages(devmem_allocation->dev,
dma_addr, npages,drivers/gpu/drm/drm_gpusvm.c new file +15
-0+ DMA_TO_DEVICE); +err_free: +if (zdd) +drm_gpusvm_zdd_put (zdd); +kvfree (buf); +err_mmunlock: +mmap_read_unlock (mm); +mmput (mm); +err_out: +return err; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_migrate_to_devmem); + +/** +* drm_gpusvm_migrate_populate_ram_pfn() - Populate RAM PFNs for a
VM area
+ * @vas: Pointer to the VM area structure, can be NULL
+ * @fault_page: Fault page
+ * @npages: Number of pages to populate
+ * @mpages: Number of pages to migrate
+ * @src_mpfn: Source array of migrate PFNs
+ * @mpfn: Array of migrate PFNs to populate
+ * @addr: Start address for PFN allocation
+ *
+ * This function populates the RAM migrate page frame numbers (PFNs)
for the
+ * specified VM area structure. It allocates and locks pages in the
VM area for
+ * RAM usage. If vas is non-NULL usealloc_page_vmafor allocation,
if NULL use
+ *alloc_pagefor allocation.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int drm_gpusvm_migrate_populate_ram_pfn(structvm_area_struct
*vas,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct page
*fault_page,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned longnpages,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long*mpages,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long
*src_mpfn,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned long*mpfn,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ unsigned longaddr)
drivers/gpu/drm/drm_gpusvm.c new file +21
-0+{ +unsigned long i; + +for (i =0 ; i < npages; ++i, addr +=PAGE_SIZE ) { +struct page *page, *src_page; + +if (!(src_mpfn[i] &MIGRATE_PFN_MIGRATE )) +continue ; + + src_page =migrate_pfn_to_page (src_mpfn[i]); +if (!src_page) +continue ; + +if (fault_page) { +if (src_page->zone_device_data != + fault_page->zone_device_data) +continue ; + } + +if (vas) + page = alloc_page_vma(GFP_HIGHUSER, vas,addr);
drivers/gpu/drm/drm_gpusvm.c new file +11
-0+else + page =alloc_page (GFP_HIGHUSER ); + +if (!page) + goto free_pages; + + mpfn[i] =migrate_pfn (page_to_pfn (page)); + } + +for (i =0 ; i < npages; ++i) { + struct page *page =
migrate_pfn_to_page(mpfn[i]);drivers/gpu/drm/drm_gpusvm.c new file +13
-0+ +if (!page) +continue ; + +WARN_ON_ONCE (!trylock_page (page)); + ++*mpages; + } + +return 0 ; + +free_pages: +for (i =0 ; i < npages; ++i) { + struct page *page =
migrate_pfn_to_page(mpfn[i]);drivers/gpu/drm/drm_gpusvm.c new file +15
-0+ +if (!page) +continue ; + +put_page (page); + mpfn[i] =0 ; + } +return -ENOMEM ; +} + +/** +* drm_gpusvm_evict_to_ram() - Evict GPU SVM range to RAM +* @devmem_allocation: Pointer to the device memory allocation +* +* Similar to __drm_gpusvm_migrate_to_ram but does not require mmap
lock and
+ * migration done via migrate_device_* functions.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+intdrm_gpusvm_evict_to_ram(struct drm_gpusvm_devmem *devmem_allocation)drivers/gpu/drm/drm_gpusvm.c new file +2
-0+{ + const struct drm_gpusvm_devmem_ops *ops =devmem_allocation-
Quoted 1 line
ops;
drivers/gpu/drm/drm_gpusvm.c new file +15
-0+unsigned long npages, mpages =0 ; +struct page **pages; +unsigned long *src, *dst; +dma_addr_t *dma_addr; +void *buf; +int i, err =0 ; +unsigned int retry_count =2 ; + + npages = devmem_allocation->size >>PAGE_SHIFT ; + +retry: +if (!mmget_not_zero (devmem_allocation->mm)) +return -EFAULT ; + + buf = kvcalloc(npages, 2 * sizeof(*src) +
sizeof(*dma_addr)drivers/gpu/drm/drm_gpusvm.c new file +10
-0+ + sizeof(*pages), GFP_KERNEL); +if (!buf) { + err = -ENOMEM ; + goto err_out; + } + src = buf; + dst = buf + (sizeof (*src) * npages); + dma_addr = buf + (2 *sizeof (*src) * npages); + pages = buf + (2 * sizeof(*src) + sizeof(*dma_addr)) *npages;
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ + err = ops->populate_devmem_pfn(devmem_allocation,npages,
src);drivers/gpu/drm/drm_gpusvm.c new file +8
-0+if (err) + goto err_free; + + err =migrate_device_pfns (src, npages); +if (err) + goto err_free; + + err = drm_gpusvm_migrate_populate_ram_pfn(NULL, NULL,npages, &mpages,
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ src, dst, 0); +if (err || !mpages) + goto err_finalize; + + err = drm_gpusvm_migrate_map_pages(devmem_allocation-Quoted 1 line
dev,
dma_addr,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ dst, npages,
DMA_FROM_DEVICE);drivers/gpu/drm/drm_gpusvm.c new file +13
-0+if (err) + goto err_finalize; + +for (i =0 ; i < npages; ++i) + pages[i] =migrate_pfn_to_page (src[i]); + + err = ops->copy_to_ram (pages, dma_addr, npages); +if (err) + goto err_finalize; + +err_finalize: +if (err) + drm_gpusvm_migration_unlock_put_pages(npages,dst);
drivers/gpu/drm/drm_gpusvm.c new file +3
-0+migrate_device_pages (src, dst, npages); +migrate_device_finalize (src, dst, npages); + drm_gpusvm_migrate_unmap_pages(devmem_allocation->dev,
dma_addr, npages,drivers/gpu/drm/drm_gpusvm.c new file +20
-0+ DMA_FROM_DEVICE); +err_free: +kvfree (buf); +err_out: +mmput_async (devmem_allocation->mm); + +if (completion_done (&devmem_allocation->detached)) +return 0 ; + +if (!err || retry_count--) { +cond_resched (); + goto retry; + } + +return err; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_evict_to_ram); + +/** +* __drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM (internal)
+ * @vas: Pointer to the VM area structure
+ * @device_private_page_owner: Device private pages owner
+ * @page: Pointer to the page for fault handling (can be NULL)
+ * @fault_addr: Fault address
+ * @size: Size of migration
+ *
+ * This internal function performs the migration of the specified
GPU SVM range
+ * to RAM. It sets up the migration, populates + dma maps RAM PFNs,
and
+ * invokes the driver-specific operations for migration to RAM.
+ *
+ * Returns:
+ * 0 on success, negative error code on failure.
+ */
+static int __drm_gpusvm_migrate_to_ram(structvm_area_struct*vas,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ void
*device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +8
-0+ struct page *page, + unsigned long fault_addr, + unsigned long size) +{ + struct migrate_vma migrate = { + .vma = vas, + .pgmap_owner = device_private_page_owner, + .flags =
MIGRATE_VMA_SELECT_DEVICE_PRIVATEQuoted 1 line
drivers/gpu/drm/drm_gpusvm.c new file +17
-0+ MIGRATE_VMA_SELECT_DEVICE_COHERENT, + .fault_page = page, + }; +struct drm_gpusvm_zdd *zdd; +const struct drm_gpusvm_devmem_ops *ops; +struct device *dev; +unsigned long npages, mpages =0 ; +struct page **pages; +dma_addr_t *dma_addr; +unsigned long start, end; +void *buf; +int i, err =0 ; + + start =ALIGN_DOWN (fault_addr, size); + end =ALIGN (fault_addr +1 , size); + +/* Corner where VMA area struct has been partially unmapped
*/drivers/gpu/drm/drm_gpusvm.c new file +10
-0+if (start < vas->vm_start) + start = vas->vm_start; +if (end > vas->vm_end) + end = vas->vm_end; + + migrate.start = start; + migrate.end = end; + npages =npages_in_range (start, end); + + buf = kvcalloc(npages, 2 * sizeof(*migrate.src) +
sizeof(*dma_addr)+drivers/gpu/drm/drm_gpusvm.c new file +7
-0+ sizeof(*pages), GFP_KERNEL); +if (!buf) { + err = -ENOMEM ; + goto err_out; + } + dma_addr = buf + (2 *sizeof (*migrate.src) * npages); + pages = buf + (2 * sizeof(*migrate.src) +
sizeof(*dma_addr))
* npages;drivers/gpu/drm/drm_gpusvm.c new file +16
-0+ + migrate.vma = vas; + migrate.src = buf; + migrate.dst = migrate.src + npages; + + err =migrate_vma_setup (&migrate); +if (err) + goto err_free; + +/* Raced with another CPU fault, nothing to do */ +if (!migrate.cpages) + goto err_free; + +if (!page) { +for (i =0 ; i < npages; ++i) { + if (!(migrate.src[i] &
MIGRATE_PFN_MIGRATE))drivers/gpu/drm/drm_gpusvm.c new file +3
-0+continue ; + + page =
migrate_pfn_to_page(migrate.src[i]);drivers/gpu/drm/drm_gpusvm.c new file +11
-0+break ; + } + +if (!page) + goto err_finalize; + } + zdd = page->zone_device_data; + ops = zdd->devmem_allocation->ops; + dev = zdd->devmem_allocation->dev; + + err = drm_gpusvm_migrate_populate_ram_pfn(vas, page,npages,
&mpages,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ migrate.src,migrate.dst,
drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ start); +if (err) + goto err_finalize; + + err = drm_gpusvm_migrate_map_pages(dev, dma_addr,migrate.dst, npages,
drivers/gpu/drm/drm_gpusvm.c new file +14
-0+ DMA_FROM_DEVICE); +if (err) + goto err_finalize; + +for (i =0 ; i < npages; ++i) + pages[i] =migrate_pfn_to_page (migrate.src[i]); + + err = ops->copy_to_ram (pages, dma_addr, npages); +if (err) + goto err_finalize; + +err_finalize: +if (err) + drm_gpusvm_migration_unlock_put_pages(npages,migrate.dst);
drivers/gpu/drm/drm_gpusvm.c new file +17
-0+migrate_vma_pages (&migrate); +migrate_vma_finalize (&migrate); +drm_gpusvm_migrate_unmap_pages (dev, dma_addr, npages, + DMA_FROM_DEVICE); +err_free: +kvfree (buf); +err_out: + +return err; +} + +/** +* drm_gpusvm_range_evict - Evict GPU SVM range +* @pagemap: Pointer to the GPU SVM structure +* @range: Pointer to the GPU SVM range to be removed +* +* This function evicts the specified GPU SVM range. This function
will not
+ * evict coherent pages.
+ *
+ * Returns:
+ * 0 on success, a negative error code on failure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +4
-0+int drm_gpusvm_range_evict (struct drm_gpusvm *gpusvm, + struct drm_gpusvm_range *range) +{ + struct mmu_interval_notifier *notifier = &range-Quoted 2 lines
notifier-
notifier;drivers/gpu/drm/drm_gpusvm.c new file +9
-0+struct hmm_range hmm_range = { + .default_flags =HMM_PFN_REQ_FAULT , + .notifier = notifier, + .start = range->itree.start, + .end = range->itree.last +1 , + .dev_private_owner =NULL , + }; +unsigned long timeout = + jiffies +
msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);drivers/gpu/drm/drm_gpusvm.c new file +2
-0+unsigned long *pfns; + unsigned long npages = npages_in_range(range-Quoted 1 line
itree.start,
drivers/gpu/drm/drm_gpusvm.c new file +2
-0+ range->itree.last +1);
drivers/gpu/drm/drm_gpusvm.c new file +7
-0+int err =0 ; +struct mm_struct *mm = gpusvm->mm; + +if (!mmget_not_zero (mm)) +return -EFAULT ; + + pfns = kvmalloc_array(npages, sizeof(*pfns),
GFP_KERNEL);drivers/gpu/drm/drm_gpusvm.c new file +6
-0+if (!pfns) +return -ENOMEM ; + + hmm_range.hmm_pfns = pfns; +while (!time_after (jiffies, timeout)) { + hmm_range.notifier_seq =
mmu_interval_read_begin(notifier);drivers/gpu/drm/drm_gpusvm.c new file +21
-0+if (time_after (jiffies, timeout)) { + err = -ETIME ; +break ; + } + +mmap_read_lock (mm); + err =hmm_range_fault (&hmm_range); +mmap_read_unlock (mm); +if (err != -EBUSY ) +break ; + } + +kvfree (pfns); +mmput (mm); + +return err; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_evict); + +/** +* drm_gpusvm_page_free() - Put GPU SVM zone device data associated
with a page
+ * @page: Pointer to the page
+ *
+ * This function is a callback used to put the GPU SVM zone device
data
+ * associated with a page when it is being released.
+ */drivers/gpu/drm/drm_gpusvm.c new file +7
-0+static void drm_gpusvm_page_free (struct page *page) +{ +drm_gpusvm_zdd_put (page->zone_device_data); +} + +/** +* drm_gpusvm_migrate_to_ram() - Migrate GPU SVM range to RAM (page
fault handler)
+ * @vmf: Pointer to the fault information structure
+ *
+ * This function is a page fault handler used to migrate a GPU SVM
range to RAM.
+ * It retrieves the GPU SVM range information from the faulting page
and invokes
+ * the internal migration function to migrate the range back to RAM.
+ *
+ * Returns:
+ *VM_FAULT_SIGBUSon failure, 0 on success.
+ */
+staticvm_fault_tdrm_gpusvm_migrate_to_ram(struct vm_fault *vmf)drivers/gpu/drm/drm_gpusvm.c new file +2
-0+{ + struct drm_gpusvm_zdd *zdd = vmf->page-Quoted 1 line
zone_device_data;drivers/gpu/drm/drm_gpusvm.c new file +4
-0+int err; + + err = __drm_gpusvm_migrate_to_ram(vmf->vma, + zdd-Quoted 1 line
device_private_page_owner,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ vmf->page, vmf-Quoted 1 line
address,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ zdd-Quoted 2 lines
devmem_allocation-
size);drivers/gpu/drm/drm_gpusvm.c new file +6
-0+ +return err ?VM_FAULT_SIGBUS :0 ; +} + +/** +* drm_gpusvm_pagemap_ops() - Device page map operations for GPU SVM
+ */drivers/gpu/drm/drm_gpusvm.c new file +7
-0+static const struct dev_pagemap_ops drm_gpusvm_pagemap_ops = { + .page_free = drm_gpusvm_page_free, + .migrate_to_ram = drm_gpusvm_migrate_to_ram, +}; + +/** +* drm_gpusvm_pagemap_ops_get() - Retrieve GPU SVM device page map
operations
+ *
+ * Returns:
+ * Pointer to the GPU SVM device page map operations structure.
+ */drivers/gpu/drm/drm_gpusvm.c new file +8
-0+const struct dev_pagemap_ops *drm_gpusvm_pagemap_ops_get (void ) +{ +return &drm_gpusvm_pagemap_ops; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_pagemap_ops_get); + +/** +* drm_gpusvm_has_mapping() - Check if GPU SVM has mapping for the
given address range
+ * @gpusvm: Pointer to the GPU SVM structure.
+ * @start: Start address
+ * @end: End address
+ *
+ * Returns:
+ * True if GPU SVM has mapping, False otherwise
+ */
+bool drm_gpusvm_has_mapping(structdrm_gpusvm*gpusvm, unsigned long
start,drivers/gpu/drm/drm_gpusvm.c new file +5
-0+ unsigned long end) +{ +struct drm_gpusvm_notifier *notifier; + + drm_gpusvm_for_each_notifier(notifier, gpusvm, start,end) {
drivers/gpu/drm/drm_gpusvm.c new file +3
-0+struct drm_gpusvm_range *range =NULL ; + + drm_gpusvm_for_each_range(range, notifier,start,
end)drivers/gpu/drm/drm_gpusvm.c new file +9
-0+return true; + } + +return false; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_has_mapping); + +/** +* drm_gpusvm_range_set_unmapped() - Mark a GPU SVM range as unmapped
+ * @range: Pointer to the GPU SVM range structure.
+ * @mmu_range: Pointer to the MMU notifier range structure.
+ *
+ * This function marks a GPU SVM range as unmapped and sets thepartial_unmapflag
+ * if the range partially falls within the provided MMU notifier range.
+ */
+void drm_gpusvm_range_set_unmapped(structdrm_gpusvm_range*range,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ const struct
mmu_notifier_range*mmu_range)drivers/gpu/drm/drm_gpusvm.c new file +2
-0+{ + lockdep_assert_held_write(&range->gpusvm-Quoted 1 line
notifier_lock);drivers/gpu/drm/drm_gpusvm.c new file +10
-0+ + range->flags.unmapped = true; +if (range->itree.start < mmu_range->start || + range->itree.last + 1 > mmu_range->end) + range->flags.partial_unmap = true; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_range_set_unmapped); + +/** +* drm_gpusvm_devmem_init() - Initialize a GPU SVM device memory allocation
+ *
+ * @dev: Pointer to the device structure which device memory allocation belongs to
+ * @mm: Pointer to themm_structfor the address space
+ * @ops: Pointer to the operations structure for GPU SVM device memory
+ * @dpagemap: The structdrm_pagemapwe're allocating from.
+ * @size: Size of device memory allocation
+ */
+void drm_gpusvm_devmem_init(structdrm_gpusvm_devmem*devmem_allocation,drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct device *dev, struct mm_struct*mm,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ const struct drm_gpusvm_devmem_ops*ops,
drivers/gpu/drm/drm_gpusvm.c new file +1
-0+ struct drm_pagemap *dpagemap, size_tsize)
drivers/gpu/drm/drm_gpusvm.c new file +12
-0+{ +init_completion (&devmem_allocation->detached); + devmem_allocation->dev = dev; + devmem_allocation->mm = mm; + devmem_allocation->ops = ops; + devmem_allocation->dpagemap = dpagemap; + devmem_allocation->size = size; +} +EXPORT_SYMBOL_GPL (drm_gpusvm_devmem_init); + +MODULE_DESCRIPTION ("DRM GPUSVM" ); +MODULE_LICENSE ("GPL" );include/drm/drm_gpusvm.h new file +24
-0@@ -0,0 +1,445 @@ +/* SPDX-License-Identifier: GPL-2.0-only OR MIT */ +/* +* Copyright © 2024 Intel Corporation +*/ + +#ifndef __DRM_GPUSVM_H__ +#define __DRM_GPUSVM_H__ + +#include <linux/kref.h> +#include <linux/interval_tree.h> +#include <linux/mmu_notifier.h> + +struct dev_pagemap_ops ; +struct drm_device ; +struct drm_gpusvm ; +struct drm_gpusvm_notifier ; +struct drm_gpusvm_ops ; +struct drm_gpusvm_range ; +struct drm_gpusvm_devmem ; +struct drm_pagemap ; +struct drm_pagemap_dma_addr ; + +/** +* struct drm_gpusvm_devmem_ops - Operations structure for GPU SVM
device memory
+ *
+ * This structure defines the operations for GPU Shared Virtual Memory (SVM)
+ * device memory. These operations are provided by the GPU driver to
manage device memory
+ * allocations and perform operations such as migration between device memory and system
+ * RAM.
+ */include/drm/drm_gpusvm.h new file +3
-0+struct drm_gpusvm_devmem_ops { +/** +* @devmem_release: Release device memory allocation (optional)
include/drm/drm_gpusvm.h new file +3
-0+* @devmem_allocation: device memory allocation +* +* Release device memory allocation and drop a reference to
deviceinclude/drm/drm_gpusvm.h new file +3
-0+* memory allocation. +*/ + void (*devmem_release)(struct drm_gpusvm_devmem
*devmem_allocation);include/drm/drm_gpusvm.h new file +3
-0+ +/** +* @populate_devmem_pfn: Populate device memory PFN (required for migration)
include/drm/drm_gpusvm.h new file +10
-0+* @devmem_allocation: device memory allocation +* @npages: Number of pages to populate +* @pfn: Array of page frame numbers to populate +* +* Populate device memory page frame numbers (PFN). +* +* Returns: +* 0 on success, a negative error code on failure. +*/ + int (*populate_devmem_pfn)(struct drm_gpusvm_devmem
*devmem_allocation,include/drm/drm_gpusvm.h new file +1
-0+ unsigned long npages,unsigned
long *pfn);include/drm/drm_gpusvm.h new file +3
-0+ +/** +* @copy_to_devmem: Copy to device memory (required for migration)
include/drm/drm_gpusvm.h new file +1
-0+* @pages: Pointer to array of device memory pages (destination)
include/drm/drm_gpusvm.h new file +14
-0+* @dma_addr: Pointer to array of DMA addresses (source) +* @npages: Number of pages to copy +* +* Copy pages to device memory. +* +* Returns: +* 0 on success, a negative error code on failure. +*/ +int (*copy_to_devmem)(struct page **pages, + dma_addr_t *dma_addr, + unsigned long npages); + +/** +* @copy_to_ram: Copy to system RAM (required for migration)
include/drm/drm_gpusvm.h new file +1
-0+* @pages: Pointer to array of device memory pages (source)
include/drm/drm_gpusvm.h new file +1
-0+* @dma_addr: Pointer to array of DMA addresses (destination)
include/drm/drm_gpusvm.h new file +14
-0+* @npages: Number of pages to copy +* +* Copy pages to system RAM. +* +* Returns: +* 0 on success, a negative error code on failure. +*/ +int (*copy_to_ram)(struct page **pages, + dma_addr_t *dma_addr, + unsigned long npages); +}; + +/** +* struct drm_gpusvm_devmem - Structure representing a GPU SVM device memory allocation
+ *
+ * @dev: Pointer to the device structure which device memory allocation belongs to
+ * @mm: Pointer to themm_structfor the address space
+ * @detached: device memory allocations is detached from device pages
+ * @ops: Pointer to the operations structure for GPU SVM device memory
+ * @dpagemap: The structdrm_pagemapof the pages this allocation
belongs to.
+ * @size: Size of device memory allocation
+ */include/drm/drm_gpusvm.h new file +13
-0+struct drm_gpusvm_devmem { +struct device *dev; +struct mm_struct *mm; +struct completion detached; +const struct drm_gpusvm_devmem_ops *ops; +struct drm_pagemap *dpagemap; +size_t size; +}; + +/** +* struct drm_gpusvm_ops - Operations structure for GPU SVM +* +* This structure defines the operations for GPU Shared Virtual Memory (SVM).
+ * These operations are provided by the GPU driver to manage SVM ranges and
+ * notifiers.
+ */include/drm/drm_gpusvm.h new file +3
-0+struct drm_gpusvm_ops { +/** +* @notifier_alloc: Allocate a GPU SVM notifier (optional)
include/drm/drm_gpusvm.h new file +5
-0+* +* Allocate a GPU SVM notifier. +* +* Returns: +* Pointer to the allocated GPU SVM notifier on success, NULL on failure.
include/drm/drm_gpusvm.h new file +6
-0+*/ +struct drm_gpusvm_notifier *(*notifier_alloc)(void ); + +/** +* @notifier_free: Free a GPU SVM notifier (optional) +* @notifier: Pointer to the GPU SVM notifier to be freed
include/drm/drm_gpusvm.h new file +4
-0+* +* Free a GPU SVM notifier. +*/ + void (*notifier_free)(struct drm_gpusvm_notifier*notifier);
include/drm/drm_gpusvm.h new file +9
-0+ +/** +* @range_alloc: Allocate a GPU SVM range (optional) +* @gpusvm: Pointer to the GPU SVM +* +* Allocate a GPU SVM range. +* +* Returns: +* Pointer to the allocated GPU SVM range on success, NULL
on failure.include/drm/drm_gpusvm.h new file +2
-0+*/ + struct drm_gpusvm_range *(*range_alloc)(struct
drm_gpusvm
*gpusvm);include/drm/drm_gpusvm.h new file +14
-0+ +/** +* @range_free: Free a GPU SVM range (optional) +* @range: Pointer to the GPU SVM range to be freed +* +* Free a GPU SVM range. +*/ +void (*range_free)(struct drm_gpusvm_range *range); + +/** +* @invalidate: Invalidate GPU SVM notifier (required) +* @gpusvm: Pointer to the GPU SVM +* @notifier: Pointer to the GPU SVM notifier +* @mmu_range: Pointer to the mmu_notifier_range structure
include/drm/drm_gpusvm.h new file +2
-0+* +* Invalidate the GPU page tables. It can safely walk the
notifier rangeinclude/drm/drm_gpusvm.h new file +1
-0+* RB tree/list in this function. Called while holding the
notifier lock.include/drm/drm_gpusvm.h new file +4
-0+*/ +void (*invalidate)(struct drm_gpusvm *gpusvm, + struct drm_gpusvm_notifier *notifier, + const struct mmu_notifier_range
*mmu_range);include/drm/drm_gpusvm.h new file +4
-0+}; + +/** +* struct drm_gpusvm_notifier - Structure representing a GPU SVM notifier
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: MMU interval notifier
+ * @itree: Interval tree node for the notifier (inserted in GPU SVM)
+ * @entry: List entry to fast interval tree traversal
+ * @root: Cached root node of the RB tree containing ranges
+ * @range_list: List head containing of ranges in the same order they appear in
+ * interval tree. This is useful to keep iterating ranges while
+ * doing modifications to RB tree.
+ * @flags.removed: Flag indicating whether the MMU interval notifier
has been
+ * removed
+ *
+ * This structure represents a GPU SVM notifier.
+ */include/drm/drm_gpusvm.h new file +14
-0+struct drm_gpusvm_notifier { +struct drm_gpusvm *gpusvm; +struct mmu_interval_notifier notifier; +struct interval_tree_node itree; +struct list_head entry; +struct rb_root_cached root; +struct list_head range_list; +struct { +u32 removed :1 ; + } flags; +}; + +/** +* struct drm_gpusvm_range - Structure representing a GPU SVM range
+ *
+ * @gpusvm: Pointer to the GPU SVM structure
+ * @notifier: Pointer to the GPU SVM notifier
+ * @refcount: Reference count for the range
+ * @itree: Interval tree node for the range (inserted in GPU SVM notifier)
+ * @entry: List entry to fast interval tree traversal
+ * @notifier_seq: Notifier sequence number of the range's pages
+ * @dma_addr: DMA address array
+ * @dpagemap: The structdrm_pagemapof the device pages we're dma-
mapping.
+ * Note this is assuming only onedrm_pagemapper range
is allowed.
+ * @flags.migrate_devmem: Flag indicating whether the range can be
migrated to device memory
+ * @flags.unmapped: Flag indicating if the range has been unmapped
+ * @flags.partial_unmap: Flag indicating if the range has been partially unmapped
+ * @flags.has_devmem_pages: Flag indicating if the range has devmem
pages
+ * @flags.has_dma_mapping: Flag indicating if the range has a DMA
mapping
+ *
+ * This structure represents a GPU SVM range used for tracking memory ranges
+ * mapped in a DRM device.
+ */include/drm/drm_gpusvm.h new file +13
-0+struct drm_gpusvm_range { +struct drm_gpusvm *gpusvm; +struct drm_gpusvm_notifier *notifier; +struct kref refcount; +struct interval_tree_node itree; +struct list_head entry; +unsigned long notifier_seq; +struct drm_pagemap_dma_addr *dma_addr; +struct drm_pagemap *dpagemap; +struct { +/* All flags below must be set upon creation */ +u16 migrate_devmem :1 ; +/* All flags below must be set / cleared under notifier lock */
include/drm/drm_gpusvm.h new file +19
-0+ u16 unmapped : 1; + u16 partial_unmap : 1; + u16 has_devmem_pages : 1; + u16 has_dma_mapping : 1; + } flags; +}; + +/** +* struct drm_gpusvm - GPU SVM structure +* +* @name: Name of the GPU SVM +* @drm: Pointer to the DRM device structure +* @mm: Pointer to the mm_struct for the address space +* @device_private_page_owner: Device private pages owner +* @mm_start: Start address of GPU SVM +* @mm_range: Range of the GPU SVM +* @notifier_size: Size of individual notifiers +* @ops: Pointer to the operations structure for GPU SVM +* @chunk_sizes: Pointer to the array of chunk sizes used in 30 lines not shown · raw message
include/drm/drm_gpusvm.h new file +17
-0+struct drm_gpusvm { +const char *name; +struct drm_device *drm; +struct mm_struct *mm; +void *device_private_page_owner; +unsigned long mm_start; +unsigned long mm_range; +unsigned long notifier_size; +const struct drm_gpusvm_ops *ops; +const unsigned long *chunk_sizes; +int num_chunks; +struct rw_semaphore notifier_lock; +struct rb_root_cached root; +struct list_head notifier_list; +#ifdef CONFIG_LOCKDEP +/** Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
2 lines not shown · raw message
include/drm/drm_gpusvm.h new file +10
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
12 lines not shown · raw message
include/drm/drm_gpusvm.h new file +10
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +1
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +3
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +11
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +14
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +4
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +3
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +1
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +4
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +1
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +2
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +4
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
2 lines not shown · raw message
include/drm/drm_gpusvm.h new file +4
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
2 lines not shown · raw message
include/drm/drm_gpusvm.h new file +3
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +1
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
2 lines not shown · raw message
include/drm/drm_gpusvm.h new file +2
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +1
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +1
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +1
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +4
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
10 lines not shown · raw message
include/drm/drm_gpusvm.h new file +4
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +1
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
1 line not shown · raw message
include/drm/drm_gpusvm.h new file +5
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Quoted 5 lines
3 lines not shown · raw message
Quoted 8 lines
6 lines not shown · raw message
3 lines not shown · raw message
Parts of this message are not shown here. View the raw message.