cgroups: Add support for pinned device memory
Subsystems (8)
Maarten Lankhorst
maintainercover cgroups: Add support for pinned device memory
When exporting dma-bufs to other devices, even when it is allowed to use move_notify in some drivers, performance will degrade severely when eviction happens.
A perticular example where this can happen is in a multi-card setup, where PCI-E peer-to-peer is used to prevent using access to system memory.
If the buffer is evicted to system memory, not only the evicting GPU wher the buffer resided is affected, but it will also stall the GPU that is waiting on the buffer.
It also makes sense for long running jobs not to be preempted by having its buffers evicted, so it will make sense to have the ability to pin from system memory too.
This is dependant on patches by Dave Airlie, so it's not part of this series yet. But I'm planning on extending pinning to the memory cgroup controller in the future to handle this case.
Implementation details:
For each cgroup up until the root cgroup, the 'min' limit is checked against currently effectively pinned value. If the value will go above 'min', the pinning attempt is rejected.
Pinned memory is handled slightly different and affects calculating effective min/low values. Pinned memory is subtracted from both, and needs to be added afterwards when calculating.
This is because increasing the amount of pinned memory, the amount of free min/low memory decreases for all cgroups that are part of the hierarchy.
Maarten Lankhorst (3):
page_counter: Allow for pinning some amount of memory cgroup/dmem: Implement pinning device memory
drm/xe: Add DRM_XE_GEM_CREATE_FLAG_PINNED flag and implementation
drivers/gpu/drm/xe/xe_bo.c | 66 +++++++++++++++++++++- drivers/gpu/drm/xe/xe_dma_buf.c | 10 +++- include/linux/cgroup_dmem.h | 2 + include/linux/page_counter.h | 8 +++ include/uapi/drm/xe_drm.h | 10 +++- kernel/cgroup/dmem.c | 57 ++++++++++++++++++- mm/page_counter.c | 98 ++++++++++++++++++++++++++++++--- 7 files changed, 237 insertions(+), 14 deletions(-)
Signature · 1 line
2.50.0
Maarten Lankhorst
maintainerRFC 1/3 page_counter: Allow for pinning some amount of memory
Add a pinned member, and use it for implementing pinning accounting. Memory to be pinned has to already be accounted for as normally used, and only memory up to the 'min' limit is allowed to be pinned.
This limit is chosen because cgroups already guarantees that memory up to that limit will not evicted.
Pinned memory affects min and low calculations, so alter those slightly.
--- include/linux/page_counter.h | 8 +++ mm/page_counter.c | 98 +++++++++++++++++++++++++++++++++--- 2 files changed, 99 insertions(+), 7 deletions(-)
include/linux/page_counter.h +8 -0
@@ -13,6 +13,7 @@ struct page_counter {* v2. The memcg->memory.usage is a hot member of struct mem_cgroup. */ atomic_long_t usage; +atomic_long_t pinned;unsigned long failcnt;/* v1-only field */ CACHELINE_PADDING (_pad1_);
@@ -68,11 +69,18 @@ static inline unsigned long page_counter_read(struct page_counter *counter)return atomic_long_read (&counter->usage); } +static inline unsigned long page_counter_pinned (struct page_counter *counter) +{ +return atomic_long_read (&counter->pinned); +} +void page_counter_cancel (struct page_counter *counter,unsigned long nr_pages);void page_counter_charge (struct page_counter *counter,unsigned long nr_pages);bool page_counter_try_charge (struct page_counter *counter,unsigned long nr_pages,struct page_counter **fail); +bool page_counter_try_pin (struct page_counter *counter,unsigned long nr_pages); +void page_counter_unpin (struct page_counter *counter,unsigned long nr_pages);void page_counter_uncharge (struct page_counter *counter,unsigned long nr_pages);void page_counter_set_min (struct page_counter *counter,unsigned long nr_pages);void page_counter_set_low (struct page_counter *counter,unsigned long nr_pages);
mm/page_counter.c +91 -7
127 lines not on this page · raw message
Signature · 1 line
2.50.0
Maarten Lankhorst
maintainerRFC 2/3 cgroup/dmem: Implement pinning device memory
In reply to Maarten LankhorstAdd a function to pin, and to unipn memory and adjust the calculations in dmem_cgroup_state_evict_valuable().
--- include/linux/cgroup_dmem.h | 2 ++ kernel/cgroup/dmem.c | 57 +++++++++++++++++++++++++++++++++++-- 2 files changed, 56 insertions(+), 3 deletions(-)
include/linux/cgroup_dmem.h +2 -0
@@ -21,6 +21,8 @@ int dmem_cgroup_try_charge(struct dmem_cgroup_region *region, u64 size,struct dmem_cgroup_pool_state **ret_pool, struct dmem_cgroup_pool_state **ret_limit_pool);void dmem_cgroup_uncharge (struct dmem_cgroup_pool_state *pool,u64 size); +int dmem_cgroup_try_pin (struct dmem_cgroup_pool_state *pool,u64 size); +void dmem_cgroup_unpin (struct dmem_cgroup_pool_state *pool,u64 size);bool dmem_cgroup_state_evict_valuable (struct dmem_cgroup_pool_state *limit_pool,struct dmem_cgroup_pool_state *test_pool,bool ignore_low,bool *ret_hit_low);
kernel/cgroup/dmem.c +54 -3
101 lines not on this page · raw message
Signature · 1 line
2.50.0
Maarten Lankhorst
maintainerRFC 3/3 drm/xe: Add DRM_XE_GEM_CREATE_FLAG_PINNED flag and implementation
In reply to Maarten LankhorstAdd an option to pin memory through the science of cgroup accounting. A bo will be pinned for its entire lifetime, and this allows buffers that are pinned for dma-buf export without requiring the pinning to be done at the dma-buf layer for all devices.
For now only implement VRAM pinning. Dave Airlie has a series to implement memcg accounting for the GPU but that is not ready yet.
--- drivers/gpu/drm/xe/xe_bo.c | 66 ++++++++++++++++++++++++++++++++- drivers/gpu/drm/xe/xe_dma_buf.c | 10 ++++- include/uapi/drm/xe_drm.h | 10 ++++- 3 files changed, 82 insertions(+), 4 deletions(-)
drivers/gpu/drm/xe/xe_bo.c +64 -2
@@ -5,6 +5,7 @@#include "xe_bo.h" +#include <linux/cgroup_dmem.h> #include <linux/dma-buf.h> #include <linux/nospec.h>
@@ -208,7 +209,8 @@ static bool force_contiguous(u32 bo_flags)* must be contiguous, also only contiguous BOs support xe_bo_vmap. */ return bo_flags &XE_BO_FLAG_NEEDS_CPU_ACCESS && - bo_flags &XE_BO_FLAG_PINNED ; + bo_flags &XE_BO_FLAG_PINNED && + !(bo_flags &XE_BO_FLAG_USER ); } static void add_vram(struct xe_device *xe, struct xe_bo *bo,
@@ -1697,6 +1699,16 @@ static void xe_gem_object_free(struct drm_gem_object *obj)ttm_bo_put (container_of(obj ,struct ttm_buffer_object ,base )); } +static void xe_bo_unpin_user (struct xe_bo *bo) +{ +xe_bo_unpin_external (bo); + +if (bo->flags &XE_BO_FLAG_SYSTEM ) +WARN_ON (1 ); +else +dmem_cgroup_unpin (bo->ttm.resource->css,xe_bo_size (bo)); +} +static void xe_gem_object_close (struct drm_gem_object *obj,struct drm_file *file_priv) {
@@ -1708,6 +1720,10 @@ static void xe_gem_object_close(struct drm_gem_object *obj,xe_bo_lock (bo, false);ttm_bo_set_bulk_move (&bo->ttm,NULL );xe_bo_unlock (bo); + }else if (bo->flags &XE_BO_FLAG_PINNED ) { +xe_bo_lock (bo, false); +xe_bo_unpin_user (bo); +xe_bo_unlock (bo); } }
@@ -2128,8 +2144,27 @@ struct xe_bo *xe_bo_create_user(struct xe_device *xe, struct xe_tile *tile,struct xe_bo *bo =__xe_bo_create_locked (xe, tile, vm, size,0 , ~0ULL , cpu_caching, ttm_bo_type_device, flags |XE_BO_FLAG_USER ,0 ); -if (!IS_ERR (bo)) +if (!IS_ERR (bo)) { +int ret =0 ; + +if (bo->flags &XE_BO_FLAG_PINNED ) { +if (bo->flags &XE_BO_FLAG_SYSTEM ) { + ret = -ENOSYS ;// TODO + }else { + ret =dmem_cgroup_try_pin (bo->ttm.resource->css, size); + } +if (!ret) + ret =xe_bo_pin_external (bo); +else if (ret == -EAGAIN ) + ret = -ENOSPC ; + } +xe_bo_unlock_vm_held (bo); +if (ret) { +xe_bo_put (bo); +return ERR_PTR (ret); + } + }return bo; }
@@ -2745,6 +2780,28 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data, args->cpu_caching == DRM_XE_GEM_CPU_CACHING_WB))return -EINVAL ; +if (XE_IOCTL_DBG (xe, args->flags &DRM_XE_GEM_CREATE_FLAG_PINNED )) { +bool pinned_flag = true; +/* Only allow a single placement for pinning */ +if (XE_IOCTL_DBG (xe, pinned_flag &&hweight32 (args->placement) !=1 )) +return -EINVAL ; + +/* Meant for exporting, do not allow a VM-local BO */ +if (XE_IOCTL_DBG (xe, pinned_flag && args->vm_id)) +return -EINVAL ; + +/* Similarly, force fail at creation time for now. We may relax this requirement later */ +if (XE_IOCTL_DBG (xe, pinned_flag && args->flags &DRM_XE_GEM_CREATE_FLAG_DEFER_BACKING )) +return -EINVAL ; + +/* Require the appropriate cgroups to be enabled. */ +if (XE_IOCTL_DBG (xe, pinned_flag && !IS_ENABLED (CONFIG_CGROUP_DMEM ) && bo_flags &XE_BO_FLAG_VRAM_MASK ) || +XE_IOCTL_DBG (xe, pinned_flag && !IS_ENABLED (CONFIG_MEMCG ) && bo_flags &XE_BO_FLAG_SYSTEM )) +return -EINVAL ; + + bo_flags |=XE_BO_FLAG_PINNED ; + } +if (args->vm_id) { vm =xe_vm_lookup (xef, args->vm_id);if (XE_IOCTL_DBG (xe, !vm))
@@ -2790,6 +2847,11 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,__xe_bo_unset_bulk_move (bo);xe_vm_unlock (vm); } +if (bo->flags &XE_BO_FLAG_PINNED ) { +xe_bo_lock (bo, false); +xe_bo_unpin_user (bo); +xe_bo_unlock (bo); + } out_put:xe_bo_put (bo); out_vm:
drivers/gpu/drm/xe/xe_dma_buf.c +9 -1
@@ -53,6 +53,11 @@ static int xe_dma_buf_pin(struct dma_buf_attachment *attach)struct xe_device *xe =xe_bo_device (bo);int ret; +if (bo->flags &XE_BO_FLAG_PINNED ) { +ttm_bo_pin (&bo->ttm); +return 0 ; + } +/* * For now only support pinning in TT memory, for two reasons: * 1) Avoid pinning in a placement not accessible to some importers.
@@ -83,7 +88,10 @@ static void xe_dma_buf_unpin(struct dma_buf_attachment *attach)struct drm_gem_object *obj = attach->dmabuf->priv;struct xe_bo *bo =gem_to_xe_bo (obj); -xe_bo_unpin_external (bo); +if (bo->flags &XE_BO_FLAG_PINNED ) +ttm_bo_unpin (&bo->ttm); +else +xe_bo_unpin_external (bo); } static struct sg_table *xe_dma_buf_map(struct dma_buf_attachment *attach,
include/uapi/drm/xe_drm.h +9 -1
@@ -765,12 +765,15 @@ struct drm_xe_device_query {* until the object is either bound to a virtual memory region via * VM_BIND or accessed by the CPU. As a result, no backing memory is * reserved at the time of GEM object creation. -* - %DRM_XE_GEM_CREATE_FLAG_SCANOUT +* - %DRM_XE_GEM_CREATE_FLAG_SCANOUT - GEM object will be used +* display framebuffer. * - %DRM_XE_GEM_CREATE_FLAG_NEEDS_VISIBLE_VRAM - When using VRAM as a * possible placement, ensure that the corresponding VRAM allocation * will always use the CPU accessible part of VRAM. This is important * for small-bar systems (on full-bar systems this gets turned into a * noop). +* - %DRM_XE_GEM_CREATE_FLAG_PINNED - Pin the backing memory permanently +* on allocation, if withing cgroups limits. * Note1: System memory can be used as an extra placement if the kernel * should spill the allocation to system memory, if space can't be made * available in the CPU accessible part of VRAM (giving the same
@@ -781,6 +784,10 @@ struct drm_xe_device_query {* need to use VRAM for display surfaces, therefore the kernel requires * setting this flag for such objects, otherwise an error is thrown on * small-bar systems. +* Note3: %DRM_XE_GEM_CREATE_FLAG_PINNED requires the BO to have only +* a single placement, no vm_id, requires (device) memory cgroups enabled, +* and is incompatible with the %DEFER_BACKING and %NEEDS_VISIBLE_VRAM +* flags. * * @cpu_caching supports the following values: * - %DRM_XE_GEM_CPU_CACHING_WB - Allocate the pages with write-back
@@ -827,6 +834,7 @@ struct drm_xe_gem_create {#define DRM_XE_GEM_CREATE_FLAG_DEFER_BACKING (1 << 0)#define DRM_XE_GEM_CREATE_FLAG_SCANOUT (1 << 1)#define DRM_XE_GEM_CREATE_FLAG_NEEDS_VISIBLE_VRAM (1 << 2) +#define DRM_XE_GEM_CREATE_FLAG_PINNED (1 << 3)/** * @flags: Flags, currently a mask of memory instances of where BO can * be placed
Signature · 1 line
2.50.0
Thomas Hellström
maintainerRe: [RFC 3/3] drm/xe: Add DRM_XE_GEM_CREATE_FLAG_PINNED flag and implementation
Hi, Maarten,
Quoted 9 lines · Maarten Lankhorst
Add an option to pin memory through the science of cgroup accounting. A bo will be pinned for its entire lifetime, and this allows buffers that are pinned for dma-buf export without requiring the pinning to be
done at the dma-buf layer for all devices.For now only implement VRAM pinning. Dave Airlie has a series to implement
memcg accounting for the GPU but that is not ready yet.
Previous discussions around this have favoured a UAPI where we pin a gpu-vm range, with a pin at mapping time, or dma-buf pin time where required, this allows for dynamic pinning and unpinning, and would avoid having separate pinning interfaces for bos and userptr.
In particular if we don't know at bo creation time which buffer objects will be exported with a method requiring pinning, how would UMD deduce what buffer objects to pin?
Thanks,
Thomas
Quoted 240 lines
Signed-off-by Maarten Lankhorst <[email protected]>---
drivers/gpu/drm/xe/xe_bo.c | 66
++++++++++++++++++++++++++++++++-
drivers/gpu/drm/xe/xe_dma_buf.c | 10 ++++-
include/uapi/drm/xe_drm.h | 10 ++++-3 files changed, 82 insertions(+), 4 deletions(-)drivers/gpu/drm/xe/xe_bo.c +0
-0@@ -5,6 +5,7 @@
#include "xe_bo.h"
+#include <linux/cgroup_dmem.h>
#include <linux/dma-buf.h>
#include <linux/nospec.h>drivers/gpu/drm/xe/xe_bo.c +0
-0@@ -208,7 +209,8 @@ static bool force_contiguous(u32 bo_flags) * must be contiguous, also only contiguous BOs support
xe_bo_vmap.
*/
returnbo_flags&XE_BO_FLAG_NEEDS_CPU_ACCESS&&drivers/gpu/drm/xe/xe_bo.c +2
-1- bo_flags & XE_BO_FLAG_PINNED; + bo_flags & XE_BO_FLAG_PINNED && + !(bo_flags & XE_BO_FLAG_USER);}
static void add_vram(structxe_device*xe, structxe_bo*bo,drivers/gpu/drm/xe/xe_bo.c +0
-0@@ -1697,6 +1699,16 @@ static void xe_gem_object_free(struct drm_gem_object *obj) ttm_bo_put(container_of(obj, struct ttm_buffer_object,base )); }drivers/gpu/drm/xe/xe_bo.c +8
-0+static void xe_bo_unpin_user (struct xe_bo *bo) +{ +xe_bo_unpin_external (bo); + +if (bo->flags &XE_BO_FLAG_SYSTEM ) +WARN_ON (1 ); +else + dmem_cgroup_unpin(bo->ttm.resource->css,
xe_bo_size(bo));drivers/gpu/drm/xe/xe_bo.c +2
-0+} +static void
xe_gem_object_close(struct drm_gem_object *obj, struct drm_file *file_priv){drivers/gpu/drm/xe/xe_bo.c +4
-0@@ -1708,6 +1720,10 @@ static void xe_gem_object_close(structdrm_gem_object *obj, xe_bo_lock(bo, false); ttm_bo_set_bulk_move(&bo->ttm, NULL); xe_bo_unlock(bo); + }else if (bo->flags &XE_BO_FLAG_PINNED ) { +xe_bo_lock (bo, false); +xe_bo_unpin_user (bo); +xe_bo_unlock (bo); } }drivers/gpu/drm/xe/xe_bo.c +0
-0@@ -2128,8 +2144,27 @@ struct xe_bo *xe_bo_create_user(struct xe_device *xe, struct xe_tile *tile, struct xe_bo *bo = __xe_bo_create_locked(xe, tile, vm, size, 0, ~0ULL, cpu_caching, ttm_bo_type_device,flags |
XE_BO_FLAG_USER, 0);drivers/gpu/drm/xe/xe_bo.c +8
-1-if (!IS_ERR (bo)) +if (!IS_ERR (bo)) { +int ret =0 ; + +if (bo->flags &XE_BO_FLAG_PINNED ) { +if (bo->flags &XE_BO_FLAG_SYSTEM ) { + ret = -ENOSYS ;// TODO + }else { + ret = dmem_cgroup_try_pin(bo-Quoted 1 line
ttm.resource->css, size);drivers/gpu/drm/xe/xe_bo.c +7
-0+ } +if (!ret) + ret =xe_bo_pin_external (bo); +else if (ret == -EAGAIN ) + ret = -ENOSPC ; + } +
xe_bo_unlock_vm_held(bo);drivers/gpu/drm/xe/xe_bo.c +5
-0+if (ret) { +xe_bo_put (bo); +return ERR_PTR (ret); + } + }
return bo;
}drivers/gpu/drm/xe/xe_bo.c +0
-0@@ -2745,6 +2780,28 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
args->cpu_caching==DRM_XE_GEM_CPU_CACHING_WB))
return-EINVAL;drivers/gpu/drm/xe/xe_bo.c +1
-0+ if (XE_IOCTL_DBG(xe, args->flags &
DRM_XE_GEM_CREATE_FLAG_PINNED)) {drivers/gpu/drm/xe/xe_bo.c +3
-0+bool pinned_flag = true; +/* Only allow a single placement for pinning */ + if (XE_IOCTL_DBG(xe, pinned_flag && hweight32(args-Quoted 1 line
placement) != 1))
drivers/gpu/drm/xe/xe_bo.c +3
-0+return -EINVAL ; + +/* Meant for exporting, do not allow a VM-local BO */
drivers/gpu/drm/xe/xe_bo.c +4
-0+if (XE_IOCTL_DBG (xe, pinned_flag && args->vm_id)) +return -EINVAL ; + +/* Similarly, force fail at creation time for now. We may relax this requirement later */
drivers/gpu/drm/xe/xe_bo.c +1
-0+ if (XE_IOCTL_DBG(xe, pinned_flag && args->flags &
DRM_XE_GEM_CREATE_FLAG_DEFER_BACKING))drivers/gpu/drm/xe/xe_bo.c +4
-0+return -EINVAL ; + +/* Require the appropriate cgroups to be enabled. */ + if (XE_IOCTL_DBG(xe, pinned_flag &&!IS_ENABLED(CONFIG_CGROUP_DMEM) &&
bo_flags&XE_BO_FLAG_VRAM_MASK) ||drivers/gpu/drm/xe/xe_bo.c +1
-0+ XE_IOCTL_DBG(xe, pinned_flag &&!IS_ENABLED(CONFIG_MEMCG) &&
bo_flags&XE_BO_FLAG_SYSTEM))drivers/gpu/drm/xe/xe_bo.c +5
-0+return -EINVAL ; + + bo_flags |=XE_BO_FLAG_PINNED ; + } +if (
args->vm_id) {
vm =xe_vm_lookup(xef, args->vm_id); if (XE_IOCTL_DBG(xe, !vm))drivers/gpu/drm/xe/xe_bo.c +5
-0@@ -2790,6 +2847,11 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data, __xe_bo_unset_bulk_move(bo); xe_vm_unlock(vm); } +if (bo->flags &XE_BO_FLAG_PINNED ) { +xe_bo_lock (bo, false); +xe_bo_unpin_user (bo); +xe_bo_unlock (bo); + }out_put:
xe_bo_put(bo);
out_vm:drivers/gpu/drm/xe/xe_dma_buf.c +0
-0@@ -53,6 +53,11 @@ static int xe_dma_buf_pin(struct dma_buf_attachment *attach) struct xe_device *xe = xe_bo_device(bo); int ret;drivers/gpu/drm/xe/xe_dma_buf.c +5
-0+if (bo->flags &XE_BO_FLAG_PINNED ) { +ttm_bo_pin (&bo->ttm); +return 0 ; + } +/*
* For now only support pinning in TT memory, for two reasons:
* 1) Avoid pinning in a placement not accessible to some importers.drivers/gpu/drm/xe/xe_dma_buf.c +0
-0@@ -83,7 +88,10 @@ static void xe_dma_buf_unpin(struct dma_buf_attachment *attach) struct drm_gem_object *obj = attach->dmabuf->priv; struct xe_bo *bo = gem_to_xe_bo(obj);drivers/gpu/drm/xe/xe_dma_buf.c +4
-1-xe_bo_unpin_external (bo); +if (bo->flags &XE_BO_FLAG_PINNED ) +ttm_bo_unpin (&bo->ttm); +else +xe_bo_unpin_external (bo);}
static structsg_table*xe_dma_buf_map(structdma_buf_attachment*attach,include/uapi/drm/xe_drm.h +0
-0@@ -765,12 +765,15 @@ struct drm_xe_device_query { * until the object is either bound to a virtual memory regionvia
*VM_BINDor accessed by the CPU. As a result, no backing memory is
* reserved at the time of GEM object creation.
- * - %DRM_XE_GEM_CREATE_FLAG_SCANOUT
+ * - %DRM_XE_GEM_CREATE_FLAG_SCANOUT - GEM object will be used
+ * display framebuffer.
* - %DRM_XE_GEM_CREATE_FLAG_NEEDS_VISIBLE_VRAM - When using VRAM as a
* possible placement, ensure that the corresponding VRAM allocation
* will always use the CPU accessible part of VRAM. This is important
* for small-bar systems (on full-bar systems this gets turned into a
* noop).
+ * - %DRM_XE_GEM_CREATE_FLAG_PINNED - Pin the backing memory permanently
+ * on allocation, if withing cgroups limits.
* Note1: System memory can be used as an extra placement if the kernel
* should spill the allocation to system memory, if space can't be made
* available in the CPU accessible part of VRAM (giving the sameinclude/uapi/drm/xe_drm.h +0
-0@@ -781,6 +784,10 @@ struct drm_xe_device_query { * need to use VRAM for display surfaces, therefore the kernelrequires
* setting this flag for such objects, otherwise an error is thrown on
* small-bar systems.
+ * Note3: %DRM_XE_GEM_CREATE_FLAG_PINNED requires the BO to have only
+ * a single placement, novm_id, requires (device) memory cgroups enabled,
+ * and is incompatible with the %DEFER_BACKING and %NEEDS_VISIBLE_VRAM
+ * flags.
*
* @cpu_caching supports the following values:
* - %DRM_XE_GEM_CPU_CACHING_WB - Allocate the pages with write- backinclude/uapi/drm/xe_drm.h +1
-0@@ -827,6 +834,7 @@ struct drm_xe_gem_create { #define DRM_XE_GEM_CREATE_FLAG_DEFER_BACKING (1 << 0) #define DRM_XE_GEM_CREATE_FLAG_SCANOUT (1 << 1) #define DRM_XE_GEM_CREATE_FLAG_NEEDS_VISIBLE_VRAM (1 << 2) +#define DRM_XE_GEM_CREATE_FLAG_PINNED (1 << 3) /*** @flags: Flags, currently a mask of memory instances of where BO can
* be placed
Maarten Lankhorst
maintainerRe: [RFC 3/3] drm/xe: Add DRM_XE_GEM_CREATE_FLAG_PINNED flag and implementation
Hey,
Den 2025-08-19 kl. 18:22, skrev Thomas Hellström:
Quoted 18 lines
Hi, Maarten,
Quoted 9 lines · Maarten Lankhorst
Add an option to pin memory through the science of cgroup accounting. A bo will be pinned for its entire lifetime, and this allows buffers that are pinned for dma-buf export without requiring the pinning to be
done at the dma-buf layer for all devices.For now only implement VRAM pinning. Dave Airlie has a series to implement
memcg accounting for the GPU but that is not ready yet.Previous discussions around this have favoured a UAPI where we pin a gpu-vm range, with a pin at mapping time, or dma-buf pin time where required, this allows for dynamic pinning and unpinning, and would avoid having separate pinning interfaces for bos and userptr.
Quoted 7 lines
In particular if we don't know at bo creation time which buffer objects will be exported with a method requiring pinning, how would UMD deduce what buffer objects to pin?
Thanks,
Thomas
For discussion purposes, it seems compute preferred pinning at allocation time,
so I wanted to propose that solution since it's easiest to implement.
I will change it for the next version, but I'm very interested in feedback on the adjustments I made in cgroups. Those are the ones that will have the most impact, and changes how effective min/low is calculated when memory pinned is involved.
Kind regards,
~Maarten
kernel test robot
build botRe: [RFC 3/3] drm/xe: Add DRM_XE_GEM_CREATE_FLAG_PINNED flag and implementation
In reply to Maarten Lankhorstkernel test robot noticed the following build errors:
Hi Maarten,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on drm-xe/drm-xe-next]
[also build test ERROR on next-20250819]
[cannot apply to tj-cgroup/for-next akpm-mm/mm-everything linus/master v6.17-rc2] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Maarten-Lankhorst/page_counter-Allow-for-pinning-some-amount-of-memory/20250819-195117 base: https://gitlab.freedesktop.org/drm/xe/kernel.git drm-xe-next patch link: https://lore.kernel.org/r/20250819114932.597600-8-dev%40lankhorst.se patch subject: [RFC 3/3] drm/xe: Add DRM_XE_GEM_CREATE_FLAG_PINNED flag and implementation config: i386-buildonly-randconfig-003-20250820 (https://download.01.org/0day-ci/archive/20250820/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14+deb12u1) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250820/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
drivers/gpu/drm/xe/xe_bo.c: In function 'xe_bo_unpin_user':
>> drivers/gpu/drm/xe/xe_bo.c:1709:17: error: implicit declaration of function 'dmem_cgroup_unpin'; did you mean 'mem_cgroup_init'? [-Werror=implicit-function-declaration]
1709 | dmem_cgroup_unpin(bo->ttm.resource->css, xe_bo_size(bo));
| ^~~~~~~~~~~~~~~~~
| mem_cgroup_init
drivers/gpu/drm/xe/xe_bo.c: In function 'xe_bo_create_user':
>> drivers/gpu/drm/xe/xe_bo.c:2154:39: error: implicit declaration of function 'dmem_cgroup_try_pin'; did you mean 'mem_cgroup_tryget'? [-Werror=implicit-function-declaration]
2154 | ret = dmem_cgroup_try_pin(bo->ttm.resource->css, size);
| ^~~~~~~~~~~~~~~~~~~
| mem_cgroup_trygetcc1: some warnings being treated as errors
vim +1709 drivers/gpu/drm/xe/xe_bo.c
1701
1702 static void xe_bo_unpin_user(struct xe_bo *bo)
1703 {
1704 xe_bo_unpin_external(bo);
1705
1706 if (bo->flags & XE_BO_FLAG_SYSTEM)
1707 WARN_ON(1);
1708 else
> 1709 dmem_cgroup_unpin(bo->ttm.resource->css, xe_bo_size(bo));
1710 }
1711
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Patchwork
CI✗ CI.checkpatch: warning for cgroups: Add support for pinned device memory
In reply to Maarten Lankhorst== Series Details ==
== Series Details ==
Series: cgroups: Add support for pinned device memory
URL : https://patchwork.freedesktop.org/series/153134/
State : warning
== Summary ==
+ KERNEL=/kernel
+ git clone https://gitlab.freedesktop.org/drm/maintainer-tools mt Cloning into 'mt'...
warning: redirecting to https://gitlab.freedesktop.org/drm/maintainer-tools.git/
+ git -C mt rev-list -n1 origin/master 553439844b6500767ce8aef522cfe9fbb7ece541
+ cd /kernel
+ git config --global --add safe.directory /kernel
+ git log -n1
commit 6adbd62f5794fdbd7fcbd26b53705d8e75f5b88b
Author: Maarten Lankhorst <[email protected]> Date: Tue Aug 19 13:49:36 2025 +0200
drm/xe: Add DRM_XE_GEM_CREATE_FLAG_PINNED flag and implementation
Add an option to pin memory through the science of cgroup accounting. A bo will be pinned for its entire lifetime, and this allows buffers that are pinned for dma-buf export without requiring the pinning to be done at the dma-buf layer for all devices.
For now only implement VRAM pinning. Dave Airlie has a series to implement memcg accounting for the GPU but that is not ready yet.
+ /mt/dim checkpatch 8093b67375779a975c13b9a23c8fcc25cb1e9c83 drm-intel d77dfce28451 page_counter: Allow for pinning some amount of memory -:77: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis #77: FILE: mm/page_counter.c:208:
+bool page_counter_try_pin(struct page_counter *counter,
+ unsigned long nr_pages)
-:107: WARNING:DATA_RACE: data_race without comment #107: FILE: mm/page_counter.c:238:
+ data_race(c->failcnt++);
total: 0 errors, 1 warnings, 1 checks, 152 lines checked33d0ce53c33f cgroup/dmem: Implement pinning device memory
-:97: WARNING:TYPO_SPELLING: 'succesfully' may be misspelled - perhaps 'successfully'? #97: FILE: kernel/cgroup/dmem.c:674:
+ * If the function succeeds, the memory is succesfully accounted as being pinned.
^^^^^^^^^^^
-:106: CHECK:BRACES: Blank lines aren't necessary before a close brace '}' #106: FILE: kernel/cgroup/dmem.c:683:
+
+}
total: 0 errors, 1 warnings, 1 checks, 109 lines checked 6adbd62f5794 drm/xe: Add DRM_XE_GEM_CREATE_FLAG_PINNED flag and implementation -:76: WARNING:BRACES: braces {} are not necessary for any arm of this statement #76: FILE: drivers/gpu/drm/xe/xe_bo.c:2151:
+ if (bo->flags & XE_BO_FLAG_SYSTEM) {
[...]
+ } else {
[...]
-:77: WARNING:ENOSYS: ENOSYS means 'invalid syscall nr' and nothing else #77: FILE: drivers/gpu/drm/xe/xe_bo.c:2152:
+ ret = -ENOSYS; // TODO
-:110: WARNING:LONG_LINE_COMMENT: line length of 105 exceeds 100 columns #110: FILE: drivers/gpu/drm/xe/xe_bo.c:2793:
+ /* Similarly, force fail at creation time for now. We may relax this requirement later */
-:111: WARNING:LONG_LINE: line length of 104 exceeds 100 columns #111: FILE: drivers/gpu/drm/xe/xe_bo.c:2794:
+ if (XE_IOCTL_DBG(xe, pinned_flag && args->flags & DRM_XE_GEM_CREATE_FLAG_DEFER_BACKING))
-:115: WARNING:LONG_LINE: line length of 122 exceeds 100 columns #115: FILE: drivers/gpu/drm/xe/xe_bo.c:2798:
+ if (XE_IOCTL_DBG(xe, pinned_flag && !IS_ENABLED(CONFIG_CGROUP_DMEM) && bo_flags & XE_BO_FLAG_VRAM_MASK) ||
-:116: WARNING:LONG_LINE: line length of 111 exceeds 100 columns #116: FILE: drivers/gpu/drm/xe/xe_bo.c:2799:
+ XE_IOCTL_DBG(xe, pinned_flag && !IS_ENABLED(CONFIG_MEMCG) && bo_flags & XE_BO_FLAG_SYSTEM))
-:182: WARNING:TYPO_SPELLING: 'withing' may be misspelled - perhaps 'within'? #182: FILE: include/uapi/drm/xe_drm.h:776:
+ * on allocation, if withing cgroups limits.
^^^^^^^
total: 0 errors, 7 warnings, 0 checks, 164 lines checked
Patchwork
CI✗ CI.KUnit: failure for cgroups: Add support for pinned device memory
In reply to Maarten Lankhorst== Series Details ==
== Series Details ==
Series: cgroups: Add support for pinned device memory
URL : https://patchwork.freedesktop.org/series/153134/
State : failure
== Summary ==
+ trap cleanup EXIT
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/xe/.kunitconfig
ERROR:root:../drivers/gpu/drm/xe/xe_bo.c: In function ‘xe_bo_unpin_user’:../drivers/gpu/drm/xe/xe_bo.c:1709:17: error: implicit declaration of function ‘dmem_cgroup_unpin’; did you mean ‘mem_cgroup_init’? [-Werror=implicit-function-declaration]
1709 | dmem_cgroup_unpin(bo->ttm.resource->css, xe_bo_size(bo));
| ^~~~~~~~~~~~~~~~~
| mem_cgroup_init
../drivers/gpu/drm/xe/xe_bo.c: In function ‘xe_bo_create_user’:
../drivers/gpu/drm/xe/xe_bo.c:2154:39: error: implicit declaration of function ‘dmem_cgroup_try_pin’; did you mean ‘mem_cgroup_tryget’? [-Werror=implicit-function-declaration]
2154 | ret = dmem_cgroup_try_pin(bo->ttm.resource->css, size);
| ^~~~~~~~~~~~~~~~~~~
| mem_cgroup_trygetcc1: some warnings being treated as errors
make[7]: *** [../scripts/Makefile.build:287: drivers/gpu/drm/xe/xe_bo.o] Error 1 make[7]: *** Waiting for unfinished jobs....
make[6]: *** [../scripts/Makefile.build:556: drivers/gpu/drm/xe] Error 2 make[6]: *** Waiting for unfinished jobs....
make[5]: *** [../scripts/Makefile.build:556: drivers/gpu/drm] Error 2 make[4]: *** [../scripts/Makefile.build:556: drivers/gpu] Error 2 make[4]: *** Waiting for unfinished jobs....
make[3]: *** [../scripts/Makefile.build:556: drivers] Error 2 make[3]: *** Waiting for unfinished jobs....
make[2]: *** [/kernel/Makefile:2011: .] Error 2
make[1]: *** [/kernel/Makefile:248: __sub-make] Error 2
make: *** [Makefile:248: __sub-make] Error 2
[16:35:23] Configuring KUnit Kernel ...
Generating .config ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig [16:35:27] Building KUnit Kernel ... Populating config with: $ make ARCH=um O=.kunit olddefconfig Building with: $ make all compile_commands.json scripts_gdb ARCH=um O=.kunit --jobs=48 + cleanup ++ stat -c %u:%g /kernel + chown -R 1003:1003 /kernel
Michal Koutný
maintainerHello Maarten.
Quoted 5 lines · Maarten Lankhorst
Implementation details:
For each cgroup up until the root cgroup, the 'min' limit is checked against currently effectively pinned value. If the value will go above 'min', the pinning attempt is rejected.
How is pinning different from setting a 'min' limit (from a user perspective)?
Quoted 8 lines
Pinned memory is handled slightly different and affects calculating effective min/low values. Pinned memory is subtracted from both, and needs to be added afterwards when calculating.
This is because increasing the amount of pinned memory, the amount of free min/low memory decreases for all cgroups that are part of the hierarchy.
What is supposed to happen with pinned memory after cgroup removal? I find the page_counter changes little bit complex without understanding of the difference between min and pinned. Should this be conceptually similar to memory.stat:unevictable? Or rather mlock(2)? So far neither of those needed interaction with min/low values (in memcg).
Thanks,
Michal
Maarten Lankhorst
maintainerHey,
Den 2025-08-26 kl. 16:20, skrev Michal Koutný:
Quoted 11 lines
Hello Maarten.
Quoted 5 lines · Maarten Lankhorst
Implementation details:
For each cgroup up until the root cgroup, the 'min' limit is checked against currently effectively pinned value. If the value will go above 'min', the pinning attempt is rejected.
How is pinning different from setting a 'min' limit (from a user perspective)?
It's related, in fact you have to set the 'min' limit first.
The 'pinned' allows you to pick /which/ memory falls under the 'min' limit.
Quoted 10 lines
Quoted 8 lines
Pinned memory is handled slightly different and affects calculating effective min/low values. Pinned memory is subtracted from both, and needs to be added afterwards when calculating.
This is because increasing the amount of pinned memory, the amount of free min/low memory decreases for all cgroups that are part of the hierarchy.
What is supposed to happen with pinned memory after cgroup removal?
I think for accounting purposes pinned memory stays pinned, otherwise the idea of pinning is lost. However when you kill all processes in the cgroup, that should solve itself eventually.
Quoted 4 lines
I find the
page_counterchanges little bit complex without understanding of the difference between min and pinned. Should this be conceptually similar to memory.stat:unevictable? Or rather mlock(2)? So far neither of those needed interaction with min/low values (in memcg).
You could in theory implement mlockall using the 'min' values too.
The page counter changes implement the following:
Lets say you have this tree with 'min' values.
/ '5' A
X'6' -- '5' B
\ '5' CEffective min without pinned pages:
/ '2' A
X'6' -- '2' B
\ '2' CNow 'B' pins 3 pages:
Effective min:
/ '1' A
X'3+3p' -- '1' B (1 + 3 pinned pages makes effective min 4)
\ '1' CSame for applies to effective 'low' calculations.
Kind regards,
~Maarten
Quoted 21 lines · Maarten Lankhorst
When exporting dma-bufs to other devices, even when it is allowed to use
move_notifyin some drivers, performance will degrade severely when eviction happens.A perticular example where this can happen is in a multi-card setup, where PCI-E peer-to-peer is used to prevent using access to system memory.
If the buffer is evicted to system memory, not only the evicting GPU wher the buffer resided is affected, but it will also stall the GPU that is waiting on the buffer.
It also makes sense for long running jobs not to be preempted by having its buffers evicted, so it will make sense to have the ability to pin from system memory too.
This is dependant on patches by Dave Airlie, so it's not part of this series yet. But I'm planning on extending pinning to the memory cgroup controller in the future to handle this case.
Implementation details:
Quoted 7 lines
For each cgroup up until the root cgroup, the 'min' limit is checked against currently effectively pinned value. If the value will go above 'min', the pinning attempt is rejected.
Pinned memory is handled slightly different and affects calculating effective min/low values. Pinned memory is subtracted from both, and needs to be added afterwards when calculating.
The term "pinning" is overloaded, and frequently we refer to pin_user_pages() and friends.
So I'm wondering if there is an alternative term to describe what you want to achieve.
Is it something like "unevictable" ?
Signature · 3 lines
Cheers David / dhildenb
Maarten Lankhorst
maintainerHello David,
Den 2025-09-01 kl. 14:25, skrev David Hildenbrand:
Quoted 30 lines · Maarten Lankhorst
Quoted 21 lines
When exporting dma-bufs to other devices, even when it is allowed to use
move_notifyin some drivers, performance will degrade severely when eviction happens.A perticular example where this can happen is in a multi-card setup, where PCI-E peer-to-peer is used to prevent using access to system memory.
If the buffer is evicted to system memory, not only the evicting GPU wher the buffer resided is affected, but it will also stall the GPU that is waiting on the buffer.
It also makes sense for long running jobs not to be preempted by having its buffers evicted, so it will make sense to have the ability to pin from system memory too.
This is dependant on patches by Dave Airlie, so it's not part of this series yet. But I'm planning on extending pinning to the memory cgroup controller in the future to handle this case.
Implementation details:
Quoted 7 lines
For each cgroup up until the root cgroup, the 'min' limit is checked against currently effectively pinned value. If the value will go above 'min', the pinning attempt is rejected.
Pinned memory is handled slightly different and affects calculating effective min/low values. Pinned memory is subtracted from both, and needs to be added afterwards when calculating.
Quoted 5 lines
The term "pinning" is overloaded, and frequently we refer to
pin_user_pages()and friends.So I'm wondering if there is an alternative term to describe what you want to achieve.
Is it something like "unevictable" ?
It could be required to include a call pin_user_pages(), in case a process wants to pin from a user's address space to the gpu.
It's not done yet, but it wouldn't surprise me if we want to include it in the future. Functionally it's similar to mlock() and related functions.
Perhaps call it mlocked instead?
Kind regards,
~Maarten Lankhorst
Thomas Hellström
maintainerHi,
Quoted 57 lines · Maarten Lankhorst
Hello David,
Den 2025-09-01 kl. 14:25, skrev David Hildenbrand:
Quoted 43 lines · Maarten Lankhorst
Quoted 37 lines
When exporting dma-bufs to other devices, even when it is allowed to use
move_notifyin some drivers, performance will degrade severely when
eviction happens.A perticular example where this can happen is in a multi-card setup,
where PCI-E peer-to-peer is used to prevent using access to system memory.If the buffer is evicted to system memory, not only the evicting GPU wher
the buffer resided is affected, but it will also stall the GPU that is
waiting on the buffer.It also makes sense for long running jobs not to be preempted by having
its buffers evicted, so it will make sense to have the ability to pin
from system memory too.This is dependant on patches by Dave Airlie, so it's not part of this
series yet. But I'm planning on extending pinning to the memory cgroup
controller in the future to handle this case.Implementation details:
For each cgroup up until the root cgroup, the 'min' limit is checked
against currently effectively pinned value. If the value will go above
'min', the pinning attempt is rejected.Quoted 4 lines
Pinned memory is handled slightly different and affects calculating
effective min/low values. Pinned memory is subtracted from both, and needs to be added afterwards when calculating.Quoted 7 lines
The term "pinning" is overloaded, and frequently we refer to
pin_user_pages()and friends.So I'm wondering if there is an alternative term to describe what you want to achieve.
Is it something like "unevictable" ?
It could be required to include a call
pin_user_pages(), in case a process wants to pin
from a user's address space to the gpu.
Quoted 5 lines
It's not done yet, but it wouldn't surprise me if we want to include it in the future.
Functionally it's similar tomlock()and related functions.Perhaps call it mlocked instead?
I was under the impression that mlocked() memory can be migrated to other physical memory but not to swap? whereas pinned memory needs to remain the exact same physical memory.
IMO "pinned" is pretty established within GPU drivers (dma-buf, TTM) and essentially means the same as "pin" in "pin_user_pages", so inventing a new name would probably cause even more confusion?
Thanks,
Thomas
Quoted 3 lines
Kind regards,
~Maarten Lankhorst
Quoted 57 lines · Thomas Hellström
Hi,
Quoted 54 lines · Maarten Lankhorst
Hello David,
Den 2025-09-01 kl. 14:25, skrev David Hildenbrand:
Quoted 43 lines · Maarten Lankhorst
Quoted 37 lines
When exporting dma-bufs to other devices, even when it is allowed to use
move_notifyin some drivers, performance will degrade severely when
eviction happens.A perticular example where this can happen is in a multi-card setup,
where PCI-E peer-to-peer is used to prevent using access to system memory.If the buffer is evicted to system memory, not only the evicting GPU wher
the buffer resided is affected, but it will also stall the GPU that is
waiting on the buffer.It also makes sense for long running jobs not to be preempted by having
its buffers evicted, so it will make sense to have the ability to pin
from system memory too.This is dependant on patches by Dave Airlie, so it's not part of this
series yet. But I'm planning on extending pinning to the memory cgroup
controller in the future to handle this case.Implementation details:
For each cgroup up until the root cgroup, the 'min' limit is checked
against currently effectively pinned value. If the value will go above
'min', the pinning attempt is rejected.Quoted 4 lines
Pinned memory is handled slightly different and affects calculating
effective min/low values. Pinned memory is subtracted from both, and needs to be added afterwards when calculating.Quoted 7 lines
The term "pinning" is overloaded, and frequently we refer to
pin_user_pages()and friends.So I'm wondering if there is an alternative term to describe what you want to achieve.
Is it something like "unevictable" ?
It could be required to include a call
pin_user_pages(), in case a
We'll only care about long-term pinnings (i.e., FOLL_LONGTERM). Ordinary short-term pinning is just fine.
(see how even "pinning" is overloaded? :) )
Quoted 6 lines
Quoted 6 lines
process wants to pin
from a user's address space to the gpu.It's not done yet, but it wouldn't surprise me if we want to include it in the future.
Functionally it's similar tomlock()and related functions.
Traditionally, vfio, io_uring and rdma do exactly that: they use GUP to longterm pin and then account that memory towards RLIMIT_MEMLOCK.
If you grep for "rlimit(RLIMIT_MEMLOCK)", you'll see what I mean.
There are known issues with that: imagine long-term pinning the same folio through GUP with 2 interfaces (e.g., vfio, io_uring, rdma), or within the same interface.
You'd account the memory multiple times, which is horrible. And so far there is no easy way out.
Quoted 6 lines
Quoted 2 lines
Perhaps call it mlocked instead?
I was under the impression that
mlocked()memory can be migrated to other physical memory but not to swap? whereas pinned memory needs to remain the exact same physical memory.
Yes, exactly.
Quoted 4 lines
IMO "pinned" is pretty established within GPU drivers (dma-buf, TTM) and essentially means the same as "pin" in "
pin_user_pages", so inventing a new name would probably cause even more confusion?
If it's the same thing, absolutely. But Marteen said "It's not done yet, but it wouldn't surprise me if we want to include it in the future".
So how is the memory we are talking about in this series "pinned" ?
Signature · 3 lines
Cheers David / dhildenb
Thomas Hellström
maintainerQuoted 115 lines · David Hildenbrand
Quoted 69 lines · Thomas Hellström
Hi,
Quoted 66 lines · Maarten Lankhorst
Hello David,
Den 2025-09-01 kl. 14:25, skrev David Hildenbrand:
Quoted 53 lines · Maarten Lankhorst
Quoted 46 lines
When exporting dma-bufs to other devices, even when it is allowed
to usemove_notifyin some drivers, performance will degrade severely
when
eviction happens.A perticular example where this can happen is in a multi-card setup,
where PCI-E peer-to-peer is used to prevent using access to system memory.If the buffer is evicted to system memory, not only the evicting
GPU wher
the buffer resided is affected, but it will also stall the GPU
that is
waiting on the buffer.It also makes sense for long running jobs not to be preempted by
having
its buffers evicted, so it will make sense to have the ability to
pin
from system memory too.This is dependant on patches by Dave Airlie, so it's not part of
this
series yet. But I'm planning on extending pinning to the memory
cgroup
controller in the future to handle this case.Implementation details:
For each cgroup up until the root cgroup, the 'min' limit is checked
against currently effectively pinned value. If the value will go
above
'min', the pinning attempt is rejected.Quoted 5 lines
Pinned memory is handled slightly different and affects calculating
effective min/low values. Pinned memory is subtracted from both,
and needs to be added afterwards when calculating.Quoted 8 lines
The term "pinning" is overloaded, and frequently we refer to
pin_user_pages()and friends.So I'm wondering if there is an alternative term to describe what
you want to achieve.Is it something like "unevictable" ?
It could be required to include a call
pin_user_pages(), in case aWe'll only care about long-term pinnings (i.e.,
FOLL_LONGTERM). Ordinary
short-term pinning is just fine.(see how even "pinning" is overloaded? :) )
Quoted 7 lines
Quoted 7 lines
process wants to pin
from a user's address space to the gpu.It's not done yet, but it wouldn't surprise me if we want to include
it in the future.
Functionally it's similar tomlock()and related functions.Traditionally, vfio,
io_uringand rdma do exactly that: they use GUP to
longterm pin and then account that memory towardsRLIMIT_MEMLOCK.If you grep for "
rlimit(RLIMIT_MEMLOCK)", you'll see what I mean.There are known issues with that: imagine long-term pinning the same folio through GUP with 2 interfaces (e.g., vfio,
io_uring, rdma), or within the same interface.You'd account the memory multiple times, which is horrible. And so far
there is no easy way out.Quoted 7 lines
Quoted 2 lines
Perhaps call it mlocked instead?
I was under the impression that
mlocked()memory can be migrated to other physical memory but not to swap? whereas pinned memory needs to
remain the exact same physical memory.Yes, exactly.
Quoted 5 lines
IMO "pinned" is pretty established within GPU drivers (dma-buf, TTM)
and essentially means the same as "pin" in "pin_user_pages", so inventing a new name would probably cause even more confusion?
Quoted 5 lines
If it's the same thing, absolutely. But Marteen said "It's not done yet,
but it wouldn't surprise me if we want to include it in the future".So how is the memory we are talking about in this series "pinned" ?
Reading the cover-letter from Maarten, he only talks about pinning affecting performance, which would be similar to user-space calling mlock(), although I doubt that moving content to other physical pages within the same memory type will be a near-term use-case.
However what's more important are situation where a device (like RDMA) needs to pin, because it can't handle the case where access is interrupted and content transferred to another physical location.
Perhaps Maarten could you elaborate whether this series is intended for both these use-cases?
/Thomas
Quoted 2 lines
Maarten Lankhorst
maintainerHey,
Den 2025-09-02 kl. 15:42, skrev Thomas Hellström:
Quoted 127 lines · David Hildenbrand
Quoted 115 lines · Thomas Hellström
Quoted 69 lines
Hi,
Quoted 66 lines · Maarten Lankhorst
Hello David,
Den 2025-09-01 kl. 14:25, skrev David Hildenbrand:
Quoted 53 lines · Maarten Lankhorst
Quoted 46 lines
When exporting dma-bufs to other devices, even when it is allowed
to usemove_notifyin some drivers, performance will degrade severely
when
eviction happens.A perticular example where this can happen is in a multi-card setup,
where PCI-E peer-to-peer is used to prevent using access to system memory.If the buffer is evicted to system memory, not only the evicting
GPU wher
the buffer resided is affected, but it will also stall the GPU
that is
waiting on the buffer.It also makes sense for long running jobs not to be preempted by
having
its buffers evicted, so it will make sense to have the ability to
pin
from system memory too.This is dependant on patches by Dave Airlie, so it's not part of
this
series yet. But I'm planning on extending pinning to the memory
cgroup
controller in the future to handle this case.Implementation details:
For each cgroup up until the root cgroup, the 'min' limit is checked
against currently effectively pinned value. If the value will go
above
'min', the pinning attempt is rejected.Quoted 5 lines
Pinned memory is handled slightly different and affects calculating
effective min/low values. Pinned memory is subtracted from both,
and needs to be added afterwards when calculating.Quoted 8 lines
The term "pinning" is overloaded, and frequently we refer to
pin_user_pages()and friends.So I'm wondering if there is an alternative term to describe what
you want to achieve.Is it something like "unevictable" ?
It could be required to include a call
pin_user_pages(), in case aWe'll only care about long-term pinnings (i.e.,
FOLL_LONGTERM). Ordinary
short-term pinning is just fine.(see how even "pinning" is overloaded? :) )
Quoted 7 lines
Quoted 7 lines
process wants to pin
from a user's address space to the gpu.It's not done yet, but it wouldn't surprise me if we want to include
it in the future.
Functionally it's similar tomlock()and related functions.Traditionally, vfio,
io_uringand rdma do exactly that: they use GUP to
longterm pin and then account that memory towardsRLIMIT_MEMLOCK.If you grep for "
rlimit(RLIMIT_MEMLOCK)", you'll see what I mean.There are known issues with that: imagine long-term pinning the same folio through GUP with 2 interfaces (e.g., vfio,
io_uring, rdma), or within the same interface.You'd account the memory multiple times, which is horrible. And so far
there is no easy way out.Quoted 7 lines
Quoted 2 lines
Perhaps call it mlocked instead?
I was under the impression that
mlocked()memory can be migrated to other physical memory but not to swap? whereas pinned memory needs to
remain the exact same physical memory.Yes, exactly.
Quoted 5 lines
IMO "pinned" is pretty established within GPU drivers (dma-buf, TTM)
and essentially means the same as "pin" in "pin_user_pages", so inventing a new name would probably cause even more confusion?Quoted 5 lines
If it's the same thing, absolutely. But Marteen said "It's not done yet,
but it wouldn't surprise me if we want to include it in the future".So how is the memory we are talking about in this series "pinned" ?
Reading the cover-letter from Maarten, he only talks about pinning affecting performance, which would be similar to user-space calling
mlock(), although I doubt that moving content to other physical pages within the same memory type will be a near-term use-case.
Quoted 6 lines
However what's more important are situation where a device (like RDMA) needs to pin, because it can't handle the case where access is interrupted and content transferred to another physical location.
Perhaps Maarten could you elaborate whether this series is intended for both these use-cases?
Yeah, this is definitely for the latter case too.
It's a performance optimization for the generic case, and very nice to have for the second case, to prevent unlimited vram pinning. With cgroups, we would be able to limit the amounts of used memory there.
Kind regards,
~Maarten
Hi,
Quoted 19 lines · Maarten Lankhorst
When exporting dma-bufs to other devices, even when it is allowed to use
move_notifyin some drivers, performance will degrade severely when eviction happens.A perticular example where this can happen is in a multi-card setup, where PCI-E peer-to-peer is used to prevent using access to system memory.
If the buffer is evicted to system memory, not only the evicting GPU wher the buffer resided is affected, but it will also stall the GPU that is waiting on the buffer.
It also makes sense for long running jobs not to be preempted by having its buffers evicted, so it will make sense to have the ability to pin from system memory too.
This is dependant on patches by Dave Airlie, so it's not part of this series yet. But I'm planning on extending pinning to the memory cgroup controller in the future to handle this case.
Quoted 5 lines
Implementation details:
For each cgroup up until the root cgroup, the 'min' limit is checked against currently effectively pinned value. If the value will go above 'min', the pinning attempt is rejected.
Why do you want to reject pins in this case? What happens in desktop usecases (e.g. PRIME buffer sharing)? AFAIU, you kind of need to be able to pin buffers and export them to other devices for that whole thing to work, right? If the user doesn't explicitly set a min value, wouldn't the value being zero mean any pins will be rejected (and thus PRIME would break)?
If your objective is to prevent pinned buffers from being evicted, perhaps you could instead make TTM try to avoid evicting pinned buffers and prefer unpinned buffers as long as there are unpinned buffers to evict? As long as the total amount of pinned memory stays below min, no pinned buffers should get evicted with that either.
Best,
Natalie
Quoted 23 lines
Pinned memory is handled slightly different and affects calculating effective min/low values. Pinned memory is subtracted from both, and needs to be added afterwards when calculating.
This is because increasing the amount of pinned memory, the amount of free min/low memory decreases for all cgroups that are part of the hierarchy.
Maarten Lankhorst (3):
page_counter: Allow for pinning some amount of memory cgroup/dmem: Implement pinning device memory
drm/xe: AddDRM_XE_GEM_CREATE_FLAG_PINNEDflag and implementationdrivers/gpu/drm/xe/xe_bo.c | 66 +++++++++++++++++++++- drivers/gpu/drm/xe/xe_dma_buf.c | 10 +++- include/linux/cgroup_dmem.h | 2 + include/linux/page_counter.h | 8 +++ include/uapi/drm/xe_drm.h | 10 +++- kernel/cgroup/dmem.c | 57 ++++++++++++++++++- mm/page_counter.c | 98 ++++++++++++++++++++++++++++++--- 7 files changed, 237 insertions(+), 14 deletions(-)
Thomas Hellström
maintainerHi,
Quoted 39 lines · Natalie Vock
Hi,
Quoted 28 lines · Maarten Lankhorst
When exporting dma-bufs to other devices, even when it is allowed to use
move_notifyin some drivers, performance will degrade severely when eviction happens.A perticular example where this can happen is in a multi-card setup,
where PCI-E peer-to-peer is used to prevent using access to system memory.If the buffer is evicted to system memory, not only the evicting GPU wher
the buffer resided is affected, but it will also stall the GPU that is
waiting on the buffer.It also makes sense for long running jobs not to be preempted by having
its buffers evicted, so it will make sense to have the ability to pin
from system memory too.This is dependant on patches by Dave Airlie, so it's not part of this
series yet. But I'm planning on extending pinning to the memory cgroup
controller in the future to handle this case.Quoted 7 lines
Implementation details:
For each cgroup up until the root cgroup, the 'min' limit is checked
against currently effectively pinned value. If the value will go above
'min', the pinning attempt is rejected.
Quoted 8 lines
Why do you want to reject pins in this case? What happens in desktop usecases (e.g. PRIME buffer sharing)? AFAIU, you kind of need to be able
to pin buffers and export them to other devices for that whole thing to
work, right? If the user doesn't explicitly set a min value, wouldn't the value being zero mean any pins will be rejected (and thus PRIME would break)?
That's really the point. If an unprivileged malicious process is allowed to pin arbitrary amounts of memory, thats a DOS vector.
However drivers that allow unlimited pinning today need to take care when implementing restrictions to avoid regressions. Like perhaps adding this behind a config option.
That said, IMO dma-buf clients should implement move_notify() whenever possible to provide an option to avoid pinning unless necessary.
/Thomas
Quoted 41 lines
If your objective is to prevent pinned buffers from being evicted, perhaps you could instead make TTM try to avoid evicting pinned buffers
and prefer unpinned buffers as long as there are unpinned buffers to evict? As long as the total amount of pinned memory stays below min, no
pinned buffers should get evicted with that either.Best,
NatalieQuoted 26 lines
Pinned memory is handled slightly different and affects calculating effective min/low values. Pinned memory is subtracted from both, and needs to be added afterwards when calculating.
This is because increasing the amount of pinned memory, the amount of
free min/low memory decreases for all cgroups that are part of the hierarchy.Maarten Lankhorst (3):
page_counter: Allow for pinning some amount of memory cgroup/dmem: Implement pinning device memory drm/xe: AddDRM_XE_GEM_CREATE_FLAG_PINNEDflag and implementationdrivers/gpu/drm/xe/xe_bo.c | 66 +++++++++++++++++++++-
drivers/gpu/drm/xe/xe_dma_buf.c| 10 +++- include/linux/cgroup_dmem.h | 2 + include/linux/page_counter.h | 8 +++ include/uapi/drm/xe_drm.h | 10 +++- kernel/cgroup/dmem.c | 57 ++++++++++++++++++- mm/page_counter.c | 98
++++++++++++++++++++++++++++++---7 files changed, 237 insertions(+), 14 deletions(-)
Maarten Lankhorst
maintainerHello Nathalie,
Den 2025-09-01 kl. 14:45, skrev Natalie Vock:
Quoted 24 lines
Hi,
Quoted 21 lines · Maarten Lankhorst
When exporting dma-bufs to other devices, even when it is allowed to use
move_notifyin some drivers, performance will degrade severely when eviction happens.A perticular example where this can happen is in a multi-card setup, where PCI-E peer-to-peer is used to prevent using access to system memory.
If the buffer is evicted to system memory, not only the evicting GPU wher the buffer resided is affected, but it will also stall the GPU that is waiting on the buffer.
It also makes sense for long running jobs not to be preempted by having its buffers evicted, so it will make sense to have the ability to pin from system memory too.
This is dependant on patches by Dave Airlie, so it's not part of this series yet. But I'm planning on extending pinning to the memory cgroup controller in the future to handle this case.
Implementation details:
Quoted 7 lines
Quoted 3 lines
For each cgroup up until the root cgroup, the 'min' limit is checked against currently effectively pinned value. If the value will go above 'min', the pinning attempt is rejected.
Why do you want to reject pins in this case? What happens in desktop usecases (e.g. PRIME buffer sharing)? AFAIU, you kind of need to be able to pin buffers and export them to other devices for that whole thing to work, right? If the user doesn't explicitly set a min value, wouldn't the value being zero mean any pins will be rejected (and thus PRIME would break)?
If your objective is to prevent pinned buffers from being evicted, perhaps you could instead make TTM try to avoid evicting pinned buffers and prefer unpinned buffers as long as there are unpinned buffers to evict? As long as the total amount of pinned memory stays below min, no pinned buffers should get evicted with that either.
That would be setting an eviction priority, that can be done but that gives no guarantee memory will not be evicted.
Kind regards,
~Maarten