drm/xe: Prioritize BO eviction based on VM exec queue priority
Subsystems (2)
Matthew Brost
maintainerv2 cover drm/xe: Prioritize BO eviction based on VM exec queue priority
This series ties a BO's TTM LRU priority (i.e. how likely it is to be picked for eviction/shrinking under memory pressure) to how userspace is actually using the VM and BOs it belongs to, instead of treating every BO the same.
The reasoning behind the heuristic:
- Kernel BOs, and any BO not private to a single VM (no vm_id, e.g. shared/exported buffers), are given the highest priority so they are the least likely to be evicted -- these are foundational or shared resources.
- BOs private to a VM are then prioritized based on the highest priority band userspace has assigned to any exec queue attached to that VM: a VM with a HIGH priority exec queue keeps its BOs at the highest priority (matching kernel/shared BOs), a VM with only NORMAL priority queues (or none yet) gets the next priority band down, and a VM with only LOW priority queues gets the band below that.
- The lowest priority band is reserved exclusively for BOs (private or shared) that have no VMA mappings left anywhere. This lets userspace use an unbind as a signal of "drop this from my working set, but keep the backing store around in case I need it again" -- the BO is retained, just made a preferred candidate for eviction/shrinking until it is rebound.
Patch 1 adds the accounting: a per-priority-band count of exec queues attached to each user VM. Patch 2 introduces the actual heuristic and uses it both to set a BO's initial priority at creation time and to bulk re-prioritize a VM's existing private BOs when an exec queue is added or removed. Patch 3 fixes up staleness that these earlier mechanisms miss: BOs created before any exec queue exists, or bound late via VM_BIND, and BOs not tied to a single VM that get demoted to the lowest priority and then rebound. Patch 4 introduces the lowest-priority demotion itself, applied when a BO loses its last VMA mapping.
Matt
Matthew Brost (4):
drm/xe: Track exec queue priority band counts for user VMs drm/xe: Re-prioritize VM's private BOs when adding an exec queue drm/xe: Fix up BO TTM priority for late VM binds
drm/xe: Lower BO TTM priority to lowest when fully unmapped from VM
drivers/gpu/drm/xe/xe_bo.c | 18 ++- drivers/gpu/drm/xe/xe_bo.h | 3 +- drivers/gpu/drm/xe/xe_bo_types.h | 3 + drivers/gpu/drm/xe/xe_dma_buf.c | 2 +- drivers/gpu/drm/xe/xe_vm.c | 205 +++++++++++++++++++++++++++++-- drivers/gpu/drm/xe/xe_vm.h | 1 + drivers/gpu/drm/xe/xe_vm_types.h | 7 ++ 7 files changed, 221 insertions(+), 18 deletions(-)
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v2 1/4 drm/xe: Track exec queue priority band counts for user VMs
Add a per-priority-band count (LOW/NORMAL/HIGH) of exec queues attached to a user VM in xe_vm_add_exec_queue()/xe_vm_remove_exec_queue(), protected by vm->exec_queues.lock in write mode. KERNEL priority is not counted since it is not possible for user exec queues.
This count is needed so a VM's private buffer objects can later be prioritized for eviction based on the priority userspace has assigned to the exec queues using that VM (i.e. work submitted at a higher priority band implies its buffers are more important to keep resident), rather than treating every VM the same.
Also move the has_ctx_tlb_inval check inside the lock in xe_vm_add_exec_queue() so the list add/count update is properly gated under the same critical section, and make xe_vm_remove_exec_queue() bail out early if q->xef is not set, rather than relying on callers to check this.
--- drivers/gpu/drm/xe/xe_vm.c | 31 ++++++++++++++++++++----------- drivers/gpu/drm/xe/xe_vm_types.h | 7 +++++++ 2 files changed, 27 insertions(+), 11 deletions(-)
drivers/gpu/drm/xe/xe_vm.c +20 -11
@@ -4933,8 +4933,9 @@ int xe_vm_alloc_cpu_addr_mirror_vma(struct xe_vm *vm, uint64_t start, uint64_t r* @vm: The VM. * @q: The exec_queue * -* Add exec queue to VM, skipped if the device does not have context based TLB -* invalidations. +* Track exec queue's priority band count for the VM. Also links the exec +* queue onto the per-GT list, skipped if the device does not have context +* based TLB invalidations. */ void xe_vm_add_exec_queue (struct xe_vm *vm,struct xe_exec_queue *q) {
@@ -4948,12 +4949,14 @@ void xe_vm_add_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q)xe_assert (xe, vm->xef);xe_assert (xe, vm == q->vm); -if (!xe->info.has_ctx_tlb_inval) -return ; -down_write (&vm->exec_queues.lock); -list_add (&q->vm_exec_queue_link, &vm->exec_queues.list[q->gt->info.id]); - ++vm->exec_queues.count[q->gt->info.id]; +if (xe->info.has_ctx_tlb_inval) { +list_add (&q->vm_exec_queue_link, &vm->exec_queues.list[q->gt->info.id]); + ++vm->exec_queues.count[q->gt->info.id]; + } +if (q->sched_props.priority >=XE_EXEC_QUEUE_PRIORITY_LOW && + q->sched_props.priority <=XE_EXEC_QUEUE_PRIORITY_HIGH ) + ++vm->exec_queues.priority_count[q->sched_props.priority];up_write (&vm->exec_queues.lock); }
@@ -4962,18 +4965,24 @@ void xe_vm_add_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q)* @vm: The VM. * @q: The exec_queue * -* Remove exec queue from VM, skipped if the device does not have context based -* TLB invalidations. +* Untrack exec queue's priority band count for the VM. Also unlinks the +* exec queue from the per-GT list, skipped if the device does not have +* context based TLB invalidations. No-op if @q is not a user exec queue, +* or is a VM exec queue (VM exec queues are never tracked by +* xe_vm_add_exec_queue()). */ void xe_vm_remove_exec_queue (struct xe_vm *vm,struct xe_exec_queue *q) { -if (!vm->xe->info.has_ctx_tlb_inval) +if (!q->xef || (q->flags &EXEC_QUEUE_FLAG_VM ))return ;down_write (&vm->exec_queues.lock); -if (!list_empty (&q->vm_exec_queue_link)) { +if (vm->xe->info.has_ctx_tlb_inval && !list_empty (&q->vm_exec_queue_link)) {list_del (&q->vm_exec_queue_link); --vm->exec_queues.count[q->gt->info.id]; } +if (q->sched_props.priority >=XE_EXEC_QUEUE_PRIORITY_LOW && + q->sched_props.priority <=XE_EXEC_QUEUE_PRIORITY_HIGH ) + --vm->exec_queues.priority_count[q->sched_props.priority];up_write (&vm->exec_queues.lock); }
drivers/gpu/drm/xe/xe_vm_types.h +7 -0
@@ -16,6 +16,7 @@#include <linux/scatterlist.h> #include "xe_device_types.h" +#include "xe_exec_queue_types.h" #include "xe_pt_types.h" #include "xe_range_fence.h" #include "xe_tlb_inval_types.h"
@@ -342,6 +343,12 @@ struct xe_vm {* per GT */ int count[XE_MAX_TILES_PER_DEVICE *XE_MAX_GT_PER_TILE ]; +/** +* @exec_queues.priority_count: count of exec queues attached +* to this VM, per priority band (LOW / NORMAL / HIGH only, +* KERNEL priority is not possible for user exec queues) +*/ +int priority_count[XE_EXEC_QUEUE_PRIORITY_HIGH +1 ];/** @exec_queues.lock: lock to protect exec_queues list */ struct rw_semaphore lock; } exec_queues;
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v2 2/4 drm/xe: Re-prioritize VM's private BOs when adding an exec queue
In reply to Matthew BrostWhen a higher (or lower) priority exec queue is added to a VM, the VM's existing private BOs were left at whatever TTM LRU priority they had at creation time, even though the newly added queue changes what xe_vm_bo_priority() would now compute for that VM. This means a VM's buffers would not reflect the actual importance userspace has now assigned to work on that VM until new BOs happen to be allocated, leaving stale, and potentially too-low, eviction priority on long-lived buffers.
After dropping vm->exec_queues.lock in xe_vm_add_exec_queue(), take the VM's dma-resv (xe_vm_lock()) and walk all of the VM's private BOs (bo->vm == vm, skipping extobjs shared with other VMs), updating each one's TTM priority to the newly computed level and moving it to the tail of its new priority's LRU list.
--- drivers/gpu/drm/xe/xe_bo.c | 18 +++-- drivers/gpu/drm/xe/xe_bo.h | 3 +- drivers/gpu/drm/xe/xe_bo_types.h | 3 + drivers/gpu/drm/xe/xe_dma_buf.c | 2 +- drivers/gpu/drm/xe/xe_vm.c | 126 ++++++++++++++++++++++++++++++- drivers/gpu/drm/xe/xe_vm.h | 1 + 6 files changed, 145 insertions(+), 8 deletions(-)
drivers/gpu/drm/xe/xe_bo.c +13 -5
@@ -1346,7 +1346,8 @@ int xe_bo_notifier_prepare_pinned(struct xe_bo *bo)if (bo->flags &XE_BO_FLAG_PINNED_NORESTORE )break ; - backup =xe_bo_init_locked (xe,NULL ,NULL , bo->ttm.base.resv,NULL ,xe_bo_size (bo), + backup =xe_bo_init_locked (xe,NULL ,NULL ,NULL , bo->ttm.base.resv,NULL , +xe_bo_size (bo),DRM_XE_GEM_CPU_CACHING_WB , ttm_bo_type_kernel,XE_BO_FLAG_SYSTEM |XE_BO_FLAG_NEEDS_CPU_ACCESS |XE_BO_FLAG_PINNED , &exec);
@@ -1486,7 +1487,7 @@ int xe_bo_evict_pinned(struct xe_bo *bo)break ;if (!backup) { - backup = xe_bo_init_locked(xe, NULL, NULL, bo->ttm.base.resv, NULL, + backup = xe_bo_init_locked(xe, NULL, NULL, NULL, bo->ttm.base.resv, NULL, xe_bo_size(bo), DRM_XE_GEM_CPU_CACHING_WB, ttm_bo_type_kernel, XE_BO_FLAG_SYSTEM | XE_BO_FLAG_NEEDS_CPU_ACCESS |
@@ -2276,6 +2277,9 @@ void xe_bo_free(struct xe_bo *bo)* if the function should allocate a new one. * @tile: The tile to select for migration of this bo, and the tile used for * GGTT binding if any. Only to be non-NULL for ttm_bo_type_kernel bos. +* @vm: The VM this bo will be associated with, used to determine the bo's +* TTM LRU priority based on the priority band of the VM's exec queues, or +* NULL if the bo is not associated with a user VM. * @resv: Pointer to a locked shared reservation object to use for this bo, * or NULL for the xe_bo to use its own. * @bulk: The bulk move to use for LRU bumping, or NULL for external bos.
@@ -2291,7 +2295,8 @@ void xe_bo_free(struct xe_bo *bo)* Return: The buffer object on success. Negative error pointer on failure. */ struct xe_bo *xe_bo_init_locked (struct xe_device *xe,struct xe_bo *bo, -struct xe_tile *tile,struct dma_resv *resv, +struct xe_tile *tile,struct xe_vm *vm, +struct dma_resv *resv,struct ttm_lru_bulk_move *bulk,size_t size,u16 cpu_caching,enum ttm_bo_type type,u32 flags,struct drm_exec *exec)
@@ -2353,7 +2358,10 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo, bo->flags = flags; bo->cpu_caching = cpu_caching; bo->ttm.base.funcs = &xe_gem_object_funcs; - bo->ttm.priority =XE_BO_PRIORITY_NORMAL ; +if (type != ttm_bo_type_device || !vm) + bo->ttm.priority =XE_BO_PRIORITY_HIGHEST ; +else + bo->ttm.priority =xe_vm_bo_priority (vm);INIT_LIST_HEAD (&bo->pinned_link);#ifdef CONFIG_PROC_FS INIT_LIST_HEAD (&bo->client_link);
@@ -2496,7 +2504,7 @@ __xe_bo_create_locked(struct xe_device *xe, } } - bo = xe_bo_init_locked(xe, bo, tile, vm ? xe_vm_resv(vm) : NULL, + bo = xe_bo_init_locked(xe, bo, tile, vm, vm ? xe_vm_resv(vm) : NULL, vm && !xe_vm_in_fault_mode(vm) && flags & XE_BO_FLAG_USER ? &vm->lru_bulk_move : NULL, size,
drivers/gpu/drm/xe/xe_bo.h +2 -1
@@ -115,7 +115,8 @@ struct xe_bo *xe_bo_alloc(void);void xe_bo_free (struct xe_bo *bo);struct xe_bo *xe_bo_init_locked (struct xe_device *xe,struct xe_bo *bo, -struct xe_tile *tile,struct dma_resv *resv, +struct xe_tile *tile,struct xe_vm *vm, +struct dma_resv *resv,struct ttm_lru_bulk_move *bulk,size_t size,u16 cpu_caching,enum ttm_bo_type type,u32 flags,struct drm_exec *exec);
drivers/gpu/drm/xe/xe_bo_types.h +3 -0
@@ -24,7 +24,10 @@ struct xe_vm;#define XE_BO_MAX_PLACEMENTS 3/* TODO: To be selected with VM_MADVISE */ +#define XE_BO_PRIORITY_LOW 0#define XE_BO_PRIORITY_NORMAL 1 +#define XE_BO_PRIORITY_HIGH 2 +#define XE_BO_PRIORITY_HIGHEST 3/** * struct xe_bo - Xe buffer object
drivers/gpu/drm/xe/xe_dma_buf.c +1 -1
@@ -300,7 +300,7 @@ xe_dma_buf_create_obj(struct drm_device *dev, struct dma_buf *dma_buf)if (ret)break ; - bo =xe_bo_init_locked (xe,NULL ,NULL , resv,NULL , dma_buf->size, + bo =xe_bo_init_locked (xe,NULL ,NULL ,NULL , resv,NULL , dma_buf->size,0 ,/* Will require 1way or 2way for vm_bind */ ttm_bo_type_sg,XE_BO_FLAG_SYSTEM , &exec);drm_exec_retry_on_contention (&exec);
drivers/gpu/drm/xe/xe_vm.c +125 -1
171 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm.h +1 -0
@@ -305,6 +305,7 @@ void xe_vm_kill(struct xe_vm *vm, bool unlocked);void xe_vm_add_exec_queue (struct xe_vm *vm,struct xe_exec_queue *q);void xe_vm_remove_exec_queue (struct xe_vm *vm,struct xe_exec_queue *q); +int xe_vm_bo_priority (struct xe_vm *vm);/** * xe_vm_assert_held(vm) - Assert that the vm's reservation object is held.
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v2 2/4] drm/xe: Re-prioritize VM's private BOs when adding an exec queue
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider: - [Medium] Concurrent exec queue additions and removals create a Time-of-Check to Time-of-Use (TOCTOU)…
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Medium] Concurrent exec queue additions and removals create a Time-of-Check to Time-of-Use (TOCTOU) race, leaving BOs with stale TTM priorities.
- [Medium] The BO priority evaluation incorrectly prioritizes LOW priority exec queues over NORMAL priority ones, causing unintended priority downgrades.
- [Medium] Unbound private BOs miss dynamic priority updates and permanently retain stale priorities.
- [Medium] The patch omits the intended logic to downgrade completely unmapped BOs to XE_BO_PRIORITY_LOW in xe_vma_destroy(). --
--- Patch [2]: [PATCH v2 2/4] drm/xe: Re-prioritize VM's private BOs when adding an exec queue --- commit 45971462729146ee32e2c293b500a521f745e5fe
Author: Matthew Brost <[email protected]>
Subject: drm/xe: Re-prioritize VM's private BOs when adding an exec queue
This commit updates the TTM LRU eviction priority of a VM's private BOs when exec queues are added or removed. It iterates over the VM's active VMAs to apply the new priority based on the highest priority band among the VM's attached exec queues.
[ ... ]
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0
[ ... ]
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.c +5
-0@@ -4912,6 +4944,87 @@ int xe_vm_alloc_cpu_addr_mirror_vma(struct xe_vm *vm, uint64_t start, uint64_t rreturn xe_vm_alloc_vma (vm, &map_req, false); } +/* +* Determine the TTM LRU priority to use for BOs private to @vm, based on the +* highest priority band among exec queues currently attached to @vm. +* +* A VM with a HIGH priority exec queue gets XE_BO_PRIORITY_HIGHEST, same as
drivers/gpu/drm/xe/xe_vm.c int xe_vm_alloc_cpu_addr_mirror_vma(struct xe_vm *vm, uint64_t start, uint64_t r · line 4961
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm.c +10
-0+* BOs that are never subject to this heuristic at all (kernel or shared +* BOs) -- userspace has indicated its most important work lives here, so +* treat these BOs as equally important. Every other band is then shifted up +* one level from its "intuitive" mapping: a NORMAL priority exec queue (or no +* exec queues at all yet) yields XE_BO_PRIORITY_HIGH, and a LOW priority exec +* queue yields XE_BO_PRIORITY_NORMAL. XE_BO_PRIORITY_LOW is intentionally +* never returned here -- it is reserved for xe_vma_destroy() to apply once a +* BO has no VMAs left, letting userspace signal "drop this from my working +* set" via unbind while still retaining the backing store, without this +* heuristic immediately promoting it back up on the next bind.
[Severity: Medium]
Is the logic to downgrade completely unmapped BOs to XE_BO_PRIORITY_LOW missing from this patch? The comment states it is reserved for xe_vma_destroy() to apply once a BO has no VMAs left, but the implementation to actually apply it does not seem to be included in this changeset.
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c +5
-0+* +* Must be called with @vm's exec_queues.lock held, in either read or write +* mode. +*/ +static int xe_vm_bo_priority_locked (struct xe_vm *vm)
drivers/gpu/drm/xe/xe_vm.c int xe_vm_alloc_cpu_addr_mirror_vma(struct xe_vm *vm, uint64_t start, uint64_t r · line 4992
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm.c +10
-0+{ +lockdep_assert_held (&vm->exec_queues.lock); + +if (vm->exec_queues.priority_count[XE_EXEC_QUEUE_PRIORITY_HIGH ]) +return XE_BO_PRIORITY_HIGHEST ; +if (vm->exec_queues.priority_count[XE_EXEC_QUEUE_PRIORITY_LOW ]) +return XE_BO_PRIORITY_NORMAL ; + +return XE_BO_PRIORITY_HIGH ; +}
[Severity: Medium]
Does this logic incorrectly prioritize LOW priority exec queues over NORMAL priority ones? If a VM has both a NORMAL and a LOW priority queue, it appears this will return XE_BO_PRIORITY_NORMAL instead of checking for the presence of a NORMAL queue first, which might unfairly downgrade the VM's priority band.
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c +3
-0+ +/** +* xe_vm_bo_priority() - Determine TTM LRU priority for a VM's private BOs
[ ... ]
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c +8
-0+/* +* Update the TTM LRU priority of all BOs private to @vm to @priority, moving +* each to the tail of its new priority level's LRU list. BOs shared with +* other VMs (extobjs) are left untouched. +*/ +static void xe_vm_update_bo_priority (struct xe_vm *vm,int priority) +{ +struct drm_gpuva *gpuva;
drivers/gpu/drm/xe/xe_vm.c int xe_vm_alloc_cpu_addr_mirror_vma(struct xe_vm *vm, uint64_t start, uint64_t r · line 5031
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm.c +10
-0+ +/* +* vm->lock is the outer most lock and serializes against concurrent +* VMA insertion/removal (bind/unbind), so it must be held to safely +* walk vm->gpuvm here. xe_vm_lock() (the VM's dma-resv) is taken +* inside it, as it protects each private BO's bo->ttm.priority. +*/ +down_read (&vm->lock); +xe_vm_lock (vm, false); +drm_gpuvm_for_each_va (gpuva, &vm->gpuvm) {
[Severity: Medium]
Can unbound private BOs miss dynamic priority updates here? Since this loop iterates over mapped VMAs using drm_gpuvm_for_each_va(), any allocated but not yet bound BOs would be skipped and might permanently retain their stale creation priorities.
Quoted 11 lines
drivers/gpu/drm/xe/xe_vm.c +11
-0+struct xe_vma *vma =gpuva_to_vma (gpuva); +struct xe_bo *bo =xe_vma_bo (vma); + +if (!bo || bo->vm != vm || bo->ttm.priority == priority) +continue ; + +xe_bo_update_ttm_priority (bo, priority); + } +xe_vm_unlock (vm); +up_read (&vm->lock); +}
[ ... ]
drivers/gpu/drm/xe/xe_vm.c void xe_vm_add_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q) · line 5062
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.c +3
-0@@ -4941,7 +5055,10 @@ void xe_vm_add_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q)if (q->sched_props.priority >=XE_EXEC_QUEUE_PRIORITY_LOW && q->sched_props.priority <=XE_EXEC_QUEUE_PRIORITY_HIGH ) ++vm->exec_queues.priority_count[q->sched_props.priority]; + priority =xe_vm_bo_priority_locked (vm);up_write (&vm->exec_queues.lock); + +xe_vm_update_bo_priority (vm, priority); }
[Severity: Medium]
Is there a race condition between taking the priority snapshot and applying it?
If multiple threads are concurrently adding and removing exec queues, their execution could interleave between dropping exec_queues.lock and acquiring vm->lock inside xe_vm_update_bo_priority().
Could the last thread to acquire vm->lock apply a stale priority snapshot that no longer reflects the current queue state?
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0/**
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Matthew Brost
maintainerPATCH v2 3/4 drm/xe: Fix up BO TTM priority for late VM binds
In reply to Matthew BrostA BO's TTM priority is only ever set once, at creation time, based on the priority band of exec queues attached to its VM at that moment (xe_vm_bo_priority()), or XE_BO_PRIORITY_HIGHEST for an extobj not tied to any single VM. This misses the common flow of VM create -> BO create -> exec queue create -> VM bind: the BO is created before any exec queue exists on the VM (so it gets XE_BO_PRIORITY_HIGH, the "no exec queues yet" default), the exec queue is added afterwards (xe_vm_add_exec_queue() only re-prioritizes BOs already bound to the VM at that time), and only then is the BO bound into the VM via VM_BIND, leaving it stuck at a stale priority that no longer reflects the VM's exec queues.
It also misses a second flow specific to extobjs: create BO with no vm_id (bo->vm == NULL) -> bind -> unbind -> rebind. The unbind can lower the BO's priority to XE_BO_PRIORITY_LOW once it has no VMA mappings left anywhere (see xe_vma_destroy()), but on rebind nothing brought it back up, leaving a still-useful, freshly rebound extobj stuck at the lowest priority, an easy target for eviction/shrinking.
Add op_update_bo_priority(), called from vm_bind_ioctl_ops_fini() for each DRM_GPUVA_OP_MAP operation. For a BO private to the VM being bound (bo->vm == vm), recompute the current priority band via xe_vm_bo_priority() and, if it differs, update bo->ttm.priority and move the BO to the tail of its new priority's LRU list. For an extobj (bo->vm == NULL, potentially shared with other VMs), instead restore it directly to XE_BO_PRIORITY_HIGHEST, since it is now in active use again and extobjs aren't tied to any single VM's exec queue priority band. This runs under the VM's dma-resv lock, already held via drm_exec in vm_bind_ioctl_ops_execute() since private BOs share the VM's reservation object, and extobjs are locked as part of the same bind operation.
--- drivers/gpu/drm/xe/xe_vm.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
drivers/gpu/drm/xe/xe_vm.c +34 -0
@@ -3554,6 +3554,38 @@ static void op_add_ufence(struct xe_vm *vm, struct xe_vma_op *op, } } +/* +* A BO's TTM priority is set when it is created, based on the priority band +* of @vm's exec queues at that time (for a private BO), or to the highest +* level (for an extobj, bo->vm == NULL). If a BO is (re)mapped into @vm +* afterwards, its priority may be stale: a private BO's VM may have gained +* or lost exec queues since, and an extobj may have been lowered to +* XE_BO_PRIORITY_LOW by xe_vma_destroy() after losing all of its VMA +* mappings. Bring it in line with the current priority band here. +*/ +static void op_update_bo_priority (struct xe_vm *vm,struct xe_vma_op *op) +{ +struct xe_bo *bo; +int priority; + +if (op->base.op !=DRM_GPUVA_OP_MAP ) +return ; + + bo =xe_vma_bo (op->map.vma); +if (!bo) +return ; + +/* Private BOs are only ever bound to the VM they were created for. */ +xe_assert (vm->xe, !bo->vm || bo->vm == vm); + + priority = bo->vm ?xe_vm_bo_priority (vm) :XE_BO_PRIORITY_HIGHEST ; +if (bo->ttm.priority == priority) +return ; + +xe_vm_assert_held (vm); +xe_bo_update_ttm_priority (bo, priority); +} +static void vm_bind_ioctl_ops_fini (struct xe_vm *vm,struct xe_vma_ops *vops,struct dma_fence *fence) {
@@ -3566,6 +3598,8 @@ static void vm_bind_ioctl_ops_fini(struct xe_vm *vm, struct xe_vma_ops *vops,if (ufence)op_add_ufence (vm, op, ufence); +op_update_bo_priority (vm, op); +if (op->base.op ==DRM_GPUVA_OP_UNMAP )xe_vma_destroy (gpuva_to_vma (op->base.unmap.va), fence);else if (op->base.op ==DRM_GPUVA_OP_REMAP )
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v2 3/4] drm/xe: Fix up BO TTM priority for late VM binds
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [High] Data race and TTM LRU corruption due to missing invariant check for cross-VM binds of private BOs.
Pre-existing issues:
- [Medium] Meaningful code and commit message mismatch (Incomplete Implementation of extobj priority lowering). --
--- Patch [3]: [PATCH v2 3/4] drm/xe: Fix up BO TTM priority for late VM binds --- commit b5439d40abd0e7b0ac5ecc4fc4045178d3dcfcb7
Author: Matthew Brost <[email protected]>
drm/xe: Fix up BO TTM priority for late VM binds
This patch updates the BO TTM priority when a BO is (re)mapped into a VM. It recalculates the priority band based on the VM's current exec queues or sets it to the highest priority for external objects.
Quoted 12 lines
A BO's TTM priority is only ever set once, at creation time, based on the priority band of exec queues attached to its VM at that moment (
xe_vm_bo_priority()), orXE_BO_PRIORITY_HIGHESTfor an extobj not tied to any single VM. This misses the common flow of VM create -> BO create -> exec queue create -> VM bind: the BO is created before any exec queue exists on the VM (so it getsXE_BO_PRIORITY_HIGH, the "no exec queues yet" default), the exec queue is added afterwards (xe_vm_add_exec_queue()only re-prioritizes BOs already bound to the VM at that time), and only then is the BO bound into the VM viaVM_BIND, leaving it stuck at a stale priority that no longer reflects the VM's exec queues.
Quoted 4 lines
It also misses a second flow specific to extobjs: create BO with no
vm_id(bo->vm== NULL) -> bind -> unbind -> rebind. The unbind can lower the BO's priority toXE_BO_PRIORITY_LOWonce it has no VMA mappings left anywhere (seexe_vma_destroy()), but on rebind nothing
[Severity: Medium]
This is a pre-existing issue, but I don't see any logic in xe_vma_destroy() or elsewhere in the codebase that actually lowers an extobj's priority to XE_BO_PRIORITY_LOW.
Because extobjs appear to permanently retain XE_BO_PRIORITY_HIGHEST, won't the fix in op_update_bo_priority() always hit the early return for extobjs, missing the intended working set eviction optimization entirely?
Quoted 4 lines
brought it back up, leaving a still-useful, freshly rebound extobj stuck at the lowest priority, an easy target for eviction/shrinking.
Add
op_update_bo_priority(), called fromvm_bind_ioctl_ops_fini()for
[ ... ]
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0
[ ... ]
Quoted 17 lines
drivers/gpu/drm/xe/xe_vm.c +13
-0@@ -3538,6 +3538,38 @@ static void op_add_ufence(struct xe_vm *vm, struct xe_vma_op *op, } } +/* +* A BO's TTM priority is set when it is created, based on the priority band +* of @vm's exec queues at that time (for a private BO), or to the highest +* level (for an extobj, bo->vm == NULL). If a BO is (re)mapped into @vm +* afterwards, its priority may be stale: a private BO's VM may have gained +* or lost exec queues since, and an extobj may have been lowered to +* XE_BO_PRIORITY_LOW by xe_vma_destroy() after losing all of its VMA +* mappings. Bring it in line with the current priority band here. +*/ +static void op_update_bo_priority (struct xe_vm *vm,struct xe_vma_op *op) +{ +struct xe_bo *bo; +int priority;
drivers/gpu/drm/xe/xe_vm.c static void op_add_ufence(struct xe_vm *vm, struct xe_vma_op *op, · line 3563
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm.c +10
-0+ +if (op->base.op !=DRM_GPUVA_OP_MAP ) +return ; + + bo =xe_vma_bo (op->map.vma); +if (!bo) +return ; + +/* Private BOs are only ever bound to the VM they were created for. */ +xe_assert (vm->xe, !bo->vm || bo->vm == vm);
[Severity: High]
Could this regression allow a data race and TTM LRU list corruption if userspace binds a private BO associated with another VM?
If userspace binds a private BO associated with VM_A to VM_B, the xe_assert() doesn't prevent execution from continuing (and may compile out in non-debug builds).
During the bind operation, vma_lock_and_validate() skips acquiring the BO's resv lock because it assumes any BO with a non-NULL bo->vm shares the target VM's lock:
drivers/gpu/drm/xe/xe_vm.c:vma_lock_and_validate() {
...
if (bo) {
if (!bo->vm)
err = drm_exec_lock_obj(exec, &bo->ttm.base);
...
}This means op_update_bo_priority() will call xe_bo_update_ttm_priority() to mutate bo->ttm.priority and TTM LRU structures without holding the required bo->ttm.base.resv lock. Can an unprivileged userspace application trigger this by passing a mismatched BO handle to the bind IOCTL?
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c +8
-0+ + priority = bo->vm ?xe_vm_bo_priority (vm) :XE_BO_PRIORITY_HIGHEST ; +if (bo->ttm.priority == priority) +return ; + +xe_vm_assert_held (vm); +xe_bo_update_ttm_priority (bo, priority); +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Matthew Brost
maintainerPATCH v2 4/4 drm/xe: Lower BO TTM priority to lowest when fully unmapped from VM
In reply to Matthew BrostAn UNMAP (or the unmap half of a REMAP) can leave a BO with no remaining VMA mappings anywhere. Such a BO is unlikely to be touched again soon, so rather than leaving it at whatever priority band it had while still in active use, drop it to XE_BO_PRIORITY_LOW, the lowest level, so it becomes one of the first candidates considered for eviction/shrinking. This applies both to BOs private to a VM and to extobjs shared across multiple VMs (bo->vm == NULL) -- an extobj with no bindings left anywhere is just as good an eviction/shrink candidate as an unmapped private BO. Imported bos are excluded, since their vma_count is never tracked (xe_bo_vma_count_inc/dec_locked() are no-ops for them), so it would never reflect anything meaningful.
This has to be done inside xe_vma_destroy() itself, right after xe_bo_vma_count_dec_locked(), rather than by its callers after the
fact: xe_vma_destroy() may drop the last reference to the BO via xe_bo_put() in xe_vma_destroy_late() (called either synchronously, or asynchronously via a fence callback), so a caller that stashed a pointer to the BO before calling xe_vma_destroy() and then dereferences it afterwards risks a use-after-free. Doing the check and update before that possible final put avoids this.
bo->purgeable.vma_count, which xe_bo_vma_count_dec_locked() just updated, tells us whether the BO has become fully unmapped. A REMAP that only shrinks a mapping still leaves vma_count > 0 via the prev/next VMA(s) created earlier in the bind, so those are correctly left untouched. This runs under the BO's dma-resv, already asserted held via xe_bo_assert_held() just above.
--- drivers/gpu/drm/xe/xe_vm.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
drivers/gpu/drm/xe/xe_vm.c +16 -0
@@ -1282,6 +1282,22 @@ static void xe_vma_destroy(struct xe_vma *vma, struct dma_fence *fence)xe_bo_vma_count_dec_locked (bo);if (vma->attr.purgeable_state ==XE_MADV_PURGEABLE_WILLNEED )xe_bo_willneed_put_locked (bo); + +/* +* @bo has just lost a VMA mapping. If it now has none left +* anywhere, it is unlikely to be used again soon, so lower +* its TTM priority to the lowest level, making it one of the +* first buffers considered for eviction/shrinking. This +* applies both to BOs private to @vm and to extobjs shared +* across multiple VMs (bo->vm == NULL), but not to imported +* bos, whose vma_count is never tracked. Must be done here, +* rather than by the caller, as this may be the last +* reference to @bo (see xe_vma_destroy_late()). +*/ +if (!drm_gem_is_imported (&bo->ttm.base) && + !bo->purgeable.vma_count && + bo->ttm.priority !=XE_BO_PRIORITY_LOW ) +xe_bo_update_ttm_priority (bo,XE_BO_PRIORITY_LOW ); }xe_vm_assert_held (vm);
Signature · 1 line
2.34.1