Skip to contentSkip to conversation

drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs

PATCH Superseded Matthew Brost 4 participants 68 messages Jul 10 – Jul 11

Subsystems (7)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Matthew Brost

author maintainer
#0 lore raw

v2 cover drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs

The overarching goal of this series is to keep user-facing IOCTLs snappy by holding the BO dma-resv lock for the shortest possible time, and to push the expensive, best-effort work - page (re)allocation, DMA mapping and placement fixups - into the background or out of the locked critical section entirely.

Two related mechanisms fall out of that goal:

- Out-of-lock backing. Page allocation and DMA mapping are the dominant cost of gem_create and of BO moves, and doing them under the dma-resv lock serializes otherwise-independent clients. This series preallocates the backing pages (and, where applicable, builds the DMA/IOVA mapping) outside the lock, then transfers the result into the BO under a brief lock hold. The user gets a fast IOCTL return; the heavy lifting happens without contending the reservation.

- Page defragmentation. A BO allocated under memory pressure keeps its scattered, sub-optimally-ordered backing for its entire lifetime, costing TLB efficiency forever. TTM grows the plumbing to track order-failure and to re-back a populated BO in place at the beneficial order, and Xe wires up a background delayed worker that promotes such BOs on the GPU once memory is available again - again, off the hot path and without stalling the submitting thread.

Since v1 [1] the series has grown considerably. The bulk of the new material is a direct result of profiling: once the defragmenter was in place, the remaining dma-resv hold times and the per-BO allocation/mapping costs showed up clearly, which motivated the out-of-lock preallocation, the IOVA-based mapping path, and the amdgpu counterpart. Rather than land the defragmenter alone, v2 folds in these optimizations since they share the same infrastructure and the same "hold dma-resv briefly, fix up in the background" architecture.

The series is organized in sections rather than described patch by patch:

- Patches 1-10 (drm/ttm): core TTM preparation - order-failure tracking, the defragmentation move, reclaim backoff, and out-of-lock page preallocation plumbing.
- Patches 11-14: other dependent drm/gpusvm and drm/xe patches this series builds on (DMA-mapping accounting, per-order DMA stats, async L2 flush, and a VM-teardown ordering fix).
- Patches 15-23 (drm/xe): the page defragmenter itself - BO tracking, the on-GPU defrag copy, xe_bo_move() handling, and the background worker with its stats and configuration.
- Patches 24-25 (drm/xe): out-of-lock system BO backing preallocation in gem_create, moving page allocation out of the dma-resv lock.
- Patches 26-32 (drm/xe): IOVA-based DMA mapping optimizations, building and finalizing the mapping outside the lock.
- Patch 33 (drm/amdgpu): the equivalent out-of-lock system BO preallocation for amdgpu, exercising the shared TTM plumbing.

Testing
=======

- 3D benchmarks on Ubuntu and on Android, with memory intentionally fragmented by a separate program at launch (plus beneficial-order error injection). BOs are initially backed at a sub-optimal order and scores start lower; the background defrag worker then promotes the backing to the beneficial order and scores climb back in line with the unfragmented baseline.
- IGT:
https://patchwork.freedesktop.org/patch/739052/?series=170046&rev=2

Matt

[1] https://patchwork.freedesktop.org/series/169053/

Cc Alex Deucher <[email protected]>
Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>

Matthew Brost (31):
drm/ttm/pool: Allow backing off reclaim at the beneficial order drm/ttm/pool: Add ttm_pool_page_order_nodma() helper drm/ttm: Record sub-optimal page order allocations in ttm_tt drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc() drm/ttm: Support defragmentation moves
drm/ttm: Add fault injection for beneficial-order allocation failures drm/ttm: Harvest beneficial-order pages on defragmentation moves drm/ttm: Bound page (re)allocation per defragmentation move drm/ttm: Preallocate beneficial-order defrag pages outside the lock drm/ttm: Add full out-of-lock preallocation for ttm_pool_alloc() drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind() drm/xe: Destroy page tables after unlinking all VMAs on VM close drm/xe: Track BOs backed at a sub-optimal page order drm/xe: Back off beneficial-order reclaim under defrag pressure drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies drm/xe: Handle defrag moves in xe_bo_move()
drm/xe: Skip self-copies for borrowed pages on defrag moves drm/xe: Add a page defragmentation worker
drm/xe: Add defrag GT stats
drm/xe: Add Kconfig.profile options for BO defrag configuration drm/xe: Defrag using out-of-lock page preallocation drm/xe: Add defrag profiling tracepoints
drm/xe: Preallocate system BO backing outside the dma-resv lock drm/xe: Add tracepoint for xe_gem_create_ioctl
drm/xe: Add IOVA-based xe_res_cursor variant
drm/xe: Use IOVA-based DMA mapping for eligible tt BOs drm/xe: Add per-device dependency scheduler for IOVA defrag finalize drm/xe: Add packed copy-step IOVA mapping for defrag drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies drm/xe: Finalize defrag-IOVA moves with post-copy job drm/amdgpu: Preallocate system BO pages outside the reservation lock

Thomas Hellström (2):
drm/gpusvm: Add a DMA-mapping accounting callback drm/xe: Add debugfs stats for DMA-mapped pages per order

 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |    4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c   |    2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       |   45 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h       |    5 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c    |    3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |    4 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       |   39 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h       |    3 +
 drivers/gpu/drm/drm_gpusvm.c                  |   17 +-
 drivers/gpu/drm/ttm/ttm_bo.c                  |  101 +-
 drivers/gpu/drm/ttm/ttm_bo_util.c             |   21 +-
 drivers/gpu/drm/ttm/ttm_pool.c                |  931 +++++++++-
 drivers/gpu/drm/ttm/ttm_tt.c                  |   67 +
 drivers/gpu/drm/xe/Kconfig.profile            |   40 +
 drivers/gpu/drm/xe/tests/xe_bo.c              |    8 +-
 drivers/gpu/drm/xe/tests/xe_dma_buf.c         |    2 +-
 drivers/gpu/drm/xe/tests/xe_migrate.c         |   12 +-
 drivers/gpu/drm/xe/xe_bo.c                    | 1632 ++++++++++++++++-
 drivers/gpu/drm/xe/xe_bo.h                    |   15 +-
 drivers/gpu/drm/xe/xe_bo_types.h              |    6 +
 drivers/gpu/drm/xe/xe_debugfs.c               |   26 +
 drivers/gpu/drm/xe/xe_device.c                |   35 +
 drivers/gpu/drm/xe/xe_device_types.h          |   57 +
 drivers/gpu/drm/xe/xe_dma_buf.c               |    2 +-
 drivers/gpu/drm/xe/xe_ggtt.c                  |    2 +-
 drivers/gpu/drm/xe/xe_gt_stats.c              |    5 +
 drivers/gpu/drm/xe/xe_gt_stats_types.h        |   17 +
 drivers/gpu/drm/xe/xe_migrate.c               |  531 +++++-
 drivers/gpu/drm/xe/xe_migrate.h               |   17 +
 drivers/gpu/drm/xe/xe_pt.c                    |    2 +-
 drivers/gpu/drm/xe/xe_res_cursor.h            |   56 +-
 drivers/gpu/drm/xe/xe_svm.c                   |   36 +-
 drivers/gpu/drm/xe/xe_svm.h                   |    3 +-
 drivers/gpu/drm/xe/xe_trace_bo.h              |   90 +
 drivers/gpu/drm/xe/xe_userptr.c               |   55 +
 drivers/gpu/drm/xe/xe_userptr.h               |    1 +
 drivers/gpu/drm/xe/xe_vm.c                    |  149 +-
 drivers/gpu/drm/xe/xe_vm.h                    |    5 +
 include/drm/drm_gpusvm.h                      |   19 +
 include/drm/ttm/ttm_bo.h                      |   64 +
 include/drm/ttm/ttm_pool.h                    |   46 +
 include/drm/ttm/ttm_tt.h                      |   43 +-
 42 files changed, 4029 insertions(+), 189 deletions(-)
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

PATCH v2 1/33 drm/ttm/pool: Allow backing off reclaim at the beneficial order

Allocating at the pool's beneficial order normally uses __GFP_RETRY_MAYFAIL to push hard for a contiguous high-order page. When beneficial-order allocations have recently been failing, this retry just burns CPU cycles in reclaim/compaction before falling back to a smaller order anyway.

Add a beneficial_reclaim_backoff flag to struct ttm_operation_ctx and plumb it through to ttm_pool_alloc_page(). When set, the beneficial order is treated like the orders we already consider not worth stalling
for: __GFP_RECLAIM is cleared so the allocation skips reclaim/compaction entirely, and the __GFP_NORETRY clearing / __GFP_RETRY_MAYFAIL promotion is skipped. Together this makes the allocator back off quickly to a smaller order instead of stalling.

This is a no-op until a caller opts in.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 12 ++++++++----
 include/drm/ttm/ttm_bo.h       |  7 +++++++
 2 files changed, 15 insertions(+), 4 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +8 -4
index 3d5f2ae0a456..dbe977412a81 100644
@@ -145,7 +145,8 @@ static int ttm_pool_nid(struct ttm_pool *pool)
 
 /* Allocate pages of size 1 << order with the given gfp_flags */
 static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
-					unsigned int order)
+					unsigned int order,
+					bool beneficial_reclaim_backoff)
 {
 	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
 	unsigned long attr = DMA_ATTR_FORCE_CONTIGUOUS;
@@ -165,10 +166,12 @@ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
 	 * Do not add latency to the allocation path for allocations orders
 	 * device tolds us do not bring them additional performance gains.
 	 */
-	if (order && beneficial_order && order != beneficial_order)
+	if (order && (beneficial_reclaim_backoff ||
+		      (beneficial_order && order != beneficial_order)))
 		gfp_flags &= ~__GFP_RECLAIM;
 
-	if (beneficial_order && order == beneficial_order) {
+	if (!beneficial_reclaim_backoff &&
+	    beneficial_order && order == beneficial_order) {
 		gfp_flags &= ~__GFP_NORETRY;
 		gfp_flags |= __GFP_RETRY_MAYFAIL;
 	}
@@ -814,7 +817,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		if (!p) {
 			page_caching = ttm_cached;
 			allow_pools = false;
-			p = ttm_pool_alloc_page(pool, gfp_flags, order);
+			p = ttm_pool_alloc_page(pool, gfp_flags, order,
+						ctx->beneficial_reclaim_backoff);
 		}
 		/* If that fails, lower the order if possible and retry. */
 		if (!p) {
include/drm/ttm/ttm_bo.h +7 -0
index 8310bc3d55f9..fd814b5cf535 100644
@@ -187,6 +187,13 @@ struct ttm_operation_ctx {
 	 * when multiple BOs share the same reservation object @resv.
 	 */
 	bool allow_res_evict;
+	/**
+	 * @beneficial_reclaim_backoff: Back off from direct reclaim at the
+	 * pool's beneficial order. Useful when beneficial-order allocation has
+	 * recently been failing, to avoid burning CPU cycles in
+	 * reclaim/compaction before falling back to a smaller order anyway.
+	 */
+	bool beneficial_reclaim_backoff;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH v2 01/33] drm/ttm/pool: Allow backing off reclaim at the beneficial order

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] `beneficial_reclaim_backoff` unintentionally disables reclaim for all non-zero orders if the…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] beneficial_reclaim_backoff unintentionally disables reclaim for all non-zero orders if the pool has no beneficial order. --

--- Patch [1]: [PATCH v2 01/33] drm/ttm/pool: Allow backing off reclaim at the beneficial order --- commit 33813073a08d456967f60dbd0f79ceb761550bda
Author: Matthew Brost <[email protected]>

drm/ttm/pool: Allow backing off reclaim at the beneficial order

This commit adds a beneficial_reclaim_backoff flag to struct ttm_operation_ctx. When set, it allows the allocator to skip reclaim and compaction at the pool's beneficial order, falling back to smaller orders quickly instead of stalling.

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_pool.c +0 -0
index 3d5f2ae0a4564..dbe977412a811 100644

[ ... ]

drivers/gpu/drm/ttm/ttm_pool.c static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags, · line 171

Quoted 8 lines
drivers/gpu/drm/ttm/ttm_pool.c +2 -1
@@ -165,10 +166,12 @@ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
 	 * Do not add latency to the allocation path for allocations orders
 	 * device tolds us do not bring them additional performance gains.
 	 */
-	if (order && beneficial_order && order != beneficial_order)
+	if (order && (beneficial_reclaim_backoff ||
+		      (beneficial_order && order != beneficial_order)))
 		gfp_flags &= ~__GFP_RECLAIM;

[Severity: Medium]
Does this unintentionally disable reclaim for all non-zero orders if the pool has no configured beneficial order?

If a caller sets ctx->beneficial_reclaim_backoff = true for an allocation on a TTM pool where ttm_pool_beneficial_order(pool) == 0, the condition above evaluates to true for any order > 0. This unconditionally clears __GFP_RECLAIM.

High-order page allocations would then completely skip direct and kswapd reclaim, causing them to fail prematurely and fall back to order-0 allocations. This seems to defeat the purpose of high-order pools when no beneficial order is specified.

Should this condition explicitly require beneficial_order to be non-zero when evaluating beneficial_reclaim_backoff?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
maintainer
#3 lore raw

Re: [PATCH v2 01/33] drm/ttm/pool: Allow backing off reclaim at the beneficial order

In reply to Matthew Brost
Quoted 6 lines · Matthew Brost

Allocating at the pool's beneficial order normally uses __GFP_RETRY_MAYFAIL to push hard for a contiguous high-order page. When beneficial-order allocations have recently been failing, this retry just burns CPU cycles in reclaim/compaction before falling back to a smaller order anyway.

Quoted 7 lines

Add a beneficial_reclaim_backoff flag to struct ttm_operation_ctx and plumb it through to ttm_pool_alloc_page(). When set, the beneficial order is treated like the orders we already consider not worth stalling
for: __GFP_RECLAIM is cleared so the allocation skips reclaim/compaction entirely, and the __GFP_NORETRY clearing / __GFP_RETRY_MAYFAIL promotion is skipped. Together this makes the allocator back off quickly to a smaller order instead of stalling.

That makes no sense at all.

You first give a beneficial order from the driver and then later overwrite it.

What could be done is to make the beneficial order part of the context in the first place, but I don't get what the use case for this should be.

It looks like you propose that TTM should accept smaller chunks of memory initially and then comes back later on and re-allocates things in huge pages.

If that is really the case then I would clearly reject the whole idea. De-fragmentation of smaller pages into larger ones is the job of the core memory management and *not* TTM.

Regards,
Christian.

Quoted 80 lines

This is a no-op until a caller opts in.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 12 ++++++++----
 include/drm/ttm/ttm_bo.h       |  7 +++++++
 2 files changed, 15 insertions(+), 4 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +8 -4
index 3d5f2ae0a456..dbe977412a81 100644
@@ -145,7 +145,8 @@ static int ttm_pool_nid(struct ttm_pool *pool)
 
 /* Allocate pages of size 1 << order with the given gfp_flags */
 static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
-					unsigned int order)
+					unsigned int order,
+					bool beneficial_reclaim_backoff)
 {
 	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
 	unsigned long attr = DMA_ATTR_FORCE_CONTIGUOUS;
@@ -165,10 +166,12 @@ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
 	 * Do not add latency to the allocation path for allocations orders
 	 * device tolds us do not bring them additional performance gains.
 	 */
-	if (order && beneficial_order && order != beneficial_order)
+	if (order && (beneficial_reclaim_backoff ||
+		      (beneficial_order && order != beneficial_order)))
 		gfp_flags &= ~__GFP_RECLAIM;
 
-	if (beneficial_order && order == beneficial_order) {
+	if (!beneficial_reclaim_backoff &&
+	    beneficial_order && order == beneficial_order) {
 		gfp_flags &= ~__GFP_NORETRY;
 		gfp_flags |= __GFP_RETRY_MAYFAIL;
 	}
@@ -814,7 +817,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		if (!p) {
 			page_caching = ttm_cached;
 			allow_pools = false;
-			p = ttm_pool_alloc_page(pool, gfp_flags, order);
+			p = ttm_pool_alloc_page(pool, gfp_flags, order,
+						ctx->beneficial_reclaim_backoff);
 		}
 		/* If that fails, lower the order if possible and retry. */
 		if (!p) {
include/drm/ttm/ttm_bo.h +7 -0
index 8310bc3d55f9..fd814b5cf535 100644
@@ -187,6 +187,13 @@ struct ttm_operation_ctx {
 	 * when multiple BOs share the same reservation object @resv.
 	 */
 	bool allow_res_evict;
+	/**
+	 * @beneficial_reclaim_backoff: Back off from direct reclaim at the
+	 * pool's beneficial order. Useful when beneficial-order allocation has
+	 * recently been failing, to avoid burning CPU cycles in
+	 * reclaim/compaction before falling back to a smaller order anyway.
+	 */
+	bool beneficial_reclaim_backoff;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
author maintainer
#4 lore raw

Re: [PATCH v2 01/33] drm/ttm/pool: Allow backing off reclaim at the beneficial order

Quoted 15 lines · Christian König
Quoted 6 lines · Matthew Brost

Allocating at the pool's beneficial order normally uses __GFP_RETRY_MAYFAIL to push hard for a contiguous high-order page. When beneficial-order allocations have recently been failing, this retry just burns CPU cycles in reclaim/compaction before falling back to a smaller order anyway.

Quoted 7 lines

Add a beneficial_reclaim_backoff flag to struct ttm_operation_ctx and plumb it through to ttm_pool_alloc_page(). When set, the beneficial order is treated like the orders we already consider not worth stalling
for: __GFP_RECLAIM is cleared so the allocation skips reclaim/compaction entirely, and the __GFP_NORETRY clearing / __GFP_RETRY_MAYFAIL promotion is skipped. Together this makes the allocator back off quickly to a smaller order instead of stalling.

Quoted 6 lines

That makes no sense at all.

You first give a beneficial order from the driver and then later overwrite it.

What could be done is to make the beneficial order part of the context in the first place, but I don't get what the use case for this should be.

In this series, the use case is that BOs have already ended up in the driver-side defragmenter due to suboptimal placement for one reason or another. In my testing, it is fairly easy for higher-order allocations using GFP_RETRY_MAYFAIL to still fail. If the defragmenter already has several BOs queued, there is little point in continuing to try aggressively for higher-order allocations. In that case, it makes sense to back off, accept smaller pages, and fix things up later through defragmentation.

I could drop this part, but in my opinion it does make a reasonable amount of sense.

Quoted 4 lines

It looks like you propose that TTM should accept smaller chunks of memory initially and then comes back later on and re-allocates things in huge pages.

If that is really the case then I would clearly reject the whole idea. De-fragmentation of smaller pages into larger ones is the job of the core memory management and *not* TTM.

Of course the core MM subsystem defragments memory. However, the driver-side defragmenter serves a different purpose. It relies on the core MM making larger allocations available, then coordinates TTM state updates, asynchronous copies, and mapping updates within the execution pipeline using fences.

What the defragmenter solves is the "once placed badly, forever placed badly" problem. Without it, allocations that end up fragmented or otherwise suboptimal tend to remain that way indefinitely. The defragmenter allows those allocations to be reorganized over time as better placement opportunities become available.

TTM is simply a transaction layer that does what the policy layer (the driver) instructs it to do. It is not responsible for making policy decisions itself, nor should it attempt to impose them on individual drivers.

Matt

Quoted 84 lines

Regards,
Christian.

Quoted 80 lines

This is a no-op until a caller opts in.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 12 ++++++++----
 include/drm/ttm/ttm_bo.h       |  7 +++++++
 2 files changed, 15 insertions(+), 4 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +8 -4
index 3d5f2ae0a456..dbe977412a81 100644
@@ -145,7 +145,8 @@ static int ttm_pool_nid(struct ttm_pool *pool)
 
 /* Allocate pages of size 1 << order with the given gfp_flags */
 static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
-					unsigned int order)
+					unsigned int order,
+					bool beneficial_reclaim_backoff)
 {
 	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
 	unsigned long attr = DMA_ATTR_FORCE_CONTIGUOUS;
@@ -165,10 +166,12 @@ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
 	 * Do not add latency to the allocation path for allocations orders
 	 * device tolds us do not bring them additional performance gains.
 	 */
-	if (order && beneficial_order && order != beneficial_order)
+	if (order && (beneficial_reclaim_backoff ||
+		      (beneficial_order && order != beneficial_order)))
 		gfp_flags &= ~__GFP_RECLAIM;
 
-	if (beneficial_order && order == beneficial_order) {
+	if (!beneficial_reclaim_backoff &&
+	    beneficial_order && order == beneficial_order) {
 		gfp_flags &= ~__GFP_NORETRY;
 		gfp_flags |= __GFP_RETRY_MAYFAIL;
 	}
@@ -814,7 +817,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		if (!p) {
 			page_caching = ttm_cached;
 			allow_pools = false;
-			p = ttm_pool_alloc_page(pool, gfp_flags, order);
+			p = ttm_pool_alloc_page(pool, gfp_flags, order,
+						ctx->beneficial_reclaim_backoff);
 		}
 		/* If that fails, lower the order if possible and retry. */
 		if (!p) {
include/drm/ttm/ttm_bo.h +7 -0
index 8310bc3d55f9..fd814b5cf535 100644
@@ -187,6 +187,13 @@ struct ttm_operation_ctx {
 	 * when multiple BOs share the same reservation object @resv.
 	 */
 	bool allow_res_evict;
+	/**
+	 * @beneficial_reclaim_backoff: Back off from direct reclaim at the
+	 * pool's beneficial order. Useful when beneficial-order allocation has
+	 * recently been failing, to avoid burning CPU cycles in
+	 * reclaim/compaction before falling back to a smaller order anyway.
+	 */
+	bool beneficial_reclaim_backoff;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
author maintainer
#5 lore raw

PATCH v2 2/33 drm/ttm/pool: Add ttm_pool_page_order_nodma() helper

In reply to Matthew Brost

TTM records the allocation order of a multi-order page in page->private at allocation time. For pools that do not use dma_alloc the order is stored directly, while dma_alloc pools store a struct ttm_pool_dma pointer there instead. Drivers that do their own DMA mapping (e.g. Xe) need the per-page order to walk a populated ttm_tt one chunk at a time, but cannot rely on folio_order(): TTM allocates high-order pages with alloc_pages_node() without __GFP_COMP, so they are not compound and folio_order() always returns 0.

Expose ttm_pool_page_order_nodma(), a thin wrapper that returns the order recorded in page->private. It is only valid for pages from a pool that does not use dma_alloc; whether a TTM device uses dma_alloc is fixed at ttm device init time, so callers know from their device configuration that this helper applies. Use it from the existing internal ttm_pool_page_order() to keep a single source of truth.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 25 ++++++++++++++++++++++++-
 include/drm/ttm/ttm_pool.h     |  2 ++
 2 files changed, 26 insertions(+), 1 deletion(-)
drivers/gpu/drm/ttm/ttm_pool.c +24 -1
index dbe977412a81..d34592d4dbc7 100644
@@ -461,6 +461,29 @@ static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free)
 	return num_pages;
 }
 
+/**
+ * ttm_pool_page_order_nodma() - Allocation order of a non-dma_alloc TTM page
+ * @p: The page to query.
+ *
+ * Return the allocation order that the TTM pool allocator recorded in
+ * @p->private at allocation time. This only works for pages that were
+ * allocated from a pool which does *not* use dma_alloc (i.e. pages backed by
+ * alloc_pages_node()), where TTM stores the order directly in page->private.
+ *
+ * For dma_alloc pools, page->private instead holds a struct ttm_pool_dma
+ * pointer and this helper must not be used. Whether a TTM device uses
+ * dma_alloc is fixed at ttm device init time, so callers are expected to know
+ * from their TTM device configuration that their pages are not dma_alloc
+ * backed before using this helper.
+ *
+ * Return: The allocation order of the page.
+ */
+unsigned int ttm_pool_page_order_nodma(struct page *p)
+{
+	return p->private;
+}
+EXPORT_SYMBOL(ttm_pool_page_order_nodma);
+
 /* Return the allocation order based for a page */
 static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
 {
@@ -470,7 +493,7 @@ static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
 		return dma->vaddr & ~PAGE_MASK;
 	}
 
-	return p->private;
+	return ttm_pool_page_order_nodma(p);
 }
 
 /*
include/drm/ttm/ttm_pool.h +2 -0
index 26ee592e1994..753203980e2c 100644
@@ -94,6 +94,8 @@ long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *ttm,
 int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			       const struct ttm_operation_ctx *ctx);
 
+unsigned int ttm_pool_page_order_nodma(struct page *p);
+
 int ttm_pool_mgr_init(unsigned long num_pages);
 void ttm_pool_mgr_fini(void);
 
Signature · 1 line
2.34.1
maintainer
#6 lore raw

Re: [PATCH v2 02/33] drm/ttm/pool: Add ttm_pool_page_order_nodma() helper

Quoted 8 lines · Matthew Brost

TTM records the allocation order of a multi-order page in page->private at allocation time. For pools that do not use dma_alloc the order is stored directly, while dma_alloc pools store a struct ttm_pool_dma pointer there instead. Drivers that do their own DMA mapping (e.g. Xe) need the per-page order to walk a populated ttm_tt one chunk at a time, but cannot rely on folio_order(): TTM allocates high-order pages with alloc_pages_node() without __GFP_COMP, so they are not compound and folio_order() always returns 0.

Well what is the justification of XE doing the DMA mapping themselves?

Regards,
Christian.

Quoted 84 lines

Expose ttm_pool_page_order_nodma(), a thin wrapper that returns the order recorded in page->private. It is only valid for pages from a pool that does not use dma_alloc; whether a TTM device uses dma_alloc is fixed at ttm device init time, so callers know from their device configuration that this helper applies. Use it from the existing internal ttm_pool_page_order() to keep a single source of truth.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 25 ++++++++++++++++++++++++-
 include/drm/ttm/ttm_pool.h     |  2 ++
 2 files changed, 26 insertions(+), 1 deletion(-)
drivers/gpu/drm/ttm/ttm_pool.c +24 -1
index dbe977412a81..d34592d4dbc7 100644
@@ -461,6 +461,29 @@ static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free)
 	return num_pages;
 }
 
+/**
+ * ttm_pool_page_order_nodma() - Allocation order of a non-dma_alloc TTM page
+ * @p: The page to query.
+ *
+ * Return the allocation order that the TTM pool allocator recorded in
+ * @p->private at allocation time. This only works for pages that were
+ * allocated from a pool which does *not* use dma_alloc (i.e. pages backed by
+ * alloc_pages_node()), where TTM stores the order directly in page->private.
+ *
+ * For dma_alloc pools, page->private instead holds a struct ttm_pool_dma
+ * pointer and this helper must not be used. Whether a TTM device uses
+ * dma_alloc is fixed at ttm device init time, so callers are expected to know
+ * from their TTM device configuration that their pages are not dma_alloc
+ * backed before using this helper.
+ *
+ * Return: The allocation order of the page.
+ */
+unsigned int ttm_pool_page_order_nodma(struct page *p)
+{
+	return p->private;
+}
+EXPORT_SYMBOL(ttm_pool_page_order_nodma);
+
 /* Return the allocation order based for a page */
 static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
 {
@@ -470,7 +493,7 @@ static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
 		return dma->vaddr & ~PAGE_MASK;
 	}
 
-	return p->private;
+	return ttm_pool_page_order_nodma(p);
 }
 
 /*
include/drm/ttm/ttm_pool.h +2 -0
index 26ee592e1994..753203980e2c 100644
@@ -94,6 +94,8 @@ long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *ttm,
 int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			       const struct ttm_operation_ctx *ctx);
 
+unsigned int ttm_pool_page_order_nodma(struct page *p);
+
 int ttm_pool_mgr_init(unsigned long num_pages);
 void ttm_pool_mgr_fini(void);
 
author maintainer
#7 lore raw

Re: [PATCH v2 02/33] drm/ttm/pool: Add ttm_pool_page_order_nodma() helper

Quoted 10 lines · Christian König
Quoted 8 lines · Matthew Brost

TTM records the allocation order of a multi-order page in page->private at allocation time. For pools that do not use dma_alloc the order is stored directly, while dma_alloc pools store a struct ttm_pool_dma pointer there instead. Drivers that do their own DMA mapping (e.g. Xe) need the per-page order to walk a populated ttm_tt one chunk at a time, but cannot rely on folio_order(): TTM allocates high-order pages with alloc_pages_node() without __GFP_COMP, so they are not compound and folio_order() always returns 0.

Quoted 2 lines

Well what is the justification of XE doing the DMA mapping themselves?

This maps to TTM_ALLOCATION_POOL_USE_DMA_ALLOC, which is clear in Xe, so we use this helper function to determine the page order in various places. I exported this because I'd want to have fish out the pool and use ttm_pool_page_order.

That said, Xe also owns the DMA mappings. To be honest, I didn't realize ttm_sg_tt_init() was commonly used, but having Xe manage DMA mappings has several advantages (I believe we inherited this part from Faith / Dave PoC work on Xe):

- Using an SG list or IOVA allocation gives smaller pages a contiguous address space, which can then be mapped using larger GPU pages.
- SG list or IOVA mapping requires only a single IOMMU sync for a collection of pages, whereas each dma_map_page() or dma_unmap_page call requires its own IOMMU synchronization.
- Using IOVA mapping (later in the series) allows individual page linkages to be replaced without modifying the IOVA programmed into the GPU page tables.

The second point is a fairly significant performance issue. I recently switched GPUSVM over to dma_iova_alloc(), and if I recall correctly, a chain of mappings with a single synchronization was roughly 100x faster than individual dma_map_page() calls for a 2 MiB region backed by 4 KiB pages.

It should not be quite as severe in TTM, since it generally does not use lower-order pages, but mapping something like a 32 MiB BO will still be significantly faster when using an SG list or IOVA allocation.

The TTM core should really be fixed in this area, or drivers should at least be encouraged not to use this path, since implementing the mapping on the driver side is fairly trivial.

Matt

Quoted 88 lines

Regards,
Christian.

Quoted 84 lines

Expose ttm_pool_page_order_nodma(), a thin wrapper that returns the order recorded in page->private. It is only valid for pages from a pool that does not use dma_alloc; whether a TTM device uses dma_alloc is fixed at ttm device init time, so callers know from their device configuration that this helper applies. Use it from the existing internal ttm_pool_page_order() to keep a single source of truth.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 25 ++++++++++++++++++++++++-
 include/drm/ttm/ttm_pool.h     |  2 ++
 2 files changed, 26 insertions(+), 1 deletion(-)
drivers/gpu/drm/ttm/ttm_pool.c +24 -1
index dbe977412a81..d34592d4dbc7 100644
@@ -461,6 +461,29 @@ static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free)
 	return num_pages;
 }
 
+/**
+ * ttm_pool_page_order_nodma() - Allocation order of a non-dma_alloc TTM page
+ * @p: The page to query.
+ *
+ * Return the allocation order that the TTM pool allocator recorded in
+ * @p->private at allocation time. This only works for pages that were
+ * allocated from a pool which does *not* use dma_alloc (i.e. pages backed by
+ * alloc_pages_node()), where TTM stores the order directly in page->private.
+ *
+ * For dma_alloc pools, page->private instead holds a struct ttm_pool_dma
+ * pointer and this helper must not be used. Whether a TTM device uses
+ * dma_alloc is fixed at ttm device init time, so callers are expected to know
+ * from their TTM device configuration that their pages are not dma_alloc
+ * backed before using this helper.
+ *
+ * Return: The allocation order of the page.
+ */
+unsigned int ttm_pool_page_order_nodma(struct page *p)
+{
+	return p->private;
+}
+EXPORT_SYMBOL(ttm_pool_page_order_nodma);
+
 /* Return the allocation order based for a page */
 static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
 {
@@ -470,7 +493,7 @@ static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
 		return dma->vaddr & ~PAGE_MASK;
 	}
 
-	return p->private;
+	return ttm_pool_page_order_nodma(p);
 }
 
 /*
include/drm/ttm/ttm_pool.h +2 -0
index 26ee592e1994..753203980e2c 100644
@@ -94,6 +94,8 @@ long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *ttm,
 int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			       const struct ttm_operation_ctx *ctx);
 
+unsigned int ttm_pool_page_order_nodma(struct page *p);
+
 int ttm_pool_mgr_init(unsigned long num_pages);
 void ttm_pool_mgr_fini(void);
 
author maintainer
#8 lore raw

PATCH v2 3/33 drm/ttm: Record sub-optimal page order allocations in ttm_tt

In reply to Matthew Brost

When __ttm_pool_alloc() fails to allocate a chunk at the device's beneficial order and falls back to a smaller order, the object ends up backed by a sub-optimal set of pages. Nothing currently records this, so a driver has no way to know an object would benefit from being re-backed with beneficial-order pages later.

Add TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED to struct ttm_tt page_flags, cleared at the start of every fresh allocation (in both ttm_pool_alloc() and ttm_pool_restore_and_alloc()). Set it in __ttm_pool_alloc() when an allocation at exactly the beneficial order fails before dropping to a lower order.

Drivers can use this hint to queue the buffer object for later defragmentation.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 13 +++++++++++++
 include/drm/ttm/ttm_tt.h       | 24 +++++++++++++++++++++++-
 2 files changed, 36 insertions(+), 1 deletion(-)
drivers/gpu/drm/ttm/ttm_pool.c +13 -0
index d34592d4dbc7..370d991c9882 100644
@@ -796,6 +796,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			    struct ttm_pool_alloc_state *alloc,
 			    struct ttm_pool_tt_restore *restore)
 {
+	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
 	enum ttm_caching page_caching;
 	gfp_t gfp_flags = GFP_USER;
 	pgoff_t caching_divide;
@@ -846,6 +847,16 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		/* If that fails, lower the order if possible and retry. */
 		if (!p) {
 			if (order) {
+				/*
+				 * Failing to allocate at the device's beneficial
+				 * order means we are about to back this object
+				 * with a sub-optimal (smaller order) set of
+				 * pages. Record it so the driver can later try to
+				 * defragment the object back to beneficial order.
+				 */
+				if (beneficial_order && order == beneficial_order)
+					tt->page_flags |=
+						TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 				--order;
 				page_caching = tt->caching;
 				allow_pools = true;
@@ -910,6 +921,7 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (WARN_ON(ttm_tt_is_backed_up(tt)))
 		return -EINVAL;
 
+	tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 	ttm_pool_alloc_state_init(tt, &alloc);
 
 	return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
@@ -942,6 +954,7 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (!restore) {
 		gfp_t gfp = GFP_KERNEL | __GFP_NOWARN;
 
+		tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 		ttm_pool_alloc_state_init(tt, &alloc);
 		if (ctx->gfp_retry_mayfail)
 			gfp |= __GFP_RETRY_MAYFAIL;
include/drm/ttm/ttm_tt.h +23 -1
index 406437ad674b..ce7533677d77 100644
@@ -90,6 +90,14 @@ struct ttm_tt {
 	 * TTM_TT_FLAG_BACKED_UP: TTM internal only. This is set if the
 	 * struct ttm_tt has been (possibly partially) backed up.
 	 *
+	 * TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED: Set by the TTM pool allocator
+	 * when at least one chunk that could have been allocated at the pool's
+	 * beneficial order had to fall back to a smaller order. This indicates
+	 * that a sub-optimal set of pages was chosen for this object, and can
+	 * be used by drivers to queue the buffer object for later
+	 * defragmentation. Drivers should access this via the
+	 * ttm_tt_is_beneficial_order_failed() helper.
+	 *
 	 * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only. DO NOT USE. This is
 	 * set by TTM after ttm_tt_populate() has successfully returned, and is
 	 * then unset when TTM calls ttm_tt_unpopulate().
@@ -101,8 +109,9 @@ struct ttm_tt {
 #define TTM_TT_FLAG_EXTERNAL_MAPPABLE	BIT(3)
 #define TTM_TT_FLAG_DECRYPTED		BIT(4)
 #define TTM_TT_FLAG_BACKED_UP	        BIT(5)
+#define TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED	BIT(6)
 
-#define TTM_TT_FLAG_PRIV_POPULATED	BIT(6)
+#define TTM_TT_FLAG_PRIV_POPULATED	BIT(7)
 	uint32_t page_flags;
 	/** @num_pages: Number of pages in the page array. */
 	uint32_t num_pages;
@@ -179,6 +188,19 @@ static inline void ttm_tt_clear_backed_up(struct ttm_tt *tt)
 	tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
 }
 
+/**
+ * ttm_tt_is_beneficial_order_failed() - Whether the tt is backed at a
+ * sub-optimal page order
+ * @tt: The struct ttm_tt.
+ *
+ * Return: true if the pool allocator had to fall back below the pool's
+ * beneficial order when backing this tt, false otherwise.
+ */
+static inline bool ttm_tt_is_beneficial_order_failed(const struct ttm_tt *tt)
+{
+	return tt->page_flags & TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+}
+
 /**
  * ttm_tt_create
  *
Signature · 1 line
2.34.1
author maintainer
#9 lore raw

PATCH v2 4/33 drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc()

In reply to Matthew Brost

__ttm_pool_alloc() walks the page orders in a single loop whose body is about to grow several defragmentation phases (reusing old pages, a move budget, out-of-lock preallocation and fault injection). Left inline these interleave into one deeply nested block of gotos and combinatorial conditionals that is hard to follow and extend.

Factor the loop's working state into a struct ttm_pool_alloc_iter and pull the per-order page acquisition and order-fallback into small phase helpers, so the main loop reads as a short sequence of named steps. This is a pure refactor with no functional change; subsequent patches extend the iterator with their defrag phases rather than nesting more logic into the loop.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 159 ++++++++++++++++++++++-----------
 1 file changed, 106 insertions(+), 53 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +106 -53
index 370d991c9882..6826c8ae6fc5 100644
@@ -791,18 +791,102 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest,
 	return min_t(unsigned int, highest, __fls(alloc->remaining_pages));
 }
 
+/**
+ * struct ttm_pool_alloc_iter - Working state for the __ttm_pool_alloc() loop
+ *
+ * Bundles the immutable inputs and the mutable per-order state that the
+ * allocation phase helpers share, so the main loop reads as a short sequence
+ * of named phases rather than one deeply nested block.
+ */
+struct ttm_pool_alloc_iter {
+	/* Immutable for the duration of the allocation. */
+	struct ttm_pool *pool;
+	struct ttm_tt *tt;
+	const struct ttm_operation_ctx *ctx;
+	struct ttm_pool_alloc_state *alloc;
+	struct ttm_pool_tt_restore *restore;
+	unsigned int beneficial_order;
+	gfp_t gfp_flags;
+
+	/* Mutated as the loop walks the orders. */
+	unsigned int order;
+	enum ttm_caching page_caching;
+	bool allow_pools;
+	struct page *p;
+};
+
+/*
+ * Acquire a single page for the current order, leaving it in @it->p (NULL on
+ * failure). Tries a same-order pool page, then a fresh system allocation.
+ */
+static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
+{
+	struct ttm_pool_type *pt;
+
+	/* First, try to allocate a page from a pool if one exists. */
+	it->p = NULL;
+	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
+	if (!it->p && pt && it->allow_pools)
+		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
+
+	/*
+	 * If that fails or previously failed, allocate from system. Note that
+	 * this also disallows additional pool allocations using write-back
+	 * cached pools of the same order.
+	 */
+	if (!it->p) {
+		it->page_caching = ttm_cached;
+		it->allow_pools = false;
+		it->p = ttm_pool_alloc_page(it->pool, it->gfp_flags, it->order,
+					    it->ctx->beneficial_reclaim_backoff);
+	}
+}
+
+/*
+ * The current order could not be satisfied. Lower it and retry, recording a
+ * sub-optimal backing (so the driver can later re-defrag) when we drop below
+ * the beneficial order.
+ *
+ * Returns 0 to retry the (lowered) order, or a negative errno.
+ */
+static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
+{
+	if (!it->order)
+		return -ENOMEM;
+
+	/*
+	 * Failing to allocate at the device's beneficial order means we are
+	 * about to back this object with a sub-optimal (smaller order) set of
+	 * pages. Record it so the driver can later try to defragment the object
+	 * back to beneficial order.
+	 */
+	if (it->beneficial_order && it->order == it->beneficial_order)
+		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+
+	it->order--;
+	it->page_caching = it->tt->caching;
+	it->allow_pools = true;
+
+	return 0;
+}
+
 static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			    const struct ttm_operation_ctx *ctx,
 			    struct ttm_pool_alloc_state *alloc,
 			    struct ttm_pool_tt_restore *restore)
 {
-	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
-	enum ttm_caching page_caching;
+	struct ttm_pool_alloc_iter it = {
+		.pool = pool,
+		.tt = tt,
+		.ctx = ctx,
+		.alloc = alloc,
+		.restore = restore,
+		.beneficial_order = ttm_pool_beneficial_order(pool),
+		.page_caching = tt->caching,
+		.allow_pools = true,
+	};
 	gfp_t gfp_flags = GFP_USER;
 	pgoff_t caching_divide;
-	unsigned int order;
-	bool allow_pools;
-	struct page *p;
 	int r;
 
 	WARN_ON(!alloc->remaining_pages || ttm_tt_is_populated(tt));
@@ -819,59 +903,28 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	else
 		gfp_flags |= GFP_HIGHUSER;
 
-	page_caching = tt->caching;
-	allow_pools = true;
-	for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
+	it.gfp_flags = gfp_flags;
+
+	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
-	     order = ttm_pool_alloc_find_order(order, alloc)) {
-		struct ttm_pool_type *pt;
-
-		/* First, try to allocate a page from a pool if one exists. */
-		p = NULL;
-		pt = ttm_pool_select_type(pool, page_caching, order);
-		if (pt && allow_pools)
-			p = ttm_pool_type_take(pt, ttm_pool_nid(pool));
-
-		/*
-		 * If that fails or previously failed, allocate from system.
-		 * Note that this also disallows additional pool allocations using
-		 * write-back cached pools of the same order. Consider removing
-		 * that behaviour.
-		 */
-		if (!p) {
-			page_caching = ttm_cached;
-			allow_pools = false;
-			p = ttm_pool_alloc_page(pool, gfp_flags, order,
-						ctx->beneficial_reclaim_backoff);
-		}
-		/* If that fails, lower the order if possible and retry. */
-		if (!p) {
-			if (order) {
-				/*
-				 * Failing to allocate at the device's beneficial
-				 * order means we are about to back this object
-				 * with a sub-optimal (smaller order) set of
-				 * pages. Record it so the driver can later try to
-				 * defragment the object back to beneficial order.
-				 */
-				if (beneficial_order && order == beneficial_order)
-					tt->page_flags |=
-						TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
-				--order;
-				page_caching = tt->caching;
-				allow_pools = true;
-				continue;
-			}
-			r = -ENOMEM;
-			goto error_free_all;
+	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
+		/* Acquire a page (pool / system) for this order. */
+		ttm_pool_iter_acquire_page(&it);
+		if (!it.p) {
+			r = ttm_pool_iter_lower_order(&it);
+			if (r)
+				goto error_free_all;
+			continue;
 		}
-		r = ttm_pool_page_allocated(pool, order, p, page_caching, alloc,
-					    restore);
+
+		r = ttm_pool_page_allocated(pool, it.order, it.p,
+					    it.page_caching, alloc, restore);
 		if (r)
 			goto error_free_page;
 
 		if (ttm_pool_restore_valid(restore)) {
-			r = ttm_pool_restore_commit(restore, tt->backup, ctx, alloc);
+			r = ttm_pool_restore_commit(restore, tt->backup, ctx,
+						    alloc);
 			if (r)
 				goto error_free_all;
 		}
@@ -887,7 +940,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	return 0;
 
 error_free_page:
-	ttm_pool_free_page(pool, page_caching, order, p, false);
+	ttm_pool_free_page(pool, it.page_caching, it.order, it.p, false);
 
 error_free_all:
 	if (tt->restore)
Signature · 1 line
2.34.1
#10 lore raw

Re: [PATCH v2 04/33] drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc()

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Calling __fls() with 0 causes undefined behavior when an allocation fulfills the exact remaining pages, potentially causing an infinite loop. --

--- Patch [4]: [PATCH v2 04/33] drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc() --- commit 5c371f076caffdc9b759a99aaeaf1a30e3ac5dd7
Author: Matthew Brost <[email protected]>

drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc()

This commit factors the working state of the __ttm_pool_alloc loop into a struct ttm_pool_alloc_iter. It pulls the per-order page acquisition into smaller helpers to simplify the nested block and improve readability.

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_pool.c +0 -0
index 370d991c9882b..6826c8ae6fc56 100644

[ ... ]

drivers/gpu/drm/ttm/ttm_pool.c static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, · line 910

Quoted 13 lines
drivers/gpu/drm/ttm/ttm_pool.c +4 -4
@@ -819,59 +903,28 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	else
 		gfp_flags |= GFP_HIGHUSER;
 
-	page_caching = tt->caching;
-	allow_pools = true;
-	for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
+	it.gfp_flags = gfp_flags;
+
+	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
-	     order = ttm_pool_alloc_find_order(order, alloc)) {
+	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {

[Severity: High]
This is a pre-existing issue, but does this loop increment expression evaluate __fls(0) when the allocation exactly fulfills the request?

If the loop successfully allocates the final needed pages during an iteration, alloc->remaining_pages drops to 0 inside the loop body. The C for-loop evaluates the increment expression before checking the loop condition, which calls ttm_pool_alloc_find_order().

Inside ttm_pool_alloc_find_order(), this directly executes:
min_t(unsigned int, highest, __fls(alloc->remaining_pages))

Calling __fls(0) translates to __builtin_clzl(0), which is explicitly undefined behavior in GCC and Clang.

Could this undefined behavior allow the compiler to assume alloc->remaining_pages is always non-zero because it is passed to __fls, potentially optimizing away the loop termination condition and causing an infinite loop?

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
author maintainer
#11 lore raw

PATCH v2 5/33 drm/ttm: Support defragmentation moves

In reply to Matthew Brost

Add infrastructure to let a driver re-back an already-resident BO with fresh pages (e.g. at a more beneficial order) by reusing the normal move machinery.

Add ctx->defrag, which makes ttm_bo_validate() skip the "current resource already compatible" short-circuit so a move is forced even when placement is unchanged. In ttm_bo_handle_move_mem(), when defragging a populated BO, stash the old tt, create and populate a fresh one (allocating new backing), and hand the old tt to the driver move callback via bo->defrag_old_tt for copying. On success the old tt is unpopulated and freed; if populate or the move fails, the freshly allocated backing is discharged and the BO is restored to its original tt and placement.

The old tt is handed off through bo->defrag_old_tt (reservation-lock protected) rather than the operation context. The pointer describes BO state for the duration of the move, so keeping it on the buffer object is better layering and lets ttm_bo_move_accel_cleanup() pick it up.

Because a defrag move commonly fails (beneficial-order pages are often unavailable) and then leaves the BO untouched on its original backing, defer ttm_bo_unmap_virtual() for the defrag case until the new backing has been populated and the move is about to commit. Ordinary moves still unmap up front. This avoids needlessly zapping and refaulting every userspace mapping on the common failure path where nothing moved.

Teach ttm_bo_move_accel_cleanup()'s ghost path about defrag: instead of tearing the move's source tt down synchronously, hand bo->defrag_old_tt to the ghost object so it is kept populated until the copy fence signals and then unbound and destroyed asynchronously, while the BO keeps its new tt. This lets a driver pipeline defrag teardown behind the GPU copy rather than blocking on it.

This is driver-opt-in and a no-op unless ctx->defrag is set.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_bo.c      | 83 ++++++++++++++++++++++++++++---
 drivers/gpu/drm/ttm/ttm_bo_util.c | 16 +++++-
 drivers/gpu/drm/ttm/ttm_pool.c    |  5 +-
 include/drm/ttm/ttm_bo.h          | 17 +++++++
 4 files changed, 111 insertions(+), 10 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +76 -7
index 3980f376e3ba..d05da4eb1d8a 100644
@@ -125,36 +125,93 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 {
 	struct ttm_device *bdev = bo->bdev;
 	bool old_use_tt, new_use_tt;
+	bool defrag;
 	int ret;
 
 	old_use_tt = !bo->resource || ttm_manager_type(bdev, bo->resource->mem_type)->use_tt;
 	new_use_tt = ttm_manager_type(bdev, mem->mem_type)->use_tt;
 
-	ttm_bo_unmap_virtual(bo);
+	/*
+	 * Only the BO that is the actual defrag target (moved in place via
+	 * ttm_bo_validate(), evict == false) is being defragmented. The same
+	 * ctx->defrag is threaded into evictions triggered while allocating
+	 * that target's new backing (ttm_bo_evict_alloc()), so guard against
+	 * an unrelated victim BO (evict == true) being misclassified as a
+	 * defrag move and needlessly reallocating its backing.
+	 */
+	defrag = ctx->defrag && !evict && new_use_tt && bo->ttm &&
+		 ttm_tt_is_populated(bo->ttm);
+
+	/*
+	 * Tear down userspace mappings up front for ordinary moves. A defrag
+	 * move is likely to leave the BO in place (beneficial-order allocation
+	 * commonly fails), so defer the unmap until the new backing has been
+	 * populated and the move is about to commit. This avoids a needless
+	 * zap and refault of every userspace mapping on the common failure,
+	 * where the BO keeps its original, still-mapped backing.
+	 */
+	if (!defrag)
+		ttm_bo_unmap_virtual(bo);
 
 	/*
 	 * Create and bind a ttm if required.
 	 */
 
 	if (new_use_tt) {
+		struct ttm_tt *defrag_old_tt = NULL;
+
+		/*
+		 * For a defrag move, stash the current populated tt and force a
+		 * fresh one to be created and populated (at beneficial order).
+		 * The old tt is handed to the driver move callback to copy the
+		 * contents from, and freed once the move completes.
+		 */
+		if (defrag) {
+			defrag_old_tt = bo->ttm;
+			bo->ttm = NULL;
+		}
+
 		/* Zero init the new TTM structure if the old location should
 		 * have used one as well.
 		 */
-		ret = ttm_tt_create(bo, old_use_tt);
-		if (ret)
+		ret = ttm_tt_create(bo, old_use_tt && !defrag_old_tt);
+		if (ret) {
+			if (defrag_old_tt)
+				bo->ttm = defrag_old_tt;
 			goto out_err;
+		}
 
-		if (mem->mem_type != TTM_PL_SYSTEM) {
+		if (defrag_old_tt || mem->mem_type != TTM_PL_SYSTEM) {
 			ret = ttm_bo_populate(bo, ctx);
-			if (ret)
+			if (ret) {
+				/*
+				 * Discharge the freshly allocated backing and
+				 * leave the BO on its original tt/placement.
+				 */
+				if (defrag_old_tt) {
+					ttm_tt_destroy(bo->bdev, bo->ttm);
+					bo->ttm = defrag_old_tt;
+				}
 				goto out_err;
+			}
 		}
+
+		bo->defrag_old_tt = defrag_old_tt;
 	}
 
 	ret = dma_resv_reserve_fences(bo->base.resv, 1);
 	if (ret)
 		goto out_err;
 
+	/*
+	 * A defrag move deferred its virtual unmap until the new backing was
+	 * successfully populated; tear the userspace mappings down now, before
+	 * the move swaps in the new pages, so the next fault re-establishes
+	 * them against the new backing.
+	 */
+	if (defrag)
+		ttm_bo_unmap_virtual(bo);
+
 	ret = bdev->funcs->move(bo, evict, ctx, mem, hop);
 	if (ret) {
 		if (ret == -EMULTIHOP)
@@ -162,12 +219,24 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 		goto out_err;
 	}
 
+	if (bo->defrag_old_tt) {
+		ttm_tt_unpopulate(bo->bdev, bo->defrag_old_tt);
+		ttm_tt_destroy(bo->bdev, bo->defrag_old_tt);
+		bo->defrag_old_tt = NULL;
+	}
+
 	ctx->bytes_moved += bo->base.size;
 	return 0;
 
 out_err:
-	if (!old_use_tt)
+	if (bo->defrag_old_tt) {
+		/* Failed defrag move: restore the original backing. */
+		ttm_bo_tt_destroy(bo);
+		bo->ttm = bo->defrag_old_tt;
+		bo->defrag_old_tt = NULL;
+	} else if (!old_use_tt) {
 		ttm_bo_tt_destroy(bo);
+	}
 
 	return ret;
 }
@@ -835,7 +904,7 @@ int ttm_bo_validate(struct ttm_buffer_object *bo,
 	force_space = false;
 	do {
 		/* Check whether we need to move buffer. */
-		if (bo->resource &&
+		if (bo->resource && !ctx->defrag &&
 		    ttm_resource_compatible(bo->resource, placement,
 					    force_space))
 			return 0;
drivers/gpu/drm/ttm/ttm_bo_util.c +14 -2
index 3e3c201a0222..771313162c8d 100644
@@ -626,10 +626,22 @@ static int ttm_bo_move_to_ghost(struct ttm_buffer_object *bo,
 	 * bo to be unbound and destroyed.
 	 */
 
-	if (dst_use_tt)
+	if (bo->defrag_old_tt) {
+		/*
+		 * Defrag move: @bo already points at the freshly allocated,
+		 * beneficial-order tt that the move copied into. Hand the old
+		 * tt (the move's source) to the ghost so it is kept populated
+		 * until @fence signals and then unbound and destroyed, instead
+		 * of being torn down synchronously. @bo keeps its new tt.
+		 */
+		ghost_obj->ttm = bo->defrag_old_tt;
+		ghost_obj->defrag_old_tt = NULL;
+		bo->defrag_old_tt = NULL;
+	} else if (dst_use_tt) {
 		ghost_obj->ttm = NULL;
-	else
+	} else {
 		bo->ttm = NULL;
+	}
 
 	dma_resv_unlock(&ghost_obj->base._resv);
 	ttm_bo_put(ghost_obj);
drivers/gpu/drm/ttm/ttm_pool.c +4 -1
index 6826c8ae6fc5..340855e9bcfe 100644
@@ -860,8 +860,11 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	 * pages. Record it so the driver can later try to defragment the object
 	 * back to beneficial order.
 	 */
-	if (it->beneficial_order && it->order == it->beneficial_order)
+	if (it->beneficial_order && it->order == it->beneficial_order) {
 		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+		if (it->ctx->defrag)
+			return -ENOMEM;
+	}
 
 	it->order--;
 	it->page_caching = it->tt->caching;
include/drm/ttm/ttm_bo.h +17 -0
index fd814b5cf535..f4f8a8c29f04 100644
@@ -135,6 +135,16 @@ struct ttm_buffer_object {
 	 * reservation lock.
 	 */
 	struct sg_table *sg;
+
+	/**
+	 * @defrag_old_tt: Used internally during a defrag move to hand the BO's
+	 * previous, populated struct ttm_tt to the driver's move callback so
+	 * that it can copy the contents into the freshly allocated
+	 * beneficial-order backing. Set by TTM, consumed by the driver; the old
+	 * tt is freed (or restored on failure) by TTM once the move completes.
+	 * Protected by the reservation lock.
+	 */
+	struct ttm_tt *defrag_old_tt;
 };
 
 #define TTM_BO_MAP_IOMEM_MASK 0x80
@@ -194,6 +204,13 @@ struct ttm_operation_ctx {
 	 * reclaim/compaction before falling back to a smaller order anyway.
 	 */
 	bool beneficial_reclaim_backoff;
+	/**
+	 * @defrag: Defragmentation pass. Attempt a move even when the bo's
+	 * current resource is already compatible with the requested placement,
+	 * so that the backing store can be reallocated at the pool's beneficial
+	 * order.
+	 */
+	bool defrag;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
Signature · 1 line
2.34.1
#12 lore raw

Re: [PATCH v2 05/33] drm/ttm: Support defragmentation moves

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] A defragmentation move that requires an intermediate bounce buffer (returns -EMULTIHOP)…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] A defragmentation move that requires an intermediate bounce buffer (returns -EMULTIHOP) permanently leaks the original ttm_tt, loses user data, and exposes uninitialized memory. --

--- Patch [5]: [PATCH v2 05/33] drm/ttm: Support defragmentation moves --- commit 4cef86e1c4adcce1b49f5be21dce811803bbfbeb
Author: Matthew Brost <[email protected]>

drm/ttm: Support defragmentation moves

Add infrastructure to let a driver re-back an already-resident BO with fresh pages (e.g. at a more beneficial order) by reusing the normal move machinery.

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
index 3980f376e3ba4..d05da4eb1d8a4 100644

[ ... ]

Quoted 1 line
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
@@ -125,36 +125,93 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,

[ ... ]

drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,

Quoted 7 lines
drivers/gpu/drm/ttm/ttm_bo.c +3 -0
+	if (defrag)
+		ttm_bo_unmap_virtual(bo);
+
 	ret = bdev->funcs->move(bo, evict, ctx, mem, hop);
 	if (ret) {
 		if (ret == -EMULTIHOP)
 			return ret;

[Severity: Critical]
Does returning here directly skip restoring the original backing?

If bdev->funcs->move() returns -EMULTIHOP during a defragmentation move, we return immediately instead of going to out_err. This leaves bo->ttm pointing to the new unpopulated TT, and bo->defrag_old_tt holding the original populated TT.

When ttm_bo_validate() subsequently handles the multi-hop by calling ttm_bo_bounce_temp_buffer(), it passes the same ctx where ctx->defrag is still true.

Will the nested ttm_bo_handle_move_mem() misclassify this bounce as another defrag move and stash the empty bo->ttm into bo->defrag_old_tt? This appears to overwrite the original TT, permanently leaking it and resulting in the loss of user data while exposing uninitialized memory.

Quoted 8 lines
drivers/gpu/drm/ttm/ttm_bo.c +5 -0
 		goto out_err;
 	}
 
+	if (bo->defrag_old_tt) {
+		ttm_tt_unpopulate(bo->bdev, bo->defrag_old_tt);
+		ttm_tt_destroy(bo->bdev, bo->defrag_old_tt);
+		bo->defrag_old_tt = NULL;
+	}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
author maintainer
#13 lore raw

PATCH v2 6/33 drm/ttm: Add fault injection for beneficial-order allocation failures

In reply to Matthew Brost

Add a beneficial_order_fault_inject debugfs fault attribute, mirroring backup_fault_inject, that forces allocation at (or above) the device's beneficial order to fail in __ttm_pool_alloc(). When triggered, both the pool take and the system page allocation are skipped for that order, so the allocation falls into the existing lower-the-order path that sets the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tt page flag (and bails out with -ENOMEM for defrag moves).

This lets the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tracking and the driver-side defrag path be exercised deterministically, without having to drive the system into real memory fragmentation to provoke a sub-optimal backing.

The knob is registered at /sys/kernel/debug/ttm/beneficial_order_fault_inject and compiles out when CONFIG_FAULT_INJECTION is disabled.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 35 ++++++++++++++++++++++++++++------
 1 file changed, 29 insertions(+), 6 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +29 -6
index 340855e9bcfe..34f533adb8eb 100644
@@ -53,6 +53,7 @@
 #ifdef CONFIG_FAULT_INJECTION
 #include <linux/fault-inject.h>
 static DECLARE_FAULT_ATTR(backup_fault_inject);
+static DECLARE_FAULT_ATTR(beneficial_order_fault_inject);
 #else
 #define should_fail(...) false
 #endif
@@ -812,21 +813,34 @@ struct ttm_pool_alloc_iter {
 	unsigned int order;
 	enum ttm_caching page_caching;
 	bool allow_pools;
+	bool fail_beneficial;
 	struct page *p;
 };
 
 /*
  * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries a same-order pool page, then a fresh system allocation.
+ * failure). Tries a same-order pool page, then a fresh system allocation. Fault
+ * injection can force the beneficial-order paths to "fail".
  */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
 	struct ttm_pool_type *pt;
 
-	/* First, try to allocate a page from a pool if one exists. */
 	it->p = NULL;
+
+	/*
+	 * Fault injection: pretend allocation at (or above) the device's
+	 * beneficial order failed, forcing a sub-optimal backing. Exercises the
+	 * beneficial_order_failed tracking and the driver defrag path without
+	 * driving the system into real fragmentation.
+	 */
+	it->fail_beneficial = IS_ENABLED(CONFIG_FAULT_INJECTION) &&
+		it->beneficial_order && it->order >= it->beneficial_order &&
+		should_fail(&beneficial_order_fault_inject, 1);
+
+	/* First, try to allocate a page from a pool if one exists. */
 	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
-	if (!it->p && pt && it->allow_pools)
+	if (!it->p && pt && it->allow_pools && !it->fail_beneficial)
 		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
 
 	/*
@@ -834,7 +848,7 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 	 * this also disallows additional pool allocations using write-back
 	 * cached pools of the same order.
 	 */
-	if (!it->p) {
+	if (!it->p && !it->fail_beneficial) {
 		it->page_caching = ttm_cached;
 		it->allow_pools = false;
 		it->p = ttm_pool_alloc_page(it->pool, it->gfp_flags, it->order,
@@ -851,6 +865,9 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
  */
 static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 {
+	bool at_beneficial = it->beneficial_order &&
+			     it->order == it->beneficial_order;
+
 	if (!it->order)
 		return -ENOMEM;
 
@@ -860,13 +877,16 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	 * pages. Record it so the driver can later try to defragment the object
 	 * back to beneficial order.
 	 */
-	if (it->beneficial_order && it->order == it->beneficial_order) {
+	if (it->fail_beneficial || at_beneficial) {
 		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 		if (it->ctx->defrag)
 			return -ENOMEM;
 	}
 
-	it->order--;
+	if (it->fail_beneficial)
+		it->order = it->beneficial_order - 1;
+	else
+		it->order--;
 	it->page_caching = it->tt->caching;
 	it->allow_pools = true;
 
@@ -1536,6 +1556,9 @@ int ttm_pool_mgr_init(unsigned long num_pages)
 #ifdef CONFIG_FAULT_INJECTION
 	fault_create_debugfs_attr("backup_fault_inject", ttm_debugfs_root,
 				  &backup_fault_inject);
+	fault_create_debugfs_attr("beneficial_order_fault_inject",
+				  ttm_debugfs_root,
+				  &beneficial_order_fault_inject);
 #endif
 #endif
 
Signature · 1 line
2.34.1
author maintainer
#14 lore raw

PATCH v2 7/33 drm/ttm: Harvest beneficial-order pages on defragmentation moves

In reply to Matthew Brost

A defragmentation move relocates a populated tt into a freshly allocated one so that sub-beneficial-order chunks can be upgraded to the device's beneficial order. Previously, if allocation could not complete entirely at beneficial order, the move failed and all progress was discarded.

Instead, once allocation has successfully upgraded part of the object, the remainder of the old tt is harvested to complete the new tt. Thread the BO's old, still-populated tt into the page allocator via a new ttm_operation_ctx::defrag_old_tt. While allocation is still proceeding, any existing chunk already at (or above) the beneficial order is reused directly instead of being reallocated. If allocation later has to fall back below beneficial_order after making partial progress, the remainder of the old tt is harvested instead of failing the move. This preserves the beneficial-order upgrades already made while leaving the rest of the object unchanged, allowing subsequent defragmentation attempts to make additional forward progress.

Harvesting only borrows the pages; they remain owned by the old tt until the move commits. This keeps every unwind path non-destructive: if the (re)allocation or the move later fails, the new tt is torn down without freeing the borrowed pages and the old tt is restored fully intact. Ownership of the shared pages is transferred to the new tt only once the move has committed. ttm_tt_defrag_disown_borrowed() implements both directions of the transfer by clearing a tt's slots for every page it shares with the other tt (same struct page at the same index), so that tt's teardown leaves the shared pages alone and each page is freed exactly once by its owner:

- on the alloc unwind and the failed-move paths, the new tt disowns the borrowed pages, leaving the old tt as the sole owner;
- on the successful-move paths (synchronous teardown and the pipelined ghost-object teardown), the old tt disowns the now-transferred pages, leaving the new tt as the sole owner.

Detecting shared pages by same-index struct page identity needs no extra per-tt bookkeeping, since two distinct allocations never alias. This is safe with drivers, such as Xe, that build their own per-tt sg table and DMA-map it independently: each tt maps and unmaps its own sg, so sharing the underlying pages between tts is DMA-clean.

This is a no-op unless ctx->defrag_old_tt is set, which only happens on a defragmentation move.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_bo.c      |  20 ++-
 drivers/gpu/drm/ttm/ttm_bo_util.c |   5 +
 drivers/gpu/drm/ttm/ttm_pool.c    | 220 ++++++++++++++++++++++++++++--
 drivers/gpu/drm/ttm/ttm_tt.c      |  67 +++++++++
 include/drm/ttm/ttm_bo.h          |  12 ++
 include/drm/ttm/ttm_tt.h          |   2 +
 6 files changed, 313 insertions(+), 13 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +19 -1
index d05da4eb1d8a..d5082306ab7c 100644
@@ -182,7 +182,14 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 		}
 
 		if (defrag_old_tt || mem->mem_type != TTM_PL_SYSTEM) {
+			/*
+			 * Let the page allocator harvest already
+			 * beneficial-order chunks out of the old tt to back
+			 * the new one, rather than reallocating them.
+			 */
+			ctx->defrag_old_tt = defrag_old_tt;
 			ret = ttm_bo_populate(bo, ctx);
+			ctx->defrag_old_tt = NULL;
 			if (ret) {
 				/*
 				 * Discharge the freshly allocated backing and
@@ -220,6 +227,12 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 	}
 
 	if (bo->defrag_old_tt) {
+		/*
+		 * Successful defrag move: any pages the new tt borrowed from the
+		 * old tt are now owned by the new tt. Disown them from the old
+		 * tt before it is torn down so they are not freed.
+		 */
+		ttm_tt_defrag_disown_borrowed(bo->defrag_old_tt, bo->ttm);
 		ttm_tt_unpopulate(bo->bdev, bo->defrag_old_tt);
 		ttm_tt_destroy(bo->bdev, bo->defrag_old_tt);
 		bo->defrag_old_tt = NULL;
@@ -230,7 +243,12 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 
 out_err:
 	if (bo->defrag_old_tt) {
-		/* Failed defrag move: restore the original backing. */
+		/*
+		 * Failed defrag move: drop any pages the new tt borrowed from
+		 * the old tt (still owned by the old tt) before destroying the
+		 * new tt, then restore the original backing.
+		 */
+		ttm_tt_defrag_disown_borrowed(bo->ttm, bo->defrag_old_tt);
 		ttm_bo_tt_destroy(bo);
 		bo->ttm = bo->defrag_old_tt;
 		bo->defrag_old_tt = NULL;
drivers/gpu/drm/ttm/ttm_bo_util.c +5 -0
index 771313162c8d..7cb718fc97c0 100644
@@ -633,7 +633,12 @@ static int ttm_bo_move_to_ghost(struct ttm_buffer_object *bo,
 		 * tt (the move's source) to the ghost so it is kept populated
 		 * until @fence signals and then unbound and destroyed, instead
 		 * of being torn down synchronously. @bo keeps its new tt.
+		 *
+		 * Any pages the new tt borrowed from the old tt are now owned by
+		 * the new tt; disown them from the old tt so the ghost teardown
+		 * leaves them alone.
 		 */
+		ttm_tt_defrag_disown_borrowed(bo->defrag_old_tt, bo->ttm);
 		ghost_obj->ttm = bo->defrag_old_tt;
 		ghost_obj->defrag_old_tt = NULL;
 		bo->defrag_old_tt = NULL;
drivers/gpu/drm/ttm/ttm_pool.c +208 -12

292 lines not on this page · raw message

drivers/gpu/drm/ttm/ttm_tt.c +67 -0

73 lines not on this page · raw message

include/drm/ttm/ttm_bo.h +12 -0
index f4f8a8c29f04..fcaba5fb8bf7 100644
@@ -211,6 +211,18 @@ struct ttm_operation_ctx {
 	 * order.
 	 */
 	bool defrag;
+	/**
+	 * @defrag_old_tt: During a defragmentation move, the bo's existing
+	 * (populated) tt that is being relocated. The page allocator may
+	 * harvest already beneficial-order chunks from this tt to back the new
+	 * tt instead of reallocating them, avoiding alloc/free churn for large
+	 * buffers. Harvested pages are only borrowed: they stay owned by
+	 * @defrag_old_tt until the move commits, at which point
+	 * ttm_tt_defrag_disown_borrowed() clears the old tt's slots for the
+	 * transferred pages so they are not freed when the old tt is torn down.
+	 * NULL for non-defrag operations.
+	 */
+	struct ttm_tt *defrag_old_tt;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
include/drm/ttm/ttm_tt.h +2 -0
index ce7533677d77..55c35bcf134d 100644
@@ -284,6 +284,8 @@ int ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm,
  */
 void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
 
+void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src);
+
 /**
  * ttm_tt_mark_for_clear - Mark pages for clearing on populate.
  *
Signature · 1 line
2.34.1
#15 lore raw

Re: [PATCH v2 07/33] drm/ttm: Harvest beneficial-order pages on defragmentation moves

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Critical] Defragmentation of DMA-allocated or external TTs causes massive out-of-bounds memory writes…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] Defragmentation of DMA-allocated or external TTs causes massive out-of-bounds memory writes due to misinterpreting page private data as an allocation order.
- [Critical] Misaligned chunks between the old and new TT lead to partial chunk harvesting, resulting in a severe use-after-free.
- [High] If the driver's move callback returns -EMULTIHOP, the defragmentation rollback is bypassed, causing a permanent memory leak of the original buffer object TT. --

--- Patch [7]: [PATCH v2 07/33] drm/ttm: Harvest beneficial-order pages on defragmentation moves --- commit 74e91ed102adee47c82989ab7b4f83fd70da47c2
Author: Matthew Brost <[email protected]>
drm/ttm: Harvest beneficial-order pages on defragmentation moves

A defragmentation move relocates a populated tt into a freshly allocated one so that sub-beneficial-order chunks can be upgraded to the device's beneficial order. Previously, if allocation could not complete entirely at beneficial order, the move failed and all progress was discarded. Instead, once allocation has successfully upgraded part of the object, the remainder of the old tt is harvested to complete the new tt. Thread the BO's old, still-populated tt into the page allocator via a new ttm_operation_ctx::defrag_old_tt.

Quoted 3 lines
drivers/gpu/drm/ttm/ttm_bo.c +0 -0

[ ... ]

drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,

Quoted 12 lines
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
@@ -218,10 +225,18 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 	if (defrag)
 		ttm_bo_unmap_virtual(bo);
 
 	ret = bdev->funcs->move(bo, evict, ctx, mem, hop);
 	if (ret) {
 		if (ret == -EMULTIHOP)
 			return ret;
 		goto out_err;
 	}
 
 	if (bo->defrag_old_tt) {

[Severity: High]
If bdev->funcs->move() returns -EMULTIHOP, does returning immediately bypass the out_err rollback path?

It looks like the original TT remains stashed in bo->defrag_old_tt instead of being restored.

When the move is retried, it appears bo->defrag_old_tt would be overwritten or cleared, causing the original buffer object TT and its unborrowed pages to be permanently leaked.

Quoted 3 lines
drivers/gpu/drm/ttm/ttm_pool.c +0 -0

[ ... ]

Quoted 9 lines
drivers/gpu/drm/ttm/ttm_pool.c +9 -0
+static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
+{
+	unsigned int harvest_order;
+	pgoff_t off;
+	int r;
+
+	if (!it->defrag_old_tt || !it->beneficial_order)
+		return TTM_POOL_ITER_FILL;
+

drivers/gpu/drm/ttm/ttm_pool.c struct ttm_pool_alloc_iter { · line 979

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+	off = it->tt->num_pages - it->alloc->remaining_pages;
+	harvest_order = ttm_pool_defrag_old_order(it->pool, it->defrag_old_tt,
+						  off, &it->old_cursor);
+
+	if (harvest_order < it->beneficial_order &&
+	    it->order >= it->beneficial_order)
+		return TTM_POOL_ITER_FILL;
+
+	it->order = harvest_order;
+	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order, off, it->alloc);

[Severity: Critical]
What happens if the new TT allocation size is unaligned, causing the offset into the old TT to land in the middle of a large chunk?

Could ttm_pool_iter_reuse_old() or ttm_pool_harvest_remaining() harvest a partial chunk, such as just the tail page?

If it harvests a tail page, the page order would be read as 0 since the private data is only populated on the head page.

Quoted 3 lines
drivers/gpu/drm/ttm/ttm_tt.c +0 -0

[ ... ]

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_tt.c +4 -0
+void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src)
+{
+	pgoff_t i = 0, run_start = 0;
+	bool in_run = false;

[ ... ]

drivers/gpu/drm/ttm/ttm_tt.c void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm) · line 491

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_tt.c +4 -0
+	while (i < tt->num_pages) {
+		struct page *sp = src->pages[i];
+		unsigned int order = sp ? ttm_pool_page_order_nodma(sp) : 0;
+		pgoff_t nr = 1UL << order;

[Severity: Critical]
Can defragmentation be initiated on a buffer object that uses a DMA-allocated or external TT?

Looking at ttm_bo_handle_move_mem(), it does not appear to restrict defragmentation to non-DMA pools.

If a DMA-allocated TT is defragmented, wouldn't ttm_pool_page_order_nodma() incorrectly read the DMA pointer stored in the page private data as an integer order?

This would result in a huge bitshift when calculating nr, which is then used in the loop increment or the memset() calls, potentially causing severe out-of-bounds memory writes.

drivers/gpu/drm/ttm/ttm_tt.c void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm) · line 500

Quoted 8 lines
drivers/gpu/drm/ttm/ttm_tt.c +8 -0
+		if (sp && tt->pages[i] == sp) {
+			if (!in_run) {
+				run_start = i;
+				in_run = true;
+			}
+		} else if (in_run) {
+			memset(&tt->pages[run_start], 0,
+			       (i - run_start) * sizeof(*tt->pages));

[Severity: Critical]
Following up on the potential for harvesting an unaligned tail page (noted in ttm_pool.c), if ttm_pool_iter_reuse_old() harvested a tail page, wouldn't this logic clear the pointer in the old TT for that tail page, but leave the head page pointer intact?

When the old TT is eventually destroyed, wouldn't ttm_pool_free_range() free the entire chunk back to the system (since the head page still indicates a large order) while the tail page is still actively mapped and used by the new TT, leading to a severe use-after-free regression?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
author maintainer
#16 lore raw

PATCH v2 8/33 drm/ttm: Bound page (re)allocation per defragmentation move

In reply to Matthew Brost

A defragmentation move reallocates an object's backing at the device's beneficial order and harvests whatever the allocator cannot upgrade from the old tt. For a very large object this can reallocate (and have the driver re-copy) the entire backing in one move, which is expensive and can stall concurrent work for an unbounded stretch.

Add an optional per-move byte budget so a large object can be upgraded in slices across successive moves. Plumb a new ttm_operation_ctx::defrag_bytes_remaining through to the pool allocator:
only pages actually (re)allocated at the beneficial order are charged against it; pages harvested (borrowed) from defrag_old_tt are free and do not count. Once the budget is exhausted the allocator stops upgrading and completes the new tt by harvesting the remainder of the old tt as-is, marking the tt sub-optimal (TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED) so the driver re-queues it and a later move resumes the upgrade. The slices already upgraded come back as free already-beneficial harvests on the next pass, so each move spends budget only on new forward progress.

The allocator decrements the budget as it allocates and writes the unused remainder back on success, letting the caller compute how many bytes were actually spent. A value of 0 means unlimited (upgrade the whole object in one move), preserving the previous behaviour. This is only consulted on a defragmentation move, i.e. when defrag_old_tt is set.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 101 ++++++++++++++++++++++++++++++++-
 include/drm/ttm/ttm_bo.h       |  14 +++++
 2 files changed, 112 insertions(+), 3 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +98 -3
index 5d2bbcce4eb8..a6847a16c47f 100644
@@ -77,6 +77,11 @@ struct ttm_pool_dma {
  * @dma_addr: Pointer to the next tt dma_address entry to populate if any.
  * @remaining_pages: Remaining pages to populate.
  * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @defrag_bytes_remaining: Byte budget for newly (re)allocated pages on a
+ * defragmentation move, seeded from ttm_operation_ctx::defrag_bytes_remaining.
+ * Pages harvested (borrowed) from the old tt are free and do not count; once
+ * this drops to zero the allocator stops upgrading and harvests the remainder.
+ * On completion it holds the unused budget. 0 means unlimited (no defrag cap).
  */
 struct ttm_pool_alloc_state {
 	struct page **pages;
@@ -84,6 +89,7 @@ struct ttm_pool_alloc_state {
 	dma_addr_t *dma_addr;
 	pgoff_t remaining_pages;
 	enum ttm_caching tt_caching;
+	s64 defrag_bytes_remaining;
 };
 
 /**
@@ -780,6 +786,7 @@ static void ttm_pool_alloc_state_init(const struct ttm_tt *tt,
 	alloc->dma_addr = tt->dma_address;
 	alloc->remaining_pages = tt->num_pages;
 	alloc->tt_caching = tt->caching;
+	alloc->defrag_bytes_remaining = 0;
 }
 
 /*
@@ -885,14 +892,33 @@ static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
 
 static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 				      struct ttm_tt *old_tt, pgoff_t off,
-				      struct ttm_pool_alloc_state *alloc)
+				      struct ttm_pool_alloc_state *alloc,
+				      bool *suboptimal)
 {
+	unsigned int beneficial = ttm_pool_beneficial_order(pool);
+	pgoff_t bnr = 1UL << beneficial;
+
 	while (alloc->remaining_pages) {
 		struct page *p = old_tt->pages[off];
 		unsigned int order = ttm_pool_page_order(pool, p);
 		pgoff_t nr = 1UL << order;
+		pgoff_t num_pages = off + alloc->remaining_pages;
 		int r;
 
+		/*
+		 * Only report a sub-optimal backing when a chunk we cannot
+		 * upgrade (out of budget/prealloc) sits in a beneficial-order
+		 * aligned region that fully fits in the tt: that region could
+		 * have been upgraded with more budget. A chunk below the
+		 * beneficial order is expected for the unaligned tail (the last
+		 * partial beneficial-order region) and for already beneficial-
+		 * order harvests - neither leaves real defrag work, so they must
+		 * not flag the tt for re-queue.
+		 */
+		if (beneficial && order < beneficial &&
+		    round_down(off, bnr) + bnr <= num_pages)
+			*suboptimal = true;
+
 		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
 		if (r)
 			return r;
@@ -936,6 +962,7 @@ struct ttm_pool_alloc_iter {
 	struct ttm_tt *defrag_old_tt;
 	unsigned int beneficial_order;
 	gfp_t gfp_flags;
+	bool defrag_capped;
 
 	/* Mutated as the loop walks the orders. */
 	unsigned int order;
@@ -947,6 +974,44 @@ struct ttm_pool_alloc_iter {
 	struct page *p;
 };
 
+/*
+ * Defrag move budget exhausted: the upgrade can make no further progress this
+ * pass. Snapshot @defrag_capped is set only when a byte budget was in force at
+ * entry.
+ */
+static bool ttm_pool_iter_defrag_exhausted(const struct ttm_pool_alloc_iter *it)
+{
+	if (!it->defrag_old_tt)
+		return false;
+
+	return it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0;
+}
+
+/*
+ * Complete the new tt by harvesting the rest of the old tt as-is (borrowing its
+ * pages, including any still below beneficial order). Never fall back to in-lock
+ * reclaim for the shortfall - that would reintroduce the stall we preallocated
+ * to avoid. Only re-queue (flag) the tt when the harvested remainder is
+ * genuinely sub-optimal; if every remaining old chunk was already at beneficial
+ * order the move fully upgraded the object.
+ */
+static int ttm_pool_iter_harvest_rest(struct ttm_pool_alloc_iter *it)
+{
+	pgoff_t off = it->tt->num_pages - it->alloc->remaining_pages;
+	bool suboptimal = false;
+	int r;
+
+	r = ttm_pool_harvest_remaining(it->pool, it->defrag_old_tt, off,
+				       it->alloc, &suboptimal);
+	if (r)
+		return r;
+
+	if (suboptimal)
+		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+
+	return 0;
+}
+
 /*
  * Defrag move: reuse an already beneficial-order chunk from the old tt at this
  * offset rather than reallocating it. Harvest whatever order the old chunk has
@@ -1047,11 +1112,12 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 		if (it->alloc_count && it->defrag_old_tt) {
 			pgoff_t off = it->tt->num_pages -
 				      it->alloc->remaining_pages;
+			bool suboptimal = false;
 			int r;
 
 			r = ttm_pool_harvest_remaining(it->pool,
 						       it->defrag_old_tt, off,
-						       it->alloc);
+						       it->alloc, &suboptimal);
 			if (r)
 				return r;
 
@@ -1106,10 +1172,23 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		gfp_flags |= GFP_HIGHUSER;
 
 	it.gfp_flags = gfp_flags;
+	it.defrag_capped = it.defrag_old_tt && alloc->defrag_bytes_remaining > 0;
 
 	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
+		/*
+		 * Out of defrag budget: harvest the rest of the old tt as-is and
+		 * stop (the tt is re-queued if the remainder is still
+		 * sub-optimal).
+		 */
+		if (ttm_pool_iter_defrag_exhausted(&it)) {
+			r = ttm_pool_iter_harvest_rest(&it);
+			if (r)
+				goto error_free_all;
+			break;
+		}
+
 		/* Reuse an already beneficial-order chunk from the old tt. */
 		r = ttm_pool_iter_reuse_old(&it);
 		if (r < 0)
@@ -1140,6 +1219,15 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 				goto error_free_all;
 		}
 
+		/*
+		 * Charge a newly (re)allocated chunk against the defrag move
+		 * budget. Harvested chunks borrow the old tt's pages for free
+		 * and never reach here.
+		 */
+		if (it.defrag_capped)
+			alloc->defrag_bytes_remaining -=
+				(s64)(1UL << it.order) << PAGE_SHIFT;
+
 		it.alloc_count++;
 	}
 
@@ -1189,14 +1277,21 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		   struct ttm_operation_ctx *ctx)
 {
 	struct ttm_pool_alloc_state alloc;
+	int ret;
 
 	if (WARN_ON(ttm_tt_is_backed_up(tt)))
 		return -EINVAL;
 
 	tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 	ttm_pool_alloc_state_init(tt, &alloc);
+	alloc.defrag_bytes_remaining = ctx->defrag_bytes_remaining;
+
+	ret = __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
+
+	/* Report the unused defrag budget back to the caller. */
+	ctx->defrag_bytes_remaining = alloc.defrag_bytes_remaining;
 
-	return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
+	return ret;
 }
 EXPORT_SYMBOL(ttm_pool_alloc);
 
include/drm/ttm/ttm_bo.h +14 -0
index fcaba5fb8bf7..6124d2627b47 100644
@@ -223,6 +223,20 @@ struct ttm_operation_ctx {
 	 * NULL for non-defrag operations.
 	 */
 	struct ttm_tt *defrag_old_tt;
+	/**
+	 * @defrag_bytes_remaining: Byte budget for newly allocated pages during
+	 * a defragmentation move. Only pages that are actually (re)allocated at
+	 * the beneficial order count against it; pages harvested (borrowed) from
+	 * @defrag_old_tt are free and do not. When the budget is exhausted the
+	 * pool allocator stops upgrading and completes the new tt by harvesting
+	 * the remainder of @defrag_old_tt, marking the tt sub-optimal so the
+	 * driver can resume on a later pass. The allocator decrements this as it
+	 * allocates, so on return it holds the unused budget (which the caller
+	 * can use to compute how many bytes were spent). A value of 0 means
+	 * unlimited (upgrade the whole object). Only consulted when
+	 * @defrag_old_tt is set.
+	 */
+	s64 defrag_bytes_remaining;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
Signature · 1 line
2.34.1
#17 lore raw

Re: [PATCH v2 08/33] drm/ttm: Bound page (re)allocation per defragmentation move

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Critical] `ttm_pool_harvest_remaining()` fetches uninitialized `page->private` on tail pages when the…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] ttm_pool_harvest_remaining() fetches uninitialized page->private on tail pages when the defragmentation budget is exhausted, causing a NULL pointer dereference or buddy allocator corruption.
- [Critical] Double-free and Undefined Behavior in ttm_tt_defrag_disown_borrowed() when defragmenting pools that use dma_alloc.
- [Medium] Negative defrag_bytes_remaining disables the defragmentation budget instead of treating it as exhausted, breaking batch budget tracking.
--

--- Patch [8]: [PATCH v2 08/33] drm/ttm: Bound page (re)allocation per defragmentation move --- commit 9447c53b431957799260b5fae41a105a050c3394
Author: Matthew Brost <[email protected]>
Subject: drm/ttm: Bound page (re)allocation per defragmentation move

This commit bounds page reallocations per defragmentation move to avoid stalling concurrent work for unbounded stretches when dealing with very large objects. It introduces an optional per-move byte budget to allow a large object to be upgraded in slices across successive moves.

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_pool.c +0 -0
index 5d2bbcce4eb89..a6847a16c47f1 100644

[ ... ]

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_pool.c +0 -0
@@ -885,14 +892,33 @@ static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
 
 static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 				      struct ttm_tt *old_tt, pgoff_t off,

drivers/gpu/drm/ttm/ttm_pool.c static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order, · line 903

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +5 -1
-				      struct ttm_pool_alloc_state *alloc)
+				      struct ttm_pool_alloc_state *alloc,
+				      bool *suboptimal)
 {
+	unsigned int beneficial = ttm_pool_beneficial_order(pool);
+	pgoff_t bnr = 1UL << beneficial;
+
 	while (alloc->remaining_pages) {
 		struct page *p = old_tt->pages[off];
 		unsigned int order = ttm_pool_page_order(pool, p);

[Severity: Critical]
Does this code safely handle the case where off falls inside a multi-page chunk rather than exactly on a chunk head?

During defragmentation, new_tt allocates chunks independently of old_tt's chunk boundaries. When the budget is exhausted, ttm_pool_harvest_remaining() is called with an off that may land on a tail page.

If this fetches a tail page, ttm_pool_page_order() will read its uninitialized private field. For DMA pools, this uninitialized private field is cast to a struct ttm_pool_dma pointer and dereferenced, causing a kernel panic. For non-DMA pools, it evaluates to order 0, corrupting the chunk boundaries and potentially leading to a double-free.

Quoted 1 line
drivers/gpu/drm/ttm/ttm_pool.c +0 -0
 		pgoff_t nr = 1UL << order;

[ ... ]

drivers/gpu/drm/ttm/ttm_pool.c static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, · line 1175

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_pool.c +1 -0
@@ -1106,10 +1172,23 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		gfp_flags |= GFP_HIGHUSER;
 
 	it.gfp_flags = gfp_flags;
+	it.defrag_capped = it.defrag_old_tt && alloc->defrag_bytes_remaining > 0;

[Severity: Medium]
Will this properly cap the allocation if the budget is overdrawn and becomes negative?

When the allocator decrements defrag_bytes_remaining by the size of the newly allocated chunks, a chunk can exceed the exact remaining budget, causing the value to drop below zero.

If this overdrawn negative budget is passed to the next BO in the batch, alloc->defrag_bytes_remaining > 0 will evaluate to false. The allocator will then treat the negative budget as unlimited rather than exhausted, bypassing the cap entirely and stalling concurrent work.

drivers/gpu/drm/ttm/ttm_pool.c static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, · line 1177

Quoted 2 lines
drivers/gpu/drm/ttm/ttm_pool.c +0 -0
 	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);

[ ... ]

Additionally, looking at ttm_tt_defrag_disown_borrowed() in drivers/gpu/drm/ttm/ttm_tt.c:

drivers/gpu/drm/ttm/ttm_tt.c:ttm_tt_defrag_disown_borrowed() {
    ...
    unsigned int order = sp ? ttm_pool_page_order_nodma(sp) : 0;
    pgoff_t nr = 1UL << order;
    ...
}

[Severity: Critical]
Is it safe to hardcode ttm_pool_page_order_nodma() here when defragmenting pools that use dma_alloc?

Drivers that use dma_alloc pools (e.g., amdgpu, vmwgfx) store a pointer to a struct ttm_pool_dma in the page->private field. Casting this pointer to an integer order results in a huge bit shift, which destroys the array stride. This silently fails to disown the shared pages, leading to an exploitable use-after-free or double-free when the old tt is destroyed.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
author maintainer
#18 lore raw

PATCH v2 9/33 drm/ttm: Preallocate beneficial-order defrag pages outside the lock

In reply to Matthew Brost

Defragmentation moves reallocate a BO's backing at the pool's beneficial order. On a fragmented system those high-order allocations frequently stall in direct reclaim/compaction, and they run under the BO dma-resv lock, holding up concurrent rendering. The defrag worker does not care how long the work takes; it only cares about minimizing locked time.

Add a preallocation bag so the expensive allocations can be hoisted out of the lock:

- struct ttm_pool_prealloc carries beneficial-order pages preallocated outside the lock; ttm_pool_prealloc_fill()/_fini() allocate and release them. __ttm_pool_alloc() consumes them via ctx->prealloc.
- Apply the requested cpu-caching to the bag pages during fill, so the set_memory_*() cost is paid outside the lock too; consumed pages are already cache-consistent and leftovers are restored to write-back on free. Bag pages are never highmem so the kernel mapping is always valid for set_memory_*().
- Record the exact number of sub-optimal pages on the tt (nr_suboptimal_pages) at populate time so a defrag pass can size its prealloc precisely instead of guessing.
- Once a defrag move exhausts the prealloc bag, harvest the remainder of the old tt rather than entering reclaim under the lock, and flag the tt sub-optimal so the driver re-queues it for a later pass.

This keeps all reclaim/compaction stalls and cache mode changes out of the dma-resv critical section: lock-held time then tracks only the bytes actually copied.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 243 +++++++++++++++++++++++++++++++--
 include/drm/ttm/ttm_bo.h       |  11 ++
 include/drm/ttm/ttm_pool.h     |  29 ++++
 include/drm/ttm/ttm_tt.h       |  17 +++
 4 files changed, 286 insertions(+), 14 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +229 -14
index a6847a16c47f..76e5fc54ea9a 100644
@@ -82,6 +82,9 @@ struct ttm_pool_dma {
  * Pages harvested (borrowed) from the old tt are free and do not count; once
  * this drops to zero the allocator stops upgrading and harvests the remainder.
  * On completion it holds the unused budget. 0 means unlimited (no defrag cap).
+ * @nr_suboptimal_pages: Number of pages backed below the pool's beneficial
+ * order. Recorded by the pool allocator after populate; a defrag pass reads it
+ * to size its out-of-lock preallocation exactly.
  */
 struct ttm_pool_alloc_state {
 	struct page **pages;
@@ -90,6 +93,7 @@ struct ttm_pool_alloc_state {
 	pgoff_t remaining_pages;
 	enum ttm_caching tt_caching;
 	s64 defrag_bytes_remaining;
+	u32 nr_suboptimal_pages;
 };
 
 /**
@@ -787,6 +791,7 @@ static void ttm_pool_alloc_state_init(const struct ttm_tt *tt,
 	alloc->remaining_pages = tt->num_pages;
 	alloc->tt_caching = tt->caching;
 	alloc->defrag_bytes_remaining = 0;
+	alloc->nr_suboptimal_pages = 0;
 }
 
 /*
@@ -852,7 +857,8 @@ static unsigned int ttm_pool_defrag_old_order(struct ttm_pool *pool,
  * pages stay owned by the old tt until the move commits.
  */
 static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
-				 pgoff_t off, struct ttm_pool_alloc_state *alloc)
+				 unsigned int beneficial_order, pgoff_t off,
+				 struct ttm_pool_alloc_state *alloc)
 {
 	pgoff_t nr = 1UL << order;
 	int r;
@@ -887,6 +893,9 @@ static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
 
 	alloc->caching_divide = alloc->pages;
 
+	if (order < beneficial_order)
+		alloc->nr_suboptimal_pages += 0x1 << order;
+
 	return 0;
 }
 
@@ -919,7 +928,8 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 		    round_down(off, bnr) + bnr <= num_pages)
 			*suboptimal = true;
 
-		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
+		r = ttm_pool_harvest_page(old_tt, order, beneficial, off,
+					  alloc);
 		if (r)
 			return r;
 
@@ -929,6 +939,172 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 	return 0;
 }
 
+unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool)
+{
+	return ttm_pool_beneficial_order(pool);
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_order);
+
+/*
+ * Build the gfp flags used for the high-order, possibly reclaiming, beneficial
+ * order page allocations, matching the in-line defrag alloc path.
+ */
+static gfp_t ttm_pool_prealloc_gfp(struct ttm_pool *pool)
+{
+	gfp_t gfp = GFP_USER;
+
+	/*
+	 * No highmem: prealloc applies caching in bulk via set_pages_array_*()
+	 * on the kernel mapping, so the pages must be permanently mapped.
+	 */
+	if (ttm_pool_uses_dma32(pool))
+		gfp |= GFP_DMA32;
+
+	return gfp;
+}
+
+/*
+ * Apply a tt's cpu-caching to a batch of freshly system-allocated (write-back)
+ * prealloc pages in one shot. @cpages is an unpacked array of @ncpages
+ * individual 4K pages (the constituent pages of the mixed-order chunks, expanded
+ * so set_pages_array_*() sees one entry per page), since the packed prealloc bag
+ * holds multi-order chunks. Prealloc pages are never highmem (see
+ * ttm_pool_prealloc_gfp), so their kernel mapping is valid. No-op for cached
+ * pages and on non-x86. Returns non-zero if the caching change failed (the
+ * caller drops the whole bag and falls back to in-line allocation).
+ */
+static int ttm_pool_prealloc_apply_caching(enum ttm_caching caching,
+					   struct page **cpages,
+					   unsigned int ncpages)
+{
+#ifdef CONFIG_X86
+	if (!ncpages)
+		return 0;
+
+	switch (caching) {
+	case ttm_cached:
+		break;
+	case ttm_write_combined:
+		return set_pages_array_wc(cpages, ncpages);
+	case ttm_uncached:
+		return set_pages_array_uc(cpages, ncpages);
+	}
+#endif
+	return 0;
+}
+
+/* Expand an @order chunk into its constituent 4K pages for bulk caching. */
+static void ttm_pool_prealloc_stage_caching(struct page **cpages,
+					    unsigned int *ncpages,
+					    struct page *p, unsigned int order)
+{
+	unsigned int i, nr = 1u << order;
+
+	for (i = 0; i < nr; i++)
+		cpages[(*ncpages)++] = p + i;
+}
+
+/**
+ * ttm_pool_prealloc_fill() - Preallocate beneficial-order pages outside any lock
+ * @pool: The pool to allocate from.
+ * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @pp: Prealloc bag to fill; @pp->order is set to the beneficial order.
+ * @count: Number of beneficial-order chunks to attempt.
+ *
+ * Allocate up to @count beneficial-order chunks, parking them in @pp for a
+ * later __ttm_pool_alloc() defrag move to consume under the dma-resv lock. May
+ * sleep/reclaim freely as it runs unlocked. A short fill is fine: the pool
+ * falls back to in-line allocation for the shortfall. DMA-alloc pools are not
+ * supported (count stays 0). Returns 0 (release with ttm_pool_prealloc_fini()).
+ */
+int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
+			   struct ttm_pool_prealloc *pp, unsigned int count)
+{
+	unsigned int order = ttm_pool_beneficial_order(pool);
+	gfp_t gfp = ttm_pool_prealloc_gfp(pool);
+	struct page **cpages = NULL;
+	unsigned int ncpages = 0;
+	int r;
+
+	pp->pages = NULL;
+	pp->order = order;
+	pp->caching = tt_caching;
+	pp->count = 0;
+	pp->used = 0;
+
+	/* Nothing to gain without a beneficial order or for DMA-alloc pools. */
+	if (!order || !count || ttm_pool_uses_dma_alloc(pool))
+		return 0;
+
+	pp->pages = kvzalloc_objs(*pp->pages, count);
+	if (!pp->pages)
+		return 0;
+
+	/*
+	 * Every chunk is a fresh write-back system page of @order that needs a
+	 * caching change; collect their constituent pages into an unpacked
+	 * scratch array and issue a single set_pages_array_*() after the fill.
+	 * Cached tts need no change (and non-x86 handles caching at map time).
+	 */
+	if (IS_ENABLED(CONFIG_X86) && tt_caching != ttm_cached) {
+		cpages = kvzalloc_objs(*cpages, (size_t)count << order);
+		if (!cpages) {
+			kvfree(pp->pages);
+			pp->pages = NULL;
+			return 0;
+		}
+	}
+
+	while (pp->count < count) {
+		struct page *p = ttm_pool_alloc_page(pool, gfp, order, false);
+
+		if (!p)
+			break;
+
+		pp->pages[pp->count++] = p;
+		if (cpages)
+			ttm_pool_prealloc_stage_caching(cpages, &ncpages, p,
+							order);
+	}
+
+	/*
+	 * Apply the requested caching to every collected page in one shot. On
+	 * failure the pages' caching is indeterminate, so drop the whole bag
+	 * (freeing restores write-back) and let the consumer allocate in-line.
+	 */
+	r = ttm_pool_prealloc_apply_caching(tt_caching, cpages, ncpages);
+	kvfree(cpages);
+	if (r) {
+		unsigned int i;
+
+		for (i = 0; i < pp->count; i++)
+			ttm_pool_free_page(pool, tt_caching, order,
+					   pp->pages[i], false);
+		pp->count = 0;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fill);
+
+/**
+ * ttm_pool_prealloc_fini() - Release unconsumed preallocated pages
+ * @pool: The pool the pages came from.
+ * @pp: Prealloc bag to drain. Consumed pages (< @used) are now owned by the tt.
+ */
+void ttm_pool_prealloc_fini(struct ttm_pool *pool, struct ttm_pool_prealloc *pp)
+{
+	unsigned int i;
+
+	for (i = pp->used; i < pp->count; ++i)
+		ttm_pool_free_page(pool, pp->caching, pp->order, pp->pages[i],
+				   false);
+	kvfree(pp->pages);
+	pp->pages = NULL;
+	pp->count = pp->used = 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fini);
+
 /**
  * enum ttm_pool_iter_action - Outcome of a per-order allocation phase
  * @TTM_POOL_ITER_FILL: A page (@it->p) was acquired; proceed to fill it in.
@@ -960,6 +1136,7 @@ struct ttm_pool_alloc_iter {
 	struct ttm_pool_alloc_state *alloc;
 	struct ttm_pool_tt_restore *restore;
 	struct ttm_tt *defrag_old_tt;
+	struct ttm_pool_prealloc *prealloc;
 	unsigned int beneficial_order;
 	gfp_t gfp_flags;
 	bool defrag_capped;
@@ -975,16 +1152,20 @@ struct ttm_pool_alloc_iter {
 };
 
 /*
- * Defrag move budget exhausted: the upgrade can make no further progress this
- * pass. Snapshot @defrag_capped is set only when a byte budget was in force at
- * entry.
+ * Defrag move budget exhausted, or the out-of-lock prealloc bag ran dry: the
+ * upgrade can make no further progress this pass. Snapshot @defrag_capped is
+ * set only when a byte budget was in force at entry.
  */
 static bool ttm_pool_iter_defrag_exhausted(const struct ttm_pool_alloc_iter *it)
 {
+	const struct ttm_pool_prealloc *pp = it->prealloc;
+
 	if (!it->defrag_old_tt)
 		return false;
+	if (it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0)
+		return true;
 
-	return it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0;
+	return pp && pp->count && pp->used >= pp->count;
 }
 
 /*
@@ -1041,7 +1222,8 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 		return TTM_POOL_ITER_FILL;
 
 	it->order = harvest_order;
-	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order, off, it->alloc);
+	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order,
+				  it->beneficial_order, off, it->alloc);
 	if (r)
 		return r;
 
@@ -1053,11 +1235,13 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 
 /*
  * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries a same-order pool page, then a fresh system allocation. Fault
- * injection can force the beneficial-order paths to "fail".
+ * failure). Tries, in order: a beneficial-order page preallocated outside the
+ * dma-resv lock (defrag), a same-order pool page, then a fresh system
+ * allocation. Fault injection can force the beneficial-order paths to "fail".
  */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
+	struct ttm_pool_prealloc *pp = it->prealloc;
 	struct ttm_pool_type *pt;
 
 	it->p = NULL;
@@ -1072,7 +1256,18 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 		it->beneficial_order && it->order >= it->beneficial_order &&
 		should_fail(&beneficial_order_fault_inject, 1);
 
-	/* First, try to allocate a page from a pool if one exists. */
+	/*
+	 * Defrag move: consume a beneficial-order page preallocated outside the
+	 * dma-resv lock instead of allocating one in-line (which may stall in
+	 * reclaim/compaction). Falls through once the bag is empty.
+	 */
+	if (pp && !it->fail_beneficial && it->order >= it->beneficial_order &&
+	    pp->used < pp->count) {
+		it->order = it->beneficial_order;
+		it->p = pp->pages[pp->used++];
+		it->page_caching = it->tt->caching;
+	}
+
 	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
 	if (!it->p && pt && it->allow_pools && !it->fail_beneficial)
 		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
@@ -1149,6 +1344,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		.alloc = alloc,
 		.restore = restore,
 		.defrag_old_tt = restore ? NULL : ctx->defrag_old_tt,
+		.prealloc = restore ? NULL : ctx->prealloc,
 		.beneficial_order = ttm_pool_beneficial_order(pool),
 		.page_caching = tt->caching,
 		.allow_pools = true,
@@ -1178,8 +1374,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
 		/*
-		 * Out of defrag budget: harvest the rest of the old tt as-is and
-		 * stop (the tt is re-queued if the remainder is still
+		 * Out of defrag budget/prealloc: harvest the rest of the old tt
+		 * as-is and stop (the tt is re-queued if the remainder is still
 		 * sub-optimal).
 		 */
 		if (ttm_pool_iter_defrag_exhausted(&it)) {
@@ -1196,7 +1392,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		if (r == TTM_POOL_ITER_RETRY)
 			continue;
 
-		/* Acquire a page (pool / system) for this order. */
+		/* Acquire a page (prealloc / pool / system) for this order. */
 		ttm_pool_iter_acquire_page(&it);
 		if (!it.p) {
 			r = ttm_pool_iter_lower_order(&it);
@@ -1228,6 +1424,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			alloc->defrag_bytes_remaining -=
 				(s64)(1UL << it.order) << PAGE_SHIFT;
 
+		if (it.order < it.beneficial_order)
+			alloc->nr_suboptimal_pages += 0x1 << it.order;
 		it.alloc_count++;
 	}
 
@@ -1291,6 +1489,9 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	/* Report the unused defrag budget back to the caller. */
 	ctx->defrag_bytes_remaining = alloc.defrag_bytes_remaining;
 
+	if (!ret)
+		tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
+
 	return ret;
 }
 EXPORT_SYMBOL(ttm_pool_alloc);
@@ -1343,6 +1544,15 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 
 			if (ret)
 				return ret;
+
+			/*
+			 * __ttm_pool_alloc() counts each freshly (re)allocated
+			 * chunk against nr_suboptimal_pages, but a chunk whose
+			 * backup copy was interrupted and is finished here
+			 * resumes past that loop, so account for it directly.
+			 */
+			if (restore->order < ttm_pool_beneficial_order(pool))
+				alloc.nr_suboptimal_pages += 1U << restore->order;
 		}
 		if (!alloc.remaining_pages) {
 			ret = ttm_pool_apply_caching(&alloc);
@@ -1352,11 +1562,16 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			kfree(tt->restore);
 			tt->restore = NULL;
 
+			tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
 			return 0;
 		}
 	}
 
-	return __ttm_pool_alloc(pool, tt, ctx, &alloc, restore);
+	ret = __ttm_pool_alloc(pool, tt, ctx, &alloc, restore);
+	if (!ret)
+		tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
+
+	return ret;
 }
 
 /**
include/drm/ttm/ttm_bo.h +11 -0
index 6124d2627b47..a80304f179ba 100644
@@ -237,6 +237,17 @@ struct ttm_operation_ctx {
 	 * @defrag_old_tt is set.
 	 */
 	s64 defrag_bytes_remaining;
+	/**
+	 * @prealloc: Pages preallocated outside the dma-resv lock for a defrag
+	 * move. The pool allocator consumes these instead of allocating fresh
+	 * beneficial-order pages under the lock, moving the (potentially
+	 * reclaim/compaction stalling) high-order allocations out of the
+	 * critical section. NULL means allocate in-line as usual. The pool
+	 * silently falls back to in-line allocation for any shortfall. The
+	 * allocator consumes it for beneficial-order chunks of any populate;
+	 * drivers only set it for a defrag move.
+	 */
+	struct ttm_pool_prealloc *prealloc;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
include/drm/ttm/ttm_pool.h +29 -0
index 753203980e2c..71670350eb15 100644
@@ -81,6 +81,35 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		   struct ttm_operation_ctx *ctx);
 void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt);
 
+/**
+ * struct ttm_pool_prealloc - Pages preallocated outside the dma-resv lock
+ * @pages: Array of @count beneficial-order pages (or fewer if a fill fell
+ *         short); each entry is the head page of a 1 << @order chunk.
+ * @order: Page order of every preallocated chunk.
+ * @caching: CPU caching applied to the pages, so leftovers can be restored
+ *           to write-back before being freed.
+ * @count: Number of valid entries in @pages.
+ * @used: Number of entries already consumed by the pool allocator.
+ *
+ * Defrag pages are interchangeable, so only a count of beneficial-order chunks
+ * is needed. ttm_pool_prealloc_fill() populates this outside the lock and
+ * __ttm_pool_alloc() drains it; any unused tail is released by
+ * ttm_pool_prealloc_fini().
+ */
+struct ttm_pool_prealloc {
+	struct page **pages;
+	unsigned int order;
+	enum ttm_caching caching;
+	unsigned int count;
+	unsigned int used;
+};
+
+int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
+			   struct ttm_pool_prealloc *pp, unsigned int count);
+void ttm_pool_prealloc_fini(struct ttm_pool *pool,
+			    struct ttm_pool_prealloc *pp);
+unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool);
+
 void ttm_pool_init(struct ttm_pool *pool, struct device *dev,
 		   int nid, unsigned int alloc_flags);
 void ttm_pool_fini(struct ttm_pool *pool);
include/drm/ttm/ttm_tt.h +17 -0
index 55c35bcf134d..81a5446ca693 100644
@@ -135,6 +135,12 @@ struct ttm_tt {
 	enum ttm_caching caching;
 	/** @restore: Partial restoration from backup state. TTM private */
 	struct ttm_pool_tt_restore *restore;
+	/**
+	 * @nr_suboptimal_pages: Number of pages backed below the pool's
+	 * beneficial order. Recorded by the pool allocator after populate; a
+	 * defrag pass reads it to size its out-of-lock preallocation exactly.
+	 */
+	u32 nr_suboptimal_pages;
 };
 
 /**
@@ -201,6 +207,17 @@ static inline bool ttm_tt_is_beneficial_order_failed(const struct ttm_tt *tt)
 	return tt->page_flags & TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 }
 
+/**
+ * ttm_tt_suboptimal_pages() - Pages backed below the pool's beneficial order
+ * @tt: The struct ttm_tt.
+ *
+ * Return: Number of pages a defrag move would (re)allocate at beneficial order.
+ */
+static inline u32 ttm_tt_suboptimal_pages(const struct ttm_tt *tt)
+{
+	return tt->nr_suboptimal_pages;
+}
+
 /**
  * ttm_tt_create
  *
Signature · 1 line
2.34.1
#19 lore raw

Re: [PATCH v2 09/33] drm/ttm: Preallocate beneficial-order defrag pages outside the lock

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Critical] Harvesting tail pages of a bisected chunk during defragmentation leads to Use-After-Free.…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] Harvesting tail pages of a bisected chunk during defragmentation leads to Use-After-Free.
- [High] Missing __GFP_NOWARN and __GFP_RETRY_MAYFAIL in best-effort preallocation causes log spam and potential OOM killer invocation. --

--- Patch [9]: [PATCH v2 09/33] drm/ttm: Preallocate beneficial-order defrag pages outside the lock --- commit d825f68af024a3a3838c6e5d1ecfcf5ee8717374
Author: Matthew Brost <[email protected]>
Subject: drm/ttm: Preallocate beneficial-order defrag pages outside the lock

This commit introduces a mechanism to preallocate high-order pages outside the DMA reservation lock during defragmentation moves. By using a prealloc bag, it avoids stalling in direct reclaim or compaction while holding the lock, improving concurrent rendering performance.

Quoted 3 lines
drivers/gpu/drm/ttm/ttm_pool.c +0 -0

[ ... ]

Quoted 17 lines
drivers/gpu/drm/ttm/ttm_pool.c +13 -0
@@ -929,6 +939,172 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 	return 0;
 }
 
+unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool)
+{
+	return ttm_pool_beneficial_order(pool);
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_order);
+
+/*
+ * Build the gfp flags used for the high-order, possibly reclaiming, beneficial
+ * order page allocations, matching the in-line defrag alloc path.
+ */
+static gfp_t ttm_pool_prealloc_gfp(struct ttm_pool *pool)
+{
+	gfp_t gfp = GFP_USER;

drivers/gpu/drm/ttm/ttm_pool.c static int ttm_pool_harvest_remaining(struct ttm_pool *pool, · line 964

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+
+	/*
+	 * No highmem: prealloc applies caching in bulk via set_pages_array_*()
+	 * on the kernel mapping, so the pages must be permanently mapped.
+	 */
+	if (ttm_pool_uses_dma32(pool))
+		gfp |= GFP_DMA32;
+
+	return gfp;
+}

[Severity: High]
Does this base GFP flag configuration risk invoking the OOM killer on a fragmented system?

The preallocation performed in ttm_pool_prealloc_fill() appears to be an opportunistic, best-effort operation:

ttm_pool_prealloc_fill() {
        ...
        struct page *p = ttm_pool_alloc_page(pool, gfp, order, false);

Without including __GFP_NOWARN and __GFP_RETRY_MAYFAIL here, a failure to allocate these high-order pages will trigger aggressive reclaim. This could unnecessarily invoke the OOM killer and spam the logs, even though the pool can safely fall back to in-line allocation.

Should these suppression flags be added to the preallocation gfp mask?

[ ... ]

drivers/gpu/drm/ttm/ttm_pool.c static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, · line 1381

Quoted 11 lines
drivers/gpu/drm/ttm/ttm_pool.c +2 -2
@@ -1178,8 +1374,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
 		/*
-		 * Out of defrag budget: harvest the rest of the old tt as-is and
-		 * stop (the tt is re-queued if the remainder is still
+		 * Out of defrag budget/prealloc: harvest the rest of the old tt
+		 * as-is and stop (the tt is re-queued if the remainder is still
 		 * sub-optimal).
 		 */
 		if (ttm_pool_iter_defrag_exhausted(&it)) {

[Severity: Critical]
Can this lead to memory corruption if the budget is exhausted while the offset is misaligned with the chunks in the old tt?

If the budget is exhausted here, ttm_pool_iter_harvest_rest() will call ttm_pool_harvest_remaining(), which reads the order from the current page:

ttm_pool_harvest_remaining() {
        ...
        struct page *p = old_tt->pages[off];
        unsigned int order = ttm_pool_page_order(pool, p);

If off points to a tail page of a previously allocated larger chunk, ttm_pool_page_order() will return 0 because the private field is 0 on tail pages. This causes the code to falsely harvest it as an order-0 chunk. When the old tt is later freed, it frees the entire large chunk from the head page, leaving the new tt with pointers to freed memory.

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_pool.c +5 -0
+			r = ttm_pool_iter_harvest_rest(&it);
+			if (r)
+				goto error_free_all;
+			break;
+		}

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
author maintainer
#20 lore raw

PATCH v2 10/33 drm/ttm: Add full out-of-lock preallocation for ttm_pool_alloc()

In reply to Matthew Brost

ttm_pool_prealloc_fill() lets a driver preallocate beneficial-order pages outside the dma-resv lock for a defragmentation move, so the high-order allocations that may stall in reclaim/compaction happen off the critical section. That bag is deliberately partial: it holds only interchangeable beneficial-order chunks and the allocator consumes it opportunistically, falling back to in-line allocation for everything else.

An ordinary populate of a system-memory buffer object has the same problem - the whole backing is allocated under the dma-resv lock - but no way to move it out of the critical section, because the existing bag does not cover sub-beneficial orders and does not describe a full tt.

Add a "full" flavour of the same preallocation mechanism:

- struct ttm_pool_prealloc gains a @full flag. In full mode the bag is an ordered tiling of an entire tt at mixed orders (pool pages included), rather than a set of interchangeable beneficial-order chunks. The per-chunk order is recorded on the page itself and read back at consume time, so no parallel order array is needed.

- ttm_pool_prealloc_fill_full() allocates the whole backing for a tt of @num_pages pages up front, mirroring the in-line allocator's order walk (highest order first, capped by the remaining page count). It reuses the existing unlocked helpers (ttm_pool_prealloc_gfp(), ttm_pool_set_caching()) via a new ttm_pool_take_page() helper that tries a same-order pool page before a fresh system allocation.

- ttm_pool_prealloc_fini() frees any unconsumed tail at each chunk's own order when @full.

- ttm_pool_prealloc_fill_full() takes a @beneficial_reclaim_backoff argument, plumbed to ttm_pool_alloc_page(), that lets the caller allocate higher-order chunks without aggressive reclaim/compaction. This suits a driver that upgrades suboptimal pages to beneficial order in place later (e.g. via a background defragmenter) and so has no need to stall the user context reclaiming for contiguity up front.

- __ttm_pool_alloc() consumes a full bag with a single early branch in ttm_pool_iter_acquire_page(): install the next chunk as-is (order taken from the page) and return; once the best-effort bag is drained the loop falls through to normal in-line allocation. The defrag harvest and reuse-old phases are inert here as there is no defrag_old_tt.

Like the defrag bag, this is best-effort: a short fill is fine (the pool allocates the shortfall in-line) and DMA-alloc pools are unsupported (caching is applied on the kernel mapping). No functional change for existing users; the new fill routine has no in-tree caller yet.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 232 ++++++++++++++++++++++++++++++++-
 include/drm/ttm/ttm_bo.h       |  19 +--
 include/drm/ttm/ttm_pool.h     |  27 +++-
 3 files changed, 258 insertions(+), 20 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +226 -6
index 76e5fc54ea9a..762ff1c73390 100644
@@ -1031,6 +1031,7 @@ int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 	pp->caching = tt_caching;
 	pp->count = 0;
 	pp->used = 0;
+	pp->full = false;
 
 	/* Nothing to gain without a beneficial order or for DMA-alloc pools. */
 	if (!order || !count || ttm_pool_uses_dma_alloc(pool))
@@ -1087,6 +1088,187 @@ int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 }
 EXPORT_SYMBOL(ttm_pool_prealloc_fill);
 
+/*
+ * Acquire a single *@order chunk, trying a same-order pool page first (already
+ * carries @caching) then a fresh system allocation (write-back). Runs unlocked.
+ * When @beneficial_reclaim_backoff is set, higher-order system allocations skip
+ * aggressive reclaim/compaction (see ttm_pool_alloc_page()).
+ *
+ * A fresh system page is write-back and needs the tt's caching applied later in
+ * bulk; *@need_caching is set true for it and false for a (pre-cached) pool
+ * page.
+ *
+ * On a genuine failure *@order is left untouched (the caller lowers it by one
+ * and retries). Fault injection instead pretends allocation at (or above) the
+ * beneficial order failed and lowers *@order to the beneficial order, so the
+ * caller's next attempt lands below it - mirroring the in-line acquire path's
+ * fail_beneficial jump and exercising the beneficial_order_failed tracking
+ * without driving the system into real fragmentation.
+ *
+ * Returns NULL if no page of *@order could be obtained.
+ */
+static struct page *ttm_pool_take_page(struct ttm_pool *pool,
+				       enum ttm_caching caching,
+				       unsigned int *order, gfp_t gfp,
+				       bool *need_caching,
+				       bool beneficial_reclaim_backoff)
+{
+	unsigned int beneficial = ttm_pool_beneficial_order(pool);
+	struct ttm_pool_type *pt;
+	struct page *p;
+
+	*need_caching = false;
+
+	if (IS_ENABLED(CONFIG_FAULT_INJECTION) && beneficial &&
+	    *order >= beneficial &&
+	    should_fail(&beneficial_order_fault_inject, 1)) {
+		*order = beneficial;
+		return NULL;
+	}
+
+	pt = ttm_pool_select_type(pool, caching, *order);
+	if (pt) {
+		p = ttm_pool_type_take(pt, ttm_pool_nid(pool));
+		if (p)
+			return p;	/* pool page already carries @caching */
+	}
+
+	p = ttm_pool_alloc_page(pool, gfp, *order, beneficial_reclaim_backoff);
+	if (!p)
+		return NULL;
+
+	/* Fresh system page is write-back; caller applies @caching in bulk. */
+	*need_caching = true;
+	return p;
+}
+
+/**
+ * ttm_pool_prealloc_fill_full() - Preallocate a whole tt's backing outside any
+ * lock
+ * @pool: The pool to allocate from.
+ * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @pp: Prealloc bag to fill; marked @full and tiled with mixed-order chunks.
+ * @num_pages: Number of 4K pages the tt needs backed.
+ *
+ * Unlike ttm_pool_prealloc_fill() (which parks a bag of interchangeable
+ * beneficial-order defrag chunks), this allocates the ENTIRE backing for a tt
+ * of @num_pages pages up front, at whatever mix of orders is available (pool
+ * pages included), so that the subsequent __ttm_pool_alloc() under the dma-resv
+ * lock merely installs the pages without stalling in reclaim/compaction. The
+ * chunk orders mirror the in-line allocator's order walk; each chunk's order is
+ * recorded on the page itself and read back at consume time.
+ *
+ * May sleep/reclaim freely as it runs unlocked. When
+ * @beneficial_reclaim_backoff is set, higher-order chunks are taken without
+ * aggressive reclaim/compaction: the caller relies on a background
+ * defragmenter to upgrade to beneficial-order pages in place later, so there
+ * is no need to stall the user context reclaiming for contiguity here. A short
+ * fill is fine: the pool falls back to in-line allocation for the shortfall.
+ * DMA-alloc pools are not supported (the bag stays empty). Returns 0 (release
+ * with ttm_pool_prealloc_fini()).
+ */
+int ttm_pool_prealloc_fill_full(struct ttm_pool *pool,
+				enum ttm_caching tt_caching,
+				struct ttm_pool_prealloc *pp,
+				unsigned int num_pages,
+				bool beneficial_reclaim_backoff)
+{
+	gfp_t gfp = ttm_pool_prealloc_gfp(pool) | __GFP_ZERO |
+		__GFP_RETRY_MAYFAIL | __GFP_NOWARN;
+	unsigned int highest = MAX_PAGE_ORDER;
+	unsigned int remaining = num_pages;
+	struct page **cpages = NULL;
+	unsigned int ncpages = 0;
+	int r;
+
+	pp->pages = NULL;
+	pp->order = 0;
+	pp->caching = tt_caching;
+	pp->count = 0;
+	pp->used = 0;
+	pp->full = true;
+
+	/* Caching is applied on the kernel mapping, so DMA-alloc is unsupported. */
+	if (!num_pages || ttm_pool_uses_dma_alloc(pool))
+		return 0;
+
+	/* Worst case is one order-0 chunk per page. */
+	pp->pages = kvzalloc_objs(*pp->pages, num_pages);
+	if (!pp->pages)
+		return 0;
+
+	/*
+	 * Fresh write-back chunks get their caching changed in a single
+	 * set_pages_array_*() after the fill; collect their constituent pages
+	 * into an unpacked scratch array (one entry per 4K page) since the
+	 * packed bag holds mixed-order chunks. Cached tts and pool pages need
+	 * no change (and non-x86 handles caching at map time).
+	 */
+	if (IS_ENABLED(CONFIG_X86) && tt_caching != ttm_cached) {
+		cpages = kvzalloc_objs(*cpages, num_pages);
+		if (!cpages) {
+			kvfree(pp->pages);
+			pp->pages = NULL;
+			return 0;
+		}
+	}
+
+	while (remaining) {
+		unsigned int order = min_t(unsigned int, highest,
+					   __fls(remaining));
+		bool need_caching;
+		struct page *p;
+
+		for (;;) {
+			p = ttm_pool_take_page(pool, tt_caching, &order, gfp,
+					       &need_caching,
+					       beneficial_reclaim_backoff);
+			if (p)
+				break;
+			if (!order)
+				goto apply_caching;	/* short fill */
+			/*
+			 * Genuine failure: retry one order down. A fault
+			 * injection lowered @order to the beneficial order
+			 * above, so this drops it below - landing on a
+			 * sub-optimal (defrag-worthy) chunk.
+			 */
+			order--;
+		}
+
+		pp->pages[pp->count++] = p;
+		if (cpages && need_caching)
+			ttm_pool_prealloc_stage_caching(cpages, &ncpages, p,
+							order);
+		remaining -= 1u << order;
+		highest = order;
+	}
+
+apply_caching:
+	/*
+	 * Apply the requested caching to every collected page in one shot. On
+	 * failure the pages' caching is indeterminate, so drop the whole bag
+	 * (freeing restores write-back) and let the consumer allocate in-line.
+	 */
+	r = ttm_pool_prealloc_apply_caching(tt_caching, cpages, ncpages);
+	kvfree(cpages);
+	if (r) {
+		unsigned int i;
+
+		for (i = 0; i < pp->count; i++) {
+			struct page *p = pp->pages[i];
+
+			ttm_pool_free_page(pool, tt_caching,
+					   ttm_pool_page_order_nodma(p), p,
+					   false);
+		}
+		pp->count = 0;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fill_full);
+
 /**
  * ttm_pool_prealloc_fini() - Release unconsumed preallocated pages
  * @pool: The pool the pages came from.
@@ -1096,9 +1278,13 @@ void ttm_pool_prealloc_fini(struct ttm_pool *pool, struct ttm_pool_prealloc *pp)
 {
 	unsigned int i;
 
-	for (i = pp->used; i < pp->count; ++i)
-		ttm_pool_free_page(pool, pp->caching, pp->order, pp->pages[i],
-				   false);
+	for (i = pp->used; i < pp->count; ++i) {
+		struct page *p = pp->pages[i];
+		unsigned int order = pp->full ?
+			ttm_pool_page_order_nodma(p) : pp->order;
+
+		ttm_pool_free_page(pool, pp->caching, order, p, false);
+	}
 	kvfree(pp->pages);
 	pp->pages = NULL;
 	pp->count = pp->used = 0;
@@ -1235,9 +1421,10 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 
 /*
  * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries, in order: a beneficial-order page preallocated outside the
- * dma-resv lock (defrag), a same-order pool page, then a fresh system
- * allocation. Fault injection can force the beneficial-order paths to "fail".
+ * failure). Tries, in order: a chunk from a full out-of-lock prealloc (populate
+ * of the whole tt), a beneficial-order page preallocated outside the dma-resv
+ * lock (defrag), a same-order pool page, then a fresh system allocation. Fault
+ * injection can force the beneficial-order paths to "fail".
  */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
@@ -1246,6 +1433,39 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 
 	it->p = NULL;
 
+	/*
+	 * Full prealloc: the entire backing was allocated (pool + system, at
+	 * mixed orders) outside the dma-resv lock. Install the next chunk as-is,
+	 * taking its order from the page. Falls through to in-line allocation
+	 * once the best-effort bag is drained.
+	 */
+	if (pp && pp->full && pp->used < pp->count) {
+		it->fail_beneficial = false;
+		it->p = pp->pages[pp->used++];
+		it->order = ttm_pool_page_order_nodma(it->p);
+		it->page_caching = it->tt->caching;
+
+		/*
+		 * The full prealloc tiles at whatever orders were available
+		 * without stalling in reclaim. A sub-beneficial chunk only
+		 * means a sub-optimal backing (defrag-worthy) when it sits in a
+		 * beneficial-order aligned region that fully fits in the tt -
+		 * that region could have been a beneficial-order page. A low
+		 * order in the unaligned trailing tail is expected and must not
+		 * flag the tt (mirrors ttm_pool_harvest_remaining()).
+		 */
+		if (it->beneficial_order && it->order < it->beneficial_order) {
+			pgoff_t bnr = 1UL << it->beneficial_order;
+			pgoff_t off = it->tt->num_pages -
+				      it->alloc->remaining_pages;
+
+			if (round_down(off, bnr) + bnr <= it->tt->num_pages)
+				it->tt->page_flags |=
+					TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+		}
+		return;
+	}
+
 	/*
 	 * Fault injection: pretend allocation at (or above) the device's
 	 * beneficial order failed, forcing a sub-optimal backing. Exercises the
include/drm/ttm/ttm_bo.h +11 -8
index a80304f179ba..52ec7e5c0d2d 100644
@@ -238,14 +238,17 @@ struct ttm_operation_ctx {
 	 */
 	s64 defrag_bytes_remaining;
 	/**
-	 * @prealloc: Pages preallocated outside the dma-resv lock for a defrag
-	 * move. The pool allocator consumes these instead of allocating fresh
-	 * beneficial-order pages under the lock, moving the (potentially
-	 * reclaim/compaction stalling) high-order allocations out of the
-	 * critical section. NULL means allocate in-line as usual. The pool
-	 * silently falls back to in-line allocation for any shortfall. The
-	 * allocator consumes it for beneficial-order chunks of any populate;
-	 * drivers only set it for a defrag move.
+	 * @prealloc: Pages preallocated outside the dma-resv lock. The pool
+	 * allocator consumes these instead of allocating under the lock, moving
+	 * the (potentially reclaim/compaction stalling) allocations out of the
+	 * critical section. NULL means allocate in-line as usual. Two flavours,
+	 * distinguished by ttm_pool_prealloc::full:
+	 *   - defrag bag (full=false): a bag of interchangeable beneficial-order
+	 *     chunks consumed only for beneficial-order chunks of a defrag move.
+	 *   - full tiling (full=true): the entire backing of the tt, at mixed
+	 *     orders, consumed for every chunk of an ordinary populate.
+	 * The allocator silently falls back to in-line allocation for any
+	 * shortfall.
 	 */
 	struct ttm_pool_prealloc *prealloc;
 	/**
include/drm/ttm/ttm_pool.h +21 -6
index 71670350eb15..777d5aa08f6a 100644
@@ -83,18 +83,27 @@ void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt);
 
 /**
  * struct ttm_pool_prealloc - Pages preallocated outside the dma-resv lock
- * @pages: Array of @count beneficial-order pages (or fewer if a fill fell
- *         short); each entry is the head page of a 1 << @order chunk.
- * @order: Page order of every preallocated chunk.
+ * @pages: Array of @count preallocated chunks; each entry is the head page of a
+ *         chunk. In defrag mode every chunk is @order pages; in @full mode the
+ *         chunks tile a whole tt at mixed orders and the per-chunk order is read
+ *         back from the page itself.
+ * @order: Page order of every preallocated chunk (defrag mode only; 0 in @full
+ *         mode where each chunk carries its own order).
  * @caching: CPU caching applied to the pages, so leftovers can be restored
  *           to write-back before being freed.
  * @count: Number of valid entries in @pages.
  * @used: Number of entries already consumed by the pool allocator.
+ * @full: When set the bag holds a full, mixed-order tiling of an entire tt
+ *        (produced by ttm_pool_prealloc_fill_full()) rather than a bag of
+ *        interchangeable beneficial-order defrag chunks. The pool allocator
+ *        then drains the bag in order for every populate, taking each chunk's
+ *        order from the page, and falls back to in-line allocation only for any
+ *        shortfall.
  *
  * Defrag pages are interchangeable, so only a count of beneficial-order chunks
- * is needed. ttm_pool_prealloc_fill() populates this outside the lock and
- * __ttm_pool_alloc() drains it; any unused tail is released by
- * ttm_pool_prealloc_fini().
+ * is needed. ttm_pool_prealloc_fill() / ttm_pool_prealloc_fill_full() populate
+ * this outside the lock and __ttm_pool_alloc() drains it; any unused tail is
+ * released by ttm_pool_prealloc_fini().
  */
 struct ttm_pool_prealloc {
 	struct page **pages;
@@ -102,10 +111,16 @@ struct ttm_pool_prealloc {
 	enum ttm_caching caching;
 	unsigned int count;
 	unsigned int used;
+	bool full;
 };
 
 int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 			   struct ttm_pool_prealloc *pp, unsigned int count);
+int ttm_pool_prealloc_fill_full(struct ttm_pool *pool,
+				enum ttm_caching tt_caching,
+				struct ttm_pool_prealloc *pp,
+				unsigned int num_pages,
+				bool beneficial_reclaim_backoff);
 void ttm_pool_prealloc_fini(struct ttm_pool *pool,
 			    struct ttm_pool_prealloc *pp);
 unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool);
Signature · 1 line
2.34.1
author maintainer
#21 lore raw

PATCH v2 11/33 drm/gpusvm: Add a DMA-mapping accounting callback

In reply to Matthew Brost

From: Thomas Hellström <[email protected]>

Drivers that use the GPU SVM core to DMA-map system memory for GPU access may want to keep track of how many pages they have mapped, for example to expose statistics or to detect leaks. Doing this accounting in the driver around drm_gpusvm_get_pages() / drm_gpusvm_unmap_pages() is error prone: the number, order and kind of the dma_addr[] entries can change between map and unmap (migration, partial unmaps, the iova vs non-iova paths), which makes symmetric accounting hard to get right.

Add an optional @dma_map_account callback to struct drm_gpusvm_ops and invoke it once per &drm_pagemap_addr entry at the exact points where the entry is DMA-mapped (sign == +1) in drm_gpusvm_get_pages() and unmapped (sign == -1) in __drm_gpusvm_unmap_pages(). Since the map error path and every unmap/free path funnel through __drm_gpusvm_unmap_pages() for the entries that were actually mapped, the accounting is symmetric by construction. The callback receives the full &drm_pagemap_addr so the driver can use the order and the proto field to distinguish system memory mappings from device interconnect mappings.

The callback must also be usable by the core drm_gpusvm_pages-only API (mm == NULL), as used e.g. for userptr-style mappings. Relax drm_gpusvm_init() to allow such users to pass a restricted @ops that only implements the page-level hooks; the full-SVM hooks (@invalidate) and the chunk configuration must still be unset in that mode.

Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/drm_gpusvm.c | 17 +++++++++++++++--
 include/drm/drm_gpusvm.h     | 19 +++++++++++++++++++
 2 files changed, 34 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_gpusvm.c +15 -2
index fcfe635bc195..512f52471419 100644
@@ -474,8 +474,15 @@ int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
 			return -EINVAL;
 		mmgrab(mm);
 	} else {
-		/* No full SVM mode, only core drm_gpusvm_pages API. */
-		if (ops || num_chunks || mm_range || notifier_size)
+		/*
+		 * No full SVM mode, only core drm_gpusvm_pages API. A
+		 * restricted @ops that only implements the page-level hooks
+		 * (e.g. @dma_map_account) is allowed; the full-SVM hooks must
+		 * not be set.
+		 */
+		if (num_chunks || mm_range || notifier_size)
+			return -EINVAL;
+		if (ops && ops->invalidate)
 			return -EINVAL;
 	}
 
@@ -1246,6 +1253,8 @@ static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm,
 			else if (dpagemap && dpagemap->ops->device_unmap)
 				dpagemap->ops->device_unmap(dpagemap,
 							    dev, addr);
+			if (gpusvm->ops && gpusvm->ops->dma_map_account)
+				gpusvm->ops->dma_map_account(gpusvm, addr, -1);
 			i += 1 << addr->order;
 		}
 
@@ -1654,6 +1663,10 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
 				(addr, DRM_INTERCONNECT_SYSTEM, order,
 				 dma_dir);
 		}
+		if (gpusvm->ops && gpusvm->ops->dma_map_account)
+			gpusvm->ops->dma_map_account(gpusvm,
+						     &svm_pages->dma_addr[j],
+						     1);
 		i += 1 << order;
 		num_dma_mapped = i;
 		flags.has_dma_mapping = true;
include/drm/drm_gpusvm.h +19 -0
index b7d987bf76aa..36f35816a569 100644
@@ -75,6 +75,25 @@ struct drm_gpusvm_ops {
 	void (*invalidate)(struct drm_gpusvm *gpusvm,
 			   struct drm_gpusvm_notifier *notifier,
 			   const struct mmu_notifier_range *mmu_range);
+
+	/**
+	 * @dma_map_account: Account a DMA-mapping change (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @addr: The address descriptor of the chunk being (un)mapped
+	 * @sign: +1 when @addr has just been DMA-mapped, -1 when it is being
+	 *        unmapped
+	 *
+	 * Called once per &drm_pagemap_addr entry, when the entry is
+	 * DMA-mapped by drm_gpusvm_get_pages() (@sign == +1) and when it is
+	 * unmapped (@sign == -1), so the driver can keep symmetric accounting
+	 * of the pages it has mapped for GPU access. The @addr->proto field
+	 * can be used to distinguish system-memory mappings from device
+	 * interconnect mappings. May be provided in both full SVM mode and the
+	 * core drm_gpusvm_pages-only mode.
+	 */
+	void (*dma_map_account)(struct drm_gpusvm *gpusvm,
+				const struct drm_pagemap_addr *addr,
+				int sign);
 };
 
 /**
Signature · 1 line
2.34.1
#22 lore raw

Re: [PATCH v2 11/33] drm/gpusvm: Add a DMA-mapping accounting callback

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Pre-existing issues:
- [High] Device memory mappings are leaked on the error path because dpagemap is not yet stored in svm_pages.
- [High] Reading uninitialized memory (svm_pages->dma_addr[0].dir) when unwinding an IOVA allocation failure.
- [High] System pages are incorrectly routed to device_unmap during unmapping of mixed ranges when IOVA is used. --

--- Patch [11]: [PATCH v2 11/33] drm/gpusvm: Add a DMA-mapping accounting callback --- commit 98db73f178129cc5ccc6277eaada197044dfbd3b
Author: Thomas Hellström <[email protected]>

drm/gpusvm: Add a DMA-mapping accounting callback

This commit adds an optional @dma_map_account callback to struct drm_gpusvm_ops to track mapped pages. It is invoked during mapping and unmapping to keep symmetric accounting. It also relaxes drm_gpusvm_init() to allow restricted ops without full-SVM mode.

Quoted 4 lines
drivers/gpu/drm/drm_gpusvm.c +0 -0
index fcfe635bc1956..512f52471419e 100644

[ ... ]

drivers/gpu/drm/drm_gpusvm.c static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm, · line 1255

Quoted 4 lines
drivers/gpu/drm/drm_gpusvm.c +0 -0
@@ -1246,6 +1253,8 @@ static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm,
 			else if (dpagemap && dpagemap->ops->device_unmap)
 				dpagemap->ops->device_unmap(dpagemap,
 							    dev, addr);

[Severity: High]
This isn't a bug introduced by this patch, but can this leak device memory mappings on the error path?

In drm_gpusvm_get_pages(), if an error occurs after successfully mapping one or more device pages, the code jumps to err_unmap and calls __drm_gpusvm_unmap_pages().

However, svm_pages->dpagemap is purposefully only updated after the mapping loop completes successfully. During the early error unwind, it remains NULL.

Because of this, in __drm_gpusvm_unmap_pages():

    struct drm_pagemap *dpagemap = svm_pages->dpagemap;

The dpagemap evaluated here will be NULL, causing this else if condition to fail and completely bypassing the unmap callback for all successfully mapped device pages in that chunk.

[Severity: High]
This is also a pre-existing issue, but could this incorrectly unmap system pages as device pages in a mixed chunk?

In drm_gpusvm_get_pages(), if a chunk has a mix of device and system pages (ctx->allow_mixed is true) and IOVA is used:

When iterating over the pages in __drm_gpusvm_unmap_pages(), system pages will have addr->proto == DRM_INTERCONNECT_SYSTEM. However, if use_iova is true, the previous condition:

    if (!use_iova && addr->proto == DRM_INTERCONNECT_SYSTEM)
        dma_unmap_page(...)

will evaluate to false.

Execution then falls through to this else if branch. If a previous device page set dpagemap, it will inappropriately call dpagemap->ops->device_unmap() on the system page.

Quoted 5 lines
drivers/gpu/drm/drm_gpusvm.c +2 -0
+			if (gpusvm->ops && gpusvm->ops->dma_map_account)
+				gpusvm->ops->dma_map_account(gpusvm, addr, -1);
 			i += 1 << addr->order;
 		}
 

[ ... ]

drivers/gpu/drm/drm_gpusvm.c int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm, · line 1669

Quoted 8 lines
drivers/gpu/drm/drm_gpusvm.c +4 -0
@@ -1654,6 +1663,10 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
 				(addr, DRM_INTERCONNECT_SYSTEM, order,
 				 dma_dir);
 		}
+		if (gpusvm->ops && gpusvm->ops->dma_map_account)
+			gpusvm->ops->dma_map_account(gpusvm,
+						     &svm_pages->dma_addr[j],
+						     1);

[Severity: High]
This is a pre-existing issue, but is it possible to read uninitialized memory here if the first page mapping fails?

In drm_gpusvm_get_pages(), svm_pages->dma_addr is allocated via kvmalloc_objs() without zero-initialization.

If an error occurs on the very first page (i = 0), such as an IOVA allocation failure, the code jumps to err_unmap with num_dma_mapped = 0, and __drm_gpusvm_unmap_pages() is called with npages = 0.

Inside __drm_gpusvm_unmap_pages(), if dma_use_iova(&svm_pages->state) is true, it will unconditionally call:

    dma_iova_destroy(dev, &svm_pages->state,
             svm_pages->state_offset,
             svm_pages->dma_addr[0].dir, 0);

Since npages = 0, svm_pages->dma_addr[0] was never initialized, leading to svm_pages->dma_addr[0].dir being read from uninitialized heap memory.

Quoted 3 lines
drivers/gpu/drm/drm_gpusvm.c +0 -0
 		i += 1 << order;
 		num_dma_mapped = i;
 		flags.has_dma_mapping = true;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
author maintainer
#23 lore raw

PATCH v2 12/33 drm/xe: Add debugfs stats for DMA-mapped pages per order

In reply to Matthew Brost

From: Thomas Hellström <[email protected]>

Expose per-page-order DMA mapping counts for the system memory that the xe driver maps for GPU access, split into two categories: TTM buffer objects, and the GPU SVM / userptr ranges (which share a single drm_gpusvm instance per VM).

The stats are visible at:

<debugfs>/dri/<N>/dma_mapped_pages

and are broken out into two rows (bo, svm/userptr), one column per page order, mirroring the layout of the TTM pool page_pool stat.

For TTM BOs the per-order counts are derived from the DMA scatter-gather
table: each mapped segment is physically contiguous, so its page count is split into the largest power-of-two runs (a segment of N pages contributes a 2^k chunk for the largest k with 2^k <= N, repeated for the remainder). Driving the accounting off the sg table rather than the pages[] array keeps it correct even when pages[] entries are cleared while still DMA-mapped, as happens on the defrag page-borrowing path.

For the SVM and userptr ranges the accounting is driven by the drm_gpusvm @dma_map_account callback, which fires once per dma_addr[] entry at the exact point the entry is DMA-mapped and unmapped. This makes the accounting symmetric by construction and avoids driver-side walks that could drift across migration, partial unmaps and the iova vs non-iova paths. Only DRM_INTERCONNECT_SYSTEM entries are real DMA maps and counted; device interconnect (VRAM, P2P) entries are skipped, so the counter reflects only system pages actually mapped through the DMA layer.

The callback is shared between the full SVM instance (fault mode) and the core drm_gpusvm_pages-only instance used for userptr, so userptr mappings are accounted in both the CONFIG_DRM_XE_GPUSVM=y and =n (but CONFIG_DRM_GPUSVM=y) configurations, into the same svm/userptr counter.

Since the counts are only ever consumed through debugfs, gate the counter storage and all accounting on CONFIG_DEBUG_FS so that builds without debugfs carry no extra atomic traffic or device state; the drm_gpusvm callback is simply left unregistered in that case.

On device teardown assert that every counter has returned to zero, to catch any unbalanced accounting (a leaked DMA mapping) during testing.

Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_bo.c           | 46 +++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_debugfs.c      | 26 +++++++++++++
 drivers/gpu/drm/xe/xe_device.c       | 24 ++++++++++++
 drivers/gpu/drm/xe/xe_device_types.h | 17 +++++++++
 drivers/gpu/drm/xe/xe_svm.c          | 36 +++++++++++++++++-
 drivers/gpu/drm/xe/xe_svm.h          |  3 +-
 drivers/gpu/drm/xe/xe_userptr.c      | 55 ++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_userptr.h      |  1 +
 8 files changed, 205 insertions(+), 3 deletions(-)
drivers/gpu/drm/xe/xe_bo.c +46 -0
index 85e6d9a0f575..60e0e568aa31 100644
@@ -367,6 +367,50 @@ struct xe_ttm_tt {
 	bool purgeable;
 };
 
+/*
+ * xe_tt_account_dma_pages - account DMA-mapped tt pages per order
+ * @xe: the xe device
+ * @xe_tt: the xe_ttm_tt whose sg table to walk
+ * @sign: +1 to add, -1 to subtract
+ *
+ * Walk the tt's DMA scatter-gather table and split each (physically
+ * contiguous) segment into the largest power-of-two page runs: a segment of N
+ * pages contributes a 2^k chunk for the largest k with 2^k <= N, repeated for
+ * the remainder. Accumulate the count of 2^k pages into
+ * xe->mem.dma_mapped_pages[k]. Driving this off the sg table rather than
+ * tt->pages keeps the accounting correct even when pages[] entries are cleared
+ * while still DMA-mapped (e.g. the defrag page-borrowing path).
+ */
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+static void xe_tt_account_dma_pages(struct xe_device *xe,
+				    struct xe_ttm_tt *xe_tt, int sign)
+{
+	struct scatterlist *sg;
+	unsigned int i;
+
+	for_each_sgtable_sg(xe_tt->sg, sg, i) {
+		unsigned long nr_pages = sg->length >> PAGE_SHIFT;
+
+		while (nr_pages) {
+			unsigned int order = min_t(unsigned int,
+						   __fls(nr_pages),
+						   MAX_PAGE_ORDER);
+			unsigned long chunk = 1UL << order;
+
+			atomic_long_add((long)chunk * sign,
+					&xe->mem.dma_mapped_pages[order]);
+
+			nr_pages -= chunk;
+		}
+	}
+}
+#else
+static void xe_tt_account_dma_pages(struct xe_device *xe,
+				    struct xe_ttm_tt *xe_tt, int sign)
+{
+}
+#endif
+
 static int xe_tt_map_sg(struct xe_device *xe, struct ttm_tt *tt)
 {
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
@@ -396,6 +440,7 @@ static int xe_tt_map_sg(struct xe_device *xe, struct ttm_tt *tt)
 		return ret;
 	}
 
+	xe_tt_account_dma_pages(xe, xe_tt, 1);
 	return 0;
 }
 
@@ -404,6 +449,7 @@ static void xe_tt_unmap_sg(struct xe_device *xe, struct ttm_tt *tt)
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
 
 	if (xe_tt->sg) {
+		xe_tt_account_dma_pages(xe, xe_tt, -1);
 		dma_unmap_sgtable(xe->drm.dev, xe_tt->sg,
 				  DMA_BIDIRECTIONAL, 0);
 		sg_free_table(xe_tt->sg);
drivers/gpu/drm/xe/xe_debugfs.c +26 -0
index 8c391c7b017a..97b47c5c4213 100644
@@ -6,6 +6,7 @@
 #include "xe_debugfs.h"
 
 #include <linux/debugfs.h>
+#include <linux/mmzone.h>
 #include <linux/fault-inject.h>
 #include <linux/string_helpers.h>
 
@@ -231,6 +232,28 @@ static int dgfx_pcie_link_residencies_show(struct seq_file *m, void *data)
 	return 0;
 }
 
+static int dma_mapped_pages_show(struct seq_file *m, void *data)
+{
+	struct xe_device *xe = m->private;
+	unsigned int i;
+
+	seq_printf(m, "%-11s ", "");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " ---%2u---", i);
+	seq_printf(m, "\n%-11s:", "bo");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " %8lu",
+			   atomic_long_read(&xe->mem.dma_mapped_pages[i]));
+	seq_printf(m, "\n%-11s:", "svm/userptr");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " %8lu",
+			   atomic_long_read(&xe->mem.dma_mapped_pages_svm[i]));
+	seq_puts(m, "\n");
+
+	return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(dma_mapped_pages);
+
 static const struct drm_info_list debugfs_list[] = {
 	{"info", info, 0},
 	{ .name = "sriov_info", .show = sriov_info, },
@@ -632,6 +655,9 @@ void xe_debugfs_register(struct xe_device *xe)
 	if (man)
 		ttm_resource_manager_create_debugfs(man, root, "stolen_mm");
 
+	debugfs_create_file("dma_mapped_pages", 0444, root, xe,
+			    &dma_mapped_pages_fops);
+
 	for_each_tile(tile, xe, tile_id)
 		xe_tile_debugfs_register(tile);
 
drivers/gpu/drm/xe/xe_device.c +24 -0
index ad7f3e61d457..0c4ee874a56d 100644
@@ -455,10 +455,34 @@ bool xe_device_is_admin_only(const struct xe_device *xe)
 }
 #endif
 
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+static void xe_device_assert_dma_pages_zero(struct xe_device *xe)
+{
+	unsigned int i;
+
+	/*
+	 * All BOs, userptr VMAs and SVM ranges must have been torn down by the
+	 * time the device is destroyed, so every DMA-mapped-pages counter must
+	 * have returned to zero. A non-zero value indicates unbalanced
+	 * accounting, i.e. a missing unmap-side decrement.
+	 */
+	for (i = 0; i < NR_PAGE_ORDERS; i++) {
+		xe_assert(xe, !atomic_long_read(&xe->mem.dma_mapped_pages[i]));
+		xe_assert(xe, !atomic_long_read(&xe->mem.dma_mapped_pages_svm[i]));
+	}
+}
+#else
+static void xe_device_assert_dma_pages_zero(struct xe_device *xe)
+{
+}
+#endif
+
 static void xe_device_destroy(struct drm_device *dev, void *dummy)
 {
 	struct xe_device *xe = to_xe_device(dev);
 
+	xe_device_assert_dma_pages_zero(xe);
+
 	xe_bo_dev_fini(&xe->bo_device);
 
 	if (xe->preempt_fence_wq)
drivers/gpu/drm/xe/xe_device_types.h +17 -0
index 022e08205897..95e7cea91cbf 100644
@@ -6,6 +6,7 @@
 #ifndef _XE_DEVICE_TYPES_H_
 #define _XE_DEVICE_TYPES_H_
 
+#include <linux/mmzone.h>
 #include <linux/pci.h>
 
 #include <drm/drm_device.h>
@@ -279,6 +280,22 @@ struct xe_device {
 		struct xe_shrinker *shrinker;
 		/** @mem.stolen_mgr: stolen memory manager. */
 		struct xe_ttm_stolen_mgr *stolen_mgr;
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+		/**
+		 * @mem.dma_mapped_pages: number of DMA-mapped pages per page
+		 * order currently live for this device, for TTM BOs. Only
+		 * accounted when CONFIG_DEBUG_FS is enabled, since it is solely
+		 * exposed through debugfs.
+		 */
+		atomic_long_t dma_mapped_pages[NR_PAGE_ORDERS];
+		/**
+		 * @mem.dma_mapped_pages_svm: number of DMA-mapped pages per
+		 * page order currently live for this device, for SVM ranges
+		 * and userptr VMAs (both share a single drm_gpusvm instance).
+		 * Only accounted when CONFIG_DEBUG_FS is enabled.
+		 */
+		atomic_long_t dma_mapped_pages_svm[NR_PAGE_ORDERS];
+#endif
 	} mem;
 
 	/** @sriov: device level virtualization data */
drivers/gpu/drm/xe/xe_svm.c +34 -2
index b228a737cfd6..2f639fc0cac4 100644
@@ -827,10 +827,42 @@ static int xe_svm_get_pagemaps(struct xe_vm *vm)
 }
 #endif
 
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+/*
+ * xe_svm_dma_map_account - drm_gpusvm DMA-mapping accounting callback
+ * @gpusvm: The GPU SVM the mapping belongs to
+ * @addr: The address descriptor of the chunk being (un)mapped
+ * @sign: +1 when @addr was DMA-mapped, -1 when it is being unmapped
+ *
+ * Maintain per-order counts of the system-memory pages that the full SVM
+ * instance (SVM ranges, and userptr VMAs in fault mode) has DMA-mapped for GPU
+ * access. Only DRM_INTERCONNECT_SYSTEM entries are real DMA maps and counted;
+ * device interconnect (VRAM, P2P) entries are skipped. The counts are solely
+ * exposed through debugfs, so the callback is only registered when
+ * CONFIG_DEBUG_FS is enabled.
+ */
+static void xe_svm_dma_map_account(struct drm_gpusvm *gpusvm,
+				   const struct drm_pagemap_addr *addr,
+				   int sign)
+{
+	struct xe_device *xe = gpusvm_to_vm(gpusvm)->xe;
+	unsigned int order = addr->order;
+
+	if (addr->proto != DRM_INTERCONNECT_SYSTEM)
+		return;
+
+	atomic_long_add((long)(1UL << order) * sign,
+			&xe->mem.dma_mapped_pages_svm[order]);
+}
+#endif
+
 static const struct drm_gpusvm_ops gpusvm_ops = {
 	.range_alloc = xe_svm_range_alloc,
 	.range_free = xe_svm_range_free,
 	.invalidate = xe_svm_invalidate,
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+	.dma_map_account = xe_svm_dma_map_account,
+#endif
 };
 
 static const unsigned long fault_chunk_sizes[] = {
@@ -925,8 +957,8 @@ int xe_svm_init(struct xe_vm *vm)
 		}
 	} else {
 		err = drm_gpusvm_init(&vm->svm.gpusvm, "Xe SVM (simple)",
-				      NULL, 0, 0, 0, NULL,
-				      NULL, 0);
+				      NULL, 0, 0, 0,
+				      xe_userptr_gpusvm_ops_get(), NULL, 0);
 	}
 
 	return err;
drivers/gpu/drm/xe/xe_svm.h +2 -1
index a921556d3466..edd668c41551 100644
@@ -236,7 +236,8 @@ int xe_svm_init(struct xe_vm *vm)
 {
 #if IS_ENABLED(CONFIG_DRM_GPUSVM)
 	return drm_gpusvm_init(&vm->svm.gpusvm, "Xe SVM (simple)",
-			       NULL, 0, 0, 0, NULL, NULL, 0);
+			       NULL, 0, 0, 0, NULL, NULL,
+			       xe_userptr_gpusvm_ops_get(), 0);
 #else
 	return 0;
 #endif
drivers/gpu/drm/xe/xe_userptr.c +55 -0

64 lines not on this page · raw message

drivers/gpu/drm/xe/xe_userptr.h +1 -0
index 2a3cd1b5efbb..1e9601be5713 100644
@@ -108,6 +108,7 @@ int __xe_vm_userptr_needs_repin(struct xe_vm *vm);
 int xe_vm_userptr_check_repin(struct xe_vm *vm);
 int xe_vma_userptr_pin_pages(struct xe_userptr_vma *uvma);
 int xe_vma_userptr_check_repin(struct xe_userptr_vma *uvma);
+const struct drm_gpusvm_ops *xe_userptr_gpusvm_ops_get(void);
 #else
 static inline void xe_userptr_remove(struct xe_userptr_vma *uvma) {}
 
Signature · 1 line
2.34.1
#24 lore raw

Re: [PATCH v2 12/33] drm/xe: Add debugfs stats for DMA-mapped pages per order

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:

New issues:
- [High] xe_device_assert_dma_pages_zero() is called too early in xe_device_destroy(), leading to spurious kernel assertion failures.
- [High] Uncapped addr->order in SVM and userptr DMA accounting leads to out-of-bounds array writes.
- [Low] Too many arguments to drm_gpusvm_init in xe_svm.h causes a compilation error when CONFIG_DRM_GPUSVM is enabled but CONFIG_DRM_XE_GPUSVM is disabled.

Pre-existing issues:
- [Critical] Asynchronous execution of vm_destroy_work_func can cause a Use-After-Free of the xe device structure. --

--- Patch [12]: [PATCH v2 12/33] drm/xe: Add debugfs stats for DMA-mapped pages per order --- commit 808c14d37a18acfee1cc7559a4568116c9d63f8b
Author: Thomas Hellström <[email protected]>
Subject: drm/xe: Add debugfs stats for DMA-mapped pages per order

This commit adds per-page-order DMA mapping counters for system memory accessible by the GPU. The stats cover both TTM buffer objects and SVM/userptr ranges, exposing them through debugfs.

Quoted 4 lines
drivers/gpu/drm/xe/xe_device.c +0 -0
index ad7f3e61d4579..0c4ee874a56d7 100644

[ ... ]

drivers/gpu/drm/xe/xe_device.c static void xe_device_destroy(struct drm_device *dev, void *dummy) · line 504

Quoted 7 lines
drivers/gpu/drm/xe/xe_device.c +2 -0
@@ -475,8 +499,10 @@ static void xe_device_destroy(struct drm_device *dev, void *dummy)
 {
 	struct xe_device *xe = to_xe_device(dev);
 
+	xe_device_assert_dma_pages_zero(xe);
+
 	xe_bo_dev_fini(&xe->bo_device);

[Severity: High]
Does this assertion trigger prematurely if the asynchronous BO destruction worker has pending buffers to free?

In xe_device_destroy(), xe_device_assert_dma_pages_zero() is called before xe_bo_dev_fini() flushes the async_free worker. If there are delayed destructions, wouldn't the DMA mappings still be active, causing the counters to be non-zero and triggering a kernel warning?

Quoted 4 lines
drivers/gpu/drm/xe/xe_svm.c +0 -0
index b228a737cfd6d..2f639fc0cac4e 100644

[ ... ]

drivers/gpu/drm/xe/xe_svm.c static int xe_svm_get_pagemaps(struct xe_vm *vm) · line 855

Quoted 12 lines
drivers/gpu/drm/xe/xe_svm.c +12 -0
+static void xe_svm_dma_map_account(struct drm_gpusvm *gpusvm,
+				   const struct drm_pagemap_addr *addr,
+				   int sign)
+{
+	struct xe_device *xe = gpusvm_to_vm(gpusvm)->xe;
+	unsigned int order = addr->order;
+
+	if (addr->proto != DRM_INTERCONNECT_SYSTEM)
+		return;
+
+	atomic_long_add((long)(1UL << order) * sign,
+			&xe->mem.dma_mapped_pages_svm[order]);

[Severity: High]
Can this cause an out-of-bounds write if addr->order exceeds the array bounds of dma_mapped_pages_svm?

In xe_svm_dma_map_account(), addr->order is used directly as an array index. If an unprivileged user maps a large huge page (such as a 1GB page with an order of 18) and faults it into the GPU, wouldn't this exceed the NR_PAGE_ORDERS limit and corrupt adjacent kernel memory?

Quoted 7 lines
drivers/gpu/drm/xe/xe_svm.c +2 -0
+}
+#endif
drivers/gpu/drm/xe/xe_svm.h +0 -0
index a921556d34662..edd668c415512 100644

[ ... ]

drivers/gpu/drm/xe/xe_svm.h int xe_svm_init(struct xe_vm *vm) · line 240

Quoted 7 lines
drivers/gpu/drm/xe/xe_svm.h +2 -1
@@ -236,7 +236,8 @@ int xe_svm_init(struct xe_vm *vm)
 {
 #if IS_ENABLED(CONFIG_DRM_GPUSVM)
 	return drm_gpusvm_init(&vm->svm.gpusvm, "Xe SVM (simple)",
-			       NULL, 0, 0, 0, NULL, NULL, 0);
+			       NULL, 0, 0, 0, NULL, NULL,
+			       xe_userptr_gpusvm_ops_get(), 0);

[Severity: Low]
Does this function call pass too many arguments to drm_gpusvm_init()?

In the xe_svm_init() inline stub, the xe_userptr_gpusvm_ops_get() argument was added without removing one of the NULL placeholders. Since drm_gpusvm_init() expects 9 arguments and this now provides 10, won't this cause a compilation error when CONFIG_DRM_GPUSVM is enabled but CONFIG_DRM_XE_GPUSVM is disabled?

Quoted 8 lines
drivers/gpu/drm/xe/xe_svm.h +0 -0
 #else
 	return 0;
 #endif
drivers/gpu/drm/xe/xe_userptr.c +0 -0
index 8b2d461ea0b28..79c2a81001f2e 100644

[ ... ]

drivers/gpu/drm/xe/xe_userptr.c line 41

Quoted 12 lines
drivers/gpu/drm/xe/xe_userptr.c +12 -0
+static void xe_userptr_dma_map_account(struct drm_gpusvm *gpusvm,
+				       const struct drm_pagemap_addr *addr,
+				       int sign)
+{
+	struct xe_vm *vm = container_of(gpusvm, struct xe_vm, svm.gpusvm);
+	unsigned int order = addr->order;
+
+	if (addr->proto != DRM_INTERCONNECT_SYSTEM)
+		return;
+
+	atomic_long_add((long)(1UL << order) * sign,
+			&vm->xe->mem.dma_mapped_pages_svm[order]);

[Severity: High]
Similarly to xe_svm_dma_map_account(), can this cause an out-of-bounds write if addr->order is larger than NR_PAGE_ORDERS?

Using addr->order directly as an array index without capping it could allow unprivileged user-space applications mapping large huge pages to corrupt memory when invoking xe_userptr_dma_map_account().

Quoted 1 line

+}

While reviewing the VM lifecycle code related to this patch, I noticed a separate pre-existing issue in drivers/gpu/drm/xe/xe_vm.c.

[Severity: Critical]
This is a pre-existing issue, but does queueing the destroy work asynchronously lead to a use-after-free if the last reference to the DRM device is dropped before the worker executes?

In xe_vm_free(), the destruction work is queued to system_dfl_wq:

	queue_work(system_dfl_wq, &vm->destroy_work);

If this drops the last device reference, the xe device could be freed via RCU before vm_destroy_work_func() runs. Since vm_destroy_work_func() dereferences the xe pointer (for example, when calling xe_pm_runtime_put()), wouldn't this result in a use-after-free?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
author maintainer
#25 lore raw

PATCH v2 13/33 drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind()

In reply to Matthew Brost

On L2-flush-optimized HW with a dma-resv (non-fault) VM, evicting a BO only needs to flush the L2 cache before the migration copy runs; the mappings themselves are torn down and rebuilt lazily via drm_gpuvm_bo_evict() and a subsequent rebind. Today this flush is done by waiting for the BO to go idle and then issuing a synchronous TLB invalidation per mapping VMA from inside xe_bo_trigger_rebind(). Both the idle wait and the synchronous invalidation stall the calling thread while holding the BO dma-resv lock, serializing the move behind all in-flight GPU work on the BO.

Replace this with an asynchronous flush. Add xe_vm_flush_vm_bo_tlb_async() which, for each VMA mapping the BO on each present tile, queues a TLB invalidation job on the tile migrate (kernel) exec queue. The jobs depend on the BO's in-flight GPU work, captured once as a singleton over DMA_RESV_USAGE_BOOKKEEP, so the flush only fires once the GPU is done with the current mapping. Each job's completion fence is installed into the BO's dma-resv as a DMA_RESV_USAGE_KERNEL fence, so the migration copy - which waits on the resv - waits on the flush without stalling this thread.

No PTEs are zapped and vma->tile_invalidated is left untouched: the mapping stays valid until the lazy rebind, and the only work performed here is the L2 flush. On any failure the caller falls back to the existing blocking wait-idle plus xe_vm_invalidate_vma() path.

Cc Thomas Hellström <[email protected]>
Cc Tejas Upadhyay <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_bo.c |   7 +++
 drivers/gpu/drm/xe/xe_vm.c | 125 +++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_vm.h |   5 ++
 3 files changed, 137 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +7 -0
index 60e0e568aa31..cca617cf34d1 100644
@@ -732,6 +732,13 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,
 			 */
 			if (!xe_device_is_l2_flush_optimized(xe))
 				continue;
+
+			/*
+			 * Attempt to flush L2 async, fallback to sync flush on
+			 * failure
+			 */
+			if (!xe_vm_flush_vm_bo_tlb_async(vm, bo, vm_bo))
+				continue;
 		}
 
 		if (!idle) {
drivers/gpu/drm/xe/xe_vm.c +125 -0

137 lines not on this page · raw message

drivers/gpu/drm/xe/xe_vm.h +5 -0
index c5b900f38ded..dd5b070eaede 100644
@@ -26,6 +26,8 @@ struct ttm_buffer_object;
 
 struct dma_fence;
 
+struct xe_bo;
+struct drm_gpuvm_bo;
 struct xe_exec_queue;
 struct xe_file;
 struct xe_pagefault;
@@ -254,6 +256,9 @@ int xe_vm_invalidate_vma(struct xe_vma *vma);
 
 int xe_vm_invalidate_vma_submit(struct xe_vma *vma, struct xe_tlb_inval_batch *batch);
 
+int xe_vm_flush_vm_bo_tlb_async(struct xe_vm *vm, struct xe_bo *bo,
+				struct drm_gpuvm_bo *vm_bo);
+
 int xe_vm_validate_protected(struct xe_vm *vm);
 
 static inline void xe_vm_queue_rebind_worker(struct xe_vm *vm)
Signature · 1 line
2.34.1
#26 lore raw

Re: [PATCH v2 13/33] drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind()

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] GFP_KERNEL allocation in shrinker and dma-resv critical section causes deadlocks and lockdep…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] GFP_KERNEL allocation in shrinker and dma-resv critical section causes deadlocks and lockdep splats --

--- Patch [13]: [PATCH v2 13/33] drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind() --- commit 985db7b22213ce9503872b19a27fddeba9fe24ff
Author: Matthew Brost <[email protected]>

drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind()

This commit replaces synchronous TLB invalidations during buffer object eviction with asynchronous flushes on L2-flush-optimized hardware. It queues a TLB invalidation job per VMA, which waits on the buffer's in-flight GPU work before flushing.

Quoted 3 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0

[ ... ]

drivers/gpu/drm/xe/xe_bo.c static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo, · line 741

Quoted 11 lines
drivers/gpu/drm/xe/xe_bo.c +7 -0
@@ -732,6 +732,13 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,
 			 */
 			if (!xe_device_is_l2_flush_optimized(xe))
 				continue;
+
+			/*
+			 * Attempt to flush L2 async, fallback to sync flush on
+			 * failure
+			 */
+			if (!xe_vm_flush_vm_bo_tlb_async(vm, bo, vm_bo))
+				continue;

[Severity: High]
Does this create a deadlock regression when reached from the shrinker path?

The memory shrinker can call into this path through xe_shrinker_scan() -> xe_bo_shrink() -> xe_bo_move_notify() -> xe_bo_trigger_rebind().

In this path, the shrinker operates under the fs_reclaim pseudo-lock and holds the buffer object's dma-resv lock.

Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c +0 -0

[ ... ]

drivers/gpu/drm/xe/xe_vm.c int xe_vm_invalidate_vma(struct xe_vma *vma) · line 4427

Quoted 13 lines
drivers/gpu/drm/xe/xe_vm.c +13 -0
+static int xe_vma_tlb_flush_client(struct xe_vm *vm, struct xe_vma *vma,
+				   struct xe_tile *tile, struct xe_gt *gt,
+				   struct dma_resv *resv, struct dma_fence *dep,
+				   int type)
+{
+	struct xe_exec_queue *q = xe_migrate_exec_queue(tile->migrate);
+	struct xe_tlb_inval_job *job;
+	struct dma_fence *fence;
+	int err;
+
+	job = xe_tlb_inval_job_create(q, &gt->tlb_inval,
+				      q->tlb_inval[type].dep_scheduler, vm,
+				      xe_vma_start(vma), xe_vma_end(vma), type);

[Severity: High]
Could this allocation recurse into memory reclaim and attempt to acquire the same dma-resv lock, or trigger a lockdep splat for context violations?

The xe_tlb_inval_job_create() function allocates the job using a default GFP_KERNEL allocation. Furthermore, the subsequent call to xe_tlb_inval_job_alloc_dep() explicitly calls might_alloc(GFP_KERNEL).

Since this can be reached while holding the dma-resv lock or while in the shrinker reclaim path, these GFP_KERNEL allocations appear to violate the context constraints.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
author maintainer
#27 lore raw

PATCH v2 14/33 drm/xe: Destroy page tables after unlinking all VMAs on VM close

In reply to Matthew Brost

xe_vm_close_and_put() freed the VM's page-table roots via xe_vm_pt_destroy() before destroying the "contested" VMAs - those backed by external (shared) BOs, which are deferred to xe_vma_destroy_unlocked() because unlinking them requires taking the external BO's dma-resv lock (via drm_exec), not vm->lock.

Until such a VMA is unlinked it remains reachable from its BO through drm_gem_for_each_gpuvm_bo(). A concurrent BO move that only holds the BO's dma-resv lock - TTM eviction, the shrinker, or the defrag worker - calls xe_bo_move_notify() -> xe_bo_trigger_rebind() -> xe_vm_invalidate_vma() -> xe_pt_zap_ptes(), which dereferences vm->pt_root[tile->id]. With the page tables already freed this is a use-after-free / NULL dereference, e.g.:

  Oops: general protection fault [...] kernel NULL pointer dereference
  RIP: xe_pt_zap_ptes+0x7c [xe]
  Call Trace:
   xe_vm_invalidate_vma_submit+0x64/0x130 [xe]
   xe_vm_invalidate_vma+0x27/0x60 [xe]
   xe_bo_move_notify+0x1e7/0x3d0 [xe]
   xe_bo_move+0x62d/0x1620 [xe]
   ttm_bo_handle_move_mem+0x243/0x330 [ttm]
   ttm_bo_validate+0xb4/0x1a0 [ttm]

The race is not closable with a flag or a NULL pt_root check in the invalidate path: the freeing side holds only vm->lock while the walker holds only the BO's resv, so there is no common lock to serialise on.

Fix the ordering instead: destroy the page tables only after every VMA - including the contested external-BO VMAs - has been unlinked. Once the contested loop completes no BO can reach a VMA of this VM, so freeing the page tables (under vm->lock, to satisfy xe_vm_assert_held()) is safe. xe_vm_free_scratch() still runs before the free as it reads pt_root[]->level.

Fixes dd08ebf6c352 ("drm/xe: Introduce a new DRM driver for Intel GPUs")
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_vm.c | 24 ++++++++++++++++--------
 1 file changed, 16 insertions(+), 8 deletions(-)
drivers/gpu/drm/xe/xe_vm.c +16 -8
index 4557a8a4d270..9f55d22cb985 100644
@@ -1934,15 +1934,7 @@ void xe_vm_close_and_put(struct xe_vm *vm)
 		vma->gpuva.flags |= XE_VMA_DESTROYED;
 	}
 
-	/*
-	 * All vm operations will add shared fences to resv.
-	 * The only exception is eviction for a shared object,
-	 * but even so, the unbind when evicted would still
-	 * install a fence to resv. Hence it's safe to
-	 * destroy the pagetables immediately.
-	 */
 	xe_vm_free_scratch(vm);
-	xe_vm_pt_destroy(vm);
 	xe_vm_unlock(vm);
 
 	/*
@@ -1956,6 +1948,22 @@ void xe_vm_close_and_put(struct xe_vm *vm)
 		xe_vma_destroy_unlocked(vma);
 	}
 
+	/*
+	 * Destroy the page tables only after every VMA has been unlinked. The
+	 * contested (external-BO) VMAs above are unlinked from their BO under
+	 * the BO's dma-resv lock, not vm->lock, so they can still be reached
+	 * via drm_gem_for_each_gpuvm_bo() by a concurrent BO move (eviction,
+	 * shrinker or the defrag worker) that only holds the BO's resv. Such a
+	 * move calls xe_vm_invalidate_vma() -> xe_pt_zap_ptes(), which
+	 * dereferences vm->pt_root[]; freeing the page tables while those VMAs
+	 * are still linked is a use-after-free. All vm operations add shared
+	 * fences to the resv (eviction of a shared object still installs an
+	 * unbind fence), so no GPU work outlives this point.
+	 */
+	xe_vm_lock(vm, false);
+	xe_vm_pt_destroy(vm);
+	xe_vm_unlock(vm);
+
 	xe_svm_fini(vm);
 
 	up_write(&vm->lock);
Signature · 1 line
2.34.1
author maintainer
#28 lore raw

PATCH v2 15/33 drm/xe: Track BOs backed at a sub-optimal page order

In reply to Matthew Brost

Add per-device tracking of BOs whose backing TT pages were allocated below the pool's beneficial order, as reported by the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tt page flag (read via ttm_tt_is_beneficial_order_failed()). These BOs are future candidates for a defragmentation pass that reallocates their pages at the beneficial order.

Add a mem.defrag sub-struct (lock, list, atomic count) to struct xe_device and a defrag_link to struct xe_bo, initialised at BO create and device init. Membership is updated from xe_bo_move() via xe_bo_defrag_update(): a BO is added when it is a populated, needed, non-pinned ttm_bo_type_device object flagged TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED, and removed otherwise. BOs are also removed on pin, on purge, moved to dontneed, and at destroy.

Only ttm_bo_type_device BOs are tracked.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_bo.c           | 115 +++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_bo.h           |   1 +
 drivers/gpu/drm/xe/xe_bo_types.h     |   6 ++
 drivers/gpu/drm/xe/xe_device.c       |   2 +
 drivers/gpu/drm/xe/xe_device_types.h |  19 +++++
 5 files changed, 143 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +115 -0
index cca617cf34d1..d7b0536564ab 100644
@@ -920,6 +920,9 @@ static void xe_bo_set_purgeable_shrinker(struct xe_bo *bo,
 	}
 }
 
+static void xe_bo_defrag_update(struct xe_bo *bo);
+static void xe_bo_defrag_remove(struct xe_bo *bo);
+
 /**
  * xe_bo_set_purgeable_state() - Set BO purgeable state with validation
  * @bo: Buffer object
@@ -948,6 +951,7 @@ void xe_bo_set_purgeable_state(struct xe_bo *bo,
 
 	bo->purgeable.state = new_state;
 	xe_bo_set_purgeable_shrinker(bo, new_state);
+	xe_bo_defrag_update(bo);
 }
 
 /**
@@ -997,9 +1001,100 @@ static int xe_ttm_bo_purge(struct ttm_buffer_object *ttm_bo, struct ttm_operatio
 	/* Commit the state transition only once invalidation was queued */
 	xe_bo_set_purgeable_state(bo, XE_MADV_PURGEABLE_PURGED);
 
+	/*
+	 * The backing store is gone, so the BO no longer needs defrag. Drop it
+	 * from the defrag list here so the count stays accurate even for purge
+	 * paths that don't pass through xe_bo_move() (e.g. swap_notify).
+	 */
+	xe_bo_defrag_remove(bo);
+
 	return 0;
 }
 
+/**
+ * xe_bo_defrag_init_early() - Initialize the device defrag BO tracking
+ * @xe: The xe device
+ *
+ * Initialize the list, lock and count used to track BOs whose backing TT
+ * pages were allocated at a sub-optimal order.
+ */
+void xe_bo_defrag_init_early(struct xe_device *xe)
+{
+	spin_lock_init(&xe->mem.defrag.lock);
+	INIT_LIST_HEAD(&xe->mem.defrag.list);
+	atomic_set(&xe->mem.defrag.count, 0);
+}
+
+static bool xe_bo_needs_defrag(struct xe_bo *bo)
+{
+	struct ttm_buffer_object *ttm_bo = &bo->ttm;
+	struct ttm_tt *tt = ttm_bo->ttm;
+
+	xe_bo_assert_held(bo);
+
+	return ttm_bo->type == ttm_bo_type_device && tt &&
+		!xe_bo_madv_is_dontneed(bo) &&
+		ttm_tt_is_populated(tt) &&
+		ttm_tt_is_beneficial_order_failed(tt) &&
+		ttm_bo->resource && ttm_bo->resource->mem_type == XE_PL_TT &&
+		!xe_bo_is_pinned(bo);
+}
+
+static void xe_bo_defrag_add(struct xe_bo *bo)
+{
+	struct xe_device *xe = xe_bo_device(bo);
+
+	xe_bo_assert_held(bo);
+	xe_assert(xe, xe_bo_needs_defrag(bo));
+
+	scoped_guard(spinlock, &xe->mem.defrag.lock) {
+		if (list_empty(&bo->defrag_link)) {
+			list_add_tail(&bo->defrag_link, &xe->mem.defrag.list);
+			atomic_inc(&xe->mem.defrag.count);
+		}
+	}
+}
+
+static void __xe_bo_defrag_remove(struct xe_bo *bo)
+{
+	struct xe_device *xe = xe_bo_device(bo);
+
+	guard(spinlock)(&xe->mem.defrag.lock);
+	if (!list_empty(&bo->defrag_link)) {
+		list_del_init(&bo->defrag_link);
+		atomic_dec(&xe->mem.defrag.count);
+	}
+}
+
+static void xe_bo_defrag_remove(struct xe_bo *bo)
+{
+	xe_bo_assert_held(bo);
+
+	if (list_empty(&bo->defrag_link))
+		return;
+
+	__xe_bo_defrag_remove(bo);
+}
+
+/**
+ * xe_bo_defrag_update() - Update defrag list membership for a BO
+ * @bo: The buffer object
+ *
+ * Add @bo to the device defrag list when it is a ttm_bo_type_device BO resident
+ * in XE_PL_TT with a populated TT whose pages were allocated at a sub-optimal
+ * order (ttm_tt_is_beneficial_order_failed()) and it isn't pinned. Otherwise
+ * ensure it is removed from the list.
+ */
+static void xe_bo_defrag_update(struct xe_bo *bo)
+{
+	xe_bo_assert_held(bo);
+
+	if (xe_bo_needs_defrag(bo))
+		xe_bo_defrag_add(bo);
+	else
+		xe_bo_defrag_remove(bo);
+}
+
 static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		      struct ttm_operation_ctx *ctx,
 		      struct ttm_resource *new_mem,
@@ -1225,6 +1320,8 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		xe_tt_unmap_sg(xe, ttm_bo->ttm);
 	}
 
+	xe_bo_defrag_update(bo);
+
 	return ret;
 }
 
@@ -1261,6 +1358,15 @@ static long xe_bo_shrink_purge(struct ttm_operation_ctx *ctx,
 	if (lret > 0) {
 		xe_ttm_tt_account_subtract(xe, bo->ttm);
 		update_global_total_pages(bo->bdev, -(long)tt->num_pages);
+
+		/*
+		 * The pages are gone, so a tracked BO no longer needs defrag.
+		 * This shrinker purge path frees pages without going through
+		 * xe_bo_move(), so drop any stale defrag list entry here. Ghost
+		 * bos are never tracked, so guard for real xe bos.
+		 */
+		if (xe_bo_is_xe_bo(bo))
+			xe_bo_defrag_remove(ttm_to_xe_bo(bo));
 	}
 
 	return lret;
@@ -1903,6 +2009,8 @@ static void xe_ttm_bo_destroy(struct ttm_buffer_object *ttm_bo)
 		list_del(&bo->vram_userfault_link);
 	mutex_unlock(&xe->mem_access.vram_userfault.lock);
 
+	__xe_bo_defrag_remove(bo);
+
 	kfree(bo);
 }
 
@@ -2408,6 +2516,7 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,
 	bo->ttm.base.funcs = &xe_gem_object_funcs;
 	bo->ttm.priority = XE_BO_PRIORITY_NORMAL;
 	INIT_LIST_HEAD(&bo->pinned_link);
+	INIT_LIST_HEAD(&bo->defrag_link);
 #ifdef CONFIG_PROC_FS
 	INIT_LIST_HEAD(&bo->client_link);
 #endif
@@ -3014,6 +3123,8 @@ int xe_bo_pin_external(struct xe_bo *bo, bool in_place, struct drm_exec *exec)
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_subtract(xe, bo->ttm.ttm);
 
+	xe_bo_defrag_remove(bo);
+
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
 	 * we do not need this.
@@ -3072,6 +3183,8 @@ int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec)
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_subtract(xe, bo->ttm.ttm);
 
+	xe_bo_defrag_remove(bo);
+
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
 	 * we do not need this.
@@ -3107,6 +3220,7 @@ void xe_bo_unpin_external(struct xe_bo *bo)
 	ttm_bo_unpin(&bo->ttm);
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_add(xe, bo->ttm.ttm);
+	xe_bo_defrag_update(bo);
 
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
@@ -3139,6 +3253,7 @@ void xe_bo_unpin(struct xe_bo *bo)
 	ttm_bo_unpin(&bo->ttm);
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_add(xe, bo->ttm.ttm);
+	xe_bo_defrag_update(bo);
 }
 
 /**
drivers/gpu/drm/xe/xe_bo.h +1 -0
index 6340317f7d2e..4186cb51af87 100644
@@ -224,6 +224,7 @@ int xe_bo_pin_external(struct xe_bo *bo, bool in_place, struct drm_exec *exec);
 int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec);
 void xe_bo_unpin_external(struct xe_bo *bo);
 void xe_bo_unpin(struct xe_bo *bo);
+void xe_bo_defrag_init_early(struct xe_device *xe);
 int xe_bo_validate(struct xe_bo *bo, struct xe_vm *vm, bool allow_res_evict,
 		   struct drm_exec *exec);
 
drivers/gpu/drm/xe/xe_bo_types.h +6 -0
index fcc63ae3f455..9eea9ab380e5 100644
@@ -54,6 +54,12 @@ struct xe_bo {
 	struct ttm_bo_kmap_obj kmap;
 	/** @pinned_link: link to present / evicted list of pinned BO */
 	struct list_head pinned_link;
+	/**
+	 * @defrag_link: link into @xe_device.mem.defrag.list for BOs whose
+	 * backing TT pages were allocated at a sub-optimal order. Protected by
+	 * @xe_device.mem.defrag.lock.
+	 */
+	struct list_head defrag_link;
 #ifdef CONFIG_PROC_FS
 	/**
 	 * @client: @xe_drm_client which created the bo
drivers/gpu/drm/xe/xe_device.c +2 -0
index 0c4ee874a56d..d4321419ee35 100644
@@ -601,6 +601,8 @@ int xe_device_init_early(struct xe_device *xe)
 	if (err)
 		return err;
 
+	xe_bo_defrag_init_early(xe);
+
 	xe->preempt_fence_wq = alloc_ordered_workqueue("xe-preempt-fence-wq",
 						       WQ_MEM_RECLAIM);
 	xe->ordered_wq = alloc_ordered_workqueue("xe-ordered-wq", 0);
drivers/gpu/drm/xe/xe_device_types.h +19 -0
index 95e7cea91cbf..9e6244d8f41b 100644
@@ -296,6 +296,25 @@ struct xe_device {
 		 */
 		atomic_long_t dma_mapped_pages_svm[NR_PAGE_ORDERS];
 #endif
+		/**
+		 * @mem.defrag: Tracking of BOs whose backing TT pages were
+		 * allocated at a sub-optimal (smaller than beneficial) order.
+		 *
+		 * Such BOs are candidates for a future defragmentation pass that
+		 * tries to reallocate their pages at the device's beneficial
+		 * order. Only ttm_bo_type_device BOs are tracked.
+		 */
+		struct {
+			/** @mem.defrag.lock: Protects @mem.defrag.list. */
+			spinlock_t lock;
+			/** @mem.defrag.list: List of struct xe_bo defrag_link. */
+			struct list_head list;
+			/**
+			 * @mem.defrag.count: Number of BOs currently on
+			 * @mem.defrag.list.
+			 */
+			atomic_t count;
+		} defrag;
 	} mem;
 
 	/** @sriov: device level virtualization data */
Signature · 1 line
2.34.1
#29 lore raw

Re: [PATCH v2 15/33] drm/xe: Track BOs backed at a sub-optimal page order

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] BOs moved to SYSTEM memory during a failed shrinker purge are incorrectly left on the defrag…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] BOs moved to SYSTEM memory during a failed shrinker purge are incorrectly left on the defrag list.
- [Medium] Framebuffer pinning bypasses defrag list tracking updates, violating the invariant that pinned BOs are removed from the list. --

--- Patch [15]: [PATCH v2 15/33] drm/xe: Track BOs backed at a sub-optimal page order --- commit d8c62c56c566775d903f43756778e071af0acb80
Author: Matthew Brost <[email protected]>

drm/xe: Track BOs backed at a sub-optimal page order

Add per-device tracking of BOs whose backing TT pages were allocated below the pool's beneficial order, as reported by the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tt page flag. These BOs are future candidates for a defragmentation pass that reallocates their pages at the beneficial order.

Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
index cca617cf34d13..d7b0536564ab2 100644

[ ... ]

Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +2 -0
@@ -1261,6 +1358,15 @@ static long xe_bo_shrink_purge(struct ttm_operation_ctx *ctx,
 	if (lret > 0) {
 		xe_ttm_tt_account_subtract(xe, bo->ttm);
 		update_global_total_pages(bo->bdev, -(long)tt->num_pages);
+
+		/*

drivers/gpu/drm/xe/xe_bo.c static long xe_bo_shrink_purge(struct ttm_operation_ctx *ctx, · line 1372

Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +7 -0
+		 * The pages are gone, so a tracked BO no longer needs defrag.
+		 * This shrinker purge path frees pages without going through
+		 * xe_bo_move(), so drop any stale defrag list entry here. Ghost
+		 * bos are never tracked, so guard for real xe bos.
+		 */
+		if (xe_bo_is_xe_bo(bo))
+			xe_bo_defrag_remove(ttm_to_xe_bo(bo));
 	}
 
 	return lret;

[Severity: Medium]
When xe_bo_shrink_purge() attempts to purge a BO, it uses ttm_bo_move_null() to move it to XE_PL_SYSTEM, which bypasses xe_bo_defrag_update().

If the subsequent ttm_bo_shrink() call fails (lret <= 0), this explicit xe_bo_defrag_remove() is skipped.

Could this leave a tracked BO in SYSTEM memory while incorrectly remaining on the defrag list, violating the tracking invariant and wasting worker cycles?

[ ... ]

drivers/gpu/drm/xe/xe_bo.c int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec) · line 3190

Quoted 9 lines
drivers/gpu/drm/xe/xe_bo.c +2 -0
@@ -3072,6 +3183,8 @@ int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec)
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_subtract(xe, bo->ttm.ttm);
 
+	xe_bo_defrag_remove(bo);
+
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
 	 * we do not need this.

[Severity: Medium]
Will this miss defrag list tracking updates for framebuffer BOs?

xe_fb_pin() and __xe_unpin_fb_vma() in drivers/gpu/drm/xe/display/xe_fb_pin.c call ttm_bo_pin() and ttm_bo_unpin() directly rather than using this xe_bo_pin() wrapper.

If a framebuffer BO resides in XE_PL_TT with sub-optimal page allocations, could it remain on the defrag list even when pinned, causing the defrag worker to incorrectly process pinned BOs?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
author maintainer
#30 lore raw

PATCH v2 16/33 drm/xe: Back off beneficial-order reclaim under defrag pressure

In reply to Matthew Brost

Once a meaningful number of BOs are already tracked as being backed at a sub-optimal order, continuing to push hard for beneficial-order pages during populate mostly wastes time in reclaim/compaction.

In xe_ttm_tt_populate(), set ctx->beneficial_reclaim_backoff when the device defrag count reaches XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD (2) so the TTM pool backs off and allocations make forward progress. The deferred defrag pass is responsible for upgrading these objects to beneficial order later.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_bo.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +14 -0
index d7b0536564ab..21d087170ef3 100644
@@ -41,6 +41,14 @@
 #include "xe_vm.h"
 #include "xe_vram_types.h"
 
+/*
+ * Once this many BOs are tracked on the device defrag list (i.e. were backed
+ * with a sub-optimal page order), request that the TTM pool backs off from
+ * aggressive reclaim at the beneficial order during populate, so that
+ * allocations make forward progress instead of stalling.
+ */
+#define XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD	2
+
 const char *const xe_mem_type_to_name[TTM_NUM_MEM_TYPES]  = {
 	[XE_PL_SYSTEM] = "system",
 	[XE_PL_TT] = "gtt",
@@ -598,6 +606,12 @@ static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt,
 	if (ttm_tt_is_backed_up(tt) && !xe_tt->purgeable) {
 		err = ttm_tt_restore(ttm_dev, tt, ctx);
 	} else {
+		struct xe_device *xe = ttm_to_xe_device(ttm_dev);
+
+		if (atomic_read(&xe->mem.defrag.count) >=
+		    XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD)
+			ctx->beneficial_reclaim_backoff = true;
+
 		ttm_tt_clear_backed_up(tt);
 		err = ttm_pool_alloc(&ttm_dev->pool, tt, ctx);
 	}
Signature · 1 line
2.34.1
author maintainer
#31 lore raw

PATCH v2 17/33 drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies

In reply to Matthew Brost

Add xe_migrate_copy_defrag(), a copy helper for the upcoming page defragmentation path where a single BO has its backing reallocated at a more beneficial page order and its contents must be relocated from the old pages to the new ones entirely on the GPU.

The contents are relocated in up to two passes:

- Data pass (always): copy the data pages verbatim. The pages are mapped without the compression PAT, so the (possibly compressed) bytes are copied as-is rather than being decompressed and recompressed, and no CCS aux copy is appended.

- CCS pass (need_ccs): copy the CCS aux (compression) state. Both the old and new data pages are mapped with the compression PAT so the copy engine accesses the aux state indirectly, issuing one indirect -> indirect XY_CTRL_SURF_COPY_BLT; the data blit is skipped.

The two passes are issued separately because they require opposite compression PAT settings on the data page mappings (cleared for the verbatim data copy, set for the indirect CCS access), which a single mapping cannot satisfy. Both passes run in order on the same migrate queue and are captured by the returned fence.

The source pages are passed explicitly via an sg_table argument (the old, still populated backing) since they no longer match the BO's current backing. The boolean arguments to __xe_migrate_copy() are folded into struct xe_migrate_copy_flags to keep the growing set of copy variants readable.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_migrate.c | 136 ++++++++++++++++++++++++++++----
 drivers/gpu/drm/xe/xe_migrate.h |   8 ++
 2 files changed, 128 insertions(+), 16 deletions(-)
drivers/gpu/drm/xe/xe_migrate.c +120 -16
index 92d5e81ceac2..305c6e1942fb 100644
@@ -858,13 +858,39 @@ static u32 xe_migrate_ccs_copy(struct xe_migrate *m,
 	return flush_flags;
 }
 
+/**
+ * struct xe_migrate_copy_flags - Flags for __xe_migrate_copy()
+ * @copy_only_ccs: Only copy the CCS aux state, aliasing the destination data
+ * mapping to the source (used when the data pages are shared between src/dst).
+ * @is_vram_resolve: Decompression (VRAM -> system) resolve; always uses the
+ * compression PAT index for the data copy.
+ * @defrag_ccs: Defrag move pass that migrates only the CCS aux state between
+ * two distinct page sets (old @src_sg pages -> new pages). The data blit is
+ * skipped (the data was copied by the @defrag_copy pass) and both the source
+ * and destination data pages are mapped with the compression PAT so the aux
+ * state is copied indirect -> indirect.
+ * @defrag_copy: Defrag move pass that copies the data pages between two
+ * distinct page sets (old @src_sg pages -> new pages) without touching the CCS
+ * aux state. The data pages are mapped without the compression PAT so the
+ * (possibly compressed) bytes are copied verbatim rather than being
+ * decompressed and recompressed, and no CCS aux copy is appended; the aux
+ * state is migrated separately by the @defrag_ccs pass. The two passes need
+ * opposite compression PAT settings, which is why they cannot be combined.
+ */
+struct xe_migrate_copy_flags {
+	u32 copy_only_ccs : 1;
+	u32 is_vram_resolve : 1;
+	u32 defrag_ccs : 1;
+	u32 defrag_copy : 1;
+};
+
 static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 					   struct xe_bo *src_bo,
 					   struct xe_bo *dst_bo,
 					   struct ttm_resource *src,
 					   struct ttm_resource *dst,
-					   bool copy_only_ccs,
-					   bool is_vram_resolve)
+					   struct sg_table *src_sg,
+					   struct xe_migrate_copy_flags flags)
 {
 	struct xe_gt *gt = m->tile->primary_gt;
 	struct xe_device *xe = gt_to_xe(gt);
@@ -881,20 +907,29 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 	bool src_is_vram = mem_type_is_vram(src->mem_type);
 	bool dst_is_vram = mem_type_is_vram(dst->mem_type);
 	bool type_device = src_bo->ttm.type == ttm_bo_type_device;
-	bool needs_ccs_emit = type_device && xe_migrate_needs_ccs_emit(xe);
+	bool needs_ccs_emit = type_device && xe_migrate_needs_ccs_emit(xe) &&
+		!flags.defrag_copy;
 	bool copy_ccs = xe_device_has_flat_ccs(xe) &&
-		xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo);
-	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram);
+		xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo) &&
+		!flags.defrag_copy;
+	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram) &&
+		!flags.defrag_ccs && !flags.defrag_copy;
 
 	/*
 	 * For decompression operation, always use the compression PAT index.
 	 * Otherwise, only use the compression PAT index for device memory
 	 * when copying from VRAM to system memory.
 	 */
-	bool use_comp_pat = is_vram_resolve || (type_device &&
+	bool use_comp_pat = flags.is_vram_resolve || (type_device &&
 			    xe_device_has_flat_ccs(xe) &&
 			    GRAPHICS_VER(xe) >= 20 && src_is_vram && !dst_is_vram);
 
+	/*
+	 * The defrag CCS pass migrates the aux state from the old pages
+	 * (@src_sg) to the new pages with both sides accessed indirectly, so
+	 * keep both src_is_pltt and dst_is_pltt set (no direct staging side).
+	 */
+
 	/* Copying CCS between two different BOs is not supported yet. */
 	if (XE_WARN_ON(copy_ccs && src_bo != dst_bo))
 		return ERR_PTR(-EINVAL);
@@ -903,7 +938,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		return ERR_PTR(-EINVAL);
 
 	if (!src_is_vram)
-		xe_res_first_sg(xe_bo_sg(src_bo), 0, size, &src_it);
+		xe_res_first_sg(src_sg ? src_sg : xe_bo_sg(src_bo), 0, size, &src_it);
 	else
 		xe_res_first(src, 0, size, &src_it);
 	if (!dst_is_vram)
@@ -912,7 +947,8 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		xe_res_first(dst, 0, size, &dst_it);
 
 	if (copy_system_ccs)
-		xe_res_first_sg(xe_bo_sg(src_bo), xe_bo_ccs_pages_start(src_bo),
+		xe_res_first_sg(src_sg ? src_sg : xe_bo_sg(src_bo),
+				xe_bo_ccs_pages_start(src_bo),
 				PAGE_ALIGN(xe_device_ccs_bytes(xe, size)),
 				&ccs_it);
 
@@ -942,7 +978,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		batch_size += pte_update_size(m, pte_flags, src, &src_it, &src_L0,
 					      &src_L0_ofs, &src_L0_pt, 0, 0,
 					      avail_pts);
-		if (copy_only_ccs) {
+		if (flags.copy_only_ccs) {
 			dst_L0_ofs = src_L0_ofs;
 		} else {
 			pte_flags = dst_is_vram ? PTE_UPDATE_FLAG_IS_VRAM : 0;
@@ -963,7 +999,8 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		}
 
 		/* Add copy commands size here */
-		batch_size += ((copy_only_ccs) ? 0 : emit_copy_cmd_len(xe)) +
+		batch_size += ((flags.copy_only_ccs || flags.defrag_ccs) ? 0 :
+			       emit_copy_cmd_len(xe)) +
 			((needs_ccs_emit ? EMIT_COPY_CCS_DW : 0));
 
 		bb = xe_bb_new(gt, batch_size, usm);
@@ -975,13 +1012,15 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		if (src_is_vram && xe_migrate_allow_identity(src_L0, &src_it))
 			xe_res_next(&src_it, src_L0);
 		else
-			emit_pte(m, bb, src_L0_pt, src_is_vram, copy_system_ccs || use_comp_pat,
+			emit_pte(m, bb, src_L0_pt, src_is_vram,
+				 copy_system_ccs || use_comp_pat || flags.defrag_ccs,
 				 &src_it, src_L0, src);
 
 		if (dst_is_vram && xe_migrate_allow_identity(src_L0, &dst_it))
 			xe_res_next(&dst_it, src_L0);
-		else if (!copy_only_ccs)
-			emit_pte(m, bb, dst_L0_pt, dst_is_vram, copy_system_ccs,
+		else if (!flags.copy_only_ccs)
+			emit_pte(m, bb, dst_L0_pt, dst_is_vram,
+				 copy_system_ccs || flags.defrag_ccs,
 				 &dst_it, src_L0, dst);
 
 		if (copy_system_ccs)
@@ -990,7 +1029,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		bb->cs[bb->len++] = MI_BATCH_BUFFER_END;
 		update_idx = bb->len;
 
-		if (!copy_only_ccs)
+		if (!flags.copy_only_ccs && !flags.defrag_ccs)
 			emit_copy(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0, XE_PAGE_SIZE);
 
 		if (needs_ccs_emit)
@@ -1078,7 +1117,69 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
 				  struct ttm_resource *dst,
 				  bool copy_only_ccs)
 {
-	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, copy_only_ccs, false);
+	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, NULL,
+				 (struct xe_migrate_copy_flags) {
+					 .copy_only_ccs = copy_only_ccs,
+				 });
+}
+
+/**
+ * xe_migrate_copy_defrag() - Migrate a BO's backing to a reallocated page set.
+ * @m: The migration context.
+ * @bo: The buffer object being defragmented. Both @src and @dst belong to @bo.
+ * @src: The source TTM resource (the old backing).
+ * @dst: The dst TTM resource (the freshly reallocated backing).
+ * @src_sg: Scatter-gather table of the source (old) backing pages.
+ * @need_ccs: Whether the BO has CCS aux state that must be migrated too.
+ *
+ * Used by the defrag path where a single BO has its backing reallocated at a
+ * more beneficial page order. The contents are relocated from the old pages
+ * (@src_sg) to the new pages (the BO's current backing) entirely on the GPU,
+ * in up to two passes:
+ *
+ *  - Data pass (always): copy the data pages verbatim. The pages are mapped
+ *    without the compression PAT, so the (possibly compressed) bytes are copied
+ *    as-is rather than being decompressed and recompressed, and no CCS aux copy
+ *    is appended.
+ *
+ *  - CCS pass (@need_ccs): copy the CCS aux (compression) state. Both the old
+ *    and new data pages are mapped with the compression PAT so the copy engine
+ *    accesses the aux state indirectly, issuing one indirect -> indirect
+ *    XY_CTRL_SURF_COPY_BLT; the data blit is skipped.
+ *
+ * The two passes are issued separately because they require opposite
+ * compression PAT settings on the data page mappings (cleared for the verbatim
+ * data copy, set for the indirect CCS access), which a single mapping cannot
+ * satisfy. Both passes run in order on the same migrate queue.
+ *
+ * Return: Pointer to a dma_fence representing the last copy batch, or an error
+ * pointer on failure.
+ */
+struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
+					 struct xe_bo *bo,
+					 struct ttm_resource *src,
+					 struct ttm_resource *dst,
+					 struct sg_table *src_sg,
+					 bool need_ccs)
+{
+	struct dma_fence *fence2, *fence =
+		__xe_migrate_copy(m, bo, bo, src, dst, src_sg,
+				  (struct xe_migrate_copy_flags) {
+					 .defrag_copy = true,
+					 });
+
+	if (IS_ERR(fence) || !need_ccs)
+		return fence;
+
+	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_sg,
+				   (struct xe_migrate_copy_flags) {
+				   .defrag_ccs = true,
+				   });
+	if (IS_ERR(fence2))
+		dma_fence_wait(fence, false);
+	dma_fence_put(fence);
+
+	return fence2;
 }
 
 /**
@@ -1096,7 +1197,10 @@ struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
 				     struct xe_bo *bo,
 				     struct ttm_resource *res)
 {
-	return __xe_migrate_copy(m, bo, bo, res, res, false, true);
+	return __xe_migrate_copy(m, bo, bo, res, res, NULL,
+				 (struct xe_migrate_copy_flags) {
+					 .is_vram_resolve = true,
+				 });
 }
 
 /**
drivers/gpu/drm/xe/xe_migrate.h +8 -0
index 965c45889c72..e4f996942a6d 100644
@@ -11,6 +11,7 @@
 struct dma_fence;
 struct drm_pagemap_addr;
 struct iosys_map;
+struct sg_table;
 struct ttm_resource;
 
 struct xe_bo;
@@ -132,6 +133,13 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
 				  struct ttm_resource *dst,
 				  bool copy_only_ccs);
 
+struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
+					 struct xe_bo *bo,
+					 struct ttm_resource *src,
+					 struct ttm_resource *dst,
+					 struct sg_table *src_sg,
+					 bool need_ccs);
+
 struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
 				     struct xe_bo *bo,
 				     struct ttm_resource *res);
Signature · 1 line
2.34.1
#32 lore raw

Re: [PATCH v2 17/33] drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Critical] Uninitialized variable `ccs_ofs` and kernel assertion failure in `xe_migrate_ccs_copy()`…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] Uninitialized variable ccs_ofs and kernel assertion failure in xe_migrate_ccs_copy() during defrag_ccs pass for TT memory on DGFX.
- [High] Defragmentation bypasses mandatory CCS metadata clearing for BOs without CCS, causing a security/information leak. --

--- Patch [17]: [PATCH v2 17/33] drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies --- commit b14779b7d60fb7520bb68944732a8ae6ceb089e7
Author: Matthew Brost <[email protected]>

drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies

Add xe_migrate_copy_defrag(), a copy helper for the upcoming page defragmentation path where a single BO has its backing reallocated at a more beneficial page order. The contents are relocated in up to two passes (data and CCS) because they require opposite compression PAT settings.

Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c +0 -0
index 9428dd5e7760c..956fd75454fa0 100644

[ ... ]

Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c +0 -0
@@ -905,20 +931,29 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 	bool src_is_vram = mem_type_is_vram(src->mem_type);
 	bool dst_is_vram = mem_type_is_vram(dst->mem_type);
 	bool type_device = src_bo->ttm.type == ttm_bo_type_device;

drivers/gpu/drm/xe/xe_migrate.c static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m, · line 940

Quoted 10 lines
drivers/gpu/drm/xe/xe_migrate.c +7 -3
-	bool needs_ccs_emit = type_device && xe_migrate_needs_ccs_emit(xe);
+	bool needs_ccs_emit = type_device && xe_migrate_needs_ccs_emit(xe) &&
+		!flags.defrag_copy;
+	bool copy_ccs = xe_device_has_flat_ccs(xe) &&
-		xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo);
-	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram);
+		xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo) &&
+		!flags.defrag_copy;
+	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram) &&
+		!flags.defrag_ccs && !flags.defrag_copy;

[Severity: High]
If a BO does not use CCS, does forcing needs_ccs_emit to false during the data pass bypass the mandatory security clearing of its CCS metadata?

In xe_migrate_copy_defrag(), if need_ccs is false, it performs the data pass and returns early:

    struct dma_fence *fence =
        __xe_migrate_copy(..., (struct xe_migrate_copy_flags) {
            .defrag_copy = true,
        });

    if (IS_ERR(fence) || !need_ccs)
        return fence;

Since needs_ccs_emit is false when flags.defrag_copy is true, xe_migrate_ccs_copy() is never called to clear the CCS metadata. Could this leave stale cross-process data in the newly allocated destination pages?

[ ... ]

drivers/gpu/drm/xe/xe_migrate.c static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m, · line 1059

Quoted 9 lines
drivers/gpu/drm/xe/xe_migrate.c +1 -1
@@ -1014,7 +1053,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		bb->cs[bb->len++] = MI_BATCH_BUFFER_END;
 		update_idx = bb->len;
 
-		if (!copy_only_ccs)
+		if (!flags.copy_only_ccs && !flags.defrag_ccs)
 			emit_copy(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0, XE_PAGE_SIZE);
 
 		if (needs_ccs_emit)

[Severity: Critical]
Can this pass an uninitialized ccs_ofs to xe_migrate_ccs_copy() and trigger a kernel assertion failure on DGFX?

During a defrag_ccs pass for a TT memory BO, flags.defrag_ccs is true, which forces copy_system_ccs to false:

    bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram) &&
        !flags.defrag_ccs && !flags.defrag_copy;

Because copy_system_ccs is false, the loop skips the initialization of ccs_ofs:

    if (copy_system_ccs) {
        ...
        batch_size += pte_update_size(..., &ccs_ofs, ...);
    }

However, needs_ccs_emit remains true (since flags.defrag_copy is false). This causes xe_migrate_ccs_copy() to be called with the uninitialized ccs_ofs.

Furthermore, for TT memory on DGFX, both src_is_vram and dst_is_vram are false. These are passed as the indirect flags to xe_migrate_ccs_copy():

flush_flags = xe_migrate_ccs_copy(m, bb, src_L0_ofs,
IS_DGFX(xe) ? src_is_vram : src_is_pltt, dst_L0_ofs,
IS_DGFX(xe) ? dst_is_vram : dst_is_pltt, src_L0, ccs_ofs, copy_ccs);

Inside xe_migrate_ccs_copy(), this will trigger the assertion because both indirect flags are false:

    xe_gt_assert(gt, src_is_indirect || dst_is_indirect);
    if (!src_is_indirect)
        src_ofs = ccs_ofs;

And then it programs the GPU copy engine with the uninitialized ccs_ofs, which could lead to arbitrary GPU memory access or data corruption.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
author maintainer
#33 lore raw

PATCH v2 18/33 drm/xe: Handle defrag moves in xe_bo_move()

In reply to Matthew Brost

Teach xe_bo_move() to service a defrag move, where TTM reallocates a BO's backing at the device's beneficial order and hands the old, still populated tt to the move via the BO's defrag_old_tt.

A defrag move always has existing contents to copy, so force move_lacks_source false and fall through the XE_PL_TT -> XE_PL_TT no-op shortcut so the copy path runs. The relocation is performed entirely on the GPU via xe_migrate_copy_defrag(), reading the old pages from the stashed tt's sg table and writing the freshly reallocated backing.

Both copy passes run in order on the same migrate queue and are captured by the returned fence, so the teardown is pipelined via ttm_bo_move_accel_cleanup(): the old tt is handed to a ghost object and only unpopulated and freed once the fence signals, so the move need not block. The VF CCS attach/detach handling is extended to treat a defrag move (ctx->defrag) like the XE_PL_SYSTEM <-> XE_PL_TT transitions it mirrors.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_bo.c | 24 ++++++++++++++++++++----
 1 file changed, 20 insertions(+), 4 deletions(-)
drivers/gpu/drm/xe/xe_bo.c +20 -4
index 21d087170ef3..8e9fc8b3dffa 100644
@@ -1164,6 +1164,13 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	move_lacks_source = !old_mem || (handle_system_ccs ? (!bo->ccs_cleared) :
 					 (!mem_type_is_vram(old_mem_type) && !tt_has_data));
 
+	/*
+	 * A defrag move always copies the existing contents from the stashed
+	 * old tt into the freshly (re)allocated backing.
+	 */
+	if (ttm_bo->defrag_old_tt)
+		move_lacks_source = false;
+
 	needs_clear = (ttm && ttm->page_flags & TTM_TT_FLAG_ZERO_ALLOC) ||
 		(!ttm && ttm_bo->type == ttm_bo_type_device);
 
@@ -1200,9 +1207,12 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	/*
 	 * Failed multi-hop where the old_mem is still marked as
 	 * TTM_PL_FLAG_TEMPORARY, should just be a dummy move.
+	 *
+	 * For a defrag move the old and new tt differ, so fallthrough to the
+	 * copy path instead of treating it as a no-op.
 	 */
 	if (old_mem_type == XE_PL_TT &&
-	    new_mem->mem_type == XE_PL_TT) {
+	    new_mem->mem_type == XE_PL_TT && !ttm_bo->defrag_old_tt) {
 		ttm_bo_move_null(ttm_bo, new_mem);
 		goto out;
 	}
@@ -1273,6 +1283,12 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 			flags |= XE_MIGRATE_CLEAR_FLAG_CCS_DATA;
 
 		fence = xe_migrate_clear(migrate, bo, new_mem, flags);
+	} else if (ttm_bo->defrag_old_tt) {
+		struct xe_ttm_tt *old_xe_tt =
+			container_of(ttm_bo->defrag_old_tt, struct xe_ttm_tt, ttm);
+
+		fence = xe_migrate_copy_defrag(migrate, bo, old_mem, new_mem,
+					       old_xe_tt->sg, handle_system_ccs);
 	} else {
 		fence = xe_migrate_copy(migrate, bo, bo, old_mem, new_mem,
 					handle_system_ccs);
@@ -1309,13 +1325,13 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	 * BBs from BO as it is no longer needed.
 	 */
 	if (IS_VF_CCS_READY(xe) && old_mem_type == XE_PL_TT &&
-	    new_mem->mem_type == XE_PL_SYSTEM)
+	    (new_mem->mem_type == XE_PL_SYSTEM || ctx->defrag))
 		xe_sriov_vf_ccs_detach_bo(bo);
 
 	if (IS_VF_CCS_READY(xe) &&
 	    ((move_lacks_source && new_mem->mem_type == XE_PL_TT) ||
-	     (old_mem_type == XE_PL_SYSTEM && new_mem->mem_type == XE_PL_TT)) &&
-	    handle_system_ccs)
+	     ((old_mem_type == XE_PL_SYSTEM || ctx->defrag) &&
+	      new_mem->mem_type == XE_PL_TT)) && handle_system_ccs)
 		ret = xe_sriov_vf_ccs_attach_bo(bo);
 
 out:
Signature · 1 line
2.34.1
#34 lore raw

Re: [PATCH v2 18/33] drm/xe: Handle defrag moves in xe_bo_move()

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Pre-existing issues:
- [High] Returning an error from xe_bo_move() after the move has been successfully committed causes a Use-After-Free of the new resource in TTM core.
- [High] Live migration during an asynchronous move causes silent loss of VF CCS metadata.
--

--- Patch [18]: [PATCH v2 18/33] drm/xe: Handle defrag moves in xe_bo_move() --- commit dad45636f401d271e91aa77826c4290b4048e3c7
Author: Matthew Brost <[email protected]>

drm/xe: Handle defrag moves in xe_bo_move()

This commit adds support for defrag moves in xe_bo_move(). It allows TTM to reallocate a buffer object's backing at a beneficial order and handles the relocation entirely on the GPU via xe_migrate_copy_defrag().

Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
index 21d087170ef36..8e9fc8b3dffaf 100644

[ ... ]

drivers/gpu/drm/xe/xe_bo.c static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, · line 1329

Quoted 7 lines
drivers/gpu/drm/xe/xe_bo.c +1 -1
@@ -1309,13 +1325,13 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	 * BBs from BO as it is no longer needed.
 	 */
 	if (IS_VF_CCS_READY(xe) && old_mem_type == XE_PL_TT &&
-	    new_mem->mem_type == XE_PL_SYSTEM)
+	    (new_mem->mem_type == XE_PL_SYSTEM || ctx->defrag))
 		xe_sriov_vf_ccs_detach_bo(bo);

[Severity: High]
This isn't a bug introduced by this patch, but if a live migration is triggered concurrently with a BO defragmentation or eviction, does this synchronous CPU detachment cause a loss of CCS metadata?

Because xe_sriov_vf_ccs_detach_bo() and xe_sriov_vf_ccs_attach_bo() are called synchronously on the CPU before the GPU has finished copying the data, they immediately point the host to the new memory for CCS save and restore operations.

If a live migration occurs before the GPU copy finishes, the host might save uninitialized CCS data from the new memory, and fail to save the valid CCS data from the old memory. Upon VM restore, the GPU migrate job resumes and copies the corrupted old CCS to the new memory.

drivers/gpu/drm/xe/xe_bo.c static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, · line 1335

Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.c +2 -2
 	if (IS_VF_CCS_READY(xe) &&
 	    ((move_lacks_source && new_mem->mem_type == XE_PL_TT) ||
-	     (old_mem_type == XE_PL_SYSTEM && new_mem->mem_type == XE_PL_TT)) &&
-	    handle_system_ccs)
+	     ((old_mem_type == XE_PL_SYSTEM || ctx->defrag) &&
+	      new_mem->mem_type == XE_PL_TT)) && handle_system_ccs)
 		ret = xe_sriov_vf_ccs_attach_bo(bo);

[Severity: High]
This is a pre-existing issue, but can returning an error code here trigger a use-after-free in the TTM core?

If xe_bo_move() commits the memory move earlier in the function by calling ttm_bo_move_accel_cleanup() or ttm_bo_move_null(), new_mem is assigned to bo->resource and the old memory is handed to a ghost object.

If xe_sriov_vf_ccs_attach_bo(bo) fails with an error like -ENOMEM, or if any subsequent timeout wait fails, the error code is returned to the TTM core.

The TTM core assumes a failed move means the new resource was not adopted:

ttm_bo_validate()
ttm_bo_handle_move_mem()

    ret = ttm_bo_move()
    if (ret) {
        ...
        ttm_resource_free(bo, &res);
    }

Since bo->resource already points to the new resource, would this leave bo->resource pointing to freed memory?

Quoted 2 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
 out:
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=18
author maintainer
#35 lore raw

PATCH v2 19/33 drm/xe: Skip self-copies for borrowed pages on defrag moves

In reply to Matthew Brost

A defrag move may harvest already beneficial-order chunks from the old tt, so the freshly reallocated backing borrows (references) some of the same physical pages. For those pages the source and destination GPU mappings alias the same memory, making the data blit a pointless self-copy.

Pass the source (old) tt into __xe_migrate_copy() instead of just its sg table, so the data pass can compare the src and dst tt page arrays at the matching offset and only blit runs of pages that actually differ. A new emit_copy_chunks() helper walks each L0 chunk, skips borrowed-page runs, and coalesces contiguous to-copy pages into a single emit_copy(). It runs twice per chunk: once with a NULL batch buffer to count the blits for batch sizing, then again to emit them.

When an entire L0 chunk turns out to be borrowed, skip it outright: no batch buffer is allocated and no GPU commands are issued, the cursors are just advanced past the chunk. If every chunk of the move is borrowed the copy is a complete no-op, so hand back a signaled stub fence.

The comparison is a cheap per-page pointer test (src page != dst page) that coalesces contiguous to-copy pages into a single blit. PTEs are still emitted for partially borrowed chunks, since that setup is cheap and only the data copies are worth skipping. Only the defrag data pass is affected; the separate CCS pass still migrates all aux state.

Since the Xe tt sg lives in the xe_ttm_tt subclass rather than the base ttm_tt, add an xe_tt_sg() helper to fetch it from a struct ttm_tt.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_bo.c      |  22 +++++-
 drivers/gpu/drm/xe/xe_bo.h      |   2 +
 drivers/gpu/drm/xe/xe_migrate.c | 136 ++++++++++++++++++++++++++++----
 drivers/gpu/drm/xe/xe_migrate.h |   3 +-
 4 files changed, 142 insertions(+), 21 deletions(-)
drivers/gpu/drm/xe/xe_bo.c +18 -4
index 8e9fc8b3dffa..3b867c486ea4 100644
@@ -473,6 +473,22 @@ struct sg_table *xe_bo_sg(struct xe_bo *bo)
 	return xe_tt->sg;
 }
 
+/**
+ * xe_tt_sg() - Return the DMA scatter-gather table backing a ttm_tt.
+ * @tt: The ttm_tt to query.
+ *
+ * Like xe_bo_sg(), but takes the ttm_tt directly. Used by the defrag copy
+ * path which needs the old (source) tt's sg table without a backing xe_bo.
+ *
+ * Return: The sg table, or NULL if the tt is not currently mapped.
+ */
+struct sg_table *xe_tt_sg(struct ttm_tt *tt)
+{
+	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
+
+	return xe_tt->sg;
+}
+
 /*
  * Account ttm pages against the device shrinker's shrinkable and
  * purgeable counts.
@@ -1284,11 +1300,9 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 
 		fence = xe_migrate_clear(migrate, bo, new_mem, flags);
 	} else if (ttm_bo->defrag_old_tt) {
-		struct xe_ttm_tt *old_xe_tt =
-			container_of(ttm_bo->defrag_old_tt, struct xe_ttm_tt, ttm);
-
 		fence = xe_migrate_copy_defrag(migrate, bo, old_mem, new_mem,
-					       old_xe_tt->sg, handle_system_ccs);
+					       ttm_bo->defrag_old_tt,
+					       handle_system_ccs);
 	} else {
 		fence = xe_migrate_copy(migrate, bo, bo, old_mem, new_mem,
 					handle_system_ccs);
drivers/gpu/drm/xe/xe_bo.h +2 -0
index 4186cb51af87..32211016a722 100644
@@ -110,6 +110,7 @@ enum xe_madv_purgeable_state {
 };
 
 struct sg_table;
+struct ttm_tt;
 
 struct xe_bo *xe_bo_alloc(void);
 void xe_bo_free(struct xe_bo *bo);
@@ -547,6 +548,7 @@ void xe_bo_dev_init(struct xe_bo_dev *bo_device);
 void xe_bo_dev_fini(struct xe_bo_dev *bo_device);
 
 struct sg_table *xe_bo_sg(struct xe_bo *bo);
+struct sg_table *xe_tt_sg(struct ttm_tt *tt);
 
 /*
  * xe_sg_segment_size() - Provides upper limit for sg segment size.
drivers/gpu/drm/xe/xe_migrate.c +120 -16

237 lines not on this page · raw message

drivers/gpu/drm/xe/xe_migrate.h +2 -1
index e4f996942a6d..4094ea1c19c1 100644
@@ -13,6 +13,7 @@ struct drm_pagemap_addr;
 struct iosys_map;
 struct sg_table;
 struct ttm_resource;
+struct ttm_tt;
 
 struct xe_bo;
 struct xe_gt;
@@ -137,7 +138,7 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 					 struct xe_bo *bo,
 					 struct ttm_resource *src,
 					 struct ttm_resource *dst,
-					 struct sg_table *src_sg,
+					 struct ttm_tt *src_tt,
 					 bool need_ccs);
 
 struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
Signature · 1 line
2.34.1
#36 lore raw

Re: [PATCH v2 19/33] drm/xe: Skip self-copies for borrowed pages on defrag moves

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Out-of-bounds array access due to incorrect page shift in start_page calculation - [High] Hardware…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Out-of-bounds array access due to incorrect page shift in start_page calculation
- [High] Hardware assertion failure and command stream corruption due to using PAGE_SIZE as pitch --

--- Patch [19]: [PATCH v2 19/33] drm/xe: Skip self-copies for borrowed pages on defrag moves --- commit 0cb7423f91b7764bc47ff13572694c591c1f58e2
Author: Matthew Brost <[email protected]>

drm/xe: Skip self-copies for borrowed pages on defrag moves

A defrag move may harvest already beneficial-order chunks from the old tt, so the freshly reallocated backing borrows (references) some of the same physical pages.

Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c +0 -0
index 956fd75454fa0..450ca8bf4f3b6 100644

[ ... ]

Quoted 6 lines
drivers/gpu/drm/xe/xe_migrate.c +2 -0
@@ -835,6 +835,63 @@ static void emit_copy(struct xe_gt *gt, struct xe_bb *bb,
 		emit_xy_fast_copy(gt, bb, src_ofs, dst_ofs, size, pitch);
 }
 
+/*
+ * emit_copy_chunks() - Emit the data blits for one defrag L0 chunk, skipping

[ ... ]

drivers/gpu/drm/xe/xe_migrate.c static void emit_copy(struct xe_gt *gt, struct xe_bb *bb, · line 863

Quoted 10 lines
drivers/gpu/drm/xe/xe_migrate.c +10 -0
+		/* Coalesce a run of pages that actually need copying. */
+		while (i + run < npages &&
+		       src_tt->pages[start_page + i + run] !=
+		       dst_tt->pages[start_page + i + run])
+			run++;
+
+		if (bb)
+			emit_copy(gt, bb, src_L0_ofs + (u64)i * PAGE_SIZE,
+				  dst_L0_ofs + (u64)i * PAGE_SIZE,
+				  run * PAGE_SIZE, PAGE_SIZE);

[Severity: High]
Can the use of PAGE_SIZE here cause a hardware assertion failure and command stream corruption on systems with 64KB pages?

The PAGE_SIZE is passed as the pitch parameter to emit_copy(), which flows into emit_xy_fast_copy() where there is a hardware limit check:

xe_gt_assert(gt, pitch <= U16_MAX);

On 64KB page systems, PAGE_SIZE is 65536, which exceeds U16_MAX (65535). This will unconditionally trigger the driver assertion. Furthermore, because the pitch is bitwise OR'ed into the BLT command instruction, won't this oversized pitch overflow its bit field and corrupt adjacent fields in the GPU command?

Should this use XE_PAGE_SIZE (4096) instead, similar to the fallback path in normal emit_copy() usage?

[ ... ]

drivers/gpu/drm/xe/xe_migrate.c static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m, · line 1058

Quoted 8 lines
drivers/gpu/drm/xe/xe_migrate.c +2 -0
@@ -985,6 +1052,8 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		u64 ccs_ofs, ccs_size;
 		u32 ccs_pt;
 		u32 pte_flags;
+		pgoff_t start_page = (xe_bo_size(src_bo) - size) >> XE_PTE_SHIFT;
+		u32 ndata_blits = 1;
 
 		bool usm = xe->info.has_usm;

[Severity: High]
Is there a risk of an out-of-bounds array access here on systems with a page size larger than 4KB (like 64KB on ARM64 or PPC64)?

By shifting with XE_PTE_SHIFT (12), the index is computed in terms of 4KB GPU pages. However, this index is later passed to emit_copy_chunks() to index the src_tt->pages and dst_tt->pages arrays:

if (src_tt->pages[start_page + i] == dst_tt->pages[start_page + i]) {

Since the ttm_tt page arrays are sized based on the system's PAGE_SHIFT, wouldn't an index calculated with XE_PTE_SHIFT grow much faster than the array bounds on a 64KB page system, leading to a memory out-of-bounds read?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=19
author maintainer
#37 lore raw

PATCH v2 20/33 drm/xe: Add a page defragmentation worker

In reply to Matthew Brost

BOs whose backing TT pages were allocated below the device's beneficial order (tracked via the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tt page flag) are added to a per-device defrag list. Add a delayed worker that periodically walks this list and attempts to re-back each BO with beneficial-order pages, improving TLB efficiency for long-lived objects that were initially allocated under memory pressure.

The worker is kicked when the list first transitions to non-empty and reschedules itself while work remains. On a failed pass it backs off exponentially (XE_BO_DEFRAG_INTERVAL_MS up to XE_BO_DEFRAG_INTERVAL_MAX_MS) and retries the whole list later; the interval is reset to the default on the next fresh enqueue. The worker stops scheduling once the list drains.

A defrag move synchronously reallocates and re-copies a BO's backing store, which is not free. To avoid stalling concurrent active work when many BOs (or one very large BO) become eligible at once (e.g. after a burst of memory pressure, which would otherwise manifest as a large FPS drop while the worker churns through the whole list), cap the number of newly (re)allocated bytes processed per run to XE_BO_DEFRAG_SIZE_LIMIT and reschedule. Only the pages a move actually reallocates are charged - pages harvested (borrowed) from the old backing are free and do not count
- so the budget tracks real defrag work and an object larger than the budget is upgraded in budget-sized slices across runs. This spreads the defrag effort over time, keeping it in the background and yielding to userspace progress.

Defragmenting a BO forces a TTM move (ctx.defrag) that reallocates the backing at the beneficial order and relocates the contents on the GPU (see xe_bo_move()). The per-BO move is bounded by the run's remaining byte budget via ctx.defrag_bytes_remaining; the pool allocator reports how much it spent so the worker can charge it accurately. When beneficial-order pages cannot be obtained the new allocation is discharged during populate and the BO keeps its original backing, staying on the list for a later retry.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_bo.c           | 277 ++++++++++++++++++++++++++-
 drivers/gpu/drm/xe/xe_bo.h           |   1 +
 drivers/gpu/drm/xe/xe_device.c       |   9 +
 drivers/gpu/drm/xe/xe_device_types.h |  11 ++
 4 files changed, 294 insertions(+), 4 deletions(-)
drivers/gpu/drm/xe/xe_bo.c +273 -4
index 3b867c486ea4..6a847e42946c 100644
@@ -31,6 +31,7 @@
 #include "xe_pat.h"
 #include "xe_pm.h"
 #include "xe_preempt_fence.h"
+#include "xe_printk.h"
 #include "xe_pxp.h"
 #include "xe_res_cursor.h"
 #include "xe_shrinker.h"
@@ -49,6 +50,35 @@
  */
 #define XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD	2
 
+/*
+ * Maximum number of bytes of newly (re)allocated backing the defrag worker will
+ * produce in a single run before yielding and rescheduling itself.
+ *
+ * A defrag move synchronously reallocates and re-copies a BO's backing store,
+ * which is not free. If a large number of BOs (or one very large BO) become
+ * eligible at once (e.g. after a burst of memory pressure), processing them all
+ * in one worker run would hold things up for a long, unbounded stretch and can
+ * visibly starve concurrent active work (e.g. a large FPS drop). Instead, cap
+ * the bytes actually upgraded per run and requeue, spreading the defrag effort
+ * out over time so it stays in the background and yields to userspace progress.
+ * Only pages a move truly reallocates are charged; pages harvested from the old
+ * backing are free, so an object larger than the budget is upgraded in
+ * budget-sized slices across runs.
+ */
+#define XE_BO_DEFRAG_SIZE_LIMIT			SZ_32M
+
+/* Default delay before (re)running the defrag worker, in milliseconds. */
+#define XE_BO_DEFRAG_INTERVAL_MS		25
+
+/*
+ * Upper bound for the (exponentially backed off) defrag worker interval, in
+ * milliseconds, so repeated failures don't push the retry arbitrarily far out.
+ */
+#define XE_BO_DEFRAG_INTERVAL_MAX_MS		15000	/* 15 seconds */
+
+static void xe_bo_defrag_worker(struct work_struct *w);
+static void xe_place_from_ttm_type(u32 mem_type, struct ttm_place *place);
+
 const char *const xe_mem_type_to_name[TTM_NUM_MEM_TYPES]  = {
 	[XE_PL_SYSTEM] = "system",
 	[XE_PL_TT] = "gtt",
@@ -624,8 +654,10 @@ static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt,
 	} else {
 		struct xe_device *xe = ttm_to_xe_device(ttm_dev);
 
-		if (atomic_read(&xe->mem.defrag.count) >=
-		    XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD)
+		if (ctx->defrag)
+			ctx->beneficial_reclaim_backoff = false;
+		else if (atomic_read(&xe->mem.defrag.count) >=
+			 XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD)
 			ctx->beneficial_reclaim_backoff = true;
 
 		ttm_tt_clear_backed_up(tt);
@@ -1041,18 +1073,54 @@ static int xe_ttm_bo_purge(struct ttm_buffer_object *ttm_bo, struct ttm_operatio
 	return 0;
 }
 
+static void xe_bo_defrag_fini(void *arg)
+{
+	struct xe_device *xe = arg;
+
+	disable_delayed_work_sync(&xe->mem.defrag.worker);
+}
+
 /**
  * xe_bo_defrag_init_early() - Initialize the device defrag BO tracking
  * @xe: The xe device
  *
- * Initialize the list, lock and count used to track BOs whose backing TT
- * pages were allocated at a sub-optimal order.
+ * Initialize the list, lock, count and delayed worker used to track and
+ * defragment BOs whose backing TT pages were allocated at a sub-optimal order.
+ *
+ * The devm action that cancels the worker on teardown is registered separately
+ * by xe_bo_defrag_init(), which must run after the migrate contexts the
+ * worker depends on have been initialized.
  */
 void xe_bo_defrag_init_early(struct xe_device *xe)
 {
 	spin_lock_init(&xe->mem.defrag.lock);
 	INIT_LIST_HEAD(&xe->mem.defrag.list);
 	atomic_set(&xe->mem.defrag.count, 0);
+	xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
+	INIT_DELAYED_WORK(&xe->mem.defrag.worker, xe_bo_defrag_worker);
+}
+
+/**
+ * xe_bo_defrag_init() - Register defrag worker teardown
+ * @xe: The xe device
+ *
+ * Register the devm action that disables and drains the defrag worker on
+ * device teardown. This must be called after xe_migrate_init() so that, with
+ * devm's reverse-order cleanup, the worker is stopped before the migrate
+ * contexts it relies on (via xe_bo_defrag_one() -> ttm_bo_validate() ->
+ * xe_bo_move() -> xe_migrate_copy_defrag()) are torn down.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int xe_bo_defrag_init(struct xe_device *xe)
+{
+	return devm_add_action_or_reset(xe->drm.dev, xe_bo_defrag_fini, xe);
+}
+
+static void xe_bo_defrag_schedule(struct xe_device *xe)
+{
+	schedule_delayed_work(&xe->mem.defrag.worker,
+			      msecs_to_jiffies(xe->mem.defrag.interval_ms));
 }
 
 static bool xe_bo_needs_defrag(struct xe_bo *bo)
@@ -1073,16 +1141,24 @@ static bool xe_bo_needs_defrag(struct xe_bo *bo)
 static void xe_bo_defrag_add(struct xe_bo *bo)
 {
 	struct xe_device *xe = xe_bo_device(bo);
+	bool kick = false;
 
 	xe_bo_assert_held(bo);
 	xe_assert(xe, xe_bo_needs_defrag(bo));
 
 	scoped_guard(spinlock, &xe->mem.defrag.lock) {
 		if (list_empty(&bo->defrag_link)) {
+			/* Kick the worker when the list transitions to non-empty. */
+			kick = list_empty(&xe->mem.defrag.list);
+			if (kick)
+				xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
 			list_add_tail(&bo->defrag_link, &xe->mem.defrag.list);
 			atomic_inc(&xe->mem.defrag.count);
 		}
 	}
+
+	if (kick)
+		xe_bo_defrag_schedule(xe);
 }
 
 static void __xe_bo_defrag_remove(struct xe_bo *bo)
@@ -1125,6 +1201,199 @@ static void xe_bo_defrag_update(struct xe_bo *bo)
 		xe_bo_defrag_remove(bo);
 }
 
+/*
+ * Attempt to defragment a single BO by forcing a move that reallocates its
+ * backing at the device's beneficial order. @budget bounds how many bytes of
+ * new pages the move may (re)allocate; the remainder of the object is completed
+ * by harvesting the old backing, and the BO stays flagged for a later pass. On
+ * return *@consumed holds the bytes actually (re)allocated (0 on failure or
+ * when nothing needed reallocating). Returns 0 if the BO no longer needs to be
+ * tracked (either defragmented or no longer eligible), or a negative error code
+ * if the attempt should be retried later.
+ */
+static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
+			    u64 budget, u64 *consumed)
+{
+	struct ttm_operation_ctx ctx = {
+		.gfp_retry_mayfail = true,
+		.defrag = true,
+		.defrag_bytes_remaining = budget,
+	};
+	struct ttm_buffer_object *ttm_bo = &bo->ttm;
+	struct ttm_placement placement;
+	struct ttm_place place;
+	int ret = 0;
+
+	*consumed = 0;
+
+	xe_bo_lock(bo, false);
+
+	/* Re-check eligibility under the BO lock. */
+	if (!xe_bo_needs_defrag(bo)) {
+		xe_bo_defrag_remove(bo);
+		goto unlock;
+	}
+
+	xe_place_from_ttm_type(ttm_bo->resource->mem_type, &place);
+	place.flags |= TTM_PL_FLAG_TEMPORARY;
+	placement.num_placement = 1;
+	placement.placement = &place;
+
+	/*
+	 * On success the move reallocates the backing at beneficial order and
+	 * drops the BO from the defrag list. On failure the BO keeps its
+	 * original backing and stays on the list for a later retry.
+	 */
+	ret = ttm_bo_validate(ttm_bo, &placement, &ctx);
+
+	/*
+	 * The pool allocator decremented defrag_bytes_remaining as it
+	 * (re)allocated pages, so the unused budget tells us how much was spent.
+	 * Only meaningful when the move succeeded. A successful move may spend
+	 * nothing - e.g. when the object was already fully upgraded and every
+	 * chunk is harvested (borrowed) from the old backing for free - so a
+	 * zero spend is legitimate and must be charged as zero, not the whole
+	 * budget.
+	 */
+	if (!ret) {
+		s64 spent = (s64)budget - ctx.defrag_bytes_remaining;
+
+		*consumed = spent > 0 ? spent : 0;
+	}
+
+	xe_dbg(xe, "Defrag attempt on BO size=%zu: ret=%pe consumed=%llu\n",
+	       xe_bo_size(bo), ERR_PTR(ret), *consumed);
+
+unlock:
+	xe_bo_unlock(bo);
+	return ret;
+}
+
+static void xe_bo_defrag_worker(struct work_struct *w)
+{
+	struct delayed_work *dwork = to_delayed_work(w);
+	struct xe_device *xe =
+		container_of(dwork, struct xe_device, mem.defrag.worker);
+	u64 defrag_bytes = 0;
+	bool requeue = false;
+	int idx;
+
+	if (!drm_dev_enter(&xe->drm, &idx))
+		return;
+
+	if (!xe_pm_runtime_get_if_in_use(xe)) {
+		drm_dev_exit(idx);
+		return;
+	}
+
+	/*
+	 * Process at most XE_BO_DEFRAG_SIZE_LIMIT bytes of newly (re)allocated
+	 * backing per run rather than draining the whole list in one go. Only
+	 * the pages a move actually reallocates are charged - pages harvested
+	 * (borrowed) from the old backing are free and do not count - so the
+	 * budget tracks real defrag work. A single object larger than the
+	 * budget is upgraded in slices across successive runs. Each defrag is
+	 * synchronous and relatively expensive, so bounding the work per run
+	 * keeps the worker from monopolising resources and lets concurrent
+	 * active work make progress; any remaining work is handled by a
+	 * follow-up run scheduled below.
+	 */
+	while (defrag_bytes < XE_BO_DEFRAG_SIZE_LIMIT) {
+		struct xe_bo *first, *bo;
+		u64 consumed = 0;
+		int ret;
+
+		scoped_guard(spinlock, &xe->mem.defrag.lock) {
+			bool again;
+
+			/* Progress, reset interval */
+			if (defrag_bytes)
+				xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
+
+			do {
+				first = list_first_entry_or_null(&xe->mem.defrag.list,
+								 struct xe_bo,
+								 defrag_link);
+				bo = first ? xe_bo_get_unless_zero(first) : NULL;
+				again = first && !bo;
+				if (again) {
+					list_del_init(&first->defrag_link);
+					atomic_dec(&xe->mem.defrag.count);
+				}
+			} while (again);
+		}
+
+		if (!bo)
+			break;
+
+		/*
+		 * Bound the per-BO move to the run's remaining byte budget. A
+		 * large object is upgraded in budget-sized slices across runs;
+		 * the slices already upgraded come back as free already-
+		 * beneficial harvests, so each run only spends budget on new
+		 * forward progress.
+		 */
+		ret = xe_bo_defrag_one(xe, bo, XE_BO_DEFRAG_SIZE_LIMIT - defrag_bytes,
+				       &consumed);
+		defrag_bytes += consumed;
+
+		if (ret || ttm_tt_is_beneficial_order_failed(bo->ttm.ttm)) {
+			scoped_guard(spinlock, &xe->mem.defrag.lock) {
+				if (ret)
+					/*
+					 * Abort the pass and retry the whole
+					 * list later, backing off exponentially
+					 * on every failure.
+					 */
+					xe->mem.defrag.interval_ms =
+						min(xe->mem.defrag.interval_ms * 2,
+						    (unsigned int)XE_BO_DEFRAG_INTERVAL_MAX_MS);
+				else
+					/* Progress, reset interval */
+					xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
+			}
+
+			xe_bo_put(bo);
+			requeue = true;
+			break;
+		}
+
+		xe_bo_put(bo);
+	}
+
+	/*
+	 * Decide whether to reschedule:
+	 *
+	 *  - The loop hit its per-run limit (defrag_bytes >=
+	 *    XE_BO_DEFRAG_SIZE_LIMIT): we stopped early to spread the work out,
+	 *    so requeue at the default interval (reset here since this run made
+	 *    progress) whenever the list still has entries.
+	 *
+	 *  - The loop aborted on a failed defrag (requeue): retry later at the
+	 *    backed-off interval computed above.
+	 *
+	 *  - The loop drained the list (broke on the empty list): nothing left to
+	 *    do, so stop scheduling. A fresh enqueue will kick the worker again.
+	 */
+	if (defrag_bytes >= XE_BO_DEFRAG_SIZE_LIMIT) {
+		struct xe_bo *bo;
+
+		scoped_guard(spinlock, &xe->mem.defrag.lock) {
+			bo = list_first_entry_or_null(&xe->mem.defrag.list,
+						      struct xe_bo, defrag_link);
+			xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
+		}
+
+		if (bo)
+			xe_bo_defrag_schedule(xe);
+	} else if (requeue) {
+		xe_bo_defrag_schedule(xe);
+	}
+
+	xe_pm_runtime_put(xe);
+	drm_dev_exit(idx);
+}
+
 static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		      struct ttm_operation_ctx *ctx,
 		      struct ttm_resource *new_mem,
drivers/gpu/drm/xe/xe_bo.h +1 -0
index 32211016a722..38d50f8122cb 100644
@@ -226,6 +226,7 @@ int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec);
 void xe_bo_unpin_external(struct xe_bo *bo);
 void xe_bo_unpin(struct xe_bo *bo);
 void xe_bo_defrag_init_early(struct xe_device *xe);
+int xe_bo_defrag_init(struct xe_device *xe);
 int xe_bo_validate(struct xe_bo *bo, struct xe_vm *vm, bool allow_res_evict,
 		   struct drm_exec *exec);
 
drivers/gpu/drm/xe/xe_device.c +9 -0
index d4321419ee35..df5bb94f3ecb 100644
@@ -1063,6 +1063,15 @@ int xe_device_probe(struct xe_device *xe)
 			return err;
 	}
 
+	/*
+	 * Register the defrag worker teardown now that the migrate contexts it
+	 * depends on are initialized, so devm's reverse-order cleanup stops the
+	 * worker before those contexts are torn down.
+	 */
+	err = xe_bo_defrag_init(xe);
+	if (err)
+		return err;
+
 	err = xe_pagefault_init(xe);
 	if (err)
 		return err;
drivers/gpu/drm/xe/xe_device_types.h +11 -0
index 9e6244d8f41b..6e68add4d652 100644
@@ -314,6 +314,17 @@ struct xe_device {
 			 * @mem.defrag.list.
 			 */
 			atomic_t count;
+			/**
+			 * @mem.defrag.worker: Delayed worker that walks
+			 * @mem.defrag.list trying to reallocate BO backing
+			 * store at the device's beneficial order.
+			 */
+			struct delayed_work worker;
+			/**
+			 * @mem.defrag.interval_ms: Reschedule interval for
+			 * @mem.defrag.worker, in milliseconds.
+			 */
+			unsigned int interval_ms;
 		} defrag;
 	} mem;
 
Signature · 1 line
2.34.1
#38 lore raw

Re: [PATCH v2 20/33] drm/xe: Add a page defragmentation worker

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider: - [Critical] Accessing `bo->ttm.ttm` outside of the BO's dma-resv lock can cause a NULL pointer dereference…

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Critical] Accessing bo->ttm.ttm outside of the BO's dma-resv lock can cause a NULL pointer dereference or data race.
- [Critical] The defrag delayed worker can be scheduled during device probe before its teardown is registered, leading to a UAF if the probe fails.
- [High] The defrag worker permanently stalls if the device enters runtime suspend.
- [High] The defrag worker fails to apply exponential backoff on sub-optimal fallback, causing an infinite spin loop.
--

--- Patch [20]: [PATCH v2 20/33] drm/xe: Add a page defragmentation worker --- commit 7f5402dce2516caf0e93f47b50a3f18b01fe034e
Author: Matthew Brost <[email protected]>

drm/xe: Add a page defragmentation worker

This patch adds a per-device defrag list and delayed worker to track and reallocate BOs whose backing TT pages were allocated below the device's beneficial order. The worker runs periodically and attempts to re-back BOs with beneficial-order pages, up to a maximum size budget per run.

Quoted 2 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0

[ ... ]

Quoted 5 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -1271,6 +1364,96 @@ static void xe_bo_defrag_worker(struct work_struct *w)
 	struct delayed_work *dwork = to_delayed_work(w);
 	struct xe_device *xe =
 		container_of(dwork, struct xe_device, mem.defrag.worker);
 	u64 defrag_bytes = 0;

drivers/gpu/drm/xe/xe_bo.c static void xe_bo_defrag_worker(struct work_struct *w)

Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
 	bool requeue = false;
 	int idx;
 
 	if (!drm_dev_enter(&xe->drm, &idx))
 		return;
 
 	if (!xe_pm_runtime_get_if_in_use(xe)) {
 		drm_dev_exit(idx);
 		return;
 	}

[Severity: High]
Does the defrag worker permanently stall if the device enters runtime suspend?

If the worker executes while the device is suspending, xe_pm_runtime_get_if_in_use() returns false and the worker exits immediately without rescheduling itself.

Because the list is not emptied, subsequent additions to the list in xe_bo_defrag_add() will not see an empty-to-non-empty transition, meaning xe_bo_defrag_schedule() will never be called again.

[ ... ]

drivers/gpu/drm/xe/xe_bo.c static void xe_bo_defrag_update(struct xe_bo *bo) · line 1340

Quoted 5 lines
drivers/gpu/drm/xe/xe_bo.c +5 -0
+		ret = xe_bo_defrag_one(xe, bo, XE_BO_DEFRAG_SIZE_LIMIT - defrag_bytes,
+				       &consumed);
+		defrag_bytes += consumed;
+
+		if (ret || ttm_tt_is_beneficial_order_failed(bo->ttm.ttm)) {

[Severity: Critical]
Can accessing bo->ttm.ttm here cause a NULL pointer dereference?

The xe_bo_defrag_one() function releases the BO's dma-resv lock before it returns. If a concurrent operation, such as evicting to VRAM or swapping out, destroys the TT while the BO is unlocked, checking ttm_tt_is_beneficial_order_failed(bo->ttm.ttm) might dereference a NULL pointer.

Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +4 -0
+			scoped_guard(spinlock, &xe->mem.defrag.lock) {
+				if (ret)
+					/*
+					 * Abort the pass and retry the whole

drivers/gpu/drm/xe/xe_bo.c static void xe_bo_defrag_update(struct xe_bo *bo) · line 1354

Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +10 -0
+					 * list later, backing off exponentially
+					 * on every failure.
+					 */
+					xe->mem.defrag.interval_ms =
+						min(xe->mem.defrag.interval_ms * 2,
+						    (unsigned int)XE_BO_DEFRAG_INTERVAL_MAX_MS);
+				else
+					/* Progress, reset interval */
+					xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
+			}

[Severity: High]
Does the backoff logic fail to apply when a fallback allocation succeeds?

If system memory is highly fragmented, the TTM pool allocator might fail to allocate beneficial-order pages but successfully fall back to a smaller order. In this case, xe_bo_defrag_one() returns 0 for success, but ttm_tt_is_beneficial_order_failed() remains true.

The backoff block checks if (ret). Since ret is 0, it takes the else branch, interprets the run as progress, and resets the interval to 25ms. This could result in a tight loop of repeated failed upgrades that wastes CPU time without ever backing off.

Quoted 2 lines
drivers/gpu/drm/xe/xe_device.c +0 -0

[ ... ]

drivers/gpu/drm/xe/xe_device.c int xe_device_probe(struct xe_device *xe) · line 1071

Quoted 10 lines
drivers/gpu/drm/xe/xe_device.c +6 -0
@@ -1063,6 +1063,15 @@ int xe_device_probe(struct xe_device *xe)
 			return err;
 	}
 
+	/*
+	 * Register the defrag worker teardown now that the migrate contexts it
+	 * depends on are initialized, so devm's reverse-order cleanup stops the
+	 * worker before those contexts are torn down.
+	 */
+	err = xe_bo_defrag_init(xe);

[Severity: Critical]
Can the delayed worker outlive the device if probe fails before the teardown is registered?

During early probe, components might allocate BOs. If these BOs fail to get beneficial order pages, they are added to the defrag list, which schedules the delayed worker via xe_bo_defrag_schedule().

If a probe step fails before reaching this late xe_bo_defrag_init() call, the devm cleanup will roll back and free the device without cancelling the worker. When the timer expires, the worker will access freed memory.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=20
author maintainer
#39 lore raw

PATCH v2 21/33 drm/xe: Add defrag GT stats

In reply to Matthew Brost

Add observability for the page defragmentation worker.

Introduce four primary-GT statistics counters: defrag_added_count (BOs queued onto the defrag list after being backed at a sub-optimal page order), defrag_success_count (BOs fully re-backed at the beneficial order), defrag_partial_success_count (BOs only partially re-backed at the beneficial order) defrag_failed_count (attempts that failed and left the BO on its original backing) and defrag_mb_moved. The ineligible/already-removed fast path is intentionally not counted as either success or failure.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_bo.c             | 17 +++++++++++++++++
 drivers/gpu/drm/xe/xe_gt_stats.c       |  5 +++++
 drivers/gpu/drm/xe/xe_gt_stats_types.h | 17 +++++++++++++++++
 3 files changed, 39 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +17 -0
index 6a847e42946c..c1bbcf0d21ed 100644
@@ -26,6 +26,7 @@
 #include "xe_dma_buf.h"
 #include "xe_drm_client.h"
 #include "xe_ggtt.h"
+#include "xe_gt_stats.h"
 #include "xe_map.h"
 #include "xe_migrate.h"
 #include "xe_pat.h"
@@ -1154,6 +1155,8 @@ static void xe_bo_defrag_add(struct xe_bo *bo)
 				xe->mem.defrag.interval_ms = XE_BO_DEFRAG_INTERVAL_MS;
 			list_add_tail(&bo->defrag_link, &xe->mem.defrag.list);
 			atomic_inc(&xe->mem.defrag.count);
+			xe_gt_stats_incr(xe_root_mmio_gt(xe),
+					 XE_GT_STATS_ID_DEFRAG_ADDED_COUNT, 1);
 		}
 	}
 
@@ -1264,6 +1267,20 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 	xe_dbg(xe, "Defrag attempt on BO size=%zu: ret=%pe consumed=%llu\n",
 	       xe_bo_size(bo), ERR_PTR(ret), *consumed);
 
+	if (!ret && ttm_tt_is_beneficial_order_failed(bo->ttm.ttm))
+		xe_gt_stats_incr(xe_root_mmio_gt(xe),
+				 XE_GT_STATS_ID_DEFRAG_PARTIAL_SUCCESS_COUNT,
+				 1);
+	else
+		xe_gt_stats_incr(xe_root_mmio_gt(xe),
+				 ret ? XE_GT_STATS_ID_DEFRAG_FAILED_COUNT :
+				       XE_GT_STATS_ID_DEFRAG_SUCCESS_COUNT, 1);
+
+	if (!ret)
+		xe_gt_stats_incr(xe_root_mmio_gt(xe),
+				 XE_GT_STATS_ID_DEFRAG_MB_MOVED,
+				 *consumed >> 20);
+
 unlock:
 	xe_bo_unlock(bo);
 	return ret;
drivers/gpu/drm/xe/xe_gt_stats.c +5 -0
index 789397514f3e..68e60d9500c8 100644
@@ -154,6 +154,11 @@ static const char *const stat_description[__XE_GT_STATS_NUM_IDS] = {
 	DEF_STAT_STR(PRL_2M_ENTRY_COUNT, "prl_2m_entry_count"),
 	DEF_STAT_STR(PRL_ISSUED_COUNT, "prl_issued_count"),
 	DEF_STAT_STR(PRL_ABORTED_COUNT, "prl_aborted_count"),
+	DEF_STAT_STR(DEFRAG_ADDED_COUNT, "defrag_added_count"),
+	DEF_STAT_STR(DEFRAG_SUCCESS_COUNT, "defrag_success_count"),
+	DEF_STAT_STR(DEFRAG_PARTIAL_SUCCESS_COUNT, "defrag_partial_success_count"),
+	DEF_STAT_STR(DEFRAG_FAILED_COUNT, "defrag_failed_count"),
+	DEF_STAT_STR(DEFRAG_MB_MOVED, "defrag_mb_moved"),
 };
 
 /**
drivers/gpu/drm/xe/xe_gt_stats_types.h +17 -0
index 425491bed6c4..c42b494390f8 100644
@@ -122,6 +122,18 @@
  * @XE_GT_STATS_ID_PRL_ABORTED_COUNT: Times the page reclaim process was
  *   aborted.
  *
+ * @XE_GT_STATS_ID_DEFRAG_ADDED_COUNT: Times a buffer object was added to the
+ *   page defragmentation list after being backed at a sub-optimal page order.
+ * @XE_GT_STATS_ID_DEFRAG_SUCCESS_COUNT: Times the defrag worker successfully
+ *   re-backed a buffer object at the beneficial page order.
+ * @XE_GT_STATS_ID_DEFRAG_PARTIAL_SUCCESS_COUNT: Times the defrag worker
+ *   partially re-backed a buffer object at the beneficial page order.
+ * @XE_GT_STATS_ID_DEFRAG_FAILED_COUNT: Times the defrag worker failed to
+ *   re-back a buffer object at the beneficial page order.
+ * @XE_GT_STATS_ID_DEFRAG_MB_MOVED: Data (MiB) actually (re)allocated while
+ *   re-backing buffer objects at the beneficial page order. Chunks borrowed
+ *   from the old backing for free are not counted.
+ *
  * @__XE_GT_STATS_NUM_IDS: Number of valid IDs; not a real counter.
  *
  * See Documentation/gpu/xe/xe_gt_stats.rst.
@@ -181,6 +193,11 @@ enum xe_gt_stats_id {
 	XE_GT_STATS_ID_PRL_2M_ENTRY_COUNT,
 	XE_GT_STATS_ID_PRL_ISSUED_COUNT,
 	XE_GT_STATS_ID_PRL_ABORTED_COUNT,
+	XE_GT_STATS_ID_DEFRAG_ADDED_COUNT,
+	XE_GT_STATS_ID_DEFRAG_SUCCESS_COUNT,
+	XE_GT_STATS_ID_DEFRAG_PARTIAL_SUCCESS_COUNT,
+	XE_GT_STATS_ID_DEFRAG_FAILED_COUNT,
+	XE_GT_STATS_ID_DEFRAG_MB_MOVED,
 	/* must be the last entry */
 	__XE_GT_STATS_NUM_IDS,
 };
Signature · 1 line
2.34.1
#40 lore raw

Re: [PATCH v2 21/33] drm/xe: Add defrag GT stats

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The statistic `XE_GT_STATS_ID_DEFRAG_MB_MOVED` systematically undercounts data moved because it…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The statistic XE_GT_STATS_ID_DEFRAG_MB_MOVED systematically undercounts data moved because it truncates the byte count to megabytes *before* accumulating it into the global counter. Any defragmentation move smaller than 1 MiB (e.g., re-backing a 64 KiB page) will evaluate to 0 and not be counted at all. Additionally, the sub-megabyte remainders of larger moves are permanently lost. --

--- Patch [21]: [PATCH v2 21/33] drm/xe: Add defrag GT stats --- commit a152b345237256025e6e622bf740f0941373de67
Author: Matthew Brost <[email protected]>

drm/xe: Add defrag GT stats

This commit introduces observability for the page defragmentation worker by adding several primary-GT statistics counters. These counters track the success, failure, and amount of data moved during defragmentation attempts.

Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
index 6a847e42946ce..c1bbcf0d21edb 100644

[ ... ]

Quoted 7 lines
drivers/gpu/drm/xe/xe_bo.c +3 -0
@@ -1264,6 +1267,20 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 	xe_dbg(xe, "Defrag attempt on BO size=%zu: ret=%pe consumed=%llu\n",
 	       xe_bo_size(bo), ERR_PTR(ret), *consumed);
 
+	if (!ret && ttm_tt_is_beneficial_order_failed(bo->ttm.ttm))
+		xe_gt_stats_incr(xe_root_mmio_gt(xe),
+				 XE_GT_STATS_ID_DEFRAG_PARTIAL_SUCCESS_COUNT,

drivers/gpu/drm/xe/xe_bo.c static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo, · line 1282

Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +10 -0
+				 1);
+	else
+		xe_gt_stats_incr(xe_root_mmio_gt(xe),
+				 ret ? XE_GT_STATS_ID_DEFRAG_FAILED_COUNT :
+				       XE_GT_STATS_ID_DEFRAG_SUCCESS_COUNT, 1);
+
+	if (!ret)
+		xe_gt_stats_incr(xe_root_mmio_gt(xe),
+				 XE_GT_STATS_ID_DEFRAG_MB_MOVED,
+				 *consumed >> 20);

[Severity: Medium]
Will this undercount the data moved if we truncate to megabytes before accumulation?

If the consumed amount is less than 1 MiB (e.g. re-backing a 64 KiB page), this evaluates to 0 and nothing is added to the global counter. The sub-megabyte remainders of larger moves also appear to be permanently lost on every function call.

Would it be better to accumulate the exact value in bytes (or kilobytes) and only perform the division when presenting the statistic to user-space?

Quoted 3 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
 unlock:
 	xe_bo_unlock(bo);
 	return ret;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=21
author maintainer
#41 lore raw

PATCH v2 22/33 drm/xe: Add Kconfig.profile options for BO defrag configuration

In reply to Matthew Brost

Add Kconfig.profile options to make XE_BO_DEFRAG_* defines configurable:
- DRM_XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD: Threshold for TTM reclaim backoff
- DRM_XE_BO_DEFRAG_SIZE_LIMIT: Maximum number of bytes to defrag per work run
- DRM_XE_BO_DEFRAG_INTERVAL_MS: Default delay before defrag worker runs
- DRM_XE_BO_DEFRAG_INTERVAL_MAX_MS: Upper bound for defrag worker interval

Update xe_bo.c to use these Kconfig options as defaults via #ifdef guards, maintaining backward compatibility with hardcoded values when not configured.

Additionally, disable defrag completely when XE_BO_DEFRAG_SIZE_LIMIT is set less than 2M by returning early from xe_bo_defrag_update() (and xe_bo_defrag_remove()) so BOs are never added to the defrag list or the worker kicked; xe_bo_defrag_add() and xe_bo_defrag_schedule() assert the limit is at least 2M.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-haiku-4.5
---
 drivers/gpu/drm/xe/Kconfig.profile | 40 ++++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_bo.c         | 26 +++++++++++++++++++
 2 files changed, 66 insertions(+)
drivers/gpu/drm/xe/Kconfig.profile +40 -0
index e07517d120e0..e0aade41f53e 100644
@@ -74,3 +74,43 @@ config DRM_XE_ENABLE_SCHEDTIMEOUT_LIMIT
 	  to apply to applicable user. For elevated user, all above MIN
 	  and MAX values will apply when this configuration is enable to
 	  apply limitation. By default limitation is applied.
+
+config DRM_XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD
+	int "BO defrag reclaim backoff threshold"
+	default 2
+	range 1 1000
+	help
+	  Once this many BOs are tracked on the device defrag list (i.e. were
+	  backed with a sub-optimal page order), request that the TTM pool backs
+	  off from aggressive reclaim at the beneficial order during populate,
+	  so that allocations make forward progress instead of stalling.
+
+config DRM_XE_BO_DEFRAG_SIZE_LIMIT
+	int "Maximum bytes to defrag per work run"
+	default 33554432
+	range 0 1073741824
+	help
+	  Maximum number of bytes the defrag worker will process in a single run
+	  before yielding and rescheduling itself. Set to less than large page
+	  size (2M) to disable defrag. A defrag move synchronously reallocates
+	  and re-copies a BO's backing store, which is not free. If a large
+	  number of BOs become eligible at once, processing them all in one
+	  worker run would hold things up for a long, unbounded stretch.
+	  Instead, cap the work done per run and requeue, spreading the defrag
+	  effort out over time.
+
+config DRM_XE_BO_DEFRAG_INTERVAL_MS
+	int "Default delay before defrag worker run (ms)"
+	default 25
+	range 1 1000
+	help
+	  Default delay before (re)running the defrag worker, in milliseconds.
+
+config DRM_XE_BO_DEFRAG_INTERVAL_MAX_MS
+	int "Upper bound for defrag worker interval (ms)"
+	default 15000
+	range 100 1000000
+	help
+	  Upper bound for the (exponentially backed off) defrag worker interval,
+	  in milliseconds, so repeated failures don't push the retry arbitrarily
+	  far out. 15000 ms = 15 seconds.
drivers/gpu/drm/xe/xe_bo.c +26 -0
index c1bbcf0d21ed..405316d0d116 100644
@@ -49,7 +49,11 @@
  * aggressive reclaim at the beneficial order during populate, so that
  * allocations make forward progress instead of stalling.
  */
+#ifdef CONFIG_DRM_XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD
+#define XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD	CONFIG_DRM_XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD
+#else
 #define XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD	2
+#endif
 
 /*
  * Maximum number of bytes of newly (re)allocated backing the defrag worker will
@@ -65,17 +69,31 @@
  * Only pages a move truly reallocates are charged; pages harvested from the old
  * backing are free, so an object larger than the budget is upgraded in
  * budget-sized slices across runs.
+ *
+ * Set to 0 to disable defrag completely.
  */
+#ifdef CONFIG_DRM_XE_BO_DEFRAG_SIZE_LIMIT
+#define XE_BO_DEFRAG_SIZE_LIMIT			CONFIG_DRM_XE_BO_DEFRAG_SIZE_LIMIT
+#else
 #define XE_BO_DEFRAG_SIZE_LIMIT			SZ_32M
+#endif
 
 /* Default delay before (re)running the defrag worker, in milliseconds. */
+#ifdef CONFIG_DRM_XE_BO_DEFRAG_INTERVAL_MS
+#define XE_BO_DEFRAG_INTERVAL_MS		CONFIG_DRM_XE_BO_DEFRAG_INTERVAL_MS
+#else
 #define XE_BO_DEFRAG_INTERVAL_MS		25
+#endif
 
 /*
  * Upper bound for the (exponentially backed off) defrag worker interval, in
  * milliseconds, so repeated failures don't push the retry arbitrarily far out.
  */
+#ifdef CONFIG_DRM_XE_BO_DEFRAG_INTERVAL_MAX_MS
+#define XE_BO_DEFRAG_INTERVAL_MAX_MS		CONFIG_DRM_XE_BO_DEFRAG_INTERVAL_MAX_MS
+#else
 #define XE_BO_DEFRAG_INTERVAL_MAX_MS		15000	/* 15 seconds */
+#endif
 
 static void xe_bo_defrag_worker(struct work_struct *w);
 static void xe_place_from_ttm_type(u32 mem_type, struct ttm_place *place);
@@ -1120,6 +1138,7 @@ int xe_bo_defrag_init(struct xe_device *xe)
 
 static void xe_bo_defrag_schedule(struct xe_device *xe)
 {
+	xe_assert(xe, XE_BO_DEFRAG_SIZE_LIMIT >= SZ_2M);
 	schedule_delayed_work(&xe->mem.defrag.worker,
 			      msecs_to_jiffies(xe->mem.defrag.interval_ms));
 }
@@ -1146,6 +1165,7 @@ static void xe_bo_defrag_add(struct xe_bo *bo)
 
 	xe_bo_assert_held(bo);
 	xe_assert(xe, xe_bo_needs_defrag(bo));
+	xe_assert(xe, XE_BO_DEFRAG_SIZE_LIMIT >= SZ_2M);
 
 	scoped_guard(spinlock, &xe->mem.defrag.lock) {
 		if (list_empty(&bo->defrag_link)) {
@@ -1179,6 +1199,9 @@ static void xe_bo_defrag_remove(struct xe_bo *bo)
 {
 	xe_bo_assert_held(bo);
 
+	if (XE_BO_DEFRAG_SIZE_LIMIT < SZ_2M)
+		return;
+
 	if (list_empty(&bo->defrag_link))
 		return;
 
@@ -1198,6 +1221,9 @@ static void xe_bo_defrag_update(struct xe_bo *bo)
 {
 	xe_bo_assert_held(bo);
 
+	if (XE_BO_DEFRAG_SIZE_LIMIT < SZ_2M)
+		return;
+
 	if (xe_bo_needs_defrag(bo))
 		xe_bo_defrag_add(bo);
 	else
Signature · 1 line
2.34.1
author maintainer
#42 lore raw

PATCH v2 23/33 drm/xe: Defrag using out-of-lock page preallocation

In reply to Matthew Brost

Convert xe_bo_defrag_one() to a three-phase scheme that moves the expensive beneficial-order allocations out of the BO dma-resv lock:

1. Under the lock, re-check eligibility, capture the tt cpu-caching and read the exact sub-optimal page count (ttm_tt_suboptimal_pages()), bounded by the run budget.
2. Drop the lock and preallocate that many beneficial-order chunks via ttm_pool_prealloc_fill(); reclaim/compaction stalls and the cpu cache mode change both happen here, outside the lock.
3. Re-take the lock, re-check, and validate consuming the prealloc bag. Any shortfall harvests the old backing instead of reclaiming, so the move never stalls under the lock; the BO re-queues for a later pass.

With both allocation and caching hoisted out, the BO-lock held time drops to copy-bound (median ~0.27ms, p95 ~2ms); the remaining tail is lock-wait behind active rendering, as intended.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_bo.c | 46 +++++++++++++++++++++++++++++++++++++-
 1 file changed, 45 insertions(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_bo.c +45 -1
index 405316d0d116..2f014fc9f988 100644
@@ -1249,15 +1249,55 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 		.defrag_bytes_remaining = budget,
 	};
 	struct ttm_buffer_object *ttm_bo = &bo->ttm;
+	struct ttm_pool *pool = &ttm_bo->bdev->pool;
+	struct ttm_pool_prealloc pp = {};
 	struct ttm_placement placement;
 	struct ttm_place place;
+	enum ttm_caching tt_caching;
+	unsigned int order, want = 0;
 	int ret = 0;
 
 	*consumed = 0;
 
+	/*
+	 * Phase 1: under the BO lock, re-check eligibility and estimate how
+	 * many beneficial-order chunks the move will (re)allocate, bounded by
+	 * the run's byte budget. Then drop the lock so the high-order, possibly
+	 * reclaim/compaction stalling allocations happen unlocked.
+	 */
+	xe_bo_lock(bo, false);
+	if (!xe_bo_needs_defrag(bo)) {
+		xe_bo_defrag_remove(bo);
+		xe_bo_unlock(bo);
+		goto out;
+	}
+	tt_caching = bo->ttm.ttm->caching;
+	order = ttm_pool_prealloc_order(pool);
+	if (order && ttm_bo->ttm) {
+		u32 suboptimal = ttm_tt_suboptimal_pages(ttm_bo->ttm);
+		u64 cap = min_t(u64, budget, xe_bo_size(bo)) >> PAGE_SHIFT;
+
+		/* Prealloc only the beneficial-order chunks the move replaces. */
+		want = DIV_ROUND_UP(min_t(u64, suboptimal, cap), 1UL << order);
+	}
+	xe_bo_unlock(bo);
+
+	/* Phase 2: preallocate outside the lock; */
+	if (want) {
+		ret = ttm_pool_prealloc_fill(pool, tt_caching, &pp, want);
+		if (ret || !pp.count) {
+			ret = ret ?: -ENOMEM;
+			xe_gt_stats_incr(xe_root_mmio_gt(xe),
+					 XE_GT_STATS_ID_DEFRAG_FAILED_COUNT, 1);
+			goto out_err;
+		}
+	}
+
+	/*
+	 * Phase 3: re-take the lock, re-check, and validate using the prealloc.
+	 */
 	xe_bo_lock(bo, false);
 
-	/* Re-check eligibility under the BO lock. */
 	if (!xe_bo_needs_defrag(bo)) {
 		xe_bo_defrag_remove(bo);
 		goto unlock;
@@ -1267,6 +1307,7 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 	place.flags |= TTM_PL_FLAG_TEMPORARY;
 	placement.num_placement = 1;
 	placement.placement = &place;
+	ctx.prealloc = &pp;
 
 	/*
 	 * On success the move reallocates the backing at beneficial order and
@@ -1309,6 +1350,9 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 
 unlock:
 	xe_bo_unlock(bo);
+out_err:
+	ttm_pool_prealloc_fini(pool, &pp);
+out:
 	return ret;
 }
 
Signature · 1 line
2.34.1
#43 lore raw

Re: [PATCH v2 23/33] drm/xe: Defrag using out-of-lock page preallocation

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [Low] 64-bit division using the '/' operator causes a link-time build failure on 32-bit architectures (missing __udivdi3).

Pre-existing issues:
- [High] Accessing bo->ttm.ttm without holding the BO's dma-resv lock can lead to a NULL pointer dereference or UAF if the BO is concurrently evicted or destroyed. --

--- Patch [23]: [PATCH v2 23/33] drm/xe: Defrag using out-of-lock page preallocation --- commit d11ab210d871e5fedb57edb2f956ed1814f6d5a4

Author: Matthew Brost <[email protected]>
Date:   Fri Jul 10 21:54:32 2026 +0000

drm/xe: Defrag using out-of-lock page preallocation

Convert xe_bo_defrag_one() to a three-phase scheme that moves the expensive beneficial-order allocations out of the BO dma-resv lock. This hoists allocation and caching out to reduce BO-lock held time.

Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
index 405316d0d116b..2f014fc9f988e 100644

[ ... ]

Quoted 1 line
drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -1249,15 +1249,55 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,

[ ... ]

drivers/gpu/drm/xe/xe_bo.c static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo, · line 1281

Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +6 -0
+	if (order && ttm_bo->ttm) {
+		u32 suboptimal = ttm_tt_suboptimal_pages(ttm_bo->ttm);
+		u64 cap = min_t(u64, budget, xe_bo_size(bo)) >> PAGE_SHIFT;
+
+		/* Prealloc only the beneficial-order chunks the move replaces. */
+		want = DIV_ROUND_UP(min_t(u64, suboptimal, cap), 1UL << order);
                 ^^^^^^^^^^^^

[Severity: Low]
Does this code cause a link-time build failure on 32-bit architectures?

The first argument to DIV_ROUND_UP() is a u64, and the second evaluates to an unsigned long (32-bit integer) and is not a compile-time constant. This results in a 64-bit division using the '/' operator, which emits a call to __udivdi3 on 32-bit GCC. Because the kernel doesn't provide this intrinsic, it will break the build.

Could DIV_ROUND_UP_ULL() be used here instead?

drivers/gpu/drm/xe/xe_bo.c

Quoted 2 lines
drivers/gpu/drm/xe/xe_bo.c +2 -0
+	}
+	xe_bo_unlock(bo);

[ ... ]

This is a pre-existing issue, but I noticed a potential problem in xe_bo_defrag_worker() regarding how the BO is accessed after unlocking:

drivers/gpu/drm/xe/xe_bo.c:xe_bo_defrag_worker() {
	...
		ret = xe_bo_defrag_one(xe, bo, XE_BO_DEFRAG_SIZE_LIMIT - defrag_bytes,
				       &consumed);
		defrag_bytes += consumed;

		if (ret || ttm_tt_is_beneficial_order_failed(bo->ttm.ttm)) {
	...
}

[Severity: High]
Can accessing bo->ttm.ttm here without holding the BO's dma-resv lock lead to a NULL pointer dereference or use-after-free?

xe_bo_defrag_worker() calls xe_bo_defrag_one(), which acquires the BO lock, performs the move, and then unlocks the BO before returning. After returning, the worker evaluates ttm_tt_is_beneficial_order_failed(bo->ttm.ttm) without holding the BO lock.

If another thread concurrently evicts the BO to VRAM, the system memory ttm_tt is destroyed and bo->ttm.ttm is set to NULL. Additionally, if xe_bo_defrag_one() finds the BO no longer needs defragmentation, it removes it from the list and returns 0, and bo->ttm.ttm could also be NULL at this point.

Should this check be performed while the BO lock is still held?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=23
author maintainer
#44 lore raw

PATCH v2 24/33 drm/xe: Add defrag profiling tracepoints

In reply to Matthew Brost

Add two tracepoints to profile the page defragmentation worker:

- xe_bo_defrag_worker: emitted on worker entry and exit with the current defrag list count and the bytes processed in the run.
- xe_bo_defrag_one: emitted per BO with the run budget, the prealloc fill ratio (got/want) and the lock-held, lock-wait and prealloc durations, plus the bytes consumed and the result.

These break down the cost of a defrag attempt into its locked vs unlocked phases, making it possible to confirm that the expensive beneficial-order allocation and cpu cache mode change happen outside the BO dma-resv lock and that the residual lock-held time is copy-bound (with the tail dominated by lock-wait behind active rendering).

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_bo.c       | 18 +++++++++-
 drivers/gpu/drm/xe/xe_trace_bo.h | 59 ++++++++++++++++++++++++++++++++
 2 files changed, 76 insertions(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_bo.c +17 -1
index 2f014fc9f988..a01004265349 100644
@@ -1254,7 +1254,9 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 	struct ttm_placement placement;
 	struct ttm_place place;
 	enum ttm_caching tt_caching;
-	unsigned int order, want = 0;
+	unsigned int order, want = 0, got = 0;
+	u64 lock_ns = 0, prealloc_ns = 0, lockwait_ns = 0;
+	ktime_t t0;
 	int ret = 0;
 
 	*consumed = 0;
@@ -1284,6 +1286,7 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 
 	/* Phase 2: preallocate outside the lock; */
 	if (want) {
+		t0 = ktime_get();
 		ret = ttm_pool_prealloc_fill(pool, tt_caching, &pp, want);
 		if (ret || !pp.count) {
 			ret = ret ?: -ENOMEM;
@@ -1291,12 +1294,16 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 					 XE_GT_STATS_ID_DEFRAG_FAILED_COUNT, 1);
 			goto out_err;
 		}
+		prealloc_ns = ktime_to_ns(ktime_sub(ktime_get(), t0));
 	}
 
 	/*
 	 * Phase 3: re-take the lock, re-check, and validate using the prealloc.
 	 */
+	t0 = ktime_get();
 	xe_bo_lock(bo, false);
+	lockwait_ns = ktime_to_ns(ktime_sub(ktime_get(), t0));
+	t0 = ktime_get();
 
 	if (!xe_bo_needs_defrag(bo)) {
 		xe_bo_defrag_remove(bo);
@@ -1350,9 +1357,13 @@ static int xe_bo_defrag_one(struct xe_device *xe, struct xe_bo *bo,
 
 unlock:
 	xe_bo_unlock(bo);
+	lock_ns = ktime_to_ns(ktime_sub(ktime_get(), t0));
+	got = pp.used;
 out_err:
 	ttm_pool_prealloc_fini(pool, &pp);
 out:
+	trace_xe_bo_defrag_one(bo, budget, want, got, lock_ns, prealloc_ns,
+			       lockwait_ns, *consumed, ret);
 	return ret;
 }
 
@@ -1373,6 +1384,9 @@ static void xe_bo_defrag_worker(struct work_struct *w)
 		return;
 	}
 
+	trace_xe_bo_defrag_worker(xe, 0, atomic_read(&xe->mem.defrag.count),
+				  true);
+
 	/*
 	 * Process at most XE_BO_DEFRAG_SIZE_LIMIT bytes of newly (re)allocated
 	 * backing per run rather than draining the whole list in one go. Only
@@ -1479,6 +1493,8 @@ static void xe_bo_defrag_worker(struct work_struct *w)
 
 	xe_pm_runtime_put(xe);
 	drm_dev_exit(idx);
+	trace_xe_bo_defrag_worker(xe, defrag_bytes,
+				  atomic_read(&xe->mem.defrag.count), false);
 }
 
 static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
drivers/gpu/drm/xe/xe_trace_bo.h +59 -0

65 lines not on this page · raw message

Signature · 1 line
2.34.1
#45 lore raw

Re: [PATCH v2 24/33] drm/xe: Add defrag profiling tracepoints

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] 64-bit integer division in TP_printk causes a link error (__udivdi3 undefined) on 32-bit…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] 64-bit integer division in TP_printk causes a link error (__udivdi3 undefined) on 32-bit architectures. --

--- Patch [24]: [PATCH v2 24/33] drm/xe: Add defrag profiling tracepoints --- commit 414478b1f94933fc988a5668d191a4c7976fd319
Author: Matthew Brost <[email protected]>

drm/xe: Add defrag profiling tracepoints

This commit adds two tracepoints, xe_bo_defrag_worker and xe_bo_defrag_one, to profile the page defragmentation worker and break down the cost of a defrag attempt.

Quoted 4 lines
drivers/gpu/drm/xe/xe_trace_bo.h +0 -0
index 86323cf3be2c0..a169a322464b2 100644

[ ... ]

Quoted 1 line

+TRACE_EVENT(xe_bo_defrag_one,

[ ... ]

drivers/gpu/drm/xe/xe_trace_bo.h TRACE_EVENT(xe_bo_move, · line 145

Quoted 7 lines
drivers/gpu/drm/xe/xe_trace_bo.h +7 -0
+	    TP_printk("dev=%s size=%zu budget=%llu prealloc=%u/%u lock_us=%llu lockwait_us=%llu prealloc_us=%llu consumed=%llu ret=%d",
+		      __get_str(dev), __entry->size, __entry->budget,
+		      __entry->prealloc_got, __entry->prealloc_want,
+		      __entry->lock_ns / 1000, __entry->lockwait_ns / 1000,
+		      __entry->prealloc_ns / 1000, __entry->consumed,
+		      __entry->ret)
+);

[Severity: Low]
Will doing 64-bit integer divisions (__entry->lock_ns / 1000, etc.) in TP_printk cause build failures on 32-bit architectures?

The C code generated by TP_printk for TRACE_EVENT(xe_bo_defrag_one) performs direct u64 division. On 32-bit platforms, this translates to a dependency on the __udivdi3 libgcc function. Since the kernel does not export this function, this will result in a linking error during 32-bit builds (such as when COMPILE_TEST is enabled).

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=24
author maintainer
#46 lore raw

PATCH v2 25/33 drm/xe: Preallocate system BO backing outside the dma-resv lock

In reply to Matthew Brost

xe_gem_create_ioctl() populates a buffer object's backing store inside xe_validation_guard(), i.e. under the dma-resv lock. For a system-memory buffer that means the whole page allocation - potentially stalling in reclaim or compaction - happens in the critical section, holding the reservation and blocking anything that contends on it.

Use the TTM full-preallocation mechanism to move that work out of the lock. For a system-only, non-deferred user BO, allocate the entire tt backing up front, before entering the validation guard, and hand the bag down to the populate so it merely installs the pages:

- Thread an optional struct ttm_pool_prealloc through xe_bo_create_user() -> __xe_bo_create_locked() -> xe_bo_init_locked(), which stashes it in ttm_operation_ctx::prealloc. All other callers pass NULL.

- In xe_gem_create_ioctl(), for XE_BO_FLAG_SYSTEM BOs that are not VRAM, stolen or deferred-backing, call ttm_pool_prealloc_fill_full() before the guard and ttm_pool_prealloc_fini() after it. The bag size covers both the data pages and the appended CCS backup pages, matching the tt layout in xe_ttm_tt_create(), and the caching mirrors the same path.

The fill is best-effort: on a short fill (or if it is skipped) the pool allocates the remainder in-line as before, so this is purely an optimisation with no behavioural change to the created BO. fini() only releases the unconsumed tail, so it is safe on every success, error and OOM-retry path (consumed pages are owned by the tt).

The fill requests reclaim backoff for higher-order chunks (@beneficial_reclaim_backoff): Xe runs a background defragmenter that upgrades suboptimal pages to beneficial order in place, so there is no need to stall this user context reclaiming/compacting for contiguity.

To size the preallocation before the xe_bo exists, factor the CCS-pages decision in xe_bo_needs_ccs_pages() into a parameterised helper, __xe_bo_needs_ccs_pages(), and query it directly from the ioctl.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/tests/xe_bo.c      |   8 +-
 drivers/gpu/drm/xe/tests/xe_dma_buf.c |   2 +-
 drivers/gpu/drm/xe/tests/xe_migrate.c |   6 +-
 drivers/gpu/drm/xe/xe_bo.c            | 120 ++++++++++++++++++++------
 drivers/gpu/drm/xe/xe_bo.h            |   8 +-
 drivers/gpu/drm/xe/xe_dma_buf.c       |   2 +-
 6 files changed, 109 insertions(+), 37 deletions(-)
drivers/gpu/drm/xe/tests/xe_bo.c +4 -4
index 49c95ed67d7e..78781317b420 100644
@@ -141,7 +141,7 @@ static void ccs_test_run_tile(struct xe_device *xe, struct xe_tile *tile,
 		kunit_info(test, "Testing system memory\n");
 
 	bo = xe_bo_create_user(xe, NULL, SZ_1M, DRM_XE_GEM_CPU_CACHING_WC,
-			       bo_flags, exec);
+			       bo_flags, NULL, exec);
 	if (IS_ERR(bo)) {
 		KUNIT_FAIL(test, "Failed to create bo.\n");
 		return;
@@ -220,7 +220,7 @@ static int evict_test_run_tile(struct xe_device *xe, struct xe_tile *tile, struc
 		xe_vm_lock(vm, false);
 		bo = xe_bo_create_user(xe, vm, 0x10000,
 				       DRM_XE_GEM_CPU_CACHING_WC,
-				       bo_flags, exec);
+				       bo_flags, NULL, exec);
 		xe_vm_unlock(vm);
 		if (IS_ERR(bo)) {
 			KUNIT_FAIL(test, "bo create err=%pe\n", bo);
@@ -229,7 +229,7 @@ static int evict_test_run_tile(struct xe_device *xe, struct xe_tile *tile, struc
 
 		external = xe_bo_create_user(xe, NULL, 0x10000,
 					     DRM_XE_GEM_CPU_CACHING_WC,
-					     bo_flags, NULL);
+					     bo_flags, NULL, NULL);
 		if (IS_ERR(external)) {
 			KUNIT_FAIL(test, "external bo create err=%pe\n", external);
 			goto cleanup_bo;
@@ -494,7 +494,7 @@ static int shrink_test_run_device(struct xe_device *xe)
 		/* We can create bos using WC caching here. But it is slower. */
 		bo = xe_bo_create_user(xe, NULL, XE_BO_SHRINK_SIZE,
 				       DRM_XE_GEM_CPU_CACHING_WB,
-				       XE_BO_FLAG_SYSTEM, NULL);
+				       XE_BO_FLAG_SYSTEM, NULL, NULL);
 		if (IS_ERR(bo)) {
 			if (bo != ERR_PTR(-ENOMEM) && bo != ERR_PTR(-ENOSPC) &&
 			    bo != ERR_PTR(-EINTR) && bo != ERR_PTR(-ERESTARTSYS))
drivers/gpu/drm/xe/tests/xe_dma_buf.c +1 -1
index 0be8440b3976..e4fdc247c0c7 100644
@@ -127,7 +127,7 @@ static void xe_test_dmabuf_import_same_driver(struct xe_device *xe)
 
 	kunit_info(test, "running %s\n", __func__);
 	bo = xe_bo_create_user(xe, NULL, size, DRM_XE_GEM_CPU_CACHING_WC,
-			       params->mem_mask, NULL);
+			       params->mem_mask, NULL, NULL);
 	if (IS_ERR(bo)) {
 		KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
 			   PTR_ERR(bo));
drivers/gpu/drm/xe/tests/xe_migrate.c +3 -3
index 3c1be809be82..41de648b2c86 100644
@@ -647,7 +647,7 @@ static void validate_ccs_test_run_tile(struct xe_device *xe, struct xe_tile *til
 				   DRM_XE_GEM_CPU_CACHING_WC,
 				   XE_BO_FLAG_SYSTEM |
 				   XE_BO_FLAG_NEEDS_CPU_ACCESS |
-				   XE_BO_FLAG_PINNED, NULL);
+				   XE_BO_FLAG_PINNED, NULL, NULL);
 
 	if (IS_ERR(sys_bo)) {
 		KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
@@ -673,7 +673,7 @@ static void validate_ccs_test_run_tile(struct xe_device *xe, struct xe_tile *til
 	ccs_bo = xe_bo_create_user(xe, NULL, SZ_4M,
 				   DRM_XE_GEM_CPU_CACHING_WC,
 				   bo_flags | XE_BO_FLAG_NEEDS_CPU_ACCESS |
-				   XE_BO_FLAG_PINNED, NULL);
+				   XE_BO_FLAG_PINNED, NULL, NULL);
 
 	if (IS_ERR(ccs_bo)) {
 		KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
@@ -698,7 +698,7 @@ static void validate_ccs_test_run_tile(struct xe_device *xe, struct xe_tile *til
 	vram_bo = xe_bo_create_user(xe, NULL, SZ_4M,
 				    DRM_XE_GEM_CPU_CACHING_WC,
 				    bo_flags | XE_BO_FLAG_NEEDS_CPU_ACCESS |
-				    XE_BO_FLAG_PINNED, NULL);
+				    XE_BO_FLAG_PINNED, NULL, NULL);
 	if (IS_ERR(vram_bo)) {
 		KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
 			   PTR_ERR(vram_bo));
drivers/gpu/drm/xe/xe_bo.c +94 -26

271 lines not on this page · raw message

drivers/gpu/drm/xe/xe_bo.h +6 -2
index 38d50f8122cb..18fcccd082b7 100644
@@ -111,6 +111,7 @@ enum xe_madv_purgeable_state {
 
 struct sg_table;
 struct ttm_tt;
+struct ttm_pool_prealloc;
 
 struct xe_bo *xe_bo_alloc(void);
 void xe_bo_free(struct xe_bo *bo);
@@ -119,13 +120,16 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,
 				struct xe_tile *tile, struct dma_resv *resv,
 				struct ttm_lru_bulk_move *bulk, size_t size,
 				u16 cpu_caching, enum ttm_bo_type type,
-				u32 flags, struct drm_exec *exec);
+				u32 flags, struct ttm_pool_prealloc *prealloc,
+				struct drm_exec *exec);
 struct xe_bo *xe_bo_create_locked(struct xe_device *xe, struct xe_tile *tile,
 				  struct xe_vm *vm, size_t size,
 				  enum ttm_bo_type type, u32 flags,
 				  struct drm_exec *exec);
 struct xe_bo *xe_bo_create_user(struct xe_device *xe, struct xe_vm *vm, size_t size,
-				u16 cpu_caching, u32 flags, struct drm_exec *exec);
+				u16 cpu_caching, u32 flags,
+				struct ttm_pool_prealloc *prealloc,
+				struct drm_exec *exec);
 struct xe_bo *xe_bo_create_pin_map(struct xe_device *xe, struct xe_tile *tile,
 				   struct xe_vm *vm, size_t size,
 				   enum ttm_bo_type type, u32 flags,
drivers/gpu/drm/xe/xe_dma_buf.c +1 -1
index 8a920e58245c..ec947d86a1ce 100644
@@ -302,7 +302,7 @@ xe_dma_buf_create_obj(struct drm_device *dev, struct dma_buf *dma_buf)
 
 		bo = xe_bo_init_locked(xe, NULL, NULL, resv, NULL, dma_buf->size,
 				       0, /* Will require 1way or 2way for vm_bind */
-				       ttm_bo_type_sg, XE_BO_FLAG_SYSTEM, &exec);
+				       ttm_bo_type_sg, XE_BO_FLAG_SYSTEM, NULL, &exec);
 		drm_exec_retry_on_contention(&exec);
 		if (IS_ERR(bo)) {
 			ret = PTR_ERR(bo);
Signature · 1 line
2.34.1
#47 lore raw

Re: [PATCH v2 25/33] drm/xe: Preallocate system BO backing outside the dma-resv lock

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Integer truncation in xe_device_ccs_bytes() returning u32 causes buffer overflows and physical memory corruption/panics for BOs >= 1TB. --

--- Patch [25]: [PATCH v2 25/33] drm/xe: Preallocate system BO backing outside the dma-resv lock --- commit aa051bafd6b83b01e59b139bb60870dfa1044d22
Author: Matthew Brost <[email protected]>
Subject: drm/xe: Preallocate system BO backing outside the dma-resv lock

xe_gem_create_ioctl() populates a buffer object's backing store inside xe_validation_guard(), i.e. under the dma-resv lock. This patch uses the TTM full-preallocation mechanism to move that work out of the lock.

Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
index a010042653495..d9226a3995fe4 100644

[ ... ]

Quoted 20 lines
drivers/gpu/drm/xe/xe_bo.c +16 -0
@@ -3994,6 +4009,45 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 			return -ENOENT;
 	}
 
+	/*
+	 * For system-only BOs, preallocate the whole backing outside the
+	 * dma-resv lock so the populate under the validation guard does not
+	 * stall in reclaim/compaction. The caching mirrors the system-BO path
+	 * in xe_ttm_tt_create(); the pool falls back to in-line allocation for
+	 * any shortfall, so a failed fill is harmless.
+	 */
+	if ((bo_flags & XE_BO_FLAG_SYSTEM) &&
+	    !(bo_flags & (XE_BO_FLAG_DEFER_BACKING | XE_BO_FLAG_VRAM_MASK |
+			  XE_BO_FLAG_STOLEN))) {
+		enum ttm_caching caching = ttm_cached;
+		unsigned long num_pages = args->size >> PAGE_SHIFT;
+
+		if (!IS_DGFX(xe) &&
+		    args->cpu_caching == DRM_XE_GEM_CPU_CACHING_WC)
+			caching = ttm_write_combined;

drivers/gpu/drm/xe/xe_bo.c int xe_gem_create_ioctl(struct drm_device *dev, void *data, · line 4037

Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +10 -0
+
+		/*
+		 * Cover the CCS backup pages appended to the tt (see
+		 * xe_ttm_tt_create()) so the whole tt is preallocated.
+		 */
+		if (__xe_bo_needs_ccs_pages(xe, ttm_bo_type_device, bo_flags,
+					    args->cpu_caching))
+			num_pages += DIV_ROUND_UP(xe_device_ccs_bytes(xe,
+								      args->size),
+						  PAGE_SIZE);

[Severity: High]
This is a pre-existing issue, but can this lead to an integer truncation for very large buffer sizes?

The function xe_device_ccs_bytes() in drivers/gpu/drm/xe/xe_device.c returns a u32, but takes a u64 size parameter. If a user requests a BO allocation of

Quoted 1 line

= 1TB via DRM_IOCTL_XE_GEM_CREATE with CCS enabled, the required CCS size

would be 1TB / 256 = 4GB, which silently truncates to 0 in the u32 return type.

Because this patch propagates the pattern by calling xe_device_ccs_bytes() on the user-controlled args->size, the extra_pages for the CCS backup could be calculated as 0, under-allocating the BO's SG table. Later, when xe_migrate_copy() processes the BO in chunks, it could advance the CCS cursor past the end of the under-allocated SG table, resulting in a NULL pointer dereference and kernel panic.

Should xe_device_ccs_bytes() be updated to return a u64, or should there be size limits enforced earlier in the ioctl?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=25
author maintainer
#48 lore raw

PATCH v2 26/33 drm/xe: Add tracepoint for xe_gem_create_ioctl

In reply to Matthew Brost

Add a xe_gem_create_ioctl tracepoint to help profile GEM creation, in particular the cost of populating a BO while holding the dma-resv lock versus the total IOCTL cost. This is useful for evaluating the out-of-lock preallocation path, which aims to shrink the time spent under the lock.

The tracepoint records the BO size, the requested placement mask, the CPU caching mode, the VM asid (0 when no VM is bound), the total time spent in the IOCTL, and the time spent under the dma-resv lock (the xe_validation_guard region where the lock is held).

The asid is captured at VM lookup time rather than at trace emission, since the VM reference is dropped before the tracepoint fires.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_bo.c       | 16 ++++++++++++++++
 drivers/gpu/drm/xe/xe_trace_bo.h | 31 +++++++++++++++++++++++++++++++
 2 files changed, 47 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +16 -0
index d9226a3995fe..c096f3dbd528 100644
@@ -3929,6 +3929,10 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 	bool have_prealloc = false;
 	unsigned int bo_flags;
 	u32 handle;
+	ktime_t ioctl_start = ktime_get();
+	ktime_t lock_start = ktime_set(0, 0);
+	u64 lock_ns = 0;
+	u32 asid = 0;
 	int err;
 
 	if (XE_IOCTL_DBG(xe, args->pad[0] || args->pad[1] || args->pad[2]) ||
@@ -4007,6 +4011,7 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 		vm = xe_vm_lookup(xef, args->vm_id);
 		if (XE_IOCTL_DBG(xe, !vm))
 			return -ENOENT;
+		asid = vm->usm.asid;
 	}
 
 	/*
@@ -4057,6 +4062,10 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 			if (err)
 				break;
 		}
+
+		if (lock_start == ktime_set(0, 0))
+			lock_start = ktime_get();
+
 		bo = xe_bo_create_user(xe, vm, args->size, args->cpu_caching,
 				       bo_flags, have_prealloc ? &prealloc : NULL,
 				       &exec);
@@ -4067,6 +4076,7 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 			break;
 		}
 	}
+	lock_ns = ktime_to_ns(ktime_sub(ktime_get(), lock_start));
 	if (have_prealloc)
 		ttm_pool_prealloc_fini(&xe->ttm.pool, &prealloc);
 	if (err)
@@ -4097,6 +4107,12 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 	if (vm)
 		xe_vm_put(vm);
 
+	trace_xe_gem_create_ioctl(xe, args->size, args->placement,
+				  args->cpu_caching, asid,
+				  ktime_to_ns(ktime_sub(ktime_get(),
+							ioctl_start)) / 1000,
+				  lock_ns / 1000);
+
 	return err;
 }
 
drivers/gpu/drm/xe/xe_trace_bo.h +31 -0
index a169a322464b..b2dacb5fa3f7 100644
@@ -144,6 +144,37 @@ TRACE_EVENT(xe_bo_defrag_one,
 		      __entry->ret)
 );
 
+TRACE_EVENT(xe_gem_create_ioctl,
+	    TP_PROTO(struct xe_device *xe, u64 size, u32 placement,
+		     u16 caching, u32 asid, u64 ioctl_us, u64 lock_us),
+	    TP_ARGS(xe, size, placement, caching, asid, ioctl_us, lock_us),
+
+	    TP_STRUCT__entry(
+		     __string(dev, dev_name(xe->drm.dev))
+		     __field(u64, size)
+		     __field(u32, placement)
+		     __field(u16, caching)
+		     __field(u32, asid)
+		     __field(u64, ioctl_us)
+		     __field(u64, lock_us)
+		     ),
+
+	    TP_fast_assign(
+		   __assign_str(dev);
+		   __entry->size = size;
+		   __entry->placement = placement;
+		   __entry->caching = caching;
+		   __entry->asid = asid;
+		   __entry->ioctl_us = ioctl_us;
+		   __entry->lock_us = lock_us;
+		   ),
+
+	    TP_printk("dev=%s, size=%llu, placement=0x%x, caching=%u, asid=%u, ioctl_us=%llu, lock_us=%llu",
+		      __get_str(dev), __entry->size, __entry->placement,
+		      __entry->caching, __entry->asid, __entry->ioctl_us,
+		      __entry->lock_us)
+);
+
 DECLARE_EVENT_CLASS(xe_vma,
 		    TP_PROTO(struct xe_vma *vma),
 		    TP_ARGS(vma),
Signature · 1 line
2.34.1
#49 lore raw

Re: [PATCH v2 26/33] drm/xe: Add tracepoint for xe_gem_create_ioctl

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Tracepoint metric `lock_us` incorrectly reports system uptime on early error paths and includes…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Tracepoint metric lock_us incorrectly reports system uptime on early error paths and includes lock-dropped backoff time during contention retries. --

--- Patch [26]: [PATCH v2 26/33] drm/xe: Add tracepoint for xe_gem_create_ioctl --- commit 245e602165eaf1749a023b1941ce834af1e88d5e
Author: Matthew Brost <[email protected]>

drm/xe: Add tracepoint for xe_gem_create_ioctl

This commit adds a tracepoint to xe_gem_create_ioctl to profile GEM creation, particularly tracking the time spent under the dma-resv lock versus total IOCTL time. The asid is captured at VM lookup time since the VM reference is dropped before the tracepoint fires.

Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
index d9226a3995fe4..c096f3dbd5289 100644

[ ... ]

drivers/gpu/drm/xe/xe_bo.c int xe_gem_create_ioctl(struct drm_device *dev, void *data, · line 4067

Quoted 7 lines
drivers/gpu/drm/xe/xe_bo.c +3 -0
@@ -4057,6 +4062,10 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 			if (err)
 				break;
 		}
+
+		if (lock_start == ktime_set(0, 0))
+			lock_start = ktime_get();

[Severity: Medium]
Will this include lock-dropped backoff time if there is lock contention?

If a retry occurs and the validation loop restarts, lock_start is not reset to the actual lock acquisition time because it is no longer 0.

drivers/gpu/drm/xe/xe_bo.c int xe_gem_create_ioctl(struct drm_device *dev, void *data, · line 4079

Quoted 9 lines
drivers/gpu/drm/xe/xe_bo.c +2 -0
+
 		bo = xe_bo_create_user(xe, vm, args->size, args->cpu_caching,
 				       bo_flags, have_prealloc ? &prealloc : NULL,
 				       &exec);
@@ -4067,6 +4076,7 @@ int xe_gem_create_ioctl(struct drm_device *dev, void *data,
 			break;
 		}
 	}
+	lock_ns = ktime_to_ns(ktime_sub(ktime_get(), lock_start));

[Severity: Medium]
Does this calculate a duration based on system uptime on early error paths?

If the validation loop breaks early (e.g. from an -EINTR in xe_vm_drm_exec_lock) before lock_start is set, lock_start will still be 0.

Quoted 3 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
 	if (have_prealloc)
 		ttm_pool_prealloc_fini(&xe->ttm.pool, &prealloc);
 	if (err)
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=26
author maintainer
#50 lore raw

PATCH v2 27/33 drm/xe: Add IOVA-based xe_res_cursor variant

In reply to Matthew Brost

Introduce xe_res_first_iova() to walk a buffer that was DMA mapped using the IOVA-based DMA API (dma_iova_try_alloc() and friends). Such a mapping is described by a struct dma_iova_state and is a single contiguous IOVA range, so the cursor walks it as one segment.

Add an is_iova flag to struct xe_res_cursor to distinguish this case. When set, xe_res_dma() returns the IOVA backing the current position (state->addr + cursor offset), xe_res_next() advances within the single contiguous segment, and xe_res_is_vram() returns false since IOVA mappings never point to same-device VRAM.

The flag is explicitly cleared in the existing xe_res_first(), xe_res_first_sg() and xe_res_first_dma() initializers so stale stack values cannot leak into the new path.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_res_cursor.h | 56 +++++++++++++++++++++++++++++-
 1 file changed, 55 insertions(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_res_cursor.h +55 -1
index 0522caafd89d..f0f1d699e16e 100644
@@ -24,6 +24,7 @@
 #ifndef _XE_RES_CURSOR_H_
 #define _XE_RES_CURSOR_H_
 
+#include <linux/dma-mapping.h>
 #include <linux/scatterlist.h>
 
 #include <drm/drm_pagemap.h>
@@ -67,6 +68,12 @@ struct xe_res_cursor {
 	u64 dma_start;
 	/** @dma_seg_size: Size of the current DMA segment. */
 	u64 dma_seg_size;
+	/**
+	 * @is_iova: Whether the cursor walks a single contiguous IOVA
+	 * mapping described by a struct dma_iova_state. When true, the
+	 * DMA address is @dma_start + @start and never points to VRAM.
+	 */
+	bool is_iova;
 };
 
 static struct gpu_buddy *xe_res_get_buddy(struct ttm_resource *res)
@@ -93,6 +100,7 @@ static inline void xe_res_first(struct ttm_resource *res,
 {
 	cur->sgl = NULL;
 	cur->dma_addr = NULL;
+	cur->is_iova = false;
 	if (!res)
 		goto fallback;
 
@@ -224,6 +232,7 @@ static inline void xe_res_first_sg(const struct sg_table *sg,
 	cur->dma_addr = NULL;
 	cur->sgl = sg->sgl;
 	cur->mem_type = XE_PL_TT;
+	cur->is_iova = false;
 	__xe_res_sg_next(cur);
 }
 
@@ -255,6 +264,41 @@ static inline void xe_res_first_dma(const struct drm_pagemap_addr *dma_addr,
 	__xe_res_dma_next(cur);
 	cur->sgl = NULL;
 	cur->mem_type = XE_PL_TT;
+	cur->is_iova = false;
+}
+
+/**
+ * xe_res_first_iova() - initialize a xe_res_cursor with a dma_iova_state
+ *
+ * @state: struct dma_iova_state describing a single contiguous IOVA mapping
+ * @start: Start of the range
+ * @size: Size of the range
+ * @cur: cursor object to initialize
+ *
+ * Start walking over the range of an allocation that was DMA mapped using the
+ * IOVA-based DMA API (see dma_iova_try_alloc()). Such a mapping is a single
+ * contiguous IOVA range, so the cursor walks it as one segment. Subsequent
+ * calls to xe_res_next() advance within that range and xe_res_dma() returns
+ * the IOVA backing the current position. xe_res_is_vram() always returns false
+ * for such a cursor.
+ */
+static inline void xe_res_first_iova(struct dma_iova_state *state,
+				     u64 start, u64 size,
+				     struct xe_res_cursor *cur)
+{
+	XE_WARN_ON(!state);
+	XE_WARN_ON(start + size > dma_iova_size(state));
+
+	cur->node = NULL;
+	cur->start = start;
+	cur->remaining = size;
+	cur->size = size;
+	cur->dma_addr = NULL;
+	cur->sgl = NULL;
+	cur->dma_start = state->addr;
+	cur->dma_seg_size = dma_iova_size(state);
+	cur->mem_type = XE_PL_TT;
+	cur->is_iova = true;
 }
 
 /**
@@ -283,6 +327,13 @@ static inline void xe_res_next(struct xe_res_cursor *cur, u64 size)
 		return;
 	}
 
+	if (cur->is_iova) {
+		/* Single contiguous IOVA segment. */
+		cur->start += size;
+		cur->size = cur->remaining;
+		return;
+	}
+
 	if (cur->dma_addr) {
 		cur->start += size;
 		__xe_res_dma_next(cur);
@@ -334,7 +385,7 @@ static inline void xe_res_next(struct xe_res_cursor *cur, u64 size)
  */
 static inline u64 xe_res_dma(const struct xe_res_cursor *cur)
 {
-	if (cur->dma_addr)
+	if (cur->is_iova || cur->dma_addr)
 		return cur->dma_start + cur->start;
 	else if (cur->sgl)
 		return sg_dma_address(cur->sgl) + cur->start;
@@ -351,6 +402,9 @@ static inline u64 xe_res_dma(const struct xe_res_cursor *cur)
  */
 static inline bool xe_res_is_vram(const struct xe_res_cursor *cur)
 {
+	if (cur->is_iova)
+		return false;
+
 	if (cur->dma_addr)
 		return cur->dma_addr->proto == XE_INTERCONNECT_VRAM;
 
Signature · 1 line
2.34.1
#51 lore raw

Re: [PATCH v2 27/33] drm/xe: Add IOVA-based xe_res_cursor variant

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Unreachable dead code block added for IOVA cursor segment advancement in xe_res_next(). --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Unreachable dead code block added for IOVA cursor segment advancement in xe_res_next(). --

--- Patch [27]: [PATCH v2 27/33] drm/xe: Add IOVA-based xe_res_cursor variant --- commit 8700355546957aacaa7e49fc841d6e860b3ed9e7
Author: Matthew Brost <[email protected]>

drm/xe: Add IOVA-based xe_res_cursor variant

Introduce xe_res_first_iova() to walk a buffer that was DMA mapped using the IOVA-based DMA API. Such a mapping is described by a struct dma_iova_state and is a single contiguous IOVA range, so the cursor walks it as one segment.

Quoted 4 lines
drivers/gpu/drm/xe/xe_res_cursor.h +0 -0
index 0522caafd89d7..f0f1d699e16e5 100644

[ ... ]

Quoted 19 lines
drivers/gpu/drm/xe/xe_res_cursor.h +17 -0
@@ -255,6 +264,41 @@ static inline void xe_res_first_dma(const struct drm_pagemap_addr *dma_addr,
+static inline void xe_res_first_iova(struct dma_iova_state *state,
+				     u64 start, u64 size,
+				     struct xe_res_cursor *cur)
+{
+	XE_WARN_ON(!state);
+	XE_WARN_ON(start + size > dma_iova_size(state));
+
+	cur->node = NULL;
+	cur->start = start;
+	cur->remaining = size;
+	cur->size = size;
+	cur->dma_addr = NULL;
+	cur->sgl = NULL;
+	cur->dma_start = state->addr;
+	cur->dma_seg_size = dma_iova_size(state);
+	cur->mem_type = XE_PL_TT;
+	cur->is_iova = true;
 }

[ ... ]

Quoted 4 lines
drivers/gpu/drm/xe/xe_res_cursor.h +0 -0
@@ -283,6 +327,13 @@ static inline void xe_res_next(struct xe_res_cursor *cur, u64 size)
 	if (cur->size > size) {
 		cur->size -= size;
 		cur->start += size;

drivers/gpu/drm/xe/xe_res_cursor.h static inline void xe_res_next(struct xe_res_cursor *cur, u64 size) · line 339

Quoted 10 lines
drivers/gpu/drm/xe/xe_res_cursor.h +7 -0
 		return;
 	}
 
+	if (cur->is_iova) {
+		/* Single contiguous IOVA segment. */
+		cur->start += size;
+		cur->size = cur->remaining;
+		return;
+	}
+

[Severity: Low]
Is this block unreachable dead code?

In xe_res_first_iova(), the cursor is initialized as a single segment where cur->size is equal to cur->remaining:

    cur->remaining = size;
    cur->size = size;

When xe_res_next(cur, size) is called, it first subtracts size from cur->remaining. If the entire remaining segment is consumed, it returns early via if (!cur->remaining).

If only a portion of the segment is consumed, size is strictly less than cur->size. This causes the function to always take the early return path immediately above this newly added block:

    if (cur->size > size) {
        cur->size -= size;
        cur->start += size;
        return;
    }

Because an IOVA cursor is a single contiguous segment where cur->size and cur->remaining start perfectly synchronized, does it ever need to bypass these early returns and evaluate the if (cur->is_iova) block?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=27
author maintainer
#52 lore raw

PATCH v2 28/33 drm/xe: Use IOVA-based DMA mapping for eligible tt BOs

In reply to Matthew Brost

Add support for mapping a ttm_tt's pages using the IOVA-based DMA API (dma_iova_try_alloc()/link()/sync()/unlink()/free()) as an alternative to the scatter-gather list. An IOVA mapping is a single contiguous DMA range, which is cheaper to walk and program than an sg list.

Whether the IOVA API can be used depends on the IOMMU mode, so the decision is made once per tt at xe_ttm_tt_create() time and recorded in struct xe_ttm_tt::use_iova. To keep the scope limited, only device BOs that are not GGTT-mapped are eligible: dma-buf imports manage their own mapping and GGTT bindings are programmed directly from the sg list. dma_iova_try_alloc() may still fail, in which case the tt transparently falls back to the sg list, which remains the default.

The IOVA reservation is allocated at tt creation and freed at tt destroy. xe_tt_map_iova() links the tt pages (coalescing physically contiguous runs) and syncs the IOTLB; xe_tt_unmap_iova() unlinks them but keeps the reservation so the tt can be remapped without re-allocating IOVA space. xe_tt_map()/xe_tt_unmap() dispatch between the IOVA and sg backends, and xe_bo_dma_unmap_pinned() learns about the IOVA case.

Consumers that walked the sg list via xe_res_first_sg(xe_bo_sg()) now go through the new xe_res_first_tt() helper, which initializes either an IOVA cursor (xe_res_first_iova()) or an sg cursor depending on the tt's mapping mode.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/tests/xe_migrate.c |   6 +-
 drivers/gpu/drm/xe/xe_bo.c            | 239 ++++++++++++++++++++++++--
 drivers/gpu/drm/xe/xe_bo.h            |   5 +-
 drivers/gpu/drm/xe/xe_ggtt.c          |   2 +-
 drivers/gpu/drm/xe/xe_migrate.c       |  14 +-
 drivers/gpu/drm/xe/xe_pt.c            |   2 +-
 6 files changed, 237 insertions(+), 31 deletions(-)
drivers/gpu/drm/xe/tests/xe_migrate.c +3 -3
index 41de648b2c86..119134e67c89 100644
@@ -253,7 +253,7 @@ static void xe_migrate_sanity_test(struct xe_migrate *m, struct kunit *test,
 	if (xe_bo_is_vram(pt))
 		xe_res_first(pt->ttm.resource, 0, xe_bo_size(pt), &src_it);
 	else
-		xe_res_first_sg(xe_bo_sg(pt), 0, xe_bo_size(pt), &src_it);
+		xe_res_first_tt(pt->ttm.ttm, 0, xe_bo_size(pt), &src_it);
 
 	emit_pte(m, bb, NUM_KERNEL_PDE - 1, xe_bo_is_vram(pt), false,
 		 &src_it, XE_PAGE_SIZE, pt->ttm.resource);
@@ -385,12 +385,12 @@ static struct dma_fence *blt_copy(struct xe_tile *tile,
 	bool dst_is_vram = mem_type_is_vram(dst->mem_type);
 
 	if (!src_is_vram)
-		xe_res_first_sg(xe_bo_sg(src_bo), 0, size, &src_it);
+		xe_res_first_tt(src_bo->ttm.ttm, 0, size, &src_it);
 	else
 		xe_res_first(src, 0, size, &src_it);
 
 	if (!dst_is_vram)
-		xe_res_first_sg(xe_bo_sg(dst_bo), 0, size, &dst_it);
+		xe_res_first_tt(dst_bo->ttm.ttm, 0, size, &dst_it);
 	else
 		xe_res_first(dst, 0, size, &dst_it);
 
drivers/gpu/drm/xe/xe_bo.c +222 -17

334 lines not on this page · raw message

drivers/gpu/drm/xe/xe_bo.h +3 -2
index 18fcccd082b7..3e2eaefecb8f 100644
@@ -552,8 +552,9 @@ void xe_bo_dev_init(struct xe_bo_dev *bo_device);
 
 void xe_bo_dev_fini(struct xe_bo_dev *bo_device);
 
-struct sg_table *xe_bo_sg(struct xe_bo *bo);
-struct sg_table *xe_tt_sg(struct ttm_tt *tt);
+struct xe_res_cursor;
+void xe_res_first_tt(struct ttm_tt *tt, u64 start, u64 size,
+		     struct xe_res_cursor *cur);
 
 /*
  * xe_sg_segment_size() - Provides upper limit for sg segment size.
drivers/gpu/drm/xe/xe_ggtt.c +1 -1
index 8ec23862477f..4084bc183e06 100644
@@ -699,7 +699,7 @@ static void xe_ggtt_map_bo(struct xe_ggtt *ggtt, struct xe_ggtt_node *node,
 	if (!xe_bo_is_vram(bo) && !xe_bo_is_stolen(bo)) {
 		xe_assert(xe_bo_device(bo), bo->ttm.ttm);
 
-		for (xe_res_first_sg(xe_bo_sg(bo), 0, xe_bo_size(bo), &cur);
+		for (xe_res_first_tt(bo->ttm.ttm, 0, xe_bo_size(bo), &cur);
 		     cur.remaining; xe_res_next(&cur, XE_PAGE_SIZE))
 			ggtt->pt_ops->ggtt_set_pte(ggtt, end - cur.remaining,
 						   pte | xe_res_dma(&cur));
drivers/gpu/drm/xe/xe_migrate.c +7 -7
index c1e96f105648..80fff1b5b3aa 100644
@@ -1005,16 +1005,16 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		return ERR_PTR(-EINVAL);
 
 	if (!src_is_vram)
-		xe_res_first_sg(src_tt ? xe_tt_sg(src_tt) : xe_bo_sg(src_bo), 0, size, &src_it);
+		xe_res_first_tt(src_tt ? src_tt : src_bo->ttm.ttm, 0, size, &src_it);
 	else
 		xe_res_first(src, 0, size, &src_it);
 	if (!dst_is_vram)
-		xe_res_first_sg(xe_bo_sg(dst_bo), 0, size, &dst_it);
+		xe_res_first_tt(dst_bo->ttm.ttm, 0, size, &dst_it);
 	else
 		xe_res_first(dst, 0, size, &dst_it);
 
 	if (copy_system_ccs)
-		xe_res_first_sg(src_tt ? xe_tt_sg(src_tt) : xe_bo_sg(src_bo),
+		xe_res_first_tt(src_tt ? src_tt : src_bo->ttm.ttm,
 				xe_bo_ccs_pages_start(src_bo),
 				PAGE_ALIGN(xe_device_ccs_bytes(xe, size)),
 				&ccs_it);
@@ -1383,9 +1383,9 @@ int xe_migrate_ccs_rw_copy(struct xe_tile *tile, struct xe_exec_queue *q,
 
 	ctx = &xe->sriov.vf.ccs.contexts[read_write];
 
-	xe_res_first_sg(xe_bo_sg(src_bo), 0, size, &src_it);
+	xe_res_first_tt(src_bo->ttm.ttm, 0, size, &src_it);
 
-	xe_res_first_sg(xe_bo_sg(src_bo), xe_bo_ccs_pages_start(src_bo),
+	xe_res_first_tt(src_bo->ttm.ttm, xe_bo_ccs_pages_start(src_bo),
 			PAGE_ALIGN(xe_device_ccs_bytes(xe, size)),
 			&ccs_it);
 
@@ -1588,7 +1588,7 @@ struct dma_fence *xe_migrate_vram_copy_chunk(struct xe_bo *vram_bo, u64 vram_off
 	xe_assert(xe, !range_overflows(sysmem_offset, size, (u64)sysmem_bo->ttm.base.size));
 
 	xe_res_first(vram, vram_offset, size, &vram_it);
-	xe_res_first_sg(xe_bo_sg(sysmem_bo), sysmem_offset, size, &sysmem_it);
+	xe_res_first_tt(sysmem_bo->ttm.ttm, sysmem_offset, size, &sysmem_it);
 
 	while (size) {
 		u32 pte_flags = PTE_UPDATE_FLAG_IS_VRAM;
@@ -1800,7 +1800,7 @@ struct dma_fence *xe_migrate_clear(struct xe_migrate *m,
 		clear_only_system_ccs = true;
 
 	if (!clear_vram)
-		xe_res_first_sg(xe_bo_sg(bo), 0, xe_bo_size(bo), &src_it);
+		xe_res_first_tt(bo->ttm.ttm, 0, xe_bo_size(bo), &src_it);
 	else
 		xe_res_first(src, 0, xe_bo_size(bo), &src_it);
 
drivers/gpu/drm/xe/xe_pt.c +1 -1
index 7111b5d0f6aa..185d15fb0a97 100644
@@ -797,7 +797,7 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma,
 			xe_res_first(bo->ttm.resource, xe_vma_bo_offset(vma),
 				     xe_vma_size(vma), &curs);
 		else
-			xe_res_first_sg(xe_bo_sg(bo), xe_vma_bo_offset(vma),
+			xe_res_first_tt(bo->ttm.ttm, xe_vma_bo_offset(vma),
 					xe_vma_size(vma), &curs);
 	} else if (!range) {
 		curs.size = xe_vma_size(vma);
Signature · 1 line
2.34.1
author maintainer
#53 lore raw

PATCH v2 29/33 drm/xe: Add per-device dependency scheduler for IOVA defrag finalize

In reply to Matthew Brost

A future patch performs the destination DMA mapping of a defrag move in two stages when the BO is IOVA mapped: a temporary contiguous mapping of just tht changed pages used for the GPU copy, followed by a post-copy finalize stet that tears that mapping down and links the new backing into the (transferred) IOVA reservation. The finalize step must run only after the GPU copy it depends on has completed, so it is dispatched as an xe_dep_job,

Add a per-device xe_dep_scheduler dedicated to these finalize jobs, created in xe_bo_defrag_init() (after the migrate contexts are up) and torn down in xe_bo_defrag_fini(). No users yet; the job itself is added in a later patch.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_bo.c           | 15 +++++++++++++++
 drivers/gpu/drm/xe/xe_device_types.h | 10 ++++++++++
 2 files changed, 25 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +15 -0
index f3d76ad9a105..ec034518e2ea 100644
@@ -25,6 +25,7 @@
 #include <trace/events/gpu_mem.h>
 
 #include "xe_device.h"
+#include "xe_dep_scheduler.h"
 #include "xe_dma_buf.h"
 #include "xe_drm_client.h"
 #include "xe_ggtt.h"
@@ -1300,6 +1301,10 @@ static void xe_bo_defrag_fini(void *arg)
 	struct xe_device *xe = arg;
 
 	disable_delayed_work_sync(&xe->mem.defrag.worker);
+	if (xe->mem.defrag.iova_sched) {
+		xe_dep_scheduler_fini(xe->mem.defrag.iova_sched);
+		xe->mem.defrag.iova_sched = NULL;
+	}
 }
 
 /**
@@ -1336,6 +1341,16 @@ void xe_bo_defrag_init_early(struct xe_device *xe)
  */
 int xe_bo_defrag_init(struct xe_device *xe)
 {
+#define XE_BO_MAX_IOVA_DEFRAG_JOBS	16	/* Picking a reasonable value */
+	struct xe_dep_scheduler *iova_sched;
+
+	iova_sched = xe_dep_scheduler_create(xe, NULL, "xe_iova_defrag",
+					     XE_BO_MAX_IOVA_DEFRAG_JOBS);
+	if (IS_ERR(iova_sched))
+		return PTR_ERR(iova_sched);
+	xe->mem.defrag.iova_sched = iova_sched;
+#undef XE_BO_MAX_IOVA_DEFRAG_JOBS
+
 	return devm_add_action_or_reset(xe->drm.dev, xe_bo_defrag_fini, xe);
 }
 
drivers/gpu/drm/xe/xe_device_types.h +10 -0
index 6e68add4d652..b55d549d5e30 100644
@@ -38,6 +38,7 @@
 
 struct drm_pagemap_shrinker;
 struct intel_display;
+struct xe_dep_scheduler;
 struct intel_dg_nvm_dev;
 struct xe_ggtt;
 struct xe_i2c;
@@ -325,6 +326,15 @@ struct xe_device {
 			 * @mem.defrag.worker, in milliseconds.
 			 */
 			unsigned int interval_ms;
+			/**
+			 * @mem.defrag.iova_sched: Dependency scheduler used to
+			 * run the post-copy IOVA finalize job for defrag moves
+			 * of IOVA-mapped BOs. The job tears down the temporary
+			 * copy-step IOVA mapping and links the new backing into
+			 * the (transferred) reservation once the GPU copy that
+			 * its dependencies gate has completed.
+			 */
+			struct xe_dep_scheduler *iova_sched;
 		} defrag;
 	} mem;
 
Signature · 1 line
2.34.1
#54 lore raw

Re: [PATCH v2 29/33] drm/xe: Add per-device dependency scheduler for IOVA defrag finalize

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] If `xe_dep_scheduler_create()` fails, `xe_bo_defrag_init()` returns early and skips registering…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] If xe_dep_scheduler_create() fails, xe_bo_defrag_init() returns early and skips registering the xe_bo_defrag_fini() devm action. This leaves the xe->mem.defrag.worker active, which can result in a use-after-free when the device struct is freed on probe failure. --

--- Patch [29]: [PATCH v2 29/33] drm/xe: Add per-device dependency scheduler for IOVA defrag finalize --- commit 5ebeeff5cf94ad2658593973ac47acd35100efa7
Author: Matthew Brost <[email protected]>
Subject: drm/xe: Add per-device dependency scheduler for IOVA defrag finalize

This patch adds a per-device xe_dep_scheduler dedicated to IOVA defrag finalize jobs. It initializes this scheduler in xe_bo_defrag_init() and cleans it up in xe_bo_defrag_fini().

Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
index f3d76ad9a1050..ec034518e2eae 100644

[ ... ]

drivers/gpu/drm/xe/xe_bo.c void xe_bo_defrag_init_early(struct xe_device *xe) · line 1350

Quoted 11 lines
drivers/gpu/drm/xe/xe_bo.c +7 -0
@@ -1336,6 +1341,16 @@ void xe_bo_defrag_init_early(struct xe_device *xe)
  */
 int xe_bo_defrag_init(struct xe_device *xe)
 {
+#define XE_BO_MAX_IOVA_DEFRAG_JOBS	16	/* Picking a reasonable value */
+	struct xe_dep_scheduler *iova_sched;
+
+	iova_sched = xe_dep_scheduler_create(xe, NULL, "xe_iova_defrag",
+					     XE_BO_MAX_IOVA_DEFRAG_JOBS);
+	if (IS_ERR(iova_sched))
+		return PTR_ERR(iova_sched);

[Severity: High]
If xe_dep_scheduler_create() fails, returning early here skips the registration of the xe_bo_defrag_fini() devm action below.

Does this leave the xe->mem.defrag.worker active on this probe error path? If the worker was already queued during earlier device initialization, it appears this could result in a use-after-free when the timer fires and the worker accesses the freed xe_device structure.

Quoted 5 lines
drivers/gpu/drm/xe/xe_bo.c +3 -0
+	xe->mem.defrag.iova_sched = iova_sched;
+#undef XE_BO_MAX_IOVA_DEFRAG_JOBS
+
 	return devm_add_action_or_reset(xe->drm.dev, xe_bo_defrag_fini, xe);
 }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=29
author maintainer
#55 lore raw

PATCH v2 30/33 drm/xe: Add packed copy-step IOVA mapping for defrag

In reply to Matthew Brost

Add machinery to map only changed pages during defrag when the destination TT is IOVA-mapped. Only the non-borrowed (freshly allocated) pages need copying; borrowed pages are aliases from the old TT.

Add xe_tt_iova_copy_init() to diff new vs old TT folios and record changed-page runs. Add xe_tt_map_iova_copy() to link those changed pages packed contiguously at the front of the new TT's IOVA reservation, synced and ready for GPU blits.

No users yet; wired up in later patches.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_bo.c | 364 +++++++++++++++++++++++++++++++++++++
 1 file changed, 364 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +364 -0
index ec034518e2ea..16d9e135cba2 100644
@@ -531,6 +531,31 @@ static void xe_tt_unaccount_iova(struct xe_device *xe,
 		}
 	}
 }
+
+/*
+ * xe_tt_account_iova_pages - account a tt's pages from a full folio walk
+ * @xe: the xe device
+ * @xe_tt: the xe_ttm_tt to account (its @iova_dma_pages must start zeroed)
+ *
+ * Walk @xe_tt's pages one folio at a time (the same head-page order walk used
+ * by xe_tt_map_iova()) and account each into xe->mem.dma_mapped_pages[] and
+ * @xe_tt->iova_dma_pages[]. Used to (re)derive accounting that exactly matches
+ * the pages a tt is actually mapped with.
+ */
+__maybe_unused
+static void xe_tt_account_iova_pages(struct xe_device *xe,
+				     struct xe_ttm_tt *xe_tt)
+{
+	struct ttm_tt *tt = &xe_tt->ttm;
+	unsigned long i = 0;
+
+	while (i < tt->num_pages) {
+		unsigned int order = ttm_pool_page_order_nodma(tt->pages[i]);
+
+		xe_tt_account_iova_add(xe, xe_tt, order);
+		i += 1UL << order;
+	}
+}
 #else
 static void xe_tt_account_iova_add(struct xe_device *xe,
 				   struct xe_ttm_tt *xe_tt, unsigned int order)
@@ -541,6 +566,12 @@ static void xe_tt_unaccount_iova(struct xe_device *xe,
 				 struct xe_ttm_tt *xe_tt)
 {
 }
+
+__maybe_unused
+static void xe_tt_account_iova_pages(struct xe_device *xe,
+				     struct xe_ttm_tt *xe_tt)
+{
+}
 #endif
 
 static int xe_tt_map_sg(struct xe_device *xe, struct ttm_tt *tt)
@@ -714,6 +745,339 @@ void xe_res_first_tt(struct ttm_tt *tt, u64 start, u64 size,
 		xe_res_first_sg(xe_tt->sg, start, size, cur);
 }
 
+/**
+ * struct xe_tt_iova_run - A contiguous run of changed pages in a defrag move
+ * @start: First page offset (natural ttm_tt index) of the run.
+ * @count: Number of pages in the run.
+ *
+ * Runs are folio aligned: defrag borrows whole beneficial-order chunks, so a
+ * folio is either entirely borrowed (unchanged) or entirely freshly allocated
+ * (changed), never split across the boundary.
+ */
+struct xe_tt_iova_run {
+	pgoff_t start;
+	pgoff_t count;
+};
+
+/**
+ * struct xe_tt_iova_link - A snapshot folio link descriptor for defrag relink
+ * @phys: Physical address of the folio's head page.
+ * @offset: Natural byte offset into the reservation to link the folio at.
+ * @len: Byte length of the folio (PAGE_SIZE << order).
+ *
+ * Captured at copy-init time while the destination tt's page array is valid and
+ * stable under the BO lock. The post-copy finalize job runs asynchronously,
+ * after TTM may have mutated (or torn down) the destination tt page array, so
+ * it relinks purely from these snapshots and never dereferences that array.
+ */
+struct xe_tt_iova_link {
+	phys_addr_t phys;
+	size_t offset;
+	size_t len;
+};
+
+/**
+ * struct xe_tt_iova_copy - Changed-page subset of an IOVA defrag move
+ * @runs: Array of changed-page runs, in ascending natural tt offset.
+ * @nr_runs: Number of entries in @runs.
+ * @links: Per-folio link snapshots covering the changed runs, in ascending
+ *         natural offset. Drives the asynchronous finalize relink so it never
+ *         touches the destination tt's (possibly mutated) page array.
+ * @nr_links: Number of entries in @links.
+ * @nr_changed: Total number of changed pages (sum of the run counts). This is
+ *              also the size, in pages, of the packed copy-step IOVA mapping
+ *              produced by xe_tt_map_iova_copy().
+ */
+struct xe_tt_iova_copy {
+	struct xe_tt_iova_run *runs;
+	unsigned int nr_runs;
+	struct xe_tt_iova_link *links;
+	unsigned int nr_links;
+	pgoff_t nr_changed;
+};
+
+/*
+ * xe_tt_iova_walk_changed - Walk a defrag move's changed-page runs.
+ *
+ * Iterate @new_tt one folio at a time (folio order read from TTM via
+ * ttm_pool_page_order_nodma()) and compare each folio's head page against
+ * @old_tt at the same natural offset. Pages the new backing borrowed from the
+ * old tt alias the same struct page and are unchanged; freshly allocated pages
+ * differ. Coalesce contiguous changed folios into runs and, for each run,
+ * invoke @fn. Returns the total number of changed pages.
+ */
+__maybe_unused
+static pgoff_t xe_tt_iova_walk_changed(struct ttm_tt *new_tt,
+				       struct ttm_tt *old_tt,
+				       void (*fn)(pgoff_t start, pgoff_t count,
+						  void *arg),
+				       void *arg)
+{
+	pgoff_t i = 0, n = new_tt->num_pages;
+	pgoff_t run_start = 0, nr_changed = 0;
+	bool in_run = false;
+
+	while (i < n) {
+		unsigned int order = ttm_pool_page_order_nodma(new_tt->pages[i]);
+		pgoff_t nr = 1UL << order;
+		bool changed = new_tt->pages[i] != old_tt->pages[i];
+
+		if (changed) {
+			if (!in_run) {
+				run_start = i;
+				in_run = true;
+			}
+			nr_changed += nr;
+		} else if (in_run) {
+			if (fn)
+				fn(run_start, i - run_start, arg);
+			in_run = false;
+		}
+
+		i += nr;
+	}
+
+	if (in_run && fn)
+		fn(run_start, n - run_start, arg);
+
+	return nr_changed;
+}
+
+struct xe_tt_iova_run_collect {
+	struct ttm_tt *new_tt;
+	struct xe_tt_iova_run *runs;
+	unsigned int nr_runs;
+	unsigned int runs_cap;
+	struct xe_tt_iova_link *links;
+	unsigned int nr_links;
+	unsigned int links_cap;
+	int err;
+};
+
+/* Geometrically grow @arr (element size @elem) to hold at least @need slots. */
+__maybe_unused
+static int xe_tt_iova_grow(void **arr, unsigned int *cap, unsigned int need,
+			   size_t elem)
+{
+	unsigned int new_cap;
+	void *p;
+
+	if (need <= *cap)
+		return 0;
+
+	new_cap = *cap ? *cap * 2 : 64;
+	if (new_cap < need)
+		new_cap = need;
+
+	p = krealloc_array(*arr, new_cap, elem, GFP_KERNEL);
+	if (!p)
+		return -ENOMEM;
+
+	*arr = p;
+	*cap = new_cap;
+	return 0;
+}
+
+__maybe_unused
+static void xe_tt_iova_store_run(pgoff_t start, pgoff_t count, void *arg)
+{
+	struct xe_tt_iova_run_collect *c = arg;
+	pgoff_t i = start, end = start + count;
+
+	if (c->err)
+		return;
+
+	if (xe_tt_iova_grow((void **)&c->runs, &c->runs_cap, c->nr_runs + 1,
+			    sizeof(*c->runs))) {
+		c->err = -ENOMEM;
+		return;
+	}
+	c->runs[c->nr_runs].start = start;
+	c->runs[c->nr_runs].count = count;
+	c->nr_runs++;
+
+	while (i < end) {
+		struct page *page = c->new_tt->pages[i];
+		unsigned int order = ttm_pool_page_order_nodma(page);
+
+		if (xe_tt_iova_grow((void **)&c->links, &c->links_cap,
+				    c->nr_links + 1, sizeof(*c->links))) {
+			c->err = -ENOMEM;
+			return;
+		}
+		c->links[c->nr_links].phys = page_to_phys(page);
+		c->links[c->nr_links].offset = (size_t)i << PAGE_SHIFT;
+		c->links[c->nr_links].len = PAGE_SIZE << order;
+		c->nr_links++;
+
+		i += 1UL << order;
+	}
+}
+
+/**
+ * xe_tt_iova_copy_init() - Compute the changed-page subset of a defrag move
+ * @new_tt: The freshly (re)allocated destination tt.
+ * @old_tt: The stashed source tt (bo->defrag_old_tt).
+ * @copy: Output description of the changed pages.
+ *
+ * Diff @new_tt against @old_tt and record the runs of changed (non-borrowed)
+ * pages into @copy, along with a per-folio snapshot of their physical
+ * addresses, natural offsets and lengths. The result drives both the packed
+ * copy-step IOVA mapping (xe_tt_map_iova_copy()) and the post-copy finalize
+ * that relinks the new backing into the IOVA reservation. The finalize runs
+ * asynchronously, after TTM may have mutated @new_tt->pages, so it consumes the
+ * snapshot rather than re-reading the page array. @new_tt and @old_tt must
+ * describe the same BO (equal num_pages) and both be populated.
+ *
+ * The caller owns @copy and must release it with xe_tt_iova_copy_fini().
+ *
+ * Return: 0 on success, -ENOMEM on allocation failure.
+ */
+__maybe_unused
+static int xe_tt_iova_copy_init(struct ttm_tt *new_tt, struct ttm_tt *old_tt,
+				struct xe_tt_iova_copy *copy)
+{
+	struct xe_tt_iova_run_collect c = { .new_tt = new_tt };
+
+	memset(copy, 0, sizeof(*copy));
+
+	/*
+	 * Single pass: collect the changed runs and per-folio link snapshots
+	 * directly, growing the arrays on demand. Changed runs are few (the
+	 * borrowed majority is skipped), so the geometric reallocs are rare and
+	 * this avoids a second full O(num_pages) walk just to size the arrays.
+	 */
+	copy->nr_changed = xe_tt_iova_walk_changed(new_tt, old_tt,
+						   xe_tt_iova_store_run, &c);
+	if (c.err) {
+		kfree(c.runs);
+		kfree(c.links);
+		return c.err;
+	}
+	if (!c.nr_runs)
+		return 0;
+
+	copy->runs = c.runs;
+	copy->nr_runs = c.nr_runs;
+	copy->links = c.links;
+	copy->nr_links = c.nr_links;
+
+	return 0;
+}
+
+/**
+ * xe_tt_iova_copy_fini() - Release a struct xe_tt_iova_copy
+ * @copy: The copy description to release.
+ */
+__maybe_unused
+static void xe_tt_iova_copy_fini(struct xe_tt_iova_copy *copy)
+{
+	kfree(copy->runs);
+	kfree(copy->links);
+	copy->runs = NULL;
+	copy->links = NULL;
+	copy->nr_runs = 0;
+	copy->nr_links = 0;
+	copy->nr_changed = 0;
+}
+
+/**
+ * xe_tt_map_iova_copy() - DMA map the changed-page subset for a defrag copy
+ * @xe: The xe device.
+ * @new_tt: The destination tt (must be IOVA eligible, use_iova == true).
+ * @copy: The changed-page description from xe_tt_iova_copy_init().
+ *
+ * Link only the changed pages of @new_tt into its IOVA reservation, packed
+ * contiguously at the front of the range ([0, @copy->nr_changed) pages). This
+ * provides a compact, contiguous destination for the GPU defrag copy without
+ * mapping the (majority) borrowed pages, and lets the whole temporary mapping
+ * later be torn down with a single dma_iova_destroy().
+ *
+ * The reservation's remaining tail is left unlinked. This does not establish
+ * the tt's final mapping (@iova_linked stays false); the post-copy finalize is
+ * responsible for that.
+ *
+ * Return: 0 on success, negative error code on failure (any partial linkage is
+ * unwound).
+ */
+__maybe_unused
+static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt,
+			       const struct xe_tt_iova_copy *copy)
+{
+	struct xe_ttm_tt *xe_tt = container_of(new_tt, struct xe_ttm_tt, ttm);
+	struct device *dev = xe->drm.dev;
+	size_t mapped = 0;
+	unsigned int r;
+	int ret;
+
+	xe_assert(xe, xe_tt->use_iova);
+	xe_assert(xe, !xe_tt->iova_linked);
+
+	for (r = 0; r < copy->nr_runs; r++) {
+		pgoff_t i = copy->runs[r].start;
+		pgoff_t end = i + copy->runs[r].count;
+
+		while (i < end) {
+			struct page *page = new_tt->pages[i];
+			unsigned int order = ttm_pool_page_order_nodma(page);
+			size_t len = PAGE_SIZE << order;
+
+			ret = dma_iova_link(dev, &xe_tt->iova,
+					    page_to_phys(page), mapped, len,
+					    DMA_BIDIRECTIONAL,
+					    DMA_ATTR_SKIP_CPU_SYNC);
+			if (ret)
+				goto err_unlink;
+
+			mapped += len;
+			i += 1UL << order;
+		}
+	}
+
+	ret = dma_iova_sync(dev, &xe_tt->iova, 0, mapped);
+	if (ret)
+		goto err_unlink;
+
+	return 0;
+
+err_unlink:
+	if (mapped)
+		dma_iova_unlink(dev, &xe_tt->iova, 0, mapped,
+				DMA_BIDIRECTIONAL, DMA_ATTR_SKIP_CPU_SYNC);
+	return ret;
+}
+
+/*
+ * xe_tt_iova_transfer - Transfer a full IOVA reservation old_tt -> new_tt.
+ *
+ * Move the (fully linked) IOVA reservation from the old tt to the new tt,
+ * leaving the old tt without a reservation so its destroy path does not double
+ * free. The new tt keeps its existing reservation snapshotted in the finalize
+ * job for teardown. Called at the defrag finalize job's point of no failure.
+ *
+ * Also rebalance the per-order DMA-page accounting: drop the old tt's
+ * accounting and account the pages the new tt actually ends up mapped with.
+ * The finalize relink leaves the new tt's mapping equal to new_tt->pages
+ * (borrowed pages alias the old tt; changed runs use the new pages, which may
+ * have a different order distribution), so deriving the accounting from a full
+ * folio walk of the new tt keeps xe->mem.dma_mapped_pages[] exact while the new
+ * mapping lives, rather than carrying the old tt's stale distribution.
+ */
+__maybe_unused
+static void xe_tt_iova_transfer(struct xe_device *xe,
+				struct xe_ttm_tt *new_tt,
+				struct xe_ttm_tt *old_tt)
+{
+	new_tt->iova = old_tt->iova;
+	new_tt->iova_linked = true;
+
+	xe_tt_unaccount_iova(xe, old_tt);
+	xe_tt_account_iova_pages(xe, new_tt);
+
+	old_tt->iova_linked = false;
+	old_tt->use_iova = false;
+}
+
 /*
  * Account ttm pages against the device shrinker's shrinkable and
  * purgeable counts.
Signature · 1 line
2.34.1
author maintainer
#56 lore raw

PATCH v2 31/33 drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies

In reply to Matthew Brost

A defrag move with IOVA enabled only DMA-maps the changed (non-borrowed) pages of the new tt, packed contiguously into the front of its IOVA reservation (xe_tt_map_iova_copy()). The GPU copy therefore has an asymmetric geometry: the source is read from the old tt at its natural page offsets (via the old tt's full IOVA), while the destination only exposes the changed pages, packed.

Teach __xe_migrate_copy() about this packed-destination mode:

- Add a @dst_packed_pages argument. When non-zero the destination tt cursor walks just that many packed pages and the source (not the destination) drives the chunk size, so the smaller packed prefix never clamps the source walk.

- Size the destination PTE update for each chunk from the changed-page count (xe_migrate_count_changed_pages()) rather than the full chunk, and skip the destination PTEs entirely for an all-borrowed chunk.

- Extend emit_copy_chunks() with a @pack_dst flag so each emitted run is blitted from its natural source offset into a running packed destination offset that only advances for changed runs. The all-borrowed skip leaves the packed destination cursor untouched.

Expose this via xe_migrate_copy_defrag_iova(). It is only valid when the BO needs no CCS pass, since the separate CCS pass migrates aux state for all pages and would require the full destination mapping. The existing xe_migrate_copy_defrag() (full destination mapping) remains the path for CCS BOs and the non-IOVA fallback.

No users yet; xe_bo_move() is wired to this path in a later patch.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_migrate.c | 307 ++++++++++++++++++++++++++++++--
 drivers/gpu/drm/xe/xe_migrate.h |   8 +
 2 files changed, 296 insertions(+), 19 deletions(-)
drivers/gpu/drm/xe/xe_migrate.c +288 -19
index 80fff1b5b3aa..f285a2a5a075 100644
@@ -811,6 +811,64 @@ static void emit_copy(struct xe_gt *gt, struct xe_bb *bb,
 		emit_xy_fast_copy(gt, bb, src_ofs, dst_ofs, size, pitch);
 }
 
+/*
+ * xe_migrate_count_changed_pages() - Count non-borrowed pages in a defrag chunk.
+ * @src_tt: Source (old) tt providing the page pointers.
+ * @dst_tt: Destination (new) tt providing the page pointers.
+ * @start_page: Page offset of this L0 chunk within the tt page arrays.
+ * @npages: Number of pages in the L0 chunk.
+ *
+ * A defragmentation move may harvest already beneficial-order chunks from the
+ * old tt, so the new tt references (borrows) some of the same physical pages.
+ * Count the pages that actually differ (src page != dst page) within the chunk.
+ *
+ * Return: The number of non-borrowed (changed) pages.
+ */
+static u32 xe_migrate_count_changed_pages(struct ttm_tt *src_tt,
+					  struct ttm_tt *dst_tt,
+					  pgoff_t start_page, u32 npages)
+{
+	u32 i, nchg = 0;
+
+	for (i = 0; i < npages; i++)
+		if (src_tt->pages[start_page + i] !=
+		    dst_tt->pages[start_page + i])
+			nchg++;
+
+	return nchg;
+}
+
+/*
+ * xe_migrate_leading_borrowed_pages() - Count the run of borrowed pages at the
+ * head of the remaining copy.
+ * @src_tt: Source (old) tt providing the page pointers.
+ * @dst_tt: Destination (new) tt providing the page pointers.
+ * @start_page: Page offset of the next page to copy within the tt page arrays.
+ * @max_pages: Pages left to copy.
+ *
+ * A defragmentation move borrows the already beneficial-order chunks from the
+ * old tt, so the new tt references (borrows) the same physical pages for the
+ * unchanged majority of a large object. Those pages alias on both the source
+ * and destination GPU mappings and need no copy. Count how many consecutive
+ * pages starting at @start_page are borrowed so the caller can advance past the
+ * whole run at once instead of rediscovering it one L0 chunk at a time.
+ *
+ * Return: The number of leading borrowed (src page == dst page) pages, capped
+ * at @max_pages.
+ */
+static u32 xe_migrate_leading_borrowed_pages(struct ttm_tt *src_tt,
+					     struct ttm_tt *dst_tt,
+					     pgoff_t start_page, u32 max_pages)
+{
+	u32 n = 0;
+
+	while (n < max_pages &&
+	       src_tt->pages[start_page + n] == dst_tt->pages[start_page + n])
+		n++;
+
+	return n;
+}
+
 /*
  * emit_copy_chunks() - Emit the data blits for one defrag L0 chunk, skipping
  * borrowed pages.
@@ -822,6 +880,9 @@ static void emit_copy(struct xe_gt *gt, struct xe_bb *bb,
  * @src_tt: Source (old) tt providing the page pointers.
  * @dst_tt: Destination (new) tt providing the page pointers.
  * @start_page: Page offset of this L0 chunk within the tt page arrays.
+ * @pack_dst: If true the destination only maps the changed pages packed
+ *	      contiguously, so the destination offset advances by changed-page
+ *	      runs instead of tracking the source page index.
  *
  * A defragmentation move may harvest already beneficial-order chunks from the
  * old tt, so the new tt references (borrows) some of the same physical pages.
@@ -830,16 +891,21 @@ static void emit_copy(struct xe_gt *gt, struct xe_bb *bb,
  * only blit runs of pages that actually differ (src page != dst page),
  * coalescing contiguous to-copy pages into a single blit.
  *
+ * When @pack_dst is set the source pages are read from their natural offsets
+ * (@src_L0_ofs + page index) while the destination pages were packed into a
+ * contiguous prefix, so the destination offset is a running counter that only
+ * advances for emitted (changed) runs.
+ *
  * Return: The number of blits emitted (or that would be emitted when @bb is
  * NULL).
  */
 static u32 emit_copy_chunks(struct xe_gt *gt, struct xe_bb *bb,
 			    u64 src_L0_ofs, u64 dst_L0_ofs, u32 src_L0,
 			    struct ttm_tt *src_tt, struct ttm_tt *dst_tt,
-			    pgoff_t start_page)
+			    pgoff_t start_page, bool pack_dst)
 {
 	u32 npages = src_L0 >> PAGE_SHIFT;
-	u32 i = 0, nblits = 0;
+	u32 i = 0, pack = 0, nblits = 0;
 
 	while (i < npages) {
 		u32 run = 0;
@@ -859,9 +925,71 @@ static u32 emit_copy_chunks(struct xe_gt *gt, struct xe_bb *bb,
 
 		if (bb)
 			emit_copy(gt, bb, src_L0_ofs + (u64)i * PAGE_SIZE,
-				  dst_L0_ofs + (u64)i * PAGE_SIZE,
+				  dst_L0_ofs +
+				  (u64)(pack_dst ? pack : i) * PAGE_SIZE,
 				  run * PAGE_SIZE, PAGE_SIZE);
 		nblits++;
+		pack += run;
+		i += run;
+	}
+
+	return nblits;
+}
+
+/*
+ * emit_ccs_copy_chunks() - Emit the indirect CCS aux copies for one packed
+ * defrag L0 chunk, skipping borrowed pages.
+ * @gt: The GT.
+ * @bb: The batch buffer to emit into, or NULL to only count the copies needed.
+ * @src_L0_ofs: GPU offset of the source pages mapped (comp PAT) for this chunk.
+ * @dst_L0_ofs: GPU offset of the destination pages mapped (comp PAT) for this
+ *	        chunk (the packed changed-page prefix).
+ * @src_L0: Byte size of the L0 chunk (page aligned).
+ * @src_tt: Source (old) tt providing the page pointers.
+ * @dst_tt: Destination (new) tt providing the page pointers.
+ * @start_page: Page offset of this L0 chunk within the tt page arrays.
+ *
+ * Companion to emit_copy_chunks() for the packed defrag CCS pass. The flat-CCS
+ * aux state is migrated indirectly through the data-page mappings, so it
+ * follows the physical page regardless of the (packed) GPU VA layout. Borrowed
+ * pages alias the same physical page on both sides, so their aux is already
+ * identical and is skipped; only the changed runs are copied, from the source
+ * at its natural offset to the destination at its packed offset.
+ *
+ * Return: The number of CCS copies emitted (or that would be emitted when @bb
+ * is NULL).
+ */
+static u32 emit_ccs_copy_chunks(struct xe_gt *gt, struct xe_bb *bb,
+				u64 src_L0_ofs, u64 dst_L0_ofs, u32 src_L0,
+				struct ttm_tt *src_tt, struct ttm_tt *dst_tt,
+				pgoff_t start_page)
+{
+	u32 npages = src_L0 >> PAGE_SHIFT;
+	u32 i = 0, pack = 0, nblits = 0;
+
+	while (i < npages) {
+		u32 run = 0;
+
+		/* Skip a run of borrowed pages (src and dst alias). */
+		if (src_tt->pages[start_page + i] ==
+		    dst_tt->pages[start_page + i]) {
+			i++;
+			continue;
+		}
+
+		/* Coalesce a run of pages whose aux must be migrated. */
+		while (i + run < npages &&
+		       src_tt->pages[start_page + i + run] !=
+		       dst_tt->pages[start_page + i + run])
+			run++;
+
+		if (bb)
+			emit_copy_ccs(gt, bb,
+				      dst_L0_ofs + (u64)pack * PAGE_SIZE, true,
+				      src_L0_ofs + (u64)i * PAGE_SIZE, true,
+				      run * PAGE_SIZE);
+		nblits++;
+		pack += run;
 		i += run;
 	}
 
@@ -949,6 +1077,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 					   struct ttm_resource *src,
 					   struct ttm_resource *dst,
 					   struct ttm_tt *src_tt,
+					   pgoff_t dst_packed_pages,
 					   struct xe_migrate_copy_flags flags)
 {
 	struct xe_gt *gt = m->tile->primary_gt;
@@ -981,6 +1110,23 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 	 */
 	struct ttm_tt *dst_tt = dst_bo->ttm.ttm;
 	bool skip_borrowed = flags.defrag_copy && src_tt && dst_tt;
+	/*
+	 * Defrag-IOVA copy: the destination tt only maps the changed
+	 * (non-borrowed) pages, packed contiguously into the first
+	 * @dst_packed_pages pages of its IOVA. The source is read from the old
+	 * tt at natural offsets and each changed run is blitted into the packed
+	 * destination prefix.
+	 */
+	bool pack_dst = dst_packed_pages != 0;
+	/*
+	 * Packed defrag CCS pass: migrate the flat-CCS aux state of the changed
+	 * pages only, from the source at its natural offset to the destination
+	 * packed prefix. Borrowed pages share the physical page (and thus its
+	 * aux) on both sides, so they are skipped exactly like the data pass.
+	 */
+	bool ccs_pack = flags.defrag_ccs && pack_dst && src_tt && dst_tt;
+	/* Per-run, borrowed-skipping defrag emission (data or packed CCS). */
+	bool defrag_runs = skip_borrowed || ccs_pack;
 
 	/*
 	 * For decompression operation, always use the compression PAT index.
@@ -1009,7 +1155,9 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 	else
 		xe_res_first(src, 0, size, &src_it);
 	if (!dst_is_vram)
-		xe_res_first_tt(dst_bo->ttm.ttm, 0, size, &dst_it);
+		xe_res_first_tt(dst_bo->ttm.ttm, 0,
+				pack_dst ? (u64)dst_packed_pages << PAGE_SHIFT :
+				size, &dst_it);
 	else
 		xe_res_first(dst, 0, size, &dst_it);
 
@@ -1030,29 +1178,86 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		u32 pte_flags;
 		pgoff_t start_page = (xe_bo_size(src_bo) - size) >> XE_PTE_SHIFT;
 		u32 ndata_blits = 1;
+		u32 nchg = 0;
+		u64 dst_size;
 
 		bool usm = xe->info.has_usm;
 		u32 avail_pts = max_mem_transfer_per_pass(xe) / LEVEL0_PAGE_TABLE_ENCODE_SIZE;
 
+		/*
+		 * Defrag fast-forward: the new (destination) tt borrows the
+		 * unchanged beneficial-order chunks from the old tt, so for a
+		 * large object the bulk of the copy is borrowed pages that alias
+		 * the same memory on both sides. Skip the entire leading
+		 * borrowed run in a single cursor stride rather than discovering
+		 * it one (8 MB) L0 chunk at a time - each such pass would
+		 * otherwise pay xe_migrate_res_sizes() + pte_update_size() + a
+		 * full chunk walk only to copy nothing. The non-vram tt cursors
+		 * are normally advanced by emit_pte(); replicate that with
+		 * xe_res_next(). In pack_dst mode the destination only maps the
+		 * changed pages, so its cursor must not advance for borrowed
+		 * pages.
+		 */
+		if (defrag_runs) {
+			u32 brun = xe_migrate_leading_borrowed_pages(src_tt,
+								     dst_tt,
+								     start_page,
+								     size >> PAGE_SHIFT);
+
+			if (brun) {
+				u64 brun_bytes = (u64)brun << PAGE_SHIFT;
+
+				xe_res_next(&src_it, brun_bytes);
+				if (!pack_dst)
+					xe_res_next(&dst_it, brun_bytes);
+				size -= brun_bytes;
+				continue;
+			}
+		}
+
 		src_L0 = xe_migrate_res_sizes(m, &src_it);
 		dst_L0 = xe_migrate_res_sizes(m, &dst_it);
 
 		drm_dbg(&xe->drm, "Pass %u, sizes: %llu & %llu\n",
 			pass++, src_L0, dst_L0);
 
-		src_L0 = min(src_L0, dst_L0);
+		/*
+		 * In pack_dst mode the destination is the (smaller) packed
+		 * prefix, so it must not clamp the source chunk size; the
+		 * source drives the walk and the destination is sized
+		 * independently from the changed-page count below.
+		 */
+		if (!pack_dst)
+			src_L0 = min(src_L0, dst_L0);
 
 		pte_flags = src_is_vram ? PTE_UPDATE_FLAG_IS_VRAM : 0;
 		pte_flags |= use_comp_pat ? PTE_UPDATE_FLAG_IS_COMP_PTE : 0;
 		batch_size += pte_update_size(m, pte_flags, src, &src_it, &src_L0,
 					      &src_L0_ofs, &src_L0_pt, 0, 0,
 					      avail_pts);
+		/*
+		 * The source chunk size may have been clamped above, so derive
+		 * the packed destination size from the changed-page count of
+		 * the final chunk.
+		 */
+		if (pack_dst) {
+			nchg = xe_migrate_count_changed_pages(src_tt, dst_tt,
+							      start_page,
+							      src_L0 >> PAGE_SHIFT);
+			dst_size = (u64)nchg << PAGE_SHIFT;
+		} else {
+			dst_size = src_L0;
+		}
 		if (flags.copy_only_ccs) {
 			dst_L0_ofs = src_L0_ofs;
+		} else if (pack_dst && !nchg) {
+			/* All borrowed: no destination PTEs for this chunk. */
+			dst_L0_ofs = 0;
+			dst_L0_pt = 0;
 		} else {
 			pte_flags = dst_is_vram ? PTE_UPDATE_FLAG_IS_VRAM : 0;
 			batch_size += pte_update_size(m, pte_flags, dst,
-						      &dst_it, &src_L0,
+						      &dst_it, &dst_size,
 						      &dst_L0_ofs, &dst_L0_pt,
 						      0, avail_pts, avail_pts);
 		}
@@ -1072,26 +1277,31 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		 * is split into one command per run of non-borrowed pages, so
 		 * size for the number of runs actually needed.
 		 */
-		if (skip_borrowed)
+		if (defrag_runs)
 			ndata_blits = emit_copy_chunks(gt, NULL, 0, 0, src_L0,
-						       src_tt, dst_tt, start_page);
+						       src_tt, dst_tt, start_page,
+						       pack_dst);
 
 		/*
 		 * If every page in this chunk was borrowed there is nothing to
 		 * copy: don't build a batch or issue any GPU commands, just
 		 * advance the cursors past the chunk. The non-vram tt cursors
-		 * would otherwise be advanced by emit_pte().
+		 * would otherwise be advanced by emit_pte(). In pack_dst mode
+		 * the destination consumed no packed pages, so leave its cursor
+		 * untouched.
 		 */
-		if (skip_borrowed && !ndata_blits) {
+		if (defrag_runs && !ndata_blits) {
 			xe_res_next(&src_it, src_L0);
-			xe_res_next(&dst_it, src_L0);
+			if (!pack_dst)
+				xe_res_next(&dst_it, src_L0);
 			size -= src_L0;
 			continue;
 		}
 
 		batch_size += ((flags.copy_only_ccs || flags.defrag_ccs) ? 0 :
 			       ndata_blits * emit_copy_cmd_len(xe)) +
-			((needs_ccs_emit ? EMIT_COPY_CCS_DW : 0));
+			(ccs_pack ? ndata_blits * EMIT_COPY_CCS_DW :
+			 (needs_ccs_emit ? EMIT_COPY_CCS_DW : 0));
 
 		bb = xe_bb_new(gt, batch_size, usm);
 		if (IS_ERR(bb)) {
@@ -1111,7 +1321,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		else if (!flags.copy_only_ccs)
 			emit_pte(m, bb, dst_L0_pt, dst_is_vram,
 				 copy_system_ccs || flags.defrag_ccs,
-				 &dst_it, src_L0, dst);
+				 &dst_it, dst_size, dst);
 
 		if (copy_system_ccs)
 			emit_pte(m, bb, ccs_pt, false, false, &ccs_it, ccs_size, src);
@@ -1121,16 +1331,22 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 
 		if (skip_borrowed)
 			emit_copy_chunks(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0,
-					 src_tt, dst_tt, start_page);
+					 src_tt, dst_tt, start_page, pack_dst);
 		else if (!flags.copy_only_ccs && !flags.defrag_ccs)
 			emit_copy(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0, XE_PAGE_SIZE);
 
-		if (needs_ccs_emit)
+		if (ccs_pack) {
+			if (emit_ccs_copy_chunks(gt, bb, src_L0_ofs, dst_L0_ofs,
+						 src_L0, src_tt, dst_tt,
+						 start_page))
+				flush_flags = MI_FLUSH_DW_CCS;
+		} else if (needs_ccs_emit) {
 			flush_flags = xe_migrate_ccs_copy(m, bb, src_L0_ofs,
 							  IS_DGFX(xe) ? src_is_vram : src_is_pltt,
 							  dst_L0_ofs,
 							  IS_DGFX(xe) ? dst_is_vram : dst_is_pltt,
 							  src_L0, ccs_ofs, copy_ccs);
+		}
 
 		job = xe_bb_create_migration_job(m->q, bb,
 						 xe_migrate_batch_base(m, usm),
@@ -1218,7 +1434,7 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
 				  struct ttm_resource *dst,
 				  bool copy_only_ccs)
 {
-	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, NULL,
+	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, NULL, 0,
 				 (struct xe_migrate_copy_flags) {
 					 .copy_only_ccs = copy_only_ccs,
 				 });
@@ -1267,7 +1483,7 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 					 bool need_ccs)
 {
 	struct dma_fence *fence2, *fence =
-		__xe_migrate_copy(m, bo, bo, src, dst, src_tt,
+		__xe_migrate_copy(m, bo, bo, src, dst, src_tt, 0,
 				  (struct xe_migrate_copy_flags) {
 					 .defrag_copy = true,
 					 });
@@ -1275,7 +1491,7 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 	if (IS_ERR(fence) || !need_ccs)
 		return fence;
 
-	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_tt,
+	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_tt, 0,
 				   (struct xe_migrate_copy_flags) {
 				   .defrag_ccs = true,
 				   });
@@ -1286,6 +1502,59 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 	return fence2;
 }
 
+/**
+ * xe_migrate_copy_defrag_iova() - Copy a defrag move into a packed IOVA dst.
+ * @m: The migration context.
+ * @bo: The buffer object being defragmented.
+ * @src: The source (old) TTM resource.
+ * @dst: The destination (new) TTM resource.
+ * @src_tt: The stashed source (old) tt providing the page pointers, fully
+ *	    IOVA-mapped at natural offsets.
+ * @dst_packed_pages: Number of changed pages packed into the destination tt's
+ *	    IOVA prefix by xe_tt_map_iova_copy().
+ * @need_ccs: Whether the BO has flat-CCS aux state that must be migrated too.
+ *
+ * Defrag copy variant for IOVA-mapped BOs where the destination only maps the
+ * changed (non-borrowed) pages, packed contiguously into the first
+ * @dst_packed_pages of its IOVA. The source is read from the old tt at its
+ * natural offsets and each changed run is blitted into the packed destination
+ * prefix. When @need_ccs is set a second pass migrates the flat-CCS aux state
+ * of the same changed pages indirectly through their (packed) data mappings;
+ * borrowed pages share the physical page and its aux, so only the changed runs
+ * are touched.
+ *
+ * Return: Pointer to a dma_fence representing the last copy batch, or an error
+ * pointer on failure.
+ */
+struct dma_fence *xe_migrate_copy_defrag_iova(struct xe_migrate *m,
+					      struct xe_bo *bo,
+					      struct ttm_resource *src,
+					      struct ttm_resource *dst,
+					      struct ttm_tt *src_tt,
+					      pgoff_t dst_packed_pages,
+					      bool need_ccs)
+{
+	struct dma_fence *fence2, *fence =
+		__xe_migrate_copy(m, bo, bo, src, dst, src_tt, dst_packed_pages,
+				  (struct xe_migrate_copy_flags) {
+					 .defrag_copy = true,
+				 });
+
+	if (IS_ERR(fence) || !need_ccs)
+		return fence;
+
+	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_tt,
+				   dst_packed_pages,
+				   (struct xe_migrate_copy_flags) {
+					 .defrag_ccs = true,
+				   });
+	if (IS_ERR(fence2))
+		dma_fence_wait(fence, false);
+	dma_fence_put(fence);
+
+	return fence2;
+}
+
 /**
  * xe_migrate_resolve() - Resolve and decompress a buffer object if required.
  * @m: The migrate context
@@ -1301,7 +1570,7 @@ struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
 				     struct xe_bo *bo,
 				     struct ttm_resource *res)
 {
-	return __xe_migrate_copy(m, bo, bo, res, res, NULL,
+	return __xe_migrate_copy(m, bo, bo, res, res, NULL, 0,
 				 (struct xe_migrate_copy_flags) {
 					 .is_vram_resolve = true,
 				 });
drivers/gpu/drm/xe/xe_migrate.h +8 -0
index 4094ea1c19c1..da066cdf01bf 100644
@@ -141,6 +141,14 @@ struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
 					 struct ttm_tt *src_tt,
 					 bool need_ccs);
 
+struct dma_fence *xe_migrate_copy_defrag_iova(struct xe_migrate *m,
+					      struct xe_bo *bo,
+					      struct ttm_resource *src,
+					      struct ttm_resource *dst,
+					      struct ttm_tt *src_tt,
+					      pgoff_t dst_packed_pages,
+					      bool need_ccs);
+
 struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
 				     struct xe_bo *bo,
 				     struct ttm_resource *res);
Signature · 1 line
2.34.1
#57 lore raw

Re: [PATCH v2 31/33] drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] Overloading dst_packed_pages as a boolean flag causes GPU IOMMU faults when defragmentation yields 0 changed pages.

Pre-existing issues:
- [High] Mismatch between XE_PTE_SHIFT and PAGE_SHIFT causes out-of-bounds array indexing and assertion failures on architectures with 64K pages. --

--- Patch [31]: [PATCH v2 31/33] drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies --- commit b1d39bbd2548d9ad82a4ac12f4936dc748d93736
Author: Matthew Brost <[email protected]>
Subject: drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies

This commit adds a packed-destination mode to __xe_migrate_copy for defrag moves with IOVA enabled. It allows reading from the old tt at natural offsets and writing to the new tt at packed contiguous offsets. A new xe_migrate_copy_defrag_iova entry point is exposed for this functionality.

Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c +0 -0
index 0a3a1c5ca1938..35fd0253fef30 100644

[ ... ]

drivers/gpu/drm/xe/xe_migrate.c static u32 emit_copy_chunks(struct xe_gt *gt, struct xe_bb *bb, · line 954

Quoted 8 lines
drivers/gpu/drm/xe/xe_migrate.c +2 -1
@@ -883,9 +949,71 @@ static u32 emit_copy_chunks(struct xe_gt *gt, struct xe_bb *bb,
 
 		if (bb)
 			emit_copy(gt, bb, src_L0_ofs + (u64)i * PAGE_SIZE,
-				  dst_L0_ofs + (u64)i * PAGE_SIZE,
+				  dst_L0_ofs +
+				  (u64)(pack_dst ? pack : i) * PAGE_SIZE,
 				  run * PAGE_SIZE, PAGE_SIZE);

[Severity: High]
This isn't a bug introduced by this patch, but does passing PAGE_SIZE as the pitch to emit_copy overflow the 16-bit pitch limit on systems with 64KB CPU pages? The xe_gt_assert condition in emit_xy_fast_copy expects the pitch to be <= U16_MAX, which could cause a crash on architectures like ARM64 when defragmenting BOs.

Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c +3 -0
 		nblits++;
+		pack += run;
+		i += run;
+	}

[ ... ]

drivers/gpu/drm/xe/xe_migrate.c static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m, · line 1144

Quoted 12 lines
drivers/gpu/drm/xe/xe_migrate.c +8 -0
@@ -1005,6 +1134,23 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 	 */
 	struct ttm_tt *dst_tt = dst_bo->ttm.ttm;
 	bool skip_borrowed = flags.defrag_copy && src_tt && dst_tt;
+	/*
+	 * Defrag-IOVA copy: the destination tt only maps the changed
+	 * (non-borrowed) pages, packed contiguously into the first
+	 * @dst_packed_pages pages of its IOVA. The source is read from the old
+	 * tt at natural offsets and each changed run is blitted into the packed
+	 * destination prefix.
+	 */
+	bool pack_dst = dst_packed_pages != 0;

[Severity: High]
Can evaluating pack_dst this way lead to GPU IOMMU faults when a defragmentation move yields 0 changed pages?

If a BO defragmentation successfully borrows all pages, dst_packed_pages will be 0. This makes pack_dst false, which disables the packed CCS logic. During the CCS pass, the driver might then fall back to a full BO CCS copy via xe_migrate_ccs_copy(), assuming the destination is fully mapped. However, because the destination IOVA mapping only covers the 0 changed pages, the GPU would write CCS data to unmapped IOVA addresses.

Quoted 7 lines
drivers/gpu/drm/xe/xe_migrate.c +7 -0
+	/*
+	 * Packed defrag CCS pass: migrate the flat-CCS aux state of the changed
+	 * pages only, from the source at its natural offset to the destination
+	 * packed prefix. Borrowed pages share the physical page (and thus its
+	 * aux) on both sides, so they are skipped exactly like the data pass.
+	 */
+	bool ccs_pack = flags.defrag_ccs && pack_dst && src_tt && dst_tt;

[ ... ]

drivers/gpu/drm/xe/xe_migrate.c static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m, · line 1203

Quoted 3 lines
drivers/gpu/drm/xe/xe_migrate.c +0 -0
@@ -1054,29 +1202,86 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		u32 pte_flags;
 		pgoff_t start_page = (xe_bo_size(src_bo) - size) >> XE_PTE_SHIFT;

[Severity: High]
This isn't a bug introduced by this patch, but could the start_page calculation cause out-of-bounds array indexing on architectures with 64KB pages?

The index start_page is computed using XE_PTE_SHIFT (which is 4KB), but it is later used to index ttm_tt->pages which is sized based on PAGE_SHIFT. On a 64KB page system, the calculated index would be 16x larger than the array bounds, potentially leading to memory corruption during defragmentation.

Quoted 3 lines
drivers/gpu/drm/xe/xe_migrate.c +2 -0
 		u32 ndata_blits = 1;
+		u32 nchg = 0;
+		u64 dst_size;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=31
author maintainer
#58 lore raw

PATCH v2 32/33 drm/xe: Finalize defrag-IOVA moves with post-copy job

In reply to Matthew Brost

Wire up end-to-end defrag for IOVA-mapped BOs. A defrag move is eligible whenever the destination TT is IOVA mapped. The packed-copy step moves only changed pages; the IOVA reservation is transferred to the new backing so device-visible addresses are preserved, avoiding GPU rebind overhead.

Add xe_bo_defrag_use_iova() to select the IOVA defrag path. Add xe_bo_defrag_iova_copy() to drive the move: diff new vs stashed old TT, link changed pages packed, blit them, then queue a post-copy finalize job.

The finalize job tears down the temporary packed mapping, transfers the full old TT reservation to the new one, and relinks the copied pages at natural offsets.

Borrowed pages skip the copy and simply adopt the old reservation.

Since reservation transfer preserves device-visible addresses, xe_bo_trigger_rebind() skips drm_gpuvm_bo_evict() marking; GPU PTEs stay valid.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_bo.c | 302 +++++++++++++++++++++++++++++++++++--
 1 file changed, 288 insertions(+), 14 deletions(-)
drivers/gpu/drm/xe/xe_bo.c +288 -14
index 16d9e135cba2..0eee53f4d653 100644
@@ -13,6 +13,7 @@
 #include <drm/drm_dumb_buffers.h>
 #include <drm/drm_gem_ttm_helper.h>
 #include <drm/drm_managed.h>
+#include <drm/gpu_scheduler.h>
 #include <drm/ttm/ttm_backup.h>
 #include <drm/ttm/ttm_device.h>
 #include <drm/ttm/ttm_placement.h>
@@ -25,6 +26,7 @@
 #include <trace/events/gpu_mem.h>
 
 #include "xe_device.h"
+#include "xe_dep_job_types.h"
 #include "xe_dep_scheduler.h"
 #include "xe_dma_buf.h"
 #include "xe_drm_client.h"
@@ -542,7 +544,6 @@ static void xe_tt_unaccount_iova(struct xe_device *xe,
  * @xe_tt->iova_dma_pages[]. Used to (re)derive accounting that exactly matches
  * the pages a tt is actually mapped with.
  */
-__maybe_unused
 static void xe_tt_account_iova_pages(struct xe_device *xe,
 				     struct xe_ttm_tt *xe_tt)
 {
@@ -567,7 +568,6 @@ static void xe_tt_unaccount_iova(struct xe_device *xe,
 {
 }
 
-__maybe_unused
 static void xe_tt_account_iova_pages(struct xe_device *xe,
 				     struct xe_ttm_tt *xe_tt)
 {
@@ -806,7 +806,6 @@ struct xe_tt_iova_copy {
  * differ. Coalesce contiguous changed folios into runs and, for each run,
  * invoke @fn. Returns the total number of changed pages.
  */
-__maybe_unused
 static pgoff_t xe_tt_iova_walk_changed(struct ttm_tt *new_tt,
 				       struct ttm_tt *old_tt,
 				       void (*fn)(pgoff_t start, pgoff_t count,
@@ -855,7 +854,6 @@ struct xe_tt_iova_run_collect {
 };
 
 /* Geometrically grow @arr (element size @elem) to hold at least @need slots. */
-__maybe_unused
 static int xe_tt_iova_grow(void **arr, unsigned int *cap, unsigned int need,
 			   size_t elem)
 {
@@ -878,7 +876,6 @@ static int xe_tt_iova_grow(void **arr, unsigned int *cap, unsigned int need,
 	return 0;
 }
 
-__maybe_unused
 static void xe_tt_iova_store_run(pgoff_t start, pgoff_t count, void *arg)
 {
 	struct xe_tt_iova_run_collect *c = arg;
@@ -933,7 +930,6 @@ static void xe_tt_iova_store_run(pgoff_t start, pgoff_t count, void *arg)
  *
  * Return: 0 on success, -ENOMEM on allocation failure.
  */
-__maybe_unused
 static int xe_tt_iova_copy_init(struct ttm_tt *new_tt, struct ttm_tt *old_tt,
 				struct xe_tt_iova_copy *copy)
 {
@@ -969,7 +965,6 @@ static int xe_tt_iova_copy_init(struct ttm_tt *new_tt, struct ttm_tt *old_tt,
  * xe_tt_iova_copy_fini() - Release a struct xe_tt_iova_copy
  * @copy: The copy description to release.
  */
-__maybe_unused
 static void xe_tt_iova_copy_fini(struct xe_tt_iova_copy *copy)
 {
 	kfree(copy->runs);
@@ -1000,7 +995,6 @@ static void xe_tt_iova_copy_fini(struct xe_tt_iova_copy *copy)
  * Return: 0 on success, negative error code on failure (any partial linkage is
  * unwound).
  */
-__maybe_unused
 static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt,
 			       const struct xe_tt_iova_copy *copy)
 {
@@ -1047,6 +1041,129 @@ static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt,
 	return ret;
 }
 
+/**
+ * struct xe_iova_defrag_job - Post-copy finalize job for an IOVA defrag move.
+ *
+ * After the GPU defrag copy into the packed copy-step mapping completes, this
+ * dependency job runs the CPU-side IOMMU fixups at the point of no failure:
+ * tear down the temporary packed mapping, then per changed run unlink the old
+ * pages and link the new pages into the (transferred) reservation at their
+ * natural offset.
+ */
+struct xe_iova_defrag_job {
+	/** @dep: base generic dependency Xe job. */
+	struct xe_dep_job dep;
+	/** @xe: The xe device. */
+	struct xe_device *xe;
+	/** @new_tt: The destination tt being finalized (owns the reservation). */
+	struct ttm_tt *new_tt;
+	/**
+	 * @copy_iova: Snapshot of the new tt's original IOVA reservation that
+	 * held the temporary packed copy-step mapping. Destroyed by the job
+	 * (the new tt itself adopts the old tt's reservation at job creation).
+	 */
+	struct dma_iova_state copy_iova;
+	/** @copy_bytes: Byte length of the packed copy-step mapping. */
+	size_t copy_bytes;
+	/** @copy: Changed-page description; the job owns and frees @copy.runs. */
+	struct xe_tt_iova_copy copy;
+};
+
+static struct dma_fence *xe_iova_defrag_job_run(struct xe_dep_job *dep_job)
+{
+	struct xe_iova_defrag_job *job =
+		container_of(dep_job, typeof(*job), dep);
+	struct xe_device *xe = job->xe;
+	struct device *dev = xe->drm.dev;
+	struct xe_ttm_tt *xe_tt =
+		container_of(job->new_tt, struct xe_ttm_tt, ttm);
+	unsigned int r, l;
+
+	/*
+	 * Tear down the temporary packed copy-step mapping and free its
+	 * reservation. The new tt has already adopted the old tt's full
+	 * reservation (transferred at job creation), so this only touches the
+	 * snapshot taken before the transfer.
+	 *
+	 * This mapping was the GPU blit's write destination, so tear it down
+	 * with attrs 0 (not DMA_ATTR_SKIP_CPU_SYNC): for non-coherent or
+	 * bounced (SWIOTLB) mappings the device->CPU copy-back / cache sync of
+	 * the freshly copied data happens during unlink, exactly as
+	 * xe_tt_unmap_iova() does. Skipping it would silently drop the
+	 * defragmented data on those configs.
+	 */
+	if (job->copy_bytes)
+		dma_iova_destroy(dev, &job->copy_iova, job->copy_bytes,
+				 DMA_BIDIRECTIONAL, 0);
+	else
+		dma_iova_free(dev, &job->copy_iova);
+
+	/*
+	 * For each changed run, unlink the old pages (still linked from the
+	 * adopted reservation).
+	 */
+	for (r = 0; r < job->copy.nr_runs; r++) {
+		size_t off = (size_t)job->copy.runs[r].start << PAGE_SHIFT;
+		size_t len = (size_t)job->copy.runs[r].count << PAGE_SHIFT;
+
+		dma_iova_unlink(dev, &xe_tt->iova, off, len,
+				DMA_BIDIRECTIONAL, DMA_ATTR_SKIP_CPU_SYNC);
+	}
+
+	/*
+	 * Relink the freshly copied new pages from the snapshot captured at job
+	 * creation. The destination tt's page array may have been mutated (or
+	 * torn down) by TTM since then, so it is never dereferenced here.
+	 * Borrowed pages are shared with the old tt and stay correctly linked
+	 * for free, so the snapshot only covers the changed runs.
+	 */
+	for (l = 0; l < job->copy.nr_links; l++) {
+		int ret = dma_iova_link(dev, &xe_tt->iova,
+					job->copy.links[l].phys,
+					job->copy.links[l].offset,
+					job->copy.links[l].len,
+					DMA_BIDIRECTIONAL,
+					DMA_ATTR_SKIP_CPU_SYNC);
+		/*
+		 * Point of no failure: the data is already copied and the
+		 * reservation ownership has been transferred. A relink failure
+		 * leaves a hole in the mapping; there is no safe recovery, so
+		 * just warn.
+		 */
+		drm_WARN_ON(&xe->drm, ret);
+	}
+
+	/* Sync the relinked changed runs into the device's view. */
+	for (r = 0; r < job->copy.nr_runs; r++) {
+		size_t off = (size_t)job->copy.runs[r].start << PAGE_SHIFT;
+		size_t len = (size_t)job->copy.runs[r].count << PAGE_SHIFT;
+
+		dma_iova_sync(dev, &xe_tt->iova, off, len);
+	}
+
+	/*
+	 * All work is synchronous CPU-side IOMMU fixup, so there is no hardware
+	 * fence to wait on; returning NULL makes the scheduler signal the job's
+	 * finished fence immediately.
+	 */
+	return NULL;
+}
+
+static void xe_iova_defrag_job_free(struct xe_dep_job *dep_job)
+{
+	struct xe_iova_defrag_job *job =
+		container_of(dep_job, typeof(*job), dep);
+
+	xe_tt_iova_copy_fini(&job->copy);
+	drm_sched_job_cleanup(&job->dep.drm);
+	kfree(job);
+}
+
+static const struct xe_dep_job_ops xe_iova_defrag_job_ops = {
+	.run_job = xe_iova_defrag_job_run,
+	.free_job = xe_iova_defrag_job_free,
+};
+
 /*
  * xe_tt_iova_transfer - Transfer a full IOVA reservation old_tt -> new_tt.
  *
@@ -1063,7 +1180,6 @@ static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt,
  * folio walk of the new tt keeps xe->mem.dma_mapped_pages[] exact while the new
  * mapping lives, rather than carrying the old tt's stale distribution.
  */
-__maybe_unused
 static void xe_tt_iova_transfer(struct xe_device *xe,
 				struct xe_ttm_tt *new_tt,
 				struct xe_ttm_tt *old_tt)
@@ -1078,6 +1194,135 @@ static void xe_tt_iova_transfer(struct xe_device *xe,
 	old_tt->use_iova = false;
 }
 
+/*
+ * xe_bo_defrag_iova_copy - Run an IOVA defrag move with a packed copy mapping.
+ *
+ * Only the changed (non-borrowed) pages of @new_tt are DMA mapped, packed
+ * contiguously into the front of its reservation, the GPU copy blits them from
+ * the old tt at natural offsets, and a post-copy finalize job adopts the old
+ * tt's full reservation and relinks the changed runs. Returns the finalize
+ * job's finished fence (the fence the move waits on), or an ERR_PTR on failure
+ * with all partial state unwound.
+ */
+static struct dma_fence *
+xe_bo_defrag_iova_copy(struct xe_device *xe, struct xe_migrate *migrate,
+		       struct xe_bo *bo, struct ttm_resource *old_mem,
+		       struct ttm_resource *new_mem, struct ttm_tt *old_tt,
+		       bool handle_system_ccs)
+{
+	struct ttm_tt *new_tt = bo->ttm.ttm;
+	struct xe_ttm_tt *new_xe_tt = container_of(new_tt, struct xe_ttm_tt, ttm);
+	struct xe_ttm_tt *old_xe_tt = container_of(old_tt, struct xe_ttm_tt, ttm);
+	struct drm_sched_entity *entity =
+		xe_dep_scheduler_entity(xe->mem.defrag.iova_sched);
+	struct xe_iova_defrag_job *job;
+	struct dma_fence *copy_fence;
+	struct dma_fence *fence;
+	struct xe_tt_iova_copy copy;
+	bool packed = false;
+	int err;
+
+	xe_assert(xe, new_xe_tt->use_iova && !new_xe_tt->iova_linked);
+	xe_assert(xe, old_xe_tt->use_iova && old_xe_tt->iova_linked);
+
+	err = xe_tt_iova_copy_init(new_tt, old_tt, &copy);
+	if (err)
+		return ERR_PTR(err);
+
+	if (copy.nr_changed) {
+		err = xe_tt_map_iova_copy(xe, new_tt, &copy);
+		if (err)
+			goto err_copy;
+		packed = true;
+
+		copy_fence = xe_migrate_copy_defrag_iova(migrate, bo, old_mem,
+							 new_mem, old_tt,
+							 copy.nr_changed,
+							 handle_system_ccs);
+	} else {
+		copy_fence = dma_fence_get_stub();
+	}
+	if (IS_ERR(copy_fence)) {
+		err = PTR_ERR(copy_fence);
+		goto err_unmap;
+	}
+
+	job = kzalloc(sizeof(*job), GFP_KERNEL);
+	if (!job) {
+		err = -ENOMEM;
+		goto err_fence;
+	}
+
+	job->dep.ops = &xe_iova_defrag_job_ops;
+	job->xe = xe;
+	job->new_tt = new_tt;
+	job->copy_iova = new_xe_tt->iova;
+	job->copy_bytes = (size_t)copy.nr_changed << PAGE_SHIFT;
+	job->copy = copy;
+
+	err = drm_sched_job_init(&job->dep.drm, entity, 1, NULL, 0);
+	if (err)
+		goto err_job;
+
+	err = drm_sched_job_add_dependency(&job->dep.drm,
+					   dma_fence_get(copy_fence));
+	if (err)
+		goto err_job_init;
+
+	/*
+	 * Point of no failure: transfer the old tt's full reservation to the
+	 * new tt. The finalize job (already armed below) destroys the packed
+	 * copy mapping snapshot and relinks the changed runs.
+	 */
+	xe_tt_iova_transfer(xe, new_xe_tt, old_xe_tt);
+	memset(&copy, 0, sizeof(copy)); /* ownership moved to the job */
+
+	drm_sched_job_arm(&job->dep.drm);
+	fence = dma_fence_get(&job->dep.drm.s_fence->finished);
+	drm_sched_entity_push_job(&job->dep.drm);
+
+	dma_fence_put(copy_fence);
+
+	return fence;
+
+err_job_init:
+	drm_sched_job_cleanup(&job->dep.drm);
+err_job:
+	kfree(job);
+err_fence:
+	dma_fence_put(copy_fence);
+err_unmap:
+	if (packed)
+		dma_iova_unlink(xe->drm.dev, &new_xe_tt->iova, 0,
+				(size_t)copy.nr_changed << PAGE_SHIFT,
+				DMA_BIDIRECTIONAL, DMA_ATTR_SKIP_CPU_SYNC);
+err_copy:
+	xe_tt_iova_copy_fini(&copy);
+	return ERR_PTR(err);
+}
+
+/*
+ * xe_bo_defrag_use_iova - Whether a defrag move should use the packed IOVA path.
+ *
+ * Eligible when the move is a defrag (stashed old tt) and the destination tt is
+ * IOVA mapped. The IOVA reservation is transferred to the new backing, so the
+ * device-visible addresses are preserved and no GPU rebind is needed (see
+ * xe_bo_trigger_rebind()); this makes the path worthwhile regardless of BO
+ * size. A flat-CCS aux pass, if needed, is migrated packed alongside the data
+ * (changed pages only), so it does not disqualify the path. Otherwise the
+ * full-mapping defrag path is used.
+ */
+static bool xe_bo_defrag_use_iova(struct ttm_buffer_object *ttm_bo)
+{
+	struct xe_ttm_tt *xe_tt;
+
+	if (!ttm_bo->defrag_old_tt || !ttm_bo->ttm)
+		return false;
+
+	xe_tt = container_of(ttm_bo->ttm, struct xe_ttm_tt, ttm);
+	return xe_tt->use_iova;
+}
+
 /*
  * Account ttm pages against the device shrinker's shrinkable and
  * purgeable counts.
@@ -1357,6 +1602,7 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,
 	struct drm_gem_object *obj = &bo->ttm.base;
 	struct drm_gpuvm_bo *vm_bo;
 	bool idle = false;
+	bool defrag_iova;
 	int ret = 0;
 
 	dma_resv_assert_held(bo->ttm.base.resv);
@@ -1369,12 +1615,25 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,
 		dma_resv_iter_end(&cursor);
 	}
 
+	/*
+	 * A defrag move that reallocates IOVA-mapped backing preserves the
+	 * device-visible addresses: the IOVA reservation is transferred from
+	 * the old to the new tt, so the existing GPU PTEs stay valid and the
+	 * BO must not be marked evicted (the rebind paths in xe_exec_ioctl()
+	 * and preempt_rebind_work_func() would otherwise needlessly rebind).
+	 * Preempt fences still need to be triggered, and fault-mode / L2
+	 * flush-optimized configs still need a TLB invalidation below, so only
+	 * the eviction marking is skipped.
+	 */
+	defrag_iova = xe_bo_defrag_use_iova(&bo->ttm);
+
 	drm_gem_for_each_gpuvm_bo(vm_bo, obj) {
 		struct xe_vm *vm = gpuvm_to_vm(vm_bo->vm);
 		struct drm_gpuva *gpuva;
 
 		if (!xe_vm_in_fault_mode(vm)) {
-			drm_gpuvm_bo_evict(vm_bo, true);
+			if (!defrag_iova)
+				drm_gpuvm_bo_evict(vm_bo, true);
 			/*
 			 * L2 cache may not be flushed, so ensure that is done in
 			 * xe_vm_invalidate_vma() below
@@ -1388,6 +1647,8 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,
 			 */
 			if (!xe_vm_flush_vm_bo_tlb_async(vm, bo, vm_bo))
 				continue;
+			else if (defrag_iova)
+				drm_gpuvm_bo_evict(vm_bo, true);
 		}
 
 		if (!idle) {
@@ -2096,6 +2357,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	bool needs_clear;
 	bool handle_system_ccs = (!IS_DGFX(xe) && xe_bo_needs_ccs_pages(bo) &&
 				  ttm && ttm_tt_is_populated(ttm)) ? true : false;
+	bool defrag_iova = xe_bo_defrag_use_iova(ttm_bo);
 	int ret = 0;
 
 	/*
@@ -2144,7 +2406,12 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 	needs_clear = (ttm && ttm->page_flags & TTM_TT_FLAG_ZERO_ALLOC) ||
 		(!ttm && ttm_bo->type == ttm_bo_type_device);
 
-	if (new_mem->mem_type == XE_PL_TT) {
+	/*
+	 * For a defrag move using the packed IOVA path the destination tt is
+	 * not fully mapped here; only the changed pages are mapped (packed) for
+	 * the copy, and the finalize job establishes the full mapping.
+	 */
+	if (new_mem->mem_type == XE_PL_TT && !defrag_iova) {
 		ret = xe_tt_map(xe, ttm);
 		if (ret)
 			goto out;
@@ -2254,9 +2521,16 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 
 		fence = xe_migrate_clear(migrate, bo, new_mem, flags);
 	} else if (ttm_bo->defrag_old_tt) {
-		fence = xe_migrate_copy_defrag(migrate, bo, old_mem, new_mem,
-					       ttm_bo->defrag_old_tt,
-					       handle_system_ccs);
+		if (defrag_iova)
+			fence = xe_bo_defrag_iova_copy(xe, migrate, bo, old_mem,
+						       new_mem,
+						       ttm_bo->defrag_old_tt,
+						       handle_system_ccs);
+		else
+			fence = xe_migrate_copy_defrag(migrate, bo, old_mem,
+						       new_mem,
+						       ttm_bo->defrag_old_tt,
+						       handle_system_ccs);
 	} else {
 		fence = xe_migrate_copy(migrate, bo, bo, old_mem, new_mem,
 					handle_system_ccs);
Signature · 1 line
2.34.1
#59 lore raw

Re: [PATCH v2 32/33] drm/xe: Finalize defrag-IOVA moves with post-copy job

Thank you for your contribution! Sashiko AI review found 6 potential issue(s) to consider: - [Critical] Error paths tear down the destination IOVA mapping without waiting for the submitted…

Thank you for your contribution! Sashiko AI review found 6 potential issue(s) to consider:
- [Critical] Error paths tear down the destination IOVA mapping without waiting for the submitted asynchronous GPU copy job, leading to IOMMU faults and Write-After-Free.
- [High] xe_bo_defrag_use_iova() checks only the new_tt for IOVA support, ignoring whether old_tt actually uses IOVA, leading to invalid IOVA transfer and IOMMU corruption.
- [High] TLB/L2 cache invalidation for the IOVA defrag is scheduled too early, leading to stale entries in the GPU TLB after the IOMMU update completes.
- [High] Allocating memory with GFP_KERNEL while holding the BO's dma_resv lock creates an AB-BA deadlock risk with the DRM memory shrinker.
- [High] Incomplete SWIOTLB bounce buffer handling causes the GPU to read uninitialized memory after defragmentation.
- [Medium] The return value of dma_iova_sync() is ignored in the defrag finalize job, potentially leaving the IOTLB/cache unflushed.
--

--- Patch [32]: [PATCH v2 32/33] drm/xe: Finalize defrag-IOVA moves with post-copy job --- commit 23b117a272c85c9d94147e6761afd74a17131d1f
Author: Matthew Brost <[email protected]>
Subject: drm/xe: Finalize defrag-IOVA moves with post-copy job

This commit wires up end-to-end defrag for IOVA-mapped BOs. A defrag move is eligible whenever the destination TT is IOVA mapped. The packed-copy step moves only changed pages, and a post-copy finalize job tears down the temporary packed mapping and relinks the copied pages.

Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
index 16d9e135cba2e..0eee53f4d653f 100644

[ ... ]

Quoted 9 lines
drivers/gpu/drm/xe/xe_bo.c +0 -1
@@ -855,7 +854,6 @@ struct xe_tt_iova_run_collect {
 };
 
 /* Geometrically grow @arr (element size @elem) to hold at least @need slots. */
-__maybe_unused
 static int xe_tt_iova_grow(void **arr, unsigned int *cap, unsigned int need,
 			   size_t elem)
 {

drivers/gpu/drm/xe/xe_bo.c static int xe_tt_iova_grow(void **arr, unsigned int *cap, unsigned int need, · line 870

Quoted 7 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -866,13 +864,12 @@ static int xe_tt_iova_grow(void **arr, unsigned int *cap, unsigned int need,
 		return 0;
 
 	new_cap = *cap ? *cap * 2 : 64;
 	if (new_cap < need)
 		new_cap = need;
 
 	p = krealloc_array(*arr, new_cap, elem, GFP_KERNEL);

[Severity: High]
Is it safe to use GFP_KERNEL here? This is called from xe_bo_move() while holding the buffer object's dma_resv lock.

If the memory allocator triggers direct reclaim, it can invoke the DRM memory shrinker, which will attempt to acquire dma_resv locks to evict buffer objects, leading to an AB-BA deadlock.

Quoted 2 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
 	if (!p)
 		return -ENOMEM;

[ ... ]

Quoted 3 lines
drivers/gpu/drm/xe/xe_bo.c +2 -0
@@ -1047,6 +1041,129 @@ static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt,
+static struct dma_fence *xe_iova_defrag_job_run(struct xe_dep_job *dep_job)
+{

[ ... ]

Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +4 -0
+	/*
+	 * Relink the freshly copied new pages from the snapshot captured at job
+	 * creation. The destination tt's page array may have been mutated (or
+	 * torn down) by TTM since then, so it is never dereferenced here.

drivers/gpu/drm/xe/xe_bo.c static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt, · line 1126

Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +10 -0
+	 * Borrowed pages are shared with the old tt and stay correctly linked
+	 * for free, so the snapshot only covers the changed runs.
+	 */
+	for (l = 0; l < job->copy.nr_links; l++) {
+		int ret = dma_iova_link(dev, &xe_tt->iova,
+					job->copy.links[l].phys,
+					job->copy.links[l].offset,
+					job->copy.links[l].len,
+					DMA_BIDIRECTIONAL,
+					DMA_ATTR_SKIP_CPU_SYNC);

[Severity: High]
Does skipping CPU sync here break configurations using SWIOTLB?

When using DMA_ATTR_SKIP_CPU_SYNC, the DMA API allocates a new bounce buffer for the physical page but bypasses copying the physical page contents into it.

This would cause the new IOVA mapping to point to an uninitialized bounce buffer, leading to the GPU reading uninitialized memory.

Quoted 5 lines
drivers/gpu/drm/xe/xe_bo.c +5 -0
+		/*
+		 * Point of no failure: the data is already copied and the
+		 * reservation ownership has been transferred. A relink failure
+		 * leaves a hole in the mapping; there is no safe recovery, so
+		 * just warn.

drivers/gpu/drm/xe/xe_bo.c static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt, · line 1141

Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +10 -0
+		 */
+		drm_WARN_ON(&xe->drm, ret);
+	}
+
+	/* Sync the relinked changed runs into the device's view. */
+	for (r = 0; r < job->copy.nr_runs; r++) {
+		size_t off = (size_t)job->copy.runs[r].start << PAGE_SHIFT;
+		size_t len = (size_t)job->copy.runs[r].count << PAGE_SHIFT;
+
+		dma_iova_sync(dev, &xe_tt->iova, off, len);

[Severity: Medium]
Should the return value of dma_iova_sync() be checked here?

While this is a point of no return and the job cannot be aborted gracefully, silently ignoring the error hides IOMMU cache sync failures.

A failure could result in stale IOMMU caches and GPU data corruption, so perhaps at least a drm_WARN_ON() is warranted, similar to the dma_iova_link() call above.

Quoted 1 line
drivers/gpu/drm/xe/xe_bo.c +1 -0
+	}

[ ... ]

drivers/gpu/drm/xe/xe_bo.c static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt, · line 1184

Quoted 7 lines
drivers/gpu/drm/xe/xe_bo.c +0 -1
@@ -1063,7 +1180,6 @@ static int xe_tt_map_iova_copy(struct xe_device *xe, struct ttm_tt *new_tt,
-__maybe_unused
 static void xe_tt_iova_transfer(struct xe_device *xe,
 				struct xe_ttm_tt *new_tt,
 				struct xe_ttm_tt *old_tt)
 {
 	new_tt->iova = old_tt->iova;

[Severity: High]
What happens if the original old_tt failed to allocate an IOVA mapping and fell back to an SG mapping?

If old_tt->use_iova is false, old_tt->iova would be uninitialized or empty. Copying it here would overwrite the valid new_tt->iova state, which could lead to IOMMU corruption when the finalize job attempts to link pages.

Quoted 1 line
drivers/gpu/drm/xe/xe_bo.c +0 -0
 	new_tt->iova_linked = true;

[ ... ]

Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +6 -0
+static struct dma_fence *
+xe_bo_defrag_iova_copy(struct xe_device *xe, struct xe_migrate *migrate,
+		       struct xe_bo *bo, struct ttm_resource *old_mem,
+		       struct ttm_resource *new_mem, struct ttm_tt *old_tt,
+		       bool handle_system_ccs)
+{

[ ... ]

drivers/gpu/drm/xe/xe_bo.c static void xe_tt_iova_transfer(struct xe_device *xe, · line 1250

Quoted 1 line
drivers/gpu/drm/xe/xe_bo.c +1 -0
+	job = kzalloc(sizeof(*job), GFP_KERNEL);

[Severity: High]
Could this GFP_KERNEL allocation cause a deadlock?

Just like the allocation in xe_tt_iova_grow(), this executes under the buffer object's dma_resv lock during xe_bo_move(). If memory pressure triggers the shrinker, it might try to acquire the same dma_resv lock.

Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +4 -0
+	if (!job) {
+		err = -ENOMEM;
+		goto err_fence;
+	}

[ ... ]

drivers/gpu/drm/xe/xe_bo.c static void xe_tt_iova_transfer(struct xe_device *xe, · line 1298

Quoted 11 lines
drivers/gpu/drm/xe/xe_bo.c +11 -0
+err_job_init:
+	drm_sched_job_cleanup(&job->dep.drm);
+err_job:
+	kfree(job);
+err_fence:
+	dma_fence_put(copy_fence);
+err_unmap:
+	if (packed)
+		dma_iova_unlink(xe->drm.dev, &new_xe_tt->iova, 0,
+				(size_t)copy.nr_changed << PAGE_SHIFT,
+				DMA_BIDIRECTIONAL, DMA_ATTR_SKIP_CPU_SYNC);

[Severity: Critical]
Is it safe to unlink the IOVA mapping here without waiting for copy_fence to signal?

By destroying the destination IOVA page tables immediately, the asynchronous GPU migration job might still be actively writing to those addresses.

This could result in IOMMU faults or write-after-free corruption if TTM aborts the move and frees the physical pages while the GPU blit is still running.

Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +4 -0
+err_copy:
+	xe_tt_iova_copy_fini(&copy);
+	return ERR_PTR(err);
+}

[ ... ]

drivers/gpu/drm/xe/xe_bo.c static void xe_tt_iova_transfer(struct xe_device *xe, · line 1323

Quoted 12 lines
drivers/gpu/drm/xe/xe_bo.c +12 -0
+/*
+ * xe_bo_defrag_use_iova - Whether a defrag move should use the packed IOVA path.
+ */
+static bool xe_bo_defrag_use_iova(struct ttm_buffer_object *ttm_bo)
+{
+	struct xe_ttm_tt *xe_tt;
+
+	if (!ttm_bo->defrag_old_tt || !ttm_bo->ttm)
+		return false;
+
+	xe_tt = container_of(ttm_bo->ttm, struct xe_ttm_tt, ttm);
+	return xe_tt->use_iova;

[Severity: High]
Does this check adequately ensure the old_tt is actually using IOVA?

The function verifies new_tt->use_iova, but omits checking old_tt->use_iova.

If the initial old_tt fell back to an SG mapping, this path would still return true as long as new_tt is IOVA-eligible. This would then misroute the move to xe_bo_defrag_iova_copy() and trigger the uninitialized IOVA transfer mentioned earlier.

Quoted 1 line

+}

[ ... ]

drivers/gpu/drm/xe/xe_bo.c static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo, · line 1651

Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +2 -0
@@ -1388,6 +1647,8 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,
 			 */
 			if (!xe_vm_flush_vm_bo_tlb_async(vm, bo, vm_bo))
 				continue;
+			else if (defrag_iova)
+				drm_gpuvm_bo_evict(vm_bo, true);

[Severity: High]
Can this sequence cause the GPU to access freed physical pages?

The function xe_vm_flush_vm_bo_tlb_async() flushes the TLB asynchronously based on the current fences in the reservation. However, this is executed before the migration and finalize jobs are added to the buffer object's reservation.

Because the eviction marking is skipped for defrag_iova, the userspace GPU context is not stopped. It can access the buffer object and repopulate the GPU TLB with the old IOMMU translations after this premature flush completes, but before the finalize job actually updates the IOMMU.

Quoted 1 line
drivers/gpu/drm/xe/xe_bo.c +0 -0
 		}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=32
author maintainer
#60 lore raw

PATCH v2 33/33 drm/amdgpu: Preallocate system BO pages outside the reservation lock

In reply to Matthew Brost

Populating a GTT (system) buffer object under the reservation lock can stall in reclaim and compaction while trying to satisfy beneficial-order allocations, holding the lock for the duration.

Mirror the Xe out-of-lock preallocation for pure system BOs: when a gem_create request targets AMDGPU_GEM_DOMAIN_GTT only, fill the full page backing up front via ttm_pool_prealloc_fill_full() before taking the reservation lock. The populate under the lock then simply installs these pages instead of reclaiming and compacting in the critical section. The fill is best-effort - a short fill falls back to the normal in-lock allocation for the remaining pages, and any leftover pages are released by ttm_pool_prealloc_fini() on all exit paths.

Unlike Xe, amdgpu has no background defragmenter, so the higher-order reclaim backoff is left disabled (normal reclaim). If the pool uses dma-alloc (swiotlb), the fill bails and the feature is a silent no-op.

Factor the tt pool selection into amdgpu_ttm_tt_pool_id() and amdgpu_ttm_pool() so the preallocation targets the exact pool the populate will consume, and thread an optional prealloc bag through amdgpu_gem_object_create() and amdgpu_bo_param into the populate ctx.

Cc Alex Deucher <[email protected]>
Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |  4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c   |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       | 45 +++++++++++++++----
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h       |  5 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c    |  3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |  4 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       | 39 +++++++++++-----
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h       |  3 ++
 8 files changed, 80 insertions(+), 25 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +2 -2
index 20831dbebc31..940f58848a97 100644
@@ -345,7 +345,7 @@ create_dmamap_sg_bo(struct amdgpu_device *adev,
 
 	ret = amdgpu_gem_object_create(adev, mem->bo->tbo.base.size, 1,
 			AMDGPU_GEM_DOMAIN_CPU, AMDGPU_GEM_CREATE_PREEMPTIBLE | flags,
-			ttm_bo_type_sg, mem->bo->tbo.base.resv, &gem_obj, 0);
+			ttm_bo_type_sg, mem->bo->tbo.base.resv, &gem_obj, 0, NULL);
 
 	amdgpu_bo_unreserve(mem->bo);
 
@@ -1811,7 +1811,7 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(
 		 domain_string(alloc_domain), xcp_id);
 
 	ret = amdgpu_gem_object_create(adev, aligned_size, 1, alloc_domain, alloc_flags,
-				       bo_type, NULL, &gobj, xcp_id + 1);
+				       bo_type, NULL, &gobj, xcp_id + 1, NULL);
 	if (ret) {
 		pr_debug("Failed to create BO on domain %s. ret %d\n",
 			 domain_string(alloc_domain), ret);
drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c +1 -1
index b33c300e26e2..51510e831129 100644
@@ -435,7 +435,7 @@ amdgpu_dma_buf_create_obj(struct drm_device *dev, struct dma_buf *dma_buf)
 
 	ret = amdgpu_gem_object_create(adev, dma_buf->size, PAGE_SIZE,
 				       AMDGPU_GEM_DOMAIN_CPU, flags,
-				       ttm_bo_type_sg, resv, &gobj, 0);
+				       ttm_bo_type_sg, resv, &gobj, 0, NULL);
 	if (ret)
 		goto error;
 
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +37 -8

110 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h +4 -1
index b558336bc4c6..706aeca011f1 100644
@@ -35,6 +35,8 @@
 
 extern const struct drm_gem_object_funcs amdgpu_gem_object_funcs;
 
+struct ttm_pool_prealloc;
+
 unsigned long amdgpu_gem_timeout(uint64_t timeout_ns);
 
 /*
@@ -45,7 +47,8 @@ int amdgpu_gem_object_create(struct amdgpu_device *adev, unsigned long size,
 			     int alignment, u32 initial_domain,
 			     u64 flags, enum ttm_bo_type type,
 			     struct dma_resv *resv,
-			     struct drm_gem_object **obj, int8_t xcp_id_plus1);
+			     struct drm_gem_object **obj, int8_t xcp_id_plus1,
+			     struct ttm_pool_prealloc *prealloc);
 int amdgpu_mode_dumb_create(struct drm_file *file_priv,
 			    struct drm_device *dev,
 			    struct drm_mode_create_dumb *args);
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +2 -1
index f98bfba59a2c..18c4cf3f35a5 100644
@@ -632,7 +632,8 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
 		/* We opt to avoid OOM on system pages allocations */
 		.gfp_retry_mayfail = true,
 		.allow_res_evict = bp->type != ttm_bo_type_kernel,
-		.resv = bp->resv
+		.resv = bp->resv,
+		.prealloc = bp->prealloc,
 	};
 	struct amdgpu_bo *bo;
 	unsigned long page_align, size = bp->size;
drivers/gpu/drm/amd/amdgpu/amdgpu_object.h +4 -0
index ff11a0903499..11f1d403f152 100644
@@ -45,6 +45,8 @@
 #define to_amdgpu_bo_user(abo) container_of((abo), struct amdgpu_bo_user, bo)
 #define to_amdgpu_bo_vm(abo) container_of((abo), struct amdgpu_bo_vm, bo)
 
+struct ttm_pool_prealloc;
+
 struct amdgpu_bo_param {
 	unsigned long			size;
 	int				byte_align;
@@ -58,6 +60,8 @@ struct amdgpu_bo_param {
 	void				(*destroy)(struct ttm_buffer_object *bo);
 	/* xcp partition number plus 1, 0 means any partition */
 	int8_t				xcp_id_plus1;
+	/* optional out-of-lock preallocated backing (system/GTT only) */
+	struct ttm_pool_prealloc	*prealloc;
 };
 
 /* bo virtual addresses in a vm */
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +27 -12
index b10b0878df37..c9cd4714f515 100644
@@ -1173,6 +1173,30 @@ void amdgpu_ttm_mmio_remap_free_sgt(struct device *dev,
 	kfree(sgt);
 }
 
+/*
+ * amdgpu_ttm_tt_pool_id - compute the ttm pool id backing a given xcp
+ *
+ * Mirrors the mapping used when creating the gtt ttm_tt, so callers that only
+ * have an xcp id (e.g. an out-of-lock preallocation before the bo exists) pick
+ * the same pool the populate will use.
+ */
+int32_t amdgpu_ttm_tt_pool_id(struct amdgpu_device *adev, int32_t xcp_id)
+{
+	if (adev->gmc.mem_partitions && xcp_id >= 0)
+		return KFD_XCP_MEM_ID(adev, xcp_id);
+
+	return xcp_id;
+}
+
+/* amdgpu_ttm_pool - select the ttm pool for a given pool id */
+struct ttm_pool *amdgpu_ttm_pool(struct amdgpu_device *adev, int32_t pool_id)
+{
+	if (adev->mman.ttm_pools && pool_id >= 0)
+		return &adev->mman.ttm_pools[pool_id];
+
+	return &adev->mman.bdev.pool;
+}
+
 /**
  * amdgpu_ttm_tt_create - Create a ttm_tt object for a given BO
  *
@@ -1194,10 +1218,7 @@ static struct ttm_tt *amdgpu_ttm_tt_create(struct ttm_buffer_object *bo,
 		return NULL;
 
 	gtt->gobj = &bo->base;
-	if (adev->gmc.mem_partitions && abo->xcp_id >= 0)
-		gtt->pool_id = KFD_XCP_MEM_ID(adev, abo->xcp_id);
-	else
-		gtt->pool_id = abo->xcp_id;
+	gtt->pool_id = amdgpu_ttm_tt_pool_id(adev, abo->xcp_id);
 
 	if (abo->flags & AMDGPU_GEM_CREATE_CPU_GTT_USWC)
 		caching = ttm_write_combined;
@@ -1239,10 +1260,7 @@ static int amdgpu_ttm_tt_populate(struct ttm_device *bdev,
 	if (ttm->page_flags & TTM_TT_FLAG_EXTERNAL)
 		return 0;
 
-	if (adev->mman.ttm_pools && gtt->pool_id >= 0)
-		pool = &adev->mman.ttm_pools[gtt->pool_id];
-	else
-		pool = &adev->mman.bdev.pool;
+	pool = amdgpu_ttm_pool(adev, gtt->pool_id);
 	ret = ttm_pool_alloc(pool, ttm, ctx);
 	if (ret)
 		return ret;
@@ -1284,10 +1302,7 @@ static void amdgpu_ttm_tt_unpopulate(struct ttm_device *bdev,
 
 	adev = amdgpu_ttm_adev(bdev);
 
-	if (adev->mman.ttm_pools && gtt->pool_id >= 0)
-		pool = &adev->mman.ttm_pools[gtt->pool_id];
-	else
-		pool = &adev->mman.bdev.pool;
+	pool = amdgpu_ttm_pool(adev, gtt->pool_id);
 
 	return ttm_pool_free(pool, ttm);
 }
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h +3 -0
index ff9e2e346609..f90844937b73 100644
@@ -207,6 +207,9 @@ int amdgpu_ttm_alloc_gart(struct ttm_buffer_object *bo);
 void amdgpu_ttm_recover_gart(struct ttm_buffer_object *tbo);
 uint64_t amdgpu_ttm_domain_start(struct amdgpu_device *adev, uint32_t type);
 
+int32_t amdgpu_ttm_tt_pool_id(struct amdgpu_device *adev, int32_t xcp_id);
+struct ttm_pool *amdgpu_ttm_pool(struct amdgpu_device *adev, int32_t pool_id);
+
 #if IS_ENABLED(CONFIG_DRM_AMDGPU_USERPTR)
 int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo,
 				 struct amdgpu_hmm_range *range);
Signature · 1 line
2.34.1
#61 lore raw

Re: [PATCH v2 33/33] drm/amdgpu: Preallocate system BO pages outside the reservation lock

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Critical] Preallocated pages can be configured with Write-Combined (WC) caching but assigned to a…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] Preallocated pages can be configured with Write-Combined (WC) caching but assigned to a Write-Back (WB) TTM buffer object if the platform rejects USWC, causing PAT aliasing and TTM pool corruption.
- [High] Unbounded memory preallocation allows unprivileged users to cause massive memory pressure and kernel stalls before BO size limits are checked.
--

--- Patch [33]: [PATCH v2 33/33] drm/amdgpu: Preallocate system BO pages outside the reservation lock --- commit 4ff1d390d0b654be970d2510b2e5cd29bc266121
Author: Matthew Brost <[email protected]>

drm/amdgpu: Preallocate system BO pages outside the reservation lock

This commit mirrors Xe's out-of-lock preallocation for pure system BOs in amdgpu. When a gem_create request targets only AMDGPU_GEM_DOMAIN_GTT, it fills the full page backing up front before taking the reservation lock, to prevent stalls in reclaim and compaction while holding the lock.

Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +0 -0
index 6a0699746fbcd..bef8b5bc8b892 100644

[ ... ]

Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +3 -0
@@ -443,10 +448,29 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data,
 		flags |= AMDGPU_GEM_CREATE_NO_CPU_ACCESS;
 	}
 
+	/*
+	 * For system-only (pure GTT) BOs, preallocate the whole page backing
+	 * up front, outside the reservation lock. Populate under the lock then

drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data, · line 463

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +10 -0
+	 * just installs these pages instead of reclaiming/compacting in the
+	 * critical section. Best-effort: a short fill falls back to the normal
+	 * in-lock allocation for the missing pages.
+	 */
+	if (args->in.domains == AMDGPU_GEM_DOMAIN_GTT) {
+		int32_t xcp_id = adev->gmc.mem_partitions ? fpriv->xcp_id : 0;
+		int32_t pool_id = amdgpu_ttm_tt_pool_id(adev, xcp_id);
+		enum ttm_caching caching =
+			(flags & AMDGPU_GEM_CREATE_CPU_GTT_USWC) ?
+			ttm_write_combined : ttm_cached;

[Severity: Critical]
Is it possible for the preallocated pages to be configured with write-combined caching here, but later assigned to a write-back TTM buffer object?

If an unprivileged user requests AMDGPU_GEM_CREATE_CPU_GTT_USWC on a platform where amdgpu_bo_support_uswc() returns false, the code preallocates the pages as ttm_write_combined.

Later, amdgpu_bo_create() in drivers/gpu/drm/amd/amdgpu/amdgpu_object.c strips the unsupported flag:

amdgpu_bo_create() {
    ...
    if (!amdgpu_bo_support_uswc(bo->flags))
        bo->flags &= ~AMDGPU_GEM_CREATE_CPU_GTT_USWC;
    ...
}

This creates a ttm_tt initialized as ttm_cached (write-back). When the object is eventually destroyed, ttm_pool_free() will use the write-back caching state, bypassing the required PAT restoration.

Does this leak the write-combined physical pages back into the kernel allocator and cause PAT aliasing or machine check exceptions?

drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data, · line 467

Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +4 -0
+
+		prealloc_pool = amdgpu_ttm_pool(adev, pool_id);
+		ttm_pool_prealloc_fill_full(prealloc_pool, caching, &prealloc,
+					    PFN_UP(size), false);

[Severity: High]
Could this allow unprivileged users to trigger massive memory pressure before buffer object size limits are checked?

The user-provided size is passed directly to ttm_pool_prealloc_fill_full() here, but the size validation against domain limits does not happen until later in the call chain:

drivers/gpu/drm/amd/amdgpu/amdgpu_object.c:amdgpu_bo_create() {
    ...
    if (!amdgpu_bo_validate_size(adev, size, bp->domain))
        return -ENOMEM;
    ...
}

If an unprivileged user makes an ioctl request with an excessively large size (e.g., 100 GB), ttm_pool_prealloc_fill_full() will attempt to satisfy the request, triggering intensive reclaim and compaction before the size is ever validated.

Would it be safer to validate the size against domain limits before attempting the preallocation?

Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +2 -0
+	}
+
 	if (flags & AMDGPU_GEM_CREATE_VM_ALWAYS_VALID) {
 		r = amdgpu_bo_reserve(vm->root.bo, false);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=33
#62 lore raw

✗ CI.checkpatch: warning for drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs

In reply to Matthew Brost
== Series Details ==

== Series Details ==

Series: drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs
URL : https://patchwork.freedesktop.org/series/170216/
State : warning

== Summary ==

+ KERNEL=/kernel
+ git clone https://gitlab.freedesktop.org/drm/maintainer-tools mt Cloning into 'mt'...
warning: redirecting to https://gitlab.freedesktop.org/drm/maintainer-tools.git/
+ git -C mt rev-list -n1 origin/master 061140b9bc586ae7f40abc1249c97e1cc72d1b9d
+ cd /kernel
+ git config --global --add safe.directory /kernel
+ git log -n1
commit 82ab45b799f73f937cd639d94d3b08a5b7a9a0b2

Author: Matthew Brost <[email protected]>
Date:   Fri Jul 10 14:54:42 2026 -0700

drm/amdgpu: Preallocate system BO pages outside the reservation lock

Populating a GTT (system) buffer object under the reservation lock can stall in reclaim and compaction while trying to satisfy beneficial-order allocations, holding the lock for the duration.

Mirror the Xe out-of-lock preallocation for pure system BOs: when a gem_create request targets AMDGPU_GEM_DOMAIN_GTT only, fill the full page backing up front via ttm_pool_prealloc_fill_full() before taking the reservation lock. The populate under the lock then simply installs these pages instead of reclaiming and compacting in the critical section. The fill is best-effort - a short fill falls back to the normal in-lock allocation for the remaining pages, and any leftover pages are released by ttm_pool_prealloc_fini() on all exit paths.

Unlike Xe, amdgpu has no background defragmenter, so the higher-order reclaim backoff is left disabled (normal reclaim). If the pool uses dma-alloc (swiotlb), the fill bails and the feature is a silent no-op.

Factor the tt pool selection into amdgpu_ttm_tt_pool_id() and amdgpu_ttm_pool() so the preallocation targets the exact pool the populate will consume, and thread an optional prealloc bag through amdgpu_gem_object_create() and amdgpu_bo_param into the populate ctx.

Cc Alex Deucher <[email protected]>
Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

+ /mt/dim checkpatch 6d76f5afea7406e978db1fc356a9b88d0d8989d1 drm-intel b5ea29b79e69 drm/ttm/pool: Allow backing off reclaim at the beneficial order 40519ebbeb07 drm/ttm/pool: Add ttm_pool_page_order_nodma() helper d11b3260013f drm/ttm: Record sub-optimal page order allocations in ttm_tt 09a182540850 drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc() 29b9e1034781 drm/ttm: Support defragmentation moves
545efd587d6c drm/ttm: Add fault injection for beneficial-order allocation failures 6b0f9d119b29 drm/ttm: Harvest beneficial-order pages on defragmentation moves -:296: WARNING:PREFER_FALLTHROUGH: Prefer 'fallthrough;' over fallthrough comment #296: FILE: drivers/gpu/drm/ttm/ttm_pool.c:959:
+ * fall through to a fresh allocation, or a negative errno.

total: 0 errors, 1 warnings, 0 checks, 442 lines checked ac53fd9106fb drm/ttm: Bound page (re)allocation per defragmentation move c996f6eb6f33 drm/ttm: Preallocate beneficial-order defrag pages outside the lock -:280: CHECK:MULTIPLE_ASSIGNMENTS: multiple assignments should be avoided #280: FILE: drivers/gpu/drm/ttm/ttm_pool.c:1104:
+ pp->count = pp->used = 0;

-:354: WARNING:PREFER_FALLTHROUGH: Prefer 'fallthrough;' over fallthrough comment #354: FILE: drivers/gpu/drm/ttm/ttm_pool.c:1262:
+ * reclaim/compaction). Falls through once the bag is empty.

total: 0 errors, 1 warnings, 1 checks, 451 lines checked d8e9b4e458ba drm/ttm: Add full out-of-lock preallocation for ttm_pool_alloc() -:313: WARNING:PREFER_FALLTHROUGH: Prefer 'fallthrough;' over fallthrough comment #313: FILE: drivers/gpu/drm/ttm/ttm_pool.c:1439:
+ * taking its order from the page. Falls through to in-line allocation

total: 0 errors, 1 warnings, 0 checks, 336 lines checked b64fca16eebd drm/gpusvm: Add a DMA-mapping accounting callback 07bccbda4186 drm/xe: Add debugfs stats for DMA-mapped pages per order 13e14afb16de drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind() db0dc1e13c44 drm/xe: Destroy page tables after unlinking all VMAs on VM close c4a0595dcbde drm/xe: Track BOs backed at a sub-optimal page order e1acf58015b1 drm/xe: Back off beneficial-order reclaim under defrag pressure 381ceba7708f drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies 97501f259e44 drm/xe: Handle defrag moves in xe_bo_move()
-:66: WARNING:PREFER_FALLTHROUGH: Prefer 'fallthrough;' over fallthrough comment #66: FILE: drivers/gpu/drm/xe/xe_bo.c:1211:
+ * For a defrag move the old and new tt differ, so fallthrough to the

total: 0 errors, 1 warnings, 0 checks, 54 lines checked 926f66dda2e7 drm/xe: Skip self-copies for borrowed pages on defrag moves 8c392057c15f drm/xe: Add a page defragmentation worker 5f3a08ec2b4d drm/xe: Add defrag GT stats
73f0ef10d182 drm/xe: Add Kconfig.profile options for BO defrag configuration ee0a08bca176 drm/xe: Defrag using out-of-lock page preallocation 0ad2ae8f7849 drm/xe: Add defrag profiling tracepoints
-:124: CHECK:OPEN_ENDED_LINE: Lines should not end with a '(' #124: FILE: drivers/gpu/drm/xe/xe_trace_bo.h:92:
+ TP_STRUCT__entry(

-:130: CHECK:OPEN_ENDED_LINE: Lines should not end with a '(' #130: FILE: drivers/gpu/drm/xe/xe_trace_bo.h:98:
+ TP_fast_assign(

-:147: CHECK:OPEN_ENDED_LINE: Lines should not end with a '(' #147: FILE: drivers/gpu/drm/xe/xe_trace_bo.h:115:
+ TP_STRUCT__entry(

-:159: CHECK:OPEN_ENDED_LINE: Lines should not end with a '(' #159: FILE: drivers/gpu/drm/xe/xe_trace_bo.h:127:
+ TP_fast_assign(

total: 0 errors, 0 warnings, 4 checks, 128 lines checked 1331bb746434 drm/xe: Preallocate system BO backing outside the dma-resv lock fb30f45f50c5 drm/xe: Add tracepoint for xe_gem_create_ioctl -:107: CHECK:OPEN_ENDED_LINE: Lines should not end with a '(' #107: FILE: drivers/gpu/drm/xe/xe_trace_bo.h:152:
+ TP_STRUCT__entry(

-:117: CHECK:OPEN_ENDED_LINE: Lines should not end with a '(' #117: FILE: drivers/gpu/drm/xe/xe_trace_bo.h:162:
+ TP_fast_assign(

total: 0 errors, 0 warnings, 2 checks, 83 lines checked
21e477f8ed2b drm/xe: Add IOVA-based xe_res_cursor variant c2cb5210431a drm/xe: Use IOVA-based DMA mapping for eligible tt BOs e4d12e72dbd8 drm/xe: Add per-device dependency scheduler for IOVA defrag finalize ebdffcf82b2f drm/xe: Add packed copy-step IOVA mapping for defrag e0180fc409b2 drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies dd513f622f83 drm/xe: Finalize defrag-IOVA moves with post-copy job 82ab45b799f7 drm/amdgpu: Preallocate system BO pages outside the reservation lock -:136: CHECK:PREFER_KERNEL_TYPES: Prefer kernel type 's32' over 'int32_t' #136: FILE: drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c:459:
+ int32_t xcp_id = adev->gmc.mem_partitions ? fpriv->xcp_id : 0;

-:137: CHECK:PREFER_KERNEL_TYPES: Prefer kernel type 's32' over 'int32_t' #137: FILE: drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c:460:
+ int32_t pool_id = amdgpu_ttm_tt_pool_id(adev, xcp_id);

total: 0 errors, 0 warnings, 2 checks, 248 lines checked

#63 lore raw

✓ CI.KUnit: success for drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs

In reply to Matthew Brost
== Series Details ==

== Series Details ==

Series: drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs
URL : https://patchwork.freedesktop.org/series/170216/
State : success

== Summary ==

+ trap cleanup EXIT
+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/xe/.kunitconfig [22:03:46] Configuring KUnit Kernel ...
Generating .config ...
Populating config with:

$ make ARCH=um O=.kunit olddefconfig
[22:03:50] Building KUnit Kernel ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
Building with:
$ make all compile_commands.json scripts_gdb ARCH=um O=.kunit --jobs=48
[22:04:22] Starting KUnit Kernel (1/1)...
[22:04:22] ============================================================
Running tests with:
$ .kunit/linux kunit.enable=1 mem=1G console=tty kunit_shutdown=halt
[22:04:22] ================== guc_buf (11 subtests) ===================
[22:04:22] [PASSED] test_smallest
[22:04:22] [PASSED] test_largest
[22:04:22] [PASSED] test_granular
[22:04:22] [PASSED] test_unique
[22:04:22] [PASSED] test_overlap
[22:04:22] [PASSED] test_reusable
[22:04:22] [PASSED] test_too_big
[22:04:22] [PASSED] test_flush
[22:04:22] [PASSED] test_lookup
[22:04:22] [PASSED] test_data
[22:04:22] [PASSED] test_class
[22:04:22] ===================== [PASSED] guc_buf =====================
[22:04:22] =================== guc_dbm (7 subtests) ===================
[22:04:22] [PASSED] test_empty
[22:04:22] [PASSED] test_default
[22:04:22] ======================== test_size  ========================
[22:04:22] [PASSED] 4
[22:04:22] [PASSED] 8
[22:04:22] [PASSED] 32
[22:04:22] [PASSED] 256
[22:04:22] ==================== [PASSED] test_size ====================
[22:04:22] ======================= test_reuse  ========================
[22:04:22] [PASSED] 4
[22:04:22] [PASSED] 8
[22:04:22] [PASSED] 32
[22:04:22] [PASSED] 256
[22:04:22] =================== [PASSED] test_reuse ====================
[22:04:22] =================== test_range_overlap  ====================
[22:04:22] [PASSED] 4
[22:04:22] [PASSED] 8
[22:04:22] [PASSED] 32
[22:04:22] [PASSED] 256
[22:04:22] =============== [PASSED] test_range_overlap ================
[22:04:22] =================== test_range_compact  ====================
[22:04:22] [PASSED] 4
[22:04:22] [PASSED] 8
[22:04:22] [PASSED] 32
[22:04:22] [PASSED] 256
[22:04:22] =============== [PASSED] test_range_compact ================
[22:04:22] ==================== test_range_spare  =====================
[22:04:22] [PASSED] 4
[22:04:22] [PASSED] 8
[22:04:22] [PASSED] 32
[22:04:22] [PASSED] 256
[22:04:22] ================ [PASSED] test_range_spare =================
[22:04:22] ===================== [PASSED] guc_dbm =====================
[22:04:22] =================== guc_idm (6 subtests) ===================
[22:04:22] [PASSED] bad_init
[22:04:22] [PASSED] no_init
[22:04:22] [PASSED] init_fini
[22:04:22] [PASSED] check_used
[22:04:22] [PASSED] check_quota
[22:04:22] [PASSED] check_all
[22:04:22] ===================== [PASSED] guc_idm =====================
[22:04:22] ================== no_relay (3 subtests) ===================
[22:04:22] [PASSED] xe_drops_guc2pf_if_not_ready
[22:04:22] [PASSED] xe_drops_guc2vf_if_not_ready
[22:04:22] [PASSED] xe_rejects_send_if_not_ready
[22:04:22] ==================== [PASSED] no_relay =====================
[22:04:22] ================== pf_relay (14 subtests) ==================
[22:04:22] [PASSED] pf_rejects_guc2pf_too_short
[22:04:22] [PASSED] pf_rejects_guc2pf_too_long
[22:04:22] [PASSED] pf_rejects_guc2pf_no_payload
[22:04:22] [PASSED] pf_fails_no_payload
[22:04:22] [PASSED] pf_fails_bad_origin
[22:04:22] [PASSED] pf_fails_bad_type
[22:04:22] [PASSED] pf_txn_reports_error
[22:04:22] [PASSED] pf_txn_sends_pf2guc
[22:04:22] [PASSED] pf_sends_pf2guc
[22:04:22] [SKIPPED] pf_loopback_nop (requires CONFIG_DRM_XE_DEBUG_SRIOV)
[22:04:22] [SKIPPED] pf_loopback_echo (requires CONFIG_DRM_XE_DEBUG_SRIOV)
[22:04:22] [SKIPPED] pf_loopback_fail (requires CONFIG_DRM_XE_DEBUG_SRIOV)
[22:04:22] [SKIPPED] pf_loopback_busy (requires CONFIG_DRM_XE_DEBUG_SRIOV)
[22:04:22] [SKIPPED] pf_loopback_retry (requires CONFIG_DRM_XE_DEBUG_SRIOV)
[22:04:22] ==================== [PASSED] pf_relay =====================
[22:04:22] ================== vf_relay (3 subtests) ===================
[22:04:22] [PASSED] vf_rejects_guc2vf_too_short
[22:04:22] [PASSED] vf_rejects_guc2vf_too_long
[22:04:22] [PASSED] vf_rejects_guc2vf_no_payload
[22:04:22] ==================== [PASSED] vf_relay =====================
[22:04:22] ================ pf_gt_config (9 subtests) =================
[22:04:22] [PASSED] fair_contexts_1vf
[22:04:22] [PASSED] fair_doorbells_1vf
[22:04:22] [PASSED] fair_ggtt_1vf
[22:04:22] ====================== fair_vram_1vf  ======================
[22:04:22] [PASSED] 3.50 GiB
[22:04:22] [PASSED] 11.5 GiB
[22:04:22] [PASSED] 15.5 GiB
[22:04:22] [PASSED] 31.5 GiB
[22:04:22] [PASSED] 63.5 GiB
[22:04:22] [PASSED] 1.91 GiB
[22:04:22] ================== [PASSED] fair_vram_1vf ==================
[22:04:22] ================ fair_vram_1vf_admin_only  =================
[22:04:22] [PASSED] 3.50 GiB
[22:04:22] [PASSED] 11.5 GiB
[22:04:22] [PASSED] 15.5 GiB
[22:04:22] [PASSED] 31.5 GiB
[22:04:22] [PASSED] 63.5 GiB
[22:04:22] [PASSED] 1.91 GiB
[22:04:22] ============ [PASSED] fair_vram_1vf_admin_only =============
[22:04:22] ====================== fair_contexts  ======================
[22:04:22] [PASSED] 1 VF
[22:04:22] [PASSED] 2 VFs
[22:04:22] [PASSED] 3 VFs
[22:04:22] [PASSED] 4 VFs
[22:04:22] [PASSED] 5 VFs
[22:04:22] [PASSED] 6 VFs
[22:04:22] [PASSED] 7 VFs
[22:04:22] [PASSED] 8 VFs
[22:04:22] [PASSED] 9 VFs
[22:04:22] [PASSED] 10 VFs
[22:04:22] [PASSED] 11 VFs
[22:04:22] [PASSED] 12 VFs
[22:04:22] [PASSED] 13 VFs
[22:04:22] [PASSED] 14 VFs
[22:04:22] [PASSED] 15 VFs
[22:04:22] [PASSED] 16 VFs
[22:04:22] [PASSED] 17 VFs
[22:04:22] [PASSED] 18 VFs
[22:04:22] [PASSED] 19 VFs
[22:04:22] [PASSED] 20 VFs
[22:04:22] [PASSED] 21 VFs
[22:04:22] [PASSED] 22 VFs
[22:04:22] [PASSED] 23 VFs
[22:04:22] [PASSED] 24 VFs
[22:04:22] [PASSED] 25 VFs
[22:04:22] [PASSED] 26 VFs
[22:04:22] [PASSED] 27 VFs
[22:04:22] [PASSED] 28 VFs
[22:04:22] [PASSED] 29 VFs
[22:04:22] [PASSED] 30 VFs
[22:04:22] [PASSED] 31 VFs
[22:04:22] [PASSED] 32 VFs
[22:04:22] [PASSED] 33 VFs
[22:04:22] [PASSED] 34 VFs
[22:04:22] [PASSED] 35 VFs
[22:04:22] [PASSED] 36 VFs
[22:04:22] [PASSED] 37 VFs
[22:04:22] [PASSED] 38 VFs
[22:04:22] [PASSED] 39 VFs
[22:04:22] [PASSED] 40 VFs
[22:04:22] [PASSED] 41 VFs
[22:04:22] [PASSED] 42 VFs
[22:04:22] [PASSED] 43 VFs
[22:04:22] [PASSED] 44 VFs
[22:04:22] [PASSED] 45 VFs
[22:04:22] [PASSED] 46 VFs
[22:04:22] [PASSED] 47 VFs
[22:04:22] [PASSED] 48 VFs
[22:04:22] [PASSED] 49 VFs
[22:04:22] [PASSED] 50 VFs
[22:04:22] [PASSED] 51 VFs
[22:04:22] [PASSED] 52 VFs
[22:04:22] [PASSED] 53 VFs
[22:04:22] [PASSED] 54 VFs
[22:04:22] [PASSED] 55 VFs
[22:04:22] [PASSED] 56 VFs
[22:04:22] [PASSED] 57 VFs
[22:04:22] [PASSED] 58 VFs
[22:04:22] [PASSED] 59 VFs
[22:04:22] [PASSED] 60 VFs
[22:04:22] [PASSED] 61 VFs
[22:04:22] [PASSED] 62 VFs
[22:04:22] [PASSED] 63 VFs
[22:04:22] ================== [PASSED] fair_contexts ==================
[22:04:22] ===================== fair_doorbells  ======================
[22:04:22] [PASSED] 1 VF
[22:04:22] [PASSED] 2 VFs
[22:04:22] [PASSED] 3 VFs
[22:04:22] [PASSED] 4 VFs
[22:04:22] [PASSED] 5 VFs
[22:04:22] [PASSED] 6 VFs
[22:04:22] [PASSED] 7 VFs
[22:04:22] [PASSED] 8 VFs
[22:04:22] [PASSED] 9 VFs
[22:04:22] [PASSED] 10 VFs
[22:04:22] [PASSED] 11 VFs
[22:04:22] [PASSED] 12 VFs
[22:04:22] [PASSED] 13 VFs
[22:04:22] [PASSED] 14 VFs
[22:04:22] [PASSED] 15 VFs
[22:04:22] [PASSED] 16 VFs
[22:04:22] [PASSED] 17 VFs
[22:04:22] [PASSED] 18 VFs
[22:04:22] [PASSED] 19 VFs
[22:04:22] [PASSED] 20 VFs
[22:04:22] [PASSED] 21 VFs
[22:04:22] [PASSED] 22 VFs
[22:04:22] [PASSED] 23 VFs
[22:04:22] [PASSED] 24 VFs
[22:04:22] [PASSED] 25 VFs
[22:04:22] [PASSED] 26 VFs
[22:04:22] [PASSED] 27 VFs
[22:04:22] [PASSED] 28 VFs
[22:04:22] [PASSED] 29 VFs
[22:04:22] [PASSED] 30 VFs
[22:04:22] [PASSED] 31 VFs
[22:04:22] [PASSED] 32 VFs
[22:04:22] [PASSED] 33 VFs
[22:04:22] [PASSED] 34 VFs
[22:04:22] [PASSED] 35 VFs
[22:04:22] [PASSED] 36 VFs
[22:04:22] [PASSED] 37 VFs
[22:04:22] [PASSED] 38 VFs
[22:04:22] [PASSED] 39 VFs
[22:04:22] [PASSED] 40 VFs
[22:04:22] [PASSED] 41 VFs
[22:04:22] [PASSED] 42 VFs
[22:04:22] [PASSED] 43 VFs
[22:04:22] [PASSED] 44 VFs
[22:04:22] [PASSED] 45 VFs
[22:04:22] [PASSED] 46 VFs
[22:04:22] [PASSED] 47 VFs
[22:04:22] [PASSED] 48 VFs
[22:04:22] [PASSED] 49 VFs
[22:04:22] [PASSED] 50 VFs
[22:04:22] [PASSED] 51 VFs
[22:04:22] [PASSED] 52 VFs
[22:04:22] [PASSED] 53 VFs
[22:04:22] [PASSED] 54 VFs
[22:04:22] [PASSED] 55 VFs
[22:04:22] [PASSED] 56 VFs
[22:04:22] [PASSED] 57 VFs
[22:04:22] [PASSED] 58 VFs
[22:04:22] [PASSED] 59 VFs
[22:04:22] [PASSED] 60 VFs
[22:04:22] [PASSED] 61 VFs
[22:04:22] [PASSED] 62 VFs
[22:04:22] [PASSED] 63 VFs
[22:04:22] ================= [PASSED] fair_doorbells ==================
[22:04:22] ======================== fair_ggtt  ========================
[22:04:22] [PASSED] 1 VF
[22:04:22] [PASSED] 2 VFs
[22:04:22] [PASSED] 3 VFs
[22:04:22] [PASSED] 4 VFs
[22:04:22] [PASSED] 5 VFs
[22:04:22] [PASSED] 6 VFs
[22:04:22] [PASSED] 7 VFs
[22:04:22] [PASSED] 8 VFs
[22:04:22] [PASSED] 9 VFs
[22:04:22] [PASSED] 10 VFs
[22:04:22] [PASSED] 11 VFs
[22:04:22] [PASSED] 12 VFs
[22:04:22] [PASSED] 13 VFs
[22:04:22] [PASSED] 14 VFs
[22:04:22] [PASSED] 15 VFs
[22:04:22] [PASSED] 16 VFs
[22:04:22] [PASSED] 17 VFs
[22:04:22] [PASSED] 18 VFs
[22:04:22] [PASSED] 19 VFs
[22:04:22] [PASSED] 20 VFs
[22:04:22] [PASSED] 21 VFs
[22:04:22] [PASSED] 22 VFs
[22:04:22] [PASSED] 23 VFs
[22:04:22] [PASSED] 24 VFs
[22:04:22] [PASSED] 25 VFs
[22:04:22] [PASSED] 26 VFs
[22:04:22] [PASSED] 27 VFs
[22:04:22] [PASSED] 28 VFs
[22:04:22] [PASSED] 29 VFs
[22:04:22] [PASSED] 30 VFs
[22:04:22] [PASSED] 31 VFs
[22:04:22] [PASSED] 32 VFs
[22:04:22] [PASSED] 33 VFs
[22:04:22] [PASSED] 34 VFs
[22:04:22] [PASSED] 35 VFs
[22:04:22] [PASSED] 36 VFs
[22:04:22] [PASSED] 37 VFs
[22:04:22] [PASSED] 38 VFs
[22:04:22] [PASSED] 39 VFs
[22:04:22] [PASSED] 40 VFs
[22:04:22] [PASSED] 41 VFs
[22:04:22] [PASSED] 42 VFs
[22:04:22] [PASSED] 43 VFs
[22:04:22] [PASSED] 44 VFs
[22:04:22] [PASSED] 45 VFs
[22:04:22] [PASSED] 46 VFs
[22:04:22] [PASSED] 47 VFs
[22:04:22] [PASSED] 48 VFs
[22:04:22] [PASSED] 49 VFs
[22:04:22] [PASSED] 50 VFs
[22:04:22] [PASSED] 51 VFs
[22:04:22] [PASSED] 52 VFs
[22:04:22] [PASSED] 53 VFs
[22:04:22] [PASSED] 54 VFs
[22:04:22] [PASSED] 55 VFs
[22:04:22] [PASSED] 56 VFs
[22:04:22] [PASSED] 57 VFs
[22:04:22] [PASSED] 58 VFs
[22:04:22] [PASSED] 59 VFs
[22:04:23] [PASSED] 60 VFs
[22:04:23] [PASSED] 61 VFs
[22:04:23] [PASSED] 62 VFs
[22:04:23] [PASSED] 63 VFs
[22:04:23] ==================== [PASSED] fair_ggtt ====================
[22:04:23] ======================== fair_vram  ========================
[22:04:23] [PASSED] 1 VF
[22:04:23] [PASSED] 2 VFs
[22:04:23] [PASSED] 3 VFs
[22:04:23] [PASSED] 4 VFs
[22:04:23] [PASSED] 5 VFs
[22:04:23] [PASSED] 6 VFs
[22:04:23] [PASSED] 7 VFs
[22:04:23] [PASSED] 8 VFs
[22:04:23] [PASSED] 9 VFs
[22:04:23] [PASSED] 10 VFs
[22:04:23] [PASSED] 11 VFs
[22:04:23] [PASSED] 12 VFs
[22:04:23] [PASSED] 13 VFs
[22:04:23] [PASSED] 14 VFs
[22:04:23] [PASSED] 15 VFs
[22:04:23] [PASSED] 16 VFs
[22:04:23] [PASSED] 17 VFs
[22:04:23] [PASSED] 18 VFs
[22:04:23] [PASSED] 19 VFs
[22:04:23] [PASSED] 20 VFs
[22:04:23] [PASSED] 21 VFs
[22:04:23] [PASSED] 22 VFs
[22:04:23] [PASSED] 23 VFs
[22:04:23] [PASSED] 24 VFs
[22:04:23] [PASSED] 25 VFs
[22:04:23] [PASSED] 26 VFs
[22:04:23] [PASSED] 27 VFs
[22:04:23] [PASSED] 28 VFs
[22:04:23] [PASSED] 29 VFs
[22:04:23] [PASSED] 30 VFs
[22:04:23] [PASSED] 31 VFs
[22:04:23] [PASSED] 32 VFs
[22:04:23] [PASSED] 33 VFs
[22:04:23] [PASSED] 34 VFs
[22:04:23] [PASSED] 35 VFs
[22:04:23] [PASSED] 36 VFs
[22:04:23] [PASSED] 37 VFs
[22:04:23] [PASSED] 38 VFs
[22:04:23] [PASSED] 39 VFs
[22:04:23] [PASSED] 40 VFs
[22:04:23] [PASSED] 41 VFs
[22:04:23] [PASSED] 42 VFs
[22:04:23] [PASSED] 43 VFs
[22:04:23] [PASSED] 44 VFs
[22:04:23] [PASSED] 45 VFs
[22:04:23] [PASSED] 46 VFs
[22:04:23] [PASSED] 47 VFs
[22:04:23] [PASSED] 48 VFs
[22:04:23] [PASSED] 49 VFs
[22:04:23] [PASSED] 50 VFs
[22:04:23] [PASSED] 51 VFs
[22:04:23] [PASSED] 52 VFs
[22:04:23] [PASSED] 53 VFs
[22:04:23] [PASSED] 54 VFs
[22:04:23] [PASSED] 55 VFs
[22:04:23] [PASSED] 56 VFs
[22:04:23] [PASSED] 57 VFs
[22:04:23] [PASSED] 58 VFs
[22:04:23] [PASSED] 59 VFs
[22:04:23] [PASSED] 60 VFs
[22:04:23] [PASSED] 61 VFs
[22:04:23] [PASSED] 62 VFs
[22:04:23] [PASSED] 63 VFs
[22:04:23] ==================== [PASSED] fair_vram ====================
[22:04:23] ================== [PASSED] pf_gt_config ===================
[22:04:23] ===================== lmtt (1 subtest) =====================
[22:04:23] ======================== test_ops  =========================
[22:04:23] [PASSED] 2-level
[22:04:23] [PASSED] multi-level
[22:04:23] ==================== [PASSED] test_ops =====================
[22:04:23] ====================== [PASSED] lmtt =======================
[22:04:23] ================= pf_service (11 subtests) =================
[22:04:23] [PASSED] pf_negotiate_any
[22:04:23] [PASSED] pf_negotiate_base_match
[22:04:23] [PASSED] pf_negotiate_base_newer
[22:04:23] [PASSED] pf_negotiate_base_next
[22:04:23] [SKIPPED] pf_negotiate_base_older (no older minor)
[22:04:23] [PASSED] pf_negotiate_base_prev
[22:04:23] [PASSED] pf_negotiate_latest_match
[22:04:23] [PASSED] pf_negotiate_latest_newer
[22:04:23] [PASSED] pf_negotiate_latest_next
[22:04:23] [SKIPPED] pf_negotiate_latest_older (no older minor)
[22:04:23] [SKIPPED] pf_negotiate_latest_prev (no prev major)
[22:04:23] =================== [PASSED] pf_service ====================
[22:04:23] ================= xe_guc_g2g (2 subtests) ==================
[22:04:23] ============== xe_live_guc_g2g_kunit_default  ==============
[22:04:23] ========= [SKIPPED] xe_live_guc_g2g_kunit_default ==========
[22:04:23] ============== xe_live_guc_g2g_kunit_allmem  ===============
[22:04:23] ========== [SKIPPED] xe_live_guc_g2g_kunit_allmem ==========
[22:04:23] =================== [SKIPPED] xe_guc_g2g ===================
[22:04:23] =================== xe_mocs (2 subtests) ===================
[22:04:23] ================ xe_live_mocs_kernel_kunit  ================
[22:04:23] =========== [SKIPPED] xe_live_mocs_kernel_kunit ============
[22:04:23] ================ xe_live_mocs_reset_kunit  =================
[22:04:23] ============ [SKIPPED] xe_live_mocs_reset_kunit ============
[22:04:23] ==================== [SKIPPED] xe_mocs =====================
[22:04:23] ================= xe_migrate (2 subtests) ==================
[22:04:23] ================= xe_migrate_sanity_kunit  =================
[22:04:23] ============ [SKIPPED] xe_migrate_sanity_kunit =============
[22:04:23] ================== xe_validate_ccs_kunit  ==================
[22:04:23] ============= [SKIPPED] xe_validate_ccs_kunit ==============
[22:04:23] =================== [SKIPPED] xe_migrate ===================
[22:04:23] ================== xe_dma_buf (1 subtest) ==================
[22:04:23] ==================== xe_dma_buf_kunit  =====================
[22:04:23] ================ [SKIPPED] xe_dma_buf_kunit ================
[22:04:23] =================== [SKIPPED] xe_dma_buf ===================
[22:04:23] ================= xe_bo_shrink (1 subtest) =================
[22:04:23] =================== xe_bo_shrink_kunit  ====================
[22:04:23] =============== [SKIPPED] xe_bo_shrink_kunit ===============
[22:04:23] ================== [SKIPPED] xe_bo_shrink ==================
[22:04:23] ==================== xe_bo (2 subtests) ====================
[22:04:23] ================== xe_ccs_migrate_kunit  ===================
[22:04:23] ============== [SKIPPED] xe_ccs_migrate_kunit ==============
[22:04:23] ==================== xe_bo_evict_kunit  ====================
[22:04:23] =============== [SKIPPED] xe_bo_evict_kunit ================
[22:04:23] ===================== [SKIPPED] xe_bo ======================
[22:04:23] ==================== args (13 subtests) ====================
[22:04:23] [PASSED] count_args_test
[22:04:23] [PASSED] call_args_example
[22:04:23] [PASSED] call_args_test
[22:04:23] [PASSED] drop_first_arg_example
[22:04:23] [PASSED] drop_first_arg_test
[22:04:23] [PASSED] first_arg_example
[22:04:23] [PASSED] first_arg_test
[22:04:23] [PASSED] last_arg_example
[22:04:23] [PASSED] last_arg_test
[22:04:23] [PASSED] pick_arg_example
[22:04:23] [PASSED] if_args_example
[22:04:23] [PASSED] if_args_test
[22:04:23] [PASSED] sep_comma_example
[22:04:23] ====================== [PASSED] args =======================
[22:04:23] =================== xe_pci (3 subtests) ====================
[22:04:23] ==================== check_graphics_ip  ====================
[22:04:23] [PASSED] 12.00 Xe_LP
[22:04:23] [PASSED] 12.10 Xe_LP+
[22:04:23] [PASSED] 12.55 Xe_HPG
[22:04:23] [PASSED] 12.60 Xe_HPC
[22:04:23] [PASSED] 12.70 Xe_LPG
[22:04:23] [PASSED] 12.71 Xe_LPG
[22:04:23] [PASSED] 12.74 Xe_LPG+
[22:04:23] [PASSED] 20.01 Xe2_HPG
[22:04:23] [PASSED] 20.02 Xe2_HPG
[22:04:23] [PASSED] 20.04 Xe2_LPG
[22:04:23] [PASSED] 30.00 Xe3_LPG
[22:04:23] [PASSED] 30.01 Xe3_LPG
[22:04:23] [PASSED] 30.03 Xe3_LPG
[22:04:23] [PASSED] 30.04 Xe3_LPG
[22:04:23] [PASSED] 30.05 Xe3_LPG
[22:04:23] [PASSED] 35.10 Xe3p_LPG
[22:04:23] [PASSED] 35.11 Xe3p_XPC
[22:04:23] ================ [PASSED] check_graphics_ip ================
[22:04:23] ===================== check_media_ip  ======================
[22:04:23] [PASSED] 12.00 Xe_M
[22:04:23] [PASSED] 12.55 Xe_HPM
[22:04:23] [PASSED] 13.00 Xe_LPM+
[22:04:23] [PASSED] 13.01 Xe2_HPM
[22:04:23] [PASSED] 20.00 Xe2_LPM
[22:04:23] [PASSED] 30.00 Xe3_LPM
[22:04:23] [PASSED] 30.02 Xe3_LPM
[22:04:23] [PASSED] 35.00 Xe3p_LPM
[22:04:23] [PASSED] 35.03 Xe3p_HPM
[22:04:23] ================= [PASSED] check_media_ip ==================
[22:04:23] =================== check_platform_desc  ===================
[22:04:23] [PASSED] 0x9A60 (TIGERLAKE)
[22:04:23] [PASSED] 0x9A68 (TIGERLAKE)
[22:04:23] [PASSED] 0x9A70 (TIGERLAKE)
[22:04:23] [PASSED] 0x9A40 (TIGERLAKE)
[22:04:23] [PASSED] 0x9A49 (TIGERLAKE)
[22:04:23] [PASSED] 0x9A59 (TIGERLAKE)
[22:04:23] [PASSED] 0x9A78 (TIGERLAKE)
[22:04:23] [PASSED] 0x9AC0 (TIGERLAKE)
[22:04:23] [PASSED] 0x9AC9 (TIGERLAKE)
[22:04:23] [PASSED] 0x9AD9 (TIGERLAKE)
[22:04:23] [PASSED] 0x9AF8 (TIGERLAKE)
[22:04:23] [PASSED] 0x4C80 (ROCKETLAKE)
[22:04:23] [PASSED] 0x4C8A (ROCKETLAKE)
[22:04:23] [PASSED] 0x4C8B (ROCKETLAKE)
[22:04:23] [PASSED] 0x4C8C (ROCKETLAKE)
[22:04:23] [PASSED] 0x4C90 (ROCKETLAKE)
[22:04:23] [PASSED] 0x4C9A (ROCKETLAKE)
[22:04:23] [PASSED] 0x4680 (ALDERLAKE_S)
[22:04:23] [PASSED] 0x4682 (ALDERLAKE_S)
[22:04:23] [PASSED] 0x4688 (ALDERLAKE_S)
[22:04:23] [PASSED] 0x468A (ALDERLAKE_S)
[22:04:23] [PASSED] 0x468B (ALDERLAKE_S)
[22:04:23] [PASSED] 0x4690 (ALDERLAKE_S)
[22:04:23] [PASSED] 0x4692 (ALDERLAKE_S)
[22:04:23] [PASSED] 0x4693 (ALDERLAKE_S)
[22:04:23] [PASSED] 0x46A0 (ALDERLAKE_P)
[22:04:23] [PASSED] 0x46A1 (ALDERLAKE_P)
[22:04:23] [PASSED] 0x46A2 (ALDERLAKE_P)
[22:04:23] [PASSED] 0x46A3 (ALDERLAKE_P)
[22:04:23] [PASSED] 0x46A6 (ALDERLAKE_P)
[22:04:23] [PASSED] 0x46A8 (ALDERLAKE_P)
[22:04:23] [PASSED] 0x46AA (ALDERLAKE_P)
[22:04:23] [PASSED] 0x462A (ALDERLAKE_P)
[22:04:23] [PASSED] 0x4626 (ALDERLAKE_P)
[22:04:23] [PASSED] 0x4628 (ALDERLAKE_P)
[22:04:23] [PASSED] 0x46B0 (ALDERLAKE_P)
[22:04:23] [PASSED] 0x46B1 (ALDERLAKE_P)
[22:04:23] [PASSED] 0x46B2 (ALDERLAKE_P)
[22:04:23] [PASSED] 0x46B3 (ALDERLAKE_P)
[22:04:23] [PASSED] 0x46C0 (ALDERLAKE_P)
[22:04:23] [PASSED] 0x46C1 (ALDERLAKE_P)
[22:04:23] [PASSED] 0x46C2 (ALDERLAKE_P)
[22:04:23] [PASSED] 0x46C3 (ALDERLAKE_P)
[22:04:23] [PASSED] 0x46D0 (ALDERLAKE_N)
[22:04:23] [PASSED] 0x46D1 (ALDERLAKE_N)
[22:04:23] [PASSED] 0x46D2 (ALDERLAKE_N)
[22:04:23] [PASSED] 0x46D3 (ALDERLAKE_N)
[22:04:23] [PASSED] 0x46D4 (ALDERLAKE_N)
[22:04:23] [PASSED] 0xA721 (ALDERLAKE_P)
[22:04:23] [PASSED] 0xA7A1 (ALDERLAKE_P)
[22:04:23] [PASSED] 0xA7A9 (ALDERLAKE_P)
[22:04:23] [PASSED] 0xA7AC (ALDERLAKE_P)
[22:04:23] [PASSED] 0xA7AD (ALDERLAKE_P)
[22:04:23] [PASSED] 0xA720 (ALDERLAKE_P)
[22:04:23] [PASSED] 0xA7A0 (ALDERLAKE_P)
[22:04:23] [PASSED] 0xA7A8 (ALDERLAKE_P)
[22:04:23] [PASSED] 0xA7AA (ALDERLAKE_P)
[22:04:23] [PASSED] 0xA7AB (ALDERLAKE_P)
[22:04:23] [PASSED] 0xA780 (ALDERLAKE_S)
[22:04:23] [PASSED] 0xA781 (ALDERLAKE_S)
[22:04:23] [PASSED] 0xA782 (ALDERLAKE_S)
[22:04:23] [PASSED] 0xA783 (ALDERLAKE_S)
[22:04:23] [PASSED] 0xA788 (ALDERLAKE_S)
[22:04:23] [PASSED] 0xA789 (ALDERLAKE_S)
[22:04:23] [PASSED] 0xA78A (ALDERLAKE_S)
[22:04:23] [PASSED] 0xA78B (ALDERLAKE_S)
[22:04:23] [PASSED] 0x4905 (DG1)
[22:04:23] [PASSED] 0x4906 (DG1)
[22:04:23] [PASSED] 0x4907 (DG1)
[22:04:23] [PASSED] 0x4908 (DG1)
[22:04:23] [PASSED] 0x4909 (DG1)
[22:04:23] [PASSED] 0x56C0 (DG2)
[22:04:23] [PASSED] 0x56C2 (DG2)
[22:04:23] [PASSED] 0x56C1 (DG2)
[22:04:23] [PASSED] 0x7D51 (METEORLAKE)
[22:04:23] [PASSED] 0x7DD1 (METEORLAKE)
[22:04:23] [PASSED] 0x7D41 (METEORLAKE)
[22:04:23] [PASSED] 0x7D67 (METEORLAKE)
[22:04:23] [PASSED] 0xB640 (METEORLAKE)
[22:04:23] [PASSED] 0x56A0 (DG2)
[22:04:23] [PASSED] 0x56A1 (DG2)
[22:04:23] [PASSED] 0x56A2 (DG2)
[22:04:23] [PASSED] 0x56BE (DG2)
[22:04:23] [PASSED] 0x56BF (DG2)
[22:04:23] [PASSED] 0x5690 (DG2)
[22:04:23] [PASSED] 0x5691 (DG2)
[22:04:23] [PASSED] 0x5692 (DG2)
[22:04:23] [PASSED] 0x56A5 (DG2)
[22:04:23] [PASSED] 0x56A6 (DG2)
[22:04:23] [PASSED] 0x56B0 (DG2)
[22:04:23] [PASSED] 0x56B1 (DG2)
[22:04:23] [PASSED] 0x56BA (DG2)
[22:04:23] [PASSED] 0x56BB (DG2)
[22:04:23] [PASSED] 0x56BC (DG2)
[22:04:23] [PASSED] 0x56BD (DG2)
[22:04:23] [PASSED] 0x5693 (DG2)
[22:04:23] [PASSED] 0x5694 (DG2)
[22:04:23] [PASSED] 0x5695 (DG2)
[22:04:23] [PASSED] 0x56A3 (DG2)
[22:04:23] [PASSED] 0x56A4 (DG2)
[22:04:23] [PASSED] 0x56B2 (DG2)
[22:04:23] [PASSED] 0x56B3 (DG2)
[22:04:23] [PASSED] 0x5696 (DG2)
[22:04:23] [PASSED] 0x5697 (DG2)
[22:04:23] [PASSED] 0xB69 (PVC)
[22:04:23] [PASSED] 0xB6E (PVC)
[22:04:23] [PASSED] 0xBD4 (PVC)
[22:04:23] [PASSED] 0xBD5 (PVC)
[22:04:23] [PASSED] 0xBD6 (PVC)
[22:04:23] [PASSED] 0xBD7 (PVC)
[22:04:23] [PASSED] 0xBD8 (PVC)
[22:04:23] [PASSED] 0xBD9 (PVC)
[22:04:23] [PASSED] 0xBDA (PVC)
[22:04:23] [PASSED] 0xBDB (PVC)
[22:04:23] [PASSED] 0xBE0 (PVC)
[22:04:23] [PASSED] 0xBE1 (PVC)
[22:04:23] [PASSED] 0xBE5 (PVC)
[22:04:23] [PASSED] 0x7D40 (METEORLAKE)
[22:04:23] [PASSED] 0x7D45 (METEORLAKE)
[22:04:23] [PASSED] 0x7D55 (METEORLAKE)
[22:04:23] [PASSED] 0x7D60 (METEORLAKE)
[22:04:23] [PASSED] 0x7DD5 (METEORLAKE)
[22:04:23] [PASSED] 0x6420 (LUNARLAKE)
[22:04:23] [PASSED] 0x64A0 (LUNARLAKE)
[22:04:23] [PASSED] 0x64B0 (LUNARLAKE)
[22:04:23] [PASSED] 0xE202 (BATTLEMAGE)
[22:04:23] [PASSED] 0xE209 (BATTLEMAGE)
[22:04:23] [PASSED] 0xE20B (BATTLEMAGE)
[22:04:23] [PASSED] 0xE20C (BATTLEMAGE)
[22:04:23] [PASSED] 0xE20D (BATTLEMAGE)
[22:04:23] [PASSED] 0xE210 (BATTLEMAGE)
[22:04:23] [PASSED] 0xE211 (BATTLEMAGE)
[22:04:23] [PASSED] 0xE212 (BATTLEMAGE)
[22:04:23] [PASSED] 0xE216 (BATTLEMAGE)
[22:04:23] [PASSED] 0xE220 (BATTLEMAGE)
[22:04:23] [PASSED] 0xE221 (BATTLEMAGE)
[22:04:23] [PASSED] 0xE222 (BATTLEMAGE)
[22:04:23] [PASSED] 0xE223 (BATTLEMAGE)
[22:04:23] [PASSED] 0xB080 (PANTHERLAKE)
[22:04:23] [PASSED] 0xB081 (PANTHERLAKE)
[22:04:23] [PASSED] 0xB082 (PANTHERLAKE)
[22:04:23] [PASSED] 0xB083 (PANTHERLAKE)
[22:04:23] [PASSED] 0xB084 (PANTHERLAKE)
[22:04:23] [PASSED] 0xB085 (PANTHERLAKE)
[22:04:23] [PASSED] 0xB086 (PANTHERLAKE)
[22:04:23] [PASSED] 0xB087 (PANTHERLAKE)
[22:04:23] [PASSED] 0xB08F (PANTHERLAKE)
[22:04:23] [PASSED] 0xB090 (PANTHERLAKE)
[22:04:23] [PASSED] 0xB0A0 (PANTHERLAKE)
[22:04:23] [PASSED] 0xB0B0 (PANTHERLAKE)
[22:04:23] [PASSED] 0xFD80 (PANTHERLAKE)
[22:04:23] [PASSED] 0xFD81 (PANTHERLAKE)
[22:04:23] [PASSED] 0xD740 (NOVALAKE_S)
[22:04:23] [PASSED] 0xD741 (NOVALAKE_S)
[22:04:23] [PASSED] 0xD742 (NOVALAKE_S)
[22:04:23] [PASSED] 0xD743 (NOVALAKE_S)
[22:04:23] [PASSED] 0xD745 (NOVALAKE_S)
[22:04:23] [PASSED] 0xD74A (NOVALAKE_S)
[22:04:23] [PASSED] 0xD74B (NOVALAKE_S)
[22:04:23] [PASSED] 0x674C (CRESCENTISLAND)
[22:04:23] [PASSED] 0x674D (CRESCENTISLAND)
[22:04:23] [PASSED] 0x674E (CRESCENTISLAND)
[22:04:23] [PASSED] 0x674F (CRESCENTISLAND)
[22:04:23] [PASSED] 0x6750 (CRESCENTISLAND)
[22:04:23] [PASSED] 0xD750 (NOVALAKE_P)
[22:04:23] [PASSED] 0xD751 (NOVALAKE_P)
[22:04:23] [PASSED] 0xD752 (NOVALAKE_P)
[22:04:23] [PASSED] 0xD753 (NOVALAKE_P)
[22:04:23] [PASSED] 0xD754 (NOVALAKE_P)
[22:04:23] [PASSED] 0xD755 (NOVALAKE_P)
[22:04:23] [PASSED] 0xD756 (NOVALAKE_P)
[22:04:23] [PASSED] 0xD757 (NOVALAKE_P)
[22:04:23] [PASSED] 0xD75F (NOVALAKE_P)
[22:04:23] =============== [PASSED] check_platform_desc ===============
[22:04:23] ===================== [PASSED] xe_pci ======================
[22:04:23] ============= xe_rtp_tables_test (5 subtests) ==============
[22:04:23] ================== xe_rtp_table_gt_test  ===================
[22:04:23] [PASSED] gt_was/14011060649
[22:04:23] [PASSED] gt_was/14011059788
[22:04:23] [PASSED] gt_was/14015795083
[22:04:23] [PASSED] gt_was/16021867713
[22:04:23] [PASSED] gt_was/14019449301
[22:04:23] [PASSED] gt_was/16028005424
[22:04:23] [PASSED] gt_was/14026578760
[22:04:23] [PASSED] gt_was/1409420604
[22:04:23] [PASSED] gt_was/1408615072
[22:04:23] [PASSED] gt_was/22010523718
[22:04:23] [PASSED] gt_was/14011006942
[22:04:23] [PASSED] gt_was/14014830051
[22:04:23] [PASSED] gt_was/18018781329
[22:04:23] [PASSED] gt_was/1509235366
[22:04:23] [PASSED] gt_was/18018781329
[22:04:23] [PASSED] gt_was/16016694945
[22:04:23] [PASSED] gt_was/14018575942
[22:04:23] [PASSED] gt_was/22016670082
[22:04:23] [PASSED] gt_was/22016670082
[22:04:23] [PASSED] gt_was/14017421178
[22:04:23] [PASSED] gt_was/16025250150
[22:04:23] [PASSED] gt_was/14021871409
[22:04:23] [PASSED] gt_was/16021865536
[22:04:23] [PASSED] gt_was/14021486841
[22:04:23] [PASSED] gt_was/14025160223
[22:04:23] [PASSED] gt_was/14026144927, 16029437861, 14026127056
[22:04:23] [PASSED] gt_was/14025635424
[22:04:23] [PASSED] gt_was/16028005424
[22:04:23] ============== [PASSED] xe_rtp_table_gt_test ===============
[22:04:23] ================== xe_rtp_table_gt_test  ===================
[22:04:23] [PASSED] gt_tunings/Tuning: Blend Fill Caching Optimization Disable
[22:04:23] [PASSED] gt_tunings/Tuning: 32B Access Enable
[22:04:23] [PASSED] gt_tunings/Tuning: L3 cache
[22:04:23] [PASSED] gt_tunings/Tuning: L3 cache - media
[22:04:23] [PASSED] gt_tunings/Tuning: Compression Overfetch
[22:04:23] [PASSED] gt_tunings/Tuning: Compression Overfetch - media
[22:04:23] [PASSED] gt_tunings/Tuning: Enable compressible partial write overfetch in L3
[22:04:23] [PASSED] gt_tunings/Tuning: Enable compressible partial write overfetch in L3 - media
[22:04:23] [PASSED] gt_tunings/Tuning: L2 Overfetch Compressible Only
[22:04:23] [PASSED] gt_tunings/Tuning: L2 Overfetch Compressible Only - media
[22:04:23] [PASSED] gt_tunings/Tuning: Stateless compression control
[22:04:23] [PASSED] gt_tunings/Tuning: Stateless compression control - media
[22:04:23] [PASSED] gt_tunings/Tuning: L3 RW flush all Cache
[22:04:23] [PASSED] gt_tunings/Tuning: L3 RW flush all cache - media
[22:04:23] [PASSED] gt_tunings/Tuning: Set STLB Bank Hash Mode to 4KB
[22:04:23] ============== [PASSED] xe_rtp_table_gt_test ===============
[22:04:23] ================== xe_rtp_table_oob_test  ==================
[22:04:23] [PASSED] oob_was/1607983814
[22:04:23] [PASSED] oob_was/16010904313
[22:04:23] [PASSED] oob_was/18022495364
[22:04:23] [PASSED] oob_was/22012773006
[22:04:23] [PASSED] oob_was/14014475959
[22:04:23] [PASSED] oob_was/22011391025
[22:04:23] [PASSED] oob_was/22012727170
[22:04:23] [PASSED] oob_was/22012727685
[22:04:23] [PASSED] oob_was/22016596838
[22:04:23] [PASSED] oob_was/18020744125
[22:04:23] [PASSED] oob_was/1409600907
[22:04:23] [PASSED] oob_was/22014953428
[22:04:23] [PASSED] oob_was/16017236439
[22:04:23] [PASSED] oob_was/14019821291
[22:04:23] [PASSED] oob_was/14015076503
[22:04:23] [PASSED] oob_was/14018913170
[22:04:23] [PASSED] oob_was/14018094691
[22:04:23] [PASSED] oob_was/18024947630
[22:04:23] [PASSED] oob_was/16022287689
[22:04:23] [PASSED] oob_was/13011645652
[22:04:23] [PASSED] oob_was/14022293748
[22:04:23] [PASSED] oob_was/22019794406
[22:04:23] [PASSED] oob_was/22019338487
[22:04:23] [PASSED] oob_was/16023588340
[22:04:23] [PASSED] oob_was/14019789679
[22:04:23] [PASSED] oob_was/14022866841
[22:04:23] [PASSED] oob_was/16021333562
[22:04:23] [PASSED] oob_was/14016712196
[22:04:23] [PASSED] oob_was/14015568240
[22:04:23] [PASSED] oob_was/18013179988
[22:04:23] [PASSED] oob_was/1508761755
[22:04:23] [PASSED] oob_was/16023105232
[22:04:23] [PASSED] oob_was/16026508708
[22:04:23] [PASSED] oob_was/14020001231
[22:04:23] [PASSED] oob_was/16023683509
[22:04:23] [PASSED] oob_was/14025515070
[22:04:23] [PASSED] oob_was/15015404425_disable
[22:04:23] [PASSED] oob_was/16026007364
[22:04:23] [PASSED] oob_was/14020316580
[22:04:23] [PASSED] oob_was/14025883347
[22:04:23] [PASSED] oob_was/16029380221
[22:04:23] [PASSED] oob_was/22022079272
[22:04:23] [PASSED] oob_was/16029897822
[22:04:23] ============== [PASSED] xe_rtp_table_oob_test ==============
[22:04:23] ================ xe_rtp_table_dev_oob_test  ================
[22:04:23] [PASSED] device_oob_was/22010954014
[22:04:23] [PASSED] device_oob_was/15015404425
[22:04:23] [PASSED] device_oob_was/22019338487_display
[22:04:23] [PASSED] device_oob_was/14022085890
[22:04:23] [PASSED] device_oob_was/14026539277
[22:04:23] [PASSED] device_oob_was/14026633728
[22:04:23] [PASSED] device_oob_was/14026746987
[22:04:23] [PASSED] device_oob_was/14026779378
[22:04:23] ============ [PASSED] xe_rtp_table_dev_oob_test ============
[22:04:23] ========== xe_rtp_table_missing_upper_bound_test  ==========
[22:04:23] [PASSED] register_whitelist/WaAllowPMDepthAndInvocationCountAccessFromUMD, 1408556865
[22:04:23] [PASSED] register_whitelist/1508744258, 14012131227, 1808121037
[22:04:23] [PASSED] register_whitelist/1806527549
[22:04:23] [PASSED] register_whitelist/allow_read_ctx_timestamp
[22:04:23] [PASSED] register_whitelist/allow_read_queue_timestamp
[22:04:23] [PASSED] register_whitelist/16014440446
[22:04:23] [PASSED] register_whitelist/16017236439
[22:04:23] [PASSED] register_whitelist/16020183090
[22:04:23] [PASSED] register_whitelist/14024997852
[22:04:23] [PASSED] register_whitelist/14024997852
[22:04:23] ====== [PASSED] xe_rtp_table_missing_upper_bound_test ======
[22:04:23] =============== [PASSED] xe_rtp_tables_test ================
[22:04:23] =================== xe_rtp (3 subtests) ====================
[22:04:23] =================== xe_rtp_rules_tests  ====================
[22:04:23] [PASSED] no
[22:04:23] [PASSED] yes
[22:04:23] [PASSED] no-and-no
[22:04:23] [PASSED] no-and-yes
[22:04:23] [PASSED] yes-and-no
[22:04:23] [PASSED] yes-and-yes
[22:04:23] [PASSED] no-or-no
[22:04:23] [PASSED] no-or-yes
[22:04:23] [PASSED] yes-or-no
[22:04:23] [PASSED] yes-or-yes
[22:04:23] [PASSED] no-yes-or-yes-no
[22:04:23] [PASSED] no-yes-or-yes-yes
[22:04:23] [PASSED] yes-yes-or-no-yes
[22:04:23] [PASSED] yes-yes-or-yes-yes
[22:04:23] [PASSED] no-no-or-yes-or-no
[22:04:23] [PASSED] or
[22:04:23] [PASSED] or-yes
[22:04:23] [PASSED] or-no
[22:04:23] [PASSED] yes-or
[22:04:23] [PASSED] no-or
[22:04:23] [PASSED] no-or-or-yes
[22:04:23] [PASSED] yes-or-or-no
[22:04:23] [PASSED] no-or-or-no
[22:04:23] [PASSED] missing-context-engine-class
[22:04:23] [PASSED] missing-context-engine-class-or-yes
[22:04:23] [PASSED] missing-context-engine-class-or-or-yes
[22:04:23] =============== [PASSED] xe_rtp_rules_tests ================
[22:04:23] =============== xe_rtp_process_to_sr_tests  ================
[22:04:23] [PASSED] coalesce-same-reg
[22:04:23] [PASSED] coalesce-same-reg-literal-and-func
[22:04:23] [PASSED] no-match-no-add
[22:04:23] [PASSED] two-regs-two-entries
[22:04:23] [PASSED] clr-one-set-other
[22:04:23] [PASSED] set-field
[22:04:23] [PASSED] conflict-duplicate
[22:04:23] [PASSED] conflict-not-disjoint
[22:04:23] [PASSED] conflict-not-disjoint-literal-and-func
[22:04:23] [PASSED] conflict-reg-type
[22:04:23] [PASSED] bad-mcr-reg-forced-to-regular
[22:04:23] [PASSED] bad-regular-reg-forced-to-mcr
[22:04:23] =========== [PASSED] xe_rtp_process_to_sr_tests ============
[22:04:23] ================== xe_rtp_process_tests  ===================
[22:04:23] [PASSED] active1
[22:04:23] [PASSED] active2
[22:04:23] [PASSED] active-inactive
[22:04:23] [PASSED] inactive-active
[22:04:23] [PASSED] inactive-active-inactive
[22:04:23] [PASSED] inactive-inactive-inactive
[22:04:23] ============== [PASSED] xe_rtp_process_tests ===============
[22:04:23] ===================== [PASSED] xe_rtp ======================
[22:04:23] ==================== xe_wa (1 subtest) =====================
[22:04:23] ======================== xe_wa_gt  =========================
[22:04:23] [PASSED] TIGERLAKE B0
[22:04:23] [PASSED] DG1 A0
[22:04:23] [PASSED] DG1 B0
[22:04:23] [PASSED] ALDERLAKE_S A0
[22:04:23] [PASSED] ALDERLAKE_S B0
[22:04:23] [PASSED] ALDERLAKE_S C0
[22:04:23] [PASSED] ALDERLAKE_S D0
[22:04:23] [PASSED] ALDERLAKE_P A0
[22:04:23] [PASSED] ALDERLAKE_P B0
[22:04:23] [PASSED] ALDERLAKE_P C0
[22:04:23] [PASSED] ALDERLAKE_S RPLS D0
[22:04:23] [PASSED] ALDERLAKE_P RPLU E0
[22:04:23] [PASSED] DG2 G10 C0
[22:04:23] [PASSED] DG2 G11 B1
[22:04:23] [PASSED] DG2 G12 A1
[22:04:23] [PASSED] METEORLAKE 12.70(Xe_LPG) A0 13.00(Xe_LPM+) A0
[22:04:23] [PASSED] METEORLAKE 12.71(Xe_LPG) A0 13.00(Xe_LPM+) A0
[22:04:23] [PASSED] METEORLAKE 12.74(Xe_LPG+) A0 13.00(Xe_LPM+) A0
[22:04:23] [PASSED] LUNARLAKE 20.04(Xe2_LPG) A0 20.00(Xe2_LPM) A0
[22:04:23] [PASSED] LUNARLAKE 20.04(Xe2_LPG) B0 20.00(Xe2_LPM) A0
[22:04:23] [PASSED] BATTLEMAGE 20.01(Xe2_HPG) A0 13.01(Xe2_HPM) A1
[22:04:23] [PASSED] PANTHERLAKE 30.00(Xe3_LPG) A0 30.00(Xe3_LPM) A0
[22:04:23] ==================== [PASSED] xe_wa_gt =====================
[22:04:23] ====================== [PASSED] xe_wa ======================
[22:04:23] ============================================================
[22:04:23] Testing complete. Ran 731 tests: passed: 713, skipped: 18
[22:04:23] Elapsed time: 36.930s total, 4.438s configuring, 31.826s building, 0.654s running

+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/tests/.kunitconfig [22:04:23] Configuring KUnit Kernel ...
Regenerating .config ...
Populating config with:

$ make ARCH=um O=.kunit olddefconfig
[22:04:25] Building KUnit Kernel ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
Building with:
$ make all compile_commands.json scripts_gdb ARCH=um O=.kunit --jobs=48
[22:04:50] Starting KUnit Kernel (1/1)...
[22:04:50] ============================================================
Running tests with:
$ .kunit/linux kunit.enable=1 mem=1G console=tty kunit_shutdown=halt
[22:04:50] ============ drm_test_pick_cmdline (2 subtests) ============
[22:04:50] [PASSED] drm_test_pick_cmdline_res_1920_1080_60
[22:04:50] =============== drm_test_pick_cmdline_named  ===============
[22:04:50] [PASSED] NTSC
[22:04:50] [PASSED] NTSC-J
[22:04:50] [PASSED] PAL
[22:04:50] [PASSED] PAL-M
[22:04:50] =========== [PASSED] drm_test_pick_cmdline_named ===========
[22:04:50] ============== [PASSED] drm_test_pick_cmdline ==============
[22:04:50] == drm_test_atomic_get_connector_for_encoder (1 subtest) ===
[22:04:50] [PASSED] drm_test_drm_atomic_get_connector_for_encoder
[22:04:50] ==== [PASSED] drm_test_atomic_get_connector_for_encoder ====
[22:04:50] =========== drm_validate_clone_mode (2 subtests) ===========
[22:04:50] ============== drm_test_check_in_clone_mode  ===============
[22:04:50] [PASSED] in_clone_mode
[22:04:50] [PASSED] not_in_clone_mode
[22:04:50] ========== [PASSED] drm_test_check_in_clone_mode ===========
[22:04:50] =============== drm_test_check_valid_clones  ===============
[22:04:50] [PASSED] not_in_clone_mode
[22:04:50] [PASSED] valid_clone
[22:04:50] [PASSED] invalid_clone
[22:04:50] =========== [PASSED] drm_test_check_valid_clones ===========
[22:04:50] ============= [PASSED] drm_validate_clone_mode =============
[22:04:50] ============= drm_validate_modeset (1 subtest) =============
[22:04:50] [PASSED] drm_test_check_connector_changed_modeset
[22:04:50] ============== [PASSED] drm_validate_modeset ===============
[22:04:50] ====== drm_test_bridge_get_current_state (2 subtests) ======
[22:04:50] [PASSED] drm_test_drm_bridge_get_current_state_atomic
[22:04:50] [PASSED] drm_test_drm_bridge_get_current_state_legacy
[22:04:50] ======== [PASSED] drm_test_bridge_get_current_state ========
[22:04:50] ====== drm_test_bridge_helper_reset_crtc (4 subtests) ======
[22:04:50] [PASSED] drm_test_drm_bridge_helper_reset_crtc_atomic
[22:04:50] [PASSED] drm_test_drm_bridge_helper_reset_crtc_atomic_disabled
[22:04:50] [PASSED] drm_test_drm_bridge_helper_reset_crtc_legacy
[22:04:50] [PASSED] drm_test_drm_bridge_helper_hdmi_output_bus_fmts
[22:04:50] ======== [PASSED] drm_test_bridge_helper_reset_crtc ========
[22:04:50] ============== drm_bridge_alloc (2 subtests) ===============
[22:04:50] [PASSED] drm_test_drm_bridge_alloc_basic
[22:04:50] [PASSED] drm_test_drm_bridge_alloc_get_put
[22:04:50] ================ [PASSED] drm_bridge_alloc =================
[22:04:50] ============= drm_bridge_bus_fmt (5 subtests) ==============
[22:04:50] [PASSED] drm_test_bridge_rgb_yuv_rgb
[22:04:50] [PASSED] drm_test_bridge_must_convert_to_yuv444
[22:04:50] [PASSED] drm_test_bridge_hdmi_auto_rgb
[22:04:50] [PASSED] drm_test_bridge_auto_first
[22:04:50] [PASSED] drm_test_bridge_rgb_yuv_no_path
[22:04:50] =============== [PASSED] drm_bridge_bus_fmt ================
[22:04:50] ============= drm_cmdline_parser (40 subtests) =============
[22:04:50] [PASSED] drm_test_cmdline_force_d_only
[22:04:50] [PASSED] drm_test_cmdline_force_D_only_dvi
[22:04:50] [PASSED] drm_test_cmdline_force_D_only_hdmi
[22:04:50] [PASSED] drm_test_cmdline_force_D_only_not_digital
[22:04:50] [PASSED] drm_test_cmdline_force_e_only
[22:04:50] [PASSED] drm_test_cmdline_res
[22:04:50] [PASSED] drm_test_cmdline_res_vesa
[22:04:50] [PASSED] drm_test_cmdline_res_vesa_rblank
[22:04:50] [PASSED] drm_test_cmdline_res_rblank
[22:04:50] [PASSED] drm_test_cmdline_res_bpp
[22:04:50] [PASSED] drm_test_cmdline_res_refresh
[22:04:50] [PASSED] drm_test_cmdline_res_bpp_refresh
[22:04:50] [PASSED] drm_test_cmdline_res_bpp_refresh_interlaced
[22:04:50] [PASSED] drm_test_cmdline_res_bpp_refresh_margins
[22:04:50] [PASSED] drm_test_cmdline_res_bpp_refresh_force_off
[22:04:50] [PASSED] drm_test_cmdline_res_bpp_refresh_force_on
[22:04:50] [PASSED] drm_test_cmdline_res_bpp_refresh_force_on_analog
[22:04:50] [PASSED] drm_test_cmdline_res_bpp_refresh_force_on_digital
[22:04:50] [PASSED] drm_test_cmdline_res_bpp_refresh_interlaced_margins_force_on
[22:04:50] [PASSED] drm_test_cmdline_res_margins_force_on
[22:04:50] [PASSED] drm_test_cmdline_res_vesa_margins
[22:04:50] [PASSED] drm_test_cmdline_name
[22:04:50] [PASSED] drm_test_cmdline_name_bpp
[22:04:50] [PASSED] drm_test_cmdline_name_option
[22:04:50] [PASSED] drm_test_cmdline_name_bpp_option
[22:04:50] [PASSED] drm_test_cmdline_rotate_0
[22:04:50] [PASSED] drm_test_cmdline_rotate_90
[22:04:50] [PASSED] drm_test_cmdline_rotate_180
[22:04:50] [PASSED] drm_test_cmdline_rotate_270
[22:04:50] [PASSED] drm_test_cmdline_hmirror
[22:04:50] [PASSED] drm_test_cmdline_vmirror
[22:04:50] [PASSED] drm_test_cmdline_margin_options
[22:04:50] [PASSED] drm_test_cmdline_multiple_options
[22:04:50] [PASSED] drm_test_cmdline_bpp_extra_and_option
[22:04:50] [PASSED] drm_test_cmdline_extra_and_option
[22:04:50] [PASSED] drm_test_cmdline_freestanding_options
[22:04:50] [PASSED] drm_test_cmdline_freestanding_force_e_and_options
[22:04:50] [PASSED] drm_test_cmdline_panel_orientation
[22:04:50] ================ drm_test_cmdline_invalid  =================
[22:04:50] [PASSED] margin_only
[22:04:50] [PASSED] interlace_only
[22:04:50] [PASSED] res_missing_x
[22:04:50] [PASSED] res_missing_y
[22:04:50] [PASSED] res_bad_y
[22:04:50] [PASSED] res_missing_y_bpp
[22:04:50] [PASSED] res_bad_bpp
[22:04:50] [PASSED] res_bad_refresh
[22:04:50] [PASSED] res_bpp_refresh_force_on_off
[22:04:50] [PASSED] res_invalid_mode
[22:04:50] [PASSED] res_bpp_wrong_place_mode
[22:04:50] [PASSED] name_bpp_refresh
[22:04:50] [PASSED] name_refresh
[22:04:50] [PASSED] name_refresh_wrong_mode
[22:04:50] [PASSED] name_refresh_invalid_mode
[22:04:50] [PASSED] rotate_multiple
[22:04:50] [PASSED] rotate_invalid_val
[22:04:50] [PASSED] rotate_truncated
[22:04:50] [PASSED] invalid_option
[22:04:50] [PASSED] invalid_tv_option
[22:04:50] [PASSED] truncated_tv_option
[22:04:50] ============ [PASSED] drm_test_cmdline_invalid =============
[22:04:50] =============== drm_test_cmdline_tv_options  ===============
[22:04:50] [PASSED] NTSC
[22:04:50] [PASSED] NTSC_443
[22:04:50] [PASSED] NTSC_J
[22:04:50] [PASSED] PAL
[22:04:50] [PASSED] PAL_M
[22:04:50] [PASSED] PAL_N
[22:04:50] [PASSED] SECAM
[22:04:50] [PASSED] MONO_525
[22:04:50] [PASSED] MONO_625
[22:04:50] =========== [PASSED] drm_test_cmdline_tv_options ===========
[22:04:50] =============== [PASSED] drm_cmdline_parser ================
[22:04:50] ========== drmm_connector_hdmi_init (20 subtests) ==========
[22:04:50] [PASSED] drm_test_connector_hdmi_init_valid
[22:04:50] [PASSED] drm_test_connector_hdmi_init_bpc_8
[22:04:50] [PASSED] drm_test_connector_hdmi_init_bpc_10
[22:04:50] [PASSED] drm_test_connector_hdmi_init_bpc_12
[22:04:50] [PASSED] drm_test_connector_hdmi_init_bpc_invalid
[22:04:50] [PASSED] drm_test_connector_hdmi_init_bpc_null
[22:04:50] [PASSED] drm_test_connector_hdmi_init_formats_empty
[22:04:50] [PASSED] drm_test_connector_hdmi_init_formats_no_rgb
[22:04:50] === drm_test_connector_hdmi_init_formats_yuv420_allowed  ===
[22:04:50] [PASSED] supported_formats=0x9 yuv420_allowed=1
[22:04:50] [PASSED] supported_formats=0x9 yuv420_allowed=0
[22:04:50] [PASSED] supported_formats=0x5 yuv420_allowed=1
[22:04:50] [PASSED] supported_formats=0x5 yuv420_allowed=0
[22:04:50] === [PASSED] drm_test_connector_hdmi_init_formats_yuv420_allowed ===
[22:04:50] [PASSED] drm_test_connector_hdmi_init_null_ddc
[22:04:50] [PASSED] drm_test_connector_hdmi_init_null_product
[22:04:50] [PASSED] drm_test_connector_hdmi_init_null_vendor
[22:04:50] [PASSED] drm_test_connector_hdmi_init_product_length_exact
[22:04:50] [PASSED] drm_test_connector_hdmi_init_product_length_too_long
[22:04:50] [PASSED] drm_test_connector_hdmi_init_product_valid
[22:04:50] [PASSED] drm_test_connector_hdmi_init_vendor_length_exact
[22:04:50] [PASSED] drm_test_connector_hdmi_init_vendor_length_too_long
[22:04:50] [PASSED] drm_test_connector_hdmi_init_vendor_valid
[22:04:50] ========= drm_test_connector_hdmi_init_type_valid  =========
[22:04:50] [PASSED] HDMI-A
[22:04:50] [PASSED] HDMI-B
[22:04:50] ===== [PASSED] drm_test_connector_hdmi_init_type_valid =====
[22:04:50] ======== drm_test_connector_hdmi_init_type_invalid  ========
[22:04:50] [PASSED] Unknown
[22:04:50] [PASSED] VGA
[22:04:50] [PASSED] DVI-I
[22:04:50] [PASSED] DVI-D
[22:04:50] [PASSED] DVI-A
[22:04:50] [PASSED] Composite
[22:04:50] [PASSED] SVIDEO
[22:04:50] [PASSED] LVDS
[22:04:50] [PASSED] Component
[22:04:50] [PASSED] DIN
[22:04:50] [PASSED] DP
[22:04:50] [PASSED] TV
[22:04:50] [PASSED] eDP
[22:04:50] [PASSED] Virtual
[22:04:50] [PASSED] DSI
[22:04:50] [PASSED] DPI
[22:04:50] [PASSED] Writeback
[22:04:50] [PASSED] SPI
[22:04:50] [PASSED] USB
[22:04:50] ==== [PASSED] drm_test_connector_hdmi_init_type_invalid ====
[22:04:50] ============ [PASSED] drmm_connector_hdmi_init =============
[22:04:50] ============= drmm_connector_init (3 subtests) =============
[22:04:50] [PASSED] drm_test_drmm_connector_init
[22:04:50] [PASSED] drm_test_drmm_connector_init_null_ddc
[22:04:50] ========= drm_test_drmm_connector_init_type_valid  =========
[22:04:50] [PASSED] Unknown
[22:04:50] [PASSED] VGA
[22:04:50] [PASSED] DVI-I
[22:04:50] [PASSED] DVI-D
[22:04:50] [PASSED] DVI-A
[22:04:50] [PASSED] Composite
[22:04:50] [PASSED] SVIDEO
[22:04:50] [PASSED] LVDS
[22:04:50] [PASSED] Component
[22:04:50] [PASSED] DIN
[22:04:50] [PASSED] DP
[22:04:50] [PASSED] HDMI-A
[22:04:50] [PASSED] HDMI-B
[22:04:50] [PASSED] TV
[22:04:50] [PASSED] eDP
[22:04:50] [PASSED] Virtual
[22:04:50] [PASSED] DSI
[22:04:50] [PASSED] DPI
[22:04:50] [PASSED] Writeback
[22:04:50] [PASSED] SPI
[22:04:50] [PASSED] USB
[22:04:50] ===== [PASSED] drm_test_drmm_connector_init_type_valid =====
[22:04:50] =============== [PASSED] drmm_connector_init ===============
[22:04:50] ========= drm_connector_dynamic_init (6 subtests) ==========
[22:04:50] [PASSED] drm_test_drm_connector_dynamic_init
[22:04:50] [PASSED] drm_test_drm_connector_dynamic_init_null_ddc
[22:04:50] [PASSED] drm_test_drm_connector_dynamic_init_not_added
[22:04:50] [PASSED] drm_test_drm_connector_dynamic_init_properties
[22:04:50] ===== drm_test_drm_connector_dynamic_init_type_valid  ======
[22:04:50] [PASSED] Unknown
[22:04:50] [PASSED] VGA
[22:04:50] [PASSED] DVI-I
[22:04:50] [PASSED] DVI-D
[22:04:50] [PASSED] DVI-A
[22:04:50] [PASSED] Composite
[22:04:50] [PASSED] SVIDEO
[22:04:50] [PASSED] LVDS
[22:04:50] [PASSED] Component
[22:04:50] [PASSED] DIN
[22:04:50] [PASSED] DP
[22:04:50] [PASSED] HDMI-A
[22:04:50] [PASSED] HDMI-B
[22:04:50] [PASSED] TV
[22:04:50] [PASSED] eDP
[22:04:50] [PASSED] Virtual
[22:04:50] [PASSED] DSI
[22:04:50] [PASSED] DPI
[22:04:50] [PASSED] Writeback
[22:04:50] [PASSED] SPI
[22:04:50] [PASSED] USB
[22:04:50] = [PASSED] drm_test_drm_connector_dynamic_init_type_valid ==
[22:04:50] ======== drm_test_drm_connector_dynamic_init_name  =========
[22:04:50] [PASSED] Unknown
[22:04:50] [PASSED] VGA
[22:04:50] [PASSED] DVI-I
[22:04:50] [PASSED] DVI-D
[22:04:50] [PASSED] DVI-A
[22:04:50] [PASSED] Composite
[22:04:50] [PASSED] SVIDEO
[22:04:50] [PASSED] LVDS
[22:04:50] [PASSED] Component
[22:04:50] [PASSED] DIN
[22:04:50] [PASSED] DP
[22:04:50] [PASSED] HDMI-A
[22:04:50] [PASSED] HDMI-B
[22:04:50] [PASSED] TV
[22:04:50] [PASSED] eDP
[22:04:50] [PASSED] Virtual
[22:04:50] [PASSED] DSI
[22:04:50] [PASSED] DPI
[22:04:50] [PASSED] Writeback
[22:04:50] [PASSED] SPI
[22:04:50] [PASSED] USB
[22:04:50] ==== [PASSED] drm_test_drm_connector_dynamic_init_name =====
[22:04:50] =========== [PASSED] drm_connector_dynamic_init ============
[22:04:50] ==== drm_connector_dynamic_register_early (4 subtests) =====
[22:04:50] [PASSED] drm_test_drm_connector_dynamic_register_early_on_list
[22:04:50] [PASSED] drm_test_drm_connector_dynamic_register_early_defer
[22:04:50] [PASSED] drm_test_drm_connector_dynamic_register_early_no_init
[22:04:50] [PASSED] drm_test_drm_connector_dynamic_register_early_no_mode_object
[22:04:50] ====== [PASSED] drm_connector_dynamic_register_early =======
[22:04:50] ======= drm_connector_dynamic_register (7 subtests) ========
[22:04:50] [PASSED] drm_test_drm_connector_dynamic_register_on_list
[22:04:50] [PASSED] drm_test_drm_connector_dynamic_register_no_defer
[22:04:50] [PASSED] drm_test_drm_connector_dynamic_register_no_init
[22:04:50] [PASSED] drm_test_drm_connector_dynamic_register_mode_object
[22:04:50] [PASSED] drm_test_drm_connector_dynamic_register_sysfs
[22:04:50] [PASSED] drm_test_drm_connector_dynamic_register_sysfs_name
[22:04:50] [PASSED] drm_test_drm_connector_dynamic_register_debugfs
[22:04:50] ========= [PASSED] drm_connector_dynamic_register ==========
[22:04:50] = drm_connector_attach_broadcast_rgb_property (2 subtests) =
[22:04:50] [PASSED] drm_test_drm_connector_attach_broadcast_rgb_property
[22:04:50] [PASSED] drm_test_drm_connector_attach_broadcast_rgb_property_hdmi_connector
[22:04:50] === [PASSED] drm_connector_attach_broadcast_rgb_property ===
[22:04:50] ========== drm_get_tv_mode_from_name (2 subtests) ==========
[22:04:50] ========== drm_test_get_tv_mode_from_name_valid  ===========
[22:04:50] [PASSED] NTSC
[22:04:50] [PASSED] NTSC-443
[22:04:50] [PASSED] NTSC-J
[22:04:50] [PASSED] PAL
[22:04:50] [PASSED] PAL-M
[22:04:50] [PASSED] PAL-N
[22:04:50] [PASSED] SECAM
[22:04:50] [PASSED] Mono
[22:04:50] ====== [PASSED] drm_test_get_tv_mode_from_name_valid =======
[22:04:50] [PASSED] drm_test_get_tv_mode_from_name_truncated
[22:04:50] ============ [PASSED] drm_get_tv_mode_from_name ============
[22:04:50] = drm_test_connector_hdmi_compute_mode_clock (12 subtests) =
[22:04:50] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb
[22:04:50] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb_10bpc
[22:04:50] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb_10bpc_vic_1
[22:04:50] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb_12bpc
[22:04:50] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb_12bpc_vic_1
[22:04:50] [PASSED] drm_test_drm_hdmi_compute_mode_clock_rgb_double
[22:04:50] = drm_test_connector_hdmi_compute_mode_clock_yuv420_valid  =
[22:04:50] [PASSED] VIC 96
[22:04:50] [PASSED] VIC 97
[22:04:50] [PASSED] VIC 101
[22:04:50] [PASSED] VIC 102
[22:04:50] [PASSED] VIC 106
[22:04:50] [PASSED] VIC 107
[22:04:50] === [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv420_valid ===
[22:04:50] [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv420_10_bpc
[22:04:50] [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv420_12_bpc
[22:04:50] [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv422_8_bpc
[22:04:50] [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv422_10_bpc
[22:04:50] [PASSED] drm_test_connector_hdmi_compute_mode_clock_yuv422_12_bpc
[22:04:50] === [PASSED] drm_test_connector_hdmi_compute_mode_clock ====
[22:04:50] == drm_hdmi_connector_get_broadcast_rgb_name (2 subtests) ==
[22:04:50] === drm_test_drm_hdmi_connector_get_broadcast_rgb_name  ====
[22:04:50] [PASSED] Automatic
[22:04:50] [PASSED] Full
[22:04:50] [PASSED] Limited 16:235
[22:04:50] === [PASSED] drm_test_drm_hdmi_connector_get_broadcast_rgb_name ===
[22:04:50] [PASSED] drm_test_drm_hdmi_connector_get_broadcast_rgb_name_invalid
[22:04:50] ==== [PASSED] drm_hdmi_connector_get_broadcast_rgb_name ====
[22:04:50] == drm_hdmi_connector_get_output_format_name (2 subtests) ==
[22:04:50] === drm_test_drm_hdmi_connector_get_output_format_name  ====
[22:04:50] [PASSED] RGB
[22:04:50] [PASSED] YUV 4:2:0
[22:04:50] [PASSED] YUV 4:2:2
[22:04:50] [PASSED] YUV 4:4:4
[22:04:50] === [PASSED] drm_test_drm_hdmi_connector_get_output_format_name ===
[22:04:50] [PASSED] drm_test_drm_hdmi_connector_get_output_format_name_invalid
[22:04:50] ==== [PASSED] drm_hdmi_connector_get_output_format_name ====
[22:04:50] ============= drm_damage_helper (21 subtests) ==============
[22:04:50] [PASSED] drm_test_damage_iter_no_damage
[22:04:50] [PASSED] drm_test_damage_iter_no_damage_fractional_src
[22:04:50] [PASSED] drm_test_damage_iter_no_damage_src_moved
[22:04:50] [PASSED] drm_test_damage_iter_no_damage_fractional_src_moved
[22:04:50] [PASSED] drm_test_damage_iter_no_damage_not_visible
[22:04:50] [PASSED] drm_test_damage_iter_no_damage_no_crtc
[22:04:50] [PASSED] drm_test_damage_iter_no_damage_no_fb
[22:04:50] [PASSED] drm_test_damage_iter_simple_damage
[22:04:50] [PASSED] drm_test_damage_iter_single_damage
[22:04:50] [PASSED] drm_test_damage_iter_single_damage_intersect_src
[22:04:50] [PASSED] drm_test_damage_iter_single_damage_outside_src
[22:04:50] [PASSED] drm_test_damage_iter_single_damage_fractional_src
[22:04:50] [PASSED] drm_test_damage_iter_single_damage_intersect_fractional_src
[22:04:50] [PASSED] drm_test_damage_iter_single_damage_outside_fractional_src
[22:04:50] [PASSED] drm_test_damage_iter_single_damage_src_moved
[22:04:50] [PASSED] drm_test_damage_iter_single_damage_fractional_src_moved
[22:04:50] [PASSED] drm_test_damage_iter_damage
[22:04:50] [PASSED] drm_test_damage_iter_damage_one_intersect
[22:04:50] [PASSED] drm_test_damage_iter_damage_one_outside
[22:04:50] [PASSED] drm_test_damage_iter_damage_src_moved
[22:04:50] [PASSED] drm_test_damage_iter_damage_not_visible
[22:04:50] ================ [PASSED] drm_damage_helper ================
[22:04:50] ============== drm_dp_mst_helper (3 subtests) ==============
[22:04:50] ============== drm_test_dp_mst_calc_pbn_mode  ==============
[22:04:50] [PASSED] Clock 154000 BPP 30 DSC disabled
[22:04:50] [PASSED] Clock 234000 BPP 30 DSC disabled
[22:04:50] [PASSED] Clock 297000 BPP 24 DSC disabled
[22:04:50] [PASSED] Clock 332880 BPP 24 DSC enabled
[22:04:50] [PASSED] Clock 324540 BPP 24 DSC enabled
[22:04:50] ========== [PASSED] drm_test_dp_mst_calc_pbn_mode ==========
[22:04:50] ============== drm_test_dp_mst_calc_pbn_div  ===============
[22:04:50] [PASSED] Link rate 2000000 lane count 4
[22:04:50] [PASSED] Link rate 2000000 lane count 2
[22:04:50] [PASSED] Link rate 2000000 lane count 1
[22:04:50] [PASSED] Link rate 1350000 lane count 4
[22:04:50] [PASSED] Link rate 1350000 lane count 2
[22:04:50] [PASSED] Link rate 1350000 lane count 1
[22:04:50] [PASSED] Link rate 1000000 lane count 4
[22:04:50] [PASSED] Link rate 1000000 lane count 2
[22:04:50] [PASSED] Link rate 1000000 lane count 1
[22:04:50] [PASSED] Link rate 810000 lane count 4
[22:04:50] [PASSED] Link rate 810000 lane count 2
[22:04:50] [PASSED] Link rate 810000 lane count 1
[22:04:50] [PASSED] Link rate 540000 lane count 4
[22:04:50] [PASSED] Link rate 540000 lane count 2
[22:04:50] [PASSED] Link rate 540000 lane count 1
[22:04:50] [PASSED] Link rate 270000 lane count 4
[22:04:50] [PASSED] Link rate 270000 lane count 2
[22:04:50] [PASSED] Link rate 270000 lane count 1
[22:04:50] [PASSED] Link rate 162000 lane count 4
[22:04:50] [PASSED] Link rate 162000 lane count 2
[22:04:50] [PASSED] Link rate 162000 lane count 1
[22:04:50] ========== [PASSED] drm_test_dp_mst_calc_pbn_div ===========
[22:04:50] ========= drm_test_dp_mst_sideband_msg_req_decode  =========
[22:04:50] [PASSED] DP_ENUM_PATH_RESOURCES with port number
[22:04:50] [PASSED] DP_POWER_UP_PHY with port number
[22:04:50] [PASSED] DP_POWER_DOWN_PHY with port number
[22:04:50] [PASSED] DP_ALLOCATE_PAYLOAD with SDP stream sinks
[22:04:50] [PASSED] DP_ALLOCATE_PAYLOAD with port number
[22:04:50] [PASSED] DP_ALLOCATE_PAYLOAD with VCPI
[22:04:50] [PASSED] DP_ALLOCATE_PAYLOAD with PBN
[22:04:50] [PASSED] DP_QUERY_PAYLOAD with port number
[22:04:50] [PASSED] DP_QUERY_PAYLOAD with VCPI
[22:04:50] [PASSED] DP_REMOTE_DPCD_READ with port number
[22:04:50] [PASSED] DP_REMOTE_DPCD_READ with DPCD address
[22:04:50] [PASSED] DP_REMOTE_DPCD_READ with max number of bytes
[22:04:50] [PASSED] DP_REMOTE_DPCD_WRITE with port number
[22:04:50] [PASSED] DP_REMOTE_DPCD_WRITE with DPCD address
[22:04:50] [PASSED] DP_REMOTE_DPCD_WRITE with data array
[22:04:50] [PASSED] DP_REMOTE_I2C_READ with port number
[22:04:50] [PASSED] DP_REMOTE_I2C_READ with I2C device ID
[22:04:50] [PASSED] DP_REMOTE_I2C_READ with transactions array
[22:04:50] [PASSED] DP_REMOTE_I2C_WRITE with port number
[22:04:50] [PASSED] DP_REMOTE_I2C_WRITE with I2C device ID
[22:04:50] [PASSED] DP_REMOTE_I2C_WRITE with data array
[22:04:50] [PASSED] DP_QUERY_STREAM_ENC_STATUS with stream ID
[22:04:50] [PASSED] DP_QUERY_STREAM_ENC_STATUS with client ID
[22:04:50] [PASSED] DP_QUERY_STREAM_ENC_STATUS with stream event
[22:04:50] [PASSED] DP_QUERY_STREAM_ENC_STATUS with valid stream event
[22:04:50] [PASSED] DP_QUERY_STREAM_ENC_STATUS with stream behavior
[22:04:50] [PASSED] DP_QUERY_STREAM_ENC_STATUS with a valid stream behavior
[22:04:50] ===== [PASSED] drm_test_dp_mst_sideband_msg_req_decode =====
[22:04:50] ================ [PASSED] drm_dp_mst_helper ================
[22:04:50] ================== drm_exec (7 subtests) ===================
[22:04:50] [PASSED] sanitycheck
[22:04:50] [PASSED] test_lock
[22:04:50] [PASSED] test_lock_unlock
[22:04:50] [PASSED] test_duplicates
[22:04:50] [PASSED] test_prepare
[22:04:50] [PASSED] test_prepare_array
[22:04:50] [PASSED] test_multiple_loops
[22:04:50] ==================== [PASSED] drm_exec =====================
[22:04:50] =========== drm_format_helper_test (17 subtests) ===========
[22:04:50] ============== drm_test_fb_xrgb8888_to_gray8  ==============
[22:04:50] [PASSED] single_pixel_source_buffer
[22:04:50] [PASSED] single_pixel_clip_rectangle
[22:04:50] [PASSED] well_known_colors
[22:04:50] [PASSED] destination_pitch
[22:04:50] ========== [PASSED] drm_test_fb_xrgb8888_to_gray8 ==========
[22:04:50] ============= drm_test_fb_xrgb8888_to_rgb332  ==============
[22:04:50] [PASSED] single_pixel_source_buffer
[22:04:50] [PASSED] single_pixel_clip_rectangle
[22:04:50] [PASSED] well_known_colors
[22:04:50] [PASSED] destination_pitch
[22:04:50] ========= [PASSED] drm_test_fb_xrgb8888_to_rgb332 ==========
[22:04:50] ============= drm_test_fb_xrgb8888_to_rgb565  ==============
[22:04:50] [PASSED] single_pixel_source_buffer
[22:04:50] [PASSED] single_pixel_clip_rectangle
[22:04:50] [PASSED] well_known_colors
[22:04:50] [PASSED] destination_pitch
[22:04:50] ========= [PASSED] drm_test_fb_xrgb8888_to_rgb565 ==========
[22:04:50] ============ drm_test_fb_xrgb8888_to_xrgb1555  =============
[22:04:50] [PASSED] single_pixel_source_buffer
[22:04:50] [PASSED] single_pixel_clip_rectangle
[22:04:50] [PASSED] well_known_colors
[22:04:50] [PASSED] destination_pitch
[22:04:50] ======== [PASSED] drm_test_fb_xrgb8888_to_xrgb1555 =========
[22:04:50] ============ drm_test_fb_xrgb8888_to_argb1555  =============
[22:04:50] [PASSED] single_pixel_source_buffer
[22:04:50] [PASSED] single_pixel_clip_rectangle
[22:04:50] [PASSED] well_known_colors
[22:04:50] [PASSED] destination_pitch
[22:04:50] ======== [PASSED] drm_test_fb_xrgb8888_to_argb1555 =========
[22:04:50] ============ drm_test_fb_xrgb8888_to_rgba5551  =============
[22:04:50] [PASSED] single_pixel_source_buffer
[22:04:50] [PASSED] single_pixel_clip_rectangle
[22:04:50] [PASSED] well_known_colors
[22:04:50] [PASSED] destination_pitch
[22:04:50] ======== [PASSED] drm_test_fb_xrgb8888_to_rgba5551 =========
[22:04:50] ============= drm_test_fb_xrgb8888_to_rgb888  ==============
[22:04:50] [PASSED] single_pixel_source_buffer
[22:04:50] [PASSED] single_pixel_clip_rectangle
[22:04:50] [PASSED] well_known_colors
[22:04:50] [PASSED] destination_pitch
[22:04:50] ========= [PASSED] drm_test_fb_xrgb8888_to_rgb888 ==========
[22:04:50] ============= drm_test_fb_xrgb8888_to_bgr888  ==============
[22:04:50] [PASSED] single_pixel_source_buffer
[22:04:50] [PASSED] single_pixel_clip_rectangle
[22:04:50] [PASSED] well_known_colors
[22:04:50] [PASSED] destination_pitch
[22:04:50] ========= [PASSED] drm_test_fb_xrgb8888_to_bgr888 ==========
[22:04:50] ============ drm_test_fb_xrgb8888_to_argb8888  =============
[22:04:50] [PASSED] single_pixel_source_buffer
[22:04:50] [PASSED] single_pixel_clip_rectangle
[22:04:50] [PASSED] well_known_colors
[22:04:50] [PASSED] destination_pitch
[22:04:50] ======== [PASSED] drm_test_fb_xrgb8888_to_argb8888 =========
[22:04:50] =========== drm_test_fb_xrgb8888_to_xrgb2101010  ===========
[22:04:50] [PASSED] single_pixel_source_buffer
[22:04:50] [PASSED] single_pixel_clip_rectangle
[22:04:50] [PASSED] well_known_colors
[22:04:50] [PASSED] destination_pitch
[22:04:50] ======= [PASSED] drm_test_fb_xrgb8888_to_xrgb2101010 =======
[22:04:50] =========== drm_test_fb_xrgb8888_to_argb2101010  ===========
[22:04:50] [PASSED] single_pixel_source_buffer
[22:04:50] [PASSED] single_pixel_clip_rectangle
[22:04:50] [PASSED] well_known_colors
[22:04:50] [PASSED] destination_pitch
[22:04:50] ======= [PASSED] drm_test_fb_xrgb8888_to_argb2101010 =======
[22:04:50] ============== drm_test_fb_xrgb8888_to_mono  ===============
[22:04:50] [PASSED] single_pixel_source_buffer
[22:04:50] [PASSED] single_pixel_clip_rectangle
[22:04:50] [PASSED] well_known_colors
[22:04:50] [PASSED] destination_pitch
[22:04:50] ========== [PASSED] drm_test_fb_xrgb8888_to_mono ===========
[22:04:50] ==================== drm_test_fb_swab  =====================
[22:04:50] [PASSED] single_pixel_source_buffer
[22:04:50] [PASSED] single_pixel_clip_rectangle
[22:04:50] [PASSED] well_known_colors
[22:04:50] [PASSED] destination_pitch
[22:04:50] ================ [PASSED] drm_test_fb_swab =================
[22:04:50] ============ drm_test_fb_xrgb8888_to_xbgr8888  =============
[22:04:50] [PASSED] single_pixel_source_buffer
[22:04:50] [PASSED] single_pixel_clip_rectangle
[22:04:50] [PASSED] well_known_colors
[22:04:50] [PASSED] destination_pitch
[22:04:50] ======== [PASSED] drm_test_fb_xrgb8888_to_xbgr8888 =========
[22:04:50] ============ drm_test_fb_xrgb8888_to_abgr8888  =============
[22:04:50] [PASSED] single_pixel_source_buffer
[22:04:50] [PASSED] single_pixel_clip_rectangle
[22:04:50] [PASSED] well_known_colors
[22:04:50] [PASSED] destination_pitch
[22:04:50] ======== [PASSED] drm_test_fb_xrgb8888_to_abgr8888 =========
[22:04:50] ================= drm_test_fb_clip_offset  =================
[22:04:50] [PASSED] pass through
[22:04:50] [PASSED] horizontal offset
[22:04:50] [PASSED] vertical offset
[22:04:50] [PASSED] horizontal and vertical offset
[22:04:50] [PASSED] horizontal offset (custom pitch)
[22:04:50] [PASSED] vertical offset (custom pitch)
[22:04:50] [PASSED] horizontal and vertical offset (custom pitch)
[22:04:50] ============= [PASSED] drm_test_fb_clip_offset =============
[22:04:50] =================== drm_test_fb_memcpy  ====================
[22:04:50] [PASSED] single_pixel_source_buffer: XR24 little-endian (0x34325258)
[22:04:50] [PASSED] single_pixel_source_buffer: XRA8 little-endian (0x38415258)
[22:04:50] [PASSED] single_pixel_source_buffer: YU24 little-endian (0x34325559)
[22:04:50] [PASSED] single_pixel_clip_rectangle: XB24 little-endian (0x34324258)
[22:04:50] [PASSED] single_pixel_clip_rectangle: XRA8 little-endian (0x38415258)
[22:04:50] [PASSED] single_pixel_clip_rectangle: YU24 little-endian (0x34325559)
[22:04:50] [PASSED] well_known_colors: XB24 little-endian (0x34324258)
[22:04:50] [PASSED] well_known_colors: XRA8 little-endian (0x38415258)
[22:04:50] [PASSED] well_known_colors: YU24 little-endian (0x34325559)
[22:04:50] [PASSED] destination_pitch: XB24 little-endian (0x34324258)
[22:04:50] [PASSED] destination_pitch: XRA8 little-endian (0x38415258)
[22:04:50] [PASSED] destination_pitch: YU24 little-endian (0x34325559)
[22:04:50] =============== [PASSED] drm_test_fb_memcpy ================
[22:04:50] ============= [PASSED] drm_format_helper_test ==============
[22:04:50] ================= drm_format (18 subtests) =================
[22:04:50] [PASSED] drm_test_format_block_width_invalid
[22:04:50] [PASSED] drm_test_format_block_width_one_plane
[22:04:50] [PASSED] drm_test_format_block_width_two_plane
[22:04:50] [PASSED] drm_test_format_block_width_three_plane
[22:04:50] [PASSED] drm_test_format_block_width_tiled
[22:04:50] [PASSED] drm_test_format_block_height_invalid
[22:04:50] [PASSED] drm_test_format_block_height_one_plane
[22:04:50] [PASSED] drm_test_format_block_height_two_plane
[22:04:50] [PASSED] drm_test_format_block_height_three_plane
[22:04:50] [PASSED] drm_test_format_block_height_tiled
[22:04:50] [PASSED] drm_test_format_min_pitch_invalid
[22:04:50] [PASSED] drm_test_format_min_pitch_one_plane_8bpp
[22:04:50] [PASSED] drm_test_format_min_pitch_one_plane_16bpp
[22:04:50] [PASSED] drm_test_format_min_pitch_one_plane_24bpp
[22:04:50] [PASSED] drm_test_format_min_pitch_one_plane_32bpp
[22:04:50] [PASSED] drm_test_format_min_pitch_two_plane
[22:04:50] [PASSED] drm_test_format_min_pitch_three_plane_8bpp
[22:04:50] [PASSED] drm_test_format_min_pitch_tiled
[22:04:50] =================== [PASSED] drm_format ====================
[22:04:50] ============== drm_framebuffer (10 subtests) ===============
[22:04:50] ========== drm_test_framebuffer_check_src_coords  ==========
[22:04:50] [PASSED] Success: source fits into fb
[22:04:50] [PASSED] Fail: overflowing fb with x-axis coordinate
[22:04:50] [PASSED] Fail: overflowing fb with y-axis coordinate
[22:04:50] [PASSED] Fail: overflowing fb with source width
[22:04:50] [PASSED] Fail: overflowing fb with source height
[22:04:50] ====== [PASSED] drm_test_framebuffer_check_src_coords ======
[22:04:50] [PASSED] drm_test_framebuffer_cleanup
[22:04:50] =============== drm_test_framebuffer_create  ===============
[22:04:50] [PASSED] ABGR8888 normal sizes
[22:04:50] [PASSED] ABGR8888 max sizes
[22:04:50] [PASSED] ABGR8888 pitch greater than min required
[22:04:50] [PASSED] ABGR8888 pitch less than min required
[22:04:50] [PASSED] ABGR8888 Invalid width
[22:04:50] [PASSED] ABGR8888 Invalid buffer handle
[22:04:50] [PASSED] No pixel format
[22:04:50] [PASSED] ABGR8888 Width 0
[22:04:50] [PASSED] ABGR8888 Height 0
[22:04:50] [PASSED] ABGR8888 Out of bound height * pitch combination
[22:04:50] [PASSED] ABGR8888 Large buffer offset
[22:04:50] [PASSED] ABGR8888 Buffer offset for inexistent plane
[22:04:50] [PASSED] ABGR8888 Invalid flag
[22:04:50] [PASSED] ABGR8888 Set DRM_MODE_FB_MODIFIERS without modifiers
[22:04:50] [PASSED] ABGR8888 Valid buffer modifier
[22:04:50] [PASSED] ABGR8888 Invalid buffer modifier(DRM_FORMAT_MOD_SAMSUNG_64_32_TILE)
[22:04:50] [PASSED] ABGR8888 Extra pitches without DRM_MODE_FB_MODIFIERS
[22:04:50] [PASSED] ABGR8888 Extra pitches with DRM_MODE_FB_MODIFIERS
[22:04:50] [PASSED] NV12 Normal sizes
[22:04:50] [PASSED] NV12 Max sizes
[22:04:50] [PASSED] NV12 Invalid pitch
[22:04:50] [PASSED] NV12 Invalid modifier/missing DRM_MODE_FB_MODIFIERS flag
[22:04:50] [PASSED] NV12 different  modifier per-plane
[22:04:50] [PASSED] NV12 with DRM_FORMAT_MOD_SAMSUNG_64_32_TILE
[22:04:50] [PASSED] NV12 Valid modifiers without DRM_MODE_FB_MODIFIERS
[22:04:50] [PASSED] NV12 Modifier for inexistent plane
[22:04:50] [PASSED] NV12 Handle for inexistent plane
[22:04:50] [PASSED] NV12 Handle for inexistent plane without DRM_MODE_FB_MODIFIERS
[22:04:50] [PASSED] YVU420 DRM_MODE_FB_MODIFIERS set without modifier
[22:04:50] [PASSED] YVU420 Normal sizes
[22:04:50] [PASSED] YVU420 Max sizes
[22:04:50] [PASSED] YVU420 Invalid pitch
[22:04:50] [PASSED] YVU420 Different pitches
[22:04:50] [PASSED] YVU420 Different buffer offsets/pitches
[22:04:50] [PASSED] YVU420 Modifier set just for plane 0, without DRM_MODE_FB_MODIFIERS
[22:04:50] [PASSED] YVU420 Modifier set just for planes 0, 1, without DRM_MODE_FB_MODIFIERS
[22:04:50] [PASSED] YVU420 Modifier set just for plane 0, 1, with DRM_MODE_FB_MODIFIERS
[22:04:50] [PASSED] YVU420 Valid modifier
[22:04:50] [PASSED] YVU420 Different modifiers per plane
[22:04:50] [PASSED] YVU420 Modifier for inexistent plane
[22:04:50] [PASSED] YUV420_10BIT Invalid modifier(DRM_FORMAT_MOD_LINEAR)
[22:04:50] [PASSED] X0L2 Normal sizes
[22:04:50] [PASSED] X0L2 Max sizes
[22:04:50] [PASSED] X0L2 Invalid pitch
[22:04:50] [PASSED] X0L2 Pitch greater than minimum required
[22:04:50] [PASSED] X0L2 Handle for inexistent plane
[22:04:50] [PASSED] X0L2 Offset for inexistent plane, without DRM_MODE_FB_MODIFIERS set
[22:04:50] [PASSED] X0L2 Modifier without DRM_MODE_FB_MODIFIERS set
[22:04:50] [PASSED] X0L2 Valid modifier
[22:04:50] [PASSED] X0L2 Modifier for inexistent plane
[22:04:50] =========== [PASSED] drm_test_framebuffer_create ===========
[22:04:50] [PASSED] drm_test_framebuffer_free
[22:04:50] [PASSED] drm_test_framebuffer_init
[22:04:50] [PASSED] drm_test_framebuffer_init_bad_format
[22:04:50] [PASSED] drm_test_framebuffer_init_dev_mismatch
[22:04:50] [PASSED] drm_test_framebuffer_lookup
[22:04:50] [PASSED] drm_test_framebuffer_lookup_inexistent
[22:04:50] [PASSED] drm_test_framebuffer_modifiers_not_supported
[22:04:50] ================= [PASSED] drm_framebuffer =================
[22:04:50] ================ drm_gem_shmem (8 subtests) ================
[22:04:50] [PASSED] drm_gem_shmem_test_obj_create
[22:04:50] [PASSED] drm_gem_shmem_test_obj_create_private
[22:04:50] [PASSED] drm_gem_shmem_test_pin_pages
[22:04:50] [PASSED] drm_gem_shmem_test_vmap
[22:04:50] [PASSED] drm_gem_shmem_test_get_sg_table
[22:04:50] [PASSED] drm_gem_shmem_test_get_pages_sgt
[22:04:50] [PASSED] drm_gem_shmem_test_madvise
[22:04:50] [PASSED] drm_gem_shmem_test_purge
[22:04:50] ================== [PASSED] drm_gem_shmem ==================
[22:04:50] === drm_atomic_helper_connector_hdmi_check (29 subtests) ===
[22:04:50] [PASSED] drm_test_check_broadcast_rgb_auto_cea_mode
[22:04:50] [PASSED] drm_test_check_broadcast_rgb_auto_cea_mode_vic_1
[22:04:50] [PASSED] drm_test_check_broadcast_rgb_full_cea_mode
[22:04:50] [PASSED] drm_test_check_broadcast_rgb_full_cea_mode_vic_1
[22:04:50] [PASSED] drm_test_check_broadcast_rgb_limited_cea_mode
[22:04:50] [PASSED] drm_test_check_broadcast_rgb_limited_cea_mode_vic_1
[22:04:50] ====== drm_test_check_broadcast_rgb_cea_mode_yuv420  =======
[22:04:50] [PASSED] Automatic
[22:04:50] [PASSED] Full
[22:04:50] [PASSED] Limited 16:235
[22:04:50] == [PASSED] drm_test_check_broadcast_rgb_cea_mode_yuv420 ===
[22:04:50] [PASSED] drm_test_check_broadcast_rgb_crtc_mode_changed
[22:04:50] [PASSED] drm_test_check_broadcast_rgb_crtc_mode_not_changed
[22:04:50] [PASSED] drm_test_check_disable_connector
[22:04:50] [PASSED] drm_test_check_hdmi_funcs_reject_rate
[22:04:50] [PASSED] drm_test_check_max_tmds_rate_bpc_fallback_rgb
[22:04:50] [PASSED] drm_test_check_max_tmds_rate_bpc_fallback_yuv420
[22:04:50] [PASSED] drm_test_check_max_tmds_rate_bpc_fallback_ignore_yuv422
[22:04:50] [PASSED] drm_test_check_max_tmds_rate_bpc_fallback_ignore_yuv420
[22:04:50] [PASSED] drm_test_check_driver_unsupported_fallback_yuv420
[22:04:50] [PASSED] drm_test_check_output_bpc_crtc_mode_changed
[22:04:50] [PASSED] drm_test_check_output_bpc_crtc_mode_not_changed
[22:04:50] [PASSED] drm_test_check_output_bpc_dvi
[22:04:50] [PASSED] drm_test_check_output_bpc_format_vic_1
[22:04:50] [PASSED] drm_test_check_output_bpc_format_display_8bpc_only
[22:04:50] [PASSED] drm_test_check_output_bpc_format_display_rgb_only
[22:04:50] [PASSED] drm_test_check_output_bpc_format_driver_8bpc_only
[22:04:50] [PASSED] drm_test_check_output_bpc_format_driver_rgb_only
[22:04:50] [PASSED] drm_test_check_tmds_char_rate_rgb_8bpc
[22:04:50] [PASSED] drm_test_check_tmds_char_rate_rgb_10bpc
[22:04:50] [PASSED] drm_test_check_tmds_char_rate_rgb_12bpc
[22:04:50] ============ drm_test_check_hdmi_color_format  =============
[22:04:50] [PASSED] AUTO -> RGB
[22:04:50] [PASSED] YCBCR422 -> YUV422
[22:04:50] [PASSED] YCBCR420 -> YUV420
[22:04:50] [PASSED] YCBCR444 -> YUV444
[22:04:50] [PASSED] RGB -> RGB
[22:04:50] ======== [PASSED] drm_test_check_hdmi_color_format =========
[22:04:50] ======== drm_test_check_hdmi_color_format_420_only  ========
[22:04:50] [PASSED] RGB should fail
[22:04:50] [PASSED] YUV444 should fail
[22:04:50] [PASSED] YUV422 should fail
[22:04:50] [PASSED] YUV420 should work
[22:04:50] ==== [PASSED] drm_test_check_hdmi_color_format_420_only ====
[22:04:50] ===== [PASSED] drm_atomic_helper_connector_hdmi_check ======
[22:04:50] === drm_atomic_helper_connector_hdmi_reset (6 subtests) ====
[22:04:50] [PASSED] drm_test_check_broadcast_rgb_value
[22:04:50] [PASSED] drm_test_check_bpc_8_value
[22:04:50] [PASSED] drm_test_check_bpc_10_value
[22:04:50] [PASSED] drm_test_check_bpc_12_value
[22:04:50] [PASSED] drm_test_check_format_value
[22:04:50] [PASSED] drm_test_check_tmds_char_value
[22:04:50] ===== [PASSED] drm_atomic_helper_connector_hdmi_reset ======
[22:04:50] = drm_atomic_helper_connector_hdmi_mode_valid (7 subtests) =
[22:04:50] [PASSED] drm_test_check_mode_valid
[22:04:50] [PASSED] drm_test_check_mode_valid_reject
[22:04:50] [PASSED] drm_test_check_mode_valid_reject_rate
[22:04:50] [PASSED] drm_test_check_mode_valid_reject_max_clock
[22:04:50] [PASSED] drm_test_check_mode_valid_yuv420_only_max_clock
[22:04:50] [PASSED] drm_test_check_mode_valid_reject_yuv420_only_connector
[22:04:50] [PASSED] drm_test_check_mode_valid_accept_yuv420_also_connector_rgb
[22:04:50] === [PASSED] drm_atomic_helper_connector_hdmi_mode_valid ===
[22:04:50] = drm_atomic_helper_connector_hdmi_infoframes (5 subtests) =
[22:04:50] [PASSED] drm_test_check_infoframes
[22:04:50] [PASSED] drm_test_check_reject_avi_infoframe
[22:04:50] [PASSED] drm_test_check_reject_hdr_infoframe_bpc_8
[22:04:50] [PASSED] drm_test_check_reject_hdr_infoframe_bpc_10
[22:04:50] [PASSED] drm_test_check_reject_audio_infoframe
[22:04:50] === [PASSED] drm_atomic_helper_connector_hdmi_infoframes ===
[22:04:50] ================= drm_managed (2 subtests) =================
[22:04:50] [PASSED] drm_test_managed_release_action
[22:04:50] [PASSED] drm_test_managed_run_action
[22:04:50] =================== [PASSED] drm_managed ===================
[22:04:50] =================== drm_mm (6 subtests) ====================
[22:04:50] [PASSED] drm_test_mm_init
[22:04:50] [PASSED] drm_test_mm_debug
[22:04:50] [PASSED] drm_test_mm_align32
[22:04:50] [PASSED] drm_test_mm_align64
[22:04:50] [PASSED] drm_test_mm_lowest
[22:04:50] [PASSED] drm_test_mm_highest
[22:04:50] ===================== [PASSED] drm_mm ======================
[22:04:50] ============= drm_modes_analog_tv (5 subtests) =============
[22:04:50] [PASSED] drm_test_modes_analog_tv_mono_576i
[22:04:50] [PASSED] drm_test_modes_analog_tv_ntsc_480i
[22:04:50] [PASSED] drm_test_modes_analog_tv_ntsc_480i_inlined
[22:04:50] [PASSED] drm_test_modes_analog_tv_pal_576i
[22:04:50] [PASSED] drm_test_modes_analog_tv_pal_576i_inlined
[22:04:50] =============== [PASSED] drm_modes_analog_tv ===============
[22:04:50] ============== drm_plane_helper (2 subtests) ===============
[22:04:50] =============== drm_test_check_plane_state  ================
[22:04:50] [PASSED] clipping_simple
[22:04:50] [PASSED] clipping_rotate_reflect
[22:04:50] [PASSED] positioning_simple
[22:04:50] [PASSED] upscaling
[22:04:50] [PASSED] downscaling
[22:04:50] [PASSED] rounding1
[22:04:50] [PASSED] rounding2
[22:04:50] [PASSED] rounding3
[22:04:50] [PASSED] rounding4
[22:04:50] =========== [PASSED] drm_test_check_plane_state ============
[22:04:50] =========== drm_test_check_invalid_plane_state  ============
[22:04:50] [PASSED] positioning_invalid
[22:04:50] [PASSED] upscaling_invalid
[22:04:50] [PASSED] downscaling_invalid
[22:04:50] ======= [PASSED] drm_test_check_invalid_plane_state ========
[22:04:50] ================ [PASSED] drm_plane_helper =================
[22:04:50] ====== drm_connector_helper_tv_get_modes (1 subtest) =======
[22:04:50] ====== drm_test_connector_helper_tv_get_modes_check  =======
[22:04:50] [PASSED] None
[22:04:50] [PASSED] PAL
[22:04:50] [PASSED] NTSC
[22:04:50] [PASSED] Both, NTSC Default
[22:04:50] [PASSED] Both, PAL Default
[22:04:50] [PASSED] Both, NTSC Default, with PAL on command-line
[22:04:50] [PASSED] Both, PAL Default, with NTSC on command-line
[22:04:50] == [PASSED] drm_test_connector_helper_tv_get_modes_check ===
[22:04:50] ======== [PASSED] drm_connector_helper_tv_get_modes ========
[22:04:50] ================== drm_rect (9 subtests) ===================
[22:04:50] [PASSED] drm_test_rect_clip_scaled_div_by_zero
[22:04:50] [PASSED] drm_test_rect_clip_scaled_not_clipped
[22:04:50] [PASSED] drm_test_rect_clip_scaled_clipped
[22:04:50] [PASSED] drm_test_rect_clip_scaled_signed_vs_unsigned
[22:04:50] ================= drm_test_rect_intersect  =================
[22:04:50] [PASSED] top-left x bottom-right: 2x2+1+1 x 2x2+0+0
[22:04:50] [PASSED] top-right x bottom-left: 2x2+0+0 x 2x2+1-1
[22:04:50] [PASSED] bottom-left x top-right: 2x2+1-1 x 2x2+0+0
[22:04:50] [PASSED] bottom-right x top-left: 2x2+0+0 x 2x2+1+1
[22:04:50] [PASSED] right x left: 2x1+0+0 x 3x1+1+0
[22:04:50] [PASSED] left x right: 3x1+1+0 x 2x1+0+0
[22:04:50] [PASSED] up x bottom: 1x2+0+0 x 1x3+0-1
[22:04:50] [PASSED] bottom x up: 1x3+0-1 x 1x2+0+0
[22:04:50] [PASSED] touching corner: 1x1+0+0 x 2x2+1+1
[22:04:50] [PASSED] touching side: 1x1+0+0 x 1x1+1+0
[22:04:50] [PASSED] equal rects: 2x2+0+0 x 2x2+0+0
[22:04:50] [PASSED] inside another: 2x2+0+0 x 1x1+1+1
[22:04:50] [PASSED] far away: 1x1+0+0 x 1x1+3+6
[22:04:50] [PASSED] points intersecting: 0x0+5+10 x 0x0+5+10
[22:04:50] [PASSED] points not intersecting: 0x0+0+0 x 0x0+5+10
[22:04:50] ============= [PASSED] drm_test_rect_intersect =============
[22:04:50] ================ drm_test_rect_calc_hscale  ================
[22:04:50] [PASSED] normal use
[22:04:50] [PASSED] out of max range
[22:04:50] [PASSED] out of min range
[22:04:50] [PASSED] zero dst
[22:04:50] [PASSED] negative src
[22:04:50] [PASSED] negative dst
[22:04:50] ============ [PASSED] drm_test_rect_calc_hscale ============
[22:04:50] ================ drm_test_rect_calc_vscale  ================
[22:04:50] [PASSED] normal use
[22:04:50] [PASSED] out of max range
[22:04:50] [PASSED] out of min range
[22:04:50] [PASSED] zero dst
[22:04:50] [PASSED] negative src
[22:04:50] [PASSED] negative dst
[22:04:50] ============ [PASSED] drm_test_rect_calc_vscale ============
[22:04:50] ================== drm_test_rect_rotate  ===================
[22:04:50] [PASSED] reflect-x
[22:04:50] [PASSED] reflect-y
[22:04:50] [PASSED] rotate-0
[22:04:50] [PASSED] rotate-90
[22:04:50] [PASSED] rotate-180
[22:04:50] [PASSED] rotate-270
[22:04:50] ============== [PASSED] drm_test_rect_rotate ===============
[22:04:50] ================ drm_test_rect_rotate_inv  =================
[22:04:50] [PASSED] reflect-x
[22:04:50] [PASSED] reflect-y
[22:04:50] [PASSED] rotate-0
[22:04:50] [PASSED] rotate-90
[22:04:50] [PASSED] rotate-180
[22:04:50] [PASSED] rotate-270
[22:04:50] ============ [PASSED] drm_test_rect_rotate_inv =============
[22:04:50] ==================== [PASSED] drm_rect =====================
[22:04:50] ============ drm_sysfb_modeset_test (1 subtest) ============
[22:04:50] ============ drm_test_sysfb_build_fourcc_list  =============
[22:04:50] [PASSED] no native formats
[22:04:50] [PASSED] XRGB8888 as native format
[22:04:50] [PASSED] remove duplicates
[22:04:50] [PASSED] convert alpha formats
[22:04:50] [PASSED] random formats
[22:04:50] ======== [PASSED] drm_test_sysfb_build_fourcc_list =========
[22:04:50] ============= [PASSED] drm_sysfb_modeset_test ==============
[22:04:50] ================== drm_fixp (2 subtests) ===================
[22:04:50] [PASSED] drm_test_int2fixp
[22:04:50] [PASSED] drm_test_sm2fixp
[22:04:50] ==================== [PASSED] drm_fixp =====================
[22:04:50] ============================================================
[22:04:50] Testing complete. Ran 639 tests: passed: 639
[22:04:50] Elapsed time: 26.931s total, 1.847s configuring, 24.868s building, 0.191s running

+ /kernel/tools/testing/kunit/kunit.py run --kunitconfig /kernel/drivers/gpu/drm/ttm/tests/.kunitconfig [22:04:50] Configuring KUnit Kernel ...
Regenerating .config ...
Populating config with:

$ make ARCH=um O=.kunit olddefconfig
[22:04:52] Building KUnit Kernel ...
Populating config with:
$ make ARCH=um O=.kunit olddefconfig
Building with:
$ make all compile_commands.json scripts_gdb ARCH=um O=.kunit --jobs=48
[22:05:02] Starting KUnit Kernel (1/1)...
[22:05:02] ============================================================
Running tests with:
$ .kunit/linux kunit.enable=1 mem=1G console=tty kunit_shutdown=halt
[22:05:02] ================= ttm_device (5 subtests) ==================
[22:05:02] [PASSED] ttm_device_init_basic
[22:05:02] [PASSED] ttm_device_init_multiple
[22:05:02] [PASSED] ttm_device_fini_basic
[22:05:02] [PASSED] ttm_device_init_no_vma_man
[22:05:02] ================== ttm_device_init_pools  ==================
[22:05:02] [PASSED] No DMA allocations, no DMA32 required
[22:05:02] [PASSED] DMA allocations, DMA32 required
[22:05:02] [PASSED] No DMA allocations, DMA32 required
[22:05:02] [PASSED] DMA allocations, no DMA32 required
[22:05:02] ============== [PASSED] ttm_device_init_pools ==============
[22:05:02] =================== [PASSED] ttm_device ====================
[22:05:02] ================== ttm_pool (8 subtests) ===================
[22:05:02] ================== ttm_pool_alloc_basic  ===================
[22:05:02] [PASSED] One page
[22:05:02] [PASSED] More than one page
[22:05:02] [PASSED] Above the allocation limit
[22:05:02] [PASSED] One page, with coherent DMA mappings enabled
[22:05:02] [PASSED] Above the allocation limit, with coherent DMA mappings enabled
[22:05:02] ============== [PASSED] ttm_pool_alloc_basic ===============
[22:05:02] ============== ttm_pool_alloc_basic_dma_addr  ==============
[22:05:02] [PASSED] One page
[22:05:02] [PASSED] More than one page
[22:05:02] [PASSED] Above the allocation limit
[22:05:02] [PASSED] One page, with coherent DMA mappings enabled
[22:05:02] [PASSED] Above the allocation limit, with coherent DMA mappings enabled
[22:05:02] ========== [PASSED] ttm_pool_alloc_basic_dma_addr ==========
[22:05:02] [PASSED] ttm_pool_alloc_order_caching_match
[22:05:02] [PASSED] ttm_pool_alloc_caching_mismatch
[22:05:02] [PASSED] ttm_pool_alloc_order_mismatch
[22:05:02] [PASSED] ttm_pool_free_dma_alloc
[22:05:02] [PASSED] ttm_pool_free_no_dma_alloc
[22:05:02] [PASSED] ttm_pool_fini_basic
[22:05:02] ==================== [PASSED] ttm_pool =====================
[22:05:02] ================ ttm_resource (8 subtests) =================
[22:05:02] ================= ttm_resource_init_basic  =================
[22:05:02] [PASSED] Init resource in TTM_PL_SYSTEM
[22:05:02] [PASSED] Init resource in TTM_PL_VRAM
[22:05:02] [PASSED] Init resource in a private placement
[22:05:02] [PASSED] Init resource in TTM_PL_SYSTEM, set placement flags
[22:05:02] ============= [PASSED] ttm_resource_init_basic =============
[22:05:02] [PASSED] ttm_resource_init_pinned
[22:05:02] [PASSED] ttm_resource_fini_basic
[22:05:02] [PASSED] ttm_resource_manager_init_basic
[22:05:02] [PASSED] ttm_resource_manager_usage_basic
[22:05:02] [PASSED] ttm_resource_manager_set_used_basic
[22:05:02] [PASSED] ttm_sys_man_alloc_basic
[22:05:02] [PASSED] ttm_sys_man_free_basic
[22:05:02] ================== [PASSED] ttm_resource ===================
[22:05:02] =================== ttm_tt (15 subtests) ===================
[22:05:02] ==================== ttm_tt_init_basic  ====================
[22:05:02] [PASSED] Page-aligned size
[22:05:02] [PASSED] Extra pages requested
[22:05:02] ================ [PASSED] ttm_tt_init_basic ================
[22:05:02] [PASSED] ttm_tt_init_misaligned
[22:05:02] [PASSED] ttm_tt_fini_basic
[22:05:02] [PASSED] ttm_tt_fini_sg
[22:05:02] [PASSED] ttm_tt_fini_shmem
[22:05:02] [PASSED] ttm_tt_create_basic
[22:05:02] [PASSED] ttm_tt_create_invalid_bo_type
[22:05:02] [PASSED] ttm_tt_create_ttm_exists
[22:05:02] [PASSED] ttm_tt_create_failed
[22:05:02] [PASSED] ttm_tt_destroy_basic
[22:05:02] [PASSED] ttm_tt_populate_null_ttm
[22:05:02] [PASSED] ttm_tt_populate_populated_ttm
[22:05:02] [PASSED] ttm_tt_unpopulate_basic
[22:05:02] [PASSED] ttm_tt_unpopulate_empty_ttm
[22:05:02] [PASSED] ttm_tt_swapin_basic
[22:05:02] ===================== [PASSED] ttm_tt ======================
[22:05:02] =================== ttm_bo (14 subtests) ===================
[22:05:02] =========== ttm_bo_reserve_optimistic_no_ticket  ===========
[22:05:02] [PASSED] Cannot be interrupted and sleeps
[22:05:02] [PASSED] Cannot be interrupted, locks straight away
[22:05:02] [PASSED] Can be interrupted, sleeps
[22:05:02] ======= [PASSED] ttm_bo_reserve_optimistic_no_ticket =======
[22:05:02] [PASSED] ttm_bo_reserve_locked_no_sleep
[22:05:02] [PASSED] ttm_bo_reserve_no_wait_ticket
[22:05:02] [PASSED] ttm_bo_reserve_double_resv
[22:05:02] [PASSED] ttm_bo_reserve_interrupted
[22:05:02] [PASSED] ttm_bo_reserve_deadlock
[22:05:02] [PASSED] ttm_bo_unreserve_basic
[22:05:02] [PASSED] ttm_bo_unreserve_pinned
[22:05:02] [PASSED] ttm_bo_unreserve_bulk
[22:05:02] [PASSED] ttm_bo_fini_basic
[22:05:02] [PASSED] ttm_bo_fini_shared_resv
[22:05:02] [PASSED] ttm_bo_pin_basic
[22:05:02] [PASSED] ttm_bo_pin_unpin_resource
[22:05:02] [PASSED] ttm_bo_multiple_pin_one_unpin
[22:05:02] ===================== [PASSED] ttm_bo ======================
[22:05:02] ============== ttm_bo_validate (22 subtests) ===============
[22:05:02] ============== ttm_bo_init_reserved_sys_man  ===============
[22:05:02] [PASSED] Buffer object for userspace
[22:05:02] [PASSED] Kernel buffer object
[22:05:02] [PASSED] Shared buffer object
[22:05:02] ========== [PASSED] ttm_bo_init_reserved_sys_man ===========
[22:05:02] ============== ttm_bo_init_reserved_mock_man  ==============
[22:05:02] [PASSED] Buffer object for userspace
[22:05:02] [PASSED] Kernel buffer object
[22:05:02] [PASSED] Shared buffer object
[22:05:02] ========== [PASSED] ttm_bo_init_reserved_mock_man ==========
[22:05:02] [PASSED] ttm_bo_init_reserved_resv
[22:05:02] ================== ttm_bo_validate_basic  ==================
[22:05:02] [PASSED] Buffer object for userspace
[22:05:02] [PASSED] Kernel buffer object
[22:05:02] [PASSED] Shared buffer object
[22:05:02] ============== [PASSED] ttm_bo_validate_basic ==============
[22:05:02] [PASSED] ttm_bo_validate_invalid_placement
[22:05:02] ============= ttm_bo_validate_same_placement  ==============
[22:05:02] [PASSED] System manager
[22:05:02] [PASSED] VRAM manager
[22:05:02] ========= [PASSED] ttm_bo_validate_same_placement ==========
[22:05:02] [PASSED] ttm_bo_validate_failed_alloc
[22:05:02] [PASSED] ttm_bo_validate_pinned
[22:05:02] [PASSED] ttm_bo_validate_busy_placement
[22:05:02] ================ ttm_bo_validate_multihop  =================
[22:05:02] [PASSED] Buffer object for userspace
[22:05:02] [PASSED] Kernel buffer object
[22:05:02] [PASSED] Shared buffer object
[22:05:02] ============ [PASSED] ttm_bo_validate_multihop =============
[22:05:02] ========== ttm_bo_validate_no_placement_signaled  ==========
[22:05:02] [PASSED] Buffer object in system domain, no page vector
[22:05:02] [PASSED] Buffer object in system domain with an existing page vector
[22:05:02] ====== [PASSED] ttm_bo_validate_no_placement_signaled ======
[22:05:02] ======== ttm_bo_validate_no_placement_not_signaled  ========
[22:05:02] [PASSED] Buffer object for userspace
[22:05:02] [PASSED] Kernel buffer object
[22:05:02] [PASSED] Shared buffer object
[22:05:02] ==== [PASSED] ttm_bo_validate_no_placement_not_signaled ====
[22:05:02] [PASSED] ttm_bo_validate_move_fence_signaled
[22:05:02] ========= ttm_bo_validate_move_fence_not_signaled  =========
[22:05:02] [PASSED] Waits for GPU
[22:05:02] [PASSED] Tries to lock straight away
[22:05:02] ===== [PASSED] ttm_bo_validate_move_fence_not_signaled =====
[22:05:02] [PASSED] ttm_bo_validate_swapout
[22:05:02] [PASSED] ttm_bo_validate_happy_evict
[22:05:02] [PASSED] ttm_bo_validate_all_pinned_evict
[22:05:02] [PASSED] ttm_bo_validate_allowed_only_evict
[22:05:02] [PASSED] ttm_bo_validate_deleted_evict
[22:05:02] [PASSED] ttm_bo_validate_busy_domain_evict
[22:05:02] [PASSED] ttm_bo_validate_evict_gutting
[22:05:02] [PASSED] ttm_bo_validate_recrusive_evict
[22:05:02] ================= [PASSED] ttm_bo_validate =================
[22:05:02] ============================================================
[22:05:02] Testing complete. Ran 102 tests: passed: 102
[22:05:02] Elapsed time: 11.994s total, 1.771s configuring, 10.008s building, 0.177s running

+ cleanup
++ stat -c %u:%g /kernel
+ chown -R 1003:1003 /kernel

#64 lore raw

✗ CI.checksparse: warning for drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs

In reply to Matthew Brost
== Series Details ==

== Series Details ==

Series: drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs
URL : https://patchwork.freedesktop.org/series/170216/
State : warning

== Summary ==

+ trap cleanup EXIT
+ KERNEL=/kernel
+ MT=/root/linux/maintainer-tools
+ git clone https://gitlab.freedesktop.org/drm/maintainer-tools /root/linux/maintainer-tools Cloning into '/root/linux/maintainer-tools'...
warning: redirecting to https://gitlab.freedesktop.org/drm/maintainer-tools.git/
+ make -C /root/linux/maintainer-tools
make: Entering directory '/root/linux/maintainer-tools'
cc -O2 -g -Wextra -o remap-log remap-log.c
make: Leaving directory '/root/linux/maintainer-tools'
+ cd /kernel
+ git config --global --add safe.directory /kernel
+ /root/linux/maintainer-tools/dim sparse --fast 6d76f5afea7406e978db1fc356a9b88d0d8989d1 Sparse version: 0.6.4 (Ubuntu: 0.6.4-4ubuntu3)
Fast mode used, each commit won't be checked separately. +/kernel/Makefile:1280: C=1 specified, but sparse is not available or not up to date

+ cleanup
++ stat -c %u:%g /kernel
+ chown -R 1003:1003 /kernel

#65 lore raw

✗ Xe.CI.BAT: failure for drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs

In reply to Matthew Brost
== Series Details ==

== Series Details ==

Series: drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs
URL : https://patchwork.freedesktop.org/series/170216/
State : failure

== Summary ==

CI Bug Log - changes from xe-5385-6d76f5afea7406e978db1fc356a9b88d0d8989d1_BAT -> xe-pw-170216v1_BAT
====================================================

Summary
-------

**FAILURE**

Serious unknown changes coming with xe-pw-170216v1_BAT absolutely need to be verified manually.

If you think the reported changes have nothing to do with the changes
introduced in xe-pw-170216v1_BAT, please notify your bug team ([email protected]) to allow them to document this new failure mode, which will reduce false positives in CI.

Participating hosts (12 -> 13)
------------------------------

Additional (1): bat-bmg-2

Possible new issues
-------------------

Here are the unknown changes that may have been introduced in xe-pw-170216v1_BAT:

### IGT changes ###

#### Possible regressions ####

* igt@xe_create@create-massive-size:
- bat-bmg-2: NOTRUN -> [ABORT][1]
[1]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-bmg-2/igt@[email protected]
- bat-adlp-7: [PASS][2] -> [ABORT][3]
[2]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-5385-6d76f5afea7406e978db1fc356a9b88d0d8989d1/bat-adlp-7/igt@[email protected]
[3]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-adlp-7/igt@[email protected]
- bat-bmg-vm: [PASS][4] -> [ABORT][5]
[4]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-5385-6d76f5afea7406e978db1fc356a9b88d0d8989d1/bat-bmg-vm/igt@[email protected]
[5]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-bmg-vm/igt@[email protected]
- bat-bmg-1: [PASS][6] -> [ABORT][7]
[6]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-5385-6d76f5afea7406e978db1fc356a9b88d0d8989d1/bat-bmg-1/igt@[email protected]
[7]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-bmg-1/igt@[email protected]
- bat-lnl-2: [PASS][8] -> [ABORT][9]
[8]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-5385-6d76f5afea7406e978db1fc356a9b88d0d8989d1/bat-lnl-2/igt@[email protected]
[9]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-lnl-2/igt@[email protected]
- bat-ptl-2: [PASS][10] -> [ABORT][11]
[10]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-5385-6d76f5afea7406e978db1fc356a9b88d0d8989d1/bat-ptl-2/igt@[email protected]
[11]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-ptl-2/igt@[email protected]
- bat-wcl-1: [PASS][12] -> [ABORT][13]
[12]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-5385-6d76f5afea7406e978db1fc356a9b88d0d8989d1/bat-wcl-1/igt@[email protected]
[13]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-wcl-1/igt@[email protected]
- bat-atsm-2: [PASS][14] -> [ABORT][15]
[14]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-5385-6d76f5afea7406e978db1fc356a9b88d0d8989d1/bat-atsm-2/igt@[email protected]
[15]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-atsm-2/igt@[email protected]
- bat-ptl-1: [PASS][16] -> [ABORT][17]
[16]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-5385-6d76f5afea7406e978db1fc356a9b88d0d8989d1/bat-ptl-1/igt@[email protected]
[17]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-ptl-1/igt@[email protected]
- bat-lnl-1: [PASS][18] -> [ABORT][19]
[18]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-5385-6d76f5afea7406e978db1fc356a9b88d0d8989d1/bat-lnl-1/igt@[email protected]
[19]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-lnl-1/igt@[email protected]
- bat-ptl-vm: [PASS][20] -> [ABORT][21]
[20]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-5385-6d76f5afea7406e978db1fc356a9b88d0d8989d1/bat-ptl-vm/igt@[email protected]
[21]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-ptl-vm/igt@[email protected]
- bat-wcl-2: [PASS][22] -> [ABORT][23]
[22]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-5385-6d76f5afea7406e978db1fc356a9b88d0d8989d1/bat-wcl-2/igt@[email protected]
[23]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-wcl-2/igt@[email protected]

* igt@xe_create@multigpu-create-massive-size:
- bat-bmg-3: [PASS][24] -> [ABORT][25]
[24]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-5385-6d76f5afea7406e978db1fc356a9b88d0d8989d1/bat-bmg-3/igt@[email protected]
[25]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-bmg-3/igt@[email protected]

Known issues
------------

Here are the changes found in xe-pw-170216v1_BAT that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@fbdev@nullptr:
- bat-bmg-2: NOTRUN -> [SKIP][26] ([Intel XE#2134]) +4 other tests skip
[26]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-bmg-2/igt@[email protected]

* igt@kms_addfb_basic@addfb25-y-tiled-small-legacy:
- bat-bmg-2: NOTRUN -> [SKIP][27] ([Intel XE#2233])
[27]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-bmg-2/igt@[email protected]

* igt@kms_cursor_legacy@basic-flip-after-cursor-legacy:
- bat-bmg-2: NOTRUN -> [SKIP][28] ([Intel XE#2489] / [Intel XE#3419]) +13 other tests skip
[28]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-bmg-2/igt@[email protected]

* igt@kms_flip@basic-flip-vs-modeset:
- bat-bmg-2: NOTRUN -> [SKIP][29] ([Intel XE#2482]) +3 other tests skip
[29]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-bmg-2/igt@[email protected]

* igt@kms_frontbuffer_tracking@basic:
- bat-bmg-2: NOTRUN -> [SKIP][30] ([Intel XE#2434] / [Intel XE#2548] / [Intel XE#6314])
[30]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-bmg-2/igt@[email protected]

* igt@kms_psr@psr-sprite-plane-onoff:
- bat-bmg-2: NOTRUN -> [SKIP][31] ([Intel XE#2234] / [Intel XE#2850]) +2 other tests skip
[31]: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/bat-bmg-2/igt@[email protected]

[Intel XE#2134]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/2134
[Intel XE#2233]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/2233
[Intel XE#2234]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/2234
[Intel XE#2434]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/2434
[Intel XE#2482]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/2482
[Intel XE#2489]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/2489
[Intel XE#2548]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/2548
[Intel XE#2850]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/2850
[Intel XE#3419]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/3419
[Intel XE#6314]: https://gitlab.freedesktop.org/drm/xe/kernel/issues/6314

Build changes
-------------

* Linux: xe-5385-6d76f5afea7406e978db1fc356a9b88d0d8989d1 -> xe-pw-170216v1

IGT_9002: 9002
xe-5385-6d76f5afea7406e978db1fc356a9b88d0d8989d1: 6d76f5afea7406e978db1fc356a9b88d0d8989d1
xe-pw-170216v1: 170216v1

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/intel-xe/xe-pw-170216v1/index.html

maintainer
#66 lore raw
In reply to Matthew Brost
Quoted 11 lines · Matthew Brost

The overarching goal of this series is to keep user-facing IOCTLs snappy by holding the BO dma-resv lock for the shortest possible time, and to push the expensive, best-effort work - page (re)allocation, DMA mapping and placement fixups - into the background or out of the locked critical section entirely.

Two related mechanisms fall out of that goal:

- Out-of-lock backing. Page allocation and DMA mapping are the dominant cost of gem_create and of BO moves, and doing them under the dma-resv lock serializes otherwise-independent clients.

Saying that doesn't make sense.

The DMA-resv lock is either per BO or per VM and so the clients are not even remotely independent of that lock.

Quoted 5 lines

This series preallocates
the backing pages (and, where applicable, builds the DMA/IOVA mapping) outside the lock, then transfers the result into the BO under a brief lock hold. The user gets a fast IOCTL return; the heavy lifting happens without contending the reservation.

Ok, that sounds like a really bad idea.

Why in the world would you do this? The dma_resv lock is exactly what is used to prevent something like that.

Quoted 8 lines

- Page defragmentation. A BO allocated under memory pressure keeps its scattered, sub-optimally-ordered backing for its entire lifetime, costing TLB efficiency forever. TTM grows the plumbing to track order-failure and to re-back a populated BO in place at the beneficial order, and Xe wires up a background delayed worker that promotes such BOs on the GPU once memory is available again - again, off the hot path and without stalling the submitting thread.

That could be quite beneficial, but so far falling back to low order allocation was only seen as last resort to avoid OOM.

Regards,
Christian.

Quoted 143 lines

Since v1 [1] the series has grown considerably. The bulk of the new material is a direct result of profiling: once the defragmenter was in place, the remaining dma-resv hold times and the per-BO allocation/mapping costs showed up clearly, which motivated the out-of-lock preallocation, the IOVA-based mapping path, and the amdgpu counterpart. Rather than land the defragmenter alone, v2 folds in these optimizations since they share the same infrastructure and the same "hold dma-resv briefly, fix up in the background" architecture.

The series is organized in sections rather than described patch by patch:

- Patches 1-10 (drm/ttm): core TTM preparation - order-failure tracking, the defragmentation move, reclaim backoff, and out-of-lock page preallocation plumbing.
- Patches 11-14: other dependent drm/gpusvm and drm/xe patches this series builds on (DMA-mapping accounting, per-order DMA stats, async L2 flush, and a VM-teardown ordering fix).
- Patches 15-23 (drm/xe): the page defragmenter itself - BO tracking, the on-GPU defrag copy, xe_bo_move() handling, and the background worker with its stats and configuration.
- Patches 24-25 (drm/xe): out-of-lock system BO backing preallocation in gem_create, moving page allocation out of the dma-resv lock.
- Patches 26-32 (drm/xe): IOVA-based DMA mapping optimizations, building and finalizing the mapping outside the lock.
- Patch 33 (drm/amdgpu): the equivalent out-of-lock system BO preallocation for amdgpu, exercising the shared TTM plumbing.

Testing
=======

- 3D benchmarks on Ubuntu and on Android, with memory intentionally fragmented by a separate program at launch (plus beneficial-order error injection). BOs are initially backed at a sub-optimal order and scores start lower; the background defrag worker then promotes the backing to the beneficial order and scores climb back in line with the unfragmented baseline.
- IGT:

   https://patchwork.freedesktop.org/patch/739052/?series=170046&rev=2

Matt

[1] https://patchwork.freedesktop.org/series/169053/

Cc Alex Deucher <[email protected]>
Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>

Matthew Brost (31):
drm/ttm/pool: Allow backing off reclaim at the beneficial order drm/ttm/pool: Add ttm_pool_page_order_nodma() helper drm/ttm: Record sub-optimal page order allocations in ttm_tt drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc() drm/ttm: Support defragmentation moves
drm/ttm: Add fault injection for beneficial-order allocation failures drm/ttm: Harvest beneficial-order pages on defragmentation moves drm/ttm: Bound page (re)allocation per defragmentation move drm/ttm: Preallocate beneficial-order defrag pages outside the lock drm/ttm: Add full out-of-lock preallocation for ttm_pool_alloc() drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind() drm/xe: Destroy page tables after unlinking all VMAs on VM close drm/xe: Track BOs backed at a sub-optimal page order drm/xe: Back off beneficial-order reclaim under defrag pressure drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies drm/xe: Handle defrag moves in xe_bo_move()
drm/xe: Skip self-copies for borrowed pages on defrag moves drm/xe: Add a page defragmentation worker
drm/xe: Add defrag GT stats
drm/xe: Add Kconfig.profile options for BO defrag configuration drm/xe: Defrag using out-of-lock page preallocation drm/xe: Add defrag profiling tracepoints
drm/xe: Preallocate system BO backing outside the dma-resv lock drm/xe: Add tracepoint for xe_gem_create_ioctl
drm/xe: Add IOVA-based xe_res_cursor variant
drm/xe: Use IOVA-based DMA mapping for eligible tt BOs drm/xe: Add per-device dependency scheduler for IOVA defrag finalize drm/xe: Add packed copy-step IOVA mapping for defrag drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies drm/xe: Finalize defrag-IOVA moves with post-copy job drm/amdgpu: Preallocate system BO pages outside the reservation lock

Thomas Hellström (2):
drm/gpusvm: Add a DMA-mapping accounting callback drm/xe: Add debugfs stats for DMA-mapped pages per order

 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |    4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c   |    2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       |   45 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h       |    5 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c    |    3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |    4 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       |   39 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h       |    3 +
 drivers/gpu/drm/drm_gpusvm.c                  |   17 +-
 drivers/gpu/drm/ttm/ttm_bo.c                  |  101 +-
 drivers/gpu/drm/ttm/ttm_bo_util.c             |   21 +-
 drivers/gpu/drm/ttm/ttm_pool.c                |  931 +++++++++-
 drivers/gpu/drm/ttm/ttm_tt.c                  |   67 +
 drivers/gpu/drm/xe/Kconfig.profile            |   40 +
 drivers/gpu/drm/xe/tests/xe_bo.c              |    8 +-
 drivers/gpu/drm/xe/tests/xe_dma_buf.c         |    2 +-
 drivers/gpu/drm/xe/tests/xe_migrate.c         |   12 +-
 drivers/gpu/drm/xe/xe_bo.c                    | 1632 ++++++++++++++++-
 drivers/gpu/drm/xe/xe_bo.h                    |   15 +-
 drivers/gpu/drm/xe/xe_bo_types.h              |    6 +
 drivers/gpu/drm/xe/xe_debugfs.c               |   26 +
 drivers/gpu/drm/xe/xe_device.c                |   35 +
 drivers/gpu/drm/xe/xe_device_types.h          |   57 +
 drivers/gpu/drm/xe/xe_dma_buf.c               |    2 +-
 drivers/gpu/drm/xe/xe_ggtt.c                  |    2 +-
 drivers/gpu/drm/xe/xe_gt_stats.c              |    5 +
 drivers/gpu/drm/xe/xe_gt_stats_types.h        |   17 +
 drivers/gpu/drm/xe/xe_migrate.c               |  531 +++++-
 drivers/gpu/drm/xe/xe_migrate.h               |   17 +
 drivers/gpu/drm/xe/xe_pt.c                    |    2 +-
 drivers/gpu/drm/xe/xe_res_cursor.h            |   56 +-
 drivers/gpu/drm/xe/xe_svm.c                   |   36 +-
 drivers/gpu/drm/xe/xe_svm.h                   |    3 +-
 drivers/gpu/drm/xe/xe_trace_bo.h              |   90 +
 drivers/gpu/drm/xe/xe_userptr.c               |   55 +
 drivers/gpu/drm/xe/xe_userptr.h               |    1 +
 drivers/gpu/drm/xe/xe_vm.c                    |  149 +-
 drivers/gpu/drm/xe/xe_vm.h                    |    5 +
 include/drm/drm_gpusvm.h                      |   19 +
 include/drm/ttm/ttm_bo.h                      |   64 +
 include/drm/ttm/ttm_pool.h                    |   46 +
 include/drm/ttm/ttm_tt.h                      |   43 +-
 42 files changed, 4029 insertions(+), 189 deletions(-)
author maintainer
#67 lore raw
Quoted 9 lines · Christian König
Quoted 8 lines · Matthew Brost

The overarching goal of this series is to keep user-facing IOCTLs snappy by holding the BO dma-resv lock for the shortest possible time, and to push the expensive, best-effort work - page (re)allocation, DMA mapping and placement fixups - into the background or out of the locked critical section entirely.

Two related mechanisms fall out of that goal:

Quoted 8 lines
Quoted 3 lines

- Out-of-lock backing. Page allocation and DMA mapping are the dominant cost of gem_create and of BO moves, and doing them under the dma-resv lock serializes otherwise-independent clients.

Saying that doesn't make sense.

The DMA-resv lock is either per BO or per VM and so the clients are not even remotely independent of that lock.

s/clients/threads:

Same VM:

Thread A - allocate memory under VM's dma-resv, takes a very long time Thread B - tries to submit, blocks on Thread A

Quoted 10 lines
Quoted 5 lines

This series preallocates
the backing pages (and, where applicable, builds the DMA/IOVA mapping) outside the lock, then transfers the result into the BO under a brief lock hold. The user gets a fast IOCTL return; the heavy lifting happens without contending the reservation.

Ok, that sounds like a really bad idea.

Why in the world would you do this? The dma_resv lock is exactly what is used to prevent something like that.

Allocating pages from the core MM subsystem does not really have anything to do with dma_resv. Moving pages attached to a BO, however, is related to dma_resv, and that is where TTM comes into play.

The TL;DR is that a defragmenter does not work well when page allocations are performed while holding the dma_resv lock. During the v1 implementation, I quickly noticed that the dma_resv lock could be held for up to 100 ms in the worst case, and more typically for 5–10 ms. When the frame budget at 60 Hz is only 16 ms, this is obviously a problem, and it becomes even more problematic at higher refresh rates.

After quite a bit of work, the common-case lock hold time is now below 100 µs, while the worst case is under 1 ms. At that point, a defragmenter becomes much more viable.

I'm applying the same defragmentation logic to IOCTLs. Why hold the lock any longer than necessary and introduce additional pipeline stalls?

Quoted 11 lines
Quoted 8 lines

- Page defragmentation. A BO allocated under memory pressure keeps its scattered, sub-optimally-ordered backing for its entire lifetime, costing TLB efficiency forever. TTM grows the plumbing to track order-failure and to re-back a populated BO in place at the beneficial order, and Xe wires up a background delayed worker that promotes such BOs on the GPU once memory is available again - again, off the hot path and without stalling the submitting thread.

That could be quite beneficial, but so far falling back to low order allocation was only seen as last resort to avoid OOM.

Right now, the situation is effectively "once placed badly, forever placed badly."

Matt

Quoted 147 lines

Regards,
Christian.

Quoted 143 lines

Since v1 [1] the series has grown considerably. The bulk of the new material is a direct result of profiling: once the defragmenter was in place, the remaining dma-resv hold times and the per-BO allocation/mapping costs showed up clearly, which motivated the out-of-lock preallocation, the IOVA-based mapping path, and the amdgpu counterpart. Rather than land the defragmenter alone, v2 folds in these optimizations since they share the same infrastructure and the same "hold dma-resv briefly, fix up in the background" architecture.

The series is organized in sections rather than described patch by patch:

- Patches 1-10 (drm/ttm): core TTM preparation - order-failure tracking, the defragmentation move, reclaim backoff, and out-of-lock page preallocation plumbing.
- Patches 11-14: other dependent drm/gpusvm and drm/xe patches this series builds on (DMA-mapping accounting, per-order DMA stats, async L2 flush, and a VM-teardown ordering fix).
- Patches 15-23 (drm/xe): the page defragmenter itself - BO tracking, the on-GPU defrag copy, xe_bo_move() handling, and the background worker with its stats and configuration.
- Patches 24-25 (drm/xe): out-of-lock system BO backing preallocation in gem_create, moving page allocation out of the dma-resv lock.
- Patches 26-32 (drm/xe): IOVA-based DMA mapping optimizations, building and finalizing the mapping outside the lock.
- Patch 33 (drm/amdgpu): the equivalent out-of-lock system BO preallocation for amdgpu, exercising the shared TTM plumbing.

Testing
=======

- 3D benchmarks on Ubuntu and on Android, with memory intentionally fragmented by a separate program at launch (plus beneficial-order error injection). BOs are initially backed at a sub-optimal order and scores start lower; the background defrag worker then promotes the backing to the beneficial order and scores climb back in line with the unfragmented baseline.
- IGT:

   https://patchwork.freedesktop.org/patch/739052/?series=170046&rev=2

Matt

[1] https://patchwork.freedesktop.org/series/169053/

Cc Alex Deucher <[email protected]>
Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>

Matthew Brost (31):
drm/ttm/pool: Allow backing off reclaim at the beneficial order drm/ttm/pool: Add ttm_pool_page_order_nodma() helper drm/ttm: Record sub-optimal page order allocations in ttm_tt drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc() drm/ttm: Support defragmentation moves
drm/ttm: Add fault injection for beneficial-order allocation failures drm/ttm: Harvest beneficial-order pages on defragmentation moves drm/ttm: Bound page (re)allocation per defragmentation move drm/ttm: Preallocate beneficial-order defrag pages outside the lock drm/ttm: Add full out-of-lock preallocation for ttm_pool_alloc() drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind() drm/xe: Destroy page tables after unlinking all VMAs on VM close drm/xe: Track BOs backed at a sub-optimal page order drm/xe: Back off beneficial-order reclaim under defrag pressure drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies drm/xe: Handle defrag moves in xe_bo_move()
drm/xe: Skip self-copies for borrowed pages on defrag moves drm/xe: Add a page defragmentation worker
drm/xe: Add defrag GT stats
drm/xe: Add Kconfig.profile options for BO defrag configuration drm/xe: Defrag using out-of-lock page preallocation drm/xe: Add defrag profiling tracepoints
drm/xe: Preallocate system BO backing outside the dma-resv lock drm/xe: Add tracepoint for xe_gem_create_ioctl
drm/xe: Add IOVA-based xe_res_cursor variant
drm/xe: Use IOVA-based DMA mapping for eligible tt BOs drm/xe: Add per-device dependency scheduler for IOVA defrag finalize drm/xe: Add packed copy-step IOVA mapping for defrag drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies drm/xe: Finalize defrag-IOVA moves with post-copy job drm/amdgpu: Preallocate system BO pages outside the reservation lock

Thomas Hellström (2):
drm/gpusvm: Add a DMA-mapping accounting callback drm/xe: Add debugfs stats for DMA-mapped pages per order

 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |    4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c   |    2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       |   45 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h       |    5 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c    |    3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |    4 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       |   39 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h       |    3 +
 drivers/gpu/drm/drm_gpusvm.c                  |   17 +-
 drivers/gpu/drm/ttm/ttm_bo.c                  |  101 +-
 drivers/gpu/drm/ttm/ttm_bo_util.c             |   21 +-
 drivers/gpu/drm/ttm/ttm_pool.c                |  931 +++++++++-
 drivers/gpu/drm/ttm/ttm_tt.c                  |   67 +
 drivers/gpu/drm/xe/Kconfig.profile            |   40 +
 drivers/gpu/drm/xe/tests/xe_bo.c              |    8 +-
 drivers/gpu/drm/xe/tests/xe_dma_buf.c         |    2 +-
 drivers/gpu/drm/xe/tests/xe_migrate.c         |   12 +-
 drivers/gpu/drm/xe/xe_bo.c                    | 1632 ++++++++++++++++-
 drivers/gpu/drm/xe/xe_bo.h                    |   15 +-
 drivers/gpu/drm/xe/xe_bo_types.h              |    6 +
 drivers/gpu/drm/xe/xe_debugfs.c               |   26 +
 drivers/gpu/drm/xe/xe_device.c                |   35 +
 drivers/gpu/drm/xe/xe_device_types.h          |   57 +
 drivers/gpu/drm/xe/xe_dma_buf.c               |    2 +-
 drivers/gpu/drm/xe/xe_ggtt.c                  |    2 +-
 drivers/gpu/drm/xe/xe_gt_stats.c              |    5 +
 drivers/gpu/drm/xe/xe_gt_stats_types.h        |   17 +
 drivers/gpu/drm/xe/xe_migrate.c               |  531 +++++-
 drivers/gpu/drm/xe/xe_migrate.h               |   17 +
 drivers/gpu/drm/xe/xe_pt.c                    |    2 +-
 drivers/gpu/drm/xe/xe_res_cursor.h            |   56 +-
 drivers/gpu/drm/xe/xe_svm.c                   |   36 +-
 drivers/gpu/drm/xe/xe_svm.h                   |    3 +-
 drivers/gpu/drm/xe/xe_trace_bo.h              |   90 +
 drivers/gpu/drm/xe/xe_userptr.c               |   55 +
 drivers/gpu/drm/xe/xe_userptr.h               |    1 +
 drivers/gpu/drm/xe/xe_vm.c                    |  149 +-
 drivers/gpu/drm/xe/xe_vm.h                    |    5 +
 include/drm/drm_gpusvm.h                      |   19 +
 include/drm/ttm/ttm_bo.h                      |   64 +
 include/drm/ttm/ttm_pool.h                    |   46 +
 include/drm/ttm/ttm_tt.h                      |   43 +-
 42 files changed, 4029 insertions(+), 189 deletions(-)