Skip to contentSkip to conversation

drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs

PATCH In review Matthew Brost 6 participants 122 messages Jul 11 – Nov 20 Page 1 of 3

Subsystems (7)
lore
Share & tools
lore

Matthew Brost

author maintainer
#0 lore raw

v3 cover drm/ttm, drm/xe: Minimize dma-resv hold times and defragment sub-optimally backed BOs

The overarching goal of this series is to keep user-facing IOCTLs snappy by holding the BO dma-resv lock for the shortest possible time, and to push the expensive, best-effort work - page (re)allocation, DMA mapping and placement fixups - into the background or out of the locked critical section entirely.

Two related mechanisms fall out of that goal:

- Out-of-lock backing. Page allocation and DMA mapping are the dominant cost of gem_create and of BO moves, and doing them under the dma-resv lock serializes otherwise-independent clients. This series preallocates the backing pages (and, where applicable, builds the DMA/IOVA mapping) outside the lock, then transfers the result into the BO under a brief lock hold. The user gets a fast IOCTL return; the heavy lifting happens without contending the reservation.

- Page defragmentation. A BO allocated under memory pressure keeps its scattered, sub-optimally-ordered backing for its entire lifetime, costing TLB efficiency forever. TTM grows the plumbing to track order-failure and to re-back a populated BO in place at the beneficial order, and Xe wires up a background delayed worker that promotes such BOs on the GPU once memory is available again - again, off the hot path and without stalling the submitting thread.

Since v1 [1] the series has grown considerably. The bulk of the new material is a direct result of profiling: once the defragmenter was in place, the remaining dma-resv hold times and the per-BO allocation/mapping costs showed up clearly, which motivated the out-of-lock preallocation, the IOVA-based mapping path, and the amdgpu counterpart. Rather than land the defragmenter alone, v2 folds in these optimizations since they share the same infrastructure and the same "hold dma-resv briefly, fix up in the background" architecture.

The series is organized in sections rather than described patch by patch:

- Patches 1-10 (drm/ttm): core TTM preparation - order-failure tracking, the defragmentation move, reclaim backoff, and out-of-lock page preallocation plumbing.
- Patches 11-14: other dependent drm/gpusvm and drm/xe patches this series builds on (DMA-mapping accounting, per-order DMA stats, async L2 flush, and a VM-teardown ordering fix).
- Patches 15-23 (drm/xe): the page defragmenter itself - BO tracking, the on-GPU defrag copy, xe_bo_move() handling, and the background worker with its stats and configuration.
- Patches 24-25 (drm/xe): out-of-lock system BO backing preallocation in gem_create, moving page allocation out of the dma-resv lock.
- Patches 26-32 (drm/xe): IOVA-based DMA mapping optimizations, building and finalizing the mapping outside the lock.
- Patch 33 (drm/amdgpu): the equivalent out-of-lock system BO preallocation for amdgpu, exercising the shared TTM plumbing.

Testing
=======

- 3D benchmarks on Ubuntu and on Android, with memory intentionally fragmented by a separate program at launch (plus beneficial-order error injection). BOs are initially backed at a sub-optimal order and scores start lower; the background defrag worker then promotes the backing to the beneficial order and scores climb back in line with the unfragmented baseline.
- IGT:
https://patchwork.freedesktop.org/patch/739052/?series=170046&rev=2

Matt

v3:
- Fix valid raised raised by Sashiko on v2 [2]

[1] https://patchwork.freedesktop.org/series/169053/
[2] https://sashiko.dev/#/patchset/20260710215442.2444235-1-matthew.brost%40intel.com

Cc Alex Deucher <[email protected]>
Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>

Matthew Brost (31):
drm/ttm/pool: Allow backing off reclaim at the beneficial order drm/ttm/pool: Add ttm_pool_page_order_nodma() helper drm/ttm: Record sub-optimal page order allocations in ttm_tt drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc() drm/ttm: Support defragmentation moves
drm/ttm: Add fault injection for beneficial-order allocation failures drm/ttm: Harvest beneficial-order pages on defragmentation moves drm/ttm: Bound page (re)allocation per defragmentation move drm/ttm: Preallocate beneficial-order defrag pages outside the lock drm/ttm: Add full out-of-lock preallocation for ttm_pool_alloc() drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind() drm/xe: Destroy page tables after unlinking all VMAs on VM close drm/xe: Track BOs backed at a sub-optimal page order drm/xe: Back off beneficial-order reclaim under defrag pressure drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies drm/xe: Handle defrag moves in xe_bo_move()
drm/xe: Skip self-copies for borrowed pages on defrag moves drm/xe: Add a page defragmentation worker
drm/xe: Add defrag GT stats
drm/xe: Add Kconfig.profile options for BO defrag configuration drm/xe: Defrag using out-of-lock page preallocation drm/xe: Add defrag profiling tracepoints
drm/xe: Preallocate system BO backing outside the dma-resv lock drm/xe: Add tracepoint for xe_gem_create_ioctl
drm/xe: Add IOVA-based xe_res_cursor variant
drm/xe: Use IOVA-based DMA mapping for eligible tt BOs drm/xe: Add per-device dependency scheduler for IOVA defrag finalize drm/xe: Add packed copy-step IOVA mapping for defrag drm/xe: Blit src-natural to dst-packed for defrag-IOVA copies drm/xe: Finalize defrag-IOVA moves with post-copy job drm/amdgpu: Preallocate system BO pages outside the reservation lock

Thomas Hellström (2):
drm/gpusvm: Add a DMA-mapping accounting callback drm/xe: Add debugfs stats for DMA-mapped pages per order

 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  |    4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c   |    2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c       |   73 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.h       |    5 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c    |    3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    |    4 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       |   39 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h       |    3 +
 drivers/gpu/drm/drm_gpusvm.c                  |   17 +-
 drivers/gpu/drm/ttm/ttm_bo.c                  |  118 +-
 drivers/gpu/drm/ttm/ttm_bo_util.c             |   21 +-
 drivers/gpu/drm/ttm/ttm_pool.c                |  947 ++++++++-
 drivers/gpu/drm/ttm/ttm_tt.c                  |   67 +
 drivers/gpu/drm/xe/Kconfig.profile            |   40 +
 drivers/gpu/drm/xe/tests/xe_bo.c              |    8 +-
 drivers/gpu/drm/xe/tests/xe_dma_buf.c         |    2 +-
 drivers/gpu/drm/xe/tests/xe_migrate.c         |   12 +-
 drivers/gpu/drm/xe/xe_bo.c                    | 1699 ++++++++++++++++-
 drivers/gpu/drm/xe/xe_bo.h                    |   15 +-
 drivers/gpu/drm/xe/xe_bo_types.h              |    6 +
 drivers/gpu/drm/xe/xe_debugfs.c               |   26 +
 drivers/gpu/drm/xe/xe_device.c                |   41 +
 drivers/gpu/drm/xe/xe_device_types.h          |   57 +
 drivers/gpu/drm/xe/xe_dma_buf.c               |    2 +-
 drivers/gpu/drm/xe/xe_ggtt.c                  |    2 +-
 drivers/gpu/drm/xe/xe_gt_stats.c              |    5 +
 drivers/gpu/drm/xe/xe_gt_stats_types.h        |   17 +
 drivers/gpu/drm/xe/xe_migrate.c               |  531 +++++-
 drivers/gpu/drm/xe/xe_migrate.h               |   17 +
 drivers/gpu/drm/xe/xe_pt.c                    |    2 +-
 drivers/gpu/drm/xe/xe_res_cursor.h            |   56 +-
 drivers/gpu/drm/xe/xe_svm.c                   |   44 +-
 drivers/gpu/drm/xe/xe_svm.h                   |    4 +-
 drivers/gpu/drm/xe/xe_trace_bo.h              |   90 +
 drivers/gpu/drm/xe/xe_userptr.c               |   63 +
 drivers/gpu/drm/xe/xe_userptr.h               |    1 +
 drivers/gpu/drm/xe/xe_vm.c                    |  149 +-
 drivers/gpu/drm/xe/xe_vm.h                    |    5 +
 include/drm/drm_gpusvm.h                      |   19 +
 include/drm/ttm/ttm_bo.h                      |   64 +
 include/drm/ttm/ttm_pool.h                    |   46 +
 include/drm/ttm/ttm_tt.h                      |   43 +-
 42 files changed, 4179 insertions(+), 190 deletions(-)
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

PATCH v3 1/33 drm/ttm/pool: Allow backing off reclaim at the beneficial order

Allocating at the pool's beneficial order normally uses __GFP_RETRY_MAYFAIL to push hard for a contiguous high-order page. When beneficial-order allocations have recently been failing, this retry just burns CPU cycles in reclaim/compaction before falling back to a smaller order anyway.

Add a beneficial_reclaim_backoff flag to struct ttm_operation_ctx and plumb it through to ttm_pool_alloc_page(). When set, the beneficial order is treated like the orders we already consider not worth stalling
for: __GFP_RECLAIM is cleared so the allocation skips reclaim/compaction entirely, and the __GFP_NORETRY clearing / __GFP_RETRY_MAYFAIL promotion is skipped. Together this makes the allocator back off quickly to a smaller order instead of stalling.

This is a no-op until a caller opts in.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---

v3:
- Require a valid beneficial order before backing off reclaim in ttm_pool_alloc_page() (Sashiko)

---
 drivers/gpu/drm/ttm/ttm_pool.c | 12 ++++++++----
 include/drm/ttm/ttm_bo.h       |  7 +++++++
 2 files changed, 15 insertions(+), 4 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +8 -4
index 3d5f2ae0a456..ab242ad4339e 100644
@@ -145,7 +145,8 @@ static int ttm_pool_nid(struct ttm_pool *pool)
 
 /* Allocate pages of size 1 << order with the given gfp_flags */
 static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
-					unsigned int order)
+					unsigned int order,
+					bool beneficial_reclaim_backoff)
 {
 	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
 	unsigned long attr = DMA_ATTR_FORCE_CONTIGUOUS;
@@ -165,10 +166,12 @@ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
 	 * Do not add latency to the allocation path for allocations orders
 	 * device tolds us do not bring them additional performance gains.
 	 */
-	if (order && beneficial_order && order != beneficial_order)
+	if (order && beneficial_order &&
+	    (beneficial_reclaim_backoff || order != beneficial_order))
 		gfp_flags &= ~__GFP_RECLAIM;
 
-	if (beneficial_order && order == beneficial_order) {
+	if (!beneficial_reclaim_backoff &&
+	    beneficial_order && order == beneficial_order) {
 		gfp_flags &= ~__GFP_NORETRY;
 		gfp_flags |= __GFP_RETRY_MAYFAIL;
 	}
@@ -814,7 +817,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		if (!p) {
 			page_caching = ttm_cached;
 			allow_pools = false;
-			p = ttm_pool_alloc_page(pool, gfp_flags, order);
+			p = ttm_pool_alloc_page(pool, gfp_flags, order,
+						ctx->beneficial_reclaim_backoff);
 		}
 		/* If that fails, lower the order if possible and retry. */
 		if (!p) {
include/drm/ttm/ttm_bo.h +7 -0
index 8310bc3d55f9..fd814b5cf535 100644
@@ -187,6 +187,13 @@ struct ttm_operation_ctx {
 	 * when multiple BOs share the same reservation object @resv.
 	 */
 	bool allow_res_evict;
+	/**
+	 * @beneficial_reclaim_backoff: Back off from direct reclaim at the
+	 * pool's beneficial order. Useful when beneficial-order allocation has
+	 * recently been failing, to avoid burning CPU cycles in
+	 * reclaim/compaction before falling back to a smaller order anyway.
+	 */
+	bool beneficial_reclaim_backoff;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH v3 01/33] drm/ttm/pool: Allow backing off reclaim at the beneficial order

Quoted 32 lines · Matthew Brost

Allocating at the pool's beneficial order normally uses __GFP_RETRY_MAYFAIL to push hard for a contiguous high-order page. When beneficial-order allocations have recently been failing, this retry just burns CPU cycles in reclaim/compaction before falling back to a smaller order anyway.

Add a beneficial_reclaim_backoff flag to struct ttm_operation_ctx and plumb it through to ttm_pool_alloc_page(). When set, the beneficial order is treated like the orders we already consider not worth stalling
for: __GFP_RECLAIM is cleared so the allocation skips reclaim/compaction entirely, and the __GFP_NORETRY clearing / __GFP_RETRY_MAYFAIL promotion is skipped. Together this makes the allocator back off quickly to a smaller order instead of stalling.

This is a no-op until a caller opts in.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
Quoted 5 lines

---

v3:
- Require a valid beneficial order before backing off reclaim in ttm_pool_alloc_page() (Sashiko)

LGTM, though I need to take a deeper look at the patches that use this flag.

Niranjana

Quoted 65 lines

---
drivers/gpu/drm/ttm/ttm_pool.c | 12 ++++++++----
include/drm/ttm/ttm_bo.h | 7 +++++++

2 files changed, 15 insertions(+), 4 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +8 -4
index 3d5f2ae0a456..ab242ad4339e 100644
@@ -145,7 +145,8 @@ static int ttm_pool_nid(struct ttm_pool *pool)
 
 /* Allocate pages of size 1 << order with the given gfp_flags */
 static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
-					unsigned int order)
+					unsigned int order,
+					bool beneficial_reclaim_backoff)
 {
 	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
 	unsigned long attr = DMA_ATTR_FORCE_CONTIGUOUS;
@@ -165,10 +166,12 @@ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
 	 * Do not add latency to the allocation path for allocations orders
 	 * device tolds us do not bring them additional performance gains.
 	 */
-	if (order && beneficial_order && order != beneficial_order)
+	if (order && beneficial_order &&
+	    (beneficial_reclaim_backoff || order != beneficial_order))
 		gfp_flags &= ~__GFP_RECLAIM;
 
-	if (beneficial_order && order == beneficial_order) {
+	if (!beneficial_reclaim_backoff &&
+	    beneficial_order && order == beneficial_order) {
 		gfp_flags &= ~__GFP_NORETRY;
 		gfp_flags |= __GFP_RETRY_MAYFAIL;
 	}
@@ -814,7 +817,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		if (!p) {
 			page_caching = ttm_cached;
 			allow_pools = false;
-			p = ttm_pool_alloc_page(pool, gfp_flags, order);
+			p = ttm_pool_alloc_page(pool, gfp_flags, order,
+						ctx->beneficial_reclaim_backoff);
 		}
 		/* If that fails, lower the order if possible and retry. */
 		if (!p) {
include/drm/ttm/ttm_bo.h +7 -0
index 8310bc3d55f9..fd814b5cf535 100644
@@ -187,6 +187,13 @@ struct ttm_operation_ctx {
 	 * when multiple BOs share the same reservation object @resv.
 	 */
 	bool allow_res_evict;
+	/**
+	 * @beneficial_reclaim_backoff: Back off from direct reclaim at the
+	 * pool's beneficial order. Useful when beneficial-order allocation has
+	 * recently been failing, to avoid burning CPU cycles in
+	 * reclaim/compaction before falling back to a smaller order anyway.
+	 */
+	bool beneficial_reclaim_backoff;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
Signature · 1 line
2.34.1
author maintainer
#3 lore raw

Re: [PATCH v3 01/33] drm/ttm/pool: Allow backing off reclaim at the beneficial order

Quoted 33 lines · Niranjana Vishwanathapura
Quoted 32 lines · Matthew Brost

Allocating at the pool's beneficial order normally uses __GFP_RETRY_MAYFAIL to push hard for a contiguous high-order page. When beneficial-order allocations have recently been failing, this retry just burns CPU cycles in reclaim/compaction before falling back to a smaller order anyway.

Add a beneficial_reclaim_backoff flag to struct ttm_operation_ctx and plumb it through to ttm_pool_alloc_page(). When set, the beneficial order is treated like the orders we already consider not worth stalling
for: __GFP_RECLAIM is cleared so the allocation skips reclaim/compaction entirely, and the __GFP_NORETRY clearing / __GFP_RETRY_MAYFAIL promotion is skipped. Together this makes the allocator back off quickly to a smaller order instead of stalling.

This is a no-op until a caller opts in.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
Quoted 8 lines
Quoted 5 lines

---

v3:
- Require a valid beneficial order before backing off reclaim in ttm_pool_alloc_page() (Sashiko)

LGTM, though I need to take a deeper look at the patches that use this flag.

If it isn't clear - the end result of this series is all foreground tasks will set this flag to avoid jank (i.e., no entering reclaim at higher orders) on the UI and defragmenter in background clears this flag and does aggresive reclaim pruning in higher order pages iteratively.

Some small tweaks to this algorithm will made in the next rev.

Matt

Quoted 67 lines

Niranjana

Quoted 65 lines

---
drivers/gpu/drm/ttm/ttm_pool.c | 12 ++++++++----
include/drm/ttm/ttm_bo.h | 7 +++++++

2 files changed, 15 insertions(+), 4 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +8 -4
index 3d5f2ae0a456..ab242ad4339e 100644
@@ -145,7 +145,8 @@ static int ttm_pool_nid(struct ttm_pool *pool)
 
 /* Allocate pages of size 1 << order with the given gfp_flags */
 static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
-					unsigned int order)
+					unsigned int order,
+					bool beneficial_reclaim_backoff)
 {
 	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
 	unsigned long attr = DMA_ATTR_FORCE_CONTIGUOUS;
@@ -165,10 +166,12 @@ static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags,
 	 * Do not add latency to the allocation path for allocations orders
 	 * device tolds us do not bring them additional performance gains.
 	 */
-	if (order && beneficial_order && order != beneficial_order)
+	if (order && beneficial_order &&
+	    (beneficial_reclaim_backoff || order != beneficial_order))
 		gfp_flags &= ~__GFP_RECLAIM;
 
-	if (beneficial_order && order == beneficial_order) {
+	if (!beneficial_reclaim_backoff &&
+	    beneficial_order && order == beneficial_order) {
 		gfp_flags &= ~__GFP_NORETRY;
 		gfp_flags |= __GFP_RETRY_MAYFAIL;
 	}
@@ -814,7 +817,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		if (!p) {
 			page_caching = ttm_cached;
 			allow_pools = false;
-			p = ttm_pool_alloc_page(pool, gfp_flags, order);
+			p = ttm_pool_alloc_page(pool, gfp_flags, order,
+						ctx->beneficial_reclaim_backoff);
 		}
 		/* If that fails, lower the order if possible and retry. */
 		if (!p) {
include/drm/ttm/ttm_bo.h +7 -0
index 8310bc3d55f9..fd814b5cf535 100644
@@ -187,6 +187,13 @@ struct ttm_operation_ctx {
 	 * when multiple BOs share the same reservation object @resv.
 	 */
 	bool allow_res_evict;
+	/**
+	 * @beneficial_reclaim_backoff: Back off from direct reclaim at the
+	 * pool's beneficial order. Useful when beneficial-order allocation has
+	 * recently been failing, to avoid burning CPU cycles in
+	 * reclaim/compaction before falling back to a smaller order anyway.
+	 */
+	bool beneficial_reclaim_backoff;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
Signature · 1 line
2.34.1
author maintainer
#4 lore raw

PATCH v3 2/33 drm/ttm/pool: Add ttm_pool_page_order_nodma() helper

In reply to Matthew Brost

TTM records the allocation order of a multi-order page in page->private at allocation time. For pools that do not use dma_alloc the order is stored directly, while dma_alloc pools store a struct ttm_pool_dma pointer there instead. Drivers that do their own DMA mapping (e.g. Xe) need the per-page order to walk a populated ttm_tt one chunk at a time, but cannot rely on folio_order(): TTM allocates high-order pages with alloc_pages_node() without __GFP_COMP, so they are not compound and folio_order() always returns 0.

Expose ttm_pool_page_order_nodma(), a thin wrapper that returns the order recorded in page->private. It is only valid for pages from a pool that does not use dma_alloc; whether a TTM device uses dma_alloc is fixed at ttm device init time, so callers know from their device configuration that this helper applies. Use it from the existing internal ttm_pool_page_order() to keep a single source of truth.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 25 ++++++++++++++++++++++++-
 include/drm/ttm/ttm_pool.h     |  2 ++
 2 files changed, 26 insertions(+), 1 deletion(-)
drivers/gpu/drm/ttm/ttm_pool.c +24 -1
index ab242ad4339e..5b7d7f5ae53d 100644
@@ -461,6 +461,29 @@ static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free)
 	return num_pages;
 }
 
+/**
+ * ttm_pool_page_order_nodma() - Allocation order of a non-dma_alloc TTM page
+ * @p: The page to query.
+ *
+ * Return the allocation order that the TTM pool allocator recorded in
+ * @p->private at allocation time. This only works for pages that were
+ * allocated from a pool which does *not* use dma_alloc (i.e. pages backed by
+ * alloc_pages_node()), where TTM stores the order directly in page->private.
+ *
+ * For dma_alloc pools, page->private instead holds a struct ttm_pool_dma
+ * pointer and this helper must not be used. Whether a TTM device uses
+ * dma_alloc is fixed at ttm device init time, so callers are expected to know
+ * from their TTM device configuration that their pages are not dma_alloc
+ * backed before using this helper.
+ *
+ * Return: The allocation order of the page.
+ */
+unsigned int ttm_pool_page_order_nodma(struct page *p)
+{
+	return p->private;
+}
+EXPORT_SYMBOL(ttm_pool_page_order_nodma);
+
 /* Return the allocation order based for a page */
 static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
 {
@@ -470,7 +493,7 @@ static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
 		return dma->vaddr & ~PAGE_MASK;
 	}
 
-	return p->private;
+	return ttm_pool_page_order_nodma(p);
 }
 
 /*
include/drm/ttm/ttm_pool.h +2 -0
index 26ee592e1994..753203980e2c 100644
@@ -94,6 +94,8 @@ long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *ttm,
 int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			       const struct ttm_operation_ctx *ctx);
 
+unsigned int ttm_pool_page_order_nodma(struct page *p);
+
 int ttm_pool_mgr_init(unsigned long num_pages);
 void ttm_pool_mgr_fini(void);
 
Signature · 1 line
2.34.1
reviewer Reviewed-by on v3 2/33
#5 lore raw

Re: [PATCH v3 02/33] drm/ttm/pool: Add ttm_pool_page_order_nodma() helper

Quoted 16 lines · Matthew Brost

TTM records the allocation order of a multi-order page in page->private at allocation time. For pools that do not use dma_alloc the order is stored directly, while dma_alloc pools store a struct ttm_pool_dma pointer there instead. Drivers that do their own DMA mapping (e.g. Xe) need the per-page order to walk a populated ttm_tt one chunk at a time, but cannot rely on folio_order(): TTM allocates high-order pages with alloc_pages_node() without __GFP_COMP, so they are not compound and folio_order() always returns 0.

Expose ttm_pool_page_order_nodma(), a thin wrapper that returns the order recorded in page->private. It is only valid for pages from a pool that does not use dma_alloc; whether a TTM device uses dma_alloc is fixed at ttm device init time, so callers know from their device configuration that this helper applies. Use it from the existing internal ttm_pool_page_order() to keep a single source of truth.

Quoted 15 lines
Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
Reviewed-by on v3 2/33
#6 lore raw

Re: [PATCH v3 02/33] drm/ttm/pool: Add ttm_pool_page_order_nodma() helper

In reply to Matthew Brost
Quoted 56 lines · Matthew Brost

TTM records the allocation order of a multi-order page in page->private at allocation time. For pools that do not use dma_alloc the order is stored directly, while dma_alloc pools store a struct ttm_pool_dma pointer there instead. Drivers that do their own DMA mapping (e.g. Xe) need the per-page order to walk a populated ttm_tt one chunk at a time, but cannot rely on folio_order(): TTM allocates high-order pages with alloc_pages_node() without __GFP_COMP, so they are not compound and folio_order() always returns 0.

Expose ttm_pool_page_order_nodma(), a thin wrapper that returns the order recorded in page->private. It is only valid for pages from a pool that does not use dma_alloc; whether a TTM device uses dma_alloc is fixed at ttm device init time, so callers know from their device configuration that this helper applies. Use it from the existing internal ttm_pool_page_order() to keep a single source of truth.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---
drivers/gpu/drm/ttm/ttm_pool.c | 25 ++++++++++++++++++++++++-
include/drm/ttm/ttm_pool.h | 2 ++

2 files changed, 26 insertions(+), 1 deletion(-)
drivers/gpu/drm/ttm/ttm_pool.c +12 -0
index ab242ad4339e..5b7d7f5ae53d 100644
@@ -461,6 +461,29 @@ static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free)
 	return num_pages;
 }
 
+/**
+ * ttm_pool_page_order_nodma() - Allocation order of a non-dma_alloc TTM page
+ * @p: The page to query.
+ *
+ * Return the allocation order that the TTM pool allocator recorded in
+ * @p->private at allocation time. This only works for pages that were
+ * allocated from a pool which does *not* use dma_alloc (i.e. pages backed by
+ * alloc_pages_node()), where TTM stores the order directly in page->private.
+ *
+ * For dma_alloc pools, page->private instead holds a struct ttm_pool_dma
+ * pointer and this helper must not be used. Whether a TTM device uses
+ * dma_alloc is fixed at ttm device init time, so callers are expected to know

drivers/gpu/drm/ttm/ttm_pool.c static unsigned int ttm_pool_shrink(int nid, unsigned long num_to_free) · line 485

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+ * from their TTM device configuration that their pages are not dma_alloc
+ * backed before using this helper.
+ *
+ * Return: The allocation order of the page.
+ */
+unsigned int ttm_pool_page_order_nodma(struct page *p)
+{
+	return p->private;
+}
+EXPORT_SYMBOL(ttm_pool_page_order_nodma);

NIT...make page pointer const?
LGTM.

Quoted 29 lines
drivers/gpu/drm/ttm/ttm_pool.c +2 -1
+
 /* Return the allocation order based for a page */
 static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
 {
@@ -470,7 +493,7 @@ static unsigned int ttm_pool_page_order(struct ttm_pool *pool, struct page *p)
 		return dma->vaddr & ~PAGE_MASK;
 	}
 
-	return p->private;
+	return ttm_pool_page_order_nodma(p);
 }
 
 /*
include/drm/ttm/ttm_pool.h +2 -0
index 26ee592e1994..753203980e2c 100644
@@ -94,6 +94,8 @@ long ttm_pool_backup(struct ttm_pool *pool, struct ttm_tt *ttm,
 int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			       const struct ttm_operation_ctx *ctx);
 
+unsigned int ttm_pool_page_order_nodma(struct page *p);
+
 int ttm_pool_mgr_init(unsigned long num_pages);
 void ttm_pool_mgr_fini(void);
 
Signature · 1 line
2.34.1
author maintainer
#7 lore raw

PATCH v3 3/33 drm/ttm: Record sub-optimal page order allocations in ttm_tt

In reply to Matthew Brost

When __ttm_pool_alloc() fails to allocate a chunk at the device's beneficial order and falls back to a smaller order, the object ends up backed by a sub-optimal set of pages. Nothing currently records this, so a driver has no way to know an object would benefit from being re-backed with beneficial-order pages later.

Add TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED to struct ttm_tt page_flags, cleared at the start of every fresh allocation (in both ttm_pool_alloc() and ttm_pool_restore_and_alloc()). Set it in __ttm_pool_alloc() when an allocation at exactly the beneficial order fails before dropping to a lower order.

Drivers can use this hint to queue the buffer object for later defragmentation.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 13 +++++++++++++
 include/drm/ttm/ttm_tt.h       | 24 +++++++++++++++++++++++-
 2 files changed, 36 insertions(+), 1 deletion(-)
drivers/gpu/drm/ttm/ttm_pool.c +13 -0
index 5b7d7f5ae53d..e9175c0290d7 100644
@@ -796,6 +796,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			    struct ttm_pool_alloc_state *alloc,
 			    struct ttm_pool_tt_restore *restore)
 {
+	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
 	enum ttm_caching page_caching;
 	gfp_t gfp_flags = GFP_USER;
 	pgoff_t caching_divide;
@@ -846,6 +847,16 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		/* If that fails, lower the order if possible and retry. */
 		if (!p) {
 			if (order) {
+				/*
+				 * Failing to allocate at the device's beneficial
+				 * order means we are about to back this object
+				 * with a sub-optimal (smaller order) set of
+				 * pages. Record it so the driver can later try to
+				 * defragment the object back to beneficial order.
+				 */
+				if (beneficial_order && order == beneficial_order)
+					tt->page_flags |=
+						TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 				--order;
 				page_caching = tt->caching;
 				allow_pools = true;
@@ -910,6 +921,7 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (WARN_ON(ttm_tt_is_backed_up(tt)))
 		return -EINVAL;
 
+	tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 	ttm_pool_alloc_state_init(tt, &alloc);
 
 	return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
@@ -942,6 +954,7 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (!restore) {
 		gfp_t gfp = GFP_KERNEL | __GFP_NOWARN;
 
+		tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 		ttm_pool_alloc_state_init(tt, &alloc);
 		if (ctx->gfp_retry_mayfail)
 			gfp |= __GFP_RETRY_MAYFAIL;
include/drm/ttm/ttm_tt.h +23 -1
index 406437ad674b..ce7533677d77 100644
@@ -90,6 +90,14 @@ struct ttm_tt {
 	 * TTM_TT_FLAG_BACKED_UP: TTM internal only. This is set if the
 	 * struct ttm_tt has been (possibly partially) backed up.
 	 *
+	 * TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED: Set by the TTM pool allocator
+	 * when at least one chunk that could have been allocated at the pool's
+	 * beneficial order had to fall back to a smaller order. This indicates
+	 * that a sub-optimal set of pages was chosen for this object, and can
+	 * be used by drivers to queue the buffer object for later
+	 * defragmentation. Drivers should access this via the
+	 * ttm_tt_is_beneficial_order_failed() helper.
+	 *
 	 * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only. DO NOT USE. This is
 	 * set by TTM after ttm_tt_populate() has successfully returned, and is
 	 * then unset when TTM calls ttm_tt_unpopulate().
@@ -101,8 +109,9 @@ struct ttm_tt {
 #define TTM_TT_FLAG_EXTERNAL_MAPPABLE	BIT(3)
 #define TTM_TT_FLAG_DECRYPTED		BIT(4)
 #define TTM_TT_FLAG_BACKED_UP	        BIT(5)
+#define TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED	BIT(6)
 
-#define TTM_TT_FLAG_PRIV_POPULATED	BIT(6)
+#define TTM_TT_FLAG_PRIV_POPULATED	BIT(7)
 	uint32_t page_flags;
 	/** @num_pages: Number of pages in the page array. */
 	uint32_t num_pages;
@@ -179,6 +188,19 @@ static inline void ttm_tt_clear_backed_up(struct ttm_tt *tt)
 	tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
 }
 
+/**
+ * ttm_tt_is_beneficial_order_failed() - Whether the tt is backed at a
+ * sub-optimal page order
+ * @tt: The struct ttm_tt.
+ *
+ * Return: true if the pool allocator had to fall back below the pool's
+ * beneficial order when backing this tt, false otherwise.
+ */
+static inline bool ttm_tt_is_beneficial_order_failed(const struct ttm_tt *tt)
+{
+	return tt->page_flags & TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+}
+
 /**
  * ttm_tt_create
  *
Signature · 1 line
2.34.1
#8 lore raw

Re: [PATCH v3 03/33] drm/ttm: Record sub-optimal page order allocations in ttm_tt

Quoted 51 lines · Matthew Brost

When __ttm_pool_alloc() fails to allocate a chunk at the device's beneficial order and falls back to a smaller order, the object ends up backed by a sub-optimal set of pages. Nothing currently records this, so a driver has no way to know an object would benefit from being re-backed with beneficial-order pages later.

Add TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED to struct ttm_tt page_flags, cleared at the start of every fresh allocation (in both ttm_pool_alloc() and ttm_pool_restore_and_alloc()). Set it in __ttm_pool_alloc() when an allocation at exactly the beneficial order fails before dropping to a lower order.

Drivers can use this hint to queue the buffer object for later defragmentation.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---
drivers/gpu/drm/ttm/ttm_pool.c | 13 +++++++++++++
include/drm/ttm/ttm_tt.h | 24 +++++++++++++++++++++++-

2 files changed, 36 insertions(+), 1 deletion(-)
drivers/gpu/drm/ttm/ttm_pool.c +1 -0
index 5b7d7f5ae53d..e9175c0290d7 100644
@@ -796,6 +796,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			    struct ttm_pool_alloc_state *alloc,
 			    struct ttm_pool_tt_restore *restore)
 {
+	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
 	enum ttm_caching page_caching;
 	gfp_t gfp_flags = GFP_USER;
 	pgoff_t caching_divide;
@@ -846,6 +847,16 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		/* If that fails, lower the order if possible and retry. */
 		if (!p) {
 			if (order) {

drivers/gpu/drm/ttm/ttm_pool.c static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, · line 859

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+				/*
+				 * Failing to allocate at the device's beneficial
+				 * order means we are about to back this object
+				 * with a sub-optimal (smaller order) set of
+				 * pages. Record it so the driver can later try to
+				 * defragment the object back to beneficial order.
+				 */
+				if (beneficial_order && order == beneficial_order)
+					tt->page_flags |=
+						TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;

I haven't looked at beneficial_reclaim_backoff usecase yet, but this flag gets set irrespective of whether the allocation failure resulted with/without the beneficial_reclaim_backoff request from the driver. Is that intended?

Other than that, the change here LGTM, though I still need to look at how driver uses this flag in more detail.

Niranjana

Quoted 72 lines
drivers/gpu/drm/ttm/ttm_pool.c +2 -0
 				--order;
 				page_caching = tt->caching;
 				allow_pools = true;
@@ -910,6 +921,7 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (WARN_ON(ttm_tt_is_backed_up(tt)))
 		return -EINVAL;
 
+	tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 	ttm_pool_alloc_state_init(tt, &alloc);
 
 	return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
@@ -942,6 +954,7 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (!restore) {
 		gfp_t gfp = GFP_KERNEL | __GFP_NOWARN;
 
+		tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 		ttm_pool_alloc_state_init(tt, &alloc);
 		if (ctx->gfp_retry_mayfail)
 			gfp |= __GFP_RETRY_MAYFAIL;
include/drm/ttm/ttm_tt.h +23 -1
index 406437ad674b..ce7533677d77 100644
@@ -90,6 +90,14 @@ struct ttm_tt {
 	 * TTM_TT_FLAG_BACKED_UP: TTM internal only. This is set if the
 	 * struct ttm_tt has been (possibly partially) backed up.
 	 *
+	 * TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED: Set by the TTM pool allocator
+	 * when at least one chunk that could have been allocated at the pool's
+	 * beneficial order had to fall back to a smaller order. This indicates
+	 * that a sub-optimal set of pages was chosen for this object, and can
+	 * be used by drivers to queue the buffer object for later
+	 * defragmentation. Drivers should access this via the
+	 * ttm_tt_is_beneficial_order_failed() helper.
+	 *
 	 * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only. DO NOT USE. This is
 	 * set by TTM after ttm_tt_populate() has successfully returned, and is
 	 * then unset when TTM calls ttm_tt_unpopulate().
@@ -101,8 +109,9 @@ struct ttm_tt {
 #define TTM_TT_FLAG_EXTERNAL_MAPPABLE	BIT(3)
 #define TTM_TT_FLAG_DECRYPTED		BIT(4)
 #define TTM_TT_FLAG_BACKED_UP	        BIT(5)
+#define TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED	BIT(6)
 
-#define TTM_TT_FLAG_PRIV_POPULATED	BIT(6)
+#define TTM_TT_FLAG_PRIV_POPULATED	BIT(7)
 	uint32_t page_flags;
 	/** @num_pages: Number of pages in the page array. */
 	uint32_t num_pages;
@@ -179,6 +188,19 @@ static inline void ttm_tt_clear_backed_up(struct ttm_tt *tt)
 	tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
 }
 
+/**
+ * ttm_tt_is_beneficial_order_failed() - Whether the tt is backed at a
+ * sub-optimal page order
+ * @tt: The struct ttm_tt.
+ *
+ * Return: true if the pool allocator had to fall back below the pool's
+ * beneficial order when backing this tt, false otherwise.
+ */
+static inline bool ttm_tt_is_beneficial_order_failed(const struct ttm_tt *tt)
+{
+	return tt->page_flags & TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+}
+
 /**
 * ttm_tt_create
 *
Signature · 1 line
2.34.1
author maintainer
#9 lore raw

Re: [PATCH v3 03/33] drm/ttm: Record sub-optimal page order allocations in ttm_tt

Quoted 58 lines · Niranjana Vishwanathapura
Quoted 57 lines · Matthew Brost

When __ttm_pool_alloc() fails to allocate a chunk at the device's beneficial order and falls back to a smaller order, the object ends up backed by a sub-optimal set of pages. Nothing currently records this, so a driver has no way to know an object would benefit from being re-backed with beneficial-order pages later.

Add TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED to struct ttm_tt page_flags, cleared at the start of every fresh allocation (in both ttm_pool_alloc() and ttm_pool_restore_and_alloc()). Set it in __ttm_pool_alloc() when an allocation at exactly the beneficial order fails before dropping to a lower order.

Drivers can use this hint to queue the buffer object for later defragmentation.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---
drivers/gpu/drm/ttm/ttm_pool.c | 13 +++++++++++++
include/drm/ttm/ttm_tt.h | 24 +++++++++++++++++++++++-

2 files changed, 36 insertions(+), 1 deletion(-)
drivers/gpu/drm/ttm/ttm_pool.c +7 -0
index 5b7d7f5ae53d..e9175c0290d7 100644
@@ -796,6 +796,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			    struct ttm_pool_alloc_state *alloc,
 			    struct ttm_pool_tt_restore *restore)
 {
+	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
 	enum ttm_caching page_caching;
 	gfp_t gfp_flags = GFP_USER;
 	pgoff_t caching_divide;
@@ -846,6 +847,16 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		/* If that fails, lower the order if possible and retry. */
 		if (!p) {
 			if (order) {
+				/*
+				 * Failing to allocate at the device's beneficial
+				 * order means we are about to back this object
+				 * with a sub-optimal (smaller order) set of
+				 * pages. Record it so the driver can later try to
+				 * defragment the object back to beneficial order.
Quoted 9 lines

drivers/gpu/drm/ttm/ttm_pool.c static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, · line 859

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_pool.c +4 -0
+				 */
+				if (beneficial_order && order == beneficial_order)
+					tt->page_flags |=
+						TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;

I haven't looked at beneficial_reclaim_backoff usecase yet, but this flag gets set irrespective of whether the allocation failure resulted with/without the beneficial_reclaim_backoff request from the driver. Is that intended?

Yes. The end goal (the series on list needs a slight Xe side tweak) it everything in the foreground (new memory allocations, swapin after shrinking, etc..) sets beneficial_reclaim_backoff (no reclaim). In that case if we don't get optimial pages, the defragmenter in the background does aggresive reclaim and then prunes in pages it finds at beneficial_order in a way we don't miss page flips.

Matt

Quoted 77 lines

Other than that, the change here LGTM, though I still need to look at how driver uses this flag in more detail.

Niranjana

Quoted 72 lines
drivers/gpu/drm/ttm/ttm_pool.c +2 -0
 				--order;
 				page_caching = tt->caching;
 				allow_pools = true;
@@ -910,6 +921,7 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (WARN_ON(ttm_tt_is_backed_up(tt)))
 		return -EINVAL;
 
+	tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 	ttm_pool_alloc_state_init(tt, &alloc);
 
 	return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
@@ -942,6 +954,7 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (!restore) {
 		gfp_t gfp = GFP_KERNEL | __GFP_NOWARN;
 
+		tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 		ttm_pool_alloc_state_init(tt, &alloc);
 		if (ctx->gfp_retry_mayfail)
 			gfp |= __GFP_RETRY_MAYFAIL;
include/drm/ttm/ttm_tt.h +23 -1
index 406437ad674b..ce7533677d77 100644
@@ -90,6 +90,14 @@ struct ttm_tt {
 	 * TTM_TT_FLAG_BACKED_UP: TTM internal only. This is set if the
 	 * struct ttm_tt has been (possibly partially) backed up.
 	 *
+	 * TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED: Set by the TTM pool allocator
+	 * when at least one chunk that could have been allocated at the pool's
+	 * beneficial order had to fall back to a smaller order. This indicates
+	 * that a sub-optimal set of pages was chosen for this object, and can
+	 * be used by drivers to queue the buffer object for later
+	 * defragmentation. Drivers should access this via the
+	 * ttm_tt_is_beneficial_order_failed() helper.
+	 *
 	 * TTM_TT_FLAG_PRIV_POPULATED: TTM internal only. DO NOT USE. This is
 	 * set by TTM after ttm_tt_populate() has successfully returned, and is
 	 * then unset when TTM calls ttm_tt_unpopulate().
@@ -101,8 +109,9 @@ struct ttm_tt {
 #define TTM_TT_FLAG_EXTERNAL_MAPPABLE	BIT(3)
 #define TTM_TT_FLAG_DECRYPTED		BIT(4)
 #define TTM_TT_FLAG_BACKED_UP	        BIT(5)
+#define TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED	BIT(6)
 
-#define TTM_TT_FLAG_PRIV_POPULATED	BIT(6)
+#define TTM_TT_FLAG_PRIV_POPULATED	BIT(7)
 	uint32_t page_flags;
 	/** @num_pages: Number of pages in the page array. */
 	uint32_t num_pages;
@@ -179,6 +188,19 @@ static inline void ttm_tt_clear_backed_up(struct ttm_tt *tt)
 	tt->page_flags &= ~TTM_TT_FLAG_BACKED_UP;
 }
 
+/**
+ * ttm_tt_is_beneficial_order_failed() - Whether the tt is backed at a
+ * sub-optimal page order
+ * @tt: The struct ttm_tt.
+ *
+ * Return: true if the pool allocator had to fall back below the pool's
+ * beneficial order when backing this tt, false otherwise.
+ */
+static inline bool ttm_tt_is_beneficial_order_failed(const struct ttm_tt *tt)
+{
+	return tt->page_flags & TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+}
+
 /**
 * ttm_tt_create
 *
Signature · 1 line
2.34.1
author maintainer
#10 lore raw

PATCH v3 4/33 drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc()

In reply to Matthew Brost

__ttm_pool_alloc() walks the page orders in a single loop whose body is about to grow several defragmentation phases (reusing old pages, a move budget, out-of-lock preallocation and fault injection). Left inline these interleave into one deeply nested block of gotos and combinatorial conditionals that is hard to follow and extend.

Factor the loop's working state into a struct ttm_pool_alloc_iter and pull the per-order page acquisition and order-fallback into small phase helpers, so the main loop reads as a short sequence of named steps. This is a pure refactor with no functional change; subsequent patches extend the iterator with their defrag phases rather than nesting more logic into the loop.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 159 ++++++++++++++++++++++-----------
 1 file changed, 106 insertions(+), 53 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +106 -53
index e9175c0290d7..0ca63f5ce73c 100644
@@ -791,18 +791,102 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest,
 	return min_t(unsigned int, highest, __fls(alloc->remaining_pages));
 }
 
+/**
+ * struct ttm_pool_alloc_iter - Working state for the __ttm_pool_alloc() loop
+ *
+ * Bundles the immutable inputs and the mutable per-order state that the
+ * allocation phase helpers share, so the main loop reads as a short sequence
+ * of named phases rather than one deeply nested block.
+ */
+struct ttm_pool_alloc_iter {
+	/* Immutable for the duration of the allocation. */
+	struct ttm_pool *pool;
+	struct ttm_tt *tt;
+	const struct ttm_operation_ctx *ctx;
+	struct ttm_pool_alloc_state *alloc;
+	struct ttm_pool_tt_restore *restore;
+	unsigned int beneficial_order;
+	gfp_t gfp_flags;
+
+	/* Mutated as the loop walks the orders. */
+	unsigned int order;
+	enum ttm_caching page_caching;
+	bool allow_pools;
+	struct page *p;
+};
+
+/*
+ * Acquire a single page for the current order, leaving it in @it->p (NULL on
+ * failure). Tries a same-order pool page, then a fresh system allocation.
+ */
+static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
+{
+	struct ttm_pool_type *pt;
+
+	/* First, try to allocate a page from a pool if one exists. */
+	it->p = NULL;
+	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
+	if (!it->p && pt && it->allow_pools)
+		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
+
+	/*
+	 * If that fails or previously failed, allocate from system. Note that
+	 * this also disallows additional pool allocations using write-back
+	 * cached pools of the same order.
+	 */
+	if (!it->p) {
+		it->page_caching = ttm_cached;
+		it->allow_pools = false;
+		it->p = ttm_pool_alloc_page(it->pool, it->gfp_flags, it->order,
+					    it->ctx->beneficial_reclaim_backoff);
+	}
+}
+
+/*
+ * The current order could not be satisfied. Lower it and retry, recording a
+ * sub-optimal backing (so the driver can later re-defrag) when we drop below
+ * the beneficial order.
+ *
+ * Returns 0 to retry the (lowered) order, or a negative errno.
+ */
+static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
+{
+	if (!it->order)
+		return -ENOMEM;
+
+	/*
+	 * Failing to allocate at the device's beneficial order means we are
+	 * about to back this object with a sub-optimal (smaller order) set of
+	 * pages. Record it so the driver can later try to defragment the object
+	 * back to beneficial order.
+	 */
+	if (it->beneficial_order && it->order == it->beneficial_order)
+		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+
+	it->order--;
+	it->page_caching = it->tt->caching;
+	it->allow_pools = true;
+
+	return 0;
+}
+
 static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			    const struct ttm_operation_ctx *ctx,
 			    struct ttm_pool_alloc_state *alloc,
 			    struct ttm_pool_tt_restore *restore)
 {
-	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
-	enum ttm_caching page_caching;
+	struct ttm_pool_alloc_iter it = {
+		.pool = pool,
+		.tt = tt,
+		.ctx = ctx,
+		.alloc = alloc,
+		.restore = restore,
+		.beneficial_order = ttm_pool_beneficial_order(pool),
+		.page_caching = tt->caching,
+		.allow_pools = true,
+	};
 	gfp_t gfp_flags = GFP_USER;
 	pgoff_t caching_divide;
-	unsigned int order;
-	bool allow_pools;
-	struct page *p;
 	int r;
 
 	WARN_ON(!alloc->remaining_pages || ttm_tt_is_populated(tt));
@@ -819,59 +903,28 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	else
 		gfp_flags |= GFP_HIGHUSER;
 
-	page_caching = tt->caching;
-	allow_pools = true;
-	for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
+	it.gfp_flags = gfp_flags;
+
+	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
-	     order = ttm_pool_alloc_find_order(order, alloc)) {
-		struct ttm_pool_type *pt;
-
-		/* First, try to allocate a page from a pool if one exists. */
-		p = NULL;
-		pt = ttm_pool_select_type(pool, page_caching, order);
-		if (pt && allow_pools)
-			p = ttm_pool_type_take(pt, ttm_pool_nid(pool));
-
-		/*
-		 * If that fails or previously failed, allocate from system.
-		 * Note that this also disallows additional pool allocations using
-		 * write-back cached pools of the same order. Consider removing
-		 * that behaviour.
-		 */
-		if (!p) {
-			page_caching = ttm_cached;
-			allow_pools = false;
-			p = ttm_pool_alloc_page(pool, gfp_flags, order,
-						ctx->beneficial_reclaim_backoff);
-		}
-		/* If that fails, lower the order if possible and retry. */
-		if (!p) {
-			if (order) {
-				/*
-				 * Failing to allocate at the device's beneficial
-				 * order means we are about to back this object
-				 * with a sub-optimal (smaller order) set of
-				 * pages. Record it so the driver can later try to
-				 * defragment the object back to beneficial order.
-				 */
-				if (beneficial_order && order == beneficial_order)
-					tt->page_flags |=
-						TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
-				--order;
-				page_caching = tt->caching;
-				allow_pools = true;
-				continue;
-			}
-			r = -ENOMEM;
-			goto error_free_all;
+	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
+		/* Acquire a page (pool / system) for this order. */
+		ttm_pool_iter_acquire_page(&it);
+		if (!it.p) {
+			r = ttm_pool_iter_lower_order(&it);
+			if (r)
+				goto error_free_all;
+			continue;
 		}
-		r = ttm_pool_page_allocated(pool, order, p, page_caching, alloc,
-					    restore);
+
+		r = ttm_pool_page_allocated(pool, it.order, it.p,
+					    it.page_caching, alloc, restore);
 		if (r)
 			goto error_free_page;
 
 		if (ttm_pool_restore_valid(restore)) {
-			r = ttm_pool_restore_commit(restore, tt->backup, ctx, alloc);
+			r = ttm_pool_restore_commit(restore, tt->backup, ctx,
+						    alloc);
 			if (r)
 				goto error_free_all;
 		}
@@ -887,7 +940,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	return 0;
 
 error_free_page:
-	ttm_pool_free_page(pool, page_caching, order, p, false);
+	ttm_pool_free_page(pool, it.page_caching, it.order, it.p, false);
 
 error_free_all:
 	if (tt->restore)
Signature · 1 line
2.34.1
#11 lore raw

Re: [PATCH v3 04/33] drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc()

Quoted 64 lines · Matthew Brost

__ttm_pool_alloc() walks the page orders in a single loop whose body is about to grow several defragmentation phases (reusing old pages, a move budget, out-of-lock preallocation and fault injection). Left inline these interleave into one deeply nested block of gotos and combinatorial conditionals that is hard to follow and extend.

Factor the loop's working state into a struct ttm_pool_alloc_iter and pull the per-order page acquisition and order-fallback into small phase helpers, so the main loop reads as a short sequence of named steps. This is a pure refactor with no functional change; subsequent patches extend the iterator with their defrag phases rather than nesting more logic into the loop.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---
drivers/gpu/drm/ttm/ttm_pool.c | 159 ++++++++++++++++++++++-----------

1 file changed, 106 insertions(+), 53 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +24 -0
index e9175c0290d7..0ca63f5ce73c 100644
@@ -791,18 +791,102 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest,
 	return min_t(unsigned int, highest, __fls(alloc->remaining_pages));
 }
 
+/**
+ * struct ttm_pool_alloc_iter - Working state for the __ttm_pool_alloc() loop
+ *
+ * Bundles the immutable inputs and the mutable per-order state that the
+ * allocation phase helpers share, so the main loop reads as a short sequence
+ * of named phases rather than one deeply nested block.
+ */
+struct ttm_pool_alloc_iter {
+	/* Immutable for the duration of the allocation. */
+	struct ttm_pool *pool;
+	struct ttm_tt *tt;
+	const struct ttm_operation_ctx *ctx;
+	struct ttm_pool_alloc_state *alloc;
+	struct ttm_pool_tt_restore *restore;
+	unsigned int beneficial_order;
+	gfp_t gfp_flags;
+
+	/* Mutated as the loop walks the orders. */
+	unsigned int order;
+	enum ttm_caching page_caching;
+	bool allow_pools;
+	struct page *p;
+};
+

drivers/gpu/drm/ttm/ttm_pool.c static unsigned int ttm_pool_alloc_find_order(unsigned int highest, · line 827

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+/*
+ * Acquire a single page for the current order, leaving it in @it->p (NULL on
+ * failure). Tries a same-order pool page, then a fresh system allocation.
+ */
+static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
+{
+	struct ttm_pool_type *pt;
+
+	/* First, try to allocate a page from a pool if one exists. */
+	it->p = NULL;

NIT...Looks bit odd to set this to NULL here and check for '!ti->p' below. But I see subsequent patches add more code here, probably should be fine or add a temporary comment until then.

Quoted 126 lines
drivers/gpu/drm/ttm/ttm_pool.c +62 -49
+	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
+	if (!it->p && pt && it->allow_pools)
+		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
+
+	/*
+	 * If that fails or previously failed, allocate from system. Note that
+	 * this also disallows additional pool allocations using write-back
+	 * cached pools of the same order.
+	 */
+	if (!it->p) {
+		it->page_caching = ttm_cached;
+		it->allow_pools = false;
+		it->p = ttm_pool_alloc_page(it->pool, it->gfp_flags, it->order,
+					    it->ctx->beneficial_reclaim_backoff);
+	}
+}
+
+/*
+ * The current order could not be satisfied. Lower it and retry, recording a
+ * sub-optimal backing (so the driver can later re-defrag) when we drop below
+ * the beneficial order.
+ *
+ * Returns 0 to retry the (lowered) order, or a negative errno.
+ */
+static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
+{
+	if (!it->order)
+		return -ENOMEM;
+
+	/*
+	 * Failing to allocate at the device's beneficial order means we are
+	 * about to back this object with a sub-optimal (smaller order) set of
+	 * pages. Record it so the driver can later try to defragment the object
+	 * back to beneficial order.
+	 */
+	if (it->beneficial_order && it->order == it->beneficial_order)
+		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+
+	it->order--;
+	it->page_caching = it->tt->caching;
+	it->allow_pools = true;
+
+	return 0;
+}
+
 static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			    const struct ttm_operation_ctx *ctx,
 			    struct ttm_pool_alloc_state *alloc,
 			    struct ttm_pool_tt_restore *restore)
 {
-	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
-	enum ttm_caching page_caching;
+	struct ttm_pool_alloc_iter it = {
+		.pool = pool,
+		.tt = tt,
+		.ctx = ctx,
+		.alloc = alloc,
+		.restore = restore,
+		.beneficial_order = ttm_pool_beneficial_order(pool),
+		.page_caching = tt->caching,
+		.allow_pools = true,
+	};
 	gfp_t gfp_flags = GFP_USER;
 	pgoff_t caching_divide;
-	unsigned int order;
-	bool allow_pools;
-	struct page *p;
 	int r;
 
 	WARN_ON(!alloc->remaining_pages || ttm_tt_is_populated(tt));
@@ -819,59 +903,28 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	else
 		gfp_flags |= GFP_HIGHUSER;
 
-	page_caching = tt->caching;
-	allow_pools = true;
-	for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
+	it.gfp_flags = gfp_flags;
+
+	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
-	     order = ttm_pool_alloc_find_order(order, alloc)) {
-		struct ttm_pool_type *pt;
-
-		/* First, try to allocate a page from a pool if one exists. */
-		p = NULL;
-		pt = ttm_pool_select_type(pool, page_caching, order);
-		if (pt && allow_pools)
-			p = ttm_pool_type_take(pt, ttm_pool_nid(pool));
-
-		/*
-		 * If that fails or previously failed, allocate from system.
-		 * Note that this also disallows additional pool allocations using
-		 * write-back cached pools of the same order. Consider removing
-		 * that behaviour.
-		 */
-		if (!p) {
-			page_caching = ttm_cached;
-			allow_pools = false;
-			p = ttm_pool_alloc_page(pool, gfp_flags, order,
-						ctx->beneficial_reclaim_backoff);
-		}
-		/* If that fails, lower the order if possible and retry. */
-		if (!p) {
-			if (order) {
-				/*
-				 * Failing to allocate at the device's beneficial
-				 * order means we are about to back this object
-				 * with a sub-optimal (smaller order) set of
-				 * pages. Record it so the driver can later try to
-				 * defragment the object back to beneficial order.
-				 */
-				if (beneficial_order && order == beneficial_order)
-					tt->page_flags |=
-						TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
-				--order;
-				page_caching = tt->caching;
-				allow_pools = true;
-				continue;
-			}
-			r = -ENOMEM;
-			goto error_free_all;
+	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
+		/* Acquire a page (pool / system) for this order. */
+		ttm_pool_iter_acquire_page(&it);
+		if (!it.p) {

drivers/gpu/drm/ttm/ttm_pool.c static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, · line 921

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +7 -2
+			r = ttm_pool_iter_lower_order(&it);
+			if (r)
+				goto error_free_all;
+			continue;
 		}
-		r = ttm_pool_page_allocated(pool, order, p, page_caching, alloc,
-					    restore);
+
+		r = ttm_pool_page_allocated(pool, it.order, it.p,
+					    it.page_caching, alloc, restore);

Looks like iterator has a 'restore' field that gets set, but it is not used here or in other places. Is that intended?

Other than above comments, the patch LGTM, though need to look into other patches in this series which makes use of this iterator.

Niranjana

Quoted 22 lines
drivers/gpu/drm/ttm/ttm_pool.c +2 -1
 		if (r)
 			goto error_free_page;
 
 		if (ttm_pool_restore_valid(restore)) {
-			r = ttm_pool_restore_commit(restore, tt->backup, ctx, alloc);
+			r = ttm_pool_restore_commit(restore, tt->backup, ctx,
+						    alloc);
 			if (r)
 				goto error_free_all;
 		}
@@ -887,7 +940,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	return 0;
 

error_free_page:

drivers/gpu/drm/ttm/ttm_pool.c +1 -1
-	ttm_pool_free_page(pool, page_caching, order, p, false);
+	ttm_pool_free_page(pool, it.page_caching, it.order, it.p, false);

error_free_all:

	if (tt->restore)
Signature · 1 line
2.34.1
author maintainer
#12 lore raw

Re: [PATCH v3 04/33] drm/ttm: Introduce ttm_pool_alloc_iter for __ttm_pool_alloc()

Quoted 71 lines · Niranjana Vishwanathapura
Quoted 70 lines · Matthew Brost

__ttm_pool_alloc() walks the page orders in a single loop whose body is about to grow several defragmentation phases (reusing old pages, a move budget, out-of-lock preallocation and fault injection). Left inline these interleave into one deeply nested block of gotos and combinatorial conditionals that is hard to follow and extend.

Factor the loop's working state into a struct ttm_pool_alloc_iter and pull the per-order page acquisition and order-fallback into small phase helpers, so the main loop reads as a short sequence of named steps. This is a pure refactor with no functional change; subsequent patches extend the iterator with their defrag phases rather than nesting more logic into the loop.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---
drivers/gpu/drm/ttm/ttm_pool.c | 159 ++++++++++++++++++++++-----------

1 file changed, 106 insertions(+), 53 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +30 -0
index e9175c0290d7..0ca63f5ce73c 100644
@@ -791,18 +791,102 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest,
 	return min_t(unsigned int, highest, __fls(alloc->remaining_pages));
 }
 
+/**
+ * struct ttm_pool_alloc_iter - Working state for the __ttm_pool_alloc() loop
+ *
+ * Bundles the immutable inputs and the mutable per-order state that the
+ * allocation phase helpers share, so the main loop reads as a short sequence
+ * of named phases rather than one deeply nested block.
+ */
+struct ttm_pool_alloc_iter {
+	/* Immutable for the duration of the allocation. */
+	struct ttm_pool *pool;
+	struct ttm_tt *tt;
+	const struct ttm_operation_ctx *ctx;
+	struct ttm_pool_alloc_state *alloc;
+	struct ttm_pool_tt_restore *restore;
+	unsigned int beneficial_order;
+	gfp_t gfp_flags;
+
+	/* Mutated as the loop walks the orders. */
+	unsigned int order;
+	enum ttm_caching page_caching;
+	bool allow_pools;
+	struct page *p;
+};
+
+/*
+ * Acquire a single page for the current order, leaving it in @it->p (NULL on
+ * failure). Tries a same-order pool page, then a fresh system allocation.
+ */
+static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
+{
Quoted 9 lines

drivers/gpu/drm/ttm/ttm_pool.c static unsigned int ttm_pool_alloc_find_order(unsigned int highest, · line 827

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_pool.c +4 -0
+	struct ttm_pool_type *pt;
+
+	/* First, try to allocate a page from a pool if one exists. */
+	it->p = NULL;

NIT...Looks bit odd to set this to NULL here and check for '!ti->p' below. But I see subsequent patches add more code here, probably should be fine or add a temporary comment until then.

I'm not a huge fan of temporary comments and the structure does make more sense in subsequent patches.

Quoted 131 lines
Quoted 131 lines
drivers/gpu/drm/ttm/ttm_pool.c +66 -49
+	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
+	if (!it->p && pt && it->allow_pools)
+		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
+
+	/*
+	 * If that fails or previously failed, allocate from system. Note that
+	 * this also disallows additional pool allocations using write-back
+	 * cached pools of the same order.
+	 */
+	if (!it->p) {
+		it->page_caching = ttm_cached;
+		it->allow_pools = false;
+		it->p = ttm_pool_alloc_page(it->pool, it->gfp_flags, it->order,
+					    it->ctx->beneficial_reclaim_backoff);
+	}
+}
+
+/*
+ * The current order could not be satisfied. Lower it and retry, recording a
+ * sub-optimal backing (so the driver can later re-defrag) when we drop below
+ * the beneficial order.
+ *
+ * Returns 0 to retry the (lowered) order, or a negative errno.
+ */
+static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
+{
+	if (!it->order)
+		return -ENOMEM;
+
+	/*
+	 * Failing to allocate at the device's beneficial order means we are
+	 * about to back this object with a sub-optimal (smaller order) set of
+	 * pages. Record it so the driver can later try to defragment the object
+	 * back to beneficial order.
+	 */
+	if (it->beneficial_order && it->order == it->beneficial_order)
+		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+
+	it->order--;
+	it->page_caching = it->tt->caching;
+	it->allow_pools = true;
+
+	return 0;
+}
+
 static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			    const struct ttm_operation_ctx *ctx,
 			    struct ttm_pool_alloc_state *alloc,
 			    struct ttm_pool_tt_restore *restore)
 {
-	const unsigned int beneficial_order = ttm_pool_beneficial_order(pool);
-	enum ttm_caching page_caching;
+	struct ttm_pool_alloc_iter it = {
+		.pool = pool,
+		.tt = tt,
+		.ctx = ctx,
+		.alloc = alloc,
+		.restore = restore,
+		.beneficial_order = ttm_pool_beneficial_order(pool),
+		.page_caching = tt->caching,
+		.allow_pools = true,
+	};
 	gfp_t gfp_flags = GFP_USER;
 	pgoff_t caching_divide;
-	unsigned int order;
-	bool allow_pools;
-	struct page *p;
 	int r;
 
 	WARN_ON(!alloc->remaining_pages || ttm_tt_is_populated(tt));
@@ -819,59 +903,28 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	else
 		gfp_flags |= GFP_HIGHUSER;
 
-	page_caching = tt->caching;
-	allow_pools = true;
-	for (order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
+	it.gfp_flags = gfp_flags;
+
+	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
-	     order = ttm_pool_alloc_find_order(order, alloc)) {
-		struct ttm_pool_type *pt;
-
-		/* First, try to allocate a page from a pool if one exists. */
-		p = NULL;
-		pt = ttm_pool_select_type(pool, page_caching, order);
-		if (pt && allow_pools)
-			p = ttm_pool_type_take(pt, ttm_pool_nid(pool));
-
-		/*
-		 * If that fails or previously failed, allocate from system.
-		 * Note that this also disallows additional pool allocations using
-		 * write-back cached pools of the same order. Consider removing
-		 * that behaviour.
-		 */
-		if (!p) {
-			page_caching = ttm_cached;
-			allow_pools = false;
-			p = ttm_pool_alloc_page(pool, gfp_flags, order,
-						ctx->beneficial_reclaim_backoff);
-		}
-		/* If that fails, lower the order if possible and retry. */
-		if (!p) {
-			if (order) {
-				/*
-				 * Failing to allocate at the device's beneficial
-				 * order means we are about to back this object
-				 * with a sub-optimal (smaller order) set of
-				 * pages. Record it so the driver can later try to
-				 * defragment the object back to beneficial order.
-				 */
-				if (beneficial_order && order == beneficial_order)
-					tt->page_flags |=
-						TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
-				--order;
-				page_caching = tt->caching;
-				allow_pools = true;
-				continue;
-			}
-			r = -ENOMEM;
-			goto error_free_all;
+	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
+		/* Acquire a page (pool / system) for this order. */
+		ttm_pool_iter_acquire_page(&it);
+		if (!it.p) {
+			r = ttm_pool_iter_lower_order(&it);
+			if (r)
+				goto error_free_all;
+			continue;
 		}
Quoted 9 lines

drivers/gpu/drm/ttm/ttm_pool.c static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, · line 921

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_pool.c +3 -2
-		r = ttm_pool_page_allocated(pool, order, p, page_caching, alloc,
-					    restore);
+
+		r = ttm_pool_page_allocated(pool, it.order, it.p,
+					    it.page_caching, alloc, restore);

Looks like iterator has a 'restore' field that gets set, but it is not used here or in other places. Is that intended?

It is not intended. 'restore' should be not be in the iterator as it is unused. Will remove.

Matt

Quoted 27 lines

Other than above comments, the patch LGTM, though need to look into other patches in this series which makes use of this iterator.

Niranjana

Quoted 22 lines
drivers/gpu/drm/ttm/ttm_pool.c +2 -1
 		if (r)
 			goto error_free_page;
 
 		if (ttm_pool_restore_valid(restore)) {
-			r = ttm_pool_restore_commit(restore, tt->backup, ctx, alloc);
+			r = ttm_pool_restore_commit(restore, tt->backup, ctx,
+						    alloc);
 			if (r)
 				goto error_free_all;
 		}
@@ -887,7 +940,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	return 0;
 

error_free_page:

drivers/gpu/drm/ttm/ttm_pool.c +1 -1
-	ttm_pool_free_page(pool, page_caching, order, p, false);
+	ttm_pool_free_page(pool, it.page_caching, it.order, it.p, false);

error_free_all:

	if (tt->restore)
Signature · 1 line
2.34.1
author maintainer
#13 lore raw

PATCH v3 5/33 drm/ttm: Support defragmentation moves

In reply to Matthew Brost

Add infrastructure to let a driver re-back an already-resident BO with fresh pages (e.g. at a more beneficial order) by reusing the normal move machinery.

Add ctx->defrag, which makes ttm_bo_validate() skip the "current resource already compatible" short-circuit so a move is forced even when placement is unchanged. In ttm_bo_handle_move_mem(), when defragging a populated BO, stash the old tt, create and populate a fresh one (allocating new backing), and hand the old tt to the driver move callback via bo->defrag_old_tt for copying. On success the old tt is unpopulated and freed; if populate or the move fails, the freshly allocated backing is discharged and the BO is restored to its original tt and placement.

The old tt is handed off through bo->defrag_old_tt (reservation-lock protected) rather than the operation context. The pointer describes BO state for the duration of the move, so keeping it on the buffer object is better layering and lets ttm_bo_move_accel_cleanup() pick it up.

Because a defrag move commonly fails (beneficial-order pages are often unavailable) and then leaves the BO untouched on its original backing, defer ttm_bo_unmap_virtual() for the defrag case until the new backing has been populated and the move is about to commit. Ordinary moves still unmap up front. This avoids needlessly zapping and refaulting every userspace mapping on the common failure path where nothing moved.

Teach ttm_bo_move_accel_cleanup()'s ghost path about defrag: instead of tearing the move's source tt down synchronously, hand bo->defrag_old_tt to the ghost object so it is kept populated until the copy fence signals and then unbound and destroyed asynchronously, while the BO keeps its new tt. This lets a driver pipeline defrag teardown behind the GPU copy rather than blocking on it.

This is driver-opt-in and a no-op unless ctx->defrag is set.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---

v3:
- Reject defrag moves needing a multihop bounce: WARN, restore backing, -EINVAL (Sashiko)

---
 drivers/gpu/drm/ttm/ttm_bo.c      | 100 +++++++++++++++++++++++++++---
 drivers/gpu/drm/ttm/ttm_bo_util.c |  16 ++++-
 drivers/gpu/drm/ttm/ttm_pool.c    |   5 +-
 include/drm/ttm/ttm_bo.h          |  17 +++++
 4 files changed, 127 insertions(+), 11 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +92 -8
index 3980f376e3ba..c1629b4c4f0f 100644
@@ -125,49 +125,133 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 {
 	struct ttm_device *bdev = bo->bdev;
 	bool old_use_tt, new_use_tt;
+	bool defrag;
 	int ret;
 
 	old_use_tt = !bo->resource || ttm_manager_type(bdev, bo->resource->mem_type)->use_tt;
 	new_use_tt = ttm_manager_type(bdev, mem->mem_type)->use_tt;
 
-	ttm_bo_unmap_virtual(bo);
+	/*
+	 * Only the BO that is the actual defrag target (moved in place via
+	 * ttm_bo_validate(), evict == false) is being defragmented. The same
+	 * ctx->defrag is threaded into evictions triggered while allocating
+	 * that target's new backing (ttm_bo_evict_alloc()), so guard against
+	 * an unrelated victim BO (evict == true) being misclassified as a
+	 * defrag move and needlessly reallocating its backing.
+	 */
+	defrag = ctx->defrag && !evict && new_use_tt && bo->ttm &&
+		 ttm_tt_is_populated(bo->ttm);
+
+	/*
+	 * Tear down userspace mappings up front for ordinary moves. A defrag
+	 * move is likely to leave the BO in place (beneficial-order allocation
+	 * commonly fails), so defer the unmap until the new backing has been
+	 * populated and the move is about to commit. This avoids a needless
+	 * zap and refault of every userspace mapping on the common failure,
+	 * where the BO keeps its original, still-mapped backing.
+	 */
+	if (!defrag)
+		ttm_bo_unmap_virtual(bo);
 
 	/*
 	 * Create and bind a ttm if required.
 	 */
 
 	if (new_use_tt) {
+		struct ttm_tt *defrag_old_tt = NULL;
+
+		/*
+		 * For a defrag move, stash the current populated tt and force a
+		 * fresh one to be created and populated (at beneficial order).
+		 * The old tt is handed to the driver move callback to copy the
+		 * contents from, and freed once the move completes.
+		 */
+		if (defrag) {
+			defrag_old_tt = bo->ttm;
+			bo->ttm = NULL;
+		}
+
 		/* Zero init the new TTM structure if the old location should
 		 * have used one as well.
 		 */
-		ret = ttm_tt_create(bo, old_use_tt);
-		if (ret)
+		ret = ttm_tt_create(bo, old_use_tt && !defrag_old_tt);
+		if (ret) {
+			if (defrag_old_tt)
+				bo->ttm = defrag_old_tt;
 			goto out_err;
+		}
 
-		if (mem->mem_type != TTM_PL_SYSTEM) {
+		if (defrag_old_tt || mem->mem_type != TTM_PL_SYSTEM) {
 			ret = ttm_bo_populate(bo, ctx);
-			if (ret)
+			if (ret) {
+				/*
+				 * Discharge the freshly allocated backing and
+				 * leave the BO on its original tt/placement.
+				 */
+				if (defrag_old_tt) {
+					ttm_tt_destroy(bo->bdev, bo->ttm);
+					bo->ttm = defrag_old_tt;
+				}
 				goto out_err;
+			}
 		}
+
+		bo->defrag_old_tt = defrag_old_tt;
 	}
 
 	ret = dma_resv_reserve_fences(bo->base.resv, 1);
 	if (ret)
 		goto out_err;
 
+	/*
+	 * A defrag move deferred its virtual unmap until the new backing was
+	 * successfully populated; tear the userspace mappings down now, before
+	 * the move swaps in the new pages, so the next fault re-establishes
+	 * them against the new backing.
+	 */
+	if (defrag)
+		ttm_bo_unmap_virtual(bo);
+
 	ret = bdev->funcs->move(bo, evict, ctx, mem, hop);
 	if (ret) {
-		if (ret == -EMULTIHOP)
+		if (ret == -EMULTIHOP) {
+			/*
+			 * A defrag move targets the BO's current placement and
+			 * must complete in a single hop. A driver requesting a
+			 * bounce here is unsupported: restore the original
+			 * backing and fail the defrag rather than retrying the
+			 * multihop with ctx->defrag still set, which would
+			 * misclassify the bounce as another defrag move and
+			 * overwrite/leak bo->defrag_old_tt.
+			 */
+			if (unlikely(defrag)) {
+				WARN_ON_ONCE(1);
+				ret = -EINVAL;
+				goto out_err;
+			}
 			return ret;
+		}
 		goto out_err;
 	}
 
+	if (bo->defrag_old_tt) {
+		ttm_tt_unpopulate(bo->bdev, bo->defrag_old_tt);
+		ttm_tt_destroy(bo->bdev, bo->defrag_old_tt);
+		bo->defrag_old_tt = NULL;
+	}
+
 	ctx->bytes_moved += bo->base.size;
 	return 0;
 
 out_err:
-	if (!old_use_tt)
+	if (bo->defrag_old_tt) {
+		/* Failed defrag move: restore the original backing. */
+		ttm_bo_tt_destroy(bo);
+		bo->ttm = bo->defrag_old_tt;
+		bo->defrag_old_tt = NULL;
+	} else if (!old_use_tt) {
 		ttm_bo_tt_destroy(bo);
+	}
 
 	return ret;
 }
@@ -835,7 +919,7 @@ int ttm_bo_validate(struct ttm_buffer_object *bo,
 	force_space = false;
 	do {
 		/* Check whether we need to move buffer. */
-		if (bo->resource &&
+		if (bo->resource && !ctx->defrag &&
 		    ttm_resource_compatible(bo->resource, placement,
 					    force_space))
 			return 0;
drivers/gpu/drm/ttm/ttm_bo_util.c +14 -2
index 3e3c201a0222..771313162c8d 100644
@@ -626,10 +626,22 @@ static int ttm_bo_move_to_ghost(struct ttm_buffer_object *bo,
 	 * bo to be unbound and destroyed.
 	 */
 
-	if (dst_use_tt)
+	if (bo->defrag_old_tt) {
+		/*
+		 * Defrag move: @bo already points at the freshly allocated,
+		 * beneficial-order tt that the move copied into. Hand the old
+		 * tt (the move's source) to the ghost so it is kept populated
+		 * until @fence signals and then unbound and destroyed, instead
+		 * of being torn down synchronously. @bo keeps its new tt.
+		 */
+		ghost_obj->ttm = bo->defrag_old_tt;
+		ghost_obj->defrag_old_tt = NULL;
+		bo->defrag_old_tt = NULL;
+	} else if (dst_use_tt) {
 		ghost_obj->ttm = NULL;
-	else
+	} else {
 		bo->ttm = NULL;
+	}
 
 	dma_resv_unlock(&ghost_obj->base._resv);
 	ttm_bo_put(ghost_obj);
drivers/gpu/drm/ttm/ttm_pool.c +4 -1
index 0ca63f5ce73c..d045105afc24 100644
@@ -860,8 +860,11 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	 * pages. Record it so the driver can later try to defragment the object
 	 * back to beneficial order.
 	 */
-	if (it->beneficial_order && it->order == it->beneficial_order)
+	if (it->beneficial_order && it->order == it->beneficial_order) {
 		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+		if (it->ctx->defrag)
+			return -ENOMEM;
+	}
 
 	it->order--;
 	it->page_caching = it->tt->caching;
include/drm/ttm/ttm_bo.h +17 -0
index fd814b5cf535..f4f8a8c29f04 100644
@@ -135,6 +135,16 @@ struct ttm_buffer_object {
 	 * reservation lock.
 	 */
 	struct sg_table *sg;
+
+	/**
+	 * @defrag_old_tt: Used internally during a defrag move to hand the BO's
+	 * previous, populated struct ttm_tt to the driver's move callback so
+	 * that it can copy the contents into the freshly allocated
+	 * beneficial-order backing. Set by TTM, consumed by the driver; the old
+	 * tt is freed (or restored on failure) by TTM once the move completes.
+	 * Protected by the reservation lock.
+	 */
+	struct ttm_tt *defrag_old_tt;
 };
 
 #define TTM_BO_MAP_IOMEM_MASK 0x80
@@ -194,6 +204,13 @@ struct ttm_operation_ctx {
 	 * reclaim/compaction before falling back to a smaller order anyway.
 	 */
 	bool beneficial_reclaim_backoff;
+	/**
+	 * @defrag: Defragmentation pass. Attempt a move even when the bo's
+	 * current resource is already compatible with the requested placement,
+	 * so that the backing store can be reallocated at the pool's beneficial
+	 * order.
+	 */
+	bool defrag;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
Signature · 1 line
2.34.1
#14 lore raw

Re: [PATCH v3 05/33] drm/ttm: Support defragmentation moves

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Critical] Data loss due to `ttm_bo_move_memcpy` and driver hardware copy routines ignoring…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] Data loss due to ttm_bo_move_memcpy and driver hardware copy routines ignoring bo->defrag_old_tt.
- [Critical] Asynchronous GPU copy Use-After-Free due to synchronous teardown on defrag move failure.
- [High] Victim BOs during multihop bounce moves are misclassified as defrag targets.
--

--- Patch [5]: [PATCH v3 05/33] drm/ttm: Support defragmentation moves --- commit fba7dcbd46a60b9e14c15b0602b46ebb23a7bc37
Author: Matthew Brost <[email protected]>
Subject: drm/ttm: Support defragmentation moves

This commit introduces infrastructure to re-back an already-resident buffer object with fresh pages at a more beneficial order. It stashes the old backing in a new defrag_old_tt pointer and creates a fresh one during the move.

Quoted 13 lines
drivers/gpu/drm/ttm/ttm_bo.c +1 -1
@@ -125,49 +125,133 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 {
 	struct ttm_device *bdev = bo->bdev;
 	bool old_use_tt, new_use_tt;
+	bool defrag;
 	int ret;
 
 	old_use_tt = !bo->resource || ttm_manager_type(bdev, bo->resource->mem_type)->use_tt;
 	new_use_tt = ttm_manager_type(bdev, mem->mem_type)->use_tt;
 
-	ttm_bo_unmap_virtual(bo);

drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, · line 143

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_bo.c +10 -0
+	/*
+	 * Only the BO that is the actual defrag target (moved in place via
+	 * ttm_bo_validate(), evict == false) is being defragmented. The same
+	 * ctx->defrag is threaded into evictions triggered while allocating
+	 * that target's new backing (ttm_bo_evict_alloc()), so guard against
+	 * an unrelated victim BO (evict == true) being misclassified as a
+	 * defrag move and needlessly reallocating its backing.
+	 */
+	defrag = ctx->defrag && !evict && new_use_tt && bo->ttm &&
+		 ttm_tt_is_populated(bo->ttm);

[Severity: High]
Are victim buffer objects misclassified as defragmentation targets during multihop bounce moves?

When a victim BO's eviction requires a multihop bounce, ttm_bo_evict() calls ttm_bo_bounce_temp_buffer(), which subsequently calls ttm_bo_handle_move_mem() with evict = false.

If ctx->defrag is true from the surrounding defragmentation pass, this condition will evaluate to true for the bounce move.

Could this unnecessarily reallocate the victim's backing, and subsequently trigger the WARN_ON_ONCE below if a bounce is requested?

[ ... ]

drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, · line 215

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_bo.c +9 -0
+	/*
+	 * A defrag move deferred its virtual unmap until the new backing was
+	 * successfully populated; tear the userspace mappings down now, before
+	 * the move swaps in the new pages, so the next fault re-establishes
+	 * them against the new backing.
+	 */
+	if (defrag)
+		ttm_bo_unmap_virtual(bo);
+
 	ret = bdev->funcs->move(bo, evict, ctx, mem, hop);

[Severity: Critical]
Does this result in data loss if the driver falls back to the generic copy routines?

If a driver opts into defragmentation via ctx->defrag = true, ttm_bo_handle_move_mem() allocates a new TT backing (bo->ttm) and stashes the old data in bo->defrag_old_tt before calling the driver's move callback.

If the driver falls back to the generic ttm_bo_move_memcpy() or uses a hardware copy that directly relies on bo->ttm:

drivers/gpu/drm/ttm/ttm_bo_util.c:ttm_bo_move_memcpy() {
    ...
    dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm);
    ...
    src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm);
    ...
}

Won't both the source and destination initialize using the newly allocated uninitialized bo->ttm, copying garbage onto itself, while the actual data in bo->defrag_old_tt is ignored and eventually freed?

[ ... ]

drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, · line 249

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_bo.c +3 -1
 out_err:
-	if (!old_use_tt)
+	if (bo->defrag_old_tt) {
+		/* Failed defrag move: restore the original backing. */
+		ttm_bo_tt_destroy(bo);

[Severity: Critical]
Can this synchronously free the backing memory while an asynchronous GPU copy is still executing?

During a defrag move, the driver's move callback may launch an asynchronous hardware copy to populate the new TT backing.

If a subsequent cleanup step fails (for example, ttm_bo_move_accel_cleanup() returning -ENOMEM), the failure falls through to this out_err block.

If the memory is synchronously freed here using ttm_bo_tt_destroy(), won't the GPU continue executing the copy into the now-freed system pages, causing memory corruption?

Quoted 8 lines
drivers/gpu/drm/ttm/ttm_bo.c +4 -0
+		bo->ttm = bo->defrag_old_tt;
+		bo->defrag_old_tt = NULL;
+	} else if (!old_use_tt) {
 		ttm_bo_tt_destroy(bo);
+	}
 
 	return ret;
 }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
#15 lore raw

Re: [PATCH v3 05/33] drm/ttm: Support defragmentation moves

In reply to Matthew Brost
Quoted 79 lines · Matthew Brost

Add infrastructure to let a driver re-back an already-resident BO with fresh pages (e.g. at a more beneficial order) by reusing the normal move machinery.

Add ctx->defrag, which makes ttm_bo_validate() skip the "current resource already compatible" short-circuit so a move is forced even when placement is unchanged. In ttm_bo_handle_move_mem(), when defragging a populated BO, stash the old tt, create and populate a fresh one (allocating new backing), and hand the old tt to the driver move callback via bo->defrag_old_tt for copying. On success the old tt is unpopulated and freed; if populate or the move fails, the freshly allocated backing is discharged and the BO is restored to its original tt and placement.

The old tt is handed off through bo->defrag_old_tt (reservation-lock protected) rather than the operation context. The pointer describes BO state for the duration of the move, so keeping it on the buffer object is better layering and lets ttm_bo_move_accel_cleanup() pick it up.

Because a defrag move commonly fails (beneficial-order pages are often unavailable) and then leaves the BO untouched on its original backing, defer ttm_bo_unmap_virtual() for the defrag case until the new backing has been populated and the move is about to commit. Ordinary moves still unmap up front. This avoids needlessly zapping and refaulting every userspace mapping on the common failure path where nothing moved.

Teach ttm_bo_move_accel_cleanup()'s ghost path about defrag: instead of tearing the move's source tt down synchronously, hand bo->defrag_old_tt to the ghost object so it is kept populated until the copy fence signals and then unbound and destroyed asynchronously, while the BO keeps its new tt. This lets a driver pipeline defrag teardown behind the GPU copy rather than blocking on it.

This is driver-opt-in and a no-op unless ctx->defrag is set.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---

v3:
- Reject defrag moves needing a multihop bounce: WARN, restore backing, -EINVAL (Sashiko)
---

drivers/gpu/drm/ttm/ttm_bo.c      | 100 +++++++++++++++++++++++++++---
drivers/gpu/drm/ttm/ttm_bo_util.c |  16 ++++-
drivers/gpu/drm/ttm/ttm_pool.c    |   5 +-
include/drm/ttm/ttm_bo.h          |  17 +++++
4 files changed, 127 insertions(+), 11 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +2 -1
index 3980f376e3ba..c1629b4c4f0f 100644
@@ -125,49 +125,133 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 {
 	struct ttm_device *bdev = bo->bdev;
 	bool old_use_tt, new_use_tt;
+	bool defrag;
 	int ret;
 
 	old_use_tt = !bo->resource || ttm_manager_type(bdev, bo->resource->mem_type)->use_tt;
 	new_use_tt = ttm_manager_type(bdev, mem->mem_type)->use_tt;
 
-	ttm_bo_unmap_virtual(bo);
+	/*

drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, · line 144

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_bo.c +10 -0
+	 * Only the BO that is the actual defrag target (moved in place via
+	 * ttm_bo_validate(), evict == false) is being defragmented. The same
+	 * ctx->defrag is threaded into evictions triggered while allocating
+	 * that target's new backing (ttm_bo_evict_alloc()), so guard against
+	 * an unrelated victim BO (evict == true) being misclassified as a
+	 * defrag move and needlessly reallocating its backing.
+	 */
+	defrag = ctx->defrag && !evict && new_use_tt && bo->ttm &&
+		 ttm_tt_is_populated(bo->ttm);
+

The sashiko reports a bug about '!evict' check which seems legitimate.

Quoted 64 lines
drivers/gpu/drm/ttm/ttm_bo.c +41 -4
+	/*
+	 * Tear down userspace mappings up front for ordinary moves. A defrag
+	 * move is likely to leave the BO in place (beneficial-order allocation
+	 * commonly fails), so defer the unmap until the new backing has been
+	 * populated and the move is about to commit. This avoids a needless
+	 * zap and refault of every userspace mapping on the common failure,
+	 * where the BO keeps its original, still-mapped backing.
+	 */
+	if (!defrag)
+		ttm_bo_unmap_virtual(bo);
 
 	/*
 	 * Create and bind a ttm if required.
 	 */
 
 	if (new_use_tt) {
+		struct ttm_tt *defrag_old_tt = NULL;
+
+		/*
+		 * For a defrag move, stash the current populated tt and force a
+		 * fresh one to be created and populated (at beneficial order).
+		 * The old tt is handed to the driver move callback to copy the
+		 * contents from, and freed once the move completes.
+		 */
+		if (defrag) {
+			defrag_old_tt = bo->ttm;
+			bo->ttm = NULL;
+		}
+
 		/* Zero init the new TTM structure if the old location should
 		 * have used one as well.
 		 */
-		ret = ttm_tt_create(bo, old_use_tt);
-		if (ret)
+		ret = ttm_tt_create(bo, old_use_tt && !defrag_old_tt);
+		if (ret) {
+			if (defrag_old_tt)
+				bo->ttm = defrag_old_tt;
 			goto out_err;
+		}
 
-		if (mem->mem_type != TTM_PL_SYSTEM) {
+		if (defrag_old_tt || mem->mem_type != TTM_PL_SYSTEM) {
 			ret = ttm_bo_populate(bo, ctx);
-			if (ret)
+			if (ret) {
+				/*
+				 * Discharge the freshly allocated backing and
+				 * leave the BO on its original tt/placement.
+				 */
+				if (defrag_old_tt) {
+					ttm_tt_destroy(bo->bdev, bo->ttm);
+					bo->ttm = defrag_old_tt;
+				}
 				goto out_err;
+			}
 		}
+
+		bo->defrag_old_tt = defrag_old_tt;
 	}
 
 	ret = dma_resv_reserve_fences(bo->base.resv, 1);
 	if (ret)
 		goto out_err;

drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, · line 214

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_bo.c +9 -0
 
+	/*
+	 * A defrag move deferred its virtual unmap until the new backing was
+	 * successfully populated; tear the userspace mappings down now, before
+	 * the move swaps in the new pages, so the next fault re-establishes
+	 * them against the new backing.
+	 */
+	if (defrag)
+		ttm_bo_unmap_virtual(bo);
+

NIT...would it be good to always do virtual unmap here defrag or not?

drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, · line 215

Quoted 1 line
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
 	ret = bdev->funcs->move(bo, evict, ctx, mem, hop);

Sashiko reports a bug here when ttm_bo_move_memcpy() is used as move() function. Though xe doesn't use that function, it is perhaps good to guard against.

Quoted 32 lines
drivers/gpu/drm/ttm/ttm_bo.c +22 -1
 	if (ret) {
-		if (ret == -EMULTIHOP)
+		if (ret == -EMULTIHOP) {
+			/*
+			 * A defrag move targets the BO's current placement and
+			 * must complete in a single hop. A driver requesting a
+			 * bounce here is unsupported: restore the original
+			 * backing and fail the defrag rather than retrying the
+			 * multihop with ctx->defrag still set, which would
+			 * misclassify the bounce as another defrag move and
+			 * overwrite/leak bo->defrag_old_tt.
+			 */
+			if (unlikely(defrag)) {
+				WARN_ON_ONCE(1);
+				ret = -EINVAL;
+				goto out_err;
+			}
 			return ret;
+		}
 		goto out_err;
 	}
 
+	if (bo->defrag_old_tt) {
+		ttm_tt_unpopulate(bo->bdev, bo->defrag_old_tt);
+		ttm_tt_destroy(bo->bdev, bo->defrag_old_tt);
+		bo->defrag_old_tt = NULL;
+	}
+
 	ctx->bytes_moved += bo->base.size;
 	return 0;
 

out_err:

drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, · line 252

Quoted 7 lines
drivers/gpu/drm/ttm/ttm_bo.c +6 -1
-	if (!old_use_tt)
+	if (bo->defrag_old_tt) {
+		/* Failed defrag move: restore the original backing. */
+		ttm_bo_tt_destroy(bo);
+		bo->ttm = bo->defrag_old_tt;
+		bo->defrag_old_tt = NULL;
+	} else if (!old_use_tt) {

Another sashiko reported bug about tearing down the old pages while an async copy is still in progress. Looks like xe driver waits for move to complete before returning, so should be fine. But probably worth adding a documentation for defrag_old_tt that "a defrag-aware move callback must not return an error while a copy into the new backing is still in flight

Niranjana

Quoted 48 lines
		ttm_bo_tt_destroy(bo);
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
+	}
	return ret;
}
drivers/gpu/drm/ttm/ttm_bo.c +1 -1
@@ -835,7 +919,7 @@ int ttm_bo_validate(struct ttm_buffer_object *bo,
 	force_space = false;
 	do {
 		/* Check whether we need to move buffer. */
-		if (bo->resource &&
+		if (bo->resource && !ctx->defrag &&
 		    ttm_resource_compatible(bo->resource, placement,
 					    force_space))
 			return 0;
drivers/gpu/drm/ttm/ttm_bo_util.c +14 -2
index 3e3c201a0222..771313162c8d 100644
@@ -626,10 +626,22 @@ static int ttm_bo_move_to_ghost(struct ttm_buffer_object *bo,
 	 * bo to be unbound and destroyed.
 	 */
 
-	if (dst_use_tt)
+	if (bo->defrag_old_tt) {
+		/*
+		 * Defrag move: @bo already points at the freshly allocated,
+		 * beneficial-order tt that the move copied into. Hand the old
+		 * tt (the move's source) to the ghost so it is kept populated
+		 * until @fence signals and then unbound and destroyed, instead
+		 * of being torn down synchronously. @bo keeps its new tt.
+		 */
+		ghost_obj->ttm = bo->defrag_old_tt;
+		ghost_obj->defrag_old_tt = NULL;
+		bo->defrag_old_tt = NULL;
+	} else if (dst_use_tt) {
 		ghost_obj->ttm = NULL;
-	else
+	} else {
 		bo->ttm = NULL;
+	}
 
 	dma_resv_unlock(&ghost_obj->base._resv);
 	ttm_bo_put(ghost_obj);
drivers/gpu/drm/ttm/ttm_pool.c +0 -0
index 0ca63f5ce73c..d045105afc24 100644

drivers/gpu/drm/ttm/ttm_pool.c static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it) · line 867

Quoted 9 lines
drivers/gpu/drm/ttm/ttm_pool.c +4 -1
@@ -860,8 +860,11 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	 * pages. Record it so the driver can later try to defragment the object
 	 * back to beneficial order.
 	 */
-	if (it->beneficial_order && it->order == it->beneficial_order)
+	if (it->beneficial_order && it->order == it->beneficial_order) {
 		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+		if (it->ctx->defrag)
+			return -ENOMEM;
+	}

NIT...probably good to add a comment here that says something like, 'return error here as the purpose of defragmentation is to get beneficial order'.

Quoted 41 lines
drivers/gpu/drm/ttm/ttm_pool.c +0 -0
 	it->order--;
 	it->page_caching = it->tt->caching;
include/drm/ttm/ttm_bo.h +17 -0
index fd814b5cf535..f4f8a8c29f04 100644
@@ -135,6 +135,16 @@ struct ttm_buffer_object {
 	 * reservation lock.
 	 */
 	struct sg_table *sg;
+
+	/**
+	 * @defrag_old_tt: Used internally during a defrag move to hand the BO's
+	 * previous, populated struct ttm_tt to the driver's move callback so
+	 * that it can copy the contents into the freshly allocated
+	 * beneficial-order backing. Set by TTM, consumed by the driver; the old
+	 * tt is freed (or restored on failure) by TTM once the move completes.
+	 * Protected by the reservation lock.
+	 */
+	struct ttm_tt *defrag_old_tt;
 };
 
 #define TTM_BO_MAP_IOMEM_MASK 0x80
@@ -194,6 +204,13 @@ struct ttm_operation_ctx {
 	 * reclaim/compaction before falling back to a smaller order anyway.
 	 */
 	bool beneficial_reclaim_backoff;
+	/**
+	 * @defrag: Defragmentation pass. Attempt a move even when the bo's
+	 * current resource is already compatible with the requested placement,
+	 * so that the backing store can be reallocated at the pool's beneficial
+	 * order.
+	 */
+	bool defrag;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
Signature · 1 line
2.34.1
author maintainer
#16 lore raw

Re: [PATCH v3 05/33] drm/ttm: Support defragmentation moves

Quoted 78 lines · Niranjana Vishwanathapura
Quoted 77 lines · Matthew Brost

Add infrastructure to let a driver re-back an already-resident BO with fresh pages (e.g. at a more beneficial order) by reusing the normal move machinery.

Add ctx->defrag, which makes ttm_bo_validate() skip the "current resource already compatible" short-circuit so a move is forced even when placement is unchanged. In ttm_bo_handle_move_mem(), when defragging a populated BO, stash the old tt, create and populate a fresh one (allocating new backing), and hand the old tt to the driver move callback via bo->defrag_old_tt for copying. On success the old tt is unpopulated and freed; if populate or the move fails, the freshly allocated backing is discharged and the BO is restored to its original tt and placement.

The old tt is handed off through bo->defrag_old_tt (reservation-lock protected) rather than the operation context. The pointer describes BO state for the duration of the move, so keeping it on the buffer object is better layering and lets ttm_bo_move_accel_cleanup() pick it up.

Because a defrag move commonly fails (beneficial-order pages are often unavailable) and then leaves the BO untouched on its original backing, defer ttm_bo_unmap_virtual() for the defrag case until the new backing has been populated and the move is about to commit. Ordinary moves still unmap up front. This avoids needlessly zapping and refaulting every userspace mapping on the common failure path where nothing moved.

Teach ttm_bo_move_accel_cleanup()'s ghost path about defrag: instead of tearing the move's source tt down synchronously, hand bo->defrag_old_tt to the ghost object so it is kept populated until the copy fence signals and then unbound and destroyed asynchronously, while the BO keeps its new tt. This lets a driver pipeline defrag teardown behind the GPU copy rather than blocking on it.

This is driver-opt-in and a no-op unless ctx->defrag is set.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---

v3:
- Reject defrag moves needing a multihop bounce: WARN, restore backing, -EINVAL (Sashiko)
---

drivers/gpu/drm/ttm/ttm_bo.c      | 100 +++++++++++++++++++++++++++---
drivers/gpu/drm/ttm/ttm_bo_util.c |  16 ++++-
drivers/gpu/drm/ttm/ttm_pool.c    |   5 +-
include/drm/ttm/ttm_bo.h          |  17 +++++
4 files changed, 127 insertions(+), 11 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
index 3980f376e3ba..c1629b4c4f0f 100644
@@ -125,49 +125,133 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 {
 	struct ttm_device *bdev = bo->bdev;
 	bool old_use_tt, new_use_tt;
+	bool defrag;
 	int ret;
 
 	old_use_tt = !bo->resource || ttm_manager_type(bdev, bo->resource->mem_type)->use_tt;
 	new_use_tt = ttm_manager_type(bdev, mem->mem_type)->use_tt;
 
Quoted 15 lines

drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, · line 144

Quoted 12 lines
drivers/gpu/drm/ttm/ttm_bo.c +11 -1
-	ttm_bo_unmap_virtual(bo);
+	/*
+	 * Only the BO that is the actual defrag target (moved in place via
+	 * ttm_bo_validate(), evict == false) is being defragmented. The same
+	 * ctx->defrag is threaded into evictions triggered while allocating
+	 * that target's new backing (ttm_bo_evict_alloc()), so guard against
+	 * an unrelated victim BO (evict == true) being misclassified as a
+	 * defrag move and needlessly reallocating its backing.
+	 */
+	defrag = ctx->defrag && !evict && new_use_tt && bo->ttm &&
+		 ttm_tt_is_populated(bo->ttm);
+

The sashiko reports a bug about '!evict' check which seems legitimate.

See below - we don't allow multi-hop if ctx->defrag == true - i.e., we pop a warning if driver tries to do this. Thus the Sashiko feedback isn't right.

Quoted 65 lines
Quoted 65 lines
drivers/gpu/drm/ttm/ttm_bo.c +41 -4
+	/*
+	 * Tear down userspace mappings up front for ordinary moves. A defrag
+	 * move is likely to leave the BO in place (beneficial-order allocation
+	 * commonly fails), so defer the unmap until the new backing has been
+	 * populated and the move is about to commit. This avoids a needless
+	 * zap and refault of every userspace mapping on the common failure,
+	 * where the BO keeps its original, still-mapped backing.
+	 */
+	if (!defrag)
+		ttm_bo_unmap_virtual(bo);
 
 	/*
 	 * Create and bind a ttm if required.
 	 */
 
 	if (new_use_tt) {
+		struct ttm_tt *defrag_old_tt = NULL;
+
+		/*
+		 * For a defrag move, stash the current populated tt and force a
+		 * fresh one to be created and populated (at beneficial order).
+		 * The old tt is handed to the driver move callback to copy the
+		 * contents from, and freed once the move completes.
+		 */
+		if (defrag) {
+			defrag_old_tt = bo->ttm;
+			bo->ttm = NULL;
+		}
+
 		/* Zero init the new TTM structure if the old location should
 		 * have used one as well.
 		 */
-		ret = ttm_tt_create(bo, old_use_tt);
-		if (ret)
+		ret = ttm_tt_create(bo, old_use_tt && !defrag_old_tt);
+		if (ret) {
+			if (defrag_old_tt)
+				bo->ttm = defrag_old_tt;
 			goto out_err;
+		}
 
-		if (mem->mem_type != TTM_PL_SYSTEM) {
+		if (defrag_old_tt || mem->mem_type != TTM_PL_SYSTEM) {
 			ret = ttm_bo_populate(bo, ctx);
-			if (ret)
+			if (ret) {
+				/*
+				 * Discharge the freshly allocated backing and
+				 * leave the BO on its original tt/placement.
+				 */
+				if (defrag_old_tt) {
+					ttm_tt_destroy(bo->bdev, bo->ttm);
+					bo->ttm = defrag_old_tt;
+				}
 				goto out_err;
+			}
 		}
+
+		bo->defrag_old_tt = defrag_old_tt;
 	}
 
 	ret = dma_resv_reserve_fences(bo->base.resv, 1);
 	if (ret)
 		goto out_err;
 
Quoted 12 lines

drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, · line 214

Quoted 9 lines
drivers/gpu/drm/ttm/ttm_bo.c +9 -0
+	/*
+	 * A defrag move deferred its virtual unmap until the new backing was
+	 * successfully populated; tear the userspace mappings down now, before
+	 * the move swaps in the new pages, so the next fault re-establishes
+	 * them against the new backing.
+	 */
+	if (defrag)
+		ttm_bo_unmap_virtual(bo);
+

NIT...would it be good to always do virtual unmap here defrag or not?

We could unconditionally move the ttm_bo_unmap_virtual here. I didn't do that to not mess non-defrag behavior but it looks harmless to unconditionally shuffle around.

Quoted 6 lines

drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, · line 215

Quoted 1 line
drivers/gpu/drm/ttm/ttm_bo.c +0 -0
 	ret = bdev->funcs->move(bo, evict, ctx, mem, hop);

Sashiko reports a bug here when ttm_bo_move_memcpy() is used as move() function. Though xe doesn't use that function, it is perhaps good to guard against.

What is suggestion to guard against this?

Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/ttm/ttm_bo.c +3 -1
 	if (ret) {
-		if (ret == -EMULTIHOP)
+		if (ret == -EMULTIHOP) {
+			/*
+			 * A defrag move targets the BO's current placement and
Quoted 8 lines

drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, · line 227

Quoted 8 lines
drivers/gpu/drm/ttm/ttm_bo.c +8 -0
+			 * must complete in a single hop. A driver requesting a
+			 * bounce here is unsupported: restore the original
+			 * backing and fail the defrag rather than retrying the
+			 * multihop with ctx->defrag still set, which would
+			 * misclassify the bounce as another defrag move and
+			 * overwrite/leak bo->defrag_old_tt.
+			 */
+			if (unlikely(defrag)) {

Here is what makes -EMULTIHOP (Sashiko evict issue) impossible.

Quoted 18 lines
Quoted 18 lines
drivers/gpu/drm/ttm/ttm_bo.c +11 -0
+				WARN_ON_ONCE(1);
+				ret = -EINVAL;
+				goto out_err;
+			}
 			return ret;
+		}
 		goto out_err;
 	}
 
+	if (bo->defrag_old_tt) {
+		ttm_tt_unpopulate(bo->bdev, bo->defrag_old_tt);
+		ttm_tt_destroy(bo->bdev, bo->defrag_old_tt);
+		bo->defrag_old_tt = NULL;
+	}
+
 	ctx->bytes_moved += bo->base.size;
 	return 0;
 
Quoted 15 lines

drivers/gpu/drm/ttm/ttm_bo.c static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo, · line 252

Quoted 8 lines

out_err:

drivers/gpu/drm/ttm/ttm_bo.c +6 -1
-	if (!old_use_tt)
+	if (bo->defrag_old_tt) {
+		/* Failed defrag move: restore the original backing. */
+		ttm_bo_tt_destroy(bo);
+		bo->ttm = bo->defrag_old_tt;
+		bo->defrag_old_tt = NULL;
+	} else if (!old_use_tt) {

Another sashiko reported bug about tearing down the old pages while an async copy is still in progress. Looks like xe driver waits for move to complete before returning, so should be fine. But probably worth adding a documentation for defrag_old_tt that "a defrag-aware move callback must not return an error while a copy into the new backing is still in flight

I think it is implied in the ->move() contract if one fails after triggering an async move, you must wait on async move before returning or TTM will explode. This appears to be existing behavior too.

Matt

Quoted 104 lines

Niranjana

Quoted 48 lines
		ttm_bo_tt_destroy(bo);
drivers/gpu/drm/ttm/ttm_bo.c +1 -0
+	}
	return ret;
}
drivers/gpu/drm/ttm/ttm_bo.c +1 -1
@@ -835,7 +919,7 @@ int ttm_bo_validate(struct ttm_buffer_object *bo,
 	force_space = false;
 	do {
 		/* Check whether we need to move buffer. */
-		if (bo->resource &&
+		if (bo->resource && !ctx->defrag &&
 		    ttm_resource_compatible(bo->resource, placement,
 					    force_space))
 			return 0;
drivers/gpu/drm/ttm/ttm_bo_util.c +14 -2
index 3e3c201a0222..771313162c8d 100644
@@ -626,10 +626,22 @@ static int ttm_bo_move_to_ghost(struct ttm_buffer_object *bo,
 	 * bo to be unbound and destroyed.
 	 */
 
-	if (dst_use_tt)
+	if (bo->defrag_old_tt) {
+		/*
+		 * Defrag move: @bo already points at the freshly allocated,
+		 * beneficial-order tt that the move copied into. Hand the old
+		 * tt (the move's source) to the ghost so it is kept populated
+		 * until @fence signals and then unbound and destroyed, instead
+		 * of being torn down synchronously. @bo keeps its new tt.
+		 */
+		ghost_obj->ttm = bo->defrag_old_tt;
+		ghost_obj->defrag_old_tt = NULL;
+		bo->defrag_old_tt = NULL;
+	} else if (dst_use_tt) {
 		ghost_obj->ttm = NULL;
-	else
+	} else {
 		bo->ttm = NULL;
+	}
 
 	dma_resv_unlock(&ghost_obj->base._resv);
 	ttm_bo_put(ghost_obj);
drivers/gpu/drm/ttm/ttm_pool.c +0 -0
index 0ca63f5ce73c..d045105afc24 100644

drivers/gpu/drm/ttm/ttm_pool.c static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it) · line 867

Quoted 9 lines
drivers/gpu/drm/ttm/ttm_pool.c +4 -1
@@ -860,8 +860,11 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	 * pages. Record it so the driver can later try to defragment the object
 	 * back to beneficial order.
 	 */
-	if (it->beneficial_order && it->order == it->beneficial_order)
+	if (it->beneficial_order && it->order == it->beneficial_order) {
 		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+		if (it->ctx->defrag)
+			return -ENOMEM;
+	}

NIT...probably good to add a comment here that says something like, 'return error here as the purpose of defragmentation is to get beneficial order'.

Quoted 41 lines
drivers/gpu/drm/ttm/ttm_pool.c +0 -0
 	it->order--;
 	it->page_caching = it->tt->caching;
include/drm/ttm/ttm_bo.h +17 -0
index fd814b5cf535..f4f8a8c29f04 100644
@@ -135,6 +135,16 @@ struct ttm_buffer_object {
 	 * reservation lock.
 	 */
 	struct sg_table *sg;
+
+	/**
+	 * @defrag_old_tt: Used internally during a defrag move to hand the BO's
+	 * previous, populated struct ttm_tt to the driver's move callback so
+	 * that it can copy the contents into the freshly allocated
+	 * beneficial-order backing. Set by TTM, consumed by the driver; the old
+	 * tt is freed (or restored on failure) by TTM once the move completes.
+	 * Protected by the reservation lock.
+	 */
+	struct ttm_tt *defrag_old_tt;
 };
 
 #define TTM_BO_MAP_IOMEM_MASK 0x80
@@ -194,6 +204,13 @@ struct ttm_operation_ctx {
 	 * reclaim/compaction before falling back to a smaller order anyway.
 	 */
 	bool beneficial_reclaim_backoff;
+	/**
+	 * @defrag: Defragmentation pass. Attempt a move even when the bo's
+	 * current resource is already compatible with the requested placement,
+	 * so that the backing store can be reallocated at the pool's beneficial
+	 * order.
+	 */
+	bool defrag;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
Signature · 1 line
2.34.1
author maintainer
#17 lore raw

PATCH v3 6/33 drm/ttm: Add fault injection for beneficial-order allocation failures

In reply to Matthew Brost

Add a beneficial_order_fault_inject debugfs fault attribute, mirroring backup_fault_inject, that forces allocation at (or above) the device's beneficial order to fail in __ttm_pool_alloc(). When triggered, both the pool take and the system page allocation are skipped for that order, so the allocation falls into the existing lower-the-order path that sets the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tt page flag (and bails out with -ENOMEM for defrag moves).

This lets the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tracking and the driver-side defrag path be exercised deterministically, without having to drive the system into real memory fragmentation to provoke a sub-optimal backing.

The knob is registered at /sys/kernel/debug/ttm/beneficial_order_fault_inject and compiles out when CONFIG_FAULT_INJECTION is disabled.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 35 ++++++++++++++++++++++++++++------
 1 file changed, 29 insertions(+), 6 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +29 -6
index d045105afc24..2598d2f33767 100644
@@ -53,6 +53,7 @@
 #ifdef CONFIG_FAULT_INJECTION
 #include <linux/fault-inject.h>
 static DECLARE_FAULT_ATTR(backup_fault_inject);
+static DECLARE_FAULT_ATTR(beneficial_order_fault_inject);
 #else
 #define should_fail(...) false
 #endif
@@ -812,21 +813,34 @@ struct ttm_pool_alloc_iter {
 	unsigned int order;
 	enum ttm_caching page_caching;
 	bool allow_pools;
+	bool fail_beneficial;
 	struct page *p;
 };
 
 /*
  * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries a same-order pool page, then a fresh system allocation.
+ * failure). Tries a same-order pool page, then a fresh system allocation. Fault
+ * injection can force the beneficial-order paths to "fail".
  */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
 	struct ttm_pool_type *pt;
 
-	/* First, try to allocate a page from a pool if one exists. */
 	it->p = NULL;
+
+	/*
+	 * Fault injection: pretend allocation at (or above) the device's
+	 * beneficial order failed, forcing a sub-optimal backing. Exercises the
+	 * beneficial_order_failed tracking and the driver defrag path without
+	 * driving the system into real fragmentation.
+	 */
+	it->fail_beneficial = IS_ENABLED(CONFIG_FAULT_INJECTION) &&
+		it->beneficial_order && it->order >= it->beneficial_order &&
+		should_fail(&beneficial_order_fault_inject, 1);
+
+	/* First, try to allocate a page from a pool if one exists. */
 	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
-	if (!it->p && pt && it->allow_pools)
+	if (!it->p && pt && it->allow_pools && !it->fail_beneficial)
 		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
 
 	/*
@@ -834,7 +848,7 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 	 * this also disallows additional pool allocations using write-back
 	 * cached pools of the same order.
 	 */
-	if (!it->p) {
+	if (!it->p && !it->fail_beneficial) {
 		it->page_caching = ttm_cached;
 		it->allow_pools = false;
 		it->p = ttm_pool_alloc_page(it->pool, it->gfp_flags, it->order,
@@ -851,6 +865,9 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
  */
 static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 {
+	bool at_beneficial = it->beneficial_order &&
+			     it->order == it->beneficial_order;
+
 	if (!it->order)
 		return -ENOMEM;
 
@@ -860,13 +877,16 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	 * pages. Record it so the driver can later try to defragment the object
 	 * back to beneficial order.
 	 */
-	if (it->beneficial_order && it->order == it->beneficial_order) {
+	if (it->fail_beneficial || at_beneficial) {
 		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 		if (it->ctx->defrag)
 			return -ENOMEM;
 	}
 
-	it->order--;
+	if (it->fail_beneficial)
+		it->order = it->beneficial_order - 1;
+	else
+		it->order--;
 	it->page_caching = it->tt->caching;
 	it->allow_pools = true;
 
@@ -1536,6 +1556,9 @@ int ttm_pool_mgr_init(unsigned long num_pages)
 #ifdef CONFIG_FAULT_INJECTION
 	fault_create_debugfs_attr("backup_fault_inject", ttm_debugfs_root,
 				  &backup_fault_inject);
+	fault_create_debugfs_attr("beneficial_order_fault_inject",
+				  ttm_debugfs_root,
+				  &beneficial_order_fault_inject);
 #endif
 #endif
 
Signature · 1 line
2.34.1
#18 lore raw

Re: [PATCH v3 06/33] drm/ttm: Add fault injection for beneficial-order allocation failures

Quoted 68 lines · Matthew Brost

Add a beneficial_order_fault_inject debugfs fault attribute, mirroring backup_fault_inject, that forces allocation at (or above) the device's beneficial order to fail in __ttm_pool_alloc(). When triggered, both the pool take and the system page allocation are skipped for that order, so the allocation falls into the existing lower-the-order path that sets the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tt page flag (and bails out with -ENOMEM for defrag moves).

This lets the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tracking and the driver-side defrag path be exercised deterministically, without having to drive the system into real memory fragmentation to provoke a sub-optimal backing.

The knob is registered at /sys/kernel/debug/ttm/beneficial_order_fault_inject and compiles out when CONFIG_FAULT_INJECTION is disabled.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---
drivers/gpu/drm/ttm/ttm_pool.c | 35 ++++++++++++++++++++++++++++------

1 file changed, 29 insertions(+), 6 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +4 -2
index d045105afc24..2598d2f33767 100644
@@ -53,6 +53,7 @@
 #ifdef CONFIG_FAULT_INJECTION
 #include <linux/fault-inject.h>
 static DECLARE_FAULT_ATTR(backup_fault_inject);
+static DECLARE_FAULT_ATTR(beneficial_order_fault_inject);
 #else
 #define should_fail(...) false
 #endif
@@ -812,21 +813,34 @@ struct ttm_pool_alloc_iter {
 	unsigned int order;
 	enum ttm_caching page_caching;
 	bool allow_pools;
+	bool fail_beneficial;
 	struct page *p;
 };
 
 /*
 * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries a same-order pool page, then a fresh system allocation.
+ * failure). Tries a same-order pool page, then a fresh system allocation. Fault
+ * injection can force the beneficial-order paths to "fail".
 */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
 	struct ttm_pool_type *pt;
 
-	/* First, try to allocate a page from a pool if one exists. */
 	it->p = NULL;

drivers/gpu/drm/ttm/ttm_pool.c struct ttm_pool_alloc_iter { · line 839

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+
+	/*
+	 * Fault injection: pretend allocation at (or above) the device's
+	 * beneficial order failed, forcing a sub-optimal backing. Exercises the
+	 * beneficial_order_failed tracking and the driver defrag path without
+	 * driving the system into real fragmentation.
+	 */
+	it->fail_beneficial = IS_ENABLED(CONFIG_FAULT_INJECTION) &&
+		it->beneficial_order && it->order >= it->beneficial_order &&
+		should_fail(&beneficial_order_fault_inject, 1);

NIT..Looks like IS_ENABLED(CONFIG_FAULT_INJECTION) check here is redundant, but other fault inject case is also using it, so, should be fine I guess.

This patch LGTM.

Niranjana

Quoted 59 lines
drivers/gpu/drm/ttm/ttm_pool.c +15 -4
+
+	/* First, try to allocate a page from a pool if one exists. */
 	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
-	if (!it->p && pt && it->allow_pools)
+	if (!it->p && pt && it->allow_pools && !it->fail_beneficial)
 		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
 
 	/*
@@ -834,7 +848,7 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 	 * this also disallows additional pool allocations using write-back
 	 * cached pools of the same order.
 	 */
-	if (!it->p) {
+	if (!it->p && !it->fail_beneficial) {
 		it->page_caching = ttm_cached;
 		it->allow_pools = false;
 		it->p = ttm_pool_alloc_page(it->pool, it->gfp_flags, it->order,
@@ -851,6 +865,9 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 */
 static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 {
+	bool at_beneficial = it->beneficial_order &&
+			     it->order == it->beneficial_order;
+
 	if (!it->order)
 		return -ENOMEM;
 
@@ -860,13 +877,16 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	 * pages. Record it so the driver can later try to defragment the object
 	 * back to beneficial order.
 	 */
-	if (it->beneficial_order && it->order == it->beneficial_order) {
+	if (it->fail_beneficial || at_beneficial) {
 		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 		if (it->ctx->defrag)
 			return -ENOMEM;
 	}
 
-	it->order--;
+	if (it->fail_beneficial)
+		it->order = it->beneficial_order - 1;
+	else
+		it->order--;
 	it->page_caching = it->tt->caching;
 	it->allow_pools = true;
 
@@ -1536,6 +1556,9 @@ int ttm_pool_mgr_init(unsigned long num_pages)
 #ifdef CONFIG_FAULT_INJECTION
 	fault_create_debugfs_attr("backup_fault_inject", ttm_debugfs_root,
 				  &backup_fault_inject);
+	fault_create_debugfs_attr("beneficial_order_fault_inject",
+				  ttm_debugfs_root,
+				  &beneficial_order_fault_inject);
 #endif
 #endif
 
Signature · 1 line
2.34.1
author maintainer
#19 lore raw

Re: [PATCH v3 06/33] drm/ttm: Add fault injection for beneficial-order allocation failures

Quoted 67 lines · Niranjana Vishwanathapura
Quoted 66 lines · Matthew Brost

Add a beneficial_order_fault_inject debugfs fault attribute, mirroring backup_fault_inject, that forces allocation at (or above) the device's beneficial order to fail in __ttm_pool_alloc(). When triggered, both the pool take and the system page allocation are skipped for that order, so the allocation falls into the existing lower-the-order path that sets the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tt page flag (and bails out with -ENOMEM for defrag moves).

This lets the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tracking and the driver-side defrag path be exercised deterministically, without having to drive the system into real memory fragmentation to provoke a sub-optimal backing.

The knob is registered at /sys/kernel/debug/ttm/beneficial_order_fault_inject and compiles out when CONFIG_FAULT_INJECTION is disabled.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---
drivers/gpu/drm/ttm/ttm_pool.c | 35 ++++++++++++++++++++++++++++------

1 file changed, 29 insertions(+), 6 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +4 -1
index d045105afc24..2598d2f33767 100644
@@ -53,6 +53,7 @@
 #ifdef CONFIG_FAULT_INJECTION
 #include <linux/fault-inject.h>
 static DECLARE_FAULT_ATTR(backup_fault_inject);
+static DECLARE_FAULT_ATTR(beneficial_order_fault_inject);
 #else
 #define should_fail(...) false
 #endif
@@ -812,21 +813,34 @@ struct ttm_pool_alloc_iter {
 	unsigned int order;
 	enum ttm_caching page_caching;
 	bool allow_pools;
+	bool fail_beneficial;
 	struct page *p;
 };
 
 /*
 * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries a same-order pool page, then a fresh system allocation.
+ * failure). Tries a same-order pool page, then a fresh system allocation. Fault
+ * injection can force the beneficial-order paths to "fail".
 */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
 	struct ttm_pool_type *pt;
 
Quoted 16 lines

drivers/gpu/drm/ttm/ttm_pool.c struct ttm_pool_alloc_iter { · line 839

Quoted 12 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -1
-	/* First, try to allocate a page from a pool if one exists. */
 	it->p = NULL;
+
+	/*
+	 * Fault injection: pretend allocation at (or above) the device's
+	 * beneficial order failed, forcing a sub-optimal backing. Exercises the
+	 * beneficial_order_failed tracking and the driver defrag path without
+	 * driving the system into real fragmentation.
+	 */
+	it->fail_beneficial = IS_ENABLED(CONFIG_FAULT_INJECTION) &&
+		it->beneficial_order && it->order >= it->beneficial_order &&
+		should_fail(&beneficial_order_fault_inject, 1);

NIT..Looks like IS_ENABLED(CONFIG_FAULT_INJECTION) check here is redundant, but other fault inject case is also using it, so, should be fine I guess.

I believe the thinking is immediate short circuit on a static Kconfig so the compiler can optimize all of this out. Maybe it should_fail is enough but following the style that exists in this file for other injection points.

Matt

Quoted 63 lines

This patch LGTM.

Niranjana

Quoted 59 lines
drivers/gpu/drm/ttm/ttm_pool.c +15 -4
+
+	/* First, try to allocate a page from a pool if one exists. */
 	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
-	if (!it->p && pt && it->allow_pools)
+	if (!it->p && pt && it->allow_pools && !it->fail_beneficial)
 		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
 
 	/*
@@ -834,7 +848,7 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 	 * this also disallows additional pool allocations using write-back
 	 * cached pools of the same order.
 	 */
-	if (!it->p) {
+	if (!it->p && !it->fail_beneficial) {
 		it->page_caching = ttm_cached;
 		it->allow_pools = false;
 		it->p = ttm_pool_alloc_page(it->pool, it->gfp_flags, it->order,
@@ -851,6 +865,9 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 */
 static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 {
+	bool at_beneficial = it->beneficial_order &&
+			     it->order == it->beneficial_order;
+
 	if (!it->order)
 		return -ENOMEM;
 
@@ -860,13 +877,16 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	 * pages. Record it so the driver can later try to defragment the object
 	 * back to beneficial order.
 	 */
-	if (it->beneficial_order && it->order == it->beneficial_order) {
+	if (it->fail_beneficial || at_beneficial) {
 		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 		if (it->ctx->defrag)
 			return -ENOMEM;
 	}
 
-	it->order--;
+	if (it->fail_beneficial)
+		it->order = it->beneficial_order - 1;
+	else
+		it->order--;
 	it->page_caching = it->tt->caching;
 	it->allow_pools = true;
 
@@ -1536,6 +1556,9 @@ int ttm_pool_mgr_init(unsigned long num_pages)
 #ifdef CONFIG_FAULT_INJECTION
 	fault_create_debugfs_attr("backup_fault_inject", ttm_debugfs_root,
 				  &backup_fault_inject);
+	fault_create_debugfs_attr("beneficial_order_fault_inject",
+				  ttm_debugfs_root,
+				  &beneficial_order_fault_inject);
 #endif
 #endif
 
Signature · 1 line
2.34.1
author maintainer
#20 lore raw

PATCH v3 7/33 drm/ttm: Harvest beneficial-order pages on defragmentation moves

In reply to Matthew Brost

A defragmentation move relocates a populated tt into a freshly allocated one so that sub-beneficial-order chunks can be upgraded to the device's beneficial order. Previously, if allocation could not complete entirely at beneficial order, the move failed and all progress was discarded.

Instead, once allocation has successfully upgraded part of the object, the remainder of the old tt is harvested to complete the new tt. Thread the BO's old, still-populated tt into the page allocator via a new ttm_operation_ctx::defrag_old_tt. While allocation is still proceeding, any existing chunk already at (or above) the beneficial order is reused directly instead of being reallocated. If allocation later has to fall back below beneficial_order after making partial progress, the remainder of the old tt is harvested instead of failing the move. This preserves the beneficial-order upgrades already made while leaving the rest of the object unchanged, allowing subsequent defragmentation attempts to make additional forward progress.

Harvesting only borrows the pages; they remain owned by the old tt until the move commits. This keeps every unwind path non-destructive: if the (re)allocation or the move later fails, the new tt is torn down without freeing the borrowed pages and the old tt is restored fully intact. Ownership of the shared pages is transferred to the new tt only once the move has committed. ttm_tt_defrag_disown_borrowed() implements both directions of the transfer by clearing a tt's slots for every page it shares with the other tt (same struct page at the same index), so that tt's teardown leaves the shared pages alone and each page is freed exactly once by its owner:

- on the alloc unwind and the failed-move paths, the new tt disowns the borrowed pages, leaving the old tt as the sole owner;
- on the successful-move paths (synchronous teardown and the pipelined ghost-object teardown), the old tt disowns the now-transferred pages, leaving the new tt as the sole owner.

Detecting shared pages by same-index struct page identity needs no extra per-tt bookkeeping, since two distinct allocations never alias. This is safe with drivers, such as Xe, that build their own per-tt sg table and DMA-map it independently: each tt maps and unmaps its own sg, so sharing the underlying pages between tts is DMA-clean.

This is a no-op unless ctx->defrag_old_tt is set, which only happens on a defragmentation move.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---

v3:
- Document the chunk-head invariant read by
ttm_pool_harvest_remaining() (Sashiko)

---
 drivers/gpu/drm/ttm/ttm_bo.c      |  20 ++-
 drivers/gpu/drm/ttm/ttm_bo_util.c |   5 +
 drivers/gpu/drm/ttm/ttm_pool.c    | 235 ++++++++++++++++++++++++++++--
 drivers/gpu/drm/ttm/ttm_tt.c      |  67 +++++++++
 include/drm/ttm/ttm_bo.h          |  12 ++
 include/drm/ttm/ttm_tt.h          |   2 +
 6 files changed, 328 insertions(+), 13 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +19 -1
index c1629b4c4f0f..ab2e637a017f 100644
@@ -182,7 +182,14 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 		}
 
 		if (defrag_old_tt || mem->mem_type != TTM_PL_SYSTEM) {
+			/*
+			 * Let the page allocator harvest already
+			 * beneficial-order chunks out of the old tt to back
+			 * the new one, rather than reallocating them.
+			 */
+			ctx->defrag_old_tt = defrag_old_tt;
 			ret = ttm_bo_populate(bo, ctx);
+			ctx->defrag_old_tt = NULL;
 			if (ret) {
 				/*
 				 * Discharge the freshly allocated backing and
@@ -235,6 +242,12 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 	}
 
 	if (bo->defrag_old_tt) {
+		/*
+		 * Successful defrag move: any pages the new tt borrowed from the
+		 * old tt are now owned by the new tt. Disown them from the old
+		 * tt before it is torn down so they are not freed.
+		 */
+		ttm_tt_defrag_disown_borrowed(bo->defrag_old_tt, bo->ttm);
 		ttm_tt_unpopulate(bo->bdev, bo->defrag_old_tt);
 		ttm_tt_destroy(bo->bdev, bo->defrag_old_tt);
 		bo->defrag_old_tt = NULL;
@@ -245,7 +258,12 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 
 out_err:
 	if (bo->defrag_old_tt) {
-		/* Failed defrag move: restore the original backing. */
+		/*
+		 * Failed defrag move: drop any pages the new tt borrowed from
+		 * the old tt (still owned by the old tt) before destroying the
+		 * new tt, then restore the original backing.
+		 */
+		ttm_tt_defrag_disown_borrowed(bo->ttm, bo->defrag_old_tt);
 		ttm_bo_tt_destroy(bo);
 		bo->ttm = bo->defrag_old_tt;
 		bo->defrag_old_tt = NULL;
drivers/gpu/drm/ttm/ttm_bo_util.c +5 -0
index 771313162c8d..7cb718fc97c0 100644
@@ -633,7 +633,12 @@ static int ttm_bo_move_to_ghost(struct ttm_buffer_object *bo,
 		 * tt (the move's source) to the ghost so it is kept populated
 		 * until @fence signals and then unbound and destroyed, instead
 		 * of being torn down synchronously. @bo keeps its new tt.
+		 *
+		 * Any pages the new tt borrowed from the old tt are now owned by
+		 * the new tt; disown them from the old tt so the ghost teardown
+		 * leaves them alone.
 		 */
+		ttm_tt_defrag_disown_borrowed(bo->defrag_old_tt, bo->ttm);
 		ghost_obj->ttm = bo->defrag_old_tt;
 		ghost_obj->defrag_old_tt = NULL;
 		bo->defrag_old_tt = NULL;
drivers/gpu/drm/ttm/ttm_pool.c +223 -12

307 lines not on this page · raw message

drivers/gpu/drm/ttm/ttm_tt.c +67 -0

73 lines not on this page · raw message

include/drm/ttm/ttm_bo.h +12 -0
index f4f8a8c29f04..fcaba5fb8bf7 100644
@@ -211,6 +211,18 @@ struct ttm_operation_ctx {
 	 * order.
 	 */
 	bool defrag;
+	/**
+	 * @defrag_old_tt: During a defragmentation move, the bo's existing
+	 * (populated) tt that is being relocated. The page allocator may
+	 * harvest already beneficial-order chunks from this tt to back the new
+	 * tt instead of reallocating them, avoiding alloc/free churn for large
+	 * buffers. Harvested pages are only borrowed: they stay owned by
+	 * @defrag_old_tt until the move commits, at which point
+	 * ttm_tt_defrag_disown_borrowed() clears the old tt's slots for the
+	 * transferred pages so they are not freed when the old tt is torn down.
+	 * NULL for non-defrag operations.
+	 */
+	struct ttm_tt *defrag_old_tt;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
include/drm/ttm/ttm_tt.h +2 -0
index ce7533677d77..55c35bcf134d 100644
@@ -284,6 +284,8 @@ int ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm,
  */
 void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
 
+void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src);
+
 /**
  * ttm_tt_mark_for_clear - Mark pages for clearing on populate.
  *
Signature · 1 line
2.34.1
#21 lore raw

Re: [PATCH v3 07/33] drm/ttm: Harvest beneficial-order pages on defragmentation moves

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Critical] Defragmentation of DMA-alloc or external buffer objects causes out-of-bounds memory…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] Defragmentation of DMA-alloc or external buffer objects causes out-of-bounds memory corruption.
- [Low] Comment falsely documents non-existent capping and prealloc logic.
--

--- Patch [7]: [PATCH v3 07/33] drm/ttm: Harvest beneficial-order pages on defragmentation moves --- commit 6003b76a02bde0abed96778f39a94035b5bc6acd
Author: Matthew Brost <[email protected]>

drm/ttm: Harvest beneficial-order pages on defragmentation moves

A defragmentation move relocates a populated tt into a freshly allocated one so that sub-beneficial-order chunks can be upgraded to the device's beneficial order.

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_pool.c +0 -0
index 2598d2f33767e..52f5e64f05ddc 100644

[ ... ]

Quoted 9 lines
drivers/gpu/drm/ttm/ttm_pool.c +9 -0
+static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
+				      struct ttm_tt *old_tt, pgoff_t off,
+				      struct ttm_pool_alloc_state *alloc)
+{
+	/*
+	 * @off is always an old-tt chunk head here, never a tail page, so
+	 * reading the per-chunk order below is safe. This holds because:
+	 *  - Old chunks are self-aligned: greedy largest-power-of-two fill
+	 *    places an order-k chunk at a 2^k-aligned offset, so any

drivers/gpu/drm/ttm/ttm_pool.c static unsigned int ttm_pool_alloc_find_order(unsigned int highest, · line 904

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+	 *    beneficial-order-aligned offset is a chunk boundary.
+	 *  - Fresh defrag allocations are capped at the beneficial order (see
+	 *    ttm_pool_iter_acquire_page()), so they advance @off in beneficial
+	 *    steps and never overshoot into the middle of a larger old chunk;
+	 *    super-beneficial old chunks are always harvested whole by
+	 *    ttm_pool_iter_reuse_old().
+	 *  - We are only reached once the defrag budget/prealloc is exhausted,
+	 *    which in the capped/prealloc config leaves @off beneficial-aligned
+	 *    (or, in the sub-beneficial tail, already walking old boundaries).
+	 */

[Severity: Low]
Are the invariants described in this comment block referring to code that is missing from this series?

The documentation states that allocations are capped and references ttm_pool_iter_acquire_page(), but that function does not appear to contain any capping logic.

The comment also refers to a defrag budget/prealloc exhaustion, but those mechanisms do not seem to be present in the code.

Quoted 2 lines
drivers/gpu/drm/ttm/ttm_pool.c +2 -0
+	while (alloc->remaining_pages) {
+		struct page *p = old_tt->pages[off];

[ ... ]

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_tt.c +0 -0
index b645a18181843..2388ff794a594 100644

[ ... ]

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_tt.c +4 -0
+void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src)
+{
+	pgoff_t i = 0, run_start = 0;
+	bool in_run = false;

[ ... ]

drivers/gpu/drm/ttm/ttm_tt.c void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm) · line 491

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_tt.c +4 -0
+	while (i < tt->num_pages) {
+		struct page *sp = src->pages[i];
+		unsigned int order = sp ? ttm_pool_page_order_nodma(sp) : 0;
+		pgoff_t nr = 1UL << order;

[Severity: Critical]
Can this lead to a massive out-of-bounds memset and memcpy when defragmentation targets a dma-alloc pool or externally allocated buffer object?

If a defragmentation move is initiated in ttm_bo_handle_move_mem() on a buffer object using a dma-alloc pool or with TTM_TT_FLAG_EXTERNAL set, page private data often contains a kernel pointer or arbitrary foreign data instead of an allocation order.

Since ttm_pool_page_order_nodma() treats this data as an integer, the resulting order calculation would yield an astronomically large chunk size for nr. This size is then used in the memset operation immediately below, and in the memcpy operation within ttm_pool_harvest_page():

drivers/gpu/drm/ttm/ttm_pool.c:ttm_pool_harvest_page() {
    ...
    memcpy(alloc->pages, &old_tt->pages[off], nr * sizeof(*alloc->pages));
    ...
}

Does ttm_bo_handle_move_mem() need validation to block defragmentation on these configurations?

Quoted 9 lines
drivers/gpu/drm/ttm/ttm_tt.c +9 -0
+
+		if (sp && tt->pages[i] == sp) {
+			if (!in_run) {
+				run_start = i;
+				in_run = true;
+			}
+		} else if (in_run) {
+			memset(&tt->pages[run_start], 0,
+			       (i - run_start) * sizeof(*tt->pages));
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
#22 lore raw

Re: [PATCH v3 07/33] drm/ttm: Harvest beneficial-order pages on defragmentation moves

In reply to Matthew Brost
Quoted 169 lines · Matthew Brost

A defragmentation move relocates a populated tt into a freshly allocated one so that sub-beneficial-order chunks can be upgraded to the device's beneficial order. Previously, if allocation could not complete entirely at beneficial order, the move failed and all progress was discarded.

Instead, once allocation has successfully upgraded part of the object, the remainder of the old tt is harvested to complete the new tt. Thread the BO's old, still-populated tt into the page allocator via a new ttm_operation_ctx::defrag_old_tt. While allocation is still proceeding, any existing chunk already at (or above) the beneficial order is reused directly instead of being reallocated. If allocation later has to fall back below beneficial_order after making partial progress, the remainder of the old tt is harvested instead of failing the move. This preserves the beneficial-order upgrades already made while leaving the rest of the object unchanged, allowing subsequent defragmentation attempts to make additional forward progress.

Harvesting only borrows the pages; they remain owned by the old tt until the move commits. This keeps every unwind path non-destructive: if the (re)allocation or the move later fails, the new tt is torn down without freeing the borrowed pages and the old tt is restored fully intact. Ownership of the shared pages is transferred to the new tt only once the move has committed. ttm_tt_defrag_disown_borrowed() implements both directions of the transfer by clearing a tt's slots for every page it shares with the other tt (same struct page at the same index), so that tt's teardown leaves the shared pages alone and each page is freed exactly once by its owner:

- on the alloc unwind and the failed-move paths, the new tt disowns the borrowed pages, leaving the old tt as the sole owner;
- on the successful-move paths (synchronous teardown and the pipelined ghost-object teardown), the old tt disowns the now-transferred pages, leaving the new tt as the sole owner.

Detecting shared pages by same-index struct page identity needs no extra per-tt bookkeeping, since two distinct allocations never alias. This is safe with drivers, such as Xe, that build their own per-tt sg table and DMA-map it independently: each tt maps and unmaps its own sg, so sharing the underlying pages between tts is DMA-clean.

This is a no-op unless ctx->defrag_old_tt is set, which only happens on a defragmentation move.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---

v3:
- Document the chunk-head invariant read by
ttm_pool_harvest_remaining() (Sashiko)
---

drivers/gpu/drm/ttm/ttm_bo.c      |  20 ++-
drivers/gpu/drm/ttm/ttm_bo_util.c |   5 +
drivers/gpu/drm/ttm/ttm_pool.c    | 235 ++++++++++++++++++++++++++++--
drivers/gpu/drm/ttm/ttm_tt.c      |  67 +++++++++
include/drm/ttm/ttm_bo.h          |  12 ++
include/drm/ttm/ttm_tt.h          |   2 +
6 files changed, 328 insertions(+), 13 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +13 -0
index c1629b4c4f0f..ab2e637a017f 100644
@@ -182,7 +182,14 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 		}
 
 		if (defrag_old_tt || mem->mem_type != TTM_PL_SYSTEM) {
+			/*
+			 * Let the page allocator harvest already
+			 * beneficial-order chunks out of the old tt to back
+			 * the new one, rather than reallocating them.
+			 */
+			ctx->defrag_old_tt = defrag_old_tt;
 			ret = ttm_bo_populate(bo, ctx);
+			ctx->defrag_old_tt = NULL;
 			if (ret) {
 				/*
 				 * Discharge the freshly allocated backing and
@@ -235,6 +242,12 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 	}
 
 	if (bo->defrag_old_tt) {
+		/*
+		 * Successful defrag move: any pages the new tt borrowed from the
+		 * old tt are now owned by the new tt. Disown them from the old
+		 * tt before it is torn down so they are not freed.
+		 */
+		ttm_tt_defrag_disown_borrowed(bo->defrag_old_tt, bo->ttm);
 		ttm_tt_unpopulate(bo->bdev, bo->defrag_old_tt);
 		ttm_tt_destroy(bo->bdev, bo->defrag_old_tt);
 		bo->defrag_old_tt = NULL;
@@ -245,7 +258,12 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 

out_err:

	if (bo->defrag_old_tt) {
drivers/gpu/drm/ttm/ttm_bo.c +6 -1
-		/* Failed defrag move: restore the original backing. */
+		/*
+		 * Failed defrag move: drop any pages the new tt borrowed from
+		 * the old tt (still owned by the old tt) before destroying the
+		 * new tt, then restore the original backing.
+		 */
+		ttm_tt_defrag_disown_borrowed(bo->ttm, bo->defrag_old_tt);
		ttm_bo_tt_destroy(bo);
		bo->ttm = bo->defrag_old_tt;
		bo->defrag_old_tt = NULL;
drivers/gpu/drm/ttm/ttm_bo_util.c +5 -0
index 771313162c8d..7cb718fc97c0 100644
@@ -633,7 +633,12 @@ static int ttm_bo_move_to_ghost(struct ttm_buffer_object *bo,
 		 * tt (the move's source) to the ghost so it is kept populated
 		 * until @fence signals and then unbound and destroyed, instead
 		 * of being torn down synchronously. @bo keeps its new tt.
+		 *
+		 * Any pages the new tt borrowed from the old tt are now owned by
+		 * the new tt; disown them from the old tt so the ghost teardown
+		 * leaves them alone.
 		 */
+		ttm_tt_defrag_disown_borrowed(bo->defrag_old_tt, bo->ttm);
 		ghost_obj->ttm = bo->defrag_old_tt;
 		ghost_obj->defrag_old_tt = NULL;
 		bo->defrag_old_tt = NULL;
drivers/gpu/drm/ttm/ttm_pool.c +25 -0
index 2598d2f33767..52f5e64f05dd 100644
@@ -792,6 +792,148 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest,
 	return min_t(unsigned int, highest, __fls(alloc->remaining_pages));
 }
 
+/*
+ * Defragmentation page harvesting.
+ *
+ * A defrag move relocates a populated tt into a freshly allocated one in order
+ * to upgrade sub-beneficial-order chunks to the device's beneficial order.
+ * Chunks in the old tt (ctx->defrag_old_tt) that are ALREADY at the beneficial
+ * order do not need to be reallocated: their physical pages can simply be
+ * transferred ("harvested") into the new tt at the same offset, avoiding the
+ * alloc/free churn that is significant for large buffers.
+ *
+ * Harvesting only borrows the pages: the new tt simply references the old tt's
+ * pages, which remain owned by the old tt. This keeps the unwind paths simple -
+ * if the (re)allocation or the move later fails, the new tt is torn down without
+ * freeing the borrowed pages and the old tt is restored fully intact. Ownership
+ * of the shared pages is transferred to the new tt only once the move has
+ * committed, via ttm_tt_defrag_disown_borrowed() (see ttm_tt.c).
+ */
+
+/*
+ * Return the allocation order of the old tt chunk whose head sits at @off, or 0
+ * if @off is not an old chunk head (it falls inside a chunk, or the slot is
+ * empty / backed up). @cursor monotonically tracks the old chunk boundary and
+ * must be advanced past every offset that has already been queried; a page
+ * order may only be read from a chunk head page, hence the lazy walk.
+ */

drivers/gpu/drm/ttm/ttm_pool.c static unsigned int ttm_pool_alloc_find_order(unsigned int highest, · line 829

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+static unsigned int ttm_pool_defrag_old_order(struct ttm_pool *pool,
+					      struct ttm_tt *old_tt,
+					      pgoff_t off, pgoff_t *cursor)
+{
+	struct page *p;
+
+	while (*cursor < off) {
+		p = old_tt->pages[*cursor];
+		if (!p || ttm_backup_page_ptr_is_handle(p))
+			return 0;
Assisted-by LLM

ttm_backup_page_ptr_is_handle() is only check here, but not in the harvest function. Probably we won't endup with ptr_is_handle() case during defrag, but thisasymmetry makes it bit confusing.

Quoted 77 lines
drivers/gpu/drm/ttm/ttm_pool.c +77 -0
+		*cursor += 1UL << ttm_pool_page_order(pool, p);
+	}
+
+	if (*cursor != off)
+		return 0;
+
+	p = old_tt->pages[off];
+	if (!p || ttm_backup_page_ptr_is_handle(p))
+		return 0;
+
+	return ttm_pool_page_order(pool, p);
+}
+
+/*
+ * Borrow an already beneficial-order chunk from the old tt into @alloc. The
+ * pages stay owned by the old tt until the move commits.
+ */
+static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
+				 pgoff_t off, struct ttm_pool_alloc_state *alloc)
+{
+	pgoff_t nr = 1UL << order;
+	int r;
+
+	/*
+	 * A harvested chunk keeps the (same) bo's caching, so the deferred
+	 * caching range is consistent here; flush it before committing, exactly
+	 * like the caching-consistent path in ttm_pool_page_allocated().
+	 */
+	r = ttm_pool_apply_caching(alloc);
+	if (r)
+		return r;
+
+	/*
+	 * The borrowed chunk's page pointers (consecutive struct pages of one
+	 * compound allocation) and dma addresses already sit contiguously in
+	 * the old tt at @off. Copy the whole run with memcpy() instead of the
+	 * per-page increment loop in ttm_pool_allocated_page_commit(): for a
+	 * large defrag the borrowed pages dominate the new tt, so this turns
+	 * the populate fill from O(num_pages) individual stores into a handful
+	 * of bulk copies.
+	 */
+	memcpy(alloc->pages, &old_tt->pages[off], nr * sizeof(*alloc->pages));
+	alloc->pages += nr;
+	alloc->remaining_pages -= nr;
+
+	if (alloc->dma_addr) {
+		memcpy(alloc->dma_addr, &old_tt->dma_address[off],
+		       nr * sizeof(*alloc->dma_addr));
+		alloc->dma_addr += nr;
+	}
+
+	alloc->caching_divide = alloc->pages;
+
+	return 0;
+}
+
+static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
+				      struct ttm_tt *old_tt, pgoff_t off,
+				      struct ttm_pool_alloc_state *alloc)
+{
+	/*
+	 * @off is always an old-tt chunk head here, never a tail page, so
+	 * reading the per-chunk order below is safe. This holds because:
+	 *  - Old chunks are self-aligned: greedy largest-power-of-two fill
+	 *    places an order-k chunk at a 2^k-aligned offset, so any
+	 *    beneficial-order-aligned offset is a chunk boundary.
+	 *  - Fresh defrag allocations are capped at the beneficial order (see
+	 *    ttm_pool_iter_acquire_page()), so they advance @off in beneficial
+	 *    steps and never overshoot into the middle of a larger old chunk;
+	 *    super-beneficial old chunks are always harvested whole by
+	 *    ttm_pool_iter_reuse_old().
+	 *  - We are only reached once the defrag budget/prealloc is exhausted,
+	 *    which in the capped/prealloc config leaves @off beneficial-aligned
+	 *    (or, in the sub-beneficial tail, already walking old boundaries).
+	 */
+	while (alloc->remaining_pages) {
+		struct page *p = old_tt->pages[off];

drivers/gpu/drm/ttm/ttm_pool.c static unsigned int ttm_pool_alloc_find_order(unsigned int highest, · line 916

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+		unsigned int order = ttm_pool_page_order(pool, p);
+		pgoff_t nr = 1UL << order;
+		int r;
+
+		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
+		if (r)
+			return r;
+
+		off += nr;
+	}

The 'off' being always at chunk head is documented, but not checked/enforced in the code. Though currently, we won't have it otherwise, but tomorrow if we do for some reason, it will go unnoticed. Probably worth adding a warning here.

Quoted 8 lines
drivers/gpu/drm/ttm/ttm_pool.c +8 -0
+
+	return 0;
+}
+
+/**
+ * enum ttm_pool_iter_action - Outcome of a per-order allocation phase
+ * @TTM_POOL_ITER_FILL: A page (@it->p) was acquired; proceed to fill it in.
+ * @TTM_POOL_ITER_RETRY: Loop state was adjusted (e.g. order lowered or a chunk

drivers/gpu/drm/ttm/ttm_pool.c static unsigned int ttm_pool_alloc_find_order(unsigned int highest, · line 934

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+ *	harvested); re-enter the loop body for the next order.
+ * @TTM_POOL_ITER_STOP: The allocation is complete; stop the loop.
+ *
+ * Phase helpers return one of these (>= 0) on success, or a negative errno on
+ * failure (the caller then unwinds via error_free_all).
+ */
+enum ttm_pool_iter_action {
+	TTM_POOL_ITER_FILL,
+	TTM_POOL_ITER_RETRY,
+	TTM_POOL_ITER_STOP,

NIT...STOP is really not needed here, RETRY will do as the loop is expected to end there. But it is fine.

Quoted 209 lines
drivers/gpu/drm/ttm/ttm_pool.c +83 -12
+};
+
 /**
 * struct ttm_pool_alloc_iter - Working state for the __ttm_pool_alloc() loop
 *
@@ -806,6 +948,7 @@ struct ttm_pool_alloc_iter {
 	const struct ttm_operation_ctx *ctx;
 	struct ttm_pool_alloc_state *alloc;
 	struct ttm_pool_tt_restore *restore;
+	struct ttm_tt *defrag_old_tt;
 	unsigned int beneficial_order;
 	gfp_t gfp_flags;
 
@@ -814,9 +957,50 @@ struct ttm_pool_alloc_iter {
 	enum ttm_caching page_caching;
 	bool allow_pools;
 	bool fail_beneficial;
+	unsigned int alloc_count;
+	pgoff_t old_cursor;
 	struct page *p;
 };
 
+/*
+ * Defrag move: reuse an already beneficial-order chunk from the old tt at this
+ * offset rather than reallocating it. Harvest whatever order the old chunk has
+ * (>= beneficial order), which may exceed the order the new tt would otherwise
+ * pick. An order below beneficial_order is only reused for the unaligned tail
+ * (where there is no benefit to reallocating). Only borrows the pages; the alloc
+ * unwind path can drop them again.
+ *
+ * Returns TTM_POOL_ITER_RETRY when a chunk was harvested, TTM_POOL_ITER_FILL to
+ * fall through to a fresh allocation, or a negative errno.
+ */
+static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
+{
+	unsigned int harvest_order;
+	pgoff_t off;
+	int r;
+
+	if (!it->defrag_old_tt || !it->beneficial_order)
+		return TTM_POOL_ITER_FILL;
+
+	off = it->tt->num_pages - it->alloc->remaining_pages;
+	harvest_order = ttm_pool_defrag_old_order(it->pool, it->defrag_old_tt,
+						  off, &it->old_cursor);
+
+	if (harvest_order < it->beneficial_order &&
+	    it->order >= it->beneficial_order)
+		return TTM_POOL_ITER_FILL;
+
+	it->order = harvest_order;
+	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order, off, it->alloc);
+	if (r)
+		return r;
+
+	it->page_caching = it->tt->caching;
+	it->allow_pools = true;
+
+	return TTM_POOL_ITER_RETRY;
+}
+
 /*
 * Acquire a single page for the current order, leaving it in @it->p (NULL on
 * failure). Tries a same-order pool page, then a fresh system allocation. Fault
@@ -859,9 +1043,10 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 /*
 * The current order could not be satisfied. Lower it and retry, recording a
 * sub-optimal backing (so the driver can later re-defrag) when we drop below
- * the beneficial order.
+ * the beneficial order. For a defrag move that has already made partial
+ * progress, harvest the remaining old pages and stop instead.
 *
- * Returns 0 to retry the (lowered) order, or a negative errno.
+ * Returns TTM_POOL_ITER_RETRY, TTM_POOL_ITER_STOP, or a negative errno.
 */
 static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 {
@@ -871,16 +1056,24 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	if (!it->order)
 		return -ENOMEM;
 
-	/*
-	 * Failing to allocate at the device's beneficial order means we are
-	 * about to back this object with a sub-optimal (smaller order) set of
-	 * pages. Record it so the driver can later try to defragment the object
-	 * back to beneficial order.
-	 */
 	if (it->fail_beneficial || at_beneficial) {
 		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
-		if (it->ctx->defrag)
+
+		if (it->alloc_count && it->defrag_old_tt) {
+			pgoff_t off = it->tt->num_pages -
+				      it->alloc->remaining_pages;
+			int r;
+
+			r = ttm_pool_harvest_remaining(it->pool,
+						       it->defrag_old_tt, off,
+						       it->alloc);
+			if (r)
+				return r;
+
+			return TTM_POOL_ITER_STOP;
+		} else if (it->defrag_old_tt) {
 			return -ENOMEM;
+		}
 	}
 
 	if (it->fail_beneficial)
@@ -890,7 +1083,7 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	it->page_caching = it->tt->caching;
 	it->allow_pools = true;
 
-	return 0;
+	return TTM_POOL_ITER_RETRY;
 }
 
 static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
@@ -904,6 +1097,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		.ctx = ctx,
 		.alloc = alloc,
 		.restore = restore,
+		.defrag_old_tt = restore ? NULL : ctx->defrag_old_tt,
 		.beneficial_order = ttm_pool_beneficial_order(pool),
 		.page_caching = tt->caching,
 		.allow_pools = true,
@@ -931,13 +1125,22 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
+		/* Reuse an already beneficial-order chunk from the old tt. */
+		r = ttm_pool_iter_reuse_old(&it);
+		if (r < 0)
+			goto error_free_all;
+		if (r == TTM_POOL_ITER_RETRY)
+			continue;
+
 		/* Acquire a page (pool / system) for this order. */
 		ttm_pool_iter_acquire_page(&it);
 		if (!it.p) {
 			r = ttm_pool_iter_lower_order(&it);
-			if (r)
+			if (r < 0)
 				goto error_free_all;
-			continue;
+			if (r == TTM_POOL_ITER_RETRY)
+				continue;
+			break;	/* TTM_POOL_ITER_STOP */
 		}
 
 		r = ttm_pool_page_allocated(pool, it.order, it.p,
@@ -951,6 +1154,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			if (r)
 				goto error_free_all;
 		}
+
+		it.alloc_count++;
 	}
 
 	r = ttm_pool_apply_caching(alloc);
@@ -969,6 +1174,12 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (tt->restore)
 		return r;
 
+	/*
+	 * Drop any pages borrowed from the old tt during a defrag move: they are
+	 * still owned by the old tt and must not be freed here.
+	 */
+	ttm_tt_defrag_disown_borrowed(tt, it.defrag_old_tt);
+
 	caching_divide = alloc->caching_divide - tt->pages;
 	ttm_pool_free_range(pool, tt, tt->caching, 0, caching_divide);
 	ttm_pool_free_range(pool, tt, ttm_cached, caching_divide,
drivers/gpu/drm/ttm/ttm_tt.c +28 -0
index b645a1818184..2388ff794a59 100644
@@ -450,6 +450,73 @@ void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm)
 }
 EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_unpopulate);
 
+/**
+ * ttm_tt_defrag_disown_borrowed() - Drop borrowed defrag pages from a tt
+ * @tt: The tt to drop the borrowed page references from.
+ * @src: The tt the pages may have been harvested from, or NULL.
+ *
+ * During a defragmentation move the freshly (re)allocated tt may harvest
+ * already beneficial-order chunks from the BO's old tt, sharing the underlying
+ * pages instead of reallocating them. Until the move commits these pages stay
+ * owned by the old tt. Clearing a tt's slots for every page it shares with
+ * @src (the same struct page at the same index) makes that tt's teardown leave
+ * the shared pages alone, so they are freed exactly once by their owner.
+ *
+ * This is used both to unwind a failed defrag move (disown the borrowed pages
+ * from the new tt, leaving the old tt intact) and to commit a successful one
+ * (disown the now-transferred pages from the old tt, leaving them owned by the
+ * new tt).
+ */
+void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src)
+{
+	pgoff_t i = 0, run_start = 0;
+	bool in_run = false;
+
+	if (!tt || !src)
+		return;
+
+	/*
+	 * Borrowing happens at chunk granularity: a borrowed chunk shares the
+	 * same struct page on both tts across its whole allocation order, while

drivers/gpu/drm/ttm/ttm_tt.c void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm) · line 490

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_tt.c +10 -0
+	 * a non-borrowed chunk differs across its whole order. Stride by @src's
+	 * chunk order instead of walking every page, coalescing the borrowed
+	 * (matching) chunks into runs and clearing each run with a single
+	 * memset rather than a per-page store. @src is a fully populated,
+	 * non-dma_alloc tt (the defrag path), so its page order is recorded in
+	 * page->private and read via ttm_pool_page_order_nodma().
+	 */
+	while (i < tt->num_pages) {
+		struct page *sp = src->pages[i];
+		unsigned int order = sp ? ttm_pool_page_order_nodma(sp) : 0;

Here ttm_pool_page_order_nodma() is used where as in harvest funciton, ttm_pool_page_order() is used. Though it doesn't matter to Xe, this asymmetry makes it bit confusing.

Niranjana

Quoted 71 lines
drivers/gpu/drm/ttm/ttm_tt.c +29 -0
+		pgoff_t nr = 1UL << order;
+
+		if (sp && tt->pages[i] == sp) {
+			if (!in_run) {
+				run_start = i;
+				in_run = true;
+			}
+		} else if (in_run) {
+			memset(&tt->pages[run_start], 0,
+			       (i - run_start) * sizeof(*tt->pages));
+			if (tt->dma_address)
+				memset(&tt->dma_address[run_start], 0,
+				       (i - run_start) *
+				       sizeof(*tt->dma_address));
+			in_run = false;
+		}
+
+		i += nr;
+	}
+
+	if (in_run) {
+		memset(&tt->pages[run_start], 0,
+		       (i - run_start) * sizeof(*tt->pages));
+		if (tt->dma_address)
+			memset(&tt->dma_address[run_start], 0,
+			       (i - run_start) * sizeof(*tt->dma_address));
+	}
+}
+
 #ifdef CONFIG_DEBUG_FS
 
 /* Test the shrinker functions and dump the result */
include/drm/ttm/ttm_bo.h +12 -0
index f4f8a8c29f04..fcaba5fb8bf7 100644
@@ -211,6 +211,18 @@ struct ttm_operation_ctx {
 	 * order.
 	 */
 	bool defrag;
+	/**
+	 * @defrag_old_tt: During a defragmentation move, the bo's existing
+	 * (populated) tt that is being relocated. The page allocator may
+	 * harvest already beneficial-order chunks from this tt to back the new
+	 * tt instead of reallocating them, avoiding alloc/free churn for large
+	 * buffers. Harvested pages are only borrowed: they stay owned by
+	 * @defrag_old_tt until the move commits, at which point
+	 * ttm_tt_defrag_disown_borrowed() clears the old tt's slots for the
+	 * transferred pages so they are not freed when the old tt is torn down.
+	 * NULL for non-defrag operations.
+	 */
+	struct ttm_tt *defrag_old_tt;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
include/drm/ttm/ttm_tt.h +2 -0
index ce7533677d77..55c35bcf134d 100644
@@ -284,6 +284,8 @@ int ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm,
 */
 void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
 
+void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src);
+
 /**
 * ttm_tt_mark_for_clear - Mark pages for clearing on populate.
 *
Signature · 1 line
2.34.1
author maintainer
#23 lore raw

Re: [PATCH v3 07/33] drm/ttm: Harvest beneficial-order pages on defragmentation moves

Quoted 174 lines · Niranjana Vishwanathapura
Quoted 173 lines · Matthew Brost

A defragmentation move relocates a populated tt into a freshly allocated one so that sub-beneficial-order chunks can be upgraded to the device's beneficial order. Previously, if allocation could not complete entirely at beneficial order, the move failed and all progress was discarded.

Instead, once allocation has successfully upgraded part of the object, the remainder of the old tt is harvested to complete the new tt. Thread the BO's old, still-populated tt into the page allocator via a new ttm_operation_ctx::defrag_old_tt. While allocation is still proceeding, any existing chunk already at (or above) the beneficial order is reused directly instead of being reallocated. If allocation later has to fall back below beneficial_order after making partial progress, the remainder of the old tt is harvested instead of failing the move. This preserves the beneficial-order upgrades already made while leaving the rest of the object unchanged, allowing subsequent defragmentation attempts to make additional forward progress.

Harvesting only borrows the pages; they remain owned by the old tt until the move commits. This keeps every unwind path non-destructive: if the (re)allocation or the move later fails, the new tt is torn down without freeing the borrowed pages and the old tt is restored fully intact. Ownership of the shared pages is transferred to the new tt only once the move has committed. ttm_tt_defrag_disown_borrowed() implements both directions of the transfer by clearing a tt's slots for every page it shares with the other tt (same struct page at the same index), so that tt's teardown leaves the shared pages alone and each page is freed exactly once by its owner:

- on the alloc unwind and the failed-move paths, the new tt disowns the borrowed pages, leaving the old tt as the sole owner;
- on the successful-move paths (synchronous teardown and the pipelined ghost-object teardown), the old tt disowns the now-transferred pages, leaving the new tt as the sole owner.

Detecting shared pages by same-index struct page identity needs no extra per-tt bookkeeping, since two distinct allocations never alias. This is safe with drivers, such as Xe, that build their own per-tt sg table and DMA-map it independently: each tt maps and unmaps its own sg, so sharing the underlying pages between tts is DMA-clean.

This is a no-op unless ctx->defrag_old_tt is set, which only happens on a defragmentation move.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---

v3:
- Document the chunk-head invariant read by
ttm_pool_harvest_remaining() (Sashiko)
---

drivers/gpu/drm/ttm/ttm_bo.c      |  20 ++-
drivers/gpu/drm/ttm/ttm_bo_util.c |   5 +
drivers/gpu/drm/ttm/ttm_pool.c    | 235 ++++++++++++++++++++++++++++--
drivers/gpu/drm/ttm/ttm_tt.c      |  67 +++++++++
include/drm/ttm/ttm_bo.h          |  12 ++
include/drm/ttm/ttm_tt.h          |   2 +
6 files changed, 328 insertions(+), 13 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +13 -0
index c1629b4c4f0f..ab2e637a017f 100644
@@ -182,7 +182,14 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 		}
 
 		if (defrag_old_tt || mem->mem_type != TTM_PL_SYSTEM) {
+			/*
+			 * Let the page allocator harvest already
+			 * beneficial-order chunks out of the old tt to back
+			 * the new one, rather than reallocating them.
+			 */
+			ctx->defrag_old_tt = defrag_old_tt;
 			ret = ttm_bo_populate(bo, ctx);
+			ctx->defrag_old_tt = NULL;
 			if (ret) {
 				/*
 				 * Discharge the freshly allocated backing and
@@ -235,6 +242,12 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 	}
 
 	if (bo->defrag_old_tt) {
+		/*
+		 * Successful defrag move: any pages the new tt borrowed from the
+		 * old tt are now owned by the new tt. Disown them from the old
+		 * tt before it is torn down so they are not freed.
+		 */
+		ttm_tt_defrag_disown_borrowed(bo->defrag_old_tt, bo->ttm);
 		ttm_tt_unpopulate(bo->bdev, bo->defrag_old_tt);
 		ttm_tt_destroy(bo->bdev, bo->defrag_old_tt);
 		bo->defrag_old_tt = NULL;
@@ -245,7 +258,12 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 

out_err:

	if (bo->defrag_old_tt) {
drivers/gpu/drm/ttm/ttm_bo.c +6 -1
-		/* Failed defrag move: restore the original backing. */
+		/*
+		 * Failed defrag move: drop any pages the new tt borrowed from
+		 * the old tt (still owned by the old tt) before destroying the
+		 * new tt, then restore the original backing.
+		 */
+		ttm_tt_defrag_disown_borrowed(bo->ttm, bo->defrag_old_tt);
		ttm_bo_tt_destroy(bo);
		bo->ttm = bo->defrag_old_tt;
		bo->defrag_old_tt = NULL;
drivers/gpu/drm/ttm/ttm_bo_util.c +5 -0
index 771313162c8d..7cb718fc97c0 100644
@@ -633,7 +633,12 @@ static int ttm_bo_move_to_ghost(struct ttm_buffer_object *bo,
 		 * tt (the move's source) to the ghost so it is kept populated
 		 * until @fence signals and then unbound and destroyed, instead
 		 * of being torn down synchronously. @bo keeps its new tt.
+		 *
+		 * Any pages the new tt borrowed from the old tt are now owned by
+		 * the new tt; disown them from the old tt so the ghost teardown
+		 * leaves them alone.
 		 */
+		ttm_tt_defrag_disown_borrowed(bo->defrag_old_tt, bo->ttm);
 		ghost_obj->ttm = bo->defrag_old_tt;
 		ghost_obj->defrag_old_tt = NULL;
 		bo->defrag_old_tt = NULL;
drivers/gpu/drm/ttm/ttm_pool.c +29 -0
index 2598d2f33767..52f5e64f05dd 100644
@@ -792,6 +792,148 @@ static unsigned int ttm_pool_alloc_find_order(unsigned int highest,
 	return min_t(unsigned int, highest, __fls(alloc->remaining_pages));
 }
 
+/*
+ * Defragmentation page harvesting.
+ *
+ * A defrag move relocates a populated tt into a freshly allocated one in order
+ * to upgrade sub-beneficial-order chunks to the device's beneficial order.
+ * Chunks in the old tt (ctx->defrag_old_tt) that are ALREADY at the beneficial
+ * order do not need to be reallocated: their physical pages can simply be
+ * transferred ("harvested") into the new tt at the same offset, avoiding the
+ * alloc/free churn that is significant for large buffers.
+ *
+ * Harvesting only borrows the pages: the new tt simply references the old tt's
+ * pages, which remain owned by the old tt. This keeps the unwind paths simple -
+ * if the (re)allocation or the move later fails, the new tt is torn down without
+ * freeing the borrowed pages and the old tt is restored fully intact. Ownership
+ * of the shared pages is transferred to the new tt only once the move has
+ * committed, via ttm_tt_defrag_disown_borrowed() (see ttm_tt.c).
+ */
+
+/*
+ * Return the allocation order of the old tt chunk whose head sits at @off, or 0
+ * if @off is not an old chunk head (it falls inside a chunk, or the slot is
+ * empty / backed up). @cursor monotonically tracks the old chunk boundary and
+ * must be advanced past every offset that has already been queried; a page
+ * order may only be read from a chunk head page, hence the lazy walk.
+ */
+static unsigned int ttm_pool_defrag_old_order(struct ttm_pool *pool,
+					      struct ttm_tt *old_tt,
+					      pgoff_t off, pgoff_t *cursor)
+{
Quoted 9 lines

drivers/gpu/drm/ttm/ttm_pool.c static unsigned int ttm_pool_alloc_find_order(unsigned int highest, · line 829

Quoted 6 lines
drivers/gpu/drm/ttm/ttm_pool.c +6 -0
+	struct page *p;
+
+	while (*cursor < off) {
+		p = old_tt->pages[*cursor];
+		if (!p || ttm_backup_page_ptr_is_handle(p))
+			return 0;
Assisted-by LLM

Yes.

Quoted 4 lines

ttm_backup_page_ptr_is_handle() is only check here, but not in the harvest function. Probably we won't endup with ptr_is_handle() case during defrag, but thisasymmetry makes it bit confusing.

In Xe we only allow defrag in TT placement, so ttm_backup_page_ptr_is_handle probably could be omitted here.

Quoted 82 lines
Quoted 82 lines
drivers/gpu/drm/ttm/ttm_pool.c +82 -0
+		*cursor += 1UL << ttm_pool_page_order(pool, p);
+	}
+
+	if (*cursor != off)
+		return 0;
+
+	p = old_tt->pages[off];
+	if (!p || ttm_backup_page_ptr_is_handle(p))
+		return 0;
+
+	return ttm_pool_page_order(pool, p);
+}
+
+/*
+ * Borrow an already beneficial-order chunk from the old tt into @alloc. The
+ * pages stay owned by the old tt until the move commits.
+ */
+static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
+				 pgoff_t off, struct ttm_pool_alloc_state *alloc)
+{
+	pgoff_t nr = 1UL << order;
+	int r;
+
+	/*
+	 * A harvested chunk keeps the (same) bo's caching, so the deferred
+	 * caching range is consistent here; flush it before committing, exactly
+	 * like the caching-consistent path in ttm_pool_page_allocated().
+	 */
+	r = ttm_pool_apply_caching(alloc);
+	if (r)
+		return r;
+
+	/*
+	 * The borrowed chunk's page pointers (consecutive struct pages of one
+	 * compound allocation) and dma addresses already sit contiguously in
+	 * the old tt at @off. Copy the whole run with memcpy() instead of the
+	 * per-page increment loop in ttm_pool_allocated_page_commit(): for a
+	 * large defrag the borrowed pages dominate the new tt, so this turns
+	 * the populate fill from O(num_pages) individual stores into a handful
+	 * of bulk copies.
+	 */
+	memcpy(alloc->pages, &old_tt->pages[off], nr * sizeof(*alloc->pages));
+	alloc->pages += nr;
+	alloc->remaining_pages -= nr;
+
+	if (alloc->dma_addr) {
+		memcpy(alloc->dma_addr, &old_tt->dma_address[off],
+		       nr * sizeof(*alloc->dma_addr));
+		alloc->dma_addr += nr;
+	}
+
+	alloc->caching_divide = alloc->pages;
+
+	return 0;
+}
+
+static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
+				      struct ttm_tt *old_tt, pgoff_t off,
+				      struct ttm_pool_alloc_state *alloc)
+{
+	/*
+	 * @off is always an old-tt chunk head here, never a tail page, so
+	 * reading the per-chunk order below is safe. This holds because:
+	 *  - Old chunks are self-aligned: greedy largest-power-of-two fill
+	 *    places an order-k chunk at a 2^k-aligned offset, so any
+	 *    beneficial-order-aligned offset is a chunk boundary.
+	 *  - Fresh defrag allocations are capped at the beneficial order (see
+	 *    ttm_pool_iter_acquire_page()), so they advance @off in beneficial
+	 *    steps and never overshoot into the middle of a larger old chunk;
+	 *    super-beneficial old chunks are always harvested whole by
+	 *    ttm_pool_iter_reuse_old().
+	 *  - We are only reached once the defrag budget/prealloc is exhausted,
+	 *    which in the capped/prealloc config leaves @off beneficial-aligned
+	 *    (or, in the sub-beneficial tail, already walking old boundaries).
+	 */
+	while (alloc->remaining_pages) {
+		struct page *p = old_tt->pages[off];
+		unsigned int order = ttm_pool_page_order(pool, p);
+		pgoff_t nr = 1UL << order;
+		int r;
+
+		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
Quoted 8 lines

drivers/gpu/drm/ttm/ttm_pool.c static unsigned int ttm_pool_alloc_find_order(unsigned int highest, · line 916

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_pool.c +5 -0
+		if (r)
+			return r;
+
+		off += nr;
+	}

The 'off' being always at chunk head is documented, but not checked/enforced in the code. Though currently, we won't have it otherwise, but tomorrow if

I don't know how to enforce this. p->private is the chunk order at the head page only but an order of 0 is a valid value, i.e., I'm not sure how to implement 'page is chunk head' in the current code.

Quoted 16 lines

we do for some reason, it will go unnoticed. Probably worth adding a warning here.

Quoted 13 lines
drivers/gpu/drm/ttm/ttm_pool.c +13 -0
+
+	return 0;
+}
+
+/**
+ * enum ttm_pool_iter_action - Outcome of a per-order allocation phase
+ * @TTM_POOL_ITER_FILL: A page (@it->p) was acquired; proceed to fill it in.
+ * @TTM_POOL_ITER_RETRY: Loop state was adjusted (e.g. order lowered or a chunk
+ *	harvested); re-enter the loop body for the next order.
+ * @TTM_POOL_ITER_STOP: The allocation is complete; stop the loop.
+ *
+ * Phase helpers return one of these (>= 0) on success, or a negative errno on
+ * failure (the caller then unwinds via error_free_all).
Quoted 9 lines

drivers/gpu/drm/ttm/ttm_pool.c static unsigned int ttm_pool_alloc_find_order(unsigned int highest, · line 934

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_pool.c +5 -0
+ */
+enum ttm_pool_iter_action {
+	TTM_POOL_ITER_FILL,
+	TTM_POOL_ITER_RETRY,
+	TTM_POOL_ITER_STOP,

NIT...STOP is really not needed here, RETRY will do as the loop is expected to end there. But it is fine.

Not following this - __ttm_pool_alloc look at TTM_POOL_ITER_RETRY vs STOP.

Quoted 215 lines
Quoted 215 lines
drivers/gpu/drm/ttm/ttm_pool.c +83 -12
+};
+
 /**
 * struct ttm_pool_alloc_iter - Working state for the __ttm_pool_alloc() loop
 *
@@ -806,6 +948,7 @@ struct ttm_pool_alloc_iter {
 	const struct ttm_operation_ctx *ctx;
 	struct ttm_pool_alloc_state *alloc;
 	struct ttm_pool_tt_restore *restore;
+	struct ttm_tt *defrag_old_tt;
 	unsigned int beneficial_order;
 	gfp_t gfp_flags;
 
@@ -814,9 +957,50 @@ struct ttm_pool_alloc_iter {
 	enum ttm_caching page_caching;
 	bool allow_pools;
 	bool fail_beneficial;
+	unsigned int alloc_count;
+	pgoff_t old_cursor;
 	struct page *p;
 };
 
+/*
+ * Defrag move: reuse an already beneficial-order chunk from the old tt at this
+ * offset rather than reallocating it. Harvest whatever order the old chunk has
+ * (>= beneficial order), which may exceed the order the new tt would otherwise
+ * pick. An order below beneficial_order is only reused for the unaligned tail
+ * (where there is no benefit to reallocating). Only borrows the pages; the alloc
+ * unwind path can drop them again.
+ *
+ * Returns TTM_POOL_ITER_RETRY when a chunk was harvested, TTM_POOL_ITER_FILL to
+ * fall through to a fresh allocation, or a negative errno.
+ */
+static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
+{
+	unsigned int harvest_order;
+	pgoff_t off;
+	int r;
+
+	if (!it->defrag_old_tt || !it->beneficial_order)
+		return TTM_POOL_ITER_FILL;
+
+	off = it->tt->num_pages - it->alloc->remaining_pages;
+	harvest_order = ttm_pool_defrag_old_order(it->pool, it->defrag_old_tt,
+						  off, &it->old_cursor);
+
+	if (harvest_order < it->beneficial_order &&
+	    it->order >= it->beneficial_order)
+		return TTM_POOL_ITER_FILL;
+
+	it->order = harvest_order;
+	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order, off, it->alloc);
+	if (r)
+		return r;
+
+	it->page_caching = it->tt->caching;
+	it->allow_pools = true;
+
+	return TTM_POOL_ITER_RETRY;
+}
+
 /*
 * Acquire a single page for the current order, leaving it in @it->p (NULL on
 * failure). Tries a same-order pool page, then a fresh system allocation. Fault
@@ -859,9 +1043,10 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 /*
 * The current order could not be satisfied. Lower it and retry, recording a
 * sub-optimal backing (so the driver can later re-defrag) when we drop below
- * the beneficial order.
+ * the beneficial order. For a defrag move that has already made partial
+ * progress, harvest the remaining old pages and stop instead.
 *
- * Returns 0 to retry the (lowered) order, or a negative errno.
+ * Returns TTM_POOL_ITER_RETRY, TTM_POOL_ITER_STOP, or a negative errno.
 */
 static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 {
@@ -871,16 +1056,24 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	if (!it->order)
 		return -ENOMEM;
 
-	/*
-	 * Failing to allocate at the device's beneficial order means we are
-	 * about to back this object with a sub-optimal (smaller order) set of
-	 * pages. Record it so the driver can later try to defragment the object
-	 * back to beneficial order.
-	 */
 	if (it->fail_beneficial || at_beneficial) {
 		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
-		if (it->ctx->defrag)
+
+		if (it->alloc_count && it->defrag_old_tt) {
+			pgoff_t off = it->tt->num_pages -
+				      it->alloc->remaining_pages;
+			int r;
+
+			r = ttm_pool_harvest_remaining(it->pool,
+						       it->defrag_old_tt, off,
+						       it->alloc);
+			if (r)
+				return r;
+
+			return TTM_POOL_ITER_STOP;
+		} else if (it->defrag_old_tt) {
 			return -ENOMEM;
+		}
 	}
 
 	if (it->fail_beneficial)
@@ -890,7 +1083,7 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 	it->page_caching = it->tt->caching;
 	it->allow_pools = true;
 
-	return 0;
+	return TTM_POOL_ITER_RETRY;
 }
 
 static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
@@ -904,6 +1097,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		.ctx = ctx,
 		.alloc = alloc,
 		.restore = restore,
+		.defrag_old_tt = restore ? NULL : ctx->defrag_old_tt,
 		.beneficial_order = ttm_pool_beneficial_order(pool),
 		.page_caching = tt->caching,
 		.allow_pools = true,
@@ -931,13 +1125,22 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
+		/* Reuse an already beneficial-order chunk from the old tt. */
+		r = ttm_pool_iter_reuse_old(&it);
+		if (r < 0)
+			goto error_free_all;
+		if (r == TTM_POOL_ITER_RETRY)
+			continue;
+
 		/* Acquire a page (pool / system) for this order. */
 		ttm_pool_iter_acquire_page(&it);
 		if (!it.p) {
 			r = ttm_pool_iter_lower_order(&it);
-			if (r)
+			if (r < 0)
 				goto error_free_all;
-			continue;
+			if (r == TTM_POOL_ITER_RETRY)
+				continue;
+			break;	/* TTM_POOL_ITER_STOP */
 		}
 
 		r = ttm_pool_page_allocated(pool, it.order, it.p,
@@ -951,6 +1154,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			if (r)
 				goto error_free_all;
 		}
+
+		it.alloc_count++;
 	}
 
 	r = ttm_pool_apply_caching(alloc);
@@ -969,6 +1174,12 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	if (tt->restore)
 		return r;
 
+	/*
+	 * Drop any pages borrowed from the old tt during a defrag move: they are
+	 * still owned by the old tt and must not be freed here.
+	 */
+	ttm_tt_defrag_disown_borrowed(tt, it.defrag_old_tt);
+
 	caching_divide = alloc->caching_divide - tt->pages;
 	ttm_pool_free_range(pool, tt, tt->caching, 0, caching_divide);
 	ttm_pool_free_range(pool, tt, ttm_cached, caching_divide,
drivers/gpu/drm/ttm/ttm_tt.c +34 -0
index b645a1818184..2388ff794a59 100644
@@ -450,6 +450,73 @@ void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm)
 }
 EXPORT_SYMBOL_FOR_TESTS_ONLY(ttm_tt_unpopulate);
 
+/**
+ * ttm_tt_defrag_disown_borrowed() - Drop borrowed defrag pages from a tt
+ * @tt: The tt to drop the borrowed page references from.
+ * @src: The tt the pages may have been harvested from, or NULL.
+ *
+ * During a defragmentation move the freshly (re)allocated tt may harvest
+ * already beneficial-order chunks from the BO's old tt, sharing the underlying
+ * pages instead of reallocating them. Until the move commits these pages stay
+ * owned by the old tt. Clearing a tt's slots for every page it shares with
+ * @src (the same struct page at the same index) makes that tt's teardown leave
+ * the shared pages alone, so they are freed exactly once by their owner.
+ *
+ * This is used both to unwind a failed defrag move (disown the borrowed pages
+ * from the new tt, leaving the old tt intact) and to commit a successful one
+ * (disown the now-transferred pages from the old tt, leaving them owned by the
+ * new tt).
+ */
+void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src)
+{
+	pgoff_t i = 0, run_start = 0;
+	bool in_run = false;
+
+	if (!tt || !src)
+		return;
+
+	/*
+	 * Borrowing happens at chunk granularity: a borrowed chunk shares the
+	 * same struct page on both tts across its whole allocation order, while
+	 * a non-borrowed chunk differs across its whole order. Stride by @src's
+	 * chunk order instead of walking every page, coalescing the borrowed
+	 * (matching) chunks into runs and clearing each run with a single
+	 * memset rather than a per-page store. @src is a fully populated,
+	 * non-dma_alloc tt (the defrag path), so its page order is recorded in
+	 * page->private and read via ttm_pool_page_order_nodma().
Quoted 9 lines

drivers/gpu/drm/ttm/ttm_tt.c void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm) · line 490

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_tt.c +4 -0
+	 */
+	while (i < tt->num_pages) {
+		struct page *sp = src->pages[i];
+		unsigned int order = sp ? ttm_pool_page_order_nodma(sp) : 0;

Here ttm_pool_page_order_nodma() is used where as in harvest funciton, ttm_pool_page_order() is used. Though it doesn't matter to Xe, this asymmetry makes it bit confusing.

We don't have the pool here, hence we use ttm_pool_page_order_nodma.

Defrag isn't supported on dma_alloc pages, thus is safe.

Matt

Quoted 73 lines

Niranjana

Quoted 71 lines
drivers/gpu/drm/ttm/ttm_tt.c +29 -0
+		pgoff_t nr = 1UL << order;
+
+		if (sp && tt->pages[i] == sp) {
+			if (!in_run) {
+				run_start = i;
+				in_run = true;
+			}
+		} else if (in_run) {
+			memset(&tt->pages[run_start], 0,
+			       (i - run_start) * sizeof(*tt->pages));
+			if (tt->dma_address)
+				memset(&tt->dma_address[run_start], 0,
+				       (i - run_start) *
+				       sizeof(*tt->dma_address));
+			in_run = false;
+		}
+
+		i += nr;
+	}
+
+	if (in_run) {
+		memset(&tt->pages[run_start], 0,
+		       (i - run_start) * sizeof(*tt->pages));
+		if (tt->dma_address)
+			memset(&tt->dma_address[run_start], 0,
+			       (i - run_start) * sizeof(*tt->dma_address));
+	}
+}
+
 #ifdef CONFIG_DEBUG_FS
 
 /* Test the shrinker functions and dump the result */
include/drm/ttm/ttm_bo.h +12 -0
index f4f8a8c29f04..fcaba5fb8bf7 100644
@@ -211,6 +211,18 @@ struct ttm_operation_ctx {
 	 * order.
 	 */
 	bool defrag;
+	/**
+	 * @defrag_old_tt: During a defragmentation move, the bo's existing
+	 * (populated) tt that is being relocated. The page allocator may
+	 * harvest already beneficial-order chunks from this tt to back the new
+	 * tt instead of reallocating them, avoiding alloc/free churn for large
+	 * buffers. Harvested pages are only borrowed: they stay owned by
+	 * @defrag_old_tt until the move commits, at which point
+	 * ttm_tt_defrag_disown_borrowed() clears the old tt's slots for the
+	 * transferred pages so they are not freed when the old tt is torn down.
+	 * NULL for non-defrag operations.
+	 */
+	struct ttm_tt *defrag_old_tt;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
include/drm/ttm/ttm_tt.h +2 -0
index ce7533677d77..55c35bcf134d 100644
@@ -284,6 +284,8 @@ int ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm,
 */
 void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
 
+void ttm_tt_defrag_disown_borrowed(struct ttm_tt *tt, const struct ttm_tt *src);
+
 /**
 * ttm_tt_mark_for_clear - Mark pages for clearing on populate.
 *
Signature · 1 line
2.34.1
author maintainer
#24 lore raw

PATCH v3 8/33 drm/ttm: Bound page (re)allocation per defragmentation move

In reply to Matthew Brost

A defragmentation move reallocates an object's backing at the device's beneficial order and harvests whatever the allocator cannot upgrade from the old tt. For a very large object this can reallocate (and have the driver re-copy) the entire backing in one move, which is expensive and can stall concurrent work for an unbounded stretch.

Add an optional per-move byte budget so a large object can be upgraded in slices across successive moves. Plumb a new ttm_operation_ctx::defrag_bytes_remaining through to the pool allocator:
only pages actually (re)allocated at the beneficial order are charged against it; pages harvested (borrowed) from defrag_old_tt are free and do not count. Once the budget is exhausted the allocator stops upgrading and completes the new tt by harvesting the remainder of the old tt as-is, marking the tt sub-optimal (TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED) so the driver re-queues it and a later move resumes the upgrade. The slices already upgraded come back as free already-beneficial harvests on the next pass, so each move spends budget only on new forward progress.

The allocator decrements the budget as it allocates and writes the unused remainder back on success, letting the caller compute how many bytes were actually spent. A value of 0 means unlimited (upgrade the whole object in one move), preserving the previous behaviour. This is only consulted on a defragmentation move, i.e. when defrag_old_tt is set.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 101 ++++++++++++++++++++++++++++++++-
 include/drm/ttm/ttm_bo.h       |  14 +++++
 2 files changed, 112 insertions(+), 3 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +98 -3
index 52f5e64f05dd..1673e81b8df8 100644
@@ -77,6 +77,11 @@ struct ttm_pool_dma {
  * @dma_addr: Pointer to the next tt dma_address entry to populate if any.
  * @remaining_pages: Remaining pages to populate.
  * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @defrag_bytes_remaining: Byte budget for newly (re)allocated pages on a
+ * defragmentation move, seeded from ttm_operation_ctx::defrag_bytes_remaining.
+ * Pages harvested (borrowed) from the old tt are free and do not count; once
+ * this drops to zero the allocator stops upgrading and harvests the remainder.
+ * On completion it holds the unused budget. 0 means unlimited (no defrag cap).
  */
 struct ttm_pool_alloc_state {
 	struct page **pages;
@@ -84,6 +89,7 @@ struct ttm_pool_alloc_state {
 	dma_addr_t *dma_addr;
 	pgoff_t remaining_pages;
 	enum ttm_caching tt_caching;
+	s64 defrag_bytes_remaining;
 };
 
 /**
@@ -780,6 +786,7 @@ static void ttm_pool_alloc_state_init(const struct ttm_tt *tt,
 	alloc->dma_addr = tt->dma_address;
 	alloc->remaining_pages = tt->num_pages;
 	alloc->tt_caching = tt->caching;
+	alloc->defrag_bytes_remaining = 0;
 }
 
 /*
@@ -885,8 +892,12 @@ static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
 
 static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 				      struct ttm_tt *old_tt, pgoff_t off,
-				      struct ttm_pool_alloc_state *alloc)
+				      struct ttm_pool_alloc_state *alloc,
+				      bool *suboptimal)
 {
+	unsigned int beneficial = ttm_pool_beneficial_order(pool);
+	pgoff_t bnr = 1UL << beneficial;
+
 	/*
 	 * @off is always an old-tt chunk head here, never a tail page, so
 	 * reading the per-chunk order below is safe. This holds because:
@@ -906,8 +917,23 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 		struct page *p = old_tt->pages[off];
 		unsigned int order = ttm_pool_page_order(pool, p);
 		pgoff_t nr = 1UL << order;
+		pgoff_t num_pages = off + alloc->remaining_pages;
 		int r;
 
+		/*
+		 * Only report a sub-optimal backing when a chunk we cannot
+		 * upgrade (out of budget/prealloc) sits in a beneficial-order
+		 * aligned region that fully fits in the tt: that region could
+		 * have been upgraded with more budget. A chunk below the
+		 * beneficial order is expected for the unaligned tail (the last
+		 * partial beneficial-order region) and for already beneficial-
+		 * order harvests - neither leaves real defrag work, so they must
+		 * not flag the tt for re-queue.
+		 */
+		if (beneficial && order < beneficial &&
+		    round_down(off, bnr) + bnr <= num_pages)
+			*suboptimal = true;
+
 		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
 		if (r)
 			return r;
@@ -951,6 +977,7 @@ struct ttm_pool_alloc_iter {
 	struct ttm_tt *defrag_old_tt;
 	unsigned int beneficial_order;
 	gfp_t gfp_flags;
+	bool defrag_capped;
 
 	/* Mutated as the loop walks the orders. */
 	unsigned int order;
@@ -962,6 +989,44 @@ struct ttm_pool_alloc_iter {
 	struct page *p;
 };
 
+/*
+ * Defrag move budget exhausted: the upgrade can make no further progress this
+ * pass. Snapshot @defrag_capped is set only when a byte budget was in force at
+ * entry.
+ */
+static bool ttm_pool_iter_defrag_exhausted(const struct ttm_pool_alloc_iter *it)
+{
+	if (!it->defrag_old_tt)
+		return false;
+
+	return it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0;
+}
+
+/*
+ * Complete the new tt by harvesting the rest of the old tt as-is (borrowing its
+ * pages, including any still below beneficial order). Never fall back to in-lock
+ * reclaim for the shortfall - that would reintroduce the stall we preallocated
+ * to avoid. Only re-queue (flag) the tt when the harvested remainder is
+ * genuinely sub-optimal; if every remaining old chunk was already at beneficial
+ * order the move fully upgraded the object.
+ */
+static int ttm_pool_iter_harvest_rest(struct ttm_pool_alloc_iter *it)
+{
+	pgoff_t off = it->tt->num_pages - it->alloc->remaining_pages;
+	bool suboptimal = false;
+	int r;
+
+	r = ttm_pool_harvest_remaining(it->pool, it->defrag_old_tt, off,
+				       it->alloc, &suboptimal);
+	if (r)
+		return r;
+
+	if (suboptimal)
+		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+
+	return 0;
+}
+
 /*
  * Defrag move: reuse an already beneficial-order chunk from the old tt at this
  * offset rather than reallocating it. Harvest whatever order the old chunk has
@@ -1062,11 +1127,12 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 		if (it->alloc_count && it->defrag_old_tt) {
 			pgoff_t off = it->tt->num_pages -
 				      it->alloc->remaining_pages;
+			bool suboptimal = false;
 			int r;
 
 			r = ttm_pool_harvest_remaining(it->pool,
 						       it->defrag_old_tt, off,
-						       it->alloc);
+						       it->alloc, &suboptimal);
 			if (r)
 				return r;
 
@@ -1121,10 +1187,23 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		gfp_flags |= GFP_HIGHUSER;
 
 	it.gfp_flags = gfp_flags;
+	it.defrag_capped = it.defrag_old_tt && alloc->defrag_bytes_remaining > 0;
 
 	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
+		/*
+		 * Out of defrag budget: harvest the rest of the old tt as-is and
+		 * stop (the tt is re-queued if the remainder is still
+		 * sub-optimal).
+		 */
+		if (ttm_pool_iter_defrag_exhausted(&it)) {
+			r = ttm_pool_iter_harvest_rest(&it);
+			if (r)
+				goto error_free_all;
+			break;
+		}
+
 		/* Reuse an already beneficial-order chunk from the old tt. */
 		r = ttm_pool_iter_reuse_old(&it);
 		if (r < 0)
@@ -1155,6 +1234,15 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 				goto error_free_all;
 		}
 
+		/*
+		 * Charge a newly (re)allocated chunk against the defrag move
+		 * budget. Harvested chunks borrow the old tt's pages for free
+		 * and never reach here.
+		 */
+		if (it.defrag_capped)
+			alloc->defrag_bytes_remaining -=
+				(s64)(1UL << it.order) << PAGE_SHIFT;
+
 		it.alloc_count++;
 	}
 
@@ -1204,14 +1292,21 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		   struct ttm_operation_ctx *ctx)
 {
 	struct ttm_pool_alloc_state alloc;
+	int ret;
 
 	if (WARN_ON(ttm_tt_is_backed_up(tt)))
 		return -EINVAL;
 
 	tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 	ttm_pool_alloc_state_init(tt, &alloc);
+	alloc.defrag_bytes_remaining = ctx->defrag_bytes_remaining;
+
+	ret = __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
+
+	/* Report the unused defrag budget back to the caller. */
+	ctx->defrag_bytes_remaining = alloc.defrag_bytes_remaining;
 
-	return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
+	return ret;
 }
 EXPORT_SYMBOL(ttm_pool_alloc);
 
include/drm/ttm/ttm_bo.h +14 -0
index fcaba5fb8bf7..6124d2627b47 100644
@@ -223,6 +223,20 @@ struct ttm_operation_ctx {
 	 * NULL for non-defrag operations.
 	 */
 	struct ttm_tt *defrag_old_tt;
+	/**
+	 * @defrag_bytes_remaining: Byte budget for newly allocated pages during
+	 * a defragmentation move. Only pages that are actually (re)allocated at
+	 * the beneficial order count against it; pages harvested (borrowed) from
+	 * @defrag_old_tt are free and do not. When the budget is exhausted the
+	 * pool allocator stops upgrading and completes the new tt by harvesting
+	 * the remainder of @defrag_old_tt, marking the tt sub-optimal so the
+	 * driver can resume on a later pass. The allocator decrements this as it
+	 * allocates, so on return it holds the unused budget (which the caller
+	 * can use to compute how many bytes were spent). A value of 0 means
+	 * unlimited (upgrade the whole object). Only consulted when
+	 * @defrag_old_tt is set.
+	 */
+	s64 defrag_bytes_remaining;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
Signature · 1 line
2.34.1
#25 lore raw

Re: [PATCH v3 08/33] drm/ttm: Bound page (re)allocation per defragmentation move

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Negative defrag budget values are incorrectly treated as unlimited instead of exhausted. --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Negative defrag budget values are incorrectly treated as unlimited instead of exhausted. --

--- Patch [8]: [PATCH v3 08/33] drm/ttm: Bound page (re)allocation per defragmentation move --- commit 933e390472c0331c941d5caa6b241841ec7bb236
Author: Matthew Brost <[email protected]>

drm/ttm: Bound page (re)allocation per defragmentation move

This commit introduces a per-move byte budget to bound the reallocation size during TTM object defragmentation. It updates the pool allocator to track upgraded pages against this budget, stopping the upgrade process and harvesting the remaining old pages once the budget is exhausted.

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_pool.c +0 -0
index 52f5e64f05ddc..1673e81b8df81 100644

[ ... ]

drivers/gpu/drm/ttm/ttm_pool.c static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, · line 1190

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_pool.c +1 -0
@@ -1121,10 +1187,23 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		gfp_flags |= GFP_HIGHUSER;
 
 	it.gfp_flags = gfp_flags;
+	it.defrag_capped = it.defrag_old_tt && alloc->defrag_bytes_remaining > 0;

[Severity: Medium]
Does this initialization correctly handle negative budget values?

If a caller uses the same ttm_operation_ctx for multiple defragmentation moves, the first move that overshoots the budget can leave defrag_bytes_remaining as a negative value.

When this negative value is passed to the next __ttm_pool_alloc() call, the alloc->defrag_bytes_remaining > 0 check evaluates to false.

Will this disable the cap entirely (defrag_capped = false) and cause the allocator to perform an unlimited upgrade, potentially reintroducing the unbounded stall this budget was designed to prevent? Should this condition be != 0 instead to properly enforce negative or exhausted budgets?

Quoted 3 lines
drivers/gpu/drm/ttm/ttm_pool.c +0 -0
 	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
#26 lore raw

Re: [PATCH v3 08/33] drm/ttm: Bound page (re)allocation per defragmentation move

In reply to Matthew Brost
Quoted 5 lines · Matthew Brost

A defragmentation move reallocates an object's backing at the device's beneficial order and harvests whatever the allocator cannot upgrade from the old tt. For a very large object this can reallocate (and have the driver re-copy) the entire backing in one move, which is expensive and can stall concurrent work for an unbounded stretch.

Assisted-by LLM

NIT...So, a time budge would also be helpful here along with the size budge?

Quoted 86 lines

Add an optional per-move byte budget so a large object can be upgraded in slices across successive moves. Plumb a new ttm_operation_ctx::defrag_bytes_remaining through to the pool allocator:
only pages actually (re)allocated at the beneficial order are charged against it; pages harvested (borrowed) from defrag_old_tt are free and do not count. Once the budget is exhausted the allocator stops upgrading and completes the new tt by harvesting the remainder of the old tt as-is, marking the tt sub-optimal (TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED) so the driver re-queues it and a later move resumes the upgrade. The slices already upgraded come back as free already-beneficial harvests on the next pass, so each move spends budget only on new forward progress.

The allocator decrements the budget as it allocates and writes the unused remainder back on success, letting the caller compute how many bytes were actually spent. A value of 0 means unlimited (upgrade the whole object in one move), preserving the previous behaviour. This is only consulted on a defragmentation move, i.e. when defrag_old_tt is set.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---
drivers/gpu/drm/ttm/ttm_pool.c | 101 ++++++++++++++++++++++++++++++++-
include/drm/ttm/ttm_bo.h | 14 +++++

2 files changed, 112 insertions(+), 3 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +12 -1
index 52f5e64f05dd..1673e81b8df8 100644
@@ -77,6 +77,11 @@ struct ttm_pool_dma {
 * @dma_addr: Pointer to the next tt dma_address entry to populate if any.
 * @remaining_pages: Remaining pages to populate.
 * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @defrag_bytes_remaining: Byte budget for newly (re)allocated pages on a
+ * defragmentation move, seeded from ttm_operation_ctx::defrag_bytes_remaining.
+ * Pages harvested (borrowed) from the old tt are free and do not count; once
+ * this drops to zero the allocator stops upgrading and harvests the remainder.
+ * On completion it holds the unused budget. 0 means unlimited (no defrag cap).
 */
 struct ttm_pool_alloc_state {
 	struct page **pages;
@@ -84,6 +89,7 @@ struct ttm_pool_alloc_state {
 	dma_addr_t *dma_addr;
 	pgoff_t remaining_pages;
 	enum ttm_caching tt_caching;
+	s64 defrag_bytes_remaining;
 };
 
 /**
@@ -780,6 +786,7 @@ static void ttm_pool_alloc_state_init(const struct ttm_tt *tt,
 	alloc->dma_addr = tt->dma_address;
 	alloc->remaining_pages = tt->num_pages;
 	alloc->tt_caching = tt->caching;
+	alloc->defrag_bytes_remaining = 0;
 }
 
 /*
@@ -885,8 +892,12 @@ static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
 
 static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 				      struct ttm_tt *old_tt, pgoff_t off,
-				      struct ttm_pool_alloc_state *alloc)
+				      struct ttm_pool_alloc_state *alloc,
+				      bool *suboptimal)
 {
+	unsigned int beneficial = ttm_pool_beneficial_order(pool);
+	pgoff_t bnr = 1UL << beneficial;
+
 	/*
 	 * @off is always an old-tt chunk head here, never a tail page, so
 	 * reading the per-chunk order below is safe. This holds because:

drivers/gpu/drm/ttm/ttm_pool.c static int ttm_pool_harvest_remaining(struct ttm_pool *pool, · line 920

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_pool.c +1 -0
@@ -906,8 +917,23 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 		struct page *p = old_tt->pages[off];
 		unsigned int order = ttm_pool_page_order(pool, p);
 		pgoff_t nr = 1UL << order;
+		pgoff_t num_pages = off + alloc->remaining_pages;

num_pages seems loop invariant. May be move it out of the loop as total_pages or something?

Quoted 73 lines
drivers/gpu/drm/ttm/ttm_pool.c +53 -0
 		int r;
 
+		/*
+		 * Only report a sub-optimal backing when a chunk we cannot
+		 * upgrade (out of budget/prealloc) sits in a beneficial-order
+		 * aligned region that fully fits in the tt: that region could
+		 * have been upgraded with more budget. A chunk below the
+		 * beneficial order is expected for the unaligned tail (the last
+		 * partial beneficial-order region) and for already beneficial-
+		 * order harvests - neither leaves real defrag work, so they must
+		 * not flag the tt for re-queue.
+		 */
+		if (beneficial && order < beneficial &&
+		    round_down(off, bnr) + bnr <= num_pages)
+			*suboptimal = true;
+
 		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
 		if (r)
 			return r;
@@ -951,6 +977,7 @@ struct ttm_pool_alloc_iter {
 	struct ttm_tt *defrag_old_tt;
 	unsigned int beneficial_order;
 	gfp_t gfp_flags;
+	bool defrag_capped;
 
 	/* Mutated as the loop walks the orders. */
 	unsigned int order;
@@ -962,6 +989,44 @@ struct ttm_pool_alloc_iter {
 	struct page *p;
 };
 
+/*
+ * Defrag move budget exhausted: the upgrade can make no further progress this
+ * pass. Snapshot @defrag_capped is set only when a byte budget was in force at
+ * entry.
+ */
+static bool ttm_pool_iter_defrag_exhausted(const struct ttm_pool_alloc_iter *it)
+{
+	if (!it->defrag_old_tt)
+		return false;
+
+	return it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0;
+}
+
+/*
+ * Complete the new tt by harvesting the rest of the old tt as-is (borrowing its
+ * pages, including any still below beneficial order). Never fall back to in-lock
+ * reclaim for the shortfall - that would reintroduce the stall we preallocated
+ * to avoid. Only re-queue (flag) the tt when the harvested remainder is
+ * genuinely sub-optimal; if every remaining old chunk was already at beneficial
+ * order the move fully upgraded the object.
+ */
+static int ttm_pool_iter_harvest_rest(struct ttm_pool_alloc_iter *it)
+{
+	pgoff_t off = it->tt->num_pages - it->alloc->remaining_pages;
+	bool suboptimal = false;
+	int r;
+
+	r = ttm_pool_harvest_remaining(it->pool, it->defrag_old_tt, off,
+				       it->alloc, &suboptimal);
+	if (r)
+		return r;
+
+	if (suboptimal)
+		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+
+	return 0;
+}
+
 /*
 * Defrag move: reuse an already beneficial-order chunk from the old tt at this
 * offset rather than reallocating it. Harvest whatever order the old chunk has

drivers/gpu/drm/ttm/ttm_pool.c static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it) · line 1135

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +2 -1
@@ -1062,11 +1127,12 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 		if (it->alloc_count && it->defrag_old_tt) {
 			pgoff_t off = it->tt->num_pages -
 				      it->alloc->remaining_pages;
+			bool suboptimal = false;
 			int r;
 
 			r = ttm_pool_harvest_remaining(it->pool,
 						       it->defrag_old_tt, off,
-						       it->alloc);
+						       it->alloc, &suboptimal);

As suboptimal is not required here, probably send a NULL here and have ttm_pool_harvest_remaining handle that NULL.

Quoted 78 lines
drivers/gpu/drm/ttm/ttm_pool.c +30 -1
 			if (r)
 				return r;
 
@@ -1121,10 +1187,23 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		gfp_flags |= GFP_HIGHUSER;
 
 	it.gfp_flags = gfp_flags;
+	it.defrag_capped = it.defrag_old_tt && alloc->defrag_bytes_remaining > 0;
 
 	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
+		/*
+		 * Out of defrag budget: harvest the rest of the old tt as-is and
+		 * stop (the tt is re-queued if the remainder is still
+		 * sub-optimal).
+		 */
+		if (ttm_pool_iter_defrag_exhausted(&it)) {
+			r = ttm_pool_iter_harvest_rest(&it);
+			if (r)
+				goto error_free_all;
+			break;
+		}
+
 		/* Reuse an already beneficial-order chunk from the old tt. */
 		r = ttm_pool_iter_reuse_old(&it);
 		if (r < 0)
@@ -1155,6 +1234,15 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 				goto error_free_all;
 		}
 
+		/*
+		 * Charge a newly (re)allocated chunk against the defrag move
+		 * budget. Harvested chunks borrow the old tt's pages for free
+		 * and never reach here.
+		 */
+		if (it.defrag_capped)
+			alloc->defrag_bytes_remaining -=
+				(s64)(1UL << it.order) << PAGE_SHIFT;
+
 		it.alloc_count++;
 	}
 
@@ -1204,14 +1292,21 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		   struct ttm_operation_ctx *ctx)
 {
 	struct ttm_pool_alloc_state alloc;
+	int ret;
 
 	if (WARN_ON(ttm_tt_is_backed_up(tt)))
 		return -EINVAL;
 
 	tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 	ttm_pool_alloc_state_init(tt, &alloc);
+	alloc.defrag_bytes_remaining = ctx->defrag_bytes_remaining;
+
+	ret = __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
+
+	/* Report the unused defrag budget back to the caller. */
+	ctx->defrag_bytes_remaining = alloc.defrag_bytes_remaining;
 
-	return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
+	return ret;
 }
 EXPORT_SYMBOL(ttm_pool_alloc);
 
include/drm/ttm/ttm_bo.h +4 -0
index fcaba5fb8bf7..6124d2627b47 100644
@@ -223,6 +223,20 @@ struct ttm_operation_ctx {
 	 * NULL for non-defrag operations.
 	 */
 	struct ttm_tt *defrag_old_tt;
+	/**
+	 * @defrag_bytes_remaining: Byte budget for newly allocated pages during
+	 * a defragmentation move. Only pages that are actually (re)allocated at
+	 * the beneficial order count against it; pages harvested (borrowed) from

include/drm/ttm/ttm_bo.h struct ttm_operation_ctx { · line 239

Quoted 10 lines
include/drm/ttm/ttm_bo.h +10 -0
+	 * @defrag_old_tt are free and do not. When the budget is exhausted the
+	 * pool allocator stops upgrading and completes the new tt by harvesting
+	 * the remainder of @defrag_old_tt, marking the tt sub-optimal so the
+	 * driver can resume on a later pass. The allocator decrements this as it
+	 * allocates, so on return it holds the unused budget (which the caller
+	 * can use to compute how many bytes were spent). A value of 0 means
+	 * unlimited (upgrade the whole object). Only consulted when
+	 * @defrag_old_tt is set.
+	 */
+	s64 defrag_bytes_remaining;

If user specifies -ve value, currently it means unlimited. So, may be update the documentation here to say "A value of <= 0 means unlimited"? Or, better only accept

Quoted 1 line

=0 value from the user ctx here.

Niranjana

Quoted 6 lines

/**
* @resv: Reservation object to be used together with
* @allow_res_evict.

Signature · 1 line
2.34.1
author maintainer
#27 lore raw

Re: [PATCH v3 08/33] drm/ttm: Bound page (re)allocation per defragmentation move

Quoted 11 lines · Niranjana Vishwanathapura
Quoted 5 lines · Matthew Brost

A defragmentation move reallocates an object's backing at the device's beneficial order and harvests whatever the allocator cannot upgrade from the old tt. For a very large object this can reallocate (and have the driver re-copy) the entire backing in one move, which is expensive and can stall concurrent work for an unbounded stretch.

Assisted-by LLM

NIT...So, a time budge would also be helpful here along with the size budge?

That could be option but a size budget seems like reasonable place to start.

Quoted 86 lines
Quoted 86 lines

Add an optional per-move byte budget so a large object can be upgraded in slices across successive moves. Plumb a new ttm_operation_ctx::defrag_bytes_remaining through to the pool allocator:
only pages actually (re)allocated at the beneficial order are charged against it; pages harvested (borrowed) from defrag_old_tt are free and do not count. Once the budget is exhausted the allocator stops upgrading and completes the new tt by harvesting the remainder of the old tt as-is, marking the tt sub-optimal (TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED) so the driver re-queues it and a later move resumes the upgrade. The slices already upgraded come back as free already-beneficial harvests on the next pass, so each move spends budget only on new forward progress.

The allocator decrements the budget as it allocates and writes the unused remainder back on success, letting the caller compute how many bytes were actually spent. A value of 0 means unlimited (upgrade the whole object in one move), preserving the previous behaviour. This is only consulted on a defragmentation move, i.e. when defrag_old_tt is set.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---
drivers/gpu/drm/ttm/ttm_pool.c | 101 ++++++++++++++++++++++++++++++++-
include/drm/ttm/ttm_bo.h | 14 +++++

2 files changed, 112 insertions(+), 3 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +12 -1
index 52f5e64f05dd..1673e81b8df8 100644
@@ -77,6 +77,11 @@ struct ttm_pool_dma {
 * @dma_addr: Pointer to the next tt dma_address entry to populate if any.
 * @remaining_pages: Remaining pages to populate.
 * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @defrag_bytes_remaining: Byte budget for newly (re)allocated pages on a
+ * defragmentation move, seeded from ttm_operation_ctx::defrag_bytes_remaining.
+ * Pages harvested (borrowed) from the old tt are free and do not count; once
+ * this drops to zero the allocator stops upgrading and harvests the remainder.
+ * On completion it holds the unused budget. 0 means unlimited (no defrag cap).
 */
 struct ttm_pool_alloc_state {
 	struct page **pages;
@@ -84,6 +89,7 @@ struct ttm_pool_alloc_state {
 	dma_addr_t *dma_addr;
 	pgoff_t remaining_pages;
 	enum ttm_caching tt_caching;
+	s64 defrag_bytes_remaining;
 };
 
 /**
@@ -780,6 +786,7 @@ static void ttm_pool_alloc_state_init(const struct ttm_tt *tt,
 	alloc->dma_addr = tt->dma_address;
 	alloc->remaining_pages = tt->num_pages;
 	alloc->tt_caching = tt->caching;
+	alloc->defrag_bytes_remaining = 0;
 }
 
 /*
@@ -885,8 +892,12 @@ static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
 
 static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 				      struct ttm_tt *old_tt, pgoff_t off,
-				      struct ttm_pool_alloc_state *alloc)
+				      struct ttm_pool_alloc_state *alloc,
+				      bool *suboptimal)
 {
+	unsigned int beneficial = ttm_pool_beneficial_order(pool);
+	pgoff_t bnr = 1UL << beneficial;
+
 	/*
 	 * @off is always an old-tt chunk head here, never a tail page, so
 	 * reading the per-chunk order below is safe. This holds because:
Quoted 9 lines

drivers/gpu/drm/ttm/ttm_pool.c static int ttm_pool_harvest_remaining(struct ttm_pool *pool, · line 920

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_pool.c +1 -0
@@ -906,8 +917,23 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 		struct page *p = old_tt->pages[off];
 		unsigned int order = ttm_pool_page_order(pool, p);
 		pgoff_t nr = 1UL << order;
+		pgoff_t num_pages = off + alloc->remaining_pages;

num_pages seems loop invariant. May be move it out of the loop as total_pages or something?

No, both 'off' and 'alloc->remaining_pages' change each iteration of the loop.

Quoted 11 lines
Quoted 11 lines
drivers/gpu/drm/ttm/ttm_pool.c +9 -0
 		int r;
 
+		/*
+		 * Only report a sub-optimal backing when a chunk we cannot
+		 * upgrade (out of budget/prealloc) sits in a beneficial-order
+		 * aligned region that fully fits in the tt: that region could
+		 * have been upgraded with more budget. A chunk below the
+		 * beneficial order is expected for the unaligned tail (the last
+		 * partial beneficial-order region) and for already beneficial-
+		 * order harvests - neither leaves real defrag work, so they must
+		 * not flag the tt for re-queue.
Quoted 8 lines

drivers/gpu/drm/ttm/ttm_pool.c static int ttm_pool_harvest_remaining(struct ttm_pool *pool, · line 939

Quoted 8 lines
drivers/gpu/drm/ttm/ttm_pool.c +5 -0
+		 */
+		if (beneficial && order < beneficial &&
+		    round_down(off, bnr) + bnr <= num_pages)
+			*suboptimal = true;
+
 		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
 		if (r)
 			return r;

e.g., below here (unseen) 'off' is incremented by 'nr'.

Quoted 60 lines
Quoted 60 lines
drivers/gpu/drm/ttm/ttm_pool.c +40 -0
@@ -951,6 +977,7 @@ struct ttm_pool_alloc_iter {
 	struct ttm_tt *defrag_old_tt;
 	unsigned int beneficial_order;
 	gfp_t gfp_flags;
+	bool defrag_capped;
 
 	/* Mutated as the loop walks the orders. */
 	unsigned int order;
@@ -962,6 +989,44 @@ struct ttm_pool_alloc_iter {
 	struct page *p;
 };
 
+/*
+ * Defrag move budget exhausted: the upgrade can make no further progress this
+ * pass. Snapshot @defrag_capped is set only when a byte budget was in force at
+ * entry.
+ */
+static bool ttm_pool_iter_defrag_exhausted(const struct ttm_pool_alloc_iter *it)
+{
+	if (!it->defrag_old_tt)
+		return false;
+
+	return it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0;
+}
+
+/*
+ * Complete the new tt by harvesting the rest of the old tt as-is (borrowing its
+ * pages, including any still below beneficial order). Never fall back to in-lock
+ * reclaim for the shortfall - that would reintroduce the stall we preallocated
+ * to avoid. Only re-queue (flag) the tt when the harvested remainder is
+ * genuinely sub-optimal; if every remaining old chunk was already at beneficial
+ * order the move fully upgraded the object.
+ */
+static int ttm_pool_iter_harvest_rest(struct ttm_pool_alloc_iter *it)
+{
+	pgoff_t off = it->tt->num_pages - it->alloc->remaining_pages;
+	bool suboptimal = false;
+	int r;
+
+	r = ttm_pool_harvest_remaining(it->pool, it->defrag_old_tt, off,
+				       it->alloc, &suboptimal);
+	if (r)
+		return r;
+
+	if (suboptimal)
+		it->tt->page_flags |= TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+
+	return 0;
+}
+
 /*
 * Defrag move: reuse an already beneficial-order chunk from the old tt at this
 * offset rather than reallocating it. Harvest whatever order the old chunk has
@@ -1062,11 +1127,12 @@ static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it)
 		if (it->alloc_count && it->defrag_old_tt) {
 			pgoff_t off = it->tt->num_pages -
 				      it->alloc->remaining_pages;
+			bool suboptimal = false;
 			int r;
 
Quoted 8 lines

drivers/gpu/drm/ttm/ttm_pool.c static int ttm_pool_iter_lower_order(struct ttm_pool_alloc_iter *it) · line 1135

Quoted 4 lines
drivers/gpu/drm/ttm/ttm_pool.c +1 -1
 			r = ttm_pool_harvest_remaining(it->pool,
 						       it->defrag_old_tt, off,
-						       it->alloc);
+						       it->alloc, &suboptimal);

As suboptimal is not required here, probably send a NULL here and have ttm_pool_harvest_remaining handle that NULL.

That is better will change.

Quoted 83 lines
Quoted 83 lines
drivers/gpu/drm/ttm/ttm_pool.c +30 -1
 			if (r)
 				return r;
 
@@ -1121,10 +1187,23 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		gfp_flags |= GFP_HIGHUSER;
 
 	it.gfp_flags = gfp_flags;
+	it.defrag_capped = it.defrag_old_tt && alloc->defrag_bytes_remaining > 0;
 
 	for (it.order = ttm_pool_alloc_find_order(MAX_PAGE_ORDER, alloc);
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
+		/*
+		 * Out of defrag budget: harvest the rest of the old tt as-is and
+		 * stop (the tt is re-queued if the remainder is still
+		 * sub-optimal).
+		 */
+		if (ttm_pool_iter_defrag_exhausted(&it)) {
+			r = ttm_pool_iter_harvest_rest(&it);
+			if (r)
+				goto error_free_all;
+			break;
+		}
+
 		/* Reuse an already beneficial-order chunk from the old tt. */
 		r = ttm_pool_iter_reuse_old(&it);
 		if (r < 0)
@@ -1155,6 +1234,15 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 				goto error_free_all;
 		}
 
+		/*
+		 * Charge a newly (re)allocated chunk against the defrag move
+		 * budget. Harvested chunks borrow the old tt's pages for free
+		 * and never reach here.
+		 */
+		if (it.defrag_capped)
+			alloc->defrag_bytes_remaining -=
+				(s64)(1UL << it.order) << PAGE_SHIFT;
+
 		it.alloc_count++;
 	}
 
@@ -1204,14 +1292,21 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		   struct ttm_operation_ctx *ctx)
 {
 	struct ttm_pool_alloc_state alloc;
+	int ret;
 
 	if (WARN_ON(ttm_tt_is_backed_up(tt)))
 		return -EINVAL;
 
 	tt->page_flags &= ~TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 	ttm_pool_alloc_state_init(tt, &alloc);
+	alloc.defrag_bytes_remaining = ctx->defrag_bytes_remaining;
+
+	ret = __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
+
+	/* Report the unused defrag budget back to the caller. */
+	ctx->defrag_bytes_remaining = alloc.defrag_bytes_remaining;
 
-	return __ttm_pool_alloc(pool, tt, ctx, &alloc, NULL);
+	return ret;
 }
 EXPORT_SYMBOL(ttm_pool_alloc);
 
include/drm/ttm/ttm_bo.h +9 -0
index fcaba5fb8bf7..6124d2627b47 100644
@@ -223,6 +223,20 @@ struct ttm_operation_ctx {
 	 * NULL for non-defrag operations.
 	 */
 	struct ttm_tt *defrag_old_tt;
+	/**
+	 * @defrag_bytes_remaining: Byte budget for newly allocated pages during
+	 * a defragmentation move. Only pages that are actually (re)allocated at
+	 * the beneficial order count against it; pages harvested (borrowed) from
+	 * @defrag_old_tt are free and do not. When the budget is exhausted the
+	 * pool allocator stops upgrading and completes the new tt by harvesting
+	 * the remainder of @defrag_old_tt, marking the tt sub-optimal so the
+	 * driver can resume on a later pass. The allocator decrements this as it
+	 * allocates, so on return it holds the unused budget (which the caller
Quoted 9 lines

include/drm/ttm/ttm_bo.h struct ttm_operation_ctx { · line 239

Quoted 5 lines
include/drm/ttm/ttm_bo.h +5 -0
+	 * can use to compute how many bytes were spent). A value of 0 means
+	 * unlimited (upgrade the whole object). Only consulted when
+	 * @defrag_old_tt is set.
+	 */
+	s64 defrag_bytes_remaining;

If user specifies -ve value, currently it means unlimited. So, may be update the documentation here to say "A value of <= 0 means unlimited"? Or, better only accept

Quoted 1 line

=0 value from the user ctx here.

How about I reject user input defrag_bytes_remaining < 0?

Matt

Quoted 9 lines

Niranjana

Quoted 6 lines

/**
* @resv: Reservation object to be used together with
* @allow_res_evict.

Signature · 1 line
2.34.1
author maintainer
#28 lore raw

PATCH v3 9/33 drm/ttm: Preallocate beneficial-order defrag pages outside the lock

In reply to Matthew Brost

Defragmentation moves reallocate a BO's backing at the pool's beneficial order. On a fragmented system those high-order allocations frequently stall in direct reclaim/compaction, and they run under the BO dma-resv lock, holding up concurrent rendering. The defrag worker does not care how long the work takes; it only cares about minimizing locked time.

Add a preallocation bag so the expensive allocations can be hoisted out of the lock:

- struct ttm_pool_prealloc carries beneficial-order pages preallocated outside the lock; ttm_pool_prealloc_fill()/_fini() allocate and release them. __ttm_pool_alloc() consumes them via ctx->prealloc.
- Apply the requested cpu-caching to the bag pages during fill, so the set_memory_*() cost is paid outside the lock too; consumed pages are already cache-consistent and leftovers are restored to write-back on free. Bag pages are never highmem so the kernel mapping is always valid for set_memory_*().
- Record the exact number of sub-optimal pages on the tt (nr_suboptimal_pages) at populate time so a defrag pass can size its prealloc precisely instead of guessing.
- Once a defrag move exhausts the prealloc bag, harvest the remainder of the old tt rather than entering reclaim under the lock, and flag the tt sub-optimal so the driver re-queues it for a later pass.

This keeps all reclaim/compaction stalls and cache mode changes out of the dma-resv critical section: lock-held time then tracks only the bytes actually copied.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 244 +++++++++++++++++++++++++++++++--
 include/drm/ttm/ttm_bo.h       |  11 ++
 include/drm/ttm/ttm_pool.h     |  29 ++++
 include/drm/ttm/ttm_tt.h       |  17 +++
 4 files changed, 287 insertions(+), 14 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +230 -14
index 1673e81b8df8..da7d132c8297 100644
@@ -82,6 +82,9 @@ struct ttm_pool_dma {
  * Pages harvested (borrowed) from the old tt are free and do not count; once
  * this drops to zero the allocator stops upgrading and harvests the remainder.
  * On completion it holds the unused budget. 0 means unlimited (no defrag cap).
+ * @nr_suboptimal_pages: Number of pages backed below the pool's beneficial
+ * order. Recorded by the pool allocator after populate; a defrag pass reads it
+ * to size its out-of-lock preallocation exactly.
  */
 struct ttm_pool_alloc_state {
 	struct page **pages;
@@ -90,6 +93,7 @@ struct ttm_pool_alloc_state {
 	pgoff_t remaining_pages;
 	enum ttm_caching tt_caching;
 	s64 defrag_bytes_remaining;
+	u32 nr_suboptimal_pages;
 };
 
 /**
@@ -787,6 +791,7 @@ static void ttm_pool_alloc_state_init(const struct ttm_tt *tt,
 	alloc->remaining_pages = tt->num_pages;
 	alloc->tt_caching = tt->caching;
 	alloc->defrag_bytes_remaining = 0;
+	alloc->nr_suboptimal_pages = 0;
 }
 
 /*
@@ -852,7 +857,8 @@ static unsigned int ttm_pool_defrag_old_order(struct ttm_pool *pool,
  * pages stay owned by the old tt until the move commits.
  */
 static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
-				 pgoff_t off, struct ttm_pool_alloc_state *alloc)
+				 unsigned int beneficial_order, pgoff_t off,
+				 struct ttm_pool_alloc_state *alloc)
 {
 	pgoff_t nr = 1UL << order;
 	int r;
@@ -887,6 +893,9 @@ static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
 
 	alloc->caching_divide = alloc->pages;
 
+	if (order < beneficial_order)
+		alloc->nr_suboptimal_pages += 0x1 << order;
+
 	return 0;
 }
 
@@ -934,7 +943,8 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 		    round_down(off, bnr) + bnr <= num_pages)
 			*suboptimal = true;
 
-		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
+		r = ttm_pool_harvest_page(old_tt, order, beneficial, off,
+					  alloc);
 		if (r)
 			return r;
 
@@ -944,6 +954,173 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 	return 0;
 }
 
+unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool)
+{
+	return ttm_pool_beneficial_order(pool);
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_order);
+
+/*
+ * Build the gfp flags used for the high-order, possibly reclaiming, beneficial
+ * order page allocations, matching the in-line defrag alloc path.
+ */
+static gfp_t ttm_pool_prealloc_gfp(struct ttm_pool *pool)
+{
+	gfp_t gfp = GFP_USER;
+
+	/*
+	 * No highmem: prealloc applies caching in bulk via set_pages_array_*()
+	 * on the kernel mapping, so the pages must be permanently mapped.
+	 */
+	if (ttm_pool_uses_dma32(pool))
+		gfp |= GFP_DMA32;
+
+	return gfp;
+}
+
+/*
+ * Apply a tt's cpu-caching to a batch of freshly system-allocated (write-back)
+ * prealloc pages in one shot. @cpages is an unpacked array of @ncpages
+ * individual 4K pages (the constituent pages of the mixed-order chunks, expanded
+ * so set_pages_array_*() sees one entry per page), since the packed prealloc bag
+ * holds multi-order chunks. Prealloc pages are never highmem (see
+ * ttm_pool_prealloc_gfp), so their kernel mapping is valid. No-op for cached
+ * pages and on non-x86. Returns non-zero if the caching change failed (the
+ * caller drops the whole bag and falls back to in-line allocation).
+ */
+static int ttm_pool_prealloc_apply_caching(enum ttm_caching caching,
+					   struct page **cpages,
+					   unsigned int ncpages)
+{
+#ifdef CONFIG_X86
+	if (!ncpages)
+		return 0;
+
+	switch (caching) {
+	case ttm_cached:
+		break;
+	case ttm_write_combined:
+		return set_pages_array_wc(cpages, ncpages);
+	case ttm_uncached:
+		return set_pages_array_uc(cpages, ncpages);
+	}
+#endif
+	return 0;
+}
+
+/* Expand an @order chunk into its constituent 4K pages for bulk caching. */
+static void ttm_pool_prealloc_stage_caching(struct page **cpages,
+					    unsigned int *ncpages,
+					    struct page *p, unsigned int order)
+{
+	unsigned int i, nr = 1u << order;
+
+	for (i = 0; i < nr; i++)
+		cpages[(*ncpages)++] = p + i;
+}
+
+/**
+ * ttm_pool_prealloc_fill() - Preallocate beneficial-order pages outside any lock
+ * @pool: The pool to allocate from.
+ * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @pp: Prealloc bag to fill; @pp->order is set to the beneficial order.
+ * @count: Number of beneficial-order chunks to attempt.
+ *
+ * Allocate up to @count beneficial-order chunks, parking them in @pp for a
+ * later __ttm_pool_alloc() defrag move to consume under the dma-resv lock. May
+ * sleep/reclaim freely as it runs unlocked. A short fill is fine: the pool
+ * falls back to in-line allocation for the shortfall. DMA-alloc pools are not
+ * supported (count stays 0). Returns 0 (release with ttm_pool_prealloc_fini()).
+ */
+int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
+			   struct ttm_pool_prealloc *pp, unsigned int count)
+{
+	unsigned int order = ttm_pool_beneficial_order(pool);
+	gfp_t gfp = ttm_pool_prealloc_gfp(pool);
+	struct page **cpages = NULL;
+	unsigned int ncpages = 0;
+	int r;
+
+	pp->pages = NULL;
+	pp->order = order;
+	pp->caching = tt_caching;
+	pp->count = 0;
+	pp->used = 0;
+
+	/* Nothing to gain without a beneficial order or for DMA-alloc pools. */
+	if (!order || !count || ttm_pool_uses_dma_alloc(pool))
+		return 0;
+
+	pp->pages = kvzalloc_objs(*pp->pages, count);
+	if (!pp->pages)
+		return 0;
+
+	/*
+	 * Every chunk is a fresh write-back system page of @order that needs a
+	 * caching change; collect their constituent pages into an unpacked
+	 * scratch array and issue a single set_pages_array_*() after the fill.
+	 * Cached tts need no change (and non-x86 handles caching at map time).
+	 */
+	if (IS_ENABLED(CONFIG_X86) && tt_caching != ttm_cached) {
+		cpages = kvzalloc_objs(*cpages, (size_t)count << order);
+		if (!cpages) {
+			kvfree(pp->pages);
+			pp->pages = NULL;
+			return 0;
+		}
+	}
+
+	while (pp->count < count) {
+		struct page *p = ttm_pool_alloc_page(pool, gfp, order, false);
+
+		if (!p)
+			break;
+
+		pp->pages[pp->count++] = p;
+		if (cpages)
+			ttm_pool_prealloc_stage_caching(cpages, &ncpages, p,
+							order);
+	}
+
+	/*
+	 * Apply the requested caching to every collected page in one shot. On
+	 * failure the pages' caching is indeterminate, so drop the whole bag
+	 * (freeing restores write-back) and let the consumer allocate in-line.
+	 */
+	r = ttm_pool_prealloc_apply_caching(tt_caching, cpages, ncpages);
+	kvfree(cpages);
+	if (r) {
+		unsigned int i;
+
+		for (i = 0; i < pp->count; i++)
+			ttm_pool_free_page(pool, tt_caching, order,
+					   pp->pages[i], false);
+		pp->count = 0;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fill);
+
+/**
+ * ttm_pool_prealloc_fini() - Release unconsumed preallocated pages
+ * @pool: The pool the pages came from.
+ * @pp: Prealloc bag to drain. Consumed pages (< @used) are now owned by the tt.
+ */
+void ttm_pool_prealloc_fini(struct ttm_pool *pool, struct ttm_pool_prealloc *pp)
+{
+	unsigned int i;
+
+	for (i = pp->used; i < pp->count; ++i)
+		ttm_pool_free_page(pool, pp->caching, pp->order, pp->pages[i],
+				   false);
+	kvfree(pp->pages);
+	pp->pages = NULL;
+	pp->count = 0;
+	pp->used = 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fini);
+
 /**
  * enum ttm_pool_iter_action - Outcome of a per-order allocation phase
  * @TTM_POOL_ITER_FILL: A page (@it->p) was acquired; proceed to fill it in.
@@ -975,6 +1152,7 @@ struct ttm_pool_alloc_iter {
 	struct ttm_pool_alloc_state *alloc;
 	struct ttm_pool_tt_restore *restore;
 	struct ttm_tt *defrag_old_tt;
+	struct ttm_pool_prealloc *prealloc;
 	unsigned int beneficial_order;
 	gfp_t gfp_flags;
 	bool defrag_capped;
@@ -990,16 +1168,20 @@ struct ttm_pool_alloc_iter {
 };
 
 /*
- * Defrag move budget exhausted: the upgrade can make no further progress this
- * pass. Snapshot @defrag_capped is set only when a byte budget was in force at
- * entry.
+ * Defrag move budget exhausted, or the out-of-lock prealloc bag ran dry: the
+ * upgrade can make no further progress this pass. Snapshot @defrag_capped is
+ * set only when a byte budget was in force at entry.
  */
 static bool ttm_pool_iter_defrag_exhausted(const struct ttm_pool_alloc_iter *it)
 {
+	const struct ttm_pool_prealloc *pp = it->prealloc;
+
 	if (!it->defrag_old_tt)
 		return false;
+	if (it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0)
+		return true;
 
-	return it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0;
+	return pp && pp->count && pp->used >= pp->count;
 }
 
 /*
@@ -1056,7 +1238,8 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 		return TTM_POOL_ITER_FILL;
 
 	it->order = harvest_order;
-	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order, off, it->alloc);
+	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order,
+				  it->beneficial_order, off, it->alloc);
 	if (r)
 		return r;
 
@@ -1068,11 +1251,13 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 
 /*
  * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries a same-order pool page, then a fresh system allocation. Fault
- * injection can force the beneficial-order paths to "fail".
+ * failure). Tries, in order: a beneficial-order page preallocated outside the
+ * dma-resv lock (defrag), a same-order pool page, then a fresh system
+ * allocation. Fault injection can force the beneficial-order paths to "fail".
  */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
+	struct ttm_pool_prealloc *pp = it->prealloc;
 	struct ttm_pool_type *pt;
 
 	it->p = NULL;
@@ -1087,7 +1272,18 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 		it->beneficial_order && it->order >= it->beneficial_order &&
 		should_fail(&beneficial_order_fault_inject, 1);
 
-	/* First, try to allocate a page from a pool if one exists. */
+	/*
+	 * Defrag move: consume a beneficial-order page preallocated outside the
+	 * dma-resv lock instead of allocating one in-line (which may stall in
+	 * reclaim/compaction). Falls through once the bag is empty.
+	 */
+	if (pp && !it->fail_beneficial && it->order >= it->beneficial_order &&
+	    pp->used < pp->count) {
+		it->order = it->beneficial_order;
+		it->p = pp->pages[pp->used++];
+		it->page_caching = it->tt->caching;
+	}
+
 	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
 	if (!it->p && pt && it->allow_pools && !it->fail_beneficial)
 		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
@@ -1164,6 +1360,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		.alloc = alloc,
 		.restore = restore,
 		.defrag_old_tt = restore ? NULL : ctx->defrag_old_tt,
+		.prealloc = restore ? NULL : ctx->prealloc,
 		.beneficial_order = ttm_pool_beneficial_order(pool),
 		.page_caching = tt->caching,
 		.allow_pools = true,
@@ -1193,8 +1390,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
 		/*
-		 * Out of defrag budget: harvest the rest of the old tt as-is and
-		 * stop (the tt is re-queued if the remainder is still
+		 * Out of defrag budget/prealloc: harvest the rest of the old tt
+		 * as-is and stop (the tt is re-queued if the remainder is still
 		 * sub-optimal).
 		 */
 		if (ttm_pool_iter_defrag_exhausted(&it)) {
@@ -1211,7 +1408,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		if (r == TTM_POOL_ITER_RETRY)
 			continue;
 
-		/* Acquire a page (pool / system) for this order. */
+		/* Acquire a page (prealloc / pool / system) for this order. */
 		ttm_pool_iter_acquire_page(&it);
 		if (!it.p) {
 			r = ttm_pool_iter_lower_order(&it);
@@ -1243,6 +1440,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			alloc->defrag_bytes_remaining -=
 				(s64)(1UL << it.order) << PAGE_SHIFT;
 
+		if (it.order < it.beneficial_order)
+			alloc->nr_suboptimal_pages += 0x1 << it.order;
 		it.alloc_count++;
 	}
 
@@ -1306,6 +1505,9 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	/* Report the unused defrag budget back to the caller. */
 	ctx->defrag_bytes_remaining = alloc.defrag_bytes_remaining;
 
+	if (!ret)
+		tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
+
 	return ret;
 }
 EXPORT_SYMBOL(ttm_pool_alloc);
@@ -1358,6 +1560,15 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 
 			if (ret)
 				return ret;
+
+			/*
+			 * __ttm_pool_alloc() counts each freshly (re)allocated
+			 * chunk against nr_suboptimal_pages, but a chunk whose
+			 * backup copy was interrupted and is finished here
+			 * resumes past that loop, so account for it directly.
+			 */
+			if (restore->order < ttm_pool_beneficial_order(pool))
+				alloc.nr_suboptimal_pages += 1U << restore->order;
 		}
 		if (!alloc.remaining_pages) {
 			ret = ttm_pool_apply_caching(&alloc);
@@ -1367,11 +1578,16 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			kfree(tt->restore);
 			tt->restore = NULL;
 
+			tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
 			return 0;
 		}
 	}
 
-	return __ttm_pool_alloc(pool, tt, ctx, &alloc, restore);
+	ret = __ttm_pool_alloc(pool, tt, ctx, &alloc, restore);
+	if (!ret)
+		tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
+
+	return ret;
 }
 
 /**
include/drm/ttm/ttm_bo.h +11 -0
index 6124d2627b47..a80304f179ba 100644
@@ -237,6 +237,17 @@ struct ttm_operation_ctx {
 	 * @defrag_old_tt is set.
 	 */
 	s64 defrag_bytes_remaining;
+	/**
+	 * @prealloc: Pages preallocated outside the dma-resv lock for a defrag
+	 * move. The pool allocator consumes these instead of allocating fresh
+	 * beneficial-order pages under the lock, moving the (potentially
+	 * reclaim/compaction stalling) high-order allocations out of the
+	 * critical section. NULL means allocate in-line as usual. The pool
+	 * silently falls back to in-line allocation for any shortfall. The
+	 * allocator consumes it for beneficial-order chunks of any populate;
+	 * drivers only set it for a defrag move.
+	 */
+	struct ttm_pool_prealloc *prealloc;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
include/drm/ttm/ttm_pool.h +29 -0
index 753203980e2c..71670350eb15 100644
@@ -81,6 +81,35 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		   struct ttm_operation_ctx *ctx);
 void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt);
 
+/**
+ * struct ttm_pool_prealloc - Pages preallocated outside the dma-resv lock
+ * @pages: Array of @count beneficial-order pages (or fewer if a fill fell
+ *         short); each entry is the head page of a 1 << @order chunk.
+ * @order: Page order of every preallocated chunk.
+ * @caching: CPU caching applied to the pages, so leftovers can be restored
+ *           to write-back before being freed.
+ * @count: Number of valid entries in @pages.
+ * @used: Number of entries already consumed by the pool allocator.
+ *
+ * Defrag pages are interchangeable, so only a count of beneficial-order chunks
+ * is needed. ttm_pool_prealloc_fill() populates this outside the lock and
+ * __ttm_pool_alloc() drains it; any unused tail is released by
+ * ttm_pool_prealloc_fini().
+ */
+struct ttm_pool_prealloc {
+	struct page **pages;
+	unsigned int order;
+	enum ttm_caching caching;
+	unsigned int count;
+	unsigned int used;
+};
+
+int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
+			   struct ttm_pool_prealloc *pp, unsigned int count);
+void ttm_pool_prealloc_fini(struct ttm_pool *pool,
+			    struct ttm_pool_prealloc *pp);
+unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool);
+
 void ttm_pool_init(struct ttm_pool *pool, struct device *dev,
 		   int nid, unsigned int alloc_flags);
 void ttm_pool_fini(struct ttm_pool *pool);
include/drm/ttm/ttm_tt.h +17 -0
index 55c35bcf134d..81a5446ca693 100644
@@ -135,6 +135,12 @@ struct ttm_tt {
 	enum ttm_caching caching;
 	/** @restore: Partial restoration from backup state. TTM private */
 	struct ttm_pool_tt_restore *restore;
+	/**
+	 * @nr_suboptimal_pages: Number of pages backed below the pool's
+	 * beneficial order. Recorded by the pool allocator after populate; a
+	 * defrag pass reads it to size its out-of-lock preallocation exactly.
+	 */
+	u32 nr_suboptimal_pages;
 };
 
 /**
@@ -201,6 +207,17 @@ static inline bool ttm_tt_is_beneficial_order_failed(const struct ttm_tt *tt)
 	return tt->page_flags & TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 }
 
+/**
+ * ttm_tt_suboptimal_pages() - Pages backed below the pool's beneficial order
+ * @tt: The struct ttm_tt.
+ *
+ * Return: Number of pages a defrag move would (re)allocate at beneficial order.
+ */
+static inline u32 ttm_tt_suboptimal_pages(const struct ttm_tt *tt)
+{
+	return tt->nr_suboptimal_pages;
+}
+
 /**
  * ttm_tt_create
  *
Signature · 1 line
2.34.1
#29 lore raw

Re: [PATCH v3 09/33] drm/ttm: Preallocate beneficial-order defrag pages outside the lock

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] An empty preallocation bag causes a defrag move to perform all allocations in-line under the…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] An empty preallocation bag causes a defrag move to perform all allocations in-line under the dma-resv lock, causing severe reclaim/compaction stalls and defeating the design goal. --

--- Patch [9]: [PATCH v3 09/33] drm/ttm: Preallocate beneficial-order defrag pages outside the lock --- commit a99093b0428c713eac3edfaabcb993852aeabdbd
Author: Matthew Brost <[email protected]>

drm/ttm: Preallocate beneficial-order defrag pages outside the lock

This commit introduces a preallocation bag for TTM pool allocations to hoist expensive high-order allocations out of the dma-resv critical section. It records suboptimal pages during populate so a defragmentation pass can appropriately size its out-of-lock preallocation, and harvests the remainder of the old tt if the preallocation runs out.

Quoted 3 lines
drivers/gpu/drm/ttm/ttm_pool.c +0 -0

[ ... ]

Quoted 12 lines
drivers/gpu/drm/ttm/ttm_pool.c +3 -3
@@ -990,16 +1168,20 @@ struct ttm_pool_alloc_iter {
 };
 
 /*
- * Defrag move budget exhausted: the upgrade can make no further progress this
- * pass. Snapshot @defrag_capped is set only when a byte budget was in force at
- * entry.
+ * Defrag move budget exhausted, or the out-of-lock prealloc bag ran dry: the
+ * upgrade can make no further progress this pass. Snapshot @defrag_capped is
+ * set only when a byte budget was in force at entry.
  */
 static bool ttm_pool_iter_defrag_exhausted(const struct ttm_pool_alloc_iter *it)

drivers/gpu/drm/ttm/ttm_pool.c struct ttm_pool_alloc_iter { · line 1184

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +5 -1
 {
+	const struct ttm_pool_prealloc *pp = it->prealloc;
+
 	if (!it->defrag_old_tt)
 		return false;
+	if (it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0)
+		return true;
 
-	return it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0;
+	return pp && pp->count && pp->used >= pp->count;

[Severity: High]
If the out-of-lock preallocation fails to allocate any pages due to memory pressure, pp->count will be 0. Does the && pp->count condition cause this function to return false in that scenario?

If this returns false for an empty preallocation bag, __ttm_pool_alloc() will bypass the exhaustion check. The allocator would then proceed with in-line system page allocations while holding the dma-resv lock.

Because an empty bag is most likely on a fragmented system, these in-line allocations could stall aggressively in direct reclaim and compaction, which seems to bypass the goal of hoisting these allocations out of the critical section.

Could this check be updated to return pp && pp->used >= pp->count; so that an empty bag correctly aborts the defrag move?

Quoted 3 lines
drivers/gpu/drm/ttm/ttm_pool.c +0 -0
 }
 
 /*
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
#30 lore raw

Re: [PATCH v3 09/33] drm/ttm: Preallocate beneficial-order defrag pages outside the lock

In reply to Matthew Brost
Quoted 213 lines · Matthew Brost

Defragmentation moves reallocate a BO's backing at the pool's beneficial order. On a fragmented system those high-order allocations frequently stall in direct reclaim/compaction, and they run under the BO dma-resv lock, holding up concurrent rendering. The defrag worker does not care how long the work takes; it only cares about minimizing locked time.

Add a preallocation bag so the expensive allocations can be hoisted out of the lock:

- struct ttm_pool_prealloc carries beneficial-order pages preallocated outside the lock; ttm_pool_prealloc_fill()/_fini() allocate and release them. __ttm_pool_alloc() consumes them via ctx->prealloc.
- Apply the requested cpu-caching to the bag pages during fill, so the set_memory_*() cost is paid outside the lock too; consumed pages are already cache-consistent and leftovers are restored to write-back on free. Bag pages are never highmem so the kernel mapping is always valid for set_memory_*().
- Record the exact number of sub-optimal pages on the tt (nr_suboptimal_pages) at populate time so a defrag pass can size its prealloc precisely instead of guessing.
- Once a defrag move exhausts the prealloc bag, harvest the remainder of the old tt rather than entering reclaim under the lock, and flag the tt sub-optimal so the driver re-queues it for a later pass.

This keeps all reclaim/compaction stalls and cache mode changes out of the dma-resv critical section: lock-held time then tracks only the bytes actually copied.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---

drivers/gpu/drm/ttm/ttm_pool.c | 244 +++++++++++++++++++++++++++++++--
include/drm/ttm/ttm_bo.h       |  11 ++
include/drm/ttm/ttm_pool.h     |  29 ++++
include/drm/ttm/ttm_tt.h       |  17 +++
4 files changed, 287 insertions(+), 14 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +111 -2
index 1673e81b8df8..da7d132c8297 100644
@@ -82,6 +82,9 @@ struct ttm_pool_dma {
 * Pages harvested (borrowed) from the old tt are free and do not count; once
 * this drops to zero the allocator stops upgrading and harvests the remainder.
 * On completion it holds the unused budget. 0 means unlimited (no defrag cap).
+ * @nr_suboptimal_pages: Number of pages backed below the pool's beneficial
+ * order. Recorded by the pool allocator after populate; a defrag pass reads it
+ * to size its out-of-lock preallocation exactly.
 */
 struct ttm_pool_alloc_state {
 	struct page **pages;
@@ -90,6 +93,7 @@ struct ttm_pool_alloc_state {
 	pgoff_t remaining_pages;
 	enum ttm_caching tt_caching;
 	s64 defrag_bytes_remaining;
+	u32 nr_suboptimal_pages;
 };
 
 /**
@@ -787,6 +791,7 @@ static void ttm_pool_alloc_state_init(const struct ttm_tt *tt,
 	alloc->remaining_pages = tt->num_pages;
 	alloc->tt_caching = tt->caching;
 	alloc->defrag_bytes_remaining = 0;
+	alloc->nr_suboptimal_pages = 0;
 }
 
 /*
@@ -852,7 +857,8 @@ static unsigned int ttm_pool_defrag_old_order(struct ttm_pool *pool,
 * pages stay owned by the old tt until the move commits.
 */
 static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
-				 pgoff_t off, struct ttm_pool_alloc_state *alloc)
+				 unsigned int beneficial_order, pgoff_t off,
+				 struct ttm_pool_alloc_state *alloc)
 {
 	pgoff_t nr = 1UL << order;
 	int r;
@@ -887,6 +893,9 @@ static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
 
 	alloc->caching_divide = alloc->pages;
 
+	if (order < beneficial_order)
+		alloc->nr_suboptimal_pages += 0x1 << order;
+
 	return 0;
 }
 
@@ -934,7 +943,8 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 		    round_down(off, bnr) + bnr <= num_pages)
 			*suboptimal = true;
 
-		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
+		r = ttm_pool_harvest_page(old_tt, order, beneficial, off,
+					  alloc);
 		if (r)
 			return r;
 
@@ -944,6 +954,173 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 	return 0;
 }
 
+unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool)
+{
+	return ttm_pool_beneficial_order(pool);
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_order);
+
+/*
+ * Build the gfp flags used for the high-order, possibly reclaiming, beneficial
+ * order page allocations, matching the in-line defrag alloc path.
+ */
+static gfp_t ttm_pool_prealloc_gfp(struct ttm_pool *pool)
+{
+	gfp_t gfp = GFP_USER;
+
+	/*
+	 * No highmem: prealloc applies caching in bulk via set_pages_array_*()
+	 * on the kernel mapping, so the pages must be permanently mapped.
+	 */
+	if (ttm_pool_uses_dma32(pool))
+		gfp |= GFP_DMA32;
+
+	return gfp;
+}
+
+/*
+ * Apply a tt's cpu-caching to a batch of freshly system-allocated (write-back)
+ * prealloc pages in one shot. @cpages is an unpacked array of @ncpages
+ * individual 4K pages (the constituent pages of the mixed-order chunks, expanded
+ * so set_pages_array_*() sees one entry per page), since the packed prealloc bag
+ * holds multi-order chunks. Prealloc pages are never highmem (see
+ * ttm_pool_prealloc_gfp), so their kernel mapping is valid. No-op for cached
+ * pages and on non-x86. Returns non-zero if the caching change failed (the
+ * caller drops the whole bag and falls back to in-line allocation).
+ */
+static int ttm_pool_prealloc_apply_caching(enum ttm_caching caching,
+					   struct page **cpages,
+					   unsigned int ncpages)
+{
+#ifdef CONFIG_X86
+	if (!ncpages)
+		return 0;
+
+	switch (caching) {
+	case ttm_cached:
+		break;
+	case ttm_write_combined:
+		return set_pages_array_wc(cpages, ncpages);
+	case ttm_uncached:
+		return set_pages_array_uc(cpages, ncpages);
+	}
+#endif
+	return 0;
+}
+
+/* Expand an @order chunk into its constituent 4K pages for bulk caching. */
+static void ttm_pool_prealloc_stage_caching(struct page **cpages,
+					    unsigned int *ncpages,
+					    struct page *p, unsigned int order)
+{
+	unsigned int i, nr = 1u << order;
+
+	for (i = 0; i < nr; i++)
+		cpages[(*ncpages)++] = p + i;
+}
+
+/**
+ * ttm_pool_prealloc_fill() - Preallocate beneficial-order pages outside any lock
+ * @pool: The pool to allocate from.
+ * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @pp: Prealloc bag to fill; @pp->order is set to the beneficial order.
+ * @count: Number of beneficial-order chunks to attempt.
+ *
+ * Allocate up to @count beneficial-order chunks, parking them in @pp for a
+ * later __ttm_pool_alloc() defrag move to consume under the dma-resv lock. May
+ * sleep/reclaim freely as it runs unlocked. A short fill is fine: the pool
+ * falls back to in-line allocation for the shortfall. DMA-alloc pools are not
+ * supported (count stays 0). Returns 0 (release with ttm_pool_prealloc_fini()).
+ */
+int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
+			   struct ttm_pool_prealloc *pp, unsigned int count)
+{
+	unsigned int order = ttm_pool_beneficial_order(pool);
+	gfp_t gfp = ttm_pool_prealloc_gfp(pool);
+	struct page **cpages = NULL;
+	unsigned int ncpages = 0;
+	int r;
+
+	pp->pages = NULL;
+	pp->order = order;
+	pp->caching = tt_caching;
+	pp->count = 0;
+	pp->used = 0;
+
+	/* Nothing to gain without a beneficial order or for DMA-alloc pools. */
+	if (!order || !count || ttm_pool_uses_dma_alloc(pool))
+		return 0;
+
+	pp->pages = kvzalloc_objs(*pp->pages, count);
+	if (!pp->pages)

drivers/gpu/drm/ttm/ttm_pool.c static int ttm_pool_harvest_remaining(struct ttm_pool *pool, · line 1065

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+		return 0;
+
+	/*
+	 * Every chunk is a fresh write-back system page of @order that needs a
+	 * caching change; collect their constituent pages into an unpacked
+	 * scratch array and issue a single set_pages_array_*() after the fill.
+	 * Cached tts need no change (and non-x86 handles caching at map time).
+	 */
+	if (IS_ENABLED(CONFIG_X86) && tt_caching != ttm_cached) {
+		cpages = kvzalloc_objs(*cpages, (size_t)count << order);
Assisted-by LLM

NIT..cpages is tansient memory. Though it is bounded here by 'budget', may be it can be avoided altogether by using a one static 4k page in the pool and flush it with set_pages_array*() as and when it fill up?

Quoted 169 lines
drivers/gpu/drm/ttm/ttm_pool.c +89 -11
+		if (!cpages) {
+			kvfree(pp->pages);
+			pp->pages = NULL;
+			return 0;
+		}
+	}
+
+	while (pp->count < count) {
+		struct page *p = ttm_pool_alloc_page(pool, gfp, order, false);
+
+		if (!p)
+			break;
+
+		pp->pages[pp->count++] = p;
+		if (cpages)
+			ttm_pool_prealloc_stage_caching(cpages, &ncpages, p,
+							order);
+	}
+
+	/*
+	 * Apply the requested caching to every collected page in one shot. On
+	 * failure the pages' caching is indeterminate, so drop the whole bag
+	 * (freeing restores write-back) and let the consumer allocate in-line.
+	 */
+	r = ttm_pool_prealloc_apply_caching(tt_caching, cpages, ncpages);
+	kvfree(cpages);
+	if (r) {
+		unsigned int i;
+
+		for (i = 0; i < pp->count; i++)
+			ttm_pool_free_page(pool, tt_caching, order,
+					   pp->pages[i], false);
+		pp->count = 0;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fill);
+
+/**
+ * ttm_pool_prealloc_fini() - Release unconsumed preallocated pages
+ * @pool: The pool the pages came from.
+ * @pp: Prealloc bag to drain. Consumed pages (< @used) are now owned by the tt.
+ */
+void ttm_pool_prealloc_fini(struct ttm_pool *pool, struct ttm_pool_prealloc *pp)
+{
+	unsigned int i;
+
+	for (i = pp->used; i < pp->count; ++i)
+		ttm_pool_free_page(pool, pp->caching, pp->order, pp->pages[i],
+				   false);
+	kvfree(pp->pages);
+	pp->pages = NULL;
+	pp->count = 0;
+	pp->used = 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fini);
+
 /**
 * enum ttm_pool_iter_action - Outcome of a per-order allocation phase
 * @TTM_POOL_ITER_FILL: A page (@it->p) was acquired; proceed to fill it in.
@@ -975,6 +1152,7 @@ struct ttm_pool_alloc_iter {
 	struct ttm_pool_alloc_state *alloc;
 	struct ttm_pool_tt_restore *restore;
 	struct ttm_tt *defrag_old_tt;
+	struct ttm_pool_prealloc *prealloc;
 	unsigned int beneficial_order;
 	gfp_t gfp_flags;
 	bool defrag_capped;
@@ -990,16 +1168,20 @@ struct ttm_pool_alloc_iter {
 };
 
 /*
- * Defrag move budget exhausted: the upgrade can make no further progress this
- * pass. Snapshot @defrag_capped is set only when a byte budget was in force at
- * entry.
+ * Defrag move budget exhausted, or the out-of-lock prealloc bag ran dry: the
+ * upgrade can make no further progress this pass. Snapshot @defrag_capped is
+ * set only when a byte budget was in force at entry.
 */
 static bool ttm_pool_iter_defrag_exhausted(const struct ttm_pool_alloc_iter *it)
 {
+	const struct ttm_pool_prealloc *pp = it->prealloc;
+
 	if (!it->defrag_old_tt)
 		return false;
+	if (it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0)
+		return true;
 
-	return it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0;
+	return pp && pp->count && pp->used >= pp->count;
 }
 
 /*
@@ -1056,7 +1238,8 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 		return TTM_POOL_ITER_FILL;
 
 	it->order = harvest_order;
-	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order, off, it->alloc);
+	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order,
+				  it->beneficial_order, off, it->alloc);
 	if (r)
 		return r;
 
@@ -1068,11 +1251,13 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 
 /*
 * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries a same-order pool page, then a fresh system allocation. Fault
- * injection can force the beneficial-order paths to "fail".
+ * failure). Tries, in order: a beneficial-order page preallocated outside the
+ * dma-resv lock (defrag), a same-order pool page, then a fresh system
+ * allocation. Fault injection can force the beneficial-order paths to "fail".
 */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
+	struct ttm_pool_prealloc *pp = it->prealloc;
 	struct ttm_pool_type *pt;
 
 	it->p = NULL;
@@ -1087,7 +1272,18 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 		it->beneficial_order && it->order >= it->beneficial_order &&
 		should_fail(&beneficial_order_fault_inject, 1);
 
-	/* First, try to allocate a page from a pool if one exists. */
+	/*
+	 * Defrag move: consume a beneficial-order page preallocated outside the
+	 * dma-resv lock instead of allocating one in-line (which may stall in
+	 * reclaim/compaction). Falls through once the bag is empty.
+	 */
+	if (pp && !it->fail_beneficial && it->order >= it->beneficial_order &&
+	    pp->used < pp->count) {
+		it->order = it->beneficial_order;
+		it->p = pp->pages[pp->used++];
+		it->page_caching = it->tt->caching;
+	}
+
 	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
 	if (!it->p && pt && it->allow_pools && !it->fail_beneficial)
 		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
@@ -1164,6 +1360,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		.alloc = alloc,
 		.restore = restore,
 		.defrag_old_tt = restore ? NULL : ctx->defrag_old_tt,
+		.prealloc = restore ? NULL : ctx->prealloc,
 		.beneficial_order = ttm_pool_beneficial_order(pool),
 		.page_caching = tt->caching,
 		.allow_pools = true,
@@ -1193,8 +1390,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
 		/*
-		 * Out of defrag budget: harvest the rest of the old tt as-is and
-		 * stop (the tt is re-queued if the remainder is still
+		 * Out of defrag budget/prealloc: harvest the rest of the old tt
+		 * as-is and stop (the tt is re-queued if the remainder is still
 		 * sub-optimal).
 		 */
 		if (ttm_pool_iter_defrag_exhausted(&it)) {
@@ -1211,7 +1408,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		if (r == TTM_POOL_ITER_RETRY)
 			continue;
 
-		/* Acquire a page (pool / system) for this order. */
+		/* Acquire a page (prealloc / pool / system) for this order. */
 		ttm_pool_iter_acquire_page(&it);
 		if (!it.p) {
 			r = ttm_pool_iter_lower_order(&it);

drivers/gpu/drm/ttm/ttm_pool.c static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt, · line 1444

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_pool.c +2 -0
@@ -1243,6 +1440,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			alloc->defrag_bytes_remaining -=
 				(s64)(1UL << it.order) << PAGE_SHIFT;
 
+		if (it.order < it.beneficial_order)
+			alloc->nr_suboptimal_pages += 0x1 << it.order;

NIT..This counts any tail bytes also as suboptimal. Should be fine as it only leads to extra prealloc, but just noticing that TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED is not flagged for tail bytes.

Quoted 16 lines
drivers/gpu/drm/ttm/ttm_pool.c +3 -0
 		it.alloc_count++;
 	}
 
@@ -1306,6 +1505,9 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	/* Report the unused defrag budget back to the caller. */
 	ctx->defrag_bytes_remaining = alloc.defrag_bytes_remaining;
 
+	if (!ret)
+		tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
+
 	return ret;
 }
 EXPORT_SYMBOL(ttm_pool_alloc);
@@ -1358,6 +1560,15 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 
 			if (ret)

drivers/gpu/drm/ttm/ttm_pool.c int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt, · line 1571

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +9 -0
 				return ret;
+
+			/*
+			 * __ttm_pool_alloc() counts each freshly (re)allocated
+			 * chunk against nr_suboptimal_pages, but a chunk whose
+			 * backup copy was interrupted and is finished here
+			 * resumes past that loop, so account for it directly.
+			 */
+			if (restore->order < ttm_pool_beneficial_order(pool))
+				alloc.nr_suboptimal_pages += 1U << restore->order;

NIT...'1U << order' vs '0x1 << order' in other places. Should be fine, probably good to be consistent.

Quoted 88 lines
drivers/gpu/drm/ttm/ttm_pool.c +6 -1
 		}
 		if (!alloc.remaining_pages) {
 			ret = ttm_pool_apply_caching(&alloc);
@@ -1367,11 +1578,16 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			kfree(tt->restore);
 			tt->restore = NULL;
 
+			tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
 			return 0;
 		}
 	}
 
-	return __ttm_pool_alloc(pool, tt, ctx, &alloc, restore);
+	ret = __ttm_pool_alloc(pool, tt, ctx, &alloc, restore);
+	if (!ret)
+		tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
+
+	return ret;
 }
 
 /**
include/drm/ttm/ttm_bo.h +11 -0
index 6124d2627b47..a80304f179ba 100644
@@ -237,6 +237,17 @@ struct ttm_operation_ctx {
 	 * @defrag_old_tt is set.
 	 */
 	s64 defrag_bytes_remaining;
+	/**
+	 * @prealloc: Pages preallocated outside the dma-resv lock for a defrag
+	 * move. The pool allocator consumes these instead of allocating fresh
+	 * beneficial-order pages under the lock, moving the (potentially
+	 * reclaim/compaction stalling) high-order allocations out of the
+	 * critical section. NULL means allocate in-line as usual. The pool
+	 * silently falls back to in-line allocation for any shortfall. The
+	 * allocator consumes it for beneficial-order chunks of any populate;
+	 * drivers only set it for a defrag move.
+	 */
+	struct ttm_pool_prealloc *prealloc;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
include/drm/ttm/ttm_pool.h +29 -0
index 753203980e2c..71670350eb15 100644
@@ -81,6 +81,35 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		   struct ttm_operation_ctx *ctx);
 void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt);
 
+/**
+ * struct ttm_pool_prealloc - Pages preallocated outside the dma-resv lock
+ * @pages: Array of @count beneficial-order pages (or fewer if a fill fell
+ *         short); each entry is the head page of a 1 << @order chunk.
+ * @order: Page order of every preallocated chunk.
+ * @caching: CPU caching applied to the pages, so leftovers can be restored
+ *           to write-back before being freed.
+ * @count: Number of valid entries in @pages.
+ * @used: Number of entries already consumed by the pool allocator.
+ *
+ * Defrag pages are interchangeable, so only a count of beneficial-order chunks
+ * is needed. ttm_pool_prealloc_fill() populates this outside the lock and
+ * __ttm_pool_alloc() drains it; any unused tail is released by
+ * ttm_pool_prealloc_fini().
+ */
+struct ttm_pool_prealloc {
+	struct page **pages;
+	unsigned int order;
+	enum ttm_caching caching;
+	unsigned int count;
+	unsigned int used;
+};
+
+int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
+			   struct ttm_pool_prealloc *pp, unsigned int count);
+void ttm_pool_prealloc_fini(struct ttm_pool *pool,
+			    struct ttm_pool_prealloc *pp);
+unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool);
+
 void ttm_pool_init(struct ttm_pool *pool, struct device *dev,
 		   int nid, unsigned int alloc_flags);
 void ttm_pool_fini(struct ttm_pool *pool);
include/drm/ttm/ttm_tt.h +0 -0
index 55c35bcf134d..81a5446ca693 100644

include/drm/ttm/ttm_tt.h struct ttm_tt { · line 142

Quoted 8 lines
include/drm/ttm/ttm_tt.h +5 -0
@@ -135,6 +135,12 @@ struct ttm_tt {
 	enum ttm_caching caching;
 	/** @restore: Partial restoration from backup state. TTM private */
 	struct ttm_pool_tt_restore *restore;
+	/**
+	 * @nr_suboptimal_pages: Number of pages backed below the pool's
+	 * beneficial order. Recorded by the pool allocator after populate; a
+	 * defrag pass reads it to size its out-of-lock preallocation exactly.
+	 */

NIT...may be not 'exactly' as patch 23 rounds it up.

Niranjana

Quoted 25 lines
include/drm/ttm/ttm_tt.h +12 -0
+	u32 nr_suboptimal_pages;
 };
 
 /**
@@ -201,6 +207,17 @@ static inline bool ttm_tt_is_beneficial_order_failed(const struct ttm_tt *tt)
 	return tt->page_flags & TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 }
 
+/**
+ * ttm_tt_suboptimal_pages() - Pages backed below the pool's beneficial order
+ * @tt: The struct ttm_tt.
+ *
+ * Return: Number of pages a defrag move would (re)allocate at beneficial order.
+ */
+static inline u32 ttm_tt_suboptimal_pages(const struct ttm_tt *tt)
+{
+	return tt->nr_suboptimal_pages;
+}
+
 /**
 * ttm_tt_create
 *
Signature · 1 line
2.34.1
#31 lore raw

Re: [PATCH v3 09/33] drm/ttm: Preallocate beneficial-order defrag pages outside the lock

In reply to Matthew Brost
Quoted 304 lines · Matthew Brost

Defragmentation moves reallocate a BO's backing at the pool's beneficial order. On a fragmented system those high-order allocations frequently stall in direct reclaim/compaction, and they run under the BO dma-resv lock, holding up concurrent rendering. The defrag worker does not care how long the work takes; it only cares about minimizing locked time.

Add a preallocation bag so the expensive allocations can be hoisted out of the lock:

- struct ttm_pool_prealloc carries beneficial-order pages preallocated outside the lock; ttm_pool_prealloc_fill()/_fini() allocate and release them. __ttm_pool_alloc() consumes them via ctx->prealloc.
- Apply the requested cpu-caching to the bag pages during fill, so the set_memory_*() cost is paid outside the lock too; consumed pages are already cache-consistent and leftovers are restored to write-back on free. Bag pages are never highmem so the kernel mapping is always valid for set_memory_*().
- Record the exact number of sub-optimal pages on the tt (nr_suboptimal_pages) at populate time so a defrag pass can size its prealloc precisely instead of guessing.
- Once a defrag move exhausts the prealloc bag, harvest the remainder of the old tt rather than entering reclaim under the lock, and flag the tt sub-optimal so the driver re-queues it for a later pass.

This keeps all reclaim/compaction stalls and cache mode changes out of the dma-resv critical section: lock-held time then tracks only the bytes actually copied.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---

drivers/gpu/drm/ttm/ttm_pool.c | 244 +++++++++++++++++++++++++++++++--
include/drm/ttm/ttm_bo.h       |  11 ++
include/drm/ttm/ttm_pool.h     |  29 ++++
include/drm/ttm/ttm_tt.h       |  17 +++
4 files changed, 287 insertions(+), 14 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +183 -5
index 1673e81b8df8..da7d132c8297 100644
@@ -82,6 +82,9 @@ struct ttm_pool_dma {
 * Pages harvested (borrowed) from the old tt are free and do not count; once
 * this drops to zero the allocator stops upgrading and harvests the remainder.
 * On completion it holds the unused budget. 0 means unlimited (no defrag cap).
+ * @nr_suboptimal_pages: Number of pages backed below the pool's beneficial
+ * order. Recorded by the pool allocator after populate; a defrag pass reads it
+ * to size its out-of-lock preallocation exactly.
 */
 struct ttm_pool_alloc_state {
 	struct page **pages;
@@ -90,6 +93,7 @@ struct ttm_pool_alloc_state {
 	pgoff_t remaining_pages;
 	enum ttm_caching tt_caching;
 	s64 defrag_bytes_remaining;
+	u32 nr_suboptimal_pages;
 };
 
 /**
@@ -787,6 +791,7 @@ static void ttm_pool_alloc_state_init(const struct ttm_tt *tt,
 	alloc->remaining_pages = tt->num_pages;
 	alloc->tt_caching = tt->caching;
 	alloc->defrag_bytes_remaining = 0;
+	alloc->nr_suboptimal_pages = 0;
 }
 
 /*
@@ -852,7 +857,8 @@ static unsigned int ttm_pool_defrag_old_order(struct ttm_pool *pool,
 * pages stay owned by the old tt until the move commits.
 */
 static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
-				 pgoff_t off, struct ttm_pool_alloc_state *alloc)
+				 unsigned int beneficial_order, pgoff_t off,
+				 struct ttm_pool_alloc_state *alloc)
 {
 	pgoff_t nr = 1UL << order;
 	int r;
@@ -887,6 +893,9 @@ static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
 
 	alloc->caching_divide = alloc->pages;
 
+	if (order < beneficial_order)
+		alloc->nr_suboptimal_pages += 0x1 << order;
+
 	return 0;
 }
 
@@ -934,7 +943,8 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 		    round_down(off, bnr) + bnr <= num_pages)
 			*suboptimal = true;
 
-		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
+		r = ttm_pool_harvest_page(old_tt, order, beneficial, off,
+					  alloc);
 		if (r)
 			return r;
 
@@ -944,6 +954,173 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 	return 0;
 }
 
+unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool)
+{
+	return ttm_pool_beneficial_order(pool);
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_order);
+
+/*
+ * Build the gfp flags used for the high-order, possibly reclaiming, beneficial
+ * order page allocations, matching the in-line defrag alloc path.
+ */
+static gfp_t ttm_pool_prealloc_gfp(struct ttm_pool *pool)
+{
+	gfp_t gfp = GFP_USER;
+
+	/*
+	 * No highmem: prealloc applies caching in bulk via set_pages_array_*()
+	 * on the kernel mapping, so the pages must be permanently mapped.
+	 */
+	if (ttm_pool_uses_dma32(pool))
+		gfp |= GFP_DMA32;
+
+	return gfp;
+}
+
+/*
+ * Apply a tt's cpu-caching to a batch of freshly system-allocated (write-back)
+ * prealloc pages in one shot. @cpages is an unpacked array of @ncpages
+ * individual 4K pages (the constituent pages of the mixed-order chunks, expanded
+ * so set_pages_array_*() sees one entry per page), since the packed prealloc bag
+ * holds multi-order chunks. Prealloc pages are never highmem (see
+ * ttm_pool_prealloc_gfp), so their kernel mapping is valid. No-op for cached
+ * pages and on non-x86. Returns non-zero if the caching change failed (the
+ * caller drops the whole bag and falls back to in-line allocation).
+ */
+static int ttm_pool_prealloc_apply_caching(enum ttm_caching caching,
+					   struct page **cpages,
+					   unsigned int ncpages)
+{
+#ifdef CONFIG_X86
+	if (!ncpages)
+		return 0;
+
+	switch (caching) {
+	case ttm_cached:
+		break;
+	case ttm_write_combined:
+		return set_pages_array_wc(cpages, ncpages);
+	case ttm_uncached:
+		return set_pages_array_uc(cpages, ncpages);
+	}
+#endif
+	return 0;
+}
+
+/* Expand an @order chunk into its constituent 4K pages for bulk caching. */
+static void ttm_pool_prealloc_stage_caching(struct page **cpages,
+					    unsigned int *ncpages,
+					    struct page *p, unsigned int order)
+{
+	unsigned int i, nr = 1u << order;
+
+	for (i = 0; i < nr; i++)
+		cpages[(*ncpages)++] = p + i;
+}
+
+/**
+ * ttm_pool_prealloc_fill() - Preallocate beneficial-order pages outside any lock
+ * @pool: The pool to allocate from.
+ * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @pp: Prealloc bag to fill; @pp->order is set to the beneficial order.
+ * @count: Number of beneficial-order chunks to attempt.
+ *
+ * Allocate up to @count beneficial-order chunks, parking them in @pp for a
+ * later __ttm_pool_alloc() defrag move to consume under the dma-resv lock. May
+ * sleep/reclaim freely as it runs unlocked. A short fill is fine: the pool
+ * falls back to in-line allocation for the shortfall. DMA-alloc pools are not
+ * supported (count stays 0). Returns 0 (release with ttm_pool_prealloc_fini()).
+ */
+int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
+			   struct ttm_pool_prealloc *pp, unsigned int count)
+{
+	unsigned int order = ttm_pool_beneficial_order(pool);
+	gfp_t gfp = ttm_pool_prealloc_gfp(pool);
+	struct page **cpages = NULL;
+	unsigned int ncpages = 0;
+	int r;
+
+	pp->pages = NULL;
+	pp->order = order;
+	pp->caching = tt_caching;
+	pp->count = 0;
+	pp->used = 0;
+
+	/* Nothing to gain without a beneficial order or for DMA-alloc pools. */
+	if (!order || !count || ttm_pool_uses_dma_alloc(pool))
+		return 0;
+
+	pp->pages = kvzalloc_objs(*pp->pages, count);
+	if (!pp->pages)
+		return 0;
+
+	/*
+	 * Every chunk is a fresh write-back system page of @order that needs a
+	 * caching change; collect their constituent pages into an unpacked
+	 * scratch array and issue a single set_pages_array_*() after the fill.
+	 * Cached tts need no change (and non-x86 handles caching at map time).
+	 */
+	if (IS_ENABLED(CONFIG_X86) && tt_caching != ttm_cached) {
+		cpages = kvzalloc_objs(*cpages, (size_t)count << order);
+		if (!cpages) {
+			kvfree(pp->pages);
+			pp->pages = NULL;
+			return 0;
+		}
+	}
+
+	while (pp->count < count) {
+		struct page *p = ttm_pool_alloc_page(pool, gfp, order, false);
+
+		if (!p)
+			break;
+
+		pp->pages[pp->count++] = p;
+		if (cpages)
+			ttm_pool_prealloc_stage_caching(cpages, &ncpages, p,
+							order);
+	}
+
+	/*
+	 * Apply the requested caching to every collected page in one shot. On
+	 * failure the pages' caching is indeterminate, so drop the whole bag
+	 * (freeing restores write-back) and let the consumer allocate in-line.
+	 */
+	r = ttm_pool_prealloc_apply_caching(tt_caching, cpages, ncpages);
+	kvfree(cpages);
+	if (r) {
+		unsigned int i;
+
+		for (i = 0; i < pp->count; i++)
+			ttm_pool_free_page(pool, tt_caching, order,
+					   pp->pages[i], false);
+		pp->count = 0;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fill);
+
+/**
+ * ttm_pool_prealloc_fini() - Release unconsumed preallocated pages
+ * @pool: The pool the pages came from.
+ * @pp: Prealloc bag to drain. Consumed pages (< @used) are now owned by the tt.
+ */
+void ttm_pool_prealloc_fini(struct ttm_pool *pool, struct ttm_pool_prealloc *pp)
+{
+	unsigned int i;
+
+	for (i = pp->used; i < pp->count; ++i)
+		ttm_pool_free_page(pool, pp->caching, pp->order, pp->pages[i],
+				   false);
+	kvfree(pp->pages);
+	pp->pages = NULL;
+	pp->count = 0;
+	pp->used = 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fini);
+
 /**
 * enum ttm_pool_iter_action - Outcome of a per-order allocation phase
 * @TTM_POOL_ITER_FILL: A page (@it->p) was acquired; proceed to fill it in.
@@ -975,6 +1152,7 @@ struct ttm_pool_alloc_iter {
 	struct ttm_pool_alloc_state *alloc;
 	struct ttm_pool_tt_restore *restore;
 	struct ttm_tt *defrag_old_tt;
+	struct ttm_pool_prealloc *prealloc;
 	unsigned int beneficial_order;
 	gfp_t gfp_flags;
 	bool defrag_capped;
@@ -990,16 +1168,20 @@ struct ttm_pool_alloc_iter {
 };
 
 /*
- * Defrag move budget exhausted: the upgrade can make no further progress this
- * pass. Snapshot @defrag_capped is set only when a byte budget was in force at
- * entry.
+ * Defrag move budget exhausted, or the out-of-lock prealloc bag ran dry: the
+ * upgrade can make no further progress this pass. Snapshot @defrag_capped is
+ * set only when a byte budget was in force at entry.
 */
 static bool ttm_pool_iter_defrag_exhausted(const struct ttm_pool_alloc_iter *it)

drivers/gpu/drm/ttm/ttm_pool.c struct ttm_pool_alloc_iter { · line 1184

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +5 -1
 {
+	const struct ttm_pool_prealloc *pp = it->prealloc;
+
 	if (!it->defrag_old_tt)
 		return false;
+	if (it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0)
+		return true;
 
-	return it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0;
+	return pp && pp->count && pp->used >= pp->count;
Assisted-by LLM

Also, there is a sashiko reported issue here which suggests to dropping 'pp->count' check here. Though, we won't endup here with pp->count as 0 and even if we do it gets handled properly, it seems it is good to drop it so that it is explicitly handled.

Niranjana

Quoted 230 lines
drivers/gpu/drm/ttm/ttm_pool.c +42 -8
 }
 
 /*
@@ -1056,7 +1238,8 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 		return TTM_POOL_ITER_FILL;
 
 	it->order = harvest_order;
-	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order, off, it->alloc);
+	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order,
+				  it->beneficial_order, off, it->alloc);
 	if (r)
 		return r;
 
@@ -1068,11 +1251,13 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 
 /*
 * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries a same-order pool page, then a fresh system allocation. Fault
- * injection can force the beneficial-order paths to "fail".
+ * failure). Tries, in order: a beneficial-order page preallocated outside the
+ * dma-resv lock (defrag), a same-order pool page, then a fresh system
+ * allocation. Fault injection can force the beneficial-order paths to "fail".
 */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
+	struct ttm_pool_prealloc *pp = it->prealloc;
 	struct ttm_pool_type *pt;
 
 	it->p = NULL;
@@ -1087,7 +1272,18 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 		it->beneficial_order && it->order >= it->beneficial_order &&
 		should_fail(&beneficial_order_fault_inject, 1);
 
-	/* First, try to allocate a page from a pool if one exists. */
+	/*
+	 * Defrag move: consume a beneficial-order page preallocated outside the
+	 * dma-resv lock instead of allocating one in-line (which may stall in
+	 * reclaim/compaction). Falls through once the bag is empty.
+	 */
+	if (pp && !it->fail_beneficial && it->order >= it->beneficial_order &&
+	    pp->used < pp->count) {
+		it->order = it->beneficial_order;
+		it->p = pp->pages[pp->used++];
+		it->page_caching = it->tt->caching;
+	}
+
 	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
 	if (!it->p && pt && it->allow_pools && !it->fail_beneficial)
 		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
@@ -1164,6 +1360,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		.alloc = alloc,
 		.restore = restore,
 		.defrag_old_tt = restore ? NULL : ctx->defrag_old_tt,
+		.prealloc = restore ? NULL : ctx->prealloc,
 		.beneficial_order = ttm_pool_beneficial_order(pool),
 		.page_caching = tt->caching,
 		.allow_pools = true,
@@ -1193,8 +1390,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
 		/*
-		 * Out of defrag budget: harvest the rest of the old tt as-is and
-		 * stop (the tt is re-queued if the remainder is still
+		 * Out of defrag budget/prealloc: harvest the rest of the old tt
+		 * as-is and stop (the tt is re-queued if the remainder is still
 		 * sub-optimal).
 		 */
 		if (ttm_pool_iter_defrag_exhausted(&it)) {
@@ -1211,7 +1408,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		if (r == TTM_POOL_ITER_RETRY)
 			continue;
 
-		/* Acquire a page (pool / system) for this order. */
+		/* Acquire a page (prealloc / pool / system) for this order. */
 		ttm_pool_iter_acquire_page(&it);
 		if (!it.p) {
 			r = ttm_pool_iter_lower_order(&it);
@@ -1243,6 +1440,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			alloc->defrag_bytes_remaining -=
 				(s64)(1UL << it.order) << PAGE_SHIFT;
 
+		if (it.order < it.beneficial_order)
+			alloc->nr_suboptimal_pages += 0x1 << it.order;
 		it.alloc_count++;
 	}
 
@@ -1306,6 +1505,9 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	/* Report the unused defrag budget back to the caller. */
 	ctx->defrag_bytes_remaining = alloc.defrag_bytes_remaining;
 
+	if (!ret)
+		tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
+
 	return ret;
 }
 EXPORT_SYMBOL(ttm_pool_alloc);
@@ -1358,6 +1560,15 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 
 			if (ret)
 				return ret;
+
+			/*
+			 * __ttm_pool_alloc() counts each freshly (re)allocated
+			 * chunk against nr_suboptimal_pages, but a chunk whose
+			 * backup copy was interrupted and is finished here
+			 * resumes past that loop, so account for it directly.
+			 */
+			if (restore->order < ttm_pool_beneficial_order(pool))
+				alloc.nr_suboptimal_pages += 1U << restore->order;
 		}
 		if (!alloc.remaining_pages) {
 			ret = ttm_pool_apply_caching(&alloc);
@@ -1367,11 +1578,16 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			kfree(tt->restore);
 			tt->restore = NULL;
 
+			tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
 			return 0;
 		}
 	}
 
-	return __ttm_pool_alloc(pool, tt, ctx, &alloc, restore);
+	ret = __ttm_pool_alloc(pool, tt, ctx, &alloc, restore);
+	if (!ret)
+		tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
+
+	return ret;
 }
 
 /**
include/drm/ttm/ttm_bo.h +11 -0
index 6124d2627b47..a80304f179ba 100644
@@ -237,6 +237,17 @@ struct ttm_operation_ctx {
 	 * @defrag_old_tt is set.
 	 */
 	s64 defrag_bytes_remaining;
+	/**
+	 * @prealloc: Pages preallocated outside the dma-resv lock for a defrag
+	 * move. The pool allocator consumes these instead of allocating fresh
+	 * beneficial-order pages under the lock, moving the (potentially
+	 * reclaim/compaction stalling) high-order allocations out of the
+	 * critical section. NULL means allocate in-line as usual. The pool
+	 * silently falls back to in-line allocation for any shortfall. The
+	 * allocator consumes it for beneficial-order chunks of any populate;
+	 * drivers only set it for a defrag move.
+	 */
+	struct ttm_pool_prealloc *prealloc;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
include/drm/ttm/ttm_pool.h +29 -0
index 753203980e2c..71670350eb15 100644
@@ -81,6 +81,35 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		   struct ttm_operation_ctx *ctx);
 void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt);
 
+/**
+ * struct ttm_pool_prealloc - Pages preallocated outside the dma-resv lock
+ * @pages: Array of @count beneficial-order pages (or fewer if a fill fell
+ *         short); each entry is the head page of a 1 << @order chunk.
+ * @order: Page order of every preallocated chunk.
+ * @caching: CPU caching applied to the pages, so leftovers can be restored
+ *           to write-back before being freed.
+ * @count: Number of valid entries in @pages.
+ * @used: Number of entries already consumed by the pool allocator.
+ *
+ * Defrag pages are interchangeable, so only a count of beneficial-order chunks
+ * is needed. ttm_pool_prealloc_fill() populates this outside the lock and
+ * __ttm_pool_alloc() drains it; any unused tail is released by
+ * ttm_pool_prealloc_fini().
+ */
+struct ttm_pool_prealloc {
+	struct page **pages;
+	unsigned int order;
+	enum ttm_caching caching;
+	unsigned int count;
+	unsigned int used;
+};
+
+int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
+			   struct ttm_pool_prealloc *pp, unsigned int count);
+void ttm_pool_prealloc_fini(struct ttm_pool *pool,
+			    struct ttm_pool_prealloc *pp);
+unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool);
+
 void ttm_pool_init(struct ttm_pool *pool, struct device *dev,
 		   int nid, unsigned int alloc_flags);
 void ttm_pool_fini(struct ttm_pool *pool);
include/drm/ttm/ttm_tt.h +17 -0
index 55c35bcf134d..81a5446ca693 100644
@@ -135,6 +135,12 @@ struct ttm_tt {
 	enum ttm_caching caching;
 	/** @restore: Partial restoration from backup state. TTM private */
 	struct ttm_pool_tt_restore *restore;
+	/**
+	 * @nr_suboptimal_pages: Number of pages backed below the pool's
+	 * beneficial order. Recorded by the pool allocator after populate; a
+	 * defrag pass reads it to size its out-of-lock preallocation exactly.
+	 */
+	u32 nr_suboptimal_pages;
 };
 
 /**
@@ -201,6 +207,17 @@ static inline bool ttm_tt_is_beneficial_order_failed(const struct ttm_tt *tt)
 	return tt->page_flags & TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 }
 
+/**
+ * ttm_tt_suboptimal_pages() - Pages backed below the pool's beneficial order
+ * @tt: The struct ttm_tt.
+ *
+ * Return: Number of pages a defrag move would (re)allocate at beneficial order.
+ */
+static inline u32 ttm_tt_suboptimal_pages(const struct ttm_tt *tt)
+{
+	return tt->nr_suboptimal_pages;
+}
+
 /**
 * ttm_tt_create
 *
Signature · 1 line
2.34.1
author maintainer
#32 lore raw

Re: [PATCH v3 09/33] drm/ttm: Preallocate beneficial-order defrag pages outside the lock

Quoted 313 lines · Niranjana Vishwanathapura
Quoted 312 lines · Matthew Brost

Defragmentation moves reallocate a BO's backing at the pool's beneficial order. On a fragmented system those high-order allocations frequently stall in direct reclaim/compaction, and they run under the BO dma-resv lock, holding up concurrent rendering. The defrag worker does not care how long the work takes; it only cares about minimizing locked time.

Add a preallocation bag so the expensive allocations can be hoisted out of the lock:

- struct ttm_pool_prealloc carries beneficial-order pages preallocated outside the lock; ttm_pool_prealloc_fill()/_fini() allocate and release them. __ttm_pool_alloc() consumes them via ctx->prealloc.
- Apply the requested cpu-caching to the bag pages during fill, so the set_memory_*() cost is paid outside the lock too; consumed pages are already cache-consistent and leftovers are restored to write-back on free. Bag pages are never highmem so the kernel mapping is always valid for set_memory_*().
- Record the exact number of sub-optimal pages on the tt (nr_suboptimal_pages) at populate time so a defrag pass can size its prealloc precisely instead of guessing.
- Once a defrag move exhausts the prealloc bag, harvest the remainder of the old tt rather than entering reclaim under the lock, and flag the tt sub-optimal so the driver re-queues it for a later pass.

This keeps all reclaim/compaction stalls and cache mode changes out of the dma-resv critical section: lock-held time then tracks only the bytes actually copied.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---

drivers/gpu/drm/ttm/ttm_pool.c | 244 +++++++++++++++++++++++++++++++--
include/drm/ttm/ttm_bo.h       |  11 ++
include/drm/ttm/ttm_pool.h     |  29 ++++
include/drm/ttm/ttm_tt.h       |  17 +++
4 files changed, 287 insertions(+), 14 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +187 -5
index 1673e81b8df8..da7d132c8297 100644
@@ -82,6 +82,9 @@ struct ttm_pool_dma {
 * Pages harvested (borrowed) from the old tt are free and do not count; once
 * this drops to zero the allocator stops upgrading and harvests the remainder.
 * On completion it holds the unused budget. 0 means unlimited (no defrag cap).
+ * @nr_suboptimal_pages: Number of pages backed below the pool's beneficial
+ * order. Recorded by the pool allocator after populate; a defrag pass reads it
+ * to size its out-of-lock preallocation exactly.
 */
 struct ttm_pool_alloc_state {
 	struct page **pages;
@@ -90,6 +93,7 @@ struct ttm_pool_alloc_state {
 	pgoff_t remaining_pages;
 	enum ttm_caching tt_caching;
 	s64 defrag_bytes_remaining;
+	u32 nr_suboptimal_pages;
 };
 
 /**
@@ -787,6 +791,7 @@ static void ttm_pool_alloc_state_init(const struct ttm_tt *tt,
 	alloc->remaining_pages = tt->num_pages;
 	alloc->tt_caching = tt->caching;
 	alloc->defrag_bytes_remaining = 0;
+	alloc->nr_suboptimal_pages = 0;
 }
 
 /*
@@ -852,7 +857,8 @@ static unsigned int ttm_pool_defrag_old_order(struct ttm_pool *pool,
 * pages stay owned by the old tt until the move commits.
 */
 static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
-				 pgoff_t off, struct ttm_pool_alloc_state *alloc)
+				 unsigned int beneficial_order, pgoff_t off,
+				 struct ttm_pool_alloc_state *alloc)
 {
 	pgoff_t nr = 1UL << order;
 	int r;
@@ -887,6 +893,9 @@ static int ttm_pool_harvest_page(struct ttm_tt *old_tt, unsigned int order,
 
 	alloc->caching_divide = alloc->pages;
 
+	if (order < beneficial_order)
+		alloc->nr_suboptimal_pages += 0x1 << order;
+
 	return 0;
 }
 
@@ -934,7 +943,8 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 		    round_down(off, bnr) + bnr <= num_pages)
 			*suboptimal = true;
 
-		r = ttm_pool_harvest_page(old_tt, order, off, alloc);
+		r = ttm_pool_harvest_page(old_tt, order, beneficial, off,
+					  alloc);
 		if (r)
 			return r;
 
@@ -944,6 +954,173 @@ static int ttm_pool_harvest_remaining(struct ttm_pool *pool,
 	return 0;
 }
 
+unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool)
+{
+	return ttm_pool_beneficial_order(pool);
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_order);
+
+/*
+ * Build the gfp flags used for the high-order, possibly reclaiming, beneficial
+ * order page allocations, matching the in-line defrag alloc path.
+ */
+static gfp_t ttm_pool_prealloc_gfp(struct ttm_pool *pool)
+{
+	gfp_t gfp = GFP_USER;
+
+	/*
+	 * No highmem: prealloc applies caching in bulk via set_pages_array_*()
+	 * on the kernel mapping, so the pages must be permanently mapped.
+	 */
+	if (ttm_pool_uses_dma32(pool))
+		gfp |= GFP_DMA32;
+
+	return gfp;
+}
+
+/*
+ * Apply a tt's cpu-caching to a batch of freshly system-allocated (write-back)
+ * prealloc pages in one shot. @cpages is an unpacked array of @ncpages
+ * individual 4K pages (the constituent pages of the mixed-order chunks, expanded
+ * so set_pages_array_*() sees one entry per page), since the packed prealloc bag
+ * holds multi-order chunks. Prealloc pages are never highmem (see
+ * ttm_pool_prealloc_gfp), so their kernel mapping is valid. No-op for cached
+ * pages and on non-x86. Returns non-zero if the caching change failed (the
+ * caller drops the whole bag and falls back to in-line allocation).
+ */
+static int ttm_pool_prealloc_apply_caching(enum ttm_caching caching,
+					   struct page **cpages,
+					   unsigned int ncpages)
+{
+#ifdef CONFIG_X86
+	if (!ncpages)
+		return 0;
+
+	switch (caching) {
+	case ttm_cached:
+		break;
+	case ttm_write_combined:
+		return set_pages_array_wc(cpages, ncpages);
+	case ttm_uncached:
+		return set_pages_array_uc(cpages, ncpages);
+	}
+#endif
+	return 0;
+}
+
+/* Expand an @order chunk into its constituent 4K pages for bulk caching. */
+static void ttm_pool_prealloc_stage_caching(struct page **cpages,
+					    unsigned int *ncpages,
+					    struct page *p, unsigned int order)
+{
+	unsigned int i, nr = 1u << order;
+
+	for (i = 0; i < nr; i++)
+		cpages[(*ncpages)++] = p + i;
+}
+
+/**
+ * ttm_pool_prealloc_fill() - Preallocate beneficial-order pages outside any lock
+ * @pool: The pool to allocate from.
+ * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @pp: Prealloc bag to fill; @pp->order is set to the beneficial order.
+ * @count: Number of beneficial-order chunks to attempt.
+ *
+ * Allocate up to @count beneficial-order chunks, parking them in @pp for a
+ * later __ttm_pool_alloc() defrag move to consume under the dma-resv lock. May
+ * sleep/reclaim freely as it runs unlocked. A short fill is fine: the pool
+ * falls back to in-line allocation for the shortfall. DMA-alloc pools are not
+ * supported (count stays 0). Returns 0 (release with ttm_pool_prealloc_fini()).
+ */
+int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
+			   struct ttm_pool_prealloc *pp, unsigned int count)
+{
+	unsigned int order = ttm_pool_beneficial_order(pool);
+	gfp_t gfp = ttm_pool_prealloc_gfp(pool);
+	struct page **cpages = NULL;
+	unsigned int ncpages = 0;
+	int r;
+
+	pp->pages = NULL;
+	pp->order = order;
+	pp->caching = tt_caching;
+	pp->count = 0;
+	pp->used = 0;
+
+	/* Nothing to gain without a beneficial order or for DMA-alloc pools. */
+	if (!order || !count || ttm_pool_uses_dma_alloc(pool))
+		return 0;
+
+	pp->pages = kvzalloc_objs(*pp->pages, count);
+	if (!pp->pages)
+		return 0;
+
+	/*
+	 * Every chunk is a fresh write-back system page of @order that needs a
+	 * caching change; collect their constituent pages into an unpacked
+	 * scratch array and issue a single set_pages_array_*() after the fill.
+	 * Cached tts need no change (and non-x86 handles caching at map time).
+	 */
+	if (IS_ENABLED(CONFIG_X86) && tt_caching != ttm_cached) {
+		cpages = kvzalloc_objs(*cpages, (size_t)count << order);
+		if (!cpages) {
+			kvfree(pp->pages);
+			pp->pages = NULL;
+			return 0;
+		}
+	}
+
+	while (pp->count < count) {
+		struct page *p = ttm_pool_alloc_page(pool, gfp, order, false);
+
+		if (!p)
+			break;
+
+		pp->pages[pp->count++] = p;
+		if (cpages)
+			ttm_pool_prealloc_stage_caching(cpages, &ncpages, p,
+							order);
+	}
+
+	/*
+	 * Apply the requested caching to every collected page in one shot. On
+	 * failure the pages' caching is indeterminate, so drop the whole bag
+	 * (freeing restores write-back) and let the consumer allocate in-line.
+	 */
+	r = ttm_pool_prealloc_apply_caching(tt_caching, cpages, ncpages);
+	kvfree(cpages);
+	if (r) {
+		unsigned int i;
+
+		for (i = 0; i < pp->count; i++)
+			ttm_pool_free_page(pool, tt_caching, order,
+					   pp->pages[i], false);
+		pp->count = 0;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fill);
+
+/**
+ * ttm_pool_prealloc_fini() - Release unconsumed preallocated pages
+ * @pool: The pool the pages came from.
+ * @pp: Prealloc bag to drain. Consumed pages (< @used) are now owned by the tt.
+ */
+void ttm_pool_prealloc_fini(struct ttm_pool *pool, struct ttm_pool_prealloc *pp)
+{
+	unsigned int i;
+
+	for (i = pp->used; i < pp->count; ++i)
+		ttm_pool_free_page(pool, pp->caching, pp->order, pp->pages[i],
+				   false);
+	kvfree(pp->pages);
+	pp->pages = NULL;
+	pp->count = 0;
+	pp->used = 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fini);
+
 /**
 * enum ttm_pool_iter_action - Outcome of a per-order allocation phase
 * @TTM_POOL_ITER_FILL: A page (@it->p) was acquired; proceed to fill it in.
@@ -975,6 +1152,7 @@ struct ttm_pool_alloc_iter {
 	struct ttm_pool_alloc_state *alloc;
 	struct ttm_pool_tt_restore *restore;
 	struct ttm_tt *defrag_old_tt;
+	struct ttm_pool_prealloc *prealloc;
 	unsigned int beneficial_order;
 	gfp_t gfp_flags;
 	bool defrag_capped;
@@ -990,16 +1168,20 @@ struct ttm_pool_alloc_iter {
 };
 
 /*
- * Defrag move budget exhausted: the upgrade can make no further progress this
- * pass. Snapshot @defrag_capped is set only when a byte budget was in force at
- * entry.
+ * Defrag move budget exhausted, or the out-of-lock prealloc bag ran dry: the
+ * upgrade can make no further progress this pass. Snapshot @defrag_capped is
+ * set only when a byte budget was in force at entry.
 */
 static bool ttm_pool_iter_defrag_exhausted(const struct ttm_pool_alloc_iter *it)
 {
+	const struct ttm_pool_prealloc *pp = it->prealloc;
+
 	if (!it->defrag_old_tt)
 		return false;
+	if (it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0)
+		return true;
 
Quoted 5 lines

drivers/gpu/drm/ttm/ttm_pool.c struct ttm_pool_alloc_iter { · line 1184

Quoted 2 lines
drivers/gpu/drm/ttm/ttm_pool.c +1 -1
-	return it->defrag_capped && it->alloc->defrag_bytes_remaining <= 0;
+	return pp && pp->count && pp->used >= pp->count;
Assisted-by LLM

Yes.

Quoted 5 lines

Also, there is a sashiko reported issue here which suggests to dropping 'pp->count' check here. Though, we won't endup here with pp->count as 0 and even if we do it gets handled properly, it seems it is good to drop it so that it is explicitly handled.

Yes, that is better. We actually handle this driver side in Xe but TTM should handle this as well.

Matt

Quoted 232 lines

Niranjana

Quoted 230 lines
drivers/gpu/drm/ttm/ttm_pool.c +42 -8
 }
 
 /*
@@ -1056,7 +1238,8 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 		return TTM_POOL_ITER_FILL;
 
 	it->order = harvest_order;
-	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order, off, it->alloc);
+	r = ttm_pool_harvest_page(it->defrag_old_tt, it->order,
+				  it->beneficial_order, off, it->alloc);
 	if (r)
 		return r;
 
@@ -1068,11 +1251,13 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 
 /*
 * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries a same-order pool page, then a fresh system allocation. Fault
- * injection can force the beneficial-order paths to "fail".
+ * failure). Tries, in order: a beneficial-order page preallocated outside the
+ * dma-resv lock (defrag), a same-order pool page, then a fresh system
+ * allocation. Fault injection can force the beneficial-order paths to "fail".
 */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
+	struct ttm_pool_prealloc *pp = it->prealloc;
 	struct ttm_pool_type *pt;
 
 	it->p = NULL;
@@ -1087,7 +1272,18 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 		it->beneficial_order && it->order >= it->beneficial_order &&
 		should_fail(&beneficial_order_fault_inject, 1);
 
-	/* First, try to allocate a page from a pool if one exists. */
+	/*
+	 * Defrag move: consume a beneficial-order page preallocated outside the
+	 * dma-resv lock instead of allocating one in-line (which may stall in
+	 * reclaim/compaction). Falls through once the bag is empty.
+	 */
+	if (pp && !it->fail_beneficial && it->order >= it->beneficial_order &&
+	    pp->used < pp->count) {
+		it->order = it->beneficial_order;
+		it->p = pp->pages[pp->used++];
+		it->page_caching = it->tt->caching;
+	}
+
 	pt = ttm_pool_select_type(it->pool, it->page_caching, it->order);
 	if (!it->p && pt && it->allow_pools && !it->fail_beneficial)
 		it->p = ttm_pool_type_take(pt, ttm_pool_nid(it->pool));
@@ -1164,6 +1360,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		.alloc = alloc,
 		.restore = restore,
 		.defrag_old_tt = restore ? NULL : ctx->defrag_old_tt,
+		.prealloc = restore ? NULL : ctx->prealloc,
 		.beneficial_order = ttm_pool_beneficial_order(pool),
 		.page_caching = tt->caching,
 		.allow_pools = true,
@@ -1193,8 +1390,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	     alloc->remaining_pages;
 	     it.order = ttm_pool_alloc_find_order(it.order, alloc)) {
 		/*
-		 * Out of defrag budget: harvest the rest of the old tt as-is and
-		 * stop (the tt is re-queued if the remainder is still
+		 * Out of defrag budget/prealloc: harvest the rest of the old tt
+		 * as-is and stop (the tt is re-queued if the remainder is still
 		 * sub-optimal).
 		 */
 		if (ttm_pool_iter_defrag_exhausted(&it)) {
@@ -1211,7 +1408,7 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		if (r == TTM_POOL_ITER_RETRY)
 			continue;
 
-		/* Acquire a page (pool / system) for this order. */
+		/* Acquire a page (prealloc / pool / system) for this order. */
 		ttm_pool_iter_acquire_page(&it);
 		if (!it.p) {
 			r = ttm_pool_iter_lower_order(&it);
@@ -1243,6 +1440,8 @@ static int __ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			alloc->defrag_bytes_remaining -=
 				(s64)(1UL << it.order) << PAGE_SHIFT;
 
+		if (it.order < it.beneficial_order)
+			alloc->nr_suboptimal_pages += 0x1 << it.order;
 		it.alloc_count++;
 	}
 
@@ -1306,6 +1505,9 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 	/* Report the unused defrag budget back to the caller. */
 	ctx->defrag_bytes_remaining = alloc.defrag_bytes_remaining;
 
+	if (!ret)
+		tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
+
 	return ret;
 }
 EXPORT_SYMBOL(ttm_pool_alloc);
@@ -1358,6 +1560,15 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 
 			if (ret)
 				return ret;
+
+			/*
+			 * __ttm_pool_alloc() counts each freshly (re)allocated
+			 * chunk against nr_suboptimal_pages, but a chunk whose
+			 * backup copy was interrupted and is finished here
+			 * resumes past that loop, so account for it directly.
+			 */
+			if (restore->order < ttm_pool_beneficial_order(pool))
+				alloc.nr_suboptimal_pages += 1U << restore->order;
 		}
 		if (!alloc.remaining_pages) {
 			ret = ttm_pool_apply_caching(&alloc);
@@ -1367,11 +1578,16 @@ int ttm_pool_restore_and_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 			kfree(tt->restore);
 			tt->restore = NULL;
 
+			tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
 			return 0;
 		}
 	}
 
-	return __ttm_pool_alloc(pool, tt, ctx, &alloc, restore);
+	ret = __ttm_pool_alloc(pool, tt, ctx, &alloc, restore);
+	if (!ret)
+		tt->nr_suboptimal_pages = alloc.nr_suboptimal_pages;
+
+	return ret;
 }
 
 /**
include/drm/ttm/ttm_bo.h +11 -0
index 6124d2627b47..a80304f179ba 100644
@@ -237,6 +237,17 @@ struct ttm_operation_ctx {
 	 * @defrag_old_tt is set.
 	 */
 	s64 defrag_bytes_remaining;
+	/**
+	 * @prealloc: Pages preallocated outside the dma-resv lock for a defrag
+	 * move. The pool allocator consumes these instead of allocating fresh
+	 * beneficial-order pages under the lock, moving the (potentially
+	 * reclaim/compaction stalling) high-order allocations out of the
+	 * critical section. NULL means allocate in-line as usual. The pool
+	 * silently falls back to in-line allocation for any shortfall. The
+	 * allocator consumes it for beneficial-order chunks of any populate;
+	 * drivers only set it for a defrag move.
+	 */
+	struct ttm_pool_prealloc *prealloc;
 	/**
 	 * @resv: Reservation object to be used together with
 	 * @allow_res_evict.
include/drm/ttm/ttm_pool.h +29 -0
index 753203980e2c..71670350eb15 100644
@@ -81,6 +81,35 @@ int ttm_pool_alloc(struct ttm_pool *pool, struct ttm_tt *tt,
 		   struct ttm_operation_ctx *ctx);
 void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt);
 
+/**
+ * struct ttm_pool_prealloc - Pages preallocated outside the dma-resv lock
+ * @pages: Array of @count beneficial-order pages (or fewer if a fill fell
+ *         short); each entry is the head page of a 1 << @order chunk.
+ * @order: Page order of every preallocated chunk.
+ * @caching: CPU caching applied to the pages, so leftovers can be restored
+ *           to write-back before being freed.
+ * @count: Number of valid entries in @pages.
+ * @used: Number of entries already consumed by the pool allocator.
+ *
+ * Defrag pages are interchangeable, so only a count of beneficial-order chunks
+ * is needed. ttm_pool_prealloc_fill() populates this outside the lock and
+ * __ttm_pool_alloc() drains it; any unused tail is released by
+ * ttm_pool_prealloc_fini().
+ */
+struct ttm_pool_prealloc {
+	struct page **pages;
+	unsigned int order;
+	enum ttm_caching caching;
+	unsigned int count;
+	unsigned int used;
+};
+
+int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
+			   struct ttm_pool_prealloc *pp, unsigned int count);
+void ttm_pool_prealloc_fini(struct ttm_pool *pool,
+			    struct ttm_pool_prealloc *pp);
+unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool);
+
 void ttm_pool_init(struct ttm_pool *pool, struct device *dev,
 		   int nid, unsigned int alloc_flags);
 void ttm_pool_fini(struct ttm_pool *pool);
include/drm/ttm/ttm_tt.h +17 -0
index 55c35bcf134d..81a5446ca693 100644
@@ -135,6 +135,12 @@ struct ttm_tt {
 	enum ttm_caching caching;
 	/** @restore: Partial restoration from backup state. TTM private */
 	struct ttm_pool_tt_restore *restore;
+	/**
+	 * @nr_suboptimal_pages: Number of pages backed below the pool's
+	 * beneficial order. Recorded by the pool allocator after populate; a
+	 * defrag pass reads it to size its out-of-lock preallocation exactly.
+	 */
+	u32 nr_suboptimal_pages;
 };
 
 /**
@@ -201,6 +207,17 @@ static inline bool ttm_tt_is_beneficial_order_failed(const struct ttm_tt *tt)
 	return tt->page_flags & TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
 }
 
+/**
+ * ttm_tt_suboptimal_pages() - Pages backed below the pool's beneficial order
+ * @tt: The struct ttm_tt.
+ *
+ * Return: Number of pages a defrag move would (re)allocate at beneficial order.
+ */
+static inline u32 ttm_tt_suboptimal_pages(const struct ttm_tt *tt)
+{
+	return tt->nr_suboptimal_pages;
+}
+
 /**
 * ttm_tt_create
 *
Signature · 1 line
2.34.1
author maintainer
#33 lore raw

PATCH v3 10/33 drm/ttm: Add full out-of-lock preallocation for ttm_pool_alloc()

In reply to Matthew Brost

ttm_pool_prealloc_fill() lets a driver preallocate beneficial-order pages outside the dma-resv lock for a defragmentation move, so the high-order allocations that may stall in reclaim/compaction happen off the critical section. That bag is deliberately partial: it holds only interchangeable beneficial-order chunks and the allocator consumes it opportunistically, falling back to in-line allocation for everything else.

An ordinary populate of a system-memory buffer object has the same problem - the whole backing is allocated under the dma-resv lock - but no way to move it out of the critical section, because the existing bag does not cover sub-beneficial orders and does not describe a full tt.

Add a "full" flavour of the same preallocation mechanism:

- struct ttm_pool_prealloc gains a @full flag. In full mode the bag is an ordered tiling of an entire tt at mixed orders (pool pages included), rather than a set of interchangeable beneficial-order chunks. The per-chunk order is recorded on the page itself and read back at consume time, so no parallel order array is needed.

- ttm_pool_prealloc_fill_full() allocates the whole backing for a tt of @num_pages pages up front, mirroring the in-line allocator's order walk (highest order first, capped by the remaining page count). It reuses the existing unlocked helpers (ttm_pool_prealloc_gfp(), ttm_pool_set_caching()) via a new ttm_pool_take_page() helper that tries a same-order pool page before a fresh system allocation.

- ttm_pool_prealloc_fini() frees any unconsumed tail at each chunk's own order when @full.

- ttm_pool_prealloc_fill_full() takes a @beneficial_reclaim_backoff argument, plumbed to ttm_pool_alloc_page(), that lets the caller allocate higher-order chunks without aggressive reclaim/compaction. This suits a driver that upgrades suboptimal pages to beneficial order in place later (e.g. via a background defragmenter) and so has no need to stall the user context reclaiming for contiguity up front.

- __ttm_pool_alloc() consumes a full bag with a single early branch in ttm_pool_iter_acquire_page(): install the next chunk as-is (order taken from the page) and return; once the best-effort bag is drained the loop falls through to normal in-line allocation. The defrag harvest and reuse-old phases are inert here as there is no defrag_old_tt.

Like the defrag bag, this is best-effort: a short fill is fine (the pool allocates the shortfall in-line) and DMA-alloc pools are unsupported (caching is applied on the kernel mapping). No functional change for existing users; the new fill routine has no in-tree caller yet.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/ttm/ttm_pool.c | 232 ++++++++++++++++++++++++++++++++-
 include/drm/ttm/ttm_bo.h       |  19 +--
 include/drm/ttm/ttm_pool.h     |  27 +++-
 3 files changed, 258 insertions(+), 20 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +226 -6
index da7d132c8297..4746f5d8cd7e 100644
@@ -1046,6 +1046,7 @@ int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 	pp->caching = tt_caching;
 	pp->count = 0;
 	pp->used = 0;
+	pp->full = false;
 
 	/* Nothing to gain without a beneficial order or for DMA-alloc pools. */
 	if (!order || !count || ttm_pool_uses_dma_alloc(pool))
@@ -1102,6 +1103,187 @@ int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 }
 EXPORT_SYMBOL(ttm_pool_prealloc_fill);
 
+/*
+ * Acquire a single *@order chunk, trying a same-order pool page first (already
+ * carries @caching) then a fresh system allocation (write-back). Runs unlocked.
+ * When @beneficial_reclaim_backoff is set, higher-order system allocations skip
+ * aggressive reclaim/compaction (see ttm_pool_alloc_page()).
+ *
+ * A fresh system page is write-back and needs the tt's caching applied later in
+ * bulk; *@need_caching is set true for it and false for a (pre-cached) pool
+ * page.
+ *
+ * On a genuine failure *@order is left untouched (the caller lowers it by one
+ * and retries). Fault injection instead pretends allocation at (or above) the
+ * beneficial order failed and lowers *@order to the beneficial order, so the
+ * caller's next attempt lands below it - mirroring the in-line acquire path's
+ * fail_beneficial jump and exercising the beneficial_order_failed tracking
+ * without driving the system into real fragmentation.
+ *
+ * Returns NULL if no page of *@order could be obtained.
+ */
+static struct page *ttm_pool_take_page(struct ttm_pool *pool,
+				       enum ttm_caching caching,
+				       unsigned int *order, gfp_t gfp,
+				       bool *need_caching,
+				       bool beneficial_reclaim_backoff)
+{
+	unsigned int beneficial = ttm_pool_beneficial_order(pool);
+	struct ttm_pool_type *pt;
+	struct page *p;
+
+	*need_caching = false;
+
+	if (IS_ENABLED(CONFIG_FAULT_INJECTION) && beneficial &&
+	    *order >= beneficial &&
+	    should_fail(&beneficial_order_fault_inject, 1)) {
+		*order = beneficial;
+		return NULL;
+	}
+
+	pt = ttm_pool_select_type(pool, caching, *order);
+	if (pt) {
+		p = ttm_pool_type_take(pt, ttm_pool_nid(pool));
+		if (p)
+			return p;	/* pool page already carries @caching */
+	}
+
+	p = ttm_pool_alloc_page(pool, gfp, *order, beneficial_reclaim_backoff);
+	if (!p)
+		return NULL;
+
+	/* Fresh system page is write-back; caller applies @caching in bulk. */
+	*need_caching = true;
+	return p;
+}
+
+/**
+ * ttm_pool_prealloc_fill_full() - Preallocate a whole tt's backing outside any
+ * lock
+ * @pool: The pool to allocate from.
+ * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @pp: Prealloc bag to fill; marked @full and tiled with mixed-order chunks.
+ * @num_pages: Number of 4K pages the tt needs backed.
+ *
+ * Unlike ttm_pool_prealloc_fill() (which parks a bag of interchangeable
+ * beneficial-order defrag chunks), this allocates the ENTIRE backing for a tt
+ * of @num_pages pages up front, at whatever mix of orders is available (pool
+ * pages included), so that the subsequent __ttm_pool_alloc() under the dma-resv
+ * lock merely installs the pages without stalling in reclaim/compaction. The
+ * chunk orders mirror the in-line allocator's order walk; each chunk's order is
+ * recorded on the page itself and read back at consume time.
+ *
+ * May sleep/reclaim freely as it runs unlocked. When
+ * @beneficial_reclaim_backoff is set, higher-order chunks are taken without
+ * aggressive reclaim/compaction: the caller relies on a background
+ * defragmenter to upgrade to beneficial-order pages in place later, so there
+ * is no need to stall the user context reclaiming for contiguity here. A short
+ * fill is fine: the pool falls back to in-line allocation for the shortfall.
+ * DMA-alloc pools are not supported (the bag stays empty). Returns 0 (release
+ * with ttm_pool_prealloc_fini()).
+ */
+int ttm_pool_prealloc_fill_full(struct ttm_pool *pool,
+				enum ttm_caching tt_caching,
+				struct ttm_pool_prealloc *pp,
+				unsigned int num_pages,
+				bool beneficial_reclaim_backoff)
+{
+	gfp_t gfp = ttm_pool_prealloc_gfp(pool) | __GFP_ZERO |
+		__GFP_RETRY_MAYFAIL | __GFP_NOWARN;
+	unsigned int highest = MAX_PAGE_ORDER;
+	unsigned int remaining = num_pages;
+	struct page **cpages = NULL;
+	unsigned int ncpages = 0;
+	int r;
+
+	pp->pages = NULL;
+	pp->order = 0;
+	pp->caching = tt_caching;
+	pp->count = 0;
+	pp->used = 0;
+	pp->full = true;
+
+	/* Caching is applied on the kernel mapping, so DMA-alloc is unsupported. */
+	if (!num_pages || ttm_pool_uses_dma_alloc(pool))
+		return 0;
+
+	/* Worst case is one order-0 chunk per page. */
+	pp->pages = kvzalloc_objs(*pp->pages, num_pages);
+	if (!pp->pages)
+		return 0;
+
+	/*
+	 * Fresh write-back chunks get their caching changed in a single
+	 * set_pages_array_*() after the fill; collect their constituent pages
+	 * into an unpacked scratch array (one entry per 4K page) since the
+	 * packed bag holds mixed-order chunks. Cached tts and pool pages need
+	 * no change (and non-x86 handles caching at map time).
+	 */
+	if (IS_ENABLED(CONFIG_X86) && tt_caching != ttm_cached) {
+		cpages = kvzalloc_objs(*cpages, num_pages);
+		if (!cpages) {
+			kvfree(pp->pages);
+			pp->pages = NULL;
+			return 0;
+		}
+	}
+
+	while (remaining) {
+		unsigned int order = min_t(unsigned int, highest,
+					   __fls(remaining));
+		bool need_caching;
+		struct page *p;
+
+		for (;;) {
+			p = ttm_pool_take_page(pool, tt_caching, &order, gfp,
+					       &need_caching,
+					       beneficial_reclaim_backoff);
+			if (p)
+				break;
+			if (!order)
+				goto apply_caching;	/* short fill */
+			/*
+			 * Genuine failure: retry one order down. A fault
+			 * injection lowered @order to the beneficial order
+			 * above, so this drops it below - landing on a
+			 * sub-optimal (defrag-worthy) chunk.
+			 */
+			order--;
+		}
+
+		pp->pages[pp->count++] = p;
+		if (cpages && need_caching)
+			ttm_pool_prealloc_stage_caching(cpages, &ncpages, p,
+							order);
+		remaining -= 1u << order;
+		highest = order;
+	}
+
+apply_caching:
+	/*
+	 * Apply the requested caching to every collected page in one shot. On
+	 * failure the pages' caching is indeterminate, so drop the whole bag
+	 * (freeing restores write-back) and let the consumer allocate in-line.
+	 */
+	r = ttm_pool_prealloc_apply_caching(tt_caching, cpages, ncpages);
+	kvfree(cpages);
+	if (r) {
+		unsigned int i;
+
+		for (i = 0; i < pp->count; i++) {
+			struct page *p = pp->pages[i];
+
+			ttm_pool_free_page(pool, tt_caching,
+					   ttm_pool_page_order_nodma(p), p,
+					   false);
+		}
+		pp->count = 0;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fill_full);
+
 /**
  * ttm_pool_prealloc_fini() - Release unconsumed preallocated pages
  * @pool: The pool the pages came from.
@@ -1111,9 +1293,13 @@ void ttm_pool_prealloc_fini(struct ttm_pool *pool, struct ttm_pool_prealloc *pp)
 {
 	unsigned int i;
 
-	for (i = pp->used; i < pp->count; ++i)
-		ttm_pool_free_page(pool, pp->caching, pp->order, pp->pages[i],
-				   false);
+	for (i = pp->used; i < pp->count; ++i) {
+		struct page *p = pp->pages[i];
+		unsigned int order = pp->full ?
+			ttm_pool_page_order_nodma(p) : pp->order;
+
+		ttm_pool_free_page(pool, pp->caching, order, p, false);
+	}
 	kvfree(pp->pages);
 	pp->pages = NULL;
 	pp->count = 0;
@@ -1251,9 +1437,10 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 
 /*
  * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries, in order: a beneficial-order page preallocated outside the
- * dma-resv lock (defrag), a same-order pool page, then a fresh system
- * allocation. Fault injection can force the beneficial-order paths to "fail".
+ * failure). Tries, in order: a chunk from a full out-of-lock prealloc (populate
+ * of the whole tt), a beneficial-order page preallocated outside the dma-resv
+ * lock (defrag), a same-order pool page, then a fresh system allocation. Fault
+ * injection can force the beneficial-order paths to "fail".
  */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
@@ -1262,6 +1449,39 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 
 	it->p = NULL;
 
+	/*
+	 * Full prealloc: the entire backing was allocated (pool + system, at
+	 * mixed orders) outside the dma-resv lock. Install the next chunk as-is,
+	 * taking its order from the page. Falls through to in-line allocation
+	 * once the best-effort bag is drained.
+	 */
+	if (pp && pp->full && pp->used < pp->count) {
+		it->fail_beneficial = false;
+		it->p = pp->pages[pp->used++];
+		it->order = ttm_pool_page_order_nodma(it->p);
+		it->page_caching = it->tt->caching;
+
+		/*
+		 * The full prealloc tiles at whatever orders were available
+		 * without stalling in reclaim. A sub-beneficial chunk only
+		 * means a sub-optimal backing (defrag-worthy) when it sits in a
+		 * beneficial-order aligned region that fully fits in the tt -
+		 * that region could have been a beneficial-order page. A low
+		 * order in the unaligned trailing tail is expected and must not
+		 * flag the tt (mirrors ttm_pool_harvest_remaining()).
+		 */
+		if (it->beneficial_order && it->order < it->beneficial_order) {
+			pgoff_t bnr = 1UL << it->beneficial_order;
+			pgoff_t off = it->tt->num_pages -
+				      it->alloc->remaining_pages;
+
+			if (round_down(off, bnr) + bnr <= it->tt->num_pages)
+				it->tt->page_flags |=
+					TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+		}
+		return;
+	}
+
 	/*
 	 * Fault injection: pretend allocation at (or above) the device's
 	 * beneficial order failed, forcing a sub-optimal backing. Exercises the
include/drm/ttm/ttm_bo.h +11 -8
index a80304f179ba..52ec7e5c0d2d 100644
@@ -238,14 +238,17 @@ struct ttm_operation_ctx {
 	 */
 	s64 defrag_bytes_remaining;
 	/**
-	 * @prealloc: Pages preallocated outside the dma-resv lock for a defrag
-	 * move. The pool allocator consumes these instead of allocating fresh
-	 * beneficial-order pages under the lock, moving the (potentially
-	 * reclaim/compaction stalling) high-order allocations out of the
-	 * critical section. NULL means allocate in-line as usual. The pool
-	 * silently falls back to in-line allocation for any shortfall. The
-	 * allocator consumes it for beneficial-order chunks of any populate;
-	 * drivers only set it for a defrag move.
+	 * @prealloc: Pages preallocated outside the dma-resv lock. The pool
+	 * allocator consumes these instead of allocating under the lock, moving
+	 * the (potentially reclaim/compaction stalling) allocations out of the
+	 * critical section. NULL means allocate in-line as usual. Two flavours,
+	 * distinguished by ttm_pool_prealloc::full:
+	 *   - defrag bag (full=false): a bag of interchangeable beneficial-order
+	 *     chunks consumed only for beneficial-order chunks of a defrag move.
+	 *   - full tiling (full=true): the entire backing of the tt, at mixed
+	 *     orders, consumed for every chunk of an ordinary populate.
+	 * The allocator silently falls back to in-line allocation for any
+	 * shortfall.
 	 */
 	struct ttm_pool_prealloc *prealloc;
 	/**
include/drm/ttm/ttm_pool.h +21 -6
index 71670350eb15..777d5aa08f6a 100644
@@ -83,18 +83,27 @@ void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt);
 
 /**
  * struct ttm_pool_prealloc - Pages preallocated outside the dma-resv lock
- * @pages: Array of @count beneficial-order pages (or fewer if a fill fell
- *         short); each entry is the head page of a 1 << @order chunk.
- * @order: Page order of every preallocated chunk.
+ * @pages: Array of @count preallocated chunks; each entry is the head page of a
+ *         chunk. In defrag mode every chunk is @order pages; in @full mode the
+ *         chunks tile a whole tt at mixed orders and the per-chunk order is read
+ *         back from the page itself.
+ * @order: Page order of every preallocated chunk (defrag mode only; 0 in @full
+ *         mode where each chunk carries its own order).
  * @caching: CPU caching applied to the pages, so leftovers can be restored
  *           to write-back before being freed.
  * @count: Number of valid entries in @pages.
  * @used: Number of entries already consumed by the pool allocator.
+ * @full: When set the bag holds a full, mixed-order tiling of an entire tt
+ *        (produced by ttm_pool_prealloc_fill_full()) rather than a bag of
+ *        interchangeable beneficial-order defrag chunks. The pool allocator
+ *        then drains the bag in order for every populate, taking each chunk's
+ *        order from the page, and falls back to in-line allocation only for any
+ *        shortfall.
  *
  * Defrag pages are interchangeable, so only a count of beneficial-order chunks
- * is needed. ttm_pool_prealloc_fill() populates this outside the lock and
- * __ttm_pool_alloc() drains it; any unused tail is released by
- * ttm_pool_prealloc_fini().
+ * is needed. ttm_pool_prealloc_fill() / ttm_pool_prealloc_fill_full() populate
+ * this outside the lock and __ttm_pool_alloc() drains it; any unused tail is
+ * released by ttm_pool_prealloc_fini().
  */
 struct ttm_pool_prealloc {
 	struct page **pages;
@@ -102,10 +111,16 @@ struct ttm_pool_prealloc {
 	enum ttm_caching caching;
 	unsigned int count;
 	unsigned int used;
+	bool full;
 };
 
 int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 			   struct ttm_pool_prealloc *pp, unsigned int count);
+int ttm_pool_prealloc_fill_full(struct ttm_pool *pool,
+				enum ttm_caching tt_caching,
+				struct ttm_pool_prealloc *pp,
+				unsigned int num_pages,
+				bool beneficial_reclaim_backoff);
 void ttm_pool_prealloc_fini(struct ttm_pool *pool,
 			    struct ttm_pool_prealloc *pp);
 unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool);
Signature · 1 line
2.34.1
#34 lore raw

Re: [PATCH v3 10/33] drm/ttm: Add full out-of-lock preallocation for ttm_pool_alloc()

Quoted 163 lines · Matthew Brost

ttm_pool_prealloc_fill() lets a driver preallocate beneficial-order pages outside the dma-resv lock for a defragmentation move, so the high-order allocations that may stall in reclaim/compaction happen off the critical section. That bag is deliberately partial: it holds only interchangeable beneficial-order chunks and the allocator consumes it opportunistically, falling back to in-line allocation for everything else.

An ordinary populate of a system-memory buffer object has the same problem - the whole backing is allocated under the dma-resv lock - but no way to move it out of the critical section, because the existing bag does not cover sub-beneficial orders and does not describe a full tt.

Add a "full" flavour of the same preallocation mechanism:

- struct ttm_pool_prealloc gains a @full flag. In full mode the bag is an ordered tiling of an entire tt at mixed orders (pool pages included), rather than a set of interchangeable beneficial-order chunks. The per-chunk order is recorded on the page itself and read back at consume time, so no parallel order array is needed.

- ttm_pool_prealloc_fill_full() allocates the whole backing for a tt of @num_pages pages up front, mirroring the in-line allocator's order walk (highest order first, capped by the remaining page count). It reuses the existing unlocked helpers (ttm_pool_prealloc_gfp(), ttm_pool_set_caching()) via a new ttm_pool_take_page() helper that tries a same-order pool page before a fresh system allocation.

- ttm_pool_prealloc_fini() frees any unconsumed tail at each chunk's own order when @full.

- ttm_pool_prealloc_fill_full() takes a @beneficial_reclaim_backoff argument, plumbed to ttm_pool_alloc_page(), that lets the caller allocate higher-order chunks without aggressive reclaim/compaction. This suits a driver that upgrades suboptimal pages to beneficial order in place later (e.g. via a background defragmenter) and so has no need to stall the user context reclaiming for contiguity up front.

- __ttm_pool_alloc() consumes a full bag with a single early branch in ttm_pool_iter_acquire_page(): install the next chunk as-is (order taken from the page) and return; once the best-effort bag is drained the loop falls through to normal in-line allocation. The defrag harvest and reuse-old phases are inert here as there is no defrag_old_tt.

Like the defrag bag, this is best-effort: a short fill is fine (the pool allocates the shortfall in-line) and DMA-alloc pools are unsupported (caching is applied on the kernel mapping). No functional change for existing users; the new fill routine has no in-tree caller yet.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---

drivers/gpu/drm/ttm/ttm_pool.c | 232 ++++++++++++++++++++++++++++++++-
include/drm/ttm/ttm_bo.h       |  19 +--
include/drm/ttm/ttm_pool.h     |  27 +++-
3 files changed, 258 insertions(+), 20 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +78 -0
index da7d132c8297..4746f5d8cd7e 100644
@@ -1046,6 +1046,7 @@ int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 	pp->caching = tt_caching;
 	pp->count = 0;
 	pp->used = 0;
+	pp->full = false;
 
 	/* Nothing to gain without a beneficial order or for DMA-alloc pools. */
 	if (!order || !count || ttm_pool_uses_dma_alloc(pool))
@@ -1102,6 +1103,187 @@ int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 }
 EXPORT_SYMBOL(ttm_pool_prealloc_fill);
 
+/*
+ * Acquire a single *@order chunk, trying a same-order pool page first (already
+ * carries @caching) then a fresh system allocation (write-back). Runs unlocked.
+ * When @beneficial_reclaim_backoff is set, higher-order system allocations skip
+ * aggressive reclaim/compaction (see ttm_pool_alloc_page()).
+ *
+ * A fresh system page is write-back and needs the tt's caching applied later in
+ * bulk; *@need_caching is set true for it and false for a (pre-cached) pool
+ * page.
+ *
+ * On a genuine failure *@order is left untouched (the caller lowers it by one
+ * and retries). Fault injection instead pretends allocation at (or above) the
+ * beneficial order failed and lowers *@order to the beneficial order, so the
+ * caller's next attempt lands below it - mirroring the in-line acquire path's
+ * fail_beneficial jump and exercising the beneficial_order_failed tracking
+ * without driving the system into real fragmentation.
+ *
+ * Returns NULL if no page of *@order could be obtained.
+ */
+static struct page *ttm_pool_take_page(struct ttm_pool *pool,
+				       enum ttm_caching caching,
+				       unsigned int *order, gfp_t gfp,
+				       bool *need_caching,
+				       bool beneficial_reclaim_backoff)
+{
+	unsigned int beneficial = ttm_pool_beneficial_order(pool);
+	struct ttm_pool_type *pt;
+	struct page *p;
+
+	*need_caching = false;
+
+	if (IS_ENABLED(CONFIG_FAULT_INJECTION) && beneficial &&
+	    *order >= beneficial &&
+	    should_fail(&beneficial_order_fault_inject, 1)) {
+		*order = beneficial;
+		return NULL;
+	}
+
+	pt = ttm_pool_select_type(pool, caching, *order);
+	if (pt) {
+		p = ttm_pool_type_take(pt, ttm_pool_nid(pool));
+		if (p)
+			return p;	/* pool page already carries @caching */
+	}
+
+	p = ttm_pool_alloc_page(pool, gfp, *order, beneficial_reclaim_backoff);
+	if (!p)
+		return NULL;
+
+	/* Fresh system page is write-back; caller applies @caching in bulk. */
+	*need_caching = true;
+	return p;
+}
+
+/**
+ * ttm_pool_prealloc_fill_full() - Preallocate a whole tt's backing outside any
+ * lock
+ * @pool: The pool to allocate from.
+ * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @pp: Prealloc bag to fill; marked @full and tiled with mixed-order chunks.
+ * @num_pages: Number of 4K pages the tt needs backed.
+ *
+ * Unlike ttm_pool_prealloc_fill() (which parks a bag of interchangeable
+ * beneficial-order defrag chunks), this allocates the ENTIRE backing for a tt
+ * of @num_pages pages up front, at whatever mix of orders is available (pool
+ * pages included), so that the subsequent __ttm_pool_alloc() under the dma-resv
+ * lock merely installs the pages without stalling in reclaim/compaction. The
+ * chunk orders mirror the in-line allocator's order walk; each chunk's order is
+ * recorded on the page itself and read back at consume time.
+ *
+ * May sleep/reclaim freely as it runs unlocked. When
+ * @beneficial_reclaim_backoff is set, higher-order chunks are taken without
+ * aggressive reclaim/compaction: the caller relies on a background
+ * defragmenter to upgrade to beneficial-order pages in place later, so there
+ * is no need to stall the user context reclaiming for contiguity here. A short
+ * fill is fine: the pool falls back to in-line allocation for the shortfall.
+ * DMA-alloc pools are not supported (the bag stays empty). Returns 0 (release

drivers/gpu/drm/ttm/ttm_pool.c int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching, · line 1192

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+ * with ttm_pool_prealloc_fini()).
+ */
+int ttm_pool_prealloc_fill_full(struct ttm_pool *pool,
+				enum ttm_caching tt_caching,
+				struct ttm_pool_prealloc *pp,
+				unsigned int num_pages,
+				bool beneficial_reclaim_backoff)
+{
+	gfp_t gfp = ttm_pool_prealloc_gfp(pool) | __GFP_ZERO |
+		__GFP_RETRY_MAYFAIL | __GFP_NOWARN;
Assisted-by LLM

__GFP_RETRY_MAYFAIL is bit confusing here as as ttm_pool_alloc_page() is expected to suppy it (patch 1). I see that only applies to beneficial order, but here we have mixed-order and hence it is set here. May be add some documentation to clarify it?

Quoted 21 lines
drivers/gpu/drm/ttm/ttm_pool.c +21 -0
+	unsigned int highest = MAX_PAGE_ORDER;
+	unsigned int remaining = num_pages;
+	struct page **cpages = NULL;
+	unsigned int ncpages = 0;
+	int r;
+
+	pp->pages = NULL;
+	pp->order = 0;
+	pp->caching = tt_caching;
+	pp->count = 0;
+	pp->used = 0;
+	pp->full = true;
+
+	/* Caching is applied on the kernel mapping, so DMA-alloc is unsupported. */
+	if (!num_pages || ttm_pool_uses_dma_alloc(pool))
+		return 0;
+
+	/* Worst case is one order-0 chunk per page. */
+	pp->pages = kvzalloc_objs(*pp->pages, num_pages);
+	if (!pp->pages)
+		return 0;

drivers/gpu/drm/ttm/ttm_pool.c int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching, · line 1223

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+
+	/*
+	 * Fresh write-back chunks get their caching changed in a single
+	 * set_pages_array_*() after the fill; collect their constituent pages
+	 * into an unpacked scratch array (one entry per 4K page) since the
+	 * packed bag holds mixed-order chunks. Cached tts and pool pages need
+	 * no change (and non-x86 handles caching at map time).
+	 */
+	if (IS_ENABLED(CONFIG_X86) && tt_caching != ttm_cached) {
+		cpages = kvzalloc_objs(*cpages, num_pages);

Same comment as in patch 9.
cpages is tansient memory and unlike patch 9, it is not bounded by 'budget' here. may be it can be avoided altogether by using a one static 4k page in the pool and flush it with set_pages_array*() as and when it fills up?

Quoted 52 lines
drivers/gpu/drm/ttm/ttm_pool.c +52 -0
+		if (!cpages) {
+			kvfree(pp->pages);
+			pp->pages = NULL;
+			return 0;
+		}
+	}
+
+	while (remaining) {
+		unsigned int order = min_t(unsigned int, highest,
+					   __fls(remaining));
+		bool need_caching;
+		struct page *p;
+
+		for (;;) {
+			p = ttm_pool_take_page(pool, tt_caching, &order, gfp,
+					       &need_caching,
+					       beneficial_reclaim_backoff);
+			if (p)
+				break;
+			if (!order)
+				goto apply_caching;	/* short fill */
+			/*
+			 * Genuine failure: retry one order down. A fault
+			 * injection lowered @order to the beneficial order
+			 * above, so this drops it below - landing on a
+			 * sub-optimal (defrag-worthy) chunk.
+			 */
+			order--;
+		}
+
+		pp->pages[pp->count++] = p;
+		if (cpages && need_caching)
+			ttm_pool_prealloc_stage_caching(cpages, &ncpages, p,
+							order);
+		remaining -= 1u << order;
+		highest = order;
+	}
+
+apply_caching:
+	/*
+	 * Apply the requested caching to every collected page in one shot. On
+	 * failure the pages' caching is indeterminate, so drop the whole bag
+	 * (freeing restores write-back) and let the consumer allocate in-line.
+	 */
+	r = ttm_pool_prealloc_apply_caching(tt_caching, cpages, ncpages);
+	kvfree(cpages);
+	if (r) {
+		unsigned int i;
+
+		for (i = 0; i < pp->count; i++) {
+			struct page *p = pp->pages[i];
+

drivers/gpu/drm/ttm/ttm_pool.c int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching, · line 1285

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+			ttm_pool_free_page(pool, tt_caching,
+					   ttm_pool_page_order_nodma(p), p,
+					   false);
+		}
+		pp->count = 0;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fill_full);

It has lot of code common with ttm_pool_prealloc_fill(). May be extract common code to separate function/s?

Quoted 40 lines
drivers/gpu/drm/ttm/ttm_pool.c +13 -6
+
 /**
 * ttm_pool_prealloc_fini() - Release unconsumed preallocated pages
 * @pool: The pool the pages came from.
@@ -1111,9 +1293,13 @@ void ttm_pool_prealloc_fini(struct ttm_pool *pool, struct ttm_pool_prealloc *pp)
 {
 	unsigned int i;
 
-	for (i = pp->used; i < pp->count; ++i)
-		ttm_pool_free_page(pool, pp->caching, pp->order, pp->pages[i],
-				   false);
+	for (i = pp->used; i < pp->count; ++i) {
+		struct page *p = pp->pages[i];
+		unsigned int order = pp->full ?
+			ttm_pool_page_order_nodma(p) : pp->order;
+
+		ttm_pool_free_page(pool, pp->caching, order, p, false);
+	}
 	kvfree(pp->pages);
 	pp->pages = NULL;
 	pp->count = 0;
@@ -1251,9 +1437,10 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 
 /*
 * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries, in order: a beneficial-order page preallocated outside the
- * dma-resv lock (defrag), a same-order pool page, then a fresh system
- * allocation. Fault injection can force the beneficial-order paths to "fail".
+ * failure). Tries, in order: a chunk from a full out-of-lock prealloc (populate
+ * of the whole tt), a beneficial-order page preallocated outside the dma-resv
+ * lock (defrag), a same-order pool page, then a fresh system allocation. Fault
+ * injection can force the beneficial-order paths to "fail".
 */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
@@ -1262,6 +1449,39 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 
 	it->p = NULL;
 
+	/*

drivers/gpu/drm/ttm/ttm_pool.c static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it) · line 1462

Quoted 10 lines
drivers/gpu/drm/ttm/ttm_pool.c +10 -0
+	 * Full prealloc: the entire backing was allocated (pool + system, at
+	 * mixed orders) outside the dma-resv lock. Install the next chunk as-is,
+	 * taking its order from the page. Falls through to in-line allocation
+	 * once the best-effort bag is drained.
+	 */
+	if (pp && pp->full && pp->used < pp->count) {
+		it->fail_beneficial = false;
+		it->p = pp->pages[pp->used++];
+		it->order = ttm_pool_page_order_nodma(it->p);
+		it->page_caching = it->tt->caching;

Same comment as in 23.
We should ensure caching has not changed here after prefill (ie., pp->caching == it->tt->caching). Though it is not expected to change in gem_create_ioctl call (unlike defragmentation case of patch 23), probably it is still better to check to ensure TTM prealloc() and pool_alloc_page() interfaces work correctly together?

Quoted 113 lines
drivers/gpu/drm/ttm/ttm_pool.c +22 -0
+
+		/*
+		 * The full prealloc tiles at whatever orders were available
+		 * without stalling in reclaim. A sub-beneficial chunk only
+		 * means a sub-optimal backing (defrag-worthy) when it sits in a
+		 * beneficial-order aligned region that fully fits in the tt -
+		 * that region could have been a beneficial-order page. A low
+		 * order in the unaligned trailing tail is expected and must not
+		 * flag the tt (mirrors ttm_pool_harvest_remaining()).
+		 */
+		if (it->beneficial_order && it->order < it->beneficial_order) {
+			pgoff_t bnr = 1UL << it->beneficial_order;
+			pgoff_t off = it->tt->num_pages -
+				      it->alloc->remaining_pages;
+
+			if (round_down(off, bnr) + bnr <= it->tt->num_pages)
+				it->tt->page_flags |=
+					TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+		}
+		return;
+	}
+
 	/*
 	 * Fault injection: pretend allocation at (or above) the device's
 	 * beneficial order failed, forcing a sub-optimal backing. Exercises the
include/drm/ttm/ttm_bo.h +11 -8
index a80304f179ba..52ec7e5c0d2d 100644
@@ -238,14 +238,17 @@ struct ttm_operation_ctx {
 	 */
 	s64 defrag_bytes_remaining;
 	/**
-	 * @prealloc: Pages preallocated outside the dma-resv lock for a defrag
-	 * move. The pool allocator consumes these instead of allocating fresh
-	 * beneficial-order pages under the lock, moving the (potentially
-	 * reclaim/compaction stalling) high-order allocations out of the
-	 * critical section. NULL means allocate in-line as usual. The pool
-	 * silently falls back to in-line allocation for any shortfall. The
-	 * allocator consumes it for beneficial-order chunks of any populate;
-	 * drivers only set it for a defrag move.
+	 * @prealloc: Pages preallocated outside the dma-resv lock. The pool
+	 * allocator consumes these instead of allocating under the lock, moving
+	 * the (potentially reclaim/compaction stalling) allocations out of the
+	 * critical section. NULL means allocate in-line as usual. Two flavours,
+	 * distinguished by ttm_pool_prealloc::full:
+	 *   - defrag bag (full=false): a bag of interchangeable beneficial-order
+	 *     chunks consumed only for beneficial-order chunks of a defrag move.
+	 *   - full tiling (full=true): the entire backing of the tt, at mixed
+	 *     orders, consumed for every chunk of an ordinary populate.
+	 * The allocator silently falls back to in-line allocation for any
+	 * shortfall.
 	 */
 	struct ttm_pool_prealloc *prealloc;
 	/**
include/drm/ttm/ttm_pool.h +21 -6
index 71670350eb15..777d5aa08f6a 100644
@@ -83,18 +83,27 @@ void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt);
 
 /**
 * struct ttm_pool_prealloc - Pages preallocated outside the dma-resv lock
- * @pages: Array of @count beneficial-order pages (or fewer if a fill fell
- *         short); each entry is the head page of a 1 << @order chunk.
- * @order: Page order of every preallocated chunk.
+ * @pages: Array of @count preallocated chunks; each entry is the head page of a
+ *         chunk. In defrag mode every chunk is @order pages; in @full mode the
+ *         chunks tile a whole tt at mixed orders and the per-chunk order is read
+ *         back from the page itself.
+ * @order: Page order of every preallocated chunk (defrag mode only; 0 in @full
+ *         mode where each chunk carries its own order).
 * @caching: CPU caching applied to the pages, so leftovers can be restored
 *           to write-back before being freed.
 * @count: Number of valid entries in @pages.
 * @used: Number of entries already consumed by the pool allocator.
+ * @full: When set the bag holds a full, mixed-order tiling of an entire tt
+ *        (produced by ttm_pool_prealloc_fill_full()) rather than a bag of
+ *        interchangeable beneficial-order defrag chunks. The pool allocator
+ *        then drains the bag in order for every populate, taking each chunk's
+ *        order from the page, and falls back to in-line allocation only for any
+ *        shortfall.
 *
 * Defrag pages are interchangeable, so only a count of beneficial-order chunks
- * is needed. ttm_pool_prealloc_fill() populates this outside the lock and
- * __ttm_pool_alloc() drains it; any unused tail is released by
- * ttm_pool_prealloc_fini().
+ * is needed. ttm_pool_prealloc_fill() / ttm_pool_prealloc_fill_full() populate
+ * this outside the lock and __ttm_pool_alloc() drains it; any unused tail is
+ * released by ttm_pool_prealloc_fini().
 */
 struct ttm_pool_prealloc {
 	struct page **pages;
@@ -102,10 +111,16 @@ struct ttm_pool_prealloc {
 	enum ttm_caching caching;
 	unsigned int count;
 	unsigned int used;
+	bool full;
 };
 
 int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 			   struct ttm_pool_prealloc *pp, unsigned int count);
+int ttm_pool_prealloc_fill_full(struct ttm_pool *pool,
+				enum ttm_caching tt_caching,
+				struct ttm_pool_prealloc *pp,
+				unsigned int num_pages,
+				bool beneficial_reclaim_backoff);
 void ttm_pool_prealloc_fini(struct ttm_pool *pool,
 			    struct ttm_pool_prealloc *pp);
 unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool);
Signature · 1 line
2.34.1
author maintainer
#35 lore raw

Re: [PATCH v3 10/33] drm/ttm: Add full out-of-lock preallocation for ttm_pool_alloc()

Quoted 168 lines · Niranjana Vishwanathapura
Quoted 167 lines · Matthew Brost

ttm_pool_prealloc_fill() lets a driver preallocate beneficial-order pages outside the dma-resv lock for a defragmentation move, so the high-order allocations that may stall in reclaim/compaction happen off the critical section. That bag is deliberately partial: it holds only interchangeable beneficial-order chunks and the allocator consumes it opportunistically, falling back to in-line allocation for everything else.

An ordinary populate of a system-memory buffer object has the same problem - the whole backing is allocated under the dma-resv lock - but no way to move it out of the critical section, because the existing bag does not cover sub-beneficial orders and does not describe a full tt.

Add a "full" flavour of the same preallocation mechanism:

- struct ttm_pool_prealloc gains a @full flag. In full mode the bag is an ordered tiling of an entire tt at mixed orders (pool pages included), rather than a set of interchangeable beneficial-order chunks. The per-chunk order is recorded on the page itself and read back at consume time, so no parallel order array is needed.

- ttm_pool_prealloc_fill_full() allocates the whole backing for a tt of @num_pages pages up front, mirroring the in-line allocator's order walk (highest order first, capped by the remaining page count). It reuses the existing unlocked helpers (ttm_pool_prealloc_gfp(), ttm_pool_set_caching()) via a new ttm_pool_take_page() helper that tries a same-order pool page before a fresh system allocation.

- ttm_pool_prealloc_fini() frees any unconsumed tail at each chunk's own order when @full.

- ttm_pool_prealloc_fill_full() takes a @beneficial_reclaim_backoff argument, plumbed to ttm_pool_alloc_page(), that lets the caller allocate higher-order chunks without aggressive reclaim/compaction. This suits a driver that upgrades suboptimal pages to beneficial order in place later (e.g. via a background defragmenter) and so has no need to stall the user context reclaiming for contiguity up front.

- __ttm_pool_alloc() consumes a full bag with a single early branch in ttm_pool_iter_acquire_page(): install the next chunk as-is (order taken from the page) and return; once the best-effort bag is drained the loop falls through to normal in-line allocation. The defrag harvest and reuse-old phases are inert here as there is no defrag_old_tt.

Like the defrag bag, this is best-effort: a short fill is fine (the pool allocates the shortfall in-line) and DMA-alloc pools are unsupported (caching is applied on the kernel mapping). No functional change for existing users; the new fill routine has no in-tree caller yet.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---

drivers/gpu/drm/ttm/ttm_pool.c | 232 ++++++++++++++++++++++++++++++++-
include/drm/ttm/ttm_bo.h       |  19 +--
include/drm/ttm/ttm_pool.h     |  27 +++-
3 files changed, 258 insertions(+), 20 deletions(-)
drivers/gpu/drm/ttm/ttm_pool.c +82 -0
index da7d132c8297..4746f5d8cd7e 100644
@@ -1046,6 +1046,7 @@ int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 	pp->caching = tt_caching;
 	pp->count = 0;
 	pp->used = 0;
+	pp->full = false;
 
 	/* Nothing to gain without a beneficial order or for DMA-alloc pools. */
 	if (!order || !count || ttm_pool_uses_dma_alloc(pool))
@@ -1102,6 +1103,187 @@ int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 }
 EXPORT_SYMBOL(ttm_pool_prealloc_fill);
 
+/*
+ * Acquire a single *@order chunk, trying a same-order pool page first (already
+ * carries @caching) then a fresh system allocation (write-back). Runs unlocked.
+ * When @beneficial_reclaim_backoff is set, higher-order system allocations skip
+ * aggressive reclaim/compaction (see ttm_pool_alloc_page()).
+ *
+ * A fresh system page is write-back and needs the tt's caching applied later in
+ * bulk; *@need_caching is set true for it and false for a (pre-cached) pool
+ * page.
+ *
+ * On a genuine failure *@order is left untouched (the caller lowers it by one
+ * and retries). Fault injection instead pretends allocation at (or above) the
+ * beneficial order failed and lowers *@order to the beneficial order, so the
+ * caller's next attempt lands below it - mirroring the in-line acquire path's
+ * fail_beneficial jump and exercising the beneficial_order_failed tracking
+ * without driving the system into real fragmentation.
+ *
+ * Returns NULL if no page of *@order could be obtained.
+ */
+static struct page *ttm_pool_take_page(struct ttm_pool *pool,
+				       enum ttm_caching caching,
+				       unsigned int *order, gfp_t gfp,
+				       bool *need_caching,
+				       bool beneficial_reclaim_backoff)
+{
+	unsigned int beneficial = ttm_pool_beneficial_order(pool);
+	struct ttm_pool_type *pt;
+	struct page *p;
+
+	*need_caching = false;
+
+	if (IS_ENABLED(CONFIG_FAULT_INJECTION) && beneficial &&
+	    *order >= beneficial &&
+	    should_fail(&beneficial_order_fault_inject, 1)) {
+		*order = beneficial;
+		return NULL;
+	}
+
+	pt = ttm_pool_select_type(pool, caching, *order);
+	if (pt) {
+		p = ttm_pool_type_take(pt, ttm_pool_nid(pool));
+		if (p)
+			return p;	/* pool page already carries @caching */
+	}
+
+	p = ttm_pool_alloc_page(pool, gfp, *order, beneficial_reclaim_backoff);
+	if (!p)
+		return NULL;
+
+	/* Fresh system page is write-back; caller applies @caching in bulk. */
+	*need_caching = true;
+	return p;
+}
+
+/**
+ * ttm_pool_prealloc_fill_full() - Preallocate a whole tt's backing outside any
+ * lock
+ * @pool: The pool to allocate from.
+ * @tt_caching: The requested cpu-caching for the pages allocated.
+ * @pp: Prealloc bag to fill; marked @full and tiled with mixed-order chunks.
+ * @num_pages: Number of 4K pages the tt needs backed.
+ *
+ * Unlike ttm_pool_prealloc_fill() (which parks a bag of interchangeable
+ * beneficial-order defrag chunks), this allocates the ENTIRE backing for a tt
+ * of @num_pages pages up front, at whatever mix of orders is available (pool
+ * pages included), so that the subsequent __ttm_pool_alloc() under the dma-resv
+ * lock merely installs the pages without stalling in reclaim/compaction. The
+ * chunk orders mirror the in-line allocator's order walk; each chunk's order is
+ * recorded on the page itself and read back at consume time.
+ *
+ * May sleep/reclaim freely as it runs unlocked. When
+ * @beneficial_reclaim_backoff is set, higher-order chunks are taken without
+ * aggressive reclaim/compaction: the caller relies on a background
+ * defragmenter to upgrade to beneficial-order pages in place later, so there
+ * is no need to stall the user context reclaiming for contiguity here. A short
+ * fill is fine: the pool falls back to in-line allocation for the shortfall.
+ * DMA-alloc pools are not supported (the bag stays empty). Returns 0 (release
+ * with ttm_pool_prealloc_fini()).
+ */
+int ttm_pool_prealloc_fill_full(struct ttm_pool *pool,
+				enum ttm_caching tt_caching,
Quoted 9 lines

drivers/gpu/drm/ttm/ttm_pool.c int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching, · line 1192

Quoted 6 lines
drivers/gpu/drm/ttm/ttm_pool.c +6 -0
+				struct ttm_pool_prealloc *pp,
+				unsigned int num_pages,
+				bool beneficial_reclaim_backoff)
+{
+	gfp_t gfp = ttm_pool_prealloc_gfp(pool) | __GFP_ZERO |
+		__GFP_RETRY_MAYFAIL | __GFP_NOWARN;
Assisted-by LLM

Yes.

Quoted 5 lines

__GFP_RETRY_MAYFAIL is bit confusing here as as ttm_pool_alloc_page() is expected to suppy it (patch 1). I see that only applies to beneficial order, but here we have mixed-order and hence it is set here. May be add some documentation to clarify it?

Ye, this is a bit confusing. ttm_pool_alloc_page will change these flags based beneficial_reclaim_backoff / beneficial_order. In general I chose to set __GFP_RETRY_MAYFAIL here as a default to avoid the prealloc to avoid any chance of triggering OOM - i.e., if prealloc fails for whatever reason we fall back the traditional TTM alloc path which I think is correct.

So maybe '__GFP_RETRY_MAYFAIL is set to never trigger OOM on prealloc'.

Quoted 26 lines
Quoted 26 lines
drivers/gpu/drm/ttm/ttm_pool.c +26 -0
+	unsigned int highest = MAX_PAGE_ORDER;
+	unsigned int remaining = num_pages;
+	struct page **cpages = NULL;
+	unsigned int ncpages = 0;
+	int r;
+
+	pp->pages = NULL;
+	pp->order = 0;
+	pp->caching = tt_caching;
+	pp->count = 0;
+	pp->used = 0;
+	pp->full = true;
+
+	/* Caching is applied on the kernel mapping, so DMA-alloc is unsupported. */
+	if (!num_pages || ttm_pool_uses_dma_alloc(pool))
+		return 0;
+
+	/* Worst case is one order-0 chunk per page. */
+	pp->pages = kvzalloc_objs(*pp->pages, num_pages);
+	if (!pp->pages)
+		return 0;
+
+	/*
+	 * Fresh write-back chunks get their caching changed in a single
+	 * set_pages_array_*() after the fill; collect their constituent pages
+	 * into an unpacked scratch array (one entry per 4K page) since the
Quoted 8 lines

drivers/gpu/drm/ttm/ttm_pool.c int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching, · line 1223

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_pool.c +5 -0
+	 * packed bag holds mixed-order chunks. Cached tts and pool pages need
+	 * no change (and non-x86 handles caching at map time).
+	 */
+	if (IS_ENABLED(CONFIG_X86) && tt_caching != ttm_cached) {
+		cpages = kvzalloc_objs(*cpages, num_pages);

Same comment as in patch 9.
cpages is tansient memory and unlike patch 9, it is not bounded by 'budget' here.

It is not bounded by any budget rather by the number pages in the BO size.

Quoted 3 lines

may be it can be avoided altogether by using a one static 4k page in the pool and flush it with set_pages_array*() as and when it fills up?

I think that is the wrong trade. set_pages_array on x86 does some hardware interaction which is rather expensive, thus the entire reason TTM pool exist to avoid calling set_pages_array on every alloc / free. i.e., more tansient memory vs 1 hardware interaction is the right trade.

Quoted 57 lines
Quoted 57 lines
drivers/gpu/drm/ttm/ttm_pool.c +57 -0
+		if (!cpages) {
+			kvfree(pp->pages);
+			pp->pages = NULL;
+			return 0;
+		}
+	}
+
+	while (remaining) {
+		unsigned int order = min_t(unsigned int, highest,
+					   __fls(remaining));
+		bool need_caching;
+		struct page *p;
+
+		for (;;) {
+			p = ttm_pool_take_page(pool, tt_caching, &order, gfp,
+					       &need_caching,
+					       beneficial_reclaim_backoff);
+			if (p)
+				break;
+			if (!order)
+				goto apply_caching;	/* short fill */
+			/*
+			 * Genuine failure: retry one order down. A fault
+			 * injection lowered @order to the beneficial order
+			 * above, so this drops it below - landing on a
+			 * sub-optimal (defrag-worthy) chunk.
+			 */
+			order--;
+		}
+
+		pp->pages[pp->count++] = p;
+		if (cpages && need_caching)
+			ttm_pool_prealloc_stage_caching(cpages, &ncpages, p,
+							order);
+		remaining -= 1u << order;
+		highest = order;
+	}
+
+apply_caching:
+	/*
+	 * Apply the requested caching to every collected page in one shot. On
+	 * failure the pages' caching is indeterminate, so drop the whole bag
+	 * (freeing restores write-back) and let the consumer allocate in-line.
+	 */
+	r = ttm_pool_prealloc_apply_caching(tt_caching, cpages, ncpages);
+	kvfree(cpages);
+	if (r) {
+		unsigned int i;
+
+		for (i = 0; i < pp->count; i++) {
+			struct page *p = pp->pages[i];
+
+			ttm_pool_free_page(pool, tt_caching,
+					   ttm_pool_page_order_nodma(p), p,
+					   false);
+		}
+		pp->count = 0;
Quoted 8 lines

drivers/gpu/drm/ttm/ttm_pool.c int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching, · line 1285

Quoted 5 lines
drivers/gpu/drm/ttm/ttm_pool.c +5 -0
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL(ttm_pool_prealloc_fill_full);

It has lot of code common with ttm_pool_prealloc_fill(). May be extract common code to separate function/s?

I see what you mean, but the algorithms are different. One tries to obtain N beneficially ordered pages, while the other fills an array with any set of pages regardless of order.

I'm hesitant to force the code into helpers or create a single unified function with conditionals just to save 70 lines of code. In my opinion, the abstraction is already at the right level.

Quoted 50 lines
Quoted 50 lines
drivers/gpu/drm/ttm/ttm_pool.c +22 -6
+
 /**
 * ttm_pool_prealloc_fini() - Release unconsumed preallocated pages
 * @pool: The pool the pages came from.
@@ -1111,9 +1293,13 @@ void ttm_pool_prealloc_fini(struct ttm_pool *pool, struct ttm_pool_prealloc *pp)
 {
 	unsigned int i;
 
-	for (i = pp->used; i < pp->count; ++i)
-		ttm_pool_free_page(pool, pp->caching, pp->order, pp->pages[i],
-				   false);
+	for (i = pp->used; i < pp->count; ++i) {
+		struct page *p = pp->pages[i];
+		unsigned int order = pp->full ?
+			ttm_pool_page_order_nodma(p) : pp->order;
+
+		ttm_pool_free_page(pool, pp->caching, order, p, false);
+	}
 	kvfree(pp->pages);
 	pp->pages = NULL;
 	pp->count = 0;
@@ -1251,9 +1437,10 @@ static int ttm_pool_iter_reuse_old(struct ttm_pool_alloc_iter *it)
 
 /*
 * Acquire a single page for the current order, leaving it in @it->p (NULL on
- * failure). Tries, in order: a beneficial-order page preallocated outside the
- * dma-resv lock (defrag), a same-order pool page, then a fresh system
- * allocation. Fault injection can force the beneficial-order paths to "fail".
+ * failure). Tries, in order: a chunk from a full out-of-lock prealloc (populate
+ * of the whole tt), a beneficial-order page preallocated outside the dma-resv
+ * lock (defrag), a same-order pool page, then a fresh system allocation. Fault
+ * injection can force the beneficial-order paths to "fail".
 */
 static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 {
@@ -1262,6 +1449,39 @@ static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it)
 
 	it->p = NULL;
 
+	/*
+	 * Full prealloc: the entire backing was allocated (pool + system, at
+	 * mixed orders) outside the dma-resv lock. Install the next chunk as-is,
+	 * taking its order from the page. Falls through to in-line allocation
+	 * once the best-effort bag is drained.
+	 */
+	if (pp && pp->full && pp->used < pp->count) {
+		it->fail_beneficial = false;
+		it->p = pp->pages[pp->used++];
+		it->order = ttm_pool_page_order_nodma(it->p);
Quoted 9 lines

drivers/gpu/drm/ttm/ttm_pool.c static void ttm_pool_iter_acquire_page(struct ttm_pool_alloc_iter *it) · line 1462

Quoted 1 line
drivers/gpu/drm/ttm/ttm_pool.c +1 -0
+		it->page_caching = it->tt->caching;

Same comment as in 23.
We should ensure caching has not changed here after prefill (ie., pp->caching == it->tt->caching). Though it is not expected to change in gem_create_ioctl call (unlike defragmentation case of patch 23), probably it is still better to check to ensure TTM prealloc() and pool_alloc_page() interfaces work correctly together?

We don't have formalized assert framework in TTM like we do in Xe but perhaps we should - I'd like add one in a follow up as there as so many undocumented invarients in TTM...

So...

/* TODO: formalize invarients in a common TTM assert framework */ WARN_ON(it->tt->caching != pp->pp->caching)?

Matt

Quoted 113 lines
Quoted 113 lines
drivers/gpu/drm/ttm/ttm_pool.c +22 -0
+
+		/*
+		 * The full prealloc tiles at whatever orders were available
+		 * without stalling in reclaim. A sub-beneficial chunk only
+		 * means a sub-optimal backing (defrag-worthy) when it sits in a
+		 * beneficial-order aligned region that fully fits in the tt -
+		 * that region could have been a beneficial-order page. A low
+		 * order in the unaligned trailing tail is expected and must not
+		 * flag the tt (mirrors ttm_pool_harvest_remaining()).
+		 */
+		if (it->beneficial_order && it->order < it->beneficial_order) {
+			pgoff_t bnr = 1UL << it->beneficial_order;
+			pgoff_t off = it->tt->num_pages -
+				      it->alloc->remaining_pages;
+
+			if (round_down(off, bnr) + bnr <= it->tt->num_pages)
+				it->tt->page_flags |=
+					TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED;
+		}
+		return;
+	}
+
 	/*
 	 * Fault injection: pretend allocation at (or above) the device's
 	 * beneficial order failed, forcing a sub-optimal backing. Exercises the
include/drm/ttm/ttm_bo.h +11 -8
index a80304f179ba..52ec7e5c0d2d 100644
@@ -238,14 +238,17 @@ struct ttm_operation_ctx {
 	 */
 	s64 defrag_bytes_remaining;
 	/**
-	 * @prealloc: Pages preallocated outside the dma-resv lock for a defrag
-	 * move. The pool allocator consumes these instead of allocating fresh
-	 * beneficial-order pages under the lock, moving the (potentially
-	 * reclaim/compaction stalling) high-order allocations out of the
-	 * critical section. NULL means allocate in-line as usual. The pool
-	 * silently falls back to in-line allocation for any shortfall. The
-	 * allocator consumes it for beneficial-order chunks of any populate;
-	 * drivers only set it for a defrag move.
+	 * @prealloc: Pages preallocated outside the dma-resv lock. The pool
+	 * allocator consumes these instead of allocating under the lock, moving
+	 * the (potentially reclaim/compaction stalling) allocations out of the
+	 * critical section. NULL means allocate in-line as usual. Two flavours,
+	 * distinguished by ttm_pool_prealloc::full:
+	 *   - defrag bag (full=false): a bag of interchangeable beneficial-order
+	 *     chunks consumed only for beneficial-order chunks of a defrag move.
+	 *   - full tiling (full=true): the entire backing of the tt, at mixed
+	 *     orders, consumed for every chunk of an ordinary populate.
+	 * The allocator silently falls back to in-line allocation for any
+	 * shortfall.
 	 */
 	struct ttm_pool_prealloc *prealloc;
 	/**
include/drm/ttm/ttm_pool.h +21 -6
index 71670350eb15..777d5aa08f6a 100644
@@ -83,18 +83,27 @@ void ttm_pool_free(struct ttm_pool *pool, struct ttm_tt *tt);
 
 /**
 * struct ttm_pool_prealloc - Pages preallocated outside the dma-resv lock
- * @pages: Array of @count beneficial-order pages (or fewer if a fill fell
- *         short); each entry is the head page of a 1 << @order chunk.
- * @order: Page order of every preallocated chunk.
+ * @pages: Array of @count preallocated chunks; each entry is the head page of a
+ *         chunk. In defrag mode every chunk is @order pages; in @full mode the
+ *         chunks tile a whole tt at mixed orders and the per-chunk order is read
+ *         back from the page itself.
+ * @order: Page order of every preallocated chunk (defrag mode only; 0 in @full
+ *         mode where each chunk carries its own order).
 * @caching: CPU caching applied to the pages, so leftovers can be restored
 *           to write-back before being freed.
 * @count: Number of valid entries in @pages.
 * @used: Number of entries already consumed by the pool allocator.
+ * @full: When set the bag holds a full, mixed-order tiling of an entire tt
+ *        (produced by ttm_pool_prealloc_fill_full()) rather than a bag of
+ *        interchangeable beneficial-order defrag chunks. The pool allocator
+ *        then drains the bag in order for every populate, taking each chunk's
+ *        order from the page, and falls back to in-line allocation only for any
+ *        shortfall.
 *
 * Defrag pages are interchangeable, so only a count of beneficial-order chunks
- * is needed. ttm_pool_prealloc_fill() populates this outside the lock and
- * __ttm_pool_alloc() drains it; any unused tail is released by
- * ttm_pool_prealloc_fini().
+ * is needed. ttm_pool_prealloc_fill() / ttm_pool_prealloc_fill_full() populate
+ * this outside the lock and __ttm_pool_alloc() drains it; any unused tail is
+ * released by ttm_pool_prealloc_fini().
 */
 struct ttm_pool_prealloc {
 	struct page **pages;
@@ -102,10 +111,16 @@ struct ttm_pool_prealloc {
 	enum ttm_caching caching;
 	unsigned int count;
 	unsigned int used;
+	bool full;
 };
 
 int ttm_pool_prealloc_fill(struct ttm_pool *pool, enum ttm_caching tt_caching,
 			   struct ttm_pool_prealloc *pp, unsigned int count);
+int ttm_pool_prealloc_fill_full(struct ttm_pool *pool,
+				enum ttm_caching tt_caching,
+				struct ttm_pool_prealloc *pp,
+				unsigned int num_pages,
+				bool beneficial_reclaim_backoff);
 void ttm_pool_prealloc_fini(struct ttm_pool *pool,
 			    struct ttm_pool_prealloc *pp);
 unsigned int ttm_pool_prealloc_order(struct ttm_pool *pool);
Signature · 1 line
2.34.1
author maintainer
#36 lore raw

PATCH v3 11/33 drm/gpusvm: Add a DMA-mapping accounting callback

In reply to Matthew Brost

From: Thomas Hellström <[email protected]>

Drivers that use the GPU SVM core to DMA-map system memory for GPU access may want to keep track of how many pages they have mapped, for example to expose statistics or to detect leaks. Doing this accounting in the driver around drm_gpusvm_get_pages() / drm_gpusvm_unmap_pages() is error prone: the number, order and kind of the dma_addr[] entries can change between map and unmap (migration, partial unmaps, the iova vs non-iova paths), which makes symmetric accounting hard to get right.

Add an optional @dma_map_account callback to struct drm_gpusvm_ops and invoke it once per &drm_pagemap_addr entry at the exact points where the entry is DMA-mapped (sign == +1) in drm_gpusvm_get_pages() and unmapped (sign == -1) in __drm_gpusvm_unmap_pages(). Since the map error path and every unmap/free path funnel through __drm_gpusvm_unmap_pages() for the entries that were actually mapped, the accounting is symmetric by construction. The callback receives the full &drm_pagemap_addr so the driver can use the order and the proto field to distinguish system memory mappings from device interconnect mappings.

The callback must also be usable by the core drm_gpusvm_pages-only API (mm == NULL), as used e.g. for userptr-style mappings. Relax drm_gpusvm_init() to allow such users to pass a restricted @ops that only implements the page-level hooks; the full-SVM hooks (@invalidate) and the chunk configuration must still be unset in that mode.

Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/drm_gpusvm.c | 17 +++++++++++++++--
 include/drm/drm_gpusvm.h     | 19 +++++++++++++++++++
 2 files changed, 34 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_gpusvm.c +15 -2
index fcfe635bc195..512f52471419 100644
@@ -474,8 +474,15 @@ int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
 			return -EINVAL;
 		mmgrab(mm);
 	} else {
-		/* No full SVM mode, only core drm_gpusvm_pages API. */
-		if (ops || num_chunks || mm_range || notifier_size)
+		/*
+		 * No full SVM mode, only core drm_gpusvm_pages API. A
+		 * restricted @ops that only implements the page-level hooks
+		 * (e.g. @dma_map_account) is allowed; the full-SVM hooks must
+		 * not be set.
+		 */
+		if (num_chunks || mm_range || notifier_size)
+			return -EINVAL;
+		if (ops && ops->invalidate)
 			return -EINVAL;
 	}
 
@@ -1246,6 +1253,8 @@ static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm,
 			else if (dpagemap && dpagemap->ops->device_unmap)
 				dpagemap->ops->device_unmap(dpagemap,
 							    dev, addr);
+			if (gpusvm->ops && gpusvm->ops->dma_map_account)
+				gpusvm->ops->dma_map_account(gpusvm, addr, -1);
 			i += 1 << addr->order;
 		}
 
@@ -1654,6 +1663,10 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
 				(addr, DRM_INTERCONNECT_SYSTEM, order,
 				 dma_dir);
 		}
+		if (gpusvm->ops && gpusvm->ops->dma_map_account)
+			gpusvm->ops->dma_map_account(gpusvm,
+						     &svm_pages->dma_addr[j],
+						     1);
 		i += 1 << order;
 		num_dma_mapped = i;
 		flags.has_dma_mapping = true;
include/drm/drm_gpusvm.h +19 -0
index b7d987bf76aa..36f35816a569 100644
@@ -75,6 +75,25 @@ struct drm_gpusvm_ops {
 	void (*invalidate)(struct drm_gpusvm *gpusvm,
 			   struct drm_gpusvm_notifier *notifier,
 			   const struct mmu_notifier_range *mmu_range);
+
+	/**
+	 * @dma_map_account: Account a DMA-mapping change (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @addr: The address descriptor of the chunk being (un)mapped
+	 * @sign: +1 when @addr has just been DMA-mapped, -1 when it is being
+	 *        unmapped
+	 *
+	 * Called once per &drm_pagemap_addr entry, when the entry is
+	 * DMA-mapped by drm_gpusvm_get_pages() (@sign == +1) and when it is
+	 * unmapped (@sign == -1), so the driver can keep symmetric accounting
+	 * of the pages it has mapped for GPU access. The @addr->proto field
+	 * can be used to distinguish system-memory mappings from device
+	 * interconnect mappings. May be provided in both full SVM mode and the
+	 * core drm_gpusvm_pages-only mode.
+	 */
+	void (*dma_map_account)(struct drm_gpusvm *gpusvm,
+				const struct drm_pagemap_addr *addr,
+				int sign);
 };
 
 /**
Signature · 1 line
2.34.1
#37 lore raw

Re: [PATCH v3 11/33] drm/gpusvm: Add a DMA-mapping accounting callback

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Pre-existing issues:
- [High] Uninitialized memory read of svm_pages->dma_addr[0].dir passed to dma_iova_destroy() on the error path.
- [High] System memory pages are erroneously unmapped using the device-specific unmap callback in mixed SVM ranges.
- [High] Device memory DMA mappings are leaked on the error path because svm_pages->dpagemap is assigned too late. --

--- Patch [11]: [PATCH v3 11/33] drm/gpusvm: Add a DMA-mapping accounting callback --- commit c12d7531ab5aab8f1172b95a5e7473303dec85e1
Author: Thomas Hellström <[email protected]>

drm/gpusvm: Add a DMA-mapping accounting callback

The patch adds a dma_map_account callback to drm_gpusvm_ops to allow drivers to symmetrically account for DMA-mapped pages during map and unmap operations.

Quoted 4 lines
drivers/gpu/drm/drm_gpusvm.c +0 -0
index fcfe635bc1956..512f52471419e 100644

[ ... ]

drivers/gpu/drm/drm_gpusvm.c static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm, · line 1255

Quoted 4 lines
drivers/gpu/drm/drm_gpusvm.c +0 -0
@@ -1246,6 +1253,8 @@ static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm,
 			else if (dpagemap && dpagemap->ops->device_unmap)
 				dpagemap->ops->device_unmap(dpagemap,
 							    dev, addr);

[Severity: High]
This isn't a bug introduced by this patch, but does the error path in drm_gpusvm_get_pages() trigger an uninitialized memory read here?

If the first page's mapping fails (e.g., dma_iova_link() fails), the code jumps to the err_unmap label with num_dma_mapped = 0. Because dma_iova_try_alloc() succeeded, use_iova evaluates to true in __drm_gpusvm_unmap_pages(). This unconditionally executes dma_iova_destroy() just before this loop, reading svm_pages->dma_addr[0].dir by value:

    if (use_iova)
        dma_iova_destroy(dev, &svm_pages->state, svm_pages->state_offset,
                         svm_pages->dma_addr[0].dir, 0);

Since svm_pages->dma_addr was allocated using kvmalloc_objs without zero-initialization, this passes an uninitialized enum dma_data_direction into a kernel DMA API, which can trigger KMSAN panics or BUG_ON checks in DMA debugging routines.

[Severity: High]
This is a pre-existing issue, but does this logic misroute system memory pages to the device unmap callback?

When ctx->allow_mixed is true, an SVM range can contain both system and device pages. In this scenario, use_iova evaluates to true, and svm_pages->dpagemap is initialized to the device's pagemap. During __drm_gpusvm_unmap_pages(), the loop iterates over all pages. For a system page, addr->proto == DRM_INTERCONNECT_SYSTEM is true, but !use_iova evaluates to false.

This causes the execution to fall through to the else if condition without validating the page type:

    else if (dpagemap && dpagemap->ops->device_unmap)
        dpagemap->ops->device_unmap(dpagemap, dev, addr);

This incorrectly executes the device's unmap callback on a system memory page (containing an IOVA address), which seems likely to cause memory corruption or a crash.

[Severity: High]
This is also a pre-existing issue, but are device memory DMA mappings leaked on the error path because svm_pages->dpagemap is assigned too late?

If an error occurs during drm_gpusvm_get_pages() after successfully mapping one or more device private pages, the code jumps to the err_unmap cleanup path. However, svm_pages->dpagemap is only updated at the very end of the success path:

    if (pagemap) {
        flags.has_devmem_pages = true;
        drm_pagemap_get(dpagemap);
        drm_pagemap_put(svm_pages->dpagemap);
        svm_pages->dpagemap = dpagemap;
    }

Consequently, inside __drm_gpusvm_unmap_pages(), the local dpagemap variable is NULL, causing the condition dpagemap && dpagemap->ops->device_unmap to evaluate to false. The device unmap callback is skipped, leaking the device memory mappings. The newly added dma_map_account callback will un-account these leaked pages, causing internal accounting to irreversibly drift from the hardware state.

Quoted 4 lines
drivers/gpu/drm/drm_gpusvm.c +2 -0
+			if (gpusvm->ops && gpusvm->ops->dma_map_account)
+				gpusvm->ops->dma_map_account(gpusvm, addr, -1);
 			i += 1 << addr->order;
 		}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
Reviewed-by on v3 11/33
#38 lore raw

Re: [PATCH v3 11/33] drm/gpusvm: Add a DMA-mapping accounting callback

In reply to Matthew Brost
Quoted 20 lines · Matthew Brost

From: Thomas Hellström <[email protected]>

Drivers that use the GPU SVM core to DMA-map system memory for GPU access may want to keep track of how many pages they have mapped, for example to expose statistics or to detect leaks. Doing this accounting in the driver around drm_gpusvm_get_pages() / drm_gpusvm_unmap_pages() is error prone: the number, order and kind of the dma_addr[] entries can change between map and unmap (migration, partial unmaps, the iova vs non-iova paths), which makes symmetric accounting hard to get right.

Add an optional @dma_map_account callback to struct drm_gpusvm_ops and invoke it once per &drm_pagemap_addr entry at the exact points where the entry is DMA-mapped (sign == +1) in drm_gpusvm_get_pages() and unmapped (sign == -1) in __drm_gpusvm_unmap_pages(). Since the map error path and every unmap/free path funnel through __drm_gpusvm_unmap_pages() for the entries that were actually mapped, the accounting is symmetric by construction. The callback receives the full &drm_pagemap_addr so the driver can use the order and the proto field to distinguish system memory mappings from device interconnect mappings.

Quoted 8 lines

The callback must also be usable by the core drm_gpusvm_pages-only API (mm == NULL), as used e.g. for userptr-style mappings. Relax drm_gpusvm_init() to allow such users to pass a restricted @ops that only implements the page-level hooks; the full-SVM hooks (@invalidate) and the chunk configuration must still be unset in that mode.

Assisted-by GitHub_Copilot:claude-opus-4.8

Patches 11 and 12 can be spinned out as a separate patch series (for fast tracking).

LGTM.

Quoted 80 lines

---
drivers/gpu/drm/drm_gpusvm.c | 17 +++++++++++++++--
include/drm/drm_gpusvm.h | 19 +++++++++++++++++++

2 files changed, 34 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_gpusvm.c +15 -2
index fcfe635bc195..512f52471419 100644
@@ -474,8 +474,15 @@ int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
 			return -EINVAL;
 		mmgrab(mm);
 	} else {
-		/* No full SVM mode, only core drm_gpusvm_pages API. */
-		if (ops || num_chunks || mm_range || notifier_size)
+		/*
+		 * No full SVM mode, only core drm_gpusvm_pages API. A
+		 * restricted @ops that only implements the page-level hooks
+		 * (e.g. @dma_map_account) is allowed; the full-SVM hooks must
+		 * not be set.
+		 */
+		if (num_chunks || mm_range || notifier_size)
+			return -EINVAL;
+		if (ops && ops->invalidate)
 			return -EINVAL;
 	}
 
@@ -1246,6 +1253,8 @@ static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm,
 			else if (dpagemap && dpagemap->ops->device_unmap)
 				dpagemap->ops->device_unmap(dpagemap,
 							    dev, addr);
+			if (gpusvm->ops && gpusvm->ops->dma_map_account)
+				gpusvm->ops->dma_map_account(gpusvm, addr, -1);
 			i += 1 << addr->order;
 		}
 
@@ -1654,6 +1663,10 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
 				(addr, DRM_INTERCONNECT_SYSTEM, order,
 				 dma_dir);
 		}
+		if (gpusvm->ops && gpusvm->ops->dma_map_account)
+			gpusvm->ops->dma_map_account(gpusvm,
+						     &svm_pages->dma_addr[j],
+						     1);
 		i += 1 << order;
 		num_dma_mapped = i;
 		flags.has_dma_mapping = true;
include/drm/drm_gpusvm.h +19 -0
index b7d987bf76aa..36f35816a569 100644
@@ -75,6 +75,25 @@ struct drm_gpusvm_ops {
 	void (*invalidate)(struct drm_gpusvm *gpusvm,
 			   struct drm_gpusvm_notifier *notifier,
 			   const struct mmu_notifier_range *mmu_range);
+
+	/**
+	 * @dma_map_account: Account a DMA-mapping change (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @addr: The address descriptor of the chunk being (un)mapped
+	 * @sign: +1 when @addr has just been DMA-mapped, -1 when it is being
+	 *        unmapped
+	 *
+	 * Called once per &drm_pagemap_addr entry, when the entry is
+	 * DMA-mapped by drm_gpusvm_get_pages() (@sign == +1) and when it is
+	 * unmapped (@sign == -1), so the driver can keep symmetric accounting
+	 * of the pages it has mapped for GPU access. The @addr->proto field
+	 * can be used to distinguish system-memory mappings from device
+	 * interconnect mappings. May be provided in both full SVM mode and the
+	 * core drm_gpusvm_pages-only mode.
+	 */
+	void (*dma_map_account)(struct drm_gpusvm *gpusvm,
+				const struct drm_pagemap_addr *addr,
+				int sign);
 };
 
 /**
Signature · 1 line
2.34.1
#39 lore raw

Re: [PATCH v3 11/33] drm/gpusvm: Add a DMA-mapping accounting callback

Quoted 27 lines · Niranjana Vishwanathapura
Quoted 26 lines · Matthew Brost

From: Thomas Hellström <[email protected]>

Drivers that use the GPU SVM core to DMA-map system memory for GPU access may want to keep track of how many pages they have mapped, for example to expose statistics or to detect leaks. Doing this accounting in the driver around drm_gpusvm_get_pages() / drm_gpusvm_unmap_pages() is error prone: the number, order and kind of the dma_addr[] entries can change between map and unmap (migration, partial unmaps, the iova vs non-iova paths), which makes symmetric accounting hard to get right.

Add an optional @dma_map_account callback to struct drm_gpusvm_ops and invoke it once per &drm_pagemap_addr entry at the exact points where the entry is DMA-mapped (sign == +1) in drm_gpusvm_get_pages() and unmapped (sign == -1) in __drm_gpusvm_unmap_pages(). Since the map error path and every unmap/free path funnel through __drm_gpusvm_unmap_pages() for the entries that were actually mapped, the accounting is symmetric by construction. The callback receives the full &drm_pagemap_addr so the driver can use the order and the proto field to distinguish system memory mappings from device interconnect mappings.

The callback must also be usable by the core drm_gpusvm_pages-only API (mm == NULL), as used e.g. for userptr-style mappings. Relax drm_gpusvm_init() to allow such users to pass a restricted @ops that only implements the page-level hooks; the full-SVM hooks (@invalidate) and the chunk configuration must still be unset in that mode.

Quoted 8 lines
Quoted 2 lines
Assisted-by GitHub_Copilot:claude-opus-4.8

Patches 11 and 12 can be spinned out as a separate patch series (for fast tracking).

LGTM.

One other point below...

Quoted 28 lines
Quoted 28 lines

---
drivers/gpu/drm/drm_gpusvm.c | 17 +++++++++++++++--
include/drm/drm_gpusvm.h | 19 +++++++++++++++++++

2 files changed, 34 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_gpusvm.c +9 -2
index fcfe635bc195..512f52471419 100644
@@ -474,8 +474,15 @@ int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
 			return -EINVAL;
 		mmgrab(mm);
 	} else {
-		/* No full SVM mode, only core drm_gpusvm_pages API. */
-		if (ops || num_chunks || mm_range || notifier_size)
+		/*
+		 * No full SVM mode, only core drm_gpusvm_pages API. A
+		 * restricted @ops that only implements the page-level hooks
+		 * (e.g. @dma_map_account) is allowed; the full-SVM hooks must
+		 * not be set.
+		 */
+		if (num_chunks || mm_range || notifier_size)
+			return -EINVAL;
+		if (ops && ops->invalidate)
 			return -EINVAL;
 	}
 
Quoted 6 lines

drivers/gpu/drm/drm_gpusvm.c static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm, · line 1257

Quoted 6 lines
drivers/gpu/drm/drm_gpusvm.c +2 -0
@@ -1246,6 +1253,8 @@ static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm,
 			else if (dpagemap && dpagemap->ops->device_unmap)
 				dpagemap->ops->device_unmap(dpagemap,
 							    dev, addr);
+			if (gpusvm->ops && gpusvm->ops->dma_map_account)
+				gpusvm->ops->dma_map_account(gpusvm, addr, -1);

In the latest code, there is use_iova path and unamp there doesn't seem to come here for that case. Something to take care of during rebase.

Niranjana

Quoted 47 lines
Quoted 47 lines
drivers/gpu/drm/drm_gpusvm.c +4 -0
 			i += 1 << addr->order;
 		}
 
@@ -1654,6 +1663,10 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
 				(addr, DRM_INTERCONNECT_SYSTEM, order,
 				 dma_dir);
 		}
+		if (gpusvm->ops && gpusvm->ops->dma_map_account)
+			gpusvm->ops->dma_map_account(gpusvm,
+						     &svm_pages->dma_addr[j],
+						     1);
 		i += 1 << order;
 		num_dma_mapped = i;
 		flags.has_dma_mapping = true;
include/drm/drm_gpusvm.h +19 -0
index b7d987bf76aa..36f35816a569 100644
@@ -75,6 +75,25 @@ struct drm_gpusvm_ops {
 	void (*invalidate)(struct drm_gpusvm *gpusvm,
 			   struct drm_gpusvm_notifier *notifier,
 			   const struct mmu_notifier_range *mmu_range);
+
+	/**
+	 * @dma_map_account: Account a DMA-mapping change (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @addr: The address descriptor of the chunk being (un)mapped
+	 * @sign: +1 when @addr has just been DMA-mapped, -1 when it is being
+	 *        unmapped
+	 *
+	 * Called once per &drm_pagemap_addr entry, when the entry is
+	 * DMA-mapped by drm_gpusvm_get_pages() (@sign == +1) and when it is
+	 * unmapped (@sign == -1), so the driver can keep symmetric accounting
+	 * of the pages it has mapped for GPU access. The @addr->proto field
+	 * can be used to distinguish system-memory mappings from device
+	 * interconnect mappings. May be provided in both full SVM mode and the
+	 * core drm_gpusvm_pages-only mode.
+	 */
+	void (*dma_map_account)(struct drm_gpusvm *gpusvm,
+				const struct drm_pagemap_addr *addr,
+				int sign);
 };
 
 /**
Signature · 1 line
2.34.1
author maintainer
#40 lore raw

Re: [PATCH v3 11/33] drm/gpusvm: Add a DMA-mapping accounting callback

Quoted 67 lines · Niranjana Vishwanathapura
Quoted 27 lines · Niranjana Vishwanathapura
Quoted 26 lines · Matthew Brost

From: Thomas Hellström <[email protected]>

Drivers that use the GPU SVM core to DMA-map system memory for GPU access may want to keep track of how many pages they have mapped, for example to expose statistics or to detect leaks. Doing this accounting in the driver around drm_gpusvm_get_pages() / drm_gpusvm_unmap_pages() is error prone: the number, order and kind of the dma_addr[] entries can change between map and unmap (migration, partial unmaps, the iova vs non-iova paths), which makes symmetric accounting hard to get right.

Add an optional @dma_map_account callback to struct drm_gpusvm_ops and invoke it once per &drm_pagemap_addr entry at the exact points where the entry is DMA-mapped (sign == +1) in drm_gpusvm_get_pages() and unmapped (sign == -1) in __drm_gpusvm_unmap_pages(). Since the map error path and every unmap/free path funnel through __drm_gpusvm_unmap_pages() for the entries that were actually mapped, the accounting is symmetric by construction. The callback receives the full &drm_pagemap_addr so the driver can use the order and the proto field to distinguish system memory mappings from device interconnect mappings.

The callback must also be usable by the core drm_gpusvm_pages-only API (mm == NULL), as used e.g. for userptr-style mappings. Relax drm_gpusvm_init() to allow such users to pass a restricted @ops that only implements the page-level hooks; the full-SVM hooks (@invalidate) and the chunk configuration must still be unset in that mode.

Quoted 8 lines
Quoted 2 lines
Assisted-by GitHub_Copilot:claude-opus-4.8

Patches 11 and 12 can be spinned out as a separate patch series (for fast tracking).

LGTM.

One other point below...

Quoted 28 lines
Quoted 28 lines

---
drivers/gpu/drm/drm_gpusvm.c | 17 +++++++++++++++--
include/drm/drm_gpusvm.h | 19 +++++++++++++++++++

2 files changed, 34 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_gpusvm.c +9 -2
index fcfe635bc195..512f52471419 100644
@@ -474,8 +474,15 @@ int drm_gpusvm_init(struct drm_gpusvm *gpusvm,
 			return -EINVAL;
 		mmgrab(mm);
 	} else {
-		/* No full SVM mode, only core drm_gpusvm_pages API. */
-		if (ops || num_chunks || mm_range || notifier_size)
+		/*
+		 * No full SVM mode, only core drm_gpusvm_pages API. A
+		 * restricted @ops that only implements the page-level hooks
+		 * (e.g. @dma_map_account) is allowed; the full-SVM hooks must
+		 * not be set.
+		 */
+		if (num_chunks || mm_range || notifier_size)
+			return -EINVAL;
+		if (ops && ops->invalidate)
 			return -EINVAL;
 	}
 
Quoted 10 lines
Quoted 6 lines

drivers/gpu/drm/drm_gpusvm.c static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm, · line 1257

Quoted 6 lines
drivers/gpu/drm/drm_gpusvm.c +2 -0
@@ -1246,6 +1253,8 @@ static void __drm_gpusvm_unmap_pages(struct drm_gpusvm *gpusvm,
 			else if (dpagemap && dpagemap->ops->device_unmap)
 				dpagemap->ops->device_unmap(dpagemap,
 							    dev, addr);
+			if (gpusvm->ops && gpusvm->ops->dma_map_account)
+				gpusvm->ops->dma_map_account(gpusvm, addr, -1);

In the latest code, there is use_iova path and unamp there doesn't seem to come here for that case. Something to take care of during rebase.

Yep. I have this fixed also because of inline mappings some further adjustments are needed.

Posting now with this fixed.

Matt

Quoted 49 lines

Niranjana

Quoted 47 lines
Quoted 47 lines
drivers/gpu/drm/drm_gpusvm.c +4 -0
 			i += 1 << addr->order;
 		}
 
@@ -1654,6 +1663,10 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
 				(addr, DRM_INTERCONNECT_SYSTEM, order,
 				 dma_dir);
 		}
+		if (gpusvm->ops && gpusvm->ops->dma_map_account)
+			gpusvm->ops->dma_map_account(gpusvm,
+						     &svm_pages->dma_addr[j],
+						     1);
 		i += 1 << order;
 		num_dma_mapped = i;
 		flags.has_dma_mapping = true;
include/drm/drm_gpusvm.h +19 -0
index b7d987bf76aa..36f35816a569 100644
@@ -75,6 +75,25 @@ struct drm_gpusvm_ops {
 	void (*invalidate)(struct drm_gpusvm *gpusvm,
 			   struct drm_gpusvm_notifier *notifier,
 			   const struct mmu_notifier_range *mmu_range);
+
+	/**
+	 * @dma_map_account: Account a DMA-mapping change (optional)
+	 * @gpusvm: Pointer to the GPU SVM
+	 * @addr: The address descriptor of the chunk being (un)mapped
+	 * @sign: +1 when @addr has just been DMA-mapped, -1 when it is being
+	 *        unmapped
+	 *
+	 * Called once per &drm_pagemap_addr entry, when the entry is
+	 * DMA-mapped by drm_gpusvm_get_pages() (@sign == +1) and when it is
+	 * unmapped (@sign == -1), so the driver can keep symmetric accounting
+	 * of the pages it has mapped for GPU access. The @addr->proto field
+	 * can be used to distinguish system-memory mappings from device
+	 * interconnect mappings. May be provided in both full SVM mode and the
+	 * core drm_gpusvm_pages-only mode.
+	 */
+	void (*dma_map_account)(struct drm_gpusvm *gpusvm,
+				const struct drm_pagemap_addr *addr,
+				int sign);
 };
 
 /**
Signature · 1 line
2.34.1
author maintainer
#41 lore raw

PATCH v3 12/33 drm/xe: Add debugfs stats for DMA-mapped pages per order

In reply to Matthew Brost

From: Thomas Hellström <[email protected]>

Expose per-page-order DMA mapping counts for the system memory that the xe driver maps for GPU access, split into two categories: TTM buffer objects, and the GPU SVM / userptr ranges (which share a single drm_gpusvm instance per VM).

The stats are visible at:

<debugfs>/dri/<N>/dma_mapped_pages

and are broken out into two rows (bo, svm/userptr), one column per page order, mirroring the layout of the TTM pool page_pool stat.

For TTM BOs the per-order counts are derived from the DMA scatter-gather
table: each mapped segment is physically contiguous, so its page count is split into the largest power-of-two runs (a segment of N pages contributes a 2^k chunk for the largest k with 2^k <= N, repeated for the remainder). Driving the accounting off the sg table rather than the pages[] array keeps it correct even when pages[] entries are cleared while still DMA-mapped, as happens on the defrag page-borrowing path.

For the SVM and userptr ranges the accounting is driven by the drm_gpusvm @dma_map_account callback, which fires once per dma_addr[] entry at the exact point the entry is DMA-mapped and unmapped. This makes the accounting symmetric by construction and avoids driver-side walks that could drift across migration, partial unmaps and the iova vs non-iova paths. Only DRM_INTERCONNECT_SYSTEM entries are real DMA maps and counted; device interconnect (VRAM, P2P) entries are skipped, so the counter reflects only system pages actually mapped through the DMA layer.

The callback is shared between the full SVM instance (fault mode) and the core drm_gpusvm_pages-only instance used for userptr, so userptr mappings are accounted in both the CONFIG_DRM_XE_GPUSVM=y and =n (but CONFIG_DRM_GPUSVM=y) configurations, into the same svm/userptr counter.

Since the counts are only ever consumed through debugfs, gate the counter storage and all accounting on CONFIG_DEBUG_FS so that builds without debugfs carry no extra atomic traffic or device state; the drm_gpusvm callback is simply left unregistered in that case.

On device teardown assert that every counter has returned to zero, to catch any unbalanced accounting (a leaked DMA mapping) during testing.

Assisted-by GitHub_Copilot:claude-opus-4.8

---

v3:
- Fix drm_gpusvm_init() argument order, clamp DMA-page order to NR_PAGE_ORDERS - 1, and drain DMA-page accounting after xe_bo_dev_fini() (Sashiko)

---
 drivers/gpu/drm/xe/xe_bo.c           | 46 ++++++++++++++++++++
 drivers/gpu/drm/xe/xe_debugfs.c      | 26 ++++++++++++
 drivers/gpu/drm/xe/xe_device.c       | 30 +++++++++++++
 drivers/gpu/drm/xe/xe_device_types.h | 17 ++++++++
 drivers/gpu/drm/xe/xe_svm.c          | 44 ++++++++++++++++++-
 drivers/gpu/drm/xe/xe_svm.h          |  4 +-
 drivers/gpu/drm/xe/xe_userptr.c      | 63 ++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_userptr.h      |  1 +
 8 files changed, 228 insertions(+), 3 deletions(-)
drivers/gpu/drm/xe/xe_bo.c +46 -0
index 85e6d9a0f575..60e0e568aa31 100644
@@ -367,6 +367,50 @@ struct xe_ttm_tt {
 	bool purgeable;
 };
 
+/*
+ * xe_tt_account_dma_pages - account DMA-mapped tt pages per order
+ * @xe: the xe device
+ * @xe_tt: the xe_ttm_tt whose sg table to walk
+ * @sign: +1 to add, -1 to subtract
+ *
+ * Walk the tt's DMA scatter-gather table and split each (physically
+ * contiguous) segment into the largest power-of-two page runs: a segment of N
+ * pages contributes a 2^k chunk for the largest k with 2^k <= N, repeated for
+ * the remainder. Accumulate the count of 2^k pages into
+ * xe->mem.dma_mapped_pages[k]. Driving this off the sg table rather than
+ * tt->pages keeps the accounting correct even when pages[] entries are cleared
+ * while still DMA-mapped (e.g. the defrag page-borrowing path).
+ */
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+static void xe_tt_account_dma_pages(struct xe_device *xe,
+				    struct xe_ttm_tt *xe_tt, int sign)
+{
+	struct scatterlist *sg;
+	unsigned int i;
+
+	for_each_sgtable_sg(xe_tt->sg, sg, i) {
+		unsigned long nr_pages = sg->length >> PAGE_SHIFT;
+
+		while (nr_pages) {
+			unsigned int order = min_t(unsigned int,
+						   __fls(nr_pages),
+						   MAX_PAGE_ORDER);
+			unsigned long chunk = 1UL << order;
+
+			atomic_long_add((long)chunk * sign,
+					&xe->mem.dma_mapped_pages[order]);
+
+			nr_pages -= chunk;
+		}
+	}
+}
+#else
+static void xe_tt_account_dma_pages(struct xe_device *xe,
+				    struct xe_ttm_tt *xe_tt, int sign)
+{
+}
+#endif
+
 static int xe_tt_map_sg(struct xe_device *xe, struct ttm_tt *tt)
 {
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
@@ -396,6 +440,7 @@ static int xe_tt_map_sg(struct xe_device *xe, struct ttm_tt *tt)
 		return ret;
 	}
 
+	xe_tt_account_dma_pages(xe, xe_tt, 1);
 	return 0;
 }
 
@@ -404,6 +449,7 @@ static void xe_tt_unmap_sg(struct xe_device *xe, struct ttm_tt *tt)
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
 
 	if (xe_tt->sg) {
+		xe_tt_account_dma_pages(xe, xe_tt, -1);
 		dma_unmap_sgtable(xe->drm.dev, xe_tt->sg,
 				  DMA_BIDIRECTIONAL, 0);
 		sg_free_table(xe_tt->sg);
drivers/gpu/drm/xe/xe_debugfs.c +26 -0
index 8c391c7b017a..97b47c5c4213 100644
@@ -6,6 +6,7 @@
 #include "xe_debugfs.h"
 
 #include <linux/debugfs.h>
+#include <linux/mmzone.h>
 #include <linux/fault-inject.h>
 #include <linux/string_helpers.h>
 
@@ -231,6 +232,28 @@ static int dgfx_pcie_link_residencies_show(struct seq_file *m, void *data)
 	return 0;
 }
 
+static int dma_mapped_pages_show(struct seq_file *m, void *data)
+{
+	struct xe_device *xe = m->private;
+	unsigned int i;
+
+	seq_printf(m, "%-11s ", "");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " ---%2u---", i);
+	seq_printf(m, "\n%-11s:", "bo");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " %8lu",
+			   atomic_long_read(&xe->mem.dma_mapped_pages[i]));
+	seq_printf(m, "\n%-11s:", "svm/userptr");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " %8lu",
+			   atomic_long_read(&xe->mem.dma_mapped_pages_svm[i]));
+	seq_puts(m, "\n");
+
+	return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(dma_mapped_pages);
+
 static const struct drm_info_list debugfs_list[] = {
 	{"info", info, 0},
 	{ .name = "sriov_info", .show = sriov_info, },
@@ -632,6 +655,9 @@ void xe_debugfs_register(struct xe_device *xe)
 	if (man)
 		ttm_resource_manager_create_debugfs(man, root, "stolen_mm");
 
+	debugfs_create_file("dma_mapped_pages", 0444, root, xe,
+			    &dma_mapped_pages_fops);
+
 	for_each_tile(tile, xe, tile_id)
 		xe_tile_debugfs_register(tile);
 
drivers/gpu/drm/xe/xe_device.c +30 -0
index ad7f3e61d457..97cc2d185e45 100644
@@ -455,12 +455,42 @@ bool xe_device_is_admin_only(const struct xe_device *xe)
 }
 #endif
 
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+static void xe_device_assert_dma_pages_zero(struct xe_device *xe)
+{
+	unsigned int i;
+
+	/*
+	 * All BOs, userptr VMAs and SVM ranges must have been torn down by the
+	 * time the device is destroyed, so every DMA-mapped-pages counter must
+	 * have returned to zero. A non-zero value indicates unbalanced
+	 * accounting, i.e. a missing unmap-side decrement.
+	 */
+	for (i = 0; i < NR_PAGE_ORDERS; i++) {
+		xe_assert(xe, !atomic_long_read(&xe->mem.dma_mapped_pages[i]));
+		xe_assert(xe, !atomic_long_read(&xe->mem.dma_mapped_pages_svm[i]));
+	}
+}
+#else
+static void xe_device_assert_dma_pages_zero(struct xe_device *xe)
+{
+}
+#endif
+
 static void xe_device_destroy(struct drm_device *dev, void *dummy)
 {
 	struct xe_device *xe = to_xe_device(dev);
 
 	xe_bo_dev_fini(&xe->bo_device);
 
+	/*
+	 * Assert the DMA-mapped-pages accounting only after xe_bo_dev_fini()
+	 * has flushed the async BO-free worker: BOs pending async free still
+	 * hold their DMA mappings (and counts) until the worker runs, so
+	 * asserting before the flush could trip a false positive.
+	 */
+	xe_device_assert_dma_pages_zero(xe);
+
 	if (xe->preempt_fence_wq)
 		destroy_workqueue(xe->preempt_fence_wq);
 
drivers/gpu/drm/xe/xe_device_types.h +17 -0
index 022e08205897..95e7cea91cbf 100644
@@ -6,6 +6,7 @@
 #ifndef _XE_DEVICE_TYPES_H_
 #define _XE_DEVICE_TYPES_H_
 
+#include <linux/mmzone.h>
 #include <linux/pci.h>
 
 #include <drm/drm_device.h>
@@ -279,6 +280,22 @@ struct xe_device {
 		struct xe_shrinker *shrinker;
 		/** @mem.stolen_mgr: stolen memory manager. */
 		struct xe_ttm_stolen_mgr *stolen_mgr;
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+		/**
+		 * @mem.dma_mapped_pages: number of DMA-mapped pages per page
+		 * order currently live for this device, for TTM BOs. Only
+		 * accounted when CONFIG_DEBUG_FS is enabled, since it is solely
+		 * exposed through debugfs.
+		 */
+		atomic_long_t dma_mapped_pages[NR_PAGE_ORDERS];
+		/**
+		 * @mem.dma_mapped_pages_svm: number of DMA-mapped pages per
+		 * page order currently live for this device, for SVM ranges
+		 * and userptr VMAs (both share a single drm_gpusvm instance).
+		 * Only accounted when CONFIG_DEBUG_FS is enabled.
+		 */
+		atomic_long_t dma_mapped_pages_svm[NR_PAGE_ORDERS];
+#endif
 	} mem;
 
 	/** @sriov: device level virtualization data */
drivers/gpu/drm/xe/xe_svm.c +42 -2

60 lines not on this page · raw message

drivers/gpu/drm/xe/xe_svm.h +3 -1
index a921556d3466..8e347137a7a6 100644
@@ -194,6 +194,7 @@ struct drm_pagemap *xe_drm_pagemap_from_fd(int fd, u32 region_instance);
 
 #else
 #include <linux/interval_tree.h>
+#include "xe_userptr.h"
 #include "xe_vm.h"
 
 struct drm_pagemap_addr;
@@ -236,7 +237,8 @@ int xe_svm_init(struct xe_vm *vm)
 {
 #if IS_ENABLED(CONFIG_DRM_GPUSVM)
 	return drm_gpusvm_init(&vm->svm.gpusvm, "Xe SVM (simple)",
-			       NULL, 0, 0, 0, NULL, NULL, 0);
+			       NULL, 0, 0, 0,
+			       xe_userptr_gpusvm_ops_get(), NULL, 0);
 #else
 	return 0;
 #endif
drivers/gpu/drm/xe/xe_userptr.c +63 -0

72 lines not on this page · raw message

drivers/gpu/drm/xe/xe_userptr.h +1 -0
index 2a3cd1b5efbb..1e9601be5713 100644
@@ -108,6 +108,7 @@ int __xe_vm_userptr_needs_repin(struct xe_vm *vm);
 int xe_vm_userptr_check_repin(struct xe_vm *vm);
 int xe_vma_userptr_pin_pages(struct xe_userptr_vma *uvma);
 int xe_vma_userptr_check_repin(struct xe_userptr_vma *uvma);
+const struct drm_gpusvm_ops *xe_userptr_gpusvm_ops_get(void);
 #else
 static inline void xe_userptr_remove(struct xe_userptr_vma *uvma) {}
 
Signature · 1 line
2.34.1
#42 lore raw

Re: [PATCH v3 12/33] drm/xe: Add debugfs stats for DMA-mapped pages per order

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [Critical] Missing decrement of DMA mapped pages counter in xe_bo_dma_unmap_pinned() leads to leaked accounting and kernel assertion failure during device teardown.

Pre-existing issues:
- [High] xe_bo_dev_fini() is called before xe->destroy_wq is flushed, leading to Use-After-Free of the TTM device and false positive assertion failures on device teardown. --

--- Patch [12]: [PATCH v3 12/33] drm/xe: Add debugfs stats for DMA-mapped pages per order --- commit 670e42692f03a24b49683eec7ff3afb39e28a858
Author: Thomas Hellström <[email protected]>
Subject: drm/xe: Add debugfs stats for DMA-mapped pages per order

Expose per-page-order DMA mapping counts for the system memory that the xe driver maps for GPU access, split into two categories: TTM buffer objects, and the GPU SVM / userptr ranges.

Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
index 85e6d9a0f575b..60e0e568aa31f 100644

[ ... ]

drivers/gpu/drm/xe/xe_bo.c static void xe_tt_unmap_sg(struct xe_device *xe, struct ttm_tt *tt) · line 455

Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.c +1 -0
@@ -404,6 +449,7 @@ static void xe_tt_unmap_sg(struct xe_device *xe, struct ttm_tt *tt)
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
 
 	if (xe_tt->sg) {
+		xe_tt_account_dma_pages(xe, xe_tt, -1);
 		dma_unmap_sgtable(xe->drm.dev, xe_tt->sg,
 				  DMA_BIDIRECTIONAL, 0);
 		sg_free_table(xe_tt->sg);

[Severity: Critical]
Will this accounting leak if the scatter-gather table is unmapped elsewhere?

Looking at xe_bo_dma_unmap_pinned() in drivers/gpu/drm/xe/xe_bo.c, it unmaps and frees the sg table without updating the accounting:

	} else if (xe_tt->sg) {
		dma_unmap_sgtable(ttm_to_xe_device(ttm_bo->bdev)->drm.dev,
				  xe_tt->sg,
				  DMA_BIDIRECTIONAL, 0);
		sg_free_table(xe_tt->sg);
		xe_tt->sg = NULL;
	}

Because xe_bo_dma_unmap_pinned() clears xe_tt->sg, a later call to xe_tt_unmap_sg() will see xe_tt->sg as NULL and skip the decrement, leading to a permanent accounting leak and a guaranteed assertion failure on device teardown.

Quoted 4 lines
drivers/gpu/drm/xe/xe_device.c +0 -0
index ad7f3e61d4579..97cc2d185e45d 100644

[ ... ]

Quoted 6 lines

static void xe_device_destroy(struct drm_device *dev, void *dummy) {

drivers/gpu/drm/xe/xe_device.c +0 -0
 	struct xe_device *xe = to_xe_device(dev);
 
 	xe_bo_dev_fini(&xe->bo_device);
 

drivers/gpu/drm/xe/xe_device.c bool xe_device_is_admin_only(const struct xe_device *xe) · line 495

Quoted 10 lines
drivers/gpu/drm/xe/xe_device.c +8 -0
+	/*
+	 * Assert the DMA-mapped-pages accounting only after xe_bo_dev_fini()
+	 * has flushed the async BO-free worker: BOs pending async free still
+	 * hold their DMA mappings (and counts) until the worker runs, so
+	 * asserting before the flush could trip a false positive.
+	 */
+	xe_device_assert_dma_pages_zero(xe);
+
 	if (xe->preempt_fence_wq)
 		destroy_workqueue(xe->preempt_fence_wq);

[Severity: High]
This is a pre-existing issue, but will this new assertion falsely trip due to the teardown order?

Because xe->destroy_wq is destroyed after this assertion, and work items in xe->destroy_wq might still hold final references to BOs, those BOs won't be unmapped until the workqueue is drained.

Since the assertion now runs before the workqueues drop their BO references, it will incorrectly trigger because those BOs still have active DMA mappings.

Additionally, since xe_bo_dev_fini() flushes the async BO-free worker before the workqueues drop BOs, the worker can run concurrently with or after ttm_device_fini(), which is a pre-existing use-after-free risk exposed by this change.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
#43 lore raw

Re: [PATCH v3 12/33] drm/xe: Add debugfs stats for DMA-mapped pages per order

In reply to Matthew Brost
Quoted 41 lines · Matthew Brost

From: Thomas Hellström <[email protected]>

Expose per-page-order DMA mapping counts for the system memory that the xe driver maps for GPU access, split into two categories: TTM buffer objects, and the GPU SVM / userptr ranges (which share a single drm_gpusvm instance per VM).

The stats are visible at:

<debugfs>/dri/<N>/dma_mapped_pages

and are broken out into two rows (bo, svm/userptr), one column per page order, mirroring the layout of the TTM pool page_pool stat.

For TTM BOs the per-order counts are derived from the DMA scatter-gather
table: each mapped segment is physically contiguous, so its page count is split into the largest power-of-two runs (a segment of N pages contributes a 2^k chunk for the largest k with 2^k <= N, repeated for the remainder). Driving the accounting off the sg table rather than the pages[] array keeps it correct even when pages[] entries are cleared while still DMA-mapped, as happens on the defrag page-borrowing path.

For the SVM and userptr ranges the accounting is driven by the drm_gpusvm @dma_map_account callback, which fires once per dma_addr[] entry at the exact point the entry is DMA-mapped and unmapped. This makes the accounting symmetric by construction and avoids driver-side walks that could drift across migration, partial unmaps and the iova vs non-iova paths. Only DRM_INTERCONNECT_SYSTEM entries are real DMA maps and counted; device interconnect (VRAM, P2P) entries are skipped, so the counter reflects only system pages actually mapped through the DMA layer.

The callback is shared between the full SVM instance (fault mode) and the core drm_gpusvm_pages-only instance used for userptr, so userptr mappings are accounted in both the CONFIG_DRM_XE_GPUSVM=y and =n (but CONFIG_DRM_GPUSVM=y) configurations, into the same svm/userptr counter.

Since the counts are only ever consumed through debugfs, gate the counter storage and all accounting on CONFIG_DEBUG_FS so that builds without debugfs carry no extra atomic traffic or device state; the drm_gpusvm callback is simply left unregistered in that case.

Quoted 6 lines

On device teardown assert that every counter has returned to zero, to catch any unbalanced accounting (a leaked DMA mapping) during testing.

Assisted-by GitHub_Copilot:claude-opus-4.8

Patches 11 and 12 can be spinned out as a separate patch series (for fast tracking).

Also, dma map/unmap of dmabuf imported BOs are skipped here (rightly so). But perhaps add a comments in commit message that either self-import returns the same BO and BOs of other device gets counted on exporter side when it gets first migrated to TT first.

Quoted 48 lines

---

v3:
- Fix drm_gpusvm_init() argument order, clamp DMA-page order to NR_PAGE_ORDERS - 1, and drain DMA-page accounting after xe_bo_dev_fini() (Sashiko)
---

drivers/gpu/drm/xe/xe_bo.c           | 46 ++++++++++++++++++++
drivers/gpu/drm/xe/xe_debugfs.c      | 26 ++++++++++++
drivers/gpu/drm/xe/xe_device.c       | 30 +++++++++++++
drivers/gpu/drm/xe/xe_device_types.h | 17 ++++++++
drivers/gpu/drm/xe/xe_svm.c          | 44 ++++++++++++++++++-
drivers/gpu/drm/xe/xe_svm.h          |  4 +-
drivers/gpu/drm/xe/xe_userptr.c      | 63 ++++++++++++++++++++++++++++
drivers/gpu/drm/xe/xe_userptr.h      |  1 +
8 files changed, 228 insertions(+), 3 deletions(-)
drivers/gpu/drm/xe/xe_bo.c +23 -0
index 85e6d9a0f575..60e0e568aa31 100644
@@ -367,6 +367,50 @@ struct xe_ttm_tt {
 	bool purgeable;
 };
 
+/*
+ * xe_tt_account_dma_pages - account DMA-mapped tt pages per order
+ * @xe: the xe device
+ * @xe_tt: the xe_ttm_tt whose sg table to walk
+ * @sign: +1 to add, -1 to subtract
+ *
+ * Walk the tt's DMA scatter-gather table and split each (physically
+ * contiguous) segment into the largest power-of-two page runs: a segment of N
+ * pages contributes a 2^k chunk for the largest k with 2^k <= N, repeated for
+ * the remainder. Accumulate the count of 2^k pages into
+ * xe->mem.dma_mapped_pages[k]. Driving this off the sg table rather than
+ * tt->pages keeps the accounting correct even when pages[] entries are cleared
+ * while still DMA-mapped (e.g. the defrag page-borrowing path).
+ */
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+static void xe_tt_account_dma_pages(struct xe_device *xe,
+				    struct xe_ttm_tt *xe_tt, int sign)
+{
+	struct scatterlist *sg;
+	unsigned int i;
+
+	for_each_sgtable_sg(xe_tt->sg, sg, i) {
+		unsigned long nr_pages = sg->length >> PAGE_SHIFT;

drivers/gpu/drm/xe/xe_bo.c struct xe_ttm_tt { · line 402

Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +10 -0
+
+		while (nr_pages) {
+			unsigned int order = min_t(unsigned int,
+						   __fls(nr_pages),
+						   MAX_PAGE_ORDER);
+			unsigned long chunk = 1UL << order;
+
+			atomic_long_add((long)chunk * sign,
+					&xe->mem.dma_mapped_pages[order]);
+

NIT: As there is an atomic operation involved, instead of doing it under CONFIG_DEBUG_FS, should we be doing it under a separate xe specific debug config?

Quoted 23 lines
drivers/gpu/drm/xe/xe_bo.c +12 -0
+			nr_pages -= chunk;
+		}
+	}
+}
+#else
+static void xe_tt_account_dma_pages(struct xe_device *xe,
+				    struct xe_ttm_tt *xe_tt, int sign)
+{
+}
+#endif
+
 static int xe_tt_map_sg(struct xe_device *xe, struct ttm_tt *tt)
 {
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
@@ -396,6 +440,7 @@ static int xe_tt_map_sg(struct xe_device *xe, struct ttm_tt *tt)
 		return ret;
 	}
 
+	xe_tt_account_dma_pages(xe, xe_tt, 1);
 	return 0;
 }
 

drivers/gpu/drm/xe/xe_bo.c static void xe_tt_unmap_sg(struct xe_device *xe, struct ttm_tt *tt) · line 455

Quoted 7 lines
drivers/gpu/drm/xe/xe_bo.c +1 -0
@@ -404,6 +449,7 @@ static void xe_tt_unmap_sg(struct xe_device *xe, struct ttm_tt *tt)
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
 
 	if (xe_tt->sg) {
+		xe_tt_account_dma_pages(xe, xe_tt, -1);
 		dma_unmap_sgtable(xe->drm.dev, xe_tt->sg,
 				  DMA_BIDIRECTIONAL, 0);
 		sg_free_table(xe_tt->sg);

There is another dma_unamp_sgtable() in xe_bo_dma_unmap_pinned(). I guess we should account for that unmap path also.
I am seeing sashiko reported this as well.

Quoted 24 lines
drivers/gpu/drm/xe/xe_debugfs.c +9 -0
index 8c391c7b017a..97b47c5c4213 100644
@@ -6,6 +6,7 @@
 #include "xe_debugfs.h"
 
 #include <linux/debugfs.h>
+#include <linux/mmzone.h>
 #include <linux/fault-inject.h>
 #include <linux/string_helpers.h>
 
@@ -231,6 +232,28 @@ static int dgfx_pcie_link_residencies_show(struct seq_file *m, void *data)
 	return 0;
 }
 
+static int dma_mapped_pages_show(struct seq_file *m, void *data)
+{
+	struct xe_device *xe = m->private;
+	unsigned int i;
+
+	seq_printf(m, "%-11s ", "");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " ---%2u---", i);

drivers/gpu/drm/xe/xe_debugfs.c static int dgfx_pcie_link_residencies_show(struct seq_file *m, void *data) · line 252

Quoted 10 lines
drivers/gpu/drm/xe/xe_debugfs.c +10 -0
+	seq_printf(m, "\n%-11s:", "bo");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " %8lu",
+			   atomic_long_read(&xe->mem.dma_mapped_pages[i]));
+	seq_printf(m, "\n%-11s:", "svm/userptr");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " %8lu",
+			   atomic_long_read(&xe->mem.dma_mapped_pages_svm[i]));
+	seq_puts(m, "\n");
+

NIT: As order-10 also prints higher order (1GB) pages, may be print it as '>=10'?

NIT: This is a rough snapshot as array elements could be changing as we print. I guess that is fine.

Quoted 29 lines
drivers/gpu/drm/xe/xe_debugfs.c +7 -0
+	return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(dma_mapped_pages);
+
 static const struct drm_info_list debugfs_list[] = {
 	{"info", info, 0},
 	{ .name = "sriov_info", .show = sriov_info, },
@@ -632,6 +655,9 @@ void xe_debugfs_register(struct xe_device *xe)
 	if (man)
 		ttm_resource_manager_create_debugfs(man, root, "stolen_mm");
 
+	debugfs_create_file("dma_mapped_pages", 0444, root, xe,
+			    &dma_mapped_pages_fops);
+
 	for_each_tile(tile, xe, tile_id)
 		xe_tile_debugfs_register(tile);
 
drivers/gpu/drm/xe/xe_device.c +4 -0
index ad7f3e61d457..97cc2d185e45 100644
@@ -455,12 +455,42 @@ bool xe_device_is_admin_only(const struct xe_device *xe)
 }
 #endif
 
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+static void xe_device_assert_dma_pages_zero(struct xe_device *xe)
+{
+	unsigned int i;

drivers/gpu/drm/xe/xe_device.c bool xe_device_is_admin_only(const struct xe_device *xe) · line 471

Quoted 10 lines
drivers/gpu/drm/xe/xe_device.c +10 -0
+
+	/*
+	 * All BOs, userptr VMAs and SVM ranges must have been torn down by the
+	 * time the device is destroyed, so every DMA-mapped-pages counter must
+	 * have returned to zero. A non-zero value indicates unbalanced
+	 * accounting, i.e. a missing unmap-side decrement.
+	 */
+	for (i = 0; i < NR_PAGE_ORDERS; i++) {
+		xe_assert(xe, !atomic_long_read(&xe->mem.dma_mapped_pages[i]));
+		xe_assert(xe, !atomic_long_read(&xe->mem.dma_mapped_pages_svm[i]));

This also probably good to do it under a separate xe DEBUG config instead of CONFIG_DEBUG_FS.

Niranjana

Quoted 240 lines
drivers/gpu/drm/xe/xe_device.c +16 -0
+	}
+}
+#else
+static void xe_device_assert_dma_pages_zero(struct xe_device *xe)
+{
+}
+#endif
+
 static void xe_device_destroy(struct drm_device *dev, void *dummy)
 {
 	struct xe_device *xe = to_xe_device(dev);
 
 	xe_bo_dev_fini(&xe->bo_device);
 
+	/*
+	 * Assert the DMA-mapped-pages accounting only after xe_bo_dev_fini()
+	 * has flushed the async BO-free worker: BOs pending async free still
+	 * hold their DMA mappings (and counts) until the worker runs, so
+	 * asserting before the flush could trip a false positive.
+	 */
+	xe_device_assert_dma_pages_zero(xe);
+
 	if (xe->preempt_fence_wq)
 		destroy_workqueue(xe->preempt_fence_wq);
 
drivers/gpu/drm/xe/xe_device_types.h +17 -0
index 022e08205897..95e7cea91cbf 100644
@@ -6,6 +6,7 @@
 #ifndef _XE_DEVICE_TYPES_H_
 #define _XE_DEVICE_TYPES_H_
 
+#include <linux/mmzone.h>
 #include <linux/pci.h>
 
 #include <drm/drm_device.h>
@@ -279,6 +280,22 @@ struct xe_device {
 		struct xe_shrinker *shrinker;
 		/** @mem.stolen_mgr: stolen memory manager. */
 		struct xe_ttm_stolen_mgr *stolen_mgr;
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+		/**
+		 * @mem.dma_mapped_pages: number of DMA-mapped pages per page
+		 * order currently live for this device, for TTM BOs. Only
+		 * accounted when CONFIG_DEBUG_FS is enabled, since it is solely
+		 * exposed through debugfs.
+		 */
+		atomic_long_t dma_mapped_pages[NR_PAGE_ORDERS];
+		/**
+		 * @mem.dma_mapped_pages_svm: number of DMA-mapped pages per
+		 * page order currently live for this device, for SVM ranges
+		 * and userptr VMAs (both share a single drm_gpusvm instance).
+		 * Only accounted when CONFIG_DEBUG_FS is enabled.
+		 */
+		atomic_long_t dma_mapped_pages_svm[NR_PAGE_ORDERS];
+#endif
 	} mem;
 
 	/** @sriov: device level virtualization data */
drivers/gpu/drm/xe/xe_svm.c +42 -2
index b228a737cfd6..ebe48cb76ad9 100644
@@ -827,10 +827,50 @@ static int xe_svm_get_pagemaps(struct xe_vm *vm)
 }
 #endif
 
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+/*
+ * xe_svm_dma_map_account - drm_gpusvm DMA-mapping accounting callback
+ * @gpusvm: The GPU SVM the mapping belongs to
+ * @addr: The address descriptor of the chunk being (un)mapped
+ * @sign: +1 when @addr was DMA-mapped, -1 when it is being unmapped
+ *
+ * Maintain per-order counts of the system-memory pages that the full SVM
+ * instance (SVM ranges, and userptr VMAs in fault mode) has DMA-mapped for GPU
+ * access. Only DRM_INTERCONNECT_SYSTEM entries are real DMA maps and counted;
+ * device interconnect (VRAM, P2P) entries are skipped. The counts are solely
+ * exposed through debugfs, so the callback is only registered when
+ * CONFIG_DEBUG_FS is enabled.
+ */
+static void xe_svm_dma_map_account(struct drm_gpusvm *gpusvm,
+				   const struct drm_pagemap_addr *addr,
+				   int sign)
+{
+	struct xe_device *xe = gpusvm_to_vm(gpusvm)->xe;
+	unsigned int order = addr->order;
+
+	if (addr->proto != DRM_INTERCONNECT_SYSTEM)
+		return;
+
+	/*
+	 * The CPU-PTE map order reported by hmm can exceed MAX_PAGE_ORDER
+	 * (e.g. an order-18 1GB mapping), so clamp the array index to avoid an
+	 * out-of-bounds access. Clamping is symmetric across the +1/-1 calls,
+	 * so the per-order counters stay balanced.
+	 */
+	order = min(order, (unsigned int)NR_PAGE_ORDERS - 1);
+
+	atomic_long_add((long)(1UL << addr->order) * sign,
+			&xe->mem.dma_mapped_pages_svm[order]);
+}
+#endif
+
 static const struct drm_gpusvm_ops gpusvm_ops = {
 	.range_alloc = xe_svm_range_alloc,
 	.range_free = xe_svm_range_free,
 	.invalidate = xe_svm_invalidate,
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+	.dma_map_account = xe_svm_dma_map_account,
+#endif
 };
 
 static const unsigned long fault_chunk_sizes[] = {
@@ -925,8 +965,8 @@ int xe_svm_init(struct xe_vm *vm)
 		}
 	} else {
 		err = drm_gpusvm_init(&vm->svm.gpusvm, "Xe SVM (simple)",
-				      NULL, 0, 0, 0, NULL,
-				      NULL, 0);
+				      NULL, 0, 0, 0,
+				      xe_userptr_gpusvm_ops_get(), NULL, 0);
 	}
 
 	return err;
drivers/gpu/drm/xe/xe_svm.h +3 -1
index a921556d3466..8e347137a7a6 100644
@@ -194,6 +194,7 @@ struct drm_pagemap *xe_drm_pagemap_from_fd(int fd, u32 region_instance);
 
 #else
 #include <linux/interval_tree.h>
+#include "xe_userptr.h"
 #include "xe_vm.h"
 
 struct drm_pagemap_addr;
@@ -236,7 +237,8 @@ int xe_svm_init(struct xe_vm *vm)
 {
 #if IS_ENABLED(CONFIG_DRM_GPUSVM)
 	return drm_gpusvm_init(&vm->svm.gpusvm, "Xe SVM (simple)",
-			       NULL, 0, 0, 0, NULL, NULL, 0);
+			       NULL, 0, 0, 0,
+			       xe_userptr_gpusvm_ops_get(), NULL, 0);
 #else
 	return 0;
 #endif
drivers/gpu/drm/xe/xe_userptr.c +63 -0
index 8b2d461ea0b2..6b9e1530ead4 100644
@@ -8,9 +8,72 @@
 
 #include <linux/mm.h>
 
+#include <drm/drm_pagemap.h>
+
 #include "xe_tlb_inval.h"
 #include "xe_trace_bo.h"
 
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+/*
+ * xe_userptr_dma_map_account - drm_gpusvm DMA-mapping accounting callback
+ * @gpusvm: The GPU SVM the mapping belongs to
+ * @addr: The address descriptor of the chunk being (un)mapped
+ * @sign: +1 when @addr was DMA-mapped, -1 when it is being unmapped
+ *
+ * Account, per page order, the system-memory pages DMA-mapped for GPU access
+ * through the drm_gpusvm_pages instance shared by userptr (and, in fault mode,
+ * SVM). Only DRM_INTERCONNECT_SYSTEM entries are real DMA maps and counted;
+ * device interconnect (VRAM, P2P) entries are skipped. The counts are solely
+ * exposed through debugfs, so the callback is only registered when
+ * CONFIG_DEBUG_FS is enabled.
+ */
+static void xe_userptr_dma_map_account(struct drm_gpusvm *gpusvm,
+				       const struct drm_pagemap_addr *addr,
+				       int sign)
+{
+	struct xe_vm *vm = container_of(gpusvm, struct xe_vm, svm.gpusvm);
+	unsigned int order = addr->order;
+
+	if (addr->proto != DRM_INTERCONNECT_SYSTEM)
+		return;
+
+	/*
+	 * The CPU-PTE map order reported by hmm can exceed MAX_PAGE_ORDER
+	 * (e.g. an order-18 1GB mapping), so clamp the array index to avoid an
+	 * out-of-bounds access. Clamping is symmetric across the +1/-1 calls,
+	 * so the per-order counters stay balanced.
+	 */
+	order = min(order, (unsigned int)NR_PAGE_ORDERS - 1);
+
+	atomic_long_add((long)(1UL << addr->order) * sign,
+			&vm->xe->mem.dma_mapped_pages_svm[order]);
+}
+
+static const struct drm_gpusvm_ops xe_userptr_gpusvm_ops = {
+	.dma_map_account = xe_userptr_dma_map_account,
+};
+#endif
+
+/**
+ * xe_userptr_gpusvm_ops_get() - Accounting ops for the simple gpusvm instance
+ *
+ * The core drm_gpusvm_pages-only instance used for userptr (and, in fault
+ * mode, shared with SVM) is initialised with these ops so that DMA-mapped
+ * system pages are accounted and exposed through debugfs. Returns NULL when
+ * CONFIG_DEBUG_FS is disabled, in which case no accounting is kept and the
+ * instance is initialised without ops.
+ *
+ * Return: Pointer to the restricted &drm_gpusvm_ops, or NULL.
+ */
+const struct drm_gpusvm_ops *xe_userptr_gpusvm_ops_get(void)
+{
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+	return &xe_userptr_gpusvm_ops;
+#else
+	return NULL;
+#endif
+}
+
 static void xe_userptr_assert_in_notifier(struct xe_vm *vm)
 {
 	lockdep_assert(lockdep_is_held_type(&vm->svm.gpusvm.notifier_lock, 0) ||
drivers/gpu/drm/xe/xe_userptr.h +1 -0
index 2a3cd1b5efbb..1e9601be5713 100644
@@ -108,6 +108,7 @@ int __xe_vm_userptr_needs_repin(struct xe_vm *vm);
 int xe_vm_userptr_check_repin(struct xe_vm *vm);
 int xe_vma_userptr_pin_pages(struct xe_userptr_vma *uvma);
 int xe_vma_userptr_check_repin(struct xe_userptr_vma *uvma);
+const struct drm_gpusvm_ops *xe_userptr_gpusvm_ops_get(void);
 #else
 static inline void xe_userptr_remove(struct xe_userptr_vma *uvma) {}
 
Signature · 1 line
2.34.1
author maintainer
#44 lore raw

Re: [PATCH v3 12/33] drm/xe: Add debugfs stats for DMA-mapped pages per order

Quoted 45 lines · Niranjana Vishwanathapura
Quoted 44 lines · Matthew Brost

From: Thomas Hellström <[email protected]>

Expose per-page-order DMA mapping counts for the system memory that the xe driver maps for GPU access, split into two categories: TTM buffer objects, and the GPU SVM / userptr ranges (which share a single drm_gpusvm instance per VM).

The stats are visible at:

<debugfs>/dri/<N>/dma_mapped_pages

and are broken out into two rows (bo, svm/userptr), one column per page order, mirroring the layout of the TTM pool page_pool stat.

For TTM BOs the per-order counts are derived from the DMA scatter-gather
table: each mapped segment is physically contiguous, so its page count is split into the largest power-of-two runs (a segment of N pages contributes a 2^k chunk for the largest k with 2^k <= N, repeated for the remainder). Driving the accounting off the sg table rather than the pages[] array keeps it correct even when pages[] entries are cleared while still DMA-mapped, as happens on the defrag page-borrowing path.

For the SVM and userptr ranges the accounting is driven by the drm_gpusvm @dma_map_account callback, which fires once per dma_addr[] entry at the exact point the entry is DMA-mapped and unmapped. This makes the accounting symmetric by construction and avoids driver-side walks that could drift across migration, partial unmaps and the iova vs non-iova paths. Only DRM_INTERCONNECT_SYSTEM entries are real DMA maps and counted; device interconnect (VRAM, P2P) entries are skipped, so the counter reflects only system pages actually mapped through the DMA layer.

The callback is shared between the full SVM instance (fault mode) and the core drm_gpusvm_pages-only instance used for userptr, so userptr mappings are accounted in both the CONFIG_DRM_XE_GPUSVM=y and =n (but CONFIG_DRM_GPUSVM=y) configurations, into the same svm/userptr counter.

Since the counts are only ever consumed through debugfs, gate the counter storage and all accounting on CONFIG_DEBUG_FS so that builds without debugfs carry no extra atomic traffic or device state; the drm_gpusvm callback is simply left unregistered in that case.

On device teardown assert that every counter has returned to zero, to catch any unbalanced accounting (a leaked DMA mapping) during testing.

Quoted 7 lines
Quoted 3 lines
Assisted-by GitHub_Copilot:claude-opus-4.8

Patches 11 and 12 can be spinned out as a separate patch series (for fast tracking).

Yep, spliting these out now.

Quoted 5 lines

Also, dma map/unmap of dmabuf imported BOs are skipped here (rightly so). But perhaps add a comments in commit message that either self-import returns the same BO and BOs of other device gets counted on exporter side when it gets first migrated to TT first.

Sure.

Quoted 54 lines
Quoted 54 lines

---

v3:
- Fix drm_gpusvm_init() argument order, clamp DMA-page order to NR_PAGE_ORDERS - 1, and drain DMA-page accounting after xe_bo_dev_fini() (Sashiko)
---

drivers/gpu/drm/xe/xe_bo.c           | 46 ++++++++++++++++++++
drivers/gpu/drm/xe/xe_debugfs.c      | 26 ++++++++++++
drivers/gpu/drm/xe/xe_device.c       | 30 +++++++++++++
drivers/gpu/drm/xe/xe_device_types.h | 17 ++++++++
drivers/gpu/drm/xe/xe_svm.c          | 44 ++++++++++++++++++-
drivers/gpu/drm/xe/xe_svm.h          |  4 +-
drivers/gpu/drm/xe/xe_userptr.c      | 63 ++++++++++++++++++++++++++++
drivers/gpu/drm/xe/xe_userptr.h      |  1 +
8 files changed, 228 insertions(+), 3 deletions(-)
drivers/gpu/drm/xe/xe_bo.c +29 -0
index 85e6d9a0f575..60e0e568aa31 100644
@@ -367,6 +367,50 @@ struct xe_ttm_tt {
 	bool purgeable;
 };
 
+/*
+ * xe_tt_account_dma_pages - account DMA-mapped tt pages per order
+ * @xe: the xe device
+ * @xe_tt: the xe_ttm_tt whose sg table to walk
+ * @sign: +1 to add, -1 to subtract
+ *
+ * Walk the tt's DMA scatter-gather table and split each (physically
+ * contiguous) segment into the largest power-of-two page runs: a segment of N
+ * pages contributes a 2^k chunk for the largest k with 2^k <= N, repeated for
+ * the remainder. Accumulate the count of 2^k pages into
+ * xe->mem.dma_mapped_pages[k]. Driving this off the sg table rather than
+ * tt->pages keeps the accounting correct even when pages[] entries are cleared
+ * while still DMA-mapped (e.g. the defrag page-borrowing path).
+ */
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+static void xe_tt_account_dma_pages(struct xe_device *xe,
+				    struct xe_ttm_tt *xe_tt, int sign)
+{
+	struct scatterlist *sg;
+	unsigned int i;
+
+	for_each_sgtable_sg(xe_tt->sg, sg, i) {
+		unsigned long nr_pages = sg->length >> PAGE_SHIFT;
+
+		while (nr_pages) {
+			unsigned int order = min_t(unsigned int,
+						   __fls(nr_pages),
+						   MAX_PAGE_ORDER);
+			unsigned long chunk = 1UL << order;
Quoted 9 lines

drivers/gpu/drm/xe/xe_bo.c struct xe_ttm_tt { · line 402

Quoted 4 lines
drivers/gpu/drm/xe/xe_bo.c +4 -0
+
+			atomic_long_add((long)chunk * sign,
+					&xe->mem.dma_mapped_pages[order]);
+

NIT: As there is an atomic operation involved, instead of doing it under CONFIG_DEBUG_FS, should we be doing it under a separate xe specific debug config?

I think GT stats are just gated by CONFIG_DEBUG_FS, I say this is roughly idea. If someone complains about perf we could change this per-cpu counter like GT stats or I guess add different Kconfig gate?

Quoted 25 lines
Quoted 25 lines
drivers/gpu/drm/xe/xe_bo.c +12 -0
+			nr_pages -= chunk;
+		}
+	}
+}
+#else
+static void xe_tt_account_dma_pages(struct xe_device *xe,
+				    struct xe_ttm_tt *xe_tt, int sign)
+{
+}
+#endif
+
 static int xe_tt_map_sg(struct xe_device *xe, struct ttm_tt *tt)
 {
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
@@ -396,6 +440,7 @@ static int xe_tt_map_sg(struct xe_device *xe, struct ttm_tt *tt)
 		return ret;
 	}
 
+	xe_tt_account_dma_pages(xe, xe_tt, 1);
 	return 0;
 }
 
@@ -404,6 +449,7 @@ static void xe_tt_unmap_sg(struct xe_device *xe, struct ttm_tt *tt)
 	struct xe_ttm_tt *xe_tt = container_of(tt, struct xe_ttm_tt, ttm);
 
Quoted 9 lines

drivers/gpu/drm/xe/xe_bo.c static void xe_tt_unmap_sg(struct xe_device *xe, struct ttm_tt *tt) · line 455

Quoted 5 lines
drivers/gpu/drm/xe/xe_bo.c +1 -0
 	if (xe_tt->sg) {
+		xe_tt_account_dma_pages(xe, xe_tt, -1);
 		dma_unmap_sgtable(xe->drm.dev, xe_tt->sg,
 				  DMA_BIDIRECTIONAL, 0);
 		sg_free_table(xe_tt->sg);

There is another dma_unamp_sgtable() in xe_bo_dma_unmap_pinned(). I guess we should account for that unmap path also.
I am seeing sashiko reported this as well.

I see dma_buf_unmap_attachment and call to xe_tt_unmap_sg but I'm on drm-tip now, so at in what I'll post shortly this I don't think this is an issue.

Quoted 75 lines
Quoted 25 lines
drivers/gpu/drm/xe/xe_debugfs.c +9 -0
index 8c391c7b017a..97b47c5c4213 100644
@@ -6,6 +6,7 @@
 #include "xe_debugfs.h"
 
 #include <linux/debugfs.h>
+#include <linux/mmzone.h>
 #include <linux/fault-inject.h>
 #include <linux/string_helpers.h>
 
@@ -231,6 +232,28 @@ static int dgfx_pcie_link_residencies_show(struct seq_file *m, void *data)
 	return 0;
 }
 
+static int dma_mapped_pages_show(struct seq_file *m, void *data)
+{
+	struct xe_device *xe = m->private;
+	unsigned int i;
+
+	seq_printf(m, "%-11s ", "");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " ---%2u---", i);

drivers/gpu/drm/xe/xe_debugfs.c static int dgfx_pcie_link_residencies_show(struct seq_file *m, void *data) · line 252

Quoted 10 lines
drivers/gpu/drm/xe/xe_debugfs.c +10 -0
+	seq_printf(m, "\n%-11s:", "bo");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " %8lu",
+			   atomic_long_read(&xe->mem.dma_mapped_pages[i]));
+	seq_printf(m, "\n%-11s:", "svm/userptr");
+	for (i = 0; i < NR_PAGE_ORDERS; i++)
+		seq_printf(m, " %8lu",
+			   atomic_long_read(&xe->mem.dma_mapped_pages_svm[i]));
+	seq_puts(m, "\n");
+

NIT: As order-10 also prints higher order (1GB) pages, may be print it as '>=10'?

NIT: This is a rough snapshot as array elements could be changing as we print. I guess that is fine.

Quoted 34 lines
drivers/gpu/drm/xe/xe_debugfs.c +7 -0
+	return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(dma_mapped_pages);
+
 static const struct drm_info_list debugfs_list[] = {
 	{"info", info, 0},
 	{ .name = "sriov_info", .show = sriov_info, },
@@ -632,6 +655,9 @@ void xe_debugfs_register(struct xe_device *xe)
 	if (man)
 		ttm_resource_manager_create_debugfs(man, root, "stolen_mm");
 
+	debugfs_create_file("dma_mapped_pages", 0444, root, xe,
+			    &dma_mapped_pages_fops);
+
 	for_each_tile(tile, xe, tile_id)
 		xe_tile_debugfs_register(tile);
 
drivers/gpu/drm/xe/xe_device.c +9 -0
index ad7f3e61d457..97cc2d185e45 100644
@@ -455,12 +455,42 @@ bool xe_device_is_admin_only(const struct xe_device *xe)
 }
 #endif
 
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+static void xe_device_assert_dma_pages_zero(struct xe_device *xe)
+{
+	unsigned int i;
+
+	/*
+	 * All BOs, userptr VMAs and SVM ranges must have been torn down by the
+	 * time the device is destroyed, so every DMA-mapped-pages counter must
+	 * have returned to zero. A non-zero value indicates unbalanced
Quoted 9 lines

drivers/gpu/drm/xe/xe_device.c bool xe_device_is_admin_only(const struct xe_device *xe) · line 471

Quoted 5 lines
drivers/gpu/drm/xe/xe_device.c +5 -0
+	 * accounting, i.e. a missing unmap-side decrement.
+	 */
+	for (i = 0; i < NR_PAGE_ORDERS; i++) {
+		xe_assert(xe, !atomic_long_read(&xe->mem.dma_mapped_pages[i]));
+		xe_assert(xe, !atomic_long_read(&xe->mem.dma_mapped_pages_svm[i]));

This also probably good to do it under a separate xe DEBUG config instead of CONFIG_DEBUG_FS.

I can make this change but IMO this is needed. I would think we'd want these stats on all production systems for debug, i.e., CONFIG_DEBUG_FS is likely the right Kconfig.

Matt

Quoted 242 lines

Niranjana

Quoted 240 lines
drivers/gpu/drm/xe/xe_device.c +16 -0
+	}
+}
+#else
+static void xe_device_assert_dma_pages_zero(struct xe_device *xe)
+{
+}
+#endif
+
 static void xe_device_destroy(struct drm_device *dev, void *dummy)
 {
 	struct xe_device *xe = to_xe_device(dev);
 
 	xe_bo_dev_fini(&xe->bo_device);
 
+	/*
+	 * Assert the DMA-mapped-pages accounting only after xe_bo_dev_fini()
+	 * has flushed the async BO-free worker: BOs pending async free still
+	 * hold their DMA mappings (and counts) until the worker runs, so
+	 * asserting before the flush could trip a false positive.
+	 */
+	xe_device_assert_dma_pages_zero(xe);
+
 	if (xe->preempt_fence_wq)
 		destroy_workqueue(xe->preempt_fence_wq);
 
drivers/gpu/drm/xe/xe_device_types.h +17 -0
index 022e08205897..95e7cea91cbf 100644
@@ -6,6 +6,7 @@
 #ifndef _XE_DEVICE_TYPES_H_
 #define _XE_DEVICE_TYPES_H_
 
+#include <linux/mmzone.h>
 #include <linux/pci.h>
 
 #include <drm/drm_device.h>
@@ -279,6 +280,22 @@ struct xe_device {
 		struct xe_shrinker *shrinker;
 		/** @mem.stolen_mgr: stolen memory manager. */
 		struct xe_ttm_stolen_mgr *stolen_mgr;
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+		/**
+		 * @mem.dma_mapped_pages: number of DMA-mapped pages per page
+		 * order currently live for this device, for TTM BOs. Only
+		 * accounted when CONFIG_DEBUG_FS is enabled, since it is solely
+		 * exposed through debugfs.
+		 */
+		atomic_long_t dma_mapped_pages[NR_PAGE_ORDERS];
+		/**
+		 * @mem.dma_mapped_pages_svm: number of DMA-mapped pages per
+		 * page order currently live for this device, for SVM ranges
+		 * and userptr VMAs (both share a single drm_gpusvm instance).
+		 * Only accounted when CONFIG_DEBUG_FS is enabled.
+		 */
+		atomic_long_t dma_mapped_pages_svm[NR_PAGE_ORDERS];
+#endif
 	} mem;
 
 	/** @sriov: device level virtualization data */
drivers/gpu/drm/xe/xe_svm.c +42 -2
index b228a737cfd6..ebe48cb76ad9 100644
@@ -827,10 +827,50 @@ static int xe_svm_get_pagemaps(struct xe_vm *vm)
 }
 #endif
 
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+/*
+ * xe_svm_dma_map_account - drm_gpusvm DMA-mapping accounting callback
+ * @gpusvm: The GPU SVM the mapping belongs to
+ * @addr: The address descriptor of the chunk being (un)mapped
+ * @sign: +1 when @addr was DMA-mapped, -1 when it is being unmapped
+ *
+ * Maintain per-order counts of the system-memory pages that the full SVM
+ * instance (SVM ranges, and userptr VMAs in fault mode) has DMA-mapped for GPU
+ * access. Only DRM_INTERCONNECT_SYSTEM entries are real DMA maps and counted;
+ * device interconnect (VRAM, P2P) entries are skipped. The counts are solely
+ * exposed through debugfs, so the callback is only registered when
+ * CONFIG_DEBUG_FS is enabled.
+ */
+static void xe_svm_dma_map_account(struct drm_gpusvm *gpusvm,
+				   const struct drm_pagemap_addr *addr,
+				   int sign)
+{
+	struct xe_device *xe = gpusvm_to_vm(gpusvm)->xe;
+	unsigned int order = addr->order;
+
+	if (addr->proto != DRM_INTERCONNECT_SYSTEM)
+		return;
+
+	/*
+	 * The CPU-PTE map order reported by hmm can exceed MAX_PAGE_ORDER
+	 * (e.g. an order-18 1GB mapping), so clamp the array index to avoid an
+	 * out-of-bounds access. Clamping is symmetric across the +1/-1 calls,
+	 * so the per-order counters stay balanced.
+	 */
+	order = min(order, (unsigned int)NR_PAGE_ORDERS - 1);
+
+	atomic_long_add((long)(1UL << addr->order) * sign,
+			&xe->mem.dma_mapped_pages_svm[order]);
+}
+#endif
+
 static const struct drm_gpusvm_ops gpusvm_ops = {
 	.range_alloc = xe_svm_range_alloc,
 	.range_free = xe_svm_range_free,
 	.invalidate = xe_svm_invalidate,
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+	.dma_map_account = xe_svm_dma_map_account,
+#endif
 };
 
 static const unsigned long fault_chunk_sizes[] = {
@@ -925,8 +965,8 @@ int xe_svm_init(struct xe_vm *vm)
 		}
 	} else {
 		err = drm_gpusvm_init(&vm->svm.gpusvm, "Xe SVM (simple)",
-				      NULL, 0, 0, 0, NULL,
-				      NULL, 0);
+				      NULL, 0, 0, 0,
+				      xe_userptr_gpusvm_ops_get(), NULL, 0);
 	}
 
 	return err;
drivers/gpu/drm/xe/xe_svm.h +3 -1
index a921556d3466..8e347137a7a6 100644
@@ -194,6 +194,7 @@ struct drm_pagemap *xe_drm_pagemap_from_fd(int fd, u32 region_instance);
 
 #else
 #include <linux/interval_tree.h>
+#include "xe_userptr.h"
 #include "xe_vm.h"
 
 struct drm_pagemap_addr;
@@ -236,7 +237,8 @@ int xe_svm_init(struct xe_vm *vm)
 {
 #if IS_ENABLED(CONFIG_DRM_GPUSVM)
 	return drm_gpusvm_init(&vm->svm.gpusvm, "Xe SVM (simple)",
-			       NULL, 0, 0, 0, NULL, NULL, 0);
+			       NULL, 0, 0, 0,
+			       xe_userptr_gpusvm_ops_get(), NULL, 0);
 #else
 	return 0;
 #endif
drivers/gpu/drm/xe/xe_userptr.c +63 -0
index 8b2d461ea0b2..6b9e1530ead4 100644
@@ -8,9 +8,72 @@
 
 #include <linux/mm.h>
 
+#include <drm/drm_pagemap.h>
+
 #include "xe_tlb_inval.h"
 #include "xe_trace_bo.h"
 
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+/*
+ * xe_userptr_dma_map_account - drm_gpusvm DMA-mapping accounting callback
+ * @gpusvm: The GPU SVM the mapping belongs to
+ * @addr: The address descriptor of the chunk being (un)mapped
+ * @sign: +1 when @addr was DMA-mapped, -1 when it is being unmapped
+ *
+ * Account, per page order, the system-memory pages DMA-mapped for GPU access
+ * through the drm_gpusvm_pages instance shared by userptr (and, in fault mode,
+ * SVM). Only DRM_INTERCONNECT_SYSTEM entries are real DMA maps and counted;
+ * device interconnect (VRAM, P2P) entries are skipped. The counts are solely
+ * exposed through debugfs, so the callback is only registered when
+ * CONFIG_DEBUG_FS is enabled.
+ */
+static void xe_userptr_dma_map_account(struct drm_gpusvm *gpusvm,
+				       const struct drm_pagemap_addr *addr,
+				       int sign)
+{
+	struct xe_vm *vm = container_of(gpusvm, struct xe_vm, svm.gpusvm);
+	unsigned int order = addr->order;
+
+	if (addr->proto != DRM_INTERCONNECT_SYSTEM)
+		return;
+
+	/*
+	 * The CPU-PTE map order reported by hmm can exceed MAX_PAGE_ORDER
+	 * (e.g. an order-18 1GB mapping), so clamp the array index to avoid an
+	 * out-of-bounds access. Clamping is symmetric across the +1/-1 calls,
+	 * so the per-order counters stay balanced.
+	 */
+	order = min(order, (unsigned int)NR_PAGE_ORDERS - 1);
+
+	atomic_long_add((long)(1UL << addr->order) * sign,
+			&vm->xe->mem.dma_mapped_pages_svm[order]);
+}
+
+static const struct drm_gpusvm_ops xe_userptr_gpusvm_ops = {
+	.dma_map_account = xe_userptr_dma_map_account,
+};
+#endif
+
+/**
+ * xe_userptr_gpusvm_ops_get() - Accounting ops for the simple gpusvm instance
+ *
+ * The core drm_gpusvm_pages-only instance used for userptr (and, in fault
+ * mode, shared with SVM) is initialised with these ops so that DMA-mapped
+ * system pages are accounted and exposed through debugfs. Returns NULL when
+ * CONFIG_DEBUG_FS is disabled, in which case no accounting is kept and the
+ * instance is initialised without ops.
+ *
+ * Return: Pointer to the restricted &drm_gpusvm_ops, or NULL.
+ */
+const struct drm_gpusvm_ops *xe_userptr_gpusvm_ops_get(void)
+{
+#if IS_ENABLED(CONFIG_DEBUG_FS)
+	return &xe_userptr_gpusvm_ops;
+#else
+	return NULL;
+#endif
+}
+
 static void xe_userptr_assert_in_notifier(struct xe_vm *vm)
 {
 	lockdep_assert(lockdep_is_held_type(&vm->svm.gpusvm.notifier_lock, 0) ||
drivers/gpu/drm/xe/xe_userptr.h +1 -0
index 2a3cd1b5efbb..1e9601be5713 100644
@@ -108,6 +108,7 @@ int __xe_vm_userptr_needs_repin(struct xe_vm *vm);
 int xe_vm_userptr_check_repin(struct xe_vm *vm);
 int xe_vma_userptr_pin_pages(struct xe_userptr_vma *uvma);
 int xe_vma_userptr_check_repin(struct xe_userptr_vma *uvma);
+const struct drm_gpusvm_ops *xe_userptr_gpusvm_ops_get(void);
 #else
 static inline void xe_userptr_remove(struct xe_userptr_vma *uvma) {}
 
Signature · 1 line
2.34.1
author maintainer
#45 lore raw

PATCH v3 13/33 drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind()

In reply to Matthew Brost

On L2-flush-optimized HW with a dma-resv (non-fault) VM, evicting a BO only needs to flush the L2 cache before the migration copy runs; the mappings themselves are torn down and rebuilt lazily via drm_gpuvm_bo_evict() and a subsequent rebind. Today this flush is done by waiting for the BO to go idle and then issuing a synchronous TLB invalidation per mapping VMA from inside xe_bo_trigger_rebind(). Both the idle wait and the synchronous invalidation stall the calling thread while holding the BO dma-resv lock, serializing the move behind all in-flight GPU work on the BO.

Replace this with an asynchronous flush. Add xe_vm_flush_vm_bo_tlb_async() which, for each VMA mapping the BO on each present tile, queues a TLB invalidation job on the tile migrate (kernel) exec queue. The jobs depend on the BO's in-flight GPU work, captured once as a singleton over DMA_RESV_USAGE_BOOKKEEP, so the flush only fires once the GPU is done with the current mapping. Each job's completion fence is installed into the BO's dma-resv as a DMA_RESV_USAGE_KERNEL fence, so the migration copy - which waits on the resv - waits on the flush without stalling this thread.

No PTEs are zapped and vma->tile_invalidated is left untouched: the mapping stays valid until the lazy rebind, and the only work performed here is the L2 flush. On any failure the caller falls back to the existing blocking wait-idle plus xe_vm_invalidate_vma() path.

Cc Thomas Hellström <[email protected]>
Cc Tejas Upadhyay <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---

v3:
- Gate the asynchronous L2 flush on !ctx->no_wait_gpu (Sashiko)

---
 drivers/gpu/drm/xe/xe_bo.c |  13 ++++
 drivers/gpu/drm/xe/xe_vm.c | 125 +++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_vm.h |   5 ++
 3 files changed, 143 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +13 -0
index 60e0e568aa31..4ee48cdc02ac 100644
@@ -732,6 +732,19 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo,
 			 */
 			if (!xe_device_is_l2_flush_optimized(xe))
 				continue;
+
+			/*
+			 * Attempt to flush L2 async, fallback to sync flush on
+			 * failure. Skip the async attempt under no_wait_gpu
+			 * (e.g. the shrinker's reclaim pass): job creation
+			 * allocates with GFP_KERNEL, which must not run while
+			 * holding the dma-resv lock under fs_reclaim. The sync
+			 * fallback below honors no_wait_gpu (returns -EBUSY if
+			 * the BO is not idle).
+			 */
+			if (!ctx->no_wait_gpu &&
+			    !xe_vm_flush_vm_bo_tlb_async(vm, bo, vm_bo))
+				continue;
 		}
 
 		if (!idle) {
drivers/gpu/drm/xe/xe_vm.c +125 -0

137 lines not on this page · raw message

drivers/gpu/drm/xe/xe_vm.h +5 -0
index c5b900f38ded..dd5b070eaede 100644
@@ -26,6 +26,8 @@ struct ttm_buffer_object;
 
 struct dma_fence;
 
+struct xe_bo;
+struct drm_gpuvm_bo;
 struct xe_exec_queue;
 struct xe_file;
 struct xe_pagefault;
@@ -254,6 +256,9 @@ int xe_vm_invalidate_vma(struct xe_vma *vma);
 
 int xe_vm_invalidate_vma_submit(struct xe_vma *vma, struct xe_tlb_inval_batch *batch);
 
+int xe_vm_flush_vm_bo_tlb_async(struct xe_vm *vm, struct xe_bo *bo,
+				struct drm_gpuvm_bo *vm_bo);
+
 int xe_vm_validate_protected(struct xe_vm *vm);
 
 static inline void xe_vm_queue_rebind_worker(struct xe_vm *vm)
Signature · 1 line
2.34.1
reviewer
#46 lore raw

Re: [PATCH v3 13/33] drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind()

Quoted 24 lines · Matthew Brost

On L2-flush-optimized HW with a dma-resv (non-fault) VM, evicting a BO only needs to flush the L2 cache before the migration copy runs; the mappings themselves are torn down and rebuilt lazily via drm_gpuvm_bo_evict() and a subsequent rebind. Today this flush is done by waiting for the BO to go idle and then issuing a synchronous TLB invalidation per mapping VMA from inside xe_bo_trigger_rebind(). Both the idle wait and the synchronous invalidation stall the calling thread while holding the BO dma-resv lock, serializing the move behind all in-flight GPU work on the BO.

Replace this with an asynchronous flush. Add xe_vm_flush_vm_bo_tlb_async() which, for each VMA mapping the BO on each present tile, queues a TLB invalidation job on the tile migrate (kernel) exec queue. The jobs depend on the BO's in-flight GPU work, captured once as a singleton over DMA_RESV_USAGE_BOOKKEEP, so the flush only fires once the GPU is done with the current mapping. Each job's completion fence is installed into the BO's dma-resv as a DMA_RESV_USAGE_KERNEL fence, so the migration copy - which waits on the resv - waits on the flush without stalling this thread.

No PTEs are zapped and vma->tile_invalidated is left untouched: the mapping stays valid until the lazy rebind, and the only work performed here is the L2 flush. On any failure the caller falls back to the existing blocking wait-idle plus xe_vm_invalidate_vma() path.

Quoted 4 lines
Cc Thomas Hellström <[email protected]>
Cc Tejas Upadhyay <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

I think this one was split out right? But not merged yet?

author maintainer
#47 lore raw

Re: [PATCH v3 13/33] drm/xe: Flush L2 asynchronously in xe_bo_trigger_rebind()

Quoted 25 lines · Matthew Auld
Quoted 24 lines · Matthew Brost

On L2-flush-optimized HW with a dma-resv (non-fault) VM, evicting a BO only needs to flush the L2 cache before the migration copy runs; the mappings themselves are torn down and rebuilt lazily via drm_gpuvm_bo_evict() and a subsequent rebind. Today this flush is done by waiting for the BO to go idle and then issuing a synchronous TLB invalidation per mapping VMA from inside xe_bo_trigger_rebind(). Both the idle wait and the synchronous invalidation stall the calling thread while holding the BO dma-resv lock, serializing the move behind all in-flight GPU work on the BO.

Replace this with an asynchronous flush. Add xe_vm_flush_vm_bo_tlb_async() which, for each VMA mapping the BO on each present tile, queues a TLB invalidation job on the tile migrate (kernel) exec queue. The jobs depend on the BO's in-flight GPU work, captured once as a singleton over DMA_RESV_USAGE_BOOKKEEP, so the flush only fires once the GPU is done with the current mapping. Each job's completion fence is installed into the BO's dma-resv as a DMA_RESV_USAGE_KERNEL fence, so the migration copy - which waits on the resv - waits on the flush without stalling this thread.

No PTEs are zapped and vma->tile_invalidated is left untouched: the mapping stays valid until the lazy rebind, and the only work performed here is the L2 flush. On any failure the caller falls back to the existing blocking wait-idle plus xe_vm_invalidate_vma() path.

Quoted 6 lines
Quoted 4 lines
Cc Thomas Hellström <[email protected]>
Cc Tejas Upadhyay <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

I think this one was split out right? But not merged yet?

Yes, we can split this one out and in fact did earlier [1].

Let me respin this one as standalone patch today.

Matt

[1] https://patchwork.freedesktop.org/series/170046/

author maintainer
#48 lore raw

PATCH v3 14/33 drm/xe: Destroy page tables after unlinking all VMAs on VM close

In reply to Matthew Brost

xe_vm_close_and_put() freed the VM's page-table roots via xe_vm_pt_destroy() before destroying the "contested" VMAs - those backed by external (shared) BOs, which are deferred to xe_vma_destroy_unlocked() because unlinking them requires taking the external BO's dma-resv lock (via drm_exec), not vm->lock.

Until such a VMA is unlinked it remains reachable from its BO through drm_gem_for_each_gpuvm_bo(). A concurrent BO move that only holds the BO's dma-resv lock - TTM eviction, the shrinker, or the defrag worker - calls xe_bo_move_notify() -> xe_bo_trigger_rebind() -> xe_vm_invalidate_vma() -> xe_pt_zap_ptes(), which dereferences vm->pt_root[tile->id]. With the page tables already freed this is a use-after-free / NULL dereference, e.g.:

  Oops: general protection fault [...] kernel NULL pointer dereference
  RIP: xe_pt_zap_ptes+0x7c [xe]
  Call Trace:
   xe_vm_invalidate_vma_submit+0x64/0x130 [xe]
   xe_vm_invalidate_vma+0x27/0x60 [xe]
   xe_bo_move_notify+0x1e7/0x3d0 [xe]
   xe_bo_move+0x62d/0x1620 [xe]
   ttm_bo_handle_move_mem+0x243/0x330 [ttm]
   ttm_bo_validate+0xb4/0x1a0 [ttm]

The race is not closable with a flag or a NULL pt_root check in the invalidate path: the freeing side holds only vm->lock while the walker holds only the BO's resv, so there is no common lock to serialise on.

Fix the ordering instead: destroy the page tables only after every VMA - including the contested external-BO VMAs - has been unlinked. Once the contested loop completes no BO can reach a VMA of this VM, so freeing the page tables (under vm->lock, to satisfy xe_vm_assert_held()) is safe. xe_vm_free_scratch() still runs before the free as it reads pt_root[]->level.

Fixes dd08ebf6c352 ("drm/xe: Introduce a new DRM driver for Intel GPUs")
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_vm.c | 24 ++++++++++++++++--------
 1 file changed, 16 insertions(+), 8 deletions(-)
drivers/gpu/drm/xe/xe_vm.c +16 -8
index 4557a8a4d270..9f55d22cb985 100644
@@ -1934,15 +1934,7 @@ void xe_vm_close_and_put(struct xe_vm *vm)
 		vma->gpuva.flags |= XE_VMA_DESTROYED;
 	}
 
-	/*
-	 * All vm operations will add shared fences to resv.
-	 * The only exception is eviction for a shared object,
-	 * but even so, the unbind when evicted would still
-	 * install a fence to resv. Hence it's safe to
-	 * destroy the pagetables immediately.
-	 */
 	xe_vm_free_scratch(vm);
-	xe_vm_pt_destroy(vm);
 	xe_vm_unlock(vm);
 
 	/*
@@ -1956,6 +1948,22 @@ void xe_vm_close_and_put(struct xe_vm *vm)
 		xe_vma_destroy_unlocked(vma);
 	}
 
+	/*
+	 * Destroy the page tables only after every VMA has been unlinked. The
+	 * contested (external-BO) VMAs above are unlinked from their BO under
+	 * the BO's dma-resv lock, not vm->lock, so they can still be reached
+	 * via drm_gem_for_each_gpuvm_bo() by a concurrent BO move (eviction,
+	 * shrinker or the defrag worker) that only holds the BO's resv. Such a
+	 * move calls xe_vm_invalidate_vma() -> xe_pt_zap_ptes(), which
+	 * dereferences vm->pt_root[]; freeing the page tables while those VMAs
+	 * are still linked is a use-after-free. All vm operations add shared
+	 * fences to the resv (eviction of a shared object still installs an
+	 * unbind fence), so no GPU work outlives this point.
+	 */
+	xe_vm_lock(vm, false);
+	xe_vm_pt_destroy(vm);
+	xe_vm_unlock(vm);
+
 	xe_svm_fini(vm);
 
 	up_write(&vm->lock);
Signature · 1 line
2.34.1
#49 lore raw

Re: [PATCH v3 14/33] drm/xe: Destroy page tables after unlinking all VMAs on VM close

Quoted 35 lines · Matthew Brost

xe_vm_close_and_put() freed the VM's page-table roots via xe_vm_pt_destroy() before destroying the "contested" VMAs - those backed by external (shared) BOs, which are deferred to xe_vma_destroy_unlocked() because unlinking them requires taking the external BO's dma-resv lock (via drm_exec), not vm->lock.

Until such a VMA is unlinked it remains reachable from its BO through drm_gem_for_each_gpuvm_bo(). A concurrent BO move that only holds the BO's dma-resv lock - TTM eviction, the shrinker, or the defrag worker - calls xe_bo_move_notify() -> xe_bo_trigger_rebind() -> xe_vm_invalidate_vma() -> xe_pt_zap_ptes(), which dereferences vm->pt_root[tile->id]. With the page tables already freed this is a use-after-free / NULL dereference, e.g.:

 Oops: general protection fault [...] kernel NULL pointer dereference
 RIP: xe_pt_zap_ptes+0x7c [xe]
 Call Trace:
  xe_vm_invalidate_vma_submit+0x64/0x130 [xe]
  xe_vm_invalidate_vma+0x27/0x60 [xe]
  xe_bo_move_notify+0x1e7/0x3d0 [xe]
  xe_bo_move+0x62d/0x1620 [xe]
  ttm_bo_handle_move_mem+0x243/0x330 [ttm]
  ttm_bo_validate+0xb4/0x1a0 [ttm]

The race is not closable with a flag or a NULL pt_root check in the invalidate path: the freeing side holds only vm->lock while the walker holds only the BO's resv, so there is no common lock to serialise on.

Fix the ordering instead: destroy the page tables only after every VMA - including the contested external-BO VMAs - has been unlinked. Once the contested loop completes no BO can reach a VMA of this VM, so freeing the page tables (under vm->lock, to satisfy xe_vm_assert_held()) is safe. xe_vm_free_scratch() still runs before the free as it reads pt_root[]->level.

Quoted 3 lines
Fixes dd08ebf6c352 ("drm/xe: Introduce a new DRM driver for Intel GPUs")
Assisted-by GitHub_Copilot:claude-opus-4.8

Looks like this change is already merged with below commit "drm/xe/vm: nuke PTs only after unlinking contested VMAs"

So, needs to be dropped in this series.

Niranjana

Quoted 50 lines

---
drivers/gpu/drm/xe/xe_vm.c | 24 ++++++++++++++++--------

1 file changed, 16 insertions(+), 8 deletions(-)
drivers/gpu/drm/xe/xe_vm.c +16 -8
index 4557a8a4d270..9f55d22cb985 100644
@@ -1934,15 +1934,7 @@ void xe_vm_close_and_put(struct xe_vm *vm)
 		vma->gpuva.flags |= XE_VMA_DESTROYED;
 	}
 
-	/*
-	 * All vm operations will add shared fences to resv.
-	 * The only exception is eviction for a shared object,
-	 * but even so, the unbind when evicted would still
-	 * install a fence to resv. Hence it's safe to
-	 * destroy the pagetables immediately.
-	 */
 	xe_vm_free_scratch(vm);
-	xe_vm_pt_destroy(vm);
 	xe_vm_unlock(vm);
 
 	/*
@@ -1956,6 +1948,22 @@ void xe_vm_close_and_put(struct xe_vm *vm)
 		xe_vma_destroy_unlocked(vma);
 	}
 
+	/*
+	 * Destroy the page tables only after every VMA has been unlinked. The
+	 * contested (external-BO) VMAs above are unlinked from their BO under
+	 * the BO's dma-resv lock, not vm->lock, so they can still be reached
+	 * via drm_gem_for_each_gpuvm_bo() by a concurrent BO move (eviction,
+	 * shrinker or the defrag worker) that only holds the BO's resv. Such a
+	 * move calls xe_vm_invalidate_vma() -> xe_pt_zap_ptes(), which
+	 * dereferences vm->pt_root[]; freeing the page tables while those VMAs
+	 * are still linked is a use-after-free. All vm operations add shared
+	 * fences to the resv (eviction of a shared object still installs an
+	 * unbind fence), so no GPU work outlives this point.
+	 */
+	xe_vm_lock(vm, false);
+	xe_vm_pt_destroy(vm);
+	xe_vm_unlock(vm);
+
 	xe_svm_fini(vm);
 
 	up_write(&vm->lock);
Signature · 1 line
2.34.1
author maintainer
#50 lore raw

Re: [PATCH v3 14/33] drm/xe: Destroy page tables after unlinking all VMAs on VM close

Quoted 36 lines · Niranjana Vishwanathapura
Quoted 35 lines · Matthew Brost

xe_vm_close_and_put() freed the VM's page-table roots via xe_vm_pt_destroy() before destroying the "contested" VMAs - those backed by external (shared) BOs, which are deferred to xe_vma_destroy_unlocked() because unlinking them requires taking the external BO's dma-resv lock (via drm_exec), not vm->lock.

Until such a VMA is unlinked it remains reachable from its BO through drm_gem_for_each_gpuvm_bo(). A concurrent BO move that only holds the BO's dma-resv lock - TTM eviction, the shrinker, or the defrag worker - calls xe_bo_move_notify() -> xe_bo_trigger_rebind() -> xe_vm_invalidate_vma() -> xe_pt_zap_ptes(), which dereferences vm->pt_root[tile->id]. With the page tables already freed this is a use-after-free / NULL dereference, e.g.:

 Oops: general protection fault [...] kernel NULL pointer dereference
 RIP: xe_pt_zap_ptes+0x7c [xe]
 Call Trace:
  xe_vm_invalidate_vma_submit+0x64/0x130 [xe]
  xe_vm_invalidate_vma+0x27/0x60 [xe]
  xe_bo_move_notify+0x1e7/0x3d0 [xe]
  xe_bo_move+0x62d/0x1620 [xe]
  ttm_bo_handle_move_mem+0x243/0x330 [ttm]
  ttm_bo_validate+0xb4/0x1a0 [ttm]

The race is not closable with a flag or a NULL pt_root check in the invalidate path: the freeing side holds only vm->lock while the walker holds only the BO's resv, so there is no common lock to serialise on.

Fix the ordering instead: destroy the page tables only after every VMA - including the contested external-BO VMAs - has been unlinked. Once the contested loop completes no BO can reach a VMA of this VM, so freeing the page tables (under vm->lock, to satisfy xe_vm_assert_held()) is safe. xe_vm_free_scratch() still runs before the free as it reads pt_root[]->level.

Quoted 9 lines
Quoted 3 lines
Fixes dd08ebf6c352 ("drm/xe: Introduce a new DRM driver for Intel GPUs")
Assisted-by GitHub_Copilot:claude-opus-4.8

Looks like this change is already merged with below commit "drm/xe/vm: nuke PTs only after unlinking contested VMAs"

So, needs to be dropped in this series.

Yes, Auld got to this one so it can be droppped.

Matt

Quoted 52 lines

Niranjana

Quoted 50 lines

---
drivers/gpu/drm/xe/xe_vm.c | 24 ++++++++++++++++--------

1 file changed, 16 insertions(+), 8 deletions(-)
drivers/gpu/drm/xe/xe_vm.c +16 -8
index 4557a8a4d270..9f55d22cb985 100644
@@ -1934,15 +1934,7 @@ void xe_vm_close_and_put(struct xe_vm *vm)
 		vma->gpuva.flags |= XE_VMA_DESTROYED;
 	}
 
-	/*
-	 * All vm operations will add shared fences to resv.
-	 * The only exception is eviction for a shared object,
-	 * but even so, the unbind when evicted would still
-	 * install a fence to resv. Hence it's safe to
-	 * destroy the pagetables immediately.
-	 */
 	xe_vm_free_scratch(vm);
-	xe_vm_pt_destroy(vm);
 	xe_vm_unlock(vm);
 
 	/*
@@ -1956,6 +1948,22 @@ void xe_vm_close_and_put(struct xe_vm *vm)
 		xe_vma_destroy_unlocked(vma);
 	}
 
+	/*
+	 * Destroy the page tables only after every VMA has been unlinked. The
+	 * contested (external-BO) VMAs above are unlinked from their BO under
+	 * the BO's dma-resv lock, not vm->lock, so they can still be reached
+	 * via drm_gem_for_each_gpuvm_bo() by a concurrent BO move (eviction,
+	 * shrinker or the defrag worker) that only holds the BO's resv. Such a
+	 * move calls xe_vm_invalidate_vma() -> xe_pt_zap_ptes(), which
+	 * dereferences vm->pt_root[]; freeing the page tables while those VMAs
+	 * are still linked is a use-after-free. All vm operations add shared
+	 * fences to the resv (eviction of a shared object still installs an
+	 * unbind fence), so no GPU work outlives this point.
+	 */
+	xe_vm_lock(vm, false);
+	xe_vm_pt_destroy(vm);
+	xe_vm_unlock(vm);
+
 	xe_svm_fini(vm);
 
 	up_write(&vm->lock);
Signature · 1 line
2.34.1
author maintainer
#51 lore raw

PATCH v3 15/33 drm/xe: Track BOs backed at a sub-optimal page order

In reply to Matthew Brost

Add per-device tracking of BOs whose backing TT pages were allocated below the pool's beneficial order, as reported by the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tt page flag (read via ttm_tt_is_beneficial_order_failed()). These BOs are future candidates for a defragmentation pass that reallocates their pages at the beneficial order.

Add a mem.defrag sub-struct (lock, list, atomic count) to struct xe_device and a defrag_link to struct xe_bo, initialised at BO create and device init. Membership is updated from xe_bo_move() via xe_bo_defrag_update(): a BO is added when it is a populated, needed, non-pinned ttm_bo_type_device object flagged TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED, and removed otherwise. BOs are also removed on pin, on purge, moved to dontneed, and at destroy.

Only ttm_bo_type_device BOs are tracked.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_bo.c           | 115 +++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_bo.h           |   1 +
 drivers/gpu/drm/xe/xe_bo_types.h     |   6 ++
 drivers/gpu/drm/xe/xe_device.c       |   2 +
 drivers/gpu/drm/xe/xe_device_types.h |  19 +++++
 5 files changed, 143 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +115 -0
index 4ee48cdc02ac..ff324caab6da 100644
@@ -926,6 +926,9 @@ static void xe_bo_set_purgeable_shrinker(struct xe_bo *bo,
 	}
 }
 
+static void xe_bo_defrag_update(struct xe_bo *bo);
+static void xe_bo_defrag_remove(struct xe_bo *bo);
+
 /**
  * xe_bo_set_purgeable_state() - Set BO purgeable state with validation
  * @bo: Buffer object
@@ -954,6 +957,7 @@ void xe_bo_set_purgeable_state(struct xe_bo *bo,
 
 	bo->purgeable.state = new_state;
 	xe_bo_set_purgeable_shrinker(bo, new_state);
+	xe_bo_defrag_update(bo);
 }
 
 /**
@@ -1003,9 +1007,100 @@ static int xe_ttm_bo_purge(struct ttm_buffer_object *ttm_bo, struct ttm_operatio
 	/* Commit the state transition only once invalidation was queued */
 	xe_bo_set_purgeable_state(bo, XE_MADV_PURGEABLE_PURGED);
 
+	/*
+	 * The backing store is gone, so the BO no longer needs defrag. Drop it
+	 * from the defrag list here so the count stays accurate even for purge
+	 * paths that don't pass through xe_bo_move() (e.g. swap_notify).
+	 */
+	xe_bo_defrag_remove(bo);
+
 	return 0;
 }
 
+/**
+ * xe_bo_defrag_init_early() - Initialize the device defrag BO tracking
+ * @xe: The xe device
+ *
+ * Initialize the list, lock and count used to track BOs whose backing TT
+ * pages were allocated at a sub-optimal order.
+ */
+void xe_bo_defrag_init_early(struct xe_device *xe)
+{
+	spin_lock_init(&xe->mem.defrag.lock);
+	INIT_LIST_HEAD(&xe->mem.defrag.list);
+	atomic_set(&xe->mem.defrag.count, 0);
+}
+
+static bool xe_bo_needs_defrag(struct xe_bo *bo)
+{
+	struct ttm_buffer_object *ttm_bo = &bo->ttm;
+	struct ttm_tt *tt = ttm_bo->ttm;
+
+	xe_bo_assert_held(bo);
+
+	return ttm_bo->type == ttm_bo_type_device && tt &&
+		!xe_bo_madv_is_dontneed(bo) &&
+		ttm_tt_is_populated(tt) &&
+		ttm_tt_is_beneficial_order_failed(tt) &&
+		ttm_bo->resource && ttm_bo->resource->mem_type == XE_PL_TT &&
+		!xe_bo_is_pinned(bo);
+}
+
+static void xe_bo_defrag_add(struct xe_bo *bo)
+{
+	struct xe_device *xe = xe_bo_device(bo);
+
+	xe_bo_assert_held(bo);
+	xe_assert(xe, xe_bo_needs_defrag(bo));
+
+	scoped_guard(spinlock, &xe->mem.defrag.lock) {
+		if (list_empty(&bo->defrag_link)) {
+			list_add_tail(&bo->defrag_link, &xe->mem.defrag.list);
+			atomic_inc(&xe->mem.defrag.count);
+		}
+	}
+}
+
+static void __xe_bo_defrag_remove(struct xe_bo *bo)
+{
+	struct xe_device *xe = xe_bo_device(bo);
+
+	guard(spinlock)(&xe->mem.defrag.lock);
+	if (!list_empty(&bo->defrag_link)) {
+		list_del_init(&bo->defrag_link);
+		atomic_dec(&xe->mem.defrag.count);
+	}
+}
+
+static void xe_bo_defrag_remove(struct xe_bo *bo)
+{
+	xe_bo_assert_held(bo);
+
+	if (list_empty(&bo->defrag_link))
+		return;
+
+	__xe_bo_defrag_remove(bo);
+}
+
+/**
+ * xe_bo_defrag_update() - Update defrag list membership for a BO
+ * @bo: The buffer object
+ *
+ * Add @bo to the device defrag list when it is a ttm_bo_type_device BO resident
+ * in XE_PL_TT with a populated TT whose pages were allocated at a sub-optimal
+ * order (ttm_tt_is_beneficial_order_failed()) and it isn't pinned. Otherwise
+ * ensure it is removed from the list.
+ */
+static void xe_bo_defrag_update(struct xe_bo *bo)
+{
+	xe_bo_assert_held(bo);
+
+	if (xe_bo_needs_defrag(bo))
+		xe_bo_defrag_add(bo);
+	else
+		xe_bo_defrag_remove(bo);
+}
+
 static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		      struct ttm_operation_ctx *ctx,
 		      struct ttm_resource *new_mem,
@@ -1231,6 +1326,8 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		xe_tt_unmap_sg(xe, ttm_bo->ttm);
 	}
 
+	xe_bo_defrag_update(bo);
+
 	return ret;
 }
 
@@ -1267,6 +1364,15 @@ static long xe_bo_shrink_purge(struct ttm_operation_ctx *ctx,
 	if (lret > 0) {
 		xe_ttm_tt_account_subtract(xe, bo->ttm);
 		update_global_total_pages(bo->bdev, -(long)tt->num_pages);
+
+		/*
+		 * The pages are gone, so a tracked BO no longer needs defrag.
+		 * This shrinker purge path frees pages without going through
+		 * xe_bo_move(), so drop any stale defrag list entry here. Ghost
+		 * bos are never tracked, so guard for real xe bos.
+		 */
+		if (xe_bo_is_xe_bo(bo))
+			xe_bo_defrag_remove(ttm_to_xe_bo(bo));
 	}
 
 	return lret;
@@ -1909,6 +2015,8 @@ static void xe_ttm_bo_destroy(struct ttm_buffer_object *ttm_bo)
 		list_del(&bo->vram_userfault_link);
 	mutex_unlock(&xe->mem_access.vram_userfault.lock);
 
+	__xe_bo_defrag_remove(bo);
+
 	kfree(bo);
 }
 
@@ -2414,6 +2522,7 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,
 	bo->ttm.base.funcs = &xe_gem_object_funcs;
 	bo->ttm.priority = XE_BO_PRIORITY_NORMAL;
 	INIT_LIST_HEAD(&bo->pinned_link);
+	INIT_LIST_HEAD(&bo->defrag_link);
 #ifdef CONFIG_PROC_FS
 	INIT_LIST_HEAD(&bo->client_link);
 #endif
@@ -3020,6 +3129,8 @@ int xe_bo_pin_external(struct xe_bo *bo, bool in_place, struct drm_exec *exec)
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_subtract(xe, bo->ttm.ttm);
 
+	xe_bo_defrag_remove(bo);
+
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
 	 * we do not need this.
@@ -3078,6 +3189,8 @@ int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec)
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_subtract(xe, bo->ttm.ttm);
 
+	xe_bo_defrag_remove(bo);
+
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
 	 * we do not need this.
@@ -3113,6 +3226,7 @@ void xe_bo_unpin_external(struct xe_bo *bo)
 	ttm_bo_unpin(&bo->ttm);
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_add(xe, bo->ttm.ttm);
+	xe_bo_defrag_update(bo);
 
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
@@ -3145,6 +3259,7 @@ void xe_bo_unpin(struct xe_bo *bo)
 	ttm_bo_unpin(&bo->ttm);
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_add(xe, bo->ttm.ttm);
+	xe_bo_defrag_update(bo);
 }
 
 /**
drivers/gpu/drm/xe/xe_bo.h +1 -0
index 6340317f7d2e..4186cb51af87 100644
@@ -224,6 +224,7 @@ int xe_bo_pin_external(struct xe_bo *bo, bool in_place, struct drm_exec *exec);
 int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec);
 void xe_bo_unpin_external(struct xe_bo *bo);
 void xe_bo_unpin(struct xe_bo *bo);
+void xe_bo_defrag_init_early(struct xe_device *xe);
 int xe_bo_validate(struct xe_bo *bo, struct xe_vm *vm, bool allow_res_evict,
 		   struct drm_exec *exec);
 
drivers/gpu/drm/xe/xe_bo_types.h +6 -0
index fcc63ae3f455..9eea9ab380e5 100644
@@ -54,6 +54,12 @@ struct xe_bo {
 	struct ttm_bo_kmap_obj kmap;
 	/** @pinned_link: link to present / evicted list of pinned BO */
 	struct list_head pinned_link;
+	/**
+	 * @defrag_link: link into @xe_device.mem.defrag.list for BOs whose
+	 * backing TT pages were allocated at a sub-optimal order. Protected by
+	 * @xe_device.mem.defrag.lock.
+	 */
+	struct list_head defrag_link;
 #ifdef CONFIG_PROC_FS
 	/**
 	 * @client: @xe_drm_client which created the bo
drivers/gpu/drm/xe/xe_device.c +2 -0
index 97cc2d185e45..bb93425a5c70 100644
@@ -607,6 +607,8 @@ int xe_device_init_early(struct xe_device *xe)
 	if (err)
 		return err;
 
+	xe_bo_defrag_init_early(xe);
+
 	xe->preempt_fence_wq = alloc_ordered_workqueue("xe-preempt-fence-wq",
 						       WQ_MEM_RECLAIM);
 	xe->ordered_wq = alloc_ordered_workqueue("xe-ordered-wq", 0);
drivers/gpu/drm/xe/xe_device_types.h +19 -0
index 95e7cea91cbf..9e6244d8f41b 100644
@@ -296,6 +296,25 @@ struct xe_device {
 		 */
 		atomic_long_t dma_mapped_pages_svm[NR_PAGE_ORDERS];
 #endif
+		/**
+		 * @mem.defrag: Tracking of BOs whose backing TT pages were
+		 * allocated at a sub-optimal (smaller than beneficial) order.
+		 *
+		 * Such BOs are candidates for a future defragmentation pass that
+		 * tries to reallocate their pages at the device's beneficial
+		 * order. Only ttm_bo_type_device BOs are tracked.
+		 */
+		struct {
+			/** @mem.defrag.lock: Protects @mem.defrag.list. */
+			spinlock_t lock;
+			/** @mem.defrag.list: List of struct xe_bo defrag_link. */
+			struct list_head list;
+			/**
+			 * @mem.defrag.count: Number of BOs currently on
+			 * @mem.defrag.list.
+			 */
+			atomic_t count;
+		} defrag;
 	} mem;
 
 	/** @sriov: device level virtualization data */
Signature · 1 line
2.34.1
#52 lore raw

Re: [PATCH v3 15/33] drm/xe: Track BOs backed at a sub-optimal page order

Quoted 120 lines · Matthew Brost

Add per-device tracking of BOs whose backing TT pages were allocated below the pool's beneficial order, as reported by the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tt page flag (read via ttm_tt_is_beneficial_order_failed()). These BOs are future candidates for a defragmentation pass that reallocates their pages at the beneficial order.

Add a mem.defrag sub-struct (lock, list, atomic count) to struct xe_device and a defrag_link to struct xe_bo, initialised at BO create and device init. Membership is updated from xe_bo_move() via xe_bo_defrag_update(): a BO is added when it is a populated, needed, non-pinned ttm_bo_type_device object flagged TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED, and removed otherwise. BOs are also removed on pin, on purge, moved to dontneed, and at destroy.

Only ttm_bo_type_device BOs are tracked.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---

drivers/gpu/drm/xe/xe_bo.c           | 115 +++++++++++++++++++++++++++
drivers/gpu/drm/xe/xe_bo.h           |   1 +
drivers/gpu/drm/xe/xe_bo_types.h     |   6 ++
drivers/gpu/drm/xe/xe_device.c       |   2 +
drivers/gpu/drm/xe/xe_device_types.h |  19 +++++
5 files changed, 143 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +55 -0
index 4ee48cdc02ac..ff324caab6da 100644
@@ -926,6 +926,9 @@ static void xe_bo_set_purgeable_shrinker(struct xe_bo *bo,
 	}
 }
 
+static void xe_bo_defrag_update(struct xe_bo *bo);
+static void xe_bo_defrag_remove(struct xe_bo *bo);
+
 /**
 * xe_bo_set_purgeable_state() - Set BO purgeable state with validation
 * @bo: Buffer object
@@ -954,6 +957,7 @@ void xe_bo_set_purgeable_state(struct xe_bo *bo,
 
 	bo->purgeable.state = new_state;
 	xe_bo_set_purgeable_shrinker(bo, new_state);
+	xe_bo_defrag_update(bo);
 }
 
 /**
@@ -1003,9 +1007,100 @@ static int xe_ttm_bo_purge(struct ttm_buffer_object *ttm_bo, struct ttm_operatio
 	/* Commit the state transition only once invalidation was queued */
 	xe_bo_set_purgeable_state(bo, XE_MADV_PURGEABLE_PURGED);
 
+	/*
+	 * The backing store is gone, so the BO no longer needs defrag. Drop it
+	 * from the defrag list here so the count stays accurate even for purge
+	 * paths that don't pass through xe_bo_move() (e.g. swap_notify).
+	 */
+	xe_bo_defrag_remove(bo);
+
 	return 0;
 }
 
+/**
+ * xe_bo_defrag_init_early() - Initialize the device defrag BO tracking
+ * @xe: The xe device
+ *
+ * Initialize the list, lock and count used to track BOs whose backing TT
+ * pages were allocated at a sub-optimal order.
+ */
+void xe_bo_defrag_init_early(struct xe_device *xe)
+{
+	spin_lock_init(&xe->mem.defrag.lock);
+	INIT_LIST_HEAD(&xe->mem.defrag.list);
+	atomic_set(&xe->mem.defrag.count, 0);
+}
+
+static bool xe_bo_needs_defrag(struct xe_bo *bo)
+{
+	struct ttm_buffer_object *ttm_bo = &bo->ttm;
+	struct ttm_tt *tt = ttm_bo->ttm;
+
+	xe_bo_assert_held(bo);
+
+	return ttm_bo->type == ttm_bo_type_device && tt &&
+		!xe_bo_madv_is_dontneed(bo) &&
+		ttm_tt_is_populated(tt) &&
+		ttm_tt_is_beneficial_order_failed(tt) &&
+		ttm_bo->resource && ttm_bo->resource->mem_type == XE_PL_TT &&
+		!xe_bo_is_pinned(bo);
+}
+
+static void xe_bo_defrag_add(struct xe_bo *bo)
+{
+	struct xe_device *xe = xe_bo_device(bo);
+
+	xe_bo_assert_held(bo);
+	xe_assert(xe, xe_bo_needs_defrag(bo));
+
+	scoped_guard(spinlock, &xe->mem.defrag.lock) {
+		if (list_empty(&bo->defrag_link)) {
+			list_add_tail(&bo->defrag_link, &xe->mem.defrag.list);
+			atomic_inc(&xe->mem.defrag.count);
+		}
+	}
+}
+

drivers/gpu/drm/xe/xe_bo.c static int xe_ttm_bo_purge(struct ttm_buffer_object *ttm_bo, struct ttm_operatio · line 1073

Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +10 -0
+static void __xe_bo_defrag_remove(struct xe_bo *bo)
+{
+	struct xe_device *xe = xe_bo_device(bo);
+
+	guard(spinlock)(&xe->mem.defrag.lock);
+	if (!list_empty(&bo->defrag_link)) {
+		list_del_init(&bo->defrag_link);
+		atomic_dec(&xe->mem.defrag.count);
+	}
+}
Assisted-by LLM

NIT...probably worth adding a comment saying this version can be called without holding bo dma-resv lock?

Quoted 102 lines
drivers/gpu/drm/xe/xe_bo.c +49 -0
+
+static void xe_bo_defrag_remove(struct xe_bo *bo)
+{
+	xe_bo_assert_held(bo);
+
+	if (list_empty(&bo->defrag_link))
+		return;
+
+	__xe_bo_defrag_remove(bo);
+}
+
+/**
+ * xe_bo_defrag_update() - Update defrag list membership for a BO
+ * @bo: The buffer object
+ *
+ * Add @bo to the device defrag list when it is a ttm_bo_type_device BO resident
+ * in XE_PL_TT with a populated TT whose pages were allocated at a sub-optimal
+ * order (ttm_tt_is_beneficial_order_failed()) and it isn't pinned. Otherwise
+ * ensure it is removed from the list.
+ */
+static void xe_bo_defrag_update(struct xe_bo *bo)
+{
+	xe_bo_assert_held(bo);
+
+	if (xe_bo_needs_defrag(bo))
+		xe_bo_defrag_add(bo);
+	else
+		xe_bo_defrag_remove(bo);
+}
+
 static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		      struct ttm_operation_ctx *ctx,
 		      struct ttm_resource *new_mem,
@@ -1231,6 +1326,8 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		xe_tt_unmap_sg(xe, ttm_bo->ttm);
 	}
 
+	xe_bo_defrag_update(bo);
+
 	return ret;
 }
 
@@ -1267,6 +1364,15 @@ static long xe_bo_shrink_purge(struct ttm_operation_ctx *ctx,
 	if (lret > 0) {
 		xe_ttm_tt_account_subtract(xe, bo->ttm);
 		update_global_total_pages(bo->bdev, -(long)tt->num_pages);
+
+		/*
+		 * The pages are gone, so a tracked BO no longer needs defrag.
+		 * This shrinker purge path frees pages without going through
+		 * xe_bo_move(), so drop any stale defrag list entry here. Ghost
+		 * bos are never tracked, so guard for real xe bos.
+		 */
+		if (xe_bo_is_xe_bo(bo))
+			xe_bo_defrag_remove(ttm_to_xe_bo(bo));
 	}
 
 	return lret;
@@ -1909,6 +2015,8 @@ static void xe_ttm_bo_destroy(struct ttm_buffer_object *ttm_bo)
 		list_del(&bo->vram_userfault_link);
 	mutex_unlock(&xe->mem_access.vram_userfault.lock);
 
+	__xe_bo_defrag_remove(bo);
+
 	kfree(bo);
 }
 
@@ -2414,6 +2522,7 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,
 	bo->ttm.base.funcs = &xe_gem_object_funcs;
 	bo->ttm.priority = XE_BO_PRIORITY_NORMAL;
 	INIT_LIST_HEAD(&bo->pinned_link);
+	INIT_LIST_HEAD(&bo->defrag_link);
 #ifdef CONFIG_PROC_FS
 	INIT_LIST_HEAD(&bo->client_link);
 #endif
@@ -3020,6 +3129,8 @@ int xe_bo_pin_external(struct xe_bo *bo, bool in_place, struct drm_exec *exec)
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_subtract(xe, bo->ttm.ttm);
 
+	xe_bo_defrag_remove(bo);
+
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
 	 * we do not need this.
@@ -3078,6 +3189,8 @@ int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec)
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_subtract(xe, bo->ttm.ttm);
 
+	xe_bo_defrag_remove(bo);
+
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
 	 * we do not need this.
@@ -3113,6 +3226,7 @@ void xe_bo_unpin_external(struct xe_bo *bo)
 	ttm_bo_unpin(&bo->ttm);
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_add(xe, bo->ttm.ttm);
+	xe_bo_defrag_update(bo);
 
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking

drivers/gpu/drm/xe/xe_bo.c void xe_bo_unpin(struct xe_bo *bo) · line 3263

Quoted 5 lines
drivers/gpu/drm/xe/xe_bo.c +1 -0
@@ -3145,6 +3259,7 @@ void xe_bo_unpin(struct xe_bo *bo)
 	ttm_bo_unpin(&bo->ttm);
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_add(xe, bo->ttm.ttm);
+	xe_bo_defrag_update(bo);
 }

Looks like list update needs to be called from 2 other places,

1. ttm_bo_populate() getting called from CPU fault path and from TTM fault never calls xe_bo_defrag_update().
2. In xe_bo_shrink() if ttm_bo_shrink() returns >0 value. This will inflate the defrag.count value.

Quoted 60 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
 /**
drivers/gpu/drm/xe/xe_bo.h +1 -0
index 6340317f7d2e..4186cb51af87 100644
@@ -224,6 +224,7 @@ int xe_bo_pin_external(struct xe_bo *bo, bool in_place, struct drm_exec *exec);
 int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec);
 void xe_bo_unpin_external(struct xe_bo *bo);
 void xe_bo_unpin(struct xe_bo *bo);
+void xe_bo_defrag_init_early(struct xe_device *xe);
 int xe_bo_validate(struct xe_bo *bo, struct xe_vm *vm, bool allow_res_evict,
 		   struct drm_exec *exec);
 
drivers/gpu/drm/xe/xe_bo_types.h +6 -0
index fcc63ae3f455..9eea9ab380e5 100644
@@ -54,6 +54,12 @@ struct xe_bo {
 	struct ttm_bo_kmap_obj kmap;
 	/** @pinned_link: link to present / evicted list of pinned BO */
 	struct list_head pinned_link;
+	/**
+	 * @defrag_link: link into @xe_device.mem.defrag.list for BOs whose
+	 * backing TT pages were allocated at a sub-optimal order. Protected by
+	 * @xe_device.mem.defrag.lock.
+	 */
+	struct list_head defrag_link;
 #ifdef CONFIG_PROC_FS
 	/**
 	 * @client: @xe_drm_client which created the bo
drivers/gpu/drm/xe/xe_device.c +2 -0
index 97cc2d185e45..bb93425a5c70 100644
@@ -607,6 +607,8 @@ int xe_device_init_early(struct xe_device *xe)
 	if (err)
 		return err;
 
+	xe_bo_defrag_init_early(xe);
+
 	xe->preempt_fence_wq = alloc_ordered_workqueue("xe-preempt-fence-wq",
 						       WQ_MEM_RECLAIM);
 	xe->ordered_wq = alloc_ordered_workqueue("xe-ordered-wq", 0);
drivers/gpu/drm/xe/xe_device_types.h +8 -0
index 95e7cea91cbf..9e6244d8f41b 100644
@@ -296,6 +296,25 @@ struct xe_device {
 		 */
 		atomic_long_t dma_mapped_pages_svm[NR_PAGE_ORDERS];
 #endif
+		/**
+		 * @mem.defrag: Tracking of BOs whose backing TT pages were
+		 * allocated at a sub-optimal (smaller than beneficial) order.
+		 *
+		 * Such BOs are candidates for a future defragmentation pass that
+		 * tries to reallocate their pages at the device's beneficial
+		 * order. Only ttm_bo_type_device BOs are tracked.
+		 */

drivers/gpu/drm/xe/xe_device_types.h struct xe_device { · line 316

Quoted 10 lines
drivers/gpu/drm/xe/xe_device_types.h +10 -0
+		struct {
+			/** @mem.defrag.lock: Protects @mem.defrag.list. */
+			spinlock_t lock;
+			/** @mem.defrag.list: List of struct xe_bo defrag_link. */
+			struct list_head list;
+			/**
+			 * @mem.defrag.count: Number of BOs currently on
+			 * @mem.defrag.list.
+			 */
+			atomic_t count;

NIT...Not sure if we need it to be atomic. Count is only used in patch 16 and usage is approximation anyhow (doesn't guarantee anything as list lenght can be changing).

Niranjana

Quoted 7 lines
drivers/gpu/drm/xe/xe_device_types.h +1 -0
+		} defrag;
 	} mem;
 
 	/** @sriov: device level virtualization data */
Signature · 1 line
2.34.1
author maintainer
#53 lore raw

Re: [PATCH v3 15/33] drm/xe: Track BOs backed at a sub-optimal page order

Quoted 125 lines · Niranjana Vishwanathapura
Quoted 124 lines · Matthew Brost

Add per-device tracking of BOs whose backing TT pages were allocated below the pool's beneficial order, as reported by the TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED tt page flag (read via ttm_tt_is_beneficial_order_failed()). These BOs are future candidates for a defragmentation pass that reallocates their pages at the beneficial order.

Add a mem.defrag sub-struct (lock, list, atomic count) to struct xe_device and a defrag_link to struct xe_bo, initialised at BO create and device init. Membership is updated from xe_bo_move() via xe_bo_defrag_update(): a BO is added when it is a populated, needed, non-pinned ttm_bo_type_device object flagged TTM_TT_FLAG_BENEFICIAL_ORDER_FAILED, and removed otherwise. BOs are also removed on pin, on purge, moved to dontneed, and at destroy.

Only ttm_bo_type_device BOs are tracked.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---

drivers/gpu/drm/xe/xe_bo.c           | 115 +++++++++++++++++++++++++++
drivers/gpu/drm/xe/xe_bo.h           |   1 +
drivers/gpu/drm/xe/xe_bo_types.h     |   6 ++
drivers/gpu/drm/xe/xe_device.c       |   2 +
drivers/gpu/drm/xe/xe_device_types.h |  19 +++++
5 files changed, 143 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +59 -0
index 4ee48cdc02ac..ff324caab6da 100644
@@ -926,6 +926,9 @@ static void xe_bo_set_purgeable_shrinker(struct xe_bo *bo,
 	}
 }
 
+static void xe_bo_defrag_update(struct xe_bo *bo);
+static void xe_bo_defrag_remove(struct xe_bo *bo);
+
 /**
 * xe_bo_set_purgeable_state() - Set BO purgeable state with validation
 * @bo: Buffer object
@@ -954,6 +957,7 @@ void xe_bo_set_purgeable_state(struct xe_bo *bo,
 
 	bo->purgeable.state = new_state;
 	xe_bo_set_purgeable_shrinker(bo, new_state);
+	xe_bo_defrag_update(bo);
 }
 
 /**
@@ -1003,9 +1007,100 @@ static int xe_ttm_bo_purge(struct ttm_buffer_object *ttm_bo, struct ttm_operatio
 	/* Commit the state transition only once invalidation was queued */
 	xe_bo_set_purgeable_state(bo, XE_MADV_PURGEABLE_PURGED);
 
+	/*
+	 * The backing store is gone, so the BO no longer needs defrag. Drop it
+	 * from the defrag list here so the count stays accurate even for purge
+	 * paths that don't pass through xe_bo_move() (e.g. swap_notify).
+	 */
+	xe_bo_defrag_remove(bo);
+
 	return 0;
 }
 
+/**
+ * xe_bo_defrag_init_early() - Initialize the device defrag BO tracking
+ * @xe: The xe device
+ *
+ * Initialize the list, lock and count used to track BOs whose backing TT
+ * pages were allocated at a sub-optimal order.
+ */
+void xe_bo_defrag_init_early(struct xe_device *xe)
+{
+	spin_lock_init(&xe->mem.defrag.lock);
+	INIT_LIST_HEAD(&xe->mem.defrag.list);
+	atomic_set(&xe->mem.defrag.count, 0);
+}
+
+static bool xe_bo_needs_defrag(struct xe_bo *bo)
+{
+	struct ttm_buffer_object *ttm_bo = &bo->ttm;
+	struct ttm_tt *tt = ttm_bo->ttm;
+
+	xe_bo_assert_held(bo);
+
+	return ttm_bo->type == ttm_bo_type_device && tt &&
+		!xe_bo_madv_is_dontneed(bo) &&
+		ttm_tt_is_populated(tt) &&
+		ttm_tt_is_beneficial_order_failed(tt) &&
+		ttm_bo->resource && ttm_bo->resource->mem_type == XE_PL_TT &&
+		!xe_bo_is_pinned(bo);
+}
+
+static void xe_bo_defrag_add(struct xe_bo *bo)
+{
+	struct xe_device *xe = xe_bo_device(bo);
+
+	xe_bo_assert_held(bo);
+	xe_assert(xe, xe_bo_needs_defrag(bo));
+
+	scoped_guard(spinlock, &xe->mem.defrag.lock) {
+		if (list_empty(&bo->defrag_link)) {
+			list_add_tail(&bo->defrag_link, &xe->mem.defrag.list);
+			atomic_inc(&xe->mem.defrag.count);
+		}
+	}
+}
+
+static void __xe_bo_defrag_remove(struct xe_bo *bo)
+{
+	struct xe_device *xe = xe_bo_device(bo);
+
Quoted 9 lines

drivers/gpu/drm/xe/xe_bo.c static int xe_ttm_bo_purge(struct ttm_buffer_object *ttm_bo, struct ttm_operatio · line 1073

Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +6 -0
+	guard(spinlock)(&xe->mem.defrag.lock);
+	if (!list_empty(&bo->defrag_link)) {
+		list_del_init(&bo->defrag_link);
+		atomic_dec(&xe->mem.defrag.count);
+	}
+}
Assisted-by LLM

Yes.

Quoted 3 lines

NIT...probably worth adding a comment saying this version can be called without holding bo dma-resv lock?

It should be self documenting with lockdep but can add this.

Quoted 110 lines
Quoted 102 lines
drivers/gpu/drm/xe/xe_bo.c +49 -0
+
+static void xe_bo_defrag_remove(struct xe_bo *bo)
+{
+	xe_bo_assert_held(bo);
+
+	if (list_empty(&bo->defrag_link))
+		return;
+
+	__xe_bo_defrag_remove(bo);
+}
+
+/**
+ * xe_bo_defrag_update() - Update defrag list membership for a BO
+ * @bo: The buffer object
+ *
+ * Add @bo to the device defrag list when it is a ttm_bo_type_device BO resident
+ * in XE_PL_TT with a populated TT whose pages were allocated at a sub-optimal
+ * order (ttm_tt_is_beneficial_order_failed()) and it isn't pinned. Otherwise
+ * ensure it is removed from the list.
+ */
+static void xe_bo_defrag_update(struct xe_bo *bo)
+{
+	xe_bo_assert_held(bo);
+
+	if (xe_bo_needs_defrag(bo))
+		xe_bo_defrag_add(bo);
+	else
+		xe_bo_defrag_remove(bo);
+}
+
 static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		      struct ttm_operation_ctx *ctx,
 		      struct ttm_resource *new_mem,
@@ -1231,6 +1326,8 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict,
 		xe_tt_unmap_sg(xe, ttm_bo->ttm);
 	}
 
+	xe_bo_defrag_update(bo);
+
 	return ret;
 }
 
@@ -1267,6 +1364,15 @@ static long xe_bo_shrink_purge(struct ttm_operation_ctx *ctx,
 	if (lret > 0) {
 		xe_ttm_tt_account_subtract(xe, bo->ttm);
 		update_global_total_pages(bo->bdev, -(long)tt->num_pages);
+
+		/*
+		 * The pages are gone, so a tracked BO no longer needs defrag.
+		 * This shrinker purge path frees pages without going through
+		 * xe_bo_move(), so drop any stale defrag list entry here. Ghost
+		 * bos are never tracked, so guard for real xe bos.
+		 */
+		if (xe_bo_is_xe_bo(bo))
+			xe_bo_defrag_remove(ttm_to_xe_bo(bo));
 	}
 
 	return lret;
@@ -1909,6 +2015,8 @@ static void xe_ttm_bo_destroy(struct ttm_buffer_object *ttm_bo)
 		list_del(&bo->vram_userfault_link);
 	mutex_unlock(&xe->mem_access.vram_userfault.lock);
 
+	__xe_bo_defrag_remove(bo);
+
 	kfree(bo);
 }
 
@@ -2414,6 +2522,7 @@ struct xe_bo *xe_bo_init_locked(struct xe_device *xe, struct xe_bo *bo,
 	bo->ttm.base.funcs = &xe_gem_object_funcs;
 	bo->ttm.priority = XE_BO_PRIORITY_NORMAL;
 	INIT_LIST_HEAD(&bo->pinned_link);
+	INIT_LIST_HEAD(&bo->defrag_link);
 #ifdef CONFIG_PROC_FS
 	INIT_LIST_HEAD(&bo->client_link);
 #endif
@@ -3020,6 +3129,8 @@ int xe_bo_pin_external(struct xe_bo *bo, bool in_place, struct drm_exec *exec)
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_subtract(xe, bo->ttm.ttm);
 
+	xe_bo_defrag_remove(bo);
+
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
 	 * we do not need this.
@@ -3078,6 +3189,8 @@ int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec)
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_subtract(xe, bo->ttm.ttm);
 
+	xe_bo_defrag_remove(bo);
+
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking
 	 * we do not need this.
@@ -3113,6 +3226,7 @@ void xe_bo_unpin_external(struct xe_bo *bo)
 	ttm_bo_unpin(&bo->ttm);
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_add(xe, bo->ttm.ttm);
+	xe_bo_defrag_update(bo);
 
 	/*
 	 * FIXME: If we always use the reserve / unreserve functions for locking

drivers/gpu/drm/xe/xe_bo.c void xe_bo_unpin(struct xe_bo *bo) · line 3263

Quoted 5 lines
drivers/gpu/drm/xe/xe_bo.c +1 -0
@@ -3145,6 +3259,7 @@ void xe_bo_unpin(struct xe_bo *bo)
 	ttm_bo_unpin(&bo->ttm);
 	if (bo->ttm.ttm && ttm_tt_is_populated(bo->ttm.ttm))
 		xe_ttm_tt_account_add(xe, bo->ttm.ttm);
+	xe_bo_defrag_update(bo);
 }

Looks like list update needs to be called from 2 other places,

Quoted 2 lines

1. ttm_bo_populate() getting called from CPU fault path and from TTM fault never calls xe_bo_defrag_update().

Yes. We also need to strip out reclaim flags in CPU fault path as this a foreground op.

Quoted 3 lines

2. In xe_bo_shrink() if ttm_bo_shrink() returns >0 value. This will inflate the defrag.count value.

This routes through ttm_bo_validate which ends up in xe_bo_move so I think this case is covered.

Quoted 66 lines
Quoted 66 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
 /**
drivers/gpu/drm/xe/xe_bo.h +1 -0
index 6340317f7d2e..4186cb51af87 100644
@@ -224,6 +224,7 @@ int xe_bo_pin_external(struct xe_bo *bo, bool in_place, struct drm_exec *exec);
 int xe_bo_pin(struct xe_bo *bo, struct drm_exec *exec);
 void xe_bo_unpin_external(struct xe_bo *bo);
 void xe_bo_unpin(struct xe_bo *bo);
+void xe_bo_defrag_init_early(struct xe_device *xe);
 int xe_bo_validate(struct xe_bo *bo, struct xe_vm *vm, bool allow_res_evict,
 		   struct drm_exec *exec);
 
drivers/gpu/drm/xe/xe_bo_types.h +6 -0
index fcc63ae3f455..9eea9ab380e5 100644
@@ -54,6 +54,12 @@ struct xe_bo {
 	struct ttm_bo_kmap_obj kmap;
 	/** @pinned_link: link to present / evicted list of pinned BO */
 	struct list_head pinned_link;
+	/**
+	 * @defrag_link: link into @xe_device.mem.defrag.list for BOs whose
+	 * backing TT pages were allocated at a sub-optimal order. Protected by
+	 * @xe_device.mem.defrag.lock.
+	 */
+	struct list_head defrag_link;
 #ifdef CONFIG_PROC_FS
 	/**
 	 * @client: @xe_drm_client which created the bo
drivers/gpu/drm/xe/xe_device.c +2 -0
index 97cc2d185e45..bb93425a5c70 100644
@@ -607,6 +607,8 @@ int xe_device_init_early(struct xe_device *xe)
 	if (err)
 		return err;
 
+	xe_bo_defrag_init_early(xe);
+
 	xe->preempt_fence_wq = alloc_ordered_workqueue("xe-preempt-fence-wq",
 						       WQ_MEM_RECLAIM);
 	xe->ordered_wq = alloc_ordered_workqueue("xe-ordered-wq", 0);
drivers/gpu/drm/xe/xe_device_types.h +14 -0
index 95e7cea91cbf..9e6244d8f41b 100644
@@ -296,6 +296,25 @@ struct xe_device {
 		 */
 		atomic_long_t dma_mapped_pages_svm[NR_PAGE_ORDERS];
 #endif
+		/**
+		 * @mem.defrag: Tracking of BOs whose backing TT pages were
+		 * allocated at a sub-optimal (smaller than beneficial) order.
+		 *
+		 * Such BOs are candidates for a future defragmentation pass that
+		 * tries to reallocate their pages at the device's beneficial
+		 * order. Only ttm_bo_type_device BOs are tracked.
+		 */
+		struct {
+			/** @mem.defrag.lock: Protects @mem.defrag.list. */
+			spinlock_t lock;
+			/** @mem.defrag.list: List of struct xe_bo defrag_link. */
+			struct list_head list;
+			/**
Quoted 8 lines

drivers/gpu/drm/xe/xe_device_types.h struct xe_device { · line 316

Quoted 4 lines
drivers/gpu/drm/xe/xe_device_types.h +4 -0
+			 * @mem.defrag.count: Number of BOs currently on
+			 * @mem.defrag.list.
+			 */
+			atomic_t count;

NIT...Not sure if we need it to be atomic. Count is only used in patch 16 and usage is approximation anyhow (doesn't guarantee anything as list lenght can be changing).

Right - incr / decr are protected by the spin lock along with the list.

I think count might actually be able to be dropped as target has slighly moved - foreground just never does reclaim, thus XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD / count isn't needed.

Matt

Quoted 10 lines

Niranjana

Quoted 7 lines
drivers/gpu/drm/xe/xe_device_types.h +1 -0
+		} defrag;
 	} mem;
 
 	/** @sriov: device level virtualization data */
Signature · 1 line
2.34.1
author maintainer
#54 lore raw

PATCH v3 16/33 drm/xe: Back off beneficial-order reclaim under defrag pressure

In reply to Matthew Brost

Once a meaningful number of BOs are already tracked as being backed at a sub-optimal order, continuing to push hard for beneficial-order pages during populate mostly wastes time in reclaim/compaction.

In xe_ttm_tt_populate(), set ctx->beneficial_reclaim_backoff when the device defrag count reaches XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD (2) so the TTM pool backs off and allocations make forward progress. The deferred defrag pass is responsible for upgrading these objects to beneficial order later.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_bo.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +14 -0
index ff324caab6da..02395968d613 100644
@@ -41,6 +41,14 @@
 #include "xe_vm.h"
 #include "xe_vram_types.h"
 
+/*
+ * Once this many BOs are tracked on the device defrag list (i.e. were backed
+ * with a sub-optimal page order), request that the TTM pool backs off from
+ * aggressive reclaim at the beneficial order during populate, so that
+ * allocations make forward progress instead of stalling.
+ */
+#define XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD	2
+
 const char *const xe_mem_type_to_name[TTM_NUM_MEM_TYPES]  = {
 	[XE_PL_SYSTEM] = "system",
 	[XE_PL_TT] = "gtt",
@@ -598,6 +606,12 @@ static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt,
 	if (ttm_tt_is_backed_up(tt) && !xe_tt->purgeable) {
 		err = ttm_tt_restore(ttm_dev, tt, ctx);
 	} else {
+		struct xe_device *xe = ttm_to_xe_device(ttm_dev);
+
+		if (atomic_read(&xe->mem.defrag.count) >=
+		    XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD)
+			ctx->beneficial_reclaim_backoff = true;
+
 		ttm_tt_clear_backed_up(tt);
 		err = ttm_pool_alloc(&ttm_dev->pool, tt, ctx);
 	}
Signature · 1 line
2.34.1
#55 lore raw

Re: [PATCH v3 16/33] drm/xe: Back off beneficial-order reclaim under defrag pressure

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The beneficial-order reclaim backoff optimization is inadvertently skipped when restoring…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The beneficial-order reclaim backoff optimization is inadvertently skipped when restoring backed-up buffer objects (swap-ins). --

--- Patch [16]: [PATCH v3 16/33] drm/xe: Back off beneficial-order reclaim under defrag pressure --- commit 2099c6d8da1f6677d9db38b2ffe92ba3eb96f8f0
Author: Matthew Brost <[email protected]>

drm/xe: Back off beneficial-order reclaim under defrag pressure

Once a meaningful number of BOs are tracked as being backed at a sub-optimal order, this commit configures the TTM pool to back off from aggressive reclaim during populate. This allows allocations to make forward progress, leaving upgrades to beneficial orders for a deferred defrag pass.

Quoted 3 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0

[ ... ]

drivers/gpu/drm/xe/xe_bo.c static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt, · line 614

Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +6 -0
@@ -598,6 +606,12 @@ static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt,
 	if (ttm_tt_is_backed_up(tt) && !xe_tt->purgeable) {
 		err = ttm_tt_restore(ttm_dev, tt, ctx);
 	} else {
+		struct xe_device *xe = ttm_to_xe_device(ttm_dev);
+
+		if (atomic_read(&xe->mem.defrag.count) >=
+		    XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD)
+			ctx->beneficial_reclaim_backoff = true;
+

[Severity: Medium]
Does this unintentionally skip the backoff optimization for swap-ins?

Because the beneficial_reclaim_backoff flag is only set inside the else branch, any swapped-out buffer object being restored via ttm_tt_restore() under fragmentation pressure will miss this optimization.

Without the flag, ttm_tt_restore() will continue pushing hard for beneficial-order pages, potentially causing the exact reclaim and compaction stalls this commit intended to avoid.

Would it make sense to move the device defrag count check and flag evaluation above the ttm_tt_is_backed_up() check so both paths benefit from the optimization?

Quoted 3 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
 		ttm_tt_clear_backed_up(tt);
 		err = ttm_pool_alloc(&ttm_dev->pool, tt, ctx);
 	}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
#56 lore raw

Re: [PATCH v3 16/33] drm/xe: Back off beneficial-order reclaim under defrag pressure

In reply to Matthew Brost
Quoted 49 lines · Matthew Brost

Once a meaningful number of BOs are already tracked as being backed at a sub-optimal order, continuing to push hard for beneficial-order pages during populate mostly wastes time in reclaim/compaction.

In xe_ttm_tt_populate(), set ctx->beneficial_reclaim_backoff when the device defrag count reaches XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD (2) so the TTM pool backs off and allocations make forward progress. The deferred defrag pass is responsible for upgrading these objects to beneficial order later.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---
drivers/gpu/drm/xe/xe_bo.c | 14 ++++++++++++++

1 file changed, 14 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +8 -0
index ff324caab6da..02395968d613 100644
@@ -41,6 +41,14 @@
 #include "xe_vm.h"
 #include "xe_vram_types.h"
 
+/*
+ * Once this many BOs are tracked on the device defrag list (i.e. were backed
+ * with a sub-optimal page order), request that the TTM pool backs off from
+ * aggressive reclaim at the beneficial order during populate, so that
+ * allocations make forward progress instead of stalling.
+ */
+#define XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD	2
+
 const char *const xe_mem_type_to_name[TTM_NUM_MEM_TYPES]  = {
 	[XE_PL_SYSTEM] = "system",
 	[XE_PL_TT] = "gtt",

drivers/gpu/drm/xe/xe_bo.c static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt, · line 607

Quoted 2 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -598,6 +606,12 @@ static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt,
 	if (ttm_tt_is_backed_up(tt) && !xe_tt->purgeable) {
 		err = ttm_tt_restore(ttm_dev, tt, ctx);
Assisted-by LLM

Don't we need to to backoff in restore path also? If it is intentional, probably add a comment?

drivers/gpu/drm/xe/xe_bo.c static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt, · line 614

Quoted 7 lines
drivers/gpu/drm/xe/xe_bo.c +6 -0
 	} else {
+		struct xe_device *xe = ttm_to_xe_device(ttm_dev);
+
+		if (atomic_read(&xe->mem.defrag.count) >=
+		    XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD)
+			ctx->beneficial_reclaim_backoff = true;
+

- beneficial_reclaim_backoff is not cleared if count comes below threshold. If the same ctx is used for multiple BOs, they all will backoff even when count has moved below threshold.

- Looks like self-reinforcing feedback loop causes spike in backoff.

count ≥ 2 → backoff on → new populates land sub-optimal → BENEFICIAL_ORDER_FAILED → added to defrag list → count stays ≥ 2 → backoff stays on → ...

Is it intentional? If so, may be a comment explaining it would be helpful?

- Will there be any value is heuristically adjusting threshold instead of using static value (2)?

Niranjana

Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
 		ttm_tt_clear_backed_up(tt);
 		err = ttm_pool_alloc(&ttm_dev->pool, tt, ctx);
 	}
Signature · 1 line
2.34.1
author maintainer
#57 lore raw

Re: [PATCH v3 16/33] drm/xe: Back off beneficial-order reclaim under defrag pressure

Quoted 55 lines · Niranjana Vishwanathapura
Quoted 49 lines · Matthew Brost

Once a meaningful number of BOs are already tracked as being backed at a sub-optimal order, continuing to push hard for beneficial-order pages during populate mostly wastes time in reclaim/compaction.

In xe_ttm_tt_populate(), set ctx->beneficial_reclaim_backoff when the device defrag count reaches XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD (2) so the TTM pool backs off and allocations make forward progress. The deferred defrag pass is responsible for upgrading these objects to beneficial order later.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8

---
drivers/gpu/drm/xe/xe_bo.c | 14 ++++++++++++++

1 file changed, 14 insertions(+)
drivers/gpu/drm/xe/xe_bo.c +8 -0
index ff324caab6da..02395968d613 100644
@@ -41,6 +41,14 @@
 #include "xe_vm.h"
 #include "xe_vram_types.h"
 
+/*
+ * Once this many BOs are tracked on the device defrag list (i.e. were backed
+ * with a sub-optimal page order), request that the TTM pool backs off from
+ * aggressive reclaim at the beneficial order during populate, so that
+ * allocations make forward progress instead of stalling.
+ */
+#define XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD	2
+
 const char *const xe_mem_type_to_name[TTM_NUM_MEM_TYPES]  = {
 	[XE_PL_SYSTEM] = "system",
 	[XE_PL_TT] = "gtt",

drivers/gpu/drm/xe/xe_bo.c static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt, · line 607

Quoted 2 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
@@ -598,6 +606,12 @@ static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt,
 	if (ttm_tt_is_backed_up(tt) && !xe_tt->purgeable) {
 		err = ttm_tt_restore(ttm_dev, tt, ctx);
Assisted-by LLM
Quoted 3 lines

Don't we need to to backoff in restore path also? If it is intentional, probably add a comment?

Yes, we do need backoff in the restore path. There a couple other places in this series that miss the backoff too (e.g., CPU fault) - the divide we want is anything the foreground no reclaim, and only reclaim in background outside of dma-resv (defragmenter).

Quoted 14 lines

drivers/gpu/drm/xe/xe_bo.c static int xe_ttm_tt_populate(struct ttm_device *ttm_dev, struct ttm_tt *tt, · line 614

Quoted 7 lines
drivers/gpu/drm/xe/xe_bo.c +6 -0
 	} else {
+		struct xe_device *xe = ttm_to_xe_device(ttm_dev);
+
+		if (atomic_read(&xe->mem.defrag.count) >=
+		    XE_BO_DEFRAG_RECLAIM_BACKOFF_THRESHOLD)
+			ctx->beneficial_reclaim_backoff = true;
+

- beneficial_reclaim_backoff is not cleared if count comes below threshold. If the same ctx is used for multiple BOs, they all will backoff even when count has moved below threshold.

- Looks like self-reinforcing feedback loop causes spike in backoff.

Quoted 8 lines

count ≥ 2 → backoff on → new populates land sub-optimal → BENEFICIAL_ORDER_FAILED → added to defrag list → count stays ≥ 2 → backoff stays on → ...

Is it intentional? If so, may be a comment explaining it would be helpful?

- Will there be any value is heuristically adjusting threshold instead of using static value (2)?

As mentiond above / last patch - I'm just going rip this out have clear divide between foreground / background setting the reclaim flags correctly.

Matt

Quoted 8 lines

Niranjana

Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +0 -0
 		ttm_tt_clear_backed_up(tt);
 		err = ttm_pool_alloc(&ttm_dev->pool, tt, ctx);
 	}
Signature · 1 line
2.34.1
author maintainer
#58 lore raw

PATCH v3 17/33 drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies

In reply to Matthew Brost

Add xe_migrate_copy_defrag(), a copy helper for the upcoming page defragmentation path where a single BO has its backing reallocated at a more beneficial page order and its contents must be relocated from the old pages to the new ones entirely on the GPU.

The contents are relocated in up to two passes:

- Data pass (always): copy the data pages verbatim. The pages are mapped without the compression PAT, so the (possibly compressed) bytes are copied as-is rather than being decompressed and recompressed, and no CCS aux copy is appended.

- CCS pass (need_ccs): copy the CCS aux (compression) state. Both the old and new data pages are mapped with the compression PAT so the copy engine accesses the aux state indirectly, issuing one indirect -> indirect XY_CTRL_SURF_COPY_BLT; the data blit is skipped.

The two passes are issued separately because they require opposite compression PAT settings on the data page mappings (cleared for the verbatim data copy, set for the indirect CCS access), which a single mapping cannot satisfy. Both passes run in order on the same migrate queue and are captured by the returned fence.

The source pages are passed explicitly via an sg_table argument (the old, still populated backing) since they no longer match the BO's current backing. The boolean arguments to __xe_migrate_copy() are folded into struct xe_migrate_copy_flags to keep the growing set of copy variants readable.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
 drivers/gpu/drm/xe/xe_migrate.c | 136 ++++++++++++++++++++++++++++----
 drivers/gpu/drm/xe/xe_migrate.h |   8 ++
 2 files changed, 128 insertions(+), 16 deletions(-)
drivers/gpu/drm/xe/xe_migrate.c +120 -16
index 92d5e81ceac2..305c6e1942fb 100644
@@ -858,13 +858,39 @@ static u32 xe_migrate_ccs_copy(struct xe_migrate *m,
 	return flush_flags;
 }
 
+/**
+ * struct xe_migrate_copy_flags - Flags for __xe_migrate_copy()
+ * @copy_only_ccs: Only copy the CCS aux state, aliasing the destination data
+ * mapping to the source (used when the data pages are shared between src/dst).
+ * @is_vram_resolve: Decompression (VRAM -> system) resolve; always uses the
+ * compression PAT index for the data copy.
+ * @defrag_ccs: Defrag move pass that migrates only the CCS aux state between
+ * two distinct page sets (old @src_sg pages -> new pages). The data blit is
+ * skipped (the data was copied by the @defrag_copy pass) and both the source
+ * and destination data pages are mapped with the compression PAT so the aux
+ * state is copied indirect -> indirect.
+ * @defrag_copy: Defrag move pass that copies the data pages between two
+ * distinct page sets (old @src_sg pages -> new pages) without touching the CCS
+ * aux state. The data pages are mapped without the compression PAT so the
+ * (possibly compressed) bytes are copied verbatim rather than being
+ * decompressed and recompressed, and no CCS aux copy is appended; the aux
+ * state is migrated separately by the @defrag_ccs pass. The two passes need
+ * opposite compression PAT settings, which is why they cannot be combined.
+ */
+struct xe_migrate_copy_flags {
+	u32 copy_only_ccs : 1;
+	u32 is_vram_resolve : 1;
+	u32 defrag_ccs : 1;
+	u32 defrag_copy : 1;
+};
+
 static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 					   struct xe_bo *src_bo,
 					   struct xe_bo *dst_bo,
 					   struct ttm_resource *src,
 					   struct ttm_resource *dst,
-					   bool copy_only_ccs,
-					   bool is_vram_resolve)
+					   struct sg_table *src_sg,
+					   struct xe_migrate_copy_flags flags)
 {
 	struct xe_gt *gt = m->tile->primary_gt;
 	struct xe_device *xe = gt_to_xe(gt);
@@ -881,20 +907,29 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 	bool src_is_vram = mem_type_is_vram(src->mem_type);
 	bool dst_is_vram = mem_type_is_vram(dst->mem_type);
 	bool type_device = src_bo->ttm.type == ttm_bo_type_device;
-	bool needs_ccs_emit = type_device && xe_migrate_needs_ccs_emit(xe);
+	bool needs_ccs_emit = type_device && xe_migrate_needs_ccs_emit(xe) &&
+		!flags.defrag_copy;
 	bool copy_ccs = xe_device_has_flat_ccs(xe) &&
-		xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo);
-	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram);
+		xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo) &&
+		!flags.defrag_copy;
+	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram) &&
+		!flags.defrag_ccs && !flags.defrag_copy;
 
 	/*
 	 * For decompression operation, always use the compression PAT index.
 	 * Otherwise, only use the compression PAT index for device memory
 	 * when copying from VRAM to system memory.
 	 */
-	bool use_comp_pat = is_vram_resolve || (type_device &&
+	bool use_comp_pat = flags.is_vram_resolve || (type_device &&
 			    xe_device_has_flat_ccs(xe) &&
 			    GRAPHICS_VER(xe) >= 20 && src_is_vram && !dst_is_vram);
 
+	/*
+	 * The defrag CCS pass migrates the aux state from the old pages
+	 * (@src_sg) to the new pages with both sides accessed indirectly, so
+	 * keep both src_is_pltt and dst_is_pltt set (no direct staging side).
+	 */
+
 	/* Copying CCS between two different BOs is not supported yet. */
 	if (XE_WARN_ON(copy_ccs && src_bo != dst_bo))
 		return ERR_PTR(-EINVAL);
@@ -903,7 +938,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		return ERR_PTR(-EINVAL);
 
 	if (!src_is_vram)
-		xe_res_first_sg(xe_bo_sg(src_bo), 0, size, &src_it);
+		xe_res_first_sg(src_sg ? src_sg : xe_bo_sg(src_bo), 0, size, &src_it);
 	else
 		xe_res_first(src, 0, size, &src_it);
 	if (!dst_is_vram)
@@ -912,7 +947,8 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		xe_res_first(dst, 0, size, &dst_it);
 
 	if (copy_system_ccs)
-		xe_res_first_sg(xe_bo_sg(src_bo), xe_bo_ccs_pages_start(src_bo),
+		xe_res_first_sg(src_sg ? src_sg : xe_bo_sg(src_bo),
+				xe_bo_ccs_pages_start(src_bo),
 				PAGE_ALIGN(xe_device_ccs_bytes(xe, size)),
 				&ccs_it);
 
@@ -942,7 +978,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		batch_size += pte_update_size(m, pte_flags, src, &src_it, &src_L0,
 					      &src_L0_ofs, &src_L0_pt, 0, 0,
 					      avail_pts);
-		if (copy_only_ccs) {
+		if (flags.copy_only_ccs) {
 			dst_L0_ofs = src_L0_ofs;
 		} else {
 			pte_flags = dst_is_vram ? PTE_UPDATE_FLAG_IS_VRAM : 0;
@@ -963,7 +999,8 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		}
 
 		/* Add copy commands size here */
-		batch_size += ((copy_only_ccs) ? 0 : emit_copy_cmd_len(xe)) +
+		batch_size += ((flags.copy_only_ccs || flags.defrag_ccs) ? 0 :
+			       emit_copy_cmd_len(xe)) +
 			((needs_ccs_emit ? EMIT_COPY_CCS_DW : 0));
 
 		bb = xe_bb_new(gt, batch_size, usm);
@@ -975,13 +1012,15 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		if (src_is_vram && xe_migrate_allow_identity(src_L0, &src_it))
 			xe_res_next(&src_it, src_L0);
 		else
-			emit_pte(m, bb, src_L0_pt, src_is_vram, copy_system_ccs || use_comp_pat,
+			emit_pte(m, bb, src_L0_pt, src_is_vram,
+				 copy_system_ccs || use_comp_pat || flags.defrag_ccs,
 				 &src_it, src_L0, src);
 
 		if (dst_is_vram && xe_migrate_allow_identity(src_L0, &dst_it))
 			xe_res_next(&dst_it, src_L0);
-		else if (!copy_only_ccs)
-			emit_pte(m, bb, dst_L0_pt, dst_is_vram, copy_system_ccs,
+		else if (!flags.copy_only_ccs)
+			emit_pte(m, bb, dst_L0_pt, dst_is_vram,
+				 copy_system_ccs || flags.defrag_ccs,
 				 &dst_it, src_L0, dst);
 
 		if (copy_system_ccs)
@@ -990,7 +1029,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		bb->cs[bb->len++] = MI_BATCH_BUFFER_END;
 		update_idx = bb->len;
 
-		if (!copy_only_ccs)
+		if (!flags.copy_only_ccs && !flags.defrag_ccs)
 			emit_copy(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0, XE_PAGE_SIZE);
 
 		if (needs_ccs_emit)
@@ -1078,7 +1117,69 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
 				  struct ttm_resource *dst,
 				  bool copy_only_ccs)
 {
-	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, copy_only_ccs, false);
+	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, NULL,
+				 (struct xe_migrate_copy_flags) {
+					 .copy_only_ccs = copy_only_ccs,
+				 });
+}
+
+/**
+ * xe_migrate_copy_defrag() - Migrate a BO's backing to a reallocated page set.
+ * @m: The migration context.
+ * @bo: The buffer object being defragmented. Both @src and @dst belong to @bo.
+ * @src: The source TTM resource (the old backing).
+ * @dst: The dst TTM resource (the freshly reallocated backing).
+ * @src_sg: Scatter-gather table of the source (old) backing pages.
+ * @need_ccs: Whether the BO has CCS aux state that must be migrated too.
+ *
+ * Used by the defrag path where a single BO has its backing reallocated at a
+ * more beneficial page order. The contents are relocated from the old pages
+ * (@src_sg) to the new pages (the BO's current backing) entirely on the GPU,
+ * in up to two passes:
+ *
+ *  - Data pass (always): copy the data pages verbatim. The pages are mapped
+ *    without the compression PAT, so the (possibly compressed) bytes are copied
+ *    as-is rather than being decompressed and recompressed, and no CCS aux copy
+ *    is appended.
+ *
+ *  - CCS pass (@need_ccs): copy the CCS aux (compression) state. Both the old
+ *    and new data pages are mapped with the compression PAT so the copy engine
+ *    accesses the aux state indirectly, issuing one indirect -> indirect
+ *    XY_CTRL_SURF_COPY_BLT; the data blit is skipped.
+ *
+ * The two passes are issued separately because they require opposite
+ * compression PAT settings on the data page mappings (cleared for the verbatim
+ * data copy, set for the indirect CCS access), which a single mapping cannot
+ * satisfy. Both passes run in order on the same migrate queue.
+ *
+ * Return: Pointer to a dma_fence representing the last copy batch, or an error
+ * pointer on failure.
+ */
+struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
+					 struct xe_bo *bo,
+					 struct ttm_resource *src,
+					 struct ttm_resource *dst,
+					 struct sg_table *src_sg,
+					 bool need_ccs)
+{
+	struct dma_fence *fence2, *fence =
+		__xe_migrate_copy(m, bo, bo, src, dst, src_sg,
+				  (struct xe_migrate_copy_flags) {
+					 .defrag_copy = true,
+					 });
+
+	if (IS_ERR(fence) || !need_ccs)
+		return fence;
+
+	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_sg,
+				   (struct xe_migrate_copy_flags) {
+				   .defrag_ccs = true,
+				   });
+	if (IS_ERR(fence2))
+		dma_fence_wait(fence, false);
+	dma_fence_put(fence);
+
+	return fence2;
 }
 
 /**
@@ -1096,7 +1197,10 @@ struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
 				     struct xe_bo *bo,
 				     struct ttm_resource *res)
 {
-	return __xe_migrate_copy(m, bo, bo, res, res, false, true);
+	return __xe_migrate_copy(m, bo, bo, res, res, NULL,
+				 (struct xe_migrate_copy_flags) {
+					 .is_vram_resolve = true,
+				 });
 }
 
 /**
drivers/gpu/drm/xe/xe_migrate.h +8 -0
index 965c45889c72..e4f996942a6d 100644
@@ -11,6 +11,7 @@
 struct dma_fence;
 struct drm_pagemap_addr;
 struct iosys_map;
+struct sg_table;
 struct ttm_resource;
 
 struct xe_bo;
@@ -132,6 +133,13 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
 				  struct ttm_resource *dst,
 				  bool copy_only_ccs);
 
+struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
+					 struct xe_bo *bo,
+					 struct ttm_resource *src,
+					 struct ttm_resource *dst,
+					 struct sg_table *src_sg,
+					 bool need_ccs);
+
 struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
 				     struct xe_bo *bo,
 				     struct ttm_resource *res);
Signature · 1 line
2.34.1
reviewer
#59 lore raw

Re: [PATCH v3 17/33] drm/xe: Add xe_migrate_copy_defrag() for on-GPU defrag copies

Quoted 12 lines · Matthew Brost

Add xe_migrate_copy_defrag(), a copy helper for the upcoming page defragmentation path where a single BO has its backing reallocated at a more beneficial page order and its contents must be relocated from the old pages to the new ones entirely on the GPU.

The contents are relocated in up to two passes:

- Data pass (always): copy the data pages verbatim. The pages are mapped without the compression PAT, so the (possibly compressed) bytes are copied as-is rather than being decompressed and recompressed, and no CCS aux copy is appended.

Quoted 4 lines

- CCS pass (need_ccs): copy the CCS aux (compression) state. Both the old and new data pages are mapped with the compression PAT so the copy engine accesses the aux state indirectly, issuing one indirect -> indirect XY_CTRL_SURF_COPY_BLT; the data blit is skipped.

Just to double check, there were no issues with this flow for a compressed surface? Not 100% sure if reading from the src with uncompressed PAT will zero out the CCS state (update to uncompressed status), such that when you do the CCS pass it is already gone. Quite sure it does that for a write, but I guess not for a read, so we are fine here?

Quoted 288 lines

The two passes are issued separately because they require opposite compression PAT settings on the data page mappings (cleared for the verbatim data copy, set for the indirect CCS access), which a single mapping cannot satisfy. Both passes run in order on the same migrate queue and are captured by the returned fence.

The source pages are passed explicitly via an sg_table argument (the old, still populated backing) since they no longer match the BO's current backing. The boolean arguments to __xe_migrate_copy() are folded into struct xe_migrate_copy_flags to keep the growing set of copy variants readable.

Cc Carlos Santa <[email protected]>
Cc Ryan Neph <[email protected]>
Cc Christian Koenig <[email protected]>
Cc Huang Rui <[email protected]>
Cc Matthew Auld <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Thomas Hellström <[email protected]>
Assisted-by GitHub_Copilot:claude-opus-4.8
---
  drivers/gpu/drm/xe/xe_migrate.c | 136 ++++++++++++++++++++++++++++----
  drivers/gpu/drm/xe/xe_migrate.h |   8 ++
  2 files changed, 128 insertions(+), 16 deletions(-)
drivers/gpu/drm/xe/xe_migrate.c +120 -16
index 92d5e81ceac2..305c6e1942fb 100644
@@ -858,13 +858,39 @@ static u32 xe_migrate_ccs_copy(struct xe_migrate *m,
  	return flush_flags;
  }
 
+/**
+ * struct xe_migrate_copy_flags - Flags for __xe_migrate_copy()
+ * @copy_only_ccs: Only copy the CCS aux state, aliasing the destination data
+ * mapping to the source (used when the data pages are shared between src/dst).
+ * @is_vram_resolve: Decompression (VRAM -> system) resolve; always uses the
+ * compression PAT index for the data copy.
+ * @defrag_ccs: Defrag move pass that migrates only the CCS aux state between
+ * two distinct page sets (old @src_sg pages -> new pages). The data blit is
+ * skipped (the data was copied by the @defrag_copy pass) and both the source
+ * and destination data pages are mapped with the compression PAT so the aux
+ * state is copied indirect -> indirect.
+ * @defrag_copy: Defrag move pass that copies the data pages between two
+ * distinct page sets (old @src_sg pages -> new pages) without touching the CCS
+ * aux state. The data pages are mapped without the compression PAT so the
+ * (possibly compressed) bytes are copied verbatim rather than being
+ * decompressed and recompressed, and no CCS aux copy is appended; the aux
+ * state is migrated separately by the @defrag_ccs pass. The two passes need
+ * opposite compression PAT settings, which is why they cannot be combined.
+ */
+struct xe_migrate_copy_flags {
+	u32 copy_only_ccs : 1;
+	u32 is_vram_resolve : 1;
+	u32 defrag_ccs : 1;
+	u32 defrag_copy : 1;
+};
+
  static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
  					   struct xe_bo *src_bo,
  					   struct xe_bo *dst_bo,
  					   struct ttm_resource *src,
  					   struct ttm_resource *dst,
-					   bool copy_only_ccs,
-					   bool is_vram_resolve)
+					   struct sg_table *src_sg,
+					   struct xe_migrate_copy_flags flags)
  {
  	struct xe_gt *gt = m->tile->primary_gt;
  	struct xe_device *xe = gt_to_xe(gt);
@@ -881,20 +907,29 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
  	bool src_is_vram = mem_type_is_vram(src->mem_type);
  	bool dst_is_vram = mem_type_is_vram(dst->mem_type);
  	bool type_device = src_bo->ttm.type == ttm_bo_type_device;
-	bool needs_ccs_emit = type_device && xe_migrate_needs_ccs_emit(xe);
+	bool needs_ccs_emit = type_device && xe_migrate_needs_ccs_emit(xe) &&
+		!flags.defrag_copy;
  	bool copy_ccs = xe_device_has_flat_ccs(xe) &&
-		xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo);
-	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram);
+		xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo) &&
+		!flags.defrag_copy;
+	bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram) &&
+		!flags.defrag_ccs && !flags.defrag_copy;
 
  	/*
  	 * For decompression operation, always use the compression PAT index.
  	 * Otherwise, only use the compression PAT index for device memory
  	 * when copying from VRAM to system memory.
  	 */
-	bool use_comp_pat = is_vram_resolve || (type_device &&
+	bool use_comp_pat = flags.is_vram_resolve || (type_device &&
  			    xe_device_has_flat_ccs(xe) &&
  			    GRAPHICS_VER(xe) >= 20 && src_is_vram && !dst_is_vram);
 
+	/*
+	 * The defrag CCS pass migrates the aux state from the old pages
+	 * (@src_sg) to the new pages with both sides accessed indirectly, so
+	 * keep both src_is_pltt and dst_is_pltt set (no direct staging side).
+	 */
+
  	/* Copying CCS between two different BOs is not supported yet. */
  	if (XE_WARN_ON(copy_ccs && src_bo != dst_bo))
  		return ERR_PTR(-EINVAL);
@@ -903,7 +938,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
  		return ERR_PTR(-EINVAL);
 
  	if (!src_is_vram)
-		xe_res_first_sg(xe_bo_sg(src_bo), 0, size, &src_it);
+		xe_res_first_sg(src_sg ? src_sg : xe_bo_sg(src_bo), 0, size, &src_it);
  	else
  		xe_res_first(src, 0, size, &src_it);
  	if (!dst_is_vram)
@@ -912,7 +947,8 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
  		xe_res_first(dst, 0, size, &dst_it);
 
  	if (copy_system_ccs)
-		xe_res_first_sg(xe_bo_sg(src_bo), xe_bo_ccs_pages_start(src_bo),
+		xe_res_first_sg(src_sg ? src_sg : xe_bo_sg(src_bo),
+				xe_bo_ccs_pages_start(src_bo),
  				PAGE_ALIGN(xe_device_ccs_bytes(xe, size)),
  				&ccs_it);
 
@@ -942,7 +978,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
  		batch_size += pte_update_size(m, pte_flags, src, &src_it, &src_L0,
  					      &src_L0_ofs, &src_L0_pt, 0, 0,
  					      avail_pts);
-		if (copy_only_ccs) {
+		if (flags.copy_only_ccs) {
  			dst_L0_ofs = src_L0_ofs;
  		} else {
  			pte_flags = dst_is_vram ? PTE_UPDATE_FLAG_IS_VRAM : 0;
@@ -963,7 +999,8 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
  		}
 
  		/* Add copy commands size here */
-		batch_size += ((copy_only_ccs) ? 0 : emit_copy_cmd_len(xe)) +
+		batch_size += ((flags.copy_only_ccs || flags.defrag_ccs) ? 0 :
+			       emit_copy_cmd_len(xe)) +
  			((needs_ccs_emit ? EMIT_COPY_CCS_DW : 0));
 
  		bb = xe_bb_new(gt, batch_size, usm);
@@ -975,13 +1012,15 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
  		if (src_is_vram && xe_migrate_allow_identity(src_L0, &src_it))
  			xe_res_next(&src_it, src_L0);
  		else
-			emit_pte(m, bb, src_L0_pt, src_is_vram, copy_system_ccs || use_comp_pat,
+			emit_pte(m, bb, src_L0_pt, src_is_vram,
+				 copy_system_ccs || use_comp_pat || flags.defrag_ccs,
  				 &src_it, src_L0, src);
 
  		if (dst_is_vram && xe_migrate_allow_identity(src_L0, &dst_it))
  			xe_res_next(&dst_it, src_L0);
-		else if (!copy_only_ccs)
-			emit_pte(m, bb, dst_L0_pt, dst_is_vram, copy_system_ccs,
+		else if (!flags.copy_only_ccs)
+			emit_pte(m, bb, dst_L0_pt, dst_is_vram,
+				 copy_system_ccs || flags.defrag_ccs,
  				 &dst_it, src_L0, dst);
 
  		if (copy_system_ccs)
@@ -990,7 +1029,7 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
  		bb->cs[bb->len++] = MI_BATCH_BUFFER_END;
  		update_idx = bb->len;
 
-		if (!copy_only_ccs)
+		if (!flags.copy_only_ccs && !flags.defrag_ccs)
  			emit_copy(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0, XE_PAGE_SIZE);
 
  		if (needs_ccs_emit)
@@ -1078,7 +1117,69 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
  				  struct ttm_resource *dst,
  				  bool copy_only_ccs)
  {
-	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, copy_only_ccs, false);
+	return __xe_migrate_copy(m, src_bo, dst_bo, src, dst, NULL,
+				 (struct xe_migrate_copy_flags) {
+					 .copy_only_ccs = copy_only_ccs,
+				 });
+}
+
+/**
+ * xe_migrate_copy_defrag() - Migrate a BO's backing to a reallocated page set.
+ * @m: The migration context.
+ * @bo: The buffer object being defragmented. Both @src and @dst belong to @bo.
+ * @src: The source TTM resource (the old backing).
+ * @dst: The dst TTM resource (the freshly reallocated backing).
+ * @src_sg: Scatter-gather table of the source (old) backing pages.
+ * @need_ccs: Whether the BO has CCS aux state that must be migrated too.
+ *
+ * Used by the defrag path where a single BO has its backing reallocated at a
+ * more beneficial page order. The contents are relocated from the old pages
+ * (@src_sg) to the new pages (the BO's current backing) entirely on the GPU,
+ * in up to two passes:
+ *
+ *  - Data pass (always): copy the data pages verbatim. The pages are mapped
+ *    without the compression PAT, so the (possibly compressed) bytes are copied
+ *    as-is rather than being decompressed and recompressed, and no CCS aux copy
+ *    is appended.
+ *
+ *  - CCS pass (@need_ccs): copy the CCS aux (compression) state. Both the old
+ *    and new data pages are mapped with the compression PAT so the copy engine
+ *    accesses the aux state indirectly, issuing one indirect -> indirect
+ *    XY_CTRL_SURF_COPY_BLT; the data blit is skipped.
+ *
+ * The two passes are issued separately because they require opposite
+ * compression PAT settings on the data page mappings (cleared for the verbatim
+ * data copy, set for the indirect CCS access), which a single mapping cannot
+ * satisfy. Both passes run in order on the same migrate queue.
+ *
+ * Return: Pointer to a dma_fence representing the last copy batch, or an error
+ * pointer on failure.
+ */
+struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
+					 struct xe_bo *bo,
+					 struct ttm_resource *src,
+					 struct ttm_resource *dst,
+					 struct sg_table *src_sg,
+					 bool need_ccs)
+{
+	struct dma_fence *fence2, *fence =
+		__xe_migrate_copy(m, bo, bo, src, dst, src_sg,
+				  (struct xe_migrate_copy_flags) {
+					 .defrag_copy = true,
+					 });
+
+	if (IS_ERR(fence) || !need_ccs)
+		return fence;
+
+	fence2 = __xe_migrate_copy(m, bo, bo, src, dst, src_sg,
+				   (struct xe_migrate_copy_flags) {
+				   .defrag_ccs = true,
+				   });
+	if (IS_ERR(fence2))
+		dma_fence_wait(fence, false);
+	dma_fence_put(fence);
+
+	return fence2;
  }
 
  /**
@@ -1096,7 +1197,10 @@ struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
  				     struct xe_bo *bo,
  				     struct ttm_resource *res)
  {
-	return __xe_migrate_copy(m, bo, bo, res, res, false, true);
+	return __xe_migrate_copy(m, bo, bo, res, res, NULL,
+				 (struct xe_migrate_copy_flags) {
+					 .is_vram_resolve = true,
+				 });
  }
 
  /**
drivers/gpu/drm/xe/xe_migrate.h +8 -0
index 965c45889c72..e4f996942a6d 100644
@@ -11,6 +11,7 @@
  struct dma_fence;
  struct drm_pagemap_addr;
  struct iosys_map;
+struct sg_table;
  struct ttm_resource;
 
  struct xe_bo;
@@ -132,6 +133,13 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m,
  				  struct ttm_resource *dst,
  				  bool copy_only_ccs);
 
+struct dma_fence *xe_migrate_copy_defrag(struct xe_migrate *m,
+					 struct xe_bo *bo,
+					 struct ttm_resource *src,
+					 struct ttm_resource *dst,
+					 struct sg_table *src_sg,
+					 bool need_ccs);
+
  struct dma_fence *xe_migrate_resolve(struct xe_migrate *m,
  				     struct xe_bo *bo,
  				     struct ttm_resource *res);